Dangers of proof by contradiction (advice for beginners)

Proof by contradiction is a powerful mathematical technique: if you want to prove X, start by assuming X is false and then derive consequences. If you reach a contradiction with something you know is true, then the only possible problem can be in your initial assumption that X is false. Therefore, X must be true.

Unfortunately, this proof technique can really cause problems for beginners. Typically, what happens is that the proof starts off quite reasonably, and then gets lost in a maze of complexity. Somewhere in the confusion, a mistake is made, which leads to a contradiction. Then it looks like the proof is done, but unfortunately the contradiction has nothing to do with the initial assumption, and comes solely from the mistake in the middle. Even if someone points out the mistake and the author tries to correct it, the same thing can happen again (either with a new mistake in the corrected version, or with a different mistake that was already present in the first attempt).

The upshot is that proofs by contradiction aren't trustworthy unless everyone can be confident that the contradiction isn't coming from a mistake. How can one create such confidence? Unfortunately, I don't have much of an answer. Two things are obvious: work out the details carefully and write clearly. Beyond that, all I can do is offer some advice about structuring a proof by contradiction.

The tricky thing is that it is hard to test statements within a proof by contradiction for plausibility. If they sound weird, or even false, is that because you've made a mistake, or because you've reached the contradiction? To avoid this issue, it's often best to try to build up as much of the framework for the proof as possible outside of the actual proof by contradiction. For example, often the proof will use a number of different lemmas. If you can, I recommend stating and proving them first, before making the initial assumption one hopes to disprove. Sometimes it may take some work to disentangle the lemma statements from that assumption, but it can be well worth it, since then the lemmas end up as statements that are really supposed to be true and can be checked and evaluated independently of the rest of the proof. If you make the actual proof by contradiction as short and simple as possible, then it will help you avoid mistakes and help other people check your work.