There's no reason why amateurs can't make worthwhile research contributions in mathematics. It has happened many times in the past, and I know of several cases today. I don't have time to offer a lot of personal advice and guidance, but I figured I'd post some general advice here. I've aimed it at people who think they've already solved famous problems, since those are the sort that typically write, but it should be equally useful for people with more modest aspirations. I focus on the mechanics of how to do literature searches, write papers, and publish them, because I have less to say about the deeper issue of how to do research. None of this advice is specific to amateurs, but professionals already learn all these things from their advisors.
Develop a track record
If you appear out of nowhere claiming to have solved a famous open problem, nobody will pay any attention. In principle you might be right, but many people claim to have done this and virtually all of them are wrong. If you want anyone to take your work seriously, you need to develop a track record that separates you from the cranks.
The easiest way to do this is to publish some other papers. They don't have to be deep or profound, just to show that you can make a serious, uncontroversial contribution to an area some other mathematicians care about. If you can't in fact do this, and all you can do is write controversial papers, then you should start worrying that you're deluding yourself about the quality of your papers.
One common misconception is that other researchers have an obligation to evaluate your work, and that it's unprofessional and unfair of them to ignore it. There's a kernel of truth in that, since once you've got a good track record and are circulating a clear manuscript your work shouldn't be entirely ignored (it might still be reasonable to dismiss it as nonsense, if that happens to be the case). However, it's ridiculous to argue that all proposed solutions to famous problems must either be accepted as true or be refuted to the satisfaction of the author. The mathematical community couldn't function under such a constraint.
Along the way to developing a convincing track record you'll most likely realize that your purported solution was incorrect or incomplete, but that's another side benefit.
Use standard terminology and language
It's amazing how many people introduce their own terminology or notation, on the grounds that it's better than the existing options. That's usually debatable, but even when it's objectively true doing this will just make it less likely that anyone will actually read your work. If you must introduce new notation, explain clearly how it is related to the standard notation.
Use MathSciNet to search for references
MathSciNet is the online version of the Mathematical Reviews journal. It's a database of virtually every published mathematical paper from the last fifty years, with brief reviews. Unfortunately a subscription is required, but you can access it at any college or university library. They will likely have it on paper too, but that is harder to search.
Using MathSciNet you can look up reviews of specific papers (the reviews typically summarize the contents), find all papers by a given author, search by topic, find papers whose titles or reviews include a certain word or phrase, find all reviews that refer to a certain paper, etc. It's a tremendously powerful tool for searching the mathematical literature.
If you're not using MathSciNet, and you're not already an expert on the literature, then you have no idea what's out there. It's important to spend a few hours searching to see what sorts of references you can dig up. (Then look them up and study them, of course.)
Write a clear introduction to your paper
Most people write terrible introductions to their first papers. I certainly did: the first research paper I ever submitted was rejected, and I felt like the referee's comments didn't do it justice. Things I considered strengths of the paper were cited as weaknesses, and the referee just didn't seem to share my point of view. A large part of the problem was with the introduction. If you want people to look at things from a certain point of view, or to appreciate certain aspects of your work, then you need to present this clearly at the start of the paper. I rewrote the introduction and submitted the paper to a comparable journal, where it was accepted. In hindsight I don't think the revised introduction is that great either, but the one it replaced was terrible, and the revisions at least put the results in context.
What you need to remember is that if you're writing a paper on something, you've almost certainly spent more time thinking about this specific question than the referees have, even if they are more broadly knowledgeable in the field. You need to explain what your paper is on, what it's related to and why the reader should care, what sorts of techniques you use, etc. Think of the introduction as laying out the case for why the paper should be published and, later, why it should be read. (However, don't brag or exaggerate, and don't lay out this case explicitly. Do it implicitly just by explaining the ideas and context.)
Follow established conventions for mathematical writing
Part of the reason why it is hard to write mathematics clearly is that things that seem obvious to the author may be quite mysterious to readers. You know perfectly well what you meant, and it is hard to see it through someone else's eyes. An important step in avoiding this problem is to get feedback from other people (see below), but fortunately there are a number of stylistic conventions that can help. In particular, they will help you avoid common sources of ambiguity, confusing or difficult to read constructions, etc.
You can find some online guides to mathematical writing here, here, and here. I urge you to read all three, and to think about them carefully. It can make a big difference in the readability of your papers.
You should also make sure your paper is in a reasonably standard format. In particular, it should include a one-paragraph abstract that briefly summarizes the contents. Theoretical mathematics papers generally do not include a conclusions section at the end. (It's not out of the question, but don't force your manuscript into a generic "science paper" format.)
Get feedback on your manuscript from friends or collaborators
As I mentioned above, you need to get feedback from friends, or from colleagues with a good reason to study your work carefully (for example, collaborators of yours). This is a crucial step for clearing up ambiguities, figuring out where more detail is needed, etc., and it's something I always do. I'm skeptical that anybody can publish well-written research papers without getting feedback from others.
The referees don't play this role. If you haven't already fixed most of these problems before submission, you're making things unnecessarily hard for the referees and thereby increasing the chances of rejection.
If you can't get anyone to read your paper now, what makes you think anyone will want to read it after publication? That's not an entirely fair question, because you might just not know anyone interested in the topic even if there are many such people in the world. However, it's something worth thinking about.
Unfortunately I don't have time to give this sort of feedback to strangers. Friends are typically the easiest source of feedback, especially if you can return the favor. Be sure to ask them to give criticism. (You want them to say "I have no idea what you're talking about in Section 2," not "Wow, Section 2 was really deep and impressive.")
It's not as good as getting feedback from other people, but one helpful thing is to set your manuscript aside for a few months and then take a look at it. Often things that seemed clear when you were writing them will already have started to look cryptic, in which case you should rewrite them more clearly.
Submit papers to journals professionally
If you are going to submit a paper to a journal, you should do a professional job of it. If possible, you should write your paper using LaTeX (the TeX users group has information). That's not necessary, but it can't hurt to make it look professional, and it means the journal won't need to retype your paper if it is accepted for publication (which can introduce errors). Almost all research papers in mathematics use LaTeX.
When you are ready to submit the paper, keep the following in mind:
- Choose a journal that publishes papers like yours (in terms of topic, style, length, importance, etc.). If you're not sure, go to the library and flip through some back issues until you find one that looks suitable. A good starting point is the journals that published the papers you cite. Each issue of a journal should explain where to send submissions.
- You cannot submit the same paper to two journals at once. If you are caught doing this, the editors will be angry with you and it will be hard to get your paper published. (You should also not submit simultaneously to two different editors at the same journal.)
- If your work makes use of computer programs you wrote, refers to unpublished manuscripts, etc., you should provide copies for the referee.
- If a couple of weeks go by after submission and you haven't heard anything, it's reasonable to check whether your paper was received.
- When a paper has been sent to referees, you may well have to wait months to hear back. Once six months have passed it may be reasonable to inquire about the status of your submission, but you should wait at least that long. For lengthy or complicated papers refereeing may take even longer.
- Often the referee's report will include comments for the author, ranging from pointing out typos and minor errors to substantive suggestions. This can be quite useful, even if you are unhappy with the referee's recommendation.
- If your paper is rejected, you can submit it to another journal, but you should not resubmit a revised version to the same journal unless they explicitly invite you to do so (which rarely happens).
- Sometimes a paper might be rejected just because it doesn't meet the journal's high standards, not because there's anything really wrong with it. However, if a referee makes any substantive criticisms, it's almost always a mistake to submit the paper to another journal without making some revisions.