Hi there. I am not yet a full-fledged researcher in quantum information, but have been in the area for several years and will be defending my Master's thesis on resource theories in quantum information in a couple of weeks. Since I am in a math department, however, I feel qualified to answer your question.

The math you need to know for quantum information depends heavily on what you want to do. There are roughly three types of people who do quantum information: computer scientists, mathematicians, and physicists. I hope others will forgive me for this simplification. The computer scientists study what can be done with a quantum computer should one become available to us. This is the least intensive in terms of mathematical prerequisites, though it is no less challenging than other areas of quantum information (and, indeed, I find it the hardest part). To understand the basics of these quantum algorithms, linear algebra (also called matrix algebra) is often enough. More advanced algebra is a definite plus for this subject and for the study of algorithms in general.

The mathematician studying quantum information needs to understand matrix analysis. This requires matrix algebra (unsurprisingly), but also quite advanced knowledge of calculus. What they call calculus in school is really not quite what is necessary, however. The proper name (according to mathematicians, at least) is mathematical analysis, and this subject feels a bit more like high level algebra (though you still compute derivatives and such). I argue that functional analysis is essential to the proper understanding of mathematical quantum information and of quantum mechanics in general (otherwise you will likely not appreciate the correspondence between the Schroedinger and Heisenberg pictures of quantum mechanics). Entanglement witnesses, for example, may be seen to exist as a consequence of the Hahn-Banach Theorem.

The physicist will require the most of what you call calculus. The physicist's job in quantum information is to figure out how to implement all the wacky quantum information protocols devised by the cunning computer scientists and mathematicians. This means describing physical systems using quantum mechanics, which means solving Schroedinger's equation. Schroedinger's equation is a partial differential equation in many complex variables, which means you're probably going to be programming a computer to do it for you. That means you would need to know partial differential equations, numerical analysis, and at least one programming language (MATLAB or C++ are the typical choices, though I have gotten away with only Mathematica... probably not for much longer). Each subject has prerequisites such as the ones you have already discussed.

This all sounds daunting, no doubt. Please keep in mind that it generally takes years to gain all this knowledge, and it doesn't seem like so much when you are spending a few years on it. Moreover, the skills you gain in preparing for a career in quantum information have broader application. The mathematics used to understand quantum information are quite important to companies such as Google. The PageRank algorithm, for example, is understandable to most quantum information scientists including Michael Nielsen -- http://michaelnielsen.org/blog/using-your-laptop-to-compute-pagerank-for...

Some parting advice: if you're serious about quantum information, find someone to coach you. Go to a university that has a strong research group in quantum information, make friends with the other students (so you can study with them), do more research projects, and go to seminars. Best of luck, and I hope to see you at future conferences!