Classical information

Classical information was first defined rigorously by [[Claude Shannon]]. Information is equal to how much communication is needed to convey it. Roughly speaking, if one has a list of possible messages you might want to convey, then the information of the messages is how much communication is required to tell someone which of the messages from the list you wish to communicate. We denote the messages by a [[random variable]] $X$. This is a list of messages $\backslash \{x\_1,\; x\_2,\; x\_3...x\_m\backslash \}$ each one occuring with probability $\backslash \{p(x\_1),\; p(x\_2),\; p(x\_3)...p(x\_m)\backslash \}$. We denote this probability distribution by $P\_X$. Shannon showed that the number of [[bits]] needed to convey which message occurs is given by the [[Shannon entropy]] $H(X)=-\backslash sum\; p(x)\; \backslash log\; p(x)$. Essentially, he showed that if one has $n$ messages, then one can [[compress]] this information onto a space of dimension just over $2^\{n\{H(X)\}\}$ such that with high probability the information is sent faithfully. ==Reference== * C. E. Shannon, [http://cm.bell-labs.com/cm/ms/what/shannonday/shannon1948.pdf A mathematical theory of communication]''[[Bell System Technical Journal]]'', vol. 27, pp. 379–423 and 623–656, (July and October, 1948) [[Category:Classical Information Theory]]