Question

I know about Wikipedia and MacKay’s Information Theory, Inference, and Learning Algorithms (is it appropriate as textbook?). A textbook starting with Shannon’s entropy and going through Conditional entropy and Mutual information is sought… Any idea? If you are following such a course at your university, which textbook is used?

Thanks.

Answer 1

I used the following textbook during my studies in CS at EPFL. IMO, it’s well written, with good explainations, and covers more than enough for an introduction to the domain.

Elements of Information Theory

EDIT: For further reading, here are some other readings that my professor did recommend. I did not read them (shame on me), so I can’t say if they’re good or not.

1. R. G. Gallager, Information Theory and Reliable Communication, Wiley, 1968.
2. D. MacKay, Information Theory, Inference & Learning Algorithms, Cambridge University Press, 2008. (you already mentioned it)
3. I. Csiszar and J. Korner, Information Theory: Coding Theorems for Discrete Memoryless Systems, Akademiai Kiado, 1997.
4. C. E. Shannon, The Mathematical Theory of Communication