Page plan · 15 pages

Written for: Engineering and computer science students in a reading group

1. 1

   A Mathematical Theory of Communication, Shannon 1948

   • Today we read Shannon's 1948 paper, A Mathematical Theory of Communication.
   • It appeared in the Bell System Technical Journal and founded information theory.
   • We treat each slide as a blueprint sheet: one component of the theory at a time.
   • Goal: leave able to explain entropy, capacity and the coding theorem.
2. 2

   Route through the paper: from noisy channels to the coding theorem

   • The route starts with the problem of noise and ends at the noisy-channel coding theorem.
   • Along the way we define information, entropy, sources with memory and capacity.
   • Each stop adds a tool that the coding theorem needs.
   • Later sheets cover limits of the theory and its descendants in practice.
3. 3

   The engineering problem: messages corrupted by noise

   • Engineering problem: reproduce at one point a message selected at another point.
   • Noise on a channel corrupts symbols, so the receiver may decode wrongly.
   • Before Shannon, more reliability was assumed to need slower or more costly transmission.
   • Shannon ignores meaning and focuses on what can be sent reliably.
4. 4

   The core idea: information as choice among possible messages

   • Information measures the choice among possible messages, not their meaning.
   • A message from a larger set of equally likely options carries more information.
   • The actual message is semantically irrelevant to the engineering problem.
   • Probabilities of messages are what the mathematical theory is built on.
5. 5

   Shannon's general communication system, box by box

   • The system has an information source, transmitter, channel, receiver and destination.
   • The transmitter encodes the message into a signal suited to the channel.
   • A noise source corrupts the signal inside the channel.
   • The receiver decodes the received signal back into a message for the destination.
6. 6

   Measuring information in bits with entropy

   • Entropy H = -Σ p log p measures average uncertainty of a source.
   • With base-2 logarithms the unit is the bit.
   • A fair coin flip has exactly 1 bit of entropy.
   • Entropy is highest when outcomes are equally likely and zero when the outcome is certain.
7. 7

   Sources with memory: English needs fewer bits once context is known

   • Each added level of context lowers the estimated entropy of English.
   • With a 27-letter alphabet, entropy falls from 4.76 bits per letter at F0 to 2.14 at the word level.
   • With a 26-letter alphabet, it falls from 4.7 to 2.62 bits per letter.
   • The drop shows English is highly predictable, so it holds redundancy.
   • These figures come from Shannon's 1951 follow-up paper, not the 1948 paper.

   Line chart: Entropy of printed English as more context is counted (Shannon, 1951). Source: Prediction and Entropy of Printed English (C. E. Shannon, Bell System Technical Journal 30:1, 1951) · checked 2026-10-10

8. 8

   Three uses of entropy: compression, redundancy and rate

   • Compression: entropy is the lowest average bits per symbol for lossless coding of a source.
   • Redundancy is the gap between the maximum possible entropy and the source's actual entropy.
   • Rate is how many bits per second the source produces after ideal coding.
   • These three ideas together tell us how much a source can shrink.
9. 9

   Channel capacity: the most a noisy line can carry

   • Channel capacity C is the maximum rate of information a channel can carry reliably.
   • It is defined by maximizing mutual information between input and output.
   • Noise lowers capacity, but it does not reduce it to zero.
   • For a band-limited Gaussian channel, capacity depends on bandwidth and signal-to-noise ratio.
10. 10

    Why clever coding beats sending everything twice

    • Repeating every bit cuts the rate and still leaves some chance of error.
    • Sending more copies lowers errors only by driving the rate toward zero.
    • Shannon showed that well-designed codes over long blocks can do far better.
    • Block codes spread redundancy cleverly to correct errors at a useful rate.
11. 11

    The noisy-channel coding theorem, stated plainly

    • If the rate R is below capacity C, codes exist with arbitrarily small error probability.
    • If R is above C, reliable communication is impossible.
    • The result holds for the noisy discrete channel described in the paper.
    • Reliability is achieved by coding over long blocks, not by raising power alone.
12. 12

    What the theorem guarantees and what it leaves open

    • The theorem guarantees that good codes exist below capacity.
    • It does not give an explicit, practical construction of those codes.
    • Long block lengths may bring delay and decoding complexity.
    • Finding codes that approach capacity became a major research problem afterward.
13. 13

    Assumptions behind the theory and where they break

    • The theory assumes known probability models of the source and channel.
    • It models stationary, ergodic sources, which real data may not be.
    • It ignores meaning, so it cannot judge whether a message is useful.
    • Real systems face delay limits, changing channels and finite code lengths.
14. 14

    From the 1948 paper to modems, phones and deep-space links

    • Modems use error-correcting codes to push data rates closer to channel capacity.
    • Phone and wireless systems rely on coding and compression ideas from the paper.
    • Deep-space links use strong error correction because signals are very weak.
    • Modern codes are judged by how near they come to Shannon's limit.
15. 15

    Closing sheet: questions for the reading group

    • Why does Shannon leave meaning out, and what does that gain us?
    • Where does the entropy estimate for English depend on modeling choices?
    • How would you design a code at a rate just below capacity?
    • Which assumption of the theory is most fragile in a system you know?