The Noisy Channel. Why Reliable Communication Is Always Possible

Post 2 of 5 in the series Shannon’s Blueprints — Five Ideas That Built the Foundation of AI

A signal passing through a noisy channel, distorted in transmission and reconstructed through redundancy and error correction. Image generated by DALL-E.

Introduction

Post one established what information is. Shannon defined it as resolved uncertainty, measured in bits. The formula runs inside every language model trained today, in the form of cross-entropy loss.

This post moves from the source to the channel. A source produces messages. A channel carries them. The question post one left open is what happens when the channel is imperfect, when noise corrupts the signal before it arrives.

Shannon’s answer to that question surprised everyone in 1948. It still surprises people today. And it has a direct line to three foundational ideas in modern AI.

This is post two of five. Each post takes one of Shannon’s core ideas, traces how he derived it, and connects it directly to modern AI. The lineage is exact.

What Noise Actually Is

In Shannon’s framework, noise is defined precisely. It is the probability that a symbol arrives at the receiver different from what was sent. A channel takes an input symbol and produces an output, and the transition probabilities between them fully describe its noise characteristics.

The simplest model is the binary symmetric channel. A sender transmits a single bit, 0 or 1. The channel flips it with probability p and passes it through unchanged with probability 1 − p. If p = 0.1, one in ten bits gets flipped. If p = 0.5, the channel is pure noise, where the output is completely unrelated to the input.

This is an abstraction, but a useful one. Thermal noise in a wire, interference in a radio signal, a scratched optical disc, a corrupted storage medium, all of these reduce to the same mathematical structure. A symbol goes in. A (possibly different) symbol comes out.

The engineering question before Shannon was how reliably information could be transmitted over a noisy channel. The answer that stood in 1948 was that to transmit more reliably, transmission speed had to slow down. More reliability meant slower transmission. Less noise tolerance meant faster transmission. Engineers accepted this as a fixed constraint of physics.

Shannon showed that this trade-off was an illusion. Reliable communication at a fixed rate is always possible below capacity.

Noisy Channel Coding Theorem

Shannon defined channel capacity C as the maximum rate at which information can be transmitted reliably over a given channel. For the binary symmetric channel with flip probability p, the capacity is 1 − H(p), where H(p) = −p log₂ p − (1 − p) log₂(1 − p) is the entropy of a single bit flip.

Shannon’s theorem states that for any noise level and any transmission rate R below the channel capacity C, there exist encoding and decoding schemes that allow the probability of error to be made arbitrarily small. Transmit below capacity, and reliable communication is achievable. Transmit above capacity, and reliable communication is impossible.

The capacity C defines a sharp theoretical threshold with two sides.

The surprising part was the first half. Before 1948, engineers believed that reliable communication required the error probability to approach zero only as the transmission rate approached zero, meaning the transmission rate had to drop indefinitely to get rid of errors entirely. Shannon proved the opposite. At any rate below C, reliable communication is achievable over even a very noisy channel. Redundancy does the job, and that redundancy is introduced through encoding.

Instead of sending each bit independently and hoping it survives the channel, a block of k bits is encoded into a longer codeword of n bits, where the extra n − k bits carry structured redundancy. The receiver, seeing a corrupted version of the codeword, uses the redundancy to deduce what was actually sent. Shannon proved that for any channel, such codes exist that make this work, pushing error rates as close to zero as desired, as long as the transmission rate R = k/n, the ratio of information bits to total bits sent, stays below C.

The Proof That Changed How Engineers Think

Shannon’s proof deserves attention beyond the result itself. He showed that good codes must exist by proving that a randomly constructed code performs well on average. If the average code achieves near-zero error probability, at least one specific code must be at least as good. Shannon proved existence without constructing anything. Finding efficient codes in practice was a separate problem that took decades of coding theory to solve.

This was an unusual style of argument for engineers in 1948. Engineers were used to building things they could describe explicitly. Shannon showed that a description was not required.

This probabilistic existence proof became a template for later theoretical results in computer science and mathematics. The principle that random objects have good average-case properties, and therefore good specific instances must exist, appears throughout complexity theory, combinatorics, and the theory of learning. Shannon introduced it to the engineering world.

Why Repetition Is the Wrong Approach

Suppose a channel flips one in ten bits. The naive approach is repetition. Send each bit three times. Take a majority vote. Two out of three copies agree on the correct value, and the probability of a decoding error drops significantly. Send each bit five times, and the error rate drops further.

This feels like the right approach. But it is expensive. Sending each bit five times means the effective transmission rate is one fifth of the channel’s raw speed. To get the error rate arbitrarily close to zero by repetition alone, the transmission rate must go to zero. This matched what engineers expected. Reliability comes at the cost of speed.

Shannon showed this intuition is wrong. Repetition is an inefficient code. With smarter codes, ones that spread redundancy across longer blocks rather than repeating individual bits, it is possible to achieve arbitrarily small error rates while transmitting at a fixed rate below capacity. The capacity of a binary symmetric channel with p = 0.1 is about 0.53 bits per channel use. That means it is possible, in principle, to transmit at 53% of the raw speed with near-zero errors. Repetition codes cannot get anywhere near this. Capacity-approaching codes can.

The reason the result was so counterintuitive is that it requires thinking about long blocks of symbols jointly. The redundancy that makes reliable communication possible is distributed across the whole block. Shannon’s framework demanded a shift from thinking about signals to thinking about probability distributions over sequences.

From Theory to Practice. The Codes Shannon Made Possible

The simplest form of redundancy is the parity bit. Given a block of data bits, one extra bit is added so that the total number of 1s is always even. If a single bit gets flipped during transmission, the parity no longer holds and the receiver knows an error occurred. The limitation is fundamental. A parity bit can detect a single error but cannot locate it, so the receiver knows something went wrong but not which bit to fix. Worse, if two bits flip, the parity still checks out and the error goes undetected entirely.

The solution is more parity checks, each covering a different subset of bits. With enough overlapping checks, the receiver can triangulate exactly which bit was corrupted and correct it. Richard Hamming at Bell Labs formalized this in 1950, producing codes that correct any single-bit error in a block. Shannon had proved such codes must exist. Hamming showed how to build them.

The remaining problem was efficiency. Simple codes like Hamming’s require a lot of overhead for modest error correction capability. Getting close to the Shannon limit means transmitting at high rates with very low error. Achieving that requires codes that operate over very long blocks with many lightweight, overlapping parity checks rather than a few heavy ones. Low-Density Parity-Check codes, developed by Robert Gallager in the 1960s but largely forgotten until their rediscovery in the 1990s, take exactly this approach. The sparse structure of their parity checks keeps decoding computationally tractable even at large block sizes, which is why they can operate within a fraction of a decibel of the Shannon limit.

Reed-Solomon codes solve a different version of the problem. Rather than operating on individual bits, they operate on groups of bits treated as symbols in a larger alphabet. This makes them particularly effective against burst errors, where a scratch on an optical disc or a bad sector on a hard drive corrupts a contiguous stretch of data. A Reed-Solomon decoder can reconstruct the original data as long as enough uncorrupted symbols remain. Every CD, DVD, and QR code uses Reed-Solomon error correction. So does deep-space communication with NASA probes.

These codes are the practical realization of what Shannon proved was possible. Redundancy enabling reliable communication under noise, the core of his theorem, also appears inside the AI systems that those pipelines feed.

The AI Connection. Dropout Regularization

Dropout is a regularization technique for neural networks (Srivastava et al., 2014). During training, each neuron is randomly set to zero with probability p on every forward pass. At inference, all neurons are active and weights are scaled to compensate.

The stated motivation was to prevent co-adaptation, where neurons become over-reliant on each other. A network trained with dropout cannot rely on any single neuron being present, so it is forced to learn distributed representations. The result is a more robust network that generalizes better.

This can be understood through Shannon’s lens. Dropout injects noise directly into the computation. Each forward pass is a degraded version of the network, where parts of the signal are removed with probability p. The network is being trained under conditions analogous to a noisy channel.

Shannon showed that reliable communication over noise requires redundancy. Information must be encoded in a way that survives corruption. Dropout induces exactly that pressure. Representations cannot be concentrated in a single neuron, because that neuron may disappear. The network learns to spread information across many neurons, creating redundancy that allows the signal to survive.

There is a second effect worth noting. By removing neurons on each pass, dropout reduces the effective capacity of the network during training, forcing the model to operate under constraint rather than relying on a fixed high-capacity pathway to memorize the data.

In Shannon’s framework, reliable communication requires operating below channel capacity. The constraint is what makes reliable transmission possible in the presence of noise. Dropout introduces a different kind of constraint, one that limits the model’s ability to overfit. In both cases, the constraint discourages high-capacity solutions and favors representations that remain stable under perturbation.

The AI Connection. Attention Heads and Distributed Redundancy

A transformer processes input through multiple attention heads operating in parallel. Each head computes its own attention pattern over the sequence, and their outputs are combined before passing to the next layer.

A common explanation is that different heads specialize, with one capturing syntax, another semantic similarity, and another positional structure. This is often true, and yet it is an incomplete picture.

Empirical work on models like GPT-2 shows that many attention heads are highly redundant. Large fractions can be removed with minimal performance impact on certain tasks. In some cases, up to 90% of heads can be pruned with negligible degradation (Michel et al., 2019). Multiple heads frequently encode overlapping information in slightly different ways.

This redundancy reflects the same principle Shannon identified. When a system must operate under uncertainty, robust representations are distributed rather than localized. A transformer does not know in advance which patterns will matter for a given input. By processing the sequence through multiple heads, it builds overlapping views of the same signal.

When some heads contribute little useful information for a particular input, others still carry the signal forward. The representation degrades gracefully rather than collapsing. The structural response mirrors Shannon’s insight even without a literal noisy channel. Redundancy makes the system robust to variation, uncertainty, and partial failure.

The AI Connection. Error Correction in Data Pipelines

The connection between Shannon’s work and error correction is direct. The codes used to store and transmit the data that trains modern AI systems are descendants of the coding theory his work made possible.

Every hard drive and SSD uses error-correcting codes, typically Reed-Solomon or LDPC variants, to recover from bit errors during storage and retrieval (MacKay, 2003). Without them, large-scale training would operate on silently corrupted data, with no way to detect or recover from errors.

Every network packet carrying training data between machines relies on error detection and recovery. Protocols like TCP ensure reliable delivery over an unreliable IP layer through acknowledgment and retransmission. The packet either arrives intact or is sent again until it does.

Training runs for large models consume petabytes of data across distributed systems over weeks or months. Storage devices, network links, and memory systems are all, at the physical level, noisy channels. Shannon’s theory is what makes it possible to treat them as reliable. Without it, large-scale training would fail unpredictably, with corrupted gradients and undetected data loss.

Conclusion

Shannon’s noisy channel theorem established something that felt impossible. Reliable communication over any channel, at any noise level, as long as the transmission rate stays below capacity. The key was structured redundancy distributed across long blocks, a fundamentally different approach from naive repetition.

Shannon did not just solve communication. He identified a general pattern. Systems that operate under noise become reliable through structured redundancy.

Dropout trains models under conditions analogous to a noisy channel, forcing representations that survive when neurons disappear. Multi-head attention builds overlapping views of the same signal, degrading gracefully when individual heads contribute little. Error-correcting codes ensure that the data those models train on arrives intact.

Shannon solved the channel problem in 1948. The channel is everywhere.

The next post moves to compression and prediction. Shannon ran experiments where he had people guess the next letter of English text, and showed mathematically that the better a system predicts, the better it can compress. That equivalence is the most direct line to large language models.

References:

Shannon, C.E. (1948). “A Mathematical Theory of Communication.” Bell System Technical Journal, 27(3–4), 379–423, 623–656. https://doi.org/10.1002/j.1538-7305.1948.tb01338.x

Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. (2014). “Dropout: A Simple Way to Prevent Neural Networks from Overfitting.” Journal of Machine Learning Research, 15(1), 1929–1958. https://www.jmlr.org/papers/volume15/srivastava14a/srivastava14a.pdf

Michel, P., Levy, O., & Neubig, G. (2019). “Are Sixteen Heads Really Better than One?” Advances in Neural Information Processing Systems, 32. https://arxiv.org/abs/1905.10650

MacKay, D.J.C. (2003). Information Theory, Inference, and Learning Algorithms. Cambridge University Press. https://www.inference.org.uk/mackay/itila/