The Information Bottleneck

From a 1999 compression principle to a contested theory of why deep learning works

A wide input compressed through a narrow channel, where most detail is discarded and only the part that predicts the target passes through. Image generated by DALL·E.

Introduction

In 1999, Naftali Tishby, Fernando Pereira, and William Bialek asked a question. Given some input data, which parts of it matter for prediction, and which parts can be discarded? They proposed a framework called the information bottleneck and presented it at a conference on communication and computing.

The Shannon series on this blog has argued that modern AI runs inside mathematical structures designed by Claude Shannon. An earlier post in Shannon’s series showed that prediction and compression are the same operation. A good predictor is also a good compressor. But that argument stopped at the level of sequences. It did not explain what a useful internal representation should preserve. The information bottleneck takes the next step. It asks what information a representation keeps from the input, and what information it throws away.

The method began as a tool for clustering and representation learning. Years later, it was proposed as an explanation for why deep neural networks work. The proposal attracted attention, followed by criticism and debate. The bottleneck framework itself endured. What remained controversial was whether neural networks actually train according to the dynamics the theory proposed.

This post is organized around a set of questions. They trace what the bottleneck is, why it was connected to deep learning, and what survived the dispute.

What problem does the information bottleneck actually solve?

Compression throws something away. The question is what. A photograph saved as a small file loses fine detail. A summary of a long report loses most of the sentences. In both cases someone decided which losses were acceptable. That decision is the hard part, and for a long time people made it by hand.

Information theory had a formal version of this problem. It was called rate-distortion theory, developed by Claude Shannon in the 1950s. Rate is how many bits the compressed version uses. Distortion is how far the compressed version is from the original. The theory describes the best possible trade-off between the two.

But the theory does not say what distortion means. A person has to choose a distortion measure first. For images that might be squared pixel error. For sound it might be something else. The measure comes first. The theory runs after that choice. Pick a poor measure and compression keeps the wrong things.

Tishby and his coauthors changed the problem by introducing a second variable. Call the data X and the thing worth predicting Y. For face images, X is the pixels and Y is the name of the person. For speech, X is the sound and Y is the word.

Relevance no longer comes from a person choosing a distortion measure. It comes from Y. The optimal compressed version of X keeps the information that X carries about Y and discards the rest. The distortion measure does not come first anymore. It comes from the joint statistics of X and Y.

The method turns this idea into an optimization problem. It looks for a compressed representation, called T, that solves one optimization problem. Make the mutual information between T and X small, which forces compression. Keep the mutual information between T and Y large, which preserves prediction. The two goals pull against each other.

Written as one objective, the method maximizes I(T;Y) − β I(T;X). The number β sets the exchange rate between prediction and compression. A large β throws away more information. A small β keeps more of it. As β increases, a curve appears. Each point on the curve is the best trade-off for that setting. Tishby and his coauthors derived a set of self-consistent equations that trace this curve.

There is an older name for what T becomes near the prediction end of that curve. A sufficient statistic is a summary of data that keeps everything relevant to a question and nothing more. Classical statistics defined it only for narrow cases, distributions with a fixed mathematical form. The information bottleneck extends the idea to any joint distribution. T is the closest thing to a sufficient statistic that a given level of compression allows.

The connection to the earlier post is direct. Predicting well and compressing well are the same task. The bottleneck adds the missing piece. It says which information a representation should keep when it cannot keep all of it.

Why would the information bottleneck describe a neural network?

A neural network processes input one layer at a time. Pixels enter at the first layer. The last layer outputs a class label. Each layer in between holds an intermediate representation. The pixels at the input and the label at the output are fixed by the task. The network learns the representations in the middle during training.

Naftali Tishby and Noga Zaslavsky made the connection in 2015. They proposed that each hidden layer of a network is a candidate T in the bottleneck objective. The input X enters the first layer. The label Y appears at the output. Each layer in between produces its own representation.

Each representation has some mutual information with X and some mutual information with Y. Those two numbers place the layer at a single point. Plot the points and each layer becomes a dot. Each network becomes a sequence of dots. The plane that holds those dots is called the information plane.

The architecture itself limits what can happen in this plane. Information moves from input to output through a chain of transformations. Each transformation depends only on the previous layer. Information theory contains a basic result called the data processing inequality. A function of a variable cannot carry more information about a target than the variable already carries.

As information moves through the network, information about X can only decrease from layer to layer. Information about Y cannot increase from layer to layer either. A layer cannot create relevance that earlier layers lost. It can only preserve information or discard it.

The information plane has a boundary. No layer can sit above the curve set by the joint statistics of X and Y. The bottleneck curve, traced as β increases, sets the upper limit. Every real layer sits on or below it.

The proposal turned a vague idea into a measurable one. Researchers had long said that deep networks learn representations of increasing abstraction. The information plane made the claim measurable. A layer that approaches the bottleneck curve keeps the information about Y that prediction needs and discards information about X that prediction does not need. A layer that moves away from the curve does the opposite.

Training, if it is doing useful work, should push each layer toward the bottleneck curve. A framework that began as a clustering method had become a way to test neural networks. A network with finite layers and finite data was supposed to approximate the optimal trade-off described by the bottleneck. Whether real networks actually do this became the next question.

What did the two-phase picture claim about deep learning?

Ravid Shwartz-Ziv and Naftali Tishby published a paper in 2017 titled Opening the Black Box of Deep Neural Networks via Information. The paper proposed a mechanism for why deep learning works. The mechanism appeared on the information plane.

They trained a small fully connected network with seven layers, narrowing from twelve units to one on a synthetic classification task. After each training step they estimated, for every layer, the mutual information with the input and the mutual information with the label. They plotted the trajectories. The picture they reported looked the same across many runs.

Each layer started near the origin of the plane, with little information about either variable. During the first part of training, the layers moved up and to the right, gaining information about the label and about the input at the same time. They called this the fitting phase. It lasted a few hundred steps.

Then something changed. The layers stopped gaining information about the input and began losing it, while information about the label stayed high. The trajectory bent leftward on the information plane. They called this the compression phase. It lasted thousands of steps, longer than the fitting phase.

The paper went further than describing the trajectory. It tied the compression phase to generalization. A network that overfits its training data keeps too much information about the input, including information that does not help predict the label. Compression throws that information away.

The paper proposed that most of training time is spent compressing representations. Generalization improves during the compression phase, not the fitting phase. The paper also proposed a cause for the compression. Stochastic gradient descent adds noise to each update. Over many steps, the noise behaves like diffusion. Diffusion in weight space pushes layers toward representations that depend on the input only through what predicts the label. The paper proposed that the mechanism behind generalization was a process driven by noise, similar to diffusion in physics.

The claim drew attention because it offered something the field had been missing. Benchmark papers report what a network achieves. They do not say why. Generalization in deep learning had been studied through learning theory, through capacity measures, and through empirical bounds. None of those approaches gave a picture of what happens inside a network during training.

The two-phase account did. It said, in measurable terms, that a network first learns and then forgets, and that the forgetting is what makes it generalize. The picture was simple enough to draw. The mechanism was specific enough to test. The paper was widely cited within a year of release.

Why did the compression phase fail to hold up?

Andrew Saxe and his coauthors published a response at the 2018 International Conference on Learning Representations. The paper was titled On the Information Bottleneck Theory of Deep Learning. It examined the main claims one at a time with new experiments.

The first claim was that a compression phase exists. Saxe and his coauthors reproduced the original setup and confirmed the picture, but only for networks that used tanh activations. Tanh saturates at both ends of its range. Once a neuron’s input becomes large in absolute value, the output stays near plus or minus one and small changes in input produce almost no change in output.

The estimator used by Shwartz-Ziv and Tishby bins the activations to compute mutual information. Saturation pushes many activations into the same bin, which makes the estimated mutual information with the input fall. When the same networks used ReLU activations instead, which do not saturate at the positive end, no compression phase appeared. The trajectories moved up and to the right and stayed there. The networks still generalized well. The compression phase depended strongly on the activation function. It was not a universal property of deep learning.

The second claim was that compression causes generalization. Saxe and his coauthors built networks that did not compress and showed that they generalized. They also built networks that did compress and showed that the timing of compression had no consistent relationship to the timing of generalization.

They studied a linear network, where the optimization has a closed-form solution, and showed that generalization proceeds without any compression at all. The link from compression to generalization, the central claim made by Shwartz-Ziv and Tishby, did not survive controlled tests.

The third claim was that stochastic gradient descent causes the compression through noise that behaves like diffusion. The argument required compression to depend on stochasticity. Saxe and his coauthors trained the same networks with full-batch gradient descent, which has no stochasticity at all. The compression phase still appeared when the activations were tanh. It still failed to appear when the activations were ReLU. Whatever caused the phase, it was not the noise. The proposed mechanism did not hold.

The 2017 paper described a unified picture with a single mechanism. The rebuttal did not deny that some trajectories on the information plane look the way Shwartz-Ziv and Tishby described. It denied that the picture meant what the original paper claimed.

A phase tied to one activation function. A correlation that breaks under controlled testing. A proposed cause that disappears when the supposed cause is removed. The three claims that made the picture interesting failed under closer examination.

Where does the information bottleneck stand now?

The framework introduced in 1999 has held up well. It still produces useful results in clustering. It still serves as an objective when the goal is to find a representation that throws away input detail while keeping label information. Variants of the objective appear in modern representation learning, often under different names. The framework formalizes a real problem.

The proposal that the information bottleneck explains deep learning has not held up. The Saxe rebuttal still stands, and later work has supported its conclusions more than the original two-phase picture.

Later studies showed that, for deterministic networks with continuous variables, mutual information is formally infinite. They also showed that the estimator-dependent quantities plotted in the 2017 paper are not the mutual information values they appear to be. The information plane remains a useful way to visualize representations. Present evidence does not support it as a hidden mechanism for why deep learning works.

The Shannon series argued that modern AI runs inside structures designed by Claude Shannon. Cross-entropy loss and channel capacity are part of those structures. So is the equivalence of prediction and compression. The information bottleneck extends them by asking what a representation should keep when it cannot keep all of it.

That extension has lasted. As a definition of relevant information, the framework remains useful. As a theory of deep learning, it is one of several proposals that identified an important object and gave a partial description of how it behaves. The question of why deep networks generalize remains open.

Conclusion

The information bottleneck began as a method for deciding what to throw away. Tishby and his coauthors defined relevance through a second variable and turned compression into an optimization problem with a clear objective. The framework gave classical statistics a broader version of the sufficient statistic. It also gave information theory a rate-distortion problem where the distortion measure no longer had to be chosen first.

The attempt to apply the framework to deep learning produced one of the most discussed proposals of the last decade. The proposal had three parts: a two-phase picture, a link between compression and generalization, and a mechanism tied to stochastic gradient descent. Researchers tested each part. The framework survived as a definition. The specific account of deep learning did not survive the tests.

References:

Tishby, N., Pereira, F. C., & Bialek, W. (1999). The information bottleneck method. Proceedings of the 37th Annual Allerton Conference on Communication, Control, and Computing, 368–377. https://arxiv.org/abs/physics/0004057

Tishby, N., & Zaslavsky, N. (2015). Deep learning and the information bottleneck principle. IEEE Information Theory Workshop (ITW), 1–5. https://doi.org/10.1109/ITW.2015.7133169

Shwartz-Ziv, R., & Tishby, N. (2017). Opening the black box of deep neural networks via information. https://arxiv.org/abs/1703.00810

Saxe, A. M., Bansal, Y., Dapello, J., Advani, M., Kolchinsky, A., Tracey, B. D., & Cox, D. D. (2019). On the information bottleneck theory of deep learning. Journal of Statistical Mechanics: Theory and Experiment, 2019(12), 124020. https://doi.org/10.1088/1742-5468/ab3985