Entropy, KL Divergence, and Binary Cross-Entropy: An Information-Theoretic View of Loss

In the field of machine learning, loss functions are more than just mathematical tools; they are the language that models use to learn from data. A deep connection to information theory, particularly the concepts of entropy, cross-entropy, and KL divergence, is at the heart of many popular loss functions.

This article will take you on a step-by-step journey, starting with the foundations of entropy as a measure of uncertainty and moving through KL divergence and cross-entropy before narrowing down to binary cross-entropy loss, which is used in classification tasks.

By the end, we’ll not only understand why binary cross-entropy is the standard choice for logistic regression and neural networks, but also how its roots in information theory give it a solid theoretical foundation.

Let’s start with the definition of a loss function and its relation with entropy.

A loss function is a mathematical tool that measures the error between predictions and true values, guiding the model to improve its parameters. Its connection to information theory comes from the use of concepts like entropy (uncertainty)and KL divergence, which quantify how much information is lost when one probability distribution is used to approximate another.

Shannon Entropy: Measuring Uncertainty

Shannon Entropy measures the average amount of uncertainty (or information) in a probability distribution p. It tells us how unpredictable the outcome of a random variable is, and this can be expressed with the following formula:

Shannon Entropy

Cross-Entropy and KL Divergence

As explained above, Shannon Entropy H(p) measures the average uncertainty (or information) in the true distribution p. Building on this, Cross-Entropy H(p,q) measures the uncertainty we face when we use a model distribution q instead of the true distribution p:

Cross Entropy

The difference between cross-entropy and the entropy of the true distribution is captured by the Kullback–Leibler (KL) Divergence, defined as:

KL Divergence

Expanding the KL divergence gives us:

If we plug in the entropy and cross-entropy definitions, we obtain:

From this relation, we can write the cross-entropy as the sum of the entropy of the true distribution and the KL Divergence (the extra cost) between the true and model distributions. It measures how much information is lost when you use q to approximate p:

As you might guess, the entropy term H(p) is determined solely by the data and does not depend on the model, meaning the model cannot change it. For this reason, when we minimise the cross-entropy H(p,q), we are effectively minimising the Kullback-Leibler Divergence(the extra cost). In practice, reducing cross-entropy directly reduces KL Divergence, which brings the model’s predicted distribution closer to the true distribution p.

As shown in the graph, KL Divergence measures the gap between the true distribution p(x) (in red) and the model distribution q(x) (in blue). Minimising cross-entropy therefore reduces this gap, making the two distributions overlap more closely.

Binary Cross-Entropy (BCE)

Binary Cross-Entropy (log loss) is a loss function used in binary classification problems and is directly derived from the concept of entropy. While entropy measures the inherent uncertainty in the true distribution of labels, binary cross-entropy quantifies how much additional uncertainty is introduced when using the model’s predicted probabilities instead.

In the case of binary classification, the true distribution is represented by the observed label y, and the model predicts a probability y_hat for the positive class. The cross-entropy, therefore, takes the form

Replacing the true distribution p with the observed label y, we obtain the Binary Cross-Entropy (BCE) loss function:

Thus, BCE is not just an arbitrary loss function but directly arises from the general definition of cross-entropy. Minimising BCE is equivalent to minimising the KL Divergence between the true label distribution and the predicted probabilities, thereby bringing the model’s predictions y_hat closer to the true distribution.

As shown in the figure above, BCE captures how the model behaves in binary classification. When the predicted probability (y_hat) aligns with the true label, the loss remains close to zero. But as the model becomes confidently wrong, the loss shoots up towards infinity, heavily penalising such mistakes. This directly reflects its connection to KL Divergence: the larger the gap between the true and predicted distributions, the higher the loss.

Conclusion

In conclusion, loss functions are deeply rooted in information theory: we started from entropy as a measure of uncertainty, extended it to cross-entropy and KL Divergence, and showed how these concepts ground modern approaches to measuring error. Binary Cross-Entropy naturally arises in classification, and through gradients and optimisation, minimising it aligns model predictions with the true distribution.

References:

• https://medium.com/@yalcinselcuk0/information-processing-and-communication-a-beginners-guide-to-information-processing-7d2f7e60f646
• https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence