Entropy & Cross Entropy in Machine Learning: A Beginner-Friendly Guide with Coins, Chickens, and 20 Questions

Your understanding of Entropy is just 1 question away.

Imagine a 20-example data set containing living and non-living things, and I have chosen one particular item from that set. Now it’s your turn to predict the right answer.

The 1 Bit Example: A Coin Flip

Before we look at the 20-item dataset, let’s start simple: a coin flip. How many yes/no questions do we need to determine if the coin shows Heads or Tails? Just one question: “Is it Head?” This immediately tells us the outcome.

Expanding to 20 questions

Similarly, now dataset with

[embed]

Example:

Question 1

“Does it move on its own?”

YES → Dog, Cat, Lizard, Fish, Human, Butterfly, Whale (7 items) NO → All others (13 items)

Next:

Question 2

(If YES) “Is it warm-blooded?”

YES → Dog, Cat, Human, Whale (4 items) NO → Fish, Butterfly, Lizard (3 items)

Intuition: Information and Surprise

Each question reduces uncertainty → exactly what entropy teaches. Entropy is the measure of average information contained in the data. Here, the information is the measure of how surprised we are if we get an outcome.

Red and orange chicken

For example, as shown in the picture, if we randomly take one chicken, how surprised will we be when we get a red chicken? And similarly, if we randomly take out one chicken, how surprised will we be when we get orange chicken? Getting orange chicken is more surprising than red chicken. Now, let’s see it probabilistically. The probability of randomly catching a red chicken in a group of 9 chickens is 8/9, and the probability of randomly catching an orange chicken is 1/9. As we can see, as the probability of a certain outcome increases, the information content decreases, and as the probability of a certain outcome decreases, the information content increases. Information and probability have an intuitive inverse relationship: rare events carry more information, common events carry less. Formally, we can think of:

i.e. Information(x) α 1/ P(x)

Shannon’s Rule and Logarithm

But here is one caveat as per Shannon’s rule for information. The information of two independent events adds up instead of multiplying. But the probability of two independent events gets multiplied.

P(X and Y) = P(X) P(Y)*

But,

I(X and Y) = I(X) + I(Y)

So, to make mathematics consistent and the product of probability and the additivity of information in sync. We use logarithms.

Now,

I(X and Y) = 1/ (P(X) P(Y))*

log2(1/(P(X)P(Y))) = -(log2(P(X)) + log2(P(Y)))*

If I(X) = log2(1/P(X)) = -log2(P(X))

Then I(X and Y) = I(X) + I(Y)

Now the average surprise is an entropy.

E = ∑(P(Xi)I(Xi))

As the uncertainty increases, the entropy increases, and as the uncertainty decreases, the entropy decreases. If we have a coin with heads and tails, the entropy is 1. If we have a coin with both sides heads, the entropy is 0.

Fair Coin

E(with fair coin) = -P(head)log2(P(head))-P(tail)log2(P(tail))

= 0.5 + 0.5 = 1

Two Headed Coin

E(with coin having both side head)= -1 0 - 00

= 0

Cross Entropy

Now, what is cross-entropy?

What if we have created a model that is given some input and it predicts the output based on the formulation of the model? How can we test it? For this, we use cross-entropy. Cross-entropy is the measure of the difference between actual and predicted output.

For example, if the probability of today raining is 0.9, but the model showed the score of 0.1, then the cross-entropy is higher. This shows that our model is worse than what it actually is.

In other words, cross-entropy is the measure of how surprised our model will be if it is shown the actual output.

Further, taking this analogy to our earliest example of 20 questions.

Entropy is the minimum number of optimal yes/no questions required to identify the truth (assuming perfect knowledge).

Cross-entropy is the number of questions your model asks based on its own understanding, which may be wrong.

The difference between them is how inefficient (or stupid) your questioning strategy is.

Cross-entropy measures how surprised the model is when shown the actual outcome. If the model assigns a high probability to the true label, the surprise is low. If the model assigns a low probability, the surprise is high — indicating the model failed to capture the true pattern.

mathematically,

H(x) = -P(X)log(Q(X))

Binary Cross-entropy

For binary classification, we use binary cross-entropy (BCE):

BCE= -ylog2(y_hat)-(1-y)log2(1-y_hat)

Here, “y” is the true outcome and “y_hat” is the predicted probability. BCE measures how far the model’s prediction is from the truth.

Cross entropy is crucial in compression and machine learning:

• Compression: common events require fewer bits; rare events require more bits.
• Machine Learning: cross-entropy loss tells us how well the model predicts unknown outcomes.