Cross-Entropy Loss Explained (With a Binary Classification Example)

If you’ve worked with classification models — logistic regression, neural networks, or deep learning — you’ve almost certainly seen cross-entropy loss. It’s everywhere. But why does it work so well, and what is it actually computing?

In this post, we’ll break down cross-entropy as a loss function, build intuition from first principles, and then walk step-by-step through the binary cross-entropy case with concrete examples.

Why do we need a special loss for classification?

In classification, models don’t just output labels — they output probabilities.

For example:

• “This email is spam” → 0.93
• “This image contains a cat” → 0.12

A good loss function should:

1. Reward high confidence when correct
2. Penalize high confidence when wrong
3. Be smooth and differentiable for optimization

Cross-entropy does exactly this.

The intuition behind cross-entropy

At a high level, cross-entropy measures how far your predicted probability distribution is from the true distribution.

• If your predictions closely match reality → low loss
• If your predictions confidently contradict reality → very high loss

From an information theory perspective, cross-entropy answers the question:

“How surprised is the model by what actually happened?”

The more surprised the model is, the larger the loss.

Binary classification setup

Let’s focus on the simplest case: binary outcomes.

• True label: y∈{0,1}
• Model prediction:

So:

• p^​ = predicted probability of class 1
• 1−p^​ = predicted probability of class 0

Binary Cross-Entropy (BCE) loss

The binary cross-entropy loss for a single observation is:

This single formula neatly handles both classes.

Understanding the formula by cases

Case 1: When the true label is 1

If y=1 the loss simplifies to:

If the model assigns a tiny probability to an event that actually occurred, it is heavily penalized.

Case 2: When the true label is 0

If y=0

Predicting “almost certainly yes” when the answer is “no” is disastrous — and cross-entropy makes sure the model feels that pain.

Why the logarithm matters

The logarithm plays a crucial role:

• It magnifies penalties for confident wrong predictions
• It keeps the loss smooth and differentiable
• It turns probability products into sums (great for optimization)

Without the log, the model wouldn’t be pushed strongly enough toward correct probability estimates.

Connection to logistic regression

In logistic regression, predictions are produced using the sigmoid function:

When we pair:

• Sigmoid activation
• Binary cross-entropy loss

we are effectively performing maximum likelihood estimation for a Bernoulli random variable.

This is why BCE is the natural loss function for binary classification.

Loss over a full dataset

For a dataset with nnn observations, the average loss is:

Training a model simply means minimizing this quantity.

Key takeaway

Binary cross-entropy measures how surprised your model is by the true outcome — and it punishes confident wrong predictions much more than uncertain ones.

That single property is what makes it so powerful for classification.