Day 10 of 30: [Softmax and Logits — Classification Output Layer] (Deep Learning Challenge) — First code, then understand

.

First code, then understand — Day 10 of 30: [Softmax and Logits — Classification Output Layer] (Deep Learning Challenge).

Includes: NO source code on GitHub — sorry ;) some math formulas

Softmax output across varying logits

Table of Contents

1. What are logits? a) Code example
2. Why softmax?
3. What is softmax? a) Manual softmax in Python b) Why softmax -> logits -> probabilities? c) Why do we need probabilities? d) Without probabilities, loss wouldn’t work! e) But the one-hot encoded isn’t a probability. f) Is One-Hot a probability distribution? g) Cross-Entropy Formula with One-Hot Label

1. What are logits?

| Logits — raw, unnormalized output of neural network before applying softmax. | E.g.: logits = [2.0, 1.0, 0.1] <- not probabilities!

• Logits are used during training for numerical stability,
• Loss functions like ‘CrossEntropyLoss’ in PyTorch expect logits, not probabilities.

1. a) Code example

> PyTorch

import torch
import torch.nn.functional as F

# Raw model output (logits)
logits = torch.tensor([[2.0, 1.0, 0.1]])

# Apply softmax to convert logits into probabilities
probs = F.softmax(logits, dim=1)

> TensorFlow/Keras

import numpy as np
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Softmax

# Example: 3-class classification output
model = Sequential([
    Dense(3, input_shape=(4,)),  # Last layer with 3 neurons
    Softmax()
])

# Fake input: 1 sample with 4 features
x = np.array([[0.5, 0.2, 0.1, 0.7]])
output = model.predict(x)

2. Why softmax?

• Converts logits into probability distribution,
• Each y_hat[i] in (1, 0), and the sum is 1,
• Squashes large positive values and higlights the most probable class,

Softmax formula: y_hat_i = exp(z_i) / sum_j(exp(z_j))

| If logits = [9.0, 1.0, 0.1], softmax will be skewed toward the first class. | Numerical stability trick: y_hat_i = exp(z_i — max(z)) / sum_j(exp(z_j — max(z))). | This version subtracts max(z) from all logits to improve numerical stability (avoids large exponentials).

3. What is softmax?

• Converts logits into a probability distribution

• Takes raw outputs (logits) from the final layer,
• Transforms them into values in the range [0, 1],
• Interpretable as class probabilities.

• Ensures outputs sum to 1

• Output vector: y_hat = [p1, p2, …, pn], where sum_i(p_i) = 1
• Makes it suitable for multi-class classification

• Used in multi-class classification

• Output layer for: Image classification, Text classification, ** Any task with mutually exclusive classes.

• Activation function

• Like ReLU or sigmoid, but for the output layer only,
• It’s differentiable -> supports gradient-based learning.

3. a) Manual softmax in Python

import numpy as np

def softmax(logits): shifted = logits — np.max(logits) # For numerical stability exps = np.exp(shifted) return exps / np.sum(exps)

logits = np.array([2.0, 1.0, 0.1]) softmax(logits)

3. b) Why softmax -> logits -> probabilities?

• Because in classification tasks, we want the model’s output to express:

• “How likely is this input to belong to class k?”,
• “This image is 90% cat, 10% dog”.

• This is naturally expressed using probabilities — values between 0 and 1, summing to 1.

3. c) Why do we need probabilities?

Because in classification: We want… -> “This is a cat with 85% confidence” So we use… -> Probabilities (e.g.: softmax)

We want… -> “Most likely class is Dog” So we use… -> argmax(probabilities)

We want… -> Measure “how wrong” we were So we use… -> CrossEntropyLoss(prob, true)

3. d) Without probabilities, loss wouldn’t work!

• Cross-Entropy Loss measures the distance between two probability distributions:

• Model’s prediction (via softmax),
• True label (one-hot encoded).

That’s why softmax is critical — it turns unbounded logits into a probabilistic interpretation, so we can measure how wrong the model was.

• Side note:

• During training, PyTorch/Keras apply CrossEntropyLoss directly to logits — it combines log_softmax + NLLLoss internally for numerical stability.
• But during prediction, we apply softmax manually to interpret the results.

Concept — Logits Description — Raw scores (not interpretable)

Concept — Softmax Description — Converts logits into probabilities summing to 1

Concept — Probabilities Description — Let us say “how likely” an input belongs to each class

Concept — Needed for Description — Loss functions, interpretability, decision-making

3. e) But the one-hot encoded isn’t a probability.

• How one-hot encoder works?

• Let’s say we have 3 classes: cat, dog, rabbit.
• If the true label is dog, then the one-hot encoded vector is: [0, 1, 0] # cat=0, dog=1, rabbit=0

• Key properties of one-hot encoding:

• It’s a vector with exactly one 1, and all others are 0,
• Length = number of classes.
• It represents certainty: “100% sure this is class 1”.

3. f) Is One-Hot a probability distribution?

| Yes, technically — it’s a special case.

• A one-hot vector is a degenerate (discrete) probability distribution:

• All probability mass is on one class (1),
• Others are zero (0).

• So it acts like a probability distribution:

• All values in [0, 1],
• Sum = 1.

• That’s why we can compute cross-entropy between:

• model_prob = [0.7, 0.2, 0.1]
• true_label = [0, 1, 0] ← one-hot

3. g) Cross-Entropy Formula with One-Hot Label

• Let:

• y = one-hot true label (e.g., [0, 1, 0])
• y_hat = predicted softmax output (e.g., [0.7, 0.2, 0.1])

• Then:

• CrossEntropy = -sum(y_i * log(y_hat_i)) = -log(y_hat_true_class),
• Because y_i is zero for all but the true class.

So in effect: - Cross-entropy compares:

• A true distribution = one-hot encoded
• A predicted distribution = softmax output

• The lower the predicted probability for the correct class → the higher the loss.

Real-World Analogy - True label: “It’s definitely a dog.” -> [0, 1, 0],

• Model says: ‘Maybe cat: 70%, dog: 20%, rabbit: 10%” -> [0.7, 0.2, 0.1],
• Cross-entropy punishes it for being wrong about dog.

SOURCE CODE on my GitHub — machinelearning-maverick https://github.com/machinelearning-maverick/deep-learning-challenge/

Let me know in the comments what do you think about this challenge! More content in the comments (LinkedIn post characters limit)…

30DaysChallengeWithDeepLearning #Softmax #Logits #DeepLearning #NeuralNetworks #Classification #CrossEntropy #ActivationFunction #PyTorch #Keras #MLChallenge