What is the Softmax Function?

What is the Softmax Function?

The Softmax function converts a list of numbers (called logits) into probabilities.

For example, suppose a neural network outputs:

[2.0, 1.0, 0.1]

These values are just scores. They are not probabilities because:

• They can be negative or positive.
• They do not sum to 1.

Softmax transforms them into:

[0.659, 0.242, 0.099]

Now:

• Every value is between 0 and 1.
• The sum of all values is exactly 1.
• They can be interpreted as probabilities.

The Softmax formula is:

Where:

• zi= score (logit) for class i
• e = Euler’s number (≈ 2.718)
• K = total number of classes

Why Do We Need Softmax?

Imagine you build a model to classify images into:

1. Cat
2. Dog
3. Bird

The final layer outputs:

Cat  = 4.5
Dog  = 2.3
Bird = 0.8

These values don’t tell us probabilities.

After applying Softmax:

Cat  = 0.87
Dog  = 0.10
Bird = 0.03

Now we can say:

The model is 87% confident that the image is a cat.

This makes the output understandable for humans.

Why Use the Exponential Function?

A common question is:

Why don’t we simply divide by the sum.

Suppose logits are:

[5, 4, 1]

If we directly normalize:

[0.5, 0.4, 0.1]

The difference between 5 and 1 is not emphasized much.

Softmax first applies the exponential:

e^5 = 148.4
e^4 = 54.6
e^1 = 2.7

Now:

[148.4, 54.6, 2.7]

After normalization:

[0.72, 0.27, 0.01]

The largest score becomes much more dominant.

This helps the model make clearer decisions.

Where is Softmax Used?

1. Image Classification

Classes:

Cat
Dog
Horse
Bird

The model predicts the probability of each class.

2. Large Language Models (LLMs)

When you type:

The capital of France is

The model produces scores for every word in its vocabulary.

Softmax converts these scores into probabilities:

Paris  -> 0.85
London -> 0.05
Rome   -> 0.03
...

Then the next token is selected based on these probabilities.

3. Sentiment Analysis

Positive
Neutral
Negative

Softmax provides probabilities for each sentiment.

Author

Name : Tirth Patel Github : Tirth9978