AI/ML Under the Hood — Part 19: Neural Network Output Layers — Where Learning Begins
From Gaussian Likelihoods to Cross-Entropy: Understanding How Neural Networks Learn Through Their Outputs
AI/ML Under the Hood — Part 19: Neural Network Output Layers — Where Learning Begins
Continue reading the complete article here — free access. 🔗 New to the series? Start here: AI/ML Under the Hood — Learning Guide
A neural network, in the context of supervised learning, is trained using data and later performs inference on its own. During training, each example flows through the network in what is known as a forward pass: the input enters the input layer, passes through the hidden layers, and produces an output at the output layer.
The structure of such a network is typically visualized as layers of interconnected neurons:
A feedforward neural network: input layer (ℓ = 0), hidden layers (0 < ℓ < L), and the output layer (ℓ = L).
At first glance, this diagram suggests a simple left-to-right flow of information. But this view hides what truly matters.
The output layer is not just where the network produces its prediction — it is where that prediction is evaluated. The model’s output is compared to the correct answer, an error is computed, and that error is propagated back through the network to adjust all internal parameters.
In this sense, the output layer is both the end of the forward pass and the starting point of learning.
To better understand this, it is useful to momentarily ignore the network's internal complexity.
Instead of focusing on the many hidden layers, we can treat them as a single component — a “black box” that transforms inputs into outputs.
The most important design decision in a supervised neural network is not how many layers it has or how they are connected — but what the output represents.
Whether the network predicts a continuous value, a probability, or a distribution over multiple classes determines:
- How we measure error
- How gradients behave
- and ultimately, how the model learns
The network may appear complex internally, but all its complexity serves a single purpose: producing an output that can be meaningfully compared to reality.
Output Layer Types
However, despite this apparent variety, most output layers are built on three fundamental forms:
- Regression — predicting a continuous value
- Binary classification — predicting a probability between two classes
- Multi-class classification — distributing probability across multiple classes
These are not arbitrary choices. Each form reflects a different type of question we ask the model:
- How much? → Regression
- Which side? → Binary classification
- Which one? → Multi-class classification
While implementations may vary, these three forms capture the essential ways in which a neural network expresses its output in supervised learning.
Output Layer Type: Regression
In regression tasks, the goal of the network is to predict a continuous value.
The internal layers produce a raw output — typically a weighted sum of the values flowing through the network. At the output layer, this value is passed through an identity function, leaving it unchanged.
In other words, the model outputs a real number directly.
This number can represent quantities such as the number of machines to purchase, the required dosage of a treatment, a price, or a measurement. The prediction is not a choice or a probability — it is a value.
When modeling a regression problem from a probabilistic perspective, we assume that the target value is drawn from a normal (Gaussian) distribution centered around the model’s prediction:
In this view, the model’s prediction defines the center of a bell-shaped curve, and all possible values of yₙ lie along this distribution. The curve represents the probability density, indicating how plausible each value is.
The density of observing a specific value yₙ is given by:
Values close to the prediction have higher density and are considered more plausible, whereas values farther away have lower density.
The likelihood based on this probability density measures how plausible the observed dataset is under the model.
For mathematical convenience, we take the logarithm of the likelihood. Nothing about the objective changes — we are still evaluating how well the model explains the data — but the expression becomes clearer and easier to handle.
The result is very interesting:
Our objective is to find θ for which the Log Likelihood is the maximum, hence maximum likelihood. Hence, our objective is to find the θ for which the Squared Error term is the minimum (Negative Log-Likelihood). This is all based on simple math.
In Linear Regression, as explained in Part 16, the model prediction is:
where θ represents the model parameters.
And therefore, finding θ in the Linear Regression expression is a closed solution:
But in Neural Networks, the prediction is not linear. Even though the activation function in the output layer is the identity function, the activation functions in the hidden layers are not linear.
where θ represents the entire neural network processing flow
Hence, a closed-form solution, as in linear regression, is not possible. The alternative is applying Gradient Descent.
The Cost Function
In previous parts of this series, including linear regression and logistic regression, we already minimized an objective derived from the Negative Log-Likelihood. Although we did not explicitly use this terminology, in neural network literature, this minimized quantity is commonly referred to as the cost function.
The loss function measures the error for a single training example:
Since the network learns from an entire dataset, the losses are aggregated into a single expression known as the cost function:
In the case of regression, this becomes:
Once the cost function is defined, we compute its gradient with respect to the network parameters and iteratively update them in the opposite direction of the gradient:
This small operation means quite a lot. We are basically calculating the cost function gradient with respect to all the parameters that compose θ, hence:
These gradients are derived from the δ̲ values, which are computed using backward propagation. The process begins by calculating the δ̲⁽ᴸ⁾ of the output layer. Based on this output error signal, the δ̲ values of the hidden layers are then computed while propagating backward through the network.
Once δ̲⁽ᴸ⁾ is known, we can compute the gradients of the output layer parameters:
So, how do we calculate δ̲⁽ᴸ⁾ itself? The calculation begins at the output layer, which receives the activations from the previous layer and computes the weighted input vector s̲⁽ᴸ⁾.
The identity function I is then applied, producing the output vector x̲⁽ᴸ⁾.
and
δ̲⁽ᴸ⁾ is the gradient of the cost function with respect to the weighted input vector s̲⁽ᴸ⁾ reaching the output layer.
so
Element-wise, this becomes:
This backward propagation mechanism enables the network to compute gradients for all parameters and iteratively improve them using Gradient Descent, as demonstrated through a numerical example in Part 13.
Output Layer Type: Binary classification
In binary classification tasks, the objective of the network is not to predict a continuous value but rather to predict the probability of one of two possible outcomes: y ∈ {0, 1}.
These choices may represent:
- black [y=0] or white [y=1]
- false [y=0] or true [y=1]
- not spam [y=0] or spam [y=1]
The network therefore produces a value between 0 and 1 representing the probability that the input belongs to one of the two classes.
Unlike regression, where the output layer uses the identity function and can produce any real number, binary classification requires a bounded output. For this purpose, neural networks commonly use the sigmoid activation function:
This function transforms any real-valued input into the interval:
Large positive values approach 1, while large negative values approach 0.
The network output can therefore be interpreted as a probability:
and consequently:
From a probabilistic perspective, binary classification assumes that the target variable follows a Bernoulli distribution:
As discussed in Part 17, binary classification can be modeled probabilistically using the Bernoulli distribution and Maximum Likelihood Estimation.
Unlike the Gaussian case in regression, the Bernoulli distribution is discrete rather than continuous. Therefore, we use a probability mass function rather than a probability density function.
The network predicts the probability that a sample belongs to class y=1:
Therefore:
and:
These two cases can be combined into a single compact expression:
The likelihood of the entire dataset is therefore:
For mathematical convenience, we again take the logarithm:
As in regression, training minimizes the Negative Log-Likelihood:
For a single sample, this expression defines the Binary Cross-Entropy loss function:
Aggregating the loss across the dataset yields the corresponding cost function:
As in regression, the objective of training is to find the parameter set that minimizes the cost function.
Since no closed-form solution exists, the network again relies on Gradient Descent, iteratively updating the parameters in the direction opposite to the gradient:
However, an important challenge now emerges.
When the sigmoid activation becomes saturated in this case, very close to 0 or 1, its derivative approaches zero:
Similar to regression, we need to calculate the δ̲⁽ᴸ⁾ of the output layer in order to begin the backpropagation.
For different output activation functions, δ̲⁽ᴸ⁾ will differ, as seen in Part 13, where the hyperbolic tangent is used instead of the sigmoid.
For very large positive or negative inputs, the sigmoid output approaches 1 or 0, causing the curve to become nearly flat.
As a result, the gradients flowing backward through the network may become extremely small. Consequently, parameter updates become tiny and learning slows dramatically.
This phenomenon is known as the vanishing gradient problem.
Over the years, several techniques have been developed to mitigate the vanishing gradient problem.
One important improvement was replacing sigmoid activations in hidden layers with activation functions that do not saturate as easily, such as the ReLU (Rectified Linear Unit):
Unlike the sigmoid function, ReLU does not compress large positive values into a narrow range between 0 and 1. Consequently, its derivative remains large across a wider region:
This allows gradients to propagate more effectively through deep networks.
Additional techniques such as:
- careful weight initialization
- normalization methods
- residual connections
Further improved gradient flow and enabled the successful training of very deep neural networks.
Today, sigmoid activations are used far less frequently in hidden layers. Instead, modern neural networks commonly use activation functions such as ReLU in hidden layers, since they allow gradients to propagate more effectively and significantly reduce the vanishing gradient problem.
However, sigmoid activations remain highly useful in binary classification output layers because their outputs naturally represent probabilities between 0 and 1.
Using ReLU in the output layer of a binary classifier would be problematic, since negative inputs collapse to 0, preventing the output from representing a smooth probability distribution across the interval [0,1][0,1][0,1].
Output Layer Type: Multi-class classification (Softmax)
In many real-world problems, the network must choose not between two possibilities, but among many possible classes.
Examples include:
- cat, dog, or bird
- car, truck, or motorcycle
- digits 0 to 9
In such cases, the objective of the network is not to predict a single probability, but rather a probability distribution across multiple possible classes.
If there are K classes, the network produces K output values:
These values are often called logits.
Unlike binary classification, where a sigmoid function independently maps a single value into the interval [0,1], multi-class classification requires all outputs to compete with one another.
The probabilities must satisfy two conditions:
and:
For this purpose, neural networks commonly use the Softmax activation function:
where:
represents the raw score produced for class k.
Unlike sigmoid activation, where each output is treated independently, Softmax introduces competition between classes.
Increasing the probability of one class necessarily decreases the probabilities of the others.
Unlike binary classification, where each output can be treated independently, Softmax creates competition between classes. The network is not merely learning whether a class is correct in isolation — it is learning which class should dominate all alternatives simultaneously.
From a probabilistic perspective, multi-class classification assumes that the target variable follows a categorical distribution:
For each training example, the correct class is commonly represented using one-hot encoding:
where only the correct class contains the value 1.
If the network predicts:
then the model assigns probability 0.7 to the correct class.
The likelihood of a single sample can therefore be written compactly as:
The likelihood of the entire dataset becomes:
However, directly computing exponentials can create numerical instability. If one of the logits becomes very large, the exponential term may exceed the numerical range that computers can reliably represent.
To avoid this problem, implementations commonly subtract the maximum logit before applying the exponential function:
Since the same value is subtracted from all logits, the resulting probabilities remain unchanged, while the computation becomes numerically stable.
As before, we take the logarithm for mathematical convenience:
Training minimizes the Negative Log-Likelihood:
For a single training example, this defines the Categorical Cross-Entropy loss function:
Aggregating the losses across the dataset yields the corresponding cost function:
The term
is known as the log-sum-exp expression. It behaves like a smooth approximation of the maximum logit:
This means that the Softmax-Cross-Entropy loss can be understood as comparing the correct class score against the strongest competing score. The model is rewarded when the correct class logit exceeds the other logits.
As in regression and binary classification, the objective is to find the parameter set that minimizes the cost function.
Since no closed-form solution exists, the network again relies on Gradient Descent:
Similar to regression and binary classification, backpropagation begins by calculating the error signal at the output layer.
However, in the Softmax case, the situation becomes slightly more complex.
For each training sample n, the network now produces K output logits:
which are transformed by the Softmax function into a probability distribution over all classes.
As a result, the output layer error signal is no longer a vector of size N, but rather a matrix containing one error term per sample and per class:
where:
Unlike the sigmoid case, the Softmax outputs are not independent. Each probability depends on all logits simultaneously:
Therefore, changing a single logit affects the probabilities of all classes.
At first glance, this suggests that the derivative should become significantly more complicated.
Surprisingly, when the Softmax activation function is combined with the Cross-Entropy loss function, many terms cancel during differentiation, producing a remarkably elegant result:
This simplification is one of the reasons why the Softmax + Cross-Entropy combination became the standard approach for multi-class neural network classification.
Although neural networks may contain millions or even billions of parameters, the learning process always begins at the same place: the output layer.
Whether predicting a continuous value, a binary probability, or a probability distribution across many classes, the output layer defines how the network interprets reality. From this interpretation emerges the probabilistic model, the likelihood function, the cost function, and ultimately the gradients that drive learning itself.
The hidden layers may contain most of the computational complexity, but the output layer defines the meaning of the prediction. It determines what the network is trying to express, how error is measured, and how information flows backward through the system during learning.
In this sense, the output layer is not merely the end of the neural network. It is the mathematical bridge between prediction, probability, and optimization.
🤝
Kobi Toueg Principal Software Developer | Software Security | Mobile & Video Systems
LinkedIn: https://www.linkedin.com/in/kobi-toueg
Explore the Series: AI/ML Under the Hood
메타데이터
- post_id
- 044ca0568489
- slug
- ai-ml-under-the-hood-part-19-neural-network-output-layers-where-learning-begins-044ca0568489
- url
- https://medium.com/the-thoughtful-engineer/ai-ml-under-the-hood-part-19-neural-network-output-layers-where-learning-begins-044ca0568489
- canonical_url
- https://medium.com/the-thoughtful-engineer/ai-ml-under-the-hood-part-19-neural-network-output-layers-where-learning-begins-044ca0568489
- author_url
- https://medium.com/@kobi.toueg
- status
- ok
- fetched_at
- 2026-06-15 20:49:13
