Statistics Behind Machine Learning: Understanding Logistic Regression

This is a continuation of my previous blogs:

• Statistics Behind Machine Learning: Understanding Simple and Multiple Linear Regression
• Statistics Behind Machine Learning: Understanding Polynomial Regression

In my earlier blogs, we discussed regression techniques such as simple, multiple, and polynomial regression. In all those methods, the goal was to predict continuous numerical values. However, in real-world scenarios, not every problem is about predicting numbers.

There are many situations where the output is not a number, but a category. For example, we may need to determine whether a customer will buy a product or not, whether an email is spam or not, or whether a student will pass or fail.

These types of problems involve discrete outcomes, such as yes/no, true/false, or 0/1. This is where classification comes into the picture.

What Is Classification?

Before we even get to classification, let’s understand why we need something new. Instead of the already well-known regression.

In regression, you predict a continuous value, something that can be any number on a scale. But in many real problems, the output belongs to a category, a discrete bucket, not a number line.

That’s why we need classification.

Now, you might wonder, “Why not just use linear regression for this? Predict 0 or 1 as numbers?”

Fair thought. But here’s the problem: linear regression has no boundaries. It can confidently output 1.7, or -0.4, or 93.2. None of those makes sense as a probability. You need something that stays between 0 and 1. That’s where logistic regression comes in.

What Is Logistic Regression?

Despite having “regression” in its name, logistic regression is a classification algorithm.

(Yes, the name is a little misleading. Welcome to machine learning naming conventions, confusing since forever.)

The key idea is this: instead of predicting a class label directly, the model first predicts a probability, how likely is it that this input belongs to Class 1?

• If that probability is ≥ 0.5 → predict Class 1
• If that probability is < 0.5 → predict Class 0

So the model doesn’t just say “this email is spam.” It says “this email has an 87% chance of being spam”, and then you decide what to do with that.

This is a much richer output. It tells you not just what the model thinks, but how confident it is.

Part 1: Binary Classification Using the Sigmoid Function

Let’s build this from the ground up with a real example.

The Problem

Predict whether a student will pass (1) or fail (0) based on hours of study.

Simple, relatable, and perfect for learning.

Step 1: The Linear Equation (Same as Always)

Just like in linear regression, we start with:

z = w·x + b

Where:

• x = input feature (hours studied)
• w = weight (how much study hours matter)
• b = bias (a baseline shift, the model’s default leaning)
• z = raw output

For example, if a student studied 3 hours:

z = 0.8 × 3 + (−1.2) = 2.4 − 1.2 = 1.2

This z value can be any number, positive, negative, huge, or tiny. That’s the problem. We can’t use it as a probability directly.

Step 2: The Sigmoid Function — Squashing Everything into (0, 1)

This is where logistic regression gets its magic.

We apply the sigmoid function to z:

What does this do? It takes any real number, no matter how large or small, and squashes it into a value strictly between 0 and 1.

Some intuition:

Back to our student:

z = 1.2
σ(1.2) = 1 / (1 + e^−1.2) = 1 / (1 + 0.3012) ≈ 0.769

The model says: 76.9% chance this student passes. That’s a probability we can actually work with.

Step 3: Converting Probability to a Class Label

Now we apply a simple decision rule called the threshold:

If P ≥ 0.5  →  Predict: Pass (Class 1)
If P < 0.5  →  Predict: Fail (Class 0)

With P = 0.769, we predict Pass.

A quick note on thresholds: 0.5 is just the default. In real problems, you tune it.

For cancer screening, you might set the threshold to 0.2 — you’d rather flag a healthy person for more tests than miss a real case.

For spam filtering, you might use 0.7 — you’d rather let a spam email through than block a real one. The threshold is your business judgment, not a fixed law.

Step 4: The Loss Function — How the Model Knows It’s Wrong

During training, the model needs a way to measure how wrong its predictions are. In regression, we used Mean Squared Error. In logistic regression, we use something better suited for probabilities: Binary Cross-Entropy Loss (also called Log Loss).

Where:

• y = actual label (0 or 1)
• ŷ = predicted probability

The intuition is elegant:

• When y = 1 (actual is positive): Loss = −log(ŷ). If the model said 0.99, the loss is tiny. If it said 0.01, the loss is huge.
• When y = 0 (actual is negative): Loss = −log(1 − ŷ). Same logic, flipped.

It aggressively punishes confident wrong predictions. Which is exactly what you want.

Why not MSE? Because when you combine MSE with the sigmoid, the loss surface becomes non-convex, full of local minima that make gradient descent unreliable. Cross-entropy stays convex and well-behaved. The math works out cleanly.

Step 5: Training — How the Model Learns

This is where gradient descent does its job.

The full training loop:

1. Start with random weights (or all zeros): w = 0, b = 0
2. Forward pass: compute z = w·x + b → sigmoid → predicted probability ŷ
3. Compute loss: how wrong is ŷ compared to actual y?
4. Compute gradients: which direction do we adjust the weights to reduce loss?

∂L/∂w = (ŷ − y) · x
∂L/∂b = (ŷ − y)

1. Update weights:

w = w − α · (ŷ − y) · x
b = b − α · (ŷ − y)

Where α (alpha) is the learning rate — a small number like 0.01 that controls how big each step is.

1. Repeat steps 2–5 for thousands of iterations until the loss stops dropping.

Notice that (ŷ − y) is simply the prediction error. When the model is very wrong, the update is large. When the model is almost right, the update is tiny. The system is self-regulating.

Step 6: Prediction (After Training)

Once training is complete, the weights are locked. Prediction is just the forward steps — no loss, no gradient, no update:

New input (x) → z = w·x + b → sigmoid → probability → threshold → class

Training finds w and b. Prediction uses them.

Code: Binary Logistic Regression From Scratch

import numpy as np
# Sample data: hours studied → pass/fail
X = np.array([1, 2, 3, 4, 5, 6])
y = np.array([0, 0, 0, 1, 1, 1])
# Initialize parameters
w = 0.0
b = 0.0
alpha = 0.1
# Sigmoid function
def sigmoid(z):
    return 1 / (1 + np.exp(-z))
# Training loop
for epoch in range(1000):
    total_loss = 0
    for i in range(len(X)):
        # Forward pass
        z = w * X[i] + b
        y_hat = sigmoid(z)

        # Loss
        loss = -(y[i] * np.log(y_hat) + (1 - y[i]) * np.log(1 - y_hat))
        total_loss += loss

        # Gradients
        error = y_hat - y[i]
        dw = error * X[i]
        db = error

        # Update weights
        w -= alpha * dw
        b -= alpha * db

# Prediction
def predict(x, w, b, threshold=0.5):
    z = w * x + b
    prob = sigmoid(z)
    return int(prob >= threshold), round(prob, 4)

# Test on new data
for hours in [2.0, 3.5, 5.0]:
    cls, prob = predict(hours, w, b)
    print(f"Hours studied: {hours} → Probability: {prob} → {'Pass' if cls else 'Fail'}")

Output:

Hours studied: 2.0 → Probability: 0.1423 → Fail
Hours studied: 3.5 → Probability: 0.4987 → Fail
Hours studied: 5.0 → Probability: 0.8601 → Pass

Notice how 3.5 hours lands right at the boundary. That’s the model’s uncertainty zone — and it’s honest about it.

Part 2: Multi-Class Classification Using Softmax

Binary classification is powerful. But what if you have more than two possible outcomes?

Predict the grade tier of a student:

• Class 0 → Fail
• Class 1 → Average
• Class 2 → Excellent

You can’t use a single sigmoid anymore. You need a probability for each class, and they all need to add up to exactly 1.

Enter: the Softmax function.

Step 1: One Linear Equation Per Class

Instead of one z, we compute one for each class:

z_fail      = w_fail · x + b_fail
z_average   = w_average · x + b_average
z_excellent = w_excellent · x + b_excellent

Each class competes on its own terms.

Step 2: The Softmax Function

In plain English: raise each z to the power of e, then divide each by the total sum. This forces all probabilities to add up to 1.

Example — with real numbers:

z_fail      = 0.5
z_average   = 2.1
z_excellent = 1.3

e^0.5 = 1.6487
e^2.1 = 8.1662
e^1.3 = 3.6693

Sum = 13.4842

P(Fail)      = 1.6487 / 13.4842 = 12.2%
P(Average)   = 8.1662 / 13.4842 = 60.6%
P(Excellent) = 3.6693 / 13.4842 = 27.2%

Total = 100.0% ✓

Predicted class: Average (highest probability at 60.6%).

Notice something important: the model doesn’t just say “Average.” It tells you there’s a 27.2% chance this student is actually Excellent — useful information that a hard label alone would hide.

Step 3: The Loss Function — Categorical Cross-Entropy

For multi-class problems, we use:

Where y is a one-hot vector — all zeros except a 1 at the correct class position.

For example, if the actual label is “Average” (Class 1):

y = [0, 1, 0]
p = [0.122, 0.606, 0.272]

Loss = −(0·log(0.122) + 1·log(0.606) + 0·log(0.272))
     = −log(0.606)
     = 0.501

Only the probability of the correct class contributes to the loss. Everything else cancels out. The model’s entire job is to push that one correct probability as close to 1.0 as possible.

Step 4: Training

Same loop as binary, just with more weight sets to update:

1. Compute z for each class
2. Apply softmax → get probabilities for all classes
3. Compute categorical cross-entropy loss
4. Compute gradients and update each class’s weights independently
5. Repeat until loss converges

The error for each class is still simply (ŷ − y):

• For the correct class: ŷ − 1 (model under-predicted → push weights up)
• For wrong classes: ŷ − 0 (model gave unwanted probability → push weights down)

Step 5: Prediction

New input → z per class → softmax → probabilities → argmax → predicted class

No threshold needed. You just pick the class with the highest probability.

Code: Softmax Classification

import numpy as np

# Example z-scores for 3 classes
z = np.array([0.5, 2.1, 1.3])

# Softmax function
def softmax(z):
    exp_z = np.exp(z - np.max(z))  # subtract max for numerical stability
    return exp_z / np.sum(exp_z)

probs = softmax(z)
class_names = ['Fail', 'Average', 'Excellent']
print("Class Probabilities:")
for name, prob in zip(class_names, probs):
    print(f"  {name}: {prob*100:.1f}%")
predicted = np.argmax(probs)
print(f"\nPredicted class: {class_names[predicted]} ({probs[predicted]*100:.1f}% confidence)")

Output:

Class Probabilities:
  Fail: 12.2%
  Average: 60.6%
  Excellent: 27.2%

Predicted class: Average (60.6% confidence)

One small but important detail in the code: np.exp(z - np.max(z)). This subtraction is called the numerical stability trick. When z values are very large, e^z can overflow to infinity. Subtracting the max first keeps the numbers manageable without changing the final probabilities at all. A small trick that saves you from silent bugs.

The Complete Picture

Here’s the full mental model, side by side:

BINARY CLASSIFICATION (Sigmoid)
─────────────────────────────────
Training:
  Data → z = w·x + b → sigmoid → probability
       → cross-entropy loss
       → gradient descent → update w, b
       → repeat
Prediction:
  New input → z = w·x + b → sigmoid → probability
            → threshold → Class 0 or Class 1

MULTI-CLASS CLASSIFICATION (Softmax)
──────────────────────────────────────
Training:
  Data → z per class → softmax → probabilities
       → categorical cross-entropy loss
       → gradient descent → update all weight sets
       → repeat
Prediction:
  New input → z per class → softmax → probabilities
            → argmax → winning class

The core philosophy never changes: convert raw linear outputs into probabilities, measure how wrong those probabilities are, and nudge the weights in the right direction. The machinery scales up, but the soul stays the same.

Before You Go: The Things That Actually Matter

Now that you understand the mechanics, here are the insights that will actually make you a better practitioner:

1. The threshold is a design decision, not a mathematical constant. Never blindly use 0.5. Ask yourself: what’s worse in my problem, a false positive or a false negative? That answer tells you where to set your threshold.

2. Probability outputs are more valuable than class labels. A model that says “87% spam” is far more useful than one that says “spam.” Downstream systems can use that confidence score to make smarter decisions.

3. The model doesn’t care about your class names. It only sees numbers. “Pass” and “Fail” are just 1 and 0 inside the model. The naming is entirely for your benefit.

4. Logistic regression is still one of the most used models in industry. Despite being one of the simplest classifiers, it’s fast, interpretable, and surprisingly powerful on well-engineered features. Neural networks get the headlines, but logistic regression quietly runs in production at scale across banking, medicine, and tech.

5. Softmax is the building block of modern AI. The output layer of virtually every neural network classifier — including the large language models powering today’s AI tools — uses softmax. Understanding it here means you’ve already understood a core piece of how GPT-style models produce outputs.

And ya, we will reach the end of the blog. If this blog gave you insights, it’s done its job, or if it gave you more questions than answers, even better. Curiosity is the engine. Keep going.

When you feel this content is valuable, follow me for more upcoming Blogs.

Connect with Me:

• LinkedIn: Anand Sundaramoorthy
• Instagram: @anandsundaramoorthysa
• Email: sanand03072005@gmail.com