Demystifying Logit, Probit, Sigmoid Functions, and Odds: Foundations of Binary Classification

Binary classification — determining whether an email is spam, whether a customer will churn, or whether a loan will default — is fundamental to modern data science. Behind these predictions lie elegant mathematical concepts that transform messy real-world probabilities into workable linear models. This article unpacks the essential mathematics of logistic and probit regression, making these powerful tools accessible to students and practitioners.

1. From Probabilities to Odds: Why We Need a Different Scale

Understanding Odds

While probabilities range from 0 to 1, this bounded scale creates challenges for linear modeling. Enter odds — a more flexible representation:

Odds = p / (1 — p)

Where p is the probability of success.

Example: If a team has an 80% chance of winning (p = 0.8):

• Odds = 0.8 / 0.2 = 4:1
• Interpretation: They’re 4 times more likely to win than lose

The Log-Odds (Logit) Transformation

The genius move is taking the logarithm of odds:

Logit(p) = log(p / (1 — p))

This transformation achieves something remarkable:

• Input: Probability bounded between 0 and 1
• Output: Any real number from -∞ to +∞

This unbounded scale allows us to use linear regression techniques on probability problems — a mathematical bridge between two worlds.

2. The Linear Predictor: Xβ

At the heart of both logit and probit models lies the linear predictor Xβ:

• X: Matrix of features (age, income, credit score, etc.)
• β: Vector of coefficients (weights for each feature)
• Xβ: The weighted sum that produces a single score

Think of Xβ as your model’s “confidence score” before converting to probability. A highly positive Xβ suggests high probability; negative suggests low probability.

3. Logistic Regression: The Sigmoid Solution

The Model

Logistic regression models the relationship:

log(p / (1 — p)) = Xβ

But we need probabilities, not log-odds. Enter the sigmoid function:

The Sigmoid Function

σ(x) = 1 / (1 + e^(-x))

This S-shaped curve elegantly maps any real number to a probability:

• As x → -∞, σ(x) → 0
• As x → +∞, σ(x) → 1
• At x = 0, σ(x) = 0.5

The sigmoid is actually the inverse of the logit function — they undo each other perfectly.

4. Probit Regression: The Normal Distribution Approach

A Different Philosophy

While logistic regression uses the logistic distribution, probit regression assumes errors follow a normal distribution:

Φ^(-1)(p) = Xβ

Where Φ^(-1) is the inverse of the cumulative normal distribution function.

Intuition

Imagine a latent (hidden) variable Y* = Xβ + ε, where ε follows a standard normal distribution. We observe:

• Success if Y* > 0
• Failure if Y* ≤ 0

The probability of success becomes the area under the normal curve.

5. Comparing Approaches: When to Use Which?

| Aspect | Logistic Regression | Probit Regression |
|--------|-------------------|------------------|
| **Mathematical Basis** | Logistic distribution | Normal distribution |
| **Transformation** | Sigmoid function | Cumulative normal CDF |
| **Tail Behavior** | Heavier tails (more sensitive to extremes) | Lighter tails (approaches 0 and 1 faster) |
| **Interpretability** | Coefficients relate to odds ratios | Less intuitive coefficient interpretation |
| **Computational Efficiency** | Faster (closed-form derivatives) | Slower (requires numerical integration) |
| **Common Applications** | Machine learning, web analytics, medical diagnosis | Economics, psychometrics, biostatistics |

6. Practical Insights

Why Logistic Regression Dominates

1. Interpretability: A one-unit change in X multiplies the odds by e^β
2. Computational simplicity: Faster training and prediction
3. Robustness: Performs well even when normality assumptions are violated

When Probit Shines

1. Theoretical justification: When the underlying process truly follows a normal distribution
2. Dose-response studies: Common in toxicology where responses follow normal tolerance distributions
3. Structural equation modeling: Better integration with other normal-based models

7. Real-World Example

Consider predicting loan default based on credit score:

Logistic Model:

• log(p/(1-p)) = -5 + 0.01 × Credit Score
• For a 700 credit score: log-odds = -5 + 7 = 2
• Probability = σ(2) = 1/(1 + e^(-2)) ≈ 0.88
• Interpretation: 88% chance of repayment

Key Insight: The same credit score change has different effects at different probability levels — the non-linear nature of these transformations captures the reality that moving from 50% to 60% probability is different from moving from 90% to 100%.

Conclusion

The journey from probabilities to odds, through logarithms and sigmoid curves, isn’t just mathematical elegance — it’s practical necessity. These transformations allow us to:

1. Apply linear modeling to non-linear problems
2. Ensure predictions stay within valid probability bounds
3. Capture the nuanced relationship between predictors and binary outcomes

Whether you choose logistic or probit regression, understanding these foundations empowers you to build better models, interpret results correctly, and tackle increasingly complex classification challenges. As you advance in machine learning, you’ll find these concepts recurring in neural networks, deep learning, and beyond — making them truly foundational knowledge for any data scientist.