VM‑UniqThm: The Uniqueness Law for L2‑Regularized Logistic Regression

By: Valarmathi Ganessin VM 01/01/2026

Abstract

Logistic regression is one of the most widely used models in data science for binary classification. Despite its simplicity, practitioners often observe unstable behaviour when the model is trained without regularization: different optimizers, different initializations, or even different data shuffles can lead to different weight solutions and therefore different predictions.

This paper provides a clean, intuitive, example‑driven explanation of why L2‑regularized logistic regression always converges to a unique, stable, reproducible solution. We introduce the VM‑UniqThm, a pedagogical framing that connects strict convexity to practical data‑science stability. Through one‑feature and three‑feature examples, visual diagrams, and human‑readable formulas, we demonstrate how L2 transforms logistic regression from an unstable model into a mathematically well‑posed and reproducible system.

“Same weights → same probabilities → same predictions → same performance.”

1. Introduction

Logistic regression is foundational in machine learning, powering applications such as fraud detection, medical diagnosis, credit scoring, and risk modelling. However, logistic regression without regularization can behave unpredictably:

• Weights may diverge
• Solutions may not be unique
• Predictions may change across runs
• Optimizers may converge to different points

This instability is especially visible when:

• Features are correlated
• Data is linearly separable
• The model is high‑dimensional

L2 regularization fixes all of this.

This document explains why, using:

• Intuition
• Examples
• Visuals
• Proof
• Practical interpretation

This is the essence of VM‑UniqThm.

“In real‑world systems, reproducibility is not optional — it is essential.”

2. What Is Logistic Regression?

Logistic regression predicts a probability:

p = σ(wᵀx)

where the sigmoid function is:

σ(z) = 1 / (1 + e^(−z))

Sigmoid Visual

Decision Rule

• If p > 0.5 → Class 1
• If p ≤ 0.5 → Class 0

3. Logistic Loss Function

For one sample:

ℓ(w; xᵢ, yᵢ) = −[yᵢ · log(σ(wᵀxᵢ)) + (1 − yᵢ) · log(1 − σ(wᵀxᵢ))]

Total loss:

L(w) = Σᵢ₌₁ⁿ ℓ(w; xᵢ, yᵢ)

Loss Shape (Convex but not strictly convex)

Loss

This “flat bottom” is the reason logistic regression can have multiple solutions.

4. One‑Feature Example (Before L2)

Dataset:

Try different weights:

Observation

The loss is almost flat around w = 1 to 3.

Different optimizers may land at different points.

Visual

Code

Loss

This is why logistic regression without L2 is unstable.

5. Three‑Feature Example (Before L2)

Try weight vectors:

Observation

Many weight vectors give almost the same loss.

This means:

• No unique solution
• Different runs → different weights
• Predictions may change

6. What Is Regularization?

Regularization adds a penalty to the loss to control model complexity.

Why we need it

• Prevent overfitting
• Prevent weight explosion
• Improve generalization
• Improve stability
• Ensure uniqueness

Where it is used

• Logistic regression
• Linear regression
• Neural networks
• SVMs
• Any model with weights

7. L1 vs L2 Regularization

L1 Regularization

λ‖w‖₁ = λ ( |w₁| + |w₂| + |w₃| + … )

• Produces sparse weights
• Performs feature selection
• Not strictly convex
• Does not guarantee uniqueness

L2 Regularization

(λ / 2) · ‖w‖²

• Smooth penalty
• Shrinks weights
• Keeps all features
• Strictly convex
• Guarantees unique solution

Visual Comparison

Code

“L1 chooses features; L2 stabilizes models.”

8. L2 Regularization in Logistic Regression

The new loss:

L(w) = ∑_{i=1}^{n} ℓ(w; x_i, y_i) + (λ/2)‖w‖²

Effect

• Removes flat regions
• Makes the loss strictly convex
• Guarantees one unique minimum

Visual

Code

“Small weights create simpler models, and simpler models overfit less — this is why L2 improves generalization.”

9. One‑Feature Example (After L2)

Using the same dataset:

Observation

Now the loss has one clear minimum at w = 1.

Visual

Code

Loss

Every optimizer will land at w = 1.

One-Feature Loss Curve (No L2 vs L2)

Key Insight: Without L2, gradient descent paths diverge to different solutions depending on initialization. With L2, all paths converge to the same unique minimum, confirming the stability guaranteed by VM‑UniqTh

Figure 1: Logistic Loss Curve Before and After L2 Regularization Loss vs weight (w) for x = [1, 2, 3], y = [0, 0, 1]. Without L2, the curve is flat and admits multiple minimizers. With L2 (λ = 1), the curve becomes strictly convex, enforcing a unique solution.

Key Insights:

• Without L2: flat regions → multiple valid solutions
• With L2: steep, convex curve → unique minimizer
• Regularization stabilizes optimization and improves reproducibility

Key Insight: Without L2, gradient descent paths diverge to different solutions depending on initialization. With L2, all paths converge to the same unique minimum, confirming the stability guaranteed by VM‑UniqTh

Key Insight: Without L2, the loss curve is flat around the minimum, allowing many different weight values to achieve nearly the same loss. With L2, the curve becomes steep and strictly convex, forcing the optimizer toward one unique, stable solution.

10. Three‑Feature Example (After L2)

Try weight vectors:

Observation

Unique minimum at (1, 1, 1).

Every optimizer → same weights → same predictions.

mage 2: 3D Loss Surfaces (No L2 vs L2)

Figure 2: Loss Surface Comparison for Three-Feature Logistic Regression

Left: Without L2, the surface is shallow with broad valleys. Right: With L2, the surface is sharply convex, penalizing large weights and enforcing uniqueness.

Key Insights:

• No L2: flat valleys → optimizer may wander
• With L2: steep curvature → optimizer converges reliably
• L2 transforms the loss landscape into a well-posed basin

11. Logistic Regression WITHOUT L2 vs WITH L2

Table 1: Demonstrating Stability Before and After L2 Regularization (3-Feature Logistic Regression)

1. Without L2 Regularization (λ = 0): Optimization is fragile and inconsistent

• Different optimizers (GD, SGD, Momentum) yield different final weights for the same dataset.
• Changing the initialization leads to different predictions and loss values.
• The model is sensitive to optimizer noise and data shuffling, especially with SGD.
• This confirms that the loss surface is not strictly convex, allowing multiple minimizers.

2. With L2 Regularization (λ = 1): Optimization becomes stable and reproducible

• All optimizers converge to the same final weight vector, regardless of initialization.
• Predictions and loss values are identical across all runs.
• The model becomes robust to optimizer choice and random shuffling.
• This demonstrates that L2 regularization makes the loss strictly convex, enforcing uniqueness.

3. Momentum and SGD behave like GD once L2 is applied

• Even noisy optimizers like SGD converge to the same solution.
• This shows that regularization dominates optimizer variance, stabilizing the training process.

4. Practical Implication

• Without L2: model is unpredictable, initialization-dependent, and non-reproducible.
• With L2: model is stable, well-posed, and scientifically reproducible.
• This table visually confirms the VM-UniqThm: L2 regularization guarantees uniqueness in logistic regression.

This is the heart of VM‑UniqThm.

Figure 3: Animated Transition from Unregularized to Regularized Loss Curve Interpolated loss curve showing how L2 regularization steepens and convexifies the landscape. The morph illustrates the continuous transformation from unstable to stable optimization.

Key Insights:

• Visualizes the effect of increasing λ
• Shows how regularization reshapes the loss
• Highlights the path from ambiguity to uniqueness

Table2: Loss & Predictions for Different Weight Values

. L2 regularization increases the loss for large weights

• As ∣w∣ grows, the L2 penalty λ2∥w∥2 becomes dominant.
• This is why the L2 loss is much higher for w=−4,−2,2,3.
• It discourages extreme weight values and stabilizes the model.

2. Predictions remain identical with and without L2

• For each value of w, the predicted probabilities are exactly the same.
• This is because L2 affects the loss, not the forward pass.
• The model’s mapping Xw→σ(Xw) is unchanged.

3. L2 influences optimization, not inference

• Even though predictions are identical, the optimization landscape changes dramatically.
• L2 makes the loss surface steeper and more convex, which leads to:
• unique minimizers
• stable convergence
• reproducible training outcomes

4. Practical takeaway

• Without L2: the loss is flatter → multiple solutions possible.
• With L2: the loss penalizes large weights → optimization becomes well‑posed.
• This table provides the first numerical evidence supporting VM‑UniqThm.

Gradient Descent Convergence: Comparing Final Weights With and Without L2 Regularization (3-Feature Model)

Table: Final Weights After Gradient Descent

1. Without L2 Regularization (λ = 0): Final weights depend on initialization

• Each initial weight vector leads to a different final solution.
• The optimizer converges to different local minima, even though the loss function is smooth.
• This confirms that the unregularized logistic loss is not strictly convex, allowing multiple minimizers.

2. With L2 Regularization (λ = 1): Final weights are identical across all initializations

• All three initializations converge to exactly the same final weight vector.
• This demonstrates that L2 regularization makes the loss strictly convex, enforcing a unique global minimum.
• The optimizer becomes initialization-independent, which is critical for reproducibility.

3. Practical Implication

• Without L2: model behavior is unpredictable and sensitive to initialization.
• With L2: model behavior is stable, reproducible, and mathematically well-posed.

This table provides direct numerical evidence for the VM‑UniqThm:

• L2 regularization guarantees uniqueness in logistic regression.

Python Libraries Required

To reproduce all experiments, visualizations, and tables in this article, the following Python libraries are required:

numpy matplotlib scikit-learn jupyter

Install all dependencies using:

pip install numpy matplotlib scikit-learn jupyter

GitHub Repository (request access if needed): https://github.com/Srivalarmathi/L2-Regularisation-for-Logistic-Regression

12. VM‑UniqThm (Main Theorem)

Theorem:

For any λ > 0, the L2‑regularized logistic regression loss is strictly convex and therefore has a unique global minimizer.

13. Proof (Readable)

Step 1 — Logistic loss is convex

d²ℓ(z) / dz² = σ(z) · (1 − σ(z)) ≥ 0

Step 2 — L2 penalty is strictly convex

∇²[(λ / 2) · ‖w‖²] = λ · I

Step 3 — Sum is strictly convex

Convex + strictly convex = strictly convex.

Step 4 — Strict convexity → unique minimizer

Therefore logistic regression with L2 has one unique solution.

14. Practical Data‑Science Interpretation

L2 gives:

• Stable training
• Reproducible results
• Predictable behaviour
• No optimizer sensitivity
• No weight explosion
• Better generalization

This is why every production logistic regression model uses L2.

“A unique solution means the model behaves the same everywhere — same optimizer, same shuffle, same randomness, same outcome.”

15. Related Work

Logistic regression has been extensively studied in statistics, machine learning, and convex optimization. Classical texts on generalized linear models establish the convexity of the logistic loss and discuss regularization as a means to prevent overfitting. Optimization literature further analyses the role of L2 regularization in ensuring numerical stability and improving generalization. However, most treatments present these results in a highly mathematical or abstract form, without connecting them to the practical behaviour observed by data‑science practitioners.

Several works discuss the non‑uniqueness of logistic regression solutions in the absence of regularization, particularly in the presence of multicollinearity or linearly separable data. Other studies highlight the empirical benefits of L2 regularization but do not provide a unified, intuitive explanation of why the solution becomes unique. To the best of our knowledge, no prior work has presented a clear, example‑driven, and pedagogically structured explanation linking strict convexity to reproducibility in logistic regression. This gap motivates the VM‑UniqThm formulation.

“Unlike prior work, VM‑UniqThm unifies theory, intuition, and practical behaviour into a single explanatory framework.”

16. Novelty Statement

While the strict convexity of L2‑regularized logistic regression is known in theory, existing explanations are either purely mathematical or disconnected from practical machine‑learning workflows. The novelty of this work lies in presenting the result in a unified, intuitive, and accessible manner. We introduce the VM‑UniqThm as a pedagogical framework that connects the mathematical property of strict convexity to the practical outcomes of stability, reproducibility, and consistent predictions. By combining formal proof, geometric intuition, and concrete numerical examples, this work provides a uniquely clear and practitioner‑friendly understanding of why L2‑regularized logistic regression always converges to a single, stable solution.

17. Contribution Summary

This work makes the following contributions:

A clear and intuitive explanation of logistic regression

1. We present the model, sigmoid function, and loss formulation in a way that is accessible to both beginners and practitioners.

A unified view of regularization

2. We explain why regularization is needed, how L2 differs from L1, and how these choices affect model behaviour.

A rigorous yet simple proof of strict convexity

3. We show that adding the L2 penalty makes the total loss strictly convex, guaranteeing a unique global minimizer.

Practical examples demonstrating uniqueness

4. Using 1‑feature and 3‑feature examples, we illustrate how L2 regularization leads to stable, reproducible solutions.

The VM‑UniqThm formulation

5. We introduce a named, structured explanation that connects theory to practice, making the concept easier to teach, understand, and apply.

18. Future Work

Several directions can extend the ideas presented in this work:

Multi‑class logistic regression

1. Extending the VM‑UniqThm framework to SoftMax regression and analysing strict convexity in higher‑dimensional probability spaces.

Generalized linear models (GLMs)

2. Investigating whether similar uniqueness guarantees hold for other GLMs under L2 regularization.

Alternative regularization schemes

3. Exploring the uniqueness properties of elastic net, group lasso, and non‑convex penalties such as SCAD or MCP.

Optimization algorithms

4. Studying how different solvers behave under strictly convex objectives and whether convergence rates improve with L2 regularization.

Practical reproducibility studies

5. Empirically validating VM‑UniqThm across real‑world datasets, feature scales, and training pipelines.

“Understanding uniqueness in other regularized models may lead to a broader theory of reproducible machine learning.”

19. Conclusion

With L2 regularization, logistic regression always converges to the same weight vector — and the same weights produce the same probabilities, the same predictions, and the same performance, regardless of optimizer, data shuffle, or training conditions. Small, stable weights create a simple model that generalizes well and avoids overfitting. This is the practical essence of the VM‑UniqThm.

Same weights → same probabilities → same predictions → same performance.

L2 makes logistic regression stable, simple, and uniquely reproducible.

This is the essence of VM‑UniqThm.

Valarmathi Ganessin VM, Author of VM‑UniqThm