Beyond the Average: Understanding Quantile Regression (and Why IRLS Matters in Production)

Most machine learning systems are built to answer a deceptively simple question: “What is the expected outcome?”

This works well — until it doesn’t.

In many real-world systems, the average case is not what hurts you. What hurts you is the tail:

• The delivery that arrives much later than expected
• The system request that takes far longer than usual
• The financial outcome that deviates sharply from the norm

If you only model the mean, you are effectively blind to these risks. This is where quantile regression becomes essential. It allows you to directly model the distribution of outcomes, especially the parts that matter most when things go wrong.

In this post, we’ll go deeper into:

• What quantile regression really means (beyond textbook definitions)
• How it works and why it behaves differently from standard models
• Where it shows up across industries
• What makes it tricky in production systems
• And how Iteratively Reweighted Least Squares (IRLS) helps make it practical

Rethinking the Prediction Problem

When we train a standard regression model using mean squared error, we are implicitly asking the model to estimate:

In other words, we want the average outcome conditioned on features.

This works if:

• The data is symmetric
• The variance is stable
• The cost of overestimation and underestimation is the same

But in practice, none of these assumptions hold.

Imagine predicting delivery times. If your model predicts 30 minutes on average, but occasionally the delivery takes 90 minutes, the average is not particularly helpful for user experience. What users care about is something closer to: “How long will it take in the worst reasonable case?”

Quantile regression answers exactly that.

This is the τ-th quantile of the outcome. For example:

• τ = 0.5 → median
• τ = 0.95 → 95th percentile
• τ = 0.99 → 99th percentile

A p95 prediction tells you: “95% of outcomes will fall below this value.”

This is fundamentally a different question — and often a more useful one.

Why Quantiles Change How Models Behave

The key to quantile regression lies in how it treats errors.

In standard regression, errors are squared. This makes the model care equally about overestimation and underestimation. But in many business scenarios, these two types of errors are not equally costly.

Consider again the delivery example:

• If you overestimate delivery time (say 50 minutes instead of 40), the user might be pleasantly surprised
• If you underestimate (say 30 minutes but it takes 50), the user is frustrated

Quantile regression encodes this asymmetry directly into the loss function.

It uses what is known as the pinball loss, which penalizes underestimation and overestimation differently depending on the quantile being targeted. For high quantiles like p95:

• Underestimating is penalized heavily
• Overestimating is penalized lightly

This shifts the model’s behavior. It becomes intentionally conservative, pushing predictions upward to ensure coverage of most outcomes.

A helpful way to think about this is:

• Mean regression tries to be accurate on average
• Quantile regression tries to be safe with respect to a target percentile

A More Concrete Example

Let’s say you are modeling system latency.

A mean model might tell you:

“Average latency is 120ms”

But your users don’t experience averages — they experience individual requests. If 5% of requests take 800ms, those are the ones that define perceived performance.

A p95 model might predict:

“95% of requests will complete within 400ms”

A p99 model might say:

“99% of requests will complete within 900ms”

These numbers directly inform:

• SLA commitments
• Capacity planning
• Alerting thresholds

The mean alone cannot do that.

Where Quantile Regression Shows Up in Practice

Quantile regression is widely used in domains where risk, reliability, or variability matter.

In logistics and operations, it helps estimate late deliveries and build buffers into planning systems. Instead of optimizing for average delivery time, companies optimize for consistency and predictability.

In infrastructure and distributed systems, p95 and p99 latency are standard metrics. Engineering teams design systems not for the average request, but for the tail behavior that affects user experience.

In finance, quantile-based measures like Value at Risk (VaR) are foundational. These explicitly focus on worst-case scenarios over a given time horizon.

In marketplaces and pricing systems, quantile models help anticipate high-demand or high-cost situations, enabling safer pricing strategies and avoiding underestimation of costs.

Even in healthcare, quantile regression is used to model extreme outcomes, such as unusually long hospital stays, which are critical for resource planning.

Across all these domains, the common theme is clear:

When variability matters, modeling the distribution is more important than modeling the mean.

The Computational Challenge

While the idea of quantile regression is conceptually simple, it is computationally more challenging than standard regression.

The pinball loss is not smooth in the same way as squared error, and it does not lead to a clean closed-form solution. This means we need iterative optimization methods.

One particularly elegant and practical approach is Iteratively Reweighted Least Squares (IRLS).

Understanding IRLS Through Intuition

IRLS is based on a powerful idea:

If a problem is hard to solve directly, solve a sequence of easier problems that approximate it.

In this case, the “easy problem” is least squares, which we know how to solve efficiently and at scale.

IRLS works as follows:

• Start with an initial model
• Look at how wrong the predictions are (the residuals)
• Assign a weight to each data point based on those errors
• Solve a weighted least squares problem
• Repeat the process with updated weights

Over time, the weights adjust in a way that makes the solution converge to the desired objective — in this case, the quantile loss.

How IRLS Enables Quantile Regression

For quantile regression, the weights are designed to reflect the asymmetry of the pinball loss.

Points that are under-predicted (i.e., actual value is above prediction) receive higher weight when modeling high quantiles like p95. This forces the model to pay more attention to these cases and push predictions upward.

Points that are over-predicted receive lower weight, meaning the model is less concerned about them.

Through repeated iterations, this weighting scheme reshapes the least squares objective into something that closely approximates quantile regression.

An intuitive way to interpret IRLS in this context is:

The model gradually learns which mistakes matter more — and focuses on correcting those.

Why IRLS Matters in Production Systems

From a production standpoint, IRLS is valuable because it allows teams to reuse well-optimized least squares solvers, which are often highly scalable and numerically stable.

It also provides a level of interpretability. By inspecting the weights, you can understand which data points are influencing the model most strongly. This can be useful for debugging and for building trust with stakeholders.

However, IRLS is not without its challenges. Convergence can be sensitive to initialization, and performance can degrade when modeling extreme quantiles like p99, where data is sparse and noisy.

What Makes Quantile Regression Tricky in Production

While quantile regression is powerful, deploying it in real systems requires careful consideration.

One of the biggest challenges is tail noise. By definition, p95 or p99 focuses on a small fraction of the data. This makes estimates inherently noisy and sensitive to outliers. In practice, this can lead to instability across model retraining cycles.

Another issue is quantile crossing, where independently trained models produce inconsistent outputs (e.g., a p95 prediction lower than a p50 prediction). This violates basic logic and requires additional constraints or post-processing to fix.

Data sparsity is also a concern, especially at granular levels such as per-user or per-item predictions. In these cases, there may not be enough data to reliably estimate tail behavior. Teams often address this using hierarchical models or by backing off to more aggregated levels.

Distribution shift poses another challenge. Because quantile models focus on the tails, they are particularly sensitive to changes in the underlying distribution. This makes monitoring and retraining more critical compared to mean-based models.

Finally, evaluation requires a different mindset. Traditional metrics like MSE are no longer appropriate. Instead, teams must measure calibration, ensuring that the predicted quantiles actually match observed frequencies over time.

A Practical Mental Model

For practitioners and product managers alike, it helps to reframe quantile regression in simple terms:

• Mean models answer: “What usually happens?”
• Quantile models answer: “What happens when things go wrong?”

This shift is subtle but profound.

It enables:

• Safer automation
• More reliable systems
• Better user experience under uncertainty

When Should You Use Quantile Regression?

Quantile regression is particularly valuable when:

• Tail risk matters more than average performance
• You need to provide guarantees (e.g., SLAs)
• The data distribution is skewed or heavy-tailed

However, it may not be the right tool when:

• Data is limited and tails are too noisy
• Stability is more important than conservativeness
• The added complexity cannot be justified by business impact

Final Thoughts

Quantile regression represents a shift in how we think about prediction.

Instead of optimizing for the average case, it allows us to design systems that are robust to variability and resilient to worst-case scenarios.

And while the underlying mathematics can be complex, tools like Iteratively Reweighted Least Squares (IRLS) make it practical to implement at scale.

In modern production systems, where reliability and user trust are critical, this is not just a modeling choice.

It is a product decision about how much risk you are willing to take — and how well you can manage it.