Time Series Forecasting Metrics: A Practical Guide

How to pick, interpret, and defend the right error metrics for your forecasts

[embed]Google Colab Edit descriptioncolab.research.google.com

This notebook contains all the components covered in the article. It is recommended to use it to reproduce and practice everything discussed here.

For more time series related content, you can also check github repository

If you build forecasts, you’ll get asked two questions:

“How good is it?”

“Good for what?”

This article answers both.

1. Define the main time-series metrics (with plain-English intuition).
2. Map metrics to real-world scenarios (intermittent demand, zeros, probabilistic forecasts, etc.).
3. End with a short, opinionated shortlist you can use by default.

## Notation (keep it simple)

- `y_t`: actual at time t
- `ŷ_t`: forecast at time t
- `e_t = y_t − ŷ_t`: residual / error
- N: number of forecasted points
- `naive_t`: naive baseline

Part 1 — The Metrics (what they measure, when they fail)

1.1. Absolute & Squared Error Family

• MAE = mean absolute error Robust to outliers vs MSE. Great for median-like behavior.
• MSE = mean squared error — Penalizes large errors heavily. Smooth, differentiable. Sensitive to outliers.
• RMSE = sqrt(MSE) — Same units as target. Popular when large misses are especially costly.

Since this is not Machine Learning 101, I assume everyone reading this article are familiar with this topic.

1.2. Percentage Error Family

Absolute and squared error metrics (like MAE and RMSE) have a key limitation: they are Not comparable across scales.

1. RMSE gives the same value regardless of scale — whether the target series is around 10, 100, or 1000. As you can see the first plot below.
2. But the meaning of the same error changes with scale — an error of 2 is very large when the true value is 10, but almost negligible when the true value is 1000.
3. Unfortunately, RMSE does not reflect this difference — it treats both cases as equally severe, even though their practical impact is very different.

To address this, percentage-based metrics are introduced:

1. Percentage-based metrics normalize the error by the magnitude of the target — for example, dividing by y_t or by the average of y_t and ŷ_t.
2. This makes the error relative rather than absolute — the same raw error will appear smaller when the target is large, and larger when the target is small.
3. As a result, we can fairly compare performance across different scales — and better judge whether an error is truly significant in context. As you can see from the plot above.

However, this type metrics would be a disaster when y_t/ŷ_t are near 0.

MAPE
- Explodes near zero, 
- undefined at `y_t=0`, 
- biases against under-forecasting when `y_t` small.

MASE
- Less extreme than MAPE, but still quirky when both are small.

When to use:

• Avoid if zeros/small values exist.
• Consider only for strictly positive data with meaningful percent interpretation.

1.3. Extrinsic Metrics Family (compare to a naive baseline)

One of the my favorite metrics in regression is R², which expresses how much of the variance in the data is explained by the model. While it doesn’t directly show how close predictions are to actual values, it does indicate how well the model performs relative to the inherent variability of the data.

In time series forecasting, we have a similar concept through scaled error metrics (such as MASE).

• Instead of comparing explained variance, these metrics compare your model’s error against a simple baseline forecast (like a naive or seasonal naive model).
• This allows us to judge whether a carefully tuned model actually outperforms a trivial benchmark, giving a relative sense of usefulness just like R² does in regression.
• To be noticed, scaled error is based on in-sample MAE, which means the trainning data results.

Below example explained this difference:

Case 1 — Flat series (easy):

1. The series barely changes → a (seasonal) naïve baseline already performs extremely well.
2. Advanced model’s MAE is low in absolute terms, but the baseline’s MAE is even lower.
3. Result: MASE = MAE_model / MAE_baseline ≥ 1 (i.e., “high”). The task is trivial, and the model doesn’t beat the baseline.

Case 2 — Volatile series (hard):

1. The series fluctuates a lot → the naïve baseline performs poorly.
2. Your model still has a relatively large MAE (because the data are noisy). However, it reduces error substantially versus the baseline.
3. Result: MASE < 1 (i.e., “low”). Despite a big MAE, the model is strong relative to task difficulty.

1.4. Weighted Aggregations

WAPE** (Weighted Absolute Percentage Error), also called MAE over sum:

WAPE = (∑|e_t|) / (∑|y_t|)

— Handles zeros better than MAPE (still problematic if total demand near zero). — Weight by revenue, volume, or hour-of-day importance. “Across all products/periods, our forecasts were off by 8.78% of total demand.”

1.5. Bias & Directionality

Forecast bias measures whether a model consistently over-forecasts or under-forecasts.

1. Over-forecasting → the predicted values tend to be higher than the actual values.
2. Under-forecasting → the predicted values tend to be lower than the actual values.
3. By tracking bias, we can see not just how large the errors are, but whether the model systematically leans in one direction.

Part 2 — Which Metric for Which Scenario?

2.1. Single series, no zeros, comparable scale

For the simplest case — when (1) there is only one time series, (2) its values stay within a narrow range, and (3) there are no zeros or near-zeros — you can rely on the simplest error metrics.

2.2. Many series, different scales | Prefer MASE & RMSSE

When you are dealing with multiple time series of very different magnitudes (for example, stock prices ranging from $10 to $1000), using absolute error metrics such as MAE or RMSE is not a good choice.

As the table shows, MAE and RMSE naturally increase with the scale of the series:

• For Series A (~10s), a MAE of 1.3 is quite large relative to the scale.
• For Series C (~1000s), a MAE of 20.9 looks big in absolute terms, but is actually tiny compared to the magnitude of the series.

Summary metrics:
              Series     MAE    RMSE   MASE  RMSSE
0    Series A (~10s)   1.266   1.581  1.030  1.164
1   Series B (~100s)   6.383   7.191  1.039  1.055
2  Series C (~1000s)  20.934  25.319  1.060  1.071

To make results comparable across series, you should include scaled metrics such as MASE or RMSSE. These normalize errors relative to a baseline, and as shown in the table, they remain close to 1 across all series — clearly indicating that forecast performance is consistent regardless of magnitude.

2.3. Zeros or near-zeros present

The biggest limitation of percentage error metrics is that they cannot handle values near zero. Since the actual value appears in the denominator, if it is very small or zero, the error explodes (as shown in the example below).

When series near Zeros:
  MAE RMSE MAPE      RMSPE   WAPE  MASE RMSSE
  1.1 1.237 482.932 809.401 27.16 0.137 0.146

When series contain Zeros:
   MAE RMSE  MAPE RMSPE  WAPE   MASE RMSSE
 1.125 1.275 NaN   NaN  28.125 0.138 0.149

In such cases, percentage error is not recommended. Symmetric percentage error reduces the problem somewhat by normalizing against both actual and forecast values, but it still remains vulnerable when values are close to zero.

2.4. Stakeholders want “% of demand” summary

It is highly recommended to use WAPE (Weighted Absolute Percentage Error) since it ties directly to business impact.

With WAPE, you can clearly communicate results in business terms. For example:

“Across all products and periods, our forecasts were off by 8.78% of total demand.”

This phrasing not only explains model performance but also translates it into a meaningful business insight that stakeholders can immediately understand.

Part 3 — A defensible shortlist (what to use by default)

If you must pick just a few:

1. Classical

• MAE (Mean Absolute Error)
• RMSE (Root Mean Squared Error)

2. Scale-independent / Percentage-based

• sMAPE (Symmetric Mean Absolute Percentage Error)
• WAPE (Weighted Absolute Percentage Error)

3. Benchmark-relative

• MASE (Mean Absolute Scaled Error)
• RMSSE (Root Mean Squared Scaled Error)

4. Over/under-forecasting

• Forecast Bias (FB)
• Tracking Signal (TS)

[embed]time-series-analysis-toolkit/src/Evaluation/metrics.py at main ·… A comprehensive toolkit for time series analysis, including scripts for visualizing results, detecting stationarity…github.com

This github code contains all above metrics, you can directly used in your project.

Practice

If interested, you can use this real world example to better practice these metrics.

Reference

[embed]Forecast Evaluation for Data Scientists: Common Pitfalls and Best Practices Machine Learning (ML) and Deep Learning (DL) methods are increasingly replacing traditional methods in many domains…arxiv.org

This paper listed a comprehensive method to find the appropriate metrcis based on your business requirement.