Building a hybrid demand forecasting service: classical ML for the number, an LLM for the why

Your gradient-boosted model says unit sales for a product will jump 30% next week. The category strategist asks one question: why? A pure forecaster has no answer. It returns a number, not a reason.

This article is a design blueprint for a retail demand forecasting service where classical ML produces the number and an LLM produces the context around it. I have not deployed this. Everything below is how I would wire it, the trade-offs I would weigh, and the Azure services I would target. The code is illustrative, not output I ran. I am still ramping up on the Microsoft stack, so I will be explicit about what is documented versus what I have actually touched (nothing, on Azure, yet).

The trap: retail forecasting is not a “normal” ML problem

The tempting move is to load a sales dataset, throw it at XGBoost with a train/test split, and read off an R². That produces a number that looks good and lies to you.

Two things make retail demand different from a generic tabular problem:

• Time has a direction. A random train/test split puts next month’s data into training and last month’s into test. The model literally sees the future, and your error estimate is fantasy.
• The number is not the deliverable. A strategist acting on a forecast needs to know whether the spike is a promo, a holiday, a supply story, or noise. That explanation is not something a regression coefficient hands you.

So the design has two halves that get evaluated separately: a time-series model that has to survive honest backtesting, and an LLM layer that either feeds the model interpretable signals or narrates its output. Neither half is optional, and neither excuses sloppiness in the other.

The data problem nobody mentions

Here is the honest starting point. No single well-known Kaggle dataset gives you both a clean demand time series and matching free-form text.

The canonical demand competitions carry rich structured signals but essentially no prose:

• Rossmann Store Sales: daily sales for 1,115 drugstores. The target is Sales; the signals are Promo, Promo2, PromoInterval, StateHoliday (a/b/c/0), and SchoolHoliday. No text column.
• Corporación Favorita: Ecuadorian grocery chain, forecasting unit_sales per (date, store_nbr, item_nbr) 16 days out, with store metadata, item metadata, transaction counts, and daily oil price. Around 125.5M training rows across 54 stores (treat that count as approximate until re-checked on the data page).
• M5 (Walmart): hierarchical daily sales, 28 days ahead, 3,049 products across 3 categories and 10 stores in CA/TX/WI, 42,840 series across 12 aggregation levels. The competition metric was Weighted RMSSE. The calendar ships event_name/event_type as categoricals, not text to summarize.

The review datasets are the mirror image: Amazon review sets carry review text, a 1 to 5 star rating, helpfulness, and timestamps, but no store-level demand.

That tension is the whole point, so I will not hand-wave it. There are two honest paths.

Option A, self-contained (my default for reproducibility). Use a review dataset and forecast a derived series. The McAuley Lab Amazon Reviews 2023 set has 571.54M reviews across 48M items and 33 categories, spanning May 1996 to September 2023, with second-level timestamps. Those timestamps let you build a per-product or per-category review-volume series and a mean-rating trend, then enrich each period with sentiment and aspects extracted from the same reviews. The text and the series come from one source, no external join. (The dataset card does not state a license in an easy-to-quote form. Verify licensing before you redistribute anything.)

Option B, explicit join (clearly labeled as a design choice). Take a real demand series (Favorita or Rossmann) and join a text source. The lowest-friction real text is Favorita’s holidays_events.csv, which ships an actual description field plus type (Holiday, Transfer, Additional, Bridge, Work Day, Event), locale, and a transferred boolean. The docs' own example: "Independencia de Guayaquil" was transferred from 2012-10-09 to 2012-10-12, and a transferred holiday behaves more like a normal day. This is the strongest case of text living inside a demand dataset, but be honest about what it is: short calendar-event labels, not narrative reviews. Any richer external join (news headlines, for example) is a design decision you own, and you must cite a concrete source rather than pretend a dataset ships one.

The one thing you cannot do is claim a single mainstream dataset conveniently has both. It does not.

Feature engineering for time series

Whichever path you take, the classical model eats tabular features. The three families that matter for demand are lags, rolling windows, and calendar signals, plus event flags from whatever text join you chose.

import pandas as pd

def build_features(df: pd.DataFrame, target: str = "unit_sales") -> pd.DataFrame:
    # df is sorted by date, one row per (date, series_id)
    g = df.groupby("series_id")[target]

    # 1. Lags: yesterday, one week back, four weeks back
    for lag in (1, 7, 28):
        df[f"lag_{lag}"] = g.shift(lag)

    # 2. Rolling windows on the trailing history.
    #    shift(1) first so today's value never leaks into its own feature.
    shifted = g.shift(1)
    for window in (7, 28):
        df[f"roll_mean_{window}"] = shifted.rolling(window).mean()
        df[f"roll_std_{window}"] = shifted.rolling(window).std()

    # 3. Calendar / seasonality
    df["dow"] = df["date"].dt.dayofweek
    df["month"] = df["date"].dt.month
    df["is_weekend"] = df["dow"].isin([5, 6]).astype(int)

    # 4. Event flag from the joined holidays_events description (Option B)
    df["is_event"] = df["event_type"].notna().astype(int)

    return df

The one line that separates a working feature set from a leaky one is shift(1) before the rolling window. A rolling(7).mean() computed straight on the target includes today's value, which the model does not have at prediction time. Shift the series back one step first, then roll. Every lag and window here is computed only from the past, which is exactly what the model will have in production.

The classical model, and backtesting that does not lie

Use gradient boosting (XGBoost or LightGBM) for the tabular features, or Prophet if you want an additive trend/seasonality decomposition. Cite their official docs for anything version-specific; I am treating them as interchangeable engines here.

The part that earns trust is validation. For time series, the split has to respect order. Scikit-learn’s TimeSeriesSplit produces train/test indices in time order: in the k-th split the first k folds are training and the (k+1)-th is test, so each training set is a superset of the previous one. That is expanding-window walk-forward, the same idea Hyndman and Athanasopoulos call evaluation on a rolling forecasting origin.

A shuffled split lets the model peek at the future; expanding-window walk-forward keeps training in the past and testing in the future, with gap=7 as a leakage buffer.

from sklearn.model_selection import TimeSeriesSplit
from sklearn.metrics import mean_absolute_error
import numpy as np

# 5 expanding-window folds, with a 7-day gap so the test set never
# borrows from rows adjacent to training (a leakage buffer).
tscv = TimeSeriesSplit(n_splits=5, gap=7)

maes = []
for train_idx, test_idx in tscv.split(X):
    model.fit(X.iloc[train_idx], y.iloc[train_idx])
    pred = model.predict(X.iloc[test_idx])
    maes.append(mean_absolute_error(y.iloc[test_idx], pred))

print(f"walk-forward MAE per fold: {np.round(maes, 2)}")

The gap=7 parameter inserts a buffer between train and test so dependent samples right at the boundary do not leak. Why bother with all this instead of a shuffled split? Random shuffling scatters future observations into training and past ones into test. The model sees the future, and your generalization estimate is worthless. Time-ordered splitting is the only way to get an honest read.

A number in isolation still tells you nothing. Always report against a baseline. The two standard reference forecasts are naïve (repeat the last value) and seasonal-naïve (repeat the value from one season back). Then use a metric that is anchored to that baseline:

• MAE, MSE, RMSE are scale-dependent. Fine within one series, useless for comparing across products with different volumes.
• MAPE is scale-independent but only sensible when values are well above zero. It is undefined at zero and explodes near it, which is common in intermittent retail demand.
• sMAPE is a symmetric variant of MAPE.
• MASE scales the error by the in-sample MAE of the (seasonal-)naïve method. MASE = 1 means “as good as naïve,” below 1 beats it, above 1 is worse. It is scale-free, comparable across series, and defined even for intermittent or near-zero data. Hyndman and Koehler (2006) recommend it as the best general-purpose accuracy measure.
• WAPE shows up in retail practice for volume-weighting error across items. I would pin down a primary definition before printing its formula, since the sources I trust for the others do not define it.

Here is MASE in code, so the baseline is not just talked about but computed:

import numpy as np

def mase(y_true, y_pred, y_train, season: int = 7) -> float:
    # denominator: in-sample MAE of the seasonal-naive forecast
    naive_errors = np.abs(y_train[season:] - y_train[:-season])
    scale = naive_errors.mean()
    return np.mean(np.abs(y_true - y_pred)) / scale

If your fancy model cannot beat seasonal-naïve on MASE, you do not have a forecasting service. You have a slower way to repeat last week.

Gluing the LLM to the ML

The LLM enters in two distinct roles. Keep them separate, because they are evaluated differently.

Role 1: the LLM as a feature generator. This is the part with peer-reviewed backing. Balek, Sýkora, Sklenák, and Kliegr (2024) used an LLM to extract a small set of human-interpretable features from text (things like methodological rigor, novelty, grammatical correctness). On citation-rate prediction (CORD-19) and a 5-class expert grade (M17+), 62 LLM-generated features matched SciBERT’s 768-dimensional embeddings while staying interpretable and usable by rule learners. The motivation is exactly ours: embeddings are high-dimensional and opaque, whereas a handful of named signals slot cleanly into a tabular model and can be reasoned about.

Applied to demand, the LLM reads the review text (Option A) or the event descriptions (Option B) and returns a compact, structured set of signals per period. Those become columns for the forecaster.

import json

PROMPT = """You label retail product reviews. For each review return:
- sentiment: one of {negative, neutral, positive}
- aspects: up to 3 short tags (e.g. "shipping", "sizing", "price")
Return JSON only, one object per review."""

def enrich_reviews(reviews: list[str], client) -> list[dict]:
    # `client` is your model client (a Foundry-hosted model in the
    # target architecture). One batched call per chunk of reviews.
    raw = client.complete(system=PROMPT, user=json.dumps(reviews))
    return json.loads(raw)  # validate against a schema before trusting it

def aggregate_period(labels: list[dict]) -> dict:
    # Roll per-review labels up into features for one time step.
    n = len(labels) or 1
    pos = sum(1 for l in labels if l["sentiment"] == "positive")
    neg = sum(1 for l in labels if l["sentiment"] == "negative")
    return {
        "review_volume": len(labels),
        "share_positive": pos / n,
        "share_negative": neg / n,
    }

The forecaster now has share_positive, share_negative, and review_volume alongside its lags and calendar flags. The text has become a small number of interpretable columns, which is the whole pattern from the Balek paper.

Role 2: the LLM as a narrator. After the model predicts, the LLM writes the explanation the strategist asked for: this product is up because a recognized holiday falls in the window and the trailing sentiment turned positive. I want to be modest here. I have no primary source proving “LLM writes forecast narrative” is a validated technique, so I treat it as a product feature whose trustworthiness is established by evaluation, not by citation. The narrative is only as good as its grounding, which is the next section.

Evaluating both sides

The forecast side you already have: walk-forward MAE, RMSE, and MASE against the naïve baselines. The LLM side needs two different regimes.

Enrichment quality (classification). When the LLM emits labels (a sentiment class, an aspect tag), evaluate it like any classifier: accuracy, precision/recall, agreement against a labeled sample. This is standard supervised evaluation, and it is implicit in the Balek methodology where LLM features are validated against known targets. Hand-label a few hundred reviews and measure.

Narrative quality (groundedness). A generated explanation can be fluent and wrong. It can invent a promo that did not happen. Azure AI Foundry (now branded Microsoft Foundry; more on the rename below) defines a suite of RAG/generation evaluators for exactly this. The one that matters most here is Groundedness: it measures whether the response aligns with the given context without fabricating content, the precision aspect of “does not contain anything outside the grounding context.” It is LLM-judge based, needs the response plus the context, and returns a 1 to 5 score with a default pass threshold of 3. Groundedness is sometimes referred to as faithfulness.

from azure.ai.evaluation import GroundednessEvaluator  # verify SDK version on PyPI

groundedness = GroundednessEvaluator(model_config)

result = groundedness(
    response=narrative,        # the generated explanation
    context=forecast_context,  # the numbers + features it must stick to
)
# result -> {"groundedness": <1-5 score>}; treat >= 3 as pass

Two caveats I want stated plainly. First, Groundedness Pro (a stricter consistency check powered by Azure AI Content Safety, returning a boolean) and Response Completeness are preview features, so their behavior may change. Second, I have not run this SDK; the version pinning is something you verify on PyPI, not something I am asserting from experience.

Drift and monitoring. Both halves decay. The forecaster’s MASE will creep up as buying behavior shifts, so backtest on a rolling window and alarm when it crosses your naïve baseline. The enrichment classifier drifts as products and language change, so re-check accuracy against fresh labels on a schedule. The narrator’s groundedness score is a live guardrail: sample generated narratives in production, score them, and gate anything below threshold. None of this is exotic; it is the same discipline you would apply to any model you actually depend on.

Ship it as a service, not a notebook

A forecast that lives in a notebook helps nobody. The deliverable is an endpoint that takes a series and a horizon and returns the number plus the narrative.

from fastapi import FastAPI
from pydantic import BaseModel

app = FastAPI()

class ForecastRequest(BaseModel):
    series_id: str
    horizon: int = 14

class ForecastResponse(BaseModel):
    series_id: str
    horizon: int
    point_forecast: list[float]
    narrative: str

@app.post("/forecast", response_model=ForecastResponse)
def forecast(req: ForecastRequest) -> ForecastResponse:
    history = load_history(req.series_id)
    features = build_features(history)
    point = model.predict(features.tail(req.horizon))
    narrative = narrate(req.series_id, point, features)  # LLM call, grounded
    return ForecastResponse(
        series_id=req.series_id,
        horizon=req.horizon,
        point_forecast=point.tolist(),
        narrative=narrative,
    )

This is the conceptual shape. Pydantic models pin the contract at the edges, the classical model produces point_forecast, and the LLM produces narrative from the same features. In production the two would likely be separate deployments called by this orchestration layer, which is where the Azure mapping comes in.

Mapping it onto Azure

I want to be upfront: I have no Azure access and have not deployed any of this. What follows is a mapping onto the services Microsoft documents for each concern, so the blueprint has a concrete deployment target rather than a hand-wave. Treat it as a reading of the official docs, not a report from a running system.

The pipeline mapped onto Azure: data lives in Fabric/OneLake, the classical model serves from an Azure ML managed online endpoint, and the enrichment/narrative agent runs on Foundry Agent Service. The Azure band is a reading of the docs, not a running system.

Data layer: Microsoft Fabric and OneLake. OneLake is Fabric’s single, unified, logical data lake for the whole organization. It comes automatically with every Fabric tenant, with no infrastructure to manage, and its analytics engines work on data in place without copying. A Fabric Lakehouse combines data-lake scalability with warehouse querying, holding structured and unstructured data together under Delta Lake. That is where the demand series, the raw text, and the engineered features would live, with the bronze/silver/gold medallion pattern Fabric documents for exactly this kind of layering.

Classical model: Azure ML managed online endpoints. These deploy a model on managed CPU/GPU compute and handle serving, scaling, securing, and monitoring, which Microsoft describes as the recommended way to do real-time inference in Azure ML. There is no surcharge beyond compute and networking, auth supports key, Azure ML token, or Microsoft Entra token, and blue/green safe rollout is documented. The XGBoost/LightGBM/Prophet forecaster sits behind one of these.

LLM layer: Foundry Agent Service. This is the managed runtime for the enrichment and narrative agent. It lets you use any supported model from the Foundry catalog through the Responses API, with two agent types: Prompt agents (Foundry runs it, no containers to manage) and Hosted agents (your code, packaged as a container, on a managed endpoint with scaling, identity, and observability). It ships enterprise controls including guardrails against unsafe output and prompt-injection.

One honesty note on naming. Microsoft Learn now uses “Microsoft Foundry” and “Foundry Agent Service,” while older material says “Azure AI Foundry.” The product naming is in flux, so I am stating the current name and flagging the rename rather than mixing them silently. And, again, several evaluators I would lean on are in preview.

Conclusion

A hybrid forecasting service is two disciplines bolted together, and each has to hold on its own. The classical model earns trust through walk-forward backtesting and a MASE that beats seasonal-naïve, not through a shuffled split that lets it peek at the future. The LLM earns trust in two separate ways: as a feature generator measured against labels, and as a narrator measured by groundedness. Confuse those two evaluation regimes and you will ship something that sounds confident and is wrong.

If you want to build toward this, start with the honest data decision: pick Option A (derive a series from a review dataset, self-contained) or Option B (join a demand series to a real text source and say out loud that the join is your design choice). Get walk-forward MASE beating the baseline before you add a single LLM feature. Everything else, the enrichment, the narrative, the Azure endpoints, is layered on a forecaster you already trust.

Connect and Collaborate

I write about GenAI engineering: agents, RAG, and the back-end that makes them work. If this was useful, connect with me on LinkedIn and follow my code on GitHub. Questions and corrections are welcome, especially from anyone who has run this stack on Azure in anger.

Recommended Reading

• LLM-based feature generation from text for interpretable machine learning (Balek et al., 2024) — https://arxiv.org/abs/2409.07132
• Forecasting: Principles and Practice (3rd ed.), §5.10 Time-series cross-validation — https://otexts.com/fpp3/tscv.html
• Hyndman & Koehler, “Another look at measures of forecast accuracy” (MASE) — https://robjhyndman.com/papers/mase.pdf
• RAG Evaluators for Generative AI (Microsoft Foundry) — https://learn.microsoft.com/en-us/azure/foundry/concepts/evaluation-evaluators/rag-evaluators

Sources

• Rossmann Store Sales — https://www.kaggle.com/c/rossmann-store-sales/data
• Corporación Favorita Grocery Sales Forecasting — https://www.kaggle.com/c/favorita-grocery-sales-forecasting
• Store Sales — Time Series Forecasting (Favorita playground, holidays_events.csv) — https://www.kaggle.com/competitions/store-sales-time-series-forecasting/data
• M5 Forecasting — Accuracy — https://www.kaggle.com/competitions/m5-forecasting-accuracy
• M5 accuracy competition results paper (Makridakis et al.) — https://statmodeling.stat.columbia.edu/wp-content/uploads/2021/10/M5_accuracy_competition.pdf
• Amazon US Customer Reviews Dataset — https://www.kaggle.com/datasets/cynthiarempel/amazon-us-customer-reviews-dataset
• Amazon Reviews (polarity) — https://www.kaggle.com/datasets/kritanjalijain/amazon-reviews
• McAuley-Lab Amazon-Reviews-2023 — https://huggingface.co/datasets/McAuley-Lab/Amazon-Reviews-2023
• Amazon Reviews 2023 project site — https://amazon-reviews-2023.github.io/
• Hou et al., “Bridging Language and Items for Retrieval and Recommendation” — https://arxiv.org/abs/2403.03952
• scikit-learn TimeSeriesSplit — https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.TimeSeriesSplit.html
• scikit-learn Cross-validation guide — https://scikit-learn.org/stable/modules/cross_validation.html
• Forecasting: Principles and Practice (3rd ed.), §5.10 — https://otexts.com/fpp3/tscv.html
• Hyndman, time-series cross-validation — https://robjhyndman.com/hyndsight/tscv/
• Forecasting: Principles and Practice (3rd ed.) — https://otexts.com/fpp3/
• Hyndman & Koehler, “Another look at measures of forecast accuracy” — https://robjhyndman.com/papers/mase.pdf
• Hyndman, measuring forecast accuracy — https://robjhyndman.com/papers/forecast-accuracy.pdf
• forecast::accuracy reference — https://pkg.robjhyndman.com/forecast/reference/accuracy.default.html
• Balek et al., “LLM-based feature generation from text for interpretable machine learning” — https://arxiv.org/abs/2409.07132
• Advanced feature engineering with LLM embeddings (MachineLearningMastery) — https://machinelearningmastery.com/7-advanced-feature-engineering-tricks-for-text-data-using-llm-embeddings/
• RAG Evaluators for Generative AI (Microsoft Foundry) — https://learn.microsoft.com/en-us/azure/foundry/concepts/evaluation-evaluators/rag-evaluators
• Develop a RAG Solution: LLM End-to-End Evaluation (Azure Architecture Center) — https://learn.microsoft.com/en-us/azure/architecture/ai-ml/guide/rag/rag-llm-evaluation-phase
• OneLake overview — https://learn.microsoft.com/en-us/fabric/onelake/onelake-overview
• What is a lakehouse? (Fabric) — https://learn.microsoft.com/en-us/fabric/data-engineering/lakehouse-overview
• OneLake medallion lakehouse architecture — https://learn.microsoft.com/en-us/fabric/onelake/onelake-medallion-lakehouse-architecture
• Online endpoints for real-time inference (Azure ML) — https://learn.microsoft.com/en-us/azure/machine-learning/concept-endpoints-online?view=azureml-api-2
• Deploy to online endpoints (Azure ML) — https://learn.microsoft.com/en-us/azure/machine-learning/how-to-deploy-online-endpoints?view=azureml-api-2
• Safe rollout for online endpoints (Azure ML) — https://learn.microsoft.com/en-us/azure/machine-learning/how-to-safely-rollout-online-endpoints?view=azureml-api-2
• What is Foundry Agent Service? — https://learn.microsoft.com/en-us/azure/foundry/agents/overview