The champion model changes every 90 days
We ran 21,600 backtests — five forecasting models, across nine retail catalogs and six planning horizons — to answer a simple question: which algorithm should power a demand forecast? The honest answer is that no single model wins for long. Here is what actually won, where, and why it matters for how you should think about champion selection.
buffers.ai Benchmark Lab · July 2026
Why "best model" is the wrong question
buffers.ai doesn't hard-code a forecasting algorithm. Every SKU is backtested against a slate of candidates — a pretrained foundation model, a gradient-boosted tree model, a classical statistical model, and two simple baselines — and a composite score picks a champion per SKU, weighted mostly by weighted absolute percentage error (WAPE), with bias, safety-stock impact, and achieved service level as secondary tie-breakers.
That per-SKU approach is deliberately un-opinionated: it doesn't assume any one model is universally best. This report asks the aggregate version of that question anyway — at scale, across many catalogs and many horizons, does a pattern emerge in which model tends to win, and under what conditions does that tendency flip?
Qualification gate: a SKU needs at least 20 non-zero-demand days before the evaluation window starts to be considered backtestable at all.
Thinner or brand-new SKUs are scored through a separate cold-start/cohort path and are outside this report's scope — this is Track A, the mature-SKU model comparison. 720 SKUs cleared this gate across the nine catalogs studied here.
The headline result
Across all 4,320 SKU × horizon combinations, LightGBM won the plurality of the time — 36.2% — narrowly ahead of a plain moving-average baseline at 31.9%. Together those two account for two out of every three wins. The three more elaborate models — Prophet, TimeSFM, and exponential smoothing — split the remaining third, and none individually clears 14%.
| Model | Win rate | Median WAPE | Median |bias| |
|---|---|---|---|
| LightGBM | 36.2% | 0.321 | 0.173 |
| MovingAverage | 31.9% | 0.328 | 0.128 |
| Prophet | 13.4% | 0.700 | 0.416 |
| TimeSFM | 12.4% | 0.554 | 0.486 |
| ExpSmoothing | 6.2% | 0.739 | 0.495 |
Win rate = share of the 4,320 SKU × horizon combinations where a model produced the lowest WAPE among the five candidates. WAPE and bias are medians across all combinations that model was scored on, win or not — the two winningest models also post the two lowest error and bias figures, so the ranking isn't an artifact of the scoring rule.
The horizon effect
The aggregate hides a crossover. LightGBM's edge is largest at 30–60 days — winning roughly 47% of SKUs — and erodes at every horizon after that. The moving-average baseline is the mirror image: barely competitive at 30 days (24.6%), it overtakes LightGBM around 120–150 days and wins the plurality of SKUs by 180 days (42.6%).
Why the crossover happens: near-term demand carries structure a model can learn from — day-of-week effects, a recent promotional lift, short-term trend — and LightGBM's lag features exploit exactly that. Far enough out, that structure decays faster than the model's ability to extrapolate it, and a plain average of recent history becomes hard to beat with a model that's effectively extrapolating noise.
What the real client revealed
Eight of the nine catalogs in this study are synthetic, generated by buffers.ai's internal Synthesis engine. The ninth is one real, anonymized retail client's production sales history. Comparing that one real catalog against the synthetic-only aggregate exposes a gap the aggregate alone hides.
| Model | Win rate — real client | Win rate — synthetic (avg) | Difference |
|---|---|---|---|
| LightGBM | 25.2% | 37.6% | -12.4pp |
| MovingAverage | 19.7% | 33.5% | -13.8pp |
| Prophet | 15.9% | 13.1% | +2.8pp |
| TimeSFM | 25.0% | 10.7% | +14.3pp |
| ExpSmoothing | 14.2% | 5.2% | +9.0pp |
The TimeSFM gap: TimeSFM — a large pretrained forecasting foundation model — wins almost never on synthetic data (10.7%) but ties LightGBM for the top spot on the one real dataset (25.0% vs. 25.2%). Prophet and exponential smoothing show the same pattern in miniature: both do measurably better on real data than on synthetic.
Read literally, this is one client, not a population — not proof of anything universal. But the size and consistency of the gap is a caution against trusting synthetic benchmarks alone when evaluating a large pretrained model: synthetic catalogs, however carefully constructed, may lack the messy structure — irregular promotions, supply disruptions, genuine regime changes — that a model like TimeSFM is specifically built to absorb.
Recommended model, by catalog and horizon
The practical output of this study isn't a universal recommendation — it's a diagnostic. Each cell below shows the model that won the most SKUs for that catalog and horizon, and the share of SKUs it won by.
| Catalog | 30d | 60d | 90d | 120d | 150d | 180d |
|---|---|---|---|---|---|---|
| Real client (anonymized) | LightGBM25% | TimeSFM24% | TimeSFM25% | LightGBM29% | TimeSFM25% | TimeSFM28% |
| Basic fashion | LightGBM51% | LightGBM48% | LightGBM36% | MovingAverage33% | MovingAverage42% | MovingAverage46% |
| Luxury | LightGBM36% | LightGBM44% | LightGBM41% | LightGBM34% | MovingAverage33% | MovingAverage40% |
| Footwear | LightGBM51% | LightGBM41% | LightGBM35% | MovingAverage38% | MovingAverage46% | MovingAverage52% |
| Beauty | LightGBM49% | LightGBM59% | LightGBM45% | MovingAverage41% | MovingAverage55% | MovingAverage54% |
| Home | LightGBM57% | LightGBM57% | LightGBM42% | LightGBM42% | LightGBM35% | LightGBM39% |
| Mass retail | LightGBM55% | LightGBM55% | LightGBM42% | MovingAverage45% | MovingAverage57% | MovingAverage54% |
| Sporting goods | LightGBM63% | LightGBM56% | LightGBM48% | LightGBM42% | MovingAverage46% | MovingAverage49% |
| Bags & accessories | LightGBM44% | LightGBM44% | LightGBM46% | LightGBM30% | MovingAverage39% | MovingAverage36% |
Percentages are the winning model's share of that cell's ~80 qualified SKUs; the remainder splits across the other four candidates. The real-client row is highlighted — note it's the only row where TimeSFM (not MovingAverage) picks up the long-horizon wins.
Methodology & limits
- Data
- Eight vertical catalogs synthesized via buffers.ai's internal Synthesis engine — basic fashion, luxury, footwear, beauty, home, mass retail, sporting goods, and bags & accessories — 500 SKUs generated per catalog with a lifecycle mix skewed toward more new/thin SKUs than a realistic catalog, so qualification filtering still leaves enough sample per bucket. One real, anonymized retail client, sourced from production sales history and labelled "real client" throughout.
- Sample
- 80 qualified SKUs randomly sampled per catalog (720 total), each backtested at six horizons (30 / 60 / 90 / 120 / 150 / 180 days) against all five candidate models — 4,320 SKU × horizon combinations, 21,600 individual model fits.
- Scoring
- "Won" means lowest WAPE among the five candidates for that SKU × horizon — a simpler proxy than buffers.ai's live composite ranking, which additionally weighs bias, safety-stock impact, and achieved service level. The simpler rule is used here for a cleaner aggregate read.
- Coverage
- About 4.6% of individual model fits did not produce a usable forecast — most often a model needing more history than a given SKU/horizon combination had — and were excluded rather than imputed.
- One real client is one data point, not a population — treat section 04's finding directionally, not as a universal claim about foundation models.
- Synthetic catalogs, however varied, are generated by one internal engine and inherit its assumptions about seasonality and promotional structure.
- This report covers only Track A (qualified, mature SKUs). Cold-start and thin-history SKUs are scored through a separate cohort model and are not represented here.