When Live Doesn't Match the Backtest: A Diagnosis Checklist
The Fluxy Team
-When Live Doesn't Match the Backtest: A Diagnosis Checklist
It's the systematic trader's recurring nightmare: three weeks live, and the strategy that backtested at a 1.8 Sharpe is flat-to-down while the backtest, re-run over the same three weeks, claims you should be up 4%. Is the edge gone? Was it ever real?
Sometimes, yes, the edge was overfit ghost. But having diagnosed this divergence many times — on our own book — we can tell you the boring truth: most backtest-live gaps have a specific mechanical cause, and they're findable if you check in the right order. Cheapest checks first.
1. Costs and fees (five minutes)
Sum your actual paid fees and compare to the backtest's cost assumptions. Taker fees where you assumed maker, a venue fee tier you don't actually have, or funding paid that the backtest didn't model — on a strategy that trades daily, 5 basis points of unmodeled cost per turn is roughly 12% a year of divergence, which is most strategies' entire edge. This is the most common cause and the easiest to verify, which is why it's first.
2. Fill assumptions vs. fill reality (an hour)
Pull your live fills and compare each to the bar the backtest would have filled at. Backtests fill at the close (or a modeled price); live orders eat spread and slippage, and in thin books they move the price. The telltale: divergence that scales with position size and clusters in fast markets. If your average live fill is 8bps worse than modeled and you turn the book over twice a week, there's your gap.
3. Timing skew (an hour)
The backtest acts on the bar close; the live runner computes, sizes, and sends orders some seconds or minutes later. In trending micro-moments that skew is systematically against you (you buy after the up-bar finished going up). The telltale: divergence concentrated in your highest-momentum entries. This is why running the same engine live and in research matters — it makes the skew measurable instead of invisible.
4. Universe drift (thirty minutes)
Is live trading the same instrument set the backtest tested? Symbols get delisted, filtered by liquidity gates, or blocked by venue quirks — and the ones that drop out are rarely random. A backtest over a universe that live can't fully trade is survivorship bias in real time.
5. Data revisions (thirty minutes)
The bar your live runner saw at decision time and the bar your historical database holds now are not always the same bar — late prints, venue corrections, and in-progress bars all revise history slightly. Re-run the backtest over the live window and diff the signals, not the returns. If the signals differ, your divergence starts at the data layer, not the market.
6. State divergence (the scary one)
Does the engine's view of your positions match the venue's? A partial fill it didn't register, a manual trade, a hedge leg that didn't complete — now the strategy is sizing from a book that doesn't exist. This one compounds silently, which is why reconciliation runs as a pre-trade guard on our executor rather than as a weekly report.
7. Only now: maybe the edge
If fees match, fills match, timing is tight, the universe is intact, signals reproduce, and state reconciles — then you're allowed to suspect the edge itself. Even then, three weeks is noise for most strategies; compare against the backtest's own distribution of three-week windows before declaring death. A strategy whose live period sits inside its backtested 30th–70th percentile isn't diverging. It's just Tuesday.
The meta-lesson: every one of these checks requires that your backtest and your live system be comparable — same engine, same data lineage, logged fills, reconciled state. That comparability isn't a nice-to-have. It's the entire diagnostic toolkit, and it has to be built in before you go live.
One engine from research to live — so when they disagree, you can find out why. See how the pipeline works →
Related reading