Reading a Data-Quality Report: The Numbers Behind the Numbers
The Fluxy Team
-Reading a Data-Quality Report: The Numbers Behind the Numbers
A backtest's equity curve answers "what would this strategy have earned on this data?" The question almost nobody asks is the one underneath: how much data was actually there? Crypto data is gappy — feeds drop, venues have outages, scrapes fail, symbols appear mid-history — and a backtest engine has to do something on every missing print. What it does, and whether it tells you, separates research platforms from random-number generators.
Our engine's answer is to count everything it couldn't do and put the tally next to the results. This post is how we read that tally.
The numbers that matter
Bar coverage. Of the bars the strategy's window implies, how many actually exist? A 97% coverage number on a major perp is normal (maintenance windows, brief outages). 80% coverage means a fifth of your test is imagination — and worse, the missing fifth is rarely random. Outages cluster in exactly the violent moments your drawdown statistics depend on.
Stale-feed returns. Every auxiliary series in our engine carries a freshness TTL derived from its cadence; a value past its TTL comes back as no-data rather than being forward-filled forever. The report counts how often a strategy asked for a series and got None. A high count isn't automatically bad — a strategy that notices stale data and holds is behaving correctly — but it tells you how much of the window your signal was actually live.
Skipped funding. For perp strategies, funding is P&L, and our own market data has real funding gaps (we've published them). The skipped_funding counter says how many settlements couldn't be charged for lack of a print. A carry strategy with 30% skipped funding has an equity curve that's 30% fiction — that's a strategy to re-run on better coverage, not to deploy.
Symbol coverage asymmetry. In multi-asset tests, per-symbol coverage matters more than the average. A basket backtest where one leg has half the history of the others isn't testing the basket — it's testing a time-varying basket that happens to share a name with yours.
The judgment calls
There's no universal threshold, but our internal rules of thumb:
- Headline metrics need >95% bar coverage on every instrument that matters. Below that, treat Sharpe and drawdown as estimates with error bars you can't compute.
- Any gap overlapping a drawdown is disqualifying for that drawdown's statistics. If the worst week of the test is also the worst-covered week, the real drawdown is unknown — and probably worse.
- Fill-rate on the signal matters more than on the price. Prices interpolate tolerably; signals don't. A momentum signal computed over a gap is a different signal.
- When two runs disagree, believe the one with the better data report. We've watched a "worse" strategy variant outperform live because its backtest was the honest one.
Why this is a report and not a silent fix
The tempting engineering move is to make gaps invisible: forward-fill everything, interpolate, smooth. The result is a backtest that always runs and never warns — maximally convenient, minimally true. We made the opposite choice: gaps surface as no-data, skips are counted, and the counts ship with every run, because a number you can't defend is worse than no number.
The habit to build is simple: read the data report first. Thirty seconds there tells you whether the next ten minutes of reading metrics is worth your time — and it's the first thing an allocator's diligence will ask about anyway.
Every backtest ships its data-quality report — coverage, staleness, skipped settlements — next to the metrics. Run one and read it →
Related reading