The AI Wrote My Strategy. Here's How to Find Out If It's Any Good
The Fluxy Team
-The AI Wrote My Strategy. Here's How to Find Out If It's Any Good
Ask a modern LLM for a crypto trading strategy and you'll get one: clean code, sensible-sounding logic, a confident paragraph about why it works. The problem is not that the code is bad — it often runs fine. The problem is that fluency hacks your judgment. A strategy you wrote yourself gets your full skepticism; a strategy delivered in polished prose with an explanation arrives pre-trusted. It shouldn't.
We build AI strategy generation into our own platform, so this isn't AI-skepticism — it's the validation gauntlet we wired in because we believe in the tool. Whatever platform you use, an AI-written strategy should survive all five of these before it sees money.
1. Does it even run against real data?
The most common failure is the quietest: code that references columns, symbols, or API methods that don't exist in your actual environment. It parses, it looks right, it dies at runtime — or worse, half-runs. This is a solved problem when generation is grounded (our editor feeds the model the real dataset catalog and sample rows, and validates every generated strategy with an actual backtest before showing it). If your tool doesn't do that, step one is manual: run it and read the errors, before reading the equity curve.
2. Is the backtest honest, or just present?
"It backtests well" means nothing until you know what the backtest charges and forbids. The checklist is the same as for human code, applied with more suspicion: point-in-time data (no look-ahead), funding charged on perps, fees and slippage modeled, delisted assets present rather than erased. LLMs trained on tutorial code reproduce tutorial sins — the classic is computing an indicator over the full series (future included) and trading on it historically. An honest engine makes that impossible; a notebook does not.
3. The out-of-sample verdict
AI-generated strategies are overfitting machines in a specific way: the model has read thousands of strategies and tends to produce what looked good historically — parameters and patterns that are, in effect, pre-fit to the past. So the in-sample/out-of-sample split isn't optional here; it's the whole exam. Generate, test in-sample, then judge only on data the generation never touched. If OOS collapses, the AI gave you a description of history, not a strategy. Regenerating five variants and keeping the best OOS performer is also overfitting — now you're the optimizer. Pick the thesis first, then validate once.
4. Would you understand it at 3am?
Read the code until you can say what it does in one sentence without the AI's explanation. This isn't ceremony: when the strategy is live and doing something weird during a flash move, the person debugging it is you, at the worst possible hour. A strategy you can't explain is a strategy you can't supervise — and "the AI wrote it" is not an incident report your future self will accept. If you can't get to that one sentence, simplify until you can. (Our glossary exists partly for this — every term the code leans on, defined the way the engine means it.)
5. Sizing was never the AI's job
Whatever the generated code says about position size, override it with your own risk budget: vol targeting, leverage caps, portfolio limits. Generation is good at signal logic and bad at knowing your account, your other positions, and your tolerance. Keep those human, keep them mechanical, and keep them outside the generated code where a regeneration can't silently change them.
The role AI actually earns
Pass all five and what has the AI done? Removed the blank-page tax, the syntax tax, and a few hours of boilerplate — genuinely valuable. What it hasn't done is create an edge or absorb responsibility. The honest framing: AI moves you from idea to testable implementation in minutes; the testing is still the product. Platforms that sell the first half without building the second are selling exactly the trap this post is about.
Describe a strategy; get real code, grounded in your data, validated by a real backtest before you see it. Try the AI editor →
Related reading