Research

Backtesting a crypto trading bot: seven pitfalls that inflate results

A backtest shows how a strategy would have behaved on past data, not how it will behave. Seven common pitfalls that make results look better than they are.

Published

CryptoIQ Sentinel Journal — AI trading systems and decision receipts

What a backtest can and cannot show

A backtest replays a strategy’s rules over historical data and records what it would have done. Done carefully, it can rule out ideas that never worked and expose obvious flaws. It cannot show that a strategy will work going forward. Every pitfall below is a reason to treat a published backtest as a claim to examine, not a result to trust.

Seven pitfalls

  1. Overfitting. Try enough parameter settings and one will fit the past by chance. Bailey and colleagues’ paper The Probability of Backtest Overfitting examines this problem. Ask how many variants were tried, not only which one won.
  2. Look-ahead bias. Using information the system could not have had at the time, such as a candle’s final close while the candle was still open, or data that was revised later.
  3. Ignoring costs. Fees, spreads, slippage and funding all reduce results, and the effect grows with trade frequency.
  4. Survivorship and selection. Testing only assets that still exist and are liquid today hides the ones that failed. Choosing the test period after seeing the results does the same.
  5. Unrealistic fills. Assuming every order fills at the quoted price ignores order-book depth and partial fills, especially for larger sizes or thin markets.
  6. Too little data or one regime. A strategy tested only in a rising market has not been tested. Count the number of independent trades, not just the number of days.
  7. No out-of-sample test. Evaluate on data the design never touched. Even then, a single holdout is weak evidence.

A worked example of costs (hypothetical)

Hypothetical numbers, not a CryptoIQ result. A strategy gains an average of 0.30% per trade before costs. If round-trip fees and slippage total 0.25% per trade, the average after costs is 0.05% — 83% of the apparent edge gone. If costs turn out to be 0.35% in live conditions, the average is negative. A small change in an assumption can flip the sign, which is why backtests should state their cost assumptions.

Questions to ask of any backtest

  • What data, from which venue, over what period, at what interval?
  • How many parameter combinations were tested, and how was the reported one chosen?
  • What costs and fills were assumed?
  • Was there a separate period the design never saw?
  • What was the worst drawdown, and how was it measured? See max drawdown.
  • Is the number labelled as simulated, and kept apart from live results?

How we label results

In CryptoIQ Sentinel, Replay (historical playback), Paper (simulated execution), Shadow (observation without submitting) and Live are separate modes, and a record from one is not presented as another. Our policy is that no win-rate, profit or edge figure appears in marketing copy; anything about performance should resolve to the public proof feed or a specific receipt. A historical result describes the past record only. It is not a prediction, and trading involves a risk of loss.

Frequently asked questions

What is backtesting a trading bot?

Backtesting replays a bot’s rules over historical market data to see what it would have done. It can reveal flaws, but it does not show that the strategy will work in the future.

Why do backtests look better than live results?

Common reasons are overfitting to past data, look-ahead bias, ignored fees and slippage, unrealistic fills, survivorship bias and testing in only one market regime.

What is overfitting in a trading strategy?

Overfitting means tuning a strategy so closely to past data that it captures noise rather than a repeatable pattern. Testing many parameter settings and reporting only the best one is a common cause.

Is paper trading the same as backtesting?

No. Backtesting replays historical data. Paper trading runs a strategy in simulated execution against current market conditions. Both are simulated, and neither is a live result.

Keep reading

Risk and about this note

Educational material, not investment, legal or tax advice. Trading crypto assets involves substantial risk of loss, and leverage can multiply losses. Nothing here predicts or promises any return, fill or outcome, and no CryptoIQ performance figure is stated. Past results do not guarantee future performance.

Published by CryptoIQ. Prepared with AI assistance. No independent specialist review is claimed. Linked sources explain concepts and do not endorse CryptoIQ.