Data Snooping: Test 200 Strategies and the Best One May Be Pure Luck

Data snooping, or multiple testing, is what happens when you try many strategies on the same data and keep the one with the best result. The more you try, the more likely you are to find one that looks good by accident rather than because it has a real edge. It differs from the overfitting we covered before: even if every variant is simple and nothing is finely tuned, simply picking the best of many can fool you. This article simulates it with real numbers, then looks at the tools researchers use to correct for it.

Test 100, and 5 pass by luck

A statistical test at the 5% level asks: “if this system had no edge at all, how likely is a result this good by luck alone?” If it’s below 5%, we call it a pass.

That threshold was built for one test. If you test 100 variants that have no edge at all:

1. Expected number of lucky passes = 100 × 5% = 5
2. Chance that none of them passes = 0.95 to the power of 100 ≈ 0.6%
3. Chance that at least one passes ≈ 99.4%

Put simply, if you try enough, you will find a “statistically significant system”, whether or not a good one is in the pile. What is overfitting covers this for EA optimisation. Here we go deeper: how much the result gets inflated, and what to do about it.

Worked example: 200 strategies with no edge

We built 200 hypothetical strategies with a node script. Each takes 100 trades. Every trade has a 50% chance of +1R and a 50% chance of −1R (R is your risk per trade, for example 1% of the account). No spread, and not one of them has any edge. Every strategy’s expected result is 0.

With W wins, the total is W − (100 − W) = 2W − 100 R.

Pass threshold: 59 or more wins (+18R or more), because the chance that a fair coin lands heads 59 or more times out of 100 is 4.4%, the closest to 5% without going over.

Results:

• Average total across the 200 = −0.4R, ranging from −24R to +30R
• 89 positive, 98 negative, 13 exactly flat
• 10 pass the test. The expected count is 200 × 4.43% = 8.9, right where it should be

The best one won 65 of 100 trades. Seen in a backtest report, that would look exciting.

1. Total = 2 × 65 − 100 = +30R
2. Average per trade = 30 ÷ 100 = 0.30R
3. Standard deviation per trade ≈ 0.959R
4. Per-trade Sharpe = 0.30 ÷ 0.959 ≈ 0.31
5. t-stat = 0.31 × √100 ≈ 3.13. The chance of a fair coin landing heads 65 or more times is 0.18%

Looked at alone, you would say it passes by a mile. But it’s a fair coin. We simply picked the coin that happened to land heads most often out of 200.

The best one meets new data

Now every strategy trades another 100 trades on data that was not used to pick it (out-of-sample).

• The best one goes from +30R to −8R (46 wins)
• The top 10 average goes from +21.4R to +8R

In this run, the top 10 stayed positive on average, and 3 of the 10 that passed passed again, more than luck usually gives. We did not re-roll the random numbers to get a nicer picture, because re-running until you get the picture you want is just another form of data snooping. What tells the truth is repeating the same experiment many times.

We repeated this 200-strategy experiment 2,000 times:

• The best strategy in each run averaged +27.2R in-sample, and never below +18R
• The same strategy on new data averaged 0.0R, and was positive in only 45.9% of runs
• The top 10 also averaged 0.0R on new data
• Strategies that passed the test passed again on new data 4.4% of the time, exactly a fair coin’s chance

100 new trades carry plenty of luck too. Each strategy’s 100-trade total has a standard deviation of about 10R from luck alone. So a single confirmation isn’t enough. You need to see how far the confirmation result sits from what luck could produce.

How much does the best of N get inflated

The more you try, the higher the best result climbs on its own, even though none of them is good. We simulated the average best result (2,000 runs per row) and compared it with the formula for the expected maximum of N normal random numbers that Bailey and López de Prado use in their Deflated Sharpe ratio work.

Variants tried (N)	Best, simulated	Best, formula
1	−0.2R	0R
10	+15.4R	+15.7R
100	+25.0R	+25.3R
200	+27.2R	+27.7R
1,000	+32.3R	+32.6R

(The formula gives standard deviations, multiplied by 10R, the standard deviation of a 100-trade total. N = 1 means no selection, so the expected value is 0.)

The number grows fast at first, then more slowly, but it never stops. Going from 200 to 1,000 variants gives the fake winner about 5R more for free. In per-trade Sharpe terms, the best of 200 strategies with nothing in them is expected to reach about 0.28. Ours reached 0.31, barely above luck.

Compare the best result with “the best that luck can produce from this many tries”, not with zero.

Bonferroni: the more you test, the higher the bar

The simplest fix is the Bonferroni correction: divide the threshold by the number of variants.

• 200 variants: threshold = 5% ÷ 200 = 0.025%
• You need 68 or more wins out of 100 (0.020% by luck). 67 wins still has a 0.044% chance by luck, above the threshold
• Our best strategy, with 65 wins, fails

Bonferroni’s weakness is that it is too strict when the variants are similar. MA 20 and MA 21 are nearly the same system, and counting them as two full tests can throw out a good one. In practice, though, retail traders far more often count too few than are too strict.

Tools from the research

• White’s Reality Check (Halbert White, 2000). Instead of testing one variant at a time, it tests whether “the best of everything you tried beats what luck can produce”, by resampling the data (bootstrap) and seeing how good the best variant gets in each resampled run.
• Hansen’s SPA test (Peter Hansen, 2005) improves on the Reality Check so it is not weakened when the pile contains many poor or irrelevant variants.
• Harvey, Liu and Zhu (2016), in “…and the Cross-Section of Expected Returns”, reviewed the hundreds of factors proposed to explain stock returns and argued that a new factor should clear a t-stat of about 3, not the customary 2, because the field has already tried so many. Note from our example: the best of 200 strategies scored t = 3.13, clearing 3 while being pure luck. 3 is not a magic number. The more you try, the higher the bar must go.
• The Deflated Sharpe ratio (Bailey and López de Prado, 2014) asks whether your Sharpe beats the highest Sharpe luck would produce from N trials, and also allows for results that are not normally distributed.

We computed it for our best strategy (using the Probabilistic Sharpe ratio formula, adjusted for the skew and kurtosis of the per-trade results):

• Against a Sharpe of 0 (as if only one variant was tried): probability that the true Sharpe is above 0 ≈ 99.8%. Looks certain
• Against the 0.28 that luck delivers from 200 tries: it drops to ≈ 63%, far below the 95% usually required

Same numbers. Just by admitting how many variants were tried, the conclusion flips.

Lessons from our own backtests

925 false passes. In our backtests, we counted every time gold touched various Fibonacci levels in M1 bar data from 2022 to 2026, 62,867 events, and ranked 8,078 combinations (level × direction × TP × SL) against random entry. The first round had 925 combinations “beating random”, and all of them were fake. The script dropped events whose outcome wasn’t known within 5 hours, and some combinations lost 30–70% of their events that way. Near-TP, far-SL combinations then looked like they won far more than they did, because the times price reached a nearby target finished quickly and were counted, while the times price didn’t head for the target tended to hang around until they were dropped. Keeping only combinations with under 5% unfinished left 40, which still had to be confirmed on the held-back years. A measurement flaw combined with thousands of tries can produce hundreds of fake winners.

70 lines on the screen. Once we saw price turn exactly at the 261.8 level of an earlier swing, and it looked as if old swings’ Fibonacci lines still mattered. But draw the lines for 10 earlier swings, several levels each, and there are about 70 lines on screen. One of them will always be near a turning point. That is choosing the example after seeing the result. So we measured all 241,000 touches (gold, M1) against nearby non-Fibonacci placebo levels. The 261.8 level bounced 41% of the time, the placebo levels 42%. No Fibonacci level bounced meaningfully more often than the placebos. More on this in Hindsight bias.

The control has to look the same. A comparison or placebo has to be built like the thing you test in every way except the part you claim is the edge. We once used a control that wasn’t built the same way and got a result that beat it, which turned out to be an artefact of the control itself.

Rules written in advance can still be broken. Our research log set rules from the start: write each idea down before running it, never change definitions midway, use 2022–2024 to search and 2025–2026 only to confirm. After a while we started choosing filters from the 2026 results anyway, which turned that year into selection data, and several filters turned out to work only in that one year.

The 2022–2025 data in this work is bar data or generated ticks. Only 2026 is real tick data. Backtest and past results do not predict future results.

Practical rules

1. Count every variant you try. Log every run, including the ones you dropped. That count is the N you judge by. If you tried 50, the best result must be compared with what luck produces from 50.
2. Write the rules before looking. Record the hypothesis, definitions and pass criteria in advance (pre-registration). If you change the criteria after seeing results, treat it as a new experiment.
3. Hold data back and touch it once. If you look at the confirmation data and then go back and change the system, it has become search data. Walk-forward testing gives you several confirmation periods.
4. Prefer simple. Fewer settings means fewer variants, a smaller N, and results you can trust more.
5. Compare with a placebo that looks the same, for example random entries with the same SL, TP and number of trades. If it doesn’t clearly beat the placebo, there is no evidence yet.
6. Raise the bar with N. Use a rough Bonferroni, or compare with the best-of-N table above, and expect live results to come in below the backtest. See Why live results usually trail the backtest.

Data that hasn’t happened yet is the only data you are guaranteed not to have peeked at.

Common mistakes

• Reporting only the best variant without saying how many were tried, whether reading someone else’s results or logging your own. Without N, the number can’t be read.
• Counting only Optimize runs and forgetting pairs, timeframes, trading hours and the ideas you tested by eye on the chart.
• Peeking at the confirmation data again and again. Every look followed by a change uses it up a little more.
• Trusting t > 3 or a high Sharpe without knowing N. Our example got t = 3.13 from a fair coin.
• Dropping unfinished trades from the count. Results tilt toward near-TP, far-SL systems straight away.
• Re-running a simulation until you like the picture. Choosing simulation results is data snooping too.
• Treating one short confirmation as a verdict. A 100-trade total with no edge has a standard deviation of about 10R from luck alone. Look at the spread of outcomes with Monte Carlo testing.