How it unfolds, step by step
The setup
On this chart, rules are tested one after another against data containing NO signal whatsoever. The true hit rate of every one of them is exactly 50%. The line is the best result found so far.
What happens next
The first rule lands near 50%, as it must. By the tenth rule the best-so-far is around 58%. By the fortieth it is above 62%. Nothing was learned in between. The line rises purely because the search continued.
What it means — and does not
A 62% result, presented on its own, looks exactly like a real discovery. What separates them is a fact about the process, not about the number. It is how many rules were tried before this one was reported. That is why the honest form of a search result states the number of things searched. It is also why a result that survives correction for that search is worth far more than a bigger number that does not.
- Best hit rate found so far (%)
- The true rate of every rule here · 50
Forty rules over data with NO signal. Every true rate is 50%. The rising line is the search, not a discovery.
Show the underlying data
| Rules tested | Best hit rate found so far (%) |
|---|---|
| 1 | 50.4 |
| 2 | 52.1 |
| 4 | 54.3 |
| 6 | 55 |
| 8 | 56.8 |
| 10 | 57.9 |
| 14 | 58.6 |
| 18 | 59.4 |
| 22 | 60.1 |
| 26 | 60.7 |
| 30 | 61.2 |
| 35 | 61.8 |
| 40 | 62.4 |
What this does not establish
The reading is that the best-of-N is inflated. The way to refute it for a specific rule is to test it on data the search never touched. A rule that holds up on data it was not selected from was not selected by chance. One that collapses was. The level that settles it is not on this chart at all. It is evidence, which is the next page.
Things to check
4How many variations were tried before this one was reported?
Was the reported number corrected for that search, or quoted raw?
Has the rule been tested on data that played no part in choosing it?
The best result from many searches can come from chance alone.
Common mistakes
2Reporting the best rule found without reporting how many were examined. The second number is what gives the first any meaning.
Treating a threshold that was tuned to the data as if it had been chosen in advance.
Forty rules are tested against data containing no signal at all. What does the best of them look like?
- About 50%, since none of them works
- Better than 50%, purely because forty were examined
- Unpredictable, because a result drawn from pure noise could land anywhere
Show the answer
B. Better than 50%, purely because forty were examined
The reported figure is a maximum over forty draws, and a maximum is biased upward by construction even when every underlying rate is exactly one half. That is why the number of variations examined has to be reported alongside the result.
Frequently asked
3How do I correct for the number of rules tried?
By judging the result against the distribution of the BEST outcome under no signal, rather than against a single rule's distribution. That is what a search-corrected figure means, and it is always a harder bar than the uncorrected one.
Does this only apply to formal backtests?
No. Trying thresholds by eye until a chart looks right is the same search with no record of how many were tried, which makes it worse rather than better.
What is the fix?
Write down the rule before testing it, count the variations tried, correct for them, and confirm on data that took no part in the choice.