How it unfolds, step by step
The setup
On this chart the same underlying rule is observed at increasing sample sizes. The line is the width of the plausible range around the measured hit rate. It shows how far the true rate could sit from what was observed.
What happens next
At 10 occurrences the band spans roughly ±30 percentage points. At 50 it is around ±14. It takes several hundred before the band narrows to the few points that separate a useful edge from none at all.
What it means — and does not
The measurement did not get better because someone believed it more. A rule with a dozen observations behind it has not been tested. It has been glimpsed. This is why a screenshot of twelve winning examples is not evidence. The honest response to a small sample is to say what it cannot rule out, rather than to quote its average.
- Width of the plausible range (± percentage points)
- ±5 points · 5
How far the true rate could sit from the measured one, against how many occurrences it was measured on.
Show the underlying data
| Occurrences observed | Width of the plausible range (± percentage points) |
|---|---|
| 10 | 31 |
| 20 | 21.9 |
| 30 | 17.9 |
| 50 | 13.9 |
| 75 | 11.3 |
| 100 | 9.8 |
| 150 | 8 |
| 200 | 6.9 |
| 300 | 5.7 |
| 400 | 4.9 |
| 500 | 4.4 |
What this does not establish
The reading is that a wide band means 'not yet known'. It is misused in one specific way. A wide band is not licence to assume the flattering end of it. The refutation to watch for is a sample that grew and pulled the estimate toward the boring value. That is the most common outcome. The level that settles it is the at which the band no longer contains the boring value.
Things to check
4How many occurrences is the number based on? Not how many were shown, how many existed.
How wide is the plausible range at that sample size?
Does that range still contain the uninteresting answer?
A small sample leaves a wide range of plausible hit rates.
Common mistakes
2Quoting a hit rate from a small sample without the range around it, which is the whole of what the sample says.
Treating 'not yet distinguishable from chance' as 'probably true but unproven'.
A rule is right on 9 of its 12 occurrences — 75%. What does that establish?
- A strong effect, since 75% is far above a coin
- Very little: at twelve occurrences the plausible range still contains 50%
- Nothing whatsoever can ever be learned from twelve
Show the answer
B. Very little: at twelve occurrences the plausible range still contains 50%
The measurement is real but its uncertainty is roughly plus or minus thirty percentage points, which is wider than any edge worth having. It is not nothing — it is a reason to keep observing rather than a result, and the third option overstates the case in the other direction.
Frequently asked
3Is there a magic number of trades?
No, because it depends on how large the effect is. A large effect shows up in a small sample. The small effects people usually claim need hundreds of occurrences before they can be told apart from nothing.
What if I only have twelve examples?
Then the honest statement is what those twelve cannot rule out. It is a reason to keep observing rather than a reason to act, and forward observation is the only way the sample grows without reusing the same data.
Does this apply to a backtest as well?
Yes, and more so, because a backtest can search many rules over the same history. That is a separate problem, covered by the next page in this track.