Is this pattern real?
The harder half. How to tell a finding from a coincidence.
What this track is for
This track is about judging whether a pattern is real. It is not about learning what one is. Four questions do most of the work. How many times has this been seen? , without the rule? How many rules were tried before this one was reported? And what does the same analysis produce when nothing is going on? It ends on the only test that can refute a rule.
Steps6Each says why it follows the last
You will be able to5Things to read and recognise
Comes afterWhat a trade has to clearRead that track first
The track, in order
6- What a Chart Can and Cannot Tell You: Observation, Interpretation, ClaimStart by separating the three things a chart claim can be: something observed, something interpreted, or something actually tested.A chart is a record of what already happened. It is drawn to one chosen timeframe. It can support an observation, such as a close above an earlier high. It can also suggest an interpretation of that. It cannot, on its own, prove that the interpretation is worth acting on. Those are three different levels of claim. Each one needs more evidence than the one before.
- Sample Size: What Twelve Occurrences Actually ProveNow the question that comes before every other one. A dozen occurrences carries an uncertainty wider than any edge worth having.A hit rate measured on a small number of occurrences comes with an uncertainty band around it. At a dozen occurrences that band is huge, wide enough to contain both a strong edge and a losing rule at the same time. Twelve occurrences at 75% is entirely consistent with an underlying rate of 50%. The number to look at is not the percentage. It is how much the percentage could move if you saw the next twelve.
- Base Rate First: What the Market Did AnywayThen the benchmark. A rule is only credited with the difference between its rate and what happened anyway. That turns most headline percentages into small effects.Every claimed hit rate has to be read against what happened without the rule. If a market closed higher on 55% of days, a rule that is right 60% of the time has added five percentage points, not sixty. The base rate is the first number to establish, and the last one most people look for. Skipping it is how an ordinary result gets published as a discovery.
- Search Enough Rules and One Will Fit by ChanceThe next question is about the process, not the rule. Test forty rules on data with no signal in it and the best one still looks impressive.If you test forty rules against data with no signal in it at all, the best of the forty will still look good. Not because it works, but because you looked forty times. The number that gets reported is the maximum of a search, and a maximum is biased upward by its very nature. This is the single most common reason a rule that tested well stops working the moment it meets new data.
- The Shuffle Test: Comparing a Result Against NothingSo build something to compare against. Break the link between rule and outcome. Re-run it, and see whether the real result is unusual at all.A number on its own cannot be judged. The shuffle test builds something to judge it against. Take the same data, randomly break the link between the rule and the outcomes, re-run it, and record the result. Do that a few hundred times and you have a picture of what this rule produces when it means nothing at all. If the real result sits comfortably inside that picture, the result IS nothing at all.
- Forward Versus Backward: A Rule Tested on Its Own Data Is UntestedFinish with the only test that can actually refute a rule: data it played no part in choosing. Everything before this describes. Only this one checks.A rule discovered in a stretch of data and then judged on that same stretch has not been tested. The data already knows the answer, because it is what suggested the rule. The only evidence that can refute a rule is data that played no part in choosing it. In practice that means observing forward, from today, with the rule written down first.
When you finish this track
Next track: Investing money, not trading it
Contributions, units, corporate actions, and the arithmetic that decides what a return means.
See what that track covers →All tracks and the full library
Every ordered track, plus the searchable index of explainers and worked case studies behind them.
Open the learning index →Base Rate
The base rate is how often an outcome occurs.
Read more