How it unfolds, step by step
The setup
On this chart the same rule is re-run thirty times with the outcomes shuffled. Any relationship between the two has been destroyed on purpose. Each point is one shuffle's result.
What happens next
The shuffled results scatter between roughly 44% and 58%. That spread is what pure chance produces on this data, with this many observations. The dashed line is the real, unshuffled result: 57%.
What it means — and does not
The real result sits inside the noise. It is a perfectly respectable-looking 57%, and the shuffles reached 58% with no relationship present at all. The test has not proved the rule is worthless. It has shown that this evidence cannot tell the two apart. That is a different statement from a percentage on its own, and a much more useful one.
- Shuffled result (%)
- The real result: 57% · 57
Thirty runs with the link between rule and outcome destroyed. The real result sits inside the scatter.
Show the underlying data
| Shuffle number | Shuffled result (%) |
|---|---|
| 1 | 51.2 |
| 2 | 47.8 |
| 3 | 55.1 |
| 4 | 49.6 |
| 5 | 53.4 |
| 6 | 44.9 |
| 7 | 57.8 |
| 8 | 50.3 |
| 9 | 46.2 |
| 10 | 52.7 |
| 12 | 56.4 |
| 14 | 48.1 |
| 16 | 54 |
| 18 | 45.7 |
| 20 | 51.9 |
| 22 | 58.2 |
| 24 | 47.3 |
| 26 | 53.1 |
| 28 | 49 |
| 30 | 55.6 |
What this does not establish
The test is only as good as what the shuffle destroys. Shuffling outcomes while leaving the time ordering intact keeps any drift in the underlying. A rule that is simply long a rising market can then beat that null while being nothing of the kind. The check is to ask what the shuffle preserved. A null that accidentally keeps the very effect being tested is the level at which this whole method quietly fails.
Things to check
4What exactly does the shuffle break, and what does it leave intact?
How many shuffles? Enough to see the tail, not just the middle.
Where does the real result sit in that distribution?
Compare the observed result with the shuffled results before drawing a conclusion.
Common mistakes
2Running a handful of shuffles and reading the middle, when the question is about the tail.
Building a null that preserves the effect under test, which makes any result look ordinary and is the mirror-image error.
A real result of 57% sits inside a shuffled spread that runs from 44% to 58%. What has the test shown?
- That the rule is worthless
- That the rule works, since 57% is above the average of the shuffled results
- That this evidence cannot tell the rule apart from chance
Show the answer
C. That this evidence cannot tell the rule apart from chance
A null model says what the same analysis produces when nothing is going on. A result inside that spread has not been distinguished from nothing — which is a statement about the evidence, not a verdict that the rule is worthless.
Frequently asked
3How many shuffles are enough?
Enough that the tail is filled in, not just the middle. Thirty is enough to draw. Several hundred to a few thousand is usual before a claim rests on where the real result sat.
What does it mean if the real result is outside the scatter?
That chance alone rarely produces it on this data, which is evidence and not proof. It still has to survive the search correction and confirmation on data it was not selected from.
Is this the same as a backtest?
No. A backtest asks what a rule would have produced. A shuffle test asks what the same rule produces when it cannot possibly be working, which is the comparison that makes the first number readable.