Null Model (Shuffle Test)
Also called: null model, shuffle test, permutation test, randomisation test, null hypothesis, the null.
How it is read
A number cannot be judged on its own. A null model is what gives it something to be judged against. Say the real result sits comfortably inside the shuffled spread. The evidence then cannot tell the rule apart from nothing. That is a far more useful statement than the percentage was. A result outside the spread is evidence. It still has to survive correction for how many rules were searched. It also has to be confirmed on data that did not select it.
What it does not tell you
A null model is only as good as what its shuffle destroys. A shuffle that leaves the time ordering intact preserves any drift in the underlying. So a rule that is simply long a rising market can beat that null. It can do that while being nothing of the kind. The mirror-image error is a null that accidentally preserves the effect under test. That makes every genuine result look ordinary. Always ask what the shuffle kept, not just what it broke.
Related concepts
The Shuffle Test: Comparing a Result Against Nothing
A number on its own cannot be judged. The shuffle test builds something to judge it against. Take the same data, randomly break the link between the rule and the outcomes, re-run it, and record the result. Do that a few hundred times and you have a picture of what this rule produces when it means nothing at all. If the real result sits comfortably inside that picture, the result IS nothing at all.
Open the lesson