Out-of-Sample Test
Also called: out of sample, out-of-sample, forward test, walk forward, holdout, in sample vs out of sample.
How it is read
This is the only check that can actually refute a rule. That is why a strong in-sample curve is a description of history. It is not evidence. In practice the cleanest version is forward observation. That means you start the clock now, not in the past. Write the rule down. Then watch what it does from today. A rule that holds up on data it was not selected from was not selected by chance. One that flattens at the moment the data becomes new was.
What it does not tell you
One out-of-sample stretch is itself a sample, and a small one. So a disappointing result is not proof of failure. A good in-sample result was not proof of success either. The larger danger is procedural. Adjusting the rule after seeing the out-of-sample result converts the only genuine test back into fitting. Once that is done, there is nothing left to check the rule against.
Related concepts
Search Enough Rules and One Will Fit by Chance
If you test forty rules against data with no signal in it at all, the best of the forty will still look good. Not because it works, but because you looked forty times. The number that gets reported is the maximum of a search, and a maximum is biased upward by its very nature. This is the single most common reason a rule that tested well stops working the moment it meets new data.
Open the lesson