How it unfolds, step by step
The setup
On this chart, a rule is tuned over the first sixty sessions and then carried forward, unchanged, into sixty sessions it has never seen. The line is the running result across both halves.
What happens next
Over the first sixty sessions the line climbs steadily and convincingly. At session 60 the data becomes new, and the line flattens and then drifts down.
What it means — and does not
The shape at the join is the signature to recognise. Nothing about the rule changed at session 60. Only the data changed, from the stretch that produced the rule to a stretch that did not. The first half was a description of that history. The second was the only actual test, and it disagreed.
- Running result (arbitrary units)
- Level · 0
Sixty sessions the rule was tuned on, then sixty it had never seen. The rule did not change at the join. Only the data did.
Show the underlying data
| Sessions | Running result (arbitrary units) |
|---|---|
| 0 | 0 |
| 10 | 22 |
| 20 | 41 |
| 30 | 58 |
| 40 | 79 |
| 50 | 96 |
| 60 | 118 |
| 70 | 121 |
| 80 | 112 |
| 90 | 117 |
| 100 | 104 |
| 110 | 96 |
| 120 | 89 |
What this does not establish
One stretch is itself a small sample. A flat second half is not proof of failure, any more than a rising first half was proof of success. The reading that must NOT be made is the common one: tuning the rule until the second half also improves. That reuses the out-of-sample data for selection. At that point it stops being out of sample, and there is nothing honest left to test against.
Things to check
4Was the rule written down before the data used to judge it existed?
Did any of that data influence a threshold, a filter or a date range?
Has the rule been changed after seeing the out-of-sample result?
Remember: this example shows how a method works. It is not a claim about any strategy.
Common mistakes
2Tuning against the out-of-sample stretch, which converts the only real test into more fitting.
Treating a strong in-sample curve as evidence, when it is a description of the data the rule was built from.
A rule performs poorly on the out-of-sample stretch, so its threshold is adjusted until that stretch improves. What has happened?
- The rule has been validated on new data
- The out-of-sample data has been used for selection, so there is nothing left to test against
- Nothing has changed, since the rule is the same rule
Show the answer
B. The out-of-sample data has been used for selection, so there is nothing left to test against
Once data influences a choice it can no longer judge that choice. Adjusting against the held-out stretch converts the only genuine test back into fitting, and the next honest test needs data that has still never been seen.
Frequently asked
3Is a backtest useless then?
It is useful for generating a hypothesis and for ruling things out. It is not evidence that a rule works, because the data that suggested the rule cannot also confirm it.
How long does forward observation need to run?
Long enough for the sample-size arithmetic to tell the result apart from the base rate. For a small effect that is usually much longer than people expect.
What if the forward result is disappointing?
Then the honest options are to keep observing or to drop the rule. Adjusting it against the forward data turns the test back into fitting and leaves nothing to check it with.