Skip to content
Is the market open today? Today
SearchCtrl K
EnglishHinglish
My learning
Is the market open today? Today

Find your next idea

Glossary term

Overfitting

Also called: overfitting, curve fitting, data mining bias, multiple comparisons, p-hacking.

Overfitting is arriving at a rule that describes only the data it was built from. It does not describe anything that repeats. The rule fits the past and then fails on new data. It happens whenever the same data is used both to choose the rule and to judge it. That includes tuning thresholds. It includes trying many variations and reporting the best. It includes adjusting a chart's parameters until it looks convincing.

How it is read

The mechanism is a maximum. Test forty rules against data containing no signal at all. The best of the forty will still look impressive. A maximum over many draws is biased upward by construction. So the number that matters alongside any reported result is how many variations were examined before it. A result corrected for that search is worth far more than a larger number that was not. Trying thresholds by eye is the same search. It just keeps no record of how many were tried.

What it does not tell you

Overfitting is a property of the PROCESS, not of the rule. So it cannot be detected by looking at the rule. It cannot be detected by looking at how sensible its reasoning sounds either. The only reliable check is data that took no part in choosing it. Correcting for a search is also not a rescue. It makes a result honest. It does not make a weak one strong.

Out-of-Sample Test

Definition →

Sample Size

Definition →

Null Model (Shuffle Test)

Definition →Track

Is this pattern real?

Open the track →
Reference. Standard statistical practice on multiple comparisons and model selection.
The lesson that uses this term

Search Enough Rules and One Will Fit by Chance

If you test forty rules against data with no signal in it at all, the best of the forty will still look good. Not because it works, but because you looked forty times. The number that gets reported is the maximum of a search, and a maximum is biased upward by its very nature. This is the single most common reason a rule that tested well stops working the moment it meets new data.

Open the lesson