VALIDATION FOUNDATION
What out-of-sample validation can and cannot tell you.
Out-of-sample validation tests a frozen specification on data that did not influence its selection. It can expose overfitting and instability, but it cannot prove that future trading will be profitable.
Before you begin
Prerequisites
Learning objectives
- Explain: The separation must be real
- Explain: What a clean run records
- Explain: What the result means
The separation must be real
Research data can form a hypothesis and select one fixed specification. The held-out period stays unseen until logic, parameters, session rules, costs and data sources are recorded.
Example: if a model is selected with data through June, July cannot remain OOS after its result is used to change a threshold.
What a clean run records
A reproducible run preserves the protocol before evaluation and keeps failed results.
- Frozen code and parameter hashes
- Data period, source and cost assumptions
- A pass/fail rule written before the result
- Every attempted specification, not only the winner
What the result means
A clean result only shows whether one frozen hypothesis survived one unseen period under stated assumptions. It cannot remove market, execution, platform or data risk; release monitoring must still challenge live operations.
Common mistakes
- Reading the result without the stated scope and assumptions.
- Changing the rule after seeing an outcome while still calling the data unseen evidence.
Trader Checklist
- Frozen code and parameter hashes
- Data period, source and cost assumptions
- A pass/fail rule written before the result
- Every attempted specification, not only the winner
Practice exercise
Choose one of your own trading examples and write one page of rules, evidence and stopping conditions using the principles in "What out-of-sample validation can and cannot tell you.".
What evidence would overturn the conclusion
The conclusion should be overturned or narrowed if a key assumption cannot be reproduced inside the supported scope, the control cannot be executed, or new out-of-sample evidence repeatedly contradicts it.