VALIDATION DESIGN
Out-of-sample validation vs. walk-forward optimization.
A single OOS test evaluates one frozen specification once; walk-forward analysis repeats selection and evaluation across ordered windows. They answer different questions and are not interchangeable labels.
Before you begin
Learning objectives
- Explain: One holdout, one decision
- Explain: Repeated windows
- Explain: Where leakage enters
One holdout, one decision
A conventional OOS design separates a final period, freezes the candidate and evaluates it once. It is simple to audit but supplies only one market path.
Repeated windows
Walk-forward might select on January to March and test April, then select on February to April and test May. The combined result describes a repeatable selection process, not one permanently fixed parameter set.
Where leakage enters
Choosing window length, objective or constraints after comparing the final result leaks information into design.
- Keep time order intact
- Record window rules before the run
- Use a final holdout for one release candidate
- Do not select a workflow from its final test curve
Common mistakes
- Reading the result without the stated scope and assumptions.
- Changing the rule after seeing an outcome while still calling the data unseen evidence.
Trader Checklist
- Keep time order intact
- Record window rules before the run
- Use a final holdout for one release candidate
- Do not select a workflow from its final test curve
Practice exercise
Choose one of your own trading examples and write one page of rules, evidence and stopping conditions using the principles in "Out-of-sample validation vs. walk-forward optimization.".
What evidence would overturn the conclusion
The conclusion should be overturned or narrowed if a key assumption cannot be reproduced inside the supported scope, the control cannot be executed, or new out-of-sample evidence repeatedly contradicts it.