RESEARCH CONTROL
Why retuning after OOS invalidates the original test.
Once an OOS result changes code, parameters or eligibility rules, that period has influenced selection. The revised model may be tested again, but the original period can no longer provide independent evidence for it.
Before you begin
Prerequisites
Learning objectives
- Explain: Feedback changes the role of data
- Explain: A rename does not reset evidence
- Explain: What to do after failure
Feedback changes the role of data
Suppose threshold 20 fails in the holdout and becomes 24 because the loss was visible. Even if 24 passes, the holdout helped select it and has become research data.
A rename does not reset evidence
Creating version 1.1 or moving a date boundary does not make the same observations unseen. Independence depends on information flow, not filenames.
What to do after failure
Record the failure, explain the revision and create a new candidate.
- Do not overwrite the failed protocol
- Use genuinely later or untouched evidence
- Limit research restarts
- Bind each test to artifact and source hashes
Common mistakes
- Reading the result without the stated scope and assumptions.
- Changing the rule after seeing an outcome while still calling the data unseen evidence.
Trader Checklist
- Do not overwrite the failed protocol
- Use genuinely later or untouched evidence
- Limit research restarts
- Bind each test to artifact and source hashes
Practice exercise
Choose one of your own trading examples and write one page of rules, evidence and stopping conditions using the principles in "Why retuning after OOS invalidates the original test.".
What evidence would overturn the conclusion
The conclusion should be overturned or narrowed if a key assumption cannot be reproduced inside the supported scope, the control cannot be executed, or new out-of-sample evidence repeatedly contradicts it.