Out-of-sample testing

By Jarosław Wasiński • Published: May 25, 2026 • Updated: Oct 11, 2026

Data withheld from strategy selection and parameter fitting, then used to assess performance on unseen observations. Once results guide further changes, that data is no longer a clean holdout for the revised strategy. No fixed percentage or trade count guarantees a meaningful test; avoid leakage and include realistic costs.

Example. Rules are developed on 2020–2023 data, selected with separate 2024 validation, and the frozen version is evaluated on an untouched 2025 test. Development expectancy of +0.30R versus +0.08R in the test does not identify one cause. If a filter changes after reviewing 2025, that year is no longer clean OOS for the revised version.

The sample must be separate from selection

OOS means data not used to select the strategy being evaluated, rather than simply a differently named file. Choosing parameters, instruments or exceptions after seeing the test also uses its information. Further research cycles can be conducted transparently, but distinguish developmental validation from a final test. Keep decisions and versions so it is clear what the model’s developer had genuinely not seen.

Chronology and data leakage

Random shuffling in market time series can mix future and past information. Fit scaling, feature selection and thresholds only with data available before the test. Trades or labels covering a future interval can cross sample boundaries; define exclusions appropriate to their horizons. A calendar gap alone does not eliminate every leak. See scikit-learn’s data-leakage documentation for examples.

What does a weaker result mean?

Possible causes include overfitting, changed market conditions, higher costs, an implementation error or small-sample uncertainty. Investigate diagnoses rather than automatically declaring overfitting. Report completed-trade count, dispersion and all charges. A short positive OOS period is not a profitability certificate either; it remains an estimate under a particular model and measurement window.

Documentation makes the test reproducible

Retain code version, source data, publication schedule and product specification. Define how open positions crossing sample boundaries are handled and how indicators receive warm-up history. Earlier observations used to calculate an indicator are different from future results used to select parameters. The record should reproduce the exact test version without silently inserting knowledge from a later period into the previous evaluation.

A practical protocol

Record sample boundaries, information-availability timestamps, evaluation criteria and the action plan before reading results. Keep full costs and comparable sizing. If the test informs changes, arrange a new honest assessment on later data rather than claiming the previous observations remain untouched. In the course: backtesting and forward testing. Source checked on 11 October 2026.

Diagram and worked example

Educational infographic: Out-of-sample testing — Educational diagram; model data and assumptions are stated in the graphic.
Open the full-size graphic

FAQ

Is a 70/30 split always sufficient?

No. The appropriate split depends on horizon, observation count, dependence and the selection process.

Can I inspect OOS repeatedly?

Yes, but if that affects the model or selection, the observations become research data rather than a clean final test.

← Back to the full glossary