The systems graveyard — what failed strategies can teach you
Studying an unsuccessful strategy can improve your next research project, provided you can reconstruct its rules, data and order execution. A loss alone does not explain whether the assumptions, test or execution failed, or whether market conditions changed. This lesson shows how to investigate weaker results, separate confirmed errors from hypotheses and identify questions the evidence still cannot answer. Two worked examples and a case-review template help you prepare your own account of what happened. That gives you a clearer basis for further research, a pause or closing the project, without assigning causes the data do not support. We reconcile the data before choosing the next action.

Choose a claim that the archive can actually test
An old report with a positive total can still be unusable. An old negative report can contain a valid implementation. Start the review with a specific claim about a named version and its evidence, rather than deciding that the strategy must either be revived or buried.
Our synthetic archive contains historical simulation version R1, with six completed simulated trades and a total of USD 18 net under a fixed, fully stated cost model. There are no live executions in this case. Its entry rule requires the relevant input to be available by 10:00 on the decision clock. A later archive check finds that one input used for a selected trade first became available at 10:03. The rule was written before this review; it is not being changed to exclude an inconvenient outcome.
Assume the source timestamp is verified as availability time, not merely a file-modification time or an unrelated timezone. That distinction is essential in a real investigation. Here it lets us ask a narrow reproducibility question: could this particular historical decision have been made using the information permitted by the frozen rule?
Separate the verified error from the repaired result
| Record | Trades | Net total in USD |
|---|---|---|
| Original historical simulation | 6 | 18 |
| Trade requiring the late input | 1 | 12 |
| Controlled corrected replay under the supplied assumptions | 5 | 6 |
The input was available three minutes after the required cutoff. That selected trade therefore cannot remain an admissible entry under the stated rule. In this controlled example, replaying the corrected logic removes that trade and changes no other signal, quantity, cost or outcome. Its USD 12 net contribution disappears: 18−12=USD 6 across the remaining five simulated trades.
This is not a general rule that a faulty trade can always be subtracted without rerunning the test. In another system, removing one entry could change available capital, later signals, overlapping positions or costs. Our assumption deliberately fixes those other effects. A real repair needs an actual replay and reconciliation of any downstream differences, with both reports preserved.
Do not mistake a positive corrected sum for a successful diagnosis
The corrected USD 6 total is still positive. It establishes the total under the supplied repaired simulation, not a durable expected return. We have no individual-result dispersion, complete research-trial history or independent evaluation sample. The example neither estimates a confidence interval from the aggregate nor concludes that the remaining strategy is profitable.
The official Pine Script data-request documentation gives a concrete platform example of historical requests exposing values before they were knowable. The general audit question is availability, regardless of platform. A later database value that looks reasonable cannot be assumed to have existed at the earlier decision time.
Keep two findings distinct: the original R1 implementation used unavailable information, and the revised result still lacks the evidence required for a performance claim. The first finding is supported by the stipulated timestamp and rule. The second is an evidence limitation. Neither needs a story about a trader’s character or a declaration that the market permanently stopped working.
Retain what the next project can use
Preserve the original code, data version, parameter settings, decision records, timing evidence and report. Add the repaired version and a concise difference log. State the downstream-invariance assumption if using this small reconciliation as a teaching example; do not hide it in the final total.
Had a real account contained the original executions, repairing a historical model would not erase their cash effects. The account ledger and the corrected research result would remain different records. This case is simulation-only, so no actual loss or transaction is removed. That boundary should remain clear when a review is shared with another researcher.
A reasonable next decision can be to pause, close the project or design a fresh evaluation after the repair. The historical error does not oblige further spending or live experimentation. Continue to the longer original cases for costs, sizing, uncertainty and project closure. The useful inheritance from a failed report is a reproducible explanation of what was verified and what remains unknown, rather than a new promise built around a cleaner chart.
Three exercises with worked answers
The exercise data are hypothetical. Choose an answer, then check the calculation.
Continue to the full case-review framework
The original detailed cases, diagrams and uncertainty discussion remain intact. Use the availability example to distinguish a verified implementation defect, its corrected report and the still-open performance question.
1. What actually stopped working?
“The strategy stopped working” can mean several things: the latest period lost money, costs exceed the benefit shown in a simulation, the rules cannot be executed, or further research no longer fits the budget. These are not equivalent findings. Before looking for an explanation, record what you are assessing, over which period and against which expectation.
The unit of analysis might be an entry rule, a complete versioned strategy, a portfolio, an account or a firm’s business. An account containing several strategies and changing exposure does not isolate one rule’s performance. Breaching a commercial prop programme’s limit can end participation without settling the strategy’s expected value. A firm’s insolvency requires analysis of its liabilities, liquidity and governance, not just a trade chart.
A market edge is used here as a hypothesis that a defined approach has positive expected value after costs under specified conditions. A sample mean only estimates it. Even a positive result outside the data used to design the strategy does not guarantee repeatability.
Distinguish insufficient support for the initial hypothesis, deterioration in observed results and a project that is no longer worth its cost to the owner. These can coexist. None justifies a story about the “death of an edge” when it is unclear what was established beforehand.
2. A version history, not an expiry date
Strategies do not share a useful life measured in a fixed number of months. Nor must parameters change simply because they have remained unchanged for a long time. You need a record showing which rules were evaluated and what information was available then.
- Hypothesis: describe the mechanism, instrument, decision times, operating conditions and circumstances that would contradict the idea.
- Test: preserve the code or rule version, data source and scope, costs and list of variants tried. Separate design data from evaluation data.
- Observation: label the environment — historical data, demo or live account — and record deviations from the assumptions.
- Review: record the decision and its basis. A rule change creates a new version; correcting a calculation error also requires recalculating the comparisons.
- Archive: retain versions rejected before deployment too. Not every hypothesis has to become a traded strategy.
This is an order for documentation, not a career timetable. Moving to the next step is not automatic approval for live trading. 15.4 — backtesting and forward testing covers test design; here we use the evidence to explain an unsuccessful attempt cautiously.
First check whether the original result was credible
- Fitting to the data: from how many rules was the winner selected? Repeatedly consulting the same test set influences selection. Calling it “out-of-sample” does not remove that problem.[1]
- Future information: did a signal use a candle’s closing price before it closed, a subsequent data revision or an indicator that changes its historical values? Check when information became available, not just the chart’s appearance.
- Selection: were difficult periods, instruments or abandoned trials excluded? Define the population and keep a record of exclusions.
- Executable assumptions: do prices, commissions, financing and order-size restrictions represent the environment being studied?
A weaker later result is a reason to check these issues, not automatic proof of overfitting. Conversely, a confirmed timestamp error can invalidate a test without waiting for another series of trades.
3. Markets, costs and execution
Changes in volatility, liquidity or costs can change strategy performance. Attributing a particular effect requires more than dates lining up. Compare the same rules, instruments and decision windows, allowing for changes in size and measurement.
| Hypothesis | Evidence needed | Limit of the conclusion |
|---|---|---|
| Different market conditions | Results for an unchanged version under conditions defined before evaluation; distributions of volatility, trend, timing and events. | A filter added after seeing losses is a new hypothesis, not an independent explanation of the same data. |
| Higher costs | Execution reports, dated tariffs, commissions, financing and comparable price benchmarks. | You can calculate a cost difference without inferring the provider’s intentions. |
| Worse execution of larger orders | Submission time, benchmark price, order size, partial fills, rejections and liquidity conditions. | More slippage may coincide with a different market. Comparing two averages alone does not isolate the size effect. |
| More competition for the same effect | Evidence of changed participant behaviour and achievable prices, not just a popular strategy description. | Publishing a setup followed by a loss does not prove that competitors removed the edge. |
| A changed instrument or contract | Product specifications and agreements before and after the change, P/L calculation rules and trading availability. | A change can prevent execution of one version without invalidating similar methods on other products. |
Do not deduct spread and slippage twice. In a report based on actual fills, the difference between purchase and sale prices already reflects the spread and achieved slippage. Add only charges not included in that P/L field. A backtest based on theoretical mid-prices is different: it still needs an appropriate execution-cost model.
4. Rules, sizing and cash flows
A departure from the plan is established when you can identify the rule in force and the decision that contradicted it. That does not automatically explain the entire loss. Equally, a profitable off-plan trade does not establish that the change was beneficial. Keep both records: financial outcome and rule compliance.
Not every modification after a short sample is a mistake. Correcting an incorrectly calculated commission may be justified immediately. Fitting parameters to recent losses without a recorded hypothesis is different. In that case, label the new version and do not present its improvement on the same data as independent validation.
The same trade sequence, different drawdowns
Synthetic example, not an account history. Five consecutive trades each finish at −1R net. Each trade’s initial risk is a fixed fraction f of current equity. Positions do not overlap and there are no deposits or withdrawals. In this simplification, equity after the sequence is C₅ = C₀ × (1 − f)⁵.
| Risk fraction f | Decline from C₀ | Drawdown |
|---|---|---|
| 1% | 1 − 0.99⁵ | 4.90% |
| 1.5% | 1 − 0.985⁵ | 7.28% |
| 2% | 1 − 0.98⁵ | 9.61% |
Only sizing changes in this model, not the R outcomes. The larger capital decline is therefore not evidence of worse entry rules. In actual trading, a larger position can also change costs and execution, so identical R outcomes would need checking. These risk levels are comparison assumptions, not recommendations.
A deposit does not undo a loss
A second synthetic example (PLN: Polish złoty). Equity falls from PLN 100,000 to PLN 80,000, followed by a PLN 20,000 deposit. The account again shows PLN 100,000, but trading P/L remains negative: 100,000 − 100,000 − 20,000 = −PLN 20,000. With the deposit at the end of the period, the return before it is −20%. If the plan sets risk at 1% of current equity, the risk amount increases after the deposit from PLN 800 to PLN 1,000, or 25%.
The deposit itself establishes neither an error nor a motivation; it may follow a prior plan. It must not be counted as profit or the restored balance used as evidence of strategy improvement. 16.5 — documenting performance develops the distinction between returns and external cash flows.
5. A review template and two worked cases
Start by reconciling the data to the source report. Then reconstruct the applicable rules, check costs and exposure, and only then compare results under different conditions. 15.10 — strategy audit covers the complete review cycle. This template serves a narrower purpose: recording what an unsuccessful attempt actually taught you.
| Field | What to record |
|---|---|
| ID and version | Rules or code, instrument, environment and period. Separate the strategy from the account in which it was used. |
| Hypothesis | What effect was expected, under which conditions and after which costs? |
| Prior evidence | Material existing before the period being assessed: data, test results, dates and number of variants tried. If none exists, say so. |
| Observation | What happened in the sample? Trade count, result with units, costs, cash flows and departures from the rules. |
| Confirmed error | Specific evidence, such as an omitted commission. If no error is established, do not fill the field with a guess. |
| Causal hypotheses | Possible explanations and alternatives. They may coexist; plausibility alone does not establish any of them. |
| Missing evidence | What could distinguish the explanations? Can it be obtained, or does it no longer exist? |
| Decision | What will you do now, until when and on what basis? Separate protective action from assessment of the edge. |
| Lesson for the next attempt | One specific safeguard or research question arising from this case, without generalising to the whole market. |
Case A: a positive result disappears when costs are counted correctly
All figures are synthetic. We assess version A1 of a EUR/USD strategy, tested on 100 trades from 1 April to 30 June 2026. The simulation uses mid-prices, a fixed position of 1 lot = EUR 100,000 and initial price risk equivalent to a 5-pip move. At a pip size of 0.0001 USD per EUR, the pip value is USD 10 and 1R = USD 50. Total P/L before costs is USD 800, averaging USD 8 = +0.16R per trade.
The assumed constant spread is 0.3 pips and the round-trip commission is USD 6 per lot. There is no overnight financing, slippage or change in size. Taxes and fixed project costs are outside this trading-P/L calculation. The 0.3-pip spread is the cost of the complete round trip relative to mid-prices in this model; it is not charged twice.
| Item | Per trade | In R | 100 trades |
|---|---|---|---|
| P/L before costs | USD 8 | +0.16R | USD 800 |
| Spread: 0.3 × USD 10 | USD 3 cost | −0.06R | −USD 300 |
| Originally reported result | USD 5 | +0.10R | USD 500 |
| Omitted commission | USD 6 cost | −0.12R | −USD 600 |
| Result after both costs | −USD 1 | −0.02R | −USD 100 |
Confirmed: the report omitted commission despite including it in the cost specification. The correction explains the full difference between the reported +USD 500 and corrected −USD 100. This is not an account of an edge later taken away by the market; the earlier report was incomplete.
Not established: whether the rules have positive expected value on other data with achievable execution. Individual-result dispersion and the history of variant selection are unavailable, so we do not calculate a confidence interval from the total alone. A sample of 100 is not a threshold confirming an edge. Decision in this example: do not launch the version on the basis of the old report; correct the cost model and repeat the evaluation, retaining both reports. Lesson: reconcile every cost component before interpreting the result.
Under the same assumptions, a commission-free alternative with a 1.2-pip spread would cost USD 12 rather than USD 9 per trade. The difference is 3 / 50 = 0.06R, not 0.18R. No commission does not necessarily mean a lower total cost. This compares two models; it is not a broker recommendation.
Case A1: 100 trades, total spread USD 300 and commission USD 600. Bars are successive P/L totals, not components to add together again. Other costs are excluded as specified in the example.
Case B: a loss does not establish that an edge has disappeared
Synthetic example. For unchanged version B1, we analyse a period selected in advance: 1–31 July 2026. Thirty trades total −6R net, giving a mean of −6 / 30 = −0.20R. Assume sample standard deviation s = 1.40R. Each recorded R-multiple is P/L after trading costs divided by the risk amount fixed before entry. Taxes and fixed project costs are excluded.
For this calculation, assume independent observations from the same normal distribution. A two-sided 95% confidence interval for the mean, with unknown population standard deviation, uses Student’s t distribution:[3]
mean ± t₀.₉₇₅;₂₉ × s / √n−0.20 ± 2.04523 × 1.40 / √30 ≈ −0.20 ± 0.523R
Interval: approximately −0.723R to +0.323R.
The interval includes negative and positive values. Under the assumed model, this sample does not reject a zero mean at the 5% significance level. It establishes neither the absence of an edge nor that the loss was “just bad luck”. Comparable earlier data are also missing, so we have not estimated a change in the edge.
Real observations can be dependent, asymmetrical and drawn from changing market conditions. The simple t interval may then be inappropriate; adding trades alone does not fix those assumptions. Prior selection of windows and repeated testing also need consideration. A confidence level is not the probability that a strategy works. The ASA cautions against replacing reasoning with one statistical threshold.[4]
Decision in this example: continue observation in simulation and obtain reference data and cost checks. This limits research expenditure; it is not a finding that an edge has disappeared. Lesson: report the negative total with its uncertainty instead of inventing a cause.
Case B1: n = 30, s = 1.40R, assuming independent observations from the same normal distribution. R is net P/L divided by initial risk fixed before entry. The interval does not establish that there is no edge or that the loss was “bad luck”.
What can Monte Carlo show — and what can it not show?
Simulations can compare a drawdown with the outcomes of a defined model. They cannot unconditionally determine whether a system is “alive”. Disclose the reference data, dependence assumptions, path length, sizing rules, costs and number of simulations. The reference sample should not contain the period being assessed or be selected solely because it looks best.
With non-overlapping trades, no external flows and a fixed risk fraction f, the equity model is Cᵢ = Cᵢ₋₁ × (1 + f × Rᵢ), not a product of the R-multiples alone. Include initial equity C₀ in the running peak so that a first-trade loss is not missed. If the model permits complete capital depletion, state when a path ends instead of continuing through negative equity. A drawdown magnitude below the simulated 95th percentile of drawdown magnitudes only meets that particular model criterion; it does not establish safety or explain the loss.
6. Avoiding selective stories
A winning strategy’s story does not tell you how many similar attempts failed. A failure story can be selective too. Survivorship bias arises when only objects that survived a selection process remain available for assessment. A register of successful and abandoned variants helps reveal the scope of that selection.
Your own dataset can also be selected: recent data are not automatically unbiased, and a strategy you designed yourself is not immune to overfitting. Record every trial within the declared project, its start date, rule changes and reason for ending. When assessing someone’s complete performance history, define the account coverage; do not treat one account as their entire record.
An interview or screenshot can suggest a research question. Without the complete data scope, it cannot estimate success rates, causes of failure or the time needed to achieve results. Treat missing data as a limitation, not as evidence that the author is dishonest.
7. Scenarios to investigate — not a ranking of “dead systems”
The following are hypothetical situations, not a study of any national trading population. Each identifies a question to ask before drawing a conclusion.
- Scalping with a tight stop: is the full cost small relative to R, not just small in pips?
- Trend following with a few large winners: was dependence on rare moves part of the hypothesis? Profit concentration alone does not establish an error, but its risk and sampling uncertainty need examination.
- Adding to losing positions: what were total exposure and the stopping condition? A high win rate for individual closures does not describe the risk of the whole arrangement.
- A news-release strategy: does the test use information available at decision time and achievable execution prices?
- A programme with a loss limit: are you assessing strategy rules or compliance with a particular programme’s terms? Record whether the environment is simulated or live.
We assign no “death date” to these methods. The object of review is a particular version, dataset and execution process.
Protective action can precede a finding about the cause. Suspending exposure does not automatically establish that an edge has disappeared.
8. Acting on the findings
The states below organise a decision. They are not a validated scale or a capital-allocation algorithm. The review date follows the research plan and the ability to carry it out, not an obligation to endure more months of losses.
Leave the rules unchanged
The review found no reason to modify them within the scope examined. Record that scope and the unknowns. “No error found” does not mean “safety established”. The project can remain in simulation.
Gather further observations
Specify which evidence would distinguish the hypotheses. Collecting more observations without a research question does not resolve the problem. Many gaps can be investigated on historical data or demo.
Reduce exposure
If the project already involves live trading, reducing risk is a separate protective decision. There is no common rule to “always halve the previous position”. Smaller exposure can still lose money and does not replace correcting a defect.
Suspend trading
Do not wait for a required n when a limit has been breached, orders cannot be controlled or important costs remain unknown. Apply the prepared risk procedure to existing positions; closing the application does not close positions or cancel pending orders.
Archive the version
End work on a version if you do not want to finance further research, cannot reconstruct essential data or the idea no longer fits your constraints. Preserve the reason: “insufficient evidence” should not turn into “the market destroyed the system” years later.
9. More research or closing the project?
Returning to an old idea can make sense as a new research stage if you can identify what another trial would teach you and how you would limit its cost. Changing the broker, filter or decision window does not prove the problem is fixed. A comparison must account for other conditions that changed at the same time.
| What is known | Possible next step | What not to infer |
|---|---|---|
| A cost or code error is confirmed. | Correct it, recalculate the same sample, then evaluate on data not used to design the correction. | A better corrected result does not guarantee an out-of-sample edge. |
| Results deteriorated; the cause is unresolved. | Investigate alternative explanations, continue simulation or close the project because of research cost. | Uncertainty does not require continued trading. |
| The rules cannot be executed under current conditions. | Suspend this version; consider a new product or execution specification. | The old backtest cannot simply be transferred to new conditions. |
| The earlier result cannot be reconstructed. | Archive it with the missing evidence documented, or rebuild a test from an explicit hypothesis. | Missing material does not establish that all similar strategies are ineffective. |
10. What to retain for the next project
Attach the data files, cost specification, code or playbook, reports before and after the correction, and variant register to the case review. Organise dates, named time zones and filenames so another person can reconstruct the analysis. Remove credentials and anonymise personal information before sharing.
Close the note with three sentences: what was established, what remains unresolved and what follows for the next attempt. For example: “The test omitted commission. No post-correction edge was estimated on independent data. Before the next evaluation, I will check the report against a complete cost-component table.”
You need not design another strategy immediately. You can return to education, pause or end the project. For questions about time and commitments, use 16.3 — assessing your circumstances. For supporting a household without trading withdrawals, use the model in 16.8. The number of abandoned strategies alone neither measures competence nor determines the length of a future career.
FAQ — frequently asked questions
After how many losses should I change a Forex strategy?
How can I distinguish drawdown from deterioration of an edge?
Can I revisit a previously rejected system?
Is a positive average enough to continue?
What should I examine in a commercial prop programme?
How does a note on a retired version differ from an audit?
Sources and their scope
- Bailey, Borwein, López de Prado and Zhu, The Probability of Backtest Overfitting, author version, February 2015 (PDF). Strategy-selection risk across multiple tests, not an expiry date or a universal qualification threshold for trading.
- Reserve Bank of Australia, Box B: The Recent Japanese Yen Flash Event, February 2019. The event and limitations of causal explanation, not the history of a particular strategy.
- NIST/SEMATECH, Confidence Limits for the Mean. The t-interval formula and confidence-level interpretation. Application requires checking model assumptions.
- American Statistical Association, Statement on Statistical Significance and P-Values (PDF), 7 March 2016. Limits of drawing conclusions from a single statistical threshold.
Sources checked on 8 September 2026. Cases A, B and C, the sizing comparison and the deposit calculation are illustrative educational models, not investment results. The template and decision states are editorial tools, not a validated test. This material is educational, not an investment recommendation.