Brier score
Before each of four observations, forecast the probability that EUR/USD will close the next hour above the current Bid. Forecasts are 0.8, 0.6, 0.3 and 0.1; realised event labels are 1, 0, 1 and 0. Under the binary 0–1 convention, the Brier score is (0.04 + 0.36 + 0.49 + 0.01) / 4 = 0.225. Constant 0.5 forecasts for the same events score 0.25. This demonstrates arithmetic; four observations do not establish future forecasting quality.
Define the event and forecast time first
The label 1 must mean the same condition, horizon and data source in every observation. Record the forecast before learning the outcome. Do not mix an hourly price rise with reaching a target before a stop: these are different events. Specify handling of missing data and ties. Overlapping horizons can create dependent observations, so the number of forecasts need not equal the number of independent pieces of information.
Use an appropriate comparison
Lower is better on the same sample under the same convention. Predicting 0.5 always scores 0.25, but for rare events a frequency estimated from earlier data may be a better baseline. Do not select it using test outcomes. Record the library’s convention too: a sum over all classes or a multiclass formulation may have a different scale. Check the positive-class label rather than assuming a probability column refers to an upward move.
A good score is not a profitable strategy
The Brier score evaluates probability forecasts, not cash trading results. It reflects more than calibration alone; inspect calibration and stability out of sample separately. Even a genuine 60% probability with payoffs of +1R or −2R gives expectancy of 0.6 × 1R − 0.4 × 2R = −0.2R before costs. A trading decision additionally requires payoffs, execution, sizing and risk constraints.
Further learning and sources
10.1 Trading under uncertainty – how to judge decisions whose outcome cannot be known; 11.9 AI and LLMs in trading – applications, limits and risks; 15.9 Forex Trade Journal — R-Multiple, Expectancy, SQN, and MAE/MFE.
Primary sources (checked 2026-10-11): scikit-learn — brier_score_loss; scikit-learn — probability forecast evaluation.
