← Research

Parity study — why the backtest correlation collapsed

Generated by parity_study.py, 2026-07-08. Luck SD from 200,000 simulated 82-game seasons (p_win=0.5, OT share 0.22): 8.49 points.

1. League compression: SD of team points, 2010-11 to 2025-26

Each team's points normalized to an 82-game pace (short seasons 2012-13, 2019-20, 2020-21 included at pace).

SeasonTeamsGP/teamSD(pts/82)
2010-113082.013.27
2011-123082.011.73
2012-133048.016.46
2013-143082.015.26
2014-153082.015.91
2015-163082.012.86
2016-173082.015.13
2017-183182.015.44
2018-193182.013.65
2019-203169.814.12
2020-213156.019.27
2021-223282.020.28
2022-233282.018.90
2023-243282.017.63
2024-253282.014.84
2025-263282.013.18

2010-2019 mean SD 14.38 (range 11.73-16.46); 2024-25 14.84, 2025-26 13.18.

2. Backtest decomposition

signal SD = sqrt(SD(actual)^2 − luck SD^2); luck share = luck var / actual var; r ceiling = signal SD / SD(actual) — the correlation a perfect strength oracle would score; err_sig SD = sqrt(RMSE^2 − luck SD^2) — the model's error on true strength; r expected = corr implied by that signal + that error; r true-inputs = corr of the end-of-season-known-inputs prediction with actual points.

SeasonSD actSD predMAERMSEr (Pearson)rho (Spearman)signal SDluck sharer ceilingerr_sig SDr expectedr true-inputs
2022-2318.98.311.514.00.7140.70716.920%0.89311.10.7460.908
2023-2417.68.910.713.20.6710.62615.523%0.87610.10.7330.874
2024-2514.87.511.013.30.4260.38512.233%0.82010.20.6290.878
2025-2613.26.210.612.80.2730.21910.141%0.7659.60.5540.844

Compression-only counterfactual

Freeze the model's information at its 2022-23 level (implied info-error SD 12.7 pts, from r = Vs/sqrt((Vs+Ve)(Vs+Vl)) calibrated on 2022-23) and let only the signal SD move:

Seasonsignal SDobserved rcompression-only rSE(r), n=32
2022-2316.90.7140.7140.09
2023-2415.50.6710.6770.10
2024-2512.20.4260.5680.15
2025-2610.10.2730.4750.17

3. Was the 2025-26 projection worse? Churn covariates

SeasonRMSEerr_sig SDmean NNoDatamean RetSharer(err, NNoData)r(err, RetShare)
2022-2314.011.10.60.79-0.09-0.19
2023-2413.210.10.50.800.05-0.22
2024-2513.310.20.50.78-0.05-0.15
2025-2612.89.60.70.81-0.050.15

Pooled across 128 team-seasons: r(|err|, NNoData) = -0.04, r(|err|, RetShare) = -0.12.

Variant comparison (RMSE / Pearson r):

Variant2022-232023-242024-252025-26
base14.0 / 0.7113.2 / 0.6713.3 / 0.4312.8 / 0.27
age14.0 / 0.7213.3 / 0.6713.2 / 0.4312.7 / 0.29
ageG14.0 / 0.7213.3 / 0.6713.2 / 0.4312.6 / 0.30

4. Verdict

Mostly parity compression; the model's absolute accuracy did not degrade. The model's error actually improved (RMSE 14.0 -> 12.8, MAE 11.5 -> 10.6), but the spread it is trying to rank shrank dramatically: SD of actual points fell 18.9 -> 13.2, and after removing the 8.5-point schedule-luck floor the true strength SD fell 16.9 -> 10.1; luck now accounts for 41% of actual-points variance vs 20% in 2022-23, and a perfect strength oracle tops out at r = 0.76 (was 0.89). The historical table confirms 2024-26 is the tightest league since at least 2010 (SD82 14.8 and 13.2 vs 2010-2019 mean 14.4, and vs 18.9-20.3 in 2021-23). Attribution: freezing the model's information at its 2022-23 level and shrinking only the signal predicts r = 0.48 for 2025-26 — i.e. compression alone accounts for 0.24 of the 0.44 Pearson-r drop (54%). The residual 0.20 is only ~1.2x the n=32 sampling SE of r (0.17), so evidence of genuine relative degradation is weak — and three independent checks say the model itself held up: the signal-error SD (sqrt(RMSE^2 - luck^2)) fell 11.1 -> 9.6; the true-inputs prediction still ranks well (r 0.84 in 2025-26 vs 0.91 in 2022-23), so the input projections, not the points model, carry what residual there is; and |error| is uncorrelated with roster churn (pooled r vs NNoData -0.04, vs RetShare -0.12), killing the churn/rookie alternative.

2026-27 implications. The 2026-27 predicted-standings spread is narrow — SD of ExpPts is 5.0 vs a recent true-signal SD of ~11.1. That gap is correct behavior, not a bug: a point forecast should be shrunk toward the mean by its own uncertainty (the backtest's SD(pred) was likewise ~6 vs SD(act) 13-19), and in a compressed league the shrinkage is heavier. Do not widen ExpPts to match the actual standings spread. STRENGTH_SD = 10.5 represents strength uncertainty beyond schedule luck; the backtest's realized signal-error SD averaged 9.9 over 2024-26, so 10.5 remains about right, if a shade conservative (implied total per-team spread sqrt(10.5^2 + 8.5^2) = 13.5 vs realized RMSE 12.8; 9.5-10 would match exactly — not worth retuning). The practical consequence is humility on playoff odds: with true-strength SD ~11 points against a luck SD of 8.5 plus ~10 points of projection error, mid-table playoff probabilities belong in the 35-65% band and predicted ranks in the middle two quartiles are close to interchangeable — the sim's wide PlayoffProb bands are a feature of the era, not a model defect.