Ballpark Genius predicts MLB game outcomes — win probability, projected score, the “Pitcher Edge” and “Offense Edge” breakdowns you see on a game page. Behind that is a nightly champion/challenger loop: tune a candidate model, evaluate it against a fixed holdout set, and promote it only if it’s genuinely better. Over the course of about a week, I found and fixed three separate bugs in that loop — each one plausible-looking, each one silently defeating the whole point of having a holdout set in the first place.

The prediction surface this loop feeds — a real game, produced by the champion model after all three fixes below.

Bug 1: the eval set that stopped growing

The gate that decides whether to promote a new model runs entirely against games flagged held_out = true. That flag is a persisted boolean, and it was only ever set by a one-off backfill script — the daily import path that adds new games as the season progresses never set it. So the holdout set froze at whatever the last manual backfill happened to cover: 120 games, static, while every game imported afterward leaked into the training set instead. Two eval reports run a month apart came back byte-identical — same 120 games, same Brier score to four decimal places — because the loop was judging every new candidate against a yardstick that hadn’t moved in a month.

The fix was to stop treating “mark the holdout set” as a manual, one-time step: importGames now assigns heldOut at insert time for every game, using the same holdout-selection rule the backfill script used, so the eval set grows deterministically with every import and can’t drift out of sync again. Re-running the backfill once to catch the 67 games that had already leaked brought the holdout set current — 187 games instead of 120.

Bug 2: a fixed threshold that noise could clear

With a live holdout set, the next problem was the promotion gate itself. It compared aggregate composite scores against a flat epsilon = 0.005 — if the candidate beat the champion by more than that, it got promoted. But per-game Brier-score noise on a ~180-game holdout runs around ±0.02, several times larger than the threshold meant to detect real improvement. The gate was mostly reacting to noise: on at least one occasion it treated a +0.025 swing that was almost certainly sampling variance as a genuine, promotion-worthy gain. And because the epsilon was fixed, a growing holdout set — which should make the gate more discerning as more data comes in — never actually lowered the bar for what counted as a real signal.

The fix replaced the flat threshold with a paired one-sided significance test on per-game composite scores: compute the champion-minus-candidate delta on the same holdout games for both models, and promote only when the lower bound of a 95% confidence interval on that delta clears zero. As the holdout set grows, the standard error shrinks, so smaller real gains become detectable over time — the gate gets sharper with more data instead of staying flat forever. A dry run against the live 180-game holdout immediately started returning honest reasons instead of noisy verdicts: a worse candidate failed on a negative lower bound; a directionally-better model (a real +0.0010 edge, ±0.0066) correctly failed as not yet statistically significant rather than being promoted on what was still mostly noise.

Bug 3: the tuner that optimized for the wrong exam

Fixing the gate exposed the last problem. The coordinate-descent tuner that produces each candidate ran to convergence minimizing composite score on a single ~785-game walk-forward sample, then got graded on the separate holdout set. That’s an overfitting setup by construction: the tuner wins on the sample it was allowed to see and loses on the one it wasn’t — in this case, +0.017 on walk-forward against −0.023 on holdout, which now correctly failed the significance gate every single time.

The fix added an explicit shrinkage-to-champion penalty to the tuning objective — composite + λ · penalty, where the penalty is the range-normalized distance from the current champion’s weights — and chose λ by k-fold cross-validation rather than by hand: for each candidate λ, tune on k−1 folds and score the held-back fold on pure composite (no penalty term), and keep the λ whose out-of-fold score is best. λ = 0 stays in the search grid, so a genuinely generalizing tune is never blocked from winning outright.

On the same live holdout, the heuristic candidate’s performance went from −0.0234 (overfit) to +0.0003 — and the CV procedure chose λ = 2, which is the tuner correctly concluding that the data doesn’t support moving off the champion at all. That microscopic +0.0003 edge also correctly fails a companion fix from the same pass: a minimum-effect-size floor on the significance gate, added after noticing that heavy shrinkage makes a candidate converge toward the champion so closely that its paired variance collapses too, letting a statistically “significant” but practically meaningless edge slip through a bare > 0 check. Requiring the confidence interval to clear a real minimum, not just zero, closed that gap.

Why the order of these fixes mattered

None of the three bugs were individually exotic — a flag that never got set, a threshold that didn’t account for sample noise, an objective that didn’t match its own exam. What made this a multi-week chase rather than a single fix is that each one made the next one look like the real problem. A frozen holdout set makes a noisy gate look stable, because it’s comparing against the same games every time. A noisy gate makes an overfitting tuner look successful, because noise-driven promotions occasionally go through. Only once the eval set was current and the gate was honestly measuring uncertainty did the tuner’s overfitting become visible as its own distinct bug rather than getting attributed to “the gate is too strict” or “the holdout is too small.”

The loop is mechanically sound now — it evaluates on a live holdout, requires statistically real improvement at a real effect size, and tunes candidates against an objective that generalizes instead of memorizing. heuristic_v9 is already near-optimal for its current feature set, and neither a re-tuned version of itself nor a logistic-regression challenger beats it meaningfully right now, which is the gate correctly reporting a true negative rather than a broken positive. Getting further gains from here is a features-and-data problem. It’s no longer a loop-machinery problem, and proving that distinction was most of the actual work.

— James