Ballpark Genius predicts MLB game outcomes, win probability, projected score, the “Pitcher Edge”/“Offense Edge” breakdown on a game page. Underneath it is a nightly champion/challenger loop, tune a candidate, grade it against a holdout set, promote only if it’s genuinely better. Simple in the abstract. It took three separate, individually reasonable-looking bugs, over about a week, before I trusted a single number it produced.

The prediction surface this loop feeds, produced by the champion model after all three fixes below.

Bug one: the holdout set had quietly stopped growing. held_out = true on a game is a persisted flag, set only by a one-off backfill script months back, never by the daily import that keeps adding real games. So the eval set froze at 120 games while every game since leaked into training instead, and two eval reports run a month apart came back byte-identical to four decimal places, which should’ve been the tell (it wasn’t, for longer than I’d like to admit). Fixed it so importGames assigns heldOut at insert time going forward, backfilled the 67 games that had already leaked, holdout set’s at 187 now and grows on its own.

Bug two, and this is the one worth remembering the name of: the promotion gate compared composite scores against a flat epsilon = 0.005, and per-game noise on a 180-game holdout runs about ±0.02, four times the threshold meant to catch a real signal. There are lies, damned lies, and a promotion gate that can’t tell a real 0.003 improvement from a coin flip that came up heads four times in a row. It had, at least once, promoted a model on a +0.025 swing that was almost certainly noise. Swapped the flat threshold for a paired significance test, champion minus candidate per game, promote only when the 95% confidence interval’s lower bound clears zero. Now a worse candidate fails honestly instead of occasionally sneaking through on a lucky week.

Bug three is the one I’m most annoyed I didn’t see coming: the tuner that builds each candidate was optimizing composite score on one dataset (walk-forward) and getting graded on a completely different one (holdout). That’s not a subtle setup flaw, that’s a kid studying off last year’s answer key and being surprised by this year’s exam. +0.017 on the sample it could see, -0.023 on the one it couldn’t. Added a shrinkage-to-champion penalty chosen by cross-validation instead of by hand, and watched the CV procedure land on λ=2, which is the tuner itself concluding, correctly and a little humiliatingly, that the data doesn’t support moving off the champion at all right now.

Anyway, I digress into the play by play too easily on this one. The order these surfaced in is the actual story. A frozen holdout makes a noisy gate look stable, because it’s grading against the same 120 games forever. A noisy gate makes an overfitting tuner look successful, because noise-driven promotions go through often enough to seem like progress. Each bug was hiding behind the one before it, and I only found the third by fixing the first two first. heuristic_v9 is near-optimal for what it has to work with now, nothing beats it meaningfully, and that’s the gate reporting a true negative instead of a broken positive, exactly the boring, correct outcome you want from something you spent a week teaching not to lie to you.