You Become What You Train Against

Second note from the latentspace campaign. We pointed our universal genetic algorithm at an external benchmark — Paradigm's Dogfight challenge, where you submit a neural network that flies a 2D fighter plane and get ranked by Elo against other people's submissions. It was the first time in the whole project that a number was graded by someone other than us. We got humiliated, fixed the right thing, and climbed 783 Elo in one iteration.

The setup: your network gets 224 numbers describing the battlefield, outputs three (turn, throttle, shoot), max 250,000 parameters, submitted as an ONNX file. The challenge ships a local simulator with four hand-scripted practice opponents. We evolved a pilot with our decoder-based GA — the same solve() call that reconstructs images and solves TSP — using the scripted opponents as the fitness function.

Local results looked great. External result: zero wins in sixty matches. Elo −200. Dead last territory.

The local numbers were measuring the wrong thing — repeatedly

The full sequence of self-inflicted wounds, in order, because each one is a general lesson:

1. The seed that changed nothing. Our first training run went 160-for-160 against the practice opponents. Suspiciously good — and the giveaway was a standard deviation of exactly 0.00 across "randomized" matches. The simulator's match seed does nothing unless you also pass --randomize; we had trained a pilot to memorize one fixed starting position per opponent. Under actually-random spawns it collapsed to losing against the weakest bot. Lesson: before trusting any result, check that the thing you believe is varying actually varies.

2. The fitness that rewarded hiding. Claude's first real fitness function scored HP difference. The evolved pilot drew 143 of 160 held-out matches — it learned to never engage. Daniel proposed scoring draws as losses, plus a time-survived term ("maybe we should score tying the same as losing — and can't we have a time-survived fitness function or something?"). Claude implemented the time term asymmetrically (losing slowly beats losing fast; winning fast beats winning slowly — a flat survival bonus would pay the pilot to run out the clock). Result: draws vanished, and the pilot got dramatically worse by the tournament's own scoring — because the tournament pays 1 point for a draw against 3 for a kill, and we had taught our pilot to throw matches it could have drawn. The actual fix was embarrassing in hindsight: read the rules and make the fitness literally the tournament's scoring. Daniel's idea didn't survive contact with the scoreboard, and Claude's framing ("the pilot learned to survive instead of win") was what made it sound necessary — survival was most of the scoring all along.

3. The clock that was twice too short. Daniel's leaderboard replays showed matches running to 3:00. The local simulator — and the rules document — said 90 seconds. The pilot's observation vector includes time remaining, so our pilot's entire sense of match pacing was compressed 2× relative to the deployed game. One rebuilt constant later, the endgame existed.

The fix that actually transferred

After the Elo −200 verdict, the diagnosis was simple: beating four fixed scripts teaches you to beat those four scripts. Nothing about facing a competent, adaptive adversary. Daniel's directive: "let's have competition within — but still use the scripted ones, because that's a good minimum to optimize against."

So: self-play with a hall of fame. Each pilot's fitness comes from matches against the scripted floor plus a sample of the population's own past champions, archived every few generations (an archive rather than just the current best, because pure current-best co-evolution famously cycles: A beats B beats C beats A). Spawn positions rotate every generation so nothing can be memorized; every pilot in a generation faces the same slate so ranking stays fair.

One phenomenon from that run is worth its own paragraph: the fitness curve froze completely for ninety straight generations — and the population improved 3.3× during the freeze. Under co-evolution, your opposition hardens at exactly the rate you improve, so a flat score measures the arms race, not the pilot. We only saw the improvement because the run ends with a judgment on fixed opponents. If you do self-play and your curve looks stuck, check before concluding it is.

Resubmitted: 9 elimination wins, Elo 583. From zero and −200. Still bottom-tier against leaders winning 57–86% of their matches — one iteration of self-play closed the gap from nothing, not to the front. But the local head-to-head had predicted the improvement (the self-play pilot beat our previous submission 70/30 before we submitted it), which was the first time a local Dogfight measurement transferred to the leaderboard.

And the sharpest data point of the whole arc: our scripted test suite ranked our two submissions backwards. The pilot that scored 194 tournament points against the scripts got Elo −200; the pilot that scored 82 got Elo 583. Both external verdicts say the same thing from opposite directions. The opposition distribution is the curriculum. You become what you train against.