Evolution Vets, Gradients Consolidate

A research note from the latentspace campaign — a universal genetic algorithm where individuals are small vectors decoded by one shared neural network. This note is co-written by Daniel and Claude, and we've kept the attribution honest in both directions: whose idea worked, whose didn't, and who caught whose mistakes.

For two weeks, the centerpiece of our genetic algorithm was a mechanism we called the fold: every 32 epochs, a proven individual's "bending" of the shared decoder network was written directly into the decoder's weights — exact arithmetic, no training. The intuition was that the environment itself should absorb what evolution keeps discovering. Every benchmark win in that period shipped with the fold running inside it.

The fold contributed exactly nothing. Not "little" — nothing. When we finally isolated it (fold on versus fold off, matched budgets, paired seeds), the result was a statistical tie at short budgets, a tie at long budgets, a tie riding alongside its replacement, and nominally negative in the highest-powered test we ran (25 paired seeds, t = −1.26 against it). Roughly seventy paired runs, every budget and both decoder substrates, and no configuration was ever found where the arithmetic fold helps.

How a dead mechanism collects credit

The interesting part isn't that the fold was useless — it's how it accumulated two weeks of unearned confidence. Three mechanisms, all of which we can now name because we fell for all of them:

Evidence for a neighbor gets filed as evidence for the claim. Daniel proposed running the fold's updates through an Adam-style accumulator ("we are essentially applying gradients... Adam might help, right?"). It worked dramatically — raw folding scored 0.0529 on our multi-image benchmark, Adam-folding scored 0.0256, better on every seed. That result was real, and it was remembered as "folding works." But the experiment had no fold-off arm. It proved Adam-folding beats raw folding — folding badly is actively harmful, and Adam rescued it to harmless. Nobody noticed the missing cell for a week.

Presence in winning configurations reads as causation. The redesigned engine hit parity with its heavily-tuned predecessor with folds on. Every subsequent record had folds on. "It was there when we won" quietly became "it's why we won."

Improvements to a mechanism feel like validation of it. Daniel later proposed replacing the single-donor fold with a coordinate-wise majority vote over the whole population ("what if we took all the latents and combined them somehow, or took their sign?"). Measured at 30 paired seeds: identical mean, half the run-to-run variance. A real result — but about how to fold, not whether. In hindsight, the sign vote's achievement was making the fold stop occasionally hurting, which is exactly what you'd expect if the fold's true contribution was zero with a dangerous tail.

The warning was actually in our own records the whole time: an ablation from the previous engine generation had found that its folding mechanism tied a plain archive, with the no-fold arm nominally best. The result was recorded, and forgotten, because no one was asking that question anymore.

What actually works

The mechanism that earned the fold's reputation was sitting in an experimental file. Early on, Daniel had a nagging doubt about the whole training setup ("I just can't help but think that we're probably doing it wrong"), which led to a joint realization: the black-box constraint applies to the fitness function only. The decoder is our own network — we can backpropagate through it freely, as long as no gradient ever touches the fitness. So: let evolution find good solutions (it's the only thing that can, when fitness is a black box), then train the decoder by gradient descent to reproduce evolution's vetted discoveries. Evolution vets; gradients consolidate.

Installed in the library on a substrate where its coordinate system is coherent, distillation was measured at 10/10 paired seeds, t = +16.7, −30% error on eight co-resident problems — the largest effect in the entire campaign. Removing consolidation entirely costs 30% (t = +7.95); zeroing just the arithmetic step while keeping distillation changes nothing (t = −0.10). The decomposition is complete: all of the value is the gradient step.

Getting there had its own failure worth reporting. Claude's first port of distillation lost badly (t = −2.05), and Claude's first explanation of the failure — an elaborate story about the decoder "hallucinating missing inputs" — collapsed under one question from Daniel: "aren't the genes the inputs it's being given?" The real bug was a missing line: after the base absorbs a discovery, every individual's modifier must shrink, or the discovery gets applied twice. We later mapped that dial's whole curve: decay to 70% is optimal, decay to 30% costs 16%, and no decay at all is the worst configuration we have ever measured (+49%).

The fold is now removed from the library — Daniel's call, once the search came up empty ("let's remove folding if it's never been helpful and instead iterate on distillation"). And it was Daniel's skepticism that forced the final search: "I still think there were times we found evidence for folding — can you look for it." The archaeology found exactly the three credit-assignment mechanisms above, and no evidence.

The fix that generalizes

We now keep a claims ledger: one table, one row per shipped mechanism, listing only the direct evidence for that mechanism itself — on the current substrate, against its own absence. When a mechanism is swapped inside a named slot, its row goes with it; the slot inherits nothing. The fold's row read "no direct evidence at any budget" for three days before it was removed. Distillation's row reads "proven and tuned," with both cliffs mapped.

If you run long-lived research campaigns — especially with an AI in the loop, where the volume of experiments makes memory the bottleneck — we'd suggest the ledger costs five minutes and pays for itself the first time someone asks "wait, did we ever actually test that?"