Watching Evolution Find an Apple

I've been chasing an idea for a while: a universal genetic algorithm. A normal GA needs custom mutation and crossover for every kind of solution — noise for pixel arrays, swap operations for routes, something else again for 3D models or programs. Every new problem means new operator engineering, which is exactly the thing a "general" algorithm shouldn't need.

The version of the idea that survived twenty rounds of benchmarking is this: never mutate the solution itself. Every individual in the population carries a small genome (a few dozen numbers) plus its own small neural network — a decoder — that translates the genome into a solution. Evolution only ever operates on genomes and network weights, which are just tensors, so the same mutation works for any problem on earth. The only problem-specific things you supply are the fitness function, the output shape, and (optionally) the decoder architecture — things you had to supply anyway.

This week I pointed it at a photograph.

The experiment

The task: match a hidden 96×96 full-color photo of an apple — 27,648 values. The solver receives a fitness score (mean squared error) and nothing else. No gradients, no peek at the target, and no pixel is ever mutated: every image below was decoded from an evolved genome.

That's one run, left to right, from random weights to the final result after 150,000 fitness evaluations (the rightmost tile is the hidden target). The population is 32 individuals, each with its own genome and its own little convolutional decoder; children get noisy copies of both.

The showdown

Same photo, same budget, three competitors:

Left to right: the hidden target; pure decoder evolution (final error 0.0049); a hybrid that switches mid-run to CMA-ES searching a compressed gene space (0.0112); and a traditional GA mutating pixels directly (0.1200 — 24× worse). CMA-ES applied directly to pixels couldn't compete at all: at 27,648 dimensions its covariance matrix is unlearnable in any reasonable budget.

The traditional GA's picture is drowning in confetti because per-pixel mutation never stops injecting noise. The evolved-decoder images are smooth for a structural reason: everything they can express must come out of a decoder, and noise was never in their vocabulary.

What surprised me

Three things from this run that I didn't expect:

The simplest version won. We spent many rounds building a clever pipeline — explore, then compress the best solutions into a compact gene space, then let CMA-ES exploit it. At small budgets that pipeline is worth 4× and beats a traditional GA 10 seeds to 0. But on this long run, a deterministic counterfactual (same seed, hand-off disabled) showed the switch was a mistake: CMA-ES sprints for ~25,000 evaluations, then flattens against the ceiling of its frozen gene space, while plain decoder evolution keeps compounding and finishes 2.3× ahead. Freezing knowledge helps when evaluations are scarce and caps you when they aren't.

The leaf never turned green. The apple has a small green leaf — 3.5% of the pixels. None of the decoder-based runs ever produced green there, not even pure evolution with 150,000 evaluations and total freedom. The bias lives in the lineages themselves: every individual descends from ancestors that paint dark-shapes-on-light, and weight mutations never escaped that basin. Meanwhile the crude pixel GA — unbiased by construction — was slowly finding the green. A decoder's built-in preferences are the source of both its 24× win and its blind spots, and fixing the blind spots looks like a diversity problem, not a budget one.

Honest benchmarking keeps flipping the story. The original version of this idea (a single random decoder that trains on its own outputs) failed every test we threw at it, and most of what we tried since failed too — including several ideas I was sure about. What survives is narrow and specific: evolve the decoders, keep their errors independent, match the architecture to the modality, and know which regime you're in. Every claim above comes from paired-seed runs with exact evaluation budgets, and the negative results are in the repo alongside the wins.

Where this goes

The whole thing is open source — the solver is one call (solve(fitness_fn, output_shape, architecture)), and the repo includes all twenty rounds of benchmarks, raw per-seed results, and a FINDINGS.md that documents the failures as carefully as the successes:

github.com/dadukhankevin/latentspace

Next up: a scheduler that allocates budget between exploration and exploitation by their measured improvement rates instead of a fixed plan, lineage-diversity mechanisms to fix the green-leaf problem, and the one frontier nothing latent has ever won — discrete problems.