If you kept the best of 200 prompts, that score is inflated

Prompt optimisation has a selection problem nobody corrects for, and it is the same error that kills trading backtests. Here is how much of your winner's margin the search itself explains.

You try 200 prompt variants, keep the one scoring 0.87, and report 0.87. But you did not run one experiment — you ran best-of-200, and the winner's score carries the search inside it.

Draw 200 samples from a pool with no real differences at all and the top one still sits about 2.6 standard deviations above the rest, purely by luck. We tested exactly that: 200 variants from an identical distribution. The winner's margin came out at 2.87 sd against the 2.61 noise alone would hand you. Almost the entire margin was the search.

The same arithmetic kills trading backtests, which is where we met it. A strategy search here advertised a 10% false-discovery rate; shifting the signals against price so the true answer was known to be nothing, it measured 39%.

The fix is boring and it works: re-score the winner on data it was never selected on. Selection cannot inflate a score it did not choose. If that is impossible, at minimum compare the winner's margin against what best-of-N returns from noise.

One more thing worth knowing. If the spread between your variants is smaller than the noise in a single variant's score, you cannot rank them apart at all — and the answer is more trials per variant, not more variants.