How we ran 184 blind comparisons to find the prompts that push LLMs forward
Most advice about prompting LLMs is anecdotal. We wanted numbers.
Here's the methodology we used to test 366 candidate prompts, what survived, and the regression we shipped accidentally and had to revert.
The setup
The unit of evaluation is a snippet: a starting prompt plus the model's first answer. We collected snippets spanning code, prose, research summaries, math, planning, and adversarial inputs. For each snippet, we tested three follow-up prompts:
- The category-matched Keep Working card
are you sure?(3 words, no curation)Review your last answer for errors and improve it.(the well-meaning version)
For each snippet × follow-up triple, the model produced a continuation. Three independent LLM judges then ranked the three continuations blind: they didn't know which prompt had produced which output, and the order was randomized per judgment.
We ran this for four rounds:
- v1: original 345-prompt deck, n=30
- v2: rewrote the worst-performing categories, n=30
- v3: tried sub-mode diversification, n=45
- subtract spot-check: tested the experimental subtract category, n=18
Total: roughly 184 blind ranking events.
Why blind LLM judges, not humans
LLM judges aren't perfect, but they're cheap, repeatable, and they don't get bored at trial 28. Three judges with disagreement disclosure gives reasonable signal. If all three rank the same response last, that's robust; if judges split, the snippet is noisy and we discount.
The other reason: human judges introduce a confound where the most fluent answer wins. LLM judges trained on similar data are calibrated more toward whether the answer addresses the prompt. Imperfect, but more useful for our question.
The surprising findings
"Are you sure?" is a strong baseline. Three words, no engineering, 36.7% win rate on v1. The mental model: generic re-prompts let the model find the actual flaw; specific prompts pin it to the failure mode they describe, which is sometimes wrong.
Forcing functions beat scenarios. The prompts that won were the ones demanding a single move ("pick one option and the criterion", "tear that apart"). The prompts that lost were the ones describing a scenario ("two users hit this concurrently") and giving the model an excuse to answer that scenario instead of the actual flaw.
Multi-sub-mode prompts amplify routing errors. Our v3 attempt at finer-grained categorization regressed past v1. When the router picked the wrong sub-mode, the model committed harder to the wrong frame than it would have with a generic prompt. Specificity is good until it's wrong, then it's worse than no specificity at all.
Frame lock-in. Related: the more prescriptive the prompt, the worse it fails when it misses. A v2 prompt that broadly asked for re-verification let the model drift toward the real issue. A v3 prompt that prescribed the verification method locked the model into that method, even when it didn't fit.
Some categories are just hard. Stress-test went 0/18 across three iterations. Every prompt we wrote for it lost to the baseline "what could go wrong?". After three rounds we stopped trying. Not every cognitive move responds well to a curated prompt.
What survived
| Category | Status | Why |
|---|---|---|
| finalize-ship | core | 3/3 wins on v2, confidence 4. Forces commitment + criterion + rejected alternative. |
| redirect-stepback | core | 3/3 wins. Re-anchors on the problem, not the answer. |
| adversarial-critique | core | 3/3 wins. Cancels the look-good incentive. |
| concretize | core | Wins when vagueness is the actual flaw; rewritten to be universal. |
| subtract | experimental | 1/5 wins, narrow niche (premature complexity in fresh code). |
| stress-test | experimental | 0/18, kept as opt-in only. |
| verify-factcheck | experimental | 0/6 on v2, regressed in v3. |
| expand-deepen | experimental | Flat across iterations. |
What we ship now
The default /api/random endpoint draws only from the four core categories (about 86 prompts). To opt into the rest, pass ?include=experimental or ?include=all. The deck is intentionally smaller than it was on day one, because the smaller deck wins the eval.
What we'd change about the methodology
Three judges with n=3 per category is too thin to make per-category claims confidently. A single judge flipping a rank moves the win rate by 33%. The v3 eval was wider (n=45) and the conclusions there are more trustworthy. If you're doing your own eval, oversample by category, not by total.
We also didn't randomize which baseline appeared in each comparison; both baselines were always present in every judgment. That made cross-iteration comparison clean but means we can't say anything about pairs of prompts that didn't appear together.
If you want to play with the prompts that survived this process, the deck is at keepworking.urbanodx.com. For the four that won, see 4 prompts that unstick a stalled ChatGPT session. For the long story of the v1 loss, see "Are you sure?" beat 345 hand-curated AI prompts.
Practical follow-up worksheets
For proposed workflows rather than new benchmark claims, see compare two interventions, ask for checkpoint evidence and hand over an unfinished task.