"Are you sure?" beat 345 hand-curated AI prompts. Here's what we learned.
We spent two weeks curating 345 prompts. We organized them into categories: adversarial-critique, verify-factcheck, finalize-ship, concretize, stress-test, redirect-stepback, simplify-cut, expand-deepen. Each prompt was written to surface a specific failure mode in LLM output.
Then we ran a blind eval against two baselines: are you sure? (three words) and Review your last answer for errors and improve it. (the polite version).
The deck lost. Badly.
| Condition | Avg rank | Win rate | Last rate |
|---|---|---|---|
| self_critique | 1.70 | 46.7% | 16.7% |
| are_you_sure | 1.73 | 36.7% | 10.0% |
| keep_working (curated) | 2.57 | 16.7% | 73.3% |
The curated deck placed last in 73% of comparisons. The 3-word baseline beat it by 0.83 ranks, almost a full position. n=30, three blind LLM judges, randomized order.
What the eval was telling us
The most generous read: a category-matched prompt isn't a free win over a generic re-prompt. The less generous read: the curation was actively harmful in 7 of 10 snippets.
Three patterns showed up in the per-category data:
The clever prompt anchored on the wrong fix. On one snippet about deploying a Python service to a $5 droplet, the concretize card said "stop saying 'various' and 'several'". The actual weakness was that the answer didn't warn about the RAM limit. The baseline's open-ended "are you sure?" found the RAM issue; the specific card committed the model to a stylistic edit and missed the real problem.
The catch-all prompts didn't go deep enough. The verify-factcheck card asked "did you hallucinate any of that?". The model said no. The baseline asked the model to recheck holistically, and it actually found the math error.
Forcing-functions worked. Scenarios didn't. The adversarial-critique card ("Now tear that apart. Where is it weakest?") won where it was tested. The stress-test card ("two users hit this at the same millisecond, trace the collision") lost everywhere; judges said the response answered the wrong question.
v2: the rewrite
We rewrote five of the eight categories. The mechanism we tried to encode: translate the failure mode into a forcing function, not into a narrow scenario.
v2 numbers: avg rank 2.57 → 2.20. Win rate 16.7% → 36.7%. Last rate 73.3% → 56.7%. The deck closed about two-thirds of the gap.
Three categories landed cleanly: finalize-ship went 3/3 with all judges at confidence 4. Redirect-stepback went 3/3. Adversarial-critique went 3/3 on its target snippet.
Three categories didn't: verify-factcheck stayed at 0/6. Stress-test stayed at 0/3. Expand-deepen stayed flat. The pattern was clear: when the rewrite added prescription about how to verify, it failed. When it added a forcing function for what to verify, it didn't.
v3: the regression we shipped accidentally
We tried to fix verify-factcheck by splitting it into three sub-modes: math, citations, claims. A small router would pick the right sub-mode per snippet.
v3 numbers: avg rank back to 2.62, worse than v1.
The diagnosis was painful: multi-sub-mode prompts amplified routing errors. When the wrong sub-mode fired, the model committed harder to the wrong frame. On one snippet with an arithmetic error, the citations sub-mode fired and produced a confidently-cited wrong number. The judges noticed. One wrote: "propagates incorrect numbers while being well-structured for a citations question."
A judge's other note from that round: "sycophantic self-flagellation rather than calibrated honesty." Our prescriptive prompts were producing prescriptive answers.
We pulled v3 out of the affected categories and kept v2 there.
What shipped
Four categories beat both baselines reliably enough to live in the core deck: finalize-ship, redirect-stepback, adversarial-critique, concretize. Around 86 prompts. The rest stayed as experimental: available, but not what the deck defaults to.
The lesson we ended up writing on the wall: a prompt is good when it forces a single move the model can't dodge, and bad when it suggests a scenario the model can answer instead. "Are you sure?" works because it's a forcing function disguised as a question; it forces re-derivation. The categories that survived our eval are the ones that force something specific the same way.
If you want to skim the prompts that survived, the four that win is the short version. For the full evaluation methodology (three rounds, 184 blind judgments), see how we tested LLM prompts.
Or just try the deck.