← Keep Working

Compare two prompt interventions on the same task

When one prompt produces an appealing answer, it is tempting to keep polishing that wording. A more useful question is whether the intervention helps with the specific failure you are trying to correct. Start with a comparison small enough to inspect completely.

The method below is a proposed personal experiment, not a new Keep Working benchmark. It makes no claim that the deck outperforms a plain request.

Freeze the starting point

Save a short task, the incomplete answer and the failure you want to address. Remove private material before sharing it. Keep the model settings and available tools the same for both candidates where the system permits that control.

For a fictional example, an answer recommends three options but never states the decision criteria. The task is to make the comparison usable, not to produce a longer answer. Write that distinction before trying either intervention.

Give the plain baseline a fair chance

Compare one selected deck card with a direct baseline such as “State the decision criteria and apply them to each option.” Begin each attempt from the same saved context. Do not let one candidate see the other answer or benefit from extra explanations.

The deck API lets you inspect the current card text. Record the exact text you used, rather than only a category name: the category contains multiple cards. A saved card is a reproducible input, not a guarantee of a reproducible model output.

Decide what counts before scoring

Use a short rubric connected to the original failure. For the fictional comparison, ask whether the answer names relevant criteria, applies them consistently, identifies missing information and supports a next decision. Mark unsupported new claims as a separate problem.

Avoid awarding points for length or forceful wording unless those qualities are actually part of the task. If both answers remain unusable, record that result. A winner between two poor answers is not automatically a useful intervention.

Hide the label during review

If practical, ask a reviewer to inspect the two answers without knowing which prompt produced each one. Keep the task and rubric visible. This reduces an obvious source of preference for the wording you wrote, although a small informal comparison still has many limitations.

Repeat with other starting examples before treating the result as a habit worth adopting. Record each example separately. One successful correction on a list of options does not demonstrate better performance on code, factual research or long autonomous work.

Keep the result modest

A useful record has the input, both interventions, both outputs, rubric, observations and limitations. “The direct baseline supplied the missing criteria in this example” is a defensible observation. “This prompt makes models reliable” is a much larger claim.

Use the evidence checkpoint when the next question concerns actual task completion. Use this comparison only when choosing the intervention itself. If the plain request works well, keeping it is a valid result; the purpose of the experiment is a better working process, not a more elaborate prompt collection.