Build a Small Regression Set Before Editing Your Prompt

A practical approach

Step 1

Collect normal inputs, boundary cases and previously observed failures. Remove sensitive details and keep the expected behaviour beside each input.

Step 2

Score separate dimensions such as factual support, completeness and format validity. Avoid collapsing every judgement into a vague ‘good answer’ label.

Step 3

Run the old and proposed prompts on the same set. Review regressions before accepting an aggregate improvement, especially when errors have different consequences.

Worked example

A summarizer is revised to be shorter. It performs well on simple notes but begins omitting action owners from meetings with several speakers. The regression set exposes this because ‘preserves owner for each action’ is a separate check, rather than one part of an overall style score.

A mistake to avoid

Repeatedly tuning on the same examples can overfit the prompt to the set. Keep some held-out cases and periodically add fresh real-world failures. Store the prompt version and relevant configuration with the result so comparisons remain interpretable.

Try it yourself

Create twelve cases for a task you use frequently: four ordinary, four difficult and four previously mishandled. Write the expected outcome before making the next prompt change.

Your implementation checklist

• Expected outcomes are written in advance.
• Old and new prompts use the same cases.
• High-consequence regressions are reviewed separately.
• Some evaluation cases remain held out.