Test Retrieval Before Blaming the Answer Model

A practical approach

Step 1

Create questions with labelled supporting documents and passages. Include paraphrases, exact identifiers and questions the collection cannot answer.

Step 2

Inspect retrieval results without generation. Track useful evidence in the returned set and note irrelevant or duplicated passages.

Step 3

Feed known-good evidence to the answer step. This isolates writing and grounding errors from indexing, parsing or search failures.

Worked example

A learner asks about the prerequisite for an advanced course. The source states that API experience is required, but retrieval returns an introductory course page instead. A more capable model may still answer incorrectly because the needed evidence is absent. Fixing the audience filter or query handling addresses the actual failure.

A mistake to avoid

A single combined score can hide different failure modes. Retrieval may improve while generated answers become less cautious, or vice versa. Keep question-level traces so the team can identify which stage needs attention.

Try it yourself

Label twenty questions with the passage that should support each answer. Run retrieval alone, then generate answers from the labelled passages. Compare the two error lists before choosing your next improvement.

Your implementation checklist

• Questions have labelled supporting evidence.
• Unanswerable questions are included.
• Retrieval and generation are assessed separately.
• Failures are assigned to a specific stage.