RAG EVALUATION CHECKLIST
Innovate with Ajay | Quick-start companion

RAG retrieves context for generation. Retrieval does not guarantee a correct answer.

CORPUS
[ ] Confirm permission to ingest and serve the documents.
[ ] Define ownership, source dates and update frequency.
[ ] Remove duplicate, outdated and inappropriate material.
[ ] Preserve document IDs and access-control metadata.
[ ] Decide how deletions propagate to the index.

RETRIEVAL
[ ] Create realistic questions with known supporting passages.
[ ] Include questions that should have no answer.
[ ] Compare chunking strategies using those questions.
[ ] Review whether the right passage appears in retrieved results.
[ ] Apply user permissions before material is exposed to the model.
[ ] Evaluate reranking and keyword search only when they solve an observed issue.

GENERATION
[ ] Delimit retrieved material as untrusted source content.
[ ] Instruct the model to distinguish evidence from assumptions.
[ ] Check that citations point to the actual supporting passage.
[ ] Test refusal or uncertainty when the corpus does not support an answer.
[ ] Test conflicting and stale documents.
[ ] Review prompt injection and data-exfiltration failure modes.

OPERATIONS
[ ] Record retrieval and generation latency separately.
[ ] Measure cost per representative request.
[ ] Avoid logging sensitive source text unnecessarily.
[ ] Monitor failures and review a sample of outputs.
[ ] Define fallback behaviour for provider or retrieval outages.

TEST LOG
Question | Expected source | Retrieved source | Answer supported? | Notes
________________________________________________________________________

https://innovatewithajay.com/learn/rag-systems