Design a Document Knowledge Assistant You Can Evaluate

Define the assistant’s scope

Use a small collection of course guides as the worked scenario. The assistant answers questions about prerequisites, curriculum and learning format. It does not confirm unlisted dates, create enrollments or answer private administrative questions. This narrow scope gives you a concrete way to judge whether the system is useful before adding more documents or tools.

Name the document owner and the intended readers. Decide which sources are current and which are historical. If two guides disagree, resolve the business question or mark the conflict explicitly; do not expect retrieval ranking to decide which policy is authoritative.

Create a source register

Give each document a stable identifier and record its title, source URL, owner, version, audience and access scope. Keep upload time separate from the date the content became effective. This register becomes the connection between the original material, its derived chunks and the citations shown to readers.

document_id: course-api-foundations
version: reviewed-2026-10-04
audience: beginner-developers
access_scope: public
owner: programme-editor
source_section: prerequisites

The values above are illustrative metadata, not claims about an actual production record. In your implementation, derive access scope from trusted application data. A user-supplied label should not be able to make a private document public.

Inspect parsing before indexing

Parse a representative sample and compare the output with the original documents. Check headings, lists, tables, footnotes and links. If the parser drops a prerequisite table or separates an exception from its rule, indexing the broken text will only make the error easier to retrieve.

Start with a simple chunking strategy and preserve source and section identifiers. Compare it with a structure-aware alternative using real questions. Avoid choosing chunk size from a universal rule of thumb; the right trade-off depends on document structure, retrieval behaviour and the evidence required for your questions.

Evaluate retrieval independently

Write questions and label the passages that should support each answer. Include paraphrases, exact course names, ambiguous requests and questions the collection cannot answer. Inspect the retrieved set before generating a response. Record whether the essential passage is present and whether irrelevant material could confuse the answer step.

Generate supported answers with useful citations

Give the answer stage only the relevant authorized evidence and the current question. Ask it to distinguish supported facts from uncertainty. Resolve citations from trusted source metadata rather than accepting invented URLs. Place citations close to the claims they support and retain qualifiers such as recommended versus required.

An answer can be partially useful. If the curriculum is known but the next session date is not, explain the curriculum and identify the scheduling gap. Do not convert a topic listing into a confirmed event. Test that citations open for the intended reader and lead to the relevant section or document.

Handle updates, deletion and access changes

Track which chunks and cached answers depend on each source version. An update process should replace changed material, remove deleted material and reflect access changes. Use a controlled version switch or completion marker so readers do not receive a half-updated mixture of old and new content.

Run an update drill: change one prerequisite, delete one document and restrict one document. Ask questions affected by each change and inspect the retrieved source IDs. A successful ingestion job is not sufficient evidence; the answer behaviour must reflect the updated source state.

Decide whether the pilot is ready

• Source owners and authoritative versions are recorded.
• Parsed tables and exceptions have been inspected.
• Retrieval and answer quality are evaluated separately.
• Unanswerable and unauthorized questions have defined outcomes.
• Citations support the exact claim and respect access.
• Update and deletion drills affect derived data correctly.

Begin with a small reviewed audience. Collect failures by stage: source quality, parsing, retrieval, generation or presentation. Improve the stage supported by evidence instead of repeatedly changing the prompt. Expand the collection only when the ownership and evaluation process can grow with it.