Recap and final exam
Ten modules to go from a raw language model to a document assistant that cites its sources and refuses to invent them. Here is the course condensed, then the threads running through it, then the diagnostic tree, then the exam.
The course at a glance
| Module | What to retain |
|---|---|
| 1. Why ground in documents | RAG fixes frozen knowledge, hallucinations and confidentiality; fine-tuning teaches style, RAG teaches facts |
| 2. Extraction | PDFs encode glyphs not structure; scans need OCR; keep source, page, section and access class as metadata |
| 3. Chunking | 200–400 tokens with 10–20 % overlap, tables as atomic blocks, prepend heading trail; hierarchical wins on long documents |
| 4. Embeddings and index | Multilingual open model, unit-norm vectors, HNSW in Chroma or pgvector, filter by metadata every time |
| 5. Hybrid search | Dense loses on acronyms and identifiers; BM25 catches them; reciprocal-rank fusion merges the two by rank |
| 6. Reranking | Bi-encoders scan, cross-encoders rescore a shortlist; MMR and per-source cap prevent five near-identical passages |
| 7. Prompt and citations | Abstention rule near the top, [S<n>] labels as a contract, strongest passages at the ends of the context |
| 8. Evaluation | Recall × faithfulness table diagnoses most problems; LLM-as-judge with a different family, temperature 0 |
| 9. Cost and cache | LLM is over 90 % of the bill; cache embeddings by hash and answers by (question, user context) |
| 10. Project | Permissions at retrieval, ingestion on a schedule, weekly log review, ten-line deployment checklist |
The threads running through the course
Metadata beats cleverness. Half the failures a RAG pipeline exhibits — the wrong department's procedure, the retired version, a passage in the wrong language — are fixed not by a better model but by a where= clause. The extraction step of module 2 is where those clauses become possible, and skipping metadata there costs three months later.
Retrieval is upstream, and failures propagate. No reranker, no prompt, no evaluation trick can recover a passage that was never retrieved. Recall at is the metric to watch first, and to fix first when it drops. Modules 3, 4 and 5 all serve this one number.
The abstention answer is a feature. A model that always answers is a fabrication generator, and the abstention string of module 7 is the single largest quality lever in the whole course. Every downstream metric — faithfulness, trust, the user's willingness to keep using the tool — depends on it being enforced.
Cost lives in tokens, latency lives in the LLM. Modules 8 and 9 together turn a working prototype into a maintainable service. Break the cost into terms, cache the deterministic parts, log the hashes, review the numbers weekly.
The diagnostic tree: the answer is wrong, where do I look?
Answer is wrong.
├── The gold source is NOT in top-10 retrieval → RECALL problem
│ ├── Chunking splits or dilutes the answer → module 3 (tables, overlap, headings)
│ ├── Embedding does not represent the terms → module 4 (model, prefix, dimension)
│ └── Query rewrite dropped a key term → module 5 (rewriter, HyDE)
│
├── The gold source IS in top-10, NOT in top-5 → RERANK problem → module 6
│ ├── Cross-encoder truncates the passage → raise max_length
│ ├── MMR removed the right chunk → lower lambda_ toward 1.0
│ └── Per-source cap dropped a duplicate → check the tied hits
│
├── The gold source IS in the prompt → FAITHFULNESS problem → module 7
│ ├── Model ignores the abstention rule → give it the exact string to output
│ ├── Model blends contradictory passages → add the recency + status rule
│ ├── Answer sits at the middle of context → reorder with bracketing
│ └── Temperature too high → set 0.0 to 0.2
│
└── User asked something the corpus does not cover → ABSTENTION problem
├── Assistant answered anyway → module 7 (abstention rule)
└── Assistant abstained, user surprised → module 10 (interface, human path)
Print this. Read it every time an answer comes back wrong. The diagnostic never depends on the model of the moment; it depends on the pipeline stages, which are stable.
The final exam
The exam has 40 questions covering all ten modules: the distinction between RAG, fine-tuning and long context; extracting from PDF, HTML and office documents while keeping metadata; chunking strategies and the table-splitting failure; choosing an embedding model and building an HNSW index; combining dense and BM25 with reciprocal-rank fusion; using a cross-encoder to rerank a shortlist; assembling a prompt with citations and the abstention rule; evaluating recall, faithfulness and coverage; caching embeddings and answers, and breaking down cost per question; wiring the whole thing into a service with permissions.
Several questions present situations to diagnose: an accident-reporting question whose top hits come from the wrong procedure, a chunker that splits a role-and-responsibility table across two chunks, a retriever that returns a confidential passage to a public user, an assistant that answers a question the corpus does not contain, an invoice that doubled overnight without any suspicious commit, and a set of answers that quote non-existent citation indices. Judgement is what gets assessed, not memorised library signatures.
On success, your certificate of completion is issued immediately; its number can be verified by any third party on the platform.
Take the table above and, for each row, ask yourself "what symptom would I see if I got this wrong?". If you can explain why an acronym is unfindable by dense search, why the abstention rule must be near the top of the prompt, why permissions are enforced at retrieval and not at rendering, and why the LLM cost dominates the bill by an order of magnitude, you are ready. Good luck!
Final exam
Ready to validate this course?
40 questions drawn at random from the course bank · passing score 70% · verifiable PDF certificate issued immediately on success.
Start the examYou need to be signed in to your InSkillML account with an active subscription. You can also start the exam from My courses.