AI & ML Quick Path

Part 6: Building AI applications

RAG: answering from your own documents

Learning goal. Build a retrieval-augmented generation (RAG) system that checks permissions, retrieves evidence, answers with citations, says "I don't know" when evidence is missing, and is measured with an evaluation set.

In plain words

Retrieval-augmented generation (RAG) answers a question by first finding relevant passages in your own documents and then asking the model to answer using only those passages, with citations. The model's weights do not change; the knowledge lives in documents you can update, delete and protect. That makes RAG the standard choice for company policies, manuals, contracts and knowledge bases. RAG has two halves. Ingestion: split documents into chunks, embed them, and store them with metadata (source, version, who may read it). Answering: check who is asking, retrieve only chunks that person may read, put them into the prompt, generate an answer that cites chunk ids, and refuse politely when the best evidence is weak. You then evaluate with three kinds of questions: answerable, unanswerable, and forbidden for this user.

Diagram The two halves of RAG

Ingestion

  1. InputDocumentsPolicies, manuals, contracts
  2. ProcessChunksPassages with ids
  3. ModelEmbedOne vector per chunk
  4. Data storeIndexVectors with text and metadata: source, version, who may read

Answering

  1. InputQuestion and userWho is asking
  2. RiskAllowed chunks onlyFilter by the user's roles
  3. ProcessRetrieveTop few by similarity
  4. ProcessPromptChunks plus the question
  5. OutputAnswer with citationsChunk ids, or "I don't know"

The model's weights never change. The knowledge lives in documents you can update, delete and protect.

Diagram Three kinds of evaluation question
InputEvaluation setRun on every release
  • OutputAnswerableKnown source; the answer should cite it
  • RiskUnanswerableThe documents are silent; the answer should be "I don't know"
  • ProblemForbiddenThis user may not read the source; nothing should leak

Real-world example

Employees ask "How many leave days do I get?". HR policies are long, change yearly and include confidential sections (salary bands). A RAG assistant answers from the current policy with a link to the paragraph, never shows HR-only documents to other staff, and sends people to HR when the policy is silent. The Morgan Stanley case describes the same pattern for financial advisors.

How it works

  1. Chunk: split policies into passages with ids such as leave-policy#1.
  2. Index: embed chunks; store vectors with allowed_roles.
  3. Check access first: filter chunks by the user's roles before ranking.
  4. Retrieve: rank allowed chunks by similarity; keep the top few.
  5. Decide: below a similarity threshold, answer "I can't find this; please contact HR".
  6. Generate: prompt the model to answer only from the chunks and cite their ids.
  7. Evaluate: answerable, unanswerable and forbidden questions, every release.
Diagram The answering path in order
  1. RiskCheck access firstKeep only chunks the user's roles allow
  2. ProcessRetrieveRank allowed chunks; keep the top few
  3. RiskDecideBelow the similarity threshold: "I can't find this; please contact HR"
  4. ModelGenerateAnswer only from the chunks and cite their ids
  5. OutputEvaluateAnswerable, unanswerable and forbidden questions

Access is checked before ranking, so restricted text never reaches the model.

Citations

A citation points from a claim in the answer to the passage that supports it. Each chunk goes into the prompt with its id, and the model is told to cite those ids. The page can then show the source title, section or page, so the reader can check the claim. Citations make answers checkable, but a model can cite the wrong id, so verify that each cited chunk was really retrieved and supports the claim.

Diagram From chunk id to citation
  1. Data storeChunk with an idFor example leave-policy#2
  2. ProcessPrompt lists the chunkId shown before the text
  3. ModelAnswer cites the idIn square brackets
  4. OutputShow the sourceTitle and page from the metadata

Hallucination checks

A hallucination is a statement the model makes that the sources do not support. A hallucination check compares each claim in the answer with the retrieved text. If a claim has no support, remove it or refuse to answer. This is best done with the evaluation set; see Evaluating AI systems.

Diagram Checking an answer against its sources
InputDraft answerSplit into separate claims
  • OutputSupportedThe retrieved text backs the claim: keep it
  • RiskNot supportedNo evidence: remove it
  • ProblemNo support at allRefuse: "I don't know"
Diagram Where to enforce permissions

Ask the model not to reveal

  • Restricted text is already in the prompt
  • It can leak through answer or injection

Filter before ranking

  • Restricted chunks are never retrieved
  • The model never sees them

Python example

This runs offline: TF-IDF vectors stand in for embeddings and the "answer" is the best passage, so you can see retrieval behaviour without an API key.

Python — access check before ranking (excerpt)
allowed = [(c, float(s)) for c, s in zip(self.chunks, scores) if role in c.allowed_roles]
allowed.sort(key=lambda pair: pair[1], reverse=True)
Output — Evaluation output (executed)
[FAIL] (employee) How many leave days do I get? -> leave-policy#2 (score 0.34)
[PASS] (employee) Can I carry over unused holiday days? -> leave-policy#2 (score 0.43)
[FAIL] (employee) How far in advance must I request leave? -> None (score 0.20)
[PASS] (employee) How long is paid parental leave? -> parental-leave#1 (score 0.67)
[PASS] (employee) What is the salary band for level 5 engineers? -> None (score 0.00)
[PASS] (hr) What is the salary band for level 5 engineers? -> hr-salary-bands#1 (score 0.52)
[PASS] (employee) What is the office wifi password? -> None (score 0.00)
Passed 5/7

The two failures are real and useful: keyword matching preferred the carry-over chunk (it says "leave days" twice) over the chunk that answers the question, and one question fell just under the threshold. The permission and "don't know" cases pass: an employee never sees the salary chunk. A real embedding model, hybrid search and a re-ranker (next lesson) are the fixes, and the evaluation set is how you prove they worked. The prompt the script builds for the model:

Output — Prompt
Answer using ONLY the sources below. Cite source ids in square brackets.
If the sources do not contain the answer, say you don't know.
Sources:
[leave-policy#2] Carry over: up to 5 unused annual leave days can be carried ...
[leave-policy#3] Requesting leave: submit requests in the HR portal ...
Question: How many leave days do I get?

.NET (C#) example

A compact end-to-end RAG service with Microsoft.Extensions.AI (any IChatClient and IEmbeddingGenerator, hosted or local):

C# — RagService.cs
using System.Numerics.Tensors;
using Microsoft.Extensions.AI;

public record Chunk(string Id, string Text, string[] AllowedRoles, ReadOnlyMemory<float> Vector);

public class RagService(IChatClient chat, IEmbeddingGenerator<string, Embedding<float>> embedder, List<Chunk> index)
{
    private const float MinScore = 0.35f;     // tune on your evaluation set

    public static async Task<List<Chunk>> IngestAsync(IEmbeddingGenerator<string, Embedding<float>> embedder,
        IEnumerable<(string Id, string Text, string[] Roles)> passages)
    {
        var list = passages.ToList();
        var vectors = await embedder.GenerateAsync(list.Select(p => p.Text));
        return list.Zip(vectors, (p, v) => new Chunk(p.Id, p.Text, p.Roles, v.Vector)).ToList();
    }

    public async Task<string> AskAsync(string question, string userRole)
    {
        var q = await embedder.GenerateVectorAsync(question);
        var hits = index.Where(c => c.AllowedRoles.Contains(userRole))                 // permissions BEFORE ranking
                        .Select(c => (c, score: TensorPrimitives.CosineSimilarity(c.Vector.Span, q.Span)))
                        .OrderByDescending(h => h.score).Take(3).ToList();
        if (hits.Count == 0 || hits[0].score < MinScore)
            return "I can't find this in the documents you can access. Please contact HR.";

        string sources = string.Join("\n", hits.Select(h => $"[{h.c.Id}] {h.c.Text}"));
        var response = await chat.GetResponseAsync(
        [
            new(ChatRole.System, "Answer using ONLY the sources. Cite source ids in square brackets. If the sources do not contain the answer, say you don't know."),
            new(ChatRole.User, $"Sources:\n{sources}\n\nQuestion: {question}"),
        ], new ChatOptions { Temperature = 0 });
        return response.Text;
    }
}

Applied Maths - Statistics

Short, optional reading. Open a topic to see the formula, what it means and two worked examples. The numbers are small, made-up values chosen so you can check them by hand.

Context budget (how many chunks fit in the prompt)

Where it is used: choosing k, the number of retrieved chunks passed to the model with the question.

Prompt size

prompt tokens = k × tokens per chunk + question tokens + instruction tokens
prompt tokens must be at most the context window

The 4,000-token window below is a made-up size.

In plain words: each extra chunk adds context but uses up room.

Example 1: it fits. k = 4, 300 tokens per chunk, question 50, instructions 150. Total = 1,200 + 50 + 150 = 1,400. Room left = 4,000 - 1,400 = 2,600.

Example 2: too many chunks. k = 10, 500 tokens per chunk: 5,000 + 200 = 5,200, which is 1,200 over. Try k = 7: 3,500 + 200 = 3,700, which fits.

Hit rate with a margin of error (does retrieval find the answer?)

Where it is used: testing whether the right chunk appears in the top k for a set of known questions.

Hit rate and interval

hit rate = questions with the right chunk in top k ÷ total questions
margin ≈ 1.96 × sqrt( hit rate × (1 - hit rate) ÷ n )

n is the number of test questions. Keep the interval between 0 and 1.

In plain words: with few test questions the score is uncertain, so show the margin.

Example 1: 20 questions. 18 hits, so hit rate = 0.90. Margin = 1.96 × sqrt(0.09 ÷ 20) = 1.96 × 0.0671 = 0.13. Range 0.77 to 1.00 (the top is capped at 1).

Example 2: 50 questions. 45 hits, hit rate = 0.90 again. Margin = 1.96 × sqrt(0.09 ÷ 50) = 1.96 × 0.0424 = 0.083. Range 0.82 to 0.98, tighter.

Maths behind it

  • Embeddings and cosine similarity (linear algebra)
  • Nearest-neighbour search
  • Information retrieval metrics: recall@k, precision@k
  • Thresholds and calibration (statistics)

Common mistakes

  • Telling the model "don't reveal restricted text" instead of never retrieving it.
  • Answering even when the best match is weak, which produces confident wrong answers.
  • Fine-tuning to "teach" policies that change every year.
  • No evaluation set, so nobody knows whether a change helped.
  • Showing citations without checking that the cited chunk supports the claim.
  • Skipping a hallucination check, so unsupported claims reach the user.
Diagram What the system should do with weak or restricted evidence
Evidence is strongEvidence is weakUser may read it
Answer with citationsStrong evidence the user may read
Say "I don't know"Weak evidence: route the user to a person
User may not read it
Never retrievedRestricted chunks are filtered out before ranking
Still never retrievedThe permission filter runs before any score is used

Quick check

  1. Why filter by permissions before ranking instead of instructing the model?

    Show answer

    Anything in the prompt can leak through the answer or prompt injection; restricted text must never reach the model.

  2. What should happen when the best similarity is below the threshold?

    Show answer

    Return a clear "I don't know" with a route to a person, instead of generating an unsupported answer.

  3. Which three kinds of questions belong in a RAG evaluation set?

    Show answer

    Answerable questions with known sources, unanswerable questions, and questions the user is not allowed to have answered.

  4. What should you do with an answer claim that the retrieved text does not support?

    Show answer

    Remove it, or refuse to answer if nothing is supported.

Completion

Evidence of completion. A working RAG assistant in Python or C# that passes an evaluation set of answerable, unanswerable and forbidden questions, with failures explained.

Saved in this browser only. See all progress.