AI & ML Quick Path

Capstone 4 of 8

Company knowledge assistant (RAG)

AI applicationIntermediateEffort 20–35 h (estimate)

Business requirement

Employees ask questions about HR and IT policies; the assistant answers only from approved documents, cites the passage and says "I don’t know" when the documents do not cover it.

Lessons to finish first

Architecture

  1. Policy documents (written by you)
  2. Chunking with overlap and metadata
  3. Embeddings and a vector index
  4. Hybrid search (keywords + meaning)
  5. Prompt with retrieved chunks and citation rules
  6. LLM answer with citations
  7. Evaluation set and regression run

Build steps

  1. Write 8 to 12 short policy documents (leave, expenses, laptops, passwords, travel).
  2. Chunk them (for example 200 words with 40 words of overlap) and keep title and section as metadata.
  3. Embed the chunks and store them in a vector index (in memory first, then a database).
  4. Add keyword search and combine both rankings (reciprocal rank fusion).
  5. Write the prompt: answer only from the sources, cite them, say "I don’t know" otherwise.
  6. Create 30 test questions with expected sources, including 5 that the documents cannot answer.
  7. Measure retrieval hit rate, citation correctness and correct refusals; fix the worst failure and re-measure.

Python route and .NET route

Python
An embedding model (local through Ollama or a hosted API), NumPy or a vector database, LangChain optional. Start from the leave-policy RAG example.
.NET (C#)
Microsoft.Extensions.AI (IEmbeddingGenerator, IChatClient), Microsoft.Extensions.VectorData with an in-memory or PostgreSQL pgvector store, ASP.NET Core endpoint.

Example input and output

Input
"How many days of annual leave do new staff get?"
Output
"New staff get 25 days a year [leave-policy §2]."

Data

Documents you write yourself. Do not index real company documents without permission.

Starter file structure

Structure (Python file | .NET project)
rag/
  docs/*.md
  ingest.py | Ingest/Program.cs
  ask.py | Api/Program.cs
  eval/questions.jsonl
  eval/run_eval.py
  README.md

Tests

  • Retrieval: expected chunk in the top 5 for at least 90% of answerable questions
  • All 5 unanswerable questions get "I don’t know"
  • Every answer contains a valid citation

How it will be judged

  • Chunking choice justified
  • Evaluation set exists before tuning
  • Refusal behaviour tested
  • Prompt-injection test document included

Security and privacy

Treat document text as data, not instructions; filter results by the user’s permissions before they reach the model; keep API keys in environment variables or user secrets.

Definition of done

  • Evaluation report with the three metrics
  • One fixed failure documented before and after
  • No answer without a citation

Stretch goals

  • Permission filtering by department
  • Re-ranking with a cross-encoder
  • Streaming answers in a web UI

What to show an employer

Evaluation table plus a recorded demo of a cited answer and a correct refusal.

Completion

Saved in this browser only.