Capstone 4 of 8
Company knowledge assistant (RAG)
Business requirement
Employees ask questions about HR and IT policies; the assistant answers only from approved documents, cites the passage and says "I don’t know" when the documents do not cover it.
Lessons to finish first
Architecture
- Policy documents (written by you)
- Chunking with overlap and metadata
- Embeddings and a vector index
- Hybrid search (keywords + meaning)
- Prompt with retrieved chunks and citation rules
- LLM answer with citations
- Evaluation set and regression run
Build steps
- Write 8 to 12 short policy documents (leave, expenses, laptops, passwords, travel).
- Chunk them (for example 200 words with 40 words of overlap) and keep title and section as metadata.
- Embed the chunks and store them in a vector index (in memory first, then a database).
- Add keyword search and combine both rankings (reciprocal rank fusion).
- Write the prompt: answer only from the sources, cite them, say "I don’t know" otherwise.
- Create 30 test questions with expected sources, including 5 that the documents cannot answer.
- Measure retrieval hit rate, citation correctness and correct refusals; fix the worst failure and re-measure.
Python route and .NET route
- Python
- An embedding model (local through Ollama or a hosted API), NumPy or a vector database, LangChain optional. Start from the leave-policy RAG example.
- .NET (C#)
- Microsoft.Extensions.AI (IEmbeddingGenerator, IChatClient), Microsoft.Extensions.VectorData with an in-memory or PostgreSQL pgvector store, ASP.NET Core endpoint.
Example input and output
- Input
"How many days of annual leave do new staff get?"- Output
"New staff get 25 days a year [leave-policy §2]."
Data
Documents you write yourself. Do not index real company documents without permission.
Starter file structure
rag/
docs/*.md
ingest.py | Ingest/Program.cs
ask.py | Api/Program.cs
eval/questions.jsonl
eval/run_eval.py
README.mdTests
- Retrieval: expected chunk in the top 5 for at least 90% of answerable questions
- All 5 unanswerable questions get "I don’t know"
- Every answer contains a valid citation
How it will be judged
- Chunking choice justified
- Evaluation set exists before tuning
- Refusal behaviour tested
- Prompt-injection test document included
Security and privacy
Treat document text as data, not instructions; filter results by the user’s permissions before they reach the model; keep API keys in environment variables or user secrets.
Definition of done
- Evaluation report with the three metrics
- One fixed failure documented before and after
- No answer without a citation
Stretch goals
- Permission filtering by department
- Re-ranking with a cross-encoder
- Streaming answers in a web UI
What to show an employer
Evaluation table plus a recorded demo of a cited answer and a correct refusal.
Completion
Saved in this browser only.