AI & ML Quick Path

Part 6: Building AI applications

Advanced RAG: chunking, hybrid search, re-ranking and evaluation

Learning goal. Improve RAG quality with better chunking, hybrid keyword-plus-vector search, reciprocal rank fusion, re-ranking, query rewriting and metadata filters, and measure each change.

In plain words

The model cannot cite a passage it never receives, so retrieval quality decides RAG quality. Five techniques fix most problems. Chunking: split by meaning (paragraphs or headings) rather than blindly every N words, add a little overlap, and put the document title in each chunk. Hybrid search: run keyword search (BM25 finds exact codes, names and rare words) and vector search (finds meaning) together and merge the lists, commonly with reciprocal rank fusion (RRF), which rewards items ranked high in either list. Re-ranking: a slower, more accurate model re-scores the top 20–50 candidates. Query rewriting: an LLM turns a vague or conversational question into a clear search query, or into several sub-questions. Metadata filters: restrict by permission, product, region or date before ranking. Some teams also use knowledge graphs (GraphRAG) for questions about relationships. Measure every change with retrieval metrics (is the right passage in the top k?) and answer metrics (is the answer faithful to the sources?).

Diagram Five techniques that improve retrieval
InputRetrieval qualityDecides RAG quality
  • ProcessChunkingSplit by meaning, small overlap, title in each chunk
  • ProcessHybrid searchKeyword plus vector, merged
  • ModelRe-rankingA slower, more accurate model re-scores candidates
  • ModelQuery rewritingAn LLM makes the question a clear search query
  • RiskMetadata filtersPermission, product, region or date before ranking

The model cannot cite a passage it never receives.

Diagram Fixed-size and meaning-based chunks

Fixed 25-word chunks

  • 4 chunks
  • A cut can fall mid-rule, as in "Carry over. Up"

Paragraph chunks

  • 4 chunks, each starting with its heading
  • Each rule stays whole

Overlap

  • 6 chunks 25-word chunks with 10-word overlap

Output from the executed chunking demo on an 80-word policy.

Diagram Hybrid search
  • ProcessKeyword searchBM25 ranking
  • ModelVector searchRanking by meaning
OutputFused rankingOne merged list

Keyword search finds exact codes, names and rare words. Vector search finds meaning.

Diagram Reciprocal rank fusion in words
  1. InputTwo ranked listsOne from keyword search, one from vector search
  2. ProcessGive each position a scoreHigher rank gives a bigger score
  3. ProcessAdd the scores per documentA document high in either list rises
  4. OutputSortHighest total first

Merge by rank, not by raw score, because the two scores are on different scales.

Real-world example

A manufacturer's maintenance assistant must answer "error E-4021 on pump PX-7". Vector search returned general pump articles; keyword search found the exact error code. Hybrid search plus a re-ranker put the right procedure first, and the team proved it with a 60-question retrieval test.

How it works

  1. Chunk by structure; add titles; test chunk size on your evaluation set.
  2. Store text, vectors, keyword index and metadata.
  3. At query time: optionally rewrite the question; apply metadata filters.
  4. Run keyword and vector search (top 20 each); merge with RRF.
  5. Re-rank to the top 5; send those to the model.
  6. Track retrieval hit rate and answer faithfulness for every change.
Diagram Query time pipeline

Retrieve

  1. InputQuestionMay be rewritten first
  2. RiskMetadata filtersPermission, product, region, date
  3. ProcessKeyword and vector searchTop 20 each
  4. ProcessMerge with RRFBy rank

Answer and measure

  1. ModelRe-rankDown to the top 5
  2. ModelLLM answerUses only those 5
  3. OutputRetrieval hit rateIs the right passage in the top k?
  4. OutputAnswer faithfulnessDoes the answer match the sources?

Metadata filters apply before ranking. Retrieval metrics and answer metrics are measured separately.

Diagram Candidates shrink at each step
  1. ProcessKeyword searchTop 20
  2. ProcessVector searchTop 20
  3. ProcessMerge with RRFOne list
  4. ModelRe-rankerTop 5
  5. OutputLLMAnswers from 5 passages

Counts are the example values from the steps above.

  • Semantic search: search by meaning using embeddings rather than exact words; the retrieval step of RAG.
  • HyDE (hypothetical document embeddings): ask the model to write a fake answer first, then search with the embedding of that answer. It can help when questions are short and documents are long.
Diagram HyDE
  1. InputShort questionHard to match to long documents
  2. ModelModel writes a fake answerA hypothetical document
  3. ProcessEmbed the fake answerUse that vector to search
  4. OutputReal passagesFound by similarity
  • Agentic RAG: an agent decides when to search, what to search for, and whether to search again after reading results. More flexible for complex questions, slower and harder to test than a fixed pipeline.
Diagram Agentic RAG
  1. ModelDecide what to search forThe agent chooses
  2. ProcessSearchRetrieve passages
  3. ModelRead the resultsAre they enough?

Search again until the evidence is enough

More flexible than a fixed pipeline, but slower and harder to test.

Chunking in more detail

A chunk is one passage that is embedded and retrieved as a unit. Too small and it loses the context that makes it meaningful. Too large and it mixes several topics, so its vector becomes vague and it fills the context window. Split along the document's own structure where you can, and test sizes on your evaluation set. Store the document title, section heading and page number with each chunk, so an answer can cite where it came from.

Diagram What one chunk carries
Data storeDocumentTitle, version, source
ProcessSectionHeading, page number
OutputChunk textOne passage, with a little overlap

The same pieces are stored with every chunk.

Re-ranking

Search over the whole collection must be fast, so it compares the question and each passage separately. A re-ranker is a model that reads the question and one candidate passage together. It is slower but usually more accurate at judging relevance. So you run it only on the short list from search, then keep the best few.

Diagram First search and re-ranker

First search

  • Looks at the whole collection
  • Fast, so it can be a little rough

Re-ranker

  • Looks only at the short list
  • Reads the question and passage together

Query rewriting

Users type follow-ups such as "and for contractors?", which mean nothing on their own. Query rewriting asks a model to turn the question, with the conversation so far, into a stand-alone search query. You can also expand the query with alternative wordings, or split a compound question into sub-questions and search for each. HyDE, described above, is another rewriting trick. Rewrites can drift, so keep codes and names exactly and measure the effect.

Diagram One question becomes several searches
InputFollow-up question"And for contractors?"
  • ModelStand-alone queryAdds the missing topic from the conversation
  • ModelAlternative wordingAnother way to say the same thing
  • ModelSub-questionOne part of a compound question

Results from all searches are merged, for example with RRF.

Context packing

Context packing is choosing and arranging the retrieved chunks that go into the prompt. Remove duplicates and near-copies. Keep only chunks that fit the context window (the amount of text the model can read at once), and leave room for the answer. Models can pay less attention to the middle of a long prompt, so put the strongest evidence at the start and the end. Label each chunk with its source id so the answer can cite it.

Diagram Packing the prompt
  1. ProcessInstructionsAnswer only from the sources; cite ids
  2. OutputBest evidenceStrongest chunks first, each with a source id
  3. Data storeWeaker evidenceFewer, shorter chunks in the middle
  4. OutputNext-best evidenceAnother strong chunk near the end
  5. InputQuestion and room for the answerLeave space for the reply

Total size must fit the context window.

Parsing PDFs

Ingestion starts with getting text out of files, and bad parsing silently ruins retrieval. A text PDF contains real text that a parser can read. A scanned PDF is only pictures of pages, so it needs OCR (optical character recognition, software that reads text from images), which can make mistakes. Tables, headers, footers and multi-column layouts can scramble the reading order or repeat page furniture in every chunk. Keep page numbers and headings as metadata, and read a sample of parsed output by eye before indexing.

Diagram Which PDFs need more care
Text PDFScanned PDFSimple layout
Usually fineRead a sample to check
Needs OCRCheck OCR errors on a sample
Complex layout
Check table and column orderTables and columns can scramble the text
Hardest caseOCR plus layout problems; inspect closely

Re-indexing

The index goes stale. Documents change, you may choose a new embedding model, or you may change the chunking rules. Re-indexing means refreshing or rebuilding the index. For changed documents, update only those (an incremental update) and delete the chunks of documents that were removed, or they will still be retrieved. A new embedding model or new chunking rules need a full rebuild, because old and new vectors cannot be mixed. Keep versions of the index, and compare old and new on the evaluation set before switching.

Diagram Re-indexing cycle
  1. InputDetect a changeEdited or removed documents, new model, new chunking rules
  2. ProcessRebuild or updateIncremental for documents; full for a new model
  3. ModelEvaluate old and newSame evaluation set for both
  4. OutputSwitchOnly if the new index is no worse

Repeat whenever something changes

Keep the old index until the new one has passed.

Retrieval evals

A retrieval eval measures only whether the right chunk was found, not whether the final answer reads well. For each question in your evaluation set, record which chunk or document should be retrieved. Then check whether it appears in the top k results. If retrieval fails, no prompt can fix the answer. If retrieval succeeds but the answer is wrong, look at packing or the prompt.

Diagram Two separate measurements

Retrieval

  1. InputEvaluation questionWith the known right chunk
  2. ProcessSearch resultsTop k chunks
  3. OutputHit or missWas the right chunk in the top k?

Answer

  1. ProcessRetrieved chunksSent to the model
  2. ModelGenerated answerWritten from those chunks
  3. OutputAnswer checkFaithful and correct?

GraphRAG

GraphRAG builds a graph from your documents: the things they mention (entities such as people, products or places) as nodes, and the relationships between them as links. Retrieval can then follow connections, or use summaries of groups of related items, to answer questions about how things connect or what the whole collection is about. Building the graph takes extra effort and processing. It is only worth it when relationships matter. For plain look-up questions, ordinary RAG is simpler.

Diagram GraphRAG
  1. InputDocumentsThe source text
  2. ModelExtract entities and relationshipsA model reads the text and lists them
  3. Data storeBuild a graphEntities as nodes, relationships as links
  4. ProcessFollow connectionsGather related facts for the question
  5. OutputAnswerWritten from the connected evidence

More effort to build than ordinary RAG.

Python example

Three ways to chunk the same 80-word policy:

Output — Output of quickpath.py chunking (executed)
80 words | fixed 25-word chunks: 4 | example cut: ...'roportion to their hours. Carry over. Up'
paragraph chunks: 4 | each starts with a heading: ['Annual leave', 'Carry over', 'Requesting leave', 'Sick leave']
25-word chunks with 10-word overlap: 6

Fixed-size chunks cut a section in half ("Carry over. Up" ends one chunk). Paragraph chunks keep each rule whole with its heading.

Merging keyword and vector rankings with RRF (score = sum of 1 ÷ (60 + rank) over both lists):

Output — Output of quickpath.py hybrid (executed)
query 'E-4021': keyword ['kb1', 'kb2', 'kb3', 'kb4'] | vector ['kb1', 'kb3', 'kb2', 'kb4'] | fused (RRF) ['kb1', 'kb2', 'kb3', 'kb4']
query "pump won't power up": keyword ['kb4', 'kb2', 'kb1', 'kb3'] | vector ['kb4', 'kb2', 'kb1', 'kb3'] | fused (RRF) ['kb4', 'kb2', 'kb1', 'kb3']

To run offline, this demo uses character patterns as a stand-in for an embedding model, so the two lists are similar; with a real embedding model, paraphrased questions ("machine running too hot") produce clearly different vector results, which is where hybrid search earns its place. The merge code is the same either way:

Python — reciprocal rank fusion
rrf = collections.Counter()
for ranking in (keyword_ranking, vector_ranking):
    for position, doc_id in enumerate(ranking, 1):
        rrf[doc_id] += 1 / (60 + position)
fused = [d for d, _ in rrf.most_common()]

.NET (C#) example

RRF and an LLM query rewriter in C#:

C# — HybridSearch.cs
using Microsoft.Extensions.AI;

public static class HybridSearch
{
    public static List<string> ReciprocalRankFusion(params IReadOnlyList<string>[] rankings)
    {
        var scores = new Dictionary<string, double>();
        foreach (var ranking in rankings)
            for (int i = 0; i < ranking.Count; i++)
                scores[ranking[i]] = scores.GetValueOrDefault(ranking[i]) + 1.0 / (60 + i + 1);
        return scores.OrderByDescending(kv => kv.Value).Select(kv => kv.Key).ToList();
    }

    public static async Task<string> RewriteQueryAsync(IChatClient chat, string conversation, string question)
    {
        var response = await chat.GetResponseAsync(
        [
            new(ChatRole.System, "Rewrite the user's last question as one short, self-contained search query. Keep codes and names exactly. Reply with the query only."),
            new(ChatRole.User, $"Conversation:\n{conversation}\n\nLast question: {question}"),
        ], new ChatOptions { Temperature = 0 });
        return response.Text.Trim();
    }
}
// var fused = HybridSearch.ReciprocalRankFusion(keywordIds, vectorIds).Take(20).ToList();  then re-rank the top 20.

For keyword search in .NET use your database's full-text search (PostgreSQL, SQL Server, Azure AI Search or Elasticsearch).

Applied Maths - Statistics

Short, optional reading. Open a topic to see the formula, what it means and two worked examples. The numbers are small, made-up values chosen so you can check them by hand.

Recall@k and MRR (did the right chunk come back, and how high)

Where it is used: evaluating a retriever or re-ranker on questions whose correct chunk is known.

Recall@k and MRR

recall@k = questions with the right chunk in the top k ÷ total questions
MRR = average of (1 ÷ rank of the first right chunk)

Rank 1 is the top result. MRR means mean reciprocal rank. If the right chunk is missing, its score is 0.

In plain words: recall@k asks "was it in the top k?". MRR rewards putting it nearer the top.

Example 1: recall@3. Three questions. The right chunk is at rank 1, rank 3 and rank 8. Top 3 holds two of three, so recall@3 = 2 ÷ 3 = 0.67. Recall@10 = 3 ÷ 3 = 1.00.

Example 2: MRR. Same ranks: 1 ÷ 1 = 1, 1 ÷ 3 = 0.333, 1 ÷ 8 = 0.125. MRR = (1 + 0.333 + 0.125) ÷ 3 = 1.458 ÷ 3 = 0.486.

Reciprocal rank fusion (mixing keyword and vector results)

Where it is used: hybrid search, to merge two ranked lists into one.

Reciprocal rank fusion

score of a chunk = sum over lists of 1 ÷ (60 + rank in that list)

60 is a common smoothing constant. A chunk missing from a list adds nothing for it.

In plain words: chunks that rank well in both lists get the highest combined score.

Example 1: in both lists. Chunk A is rank 1 in keyword and rank 3 in vector: 1 ÷ 61 + 1 ÷ 63 = 0.01639 + 0.01587 = 0.03227. Chunk B is rank 2 in keyword and rank 1 in vector: 1 ÷ 62 + 1 ÷ 61 = 0.01613 + 0.01639 = 0.03252. B wins.

Example 2: in one list only. Chunk C is rank 2 in vector only: 1 ÷ 62 = 0.01613. That is below both A and B.

Chunk overlap (how many chunks you get)

Where it is used: the chunking step, where neighbouring chunks share some tokens so ideas are not cut in half.

Number of chunks

step = chunk size - overlap
chunks = round up of (document tokens - overlap) ÷ step

Sizes are in tokens. This holds when the document is at least one chunk long.

In plain words: more overlap means more chunks and more stored text.

Example 1: with overlap. 1,000 tokens, chunk size 200, overlap 50. Step = 150. Chunks = round up of 950 ÷ 150 = 6.33, so 7 chunks. Stored tokens up to 7 × 200 = 1,400.

Example 2: no overlap. Same document, overlap 0. Step = 200. Chunks = 1,000 ÷ 200 = 5 exactly. Overlap cost 2 more chunks here.

Maths behind it

  • BM25 (probabilistic information retrieval)
  • Cosine similarity of embeddings
  • Reciprocal rank fusion
  • Cross-encoder re-ranking
  • Retrieval metrics: recall@k, mean reciprocal rank

Common mistakes

  • Merging raw keyword and vector scores, which are on different scales; merge by rank instead.
  • Changing chunk size, embedding model and prompt at once, so you cannot tell what helped.
  • Only measuring final answers; measure retrieval separately to find where failures start.
  • Trusting PDF parsing without looking: scrambled columns, repeated footers or bad OCR silently ruin retrieval.
  • Never re-indexing, so changed or deleted documents are still retrieved, or mixing old and new embedding vectors.
  • Stuffing in too many chunks: duplicates and weak evidence crowd out the best chunks and the room for the answer.
Diagram Changing and measuring

Risky

  • Chunks, model and prompt changed together
  • Only the final answer is measured
  • Raw keyword and vector scores merged

Better

  • One change at a time
  • Retrieval and answers measured separately
  • Lists merged by rank

Quick check

  1. Why combine keyword and vector search?

    Show answer

    Keyword search finds exact codes and rare terms; vector search finds meaning. Together they miss less.

  2. How does reciprocal rank fusion score a document?

    Show answer

    It adds 1 ÷ (60 + rank) from each list, so documents ranked high in any list rise to the top.

  3. What does a re-ranker do?

    Show answer

    It re-scores a short list of candidates with a slower, more accurate model before they go to the LLM.

  4. Why compare an old and a new index on the evaluation set before switching?

    Show answer

    A rebuild with a new model or new chunking rules can make some questions worse; the evaluation set shows this before users see it.

  5. Why measure retrieval separately from the final answer?

    Show answer

    If the right chunk is never found, no prompt can fix the answer, so you need to know which stage failed.

Completion

Evidence of completion. A before-and-after table of retrieval hit rate (correct passage in top 5) on at least 30 questions for vector-only versus hybrid retrieval with re-ranking.

Saved in this browser only. See all progress.