Part 6: Building AI applications
Embeddings and vector search
Learning goal. Explain embeddings, find similar text by cosine similarity, and store and search vectors in a vector database from Python and C#.
In plain words
An embedding is a list of numbers (a vector) that represents the meaning of a piece of text, an image or a product. An embedding model is trained so that similar meanings get similar vectors: "vacation days" and "annual leave" end up close even though they share no words. Cosine similarity measures how close two vectors point (1 means same direction, 0 unrelated). Vector search embeds all your passages once, stores them, embeds each question, and returns the passages with the highest similarity. A vector database (pgvector in PostgreSQL, Azure AI Search, Qdrant, Elasticsearch and others) stores vectors with their text and metadata, and uses approximate nearest-neighbour (ANN) indexes such as HNSW to search millions of vectors quickly. Always embed questions and documents with the same model.
- InputTextFor example "vacation days"
- ModelEmbedding modelTrained so similar meanings get similar vectors
- OutputVectorA list of numbers that represents the meaning
Real vectors have hundreds or thousands of numbers.
Close
- "vacation days" and "annual leave policy" 0.984 although they share no words
- "vacation days" and "vacation days" 1.000
Far
- "vacation days" and "printer is jammed" 0.170
Toy 3-number vectors from the executed demo below. 1 means the same direction, 0 means unrelated.
- Data storeVectorThe numbers for each passage
- Data storeTextThe passage itself
- Data storeMetadataSource, permissions, date, used for filtering
- ProcessANN indexApproximate nearest-neighbour index such as HNSW for fast search over millions of vectors
Prepare once
- InputDocumentsYour passages
- ModelEmbed each passageWith the embedding model
- Data storeStoreVector, text and metadata
For each question
- InputQuestionThe user's words
- ModelEmbed the questionSame model
- ProcessSearchHighest similarity, filtered by metadata
- OutputTop passagesReturned with their text
Both lanes must use the same embedding model.
Real-world example
An IT help desk wants "my laptop won't connect to wifi" to find the article "Troubleshooting wireless network access". Keyword search misses it; vector search finds it because the meanings are close.
How it works
- Choose an embedding model (hosted API or a local model).
- Split documents into passages; embed each; store vector + text + metadata (source, permissions, date).
- Embed the user's question with the same model.
- Find the most similar passages (top k), filtered by metadata.
- Measure: for real questions, is the right passage in the top 3?
Keyword search
- Looks for the same words
- Misses the wireless article "my laptop won't connect to wifi" shares no words with it
Vector search
- Compares meaning
- Finds the article because the meanings are close
For the help-desk example above.
- ModelChoose a modelHosted API or local model
- ProcessEmbed passagesSplit documents first
- Data storeStoreVector, text and metadata
- ModelEmbed the questionSame model
- ProcessTop k, filteredMost similar passages the user may see
- OutputMeasureIs the right passage in the top 3?
Related concepts
Embedding models
An embedding model is a trained model that turns text into a vector. Different models make different vectors, so vectors from two models cannot be compared. To choose one, consider which languages your documents use, the length of text it accepts, how many numbers each vector has (which affects storage), whether it is hosted or runs locally, and cost. Then test two or three candidates on your own questions. Whatever you choose, embed documents and questions with that same model. If you change it later, you must re-embed everything.
- ProcessLanguagesDoes it handle the languages you have?
- ProcessText lengthHow much text can it take at once?
- RiskHosted or localData rules, cost and speed
- OutputYour own testIs the right passage found?
Test candidates on your own questions before deciding.
Vector databases
A vector database stores vectors together with their text and metadata, and is built to find the nearest vectors quickly. Instead of comparing the question with every vector, it uses an approximate nearest-neighbour (ANN) index, which gives very close answers much faster than checking everything. It also applies metadata filters and handles inserts, updates and deletes.
- InputQuery vectorThe embedded question
- RiskMetadata filterKeep only allowed rows
- ProcessANN indexJump to likely neighbours instead of scanning all
- OutputTop k rowsVector, text and metadata
pgvector
pgvector is an extension for the PostgreSQL database. It adds a vector column type and similarity search. Teams that already run PostgreSQL can keep vectors beside their other data, in the same tables, with the same backups, access control and queries. A dedicated vector database may still be chosen when vector search is the main workload, or when you need features that a general database lacks. Compare both on your own data and your own questions.
One system holds both kinds of data.
Metadata filters
Metadata is information about a passage rather than its text: source, date, language, permissions or product. A metadata filter limits the search to passages that match, for example only the current product, in the user's language, that the user may read. Filtering can happen before the vector search or during it. Permission filters must never be left to the model, so apply them in the search itself.
No filter
- Old versions and other products compete
- Restricted passages can be returned
With filter
- Only matching source, date, language and product
- Only passages this user may read
Python example
Toy 3-number vectors (real ones have hundreds or thousands of numbers):
cos('vacation days', 'annual leave policy') = 0.984
cos('vacation days', 'vacation days') = 1.000
cos('vacation days', 'printer is jammed') = 0.170With a real embedding model through an OpenAI-compatible API (local Ollama here):
import numpy as np
from openai import OpenAI
client = OpenAI(base_url="http://localhost:11434/v1", api_key="not-needed-locally")
def embed(texts): return np.array([d.embedding for d in client.embeddings.create(model="all-minilm", input=texts).data])
docs = ["Annual leave: 25 days per year", "Printer setup guide", "Wireless network troubleshooting"]
D = embed(docs); q = embed(["my laptop won't connect to wifi"])[0]
scores = D @ q / (np.linalg.norm(D, axis=1) * np.linalg.norm(q))
print(docs[int(scores.argmax())]).NET (C#) example
IEmbeddingGenerator from Microsoft.Extensions.AI plus fast cosine similarity from System.Numerics.Tensors:
using System.Numerics.Tensors;
using Microsoft.Extensions.AI;
using OllamaSharp;
IEmbeddingGenerator<string, Embedding<float>> embedder = new OllamaApiClient(new Uri("http://localhost:11434"), "all-minilm");
string[] docs = { "Annual leave: 25 days per year", "Printer setup guide", "Wireless network troubleshooting" };
var docVectors = await embedder.GenerateAsync(docs);
var question = await embedder.GenerateVectorAsync("my laptop won't connect to wifi");
var best = docs.Zip(docVectors, (text, e) => (text, score: TensorPrimitives.CosineSimilarity(e.Vector.Span, question.Span)))
.OrderByDescending(x => x.score).First();
Console.WriteLine($"{best.text} ({best.score:F3})");Method names have changed between package versions (older versions call it GenerateEmbeddingVectorAsync); check the version you install. For storage, use Microsoft.Extensions.VectorData connectors or PostgreSQL with pgvector through Npgsql.
Applied Maths - Statistics
Short, optional reading. Open a topic to see the formula, what it means and two worked examples. The numbers are small, made-up values chosen so you can check them by hand.
Cosine similarity (how close two meanings are)
Where it is used: vector search, to score how close a question embedding is to each stored chunk embedding.
Cosine similarity
cosine = (a1×b1 + a2×b2 + ... + an×bn) ÷ (length of a × length of b)
length of a = sqrt(a1^2 + a2^2 + ... + an^2)a and b are two embeddings (lists of numbers). The top part is the dot product.
In plain words: it measures the angle between two vectors. 1 means the same direction (similar meaning), 0 means unrelated, -1 means opposite.
Example 1: similar texts. Tiny embeddings a = (3, 4) and b = (4, 3). Dot product = 3×4 + 4×3 = 24. Lengths: sqrt(9 + 16) = 5 and sqrt(16 + 9) = 5. Cosine = 24 ÷ (5 × 5) = 24 ÷ 25 = 0.96, so very close.
Example 2: unrelated texts. a = (1, 0) and b = (0, 2). Dot product = 1×0 + 0×2 = 0. Cosine = 0 ÷ (1 × 2) = 0, so no similarity.
Index size (how much memory the vectors need)
Where it is used: planning how many chunks a vector store can hold before you need a bigger machine or an approximate index.
Memory for stored vectors
bytes = number of chunks × dimensions × bytes per numberA 32-bit float uses 4 bytes. Dimensions is the length of each embedding.
In plain words: multiply how many vectors you have by how long each one is, then by the size of one number.
Example 1: a small help centre. 100,000 chunks × 384 dimensions × 4 bytes = 153,600,000 bytes, about 153.6 MB.
Example 2: a large archive. 1,000,000 chunks × 1,536 dimensions × 4 bytes = 6,144,000,000 bytes, about 6.1 GB. That is 40 times more memory, so size choices matter.
Maths behind it
- Vectors and vector spaces (linear algebra)
- Dot product and cosine similarity
- Approximate nearest-neighbour search (HNSW graphs)
- Contrastive learning (how embedding models are trained)
Common mistakes
- Embedding documents with one model and questions with another.
- Relying on vectors alone for exact codes and names (combine with keyword search; see Advanced RAG).
- Storing vectors without permission metadata, so you cannot filter what each user may see.
- Changing the embedding model without re-embedding every stored passage.
- Choosing a model or a database by reputation instead of testing it on your own questions.
Different models
- Documents embedded with model A
- Questions embedded with model B
- Vectors do not line up, so results are poor
Same model
- Documents and questions both use model A
- Similar meanings get similar vectors
Quick check
Why can vector search find "annual leave" for a question about "vacation days"?
Show answer
Embeddings represent meaning, so different words with similar meaning get close vectors.
What does a vector database add over a simple list?
Show answer
Fast approximate search over millions of vectors, plus storage of text and metadata for filtering.
What must stay the same for documents and questions?
Show answer
The embedding model.
Why might a team choose pgvector?
Show answer
It adds vector search to PostgreSQL, so vectors can live beside the team's other data without a new system to run.
Completion
Evidence of completion. Twenty passages embedded with a real embedding model, five differently worded questions, and the rank of the correct passage for each.
Saved in this browser only. See all progress.