AI & ML Quick Path

Part 5: Large language models

Running language models on your own machine

Learning goal. Run an open-weight model locally, understand quantization and memory needs, and call the local model from Python and .NET through the same interfaces used for hosted models.

In plain words

You do not always need a cloud API. Open-weight models can run on your laptop, a workstation or your own servers, which keeps data in-house and can lower cost. Memory is the limit: model weights need parameters × bytes per weight. Quantization stores weights in fewer bits (8-bit, 4-bit) so a model fits in less memory, with a small loss in quality that you must measure. Popular tools: Ollama and llama.cpp run quantized models (GGUF files) on laptops and desktops, with or without a GPU; ONNX Runtime GenAI runs models in many languages; vLLM serves models to many users on GPUs. Many of these expose an OpenAI-compatible API, so the same code works for a local or hosted model by changing the address. A rough guide: a 7–8-billion-parameter model at 4 bits needs about 4–5 GB just for weights, plus memory for the conversation (the KV cache).

Diagram Local model stack
  1. InputYour Python or .NET clientPoints at the local address
  2. ProcessOpenAI-compatible APISame code for local or hosted; change the address
  3. ProcessRunnerOllama, llama.cpp, ONNX Runtime GenAI or vLLM
  4. ModelQuantized model fileFor example GGUF; weights in 8-bit or 4-bit
  5. Data storeHardwareLaptop, workstation or your own server

Your code talks to the runner, which loads the model file.

Diagram Hosted API versus local model

Hosted API

  • Data goes to the provider
  • Used through the provider's API

Local open-weight model

  • Data stays in-house
  • Can lower cost
  • Memory is the limit
Diagram Which runner for which job

Ollama and llama.cpp

  • Run quantized models (GGUF files)
  • Laptops and desktops, with or without a GPU

ONNX Runtime GenAI

  • Runs models in many languages

vLLM

  • Serves models to many users
  • Runs on GPUs
Diagram Quantization trade-off

16-bit weights

  • 16.0 GB
  • No quantization

8-bit weights

  • 8.0 GB

4-bit weights

  • 4.0 GB
  • Small quality loss to measure

Weights for an 8-billion-parameter model, from the executed memory run.

Diagram Memory a local model needs
  • ModelWeightsParameters times bytes per weight
  • Data storeKV cacheMemory for the conversation; grows with its length
OutputTotal memory neededMust fit your machine

A 7-8 billion parameter model at 4 bits needs about 4-5 GB just for weights.

Real-world example

A law firm cannot send client documents to an outside service. It runs an open model on its own GPU server with a document search system, so drafts and summaries never leave the building, and checks quality on its own test set before rollout.

Diagram Documents stay inside the building
  1. InputClient documentsCannot go to an outside service
  2. ModelOpen model on own GPU serverWith a document search system
  3. OutputDrafts and summariesNever leave the building

How it works

  1. Pick a model whose licence allows your use and whose size fits your memory.
  2. Choose a quantization level (for example 4-bit) and download it through your runner.
  3. Start the runner; it exposes a local API.
  4. Point your Python or .NET client at the local address.
  5. Measure speed (tokens per second), memory and quality on your own examples.
Diagram Setting up a local model
  1. InputPick a modelLicence allows your use; size fits your memory
  2. ProcessChoose quantizationFor example 4-bit; download through your runner
  3. ProcessStart the runnerIt exposes a local API
  4. ProcessPoint your clientPython or .NET, at the local address
  5. OutputMeasureTokens per second, memory and quality on your examples
  • Greedy decoding: always pick the most likely next token. Fast and repeatable, but can be dull or repetitive.
  • Beam search: keep the few best partial answers at each step and pick the best complete one. Common in translation and speech recognition; rarely used for chat.
  • Speculative decoding: a small, fast "draft" model guesses several tokens ahead and the large model checks them in one pass. The output is the same as the large model's, but generation is faster. Runtimes such as vLLM and llama.cpp support it.
  • Continuous batching: a serving technique that adds new requests into a running batch as others finish, so the GPU stays busy. It is the main reason dedicated LLM servers handle many users efficiently.
Diagram Ways to choose the next token

Greedy decoding

  • Always the most likely token
  • Fast and repeatable, can be dull

Beam search

  • Keeps the few best partial answers
  • Common in translation and speech recognition
Diagram Speculative decoding
  1. ProcessDraft model guessesA small, fast model guesses several tokens ahead
  2. ModelLarge model checksChecks the guesses in one pass
  3. OutputAccepted tokensSame output as the large model alone

The output is the same as the large model's, but generation is faster.

Diagram Continuous batching

Without it

  • Batch waits for all requests to finish
  • GPU can sit idle

With it

  • New requests join the batch as others finish
  • GPU stays busy

Python example

Memory for an 8-billion-parameter model by precision:

Output — Output of memory_estimate.py (executed)
Weights only, 8B parameters:
       fp32:  32.0 GB  ( 29.8 GiB)
  bf16/fp16:  16.0 GB  ( 14.9 GiB)
       int8:   8.0 GB  (  7.5 GiB)
      4-bit:   4.0 GB  (  3.7 GiB)

Calling a local Ollama model through the OpenAI-compatible endpoint (requires pip install openai and Ollama running with a pulled model):

Python — local_chat.py
from openai import OpenAI
client = OpenAI(base_url="http://localhost:11434/v1", api_key="not-needed-locally")
reply = client.chat.completions.create(model="llama3.2",
        messages=[{"role": "user", "content": "Explain RAG in two sentences."}], temperature=0)
print(reply.choices[0].message.content)
Shell — Terminal
ollama pull llama3.2          # downloads a quantized model
ollama run llama3.2 "hello"   # quick test

.NET (C#) example

OllamaSharp implements the standard IChatClient interface from Microsoft.Extensions.AI, so business code does not change when you move between local and hosted models:

C# — Program.cs
using Microsoft.Extensions.AI;
using OllamaSharp;

IChatClient chat = new OllamaApiClient(new Uri("http://localhost:11434"), "llama3.2");

await foreach (var update in chat.GetStreamingResponseAsync("Explain RAG in two sentences."))
    Console.Write(update.Text);                      // tokens appear as they are generated
Console.WriteLine();

To switch to a hosted model, register a different IChatClient (for example the OpenAI or Azure OpenAI client); the rest of the code stays the same.

Applied Maths - Statistics

Short, optional reading. Open a topic to see the formula, what it means and two worked examples. The numbers are small, made-up values chosen so you can check them by hand.

Memory for model weights (parameters × bytes)

Where it is used: deciding whether an open-weight model fits in your RAM or GPU memory, and why quantisation helps.

Weights memory

memory in GB = parameters × bytes per parameter ÷ 1,000,000,000
bytes per parameter: 32-bit = 4, 16-bit = 2, 8-bit = 1, 4-bit = 0.5

This counts the weights only. Add room for the KV cache and the runtime.

In plain words: more parameters and more bytes per parameter mean more memory. Lower precision cuts memory in proportion.

Example 1: 8 billion parameters at 4-bit. 8,000,000,000 × 0.5 = 4,000,000,000 bytes = 4 GB. At 16-bit it is 8,000,000,000 × 2 = 16 GB, so 4-bit needs a quarter of the memory.

Example 2: a 70 billion model (made-up case). At 16-bit: 70 × 2 = 140 GB, far above a 24 GB GPU. At 4-bit: 70 × 0.5 = 35 GB, which still does not fit in 24 GB. Some size and precision combinations will never fit a given machine.

KV cache size (memory that grows with the conversation)

Where it is used: the extra memory used while generating, which stores earlier attention keys and values so they are not recomputed.

KV cache memory

bytes per token = 2 × layers × KV heads × head size × bytes per value
KV cache = bytes per token × tokens in context × batch size

The 2 is one set for keys and one for values. The model shape below (32 layers, 8 KV heads, head size 128, 16-bit values) is an assumed example.

In plain words: every token in the context keeps a small block of memory, so a longer context or more users at once means more memory.

Example 1: one long chat. Bytes per token = 2 × 32 × 8 × 128 × 2 = 131,072 (128 KiB). For 8,192 tokens: 8,192 × 131,072 = 1,073,741,824 bytes, about 1.07 GB.

Example 2: eight chats at once. The same context with batch size 8: 1.07 GB × 8 = about 8.59 GB. This is why a server that handles many users needs far more memory than the weights alone.

Maths behind it

  • Quantization (number formats and rounding)
  • Memory arithmetic: parameters × bits ÷ 8
  • KV cache size (grows with context length and number of users)

Common mistakes

  • Picking a model that just fits the weights, then running out of memory with long conversations.
  • Assuming a quantized model is as good as the original without testing on your tasks.
  • Ignoring the model's licence terms for commercial use.
Diagram Fitting a model in memory

Weights only

  • Model just fits
  • Long conversations run out of memory

Weights plus KV cache

  • Leave room for the conversation
  • Test the quantized model on your tasks

Quick check

  1. About how much memory do the weights of an 8B model need at 4 bits?

    Show answer

    About 4 GB, before the KV cache and runtime overhead.

  2. Why is an OpenAI-compatible local API useful?

    Show answer

    The same client code can call a local or a hosted model by changing only the address and model name.

  3. Name one reason a company runs models locally.

    Show answer

    To keep sensitive data in-house (also: predictable cost, offline use, full control).

Completion

Evidence of completion. A local model answering through both a Python and a C# client, with memory use and tokens per second noted for two quantization levels.

Saved in this browser only. See all progress.