From laptop to big hardware: scaling training
Learning goal. Estimate memory for training and inference, and choose the right scaling tools (GPUs, mixed precision, gradient accumulation and checkpointing, data and model parallelism, FSDP) as models grow.
In plain words
The same training loop scales from a laptop to thousands of GPUs, but memory and time become the limits. Memory has several parts: the weights (parameters × bytes each; 2 bytes in 16-bit formats), the gradients, the optimiser state (AdamW keeps extra numbers per weight) and the activations saved for the backward pass. A common rule of thumb for full training with AdamW is about 16 bytes per parameter before activations. Tools to fit and speed up training, from easiest to hardest: use a GPU; use mixed precision (BF16); gradient accumulation (add up gradients from several small batches to act like one big batch); gradient checkpointing (recompute activations instead of storing them); data parallelism (copies of the model on several GPUs, each with different data); sharded data parallelism (FSDP or DeepSpeed ZeRO split weights, gradients and optimiser state across GPUs); and tensor and pipeline parallelism (split the model itself across GPUs). For fine-tuning, LoRA and QLoRA cut memory far more than any of these.
- ModelWeightsParameters times bytes each; 2 bytes in 16-bit formats
- ProcessGradientsOne per weight, from the backward pass
- Data storeOptimiser stateAdamW keeps extra numbers per weight
- Data storeActivationsSaved for the backward pass
Full training with AdamW is about 16 bytes per parameter before activations.
- ProcessGPUUse one first
- ProcessMixed precision (BF16)Smaller numbers, less memory
- ProcessGradient accumulationAdd gradients from small batches to act like a big batch
- ProcessGradient checkpointingRecompute activations instead of storing them
- ModelData parallelismModel copies on several GPUs, different data each
- ModelSharded data parallelism (FSDP, ZeRO)Split weights, gradients and optimiser state
- ModelTensor and pipeline parallelismSplit the model itself across GPUs
Use the easy tools first; move down only when memory or time forces you.
Data parallelism
- Full copy of the model on each GPU
- Each GPU gets different data
- Adds speed, not memory
Sharded (FSDP, ZeRO)
- Weights, gradients and optimiser state split
- Each GPU holds a share
- Reduces memory per GPU
Tensor and pipeline
- The model itself is split
- Different GPUs hold different parts
Full fine-tuning
- All weights trained
- Gradients and optimiser state for everything
LoRA and QLoRA
- Frozen model, small added parts trained
- Cuts memory far more than the other tools
Real-world example
A team wants to fine-tune an 8-billion-parameter model. Full fine-tuning would need about 128 GB for training state alone, more than one 80 GB GPU. Options: shard across 8 GPUs with FSDP (about 16 GB each), or use LoRA on one GPU. They choose LoRA for the first version because it is cheaper and faster to iterate.
- ModelFSDP on 8 GPUsAbout 16 GB each
- OutputLoRA on one GPUChosen for version one: cheaper, faster to iterate
Numbers are the ones in this lesson's text.
How it works
- Estimate memory: weights, gradients, optimiser state, activations.
- If it fits one GPU: use mixed precision; add gradient accumulation for larger effective batches.
- If activations are the problem: gradient checkpointing.
- If weights plus optimiser state are the problem: FSDP or ZeRO across GPUs, or switch to LoRA/QLoRA.
- For speed: data parallelism across GPUs; measure tokens per second and cost.
- Checkpoint often; long jobs on many GPUs fail more often.
- InputEstimate memoryWeights, gradients, optimiser state, activations
- ProcessFits one GPU?Use mixed precision; add gradient accumulation
- ProcessActivations too big?Gradient checkpointing
- ModelWeights plus optimiser too big?FSDP or ZeRO, or LoRA and QLoRA
- OutputNeed speed?Data parallelism; measure tokens per second and cost
Without it
- One update per batch
- A larger batch needs more memory
With it
- Gradients from small batches are added up
- One update acts like a larger batch
Python example
Memory arithmetic for an 8-billion-parameter model:
Weights only, 8B parameters:
fp32: 32.0 GB ( 29.8 GiB)
bf16/fp16: 16.0 GB ( 14.9 GiB)
int8: 8.0 GB ( 7.5 GiB)
4-bit: 4.0 GB ( 3.7 GiB)
Full fine-tuning state (AdamW, mixed precision, no activations): 128 GB
KV cache per token (bf16): 128 KiB
8192 tokens x batch 1: 1.07 GB
32768 tokens x batch 1: 4.29 GB
8192 tokens x batch 8: 8.59 GBMixed precision, gradient accumulation and multi-GPU launch in PyTorch:
scaler_dtype = torch.bfloat16
for i, (xb, yb) in enumerate(loader):
with torch.autocast("cuda", dtype=scaler_dtype): # mixed precision
loss = model(xb.cuda(), yb.cuda()) / accum_steps
loss.backward()
if (i + 1) % accum_steps == 0: # gradient accumulation
optimizer.step(); optimizer.zero_grad()
# Multi-GPU: wrap the model with torch.distributed.fsdp.FullyShardedDataParallel and launch with
# torchrun --nproc_per_node=8 train_scaled.py.NET (C#) example
Large-scale training is almost always done in Python. .NET teams typically plan jobs, launch them on cloud GPUs and use the trained model. A memory planner in plain C#:
double parameters = 8e9;
int gpus = 8; double gpuMemoryGb = 80;
double weights16 = parameters * 2 / 1e9;
double fullTraining = parameters * 16 / 1e9; // AdamW, mixed precision, before activations
double perGpuDdp = fullTraining; // data parallel: every GPU holds everything
double perGpuFsdp = fullTraining / gpus; // sharded: state split across GPUs
Console.WriteLine($"weights (16-bit): {weights16:F0} GB, full training state: {fullTraining:F0} GB");
Console.WriteLine($"data parallel per GPU: {perGpuDdp:F0} GB -> {(perGpuDdp < gpuMemoryGb ? "fits" : "does not fit")}");
Console.WriteLine($"FSDP per GPU: {perGpuFsdp:F0} GB -> {(perGpuFsdp < gpuMemoryGb ? "fits (plus activations)" : "does not fit")}");Applied Maths - Statistics
Short, optional reading. Open a topic to see the formula, what it means and two worked examples. The numbers are small, made-up values chosen so you can check them by hand.
Training memory per GPU (data parallel and sharded)
Where it is used: choosing between plain data parallelism and sharding (FSDP, ZeRO) when full fine-tuning does not fit on one GPU.
Memory per GPU
full training state ≈ parameters × 16 bytes (before activations)
data parallel, per GPU = full training state
sharded (FSDP), per GPU ≈ full training state ÷ number of GPUsThe 16 bytes per parameter is a rule of thumb for weights, gradients and AdamW optimiser state with mixed precision. Activations come on top.
In plain words: with data parallelism every GPU holds a full copy. With sharding the copy is split, so each GPU holds a share.
Example 1: sharded across 8 GPUs. An 8 billion parameter model: 8,000,000,000 × 16 = 128 GB. Split over 8 GPUs: 128 ÷ 8 = 16 GB each. This fits in an 80 GB GPU with room for activations.
Example 2: plain data parallel. Every GPU needs the whole 128 GB. 128 GB is more than 80 GB, so it does not fit, no matter how many GPUs you add.
Effective batch size (gradient accumulation)
Where it is used: the accumulation loop in the PyTorch example, which gives a large batch without large memory.
Effective batch size
effective batch = micro-batch size × accumulation steps × number of GPUsThe micro-batch is the number of examples one GPU processes in one forward and backward pass.
In plain words: the optimiser takes one step only after several small batches have added up their gradients, so it behaves like one big batch.
Example 1: eight GPUs. Micro-batch 4, accumulation 8, GPUs 8: 4 × 8 × 8 = 256 examples per optimiser step.
Example 2: one GPU. Micro-batch 4, accumulation 16, one GPU: 4 × 16 × 1 = 64 examples per step. Dividing the loss by the accumulation steps keeps the gradient the same size as it would be for one batch of 64.
Maths behind it
- Memory arithmetic: parameters × bytes per value
- Floating-point formats: FP32, BF16, FP16, INT8, 4-bit
- Gradient accumulation (averaging gradients)
- Parallel computing: data, tensor and pipeline parallelism
- Communication cost (all-reduce)
Common mistakes
- Expecting data parallelism to reduce memory per GPU; it only adds speed.
- Assuming 8 GPUs give 8 times the speed; communication overhead reduces it.
- Long runs without frequent checkpoints.
Expected
- Data parallelism lowers memory per GPU
- 8 GPUs give 8 times the speed
Reality
- Data parallelism only adds speed (full copies)
- Communication overhead reduces the speed-up
Quick check
Does data parallelism reduce memory per GPU?
Show answer
No. Each GPU holds a full copy; it increases speed. Sharding (FSDP, ZeRO) reduces memory per GPU.
What does gradient accumulation do?
Show answer
Adds up gradients from several small batches before one update, acting like a larger batch without more memory.
About how much memory does full fine-tuning of an 8B model need for training state?
Show answer
About 128 GB (16 bytes per parameter) before activations.
Completion
Evidence of completion. A written plan to train or fine-tune a named model size on a named GPU setup, with memory arithmetic and the chosen techniques justified.
Saved in this browser only. See all progress.