Capstone 6 of 8
Train a small language model on your laptop
Business requirement
Understand language models by building one: train a small transformer on a text collection on your own laptop, then plan how the same code would scale on GPUs.
Lessons to finish first
Architecture
- Text collection (public-domain or your own writing)
- Tokenizer (characters first, then BPE)
- Small decoder-only transformer
- Training loop with validation loss
- Sampling with temperature
- Scale-up plan (bigger model, more data, more GPUs)
Build steps
- Collect 1 to 5 MB of text you are allowed to use and split it 90/10 into train and validation.
- Start with a character tokenizer; later train a small BPE tokenizer and compare.
- Build a decoder-only transformer with about 1 to 10 million parameters.
- Train on CPU or a laptop GPU; plot training and validation loss; stop when validation loss stops improving.
- Generate text at temperatures 0.5, 1.0 and 1.5 and describe the difference.
- Write the scale-up plan: model size, tokens, GPU memory estimate, and which code changes (mixed precision, data parallel) are needed.
Python route and .NET route
- Python
- NumPy version first (the tiny character model example), then PyTorch (nn.TransformerEncoderLayer with a causal mask, or your own attention block).
- .NET (C#)
- TorchSharp provides the same building blocks in C#; Microsoft.ML.Tokenizers for BPE. Training in Python and running inference from .NET through ONNX is also common.
Example input and output
- Input
prompt "The ship", temperature 0.8- Output
about 200 generated characters
Data
Public-domain books or your own text. Do not train on text you do not have the right to use.
Starter file structure
tinylm/
data/
tokenizer.py
model.py
train.py
sample.py
SCALE_PLAN.mdTests
- Validation loss decreases and is reported with the parameter count
- Generation is reproducible with a fixed seed
- Memory estimate written for a 10x larger model
How it will be judged
- Loss curves plotted and explained
- Overfitting noticed if it happens
- Scale-up plan is concrete and realistic
Security and privacy
Check data licences; remove personal data from any text you collect.
Definition of done
- Trained model, loss plot and samples
- Comparison of character and BPE tokenizers
- Written scale-up plan
Stretch goals
- Fine-tune a small open model with LoRA and compare
- Export to ONNX and run from C#
- Measure tokens per second
What to show an employer
Loss plot, three samples at different temperatures and the scale-up plan.
Completion
Saved in this browser only.