AI & ML Quick Path

Capstone 6 of 8

Train a small language model on your laptop

Language modelsAdvancedEffort 25–40 h (estimate)

Business requirement

Understand language models by building one: train a small transformer on a text collection on your own laptop, then plan how the same code would scale on GPUs.

Lessons to finish first

Architecture

  1. Text collection (public-domain or your own writing)
  2. Tokenizer (characters first, then BPE)
  3. Small decoder-only transformer
  4. Training loop with validation loss
  5. Sampling with temperature
  6. Scale-up plan (bigger model, more data, more GPUs)

Build steps

  1. Collect 1 to 5 MB of text you are allowed to use and split it 90/10 into train and validation.
  2. Start with a character tokenizer; later train a small BPE tokenizer and compare.
  3. Build a decoder-only transformer with about 1 to 10 million parameters.
  4. Train on CPU or a laptop GPU; plot training and validation loss; stop when validation loss stops improving.
  5. Generate text at temperatures 0.5, 1.0 and 1.5 and describe the difference.
  6. Write the scale-up plan: model size, tokens, GPU memory estimate, and which code changes (mixed precision, data parallel) are needed.

Python route and .NET route

Python
NumPy version first (the tiny character model example), then PyTorch (nn.TransformerEncoderLayer with a causal mask, or your own attention block).
.NET (C#)
TorchSharp provides the same building blocks in C#; Microsoft.ML.Tokenizers for BPE. Training in Python and running inference from .NET through ONNX is also common.

Example input and output

Input
prompt "The ship", temperature 0.8
Output
about 200 generated characters

Data

Public-domain books or your own text. Do not train on text you do not have the right to use.

Starter file structure

Structure (Python file | .NET project)
tinylm/
  data/
  tokenizer.py
  model.py
  train.py
  sample.py
  SCALE_PLAN.md

Tests

  • Validation loss decreases and is reported with the parameter count
  • Generation is reproducible with a fixed seed
  • Memory estimate written for a 10x larger model

How it will be judged

  • Loss curves plotted and explained
  • Overfitting noticed if it happens
  • Scale-up plan is concrete and realistic

Security and privacy

Check data licences; remove personal data from any text you collect.

Definition of done

  • Trained model, loss plot and samples
  • Comparison of character and BPE tokenizers
  • Written scale-up plan

Stretch goals

  • Fine-tune a small open model with LoRA and compare
  • Export to ONNX and run from C#
  • Measure tokens per second

What to show an employer

Loss plot, three samples at different temperatures and the scale-up plan.

Completion

Saved in this browser only.