Capstone 7 of 8
Private assistant running fully on a laptop
Business requirement
A law office wants a document assistant that never sends text outside the building: a local model summarises and answers questions over local files.
Lessons to finish first
Architecture
- Local model runtime (for example Ollama with a small quantised model)
- Local embeddings and vector index
- Desktop or local web UI
- Structured output for summaries
- Speed and quality measurements
Build steps
- Install a local model runtime and download two small open models of different sizes.
- Measure tokens per second and memory use for each on your laptop.
- Build summarisation with structured output (title, key points, deadlines) and validate the JSON.
- Reuse capstone 4’s pipeline with local embeddings for question answering over local files.
- Compare quality on 20 test questions between the two models; choose one and explain why.
- Confirm with a network monitor or by disconnecting that nothing leaves the machine.
Python route and .NET route
- Python
- Ollama Python client or an OpenAI-compatible local endpoint, Pydantic for output validation, the local summariser example as a starting point.
- .NET (C#)
- OllamaSharp behind Microsoft.Extensions.AI IChatClient and IEmbeddingGenerator; GetResponseAsync<T> for structured summaries.
Example input and output
- Input
contract_2026_04.txt- Output
{"title": "Service agreement", "key_points": [...], "deadlines": ["2026-12-01"]}
Data
Documents you write yourself; no real client files.
Starter file structure
local-assistant/
summarise.py | Summarise/Program.cs
ask.py
bench.py
README.mdTests
- Works with the network disconnected
- Invalid JSON from the model is caught and retried
- Quality and speed table for both models
How it will be judged
- Model choice justified with measurements
- Validation of every model output
- Clear statement of the hardware it was tested on
Security and privacy
All processing local; encrypt the index at rest; no telemetry from the app.
Definition of done
- Offline demo
- Measured tokens per second and memory
- Validated structured summaries
Stretch goals
- Fine-tune a LoRA adapter for the office’s summary style
- Add a GPU and measure the change
- Package as a desktop app
What to show an employer
Speed and quality table plus an offline demo recording.
Completion
Saved in this browser only.