Part 6: Building AI applications
Calling an LLM from your application
Learning goal. Call a language model from Python and C#, request structured JSON, validate it, handle failures, stream answers and control cost.
In plain words
Your application talks to an LLM through an API: you send messages and get text back. A system message sets the rules ("You summarise support emails. Reply with JSON only."); user messages carry the content. You pay per token in and out, so long prompts and long answers cost more. For anything a program will use, ask for structured output (JSON with named fields) and validate it like any untrusted input: the model can still return broken JSON or values you did not allow. Plan for failure: timeouts, rate limits, invalid output. Streaming shows the answer word by word so users are not left waiting. Keep API keys in configuration, never in code, and only send data you are allowed to share with the provider.
- InputYour applicationBuilds the messages and keeps the API key in configuration
- InputMessagesA system message with the rules plus user messages with the content
- ProcessLLM APIReceives the request and counts tokens in
- ModelModelWrites the reply one token at a time
- OutputReply textIdeally JSON with named fields; tokens out are counted
Your application sends messages and gets text back. You pay for the tokens in both directions.
- InputSystem messageRules, for example "Reply with JSON only"
- InputUser messageThe content, for example the support email
- OutputReplyText that your code must treat as untrusted
The system message sets the rules. User messages carry the content to work on.
Without streaming
- User waits with nothing on screen
- Whole answer appears at the end
With streaming
- Answer appears word by word
- User can start reading at once
Real-world example
A support team receives hundreds of emails a day. Each one is summarised with a priority and a suggested next action, so agents sort their queue faster. A person still reads the email before replying; the model saves triage time, it does not make the decision.
How it works
- Define the output contract:
summary,priority(low, medium or high) andnext_action. - Write a system message with the fields, allowed values and "do not invent details".
- Call the model with a timeout and low temperature.
- Parse and validate; reject unknown priorities, missing fields and prose around the JSON.
- On failure retry once, then send the email to a person.
- Log tokens, latency and validation failures (not full email text).
- InputOutput contractsummary, priority, next_action
- InputSystem messageFields, allowed values, "do not invent details"
- ModelCall the modelWith a timeout and low temperature
- ProcessParse and validateReject unknown values, missing fields and prose
- RiskRetry once, then a personNever loop forever
- Data storeLogTokens, latency and failures, not full email text
A person is the fallback when validation fails twice.
- OutputValidAll fields present and priority is low, medium or high
- ProblemBroken JSONCannot be parsed
- ProblemUnknown priorityA value you did not allow
- ProblemProse around the JSONExtra text before or after
Treat the reply like any untrusted input.
Related concepts
System prompts
A system prompt is the standing instruction text that comes before the user's message. It sets the role, the rules and the output format for every turn. The user message is the changing request that the rules apply to. Models usually give the system prompt more weight, but it is not a security boundary: a determined user can sometimes talk the model around it. Never put secrets in it, and enforce real limits in your own code.
System prompt
- Standing rules, role and format
- Written by you, the developer
- Same on every request
User message
- The changing request or content
- Comes from the person or the data
- Different on every request
Both are part of the same input. Neither one is a lock.
Structured outputs
Structured output means asking the model for JSON that matches a schema you define: named fields, allowed values and types. Some providers can enforce a schema while the model writes, and others only follow it as an instruction. Either way, validate the reply in your own code. If it is invalid, retry once with the error message, then hand the item to a person.
- InputAsk for JSONDescribe the schema and allowed values
- ModelModel repliesText that should be JSON
- ProcessValidateCheck fields, types and allowed values
- RiskRetry with the errorSay what was wrong, once
If still invalid, send to a person
Validation always runs in your code, never inside the model.
Tool calling
Tool calling lets the model ask your code to do something it cannot do alone, such as look up an order. You describe each function (name, purpose, inputs). The model replies with a request to call one with specific inputs. Your code runs the function, checks that the request is allowed, and sends the result back. The model then uses that result to write its answer. The model never runs code itself. See Agents for loops of many tool calls.
The model
- ModelReads the question and the tool listDecides a tool is needed
- ModelWrites a tool requestFunction name and inputs
- OutputWrites the final answerUses the tool result
Your code
- ProcessChecks the request is allowedValidate the inputs
- ProcessRuns the functionFor example an order lookup
- Data storeReturns the result to the modelAs a new message
Streaming
Streaming sends the reply to your application in small pieces as the model writes it. The user can start reading at once. The key measure is time to first token: how long before the first piece arrives. A stream can fail half way, so keep what has arrived, show that the answer is incomplete, and offer a retry. Do not act on half-finished JSON: parse only after the stream ends.
- InputRequest sentSame messages as a normal call
- ModelFirst token arrivesTime to first token is the wait so far
- OutputPieces keep arrivingShow them as they come
- RiskStream breaksKeep the partial text and mark it incomplete
- ProcessRetry or fall backTry again or show a clear message
Picking a model
There is no single best model, only the best fit for your task. Compare candidates on these points.
- Task difficulty: easy tasks rarely need the largest model.
- Quality on your own evaluation set, not on general claims.
- Latency and cost for your real traffic.
- Context length: can it hold your documents?
- Privacy and where it can run: hosted, private cloud or your own hardware.
- Licence, for open-weight models.
- Ability to follow a structure and to call tools.
Start with a strong model to find the best quality you can get, then try cheaper and smaller ones against the same test set. Re-test when a provider changes or retires a model.
- InputBuild an evaluation setReal inputs with the answers you expect
- ModelTry a strong model firstThis shows the quality ceiling
- ProcessTry cheaper modelsSame test set, same prompt
- ProcessCheck cost, speed and privacyDoes it fit your limits?
- OutputPick the smallest that passesWrite down why
- RiskRe-test on model changesProviders update models
Cost and latency
Cost grows with the tokens you send and receive. Latency is the time the user waits. Both depend on input tokens, output tokens, model size, the number of calls per task and retries. Levers: shorter prompts, prompt caching (see Prompt engineering), smaller models for easy steps, streaming (does not shorten the work, but feels faster), and batching of work that is not urgent. Measure first: log tokens and time per call. See Scaling and cost.
- InputInput tokensLong prompts and documents
- InputOutput tokensLong answers
- ModelModel sizeLarger models are slower and cost more
- ProcessNumber of callsChains and agents multiply calls
- ProblemRetriesEach retry pays again
Change the biggest driver first. Measure before guessing.
Python example
response = client.chat.completions.create(
model=os.environ["LLM_MODEL"],
messages=[{"role": "system", "content": SYSTEM}, {"role": "user", "content": email}],
response_format={"type": "json_object"}, temperature=0, timeout=30)
result = parse_summary(response.choices[0].message.content or "") # raises on invalid outputsummary: Customer received order 55812 with a cracked glass panel and needs a replacement before a client visit next Tuesday.
priority: high
next_action: Confirm the damage with a photo and arrange express replacement shipping.The validator has 4 unit tests (valid reply, unknown priority, missing field, prose instead of JSON), all passing.
.NET (C#) example
Microsoft.Extensions.AI gives one interface, IChatClient, for any provider. GetResponseAsync<T> asks for JSON matching a C# type and turns it into an object (packages Microsoft.Extensions.AI and Microsoft.Extensions.AI.OpenAI, or OllamaSharp for a local model):
using Microsoft.Extensions.AI;
using OpenAI;
var builder = WebApplication.CreateBuilder(args);
builder.Services.AddChatClient(sp =>
{
var config = sp.GetRequiredService<IConfiguration>();
return new OpenAIClient(config["Llm:ApiKey"]).GetChatClient(config["Llm:Model"]).AsIChatClient();
});
var app = builder.Build();
app.MapPost("/summaries", async (SummaryRequest request, IChatClient chat, CancellationToken ct) =>
{
var messages = new List<ChatMessage>
{
new(ChatRole.System, "Summarise the support email. priority must be low, medium or high. Do not invent details."),
new(ChatRole.User, request.EmailText),
};
ChatResponse<EmailSummary> response = await chat.GetResponseAsync<EmailSummary>(messages, cancellationToken: ct);
if (!response.TryGetResult(out var s) || s is null || s.Priority is not ("low" or "medium" or "high"))
return Results.Problem("Invalid model output; route this email to a person.", statusCode: 502);
Console.WriteLine($"tokens in {response.Usage?.InputTokenCount}, out {response.Usage?.OutputTokenCount}");
return Results.Ok(s);
});
app.Run();
public record SummaryRequest(string EmailText);
public record EmailSummary(string Summary, string Priority, string NextAction);Store the key with dotnet user-secrets set "Llm:ApiKey" "<key>" locally and a secret store in production.
Applied Maths - Statistics
Short, optional reading. Open a topic to see the formula, what it means and two worked examples. The numbers are small, made-up values chosen so you can check them by hand.
Token cost (what a call costs)
Where it is used: budgeting calls to an LLM API, which bills input and output tokens separately.
Cost of one call
cost = input tokens × input price + output tokens × output pricePrices are per token. Below, the made-up prices are 2 units per million input tokens and 8 units per million output tokens. They are not real prices.
In plain words: count the tokens in and out, and multiply each by its price. Output tokens usually cost more.
Example 1: one call. 1,500 input tokens and 300 output tokens. Input = 1,500 × 2 ÷ 1,000,000 = 0.0030. Output = 300 × 8 ÷ 1,000,000 = 0.0024. Total = 0.0054 units.
Example 2: one day of traffic. 10,000 such calls: 10,000 × 0.0054 = 54 units per day. Halving the input tokens saves 10,000 × 0.0015 = 15 units per day.
Expected attempts with retries (when calls sometimes fail)
Where it is used: planning retries for rate-limit or timeout errors.
Expected attempts
expected attempts = 1 ÷ (1 - p)p is the chance that one attempt fails (between 0 and 1), assuming retries continue until one works.
In plain words: the more often a call fails, the more tries you need on average, and the more you pay in time.
Example 1: occasional failures. p = 0.2. Expected attempts = 1 ÷ 0.8 = 1.25. If one attempt takes 2 seconds, the average is 2 × 1.25 = 2.5 seconds.
Example 2: a bad hour. p = 0.5. Expected attempts = 1 ÷ 0.5 = 2, so the average is 2 × 2 = 4 seconds. That is why you cap retries and back off.
Maths behind it
- Probability and sampling (temperature)
- Tokenisation (byte-pair encoding)
- None needed for validation: it is ordinary software engineering
Common mistakes
- Trusting model output without validation.
- Hard-coding API keys or logging full customer emails.
- No timeout, so one slow call blocks the whole request pipeline.
- Ignoring cost: long prompts sent thousands of times a day add up.
- Treating the system prompt as a security boundary or putting secrets in it.
- Choosing a model by reputation instead of testing it on your own evaluation set.
- Acting on half-finished streamed JSON instead of waiting for the end of the stream.
Risky
- Output used without validation
- API key written in code
- Full customer emails in logs
- No timeout on the call
Safe
- Output validated first
- Key kept in configuration
- Log tokens and latency only
- Timeout set on every call
Quick check
Why validate JSON you asked the model to produce?
Show answer
The model can still return broken JSON, missing fields or values outside the allowed set; treat output as untrusted input.
What should happen when validation fails twice?
Show answer
Stop retrying, route the item to a person and log the reason.
What is the benefit of
IChatClientin .NET?Show answer
Business code does not depend on one provider; you can switch providers or use a local model by configuration.
How should you choose between a strong model and a cheaper one?
Show answer
Test a strong model first to find the best quality, then try cheaper models on the same evaluation set and keep the smallest one that passes.
Completion
Evidence of completion. The same support-email summarizer working in Python and C#, rejecting invalid output, with token use and latency logged for 20 test emails.
Saved in this browser only. See all progress.