AI & ML Quick Path

Part 6: Building AI applications

Calling an LLM from your application

Learning goal. Call a language model from Python and C#, request structured JSON, validate it, handle failures, stream answers and control cost.

In plain words

Your application talks to an LLM through an API: you send messages and get text back. A system message sets the rules ("You summarise support emails. Reply with JSON only."); user messages carry the content. You pay per token in and out, so long prompts and long answers cost more. For anything a program will use, ask for structured output (JSON with named fields) and validate it like any untrusted input: the model can still return broken JSON or values you did not allow. Plan for failure: timeouts, rate limits, invalid output. Streaming shows the answer word by word so users are not left waiting. Keep API keys in configuration, never in code, and only send data you are allowed to share with the provider.

Diagram One request to an LLM API
  1. InputYour applicationBuilds the messages and keeps the API key in configuration
  2. InputMessagesA system message with the rules plus user messages with the content
  3. ProcessLLM APIReceives the request and counts tokens in
  4. ModelModelWrites the reply one token at a time
  5. OutputReply textIdeally JSON with named fields; tokens out are counted

Your application sends messages and gets text back. You pay for the tokens in both directions.

Diagram What goes into the messages
  1. InputSystem messageRules, for example "Reply with JSON only"
  2. InputUser messageThe content, for example the support email
  3. OutputReplyText that your code must treat as untrusted

The system message sets the rules. User messages carry the content to work on.

Diagram Waiting versus streaming

Without streaming

  • User waits with nothing on screen
  • Whole answer appears at the end

With streaming

  • Answer appears word by word
  • User can start reading at once

Real-world example

A support team receives hundreds of emails a day. Each one is summarised with a priority and a suggested next action, so agents sort their queue faster. A person still reads the email before replying; the model saves triage time, it does not make the decision.

How it works

  1. Define the output contract: summary, priority (low, medium or high) and next_action.
  2. Write a system message with the fields, allowed values and "do not invent details".
  3. Call the model with a timeout and low temperature.
  4. Parse and validate; reject unknown priorities, missing fields and prose around the JSON.
  5. On failure retry once, then send the email to a person.
  6. Log tokens, latency and validation failures (not full email text).
Diagram The summarizer step by step
  1. InputOutput contractsummary, priority, next_action
  2. InputSystem messageFields, allowed values, "do not invent details"
  3. ModelCall the modelWith a timeout and low temperature
  4. ProcessParse and validateReject unknown values, missing fields and prose
  5. RiskRetry once, then a personNever loop forever
  6. Data storeLogTokens, latency and failures, not full email text

A person is the fallback when validation fails twice.

Diagram Validating the model reply
ModelModel replyJSON text from the model
  • OutputValidAll fields present and priority is low, medium or high
  • ProblemBroken JSONCannot be parsed
  • ProblemUnknown priorityA value you did not allow
  • ProblemProse around the JSONExtra text before or after

Treat the reply like any untrusted input.

System prompts

A system prompt is the standing instruction text that comes before the user's message. It sets the role, the rules and the output format for every turn. The user message is the changing request that the rules apply to. Models usually give the system prompt more weight, but it is not a security boundary: a determined user can sometimes talk the model around it. Never put secrets in it, and enforce real limits in your own code.

Diagram System prompt versus user message

System prompt

  • Standing rules, role and format
  • Written by you, the developer
  • Same on every request

User message

  • The changing request or content
  • Comes from the person or the data
  • Different on every request

Both are part of the same input. Neither one is a lock.

Structured outputs

Structured output means asking the model for JSON that matches a schema you define: named fields, allowed values and types. Some providers can enforce a schema while the model writes, and others only follow it as an instruction. Either way, validate the reply in your own code. If it is invalid, retry once with the error message, then hand the item to a person.

Diagram Structured output with a retry
  1. InputAsk for JSONDescribe the schema and allowed values
  2. ModelModel repliesText that should be JSON
  3. ProcessValidateCheck fields, types and allowed values
  4. RiskRetry with the errorSay what was wrong, once

If still invalid, send to a person

Validation always runs in your code, never inside the model.

Tool calling

Tool calling lets the model ask your code to do something it cannot do alone, such as look up an order. You describe each function (name, purpose, inputs). The model replies with a request to call one with specific inputs. Your code runs the function, checks that the request is allowed, and sends the result back. The model then uses that result to write its answer. The model never runs code itself. See Agents for loops of many tool calls.

Diagram Who does what in a tool call

The model

  1. ModelReads the question and the tool listDecides a tool is needed
  2. ModelWrites a tool requestFunction name and inputs
  3. OutputWrites the final answerUses the tool result

Your code

  1. ProcessChecks the request is allowedValidate the inputs
  2. ProcessRuns the functionFor example an order lookup
  3. Data storeReturns the result to the modelAs a new message

Streaming

Streaming sends the reply to your application in small pieces as the model writes it. The user can start reading at once. The key measure is time to first token: how long before the first piece arrives. A stream can fail half way, so keep what has arrived, show that the answer is incomplete, and offer a retry. Do not act on half-finished JSON: parse only after the stream ends.

Diagram Streaming and a failure mid-stream
  1. InputRequest sentSame messages as a normal call
  2. ModelFirst token arrivesTime to first token is the wait so far
  3. OutputPieces keep arrivingShow them as they come
  4. RiskStream breaksKeep the partial text and mark it incomplete
  5. ProcessRetry or fall backTry again or show a clear message

Picking a model

There is no single best model, only the best fit for your task. Compare candidates on these points.

  • Task difficulty: easy tasks rarely need the largest model.
  • Quality on your own evaluation set, not on general claims.
  • Latency and cost for your real traffic.
  • Context length: can it hold your documents?
  • Privacy and where it can run: hosted, private cloud or your own hardware.
  • Licence, for open-weight models.
  • Ability to follow a structure and to call tools.

Start with a strong model to find the best quality you can get, then try cheaper and smaller ones against the same test set. Re-test when a provider changes or retires a model.

Diagram Choosing a model step by step
  1. InputBuild an evaluation setReal inputs with the answers you expect
  2. ModelTry a strong model firstThis shows the quality ceiling
  3. ProcessTry cheaper modelsSame test set, same prompt
  4. ProcessCheck cost, speed and privacyDoes it fit your limits?
  5. OutputPick the smallest that passesWrite down why
  6. RiskRe-test on model changesProviders update models

Cost and latency

Cost grows with the tokens you send and receive. Latency is the time the user waits. Both depend on input tokens, output tokens, model size, the number of calls per task and retries. Levers: shorter prompts, prompt caching (see Prompt engineering), smaller models for easy steps, streaming (does not shorten the work, but feels faster), and batching of work that is not urgent. Measure first: log tokens and time per call. See Scaling and cost.

Diagram What drives cost and latency
OutputCost and latencyWhat your user and your bill feel
  • InputInput tokensLong prompts and documents
  • InputOutput tokensLong answers
  • ModelModel sizeLarger models are slower and cost more
  • ProcessNumber of callsChains and agents multiply calls
  • ProblemRetriesEach retry pays again

Change the biggest driver first. Measure before guessing.

Python example

Python — summarize.py (excerpt; any OpenAI-compatible endpoint, configured by environment variables)
response = client.chat.completions.create(
    model=os.environ["LLM_MODEL"],
    messages=[{"role": "system", "content": SYSTEM}, {"role": "user", "content": email}],
    response_format={"type": "json_object"}, temperature=0, timeout=30)
result = parse_summary(response.choices[0].message.content or "")   # raises on invalid output
Output — Output of summarize.py --offline (executed: validates a sample model reply without calling a model)
summary:     Customer received order 55812 with a cracked glass panel and needs a replacement before a client visit next Tuesday.
priority:    high
next_action: Confirm the damage with a photo and arrange express replacement shipping.

The validator has 4 unit tests (valid reply, unknown priority, missing field, prose instead of JSON), all passing.

.NET (C#) example

Microsoft.Extensions.AI gives one interface, IChatClient, for any provider. GetResponseAsync<T> asks for JSON matching a C# type and turns it into an object (packages Microsoft.Extensions.AI and Microsoft.Extensions.AI.OpenAI, or OllamaSharp for a local model):

C# — Program.cs (ASP.NET Core minimal API)
using Microsoft.Extensions.AI;
using OpenAI;

var builder = WebApplication.CreateBuilder(args);
builder.Services.AddChatClient(sp =>
{
    var config = sp.GetRequiredService<IConfiguration>();
    return new OpenAIClient(config["Llm:ApiKey"]).GetChatClient(config["Llm:Model"]).AsIChatClient();
});
var app = builder.Build();

app.MapPost("/summaries", async (SummaryRequest request, IChatClient chat, CancellationToken ct) =>
{
    var messages = new List<ChatMessage>
    {
        new(ChatRole.System, "Summarise the support email. priority must be low, medium or high. Do not invent details."),
        new(ChatRole.User, request.EmailText),
    };
    ChatResponse<EmailSummary> response = await chat.GetResponseAsync<EmailSummary>(messages, cancellationToken: ct);
    if (!response.TryGetResult(out var s) || s is null || s.Priority is not ("low" or "medium" or "high"))
        return Results.Problem("Invalid model output; route this email to a person.", statusCode: 502);
    Console.WriteLine($"tokens in {response.Usage?.InputTokenCount}, out {response.Usage?.OutputTokenCount}");
    return Results.Ok(s);
});
app.Run();

public record SummaryRequest(string EmailText);
public record EmailSummary(string Summary, string Priority, string NextAction);

Store the key with dotnet user-secrets set "Llm:ApiKey" "<key>" locally and a secret store in production.

Applied Maths - Statistics

Short, optional reading. Open a topic to see the formula, what it means and two worked examples. The numbers are small, made-up values chosen so you can check them by hand.

Token cost (what a call costs)

Where it is used: budgeting calls to an LLM API, which bills input and output tokens separately.

Cost of one call

cost = input tokens × input price + output tokens × output price

Prices are per token. Below, the made-up prices are 2 units per million input tokens and 8 units per million output tokens. They are not real prices.

In plain words: count the tokens in and out, and multiply each by its price. Output tokens usually cost more.

Example 1: one call. 1,500 input tokens and 300 output tokens. Input = 1,500 × 2 ÷ 1,000,000 = 0.0030. Output = 300 × 8 ÷ 1,000,000 = 0.0024. Total = 0.0054 units.

Example 2: one day of traffic. 10,000 such calls: 10,000 × 0.0054 = 54 units per day. Halving the input tokens saves 10,000 × 0.0015 = 15 units per day.

Expected attempts with retries (when calls sometimes fail)

Where it is used: planning retries for rate-limit or timeout errors.

Expected attempts

expected attempts = 1 ÷ (1 - p)

p is the chance that one attempt fails (between 0 and 1), assuming retries continue until one works.

In plain words: the more often a call fails, the more tries you need on average, and the more you pay in time.

Example 1: occasional failures. p = 0.2. Expected attempts = 1 ÷ 0.8 = 1.25. If one attempt takes 2 seconds, the average is 2 × 1.25 = 2.5 seconds.

Example 2: a bad hour. p = 0.5. Expected attempts = 1 ÷ 0.5 = 2, so the average is 2 × 2 = 4 seconds. That is why you cap retries and back off.

Maths behind it

  • Probability and sampling (temperature)
  • Tokenisation (byte-pair encoding)
  • None needed for validation: it is ordinary software engineering

Common mistakes

  • Trusting model output without validation.
  • Hard-coding API keys or logging full customer emails.
  • No timeout, so one slow call blocks the whole request pipeline.
  • Ignoring cost: long prompts sent thousands of times a day add up.
  • Treating the system prompt as a security boundary or putting secrets in it.
  • Choosing a model by reputation instead of testing it on your own evaluation set.
  • Acting on half-finished streamed JSON instead of waiting for the end of the stream.
Diagram Risky and safe habits

Risky

  • Output used without validation
  • API key written in code
  • Full customer emails in logs
  • No timeout on the call

Safe

  • Output validated first
  • Key kept in configuration
  • Log tokens and latency only
  • Timeout set on every call

Quick check

  1. Why validate JSON you asked the model to produce?

    Show answer

    The model can still return broken JSON, missing fields or values outside the allowed set; treat output as untrusted input.

  2. What should happen when validation fails twice?

    Show answer

    Stop retrying, route the item to a person and log the reason.

  3. What is the benefit of IChatClient in .NET?

    Show answer

    Business code does not depend on one provider; you can switch providers or use a local model by configuration.

  4. How should you choose between a strong model and a cheaper one?

    Show answer

    Test a strong model first to find the best quality, then try cheaper models on the same evaluation set and keep the smallest one that passes.

Completion

Evidence of completion. The same support-email summarizer working in Python and C#, rejecting invalid output, with token use and latency logged for 20 test emails.

Saved in this browser only. See all progress.