Production architecture for AI systems
Learning goal. Design a complete AI system (a .NET application calling a Python model service and an LLM), with containers, secrets, fallbacks, security and a clear checklist, so you can plan any industry case.
In plain words
In industry an AI feature is rarely one program. It is a small system: a front end, an application back end (often ASP.NET Core in .NET companies), one or more model services (often Python, because most ML libraries are Python-first), an LLM provider (hosted API or a local model), a vector database for RAG, a data store, and monitoring. The parts talk through clear contracts (agreed request and response formats, with a version number such as /v1/). Each part is packaged in a container (a portable box holding the app and everything it needs) so it runs the same on a laptop and in the cloud. Good design plans for failure: every call to a model has a timeout, a retry for temporary errors, and a fallback (a simpler rule, a cached answer or a human) so the business keeps running when the model is down. Secrets such as API keys live in a secret store or environment variables, never in code. This lesson pulls everything in the guide together into one design you can reuse.
- InputFront endwhat the user sees
- ProcessApplication back endoften ASP.NET Core in .NET companies
- ModelModel servicesoften Python, where most ML libraries live
- ModelLLM providerhosted API or a local model
- Data storeVector database and data storedocuments for RAG, tickets and decisions
- OutputMonitoringlogs, metrics, dashboards and alerts
Each part talks to the next through a versioned contract.
The same box runs on a laptop and in the cloud.
- ProcessCall the modelwith a timeout
- RiskTemporary errorthe call fails or is too slow
- ProcessRetryfor temporary errors
- OutputFallbacka simpler rule, a cached answer or a human
The business keeps running when the model is down.
In code or the front end
- API keys visible to anyone with the source
- Hard to rotate
Secret store or environment variables
- Keys kept outside the code
- .NET user secrets for development
Real-world example
A support desk built on .NET wants tickets classified and draft replies written. The ASP.NET Core app stays the system of record. It calls a Python classifier service for the ticket category and an LLM with RAG over the help-centre articles for the draft. If the classifier times out, the ticket goes to a general queue for manual triage. Drafts are never sent automatically; an agent approves each one. Public examples of companies running AI assistants over their own knowledge include Morgan Stanley and Klarna; their internal designs are not public, so treat this one as a teaching design.
- InputTicket arrivesinto the ASP.NET Core app
- ModelPython classifiergives the ticket category
- RiskIf it times outticket goes to a general queue for manual triage
- ModelLLM with RAGdrafts a reply from help-centre articles
- OutputAgent approvesdrafts are never sent automatically
A teaching design, not a description of any real company.
How it works
- Start from the decision: what does the system decide or produce, for whom, and what does a mistake cost?
- Pick the simplest approach that can work: rules, classic ML, an LLM, RAG, or an agent (Solve any case walks through this choice).
- Draw the components and the data flow; mark where personal data goes.
- Define contracts: versioned JSON request and response with types and limits.
- Plan failure: timeouts, retries with backoff, circuit breakers, and a fallback for every AI call.
- Secure it: authentication between services, least-privilege tools, secrets in a vault, input and output checks.
- Package and deploy: containers, health checks, autoscaling, staged rollout.
- Monitor and improve: the signals from Monitoring, evaluation runs, and a feedback loop.
The design in this lesson, as a text diagram:
[Web UI] -> [ASP.NET Core API] --(HTTP /v1/classify, 5 s timeout, retry)--> [Python model service (FastAPI + model file)]
| \-- fallback: manual triage queue
|--(IChatClient)--> [LLM: hosted API or local Ollama]
|--(embeddings + search)--> [Vector store: help-centre chunks]
|--> [SQL database: tickets, decisions, model_version]
\--> [Logs + metrics (OpenTelemetry) -> dashboards and alerts]
Secrets: environment variables or a secret store. Every AI output is reviewed by an agent before sending.- ModelPython model serviceHTTP /v1/classify, 5 s timeout, retry
- ModelLLMhosted API or local Ollama
- Data storeVector storehelp-centre chunks
- Data storeSQL databasetickets, decisions, model_version
- OutputLogs and metricsdashboards and alerts
The same design as the text diagram above.
Decide
- InputStart from the decisionwho, what, cost of a mistake
- ProcessPick the simplest approachrules, ML, LLM, RAG or agent
Build
- ProcessDraw components and data flowmark personal data
- ProcessDefine contractsversioned JSON
- RiskPlan failuretimeouts, retries, fallback
- RiskSecure itauthentication, vault, checks
Run
- OutputPackage and deploycontainers, health checks, rollout
- OutputMonitor and improvesignals, evaluation, feedback
Related concepts
Retries and fallbacks
A transient error is a short-lived failure, such as a timeout or a "try again later" reply. Retry it, but wait longer each time (backoff) and stop after a set number of tries. Never retry forever. Set a timeout on each call so a slow service cannot block the user. If the retries fail, use a fallback: another model, a simpler rule or a safe message. Be careful with idempotency: an operation is idempotent if doing it twice has the same effect as doing it once. Give each request an id, so a retry cannot, for example, create the same order twice.
- ProcessCall with a timeouta slow call counts as a failure
- RiskTransient errortimeout or try-again-later reply
- ProcessWait, then retrylonger wait each time, same request id
- ProblemLimit reachedgive up retrying
- OutputFallbackanother model, a simple rule or a safe message
Stop after the agreed limit of tries
Secrets management
A secret is a value that grants access, such as an API key or a password. Keep secrets in environment variables, .NET user secrets (development) or a secret store, never in code or in Git. Rotate them, meaning replace them on a schedule and at once if one leaks. Use separate secrets for each environment (development, test, production), so a leak in one does not open the others.
Code and Git never hold the secret itself.
Deploying to the cloud
Moving an AI app from a laptop to the cloud means packaging it in a container and running it somewhere managed. Common options are a managed app service, a virtual machine (a rented computer you manage yourself) or a serverless function (code that runs only when called). Keep configuration and secrets outside the image. Add a health check so the platform can restart a broken copy, plus scaling rules, logs and a spending limit with alerts. Choose the option from your needs; check each provider's own documents for details and prices.
Package
- ProcessBuild a container imageapp and dependencies, no secrets inside
- Data storeStore the imagein a registry the platform can reach
Run
- ModelManaged service, virtual machine or serverlesspick by need
- Data storeConfiguration and secrets injected at startfrom outside the image
Operate
- OutputHealth checks and scalingrestart broken copies, add copies under load
- OutputLogs and a cost limit with alertssee problems and spending early
Python example
The Python side: a versioned model service with service-to-service authentication and a health check:
import os
from fastapi import FastAPI, Header, HTTPException
from pydantic import BaseModel, Field
app = FastAPI(title="ticket-classifier", version="1.0.0")
MODEL_VERSION = "ticket-tfidf-logreg-2026-10"
class ClassifyRequest(BaseModel):
text: str = Field(min_length=1, max_length=5000)
request_id: str = Field(min_length=1, max_length=64)
class ClassifyResponse(BaseModel):
label: str
score: float = Field(ge=0, le=1)
model_version: str
request_id: str
@app.post("/v1/classify", response_model=ClassifyResponse)
def classify(req: ClassifyRequest, x_api_key: str = Header(...)):
if x_api_key != os.environ.get("SERVICE_API_KEY"):
raise HTTPException(status_code=401, detail="invalid service credentials")
label, score = predict(req.text) # model loaded once at start-up
return ClassifyResponse(label=label, score=score, model_version=MODEL_VERSION, request_id=req.request_id)
@app.get("/health")
def health():
return {"status": "ok", "model_version": MODEL_VERSION}A container file for it:
FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY classifier_service.py model.joblib ./
USER 1000
EXPOSE 8001
CMD ["uvicorn", "classifier_service:app", "--host", "0.0.0.0", "--port", "8001"]The /health and /predict pattern was shown running in Serving a model.
.NET (C#) example
The .NET side: a typed HTTP client with a timeout, standard resilience (retries, circuit breaker) and a fallback. Note the explicit JSON names, because Python uses snake_case and C# uses PascalCase.
using System.Net.Http.Json;
using System.Text.Json.Serialization;
// Program.cs:
// builder.Services.AddHttpClient<ClassifierClient>(c =>
// {
// c.BaseAddress = new Uri(builder.Configuration["Classifier:BaseUrl"]!);
// c.Timeout = TimeSpan.FromSeconds(5);
// }).AddStandardResilienceHandler(); // package Microsoft.Extensions.Http.Resilience
public sealed record ClassifyResponse(
[property: JsonPropertyName("label")] string Label,
[property: JsonPropertyName("score")] double Score,
[property: JsonPropertyName("model_version")] string ModelVersion,
[property: JsonPropertyName("request_id")] string RequestId);
public sealed class ClassifierClient(HttpClient http, IConfiguration config, ILogger<ClassifierClient> log)
{
public async Task<ClassifyResponse?> ClassifyAsync(string text, CancellationToken ct)
{
try
{
using var msg = new HttpRequestMessage(HttpMethod.Post, "/v1/classify")
{
Content = JsonContent.Create(new { text, request_id = Guid.NewGuid().ToString("N") })
};
msg.Headers.Add("X-Api-Key", config["Classifier:ApiKey"]); // from user secrets or a vault
using var res = await http.SendAsync(msg, ct);
if (!res.IsSuccessStatusCode) return null;
var result = await res.Content.ReadFromJsonAsync<ClassifyResponse>(cancellationToken: ct);
return result is { Score: >= 0 and <= 1 } ? result : null; // validate before use
}
catch (Exception ex) when (ex is HttpRequestException or TaskCanceledException)
{
log.LogWarning(ex, "Classifier unavailable; falling back to manual triage");
return null; // caller routes to the manual queue
}
}
}The full pair of files is in the guide's examples/hybrid-service folder.
Design checklist for any case (use it for capstones and real projects):
- Decision, users and cost of a wrong answer are written down.
- Simplest workable approach chosen and the reason recorded.
- Baseline measured; evaluation set exists before building.
- Contracts versioned; every AI call has timeout, retry and fallback.
- Personal data mapped and minimised; secrets outside code; least-privilege tools; human approval for consequential actions.
- Monitoring: health, drift, quality, latency, cost, each with threshold and owner.
- Rollout plan (shadow or canary) and rollback plan.
- Cost estimate per 1,000 requests and a monthly budget alert.
Applied Maths - Statistics
Short, optional reading. Open a topic to see the formula, what it means and two worked examples. The numbers are small, made-up values chosen so you can check them by hand.
Availability of chained services (how reliable the whole system is)
Where it is used: estimating how often the support desk can produce a draft when the app, the classifier and the LLM must all work.
Availability
chain of services (all must work): availability = a1 × a2 × ... × an
two copies of one service (one is enough): availability = 1 - (1 - a) × (1 - a)
a = the share of time a service works, written as a decimal (99.9% = 0.999)In plain words: when every part is needed, the system is less reliable than its weakest part. A spare copy makes the failure chance much smaller.
Example 1: three services in a chain. App 0.9995, classifier 0.999, LLM 0.995. Availability = 0.9995 × 0.999 × 0.995 = 0.9935 (about 99.35%). That is roughly 0.65% of the time, or about 57 hours a year (0.0065 × 8,760 hours), without a draft.
Example 2: a spare copy. One classifier copy works 99% of the time (a = 0.99). Two copies: 1 - (0.01 × 0.01) = 1 - 0.0001 = 0.9999, or 99.99%.
Exponential backoff (waiting longer after each failed call)
Where it is used: retrying the classifier or LLM call after a timeout without flooding a service that is struggling.
Exponential backoff
wait before retry k = base × 2^(k - 1), but never more than the cap
total waiting = the sum of the waitsIn plain words: each retry waits twice as long as the one before, up to a limit.
Example 1: four retries. Base 0.5 seconds, no cap. Waits are 0.5, 1, 2 and 4 seconds. Total = 0.5 + 1 + 2 + 4 = 7.5 seconds.
Example 2: with a cap. Base 0.5 seconds, cap 5 seconds, five retries. Waits are 0.5, 1, 2, 4 and then 8 is cut to 5. Total = 0.5 + 1 + 2 + 4 + 5 = 12.5 seconds. After that, send the ticket to the general queue.
Maths behind it
- Queueing theory and capacity planning
- Probability of failure (availability of chained services)
- Exponential backoff
Common mistakes
- One giant service that mixes web app, model training and serving.
- No fallback, so a model outage stops the business process.
- Mismatched JSON naming between Python and .NET, giving silent nulls.
- API keys in source code or in the front end.
- Letting an LLM take actions with no human approval.
- Retrying without a limit or a growing wait, or retrying an action that is not idempotent.
- Using the same secrets in development and production, and never rotating them.
One giant service
- Web app, training and serving mixed together
- A model outage stops everything
Separate services
- Each part has its own versioned contract
- Each side changes independently
- A fallback keeps the process running
Quick check
Why do many .NET companies call a separate Python service for ML?
Show answer
Most ML libraries are Python-first; a separate service with a versioned contract lets each side use its best tools and change independently.
What should happen when the model service times out?
Show answer
A fallback: a simpler rule, a cached answer or a human queue, so the business process continues; log the event.
Where should API keys live?
Show answer
In environment variables, .NET user secrets for development, or a secret store in production; never in code or the browser.
Why does idempotency matter when you retry?
Show answer
A retry may repeat an action that already worked; if the action is idempotent, the repeat has no extra effect.
Completion
Evidence of completion. An architecture diagram and one-page design note for a capstone, covering components, contract, failure modes, security, monitoring and cost.
Saved in this browser only. See all progress.