AI & ML Quick Path

Part 7: Production

Production architecture for AI systems

Learning goal. Design a complete AI system (a .NET application calling a Python model service and an LLM), with containers, secrets, fallbacks, security and a clear checklist, so you can plan any industry case.

In plain words

In industry an AI feature is rarely one program. It is a small system: a front end, an application back end (often ASP.NET Core in .NET companies), one or more model services (often Python, because most ML libraries are Python-first), an LLM provider (hosted API or a local model), a vector database for RAG, a data store, and monitoring. The parts talk through clear contracts (agreed request and response formats, with a version number such as /v1/). Each part is packaged in a container (a portable box holding the app and everything it needs) so it runs the same on a laptop and in the cloud. Good design plans for failure: every call to a model has a timeout, a retry for temporary errors, and a fallback (a simpler rule, a cached answer or a human) so the business keeps running when the model is down. Secrets such as API keys live in a secret store or environment variables, never in code. This lesson pulls everything in the guide together into one design you can reuse.

Diagram The parts of an AI system
  1. InputFront endwhat the user sees
  2. ProcessApplication back endoften ASP.NET Core in .NET companies
  3. ModelModel servicesoften Python, where most ML libraries live
  4. ModelLLM providerhosted API or a local model
  5. Data storeVector database and data storedocuments for RAG, tickets and decisions
  6. OutputMonitoringlogs, metrics, dashboards and alerts

Each part talks to the next through a versioned contract.

Diagram A container packages everything the app needs
Data storeCloud or laptopwhere the container runs
ProcessContainera portable box
ModelApp and its dependencieseverything it needs to run

The same box runs on a laptop and in the cloud.

Diagram Planning for failure on every model call
  1. ProcessCall the modelwith a timeout
  2. RiskTemporary errorthe call fails or is too slow
  3. ProcessRetryfor temporary errors
  4. OutputFallbacka simpler rule, a cached answer or a human

The business keeps running when the model is down.

Diagram Where secrets live

In code or the front end

  • API keys visible to anyone with the source
  • Hard to rotate

Secret store or environment variables

  • Keys kept outside the code
  • .NET user secrets for development

Real-world example

A support desk built on .NET wants tickets classified and draft replies written. The ASP.NET Core app stays the system of record. It calls a Python classifier service for the ticket category and an LLM with RAG over the help-centre articles for the draft. If the classifier times out, the ticket goes to a general queue for manual triage. Drafts are never sent automatically; an agent approves each one. Public examples of companies running AI assistants over their own knowledge include Morgan Stanley and Klarna; their internal designs are not public, so treat this one as a teaching design.

Diagram The support desk design
  1. InputTicket arrivesinto the ASP.NET Core app
  2. ModelPython classifiergives the ticket category
  3. RiskIf it times outticket goes to a general queue for manual triage
  4. ModelLLM with RAGdrafts a reply from help-centre articles
  5. OutputAgent approvesdrafts are never sent automatically

A teaching design, not a description of any real company.

How it works

  1. Start from the decision: what does the system decide or produce, for whom, and what does a mistake cost?
  2. Pick the simplest approach that can work: rules, classic ML, an LLM, RAG, or an agent (Solve any case walks through this choice).
  3. Draw the components and the data flow; mark where personal data goes.
  4. Define contracts: versioned JSON request and response with types and limits.
  5. Plan failure: timeouts, retries with backoff, circuit breakers, and a fallback for every AI call.
  6. Secure it: authentication between services, least-privilege tools, secrets in a vault, input and output checks.
  7. Package and deploy: containers, health checks, autoscaling, staged rollout.
  8. Monitor and improve: the signals from Monitoring, evaluation runs, and a feedback loop.

The design in this lesson, as a text diagram:

Output — Architecture
[Web UI] -> [ASP.NET Core API] --(HTTP /v1/classify, 5 s timeout, retry)--> [Python model service (FastAPI + model file)]
                 |                                                        \-- fallback: manual triage queue
                 |--(IChatClient)--> [LLM: hosted API or local Ollama]
                 |--(embeddings + search)--> [Vector store: help-centre chunks]
                 |--> [SQL database: tickets, decisions, model_version]
                 \--> [Logs + metrics (OpenTelemetry) -> dashboards and alerts]
Secrets: environment variables or a secret store. Every AI output is reviewed by an agent before sending.
Diagram The ASP.NET Core API at the centre
ProcessASP.NET Core APIthe system of record
  • ModelPython model serviceHTTP /v1/classify, 5 s timeout, retry
  • ModelLLMhosted API or local Ollama
  • Data storeVector storehelp-centre chunks
  • Data storeSQL databasetickets, decisions, model_version
  • OutputLogs and metricsdashboards and alerts

The same design as the text diagram above.

Diagram Eight design steps in three groups

Decide

  1. InputStart from the decisionwho, what, cost of a mistake
  2. ProcessPick the simplest approachrules, ML, LLM, RAG or agent

Build

  1. ProcessDraw components and data flowmark personal data
  2. ProcessDefine contractsversioned JSON
  3. RiskPlan failuretimeouts, retries, fallback
  4. RiskSecure itauthentication, vault, checks

Run

  1. OutputPackage and deploycontainers, health checks, rollout
  2. OutputMonitor and improvesignals, evaluation, feedback

Retries and fallbacks

A transient error is a short-lived failure, such as a timeout or a "try again later" reply. Retry it, but wait longer each time (backoff) and stop after a set number of tries. Never retry forever. Set a timeout on each call so a slow service cannot block the user. If the retries fail, use a fallback: another model, a simpler rule or a safe message. Be careful with idempotency: an operation is idempotent if doing it twice has the same effect as doing it once. Give each request an id, so a retry cannot, for example, create the same order twice.

Diagram Retry with growing waits, then fall back
  1. ProcessCall with a timeouta slow call counts as a failure
  2. RiskTransient errortimeout or try-again-later reply
  3. ProcessWait, then retrylonger wait each time, same request id
  4. ProblemLimit reachedgive up retrying
  5. OutputFallbackanother model, a simple rule or a safe message

Stop after the agreed limit of tries

Secrets management

A secret is a value that grants access, such as an API key or a password. Keep secrets in environment variables, .NET user secrets (development) or a secret store, never in code or in Git. Rotate them, meaning replace them on a schedule and at once if one leaks. Use separate secrets for each environment (development, test, production), so a leak in one does not open the others.

Diagram Where a secret is kept
Data storeSecret storethe production home, with access control
ProcessEnvironment variables or user secretsread by the app at start-up
ModelApplication coderefers to a setting name, not the value

Code and Git never hold the secret itself.

Deploying to the cloud

Moving an AI app from a laptop to the cloud means packaging it in a container and running it somewhere managed. Common options are a managed app service, a virtual machine (a rented computer you manage yourself) or a serverless function (code that runs only when called). Keep configuration and secrets outside the image. Add a health check so the platform can restart a broken copy, plus scaling rules, logs and a spending limit with alerts. Choose the option from your needs; check each provider's own documents for details and prices.

Diagram From laptop to cloud

Package

  1. ProcessBuild a container imageapp and dependencies, no secrets inside
  2. Data storeStore the imagein a registry the platform can reach

Run

  1. ModelManaged service, virtual machine or serverlesspick by need
  2. Data storeConfiguration and secrets injected at startfrom outside the image

Operate

  1. OutputHealth checks and scalingrestart broken copies, add copies under load
  2. OutputLogs and a cost limit with alertssee problems and spending early

Python example

The Python side: a versioned model service with service-to-service authentication and a health check:

Python — classifier_service.py (FastAPI; not run here)
import os
from fastapi import FastAPI, Header, HTTPException
from pydantic import BaseModel, Field

app = FastAPI(title="ticket-classifier", version="1.0.0")
MODEL_VERSION = "ticket-tfidf-logreg-2026-10"

class ClassifyRequest(BaseModel):
    text: str = Field(min_length=1, max_length=5000)
    request_id: str = Field(min_length=1, max_length=64)

class ClassifyResponse(BaseModel):
    label: str
    score: float = Field(ge=0, le=1)
    model_version: str
    request_id: str

@app.post("/v1/classify", response_model=ClassifyResponse)
def classify(req: ClassifyRequest, x_api_key: str = Header(...)):
    if x_api_key != os.environ.get("SERVICE_API_KEY"):
        raise HTTPException(status_code=401, detail="invalid service credentials")
    label, score = predict(req.text)          # model loaded once at start-up
    return ClassifyResponse(label=label, score=score, model_version=MODEL_VERSION, request_id=req.request_id)

@app.get("/health")
def health():
    return {"status": "ok", "model_version": MODEL_VERSION}

A container file for it:

Output — Dockerfile (not built here)
FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY classifier_service.py model.joblib ./
USER 1000
EXPOSE 8001
CMD ["uvicorn", "classifier_service:app", "--host", "0.0.0.0", "--port", "8001"]

The /health and /predict pattern was shown running in Serving a model.

.NET (C#) example

The .NET side: a typed HTTP client with a timeout, standard resilience (retries, circuit breaker) and a fallback. Note the explicit JSON names, because Python uses snake_case and C# uses PascalCase.

C# — ClassifierClient.cs
using System.Net.Http.Json;
using System.Text.Json.Serialization;

// Program.cs:
// builder.Services.AddHttpClient<ClassifierClient>(c =>
// {
//     c.BaseAddress = new Uri(builder.Configuration["Classifier:BaseUrl"]!);
//     c.Timeout = TimeSpan.FromSeconds(5);
// }).AddStandardResilienceHandler();          // package Microsoft.Extensions.Http.Resilience

public sealed record ClassifyResponse(
    [property: JsonPropertyName("label")] string Label,
    [property: JsonPropertyName("score")] double Score,
    [property: JsonPropertyName("model_version")] string ModelVersion,
    [property: JsonPropertyName("request_id")] string RequestId);

public sealed class ClassifierClient(HttpClient http, IConfiguration config, ILogger<ClassifierClient> log)
{
    public async Task<ClassifyResponse?> ClassifyAsync(string text, CancellationToken ct)
    {
        try
        {
            using var msg = new HttpRequestMessage(HttpMethod.Post, "/v1/classify")
            {
                Content = JsonContent.Create(new { text, request_id = Guid.NewGuid().ToString("N") })
            };
            msg.Headers.Add("X-Api-Key", config["Classifier:ApiKey"]);   // from user secrets or a vault
            using var res = await http.SendAsync(msg, ct);
            if (!res.IsSuccessStatusCode) return null;
            var result = await res.Content.ReadFromJsonAsync<ClassifyResponse>(cancellationToken: ct);
            return result is { Score: >= 0 and <= 1 } ? result : null;     // validate before use
        }
        catch (Exception ex) when (ex is HttpRequestException or TaskCanceledException)
        {
            log.LogWarning(ex, "Classifier unavailable; falling back to manual triage");
            return null;                                                  // caller routes to the manual queue
        }
    }
}

The full pair of files is in the guide's examples/hybrid-service folder.

Design checklist for any case (use it for capstones and real projects):

  • Decision, users and cost of a wrong answer are written down.
  • Simplest workable approach chosen and the reason recorded.
  • Baseline measured; evaluation set exists before building.
  • Contracts versioned; every AI call has timeout, retry and fallback.
  • Personal data mapped and minimised; secrets outside code; least-privilege tools; human approval for consequential actions.
  • Monitoring: health, drift, quality, latency, cost, each with threshold and owner.
  • Rollout plan (shadow or canary) and rollback plan.
  • Cost estimate per 1,000 requests and a monthly budget alert.

Applied Maths - Statistics

Short, optional reading. Open a topic to see the formula, what it means and two worked examples. The numbers are small, made-up values chosen so you can check them by hand.

Availability of chained services (how reliable the whole system is)

Where it is used: estimating how often the support desk can produce a draft when the app, the classifier and the LLM must all work.

Availability

chain of services (all must work):  availability = a1 × a2 × ... × an
two copies of one service (one is enough):  availability = 1 - (1 - a) × (1 - a)
a = the share of time a service works, written as a decimal (99.9% = 0.999)

In plain words: when every part is needed, the system is less reliable than its weakest part. A spare copy makes the failure chance much smaller.

Example 1: three services in a chain. App 0.9995, classifier 0.999, LLM 0.995. Availability = 0.9995 × 0.999 × 0.995 = 0.9935 (about 99.35%). That is roughly 0.65% of the time, or about 57 hours a year (0.0065 × 8,760 hours), without a draft.

Example 2: a spare copy. One classifier copy works 99% of the time (a = 0.99). Two copies: 1 - (0.01 × 0.01) = 1 - 0.0001 = 0.9999, or 99.99%.

Exponential backoff (waiting longer after each failed call)

Where it is used: retrying the classifier or LLM call after a timeout without flooding a service that is struggling.

Exponential backoff

wait before retry k = base × 2^(k - 1), but never more than the cap
total waiting = the sum of the waits

In plain words: each retry waits twice as long as the one before, up to a limit.

Example 1: four retries. Base 0.5 seconds, no cap. Waits are 0.5, 1, 2 and 4 seconds. Total = 0.5 + 1 + 2 + 4 = 7.5 seconds.

Example 2: with a cap. Base 0.5 seconds, cap 5 seconds, five retries. Waits are 0.5, 1, 2, 4 and then 8 is cut to 5. Total = 0.5 + 1 + 2 + 4 + 5 = 12.5 seconds. After that, send the ticket to the general queue.

Maths behind it

  • Queueing theory and capacity planning
  • Probability of failure (availability of chained services)
  • Exponential backoff

Common mistakes

  • One giant service that mixes web app, model training and serving.
  • No fallback, so a model outage stops the business process.
  • Mismatched JSON naming between Python and .NET, giving silent nulls.
  • API keys in source code or in the front end.
  • Letting an LLM take actions with no human approval.
  • Retrying without a limit or a growing wait, or retrying an action that is not idempotent.
  • Using the same secrets in development and production, and never rotating them.
Diagram One giant service versus separate parts

One giant service

  • Web app, training and serving mixed together
  • A model outage stops everything

Separate services

  • Each part has its own versioned contract
  • Each side changes independently
  • A fallback keeps the process running

Quick check

  1. Why do many .NET companies call a separate Python service for ML?

    Show answer

    Most ML libraries are Python-first; a separate service with a versioned contract lets each side use its best tools and change independently.

  2. What should happen when the model service times out?

    Show answer

    A fallback: a simpler rule, a cached answer or a human queue, so the business process continues; log the event.

  3. Where should API keys live?

    Show answer

    In environment variables, .NET user secrets for development, or a secret store in production; never in code or the browser.

  4. Why does idempotency matter when you retry?

    Show answer

    A retry may repeat an action that already worked; if the action is idempotent, the repeat has no extra effect.

Completion

Evidence of completion. An architecture diagram and one-page design note for a capstone, covering components, contract, failure modes, security, monitoring and cost.

Saved in this browser only. See all progress.