Serving a model as an API
Learning goal. Turn a saved model into a web API that loads the model once, validates input, returns a versioned answer and reports its health, in Python and in ASP.NET Core.
In plain words
A trained model sitting in a file helps nobody. Serving means putting it behind an API (a web address other programs can call) so a website, a mobile app or another service can send it data and get a prediction back. A good prediction API does five things: it loads the model once when the service starts (loading it on every request is slow), it checks the input and rejects bad requests with a clear error, it returns the model version with every answer so you can trace which model made which decision, it offers a health check so the hosting platform knows the service is alive, and it is measured for speed. There are two serving styles. Online (real-time) serving answers one request in milliseconds, for example a fraud check during payment. Batch serving scores millions of rows on a schedule, for example a nightly churn score for every customer. Pick batch whenever the answer is not needed immediately, because it is simpler and cheaper.
- InputClient appa website, mobile app or other service
- ProcessAPI servicechecks the input and rejects bad requests
- ModelModel in memoryloaded once at start-up
- OutputAnswerprediction plus the model version
The model is already in memory when the request arrives.
- ProcessLoads the model onceat start-up, not on every request
- RiskChecks the inputclear error for bad requests
- OutputReturns the model versionso every decision can be traced
- OutputOffers a health checkso the platform knows it is alive
- OutputIs measuredmedian (p50) and slow tail (p95)
Five jobs, one service.
Online (real-time)
- Answers one request at a time
- Needs an answer in milliseconds
- Example: fraud check during payment
Batch
- Scores millions of rows on a schedule
- Nobody waits for the answer
- Example: nightly churn score for every customer
- Simpler and cheaper, so pick it when you can
Real-world example
A telecom company trains a churn model. The customer-care screen calls POST /predict when an agent opens a customer record and shows "High churn risk (0.93), model churn-logreg-v3". Every night a batch job also scores all customers so the retention team gets a ranked call list each morning. Both use the same saved model file.
- OutputOnline: POST /predictagent opens a customer record
- ProcessNightly batch jobscores all customers
- OutputRanked call listready for the retention team each morning
Same model file, different timing.
How it works
- Train and save the model (for example
model.joblibin Python ormodel.zipin ML.NET), together with its version name. - Write a small web service. At start-up, load the model into memory once.
- Define the input contract: field names, types and allowed ranges. Reject anything else with HTTP 400.
- In the
/predicthandler, turn the input into the same feature layout used in training, call the model and return the result plusmodel_version. - Add
/healththat returns{"status": "ok"}so the platform can restart a broken instance. - Measure latency: send many requests and record the median (p50) and the slow tail (p95). Users feel the tail.
- Put it in a container and deploy it (see Production architecture).
Once, at start-up
- ModelLoad the saved modelinto memory
- ProcessRead the version namefor example churn-logreg-v3
- OutputOpen /healthstatus ok
For every request
- RiskValidate the inputwrong type gives HTTP 400
- ProcessBuild the featuressame layout as training
- ModelCall the modelalready in memory
- OutputReturn result and versionthe answer
Do the slow work once, the quick work every time.
Python example
This demo starts a real HTTP server with Python's standard library, loads a churn model once, and calls it:
GET /health -> 200 {'status': 'ok', 'model_version': 'churn-logreg-v3'}
POST /predict -> 200 {'churn_probability': 0.928, 'model_version': 'churn-logreg-v3'}
POST /predict -> 400 {'error': 'need tenure_months, monthly_charge, support_calls as numbers'}
200 requests on this machine: p50 0.9 ms p95 1.1 msThe second call is a new customer with 3 months of tenure, a high bill and 6 support calls, so the risk is high. The third call sends "three" instead of a number, and the service refuses it instead of crashing. The latency is tiny because this is a small model on the same machine; over a network add the network time.
In real projects you write the same thing with FastAPI, which validates input for you:
import joblib
from fastapi import FastAPI
from pydantic import BaseModel, Field
app = FastAPI()
model = joblib.load("model.joblib") # once, at start-up
MODEL_VERSION = "churn-logreg-v3"
class Customer(BaseModel):
tenure_months: float = Field(ge=0, le=600)
monthly_charge: float = Field(ge=0)
support_calls: int = Field(ge=0, le=100)
@app.get("/health")
def health():
return {"status": "ok", "model_version": MODEL_VERSION}
@app.post("/predict")
def predict(c: Customer): # invalid input -> automatic 422 error
p = model.predict_proba([[c.tenure_months, c.monthly_charge, c.support_calls]])[0, 1]
return {"churn_probability": round(float(p), 3), "model_version": MODEL_VERSION}
# run: uvicorn serve:app --port 8000.NET (C#) example
In ASP.NET Core, PredictionEnginePool (package Microsoft.Extensions.ML) loads an ML.NET model once and is safe to use from many requests at the same time. A single PredictionEngine is not thread-safe, so never share one across requests.
using Microsoft.Extensions.ML;
using Microsoft.ML.Data;
var builder = WebApplication.CreateBuilder(args);
builder.Services.AddPredictionEnginePool<Customer, ChurnPrediction>()
.FromFile(modelName: "churn", filePath: "model.zip", watchForChanges: true);
builder.Services.AddHealthChecks();
var app = builder.Build();
const string ModelVersion = "churn-fasttree-v3";
app.MapHealthChecks("/health");
app.MapPost("/predict", (Customer c, PredictionEnginePool<Customer, ChurnPrediction> pool) =>
{
if (c.TenureMonths is < 0 or > 600 || c.MonthlyCharge < 0 || c.SupportCalls < 0)
return Results.BadRequest(new { error = "values out of range" });
var p = pool.Predict(modelName: "churn", example: c);
return Results.Ok(new { churnProbability = Math.Round(p.Probability, 3), modelVersion = ModelVersion });
});
app.Run();
public sealed class Customer
{
public float TenureMonths { get; set; }
public float MonthlyCharge { get; set; }
public float SupportCalls { get; set; }
}
public sealed class ChurnPrediction
{
[ColumnName("PredictedLabel")] public bool Churn { get; set; }
public float Probability { get; set; }
}watchForChanges: true reloads the model when the file is replaced, so you can ship a new model without restarting. The model must have been trained with the same column names as Customer.
Applied Maths - Statistics
Short, optional reading. Open a topic to see the formula, what it means and two worked examples. The numbers are small, made-up values chosen so you can check them by hand.
Percentiles (p50 and p95 latency)
Where it is used: reporting how fast the POST /predict endpoint answers, instead of one average.
Percentile (nearest rank)
sort the response times from smallest to largest
pK = the value at rank ceil(K ÷ 100 × n) (n = number of requests, ceil = round up)In plain words: p50 is the typical request. p95 is the time that 95% of requests beat, so it shows the slow ones.
Example 1: p50 and p95. Ten response times in ms, sorted: 12, 13, 13, 14, 15, 15, 16, 18, 25, 240. p50 rank = ceil(0.5 × 10) = 5, so p50 = 15 ms. p95 rank = ceil(0.95 × 10) = 10, so p95 = 240 ms. The mean is 381 ÷ 10 = 38.1 ms, which describes neither.
Example 2: p90 can miss the tail. On the same list, p90 rank = ceil(0.9 × 10) = 9, so p90 = 25 ms. One slow request in ten is invisible at p90 but visible at p95. Choose the percentile that matches your promise to users.
Utilisation and waiting (why a busy server slows sharply)
Where it is used: deciding how much load one copy of the API can take before latency becomes a problem.
Utilisation and time in system (simple one-server queue)
utilisation = arrival rate ÷ service rate
average time in system = 1 ÷ (service rate - arrival rate)
service rate = requests the server can finish per second; arrival rate must be lowerIn plain words: as arrivals get close to what the server can finish, the wait grows very quickly. This is a simplified model that assumes random arrivals.
Example 1: half busy. The server finishes 50 requests per second; 25 arrive per second. Utilisation = 25 ÷ 50 = 50%. Time = 1 ÷ (50 - 25) = 0.04 s = 40 ms.
Example 2: nearly full. 49 arrive per second. Utilisation = 49 ÷ 50 = 98%. Time = 1 ÷ (50 - 49) = 1 s, which is 25 times slower. Add capacity well before this point.
Maths behind it
- Percentiles (p50, p95) for latency
- Queueing theory (why latency rises sharply as a server gets busy)
Common mistakes
- Loading the model inside the request handler, so every request pays the loading cost.
- Training with one feature order and serving with another; the model silently gives nonsense.
- Reporting only average latency; one slow request in twenty is what users remember.
- Choosing real-time serving when a nightly batch would do.
Inside the request handler
- Every request pays the loading cost
- Slow for every user
Once at start-up
- Loaded a single time
- Each request only runs the prediction
Quick check
Why must the API return the model version with each prediction?
Show answer
So every decision can be traced to the exact model that made it, which you need for debugging, audits and rollbacks.
When is batch serving better than online serving?
Show answer
When the answer is not needed immediately (for example a nightly score for every customer); it is simpler, cheaper and easier to monitor.
Why use PredictionEnginePool instead of one PredictionEngine in ASP.NET Core?
Show answer
A PredictionEngine is not thread-safe; the pool gives each concurrent request a safe engine and loads the model once.
Completion
Evidence of completion. A running prediction API with /health and /predict, input validation that returns 400 for bad input, and a latency measurement (p50 and p95) from your machine.
Saved in this browser only. See all progress.