AI & ML Quick Path

Part 7: Production

"MLOps: versioning, pipelines and safe releases"

Learning goal. Make model training repeatable and releases safe by versioning data, code and models, recording metrics in a registry, and promoting a new model only when it passes a gate.

In plain words

MLOps is DevOps for machine learning: the habits and tools that make models repeatable, traceable and safe to change. Normal software has one moving part, the code. ML has three: code, data and the model produced from them. If any one changes, the result changes. So you version all three: code in Git, data by a snapshot or a content hash, and models in a model registry (a catalogue of every trained model with its metrics, the data it saw and its status such as "candidate", "production" or "archived"). Training runs as a pipeline (a script or workflow that goes from raw data to a saved, tested model with no manual steps), so anyone can rerun it. A new model is released only if it passes a promotion gate: written rules such as "must beat the current production model, the champion, on the same test set by a clear margin, and must pass fairness and speed checks". Safe rollout methods include shadow mode (the new model runs on live traffic but its answers are only logged), canary release (a small share of users get it first) and A/B testing (a planned comparison of business results). Always keep the previous model ready for rollback.

Diagram Three moving parts in ML
  • InputCodeversioned in Git
  • Data storeDataa snapshot or a content hash
  • ModelModelkept in the model registry
OutputThe result you getpredictions and decisions

Change any one and the result changes, so version all three.

Diagram Life of a model in the registry
  1. ModelTrainedmodel saved with metrics and data hash
  2. ProcessCandidatewaiting to be compared
  3. OutputProductionthe champion serving users
  4. Data storeArchivedreplaced, but kept for rollback

The registry records metrics, data seen and status for each model.

Diagram The promotion gate
  1. ModelCandidate modelnewly trained
  2. InputFrozen test setnever used for training
  3. ProcessGate rulesbeat the champion by a clear margin, pass fairness and speed checks
  4. OutputDecisionpromote or reject

Same test set for both models, rules written down in advance.

Diagram Three safe ways to release a new model

Shadow mode

  • New model runs on live traffic
  • Its answers are only logged
  • Users never see them

Canary release

  • A small share of users get the new model first
  • Watch the metrics before widening

A/B testing

  • A planned comparison of business results
  • Two groups, one model each

Real-world example

A lender retrains its credit-risk model every month. The pipeline pulls a dated data snapshot, trains, evaluates and writes everything to the registry. One month, a more complex model scores slightly lower than the champion on the frozen test set. The gate rejects it automatically, and the champion stays in production. Months later an auditor asks why a loan was refused in March; the registry shows exactly which model, data snapshot and threshold were used.

Diagram The lender's monthly cycle
  1. InputDated data snapshotpulled by the pipeline
  2. ProcessTrain and evaluateon the frozen test set
  3. Data storeWrite to the registrymodel, data snapshot, threshold
  4. RiskGatecomplex model scored slightly lower
  5. OutputChampion staysthe new model is rejected

How it works

  1. Freeze a test set that no candidate model ever trains on.
  2. Write the pipeline: load data snapshot, record its hash, prepare features, train, evaluate, save.
  3. Log every run: parameters, metrics, data hash, code commit, and the model file.
  4. Compare the candidate with the champion on the same test set; apply the gate.
  5. If it passes, run it in shadow mode or as a canary; watch business and technical metrics.
  6. Promote it to production, keep the old model for rollback, and record the decision.
  7. Schedule retraining (time-based or triggered by drift, see Monitoring).
Diagram The MLOps pipeline
  1. InputFreeze test setno candidate trains on it
  2. ProcessTrainsnapshot, hash, features
  3. Data storeLog the runparameters, metrics, data hash, commit
  4. ProcessCompare with championapply the gate
  5. RiskShadow or canarywatch the metrics
  6. OutputPromotekeep the old model for rollback

Retrain on a schedule or when drift is detected

  • Feature store: a shared system that computes features once and serves the same values for training (history) and for live predictions (latest values). It prevents training-serving skew, where the live feature is computed differently from the training one.
Diagram Feature store
Data storeFeature storecomputes each feature once
  • ProcessTraininghistorical values
  • OutputLive predictionslatest values

One definition of each feature, used in both places.

  • Data versioning tools keep dataset snapshots next to code versions so a run can be reproduced exactly.
  • CI/CD for ML: automated tests on data (schema, ranges), on the model (gate metrics) and on the service (API tests) every time something changes.
Diagram CI/CD for ML
InputSomething changesdata, code or model
  • ProcessData testsschema and ranges
  • ProcessModel testsgate metrics
  • ProcessService testsAPI tests

Every change triggers all three kinds of test.

Python example

A tiny registry: three candidates are trained on the same data (same hash) and must beat the champion by at least 0.005 AUC (a ranking-quality score from 0.5 to 1, see Evaluation):

Output — Output of quickpath.py registry (executed)
{"model": "v3 logistic regression", "data_sha256": "f87b8c5b834b", "test_auc": 0.877, "decision": "promote"}
{"model": "v4 gradient boosting", "data_sha256": "f87b8c5b834b", "test_auc": 0.874, "decision": "reject (not better than champion)"}
{"model": "v5 gradient boosting, 2 trees", "data_sha256": "f87b8c5b834b", "test_auc": 0.848, "decision": "reject (not better than champion)"}
champion: v3 logistic regression

The lesson in this output: the more advanced model is not automatically better. On this data the simple model wins, and the gate protects production from a downgrade.

With MLflow, a popular open-source tracking tool, the same logging looks like this:

Python — train.py (MLflow; not run here)
import mlflow, mlflow.sklearn
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.metrics import roc_auc_score

mlflow.set_experiment("churn")
with mlflow.start_run():
    params = {"n_estimators": 200, "max_depth": 3}
    model = GradientBoostingClassifier(**params).fit(X_train, y_train)
    auc = roc_auc_score(y_test, model.predict_proba(X_test)[:, 1])
    mlflow.log_params(params)
    mlflow.log_param("data_sha256", data_hash)
    mlflow.log_metric("test_auc", auc)
    mlflow.sklearn.log_model(model, "model", registered_model_name="churn")

.NET (C#) example

ML.NET has no built-in registry, so a .NET team often stores a small manifest next to each model file (or uses MLflow or a cloud registry through its API). This console step trains, evaluates and applies the gate:

C# — TrainAndGate.cs
using System.Security.Cryptography;
using System.Text.Json;
using Microsoft.ML;
using Microsoft.ML.Data;

var ml = new MLContext(seed: 0);
var data = ml.Data.LoadFromTextFile<Customer>("data/churn-2026-10.csv", hasHeader: true, separatorChar: ',');
string dataHash = Convert.ToHexString(SHA256.HashData(File.ReadAllBytes("data/churn-2026-10.csv")))[..12];
var split = ml.Data.TrainTestSplit(data, testFraction: 0.3, seed: 0);

var pipeline = ml.Transforms.Concatenate("Features", "TenureMonths", "MonthlyCharge", "SupportCalls")
    .Append(ml.BinaryClassification.Trainers.FastTree());
var model = pipeline.Fit(split.TrainSet);
double auc = ml.BinaryClassification.Evaluate(model.Transform(split.TestSet)).AreaUnderRocCurve;

var champion = JsonSerializer.Deserialize<Manifest>(File.ReadAllText("models/production.json"))!;
bool promote = auc >= champion.TestAuc + 0.005;
var manifest = new Manifest($"churn-{DateTime.UtcNow:yyyyMMdd-HHmm}", dataHash, auc, promote ? "promote" : "reject");
File.WriteAllText($"models/{manifest.Version}.json", JsonSerializer.Serialize(manifest));
ml.Model.Save(model, data.Schema, $"models/{manifest.Version}.zip");
if (promote) File.Copy($"models/{manifest.Version}.json", "models/production.json", overwrite: true);
Console.WriteLine($"{manifest.Version}: AUC {auc:F3} vs champion {champion.TestAuc:F3} -> {manifest.Decision}");

public sealed record Manifest(string Version, string DataSha256, double TestAuc, string Decision);

public sealed class Customer
{
    [LoadColumn(0)] public float TenureMonths { get; set; }
    [LoadColumn(1)] public float MonthlyCharge { get; set; }
    [LoadColumn(2)] public float SupportCalls { get; set; }
    [LoadColumn(3), ColumnName("Label")] public bool Churned { get; set; }   // trainers look for "Label" by default
}

Run it in a CI pipeline (for example GitHub Actions or Azure DevOps) so training is never a manual, unrepeatable step.

Applied Maths - Statistics

Short, optional reading. Open a topic to see the formula, what it means and two worked examples. The numbers are small, made-up values chosen so you can check them by hand.

Uncertainty of a test score (is a small difference real?)

Where it is used: the release gate, when a new model scores slightly differently from the champion on the frozen test set.

Standard error of an accuracy

standard error = sqrt( p × (1 - p) / n )
rough 95% range = p ± 1.96 × standard error
p = accuracy as a decimal, n = number of test rows

In plain words: a score from a limited test set is a little noisy. The standard error says how much. Differences smaller than the range may be luck.

Example 1: 400 test rows. Accuracy p = 0.90. Standard error = sqrt(0.9 × 0.1 / 400) = sqrt(0.000225) = 0.015. Range = 0.90 ± 1.96 × 0.015 = 0.90 ± 0.029. A challenger at 0.91 is inside that range, so the gain is not proven.

Example 2: 2,500 test rows. Same p = 0.90. Standard error = sqrt(0.09 / 2500) = 0.006. Range = 0.90 ± 1.96 × 0.006 = 0.90 ± 0.012. A bigger test set gives a tighter range.

Comparing two groups in an A/B test (a two-proportion z-score)

Where it is used: a canary or A/B release, comparing the champion and the new model on live traffic.

Two-proportion z-score

p = (successes A + successes B) / (n A + n B)
standard error = sqrt( p × (1 - p) × (1/n A + 1/n B) )
z = (rate B - rate A) / standard error

In plain words: z says how many standard errors apart the two rates are. As a common rule of thumb, a z above about 2 (or below -2) is unlikely to be luck alone.

Example 1: 1,000 users each. A: 100 successes (10%). B: 130 successes (13%). p = 230 / 2000 = 0.115. Standard error = sqrt(0.115 × 0.885 × 0.002) = sqrt(0.00020355) = 0.0143. z = 0.03 / 0.0143 = 2.1. Likely a real difference.

Example 2: only 100 users each. A: 10 successes, B: 13 successes. The rates are the same, 10% and 13%. p = 23 / 200 = 0.115. Standard error = sqrt(0.101775 × 0.02) = 0.0451. z = 0.03 / 0.0451 = 0.66. Too small to tell; keep testing.

Maths behind it

  • Cryptographic hash functions (fingerprinting data)
  • Hypothesis testing and statistical significance (A/B tests)
  • ROC AUC (comparing candidates)

Common mistakes

  • Comparing a new model on a different test set from the champion's.
  • Not recording which data trained a model, so results cannot be reproduced.
  • Promoting a model because it is newer or more complex, not because it is better.
  • No rollback plan.
Diagram Fair and unfair model comparison

Different test sets

  • Candidate and champion scored on different data
  • The comparison means nothing

Same frozen test set

  • Both models scored on identical data
  • The gate decision can be trusted

Quick check

  1. What three things must be versioned in ML, and why?

    Show answer

    Code, data and the model, because changing any one of them changes the result.

  2. What is shadow mode?

    Show answer

    The new model runs on real traffic but its answers are only logged and compared, not shown to users.

  3. In the registry demo, why was gradient boosting rejected?

    Show answer

    Its test AUC (0.874) did not beat the champion (0.877) by the required margin.

Completion

Evidence of completion. A training pipeline that records data hash, parameters and metrics for every run, plus a promotion rule that rejected at least one candidate.

Saved in this browser only. See all progress.