AI & ML Quick Path

Part 6: Building AI applications

Evaluating AI applications

Learning goal. Build an evaluation set, write automatic checks, use an LLM as a judge carefully, detect hallucinations, and compare versions with A/B tests.

In plain words

AI output varies, and changing a prompt, model or document can quietly break things. An evaluation (eval) is a repeatable test on fixed inputs. An evaluation set holds realistic inputs with expected outcomes: the right category, facts that must appear, the source to cite, or "should refuse". Automatic checks come first: valid JSON, allowed values, correct citation, forbidden text absent. For qualities such as helpfulness or tone, a person or an LLM-as-judge (a model scoring answers against a rubric) can score; check the judge against human scores before trusting it. Hallucination checks test that every claim is supported by the sources. In production, an A/B test shows two versions to different users and compares results, with a statistical test so you do not mistake noise for improvement. Run evals in CI and block releases that fall below thresholds.

Diagram An eval, step by step
  1. InputEvaluation setrealistic inputs with expected outcomes
  2. ProcessRun the applicationsame inputs every time
  3. ProcessScore the outputsautomatic checks, then judge or person
  4. OutputCompare with thresholdsblock the release if below

An eval is a repeatable test on fixed inputs.

Diagram What an evaluation set can expect
  1. InputThe right categoryfor example the correct ticket priority
  2. InputFacts that must appearin the answer
  3. InputThe source to citethe answer names a known source
  4. InputShould refusethe correct outcome is to decline

Each case states one kind of expected outcome.

Diagram Three ways to score an output

Automatic checks

  • Valid JSON, allowed values
  • Correct citation
  • Forbidden text absent
  • Cheap, fast, repeatable

LLM-as-judge

  • A model scores against a rubric
  • For helpfulness or tone
  • Check it against human scores first

Person

  • Scores qualities directly
  • Used to check the judge
Diagram Hallucination check
  1. OutputAnswerthe model's claims
  2. ProcessCompare with sourcesis each claim backed?
  3. RiskSupported or notunsupported claims are flagged

A hallucination check tests that every claim is supported by the sources.

Real-world example

A team switched to a cheaper model. The eval showed JSON validity unchanged but priority accuracy on urgent tickets down from 88% to 74%. They kept the old model for that step and used the cheaper one elsewhere: a saving without a quality drop.

Diagram What the eval showed for the cheaper model

JSON validity

  • Unchanged with the cheaper model

Priority accuracy on urgent tickets

  • Down from 88% to 74%

Decision

  • Keep the old model for that step
  • Use the cheaper one elsewhere

Numbers are from the example in the text.

How it works

  1. Collect 50–200 realistic cases, including edge cases and past failures.
  2. Write expected outcomes and automatic checks.
  3. Add an LLM judge only where needed; compare it with human labels on a sample.
  4. Record a baseline; set thresholds; run on every change.
  5. Read failures, not only scores; add new failures to the set.
  6. In production, A/B test important changes with enough users.
Diagram The eval cycle
  1. InputCollect 50-200 casesinclude edge cases and past failures
  2. ProcessWrite expected outcomesand automatic checks
  3. ModelAdd a judge only where neededcompare with human labels
  4. ProcessBaseline and thresholdsrun on every change
  5. OutputRead the failuresnot only the scores

Add new failures to the set and run again

Read failures, not only scores.

Diagram Offline eval and online A/B test

Before release (eval)

  1. InputFixed evaluation set
  2. ProcessChecks and thresholds
  3. OutputRelease or block

In production (A/B test)

  1. InputVersion A and version B shown to different users
  2. ProcessCompare results with a statistical test
  3. OutputKeep the better version

Eval datasets

An eval dataset is a saved set of questions, each with an expected answer or a rule the answer must follow. Build it from real cases: past tickets, real user questions and failures you have seen. Include hard cases and forbidden cases, where the right outcome is to refuse or to leave something out. Keep a version number, so you know which dataset produced which score. Split it in two: a part you may study and tune on, and a test part you never tune on. If you tune on the test part, the score no longer tells you how the system will do on new questions.

Diagram Building an eval dataset
  • InputReal casestickets, user questions, past failures
  • RiskHard casesunusual, vague or tricky requests
  • ProblemForbidden casesthe right outcome is to refuse
Data storeVersioned eval dataseta tuning part and a test part

Version it, and keep a test part you never tune on.

LLM as judge

An LLM as judge is a second model that scores answers against a written rubric, such as "is every claim supported by the sources?". It is useful for qualities that simple checks cannot see. It needs checks of its own. First, compare its scores with human scores on a sample and confirm they agree. Second, watch for bias, such as favouring longer answers or answers that sound confident. Third, use a different model from the one being judged when possible, because a model may rate its own style too kindly.

Diagram Checking an LLM judge
  1. InputWrite a rubricclear scoring rules
  2. ModelJudge scores a samplea different model when possible
  3. InputPeople score the same samplethe reference
  4. ProcessCompare the scoresdo judge and people agree?
  5. RiskLook for biasfor example favouring long answers

Re-check the judge when the rubric or the judge model changes

Regression tests

A regression is a quality drop caused by a change. A regression test re-runs the eval set whenever a prompt, model, retrieval setting or piece of code changes. Agree a quality bar first, for example a minimum score for each check. If the new result falls below the bar, the release fails. Every failure found in production becomes a new test case, so the same mistake cannot return unnoticed.

Diagram Regression gate before release
  1. InputA changeprompt, model, retrieval setting or code
  2. ProcessRe-run the eval setsame cases, same checks
  3. RiskBelow the agreed bar?compare with the thresholds
  4. OutputRelease or failfail the release if quality dropped

Prompt A/B tests

A prompt A/B test sends a share of real traffic to a new prompt version (B) while the rest keeps the current one (A). Compare measures agreed in advance, such as task success, user feedback and cost. Split users fairly at random, and collect enough of them before you decide; a small sample can mislead. Guard against harm: start with a small share, and stop B quickly if it does worse on safety or errors.

Diagram Splitting traffic in a prompt A/B test
InputIncoming requestsreal traffic, split at random
  • ModelPrompt Athe current version, most of the traffic
  • ModelPrompt Bthe new version, a small share at first
  • OutputCompare agreed measuressuccess, feedback, cost, errors

Start with a small share for B, then grow it if B holds up.

Python example

Output — Output of applied_basics.py evals (executed)
t1: json=ok priority=ok fact=ok
t2: json=ok priority=ok fact=ok
t3: json=FAIL priority=FAIL fact=FAIL
t4: json=ok priority=FAIL fact=ok
json valid 3/4, priority correct 2/4, all checks pass 2/4

An A/B test of two versions with 2,000 users each (10.0% versus 12.0% success):

Output — Output of math_for_ai.py abtest (executed)
A 0.100 vs B 0.120: z=2.02, two-sided p=0.043

If the versions were truly equal, a gap this big would appear about 4% of the time by chance, so B is very likely better, though only just past the usual 5% line.

YAML — CI thresholds (example)
json_valid: 1.00
priority_accuracy: 0.85
forbidden_disclosure: 0      # cases where restricted text appeared

.NET (C#) example

An LLM-as-judge with a typed verdict, plus a deterministic citation check:

C# — Judge.cs
using System.Text.RegularExpressions;
using Microsoft.Extensions.AI;

public record Verdict(bool Supported, int Helpfulness, string Reason);   // Helpfulness 1-5

public static class Evals
{
    public static bool CitesKnownSource(string answer, ISet<string> sourceIds) =>
        Regex.Matches(answer, @"\[([\w#-]+)\]").Select(m => m.Groups[1].Value).Any(sourceIds.Contains);

    public static async Task<Verdict?> JudgeAsync(IChatClient judge, string question, string sources, string answer)
    {
        var response = await judge.GetResponseAsync<Verdict>(
        [
            new(ChatRole.System, "You grade answers. Supported = every claim is backed by the sources. Helpfulness 1-5. Be strict."),
            new(ChatRole.User, $"Question: {question}\nSources:\n{sources}\nAnswer: {answer}"),
        ], new ChatOptions { Temperature = 0 });
        return response.TryGetResult(out var v) ? v : null;
    }
}

Microsoft also publishes Microsoft.Extensions.AI.Evaluation libraries with ready-made quality evaluators for .NET.

Applied Maths - Statistics

Short, optional reading. Open a topic to see the formula, what it means and two worked examples. The numbers are small, made-up values chosen so you can check them by hand.

Accuracy with a confidence interval (how sure is the score)

Where it is used: reporting a pass rate from a test set of questions without over-claiming.

Accuracy and 95% interval

p = correct ÷ total
standard error = sqrt( p × (1 - p) ÷ n )
interval ≈ p ± 1.96 × standard error

n is the number of test cases. 1.96 gives roughly 95% confidence.

In plain words: a score on a few cases is shaky. The interval shows the range the true pass rate could plausibly sit in. More test cases make it narrower.

Example 1: 50 test cases. 40 pass, so p = 40 ÷ 50 = 0.80. Standard error = sqrt(0.8 × 0.2 ÷ 50) = sqrt(0.0032) = 0.0566. Margin = 1.96 × 0.0566 = 0.111. Interval is about 0.69 to 0.91.

Example 2: 200 test cases. 160 pass, p = 0.80 again. Standard error = sqrt(0.16 ÷ 200) = sqrt(0.0008) = 0.0283. Margin = 1.96 × 0.0283 = 0.055. Interval is about 0.745 to 0.855, much tighter.

Precision and recall (two kinds of mistake)

Where it is used: scoring a checker, such as a judge that flags unsafe or wrong answers.

Precision and recall

precision = TP ÷ (TP + FP)
recall    = TP ÷ (TP + FN)

TP = true positives (flagged and really bad). FP = false positives (flagged but fine). FN = false negatives (bad but missed).

In plain words: precision asks "when it flags, is it right?". Recall asks "of all the real problems, how many did it catch?".

Example 1: a balanced checker. TP = 8, FP = 2, FN = 4. Precision = 8 ÷ 10 = 0.80. Recall = 8 ÷ 12 = 0.67.

Example 2: flag everything. 100 answers, 12 really bad. The checker flags all 100: TP = 12, FP = 88, FN = 0. Recall = 12 ÷ 12 = 1.00, but precision = 12 ÷ 100 = 0.12. A perfect recall alone can hide a useless checker.

Maths behind it

  • Accuracy and agreement rates (statistics)
  • Hypothesis tests and p-values (A/B testing)
  • Confidence intervals and sample size
  • Inter-rater agreement (judge versus humans)

Common mistakes

  • Judging a change by a few hand-picked examples.
  • Trusting an LLM judge that was never compared with human scores.
  • Stopping an A/B test the moment it looks good, which inflates false wins.
  • Tuning the prompt on the test part of the eval set, so the score looks better than real life.
  • Letting an LLM judge prefer long answers, or judging a model with itself.
  • Skipping the regression run for a "small" prompt change, and not adding production failures as new tests.
Diagram Judging a change

Weak habits

  • A few hand-picked examples
  • An LLM judge never compared with humans
  • Stopping an A/B test as soon as it looks good

Sound habits

  • A realistic evaluation set
  • Judge checked against human scores
  • Enough users, then a statistical test

Quick check

  1. Why prefer automatic deterministic checks where possible?

    Show answer

    They are cheap, fast, repeatable and unambiguous; judge models add cost and their own errors.

  2. What does p = 0.043 mean in an A/B test?

    Show answer

    If there were no real difference, a gap this large would appear about 4.3% of the time by chance.

  3. How do you make an LLM judge trustworthy?

    Show answer

    Measure its agreement with human scores on a sample before relying on it, and re-check it over time.

  4. Why keep a test part of the eval set that you never tune on?

    Show answer

    If you tune on it, the score stops showing how the system will do on new questions.

  5. What should happen to a failure found in production?

    Show answer

    It becomes a new test case, and the regression run fails any release that repeats it.

Completion

Evidence of completion. An evaluation script with thresholds that runs on every change, plus one regression it caught.

Saved in this browser only. See all progress.