Part 6: Building AI applications
Evaluating AI applications
Learning goal. Build an evaluation set, write automatic checks, use an LLM as a judge carefully, detect hallucinations, and compare versions with A/B tests.
In plain words
AI output varies, and changing a prompt, model or document can quietly break things. An evaluation (eval) is a repeatable test on fixed inputs. An evaluation set holds realistic inputs with expected outcomes: the right category, facts that must appear, the source to cite, or "should refuse". Automatic checks come first: valid JSON, allowed values, correct citation, forbidden text absent. For qualities such as helpfulness or tone, a person or an LLM-as-judge (a model scoring answers against a rubric) can score; check the judge against human scores before trusting it. Hallucination checks test that every claim is supported by the sources. In production, an A/B test shows two versions to different users and compares results, with a statistical test so you do not mistake noise for improvement. Run evals in CI and block releases that fall below thresholds.
- InputEvaluation setrealistic inputs with expected outcomes
- ProcessRun the applicationsame inputs every time
- ProcessScore the outputsautomatic checks, then judge or person
- OutputCompare with thresholdsblock the release if below
An eval is a repeatable test on fixed inputs.
- InputThe right categoryfor example the correct ticket priority
- InputFacts that must appearin the answer
- InputThe source to citethe answer names a known source
- InputShould refusethe correct outcome is to decline
Each case states one kind of expected outcome.
Automatic checks
- Valid JSON, allowed values
- Correct citation
- Forbidden text absent
- Cheap, fast, repeatable
LLM-as-judge
- A model scores against a rubric
- For helpfulness or tone
- Check it against human scores first
Person
- Scores qualities directly
- Used to check the judge
- OutputAnswerthe model's claims
- ProcessCompare with sourcesis each claim backed?
- RiskSupported or notunsupported claims are flagged
A hallucination check tests that every claim is supported by the sources.
Real-world example
A team switched to a cheaper model. The eval showed JSON validity unchanged but priority accuracy on urgent tickets down from 88% to 74%. They kept the old model for that step and used the cheaper one elsewhere: a saving without a quality drop.
JSON validity
- Unchanged with the cheaper model
Priority accuracy on urgent tickets
- Down from 88% to 74%
Decision
- Keep the old model for that step
- Use the cheaper one elsewhere
Numbers are from the example in the text.
How it works
- Collect 50–200 realistic cases, including edge cases and past failures.
- Write expected outcomes and automatic checks.
- Add an LLM judge only where needed; compare it with human labels on a sample.
- Record a baseline; set thresholds; run on every change.
- Read failures, not only scores; add new failures to the set.
- In production, A/B test important changes with enough users.
- InputCollect 50-200 casesinclude edge cases and past failures
- ProcessWrite expected outcomesand automatic checks
- ModelAdd a judge only where neededcompare with human labels
- ProcessBaseline and thresholdsrun on every change
- OutputRead the failuresnot only the scores
Add new failures to the set and run again
Read failures, not only scores.
Before release (eval)
- InputFixed evaluation set
- ProcessChecks and thresholds
- OutputRelease or block
In production (A/B test)
- InputVersion A and version B shown to different users
- ProcessCompare results with a statistical test
- OutputKeep the better version
Related concepts
Eval datasets
An eval dataset is a saved set of questions, each with an expected answer or a rule the answer must follow. Build it from real cases: past tickets, real user questions and failures you have seen. Include hard cases and forbidden cases, where the right outcome is to refuse or to leave something out. Keep a version number, so you know which dataset produced which score. Split it in two: a part you may study and tune on, and a test part you never tune on. If you tune on the test part, the score no longer tells you how the system will do on new questions.
- InputReal casestickets, user questions, past failures
- RiskHard casesunusual, vague or tricky requests
- ProblemForbidden casesthe right outcome is to refuse
Version it, and keep a test part you never tune on.
LLM as judge
An LLM as judge is a second model that scores answers against a written rubric, such as "is every claim supported by the sources?". It is useful for qualities that simple checks cannot see. It needs checks of its own. First, compare its scores with human scores on a sample and confirm they agree. Second, watch for bias, such as favouring longer answers or answers that sound confident. Third, use a different model from the one being judged when possible, because a model may rate its own style too kindly.
- InputWrite a rubricclear scoring rules
- ModelJudge scores a samplea different model when possible
- InputPeople score the same samplethe reference
- ProcessCompare the scoresdo judge and people agree?
- RiskLook for biasfor example favouring long answers
Re-check the judge when the rubric or the judge model changes
Regression tests
A regression is a quality drop caused by a change. A regression test re-runs the eval set whenever a prompt, model, retrieval setting or piece of code changes. Agree a quality bar first, for example a minimum score for each check. If the new result falls below the bar, the release fails. Every failure found in production becomes a new test case, so the same mistake cannot return unnoticed.
- InputA changeprompt, model, retrieval setting or code
- ProcessRe-run the eval setsame cases, same checks
- RiskBelow the agreed bar?compare with the thresholds
- OutputRelease or failfail the release if quality dropped
Prompt A/B tests
A prompt A/B test sends a share of real traffic to a new prompt version (B) while the rest keeps the current one (A). Compare measures agreed in advance, such as task success, user feedback and cost. Split users fairly at random, and collect enough of them before you decide; a small sample can mislead. Guard against harm: start with a small share, and stop B quickly if it does worse on safety or errors.
- ModelPrompt Athe current version, most of the traffic
- ModelPrompt Bthe new version, a small share at first
- OutputCompare agreed measuressuccess, feedback, cost, errors
Start with a small share for B, then grow it if B holds up.
Python example
t1: json=ok priority=ok fact=ok
t2: json=ok priority=ok fact=ok
t3: json=FAIL priority=FAIL fact=FAIL
t4: json=ok priority=FAIL fact=ok
json valid 3/4, priority correct 2/4, all checks pass 2/4An A/B test of two versions with 2,000 users each (10.0% versus 12.0% success):
A 0.100 vs B 0.120: z=2.02, two-sided p=0.043If the versions were truly equal, a gap this big would appear about 4% of the time by chance, so B is very likely better, though only just past the usual 5% line.
json_valid: 1.00
priority_accuracy: 0.85
forbidden_disclosure: 0 # cases where restricted text appeared.NET (C#) example
An LLM-as-judge with a typed verdict, plus a deterministic citation check:
using System.Text.RegularExpressions;
using Microsoft.Extensions.AI;
public record Verdict(bool Supported, int Helpfulness, string Reason); // Helpfulness 1-5
public static class Evals
{
public static bool CitesKnownSource(string answer, ISet<string> sourceIds) =>
Regex.Matches(answer, @"\[([\w#-]+)\]").Select(m => m.Groups[1].Value).Any(sourceIds.Contains);
public static async Task<Verdict?> JudgeAsync(IChatClient judge, string question, string sources, string answer)
{
var response = await judge.GetResponseAsync<Verdict>(
[
new(ChatRole.System, "You grade answers. Supported = every claim is backed by the sources. Helpfulness 1-5. Be strict."),
new(ChatRole.User, $"Question: {question}\nSources:\n{sources}\nAnswer: {answer}"),
], new ChatOptions { Temperature = 0 });
return response.TryGetResult(out var v) ? v : null;
}
}Microsoft also publishes Microsoft.Extensions.AI.Evaluation libraries with ready-made quality evaluators for .NET.
Applied Maths - Statistics
Short, optional reading. Open a topic to see the formula, what it means and two worked examples. The numbers are small, made-up values chosen so you can check them by hand.
Accuracy with a confidence interval (how sure is the score)
Where it is used: reporting a pass rate from a test set of questions without over-claiming.
Accuracy and 95% interval
p = correct ÷ total
standard error = sqrt( p × (1 - p) ÷ n )
interval ≈ p ± 1.96 × standard errorn is the number of test cases. 1.96 gives roughly 95% confidence.
In plain words: a score on a few cases is shaky. The interval shows the range the true pass rate could plausibly sit in. More test cases make it narrower.
Example 1: 50 test cases. 40 pass, so p = 40 ÷ 50 = 0.80. Standard error = sqrt(0.8 × 0.2 ÷ 50) = sqrt(0.0032) = 0.0566. Margin = 1.96 × 0.0566 = 0.111. Interval is about 0.69 to 0.91.
Example 2: 200 test cases. 160 pass, p = 0.80 again. Standard error = sqrt(0.16 ÷ 200) = sqrt(0.0008) = 0.0283. Margin = 1.96 × 0.0283 = 0.055. Interval is about 0.745 to 0.855, much tighter.
Precision and recall (two kinds of mistake)
Where it is used: scoring a checker, such as a judge that flags unsafe or wrong answers.
Precision and recall
precision = TP ÷ (TP + FP)
recall = TP ÷ (TP + FN)TP = true positives (flagged and really bad). FP = false positives (flagged but fine). FN = false negatives (bad but missed).
In plain words: precision asks "when it flags, is it right?". Recall asks "of all the real problems, how many did it catch?".
Example 1: a balanced checker. TP = 8, FP = 2, FN = 4. Precision = 8 ÷ 10 = 0.80. Recall = 8 ÷ 12 = 0.67.
Example 2: flag everything. 100 answers, 12 really bad. The checker flags all 100: TP = 12, FP = 88, FN = 0. Recall = 12 ÷ 12 = 1.00, but precision = 12 ÷ 100 = 0.12. A perfect recall alone can hide a useless checker.
Maths behind it
- Accuracy and agreement rates (statistics)
- Hypothesis tests and p-values (A/B testing)
- Confidence intervals and sample size
- Inter-rater agreement (judge versus humans)
Common mistakes
- Judging a change by a few hand-picked examples.
- Trusting an LLM judge that was never compared with human scores.
- Stopping an A/B test the moment it looks good, which inflates false wins.
- Tuning the prompt on the test part of the eval set, so the score looks better than real life.
- Letting an LLM judge prefer long answers, or judging a model with itself.
- Skipping the regression run for a "small" prompt change, and not adding production failures as new tests.
Weak habits
- A few hand-picked examples
- An LLM judge never compared with humans
- Stopping an A/B test as soon as it looks good
Sound habits
- A realistic evaluation set
- Judge checked against human scores
- Enough users, then a statistical test
Quick check
Why prefer automatic deterministic checks where possible?
Show answer
They are cheap, fast, repeatable and unambiguous; judge models add cost and their own errors.
What does p = 0.043 mean in an A/B test?
Show answer
If there were no real difference, a gap this large would appear about 4.3% of the time by chance.
How do you make an LLM judge trustworthy?
Show answer
Measure its agreement with human scores on a sample before relying on it, and re-check it over time.
Why keep a test part of the eval set that you never tune on?
Show answer
If you tune on it, the score stops showing how the system will do on new questions.
What should happen to a failure found in production?
Show answer
It becomes a new test case, and the regression run fails any release that repeats it.
Completion
Evidence of completion. An evaluation script with thresholds that runs on every change, plus one regression it caught.
Saved in this browser only. See all progress.