Part 6: Building AI applications
Tools and agents with human approval
Learning goal. Let a model call tools safely, design an agent workflow with explicit state, permission checks, human approval, retries, idempotency and an audit trail, in Python and C#.
In plain words
Tool calling lets a model ask your application to run a function: you describe each tool (name, purpose, inputs), and instead of text the model returns "call get_order_status with order A-1001". Your code checks the request, checks the user is allowed, runs it and returns the result; the model never executes anything itself. An agent is a system where the model chooses steps and tools within limits that your code sets, keeping state between steps. For business processes, the safest design is a mostly fixed workflow with model-powered steps (understand the request, draft a message), not an open-ended loop. Actions that change the world, especially money, need permission checks, human approval, idempotency (repeating a request does not repeat the refund), retries for temporary failures, and an audit log. The Model Context Protocol (MCP) is an open standard for exposing tools and data to AI applications (see Frameworks).
- ModelModel asks for a toolFor example call get_order_status with order A-1001
- RiskYour code checks the requestValid inputs? Is the user allowed?
- ProcessYour code runs the toolThe model never executes anything itself
- OutputResult goes back to the modelIt writes the reply
Repeat while the model asks for more tools
The model only asks. Your code checks, runs the tool and returns the result.
Mostly fixed workflow
- Safest for business processes
- Model steps: understand, draft a message
Open-ended loop
- The model chooses every step
- Harder to control and test
- RiskPermission checkIs this user allowed?
- RiskHuman approvalA person approves or declines
- ProcessIdempotencyRepeating a request does not repeat the refund
- ProcessRetriesFor temporary failures
- Data storeAudit logEvery step and decision recorded
Real-world example
The Klarna case describes an assistant handling customer-service chats including refunds and returns. A typical safe design for such a system: the assistant checks the order and the policy, drafts the refund, and money moves only after rules or a person approve it, with every step logged.
How it works
- Understand: the model extracts intent and order id (structured output).
- Verify: load the order with the user's identity from the session; reject if not theirs.
- Policy: reject if outside the 30-day window.
- Draft: create the proposed action, such as refund 59.90.
- Approve: pause until a person approves or declines; the pause must survive restarts.
- Execute: call the payment service with an idempotency key; retry temporary failures.
- Audit: record every step and decision.
- ModelUnderstandModel extracts intent and order id
- RiskVerifyLoad the order with the session identity; reject if not theirs
- RiskPolicyReject outside the 30-day window
- ProcessDraftPropose the action, such as refund 59.90
- RiskApprovePause until a person approves or declines
- ProcessExecutePayment call with an idempotency key; retry temporary failures
- Data storeAuditRecord every step and decision
Steps 1 to 4 can stop the request. Step 5 waits for a person and must survive restarts.
- OutputRefundedHappy path, or a flaky service that recovers
- RiskDeclined by reviewerPending; customer will be contacted
- ProblemRejectedNot the owner, or outside the window
- ProblemFailedPayment service down; queued for manual follow-up
Results from the executed run of the Python workflow.
Related concepts
- Multi-agent systems: several agents with different roles (for example a planner, a researcher and a reviewer) passing work between them. They can help with large tasks but add cost, latency and new failure modes. Start with one agent and a clear workflow; split only when one agent clearly struggles.
- ModelResearcherFinds the facts
- ModelReviewerChecks the result
Helpful for large tasks, but adds cost, latency and new failure modes.
- Planning: the agent writes a list of steps before acting, then follows and updates it.
- ModelWrite a list of stepsBefore acting
- ProcessDo the next stepUse tools if needed
- ModelUpdate the listBased on what happened
Repeat until the plan is done
- Memory: short-term memory is the conversation so far; long-term memory is facts saved to a store and retrieved later (often with RAG).
Short-term
- The conversation so far
Long-term
- Facts saved to a store
- Retrieved later, often with RAG
- AI coding assistants (code completion and coding agents in the editor) are agents specialised for software: their tools read files, run tests and propose changes for a developer to review.
The sections below go deeper into each idea, in the order you will meet them when you build an agent.
The agent loop
An agent runs in a loop. It observes what it knows so far (the goal, the history, the latest result). It decides what to do next. It calls a tool through your code, then reads the result, and goes round again. The loop ends when the model says the goal is done, or when your code hits a limit such as a maximum number of steps. The limit is not optional: without it, a confused agent can loop for ever and spend money.
- InputObserveRead the goal, the history and the last result
- ModelDecideThe model picks the next step or says it is done
- ProcessCall a toolYour code checks the request and runs it
- Data storeRead the resultThe result is added to the history
Repeat until done or a limit is hit
Your code owns the loop and the stop conditions.
- OutputModel says it is doneA final answer, no more tool requests
- RiskStep limit reachedYour code stops the run after a set number of steps
- RiskBudget or time limit reachedStop when the allowed cost or time is used
- OutputReport to the userGive the answer, or say clearly why it stopped
Designing tools
The model only sees your tool descriptions, so they are part of your design. Give each tool a clear name and a short description that says when to use it. Keep the purpose narrow: one tool, one job. Use typed inputs (for example an order id as text, an amount as a number) and validate them in code. Return helpful errors that tell the model what to fix, such as "order not found; check the id". Apply least privilege: a tool gets only the access its job needs. Make tools idempotent where possible, so a repeat does not repeat the effect. Return only what the model needs, because every extra field uses context and may leak data.
- InputName and descriptionClear, says when to use it
- RiskTyped and validated inputsWrong types are rejected before anything runs
- RiskLeast privilegeOnly the access this one job needs
- ProcessIdempotent actionA repeat does not repeat the effect
- OutputSmall, helpful resultOnly what the model needs, with clear errors
Each layer is a design decision you make.
Too broad
- run_anything(command)
- Can read or change any data
- Errors are vague
Narrow
- get_order_status(order_id)
- Reads one customer's own orders
- Errors say what to fix
MCP, the Model Context Protocol
MCP is one standard way to plug tools and data into many AI applications. A tool is written once as an MCP server, and any AI app that speaks MCP can use it, instead of each app needing its own custom connector. The detail, with examples, is in Frameworks. The design rules above still apply to every MCP tool.
- ModelAI app AUses the tools through MCP
- ModelAI app BUses the same tools through MCP
Planning
Planning means breaking a goal into steps. There are two styles. In plan then act, the agent writes the whole plan first and then follows it. This is easy to review, but the plan can go out of date. In act step by step, the agent decides each step after seeing the last result. This adapts well but can drift. In both styles, when a step fails the agent should re-plan: change the remaining steps instead of repeating the failed one blindly.
Plan then act
- Write all steps first
- Easy for a person to review
- Can go stale as facts change
Act step by step
- Decide after each result
- Adapts to new facts
- Can drift from the goal
- ProcessRun a stepCall the tool
- ProblemStep failsFor example an order is not found
- ModelRead the errorWork out what changed
- ModelChange the remaining planTry another route, or ask a person
Memory
Short-term memory is what sits inside the context window (the text the model can see in one call): the conversation and a scratchpad of notes the agent writes for itself. It disappears when the run ends and is limited by the window size. Long-term memory is facts saved outside the model, for example in a database, and retrieved later when relevant. There are risks. A memory can be stale (no longer true) or simply wrong, and the agent will trust it. Memories can also hold private data, so decide what may be stored, who can read it, and how a user can delete it.
- ProcessContext windowConversation and scratchpad for this run
- ProcessRetrieval stepFinds saved facts that match the task
- Data storeMemory storeLong-term facts saved outside the model
Short-term memory is in the context window. Long-term memory is outside it.
Multi-agent systems
In a multi-agent system, several agents with different roles work on one task and pass work between them. The roles can be fixed, for example a planner, a researcher and a reviewer. Each agent has its own instructions and often its own tools. This can help on large tasks, but every extra agent adds cost, delay and a new place for errors to hide. Start with one agent, and split only when one agent clearly struggles.
Planner
- Split the goal into tasks
- Hand each task to a worker
Researcher
- Find the facts for each task
- Report what was found
Reviewer
- Check the result before it goes out
- Send it back if it is wrong
Subagents
A subagent is a helper that a main agent starts for one focused sub-task. The subagent has its own fresh context and its own tools. When it finishes, it returns only a short result, not its whole working. This keeps the main agent's context small and clean. It differs from plain multi-agent: here there is a clear parent that delegates and receives the result, while in a general multi-agent system the agents may be peers that pass work around.
- OutputSubagent 2 resultA short summary
- ModelMain agentCombines the results and continues
Handoffs
A handoff is when one agent passes the task to a more suitable agent, or to a person. The sender must pass the context the next one needs: the goal, what was done, what is still open, and why it is being handed over. Without this, the receiver starts blind and the user has to repeat everything. A handoff to a person is the same idea as human approval, which is covered next.
- RiskAgent cannot finish the taskOut of scope, unsure, or too risky
- ProcessPackage the contextGoal, steps done, open questions, reason
- OutputPass to a better agent or a personThe receiver can carry on without asking again
Human in the loop
Human in the loop means a person approves risky actions before they happen, such as paying money or deleting data, as in the refund workflow above. Your code decides which actions need approval, never the model. The person should see what will happen and why. The waiting state must be saved, so that the approval can come hours later. A request for approval is one kind of handoff, and the person's answer goes back into the run.
Agent
- Prepare the action and explain why
- Pause and wait
Your code
- Mark the action as needing approval
- Run it only after a yes
Person
- Read the request
- Approve or decline
Sandboxing
A sandbox is an isolated environment where agent-generated code or tool actions run with limited files, network access, time and permissions. If the agent makes a mistake, or is tricked, the damage stays inside the sandbox and cannot reach real systems. Give the sandbox only the files it needs, block or restrict the network, set a time limit, and throw it away afterwards.
Computer use
Computer use means a model looks at screenshots and controls the mouse and keyboard to operate ordinary software, the way a person would. It is useful when a program has no API. It is slower and less reliable than an API, because it has to read the screen and click the right place. So use an API when one exists. Because the agent can click anything it can see, run it in a sandbox and ask for approval before risky actions.
- InputTake a screenshotSee what is on the screen
- ModelModel decidesChoose a click, a key press or scrolling
- ProcessClick or typeMouse and keyboard actions
Repeat until the task is done
API, when one exists
- Faster
- More reliable
- Clear inputs and outputs
Computer use
- Slower
- Can misread the screen
- Needs a sandbox and approval
Skills
A skill is a reusable package of instructions, and optionally scripts or reference files, that an agent loads only when the task needs it. The agent sees just a short name and description of each skill at first. It opens the full skill when a task matches. This keeps the context small, even when the agent has many skills.
- InputShort name and descriptionAlways visible to the agent
- ProcessFull instructionsLoaded only when the task matches
- Data storeScripts and reference filesUsed only if the instructions call for them
Only the short description is always in context.
Durable workflows
A durable workflow is a long-running agent or workflow run that saves its state at each step. If the program crashes, restarts, or waits days for a human approval, it can resume from the last saved step instead of starting over. Steps may run more than once after a crash, so use retries for temporary failures and make every step idempotent, so a repeat does no harm.
- ProcessRun a stepDo the work
- Data storeSave the stateWrite progress to storage
- RiskCrash, restart or long waitThe process may stop at any time
- OutputResume from the last saved stepDo not start over
Guardrails for agents
Guardrails are limits that your code enforces, not requests in a prompt. Examples: only the tools the task needs, a budget for cost, a maximum number of steps, and checks on the output before it is used. Details are in Security.
- RiskTool limitsOnly the tools this task needs
- RiskBudgetA cap on cost and time
- RiskStep limitA maximum number of loop rounds
- RiskOutput checksCheck the result before it is used
Prompt injection and agents
Prompt injection means text that tries to give the model new orders, for example hidden in a web page or an email. It is covered in Security. It is riskier for agents, because an agent reads untrusted content and also has tools that can act. A hidden instruction in a document could make the agent send data or change something. Treat all fetched content as data, and limit what the tools can do.
- InputAgent reads untrusted contentA web page, email or file
- ProblemHidden instruction inside itText that looks like an order
- RiskAgent has a powerful toolIt can send, change or delete
- OutputLimits stop the harmLeast privilege, approval and a sandbox
Python example
The whole workflow in plain Python, run against every path:
happy path -> refunded 59.90 as R-0001
reviewer declines -> pending: refund declined by reviewer; customer will be contacted
not the owner -> rejected: you can only return your own orders
outside window -> rejected: delivered 90 days ago (limit 30)
flaky service recovers -> refunded 59.90 as R-0001
service down -> failed: payment service unavailable; queued for manual follow-up
duplicate request -> refunded 59.90 as R-0001 / refunded 59.90 as R-0001; refunds issued: 1decision = approver(order_id, draft) # human in the loop
if not decision:
return "pending: refund declined by reviewer; customer will be contacted"
refund_id = with_retries(lambda: payments.refund(order_id, draft["amount"], f"refund:{order_id}")) # idempotency keyIn LangGraph (examples/agent-returns/langgraph_returns.py) the approval step calls interrupt(...); a checkpointer saves the state, and Command(resume=True) continues it later, even after a restart.
.NET (C#) example
Microsoft Agent Framework: read-only tools run automatically, the refund tool requires approval (packages for Agent Framework and Microsoft.Extensions.AI; API names as documented by Microsoft):
// The approval types are marked experimental in some Microsoft.Extensions.AI versions (diagnostic MEAI001),
// and newer versions may rename them; check the names for the package version you install.
#pragma warning disable MEAI001
using System.ComponentModel;
using Microsoft.Agents.AI;
using Microsoft.Extensions.AI;
[Description("Get status of one of the current customer's orders.")]
static string GetOrderStatus(string orderId) => Orders.StatusFor(Session.CustomerId, orderId); // identity from the session
[Description("Refund an order. Requires human approval.")]
static string RefundOrder(string orderId) => Payments.Refund(orderId, idempotencyKey: $"refund:{orderId}");
AIAgent agent = chatClient.AsAIAgent(
instructions: "You help customers with returns. Use tools for facts. Never promise a refund before it is approved.",
tools: [AIFunctionFactory.Create(GetOrderStatus), new ApprovalRequiredAIFunction(AIFunctionFactory.Create(RefundOrder))]);
AgentSession session = await agent.CreateSessionAsync();
AgentResponse response = await agent.RunAsync("Please refund order A-1001", session);
foreach (var request in response.Messages.SelectMany(m => m.Contents).OfType<FunctionApprovalRequestContent>())
{
bool approved = await ReviewQueue.AskAsync(request.FunctionCall.Name, request.FunctionCall.Arguments); // your approval UI
Console.WriteLine(await agent.RunAsync(new ChatMessage(ChatRole.User, [request.CreateResponse(approved)]), session));
}Applied Maths - Statistics
Short, optional reading. Open a topic to see the formula, what it means and two worked examples. The numbers are small, made-up values chosen so you can check them by hand.
Success across a chain of steps (why long agents fail more)
Where it is used: estimating how often an agent finishes a multi-step task when each tool call can go wrong.
Chain success
chance all steps work = p × p × ... × p = p^np is the chance that one step works. n is the number of steps. This assumes steps fail independently.
In plain words: small error rates add up over many steps.
Example 1: 5 steps. p = 0.95. 0.95^5 = 0.95 × 0.95 × 0.95 × 0.95 × 0.95 = 0.774, so about 77%.
Example 2: 10 steps. p = 0.90. 0.9^10 = 0.349, so only about 35%. This is why agents need checks and human approval.
Retries (at least one success)
Where it is used: deciding how many times an agent may retry a failing tool call.
Chance of success within n tries
chance = 1 - (1 - p)^n
expected cost = cost per try × expected number of triesp is the chance a single try works. The tries are assumed independent.
In plain words: retries help a lot when failures are random, but each try costs time and money.
Example 1: reliable tool. p = 0.7, n = 3. All three fail with chance 0.3^3 = 0.027. Chance of success = 1 - 0.027 = 0.973.
Example 2: weak tool. p = 0.3, n = 3. All fail with chance 0.7^3 = 0.343. Chance of success = 1 - 0.343 = 0.657. Retrying does not fix a tool that is mostly broken.
Maths behind it
- State machines and workflow graphs (computer science)
- Exponential backoff for retries
- Idempotency (distributed systems)
- No special maths: agents are mostly careful software engineering
Common mistakes
- Letting the model decide whether approval is needed; enforce it in code.
- Taking the customer id from the model's tool arguments instead of the logged-in session.
- Retrying permission or policy rejections as if they were temporary errors.
- Keeping pending approvals only in memory, so a restart loses them.
- Giving an agent a tool with too much power (for example "run any command") instead of narrow tools with least privilege.
- Having no step, time or cost limit, so a confused agent loops for ever.
- Running agent-generated code, or computer use, outside a sandbox.
Wrong
- The model decides whether approval is needed
- Customer id taken from model arguments
- Policy rejections are retried
- Pending approvals kept in memory only
Right
- Code enforces approval
- Customer id comes from the logged-in session
- Only temporary errors are retried
- Pending approvals survive a restart
Quick check
Who actually runs a tool the model asks for?
Show answer
Your application, after validating the request and checking permissions.
Why does the refund call need an idempotency key?
Show answer
Retries and duplicate requests could otherwise issue more than one refund.
Which failures should be retried automatically?
Show answer
Temporary ones such as timeouts or "service unavailable", a limited number of times with increasing waits.
Why run agent-generated code in a sandbox?
Show answer
So that a mistake, or a trick by hidden instructions, cannot damage real files, accounts or networks.
How is a subagent different from plain multi-agent work?
Show answer
A subagent is started by a main agent for one sub-task, uses its own context and returns only a short result.
Completion
Evidence of completion. A refund workflow (in either language) passing scripted tests for success, decline, wrong owner, outside policy, service failure and duplicate requests.
Saved in this browser only. See all progress.