AI Agents That Earn Their Keep: A Field Guide for Production

Most AI agent demos look impressive. Most AI agents in production are unrecognisable from the demo: narrower in scope, more constrained, more carefully integrated, with more human review than the marketing suggested. The work between the demo and the production system is where the value lives. This is what that work involves, and where it most often goes wrong.

What an agent actually is

Strip the marketing away. An AI agent is a system that takes a task, decomposes it into steps, calls tools or APIs to execute those steps, and produces an output. It is software with reasoning loops and external action. The interesting question for any business is not whether agents are powerful. They are. The interesting question is what work they should be allowed to do without human review.

The four scope decisions that matter

Most agent projects either succeed or fail at scoping, before code is written. There are four decisions that disproportionately drive the outcome.

1. What can the agent do without human review?

Be explicit. 'The agent drafts a response to a customer enquiry. A human sends.' 'The agent classifies a support ticket and routes. A human resolves.' 'The agent reschedules an appointment within these specific parameters and notifies. A human handles anything outside.' Agents that operate within a tight envelope ship faster, fail more safely, and earn the trust they need to expand the envelope later.

2. What does failure look like, and how do we know?

Failure for a chatbot is rude or confusing output. Failure for a triage agent is a mis-routed urgent request. Failure for a financial decision-support agent is a misleading recommendation that influences a credit decision. Each has a different cost and a different detection mechanism. Define failure before you ship.

3. What is the integration surface?

The agent reads from where, writes to where, and which systems does it touch in between? Agents that pretend to be standalone usually are not. Agents that explicitly model their integration footprint are easier to test, deploy, and govern.

4. Who is the responsible owner in the operation?

Not the AI team. The operational owner (the head of customer service, the operations manager, the credit chief) whose KPI moves when the agent moves. If you cannot name them at scoping, the agent will land badly when it ships.

Architecture in production

We will not pretend there is one architecture for all agent work. There are patterns we use repeatedly because they survive contact with operations.

• ·A reasoning loop that can plan, act, observe, and revise, usually built on Claude or similar reasoning-capable models, with explicit tool definitions for every external action.
• ·Tool calls that are narrow and idempotent. The agent can call 'get customer record by ID' but not 'do whatever you need to'. Each tool has a defined input, output, error mode, and rate limit.
• ·An evaluation harness that runs against a curated set of historical and synthetic cases. Every prompt change, model change, or tool change passes through the harness before deploy.
• ·Decision logs at the level needed by audit and the operational owner. What did the agent do, when, with what inputs, and what was the outcome.
• ·A human-in-the-loop surface for the cases the agent is not allowed to close on its own. Designed in the workflow, not bolted on afterwards.
• ·Drift monitoring: accuracy, latency, refusal rate, escalation rate. Agents drift quietly. The monitoring is what catches it before customers do.

Where agent projects most often go wrong

Three failure modes recur with depressing frequency.

Building the demo, not the system

Demos optimise for the impressive case. Production systems optimise for the average case while handling the edge cases safely. A demo that works on five examples is not evidence that the system will hold up across thousands. The fix is to build the evaluation harness before the agent and use it to drive scope.

Skipping the integration work

An agent that reads and writes to your CRM, ERP, or operational systems lives or dies by the integration. Vendor APIs have rate limits, authentication models, and data shape constraints that look trivial in scoping and become major engineering problems in build. Plan for integration as a primary engineering effort, not a final-week task.

Underestimating adoption

Operations teams are pragmatic. If the agent does not save them time on day one, they will route around it. Adoption is not a launch problem. It is a scoping problem. The scope has to be drawn so the agent saves clear time on the first day, not on the third quarter.

When agents are not the answer

Not every workload needs an agent. If the task is genuinely a single transformation (extract this from this document, summarise this email, generate this draft) a simple model call is cheaper, faster, and easier to govern. Agents earn their complexity when the task involves planning, multiple tool calls, or conditional logic that would be hard to specify in advance. If the work fits in a single prompt, it should stay in a single prompt.