Your agent worked in the demo. It read the ticket, called three APIs, and posted a clean summary. Then you shipped it. A week later it emailed the same customer twice, burned a day’s token budget on a retry loop, and handed your frontend a payload it could not parse.
That gap between a working prototype and a reliable agent is where most teams get stuck. The model is rarely the culprit. The trouble sits in the part that gets treated as plumbing: the API calls the agent makes on the way to an answer. An agent is a loop of tool calls, and every tool call is an HTTP request that can fail, throttle, time out, or return something you did not expect. Skip testing those calls the way you test any production API, and your agent is one bad response away from an incident.
Here is the reassuring part: agent reliability is testable. You do not have to trust the model to behave. You exercise the failure paths on purpose, before your users find them for you. This guide sorts agent failures into five modes and shows how to catch each one. The work happens at the API boundary, so you need a platform you can point at the agent’s dependencies to design the contract, mock the failures, and check what comes back. Apidog covers that job, and it runs through the examples below.
Agents fail at the API boundary, not in the prompt
When an agent misbehaves in production, the instinct is to edit the prompt. Sometimes that helps. More often the failure has nothing to do with wording. The agent asked a real API for something, and the response was slow, malformed, rate-limited, or a different shape than the agent expected. The model then reasoned over bad input and did something confident and wrong.
Look at what a single agent step involves. The model picks a tool. Your code turns that choice into an HTTP request. An external service answers. Your code feeds the result back to the model. Four handoffs, and three of them are ordinary API integration, not machine learning. That is good news, because API integration is a solved testing problem. You already know how to mock a slow endpoint or assert on a JSON schema. Agents raise the stakes, because the model acts on whatever it receives instead of throwing a clean exception.
So the reliability question is not “is the model smart enough.” It is “have I tested every way the agent’s API calls can go sideways.” Five modes cover most of it.
Failure mode 1: tool calls that drift from the contract
The most common agent failure is a tool call that does not match the API it is calling. The model invents a parameter, drops a required field, sends a string where the schema wants an integer, or calls the right endpoint with arguments that make no sense. Say a booking agent calls POST /reservations with guests: "two" instead of guests: 2. The API returns a 400, or worse, a 200 with an error buried in the body, and the agent keeps going as if it succeeded.
You catch this by testing the tool call as a contract. Define the schema for each tool the agent can invoke, then assert that the outgoing request matches: required fields present, types correct, enums valid. When the agent produces a call that breaks the contract, you want that to fail loudly in a test, not quietly in production. Our walkthrough on testing an AI agent’s tool calls goes deep on this, and the broader method for testing agents that call your APIs covers the setup end to end.
The practical move: capture the tool schemas your agent uses, load them into Apidog, and run the agent’s real tool calls against those definitions. Mismatches surface as validation failures naming the exact field that broke.
Failure mode 2: upstream errors and rate limits
Every external call the agent makes can return a 429, a 500, or nothing at all before a timeout. A well-built agent handles these with retries and backoff. A fragile one either gives up on the first error or, more dangerously, retries so hard that it triggers more throttling and spins in a loop that drains your budget. The most-asked question in the Anthropic SDK discussion board is literally about patterns for agent error recovery, which tells you how common this pain is.
You cannot test recovery against a healthy API, because a healthy API never returns the errors you need to handle. This is where mocking earns its keep. Stand up a mock of the agent’s dependency and program a sequence: a 429 with a Retry-After header, then a 500, then a success. Now watch what your agent does. Does it back off with jitter? Does it respect the header? Does it give up gracefully after a sensible number of attempts, or open a circuit breaker so it stops hammering a service that is clearly down? And if a retried action is not idempotent, does a repeat double-charge or double-send? An idempotency key is what makes a retry safe to repeat.
Rate limits deserve their own rehearsal. If you have not seen how your agent behaves when a provider throttles it, read our guide on what a rate-limit-exceeded response means and then simulate it. The dedicated guide on AI agent error recovery covers retry, timeout, backoff, and circuit-breaker patterns in full.
Failure mode 3: non-deterministic output
Set temperature to zero and you still will not get byte-identical output across runs. Developers rediscover this constantly; there is a long vLLM thread about seeds and temperature not being enough for reproducibility. Hardware, batching, and provider-side changes all introduce variation. If your tests assert on exact strings, they turn flaky, and flaky tests get ignored, which is worse than no tests. Our rundown of what causes flaky tests applies directly here.
The fix is to assert on structure and meaning, not on exact text. Check that the response validates against a JSON schema. Check that the tool call has the right shape and the right target. Check that a numeric answer falls inside a sensible range. Check that required keys exist and that forbidden fields are absent. A test that says “the reply contains a total between 0 and the cart value” survives the model’s natural variation while still catching a real regression. The guide on testing non-deterministic AI agents lays out the full set of strategies, and the piece on how agent memory works shows why state makes this harder.
Failure mode 4: runaway cost
Agents loop, and loops cost money. A single stuck agent that retries a failing call a few thousand times can turn a small bill into a large one overnight. One field report in the SDK discussions described dropping an agent’s cost from 500 dollars a month to 80 without losing quality, which shows both how fast cost creeps and how much slack usually hides in the design.
Cost is a reliability problem, not only a finance one, because the bugs that waste money (loops, redundant calls, oversized context) also make the agent slow and unpredictable. Track tokens per run, cap the budget per task, and cache what you can. For the command-line side of this, our guide on reducing agent token costs has concrete levers. When you test recovery paths against a mock, watch the call count too. An agent that succeeds but makes forty calls to get there is a cost incident waiting to happen.
Failure mode 5: missing guardrails
The failures that hurt most are the ones where the agent does exactly what it was told and the result is still bad. It sends the email, deletes the record, or places the order, because nothing stood between the model’s decision and the live action. The SDK boards have a memorable thread built around an agent that fired off a three in the morning email to someone’s boss. Funny once, expensive twice.
Guardrails are the seatbelt. Put an allowlist on the actions an agent can take without approval. Gate destructive or irreversible calls behind a human confirmation. Give the agent a dry-run mode that describes what it would do without doing it. Then test that the guardrail holds: mock the side-effecting endpoint, run the agent, and assert that it hits the confirmation path rather than the live action. Security guidance like the OWASP Top 10 for LLM applications is a solid checklist for what to protect. The guide on AI agent guardrails covers approval gates and blast-radius control in depth.
How to structure an agent test
The five modes share one testing shape, and you can reuse it:
- Capture the tool schemas your agent can call, so you have a contract to assert against.
- Mock each dependency so you control timing, status codes, and bodies, and so you avoid real side effects.
- Drive the agent through the scenario, including the unhappy paths a live API will not produce on demand.
- Assert on what the agent sent and how it reacted: request shape, recovery behavior, call count, and whether guardrails fired.
Run that loop for one tool, then add the next. The setup pays for itself the first time it catches a broken tool call before a user does.
The agent reliability checklist
Before an agent goes to production, walk this list:
- Every tool call is validated against a schema, and contract violations fail a test.
- Upstream 429, 500, and timeout responses are simulated, and the agent recovers with backoff.
- Retried actions are idempotent, so a repeat cannot double-charge or double-send.
- Tests assert on structure and meaning, not exact strings, so normal variation does not cause flakiness.
- Token use per run is measured, and a budget cap stops runaway loops.
- Destructive actions sit behind an allowlist or a human approval gate.
- The guardrail path is tested with a mock, not assumed.
Tick all seven and you have tested the ways agents break in production.
Where Apidog fits (and where it does not)
Be clear about the tool’s job. Apidog is not an agent framework, a model host, or an evaluation harness. It does not build or run your agent. What it does is own the API layer your agent depends on, which is exactly where these failures live.
In practice that means three things. You design and store the contracts for the tools your agent calls, so you can validate outgoing requests against them. You mock those dependencies and program the failure responses (429, 500, timeout, malformed body) that a live API will not produce on demand, so you can rehearse recovery. And you write assertions on the responses (schema, shape, ranges, required keys) that survive non-deterministic output. That is the honest fit: Apidog tests the APIs your agent calls, mocks the failures you need to handle, and checks what comes back. Our overview of agentic AI testing places this in the wider QA picture.
Frequently asked questions
Is agent reliability a model problem or an engineering problem? Mostly engineering. Model choice matters, but the failures that cause incidents (bad tool calls, unhandled rate limits, missing guardrails) are integration and testing problems you can solve without changing the model.
Can I test an agent without hitting the real APIs it calls? Yes, and you should. Mock the dependencies so you can force error responses, control timing, and avoid side effects. That is the only reliable way to test recovery and guardrail paths.
How do I write tests when the output changes every run? Assert on structure and meaning instead of exact text. Validate the response against a schema, check the tool call shape, and use ranges for numbers. The testing non-deterministic AI agents guide covers this in detail.
What should I test first? Guardrails on destructive actions, then error recovery. Those two protect you from the most expensive failures: an agent taking a harmful action, or an agent looping and draining your budget.
Start with one failure mode
You do not have to test all five modes at once. Pick the one that scares you most, usually guardrails or error recovery, and rehearse it against a mock this week. Program the failure, run the agent, and watch what it does. The first time you see your agent handle a simulated 429 with clean backoff instead of a budget-draining loop, you will trust it more, and for a better reason than a green demo.
Download Apidog to design the contracts, mock the failures, and assert on the responses your agent depends on.
