Your test passed on Monday. Same input, same code, temperature=0. On Tuesday it failed, and you changed nothing. The assertion checked for an exact string, and the model returned the same answer worded a little differently. The test is red, the agent is fine, and now you’re debugging your test suite instead of your product.
This is the tax on testing anything that calls a language model. The output moves, even when you told it not to. Set the temperature to zero and you still won’t get byte-identical responses across runs. Most developers learn this the hard way, once, and then rewrite how they test. This guide shows you how to write assertions that hold up when the text underneath them keeps shifting. It’s the deep dive on failure mode three from our guide to why AI agents break in production.
Why temperature=0 doesn’t mean deterministic
Temperature controls how the model samples the next token. At zero it takes the most probable token every time, which feels like it should be reproducible. It isn’t, and the reasons sit below the model.
Floating-point math is not associative on a GPU. Add the same numbers in a different order and you get a slightly different result in the last decimal place. That tiny difference can tip which token ranks highest, and one different token changes everything after it. The order of those additions depends on how the provider batches your request with other traffic, which hardware runs it, and which kernel version is deployed that day. You control none of it.
Providers also change things on their end. They swap GPUs, update inference libraries, re-quantize weights, and route your call to a different region. A long vLLM discussion walks through why a fixed seed and temperature=0 still aren’t enough for bitwise reproducibility. The short version: determinism is a property of the whole serving stack, not a flag you set in your request.
So stop treating identical output as the baseline. The model gives you an answer that means the same thing, worded however it lands this run. Your tests have to accept that.
Exact-string assertions turn your suite flaky
Here’s the trap. You write assert response == "Your order total is $42.00." because that’s what came back the first time. It passes. Then the model returns “Your total comes to $42.00” and the test fails on a correct answer.
A test that fails on a correct answer is worse than no test at all. The team learns that this suite cries wolf. People re-run it until it goes green, then they stop reading the failures, then they miss the real regression buried in the noise. Flaky tests don’t only waste time, they erode trust in the whole suite, and we’ve written before about what causes flaky tests and why they spread. Non-deterministic output is one of the fastest ways to manufacture them.
The instinct is to pin the output harder: capture the exact string, snapshot it, diff against it. That makes the flakiness worse, because you’ve coupled your test to the one thing guaranteed to change. You need the opposite move.
Assert on structure and meaning, not exact text
The output varies, but the contract underneath it should not. A support agent might phrase a refund confirmation a hundred ways, yet every valid response carries the same facts: a refund amount, an order id, a status. Test the facts, not the phrasing.
That’s the whole shift. Stop asking “did the model say exactly this” and start asking “does the response have the right shape, the right fields, and values in the right range.” Those properties survive rewording. A real regression, a missing field, a number out of bounds, a malformed payload, still trips the assertion. Here are the strategies that put this into practice.
Validate the response against a JSON schema
If your agent returns structured data, define a JSON schema for it and validate every response against that schema. The schema checks types, required fields, allowed enums, and formats without caring about specific values. A status field must be one of refunded, pending, or denied. An order_id must match your id pattern. An amount must be a number, not a string.
This is the strongest single assertion you can write against a non-deterministic response, because it catches the failures that hurt: the model dropped a field, nested the object wrong, or returned prose where you expected JSON. Load your response schema into Apidog and validate the agent’s live responses against it. A mismatch names the exact field that broke, not a 400-character string diff.
Assert the tool call has the right shape and target
When your agent decides to call a tool, test the call, not the sentence that led to it. Assert on three things: it picked the right tool, it aimed at the right target, and the payload matches the tool’s schema. A booking agent calling POST /reservations should send guests as an integer and a valid date, whatever natural-language reasoning produced that call.
This is the same discipline as validating a response body, applied to the outgoing request. Check that required parameters exist, types are correct, and no invented fields sneaked in. The end-to-end method for testing an agent’s API calls covers capturing those tool schemas and asserting against them. A tool-call payload has a contract even when the wording around it doesn’t.
Use numeric ranges instead of exact values
For any number the model produces or passes through, assert a range, not a value. A shopping-cart agent computes a total. You don’t know the exact figure across every run, cart, and tax rule, but you know it can’t be negative and can’t exceed the cart value plus max shipping and tax. So assert that: the reply contains a total between 0 and that ceiling.
That single bound catches the failures that matter, a negative total, a total ten times too large, a total of zero on a full cart, while ignoring variation you don’t care about. Ranges work for confidence scores, item counts, token usage, latency budgets, and any derived figure. Pick the widest bound that still fails on a genuine bug.
Check required keys exist and forbidden fields are absent
Two cheap assertions carry a lot of weight. First, the keys you depend on are present and non-null. Second, the keys that must never appear are absent. An agent handling a support ticket should return a resolution, and it should never leak an internal_notes or raw_prompt field to the customer.
Presence-and-absence checks are immune to rewording by design, since they test the skeleton of the response, not its content. They’re also your cheapest guard against a whole class of privacy leaks, where the model helpfully includes a field it should have kept private.
Use semantic and threshold checks for free text
Sometimes the payload is prose and you still need to test it. Exact match won’t work, so check properties instead. Does the reply contain the order number you passed in? Does it stay under a length cap? Does it avoid a denylist of phrases you never want sent to a user?
When you genuinely need to test meaning, compare by embedding similarity against a reference answer and assert the score clears a threshold, rather than demanding the strings match. Treat these semantic checks as a coarse gate, not a precise one. They catch a response that wandered off topic. They won’t catch a subtle factual error, so pair them with the structural assertions above.
Snapshot ranges, not exact snapshots
Snapshot testing still has a place, as long as you snapshot the stable parts. Freeze the shape of the response, the set of keys, the types, the enum values, and let the free-flowing fields vary within bounds. In practice your snapshot records “this response has keys a, b, c, with b in this range and c from this set” rather than a frozen blob of exact text. When the snapshot breaks, it breaks on a structural change worth reviewing, not on a synonym.
State and memory make this harder
Everything above assumes one request in, one response out. Agents don’t work that way. They carry memory across turns, and that state multiplies the sources of variation.
A stateful agent’s answer depends on what it retrieved, what it stored earlier, and what order previous turns ran in. Two runs of the same conversation can diverge because a retrieval step ranked documents differently, or because a summary written on turn two shaped the reasoning on turn five. Now your output varies for two compounding reasons: the model’s own non-determinism, and a different starting state. Our explainer on how AI agent memory works walks through where that state lives and how it’s built.
Two habits keep this testable. First, control the state you can. Seed the agent’s memory to a known starting point before each test, so you’re varying one thing and not two. Second, assert on invariants that hold regardless of path. A running balance should never go negative. A conversation that booked one flight should end with exactly one reservation, however many turns it took. Path-independent assertions are the ones that survive a stateful, non-deterministic agent.
Mock the dependencies so the test repeats
You can’t rehearse any of this against live third-party APIs. They rate-limit you, they change their data, and they add a second source of randomness on top of the model. To get a repeatable test, pin everything that isn’t the behavior you’re testing.
Mock the APIs the agent calls and program fixed responses. Now the payment API always returns the same receipt, the search API always returns the same three results, and the only thing left moving is the agent’s own reasoning, which is what you want to observe. A mocked dependency also lets you force the edge cases a healthy API won’t produce on demand, then assert the agent handles them. Point Apidog at the agent’s dependencies to stand up those mocks with stable, controllable bodies, and pair them with the schema assertions above. This sits inside the broader practice of agentic AI testing, where mocking and assertion work together.
Where Apidog fits (and where it doesn’t)
Be exact about the tool’s job. Apidog is an API design, testing, and mocking platform. It is not an agent framework, a model host, an agent runtime, or an evaluation and observability platform. It doesn’t build your agent, run it, orchestrate its steps, or score its reasoning.
What it does own is the API layer your agent talks to, and that’s where these tests live. Two honest fits. You write assertions on the agent’s API responses (schema validation, response shape, numeric ranges, required and forbidden keys, tool-call payload shape) that survive non-deterministic output. And you mock the agent’s dependencies so a test runs the same way twice. That’s the seam Apidog fills: the contract on the requests and responses, not the model that produces them.
Test the contract, not the wording
Non-determinism isn’t a bug you can configure away. It’s a property of running a language model, and temperature=0 doesn’t turn it off. The teams that ship reliable agents stopped fighting it. They test the things that stay constant, the schema, the shape, the ranges, the required fields, and they let the wording move. Do that and your suite goes quiet in the good way: it stays green while the text drifts, and it turns red only when something is broken.
Pick one flaky assertion in your suite this week and rewrite it as a schema-and-range check. Download Apidog to validate your agent’s responses against a contract and mock the dependencies that make tests repeatable.
