Your agent called the payment endpoint. The request went through, the charge landed, and then the response timed out on the way back. The agent never saw a 200, so it did what you told it to do on failure: it retried. Now the customer has been charged twice, and nothing in your logs looks like an error.
This is the failure mode that separates agents from ordinary API clients. A human clicking “Pay” once sees a spinner and waits. An agent in a retry loop sees silence and tries again, sometimes three or four times in a row, faster than any person could. Every retry policy you add to make the agent more reliable also makes duplicate writes more likely. The fix is idempotency: making a repeated request produce the same result as a single request.
This guide covers what idempotency means at the HTTP level, how to generate keys an agent can actually reuse, what the server has to store to honor them, and how to test the whole thing before a real customer gets billed twice. If you have not read our pillar on why AI agents break in production, duplicate writes are the failure mode hiding underneath most “the agent did it twice” reports.
Apidog shows up in the testing half of this. Idempotency is something you build into your API and your agent’s tool layer. What you need afterward is a way to fire the same request twice and prove the second one changed nothing, which is a test you can save and run in CI.
Why agents break idempotency more often than people do
Three things about agent traffic make duplicates common.
The first is retry volume. Agent frameworks retry aggressively by default because transient network failures are the most common cause of a broken run. Our guide to agent error recovery walks through backoff and circuit breakers, and every technique in it increases the number of times a given request hits your server.
The second is the ambiguity of a timeout. When a request times out, the client learns nothing about whether the server processed it. A 504 from a proxy could mean the write never happened or that it happened and the response got lost. Humans usually check before retrying. Agents usually do not, because “check first” is an extra tool call the model has to decide to make.
The third is the loop. An agent that fails a task may restart the whole task, not just the failed step. If step one creates an order and step four fails, a naive restart creates a second order. This is where multi-step agents differ sharply from a script: the retry boundary is fuzzy, and the model, not your code, decides where it begins.
Put those together and you get the shape of the problem. It is not that agents send bad requests. They send correct requests more than once.
What idempotency actually guarantees
An operation is idempotent when performing it many times has the same effect as performing it once. GET, PUT, and DELETE are defined as idempotent in RFC 9110, the HTTP semantics spec. POST is not, which is exactly why the dangerous operations tend to be POST calls: create an order, send a message, start a transfer.
Two clarifications save a lot of confusion.
Idempotent is not the same as safe. A safe method changes nothing. DELETE is idempotent but destructive: calling it five times leaves the resource deleted, same as calling it once, but the resource is still gone. Agents need both properties sorted separately, which is the argument our post on least-privilege API keys for agents makes from the credentials side.
Idempotent is also not the same as identical response. The second call may return the stored result of the first, and it may return a different status code. What must not change is the state on the server. One charge. One order. One email.
Idempotency keys: the pattern that makes POST safe
The standard fix is a client-generated key sent with the request. The server records the key alongside the result, and any later request carrying the same key returns the recorded result instead of doing the work again.
Stripe popularized the header, and the Stripe idempotency documentation is still the clearest description of the semantics. There is also an IETF effort to standardize it as the Idempotency-Key header field, which is worth reading before you invent your own header name.
The request looks like this:
POST /v1/payments HTTP/1.1
Host: api.yourservice.com
Authorization: Bearer sk_live_...
Idempotency-Key: 9f2b7c14-6d3a-4b18-9d55-1e2a7c0b4f31
Content-Type: application/json
{
"amount": 4900,
"currency": "usd",
"customer_id": "cus_8812",
"description": "Pro plan, August"
}
The key is a UUID. It has no meaning to the server beyond “this is the same logical operation.” The server stores it, along with a fingerprint of the request body and the response it produced.
Generating a key the agent can reuse
Here is where most agent implementations go wrong. If the tool wrapper generates a fresh UUID on every call, the key changes on every retry, and idempotency does nothing. The key must be tied to the logical operation, not to the HTTP attempt.
The rule: generate the key when the agent decides to perform an action, and hold it for every retry of that decision.
import uuid
class PaymentTool:
def __init__(self, client):
self.client = client
self._keys = {}
def charge(self, task_id, step_id, amount, customer_id):
# One key per (task, step). Retries of the same step reuse it.
op = f"{task_id}:{step_id}"
if op not in self._keys:
self._keys[op] = str(uuid.uuid4())
return self.client.post(
"/v1/payments",
headers={"Idempotency-Key": self._keys[op]},
json={"amount": amount, "customer_id": customer_id},
)
A deterministic key works too, and it survives process restarts, which a dictionary in memory does not:
import hashlib
def idempotency_key(task_id: str, step_id: str, payload: dict) -> str:
raw = f"{task_id}|{step_id}|{sorted(payload.items())}"
return hashlib.sha256(raw.encode()).hexdigest()[:32]
Derive the key from the task run and the step, never from a timestamp or a random value regenerated per attempt. If the agent restarts the whole task and genuinely intends a new charge, the task ID changes and so does the key. That is the behavior you want.
What the server has to do
Handling the header correctly takes more than a lookup. A working implementation does four things:
- On arrival, try to claim the key. Insert it into a table with a unique constraint before doing any work. If the insert fails, another attempt owns it.
- If the key exists and the stored request fingerprint differs, reject with
422. Same key with a different body means a client bug, and silently returning the old result would hide it. - If the key exists and the first attempt is still in flight, return
409so the caller backs off rather than racing. - When the work finishes, store the status code and body against the key, then return it for every later hit.
CREATE TABLE idempotency_records (
key TEXT PRIMARY KEY,
request_hash TEXT NOT NULL,
state TEXT NOT NULL, -- in_progress | completed
response_status INT,
response_body JSONB,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
expires_at TIMESTAMPTZ NOT NULL
);
Set an expiry. Twenty-four hours covers any realistic retry window, and keeping keys forever turns the table into a liability. Stripe expires keys after 24 hours, which is a reasonable default to copy.
Testing that the second call changes nothing
Building idempotency is half the work. Proving it holds is the other half, and it is the half that gets skipped, because the happy path looks identical whether or not the feature works.
The test is simple to describe: send the request, capture the result, send the exact same request again, and assert the server did not do the work twice. The hard part is the last assertion, because the response alone will not tell you. Two successful charges both return 200.
So assert on state, not on the response:
- The second response body matches the first, including the resource ID. A new ID means a new resource was created.
- A follow-up
GETon the collection returns one record, not two. - Any counter or balance moved once.
In Apidog you can wire this as a test scenario: step one sends the POST with a fixed Idempotency-Key, step two repeats it, and step three lists the resource and asserts the count. Save the response ID from step one into a variable and assert step two returns the same value. Because the whole scenario is stored, it runs in CI on every change to the payment path, which is where regressions actually appear. The same technique carries over to the broader patterns in our API contract testing guide.
Two more cases worth covering, because they catch real bugs:
- Same key, different body. Expect
422, not a silent success. - Concurrent duplicates. Fire both requests at once and confirm exactly one wins. This catches the missing unique constraint that a sequential test will never surface.
Mocking helps here too. If you are still building the agent and the payment API does not exist yet, mock it with an idempotency-aware response so the agent’s retry logic gets exercised early. Our post on why agents should hit mocks instead of production makes the wider case for that habit.
When you cannot add a key
Sometimes the API is not yours and it has no idempotency support. You still have options, in rough order of preference.
Make the operation naturally idempotent. A PUT to a resource path the client chooses is idempotent by construction: PUT /orders/{client_order_id}. If you control the API design, prefer this over POST plus a header. It needs no extra table.
Check before writing. Have the agent query for an existing record with the same natural key before it creates one. This is weaker, because a race between the check and the write can still produce two records, but it removes the common timeout case.
Deduplicate downstream. If the write is a message or an event, put the deduplication in the consumer. Attach a stable message ID and have the consumer drop repeats. This is standard practice in event-driven systems and pairs with the guidance in our reliable webhooks guide.
Gate the action. For operations that are genuinely irreversible and cannot be made idempotent, put a human in front. That is the approval-gate pattern from our post on AI agent guardrails, and it is the right answer when the cost of a duplicate is high enough.
Know which run did what
Idempotency stops the duplicate. It does not tell you which attempt created the record, and that is the question you get asked after an incident.
Keep the run identity attached to the work. When the agent is your own service, that means the task ID and step ID from the key derivation above, logged with every attempt. When the agent is a coding runtime executing assigned work, the platform usually holds it for you: in Sharkly, each run is attached to the Task it came from, with its execution state and result stored alongside the comment thread, so a repeated write traces back to a specific run rather than to an anonymous retry.
A checklist before you ship
- Every non-idempotent tool the agent can call requires an idempotency key, and the tool wrapper refuses to send without one.
- Keys derive from the task and step, not from the attempt.
- The server claims the key before doing work, not after.
- Same key with a different payload returns an error instead of the cached response.
- Concurrent duplicates are handled by a database constraint, not by application timing.
- A saved test proves the second call changes nothing, and it runs in CI.
- Keys expire on a schedule and the table is cleaned up.
Work through that list and the double-charge story stops being possible, which means your retry policy can get more aggressive rather than less. That is the real payoff: idempotency is what lets you make an agent resilient without making it dangerous.
Frequently asked questions
Do I need idempotency keys for read-only tools? No. GET requests are already idempotent and safe, so retrying one costs you a little latency and nothing else. Reserve keys for calls that create, charge, send, or otherwise change state.
Where should the key be generated, in the agent or in the tool wrapper? In the tool wrapper, keyed off the agent’s task and step identifiers. Letting the model generate the key is a mistake: models regenerate values on retries and can produce collisions across tasks.
What status code should a repeated request return? Return the stored status from the original call, so a second POST that first returned 201 returns 201 again with the same body. Some APIs add a header such as Idempotent-Replay: true to mark the repeat, which is useful for debugging and harmless to clients that ignore it.
How long should keys be kept? Twenty-four hours covers nearly every retry window. Longer retention rarely helps and grows the table without bound. If a client retries after the window, treat it as a new operation.
Does this replace transactions? No. Idempotency keys stop duplicate requests from producing duplicate effects. Transactions keep a single request atomic. You need both, and the key claim should be written in the same transaction as the work whenever your database allows it.
How do I test this without a real payment provider? Point the agent at a mock that implements the key semantics, including the 422 on payload mismatch. Our guide to testing AI agents against mocked APIs covers the setup, and Download Apidog if you want the mock and the retry test living in the same project.
