TL;DR
Design reliable webhooks with exponential backoff retry (5-10 attempts), idempotency keys, HMAC signature verification, and 5-second timeouts. Return 2xx immediately, process asynchronously. Modern PetstoreAPI implements webhooks for order updates, pet adoptions, and payment notifications with full retry and security.
Introduction
You send a webhook to notify a client that their pet was adopted. The client’s server is down. Your webhook fails. Do you retry? How many times? What if the client receives the webhook twice and charges the customer twice?
Webhooks are HTTP callbacks that push events to client URLs. They’re simple in theory but complex in practice. Networks fail, servers crash, and clients have bugs. Production webhooks need retry logic, idempotency, security, and monitoring.
Modern PetstoreAPI implements production-ready webhooks for order updates, pet adoptions, and payment notifications. Every webhook includes retry logic, signature verification, and idempotency.
In this guide, you’ll learn how to design reliable webhooks using Modern PetstoreAPI patterns.
Webhook Basics
Webhooks are HTTP POST requests sent to client-provided URLs when events occur.
How Webhooks Work
1. Client registers webhook URL:
POST /webhooks
{
"url": "https://client.com/webhooks/petstore",
"events": ["pet.adopted", "order.completed"]
}
2. Event occurs (pet adopted)
3. Server sends webhook:
POST https://client.com/webhooks/petstore
Content-Type: application/json
X-Webhook-Signature: sha256=abc123...
{
"event": "pet.adopted",
"data": {
"petId": "019b4132",
"userId": "user-456",
"timestamp": "2026-03-13T10:30:00Z"
}
}
4. Client responds:
200 OK
The Reliability Problem
Webhooks can fail for many reasons:
- Client server is down
- Network timeout
- Client returns 500 error
- Client is slow (takes 30 seconds)
- Client receives webhook but crashes before processing
Without retry logic, events are lost. Without idempotency, duplicate webhooks cause duplicate actions.
Retry Logic with Exponential Backoff
Retry failed webhooks with increasing delays.
Exponential Backoff Strategy
Attempt 1: Immediate
Attempt 2: 1 second later
Attempt 3: 2 seconds later
Attempt 4: 4 seconds later
Attempt 5: 8 seconds later
Attempt 6: 16 seconds later
Why exponential? If the client is down, hammering it with retries won’t help. Exponential backoff gives time for recovery.
Implementation
async function sendWebhook(url, payload, attempt = 1, maxAttempts = 6) {
try {
const response = await fetch(url, {
method: 'POST',
headers: {
'Content-Type': 'application/json',
'X-Webhook-Signature': generateSignature(payload)
},
body: JSON.stringify(payload),
timeout: 5000 // 5 second timeout
});
if (response.ok) {
return { success: true, attempt };
}
// Retry on 5xx errors
if (response.status >= 500 && attempt < maxAttempts) {
const delay = Math.pow(2, attempt - 1) * 1000;
await sleep(delay);
return sendWebhook(url, payload, attempt + 1, maxAttempts);
}
// Don't retry 4xx errors (client error)
return { success: false, status: response.status };
} catch (error) {
// Network error or timeout - retry
if (attempt < maxAttempts) {
const delay = Math.pow(2, attempt - 1) * 1000;
await sleep(delay);
return sendWebhook(url, payload, attempt + 1, maxAttempts);
}
return { success: false, error: error.message };
}
}
When to Retry
Retry on:
- 5xx server errors (500, 502, 503, 504)
- Network timeouts
- Connection refused
- DNS failures
Don’t retry on:
- 4xx client errors (400, 401, 404) - client won’t fix these
- 2xx success - already succeeded
Dead Letter Queue
After max retries, move failed webhooks to a dead letter queue for manual review:
if (!result.success) {
await deadLetterQueue.add({
url,
payload,
attempts: maxAttempts,
lastError: result.error,
timestamp: new Date()
});
}
Idempotency for Duplicate Prevention
Clients might receive the same webhook multiple times. Idempotency prevents duplicate processing.
Idempotency Keys
Include a unique ID with each webhook:
{
"id": "webhook_019b4132",
"event": "pet.adopted",
"data": {...}
}
Client stores processed IDs:
app.post('/webhooks/petstore', async (req, res) => {
const webhookId = req.body.id;
// Check if already processed
const processed = await db.webhooks.findOne({ id: webhookId });
if (processed) {
return res.status(200).json({ message: 'Already processed' });
}
// Process webhook
await processPetAdoption(req.body.data);
// Mark as processed
await db.webhooks.insert({ id: webhookId, processedAt: new Date() });
res.status(200).json({ message: 'Processed' });
});
Idempotent Operations
Design operations to be idempotent:
Bad (not idempotent):
// Charging twice causes double charge
await chargeCustomer(userId, amount);
Good (idempotent):
// Charging with idempotency key prevents double charge
await chargeCustomer(userId, amount, { idempotencyKey: webhookId });
Signature Verification for Security
Verify webhooks come from your API, not an attacker.
HMAC Signature
Generate signature using shared secret:
// Server generates signature
const crypto = require('crypto');
function generateSignature(payload, secret) {
const hmac = crypto.createHmac('sha256', secret);
hmac.update(JSON.stringify(payload));
return hmac.digest('hex');
}
// Include in header
headers['X-Webhook-Signature'] = `sha256=${generateSignature(payload, webhookSecret)}`;
Client verifies signature:
function verifySignature(payload, signature, secret) {
const expected = generateSignature(payload, secret);
return crypto.timingSafeEqual(
Buffer.from(signature),
Buffer.from(`sha256=${expected}`)
);
}
app.post('/webhooks/petstore', (req, res) => {
const signature = req.headers['x-webhook-signature'];
if (!verifySignature(req.body, signature, WEBHOOK_SECRET)) {
return res.status(401).json({ error: 'Invalid signature' });
}
// Process webhook
...
});
Timestamp Validation
Include timestamp to prevent replay attacks:
{
"id": "webhook_019b4132",
"timestamp": "2026-03-13T10:30:00Z",
"event": "pet.adopted",
"data": {...}
}
Reject old webhooks:
const webhookAge = Date.now() - new Date(req.body.timestamp);
if (webhookAge > 5 * 60 * 1000) { // 5 minutes
return res.status(400).json({ error: 'Webhook too old' });
}
Timeout Handling
Set aggressive timeouts to prevent slow clients from blocking your system.
5-Second Timeout
const response = await fetch(url, {
method: 'POST',
body: JSON.stringify(payload),
timeout: 5000 // 5 seconds
});
Why 5 seconds? Webhooks should return immediately. If a client takes longer, they’re doing synchronous processing (wrong pattern).
Async Processing Pattern
Bad (synchronous):
app.post('/webhooks/petstore', async (req, res) => {
// This takes 30 seconds - webhook will timeout
await processOrder(req.body.data);
await sendEmail(req.body.data);
await updateInventory(req.body.data);
res.status(200).json({ message: 'Processed' });
});
Good (asynchronous):
app.post('/webhooks/petstore', async (req, res) => {
// Return immediately
res.status(200).json({ message: 'Received' });
// Process asynchronously
queue.add('process-webhook', req.body);
});
How Modern PetstoreAPI Implements Webhooks
Modern PetstoreAPI implements production-ready webhooks.
Webhook Events
pet.adopted - Pet was adopted
pet.status_changed - Pet status changed
order.created - Order created
order.completed - Order completed
payment.succeeded - Payment succeeded
payment.failed - Payment failed
Webhook Payload
{
"id": "webhook_019b4132-70aa-764f-b315-e2803d882a24",
"event": "pet.adopted",
"timestamp": "2026-03-13T10:30:00Z",
"data": {
"petId": "019b4132-70aa-764f-b315-e2803d882a24",
"userId": "user-456",
"orderId": "order-789",
"adoptionDate": "2026-03-13"
},
"apiVersion": "v1"
}
Retry Configuration
- Max attempts: 10
- Backoff: Exponential (1s, 2s, 4s, 8s, 16s, 32s, 64s, 128s, 256s, 512s)
- Total retry window: ~17 minutes
- Dead letter queue after max retries
Security
- HMAC-SHA256 signature in
X-Webhook-Signatureheader - Timestamp validation (reject > 5 minutes old)
- HTTPS required for webhook URLs
Testing Webhooks with Apidog
Apidog supports webhook testing.
Test Webhook Delivery
- Create mock webhook endpoint in Apidog
- Register endpoint with PetstoreAPI
- Trigger event (adopt pet)
- Verify webhook received
- Check payload format
Test Signature Verification
// Apidog test script
const signature = pm.request.headers.get('X-Webhook-Signature');
const payload = pm.request.body.raw;
const secret = pm.environment.get('WEBHOOK_SECRET');
const expected = generateSignature(payload, secret);
pm.test('Signature valid', () => {
pm.expect(signature).to.equal(`sha256=${expected}`);
});
Test Retry Logic
- Return 500 error from mock endpoint
- Verify retry attempts with exponential backoff
- Check dead letter queue after max retries
Test Idempotency
- Receive webhook
- Return 200
- Receive same webhook again (simulated retry)
- Verify no duplicate processing
Conclusion
Reliable webhooks require:
- Exponential backoff retry (5-10 attempts)
- Idempotency keys to prevent duplicates
- HMAC signature verification for security
- 5-second timeouts
- Async processing on client side
- Dead letter queue for failed webhooks
Modern PetstoreAPI implements all these patterns. Check the webhook documentation for complete examples.
Test your webhooks with Apidog to verify retry logic, signatures, and idempotency before going to production.
FAQ
How many retry attempts should webhooks have?
5-10 attempts with exponential backoff. This covers temporary outages (5-17 minutes) without overwhelming the client.
Should webhooks retry on 4xx errors?
No. 4xx errors indicate client problems (bad URL, authentication failure). Retrying won’t fix these. Only retry 5xx errors and network failures.
How long should webhook timeouts be?
5 seconds maximum. Clients should return 200 immediately and process asynchronously. Longer timeouts indicate the client is doing synchronous processing.
What if a client never responds to webhooks?
After max retries, move to dead letter queue. Alert the client via email. Consider disabling webhooks for that client after repeated failures.
Should webhook URLs be HTTPS?
Yes, always require HTTPS. HTTP webhooks can be intercepted and modified. Modern PetstoreAPI rejects HTTP webhook URLs.
How do you prevent replay attacks?
Include timestamp in payload and reject webhooks older than 5 minutes. Combine with signature verification.
Can clients request webhook redelivery?
Yes. Modern PetstoreAPI provides an endpoint to redeliver specific webhooks: POST /webhooks/{id}/redeliver
How do you test webhooks locally?
Use tools like ngrok to expose localhost to the internet, or use Apidog’s mock server to simulate webhook endpoints during development.



