How Should You Design Reliable Webhooks?

Webhooks are unreliable by nature. Learn how to design production-ready webhooks with retry logic, idempotency, signature verification, and timeout handling using Modern PetstoreAPI patterns.

Ashley Innocent

Ashley Innocent

13 March 2026

How Should You Design Reliable Webhooks?

TL;DR

Design reliable webhooks with exponential backoff retry (5-10 attempts), idempotency keys, HMAC signature verification, and 5-second timeouts. Return 2xx immediately, process asynchronously. Modern PetstoreAPI implements webhooks for order updates, pet adoptions, and payment notifications with full retry and security.

Introduction

You send a webhook to notify a client that their pet was adopted. The client’s server is down. Your webhook fails. Do you retry? How many times? What if the client receives the webhook twice and charges the customer twice?

Webhooks are HTTP callbacks that push events to client URLs. They’re simple in theory but complex in practice. Networks fail, servers crash, and clients have bugs. Production webhooks need retry logic, idempotency, security, and monitoring.

Modern PetstoreAPI implements production-ready webhooks for order updates, pet adoptions, and payment notifications. Every webhook includes retry logic, signature verification, and idempotency.

💡
If you’re building or testing webhooks, Apidog helps you test webhook delivery, validate signatures, and simulate failure scenarios. You can test retry logic and verify idempotency handling.
button

In this guide, you’ll learn how to design reliable webhooks using Modern PetstoreAPI patterns.

Webhook Basics

Webhooks are HTTP POST requests sent to client-provided URLs when events occur.

How Webhooks Work

1. Client registers webhook URL:

POST /webhooks
{
  "url": "https://client.com/webhooks/petstore",
  "events": ["pet.adopted", "order.completed"]
}

2. Event occurs (pet adopted)

3. Server sends webhook:

POST https://client.com/webhooks/petstore
Content-Type: application/json
X-Webhook-Signature: sha256=abc123...

{
  "event": "pet.adopted",
  "data": {
    "petId": "019b4132",
    "userId": "user-456",
    "timestamp": "2026-03-13T10:30:00Z"
  }
}

4. Client responds:

200 OK

The Reliability Problem

Webhooks can fail for many reasons:

Without retry logic, events are lost. Without idempotency, duplicate webhooks cause duplicate actions.

Retry Logic with Exponential Backoff

Retry failed webhooks with increasing delays.

Exponential Backoff Strategy

Attempt 1: Immediate
Attempt 2: 1 second later
Attempt 3: 2 seconds later
Attempt 4: 4 seconds later
Attempt 5: 8 seconds later
Attempt 6: 16 seconds later

Why exponential? If the client is down, hammering it with retries won’t help. Exponential backoff gives time for recovery.

Implementation

async function sendWebhook(url, payload, attempt = 1, maxAttempts = 6) {
  try {
    const response = await fetch(url, {
      method: 'POST',
      headers: {
        'Content-Type': 'application/json',
        'X-Webhook-Signature': generateSignature(payload)
      },
      body: JSON.stringify(payload),
      timeout: 5000 // 5 second timeout
    });

    if (response.ok) {
      return { success: true, attempt };
    }

    // Retry on 5xx errors
    if (response.status >= 500 && attempt < maxAttempts) {
      const delay = Math.pow(2, attempt - 1) * 1000;
      await sleep(delay);
      return sendWebhook(url, payload, attempt + 1, maxAttempts);
    }

    // Don't retry 4xx errors (client error)
    return { success: false, status: response.status };

  } catch (error) {
    // Network error or timeout - retry
    if (attempt < maxAttempts) {
      const delay = Math.pow(2, attempt - 1) * 1000;
      await sleep(delay);
      return sendWebhook(url, payload, attempt + 1, maxAttempts);
    }
    return { success: false, error: error.message };
  }
}

When to Retry

Retry on:

Don’t retry on:

Dead Letter Queue

After max retries, move failed webhooks to a dead letter queue for manual review:

if (!result.success) {
  await deadLetterQueue.add({
    url,
    payload,
    attempts: maxAttempts,
    lastError: result.error,
    timestamp: new Date()
  });
}

Idempotency for Duplicate Prevention

Clients might receive the same webhook multiple times. Idempotency prevents duplicate processing.

Idempotency Keys

Include a unique ID with each webhook:

{
  "id": "webhook_019b4132",
  "event": "pet.adopted",
  "data": {...}
}

Client stores processed IDs:

app.post('/webhooks/petstore', async (req, res) => {
  const webhookId = req.body.id;

  // Check if already processed
  const processed = await db.webhooks.findOne({ id: webhookId });
  if (processed) {
    return res.status(200).json({ message: 'Already processed' });
  }

  // Process webhook
  await processPetAdoption(req.body.data);

  // Mark as processed
  await db.webhooks.insert({ id: webhookId, processedAt: new Date() });

  res.status(200).json({ message: 'Processed' });
});

Idempotent Operations

Design operations to be idempotent:

Bad (not idempotent):

// Charging twice causes double charge
await chargeCustomer(userId, amount);

Good (idempotent):

// Charging with idempotency key prevents double charge
await chargeCustomer(userId, amount, { idempotencyKey: webhookId });

Signature Verification for Security

Verify webhooks come from your API, not an attacker.

HMAC Signature

Generate signature using shared secret:

// Server generates signature
const crypto = require('crypto');

function generateSignature(payload, secret) {
  const hmac = crypto.createHmac('sha256', secret);
  hmac.update(JSON.stringify(payload));
  return hmac.digest('hex');
}

// Include in header
headers['X-Webhook-Signature'] = `sha256=${generateSignature(payload, webhookSecret)}`;

Client verifies signature:

function verifySignature(payload, signature, secret) {
  const expected = generateSignature(payload, secret);
  return crypto.timingSafeEqual(
    Buffer.from(signature),
    Buffer.from(`sha256=${expected}`)
  );
}

app.post('/webhooks/petstore', (req, res) => {
  const signature = req.headers['x-webhook-signature'];

  if (!verifySignature(req.body, signature, WEBHOOK_SECRET)) {
    return res.status(401).json({ error: 'Invalid signature' });
  }

  // Process webhook
  ...
});

Timestamp Validation

Include timestamp to prevent replay attacks:

{
  "id": "webhook_019b4132",
  "timestamp": "2026-03-13T10:30:00Z",
  "event": "pet.adopted",
  "data": {...}
}

Reject old webhooks:

const webhookAge = Date.now() - new Date(req.body.timestamp);
if (webhookAge > 5 * 60 * 1000) { // 5 minutes
  return res.status(400).json({ error: 'Webhook too old' });
}

Timeout Handling

Set aggressive timeouts to prevent slow clients from blocking your system.

5-Second Timeout

const response = await fetch(url, {
  method: 'POST',
  body: JSON.stringify(payload),
  timeout: 5000 // 5 seconds
});

Why 5 seconds? Webhooks should return immediately. If a client takes longer, they’re doing synchronous processing (wrong pattern).

Async Processing Pattern

Bad (synchronous):

app.post('/webhooks/petstore', async (req, res) => {
  // This takes 30 seconds - webhook will timeout
  await processOrder(req.body.data);
  await sendEmail(req.body.data);
  await updateInventory(req.body.data);

  res.status(200).json({ message: 'Processed' });
});

Good (asynchronous):

app.post('/webhooks/petstore', async (req, res) => {
  // Return immediately
  res.status(200).json({ message: 'Received' });

  // Process asynchronously
  queue.add('process-webhook', req.body);
});

How Modern PetstoreAPI Implements Webhooks

Modern PetstoreAPI implements production-ready webhooks.

Webhook Events

pet.adopted - Pet was adopted
pet.status_changed - Pet status changed
order.created - Order created
order.completed - Order completed
payment.succeeded - Payment succeeded
payment.failed - Payment failed

Webhook Payload

{
  "id": "webhook_019b4132-70aa-764f-b315-e2803d882a24",
  "event": "pet.adopted",
  "timestamp": "2026-03-13T10:30:00Z",
  "data": {
    "petId": "019b4132-70aa-764f-b315-e2803d882a24",
    "userId": "user-456",
    "orderId": "order-789",
    "adoptionDate": "2026-03-13"
  },
  "apiVersion": "v1"
}

Retry Configuration

Security

Testing Webhooks with Apidog

Apidog supports webhook testing.

Test Webhook Delivery

  1. Create mock webhook endpoint in Apidog
  2. Register endpoint with PetstoreAPI
  3. Trigger event (adopt pet)
  4. Verify webhook received
  5. Check payload format

Test Signature Verification

// Apidog test script
const signature = pm.request.headers.get('X-Webhook-Signature');
const payload = pm.request.body.raw;
const secret = pm.environment.get('WEBHOOK_SECRET');

const expected = generateSignature(payload, secret);
pm.test('Signature valid', () => {
  pm.expect(signature).to.equal(`sha256=${expected}`);
});

Test Retry Logic

  1. Return 500 error from mock endpoint
  2. Verify retry attempts with exponential backoff
  3. Check dead letter queue after max retries

Test Idempotency

  1. Receive webhook
  2. Return 200
  3. Receive same webhook again (simulated retry)
  4. Verify no duplicate processing

Conclusion

Reliable webhooks require:

Modern PetstoreAPI implements all these patterns. Check the webhook documentation for complete examples.

Test your webhooks with Apidog to verify retry logic, signatures, and idempotency before going to production.

button

FAQ

How many retry attempts should webhooks have?

5-10 attempts with exponential backoff. This covers temporary outages (5-17 minutes) without overwhelming the client.

Should webhooks retry on 4xx errors?

No. 4xx errors indicate client problems (bad URL, authentication failure). Retrying won’t fix these. Only retry 5xx errors and network failures.

How long should webhook timeouts be?

5 seconds maximum. Clients should return 200 immediately and process asynchronously. Longer timeouts indicate the client is doing synchronous processing.

What if a client never responds to webhooks?

After max retries, move to dead letter queue. Alert the client via email. Consider disabling webhooks for that client after repeated failures.

Should webhook URLs be HTTPS?

Yes, always require HTTPS. HTTP webhooks can be intercepted and modified. Modern PetstoreAPI rejects HTTP webhook URLs.

How do you prevent replay attacks?

Include timestamp in payload and reject webhooks older than 5 minutes. Combine with signature verification.

Can clients request webhook redelivery?

Yes. Modern PetstoreAPI provides an endpoint to redeliver specific webhooks: POST /webhooks/{id}/redeliver

How do you test webhooks locally?

Use tools like ngrok to expose localhost to the internet, or use Apidog’s mock server to simulate webhook endpoints during development.

Explore more

What is Tokenization? The Ultimate Guide to API Security

What is Tokenization? The Ultimate Guide to API Security

Tokenization is a powerful method to secure sensitive data by replacing it with non-sensitive tokens. In this guide, we explore the core concepts of tokenization, compare it with encryption, review key benefits and use cases, and show how to design and test secure APIs using Apidog.

13 March 2026

How Do You Build Event-Driven APIs with Webhooks and Message Queues?

How Do You Build Event-Driven APIs with Webhooks and Message Queues?

Event-driven APIs decouple services and enable asynchronous processing. Learn how to combine webhooks, message queues, and event buses with Modern PetstoreAPI patterns.

13 March 2026

How Do You Stream API Responses Using Server-Sent Events (SSE)?

How Do You Stream API Responses Using Server-Sent Events (SSE)?

Server-Sent Events let you stream API responses in real-time. Learn how to implement SSE for live updates, AI streaming, and progress tracking with Modern PetstoreAPI examples.

13 March 2026

Practice API Design-first in Apidog

Discover an easier way to build and use APIs