How to Handle Duplicate Webhooks: The Replay Test

Short answer: assume every webhook will arrive more than once, late, and out of order, because senders deliver at-least-once and retry whenever they are unsure. Store each delivery id in a processed-events table with a unique constraint, write that row in the same transaction as the state change, and make the change itself an upsert on the partner's own object id that refuses to overwrite newer data. If the partner does not sign its webhooks, treat each one as a nudge to re-read the object from their API. Then prove it with one test: every delivery sent three times, out of order, after a restart, must leave the state unchanged.

A founder who had built a two-way sync with a partner summed up where the work went: "most of the work was token refresh, idempotent webhooks, and mapping fields that both sides rename without warning." Nobody mentioned the build. The build is the easy part now; any coding agent writes a webhook endpoint in minutes. What it writes by default is a handler that works perfectly on the first delivery and corrupts data on the second. This guide is about the second.

Why do webhooks arrive more than once?

Because the sender cannot know whether you got it. A webhook is an HTTP request, and the sender only sees the response. Every one of these looks identical from their side:

• Your endpoint was down, so the request failed.
• Your endpoint took longer than their timeout, even though it finished the work.
• Your endpoint returned 200, but the response was lost on the way back.
• Your endpoint returned a 500 after writing half the change.

The safe move for a sender is to retry, so they all do. Retry schedules vary widely: some back off over hours, some retry every minute for a day, which is up to 1,440 copies of one event if your endpoint stays broken. Add manual "resend" buttons in the partner's dashboard, your own deploys dropping in-flight requests, and parallel workers on their side, and you also get out-of-order delivery: "invoice paid" can land before "invoice created".

Exactly-once delivery does not exist across a network. Exactly-once effect does, and it is your job, not the sender's.

Step 1: a processed-events table with a unique constraint

Most senders put a unique id on each event or delivery (a header or an id field). Record it, and let the database, not your code, decide whether you have seen it before:

create table processed_webhooks (
  provider     text        not null,
  event_id     text        not null,
  tenant_id    uuid        not null,
  received_at  timestamptz not null default now(),
  primary key (provider, event_id)
);

-- prune well after the sender's retry window has closed
delete from processed_webhooks where received_at < now() - interval '14 days';

The handler inserts that row and applies the change in one transaction:

async function handleEvent(evt: PartnerEvent, tenantId: string) {
  await db.transaction(async (tx) => {
    const res = await tx.query(
      "insert into processed_webhooks (provider, event_id, tenant_id) " +
      "values ($1, $2, $3) on conflict do nothing",
      ["partner", evt.id, tenantId],
    );
    if (res.rowCount === 0) return; // already handled: do nothing

    await applyChange(tx, tenantId, evt); // same transaction
  });
  // reply 200 whether it was new or a duplicate
}

Three details decide whether this actually works:

1. Same transaction. If you record the id and then the change fails, the retry is skipped and the change is lost forever. If you apply the change and then fail to record the id, the retry applies it twice. One transaction makes both happen or neither.
2. The unique constraint, not a SELECT. "Check if it exists, then insert" has a race: two copies arriving in the same second both see nothing and both proceed. on conflict do nothing lets the database pick exactly one winner.
3. Return 2xx for duplicates. An error status tells the sender the delivery failed, which earns you another retry. A duplicate is a success that already happened.

Also: acknowledge fast. If processing is slow, write the raw event to a queue table (deduped the same way), return 200, and process it in a worker. A slow handler is the most common cause of duplicates in the first place.

Step 2: make the change itself idempotent

The event id stops the same delivery being applied twice. It does not stop two different events describing the same object, or an old event arriving after a new one. For that, the write needs to be an upsert keyed on the partner's own id, guarded by their version or updated-at timestamp:

insert into partner_invoices (tenant_id, partner_invoice_id, status, partner_updated_at)
values ($1, $2, $3, $4)
on conflict (tenant_id, partner_invoice_id) do update
  set status             = excluded.status,
      partner_updated_at = excluded.partner_updated_at
  where partner_invoices.partner_updated_at < excluded.partner_updated_at;

Never insert a row per webhook; that is how one invoice becomes three. And never use your own receive time as the version: the whole problem is that receive order is not event order.

Side effects outside the database (sending an email, charging a card, provisioning an account) need the same discipline. Write them as rows in an outbox table inside the transaction and let a separate worker perform them, so a replay cannot send a second email. When the side effect is a call to the partner's API and they accept an Idempotency-Key header, send one derived from the event id, so your own retries are safe on their side too.

Should you trust the webhook payload at all?

Only if it is signed and you verified the signature. A signed webhook carries an HMAC of the raw request body computed with a shared secret; you recompute it, compare in constant time, and reject anything with a timestamp outside a few minutes. Verify against the raw bytes, before any JSON parsing reformats them.

Many smaller APIs do not sign their webhooks at all. Then anyone who learns the URL can post to it, and the payload is just a claim. The fix is to stop reading it as data:

// The webhook is a nudge: "something about this object changed".
async function onPartnerWebhook(body: { account_id: string; object_id: string }) {
  const tenant = await tenants.findByPartnerAccount(body.account_id);
  if (tenant) {
    // one pending job per object: ten nudges in a row become one re-read
    await jobs.enqueueUnique("resync:" + tenant.id + ":" + body.object_id, {
      tenantId: tenant.id,
      objectId: body.object_id,
    });
  }
  return new Response(null, { status: 200 }); // same answer either way
}

async function resync({ tenantId, objectId }: ResyncJob) {
  const fresh = await partnerApi(tenantId).getInvoice(objectId); // authenticated read
  await upsertInvoice(tenantId, fresh); // the guarded upsert from step 2
}

This pattern quietly solves three problems. A forged nudge costs one API call and changes nothing, because you only ever write what the partner's API returns under that tenant's own credentials. Ordering stops mattering, because a re-read always fetches the current state. And duplicates collapse, because ten nudges for one object become one job. It is worth using even for signed webhooks when the partner retries aggressively.

If the partner has no event log or cursor to catch up from after an outage, keep a high-water mark per tenant (the latest updated_at you have stored) and run a periodic "changed since" read with a small overlap window. Webhooks make you fast; the catch-up read makes you correct when the webhooks were missed.

The replay test: how do you prove a webhook handler is idempotent?

Everything above can be written wrong in a way that passes every normal test, because normal tests deliver each event once, in order. So write the test that delivers them the way production does:

Every delivery sent three times, out of order, after a restart, must leave the state unchanged.

test("replay: 3x, shuffled, across a restart", async () => {
  const deliveries = loadRecordedDeliveries(); // real payloads, captured from staging

  // the reference: each event once, in order
  for (const d of deliveries) await deliver(d);
  await drainQueue();
  const expected = await snapshotState(); // rows + outbox side effects

  await resetState();

  // the real world: three copies of everything, any order, a crash in the middle
  const chaos = shuffle([...deliveries, ...deliveries, ...deliveries]);
  const half = Math.floor(chaos.length / 2);
  for (const d of chaos.slice(0, half)) await deliver(d);
  await restartWorker(); // drop in-memory caches, kill in-flight jobs
  for (const d of chaos.slice(half)) await deliver(d);
  await drainQueue();

  expect(await snapshotState()).toEqual(expected);
});

What each part catches:

If you use the nudge pattern, your fake partner API should return the final state of each object in both runs, which is exactly what the real API would do. Run the test in CI on every change to the integration, not once at launch. The handler that passed in March is not the handler someone edited in June.

A checklist before you ship a webhook handler

1. Delivery ids recorded under a unique constraint, in the same transaction as the change.
2. Writes are upserts on the partner's object id, guarded by their version or timestamp.
3. External side effects go through an outbox, never straight from the handler.
4. Duplicates get a 2xx. Slow work goes to a queue so the endpoint answers quickly.
5. Signed webhooks verified on the raw body; unsigned ones only trigger a re-read.
6. A per-tenant catch-up read for the deliveries that never arrive.
7. The replay test in CI.

Where this fits in an integration partnership

Duplicate deliveries are one of seven integration problems that show up after launch, and they are the most mechanical of the seven. That is good news: it is pure engineering, your own coding agent can write all of the above, and the replay test tells you whether it did. The harder problems are the ones a test cannot catch, like who answers the customer when the partner is down, or whose numbers settle a revenue share.

On Ordana, members build their integrations themselves with their own coding agent, from a prompt Ordana writes; Ordana matches the partners, puts the terms in writing and tracks the revenue share. We would rather hand you a test that proves your glue is safe than pretend to own it.

Related reading:

• 7 Common SaaS Integration Problems (and How to Fix Each) — duplicate deliveries alongside credentials, schema drift, outages, attribution, exit and churn.
• The 3 Types of SaaS Integration: Placement, Capability, Handoff — which kinds of integration depend on webhooks at all.
• Integration partnerships for SaaS — choosing the partner before you write the handler.