Retry & delivery

Cobuntu webhooks are at-least-once with exponential backoff. Your receiver can be temporarily down (deploy rolling, rate-limit window, transient blip) — Cobuntu retries on a schedule that gives you ~28 hours of recovery time before giving up.

What counts as a successful delivery

• HTTP 2xx status returned within 10 seconds.
• Body content is ignored — we don't parse your response.

Anything else (non-2xx, timeout, connection error, TLS handshake failure) counts as a failure and triggers the next retry.

Backoff schedule

After the 6th attempt fails, the delivery is marked PERMANENTLY_FAILED. The event isn't retried automatically again, but you can manually replay it from cobuntu-admin → Integrations → Webhook deliveries (a button shows up on failed rows).

Manual replay

In admin → Integrations → Webhook deliveries:

1. Filter by FAILED or PERMANENTLY_FAILED status.
2. Click into a delivery → see the request payload + each attempt's response.
3. Click Resend — Cobuntu retries immediately, starting a fresh attempt counter.

Replays carry the same deliveryId as the original. If your receiver is properly idempotent (de-duping on deliveryId), a replay of a delivery you already processed successfully is a no-op.

Idempotency — your job

Webhooks are at-least-once. That means: under normal conditions each event fires once. Under degraded conditions (network blips, your receiver returning 5xx then recovering, manual replays), the same deliveryId can arrive twice or more.

Make your handler idempotent. The simplest pattern:

async function handleWebhook(req: Request) {
  const event = req.body;  // {event, deliveryId, data, …}
  // 1. Have we seen this deliveryId before?
  const seen = await db.webhookSeen.findUnique({
    where: { deliveryId: event.deliveryId }
  });
  if (seen) {
    return new Response("OK (replay)", { status: 200 });
  }
  // 2. Mark seen FIRST, in the same transaction as your side effect.
  await db.$transaction([
    db.webhookSeen.create({ data: { deliveryId: event.deliveryId } }),
    doSideEffect(event),
  ]);
  return new Response("OK", { status: 200 });
}

The webhookSeen table is just (deliveryId text primary key, receivedAt timestamptz). Insert + side effect in one transaction means: either both succeed (200) or both roll back (5xx → Cobuntu retries → next time the dedup wins).

What NOT to do

• Don't return 2xx then process async. If your async job fails, the event is silently lost; Cobuntu thinks delivery succeeded. Process inline OR enqueue + use the dedup table.
• Don't return 4xx for transient issues. 4xx means "this request will never succeed" — Cobuntu still retries (we don't distinguish 4xx semantics), but it suggests a bug if the request is well-formed. Return 5xx for "I'm temporarily down" so it's clear in your logs why retries are happening.
• Don't process in order. Webhook delivery is at-least-once, not ordered. A member.approved can arrive before the member.requested that preceded it (e.g. if the first delivery failed and the second one to a different event was first to succeed). Use timestamps in the payload, not arrival order.

Where to look when things break

The Recent deliveries tab in cobuntu-admin has, per delivery:

• Delivery ID — search your receiver's logs for this to find your side
• Event name + Created at
• All attempts with: timestamp, response status, response body, response time
• Final status — SUCCEEDED, RETRYING, FAILED, PERMANENTLY_FAILED
• Resend button

If your receiver is healthy but PERMANENTLY_FAILED is climbing, filter by event name to identify which events you can't process — often it's a specific event with a payload shape your handler doesn't expect.