> ## Documentation Index
> Fetch the complete documentation index at: https://docs.custral.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Delivery and retries

> What counts as success, how long Custral waits, how often it retries, and what your handler has to tolerate.

## What counts as success

Any `2xx`. Custral reads the status code and nothing else: the response body is
ignored, and a redirect is not followed.

Anything else, including a redirect, a timeout or a connection failure, is a
failed attempt and is retried.

| Your response                 | Outcome                                                                    |
| ----------------------------- | -------------------------------------------------------------------------- |
| `200`, `201`, `202`, `204`    | Success. Delivery closed.                                                  |
| `3xx`                         | Failed as `redirect_not_followed`. Retried against the URL you registered. |
| `4xx`                         | Failed. Retried.                                                           |
| `5xx`                         | Failed. Retried.                                                           |
| No response within 10 seconds | Failed. Retried.                                                           |

<Tip>
  **Acknowledge first, work second.** The 10 second timeout covers your whole
  response, so push slow work onto your own queue and return `200` immediately.
  A handler that calls three other APIs before responding will eventually cross
  the line, and Custral will retry an event you already processed.
</Tip>

### Redirects are not followed

A delivery is a signed `POST`. Following a redirect would send that signed body
to whatever address the `Location` header names, so Custral stops at the `3xx`,
records it, and retries the URL you registered. It never requests the
`Location`.

The common causes are an `http://` URL on a server that forces `https://`, a
trailing slash your framework adds or strips, and an apex domain that redirects
to `www`. Register the URL your endpoint redirects to.

## The retry schedule

Custral makes up to **8 attempts over about 24 hours**: the first as soon as the
event fires, then 7 retries with a longer wait before each one. The short waits
cover a dropped connection or a process restart. The long ones cover a deploy or
an outage that runs for hours.

| Attempt | Wait after the previous failure | Roughly this long after the first attempt |
| ------- | ------------------------------- | ----------------------------------------- |
| 1       | None                            | Immediately                               |
| 2       | 5 seconds                       | 5 seconds                                 |
| 3       | 30 seconds                      | 35 seconds                                |
| 4       | 2 minutes                       | 2 minutes 35 seconds                      |
| 5       | 10 minutes                      | 12 minutes 35 seconds                     |
| 6       | 1 hour                          | 1 hour 13 minutes                         |
| 7       | 5 hours                         | 6 hours 13 minutes                        |
| 8       | 18 hours                        | 24 hours 13 minutes                       |

Each wait starts when the previous attempt fails, so an attempt that hits the 10
second timeout adds those 10 seconds to the times above. After an outage, a
delivery can arrive hours after your endpoint is back online.

<Warning>
  If all 8 attempts fail, the delivery is not tried again and there is no way
  to replay it by hand. That takes an outage, or a handler that keeps returning
  errors, lasting longer than about a day. If you cannot afford to miss an
  event, reconcile from the [API](/dev/api-reference/records) after a long
  outage.
</Warning>

Two failures skip the retries entirely, because retrying cannot help:

* The endpoint's URL, or an address its hostname resolves to, does not pass the
  [safety check](/dev/webhooks/security#what-custral-will-and-will-not-call).
* The subscription was deleted or set inactive after the event was queued.

## Your handler must be idempotent

Assume every event can arrive **more than once**. A retry after a timeout is the
common case: your server did the work, the response was slow, Custral never saw
it, and the same delivery lands again.

Deduplicate on the envelope's `id`, which is stable across every attempt of one
delivery:

```ts theme={null}
custral.on("record.created", async (event) => {
  // `event.id` is the whd_… delivery id, the same on every retry.
  const fresh = await claim(event.id);
  if (!fresh) return; // Already handled.

  await doTheWork(event.data);
});
```

<Note>
  `createdAt` is **not** stable across retries. It is stamped per attempt, so it
  is useful for measuring delivery latency and useless as a deduplication key.
</Note>

## Order is not guaranteed

Deliveries are processed concurrently, so two events about the same record can
arrive out of order. Do not infer sequence from arrival.

If order matters, treat the payload as a notification that something changed and
read the current state back:

```ts theme={null}
custral.on("record.updated", async (event) => {
  const record = await custral.records.retrieve({
    object: "deals",
    id: event.data.recordId as string,
  });
  // `record` is authoritative. The payload told you to look.
});
```

## Fan-out

One event delivers once **per matching subscription**. Three endpoints
subscribed to `record.created` produce three deliveries with three separate ids,
each retried independently. An endpoint that is down does not hold up the others.

## The delivery log

Every attempt is recorded against the subscription before anything is sent, so a
delivery survives a worker restart. Open an endpoint under **Settings → Developers → Webhooks** to read the last 100, newest first.

| Field            | What it tells you                                                                                                 |
| ---------------- | ----------------------------------------------------------------------------------------------------------------- |
| `status`         | `pending`, `success`, or `failed`                                                                                 |
| `attempts`       | How many times it has been tried                                                                                  |
| `responseStatus` | The HTTP status your endpoint returned on the **last** attempt, or empty when it did not answer                   |
| `error`          | A short code for what went wrong, such as `http_500` or `redirect_not_followed`. See [Error codes](#error-codes). |
| `deliveredAt`    | When it finally succeeded, if it did                                                                              |
| `payload`        | The `data` that was sent                                                                                          |

A `failed` row is not final while `attempts` is below 8. Another attempt is
still coming, unless `error` is `webhook_inactive_or_missing` or starts with
`unsafe_url:`, which are never retried.

<Note>
  The log holds **one row per delivery, not per attempt**. `responseStatus` and
  `error` describe the most recent try, so you can see that a delivery failed
  three times and what the last failure was, but not the individual history of
  each attempt.
</Note>

## Error codes

`error` holds a code, never an error message. A failed connection's message can
include a network address, and everyone in the workspace can read the delivery
log. Transport failures carry the underlying system code after a colon when
there is one.

| Code                          | What happened                                                                                                                                                           | Retried |
| ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- |
| `http_<status>`               | Your endpoint answered with that status.                                                                                                                                | Yes     |
| `redirect_not_followed`       | Your endpoint answered with a `3xx`. See [Redirects are not followed](#redirects-are-not-followed).                                                                     | Yes     |
| `timeout`                     | No response within 10 seconds.                                                                                                                                          | Yes     |
| `connection_failed`           | The connection was refused, reset, or could not be made, such as `connection_failed:ECONNREFUSED`.                                                                      | Yes     |
| `tls_error`                   | The TLS handshake failed, such as `tls_error:CERT_HAS_EXPIRED`.                                                                                                         | Yes     |
| `dns_lookup_failed`           | The hostname did not resolve, such as `dns_lookup_failed:ENOTFOUND`.                                                                                                    | Yes     |
| `delivery_failed`             | Any other failure to deliver.                                                                                                                                           | Yes     |
| `unsafe_url:<reason>`         | Custral refused the URL, or an address its hostname resolved to. See [What Custral will and will not call](/dev/webhooks/security#what-custral-will-and-will-not-call). | No      |
| `webhook_inactive_or_missing` | The endpoint was deleted or set inactive.                                                                                                                               | No      |

## Limits worth knowing

|                                     |                                                                |
| ----------------------------------- | -------------------------------------------------------------- |
| Request timeout                     | 10 seconds                                                     |
| Redirects                           | Not followed. Recorded as `redirect_not_followed` and retried. |
| Attempts                            | 8, over about 24 hours                                         |
| Delivery log retained               | Last 100 per endpoint, newest first                            |
| Replay a delivery by hand           | Not available yet                                              |
| Auto-disable after repeated failure | Not implemented. A dead endpoint keeps being tried.            |

## What's next

<CardGroup cols={2}>
  <Card title="Troubleshooting" icon="wrench" href="/dev/webhooks/troubleshooting">
    Nothing arriving, or arriving and failing.
  </Card>

  <Card title="Managing subscriptions" icon="sliders" href="/dev/webhooks/managing">
    Create, rotate, and delete endpoints.
  </Card>
</CardGroup>


## Related topics

- [Quickstart](/dev/webhooks/quickstart.md)
- [Webhooks](/dev/webhooks/overview.md)
- [Verifying signatures](/dev/webhooks/security.md)
- [Troubleshooting](/dev/webhooks/troubleshooting.md)
- [Managing subscriptions](/dev/webhooks/managing.md)