Designing a webhook consumer that does not drop call events

A webhook is a request your provider sends to your server whenever something happens, such as a call starting, being answered or ending. The first version of a consumer is easy: parse the body, update a record, return 200. It works in testing. In production it meets slow databases, retries, duplicates and events that arrive in the wrong order. This post covers the handful of habits that keep call events from being lost or counted twice.

Verify the signature first

Your endpoint is a public URL, so anyone can send it a request that looks like a call event. Tizbo includes a signature in the Tizbo-Signature header so you can check the request came from Tizbo. The common scheme is an HMAC of the raw request body using a shared secret. Confirm the exact format in the API documentation, then follow these rules.

• Compute the signature over the raw bytes of the body, before any JSON parsing. Re-serializing a parsed body can change whitespace or key order and break the match.
• Compare using a constant-time function, so the comparison does not leak how many characters matched.
• Reject the request with a 4xx status if the signature is missing or wrong, and do no other work.
• Keep the secret in your secret store, not in the repository, and plan how to rotate it.

// Pseudocode: verify before anything else
const raw = await readRawBody(request);
const given = request.headers['tizbo-signature'];
const expected = hmacSha256(SECRET, raw);
if (!timingSafeEqual(given, expected)) return respond(401);

Respond fast, work later

Providers wait a limited time for your answer. If your handler writes to three tables and calls another service before replying, a slow moment makes the request time out, the provider assumes it failed, and it sends the event again. The remedy is to do the minimum before replying: verify, write the raw event to a durable queue or table, and return 2xx. A separate worker takes events from the queue and does the real work.

// The handler does three things only
verify(request);
await queue.add({ id: event.id, body: raw });
return respond(200);

This also gives you a place to replay events when your own code has a bug. If the raw events are stored, a fix can be run against yesterday's data.

Make processing safe to repeat

Delivery is usually at-least-once. That means the same event can arrive twice, because your 200 got lost or a retry crossed the reply. Your processing must give the same result whether it runs once or three times. Store the event ID and check it before acting.

// Idempotent processing
if (await seen(event.id)) return;
await applyEvent(event);
await markSeen(event.id);

Do the check and the write in one database transaction if you can, or use a unique constraint on the event ID so the second insert fails harmlessly. Be careful with side effects that cannot be undone, such as sending a text message. Record that the message was sent against the call, and check the record first.

Do not assume order

A call produces a sequence of events: started, answered, ended. Your endpoint may receive "ended" before "answered" if the network or a retry reorders them. Design for that.

• Key your records on the call ID, and create the record on whichever event arrives first.
• Treat each event as setting its own fields: "answered" sets the answer time, "ended" sets the end time and duration.
• Use the timestamp inside the event to decide which update is newer, and ignore the time you received it.
• Do not derive state by counting events. Derive it from the fields that are present.

Plan for events you miss

Even a careful consumer will sometimes be down. An outage, a bad deployment or an expired certificate will make your endpoint unreachable for a while, and providers retry for a limited time and then stop. Decide in advance how you will fill gaps.

• Alert when no events arrive for longer than a normal quiet period for your traffic.
• Alert on a rising count of failed or rejected requests.
• Reconcile on a schedule by comparing your call records with the call list from the REST API, and fetching anything that is missing.
• Keep the raw events for long enough to investigate problems, and delete them on a schedule you have chosen.

A worked example

Okafor Logistics logs every dispatch call against a delivery in its own system. Its first consumer wrote straight to the database inside the request handler. During a busy morning the database slowed down, requests timed out, the provider retried, and some deliveries showed two "call ended" entries and a duration that was twice what it should have been.

The fix was small. The handler now verifies the signature, stores the raw event with its ID in a table that has a unique constraint on that ID, and returns 200. A worker reads the table and updates the delivery record, setting fields by call ID. A nightly job compares the previous day's calls from the REST API against its own records and fetches anything missing. The team found two missing events the first night, both from a deployment window.

A short checklist

1. Is the signature verified on the raw body, in constant time, before anything else?
2. Does the handler reply in well under a second, with the work moved to a queue?
3. Does a repeated event leave the system unchanged?
4. Does an event that arrives early or late still produce the right record?
5. Do you notice, within a day, when events stop arriving?
6. Can you replay stored events after fixing a bug?

Test the awkward cases deliberately in your sandbox account. Send the same event twice, send events backwards and stop your endpoint for ten minutes. If each of those behaves sensibly, production will rarely surprise you.