Reliable notifications: one HTTP call and everything behind it

The requirement fits on one line:

When an alert fires, the on-call engineer must find out.

And at first glance, so does the implementation:

slack.PostMessage(channel, text)

This post is about what happens when you add the word “reliably” to that line. Every answer opens the next question, and every next question is worse than the last. We walked this corridor end to end while rewriting delivery in TokayOps, and below are the doors in the order they opened.

Door one: where to

Before you send anything, you need an address. And the address of a notification is not a string in a config file.

A page is addressed not to a channel but to whoever is on call. The schedule says who that is, and the answer depends on the moment. Then the on-call is a person, and the message goes to their Slack or Telegram, so you need a linked account. It may not exist: the person just joined the team and has linked nothing yet.

Already here there are two different kinds of “it didn’t work”:

• nowhere to send: the person has no linked account, the integration is off, there is no token. No network call happened and none will; a retry changes nothing;
• the provider said no: the channel was deleted, the bot was kicked, the token was revoked. The call happened, and the answer is final.

Neither is a reason to try again in thirty seconds. Both are a reason for someone to find out that the page did not go out. Keep that in mind: “not sent” has to be visible, not just written to a log.

We have an address. We send. We get context deadline exceeded.

Door two: did it go out

The usual delivery model is binary: it worked or it didn’t. It holds up until the first timeout.

A timeout is not “it didn’t work”. It is “no answer came back”. The request might never have left, or it might have left in full, been processed, and Slack is already showing the message to the on-call - we just never saw the response. Those are two different facts, and they call for opposite actions.

The transport actually knows more than “error”:

definitely not sent   connection refused, DNS, TLS, canceled before the write
                      -> no effect, a retry is free
possibly sent         timeout after the write, connection dropped mid-flight
                      -> there may be an effect, a retry may send a second message
provider responded    -> what the response means only the provider's dialect knows

Call the first two cases both “error” and you have to pick one of two lies. Call everything “not sent” and you will silently count as lost the pages that actually went out. Call everything “unknown” and you will create duplicates where a duplicate is impossible.

So an attempt gets a third outcome - unknown. Neither failure nor success. And it is not rare: on a bad day, when the provider is slow, it is the most common one.

The second consequence is less pleasant. If the process dies between the POST and writing the result to the database, the effect exists outside and leaves no trace inside. So the attempt is recorded before the call. Every call that may have happened has a row saying it may have happened. The converse does not hold: the process may have died before the call, too. That means an abandoned attempt cannot be closed as “not sent” - its honest outcome is also “unknown”.

So we don’t know. Try again?

Door three: how many times

The first thing everyone writes is a counter:

MaxAttempts: 3

We had two of these, 3 and 8, in two different places, and neither meant anything. A counter answers “how many times to try”, but the real question is different: “when does this page stop being needed”.

And that question has no numeric answer. A page is needed until the incident is acknowledged. It stops being needed when it is acknowledged, when the alert resolves, when an operator decides to cancel it. Or when it has expired: a “your shift has started” message delivered two days late does more harm than one never delivered.

A counter knows nothing about whether the page is still needed. It just runs out. In our case, after the third attempt the job got the status failed, and that was it: there was no alert on it, and the UI did not show such jobs.

So we separated frequency from lifetime. How often to knock is decided by a policy: exponential backoff with jitter and a ceiling. How long the system owes the delivery is decided by domain conditions: delivered, canceled, expired, decided by a human.

We try again. But the previous attempt ended in “unknown”, and it may have arrived.

Door four: the duplicate

A retry after “unknown” can produce a second message. The usual defense is an idempotency key - and this is where you find out that half the providers don’t accept one.

A webhook can: every retry carries the same X-Tokay-Event-ID, and the subscriber drops what it has already seen. Slack and Telegram can’t: sending a message has no parameter by which they would recognize a retry. For them a duplicate after “unknown” is not a bug but part of the contract.

So you have to choose which way to err. And there is no single right answer:

• a page to the on-call: a duplicate is cheaper than a loss. Waking someone twice is annoying; not waking them is an incident;
• a phone call or an SMS: a duplicate costs money and patience;
• a command to an external system, starting something expensive: a duplicate can be dangerous.

So “which way to err” is a property of the kind of message, not a global setting. For notifications: retry and accept a duplicate. Where a duplicate is dangerous: stop and call a human.

Where a key exists, it is also easy to get wrong. It feels natural for the key to change with the content: the message changed, so a new key. But then an unknown send of version one, retried as a send of version two, reaches the provider with a new key - and the provider rightfully creates a second object. The key has to be stable within one external effect, not within one version of the text.

Fine, we retried, and the provider answered 200 OK. Done?

Door five: “accepted” is not “delivered”

For Slack, Telegram and a webhook, almost. The provider accepted it, and the message is in the channel.

For SMS and phone calls, no. A gateway that answered 200 has put the message in a carrier’s queue. Between acceptance and delivery sits someone else’s system and minutes of time, and the provider will tell you the result later, in a separate callback. If it tells you at all.

We don’t have those channels yet, but the model has to allow for them from the start: the outcome of an attempt (“the provider accepted it”) and the fate of the notification (“it arrived”) are different things. Otherwise the first SMS channel will force a rewrite of everything that reads the word “success”. And separately: a delivery callback can arrive before you have recorded the id of the message it refers to. So an incoming event is stored first and matched second.

We sent it. A minute later the on-call pressed “Acknowledge”. The card has to change color.

Door six: the message has to be updated

A page in a channel is not one message but a card that lives alongside the incident. A new alert arrives - it has to appear on the card. Someone acknowledges - the card turns yellow. The incident is resolved - it turns green. Under the card there is a thread with the history, and at the end a “Resolved” reply goes into the thread, because editing a message is silent, while a new reply in a thread notifies everyone who has posted in it.

This is where it gets harder.

Updating something that may not exist. Creating the card ended in “unknown”. We don’t have the message id. An ack arrives. What do we edit?

The answer that worked for us: don’t keep a queue of edits, keep the desired state. The card has a revision number and a snapshot of how it should look. Every attempt sends the latest revision. If there are no message coordinates, it is a create, and the card is created already yellow. If there are, it is an edit. An ack during “unknown” does not produce a separate operation; it just moves the target.

The price was named above: if the “unknown” create actually went through, there will be two cards in the channel. Slack does not deduplicate.

Two writers of one card. The Acknowledge button in Slack used to redraw the card itself - fast and pleasant. At the same time a background worker was applying the revision with a new alert. There was no order between them:

an ack and a new alert arrive almost together
the worker applies the revision with the new alert
the button's response lands after it, rendered BEFORE the alert
-> the alert disappears from the card until the next edit

The only cure is a single owner. The button now answers the person who pressed it with a confirmation, and the worker recolors the card. The person sees the answer immediately; the card turns yellow a second later. That second is what it costs for the card to always be correct.

A success that looks like an error. Telegram answers an edit that changes nothing with 400 message is not modified. Without message coordinates. If the rule says “accepted without coordinates means unknown”, and unknown is retried, you get an endless loop: edit, 400, unknown, retry, 400. In fact the provider is saying the best thing it can: what is out there is already exactly what we want. That is a success. Response classification has to know what kind of call it is classifying: the same response means different things for a create and for an edit.

A thread before its card. A reply in a thread has to be addressed to a message that already has an id. Until the card is out, the thread waits - and does not count as late, otherwise the “page is late” alert would fire because of the thread rather than the page. If the card never goes out, the thread is closed with a clear reason instead of hanging forever.

A link to something that doesn’t exist yet. The direct message to the on-call should link to the card in the channel. The card and the DM are sent at the same time, and a DM with zero delay started about 20 ms before the card got its id. On a test environment that gave zero DMs with a link out of seventeen. “Sometimes without a link” turned out to be “never with a link” in practice. Now the DM waits for the card’s first attempt and goes out after it: with a link if the card went out, without one if it didn’t.

All of this is one machine in one process. And there are two processes.

Door seven: the second instance

TokayOps runs as several instances at once, and each of them sends notifications. So one notification can be picked up by two workers.

The standard answer is a lease. A worker takes the notification in the database for a while (FOR UPDATE SKIP LOCKED, a lease expiry), and the others don’t see it. The lease expired - so the worker died, and the notification can be taken again.

“So the worker died” is not true. The worker may not have died but frozen: a GC pause, an overloaded node, a provider that takes 40 seconds to answer. The lease expired, the second instance took the notification, and then the first one woke up with a response in hand.

Three rules follow.

A lease has a number. Every time a lease is granted it gets a new random token, and a result can only be written with your own token. The worker that woke up comes with the old token and hits a compare-and-set. This is called fencing, and without it a “lease” is only a convention that nothing enforces.

Time comes from the database. Whether a lease has expired is decided by the clock of one machine - the database. The clocks of two instances drift, and “not expired yet” on one with “already expired” on the other means two owners of one notification.

An attempt taken over mid-send is “unknown”. If the lease expired when the attempt was already recorded but had no result yet, the call may have happened. The second worker first closes the hanging attempt as “unknown”, and only then decides what to do next. Not “not sent”, and not “retry as if nothing happened”.

And the first worker’s late response is not an error. The provider answered, slowly, and that answer is worth keeping in the attempt’s history: it explains where the duplicate came from. Treating it as a violation would mean raising a “there is a bug here” alert every time Slack is slow.

One less obvious detail: what a taken-over attempt turns into depends on the kind of message. A page is retried. Something that must not be duplicated stops and waits for a human.

Door eight: the human

After all the rules there is a remainder - notifications the machine honestly doesn’t know what to do with. “Unknown” where a duplicate is not allowed. A permanent rejection. An expired deadline.

For these, a human has one decision with three options:

treat as delivered    the risk is recorded explicitly
cancel                stop trying
send again            new address, new key, new external effect

In the UI, “delivered” and “an operator decided to treat it as delivered” have to look different. In an incident review, someone will ask.

“Treat as delivered” after the provider has said no is not allowed. That is no longer an accepted risk, it is simply false.

What you can actually promise

After the corridor you end up with a statement you can put in the documentation without embarrassment:

An accepted notification does not disappear silently. Retryable and ambiguous notifications are kept and retried until success, explicit cancellation or expiry. A delivery that cannot be fixed gets a visible terminal status, an alert and an operator decision. Where a provider does not support idempotency, an ambiguous result may produce a rare duplicate, and the reason for it is visible in the attempt history.

There is no exactly-once here: nobody can guarantee it on top of someone else’s API. And there is no word “delivered” either. 200 OK means Slack accepted the message, not that a person read it.

The whole corridor

This is what the single line from the beginning turned into:

slack.PostMessage(channel, text)

 where to          schedule, linked account; "nowhere" is not a network error
 did it go out     three outcomes; the attempt is recorded before the call
 how many times    a lifetime, not a counter; giving up means saying so
 duplicate         a key per effect, not per version; which way to err, by kind
 accepted          is not delivered; a callback can overtake the id
 update            desired state, one writer, a success with status 400,
                   a thread and a link to what doesn't exist yet
 two instances     a numbered lease, time from the database, takeover = unknown
 the human         one decision, the risk recorded, "no" never becomes "yes"

None of these doors was invented in advance. Each one opened because the answer to the previous question wasn’t enough. If you are building a system like this from scratch, you will walk the same corridor anyway. The only question is whether you do it on paper or during an incident.