Blog

Engineering 12 min

On-call scheduling from scratch: what to decide before the first line of code

An on-call schedule looks like a function of the current moment and behaves like a function of its own history. The design decisions that follow from that: revisions, phase anchors, calendar arithmetic, and overrides.

On Thursday afternoon somebody opened the schedule settings, hit Save without changing anything, and on-call jumped from Cara to Ana.

The schedule was built the way schedules usually get built on the first evening:

people  = [Ana, Ben, Cara]
week    = weeks_since(config_created_at, now)
on_call = people[week % 3]

The position in the queue was never stored. It was derived from the age of the config. The config was created on Monday, 3 August 2026; by Thursday the 20th two full weeks had passed, 2 % 3 = 2, so Cara was on call.

Saving created a new config with a new creation time. Its age was zero, 0 % 3 = 0, and Ana was on call.

The patch is obvious: do not recreate the config when nothing changed. Behind it stands a queue of further questions. What happens when someone is added in the middle of another person’s shift? When the handoff time moves? Should today’s edit change yesterday’s calendar? Who gets woken on the night the clocks change?

An on-call schedule looks like a function of the current moment and behaves like a function of its own history. Almost every expensive mistake here starts with designing for the first when you need the second.

One schedule, three questions

The system has to answer three different questions.

Who is on call now? Incident routing needs this.

Who was on call then? Incident review and audit need this. Today’s edit must not change yesterday’s calendar.

Who will be on call? People need this: “can I fly out on the twentieth.”

The naive model answers only the first question, and only until the first edit. History is usually remembered after an unpleasant incident, and it cannot be backfilled: whatever was not recorded is gone.

So instead of one mutable config you need a chain of immutable revisions:

rev 1: [3 August, 20 August 14:00)   old rules
rev 2: [20 August 14:00, infinity)   new rules

At exactly 14:00 the second revision is already in force. To find who was on call at time T, the system picks the revision that was in force at T, computes its rotation, and applies temporary overrides on top.

revision + overrides + moment T -> assignments

The same computation has to answer for the current on-call, the past calendar, and the forecast. Otherwise three versions of the logic will inevitably drift apart.

The queue element is not a person

Teams change, and a schedule lives for years. There is never a convenient moment to edit it: somebody is always on call.

Say Ben leaves and Dan comes in to replace him:

replace in place:  [Ana, Ben, Cara] -> [Ana, Dan, Cara]
remove and append: [Ana, Ben, Cara] -> [Ana, Cara, Dan]

In the first case Dan inherits Ben’s position. In the second, Cara slides into someone else’s position and all of her future shifts move by a week.

An array of names does not distinguish these intents. Which means the queue element is not a person but a position with a stable ID that currently holds someone. A position can hold several people, so it is usually called a group.

A stable ID lets you tell three operations apart:

  • change the membership of an existing group;
  • reorder groups;
  • delete one group and create another.

The model passes the stress test if adding a person to the active group does not shift the queue, a replacement inherits the previous position, and an empty save changes nothing.

Rule and phase

The bug from the top of this article is cured by separating two kinds of state.

The rule describes the cycle: the groups, their order, a daily or weekly cadence, the handoff time, and the time zone.

The phase ties the cycle to the calendar: the moment of one handoff and the position of the group that took over at it.

phase anchor = (Monday 17 August, 11:00; position 2 - Cara)

This is a mathematical anchor, not a record of a real handoff: it may sit earlier than the start of the revision, and the revision still applies only from the moment it was created.

Who is on call at T is computed like this: count the calendar handoffs between the anchor and T, then step that many positions through the queue.

An ordinary edit copies the phase. Change a Slack group or add a person to Cara’s group, and the same group stays active.

Changing the grid itself requires a new anchor. One reasonable contract:

  • handoff time moved - keep the current group on the new grid;
  • active group deleted - pick the next surviving one;
  • queue rebuilt from scratch - choose explicitly who goes first.

These are not universal rules, they are product decisions. If you do not write them down in advance, the decisions will appear anyway, in the arbitrary order of if statements in the code.

The phase can be stored differently: keep the current on-call and the time of the next handoff, and let a scheduler flip them on a timer. For answering “who is on call now” that is simpler. The bill arrives elsewhere: the flip has to survive downtime, must not fire twice, and has to leave enough data behind for the past and the future.

A calendar is not a duration

People say “tomorrow at 11:00” and “next Monday”. Code quietly substitutes +24h and +168h.

In 2026 a handoff on 28 March at 11:00 Berlin time happens at 10:00 UTC. The next day Germany moves to summer time, and 11:00 in Berlin is now 09:00 UTC. There are 23 hours between the handoffs. The +24h version produces 12:00 local, and the handoff drifts by an hour until autumn.

With a weekly rotation even the position count breaks. Between Monday 23 March at 11:00 and the following Monday at 11:00 there are only 167 hours. The formula (T - anchor) / 168h returns zero whole weeks at the moment of the actual handoff. The pager keeps waking the previous shift for another hour.

The correct algorithm works in the local calendar: take the next date, put 11:00 on it, and only then convert the result to UTC.

But not every local time exists. At the spring transition Berlin jumps from 02:00 to 03:00, so 02:30 does not exist there on 29 March 2026. In autumn the hour from 02:00 to 03:00 repeats, so 02:30 on 25 October names two different moments.

For a shift handoff this is a rare case: handoffs are usually scheduled during the day. But a temporary override can be started by a user at any hour. A non-existent time has to be rejected, and for the repeated hour you have to ask which of the two 02:30s was meant, otherwise the library will silently pick one for you.

Sometimes a whole date disappears. In 2011 Pacific/Apia crossed the date line: 29 December was followed immediately by the 31st.

“30 December, 11:00” simply does not exist there. A library will silently turn it into the 29th or the 31st, and the grid gets a duplicate or an empty slot. Code that stubbornly asks for the 30th will loop forever.

Testing this kind of code only in UTC is useless. Berlin catches +24h, Australia/Lord_Howe catches the assumption that clocks always shift by exactly an hour, and Pacific/Apia catches mishandling of a vanished date.

Editing mid-shift

Cara’s shift started on Monday at 11:00. On Thursday at 14:00, Eve was added to her group. From when is Eve on call? From 14:00. When did the shift start? Monday at 11:00.

rotation slot:  Mon 17th 11:00 - Mon 24th 11:00
assignment:     Thu 20th 14:00 - Mon 24th 11:00

The first boundary is what the next handoff needs. The second is what audit and the “on call since” line need. The existing members do not need another “your shift started” message, but Eve may need a separate “you were added to the active shift”.

An override is a layer on top of the rotation

Vacation or illness does not change the queue. They temporarily mask the result of the rotation:

08:00 - 13:00  rotation
13:00 - 16:00  override
16:00 - 20:00  rotation

The renderer builds the rotation first, then cuts the override interval out of it. That is why an override does not take part in computing the next position and survives a config edit underneath it.

The past, meanwhile, cannot be un-happened:

  • a future override can be cancelled;
  • an active one can be ended now, preserving the part that already elapsed;
  • a finished one cannot be rewritten.

Audit needs two times: the interval it applies to and the moment it was recorded. You do not have to answer “what did the system know a week ago” on day one, but versions must not be overwritten: an edit adds a new record, while the old one keeps its interval, author, and reason.

Read and write as a whole

It is convenient to keep the historical renderer a pure function:

(revisions, overrides, range) -> assignments

All of its inputs have to be read from a single database snapshot. PostgreSQL never shows half a transaction, but under READ COMMITTED two consecutive SELECTs can see different committed states. Mixing state N with state N+1 yields a version N+1/2 that never existed in the database. A short read-only REPEATABLE READ transaction gives the renderer one snapshot.

Saving is not a set of CRUD calls either, but a single command:

  1. validate the expected version and the input;
  2. decide what happens to the phase;
  3. close the current revision and insert the new one in one transaction.

After normalization, identical configs produce a no-op: no revision, no change of on-call, no notifications. A concurrent save gets a version conflict rather than a silent last-write-wins.

The database backs up the code: a unique index allows only one open revision, and an exclusion constraint on the intervals prevents two revisions from being in force at the same time. A future bug becomes a loud failure instead of quiet corruption of history.

Machines read the schedule

The schedule is used by incident routing, escalations, handoff notifications, and chat group sync. All of them have to get their on-call from one computation, not assemble their own versions of the logic from raw tables.

The notifier has a trap: on its first tick it has no previous state, so creating a schedule is indistinguishable from a handoff. Notifying everyone or silently recording the state are both acceptable, but the choice has to be explicit.

And the most important one: “nobody is on call” and “we could not find out” are different results. A database error turned into an empty list sends the incident into silence.

someone is on call
genuinely nobody is on call
could not get an answer

The last state has to be handled as an error, loudly.

Sometimes the honest answer is “I don’t know”

History may have started being recorded after a migration. Then the boundary has to be stored explicitly: history_complete_from = 2026-05-01T00:00:00Z.

Before it, the honest answer is “I don’t know”. After it, the system owes you a complete history. If two revisions both claim the same moment, that is no longer a caveat but data corruption: better to refuse to build the calendar than to show a plausible lie.

What to decide before the code

  1. Do you need the answer to “who was on call then”?
  2. What is the stable identity of a group?
  3. Which edits preserve the phase and which create a new one?
  4. Which zone is the schedule computed in, and which one is time displayed in?
  5. How are future, active, and finished overrides edited?
  6. What counts as a no-op?
  7. How is concurrent editing resolved?
  8. How do you tell “nobody is on call” from “we could not find out”?
  9. What does the system answer before the start of trustworthy history?

The minimal test set follows directly from these decisions: a save with no changes, a membership edit mid-shift, two concurrent writes, an override on a revision boundary, clock-change days, Pacific/Apia, and a failure to read the on-call.

Why this gets noticed late

Almost every wrong decision looks like it works at first. +24h breaks twice a year, and then stays wrong every day until the next transition. Deriving the position from the age of the config breaks only on save. Overwriting the old version breaks on the first question about the past.

The bill arrives months later, and it picks its own moment: clocks change at night, and incidents happen at 03:14.

Back to Cara and Ana. An empty Save is now a no-op. Adding Eve to the active group creates a new revision but preserves the phase: Cara and Eve are on call, and on Monday the shift moves to Ana. In the past calendar there is still no Eve.

Behind picking the next name in a circle there turned out to be a system of facts: which rules were in force, where the queue stood, and which exceptions had been agreed. That is what you need to define before the first line of code.