607 alert emails: how we stopped our exchange monitoring from crying wolf

On 22 October the inbox that receives our infrastructure alerts held 607 messages from the EX5 admin monitor. Almost none of them needed a person. The same few conditions opened, closed and opened again, and every change went out twice: once by email and once to the team’s Telegram channel.

An alert channel in that state is worse than no channel. People mute it, and the one message that matters arrives in a muted chat. This is what was making the noise, and the rules we changed.

One minute in, one minute out

The monitor runs once a minute. It checks node heights against what the deposit scanners have seen, container states, the message queue, the secrets store, the KYC provider and the price feeds. Until this month every rule had the same shape: if the condition is true on this tick, open an incident and notify; if it is false on the next tick, close the incident and notify again.

That works for a condition that is clearly on or clearly off. It falls apart for anything that sits near a threshold, and a surprising number of things do.

Where the 607 came from

A Solana lag threshold set in the wrong unit. The rule compared the chain head with the height our scanner had processed, in slots, and alerted above 50. Fifty Solana slots is about 11 seconds. Our scanner polls on an interval, so the head it sees moves forward once per round, roughly 280 slots at a time. For a few seconds after every round the “lag” read high, then dropped. The incident opened and closed all day, and about 12% of ticks landed in the high part of the sawtooth.

The threshold is now 1,500 slots. The rule for picking it: it must be well above one poll round (poll interval × about 4.5 slots per second), and low enough that a real stall still pages within minutes. Avalanche had the same sawtooth in blocks.

A planned reboot. We restarted the host at midnight for a kernel update. For a few minutes the secrets store was sealed, the message queue was starting, containers were restarting and some consumers had not reconnected yet. Each of those is a real incident if it lasts. During a reboot each one lasted a minute or two, and each produced an open and a close in two channels.

Price feeds for thinly traded tokens. Some tokens trade on exactly three venues, and our price rule wanted three sources. One venue having a bad minute meant “not enough sources”, then “recovered”. The price guards are their own story; we wrote it up in When three price sources are not enough.

A probe counted twice. Earlier in October the KYC provider check was reading a probe result cached for five minutes, on a one-minute tick, and counting one cached timeout as several failures. A single slow response from the provider became an alert, while customer verifications over the same 48 hours had zero errors. We fixed that one first, which is how we noticed the rest.

The rules we use now

Open after consecutive measured ticks. A warning needs 3 consecutive ticks before it opens, a critical needs 2. A single bad minute on the Solana scanner or during a reboot never reaches anyone.

Close after 3 clean ticks. Closing is held to the same standard as opening. A condition that flips back and forth stays one open incident instead of becoming ten pairs of messages.

“Could not measure” freezes both counters. If a check times out or the network to the node is down, the tick neither advances the count toward opening nor counts as clean. This was the most important change. Before, an open incident closed when the check itself failed, because “I did not see the problem” was treated as “the problem is gone”. Then the next successful check reopened it. That pattern alone accounts for a good share of the flapping we had, and it teaches people that a “closed” message means nothing.

Crash loops are counted over a window. Docker shows a container as “restarting” only during the back-off between crashes. A container in a crash loop is in that state on most ticks, not all of them, so a consecutive count can miss it. That rule now fires when it sees at least 3 restarting ticks in a window of 10. One or two restarts during a boot do not fire it.

Rules that already debounce opt out. Some checks already had their own streak or grace period, for example the KYC probe with its failure streak and the chain-unreachable rule. Adding another three ticks on top would have delayed real pages for no reason, so those rules open after one tick of their own, already-debounced signal.

One message per run for the price job. The price job used to send one message per token per problem. A bad minute at one venue could produce a dozen. It now sends one summary per run.

One retry before a source counts as down. The price fetcher retries once on HTTP 429, a 5xx or a timeout. Public market data APIs fail like that all the time, and a second request a moment later almost always works.

Which channel, for what

The other half of the fix was routing. We now follow one rule: email is for incidents that need a person, the chat is for everything else.

• An incident opening goes to email and Telegram.
• A “closed” notice goes to Telegram only. Nobody needs an email to learn that something they never looked at has fixed itself.
• Informational notices never go to email. We learned this one the hard way: the first time the price brake released itself automatically, the notice went out by email, and the first person to read “price brake released, FF −38.5%” took it for an error. It was the system doing exactly what we had built it to do.

How we test it

Every rule change is checked with a self-test that replays tick sequences against the hysteresis logic: a single bad tick, two bad then good, three bad, flapping, held ticks in the middle of an incident, a crash loop with gaps. A rule that is wrong in one of these cases fails the test before it can reach anyone’s inbox.

We did not change any of the conditions themselves in this round, apart from the Solana unit. The pricing rules and the other thresholds stayed the same. What changed is when a condition becomes a message, and where that message goes.

If your alert channel is muted

If people on your team have muted the alert channel, the channel is the problem, not the people. The checklist we would start with:

1. Count opens and closes per rule for a week. A few rules usually make most of the noise.
2. For every threshold, write down its unit and the sampling interval of whatever it reads. A threshold below one sampling step will flap forever.
3. Require consecutive ticks to open and to close.
4. Treat “could not measure” as unknown. Never close an incident on it.
5. Send closes and informational notices to chat, not email.
6. Plan reboots with the monitor in mind: either mute for the window or make sure the open rule outlasts a normal restart.

The goal is not fewer alerts. It is that every alert that reaches email is worth reading.