When three price sources are not enough: guarding an exchange's reference price

Every asset listed on an exchange has a reference price in some base currency, usually USD. It is not the trading price; the order books set that. It is the number the back office uses. On the EX5 stack it drives two things that move money: withdrawal limits expressed in USD, and the hot, warm and cold wallet thresholds that decide when funds are swept from the hot wallet to colder storage. A wrong reference price either lets a customer withdraw far more than their limit, or makes the treasury sweep the wrong amount.

So the job that sets these prices is built to distrust its inputs. In October two of its safety rules turned out to be stricter than the market, and both failed the same way: they froze a price that should have moved.

How the price is built

Every 15 minutes the job asks several independent sources for each asset’s USD price: exchanges with public market data plus an aggregator. For BTC seven sources answer, and the spread between them is typically around 0.1%.

From those quotes:

1. Take the median. One broken source cannot move a median at all. That is why we use it rather than an average or a “two sources must agree” rule: with two, you cannot tell which one is wrong.
2. Reject outliers. Any quote more than 2% away from the median is dropped, and the median is recomputed from the rest. The distance of each source from the median is also the health signal the admin screen shows per source.
3. Require enough sources. If fewer than three quotes survive, nothing is written.
4. Apply the deviation brake. If the new price differs from the last written price by more than 15%, nothing is written. One bad tick, a delisted pair returning a stale number, a decimal error at a source: none of them can change the price by more than 15% in a run.

The arithmetic happens on exact decimal strings. The value ends up in a fixed-precision database column, and nothing on the way passes through a floating-point number.

All four rules made sense. Two of them broke in the first week of real use on smaller tokens.

Problem one: a token that trades on exactly three venues

Some of the tokens listed on EX5 trade on exactly three of the sources we query. GNO is on Binance, Coinbase and CoinGecko’s aggregate; DEXE and FF on Binance, KuCoin and CoinGecko. With a minimum of three, those tokens had no margin at all. One venue failing to answer, or one quote landing just outside the 2% band, left two sources, and the job refused to write.

Each refusal also raised an alert: “insufficient price sources”. Then the venue answered on the next run and the alert closed. That fed the flood of alert emails we described in 607 alert emails.

The question we asked was what a third source actually adds. The answer is arbitration: when two sources disagree, the third tells you which one is wrong. When two sources agree, there is nothing to arbitrate.

The rule is now:

• Three or more sources: as before.
• Exactly two sources whose spread is within the 2% outlier tolerance: enough.
• Two that disagree beyond 2%, or a single source: still blocked.

The deviation brake still applies to every write, including two-source ones. A two-source price that jumps 20% is blocked exactly as before.

Problem two: a brake with no release

The brake compares the new price with the last price we wrote. That is the right reference for catching a bad tick. It is also a trap: if the market really moves more than 15% and stays there, every later run compares against the old written price, sees a move of more than 15%, and blocks again. Forever.

There was no approve button. The only way out was for someone to notice the frozen price and edit that currency’s policy by hand. On a small token after a real crash, the reference price could stay at the pre-crash level until someone noticed, and with it the withdrawal limits and sweep thresholds calculated from it.

The fix treats persistence as the evidence. A blocked move is released when:

• it has been blocked for 4 consecutive runs (about an hour at 15 minutes),
• every one of those runs saw a move in the same direction,
• and every one of those runs saw a price within 5% of the current value.

The runs must really be consecutive: if the oldest of them is older than the expected window, the history is stale and nothing is released. When the conditions hold, the price is written, the release is recorded in the audit log, and a notice goes out.

The first real release came within hours of deploying the rule: FF had dropped 38.5%, from 0.1739 to 0.1069, and stayed there. An hour of consistent readings released it. A single bad tick cannot do that, because it would have to repeat itself four times, from several sources, inside a 5% band.

The notice that looked like an error

The release notice originally went out by email. The first person to read it, seeing “price brake released, FF −38.5%”, assumed something had broken. Nothing had. It was the rule working.

Release notices are now sent to the team chat only. Our general rule since then: email is for things a person must act on; information goes to chat. We wrote about that separation in the alerts post linked above.

What we would tell anyone running a price job

• Write down what the price is used for. If it touches limits or treasury thresholds, it needs guards. If it is only display, fewer guards are fine.
• Use the median, then reject outliers against it. Averages let one broken source drag the result.
• Set the minimum source count per asset, not globally. A token listed on three venues cannot meet a minimum of three on a bad day. Two sources that agree are better than a frozen price.
• Every brake needs a release. Decide what evidence counts as “the market really moved” (time, direction, consistency) and release automatically when you see it. A brake that only a person can release will eventually be forgotten.
• Record every blocked and released run. When someone asks why a price did not update, the answer should be one query away.