Blockchain RPC failover: what a normal load balancer gets wrong

The deposit scanners and withdrawal signers behind the EX5 exchange do not talk to blockchain nodes directly. They talk to a small gateway that keeps a pool of RPC endpoints per chain, measures each one, and forwards every request to the best endpoint available at that moment. Six chains go through it: Ethereum, BNB Chain, Avalanche, TRON, Solana and the XRP Ledger.

A plain HTTP load balancer with health checks looks like it could do this job. It cannot. Blockchain RPC has failure modes that look like success at the HTTP level, and successes that look like failures. And one class of request, broadcasting a signed transaction, must never be retried the way a load balancer retries everything else.

These are the rules the gateway follows, and most of them came from something going wrong first.

Scoring endpoints

Each endpoint has a score that decides the order in which it is tried:

• Success rate as a moving average. Every success or failure updates an exponentially weighted average (weight 0.3 for the newest result). New endpoints start optimistic.
• Latency against the pool. An endpoint slower than the pool’s median latency loses up to 20 points. Being fast does not earn extra; being much slower than its peers costs.
• Staleness. An endpoint whose block height is behind the pool’s median by more than the chain’s threshold is taken out of normal rotation. A fast endpoint that is three blocks behind is worse than a slow one that is current: a deposit scanner reading it will think the chain is quieter than it is.
• Operator weight. A multiplier for endpoints we trust more, for example a paid provider over a public one.

On top of the score:

• Circuit breaker. Five consecutive failures, or the success average dropping below 0.2, opens the circuit for 30 seconds. After that, three successful probes close it again.
• Quarantine for new endpoints. A newly added endpoint must pass 20 consecutive probes before it receives real traffic.
• Back-off on rate limits. An HTTP 429 sends the endpoint into a cooldown that starts at 5 seconds and doubles up to 60. Public endpoints rate-limit constantly; treating a 429 as a failure would wreck their score, and ignoring it would keep hammering them.

None of this is unusual. The interesting part is deciding what counts as a success and what counts as a failure.

An HTTP 200 can still be an outage

The XRP Ledger node software reports many of its own problems inside a normal HTTP 200 response: tooBusy, noNetwork, noCurrent and a few others mean “I am up, but I cannot answer you correctly right now”. Some public providers do the same with quotas: a 200 whose body says the request allowance is used up. A gateway that looks only at status codes sees a healthy endpoint and keeps sending traffic to it.

For a while ours did exactly that. The XRP pool kept routing to a node that answered every request with a polite error, and the monitor reported the whole chain as unreachable, although the pool had other endpoints that would have answered. The gateway now parses those response bodies per protocol, and treats the node-side errors and in-body quota messages as endpoint failures that trigger failover.

A revert is an answer, not a failure

The opposite mistake is just as easy. When a contract call reverts, the node returns a JSON-RPC error: “execution reverted”. That is the node doing its job. The request was wrong, or the contract said no. Sending the same request to another endpoint will get the same answer, and counting it against the endpoint punishes a healthy node for the caller’s mistake.

So application errors pass straight through to the caller and are never retried, with one exception: errors that mean “this node does not have that history”. A pruned node asked about an old block, an XRP server without the requested ledger, a node that has cleaned up old state. Those requests are routed to an endpoint that is known to keep full history, and the original endpoint is not penalised.

Never send a signed transaction twice by accident

Reads can be retried freely. A broadcast cannot be treated the same way. Once a signed withdrawal has reached a node, it may be in the mempool even if the response never came back.

The gateway sends a broadcast to one endpoint, the best one. It retries on another endpoint only when it is certain the transaction never left: a connection or TLS failure before the request body was sent, or an explicit 429/403 refusal. Everything else is ambiguous: a read timeout, a connection reset after sending, a 5xx, or any response at all. For those, the gateway does not retry on its own.

On all the chains we run, a re-broadcast of the same signed transaction is deduplicated by the network anyway (nonce, transaction ID or signature), so a duplicate send would not pay twice. We still treat it as a rule rather than relying on that, because the systems above the gateway (withdrawal state machines, reconciliation) are much easier to reason about when one broadcast means one attempt. Every broadcast attempt writes its own log line.

Wrong network: proven, or just unverified?

An endpoint that answers for the wrong chain is the most dangerous failure of all. A testnet endpoint in a mainnet pool, or a chain with a similar API behind the wrong URL, returns valid-looking blocks that are not the blocks you think. A 502 is far safer than an authoritative wrong answer.

So pools that can identify themselves are checked: EVM chains by chain ID, Solana by genesis hash. We keep three outcomes apart:

The endpoint says	Meaning	What happens
The expected chain ID	verified	normal traffic
A different chain ID	proven wrong network	removed from all traffic, even as a last resort
An error, no result, or nothing at all	not verified	previous result kept; a new endpoint is checked again every 15 seconds until it answers

Two bugs taught us to keep these apart. First, the check ran only for a hard-coded list of chains, so the Avalanche pool had an expected chain ID configured that was never actually checked. Second, on the chains that were checked, a 200 with a rate-limit error in the body counted as “wrong network”. The endpoint dropped out until the next hourly check. In a pool of three, a restart during a busy minute could have left one chain with no endpoints for an hour. “Did not tell us who it is” is not the same as “told us it is someone else”.

One chain-specific case belongs here too. Some TRON endpoints are API-only nodes: they answer simple reads but leave out fields that building a withdrawal needs. A withdrawal routed to one of them fails at signing time. The gateway detects these nodes and excludes them from all TRON traffic, not just from withdrawals, so no call can land there by accident.

The admin API is a money path too

The gateway has a small admin API: add an endpoint, change its weight, test a URL. During a security review of our custody stack we followed that API to its end and did not like what we found.

Anyone holding the admin key could add their own endpoint with the highest weight, making it primary for a chain. That endpoint could then serve invented blocks containing a deposit that never happened. The deposit scanner would credit it, and the customer could withdraw real funds against it. We found this by reading the code, not because anything had happened. We closed it anyway:

• Host allowlist per chain. New endpoints can only be added on reviewed hosts. Adding a host is a reviewed configuration change and a restart, not a click in a panel.
• Weight cap. Weights added at runtime are capped at 20. An existing endpoint can be lowered, never raised above the cap.
• Parse URLs the way the HTTP client does. The host is checked with the same parser the HTTP client uses, and a URL whose host two parsers read differently is rejected. Parser differences are a classic way around allowlists.
• No requests to arbitrary URLs. The “test this URL” action no longer probes hosts outside the allowlist, so it cannot be used to reach internal services.
• Never log the full URL. Paid RPC providers put the API key in the URL path. Refused and accepted requests log only the host.

What we would check in any RPC setup

• Does your health check read the response body, per protocol? A 200 is not a health signal on its own.
• Are application errors (reverts, invalid params) passed through instead of counted as endpoint failures?
• Do you compare each endpoint’s block height with the pool, not just its uptime?
• What happens when a broadcast times out? If the answer is “the load balancer retries”, find out where the retry goes.
• Do you verify chain identity, and do you keep “wrong network” separate from “could not verify”?
• Who can add an endpoint in production, and what is the highest weight they can give it?

None of this needs a large system. Our gateway is one small service per host. The work is in the failure semantics, and most of those you only learn by watching a chain misbehave at 3 a.m.