Jitter strategies to avoid fleet-wide retry storms

Exponential backoff is correct for one client and wrong for four hundred of them. Every device applying the same doubling schedule to the same triggering event retries in unison, and the resulting spikes are what turns a five-minute tower outage into a forty-minute one. The fix is a few lines of randomisation, and the choice among the available strategies changes both the peak load and the mean recovery time. This guide compares them for a field fleet, inside retry and backoff for unstable networks and the Bandwidth & Async Sync Optimization guide.

Four strategies #

Let base be the initial delay, cap the maximum, and n the attempt number. Write window = min(cap, base * 2**n).

No jitter sleeps window. Every device that failed at the same moment retries at the same moment, forever.

Full jitter sleeps uniform(0, window). This is the strongest de-correlator: the expected delay is halved, the spread is maximal, and two devices that started together are almost immediately independent.

Equal jitter sleeps window/2 + uniform(0, window/2). It keeps a guaranteed minimum delay, which matters when the retry itself is expensive — a radio wake, a TLS handshake — and still spreads the second half.

Decorrelated jitter sleeps uniform(base, previous_sleep * 3), capped. It grows from the previous actual sleep rather than from the attempt number, which produces a smoother distribution and avoids the synchronised doubling entirely.

The implementation #

# jitter.py — the four strategies, one interface, no shared state.
# Pure functions plus a small stateful class for the decorrelated variant.
# Uses random.random(); on a device where every unit must differ, seed from
# the device id at start-up rather than from the clock — devices powered on
# by the same relay share a clock to the second.
import random


def no_jitter(base: float, cap: float, attempt: int) -> float:
    return min(cap, base * (2 ** attempt))


def full_jitter(base: float, cap: float, attempt: int) -> float:
    """uniform(0, window). Strongest de-correlation, halves the mean delay."""
    return random.uniform(0.0, min(cap, base * (2 ** attempt)))


def equal_jitter(base: float, cap: float, attempt: int) -> float:
    """Half the window guaranteed, half randomised. Use when the retry itself
    is expensive and a near-zero delay would waste a radio wake."""
    window = min(cap, base * (2 ** attempt))
    return window / 2.0 + random.uniform(0.0, window / 2.0)


class DecorrelatedJitter:
    """Grows from the previous actual sleep rather than the attempt number.
    Smoothest arrival distribution of the four; no attempt counter needed."""

    __slots__ = ("base", "cap", "prev")

    def __init__(self, base: float = 1.0, cap: float = 300.0):
        self.base, self.cap, self.prev = base, cap, base

    def next_delay(self) -> float:
        self.prev = min(self.cap, random.uniform(self.base, self.prev * 3.0))
        return self.prev

    def reset(self) -> None:
        self.prev = self.base


def seed_from_device(device_id: str) -> None:
    """Two devices powered on by the same relay have the same wall clock to
    the second. Seeding from the clock therefore does NOT de-correlate them;
    seeding from the identifier does."""
    random.seed(hash(device_id) & 0xFFFFFFFF)

The seeding note is the one that catches people. A fleet installed at one site, powered from one supply, and rebooted together will produce identical “random” sequences if the generator is seeded from the time — which is precisely the fleet where the correlation matters most. Seeding from the device identifier makes every unit’s sequence deterministic, reproducible for debugging, and different from its neighbours.

Choosing among them #

The one that is never right on a fleet is no jitter, and it is the default in most retry libraries.

Beyond the retry loop #

Jitter is not only a retry concern. Any periodic action a whole fleet performs is a candidate for synchronisation, and several of them are more damaging than reconnection because they are not triggered by a shared failure — they are scheduled that way by design.

Scheduled sync windows. A fleet configured to sync “every hour on the hour” produces an hourly spike forever. Offset each device by a stable per-device value derived from its id.

Certificate rotation. Devices provisioned in one batch share an expiry, and a fraction-of-lifetime trigger fires for all of them within a day. The credential rotation guide covers the same fix.

Update checks. A device polling for updates on a fixed schedule joins the same crowd. Jitter the poll, and gate the download on a link class as well.

Metric scrapes and pushes. A push-based metrics client on a fixed interval synchronises exactly like everything else, and its spike lands on the collector rather than the broker.

The general rule: any timer that runs on more than one device should carry a per-device offset derived from its identifier. It costs one line at initialisation and it removes an entire class of load spike that is otherwise diagnosed as a platform capacity problem.

Where the storm actually forms #

It is worth being precise about the mechanism, because the intuitive explanation — “too many devices at once” — misses the part that makes it self-sustaining.

A broker or an API gateway has a finite accept rate. When arrivals exceed it, the excess connections are refused or time out. Each of those failures is itself a retry trigger, and because the failing devices all failed at the same instant, they schedule their next attempt from the same instant too. The spike does not decay; it re-forms at the next interval, slightly larger because devices that succeeded on the first spike have now dropped out and the rest are still synchronised.

That is why the recovery in the chart above takes forty minutes rather than five: each doubling produces another synchronised wave, and the waves keep failing until the fleet has thinned enough for one to fit under the accept rate. Jitter breaks the loop at the first step by ensuring the failures are not simultaneous, so the next attempts are not either.

The same reasoning explains why a longer backoff does not fix an unjittered fleet. Doubling the base delay halves the number of waves and leaves each one exactly as large; the peak, which is what exceeds the accept rate, is unchanged. Only spreading the arrivals reduces the peak.

Two secondary effects are worth knowing. The thundering herd hits TLS hardest — a handshake is far more expensive on the server than an established connection, so a spike of reconnections costs disproportionately more than the same number of publishes. And the spike is worse after a long outage than a short one, because a longer outage means more devices have accumulated in the same retry state, all with their windows capped at the same maximum. A fleet that has been offline for six hours with a 300-second cap retries in perfect five-minute unison.

Verifying it before the fleet does #

Jitter is easy to add and easy to add wrongly, and the failure is invisible on one device. Two checks catch it.

The first runs at build time: instantiate the retry policy a thousand times with different device identifiers, collect the first three delays from each, and assert the standard deviation across devices is a meaningful fraction of the mean. A policy that produces the same sequence for every identifier — because the seed was taken from the clock, or because a library caches its generator — fails this immediately.

The second runs in a simulation: replay a fleet-sized set of clients against a mock server with a fixed accept rate, and record the arrival histogram. The number to look at is the peak arrivals per second relative to that accept rate. Under 60% is comfortable; anything approaching it means the fleet will manufacture its own failures during a real recovery, and the fix is a larger cap, more jitter, or both.

Both checks take minutes and they answer a question that is otherwise unanswerable until a real outage, at which point the evidence arrives as a support ticket about a platform that “went down for forty minutes when the network came back”.

Constraint validation #

Related #

• Retry & Backoff for Unstable Networks — the retry policy and circuit breaker this jitter applies to.
• Setting exponential backoff for cloud sync retries — the sync agent that consumes these delays.
• Rotating device credentials for MQTT fleets — the same synchronisation problem on a longer timescale.