Ring buffer overwrite policies for telemetry backlogs

Every bounded buffer eventually meets a producer that will not stop, and the policy it applies at that moment decides what a week-long outage costs. This guide works through the four policies a telemetry spool can implement, the failure each one produces, and how to combine them per message class so that the answer is right for events and right for positions at the same time. It sits under store-and-forward buffering in the Bandwidth & Async Sync Optimization guide.

The four policies #

Drop newest refuses the incoming record and keeps what is already stored. It preserves the beginning of an incident and loses the end, which is right for a diagnostic stream where the first occurrence explains the fault and the thousandth adds nothing.

Drop oldest — the classic ring buffer — overwrites the head to make room. It keeps the buffer current at the cost of a hole in the history, which is right for position telemetry where the most recent location is the operationally useful one and last Tuesday’s is not.

Backpressure refuses to accept the record and signals the producer to stop. It is correct only when the producer can do something with the refusal. A UART reader cannot; a batch importer can.

Decimate keeps everything but at reduced resolution: when the buffer passes a threshold, discard every second record from the oldest region rather than dropping a contiguous span. It preserves the shape of a track across the whole outage at progressively coarser fidelity, which for a position stream is frequently better than either drop policy — a day at 60-second resolution beats twelve hours at 1-second resolution and twelve hours of nothing.

Implementing decimation #

Decimation is the least familiar of the four and the most useful for position streams, so it is worth writing out. The mechanism is a set of resolution bands: the buffer is divided by age, and each older band is thinned to a coarser spacing when space is needed.

# decimate.py — age-banded thinning for a position spool.
# Runs on the maintenance task when a partition crosses its high-water mark.
# Preserves the first and last record of every thinned run so the track's
# shape and its endpoints survive at every resolution.
BANDS = (
    # (max_age_seconds, target_spacing_seconds)
    (6 * 3600, 1),        # last 6 h: keep everything
    (24 * 3600, 5),       # 6–24 h: one every 5 s
    (72 * 3600, 30),      # 1–3 days: one every 30 s
    (float("inf"), 120),  # older: one every 2 minutes
)


def target_spacing(age_s: float) -> int:
    for max_age, spacing in BANDS:
        if age_s <= max_age:
            return spacing
    return BANDS[-1][1]


def decimate(records, now_s: float):
    """records: iterable of (offset, timestamp_s, kind, payload), oldest first.
    Yields the offsets to drop. Never drops an event record, and never drops
    two adjacent records — the track must keep its vertices."""
    keep_time = None
    prev_kept = True
    for offset, ts, kind, _payload in records:
        if kind != KIND_POSITION:
            keep_time = None                   # events always survive
            prev_kept = True
            continue
        spacing = target_spacing(now_s - ts)
        if keep_time is None or (ts - keep_time) >= spacing:
            keep_time = ts
            prev_kept = True
            continue
        if not prev_kept:
            # Refuse to drop two in a row: preserves direction changes even
            # when the spacing rule would remove them.
            keep_time = ts
            prev_kept = True
            continue
        prev_kept = False
        yield offset

The “never drop two adjacent records” rule is the detail that makes decimated tracks usable. A pure time-spacing rule removes every point between kept samples, which flattens a corner taken between two samples into a straight line. Keeping every second point at worst preserves the direction changes, so the decimated track still looks like the route rather than like a polygon of chords.

Constraint validation #

Gotchas and edge cases #

• A dropped record must be counted, always. The single most damaging property of any of these policies is silence. Export counters per partition and per policy, and sync them as telemetry in their own right — a consumer that knows 4 100 positions were decimated can interpret the gap; one that sees only the surviving records cannot.
• Decimation is not idempotent across passes. Running it repeatedly thins the same data further as it ages, which is intended, but a bug that re-runs it in a loop empties the partition. Gate the pass on the high-water mark and record the last pass time.
• Drop-oldest with variable-size records needs care. Reclaiming “enough bytes” can drop a partial record if the loop stops mid-record. Always drop whole records and recompute the freed space from actual record boundaries.
• Backpressure that cannot be honoured is worse than dropping. A producer told to stop, which cannot stop, will either block — stalling the acquisition path — or ignore the signal, in which case the buffer overflows anyway with the added latency of the failed negotiation.
• Events need their own reservation, not just their own policy. A never-drop policy on a shared partition means the partition fills with events and then refuses positions. The reservation described in the parent guide is what makes a never-drop policy safe.

Deciding which policy a stream needs #

Two questions settle it. If exactly one record of this stream survived the outage, which one would you want? If the answer is “the first”, the policy is drop newest. If it is “the last”, drop oldest. If the answer is “I would want a sample from throughout”, the policy is decimate. If the answer is “all of them, this is not negotiable”, the stream needs a reservation and a never-drop policy — and a rate low enough that the reservation can hold it.

What does the consumer do with a gap? A consumer that treats a gap as “no data” behaves differently from one that treats it as “not reported”. Position history is usually the first; a state stream — where a missing record means the state is unknown rather than unchanged — is the second, and gaps in it are far more damaging. State streams belong in the never-drop class even when their volume makes that uncomfortable, because the alternative is a consumer confidently reporting a state the device never sent.

Making the loss visible upstream #

A policy that drops records correctly and says nothing about it has solved half the problem. The consumer receives a clean sequence with an unmarked hole, and every analysis over that sequence is quietly wrong in a way that looks like a real absence of activity.

The fix is a gap record: a small synthetic message emitted into the same stream at the point where records were removed, carrying the count dropped, the time span they covered and the policy that removed them. It costs a few dozen bytes per gap and it converts an invisible hole into a described one. A consumer that sees dropped: 4 118 positions, 06:12–14:40, policy=decimate can render the period correctly, exclude it from a utilisation calculation, or ask for a resend if the data is recoverable.

Emit the gap record at drop time rather than at drain time, so that its position in the stream is where the data was, not where the device happened to notice. Give it the same durability treatment as an event — a gap record lost to a power cut leaves exactly the hole it was meant to describe.

One last consideration is what the policy does to a fleet-wide analysis. If every device decimates independently, a query over the whole fleet meets tracks at different resolutions depending on which devices had outages — and an aggregate computed naively over that mixture is weighted toward the devices that stayed connected. Recording the resolution alongside the data lets the analysis normalise; without it, the bias is invisible and systematic, and it favours exactly the devices whose behaviour was least interesting.

Related #

• Store-and-Forward Buffering — the spool and its reserved partitions.
• Sizing a durable spool on NAND flash — choosing the cap these policies defend.
• Queue-depth alerting thresholds for edge sync — noticing the pressure before a policy has to act.