Store-and-Forward Buffering

Within the Bandwidth & Async Sync Optimization guide, this page covers the component that decides whether a week-long outage is an inconvenience or a data loss: the durable spool. Every other technique in this section — compression, delta encoding, retry policy — assumes there is somewhere to put a message that cannot be sent right now, and that the somewhere survives a power cut, a watchdog reset and a firmware update.

The spool is deceptively simple to build and unusually easy to build wrongly. It is the only component on the device that must be correct across an unclean shutdown, it competes with the tile cache and the logs for a finite flash write budget, and its failure mode is silent: a spool that drops the wrong records under pressure produces a dataset with holes that nobody notices until an analyst asks why an asset has no history for a Tuesday.

Constraint mapping #

The endurance row is the one that shapes the implementation most. A spool that fsyncs after every record turns a 48-byte write into a full flash page program plus a journal update — a write amplification above 100× — and it is entirely avoidable by batching syncs at an interval derived from how much data the deployment can afford to lose on a power cut.

Implementation: the append path #

# spool.py — append-only durable spool with reserved partitions.
# One writer per partition; a single drain task reads. No mmap, no index.
# fsync is batched: a power cut loses at most `sync_interval_s` of records,
# and every record that survives is verifiable.
import os
import struct
import time
import zlib

HEADER = struct.Struct("<IHHQ")      # crc32, length, kind, monotonic_ns
HEADER_LEN = HEADER.size             # 16 bytes


class Partition:
    __slots__ = ("path", "fh", "cap", "used", "last_sync", "sync_interval",
                 "policy", "dropped")

    def __init__(self, path: str, cap_bytes: int, policy: str,
                 sync_interval_s: float = 5.0):
        self.path = path
        self.cap = cap_bytes
        self.policy = policy          # "never" | "drop_oldest" | "drop_newest"
        self.sync_interval = sync_interval_s
        self.fh = open(path, "ab", buffering=0)
        self.used = os.path.getsize(path)
        self.last_sync = time.monotonic()
        self.dropped = 0

    def append(self, payload: bytes, kind: int, now_ns: int) -> bool:
        need = HEADER_LEN + len(payload)
        if self.used + need > self.cap:
            if self.policy == "drop_newest":
                self.dropped += 1
                return False
            if self.policy == "never":
                raise SpoolFull(self.path)        # caller must shed upstream
            self._reclaim(need)                   # drop_oldest

        crc = zlib.crc32(payload) & 0xFFFFFFFF
        self.fh.write(HEADER.pack(crc, len(payload), kind, now_ns))
        self.fh.write(payload)
        self.used += need

        now = time.monotonic()
        if now - self.last_sync >= self.sync_interval:
            self.fh.flush()
            os.fsync(self.fh.fileno())
            self.last_sync = now
        return True

    def _reclaim(self, need: int):
        """Drop whole records from the head until `need` bytes are free.
        Implemented as a rewrite of the tail into a new file: on flash, one
        sequential rewrite beats punching holes in the middle of a file."""
        keep_from = self._offset_after_dropping(need)
        tmp = self.path + ".compact"
        with open(self.path, "rb") as src, open(tmp, "wb") as dst:
            src.seek(keep_from)
            while True:
                chunk = src.read(65536)
                if not chunk:
                    break
                dst.write(chunk)
            dst.flush()
            os.fsync(dst.fileno())
        self.fh.close()
        os.rename(tmp, self.path)                 # atomic
        self.fh = open(self.path, "ab", buffering=0)
        self.used = os.path.getsize(self.path)


class SpoolFull(Exception):
    """Raised only by a `never`-drop partition: the caller must stop producing."""

Implementation: the drain path and torn-tail recovery #

Reading is where the crash-safety guarantees are cashed in. Every record is validated before it is handed to the sender, and the first record that fails validation ends the readable region — because on an append-only file, damage can only be at the tail.

def read_records(path: str, from_offset: int = 0):
    """Yield (offset, kind, payload) until the first invalid record.
    A short read or a bad CRC means we reached the torn tail of a power cut."""
    with open(path, "rb") as fh:
        fh.seek(from_offset)
        offset = from_offset
        while True:
            head = fh.read(HEADER_LEN)
            if len(head) < HEADER_LEN:
                return                              # clean end of file
            crc, length, kind, ts_ns = HEADER.unpack(head)
            if length > 1 << 20:                    # implausible: torn header
                return
            payload = fh.read(length)
            if len(payload) < length:
                return                              # torn record
            if (zlib.crc32(payload) & 0xFFFFFFFF) != crc:
                return                              # corrupt record: stop here
            yield offset, kind, payload
            offset += HEADER_LEN + length


def truncate_torn_tail(path: str):
    """Called once at boot. Trims anything after the last valid record so the
    next append starts from a known-good offset."""
    last_good = 0
    for offset, _kind, payload in read_records(path):
        last_good = offset + HEADER_LEN + len(payload)
    size = os.path.getsize(path)
    if last_good < size:
        with open(path, "r+b") as fh:
            fh.truncate(last_good)
            os.fsync(fh.fileno())
        return size - last_good                     # bytes discarded
    return 0

Running truncate_torn_tail at every boot and exporting the number of bytes it discarded is the cheapest possible power-loss detector. A device that regularly discards a torn tail is losing power unexpectedly, which is a hardware or wiring finding that would otherwise take a site visit to establish.

Configuration and tuning #

• sync_interval_s decides how much a power cut costs. Five seconds at 40 records a second is 200 records — usually acceptable for position telemetry and never acceptable for events, so give the event partition a much shorter interval or sync it per record.
• Partition caps come from the outage target: rate × record size × target duration × overhead. The arithmetic is worked through in sizing a durable spool on NAND flash.
• Record size should be fixed per kind where possible. Fixed-size records make offsets computable, torn tails trivially detectable, and compaction a matter of arithmetic rather than parsing.
• Compaction trigger: reclaim in large batches, never per record. A rewrite that reclaims an eighth of the partition costs one sequential pass and amortises across thousands of appends.
• Priority order in the drain: events, then positions, then diagnostics. On a link that comes back for ninety seconds, the order decides what gets through.

The drain, and why acknowledgement order is the whole design #

Appending is the easy half. Draining is where a spool either preserves its guarantee or quietly loses records, and the difference is entirely in when the read cursor advances.

The rule is that the cursor moves only after the upstream has acknowledged the batch. Not after the send call returns, not after the socket accepted the bytes, not after a local timeout expired — after the receiving side has said, in whatever protocol is in use, that it holds the data. Everything else is an optimisation that trades correctness for a latency improvement nobody asked for.

That rule has a consequence people find uncomfortable: a link that accepts data and then fails before acknowledging causes the batch to be sent twice. Duplicates are the correct outcome. They are cheap to handle upstream with an idempotency key — a device id plus a monotonic sequence number is enough — and they are infinitely preferable to the alternative, which is a gap nobody can detect. Any store-and-forward design that fears duplicates more than gaps has its priorities inverted.

The cursor itself has to be durable, and it has to be updated in a way that cannot leave it ahead of reality. Writing it to its own small file with the fsync-then-rename discipline is sufficient, and updating it once per batch rather than once per record keeps the cost negligible. On a restart, the device reads the cursor, seeks to it, and resumes; if the cursor was lost, resuming from zero re-sends data and produces duplicates, which the idempotency key absorbs.

Batch size is the other drain parameter and it has a natural answer. Make a batch as large as the link’s typical uninterrupted window can carry — for a marginal cellular link that is often only a few kilobytes — because a batch that cannot complete before the link drops is a batch that is re-sent in full every time. Adaptive sizing helps here: start small after a reconnection, grow while batches succeed, and halve on failure. That is the same shape as congestion control, for the same reason.

Draining while the producers keep producing #

A spool that only drains when idle will never drain on a busy device, and one that drains flat out will starve the ingestion path of the flash and the CPU it needs. The workable arrangement gives the drain a budget rather than a priority: a maximum number of bytes per second and a maximum share of wall-clock time, both enforced by the drain itself.

Concretely, a drain that is allowed 20 KB/s and 30% of a second will send its batch, measure how long it took, and sleep for the remainder of its duty cycle. On a healthy link that produces a steady, unobtrusive flow. During a large backlog it produces the same steady flow — the backlog clears slower than it could, and nothing else on the device notices. That predictability is worth more than the throughput it gives up, because a device whose telemetry acquisition degrades whenever a backlog clears is a device that produces its worst data exactly after its worst outage.

The one exception is a bounded catch-up mode: when the backlog exceeds a threshold and the device is otherwise idle — stationary, no active job, mains powered — the drain may raise its budget. Gate that on real conditions rather than on a timer, and drop back the moment any of them changes.

Verification and field diagnostics #

Four numbers make the spool legible from a console: bytes used per partition, records dropped per partition since boot, torn bytes discarded at last boot, and the age of the oldest undrained record. The last is the one to alert on — a growing oldest-record age with a healthy link means the drain is stuck, which is a different fault from a backed-up queue and needs a different response.

Verify the crash safety deliberately rather than hoping for it. On the bench, cut power to the device mid-write a few dozen times under load and confirm three things each time: the spool reopens, the torn tail is trimmed, and the records that survive all validate. That test finds the mistakes that reading the code does not — a missed fsync on the rename, a buffered writer nobody flushed, a header written before its payload.

What the spool guarantees, stated plainly #

It is worth writing the guarantee down, because every component that depends on the spool depends on a precise version of it and will be built against whatever version the implementer assumed.

A record that append returned true for, and that was written more than sync_interval_s ago, survives any restart. That is the whole durability claim. It deliberately does not cover the last few seconds of records — buying that costs an fsync per record and the flash lifetime that implies — and every consumer of the spool has to be built knowing it.

Records are delivered at least once, in order, per partition. Not exactly once: the drain re-sends a batch whose acknowledgement was lost, and the upstream deduplicates on the sequence number. Not ordered across partitions: an event and a position written a millisecond apart may arrive in either order, because they travel independently.

A full partition drops according to its declared policy, and counts what it dropped. No partition ever grows past its cap, no partition’s overflow affects another, and no drop is silent.

Those three statements are worth putting in the code as a docstring and in the deployment manifest as a contract. The reason is a specific and common failure: a consumer built assuming exactly-once delivery, which then double-counts a metric every time a link drops mid-batch. That bug is not in the spool, and it will be reported against it.

The contract also sets what the spool is not. It is not a database — there is no query, no index, no random access. It is not a message broker — there is no routing, no topic, no fan-out. It is not a log for humans — its records are binary and its retention is governed by space rather than by time. Every one of those has been retrofitted into a spool at some point, and each retrofit costs the guarantee above, because each one adds a write path that the crash-recovery logic was not designed around.

Failure modes and recovery #

Testing the spool before it carries anything #

Three tests, run on the bench, cover essentially all of the failures above and none of them needs field hardware.

Fill and overflow. Drive each partition past its cap at production rate and confirm the declared policy applied, the counters incremented, and no other partition was affected. Ten minutes of runtime, and it catches the shared-space bug that otherwise appears only during a real outage.

Power cut under load. Cut supply mid-write repeatedly, and after each cycle assert the spool reopens, the torn tail is trimmed at a record boundary and every surviving record validates. Fifty cycles is enough to find a missing sync or a card whose flush is advisory.

Drain interruption. Kill the link mid-batch, repeatedly, and confirm no record is ever lost and every duplicate carries a sequence number the upstream can deduplicate on. This is the test that catches a cursor advanced one line too early — the single most consequential mistake available in this component.

Automate all three and run them in continuous integration against a loopback filesystem image. The spool is the one component here whose correctness cannot be inspected — a bug in it produces data that looks fine and is incomplete — so it is the one component that most deserves a test that runs whether or not anyone remembered to ask for it.

Related #

• Sizing a durable spool on NAND flash — turning an outage target into a byte cap.
• Ring buffer overwrite policies for telemetry backlogs — what to drop when the cap is reached.
• Crash-safe append logs with fsync budgets — the durability/wear trade in detail.
• Message Queue Management at the Edge — the broker and QoS layer the drain talks to.
• Retry & Backoff for Unstable Networks — the policy that decides when the drain tries again.