Sizing a durable spool on NAND flash

A spool size is usually chosen by picking a round number that sounds generous, and it is the wrong way to arrive at one. The right way is to state the outage the fleet must survive, derive the bytes that implies, and check the result against both the flash available and the flash’s write endurance. This guide does that arithmetic explicitly, for the spool built in store-and-forward buffering inside the Bandwidth & Async Sync Optimization guide.

The four inputs #

Post-filter message rate. Not the receiver’s fix rate — the rate that survives the filtering and dead-banding described in the parent guide. For a vehicle reporting at 1 Hz with a 5 m dead band and stop suppression, the retained rate is typically 0.2–0.6 messages a second. Measure it from a recorded day rather than estimating it; the ratio between raw and retained varies more between deployments than any other number here.

Encoded record size. The payload after quantisation and binary encoding, plus the spool’s own 16-byte header. A packed position record is 12–20 bytes of payload; an event with context is 60–120.

Outage target. The longest interval the device must survive without losing data. This is a business decision, not an engineering one, and it should be written down: “seven days” and “until the next site visit” imply very different hardware.

Overhead factor. The storage engine’s own cost. An append-only file with fixed records is close to 1.05; SQLite with an index is 1.3–1.5; a filesystem with a 4 KB block size applied to small records can be worse than either if records are not batched.

The calculation in code #

Do this at provisioning, from real measurements, and record the output in the deployment manifest:

# spool_sizing.py — derive partition caps from measured behaviour.
# Run at build time against a recorded day of traffic; the outputs go into
# the device manifest, not into a constant somewhere in the source.
from dataclasses import dataclass

DAY_S = 86_400


@dataclass(frozen=True)
class Stream:
    name: str
    msgs_per_day: float          # measured post-filter, not the raw fix rate
    record_bytes: int            # payload + 16-byte spool header
    overhead: float = 1.15       # append log; use 1.4 for SQLite with an index
    safety: float = 1.0          # multiplier for bursty streams


def cap_bytes(stream: Stream, outage_days: float) -> int:
    raw = stream.msgs_per_day * stream.record_bytes * outage_days
    return int(raw * stream.overhead * stream.safety)


def annual_write_bytes(streams, overhead_amplification: float = 3.0) -> int:
    """What the spool costs the flash per year, including amplification."""
    daily = sum(s.msgs_per_day * s.record_bytes for s in streams)
    return int(daily * 365 * overhead_amplification)


def endurance_years(card_bytes: int, p_e_cycles: int, annual_bytes: int) -> float:
    return (card_bytes * p_e_cycles) / annual_bytes


if __name__ == "__main__":
    streams = [
        Stream("positions", 34_560, 34),                 # 0.4/s
        Stream("events", 40, 96, safety=50.0),           # over-provisioned
        Stream("diagnostics", 2_880, 48),                # every 30 s
    ]
    total = 0
    for s in streams:
        c = cap_bytes(s, outage_days=7)
        total += c
        print(f"{s.name:<12} {c/1e6:6.2f} MB")
    print(f"{'total':<12} {total/1e6:6.2f} MB")

    ann = annual_write_bytes(streams)
    print(f"annual writes {ann/1e9:.2f} GB")
    print(f"endurance     {endurance_years(16e9, 3000, ann):.0f} years on a 16 GB card")

Two outputs matter and only one is usually looked at. The cap tells you whether the outage target fits the flash. The endurance figure tells you whether the write rate fits the flash’s lifetime, and it is the one that catches a design where the spool is small but rewritten constantly — a configuration that looks frugal and consumes the card faster than a large spool written once.

Constraint validation #

Gotchas and edge cases #

• Measure the retained rate, not the raw rate. Sizing from the receiver’s 1 Hz output over-provisions by a factor of three or more, which wastes flash that the tile archive needed. Sizing from an optimistic filter ratio under-provisions, which loses data. Take a recorded day from a real deployment.
• Records are not the only writers. The filesystem journal, the log rotation and any database WAL all consume the same endurance budget. The endurance calculation must include them or it will be optimistic by a factor of two to three.
• A big spool is not slower, but a full one is. Appends stay O(1) regardless of size; it is the compaction pass that scales with the partition, so a very large partition being reclaimed an eighth at a time is a long sequential write. Size compaction batches from the partition, not from a constant.
• Fixed-size records simplify everything downstream. Variable-length records make offsets uncomputable, torn tails harder to detect and compaction a parsing exercise. Where a stream can be fixed-width, fix it, even at the cost of a few wasted bytes per record.
• Do not size the diagnostics partition generously. Diagnostics are the stream most likely to grow without anyone noticing, because a debug flag left on in a firmware build can multiply its rate by fifty. A small partition with a drop-newest policy turns that mistake into a bounded loss of diagnostics rather than an evicted position history.

Reviewing the size after deployment #

Sizing is not a one-time exercise, and the signal that it needs revisiting is available for free. Export the high-water mark of each partition — the largest used-bytes figure since boot — and collect it across the fleet. Three patterns are worth acting on: a partition that has never exceeded 5% of its cap is over-provisioned and its flash could go to the tile archive; one that regularly exceeds 60% is one bad week away from dropping records; and a partition whose high-water mark climbs steadily across firmware versions has a producer whose rate is growing without anyone deciding that it should.

Re-derive the caps whenever the filter policy changes. Widening a dead band from 5 m to 15 m can halve the retained rate, which halves the spool requirement — and a fleet that never revisits the number keeps carrying storage it stopped needing two releases ago.

When the number does not fit #

Sometimes the arithmetic produces a cap the hardware cannot supply — a 90-day outage target on a node with 32 MB of usable flash, say. There are four honest responses and one dishonest one.

Reduce the rate. Widen the dead band, lengthen the sampling interval, or raise the movement threshold. This is almost always the right first move, because the rate was usually chosen by default rather than by requirement, and halving it halves the storage without changing anything a consumer can detect.

Reduce the record. Drop fields nobody reads. A telemetry record that carries a device id, a firmware version and a fixed tag on every message is carrying constants that belong in the session, not the record.

Shorten the target. Seven days may be a wish rather than a requirement. If the fleet’s worst observed outage over a year was 31 hours, a three-day target with an alert at two is defensible and fits.

Degrade explicitly. Accept that beyond the cap the device decimates, and state the resolution it will fall back to. A month at ten-minute resolution in 32 MB is achievable and is a genuinely useful dataset.

The dishonest response is to leave the cap where it is and let the filesystem fill. That converts a bounded, described data loss into an unbounded failure of whatever component happens to need disk next — usually the logs, which is precisely the component someone will want when diagnosing it.

It is worth stating the one number this whole exercise produces: the fleet’s data-loss horizon, the point past which an outage costs records. Written into the deployment document alongside the cap, it becomes something an operations team can plan against — “past 5.2 days offline, this device begins decimating” is an actionable statement, and “the spool is 15 MB” is not. Everything above exists to turn the second sentence into the first.

Related #

• Store-and-Forward Buffering — the spool this sizing configures.
• Ring buffer overwrite policies for telemetry backlogs — what happens at the cap this calculation produces.
• Pruning tile caches under a flash write budget — the other large consumer of the same endurance budget.