Edge Geospatial IoT Gateway Processing

Crash-safe append logs with fsync budgets

Durability on a field gateway is bought with flash writes, and the exchange rate is worse than it looks: a 34-byte record synced individually can cost a 4 KB page program plus a journal update, an amplification above 200×. This guide sets the exchange rate deliberately — how much data a power cut may lose, and what that permission buys back in flash lifetime and write latency — for the append log built in store-and-forward buffering, inside the Bandwidth & Async Sync Optimization guide.

What fsync actually guarantees, and what it does not

A write() returns once the bytes are in the kernel’s page cache. They may reach the device seconds later, or never if power is lost. fsync() returns once the device reports the data durable — which on consumer flash means “in the device’s own cache and covered by its capacitors”, a claim not every part honours.

Three consequences follow for an append log. First, the ordering between a payload write and any metadata that describes it is not guaranteed without a barrier, which is why the record’s own header carries its length and checksum rather than a separate index carrying them. Second, fsync on the file does not make a rename durable — the containing directory needs its own sync, a step that is skipped in most code and only matters after a power cut, which is to say only in the field. Third, some cheap cards lie: they acknowledge a flush before the data is durable. The mitigation is not code, it is a purchasing decision and a power-cut test on the bench.

Where a record can be lost between the append call and the flash A record passes through four stages: the application buffer, the kernel page cache, the device write cache, and the flash cells. A plain write reaches the page cache, so an application crash loses nothing but a power cut loses everything not yet flushed. An fsync pushes to the device cache and, on a part that honours the flush, into the cells. A cheap card that acknowledges early leaves a window where the data is only in the device cache, which a power cut still loses. Buffered writes with a five second sync interval bound that window explicitly. Four places the bytes can be, and what each survives app buffer survives: nothing a crash loses it kernel page cache survives: process crash a power cut loses it device write cache survives: kernel panic a power cut may lose it flash cells survives: everything this is what durable means write() fsync() device flush A card that acknowledges the flush early leaves the third box unprotected — undetectable in code, findable only by cutting power on a bench. The sync interval is a decision about how many records may sit in the first two boxes when the power goes.
Durability is a position in this chain, not a boolean. The interval decides how long a record is allowed to be in one of the first two boxes.

The budget, made explicit 

# fsync_budget.py — append with an explicit durability window per stream.
# Each stream states how much loss a power cut may cause; the writer converts
# that into a sync cadence and enforces it. Nothing here is best-effort.
import os
import time
from dataclasses import dataclass


@dataclass(frozen=True)
class DurabilityPolicy:
    """max_loss_records: how many records a power cut may cost this stream.
    max_loss_seconds: the same bound expressed in time; whichever binds first
    triggers the sync."""
    max_loss_records: int
    max_loss_seconds: float


POLICIES = {
    "events":      DurabilityPolicy(max_loss_records=0,   max_loss_seconds=0.0),
    "positions":   DurabilityPolicy(max_loss_records=200, max_loss_seconds=5.0),
    "diagnostics": DurabilityPolicy(max_loss_records=2000, max_loss_seconds=60.0),
}


class BudgetedWriter:
    __slots__ = ("fh", "policy", "since_sync", "last_sync", "syncs", "bytes_written")

    def __init__(self, path: str, policy: DurabilityPolicy):
        self.fh = open(path, "ab", buffering=0)
        self.policy = policy
        self.since_sync = 0
        self.last_sync = time.monotonic()
        self.syncs = 0
        self.bytes_written = 0

    def append(self, framed: bytes) -> None:
        self.fh.write(framed)
        self.bytes_written += len(framed)
        self.since_sync += 1
        if self._should_sync():
            self._sync()

    def _should_sync(self) -> bool:
        p = self.policy
        if p.max_loss_records == 0:
            return True                                   # sync every record
        if self.since_sync >= p.max_loss_records:
            return True
        return (time.monotonic() - self.last_sync) >= p.max_loss_seconds

    def _sync(self) -> None:
        os.fsync(self.fh.fileno())
        self.since_sync = 0
        self.last_sync = time.monotonic()
        self.syncs += 1

    def close(self) -> None:
        self._sync()
        self.fh.close()


def durable_rename(src: str, dst: str) -> None:
    """Rename plus the directory sync that makes it survive a power cut.
    Omitting the directory fsync is the most common durability bug in this
    whole area, and it only manifests after an unclean shutdown."""
    os.rename(src, dst)
    dir_fd = os.open(os.path.dirname(dst) or ".", os.O_DIRECTORY)
    try:
        os.fsync(dir_fd)
    finally:
        os.close(dir_fd)

Expressing the policy as “how many records may be lost” rather than as “sync every N” puts the decision where it belongs: with whoever owns the data, not with whoever wrote the writer. An event stream declaring zero acceptable loss gets a sync per record and pays for it; a diagnostics stream declaring two thousand gets one sync a minute and costs the flash almost nothing.

Constraint validation

Constraint Expected impact Mitigation built into the code
Flash endurance Per-record syncs amplify writes by two orders of magnitude Sync cadence derived from a declared loss budget; only the event stream pays the full price
Latency fsync blocks the calling thread for 2–40 ms on eMMC Batched for the high-rate streams; the event stream’s sync is on a low-rate path
Power loss Unsynced records are lost The window is explicit and bounded, and the torn tail is detectable on the next boot
Correctness A rename can be lost even after the file is synced durable_rename syncs the directory as well
CPU Sync is I/O wait, not CPU, but it holds the thread The writer runs on the spool task, never on the acquisition thread

Gotchas and edge cases

  • fsync on a file does not sync its directory entry. Creating a file, writing it, syncing it and renaming it still leaves the rename in the page cache. Every atomic-replace pattern in this section depends on durable_rename, and every one that omits the directory sync is subtly broken.
  • fdatasync is not always cheaper. It skips metadata that has not changed, which for an append that extends the file is exactly the metadata that did change. On an append log the two usually cost the same; measure before assuming otherwise.
  • A sync that fails must not be ignored. On some kernels a failed fsync marks the error consumed, so a retry returns success while the data is gone. Treat an OSError from fsync as fatal for the writer: close, reopen, and re-verify the tail rather than continuing.
  • Latency spikes are not the same as throughput loss. A 40 ms sync every five seconds is invisible in aggregate and very visible if it lands inside a 50 ms deadline. Keep the writer off any thread with a deadline, which is the same separation the async execution guide applies to compiled work.
  • Test with real power cuts. Killing the process tests the first box in the diagram and nothing else. Cutting power at the barrel jack, repeatedly, under load, is the only test that covers the other three — and it is the test that finds a card whose flush acknowledgement is a lie.
Write amplification and throughput against sync cadence Four cadences measured on a 34 byte record stream at 40 records a second. Syncing every record writes 4 096 bytes per record — an amplification of 120 — at 24 records a second maximum throughput. Every 10 records amplifies 12 times at 380 records a second. Every 200 records amplifies 1.6 times at 4 100 records a second. Every 5 seconds amplifies 1.2 times with no throughput ceiling that matters. The loss window grows in step, from zero records to 200. What each durability level costs, on a 34-byte record amplificationmax records/sloss window sync per recordevery 10every 200every 5 s 120×240 records 12×380≤9 records 1.6×4 100≤199 records 1.2×no practical limit≤200 records The first row is correct for events and ruinous for positions. That is the entire argument for per-stream policies.
Between the first and third rows there is a factor of seventy-five in flash consumed and a factor of a hundred and seventy in throughput — bought with two hundred records of exposure.

Proving it on the bench

Build the power-cut rig once and it pays for itself across every project. A relay on the supply, a script that writes at the production rate with the production policy, and a loop that cuts power at randomised intervals, reboots, and validates. Each cycle checks that the log reopens, that the torn tail is trimmed at a record boundary, that every surviving record’s checksum passes, and that the number of records lost is inside the declared budget.

Fifty cycles is usually enough to find the interesting faults: the missing directory sync, the writer that buffered when nobody expected it, the card whose flush is advisory. None of those is visible in code review, all of them are visible within an hour of automated power cutting, and each of them produces field data loss that would otherwise be attributed to something else entirely.

Framing that makes recovery decidable

The recovery routine’s job is to answer one question about the tail of a file: is this a complete record or the debris of an interrupted write? Everything about the frame format either helps or hinders that.

Four fields, in this order, make it decidable with no ambiguity. A length first, so the reader knows how much to consume before consuming it. A checksum over the payload, so a fully written record with corrupted bytes is caught. A kind tag, so a reader can skip a record type it does not understand rather than aborting. And a monotonic timestamp, so ordering survives a clock step.

Two things are deliberately absent. There is no magic number at the record boundary, because a magic number tempts a reader into resynchronising after damage — scanning forward for the next one — and on an append-only file that is never the right behaviour: damage is at the tail, and anything after it is by definition suspect. And there is no back-pointer to the previous record, because maintaining one means writing into the previous record after the fact, which turns an append-only file into a random-write one and forfeits the entire crash-safety argument.

The recovery rule that falls out is a single sentence: read forward from the last known-good offset, stop at the first record that fails length or checksum validation, truncate there. It has no special cases, it terminates, and it produces the same result whether the interruption happened between two records, in the middle of a header, or in the middle of a payload.

The 16-byte frame, and how each field participates in recovery A record frame of four fields followed by a payload. A four byte checksum validates the payload. A two byte length tells the reader how far to read. A two byte kind lets an unknown record type be skipped rather than aborting recovery. An eight byte monotonic timestamp preserves ordering across a clock step. Beneath, three interruption cases — between records, mid-header and mid-payload — all resolve to the same rule: stop at the first record that fails validation and truncate there. Four fields, one recovery rule, no special cases crc32 · 4 B length · 2 B kind · 2 B monotonic ns · 8 B payload cut between records → last record validates → truncate at its end cut mid-header → short read → truncate at the previous record's end cut mid-payload → short read or bad checksum → truncate at the previous record's end No magic number: resynchronising past damage is never correct on an append-only file.
The absence of a magic number and a back-pointer is what keeps the rule to one sentence — both of them exist to enable behaviour this format does not want.

A final note on where this discipline stops. Everything above concerns a single file on a single device, and it is sufficient for exactly that. The moment a design needs two files to be consistent with each other — a spool and a separate cursor, say — the guarantees have to be re-derived, because there is no ordering between two independent fsync calls. The way out is either to put both pieces of state in the same file, so one sync covers them, or to make one of them reconstructible from the other, so an inconsistency after a power cut has an unambiguous resolution. The cursor described in the parent guide takes the second route: if it is lost or stale, replaying from an earlier offset produces duplicates the upstream already knows how to absorb.

Related in Store-and-Forward Buffering