Field Diagnostics & Recovery

Within the Edge Operations & Observability practice, field diagnostics and recovery is the set of mechanisms that let an unattended gateway survive a fault it never expected and hand a technician the evidence to understand it afterward. A node bolted to a pole cannot be power-cycled by hand every time a spatial thread wedges or a modem driver deadlocks, so recovery has to be automatic: a hardware watchdog that reboots the SoC when liveness checks stop passing, shutdown hooks that flush buffers and close memory-mapped stores before that reboot lands, a small ring of health snapshots persisted to flash so the evidence survives the power cycle, and a local read-out a technician can pull off the device with nothing more than a laptop and a cable. The hard part is not detecting that something is wrong — it is building the recovery path so that fixing one fault never quietly creates a second, and so a transient stall (thermal throttle, a slow chunk, a modem hiccup) is never treated the same as a permanent one (a corrupted store, a bad sensor, a dead radio).

Constraint Mapping #

Every mechanism in this guide traces back to a physical limit that a rack server never has to think about: no operator watching a console, no RTC battery holding wall-clock time, and flash that wears out if you write to it carelessly. The general resource envelope is catalogued under device constraints and resource limits; the table below is the subset that bears directly on recovery design.

The number worth internalizing first is the watchdog margin: the timeout has to sit comfortably above the worst chunk time the pipeline will ever see, including under thermal throttle, or the watchdog becomes the fault it was meant to catch.

Implementation: a liveness-gated watchdog kicker #

The mistake that turns a hardware watchdog into decoration is kicking it unconditionally — on a fixed timer, regardless of whether the pipeline is actually making progress. That pattern reboots nothing, because the kick keeps happening even while the ingestion loop is wedged. The correct contract is: kick only when an explicit liveness predicate passes, and let every subsystem that matters register its own staleness budget so a stall anywhere in the pipeline withholds the kick.

import fcntl
import os
import struct
import time

WDIOC_SETTIMEOUT = 0xC0045706   # linux/watchdog.h: set timeout (seconds)
WDIOC_GETTIMEOUT = 0x80045707   # read back the value the driver actually accepted


class LivenessGatedWatchdog:
    """Opens /dev/watchdog, negotiates a timeout, and kicks it only when a
    caller-supplied liveness predicate passes. A watchdog kicked on a bare
    timer instead of a liveness check protects nothing.
    """

    def __init__(self, path="/dev/watchdog", timeout_s=20):
        self._fd = os.open(path, os.O_WRONLY)
        accepted = fcntl.ioctl(self._fd, WDIOC_SETTIMEOUT, struct.pack("i", timeout_s))
        self.timeout_s = struct.unpack("i", accepted)[0]

    def kick_if_alive(self, liveness_fn) -> bool:
        """Returns whether a kick was issued, so callers can log a near-miss
        instead of silently riding the countdown down to zero."""
        if liveness_fn():
            os.write(self._fd, b"\0")
            return True
        return False

    def close(self, expect_reboot: bool):
        # Magic close: writing 'V' immediately before close() disarms the
        # timer. Skip it deliberately when a stall is suspected so the
        # countdown keeps running through an unclean exit path.
        if not expect_reboot:
            os.write(self._fd, b"V")
        os.close(self._fd)


class LivenessTracker:
    """True only if every registered subsystem checked in within its own
    staleness budget. One stalled subsystem withholds the kick for all."""

    def __init__(self):
        self._last_seen = {}
        self._budgets = {}

    def register(self, name: str, staleness_budget_s: float):
        self._budgets[name] = staleness_budget_s
        self._last_seen[name] = time.monotonic()

    def heartbeat(self, name: str):
        self._last_seen[name] = time.monotonic()

    def alive(self) -> bool:
        now = time.monotonic()
        return all(
            (now - self._last_seen[name]) < budget
            for name, budget in self._budgets.items()
        )

The LivenessTracker is the part most implementations skip, and it is the part that matters: a single alive() flag toggled from one place tends to drift out of sync with reality, while per-subsystem budgets catch the case where the ingestion loop is fine but a worker pool has quietly wedged. Wire the tracker’s heartbeat() calls into the same points that already report progress — the chunk-dispatch loop in async execution for spatial workloads is a natural source, since a chunk that overruns its own timeout is exactly the condition the watchdog exists to catch one layer up:

watchdog = LivenessGatedWatchdog(timeout_s=20)
liveness = LivenessTracker()
liveness.register("ingest_loop", staleness_budget_s=6.0)
liveness.register("worker_pool", staleness_budget_s=10.0)

async def watchdog_kicker():
    while True:
        if not watchdog.kick_if_alive(liveness.alive):
            logging.error("liveness check failed; withholding watchdog kick")
        await asyncio.sleep(watchdog.timeout_s / 4)

Polling at roughly a quarter of the timeout gives three missed windows of slack before the hardware actually fires, which is enough margin to distinguish a single slow cycle from a genuine stall without extending the effective recovery time.

Implementation: a persistent health snapshot that survives reboot #

A watchdog reboot without evidence is a coin flip for the technician who eventually visits the site: did the node crash once, or has it been rebooting every ten minutes for a month? The fix is a small, fixed-width health snapshot written to flash on a bounded cadence, using an atomic swap so a power loss mid-write can never leave a torn file behind.

import os
import struct
import time

_STATE_PATH = "/var/lib/gateway/health_state.bin"
_STATE_TMP = _STATE_PATH + ".tmp"
_FMT = "<QIIBxxxQI"   # monotonic_ns, boot_id, reboot_count, degraded, last_good_ts, fault_code


def load_last_snapshot():
    try:
        with open(_STATE_PATH, "rb") as f:
            data = f.read(struct.calcsize(_FMT))
    except FileNotFoundError:
        return None
    if len(data) != struct.calcsize(_FMT):
        return None
    return struct.unpack(_FMT, data)


def persist_snapshot(boot_id, reboot_count, degraded, last_good_ts, fault_code):
    """Build the new state in full, fsync it, then rename over the old file.
    A reader never observes a torn write — only the previous snapshot or the
    complete new one, never a mix of the two."""
    record = struct.pack(
        _FMT, time.monotonic_ns(), boot_id, reboot_count,
        1 if degraded else 0, last_good_ts, fault_code,
    )
    fd = os.open(_STATE_TMP, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o644)
    try:
        os.write(fd, record)
        os.fsync(fd)                       # data must reach flash before the rename
    finally:
        os.close(fd)
    os.rename(_STATE_TMP, _STATE_PATH)     # atomic on the same filesystem
    dir_fd = os.open(os.path.dirname(_STATE_PATH), os.O_RDONLY)
    try:
        os.fsync(dir_fd)                   # persist the rename itself across power loss
    finally:
        os.close(dir_fd)


def append_fault_log(path, fault_code, boot_id):
    """Append-only write-ahead log of fault events, separate from the
    swapped snapshot above. O_APPEND writes are atomic at the syscall level
    for records this small, and fsync forces each entry to flash before the
    caller assumes it survived."""
    entry = struct.pack("<QII", time.monotonic_ns(), boot_id, fault_code)
    fd = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_APPEND, 0o644)
    try:
        os.write(fd, entry)
        os.fsync(fd)
    finally:
        os.close(fd)

Two different persistence disciplines are deliberately combined here. The current-state file uses the write-new-then-rename pattern because only the latest value matters and a reader should never see a half-written one. The fault log uses append-only writes because history matters there — each entry is small enough that a single write() is effectively atomic, and losing the very last unflushed entry to a power cut is an acceptable cost against the complexity of rotating a second swapped file. Reserve a fixed number of boot ids and reboot counts across restarts by reading load_last_snapshot() at startup before writing a new one, so the reboot counter increments rather than resets.

Configuration & Tuning #

• Watchdog timeout vs. worst-case chunk time. Set timeout_s above the slowest chunk the pipeline will realistically process under thermal throttle, not the median. If the async worker pool’s own timeout is 4 seconds, the hardware watchdog should sit well above the sum of one worker timeout plus one liveness poll interval — 20–30 seconds is typical headroom on a Cortex-A class gateway, tighter on a bare-metal target with a smaller thermal mass.
• Poll cadence. Check liveness at roughly a quarter of the watchdog timeout. Faster polling burns cycles for no benefit; slower polling erodes the margin between “genuinely stalled” and “hardware fires.”
• Write-ahead logging discipline. Treat the fault log as append-only and never rewrite existing entries in place — that guarantee is what lets a reader trust every entry it can see, even if the very last one is missing after a crash.
• Atomic rename cadence. Persist the current-state snapshot on state transitions (a fault raised, cleared, or the reboot counter incremented) rather than on a fixed timer, to keep flash wear proportional to real events instead of clock ticks.
• Snapshot format stability. Keep _FMT fixed-width and versioned by field count, not by adding optional fields — a reboot that lands mid-firmware-upgrade must still be able to parse whatever snapshot format the previous binary wrote.

Work through a concrete sizing example rather than guessing. On a Cortex-A7 gateway measured to process a 128-geometry chunk in 900 ms under nominal load and 3.1 s under a 78 °C thermal throttle, a stall_budget_s of 6 seconds leaves roughly double the worst observed chunk time as margin. Layer the hardware timeout_s above that: a 20-second watchdog gives more than three missed liveness polls (at the recommended quarter-timeout cadence) before the hardware fires, which is enough slack to distinguish “one unusually slow chunk” from “the loop has actually stopped.” Nodes with a smaller thermal mass, or fanless enclosures that throttle harder and faster, should re-measure this pair of numbers on-device rather than reusing values from a different enclosure design — the whole point of the margin is that it reflects the specific unit’s worst case, not a datasheet figure.

Verification & Field Diagnostics #

Confirming this pattern works means proving three things on a live device, not in a simulator: the watchdog actually protects the pipeline, the snapshot actually survives a real power cut, and a technician can actually read the result without a network. Each of these has to be tested by inducing the actual failure, because a watchdog that has never been made to fire on purpose is a watchdog whose timeout value is a guess.

1. Induce a real stall in staging. Block the liveness-tracked loop deliberately (an infinite loop, a blocked socket read) and confirm the device reboots within timeout_s plus one poll interval — not sooner, not indefinitely later.
2. Pull power mid-write. Cut power to a test unit while persist_snapshot() is running in a tight loop, then confirm on restart that load_last_snapshot() returns either the previous or the fully-written new record, never a struct-unpack error from a torn read.
3. Read the reboot counter, not just uptime. A device that reboots cleanly every few minutes looks alive from a distance while accomplishing nothing; track reboot_count against a baseline and alert on its rate, the same discipline described in the parent guide’s monitoring and observability practice.
4. Check RSS alongside the snapshot. A worker leaking memory ahead of a watchdog reboot is a different fault than a genuine deadlock; correlate the fault code against RSS growth using the profiling approach in profiling RSS and heap fragmentation on ARM gateways before assuming the watchdog alone fixed anything.
5. Confirm the local read-out with the network disabled. Pull the SIM or disable the modem entirely and verify a technician can still retrieve the degradation flag, last-good timestamp, and reboot counter over a serial cable — the mechanism covered in serial console health endpoints for field techs.

Failure Modes & Recovery #

• Crash-loop masked by the watchdog. The device reboots cleanly every few minutes and looks healthy from a distance, because each individual reboot succeeds. Detect: the persisted reboot_count climbs faster than the deployment’s baseline rate. Recover: alert on reboot rate rather than liveness alone, and inspect the fault log’s most recent entries — a repeating fault code across boots points at a permanent condition (bad sensor, corrupted config) that a reboot cannot fix, as opposed to a transient one that clears itself.
• Store corruption on reboot. A watchdog fires mid-write to the spatial index, and the next boot finds a torn page it cannot parse. Detect: the mmap-backed store fails to open, or its header checksum mismatches. Recover: this is precisely what the write-ahead log and atomic rename discipline exist to prevent for the health state; apply the same discipline — flush-then-rename, never write in place — to any other on-device store, and pair it with the shutdown hooks covered in hardware watchdog recovery for stalled pipelines so a clean shutdown path closes the mapping before a reboot can land on it.
• False stall from thermal throttle, not a real fault. A chunk that would normally finish in a second takes six under sustained heat, tripping the same liveness budget a genuine deadlock would trip. Detect: the fault log shows a stall alongside a rising scaling_cur_freq drop rather than a flatlined heartbeat counter. Recover: widen the affected subsystem’s staleness budget adaptively under throttle, the same hysteresis-first instinct behind threshold-based event mapping, rather than tightening the watchdog timeout to compensate.
• Fault reported but never reaches anyone. The device correctly detects and logs a permanent fault, but the uplink has been down for days and nobody knows. Detect: the reported-health timestamp in the queue lags the local snapshot’s monotonic time by more than one duty cycle. Recover: let the health report ride the same message queue as spatial data, with retry and backoff so it is delivered opportunistically the moment connectivity returns instead of being dropped as stale.

Across all four, the underlying discipline is the same: treat every reboot as evidence to be explained, not as a problem solved. A transient fault — throttle-induced slowness, a modem hiccup, a single dropped chunk — should clear itself after one or two recovery cycles and leave a fault log with no repeating pattern. A permanent fault — a corrupted configuration file, a failed sensor, a store that fails to reopen the same way every time — will show the identical fault code across consecutive boots, and no amount of additional reboots will fix it. Building the ring buffer and the reboot counter is what makes that distinction visible at all; without them, both look identical from the outside: a node that went quiet and came back.

Related #

• Serial Console Health Endpoints for Field Techs — reading the degradation flag and reboot counter over UART with no network.
• Hardware Watchdog Recovery for Stalled Pipelines — the ioctl-level detail behind arming, kicking, and disarming /dev/watchdog.
• Async Execution for Spatial Workloads — the worker-timeout discipline this guide’s liveness budgets are built to bound.
• Monitoring & Observability — the thermal and queue-depth signals that distinguish a real stall from a busy device.
• Edge Operations & Observability — the operational layer this recovery discipline belongs to.