Hardware watchdog recovery for stalled pipelines

The problem this page solves: a spatial ingestion thread on a Linux gateway wedges — a blocked socket read, a native FFI call that never returns, a deadlock between the async loop and a worker pool — and nothing short of a hard reset will bring it back, yet that reset must never land in the middle of a write to the on-device store. Within Edge Operations & Observability, and specifically as the ioctl-level detail behind the recovery mechanics in Field Diagnostics & Recovery, this guide covers the Linux hardware watchdog character device directly: arming it with WDIOC_SETTIMEOUT, kicking it only while liveness holds, and using the driver’s magic-close byte and a shutdown hook that flushes and closes the memory-mapped store so a forced reboot never corrupts what it was trying to save.

Watchdog selection rationale #

A software watchdog — a supervisor process that restarts a hung child — only works if something outside the hang is still scheduled to notice it. On a gateway where the hang can be a full kernel-level deadlock (a stuck driver, a wedged DMA transfer), a userspace supervisor can be starved of CPU time right alongside the process it was meant to watch. The hardware watchdog sidesteps that failure mode entirely: it is a timer in silicon (or in an always-on microcontroller companion on some SoCs) that is completely independent of the Linux scheduler, and it will force a reset regardless of what the CPU is doing, short of the SoC being fully powered off.

The Linux kernel exposes this timer as a character device, conventionally /dev/watchdog. Three properties of that device drive every design decision below:

1. Only one process may hold it open. A second open() on most drivers (including the generic softdog module and the vast majority of SoC-specific watchdog drivers) fails with EBUSY. This is a feature, not a limitation — it forces a single, unambiguous owner of the kick responsibility rather than several processes racing to decide whether the device is alive.
2. Closing it does not disarm it by default. Unless the driver was built or configured with nowaitdog/nowayout disabled, simply calling close() on the file descriptor leaves the countdown running. The only way to stop it cleanly is the magic close: writing the ASCII byte V to the device immediately before close().
3. The timeout is negotiated, not fixed. WDIOC_SETTIMEOUT asks the driver for a value in seconds; many drivers round to the nearest value their hardware counter actually supports and hand back the real number through the same ioctl call, which is why the code below always reads back what it got rather than trusting what it asked for.

The complete recovery implementation #

The design combines three responsibilities in one process: negotiate the watchdog timeout at startup, kick it only when an explicit liveness check passes, and — critically — flush and unmap the spatial store from every exit path, whether that exit is a clean shutdown signal or the deliberate decision to let the hardware timer fire.

import fcntl
import mmap
import os
import signal
import struct
import sys
import time

WDIOC_SETTIMEOUT = 0xC0045706   # linux/watchdog.h
WDIOC_GETTIMEOUT = 0x80045707


class WatchdogHandle:
    def __init__(self, path="/dev/watchdog", timeout_s=25):
        # Only one process may hold /dev/watchdog open; a second open() call
        # from another process fails with EBUSY on nearly every driver.
        self._fd = os.open(path, os.O_WRONLY)
        got = fcntl.ioctl(self._fd, WDIOC_SETTIMEOUT, struct.pack("i", timeout_s))
        self.timeout_s = struct.unpack("i", got)[0]   # trust what the driver accepted

    def kick(self):
        os.write(self._fd, b"\0")   # any write resets the countdown

    def disarm_and_close(self):
        # Magic close: writing 'V' immediately before close() stops the timer.
        # Only call this on a verified clean shutdown path.
        os.write(self._fd, b"V")
        os.close(self._fd)

    def abandon(self):
        # Close WITHOUT the magic byte: the countdown keeps running and the
        # SoC reboots at self.timeout_s regardless of what happens after
        # this call returns. Used when the process cannot trust its own
        # state enough to claim a clean shutdown.
        os.close(self._fd)


class SpatialStore:
    """Owns the mmap-backed spatial index. Every mutation happens through
    this object so there is exactly one place responsible for flushing it
    before a reboot can land on a half-written page."""

    def __init__(self, path, size_bytes):
        self._fd = os.open(path, os.O_RDWR | os.O_CREAT, 0o644)
        if os.fstat(self._fd).st_size < size_bytes:
            os.ftruncate(self._fd, size_bytes)
        self.mm = mmap.mmap(self._fd, size_bytes)

    def close(self):
        # flush() forces dirty pages out to the backing file before the
        # mapping is torn down — the msync-equivalent for Python's mmap.
        self.mm.flush()
        self.mm.close()
        os.close(self._fd)


def clean_shutdown(watchdog: WatchdogHandle, store: SpatialStore, expect_reboot: bool):
    store.close()
    if expect_reboot:
        watchdog.abandon()          # let the timer finish what the fault started
    else:
        watchdog.disarm_and_close()
    sys.exit(0)


def liveness_ok(last_ingest_ts: float, stall_budget_s: float) -> bool:
    return (time.monotonic() - last_ingest_ts) < stall_budget_s


def main():
    watchdog = WatchdogHandle(timeout_s=25)
    store = SpatialStore("/var/lib/gateway/spatial_index.mmap", 64 * 1024 * 1024)
    last_ingest_ts = time.monotonic()

    def handle_sigterm(signum, frame):
        clean_shutdown(watchdog, store, expect_reboot=False)

    signal.signal(signal.SIGTERM, handle_sigterm)

    try:
        while True:
            # ingest_chunk() would advance last_ingest_ts on every chunk it
            # finishes; omitted here since the loop body is application-specific.
            if liveness_ok(last_ingest_ts, stall_budget_s=15.0):
                watchdog.kick()
            else:
                # Withhold the kick. Do not call disarm_and_close() here —
                # the entire point is to let the hardware timer fire and
                # force a reboot the process could not achieve on its own.
                time.sleep(1.0)
                continue
            time.sleep(watchdog.timeout_s / 5)
    except KeyboardInterrupt:
        clean_shutdown(watchdog, store, expect_reboot=False)


if __name__ == "__main__":
    main()

The line that carries the whole design is the branch inside the while loop: when liveness_ok returns False, the code does not attempt to force-close anything or call abandon() immediately — it simply stops kicking and waits. The hardware timer, not the application, decides when the reboot happens, which is exactly the guarantee needed when the process itself cannot be trusted to know whether it is genuinely stuck.

Constraint validation table #

Gotchas and edge cases #

• The timeout must exceed the worst-case chunk time, not the median. If stall_budget_s or the hardware timeout_s is set from typical-case latency, a single thermally-throttled chunk that runs long but is still making progress will trip the watchdog and reboot a perfectly healthy pipeline — the same self-inflicted failure the parent guide’s async execution for spatial workloads worker timeout has to avoid one layer down.
• nowayout compiled into the kernel disables the magic close entirely. On some hardened builds, once /dev/watchdog is opened, no write of any kind can stop the timer before the process exits — check /sys/class/watchdog/watchdog0/nowayout (or the kernel config used to build the image) before assuming disarm_and_close() will work as written; if nowayout is set, the only real “disarm” is to keep kicking until an intentional reboot is desired anyway.
• A second process opening /dev/watchdog fails with EBUSY, not a warning. If a monitoring agent or a debugging script also tries to open the device for inspection, it will get EBUSY and may crash-loop itself if that error isn’t handled — read WDIOC_GETTIMEOUT state through /sys/class/watchdog/watchdog0/ sysfs attributes instead of opening the character device a second time.
• systemd’s own WatchdogSec= is a separate mechanism. If a unit file sets WatchdogSec= and expects sd_notify(WATCHDOG=1) pings, that is systemd’s software watchdog talking to the service manager, not the hardware device — running both without understanding the difference can mean two independent timers with two independent timeout assumptions, one of which nobody is tuning.
• The bootloader watchdog and the SoC watchdog can be different timers. Some platforms arm a watchdog in U-Boot or the boot ROM that is separate from the Linux-visible /dev/watchdog; confirm which timer the userspace code is actually driving, because disarming one does nothing to the other.
• abandon() must genuinely not run any recovery code afterward. The entire safety property depends on the process doing nothing further once it decides to let the timer fire — any code path that tries to “clean up a little” after calling abandon() risks running into the exact stall condition that triggered the decision in the first place, defeating the point of falling back to hardware.

Integrating with the gateway pipeline #

Run this process as the sole owner of both the watchdog device and the spatial store’s file descriptor, supervised by systemd with restart enabled so a clean exit (one that reached disarm_and_close()) still restarts promptly, while an abandoned exit rides out the hardware reboot instead:

# /etc/systemd/system/spatial-pipeline.service
[Unit]
Description=Spatial ingestion pipeline with hardware watchdog recovery
After=network.target

[Service]
ExecStart=/usr/bin/python3 /opt/gateway/pipeline_main.py
Restart=on-success
TimeoutStopSec=10
User=gateway

[Install]
WantedBy=multi-user.target

Restart=on-success deliberately does not cover the abandon() path: that exit either never returns (the hardware fires first) or the process has already accepted a reboot is coming, so there is nothing for systemd to restart before the SoC does it anyway. TimeoutStopSec=10 bounds how long systemd waits for SIGTERM’s handler to run clean_shutdown() before escalating to SIGKILL — set it below the negotiated timeout_s so an orderly service stop always wins the race against the hardware timer. For the technician-facing side of this recovery — reading the resulting reboot counter and fault code without any network access — see serial console health endpoints for field techs.

Related #

• Field Diagnostics & Recovery — the liveness-tracker and persisted-snapshot design this watchdog handle plugs into.
• Serial Console Health Endpoints for Field Techs — reading the reboot counter this recovery path produces, with no network.
• Async Execution for Spatial Workloads — the worker-level timeout that this hardware timeout has to sit above.