Serial console health endpoints for field techs

The problem this page solves is narrow and unglamorous: a technician stands next to a gateway whose modem has been dead for three weeks, has a laptop and a USB cable, and needs to know whether the device is degraded, when it last produced good output, and how many times it has rebooted — with zero network access of any kind. Within Edge Operations & Observability, and specifically as a companion to the recovery mechanics in Field Diagnostics & Recovery, this guide builds a minimal line-protocol server that answers exactly those questions over a UART or USB-serial link, on a Linux gateway running Python, with no GUI and no shell login required.

Protocol selection rationale #

A full login shell over getty is the obvious answer and the wrong one for this job. It requires a working PAM stack, a shell binary, and enough RAM headroom for an interactive session — all defensible on a workstation, all wasted weight on a 256 MB node whose only job during this interaction is to answer four questions. It also hands a technician (or anyone who plugs in a cable) a full command shell on a production device, which is a larger blast radius than the diagnostic task needs.

The alternative built here is a dedicated Python process attached directly to a serial device node — /dev/ttyGS0 on a USB gadget-serial setup, or a real UART such as /dev/ttyS0 / /dev/ttyAMA0 on a header-exposed board — that speaks a tiny newline-terminated command protocol: send STATUS, get back one line of text. No shell, no PTY allocation, no authentication surface beyond physical access to the port. This mirrors the AT-command-style debug consoles common on embedded modems and radios, and it keeps the entire attack surface and resource footprint to a few read-only commands over a link that, by construction, only a person standing at the device can reach.

The health state itself is not computed fresh for every query — it is read from the same atomically-swapped snapshot file that the parent guide’s persistent health snapshot implementation writes, so the serial endpoint and the main pipeline never race over shared memory or a lock file. The console is a read-only window onto state someone else already persisted.

import struct
import time

STATE_PATH = "/var/lib/gateway/health_state.bin"
_FMT = "<QIIBxxxQI"   # monotonic_ns, boot_id, reboot_count, degraded, last_good_ts, fault_code


def _read_state():
    """Read-only view onto the same file the main pipeline writes atomically.
    Returns None on any short read rather than raising, since the file may
    be mid-rename at the exact instant this runs."""
    try:
        with open(STATE_PATH, "rb") as f:
            data = f.read(struct.calcsize(_FMT))
    except FileNotFoundError:
        return None
    if len(data) != struct.calcsize(_FMT):
        return None
    mono_ns, boot_id, reboot_count, degraded, last_good_ts, fault_code = struct.unpack(_FMT, data)
    return {
        "boot_id": boot_id,
        "reboot_count": reboot_count,
        "degraded": bool(degraded),
        "last_good_ts": last_good_ts,
        "fault_code": fault_code,
    }


def handle_line(line: str) -> str:
    """Dispatch one command line to a plain-text reply, terminated CRLF so
    a plain serial terminal (minicom, screen, PuTTY) renders it cleanly."""
    cmd = line.strip().upper()
    if cmd in ("HELP", "?"):
        return "CMDS: STATUS LASTGOOD REBOOTS HELP\r\n"

    state = _read_state()
    if state is None:
        return "ERR no-state\r\n"

    if cmd == "STATUS":
        flag = "DEGRADED" if state["degraded"] else "OK"
        return f"STATUS {flag} fault={state['fault_code']}\r\n"
    if cmd == "LASTGOOD":
        return f"LASTGOOD {state['last_good_ts']}\r\n"
    if cmd == "REBOOTS":
        return f"REBOOTS {state['reboot_count']} boot={state['boot_id']}\r\n"
    return "ERR unknown-cmd\r\n"


def serve_forever(port="/dev/ttyGS0", baudrate=115200):
    """Blocking read loop over pyserial. A 1 s read timeout keeps the loop
    from busy-polling while still responding promptly to a technician typing
    a command and pressing enter."""
    import serial   # pyserial

    with serial.Serial(port, baudrate=baudrate, timeout=1.0) as ser:
        buf = b""
        ser.write(b"gateway health console - type HELP\r\n")
        while True:
            chunk = ser.read(64)
            if not chunk:
                continue
            buf += chunk
            while b"\n" in buf:
                line, buf = buf.split(b"\n", 1)
                reply = handle_line(line.decode("ascii", "ignore"))
                ser.write(reply.encode("ascii"))


if __name__ == "__main__":
    serve_forever()

Everything in handle_line is deliberately read-only. There is no RESET or DEGRADE=0 command exposed here, because a serial port anyone can plug into is not the place to accept mutating commands on a production node — if a field procedure genuinely needs to clear a degraded flag, that belongs behind a separate, explicitly-authenticated maintenance mode, not this diagnostic console.

Constraint validation table #

Every design choice above maps to a hardware limit on the class of node this console targets — the general envelope is catalogued under device constraints and resource limits; the table below is the subset specific to a serial diagnostic path.

Gotchas and edge cases #

• /dev/ttyGS0 does not exist until the gadget is configured. On a USB gadget-serial setup the device node only appears after the kernel’s USB gadget subsystem has enumerated against a host; a serve_forever() call that starts before that happens needs a retry loop around the initial serial.Serial() open, not a hard crash.
• Baud rate and line-ending mismatches. A technician’s terminal emulator defaulting to raw \n instead of \r\n will still parse correctly here because handle_line splits on \n and strips whitespace, but always send replies with \r\n — some terminals render bare \n with no carriage return, producing a staircase effect that looks broken even when the protocol is fine.
• Permissions on the device node. /dev/ttyGS0 and real UART nodes are typically owned by root:dialout; run the console process as a member of dialout rather than root, so a bug in the console cannot escalate into arbitrary file access elsewhere on the device.
• Concurrent access. Only one process should hold the serial port open at a time — a stray getty still configured on the same port will fight the console process for the file descriptor. Disable any conflicting systemd serial-getty@ unit for the port this console owns.
• Stale reads during a snapshot rename. _read_state() can, in the narrow window of an in-flight atomic rename, see either the old or the new file — never a torn one, because the writer never rewrites in place. Treat a short read as “try again,” not as an error worth surfacing to the technician.
• No timestamp without a network, still no RTC. LASTGOOD reports a value stamped against a monotonic clock and boot id, not wall-clock time, because the node has no RTC battery. A technician reading raw nanosecond counters needs the boot id alongside it to know whether “old” means “before the last reboot” or “very recently.”

Integrating with the gateway pipeline #

Wire the console as its own lightweight systemd unit, independent of the main spatial pipeline process, so a wedged pipeline never takes the diagnostic console down with it — the console reads a file, and files remain readable even while the writer is stuck:

# /etc/systemd/system/health-console.service
[Unit]
Description=Serial health console for field technicians
After=dev-ttyGS0.device

[Service]
ExecStart=/usr/bin/python3 /opt/gateway/health_console.py
Restart=always
RestartSec=2
User=gateway
Group=dialout

[Install]
WantedBy=multi-user.target

Restart=always matters here specifically because the console’s own failure mode (the serial device node disappearing on a USB gadget re-enumeration) should self-heal without operator intervention, mirroring the same automatic-recovery expectation the rest of this practice applies to the main pipeline. For gateways where reporting also travels over an active uplink, this serial path is not a replacement for the queued health report described in message queue management at the edge — it is the fallback for the specific case where that uplink has been gone long enough that a truck has already been dispatched, and a lighter-weight complement to the Prometheus edge exporter used when a LAN scrape is available instead of a cable.

Related #

• Field Diagnostics & Recovery — the watchdog and persisted-snapshot mechanics this console reads from.
• Hardware Watchdog Recovery for Stalled Pipelines — the reboot path whose reboot counter this console reports.
• Monitoring & Observability — the throttle-aware instrumentation feeding the state this console displays.