Edge Geospatial IoT Gateway Processing

Serial console health endpoints for field techs

The problem this page solves is narrow and unglamorous: a technician stands next to a gateway whose modem has been dead for three weeks, has a laptop and a USB cable, and needs to know whether the device is degraded, when it last produced good output, and how many times it has rebooted — with zero network access of any kind. Within Edge Operations & Observability, and specifically as a companion to the recovery mechanics in Field Diagnostics & Recovery, this guide builds a minimal line-protocol server that answers exactly those questions over a UART or USB-serial link, on a Linux gateway running Python, with no GUI and no shell login required.

Protocol selection rationale

A full login shell over getty is the obvious answer and the wrong one for this job. It requires a working PAM stack, a shell binary, and enough RAM headroom for an interactive session — all defensible on a workstation, all wasted weight on a 256 MB node whose only job during this interaction is to answer four questions. It also hands a technician (or anyone who plugs in a cable) a full command shell on a production device, which is a larger blast radius than the diagnostic task needs.

The alternative built here is a dedicated Python process attached directly to a serial device node — /dev/ttyGS0 on a USB gadget-serial setup, or a real UART such as /dev/ttyS0 / /dev/ttyAMA0 on a header-exposed board — that speaks a tiny newline-terminated command protocol: send STATUS, get back one line of text. No shell, no PTY allocation, no authentication surface beyond physical access to the port. This mirrors the AT-command-style debug consoles common on embedded modems and radios, and it keeps the entire attack surface and resource footprint to a few read-only commands over a link that, by construction, only a person standing at the device can reach.

The health state itself is not computed fresh for every query — it is read from the same atomically-swapped snapshot file that the parent guide’s persistent health snapshot implementation writes, so the serial endpoint and the main pipeline never race over shared memory or a lock file. The console is a read-only window onto state someone else already persisted.

import struct
import time

STATE_PATH = "/var/lib/gateway/health_state.bin"
_FMT = "<QIIBxxxQI"   # monotonic_ns, boot_id, reboot_count, degraded, last_good_ts, fault_code


def _read_state():
    """Read-only view onto the same file the main pipeline writes atomically.
    Returns None on any short read rather than raising, since the file may
    be mid-rename at the exact instant this runs."""
    try:
        with open(STATE_PATH, "rb") as f:
            data = f.read(struct.calcsize(_FMT))
    except FileNotFoundError:
        return None
    if len(data) != struct.calcsize(_FMT):
        return None
    mono_ns, boot_id, reboot_count, degraded, last_good_ts, fault_code = struct.unpack(_FMT, data)
    return {
        "boot_id": boot_id,
        "reboot_count": reboot_count,
        "degraded": bool(degraded),
        "last_good_ts": last_good_ts,
        "fault_code": fault_code,
    }


def handle_line(line: str) -> str:
    """Dispatch one command line to a plain-text reply, terminated CRLF so
    a plain serial terminal (minicom, screen, PuTTY) renders it cleanly."""
    cmd = line.strip().upper()
    if cmd in ("HELP", "?"):
        return "CMDS: STATUS LASTGOOD REBOOTS HELP\r\n"

    state = _read_state()
    if state is None:
        return "ERR no-state\r\n"

    if cmd == "STATUS":
        flag = "DEGRADED" if state["degraded"] else "OK"
        return f"STATUS {flag} fault={state['fault_code']}\r\n"
    if cmd == "LASTGOOD":
        return f"LASTGOOD {state['last_good_ts']}\r\n"
    if cmd == "REBOOTS":
        return f"REBOOTS {state['reboot_count']} boot={state['boot_id']}\r\n"
    return "ERR unknown-cmd\r\n"


def serve_forever(port="/dev/ttyGS0", baudrate=115200):
    """Blocking read loop over pyserial. A 1 s read timeout keeps the loop
    from busy-polling while still responding promptly to a technician typing
    a command and pressing enter."""
    import serial   # pyserial

    with serial.Serial(port, baudrate=baudrate, timeout=1.0) as ser:
        buf = b""
        ser.write(b"gateway health console - type HELP\r\n")
        while True:
            chunk = ser.read(64)
            if not chunk:
                continue
            buf += chunk
            while b"\n" in buf:
                line, buf = buf.split(b"\n", 1)
                reply = handle_line(line.decode("ascii", "ignore"))
                ser.write(reply.encode("ascii"))


if __name__ == "__main__":
    serve_forever()

Everything in handle_line is deliberately read-only. There is no RESET or DEGRADE=0 command exposed here, because a serial port anyone can plug into is not the place to accept mutating commands on a production node — if a field procedure genuinely needs to clear a degraded flag, that belongs behind a separate, explicitly-authenticated maintenance mode, not this diagnostic console.

Serial health console data path A technician's laptop connects over a USB-serial cable to the gateway's UART, which is served by a line REPL parsing STATUS, LASTGOOD, and REBOOTS commands, which in turn reads the health state that was persisted atomically by the main pipeline. Technician laptop USB-serial cable, no network UART link /dev/ttyGS0 gadget serial Line REPL STATUS / LASTGOOD / REBOOTS Health state read from atomic snapshot
Four stages, no network hop: everything left of the health state lives on the cable a technician brought.

Constraint validation table

Every design choice above maps to a hardware limit on the class of node this console targets — the general envelope is catalogued under device constraints and resource limits; the table below is the subset specific to a serial diagnostic path.

Constraint Expected impact Mitigation built into the code
RAM (256 MB–2 GB shared with a modem) A full getty/PAM/shell stack costs tens of MB just to sit idle A single Python process with a fixed-size read buffer; no shell, no PTY
CPU An interactive session invites busy-polling if written carelessly Blocking ser.read() with a 1 s timeout — the loop sleeps between reads
Latency Human-paced interaction; no real-time deadline Response time is dominated by UART baud rate, not processing — negligible either way
Power Keeping a USB gadget or UART link powered has a small but nonzero draw Enable the gadget-serial function only when a technician connects, not continuously
Attack surface A physically-accessible port is still an entry point Read-only command set; no mutating commands exposed over this channel
A health response as it appears on a technician's terminal A mock terminal frame showing the response to the status command. The first two lines carry the firmware version and build identifier. Then a block of fixed-width fields: state, uptime, last fix age, queue depth with its cap, die temperature, and the last error with its age. Each field is annotated with what a technician should conclude from it — an ageing last fix points at the receiver or antenna, a queue at its cap points at the backhaul, and a recent error with a healthy state means the device already recovered. Everything a five-minute site visit needs, in one screen /dev/ttyUSB0 · 115200 8N1 > status fw 2.7.1 build 9f31c2a 2026-05-04 state RUNNING uptime 6d 04:12:33 last_fix 3s ago sats 9 hdop 0.9 queue 412 / 20000 draining temp 58.4 C throttle no spool 3.1 / 15.0 MB writes ok last_err 4h12m ago E_MQTT_TIMEOUT (recovered) version first — it is the first question in every ticket fix ageing? look at the antenna, not the software queue near cap? the link is the fault, not the device an old error with a healthy state means it recovered Fixed column positions matter: crews scan for the shape of a healthy response long before they read the values.
One command, eight lines, no tooling. The annotations are what belongs on the laminated card in the enclosure.

Gotchas and edge cases

  • /dev/ttyGS0 does not exist until the gadget is configured. On a USB gadget-serial setup the device node only appears after the kernel’s USB gadget subsystem has enumerated against a host; a serve_forever() call that starts before that happens needs a retry loop around the initial serial.Serial() open, not a hard crash.
  • Baud rate and line-ending mismatches. A technician’s terminal emulator defaulting to raw \n instead of \r\n will still parse correctly here because handle_line splits on \n and strips whitespace, but always send replies with \r\n — some terminals render bare \n with no carriage return, producing a staircase effect that looks broken even when the protocol is fine.
  • Permissions on the device node. /dev/ttyGS0 and real UART nodes are typically owned by root:dialout; run the console process as a member of dialout rather than root, so a bug in the console cannot escalate into arbitrary file access elsewhere on the device.
  • Concurrent access. Only one process should hold the serial port open at a time — a stray getty still configured on the same port will fight the console process for the file descriptor. Disable any conflicting systemd serial-getty@ unit for the port this console owns.
  • Stale reads during a snapshot rename. _read_state() can, in the narrow window of an in-flight atomic rename, see either the old or the new file — never a torn one, because the writer never rewrites in place. Treat a short read as “try again,” not as an error worth surfacing to the technician.
  • No timestamp without a network, still no RTC. LASTGOOD reports a value stamped against a monotonic clock and boot id, not wall-clock time, because the node has no RTC battery. A technician reading raw nanosecond counters needs the boot id alongside it to know whether “old” means “before the last reboot” or “very recently.”
Keeping the console read-only and off the pipeline's critical path The pipeline publishes its state into a small shared snapshot structure guarded by a lock held for microseconds. The console task reads that snapshot and formats a response; it never calls into the pipeline, never takes a lock the pipeline waits on, and has no write path at all. A crossed-out arrow shows the design being rejected — a console command that reaches into the pipeline to query it live, which turns a diagnostic tool into a way for a technician to stall the very thing they came to inspect. The console reads a copy — never the pipeline itself ingestion pipeline writes the snapshot once a second lock held for ~2 µs state snapshot plain struct · fixed fields no allocation on read console task formats and writes to the UART read-only, fixed command set rejected: a command that queries the pipeline live a technician typing during a burst would then add latency to the path they came to diagnose A stale snapshot is a smaller problem than a stalled pipeline, and one second of staleness is invisible to a human at a terminal.
Diagnostics that can perturb the system are diagnostics people stop trusting — and, worse, stop using at the moment they are needed.

One more habit is worth building into the command set: print the firmware version and the build identifier at the top of every response, not just from a version command. A technician pasting console output into a ticket almost never includes the version, and the first question anyone asks is which build the device is running. Two extra lines of output remove an entire round trip from every remote diagnosis, and they cost nothing on a link that is a serial cable.

Keep the output stable across releases as well. Field crews learn the shape of a healthy response and spot deviations faster than any parser does, so reordering fields or renaming a label between firmware versions destroys accumulated pattern recognition for no benefit. Add new fields at the end; never repurpose an existing one.

Integrating with the gateway pipeline

Wire the console as its own lightweight systemd unit, independent of the main spatial pipeline process, so a wedged pipeline never takes the diagnostic console down with it — the console reads a file, and files remain readable even while the writer is stuck:

# /etc/systemd/system/health-console.service
[Unit]
Description=Serial health console for field technicians
After=dev-ttyGS0.device

[Service]
ExecStart=/usr/bin/python3 /opt/gateway/health_console.py
Restart=always
RestartSec=2
User=gateway
Group=dialout

[Install]
WantedBy=multi-user.target

Restart=always matters here specifically because the console’s own failure mode (the serial device node disappearing on a USB gadget re-enumeration) should self-heal without operator intervention, mirroring the same automatic-recovery expectation the rest of this practice applies to the main pipeline. For gateways where reporting also travels over an active uplink, this serial path is not a replacement for the queued health report described in message queue management at the edge — it is the fallback for the specific case where that uplink has been gone long enough that a truck has already been dispatched, and a lighter-weight complement to the Prometheus edge exporter used when a LAN scrape is available instead of a cable.

Related in Field Diagnostics & Recovery