Monitoring & Observability for Spatial Gateways

Within the Edge Operations & Observability guide, monitoring is the instrumentation layer that has to obey a stricter budget than anything it watches: on a fanless ARM gateway pulling milliwatts from a battery, the monitor must cost less than what it monitors, or it becomes the load it was built to catch. This guide covers the cheap end of that contract — reading thermal, clock, and memory state straight out of sysfs and procfs, holding a minimal metric surface in a few kilobytes of RAM, and choosing alert thresholds that fire early enough to act on without training a field technician to ignore the pager. It stops short of the wire format and the sync-path specifics; those are covered by its three companion pages below.

The design constraint that shapes every choice here is that a spatial gateway rarely has a human watching it live. A monitoring loop that samples too aggressively steals cycles from the same core running async spatial workloads, and a metric surface that grows without bound eventually competes with the pipeline it is meant to protect for the same RAM ceiling documented in device constraints and resource limits. Everything below is built against a Linux gateway — Raspberry Pi class through industrial Cortex-A modules — with sysfs and procfs available and a Python runtime already resident for the spatial pipeline.

Constraint mapping #

Every parameter in this guide traces back to a specific hardware ceiling. Before tuning a sample rate or a threshold, map it to the limit it is actually protecting:

The single hardest number to get right is the sampling interval itself: too fast and the monitor is the load; too slow and by the time a threshold trips, the pipeline it was meant to protect has already missed its own deadlines. The two implementation sections below build the sampler and the metric surface around that trade-off.

The same table scales down to microcontroller-class nodes running Zephyr or FreeRTOS, even though those boards rarely expose a POSIX-style sysfs tree. On that class of hardware the “sampler” is a direct register read of an on-die temperature sensor and the “registry” is a fixed struct written to a reserved RAM region — the discipline of bounded memory and cost-aware polling carries over even though none of the file paths below apply. The Linux-specific code in this guide assumes sysfs and procfs are present, which covers the large majority of ARM and RISC-V gateway deployments running a full OS.

Implementation: a throttle-aware sysfs sampler #

The sampler’s job is narrow on purpose: read three numbers — die temperature, current clock frequency, and this process’s resident set size — in one pass, with no allocation beyond a small tuple, and hand them to whatever decides what to do next. It never computes an average, never opens a socket, and never blocks; those responsibilities belong to the aggregation and export stages.

import time

THERMAL_ZONE = "/sys/class/thermal/thermal_zone0/temp"
CPU_FREQ = "/sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq"
SELF_STATUS = "/proc/self/status"

def read_sample():
    """Read temp (C), current clock (kHz), and this process's RSS (KB).

    Three plain file reads, no parsing beyond int()/split(). Missing files
    (no cpufreq governor, a kernel built without thermal_zone0) degrade to
    None rather than raising, so one absent sysfs node never kills the loop.
    """
    sample = {"ts": time.monotonic(), "temp_c": None, "freq_khz": None, "rss_kb": None}

    try:
        with open(THERMAL_ZONE) as f:
            sample["temp_c"] = int(f.read().strip()) / 1000.0
    except (FileNotFoundError, ValueError):
        pass

    try:
        with open(CPU_FREQ) as f:
            sample["freq_khz"] = int(f.read().strip())
    except (FileNotFoundError, ValueError):
        pass

    try:
        with open(SELF_STATUS) as f:
            for line in f:
                if line.startswith("VmRSS:"):
                    sample["rss_kb"] = int(line.split()[1])
                    break
    except FileNotFoundError:
        pass

    return sample

Three details make this edge-safe rather than a workstation script ported verbatim. First, each read is wrapped independently: a gateway missing a thermal zone (some RISC-V boards expose zones under a different index) still yields clock and memory data instead of aborting the whole sample. Second, VmRSS comes from /proc/self/status rather than a library like psutil, because pulling in a dependency to read one line of a text file is exactly the kind of cost this guide argues against. Third, the function returns a plain dict with a monotonic timestamp — never time.time(), since a gateway with no real-time-clock battery loses wall-clock time on every power cut and a monotonic clock is what hysteresis and rate calculations actually need.

Calling read_sample() on a fixed cadence, rather than continuously, is what keeps the sampler itself off the CPU budget it exists to protect:

import asyncio

SAMPLE_INTERVAL_S = 2.0

async def sampling_loop(on_sample):
    """Call on_sample(dict) once per SAMPLE_INTERVAL_S. Runs on the same
    asyncio loop as ingestion; each call is a handful of syscalls, not a
    blocking operation, so it never competes with the ingestion queue."""
    while True:
        on_sample(read_sample())
        await asyncio.sleep(SAMPLE_INTERVAL_S)

Because each read_sample() call is three short file reads, it costs microseconds even on a throttled core — the loop’s own footprint is close to zero, leaving the budget for aggregation and, occasionally, export.

The same shape extends cleanly to a handful of additional signals worth sampling on boards that expose them: /sys/class/power_supply/*/voltage_now on a battery-backed node, or a second thermal zone for a modem or GPU die that runs hotter than the CPU package under sustained load. Add each as its own guarded try/except block rather than a shared one — the point of isolating every read is that one absent path never takes the others down with it, and that property degrades quickly if reads are batched behind a single exception handler.

Implementation: a minimal metrics registry and exporter #

A metric surface earns its keep only if it stays small and fixed. The trap on a constrained node is copying a server-side pattern wholesale: dynamic labels, per-request gauges, a registry that grows every time a new device or session shows up. None of that is appropriate on a device with a few hundred megabytes of RAM total. The registry below declares every gauge once at startup, updates it in place, and renders a small, fixed-size text block on demand:

import threading
import time

class MetricsRegistry:
    """Fixed-cardinality in-memory gauge registry. Every name is declared
    once at startup; set() only ever updates an existing entry in place,
    so the registry's memory footprint never grows after boot."""

    def __init__(self):
        self._values = {}
        self._lock = threading.Lock()

    def declare(self, name: str, help_text: str):
        with self._lock:
            self._values.setdefault(name, {"help": help_text, "value": 0.0, "ts": 0.0})

    def set(self, name: str, value):
        if value is None:
            return
        with self._lock:
            entry = self._values[name]
            entry["value"] = float(value)
            entry["ts"] = time.monotonic()

    def render_text(self) -> str:
        """Prometheus-compatible text exposition, hand-rolled so a node
        that only ever serves a handful of gauges does not carry a client
        library for the privilege."""
        lines = []
        with self._lock:
            for name, entry in self._values.items():
                lines.append(f"# HELP {name} {entry['help']}")
                lines.append(f"# TYPE {name} gauge")
                lines.append(f"{name} {entry['value']:.3f}")
        return "\n".join(lines) + "\n"

registry = MetricsRegistry()
registry.declare("gateway_temp_celsius", "SoC die temperature")
registry.declare("gateway_cpu_freq_khz", "Current CPU clock frequency")
registry.declare("gateway_rss_kb", "Process resident set size")
registry.declare("gateway_queue_depth", "Pending items in the sync queue")

Wiring the sampler to the registry is a one-line callback; the registry never reaches back into sysfs itself, which keeps the read path and the storage path independently testable:

def on_sample(sample):
    registry.set("gateway_temp_celsius", sample["temp_c"])
    registry.set("gateway_cpu_freq_khz", sample["freq_khz"])
    registry.set("gateway_rss_kb", sample["rss_kb"])

Exposing render_text() over HTTP is the last mile, and the cheapest version uses the standard library’s http.server rather than a WSGI stack: a single-threaded handler that reads the cached registry and writes the response, never touching sysfs on the request path itself.

from http.server import BaseHTTPRequestHandler, HTTPServer

class MetricsHandler(BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path != "/metrics":
            self.send_response(404)
            self.end_headers()
            return
        body = registry.render_text().encode("utf-8")
        self.send_response(200)
        self.send_header("Content-Type", "text/plain; version=0.0.4")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)

    def log_message(self, fmt, *args):
        pass  # skip stdout logging per request; it is not free on flash-backed storage

This hand-rolled surface is deliberately small: four gauges, one endpoint, no dependency beyond the standard library. When a deployment needs a real client-side library with counters, histograms, and a maintained wire-format implementation, the Prometheus edge exporter for gateway metrics builds the same idea on prometheus_client and covers the cardinality and caching trade-offs that come with it.

Notice that every value declared above is a gauge, never a counter or a histogram. Gauges are the right shape for this device class because they represent instantaneous state — the value that matters is “what is the temperature right now,” not a cumulative total that has to be reset or rate-derived downstream. Counters and histograms are real tools with real uses, but they imply a scrape-and-aggregate pipeline that is out of scope for a node whose entire metric surface fits in four lines of text. If a deployment later needs rate calculations — how fast the queue is filling, for instance — compute the rate on-device from two gauge samples rather than exporting a counter and pushing the derivative downstream, the same approach its companion page takes for sync-queue alerting.

Configuration & tuning #

The whole tuning surface is three numbers and one behavioural rule:

• Sample cadence. Two seconds is a reasonable default for a Cortex-A gateway under continuous spatial load; a solar node duty-cycling at a slower rhythm can safely stretch this to ten or thirty seconds between wake windows. The rule is not a fixed number but a ratio: cadence should be short enough that a thermal trend crosses a threshold with at least two or three samples of lead time before the SoC actually throttles.
• Hysteresis band. Never alert or degrade on a single sample crossing a raw threshold. Require the smoothed value to stay above (or below) the line for a minimum dwell — a few consecutive samples — before acting, and require it to drop past a lower line, not just back under the trip point, before clearing. The exact cadence and hysteresis constants for the thermal case are worked out in sysfs thermal polling for throttle-aware pipelines.
• Registry size. Fix the gauge count at declaration time and never add a label with unbounded cardinality (per-packet IDs, per-GPS-fix timestamps). If a value needs history, keep a short fixed-length ring buffer alongside the gauge, not a growing series.
• Export cost. Cache the rendered text between reads rather than recomputing it on every scrape; a same-LAN scraper polling every 15 seconds should never trigger a fresh sysfs read, since the sampling loop already owns that cadence.

Systemd units running this loop should still set MemoryLimit= and CPUQuota= so a bug in the aggregation or export path is contained by cgroups rather than competing silently with the spatial pipeline for the same cycles.

Treat every default in this section as a starting point measured against the actual board, not a value to copy from this page verbatim. Two identical-looking Cortex-A modules from different vendors can throttle at different die temperatures depending on heatsink mass and enclosure airflow, and the “right” sample cadence on a passively cooled outdoor unit in direct sun is shorter than on the same board in a shaded equipment cabinet. Run a controlled thermal ramp during provisioning — load the CPU deliberately and watch the raw sysfs values climb — and derive the hysteresis band from what that test actually shows, rather than the numbers used as illustrations above.

Verification & field diagnostics #

Confirming the monitor is doing its job — and not becoming a problem of its own — means checking both what it reports and what it costs:

1. Loop overhead. Time read_sample() directly on the target hardware; it should cost low single-digit milliseconds. If it creeps past that, check for a sysfs path resolving through a slow overlay filesystem or a busy governor.
2. Cadence drift. Log the actual interval between samples, not just the configured one. A loop sharing a core with a busy ProcessPoolExecutor running spatial workers can silently stretch its own sampling interval under load, which is exactly the condition it is supposed to catch.
3. Threshold lead time. Replay a logged thermal ramp and confirm the hysteresis-gated alert fires with enough margin before the clock actually drops, not after.
4. Export cost. Confirm the exporter’s endpoint never triggers a live sysfs read; a scraper hitting /metrics should only ever see the last cached sample.
5. Registry size over time. Watch the registry’s memory footprint across a multi-day run. Flat is correct; any growth points at a label or history buffer that was not actually bounded.
6. Cross-check against a second source. On a first deployment to a new board revision, confirm the sysfs reading against an external reference — a thermal camera on the enclosure, or a bench power supply’s current draw as a proxy for load — at least once. A misread thermal zone that is actually the modem’s die, not the CPU’s, will look plausible in every log line while silently monitoring the wrong component.

Failure modes and recovery #

• The monitor becomes the load. A sampling interval set too tight, or an exporter that re-reads sysfs on every HTTP request, consumes the CPU and power budget it was built to protect — and on a thermal reading, actively worsens the temperature it is measuring. Detect: CPU time attributed to the monitoring process rises under the same conditions it is supposed to flag. Recover: move to duty-cycle sampling, serve exports from the cache, and treat any live re-read on the request path as a bug.
• Alert storms train the operator to ignore alarms. A threshold with no hysteresis flaps every time a noisy reading crosses the line, and a field technician who has seen fifty false pages stops reading the fifty-first. Detect: alert count per hour that does not correlate with any real degradation. Recover: add a minimum dwell and a separate, lower clearing threshold to every alert; never fire on a single sample.
• Cardinality creep. A registry that adds a label per device, per session, or per packet grows RAM usage slowly enough to escape notice until an OOM kill during an unrelated load spike. Detect: registry size measured in bytes climbing across days, not requests. Recover: audit every declare() call site for parameters that vary at runtime and collapse them into a fixed set of gauges.
• Silent sysfs absence. A kernel rebuild, a different SoC revision, or a container without /sys bind-mounted makes a path disappear, and an unguarded read crashes the whole loop instead of just one metric. Detect: the sampling loop stops entirely rather than reporting one gauge as stale. Recover: wrap every sysfs read independently, exactly as read_sample() does, so a missing node degrades one field instead of the whole monitor.
• Stale metrics look identical to healthy ones. A gauge that is never updated again after a crash in the sampling loop still renders a plausible-looking value forever, and a dashboard has no way to distinguish “healthy and unchanged” from “stopped reporting three days ago.” Detect: export the ts field alongside each value, or a derived seconds_since_last_sample gauge, and alert on staleness directly rather than trusting a frozen number. Recover: treat any gauge older than a few sample intervals as unknown, not as its last value, in both dashboards and automated threshold checks.

Related #

• Sysfs Thermal Polling for Throttle-Aware Pipelines — cadence, hysteresis, and EWMA smoothing for the thermal signal specifically.
• Prometheus Edge Exporter for Gateway Metrics — the same metric surface built on prometheus_client for a same-LAN scraper.
• Queue-Depth Alerting Thresholds for Edge Sync — high/low water marks and rate-of-change alerting for the sync path.
• Async Execution for Spatial Workloads — the scheduling discipline this sampler has to share a core with.
• Device Constraints & Resource Limits — the RAM, CPU, and thermal envelope every threshold in this guide is set against.