Sysfs Thermal Polling for Throttle-Aware Pipelines

This page solves one concrete problem: driving a spatial pipeline’s coarsening decisions from live SoC temperature and clock frequency, read directly from sysfs, on a Linux gateway (Raspberry Pi class through industrial Cortex-A modules) running the ingestion loop in Python — coarsening it before the kernel’s own thermal governor throttles the clock out from under it. Within the Edge Operations & Observability guide, and specifically as the thermal half of monitoring and observability, this is the narrow slice that turns a raw millidegree reading into a leading indicator a pipeline can actually act on.

The reason this needs its own page rather than a paragraph is that the naive version — read /sys/class/thermal/thermal_zone0/temp, compare to a trip point, react — is almost always wrong in the field. Raw thermal readings are noisy at the millisecond scale, a single spike crossing a threshold does not mean the SoC is actually in trouble, and reacting to every crossing produces exactly the alert-flapping and pipeline-thrashing this guide’s parent warns about. The fix is a smoothed signal, a cadence chosen against the pipeline’s own worst-case latency, and a hysteresis band wide enough that the coarsening decision only flips when the trend is real.

Polling cadence and hysteresis selection rationale #

Three properties decide the design here, and each maps to a specific failure if it is skipped.

Cadence must out-run the pipeline’s own reaction time. A polling interval of one to two seconds is fast enough to catch a thermal ramp on a passively cooled Cortex-A SoC — which typically takes tens of seconds to climb from idle to trip point under sustained load — while staying far below the cost of the file reads themselves. Polling every 100 ms buys no additional lead time on a ramp this slow, but does cost measurable CPU on a throttled core; polling every 10 s risks missing the window between “temperature is rising” and “clock has already dropped.”

Raw readings need smoothing, not thresholding. An exponentially weighted moving average (EWMA) turns a jittery per-sample reading into a trend line with one tunable parameter, alpha, and no history buffer to manage — the previous smoothed value and the new sample are the only state carried forward. That is the right shape for a device with no room for a rolling window of raw samples.

Hysteresis must be wider than the noise floor, in both directions. The coarsening decision needs two lines, not one: a rising threshold that trips the degraded mode, and a falling threshold — set several degrees lower — that has to be crossed before the pipeline is allowed to recover. Without the gap, a temperature oscillating right at a single trip point flips the mode every few samples, which is worse for pipeline stability than never reacting at all.

import time

THERMAL_ZONES = [
    "/sys/class/thermal/thermal_zone0/temp",
    "/sys/class/thermal/thermal_zone1/temp",
]
CPU_FREQ = "/sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq"

POLL_INTERVAL_S = 1.5
EWMA_ALPHA = 0.3          # weight on the newest sample; lower = smoother, slower to react
TRIP_C = 78.0             # enter degraded mode above this smoothed temperature
CLEAR_C = 68.0            # must fall below this before degraded mode is allowed to clear
MIN_DWELL_SAMPLES = 3      # consecutive samples required before a state change is honored

class ThermalMonitor:
    """Smoothed, hysteresis-gated thermal state for one or more thermal zones.

    Reads the hottest of the configured zones each cycle (some SoCs expose a
    GPU or modem zone separately from the CPU die), smooths it with an EWMA,
    and only flips `degraded` after MIN_DWELL_SAMPLES consecutive samples
    past the relevant line.
    """

    def __init__(self):
        self._ewma = None
        self._degraded = False
        self._dwell = 0

    def _read_hottest_zone(self):
        readings = []
        for path in THERMAL_ZONES:
            try:
                with open(path) as f:
                    readings.append(int(f.read().strip()) / 1000.0)
            except (FileNotFoundError, ValueError):
                continue
        return max(readings) if readings else None

    def _read_freq_khz(self):
        try:
            with open(CPU_FREQ) as f:
                return int(f.read().strip())
        except (FileNotFoundError, ValueError):
            return None

    def sample(self):
        """Call once per POLL_INTERVAL_S. Returns (smoothed_temp_c, freq_khz, degraded)."""
        raw = self._read_hottest_zone()
        freq = self._read_freq_khz()
        if raw is None:
            return self._ewma, freq, self._degraded

        self._ewma = raw if self._ewma is None else (
            EWMA_ALPHA * raw + (1 - EWMA_ALPHA) * self._ewma
        )

        # Which line matters depends on which side of the hysteresis band we are on:
        # look for TRIP_C while normal, look for CLEAR_C while already degraded.
        crossed = (self._ewma >= TRIP_C) if not self._degraded else (self._ewma <= CLEAR_C)

        if crossed:
            self._dwell += 1
            if self._dwell >= MIN_DWELL_SAMPLES:
                self._degraded = not self._degraded
                self._dwell = 0
        else:
            self._dwell = 0

        return self._ewma, freq, self._degraded

The dwell counter is doing the same job as a debounce circuit: it refuses to honor a threshold crossing until it has persisted for MIN_DWELL_SAMPLES consecutive cycles, which at a 1.5 s cadence means roughly 4.5 seconds of sustained trend before the pipeline’s coarsening mode actually changes. That is short enough to react before the kernel’s own governor drops the clock, and long enough that a single noisy sample near the line cannot flip the mode on its own.

The choice of EWMA_ALPHA = 0.3 deserves its own justification, since it is the one constant in this file with no obvious “correct” value. A lower alpha (0.1–0.15) produces a slower, smoother trend line that resists noise well but lags a genuine fast ramp — dangerous on a SoC that can go from idle to trip point in under thirty seconds under a sudden compute burst. A higher alpha (0.5+) tracks the raw signal closely enough that it barely smooths anything, defeating the point of the filter. The 0.3 used here is a reasonable middle ground for a one- to two-second cadence; boards with markedly different thermal mass (a large heatsinked industrial module versus a bare SoC on a small PCB) should re-derive it from a logged thermal ramp rather than trust this default, exactly as the sibling page on the metric surface recommends for every threshold in this guide.

Constraint validation table #

Gotchas and edge cases #

• Zone numbering is not portable. thermal_zone0 is the CPU die on most Broadcom and Rockchip SoCs, but some boards expose the GPU, a PMIC, or a modem thermal sensor at index 0 instead. Enumerate /sys/class/thermal/thermal_zone*/type once at startup and confirm which index is actually the CPU package before hard-coding a path.
• scaling_cur_freq can be absent. A board running the performance governor with frequency scaling disabled, or a kernel built without cpufreq, will not expose this file at all. Treat a missing read as “no signal” rather than zero — a frequency of zero would otherwise look like a full throttle event and trigger coarsening for the wrong reason.
• Millidegree units are easy to get backwards. The kernel reports thermal zone temperature in thousandths of a degree Celsius; dividing by the wrong factor silently produces a value that looks plausible (in the tens or thousands) but is off by 10x or 1000x. Always verify against a known-good reading (cat the file directly) once per board revision.
• EWMA alpha interacts with cadence. alpha and POLL_INTERVAL_S are not independent — halving the interval without adjusting alpha effectively doubles the smoothing time constant. If cadence changes during tuning, re-derive alpha rather than reusing the old value.
• Watchdog interaction. If this polling loop runs on the same asyncio loop as ingestion, a blocking file read that unexpectedly hangs (a wedged sysfs node during a kernel panic-in-progress) can starve the loop past the hardware watchdog window described in the parent guide’s recovery layer. Wrap the reads with a short asyncio.to_thread call if a board has ever shown this behavior.
• Thermal throttling can outrun even a well-tuned poll. Some SoCs implement a hardware-level thermal cutback that reacts within milliseconds of a die-temperature spike — far faster than any software polling loop, however tight. Sysfs-driven coarsening is a complement to that hardware protection, aimed at avoiding it altogether by degrading earlier and more gracefully, not a replacement for it. Never assume the software loop is the only thing standing between a workload and a throttled clock.

Integrating with the gateway pipeline #

The ThermalMonitor above is meant to sit beside the ingestion loop, not inside it. Call sample() once per cycle from the same scheduler that drives async execution for spatial workloads, and feed the returned degraded flag into the same chunk-size decision that pipeline already exposes:

import asyncio

monitor = ThermalMonitor()

async def thermal_poll_loop(on_state_change):
    prior = False
    while True:
        temp, freq, degraded = monitor.sample()
        if degraded != prior:
            on_state_change(degraded, temp, freq)
            prior = degraded
        await asyncio.sleep(POLL_INTERVAL_S)

def apply_degraded_mode(degraded: bool, temp: float, freq):
    # Wire this into the pipeline's own CHUNK_SIZE / worker count knobs.
    global CHUNK_SIZE
    CHUNK_SIZE = 32 if degraded else 128

Running the poll loop as its own coroutine, rather than folding the sysfs reads into the ingestion path directly, keeps the two concerns testable independently: the ingestion loop can be exercised with a mocked degraded flag, and the thermal loop can be replayed against a logged temperature trace without touching the pipeline at all.

Related #

• Monitoring & Observability for Spatial Gateways — the constraint mapping and metric-surface pattern this thermal signal feeds into.
• Prometheus Edge Exporter for Gateway Metrics — exposing the smoothed temperature as a scraped gauge.
• Async Execution for Spatial Workloads — the chunk-size and worker-count knobs the degraded flag actually drives.