Brownout-safe writes for battery-powered gateways

A brownout is not a power cut. The supply sags rather than disappearing, the SoC keeps running for a few hundred milliseconds at a voltage where its behaviour is undefined, and the flash controller may accept a write it cannot complete. The result is worse than a clean loss of power: a corrupted block, a filesystem that needs a fsck nobody is present to run, and occasionally a device that will not boot. This guide covers detecting the sag early and getting the storage into a safe state before the voltage reaches the danger band, inside power and duty cycling and the Edge Operations & Observability guide.

What actually happens as the rail sags #

Three thresholds matter and they are not the same voltage.

The application threshold is where the device should decide it is in trouble — typically 10–15% above the point where anything misbehaves. There is still time here to finish a write, flush a buffer and set a flag.

The undefined band begins where individual components start operating out of specification. The SoC may still execute instructions, the eMMC may still accept commands, and neither is guaranteed to do what it was asked. Writes issued in this band are the ones that corrupt.

The brown-out reset threshold is where the SoC’s own detector fires and holds it in reset. Nothing runs below it, which is safe — the danger is entirely in the band above.

The design goal is therefore to spend no time in the undefined band with a write outstanding, which means acting at the application threshold and having a bounded, short shutdown path.

Detecting the sag #

Three detection routes exist, in descending order of warning time.

A dedicated supervisor IC with a comparator on the raw supply raises a GPIO interrupt at a voltage set by a resistor divider, before any regulator downstream sags at all. It is a few pence, it gives the most warning, and it is the only option that works on a device whose SoC has no ADC.

An ADC on a divided supply, polled at 10–20 Hz. Cheap, adequate, and adds detection latency equal to half the poll interval — which is why the polling rate matters more than the ADC’s precision.

A fuel gauge or PMIC alert, where one is present. It typically reports battery state rather than the rail, which makes it good for scheduling decisions and mediocre for brownout detection, because a battery under a heavy transient load sags faster than its state of charge suggests.

# brownout.py — supply monitor and bounded quiesce.
# The monitor runs on its own thread at real-time priority: it must not be
# waiting behind the spatial worker when the rail starts falling.
import os
import threading
import time

APP_THRESHOLD_MV = 10_800
CLEAR_MV = 11_400                 # hysteresis: do not oscillate on a transient
POLL_HZ = 20


class BrownoutMonitor:
    __slots__ = ("read_mv", "quiesce", "resume", "_low", "_thread", "_stop", "events")

    def __init__(self, read_mv, quiesce, resume):
        self.read_mv = read_mv           # callable -> millivolts
        self.quiesce = quiesce           # callable, must be bounded and fast
        self.resume = resume
        self._low = False
        self._stop = threading.Event()
        self.events = 0
        self._thread = threading.Thread(target=self._run, daemon=True,
                                        name="brownout")

    def start(self):
        self._thread.start()
        try:
            os.sched_setscheduler(0, os.SCHED_FIFO, os.sched_param(10))
        except (AttributeError, PermissionError):
            pass                          # best effort; still works at normal priority

    def _run(self):
        period = 1.0 / POLL_HZ
        while not self._stop.is_set():
            mv = self.read_mv()
            if not self._low and mv <= APP_THRESHOLD_MV:
                self._low = True
                self.events += 1
                self.quiesce(mv)          # must complete in well under the window
            elif self._low and mv >= CLEAR_MV:
                self._low = False
                self.resume()
            time.sleep(period)

The quiesce path #

Everything the quiesce routine does must be bounded, and the bound has to be measured rather than assumed. The order matters as much as the content: stop creating new work first, then make the existing work durable, then record that it happened.

# quiesce.py — the bounded shutdown path. Target: under 60 ms on this hardware.
# Every step has a measured worst case; the sum is the budget.
def quiesce(mv: int, spool, recorder, tiles, state) -> None:
    # 1. Stop admitting work. Cheap, and it bounds everything after it. (~0.1 ms)
    spool.stop_accepting()
    tiles.stop_accepting()

    # 2. Record the event first, while writes are still safe, so a device that
    #    does not survive the rest still explains itself on the next boot. (~1 ms)
    recorder.record(kind=4, code=2, a=mv)          # brownout_detected
    recorder.flush()

    # 3. Make the data path durable. One fsync on an already-small buffer. (~25 ms)
    spool.flush_and_sync()

    # 4. Persist any small state that would otherwise be reconstructed. (~15 ms)
    state.flush()                                   # geofence membership, cursors

    # 5. Set a flag so the next boot knows this was a brownout, not a crash.
    #    A single small write, last, because it is the least important. (~10 ms)
    state.mark_unclean_shutdown(reason="brownout", mv=mv)


def resume(spool, tiles) -> None:
    """Only called after the rail has recovered past the clear threshold."""
    spool.start_accepting()
    tiles.start_accepting()

Recording the event before making the data durable is deliberate and slightly counter-intuitive. The recorder write is a single page in a mapped file and costs about a millisecond; the spool sync costs twenty-five. If only one of them completes, the more valuable outcome is knowing a brownout happened — because that knowledge changes the diagnosis of everything else, while a few seconds of lost telemetry does not.

Constraint validation #

Gotchas and edge cases #

• Measure the window, do not assume it. Put the device on a bench supply, ramp it down at a realistic rate, and log the time between the application threshold and the point where writes start failing. That number is hardware-specific and it is the only input to the quiesce budget that matters.
• A brownout during a brownout is normal. A marginal supply produces repeated sags, and a quiesce routine that is not idempotent will misbehave on the second one. Make every step safe to repeat, and count the events rather than assuming one.
• Capacitance buys time, and it is the cheapest fix available. A bulk capacitor on the supply extends the window measurably. If the measured window is too short for the quiesce path, adding capacitance is usually easier than making the software faster.
• Do not attempt a clean OS shutdown. A systemctl poweroff takes seconds and will not complete. The quiesce path is application-level and deliberately does not involve the init system.
• The unclean-shutdown flag must be cleared on a clean start. Otherwise every subsequent boot reports a brownout, and the counter becomes meaningless. Clear it once the pipeline reaches steady state, not at boot.

What it looks like afterwards #

The point of all this is the next boot. A device that quiesced cleanly comes back with an intact spool, a consistent state file, and a recorder whose last entries say brownout_detected mv=10740. That single line changes the diagnosis of everything else on the device — a gap in telemetry becomes explained, an unclean shutdown becomes attributed, and a pattern of them across a site becomes a wiring or battery finding rather than a software mystery.

Export the brownout counter as a first-class metric alongside the power budget figures. On a solar installation a rising count is the earliest available sign that the panel or the battery is no longer meeting the load, and it appears weeks before the device starts failing to complete its duty cycle.

Testing it without a bench supply #

Not every team has a programmable supply, and the test is worth running regardless. A relay in series with the input, driven by a spare GPIO or a cheap USB relay board, gives a hard cut rather than a ramp — which tests the reset path but not the undefined band.

For the band itself, the cheapest realistic approximation is a series resistor switched in to drop the supply under load. A few ohms on a 12 V rail feeding a device drawing 300 mA produces a sag of roughly the right magnitude and, importantly, the right shape: the voltage falls as the load draws, exactly as a tired battery behaves.

Run the test a few dozen times under a realistic write load, and check three things after each: the spool reopens with its tail intact, the state file parses, and the recorder’s last entries name the brownout. Any failure is worth chasing before deployment, because the same failure in the field is a device that needs a visit.

One further habit is worth adopting: treat a brownout as a scheduling signal, not only as a storage hazard. A device that has detected two sags in an hour is running on a supply that cannot meet its peak load, and the correct response is to reduce that peak — lengthen the duty-cycle interval, defer the next transmission, skip the tile prefetch — rather than to keep operating identically and quiesce more often. The wake-process-sleep scheduler already has the mechanism for that; the brownout counter simply becomes another input to it, alongside the state of charge.

Related #

• Power & Duty Cycling — the energy budget a brownout says is no longer being met.
• Crash-safe append logs with fsync budgets — the sync interval that bounds this quiesce path.
• Black-box flight recorder logs for post-crash analysis — where the brownout event is recorded and read back.