Edge Geospatial IoT Gateway Processing

A/B partition rollback for geospatial pipelines

A firmware update that leaves a gateway unbootable costs a site visit; one that leaves it booting but broken costs a site visit and a day of confusion first. A/B partitioning solves the first problem completely and the second one only if the definition of “booted successfully” is chosen carefully — which, on a device whose job is spatial processing, means a great deal more than “the kernel started”. This guide covers both halves, inside secure provisioning and OTA updates and the Edge Operations & Observability guide.

The scheme

Two identical rootfs partitions, A and B. One is active; the other is the update target. The bootloader holds a small, separately stored state block naming the active slot, a boot-attempt counter and a “confirmed” flag.

An update writes the inactive slot, verifies it, sets the active slot to that partition, clears the confirmed flag and sets the attempt counter to zero. On the next boot the bootloader increments the attempt counter and boots the named slot. If the running system confirms itself, the flag is set and the counter no longer matters. If the attempt counter reaches its limit without a confirmation, the bootloader reverts to the other slot permanently.

The result is that any failure — a bad image, a kernel panic, a service that will not start, a device that reboots in a loop — resolves automatically within a bounded number of attempts, without anyone visiting the site.

Boot state machine across a successful update and a failed one From a confirmed state on slot A, an update writes and verifies slot B, sets the active slot to B, clears the confirmed flag and zeroes the attempt counter. On boot the counter increments and slot B runs. In the successful path the pipeline reaches a healthy state, calls confirm, and the state becomes confirmed on B. In the failed path the confirmation never arrives; after three attempts the bootloader sets the active slot back to A and marks it confirmed, and the device returns to the previous known-good image without intervention. Every failure path ends at a bootable device A · confirmed steady state write + verify B A still running active=B, unconfirmed attempts = 0 reboot pipeline healthy confirm() called B · confirmed no confirm attempts → 3 A · confirmed reverted The bootloader's state block is a few dozen bytes and is the only thing that must survive a power cut mid-update.
The unconfirmed state is the whole mechanism. A device that cannot reach the confirmation is a device that reverts itself.

What “confirmed” has to mean

The default confirmation — set the flag from a systemd unit that runs after boot — proves the kernel started and almost nothing else. For a geospatial gateway it is worth considerably more to confirm against the pipeline’s actual function, because the failure modes that matter are ones a booting kernel does not notice.

Four checks make a defensible confirmation on this class of device:

A fix has been acquired and parsed. Not “the receiver is enumerated” — an actual position through the actual parser. A firmware change that broke the NMEA path leaves a device that boots perfectly and produces nothing.

A spatial predicate has been evaluated. Load the zone layer, run one containment test against a known point, compare against a known answer. This exercises the index, the geometry library and the coordinate transform in one call, and a mismatch means the update changed a result it should not have.

The spool accepted and drained a record. Proves the storage path and the uplink both work end to end.

Resident memory is inside its ceiling after the pipeline reaches steady state. A build whose dependencies grew by 60 MB boots fine and dies on the first busy hour; catching it at confirmation time reverts before that hour arrives.

# confirm.py — application-level boot confirmation for a spatial gateway.
# Called once, after the pipeline reports steady state or after a timeout.
# Confirms only when every check passes; otherwise it stays silent and lets
# the bootloader's attempt counter do its work.
import time

CONFIRM_DEADLINE_S = 300           # must confirm inside 5 minutes of boot
KNOWN_POINT = (-122.3321, 47.6062)
KNOWN_ZONE_ID = 17


def boot_self_check(pipeline, spool, uplink, limits) -> tuple[bool, str]:
    if not pipeline.wait_for_fix(timeout_s=180):
        return False, "no GNSS fix parsed"

    zones = pipeline.zone_index
    if zones is None:
        return False, "zone index not loaded"
    hits = pipeline.zones_containing(*KNOWN_POINT)
    if KNOWN_ZONE_ID not in hits:
        return False, f"containment regression: expected {KNOWN_ZONE_ID}, got {hits}"

    probe_id = spool.append_probe()
    if not uplink.wait_for_ack(probe_id, timeout_s=90):
        return False, "spool probe not acknowledged"

    rss = pipeline.resident_bytes()
    if rss > limits.rss_ceiling:
        return False, f"rss {rss} over ceiling {limits.rss_ceiling}"

    return True, "ok"


def run_confirmation(bootloader, pipeline, spool, uplink, limits, log):
    started = time.monotonic()
    ok, reason = boot_self_check(pipeline, spool, uplink, limits)
    elapsed = time.monotonic() - started
    if ok and elapsed < CONFIRM_DEADLINE_S:
        bootloader.confirm()
        log.info("boot confirmed in %.1fs", elapsed)
        return True
    # Do NOT confirm. Record why, and let the attempt counter revert us.
    log.error("boot NOT confirmed after %.1fs: %s", elapsed, reason)
    spool.append_event("boot_unconfirmed", {"reason": reason, "elapsed_s": elapsed})
    return False

The last three lines are the part that turns an automatic revert into a diagnosable one. The device will reboot into the old image and the reason will be sitting in the spool, ready to sync — which means the failure is attributable without anyone reproducing it.

Constraint validation

Constraint Expected impact Mitigation built into the design
Flash capacity Two rootfs partitions double the firmware footprint Data partitions are shared, not duplicated; only the OS and application are doubled
Power loss A cut during the write must not brick the device The inactive slot is the only thing written; the state block flip is a single small write
Boot loop A bad image could reboot forever Bounded attempt counter reverts permanently after three tries
Link A revert must be visible upstream The unconfirmed reason is spooled before the reboot and syncs from the reverted image
Time Confirmation must not wait indefinitely Hard 5-minute deadline; a slow confirmation is treated as a failed one

Gotchas and edge cases

  • Shared data partitions are not versioned. The spool, the tile archive and the zone layer live outside A and B, so a revert does not undo changes the new firmware made to them. If a firmware update migrates a data format, the old image must still be able to read the migrated data — or the migration must be deferred until the update is confirmed.
  • Confirmation must be idempotent. A device that reboots after confirming should not re-run the check and revert on a transient failure. Check the confirmed flag first, and skip the whole routine when it is already set.
  • The bootloader state block needs its own durability. It is small and it is written at exactly the moment a power cut is most likely. Use a redundant pair with a sequence number and a checksum, so a torn write falls back to the previous copy rather than to undefined behaviour.
  • A confirmation that depends on connectivity can fail for the wrong reason. The spool-and-ack check above will fail on a device that is legitimately offline, reverting a perfectly good image. Make that check conditional: require it when the link is up, skip it when the device has been offline since boot, and record which variant ran.
  • Reverting does not stop the rollout. Unless the fleet management side notices, the same bad image will be offered again on the next cycle. Report the revert, and have the platform quarantine a build after a threshold of reverts across the ring.
What each confirmation depth actually proves Four confirmation strategies with their coverage. Confirming from the bootloader on kernel start proves only that the image boots. Confirming from a systemd unit adds that the service started. Confirming after the pipeline reports steady state adds that configuration loaded and threads are running. Confirming after a functional self-check — a parsed fix, a known containment result, an acknowledged spool probe and an RSS check — additionally catches a broken receiver path, a geometry regression, a broken uplink and a memory regression, which are the failures that actually reach production. Depth of confirmation against the failures it catches kernel started catches: an unbootable image misses everything a booting kernel does not check service started + a crash on start-up misses a service that runs and does nothing pipeline steady + config loaded, threads running misses a wrong answer functional self-check + receiver path, geometry result, uplink, RSS ceiling the failures that reach production
The first three rows are cheap and mostly redundant. The fourth is the only one that would have caught the last real incident.

Sizing the attempt counter

Three attempts is the usual figure and it is worth deriving rather than copying. The counter has to be large enough to tolerate a transient failure — a receiver that takes two cold starts to acquire, a link that was down for the first boot’s probe — and small enough that a genuinely broken image does not keep the device out of service for long.

Multiply the boot time plus the confirmation deadline by the attempt count to get the worst-case outage from a bad update. At a 40-second boot and a 300-second deadline, three attempts is about seventeen minutes of downtime before the revert lands, which is acceptable for almost any fleet. Five attempts is nearly half an hour, which usually is not, and one attempt reverts on the first unlucky boot, which produces a fleet that never successfully updates.

Record the attempt count that was reached even on a successful confirmation. A build that consistently confirms on the second attempt rather than the first is telling you something — usually that the confirmation’s timeout is too tight for a cold receiver — long before it starts failing outright.

Testing the revert path deliberately

The revert is the feature, and it is the one part of the system that only runs when something has already gone wrong — which means it is the part least likely to have been exercised. Test it on purpose, on the bench, before the fleet needs it.

Build an image that fails its own confirmation deliberately: one that returns a wrong answer from the containment check, or one whose confirmation call is removed entirely. Deploy it to a bench device and watch the full cycle — three boots, the attempt counter reaching its limit, the bootloader reverting, and the device coming back on the previous image with the unconfirmed reason waiting in the spool.

Then repeat it with power cuts injected at each stage: during the write to the inactive slot, immediately after the state-block flip, and midway through the first boot of the new image. Each should resolve to a bootable device on one slot or the other, and none should produce a device that needs a serial console to recover.

That exercise takes an afternoon and it is the only evidence anyone will ever have that the mechanism works. The alternative is discovering the state block’s write is not atomic on the day a hundred devices take a bad build, which is the scenario the whole scheme exists to prevent.

The bootloader state block and its redundant pair The state block holds an active slot flag, a boot attempt counter, a confirmed flag and a sequence number, protected by a checksum, in 24 bytes. Two copies are held; the bootloader reads both, checks their checksums and uses whichever has the higher sequence number among the valid ones. A write updates the older copy first, so a power cut during the write always leaves at least one valid copy — and the one that survives is either the previous state or the new one, never a mixture. 24 bytes, written twice, never both at once active slot · 1 B attempts · 1 B confirmed · 1 B sequence · 4 B crc32 · 4 B copy A · seq 41 · crc ok the older copy — overwritten next copy B · seq 42 · crc ok highest valid sequence wins — this is the live state A power cut mid-write corrupts at most one copy, and the other is a complete, consistent state. Without the pair, the one moment most likely to lose power is the one moment the device cannot afford an invalid state block.
The redundant pair is four extra lines in the bootloader and it is the difference between an automatic revert and a device that will not boot at all.

Related in Secure Provisioning & OTA Updates