Secure Provisioning & OTA Updates

Within the Edge Operations & Observability guide, this page covers the two moments a field device is most likely to be lost: the first time it is configured, and every time it is updated afterwards. A gateway that cannot be updated safely will not be updated, which means it will run the firmware it shipped with for its entire service life — and a gateway that can be updated unsafely is a fleet-wide outage waiting for one bad build.

Geospatial deployments raise the stakes in a specific way. The payload is not only code: it is basemap archives measured in hundreds of megabytes, zone definitions that change what the device reports, and coordinate transform grids whose absence silently degrades accuracy. Each of those is a different size, a different update cadence and a different failure mode, and treating them as one artefact is how a 300 MB tile archive ends up being pushed over a metered link because it shared a bundle with a 4 KB threshold change.

Constraint mapping #

The fifth row is the one that shapes the architecture. On a fleet where a site visit is cheap, an update system can be optimistic. On a fleet of solar-powered nodes on hillsides, the recovery path is more important than the update path, and every design decision should be made in that order.

Core concept 1: verify before apply, always #

The rule is simple and frequently broken: nothing is applied until it has been verified in full, on the device, against a key the device already trusted before the transfer began.

That means a detached signature over the artefact, checked after the whole artefact has landed and before anything is swapped. It means the public key ships in the firmware image, not alongside the artefact — an artefact carrying its own key proves only that it is internally consistent. And it means the verification happens on the device, not at a gateway, a proxy or a build server, because those are exactly the places an attacker who has reached the network would be.

# verify.py — detached-signature verification for an update artefact.
# Ed25519 via the platform's crypto library; the public key is baked into the
# image at build time and is not updatable except by a firmware update.
# Streaming hash: never loads the artefact into memory.
import hashlib
from pathlib import Path

CHUNK = 1 << 16


def artefact_digest(path: Path) -> bytes:
    """SHA-256 over the file, read in chunks so a 3 GB archive costs 64 KB."""
    h = hashlib.sha256()
    with open(path, "rb") as fh:
        while True:
            chunk = fh.read(CHUNK)
            if not chunk:
                break
            h.update(chunk)
    return h.digest()


def verify_artefact(path: Path, sig_path: Path, public_key, expect_size: int,
                    expect_kind: str, manifest_kind: str) -> bool:
    """Every check that must pass before an artefact is allowed near the
    live system. Order matters: cheap checks first, signature last."""
    if not path.exists() or not sig_path.exists():
        return False
    if path.stat().st_size != expect_size:
        return False                       # truncated transfer
    if expect_kind != manifest_kind:
        return False                       # a basemap delivered as firmware
    digest = artefact_digest(path)
    try:
        public_key.verify(sig_path.read_bytes(), digest)
    except Exception:                      # any verification failure is fatal
        return False
    return True

The expect_kind check deserves a note because it catches a class of mistake no signature can. A correctly signed artefact of the wrong type — a zone file delivered where the device expected a threshold file — passes every cryptographic check and then corrupts the running configuration. Bind the kind into the manifest, check it, and refuse the mismatch.

Core concept 2: apply atomically, with a way back #

An update that can be half-applied will eventually be half-applied, and the device that results is the one that costs a site visit. Three patterns cover the artefact classes above.

A/B partitions for firmware: the update is written to the inactive partition, verified there, and the bootloader’s active flag is flipped as a single write. A failure at any earlier point leaves the running partition untouched. The walkthrough, including the boot-success confirmation that prevents a boot loop, is in A/B partition rollback for geospatial pipelines.

Write-beside-and-rename for files: the new artefact is written alongside the old, fsynced, and renamed over it, with the directory synced afterwards. This is the mechanism used for tile archives and index files throughout this site, and it is atomic on any POSIX filesystem.

Versioned side-by-side for data the running process holds open: the new version is written under its own name, the process is told to load it, and the old version is deleted only after the process confirms the new one is live. This is what zone definitions need, because the running pipeline has the old one mapped and cannot have it disappear underneath.

Core concept 3: provisioning is the root of everything #

The identity a device uses for the rest of its life is established once, usually on a bench, and every later security property depends on that moment being done right. Three things have to happen and none of them can be retrofitted.

The device generates its own key pair and never transmits the private half. A provisioning system that generates keys centrally and installs them has created a database that compromises every device it ever touched.

The device receives the fleet’s trusted public keys — for update signing, for the platform’s identity — in the firmware image or in a write-once region. These are the trust anchors, and their whole value comes from having arrived before the device was ever on a network.

The device’s certificate or registration is signed by an authority that verified the device’s own public key, at a moment when the device was physically controlled. Everything afterwards — credential rotation, re-enrolment, replacement — chains back to that.

The credential lifecycle from that point is covered in rotating device credentials for MQTT fleets, and the mechanics of signing the artefacts themselves in signed tile and config bundles for field updates.

Operational considerations #

Stage every rollout. A fleet-wide push is a fleet-wide outage with extra steps: send to one device, then ten, then a hundred, with a soak period between rings long enough for the failure mode you are worried about to appear. For a firmware change that is at least one full duty cycle; for a zone change it may be an hour.

Make the device report what it is running, always. Firmware version, config version, zone-layer version, basemap version, on every health response and in the telemetry stream. Half of all update incidents are resolved by discovering that the devices behaving oddly are the ones that did not take the update, and that discovery is free if the versions are already flowing.

Never let an update require a reboot to be visible. A device that has applied a config change but will not act on it until its next restart is a device in an ambiguous state, and ambiguity across a fleet of hundreds is not recoverable by inspection.

Delivering to devices that never see a network #

A meaningful fraction of geospatial deployments include nodes that have no practical link for large artefacts: a sensor in a valley with only a satellite messaging channel, a gateway on a vessel, a node whose data plan cannot carry a basemap in a decade. Those devices still need updates, and pretending otherwise produces a fleet with two software versions and no plan to converge them.

Physical delivery is the answer, and it works better than its reputation suggests provided the mechanism is the same one the network path uses. A technician arrives with a USB stick carrying the identical signed bundle; the device verifies it with the identical routine, against the identical key, and applies it through the identical path. Nothing about the update logic knows or cares where the bytes came from, which means the physical path is exercised by every network update and vice versa.

Two rules keep it safe. The device must never trust a medium — a USB stick is exactly as untrusted as a cellular link, and the signature check is what makes either acceptable. And the device must record the delivery channel alongside the applied version, because a fleet where half the devices were updated by hand needs to know which half.

The same mechanism covers the recovery case. A device that has somehow ended up with no working credential, no valid firmware and no link is recoverable by a technician with a stick and a serial console, provided the bundle format and the verification path do not assume a network. Designing for that from the start costs nothing; retrofitting it after the first stranded device costs a redesign.

Failure modes and recovery #

Staging a rollout that can be stopped #

The update mechanism decides whether a bad build is recoverable; the rollout process decides how many devices meet it first. Four rings, with a stop condition between each, is the shape that works on a field fleet.

Ring 0 — the bench. One device of each hardware variant, physically present, running the production workload. It catches the build that does not boot and the one whose new dependency does not exist on the target architecture.

Ring 1 — the canaries. Five to ten devices in the field, chosen to span the deployment’s variety: the oldest hardware revision, the worst link, the coldest site, the busiest route. Soak for at least one full duty cycle, which for a solar node means a full day-night cycle rather than an hour.

Ring 2 — a tenth of the fleet. Long enough to see anything rate-dependent: a memory leak that takes six hours, a flash write pattern that only shows up after a week.

Ring 3 — everyone. By this point the only failures left are the ones that need scale to appear, which are usually platform-side rather than device-side.

The stop condition between rings has to be automatic, because a manual gate is a gate someone will be too busy to close. Watch three signals per ring: the fraction of devices reporting the new version within the expected window, the revert count, and the change in error rate against the ring’s own pre-update baseline. Any of them crossing a threshold pauses the rollout without anybody deciding to.

What makes this work on a disconnected fleet is patience about the first signal. A device that has not reported the new version may be broken or may simply be asleep, and treating a duty-cycled node’s silence as a failure will halt every rollout you ever attempt. Set the window from the fleet’s actual connection cadence — for a node that reports daily, the window is days.

What to record about every update #

An update system is only as diagnosable as the record it leaves, and the record costs a few dozen bytes per attempt. Six fields cover every question that gets asked afterwards.

The bundle or image identifier, so an incident can be tied to a specific build rather than to a date. The outcome — applied, rejected, reverted — as an enumerated value rather than free text. The reason on anything other than success, drawn from a fixed set so it can be counted across a fleet. The channel the artefact arrived by. The duration from first byte to applied, which is the number that tells you whether the rollout window was realistic. And the versions in effect afterwards, all of them, so a device’s complete state is reconstructable from its last update record without querying it.

Spool that record like any other event, with the never-drop policy. An update record lost to a full partition is exactly the record someone will want, because the devices whose spools are full are disproportionately the devices something went wrong on.

Fleet-wide, those records answer the questions a rollout raises in the order they get asked: how many devices took it, how many rejected it and why, how many reverted, and how long the whole thing took from first push to last device. None of that requires a separate telemetry system — it is six fields on an event the device already knows how to deliver.

The same record is what makes a fleet’s update coverage measurable rather than assumed. Query for devices whose last successful update predates the current build and the answer is the list of devices that need attention — not an inference from an absence of failure reports. On a fleet of several hundred, that list is almost never empty, and the devices on it are usually the ones whose links are worst, which is to say the ones a manual process would also have missed. Reviewing that list monthly, and reaching the devices on it deliberately, is the difference between a fleet with one software version and a fleet with four nobody chose.

Related #

• Signed tile and config bundles for field updates — the bundle format and its verification path.
• A/B partition rollback for geospatial pipelines — surviving a bad firmware image without a site visit.
• Rotating device credentials for MQTT fleets — the credential lifecycle after provisioning.
• Field Diagnostics & Recovery — what a technician can do when an update goes wrong anyway.
• Offline Tile & Basemap Storage — the largest artefact this system has to deliver.