Rotating device credentials for MQTT fleets

A credential that never expires is a credential that will eventually be extracted from a device someone recovered from a scrapyard, and one that expires without a rotation path is a fleet-wide outage on a date somebody chose two years ago. This guide builds the rotation loop that avoids both, for devices that spend most of their life disconnected. It belongs to secure provisioning and OTA updates inside the Edge Operations & Observability guide.

What makes edge rotation different #

In a datacentre, rotation is a scheduled task against a service that is always reachable. On a field fleet three properties break that assumption.

Devices are offline for most of the window. A solar node that connects for four minutes a day has 0.3% of the rotation window available, and any scheme requiring a multi-step exchange has to survive being interrupted between steps.

Clocks drift. Certificate validity is a time-based construct and a gateway with a dead RTC battery has no idea what time it is. A device that cannot validate its own certificate’s expiry cannot decide when to rotate.

A failed rotation is unrecoverable remotely. If the device discards its old credential before the new one works, it has locked itself out of the only channel that could have fixed it. Every design decision below follows from that single sentence.

The rotation state machine #

# credential_rotation.py — interruptible, overlap-based credential rotation.
# The device never discards a working credential until a new one has
# authenticated successfully. Every state is persisted, so an interruption
# at any point resumes rather than restarts.
import enum
import json
import os
from pathlib import Path


class State(enum.Enum):
    STEADY = "steady"                 # one credential, working, not near expiry
    KEY_GENERATED = "key_generated"   # new key pair exists, no certificate yet
    CSR_SENT = "csr_sent"             # request submitted, awaiting issuance
    NEW_ISSUED = "new_issued"         # new certificate held, not yet proven
    NEW_PROVEN = "new_proven"         # new credential authenticated at least once


ROTATE_AT_FRACTION = 0.66             # begin at two thirds of the lifetime
MIN_OVERLAP_DAYS = 30


class Rotator:
    def __init__(self, store: Path, crypto, broker, clock):
        self.store, self.crypto, self.broker, self.clock = store, crypto, broker, clock
        self.state = self._load()

    def _load(self) -> dict:
        p = self.store / "rotation.json"
        if p.exists():
            return json.loads(p.read_text())
        return {"state": State.STEADY.value, "attempts": 0}

    def _save(self, **kw):
        self.state.update(kw)
        tmp = self.store / "rotation.json.tmp"
        tmp.write_text(json.dumps(self.state))
        os.replace(tmp, self.store / "rotation.json")     # atomic

    def tick(self) -> None:
        """Called on every connection opportunity. Advances one step at most,
        so an interruption costs one step rather than the whole rotation."""
        st = State(self.state["state"])

        if st is State.STEADY:
            if self._should_start():
                self.crypto.generate_new_keypair()        # private key never leaves
                self._save(state=State.KEY_GENERATED.value)
            return

        if st is State.KEY_GENERATED:
            csr = self.crypto.build_csr()
            if self.broker.submit_csr(csr):               # over the OLD credential
                self._save(state=State.CSR_SENT.value)
            return

        if st is State.CSR_SENT:
            cert = self.broker.poll_certificate()
            if cert is not None:
                self.crypto.store_pending_certificate(cert)
                self._save(state=State.NEW_ISSUED.value)
            return

        if st is State.NEW_ISSUED:
            # Prove the new credential on a real connection before trusting it.
            if self.broker.connect_with_pending():
                self._save(state=State.NEW_PROVEN.value, attempts=0)
            else:
                self._save(attempts=self.state["attempts"] + 1)
                if self.state["attempts"] >= 5:
                    self.crypto.discard_pending()          # bad issuance: start over
                    self._save(state=State.STEADY.value, attempts=0)
            return

        if st is State.NEW_PROVEN:
            # Only now is the old credential expendable.
            self.crypto.promote_pending()
            self.crypto.discard_old()
            self._save(state=State.STEADY.value)

    def _should_start(self) -> bool:
        remaining = self.crypto.current_cert_remaining_days(self.clock)
        if remaining is None:                              # unusable clock
            return self.state.get("forced", False)
        total = self.crypto.current_cert_total_days()
        return remaining <= max(total * (1 - ROTATE_AT_FRACTION), MIN_OVERLAP_DAYS)

The property that makes this deployable is that each tick advances at most one state and persists it. A device that connects for ninety seconds a day completes the rotation over five days, and an interruption at any point costs one step. Nothing requires the device to hold a connection through a multi-round exchange.

Constraint validation #

Gotchas and edge cases #

• The private key must never leave the device. Generate it on the device, send only the certificate signing request. A provisioning flow that ships key pairs from a central service creates a store whose compromise is fleet-wide and undetectable.
• Do not rotate all devices on the same date. Certificates issued in one provisioning batch share an expiry, so a fraction-of-lifetime trigger fires for all of them at once. Add per-device jitter of a few days, derived from the device id, so the CA and the broker see a spread rather than a spike.
• A device with no clock can still rotate — with help. When the RTC is dead the device cannot evaluate its own expiry. Let the platform signal “you should rotate” over the authenticated channel, and have the device accept that as a trigger. It is a weaker signal than the device’s own clock and it is much better than never rotating.
• connect_with_pending must be a real connection. Validating the certificate locally proves it parses, not that the broker will accept it. The difference is a wrong CA chain, a mismatched common name, or a policy the device knows nothing about — all of which pass local validation and fail at the broker.
• The old credential’s revocation is the platform’s job, not the device’s. The device discards its copy; the platform revokes it. A device that assumes revocation happened, and a platform that assumes the device discarded, produce a credential that is valid and unaccounted for.

Observability for rotation #

Three counters make a fleet’s credential state legible without querying every device: days-until-expiry as a gauge, current rotation state as an enumerated label, and failed-proof attempts as a counter. Collected fleet-wide they answer the only questions that matter — how many devices are approaching expiry without having started, how many are stuck mid-rotation, and whether any are failing to prove a newly issued credential.

The one to alert on is the first. A device that has not started rotating with thirty days left is a device that will lock itself out, and the alert has to fire early enough that someone can reach it — which on a fleet with quarterly site visits means the alert threshold is measured in months rather than days.

Planning the certificate lifetime #

The lifetime is the parameter everything else derives from, and it trades two risks against each other. A short lifetime limits how long a stolen credential is useful and forces the rotation path to be exercised frequently, which is how you find out it works. A long lifetime reduces the chance that a device which has been offline for an unusual length of time comes back to find itself expired.

For a fleet that connects daily, 90 days with a 30-day overlap is comfortable: rotation is exercised four times a year, and a device would have to be offline for a month to miss its window. For a fleet of remote nodes connecting monthly, 365 days with a 120-day overlap is more honest — the shorter option would spend its life mid-rotation.

Two numbers pin the choice. The maximum plausible offline period must be shorter than the overlap; take it from the fleet’s actual worst case over a year, not from the design intent. And the rotation success rate in the first cycle should be near total; if a meaningful fraction of devices need the platform’s forced trigger to rotate, the lifetime is short relative to how often those devices connect.

Whatever lifetime is chosen, exercise it before it matters. Force a rotation on a canary group a few weeks after provisioning rather than waiting months for the natural trigger — the failure modes are the same, and finding them on ten devices is preferable to finding them on the whole fleet on a date nobody remembered was coming.

One last operational habit: keep a small number of long-lived break-glass credentials, held offline, that authorise a technician-initiated re-enrolment over the serial console. They are not for routine use and they are not stored on any device. Their entire purpose is the case where the automatic path has failed on a device nobody can reach any other way, and having them costs nothing until the day it saves a recovery.

Related #

• Secure Provisioning & OTA Updates — the provisioning moment this lifecycle chains back to.
• Configuring MQTT QoS levels for telemetry drops — the broker connection these credentials authenticate.
• Signed tile and config bundles for field updates — the other half of the trust model, covering artefacts rather than identity.