Debouncing geofence crossings across restarts

A geofence state machine that lives only in memory is correct until the first watchdog reset, at which point it wakes up with no idea whether the asset was inside or outside, and either invents a crossing or misses one. On a fleet that restarts several times a week — a firmware update, a brownout, a stalled pipeline recovered by the watchdog — that is not an edge case, it is a recurring source of duplicate and missing events. This guide persists the crossing state and defines what to do on the way back up, inside threshold-based event mapping and the Local Spatial Processing Patterns guide.

What restart does to a crossing detector #

The detector’s job is to emit an event when the asset’s membership of a zone changes. Membership is a state, and a restart destroys it. Three behaviours are possible on the first fix after a restart and all three are wrong in some deployment.

Assume outside. The device treats every zone as unentered. An asset that was inside produces an ENTRY event it already produced before the restart — a duplicate.

Assume the current position’s membership, silently. The device evaluates membership and adopts it with no event. An asset that genuinely left during the restart — a vehicle that drove out while the device was rebooting — produces no EXIT, and the zone’s occupancy record is wrong until the next entry.

Emit nothing and refuse to decide. The device waits for a transition, which never comes if the asset stays put, so the zone’s state is unknown indefinitely.

The workable answer is none of these on their own: persist the state, restore it, and emit a reconciliation record describing what the device believes and why, so the ambiguity is upstream rather than hidden.

Persisting the state #

The state is small — one byte of membership per zone plus a timestamp — and the constraint is that it must survive a power cut, which means the same crash-safe discipline as the spool.

# fence_state.py — durable geofence membership across restarts.
# One fixed-size record per zone, written as a whole file with the
# fsync-then-rename pattern. At a few dozen zones the file is under a kilobyte,
# so a full rewrite per change is cheaper than any incremental scheme.
import json
import os
import time
from pathlib import Path


class FenceState:
    """Membership per zone id, plus the monotonic and wall-clock time it was
    last confirmed. The pair matters: monotonic orders events across a restart
    only if the boot id also matches, so both are recorded."""

    __slots__ = ("path", "boot_id", "state", "dirty")

    def __init__(self, path: Path, boot_id: str):
        self.path = path
        self.boot_id = boot_id
        self.state = self._load()
        self.dirty = False

    def _load(self) -> dict:
        if not self.path.exists():
            return {"boot_id": None, "zones": {}, "wall_utc": None}
        try:
            return json.loads(self.path.read_text())
        except (ValueError, OSError):
            # A corrupt file is treated as absent: better to reconcile than
            # to act on membership we cannot parse.
            return {"boot_id": None, "zones": {}, "wall_utc": None}

    def membership(self, zone_id: str) -> bool | None:
        """True, False, or None when this zone has never been evaluated."""
        return self.state["zones"].get(zone_id)

    def set(self, zone_id: str, inside: bool) -> None:
        if self.state["zones"].get(zone_id) is not inside:
            self.state["zones"][zone_id] = inside
            self.dirty = True

    def flush(self) -> None:
        """Called after a batch of updates, not per zone. A full rewrite of a
        sub-kilobyte file is one page; doing it per zone would be dozens."""
        if not self.dirty:
            return
        self.state["boot_id"] = self.boot_id
        self.state["wall_utc"] = time.time()
        tmp = self.path.with_suffix(".tmp")
        tmp.write_text(json.dumps(self.state))
        with open(tmp, "rb") as fh:
            os.fsync(fh.fileno())
        os.replace(tmp, self.path)
        dir_fd = os.open(self.path.parent, os.O_DIRECTORY)
        try:
            os.fsync(dir_fd)                 # the rename itself must be durable
        finally:
            os.close(dir_fd)
        self.dirty = False

    def is_stale(self, max_age_s: float = 3600.0) -> bool:
        """State older than this is not trustworthy as a membership claim —
        the asset could have gone anywhere while the device was off."""
        w = self.state.get("wall_utc")
        return w is None or (time.time() - w) > max_age_s

Reconciling on the way back up #

The first fix after a restart is where the decision is made, and the rule depends on how long the device was down.

def reconcile(state: FenceState, zone_ids, evaluate, now_s, emit):
    """Compare persisted membership against current reality, once, at boot.

    `evaluate(zone_id) -> bool` runs the exact containment test.
    `emit(kind, zone_id, **meta)` publishes an event.
    """
    stale = state.is_stale()
    for zone_id in zone_ids:
        was = state.membership(zone_id)
        now = evaluate(zone_id)

        if was is None:
            # Never evaluated: adopt silently, but say so.
            emit("fence_initialised", zone_id, inside=now)
        elif was == now:
            pass                              # nothing changed across the gap
        elif stale:
            # Too long down to claim a crossing time. Emit the transition with
            # an explicit uncertainty window rather than a precise lie.
            emit("fence_transition_uncertain", zone_id, inside=now,
                 window_start_s=state.state["wall_utc"], window_end_s=now_s)
        else:
            # Short restart: the crossing almost certainly happened during it.
            emit("fence_entry" if now else "fence_exit", zone_id,
                 at_s=now_s, inferred=True)
        state.set(zone_id, now)
    state.flush()

The distinction between a short and a stale restart is the part worth defending. A ninety-second reboot leaves almost no room for a crossing to be missed, so inferring one with the restart’s timestamp is defensible. A device that was off for six hours cannot claim anything about when a crossing happened, and an event carrying a precise but fabricated timestamp is worse than one carrying an honest window.

Constraint validation #

Gotchas and edge cases #

• Monotonic time does not survive a reboot. Comparing a monotonic timestamp from before the restart with one after is meaningless — the clock restarted too. That is what the boot id is for: two timestamps are comparable only when their boot ids match.
• A zone-layer update invalidates persisted membership. If zone 17 in the new layer is a different shape from zone 17 in the old one, the persisted “inside zone 17” claim is about a different polygon. Version the state file against the layer version and reconcile from scratch when they differ.
• Do not write state inside the fix handler. The flush involves two fsync calls and can take tens of milliseconds. Batch the updates and flush from the maintenance task, accepting that a power cut may lose the last few seconds of transitions — which reconciliation will then correct.
• Duplicate suppression belongs upstream too. Even with perfect device-side state, a re-sent batch after a lost acknowledgement will deliver the same crossing twice. The idempotency key described in store-and-forward buffering is what makes that harmless.
• Reconciliation must run before the first ordinary evaluation. If the normal crossing logic runs first, it compares against the in-memory default and emits before reconciliation has a chance to. Order the boot sequence explicitly and assert it.

Verifying it #

Test this the way the A/B rollback revert path is tested — deliberately, on a bench. Replay a recorded track through the pipeline, killing the process at scripted points, and assert the emitted event sequence matches the uninterrupted run exactly except for the flags. Any difference is a bug in the reconciliation, and it is far easier to find here than in a monthly report where one vehicle has two entries and no exit.

Export three counters as well: inferred transitions, uncertain transitions and initialisations. All three should be rare, and each of them rising has a specific meaning — frequent restarts, long outages, or a state file that is not persisting at all.

Cost of getting it wrong #

It is worth quantifying, because the effort to persist state always competes with something else. A fleet of 200 devices restarting an average of twice a week, each carrying eight zones with an average membership of one, produces roughly 200 spurious ENTRY events a week under the assume-outside strategy — about 10 000 a year, distributed unevenly and concentrated on the devices that restart most, which are the ones with the worst hardware.

Those events are indistinguishable from real ones at the platform. They inflate visit counts, they trigger arrival notifications for vehicles that never left, and they are discovered when somebody compares a report against a driver’s account. The remedy at that point is a data-cleaning exercise across a year of history with no reliable way to identify which entries were spurious.

Persisting the state costs one small file, two fsync calls per transition and about eighty lines. The comparison is not close.

Related #

• Threshold-Based Event Mapping — the crossing logic whose state this preserves.
• Configuring spatial thresholds for sensor event triggers — the hysteresis and dwell parameters that define a crossing.
• Field Diagnostics & Recovery — the restart sources this has to survive.