Queue-Depth Alerting Thresholds for Edge Sync

This page solves one concrete problem: choosing high-water and low-water alert thresholds for a store-and-forward sync queue on a constrained Linux gateway so that a genuine backlog gets flagged early, without the alert flapping every time normal jitter pushes the depth up and down. Within the Edge Operations & Observability guide, and as a companion to monitoring and observability, this is the alerting layer that sits directly on top of the queue itself — it does not move data, it decides when the depth of that queue means something a technician should know about.

The queue in question is the one described in message queue management at the edge: a local, persistent buffer that absorbs telemetry while a cellular or satellite uplink is unavailable, then drains it once the link returns. Depth in that queue is a completely normal, expected quantity that rises and falls with every outage — which is exactly what makes it a hard alerting target. A threshold tuned for “the queue is growing” without care produces a page every time the link drops for five minutes, which is not an incident; it is the system working as designed.

Threshold selection rationale #

A single fixed threshold on absolute queue depth is the wrong primitive for this signal, for the same reason a single thermal trip point is wrong: normal operation legitimately crosses any fixed line during a routine outage. Three refinements make the alert mean something.

High-water and low-water marks, not one line. Depth has to cross a high mark before an alert is armed, and fall back below a separate, lower low-water mark before the alert is allowed to clear. The gap between the two absorbs a queue that is oscillating right around a single threshold during a marginal link — exactly the flapping problem that hysteresis solves everywhere else in this guide.

Rate-of-change matters as much as the absolute level. A queue at 40% capacity that has been flat for an hour is a non-event; a queue at 15% capacity that doubled in the last five minutes is an early warning that the uplink just degraded and the backlog is about to get much worse. Alerting on the derivative catches the second case well before the absolute high-water mark would ever trip.

Depth has to be read against capacity, not a raw count. A count of 500 pending items means something different on a queue sized for 5,000 than on one sized for 600 — the alert logic below always works in fractions of the configured capacity, so the same thresholds are meaningful whether the sizing constant changes on a firmware revision or across a fleet with different flash budgets.

import time
from collections import deque

HIGH_WATER = 0.75      # arm the alert once depth exceeds 75% of capacity
LOW_WATER = 0.40        # depth must fall back below 40% before the alert clears
MIN_DWELL_S = 30.0      # sustained duration required before either transition is honored
RATE_WINDOW_S = 120.0   # look-back window for the rate-of-change check
RATE_TRIP = 0.30        # alert if depth fraction rises more than 30 percentage points in the window

class QueueDepthAlerter:
    """High/low water hysteresis plus a rate-of-change check on fractional
    queue depth. History is a bounded deque of (timestamp, fraction) pairs,
    trimmed to RATE_WINDOW_S on every sample — never allowed to grow with
    total uptime.
    """

    def __init__(self, capacity: int):
        self.capacity = capacity
        self._alarmed = False
        self._since = None       # monotonic time the current crossing began
        self._history = deque()  # (ts, fraction), oldest first

    def _trim_history(self, now):
        while self._history and now - self._history[0][0] > RATE_WINDOW_S:
            self._history.popleft()

    def _rate_trip(self, now, fraction):
        self._history.append((now, fraction))
        self._trim_history(now)
        if not self._history:
            return False
        oldest_fraction = self._history[0][1]
        return (fraction - oldest_fraction) >= RATE_TRIP

    def sample(self, depth: int) -> bool:
        """Call once per monitoring cycle with the current queue depth.
        Returns the (possibly updated) alarm state."""
        now = time.monotonic()
        fraction = depth / self.capacity

        rate_alert = self._rate_trip(now, fraction)
        level_crossed = fraction >= HIGH_WATER if not self._alarmed else fraction > LOW_WATER

        if level_crossed or rate_alert:
            if self._since is None:
                self._since = now
            elif now - self._since >= MIN_DWELL_S:
                self._alarmed = True
        else:
            self._since = None
            if self._alarmed and fraction <= LOW_WATER:
                self._alarmed = False

        return self._alarmed

The rate check and the level check are deliberately independent triggers that both feed the same dwell gate: either a sustained high fraction or a sustained fast rise is enough to arm the alert, but neither one fires instantly — MIN_DWELL_S requires the condition to hold for a real interval before the alarm state actually flips, which keeps a single noisy sample from paging anyone.

Constraint validation table #

Gotchas and edge cases #

• Capacity itself must be a real number, not a guess. HIGH_WATER and LOW_WATER are fractions of capacity, so if the queue’s configured capacity does not match its actual sizing (a flash partition smaller than assumed, a maxsize set differently in code than in the deployment config), the alert thresholds silently mean something other than what was tuned. Read capacity from the same configuration the queue itself enforces, never a hard-coded duplicate.
• Backpressure and alerting are related but not the same decision. A full queue triggering backpressure — producers blocking or telemetry being shed — is a real-time control decision that has to happen immediately; the alert armed by this page is a notification decision that should stay gated by dwell time even when backpressure has already kicked in. Do not wire the alert’s dwell gate into the backpressure trigger itself, or a real backlog will be allowed to grow unchecked while the alert waits out its dwell window.
• A queue draining faster than it fills can mask a real problem. If the low-water mark is set too close to the high-water mark, a brief successful drain clears the alert seconds before the backlog resumes growing, producing a rapid arm/clear/arm cycle that is worse than either a stable alarm or none at all. Keep real separation — the 75%/40% split above, not 75%/70%.
• Rate-of-change thresholds need re-tuning per link profile. A gateway on a stable LTE-M connection and one on a marginal satellite link have very different normal fill/drain cycles; a RATE_TRIP tuned against one will either never fire or fire constantly on the other. Derive it from a week of logged depth history on the actual link, not a value copied from another deployment.
• Clock source matters for the dwell and rate windows. Both rely on time.monotonic(), deliberately, because a gateway with no real-time-clock battery loses wall-clock time on every power cut; using time.time() here would make the dwell and rate windows meaningless across a reboot mid-outage.

Integrating with the gateway pipeline #

Wire the alerter into the same loop that already tracks the store-and-forward queue’s state, sampling on the same cadence as the rest of the monitoring surface rather than on every enqueue or dequeue:

import asyncio

alerter = QueueDepthAlerter(capacity=QUEUE_LIMIT)

async def queue_alert_loop(get_depth, on_alarm_change):
    prior = False
    while True:
        alarmed = alerter.sample(get_depth())
        if alarmed != prior:
            on_alarm_change(alarmed)
            prior = alarmed
        await asyncio.sleep(5.0)

def notify(alarmed: bool):
    GATEWAY_QUEUE_DEPTH_ALARM.set(1.0 if alarmed else 0.0)  # exported gauge, not a fresh read
    if alarmed:
        log_and_queue_health_event("sync_queue_backlog")

Export the alarm state itself as a gauge alongside the raw depth — a 0/1 value a scraper or dashboard can graph over time — rather than only emitting a one-shot log line, so a technician reviewing history after the fact can see exactly how long the backlog condition actually persisted.

The same high/low-water pattern generalizes beyond the sync queue itself. Any bounded buffer on the gateway — an ingestion queue awaiting a worker pool, a log-spool directory awaiting compression, a retry backlog awaiting a network window — faces the identical trade-off between reacting to a real backlog and flapping on routine variation. Building the alerter as a small, dependency-free class parameterized on capacity rather than hard-coded to one specific queue object means the same logic can be instantiated once per buffer that matters, each with thresholds tuned to that buffer’s own normal operating range, without duplicating the hysteresis and rate-of-change logic per call site.

Related #

• Monitoring & Observability for Spatial Gateways — the shared cadence and hysteresis discipline this alerter follows.
• Message Queue Management at the Edge — the store-and-forward queue this alerting layer watches.
• Prometheus Edge Exporter for Gateway Metrics — exposing the alarm state as a scraped gauge alongside raw depth.