Edge Geospatial IoT Gateway Processing

cgroup memory limits for spatial services

The kernel’s out-of-memory killer chooses its victim by a heuristic that has no idea which of your processes matters. On a gateway where a spatial worker and a telemetry daemon share 512 MB, that heuristic reliably picks the spatial worker’s parent, or the broker, or occasionally the thing that was about to write the diagnostic explaining why. A cgroup v2 memory limit replaces that lottery with a decision you made in advance. This guide configures it for the workloads described in device constraints and resource limits, inside the Core Edge GIS Fundamentals envelope.

What the three knobs actually do

cgroup v2 exposes three memory controls and they are frequently confused.

memory.max is a hard ceiling. A cgroup that would exceed it triggers reclaim, and if reclaim cannot free enough, the kernel OOM-kills a process inside that cgroup. Nothing outside is affected. This is the containment boundary.

memory.high is a throttle, not a limit. Crossing it puts the cgroup under aggressive reclaim and stalls its allocations, slowing it down rather than killing it. A process that crosses memory.high and then frees memory recovers with no fatality — which makes it the right place to catch a growing working set before it becomes a kill.

memory.min is a protection floor. Memory up to that figure is not reclaimed from the cgroup even under global pressure. It is what stops a large tile-cache read in one service from evicting the pages a latency-critical service needs.

Used together they produce a policy rather than a limit: the spatial worker is throttled at one figure, killed at a higher one, and the telemetry daemon’s working set is protected from both.

Three cgroup memory controls acting on one growing service A service's resident memory rising over time against three thresholds. Below memory.min at 40 megabytes its pages are protected from global reclaim. Passing memory.high at 180 megabytes it enters aggressive reclaim and its allocations are throttled, which slows it and gives a monitor time to act. Passing memory.max at 240 megabytes the kernel OOM-kills a process inside this cgroup only, leaving the telemetry daemon and the broker untouched. Without any of these, the global OOM killer would instead have chosen by heuristic across the whole system. A policy, not a limit: throttle here, kill there, protect this much 280 MB200 MB120 MB40 MB memory.max 240 MB — kill inside this cgroup memory.high 180 MB — throttle memory.min 40 MB — protected from reclaim reclaim + stall begins — the monitor has minutes here OOM inside this cgroup only Without a cgroup, the same growth kills whichever process the global heuristic dislikes most — frequently not this one.
The gap between the amber and red lines is the whole value: time in which a growing service is visibly slow rather than suddenly dead.

The unit file

On a systemd image the controls are properties on the service unit, which is the least error-prone place to put them because they survive restarts and are visible in the same place as everything else about the service.

# /etc/systemd/system/spatial-worker.service
[Unit]
Description=Spatial processing worker
After=network-online.target

[Service]
Type=notify
ExecStart=/usr/bin/python3 -m gateway.spatial
Restart=on-failure
RestartSec=5s

# --- memory policy ---------------------------------------------------
# Hard ceiling: the kernel kills inside this cgroup, never outside it.
MemoryMax=240M
# Throttle first: crossing this stalls allocations and triggers reclaim,
# which shows up as latency long before anything dies.
MemoryHigh=180M
# Never reclaim below this even under global pressure.
MemoryMin=40M
# Refuse swap entirely: on eMMC it converts a memory problem into a
# storage-wear problem and hides the symptom.
MemorySwapMax=0

# --- restart storm protection ---------------------------------------
# Five restarts in ten minutes means the fault is not transient.
StartLimitBurst=5
StartLimitIntervalSec=600

[Install]
WantedBy=multi-user.target

MemorySwapMax=0 deserves a note. On a device with swap on flash, an over-allocating service does not fail — it slows down enormously while writing swap pages to a card with a finite endurance budget. Disabling swap for the cgroup converts that into an honest, fast failure, which is almost always what a field device wants: a service that restarts in five seconds beats one that limps for an hour and wears the card out doing it.

Reading the pressure signals

The reason to set memory.high rather than only memory.max is that crossing it is observable. cgroup v2 exposes both an event counter and a pressure-stall metric, and both are worth exporting through the registry described in monitoring and observability.

# cgroup_probe.py — read a service's memory state from cgroup v2.
# Two small file reads; cheap enough for a 1 Hz sampler.
from pathlib import Path

BASE = Path("/sys/fs/cgroup/system.slice")


def memory_state(unit: str) -> dict:
    """Current usage, thresholds and pressure for one systemd unit."""
    cg = BASE / f"{unit}.service"
    out = {}
    for name in ("memory.current", "memory.high", "memory.max", "memory.peak"):
        p = cg / name
        if p.exists():
            raw = p.read_text().strip()
            out[name] = None if raw == "max" else int(raw)

    events = (cg / "memory.events").read_text().split()
    # memory.events is space-separated key/value pairs: low high max oom oom_kill
    out["events"] = {events[i]: int(events[i + 1]) for i in range(0, len(events), 2)}

    # memory.pressure: "some avg10=1.23 avg60=0.45 avg300=0.12 total=..."
    pressure = (cg / "memory.pressure").read_text().splitlines()
    for line in pressure:
        if line.startswith("some "):
            fields = dict(kv.split("=") for kv in line.split()[1:])
            out["stall_avg10_pct"] = float(fields["avg10"])
    return out


def headroom_fraction(state: dict) -> float | None:
    """How close the service is to its throttle threshold, 0.0–1.0+."""
    cur, high = state.get("memory.current"), state.get("memory.high")
    if not cur or not high:
        return None
    return cur / high

Three of those numbers are worth alerting on. events["high"] incrementing means the service crossed its throttle threshold — a leading indicator with no user-visible symptom yet. stall_avg10_pct rising above a few percent means the service is spending real time waiting on reclaim, which will show up as latency. And events["oom_kill"] being non-zero means the ceiling was reached, which by then is history rather than a warning.

Three cgroup signals ordered by how much warning each gives The memory.events high counter increments the moment the throttle threshold is crossed, typically minutes to hours before a kill, with no user-visible symptom. The pressure stall percentage rises as the service spends time in reclaim, typically seconds to minutes of warning, visible as latency. The oom_kill counter increments after the fact, giving no warning at all. A fourth row notes that without a cgroup none of these exist and the first signal is a process that is simply gone. Alert on the first, act on the second, post-mortem the third memory.events · high minutes to hours of warning no user-visible symptom yet — the best time to look memory.pressure · avg10 seconds to minutes already costing latency; shed load now memory.events · oom_kill no warning a record of what happened, not a chance to prevent it without a cgroup: none of these exist, and the first signal is a process that is simply gone — possibly not even this one
The point of the throttle threshold is that it manufactures a warning where the kernel's default behaviour provides none.

Constraint validation

Constraint Expected impact Mitigation
RAM ceiling An over-allocating service takes down unrelated ones MemoryMax confines the kill to the offending cgroup
Latency Reclaim stalls are invisible without instrumentation memory.pressure exported as a metric; the throttle threshold makes them observable
Flash Swapping on eMMC wears the card and hides the fault MemorySwapMax=0 turns it into a fast, honest failure
Restart storms A service that dies immediately on restart loops forever StartLimitBurst stops the loop and leaves the failure visible
Priority A tile read should not evict the telemetry daemon’s pages MemoryMin on the critical service protects its working set

Gotchas and edge cases

  • Sum the limits against physical memory, including the kernel. Four services each granted 240 MB on a 512 MB device have been granted twice the machine. cgroups do not enforce a global sum; they enforce per-cgroup ceilings, and over-committing them recreates exactly the global OOM situation the limits were meant to prevent.
  • MemoryMax counts page cache too. A service that memory-maps a large tile archive accumulates file-backed pages against its limit. Those are reclaimable, so the effect is usually reclaim rather than a kill — but a service whose limit is barely above its anonymous memory will spend its life reclaiming its own cache.
  • The kill target inside the cgroup is still a heuristic. For a multi-process service, the kernel picks which process to kill. If the parent is the wrong answer, set OOMPolicy= and the per-process oom_score_adj deliberately rather than hoping.
  • cgroup v1 and v2 do not mix. On a hybrid image, systemd may place the unit in a v1 hierarchy where these properties are named differently and behave differently. Check /sys/fs/cgroup/cgroup.controllers exists before trusting any of this.
  • Limits interact with the pool sizing. A process pool sized for four workers under a 240 MB ceiling gets 60 MB each including the interpreter, which is usually not enough. Derive the limit and the pool size together, as the async execution guide sets out.

Choosing the numbers

Start from a measured peak, not a guess. Run the service under production load for a full duty cycle, record memory.peak, and set MemoryHigh at about 1.3 times that figure and MemoryMax at about 1.6. That leaves room for a burst without making the ceiling so generous that it never fires.

Re-derive both after any change to the reference layer or the batch size, because both directly scale the working set. And treat a rising memory.peak across firmware versions as a defect worth investigating even when nothing has failed — the version that fits in 180 MB today is the version that meets an unusually large zone import next month.

Deciding what the ceiling protects

A memory limit is only useful if what it protects is stated. On a gateway, three things are usually worth protecting and they need different treatment.

The telemetry acquisition path must never be starved, because a fix not read from the UART is gone. Give it a MemoryMin covering its full working set and keep its cgroup separate from anything that can grow.

The spool writer must be able to write, because a spool that cannot accept a record loses it. It needs a modest MemoryMin and, more importantly, it must not share a cgroup with the spatial worker — a kill inside a shared cgroup can take the writer instead of the grower.

Everything else — tile serving, the local API, diagnostics — is genuinely expendable under pressure, and putting it in a cgroup with a low ceiling and no minimum makes it the thing that degrades first, which is the correct order.

That grouping is more valuable than any individual number. A device where every service shares one cgroup has a limit and no policy: the kill lands wherever the kernel’s heuristic points, which is the situation the limits were introduced to escape.

Three cgroups on a 512 MB gateway with their guarantees and ceilings The acquisition cgroup holds the GNSS reader and the spool writer, with a 48 megabyte protected minimum and a 96 megabyte ceiling. The spatial cgroup holds the worker pool, with no protected minimum, a 180 megabyte throttle and a 240 megabyte ceiling. The auxiliary cgroup holds tile serving, the local API and diagnostics, with no minimum and a 64 megabyte ceiling, so it is the first thing to degrade under pressure. The three ceilings sum to 400 megabytes, leaving 112 for the kernel and page cache. Grouping first, numbers second acquisition GNSS reader + spool writer min 48 MB · max 96 MB never starved: a fix not read is gone spatial worker pool high 180 MB · max 240 MB the one that grows; throttled before it is killed auxiliary tiles, local API, diagnostics max 64 MB, no minimum degrades first, by design 400 MB committed of 512, leaving headroom for the kernel and page cache — the sum is checked deliberately, not assumed.
The interesting decision is which services share a cgroup, because that is what decides who dies for whom.

Related in Device Constraints & Resource Limits