Bounding Process Pool Memory for Spatial Workers

Within the Local Spatial Processing Patterns guide, this page drills into a single line of the dispatch model described in async execution for spatial workloads: the initializer argument passed to ProcessPoolExecutor. That parent guide establishes why CPU-bound geometry work has to leave the event loop for a process pool; this page is about what stops that pool from becoming the thing that kills the gateway. A worker process handling GeoJSON parsing, coordinate transforms, and native predicate calls accumulates fragmented heap the same way any long-running C-backed process does, and on a device with 512 MB to 2 GB shared with a modem, three or four workers each drifting upward by a few megabytes per hour is a slow-motion OOM kill that will not show up in a short test run.

The fix has three independent layers, and each catches a different failure: a hard kernel-enforced ceiling per process (resource.setrlimit), a scheduled recycling of the worker itself before fragmentation accumulates (max_tasks_per_child), and an outer cgroup boundary around the whole pool that survives even if the first two layers are misconfigured (MemoryMax in a systemd unit). None of the three is sufficient alone.

Worker memory model selection rationale #

A ProcessPoolExecutor worker is a long-lived Python process that services many calls over its lifetime. Two properties of that lifetime work against a constrained gateway. First, native allocators (glibc’s malloc, or whatever libgeos/libproj link against) rarely return freed memory to the operating system — they keep it in the process’s arena for reuse, which means RSS is a ratchet that mostly only goes up as geometry sizes vary across calls. Second, a single runaway call — an unexpectedly large polygon, a malformed GeoJSON feature with a pathological ring count — can spike one worker’s memory well past what the rest of the pool needs, and without an enforced ceiling that spike is invisible to the scheduler until the kernel OOM killer picks a victim, often not the offending process.

Three controls close this gap, applied in the initializer that runs once per worker at spawn:

1. resource.setrlimit(resource.RLIMIT_AS, ...) caps the process’s total virtual address space — every mmap, every heap page, every stack. It is the broadest and safest limit because it catches leaks in native code the Python layer never sees directly. RLIMIT_DATA is the narrower alternative, capping only the traditional heap (brk) segment; it is weaker on modern glibc, which serves large allocations via mmap rather than brk and so can sidestep a RLIMIT_DATA cap entirely. Prefer RLIMIT_AS on Linux gateways for that reason.
2. max_tasks_per_child (a ProcessPoolExecutor constructor argument since Python 3.11) retires and respawns a worker after it has serviced a fixed number of calls, resetting its heap from a clean process image before fragmentation has a chance to compound. This is cheaper than it sounds: a fresh fork/exec on a Cortex-A gateway costs single-digit milliseconds, far less than the cost of an OOM kill and unplanned pool restart.
3. MemoryMax= in the systemd unit wraps the entire service — main process and every worker — in a cgroup v2 memory ceiling. If the first two layers are ever misconfigured, this is the backstop that contains the damage to one service instead of starving the whole gateway, including the modem stack and OS.

The systemd MemoryMax cgroup wraps this entire cycle as the outermost, hardest-to-misconfigure boundary.

The complete implementation #

# worker_bounds.py — hard per-worker memory ceiling plus scheduled recycling
# for a spatial ProcessPoolExecutor. Python 3.11+ for max_tasks_per_child.

import concurrent.futures
import logging
import os
import resource

log = logging.getLogger("worker_bounds")

# Sizing math: (total_RAM - OS_reserve - modem_reserve) / MAX_WORKERS, then a
# safety margin. On a 1 GB gateway: (1024 - 150 - 120) / 4 workers ~= 188 MB,
# rounded down to leave headroom for the pool's own bookkeeping.
MAX_WORKERS = min(os.cpu_count() or 2, 4)
PER_WORKER_RSS_MB = 175
RECYCLE_AFTER_TASKS = 500   # retire a worker before fragmentation compounds

def worker_init():
    """Runs once per worker process, before the first call is serviced.
    Caps virtual address space; RLIMIT_AS covers mmap'd allocations that
    RLIMIT_DATA (heap/brk only) would miss on a modern glibc."""
    limit_bytes = PER_WORKER_RSS_MB * 1024 * 1024
    soft, hard = resource.getrlimit(resource.RLIMIT_AS)
    # Never raise the hard ceiling above what the OS already allows.
    new_hard = hard if hard == resource.RLIM_INFINITY else min(hard, limit_bytes)
    resource.setrlimit(resource.RLIMIT_AS, (limit_bytes, new_hard))
    log.info("worker %d: RLIMIT_AS capped at %d MB", os.getpid(), PER_WORKER_RSS_MB)
    # Pre-load native contexts (GEOS/PROJ) here so the first real call doesn't
    # pay setup cost against the same budget as the memory it needs for data.

def build_executor() -> concurrent.futures.ProcessPoolExecutor:
    return concurrent.futures.ProcessPoolExecutor(
        max_workers=MAX_WORKERS,
        initializer=worker_init,
        max_tasks_per_child=RECYCLE_AFTER_TASKS,
    )

max_tasks_per_child is enforced by the executor itself: once a worker has completed the configured number of tasks, the pool terminates that process and spawns a replacement, running worker_init again on the new process before handing it more work. Any task already dispatched to the retiring worker still completes normally — the recycling happens on the boundary between tasks, never mid-call. Set RECYCLE_AFTER_TASKS from measured fragmentation growth, not a guess: instrument the RSS-versus-task-count curve in staging (see the field diagnostics below) and pick a number comfortably before RSS approaches the RLIMIT_AS ceiling under worst-case inputs.

The RLIMIT_AS value on its own does not prevent a worker from trying to allocate past it — it prevents that allocation from succeeding. A native call that hits the ceiling gets ENOMEM back from mmap/brk, which most C libraries turn into an allocation failure exception or a hard abort depending on how they were compiled; either way the failure is contained to that one worker rather than starving the pool, and the executor detects the dead process and raises BrokenProcessPool on the next submission.

Constraint validation table #

Gotchas and edge cases #

• RLIMIT_AS is stricter than it looks. It counts reserved address space, not just resident pages — a large mmap opened with MAP_NORESERVE or a big anonymous mapping that is mostly untouched still counts against the limit. Size the ceiling against peak virtual mapping, including any memory-mapped reference layers a worker holds open, not just expected RSS.
• RLIMIT_DATA is not a substitute. It only bounds the traditional brk heap segment. Modern glibc routes allocations above M_MMAP_THRESHOLD (128 KB by default) through mmap instead, which RLIMIT_DATA does not see at all — a worker parsing one large GeoJSON feature can blow past intended limits while RLIMIT_DATA reports nothing wrong.
• Hitting the limit mid-call is a hard failure, not a warning. Depending on how libgeos/libproj were compiled, an allocation failure inside native code can raise a Python exception via ctypes/cffi error paths, or in the worst case call abort() directly. Treat any dispatch to the pool as capable of failing outright and make the caller’s retry/backoff path — the same discipline used in retry and backoff for unstable networks — the thing that absorbs it, not a try/except that assumes graceful degradation.
• max_tasks_per_child interacts with in-flight work, not with queued work. Only completed tasks count toward the recycle threshold; a worker will not be killed mid-task to enforce the schedule. Do not rely on it as a timeout mechanism — that job belongs to the deadline pattern covered separately for the async loop side of this pipeline.
• cgroup MemoryMax breaches trigger the kernel OOM killer within the cgroup, which may reap a worker the application never got a chance to log. Pair MemoryMax with MemoryHigh= set slightly lower, which throttles rather than kills, giving the process-level limits a chance to act first.
• Sizing per-worker RSS wrong in either direction hurts. Too generous and a single leaking worker can still starve its siblings before the recycle fires; too tight and legitimate large polygons fail spuriously. Start from the sizing formula in the initializer’s comment and adjust from measured peak RSS under representative field data, not synthetic small polygons.

Integrating with the systemd unit #

The Python-level controls are necessary but not sufficient without an enclosing cgroup, configured once in the unit file that runs the gateway service:

# /etc/systemd/system/spatial-worker-pool.service
[Service]
ExecStart=/usr/bin/python3 /opt/gateway/run_pool.py
MemoryHigh=700M
MemoryMax=800M
CPUQuota=350%
Restart=on-failure
RestartSec=2

MemoryHigh gives the kernel a soft ceiling that throttles allocation and reclaims page cache before things get desperate; MemoryMax is the hard kill switch that should, in a correctly sized deployment, never actually trigger — if it does, that is the field signal to lower PER_WORKER_RSS_MB or MAX_WORKERS rather than raise MemoryMax. Restart=on-failure ensures that if the whole service is reaped, it comes back cleanly rather than leaving the gateway with no spatial processing at all until a human intervenes.

Related #

• Async Execution for Spatial Workloads — the dispatch model this worker pool serves, including chunk sizing and timeout handling.
• Asyncio Cancellation and Timeouts on Low-RAM Targets — the matching control on the event-loop side, for in-flight coroutines rather than worker processes.
• Device Constraints and Resource Limits — the RAM and CPU budgets that determine every constant in the sizing formula above.