Profiling RSS and Heap Fragmentation on ARM Gateways

This guide solves a specific and maddening symptom: a Python spatial pipeline on an ARM gateway shows resident memory climbing toward the OOM-killer threshold even though every object that should be freed reports as freed, gc.collect() runs clean, and no leak shows up in application-level bookkeeping. Within the Core Edge GIS Fundamentals framework, and specifically the memory ceilings covered under device constraints and resource limits, the missing piece is usually not a leak at all — it is glibc’s allocator holding freed memory in fragmented arenas instead of returning it to the kernel. Diagnosing that requires looking below the Python heap, at what the OS and the C allocator actually report, not just at what CPython thinks it holds.

Profiling toolchain selection rationale #

Three tools answer three different questions, and conflating them is the most common diagnostic mistake on a constrained device. /proc/<pid>/status gives VmRSS (current resident set) and VmHWM (the high-water mark RSS has ever reached in this process’s lifetime) — cheap to sample every cycle, but VmRSS includes shared library pages and mapped files, not just the heap, and VmHWM only ever grows, so it tells you the worst case but not whether memory has actually been given back since. /proc/<pid>/smaps (or the lighter smaps_rollup aggregate) breaks resident memory down by mapping — heap, stack, each shared library, anonymous mmap regions — which is what lets you confirm that a growing VmRSS is coming from the heap arena and not, say, a growing set of memory-mapped tile files. Python’s tracemalloc answers a third, narrower question: which Python allocation sites are responsible for the objects CPython’s own allocator currently has live. It cannot see glibc’s arena bookkeeping at all, because tracemalloc reports what the Python object allocator requested, not what the C library did with those requests afterward.

The gap between those views is the diagnosis. If tracemalloc’s traced total stays flat while VmRSS keeps climbing, Python is not the leak — the C allocator is holding freed chunks it never returned to the kernel. glibc’s ptmalloc allocates per-thread arenas (up to 8 × ncpus by default, tunable via MALLOC_ARENA_MAX) and each arena manages its own free list; a chunk freed in one arena cannot satisfy an allocation request routed to another, and small chunks below the mmap threshold are kept, not munmap’d, once acquired. On a multi-threaded gateway process this produces exactly the symptom above: RSS plateaus at a value well above the live Python heap and never comes back down, even under continuous gc.collect() calls, because the freed-but-retained glibc chunks are invisible to both Python’s garbage collector and to tracemalloc.

jemalloc and tcmalloc take a different position on the same trade-off: fewer, size-class-segregated arenas with active decommit/purge behavior that returns freed pages to the kernel more aggressively than default glibc tuning. That behavior is genuinely better for long-running, multi-threaded fragmentation-prone workloads, but it is not free on a 256 MB device — both ship as an extra shared library the firmware image has to carry, and both add their own per-arena metadata overhead that is only worth paying if profiling actually shows glibc’s default behavior is the bottleneck, not a hypothetical one.

A self-contained RSS and fragmentation probe #

The function below correlates all three signals in one call: kernel-reported RSS and high-water mark from /proc/self/status, a heap-versus-everything-else breakdown from smaps_rollup, and Python’s own traced total from tracemalloc. The difference between the kernel’s heap figure and tracemalloc’s traced total is the fragmentation estimate — memory glibc is holding that Python itself does not believe it owns.

# arm_gateway_memprobe.py
# Correlates kernel-reported RSS with Python-traced allocations to isolate
# glibc arena fragmentation from genuine Python-level growth.
# Standard library plus ctypes for the trim/tune calls; no third-party deps.
import ctypes
import os
import tracemalloc

_libc = ctypes.CDLL("libc.so.6", use_errno=True)

def read_status_vm(pid: int = None) -> dict:
    """Parse VmRSS and VmHWM (kB) from /proc/<pid>/status."""
    pid = pid or os.getpid()
    out = {}
    with open(f"/proc/{pid}/status") as f:
        for line in f:
            if line.startswith(("VmRSS:", "VmHWM:")):
                key, val = line.split(":")
                out[key.strip()] = int(val.strip().split()[0])  # kB
    return out

def read_smaps_rollup(pid: int = None) -> dict:
    """Parse the lightweight smaps_rollup aggregate: total Rss and the
    Anonymous figure, which is the best proxy for heap-owned pages on
    a process with no large mmap'd datasets."""
    pid = pid or os.getpid()
    out = {}
    with open(f"/proc/{pid}/smaps_rollup") as f:
        for line in f:
            if line.startswith(("Rss:", "Anonymous:")):
                key, val = line.split(":")
                out[key.strip()] = int(val.strip().split()[0])  # kB
    return out

def fragmentation_estimate_kb() -> dict:
    """Returns kernel-reported anonymous (heap-ish) RSS, Python's traced
    total, and the gap between them -- the fragmentation estimate."""
    if not tracemalloc.is_tracing():
        raise RuntimeError("call tracemalloc.start() at process startup first")
    vm = read_status_vm()
    rollup = read_smaps_rollup()
    traced_current, _ = tracemalloc.get_traced_memory()
    traced_kb = traced_current // 1024
    anon_kb = rollup.get("Anonymous", vm["VmRSS"])
    return {
        "vm_rss_kb": vm["VmRSS"],
        "vm_hwm_kb": vm["VmHWM"],
        "anon_rss_kb": anon_kb,
        "python_traced_kb": traced_kb,
        "fragmentation_estimate_kb": max(0, anon_kb - traced_kb),
    }

def malloc_trim(pad: int = 0) -> bool:
    """Ask glibc to release freed top-of-heap pages back to the kernel.
    Returns True if any memory was actually released. Cheap to call on
    an idle tick; expensive to call in a per-request hot path."""
    return bool(_libc.malloc_trim(ctypes.c_size_t(pad)))

def set_arena_max(n: int) -> None:
    """Cap glibc's per-process arena count via mallopt(M_ARENA_MAX, n).
    Fewer arenas reduce cross-thread fragmentation at the cost of more
    lock contention -- usually a good trade on a 2-4 core ARM gateway
    where allocator lock contention is rarely the bottleneck."""
    M_ARENA_MAX = -8  # from glibc's malloc.h mallopt() parameter enum
    if not _libc.mallopt(ctypes.c_int(M_ARENA_MAX), ctypes.c_int(n)):
        raise OSError("mallopt(M_ARENA_MAX) failed; must be set before first malloc arena is created")

Call tracemalloc.start() once at process startup, run the pipeline for a representative interval, then call fragmentation_estimate_kb(). A fragmentation_estimate_kb value that tracks close to zero means glibc is returning memory efficiently and any growth is genuinely Python-level; a value that climbs steadily under stable python_traced_kb is the fragmentation signature this page exists to catch.

Constraint validation table #

Gotchas and edge cases #

• VmRSS is not “your” memory. It includes shared library code pages, mapped .so files, and any memory-mapped datasets (a memory-mapped FlatGeobuf tile cache, for instance) alongside the heap. Always cross-check against smaps_rollup’s Anonymous figure, which excludes file-backed mappings and is a much closer proxy for heap-owned pages.
• VmHWM never resets. It is a lifetime high-water mark, useful for confirming a peak you suspect happened hours ago, but useless for judging whether memory has been reclaimed since — use VmRSS for “right now” and VmHWM only for post-incident forensics.
• smaps (full) is expensive; smaps_rollup is not. Parsing the full per-mapping /proc/<pid>/smaps file on every polling cycle can itself become a measurable CPU cost on a busy gateway. Reach for smaps_rollup for routine sampling and drop to full smaps only when you need to localize which specific mapping is growing.
• MALLOC_ARENA_MAX=1 trades fragmentation for contention. Forcing a single arena eliminates cross-arena fragmentation entirely but serializes every thread’s allocations behind one lock. On a single- or dual-core gateway where the CPU is rarely contended on the allocator anyway, this is usually a clear win; on a four-plus-core board running several allocation-heavy worker threads at once, measure lock wait time before committing to it.
• mallopt(M_ARENA_MAX, ...) must run before arenas are created. Calling it after the process has already spun up multiple threads and allocated across several arenas has no effect on arenas that already exist — set it as early as possible in process startup, ideally before importing any C-extension module that allocates on import.
• Switching allocators via LD_PRELOAD needs ABI matching on Yocto builds. A jemalloc or tcmalloc .so built against a different libc ABI than the target rootfs will fail to load or, worse, segfault deep in a code path unrelated to the actual allocation call. Cross-compile against the exact toolchain the image uses, and validate on target hardware before shipping a fleet-wide allocator swap.

Integrating periodic profiling into the pipeline #

Wire the probe into the same executor-offload pattern used for other CPU-bound work in async execution for spatial workloads, so sampling and trimming never block the event loop that is servicing sensor I/O:

import asyncio

async def memory_watchdog(interval_s: float = 60.0, trim_threshold_kb: int = 20_000):
    """Runs as a background task alongside the ingestion loop. Samples
    fragmentation, trims when the estimate crosses a threshold, and logs
    the result -- never blocks the loop for more than the sampling cost."""
    loop = asyncio.get_running_loop()
    while True:
        await asyncio.sleep(interval_s)
        stats = await loop.run_in_executor(None, fragmentation_estimate_kb)
        if stats["fragmentation_estimate_kb"] > trim_threshold_kb:
            released = await loop.run_in_executor(None, malloc_trim, 0)
            stats["trim_released"] = released
        # Ship stats to your metrics collector or structured log here.

Set trim_threshold_kb from a baseline captured on a freshly started, warmed-up process — trimming reactively once fragmentation exceeds a known-healthy baseline avoids paying the trim cost on every cycle while still catching genuine drift before it approaches the OOM-killer threshold documented in the parent guide’s degradation ladder.

Related #

• Device Constraints & Resource Limits — the RAM, CPU, and thermal envelope this profiling technique feeds diagnostics into.
• Async Execution for Spatial Workloads — the executor-offload pattern used to keep sampling and trimming off the event loop.
• Core Edge GIS Fundamentals — the broader field reference this guide belongs to.