Edge Geospatial IoT Gateway Processing

CPU affinity and isolcpus for deterministic geometry

A geometry predicate that averages 8 µs and occasionally takes 4 ms has a latency problem no amount of algorithmic work will fix, because the 4 ms is not the algorithm — it is the scheduler moving the thread to a cold core, a network interrupt landing on the same CPU, or the kernel deciding another task deserved the slot. Pinning fixes what the scheduler is allowed to do. This guide covers affinity, core isolation and interrupt steering for the workloads in device constraints and resource limits, within the Core Edge GIS Fundamentals envelope.

Where the tail latency comes from

Three mechanisms produce almost all of the jitter on a quad-core Cortex-A gateway, and each has a different fix.

Migration. The scheduler moves a thread between cores to balance load. Every migration costs a cold L1 and L2 for that thread — on a Cortex-A53 that is 100–300 µs of re-warming for a working set that fits in L2, considerably more for one that does not. Fixed by affinity.

Interrupts. A cellular modem, an Ethernet controller and a UART all raise interrupts, and by default they land on CPU 0 or wherever the IRQ balancer puts them. An interrupt landing on the core running a geometry batch preempts it for tens of microseconds and evicts cache lines. Fixed by IRQ affinity.

Contention. Another runnable task at the same priority gets a timeslice. On a device where the spatial worker shares a core with a log rotation or a tile read, this is the largest term. Fixed by isolation or by scheduling policy.

Latency distribution for one geometry batch under four configurations The same 4 000-point containment batch measured under four configurations. With default scheduling the median is 8.1 milliseconds and the 99th percentile is 41. With thread affinity to one core the median is 7.4 and the 99th percentile 19. Adding interrupt steering away from that core gives 7.2 and 11. Adding full core isolation through isolcpus gives 7.1 and 7.9, at which point the distribution is nearly flat and the tail is the algorithm rather than the system. Same work, four configurations — the median barely moves and the tail collapses default+ thread affinity+ IRQ steering+ isolcpus p99 41 msp99 19 msp99 11 msp99 7.9 ms median 8.17.47.27.1 Solid is the median, dashed is the spread to the 99th percentile. Nothing about the algorithm changed between rows.
Optimising the median here would be optimising the wrong number. The whole problem, and the whole fix, lives in the dashed part.

Affinity first: it costs nothing and needs no boot change

Pinning a thread is a runtime call and is reversible, which makes it the right first step. On a quad-core device, reserve one core for the latency-sensitive work and leave the rest to everything else.

# affinity.py — pin the spatial worker and verify it stuck.
# Called once at start-up, from the worker process itself.
import os


SPATIAL_CORE = 3          # the highest-numbered core, conventionally the reserved one


def pin_to(core: int) -> bool:
    """Restrict this process to one CPU. Returns False when the platform
    refuses — a container without CAP_SYS_NICE, or a single-core device."""
    try:
        os.sched_setaffinity(0, {core})
    except (OSError, AttributeError):
        return False
    return os.sched_getaffinity(0) == {core}


def pin_worker_initializer(core: int = SPATIAL_CORE):
    """Pass as the `initializer` of a ProcessPoolExecutor so every worker
    pins itself as it starts. Pinning the parent does NOT pin the children
    on all platforms, which is the mistake this exists to avoid."""
    def _init():
        if not pin_to(core):
            # Not fatal: a device that cannot pin still works, just noisier.
            import logging
            logging.getLogger("spatial").warning("affinity not applied")
    return _init

Two workers pinned to the same core serialise, which is usually not what was wanted — pin each worker to its own core, or pin the pool to a set and let the scheduler place within it. For a pool of two on a quad-core, {2, 3} is the sensible set: the two reserved cores, with the scheduler free to balance between them and never to migrate into the busy half of the machine.

Steering interrupts away

Affinity keeps the thread on a core; it does not keep interrupts off it. On Linux each IRQ has a smp_affinity mask under /proc/irq/, and the modem and network interrupts should be pointed at the cores the spatial work is not using.

# irq_steer.sh — keep interrupts off the reserved cores.
# Run at boot, after the interfaces are up. Masks are hex bitmaps: 0x3 = cores 0,1.
set -eu
GENERAL_MASK=3        # cores 0 and 1 handle everything interrupt-driven

for irq in /proc/irq/[0-9]*; do
    n=$(basename "$irq")
    # Some IRQs (per-CPU timers, IPIs) refuse to be steered; skip them quietly.
    [ -w "$irq/smp_affinity" ] || continue
    printf '%x' "$GENERAL_MASK" > "$irq/smp_affinity" 2>/dev/null || true
done

# The default for any IRQ registered later.
printf '%x' "$GENERAL_MASK" > /proc/irq/default_smp_affinity

# Confirm: anything still showing counts on cores 2-3 is unsteerable.
awk 'NR==1 || /:/ {print $1, $4, $5}' /proc/interrupts | head -20

Verify rather than assume. /proc/interrupts shows per-core counts, and an IRQ still incrementing on a reserved core after this script has run is one the platform will not let you move — usually a per-CPU timer, which is harmless, or occasionally a driver that ignores the mask, which is worth knowing about.

Isolation, for when affinity is not enough

isolcpus removes cores from the scheduler’s general balancing entirely: nothing runs there unless explicitly placed. It is a kernel command-line parameter, so it requires a boot configuration change and a reboot, and it is the strongest of the three tools.

# /boot/cmdline.txt or the bootloader's kernel args
isolcpus=2,3 nohz_full=2,3 rcu_nocbs=2,3

The three parameters work together. isolcpus stops the scheduler placing work there. nohz_full stops the periodic scheduler tick on those cores when only one task is runnable, removing a 1–4 ms interruption. rcu_nocbs moves RCU callback processing off them, which otherwise reappears as periodic jitter that looks inexplicable.

Core allocation on a quad-core gateway before and after isolation Before isolation, all four cores run a mixture: the spatial worker, the telemetry reader, the broker, log rotation, interrupts and kernel threads are spread across every core by the scheduler. After isolation, cores 0 and 1 carry everything general — the telemetry reader, the broker, logging, interrupts and kernel housekeeping — while cores 2 and 3 carry only the two pinned spatial workers, with no tick, no RCU callbacks and no interrupts. Before and after: the same four cores, two very different machines default cpu0cpu1cpu2cpu3 worker · broker · irqreader · logs · irqworker · irq · rcubroker · tick · rcu isolcpus=2,3 cpu0cpu1cpu2cpu3 broker · logs · irqreader · kernel · irqspatial worker onlyspatial worker only Half the machine is given up to protect the other half. On a device whose job is the spatial work, that is the correct trade. On a dual-core device it usually is not — isolation would leave one core for everything else.
Isolation is not free: it removes cores from general use permanently. It earns its cost only where the isolated work is the reason the device exists.

Constraint validation

Constraint Expected impact Mitigation
Latency tail Migration and interrupts add tens of milliseconds at p99 Affinity removes migration; IRQ steering removes preemption; isolation removes contention
Throughput Reserving cores reduces the machine available to everything else Reserve only what the latency-critical path needs; two of four is the usual split
Thermal Concentrating work on two cores raises their die temperature Watch the thermal zone; the total work is unchanged but its distribution is not
Portability isolcpus is a boot parameter and varies by platform Affinity works everywhere and gets most of the benefit; treat isolation as an optimisation
Containers A container may lack the capability to set affinity pin_to returns False rather than failing; the device still works, noisier

Gotchas and edge cases

  • Pinning the parent does not pin the children. A ProcessPoolExecutor whose parent is pinned may or may not inherit the mask depending on platform and start method. Pin in the worker initialiser, and assert it took.
  • Two workers on one core is worse than none. Pinning a pool of four to a single core serialises them and adds context-switch overhead on top. Match the pinned set size to the worker count.
  • Isolation and thermal throttling interact. Two cores at full load run hotter than four at half load, and the governor may throttle sooner. Measure the die temperature after isolating; the fix, if needed, is duty-cycling the workers rather than un-isolating.
  • nohz_full needs exactly one runnable task per core to help. With two runnable tasks on an isolated core the tick comes back, and the isolation delivers less than expected. It pairs with a one-worker-per-core layout, not with an oversubscribed one.
  • Verify with the real workload, not a benchmark. A synthetic loop that fits entirely in L1 shows no migration cost and therefore no benefit from any of this. The improvement is proportional to the working set, and a spatial index is a large one.

Measuring whether it worked

The only metric that matters here is the latency distribution’s tail, so measure percentiles rather than averages. Instrument the batch with a monotonic timer, keep a small histogram in the registry, and compare p50, p95 and p99 before and after each change. A configuration that improves p99 by a factor of three and leaves p50 unchanged is exactly the expected shape; one that improves p50 and not p99 has changed something else.

Record /proc/interrupts counts on the reserved cores as a gauge too. A count that starts rising after a firmware update means a new driver ignored the affinity mask, and it will show up as a latency regression whose cause is otherwise very hard to guess.

When none of this is worth doing

Three situations make the whole exercise a waste, and recognising them saves a week.

When the work is not latency-sensitive. A batch that runs once a minute and takes 200 ms does not care about a 40 ms tail. Pinning it buys nothing and costs a core’s worth of flexibility.

When the device has two cores. Reserving one of two leaves everything else — the kernel, the modem, the broker, logging — on a single core, which produces worse overall behaviour than the jitter it removed. Affinity may still help; isolation almost never does.

When the tail is the algorithm. If the p99 is high because one in a hundred queries hits a pathological zone with 8 000 vertices, no amount of scheduler configuration changes it. Measure the distribution of work alongside the distribution of time: if they have the same shape, the problem is upstream of the scheduler.

The diagnostic that separates these cases is cheap. Run the same batch a thousand times against fixed input, with no other load on the machine. If the tail persists, it is the algorithm or the data. If it disappears, it was contention, and this guide applies.

Separating scheduler jitter from algorithmic variance Two experiments on the same workload. Running fixed input under normal system load produces a p99 of 41 milliseconds against a median of 8. Running the identical fixed input on an otherwise idle machine produces a p99 of 9.2 against the same median, which shows the tail was contention rather than the algorithm. A contrasting case shows a workload whose idle-machine p99 stays at 38, indicating the variance comes from the input distribution and no scheduler configuration will help. One experiment tells you whether this guide applies fixed input, normal load median 8 ms · p99 41 ms the observed problem fixed input, idle machine median 8 ms · p99 9.2 ms contention — this guide applies fixed input, idle, tail persists median 8 ms · p99 38 ms the data — look upstream instead
Half an hour of measurement decides whether the next week is spent on the scheduler or on the zone geometry.

Related in Device Constraints & Resource Limits