Compression Strategies for Geospatial Payloads

Within the Bandwidth & Async Sync Optimization framework, payload compression is the deterministic pipeline stage that decides whether a field gateway’s spatial telemetry clears a constrained uplink or backs up until the watchdog reboots the device.

Telemetry and spatial datasets rarely fit cleanly into cellular, satellite, or LPWAN budgets. A fanless gateway parked at a remote site has to fit point clouds, vector feature updates, and sensor batches through links that drop, throttle, and meter every byte — while staying inside a 512 MB to 2 GB RAM envelope and a thermal ceiling that punishes any CPU spike. On that hardware, compression is not a post-processing afterthought; it is an architectural decision that directly sets queue depth, sync latency, and power draw. This guide lays out constraint-aware patterns for IoT gateways: a compact binary baseline, memory-safe streaming chunking, codec selection through C bindings, and the field diagnostics that tell you the pipeline is actually working on a deployed unit.

The Compression Decision, End to End #

Before reaching for an algorithm, decide where each byte of savings comes from. Three independent layers stack multiplicatively, and skipping the cheap ones wastes CPU on the expensive ones:

1. Structural — drop schema overhead by serializing to a compact binary container.
2. Semantic — drop redundant coordinates with delta encoding and fixed-point quantization, which dovetails with Delta Sync for Spatial Datasets.
3. Entropy — apply a general-purpose codec (LZ4, Zstandard, Brotli) to whatever remains.

A gateway that jumps straight to entropy coding on raw GeoJSON pays Brotli-level CPU to undo whitespace and repeated keys that a binary format removes for free. Order the layers structural → semantic → entropy and each one operates on already-reduced input.

Constraint Mapping #

Every codec knob trades one constrained resource for another. On a fanless ARM Cortex-A53/A72 gateway the binding limits are heap, sustained CPU (thermal), and the link budget itself. Map the technique to the limit it stresses before tuning, and cross-reference the broader device constraints and resource limits envelope for your specific SoC class.

The non-obvious row is flash endurance: compressing before the local queue means the store-and-forward buffer described in Message Queue Management at the Edge writes far fewer bytes to SD or eMMC during an outage, extending card life across thousands of spill cycles.

Implementation 1: Binary Serialization Baseline #

Raw shapefiles, GeoJSON, and CSV coordinate dumps inflate payloads by 40–70% through redundant schema overhead, coordinate-precision bloat, and whitespace padding. The cheapest win is structural: serialize to a compact binary container before any algorithmic compression runs.

Switching from verbose ASCII GeoJSON to FlatGeobuf reduces file size by 40–60% before a single byte of entropy coding, because FlatGeobuf uses a flat binary layout, eliminates key repetition, and enables direct memory mapping. That memory-mapping property is what makes it edge-safe: on a 2 GB gateway you can mmap a feature collection and let the kernel page geometries in on demand instead of parsing the whole document onto the heap. GeoPackage (SQLite-backed) is the alternative when you need indexed random access rather than a streaming wire format.

# Structural reduction: GeoJSON -> FlatGeobuf, streamed, no whole-file heap load.
# Threading model: pure CPU, runs in a worker thread (see asyncio offload below).
# GC note: read in fixed-size blocks so the GeoJSON string is never fully resident.
from flatgeobuf import FeatureCollection  # pip wheel ships a manylinux aarch64 build

def reserialize_to_fgb(geojson_path: str, fgb_path: str) -> int:
    fc = FeatureCollection.from_geojson_stream(geojson_path)  # incremental parse
    written = fc.write(fgb_path, index=False)  # skip packed R-tree on tiny batches
    return written  # bytes on disk; compare against source for the structural ratio

Keep coordinate precision honest at this stage. Trimming float64 longitude to the precision your sensor actually resolves is not lossy in any meaningful sense — it is aligning storage with the spatial data precision standards your deployment already operates under, and it shrinks every downstream layer.

Implementation 2: Streaming Chunking and Queue Integration #

Monolithic payloads exhaust heap and trigger OOM kills under memory pressure. A streaming chunker segments spatial data into fixed-size blocks aligned with the cellular MTU (1400–1500 bytes) or the broker’s maximum message size. Each chunk carries a lightweight header so it can be decompressed and reassembled independently:

[4B seq_id][2B compression_algo][8B bbox_minmax][2B payload_len]

The independent-decompressibility property is what makes chunks safe to interleave with the retry machinery in retry and backoff for unstable networks: a single failed chunk re-sends on its own without invalidating the rest of the batch.

import asyncio
import struct
import lz4.frame

CHUNK_SIZE = 1400  # Align with cellular MTU to avoid IP fragmentation
HEADER_FMT = "!IH8sH"  # seq_id (4B), algo_id (2B), bbox (8B), payload_len (2B)

async def stream_compress_chunker(raw_bytes, bbox, broker):
    # asyncio model: this coroutine owns chunk emission; the actual compress()
    # is fast enough at LZ4 to run inline. For zstd/Brotli, see the FFI offload.
    # GC note: slice views, not copies, keep transient allocations flat.
    seq = 0
    for i in range(0, len(raw_bytes), CHUNK_SIZE):
        chunk = raw_bytes[i:i + CHUNK_SIZE]
        compressed = lz4.frame.compress(chunk, store_size=False)
        header = struct.pack(HEADER_FMT, seq, 0x01, bbox, len(compressed))
        await broker.publish(f"geo/chunk/{seq}", header + compressed)
        seq += 1
        await asyncio.sleep(0)  # Yield so the broker can apply backpressure

When the link degrades, the local broker throttles publish() and the await points let the queue apply backpressure without blocking the ingestion thread — the gateway stays responsive to sensor polling and watchdog kicks. For batched MQTT delivery the per-chunk QoS choice matters; pair this chunker with configuring MQTT QoS levels for telemetry drops so that lossy chunks retry at the transport layer rather than forcing a full-batch resend.

Delta Encoding and Coordinate Quantization #

Absolute coordinate dumps waste bandwidth on bits that never change between fixes. Apply delta encoding to sequential telemetry points — store only the offset from the last known position — then quantize to fixed point. A 5-decimal grid resolves roughly 1.1 m at the equator, which collapses a float64 pair into an int32 pair:

$$q = \operatorname{round}(\lambda \times 10^{5}), \qquad \Delta q_i = q_i - q_{i-1}$$

Most consecutive deltas then fit in a single byte under varint encoding, so the entropy stage sees a stream of small, highly repetitive integers instead of full-width doubles. This is the same quantization discipline used for coordinate reference systems at the edge, applied to the wire format. Maintain a rolling buffer holding the last valid coordinate; on a sync failure, emit a full-geometry reset chunk rather than risk a divergent delta chain.

Implementation 3: Codec Selection Through FFI #

Entropy coding sits on a strict CPU-cycles-versus-ratio spectrum, and Python’s interpreter overhead makes pure-Python codecs unusable for high-throughput edge workloads. Bind directly to the C libraries through ctypes or cffi to bypass the interpreter and control allocation.

• LZ4 / Snappy — throughput first (>500 MB/s) at modest ratios (~2.1x). Ideal for high-frequency sensor streams where latency dominates. Use block-level compression for a predictable memory footprint; see the LZ4 Python documentation for the streaming bindings.
• Zstandard (zstd) — tunable levels 1–22. Levels 3–5 are the sweet spot for batched spatial telemetry, and dictionary training pays off when payloads share structure. The Zstandard documentation covers bounded-memory streaming and dictionary APIs.
• Brotli — exceptional ratios (~3.5x+) on structured spatial data but heavy on CPU and memory; reserve it for idle-window batch work. The record-aligned approach is detailed in applying Brotli compression to shapefile chunks.

Choosing a codec by throughput versus ratio for the target hardware profile.

When the binding does not ship a streaming Python API, a thin C shim keeps allocation off the Python heap and exceptions out of the hot path. Compile it -O2 -fno-exceptions and call it through ctypes:

/* edge_zstd.c — bounded one-shot wrapper; compile:
 *   cc -O2 -fno-exceptions -fPIC -shared edge_zstd.c -lzstd -o libedgezstd.so
 * No malloc in the hot path: caller owns dst, we never grow it. */
#include <zstd.h>
#include <stddef.h>

/* Returns compressed size, or 0 on error (dst too small / zstd failure). */
size_t edge_zstd_compress(const void *src, size_t src_len,
                          void *dst, size_t dst_cap, int level) {
    size_t r = ZSTD_compress(dst, dst_cap, src, src_len, level);
    return ZSTD_isError(r) ? 0u : r;  /* caller checks 0 and falls back to LZ4 */
}

import ctypes, asyncio

_lib = ctypes.CDLL("./libedgezstd.so")
_lib.edge_zstd_compress.restype = ctypes.c_size_t
_lib.edge_zstd_compress.argtypes = [
    ctypes.c_void_p, ctypes.c_size_t,
    ctypes.c_void_p, ctypes.c_size_t, ctypes.c_int,
]

async def zstd_compress(src: bytes, level: int = 4) -> bytes:
    # Threading model: GIL is released across the C call, so to_thread() lets
    # the event loop keep servicing sensor I/O during synchronous compression.
    dst = bytearray(len(src) + 64)  # preallocated; reuse across calls in prod
    def _run():
        n = _lib.edge_zstd_compress(bytes(src), len(src),
                                    (ctypes.c_char * len(dst)).from_buffer(dst),
                                    len(dst), level)
        return bytes(dst[:n]) if n else b""
    return await asyncio.to_thread(_run)

Wrapping the synchronous codec in asyncio.to_thread() (or a uvloop-compatible executor) prevents GIL contention from stalling the ingestion coroutine. Watch thread-pool saturation: if every compression call blocks a worker, peak ingestion can starve the queue. For workloads where compression is the bottleneck, route it through the same off-loop execution patterns used for on-device geometry filtering so the spatial and serialization stages share one bounded thread pool instead of competing for two.

Configuration and Tuning #

The knobs that matter on constrained hardware are codec level, codec window size, and chunk size — in that order of impact.

• Codec level — start at zstd 3 and raise only while the CPU has thermal headroom. Each level above 5 buys diminishing ratio for steep CPU cost; on a fanless enclosure, levels above 9 reliably trigger throttling.
• Window / block size — bound the codec window (ZSTD_c_windowLog ≈ 18–20, i.e. 256 KB–1 MB) so decompression memory stays predictable on the receiver and the gateway never allocates a multi-megabyte match window.
• Chunk size — keep CHUNK_SIZE just under the path MTU. Probe it once at boot rather than hard-coding 1400; a PPP or VPN encapsulated link can shave the usable MTU and silently force fragmentation.
• Dictionary training — for fleets emitting structurally similar batches, train a zstd dictionary offline and ship it with the firmware. A 4–16 KB dictionary can double the effective ratio on small payloads where the codec otherwise has no history to exploit.
• Build flags — compile native shims -O2 -fno-exceptions and, on ARM, add -mcpu=native (or the explicit -mcpu=cortex-a53) so the compiler emits NEON paths the codec can use.

Verification and Field Diagnostics #

A compression stage that silently degrades is worse than none — it burns CPU and ships corrupt chunks. Deploy constraint-aware metrics alongside the pipeline:

• heap_peak_mb / chunk_alloc_failures
• compression_cpu_ms / ratio_actual
• queue_depth / backpressure_events

Sample heap with tracemalloc and process RSS with psutil; if ratio_actual drifts below the codec’s expected floor, the input is already compressed or corrupt and you are wasting cycles. Validate every chunk with a CRC32C checksum before it enters the queue, and log fallback triggers to /var/log/edge-compression.log as structured JSON for remote diagnostics. To confirm the pipeline survives a degraded link before you leave the site, shape the interface and watch the queue drain:

# Inject 10% loss + 80 ms latency, then confirm chunks still reassemble.
tc qdisc add dev wwan0 root netem loss 10% delay 80ms
journalctl -u edge-compression -f | grep -E 'ratio_actual|crc_fail|backpressure'
tc qdisc del dev wwan0 root  # restore before disconnecting

A healthy run shows ratio_actual near the codec’s expected value, zero crc_fail, and queue_depth that rises during the loss window then drains — not one that climbs monotonically toward the spill threshold.

Failure Modes Specific to This Pattern #

• Heap exhaustion from un-chunked input — a single oversized feature collection slips past the chunker and the codec allocates a window larger than free RAM. Detect: chunk_alloc_failures increments. Recover: hard-cap input size at the serializer; reject and log oversized features rather than buffering them.
• Thermal throttling masquerading as link failure — a high codec level pushes the SoC into throttling, compression_cpu_ms spikes, sensor polls slip, and the symptom looks like a stalled uplink. Detect: CPU time per chunk rising while ratio is flat. Recover: drop the level adaptively. When cellular RSSI falls below −105 dBm, downgrade (zstd 5 → 1) to free CPU for the radio stack and TLS handshakes.
• Divergent delta chains — a dropped chunk corrupts the receiver’s coordinate baseline and every subsequent delta decodes to the wrong position. Detect: CRC mismatch on reassembly. Recover: trigger a full-state resync; never attempt partial recovery, which compounds spatial drift.
• Compressing already-compressed data — re-running a codec over zstd or Brotli output inflates the payload and wastes CPU. Detect: ratio_actual ≤ 1.0. Recover: tag each chunk’s compression_algo in the header and skip codecs whose magic bytes are already present.

The safe default across all four is the same: when a chunk fails to reassemble, fall back to a full-state resync rather than patching forward from a corrupted baseline.

Related guides #

• Bandwidth & Async Sync Optimization — the parent discipline this compression stage feeds into.
• Delta Sync for Spatial Datasets — the semantic-reduction layer that pairs with quantized deltas above.
• Message Queue Management at the Edge — backpressure and store-and-forward for the chunks this pipeline emits.
• Retry and backoff for unstable networks — transport-layer recovery for failed chunk sends.
• Applying Brotli compression to shapefile chunks — a worked, record-aligned implementation of the high-ratio codec path.