FlatGeobuf vs GeoArrow for the Edge Wire Format

This guide solves one narrow decision that shapes everything downstream of it: on a gateway with 256 MB–2 GB of shared RAM, should the on-disk and on-wire representation of a feature batch be FlatGeobuf or GeoArrow? Within the Bandwidth & Async Sync Optimization practice, and specifically inside the compression strategies for geospatial payloads this guide belongs to, the structural-serialization stage is the one every later layer inherits: a poor choice here either forces the gateway to load a whole feature collection to answer a single bounding-box query, or forces every downstream consumer to pay a per-feature decode tax before it can do bulk arithmetic. The two formats are not interchangeable defaults — they optimize for opposite access patterns, and a store-and-forward gateway usually needs both at different points in its pipeline.

Streaming random access vs columnar scan: the selection rationale #

FlatGeobuf is a flat binary container built on Google’s FlatBuffers wire format. Every feature is written as a self-describing FlatBuffers table, one after another, prefixed by a header that carries the schema and, optionally, a packed Hilbert R-tree spatial index. Because FlatBuffers tables are read in place — there is no unmarshal step, no intermediate object graph — a consumer can mmap the file and pull a single feature’s geometry straight out of the buffer at whatever offset the R-tree points to. That property is what makes FlatGeobuf a genuine streaming format rather than just a compact one: a gateway can answer “which features fall inside this cell’s bounding box” by walking O(log n) index nodes and touching only the matching feature bytes, never paging in the rest of the file.

GeoArrow takes the opposite bet. It encodes geometry into Apache Arrow’s columnar memory layout — coordinates live in flat, type-homogeneous buffers (a double array for x, another for y, or nested list arrays for line and polygon rings) rather than being interleaved inside a heterogeneous per-feature record. That layout is what lets a vectorized runtime — NumPy, pyarrow compute kernels, or a Rust/C++ Arrow consumer — apply one operation across every coordinate in a batch with no per-feature branch or allocation. The cost is that GeoArrow carries no spatial index of its own and is normally read or written as a whole Arrow IPC stream of record batches; a consumer generally wants the batch, or at least a full record batch, resident before it does anything useful with it.

Store-and-forward is the deciding factor for a gateway. A telemetry batch that has to survive a dropped LTE session, get re-queried by bounding box after a partial send, or resume from an arbitrary byte offset needs the random-access property FlatGeobuf’s spatial index provides — the same resumability concern that shapes delta sync for spatial datasets, just applied to the wire format instead of the diffing scheme. GeoArrow earns its keep on the other side of the pipe: once a batch is confirmed and staged for transformation — reprojection, delta computation across an entire array, or handing a table to a downstream analytics process — its columnar layout turns what would be thousands of Python-level per-feature calls into a handful of vectorized ones.

Reading a bounded window without loading the file #

The implementation below is the pattern that matters on a gateway: answer a bounding-box query against a FlatGeobuf file using its embedded index, touching only the bytes the query needs. It assumes the flatgeobuf package’s streaming reader, which exposes the packed R-tree as a generator of byte ranges rather than forcing a full parse.

# fgb_bbox_read.py
# Bounded-memory bbox query against a FlatGeobuf file using its packed R-tree.
# Target: gateways with 256 MB-2 GB RAM; peak resident bytes stay near the
# size of the matched features, not the size of the source file.
import mmap
from flatgeobuf import FeatureCollection  # streaming reader over the FlatBuffers layout

def bbox_query(fgb_path: str, min_x: float, min_y: float, max_x: float, max_y: float):
    """Yields only the features whose stored bbox intersects the query window.

    The file is mmap'd read-only so the kernel pages in index nodes and
    feature bytes on demand; nothing beyond the matched features and the
    index nodes walked to find them is ever resident.
    """
    with open(fgb_path, "rb") as f:
        with mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ) as buf:
            fc = FeatureCollection.open(buf)  # reads only the header + index root
            if not fc.has_index:
                raise ValueError("file was written with index=False; full scan required")
            for feature in fc.bbox_query(min_x, min_y, max_x, max_y):
                # feature.geometry is a zero-copy view into `buf`; the caller
                # must not retain it past the `with` block that owns the mmap.
                yield feature.id, feature.geometry, feature.properties

def count_matches(fgb_path: str, bbox: tuple[float, float, float, float]) -> int:
    """Cheap existence/size check before committing to a transfer window."""
    return sum(1 for _ in bbox_query(fgb_path, *bbox))

The load-bearing detail is fc.has_index: a FlatGeobuf written with index=False (the choice the parent compression guide makes for small batches, since the R-tree itself carries fixed overhead) degrades to a linear scan, which is the right trade for a few hundred features but the wrong one once a gateway is holding a multi-thousand-feature tile cache. Decide the index flag at write time based on expected query volume, not file size alone — an index that is never queried is pure overhead on a metered link.

Constraint validation table #

Gotchas and edge cases #

• An unindexed FlatGeobuf silently becomes a full scan. The parent compression guide’s write(fgb_path, index=False) choice is correct for small per-batch files, but if that same code path is reused for a growing tile cache without revisiting the flag, every bbox_query call degrades to reading the entire file. Gate the flag on an expected feature-count threshold, not a one-time decision at first deployment.
• GeoArrow’s coordinate arrays assume a fixed geometry type per column. Mixed point/line/polygon batches need either a union layout or separate tables per geometry type; bolting mixed geometries onto a single flat x/y array silently corrupts alignment between rings and their parent features.
• mmap’d views do not outlive the file handle. The feature.geometry view returned from bbox_query above is only valid inside the with mmap.mmap(...) block; copy any geometry you need to retain (bytes(feature.geometry)) before the block exits, or you will read freed pages.
• Arrow IPC streaming is not the same as random access. pyarrow’s streaming format lets a consumer read record batches incrementally as they arrive, which is useful for backpressure, but it does not give you an index to jump to an arbitrary spatial region the way FlatGeobuf’s R-tree does — treat GeoArrow batches as sequential, not addressable.
• Coordinate precision still has to be decided upstream. Neither format enforces a decimal budget; pack coordinates to the resolution set by your spatial data precision standards before writing either container, or the columnar arrays simply carry the same float64 noise FlatGeobuf’s tables would.

Integrating a batch handoff to GeoArrow #

Once a FlatGeobuf window has been confirmed and staged, hand the matched features to a columnar structure for the bulk work — reprojection, delta computation, or a compression pass over the whole array at once — before compressing with the codec choice covered under zstd and Brotli for MVT payloads or the record-aligned chunking used for Brotli-compressed shapefile chunks:

import pyarrow as pa

def to_geoarrow_batch(matches: list[tuple[int, tuple[float, float], dict]]) -> pa.RecordBatch:
    """Builds a columnar point batch from FlatGeobuf query results.
    Coordinates land in contiguous float64 arrays -- no per-feature Python
    objects survive past this call, so a vectorized transform downstream
    (reproject, quantize, delta) touches the arrays directly.
    """
    ids = pa.array([m[0] for m in matches], type=pa.int64())
    xs = pa.array([m[1][0] for m in matches], type=pa.float64())
    ys = pa.array([m[1][1] for m in matches], type=pa.float64())
    return pa.RecordBatch.from_arrays([ids, xs, ys], names=["id", "x", "y"])

This is the natural seam between the two formats: FlatGeobuf owns the resumable, queryable staging window on the gateway; GeoArrow owns the vectorized transform once a batch is committed to being sent. Building both into one pipeline avoids the false choice of picking a single format for every stage.

Related #

• Compression Strategies for Geospatial Payloads — the parent guide’s layered structural/semantic/entropy model this wire-format choice feeds.
• Zstd vs Brotli for MVT Payloads on Metered LTE — the entropy-coding stage that runs after either format is chosen.
• Delta Sync for Spatial Datasets — the resumability concern that makes FlatGeobuf’s index the right default for store-and-forward.