Offline Tile & Basemap Storage

Within the Core Edge GIS Fundamentals guide, this page covers the part of a field deployment that carries no analytical weight at all and still decides whether anyone trusts the device: the basemap. A gateway that computes a flawless geofence verdict and draws it over a grey rectangle will be reported as broken, and a technician staring at a blank viewport has no way to distinguish a rendering fault from a positioning fault. Tiles are the context that makes every other number on the screen legible, and on a disconnected device they have to be there before the disconnection starts.

The engineering problem is storage shape rather than cartography. A regional basemap at the zoom levels a field crew actually uses is hundreds of megabytes to a few gigabytes, the device has a read-mostly filesystem on flash with a finite write budget, and the reader has to answer “give me tile z/x/y” in single-digit milliseconds without a server process, without an index that has to be rebuilt at boot, and without paging a gigabyte of data through 512 MB of RAM. This page covers the container formats that make that possible, the read path each one implies, and the cache policies that keep the archive from destroying the flash it lives on.

Constraint mapping: what the hardware forces #

Tile storage is unusual in this discipline because it is dominated by storage characteristics rather than by CPU or RAM. The table below maps the limits that actually decide the design on a field gateway.

Two of those deserve emphasis because they are routinely designed around rather than designed for. Write endurance is the reason a read-only archive plus a small writable cache beats a single writable store: the archive is written once at provisioning, and only the small cache ever cycles. And the read-only rootfs is the reason the update mechanism for tiles looks nothing like a database migration — a new basemap is a new file, delivered and swapped atomically, exactly as described in signed tile and config bundles for field updates.

Implementation 1: a reader over a single-file archive #

The core of any on-device tile store is a function from (z, x, y) to a byte range. Both dominant formats — MBTiles, which is SQLite with a defined schema, and PMTiles, which is a flat file with an embedded directory — reduce to that. The reader below wraps the SQLite variant with the two things a naïve implementation always lacks: a bounded hot cache in front of it, and a read-only connection that cannot accidentally take a write lock on a mounted-read-only file.

# edge_tiles.py — read-only MBTiles reader with a bounded hot cache.
# Threading model: one connection per thread (SQLite objects are not shareable
# across threads by default); the cache is guarded by a plain lock held only for
# the dict operations, never across the disk read.
import sqlite3
import threading
from collections import OrderedDict

_HOT_MAX = 60                    # ≈4 MB at typical vector-tile sizes


class TileArchive:
    """One read-only MBTiles archive plus an LRU of decoded tiles."""

    def __init__(self, path: str, hot_max: int = _HOT_MAX):
        # immutable=1 tells SQLite the file cannot change under it: no WAL, no
        # locking, no shm file. That is what makes it safe on a read-only mount.
        self._uri = f"file:{path}?immutable=1&mode=ro"
        self._local = threading.local()
        self._hot: OrderedDict[tuple, bytes] = OrderedDict()
        self._hot_max = hot_max
        self._lock = threading.Lock()
        self.hits = 0
        self.misses = 0

    def _conn(self) -> sqlite3.Connection:
        conn = getattr(self._local, "conn", None)
        if conn is None:
            conn = sqlite3.connect(self._uri, uri=True, check_same_thread=False)
            # No journal, no synchronous cost: we never write.
            conn.execute("PRAGMA query_only = ON")
            conn.execute("PRAGMA mmap_size = 67108864")   # 64 MB window
            self._local.conn = conn
        return conn

    def get(self, z: int, x: int, y: int) -> bytes | None:
        # MBTiles stores rows in TMS order; XYZ y must be flipped.
        key = (z, x, (1 << z) - 1 - y)
        with self._lock:
            blob = self._hot.get(key)
            if blob is not None:
                self._hot.move_to_end(key)
                self.hits += 1
                return blob

        row = self._conn().execute(
            "SELECT tile_data FROM tiles "
            "WHERE zoom_level=? AND tile_column=? AND tile_row=?",
            key,
        ).fetchone()
        self.misses += 1
        if row is None:
            return None

        blob = row[0]
        with self._lock:
            self._hot[key] = blob
            self._hot.move_to_end(key)
            while len(self._hot) > self._hot_max:
                self._hot.popitem(last=False)
        return blob

    def hit_rate(self) -> float:
        total = self.hits + self.misses
        return self.hits / total if total else 0.0

Three details in that code are the difference between a reader that survives a field deployment and one that does not. immutable=1 is what allows the archive to live on a read-only mount without SQLite attempting to create a write-ahead log beside it; without it, the first query on a read-only filesystem fails with a disk I/O error that reads like corruption. The y-coordinate flip is the single most common bug in on-device tile serving, because it produces a map that is subtly wrong rather than obviously broken — tiles render, they are just from the wrong latitude. And the hot cache is deliberately keyed on the raw blob rather than the decoded geometry, so a tile that is re-requested during a pan costs nothing at all, not even a decompression.

Implementation 2: bounding the coverage before it ships #

The archive’s size is decided at provisioning time by a coverage policy, and getting that policy right is worth more than any runtime optimization. Tile count grows by roughly four per zoom level, so the difference between shipping to zoom 14 and zoom 16 is a factor of sixteen — usually the difference between an archive that fits and one that does not.

# coverage.py — estimate an archive before generating it.
# Pure arithmetic: run this at build time, not on the device.
import math

def tiles_in_bbox(min_lon, min_lat, max_lon, max_lat, z):
    """Number of XYZ tiles covering a bounding box at one zoom level."""
    def lon_to_x(lon):
        return int((lon + 180.0) / 360.0 * (1 << z))

    def lat_to_y(lat):
        rad = math.radians(lat)
        merc = math.log(math.tan(rad) + 1.0 / math.cos(rad))
        return int((1.0 - merc / math.pi) / 2.0 * (1 << z))

    x0, x1 = sorted((lon_to_x(min_lon), lon_to_x(max_lon)))
    y0, y1 = sorted((lat_to_y(min_lat), lat_to_y(max_lat)))
    return (x1 - x0 + 1) * (y1 - y0 + 1)


def archive_size_mb(bbox, z_min, z_max, mean_tile_kb=7.0):
    """Total archive size across a zoom range, in megabytes."""
    total = sum(tiles_in_bbox(*bbox, z) for z in range(z_min, z_max + 1))
    return total, total * mean_tile_kb / 1024.0


if __name__ == "__main__":
    corridor = (-122.55, 47.40, -122.20, 47.75)      # a metro working area
    for z_max in (13, 14, 15, 16):
        n, mb = archive_size_mb(corridor, 8, z_max)
        print(f"z8-z{z_max}: {n:>9,} tiles  {mb:8.1f} MB")

Run against a metro-scale working area, that script makes the trade explicit rather than theoretical: each additional zoom level roughly quadruples both the tile count and the archive size, and the useful maximum zoom is decided by what the operator needs to distinguish — a vehicle on a road needs zoom 14, a person next to a specific asset needs 17, and shipping 17 across a whole region instead of the few sites that need it is the most common cause of an oversized archive.

Configuration and tuning #

The knobs that matter are few, and most of them are set once at provisioning rather than tuned at runtime.

• mmap_size on the SQLite connection. A 64 MB window lets the kernel serve directory pages from the page cache without the reader managing anything; larger windows mostly buy nothing because the access pattern is sparse.
• Hot cache size. Set it from the viewport, not from available memory: two screens’ worth of tiles plus a margin. A 1080p viewport at zoom 14 is around 15 tiles, so 40–60 entries covers a pan in any direction without re-reading.
• Compression at rest. Vector tiles are already gzip-compressed inside the archive; storing them decompressed to save CPU is a losing trade on flash-bound hardware, and storing them double-compressed wastes cycles on every read.
• Zoom range per region. Ship a wide, shallow range for the whole operating area and a narrow, deep range for named sites. Two archives are easier to reason about and to update than one with an irregular coverage mask.
• PRAGMA query_only as a safety net. It turns any accidental write attempt into an immediate error at development time rather than a locked file in the field.

Updating an archive on a device that cannot be visited #

A basemap has a shelf life measured in years rather than days, which is fortunate, because replacing one on a remote gateway is the most bandwidth-expensive routine operation the device performs. Three approaches exist and they suit different fleets.

Whole-file replacement is the simplest and, for archives under a couple of hundred megabytes, usually the right answer. Write the new file beside the old one, verify its signature and size, fsync, then rename over the old path — an atomic operation on any POSIX filesystem — and reopen the reader. The device is never in a state where the archive is half-updated, and a failure at any point leaves the previous archive intact and serving. The cost is transferring the whole file even though almost none of it changed.

Additive overlays avoid the transfer entirely for the common case of adding coverage rather than correcting it. Ship a second, small archive covering the new area and have the reader consult archives in priority order, falling through to the base. This costs one extra lookup per miss and turns a 300 MB update into a 12 MB one. It accumulates: a device with nine overlays pays nine lookups for every tile outside all of them, so periodically collapsing overlays into a fresh base is part of the operational cycle rather than an optional cleanup.

Tile-level patches — shipping only the tiles that changed — sound optimal and rarely are. They require the device to write into the archive, which forfeits the read-only property that makes the archive safe on flash, and they require tracking which tiles changed between two generations of a source, which the tile pipeline usually cannot answer. Reserve this for the case where a small, known set of tiles is corrected frequently, and even then consider expressing it as a tiny overlay instead.

Bandwidth is only half the cost. An update also has to be scheduled, because a 314 MB transfer over a metered link is a month’s data allowance for the whole device, and pushing one during working hours can starve the telemetry the gateway exists to deliver. The workable pattern on a mixed fleet is to gate basemap updates on a connectivity class — deliver them only over an unmetered link the device recognises, such as a depot Wi-Fi network, and let cellular carry nothing but telemetry and configuration. Devices that never see an unmetered link get their basemap at the annual service visit on a USB stick, which is not a failure of the update system but the correct answer for that class of node.

Whichever mechanism is used, the update must be idempotent and interruptible. A device that loses power during a transfer should resume or restart without manual intervention, and applying the same update twice must be indistinguishable from applying it once. Both properties come free with whole-file replacement and have to be designed deliberately for the other two.

Verification and field diagnostics #

Tile problems are visual, which makes them easy to spot and hard to describe. Give the device the ability to answer the question itself:

1. Archive integrity at boot. Check the file’s size and the row count of the metadata table, and compare against values recorded at provisioning. A truncated archive from an interrupted update reads as “some tiles missing” rather than as a corrupt file, so a positive check is the only way to distinguish the two.
2. A coverage probe. A console command that takes a lat/lon and reports which zoom levels contain a tile for it. Field crews use this to answer “is the map missing here, or is the device lost?” without guessing.
3. Hit rate as a metric. Export the hot cache’s hit rate through the same registry as everything else — see monitoring and observability. A collapsing hit rate usually means the viewport is being driven by a position that is jumping, which is a receiver problem showing up as a storage symptom.
4. Read latency histogram. Bucket the pread times. On a healthy eMMC they cluster under a millisecond; a drift into the tens of milliseconds is the earliest available signal that the card is wearing out, long before write errors appear.

Failure modes specific to this pattern #

The last row is worth adopting as a default. A world basemap at zoom 0–6 costs a handful of megabytes and guarantees the viewport is never empty, which removes the single worst failure mode this system has — a screen that gives an operator no way to tell whether the device is working.

Related #

• PMTiles vs MBTiles for read-only edge basemaps — the container choice and what each one costs at read time.
• Serving vector tiles from a read-only SQLite cache — the local HTTP surface and its threading model.
• Pruning tile caches under a flash write budget — keeping a writable cache from wearing the card out.
• Fallback Routing & Offline Navigation — the routing graph that shares this storage budget.
• Device Constraints & Resource Limits — the flash, RAM and thermal envelope every archive has to fit inside.