Edge Geospatial IoT Gateway Processing

Choosing decimal places for field accuracy budgets

The number of decimal places a deployment stores is usually inherited from whatever the receiver happened to print, and it should be derived from an accuracy budget instead. This guide builds that budget — receiver error, mounting error, timing error and quantisation error, combined properly — and reads the storage precision off the result. It sits under spatial data precision standards in the Core Edge GIS Fundamentals guide.

The four error terms

Position error on a field device is not one number. Four independent terms combine, and knowing which dominates tells you where extra precision is wasted.

Receiver error is the largest term on uncorrected GNSS: 2–4 m for a clear sky view, 8–25 m in an urban canyon, unbounded under a metal roof. It is what HDOP times the user-equivalent range error estimates, and the receiver usually reports it.

Mounting error is the fixed offset between the antenna and the thing whose position is being reported. On a vehicle that is the distance from the roof antenna to the coupling point — often 3–6 m, and constant, which makes it correctable and frequently uncorrected.

Timing error is velocity times latency. A fix timestamped when it was computed rather than when it was sampled, on an asset moving at 25 m/s, is 2.5 m out for every 100 ms of delay. It is the term most often forgotten and the easiest to reduce.

Quantisation error is what storage precision contributes: half a step, uniformly distributed. At six decimal places that is 5.5 cm at the equator, and it is the only term the decimal-places decision affects.

Four error terms combined in quadrature for three deployment profiles Three profiles with their four error terms. An open-sky vehicle fleet has 2.8 metres of receiver error, 4 metres of uncorrected mounting offset, 1.2 metres of timing error at 100 milliseconds and 25 metres per second, and 0.06 metres of quantisation at six decimal places, combining to 5.0 metres. An urban fleet has 12 metres of receiver error dominating everything, combining to 12.7. An RTK-corrected survey rover has 0.02 metres of receiver error, no mounting offset, 0.01 of timing error and 0.06 of quantisation, so the quantisation dominates and seven decimal places are required. Errors add in quadrature — the largest term dominates and the rest barely matter receivermountingtimingquantisationcombined open-sky vehicles 6 decimal places 2.8 m4.0 m1.2 m0.06 m5.0 m urban canyon 5 decimal places is enough 12.0 m4.0 m1.2 m0.06 m12.7 m RTK survey rover 7 decimal places required 0.02 m0.00 m0.01 m0.06 m0.07 m
In the top two rows quantisation contributes nothing measurable. In the bottom row it is the dominant term, which is the only situation where a seventh decimal place is not waste.

Combining the terms

Independent errors combine in quadrature — the square root of the sum of squares — not by addition. That is why a term four times smaller than the largest contributes about 3% to the total, and why chasing it is almost always the wrong optimisation.

# accuracy_budget.py — combine the error terms and read off the precision.
# Run at design time, per deployment profile. Nothing here runs on the device.
import math

# Ground metres per degree of latitude; longitude shrinks with cos(lat).
M_PER_DEG_LAT = 111_320.0


def quantisation_error_m(decimals: int, lat_deg: float = 0.0) -> float:
    """Half a step, the worst case for a uniformly distributed rounding error.
    Latitude is the tighter axis; longitude is looser by 1/cos(lat)."""
    step_deg = 10.0 ** -decimals
    return step_deg * M_PER_DEG_LAT * 0.5


def combined_error_m(receiver_m: float, mounting_m: float,
                     velocity_ms: float, latency_s: float,
                     decimals: int) -> dict:
    timing = velocity_ms * latency_s
    quant = quantisation_error_m(decimals)
    total = math.sqrt(receiver_m ** 2 + mounting_m ** 2
                      + timing ** 2 + quant ** 2)
    return {
        "receiver": receiver_m,
        "mounting": mounting_m,
        "timing": timing,
        "quantisation": quant,
        "total": total,
        # A term contributing under 5% of the total is not worth reducing.
        "quantisation_share_pct": 100.0 * (quant ** 2) / (total ** 2),
    }


def sufficient_decimals(receiver_m: float, mounting_m: float,
                        velocity_ms: float, latency_s: float,
                        max_share_pct: float = 5.0) -> int:
    """The smallest precision whose quantisation contributes less than
    `max_share_pct` of the total error variance."""
    for d in range(3, 9):
        b = combined_error_m(receiver_m, mounting_m, velocity_ms, latency_s, d)
        if b["quantisation_share_pct"] < max_share_pct:
            return d
    return 8


if __name__ == "__main__":
    profiles = {
        "open-sky vehicle": (2.8, 4.0, 25.0, 0.05),
        "urban canyon":     (12.0, 4.0, 12.0, 0.10),
        "RTK rover":        (0.02, 0.0, 1.0, 0.01),
        "walking survey":   (3.0, 0.3, 1.4, 0.20),
    }
    for name, args in profiles.items():
        d = sufficient_decimals(*args)
        b = combined_error_m(*args, d)
        print(f"{name:<18} {d} places  total {b['total']:6.2f} m  "
              f"quant {b['quantisation']:.3f} m ({b['quantisation_share_pct']:.1f}%)")

The max_share_pct threshold is the judgement call and 5% is a defensible default: below it, halving the quantisation error changes the combined figure by less than a percent, which is smaller than the run-to-run variation in the receiver’s own performance.

What each place actually costs

Precision is not free in three separate places, and it is worth seeing all three together before adding a digit.

On the wire, in a fixed-width binary encoding, an extra decimal place costs nothing until it crosses a type boundary: microdegrees fit int32, and 10⁻⁷ degrees does not fit comfortably for longitude, so the seventh place forces int64 and doubles the coordinate field.

In the delta stream, precision costs directly. Each extra place multiplies the magnitude of every delta by ten, which adds roughly one byte per varint per coordinate. On a 1 Hz stream that is real money.

In storage and indexing, higher precision means fewer duplicate positions, which sounds good and means a stationary asset produces distinct points instead of collapsing — increasing the record count that dead-banding was meant to reduce.

Storage and wire cost per decimal place Five precisions compared. Five decimal places give 1.1 metre resolution, fit int32 with room to spare, and average 1.1 varint bytes per delta. Six give 11 centimetres, fit int32 comfortably, and average 1.9 bytes. Seven give 1.1 centimetres, no longer fit int32 for longitude so the field doubles to int64, and average 2.8 bytes. Eight give 1.1 millimetres, need int64, and average 3.7 bytes. A note records that the receiver's own error is 2.8 metres in the reference profile, so everything past five places is describing noise. Cost per place, against a receiver whose own error is 2.8 m resolutioninteger typevarint bytes per delta 5 places 1.1 mint321.1 6 places 11 cmint321.9 7 places 1.1 cmint64 — field doubles2.8 8 places 1.1 mmint643.7 Everything below the second row describes receiver noise at three times the byte cost.
The type boundary between the second and third rows is the largest single step in this table, and it is invisible until someone tries to store the seventh place.

Constraint validation

Constraint Expected impact How the budget addresses it
Bandwidth Each place adds roughly a byte per delta per coordinate Precision chosen from the error budget, not from the receiver’s output format
Storage Higher precision defeats duplicate collapsing Dead band sized above the quantisation step so stationary assets still collapse
Integer width The seventh place forces int64 for longitude Made explicit in the cost table before it becomes a schema migration
Correctness Under-precision loses real signal on corrected receivers The 5% variance-share rule catches the RTK case, where quantisation dominates
Comparability Two devices at different precisions produce incomparable data The chosen precision travels in the payload as a field

Gotchas and edge cases

  • Longitude is not latitude. A degree of longitude is 111 km at the equator and 78 km at 45° north; the same decimal place is a different ground distance. Budget against latitude, the tighter axis, and accept that longitude precision is looser away from the equator.
  • Precision is not a quality signal. A record with eight decimal places from a consumer receiver is not more trustworthy than one with five — it is the same measurement with more digits. Consumers who infer quality from digit count will do so; the accuracy field is what should carry that information.
  • Changing precision mid-fleet needs a schema field. Two firmware versions storing different precisions into one dataset produce apparent movement where a device was upgraded. Carry the precision in the payload so a consumer can normalise rather than discover it.
  • Rounding, not truncation. Truncating decimal places introduces a systematic bias toward the origin — every value moves the same direction. Rounding is unbiased, and the difference over millions of points is a measurable offset in aggregate statistics.
  • The budget changes with the deployment, not the hardware. The same device on a rooftop and in a warehouse has receiver errors an order of magnitude apart. Where a fleet spans both, budget for the better case and let the accuracy field carry the truth per fix.

Writing it down

Record the chosen precision, the profile it was derived from, and the four input terms in the deployment document — not in a constant in the firmware. Two years later, when someone asks why the data has 11 cm resolution, the answer should be a paragraph naming a receiver, a mounting, a latency and a decision, rather than an archaeology exercise through a git history.

Re-derive it when any of the four terms changes materially: a receiver upgrade, an antenna relocation, a change in reporting latency, or a deployment into a genuinely different environment. Each of those moves the dominant term, and the precision that was correct for the old dominant term is either wasteful or lossy for the new one.

Reducing the dominant term instead

Once the budget is written down it usually shows that the dominant term is not the one anyone was working on. Three of the four terms can be reduced, often cheaply, and doing so is worth more than any storage decision.

Mounting offset is correctable and usually uncorrected. Measure the vector from the antenna to the reference point once, and subtract it using the asset’s heading. On a vehicle fleet with a 4 m offset that removes the second-largest term for the cost of a heading source the device usually already has.

Timing error is a software fix. Timestamp the fix at the receiver’s own epoch — most receivers report the time of the position solution, not the time the sentence was written — and carry that through unchanged. A pipeline that stamps arrival time instead adds velocity times the parsing and queueing delay to every fix, silently.

Receiver error responds to the antenna. Moving from a patch antenna inside an enclosure to an external ground-plane antenna routinely halves the error, and it is the only lever that helps under a poor sky view. It costs a part and a hole in the housing.

The same profile before and after reducing the two correctable terms The open-sky vehicle profile has a combined error of 5.0 metres, made of 2.8 metres of receiver error, 4 metres of mounting offset, 1.2 metres of timing error and 0.06 of quantisation. Correcting the mounting offset with the heading and stamping fixes at the receiver's own epoch removes the second and third terms almost entirely, leaving 2.8 metres dominated by the receiver alone — a forty-four percent improvement with no hardware change and no extra bytes. Two software fixes, 44% less error, zero extra bytes before receiver 2.8mounting 4.0timing 1.2 combined 5.0 m after receiver 2.8mounting 0.3 · timing 0.15 combined 2.8 m Nothing about the storage precision changed. Two terms that were never measured turned out to be the majority of the error.
Writing the budget down is what makes this visible. Without it, the mounting offset is a constant nobody wrote down and the timestamp is whatever the parser assigned.

Related in Spatial Data Precision Standards