Module arrow_tile

Expand description

Arrow-based tile payload format.

A tile’s payload is one or more layers. Each layer is a single Arrow RecordBatch serialised as an Arrow IPC stream. Geometry is encoded using the GeoArrow interleaved-coordinate convention so the payload can be consumed directly by GeoArrow-aware renderers (e.g. @geoarrow/deck.gl).

§Per-layer schema

column	type	notes
`id`	`UInt64`	feature id
`start_time`	`Int64`	Unix ms, absolute
`end_time`	`Int64`	Unix ms, absolute
`geometry`	GeoArrow point / linestring / polygon	interleaved f64 lon/lat
`vertex_time`	`List<UInt16>` deltas or `List<Int64>` (nullable)	per-vertex Unix ms (optional; see [`build_vertex_time_array`])
`vertex_value`	`List<Float32>` (nullable)	per-vertex scalar, e.g. SST (optional)
`triangles`	`List<UInt16>` or `List<UInt32>`	pre-baked earcut indices, feature-local (optional; polygon only)
`<property>`	`Float64` or `Dictionary<UInt16,Utf8>`	one column per property

All layers in one tile are concatenated with a tiny frame so a tile can carry, say, a linestring layer and a point layer side by side. Two frame shapes exist, selected by EncoderConfig::format_version (the payload side of the packed format’s manifest.formatVersion — the two are bumped in lockstep, see docs/spec/stt-packed-format.md §9):

v1 (format_version: 1, the 0.3.x wire shape — frozen, byte-identical):

[u16 layer_count | ALIGNED_FRAME_FLAG]
  repeated: [u16 name_len][name utf8][u32 ipc_len][pad to 8][ipc stream bytes]

The leading u16’s top bit (ALIGNED_FRAME_FLAG) marks the aligned frame: zero padding after each ipc_len places every Arrow IPC stream at an 8-byte boundary relative to the payload start, so readers can hand the stream to an Arrow implementation zero-copy (Arrow buffers are 8-byte aligned within a stream; the stream itself must start aligned for that to survive). The pad length is not stored — readers derive it as (8 - pos % 8) % 8 from the position after ipc_len. Frames without the flag (every archive written before the flag existed) carry no padding and decode exactly as before.

v2 (format_version: 2, default for new stt-build output): a sectioned frame that hoists each layer’s Arrow IPC schema message into a per-dataset template (referenced by blake3-128 hash, resolved through the manifest’s schemas table) so the per-tile schema tax disappears, and carries the per-tile-varying metadata in a compact TILE_META section instead of the schema:

u16  0xFFFF                    # v2 escape (see FRAME_V2_ESCAPE)
u8   frame_version = 2
u8   flags = 0                 # reserved, MUST be 0
u16  layer_count
per layer:
  u16  name_len, [name utf8]
  u8   ref_kind_core           # 0 = INLINE_SCHEMA_CORE section present;
                               # 1 = the next 16 bytes are the template hash
  [16] core template hash     # iff ref_kind_core == 1
  u8   ref_kind_props          # 0/1 as above; 2 = NO props sections at all
  [16] props template hash    # iff ref_kind_props == 1
  u8   section_count
  per section (TOC): u8 tag, u32 length     # at-rest bytes, pad excluded
  [pad to 8, derived]
  per section: [section bytes][pad to 8, derived]

Reserved columns (id/times/geometry/vertex_*/triangles) form the CORE batch; property columns form the PROPS batch with its own schema/template (emitted only when properties exist). Each *_BATCH section is the IPC stream’s tail — dictionary batch(es) + record batch + end-of-stream — and a reader materialises concat(template, tail) for a stock Arrow reader. Unknown section tags are skippable via the TOC. Rows are stable-sorted by start_time at encode (after id assignment), declared by TILE_META.sorted. The v2 escape is unreachable from the v1 writer: the v1 path caps layer_count below 0x7fff, so an aligned v1 frame can never start with 0xFFFF.

Structs§

AttrQuant: Per-numeric-property quantization affine: value = o + q * s, where o is the column minimum (the dequantization offset) and s the requested ground precision (the step). Reconstruction is lossy to ≤ s/2; the reader applies the identical math (tile.ts).
ColumnarLayer: One decoded/encodable tile layer.
DecodedLayer: A decoded tile layer: its name and the raw Arrow RecordBatch.
EncoderConfig: Resolved, explicit encoder settings — the values the tile encoder reads at encode time (coordinate + attribute quantization, vector grouping, the point-elevation fold, vertex-time precision).
QuantAffine: Per-layer coordinate-quantization affine. Coordinates ship as i32 grid indices; the decoder reconstructs lon = x0 + qx*sx, lat = y0 + qy*sy. sx/sy (degrees per quantum) are sized from a target ground precision in meters at the layer’s mid-latitude, so the worst-case error is ≤ half a quantum (~meters/2). This trades GeoArrow self-describing Float64 for size (coords are the dominant, near-incompressible column) and is opt-in.
TemplateCollector: Thread-safe encode-side sink for the schema templates a v2 build produces.
TemplateRegistry: Decode-side template lookup: blake3-128 hash → raw template bytes.
TileMeta: The per-tile-varying metadata a v2 frame carries in its TILE_META section instead of the (now dataset-constant, template-resident) Arrow schema metadata. Canonical serialization (design §4.3): JSON, keys sorted — field order below IS alphabetical and the qa map is a BTreeMap — no whitespace. Readers MUST ignore unknown keys (serde’s default here), so the section can evolve additively. Presence rules: a key is present iff the corresponding feature is (qa omits non-quantized columns entirely; t0 iff a start-time column exists; vt iff delta-encoded vertex_time; vb iff a value matrix).
VectorGroup: A build-time directive to fuse several scalar numeric properties into one GPU-ready interleaved PropertyColumn::Vector. The component order is the vector’s component order (e.g. ["qx","qy","qz","qw"] → instanceQuaternions). Applied at encode time (like the quantization maps), so the producer keeps emitting plain scalar columns and the renderer still binds the result zero-copy. See set_vector_groups.

Enums§

GeometryColumn: Geometry for one layer, grouped by kind. Every feature in a layer shares one kind — the tiler emits a separate layer per geometry type.
PropertyColumn: A property column. Values are per-feature and may be missing.
VectorElem: Leaf element type of a PropertyColumn::Vector — the GPU upload type the renderer binds the decoded child buffer as.

Constants§

ALIGNED_FRAME_FLAG: Top bit of the layer frame’s leading u16: marks an aligned frame whose IPC streams are padded to 8-byte boundaries (see the module docs). The remaining 15 bits carry the layer count, so a tile may have at most 0x7fff layers. Old frames never set this bit (layer counts are tiny).
DEFAULT_VERTEX_TIME_MAX_STEP_MS: Default ceiling (ms) on the u16-delta vertex_time quantization step.
FORMAT_VERSION_V1: The frame format this encoder emits when nothing opts in explicitly — v1, the frozen 0.3.x wire shape. stt-build opts into v2 explicitly (its --format-version default is 2); stt-serve and every other existing caller stays v1 without changes (design §7).
FORMAT_VERSION_V2: The sectioned template-referencing frame (packed formatVersion 2).
FRAME_V2_ESCAPE: Leading u16 of a v2 layer frame. Deliberately unreachable from the v1 writer: the v1 encoder rejects tiles whose count | ALIGNED_FRAME_FLAG would collide with this escape, so the two frame shapes are disjoint on their first two bytes. Manifest formatVersion remains the authoritative discriminator (design ★F6); this escape is defense-in-depth.
MIN_QUANTIZE_COORDS_M: Finest coordinate-quantization precision (meters) the world-anchored grid supports: the grid is anchored at (-180, -90) with a uniform step of meters / M_PER_DEG_LAT degrees, so the largest longitude index is 360 * M_PER_DEG_LAT / meters. Below this floor (≈ 0.0187 m, ~19 mm) that index exceeds i32::MAX and quantization would silently snap far-east coordinates to wrong locations — so finer precisions are rejected.
SECTION_CORE_BATCH: v2 section tag: CORE batch IPC tail (dict batches + record batch + EOS).
SECTION_INLINE_SCHEMA_CORE: v2 section tag: full IPC schema prefix for the CORE batch (self-contained mode, ref_kind == REF_KIND_INLINE).
SECTION_INLINE_SCHEMA_PROPS: v2 section tag: full IPC schema prefix for the PROPS batch (as 0x01).
SECTION_PROPS_BATCH: v2 section tag: PROPS batch IPC tail (as 0x03, props schema).
SECTION_TILE_META: v2 section tag: the canonical TileMeta JSON.
STT_QUANT_ATTR_META_KEY: Field-metadata key flagging a numeric property column that ships as fixed-point integer indices (smallest of UInt16/Int32) instead of Float64. Its value is the AttrQuant JSON (value = o + q*s). Sibling of STT_QUANT_META_KEY for geometry; lives on the property field, so a reader reconstructs Float64 the same way it does coordinates.
STT_QUANT_META_KEY: Schema-metadata key (on the geometry field) that flags a tile whose coordinates are fixed-point i32 grid indices rather than GeoArrow Float64 lon/lat. Its value is the QuantAffine JSON; absent ⇒ standard Float64.
TRIANGLES_METADATA_KEY: Schema-metadata key set on layers that carry pre-baked triangle indices.

Functions§

decode_layer: Decode a single-layer Arrow IPC stream into a RecordBatch.
decode_tile: Decode a full tile payload (the layer frame) into its layers.
decode_tile_with_templates: decode_tile with the dataset’s TemplateRegistry, so v2 frames can resolve their 16-byte template-hash references. v1 frames decode unchanged (the registry is simply unused).
encode_layer: Encode a single layer to an Arrow IPC stream.
encode_layer_quantized: encode_layer with optional fixed-point coordinate quantization.
encode_layer_with: encode_layer with a fully-explicit EncoderConfig — no process-wide globals are read. This is the concurrency- and multi-config-safe entry point (e.g. a dynamic server fronting several datasets with different settings).
encode_tile: Encode a full tile payload (one or more layers) with the layer frame.
encode_tile_quantized: encode_tile with optional fixed-point coordinate quantization applied to every layer (see encode_layer_quantized). quantize_m = None is byte-identical to encode_tile; the other encoder settings come from the process-wide globals.
encode_tile_with: encode_tile with a fully-explicit EncoderConfig — no process-wide globals are read. The concurrency- and multi-config-safe entry point a dynamic per-request tile server uses so each dataset/request encodes with its own settings without touching shared state.
format_version: The build-global frame format version.
frame_v2_template_refs: Walk ONLY a v2 frame’s header structure — escape/version/flags/count, per-layer name + schema ref_kinds (+ their 16-byte hashes), TOC-driven section skips; no Arrow decode, no section parse — and return every template hash the frame references. The packed-format validators use this to prove each referenced hash resolves in manifest.schemas without decoding tiles. Errors (never panics) on truncated/malformed headers.
is_frame_v2: Whether a tile payload carries the v2 sectioned frame (leading u16 is the FRAME_V2_ESCAPE). The manifest’s formatVersion remains authoritative (★F6) — this sniff is defense-in-depth for the payload level.
point_elevation_column: The current point-elevation column name (empty ⇒ disabled).
quantize_attrs: The current numeric-property quantization map (a clone).
quantize_attrs_auto: Whether automatic numeric-property quantization is enabled.
quantize_coords_m: The build-global quantization precision in meters, or None when off.
set_format_version: Set the build-global frame format version for every subsequent default encode_tile call. Errors (storing nothing) on anything but 1 or 2.
set_point_elevation_column: Set the build-global point-elevation column name (see above). Empty disables.
set_quantize_attrs: Replace the build-global numeric-property quantization map.
set_quantize_attrs_auto: Enable/disable automatic range-adaptive quantization of every otherwise-raw Float64 numeric property (see [QUANTIZE_ATTRS_AUTO]). Explicit precisions in set_quantize_attrs always win.
set_quantize_coords_m: Set the build-global coordinate quantization precision in meters for every subsequent default encode_tile call. <= 0 (the default) turns it off — coordinates stay Float64 GeoArrow. See encode_layer_quantized for the size/precision trade-off. Errors (storing nothing) for a positive precision below MIN_QUANTIZE_COORDS_M, which would overflow the world grid’s longitude index.
set_template_collector: Install (or clear, with None) the build-global TemplateCollector.
set_vector_groups: Replace the build-global vector-group list.
set_vertex_time_max_step_ms: Override the u16-delta vertex_time step ceiling for every subsequent encode_layer call. Values below 1 ms clamp to 1 (every layer with a span beyond u16 milliseconds then takes the exact List<Int64> path).
template_collector: The current build-global template collector, if any.
tessellate_polygon: Tessellate one polygon feature (a list of rings) using earcut. Returns the flat triangle index list — each triple of indices is one triangle, indices are LOCAL (relative to the start of the feature’s coordinate run, where the exterior ring sits first followed by every hole).
validate_quantize_coords_m: Validate a coordinate-quantization precision against the world grid’s MIN_QUANTIZE_COORDS_M floor. meters <= 0 (quantization off) passes. Public so config-building paths (EncoderConfig consumers like stt-serve) can fail fast at startup with the same error the global setter and the encode-time guard produce.
vector_groups: The current vector-group list (a clone).
vertex_time_max_step_ms: The currently configured u16-delta vertex_time step ceiling (ms).

Type Aliases§

Coord: A single coordinate pair (lon, lat) in WGS84 degrees.