Crate mlt_core

Expand description

§`MapLibre Tile` (MLT) Rust library

The MapLibre Tile specification is mainly inspired by the Mapbox Vector Tile (MVT) specification, but has been redesigned from the ground up to address the challenges of rapidly growing geospatial data volumes and complex next-generation geospatial source formats as well as to leverage the capabilities of modern hardware and APIs. MLT is specifically designed for modern and next generation graphics APIs to enable high-performance processing and rendering of large (planet-scale) 2D and 2.5 basemaps.

In particular, MLT offers the following features:

Improved compression ratio: Up to 6x on large encoded tiles, based on a column oriented layout with recursively applied (custom) lightweight encodings. This leads to reduced latency, storage, and egress costs and, in particular, improved cache utilization.
Better decoding performance: Fast lightweight encodings which can be used in combination with SIMD/vectorization instructions.
Support for linear referencing and m-values: To efficiently support the upcoming next generation source formats such as Overture Maps (GeoParquet).
Support for 3D coordinates: i.e., elevation
Support for complex types: including nested properties, lists, and maps
Improved processing performance, based on storage and in-memory formats that are specifically designed for modern GL APIs, allowing for efficient processing on both CPU and GPU. The formats are designed to be loaded into GPU buffers with little or no additional processing.

📝 For a more in-depth exploration of MLT have a look at the following slides, watch this talk or read this paper by MLT inventor Markus Tremmel.

§Tile Structure

Top level structure of a tile is a sequence of layers, where each layer consists of (size, tag, data) tuples:

size: varint - size of the data block in bytes, including the size of the tag field
tag: varint - identifies the block type, e.g. 0x01 = feature table v1, 0x02 = raster layer, 0x03 = routing table, etc. We only define 0x01 for now.
data: u8[] - the actual data block of the specified size

This approach allows us to easily extend the format in the future by adding new block types, while keeping backward compatibility. Parsers can skip unknown block types by reading the size and moving forward accordingly. For now, we only define 0x01 for vector layers, and possibly a few more if needed.

Note the ordering: tag is after the size because it is possible to treat it as a single byte for now, until the parser supports more than 127 types. This allows the parser to efficiently skip unknown types without doing more expensive varint parsing.

§Layer 0x01 - MVT compatibility

Structure of the data if the tag above is 0x01. We should focus this tag on MVT compatibility, offering exactly what we had in MVT, but allowing for a clearly defined set of encodings and other optimizations like tessellation. No new data formats (per vertex data, nested data, 3d geometries, etc.). No extendable encodings - once finalized, 0x01 will only allow what has been specified. This will ensure that if a decoder declares “0x01” support, it will parse every specification-compliant 0x01 layer. For any new features and encodings we will simply use a new tag ID, likely reusing most of the existing encoding/decoding code.

name: string - Name of the layer
columnCount: varint - Number of columns in the layer
each column is defined as:
- columnType: varint - same idea as tag above, e.g. 1 = id, 2 = geometry, 3 = int property, etc.
- TODO…

See CONTRIBUTING.md for additional pipeline docs.

§Data Pipeline

The diagram below shows the full lifecycle of tile data — from raw bytes on the wire, through lazy parsing and optional per-column decoding, to zero-copy iteration or fully owned row-form access, and back to encoded bytes via the columnar encode pipeline.

flowchart TB
    A(["&[u8]  —  raw tile bytes"])

    subgraph DEC ["Decoding"]
        B["<b>Parser::parse_layers(&[u8])</b>
        <i>Create zero-copy views into input bytes
        for each layer with almost no memory allocations</i>"]

        C["Layer&lt;Lazy>
           Tag01(Layer01&lt;Lazy>) | Unknown
           columns = LazyParsed::Raw(RawStream)
           zero allocations for column data"]

        D["decode_all()  — or per-column:
           decode_id() / decode_geometry() / decode_properties()

           RawStream
           → physical codec: FastPFor · varint · byte-RLE
           → logical  codec: delta · zigzag · Morton · Hilbert
           → typed column buffers Vec&lt;T>"]

        E["ParsedLayer  =  Layer&lt;Parsed>
           all columns decoded into typed buffers"]
    end

    subgraph ACCESS ["Iterate  (zero-copy borrow)"]
        F["iter_features()  →  Layer01FeatureIter"]
        G["FeatureRef
           id: Option&lt;u64>
           geometry reference
           property iterator"]
    end

    H(["Layer01::into_tile()"])

    subgraph BRIDGE ["TileLayer01  (row-oriented, fully owned)"]
        I["Vec&lt;TileFeature>
           id:       Option&lt;u64>
           geometry: geo_types::Geometry&lt;i32>
           props:    Vec&lt;PropValue>"]
    end

    subgraph ENC ["Encoding"]
        J["StagedLayer01::from(TileLayer01)
           owned columnar form
           IdValues · GeometryValues · Vec&lt;StagedProperty>"]

        K["StagedLayer01::encode()
           / encode_auto() / encode_with_profile()
           per-column compression applied"]

        L["EncodedLayer01  (wire-ready)
           EncodedId · EncodedGeometry
           Vec&lt;EncodedProperty>  each = Vec&lt;EncodedStream>"]

        M["EncodedLayer::write_to()
           serialises: varint(size) + tag byte + column payloads"]
    end

    N(["Vec&lt;u8>  —  encoded tile bytes"])

    A --> B --> C --> D --> E
    E -->|"borrow"| F --> G
    E -->|"own all data"| H --> I
    I -->|"From&lt;TileLayer01>"| J --> K --> L --> M --> N

    classDef io fill:#1e5c3a,color:#e8f5e9,stroke:#0a3d22
    classDef bridge fill:#4a1c6b,color:#f3e5f5,stroke:#2d0b45
    classDef dec fill:#1a3a5c,color:#e3f2fd,stroke:#0d1f35
    classDef enc fill:#5c2a1a,color:#fbe9e7,stroke:#3d1510
    class A,N io
    class H,I bridge
    class B,C,D,E,F,G dec
    class J,K,L,M enc

Key types and their roles:

Type	Role
`Layer<Lazy>` / `Layer01<Lazy>`	Parsed frame with column byte slices still unprocessed; zero allocation beyond the parse pass
`LazyParsed<Raw, Parsed>`	Type-state wrapper: `Raw(RawStream)` before decoding, `Parsed(T)` after, `ParsingFailed` on error
`ParsedLayer` = `Layer<Parsed>`	All columns decoded; borrow-based iteration via `iter_features()`
`TileLayer01`	Row-oriented, fully owned bridge between decode and encode
`StagedLayer01`	Owned columnar data ready for compression/encoding
`EncodedLayer01`	Wire-ready columnar data; written by `write_to()` with size + tag prefix

§Tools

See the mlt tool for various ways to interact with the parser and decoder. This includes a terminal-based visualizer for exploring MLT files.

Modules§

encoder
geojson: GeoJSON -like data to represent decoded MLT data with i32 coordinates
logical
mvt: Convert MVT data to FeatureCollection or to TileLayer01

Structs§

Column: Column definition
ColumnRef: A single non-null property value for one feature, yielded by FeatureRef::iter_properties.
Decoder: Stateful decoder that enforces a per-tile memory budget during decoding.
FeatPropertyIter: Iterates the non-null properties of a feature as ColumnRef (name + value).
FeatValuesIter: Iterates every property slot of a feature as Option<PropValueRef>.
FeatureRef: A single map feature returned by Layer01FeatureIter.
GeometryValues: Parsed (decoded) geometry data
IdValues: Parsed ID values as a vector of optional 64-bit unsigned integers
IntEncoding
Layer01: Representation of an MLT feature table layer with tag 0x01 during decoding.
Layer01FeatureIter: Iterator over the features of a fully-decoded Layer01<Parsed>.
Layer01PropNamesIter: Iterates the column names of a Layer01’s properties, for any decode state.
Lazy: Lazy state: individual columns may still be raw or already decoded.
LogicalValue: Carries the stream metadata needed to perform the logical decode pass.
MortonMeta: Metadata for Morton decoding
Parsed: Fully-decoded state: all columns hold their parsed values directly.
ParsedScalar
ParsedSharedDict: Parsed shared dictionary payload shared by one or more child string properties.
ParsedSharedDictItem: A single sub-property within a shared dictionary parsed value.
ParsedStrings: Parsed string values for a single property.
Parser: Stateful parser that enforces a memory budget during parsing (binary → raw structures).
PropName: A zero-allocation two-part property name yielded by FeatureRef::iter_properties.
RawFsstData: Raw FSST-compressed data (4 streams) borrowed from input bytes.
RawGeometry: Raw geometry data as read directly from the tile (borrows from input bytes)
RawId: Unparsed ID data as read directly from the tile (borrows from input bytes)
RawPlainData: Raw plain data (length stream + data stream) borrowed from input bytes.
RawPresence: Raw presence/nullability stream borrowed from input bytes.
RawScalar: Raw scalar column (bool, integer, or float) as read directly from the tile.
RawSharedDict: Raw shared-dictionary column as read directly from the tile.
RawSharedDictItem: A single child field within a SharedDict raw column
RawStream: Representation of an encoded stream
RawStrings: Raw string column as read directly from the tile.
RleMeta: Metadata for RLE decoding TODO v2 optimizations:
StreamMeta: Metadata about an encoded stream
TileFeature: A single map feature in row form.
TileLayer01: Row-oriented working form for the optimizer.
Unknown: Unknown layer data, stored as encoded bytes

Enums§

ColumnType: Column data type, as stored in the tile
DictionaryType: Dictionary type used for a column, as stored in the tile
GeometryType: Types of geometries supported in MLT
Layer: A layer that can be one of the known types, or an unknown.
LazyParsed: Shared wrapper for values that may still be in the original (raw) format or already parsed (but still columnar). Used by: Id, Geometry, Property, and eventually - SharedDictItem
LengthType: Length type used for a column, as stored in the tile
LogicalEncoding: How should the stream be interpreted at the logical level (second pass of decoding)
LogicalTechnique: Logical encoding technique used for a column, as stored in the tile
MltError
OffsetType: Offset type used for a column, as stored in the tile
ParsedProperty: Parsed property values in a typed enum form.
PhysicalEncoding: Physical encoding used for a column, as stored in the tile
PropValue: A single typed value for one property of one feature.
PropValueRef: A borrowed, non-null per-feature property value.
RawIdValue: A sequence of raw ID values, either 32-bit or 64-bit unsigned integers
RawProperty: Raw property data as read directly from the tile.
RawSharedDictEncoding: Raw encoding payload for a SharedDict column.
RawStreamData
RawStringsEncoding: Raw encoding payload for a string column (plain, dictionary, or FSST variants).
StatType: What to calculate with Analyze::collect_statistic.
StreamType: How should the stream be interpreted at the physical level (first pass of decoding)

Traits§

Analyze: Trait for estimating various size/count metrics.
Decode
DecodeState: Type-state marker for Layer01 and related column wrappers.

Type Aliases§

Geometry: Geometry column representation, parameterized by decode state.
Id: ID column representation, parameterized by decode state.
MltRefResult
MltResult
ParsedLayer
ParsedLayer01
Property: Property column representation, parameterized by decode state.

Crate mlt_core

Crate mlt_core Copy item path

§MapLibre Tile (MLT) Rust library

§Tile Structure

§Layer 0x01 - MVT compatibility

§Data Pipeline

§Tools

Modules§

Structs§

Enums§

Traits§

Type Aliases§

Crate mlt_core

§`MapLibre Tile` (MLT) Rust library