Skip to main content

Crate mbtiles

Crate mbtiles 

Source
Expand description

§mbtiles

Book docs.rs docs GitHub crates.io version CI build

A library to help tile servers like Martin work with MBTiles files. When using as a lib, you may want to disable default features (i.e. the unused “cli” feature).

This crate also has a small utility that allows users to interact with the *.mbtiles files from the command line. See tools documentation for more information.

§Writing tiles

There are two ways to write tiles, and both work with every schema:

  • Mbtiles::insert_tiles is async and writes one batch per transaction. Each tile is a separate statement, and each statement goes through sqlx’s worker thread. It suits small and incremental writes.

  • Mbtiles::bulk_write hands a closure a MbtilesBulkWriter, which writes synchronously on the connection’s raw rusqlite handle. It reuses its prepared statements, commits every 65,536 tiles (its batch_size), and turns off synchronous until it’s done when the file holds no tiles yet, since such a file is regenerated rather than recovered after an OS crash. It suits generating whole files. A TileDedup hint per tile tells it what it needs to hash:

    • Unknown compares bytes by their hash, like insert_tiles;
    • Unique skips the hash;
    • Key(k) marks tiles the caller already knows to be identical, so they’re hashed and stored once. A key reused for different bytes fails the write.

    In the dedup-id schema, Unknown tiles use the connection’s hash index of stored blobs, which insert_tiles uses too. Once that index is in use, hinted tiles go through it as well, so it never misses a blob.

The bulk writer is exclusive by design:

  • The closure never sees the raw connection, and the connection can’t be used for anything else until the closure returns.
  • The writer can only be created by bulk_write. The closure borrows it, so it can’t be leaked or kept past the call.
  • Returning Ok commits, and an error or a panic rolls back.
  • It takes the file’s write lock up front, so other connections wait.

Two things are left to the caller:

  • A failed write keeps the tiles of earlier periodic commits. Set writer.batch_size = None to make the write all or nothing; the journal then grows with the data until the single commit. writer.checkpoint() commits whenever you choose, e.g. to bound by time how much an interrupted write loses.
  • The closure runs on the calling thread, so call bulk_write from a blocking context, not from an async worker thread.
use mbtiles::{CopyDuplicateMode, MbtError, MbtType, Mbtiles, TileCoord, TileDedup};

mbt.bulk_write(conn, MbtType::Flat, CopyDuplicateMode::Abort, |writer| {
    writer.write(TileCoord::new_unchecked(0, 0, 0), b"tile bytes", TileDedup::Unique)?;
    Ok::<_, MbtError>(())
})
.await

§Performance

The table below shows 20,000 tiles of 200 bytes, a quarter of them identical, written to a new file. Each row is the median of all runs on an Intel i9-10885H with an SSD (ext4). insert_tiles gets batches of 1,000 tiles, as in martin cp. The bar shows throughput relative to the fastest case. Normalized files with a hash view write at the same speed as those without one.

SchemaMethodTimeTiles/svs insert_tilesThroughput
flatinsert_tiles1.05 s19 k1.0×███▏
insert_tiles, synchronous=OFF714 ms28 k1.5×████▌
bulk_write, Unknown135 ms148 k7.7×████████████████████████
bulk_write, Key/Unique147 ms136 k7.1×██████████████████████
flat-with-hashinsert_tiles1.31 s15 k1.0×██▌
insert_tiles, synchronous=OFF963 ms21 k1.4×███▍
bulk_write, Unknown203 ms99 k6.4×████████████████
bulk_write, Key/Unique183 ms109 k7.1×█████████████████▊
normalizedinsert_tiles1.93 s10 k1.0×█▋
insert_tiles, synchronous=OFF1.47 s14 k1.3×██▎
bulk_write, Unknown292 ms68 k6.6×███████████▏
bulk_write, Key/Unique275 ms73 k7.0×███████████▉
dedup-idinsert_tiles2.70 s7 k1.0×█▎
insert_tiles, synchronous=OFF2.32 s9 k1.2×█▍
bulk_write, Unknown347 ms58 k7.8×█████████▍
bulk_write, Key/Unique136 ms147 k19.8×███████████████████████▉
cacheinsert_tiles1.56 s13 k1.0×██▏
insert_tiles, synchronous=OFF1.10 s18 k1.4×███
bulk_write, Unknown390 ms51 k4.0×████████▍
bulk_write, Key/Unique392 ms51 k4.0×████████▎

Most of insert_tiles’ time goes to sending each statement to sqlx’s worker thread, not to syncing the disk: turning synchronous off gains only a little. Hints matter most for the dedup-id schema. With Unknown, it must hash every tile and look the hash up; with Key and Unique, it hashes nothing. Running cargo bench -p mbtiles --bench bulk prints this table.

§Development

Any changes to SQL commands require running of just prepare-sqlite. This will install cargo sqlx command if it is not already installed, and update the ./sqlx-data.json file.

§License

Licensed under either of

§Contribution

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.

Re-exports§

pub use sqlx;

Structs§

AggTilesHasher
Computes agg_tiles_hash from the tiles as they are written, the value calc_agg_tiles_hash would read back from the file.
CacheEntryMeta
Cache metadata attached to a tile when writing it with Mbtiles::set_cached.
CachedTile
A cached tile together with its cache metadata, as returned by Mbtiles::get_cached.
Mbtiles
A reference to an MBTiles file providing low-level database operations.
MbtilesBulkWriter
Inserts tiles with prepared statements reused across tiles, in large transactions. Only Mbtiles::bulk_write creates one, and it owns it.
MbtilesCache
Connection pool for using an .mbtiles file as a writable tile cache.
MbtilesCopier
MbtilesPool
Connection pool for concurrent read access to an MBTiles file.
Metadata
Tileset metadata combining MBTiles and TileJSON specifications.
TileCoord
A single tile address in the XYZ tiling scheme.
UnixSeconds
A point in time as whole seconds since the Unix epoch (1970-01-01 UTC).

Enums§

AggHashType
CopyDuplicateMode
CopyType
HashAlgorithm
The algorithm a file’s tile hashes and agg_tiles_hash are computed with, recorded in the HASH_ALGORITHM metadata key.
IntegrityCheckType
MbtError
MbtType
MbtTypeCli
MbtWarning
Problems found while opening a file that do not stop it from being read.
NormalizedSchema
Describes the naming convention used by a normalized MBTiles schema.
PackCompression
How pack compresses tiles before storing them in the MBTiles file.
PatchType
PatchTypeCli
TileDedup
What the caller knows about a tile’s bytes, so that deduplicating schemas store repeated bytes once without hashing what they need not.
TileScheme
Tile-coordinate scheme of an on-disk tile directory tree.
UpdateZoomType

Constants§

AGG_TILES_HASH
Metadata key for the aggregate tiles hash value
AGG_TILES_HASH_AFTER_APPLY
Metadata key for a diff file, describing the eventual AGG_TILES_HASH value of the resulting tileset once the diff is applied
AGG_TILES_HASH_BEFORE_APPLY
Metadata key for a diff file, describing the expected AGG_TILES_HASH value of the tileset to which the diff will be applied.
HASH_ALGORITHM
metadata key naming the algorithm used to hash tiles and compute AGG_TILES_HASH.

Functions§

action_with_rusqlite
Runs action on the connection’s raw rusqlite handle. A transaction action leaves open would silently swallow every later statement of conn, so it is rolled back and reported instead.
anonymous_mbtiles
Create an in memory, temporary mbtile connection with the given script
apply_patch
calc_agg_tiles_hash
Compute the hash of the combined tiles in the mbtiles file tiles table/view. This should work on all mbtiles files perf MBTiles specification.
check_metadata_edit
Rejects a key that is both set and deleted, as Mbtiles::edit_metadata does. Lets a caller fail before doing work that ends with that call.
compute_min_max_zoom
Compute min and max zoom levels from the tiles table
create_cache_tables
Create the tile-cache table and the standard tiles view (if they don’t already exist).
create_dedup_id_normalized_tables
create_flat_tables
create_flat_with_hash_tables
create_metadata_table
create_normalized_tables
create_schema
Execute a schema-init .sql script, decorating each statement to match the runtime DDL.
create_tiles_with_hash_view
detach_db
Execute DETACH DATABASE command
get_patch_type
Check if MBTiles has a table or a view named bsdiffraw or bsdiffrawgz with needed fields, and return the corresponding patch type. If missing, return PatchType::Whole
has_tiles_with_hash
Check if MBTiles has a table or a view named tiles_with_hash with needed fields
init_mbtiles_schema
invert_y_value
MBTiles uses a TMS (Tile Map Service) scheme for its tile coordinates (inverted along the Y axis). This function converts Y value between TMS tile coordinate to an XYZ tile coordinate.
is_cache_tables_type
Check if the database uses the tile-cache schema: a tile_cache table with zoom_level, tile_column, tile_row, fetched, expires, etag, and tile_data columns.
is_dedup_id_normalized_tables_type
Check if MBTiles has an alternative normalized schema with tiles_shallow + tiles_data tables using integer tile_data_id instead of text tile_id.
is_empty_database
Returns true if the database is empty (no tables/indexes/…)
is_flat_tables_type
is_flat_with_hash_tables_type
is_normalized_tables_type
pack
Packs a {z}/{x}/{y}.{ext} directory tree at input_directory into a flat MBTiles file at output_file, interpreting the directory layout with scheme and compressing tiles per compression.
parse_key_value
Parse a key=value pair as given to --set-meta. The value may itself contain =.
reset_db_settings
temp_named_mbtiles
Create a named, in memory, temporary mbtile connection with the given script
unpack
Unpacks the MBTiles file at input_file into a {z}/{x}/{y}.{ext} directory tree under output_directory, laying out the y coordinate according to scheme.

Type Aliases§

MbtResult
Tile