Expand description
§mbtiles
A library to help tile servers like Martin work with MBTiles files. When using as a lib, you may want to disable default features (i.e. the unused “cli” feature).
This crate also has a small utility that allows users to interact with the *.mbtiles files from the command line. See tools documentation for more information.
§Writing tiles
There are two ways to write tiles, and both work with every schema:
-
Mbtiles::insert_tilesis async and writes one batch per transaction. Each tile is a separate statement, and each statement goes through sqlx’s worker thread. It suits small and incremental writes. -
Mbtiles::bulk_writehands a closure aMbtilesBulkWriter, which writes synchronously on the connection’s rawrusqlitehandle. It reuses its prepared statements, commits every 65,536 tiles (itsbatch_size), and turns offsynchronousuntil it’s done when the file holds no tiles yet, since such a file is regenerated rather than recovered after an OS crash. It suits generating whole files. ATileDeduphint per tile tells it what it needs to hash:Unknowncompares bytes by their hash, likeinsert_tiles;Uniqueskips the hash;Key(k)marks tiles the caller already knows to be identical, so they’re hashed and stored once. A key reused for different bytes fails the write.
In the dedup-id schema,
Unknowntiles use the connection’s hash index of stored blobs, whichinsert_tilesuses too. Once that index is in use, hinted tiles go through it as well, so it never misses a blob.
The bulk writer is exclusive by design:
- The closure never sees the raw connection, and the connection can’t be used for anything else until the closure returns.
- The writer can only be created by
bulk_write. The closure borrows it, so it can’t be leaked or kept past the call. - Returning
Okcommits, and an error or a panic rolls back. - It takes the file’s write lock up front, so other connections wait.
Two things are left to the caller:
- A failed write keeps the tiles of earlier periodic commits. Set
writer.batch_size = Noneto make the write all or nothing; the journal then grows with the data until the single commit.writer.checkpoint()commits whenever you choose, e.g. to bound by time how much an interrupted write loses. - The closure runs on the calling thread, so call
bulk_writefrom a blocking context, not from an async worker thread.
use mbtiles::{CopyDuplicateMode, MbtError, MbtType, Mbtiles, TileCoord, TileDedup};
mbt.bulk_write(conn, MbtType::Flat, CopyDuplicateMode::Abort, |writer| {
writer.write(TileCoord::new_unchecked(0, 0, 0), b"tile bytes", TileDedup::Unique)?;
Ok::<_, MbtError>(())
})
.await§Performance
The table below shows 20,000 tiles of 200 bytes, a quarter of them identical, written to a new file. Each row is the median of all runs on an Intel i9-10885H with an SSD (ext4). insert_tiles gets batches of 1,000 tiles, as in martin cp. The bar shows throughput relative to the fastest case. Normalized files with a hash view write at the same speed as those without one.
| Schema | Method | Time | Tiles/s | vs insert_tiles | Throughput |
|---|---|---|---|---|---|
| flat | insert_tiles | 1.05 s | 19 k | 1.0× | ███▏ |
insert_tiles, synchronous=OFF | 714 ms | 28 k | 1.5× | ████▌ | |
bulk_write, Unknown | 135 ms | 148 k | 7.7× | ████████████████████████ | |
bulk_write, Key/Unique | 147 ms | 136 k | 7.1× | ██████████████████████ | |
| flat-with-hash | insert_tiles | 1.31 s | 15 k | 1.0× | ██▌ |
insert_tiles, synchronous=OFF | 963 ms | 21 k | 1.4× | ███▍ | |
bulk_write, Unknown | 203 ms | 99 k | 6.4× | ████████████████ | |
bulk_write, Key/Unique | 183 ms | 109 k | 7.1× | █████████████████▊ | |
| normalized | insert_tiles | 1.93 s | 10 k | 1.0× | █▋ |
insert_tiles, synchronous=OFF | 1.47 s | 14 k | 1.3× | ██▎ | |
bulk_write, Unknown | 292 ms | 68 k | 6.6× | ███████████▏ | |
bulk_write, Key/Unique | 275 ms | 73 k | 7.0× | ███████████▉ | |
| dedup-id | insert_tiles | 2.70 s | 7 k | 1.0× | █▎ |
insert_tiles, synchronous=OFF | 2.32 s | 9 k | 1.2× | █▍ | |
bulk_write, Unknown | 347 ms | 58 k | 7.8× | █████████▍ | |
bulk_write, Key/Unique | 136 ms | 147 k | 19.8× | ███████████████████████▉ | |
| cache | insert_tiles | 1.56 s | 13 k | 1.0× | ██▏ |
insert_tiles, synchronous=OFF | 1.10 s | 18 k | 1.4× | ███ | |
bulk_write, Unknown | 390 ms | 51 k | 4.0× | ████████▍ | |
bulk_write, Key/Unique | 392 ms | 51 k | 4.0× | ████████▎ |
Most of insert_tiles’ time goes to sending each statement to sqlx’s worker thread, not to syncing the disk: turning synchronous off gains only a little. Hints matter most for the dedup-id schema. With Unknown, it must hash every tile and look the hash up; with Key and Unique, it hashes nothing. Running cargo bench -p mbtiles --bench bulk prints this table.
§Development
Any changes to SQL commands require running of just prepare-sqlite. This will install cargo sqlx command if it is not already installed, and update the ./sqlx-data.json file.
§License
Licensed under either of
- Apache License, Version 2.0 (LICENSE-APACHE or http://www.apache.org/licenses/LICENSE-2.0)
- MIT license (LICENSE-MIT or http://opensource.org/licenses/MIT) at your option.
§Contribution
Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.
Re-exports§
pub use sqlx;
Structs§
- AggTiles
Hasher - Computes
agg_tiles_hashfrom the tiles as they are written, the valuecalc_agg_tiles_hashwould read back from the file. - Cache
Entry Meta - Cache metadata attached to a tile when writing it with
Mbtiles::set_cached. - Cached
Tile - A cached tile together with its cache metadata, as returned by
Mbtiles::get_cached. - Mbtiles
- A reference to an
MBTilesfile providing low-level database operations. - Mbtiles
Bulk Writer - Inserts tiles with prepared statements reused across tiles, in large transactions. Only
Mbtiles::bulk_writecreates one, and it owns it. - Mbtiles
Cache - Connection pool for using an
.mbtilesfile as a writable tile cache. - Mbtiles
Copier - Mbtiles
Pool - Connection pool for concurrent read access to an
MBTilesfile. - Metadata
- Tileset metadata combining MBTiles and TileJSON specifications.
- Tile
Coord - A single tile address in the XYZ tiling scheme.
- Unix
Seconds - A point in time as whole seconds since the Unix epoch (1970-01-01 UTC).
Enums§
- AggHash
Type - Copy
Duplicate Mode - Copy
Type - Hash
Algorithm - The algorithm a file’s tile hashes and
agg_tiles_hashare computed with, recorded in theHASH_ALGORITHMmetadata key. - Integrity
Check Type - MbtError
- MbtType
- MbtType
Cli - MbtWarning
- Problems found while opening a file that do not stop it from being read.
- Normalized
Schema - Describes the naming convention used by a normalized
MBTilesschema. - Pack
Compression - How
packcompresses tiles before storing them in theMBTilesfile. - Patch
Type - Patch
Type Cli - Tile
Dedup - What the caller knows about a tile’s bytes, so that deduplicating schemas store repeated bytes once without hashing what they need not.
- Tile
Scheme - Tile-coordinate scheme of an on-disk tile directory tree.
- Update
Zoom Type
Constants§
- AGG_
TILES_ HASH - Metadata key for the aggregate tiles hash value
- AGG_
TILES_ HASH_ AFTER_ APPLY - Metadata key for a diff file, describing the eventual
AGG_TILES_HASHvalue of the resulting tileset once the diff is applied - AGG_
TILES_ HASH_ BEFORE_ APPLY - Metadata key for a diff file, describing the expected
AGG_TILES_HASHvalue of the tileset to which the diff will be applied. - HASH_
ALGORITHM - metadata key naming the algorithm used to hash tiles and compute
AGG_TILES_HASH.
Functions§
- action_
with_ rusqlite - Runs
actionon the connection’s raw rusqlite handle. A transactionactionleaves open would silently swallow every later statement ofconn, so it is rolled back and reported instead. - anonymous_
mbtiles - Create an in memory, temporary mbtile connection with the given
script - apply_
patch - calc_
agg_ tiles_ hash - Compute the hash of the combined tiles in the mbtiles file tiles table/view.
This should work on all mbtiles files perf
MBTilesspecification. - check_
metadata_ edit - Rejects a key that is both set and deleted, as
Mbtiles::edit_metadatadoes. Lets a caller fail before doing work that ends with that call. - compute_
min_ max_ zoom - Compute min and max zoom levels from the
tilestable - create_
cache_ tables - Create the tile-cache table and the standard
tilesview (if they don’t already exist). - create_
dedup_ id_ normalized_ tables - create_
flat_ tables - create_
flat_ with_ hash_ tables - create_
metadata_ table - create_
normalized_ tables - create_
schema - Execute a schema-init
.sqlscript, decorating each statement to match the runtime DDL. - create_
tiles_ with_ hash_ view - detach_
db - Execute
DETACH DATABASEcommand - get_
patch_ type - Check if
MBTileshas a table or a view namedbsdiffraworbsdiffrawgzwith needed fields, and return the corresponding patch type. If missing, returnPatchType::Whole - has_
tiles_ with_ hash - Check if
MBTileshas a table or a view namedtiles_with_hashwith needed fields - init_
mbtiles_ schema - invert_
y_ value MBTilesuses a TMS (Tile Map Service) scheme for its tile coordinates (inverted along the Y axis). This function converts Y value between TMS tile coordinate to an XYZ tile coordinate.- is_
cache_ tables_ type - Check if the database uses the tile-cache schema: a
tile_cachetable withzoom_level,tile_column,tile_row,fetched,expires,etag, andtile_datacolumns. - is_
dedup_ id_ normalized_ tables_ type - Check if
MBTileshas an alternative normalized schema withtiles_shallow+tiles_datatables using integertile_data_idinstead of texttile_id. - is_
empty_ database - Returns true if the database is empty (no tables/indexes/…)
- is_
flat_ tables_ type - is_
flat_ with_ hash_ tables_ type - is_
normalized_ tables_ type - pack
- Packs a
{z}/{x}/{y}.{ext}directory tree atinput_directoryinto a flatMBTilesfile atoutput_file, interpreting the directory layout withschemeand compressing tiles percompression. - parse_
key_ value - Parse a
key=valuepair as given to--set-meta. The value may itself contain=. - reset_
db_ settings - temp_
named_ mbtiles - Create a named, in memory, temporary mbtile connection with the given
script - unpack
- Unpacks the
MBTilesfile atinput_fileinto a{z}/{x}/{y}.{ext}directory tree underoutput_directory, laying out theycoordinate according toscheme.