Skip to main content

Module mongo

Module mongo 

Source
Expand description

Layer: Execution — MongoDB source engine (document store).

Unlike the three SQL engines (PostgreSQL / MySQL / SQL Server), MongoDB has no SQL, no fixed per-collection schema, and no information_schema. So the SQL-shaped read seam — chunked/keyset planning, incremental predicate building, catalog introspection — does not apply. This adapter is the OSS JSON-blob model: every document exports as exactly two columns — _id (Utf8, the document key stringified: ObjectId → hex, etc.) and document (Utf8 + the arrow.json extension type, the whole BSON document rendered as relaxed extended JSON).

Per-field typing is punted downstream (PARSE_JSON in the warehouse). Schema inference / auto-discovery is deliberately out of scope for OSS (it is the paid-tier discover mechanism); the user gets a lossless blob, not a guessed columnar schema. A future columns: projection can carve typed columns out of the blob without changing this default.

§Sync bridge (ADR-0011)

The mongodb driver is async (tokio); the Source trait is sync &mut self. Like the MSSQL/tiberius engine, each MongoSource owns a tokio runtime and block_ons every driver call — no async leaks into the runner.

§Scope

mode: full only (incremental / chunked / time-window are SQL-runner concepts and are refused with an actionable error). Within full mode, source.mongo. page_size opts into keyset (seek) paging on _id — bounded query time, per-page parts, optional cross-run resume (resume: true, typed BSON checkpoint), and parallel: N _id-range fan-out (any BSON _id). CDC (change streams) rides the canonical ChangeStream seam and is added separately.

Structs§

MongoSession
A connected MongoDB session: the async client plus the tokio runtime that drives it, so the sync Source trait can block_on every driver call (ADR-0011, mirrors MSSQL/tiberius). This is the one place that owns the async→sync bridge — the SDAM-needs-a-worker-thread runtime, the connect + ping handshake, the borrow dance — reused by the source, the harm / count probes, and rivet init.
MongoSource
MongoDB source over a MongoSession, carrying the resolved source.mongo: read options export applies.