Expand description
Layer: Execution — MongoDB source engine (document store).
Unlike the three SQL engines (PostgreSQL / MySQL / SQL Server), MongoDB has
no SQL, no fixed per-collection schema, and no information_schema. So the
SQL-shaped read seam — chunked/keyset planning, incremental predicate
building, catalog introspection — does not apply. This adapter is the
OSS JSON-blob model: every document exports as exactly two columns —
_id (Utf8, the document key stringified: ObjectId → hex, etc.) and
document (Utf8 + the arrow.json extension type, the whole BSON
document rendered as relaxed extended JSON).
Per-field typing is punted downstream (PARSE_JSON in the warehouse). Schema
inference / auto-discovery is deliberately out of scope for OSS (it is the
paid-tier discover mechanism); the user gets a lossless blob, not a guessed
columnar schema. A future columns: projection can carve typed columns out of
the blob without changing this default.
§Sync bridge (ADR-0011)
The mongodb driver is async (tokio); the Source trait is sync
&mut self. Like the MSSQL/tiberius engine, each MongoSource owns a tokio
runtime and block_ons every driver call — no async leaks into the runner.
§Scope
mode: full only (incremental / chunked / time-window are SQL-runner concepts
and are refused with an actionable error). Within full mode, source.mongo. page_size opts into keyset (seek) paging on _id — bounded query time,
per-page parts, optional cross-run resume (resume: true, typed BSON
checkpoint), and parallel: N _id-range fan-out (any BSON _id). CDC
(change streams) rides the canonical ChangeStream seam and is added
separately.
Structs§
- Mongo
Session - A connected MongoDB session: the async client plus the tokio runtime that
drives it, so the sync
Sourcetrait canblock_onevery driver call (ADR-0011, mirrors MSSQL/tiberius). This is the one place that owns the async→sync bridge — the SDAM-needs-a-worker-thread runtime, the connect + ping handshake, the borrow dance — reused by the source, the harm / count probes, andrivet init. - Mongo
Source - MongoDB source over a
MongoSession, carrying the resolvedsource.mongo:read optionsexportapplies.