Crate dre_core

Source

Expand description

DRE core: turns a project directory into a resolved, validated project, and runs it.

Re-exports§

pub use diag::Diagnostic;
pub use diag::Diagnostics;
pub use diag::Severity;

Modules§

catalog: Live relation context for integrations. Descriptions are read from the database; DRE does not create or maintain upstream comments.
codes: Every diagnostic and error code DRE reports, in one registry.
coltypes: How dre validate --live compares a declared source column’s data_type with what the database returns (an Arrow type, as the source plugin delivers it). The comparison is loose: a declared SQL type matches a family of Arrow types (bigint ≈ Int64, varchar(20) ≈ Utf8), since plugins map database types to Arrow in their own ways. Documented in docs/sources.md.
config: Reading DRE’s YAML files into typed config.
constraints: Plugin version constraints: intersecting semver requirements and detecting contradictions.
dates: Calendar values for templates: run.date, run.now, date(), datetime() and the periods built from them. Everything is worked out at render time, so compiled SQL holds literals and a rerun with the same DRE_RUN_DATE renders the same text.
diag: Validation diagnostics: every problem found in one pass, with file and line where known.
dialect: What DRE knows about each source type’s SQL: how to quote identifiers and string literals, how unquoted names compare, and which functions reach outside the database. One table, so dre catalog, dre query and the integrations reading the catalogue agree.
document: The context of a document template: what core sends the document plugin (the html and pdf formats) in a format write request’s template, and the Jinja environment the plugin builds from it. Rendering happens in the plugin; core only runs the queries and sends this.
engine: The run engine’s parts around a Binding’s run (crate::run):
error: DRE’s error type at module boundaries: a registered Code (whose Kind decides the exit code and whether trying again can help), the message, and an optional hint at the fix.
for_each: Per-row outputs (for_each): the pieces that don’t need a run. The report’s queries run once; the driver query gives one row per copy of the output, and every other query of the output is split by the same key, so each copy gets only its own slice. See run.rs for the run itself.
fs: Writing DRE’s durable state: write_atomic for every file a later run, or another tool, reads back (run_results.json, the drift snapshot, the manifest, dre.lock, plugin and package metadata). A forced stop mid-write leaves the previous or the next complete file, never broken JSON. Report outputs aren’t state and are written as they are.
lock: dre.lock: the exact plugin versions and package commits a project resolved, committed alongside it.
lookups: Lookups: small tables kept as files under lookups/ (mapping tables, code lists), used in SQL through ref('name') without ever being written to the source database by the user.
manager: The plugin manager: a static JSON registry index of plugin packages, checksum-verified downloads, side-by-side versioned installs, and dre.lock.
manifest: The project manifest, <target>/manifest.json: everything a project declares, as the engine resolves it, for orchestrators, CI and other tools. It’s a deliberate projection of the loaded Project, not a dump: built offline (no connection, no profiles, no plugins), the same bytes for the same project and the same inputs on every OS, and free of secrets and machine-specific values.
message: The built-in message format: a Binding’s query results rendered through Jinja into a short headline, a title plus text in portable Markdown (dre_protocol::markdown).
no_data: on_empty: what an output does when its queries return no rows.
numbers: Number filters for every template: number, percent, signed, currency and compact, formatted for the run’s locale:.
occurrences: Schedule occurrences: when each schedules.yml entry fires within a window, and the exact command that runs each firing (dre schedule ls). Pure computation over the loaded project: no profiles, plugins, network or writes. Storing, diffing and dispatching occurrences is the caller’s business.
options: Plugin options: a format’s output: keys and a destination entry’s keys. Core owns only the keys every output shares (format, destination, template, extension, profile, path); everything else belongs to the plugin, which checks it (the protocol’s validate). A new format or destination therefore needs no change to core.
packages: Macro packages: reusable macros from another project (e.g. dre_utils), declared under packages: in the project’s root dependencies.yml or packages.yml:
parse: The parse pass: each Binding rendered without a database, as dbt’s parse finds ref() and source(), to learn every query’s sources and the connection it runs on. Its results feed selection (-s source:), the manifest, dre ls and dre validate, none of which connect; a run repeats it with its own inputs and then holds rendering to what it found.
plugins: Finding installed plugin packages.
preflight: Jinja pre-flight: syntax checks and static scans of var(), env_var() and run.* call sites. Nothing is rendered with live values here.
profiles: profiles.yml: connection definitions, kept outside the project.
project: Loading a DRE project directory into a resolved, validated project.
query: A single read-only query on a configured connection, for external integrations.
rate: Traffic control for deliveries: a destination’s rate: (100/minute, 5/second, 20000/hour), with an optional burst:. Deliveries are spaced evenly (60/minute sends one a second); burst: N lets up to N go back to back while keeping the same average. One pacer serves every worker of a run or release, so the rate holds per destination, whatever the concurrency.
release: Held deliveries and dre release: a maker/checker step between rendering and sending.
render: Runtime Jinja rendering: one environment per connection a Binding uses (its queries’ SQL) plus one for output paths and template values. Provides run.*, var(), env_var(), run_query(), columns(), ref(), source(), target.name, connection.*, destination.*, profile(), the calendar functions in crate::dates and every macro in macros/.
rows: Typed, streaming JSON row export for integrations. Decimals retain their exact text; columns carry Arrow types so consumers need not infer types from values or CSV.
run: The run path: turn a resolved project into delivered files.
run_results: run_results.json: what one run of one Binding did. A public, versioned contract (docs/run-results.schema.json is generated from these types): adding an optional field keeps SCHEMA_VERSION; removing, renaming or re-typing one, or changing what it means, bumps it (a minor DRE release before 1.0, a major one after).
runs: A Binding’s run folders, so overlapping and interrupted runs are safe:
saved: Saved query results: each query’s result as the exact Arrow IPC stream core received (<name>.arrows), plus an index (results.json). dre run --output-json <DIR> writes them beside the typed JSON; dre run --from-results <DIR> replays them to the format plugins without running any query. Types (decimals, timezones, nulls) stay exact, and the files open directly in DuckDB, Polars or pandas.
schedule: Schedule timings: their shape, and the checks shared by dre validate and dre schedule ls. Core never fires schedules; occurrences works out when they would.
secret_columns: secret_columns: on a query entry: the values of those columns are secrets (a PDF or zip password). As a result arrives, its secret values join crate::secrets, so every surface that masks env_var() secrets masks them too: the console, dre.log, run_results.json, rows.jsonl, previews and errors. --output-json (which dre-mcp reads) gets the columns themselves replaced by *****. Delivered files and messages keep the real values.
secrets: Secret masking. The value of every environment variable named DRE_SECRET_*, and of every environment variable profiles.yml reads through env_var() into a secret field (one the plugin declares secret, or named like password, token, api_key), is replaced by ***** in everything DRE writes for people to read: the console, logs/dre.log, JSON events, run_results.json and the compiled SQL in target/compiled/. The SQL actually sent to the database is unchanged. mask_secrets: false in dre_project.yml turns masking off.
selector: Selector resolution, shared by dre validate (schedules.yml) and dre run.
settings: DRE’s settings and where each value came from.
sqlsplit: Statement splitting: lexing ;-separated SQL, not parsing it.
target: The target path: the folder DRE writes its generated files to (compiled SQL, run outputs, schema snapshots, run_results.json, the manifest). target/ in the project root unless set, highest first, by --target-path, DRE_TARGET_PATH or target_path: in dre_project.yml. It’s always a local (or mounted) path, never inside the project’s sources.
template_file: Where a template named in YAML is read from: an xlsx or html/pdf template.file, and the {% include %}/{% extends %} of a document template.
values: Arrow values as Jinja values, for run_query() rows and message results.
variables: Optional variable declarations, separate from vars so existing map values keep their meaning.
yaml: YAML loading with file/line locations for diagnostics.
zip_pack: zip: on a destination entry: the output’s files, plus every attach:ed output’s, packed flat into one zip during the run, optionally with a password (AES-256 by default, ZipCrypto on request). The zip is built in the run folder, so a held entry records it and its SHA-256, and dre release sends exactly those bytes.

Functions§

dre_home: DRE’s home directory: ~/.dre.
slash: A path with / separators on every platform, so messages and goldens read the same everywhere.
version: The version string dre --version prints. DRE has no released version yet.

Crate dre_core

Crate dre_core Copy item path

Re-exports§

Modules§

Functions§

Crate dre_core