Crate dre_core 

DRE core: turns a project directory into a resolved, validated project, and runs it.

Re-exports

pub use diag::Diagnostic;

pub use diag::Diagnostics;

pub use diag::Severity;

Modules

catalog

Live relation context for integrations. Descriptions are read from the database; DRE does not create or maintain upstream comments.

codes

Every diagnostic and error code DRE reports, in one registry.

coltypes

How dre validate --live compares a declared source column’s data_type with what the database returns (an Arrow type, as the source plugin delivers it). The comparison is loose: a declared SQL type matches a family of Arrow types (bigint ≈ Int64, varchar(20) ≈ Utf8), since plugins map database types to Arrow in their own ways. Documented in docs/sources.md.

config

Reading DRE’s YAML files into typed config.

constraints

Plugin version constraints: intersecting semver requirements and detecting contradictions.

dates

Calendar values for templates: run.date, run.now, date(), datetime() and the periods built from them. Everything is worked out at render time, so compiled SQL holds literals and a rerun with the same DRE_RUN_DATE renders the same text.

diag

Validation diagnostics: every problem found in one pass, with file and line where known.

dialect

What DRE knows about each source type’s SQL: how to quote identifiers and string literals, how unquoted names compare, and which functions reach outside the database. One table, so dre catalog, dre query and the integrations reading the catalogue agree.

document

The context of a document template: what core sends the document plugin (the html and pdf formats) in a format write request’s template, and the Jinja environment the plugin builds from it. Rendering happens in the plugin; core only runs the queries and sends this.

engine

The run engine’s parts around a Binding’s run (crate::run):

error

DRE’s error type at module boundaries: a registered Code (whose Kind decides the exit code and whether trying again can help), the message, and an optional hint at the fix.

for_each

Per-row outputs (for_each): the pieces that don’t need a run. The report’s queries run once; the driver query gives one row per copy of the output, and every other query of the output is split by the same key, so each copy gets only its own slice. See run.rs for the run itself.

fs

Writing DRE’s durable state: write_atomic for every file a later run, or another tool, reads back (run_results.json, the drift snapshot, the manifest, dre.lock, plugin and package metadata). A forced stop mid-write leaves the previous or the next complete file, never broken JSON. Report outputs aren’t state and are written as they are.

lock

dre.lock: the exact plugin versions and package commits a project resolved, committed alongside it.

lookups

Lookups: small tables kept as files under lookups/ (mapping tables, code lists), used in SQL through ref('name') without ever being written to the source database by the user.

manager

The plugin manager: a static JSON registry index of plugin packages, checksum-verified downloads, side-by-side versioned installs, and dre.lock.

manifest

The project manifest, <target>/manifest.json: everything a project declares, as the engine resolves it, for orchestrators, CI and other tools. It’s a deliberate projection of the loaded Project, not a dump: built offline (no connection, no profiles, no plugins), the same bytes for the same project and the same inputs on every OS, and free of secrets and machine-specific values.

message

The built-in message format: a Binding’s query results rendered through Jinja into a short headline, a title plus text in portable Markdown (dre_protocol::markdown).

no_data

on_empty: what an output does when its queries return no rows.

numbers

Number filters for every template: number, percent, signed, currency and compact, formatted for the run’s locale:.

occurrences

Schedule occurrences: when each schedules.yml entry fires within a window, and the exact command that runs each firing (dre schedule ls). Pure computation over the loaded project: no profiles, plugins, network or writes. Storing, diffing and dispatching occurrences is the caller’s business.

options

Plugin options: a format’s output: keys and a destination entry’s keys. Core owns only the keys every output shares (format, destination, template, extension, profile, path); everything else belongs to the plugin, which checks it (the protocol’s validate). A new format or destination therefore needs no change to core.

packages

Macro packages: reusable macros from another project (e.g. dre_utils), declared under packages: in the project’s root dependencies.yml or packages.yml:

parse

The parse pass: each Binding rendered without a database, as dbt’s parse finds ref() and source(), to learn every query’s sources and the connection it runs on. Its results feed selection (-s source:), the manifest, dre ls and dre validate, none of which connect; a run repeats it with its own inputs and then holds rendering to what it found.

plugins

Finding installed plugin packages.

preflight

Jinja pre-flight: syntax checks and static scans of var(), env_var() and run.* call sites. Nothing is rendered with live values here.

profiles

profiles.yml: connection definitions, kept outside the project.

project

Loading a DRE project directory into a resolved, validated project.

query

A single read-only query on a configured connection, for external integrations.

rate

Traffic control for deliveries: a destination’s rate: (100/minute, 5/second, 20000/hour), with an optional burst:. Deliveries are spaced evenly (60/minute sends one a second); burst: N lets up to N go back to back while keeping the same average. One pacer serves every worker of a run or release, so the rate holds per destination, whatever the concurrency.

release

Held deliveries and dre release: a maker/checker step between rendering and sending.

render

Runtime Jinja rendering: one environment per connection a Binding uses (its queries’ SQL) plus one for output paths and template values. Provides run.*, var(), env_var(), run_query(), columns(), ref(), source(), target.name, connection.*, destination.*, profile(), the calendar functions in crate::dates and every macro in macros/.

rows

Typed, streaming JSON row export for integrations. Decimals retain their exact text; columns carry Arrow types so consumers need not infer types from values or CSV.

run

The run path: turn a resolved project into delivered files.

run_results

run_results.json: what one run of one Binding did. A public, versioned contract (docs/run-results.schema.json is generated from these types): adding an optional field keeps SCHEMA_VERSION; removing, renaming or re-typing one, or changing what it means, bumps it (a minor DRE release before 1.0, a major one after).

runs

A Binding’s run folders, so overlapping and interrupted runs are safe:

saved

Saved query results: each query’s result as the exact Arrow IPC stream core received (<name>.arrows), plus an index (results.json). dre run --output-json <DIR> writes them beside the typed JSON; dre run --from-results <DIR> replays them to the format plugins without running any query. Types (decimals, timezones, nulls) stay exact, and the files open directly in DuckDB, Polars or pandas.

schedule

Schedule timings: their shape, and the checks shared by dre validate and dre schedule ls. Core never fires schedules; occurrences works out when they would.

secrets

Secret masking. The value of every environment variable named DRE_SECRET_*, and of every environment variable profiles.yml reads through env_var() into a secret field (one the plugin declares secret, or named like password, token, api_key), is replaced by ***** in everything DRE writes for people to read: the console, logs/dre.log, JSON events, run_results.json and the compiled SQL in target/compiled/. The SQL actually sent to the database is unchanged. mask_secrets: false in dre_project.yml turns masking off.

selector

Selector resolution, shared by dre validate (schedules.yml) and dre run.

settings

DRE’s settings and where each value came from.

sqlsplit

Statement splitting: lexing ;-separated SQL, not parsing it.

target

The target path: the folder DRE writes its generated files to (compiled SQL, run outputs, schema snapshots, run_results.json, the manifest). target/ in the project root unless set, highest first, by --target-path, DRE_TARGET_PATH or target_path: in dre_project.yml. It’s always a local (or mounted) path, never inside the project’s sources.

template_file

Where a template named in YAML is read from: an xlsx or html/pdf template.file, and the {% include %}/{% extends %} of a document template.

values

Arrow values as Jinja values, for run_query() rows and message results.

variables

Optional variable declarations, separate from vars so existing map values keep their meaning.

yaml

YAML loading with file/line locations for diagnostics.

Functions

dre_home

DRE’s home directory: ~/.dre.

slash

A path with / separators on every platform, so messages and goldens read the same everywhere.

version

The version string dre --version prints. DRE has no released version yet.