# readcon-db
**Mmap-backed CON/convel corpus store** (LMDB via [Heed](https://github.com/meilisearch/heed)), **non-SQL selection**, **xxHash3-128 exact match**, and **Rust / C / C++ / Python / Fortran** bindings.
Part of the **readcon ecosystem** with [`readcon-core`](https://github.com/lode-org/readcon-core) (Python package **`readcon`**):
| Crate / package | Role | Docs |
|-----------------|------|------|
| **[readcon-core](https://github.com/lode-org/readcon-core)** / **`readcon`** | CON interchange (parse/write/spec v2–v3). **XYZ/PDB/GRO → `ConFrame` via chemfiles** (`read_chemfiles*`), not ASE. Optional `to_ase` only for calculators. | Core README, `docs/orgmode/` |
| **readcon-db** / **`readcon_db`** (this repo) | mmap CON corpus: indexes (natoms, symbols, **energy range**, **forces/velocities/energy flags**), multi-reader, dedup. Blobs are **CON text** decoded with readcon-core. | [docs](https://lode-org.github.io/readcon-db/docs/), [`docs/orgmode/explanation/design.org`](docs/orgmode/explanation/design.org) |
ASE is not on the CON or XYZ read path. Optional `to_ase` is calculator hand-off only.
## Install
```bash
cargo add readcon-db
cargo install readcon-db --locked # CLI
pip install readcon-db # module readcon_db (PyPI)
# C/C++: FetchContent / meson dependency('readcon-db') / pkg-config
# headers in include/ are shipped; cbindgen is not required
# Prebuilt C ABI (no cargo): readcon-db-clib-$VER-$target.tar.gz on the GitHub Release
```
Docs: <https://lode-org.github.io/readcon-db/> · crate: <https://crates.io/crates/readcon-db>
## Quick start (from source)
```bash
git clone https://github.com/lode-org/readcon-db
cd readcon-db
cargo test --locked
cargo build --release # libreadcon_db + CLI readcon-db
```
Optional LODE sibling checkout (edit core + db together): clone both under the same parent, then create **untracked** `.cargo/config.toml` in `readcon-db`:
```toml
[patch.crates-io]
readcon-core = { path = "../readcon-core" }
```
Python extension from a checkout (`python/` + maturin):
```bash
pip install maturin
maturin develop --release --features python --manifest-path python/pyproject.toml
```
```rust
use readcon_db::{ConCorpus, Select};
let db = ConCorpus::open("/tmp/corpus")?;
db.append_trajectory_path(1, "run.con")?;
// XYZ in: use readcon-core chemfiles → ConFrame → append (see workflows)
let keys = db.select(
&Select::new()
.require_symbol("Cu")
.require_forces()
.exact_composition("Cu:2|H:2")
.fmax_range(0.0, 1.0)
.energy_range(-50.0, 0.0),
)?;
let h = db.frame_hash(keys[0])?;
```
```bash
./target/release/readcon-db ingest-dir /tmp/corpus /path/to/con_files
./target/release/readcon-db select /tmp/corpus --formula 'Cu:2|H:2' --require-forces \
--fmax-max 1.0 --energy-min -50 --energy-max 0
./target/release/readcon-db reindex /tmp/corpus
./target/release/readcon-db dedup-export /tmp/corpus --symbol Cu -o subset.xyz # only if a tool demands XYZ on disk
```
Foreign trajectories: **`readcon.read_chemfiles("traj.xyz")` → frames → ingest into readcon-db** (chemfiles-enabled build), not `ase.io.read`.
## Design
- **No SQL engine** — explicit indexes + in-process intersection, with **ASE.db-competitive screening fields** (mass, volume, PBC, reserved metadata, charge/magmom; see [design matrix](docs/orgmode/explanation/design.org)).
- **Decode via readcon-core** — CON semantics never fork.
- **Metadata indexes** — finite `energy` bins; flags for forces, velocities, energy presence.
- **xxHash3-128** on the CON text. A new `frames` value is `0xFD 0x01` plus one zstd level-1 frame of that UTF-8, written outside the LMDB transaction. An aligned run of 64 frames shares one `0xFD 0x03` zstd frame when that group is shorter than the per-frame values, and the other keys store a `0xFD 0x02` redirect. Untagged UTF-8 still reads as a 0.1.8 span. `readcon-db migrate` rewrites legacy values once and leaves an already packed value unchanged. `--no-compress` on `ingest`, `ingest-dir`, and `shard-ingest` stores the UTF-8. There is no dictionary. Both MPI ingest routes send the stored bytes. The hash is the decompressed text.
- **Many readers, one writer** (LMDB). Same-frame MPI: rank 0 of the
**caller communicator** packs RCSO and `MPI_Bcast` on that handle
(`include/readcon-db-mpi.h`, Python `bcast_packed_frame` /
`bcast_packed_frames`). The library never `MPI_Init`s and never names
the process-wide world communicator; LAMMPS / mpi4py pass the comm they
already own.
- **H5MD interchange** — `export_h5md` / `collect_h5md` writes one
`[T][N][3]` trajectory (CON stays authority). Engine dest is Å / ps /
kJ mol^{-1} Angstrom^{-1}; velocity dest is `Angstrom ps-1`.
Velocities follow D10: v1, v2, and v3.0 frames use the natural unit
`(amu*angstrom^2/eV)^0.5` whatever the time label says; v3.1 frames use
`units.time` and fail when a velocities section has no `units.time`.
`--velocity-time-unit`, Rust `H5mdCollectOpts`, the Python
`velocity_time_unit` argument, and C `rkrdb_set_velocity_time_unit`
override that rule for a file that breaks it.
The simulation clock uses `units.time` when that key is present.
A v3.1 frame with `time` or `timestep` and no `units.time` is an error.
The same override supplies that missing clock unit.
A present `units.time` still scales the clock.
v1, v2, and v3.0 clocks with no `units.time` are read as `fs`.
- **Node-local drain/join** — `shard-ingest` then `drain` to a unique
dest (`data.mdb` only, refuse overwrite), then `join-drained`.
`compact-join` joins one sharded root (`open_existing`). Campaign
ops: [`docs/orgmode/howto/campaign.org`](docs/orgmode/howto/campaign.org).
Full ABI table, logo, Sphinx docs, and site: see `docs/`, `website/`, `assets/logo/`, `CHANGELOG.md`. Fortran module notes: `fortran/README.md`, `fortran/ReadConDb/`.
## License
MIT
## Cooked SoA tier
Optional RCSO numerics in `frames_soa` (opt-in cook). Ingest and `migrate` do not cook. RCSO is
non-authoritative: CON text in `frames` is the sole authority for hash,
dedup, join/split, and reindex. User doc:
[`docs/orgmode/cooked-soa.org`](docs/orgmode/cooked-soa.org).