readcon-db

Mmap-backed CON/convel corpus store (LMDB via Heed), non-SQL selection, xxHash3-128 exact match, and Rust / C / C++ / Python / Fortran bindings.

Part of the readcon ecosystem with readcon-core (Python package readcon):

ASE is not on the CON or XYZ read path. Optional to_ase is calculator hand-off only.

Install

cargo add readcon-db
cargo install readcon-db --locked   # CLI
pip install readcon-db             # module readcon_db (PyPI)
# C/C++: FetchContent / meson dependency('readcon-db') / pkg-config
# headers in include/ are shipped; cbindgen is not required
# Prebuilt C ABI (no cargo): readcon-db-clib-$VER-$target.tar.gz on the GitHub Release

Docs: https://lode-org.github.io/readcon-db/ · crate: https://crates.io/crates/readcon-db

Quick start (from source)

git clone https://github.com/lode-org/readcon-db
cd readcon-db
cargo test --locked
cargo build --release   # libreadcon_db + CLI readcon-db

Optional LODE sibling checkout (edit core + db together): clone both under the same parent, then create untracked .cargo/config.toml in readcon-db:

[patch.crates-io]
readcon-core = { path = "../readcon-core" }

Python extension from a checkout (python/ + maturin):

pip install maturin
maturin develop --release --features python --manifest-path python/pyproject.toml

use readcon_db::{ConCorpus, Select};
let db = ConCorpus::open("/tmp/corpus")?;
db.append_trajectory_path(1, "run.con")?;
// XYZ in: use readcon-core chemfiles → ConFrame → append (see workflows)
let keys = db.select(
    &Select::new()
        .require_symbol("Cu")
        .require_forces()
        .exact_composition("Cu:2|H:2")
        .fmax_range(0.0, 1.0)
        .energy_range(-50.0, 0.0),
)?;
let h = db.frame_hash(keys[0])?;

./target/release/readcon-db ingest-dir /tmp/corpus /path/to/con_files
./target/release/readcon-db select /tmp/corpus --formula 'Cu:2|H:2' --require-forces \
    --fmax-max 1.0 --energy-min -50 --energy-max 0
./target/release/readcon-db reindex /tmp/corpus
./target/release/readcon-db dedup-export /tmp/corpus --symbol Cu -o subset.xyz  # only if a tool demands XYZ on disk

Foreign trajectories: readcon.read_chemfiles("traj.xyz") → frames → ingest into readcon-db (chemfiles-enabled build), not ase.io.read.

Design

• No SQL engine — explicit indexes + in-process intersection, with ASE.db-competitive screening fields (mass, volume, PBC, reserved metadata, charge/magmom; see design matrix).
• Decode via readcon-core — CON semantics never fork.
• Metadata indexes — finite energy bins; flags for forces, velocities, energy presence.
• xxHash3-128 on the CON text. A new frames value is 0xFD 0x01 plus one zstd level-1 frame of that UTF-8, written outside the LMDB transaction. An aligned run of 64 frames shares one 0xFD 0x03 zstd frame when that group is shorter than the per-frame values, and the other keys store a 0xFD 0x02 redirect. Untagged UTF-8 still reads as a 0.1.8 span. readcon-db migrate rewrites legacy values once and leaves an already packed value unchanged. --no-compress on ingest, ingest-dir, and shard-ingest stores the UTF-8. There is no dictionary. Both MPI ingest routes send the stored bytes. The hash is the decompressed text.
• Many readers, one writer (LMDB). Same-frame MPI: rank 0 of the caller communicator packs RCSO and MPI_Bcast on that handle (include/readcon-db-mpi.h, Python bcast_packed_frame / bcast_packed_frames). The library never MPI_Inits and never names the process-wide world communicator; LAMMPS / mpi4py pass the comm they already own.
• H5MD interchange — export_h5md / collect_h5md writes one [T][N][3] trajectory (CON stays authority). Engine dest is Å / ps / kJ mol^{{-1} Angstrom}{-1}; velocity dest is Angstrom ps-1. Velocities follow D10: v1, v2, and v3.0 frames use the natural unit (amu*angstrom^2/eV)^0.5 whatever the time label says; v3.1 frames use units.time and fail when a velocities section has no units.time. --velocity-time-unit, Rust H5mdCollectOpts, the Python velocity_time_unit argument, and C rkrdb_set_velocity_time_unit override that rule for a file that breaks it. The simulation clock uses units.time when that key is present. A v3.1 frame with time or timestep and no units.time is an error. The same override supplies that missing clock unit. A present units.time still scales the clock. v1, v2, and v3.0 clocks with no units.time are read as fs.
• Node-local drain/join — shard-ingest then drain to a unique dest (data.mdb only, refuse overwrite), then join-drained. compact-join joins one sharded root (open_existing). Campaign ops: docs/orgmode/howto/campaign.org.

Full ABI table, logo, Sphinx docs, and site: see docs/, website/, assets/logo/, CHANGELOG.md. Fortran module notes: fortran/README.md, fortran/ReadConDb/.

License

MIT

Cooked SoA tier

Optional RCSO numerics in frames_soa (opt-in cook). Ingest and migrate do not cook. RCSO is non-authoritative: CON text in frames is the sole authority for hash, dedup, join/split, and reindex. User doc: docs/orgmode/cooked-soa.org.