datasus-rs
Reader for DATASUS .dbc files (and plain .dbf) in Rust, shared by three
front ends:
| Crate | What it is |
|---|---|
crates/datasus-core |
The reader: PKWare DCL decompression + dBase parsing. No dependencies. |
crates/datasus-arrow |
Core → Apache Arrow record batches. |
crates/datasus-cli |
datasus command: convert to Parquet/CSV, inspect, decompress. |
crates/datasus-py |
datasuspy Python package (PyO3 + maturin). |
The core is a port of the C reader of the datasusr R package and is tested to return the same values for the same options (see Validation). Catalog, FTP download and cache are not part of this project; they live in datasusr.
CLI
convert options mirror read_datasus_dbc(): --select, --n-max,
--no-trim, --encoding, --no-guess-types, --col-type NAME=TYPE,
--parse-dates, --keep-names. Outputs are written to a .part file and
renamed on success; existing outputs need --force.
Python
From source: cd crates/datasus-py && maturin develop --release inside a
virtualenv.
= # pyarrow.Table
=
=
# Streaming, without pyarrow: any Arrow PyCapsule consumer works
=
Rust
use ;
let table = open?;
let columns = table.collect; // Vec<ColumnData>
To fill your own buffers without an intermediate copy, implement
datasus_core::Sink and call Table::read_rows.
Behaviour (same as datasusr 0.1.1)
- Records flagged as deleted (
*) are dropped;n_maxcounts live records. - Character fields are never null: a blank field is an empty string.
- Numeric, date and logical fields are always trimmed before parsing;
trim_wsonly affects character fields. A value ends at the first NUL. N/Ffields with no decimals are integers unless a value does not fit a 32-bit integer (then the column is double).i32::MINis not accepted as an integer (it is R'sNA).Dfields are text unlessparse_datesis set; invalid dates are null.- A header declaring more records than the file holds is a warning.
- Decompression stops at the record count declared in the header and never
goes past 4 GiB (
decode_with_limitchanges the bound), so a crafted file cannot expand without limit.
Known differences from the C reader:
- Whitespace is always the six ASCII characters of the "C" locale. The C
reader uses
isspace(), which on macOS under a UTF-8 locale also trims0xA0(NBSP). - Latin-1 text is decoded as strict ISO-8859-1. R on macOS and Windows
converts
latin1strings as Windows-1252, so bytes0x80–0x9Fcome out differently there. - With
encoding = "utf-8", invalid sequences become U+FFFD. - Hexadecimal floats and
nan(...), accepted bystrtod(), are null.
Validation
# Oracle: compare against datasusr's C reader (needs R with datasusr + arrow)
&& &&
License
MIT. blast.rs is a translation of Mark Adler's blast.c (zlib license).