rudb_csv/lib.rs
1//! The CSV reader and writer, including dialect sniffing and type inference.
2//!
3//! Rank 5 in the layer rule. See `xtask/layers.toml` and `spec/18-package-layout.md`.
4//!
5//! A CSV file says nothing about itself. The delimiter, the quote, whether the first line names the
6//! columns and what type each column holds are all conventions, and a reader that has to be told
7//! them is a reader every loader script in the world has to be rewritten for. So this sniffs, the
8//! way DuckDB does, and every rule in it was read off duckdb v1.4.1 rather than reasoned about.
9//! Where the two differ, the difference is written down next to the code, because a sniffer that
10//! quietly disagrees produces a column of the wrong type and that is a wrong answer rather than a
11//! slow query.
12//!
13//! Three pieces. [`scan`] turns bytes into fields, [`dialect`] works out the punctuation by running
14//! the scanner under each candidate and seeing which one is consistent, and [`infer`] walks a ladder
15//! of types to find the first one every value in a column fits. [`Reader`] is the three of them over
16//! a file, handing back chunks.
17//!
18//! [`combine`] is the fourth, and it is there because a read can cover more than one file.
19//! `read_csv('data/*.csv')` sniffs every file the pattern matched and combines the answers, since a
20//! file that says nothing about itself cannot be the one file whose word is taken the way a Parquet
21//! footer is.
22//!
23//! There is no writer yet. `COPY t TO 'out.csv'` is the statement that wants one and it is not
24//! bound, so a writer here would be a writer nothing calls.
25
26#![forbid(unsafe_code)]
27
28pub mod combine;
29pub mod dialect;
30pub mod infer;
31pub mod reader;
32pub mod scan;
33
34pub use combine::{across, mismatch, widen};
35pub use dialect::{Dialect, Given};
36pub use reader::Reader;
37
38/// The crate this rank belongs to, so that the layer check has something to read.
39pub const RANK: u8 = 5;