tabnas-csv (Rust)

The CSV grammar plugin for the tabnas parsing engine, crate tabnas_csv.

CSV text becomes an array of objects, one per row and keyed by the header row, or an array of arrays. Quoting follows RFC 4180 ("" escapes an embedded quote, a quoted field may hold the separator, a line break, or a control character such as a tab), the field and record separators are configurable, records can be streamed as they are parsed, and a non-strict mode lets a field body hold embedded jsonic ([1,2], {x:1}). The plugin is not standalone: it layers on the relaxed-JSON grammar of tabnas-jsonic, and reuses the engine's lexer, comment handling, and lifecycle hooks.

This is the Rust port of the canonical TypeScript implementation in ../ts; the TypeScript version is authoritative and this crate tracks it. The Go port is in ../go. All three embed the one grammar, ../csv-grammar.jsonic.

Use

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let value = tabnas_csv::parse("name,age\nAlice,30\nBob,25")?;
    assert_eq!(
        value.to_json().to_string(),
        r#"[{"name":"Alice","age":"30"},{"name":"Bob","age":"25"}]"#
    );
    Ok(())
}

parse builds one parser on first use and reuses it. Building the parser (installing the jsonic base, then the CSV grammar over it) costs far more than a small parse, so for anything but a one-off call build an instance once and keep it:

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let parser = tabnas_csv::make();
    let value = parser.parse("a,b\n\"x,y\",\"say \"\"hi\"\"\"")?;
    assert_eq!(
        value.to_json().to_string(),
        r#"[{"a":"x,y","b":"say \"hi\""}]"#
    );
    Ok(())
}

Options are a struct, with the defaults the TypeScript plugin documents. object: false gives arrays, header: false keeps the first row as data, field.separation and record.separators change the separators, and strict: false reads a field body as jsonic:

use tabnas_csv::{CsvOptions, FieldOptions};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let parser = tabnas_csv::make_with(CsvOptions {
        header: false,
        object: false,
        field: FieldOptions {
            separation: Some("|".to_string()),
            ..Default::default()
        },
        ..Default::default()
    });
    assert_eq!(
        parser.parse("a|b\nc|d")?.to_json().to_string(),
        r#"[["a","b"],["c","d"]]"#
    );

    let relaxed = tabnas_csv::make_with(CsvOptions {
        strict: false,
        ..Default::default()
    });
    assert_eq!(
        relaxed.parse("a,b,c\ntrue,[1,2],{x:1}")?.to_string(),
        r#"[{"a":true,"b":[1,2],"c":{"x":1}}]"#
    );
    Ok(())
}

The same options travel as a JSON bag through the engine's plugin mechanism, which is how the shared fixtures configure a parser per row and how a derived instance rebuilds the grammar:

use tabnas::Value;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let mut parser = tabnas_jsonic::make();
    let options = Value::from_json(&serde_json::json!({
        "field": { "exact": true },
    }));
    parser.use_plugin(tabnas_csv::plugin(), Some(options))?;
    assert_eq!(parser.parse("a,b\n1,2,3").unwrap_err().code, "csv_extra_field");
    Ok(())
}

To stream records instead of collecting them, give the options a Stream; the result is then an empty array:

use std::sync::{Arc, Mutex};
use tabnas_csv::{CsvOptions, Stream, StreamEvent};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let seen = Arc::new(Mutex::new(Vec::new()));
    let sink = Arc::clone(&seen);
    let parser = tabnas_csv::make_with(CsvOptions {
        stream: Some(Stream::new(move |event| {
            if let StreamEvent::Record(record) = event {
                sink.lock().unwrap().push(record.to_json().to_string());
            }
        })),
        ..Default::default()
    });
    parser.parse("a,b\n1,2\n3,4")?;
    assert_eq!(
        seen.lock().unwrap().as_slice(),
        [r#"{"a":"1","b":"2"}"#, r#"{"a":"3","b":"4"}"#]
    );
    Ok(())
}

Parse errors are the engine's TabnasError, re-exported as CsvError, with code, row, col and a report that shows the offending source. The codes this plugin adds are csv_extra_field and csv_missing_field, raised under field.exact; unexpected and unterminated_string come from the engine.

Install

The crate is published on crates.io as tabnas-csv, and so are the engine, tabnas-parser, and the jsonic base, tabnas-jsonic:

cargo add tabnas-csv tabnas-jsonic tabnas-parser serde_json

All four entries are needed. A crate's dependencies are not passed on to its dependents, so tabnas-csv alone does not put tabnas, tabnas-jsonic or serde_json in your extern prelude, and the examples above that name tabnas::Value, tabnas_jsonic::make or serde_json::json! would not resolve. The engine's library is named tabnas in code. Only CsvError is re-exported. serde_json is declared because the option bag is a tabnas::Value and Value::from_json takes a serde_json::Value; a program that uses only the CsvOptions struct can leave it out.

This repository's own rs/Cargo.toml names the engine, jsonic and, for the test suite's shared fixture runner, tabnas-support by path, and jsonic takes tabnas-json the same way. A build here therefore needs parser, jsonic, json and support from https://github.com/tabnas/ cloned beside it. The release workflow swaps those paths for crates.io versions when it publishes the crate.

Differences from the canonical TypeScript

Every parse result is the TypeScript one: the shared fixtures in ../test/spec, the manifest corpus in ../test/fixtures and the two third-party conformance corpora hold all three runtimes to it. What differs is the shape of the API and a few points where the host language has no way to say what JavaScript says:

• Options are a struct, or a JSON bag. make_with takes a CsvOptions; plugin() reads the bag use_plugin merges over the defaults. The three-state trim, comment, number and value are Option<bool>, with None standing for the TypeScript null.
• Streaming is a typed callback. A closure cannot travel in the option bag, so Stream sits on CsvOptions outside serialization and reaches the plugin through plugin_with and make_with. There is no 'error' event: a failed parse returns Err, as it does in Go.
• parse is a convenience the other runtimes lack. It keeps one default instance behind a OnceLock, which is the reuse the TypeScript and Go suites tell a caller to arrange by hand.
• Records keep document order. An object record is a Value::Object over an IndexMap, so the columns come back in header order without the JavaScript object's own ordering rules.
• No token descriptions. The TypeScript plugin registers human descriptions of the CSV tokens for the railroad diagram tool. The Rust engine has no such table and no diagram tool reads it, so that registration has no counterpart.
• Lone surrogates fold to U+FFFD, and the regular expression dialect is the regex crate's. Both come from the engine, and both are recorded there.

Build and test

The engine, the jsonic base, the JSON core it needs, and the fixture runner are path dependencies on sibling checkouts, so there is nothing to fetch for the build:

cargo test --all-targets && cargo test --doc

Or, from the repository root, make test-rs. For what CI would say, including formatting, Clippy and the lockfile check, run ci/rust/run.sh.

The suite runs every shared ../test/spec/*.tsv fixture through the shared runner, building a fresh parser from each row's opts column, and the whole ../test/fixtures manifest corpus. The conformance tests judge the two third-party corpora (csv-spectrum and Go's encoding/csv readTests) with the same divergence table the TypeScript and Go suites carry; they read the corpora vendored under ../test/suites, with no network, and FAIL, never skip, when a corpus is missing. Beside them are the in-language tests: the API surface, streaming, typed option values, the field.exact messages, the exported string matcher on a plain jsonic instance, the shared default parser under threads, instance reuse, the version sites, and the embedded grammar against the file on disk.

License

MIT.