Skip to main content

csv_string_matcher

Function csv_string_matcher 

Source
pub fn csv_string_matcher(
    quote: impl Into<String>,
) -> impl Fn(&Options) -> Option<ImperativeLexMatcher> + Send + Sync + 'static
Expand description

The custom CSV string matcher factory: "a""b" is the text a"b.

A reduced copy of the standard jsonic string matcher, with the doubled quote as the only escape. It is the counterpart of the TypeScript buildCsvStringMatcher export and the Go BuildCsvStringMatcher, in the shape the engine’s options.lex.match.<name>.make reference takes: register it with Tabnas::lex_match_factory_ref and name it from an options document.

use tabnas::GrammarSpec;

let mut parser = tabnas_jsonic::make();
parser.lex_match_factory_ref("@csv-string", tabnas_csv::csv_string_matcher("\""));
let spec = GrammarSpec::from_value(serde_json::json!({
    "options": { "lex": { "match": {
        "stringcsv": { "order": 100000, "make": "@csv-string" },
    } } },
}))?;
parser.grammar(&spec)?;
assert_eq!(parser.parse(r#""a""b""#)?, tabnas::Value::String("a\"b".to_string()));

The matcher runs only where the engine wants a string token, and only at the quote character. A line character inside the quotes is field text (a quoted field may span lines). Any other control character is field text too when the engine’s string.allowControl option is on, as the CSV plugin sets it, and unprintable when it is off, as it is by default on a plain instance. A quote left open at the end of the source is unterminated_string. Loop exhaustion is what detects the open quote, so an odd number of quotes (""", """"") is caught rather than read as a terminated string.

A quote that is not exactly one UTF-16 code unit leaves the matcher inert, and the factory answers None rather than installing one. The canonical opens a quoted field with quoteMap = { [options.string.quote]: true } and tests quoteMap[src[sI]], where src[sI] is ONE UTF-16 code unit, so it can only ever fire for a quote that is one code unit: the empty string, a multi-character quote such as ab, and an astral character such as U+1F600, which JavaScript holds as a surrogate PAIR, are all inert there. Go carries the same guard, with the same reasoning.