pub fn tokenize(input: &[u8]) -> Tokenizer<'_> ⓘExpand description
tokenize iterates over the CONL tokens in the input. It does not validate the structure of the file, so is suitable for using if you need error-tolerant parsing (e.g. for a linter). It continues after yielding errors. See parse for a stricter interface.