A Token is a single token in the input with a line number attached.
They are generated by parse and tokenize. Use Token::unescape to get the actual value.
parse iterates over a CONL file, returning Tokens. In the case of an error it will
yield a SyntaxError and then stop returning more tokens.
You can likely ignore Token::Newline, Token::Comment and Token::MultilineHint.
The structure of the file is validated, so you can be sure that you’ll see pairs of:
tokenize iterates over the CONL tokens in the input. It does not
validate the structure of the file, so is suitable for using if you
need error-tolerant parsing (e.g. for a linter).
It continues after yielding errors.
See parse for a stricter interface.