pub fn lex_partial(
src: &str,
version: RustyfiVersion,
) -> (Vec<Atom>, Option<LexError>)Expand description
lex_with_version, keeping the tokens produced before a failure
instead of discarding them.
The tokens are identical to lex_with_version’s up to the failure point;
past it there are none, and the returned LexError is the same one
lex_with_version would have returned. A successful lex returns
(atoms, None) and ends with Token::Eoi as usual; a failed one has no
Eoi, because the input never reached one.
Exists for the language server, where the buffer is half-typed by
definition and the commonest failures are local: {\emp with the command
not yet applied to anything is “unexpected token in an active area”, and
'<+p with no closing > is an unterminated group. Both leave every token
before the cursor perfectly well-formed, and those are exactly the tokens
that say which text area the cursor is in and which names are in scope
there. Discarding them costs completion its two most important cases.
The compiler keeps using lex_with_version: a partial token stream is
the right input for an editor question and the wrong input for a build.