Module lexical 

Lightweight lexical classification of source text.

The search tools need to tell an identifier occurrence apart from the same text inside a string literal or a comment. A full parser is not needed for that, and parsing every file is expensive, so this module scans the text once and records which bytes are code.

The scanner is deliberately conservative: it understands line comments, block comments, and string literals, which is what distinguishes a real identifier occurrence from a mention in a comment or a string. When it meets syntax it does not understand, it keeps scanning as code, so an unrecognised construct can only cause a text result to be reported as a usage, never a hit to be lost.

Structs

LanguageSyntax

The comment and string syntax of one language.

Functions

code_mask

Marks which bytes of source are code rather than string or comment text.