Expand description
Code-aware tokenizer: identifiers are kept whole and also split on camelCase, PascalCase, snake_case and digits, all lowercased.
Functionsยง
- chunk_
terms - Term frequencies for a chunk: body tokens, the symbol name boosted, and path terms once each.
- floor_
char - Floor a byte limit to a UTF-8 boundary, clamped to the text length.
- path_
terms - Terms contributed by a file path (directory and file stems).
- push_
ident - Push the whole identifier and its sub-words.
- tokenize
- Tokenize free text or code.