Skip to main content

Module tokenize

Module tokenize 

Source
Expand description

Code-aware tokenizer: identifiers are kept whole and also split on camelCase, PascalCase, snake_case and digits, all lowercased.

Functionsยง

chunk_terms
Term frequencies for a chunk: body tokens, the symbol name boosted, and path terms once each.
floor_char
Floor a byte limit to a UTF-8 boundary, clamped to the text length.
path_terms
Terms contributed by a file path (directory and file stems).
push_ident
Push the whole identifier and its sub-words.
tokenize
Tokenize free text or code.