Struct CodeLexTokenizer 

pub struct CodeLexTokenizer { /* private fields */ }

Tokenizer for code-like text.

Identifiers are Unicode alphanumeric characters plus _. Other characters are lexical boundaries. When operator indexing is enabled, recognized multi-character operators use longest-match tokenization and remaining operator characters are emitted individually.

Examples

use lance_tokenizer::{CodeLexTokenizer, TextAnalyzer, TokenStream};

let mut analyzer = TextAnalyzer::builder(CodeLexTokenizer::new(true)).build();
let mut stream = analyzer.token_stream("a::b");

assert!(stream.advance());
assert_eq!(stream.token().text, "a");
assert!(stream.advance());
assert_eq!(stream.token().text, "::");

Implementations

impl CodeLexTokenizer

pub fn new(index_operators: bool) -> Self

Trait Implementations

impl Clone for CodeLexTokenizer

fn clone(&self) -> CodeLexTokenizer

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Default for CodeLexTokenizer

fn default() -> CodeLexTokenizer

Returns the “default value” for a type.

impl Tokenizer for CodeLexTokenizer

type TokenStream<'a> = CodeLexTokenStream<'a>

Stream type emitted by the tokenizer.

fn token_stream<'a>(&'a mut self, text: &'a str) -> Self::TokenStream<'a>

Create a token stream for the provided text.