Struct PathHierarchyTokenizer 

pub struct PathHierarchyTokenizer {
    pub separator: char,
    pub replacement: Option<char>,
}

Splits filesystem paths into hierarchical tokens.

/a/b/c → /a, /a/b, /a/b/c

Matches ES path_hierarchy tokenizer.

See [[analyzers#Tokenizer]].

Fields

separator: char

replacement: Option<char>

Implementations

impl PathHierarchyTokenizer

pub fn new(separator: char, replacement: Option<char>) -> Self

Trait Implementations

impl Default for PathHierarchyTokenizer

fn default() -> Self

Returns the “default value” for a type.

impl Tokenizer for PathHierarchyTokenizer

fn tokenize(&self, text: &str, output: &mut Vec<Token>)

Tokenize text, appending tokens to output.