Function notmecab::parse_to_lexertokens

pub fn parse_to_lexertokens(
    dict: &Dict, 
    pseudo_string: &[char]
) -> Option<(Vec<LexerToken>, i64)>

Tokenizes a char slice by creating a lattice of possible tokens over it and finding the lowest-cost path over that lattice. Returns a list of LexerTokens and the cost of the tokenization.

The dictionary defines what tokens exist, how they appear in the string, their costs, and the costs of their possible connections.

Returns a vector listing the LexerTokens on the chosen path and the cost the path took. Cost can be negative.

It's possible for multiple paths to tie for the lowest cost. It's not defined which path is returned in that case.