Expand description
Tokenizes a string by creating a lattice of possible tokens over it and finding the lowest-cost path over that lattice. Returns a list of ParserToken and the cost of the tokenization.
The dictionary defines what tokens exist, how they appear in the string, their costs, and the costs of their possible connections.
Generates ParserTokens over the chosen path and returns a list of those ParserTokens and the cost the path took. Cost can be negative.
It’s possible for multiple paths to tie for the lowest cost. It’s not defined which path is returned in that case.