Struct TokenDictionary 

pub struct TokenDictionary { /* private fields */ }

Token dictionary mapping token strings to unique integer IDs.

Token IDs are assigned as follows:

• IDs 0 to len_legalese-1: Reserved for legalese tokens (high-value words)
• IDs len_legalese and above: Assigned to other tokens as encountered

The len_legalese delimiter allows the matching engine to distinguish between high-value (legalese) tokens and regular tokens.

Based on the Python ScanCode Toolkit implementation at: reference/scancode-toolkit/src/licensedcode/index.py

Implementations

impl TokenDictionary

pub fn new_with_legalese(legalese_entries: &[(&str, u16)]) -> Self

Create a new token dictionary initialized with legalese tokens.

This follows the Python ScanCode TorchToolkit pattern where the dictionary starts with pre-defined legalese words that get low IDs (high value).

Arguments

• legalese_entries - Slice of (word, token_id) pairs for legalese words

Returns

A new TokenDictionary instance with legalese tokens pre-populated

pub fn new(legalese_count: usize) -> Self

Create a new empty token dictionary (for testing).

Arguments

• legalese_count - Number of reserved legalese token IDs

Returns

A new TokenDictionary instance

pub fn intern(&mut self, token: &str) -> KnownToken

pub fn lookup(&self, token: &str) -> Option<KnownToken>

pub fn classify_query_token(&self, token: &str) -> QueryToken

pub fn token_kind(&self, token_id: TokenId) -> TokenKind

pub fn is_digit_only_token(&self, token_id: TokenId) -> bool

pub fn get_token_id(&self, token: &str) -> Option<TokenId>

Get the token ID for a token string if it exists.

Arguments

• token - The token string

Returns

Some(token_id) if the token exists, None otherwise

pub fn get(&self, token: &str) -> Option<TokenId>

Get the token ID (alias for backward compatibility).

pub const fn legalese_count(&self) -> usize

Get the number of legalese tokens.

Trait Implementations

impl Clone for TokenDictionary

fn clone(&self) -> TokenDictionary

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for TokenDictionary

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for TokenDictionary

fn default() -> Self

Returns the “default value” for a type.