Struct TokenDictionary 

pub struct TokenDictionary { /* private fields */ }

Token dictionary mapping token strings to unique integer IDs.

Token IDs are assigned as follows:

• IDs 0 to len_legalese-1: Reserved for legalese tokens (high-value words)
• IDs len_legalese and above: Assigned to other tokens as encountered

The len_legalese delimiter allows the matching engine to distinguish between high-value (legalese) tokens and regular tokens.

Based on the Python ScanCode Toolkit implementation at: reference/scancode-toolkit/src/licensedcode/index.py

Implementations

impl TokenDictionary

pub fn new_with_legalese(legalese_entries: &[(&str, u16)]) -> Self

Create a new token dictionary initialized with legalese tokens.

This follows the Python ScanCode TorchToolkit pattern where the dictionary starts with pre-defined legalese words that get low IDs (high value).

Arguments

• legalese_entries - Slice of (word, token_id) pairs for legalese words

Returns

A new TokenDictionary instance with legalese tokens pre-populated

pub fn new(legalese_count: usize) -> Self

Create a new empty token dictionary (for testing).

Arguments

• legalese_count - Number of reserved legalese token IDs

Returns

A new TokenDictionary instance

pub fn intern(&mut self, token: &str) -> KnownToken

pub fn lookup(&self, token: &str) -> Option<KnownToken>

pub fn classify_query_token(&self, token: &str) -> QueryToken

pub fn token_kind(&self, token_id: TokenId) -> TokenKind

pub fn is_digit_only_token(&self, token_id: TokenId) -> bool

pub fn get_token_id(&self, token: &str) -> Option<TokenId>

Get the token ID for a token string if it exists.

Arguments

• token - The token string

Returns

Some(token_id) if the token exists, None otherwise

pub fn get(&self, token: &str) -> Option<TokenId>

Get the token ID (alias for backward compatibility).

pub const fn legalese_count(&self) -> usize

Get the number of legalese tokens.

pub fn tokens_to_ids_len(&self) -> usize

Get the number of tokens in the dictionary.

Trait Implementations

impl Archive for TokenDictionary

where HashMap<String, TokenId>: Archive, Vec<Option<TokenMetadata>>: Archive, usize: Archive, TokenId: Archive,

const COPY_OPTIMIZATION: CopyOptimization<Self>

An optimization flag that allows the bytes of this type to be copied directly to a writer instead of calling serialize.

type Archived = ArchivedTokenDictionary

The archived representation of this type.

type Resolver = TokenDictionaryResolver

The resolver for this type. It must contain all the additional information from serializing needed to make the archived type from the normal type.

fn resolve(&self, resolver: Self::Resolver, out: Place<Self::Archived>)

Creates the archived version of this value at the given position and writes it to the given output.

impl Clone for TokenDictionary

fn clone(&self) -> TokenDictionary

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for TokenDictionary

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for TokenDictionary

fn default() -> Self

Returns the “default value” for a type.

impl<__D: Fallible + ?Sized> Deserialize<TokenDictionary, __D> for Archived<TokenDictionary>

where HashMap<String, TokenId>: Archive, <HashMap<String, TokenId> as Archive>::Archived: Deserialize<HashMap<String, TokenId>, __D>, Vec<Option<TokenMetadata>>: Archive, <Vec<Option<TokenMetadata>> as Archive>::Archived: Deserialize<Vec<Option<TokenMetadata>>, __D>, usize: Archive, <usize as Archive>::Archived: Deserialize<usize, __D>, TokenId: Archive, <TokenId as Archive>::Archived: Deserialize<TokenId, __D>,

fn deserialize( &self, deserializer: &mut __D, ) -> Result<TokenDictionary, <__D as Fallible>::Error>

Deserializes using the given deserializer

impl<__S: Fallible + ?Sized> Serialize<__S> for TokenDictionary

where HashMap<String, TokenId>: Serialize<__S>, Vec<Option<TokenMetadata>>: Serialize<__S>, usize: Serialize<__S>, TokenId: Serialize<__S>,

fn serialize( &self, serializer: &mut __S, ) -> Result<<Self as Archive>::Resolver, <__S as Fallible>::Error>

Writes the dependencies for the object and returns a resolver that can create the archived type.