Struct LicenseIndex 

pub struct LicenseIndex {

    pub dictionary: TokenDictionary,
    pub len_legalese: usize,
    pub rid_by_hash: HashMap<[u8; 20], RuleId>,
    pub rules_by_rid: Vec<Rule>,
    pub tids_by_rid: Vec<Vec<TokenId>>,
    pub rules_automaton: Automaton,
    pub unknown_automaton: Automaton,
    pub sets_by_rid: Vec<Option<TokenSet>>,
    pub rule_metadata_by_identifier: HashMap<String, IndexedRuleMetadata>,
    pub msets_by_rid: Vec<Option<TokenMultiset>>,
    pub high_sets_by_rid: Vec<Option<TokenSet>>,
    pub high_bitsets_by_rid: Vec<Option<HighBitset>>,
    pub high_postings_by_rid: HashMap<RuleId, HashMap<TokenId, Vec<usize>>>,
    pub licenses_by_key: HashMap<String, License>,
    pub rid_by_spdx_key: HashMap<String, RuleId>,
    pub unknown_spdx_rid: Option<RuleId>,
    pub rids_by_high_tid: HashMap<TokenId, HashSet<RuleId>>,
    pub spdx_license_list_version: Option<String>,
}

License index containing all data structures for efficient license detection.

The LicenseIndex holds multiple index structures that enable different matching strategies: hash-based exact matching, Aho-Corasick automaton matching, set-based candidate selection, and sequence matching.

Based on the Python ScanCode Toolkit implementation at: reference/scancode-toolkit/src/licensedcode/index.py

Index Structures

The index maintains several data structures for different matching strategies:

• Hash matching: rid_by_hash for exact hash-based matches
• Automaton matching: rules_automaton and unknown_automaton for pattern matching
• Candidate selection: sets_by_rid and msets_by_rid for set-based ranking
• Sequence matching: high_postings_by_rid for high-value token position tracking

Fields

dictionary: TokenDictionary

Token dictionary mapping token strings to integer IDs.

IDs 0 to len_legalese-1 are reserved for legalese tokens (high-value words). IDs len_legalese and above are assigned to other tokens as encountered.

len_legalese: usize

Number of legalese tokens.

Tokens with ID < len_legalese are considered high-value legalese words. Tokens with ID >= len_legalese are considered low-value tokens.

Corresponds to Python: self.len_legalese = 0 (line 185)

rid_by_hash: HashMap<[u8; 20], RuleId>

Mapping from rule hash to rule ID for hash-based exact matching.

This enables fast exact matches using a hash of the rule's token IDs. Each hash maps to exactly one rule ID.

Note: The hash is a 20-byte SHA1 digest, stored as a key in HashMap. In practice, we use a HashMap<[u8; 20], RuleId>.

Corresponds to Python: self.rid_by_hash = {} (line 216)

rules_by_rid: Vec<Rule>

Rules indexed by rule ID.

Maps rule IDs to Rule objects for quick lookup.

Corresponds to Python: self.rules_by_rid = [] (line 201)

tids_by_rid: Vec<Vec<TokenId>>

Token ID sequences indexed by rule ID.

Maps rule IDs to their token ID sequences.

Corresponds to Python: self.tids_by_rid = [] (line 204)

rules_automaton: Automaton

Aho-Corasick automaton built from all rule token sequences.

Supports efficient multi-pattern matching of token ID sequences. Used for exact matching of complete rules or rule fragments in query text.

Corresponds to Python: self.rules_automaton = match_aho.get_automaton() (line 219)

unknown_automaton: Automaton

Aho-Corasick automaton for unknown license detection.

Separate automaton used to detect license-like text that doesn't match any known rule. Populated with ngrams from all approx-matchable rules.

Corresponds to Python: self.unknown_automaton = match_unknown.get_automaton() (line 222)

sets_by_rid: Vec<Option<TokenSet>>

Token ID sets per rule for candidate selection.

Indexed densely by RuleId::raw() (mirroring Python’s sets_by_rid = [] list). Entries are None for rule IDs that hold no set (e.g. false-positive rules), so the candidate-selection hot loop reads a Vec slot instead of a hashed lookup. Use LicenseIndex::set_for_rid to access.

rule_metadata_by_identifier: HashMap<String, IndexedRuleMetadata>

msets_by_rid: Vec<Option<TokenMultiset>>

Token ID multisets per rule for candidate ranking.

Indexed densely by RuleId::raw() (mirroring Python’s msets_by_rid = []). None where no multiset exists. Use LicenseIndex::mset_for_rid.

high_sets_by_rid: Vec<Option<TokenSet>>

High-value token sets per rule for early candidate rejection.

Indexed densely by RuleId::raw(). A subset of sets_by_rid containing only high-value (legalese) token IDs, for early rejection of candidates that won’t pass the high-token threshold. None where the rule has no high-value tokens. Use LicenseIndex::high_set_for_rid.

high_bitsets_by_rid: Vec<Option<HighBitset>>

High-value token sets as fixed-width bitsets (width = len_legalese), densely indexed by RuleId::raw() and aligned with high_sets_by_rid (Some exactly where that is Some). Used by the candidate-selection high-token gate, where AND+popcount replaces the sorted-set merge walk. Use LicenseIndex::high_bitset_for_rid.

high_postings_by_rid: HashMap<RuleId, HashMap<TokenId, Vec<usize>>>

Inverted index of high-value token positions per rule.

Maps rule IDs to a mapping from high-value token IDs to their positions within the rule. Only contains positions for tokens with IDs < len_legalese.

This structure speeds up sequence matching by allowing quick lookup of where high-value tokens appear in each rule.

Corresponds to Python: self.high_postings_by_rid = [] (line 209) In Python: postings = {tid: array('h', [positions, ...])}

licenses_by_key: HashMap<String, License>

Mapping from ScanCode license key to License object.

Provides access to license metadata for building SPDX mappings and validating license expressions.

Corresponds to Python: get_licenses_db() in models.py

rid_by_spdx_key: HashMap<String, RuleId>

Mapping from SPDX license key to rule ID.

Enables direct lookup of rules by their SPDX license key, including aliases like “GPL-2.0+” -> gpl-2.0-plus.

Keys are stored lowercase for case-insensitive lookup.

Corresponds to Python: self.licenses_by_spdx_key in cache.py

unknown_spdx_rid: Option<RuleId>

Rule ID for the unknown-spdx license.

Used as a fallback when an SPDX identifier is not recognized.

Corresponds to Python: get_unknown_spdx_symbol() in cache.py

rids_by_high_tid: HashMap<TokenId, HashSet<RuleId>>

Inverted index mapping high-value token IDs to rule IDs.

This enables fast candidate selection by only examining rules that share at least one high-value (legalese) token with the query. Without this index, candidate selection would iterate over all 37,000+ rules for every file, making license detection extremely slow.

Only contains entries for tokens with ID < len_legalese (high-value tokens). Only approx-matchable rules are included in this index.

spdx_license_list_version: Option<String>

SPDX license list version used to build this index.

Implementations

impl LicenseIndex

pub fn rule(&self, id: RuleId) -> Option<&Rule>

Returns the rule for the given ID, or None if the ID is invalid or out of range.

pub fn rule_tokens(&self, id: RuleId) -> Option<&[TokenId]>

Returns the token ID sequence for the given rule, or None if the ID is invalid or out of range.

pub fn is_false_positive(&self, id: RuleId) -> bool

impl LicenseIndex

pub fn new(dictionary: TokenDictionary) -> Self

Create a new empty license index.

This constructor initializes all index structures with empty collections. The index can be populated with rules using the indexing methods (to be implemented in future phases).

Returns

A new LicenseIndex instance with empty index structures

pub fn set_for_rid(&self, rid: RuleId) -> Option<&TokenSet>

Unique-token set for a rule, or None if the rule holds no set (e.g. a false-positive rule). Dense Vec lookup by RuleId.

pub fn high_set_for_rid(&self, rid: RuleId) -> Option<&TokenSet>

High-value (legalese) token set for a rule, or None if the rule has no high-value tokens. Dense Vec lookup by RuleId.

pub fn mset_for_rid(&self, rid: RuleId) -> Option<&TokenMultiset>

Token multiset for a rule, or None if the rule holds no multiset. Dense Vec lookup by RuleId.

pub fn high_bitset_for_rid(&self, rid: RuleId) -> Option<&HighBitset>

High-value token bitset for a rule, or None if the rule has no high-value tokens. Dense Vec lookup by RuleId.

pub fn with_legalese_count(legalese_count: usize) -> Self

Create a new empty license index with the specified legalese count.

Convenience method that creates a new TokenDictionary and LicenseIndex in one call.

Arguments

• legalese_count - Number of reserved legalese token IDs

Returns

A new LicenseIndex instance with a new TokenDictionary

Trait Implementations

impl Archive for LicenseIndex

where TokenDictionary: Archive, usize: Archive, HashMap<[u8; 20], RuleId>: Archive, Vec<Rule>: Archive, Vec<Vec<TokenId>>: Archive, AsBytes: ArchiveWith<Automaton>, Vec<Option<TokenSet>>: Archive, HashMap<String, IndexedRuleMetadata>: Archive, Vec<Option<TokenMultiset>>: Archive, Vec<Option<HighBitset>>: Archive, HashMap<RuleId, HashMap<TokenId, Vec<usize>>>: Archive, HashMap<String, License>: Archive, HashMap<String, RuleId>: Archive, Option<RuleId>: Archive, HashMap<TokenId, HashSet<RuleId>>: Archive, Option<String>: Archive,

type Archived = ArchivedLicenseIndex

The archived representation of this type.

type Resolver = LicenseIndexResolver

The resolver for this type. It must contain all the additional information from serializing needed to make the archived type from the normal type.

fn resolve(&self, resolver: Self::Resolver, out: Place<Self::Archived>)

Creates the archived version of this value at the given position and writes it to the given output.

const COPY_OPTIMIZATION: CopyOptimization<Self> = _

An optimization flag that allows the bytes of this type to be copied directly to a writer instead of calling serialize.

impl Clone for LicenseIndex

fn clone(&self) -> LicenseIndex

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for LicenseIndex

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for LicenseIndex

fn default() -> Self

Returns the “default value” for a type.

impl<__D: Fallible + ?Sized> Deserialize<LicenseIndex, __D> for Archived<LicenseIndex>

where TokenDictionary: Archive, <TokenDictionary as Archive>::Archived: Deserialize<TokenDictionary, __D>, usize: Archive, <usize as Archive>::Archived: Deserialize<usize, __D>, HashMap<[u8; 20], RuleId>: Archive, <HashMap<[u8; 20], RuleId> as Archive>::Archived: Deserialize<HashMap<[u8; 20], RuleId>, __D>, Vec<Rule>: Archive, <Vec<Rule> as Archive>::Archived: Deserialize<Vec<Rule>, __D>, Vec<Vec<TokenId>>: Archive, <Vec<Vec<TokenId>> as Archive>::Archived: Deserialize<Vec<Vec<TokenId>>, __D>, AsBytes: ArchiveWith<Automaton> + DeserializeWith<<AsBytes as ArchiveWith<Automaton>>::Archived, Automaton, __D>, Vec<Option<TokenSet>>: Archive, <Vec<Option<TokenSet>> as Archive>::Archived: Deserialize<Vec<Option<TokenSet>>, __D>, HashMap<String, IndexedRuleMetadata>: Archive, <HashMap<String, IndexedRuleMetadata> as Archive>::Archived: Deserialize<HashMap<String, IndexedRuleMetadata>, __D>, Vec<Option<TokenMultiset>>: Archive, <Vec<Option<TokenMultiset>> as Archive>::Archived: Deserialize<Vec<Option<TokenMultiset>>, __D>, Vec<Option<HighBitset>>: Archive, <Vec<Option<HighBitset>> as Archive>::Archived: Deserialize<Vec<Option<HighBitset>>, __D>, HashMap<RuleId, HashMap<TokenId, Vec<usize>>>: Archive, <HashMap<RuleId, HashMap<TokenId, Vec<usize>>> as Archive>::Archived: Deserialize<HashMap<RuleId, HashMap<TokenId, Vec<usize>>>, __D>, HashMap<String, License>: Archive, <HashMap<String, License> as Archive>::Archived: Deserialize<HashMap<String, License>, __D>, HashMap<String, RuleId>: Archive, <HashMap<String, RuleId> as Archive>::Archived: Deserialize<HashMap<String, RuleId>, __D>, Option<RuleId>: Archive, <Option<RuleId> as Archive>::Archived: Deserialize<Option<RuleId>, __D>, HashMap<TokenId, HashSet<RuleId>>: Archive, <HashMap<TokenId, HashSet<RuleId>> as Archive>::Archived: Deserialize<HashMap<TokenId, HashSet<RuleId>>, __D>, Option<String>: Archive, <Option<String> as Archive>::Archived: Deserialize<Option<String>, __D>,

fn deserialize( &self, deserializer: &mut __D, ) -> Result<LicenseIndex, <__D as Fallible>::Error>

Deserializes using the given deserializer

impl<__S: Fallible + ?Sized> Serialize<__S> for LicenseIndex

where TokenDictionary: Serialize<__S>, usize: Serialize<__S>, HashMap<[u8; 20], RuleId>: Serialize<__S>, Vec<Rule>: Serialize<__S>, Vec<Vec<TokenId>>: Serialize<__S>, AsBytes: SerializeWith<Automaton, __S>, Vec<Option<TokenSet>>: Serialize<__S>, HashMap<String, IndexedRuleMetadata>: Serialize<__S>, Vec<Option<TokenMultiset>>: Serialize<__S>, Vec<Option<HighBitset>>: Serialize<__S>, HashMap<RuleId, HashMap<TokenId, Vec<usize>>>: Serialize<__S>, HashMap<String, License>: Serialize<__S>, HashMap<String, RuleId>: Serialize<__S>, Option<RuleId>: Serialize<__S>, HashMap<TokenId, HashSet<RuleId>>: Serialize<__S>, Option<String>: Serialize<__S>,

fn serialize( &self, serializer: &mut __S, ) -> Result<<Self as Archive>::Resolver, <__S as Fallible>::Error>

Writes the dependencies for the object and returns a resolver that can create the archived type.