Struct Dictionary 

pub struct Dictionary {
    pub prefix_dictionary: Arc<PrefixDictionary>,
    pub connection_cost_matrix: Arc<ConnectionCostMatrix>,
    pub character_definition: Arc<CharacterDefinition>,
    pub unknown_dictionary: Arc<UnknownDictionary>,
    pub metadata: Arc<Metadata>,
}

prefix_dictionary and connection_cost_matrix are Arc-wrapped so that Dictionary::clone() is O(1) regardless of load method (embedded, mmap, or plain filesystem read) – these two components dominate a dictionary’s memory footprint (tens to hundreds of MB), and nothing in this codebase mutates them after construction.

Fields

prefix_dictionary: Arc<PrefixDictionary>

connection_cost_matrix: Arc<ConnectionCostMatrix>

character_definition: Arc<CharacterDefinition>

unknown_dictionary: Arc<UnknownDictionary>

metadata: Arc<Metadata>

Implementations

impl Dictionary

pub fn unknown_word_details(&self, word_id: usize) -> Vec<&str>

Retrieve the detail fields (POS, etc.) for an unknown word entry.

引数

• word_id - The unknown-word entry id.

戻り値

A freshly allocated vector of the entry’s fields, or the UNK sentinel when the id is out of range or the entry is malformed. Prefer Dictionary::unknown_word_details_iter on per-token paths, which yields the same fields without allocating.

pub fn unknown_word_details_iter<'a>( &'a self, word_id: usize, ) -> DetailFields<'a>

Yields the detail fields of an unknown-word entry, borrowed from the dictionary’s own bytes.

引数

• word_id - The unknown-word entry id.

戻り値

The entry’s fields, or the UNK sentinel when the id is out of range or the entry is malformed – the fallback Dictionary::unknown_word_details has always applied.

pub fn word_details(&self, word_id: usize) -> Vec<&str>

Retrieve the detail fields (POS, etc.) for a system dictionary entry.

引数

• word_id - The system word id.

戻り値

A freshly allocated vector of the entry’s fields; empty when word_id is out of range, and the UNK sentinel when the entry is malformed. Prefer Dictionary::word_details_iter on per-token paths, which yields the same fields without allocating.

pub fn word_details_iter<'a>(&'a self, word_id: usize) -> DetailFields<'a>

Yields the detail fields of a system dictionary entry, borrowed from the dictionary’s own bytes.

引数

• word_id - The system word id.

戻り値

The entry’s fields; DetailFields::empty when word_id is out of range, and DetailFields::unk when the entry is malformed. Those two fallbacks differ, and both are what Dictionary::word_details has always returned.

pub fn load_from_path(dict_path: &Path) -> LinderaResult<Self>

Load dictionary from a directory containing dictionary files.

When the mmap feature is compiled in, the connection-cost matrix and word list are routed through memory-mapped reads by default (#879); use Dictionary::load_from_path_with_options with use_mmap = false to force eager reads.

pub fn load_from_path_with_options( dict_path: &Path, use_mmap: bool, ) -> LinderaResult<Self>

Load dictionary from a directory with options

use_mmap (when the mmap feature is enabled) routes connection_cost_matrix and prefix_dictionary through memory-mapped reads instead of plain file reads. What that buys differs per component:

• ConnectionCostMatrix reads its costs in place, with no copy and no anonymous memory: matrix.mtx already stores the values in the in-memory layout, and an mmap base is page-aligned, so the payload can be viewed as [i16] directly. Loading it is O(1) and the pages are faulted in lazily during tokenization. (Costs are also borrowed from a plain read’s buffer whenever it happens to be i16-aligned; the alignment is only guaranteed under mmap and for embedded data.)
• PrefixDictionary’s vals_data/words_idx_data/words_data are likewise mmap-backed and read lazily at lookup time.
• PrefixDictionary’s double-array trie (da) is still eagerly deserialized into owned daachorse structures, so for that component use_mmap only avoids the initial file-read syscall/allocation.

metadata, character_definition and unknown_dictionary are always plain-read regardless of this flag. Separately, Dictionary::clone() is O(1) regardless of use_mmap, since prefix_dictionary/connection_cost_matrix are Arc-wrapped.

pub fn save_to_path(&self, dict_path: &Path) -> LinderaResult<()>

Save dictionary to a directory

Trait Implementations

impl Clone for Dictionary

fn clone(&self) -> Dictionary

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.