Crate khmer_tokenizer_core 

khmer-tokenizer-core

A fast, dependency-free Khmer word segmenter.

Written Khmer does not put spaces between words, so segmentation is the first step of nearly every Khmer NLP pipeline. This crate does it in two passes:

1. Cluster pass — split_kcc groups the text into Khmer Character Clusters (a base character plus its subscripts and vowels). This keeps the segmenter from ever splitting inside an orthographic syllable.
2. Boundary pass — KhmerTokenizer walks a cluster-keyed trie to place word boundaries, using one of a few Strategy algorithms (default: greedy longest-match, falling back to a single cluster when nothing matches).

The engine is std-only (no external dependencies) and deterministic.

Quick start

use khmer_tokenizer_core::KhmerTokenizer;

let tk = KhmerTokenizer::with_default_dict();
let tokens = tk.segment("សួស្តីអ្នកទាំងអស់គ្នា");
assert_eq!(tokens, vec!["សួស្តី", "អ្នក", "ទាំងអស់គ្នា"]);

Structs

HmmModel

A trained BMES Hidden Markov Model for segmenting clusters the dictionary has no match for at all.

KhmerTokenizer

Dictionary-backed Khmer word segmenter.

Enums

Strategy

Which segmentation algorithm KhmerTokenizer::segment uses.

Constants

DEFAULT_DICT

The embedded default dictionary (59,526 words; see ATTRIBUTION.md): one word per line; blank lines and lines starting with # are ignored. Replace or extend it for your own use case — see the dictionary notes in the project README.

Functions

is_khmer

True if c falls anywhere in the Khmer Unicode block (U+1780..=U+17FF).

normalize

Canonicalize text’s Khmer encoding before segmentation. Idempotent: normalizing already-canonical (or already-normalized) text is a no-op. Byte-length-preserving: only reorders characters, never adds or removes any, so span offsets over the original text stay valid.

split_kcc

Split text into Khmer Character Clusters.