Expand description
Fast, lightweight, forward-only segmentation into Unicode extended grapheme clusters, as defined by UAX #29.
This crate does one thing: iterate extended grapheme clusters from the start of a string towards the end, as fast as the rules allow.
- Unicode 17.0, follow up-to-date standard.
- Extended grapheme clusters only. GB9a, GB9b and GB9c always apply.
no_std, nounsafe, zero dependencies.- Only ~19 KB of generated tables
It is a Rust port of the unicode-segmenter,
the JavaScript library this algorithm was originally developed for,
replicating the Intl.Segmenter behavior.
And it’s highly optmized for simple foward segmentation and counting grapheme clusters in a given text.
§Examples
use fast_grapheme_segmenter::graphemes;
let text = "a🇰🇷b";
let clusters: Vec<&str> = graphemes(text).collect();
assert_eq!(clusters, ["a", "🇰🇷", "b"]);Byte offsets of each cluster:
use fast_grapheme_segmenter::grapheme_indices;
let text = "a🇰🇷b";
let offsets: Vec<usize> = grapheme_indices(text).map(|(i, _)| i).collect();
assert_eq!(offsets, [0, 1, 9]);Or just the offsets, without slicing the clusters out:
use fast_grapheme_segmenter::grapheme_boundaries;
let offsets: Vec<usize> = grapheme_boundaries("a🇰🇷b").collect();
assert_eq!(offsets, [0, 1, 9]);Counting clusters, e.g. for a display-width-aware strlen:
use fast_grapheme_segmenter::count_graphemes;
assert_eq!(count_graphemes("🏳️🌈👩👩👧👦"), 2);Structs§
- Grapheme
Boundaries - External iterator over the byte offsets at which a string’s extended grapheme clusters start.
- Grapheme
Boundaries Utf16 - External iterator over the UTF-16 code unit offsets at which the extended grapheme clusters of a slice start.
- Grapheme
Indices - External iterator for a string’s extended grapheme clusters and their byte offsets.
- Graphemes
- External iterator for a string’s extended grapheme clusters.
Constants§
- UNICODE_
VERSION - The Unicode version whose data this crate’s tables encode.
Traits§
- Grapheme
Segmentation - Extension methods for segmenting a
strinto extended grapheme clusters.
Functions§
- count_
graphemes - Count the extended grapheme clusters of a string.
- count_
graphemes_ utf16 - Count the extended grapheme clusters of a potentially ill-formed UTF-16 slice,
as JavaScript and the
wasm32targets store their strings. - grapheme_
boundaries - Iterate over the byte offsets at which the extended grapheme clusters of a string start, without slicing the clusters out.
- grapheme_
boundaries_ utf16 - Iterate over the UTF-16 code unit offsets at which the extended grapheme clusters of a slice start, without decoding the clusters out.
- grapheme_
indices - Iterate over the extended grapheme clusters of a string with their byte offsets.
- graphemes
- Iterate over the extended grapheme clusters of a string.