Skip to main content

Crate fast_grapheme_segmenter

Crate fast_grapheme_segmenter 

Source
Expand description

Fast, lightweight, forward-only segmentation into Unicode extended grapheme clusters, as defined by UAX #29.

This crate does one thing: iterate extended grapheme clusters from the start of a string towards the end, as fast as the rules allow.

  • Unicode 17.0, follow up-to-date standard.
  • Extended grapheme clusters only. GB9a, GB9b and GB9c always apply.
  • no_std, no unsafe, zero dependencies.
  • Only ~19 KB of generated tables

It is a Rust port of the unicode-segmenter, the JavaScript library this algorithm was originally developed for, replicating the Intl.Segmenter behavior.

And it’s highly optmized for simple foward segmentation and counting grapheme clusters in a given text.

§Examples

use fast_grapheme_segmenter::graphemes;

let text = "a🇰🇷b";
let clusters: Vec<&str> = graphemes(text).collect();
assert_eq!(clusters, ["a", "🇰🇷", "b"]);

Byte offsets of each cluster:

use fast_grapheme_segmenter::grapheme_indices;

let text = "a🇰🇷b";
let offsets: Vec<usize> = grapheme_indices(text).map(|(i, _)| i).collect();
assert_eq!(offsets, [0, 1, 9]);

Or just the offsets, without slicing the clusters out:

use fast_grapheme_segmenter::grapheme_boundaries;

let offsets: Vec<usize> = grapheme_boundaries("a🇰🇷b").collect();
assert_eq!(offsets, [0, 1, 9]);

Counting clusters, e.g. for a display-width-aware strlen:

use fast_grapheme_segmenter::count_graphemes;

assert_eq!(count_graphemes("🏳️‍🌈👩‍👩‍👧‍👦"), 2);

Structs§

GraphemeBoundaries
External iterator over the byte offsets at which a string’s extended grapheme clusters start.
GraphemeBoundariesUtf16
External iterator over the UTF-16 code unit offsets at which the extended grapheme clusters of a slice start.
GraphemeIndices
External iterator for a string’s extended grapheme clusters and their byte offsets.
Graphemes
External iterator for a string’s extended grapheme clusters.

Constants§

UNICODE_VERSION
The Unicode version whose data this crate’s tables encode.

Traits§

GraphemeSegmentation
Extension methods for segmenting a str into extended grapheme clusters.

Functions§

count_graphemes
Count the extended grapheme clusters of a string.
count_graphemes_utf16
Count the extended grapheme clusters of a potentially ill-formed UTF-16 slice, as JavaScript and the wasm32 targets store their strings.
grapheme_boundaries
Iterate over the byte offsets at which the extended grapheme clusters of a string start, without slicing the clusters out.
grapheme_boundaries_utf16
Iterate over the UTF-16 code unit offsets at which the extended grapheme clusters of a slice start, without decoding the clusters out.
grapheme_indices
Iterate over the extended grapheme clusters of a string with their byte offsets.
graphemes
Iterate over the extended grapheme clusters of a string.