Crate cleft 

Unicode character properties, and grapheme, word and line segmentation.

The crate reads its own compact tables, generated from the Unicode Character Database. It is no_std, depends on nothing but core and alloc, and no lookup or scan allocates or panics, whatever the input.

Properties

core_props and rare_props return a character’s properties, each a single table lookup and a u32. CoreProps holds the ones a text layout reads for every character: the bidi class, the script and its extensions, paired brackets, and the emoji and break-related flags. RareProps holds the ones read less often: Grapheme_Cluster_Break, East_Asian_Width, Vertical_Orientation and the casing properties. line_break returns a character’s Line_Break.

Segmentation

Each segmenter reads a stream of characters with their offsets, such as Utf8Scalars or Utf16Scalars, and yields the offsets of the boundaries inside it.

• Graphemes finds extended grapheme clusters (UAX #29). GraphemeState is its state machine, for a scan that holds the characters already.
• LineBreaks finds line break opportunities (UAX #14), under the CSS tailorings LineOptions selects.
• WordBreaks finds word boundaries (UAX #29), with the options of WordOptions.

Thai, Lao, Khmer and Myanmar have no rules for breaks between words, and Chinese and Japanese words follow no rule either. Inside a run of one of them the line and word segmenters break at the words of a Dictionary, the longest at each position. Dictionaries holds one per script, compiled in with the dictionaries feature or loaded at run time.

use cleft::{LineBreaks, LineOptions, Utf8Scalars, WordBreaks, WordOptions};

let text = "Can't stop, 3.14!";
let lines: Vec<usize> = LineBreaks::new(Utf8Scalars::new(text), LineOptions::new()).collect();
assert_eq!(lines, [6, 12]);
let words: Vec<usize> = WordBreaks::new(Utf8Scalars::new(text), WordOptions::new()).collect();
assert_eq!(words, [5, 6, 10, 11, 12, 16]);

The data and rules are of the Unicode version UNICODE_VERSION names.

Features

No feature is on by default.

• css: the classes CSS defines and Unicode does not, read from the rare properties: EastAsianSpacing for text-autospace, TextSpacingClass for text-spacing-trim, EmphasisClass for text-emphasis-skip and FirstLetterClass for ::first-letter. Without it a smaller table is compiled, still read by one lookup.
• case: full case mapping, uppercase, lowercase and titlecase, with the language tailorings of CaseLanguage (about 5 KB of tables).
• dictionaries: the word dictionaries of Thai, Lao, Khmer, Myanmar, and Chinese and Japanese, generated from ICU’s word lists and compiled in (about 2.7 MB): Dictionaries::with_compiled.

License

The crate is licensed under MIT or Apache-2.0. Its tables are generated from the Unicode Character Database, under the Unicode License v3 (LICENSE-UNICODE). Its dictionaries are generated from ICU’s word lists, whose notices are in THIRD-PARTY.

Structs

BidiClass

A Bidi_Class, in ICU4C’s UCharDirection numbering.

BracketPairId

A bracket pair, which indexes the table of closing brackets.

CoreProps

The core properties of a character, packed in one word.

Dictionaries

The word dictionaries a segmenter reads, one per script, each optional.

Dictionary

One script’s words, as a validated automaton.

GraphemeProps

The properties UAX #29 reads of a character, packed in a byte.

GraphemeState

The state of a forward scan for grapheme cluster boundaries.

Graphemes

The offsets where the extended grapheme clusters of a stream of characters start.

LineBreak

A Line_Break value (UAX #14), in ICU4C’s ULineBreak numbering.

LineBreaks

The offsets where a line may break in a stream of characters.

LineOptions

The tailorings a LineBreaks applies. The default is plain UAX #14.

RareProps

The rare properties of a character, packed in one word.

ScriptId

A script in ICU4C’s UScriptCode numbering, which indexes the table of script tags.

ScriptWithExt

A character’s Script and Script_Extensions in ten bits, after ICU’s ScriptWithExtensions encoding.

Utf8Scalars

The characters of a UTF-8 string, with their byte offsets.

Utf16Scalars

The characters of a UTF-16 string, with their code unit offsets.

WordBreaks

The offsets of the word boundaries in a stream of characters.

WordOptions

The tailorings a WordBreaks applies. The default is plain UAX #29.

Enums

Bytes

The bytes of a dictionary: compiled in, or shared.

CaseLanguage

The rules a language maps case by.

DictionaryError

Why bytes are not a dictionary.

DictionaryScript

The script a dictionary finds words in.

EastAsianSpacing

A character’s class for text-autospace: UTR #59’s East_Asian_Spacing.

EmphasisClass

A character’s class for text-emphasis-skip (CSS Text Decoration 4, section 3.5), from its General_Category.

FirstLetterClass

A character’s class for ::first-letter, from its General_Category.

GraphemeClusterBreak

Grapheme_Cluster_Break (UAX #29), in ICU4C’s numbering except for Zwj.

LineStrictness

How strictly lines break, as CSS’s line-break sets it.

LineWordOption

Which breaks within words are allowed, as CSS’s word-break sets it.

TextSpacingClass

A character’s text-spacing class (CSS Text 4, section 8.5).

Constants

UNICODE_VERSION

The version of Unicode the properties and case mappings follow, as (major, minor, update).

Functions

core_props

Returns the core properties of ch: the ones a layout reads for every character.

grapheme_props

Returns the properties UAX #29 reads of ch.

line_break

Returns the Line_Break property of ch, as UAX #14 assigns it.

lowercase

Calls push with each character of the full lowercase of text.

rare_props

Returns the rare properties of ch: the ones a layout reads per cluster or behind a gate.

titlecase

Calls push with each character of segment with its first character titlecased and the rest as they are.

uppercase

Calls push with each character of the full uppercase of text.