cleft 0.1.0

Unicode character properties, and grapheme, word and line segmentation with dictionaries, case mapping and CSS classes, for no_std.
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66

# cleft

Unicode character properties, and grapheme, word and line segmentation.

cleft reads its own compact tables, generated from the Unicode Character
Database. It is `no_std`, has no dependencies, and no lookup or scan
allocates or panics, whatever the input.

- **Properties.** `core_props` and `rare_props` return a character's
  properties in one table lookup each: the bidi class, the script and its
  extensions, paired brackets, emoji and break-related flags,
  Grapheme_Cluster_Break, East_Asian_Width, Vertical_Orientation and the
  casing properties.
- **Segmentation.** `Graphemes` finds extended grapheme clusters and
  `WordBreaks` word boundaries (UAX #29), and `LineBreaks` finds line break
  opportunities (UAX #14) under the tailorings of CSS's `line-break` and
  `word-break`. Each reads UTF-8 or UTF-16 text.
- **Dictionaries.** Thai, Lao, Khmer, Myanmar, Chinese and Japanese break at
  the words of a dictionary, compiled in or loaded at run time.

```rust
use cleft::{Graphemes, LineBreaks, LineOptions, Utf8Scalars};

// The offsets where a line may break.
let text = "Can't stop, 3.14!";
let breaks: Vec<usize> = LineBreaks::new(Utf8Scalars::new(text), LineOptions::new()).collect();
assert_eq!(breaks, [6, 12]);

// The offsets where grapheme clusters start.
let text = "a\u{308}🇫🇷!";
let starts: Vec<usize> = Graphemes::new(Utf8Scalars::new(text)).collect();
assert_eq!(starts, [0, 3, 11]);
```

## Features

No feature is on by default.

| Feature | Adds |
|---|---|
| `css` | The character classes CSS defines and Unicode does not, for `text-autospace`, `text-spacing-trim`, `text-emphasis-skip` and `::first-letter`. |
| `case` | Full case mapping with language tailorings: `uppercase`, `lowercase` and `titlecase`. |
| `dictionaries` | The word dictionaries of Thai, Lao, Khmer, Myanmar, and Chinese and Japanese, compiled in (about 2.7 MB). |

## Unicode version

The tables and rules are of Unicode 18.0.0, which `UNICODE_VERSION` names.

## Minimum Rust version

cleft builds with Rust 1.88 and later.

## License

cleft's code is licensed under either of

- Apache License, Version 2.0 ([LICENSE-APACHE](LICENSE-APACHE))
- MIT license ([LICENSE-MIT](LICENSE-MIT))

at your option.

Its property and case tables are generated from the Unicode Character
Database, under the Unicode License v3 ([LICENSE-UNICODE](LICENSE-UNICODE)).
Its dictionaries are generated from ICU's word lists, under ICU's license
and the notices of the lists' sources, reproduced in
[THIRD-PARTY](THIRD-PARTY).