emoji-sanitize 0.1.0

Normalize or strip emoji-related Unicode (presentation selectors, variation selectors, zero-width joiners) from text before LLM input. Zero deps.
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60

//! # emoji-sanitize
//!
//! Strip Unicode characters that emoji rendering uses but plain-text
//! analysis doesn't need: variation selectors (U+FE00–U+FE0F),
//! the zero-width joiner (U+200D), and skin-tone modifiers
//! (U+1F3FB–U+1F3FF).
//!
//! Two modes:
//! - [`strip_modifiers`] — keep the base emoji codepoint, drop
//!   modifiers. `👨‍👩‍👧` becomes `👨👩👧`.
//! - [`strip_all_emoji`] — drop everything in the emoji blocks too.
//!
//! ## Example
//!
//! ```
//! use emoji_sanitize::{strip_modifiers, strip_all_emoji};
//! let s = "Hi \u{1F468}\u{200D}\u{1F469}\u{200D}\u{1F467}!";
//! assert_eq!(strip_modifiers(s), "Hi \u{1F468}\u{1F469}\u{1F467}!");
//! ```

#![deny(missing_docs)]

/// Strip variation selectors, ZWJ, and skin-tone modifiers — keeping
/// the base emoji codepoints.
pub fn strip_modifiers(s: &str) -> String {
    s.chars().filter(|c| !is_modifier(*c)).collect()
}

/// Strip everything we recognize as emoji or emoji-modifier.
pub fn strip_all_emoji(s: &str) -> String {
    s.chars()
        .filter(|c| !is_modifier(*c) && !is_emoji(*c))
        .collect()
}

/// True for emoji codepoints in the common ranges (BMP misc symbols,
/// SMP misc, SMP supplemental, emoticons, transport/map symbols).
pub fn is_emoji(c: char) -> bool {
    matches!(c as u32,
        0x1F300..=0x1F5FF // misc symbols / pictographs
        | 0x1F600..=0x1F64F // emoticons
        | 0x1F680..=0x1F6FF // transport / map
        | 0x1F700..=0x1F77F // alchemy
        | 0x1F780..=0x1F7FF // geometric
        | 0x1F800..=0x1F8FF // supplemental arrows-c
        | 0x1F900..=0x1F9FF // supplemental symbols/pictographs
        | 0x1FA00..=0x1FA6F // chess + extended
        | 0x1FA70..=0x1FAFF // symbols and pictographs extended-A
        | 0x2600..=0x26FF   // misc symbols
        | 0x2700..=0x27BF   // dingbats
    )
}

fn is_modifier(c: char) -> bool {
    matches!(c as u32,
        0x200D            // ZWJ
        | 0xFE00..=0xFE0F // variation selectors 1-16 (incl. VS16)
        | 0x1F3FB..=0x1F3FF // skin tones
    )
}