1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
use bitmask;
/// A [bitmask][bitmask_enum] representing the characters that make up a [Kind::Whitespace][crate::Kind::Whitespace]
/// token.
///
/// A [Token][crate::Token] with [Kind::Whitespace][crate::Kind::Whitespace] will store this data internal to the
/// token. Using [Token::whitespace_style()][crate::Token::whitespace_style()] will return this bitmask, depending on
/// what characters make up the whitespace token. By default the [Lexer][crate::Lexer] will produce combine multiple
/// whitespaces into a single [Token][crate::Token], so it is possible that
/// [Token::whitespace_style()][crate::Token::whitespace_style()] could contain all of the available bits here. With
/// [Feature::SeparateWhitespace][crate::Feature::SeparateWhitespace] the [Lexer][crate::Lexer] will produce discrete
/// tokens each which can only have one of the available bits in this bitmask.
///
/// ```
/// use css_lexer::*;
/// let mut lexer = Lexer::new(&EmptyAtomSet::ATOMS, "\n\t");
/// {
/// // This token will be collapsed Whitespace.
/// let token = lexer.advance();
/// assert_eq!(token, Kind::Whitespace);
/// // The Whitespace is comprised of many bits:
/// assert_eq!(token, Whitespace::Newline | Whitespace::Tab);
/// }
/// ```