Module words 

Word boundaries in a paragraph, as PowerPoint’s Ctrl+arrow, Ctrl+Backspace/Delete and double-click see them (observed black-box on PowerPoint for Windows, see docs/ui-parity.md, Text editing keys).

A paragraph splits into tokens. Each token is a run of one class of grapheme clusters plus the spaces that follow it:

• word: letters, digits and combining marks. An apostrophe (' or ’) joins a word when a letter or digit follows it (it's, 'tis, but dogs' ends before it); a hyphen joins when a letter or digit comes before it (a-b, 1-2, a-); ., , and : join between digits (3.14, 1,000, 3:45, but e.g. is five tokens).
• punctuation: everything else that is not a symbol; a run of it is one token (..., ?!, ),, 。。). _, @, /, #, quotes and brackets are punctuation.
• symbol: emoji and pictographs (a run of them is one token, 👍👍), ©, ®, ™.
• ideograph: each CJK ideograph, kana or Hangul syllable is a token of its own.
• tab and line break: each is a token of its own, and spaces after one form their own token instead of joining it.
• space: U+0020, no-break space, ideographic space and the other fixed-width spaces. Spaces join the token before them; spaces at the start of a paragraph (or after a tab or line break) are a token of their own.

Positions are character (Unicode scalar) offsets in the paragraph text, where a line break is one \u{b} character. A token never splits a grapheme cluster, so the caret never lands inside an emoji sequence or between a letter and its combining marks.

Functions

back_stops

The stops Ctrl+Left uses. PowerPoint’s backward scan differs from the forward one in two observed details: a no-break or other non-ASCII space is a stop of its own (unless an ASCII space follows it), and a symbol followed by punctuation (👍?) is one token.

delete_forward_end

What Ctrl+Delete removes from ch (before smart spacing): inside a token, the rest of its text; on spaces right after a token, those spaces and the next token’s text; anywhere else up to the next stop. None at the end of the text.

is_closing_punct

Closing punctuation that a smart delete pulls its word’s spaces up to (ab cd, ef → Ctrl+Delete on cd gives ab, ef).

is_sentence_punct

Punctuation that a smart delete keeps a space after when a word follows (cd...ef gh → Ctrl+Delete on ef gives cd... gh).

is_space

A space for word movement: absorbed into the token before it.

next_stop

The next stop after ch (Ctrl+Right), or None at the end of the text.

prev_stop

The previous stop before ch (Ctrl+Left), or None at the start.

stops

The word stops of a paragraph’s text in ascending order: 0, the start of every token, and the end of the text. Ctrl+Right goes to the next one.

word_at

The word a double-click at ch selects: the token under the caret without its trailing spaces; on a space just after a token, that token; in a run of spaces, the spaces.