Module tokenizer 

Tokenizer for inline markdown content.

This module provides tokenization of markdown inline content, breaking text into tokens for formatting markers, text, and special elements.

Structs

Tokenizer

Tokenizer for inline markdown content.

Enums

Token

Token types for inline markdown content.

Functions

cjk_count

Count CJK characters in a string.

is_cjk

Check if a character is CJK (Chinese, Japanese, Korean).

not_text

Check if a token string is “not text” (is a potential marker boundary).