Struct Tokenizer 

pub struct Tokenizer {
    pub process_links: bool,
    pub process_images: bool,
}

Tokenizer for inline markdown content.

Fields

process_links: bool

Whether to process links

process_images: bool

Whether to process images

Implementations

impl Tokenizer

pub fn new() -> Self

Create a new tokenizer with default settings.

pub fn with_settings(process_links: bool, process_images: bool) -> Self

Create a tokenizer with specific settings.

pub fn tokenize(&self, line: &str) -> Vec<Token>

Tokenize a line of markdown content.

This extracts links, images, footnotes, and inline formatting markers.

pub fn tokenize_inline(&self, text: &str, tokens: &mut Vec<Token>)

Tokenize inline content for formatting markers.

Trait Implementations

impl Debug for Tokenizer

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for Tokenizer

fn default() -> Tokenizer

Returns the “default value” for a type.