Skip to main content

Tokenizer

Trait Tokenizer 

Source
pub trait Tokenizer {
    // Required methods
    fn default_language(&self) -> Option<&str>;
    fn language_state(&self, language: &str) -> CodeLanguageState;
    fn tokenize(&self, code: &str, language: &str) -> Vec<CodeToken>;

    // Provided methods
    fn load_language(&self, language: &str, on_ready: Box<dyn FnOnce()>) { ... }
    fn tokenize_code_node(
        &self,
        code_node: &NodeRef,
        language: Option<&str>,
    ) -> Vec<NodeRef> { ... }
}
Expand description

Tokenizer: what the highlighter asks a highlighting engine.

Required Methods§

Source

fn default_language(&self) -> Option<&str>

defaultLanguage: the language of a code block without one, which the highlighter sets on the block. None (JS’s null) opts out: such a block keeps no language and is split into plain tokens, so a Markdown round trip keeps a fence without an info string.

Source

fn language_state(&self, language: &str) -> CodeLanguageState

Whether the tokenizer can highlight language now. JS’s loop asks the engine’s isCodeLanguageLoaded.

Source

fn tokenize(&self, code: &str, language: &str) -> Vec<CodeToken>

tokenize(code, language): code as tokens, for a loaded language.

Provided Methods§

Source

fn load_language(&self, language: &str, on_ready: Box<dyn FnOnce()>)

Loads language, whose state is CodeLanguageState::Loading, and calls on_ready once it is loaded, on the editor’s thread (Shiki’s loadCodeLanguage); the highlighter then highlights the blocks waiting for it. The default does nothing, for a tokenizer whose languages are all loaded or unsupported.

Source

fn tokenize_code_node( &self, code_node: &NodeRef, language: Option<&str>, ) -> Vec<NodeRef>

$tokenize(codeNode, language): the children that highlight code_node. The default is PrismTokenizer.$tokenize’s: language || defaultLanguage, and with neither the plain tokens of the block’s text ($plainifyCodeContent), else map_tokens_to_lexical_structure of tokenize of it ($getHighlightNodes, without Prism’s diff-<language> grammars).

Dyn Compatibility§

This trait is dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementors§