pub trait Tokenizer {
// Required methods
fn default_language(&self) -> Option<&str>;
fn language_state(&self, language: &str) -> CodeLanguageState;
fn tokenize(&self, code: &str, language: &str) -> Vec<CodeToken>;
// Provided methods
fn load_language(&self, language: &str, on_ready: Box<dyn FnOnce()>) { ... }
fn tokenize_code_node(
&self,
code_node: &NodeRef,
language: Option<&str>,
) -> Vec<NodeRef> { ... }
}Expand description
Tokenizer: what the highlighter asks a highlighting engine.
Required Methods§
Sourcefn default_language(&self) -> Option<&str>
fn default_language(&self) -> Option<&str>
defaultLanguage: the language of a code block without one, which
the highlighter sets on the block. None (JS’s null) opts out:
such a block keeps no language and is split into plain tokens, so a
Markdown round trip keeps a fence without an info string.
Sourcefn language_state(&self, language: &str) -> CodeLanguageState
fn language_state(&self, language: &str) -> CodeLanguageState
Whether the tokenizer can highlight language now. JS’s loop asks
the engine’s isCodeLanguageLoaded.
Provided Methods§
Sourcefn load_language(&self, language: &str, on_ready: Box<dyn FnOnce()>)
fn load_language(&self, language: &str, on_ready: Box<dyn FnOnce()>)
Loads language, whose state is CodeLanguageState::Loading, and
calls on_ready once it is loaded, on the editor’s thread
(Shiki’s loadCodeLanguage); the highlighter then highlights the
blocks waiting for it. The default does nothing, for a tokenizer
whose languages are all loaded or unsupported.
Sourcefn tokenize_code_node(
&self,
code_node: &NodeRef,
language: Option<&str>,
) -> Vec<NodeRef>
fn tokenize_code_node( &self, code_node: &NodeRef, language: Option<&str>, ) -> Vec<NodeRef>
$tokenize(codeNode, language): the children that highlight
code_node. The default is PrismTokenizer.$tokenize’s:
language || defaultLanguage, and with neither the plain tokens of
the block’s text ($plainifyCodeContent), else
map_tokens_to_lexical_structure of
tokenize of it ($getHighlightNodes, without
Prism’s diff-<language> grammars).
Dyn Compatibility§
This trait is dyn compatible.
In older versions of Rust, dyn compatibility was called "object safety".