pub struct PlainTokenizer {
pub default_language: Option<String>,
}Expand description
A tokenizer that knows no language: every code block is split into
plain tokens, so the indent and line-move handlers can work on it. The
default of CodeHighlighterConfig, as
Swift’s PlainCodeTokenizer is; JS’s default is PrismTokenizer, whose
grammars aren’t ported.
Fields§
§default_language: Option<String>defaultLanguage.
Trait Implementations§
Source§impl Clone for PlainTokenizer
impl Clone for PlainTokenizer
Source§impl Debug for PlainTokenizer
impl Debug for PlainTokenizer
Source§impl Default for PlainTokenizer
impl Default for PlainTokenizer
Source§impl Tokenizer for PlainTokenizer
impl Tokenizer for PlainTokenizer
Source§fn default_language(&self) -> Option<&str>
fn default_language(&self) -> Option<&str>
defaultLanguage: the language of a code block without one, which
the highlighter sets on the block. None (JS’s null) opts out:
such a block keeps no language and is split into plain tokens, so a
Markdown round trip keeps a fence without an info string.Source§fn language_state(&self, _language: &str) -> CodeLanguageState
fn language_state(&self, _language: &str) -> CodeLanguageState
Whether the tokenizer can highlight
language now. JS’s loop asks
the engine’s isCodeLanguageLoaded.Source§fn tokenize(&self, code: &str, _language: &str) -> Vec<CodeToken>
fn tokenize(&self, code: &str, _language: &str) -> Vec<CodeToken>
tokenize(code, language): code as tokens, for a loaded
language.Source§fn load_language(&self, language: &str, on_ready: Box<dyn FnOnce()>)
fn load_language(&self, language: &str, on_ready: Box<dyn FnOnce()>)
Loads
language, whose state is CodeLanguageState::Loading, and
calls on_ready once it is loaded, on the editor’s thread
(Shiki’s loadCodeLanguage); the highlighter then highlights the
blocks waiting for it. The default does nothing, for a tokenizer
whose languages are all loaded or unsupported.Source§fn tokenize_code_node(
&self,
code_node: &NodeRef,
language: Option<&str>,
) -> Vec<NodeRef>
fn tokenize_code_node( &self, code_node: &NodeRef, language: Option<&str>, ) -> Vec<NodeRef>
$tokenize(codeNode, language): the children that highlight
code_node. The default is PrismTokenizer.$tokenize’s:
language || defaultLanguage, and with neither the plain tokens of
the block’s text ($plainifyCodeContent), else
map_tokens_to_lexical_structure of
tokenize of it ($getHighlightNodes, without
Prism’s diff-<language> grammars).Auto Trait Implementations§
impl Forget for PlainTokenizer
impl Freeze for PlainTokenizer
impl Move for PlainTokenizer
impl RefUnwindSafe for PlainTokenizer
impl Send for PlainTokenizer
impl Sync for PlainTokenizer
impl Unpin for PlainTokenizer
impl UnsafeUnpin for PlainTokenizer
impl UnwindSafe for PlainTokenizer
Blanket Implementations§
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more