Skip to main content

PlainTokenizer

Struct PlainTokenizer 

Source
pub struct PlainTokenizer {
    pub default_language: Option<String>,
}
Expand description

A tokenizer that knows no language: every code block is split into plain tokens, so the indent and line-move handlers can work on it. The default of CodeHighlighterConfig, as Swift’s PlainCodeTokenizer is; JS’s default is PrismTokenizer, whose grammars aren’t ported.

Fields§

§default_language: Option<String>

defaultLanguage.

Trait Implementations§

Source§

impl Clone for PlainTokenizer

Source§

fn clone(&self) -> Self

Returns a duplicate of the value. Read more
1.0.0 (const: unstable) · Source§

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source. Read more
Source§

impl Debug for PlainTokenizer

Source§

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter. Read more
Source§

impl Default for PlainTokenizer

Source§

fn default() -> Self

Returns the “default value” for a type. Read more
Source§

impl Tokenizer for PlainTokenizer

Source§

fn default_language(&self) -> Option<&str>

defaultLanguage: the language of a code block without one, which the highlighter sets on the block. None (JS’s null) opts out: such a block keeps no language and is split into plain tokens, so a Markdown round trip keeps a fence without an info string.
Source§

fn language_state(&self, _language: &str) -> CodeLanguageState

Whether the tokenizer can highlight language now. JS’s loop asks the engine’s isCodeLanguageLoaded.
Source§

fn tokenize(&self, code: &str, _language: &str) -> Vec<CodeToken>

tokenize(code, language): code as tokens, for a loaded language.
Source§

fn load_language(&self, language: &str, on_ready: Box<dyn FnOnce()>)

Loads language, whose state is CodeLanguageState::Loading, and calls on_ready once it is loaded, on the editor’s thread (Shiki’s loadCodeLanguage); the highlighter then highlights the blocks waiting for it. The default does nothing, for a tokenizer whose languages are all loaded or unsupported.
Source§

fn tokenize_code_node( &self, code_node: &NodeRef, language: Option<&str>, ) -> Vec<NodeRef>

$tokenize(codeNode, language): the children that highlight code_node. The default is PrismTokenizer.$tokenize’s: language || defaultLanguage, and with neither the plain tokens of the block’s text ($plainifyCodeContent), else map_tokens_to_lexical_structure of tokenize of it ($getHighlightNodes, without Prism’s diff-<language> grammars).

Auto Trait Implementations§

Blanket Implementations§

Source§

impl<T> Any for T
where T: 'static + ?Sized,

Source§

fn type_id(&self) -> TypeId

Gets the TypeId of self. Read more
Source§

impl<T> Borrow<T> for T
where T: ?Sized,

Source§

fn borrow(&self) -> &T

Immutably borrows from an owned value. Read more
Source§

impl<T> BorrowMut<T> for T
where T: ?Sized,

Source§

fn borrow_mut(&mut self) -> &mut T

Mutably borrows from an owned value. Read more
Source§

impl<T> CloneToUninit for T
where T: Clone,

Source§

unsafe fn clone_to_uninit(&self, dest: *mut u8)

🔬This is a nightly-only experimental API. (clone_to_uninit)
Performs copy-assignment from self to dest. Read more
Source§

impl<T> From<T> for T

Source§

fn from(t: T) -> T

Returns the argument unchanged.

Source§

impl<T, U> Into<U> for T
where U: From<T>,

Source§

fn into(self) -> U

Calls U::from(self).

That is, this conversion is whatever the implementation of From<T> for U chooses to do.

Source§

impl<T> ToOwned for T
where T: Clone,

Source§

type Owned = T

The resulting type after obtaining ownership.
Source§

fn to_owned(&self) -> T

Creates owned data from borrowed data, usually by cloning. Read more
Source§

fn clone_into(&self, target: &mut T)

Uses borrowed data to replace owned data, usually by cloning. Read more
Source§

impl<T, U> TryFrom<U> for T
where U: Into<T>,

Source§

type Error = !

The type returned in the event of a conversion error.
Source§

fn try_from(value: U) -> Result<T, !>

Performs the conversion.
Source§

impl<T, U> TryInto<U> for T
where U: TryFrom<T>,

Source§

type Error = <U as TryFrom<T>>::Error

The type returned in the event of a conversion error.
Source§

fn try_into(self) -> Result<U, <U as TryFrom<T>>::Error>

Performs the conversion.