Trait NaturalLanguageParser 

pub trait NaturalLanguageParser: Send + Sync {
    // Required methods
    fn tokenize<'a>(
        &self,
        value: &'a str,
        offset: usize,
        alloc: &'a Allocator,
    ) -> ArenaVec<'a, Node<'a>>;
    fn tokenize_paragraph<'a>(
        &self,
        content: ArenaVec<'a, Node<'a>>,
        position: Option<Position<'a>>,
        alloc: &'a Allocator,
    ) -> Node<'a>;
    fn tokenize_root<'a>(
        &self,
        paragraph: Node<'a>,
        position: Option<Position<'a>>,
        alloc: &'a Allocator,
    ) -> Node<'a>;
}

A natural-language parser, as parse-latin’s instance is to mdast-util-to-nlcst and hast-util-to-nlcst: bridges into nlcst tokenize text with it and run its plugins.

Required Methods

fn tokenize<'a>( &self, value: &'a str, offset: usize, alloc: &'a Allocator, ) -> ArenaVec<'a, Node<'a>>

Split value into sentence content: words, white space, punctuation and symbols, each spanning its bytes from offset on.

fn tokenize_paragraph<'a>( &self, content: ArenaVec<'a, Node<'a>>, position: Option<Position<'a>>, alloc: &'a Allocator, ) -> Node<'a>

Run the sentence plugins over content as one sentence, split it into sentences after terminal markers and run the paragraph plugins: a ParagraphNode. The sentence and the paragraph start at position; without one, at their children’s.

fn tokenize_root<'a>( &self, paragraph: Node<'a>, position: Option<Position<'a>>, alloc: &'a Allocator, ) -> Node<'a>

Split a paragraph into paragraphs at white space holding a line ending and run the root plugins: a RootNode, at position or its children’s.

Dyn Compatibility

This trait is dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementors