Crate anyseg 

Segmentation of text into tokens and sentences with positions in the source text.

The text is never changed: a token or a sentence is a Span of byte offsets into the text it was cut from. Segmenter::segment gives a Document: tokens, words with the columns of CoNLL-U, sentences, paragraphs and lines, read from and written to CoNLL-U by read_conllu and Document::to_conllu. Every language is a Cargo feature: ru (Russian, on by default).

use anyseg::Segmenter;

let segmenter = Segmenter::new("ru").unwrap();
let text = "Мама мыла раму";
let tokens: Vec<&str> = segmenter.tokenize(text).iter().map(|span| span.slice(text)).collect();
assert_eq!(tokens, ["Мама", "мыла", "раму"]);
let doc = segmenter.segment(text);
assert_eq!(doc.sentence(0).span.slice(doc.text()), "Мама мыла раму");

Modules

cli

The anyseg command: segmentation of text from files and the standard input

Structs

ConlluError

A line of CoNLL-U that cannot be read

Document

A segmented text. Tokens are spans of the text, which is never changed; a token holds one word or, for a multiword token, several. Sentences hold whole tokens, paragraphs hold whole sentences, and lines are those of the paragraphs. Words have the columns of CoNLL-U, empty until set.

DocumentError

Wrong tokens, sentence starts or column values

Line

A line of a paragraph

Paragraph

A paragraph of a Document

Segmenter

Segmenter of texts in one language

Sentence

A sentence of a Document

Span

A piece of a text: &text[start..end], in bytes

UnknownLanguage

The language code is not one of Language::ALL

Enums

Column

A column of CoNLL-U that a word holds as text; head is a number and form comes from the tokens

InvalidEntry

A string that a list of the user cannot take

Language

A language whose conventions the segmenter follows

LineBreak

How a line of a paragraph begins after the line before it

Functions

read_conllu

The documents of a CoNLL-U text: a # newdoc comment starts a document and # newpar a paragraph. The text of a sentence is its # text comment or, without it, the forms of its tokens with the spaces of SpaceAfter and SpacesAfter; sentences of a paragraph are joined by a space (none after SpaceAfter=No) and paragraphs by a blank line. Comments, multiword token lines and empty nodes are kept, so that Document::to_conllu writes the text back as it was, with LF line ends and heads as plain numbers. A byte order mark at the start is skipped.