Expand description
§sentencize
Rule-based sentence boundary detection for English text — a pure-Rust, zero-dependency answer to the segmentation step usually delegated to Python (nltk / pysbd) in document, RAG and LLM-chunking pipelines.
Handles the classic pitfalls of naive split('.'):
- abbreviations (
Dr.,e.g.,U.S.), with a customizable list - decimal numbers (
3.14) and version-like tokens (v1.2.3) - ellipses (
.../…) - closing quotes and brackets after terminators (
He said "Go.") - initials (
J. R. R. Tolkien)
The splitter returns byte spans into the original text, so it is lossless and allocation-light.
§Quick start
split applies the built-in English rules in one call and borrows from
the input:
let sentences = sentencize::split("Hello world. How are you? I am fine!");
assert_eq!(sentences, ["Hello world.", "How are you?", "I am fine!"]);§Reusing a segmenter
Segmenter holds the abbreviation set, so build it once and reuse it
across documents rather than calling split in a hot loop:
use sentencize::Segmenter;
let seg = Segmenter::default();
let text = "Dr. Smith arrived at 3.5 p.m. sharp. He said \"Sit down.\" Then we began... or tried to.";
let sentences: Vec<&str> = seg.sentences(text).collect();
assert_eq!(sentences, [
"Dr. Smith arrived at 3.5 p.m. sharp.",
"He said \"Sit down.\"",
"Then we began... or tried to.",
]);§Byte spans
Segmenter::spans returns whitespace-trimmed, non-overlapping (start, end) offsets, which map each sentence back to its exact position in the
source document:
use sentencize::Segmenter;
let text = " One. Two! ";
let spans = Segmenter::default().spans(text);
assert_eq!(&text[spans[0].0..spans[0].1], "One.");
assert_eq!(&text[spans[1].0..spans[1].1], "Two!");§Custom abbreviations
Extend the built-in list with Segmenter::add_abbreviation, or replace it
wholesale with Segmenter::new. Entries are lowercase and written without
their trailing period:
use sentencize::Segmenter;
let mut seg = Segmenter::default();
seg.add_abbreviation("blvd");
let sentences: Vec<&str> = seg.sentences("Turn at Sunset Blvd. Then stop.").collect();
assert_eq!(sentences, ["Turn at Sunset Blvd. Then stop."]);§Scope
These are deterministic heuristics tuned for English prose — no model, no training data, no dependencies. Other Latin-script languages generally work, but the default abbreviation list is English.
Structs§
- Segmenter
- A configurable sentence segmenter.
Functions§
- split
- Convenience: split with the default English segmenter.