Skip to main content

Crate sentencize

Crate sentencize 

Source
Expand description

§sentencize

Rule-based sentence boundary detection for English text — a pure-Rust, zero-dependency answer to the segmentation step usually delegated to Python (nltk / pysbd) in document, RAG and LLM-chunking pipelines.

Handles the classic pitfalls of naive split('.'):

  • abbreviations (Dr., e.g., U.S.), with a customizable list
  • decimal numbers (3.14) and version-like tokens (v1.2.3)
  • ellipses (... / …)
  • closing quotes and brackets after terminators (He said "Go.")
  • initials (J. R. R. Tolkien)

The splitter returns byte spans into the original text, so it is lossless and allocation-light.

§Quick start

split applies the built-in English rules in one call and borrows from the input:

let sentences = sentencize::split("Hello world. How are you? I am fine!");
assert_eq!(sentences, ["Hello world.", "How are you?", "I am fine!"]);

§Reusing a segmenter

Segmenter holds the abbreviation set, so build it once and reuse it across documents rather than calling split in a hot loop:

use sentencize::Segmenter;

let seg = Segmenter::default();
let text = "Dr. Smith arrived at 3.5 p.m. sharp. He said \"Sit down.\" Then we began... or tried to.";
let sentences: Vec<&str> = seg.sentences(text).collect();
assert_eq!(sentences, [
    "Dr. Smith arrived at 3.5 p.m. sharp.",
    "He said \"Sit down.\"",
    "Then we began... or tried to.",
]);

§Byte spans

Segmenter::spans returns whitespace-trimmed, non-overlapping (start, end) offsets, which map each sentence back to its exact position in the source document:

use sentencize::Segmenter;

let text = "  One.  Two!  ";
let spans = Segmenter::default().spans(text);
assert_eq!(&text[spans[0].0..spans[0].1], "One.");
assert_eq!(&text[spans[1].0..spans[1].1], "Two!");

§Custom abbreviations

Extend the built-in list with Segmenter::add_abbreviation, or replace it wholesale with Segmenter::new. Entries are lowercase and written without their trailing period:

use sentencize::Segmenter;

let mut seg = Segmenter::default();
seg.add_abbreviation("blvd");
let sentences: Vec<&str> = seg.sentences("Turn at Sunset Blvd. Then stop.").collect();
assert_eq!(sentences, ["Turn at Sunset Blvd. Then stop."]);

§Scope

These are deterministic heuristics tuned for English prose — no model, no training data, no dependencies. Other Latin-script languages generally work, but the default abbreviation list is English.

Structs§

Segmenter
A configurable sentence segmenter.

Functions§

split
Convenience: split with the default English segmenter.