sentencize 0.1.0

Rule-based sentence boundary detection: split text into sentences with abbreviation, decimal, ellipsis and quote handling. Zero dependencies.
Documentation
  • Coverage
  • 100%
    7 out of 7 items documented1 out of 7 items with examples
  • Size
  • Source code size: 34.8 kB This is the summed size of all the files inside the crates.io package for this release.
  • Documentation size: 134.0 kB This is the summed size of all files generated by rustdoc for all configured targets
  • Ø build duration
  • this release: 3s Average build duration of successful builds.
  • all releases: 3s Average build duration of successful builds in releases after 2024-10-23.
  • Links
  • Homepage
  • faizanali2k05/sentencize
    16 0 0
  • crates.io
  • Dependencies
  • Versions
  • Owners
  • faizanali2k05

sentencize

crates.io docs.rs license

Rule-based sentence boundary detection for English text — a pure-Rust, zero-dependency answer to the segmentation step usually delegated to Python (nltk / pysbd) in document, RAG and LLM-chunking pipelines.

  • Zero dependencies, #![forbid(unsafe_code)], builds on Rust 1.75+.
  • Lossless: returns byte spans into the original text, so you can map a sentence back to its exact position in the source document.
  • Allocation-light: sentences are borrowed &str slices, not copies.
  • Configurable: bring your own abbreviation list, or extend the built-in one.

Installation

cargo add sentencize

Or in Cargo.toml:

[dependencies]
sentencize = "0.1"

Quick start

let sentences = sentencize::split("Dr. Smith left at 3.5 p.m. sharp. He said \"Go.\"");

assert_eq!(sentences, [
    "Dr. Smith left at 3.5 p.m. sharp.",
    "He said \"Go.\"",
]);

What it handles

Naive text.split('.') breaks on almost every real document. sentencize applies rules for the classic pitfalls:

Case Input Result
Abbreviations Dr. Smith met Mr. Jones. They talked. 2 sentences
Decimals & versions Pi is 3.14 roughly. Use version 1.2.3 now. 2 sentences
Initials J. R. R. Tolkien wrote it. I read it. 2 sentences
Ellipses Then we began... or tried to. 1 sentence
Multiple terminators Really?! Yes!!! Ok. 3 sentences
Quotes after a terminator He said "Stop." She refused. 2 sentences

A period only ends a sentence when it is followed by whitespace and a plausible sentence opener (an uppercase letter, a digit, or an opening quote/bracket) — which is what keeps e.g. apples and Sunset Blvd. Then from being mis-split.

Byte spans

spans gives you (start, end) byte offsets instead of slices. They are trimmed of surrounding whitespace and never overlap, so they are ideal for annotating or highlighting the source document:

use sentencize::Segmenter;

let text = "  One.  Two!  ";
let seg = Segmenter::default();
let spans = seg.spans(text);

assert_eq!(spans.len(), 2);
assert_eq!(&text[spans[0].0..spans[0].1], "One.");
assert_eq!(&text[spans[1].0..spans[1].1], "Two!");

Custom abbreviations

The built-in list covers common English titles, months, weekdays, units and dotted forms (e.g, i.e, u.s, ph.d, a.m …). Extend it for your domain:

use sentencize::Segmenter;

let mut seg = Segmenter::default();
seg.add_abbreviation("blvd");

let sentences: Vec<&str> = seg.sentences("Turn at Sunset Blvd. Then stop.").collect();
assert_eq!(sentences, ["Turn at Sunset Blvd. Then stop."]);

Or replace the list entirely with [Segmenter::new], which takes any iterator of lowercase abbreviations written without their trailing period.

API

Item Purpose
sentencize::split(text) One-shot split with the default English rules; returns Vec<&str>.
Segmenter::default() Segmenter with the built-in abbreviation list.
Segmenter::new(abbrs) Segmenter with a caller-supplied abbreviation list.
Segmenter::add_abbreviation(a) Add one abbreviation to an existing segmenter.
Segmenter::sentences(text) Iterator of &str sentence slices.
Segmenter::spans(text) Vec<(usize, usize)> byte spans into the original text.

Reuse a single Segmenter across many documents — building the default one allocates its abbreviation set, so constructing it per call is wasteful.

Example

cargo run --example basic

Scope

sentencize targets English prose with rule-based heuristics. It is deliberately not a statistical model: no training data, no dependencies, fully deterministic output. Text in other Latin-script languages generally works, but the abbreviation list is English by default.

License

Licensed under either of

at your option.

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in this crate by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.