sentencize
Rule-based sentence boundary detection for English text — a pure-Rust,
zero-dependency answer to the segmentation step usually delegated to Python
(nltk / pysbd) in document, RAG and LLM-chunking pipelines.
- Zero dependencies,
#![forbid(unsafe_code)], builds on Rust 1.75+. - Lossless: returns byte spans into the original text, so you can map a sentence back to its exact position in the source document.
- Allocation-light: sentences are borrowed
&strslices, not copies. - Configurable: bring your own abbreviation list, or extend the built-in one.
Installation
Or in Cargo.toml:
[]
= "0.1"
Quick start
let sentences = split;
assert_eq!;
What it handles
Naive text.split('.') breaks on almost every real document. sentencize
applies rules for the classic pitfalls:
| Case | Input | Result |
|---|---|---|
| Abbreviations | Dr. Smith met Mr. Jones. They talked. |
2 sentences |
| Decimals & versions | Pi is 3.14 roughly. Use version 1.2.3 now. |
2 sentences |
| Initials | J. R. R. Tolkien wrote it. I read it. |
2 sentences |
| Ellipses | Then we began... or tried to. |
1 sentence |
| Multiple terminators | Really?! Yes!!! Ok. |
3 sentences |
| Quotes after a terminator | He said "Stop." She refused. |
2 sentences |
A period only ends a sentence when it is followed by whitespace and a plausible
sentence opener (an uppercase letter, a digit, or an opening quote/bracket) —
which is what keeps e.g. apples and Sunset Blvd. Then from being mis-split.
Byte spans
spans gives you (start, end) byte offsets instead of slices. They are
trimmed of surrounding whitespace and never overlap, so they are ideal for
annotating or highlighting the source document:
use Segmenter;
let text = " One. Two! ";
let seg = default;
let spans = seg.spans;
assert_eq!;
assert_eq!;
assert_eq!;
Custom abbreviations
The built-in list covers common English titles, months, weekdays, units and
dotted forms (e.g, i.e, u.s, ph.d, a.m …). Extend it for your domain:
use Segmenter;
let mut seg = default;
seg.add_abbreviation;
let sentences: = seg.sentences.collect;
assert_eq!;
Or replace the list entirely with [Segmenter::new], which takes any iterator
of lowercase abbreviations written without their trailing period.
API
| Item | Purpose |
|---|---|
sentencize::split(text) |
One-shot split with the default English rules; returns Vec<&str>. |
Segmenter::default() |
Segmenter with the built-in abbreviation list. |
Segmenter::new(abbrs) |
Segmenter with a caller-supplied abbreviation list. |
Segmenter::add_abbreviation(a) |
Add one abbreviation to an existing segmenter. |
Segmenter::sentences(text) |
Iterator of &str sentence slices. |
Segmenter::spans(text) |
Vec<(usize, usize)> byte spans into the original text. |
Reuse a single Segmenter across many documents — building the default one
allocates its abbreviation set, so constructing it per call is wasteful.
Example
Scope
sentencize targets English prose with rule-based heuristics. It is
deliberately not a statistical model: no training data, no dependencies, fully
deterministic output. Text in other Latin-script languages generally works, but
the abbreviation list is English by default.
License
Licensed under either of
- Apache License, Version 2.0 (LICENSE-APACHE)
- MIT license (LICENSE-MIT)
at your option.
Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in this crate by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.