sakurs-core 0.4.0

High-performance sentence boundary detection using Delta-Stack Monoid algorithm
Documentation

sakurs-core

High-performance sentence boundary detection library using the Δ-Stack Monoid algorithm.

⚠️ API Stability Notice: This crate is pre-1.0. The current 0.3 series exposes a deliberately small public surface (the api module, re-exported at the crate root), so internal improvements no longer require breaking changes. Pin a minor version:

sakurs-core = "0.3"

Table of Contents

Features

  • Parallel Processing: near-linear multicore scaling using the Δ-Stack Monoid algorithm (measured: 252 MB/s single-threaded, 1.44 GB/s at 8 threads on plain English text)
  • Sequential Equivalence: any chunk size and thread count produce exactly the same boundaries as processing the whole text sequentially — a guaranteed, property-tested invariant
  • Language Support: English and Japanese bundled; new languages are compiled TOML configurations, no code required
  • Complex Text Support: handles nested quotes, abbreviations, and cross-chunk boundaries correctly, including candidates whose deciding context crosses a chunk edge

Quick Start

use sakurs_core::{SentenceProcessor, Input};

// Create processor with default configuration
let processor = SentenceProcessor::with_language("en")?;

// Process text
let text = "Hello world. This is a test.";
let output = processor.process(Input::from_text(text))?;

// Use the boundaries
for boundary in &output.boundaries {
    println!("Sentence ends at byte offset: {}", boundary.offset);
}

Advanced Usage

Custom Configuration

use sakurs_core::{Config, Input, SentenceProcessor};

let config = Config::builder()
    .language("ja")?          // Japanese language rules
    .threads(Some(4))         // Use 4 threads
    .chunk_size(512 * 1024)   // 512KB chunks
    .build()?;

let processor = SentenceProcessor::with_config(config)?;

Processing Files

use sakurs_core::{Input, SentenceProcessor};

let processor = SentenceProcessor::new();
let output = processor.process(Input::from_file("document.txt"))?;

println!("Found {} sentences", output.boundaries.len());
println!("Processing took {:?}", output.metadata.duration);

Streaming Preset

use sakurs_core::{Config, Input, SentenceProcessor};

// Use the streaming preset for 32KB algorithm chunks and limited threads
let processor = SentenceProcessor::with_config(Config::streaming())?;
let output = processor.process(Input::from_file("large_document.txt"))?;

Config::streaming(), Config::small_text(), and Config::large_text() are fixed presets for English; to combine a preset's chunk size/thread count with another language, use Config::builder() directly with the same chunk_size/threads values.

The streaming preset controls algorithm chunking and parallelism; it does not provide incremental I/O. Input::File and Input::Reader are currently read completely into memory before sentence detection.

Language Support

Currently bundled:

  • English (en)
  • Japanese (ja)

A language is a TOML configuration file compiled at load time into the algorithm's decision oracles — adding a language requires no code. See the main repository for documentation on adding new languages.

Algorithm

This library implements the Δ-Stack Monoid algorithm, which represents parsing state as an associative monoid and defers context-dependent decisions (abbreviations, sentence starters, enclosure suppression) whose window crosses a chunk edge until the neighboring chunk's context is available. This gives:

  1. Splitting text into chunks at arbitrary positions
  2. Processing chunks independently, in parallel
  3. Combining results in any order
  4. Guaranteed identical results to sequential processing

For detailed algorithm documentation, see the main repository.

License

MIT License. See LICENSE for details.

Links