rust-pragmatic-segmenter
Rust port of pySBD v3.1.0 and Ruby pragmatic_segmenter. Documentations
rust-pragmatic-segmenter is rule-based SBD. It uses a lot of regular expressions to separate sentences.
use Segmenter;
let segmenter = new?;
let result: = segmenter.segment.collect;
//=> vec!["Hi Mr. Kim. ", "Let's meet at 3 P.M."]
How to build
TODOs
- Perfectly match the behavior with pySBD (current: 99%)
- Support languages other than English
- Remove regexes with look around and back references
- Try Intel Hyperscan
- Fix mistakes of pySBD, possibly send patches to the upstream
- Optimize copies and allocations
- Use proper error types instead of Boxed error
- Import test cases from pySBD and ruby pragmatic_segmenter
rust-pragmatic-segmenter is primarily distributed under the terms of both the Apache License (Version 2.0) and the MIT license. See COPYRIGHT for details.