Expand description
§plato-tile-split
Text chunking engine. Splits large text into tiles with token-aware boundaries, overlap for context preservation, and code-aware splitting.
§Why Rust
Text splitting is CPU-bound string processing. Python’s string slicing creates new objects on every operation. Rust’s &str slicing is zero-copy.
| Metric | Python (str.split) | Rust (&str.split) |
|---|---|---|
| Split 1MB text | ~15ms | ~2ms |
| Memory per chunk | ~200 bytes (str obj) | ~40 bytes (&str + Vec) |
The zero-copy nature of Rust string slicing is the key advantage here.
Structs§
- Chunk
- A text chunk produced by splitting.
- Split
Config - Split configuration.
- Split
Stats - Split statistics.
- Tile
Split - The chunking engine.