rustling 0.4.0

A blazingly fast library for computational linguistics
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58

# Rustling

[![PyPI](https://img.shields.io/pypi/v/rustling.svg)](https://pypi.org/project/rustling/)
[![crates.io](https://img.shields.io/crates/v/rustling.svg)](https://crates.io/crates/rustling)

Rustling is a blazingly fast library for computational linguistics.
It is written in Rust, with Python bindings.

Documentation: [Python](https://rustling.readthedocs.io/) | [Rust](https://docs.rs/rustling)

## Features

- **Language Models** — N-gram language models with smoothing
  - `MLE` — Maximum Likelihood Estimation (no smoothing)
  - `Lidstone` — Lidstone (additive) smoothing
  - `Laplace` — Laplace (add-one) smoothing

- **Word Segmentation** — Models for segmenting unsegmented text into words
  - `LongestStringMatching` — Greedy left-to-right longest match segmenter
  - `RandomSegmenter` — Random baseline segmenter

- **Part-of-speech Tagging**
  - `AveragedPerceptronTagger` - Averaged perceptron tagger


## Performance

Benchmarked against pure Python implementataions from NLTK and wordseg.
See [`benchmarks/`](benchmarks/) for full details and reproduction scripts.

| Component | Task | Speedup | vs. |
|-----------|------|---------|-----|
| **Language Models** | Fit | **11x** | NLTK |
| | Score | **2x** | NLTK |
| | Generate | **25–39x** | NLTK |
| **Word Segmentation** | LongestStringMatching | **14x** | wordseg |
| | RandomSegmenter | **12x** | wordseg |
| **POS Tagging** | Training | **5x** | NLTK |
| | Tagging | **6x** | NLTK |


## Installation

### Python

```bash
pip install rustling
```

### Rust

```bash
cargo add rustling
```

## License

MIT License