ngram_rs 0.1.2

Facilitate creating ngrams in Rust to be used in the polars plugin.
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42

# N-Gram Generation Toolkit

A high-performance n-gram generation library with Rust core and Polars plugin integration.

## Features

- **Blazing Fast**: Optimized Rust implementation for n-gram generation
- **Memory Efficient**: Uses `Cow` (Copy-on-Write) for minimal allocations
- **Flexible N-Ranges**: Generate n-grams for multiple values of n simultaneously
- **Custom Delimiters**: Support for any string delimiter between tokens
- **Polars Integration**: Seamless integration with Polars DataFrames
- **Iterator Support**: Lazy n-gram generation for memory-constrained environments

## Components

### ngram_rs (Core Library)
The core Rust library providing:
- Three different APIs for various use cases
- Optimized implementations for common cases (unigrams, bigrams)
- Iterator-based lazy generation

## Quick Start


```rust
use ngram_rs::generate_ngrams;

let words = vec!["the", "quick", "brown", "fox"]
    .into_iter()
    .map(String::from)
    .collect::<Vec<_>>();

let ngrams = generate_ngrams(&words, &[1, 2, 3], Some(" "));
```

## Performance
The library is optimized for:

- Minimal memory allocations through Cow<str>
- Specialized implementations for unigrams and bigrams
- Efficient windowing algorithms for higher-order n-grams
- Zero-copy operations where possible