lindera-python 3.0.6

A Python binding for Lindera.
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79

# Lindera Python Examples

## Prerequisites

- Python >= 3.10
- Rust toolchain
- [Maturin](https://www.maturin.rs/) (`pip install maturin`)

## Build

Run from the `lindera-python/` directory:

```bash
cd lindera-python

# Build with embedded IPADIC dictionary
maturin develop --features embed-ipadic

# Build with training feature
maturin develop --features embed-ipadic,train

# Release build
maturin develop --release --features embed-ipadic
```

## Examples

Run from the `lindera-python/` directory after building:

### Basic tokenization

```bash
python examples/tokenize.py
```

Tokenizes Japanese text using the embedded IPADIC dictionary in "normal" mode.

### Decompose mode

```bash
python examples/tokenize_with_decompose.py
```

Tokenizes text in "decompose" mode, which decomposes compound morphemes.

### Tokenization with filters

```bash
python examples/tokenize_with_filters.py
```

Demonstrates `TokenizerBuilder` API with character filters (`unicode_normalize`, `japanese_iteration_mark`, `mapping`) and token filters (`japanese_katakana_stem`, `japanese_stop_tags`, `lowercase`, `japanese_base_form`).

### User dictionary

```bash
python examples/tokenize_with_userdict.py
```

Loads a custom user dictionary from `resources/ipadic_simple_userdic.csv` alongside the standard dictionary.

### Build dictionary from source

```bash
python examples/build_ipadic.py
```

Downloads the mecab-ipadic source tarball and builds a Lindera dictionary from it.

### Train and export (requires `train` feature)

```bash
# Build with train feature first
maturin develop --features embed-ipadic,train

python examples/train_and_export.py
```

Trains a CRF model from a sample corpus and exports dictionary files.