# Lindera
[](https://opensource.org/licenses/MIT) [](https://crates.io/crates/lindera)
A morphological analysis library in Rust. This project is forked from [kuromoji-rs](https://github.com/fulmicoton/kuromoji-rs).
Lindera aims to build a library which is easy to install and provides concise APIs for various Rust applications.
## Feature flags
As of v5.0, this crate is a pure morphological segmenter around the
`Segmenter` API. The analysis chain (character filters, token filters, and the
`Tokenizer`) lives in the companion
[`lindera-analysis`](https://crates.io/crates/lindera-analysis) crate:
```toml
[dependencies]
# Pure segmenter
lindera = "6"
# With the analysis chain (character filters, token filters, Tokenizer)
lindera = "6"
lindera-analysis = "6"
```
The main feature flags of this crate:
| `mmap` | Memory-mapped dictionary loading | Yes |
| `train` | CRF-based dictionary training (depends on `lindera-trainer`) | No |
| `embed-ipadic`, `embed-ipadic-neologd`, `embed-unidic`, `embed-sudachidict` | Embed a Japanese dictionary in the binary | No |
| `embed-ko-dic` | Embed the Korean dictionary (ko-dic) in the binary | No |
| `embed-cc-cedict`, `embed-jieba` | Embed a Chinese dictionary in the binary | No |
The examples below use `embed-ipadic`. Without an `embed-*` feature, load a
pre-built dictionary from its path instead (`load_dictionary("/path/to/ipadic")`).
See [Feature Flags](https://lindera.github.io/lindera/development/feature_flags.html)
for the full list, including the `embed-cjk*` bundles.
See the [migration guide](https://lindera.github.io/lindera/migration_v5_to_v6.html)
when upgrading from v5.
## Segmentation example
### Basic segmentation
Put the following in Cargo.toml:
```toml
[dependencies]
lindera = { version = "6", features = ["embed-ipadic"] }
```
This example covers the basic usage of Lindera as a pure segmenter — no
additional crates required.
It will:
- Create a segmenter in normal mode
- Segment the input text
- Output the tokens
```rust
use std::borrow::Cow;
use lindera::dictionary::load_dictionary;
use lindera::mode::Mode;
use lindera::segmenter::Segmenter;
use lindera::LinderaResult;
fn main() -> LinderaResult<()> {
let dictionary = load_dictionary("embedded://ipadic")?;
let segmenter = Segmenter::new(Mode::Normal, dictionary, None);
let text = "関西国際空港限定トートバッグ";
let mut tokens = segmenter.segment(Cow::Borrowed(text))?;
println!("text:\t{text}");
for token in tokens.iter_mut() {
let details = token.details().join(",");
println!("token:\t{}\t{}", token.surface.as_ref(), details);
}
Ok(())
}
```
The above example can be run as follows:
```shell
% cargo run -p lindera --features=embed-ipadic --example=segment
```
You can see the result as follows:
```text
text: 関西国際空港限定トートバッグ
token: 関西国際空港 名詞,固有名詞,組織,*,*,*,関西国際空港,カンサイコクサイクウコウ,カンサイコクサイクーコー
token: 限定 名詞,サ変接続,*,*,*,*,限定,ゲンテイ,ゲンテイ
token: トートバッグ 名詞,一般,*,*,*,*,*,*,*
```
## Tokenization examples
The `Tokenizer` and the filter chain below are provided by the
`lindera-analysis` crate:
```toml
[dependencies]
lindera = { version = "6", features = ["embed-ipadic"] }
lindera-analysis = "6"
```
### Basic tokenization
This example covers the basic usage of the tokenizer.
It will:
- Create a tokenizer in normal mode
- Tokenize the input text
- Output the tokens
```rust
use lindera::dictionary::load_dictionary;
use lindera::mode::Mode;
use lindera::segmenter::Segmenter;
use lindera_analysis::tokenizer::Tokenizer;
use lindera::LinderaResult;
fn main() -> LinderaResult<()> {
let dictionary = load_dictionary("embedded://ipadic")?;
let segmenter = Segmenter::new(Mode::Normal, dictionary, None);
let tokenizer = Tokenizer::new(segmenter);
let text = "関西国際空港限定トートバッグ";
let mut tokens = tokenizer.tokenize(text)?;
println!("text:\t{text}");
for token in tokens.iter_mut() {
let details = token.details().join(",");
println!("token:\t{}\t{}", token.surface.as_ref(), details);
}
Ok(())
}
```
The above example can be run as follows:
```shell
% cargo run -p lindera-analysis --features=embed-ipadic --example=tokenize
```
You can see the result as follows:
```text
text: 関西国際空港限定トートバッグ
token: 関西国際空港 名詞,固有名詞,組織,*,*,*,関西国際空港,カンサイコクサイクウコウ,カンサイコクサイクーコー
token: 限定 名詞,サ変接続,*,*,*,*,限定,ゲンテイ,ゲンテイ
token: トートバッグ 名詞,一般,*,*,*,*,*,*,*
```
### Tokenization with user dictionary
You can give user dictionary entries along with the default system dictionary. User dictionary should be a CSV with following format.
```csv
<surface>,<part_of_speech>,<reading>
```
Put the following in Cargo.toml:
```toml
[dependencies]
lindera = { version = "6", features = ["embed-ipadic"] }
lindera-analysis = "6"
anyhow = "1"
serde_json = "1"
```
For example:
```shell
% cat ./resources/user_dict/ipadic_simple_userdic.csv
東京スカイツリー,カスタム名詞,トウキョウスカイツリー
東武スカイツリーライン,カスタム名詞,トウブスカイツリーライン
とうきょうスカイツリー駅,カスタム名詞,トウキョウスカイツリーエキ
```
With a user dictionary, `Tokenizer` will be created as follows:
```rust
use std::fs::File;
use std::path::PathBuf;
use lindera::dictionary::{Metadata, load_dictionary, load_user_dictionary};
use lindera::error::LinderaErrorKind;
use lindera::mode::Mode;
use lindera::segmenter::Segmenter;
use lindera_analysis::tokenizer::Tokenizer;
use lindera::LinderaResult;
fn main() -> LinderaResult<()> {
let user_dict_path = PathBuf::from(env!("CARGO_MANIFEST_DIR"))
.join("../resources")
.join("user_dict")
.join("ipadic_simple_userdic.csv");
let metadata_file = PathBuf::from(env!("CARGO_MANIFEST_DIR"))
.join("../lindera-ipadic")
.join("metadata.json");
let metadata: Metadata = serde_json::from_reader(
File::open(metadata_file)
.map_err(|err| LinderaErrorKind::Io.with_error(anyhow::anyhow!(err)))
.unwrap(),
)
.map_err(|err| LinderaErrorKind::Io.with_error(anyhow::anyhow!(err)))
.unwrap();
let dictionary = load_dictionary("embedded://ipadic")?;
let user_dictionary = load_user_dictionary(user_dict_path.to_str().unwrap(), &metadata)?;
let segmenter = Segmenter::new(
Mode::Normal,
dictionary,
Some(user_dictionary), // Using the loaded user dictionary
);
// Create a tokenizer.
let tokenizer = Tokenizer::new(segmenter);
// Tokenize a text.
let text = "東京スカイツリーの最寄り駅はとうきょうスカイツリー駅です";
let mut tokens = tokenizer.tokenize(text)?;
// Print the text and tokens.
println!("text:\t{text}");
for token in tokens.iter_mut() {
let details = token.details().join(",");
println!("token:\t{}\t{}", token.surface.as_ref(), details);
}
Ok(())
}
```
The above example can be run by `cargo run --example`:
```shell
% cargo run -p lindera-analysis --features=embed-ipadic --example=tokenize_with_user_dict
text: 東京スカイツリーの最寄り駅はとうきょうスカイツリー駅です
token: 東京スカイツリー カスタム名詞,*,*,*,*,*,*,トウキョウスカイツリー,*
token: の 助詞,連体化,*,*,*,*,の,ノ,ノ
token: 最寄り駅 名詞,一般,*,*,*,*,最寄り駅,モヨリエキ,モヨリエキ
token: は 助詞,係助詞,*,*,*,*,は,ハ,ワ
token: とうきょうスカイツリー駅 カスタム名詞,*,*,*,*,*,*,トウキョウスカイツリーエキ,*
token: です 助動詞,*,*,*,特殊・デス,基本形,です,デス,デス
```
### Tokenize with filters
Put the following in Cargo.toml:
```toml
[dependencies]
lindera = { version = "6", features = ["embed-ipadic"] }
lindera-analysis = "6"
```
This example builds an analysis chain with character filters and token filters.
It will:
- Apply character filters for Unicode normalization (NFKC) and Japanese iteration marks
- Tokenize the input text with IPADIC
- Apply token filters that merge consecutive numeral tokens, convert Japanese numerals to Arabic numerals, and remove tokens by part-of-speech tag
```rust
use lindera::dictionary::load_dictionary;
use lindera::mode::Mode;
use lindera::segmenter::Segmenter;
use lindera_analysis::character_filter::BoxCharacterFilter;
use lindera_analysis::character_filter::japanese_iteration_mark::JapaneseIterationMarkCharacterFilter;
use lindera_analysis::character_filter::unicode_normalize::{
UnicodeNormalizeCharacterFilter, UnicodeNormalizeKind,
};
use lindera_analysis::token_filter::BoxTokenFilter;
use lindera_analysis::token_filter::japanese_compound_word::JapaneseCompoundWordTokenFilter;
use lindera_analysis::token_filter::japanese_number::JapaneseNumberTokenFilter;
use lindera_analysis::token_filter::japanese_stop_tags::JapaneseStopTagsTokenFilter;
use lindera_analysis::tokenizer::Tokenizer;
use lindera::LinderaResult;
fn main() -> LinderaResult<()> {
let dictionary = load_dictionary("embedded://ipadic")?;
let segmenter = Segmenter::new(
Mode::Normal,
dictionary,
None, // Assuming no user dictionary is provided
);
let unicode_normalize_char_filter =
UnicodeNormalizeCharacterFilter::new(UnicodeNormalizeKind::NFKC);
let japanese_iterration_mark_char_filter =
JapaneseIterationMarkCharacterFilter::new(true, true);
// Merge the numeral tokens only. A counter merged into the number token
// would keep the number filter below from converting it.
let japanese_compound_word_token_filter = JapaneseCompoundWordTokenFilter::new(
vec!["名詞,数".to_string()].into_iter().collect(),
Some("名詞,数".to_string()),
);
let japanese_number_token_filter =
JapaneseNumberTokenFilter::new(Some(vec!["名詞,数".to_string()].into_iter().collect()));
let japanese_stop_tags_token_filter = JapaneseStopTagsTokenFilter::new(
vec![
"接続詞".to_string(),
"助詞".to_string(),
"助詞,格助詞".to_string(),
"助詞,格助詞,一般".to_string(),
"助詞,格助詞,引用".to_string(),
"助詞,格助詞,連語".to_string(),
"助詞,係助詞".to_string(),
"助詞,副助詞".to_string(),
"助詞,間投助詞".to_string(),
"助詞,並立助詞".to_string(),
"助詞,終助詞".to_string(),
"助詞,副助詞／並立助詞／終助詞".to_string(),
"助詞,連体化".to_string(),
"助詞,副詞化".to_string(),
"助詞,特殊".to_string(),
"助動詞".to_string(),
"記号".to_string(),
"記号,一般".to_string(),
"記号,読点".to_string(),
"記号,句点".to_string(),
"記号,空白".to_string(),
"記号,括弧閉".to_string(),
"その他,間投".to_string(),
"フィラー".to_string(),
"非言語音".to_string(),
]
.into_iter()
.collect(),
);
// Create a tokenizer.
let mut tokenizer = Tokenizer::new(segmenter);
tokenizer
.append_character_filter(BoxCharacterFilter::from(unicode_normalize_char_filter))
.append_character_filter(BoxCharacterFilter::from(
japanese_iterration_mark_char_filter,
))
.append_token_filter(BoxTokenFilter::from(japanese_compound_word_token_filter))
.append_token_filter(BoxTokenFilter::from(japanese_number_token_filter))
.append_token_filter(BoxTokenFilter::from(japanese_stop_tags_token_filter));
// Tokenize a text.
let text = "Ｌｉｎｄｅｒａは形態素解析ｴﾝｼﾞﾝです。ユーザー辞書も利用可能です。";
let tokens = tokenizer.tokenize(text)?;
// Print the text and tokens.
println!("text: {text}");
for token in tokens {
println!(
"token: {:?}, start: {:?}, end: {:?}, details: {:?}",
token.surface, token.byte_start, token.byte_end, token.details
);
}
Ok(())
}
```
The above example can be run as follows:
```shell
% cargo run -p lindera-analysis --features=embed-ipadic --example=tokenize_with_filters
```
You can see the result as follows:
```text
text: Ｌｉｎｄｅｒａは形態素解析ｴﾝｼﾞﾝです。ユーザー辞書も利用可能です。
token: "Lindera", start: 0, end: 21, details: Some(["名詞", "固有名詞", "組織", "*", "*", "*", "*", "*", "*"])
token: "形態素", start: 24, end: 33, details: Some(["名詞", "一般", "*", "*", "*", "*", "形態素", "ケイタイソ", "ケイタイソ"])
token: "解析", start: 33, end: 39, details: Some(["名詞", "サ変接続", "*", "*", "*", "*", "解析", "カイセキ", "カイセキ"])
token: "エンジン", start: 39, end: 54, details: Some(["名詞", "一般", "*", "*", "*", "*", "エンジン", "エンジン", "エンジン"])
token: "ユーザー", start: 63, end: 75, details: Some(["名詞", "一般", "*", "*", "*", "*", "ユーザー", "ユーザー", "ユーザー"])
token: "辞書", start: 75, end: 81, details: Some(["名詞", "一般", "*", "*", "*", "*", "辞書", "ジショ", "ジショ"])
token: "利用", start: 84, end: 90, details: Some(["名詞", "サ変接続", "*", "*", "*", "*", "利用", "リヨウ", "リヨー"])
token: "可能", start: 90, end: 96, details: Some(["名詞", "形容動詞語幹", "*", "*", "*", "*", "可能", "カノウ", "カノー"])
```
## Configuration file
Lindera is able to read YAML format configuration files.
Specify the path to the following file in the environment variable LINDERA_CONFIG_PATH. You can use it easily without having to code the behavior of the tokenizer in Rust code.
```yaml
segmenter:
mode: "normal"
dictionary: "embedded://ipadic"
# user_dictionary: "./resources/user_dict/ipadic_simple_userdic.csv"
character_filters:
- kind: "unicode_normalize"
args:
kind: "nfkc"
- kind: "japanese_iteration_mark"
args:
normalize_kanji: true
normalize_kana: true
- kind: mapping
args:
mapping:
リンデラ: Lindera
token_filters:
- kind: "japanese_compound_word"
args:
# Merge the numeral tokens only. A counter merged into the number token keeps japanese_number from converting it.
tags:
- "名詞,数"
new_tag: "名詞,数"
- kind: "japanese_number"
args:
tags:
- "名詞,数"
- kind: "japanese_stop_tags"
args:
tags:
- "接続詞"
- "助詞"
- "助詞,格助詞"
- "助詞,格助詞,一般"
- "助詞,格助詞,引用"
- "助詞,格助詞,連語"
- "助詞,係助詞"
- "助詞,副助詞"
- "助詞,間投助詞"
- "助詞,並立助詞"
- "助詞,終助詞"
- "助詞,副助詞／並立助詞／終助詞"
- "助詞,連体化"
- "助詞,副詞化"
- "助詞,特殊"
- "助動詞"
- "記号"
- "記号,一般"
- "記号,読点"
- "記号,句点"
- "記号,空白"
- "記号,括弧閉"
- "その他,間投"
- "フィラー"
- "非言語音"
- kind: "japanese_katakana_stem"
args:
min: 3
- kind: "remove_diacritical_mark"
args:
japanese: false
```
```shell
% export LINDERA_CONFIG_PATH=./resources/config/lindera.yml
```
`TokenizerBuilder::new()` reads the file named by `LINDERA_CONFIG_PATH`. To load a file directly, use `TokenizerBuilder::from_file`:
```rust
use std::path::PathBuf;
use lindera_analysis::tokenizer::TokenizerBuilder;
use lindera::LinderaResult;
fn main() -> LinderaResult<()> {
// Load tokenizer configuration from file
let path = PathBuf::from(env!("CARGO_MANIFEST_DIR"))
.join("../resources")
.join("config")
.join("lindera.yml");
let builder = TokenizerBuilder::from_file(&path)?;
let tokenizer = builder.build()?;
let text = "Ｌｉｎｄｅｒａは形態素解析ｴﾝｼﾞﾝです。ユーザー辞書も利用可能です。".to_string();
println!("text: {text}");
let tokens = tokenizer.tokenize(&text)?;
for token in tokens {
println!(
"token: {:?}, start: {:?}, end: {:?}, details: {:?}",
token.surface, token.byte_start, token.byte_end, token.details
);
}
Ok(())
}
```
The above example can be run as follows:
```shell
% cargo run -p lindera-analysis --features=embed-ipadic --example=tokenize_with_config
```
You can see the result as follows. The `japanese_katakana_stem` filter in the configuration turns `ユーザー` into `ユーザ`:
```text
text: Ｌｉｎｄｅｒａは形態素解析ｴﾝｼﾞﾝです。ユーザー辞書も利用可能です。
token: "Lindera", start: 0, end: 21, details: Some(["名詞", "固有名詞", "組織", "*", "*", "*", "*", "*", "*"])
token: "形態素", start: 24, end: 33, details: Some(["名詞", "一般", "*", "*", "*", "*", "形態素", "ケイタイソ", "ケイタイソ"])
token: "解析", start: 33, end: 39, details: Some(["名詞", "サ変接続", "*", "*", "*", "*", "解析", "カイセキ", "カイセキ"])
token: "エンジン", start: 39, end: 54, details: Some(["名詞", "一般", "*", "*", "*", "*", "エンジン", "エンジン", "エンジン"])
token: "ユーザ", start: 63, end: 75, details: Some(["名詞", "一般", "*", "*", "*", "*", "ユーザー", "ユーザー", "ユーザー"])
token: "辞書", start: 75, end: 81, details: Some(["名詞", "一般", "*", "*", "*", "*", "辞書", "ジショ", "ジショ"])
token: "利用", start: 84, end: 90, details: Some(["名詞", "サ変接続", "*", "*", "*", "*", "利用", "リヨウ", "リヨー"])
token: "可能", start: 90, end: 96, details: Some(["名詞", "形容動詞語幹", "*", "*", "*", "*", "可能", "カノウ", "カノー"])
```
## Environment Variables
### LINDERA_BUILD_DICTIONARY_CACHE_DIR
The `LINDERA_BUILD_DICTIONARY_CACHE_DIR` environment variable designates a build-time cache directory for the embedded-dictionary build pipeline. It is read only by the dictionary crates' build scripts and has no effect at runtime.
When set, each build stores two kinds of files under `$LINDERA_BUILD_DICTIONARY_CACHE_DIR/<version>-fmt<format>/` (where `<version>` is the dictionary crate version and `<format>` is the dictionary format version):
- the downloaded distribution archive (validated with MD5; invalid files are re-downloaded)
- the built binary dictionary that gets embedded into the crate
The format version is part of the path so that a cache written by a build with a different on-disk dictionary layout is a miss rather than a stale hit.
This enables:
- **Offline builds**: once cached, subsequent builds need no network access
- **Faster builds**: download and dictionary build are skipped when valid cached files exist
- **Reproducible builds**: consistent dictionary versions across builds
Usage:
```shell
export LINDERA_BUILD_DICTIONARY_CACHE_DIR=/path/to/cache
cargo build --features=embed-ipadic
```
Notes:
- The directory is managed automatically and is safe to delete; contents are re-downloaded and rebuilt as needed
- Version subdirectories accumulate across upgrades and are not garbage-collected; old ones can be removed freely
- Setting this variable causes dictionary crates to download and build their dictionaries even when no `embed-*` feature is enabled (useful for pre-populating the cache)
> **Deprecated:** the previous name `LINDERA_DICTIONARIES_PATH` still works as a fallback (the new name wins when both are set) and will be removed in v7.0.0.
### LINDERA_CONFIG_PATH
The `LINDERA_CONFIG_PATH` environment variable specifies the path to a YAML configuration file for the tokenizer. This allows you to configure tokenizer behavior without modifying Rust code.
```shell
export LINDERA_CONFIG_PATH=./resources/config/lindera.yml
```
See the [Configuration file](#configuration-file) section for details on the configuration format.
### DOCS_RS
The `DOCS_RS` environment variable is automatically set by docs.rs when building documentation. When this variable is detected, Lindera creates dummy dictionary files instead of downloading actual dictionary data, allowing documentation to be built without network access or large file downloads.
This is primarily used internally by docs.rs and typically doesn't need to be set by users.
### LINDERA_WORKDIR
The `LINDERA_WORKDIR` environment variable is automatically set during the build process by the lindera-dictionary crate. It points to the directory containing the built dictionary data files and is used internally by dictionary crates to locate their data files.
This variable is set automatically and should not be modified by users.
## API reference
The API reference is available. Please see following URL:
- [lindera](https://docs.rs/lindera)
- [lindera-analysis](https://docs.rs/lindera-analysis)