litsea-cli-0.5.0 is not a library.

Litsea

Litsea is an extremely compact word segmentation and POS (Part-of-Speech) tagging software implemented in Rust, inspired by TinySegmenter and TinySegmenterMaker. Unlike traditional morphological analyzers such as MeCab and Lindera, Litsea does not rely on large-scale dictionaries but instead performs segmentation and POS tagging using compact pre-trained models. It features a fast and safe Rust implementation along with learners designed to be simple and highly extensible.

There is a small plant called Litsea cubeba (Aomoji) in the same camphoraceae family as Lindera (Kuromoji). This is the origin of the name Litsea.

How to build Litsea

Litsea is implemented in Rust. To build it, follow these steps:

Prerequisites

Install Rust (stable channel) from rust-lang.org.
Ensure Cargo (Rust’s package manager) is available.

Build Instructions

Clone the Repository

If you haven't already cloned the repository, run:
```
git clone https://github.com/mosuka/litsea.git
cd litsea
```
Obtain Dependencies and Build

In the project's root directory, run:
```
cargo build --release
```
The --release flag produces an optimized build.
Verify the Build

Once complete, the executable will be in the target/release folder. Verify by running:
```
./target/release/litsea --help
```

Additional Notes

Using the latest stable Rust ensures compatibility with dependencies and allows use of modern features.
Run cargo update to refresh your dependencies if needed.

How to train models

Prepare a corpus with words separated by spaces in advance.

corpus.txt

Litsea は TinySegmenter を 参考 に 開発 さ れ た 、 Rust で 実装 さ れ た 極めて コンパクト な 単語 分割 ソフトウェア です 。

Extract the information and features from the corpus:

./target/release/litsea extract ./resources/corpus.txt ./resources/features.txt

The output from the extract command is similar to:

Feature extraction completed successfully.

Train the features output by the above command using AdaBoost. Training stops if the new weak classifier’s accuracy falls below 0.001 or after 10,000 iterations.

./target/release/litsea train -t 0.001 -i 10000 ./resources/features.txt ./resources/model

The output from the train command is similar to:

finding instances...: 61 instances found
loading instances...: 61/61 instances loaded
Iteration 9999 - margin: 0.16068839956263622
Result Metrics:
  Accuracy: 100.00% ( 61 / 61 )
  Precision: 100.00% ( 24 / 24 )
  Recall: 100.00% ( 24 / 24 )
  Confusion Matrix:
    True Positives: 24
    False Positives: 0
    False Negatives: 0
    True Negatives: 37

How to segment sentences into words

Use the trained model to segment sentences:

echo "LitseaはTinySegmenterを参考に開発された、Rustで実装された極めてコンパクトな単語分割ソフトウェアです。" | ./target/release/litsea segment ./resources/model

The output will look like:

Litsea は TinySegmenter を 参考 に 開発 さ れ た 、 Rust で 実装 さ れ た 極めて コンパクト な 単語 分割 ソフトウェア です 。

How to segment sentences with POS tagging

Use the --pos flag with the segment command to perform joint word segmentation and POS tagging:

echo "LitseaはTinySegmenterを参考に開発された、Rustで実装された極めてコンパクトな単語分割ソフトウェアです。" | ./target/release/litsea segment --pos -l japanese ./resources/japanese_pos.model

The output will look like:

Litsea/PROPN は/ADP TinySegmenter/PROPN を/ADP 参考/NOUN に/ADP 開発/VERB さ/AUX れ/AUX た/AUX 、/PUNCT Rust/PROPN で/ADP 実装/VERB さ/AUX れ/AUX た/AUX 極めて/ADV コンパクト/ADJ な/AUX 単語/NOUN 分割/NOUN ソフトウェア/NOUN です/AUX 。/PUNCT

How to train POS models

Step 1: Extract POS features

Use the --pos flag with the extract command:

./target/release/litsea extract --pos -l japanese ./corpus_pos.txt ./features_pos.txt

Step 2: Train the POS model

Use the --pos flag with the train command. Use --num-epochs to set the number of training epochs:

./target/release/litsea train --pos --num-epochs 10 ./features_pos.txt ./resources/japanese_pos.model