sentencepiece 0.5.0

Binding for the sentencepiece tokenizer
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

# sentencepiece

This Rust crate is a binding for the
[sentencepiece](https://github.com/google/sentencepiece) unsupervised
text tokenizer. The [crate
documentation](https://rustdoc.danieldk.eu/sentencepiece) is available
online.

## `libsentencepiece` dependency

This crate depends on the `sentencepiece` C++ library. By default,
this dependency is treated as follows:

* If `sentencepiece` could be found with `pkg-config`, the crate will
  link against the library found through `pkg-config`.
* Otherwise, the crate's build script will do a static build of the
  `sentencepiece` library. This requires that `cmake` is available.

If you wish to override this behavior, the `sentencepiece-sys` crate
offers two features:

* `system`: always attempt to link to the `sentencepiece` library
  found with `pkg-config`.
* `static`: always do a static build of the `sentencepiece` library
  and link against that.