aleph-alpha-tokenizer 0.1.0

A fast implementation of a wordpiece-inspired tokenizer
Documentation
Build #241129 2020-04-17T20:18:36.221466+00:00
# rustc version
rustc 1.44.0-nightly (7f3df5772 2020-04-16)
# docs.rs version
docsrs 0.6.0 (bc15998 2020-04-16)

# build log
[INFO] running `"docker" "create" "-v" "/home/cratesfyi/workspace/builds/aleph-alpha-tokenizer-0.1.0/target:/opt/rustwide/target:rw,Z" "-v" "/home/cratesfyi/workspace/builds/aleph-alpha-tokenizer-0.1.0/source:/opt/rustwide/workdir:ro,Z" "-v" "/home/cratesfyi/workspace/cargo-home:/opt/rustwide/cargo-home:ro,Z" "-v" "/home/cratesfyi/workspace/rustup-home:/opt/rustwide/rustup-home:ro,Z" "-e" "SOURCE_DIR=/opt/rustwide/workdir" "-e" "MAP_USER_ID=1001" "-e" "CARGO_TARGET_DIR=/opt/rustwide/target" "-e" "RUSTFLAGS=" "-e" "RUSTDOCFLAGS=-Z unstable-options --resource-suffix -20200416-1.44.0-nightly-7f3df5772 --static-root-path / --cap-lints warn --extern-html-root-url fst=https://docs.rs/fst/0.4.1 --extern-html-root-url criterion=https://docs.rs/criterion/0.3.1" "-e" "CARGO_HOME=/opt/rustwide/cargo-home" "-e" "RUSTUP_HOME=/opt/rustwide/rustup-home" "-w" "/opt/rustwide/workdir" "-m" "3221225472" "--cpus" "2" "--network" "none" "rustops/crates-build-env" "/opt/rustwide/cargo-home/bin/cargo" "+nightly" "doc" "--lib" "--no-deps" "-j2"`
[INFO] [stderr] WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.
[INFO] [stdout] 026520e3eb833ca3dc27687b27d29ee7158bcf417ac9a57f5df755f0b1ef07f7
[INFO] running `"docker" "start" "-a" "026520e3eb833ca3dc27687b27d29ee7158bcf417ac9a57f5df755f0b1ef07f7"`
[INFO] [stderr]    Compiling fst v0.4.1
[INFO] [stderr]  Documenting aleph-alpha-tokenizer v0.1.0 (/opt/rustwide/workdir)
[INFO] [stderr]     Finished dev [unoptimized + debuginfo] target(s) in 2.47s
[INFO] running `"docker" "inspect" "026520e3eb833ca3dc27687b27d29ee7158bcf417ac9a57f5df755f0b1ef07f7"`
[INFO] running `"docker" "rm" "-f" "026520e3eb833ca3dc27687b27d29ee7158bcf417ac9a57f5df755f0b1ef07f7"`
[INFO] [stdout] 026520e3eb833ca3dc27687b27d29ee7158bcf417ac9a57f5df755f0b1ef07f7