tokenizers 0.5.0

Provides a generic Tokenizer trait and a smart Token struct
Documentation
Build #190516 2019-10-09T22:16:32.951612+00:00
# rustc version
rustc 1.40.0-nightly (2748a9fd9 2019-10-08)
# docs.rs version
docsrs 0.6.0 (8a6fcb5 2019-10-09)

# build log
[INFO] running `"docker" "create" "-v" "/home/cratesfyi/workspace/builds/tokenizers-0.5.0/target:/opt/rustwide/target:rw,Z" "-v" "/home/cratesfyi/workspace/builds/tokenizers-0.5.0/source:/opt/rustwide/workdir:ro,Z" "-v" "/home/cratesfyi/workspace/cargo-home:/opt/rustwide/cargo-home:ro,Z" "-v" "/home/cratesfyi/workspace/rustup-home:/opt/rustwide/rustup-home:ro,Z" "-e" "SOURCE_DIR=/opt/rustwide/workdir" "-e" "MAP_USER_ID=1001" "-e" "CARGO_TARGET_DIR=/opt/rustwide/target" "-e" "RUSTFLAGS=" "-e" "RUSTDOCFLAGS=-Z unstable-options --resource-suffix -20191008-1.40.0-nightly-2748a9fd9 --static-root-path / --disable-per-crate-search --extern-html-root-url structopt=https://docs.rs/structopt/0.2.18 --extern-html-root-url failure=https://docs.rs/failure/0.1.5 --extern-html-root-url exitfailure=https://docs.rs/exitfailure/0.5.1 --extern-html-root-url indicatif=https://docs.rs/indicatif/0.11.0" "-e" "CARGO_HOME=/opt/rustwide/cargo-home" "-e" "RUSTUP_HOME=/opt/rustwide/rustup-home" "-w" "/opt/rustwide/workdir" "-m" "3221225472" "--network" "none" "rustops/crates-build-env" "/opt/rustwide/cargo-home/bin/cargo" "+nightly" "doc" "--lib" "--no-deps" "--target" "x86_64-unknown-linux-gnu"`
[INFO] [stderr] WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.
[INFO] [stdout] 5ac4d30abcaffe0f9ed7553a331a9a8d1f6b566b43cdf0c1346f5f314c461829
[INFO] running `"docker" "start" "-a" "5ac4d30abcaffe0f9ed7553a331a9a8d1f6b566b43cdf0c1346f5f314c461829"`
[INFO] [stderr]  Documenting tokenizers v0.5.0 (/opt/rustwide/workdir)
[INFO] [stderr]     Finished dev [unoptimized + debuginfo] target(s) in 1.58s
[INFO] running `"docker" "inspect" "5ac4d30abcaffe0f9ed7553a331a9a8d1f6b566b43cdf0c1346f5f314c461829"`
[INFO] running `"docker" "rm" "-f" "5ac4d30abcaffe0f9ed7553a331a9a8d1f6b566b43cdf0c1346f5f314c461829"`
[INFO] [stdout] 5ac4d30abcaffe0f9ed7553a331a9a8d1f6b566b43cdf0c1346f5f314c461829