# rustc version rustc 1.100.0-nightly (2e2b193f8 2026-09-02)
# docs.rs version docsrs 0.0.0 (c6ca30c0fecf66cc5a0859ac541222ca6c912d59 2026-09-01 )
# build log [INFO] running `Command { std: "docker" "exec" "-e" "SOURCE_DIR=/opt/rustwide/workdir" "-e" "CARGO_HOME=/opt/rustwide/cargo-home" "-e" "RUSTUP_HOME=/opt/rustwide/rustup-home" "-e" "CARGO_TARGET_DIR=/opt/rustwide/target" "-e" "DOCS_RS=1" "-w" "/opt/rustwide/workdir" "--user" "1001:1001" "c74453690ab3f7b8ce43472122762d361bdff0ccbdf0886b4dc68b067e606f70" "/opt/rustwide/cargo-home/bin/cargo" "+nightly" "rustdoc" "--lib" "-Zrustdoc-map" "--all-features" "--config" "build.rustdocflags=[\"--cfg\", \"docsrs\", \"-Z\", \"unstable-options\", \"--emit=html-non-static-files\", \"--resource-suffix\", \"-20260902-1.100.0-nightly-2e2b193f8\", \"--static-root-path\", \"/-/rustdoc.static/\", \"--cap-lints\", \"warn\", \"--extern-html-root-takes-precedence\"]" "--offline" "-Zunstable-options" "--config=doc.extern-map.registries.crates-io=\"https://docs.rs/{pkg_name}/{version}/x86_64-unknown-linux-gnu\"" "-j12" "--target" "x86_64-unknown-linux-gnu" "-Zrustdoc-scrape-examples", kill_on_drop: false }` [INFO] [stderr] warning: explicit `package.readme` can be inferred [INFO] [stderr] --> Cargo.toml:192:1 [INFO] [stderr] | [INFO] [stderr] 192 | readme = "README.md" [INFO] [stderr] | ^^^^^^^^^^^^^^^^^^^^ [INFO] [stderr] | [INFO] [stderr] = note: `cargo::manual_readme` is set to `warn` by default [INFO] [stderr] help: consider removing `package.readme` [INFO] [stderr] warning: `package.homepage` is redundant with `package.repository` [INFO] [stderr] --> Cargo.toml:188:12 [INFO] [stderr] | [INFO] [stderr] 188 | homepage = "https://github.com/huggingface/tokenizers" [INFO] [stderr] | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [INFO] [stderr] ... [INFO] [stderr] 193 | repository = "https://github.com/huggingface/tokenizers" [INFO] [stderr] | ------------------------------------------- [INFO] [stderr] | [INFO] [stderr] = note: `cargo::redundant_homepage` is set to `warn` by default [INFO] [stderr] help: consider removing `package.homepage` [INFO] [stderr] warning: `tokenizers` (manifest) generated 2 warnings [INFO] [stderr] warning: target filter specified, but no targets matched; this is a no-op [INFO] [stderr] Documenting tokenizers v0.23.2 (/opt/rustwide/workdir) [INFO] [stderr] warning: unresolved link to `TokenizerImpl::with_normalizer` [INFO] [stderr] --> src/tokenizer/added_vocabulary.rs:350:43 [INFO] [stderr] | [INFO] [stderr] 350 | /// This is called automatically by [`TokenizerImpl::with_normalizer`] when the [INFO] [stderr] | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ no item named `TokenizerImpl` in scope [INFO] [stderr] | [INFO] [stderr] = note: `#[warn(rustdoc::broken_intra_doc_links)]` on by default [INFO] [stderr] [INFO] [stderr] warning: redundant explicit link target [INFO] [stderr] --> src/tokenizer/mod.rs:3:21 [INFO] [stderr] | [INFO] [stderr] 3 | //! A [`Tokenizer`](struct.Tokenizer.html) is composed of some of the following parts. [INFO] [stderr] | ----------- ^^^^^^^^^^^^^^^^^^^^^ explicit target is redundant [INFO] [stderr] | | [INFO] [stderr] | because label contains path that resolves to same destination [INFO] [stderr] | [INFO] [stderr] = note: when a link's destination is not specified, [INFO] [stderr] the label is used to resolve intra-doc links [INFO] [stderr] = note: `#[warn(rustdoc::redundant_explicit_links)]` on by default [INFO] [stderr] help: remove explicit link target [INFO] [stderr] | [INFO] [stderr] 3 - //! A [`Tokenizer`](struct.Tokenizer.html) is composed of some of the following parts. [INFO] [stderr] 3 + //! A [`Tokenizer`] is composed of some of the following parts. [INFO] [stderr] | [INFO] [stderr] [INFO] [stderr] warning: redundant explicit link target [INFO] [stderr] --> src/tokenizer/mod.rs:4:24 [INFO] [stderr] | [INFO] [stderr] 4 | //! - [`Normalizer`](trait.Normalizer.html): Takes care of the text normalization (like unicode normalization). [INFO] [stderr] | ------------ ^^^^^^^^^^^^^^^^^^^^^ explicit target is redundant [INFO] [stderr] | | [INFO] [stderr] | because label contains path that resolves to same destination [INFO] [stderr] | [INFO] [stderr] = note: when a link's destination is not specified, [INFO] [stderr] the label is used to resolve intra-doc links [INFO] [stderr] help: remove explicit link target [INFO] [stderr] | [INFO] [stderr] 4 - //! - [`Normalizer`](trait.Normalizer.html): Takes care of the text normalization (like unicode normalization). [INFO] [stderr] 4 + //! - [`Normalizer`]: Takes care of the text normalization (like unicode normalization). [INFO] [stderr] | [INFO] [stderr] [INFO] [stderr] warning: redundant explicit link target [INFO] [stderr] --> src/tokenizer/mod.rs:5:26 [INFO] [stderr] | [INFO] [stderr] 5 | //! - [`PreTokenizer`](trait.PreTokenizer.html): Takes care of the pre tokenization (ie. How to split tokens and pre-process [INFO] [stderr] | -------------- ^^^^^^^^^^^^^^^^^^^^^^^ explicit target is redundant [INFO] [stderr] | | [INFO] [stderr] | because label contains path that resolves to same destination [INFO] [stderr] | [INFO] [stderr] = note: when a link's destination is not specified, [INFO] [stderr] the label is used to resolve intra-doc links [INFO] [stderr] help: remove explicit link target [INFO] [stderr] | [INFO] [stderr] 5 - //! - [`PreTokenizer`](trait.PreTokenizer.html): Takes care of the pre tokenization (ie. How to split tokens and pre-process [INFO] [stderr] 5 + //! - [`PreTokenizer`]: Takes care of the pre tokenization (ie. How to split tokens and pre-process [INFO] [stderr] | [INFO] [stderr] [INFO] [stderr] warning: redundant explicit link target [INFO] [stderr] --> src/tokenizer/mod.rs:7:19 [INFO] [stderr] | [INFO] [stderr] 7 | //! - [`Model`](trait.Model.html): A model encapsulates the tokenization algorithm (like BPE, Word base, character [INFO] [stderr] | ------- ^^^^^^^^^^^^^^^^ explicit target is redundant [INFO] [stderr] | | [INFO] [stderr] | because label contains path that resolves to same destination [INFO] [stderr] | [INFO] [stderr] = note: when a link's destination is not specified, [INFO] [stderr] the label is used to resolve intra-doc links [INFO] [stderr] help: remove explicit link target [INFO] [stderr] | [INFO] [stderr] 7 - //! - [`Model`](trait.Model.html): A model encapsulates the tokenization algorithm (like BPE, Word base, character [INFO] [stderr] 7 + //! - [`Model`]: A model encapsulates the tokenization algorithm (like BPE, Word base, character [INFO] [stderr] | [INFO] [stderr] [INFO] [stderr] warning: redundant explicit link target [INFO] [stderr] --> src/tokenizer/mod.rs:9:27 [INFO] [stderr] | [INFO] [stderr] 9 | //! - [`PostProcessor`](trait.PostProcessor.html): Takes care of the processing after tokenization (like truncating, padding, [INFO] [stderr] | --------------- ^^^^^^^^^^^^^^^^^^^^^^^^ explicit target is redundant [INFO] [stderr] | | [INFO] [stderr] | because label contains path that resolves to same destination [INFO] [stderr] | [INFO] [stderr] = note: when a link's destination is not specified, [INFO] [stderr] the label is used to resolve intra-doc links [INFO] [stderr] help: remove explicit link target [INFO] [stderr] | [INFO] [stderr] 9 - //! - [`PostProcessor`](trait.PostProcessor.html): Takes care of the processing after tokenization (like truncating, padding, [INFO] [stderr] 9 + //! - [`PostProcessor`]: Takes care of the processing after tokenization (like truncating, padding, [INFO] [stderr] | [INFO] [stderr] [INFO] [stderr] warning: `tokenizers` (lib doc) generated 6 warnings [INFO] [stderr] Finished `dev` profile [unoptimized + debuginfo] target(s) in 1.40s [INFO] [stderr] Generated /opt/rustwide/target/x86_64-unknown-linux-gnu/doc/tokenizers/index.html [INFO] running `Command { std: "docker" "inspect" "c74453690ab3f7b8ce43472122762d361bdff0ccbdf0886b4dc68b067e606f70", kill_on_drop: false }`