Crate crate_names

Expand description

Compact, transparent artifacts of every crate name on crates.io, for typeahead and default-version lookup.

This crate has two halves:

Reading (always available): CrateNames, Descriptions, and Facets parse the published artifacts. The reader is sans-io: hand it bytes you fetched however you like.
Building (behind the build feature): build_from_dump streams a crates.io database dump tarball and produces the artifacts. Used by the scheduled GitHub Action in this repository; consumers normally never need it.

§Wire format (v2)

Artifacts are zstd-compressed TSV, one crate per line, sorted by the crate’s folded name: ASCII-lowercased, with - and _ treated as the same character (see normalize). Names are stored as spelled; only the ordering is folded. Crate names cannot contain tabs or newlines, and description whitespace is flattened, so no escaping is required.

Folding is what makes lookups work the way people type. crates.io will not let a new crate take a name that folds onto an existing one, so the folded key is unique across the registry, and the artifacts stay sorted and unique under it — queries remain two binary searches, and Tokio, tokio and tokio_util all find what you meant.

names-v2.tsv.zst: name \t default_version \t rank for every crate. rank is a log-quantized download count in 0..=255; see rank_from_downloads. Ordering by rank is meaningful, arithmetic on it is not.
descriptions-v2.tsv.zst: name \t description for every crate with a non-empty description, whitespace runs collapsed to single spaces.
facets-v1.tsv.zst: name \t keywords \t categories for every crate with at least one Cargo.toml keyword or category. Each field is a space-separated, sorted list (either may be empty); categories are crates.io slugs, nested with ::. Files are versioned independently, which is why this one is -v1 alongside the -v2 pair.

§Getting the artifacts

All are republished daily (built from that morning’s crates.io database dump) to a rolling GitHub release, at stable URLs also exposed as NAMES_URL_V2, DESCRIPTIONS_URL_V2, and FACETS_URL_V1:

The URLs redirect to the release asset, so follow redirects. Assets carry ETags: revalidate with If-None-Match rather than re-downloading — the content changes at most once a day.

let bytes = fetch(crate_names::NAMES_URL_V2);
let names = crate_names::CrateNames::from_zstd(&bytes)?;
let top_ten = names.typeahead("serd", 10);

Structs§

BuildOutput: The uncompressed artifacts plus build statistics.
CrateNames: Reader for the names artifact (names-v1.tsv.zst): every crate name with its default version and download rank.
Descriptions: Reader for the descriptions artifact (descriptions-v1.tsv.zst).
Entry: A single crate in the names artifact.
Facets: Reader for the facets artifact (facets-v1.tsv.zst): the declared Cargo.toml keywords and category slugs for every crate that has any.
FacetsEntry: One crate’s declared keywords and categories from the facets artifact.

Enums§

BuildError: Error returned when building artifacts from a dump.
Error: Error returned when parsing an artifact.

Constants§

DESCRIPTIONS_FILE_V2: Artifact file name for the descriptions table.
DESCRIPTIONS_URL_V2: Canonical public URL of the descriptions artifact; see NAMES_URL_V2.
FACETS_FILE_V1: Artifact file name for the facets (keywords and categories) table.
FACETS_URL_V1: Canonical public URL of the facets artifact; see NAMES_URL_V2.
NAMES_FILE_V2: Artifact file name for the names/versions/ranks table.
NAMES_URL_V2: Canonical public URL of the names artifact, republished daily by this repository’s scheduled workflow. Redirects (GitHub release asset), so fetch with redirect-following enabled.

Functions§

build_from_dump: Stream a crates.io database dump tarball (db-dump.tar.gz as downloaded, still gzipped) and produce the v2 artifacts.
normalize: The key a crate name is stored and searched under: ASCII-lowercased, with - and _ folded together — the same way crates.io decides whether two names collide.
rank_from_downloads: Quantize an all-time download count into a rank in 0..=255.