Module vectors 

Vector dataset access nodes via the vectordata crate.

Each node takes a dataset source string (URL, local path, or catalog name) as a const parameter and loads the dataset handle at construction time.

Source specifier formats:

• "dataset" — catalog lookup, uses default profile
• "dataset:profile" — catalog lookup with explicit profile
• "https://..." — direct URL
• "/path/to/dir" — local filesystem path

Prebuffering

Use dataset_prebuffer("source") to eagerly download all facets for a dataset before workload execution. After prebuffering, data access uses local mmap readers (zero HTTP overhead).

Cache-aware loading

For catalog-resolved datasets, the loader checks the local cache at ~/.cache/vectordata/<dataset>/ before issuing HTTP requests. Prebuffering populates this cache; subsequent loads are local.

Feature-gated behind vectordata.

Upstream API reference

See the vectordata consumer API docs for the full dataset access, catalog, caching, and prebuffer model: https://github.com/nosqlbench/vectordata-rs/blob/main/docs/sysref/02-api.md

Structs

DatasetDistanceFunction

Return the dataset’s distance function (e.g., “COSINE”, “EUCLIDEAN”). Operates on the dataset group; takes a Group handle.

DatasetFacets

Report which facets are available for the default profile of a dataset group as a comma-separated list. Expects a Group handle.

DatasetGroupOpen

dataset_group_open(source: str) -> Handle

DatasetOpen

dataset_open(source: str, facet: str) -> Handle

DatasetPrebuffer

Eagerly download all facets for a dataset profile into the local cache, returning a DatasetHandle::Group handle that downstream facet accessors take as their first argument. After this returns, every subsequent facet read served by vectordata::TestDataView hits the merkle- verified mmap fast path with no further network traffic.

DatasetProfileCount

Total number of profiles in a dataset group.

DatasetProfileNameAt

Look up a profile name by index from the canonical sorted list.

DatasetProfileNames

Comma-separated list of all profile names in canonical sort order (by base_count).

FilteredNeighborDistancesAt

Access filtered ground-truth neighbor distances. Expects an F32 handle.

FilteredNeighborIndicesAt

Access filtered ground-truth neighbor indices. Expects an I32 handle.

MatchingProfileNameAt

Name of the index-th profile matching pattern, in canonical (base-count-ascending) order. Pairs with profile_partitions’s masked-position idx_of so a sweep can prebuffer the active tier’s own ground-truth facets (dataset_prebuffer(str_concat("ds:", name))). pattern follows the literal / glob / regex promotion; the index wraps modulo the number of matching profiles; empty string if none match.

MatchingProfiles

Return profile names matching a prefix, comma-separated.

MetadataContentCount

Count of metadata content records. Expects a Generic handle.

MetadataResultsAt

Access metadata indices (variable-length matching base ordinals per query) at an index. Expects an Ivvec32 handle.

MetadataResultsCount

Return the metadata indices count (number of predicate result sets). Expects an Ivvec32 handle.

MetadataResultsLenAt

Return the length of one metadata_results record without loading the data (reads only the 4-byte header). Expects an Ivvec32 handle.

MetadataValueAt

Access a metadata value per base ordinal. Expects a Generic handle opened against the metadata_content facet.

NeighborCount

Return the per-record neighbor count (k) for an I32 neighbor-indices handle.

NeighborDistancesAt

Access ground-truth neighbor distances for a query. Expects an F32 handle.

NeighborIndicesAt

Access ground-truth neighbor indices for a query. Expects an I32 handle.

PredicateValueAt

Access a predicate value per query ordinal. Expects a Generic handle opened against the metadata_predicates facet.

ProfileBaseCount

Return the base vector count for the profile at a given index.

ProfileFacets

Return the comma-separated facet list for the profile at a given index.

ProfilePartitions

Partition a dataset’s vector space by its profiles matching a pattern, treated as cumulative size tiers (an SRD-71 partition source). One partition per masked profile, in canonical (base-count-ascending) order: partition k spans [prev_masked_base_count, this_masked_base_count) — exactly the vectors added at that tier — so a sweep’s “load only the increment since the previously loaded set” is just the partition’s [start_of(p), end_of(p)), and a partition inherently knows its start (no cross-iteration carry needed). idx_of(p) is the 0-based masked position (pairs with matching_profile_name_at to address the tier’s own ground-truth facets); count_of(p) is the number of masked tiers; base_extent is the largest masked tier’s size.

QueryCount

Alias for VectorCount kept for clarity in workloads that distinguish base/query handles by name.

QueryVectorAt

Access a query vector by index. Alias for VectorAt kept for clarity in workloads that distinguish base and query handles by name.

VectorAt

Access an f32 vector by index, returning a typed VecF32. Works on any F32 handle (base or query facet).

VectorCount

Return the count of records in the facet a handle was opened against. This is the canonical “how many vectors / queries / neighbor-rows” accessor — vector_count(base_handle) for base vectors, vector_count(query_handle) for query vectors, etc.

VectorDim

Return the dimensionality (f32 count per record) of a vector facet handle.

Functions

signatures

Signatures for vector dataset access nodes (feature-gated).