netindex

Read and write immutable IP-range and exact-ASN indexes. Queries return every matching assertion, including overlaps and duplicates, with its target and borrowed payload bytes. Use a membership query when you only need a yes/no answer, or longest-prefix lookup when you need one address or CIDR.

Use cases

Use netindex for datasets built periodically and queried repeatedly: membership feeds, network policy, and IP intelligence. It accepts unsorted IPv4 and IPv6 addresses, canonical CIDRs, inclusive ranges, and nonzero ASN keys. Payloads and metadata are opaque; applications choose their schema or use the optional codec.

Readers use owned bytes, borrowed slices, or immutable mappings. Writers build in memory or with bounded external sorting. Disjoint networks use compact blocks; overlaps use an interval index. Identical payloads share bytes without merging assertions.

Licensed under MIT or Apache-2.0, at your option.

Quick start

Add netindex to your project:

[dependencies]

netindex = "3"

See the 3.0 API documentation or the 2.1 API documentation. See upgrading from 2.x before replacing deployed readers or rebuilding files.

use netindex::{Builder, Limits, Reader, Target};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let mut builder = Builder::new(Limits::default());
    builder.push(
        Target::Network {
            address: "203.0.113.0".parse()?,
            prefix: 24,
        },
        b"evidence",
    )?;

    let bytes = builder.into_bytes(b"release metadata")?;
    let reader = Reader::open(bytes, Limits::default())?;

    let matches = reader.lookup_ip("203.0.113.42".parse()?)?;
    assert_eq!(matches.len(), 1);
    assert_eq!(matches[0].payload, b"evidence");

    Ok(())
}

Index files use .nidx, for example membership.nidx.

Reading

Open an index with Reader::open. Opening validates the entire file. Keep the backing immutable; payload bytes remain opaque until your application decodes them.

ASN methods match an exact number, with zero treated as a miss. They do not resolve an IP's ASN. IDs reflect insertion order, but matches within a query need not arrive in that order.

A visitor can decode its payload and return an application error. Its error type must implement From<netindex::Error>. If the callback itself cannot fail, return Ok::<_, netindex::Error>(()).

use netindex::{Builder, Limits, Reader, Target};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let address = "203.0.113.42".parse()?;
    let mut builder = Builder::new(Limits::default());
    builder.push(Target::Address(address), b"evidence")?;
    let reader = Reader::open(builder.into_bytes(b"")?, Limits::default())?;

    reader.visit_ip(address, |matched| {
        let text = std::str::from_utf8(matched.payload)?;
        assert_eq!(text, "evidence");

        Ok::<_, Box<dyn std::error::Error>>(())
    })?;

    Ok(())
}

Batches, membership and intersections

Reader::longest_prefix returns the most specific matching CIDR or address (treated as a host prefix). It ignores arbitrary ranges and breaks equal-prefix ties by lowest logical assertion ID. Use visit_ip when you need all matching assertions, including ranges.

Reader::visit_batch accepts borrowed Query values and calls the visitor with its input query position and each borrowed match. Delivery follows input order, then normal per-query ordering; duplicate queries run independently. There is no output buffer, reordering, thread pool or automatic cross-query SIMD policy. BatchError identifies the failed query and retains the original error. Earlier callbacks have already run; later queries do not run. Callers own output storage and any rollback policy.

The try_visit_* methods preserve visitor errors separately from a successful break. visit_intersecting accepts an address, canonical CIDR, or inclusive range. It searches only that address family, preserves duplicates and sorted target order, and includes shared aliases. Invalid queries fail before delivery.

use std::{net::Ipv4Addr, ops::ControlFlow};
use netindex::{Builder, Error, Limits, Query, Reader, Target};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let address = Ipv4Addr::new(203, 0, 113, 42).into();
    let mut builder = Builder::new(Limits::default());
    builder.push(Target::Address(address), b"example")?;
    let reader = Reader::open(builder.into_bytes(b"")?, Limits::default())?;
    let queries = [Query::Ip(address), Query::Intersecting(Target::Network {
        address: Ipv4Addr::new(203, 0, 113, 0).into(), prefix: 24,
    })];
    let mut results = Vec::new();
    reader.visit_batch(&queries, |query, row| {
        results.push((query, row));
        Ok::<_, Error>(())
    })?;
    assert_eq!(results.len(), 2);
    assert!(reader.contains_ip(address)?);
    let first = reader.try_visit_ip(address, |row| {
        Ok::<_, Error>(ControlFlow::Break(row.id))
    })?;
    assert_eq!(first, ControlFlow::Break(0));
    assert_eq!(reader.inspect()?.records, 1);
    Ok(())
}

Reader::inspect reports physical/logical counts, section encodings, exact region sizes, bitpacked block counts and allocated routing/alias capacities. It scans packed descriptors, allocates nothing and never decodes payloads. Backing length, reader object size, auxiliary allocations and released validation scratch are separate measures; they are not an estimate of process RSS or mapped residency.

Separate source files

ReaderSet also supports batch, membership, intersection and early-stop methods. Batch callback positions are query then source. Each query visits sources in source order; interval queries search all sources because point coverage masks cannot safely exclude a whole range.

ReaderSet shares up to 64 readers and uses IP coverage to skip files that cannot match. It returns every matching assertion with its source position and source-local record ID. ASN queries search every reader. Schema decoding and provider precedence remain application decisions.

The default router uses 64 KiB; each reader retains a 1 KiB coverage summary. ReaderSet::with_fine_routing uses a 1 MiB table and can skip more files; measure its extra cache footprint against your workload. Both preserve all matches.

replaced(position, reader) creates a new set sharing unchanged readers. Coarse routing reuses coverage cached during validation; fine routing scans only the replacement. Old sets and borrowed matches remain usable; the application publishes the new set. Use CoverageRouter to manage readers separately.

Memory-mapped files

Add features = ["mmap"] to your dependency for MappedReader::map_file. It checks file-size limits before mapping, validates the index, and owns the mapping independently of the file handle.

# #[cfg(feature = "mmap")]
# {
use std::fs::File;

use netindex::{Limits, MappedReader};

let file = File::open("membership.nidx")?;
// SAFETY: No process modifies or truncates this file while the reader is alive.
let reader = unsafe { MappedReader::map_file(&file, Limits::default()) }?;
let matches = reader.lookup_ip("203.0.113.42".parse()?)?;
println!("{} matching entries", matches.len());
# }
# Ok::<(), Box<dyn std::error::Error>>(())

Mapping is unsafe because changes to the file by any process can invalidate borrowed memory. Publish replacements as separate files and retain old readers until their requests finish. mapping() exposes the read-only mapping for platform access hints; no preloading or advice is applied by default.

For broad, disjoint networks, Reader::open_with_broad_routes or MappedReader::map_file_with_broad_routes adds direct routes during validation. This uses up to 32 KiB per eligible address family and preserves original IDs and payloads. Other queries use ordinary lookup; overlaps are never bypassed. Routing is opt-in because its additional check can slow narrow-network queries.

Building

Builder::into_bytes returns owned bytes; Builder::write_to writes to a caller-owned output. Metadata is stored once as opaque bytes.

Ordinary builders choose compact shared-payload references when they save space. If a deployed reader is still on 2.x, see upgrading from 2.x before writing new artifacts.

Enable external-sort for ExternalBuilder when buffering all assertions would exceed your budget. ExternalOptions bounds sorted runs, the payload cache, and temporary-disk usage. Insertion IDs are preserved. Identical payloads share their first-seen span, including when the memory cache is full or disabled. Uncached values use a disk-backed hash table with full byte comparisons in bounded chunks; the cache only accelerates matching. This trades random temporary-file I/O for complete deduplication. Table growth charges both old and new tables against the temporary-byte limit; builds may require a larger disk budget than earlier versions. Discard the builder after a failed push.

Call ExternalBuilder::set_disk_table_cache(bytes) before pushing rows to optionally cache the temporary table. This budget includes cache pages and tags, separate from the payload-key and disk limits. The cache is off by default; disk_table_cache_bytes() reports what was allocated. It does not change output bytes or skip complete payload comparisons.

ExternalBuilder::new_partitioned(directory, limits, options, partitions) is an opt-in alternative when extra table-cache memory is unavailable. It stages values into 1-64 power-of-two partitions, reuses the payload-key budget for one partition at a time, and restores first-seen payload order and insertion IDs. Complete byte comparisons remain the fallback when a partition exceeds the key budget. Extra sequential passes and temporary disk can make this slower than the ordinary builder; measure both. Fixed I/O buffers and open-file counts grow with the partition count outside the key budget. Nonzero table caches are unsupported, and cumulative artifact-byte limits can fail during finalization before output. See partitioned-build measurements.

The external writer requires empty, seekable output at position zero. Either writer can leave partial output on failure. Stage output, check success, and validate your payloads before publishing. File synchronization and release activation belong to your application.

Exact-bit compaction

Reader::write_bitpacked_to(output) optionally compacts packed IP deltas into format 6. It accepts the output of either builder, including shared-alias files, and preserves every assertion, ID, target, payload and metadata byte. It uses fixed block buffers and returns the written size. When no space is saved, it copies the original bytes. Ordinary builders continue producing formats 3/4.

Upgrade readers before using this output: readers without format-6 support reject it, including inside alias envelopes. Exact-bit access trades some validation and lookup work for smaller files; measure your workload. See FORMAT.md and bitpacking measurements. Stage output and check success before publishing; output errors can leave partial bytes.

Shared IPv4 aliases

For an index whose complete IPv4 network section is repeated under explicit IPv6 prefixes, Reader::write_shared_aliases can wrap the base in a format-5 artifact. The reader projects targets, shares borrowed payloads and preserves logical insertion IDs without copying the IPv4 rows. Windows must be disjoint from each other and from native IPv6 targets; overlaps and duplicates among source IPv4 networks still produce separate matches. The source IPv4 IDs must start at zero in sorted order. See the format rules.

This is opt-in. Ordinary builders still write formats 3/4, and readers that predate format 5 cannot open the shared artifact. The share_aliases example folds explicitly declared windows from an existing file only after verifying the projected targets, IDs, and payloads against the originals.

Shared values

Enable shared-values to share strings or byte values across records. ValuePoolBuilder::intern returns an ID for your record layout to store. Keep the encoded pool in metadata or a separate file and activate it with its records. Schemas, null handling, and numeric encoding remain yours.

ValuePool borrows arbitrary bytes. StringPool validates UTF-8 once and borrows strings without rescanning. Both pool readers allocate no heap memory. IDs are local to a pool and must not be reused with another release.

MMDB conversion

Enable mmdb for mmdb::visit_networks. It validates the MMDB and visits one network at a time. Your callback selects and encodes fields for either builder, or skips a record. Input needs random-access bytes or a mapping; a sequential download stream is insufficient.

Iteration options control aliases, empty records, and networks without data. Include aliases to preserve native alias targets. The canonical ::/96 subtree is emitted as IPv4; native-equivalent IPv6 lookups there need additional IPv6 targets or application routing. Discard the partial build on conversion failure; earlier pushes are not rolled back. See mmdb::visit_networks for examples.

Memory and concurrency

Default limits are one million assertions and 128 MiB per file. These bound logical records and file bytes, not RSS. Validation state, allocator overhead, caller buffers, and the OS file cache use additional memory. Large packed IPv4 sections add a lookup directory of up to 32 KiB per reader. Each reader retains a 1 KiB coverage summary collected during validation.

Reader<B> is Send + Sync when its backing is. Share a reader or reader set through Arc; this does not copy database bytes. Each caller owns its visitor and result storage. The crate is synchronous and creates no threads.

Parallelize independent queries or bounded batches. Opening, building, lengthy lookups, and mapped-page faults can block runtime workers. The Tokio and compio examples use bounded blocking workers and finish admitted work before returning a batch failure:

cargo run --example tokio

cargo run --example compio

Record codec

Add features = ["codec"] to your dependency for flat typed records with shared strings and bytes. Declare fields once, then encode records with that schema:

# #[cfg(feature = "codec")]
# {
use netindex::{
    Builder, Error, Limits, Reader, Target,
    codec::{Decoder, Encoder, Field, Kind, Schema, Value},
};

let limits = Limits::default();
let schema = Schema::new(
    &[Field::Required(Kind::U32), Field::Optional(Kind::String)],
    limits,
)?;
let mut encoder = Encoder::new(schema, limits);
let payload = encoder.encode(&[Value::U32(64512), Value::String("example network")])?;
let metadata = encoder.into_metadata()?;

let mut builder = Builder::new(limits);
builder.push(Target::Asn(64512), &payload)?;
let reader = Reader::open(builder.into_bytes(&metadata)?, limits)?;
let decoder = Decoder::open(reader.metadata()?, limits)?;

reader.visit_asn(64512, |matched| {
    let record = decoder.record(matched.payload)?;

    assert_eq!(record.string(1)?, Some("example network"));

    Ok::<_, Error>(())
})?;
# }
# Ok::<(), netindex::Error>(())

Pass encoded records to the index builder as payloads and store codec metadata with the index. Field::Absent omits a field known to be missing throughout the dataset; Field::Optional preserves missing versus empty or zero. Scalars retain their declared width; strings and bytes use four-byte pool IDs. The schema has no provider-specific fields, nested objects, or implicit defaults.

Opening validates metadata and shared strings. record() checks the row shape; field access checks values and references. Use validate() to check every field before publishing. Consumers must also check that the stored schema matches their expected field declarations. Records and their metadata must belong to the same dataset; in-range IDs from another pool cannot be detected automatically.

Upgrading from 2.x

Upgrade readers before writers. 3.0 readers accept existing format-3 files. The ordinary 3.0 writer can emit format 4; older readers that only support format 3 reject it. A small test artifact may still use format 3, so use explicit legacy output while any older reader must open your files.

Stage a new artifact and validate it before activation. Keep the old files if older readers need to roll back. Reader::inspect reports the outer and base formats. For layout and validation details, see FORMAT.md.

Features

All features are disabled by default. Global target-cpu=native is not required. Measure search on your workload: SIMD search can be slower than scalar for small packed blocks. simd-utf8 enables accelerated string-pool opening independently; simd includes it and also enables AVX2 search. Rust 1.95 is supported. The public error enum is non-exhaustive; messages are diagnostics, not a stable wire format.

See the file specification for encoding and validation, and the contributor guide for checks and measurement practices.

Performance

Costs vary with index shape, payload decoding, and startup requirements. In the Byway City deployment replay, netindex improved lookup tail latency while native MMDB opened and reloaded faster. See the benchmark guide for the workloads, tradeoffs, and reproduction instructions.