netindex

Read and write immutable IP-range and exact-ASN indexes. Each assertion retains its original target and an opaque payload. Queries return every match, including overlaps and duplicates.

Licensed under either MIT or Apache-2.0, at your option.

Quick start

Add netindex to your dependencies:

[dependencies]

netindex = "0.1"

use netindex::{Builder, Limits, Reader, Target};

let mut builder = Builder::new(Limits::default());

builder.push(
    Target::Network {
        address: "203.0.113.0".parse()?,
        prefix: 24,
    },
    b"evidence",
)?;

let bytes = builder.into_bytes(b"release metadata")?;

let reader = Reader::open(bytes, Limits::default())?;

let matches = reader.lookup_ip("203.0.113.42".parse()?)?;
assert_eq!(matches.len(), 1);

for matched in matches {
    assert_eq!(matched.payload, b"evidence");
}

# Ok::<(), Box<dyn std::error::Error>>(())

API overview

Reading

[Reader::open] accepts immutable AsRef<[u8]> backing: owned bytes, a borrowed slice, or a caller-owned mapping. Opening validates the complete index. Payload decoding belongs to the application.

[Reader::lookup_ip] and [Reader::lookup_asn] collect borrowed matches into a vector. Misses return an empty vector. Exact-ASN lookup does not resolve an IP's ASN. Payloads remain valid while their reader is retained.

Use [Reader::visit_ip], [Reader::visit_asn], or [Reader::visit_all] to process matches without allocating a result vector. A visitor error stops delivery and retains the application's error type:

# use netindex::{Builder, Limits, Reader, Target};
# let mut builder = Builder::new(Limits::default());
# builder.push(Target::Address("203.0.113.42".parse()?), b"evidence")?;
# let reader = Reader::open(builder.into_bytes(b"")?, Limits::default())?;
reader.visit_ip("203.0.113.42".parse()?, |matched| {
    let text = std::str::from_utf8(matched.payload)?;

    assert_eq!(text, "evidence");

    Ok::<_, Box<dyn std::error::Error>>(())
})?;

# Ok::<(), Box<dyn std::error::Error>>(())

For up to 64 files, derive each reader's coverage and pass it to [CoverageRouter::new] in reader order. Query every returned candidate. Coverage can admit misses but does not discard matches. Retain readers and their router together; rebuild the router when the reader set changes. ASN queries still visit each reader's exact-ASN index.

Building

[Builder] accepts unsorted assertions. [Builder::into_bytes] returns owned encoded bytes; [Builder::write_to] writes the same format into a caller-owned writer. Identical payloads share bytes without merging assertions. Metadata is stored once and remains opaque.

Enable external-sort for ExternalBuilder when buffering all assertions would exceed the desired working set. It sorts bounded runs on disk and preserves insertion-order IDs. ExternalOptions controls run size, payload-cache bytes, and temporary-disk usage. An uncached repeated payload may occupy additional output bytes. Failed pushes invalidate this builder; discard it.

External finalization requires an empty seekable output at position zero. Both writers can leave partial output on failure. Stage output and publish it only after successful finalization and any application payload checks. Synchronization and release activation belong to the caller.

MMDB conversion

Enable mmdb for mmdb::visit_networks. It validates a native MMDB reader, then visits one network at a time. The callback encodes application fields and pushes into Builder or ExternalBuilder; it can skip records without adding assertions. Input may be owned bytes or a mapping. It cannot be parsed directly from a sequential download stream.

Iteration options control aliases, empty records, and networks without data. Include aliased networks when preserving native IPv6 alias lookups. Conversion errors stop delivery but do not undo earlier pushes; discard the partial build. Payload schemas, metadata selection, and output publication remain caller-owned.

Memory and files

Default limits are one million assertions and 128 MiB per file. These bound logical records and bytes, not process RSS. Validation state, allocator overhead, caller buffers, and the OS file cache consume additional memory.

The crate does not open or map files. A mapping owner must keep its file unchanged for the reader's lifetime. Mapped-page faults can block a lookup even though index search performs no explicit I/O. Choose owned bytes or a mapping for the workload.

Threads and runtimes

Reader<B> is Send + Sync when its backing is. Share it through Arc or borrow it across scoped threads; sharing does not copy database bytes. Each caller owns its visitor and result storage. The crate creates no threads and uses no runtime.

Parallelize independent queries or bounded batches. Move opening, building, and substantial lookup work away from async runtime workers. Bound admitted jobs and keep permits inside blocking jobs until they finish, including after cancellation.

The Tokio and compio examples share readers with bounded blocking workers and await admitted work before returning a batch failure:

cargo run --example tokio

cargo run --example compio

Features

All features are disabled by default. Global target-cpu=native is not required. The supported toolchain is Rust 1.95. The public error enum is non-exhaustive; error messages are diagnostics, not a stable wire format.

The file specification defines the encoded layout and validation rules.

See contributor documentation for checks and benchmarks.