Crate iriq

Expand description

§iriq

iriq finds the shape of a URL: the route template behind it. Erase the parts that vary, keep the parts that don’t, and /users/123 and /users/999 both become /users/{user_id}. Point it at a pile of messy URLs and it collapses them into a small set of stable, deterministic templates.

An IRI is a URL that’s allowed to contain non-ASCII characters; if you know URLs, you know IRIs. The name is IRI Query.

This crate is both the library and the iriq command-line tool. This page covers the library; the project README covers the CLI.

cargo add iriq          # library
cargo install iriq      # CLI

Requires Rust 1.85 or newer.

§Parse, normalize, extract

The pure functions need no setup and fail only with ParseError.

use iriq::{normalize, parse, trace, Extractor};

fn main() -> Result<(), iriq::ParseError> {
    let iri = parse("https://Foo.com:443/users/123?tab=posts")?;
    assert_eq!(iri.host, "foo.com");
    assert_eq!(iri.port, None); // the scheme's default port is dropped
    assert_eq!(iri.path_segments, ["users", "123"]);
    assert_eq!(iri.canonical(), "https://foo.com/users/123?tab=posts");

    // The shape: variable parts become placeholders, and the fragment goes.
    assert_eq!(
        normalize("https://foo.com/users/123#bio")?,
        "https://foo.com/users/{user_id}"
    );

    // Pull URLs out of free text, including scheme-less ones like foo.com/users.
    let urls = Extractor::new().extract_strings("Visit https://foo.com today, or foo.com/users.");
    assert_eq!(urls.len(), 2);

    // Why each part rendered the way it did (the CLI's `-e`).
    let t = trace("https://shop.com/pricing/usd?currency=eur")?;
    assert_eq!(t.normalized, "https://shop.com/pricing/USD?currency=EUR");
    Ok(())
}

§Learn from a stream with a corpus

A Corpus observes URLs and learns from what it sees: which slots vary, which query params are enums, which values are HTTP statuses. It groups what it has seen into clusters, one per route.

use iriq::Corpus;

fn main() -> iriq::Result<()> {
    let mut corpus = Corpus::new(); // in memory
    for n in 1..=20 {
        corpus.observe(&format!("https://api.foo.com/orders/{n}?status=open"))?;
        corpus.observe(&format!("https://api.foo.com/orders/{n}?status=closed"))?;
    }

    let clusters = corpus.clusters()?;
    assert_eq!(clusters.len(), 1);
    let orders = &clusters[0];
    assert_eq!((orders.host.as_str(), orders.shape.as_str()), ("api.foo.com", "/orders/{order_id}"));
    assert_eq!(orders.count, 40);

    // One URL can't show that ?status is an enum; forty can.
    let status = &orders.param_summary()[0];
    assert_eq!((status.name.as_str(), status.ty.as_str()), ("status", "enum"));
    println!("{} {} conf {:.2}", status.name, status.ty, status.confidence);

    assert_eq!(
        corpus.normalize("https://api.foo.com/orders/99?status=open")?,
        "https://api.foo.com/orders/{order_id}?status={enum}"
    );
    Ok(())
}

The corpus changes a shape only at a position or param it has seen at least 5 times. Until then, corpus.normalize returns exactly what normalize does.

§Persist a corpus

Corpus::open(path) picks the backend by extension: .db, .sqlite and .sqlite3 are SQLite; anything else is JSON. For bulk ingest, hand observe_all a slice of parsed IRIs: on SQLite it commits about a second’s worth at a time, so other processes writing the corpus get turns in between.

use std::io::BufRead;
use iriq::{parse, Corpus};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let lines = std::io::stdin().lock().lines().collect::<Result<Vec<_>, _>>()?;
    // Skip input that isn't a URL.
    let iris: Vec<_> = lines.iter().filter_map(|line| parse(line).ok()).collect();

    let mut corpus = Corpus::open("routes.db")?;
    corpus.observe_all(&iris)?;

    corpus.save("routes.db")?; // its own path: flush in place (a .json corpus is written here)
    corpus.save("routes-export.json")?; // any other path: a JSON export
    Ok(())
}

If observe_all fails part-way, what it already committed stays. When observations must land together or not at all, wrap them in batch: it commits when the closure returns Ok and rolls back on Err or a panic. On SQLite it holds the write lock until then, so keep it short.

save exports only JSON: saving to another .db path returns Error::Unsupported and writes nothing.

SQLite is the one to share. Many processes can observe into one .db at once, taking turns with the write lock: a writer waits up to 10 seconds for it, then returns an error. reinfer and activation rebuild without the lock and take it only to install the result; a batch holds it until the closure returns.
JSON is single-writer. The file is read at open and written at save, so when two processes save the same file, the last one wins.
Activated recognizers. When another process activates a recognizer (the CLI’s --activate-above), a long-lived Corpus picks it up at the start of its next batch, and observe counts as one. Reads outside a batch (normalize, clusters, …) keep classifying with what the corpus had before. Reopen the corpus, or read inside batch (which on SQLite takes the write lock), to see the new ones.

§Errors

Every Corpus operation returns iriq::Result<T>, whose error is iriq::Error. Its Display names the corpus that failed; the underlying cause is its source(). Error is #[non_exhaustive], so a match needs a _ arm.

use std::error::Error as _;
use iriq::{Corpus, Error};

fn main() {
    // Input that doesn't parse.
    let err = Corpus::new().observe("http://").unwrap_err();
    assert!(matches!(err, Error::Parse(_)));

    // A corpus file that can't be opened.
    let path = std::env::temp_dir().join("iriq-no-such-dir/c.json");
    let err = Corpus::open(&path).unwrap_err();
    match &err {
        Error::Io { path, .. } => assert!(path.ends_with("c.json")),
        _ => unreachable!("a missing directory is an I/O error"),
    }
    eprintln!("iriq: {err}: {}", err.source().unwrap());
}

Error::Corrupt is a file that isn’t a usable corpus (a JSON file that isn’t an iriq corpus). Error::Unsupported is one this build can’t use: a SQLite corpus from a newer iriq, or any .db in a build without the sqlite feature. Error::Sqlite (only with that feature) is SQLite refusing an operation.

§Features

sqlite (on by default) bundles SQLite through rusqlite, so there’s no system library to install. If you only need parsing, extraction or normalization, turn it off to skip compiling the C library:

cargo add iriq --no-default-features

In-memory and JSON corpora still work; opening a .db returns Error::Unsupported.

§More

API docs
Project README: the CLI, and how classification and the corpus work
Changelog
License: MIT

Structs§

Cluster
Corpus
CorpusEntry
CrossHostShape
CustomType: The name of a SegmentType::Custom type. Only segment_type_from_name creates one, and never for a built-in name.
Extractor: Pulls IRIs out of free text. scheme_less (default true) also accepts bare host.tld/path forms; set it on a value from Extractor::new().
Identifier: Parsed IRI. For URN-style inputs only Scheme + NSS are populated; Host/Path are empty.
OrderedMap: Insertion-ordered string → string map. Mirrors Ruby Hash ordering for query parameters and other fields where users see the declared order.
ParamSummary
ParseError: Input that isn’t a parseable IRI. The only way the pure functions (parse, normalize, trace, explain) can fail.
Position
PositionStats
ProposalOptions: Thresholds for Corpus::propose_recognizers; a zero field takes its default. Start from ProposalOptions::default() and set fields.
RecognizerProposal
SegmentHint
SegmentPositionStat
TraceResult
TraceRow

Enums§

Classification
Error: Everything a Corpus operation can fail with.
FileKind
HostStrategy
Kind
PositionScope
SegmentType

Constants§

VERSION

Functions§

explain
normalize
normalize_identifier: Normalize an already-parsed IRI (see normalize). hints: false renders bare type placeholders ({integer}) instead of hinted names ({user_id}).
parse
path_shape_for: Convenience matching Ruby’s PathShape.for.
registrable_domain: Returns the registrable apex for a hostname (port not included). Returns the input unchanged for IPv4 literals, single-label hosts, and hosts already at 2-label apex form.
segment_type_from_name: Total version of segment_type_from_str. Ruby models types as symbols, so any name is a valid type; unknown names (activated recognizer proposals like “ghp”) become Custom types.
singularize
trace
trace_identifier: Trace an already-parsed IRI (see trace); hints as in normalize_identifier.

Type Aliases§

Result: Result with Error as the default error type.

Crate iriq

Crate iriq Copy item path

§iriq

§Parse, normalize, extract

§Learn from a stream with a corpus

§Persist a corpus

§Sharing a corpus between processes

§Errors

§Features

§More

Structs§

Enums§

Constants§

Functions§

Type Aliases§

Crate iriq