iriq

iriq finds the shape of a URL: the route template behind it. Erase the parts that vary, keep the parts that don't, and /users/123 and /users/999 both become /users/{user_id}. Point it at a pile of messy URLs and it collapses them into a small set of stable, deterministic templates.

An IRI is a URL that's allowed to contain non-ASCII characters; if you know URLs, you know IRIs. The name is IRI Query.

This crate is both the library and the iriq command-line tool. This page covers the library; the project README covers the CLI.

cargo add iriq          # library
cargo install iriq      # CLI

Requires Rust 1.85 or newer.

Parse, normalize, extract

The pure functions need no setup and fail only with ParseError.

use iriq::{normalize, parse, trace, Extractor};

fn main() -> Result<(), iriq::ParseError> {
    let iri = parse("https://Foo.com:443/users/123?tab=posts")?;
    assert_eq!(iri.host, "foo.com");
    assert_eq!(iri.port, None); // the scheme's default port is dropped
    assert_eq!(iri.path_segments, ["users", "123"]);
    assert_eq!(iri.canonical(), "https://foo.com/users/123?tab=posts");

    // The shape: variable parts become placeholders, and the fragment goes.
    assert_eq!(
        normalize("https://foo.com/users/123#bio")?,
        "https://foo.com/users/{user_id}"
    );

    // Pull URLs out of free text, including scheme-less ones like foo.com/users.
    let urls = Extractor::new().extract_strings("Visit https://foo.com today, or foo.com/users.");
    assert_eq!(urls.len(), 2);

    // Why each part rendered the way it did (the CLI's `-e`).
    let t = trace("https://shop.com/pricing/usd?currency=eur")?;
    assert_eq!(t.normalized, "https://shop.com/pricing/USD?currency=EUR");
    Ok(())
}

Learn from a stream with a corpus

A Corpus observes URLs and learns from what it sees: which slots vary, which query params are enums, which values are HTTP statuses. It groups what it has seen into clusters, one per route.

use iriq::Corpus;

fn main() -> iriq::Result<()> {
    let mut corpus = Corpus::new(); // in memory
    for n in 1..=20 {
        corpus.observe(&format!("https://api.foo.com/orders/{n}?status=open"))?;
        corpus.observe(&format!("https://api.foo.com/orders/{n}?status=closed"))?;
    }

    let clusters = corpus.clusters()?;
    assert_eq!(clusters.len(), 1);
    let orders = &clusters[0];
    assert_eq!((orders.host.as_str(), orders.shape.as_str()), ("api.foo.com", "/orders/{order_id}"));
    assert_eq!(orders.count, 40);

    // One URL can't show that ?status is an enum; forty can.
    let status = &orders.param_summary()[0];
    assert_eq!((status.name.as_str(), status.ty.as_str()), ("status", "enum"));
    println!("{} {} conf {:.2}", status.name, status.ty, status.confidence);

    assert_eq!(
        corpus.normalize("https://api.foo.com/orders/99?status=open")?,
        "https://api.foo.com/orders/{order_id}?status={enum}"
    );
    Ok(())
}

The corpus changes a shape only at a position or param it has seen at least 5 times. Until then, corpus.normalize returns exactly what normalize does.

Persist a corpus

Corpus::open(path) picks the backend by extension: .db, .sqlite and .sqlite3 are SQLite; anything else is JSON. For bulk ingest, hand observe_all a slice of parsed IRIs: on SQLite it commits about a second's worth at a time, so other processes writing the corpus get turns in between.

use std::io::BufRead;
use iriq::{parse, Corpus};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let lines = std::io::stdin().lock().lines().collect::<Result<Vec<_>, _>>()?;
    // Skip input that isn't a URL.
    let iris: Vec<_> = lines.iter().filter_map(|line| parse(line).ok()).collect();

    let mut corpus = Corpus::open("routes.db")?;
    corpus.observe_all(&iris)?;

    corpus.save("routes.db")?; // its own path: flush in place (a .json corpus is written here)
    corpus.save("routes-export.json")?; // any other path: a JSON export
    Ok(())
}

If observe_all fails part-way, what it already committed stays. When observations must land together or not at all, wrap them in batch: it commits when the closure returns Ok and rolls back on Err or a panic. On SQLite it holds the write lock until then, so keep it short.

save exports only JSON: saving to another .db path returns Error::Unsupported and writes nothing.

Sharing a corpus between processes

• SQLite is the one to share. Many processes can observe into one .db at once, taking turns with the write lock: a writer waits up to 10 seconds for it, then returns an error. reinfer and activation rebuild without the lock and take it only to install the result; a batch holds it until the closure returns.
• JSON is single-writer. The file is read at open and written at save, so when two processes save the same file, the last one wins.
• Activated recognizers. When another process activates a recognizer (the CLI's --activate-above), a long-lived Corpus picks it up at the start of its next batch, and observe counts as one. Reads outside a batch (normalize, clusters, …) keep classifying with what the corpus had before. Reopen the corpus, or read inside batch (which on SQLite takes the write lock), to see the new ones.

Errors

Every Corpus operation returns iriq::Result<T>, whose error is iriq::Error. Its Display names the corpus that failed; the underlying cause is its source(). Error is #[non_exhaustive], so a match needs a _ arm.

use std::error::Error as _;
use iriq::{Corpus, Error};

fn main() {
    // Input that doesn't parse.
    let err = Corpus::new().observe("http://").unwrap_err();
    assert!(matches!(err, Error::Parse(_)));

    // A corpus file that can't be opened.
    let path = std::env::temp_dir().join("iriq-no-such-dir/c.json");
    let err = Corpus::open(&path).unwrap_err();
    match &err {
        Error::Io { path, .. } => assert!(path.ends_with("c.json")),
        _ => unreachable!("a missing directory is an I/O error"),
    }
    eprintln!("iriq: {err}: {}", err.source().unwrap());
}

Error::Corrupt is a file that isn't a usable corpus (a JSON file that isn't an iriq corpus). Error::Unsupported is one this build can't use: a SQLite corpus from a newer iriq, or any .db in a build without the sqlite feature. Error::Sqlite (only with that feature) is SQLite refusing an operation.

Features

sqlite (on by default) bundles SQLite through rusqlite, so there's no system library to install. If you only need parsing, extraction or normalization, turn it off to skip compiling the C library:

cargo add iriq --no-default-features

In-memory and JSON corpora still work; opening a .db returns Error::Unsupported.

More

• API docs
• Project README: the CLI, and how classification and the corpus work
• Changelog
• License: MIT