Renegade

A nonparametric supervised learning library for Rust. Zero configuration, competitive with scikit-learn out of the box.

Renegade is a KNN-based learner that just works — no hyperparameters to tune, no preprocessing pipeline to configure. It handles mixed numeric and categorical features, automatically selects K, learns which features matter, and indexes data for fast queries. You add data, you get predictions.

Benchmarks

Leave-one-out cross-validation against scikit-learn's KNN with StandardScaler and tuned K:

Renegade wins 5 of 6 standard ML datasets with zero configuration. sklearn requires choosing a scaler, distance metric, and K for each dataset.

Performance

• Training is amortized — only recomputes when the dataset grows 50%. The VP-tree rebuilds independently every ~20% growth (~15ms at 10k points).
• New data points are immediately queryable without retraining.
• Instance weights support recency decay for online learning.

Quick Start

cargo add renegade-ml

use renegade_ml::{DataPoint, Renegade};

#[derive(Clone)]
struct Peer {
    distance: f64,     // network distance
    latency_ms: f64,   // recent avg latency
    origin: u8,        // region (categorical)
}

impl DataPoint for Peer {
    fn feature_distances(&self, other: &Self) -> Vec<f64> {
        vec![
            (self.distance - other.distance).abs() / 1.0,       // already [0, 1]
            (self.latency_ms - other.latency_ms).abs() / 500.0, // normalize
            if self.origin == other.origin { 0.0 } else { 1.0 },
        ]
    }

    fn feature_values(&self) -> Vec<f64> {
        vec![self.distance, self.latency_ms, self.origin as f64]
    }
}

let mut model = Renegade::new();

// Add observations (with optional recency weighting)
model.add(peer_a, success_rate_a);
model.add_weighted(peer_b, success_rate_b, 0.5);  // half weight (older observation)

// Predict — auto-selects K, learns metric, builds index
let predicted = model.predict(&query_peer);             // weighted mean
let neighbors = model.query(&query_peer);               // raw neighbors
let class_probs = neighbors.class_votes();              // classification
let extrapolated = model.predict_extrapolated(&query);  // with R² confidence
let dispersion = neighbors.dispersion();                 // agreement behind neighbors.weighted_mean()

// Expire stale data
model.retain(|_peer, _output| /* keep if recent */ true);

How It Works

Gower Distance + Auto K

Each feature contributes a distance in [0, 1]:

• Numeric: |a - b| / range
• Categorical: 0 if same, 1 if different
• Custom: edit distance, Jaccard, etc. — anything normalized to [0, 1]

K is selected automatically via leave-one-out cross-validation.

Effect-Space Metric Learning

For each feature, an isotonic regression learns its marginal effect on the output. Features that predict the output get high weight; noise features get zero weight. Distances are computed in this "effect space."

It's the same isotonic regression used to calibrate classifier probabilities, pointed sideways: instead of mapping scores → calibrated probabilities, it maps each feature → its marginal effect on the target, and the fit's R² becomes that feature's weight.

The metric is only kept when it demonstrably improves LOO error. Otherwise it falls back to simple Gower distance. The metric never hurts.

Dispersion: Do the Neighbors Agree?

A weighted mean alone can't tell you whether the neighborhood behind it agreed, or whether a few outliers happened to cancel out against a pile of unrelated noise — both can produce the same point estimate. neighbors.dispersion() (and neighbors.gaussian_dispersion(bandwidth) for the Gaussian-kernel prediction path) report weighted variance/std-dev about that same mean, plus two ways to size the evidence behind it:

• effective_n (Kish's effective sample size) answers "how many roughly independent observations back this estimate" — but it's scale-invariant, so k neighbors report effective_n ≈ k no matter how far away they are.
• weight_sum is the raw kernel mass. For gaussian_dispersion's bounded kernel it decays toward 0 as the query moves away from the training data, so it's the one to use for "is there evidence near this query" — dispersion()'s inverse-distance kernel is unbounded and doesn't have that property.

Treating neighbor count as a proxy for neighbor agreement is a real trap: a categorical feature can pull in a large, confident-looking neighborhood built from observations that don't actually agree with each other.

dispersion() always matches weighted_mean() on the same Neighbors, and gaussian_dispersion(bandwidth) always matches gaussian_weighted_mean(bandwidth) on the same Neighbors — but model.predict() picks one of the two strategies (and, for the Gaussian one, a different-sized neighbor set) automatically during training. Call model.diagnostics().kernel_bandwidth to find out which one is active if you want a dispersion figure that's guaranteed to match what predict() actually returned.

Shrinkage: How Much Should You Trust This Mean?

A small, high-variance neighborhood is exactly where weighted_mean() is least reliable — few points, disagreeing outputs. dispersion().standard_error() gives the standard error of that mean (sqrt(variance / effective_n) — effective_n, never weight_sum, for the same reason Dispersion itself uses it), and shrink_toward(prior, signal_variance) pulls the mean toward a caller-supplied prior by an amount that depends on it:

let neighbors = model.query(&query);
let dispersion = neighbors.dispersion().unwrap();

// signal_variance: how much of the dataset's spread is real local signal,
// as opposed to noise, near this particular query.
let signal_variance = model.local_signal_variance(&dispersion).unwrap();
let shrunk = dispersion.shrink_toward(global_prior, signal_variance);

shrunk.estimate  // prior + lambda * (dispersion.mean - prior)
shrunk.lambda    // 0 (fully trust the prior) .. 1 (fully trust the local mean)

model.shrink(&neighbors, prior) wires this together in one call, using the model's own local_signal_variance — the recommended entry point unless the Gaussian-kernel dispersion is what you're shrinking.

The subtlety is signal_variance. A single global "how much does the signal vary" constant gets inflated by any strongly localized effect elsewhere in the dataset, which keeps lambda high — trusting a noisy local mean — even in a flat region with no real signal at all. local_signal_variance instead compares THIS neighborhood's dispersion against global_output_variance() (the variance of every stored output): a neighborhood no tighter than the dataset as a whole has demonstrated no more than noise, so signal ≈ 0 and shrinkage is aggressive; a neighborhood far tighter than the dataset as a whole has captured something real, so signal stays high and the local mean is trusted. It's a method-of-moments estimate (the same local/global variance split a one-way ANOVA uses), not a calibrated quantity — noisiest exactly when effective_n is small.

model.predict_with_prior(&query, prior) wires the whole thing up through predict()'s own auto-selected K and kernel: train, query, and shrink toward prior in one call. This is opt-in, not predict()'s default — a mostly-flat target with a small genuinely-learnable region is exactly the case shrinkage is meant to help, and on a direct reconstruction of that case it measurably made both the flat-region AND the signal-region predictions worse, not better (see Renegade::predict_with_prior's doc comment for the numbers and the mechanism). Evaluate it on your own data before reaching for it as a default.

VP-Tree Indexing

A vantage-point tree provides exact nearest neighbor search (not approximate) with any distance function. Queries are O(log n) average case — 347× faster than brute force at 10k points.

The tree rebuilds automatically as data grows. Between rebuilds, new points are searched via a small brute-force tail scan.

Diagnostics

let diag = model.diagnostics();
// diag.optimal_k          — current K
// diag.metric_active       — whether learned metric is in use
// diag.feature_metrics     — per-feature weights and effect curves
// diag.output_stats        — min, max, mean, distinct count

let pred = model.predict_with_diagnostics(&query, k);
// pred.prediction          — predicted value
// pred.neighbors           — per-neighbor distance, output, feature breakdown

Design Philosophy

• No hyperparameters — every parameter is an opportunity for misconfiguration
• No multivariate optimization — no gradient descent, no learning rates, no convergence
• Correct by default — VP-tree gives exact results, metric fallback prevents regressions
• Online-friendly — incremental insertion, instance weighting, data eviction via retain()

Intended Use Cases

• Routing decisions based on historical peer performance (e.g., peer selection in Freenet)
• Online learning with moderate data volumes
• Mixed-type data where features are numeric, categorical, or custom
• Low-data regimes where parametric models overfit

License

LGPL-3.0-or-later

If LGPL doesn't work for your use case, alternative licensing is available — reach out on X (@sanity) or open a GitHub issue.