Skip to main content

Crate semq

Crate semq 

Source
Expand description

SEMQ: deterministic symbolic encoding of float32 vectors.

Five values and one service, backed by a C core that owns every rule:

  • CodecConfig: the rule (quant, phase or orbit).
  • Codec: encodes unit-norm float32 vectors into an Encoding.
  • Encoding: ids, canonical rows, a manifest and two identities; write/read, concat and diff.
  • Diff: added, removed and changed ids with hamming distances.
  • Floor: the envelope of variation observed in null diffs, bound to the config, id kind and reference it was measured against.
  • BuildInfo from build_info.

Errors are the six variants of Error. Handles are freed on drop and are Send + Sync; the views into an Encoding borrow it.

§Quickstart

use std::collections::BTreeMap;
use std::fs::File;

use semq::{Codec, CodecConfig, Encoding, Floor};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // The rule: quant, 4 dimensions, 4 magnitude bins per sign.
    let config = CodecConfig::quant(4, 4)?;
    let codec = Codec::new(&config)?;

    // Unit-norm float32 rows, one per id, in any order.
    let ids = [3u64, 1, 2];
    let vectors = [
        0.5f32, 0.5, 0.5, 0.5, //
        -0.5, -0.5, -0.5, -0.5, //
        1.0, 0.0, 0.0, 0.0,
    ];
    let manifest = BTreeMap::from([("encoder".to_string(), "my-model@1".to_string())]);
    let reference = codec.encode(&ids, &vectors, Some(&manifest))?;

    // Save, then load in another process: the identity is the same.
    reference.write(File::create("reference.semq")?)?;
    let loaded = Encoding::read(File::open("reference.semq")?)?;
    assert_eq!(loaded.state_id(), reference.state_id());

    // Measure the floor from a null rebuild of the reference, then
    // compare a candidate against it.
    let null = loaded.diff(&codec.encode(&ids, &vectors, Some(&manifest))?)?;
    let floor = Floor::measure([&null])?;
    let candidate = codec.encode(&ids, &vectors, Some(&manifest))?;
    let diff = loaded.diff(&candidate)?;
    println!(
        "changed {} of {} shared rows; within floor: {}",
        diff.changed().len(),
        diff.n_unchanged() + diff.changed().len() as u64,
        diff.within(&floor)?
    );
    Ok(())
}

Structs§

BuildInfo
Versions and kernel selection of the linked core.
Codec
An immutable encoder for one CodecConfig, safe to share between threads. Freed on drop.
CodecConfig
An immutable value: operator, dimension and the operator’s parameter.
Diff
The result of Encoding::diff.
DiffReport
A Diff as plain data: the report schema without a serializer.
Encoding
An immutable set of ids with one canonical row each, a manifest and two identities.
Floor
An envelope of observed variation, bound to the context it was measured in: the config, the id kind and the state_id of the reference every null diff was taken against.
FloorReport
A Floor as plain data: the semq-floor/1 schema without a serializer.
GateOptions
Which checks Diff::evaluate applies beyond those of Diff::within, and which data Floor::measure_for records for them. Every check is off by default, so a new check never changes an existing verdict.
Ids
A borrowed sequence of ids of one kind.
Verdict
The result of Diff::evaluate: whether the candidate passed, every check that failed, and the rows the per-row check flagged.

Enums§

Error
An error from the SDK. Each core status maps to exactly one variant.
Id
One owned id of either kind.
IdKind
The two id kinds. One per Encoding, never mixed.
Operator
The three operators. Values are pinned in the canonical form.
Reason
A check that failed in a Verdict.
Which
Which integrity check failed on load: the check, not a cause.

Functions§

build_info
Query the linked core.

Type Aliases§

Manifest
A manifest: UTF-8 keys to UTF-8 values, stored verbatim and never verified. Keys are at most 256 bytes, values at most 65536 bytes, and there are at most 4096 pairs; the core enforces the limits.
Result
Result with this crate’s Error.