llm-verify

English · 简体中文

Verify the LLM endpoint you are actually using: model authenticity, billing inflation, relay provenance, performance and silent downgrades.

A single binary with no runtime dependencies. Results come out as an HTML report you open in a browser.

Install

From crates.io, with a Rust toolchain (1.82+):

cargo install llm-verify

Without a toolchain, the install script fetches a prebuilt binary:

curl -fsSL https://raw.githubusercontent.com/asale-ai/llm-verify/main/install.sh | sh

On Windows: irm https://raw.githubusercontent.com/asale-ai/llm-verify/main/install.ps1 | iex

Use

llm-verify --base-url https://api.anthropic.com \
           --api-key sk-ant-... \
           --model claude-opus-4-5

It writes an HTML report and opens it.

Credentials can live in .env or the environment instead, leaving only the model on the command line:

# .env
LLM_VERIFY_BASE_URL=https://api.anthropic.com
LLM_VERIFY_API_KEY=sk-ant-...
LLM_VERIFY_MODEL=claude-opus-4-5

llm-verify

Example run

Probing gpt-5.6-sol through OpenRouter at the default balanced depth — 40 probes, 47 requests, 131.2s:

The summary that follows carries the verdict, the two axes, and the per-group scores:

Relayed / Relay: a real model, reached through one hop. Channel provenance and performance are what drag the groups down, not identity or billing.

Full HTML reports from two runs against the same model:

Same model, same verdict, two points apart — and the reports still differ. OpenRouter names itself in the response headers, so the origin is pinned to a single relay hop; the second path carries no channel markers at all, which leaves the origin undetermined rather than proven clean. Origin is a separate reading from both score and verdict.

Options

Exit codes

Usable as a CI gate as-is.

Reading the report

The verdict has two independent axes:

• Authenticity — genuine / genuine-with-defects / relayed / suspicious / counterfeit / inconclusive
• Origin — direct from vendor / cloud platform / subscription-derived / relay / reconstructed channel / undetermined

A real model behind a relay is relayed, not counterfeit. Longer path, same model.

Hard gates are facts no weighted score can excuse. Any one of them forces a suspicious verdict and exit code 2:

silent fallback · shared-pool forwarding · tier downgrade · third-party wrapper injection · cache replay · hidden prompt injection · response replay

Use it from an AI coding tool

The skill lives in this repository at skills/llm-verify/SKILL.md. Install it with skills, which supports Claude Code, Codex, Cursor, OpenCode and 70-odd other agents:

npx skills add asale-ai/llm-verify

npx skills add asale-ai/llm-verify -g              # user-level, every project
npx skills add asale-ai/llm-verify -a claude-code  # one agent only
npx skills add asale-ai/llm-verify --list          # look before installing

It is also published on ClawHub:

clawhub install @asale-ai/llm-verify

Then just ask: "is this API actually giving me what I paid for?"

The skill is only the usage guide — the llm-verify binary does the work, so you need both.

What it checks

40 probes across seven groups:

Use it as a library

The binary is a thin CLI over the crate, so an embedder gets the same probes, the same order and the same verdict — which is what lets it claim its own results and the published tool agree.

[dependencies]
llm-verify = { version = "0.4", default-features = false }

default-features = false drops the CLI half: clap, the terminal writer and the HTML renderer are dead weight in a service.

use llm_verify::{engine, probes::Cancel, Endpoint};

let cfg = engine::RunConfig::new(Endpoint {
    base_url: "https://api.anthropic.com".into(),
    api_key: std::env::var("ANTHROPIC_API_KEY")?,
    model: "claude-opus-4-5".into(),
    ..Default::default()
});
let report = engine::run(cfg, &Cancel::new(), &mut |_| {}).await?;
println!("{:?} {}", report.verdict.authenticity, report.verdict.score);

Probing through a relay

Half the suite asks about the endpoint — its error envelopes, its response headers, the token counts it reports. Behind a relay those describe the relay, not the model, and several of them deliberately send malformed or unauthenticated requests, which is not traffic you want attributed to somebody else's account.

Selection::model_only() keeps only the steps whose evidence is the generated text itself, which survives any number of hops:

let cfg = engine::RunConfig::new(endpoint)
    .model_only()
    .depth(llm_verify::Depth::Fast)
    // Chosen by the caller and recorded in the report, so a contested verdict
    // can be replayed probe for probe.
    .seed(0x5EED)
    // Spread the run out instead of issuing it as one recognisable burst.
    .pace(Duration::from_secs(20), Duration::from_secs(180));

When somebody is waiting for the answer

Selection::turbo() is model_only cut down to the probes a conclusion actually rests on, run several at a time:

let cfg = engine::RunConfig::new(endpoint).turbo(3);

The argument is how many requests the endpoint will answer at once — a statement about the far end, not about this crate. Set it too high and the run measures the endpoint's saturation instead of its behaviour, and on anything with a per-account concurrency budget it spends that budget on being examined. When in doubt, three.

Two things make overlapping safe rather than merely fast. Steps that measure a clock run alone, so a latency figure is never taken while this run has three other requests in the air. And each step draws its random payloads from its own generator, seeded from the run's seed and the step's id, so the same seed reproduces the same questions whatever order the scheduler happened to run them in.

What turbo gives up is documented probe by probe on Selection::turbo. The short version: the corroborating and merely descriptive identity probes, the dated-trivia knowledge check, and the three contract checks that catch an endpoint reconstructed from a web session. Put the last group back if you have not otherwise established what is on the far end:

Selection::turbo().plus(["max_tokens_truncation", "stop_sequence", "system_adherence"])

Progress, cancellation, and your own probes

engine::run reports each step as it starts and finishes, and takes a Cancel that is honoured between steps. Selection::with appends your own [Probe] implementations, which share the run's client, seed and observation buffers — so a private probe is indistinguishable from a built-in one both in the report and on the wire. That matters if you are policing a marketplace: everything in this repository is readable by the endpoint you are probing.

To put your own graded questions in place of the published battery — worth doing for exactly that reason — use replacing, which drops the built-in step and lets your probe answer to its id:

Selection::turbo().replacing("capability", Arc::new(my_bank))

Not minus(["capability"]).with(bank). skip applies to caller-supplied probes too, deliberately, so that spelling removes both and reports nothing: the run comes back with every other probe passing and the questions never asked.

What it costs

Measured, not estimated — tests/request_budget.rs pins these so that adding a probe cannot quietly multiply anyone's bill:

Limits

One false accusation against an honest provider costs far more than one miss. This tool abstains when the evidence is thin rather than guessing. Please know the following:

• Resolution stops at tier granularity (flagship / mid / light). Adjacent versions inside one tier — say 4.5 and 4.6 of the same line — cannot be separated.
• The tier call depends on sampling. The default depth asks 9 capability questions, and adjacent tiers can still swing on a single one. The tool abstains when the margin is narrow, which costs it some genuine downgrades. Use --depth forensic (15 questions) when the answer has to hold up. The report states how many questions the call rests on and by how much the winner beat the runner-up.
• Delivering above the claimed tier is not fraud and carries no risk weight. Only measuring below the claim counts.
• Middle layers contaminate identity fingerprints, which is why the contract layer runs first; where injection is found, identity confidence is reduced automatically.
• Server-side weights cannot be proven — only whether behaviour matches expectations.
• One run describes one moment. Gradual degradation needs periodic re-runs and comparison.
• The probes are public. That is the point of an auditable tool, and it is a ceiling: an endpoint can read this repository too, and answer these questions honestly while serving everything else from somewhere cheaper. Timing can be disguised (RunConfig::pace); the questions cannot be. Anyone using this to police supply they do not control needs probes of their own on top — see Selection::with — and should treat the result as a probability rather than a proof.

Licence

Apache-2.0