llm-verify 0.5.1

# llm-verify

**English** · [简体中文](README.zh-CN.md)

Verify the LLM endpoint you are actually using: model authenticity, billing inflation, relay provenance, performance and silent downgrades.

A single binary with no runtime dependencies. Results come out as an HTML report you open in a browser.

## Install

From crates.io, with a Rust toolchain (1.82+):

```bash
cargo install llm-verify
```

Without a toolchain, the install script fetches a prebuilt binary:

```bash
curl -fsSL https://raw.githubusercontent.com/asale-ai/llm-verify/main/install.sh | sh
```

On Windows: `irm https://raw.githubusercontent.com/asale-ai/llm-verify/main/install.ps1 | iex`

## Use

```bash
llm-verify --base-url https://api.anthropic.com \
           --api-key sk-ant-... \
           --model claude-opus-4-5
```

It writes an HTML report and opens it.

Credentials can live in `.env` or the environment instead, leaving only the model on the command line:

```bash
# .env
LLM_VERIFY_BASE_URL=https://api.anthropic.com
LLM_VERIFY_API_KEY=sk-ant-...
LLM_VERIFY_MODEL=claude-opus-4-5
```

```bash
llm-verify
```

### Example run

Probing `gpt-5.6-sol` through OpenRouter at the default `balanced` depth — 40 probes, 47 requests, 131.2s:

![Probe run](docs/ScreenShot_1.png)

The summary that follows carries the verdict, the two axes, and the per-group scores:

![Verdict summary](docs/ScreenShot_2.png)

`Relayed` / `Relay`: a real model, reached through one hop. Channel provenance and performance are what drag the groups down, not identity or billing.

Full HTML reports from two runs against the same model:

| Report | Verdict | Origin | Score |
|---|---|---|---|
| [openrouter.ai](docs/llm-verify-openrouter-ai-20260815-125833.html) ([preview](https://htmlpreview.github.io/?https://github.com/asale-ai/llm-verify/blob/main/docs/llm-verify-openrouter-ai-20260815-125833.html)) | Relayed | Relay | 92 / 100 |
| [gw.asale.ai](docs/llm-verify-gw-asale-ai-20260815-130656.html) ([preview](https://htmlpreview.github.io/?https://github.com/asale-ai/llm-verify/blob/main/docs/llm-verify-gw-asale-ai-20260815-130656.html)) | Relayed | Undetermined | 93 / 100 |

Same model, same verdict, two points apart — and the reports still differ. OpenRouter names itself in the response headers, so the origin is pinned to a single relay hop; the second path carries no channel markers at all, which leaves the origin *undetermined* rather than proven clean. Origin is a separate reading from both score and verdict.

### Options

| Flag | Meaning |
|---|---|
| `--protocol anthropic\|openai` | Inferred from the URL and model name if omitted |
| `--depth fast\|balanced\|forensic` | Default `balanced`. `forensic` samples more — slower and costlier, but firmer |
| `--turbo` | The quickest run that still reaches a verdict: 9 requests instead of 21, several at a time. Gives up the corroborating identity probes and the contract checks that catch a reconstructed endpoint |
| `--concurrency N` | Requests to keep in flight. Default 1. Only raise it against an endpoint you know answers that many at once — latency probes always run alone |
| `--claimed-model <ID>` | Use when the vendor's advertised name differs from the ID you request. This is how you check for a downgrade |
| `--lang en\|zh` | Report language. Follows the system locale, then falls back to English |
| `-o <path>` | HTML report path; pass a directory to auto-name the file |
| `--json <path>` | Also emit machine-readable JSON |
| `--no-open` | Do not open a browser |

### Exit codes

| Code | Meaning |
|---|---|
| 0 | Clean |
| 1 | Failing score, or a suspicious / counterfeit / inconclusive verdict |
| 2 | A hard gate tripped |

Usable as a CI gate as-is.

## Reading the report

The verdict has two **independent** axes:

- **Authenticity** — genuine / genuine-with-defects / relayed / suspicious / counterfeit / inconclusive
- **Origin** — direct from vendor / cloud platform / subscription-derived / relay / reconstructed channel / undetermined

A real model behind a relay is *relayed*, not *counterfeit*. Longer path, same model.

**Hard gates** are facts no weighted score can excuse. Any one of them forces a suspicious verdict and exit code 2:

silent fallback · shared-pool forwarding · tier downgrade · third-party wrapper injection · cache replay · hidden prompt injection · response replay

## Use it from an AI coding tool

The skill lives in this repository at [`skills/llm-verify/SKILL.md`](skills/llm-verify/SKILL.md). Install it with [`skills`](https://skills.sh), which supports Claude Code, Codex, Cursor, OpenCode and 70-odd other agents:

```bash
npx skills add asale-ai/llm-verify
```

```bash
npx skills add asale-ai/llm-verify -g              # user-level, every project
npx skills add asale-ai/llm-verify -a claude-code  # one agent only
npx skills add asale-ai/llm-verify --list          # look before installing
```

It is also published on [ClawHub](https://clawhub.ai):

```bash
clawhub install @asale-ai/llm-verify
```

Then just ask: *"is this API actually giving me what I paid for?"*

The skill is only the usage guide — the `llm-verify` binary does the work, so you need both.

## What it checks

40 probes across seven groups:

| Group | Question it answers |
|---|---|
| Protocol contract | Is this a genuine API channel? |
| Streaming | Does streaming follow the protocol, or arrive empty? |
| Metering & billing | Are the token counts honest, or are you overcharged? |
| Channel provenance | What relays sit on this path? |
| Performance | First-token latency, throughput and jitter |
| Model identity | Is the model behind this the one that was sold? |
| Cross-request consistency | Does the endpoint behave the same way every time? |

## Use it as a library

The binary is a thin CLI over the crate, so an embedder gets the same probes, the same order and the same verdict — which is what lets it claim its own results and the published tool agree.

```toml
[dependencies]
llm-verify = { version = "0.4", default-features = false }
```

`default-features = false` drops the CLI half: clap, the terminal writer and the HTML renderer are dead weight in a service.

```rust
use llm_verify::{engine, probes::Cancel, Endpoint};

let cfg = engine::RunConfig::new(Endpoint {
    base_url: "https://api.anthropic.com".into(),
    api_key: std::env::var("ANTHROPIC_API_KEY")?,
    model: "claude-opus-4-5".into(),
    ..Default::default()
});
let report = engine::run(cfg, &Cancel::new(), &mut |_| {}).await?;
println!("{:?} {}", report.verdict.authenticity, report.verdict.score);
```

### Probing through a relay

Half the suite asks about *the endpoint* — its error envelopes, its response headers, the token counts it reports. Behind a relay those describe the relay, not the model, and several of them deliberately send malformed or unauthenticated requests, which is not traffic you want attributed to somebody else's account.

`Selection::model_only()` keeps only the steps whose evidence is the generated text itself, which survives any number of hops:

```rust
let cfg = engine::RunConfig::new(endpoint)
    .model_only()
    .depth(llm_verify::Depth::Fast)
    // Chosen by the caller and recorded in the report, so a contested verdict
    // can be replayed probe for probe.
    .seed(0x5EED)
    // Spread the run out instead of issuing it as one recognisable burst.
    .pace(Duration::from_secs(20), Duration::from_secs(180));
```

### When somebody is waiting for the answer

`Selection::turbo()` is `model_only` cut down to the probes a conclusion actually rests on, run several at a time:

```rust
let cfg = engine::RunConfig::new(endpoint).turbo(3);
```

The argument is how many requests the endpoint will answer at once — a statement about the far end, not about this crate. Set it too high and the run measures the endpoint's saturation instead of its behaviour, and on anything with a per-account concurrency budget it spends that budget on being examined. When in doubt, three.

Two things make overlapping safe rather than merely fast. Steps that measure a clock run alone, so a latency figure is never taken while this run has three other requests in the air. And each step draws its random payloads from its own generator, seeded from the run's seed and the step's id, so the same seed reproduces the same questions whatever order the scheduler happened to run them in.

What turbo gives up is documented probe by probe on `Selection::turbo`. The short version: the corroborating and merely descriptive identity probes, the dated-trivia knowledge check, and the three contract checks that catch an endpoint reconstructed from a web session. Put the last group back if you have not otherwise established what is on the far end:

```rust
Selection::turbo().plus(["max_tokens_truncation", "stop_sequence", "system_adherence"])
```

### Progress, cancellation, and your own probes

`engine::run` reports each step as it starts and finishes, and takes a `Cancel` that is honoured between steps. `Selection::with` appends your own [`Probe`] implementations, which share the run's client, seed and observation buffers — so a private probe is indistinguishable from a built-in one both in the report and on the wire. That matters if you are policing a marketplace: everything in this repository is readable by the endpoint you are probing.

To put your own graded questions in place of the published battery — worth doing for exactly that reason — use `replacing`, which drops the built-in step and lets your probe answer to its id:

```rust
Selection::turbo().replacing("capability", Arc::new(my_bank))
```

Not `minus(["capability"]).with(bank)`. `skip` applies to caller-supplied probes too, deliberately, so that spelling removes both and reports nothing: the run comes back with every other probe passing and the questions never asked.

### What it costs

Measured, not estimated — `tests/request_budget.rs` pins these so that adding a probe cannot quietly multiply anyone's bill:

| Selection | Depth | Requests |
|---|---|---|
| `turbo` | `fast` | 9 |
| `model_only` | `fast` | 21 |
| `model_only` | `forensic` | 46 |
| everything | `balanced` | 49 |

## Limits

One false accusation against an honest provider costs far more than one miss. This tool abstains when the evidence is thin rather than guessing. Please know the following:

- **Resolution stops at tier granularity** (flagship / mid / light). Adjacent versions inside one tier — say 4.5 and 4.6 of the same line — cannot be separated.
- **The tier call depends on sampling.** The default depth asks 9 capability questions, and adjacent tiers can still swing on a single one. The tool abstains when the margin is narrow, which costs it some genuine downgrades. Use `--depth forensic` (15 questions) when the answer has to hold up. The report states how many questions the call rests on and by how much the winner beat the runner-up.
- **Delivering above the claimed tier is not fraud** and carries no risk weight. Only measuring *below* the claim counts.
- **Middle layers contaminate identity fingerprints**, which is why the contract layer runs first; where injection is found, identity confidence is reduced automatically.
- **Server-side weights cannot be proven** — only whether behaviour matches expectations.
- **One run describes one moment.** Gradual degradation needs periodic re-runs and comparison.
- **The probes are public.** That is the point of an auditable tool, and it is a ceiling: an endpoint can read this repository too, and answer these questions honestly while serving everything else from somewhere cheaper. Timing can be disguised (`RunConfig::pace`); the questions cannot be. Anyone using this to police supply they do not control needs probes of their own on top — see `Selection::with` — and should treat the result as a probability rather than a proof.

## Licence

[Apache-2.0](LICENSE)