jev-bridge

English | 简体中文

Turn any generic OpenAI-compatible inference service into a Jev-style decision scoring service, in a single 7 MB binary with no Python dependency.

What it does

Jev-style decisions are not a model format; they are a readout. This bridge implements the direct-options-v1 contract on top of any runtime that can return next-token log probabilities:

1. render a fixed decision prompt through the serving runtime's own chat template (system instruction plus a {"evidence", "criterion", "options"} JSON payload, thinking disabled);
2. take the final position of one forward pass;
3. softmax only over the 2–16 fixed uppercase answer letters A–P;
4. return a typed System One result without generating any answer text.

The upstream model, its weights and its quantization stay exactly where they are. This process copies no weights and modifies no model.

What it does not do

• It is not a reimplementation of Jev and does not impersonate it: --served-model refuses names starting with jev.
• Returned probabilities are conditional scores over the supplied options with calibrated=False. They are not operational confidence.
• It performs no calibration and recommends no thresholds. Validate on your own workload before automating anything consequential.

Supported types

Question types (POST /v1/systemone)

instructions is optional for all three; a type-specific generic question is substituted when it is omitted. Criteria values may be strings, objects, arrays or null; structured values are rendered as JSON.

Upstream runtimes

The bridge picks one of these transports at startup by probing, and refuses to start rather than guessing. See Transport probing.

Quality alignment

144 authored decisions from fastjev, scored row by row. The reference is the committed BF16 Torch run; this bridge ran llama.cpp Q8_0 over the same fixture with the same metric definition (mean_family_balanced_accuracy).

prompt_sha256 and input_tokens match on every one of the 144 rows. That is the strongest available evidence that the bridge renders the prompt byte-identically to the reference implementation and tokenizes identically. The 0.0053 quality delta is therefore attributable to Q8_0 quantization, not to the bridge.

All five argmax disagreements fall on rows where the top two options are close (top-2 margins between 0.01 and 0.24). The bridge flips two rows in the right direction and three in the wrong one, which matches the observed net delta. Per family, candidate_selection and rule_application are identical and only evidence_interpretation moves (0.7675 → 0.7516).

Row-level evidence:

• results/alignment-minicpm5-2b-authored144.json
• results/bridge-minicpm5-2b-q8-authored144.predictions.jsonl

Quick start

Build tip: if the crates.io index is slow on your network, configure a mirror in a local .cargo/config.toml. That file is gitignored.

cargo build --release

export JEV_BRIDGE_UPSTREAM_KEY='<your upstream token>'

./target/release/jev-bridge \
  --base-url https://your-llama-server.example/v1 \
  --model MiniCPM5-2B-Q8_0 \
  --served-model bridge-minicpm5-2b \
  --served-model-release-date 2026-09-22 \
  --port 8100

Probe only, without serving:

./target/release/jev-bridge --probe-only \
  --base-url https://your-llama-server.example/v1 \
  --model MiniCPM5-2B-Q8_0 \
  --served-model-release-date 2026-09-22

The probe prints the chosen transport, its endpoint, and the sixteen answer-slot token ids it resolved.

curl http://127.0.0.1:8100/v1/systemone \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "bridge-minicpm5-2b",
    "state": "I was charged twice and need a refund today.",
    "questions": {
      "department": {
        "type": "choice",
        "instructions": "Which queue should handle this request?",
        "criteria": {
          "billing": "Billing, payments, and refunds.",
          "technical": "Bugs, outages, and integrations."
        }
      }
    }
  }'

Batch scoring

--score reads fastjev-format rows and writes JSONL predictions, which is what the alignment run above used:

./target/release/jev-bridge --score \
  --input authored144.jsonl \
  --output bridge-predictions.jsonl \
  --base-url https://your-llama-server.example/v1 \
  --model MiniCPM5-2B-Q8_0 \
  --served-model bridge-minicpm5-2b \
  --served-model-release-date 2026-09-22

Input rows are {"id", "state", "question", "options": [{"id", "description"}]}; extra fields are ignored. Output rows carry probabilities, option_logprobs, input_tokens, prompt_sha256 and prompt_version. The output path must not already exist.

This path bypasses the System One criteria rendering (which prefixes each option description with id: ), so it matches the reference prompt contract exactly — that is why the alignment check can compare prompt hashes byte for byte.

Endpoints

GET /health is the first place to look when something is off: it states what the bridge actually negotiated instead of leaving you to guess.

Using it as a library

[dependencies]
jev-bridge = "0.1"
tokio = { version = "1", features = ["macros", "rt-multi-thread"] }
reqwest = { version = "0.12", features = ["json"] }
serde_json = "1"

use jev_bridge::server::{Bridge, BridgeConfig};
use jev_bridge::wire::{Row, RowOption};

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let bridge = Bridge::connect(
        reqwest::Client::new(),
        BridgeConfig {
            openai_base: "http://127.0.0.1:8080/v1".into(),
            native_base: "http://127.0.0.1:8080".into(),
            upstream_model: "MiniCPM5-2B-Q8_0".into(),
            upstream_key: None,
            served_model: "bridge-minicpm5-2b".into(),
            description: "local bridge".into(),
            release_date: "2026-09-22".into(),
            chat_template_kwargs: serde_json::json!({"enable_thinking": false}),
            max_input_tokens: None,
        },
    )
    .await?;

    let rows = vec![Row {
        id: "ticket".into(),
        state: serde_json::json!("I was charged twice."),
        question: "Which queue should handle this?".into(),
        options: vec![
            RowOption { id: "billing".into(), description: "Refunds and payments.".into() },
            RowOption { id: "sales".into(), description: "Pricing.".into() },
        ],
    }];

    for score in bridge.score_rows(&rows).await? {
        println!("{} {:?}", score.answer.id, score.answer.probabilities);
    }
    Ok(())
}

Serve the same bridge over HTTP with jev_bridge::server::router:

use std::sync::Arc;
use jev_bridge::server::{router, AppState};

let state = Arc::new(AppState { bridge, api_key: None });
let listener = tokio::net::TcpListener::bind("127.0.0.1:8100").await?;
axum::serve(listener, router(state)).await?;

Public modules:

Bodies, responses and answers are typed structs rather than loose JSON documents: the scoring request/response pairs, the System One response with its choice/noul/score enum, and the health payload are all Serialize/ Deserialize types. serde_json::Value remains only where the payload really is dynamic — the caller's state, the values inside criteria, and the Python JSON layout that the prompt contract pins byte for byte.

Running without runtime endpoints

Runtimes differ in which helper endpoints they expose. The bridge uses the runtime for both rendering and tokenization by default, and each half can be moved in-process independently:

# Render locally, tokenize through the runtime
./target/release/jev-bridge \
  --base-url http://127.0.0.1:8000/v1 \
  --model Qwen/Qwen3.5-4B \
  --served-model bridge-qwen3.5-4b \
  --served-model-release-date 2026-09-22 \
  --chat-template-file qwen3.5.chat_template.jinja \
  --chat-template-context '{"bos_token": "", "eos_token": "<|im_end|>"}'

# Both halves local: no /apply-template and no /tokenize needed
./target/release/jev-bridge \
  --base-url http://127.0.0.1:8000/v1 \
  --model Qwen/Qwen3.5-4B \
  --served-model bridge-qwen3.5-4b \
  --served-model-release-date 2026-09-22 \
  --chat-template-file qwen3.5.chat_template.jinja \
  --chat-template-context '{"bos_token": "", "eos_token": "<|im_end|>"}' \
  --tokenizer-json /models/Qwen3.5-4B/tokenizer.json

Take the model's own template and tokenizer, never a hand-written or foreign pair: a different vocabulary silently scores the wrong token ids, and a different template silently changes the prompt. For a llama.cpp server both can be read from /props:

curl -s "http://127.0.0.1:8080/props?model=$MODEL" | jq -r .chat_template > template.jinja

The template context carries the variables transformers supplies besides the messages. A template that begins with {{- bos_token }} needs bos_token there, or the rendered prompt silently lacks its BOS token.

GET /health reports renders_locally and tokenizes_locally, so you can confirm what the bridge actually negotiated.

Templates written for transformers use Python string methods (split, replace, startswith, strip, …) that the Rust minijinja lacks. The bridge supplies them with CPython semantics, including treating the argument of strip/lstrip/rstrip as a character set rather than a prefix. Local rendering was checked byte for byte against llama.cpp's own /apply-template output on a real model template.

Examples

Three runnable examples live in examples/. Each reads its configuration from the environment, so no credentials are baked in:

export JEV_BRIDGE_UPSTREAM_URL=http://127.0.0.1:8080/v1
export JEV_BRIDGE_UPSTREAM_KEY=...        # only if the server requires it
cargo run --example score_rows

Transport probing

At startup the bridge tries each transport below with a real sixteen-option probe request and adopts the first one that works:

The two vLLM transports claim the distribution is restricted to the option tokens. If a server ignores those fields and returns foreign tokens, the transport is rejected rather than silently adopted — otherwise /health would report a contract the service is not actually honouring.

Unrestricted distributions need a generous top-k: on a 2B quantized model, k=20 missed four of the sixteen answer letters and k=50 covered all of them. Unrestricted transports therefore request 4 × options (floor 32, cap 128). A missing option is always an error, never a zero.

Startup contracts

Two checks run before serving; failing either exits instead of scoring wrong numbers:

1. Single-token contract — each of A–P must be exactly one token, and /detokenize must return them as ABCDEFGHIJKLMNOP.
2. Boundary contract — appending a letter to the rendered prompt must produce exactly prompt tokens + [that letter's token]. Otherwise the prompt's last token merges with the letter and the scored logit belongs to a different token than the chosen option.

CLI reference

Credentials are read from environment variables or the command line and are never written to disk.

Alignment with fastjev

• The payload uses Python's json.dumps(..., ensure_ascii=False) separator layout (, and : ); a unit test pins this against real CPython output, and the alignment run confirms it with 144 matching prompt hashes.
• Softmax matches fastjev's helper (subtract the maximum first).
• fastjev.confidence_method is one-minus-normalized-entropy, matching fastjev's System One adapter; it is not TypeSafe's private statistic.
• Template rendering is delegated to the serving runtime (/apply-template), so the prompt is exactly what that runtime feeds the model. This is also why the chat completions endpoint must not be used: there the prompt is decided by the server's template and cannot be guaranteed to match this contract.

Testing

cargo test

52 tests: 37 unit tests (wire contract, CPython JSON layout, the three response shapes, softmax stability, missing options must error, CPython-semantics template methods) plus 15 integration tests split between tests/bridge.rs (library API, including DetailedScore fields) and tests/contract.rs (HTTP surface, local rendering, local tokenization).

Known limitations

• Decisions are serialized: several questions in one request are scored one after another, and upstream calls are not batched.
• The upstream revision cannot be pinned by this process. Reproducibility depends on the upstream's own version management.
• The alignment reference is BF16 while the bridge run is Q8_0, so the 0.0053 delta is a quantization delta, not a bridge delta. An equal-precision comparison needs a BF16 upstream.
• With a local tokenizer the bridge trusts the file it is given. A foreign tokenizer.json produces plausible-looking scores from the wrong token ids, so verify_special_tokens checks the context's special tokens against the vocabulary at startup but cannot check that the vocabulary is the model's own.