jev

A terminal REPL for shaping TypeSafe AI System One requests before you write any code: send a state plus named, typed questions and read the typed answers back.

cargo install jev-repl
jev            # with TYPESAFE_API_KEY for live answers; without it, answers are simulated

Inside

:preset triage                                   # a ready-made session to poke at
:state The payout failed again, third time.      # bare text works too
:noul is_urgent The message conveys urgency | yes: A deadline | no: Routine
:choice department Which team | billing=Payments | technical=Bugs
:score frustration How frustrated | Calm | Annoyed | Furious
<Enter>                                          # send; answers come back with their distributions

• :lesson walks an eleven-step track from "what is a noul" to what a call costs.

• :sketch (Ctrl-K) opens the whole request as one page of text, with the question types read off the punctuation, a gutter that says what each line became, and a live preview of the JSON, simulated answers, the same request as Rust, or what a call would cost:

  The payout failed again, third time this month.
  ---
  is_urgent? The message conveys urgency
    yes: A deadline or a threat to leave
  department: Which team should handle this
    billing = Payment or subscription issues
    technical = Bugs or integration problems
  frustration: How frustrated the customer appears
    Calm < Frustrated but civil < Very angry
  

• :build opens a form for one question; :json shows the exact request body, :last the raw response, :rust the session as a program against typesafe-ai-sdk.

• :cost says what a call is about to cost, per question and on both sides of the wire — a choice over eight labels comes back with eight probabilities, a score echoes its whole legend:

                          in   out
  department   choice     74    58
  frustration  score      60    93
  is_urgent    noul       32    19
  state                   40     ·
  envelope                22    16
  total                  228   186   414 tokens per call
    $0.000232 per call   ·   $0.2316 per 1,000 calls
    at $0.20/$1.00 per Mtok
  

  Rates are yours to supply, because nothing here knows what a model charges: :cost 0.20/1.00 is dollars per million tokens, input then output, and JEV_PRICE=0.20/1.00 sets the same at startup. Without them the table counts tokens and stops there. Tokens are estimated from the body — roughly four characters a token — so they are a shape, not an invoice; a live answer carries the counted usage, and the REPL prices that instead.

• :save triage.jev / :open triage.jev keep sessions as sketch pages; other paths use the request JSON.

Outside

A session shaped in the REPL and saved with :save is something a script can run. With a subcommand jev never opens a terminal: it reads a page (or stdin), prints one thing, and says with its exit status whether it worked — 0 when it did, 1 when the call or the file did not, 2 when the command line did not parse.

jev run triage.jev                       # send it; the answers, with their distributions
jev run triage.jev --json | jq .answers  # the raw response body instead
jev eval triage.jev --cases cases.jsonl  # score the rubric over states you have labelled
jev json triage.jev                      # the exact request body it would POST
jev cost triage.jev --price 0.20/1.00    # the token table, priced
jev rust triage.jev                      # the session as a program against the SDK
jev check triage.jev                     # parse only: every problem, with line numbers

The file is a .jev page or a request body, and which one it is comes from the text rather than the name, so a body piped back in works the same: jev json page.jev | jev cost. -, or no file at all, reads stdin.

jev run triage.jev --state "$(cat ticket.txt)" --json | jq '.answers.is_urgent.noul'

--state <text> sets or replaces the state, --model <name> picks the model, --threshold <0-1> says what counts as a yes for a noul, --price <in>/<out> prices the table, --timeout <seconds> bounds a live attempt, and --mock stays offline even with a key set. Without a key jev run simulates the answers and says so on stderr, so the stdout of a mock run is still the answer page.

Scoring a rubric

One call tells you what the model said about one ticket. It does not tell you where to put the threshold, or how sure a choice has to be before a script may act on it — and this README leaves both calls to you. jev eval is how you make them with data: it runs a saved page over states you have already labelled and reports what the rubric got right.

jev eval triage.jev --cases cases.jsonl --price 0.20/1.00
jev eval triage.jev --cases cases.jsonl --json | jq .questions.is_urgent.best

The cases are JSON Lines, one labelled state per line. state is what to judge — a string or any JSON — and expect names the questions on the page it is labelled for; questions it leaves out are still asked and simply not scored. An id is optional and shows up in the report.

{"id": "t-001", "state": "Stripe has been failing for 3 days, I'm losing sales", "expect": {"is_urgent": true, "department": "technical", "frustration": 2}}
{"state": {"subject": "Invoice question", "body": "Can I get a copy of last month's invoice?"}, "expect": {"department": "billing", "is_urgent": false}}

An expectation is written the way its question is answered: a noul takes true or false, a choice takes one of its option labels, and a score takes a level index or the text of one of its levels. A bad line stops the run before anything is sent, named by its line in the file.

  is_urgent    noul    40 cases · Brier 0.11
    threshold   acc   prec   rec    f1
    0.40        0.82  0.74   0.94   0.83
    0.50 *      0.85  0.80   0.89   0.84
    0.60        0.88  0.86   0.89   0.87
    best f1 at 0.60

  department   choice  40 cases · accuracy 0.78
    confidence ≥   coverage  accuracy
    0.00           1.00      0.78
    0.60           0.55      0.95
    confusion, rows expected, columns predicted
                billing  technical  sales
    billing     12       2          0

  40 cases · 40 answered · 0 errors
  4812 in / 3120 out tokens · $0.0041

That is the whole point of the table: the sweep says what a threshold buys, and the gate says what a confidence bar buys — 0.95 accuracy over 55% of the tickets, with the rest going to a person.

--cases <file> is the only new flag that is required; - reads them from stdin, which the page cannot also do. --concurrency <n> sends that many at a time (default 4), --cache <dir> keeps each response so a second run sends nothing, --max-cost <dollars> refuses a run whose estimate is above it (rates required), and --min-accuracy <0-1> exits 1 when a question scores below it. A live run prints its estimate on stderr before sending anything. --state does not apply: the cases carry the states. Exit status is 0 when every case answered and every bar was met, 1 when a case errored or a bar was missed, 2 when the command line did not parse.

Without TYPESAFE_API_KEY it starts in mock mode: answers are simulated locally (deterministic, not predictive) so the shapes can be learned offline. :key <api-key> switches to live calls.

Unofficial. Not affiliated with TypeSafe AI. MIT licensed; source and issues at aoprisan/typesafe-ai-rust-sdk.