Jev vs LLMs: Fast Structured Decisions Tested

Jev vs LLMs, answered briefly: Jev is a decision model that returns a choice plus a probability for every option, not text. Independent testing found it fast, cheap, and consistent, but it cannot read base64- or hex-encoded input without help and trails the best frontier model by about five points on a hard 77-way classification task.

Jev vs LLMs: what the comparison actually measures

Jev vs LLMs is a comparison of task fit, not raw model quality. Jev is a decision model from TypeSafe AI that answers one typed question at a time, while a large language model generates text that your code must then parse into a usable value.

The transcript that this article is built from comes from Tech With Tim, a developer-focused YouTube channel, and was published on 8 October 2026. The presenter reports four tests run over a few weeks, all through the OpenRouter gateway that routes requests to many providers under one API key. Treat those results as first-hand testing, not as an independent benchmark.

The four tests were: routing 50 generated support tickets in one batch, screening shell commands for an AI coding agent at 20 commands and then 315 encoded variants, a head-to-head on the same 50 tickets across three models, and a 1,000-message run over the public Banking77 dataset. The last one carries the most weight because Banking77 has 77 possible intents rather than three, which makes the task much harder to label correctly.

One term matters throughout. A confidence score is the model's own probability estimate for the answer it chose, returned alongside the option probabilities. Every accuracy claim below should be read next to the cost and latency that came with it under the same test conditions.

How the Jev decision API works and what it returns

The Jev decision API accepts a state plus one or more typed questions and returns structured answers rather than text. The request shape is deliberately narrow: there are three question types, so the possible outputs are enumerable in advance.

• Choice returns one option from a list you define, plus a probability for every option.

• Yes/no returns a single probability for the binary question and nothing else.

• Score places an input on an ordered scale you supply, such as low, medium, or high urgency.

In the transcript's example, the state holds a support ticket and one question of type choice asks which team should handle it, with billing, technical, and sales as the criteria. The response contains the chosen option, a confidence value, and the three probabilities. There is no prose, no markdown to Stripe, and no format drift between calls.

Instructions can reference named parts of the state inside backticks, so the question points at an exact field rather than describing it in words. The presenter openly says he is reading TypeSafe's own documentation for these mechanics, so the API behavior is vendor-described rather than independently specified. Anyone rebuilding this should read the current docs, because the request fields are the part most likely to change.

Vendor performance and pricing claims, labeled as vendor claims

TypeSafe's published claims about Jev are large and only the vendor has verified them. The presenter states on camera that he could not verify the numbers and that his own tests would differ, which is the correct way to report them.

Per the transcript, the company homepage claims Jev is 193.6 times faster and 444.6 times cheaper than LLMs on decision-style tasks. Pricing is quoted at 4.22 cents per million input tokens, phrased as $42 per billion, with output tokens free. Through OpenRouter the context window is stated as 32,000 tokens.

Those figures carry a vendor evidence label. They are not reproduced here as fact, and they should not be reused in a cost model without a fresh check against current published pricing. The independent value of the article sits in the next sections: what a developer actually measured when running the same tasks against Jev, a cheap LLM, and an expensive one.

Batch routing results: 50 tickets in one request

Batching is where the Jev cost advantage is largest, because one request can carry many questions instead of one call per item. In the transcript's first test, 50 support tickets were loaded into a single state and one choice question was created per ticket.

The presenter reports that all 50 tickets were routed in about 1.5 seconds for roughly a third of a tenth of a cent. He applied an application-side rule that any ticket with confidence below 0.85 goes to a human instead of being auto-routed, and 10 of the 50 tickets were flagged that way.

The flagged tickets were the borderline ones, including a follow-up email that could plausibly belong to any team and drew a confidence of about 0.15. The single ticket Jev got wrong overall also carried low confidence, so the threshold would have caught it before it reached a queue. That is the practical value of the confidence field: it is a routing signal for a code path, not a claim about correctness.

Guardrail test: screening shell commands before an agent runs them

Jev handled plain shell commands well and degraded badly on encoded ones until the encoding was decoded before the decision call. This section covers that two-stage test and the fix that moved the score.

The setup asks three or four yes/no questions about each command: does it delete something, does it override something, does it change production, and does it send data off the machine. If any answer comes back above 50 percent, the command stops and waits for human approval.

On 20 commands, including normal status checks and obviously risky ones, all 20 were classified correctly in under 2 seconds for roughly three-hundredths of a cent. The presenter then built 63 base commands and rewrote each five ways, giving about 315 variants.

The variants matter because base64 and hex encodings hide intent from a model that reads tokens. On normally written commands, 100 percent of risky ones were blocked. On encoded commands, about one in ten risky commands slipped through and almost a third of harmless ones were blocked, for an overall 91 percent. The presenter attributes the failure to Jev not being able to read the encoded payload.

The recommended fix came from TypeSafe's own documentation: do the mechanical work in ordinary code first. A small function detects the base64 or hex payload, decodes it, and appends the decoded command to the state before the decision call. Reported accuracy moved from 91 percent to 98.4 percent with that single change.

Where Jev beats LLMs and where it falls short

Jev wins on cost, latency, and repeatability, and loses on hard accuracy and on anything requiring reasoning over unreadable input. The comparison below uses the presenter's own figures from the 315-command guardrail test and the 1,000-message Banking77 test, so it reflects one developer's setup rather than a controlled benchmark.

Model	Guardrail accuracy (315 commands)	Banking77 intent accuracy	Median latency (Banking77)	Cost per 1M messages	Same answer on rerun
Jev with decoder	98.4%	80.0%	443 ms	~$71	98.9%
Gemini 3.1 Flash Lite	94.3%	78.5%	just over 1 s	~$150	96.3%
GPT 5.5	92.7%	84.9%	2.8 s	~$4,600	96.3%

Two caveats belong with those numbers. First, the guardrail accuracy for GPT 5.5 reflects stricter blocking, not worse judgment: it flagged safe commands such as git stash, pip install, and a feature-branch push, which the presenter counted as errors. Second, he describes the command test itself as not the best benchmark in the world, and the guardrail column has no confidence interval attached.

On the 50-ticket head-to-head, Jev is reported at 49 out of 50 correct, Gemini at 48, and GPT 5.5 at 41. Batched Jev came in roughly five times cheaper than Gemini and about 79 times cheaper than GPT 5.5 on that same set. Accuracy on hand-labeled tickets is arguable, so the cost and latency gaps are the durable part of that result.

One difference is not a matter of degree. Jev returned identical verdicts on both runs of the command test, while the two LLMs showed variance. The presenter measured rerun agreement of 98.9 percent for Jev against 96.3 percent for both LLMs across 1,000 Banking77 messages. Repeatability matters when a decision feeds an automated workflow.

Does the confidence score hold up on a hard 77-intent task?

The confidence score tracked accuracy in the test, which makes it usable as an escalation trigger. The presenter grouped all 1,000 answers by the confidence Jev reported.

When Jev reported 0.95 or higher, which covered about 60 percent of messages, it was correct 95.8 percent of the time, and accuracy declined as confidence declined. GPT 5.5 struggled on the same messages where Jev was unsure, which suggests those items are genuinely ambiguous rather than evidence of a weak model. The presenter also notes that some Banking77 labels are arguable in the first place.

This calibration evidence is the strongest single argument for the escalation pattern. A threshold on the confidence field converts an uncertain classification into a second call to a stronger model, and only the uncertain fraction pays that price.

The margins on the point estimate are worth keeping in view. Jev and Gemini overlap under 95 percent confidence intervals at this sample size, so calling one better than the other on Banking77 would overstate what 1,000 messages can show.

Confidence escalation: the cost and accuracy trade, step by step

Escalating low-confidence Jev answers to a frontier model recovers most of the accuracy gap for a fraction of the cost. The presenter ran this as a quick experiment on the same 1,000 messages and measured the result at each threshold.

• Use Jev alone: 80.0 percent accuracy, with total test cost at 7 cents.

• Escalate anything below 0.7 confidence to GPT 5.5: 165 messages escalated, accuracy 83.7 percent, cost $1.20.

• Escalate anything below 0.95: 405 messages escalated, accuracy 84.6 percent, cost $2.58.

• Send everything to GPT 5.5: 84.9 percent accuracy, cost $4.65.

The strongest result is the first escalation step. It closes roughly three-quarters of the accuracy gap between Jev and GPT 5.5 while escalating about one message in six and keeping total test cost in the low single-digit dollars. Going all the way to a 0.95 threshold adds 240 more escalations for less than one additional point.

This is the pattern the presenter says he will adopt for most of his workflows: handle the decision with Jev, act when confidence is high, and route only the uncertain cases to a larger model.

Frequently asked questions about Jev vs LLMs

• What is Jev and who makes it? Jev is a decision model from TypeSafe AI that answers choice, yes/no, and score questions instead of generating text. It is used through OpenRouter's API rather than a standalone SDK. The presenter describes it as being in beta during the testing period.

• Is Jev cheaper than an LLM for classification? Yes in this test, by a wide margin. The presenter measured about $71 per million messages for Jev against roughly $150 for Gemini 3.1 Flash Lite and about $4,600 for GPT 5.5. Those are one user's numbers, calculated from per-request costs read off the API response, so treat them as an estimate.

• Can an LLM replace Jev for guardrail checks? The reported guardrail accuracy was 98.4 percent for Jev with a decoder step against 94.3 percent for Gemini 3.1 Flash Lite and 92.7 percent for GPT 5.5. The GPT figure is depressed by strict blocking of safe commands, so read that column as a behavior difference as much as an accuracy difference.

• What are Jev's main limitations? It cannot generate text, and it could not read base64- or hex-encoded commands without a decode step adding the plaintext to the state. The presenter also flags the 32,000-token context window as a ceiling for long documents and recommends caution for critical decisions while the model is in beta.

• Should you trust the confidence score? In this test, accuracy declined as reported confidence declined, and answers at 0.95 confidence or higher were correct about 95.8 percent of the time across roughly 60 percent of messages. That pattern makes the score usable as an escalation trigger, though it was measured on a single dataset.

The decision-model pattern this comparison points to

The practical split is decided by whether a task needs generated text or one typed value. Ticket routing, policy checks, and command screening need a value, and a decision model returns that value without a parsing step or a format-drift failure mode.

The same boundary applies to the local and cloud question. The tests here all ran through OpenRouter, a hosted gateway, so every request left the machine and was billed per call. Nothing in this test demonstrates offline operation, and a self-hosted deployment would be a separate project with its own hardware and privacy properties.

For anyone building on this, the honest summary is narrow. Jev handled high-volume repetitive decisions quickly and consistently in one developer's tests, matched a cheap LLM on routing accuracy, and beat both LLMs on the command test once encoded input was decoded first. It trailed a frontier model by about five points on a hard 77-way classification task, and it cannot read what it cannot decode.

Turning this kind of testing into a written article

The useful part of a test like this is the setup, the numbers, and the caveats that a reader needs before trusting any of it. If you have that kind of walkthrough sitting inside a YouTube video, including transcripts you would otherwise rewrite by hand, it can be transcribed into a structured article with headnotes, tables, and source links intact.

With Skala Blog, you paste a YouTube URL and the video is transcribed into a draft article you can review and edit. The same comparison you ran on camera becomes something a reader can search, skim, and cite, and you keep the numbers and conditions you actually tested.

Source video