Jev has no idea how its last decision went. Every call starts from the state you send and the policy you wrote.
TypeSafe shipped Jev on 15 September and it blew up: nearly 13% of Vercel's paid AI Gateway teams called it inside 24 hours, over 100,000 people joined the Discord in a week, about 2,000 GitHub projects were using it by then, and OpenAI had a copy out within three weeks. Fair enough. A model that answers a yes/no or picks from a list in under half a second, with a real probability, for $0.042 per million input tokens, is a genuinely new tool.
What none of the launch threads say: Jev has no idea how its last decision went. Every call starts from the state you send and the policy you wrote. If the policy is missing a rule, Jev gets that case wrong today and in March, confidently, every time. We ran the numbers. On the same leads with the same pinned model and policy, Jev alone got 59 of 120 decisions wrong. Jev with a recursive self-improvement loop around it got 6. Nothing about Jev changed. This page explains the pieces, then shows the receipts.
An inbound lead desk, the kind every B2B company has. 120 leads arrive one at a time (400 in the longer runs). Each is a short message plus the fields a form captures: company size, country, email, what they asked for. Jev reads the lead and the written policy and picks one of five actions:
The leads are generated from a fixed seed, so anyone who clones the repo gets the same 120 in the same order. This is the whole policy, and every arm gets it word for word:
Then we put four things into the stream that a written policy cannot carry. Every real deployment has all four; we just made them measurable by knowing the truth behind each lead.
Here is lead 0, exactly as it reached Jev, and the review that came back on it. Note the review says what went wrong and never says "partner".
The three arms. Same model (jev-1.13.0), same leads, same policy, same reviews. The only thing that changes between rows is what Jev is handed alongside the lead and the policy.
- [M1] seen-mid-market-emea--route-to-ae (decide:route_to_ae): worked 0 of 1 time. Recent reviews: [did not work] Prospect told the AE they purchase through a local vendor with in-country invoicing; closed-lost.
- Tried on this situation: decide:route_to_ae 0/1Recommendation: none yet. Nothing has worked here and nothing has failed twice: followthe written policy unless a failure review itself says what should have been done.
No recommendation yet; the note says follow the policy unless the failure review says what should have been done. This one did, and Jev picked partner at 0.81.
- [M1] seen-mid-market-na--route-to-ae (decide:route_to_ae): worked 5 of 8 times; the last 3 did not. Recent reviews: [did not work] Prospect bought on the pricing page before the AE called back; 5 AE hours logged. [did not work] Prospect said the new plan was all they needed and bought online; the AE cycle added a week and no value. [...]
- Tried on this situation: decide:route_to_ae 5/8, lost last 3Recommendation: explore. decide:route_to_ae won 5/8 on tasks like this but lost itslast 3 [...] work through the 4 untried without repeating any
Three straight failures on an action that used to work is the signal for a change in the business. Jev took self-serve at 0.92 and the run never missed that situation again.
The lookup arm, for the record, tracked SenseLab closely on this short run: it missed lead 0 and lead 13 on the EMEA rule (SenseLab missed only lead 0), the same two personal-email leads, and the same three leads after the change. 7 wrong and $12,900 against 6 and $10,400. One decision apart. The gap opens when the conditions get realistic, which is what the rest of this page is about.
Where it gets good: Jev's probability is what makes the loop safe to act on (threshold above, escalate below), and the loop's outcome record is what keeps that probability honest once the world moves. One is a fast, cheap decision with no memory. The other is memory with no opinion. Together you get a decision that is right about last month as well as the policy, at the same latency. In Jev calls that is about two cents more per 400 decisions; SenseLab's own bill for the whole 400-lead run, every arm included, was $4.26.
A written policy is a snapshot of what someone knew the day they wrote it. Production is not. Six things happen, and each one grows with traffic and with the number of instances you run.
Jev makes the decision. Everything here happens around it, in a run SenseLab opens per decision and seals when the outcome arrives.
Each row is one thing a Jev deployment can do with the loop that it cannot do alone, and the number from the committed runs that shows it. A lookup of the 8 most similar past cases is in every comparison because it is the strongest alternative we could build, and it is close at small scale. Charts and the full table follow.
The business changes at lead 200 and the leads are reworded from 300. Jev alone pays for the unwritten rule on every lead and never notices the change. Both learners pay once for the rule and three times for the change (the three failures it takes to be sure it is a change and not a bad review); the lookup pays once more at lead 241 because its eight nearest cases were still mostly the old, confirmed outcomes.
The 120-lead stream has things in it that a written policy cannot carry. Jev alone gets every one of those leads wrong and gets the policy-covered leads right, which is the whole point: the model is fine, the missing feedback loop is the problem.
You keep your Jev call. Around it, one session per decision, five lines. templates/your_decision/decide.py in the repo is the whole thing in one file: swap in your actions and the state you already send Jev, then say how you find out whether a decision worked.
Jev was never the problem. On the 63 leads the policy covered, it made 2 mistakes in 120 and answered in about 150 ms. The 57 other misses were all the policy's fault: a rule nobody wrote down, a rule that stopped being true, and no way for either fact to reach the model. A written policy is a photo of the business on the day it was written, and Jev will apply that photo at 0.99 forever.
The loop fixes that without touching the model. It hands Jev a short, capped note on the situation in front of it, written from how that situation went before, and it keeps the note honest by crediting the lines Jev relied on when the review comes in. One failure with a review that explains itself is enough to learn a rule. Three straight failures on something that used to work is enough to notice the business moved. A wrong review costs $200 instead of flipping a rule. And because the note is a tally, it is the same 3,200 characters at lead 400 as at lead 30, where pasting the history in got Jev to refuse the call.
The bill for that, on 400 decisions, was two cents more in Jev calls and $4.26 of SenseLab. The saving on our made-up prices was $433,700. Pick your own prices; the ratio is the point.
uv run python -m core.runner --replay results-sample.env, and run it fresh.uv run python -m core.runner --scenario lead_routing --quicktemplates/your_decision/decide.py, put in your actions and the state you already send Jev, and say how you find out whether a decision worked.voice/, call it from the page, and call again as the same person.
Free tier. No credit card. Connect in minutes.
