Your customers are on the line again, and your voice agents are answering in its brightest voice, "Hi! How can I help you today?"
Voice agents repeat the same bad call because nothing tells them it failed. This is an open source one (Vapi + Jev + SenseLab) that learns from each call and treats the fourth call differently from the first. Clone it, add three keys, call it from your browser.
Live demo available here
Your customer is calling about the same $49 charge for the fourth time. The first bot told him it would drop off the next invoice. It did not. The second applied a credit, which finance quietly reversed eight days later. The third put him in a queue for 31 minutes and promised a callback that never came. Now he is on the line again, and the agent answering says, in its brightest voice, "Hi! How can I help you today?"
Nothing in that agent is broken. The speech is good. The policy says small billing disputes get resolved on the call, and the fourth bot is about to do exactly that, because nothing told it how the first three went. It has no way to find out that the step failed for this caller, so it cannot do anything about it. Every call is the first call, for Bruno and for the agent.
Here is the thing most voice agent tutorials skip: the written policy is the ceiling. Whatever decides inside your agent, a decision model like Jev or a prompt in front of an LLM, it does exactly what the policy says, including where the policy is wrong. Nothing changes until a human reads transcripts and edits a prompt. A bad call teaches your agent nothing.
We built a support line where a bad call does teach it, and the code is public: github.com/raia-live/jev-senselab-demos, the `voice/` folder. Clone it, paste your keys into `.env`, and you can call it from your browser in about ten minutes. This post is about what "self-improving" means in a voice agent, concretely, and how the pieces fit.
Three loops, at three speeds, all closed by outcomes from the calls themselves. Nobody edits anything by hand.
The fastest loop is one caller. When a step the agent took for Bruno did not hold (he rang back about the same thing, or he refused it on the call), the next decision for Bruno carries that fact, and the step is off the table for him, whatever it has done for everyone else. This is what "handle customers on a personal level" turns out to mean once you make it mechanical: the agent's learnings about one person change what it does for that person.
The second loop is one situation. Repeat billing callers on a consumer plan are a situation. Every decision in it gets a tally: resolving on the call worked 2 of 9 times, a specialist callback 6 of 6. The next caller in that situation gets a decision that weighs the tally against the written rule. When what works changes, the tally turns and the decision follows, a few calls later.
The slowest loop is the rules. A judge grades every finished call. When it pins two failures on the same written rule, a repair agent drafts a rewrite, replays it against the calls that failed, and if it would have changed them, canaries it: half the live calls read the rewrite, half keep the old rule, and the arms are compared on outcomes. A rewrite that does worse never ships.
That stack, outcome to learning to the next decision, at every level from one caller to the rulebook, is what we call recursive self-improvement, RSI for short. SenseLab is the engine that runs it. Jev is what decides. The rest of this post is how they fit, and why the agent cannot do this with either one alone.
John is the support agent for a made-up company called Northwind Cloud. Vapi runs the call, phone or browser audio, GPT-Live for the speech. John has a character and one rule: he never decides what happens about a caller's problem. At the moment of the call where something has to be done (sort it out himself, hand off to a human, book a specialist callback, apply a credit), he asks a backend, and the backend asks Jev.
Jev is TypeSafe's decision model. You hand it a state (the facts of the case, your written policy, whatever else you want weighed) and the actions on the menu, and it hands back an action with a probability for each, in about 200 ms. It does not chat. It decides, from exactly what you put in the state and nothing else, which is a feature: the decision is pinned and auditable, and what Jev gets told is the whole lever.
SenseLab holds that lever. Before the call connects, what the agent has learned about the caller goes into John's prompt, so he opens with "Hi Bruno, it's John at Northwind Cloud. I've got your file open here. I can see you've been in touch three times this past month, most recently five days ago, so no need to start from the beginning. What's going on today?" Ask him "what do you know about me?" and he tells you, in order: the duplicate, the reversed credit, the callback that never came.
When John asks what to do, Jev's state carries the written policy plus two things from SenseLab. The situation's learnings: the tally above, and any correction the repair loop has shipped for this rule. And the caller's own record: which steps were taken for them and how each came out, in their words, with one binding line naming anything that did not hold ("Not to be taken again for this caller: handled it on the call"). Jev weighs all of it and picks. John puts the pick in his own words and, when the caller asks "why would it work this time?", answers from what actually happened the last three times.
When the call ends, what came of the decision goes back to SenseLab, and all three loops move.
Every call is decided twice. Jev alone, on the written policy. And Jev with SenseLab. The caller gets the second; the page shows both columns side by side, which is the whole demo in one glance.
Here is the two-call arc we use. Call as Priya, a consumer who wants to cancel. The written policy says cancellations go to the retention team, so John says a colleague will call her back within the hour. Both columns agree. Hang up.
Call again a minute later: "It's Priya again. Nobody called me. I still want to cancel." Nothing is marked by hand. Ringing back about the same thing is the outcome: the server seals the first call's step as not having worked, writes it on her record, and tells John to own it before he says anything else. Then Jev decides again, now with "Not to be taken again for this caller: transferred the call to an agent" in front of it, and comes back with something else: the retention offer, handled on the call, or a specialist callback. The Jev-alone column still says transfer. That column is the fourth bot from Bruno's story, and it would be the fourth bot for every caller, forever.
If she refuses the step John just gave her ("no, I don't want to be transferred, cancel it now"), that is an outcome too. It goes on her record in her words, John asks Jev again in the same turn with the refusal as the new fact, and the step he was about to repeat is off the table.
Give the line thirty simulated calls first (command below) and call as Bruno. The situation loop has done its work by then: the tally for repeat billing callers says what the written rule gets wrong, the right column books the specialist callback, and the left column resolves it on the call or hands him to a human, again.
Three accounts, one tunnel.
- A Vapi account. The public key starts browser calls, the private key lets the server create the assistant. The default stack (GPT-Live) needs nothing else.
- A TypeSafe key for Jev (`TYPESAFE_API_KEY`).
- A SenseLab key (`AMFS_API_KEY`). Sign up, create a key, paste it in. The dashboard there shows every trace the agent produces while you call, and the drafts and canaries once the repair loop gets going.
- `ngrok` (or any tunnel), so Vapi can reach your laptop.
git clone https://github.com/raia-live/jev-senselab-demos && cd jev-senselab-demos
cp .env.example .env # fill in TYPESAFE_API_KEY, AMFS_API_KEY, VAPI_PUBLIC_KEY, VAPI_PRIVATE_KEY
uv sync --extra voice
ngrok http 8787 # terminal 1: copy the https URL it prints
VOICE_PUBLIC_URL=https://<that-url> uv run uvicorn voice.server:app --port 8787 # terminal 2
open http://127.0.0.1:8787
On startup the server seeds the written policy into SenseLab if it is not there, creates or updates the Vapi assistant pointed at your tunnel, and starts polling its own public URL so a dead tunnel shows up as a red bar on the page instead of as an agent that talks and never decides. Pick a caller, press Call, talk.
A line that has taken no calls has learned nothing yet, so give it some history:
uv run python -m voice.simulate --calls 30 --no-change
Thirty simulated calls take about five minutes and leave SenseLab with a tally for every situation, including the one the written policy gets wrong. Then call as Bruno and watch the two columns disagree.
No microphone? The page has a text box that runs the same decision without audio, and `voice/testcalls.py` runs scripted callers through Vapi Simulations with a judge grading the transcript.
This is the part worth understanding if you are going to build on it.
Jev answers one question very well: given this state, what do we do. It is stateless between calls and it knows nothing it was not handed. So personalising a decision is the job of building the state, per call, from what has happened to this caller and to callers like them, with the outcomes attached, kept across weeks and across whichever server happens to be answering. And improving the decisions is the job of changing what goes into that state as outcomes come in, without a human in the loop for every change. Jev does neither; it is not supposed to. SenseLab does both, and that is the whole division of labour.
Concretely, SenseLab touches a call in four places.
When the call connects, the server reads the caller's record for the prompt and opens a SenseLab run for each situation the caller could be in. The account facts give us tier and prior contacts; only the intent is unknown. Each run returns the compiled learnings for its situation, so by the time the caller has said why they are calling, the guidance is waiting and the decision costs Jev's 200 ms.
At the decision, the state Jev reads is the case, the written policy, the situation's learnings and corrections, and the caller's own record. The same call is decided a second time without the last two, for the page.
At hang-up, the decision is sealed with its outcome. The lesson is written to the situation, with the outcome credited to the entries Jev actually relied on, so a rule that said "this will work" and was wrong loses standing and a rule that happened to share the prompt does not. The caller's record gets a line about this call. Details the caller gave and John read back (an email, a number) go on it too, even if the model forgot to call the tool; the server reads the transcript and files them.
In the background, the repair loop works on the rules themselves, as described above. One draft did get held back on our runs: "escalate repeat callers to a human agent" sounded right after two failures, the canary showed those callers doing worse (half an hour of queue time each), and it never shipped. A rule pulled from a similarity index would have gone to every caller. Nobody edits `business.toml` for any of this.
You can try the obvious workarounds, and we did, in the batch harness that lives next to the voice demo.
Paste the call history into the state and it grows without bound; Jev refused the call at case 276. Keep a lookup store of past cases and you get most of the way on one worker, then lose it the moment you run three workers behind a queue, because each store sees a third of the evidence (the lookup arm lost 2.3x the dollars with three workers; SenseLab made the same six misses as the single agent and no others). And a lookup store does not know which written rule a bad outcome belongs to, so it cannot rewrite one, and it cannot tell you whether a rewrite made things worse before every caller has read it.
What Jev gets from SenseLab does not grow with the call count. A tally is one line whether it counts 9 outcomes or 9,000. The slot is capped at 3,200 characters and is the same size at call 120 as at 12,000. That is what "customise at scale" comes down to: one agent, whichever worker is deciding, with a per-caller record and a per-situation tally that stay the size of a paragraph, and a rulebook that rewrites itself under a canary.
The loop does not care what is inside the decision. It wraps a decision point:
python
run = Run(client.memory(AGENT_ID)) # one run per decision
guidance = run.begin(request, entity_path=ENTITY,
candidate_actions=ACTIONS, situation=situation)
decision = decide(state | {"guidance": guidance.text}) # Jev here; could be an LLM, a classifier, a rules table
run.on_tool_result("decide", {"action": decision.action}, action_key=f"decide:{decision.action}")
# later, when the world reports
run.learn(situation, f"decide:{decision.action}", worked, note)
run.complete(severity, outcome_ref=call_id, causal_entry_keys=cited)
Open a run with the situation and the candidates, decide with whatever decides, record the action, seal with the outcome when it arrives. Jev is a good fit because it is a pure decision function with probabilities and a pinned version, so every change over time has a cause you can look up (an outcome, a judge verdict, a canary result) and reverse. An LLM behind a prompt works too; it just makes the "what changed" question harder to answer.
The voice side is the same story. Vapi runs John here and reaches the server over plain HTTP: a tool call when a decision is due, a webhook at hang-up. Any voice stack that can hit a URL when a decision is due and another when the call ends can drive the same loop; the Vapi-specific code is the assistant definition (voice/assistant.py), the webhook handling (voice/server.py), the transcript and recording formats voice/agent.py reads at hang-up, and the test tooling that places calls through Vapi. The decide-and-seal loop does not depend on any of it. templates/your_decision/decide.py in the repo is the whole integration in one file, with no phone at all.
Everything about Northwind is in one file, voice/business.toml. Copy it and point the server at yours:
cp voice/business.toml my-business.toml
VOICE_BUSINESS=my-business.toml VOICE_PUBLIC_URL=https://<tunnel> uv run uvicorn voice.server:app --port 8787
The pieces you will touch:
[company]
name = "Northwind Cloud"
agent_name = "John"
greeting = "Hi, thanks for calling Northwind Cloud, this is John. What can I help you with?"
[[intents]] # what people call about
id = "billing_dispute"
label = "a charge they dispute or do not recognise"
needs = ["amount_usd"]
[[actions]] # what the agent can do at the decision point
id = "schedule_callback"
jev = "Book a callback from a specialist within one business day and end the call."
words = "booked a specialist callback"
then = "A specialist calls them back within one business day. Confirm the number on file."
[[policy]] # your written rules; Jev follows them until the outcomes say one stopped working
key = "policy-billing-small"
text = "Billing questions and disputes under $200: resolve in the bot."
action = "resolve_in_bot"
situations = ["billing_dispute/*/*/small"]
[outcomes]
mode = "reported" # nothing is assumed at hang-up; the world (or the caller) reports
jev is what Jev sees as the option; words is how the step is written on the caller's record afterwards; then is what the agent gets across once the step is chosen, written as facts for him to put in his own words. Keep the conditionals out of then where you can: the speaker will read "if the file explains it" out loud, and we have the transcript. action and situations on a policy row are what let the judge charge that rule with the outcome of following it; a rule without them is never blamed and never rewritten, which means it never improves either.
For a real deployment, set outcomes.mode = "reported". Then nothing is assumed when a call ends. The calls themselves close most decisions (a ring-back, a refusal, "the credit showed up"), and whatever else knows the outcome, your CRM or a survey, posts it to /api/outcome. Your customer lookup goes in one function, lookup(number) in voice/phonebook.py: return the account facts for a phone number and everything learned under that id follows.
Most of the rules in John's prompts came out of test calls, and they are the fun part of the README. A sample:
John told a caller nobody had matched that their charge was "a setup fee", from a record that did not exist. Each action now has a second wording for a stranger, and the server picks it.
Asked "what email do you have for me?", he said "let me check" and went silent for twenty seconds. He has no tool to check with. Not on file now means say so and ask.
He read the reasoner's instructions out loud: "the file either explains it or it doesn't." The reasoner now writes instructions, never lines, and the step text for a known caller has no `if` in it.
He took Bruno's email, read it back, said "I've got it", and never filed it. The hang-up net above exists because of that call.
The tunnel died mid-session and nothing complained. John kept talking, nothing was decided, and the hang-up never arrived. The server now pings itself through the public URL every few seconds and the page says when it cannot get through.
Every one of these has a line in the prompt or a fix in the code, and a test. GPT-Live is a beta and its speaker is not deterministic, so run `voice/testcalls.py` after you change a prompt; a prompt that passed once is not a prompt that passes.
The repo is raia-live/jev-senselab-demos. The voice agent is in voice/, with its own README covering the voice stacks, the phone number setup, the test harness and the repair loop in detail. Next to it are four batch scenarios (lead routing, invoice exceptions, marketing, moderation) that run the same loop without a phone, with the numbers above and a replayable results folder, and templates/your_decision/ for wrapping a decision you already have.
Start with the two-call arc. Call as Priya, hang up, call again, and watch the right column change its mind while the left one books the same transfer again. Then swap in your business file, take thirty calls, and see what your written policy gets wrong. That is the moment the agent starts getting better without you.
Free tier. No credit card. Connect in minutes.
