Why Agents Need a Cognitive Layer (And How to Build On Top of One)
We can get agents to use tools, follow instructions, coordinate through chains and graphs. What we can't easily do is get them to collectively improve at a task over time.
Today most people run coding agents on a laptop, or on one long-lived dev box the agent has been slowly redecorating for three weeks. Those are pets.
We're doing pets and cattle again. Nobody seems to have noticed.
First time it was servers. You had a box called zeus, up for four years, that nobody dared reboot, whose /etc was a museum of everyone who'd ever SSH'd in. Then containers made machines disposable and the industry rearranged itself around that.
Agent environments are next. Today most people run coding agents on a laptop, or on one long-lived dev box the agent has been slowly redecorating for three weeks. Those are pets. They don't survive an agent deciding rm -rf is a reasonable next step, and they really don't survive wanting eight of them.
So you give each task its own microVM (I used Sprites from Fly.io): one per PR, one per task, one per attempt. The agent wrecks it, you delete it, nobody files a ticket. Two years from now I think this is just how you run agents, as boring as containers are today.
Then the interesting question shows up.
Pets make you ask how to keep a machine healthy. Cattle deletes that question and hands you a better one:
Is environment 500 any better at its job than environment 1?
For a web server, no, and identical is the entire point. For an agent it should be, because the run learned something. That the suite needs TZ=UTC. That staging Postgres caps connections at 20 while the pool defaults to 50. That their pagination header lies, so you follow next until it 404s. That the fixture only works if you seed before migrating.
None of that is in your repo. Your code is in git and the agent can read it fine. This other stuff lives in the head of whoever last fought with it, a Slack thread from March, or a guy named Dave who left in April. Dave was great. Dave is gone.
And the model won't save you here. It's frozen at training time, the context window resets between runs, and the only path from "we learned something" to "the agent behaves differently" runs through a human editing a prompt. That isn't the agent learning. That's you learning, and then doing data entry.
So agent 500 makes agent 1's mistakes. Your fleet is perfectly reproducible and structurally incapable of getting better at anything, which is a strange thing to have built on purpose.
What I want is boring: spin up as many isolated environments as I like, and have each one start from what the last ones worked out. The environment is cattle. The learning isn't. And the learning is mine, not something parked in a vendor's harness waiting to evaporate the next time I switch models.
Everyone says this. I said this. It deserves an actual answer.
Sprites have a durable filesystem, and that's a real feature, but a mount gets you shared files. Shared knowledge is a different thing, and the gap between them is all of the work.
Point eight agents at one directory and you now own a concurrency story, a merge strategy, the format, which of two contradictory retry-budget notes is current, which 12 of 200 files fit in the context window, and whether January's note survived March's rewrite. Plus attribution, because "who told you that" has no good answer when the answer is "a file."
Then there's the self-own. You made your environments cattle, then gave every one of them a write dependency on a single mutable stateful thing. That's a pet. You built a pet. It's now the most important machine you operate and nobody's paging on it.
And files don't rank. Context is the budget you're actually managing, and a mount gives you a pile.
Checkpoints come up next and get the same answer. A checkpoint restores that box, which is great when a long job dies and you want it back. It doesn't make the next box smarter, and in a fan-out it's meaningless, because the boxes are independent on purpose.
Everyone does, and the progression is remarkably consistent. I've watched a few teams walk it
AGENTS.md. CLAUDE.md. A docs/gotchas.md that someone started with real hope in their heart. This is the correct first move. It costs nothing, it works, do it.
It works until the file grows. Every agent reads all of it every time, so you're paying tokens on the connection-pool note while fixing a CSS bug. Nobody deletes anything, because nobody knows what's still true. Month four it's 1,400 lines, three of which are load-bearing, and nobody knows which three.
Then the real problems. No per-fact trust, so somebody's wild guess from last spring sits in the same prose, same font, same authority as the thing that's been true for a year. No expiry. No provenance past git blame, which tells you who typed it, not whether acting on it worked.
And it's in the repo, which sounds like a feature right up until a fan-out. Eight Sprites each learn something, each edit AGENTS.md, and now you have eight conflicting diffs and a merge strategy of "a human reads all of them." For a throwaway VM's insight to survive it has to become a commit, so a branch, so a PR, so someone's Tuesday. Most insights don't clear that bar. They just die.
An agent editing the file that shapes its own prompt, mid-session, is also a little cursed. I can't formalise the objection. I've just watched it go badly.
So, embeddings. Pinecone, Qdrant, Chroma, whatever's already in the stack. Embed the notes, top-k, stuff the prompt.
Retrieval is not the problem. I should be clear about that, because we get filed under memory and it would be extremely convenient for me to claim otherwise. On our own benchmark a plain vector store hit 98.6% recall against our 99.3%, with overlapping confidence intervals, which is a tie no matter how I dress it up. If your problem is finding the relevant note, you're done.
Memory is table stakes. We ship it, it's one feature, and I'd rather concede the benchmark than build a story on a tie. But a fleet that remembers everything and learns nothing still plateaus at whatever your best agent managed in week one. It just gets there with better recall.
The problem is one step later, after the agent acts on what it found.
A row that caused an incident and a row that shipped clean keep the same score forever. Similarity scores how the text reads, not whether following it worked. The note that burned you in February ranks just as high in March, and nothing in the system is capable of noticing.
Nothing expires either, so January's note about a service you've since rewritten comes back at full weight. And UPDATE destroys evidence: when a fact changes you overwrite the row, so after a bad run you can't reconstruct what the agent actually read, which is the only question anyone asks in the postmortem.
The most defensible version of all this. CREATE EXTENSION vector, a table with a text column and a vector column, an ivfflat index. One afternoon, genuinely fine.
The table is the afternoon. The rest is the project. A second agent starts writing, so you need attribution, conflict handling, scoped access. Schema and migrations, not a query. The table grows and old rows keep ranking, so now there's a scoring job, a cron, and a scoring policy you have opinions about. Then someone bumps the embedding model and the whole corpus needs re-embedding with the index kept in sync, which recurs roughly twice a year until the end of time. All of it is yours.
What I'd defend: keep the vector store for RAG over your docs, where similarity across millions of chunks is the right hammer. Put the agent's own knowledge somewhere that tracks who wrote it, whether it worked, and what it said last month. Those three are what a system needs to get better, as opposed to what it needs to remember.
None of it about retrieval quality. Writes that supersede instead of overwrite, so v1 of config-provider-retry-budget still exists after v2 lands and I can diff them. Every entry knowing which agent and session wrote it. A confidence number that moves when a task succeeds or fails, not when someone remembers to curate. A record of what the agent read before it acted, which everybody skips and then wants very badly at 2am.
That last one turned out to be the whole thing. Ranking, auditing and training look like three unrelated features, and they need the same object: the task that came in, what the agent read, what it did, and what happened next. Keep only the answer and you can retrieve. Keep all four joined up and you can rank by what actually worked, reconstruct a bad run in a postmortem, and train on the result. One record, three jobs.
The smallest thing I could build that shows environment 2 beating environment 1 at the same task.
Two Sprites, senselab-demo-a and senselab-demo-b, same image, one shared entity path, five acts driven by a bash script. Each box gets two Python files pushed in, standard library only, because a pip install in a cold VM eats the demo clock and I have watched that kill a live demo in front of people.
The test suite is where the effort went. Four tests, two of which cannot pass unless the environment carries a setting you have no way to derive from the code:
SUBTOTAL_CENTS = 3340
TAX_RATE = Decimal("0.075")
EXPECTED_TOTAL_CENTS = 3590
3340 × 0.075 is exactly 250.5. ROUND_HALF_UP gives 251, ROUND_HALF_EVEN gives 250, and 3590 only works with banker's rounding. You cannot read that off the file. Someone in finance decided it in a meeting, and the only way an agent finds out is trying both and seeing which one the test wants.
The other is a retry budget. The fake payment sandbox rejects the first two calls of any charge, so RETRY_BUDGET has to be 3 or more. Under --load it sheds four before settling, so the real floor is 5. That gap between "passes at rest" and "passes under load" is why act 4 exists, and it's the most true-to-life detail in here.
Each agent asks what's known about the entity. The entity path carries a timestamp suffix so every run is genuinely cold and I can't accidentally demo against warm memory.
Same from B. This is where every disposable environment lives today.
A brute-forces a 6-cell matrix, two rounding modes by three retry budgets, and goes green on attempt 5:
Then it writes two entries and commits an outcome. Each write carries a why, which I care about more than the value:
Tax on the 3340c fixture lands exactly on a half cent, so half_up overcharges by 1c and test_tax_rounding fails. Finance requires banker's rounding.
A bare value is a magic number, and the next engineer through will "clean it up." A value with a reason attached survives contact with a refactor.
A second box off the same image, which has never run this suite. It asks what's known, merges the env block from every entry that has one, and runs the suite exactly once:
Five attempts became one. Note the by sprite-a-agent on both entries: B knows whose work it's standing on, which is the part a shared directory can't tell it.
There's an assertion in here I care about more than the happy path: if either key is missing, B exits with an error saying so, instead of quietly brute-forcing its way to the same answer and making the demo look fine. If the mechanism this whole thing exists to prove is broken, I want it to fail loudly in front of everyone.
B runs the soak variant, which A never touched. The inherited budget of 3 isn't enough:
It finds 5 and writes the same key again. Not a new key, not a -v2 suffix, not a comment in a markdown file underneath the old one. The same key, versioned, with a rationale that's explicit about superseding:
Supersedes the earlier budget: 3 is enough at rest but the provider sheds four calls under load, so 5 is the real floor.
This is the bit a markdown file can't do without a human in the loop, and the bit an UPDATE does by destroying the old value.
A re-reads. No restart, no redeploy, no message passed between the machines (and the history):
Two authors, two boxes, 47 seconds apart, one key that now knows more than either agent did. Then the script deletes both Sprites and that history is still sitting there, ready for box number three.
Enough demo. Here's where I think this pays for itself, and what specifically breaks without it.
CI is the obvious one. Every branch gets a Sprite, the agent fixes the build, the box dies. Without a shared layer, agent number five repeats agent number one's investigation and you eat that cost per build, forever, at a flat rate. With one, environment quirks pile up per repo and new agents inherit them. You're buying a cost per build that falls over time instead of one that doesn't.
Fan-out is my favourite and almost nobody does it. You launch eight attempts, keep the winner, and the seven failures evaporate. But "the async client deadlocks under this pool config" is worth more than the winning diff and it costs one write to keep. Record the dead ends at confidence 0.8 and the next fan-out skips those branches before it spends a token. You end up with a search that narrows every time you run it instead of starting at full width every time.
Then anything with an unwritten API contract. Scraper selectors, undocumented rate limits, the endpoint that lies about pagination, the sandbox that rejects your first two calls. Zero presence in your codebase, which is why a fresh VM can't infer it and why it's worth keeping. The failure mode without it is quiet and expensive: your agent hits a rate limit, backs off, "fixes" it with a sleep(5), and next week a different agent does the identical thing.
Multi-tenant execution splits along a nice seam. Sprites give you isolation per session, SenseLab gives you continuity per tenant. Entity path per customer, box destroyed after every request, customer context outliving the machine that served it. That split is irritating to build yourself, mostly because the two halves want opposite lifetimes and you end up fighting your own session handling. Scoped keys matter more than usual in this shape, since one leaked credential reading across tenants is the whole product on fire.
On-call is a similar shape. A clean room with connector access to prod and no standing credentials is a reasonable place to let an agent dig during an incident. What you gain is a runbook that improves every time it's used rather than rotting in Confluence, plus the ability to answer "what did it read before it did that" once everyone's calmed down.
And onboarding, which I didn't expect. What a new hire asks in week one is roughly what a cold agent needs in its first minute. If your agents have been writing that down for six months, you have a more honest answer than your README does.
The boundary, because I don't think this is universal: if the knowledge belongs in the repo, put it in the repo. A version pin goes in pyproject.toml. A lint rule goes in the config. This layer earns its keep for things that are environmental, expensive to rediscover, and needed by a different box than the one that learned them.
Slightly separate point, but a fleet of disposable VMs is about the worst place to scatter long-lived API keys. Rotating means touching every box. Auditing means reading process environments. Revoking for one agent means revoking for all of them.
Sprites connectors keep the credential once per org and route calls through a gateway that authenticates the calling Sprite via Fly's request signature. So my agent's base URL is:
https://api.sprites.dev/v1/gateway/custom_api/<connection_id>
There is no Authorization header anywhere in my agent code. The gateway attaches the key server-side. Neither Sprite ever held a SenseLab credential, and the same request from my laptop gets a 401 because it has no Fly identity headers. If you're running agent fleets, do this regardless of what you decide about any of the above.
Everything so far is versioned storage with provenance, which is useful but not interesting. The interesting bit is what the outcome data does, and it does it at two different speeds off the same record.
The fast loop closes on every single run. The agent gets briefed, does the work, commits an outcome, and the confidence on everything it read moves accordingly. No prompt edit, no redeploy, no human in the path. That's the loop the five acts above are running, and it's why box two beat box one.
The slow loop closes over months, and it ends at the model. Same record, different consumer.
Both matter, and I'd argue the fast one matters more day to day, because it's the one that pays out this afternoon.
Every entry has a confidence number, and it moves when a task lands rather than when someone curates
Note the asymmetry. A success nudges confidence up 3%, a critical failure knocks it down 15%. Trust is cheap to lose and slow to rebuild, which is roughly how it works with people too. Results are clamped to [0, 1], so successes saturate at certainty instead of drifting past it and failures floor at zero instead of going negative. The propagation runs as a Postgres trigger rather than in application code, so it still fires when a write arrives from something that isn't your app.
Decay isn't a flat timer either:
effective_half_life = base × type_multiplier × (1 + log1p(recall_count)) × (2 if outcomes else 1)
Beliefs decay at half the base rate, experiences at 1.5×, plain facts at 1×. Anything validated by a real outcome gets double the half-life, and frequently read entries resist decay through that log1p(recall_count) term. A fact with a 30-day base, ten reads and one outcome ends up at a 204-day effective half-life. Cold guesses nobody has used fade out. Hot production-tested knowledge sticks around. That's the pruning cron you didn't have to write.
Retrieval starts with fixed weights fused by reciprocal rank fusion, semantic at 0.4, keyword, temporal and confidence at 0.2 apiece. Fine as defaults, but a default can't know that for your particular service recency matters more than confidence. So amfs_retrain fits a gradient-boosted model on your outcome history, labelling entries read before clean outcomes as positive and entries read before incidents as negative, and once trained it joins the fusion with 30% weight. Under 20 samples it falls back to confidence ranking, so nothing breaks while you're still accumulating. It reports feature importances when it trains, which is the number I'd actually look at, because a ranker that decided age matters more than outcome history is telling you something uncomfortable about your data.
amfs_calibrate does the same to the multipliers in that table. The 0.85 for a critical failure is a guess about signal strength, and it shouldn't be identical for a payments service and a logging service, so the calibrator measures how often causally linked entries actually turn up in later incidents and adjusts. It'll also estimate a decay half-life from the age distribution of entries that are getting read.
Here's the bit I find genuinely neat. A decision trace already holds the task that arrived, the tool calls the agent made, and how it turned out. That's a (context, action, reward) tuple wearing a hat. Your agents have been quietly assembling a training set as a side effect of doing their jobs, and the outcome label stands in for human preference annotation, which is the expensive part of building that dataset by hand.
amfs_export_training_data hands it to you as an SFT dataset for behaviour cloning, or DPO triples where a successful and a failed trace that opened with the same tool become chosen and rejected, or reward-model labels. Take it and train wherever you like.
One footgun before you go hunting for your dataset: those exports are built from tool calls, and a trace only has the prompt side if you passed task_input to commit_outcome. Skip it and the trace records what the agent decided but not what it was asked, which is the half that makes it trainable. Those traces get dropped from the export silently, so an empty result usually means you've been committing outcomes without the task text.
Or you skip the pipeline, which is the part we actually sell. Managed Models takes the traces you've already got, spins up the training job, runs the fine-tune, scores the result against a holdout of your own decisions, and serves it behind an OpenAI-compatible endpoint. No ML team, no training infrastructure, no eval harness to write.
Under 800 lines of bash and Python, most of it narration. With a Sprites org and a SenseLab key:
KEEP=1 AMFS_SPRITES_CONNECTOR_ID=<your-connector-id> ./run-demo.sh
KEEP=1 leaves the VMs up so re-runs skip the creates. Every run mints a fresh entity path, so you always start cold.
I have the script and Gist available here: https://gist.github.com/br-raia/8c73cc34b47235cd27d3db1453bc60f1
The takeaway isn't the tax fixture. It's that the second box was better at the job than the first one, the third would be better again, and both of the first two are deleted. That's the whole trick: let the environments be cattle, and don't let the learning be.
It's worth being blunt about who ends up owning that improvement. The models are rented and interchangeable, and you'll swap yours at least once more this year. The environments are disposable on purpose. What isn't either of those things is the accumulated record of how your systems behave under your agents: which fix worked, which one caused the incident, what the run that shipped clean had read first. Nobody can sell you that one, because it's a description of your company. It's worth more every month you keep it, and past a certain size it's specific enough to train on, which is the point where the intelligence your agents run on starts being yours instead of something you rent by the token.
We got very good at making machines disposable. The thing worth keeping was never the machine.
Free tier. No credit card. Connect in minutes.
