Why Agents Need a Cognitive Layer (And How to Build On Top of One)
We can get agents to use tools, follow instructions, coordinate through chains and graphs. What we can't easily do is get them to collectively improve at a task over time.
A deploy finishes. An agent reads the logs for nine minutes and tells you one of two things. Either "cold image, ignore it," or "this looks like a product regression, here's a draft PR." How much do you trust that?
A deploy finishes. An agent reads the logs for nine minutes and tells you one of two things. Either "cold image, ignore it," or "this looks like a product regression, here's a draft PR." How much do you trust that?
And a harder question underneath it: how would you even check? For a lot of what we ask agents to do there's a right answer in a file somewhere, and you grade against it. "Was this deploy a real regression" has no such file at the moment the deploy lands. You find out hours later, or you find out because a customer did. So the usual move, score the agent against ground truth, isn't available to you when it matters.
Netflix hit the same wall doing causal inference with agents, and I keep coming back to their writeup because the shape of the answer is the one we landed on independently. You can't grade the output, so you grade the process. You make every step something a human can open and argue with, and you keep a human on the steps that can hurt.
This is how we run that against our own production. The last post was coding agents in throwaway Sprites where the point was that you delete the box and keep what it worked on and the results. This is the same idea aimed at deploys and alerts, which is a meaner problem than a tax-rounding test, because the failure mode is your own system.
The obvious build is a long-lived on-call agent. One process, a Slack socket, prod credentials, a prompt that grows a paragraph every incident. I don't want it. Not because it wouldn't work on a good day, but because of what it is on a bad one: a single box holding keys, able to merge, able to call gcloud run services update-traffic. Agents try to be helpful. Helpful plus traffic control is how you roll back a healthy revision because its replacement was still pulling.
So we don't keep one alive. A release goes out, GitHub Actions hits a webhook, we spin up a Sprite, it watches, it writes down what it saw, we delete it. Next release, new box. Sprites boot in about 2.5 seconds, so one microVM per deploy and one per alert costs nothing to think about.
That buys isolation and it costs continuity, the thing Sprite 1 learned is exactly what makes Sprite 50 worth running. Here's the failure I mean, concretely:
Cloud Run keeps serving the old revision while the new image pulls, so `/health` on the URL looks green while the new revision's startup probe times out. To a cold agent that reads like an outage. It pages you. You mute it. Two weeks later a real traceback shows up in the same shape and you've trained yourself to ignore it.
We named that class the first time it bit us: incident-2026-08-14-amfs-api-startup-probe-timeout. The whole game is getting that information in front of Sprite 50 before it opens Cloud Logging. And the honest version of the game is stronger than "remember the note." It's "remember the note, and remember whether acting on it actually worked last time," because a runbook that can't tell a good call from a bad one just launders your worst guess into policy.
`AGENTS.md` and Confluence don't do that. They don't rank, they don't know who wrote a line, and they don't get less trustworthy when the thing they told you turned out wrong. Eight Sprites editing one markdown file is a merge conflict, not a runbook. A shared volume is a pet you just built and forgot to page on.
What we run instead is the SenseLab loop, which is the continual-learning idea from the last post: brief on what the last watches worked out, do the watch, commit an outcome, and let confidence on what got read move up or down based on how it went. Nobody edits a prompt. The agent that ran yesterday is why today's is cheaper.
Netflix splits the work into three roles: a Principal that sets the constraints and owns the calls that can hurt, an Actor that does the actual investigating, and a Critic that decides whether the Actor's answer can be trusted. We ended up in the same place, with one difference worth stating up front: our critic is not a second LLM second-guessing the first. It's code and history.
The principal is the human. We write the constraints, we review draft PRs (maybe? do we? have we ever? at least we should, right?), we move traffic, and we also write investigations onto the same record the agents read. That matters more than it sounds, because it means the agent's runbook and ours are literally the same entity.
The actor is the Sprite. It briefs the entity, compares the repos, watches logs, classifies, writes a diagnosis or an incident, and sometimes drafts a PR. It runs with a viewer's view of the world and no way to change traffic, which I'll get to.
The critic is gates.py (a small file of hardcoded pass/fail rules we have set up) together with the briefing and the outcome history. When the actor says "this feels like a regression," nothing asks a model to agree. A function in that file checks whether we've seen this before and whether a prior instance of it actually resolved well, and the briefing has already put the known issues/fixes in front of the actor before it decided anything. The critic is boring on purpose. Boring is auditable.
Because there's no ground-truth label at the moment of a deploy, every run has to leave something a human can open later and disagree with. A quiet watch leaves a deploy-<sha> entry: both SHAs, the changed files, the commit messages, the promote-PR titles, and a committed outcome. A noisy one leaves a diagnosis-<component> and maybe an incident-<name>, cross-referenced to the class it belongs to. A proposed fix leaves a draft PR whose body you can read against what the agent read. You can reconstruct "what did it see before it did that" after the box is long gone. That's the substitute for an eval score, and it's the same substitute Netflix leaned on: inspectable artifacts plus a human where it counts.
Here's a deploy end to end, because the architecture makes more sense as a story than as a diagram.
`deploy.yml` POSTs both SHAs with an HMAC over the body. Two SHAs because we ship from two repos that drift: `amfs-internal` tags the images, and the open-source `amfs` is what `build-amfs-pro` actually checks out. A changelog that names one of them sends you debugging the wrong tree, which humans already do by hand and a cold agent does every single time. If the webhook URL or the signature is missing the job just skips, and if dispatch returns a 5xx the ship still succeeds. An agent watching a deploy should never be able to fail the deploy.
The thing that catches the webhook, sre-dispatch, is Cloud Run scaled to zero. It is not an agent and holds nothing interesting. It checks the HMAC (or a bearer token, since Cloud Monitoring can't sign a body), looks up deploy-<12-char-sha> so we don't watch the same ship twice, enqueues a Cloud Task, and returns 202 in milliseconds. GitHub shouldn't wait nine minutes for a log watch. That dedup lookup taught us something dumb early on: hosted SenseLab answers a missing key with HTTP 200 and {"status":"not_found"}, not a 404, so our first version read every deploy as "already handled" and never spawned anything. Parse the status, not the status code. Alerts dedup the same way on diagnosis-<component> within 30 minutes, because status-page flap and Monitoring retries will otherwise hand you a dozen investigations of one blip.
Only the Cloud Task can spawn the actor, and it proves that with an OIDC token whose audience is the execute URL and whose email has to be amfs-sre-dispatch@. It mints a viewer token good for about an hour, logging.viewer and run.viewer and nothing else, no JSON key on disk anywhere. Then it creates the Sprite, copies agent.py in, and runs it. The image has python3, curl, and claude. It does not have gcloud. So the "agent rolls back traffic" scenario I opened with isn't guarded against by a careful prompt, it's guarded against by there being no binary in the box that could do it. SenseLab and GitHub reach the agent through Sprites connectors, which keep the credentials once per org and authenticate the calling Sprite at a gateway, so there's no API key in the environment and the same request from my laptop gets a 401.
The first thing the actor does is brief amfs-internal/deploy. That's the loop, sitting at the top of the run, before Cloud Logging is even open. Then it compares both repos through GitHub's compare API, expands any "Promote to main" commits into the PR they actually carried, and writes the changelog. Then it watches amfs-api and pro-api for nine minutes, reading the serving revision and the logs, and pointedly not polling /health, because /health is the thing that lies. A quiet watch commits a success (with `task_input`, or the trace is useless later). A startup-probe timeout that matches the August key classifies as `cold_image`. Then it destroys itself, and the writes stay behind.
The timeouts are a budget, not a suggestion. Cloud Run kills the request at 900 seconds, so if the watch ran that long the finally that destroys the box never runs and you get a sre-a-* VM left up on the Fly dashboard. We set the watch to 540 to leave room for create and destroy, and we still lose that race occasionally, which is honestly the tidiest way to notice the budget slipped.
Alerts run the identical path. The status page signs the body; Cloud Monitoring sends a bearer token because it can't HMAC. Both have already waited out their own confirmation and skipped the all-fail no_data cycles, so the actor writes a diagnosis and hits the same gate. It doesn't relitigate whether the alert is real.
And the gate is where the actor's ambition meets the wall. should_draft_pr in gates.py, pinned by tests, allows a draft PR only for a product_regression we have seen before, where a prior instance of that class resolved successfully. Everything else, cold image, first sighting of a class, an SSE hiccup, gets a written entry and a Slack message and stops. Merge is ours. Traffic is ours. We don't auto-rollback.
This is the one that convinced me the loop was doing work.
Logs come in with failed to start and listen and startup probe. The briefing already carries incident-2026-08-14-amfs-api-startup-probe-timeout, so classify_from_logs returns cold_image, and should_draft_pr returns false even if you hand it product_regression and a prior success on purpose. Tests hold that line. The actor writes the incident, links it to the August key, posts to Slack, and destroys. The next deploy's briefing still has the key. Nobody pasted anything into a prompt.
The parallel to Netflix is exact. Their critic caught an overlap failure, an early-adopter bias that made a naive estimate roughly four times too big, and the value wasn't that the agent computed something, it was that a diagnostic refused to let a confident-looking answer through. Ours is the same move with smaller words. The agent is allowed to run. The dangerous action is blocked. The artifact says why. Hand that same model the raw logs and a GitHub token and it opens the PR, because a startup-probe timeout looks like an outage and /health looks like proof the new revision is live. The scaffolding is the whole difference, and it's the same model on both sides of it.
The cold start shows memory surviving a fresh box. This one shows it changing its mind.
In July a change to the retrieve path started sorting results by recency instead of confidence, so amfs_search handed back the least-trustworthy entry first. We named the class regression-2026-07-30-search-confidence-sort, an actor drafted the one-line fix, we reviewed it, merged it, and committed the outcome a success. So the class now carried exactly the record should_draft_pr looks for: a product_regression, seen before, last instance resolved well. The next time it appeared, the gate would say draft, and it did.
Except the fix didn't hold. It reordered in Python while the SQL ORDER BY still won under load, so the moment real traffic came back the regression came back with it. We committed that outcome against the same class as a minor_failure. This is the move a markdown file can't make: the entry that used to mean "this class drafts safely" didn't just sit there being wrong, it lost confidence, and the success it carried was now outweighed by a failure on the same key. should_draft_pr doesn't read "is this class known," it reads whether acting on it last time actually worked, and the answer had changed.
So the third time the sort regression showed up, classify_from_logs still called it a product_regression and the class was still "seen before," but the gate returned false. The actor wrote the diagnosis, linked it to the class, posted to Slack "drafted this before, the fix didn't hold, a human should look", and stopped. No PR. A plain store keyed by class would have drafted the same broken fix a third time, confidently, because to a lookup "known class" and "known-good fix" are one fact. They aren't. The gap between them is the whole product, and it's the thing that decayed. It's the same critic move as the cold start, aimed at the agent's own past answer: the diagnostic refused to let a confident-looking fix through, because the record said that confidence hadn't earned out.
Two speeds, one record. Same day, Sprite N briefs on Sprite N-1's writes, so a known cold image classifies in a single pass and a repeat regression can draft because a prior one already resolved well. Entries behind a quiet watch gain confidence; entries behind a misclassification lose it. Nobody curates a wiki for that to happen. The exact multipliers and the decay curve are in the environments post, which is where that machinery belongs.
The slow speed is the part that's genuinely ours to keep. Every trace is the task that came in, what the agent read, what it did, and how it turned out, which is a training example in all but name, provided you passed task_input when you committed the outcome. We skipped that once in a smoke test and got an empty export, which is the system telling you the truth: a trace with no task on it records what the agent decided but not what it was asked, and that's the half you'd train on. We are not fine-tuning an on-call model off ten deploys. We are keeping the join key so that at a few thousand we can.
The practical shape of all this: adding another watched service is a webhook and a Sprite, not another long-lived bot with a bigger prompt. The expensive part is the first real investigation of a class. After that the briefing is a cache, and it's a cache that knows which of its entries have paid off.
Everything here is two primitives you can pick up today. Sprites give you the box that boots in 2.5 seconds and dies without regret. SenseLab gives you the record that outlives it, the briefing at the top of the run, the confidence that moves with outcomes, the trace with task_input still attached. One is scale, the other is memory. You want both, because a box with no memory relearns every failure, and a memory with no boxes is just a wiki nobody updates.
Start small and mean it: one webhook, one Sprite that watches your next deploy, one commit_outcome when it's done. The first watch is the expensive one. After that, the briefing does the reading for you, a known failure classifies in a single pass, and the agent that ran this morning is why tonight's is cheaper. That's the whole compounding loop, and it starts paying on run two.
Spin up a box at sprites.dev give it a brain at SenseLab, and point a throwaway agent at a permanent record. Delete the box. Keep what it learned. Let the record compound while your infrastructure stays cattle.
Free tier. No credit card. Connect in minutes.
