observability
24 posts.
-
Your old frontend's service worker will outlive your redesign
Moving a redesigned agent console to '/' broke two things no diff showed: a cached service worker, and stream cursors that outlived a server restart.
-
Our memory ceiling logged ABORT while the file kept growing
A size ceiling enforced in one of three writers logged ABORT every cycle while the file grew. Five memory bugs where the guard lived in the wrong place.
-
Chrome refuses an iframe silently, and onload still fires
A shipped panel proxied to an opt-in process, then went blank behind frame-ancestors. Two failure classes, one five-minute check you can run on your own stack.
-
A 'new' badge is a lie unless something records when you looked
A sidebar of scheduled agents needs a per-viewer watermark, written when the list is actually on screen. Plus the surrogate-pair bug that ate an emoji title.
-
A parked human-in-the-loop prompt needs a clock, or a restart is your only exit
A workflow menu waited three days for a digit until a gateway restart freed it. Parked asks need a lazy TTL, a mismatch counter, and control words everywhere.
-
Your MCP audit table probably has no producer
The audit table existed since migration 046 and held zero rows for its whole life. What I found instrumenting both MCP client stacks, and three checks for yours.
-
Nine agents in one worktree and each one knew only the branch name
How a daemon tells a coding session when a peer touched the same files: keyed on the transcript not the pid, silent unless the sets intersect.
-
The trust label your registry declares never reaches the model
Two memory searches per turn with no shared id, a THIS BLOCK WINS block that a budget could evict, and a trust field only a commit guard read. What fixing all three took.
-
The runs panel said 'No runs recorded yet'. The ledger had every run.
A scheduler wrote a run ledger nobody served, a skill prompt dropped the user's words, and a spawnSync stalled the gateway 9.6 s. Three checks for your own stack.
-
A console leaked one CSS class and hid 11 days of receipts
Two ways an agent console lied during a redesign: a view's opt-out class nothing removed, and a receipt table no page read. With a five-minute check for each.
-
Your first-run path is the only code that runs with defaults off
A liveness ping answered by the transport, and a demo riding a toggle that ships off. Two failures in one first-run path, plus the check to run on yours.
-
Two writers, one judge: how a review queue became 90% noise
A memory-conflict queue hit 1,377 rows on a vault with 33 real conflicts. One producer had an LLM judge and dismissed 97% of its own work. The other had none.
-
Your message table is storing your UI, not your transcript
Four producers wrote into one turn and two baked HTML into storage. What it took to make the turn receipt a real event, and the SQL to check your own store.
-
A skill that finishes into a tab nobody has open did not finish
Console-mode skills completed into a workbench nobody was looking at. How results got routed to the surface the user is on, and the sibling-tree trap that cost two days.
-
The renderer existed. Nothing on that surface ever called it.
A side panel drew agent plans perfectly and could not run one. Four presses found four bugs a green suite could not see, plus the delivery check that catches them.
-
66,570 characters reached the model and nothing logged them
I built a per-turn event log that rebuilds the exact request we sent, then measured the bytes it could not account for. The first number was 66,570.
-
Your agent's tool manifest is a comment until something reads it
A scheduled agent declared six MCP tools, called none of them, and reported ok. Enforcing that declaration at fire time cut selection from 1-of-942 to 1-of-6.
-
Your MCP allowlist controls tool names, not what they return
A read-only MCP profile still returned my cwd and MEMORY.md, because the disclosure was in the result envelope, not the tool list. How to check yours.
-
Two tables both called 'skill', and nothing knew which was which
Vodou had 160 file skills and 15 console skills sharing one word. Every feature picked a table and called it the truth. Here is the seam and how to find yours.
-
A 400-byte cap crashed my memory daemon on one emoji
Two memory bugs with the same shape: a truncation that counted bytes, and a fact verifier whose 'approved everything' looked identical to 'never ran'. With a five-minute check for your own stack.
-
Per-site memory off switches only work if every reader asks the same authority
Governing an agent's page memory took three enforcement points, a soft delete with a dry run, and a live test that found four defects the unit suites missed.
-
Your agent's approval gate is probably just a warning label
We shipped plan cards, real parallel tool calls and an approval gate for an MCP agent. The gate was decoration for two days, and the fan was never parallel.
-
A silent failure looks exactly like a feature you never built
Shipping an agent surface into 22 chat sites we don't own: the Chrome gesture that doesn't survive an await, and three builds all claiming one version number.
-
Every copy of the rule agreed. That was the bug.
Four surfaces minted the same token four different ways, and all four agreed. How we found the drift, why the tests couldn't see it, and the guard that now can.