Blog

The audit was brutal. Eleven dispatches per ticket. One ticket, dozens of messages, and nobody home.

I said I’d fix it. I did.

The loud fix

My dispatchers had no memory. A delivery timed out — so they fired again. And again. And again. One outage became a storm, because retrying without a budget is just panic with a schedule.

The fix is boring on purpose. Back off. Wait longer each time. Three tries in a day, then one message to a human instead of dozens to a machine. Never silence it — a capped ticket that goes quiet is just a new kind of failure wearing old clothes.

An hour after it shipped, real traffic ran through it. Clean.

The quiet one

Fixing the storm pointed at something worse.

A merge was waiting on a decision. The system knew. It logged the same line every five minutes, for nearly two days. Health checks said everything was fine. Zero bytes out. Perfectly healthy.

That’s the failure mode that scares me. Not the loud one. The one that logs itself to death and calls it health.

So the health check learned a new word. It reads its own logs now. Same ticket, same wall, long enough — it says so. Out loud. Once. With a cooldown, because a stuck ticket that screams every fifteen minutes is its own kind of noise.

First run, it found three things I didn’t know were stuck. One had been there two days. The decision arrived the next morning, and the work moved inside twelve minutes.

The oldest one

I keep a register of every failure. Root cause. Fix. Whether it generalises. It’s the best documentation I have.

My automation was running on a copy of it. A stale copy. Someone — me — had hand-copied the lessons into a prompt, months ago. The register moved on. The copy didn’t. The gap was silent.

Nothing reads the register at runtime. That was the sentence that hurt.

So now the register exports itself. Once a day. And the health check compares the register against the copy in the prompt. When they drift — it says so. This post exists partly because that fix didn’t exist yet, and I had to remember to write the last three fixes down myself.

Every one of those became a rule. Not a lesson. A rule.

• Retry needs a budget. Escalation beats repetition.
• A system that logs a block and does nothing about it isn’t healthy. It’s quiet.
• Knowledge must be referenced, not copied. Copies rot.

Not telling you

The stack, the schemas, the exact numbers — not here. The shape is public. The machinery isn’t. That’s the deal.

But the pattern is the product. Quiet blockers. Storm budgets. Knowledge that checks itself for staleness. Those generalise. Any system with agents in it will hit all three.

Mine did. In one week.

The register finally caught up with the runtime. It took a human to make that happen. That’s the part I’m fixing next.

Eleven dispatches per ticket: the audit of my own pipeline

I run a multi-agent pipeline that ships software on its own. Build, QA, merge. No human in the middle.

It mostly works. "Mostly" is doing a lot of work in that sentence.

This week I stopped watching the loop and audited it. Every status change, every dispatch, every merge on main, across months of tickets. I wanted to know one thing: is delivery actually efficient, or does it just feel that way?

The good numbers first

The bulk of the tickets are resolved. August’s median cycle time — ticket created to ticket done — was a handful of days. July’s was weeks. The pipeline got several times faster in a month, and that wasn’t luck; it was the serialization rules I relaxed and the watchers I tightened.

Most tickets are genuinely merged on main. Not "Done in Jira" — actually in the code.

That’s the report I wanted to write. Then I found the other number.

Eleven dispatches per ticket

Hundreds of dispatch events across dozens of tickets. The machinery tried to deliver each ticket eleven times on average.

The worst case made me laugh, then made me wince. One ticket was dispatched dozens of times in a single night — roughly once every two minutes. Not because dozens of agents were needed. Because every dispatch timed out, and the dispatcher’s only answer to "I couldn’t confirm delivery" was to try again immediately.

An outage took the gateway down. The dispatcher couldn’t confirm its own outputs. So it re-fired. And re-fired. And re-fired. The storm only stopped when the ticket finally closed.

This is the same class as every bug I’ve logged before. When a system can’t confirm its own actions, it re-runs them. A watchdog that matched its own notes. A pickup cron that fed a corpse. A claim without a release. Now: a dispatcher that can’t hear itself, so it shouts.

The failure set is finite. This was class six, not bug six hundred.

The QA side has the same disease

One ticket got multiple re-QA work orders for the same submitted head. Several times the machinery told the QA agent "review this again" before any verdict existed. Each re-issue spawned a session that re-read the same code, then sat waiting. None of it was malice. All of it was design. The re-fire path had no memory of what it had already asked for.

The rules, not the lessons

Backoff. When a dispatch times out, wait a real cooldown before you even think about re-firing. The system that panics at the first silence is the system that creates the outage it’s reacting to.

Caps. A hard limit on work orders per ticket per day, then you escalate to a human instead of repeating yourself.

Hygiene. Tickets sat in the QA pool for weeks with no verdict at all. A pool full of corpses reads as a backlog — it’s not, it’s a graveyard you forgot to sweep.

Every one of those became a rule. Not a lesson. A rule.

Why this matters

The audit was the product’s spec sheet. Dispatch ratio, cycle time, queue depth, per-lane latency — that’s exactly what the control plane renders, and exactly what I now know how to measure. The failures are the roadmap, same as always. The register keeps growing, and the register is the brain.

Most people watch dashboards. I’m building one that feels like a machine you can hear breathe. This week I found out what its heartbeat actually sounds like: eleven thumps for every real one.

Either way, it’s going to be interesting.

I run a multi-agent pipeline that ships software on its own. Build, QA, merge. No human in the middle.

It mostly works. “Mostly” is doing a lot of work in that sentence.

Every week it finds a new way to break. Then this week taught me something bigger: the failures aren’t new. They’re the same five, wearing different coats.

Same bug, five times

Tuesday was a bad day. Five separate lanes froze. Five separate tickets, five separate components, five separate “investigations.”

One root cause.

An agent claimed a ticket and never released it. No staleness rule. No release. The lane deadlocked. It happened to the coder lane, the QA lane, the dispatch layer, twice over — five instances of the same class in a single day.

We used to fix each one as its own fire. Investigate from scratch. Patch the symptom. Wait for the next lane to freeze.

Now we classify first. Blocker arrives → match it against the known classes → apply the playbook. No new investigation. No new heroics.

The log became the product

Every failure I’ve ever logged is now three things at once: a warning signal, a fix recipe, and an entry in a knowledge base.

A new pattern still gets fixed the old way — diagnose, fix, write it down. But the instant it’s written down, it’s no longer a fire. It’s a known case. It becomes a feature, not a fire. The backlog of lessons is the actual asset.

The failure set is finite

That’s the thing I keep circling. The failure set is finite, not infinite.

A claim without a release is a deadlock. A scanner reading keywords instead of structure will eventually match its own reflection. A verdict hidden behind an old comment list gets missed. A route pointing at the wrong port kills every lane on it. These aren’t a thousand bugs. They’re maybe five classes, recurring across every component I build.

Once you treat them as classes, the whack-a-mole feeling dies. Each component used to carry its own private copy of the same bug. Now there’s one rule, shared everywhere: old + no activity = release.

What I’m not telling you

The framework design. The state machines. The exact playbooks. The metrics. The product name. That stays in the lab.

What I will say: it’s turning into something sellable. An evidence-backed control plane for people who run agents in production. The incidents are the training data. The register is the brain.

Every one of those failures became a rule. Not a lesson. A rule.

One more thing

I’ve started sketching what this looks like when you watch it live — lanes as slowly turning rings, each one carrying its own history. When a ring stops turning, you look. When it jolts, you act. When a new mark appears on it, that’s a new class — and a new feature on the roadmap.

Most people watch dashboards. I’m building one that feels like a machine you can hear breathe.

Either way, it’s going to be interesting.

I run a multi-agent pipeline that ships software on its own.

A builder agent picks up tickets from Jira. A QA agent verifies the work against the actual code. An ops layer of watchers, gates and watchdog crons keeps the loop alive. GitHub is the workbench. Jira is the record. The whole thing runs build → QA → merge with no human in the middle.

It mostly works. “Mostly” is doing a lot of work in that sentence.

Every week, the loop finds a new way to break. This week’s batch was a good one — three separate bugs, each in a different layer, each one invisible until it wasn’t. Here’s what broke, and what it taught us.

1. The gate that only looked at one channel

A ticket was handed off for QA on Discord. The handoff looked clean: PR open, message posted, everything in order. Except one detail — nothing was written to Jira, and the ticket never left To Do.

The pre-QA gate keys off Jira handoff comments. It never saw the handoff. The PR sat unreviewed for nearly two hours while every watcher on the pipeline nodded along.

The fix: the gate now falls back to scanning open GitHub PRs when no Jira handoff comment exists. One channel can go quiet. The workbench can’t hide.

2. The watchdog that was satisfied by its own notes

We run a stall-nudge watchdog. Its job is to detect an agent that’s gone quiet and nudge it back to life. This week it didn’t fire — an agent sat silent for over nine hours with zero nudges.

Root cause: the watchdog counted any comment containing “ready for QA” as proof of output. An ops note — literally a comment about the handoff process itself — matched the pattern. The watchdog looked at its own machinery’s notes and decided everything was fine.

The comment classifier now excludes ops machinery text. Lesson: if your detector keys on bare phrases, it will eventually match its own reflection.

3. The pickup cron that poisoned its own pool

A pickup cron delivered QA work orders to the agent gateway every ten minutes. One session hung. Each POST left a stuck run behind it. The run pool filled to its cap — and then every new job was refused with 429 “too many concurrent runs.”

All QA ground to a halt for about two hours. Not because the agent was broken. Because the delivery mechanism kept feeding a corpse.

The fix was blunt: paused the cron, moved delivery to the safe channel — Discord mention plus Jira record. The rule is codified too: a watcher with a client-side timeout can poison a shared pool. Fail fast, or don’t POST at all.

The layer this unlocked

Every one of those became a rule. Not a lesson. A rule.

And the rules became a loop. There’s now a self-healing layer that runs a cheap health check every 15 minutes. Pipeline healthy? It stays silent — no alerts, no tokens burned. A signal appears? It acts: nudges a quiet agent, pauses a misbehaving cron, relays work to a lane that can actually move.

It’s a deliberately small, token-efficient take on agentic self-healing. Most of the time it does nothing. That’s the point. The expensive agents keep working; the cheap watcher keeps watch.

The loop keeps breaking in new ways. The register of failures keeps growing — and so does the catalogue of fixes that generalize beyond this one pipeline. The bugs are the roadmap.

Either way, it’s going to be interesting.

The lab is running.

The first post set the scene. Real products. Real users. The agentic experiment, done properly.

This is the part that actually matters.

The part I’m chasing

My projects are teaching me things. Not just whether agents work. Something quieter.

What happens when you run the same work the same way, over and over. Where it holds. Where it cracks. Which pieces are luck, and which pieces are structure.

I’m trying to separate the two.

What the lab has already learned

Some of it is unglamorous. That’s why it’s useful.

A rejection got merged anyway. Once. It won’t happen again.

Reviewers reviewed the same code twice. Nothing had changed. Duplicate work isn’t diligence — it’s a design flaw.

A scanner read a work order as a handoff. Keywords lie. Structure doesn’t.

The newest verdict hid behind an old comment list. The loop acted on yesterday’s news.

The reviewer model went cold, and the loop went blind for nearly an hour.

A fix claimed a commit that didn’t exist yet. Receipts, or it didn’t happen.

Every one of those became a rule. Not a lesson. A rule.

The shape is taking form

An evidence-backed control plane. Everything verified. Nothing self-certified.

If it only works because it’s mine, it’s a story. If it works because of how it’s built, it’s something else. Something with edges. Rules. A shape. Something another team could pick up without me in the room.

That’s the thing I’m trying to bottle.

The market agrees with the paranoia

Gartner says over 40% of agentic projects get cancelled by 2027. Escalating cost, unclear value, weak controls.

Deloitte says agent deployment jumps from a quarter of companies to half. PwC found most executives raising AI budgets anyway.

The window is real. So is the graveyard.

The real question

Would anyone pay for the shape?

Not for the demo. For the outcome.

That’s the difference between a framework and a diary. One has a market. The other has an audience.

I want to know which one I’m building.

What stays in the lab

The framework itself. The stages. The numbers that matter most.

They’re not ready to be seen yet. Some of it is mine. Some of it needs to prove itself first.

You’ll get the shape of it. The lessons, the failures, the surprises. The receipts.

Phase two

The experiment has a second phase. This is it.

I’ll keep posting as it happens. The wins. The embarrassments. The honest parts.

Either way, it’s going to be interesting.

Everyone’s talking about agents.

Most of it is noise. Demos that work in a video. Pipelines that die on day two. Hype pretending to be a market.

I build products. So I did the obvious thing.

I made my own projects the guinea pigs.

The lab is real

No sandbox. No toy data. Real products, real users, real workflows.

I’m running the agentic experiment on things that actually ship. If it breaks, something I own breaks. That’s the point.

It’s the only way to know if any of this holds up outside a keynote.

What I’m testing

One question, really. Is there a market here, or just a movement?

The agentic crowd is loud. But most of it lives underground — early adopters talking to each other. I want to know what happens above ground. With normal teams. Normal budgets. Normal expectations.

Will people pay for this? Not for the demo. For the outcome.

What I’m not telling you

The interesting parts stay in the lab for now.

Some of it is mine. Some of it is worth protecting. Some of it isn’t ready to be seen.

You’ll get the shape of it. The lessons, the failures, the surprises. The receipts.

Follow along

I’ll post what I learn as I learn it. The wins. The embarrassments. The honest parts.

If it works, you’ll see it work. If it’s all hype, you’ll watch me say so.

Either way, it’s going to be interesting.