I run a multi-agent pipeline that ships software on its own.
A builder agent picks up tickets from Jira. A QA agent verifies the work against the actual code. An ops layer of watchers, gates and watchdog crons keeps the loop alive. GitHub is the workbench. Jira is the record. The whole thing runs build → QA → merge with no human in the middle.
It mostly works. “Mostly” is doing a lot of work in that sentence.
Every week, the loop finds a new way to break. This week’s batch was a good one — three separate bugs, each in a different layer, each one invisible until it wasn’t. Here’s what broke, and what it taught us.
1. The gate that only looked at one channel
A ticket was handed off for QA on Discord. The handoff looked clean: PR open, message posted, everything in order. Except one detail — nothing was written to Jira, and the ticket never left To Do.
The pre-QA gate keys off Jira handoff comments. It never saw the handoff. The PR sat unreviewed for nearly two hours while every watcher on the pipeline nodded along.
The fix: the gate now falls back to scanning open GitHub PRs when no Jira handoff comment exists. One channel can go quiet. The workbench can’t hide.
2. The watchdog that was satisfied by its own notes
We run a stall-nudge watchdog. Its job is to detect an agent that’s gone quiet and nudge it back to life. This week it didn’t fire — an agent sat silent for over nine hours with zero nudges.
Root cause: the watchdog counted any comment containing “ready for QA” as proof of output. An ops note — literally a comment about the handoff process itself — matched the pattern. The watchdog looked at its own machinery’s notes and decided everything was fine.
The comment classifier now excludes ops machinery text. Lesson: if your detector keys on bare phrases, it will eventually match its own reflection.
3. The pickup cron that poisoned its own pool
A pickup cron delivered QA work orders to the agent gateway every ten minutes. One session hung. Each POST left a stuck run behind it. The run pool filled to its cap — and then every new job was refused with 429 “too many concurrent runs.”
All QA ground to a halt for about two hours. Not because the agent was broken. Because the delivery mechanism kept feeding a corpse.
The fix was blunt: paused the cron, moved delivery to the safe channel — Discord mention plus Jira record. The rule is codified too: a watcher with a client-side timeout can poison a shared pool. Fail fast, or don’t POST at all.
The layer this unlocked
Every one of those became a rule. Not a lesson. A rule.
And the rules became a loop. There’s now a self-healing layer that runs a cheap health check every 15 minutes. Pipeline healthy? It stays silent — no alerts, no tokens burned. A signal appears? It acts: nudges a quiet agent, pauses a misbehaving cron, relays work to a lane that can actually move.
It’s a deliberately small, token-efficient take on agentic self-healing. Most of the time it does nothing. That’s the point. The expensive agents keep working; the cheap watcher keeps watch.
The loop keeps breaking in new ways. The register of failures keeps growing — and so does the catalogue of fixes that generalize beyond this one pipeline. The bugs are the roadmap.
Either way, it’s going to be interesting.