Agentic Delivery Pipeline — Michael O'Reilly

Why it exists

Coding agents are good at building things and bad at deciding what to build. So I put the deciding in front of the building, and wrote it down.

Working with Claude interactively, the same failures kept recurring: it would guess at an ambiguous requirement rather than ask, drift into files that were not its business, or report a green test run that had quietly skipped the tier that mattered. None of those are model problems. They are process problems, and process is the part of delivery I already do for a living.

The design brief

Treat the agent as a capable contributor on their first week: strong technically, no context, and no way of knowing which conventions are load-bearing. That framing decides everything else — the written-down rules, the narrow permissions, the single approval gate, and the insistence that acceptance criteria exist before any work starts.

From ticket to draft pull request

An idea becomes a planned ticket, the ticket waits for me, and an overnight job turns the ones I have approved into draft pull requests.

Human and agent
We plan the change together. The plan is not finished until it has an acceptance criteria section — a short checklist that can be checked objectively rather than argued about.
Command
A ticket-filing command turns the agreed plan into a story in the project's backlog, copying those acceptance criteria across verbatim. It asks for a priority, because priority decides build order.
Human — the gate
I review the backlog and move the tickets I actually want built into the "selected for development" status. Nothing else triggers a build.
Unattended
Overnight, a scheduled job takes the highest-priority approved tickets, builds each in a throwaway git worktree, runs that project's own QA commands, hands the result to two agents that did not build it, and opens a draft pull request.
Human
I read the pull request in the morning. The ticket stays in progress whether the run succeeded or failed — "done" means I have accepted the work, not that a pull request exists.

Why a worktree

Each ticket is built in a disposable git worktree branched from the project's base branch, and removed afterwards. The everyday checkout is never switched, never stashed and never left mid-rebase, so a job that dies at 04:00 cannot cost me a morning.

One human gate, deliberately

There is exactly one place where a person has to say yes, and it is before any code is written rather than after.

Moving a ticket into "selected for development" is the whole approval mechanism. It is a status change I make on a ticket whose acceptance criteria I agreed while I was in the room, which means the unattended job never has to interpret intent — it only has to satisfy a checklist that already exists.

What the agent may not do

The agent never moves a ticket into that status, never pushes to a base branch, never force-pushes and never merges. The draft pull request is the handover, so bypassing it would defeat the point of having a gate at all.

Layered context files

The agent reads a stack of written instructions before it does anything — general ones first, then whatever is specific to the project it is in.

Most of what an agent gets wrong is something a colleague would have told it. Writing those things down once, in the place the agent will look, turns out to be most of the work — and the same files are useful to a human joining the project.

The layers

A personal layer covers how I want to be worked with and what must never happen without asking. A project layer, committed alongside the code, covers that codebase's structure and conventions. A rules layer holds the things that are true of one machine or one workflow rather than of the project, so they do not end up in a pull request on somebody else's repository.

Skills, commands and subagents

Five small pieces of custom tooling, each written because a task kept being done inconsistently by hand.

Piece	What it does	Why it exists
Scaffold skill	Sets up a new project folder with its structure, its own instructions file, and its build configuration.	New projects were being started slightly differently each time, so the tooling never quite applied to them.
Ticket-filing command	Turns an agreed plan into a backlog story, carrying the acceptance criteria across word for word.	Criteria were being paraphrased on the way into the tracker, which is exactly where the meaning gets lost.
Regression subagent	Runs one project's regression suite in a clean context and reports what actually broke.	Test triage is long and repetitive, and it reads better done in isolation than inline in a build.
Test-engineer subagent	Decides for itself whether each acceptance criterion is genuinely proven, writes the coverage that is missing, and runs it.	The agent that built the change was also the one certifying it. A builder marking its own homework will pass itself.
Code-reviewer subagent	Reviews the finished change — the work and its tests together — and says which findings block and which are merely recorded.	Tests prove a criterion was met. They say nothing about whether the way it was met is one I would want to keep.

Deliberately small

One skill, one command, three subagents. Each replaced a task I was doing by hand and getting wrong in a specific way, and none was added speculatively — the two newest exist because a specific failure demanded them, which is the same test the first three had to pass. Adding more for their own sake would just be more surface to keep accurate.

Choosing a model per job

The work that needs judgement gets the stronger model. The work that is really just following a written procedure does not.

Planning, building, testing and reviewing all run on the stronger model at high reasoning effort, because those are the jobs where getting it wrong is expensive and hard to spot. That deliberately includes both verification roles: judging whether a criterion is actually proven, or whether a change is one worth keeping, is the most judgement-heavy work in the whole pipeline, and paying less for it would defeat the point of separating it out. Two narrow roles still run on the mid tier instead: both are rule-following rather than open-ended, with a written procedure to follow and a clear notion of correct.

The test

Would a careful person need to make a judgement call here, or only follow the steps? Judgement gets the stronger model. Following steps does not, and paying for it there buys nothing.

Four layers of guardrail

Written rules tell the agent what not to do; permissions stop it doing some of those things at all; independent agents check the work rather than taking its word for it; and the draft pull request means nothing reaches a main branch without me.

Written rules
The instruction files spell out what needs asking first — pushing, installing, deleting, anything touching a shared branch. This layer is advisory, and it is the one that catches the most.
Permission policy
An allow and deny list decides which commands run without a prompt, which prompt, and which are refused outright. Read-only inspection is free; anything that changes state is not.
Independent verification
Two agents that did not do the building decide whether it holds up. Neither is told what was built. This layer catches the thing the other three structurally cannot: work that broke no rule, needed no forbidden command, and is simply not as finished as it was reported to be.
The draft pull request
The last layer is structural rather than configured. The agent can only ever produce a branch and a draft pull request, so the worst outcome of a bad night is a pull request I close.

Why four and not one

Each layer fails differently. Written rules depend on the agent reading and following them. Permissions are reliable but blunt, and cannot tell a good git push from a bad one. Independent verification judges the work rather than the conduct, so it is the only layer that can catch a job done badly rather than done wrongly — but it depends on the verifier being genuinely uninformed, which is a property that can be lost by accident. Only the last layer holds regardless of what any agent decides, which is why the rule that it opens drafts and never merges is the one I would keep if I had to drop the other three.

The thing that builds is not the thing that verifies

Two agents sit between the build and the pull request, and neither is told what was built.

At first, one agent picked the ticket, built it, ran the checks, judged the result against the baseline, and wrote the summary describing how well it had done. Every verdict in the morning report came from the thing that had just built the work it was assessing. It was not dishonest. It was simply the only witness, and it had a stake in the answer.

So the judging moved out. A test engineer decides for itself whether each acceptance criterion is genuinely proven and writes the coverage that is missing. A reviewer then reads the finished change. Each starts cold, with the ticket and the diff and nothing else — and that omission is the mechanism rather than an oversight. A summary from the builder tells the reviewer where to look, which is the same as telling it where not to.

Build
The agent implements the ticket and gets the project's declared checks back to their baseline. Then it commits, because a reviewer handed an uncommitted change reads an empty diff.
Test
A fresh agent works out for itself which criteria are actually proven, writes the tests that are missing, and reports a verdict per criterion. A criterion it cannot prove is reported as unproven rather than quietly assumed.
Review
A second fresh agent reads the change and its tests together. Tests come first because a failing test changes the code, and reviewing before that happens means reviewing something that no longer exists.
Human
The draft pull request carries both verdicts, in their words rather than the builder's, including whatever they could not check.

What blocks and what is only recorded

Only two things stop a pull request being opened: a broken acceptance criterion, or a serious bug. Everything else — style, hesitations, pre-existing problems however bad they look — is written into the pull request for me to read in the morning. The distinction matters because a reviewer that can block on anything is a reviewer that blocks on everything, and an unattended job with nobody awake to overrule it would simply stop producing work.

The reviewer's first act was to fail the change that created it

On its first run it found that the review step had been wired to inspect the work before the work was committed. It would have been handed an empty change, found nothing to object to, and correctly reported it clean — a pull request that looked reviewed and had been reviewed by nobody. Every guard I might have added would have stayed silent, because both stages really did run. That is the failure this layer exists to catch, and it caught it on itself first.

A quality gate that had never run

A command one project declared as a required check was missing from the permission allow-list. Unattended, a permission prompt is a refusal rather than a pause, so the check was silently skipped and the job reported a pass it had never earned. I found that class of fault by hand three times before making the check automatic — which is the general lesson: a gate nothing verifies is indistinguishable from a gate that passes.

What it costs

A night's work is slower by a large factor. The earliest live run built two tickets in thirteen minutes; a single ticket is now budgeted for most of an hour, because that budget has to cover the build and both handovers rather than the build alone. The nightly limit of three tickets has not changed, but it is now a real ceiling rather than a number the job never got near. That is the trade in miniature: fewer things, actually checked.

QA judged against a baseline

A passing run means "nothing got worse", not "everything is green" — because on a real codebase those are rarely the same thing.

Every enrolled project declares its own QA commands and, next to each one, what a pass looks like there. Some suites have known failures that are stale tests rather than product bugs. An agent told to get to zero would either never finish, or would "fix" them by editing code it should not touch — so it is told the baseline instead, and told explicitly that fixing a baseline failure is not its call.

Per-project configuration

{
  "baseBranch": "<base-branch>",
  "workDir": "<subdirectory-changes-belong-in>",
  "setup": ["<restore gitignored files the tests need>"],
  "qa": [
    {
      "cmd": "<lint command>",
      "baseline": "<what a pass looks like in this project>"
    },
    {
      "cmd": "<test command>",
      "baseline": "<known failures that are not yours to fix>"
    }
  ],
  "maxFixAttempts": 3
}

The fix limit matters as much as the baseline. A new failure gets up to three attempts, and after that the job stops and writes up what it tried. There is nobody awake to interrupt an agent that has decided to keep going.

Reading the skips

A suite that skips what it cannot run reports the same green as one that passed. So the job is told to read what each command skipped, not just its exit code — a tier that skipped because its data was missing means the run is unverified, not clean.

Where it is now

The pipeline is built, running on a schedule, and has had work come out of it that I have read and merged.

Three projects are enrolled. The scheduled job runs unattended, builds the tickets I have approved into draft pull requests, and several of those have now been reviewed and merged — including changes to the pipeline's own tooling, which it built for itself. The question I could not answer when I first wrote this page — whether the work coming back is good enough to accept — has a provisional answer, and it is yes, on small well-specified tickets.

What I still cannot claim is that it holds at a harder size. Every ticket it has finished has been a few files wide with criteria I could have checked myself in an afternoon. The failure I would expect first is not bad code but a confident report about a change nobody understood well enough to specify, and I have not yet given it one of those.

Honest limitations

The gates now go further than they did. This site's checks drive a real browser and can answer whether a page overflows, renders in every theme, survives with scripting off, prints, and meets contrast — so the criteria that used to come back as "have a look before merging" are largely answered. What no gate settles is whether a design is any good, or whether a game feels right. Taste is still a person's job, and a ticket that turns on it should say so rather than let a green run stand in for an opinion.

Screenshot of a draft pull request opened by the nightly job — to add