Coding agents made CI the next bottleneck

Coding agents are helping us fix more issues and create more pull requests.

Then they exposed the next bottleneck: GitHub Actions.

Every new pull request starts another round of builds, tests, linting, and checks. On our current plan, we receive 2,000 included Actions minutes each month. We are now running through them. The limit is only part of the problem. The feedback loop itself feels slow. The code may be ready, but the pull request is still waiting for CI.

This is what happens when one stage of a system gets dramatically faster. The constraint does not disappear. It moves.

Code throughput is not delivery throughput

The useful unit is not how quickly an agent writes code. It is how quickly a change travels through the whole delivery system:

task → code → pull request → CI → review → merge

Coding agents have compressed the first part of that path. They can investigate an issue, make a change, run local checks, and open a pull request faster than we could before.

But every additional pull request creates downstream work. The checks still need compute. Dependencies still need to be restored. Tests still need to run. Artifacts still need to be uploaded. Reviewers still need a trustworthy signal before they merge.

If validation capacity does not grow with code throughput, the queue simply moves downstream.

Fast code generation with slow validation is not fast delivery.

Self-hosting moved the constraint again

We tried a self-hosted runner. It reduced our dependence on hosted minutes, but it did not give us infinite capacity. It moved the constraint onto the hardware we own.

That trade can still make sense. A self-hosted runner gives a team more control over compute, images, networking, and the security boundary. But someone also owns the machine, capacity planning, updates, failures, and scaling. A fixed runner that was adequate for human-paced development can become a queue when several agents create work in parallel.

The question changes from “How many hosted minutes are we using?” to “How much hardware and operational attention are we willing to own?”

The managed-runner category is becoming more interesting

We are currently trying Blacksmith. We may test Depot and other hosted-runner options later.

This is not a recommendation. It is too early for one.

The interesting part is the category itself. These companies are competing on more than raw CPU. They talk about concurrency, cache locality, Docker-layer reuse, networking, observability, and the work required to manage runners. Depot’s homepage even frames the build pipeline as the place where the velocity from AI can disappear while waiting for CI.

Those are vendor claims and vendor framing. Headline multipliers are not directly comparable. A warm Docker build on one repository says little about an uncached mobile build on another. Runner size, language, architecture, cache warmth, test sharding, and the baseline all change the result.

The only useful comparison is your workflow on your repository.

Measure the wait before buying faster compute

Changing the runner provider is only one possible intervention. Faster hardware can make an inefficient workflow complete sooner without removing unnecessary work.

Before migrating, I want to separate the run into parts:

• Time waiting for a runner
• Checkout and environment setup
• Dependency and cache restore
• Test and build execution
• Artifact upload
• Total billed minutes and cost
• Failure rate and rerun frequency
• Maintenance work outside the workflow

That breakdown matters because each delay has a different fix.

Queue time may point to concurrency or capacity. Repeated dependency downloads may point to cache design. Obsolete runs may be removable with GitHub Actions concurrency groups and cancel-in-progress. A job triggered for changes it does not inspect may need better path filtering. A long test phase may need sharding or a more deliberate split between fast pull-request checks and deeper post-merge validation.

Buying faster compute before measuring the pipeline risks paying to run avoidable work faster.

What I want the trial to answer

The trial should not end with “Blacksmith felt faster.” It should answer a small set of operational questions:

1. How much did feedback time change for representative pull requests?
2. Which part changed: queueing, setup, caching, or execution?
3. Did total billed compute fall, or only wall-clock time?
4. Did higher concurrency improve flow or merely increase simultaneous spend?
5. How much runner maintenance disappeared?
6. What security and data-boundary trade-offs did we introduce?

The result might be a hosted runner. It might be a better self-hosted setup. It might be a smaller workflow with fewer redundant runs. More likely, it will be a combination.

Agent productivity is a systems problem

This is the broader lesson I am taking from it.

An agent can increase output at one stage without improving the performance of the system around it. More code can mean more validation work. More pull requests can mean more review work. More parallelism can mean a larger queue at the next serial checkpoint.

So the useful productivity question is not “How much code did the agents write?”

It is “How quickly did a trustworthy change reach users?”

As coding agents become normal, CI capacity, cache design, workflow observability, and runner economics become part of the agent-adoption conversation. The model may produce the change. The rest of the delivery system still has to absorb it.

I am still curious how other teams are handling this: optimising existing workflows, scaling self-hosted runners, moving to managed providers, or redesigning which checks run at which stage.

References

• Product usage included with each GitHub plan, GitHub Docs
• GitHub Actions billing, GitHub Docs
• Concurrency in GitHub Actions, GitHub Docs
• Dependency caching, GitHub Docs
• Blacksmith runner overview, vendor documentation
• Depot GitHub Actions runners, vendor documentation