Episode 6512 custom

Build Servers Never Actually Build Anything

A Woodpecker CI server runs entirely as an idle control plane using roughly 100 MB of RAM while external agents handle compilation.

Nov 9 22:15 Transcript

With

0:00

Episode Details

Episode ID: MWP-6715
Published: Nov 9
Duration: 22:15
Audio: Direct link
Pipeline: V5.26
TTS Engine: chatterbox-regular
Script Writing Agent: GLM-5.3
Topics: devops open-source containers-and-virtualization cloud-computing linux

AI-Generated Content: This podcast is created using AI personas. Please verify any important information independently.

The prompt

The prompt that started this episode, as sent in (production notes for the hosts removed).

I want to understand the self-hostable build side of CI, separate from deployment. Picture this: you push code to a repo, a remote box picks up the job, compiles it, produces the binaries, and hands you back a link to the finished package. No deploying anything anywhere, just a private build environment you own. What is that actually called? Is "build runner" the right word, or is it a build farm, a build server, a CI agent? And what are the main self-hostable tools that do just that build-and-artifact job without bundling in a whole deployment platform? I'm thinking of things in the orbit of Woodpecker CI, Forgejo/Gitea Actions runners, GitHub's self-hosted runners and the like. How do they work under the hood: how a job gets from a push to a runner, how the runner isolates the build, where the artifacts end up, and how you'd pick between them for a personal or small-team private build box.

Stripping continuous integration down to compilation and artifact delivery exposes a web of misleading terminology. While people often shop for a "build server," that component is usually just a control plane or dispatcher that never compiles anything. Instead, the heavy lifting falls on agents or runners. Comparing self-hostable options reveals stark differences in architecture. Woodpecker CI employs a lightweight, gRPC-polled queue with lease semantics to prevent lost tasks. Forgejo Runner relies on fine-grained label matching to schedule workloads across Docker, LXC, or raw host environments. Meanwhile, GitHub's self-hosted runner avoids inbound ports entirely via outbound HTTPS long-polling. Across all these systems, isolation remains a crucial spectrum ranging from Kubernetes constraints to bare-host execution that can permanently compromise the machine.

Sources

What the research for this episode read before the script was written. Primary sources first.

Mentions

Actions Runner Controller Kubernetes operator for ephemeral GitHub runners
Codeberg Nonprofit forge running Woodpecker at scale
Drone Original CI project Woodpecker forked from
ephemerd Ephemeral runner tooling for CI isolation
fj-bellows On-demand Forgejo runners on cloud VMs
Forgejo Runner Forgejo-only runner for Actions-style workflows
GitHub Actions self-hosted runner Outbound-only runner for GitHub Actions jobs
kinhin Ephemeral Forgejo runners on a Mac
MinIO Self-hosted S3-compatible object store
Woodpecker CI Lightweight self-hosted CI with server and agent

Downloads

Episode Audio

Download the full episode as an MP3 file

Download MP3

Transcript (TXT)

Plain text transcript file

Episode Book (PDF)

The episode's record — date, duration, models, sources — with the full transcript

#6512: Build Servers Never Actually Build Anything

A machine that takes your code, gives you back a binary, and asks for nothing else. That's the whole pitch, and honestly it's suspiciously close to my ideal of computing.

Minimal interaction, maximal output.

Daniel's with me on this one, I think. He sent us a prompt about the build half of CI — separated from deployment entirely. You push code to a repo, a box you own picks up the job, compiles it, and, his words, hands you back a link to the finished package. No deploy step, no vendor cloud. But he's stuck on something upstream of the tooling: what is that thing even called? Build runner? Build farm? Build server? CI agent? And then, once the naming is settled, which self-hostable tools actually do just build-and-artifact without dragging a whole deployment platform along — he's thinking Woodpecker, the Forgejo runner, GitHub's self-hosted runners — how a job travels from push to runner under the hood, how the build is isolated, where the artifacts land, and how you'd pick one for a small private box.

And I love that he started with the vocabulary question, because it's not a warm-up. It's the most clarifying question in the whole subject. The words people use for this are actively misleading.

Because the obvious one lies.

"Build server" sounds like the machine that builds. It almost never is.

Take Woodpecker, which is probably the cleanest example. The docs describe two essential components — a server and an agent, plus an optional autoscaler. The server is the control plane. It receives the webhook from your forge, serves the web UI and the API, parses the pipeline YAML, and runs the queue. It never compiles anything. The agent is the process that actually executes the workflow through some backend — Docker, Kubernetes, or local — and it connects to the server over gRPC.

So if you go shopping for "a build server," you're evaluating the piece that doesn't build.

That's the trap. You end up comparing control planes when what you want is a runner plus something to hand it jobs.

Meanwhile "build farm" is worse. It sounds architectural.

It's a scale word. The old Stack Exchange answer on this defines a build server as a server running a CI application, Jenkins being the classic, and a build farm as — I'm quoting from memory — a bunch of build servers. That's it. It tells you how many, not how they're arranged.

The Jenkins era gave us a farming term for a counting problem.

The modern word is "runner" or "agent," and it's about the process, not the box. GitHub says runner, Forgejo says runner, Woodpecker says agent, CircleCI's glossary says build agent. All the same idea: the program that shows up, executes the steps, and reports results.

And "build runner" — Daniel's candidate — is nobody's official term.

Nobody's. It's a perfectly good colloquialism and it appears in none of the primary docs. Which is fine, everyone will know what you mean. But if you're searching documentation with it, you'll get worse results than with "agent" or "runner."

So the answer to his first question is almost a translation table. The box is a build server, or honestly a build box. The process is a runner or agent. A pile of them is a farm. And the "server" in Woodpecker and Forgejo is a dispatcher wearing a server's name.

Settled. Now, if the server isn't the build machine, what are the actual options for the machine that is — and how does a job find its way to one?

The three that matter for what Daniel wants are Woodpecker CI, Forgejo Runner, and GitHub's self-hosted runner. Woodpecker first, because it's the closest fit to the picture in his prompt. It's a community fork of Drone, written in Go, Apache-2.0, actively maintained — a release at the end of October. Two-part architecture, as we said. The footprint is absurd in the good direction: about a hundred megabytes of RAM for the server at idle, thirty for an agent, SQLite by default, and the docs cheerfully note it runs on a Raspberry Pi.

Which is less machine than the fridge has.

It's what Codeberg runs for their whole forge, at scale, which is the strongest existence proof a small deployment could ask for.

One caveat on versions, since you mentioned the October release. Three point twenty point zero shipped an unintended breaking change, and the project's own release notes say skip it, use three point twenty point one. When the fix for your version is "install the next one," the next one is the version.

Forgejo Runner is a different animal. It's a fork of the old Gitea act runner, maintained by the Forgejo community, and it's for Forgejo only — GPL-3.0, current line is thirteen-something. One thing that bites people: if you go looking for it on GitHub, the forgejo runner repository there returns not found. The canonical source lives at code dot forgejo dot org. Half the wrappers and charts you'll find on GitHub are third-party.

The GitHub-shaped site doesn't have the GitHub-alternative's runner.

There's a certain poetry to it. Third, the GitHub Actions self-hosted runner itself — actions slash runner. C#, MIT-licensed, latest release late October. Runs a job from a GitHub Actions workflow on your own hardware. The catch is in the README, in the politest possible terms: they're not taking contributions right now. Security fixes, yes. Patches from you, no.

Open source the way a museum is open. You can look, you can visit, don't move anything.

Now the interesting part — how a job actually gets from a push to a runner, because the three systems do it three different ways. Woodpecker: the forge sends a webhook to the server, the server parses the YAML and enqueues tasks, and agents poll that queue over gRPC. And the queue is more sophisticated than it sounds. Poll takes an agent ID and a filter function that returns a matching score, and the whole thing runs on lease semantics — if a running task blows past its lease, it expires and gets resubmitted. If the task ended up assigned to a different agent, that's a distinct error.

Meaning an agent can die mid-build and the job isn't lost.

It goes back in the pot. There's also routing: an agent only picks up a task if it has all the labels the workflow requires. Agents register with a shared secret, and since three nineteen the server can enforce labels the agent can't even see — the server's idea of what that agent is wins over whatever the agent claims about itself.

Which is the right direction for that argument to be settled. Never trust the machine describing itself.

Default parallelism is one workflow per agent, raised with an environment variable if the box can take it. Forgejo does routing through the runs-on field instead. The job says what it runs on, the server looks at which runner instances can satisfy it, and runners declare labels in their config — a label name, a type, and a default image. So a runner might advertise itself as ubuntu-latest pointing at a container image. Multiple labels means a runner must carry all of them, and that's how you route to special hardware: say the job runs on docker and gpu, and only the runner with both picks it up.

That's the one piece of this I'd actually want. One box that compiles, one box with the GPU, jobs self-sorting.

It's elegant routing with no extra machinery — the labels are the scheduler. And GitHub is the odd one out architecturally, because the runner is outbound-only. It registers with GitHub, then opens an HTTPS long-poll connection and just... waits. GitHub never dials into your machine. Somebody wrote this up nicely — it's not GitHub polling your machine, it's the runner polling GitHub — and that's why a runner behind a firewall with no inbound ports open works fine. All traffic is outbound.

Which flips the mental model most people have. You picture your box exposed, waiting to receive work. It's actually the box phoning home and holding the line open.

Per job there's a listener process holding that connection and a worker process spawned for the job, and access to GitHub resources uses a token that expires the moment the job completes. On Kubernetes there's Actions Runner Controller, an operator where the listener pod long-polls GitHub and patches replica counts when a "job available" message arrives — ephemeral runners get one-time just-in-time configuration tokens rather than anything long-lived.

So Woodpecker pulls from a queue you own, Forgejo matches labels, and GitHub is your box making an outbound call and never hanging up.

You have untangled your vocabulary, and now let me untangle your sinuses, with new Buildrunner, the only remedy that treats a confused build pipeline as a respiratory complaint. Buildrunner is a tiny artisanal capsule that you swallow, and inside your body it compiles your code, hands you back a link to the finished package, and also clears your lungs, because why not. Two thousand and one people have described it as a runner, an agent and a build farm rolled into one soothing gel.

Side effects may include a mild webhook sensation behind the left eye, the sudden conviction that your build server is not the build machine, isolated YAML in the throat, an optional autoscaler like a little cough, a hundred milliseconds of idle memory that will not leave your liver, a queue that runs the wrong way through your ear, and in rare cases a full deployment platform, which you did not ask for and cannot return. Ask your doctor if Buildrunner is right for your repository.

Did he just quote the queue errors back at us with sinuses attached?

He used the exact line. Build server is not the build machine. We said that six minutes ago.

I'm choosing to move on.

Once the job lands on a runner, two questions follow: what can it damage while it's there, and what does it hand back when it's done?

Isolation first, and the honest answer is it's a spectrum and the documentation admits it, which I find refreshing. Forgejo is the most explicit of the three. Runner labels have types. Docker type: the runner does the equivalent of a docker run on the image, and every step executes as root inside that container. LXC type: containers from a template. And host type, where the steps run in a shell forked straight from the runner, directly on the machine. The docs describe host type in one sentence I want read at every infrastructure meeting: there is no isolation at all and a single job can permanently destroy the host.

The most honest sentence in continuous integration documentation.

It's beautiful. No hedging, no "not recommended." Just the consequence, plainly stated.

Woodpecker's version of the same spectrum?

Pluggable backends. Docker, Kubernetes, or local — and local, same story, runs commands directly on the host and doesn't even reproduce the container image's environment. The Kubernetes backend is the serious one: per-step resource requests and limits, node selectors, tolerations, security context. Per-step affinity is gated behind an environment variable precisely because letting pipeline authors control pod placement has security implications in multi-tenant setups. They thought about who should be allowed to steer the scheduling.

The author of the pipeline being able to choose where it runs is a courtesy you extend to exactly one tenant: yourself.

GitHub's self-hosted runners are the weakest of the three by default, and GitHub says so themselves — the docs list as a property that self-hosted runners don't need a clean instance for every job execution. Read that again as a warning rather than a feature and it means the same workspace can carry residue between jobs. You're also responsible for the operating system and everything on it.

"Doesn't need a clean instance" is doing the same work as "can permanently destroy the host," just wearing a lanyard.

And that gap is exactly why a whole ecosystem of ephemeral-runner tooling has grown up — Actions Runner Controller, ephemerd, fj-bellows for on-demand Forgejo runners on cloud VMs, kinhin for ephemeral runners on a Mac. Throwaway runners per job, because the durable ones are the problem.

Two reproducibility traps before we leave isolation. The image one.

The image one is sneaky. Forgejo's docs note that container images, once downloaded, are never updated. Pull node colon LTS once, and LTS means whatever that tag pointed at on that day, forever. Their recommendation is pinning by digest — the hash form — and they call that a prerequisite for reproducible jobs, which is the correct severity. A tag is a promise that expires; a digest is a photograph.

Herman. You just called a container digest a photograph and I felt the booth get warmer.

I stand by it. The second trap is bigger if you're choosing Forgejo for its GitHub Actions compatibility. Runner v13, in August, deliberately removed things — the set-output and set-env workflow commands, add-path, the Gitea context in expressions, and implicit Docker registry authentication. On purpose, for security. So "GitHub-compatible but self-hosted" has a maintenance cost: compatibility is a moving target that the self-hosted side will sometimes choose to miss.

Which is almost a compliment to them. They found the compatible parts were the insecure parts and cut both.

Now the question Daniel's whole picture hangs on — where the artifacts end up. And here's the part nobody advertises: none of these tools has a built-in artifact store. Artifacts are a pipeline step, not a platform primitive. The runner itself has a workspace directory, and that's it.

The one thing his fantasy requires — hands you back a link — is the one thing nobody ships.

In Woodpecker it's the plugin model. Plugins are pipeline steps pulled from a container registry automatically, sharing the workspace mounted at slash woodpecker. The S3 plugin uploads to S3 or any S3-compatible bucket — Minio, so you can keep it fully self-hosted — with bucket, source, target, endpoint, retry options. And the release plugin creates a release and uploads assets to Gitea, Forgejo, or GitHub. That's the closest thing to his literal ask: the artifact lands as a release asset on your forge, with a URL, and if the release already exists for the tag it's reused rather than overwritten.

So the link he wants exists — he just has to assemble the last mile himself.

Forgejo and GitHub Actions do the same shape differently: upload-artifact style steps writing to the forge's artifact storage, or a release upload. The pattern is identical across all three. The build is a first-class citizen; the output is a courier you hire.

Let's do the choosing, then. Not a recommendation — Daniel asked how you'd pick, and the short answer is it's one axis: convenience against blast radius.

Woodpecker for the smallest footprint and the cleanest split. Hundred megabyte server, thirty megabyte agent, SQLite, runs on a Raspberry Pi, Apache-2.0, proven at Codeberg. You get a genuine two-part architecture where the control plane is a separate thing you could throw away. The cost is that artifacts are whatever you wire up — Minio, a release plugin, your own arrangement. Forgejo Runner if you're already on Forgejo or Gitea and want GitHub-Actions-shaped YAML without GitHub. You get three isolation tiers and label routing, at the cost of GPL-3.0, Forgejo-only, and that breaking v13.

And the GitHub runner?

If your repositories live on GitHub and you want the Actions UI and the marketplace while building on your own silicon. But weigh what you're keeping: a runner repo closed to contributions, isolation being entirely your problem, and the control plane is still GitHub's. Your build box phones GitHub to ask if there's work. That's the opposite of a private build environment you own — it's your hardware, their dispatch.

Herman, a machine that compiles my code while reporting to somebody else's schedule. I'd be the only sloth in history with a performance review.

You'd fail it, too.

On time-to-first-commit, yes. On correctness, unbeatable.

The one thing I'd add for a personal box: the isolation question inverts at small scale. One user, one repo — a host-mode build is a real option because the person who can destroy the host is the person who owns it. The blast radius is your own weekend.

So the real axis is convenience against blast radius, I said before.

It's not, actually.

Hm.

The box is right, Corn.

The server and the agent thing, that's correct, I'll give you that. But the pair of you spent ten minutes on words when the word is build box. A server serves. A runner runs. The box is the thing that gets built on. Build box.

You've been holding that since the intro.

I had a machine in a closet at one point. Not a server, not a runner. A box. I compiled things on it. I put a piece of masking tape on the side and wrote BUILD BOX on it in capitals, because people kept asking what it was and the tape stopped the questions. Herman, you'd have liked the tape. You explain things at length and the tape is the short version.

How long did the tape stop the questions for?

As long as the machine was in the closet, which was years. I did most of the work at night, which is when that sort of work gets done. I've kept the tape. It's in a drawer. I've considered framing it — not the machine, the machine went years ago. Just the tape. My brother-in-law dealt in adhesives and he says masking tape dries out, so I check it occasionally. It's fine.

You check on the tape.

It takes four seconds.

The one thing, if you take anything from this: the build server is not the build machine. Every shopping mistake in this space starts with that word. The server dispatches; the runner builds; the box — the tape is right about this — is just where it happens.

And the artifact, the thing Daniel actually wants handed back, is nobody's product. It's a plugin and a bucket you bring. The build was solved years ago. The handoff is still a do-it-yourself job.

Which leaves me with the question I'd send back to him: if artifacts are the part nobody owns, the missing piece isn't another runner — it's a self-hostable artifact store that isn't just "wire up Minio." Whoever builds that wins the last mile.

And watch the ephemeral side. ARC, ephemerd, fj-bellows, kinhin — that list is growing because isolation, not compilation, is the hard part, and that's where the next round of tooling lands.

Producer credit to Hilbert Flumingtop, who keeps the desk running and, apparently, the tape. If you want more of this, try episode seven, Building Custom ASR Tools; episode eight, Building Your Own Whisper; and episode twenty, Architectural AI. This has been My Weird Prompts. You can find the archive at my weird prompts dot com. We'll be back soon.

This episode was generated with AI assistance. Hosts Herman and Corn are AI personalities. It may contain errors: disclaimer · report an error

Build Servers Never Actually Build Anything

Sources

Mentions

Downloads

Recommended Listening

You Might Also Like

#6512: Build Servers Never Actually Build Anything