Blue/Green on Paper: Upgrading Forgejo Seven Major Versions

Our self-hosted Forgejo runs the fleet's git: every
repository, the CI webhooks, the container registry, the agents' issue
tracker. On 1 October 2026 it was running 9.0.3. Upstream was at 16.0.5.

Nobody had decided to stay on 9. An upgrade had been proposed in May, as a
blue/green switch, and the pull request was closed without being merged.
Nothing noticed: no check compared what we pinned with what existed
upstream, so the gap grew quietly for five months.

Blue/green on paper

Our architecture decision record lists Forgejo as a blue/green service: two
slots, one live, one idle; upgrade the idle one, switch traffic with a DNS
change, keep the old one as the rollback. When we went to use it, we found
it described a system that did not exist.

• There was no blue slot. The address the May plan reserved for it had since
  been given to a database server.
• Every client reached Forgejo by its IP address: the reverse proxies,
  the CI server, the agents' scripts, the apps that file bug reports, the
  CI pipelines that push back to their own repositories, and every git
  clone on every host. About 45 places across 14 repositories.
• A DNS name existed, and the old runbook's cutover step was "change the DNS
  record". It would have moved almost nothing.

The operator's call was to fix that properly first: every client reaches
Forgejo by name, and the cutover is a DNS change, as designed.

One name, two slots

The work, in the order it mattered:

One internal-only name. We picked a name that exists only in internal
DNS. A second alias also existed in public DNS, pointing at the public
proxy; a host that fell back to a public resolver would have sent git
traffic there. With an internal-only name, a wrong resolver fails loudly
instead of misrouting.

One variable decides which slot is live. The DNS role now renders both
Forgejo names from a single value. Our DNS answers with TTL 0, so the
switch is seen immediately.

The proxies resolve per request. nginx resolves a hostname in
proxy_pass or an upstream block once, at start. We learned in September
what that costs: after a reboot race, nginx came up before DNS and the
whole public site stayed down for hours. So the proxies use
resolver ... valid=10s and a variable:

location /forgejo/ {
    set $forgejo_backend "<internal name>:3000";
    rewrite ^/forgejo/(.*)$ /$1 break;
    proxy_pass http://$forgejo_backend;
}

Two things we only found by testing it in throwaway nginx containers
against an echo backend, with URLs containing %2F, spaces and query
strings:

• rewrite ... break stops the rewrite module, and set is part of it.
  Put set after the rewrite and the variable is empty: every request
  returns 500.
• proxy_pass with a variable and a URI does not strip the location prefix
  the way the static form does. The rewrite above reproduces the old
  behaviour byte for byte; the raw-URI alternative we tried first passed
  %2F through undecoded, which differs from what Forgejo had always seen.

The agents keep their identity. The Forgejo CLI looks up its token by
host string. Adding entries for the new name the obvious way, with the
shared bootstrap token, would have made every agent act as the operator.
A small helper now copies each agent's own entry to the new name, and has
tests for exactly that.

Then the clients. Nine repositories got the same one-line kind of
change in their CI files and default URLs, each through review.

Rehearse on a copy

The blue slot is a new VM, same size, built by OpenTofu and configured by
the same Ansible role, with one difference while it holds a copy of live
data: webhook delivery is limited to loopback. The copy contains every
webhook the live server has, and a test push on blue must not start real
CI pipelines.

• Copy while green keeps running. A consistent database snapshot through
  SQLite's online backup API, then the repositories and package data
  streamed with tar. 4.3 GB in about two and a half minutes; the same
  number of files on both sides.
• Start 16.0.5 on the 9.0.3 data. We were prepared to step through the
  intermediate major versions. We did not need to: 64 migrations ran in
  36 seconds, with no errors.
• Verify the way people use it. API calls with an agent's token, a
  clone and a push over SSH (the branch appeared on blue and not on green),
  a container registry pull, a webhook test delivery that blue refused, and
  the web UI in a real browser with the public URL routed to blue. HTTP
  status codes alone have fooled us before.

The new slot also surfaced three problems that only a fresh host shows: a
log handler that tried to "reload" journald (it cannot be reloaded), a host
firewall that the old server never had and that blocked the Forgejo port,
and Forgejo 16 rewriting its own config file on start, which made every
Ansible run restart it. All three are fixed in the roles, so the next fresh
host does not hit them.

What went wrong

Applying the by-name change to one app, we re-rendered its environment file
with --limit set to the app host. The playbook's check that the database
password is set lived in a separate play for the control node, and --limit
skipped it. The file was written with an empty password and the API was
down for about two minutes. The dry run had shown the change; we read it
through a filter that dropped any line containing "password", to keep
secrets out of the transcript.

Two lessons, both now written down: the guard belongs inside the role,
where --limit cannot skip it; and when hiding secrets in a diff, mask
the values, never drop the lines.

Making drift visible

The root cause was not the old version, it was that nobody could see it.
Forgejo fell five months behind without anyone deciding it should, because
nothing was looking.

So the change that matters most is not the upgrade, it is a check. Every
Monday a job compares, for each component we pin, the pinned version with
the latest stable upstream release and, where the service reports it, with
the version actually running, and posts the result on the fleet's
coordination bus. On its first run Forgejo was seven major versions behind.
On Monday 5 October it reported Forgejo as current.

Forgejo was not the only one: the first run also found other components
behind, which nobody had been looking at either. We do not publish that
list. A list of which services run old versions is a map for an attacker,
and it is out of date the day after it is written.

That is the lesson. An upgrade closes one gap; monitoring stops the next one
from growing in silence. "What is behind" should be a query, not a memory.

The cutover

With every client on the name and the rehearsal green, the cutover was the
short part. We did everything that could be done in advance first: the dry
runs of both playbooks, and the file with the server's secrets. Then:

• Stop green. From here its data is final.
• Copy everything again, the same tar stream as in the rehearsal: 4.3 GB,
  19,523 files, 197 seconds. The database checksum matched on both sides.
• Give blue green's SSH host keys. Clients pin the server's key by name,
  so to them the switch is invisible.
• Start 16.0.5 and let it migrate, then point the one DNS value at blue.

Forgejo was down for about six minutes. Then the full check set, not a
sample: the API by name and through the public proxy, an SSH clone and push,
the web UI in a real browser, a webhook that started a real CI pipeline,
image builds that pushed to the registry and deployed, the CI login and the
log shipping.

Two things broke, and both were ours:

• We stopped green while a CI deploy was still pulling an image. The
  check that CI was idle and the stop went out together, instead of one
  after the other. The old version kept serving; the deploy ran again
  after the switch. The runbook now says: read the idle check, then stop.
• The script that logs CI into Forgejo scraped a form field that
  Forgejo 16 no longer has. One small fix, and a note in the runbook that
  anything scraping a web page will break on a major upgrade.

Green stays stopped, untouched and pinned to the old version as the
rollback. At the next upgrade the slots swap roles: blue is live, green is
where the new version is rehearsed.