BlueGit

timfa bluegit

Files

bluegit / docs / runner.en.md

runner.en.md

runner.en.md 9.6 KiB Download Raw

bluegit-runner — separate CI runner

bluegit-runner runs BlueGit CI jobs on your own machine, in Docker, just like the built-in runner. It only makes outgoing HTTPS connections to BlueGit (long-poll); no port has to be opened. The built-in BlueGit runner stays available.

Network

  • HTTPS (port 443) to the BlueGit server for heartbeat, jobs, logs and artifacts;
  • internet access for whatever the jobs fetch (Docker images, packages).

No inbound ports are needed on the runner machine, also not behind NAT or a home router. A proxy works through HTTPS_PROXY in the service environment. The runner's own web interface listens on 127.0.0.1 only by default.

Install (Arch / AUR)

yay -S bluegit-runner

The package creates the system user bluegit-runner (member of docker), installs and enables bluegit-runner.service. The service starts once /etc/bluegit-runner/config.yml exists; an upgrade restarts it.

Install elsewhere (binary + systemd)

git clone https://git.timfalken.com/timfa/bluegit.git && cd bluegit
go build -trimpath -ldflags "-X main.version=$(date +%Y%m%d%H%M%S)" -o bluegit-runner ./cmd/bluegit-runner
sudo install -Dm755 bluegit-runner /usr/bin/bluegit-runner
sudo install -Dm644 packaging/runner/bluegit-runner.service /usr/lib/systemd/system/bluegit-runner.service
sudo install -Dm644 packaging/runner/sysusers.conf /usr/lib/sysusers.d/bluegit-runner.conf
sudo install -Dm644 packaging/runner/tmpfiles.conf /usr/lib/tmpfiles.d/bluegit-runner.conf
sudo systemd-sysusers && sudo systemd-tmpfiles --create
sudo usermod -aG docker bluegit-runner
sudo systemctl daemon-reload && sudo systemctl enable bluegit-runner

Without systemd: bluegit-runner run --config path/to/config.yml as a user that may use Docker.

Register

  • 1. In BlueGit: Settings → Runners → New registration token (account runner) or Group → Settings → Runners (group runner, maintainers). The token (bgrr_…) is valid 24 hours, single use, shown once.
  • 2. On the machine: sudo bluegit-runner register --url https://git.example.com --token bgrr_... [--name my-pc]. The runner gets its own token (bgr_…); the server stores only its hash. register starts the service and prints how to open the web interface.

Config

/etc/bluegit-runner/config.yml: url, token, name, docker, git, work_dir (/var/lib/bluegit-runner), network (bridge), concurrency (start value), ui_listen (127.0.0.1:8765, off disables the UI) and language (en, nl, de, fr). Settings changed in the web interface are stored in work_dir/settings.json.

Web interface

Every runner has its own admin screen in BlueGit style, in English, Dutch, German and French.

  • Address: http://127.0.0.1:8765 by default. Change ui_listen or use an SSH tunnel: ssh -L 8765:127.0.0.1:8765 runner-host.
  • Access: an admin token (bgru_…) created locally on first start in work_dir/ui-token (mode 0600). It is never logged; show it on the machine with sudo bluegit-runner ui-token.
  • Contents: status (online, version, server, last contact), hardware with ★ recommendations, active and recent jobs, a safe to upgrade indicator, Pause (drain) / Resume, all settings, image presets and a docs page at /docs.
settingdefault
namehostnameshown in BlueGit and job logs; synced to the server on save and every heartbeat
labels—your own labels
jobs at the same timeconcurrency1–64, live
maximum cached images20least recently used images above this are removed
maximum job duration—caps the pipeline timeout
CPUs / memory / processes per jobno limitdocker run --cpus, --memory, --pids-limit
disk budgetno limitimages, checkouts and cache together
pull policypull when missingalways or never; presets are always built locally
remove work dir after jobonoff keeps work_dir/work/<job> for debugging
remove dangling imagesondocker image prune -f after each job
maximum job cache10 GBoldest cache directories first

Recommendations assume ~2 cores and ~2 GB per job, keep 1 core and 1 GB for the host and ~10 GB disk per job.

Image presets work like on Steve, the built-in runner: image: bluegit/node is a Dockerfile built locally on first use and reused while it stays the same. Add your own as a fixed image, a Dockerfile (optionally with another FROM), or only another base image for a built-in preset.

Pause (drain) lets running jobs finish and picks up nothing new. The heartbeat reports draining; BlueGit shows paused on the runner (API status: "draining") and routes jobs to the next runner or keeps them queued.

Local API (Bearer admin token): GET /api/status, GET/PUT /api/settings, GET /api/jobs, POST /api/pause, POST /api/resume.

T=$(sudo cat /var/lib/bluegit-runner/ui-token); U=http://127.0.0.1:8765
curl -fsS -X POST -H "Authorization: Bearer $T" $U/api/pause
until curl -fsS -H "Authorization: Bearer $T" $U/api/status | grep -q '"safe_to_upgrade":true'; do sleep 10; done
sudo pacman -Syu bluegit-runner
curl -fsS -X POST -H "Authorization: Bearer $T" $U/api/resume

Built-in runner

Site administrators get the same screen for the runner inside BlueGit at /admin/runner, and the API /api/v1/admin/runner/{status,settings,jobs,pause,resume} (admin token, scope runners or api; others get 403). Settings live in runner-settings.json in the state directory. The self-update only waits for jobs that are actually running, so a paused built-in runner does not block an update; queued jobs start after resuming.

Git and API from a job (CI_JOB_TOKEN)

A job can reach its own repo without its own login, also when the repo is private. origin points to CI_PROJECT_URL.git, and the runner sets http.<project url>.extraHeader (Authorization: Bearer $CI_JOB_TOKEN) through GIT_CONFIG_* environment variables, only for this project's URL. No file is written, so it works in any image (including bluegit/base) with git ≥ 2.31. git ls-remote origin, git fetch and git clone work, as do GET API routes for the own repo with the job token (read-only). Pushing, other repos and any use after the job ends are refused.

Protected branches

A job on a protected branch receives protected secrets. It only runs on the built-in runner, global runners, group runners and account runners of users who are maintainer (or owner) of that repo, or instance admin. Opt-in runners of regular members/developers are skipped for such jobs: the job falls through to the next allowed tier (eventually the built-in runner), never to a disallowed runner, and the job log names the skipped runners. This is checked at every assignment (also on retry), so revoked rights apply immediately. Unprotected branches are unchanged.

Global runners

Instance admins register runners under Admin → Runner → Global runners (or POST /api/v1/admin/runners/registration-token, bg admin runner token). A global runner serves every repository, public and private, like the built-in runner, and comes first in the order: global runners → group/account runners and the built-in runner in the usual order. Offline or paused global runners are skipped. Manage them in the same tab, via /api/v1/admin/runners[/{id}] or bg admin runner …; non-admins get 403.

Security: a global runner receives the code and the secrets of every job it runs, from any repository. Only that job's secrets are sent, and the runner token can only clone the repo of a job it is running. Only register machines you fully trust; the UI, API (warning) and bg show this warning when registering.

Security and isolation

  • Every job runs in its own Docker container with the checkout on /workspace; services get a network per job. Per-job limits are set in the web interface.
  • Docker access is root-equivalent on that machine; account runners only get jobs of their owner's repositories, group runners only of their group.
  • The runner token is only in config.yml (0600); the server keeps a hash. It may only clone the repository of a job the runner is running right now.
  • Job secrets are masked in logs. The service runs with NoNewPrivileges, ProtectSystem=strict, ProtectHome and PrivateTmp.

Online, offline and lost runners

Heartbeat every 30 s; online means contact in the last 90 s. If a runner disappears during a job, BlueGit fails the job with runner_system_failure after 5 minutes without contact; retry rules apply.

Remove or replace

Revoke the runner in BlueGit (the token stops working immediately), then sudo systemctl disable --now bluegit-runner and remove the package. To replace a machine: register the new one, revoke the old one, and copy settings.json or use PUT /api/settings.

Troubleshooting

SymptomCause and fix
stays offlinejournalctl -u bluegit-runner -f; test curl -I https://<server>/healthz (DNS, proxy, outgoing 443).
401 from the serverrunner revoked: register again.
registration failed (403)registration token expired or used.
runner_system_failure: docker … is not availablesudo -u bluegit-runner docker info must work (group docker, docker.service).
job stays queuedrunner paused (locally or in BlueGit), all slots busy, or the runner does not serve this repository.
disk fulllower max images and job cache, enable dangling-image cleanup.
web interface does not opencheck ui_listen and ss -ltnp; lost token: sudo bluegit-runner ui-token.

Runner management through the BlueGit API for agents: agents.md.

Source
1# bluegit-runner — separate CI runner
2
3`bluegit-runner` runs BlueGit CI jobs on your own machine, in Docker, just
4like the built-in runner. It only makes **outgoing** HTTPS connections to
5BlueGit (long-poll); no port has to be opened. The built-in BlueGit runner
6stays available.
7
8## Network
9
10- HTTPS (port 443) to the BlueGit server for heartbeat, jobs, logs and artifacts;
11- internet access for whatever the jobs fetch (Docker images, packages).
12
13**No inbound ports** are needed on the runner machine, also not behind NAT
14or a home router. A proxy works through `HTTPS_PROXY` in the service
15environment. The runner's own web interface listens on `127.0.0.1` only by default.
16
17## Install (Arch / AUR)
18
19```sh
20yay -S bluegit-runner
21```
22
23The package creates the system user `bluegit-runner` (member of `docker`),
24installs and enables `bluegit-runner.service`. The service starts once
25`/etc/bluegit-runner/config.yml` exists; an upgrade restarts it.
26
27## Install elsewhere (binary + systemd)
28
29```sh
30git clone https://git.timfalken.com/timfa/bluegit.git && cd bluegit
31go build -trimpath -ldflags "-X main.version=$(date +%Y%m%d%H%M%S)" -o bluegit-runner ./cmd/bluegit-runner
32sudo install -Dm755 bluegit-runner /usr/bin/bluegit-runner
33sudo install -Dm644 packaging/runner/bluegit-runner.service /usr/lib/systemd/system/bluegit-runner.service
34sudo install -Dm644 packaging/runner/sysusers.conf /usr/lib/sysusers.d/bluegit-runner.conf
35sudo install -Dm644 packaging/runner/tmpfiles.conf /usr/lib/tmpfiles.d/bluegit-runner.conf
36sudo systemd-sysusers && sudo systemd-tmpfiles --create
37sudo usermod -aG docker bluegit-runner
38sudo systemctl daemon-reload && sudo systemctl enable bluegit-runner
39```
40
41Without systemd: `bluegit-runner run --config path/to/config.yml` as a user that may use Docker.
42
43## Register
44
45- **1.** In BlueGit: **Settings → Runners → New registration token** (account runner) or **Group → Settings → Runners** (group runner, maintainers). The token (`bgrr_…`) is valid 24 hours, single use, shown once.
46- **2.** On the machine: `sudo bluegit-runner register --url https://git.example.com --token bgrr_... [--name my-pc]`. The runner gets its own token (`bgr_…`); the server stores only its hash. `register` starts the service and prints how to open the web interface.
47
48## Config
49
50`/etc/bluegit-runner/config.yml`: `url`, `token`, `name`, `docker`, `git`,
51`work_dir` (`/var/lib/bluegit-runner`), `network` (`bridge`), `concurrency`
52(start value), `ui_listen` (`127.0.0.1:8765`, `off` disables the UI) and
53`language` (`en`, `nl`, `de`, `fr`). Settings changed in the web interface
54are stored in `work_dir/settings.json`.
55
56## Web interface
57
58Every runner has its own admin screen in BlueGit style, in English, Dutch,
59German and French.
60
61- **Address**: `http://127.0.0.1:8765` by default. Change `ui_listen` or use an SSH tunnel: `ssh -L 8765:127.0.0.1:8765 runner-host`.
62- **Access**: an admin token (`bgru_…`) created locally on first start in `work_dir/ui-token` (mode 0600). It is never logged; show it on the machine with `sudo bluegit-runner ui-token`.
63- **Contents**: status (online, version, server, last contact), hardware with ★ recommendations, active and recent jobs, a *safe to upgrade* indicator, **Pause (drain)** / **Resume**, all settings, image presets and a docs page at `/docs`.
64
65| setting | default | |
66|---|---|---|
67| name | hostname | shown in BlueGit and job logs; synced to the server on save and every heartbeat |
68| labels | — | your own labels |
69| jobs at the same time | `concurrency` | 1–64, live |
70| maximum cached images | 20 | least recently used images above this are removed |
71| maximum job duration | — | caps the pipeline timeout |
72| CPUs / memory / processes per job | no limit | `docker run --cpus`, `--memory`, `--pids-limit` |
73| disk budget | no limit | images, checkouts and cache together |
74| pull policy | pull when missing | `always` or `never`; presets are always built locally |
75| remove work dir after job | on | off keeps `work_dir/work/<job>` for debugging |
76| remove dangling images | on | `docker image prune -f` after each job |
77| maximum job cache | 10 GB | oldest cache directories first |
78
79Recommendations assume ~2 cores and ~2 GB per job, keep 1 core and 1 GB
80for the host and ~10 GB disk per job.
81
82**Image presets** work like on Steve, the built-in runner: `image:
83bluegit/node` is a Dockerfile built locally on first use and reused while it
84stays the same. Add your own as a fixed image, a Dockerfile (optionally with
85another `FROM`), or only another base image for a built-in preset.
86
87**Pause (drain)** lets running jobs finish and picks up nothing new. The
88heartbeat reports `draining`; BlueGit shows *paused on the runner* (API
89`status: "draining"`) and routes jobs to the next runner or keeps them queued.
90
91**Local API** (Bearer admin token): `GET /api/status`, `GET/PUT /api/settings`,
92`GET /api/jobs`, `POST /api/pause`, `POST /api/resume`.
93
94```sh
95T=$(sudo cat /var/lib/bluegit-runner/ui-token); U=http://127.0.0.1:8765
96curl -fsS -X POST -H "Authorization: Bearer $T" $U/api/pause
97until curl -fsS -H "Authorization: Bearer $T" $U/api/status | grep -q '"safe_to_upgrade":true'; do sleep 10; done
98sudo pacman -Syu bluegit-runner
99curl -fsS -X POST -H "Authorization: Bearer $T" $U/api/resume
100```
101
102## Built-in runner
103
104Site administrators get the same screen for the runner inside BlueGit at
105**/admin/runner**, and the API `/api/v1/admin/runner/{status,settings,jobs,pause,resume}`
106(admin token, scope `runners` or `api`; others get `403`). Settings live in
107`runner-settings.json` in the state directory. The self-update only waits for
108jobs that are actually *running*, so a paused built-in runner does not block
109an update; queued jobs start after resuming.
110
111## Git and API from a job (`CI_JOB_TOKEN`)
112
113A job can reach its own repo without its own login, also when the repo is private. `origin` points to `CI_PROJECT_URL.git`, and the runner sets `http.<project url>.extraHeader` (`Authorization: Bearer $CI_JOB_TOKEN`) through `GIT_CONFIG_*` environment variables, only for this project's URL. No file is written, so it works in any image (including `bluegit/base`) with git ≥ 2.31. `git ls-remote origin`, `git fetch` and `git clone` work, as do `GET` API routes for the own repo with the job token (read-only). Pushing, other repos and any use after the job ends are refused.
114
115## Protected branches
116
117A job on a protected branch receives protected secrets. It only runs on the built-in runner, global runners, group runners and account runners of users who are maintainer (or owner) of that repo, or instance admin. Opt-in runners of regular members/developers are skipped for such jobs: the job falls through to the next allowed tier (eventually the built-in runner), never to a disallowed runner, and the job log names the skipped runners. This is checked at every assignment (also on retry), so revoked rights apply immediately. Unprotected branches are unchanged.
118
119## Global runners
120
121Instance admins register runners under **Admin → Runner → Global runners** (or `POST /api/v1/admin/runners/registration-token`, `bg admin runner token`). A global runner serves **every repository**, public and private, like the built-in runner, and comes **first** in the order: global runners → group/account runners and the built-in runner in the usual order. Offline or paused global runners are skipped. Manage them in the same tab, via `/api/v1/admin/runners[/{id}]` or `bg admin runner …`; non-admins get `403`.
122
123**Security**: a global runner receives the code and the secrets of every job it runs, from any repository. Only that job's secrets are sent, and the runner token can only clone the repo of a job it is running. Only register machines you fully trust; the UI, API (`warning`) and `bg` show this warning when registering.
124
125## Security and isolation
126
127- Every job runs in its own Docker container with the checkout on `/workspace`; services get a network per job. Per-job limits are set in the web interface.
128- Docker access is root-equivalent on that machine; account runners only get jobs of their owner's repositories, group runners only of their group.
129- The runner token is only in `config.yml` (0600); the server keeps a hash. It may only clone the repository of a job the runner is running right now.
130- Job secrets are masked in logs. The service runs with `NoNewPrivileges`, `ProtectSystem=strict`, `ProtectHome` and `PrivateTmp`.
131
132## Online, offline and lost runners
133
134Heartbeat every **30 s**; **online** means contact in the last **90 s**.
135If a runner disappears **during a job**, BlueGit fails the job with
136`runner_system_failure` after **5 minutes** without contact; retry rules apply.
137
138## Remove or replace
139
140Revoke the runner in BlueGit (the token stops working immediately), then
141`sudo systemctl disable --now bluegit-runner` and remove the package. To
142replace a machine: register the new one, revoke the old one, and copy
143`settings.json` or use `PUT /api/settings`.
144
145## Troubleshooting
146
147| Symptom | Cause and fix |
148|---|---|
149| stays *offline* | `journalctl -u bluegit-runner -f`; test `curl -I https://<server>/healthz` (DNS, proxy, outgoing 443). |
150| `401` from the server | runner revoked: register again. |
151| `registration failed (403)` | registration token expired or used. |
152| `runner_system_failure: docker … is not available` | `sudo -u bluegit-runner docker info` must work (group `docker`, `docker.service`). |
153| job stays *queued* | runner paused (locally or in BlueGit), all slots busy, or the runner does not serve this repository. |
154| disk full | lower max images and job cache, enable dangling-image cleanup. |
155| web interface does not open | check `ui_listen` and `ss -ltnp`; lost token: `sudo bluegit-runner ui-token`. |
156
157Runner management through the BlueGit API for agents: [agents.md](agents.md#runners).