BlueGit

timfa bluegit

feat: breid rules en de ingebouwde runner uit

Ondersteun &&, ||, haakjes, regex en null in rules:if, plus workflow,
retry, parallel, services, environment en interruptible. Een push naar
de doelbranch start de merge-request-pipeline opnieuw.

Co-authored-by: tim <tim@timfalken.com>

timfa project_87923614_bot_3aff574f6f98414b945a8799374f9092 · pushed by timfa

Authored 2026-10-22 · 3 weeks ago

Hash
5fec96f0fa764ce8cc2b319cc6fdef39e000dfd6
Parent
de590787
21 files changed +1691 −134
docs/CI.md +20 −8
@@ -37,7 +37,8 @@ jobs:
37 37 | `include` | Alleen `local`. Een job met dezelfde naam wordt samengevoegd; het buitenste bestand wint per veld. |
38 38 | `variables` | Tekst, of `{value: ...}`. Een job overschrijft dezelfde naam. |
39 39 | `before_script`, `script`, `after_script` | `after_script` draait ook als `script` mislukt. |
40 | `needs` | Namen, of `{job, optional}`. `needs: []` start meteen. Weggelaten: wacht op eerdere stages. |
40 | `workflow` | `rules` zoals bij een job. Geen passende rule, of `when: never`, maakt geen pipeline. |
41 | `needs` | Namen, of `{job, optional}`. `needs: []` start meteen. Weggelaten: wacht op eerdere stages. Een `needs` op een `parallel`-job wacht op elke instantie. |
41 42 | `rules` | De eerste rule die klopt wint. `if`, `changes`, `exists` en `when`. Zonder rules gelden `only` en `except`. |
42 43 | `only`, `except` | Taknaam, `branches`, `tags`, of `/regex/`. |
43 44 | `when` | `on_success` (standaard), `on_failure`, `always`, `manual`. In een rule ook `never`. |
@@ -45,30 +46,41 @@ jobs:
45 46 | `artifacts` | `paths` en `expire_in` (`30s`, `30m`, `12h`, `14d`, `2w`, `never`). |
46 47 | `cache` | `paths` en een tekstuele `key`. Lokaal per repository, niet gedeeld met een andere machine. |
47 48 | `resource_group` | Jobs met dezelfde naam in één repository draaien niet tegelijk. |
48 | `timeout` | Overschrijft `ci.timeout_seconds` voor die job. |
49 | `timeout` | Overschrijft `ci.timeout_seconds` voor die job. De runner stopt de container als de tijd om is. |
50 | `retry` | Een getal, of `{max, when}`. `max` is 0 tot 2; hoger wordt afgekapt. `retry: 2` herhaalt elke mislukking behalve een annulering. `when` noemt `script_failure`, `runner_system_failure`, `job_execution_timeout`, `stuck_or_timeout_failure` of `always`. |
51 | `parallel` | Een getal, hoogstens 50. De jobs heten `naam 1/N` en krijgen `CI_NODE_INDEX` en `CI_NODE_TOTAL`. |
52 | `parallel:matrix` | Per lijstitem het cartesisch product, daarna aan elkaar. De job heet `naam: [waarde, waarde]`. De sleutels worden variabelen, in de volgorde uit het bestand. |
53 | `services` | Sidecar-containers op een eigen Docker-netwerk met de job. Een tekst, of `name`, `alias`, `command`, `entrypoint` en `variables`. Zonder alias is dat het imagenaam zonder registry en tag. |
54 | `environment` | Een naam, of `{name, url}`. Zet `CI_ENVIRONMENT_NAME` en `CI_ENVIRONMENT_URL`. Geen deployment en geen `on_stop`. |
55 | `interruptible` | Standaard uit. Een nieuwe pipeline op dezelfde ref annuleert oudere jobs die dit aan hebben en nog in de wachtrij staan of lopen. |
49 56 | `tags` | Bewaard bij de job. De ingebouwde runner draait elke job; er is geen tweede runner om op tags te kiezen. |
50 57 | `extends` | Eén naam of een lijst. Een job waarvan de naam met `.` begint is een sjabloon en draait niet. |
51 58
52 Een `if` is één vergelijking, zonder `&&` of `||`:
59 Een `if` volgt de GitLab-vorm. `&&` bindt sterker dan `||`. Haakjes groeperen. Een lege variabele is `null`.
53 60
54 61 ```text
55 $NAAM == "waarde"
56 $NAAM == $ANDERE
57 $NAAM == null
62 $NAAM == "waarde" && ($ANDERE == "x" || $ANDERE == null)
58 63 $NAAM != "waarde"
59 64 $NAAM =~ /reguliere-expressie/
65 $NAAM =~ /^pad\/met\/slash/
66 $NAAM =~ /hoofdletters/i
60 67 $NAAM !~ /reguliere-expressie/
68 $NAAM =~ $PATROON
61 69 ```
62 70
71 `=~` en `!~` gebruiken RE2, zonder anker tot je `^` en `$` zelf zet. Een `/` in het patroon schrijf je als `\/`. De vlag `i` staat direct achter de sluithaak. Een patroon in een variabele moet bij het beoordelen een geldige expressie zijn.
72
63 73 `changes` kijkt naar de diff van de push. Is die diff niet te bepalen, dan geldt de voorwaarde. `exists` eist dat elk pad in de commit staat.
64 74
65 75 Voorgedefinieerd, in rules en in de container: `CI`, `CI_COMMIT_SHA`, `CI_COMMIT_SHORT_SHA`, `CI_COMMIT_BRANCH`, `CI_COMMIT_TAG`, `CI_COMMIT_REF`, `CI_COMMIT_REF_NAME`, `CI_COMMIT_REF_SLUG`, `CI_COMMIT_TITLE`, `CI_DEFAULT_BRANCH`, `CI_PIPELINE_ID`, `CI_PIPELINE_SOURCE`, `CI_PIPELINE_URL`, `CI_PROJECT_PATH`, `CI_PROJECT_NAME`, `CI_PROJECT_NAMESPACE`, `CI_PROJECT_URL`, `CI_JOB_ID`, `CI_JOB_NAME`, `CI_JOB_STAGE`, `CI_JOB_URL`, `CI_JOB_TOKEN`, `CI_API_URL`, `CI_CONFIG_PATH`, `CI_SERVER_NAME`. Op een tag is `CI_COMMIT_BRANCH` leeg. `CI_PIPELINE_SOURCE` is `push`, `web` of `merge_request_event`.
66 76
67 Een merge-request-pipeline start bij het openen van de merge request en bij elke push naar de bronbranch. `CI_PIPELINE_SOURCE` is dan `merge_request_event`. Daarbij horen `CI_MERGE_REQUEST_ID`, `CI_MERGE_REQUEST_IID`, `CI_MERGE_REQUEST_TITLE`, `CI_MERGE_REQUEST_SOURCE_BRANCH_NAME`, `CI_MERGE_REQUEST_SOURCE_BRANCH_SHA`, `CI_MERGE_REQUEST_TARGET_BRANCH_NAME`, `CI_MERGE_REQUEST_TARGET_BRANCH_SHA`, `CI_MERGE_REQUEST_PROJECT_PATH` en `CI_MERGE_REQUEST_PROJECT_URL`. De pipeline draait op de commit van de bronbranch. Jobs zonder `rules` en zonder `only: merge_requests` doen daar niet aan mee. `only: merge_requests` en `except: merge_requests` werken wel. Zodra zo'n bestand jobs voor de merge request kiest, mag er pas gemerged worden als de nieuwste merge-request-pipeline geslaagd is. Een job met `allow_failure` mag mislukken, maar moet wel af zijn. Een job die nog loopt, in de wachtrij staat of op manual staat, houdt de merge tegen.
77 Een merge-request-pipeline start bij het openen van de merge request, bij elke push naar de bronbranch, en opnieuw wanneer iemand naar de doelbranch pusht. Die laatste draait nog steeds op de commit van de bronbranch, niet op een samengevoegde boom. `CI_PIPELINE_SOURCE` is dan `merge_request_event`. Daarbij horen `CI_MERGE_REQUEST_ID`, `CI_MERGE_REQUEST_IID`, `CI_MERGE_REQUEST_TITLE`, `CI_MERGE_REQUEST_SOURCE_BRANCH_NAME`, `CI_MERGE_REQUEST_SOURCE_BRANCH_SHA`, `CI_MERGE_REQUEST_TARGET_BRANCH_NAME`, `CI_MERGE_REQUEST_TARGET_BRANCH_SHA`, `CI_MERGE_REQUEST_PROJECT_PATH` en `CI_MERGE_REQUEST_PROJECT_URL`, in de rules én in de container. De pipeline draait op de commit van de bronbranch. Jobs zonder `rules` en zonder `only: merge_requests` doen daar niet aan mee. `only: merge_requests` en `except: merge_requests` werken wel. Zodra zo'n bestand jobs voor de merge request kiest, mag er pas gemerged worden als de nieuwste merge-request-pipeline geslaagd is. Een job met `allow_failure` mag mislukken, maar moet wel af zijn. Een job die nog loopt, in de wachtrij staat, op manual staat of geannuleerd is, houdt de merge tegen.
78
79 `CI_PIPELINE_ID` is tijdens het beoordelen van rules nog 0. `CI_JOB_ID`, `CI_JOB_NAME`, `CI_JOB_STAGE`, `CI_JOB_URL`, `CI_JOB_TOKEN`, `CI_NODE_INDEX`, `CI_NODE_TOTAL`, `CI_ENVIRONMENT_NAME` en `CI_ENVIRONMENT_URL` bestaan pas in de container.
68 80
69 81 ## Bewust niet
70 82
71 `services`, `pages`, `workflow`, `spec`, een include van een ander project of van een URL, `parallel`/`matrix`, trigger- en child-pipelines, `environment`, `retry`, `coverage`, `id_tokens`, `inherit` en `hooks`. Die sleutels geven een waarschuwing in de editor en houden de rest van het bestand niet tegen. Er is één runner, in het BlueGit-proces, die Docker op dezelfde host aanroept.
83 `pages`, `spec`, een include van een ander project of van een URL, trigger- en child-pipelines, `coverage`, `id_tokens`, `inherit` en `hooks`. Die sleutels geven een waarschuwing in de editor en houden de rest van het bestand niet tegen. Er is één runner, in het BlueGit-proces, die Docker op dezelfde host aanroept. `tags` kiezen die runner niet. `services` delen één netwerk met de job; er is geen healthcheck verder dan een gestarte container.
72 84
73 85 ## Editor
74 86
@@ -37,7 +37,8 @@ jobs:
37| `include` | Alleen `local`. Een job met dezelfde naam wordt samengevoegd; het buitenste bestand wint per veld. |
37| `include` | Alleen `local`. Een job met dezelfde naam wordt samengevoegd; het buitenste bestand wint per veld. |
38| `variables` | Tekst, of `{value: ...}`. Een job overschrijft dezelfde naam. |
38| `variables` | Tekst, of `{value: ...}`. Een job overschrijft dezelfde naam. |
39| `before_script`, `script`, `after_script` | `after_script` draait ook als `script` mislukt. |
39| `before_script`, `script`, `after_script` | `after_script` draait ook als `script` mislukt. |
40| `needs` | Namen, of `{job, optional}`. `needs: []` start meteen. Weggelaten: wacht op eerdere stages. |
40| `workflow` | `rules` zoals bij een job. Geen passende rule, of `when: never`, maakt geen pipeline. |
41| `needs` | Namen, of `{job, optional}`. `needs: []` start meteen. Weggelaten: wacht op eerdere stages. Een `needs` op een `parallel`-job wacht op elke instantie. |
41| `rules` | De eerste rule die klopt wint. `if`, `changes`, `exists` en `when`. Zonder rules gelden `only` en `except`. |
42| `rules` | De eerste rule die klopt wint. `if`, `changes`, `exists` en `when`. Zonder rules gelden `only` en `except`. |
42| `only`, `except` | Taknaam, `branches`, `tags`, of `/regex/`. |
43| `only`, `except` | Taknaam, `branches`, `tags`, of `/regex/`. |
43| `when` | `on_success` (standaard), `on_failure`, `always`, `manual`. In een rule ook `never`. |
44| `when` | `on_success` (standaard), `on_failure`, `always`, `manual`. In een rule ook `never`. |
@@ -45,30 +46,41 @@ jobs:
45| `artifacts` | `paths` en `expire_in` (`30s`, `30m`, `12h`, `14d`, `2w`, `never`). |
46| `artifacts` | `paths` en `expire_in` (`30s`, `30m`, `12h`, `14d`, `2w`, `never`). |
46| `cache` | `paths` en een tekstuele `key`. Lokaal per repository, niet gedeeld met een andere machine. |
47| `cache` | `paths` en een tekstuele `key`. Lokaal per repository, niet gedeeld met een andere machine. |
47| `resource_group` | Jobs met dezelfde naam in één repository draaien niet tegelijk. |
48| `resource_group` | Jobs met dezelfde naam in één repository draaien niet tegelijk. |
48| `timeout` | Overschrijft `ci.timeout_seconds` voor die job. |
49| `timeout` | Overschrijft `ci.timeout_seconds` voor die job. De runner stopt de container als de tijd om is. |
50| `retry` | Een getal, of `{max, when}`. `max` is 0 tot 2; hoger wordt afgekapt. `retry: 2` herhaalt elke mislukking behalve een annulering. `when` noemt `script_failure`, `runner_system_failure`, `job_execution_timeout`, `stuck_or_timeout_failure` of `always`. |
51| `parallel` | Een getal, hoogstens 50. De jobs heten `naam 1/N` en krijgen `CI_NODE_INDEX` en `CI_NODE_TOTAL`. |
52| `parallel:matrix` | Per lijstitem het cartesisch product, daarna aan elkaar. De job heet `naam: [waarde, waarde]`. De sleutels worden variabelen, in de volgorde uit het bestand. |
53| `services` | Sidecar-containers op een eigen Docker-netwerk met de job. Een tekst, of `name`, `alias`, `command`, `entrypoint` en `variables`. Zonder alias is dat het imagenaam zonder registry en tag. |
54| `environment` | Een naam, of `{name, url}`. Zet `CI_ENVIRONMENT_NAME` en `CI_ENVIRONMENT_URL`. Geen deployment en geen `on_stop`. |
55| `interruptible` | Standaard uit. Een nieuwe pipeline op dezelfde ref annuleert oudere jobs die dit aan hebben en nog in de wachtrij staan of lopen. |
49| `tags` | Bewaard bij de job. De ingebouwde runner draait elke job; er is geen tweede runner om op tags te kiezen. |
56| `tags` | Bewaard bij de job. De ingebouwde runner draait elke job; er is geen tweede runner om op tags te kiezen. |
50| `extends` | Eén naam of een lijst. Een job waarvan de naam met `.` begint is een sjabloon en draait niet. |
57| `extends` | Eén naam of een lijst. Een job waarvan de naam met `.` begint is een sjabloon en draait niet. |
51
58
52Een `if` is één vergelijking, zonder `&&` of `||`:
59Een `if` volgt de GitLab-vorm. `&&` bindt sterker dan `||`. Haakjes groeperen. Een lege variabele is `null`.
53
60
54```text
61```text
55$NAAM == "waarde"
62$NAAM == "waarde" && ($ANDERE == "x" || $ANDERE == null)
56$NAAM == $ANDERE
57$NAAM == null
58$NAAM != "waarde"
63$NAAM != "waarde"
59$NAAM =~ /reguliere-expressie/
64$NAAM =~ /reguliere-expressie/
65$NAAM =~ /^pad\/met\/slash/
66$NAAM =~ /hoofdletters/i
60$NAAM !~ /reguliere-expressie/
67$NAAM !~ /reguliere-expressie/
68$NAAM =~ $PATROON
61```
69```
62
70
71`=~` en `!~` gebruiken RE2, zonder anker tot je `^` en `$` zelf zet. Een `/` in het patroon schrijf je als `\/`. De vlag `i` staat direct achter de sluithaak. Een patroon in een variabele moet bij het beoordelen een geldige expressie zijn.
72
63`changes` kijkt naar de diff van de push. Is die diff niet te bepalen, dan geldt de voorwaarde. `exists` eist dat elk pad in de commit staat.
73`changes` kijkt naar de diff van de push. Is die diff niet te bepalen, dan geldt de voorwaarde. `exists` eist dat elk pad in de commit staat.
64
74
65Voorgedefinieerd, in rules en in de container: `CI`, `CI_COMMIT_SHA`, `CI_COMMIT_SHORT_SHA`, `CI_COMMIT_BRANCH`, `CI_COMMIT_TAG`, `CI_COMMIT_REF`, `CI_COMMIT_REF_NAME`, `CI_COMMIT_REF_SLUG`, `CI_COMMIT_TITLE`, `CI_DEFAULT_BRANCH`, `CI_PIPELINE_ID`, `CI_PIPELINE_SOURCE`, `CI_PIPELINE_URL`, `CI_PROJECT_PATH`, `CI_PROJECT_NAME`, `CI_PROJECT_NAMESPACE`, `CI_PROJECT_URL`, `CI_JOB_ID`, `CI_JOB_NAME`, `CI_JOB_STAGE`, `CI_JOB_URL`, `CI_JOB_TOKEN`, `CI_API_URL`, `CI_CONFIG_PATH`, `CI_SERVER_NAME`. Op een tag is `CI_COMMIT_BRANCH` leeg. `CI_PIPELINE_SOURCE` is `push`, `web` of `merge_request_event`.
75Voorgedefinieerd, in rules en in de container: `CI`, `CI_COMMIT_SHA`, `CI_COMMIT_SHORT_SHA`, `CI_COMMIT_BRANCH`, `CI_COMMIT_TAG`, `CI_COMMIT_REF`, `CI_COMMIT_REF_NAME`, `CI_COMMIT_REF_SLUG`, `CI_COMMIT_TITLE`, `CI_DEFAULT_BRANCH`, `CI_PIPELINE_ID`, `CI_PIPELINE_SOURCE`, `CI_PIPELINE_URL`, `CI_PROJECT_PATH`, `CI_PROJECT_NAME`, `CI_PROJECT_NAMESPACE`, `CI_PROJECT_URL`, `CI_JOB_ID`, `CI_JOB_NAME`, `CI_JOB_STAGE`, `CI_JOB_URL`, `CI_JOB_TOKEN`, `CI_API_URL`, `CI_CONFIG_PATH`, `CI_SERVER_NAME`. Op een tag is `CI_COMMIT_BRANCH` leeg. `CI_PIPELINE_SOURCE` is `push`, `web` of `merge_request_event`.
66
76
67Een merge-request-pipeline start bij het openen van de merge request en bij elke push naar de bronbranch. `CI_PIPELINE_SOURCE` is dan `merge_request_event`. Daarbij horen `CI_MERGE_REQUEST_ID`, `CI_MERGE_REQUEST_IID`, `CI_MERGE_REQUEST_TITLE`, `CI_MERGE_REQUEST_SOURCE_BRANCH_NAME`, `CI_MERGE_REQUEST_SOURCE_BRANCH_SHA`, `CI_MERGE_REQUEST_TARGET_BRANCH_NAME`, `CI_MERGE_REQUEST_TARGET_BRANCH_SHA`, `CI_MERGE_REQUEST_PROJECT_PATH` en `CI_MERGE_REQUEST_PROJECT_URL`. De pipeline draait op de commit van de bronbranch. Jobs zonder `rules` en zonder `only: merge_requests` doen daar niet aan mee. `only: merge_requests` en `except: merge_requests` werken wel. Zodra zo'n bestand jobs voor de merge request kiest, mag er pas gemerged worden als de nieuwste merge-request-pipeline geslaagd is. Een job met `allow_failure` mag mislukken, maar moet wel af zijn. Een job die nog loopt, in de wachtrij staat of op manual staat, houdt de merge tegen.
77Een merge-request-pipeline start bij het openen van de merge request, bij elke push naar de bronbranch, en opnieuw wanneer iemand naar de doelbranch pusht. Die laatste draait nog steeds op de commit van de bronbranch, niet op een samengevoegde boom. `CI_PIPELINE_SOURCE` is dan `merge_request_event`. Daarbij horen `CI_MERGE_REQUEST_ID`, `CI_MERGE_REQUEST_IID`, `CI_MERGE_REQUEST_TITLE`, `CI_MERGE_REQUEST_SOURCE_BRANCH_NAME`, `CI_MERGE_REQUEST_SOURCE_BRANCH_SHA`, `CI_MERGE_REQUEST_TARGET_BRANCH_NAME`, `CI_MERGE_REQUEST_TARGET_BRANCH_SHA`, `CI_MERGE_REQUEST_PROJECT_PATH` en `CI_MERGE_REQUEST_PROJECT_URL`, in de rules én in de container. De pipeline draait op de commit van de bronbranch. Jobs zonder `rules` en zonder `only: merge_requests` doen daar niet aan mee. `only: merge_requests` en `except: merge_requests` werken wel. Zodra zo'n bestand jobs voor de merge request kiest, mag er pas gemerged worden als de nieuwste merge-request-pipeline geslaagd is. Een job met `allow_failure` mag mislukken, maar moet wel af zijn. Een job die nog loopt, in de wachtrij staat, op manual staat of geannuleerd is, houdt de merge tegen.
78
79`CI_PIPELINE_ID` is tijdens het beoordelen van rules nog 0. `CI_JOB_ID`, `CI_JOB_NAME`, `CI_JOB_STAGE`, `CI_JOB_URL`, `CI_JOB_TOKEN`, `CI_NODE_INDEX`, `CI_NODE_TOTAL`, `CI_ENVIRONMENT_NAME` en `CI_ENVIRONMENT_URL` bestaan pas in de container.
68
80
69## Bewust niet
81## Bewust niet
70
82
71`services`, `pages`, `workflow`, `spec`, een include van een ander project of van een URL, `parallel`/`matrix`, trigger- en child-pipelines, `environment`, `retry`, `coverage`, `id_tokens`, `inherit` en `hooks`. Die sleutels geven een waarschuwing in de editor en houden de rest van het bestand niet tegen. Er is één runner, in het BlueGit-proces, die Docker op dezelfde host aanroept.
83`pages`, `spec`, een include van een ander project of van een URL, trigger- en child-pipelines, `coverage`, `id_tokens`, `inherit` en `hooks`. Die sleutels geven een waarschuwing in de editor en houden de rest van het bestand niet tegen. Er is één runner, in het BlueGit-proces, die Docker op dezelfde host aanroept. `tags` kiezen die runner niet. `services` delen één netwerk met de job; er is geen healthcheck verder dan een gestarte container.
72
84
73## Editor
85## Editor
74
86
docs/ROADMAP.md +1 −1
@@ -17,7 +17,7 @@ Deze basis is genoeg om repositories te bewaren, te klonen, een pipeline te draa
17 17 1. **Productie aanzetten.** Domein registreren, de schijf mounten, `ReadWritePaths` voor dat pad zetten, de ingress uit [INGRESS.md](INGRESS.md) aanzetten, eerste aanmelding en in `/setup` het pad bevestigen, `master` beschermen, deploy-geheimen zetten, één keer met de hand deployen, daarna de pipeline.
18 18 2. **SSH in de praktijk.** Hostkey publiceren en `known_hosts` vullen. De publieke poort is `ssh.public_port` op de ingress; op de mini-PC blijft `ssh.listen` configureerbaar. Foutmeldingen van de hook gaan al via stderr naar de git-client; de UX van een geweigerde push kan scherper.
19 19 3. **Merge requests verder.** Opmerkingen, een echte diff in plaats van `git log --oneline`, en een pipeline die groen moet zijn voor het samenvoegen.
20 4. **CI.** Services naast een job (een database voor tests), een cache, live logs in plaats van het log na afloop, en opruimen van verlopen artifacts. Een tweede runner op een andere machine als de mini-PC het niet trekt.
20 4. **CI.** Live logs tijdens een job, en opruimen van verlopen artifacts. Een tweede runner op een andere machine als de mini-PC het niet trekt. `workflow`, `retry`, `parallel`, `services` en `environment` horen bij de ingebouwde runner; zie [CI.md](CI.md). `tags` sturen nog geen aparte runner aan.
21 21 5. **Webhooks.** Privénetwerken zijn dicht. Nog te doen: opnieuw proberen, en een knop die een testgebeurtenis stuurt.
22 22 6. **Git.** LFS als de 4 TB-schijf grote bestanden moet dragen, signed commits, en mirrors. Force-push op een beschermde branch is nu toegestaan voor een beheerder; een aparte schakelaar hoort daarbij.
23 23 7. **Account.** Uitnodigen voordat iemand ooit is aangemeld werkt al op gebruikersnaam. Een zoekfunctie ontbreekt, omdat de auth-server geen lijst van alle gebruikers aan een site geeft. E-mail naar een gebruiker kan later via `POST /api/email` van de auth-server, zonder het adres te zien.
@@ -17,7 +17,7 @@ Deze basis is genoeg om repositories te bewaren, te klonen, een pipeline te draa
171. **Productie aanzetten.** Domein registreren, de schijf mounten, `ReadWritePaths` voor dat pad zetten, de ingress uit [INGRESS.md](INGRESS.md) aanzetten, eerste aanmelding en in `/setup` het pad bevestigen, `master` beschermen, deploy-geheimen zetten, één keer met de hand deployen, daarna de pipeline.
171. **Productie aanzetten.** Domein registreren, de schijf mounten, `ReadWritePaths` voor dat pad zetten, de ingress uit [INGRESS.md](INGRESS.md) aanzetten, eerste aanmelding en in `/setup` het pad bevestigen, `master` beschermen, deploy-geheimen zetten, één keer met de hand deployen, daarna de pipeline.
182. **SSH in de praktijk.** Hostkey publiceren en `known_hosts` vullen. De publieke poort is `ssh.public_port` op de ingress; op de mini-PC blijft `ssh.listen` configureerbaar. Foutmeldingen van de hook gaan al via stderr naar de git-client; de UX van een geweigerde push kan scherper.
182. **SSH in de praktijk.** Hostkey publiceren en `known_hosts` vullen. De publieke poort is `ssh.public_port` op de ingress; op de mini-PC blijft `ssh.listen` configureerbaar. Foutmeldingen van de hook gaan al via stderr naar de git-client; de UX van een geweigerde push kan scherper.
193. **Merge requests verder.** Opmerkingen, een echte diff in plaats van `git log --oneline`, en een pipeline die groen moet zijn voor het samenvoegen.
193. **Merge requests verder.** Opmerkingen, een echte diff in plaats van `git log --oneline`, en een pipeline die groen moet zijn voor het samenvoegen.
204. **CI.** Services naast een job (een database voor tests), een cache, live logs in plaats van het log na afloop, en opruimen van verlopen artifacts. Een tweede runner op een andere machine als de mini-PC het niet trekt.
204. **CI.** Live logs tijdens een job, en opruimen van verlopen artifacts. Een tweede runner op een andere machine als de mini-PC het niet trekt. `workflow`, `retry`, `parallel`, `services` en `environment` horen bij de ingebouwde runner; zie [CI.md](CI.md). `tags` sturen nog geen aparte runner aan.
215. **Webhooks.** Privénetwerken zijn dicht. Nog te doen: opnieuw proberen, en een knop die een testgebeurtenis stuurt.
215. **Webhooks.** Privénetwerken zijn dicht. Nog te doen: opnieuw proberen, en een knop die een testgebeurtenis stuurt.
226. **Git.** LFS als de 4 TB-schijf grote bestanden moet dragen, signed commits, en mirrors. Force-push op een beschermde branch is nu toegestaan voor een beheerder; een aparte schakelaar hoort daarbij.
226. **Git.** LFS als de 4 TB-schijf grote bestanden moet dragen, signed commits, en mirrors. Force-push op een beschermde branch is nu toegestaan voor een beheerder; een aparte schakelaar hoort daarbij.
237. **Account.** Uitnodigen voordat iemand ooit is aangemeld werkt al op gebruikersnaam. Een zoekfunctie ontbreekt, omdat de auth-server geen lijst van alle gebruikers aan een site geeft. E-mail naar een gebruiker kan later via `POST /api/email` van de auth-server, zonder het adres te zien.
237. **Account.** Uitnodigen voordat iemand ooit is aangemeld werkt al op gebruikersnaam. Een zoekfunctie ontbreekt, omdat de auth-server geen lijst van alle gebruikers aan een site geeft. E-mail naar een gebruiker kan later via `POST /api/email` van de auth-server, zonder het adres te zien.
internal/ci/eval.go +182 −58
@@ -4,6 +4,7 @@ import (
4 4 "fmt"
5 5 "path"
6 6 "regexp"
7 "strconv"
7 8 "strings"
8 9
9 10 "gitlab.com/Timfa/bluegit/internal/i18n"
@@ -147,7 +148,21 @@ func (p Pipeline) Select(ev Eval) ([]Job, error) {
147 148 pending[i].Needs = needs
148 149 out = append(out, pending[i])
149 150 }
150 return out, nil
151 return expandJobs(out)
152 }
153
154 // Allow zegt of workflow:rules deze pipeline toelaat.
155 // Zonder workflow wordt de pipeline gemaakt. Een passende rule met when:never
156 // of geen enkele passende rule slaat de pipeline over.
157 func (p Pipeline) Allow(ev Eval) (bool, error) {
158 if len(p.Workflow) == 0 {
159 return true, nil
160 }
161 if ev.Env == nil {
162 ev.Env = map[string]string{}
163 }
164 ok, _, err := (Job{Rules: p.Workflow, When: "always"}).matches(ev)
165 return ok, err
151 166 }
152 167
153 168 // ActiveJobs is Select met alleen variabelen. changes en exists gelden dan als waar.
@@ -398,7 +413,7 @@ func Gate(jobs []Job, status map[string]string, job Job) string {
398 413 switch status[d.Name] {
399 414 case "success", "skipped":
400 415 continue
401 case "failed":
416 case "failed", "canceled":
402 417 failedAny = true
403 418 if !d.AllowFailure {
404 419 failedHard = true
@@ -432,75 +447,184 @@ func Gate(jobs []Job, status map[string]string, job Job) string {
432 447 }
433 448 }
434 449
435 var (
436 reEq = regexp.MustCompile(`^\$([A-Za-z_][A-Za-z0-9_]*)\s*==\s*(?:"([^"]*)"|\$([A-Za-z_][A-Za-z0-9_]*)|null)$`)
437 reNe = regexp.MustCompile(`^\$([A-Za-z_][A-Za-z0-9_]*)\s*!=\s*(?:"([^"]*)"|\$([A-Za-z_][A-Za-z0-9_]*)|null)$`)
438 reRe = regexp.MustCompile(`^\$([A-Za-z_][A-Za-z0-9_]*)\s*=~\s*/(.*)/$`)
439 reNRe = regexp.MustCompile(`^\$([A-Za-z_][A-Za-z0-9_]*)\s*!~\s*/(.*)/$`)
440 )
450 func evalRule(expr string, env map[string]string) (bool, error) {
451 p := &ruleParser{s: expr, env: env}
452 ok, err := p.parseOr()
453 if err != nil {
454 return false, err
455 }
456 p.skip()
457 if p.i < len(p.s) {
458 return false, i18n.E("ci.err.ruleExpr", strings.TrimSpace(expr))
459 }
460 return ok, nil
461 }
441 462
442 type compiledRule struct {
443 kind string
444 name string
445 arg string
446 other string
447 null bool
448 re *regexp.Regexp
463 const maxParallel = 50
464
465 func expandJobs(jobs []Job) ([]Job, error) {
466 expanded := map[string][]string{}
467 var out []Job
468 seen := map[string]bool{}
469 for _, j := range jobs {
470 instances, err := expandOne(j)
471 if err != nil {
472 return nil, fmt.Errorf("job %s: %w", j.Name, err)
473 }
474 names := make([]string, len(instances))
475 for i, inst := range instances {
476 if seen[inst.Name] {
477 return nil, i18n.E("ci.err.jobDup", inst.Name)
478 }
479 seen[inst.Name] = true
480 names[i] = inst.Name
481 out = append(out, inst)
482 }
483 expanded[j.Name] = names
484 }
485 for i := range out {
486 out[i].Needs = rewriteNeeds(out[i].Needs, expanded)
487 out[i].Optional = rewriteNeeds(out[i].Optional, expanded)
488 }
489 return out, nil
449 490 }
450 491
451 func compileRule(expr string) (compiledRule, error) {
452 expr = strings.TrimSpace(expr)
453 if strings.Contains(expr, "&&") || strings.Contains(expr, "||") {
454 return compiledRule{}, i18n.E("ci.err.ruleCombine", expr)
492 func expandOne(j Job) ([]Job, error) {
493 if len(j.Matrix) > 0 {
494 var combos []matrixCombo
495 for _, spec := range j.Matrix {
496 combos = append(combos, cartesian(spec)...)
497 }
498 if len(combos) > maxParallel {
499 return nil, i18n.E("ci.err.parallel", len(combos), maxParallel)
500 }
501 if len(combos) == 0 {
502 return nil, i18n.E("ci.err.parallel", 0, maxParallel)
503 }
504 var out []Job
505 for i, combo := range combos {
506 cp := copyJob(j)
507 cp.Name = j.Name + ": [" + strings.Join(combo.values, ", ") + "]"
508 cp.Parallel = 0
509 cp.Matrix = nil
510 for k, v := range combo.vars {
511 cp.Variables[k] = v
512 }
513 cp.Variables["CI_NODE_INDEX"] = strconv.Itoa(i + 1)
514 cp.Variables["CI_NODE_TOTAL"] = strconv.Itoa(len(combos))
515 out = append(out, cp)
516 }
517 return out, nil
455 518 }
456 if m := reEq.FindStringSubmatch(expr); m != nil {
457 return compiledRule{kind: "eq", name: m[1], arg: m[2], other: m[3], null: m[2] == "" && m[3] == "" && strings.HasSuffix(strings.TrimSpace(expr), "null")}, nil
519 if j.Parallel <= 1 {
520 return []Job{j}, nil
458 521 }
459 if m := reNe.FindStringSubmatch(expr); m != nil {
460 return compiledRule{kind: "ne", name: m[1], arg: m[2], other: m[3], null: m[2] == "" && m[3] == "" && strings.HasSuffix(strings.TrimSpace(expr), "null")}, nil
522 if j.Parallel > maxParallel {
523 return nil, i18n.E("ci.err.parallel", j.Parallel, maxParallel)
461 524 }
462 if m := reRe.FindStringSubmatch(expr); m != nil {
463 re, err := regexp.Compile(m[2])
464 if err != nil {
465 return compiledRule{}, i18n.E("ci.err.regex", expr, err.Error())
466 }
467 return compiledRule{kind: "re", name: m[1], re: re}, nil
525 var out []Job
526 for i := 1; i <= j.Parallel; i++ {
527 cp := copyJob(j)
528 cp.Name = fmt.Sprintf("%s %d/%d", j.Name, i, j.Parallel)
529 cp.Parallel = 0
530 cp.Variables["CI_NODE_INDEX"] = strconv.Itoa(i)
531 cp.Variables["CI_NODE_TOTAL"] = strconv.Itoa(j.Parallel)
532 out = append(out, cp)
468 533 }
469 if m := reNRe.FindStringSubmatch(expr); m != nil {
470 re, err := regexp.Compile(m[2])
471 if err != nil {
472 return compiledRule{}, i18n.E("ci.err.regex", expr, err.Error())
534 return out, nil
535 }
536
537 type matrixCombo struct {
538 values []string
539 vars map[string]string
540 }
541
542 func cartesian(spec MatrixSpec) []matrixCombo {
543 if len(spec.Keys) == 0 {
544 return nil
545 }
546 var out []matrixCombo
547 var walk func(i int, vals []string, vars map[string]string)
548 walk = func(i int, vals []string, vars map[string]string) {
549 if i == len(spec.Keys) {
550 cp := map[string]string{}
551 for k, v := range vars {
552 cp[k] = v
553 }
554 out = append(out, matrixCombo{values: append([]string{}, vals...), vars: cp})
555 return
556 }
557 for _, v := range spec.Vals[i] {
558 vars[spec.Keys[i]] = v
559 walk(i+1, append(vals, v), vars)
473 560 }
474 return compiledRule{kind: "nre", name: m[1], re: re}, nil
475 561 }
476 return compiledRule{}, i18n.E("ci.err.ruleExpr", expr)
562 walk(0, nil, map[string]string{})
563 return out
477 564 }
478 565
479 func evalRule(expr string, env map[string]string) (bool, error) {
480 c, err := compileRule(expr)
481 if err != nil {
482 return false, err
566 func copyJob(j Job) Job {
567 cp := j
568 cp.Variables = map[string]string{}
569 for k, v := range j.Variables {
570 cp.Variables[k] = v
571 }
572 cp.Needs = append([]string{}, j.Needs...)
573 cp.Optional = append([]string{}, j.Optional...)
574 cp.Services = append([]Service{}, j.Services...)
575 return cp
576 }
577
578 func rewriteNeeds(list []string, expanded map[string][]string) []string {
579 if list == nil {
580 return nil
581 }
582 var out []string
583 for _, n := range list {
584 names := expanded[n]
585 if len(names) == 0 || (len(names) == 1 && names[0] == n) {
586 out = append(out, n)
587 continue
588 }
589 out = append(out, names...)
483 590 }
484 val := env[c.name]
485 right := c.arg
486 if c.other != "" {
487 right = env[c.other]
488 }
489 if c.null {
490 right = ""
491 }
492 switch c.kind {
493 case "eq":
494 return val == right, nil
495 case "ne":
496 return val != right, nil
497 case "re":
498 return c.re.MatchString(val), nil
499 case "nre":
500 return !c.re.MatchString(val), nil
501 default:
502 return false, i18n.E("ci.err.ruleExpr", expr)
591 if out == nil {
592 return []string{}
593 }
594 return out
595 }
596
597 // Expand vervangt $NAAM en ${NAAM} in een tekst, bijvoorbeeld environment:url.
598 func Expand(s string, env map[string]string) string {
599 var b strings.Builder
600 for i := 0; i < len(s); i++ {
601 if s[i] != '$' || i+1 >= len(s) {
602 b.WriteByte(s[i])
603 continue
604 }
605 if s[i+1] == '{' {
606 end := strings.IndexByte(s[i+2:], '}')
607 if end < 0 {
608 b.WriteByte('$')
609 continue
610 }
611 name := s[i+2 : i+2+end]
612 b.WriteString(env[name])
613 i += 2 + end
614 continue
615 }
616 j := i + 1
617 for j < len(s) && isIdent(s[j]) {
618 j++
619 }
620 if j == i+1 {
621 b.WriteByte('$')
622 continue
623 }
624 b.WriteString(env[s[i+1:j]])
625 i = j - 1
503 626 }
627 return b.String()
504 628 }
505 629
506 630 func contains(list []string, s string) bool {
@@ -4,6 +4,7 @@ import (
4 "fmt"
4 "fmt"
5 "path"
5 "path"
6 "regexp"
6 "regexp"
7 "strconv"
7 "strings"
8 "strings"
8
9
9 "gitlab.com/Timfa/bluegit/internal/i18n"
10 "gitlab.com/Timfa/bluegit/internal/i18n"
@@ -147,7 +148,21 @@ func (p Pipeline) Select(ev Eval) ([]Job, error) {
147 pending[i].Needs = needs
148 pending[i].Needs = needs
148 out = append(out, pending[i])
149 out = append(out, pending[i])
149 }
150 }
150 return out, nil
151 return expandJobs(out)
152}
153
154// Allow zegt of workflow:rules deze pipeline toelaat.
155// Zonder workflow wordt de pipeline gemaakt. Een passende rule met when:never
156// of geen enkele passende rule slaat de pipeline over.
157func (p Pipeline) Allow(ev Eval) (bool, error) {
158 if len(p.Workflow) == 0 {
159 return true, nil
160 }
161 if ev.Env == nil {
162 ev.Env = map[string]string{}
163 }
164 ok, _, err := (Job{Rules: p.Workflow, When: "always"}).matches(ev)
165 return ok, err
151}
166}
152
167
153// ActiveJobs is Select met alleen variabelen. changes en exists gelden dan als waar.
168// ActiveJobs is Select met alleen variabelen. changes en exists gelden dan als waar.
@@ -398,7 +413,7 @@ func Gate(jobs []Job, status map[string]string, job Job) string {
398 switch status[d.Name] {
413 switch status[d.Name] {
399 case "success", "skipped":
414 case "success", "skipped":
400 continue
415 continue
401 case "failed":
416 case "failed", "canceled":
402 failedAny = true
417 failedAny = true
403 if !d.AllowFailure {
418 if !d.AllowFailure {
404 failedHard = true
419 failedHard = true
@@ -432,75 +447,184 @@ func Gate(jobs []Job, status map[string]string, job Job) string {
432 }
447 }
433}
448}
434
449
435var (
450func evalRule(expr string, env map[string]string) (bool, error) {
436 reEq = regexp.MustCompile(`^\$([A-Za-z_][A-Za-z0-9_]*)\s*==\s*(?:"([^"]*)"|\$([A-Za-z_][A-Za-z0-9_]*)|null)$`)
451 p := &ruleParser{s: expr, env: env}
437 reNe = regexp.MustCompile(`^\$([A-Za-z_][A-Za-z0-9_]*)\s*!=\s*(?:"([^"]*)"|\$([A-Za-z_][A-Za-z0-9_]*)|null)$`)
452 ok, err := p.parseOr()
438 reRe = regexp.MustCompile(`^\$([A-Za-z_][A-Za-z0-9_]*)\s*=~\s*/(.*)/$`)
453 if err != nil {
439 reNRe = regexp.MustCompile(`^\$([A-Za-z_][A-Za-z0-9_]*)\s*!~\s*/(.*)/$`)
454 return false, err
440)
455 }
456 p.skip()
457 if p.i < len(p.s) {
458 return false, i18n.E("ci.err.ruleExpr", strings.TrimSpace(expr))
459 }
460 return ok, nil
461}
441
462
442type compiledRule struct {
463const maxParallel = 50
443 kind string
464
444 name string
465func expandJobs(jobs []Job) ([]Job, error) {
445 arg string
466 expanded := map[string][]string{}
446 other string
467 var out []Job
447 null bool
468 seen := map[string]bool{}
448 re *regexp.Regexp
469 for _, j := range jobs {
470 instances, err := expandOne(j)
471 if err != nil {
472 return nil, fmt.Errorf("job %s: %w", j.Name, err)
473 }
474 names := make([]string, len(instances))
475 for i, inst := range instances {
476 if seen[inst.Name] {
477 return nil, i18n.E("ci.err.jobDup", inst.Name)
478 }
479 seen[inst.Name] = true
480 names[i] = inst.Name
481 out = append(out, inst)
482 }
483 expanded[j.Name] = names
484 }
485 for i := range out {
486 out[i].Needs = rewriteNeeds(out[i].Needs, expanded)
487 out[i].Optional = rewriteNeeds(out[i].Optional, expanded)
488 }
489 return out, nil
449}
490}
450
491
451func compileRule(expr string) (compiledRule, error) {
492func expandOne(j Job) ([]Job, error) {
452 expr = strings.TrimSpace(expr)
493 if len(j.Matrix) > 0 {
453 if strings.Contains(expr, "&&") || strings.Contains(expr, "||") {
494 var combos []matrixCombo
454 return compiledRule{}, i18n.E("ci.err.ruleCombine", expr)
495 for _, spec := range j.Matrix {
496 combos = append(combos, cartesian(spec)...)
497 }
498 if len(combos) > maxParallel {
499 return nil, i18n.E("ci.err.parallel", len(combos), maxParallel)
500 }
501 if len(combos) == 0 {
502 return nil, i18n.E("ci.err.parallel", 0, maxParallel)
503 }
504 var out []Job
505 for i, combo := range combos {
506 cp := copyJob(j)
507 cp.Name = j.Name + ": [" + strings.Join(combo.values, ", ") + "]"
508 cp.Parallel = 0
509 cp.Matrix = nil
510 for k, v := range combo.vars {
511 cp.Variables[k] = v
512 }
513 cp.Variables["CI_NODE_INDEX"] = strconv.Itoa(i + 1)
514 cp.Variables["CI_NODE_TOTAL"] = strconv.Itoa(len(combos))
515 out = append(out, cp)
516 }
517 return out, nil
455 }
518 }
456 if m := reEq.FindStringSubmatch(expr); m != nil {
519 if j.Parallel <= 1 {
457 return compiledRule{kind: "eq", name: m[1], arg: m[2], other: m[3], null: m[2] == "" && m[3] == "" && strings.HasSuffix(strings.TrimSpace(expr), "null")}, nil
520 return []Job{j}, nil
458 }
521 }
459 if m := reNe.FindStringSubmatch(expr); m != nil {
522 if j.Parallel > maxParallel {
460 return compiledRule{kind: "ne", name: m[1], arg: m[2], other: m[3], null: m[2] == "" && m[3] == "" && strings.HasSuffix(strings.TrimSpace(expr), "null")}, nil
523 return nil, i18n.E("ci.err.parallel", j.Parallel, maxParallel)
461 }
524 }
462 if m := reRe.FindStringSubmatch(expr); m != nil {
525 var out []Job
463 re, err := regexp.Compile(m[2])
526 for i := 1; i <= j.Parallel; i++ {
464 if err != nil {
527 cp := copyJob(j)
465 return compiledRule{}, i18n.E("ci.err.regex", expr, err.Error())
528 cp.Name = fmt.Sprintf("%s %d/%d", j.Name, i, j.Parallel)
466 }
529 cp.Parallel = 0
467 return compiledRule{kind: "re", name: m[1], re: re}, nil
530 cp.Variables["CI_NODE_INDEX"] = strconv.Itoa(i)
531 cp.Variables["CI_NODE_TOTAL"] = strconv.Itoa(j.Parallel)
532 out = append(out, cp)
468 }
533 }
469 if m := reNRe.FindStringSubmatch(expr); m != nil {
534 return out, nil
470 re, err := regexp.Compile(m[2])
535}
471 if err != nil {
536
472 return compiledRule{}, i18n.E("ci.err.regex", expr, err.Error())
537type matrixCombo struct {
538 values []string
539 vars map[string]string
540}
541
542func cartesian(spec MatrixSpec) []matrixCombo {
543 if len(spec.Keys) == 0 {
544 return nil
545 }
546 var out []matrixCombo
547 var walk func(i int, vals []string, vars map[string]string)
548 walk = func(i int, vals []string, vars map[string]string) {
549 if i == len(spec.Keys) {
550 cp := map[string]string{}
551 for k, v := range vars {
552 cp[k] = v
553 }
554 out = append(out, matrixCombo{values: append([]string{}, vals...), vars: cp})
555 return
556 }
557 for _, v := range spec.Vals[i] {
558 vars[spec.Keys[i]] = v
559 walk(i+1, append(vals, v), vars)
473 }
560 }
474 return compiledRule{kind: "nre", name: m[1], re: re}, nil
475 }
561 }
476 return compiledRule{}, i18n.E("ci.err.ruleExpr", expr)
562 walk(0, nil, map[string]string{})
563 return out
477}
564}
478
565
479func evalRule(expr string, env map[string]string) (bool, error) {
566func copyJob(j Job) Job {
480 c, err := compileRule(expr)
567 cp := j
481 if err != nil {
568 cp.Variables = map[string]string{}
482 return false, err
569 for k, v := range j.Variables {
570 cp.Variables[k] = v
571 }
572 cp.Needs = append([]string{}, j.Needs...)
573 cp.Optional = append([]string{}, j.Optional...)
574 cp.Services = append([]Service{}, j.Services...)
575 return cp
576}
577
578func rewriteNeeds(list []string, expanded map[string][]string) []string {
579 if list == nil {
580 return nil
581 }
582 var out []string
583 for _, n := range list {
584 names := expanded[n]
585 if len(names) == 0 || (len(names) == 1 && names[0] == n) {
586 out = append(out, n)
587 continue
588 }
589 out = append(out, names...)
483 }
590 }
484 val := env[c.name]
591 if out == nil {
485 right := c.arg
592 return []string{}
486 if c.other != "" {
593 }
487 right = env[c.other]
594 return out
488 }
595}
489 if c.null {
596
490 right = ""
597// Expand vervangt $NAAM en ${NAAM} in een tekst, bijvoorbeeld environment:url.
491 }
598func Expand(s string, env map[string]string) string {
492 switch c.kind {
599 var b strings.Builder
493 case "eq":
600 for i := 0; i < len(s); i++ {
494 return val == right, nil
601 if s[i] != '$' || i+1 >= len(s) {
495 case "ne":
602 b.WriteByte(s[i])
496 return val != right, nil
603 continue
497 case "re":
604 }
498 return c.re.MatchString(val), nil
605 if s[i+1] == '{' {
499 case "nre":
606 end := strings.IndexByte(s[i+2:], '}')
500 return !c.re.MatchString(val), nil
607 if end < 0 {
501 default:
608 b.WriteByte('$')
502 return false, i18n.E("ci.err.ruleExpr", expr)
609 continue
610 }
611 name := s[i+2 : i+2+end]
612 b.WriteString(env[name])
613 i += 2 + end
614 continue
615 }
616 j := i + 1
617 for j < len(s) && isIdent(s[j]) {
618 j++
619 }
620 if j == i+1 {
621 b.WriteByte('$')
622 continue
623 }
624 b.WriteString(env[s[i+1:j]])
625 i = j - 1
503 }
626 }
627 return b.String()
504}
628}
505
629
506func contains(list []string, s string) bool {
630func contains(list []string, s string) bool {
internal/ci/expr.go added +337 −0
@@ -0,0 +1,337 @@
1 package ci
2
3 import (
4 "regexp"
5 "strings"
6 "unicode"
7
8 "gitlab.com/Timfa/bluegit/internal/i18n"
9 )
10
11 // ruleParser leest een GitLab-achtige rules:if.
12 // && bindt sterker dan ||. Haakjes groeperen. Een lege waarde is null.
13 type ruleParser struct {
14 s string
15 i int
16 env map[string]string
17 syntax bool
18 }
19
20 func (p *ruleParser) parseOr() (bool, error) {
21 left, err := p.parseAnd()
22 if err != nil {
23 return false, err
24 }
25 for {
26 p.skip()
27 if !p.take("||") {
28 return left, nil
29 }
30 right, err := p.parseAnd()
31 if err != nil {
32 return false, err
33 }
34 left = left || right
35 }
36 }
37
38 func (p *ruleParser) parseAnd() (bool, error) {
39 left, err := p.parseCmp()
40 if err != nil {
41 return false, err
42 }
43 for {
44 p.skip()
45 if !p.take("&&") {
46 return left, nil
47 }
48 right, err := p.parseCmp()
49 if err != nil {
50 return false, err
51 }
52 left = left && right
53 }
54 }
55
56 func (p *ruleParser) parseCmp() (bool, error) {
57 p.skip()
58 if p.peek() == '(' {
59 p.i++
60 inner, err := p.parseOr()
61 if err != nil {
62 return false, err
63 }
64 p.skip()
65 if p.peek() != ')' {
66 return false, i18n.E("ci.err.ruleExpr", strings.TrimSpace(p.s))
67 }
68 p.i++
69 return inner, nil
70 }
71 left, err := p.value()
72 if err != nil {
73 return false, err
74 }
75 p.skip()
76 op := p.operator()
77 if op == "" {
78 return false, i18n.E("ci.err.ruleExpr", strings.TrimSpace(p.s))
79 }
80 right, err := p.value()
81 if err != nil {
82 return false, err
83 }
84 return compare(op, left, right, p.syntax)
85 }
86
87 type ruleValue struct {
88 text string
89 null bool
90 re *regexp.Regexp
91 }
92
93 func (p *ruleParser) value() (ruleValue, error) {
94 p.skip()
95 if p.i >= len(p.s) {
96 return ruleValue{}, i18n.E("ci.err.ruleExpr", strings.TrimSpace(p.s))
97 }
98 switch p.s[p.i] {
99 case '$':
100 name, err := p.variable()
101 if err != nil {
102 return ruleValue{}, err
103 }
104 text := p.env[name]
105 return ruleValue{text: text, null: text == ""}, nil
106 case '"', '\'':
107 text, err := p.quoted()
108 if err != nil {
109 return ruleValue{}, err
110 }
111 return ruleValue{text: text, null: text == ""}, nil
112 case '/':
113 re, err := p.regex()
114 if err != nil {
115 return ruleValue{}, err
116 }
117 return ruleValue{re: re}, nil
118 default:
119 if p.takeWord("null") {
120 return ruleValue{null: true}, nil
121 }
122 return ruleValue{}, i18n.E("ci.err.ruleExpr", strings.TrimSpace(p.s))
123 }
124 }
125
126 func (p *ruleParser) variable() (string, error) {
127 p.i++
128 if p.i < len(p.s) && p.s[p.i] == '{' {
129 p.i++
130 start := p.i
131 for p.i < len(p.s) && p.s[p.i] != '}' {
132 p.i++
133 }
134 if p.i >= len(p.s) || start == p.i {
135 return "", i18n.E("ci.err.ruleExpr", strings.TrimSpace(p.s))
136 }
137 name := p.s[start:p.i]
138 p.i++
139 if !ident(name) {
140 return "", i18n.E("ci.err.ruleExpr", strings.TrimSpace(p.s))
141 }
142 return name, nil
143 }
144 start := p.i
145 for p.i < len(p.s) && (isIdent(p.s[p.i])) {
146 p.i++
147 }
148 if start == p.i {
149 return "", i18n.E("ci.err.ruleExpr", strings.TrimSpace(p.s))
150 }
151 return p.s[start:p.i], nil
152 }
153
154 func (p *ruleParser) quoted() (string, error) {
155 quote := p.s[p.i]
156 p.i++
157 var b strings.Builder
158 for p.i < len(p.s) {
159 c := p.s[p.i]
160 if c == quote {
161 p.i++
162 return b.String(), nil
163 }
164 if c == '\\' && quote == '"' && p.i+1 < len(p.s) {
165 p.i++
166 b.WriteByte(p.s[p.i])
167 p.i++
168 continue
169 }
170 if c == '$' && quote == '"' {
171 name, err := p.variable()
172 if err != nil {
173 return "", err
174 }
175 b.WriteString(p.env[name])
176 continue
177 }
178 b.WriteByte(c)
179 p.i++
180 }
181 return "", i18n.E("ci.err.ruleExpr", strings.TrimSpace(p.s))
182 }
183
184 func (p *ruleParser) regex() (*regexp.Regexp, error) {
185 p.i++
186 var b strings.Builder
187 for p.i < len(p.s) {
188 c := p.s[p.i]
189 if c == '\\' && p.i+1 < len(p.s) {
190 next := p.s[p.i+1]
191 if next == '/' {
192 b.WriteByte('/')
193 } else {
194 b.WriteByte('\\')
195 b.WriteByte(next)
196 }
197 p.i += 2
198 continue
199 }
200 if c == '/' {
201 p.i++
202 pat := b.String()
203 if p.i < len(p.s) && (p.s[p.i] == 'i' || p.s[p.i] == 'I') {
204 pat = "(?i)" + pat
205 p.i++
206 }
207 re, err := regexp.Compile(pat)
208 if err != nil {
209 return nil, i18n.E("ci.err.regex", strings.TrimSpace(p.s), err.Error())
210 }
211 return re, nil
212 }
213 b.WriteByte(c)
214 p.i++
215 }
216 return nil, i18n.E("ci.err.ruleExpr", strings.TrimSpace(p.s))
217 }
218
219 func (p *ruleParser) operator() string {
220 p.skip()
221 for _, op := range []string{"=~", "!~", "==", "!="} {
222 if p.take(op) {
223 return op
224 }
225 }
226 return ""
227 }
228
229 func (p *ruleParser) take(lit string) bool {
230 if strings.HasPrefix(p.s[p.i:], lit) {
231 p.i += len(lit)
232 return true
233 }
234 return false
235 }
236
237 func (p *ruleParser) takeWord(word string) bool {
238 if !strings.HasPrefix(p.s[p.i:], word) {
239 return false
240 }
241 end := p.i + len(word)
242 if end < len(p.s) && isIdent(p.s[end]) {
243 return false
244 }
245 p.i = end
246 return true
247 }
248
249 func (p *ruleParser) peek() byte {
250 p.skip()
251 if p.i >= len(p.s) {
252 return 0
253 }
254 return p.s[p.i]
255 }
256
257 func (p *ruleParser) skip() {
258 for p.i < len(p.s) && unicode.IsSpace(rune(p.s[p.i])) {
259 p.i++
260 }
261 }
262
263 func compare(op string, left, right ruleValue, syntax bool) (bool, error) {
264 switch op {
265 case "==":
266 if left.re != nil || right.re != nil {
267 return false, i18n.E("ci.err.ruleExpr", op)
268 }
269 if left.null || right.null {
270 return left.null && right.null, nil
271 }
272 return left.text == right.text, nil
273 case "!=":
274 if left.re != nil || right.re != nil {
275 return false, i18n.E("ci.err.ruleExpr", op)
276 }
277 if left.null || right.null {
278 return left.null != right.null, nil
279 }
280 return left.text != right.text, nil
281 case "=~", "!~":
282 if left.re != nil {
283 return false, i18n.E("ci.err.ruleExpr", op)
284 }
285 re := right.re
286 if re == nil && !right.null {
287 compiled, err := regexp.Compile(right.text)
288 if err != nil {
289 return false, i18n.E("ci.err.regex", right.text, err.Error())
290 }
291 re = compiled
292 }
293 if re == nil {
294 if syntax && right.null && right.text == "" && right.re == nil {
295 return false, nil
296 }
297 return false, i18n.E("ci.err.ruleExpr", op)
298 }
299 ok := re.MatchString(left.text)
300 if op == "!~" {
301 ok = !ok
302 }
303 return ok, nil
304 default:
305 return false, i18n.E("ci.err.ruleExpr", op)
306 }
307 }
308
309 // checkRule keurt de vorm goed zonder variabelen. Een patroon in een
310 // variabele wordt pas bij het draaien gecompileerd.
311 func checkRule(expr string) error {
312 p := &ruleParser{s: expr, env: map[string]string{}, syntax: true}
313 if _, err := p.parseOr(); err != nil {
314 return err
315 }
316 p.skip()
317 if p.i < len(p.s) {
318 return i18n.E("ci.err.ruleExpr", strings.TrimSpace(expr))
319 }
320 return nil
321 }
322
323 func ident(s string) bool {
324 if s == "" {
325 return false
326 }
327 for i := 0; i < len(s); i++ {
328 if !isIdent(s[i]) {
329 return false
330 }
331 }
332 return true
333 }
334
335 func isIdent(c byte) bool {
336 return c == '_' || (c >= 'A' && c <= 'Z') || (c >= 'a' && c <= 'z') || (c >= '0' && c <= '9')
337 }
@@ -0,0 +1,337 @@
1package ci
2
3import (
4 "regexp"
5 "strings"
6 "unicode"
7
8 "gitlab.com/Timfa/bluegit/internal/i18n"
9)
10
11// ruleParser leest een GitLab-achtige rules:if.
12// && bindt sterker dan ||. Haakjes groeperen. Een lege waarde is null.
13type ruleParser struct {
14 s string
15 i int
16 env map[string]string
17 syntax bool
18}
19
20func (p *ruleParser) parseOr() (bool, error) {
21 left, err := p.parseAnd()
22 if err != nil {
23 return false, err
24 }
25 for {
26 p.skip()
27 if !p.take("||") {
28 return left, nil
29 }
30 right, err := p.parseAnd()
31 if err != nil {
32 return false, err
33 }
34 left = left || right
35 }
36}
37
38func (p *ruleParser) parseAnd() (bool, error) {
39 left, err := p.parseCmp()
40 if err != nil {
41 return false, err
42 }
43 for {
44 p.skip()
45 if !p.take("&&") {
46 return left, nil
47 }
48 right, err := p.parseCmp()
49 if err != nil {
50 return false, err
51 }
52 left = left && right
53 }
54}
55
56func (p *ruleParser) parseCmp() (bool, error) {
57 p.skip()
58 if p.peek() == '(' {
59 p.i++
60 inner, err := p.parseOr()
61 if err != nil {
62 return false, err
63 }
64 p.skip()
65 if p.peek() != ')' {
66 return false, i18n.E("ci.err.ruleExpr", strings.TrimSpace(p.s))
67 }
68 p.i++
69 return inner, nil
70 }
71 left, err := p.value()
72 if err != nil {
73 return false, err
74 }
75 p.skip()
76 op := p.operator()
77 if op == "" {
78 return false, i18n.E("ci.err.ruleExpr", strings.TrimSpace(p.s))
79 }
80 right, err := p.value()
81 if err != nil {
82 return false, err
83 }
84 return compare(op, left, right, p.syntax)
85}
86
87type ruleValue struct {
88 text string
89 null bool
90 re *regexp.Regexp
91}
92
93func (p *ruleParser) value() (ruleValue, error) {
94 p.skip()
95 if p.i >= len(p.s) {
96 return ruleValue{}, i18n.E("ci.err.ruleExpr", strings.TrimSpace(p.s))
97 }
98 switch p.s[p.i] {
99 case '$':
100 name, err := p.variable()
101 if err != nil {
102 return ruleValue{}, err
103 }
104 text := p.env[name]
105 return ruleValue{text: text, null: text == ""}, nil
106 case '"', '\'':
107 text, err := p.quoted()
108 if err != nil {
109 return ruleValue{}, err
110 }
111 return ruleValue{text: text, null: text == ""}, nil
112 case '/':
113 re, err := p.regex()
114 if err != nil {
115 return ruleValue{}, err
116 }
117 return ruleValue{re: re}, nil
118 default:
119 if p.takeWord("null") {
120 return ruleValue{null: true}, nil
121 }
122 return ruleValue{}, i18n.E("ci.err.ruleExpr", strings.TrimSpace(p.s))
123 }
124}
125
126func (p *ruleParser) variable() (string, error) {
127 p.i++
128 if p.i < len(p.s) && p.s[p.i] == '{' {
129 p.i++
130 start := p.i
131 for p.i < len(p.s) && p.s[p.i] != '}' {
132 p.i++
133 }
134 if p.i >= len(p.s) || start == p.i {
135 return "", i18n.E("ci.err.ruleExpr", strings.TrimSpace(p.s))
136 }
137 name := p.s[start:p.i]
138 p.i++
139 if !ident(name) {
140 return "", i18n.E("ci.err.ruleExpr", strings.TrimSpace(p.s))
141 }
142 return name, nil
143 }
144 start := p.i
145 for p.i < len(p.s) && (isIdent(p.s[p.i])) {
146 p.i++
147 }
148 if start == p.i {
149 return "", i18n.E("ci.err.ruleExpr", strings.TrimSpace(p.s))
150 }
151 return p.s[start:p.i], nil
152}
153
154func (p *ruleParser) quoted() (string, error) {
155 quote := p.s[p.i]
156 p.i++
157 var b strings.Builder
158 for p.i < len(p.s) {
159 c := p.s[p.i]
160 if c == quote {
161 p.i++
162 return b.String(), nil
163 }
164 if c == '\\' && quote == '"' && p.i+1 < len(p.s) {
165 p.i++
166 b.WriteByte(p.s[p.i])
167 p.i++
168 continue
169 }
170 if c == '$' && quote == '"' {
171 name, err := p.variable()
172 if err != nil {
173 return "", err
174 }
175 b.WriteString(p.env[name])
176 continue
177 }
178 b.WriteByte(c)
179 p.i++
180 }
181 return "", i18n.E("ci.err.ruleExpr", strings.TrimSpace(p.s))
182}
183
184func (p *ruleParser) regex() (*regexp.Regexp, error) {
185 p.i++
186 var b strings.Builder
187 for p.i < len(p.s) {
188 c := p.s[p.i]
189 if c == '\\' && p.i+1 < len(p.s) {
190 next := p.s[p.i+1]
191 if next == '/' {
192 b.WriteByte('/')
193 } else {
194 b.WriteByte('\\')
195 b.WriteByte(next)
196 }
197 p.i += 2
198 continue
199 }
200 if c == '/' {
201 p.i++
202 pat := b.String()
203 if p.i < len(p.s) && (p.s[p.i] == 'i' || p.s[p.i] == 'I') {
204 pat = "(?i)" + pat
205 p.i++
206 }
207 re, err := regexp.Compile(pat)
208 if err != nil {
209 return nil, i18n.E("ci.err.regex", strings.TrimSpace(p.s), err.Error())
210 }
211 return re, nil
212 }
213 b.WriteByte(c)
214 p.i++
215 }
216 return nil, i18n.E("ci.err.ruleExpr", strings.TrimSpace(p.s))
217}
218
219func (p *ruleParser) operator() string {
220 p.skip()
221 for _, op := range []string{"=~", "!~", "==", "!="} {
222 if p.take(op) {
223 return op
224 }
225 }
226 return ""
227}
228
229func (p *ruleParser) take(lit string) bool {
230 if strings.HasPrefix(p.s[p.i:], lit) {
231 p.i += len(lit)
232 return true
233 }
234 return false
235}
236
237func (p *ruleParser) takeWord(word string) bool {
238 if !strings.HasPrefix(p.s[p.i:], word) {
239 return false
240 }
241 end := p.i + len(word)
242 if end < len(p.s) && isIdent(p.s[end]) {
243 return false
244 }
245 p.i = end
246 return true
247}
248
249func (p *ruleParser) peek() byte {
250 p.skip()
251 if p.i >= len(p.s) {
252 return 0
253 }
254 return p.s[p.i]
255}
256
257func (p *ruleParser) skip() {
258 for p.i < len(p.s) && unicode.IsSpace(rune(p.s[p.i])) {
259 p.i++
260 }
261}
262
263func compare(op string, left, right ruleValue, syntax bool) (bool, error) {
264 switch op {
265 case "==":
266 if left.re != nil || right.re != nil {
267 return false, i18n.E("ci.err.ruleExpr", op)
268 }
269 if left.null || right.null {
270 return left.null && right.null, nil
271 }
272 return left.text == right.text, nil
273 case "!=":
274 if left.re != nil || right.re != nil {
275 return false, i18n.E("ci.err.ruleExpr", op)
276 }
277 if left.null || right.null {
278 return left.null != right.null, nil
279 }
280 return left.text != right.text, nil
281 case "=~", "!~":
282 if left.re != nil {
283 return false, i18n.E("ci.err.ruleExpr", op)
284 }
285 re := right.re
286 if re == nil && !right.null {
287 compiled, err := regexp.Compile(right.text)
288 if err != nil {
289 return false, i18n.E("ci.err.regex", right.text, err.Error())
290 }
291 re = compiled
292 }
293 if re == nil {
294 if syntax && right.null && right.text == "" && right.re == nil {
295 return false, nil
296 }
297 return false, i18n.E("ci.err.ruleExpr", op)
298 }
299 ok := re.MatchString(left.text)
300 if op == "!~" {
301 ok = !ok
302 }
303 return ok, nil
304 default:
305 return false, i18n.E("ci.err.ruleExpr", op)
306 }
307}
308
309// checkRule keurt de vorm goed zonder variabelen. Een patroon in een
310// variabele wordt pas bij het draaien gecompileerd.
311func checkRule(expr string) error {
312 p := &ruleParser{s: expr, env: map[string]string{}, syntax: true}
313 if _, err := p.parseOr(); err != nil {
314 return err
315 }
316 p.skip()
317 if p.i < len(p.s) {
318 return i18n.E("ci.err.ruleExpr", strings.TrimSpace(expr))
319 }
320 return nil
321}
322
323func ident(s string) bool {
324 if s == "" {
325 return false
326 }
327 for i := 0; i < len(s); i++ {
328 if !isIdent(s[i]) {
329 return false
330 }
331 }
332 return true
333}
334
335func isIdent(c byte) bool {
336 return c == '_' || (c >= 'A' && c <= 'Z') || (c >= 'a' && c <= 'z') || (c >= '0' && c <= '9')
337}
internal/ci/expr_test.go added +162 −0
@@ -0,0 +1,162 @@
1 package ci
2
3 import "testing"
4
5 func TestRuleExpr(t *testing.T) {
6 env := map[string]string{
7 "CI_COMMIT_BRANCH": "master",
8 "CI_DEFAULT_BRANCH": "master",
9 "CI_COMMIT_TAG": "",
10 "CI_PIPELINE_SOURCE": "push",
11 "PATTERN": `^release/.*`,
12 "MSG": "Hello.World",
13 }
14 cases := []struct {
15 expr string
16 ok bool
17 }{
18 {`$CI_COMMIT_BRANCH == "master"`, true},
19 {`$CI_COMMIT_BRANCH == "feature"`, false},
20 {`$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH && $CI_PIPELINE_SOURCE == "push"`, true},
21 {`$CI_COMMIT_BRANCH == "nope" || $CI_PIPELINE_SOURCE == "push"`, true},
22 {`($CI_COMMIT_BRANCH == "nope" || $CI_COMMIT_BRANCH == "master") && $CI_COMMIT_TAG == null`, true},
23 {`$CI_COMMIT_TAG == null`, true},
24 {`$CI_COMMIT_BRANCH != null`, true},
25 {`$CI_COMMIT_BRANCH =~ /^master$/`, true},
26 {`$CI_COMMIT_BRANCH =~ /feature/`, false},
27 {`$CI_COMMIT_BRANCH !~ /feature/`, true},
28 {`$CI_COMMIT_BRANCH =~ /^MASTER$/i`, true},
29 {`$MSG =~ /Hello\.World/`, true},
30 {`$MSG =~ /Hello.World/`, true},
31 {`$CI_COMMIT_BRANCH =~ $PATTERN`, false},
32 {`$CI_COMMIT_BRANCH =~ /^release\/.*/ || $CI_COMMIT_BRANCH == "master"`, true},
33 {`"$CI_COMMIT_BRANCH" == "master"`, true},
34 {`${CI_COMMIT_BRANCH} == 'master'`, true},
35 }
36 for _, tc := range cases {
37 ok, err := evalRule(tc.expr, env)
38 if err != nil {
39 t.Fatalf("%s: %v", tc.expr, err)
40 }
41 if ok != tc.ok {
42 t.Fatalf("%s: got %v", tc.expr, ok)
43 }
44 }
45 if err := checkRule(`$FOO =~ $PATTERN`); err != nil {
46 t.Fatal(err)
47 }
48 if _, err := evalRule(`$CI_COMMIT_BRANCH ==`, env); err == nil {
49 t.Fatal("kapotte expressie")
50 }
51 if _, err := evalRule(`$MSG =~ /(/`, env); err == nil {
52 t.Fatal("kapotte regex")
53 }
54 if _, err := evalRule(`($CI_COMMIT_BRANCH == "master"`, env); err == nil {
55 t.Fatal("haakje")
56 }
57 }
58
59 func TestWorkflowParallelServices(t *testing.T) {
60 raw := []byte(`
61 workflow:
62 rules:
63 - if: $CI_COMMIT_BRANCH == "master" && $CI_PIPELINE_SOURCE == "push"
64 - when: never
65 default:
66 interruptible: true
67 retry: 2
68 build:
69 image: alpine:3.20
70 script: ["echo $CI_NODE_INDEX"]
71 parallel: 2
72 environment:
73 name: review
74 url: https://example.test/$CI_COMMIT_REF_SLUG
75 pack:
76 image: alpine:3.20
77 script: ["echo pack"]
78 needs: [build]
79 retry:
80 max: 5
81 when: [runner_system_failure, stuck_or_timeout_failure]
82 services:
83 - name: postgres:16
84 alias: db
85 variables:
86 POSTGRES_PASSWORD: secret
87 - redis:7
88 interruptible: false
89 matrix:
90 image: alpine:3.20
91 script: ["echo matrix"]
92 parallel:
93 matrix:
94 - PROVIDER: [aws, gcp]
95 STACK: app
96 - PROVIDER: aws
97 STACK: [backup]
98 `)
99 p, err := Parse(raw)
100 if err != nil {
101 t.Fatal(err)
102 }
103 if len(p.Workflow) != 2 {
104 t.Fatalf("workflow %d", len(p.Workflow))
105 }
106 ok, err := p.Allow(Eval{Env: map[string]string{"CI_COMMIT_BRANCH": "feature", "CI_PIPELINE_SOURCE": "push"}})
107 if err != nil || ok {
108 t.Fatalf("feature hoort geen pipeline: %v %v", ok, err)
109 }
110 ok, err = p.Allow(Eval{Env: map[string]string{"CI_COMMIT_BRANCH": "master", "CI_PIPELINE_SOURCE": "push"}})
111 if err != nil || !ok {
112 t.Fatalf("master hoort wel: %v %v", ok, err)
113 }
114 jobs, err := p.Select(Eval{Env: map[string]string{"CI_COMMIT_BRANCH": "master", "CI_PIPELINE_SOURCE": "push"}})
115 if err != nil {
116 t.Fatal(err)
117 }
118 byName := map[string]Job{}
119 for _, j := range jobs {
120 byName[j.Name] = j
121 }
122 if _, ok := byName["build 1/2"]; !ok || byName["build 1/2"].Variables["CI_NODE_TOTAL"] != "2" {
123 t.Fatalf("parallel: %v", namesOf(jobs))
124 }
125 if !byName["build 1/2"].Interruptible || byName["build 1/2"].Retry.Max != 2 {
126 t.Fatalf("default: %+v", byName["build 1/2"])
127 }
128 if byName["build 1/2"].Environment.Name != "review" || byName["build 1/2"].Environment.URL == "" {
129 t.Fatalf("environment: %+v", byName["build 1/2"].Environment)
130 }
131 pack := byName["pack"]
132 if pack.Interruptible || pack.Retry.Max != 2 || len(pack.Retry.When) != 2 {
133 t.Fatalf("retry geklemd: %+v", pack.Retry)
134 }
135 if len(pack.Needs) != 2 || pack.Needs[0] != "build 1/2" || pack.Needs[1] != "build 2/2" {
136 t.Fatalf("needs: %v", pack.Needs)
137 }
138 if len(pack.Services) != 2 || pack.Services[0].Alias != "db" || pack.Services[1].Alias != "redis" {
139 t.Fatalf("services: %+v", pack.Services)
140 }
141 if _, ok := byName["matrix: [aws, app]"]; !ok {
142 t.Fatalf("matrix: %v", namesOf(jobs))
143 }
144 if byName["matrix: [gcp, app]"].Variables["PROVIDER"] != "gcp" || byName["matrix: [aws, backup]"].Variables["STACK"] != "backup" {
145 t.Fatalf("matrix vars")
146 }
147 if got := Expand("https://example.test/$CI_COMMIT_REF_SLUG", map[string]string{"CI_COMMIT_REF_SLUG": "master"}); got != "https://example.test/master" {
148 t.Fatal(got)
149 }
150 }
151
152 func TestGateCanceled(t *testing.T) {
153 jobs := []Job{{Name: "a", Stage: "test"}, {Name: "b", Stage: "deploy"}}
154 status := map[string]string{"a": "canceled", "b": "queued"}
155 if got := Gate(jobs, status, jobs[1]); got != "skip" {
156 t.Fatalf("canceled houdt de volgende tegen: %s", got)
157 }
158 jobs[0].AllowFailure = true
159 if got := Gate(jobs, status, jobs[1]); got != "ready" {
160 t.Fatalf("allow_failure canceled: %s", got)
161 }
162 }
@@ -0,0 +1,162 @@
1package ci
2
3import "testing"
4
5func TestRuleExpr(t *testing.T) {
6 env := map[string]string{
7 "CI_COMMIT_BRANCH": "master",
8 "CI_DEFAULT_BRANCH": "master",
9 "CI_COMMIT_TAG": "",
10 "CI_PIPELINE_SOURCE": "push",
11 "PATTERN": `^release/.*`,
12 "MSG": "Hello.World",
13 }
14 cases := []struct {
15 expr string
16 ok bool
17 }{
18 {`$CI_COMMIT_BRANCH == "master"`, true},
19 {`$CI_COMMIT_BRANCH == "feature"`, false},
20 {`$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH && $CI_PIPELINE_SOURCE == "push"`, true},
21 {`$CI_COMMIT_BRANCH == "nope" || $CI_PIPELINE_SOURCE == "push"`, true},
22 {`($CI_COMMIT_BRANCH == "nope" || $CI_COMMIT_BRANCH == "master") && $CI_COMMIT_TAG == null`, true},
23 {`$CI_COMMIT_TAG == null`, true},
24 {`$CI_COMMIT_BRANCH != null`, true},
25 {`$CI_COMMIT_BRANCH =~ /^master$/`, true},
26 {`$CI_COMMIT_BRANCH =~ /feature/`, false},
27 {`$CI_COMMIT_BRANCH !~ /feature/`, true},
28 {`$CI_COMMIT_BRANCH =~ /^MASTER$/i`, true},
29 {`$MSG =~ /Hello\.World/`, true},
30 {`$MSG =~ /Hello.World/`, true},
31 {`$CI_COMMIT_BRANCH =~ $PATTERN`, false},
32 {`$CI_COMMIT_BRANCH =~ /^release\/.*/ || $CI_COMMIT_BRANCH == "master"`, true},
33 {`"$CI_COMMIT_BRANCH" == "master"`, true},
34 {`${CI_COMMIT_BRANCH} == 'master'`, true},
35 }
36 for _, tc := range cases {
37 ok, err := evalRule(tc.expr, env)
38 if err != nil {
39 t.Fatalf("%s: %v", tc.expr, err)
40 }
41 if ok != tc.ok {
42 t.Fatalf("%s: got %v", tc.expr, ok)
43 }
44 }
45 if err := checkRule(`$FOO =~ $PATTERN`); err != nil {
46 t.Fatal(err)
47 }
48 if _, err := evalRule(`$CI_COMMIT_BRANCH ==`, env); err == nil {
49 t.Fatal("kapotte expressie")
50 }
51 if _, err := evalRule(`$MSG =~ /(/`, env); err == nil {
52 t.Fatal("kapotte regex")
53 }
54 if _, err := evalRule(`($CI_COMMIT_BRANCH == "master"`, env); err == nil {
55 t.Fatal("haakje")
56 }
57}
58
59func TestWorkflowParallelServices(t *testing.T) {
60 raw := []byte(`
61workflow:
62 rules:
63 - if: $CI_COMMIT_BRANCH == "master" && $CI_PIPELINE_SOURCE == "push"
64 - when: never
65default:
66 interruptible: true
67 retry: 2
68build:
69 image: alpine:3.20
70 script: ["echo $CI_NODE_INDEX"]
71 parallel: 2
72 environment:
73 name: review
74 url: https://example.test/$CI_COMMIT_REF_SLUG
75pack:
76 image: alpine:3.20
77 script: ["echo pack"]
78 needs: [build]
79 retry:
80 max: 5
81 when: [runner_system_failure, stuck_or_timeout_failure]
82 services:
83 - name: postgres:16
84 alias: db
85 variables:
86 POSTGRES_PASSWORD: secret
87 - redis:7
88 interruptible: false
89matrix:
90 image: alpine:3.20
91 script: ["echo matrix"]
92 parallel:
93 matrix:
94 - PROVIDER: [aws, gcp]
95 STACK: app
96 - PROVIDER: aws
97 STACK: [backup]
98`)
99 p, err := Parse(raw)
100 if err != nil {
101 t.Fatal(err)
102 }
103 if len(p.Workflow) != 2 {
104 t.Fatalf("workflow %d", len(p.Workflow))
105 }
106 ok, err := p.Allow(Eval{Env: map[string]string{"CI_COMMIT_BRANCH": "feature", "CI_PIPELINE_SOURCE": "push"}})
107 if err != nil || ok {
108 t.Fatalf("feature hoort geen pipeline: %v %v", ok, err)
109 }
110 ok, err = p.Allow(Eval{Env: map[string]string{"CI_COMMIT_BRANCH": "master", "CI_PIPELINE_SOURCE": "push"}})
111 if err != nil || !ok {
112 t.Fatalf("master hoort wel: %v %v", ok, err)
113 }
114 jobs, err := p.Select(Eval{Env: map[string]string{"CI_COMMIT_BRANCH": "master", "CI_PIPELINE_SOURCE": "push"}})
115 if err != nil {
116 t.Fatal(err)
117 }
118 byName := map[string]Job{}
119 for _, j := range jobs {
120 byName[j.Name] = j
121 }
122 if _, ok := byName["build 1/2"]; !ok || byName["build 1/2"].Variables["CI_NODE_TOTAL"] != "2" {
123 t.Fatalf("parallel: %v", namesOf(jobs))
124 }
125 if !byName["build 1/2"].Interruptible || byName["build 1/2"].Retry.Max != 2 {
126 t.Fatalf("default: %+v", byName["build 1/2"])
127 }
128 if byName["build 1/2"].Environment.Name != "review" || byName["build 1/2"].Environment.URL == "" {
129 t.Fatalf("environment: %+v", byName["build 1/2"].Environment)
130 }
131 pack := byName["pack"]
132 if pack.Interruptible || pack.Retry.Max != 2 || len(pack.Retry.When) != 2 {
133 t.Fatalf("retry geklemd: %+v", pack.Retry)
134 }
135 if len(pack.Needs) != 2 || pack.Needs[0] != "build 1/2" || pack.Needs[1] != "build 2/2" {
136 t.Fatalf("needs: %v", pack.Needs)
137 }
138 if len(pack.Services) != 2 || pack.Services[0].Alias != "db" || pack.Services[1].Alias != "redis" {
139 t.Fatalf("services: %+v", pack.Services)
140 }
141 if _, ok := byName["matrix: [aws, app]"]; !ok {
142 t.Fatalf("matrix: %v", namesOf(jobs))
143 }
144 if byName["matrix: [gcp, app]"].Variables["PROVIDER"] != "gcp" || byName["matrix: [aws, backup]"].Variables["STACK"] != "backup" {
145 t.Fatalf("matrix vars")
146 }
147 if got := Expand("https://example.test/$CI_COMMIT_REF_SLUG", map[string]string{"CI_COMMIT_REF_SLUG": "master"}); got != "https://example.test/master" {
148 t.Fatal(got)
149 }
150}
151
152func TestGateCanceled(t *testing.T) {
153 jobs := []Job{{Name: "a", Stage: "test"}, {Name: "b", Stage: "deploy"}}
154 status := map[string]string{"a": "canceled", "b": "queued"}
155 if got := Gate(jobs, status, jobs[1]); got != "skip" {
156 t.Fatalf("canceled houdt de volgende tegen: %s", got)
157 }
158 jobs[0].AllowFailure = true
159 if got := Gate(jobs, status, jobs[1]); got != "ready" {
160 t.Fatalf("allow_failure canceled: %s", got)
161 }
162}
internal/ci/extra.go added +278 −0
@@ -0,0 +1,278 @@
1 package ci
2
3 import (
4 "strconv"
5 "strings"
6
7 "gopkg.in/yaml.v3"
8 )
9
10 func parseWorkflow(node *yaml.Node) ([]Rule, []Issue) {
11 if node.Kind != yaml.MappingNode {
12 return nil, []Issue{{Line: node.Line, Key: "ci.err.map"}}
13 }
14 var rules []Rule
15 var issues []Issue
16 for i := 0; i < len(node.Content); i += 2 {
17 key := node.Content[i].Value
18 val := node.Content[i+1]
19 if key == "rules" {
20 parsed, iss := parseRules(val)
21 issues = append(issues, iss...)
22 rules = parsed
23 continue
24 }
25 issues = append(issues, Issue{Line: node.Content[i].Line, Key: "ci.warn.ignored", Args: []any{"workflow." + key}, Warning: true})
26 }
27 return rules, issues
28 }
29
30 func parseRetry(node *yaml.Node) (Retry, []Issue) {
31 switch node.Kind {
32 case yaml.ScalarNode:
33 n, iss := retryMax(node)
34 if len(iss) > 0 && n == 0 && iss[0].Key != "ci.warn.retryMax" {
35 return Retry{}, iss
36 }
37 return Retry{Max: n}, iss
38 case yaml.MappingNode:
39 var retry Retry
40 var issues []Issue
41 for i := 0; i < len(node.Content); i += 2 {
42 key := node.Content[i].Value
43 val := node.Content[i+1]
44 switch key {
45 case "max":
46 n, iss := retryMax(val)
47 issues = append(issues, iss...)
48 retry.Max = n
49 case "when":
50 list, iss := retryWhen(val)
51 issues = append(issues, iss...)
52 retry.When = list
53 default:
54 issues = append(issues, Issue{Line: node.Content[i].Line, Key: "ci.warn.ignored", Args: []any{"retry." + key}, Warning: true})
55 }
56 }
57 return retry, issues
58 default:
59 return Retry{}, []Issue{{Line: node.Line, Key: "ci.err.retry"}}
60 }
61 }
62
63 func retryMax(node *yaml.Node) (int, []Issue) {
64 if node.Kind != yaml.ScalarNode {
65 return 0, []Issue{{Line: node.Line, Key: "ci.err.retry"}}
66 }
67 n, err := strconv.Atoi(strings.TrimSpace(node.Value))
68 if err != nil || n < 0 {
69 return 0, []Issue{{Line: node.Line, Key: "ci.err.retry"}}
70 }
71 if n > 2 {
72 return 2, []Issue{{Line: node.Line, Key: "ci.warn.retryMax", Args: []any{n, 2}, Warning: true}}
73 }
74 return n, nil
75 }
76
77 func retryWhen(node *yaml.Node) ([]string, []Issue) {
78 if node.Kind == yaml.ScalarNode {
79 if strings.TrimSpace(node.Value) == "" {
80 return nil, []Issue{{Line: node.Line, Key: "ci.err.retry"}}
81 }
82 return []string{node.Value}, nil
83 }
84 return scalarList(node, "ci.err.list")
85 }
86
87 func parseParallel(node *yaml.Node) (int, []MatrixSpec, []Issue) {
88 if node.Kind == yaml.ScalarNode {
89 n, err := strconv.Atoi(strings.TrimSpace(node.Value))
90 if err != nil || n < 1 {
91 return 0, nil, []Issue{{Line: node.Line, Key: "ci.err.parallelForm", Args: []any{maxParallel}}}
92 }
93 if n > maxParallel {
94 return 0, nil, []Issue{{Line: node.Line, Key: "ci.err.parallel", Args: []any{n, maxParallel}}}
95 }
96 return n, nil, nil
97 }
98 if node.Kind != yaml.MappingNode {
99 return 0, nil, []Issue{{Line: node.Line, Key: "ci.err.parallelForm", Args: []any{maxParallel}}}
100 }
101 var specs []MatrixSpec
102 var issues []Issue
103 seen := false
104 for i := 0; i < len(node.Content); i += 2 {
105 key := node.Content[i].Value
106 val := node.Content[i+1]
107 if key != "matrix" {
108 issues = append(issues, Issue{Line: node.Content[i].Line, Key: "ci.warn.ignored", Args: []any{"parallel." + key}, Warning: true})
109 continue
110 }
111 seen = true
112 parsed, iss := parseMatrix(val)
113 issues = append(issues, iss...)
114 specs = parsed
115 }
116 if !seen {
117 issues = append(issues, Issue{Line: node.Line, Key: "ci.err.parallelForm", Args: []any{maxParallel}})
118 }
119 count := 0
120 for _, spec := range specs {
121 count += len(cartesian(spec))
122 }
123 if count > maxParallel {
124 issues = append(issues, Issue{Line: node.Line, Key: "ci.err.parallel", Args: []any{count, maxParallel}})
125 }
126 return 0, specs, issues
127 }
128
129 func parseMatrix(node *yaml.Node) ([]MatrixSpec, []Issue) {
130 if node.Kind != yaml.SequenceNode || len(node.Content) == 0 {
131 return nil, []Issue{{Line: node.Line, Key: "ci.err.parallelForm", Args: []any{maxParallel}}}
132 }
133 var specs []MatrixSpec
134 var issues []Issue
135 for _, item := range node.Content {
136 if item.Kind != yaml.MappingNode {
137 issues = append(issues, Issue{Line: item.Line, Key: "ci.err.parallelForm", Args: []any{maxParallel}})
138 continue
139 }
140 spec := MatrixSpec{}
141 for i := 0; i < len(item.Content); i += 2 {
142 key := item.Content[i].Value
143 vals, iss := matrixValues(item.Content[i+1])
144 issues = append(issues, iss...)
145 if len(vals) == 0 {
146 continue
147 }
148 spec.Keys = append(spec.Keys, key)
149 spec.Vals = append(spec.Vals, vals)
150 }
151 if len(spec.Keys) == 0 {
152 issues = append(issues, Issue{Line: item.Line, Key: "ci.err.parallelForm", Args: []any{maxParallel}})
153 continue
154 }
155 specs = append(specs, spec)
156 }
157 return specs, issues
158 }
159
160 func matrixValues(node *yaml.Node) ([]string, []Issue) {
161 if node.Kind == yaml.ScalarNode {
162 return []string{node.Value}, nil
163 }
164 return scalarList(node, "ci.err.list")
165 }
166
167 func parseServices(node *yaml.Node) ([]Service, []Issue) {
168 var items []*yaml.Node
169 switch node.Kind {
170 case yaml.SequenceNode:
171 items = node.Content
172 default:
173 items = []*yaml.Node{node}
174 }
175 var out []Service
176 var issues []Issue
177 for _, item := range items {
178 switch item.Kind {
179 case yaml.ScalarNode:
180 if strings.TrimSpace(item.Value) == "" {
181 issues = append(issues, Issue{Line: item.Line, Key: "ci.err.service"})
182 continue
183 }
184 out = append(out, Service{Name: item.Value, Alias: serviceAlias(item.Value)})
185 case yaml.MappingNode:
186 svc := Service{Variables: map[string]string{}}
187 for i := 0; i < len(item.Content); i += 2 {
188 key := item.Content[i].Value
189 val := item.Content[i+1]
190 switch key {
191 case "name":
192 if val.Kind == yaml.ScalarNode {
193 svc.Name = val.Value
194 }
195 case "alias":
196 if val.Kind == yaml.ScalarNode {
197 svc.Alias = val.Value
198 }
199 case "command":
200 list, iss := scriptList(val)
201 issues = append(issues, iss...)
202 svc.Command = list
203 case "entrypoint":
204 list, iss := scriptList(val)
205 issues = append(issues, iss...)
206 svc.Entrypoint = list
207 case "variables":
208 vars, iss := scalarMap(val)
209 issues = append(issues, iss...)
210 svc.Variables = vars
211 default:
212 issues = append(issues, Issue{Line: item.Content[i].Line, Key: "ci.warn.ignored", Args: []any{"services." + key}, Warning: true})
213 }
214 }
215 if strings.TrimSpace(svc.Name) == "" {
216 issues = append(issues, Issue{Line: item.Line, Key: "ci.err.service"})
217 continue
218 }
219 if svc.Alias == "" {
220 svc.Alias = serviceAlias(svc.Name)
221 }
222 out = append(out, svc)
223 default:
224 issues = append(issues, Issue{Line: item.Line, Key: "ci.err.service"})
225 }
226 }
227 return out, issues
228 }
229
230 func serviceAlias(image string) string {
231 name := image
232 if i := strings.LastIndex(name, "/"); i >= 0 {
233 name = name[i+1:]
234 }
235 if i := strings.IndexAny(name, ":@"); i >= 0 {
236 name = name[:i]
237 }
238 name = strings.TrimSpace(name)
239 if name == "" {
240 return "service"
241 }
242 return name
243 }
244
245 func parseEnvironment(node *yaml.Node) (Environment, []Issue) {
246 switch node.Kind {
247 case yaml.ScalarNode:
248 if strings.TrimSpace(node.Value) == "" {
249 return Environment{}, []Issue{{Line: node.Line, Key: "ci.err.environment"}}
250 }
251 return Environment{Name: node.Value}, nil
252 case yaml.MappingNode:
253 var env Environment
254 var issues []Issue
255 for i := 0; i < len(node.Content); i += 2 {
256 key := node.Content[i].Value
257 val := node.Content[i+1]
258 switch key {
259 case "name":
260 if val.Kind == yaml.ScalarNode {
261 env.Name = val.Value
262 }
263 case "url":
264 if val.Kind == yaml.ScalarNode {
265 env.URL = val.Value
266 }
267 default:
268 issues = append(issues, Issue{Line: node.Content[i].Line, Key: "ci.warn.ignored", Args: []any{"environment." + key}, Warning: true})
269 }
270 }
271 if strings.TrimSpace(env.Name) == "" {
272 issues = append(issues, Issue{Line: node.Line, Key: "ci.err.environment"})
273 }
274 return env, issues
275 default:
276 return Environment{}, []Issue{{Line: node.Line, Key: "ci.err.environment"}}
277 }
278 }
@@ -0,0 +1,278 @@
1package ci
2
3import (
4 "strconv"
5 "strings"
6
7 "gopkg.in/yaml.v3"
8)
9
10func parseWorkflow(node *yaml.Node) ([]Rule, []Issue) {
11 if node.Kind != yaml.MappingNode {
12 return nil, []Issue{{Line: node.Line, Key: "ci.err.map"}}
13 }
14 var rules []Rule
15 var issues []Issue
16 for i := 0; i < len(node.Content); i += 2 {
17 key := node.Content[i].Value
18 val := node.Content[i+1]
19 if key == "rules" {
20 parsed, iss := parseRules(val)
21 issues = append(issues, iss...)
22 rules = parsed
23 continue
24 }
25 issues = append(issues, Issue{Line: node.Content[i].Line, Key: "ci.warn.ignored", Args: []any{"workflow." + key}, Warning: true})
26 }
27 return rules, issues
28}
29
30func parseRetry(node *yaml.Node) (Retry, []Issue) {
31 switch node.Kind {
32 case yaml.ScalarNode:
33 n, iss := retryMax(node)
34 if len(iss) > 0 && n == 0 && iss[0].Key != "ci.warn.retryMax" {
35 return Retry{}, iss
36 }
37 return Retry{Max: n}, iss
38 case yaml.MappingNode:
39 var retry Retry
40 var issues []Issue
41 for i := 0; i < len(node.Content); i += 2 {
42 key := node.Content[i].Value
43 val := node.Content[i+1]
44 switch key {
45 case "max":
46 n, iss := retryMax(val)
47 issues = append(issues, iss...)
48 retry.Max = n
49 case "when":
50 list, iss := retryWhen(val)
51 issues = append(issues, iss...)
52 retry.When = list
53 default:
54 issues = append(issues, Issue{Line: node.Content[i].Line, Key: "ci.warn.ignored", Args: []any{"retry." + key}, Warning: true})
55 }
56 }
57 return retry, issues
58 default:
59 return Retry{}, []Issue{{Line: node.Line, Key: "ci.err.retry"}}
60 }
61}
62
63func retryMax(node *yaml.Node) (int, []Issue) {
64 if node.Kind != yaml.ScalarNode {
65 return 0, []Issue{{Line: node.Line, Key: "ci.err.retry"}}
66 }
67 n, err := strconv.Atoi(strings.TrimSpace(node.Value))
68 if err != nil || n < 0 {
69 return 0, []Issue{{Line: node.Line, Key: "ci.err.retry"}}
70 }
71 if n > 2 {
72 return 2, []Issue{{Line: node.Line, Key: "ci.warn.retryMax", Args: []any{n, 2}, Warning: true}}
73 }
74 return n, nil
75}
76
77func retryWhen(node *yaml.Node) ([]string, []Issue) {
78 if node.Kind == yaml.ScalarNode {
79 if strings.TrimSpace(node.Value) == "" {
80 return nil, []Issue{{Line: node.Line, Key: "ci.err.retry"}}
81 }
82 return []string{node.Value}, nil
83 }
84 return scalarList(node, "ci.err.list")
85}
86
87func parseParallel(node *yaml.Node) (int, []MatrixSpec, []Issue) {
88 if node.Kind == yaml.ScalarNode {
89 n, err := strconv.Atoi(strings.TrimSpace(node.Value))
90 if err != nil || n < 1 {
91 return 0, nil, []Issue{{Line: node.Line, Key: "ci.err.parallelForm", Args: []any{maxParallel}}}
92 }
93 if n > maxParallel {
94 return 0, nil, []Issue{{Line: node.Line, Key: "ci.err.parallel", Args: []any{n, maxParallel}}}
95 }
96 return n, nil, nil
97 }
98 if node.Kind != yaml.MappingNode {
99 return 0, nil, []Issue{{Line: node.Line, Key: "ci.err.parallelForm", Args: []any{maxParallel}}}
100 }
101 var specs []MatrixSpec
102 var issues []Issue
103 seen := false
104 for i := 0; i < len(node.Content); i += 2 {
105 key := node.Content[i].Value
106 val := node.Content[i+1]
107 if key != "matrix" {
108 issues = append(issues, Issue{Line: node.Content[i].Line, Key: "ci.warn.ignored", Args: []any{"parallel." + key}, Warning: true})
109 continue
110 }
111 seen = true
112 parsed, iss := parseMatrix(val)
113 issues = append(issues, iss...)
114 specs = parsed
115 }
116 if !seen {
117 issues = append(issues, Issue{Line: node.Line, Key: "ci.err.parallelForm", Args: []any{maxParallel}})
118 }
119 count := 0
120 for _, spec := range specs {
121 count += len(cartesian(spec))
122 }
123 if count > maxParallel {
124 issues = append(issues, Issue{Line: node.Line, Key: "ci.err.parallel", Args: []any{count, maxParallel}})
125 }
126 return 0, specs, issues
127}
128
129func parseMatrix(node *yaml.Node) ([]MatrixSpec, []Issue) {
130 if node.Kind != yaml.SequenceNode || len(node.Content) == 0 {
131 return nil, []Issue{{Line: node.Line, Key: "ci.err.parallelForm", Args: []any{maxParallel}}}
132 }
133 var specs []MatrixSpec
134 var issues []Issue
135 for _, item := range node.Content {
136 if item.Kind != yaml.MappingNode {
137 issues = append(issues, Issue{Line: item.Line, Key: "ci.err.parallelForm", Args: []any{maxParallel}})
138 continue
139 }
140 spec := MatrixSpec{}
141 for i := 0; i < len(item.Content); i += 2 {
142 key := item.Content[i].Value
143 vals, iss := matrixValues(item.Content[i+1])
144 issues = append(issues, iss...)
145 if len(vals) == 0 {
146 continue
147 }
148 spec.Keys = append(spec.Keys, key)
149 spec.Vals = append(spec.Vals, vals)
150 }
151 if len(spec.Keys) == 0 {
152 issues = append(issues, Issue{Line: item.Line, Key: "ci.err.parallelForm", Args: []any{maxParallel}})
153 continue
154 }
155 specs = append(specs, spec)
156 }
157 return specs, issues
158}
159
160func matrixValues(node *yaml.Node) ([]string, []Issue) {
161 if node.Kind == yaml.ScalarNode {
162 return []string{node.Value}, nil
163 }
164 return scalarList(node, "ci.err.list")
165}
166
167func parseServices(node *yaml.Node) ([]Service, []Issue) {
168 var items []*yaml.Node
169 switch node.Kind {
170 case yaml.SequenceNode:
171 items = node.Content
172 default:
173 items = []*yaml.Node{node}
174 }
175 var out []Service
176 var issues []Issue
177 for _, item := range items {
178 switch item.Kind {
179 case yaml.ScalarNode:
180 if strings.TrimSpace(item.Value) == "" {
181 issues = append(issues, Issue{Line: item.Line, Key: "ci.err.service"})
182 continue
183 }
184 out = append(out, Service{Name: item.Value, Alias: serviceAlias(item.Value)})
185 case yaml.MappingNode:
186 svc := Service{Variables: map[string]string{}}
187 for i := 0; i < len(item.Content); i += 2 {
188 key := item.Content[i].Value
189 val := item.Content[i+1]
190 switch key {
191 case "name":
192 if val.Kind == yaml.ScalarNode {
193 svc.Name = val.Value
194 }
195 case "alias":
196 if val.Kind == yaml.ScalarNode {
197 svc.Alias = val.Value
198 }
199 case "command":
200 list, iss := scriptList(val)
201 issues = append(issues, iss...)
202 svc.Command = list
203 case "entrypoint":
204 list, iss := scriptList(val)
205 issues = append(issues, iss...)
206 svc.Entrypoint = list
207 case "variables":
208 vars, iss := scalarMap(val)
209 issues = append(issues, iss...)
210 svc.Variables = vars
211 default:
212 issues = append(issues, Issue{Line: item.Content[i].Line, Key: "ci.warn.ignored", Args: []any{"services." + key}, Warning: true})
213 }
214 }
215 if strings.TrimSpace(svc.Name) == "" {
216 issues = append(issues, Issue{Line: item.Line, Key: "ci.err.service"})
217 continue
218 }
219 if svc.Alias == "" {
220 svc.Alias = serviceAlias(svc.Name)
221 }
222 out = append(out, svc)
223 default:
224 issues = append(issues, Issue{Line: item.Line, Key: "ci.err.service"})
225 }
226 }
227 return out, issues
228}
229
230func serviceAlias(image string) string {
231 name := image
232 if i := strings.LastIndex(name, "/"); i >= 0 {
233 name = name[i+1:]
234 }
235 if i := strings.IndexAny(name, ":@"); i >= 0 {
236 name = name[:i]
237 }
238 name = strings.TrimSpace(name)
239 if name == "" {
240 return "service"
241 }
242 return name
243}
244
245func parseEnvironment(node *yaml.Node) (Environment, []Issue) {
246 switch node.Kind {
247 case yaml.ScalarNode:
248 if strings.TrimSpace(node.Value) == "" {
249 return Environment{}, []Issue{{Line: node.Line, Key: "ci.err.environment"}}
250 }
251 return Environment{Name: node.Value}, nil
252 case yaml.MappingNode:
253 var env Environment
254 var issues []Issue
255 for i := 0; i < len(node.Content); i += 2 {
256 key := node.Content[i].Value
257 val := node.Content[i+1]
258 switch key {
259 case "name":
260 if val.Kind == yaml.ScalarNode {
261 env.Name = val.Value
262 }
263 case "url":
264 if val.Kind == yaml.ScalarNode {
265 env.URL = val.Value
266 }
267 default:
268 issues = append(issues, Issue{Line: node.Content[i].Line, Key: "ci.warn.ignored", Args: []any{"environment." + key}, Warning: true})
269 }
270 }
271 if strings.TrimSpace(env.Name) == "" {
272 issues = append(issues, Issue{Line: node.Line, Key: "ci.err.environment"})
273 }
274 return env, issues
275 default:
276 return Environment{}, []Issue{{Line: node.Line, Key: "ci.err.environment"}}
277 }
278}
internal/ci/model.go +35 −0
@@ -14,6 +14,7 @@ type Pipeline struct {
14 14 Tags []string
15 15 Jobs []Job
16 16 Warnings []Issue
17 Workflow []Rule
17 18 }
18 19
19 20 // Job is één containerstap.
@@ -38,6 +39,40 @@ type Job struct {
38 39 ResourceGroup string
39 40 Only []string
40 41 Except []string
42 Retry Retry
43 Parallel int
44 Matrix []MatrixSpec
45 Services []Service
46 Environment Environment
47 Interruptible bool
48 }
49
50 // Retry is het aantal extra pogingen na een mislukking. Max is hoogstens 2.
51 // Een lege When betekent elke mislukking behalve een annulering.
52 type Retry struct {
53 Max int
54 When []string
55 }
56
57 // MatrixSpec is één item uit parallel:matrix. Keys staan in YAML-volgorde.
58 type MatrixSpec struct {
59 Keys []string
60 Vals [][]string
61 }
62
63 // Service is een sidecar-container op hetzelfde Docker-netwerk als de job.
64 type Service struct {
65 Name string
66 Alias string
67 Command []string
68 Entrypoint []string
69 Variables map[string]string
70 }
71
72 // Environment is de naam en een optionele URL. Er is geen deployment.
73 type Environment struct {
74 Name string
75 URL string
41 76 }
42 77
43 78 // Artifacts zijn paden die na een geslaagde job bewaard blijven.
@@ -14,6 +14,7 @@ type Pipeline struct {
14 Tags []string
14 Tags []string
15 Jobs []Job
15 Jobs []Job
16 Warnings []Issue
16 Warnings []Issue
17 Workflow []Rule
17}
18}
18
19
19// Job is één containerstap.
20// Job is één containerstap.
@@ -38,6 +39,40 @@ type Job struct {
38 ResourceGroup string
39 ResourceGroup string
39 Only []string
40 Only []string
40 Except []string
41 Except []string
42 Retry Retry
43 Parallel int
44 Matrix []MatrixSpec
45 Services []Service
46 Environment Environment
47 Interruptible bool
48}
49
50// Retry is het aantal extra pogingen na een mislukking. Max is hoogstens 2.
51// Een lege When betekent elke mislukking behalve een annulering.
52type Retry struct {
53 Max int
54 When []string
55}
56
57// MatrixSpec is één item uit parallel:matrix. Keys staan in YAML-volgorde.
58type MatrixSpec struct {
59 Keys []string
60 Vals [][]string
61}
62
63// Service is een sidecar-container op hetzelfde Docker-netwerk als de job.
64type Service struct {
65 Name string
66 Alias string
67 Command []string
68 Entrypoint []string
69 Variables map[string]string
70}
71
72// Environment is de naam en een optionele URL. Er is geen deployment.
73type Environment struct {
74 Name string
75 URL string
41}
76}
42
77
43// Artifacts zijn paden die na een geslaagde job bewaard blijven.
78// Artifacts zijn paden die na een geslaagde job bewaard blijven.
internal/ci/parse.go +57 −2
@@ -153,7 +153,11 @@ func parseBytes(data []byte, read func(string) ([]byte, error), stack []string)
153 153 d, iss := parseDefault(val)
154 154 issues = append(issues, iss...)
155 155 def = overlayDefault(def, d)
156 case "workflow", "services", "spec", "pages":
156 case "workflow":
157 rules, iss := parseWorkflow(val)
158 issues = append(issues, iss...)
159 p.Workflow = rules
160 case "services", "spec", "pages":
157 161 issues = append(issues, Issue{Line: keyNode.Line, Key: "ci.warn.ignored", Args: []any{key}, Warning: true})
158 162 case "jobs":
159 163 seenLegacy = true
@@ -347,6 +351,16 @@ func parseDefault(node *yaml.Node) (draft, []Issue) {
347 351 issues = append(issues, iss...)
348 352 d.job.Timeout = dur
349 353 d.present["timeout"] = true
354 case "retry":
355 retry, iss := parseRetry(val)
356 issues = append(issues, iss...)
357 d.job.Retry = retry
358 d.present["retry"] = true
359 case "interruptible":
360 b, iss := parseBool(val)
361 issues = append(issues, iss...)
362 d.job.Interruptible = b
363 d.present["interruptible"] = true
350 364 case "cache":
351 365 c, iss := parseCache(val)
352 366 issues = append(issues, iss...)
@@ -385,6 +399,14 @@ func overlayJob(base Job, d draft) Job {
385 399 set("resource_group", func() { base.ResourceGroup = d.job.ResourceGroup })
386 400 set("only", func() { base.Only = d.job.Only })
387 401 set("except", func() { base.Except = d.job.Except })
402 set("retry", func() { base.Retry = d.job.Retry })
403 set("parallel", func() {
404 base.Parallel = d.job.Parallel
405 base.Matrix = d.job.Matrix
406 })
407 set("services", func() { base.Services = d.job.Services })
408 set("environment", func() { base.Environment = d.job.Environment })
409 set("interruptible", func() { base.Interruptible = d.job.Interruptible })
388 410 if d.present["variables"] {
389 411 mergedVars := map[string]string{}
390 412 for k, v := range base.Variables {
@@ -420,6 +442,10 @@ func overlayDefault(base, extra draft) draft {
420 442 base.job.Timeout = extra.job.Timeout
421 443 case "cache":
422 444 base.job.Cache = extra.job.Cache
445 case "retry":
446 base.job.Retry = extra.job.Retry
447 case "interruptible":
448 base.job.Interruptible = extra.job.Interruptible
423 449 }
424 450 }
425 451 return base
@@ -530,6 +556,27 @@ func parseJob(name string, node *yaml.Node) (draft, []Issue) {
530 556 list, iss := extendsOf(val)
531 557 issues = append(issues, iss...)
532 558 d.extends = list
559 case "retry":
560 retry, iss := parseRetry(val)
561 issues = append(issues, iss...)
562 d.job.Retry = retry
563 case "parallel":
564 n, matrix, iss := parseParallel(val)
565 issues = append(issues, iss...)
566 d.job.Parallel = n
567 d.job.Matrix = matrix
568 case "services":
569 svcs, iss := parseServices(val)
570 issues = append(issues, iss...)
571 d.job.Services = svcs
572 case "environment":
573 env, iss := parseEnvironment(val)
574 issues = append(issues, iss...)
575 d.job.Environment = env
576 case "interruptible":
577 b, iss := parseBool(val)
578 issues = append(issues, iss...)
579 d.job.Interruptible = b
533 580 default:
534 581 if _, ok := reserved[key]; ok {
535 582 issues = append(issues, Issue{Line: node.Content[i].Line, Key: "ci.warn.ignored", Args: []any{key}, Warning: true})
@@ -616,6 +663,14 @@ func inherit(child, parent *draft) {
616 663 copyField("resource_group", func() { child.job.ResourceGroup = parent.job.ResourceGroup })
617 664 copyField("only", func() { child.job.Only = parent.job.Only })
618 665 copyField("except", func() { child.job.Except = parent.job.Except })
666 copyField("retry", func() { child.job.Retry = parent.job.Retry })
667 copyField("parallel", func() {
668 child.job.Parallel = parent.job.Parallel
669 child.job.Matrix = parent.job.Matrix
670 })
671 copyField("services", func() { child.job.Services = parent.job.Services })
672 copyField("environment", func() { child.job.Environment = parent.job.Environment })
673 copyField("interruptible", func() { child.job.Interruptible = parent.job.Interruptible })
619 674 if len(parent.job.Variables) > 0 {
620 675 merged := map[string]string{}
621 676 for k, v := range parent.job.Variables {
@@ -835,7 +890,7 @@ func parseRules(node *yaml.Node) ([]Rule, []Issue) {
835 890 continue
836 891 }
837 892 r.If = strings.TrimSpace(val.Value)
838 if _, err := compileRule(r.If); err != nil {
893 if err := checkRule(r.If); err != nil {
839 894 if ie, ok := err.(i18n.Error); ok {
840 895 issues = append(issues, Issue{Line: val.Line, Key: ie.Key, Args: ie.Args})
841 896 } else {
@@ -153,7 +153,11 @@ func parseBytes(data []byte, read func(string) ([]byte, error), stack []string)
153 d, iss := parseDefault(val)
153 d, iss := parseDefault(val)
154 issues = append(issues, iss...)
154 issues = append(issues, iss...)
155 def = overlayDefault(def, d)
155 def = overlayDefault(def, d)
156 case "workflow", "services", "spec", "pages":
156 case "workflow":
157 rules, iss := parseWorkflow(val)
158 issues = append(issues, iss...)
159 p.Workflow = rules
160 case "services", "spec", "pages":
157 issues = append(issues, Issue{Line: keyNode.Line, Key: "ci.warn.ignored", Args: []any{key}, Warning: true})
161 issues = append(issues, Issue{Line: keyNode.Line, Key: "ci.warn.ignored", Args: []any{key}, Warning: true})
158 case "jobs":
162 case "jobs":
159 seenLegacy = true
163 seenLegacy = true
@@ -347,6 +351,16 @@ func parseDefault(node *yaml.Node) (draft, []Issue) {
347 issues = append(issues, iss...)
351 issues = append(issues, iss...)
348 d.job.Timeout = dur
352 d.job.Timeout = dur
349 d.present["timeout"] = true
353 d.present["timeout"] = true
354 case "retry":
355 retry, iss := parseRetry(val)
356 issues = append(issues, iss...)
357 d.job.Retry = retry
358 d.present["retry"] = true
359 case "interruptible":
360 b, iss := parseBool(val)
361 issues = append(issues, iss...)
362 d.job.Interruptible = b
363 d.present["interruptible"] = true
350 case "cache":
364 case "cache":
351 c, iss := parseCache(val)
365 c, iss := parseCache(val)
352 issues = append(issues, iss...)
366 issues = append(issues, iss...)
@@ -385,6 +399,14 @@ func overlayJob(base Job, d draft) Job {
385 set("resource_group", func() { base.ResourceGroup = d.job.ResourceGroup })
399 set("resource_group", func() { base.ResourceGroup = d.job.ResourceGroup })
386 set("only", func() { base.Only = d.job.Only })
400 set("only", func() { base.Only = d.job.Only })
387 set("except", func() { base.Except = d.job.Except })
401 set("except", func() { base.Except = d.job.Except })
402 set("retry", func() { base.Retry = d.job.Retry })
403 set("parallel", func() {
404 base.Parallel = d.job.Parallel
405 base.Matrix = d.job.Matrix
406 })
407 set("services", func() { base.Services = d.job.Services })
408 set("environment", func() { base.Environment = d.job.Environment })
409 set("interruptible", func() { base.Interruptible = d.job.Interruptible })
388 if d.present["variables"] {
410 if d.present["variables"] {
389 mergedVars := map[string]string{}
411 mergedVars := map[string]string{}
390 for k, v := range base.Variables {
412 for k, v := range base.Variables {
@@ -420,6 +442,10 @@ func overlayDefault(base, extra draft) draft {
420 base.job.Timeout = extra.job.Timeout
442 base.job.Timeout = extra.job.Timeout
421 case "cache":
443 case "cache":
422 base.job.Cache = extra.job.Cache
444 base.job.Cache = extra.job.Cache
445 case "retry":
446 base.job.Retry = extra.job.Retry
447 case "interruptible":
448 base.job.Interruptible = extra.job.Interruptible
423 }
449 }
424 }
450 }
425 return base
451 return base
@@ -530,6 +556,27 @@ func parseJob(name string, node *yaml.Node) (draft, []Issue) {
530 list, iss := extendsOf(val)
556 list, iss := extendsOf(val)
531 issues = append(issues, iss...)
557 issues = append(issues, iss...)
532 d.extends = list
558 d.extends = list
559 case "retry":
560 retry, iss := parseRetry(val)
561 issues = append(issues, iss...)
562 d.job.Retry = retry
563 case "parallel":
564 n, matrix, iss := parseParallel(val)
565 issues = append(issues, iss...)
566 d.job.Parallel = n
567 d.job.Matrix = matrix
568 case "services":
569 svcs, iss := parseServices(val)
570 issues = append(issues, iss...)
571 d.job.Services = svcs
572 case "environment":
573 env, iss := parseEnvironment(val)
574 issues = append(issues, iss...)
575 d.job.Environment = env
576 case "interruptible":
577 b, iss := parseBool(val)
578 issues = append(issues, iss...)
579 d.job.Interruptible = b
533 default:
580 default:
534 if _, ok := reserved[key]; ok {
581 if _, ok := reserved[key]; ok {
535 issues = append(issues, Issue{Line: node.Content[i].Line, Key: "ci.warn.ignored", Args: []any{key}, Warning: true})
582 issues = append(issues, Issue{Line: node.Content[i].Line, Key: "ci.warn.ignored", Args: []any{key}, Warning: true})
@@ -616,6 +663,14 @@ func inherit(child, parent *draft) {
616 copyField("resource_group", func() { child.job.ResourceGroup = parent.job.ResourceGroup })
663 copyField("resource_group", func() { child.job.ResourceGroup = parent.job.ResourceGroup })
617 copyField("only", func() { child.job.Only = parent.job.Only })
664 copyField("only", func() { child.job.Only = parent.job.Only })
618 copyField("except", func() { child.job.Except = parent.job.Except })
665 copyField("except", func() { child.job.Except = parent.job.Except })
666 copyField("retry", func() { child.job.Retry = parent.job.Retry })
667 copyField("parallel", func() {
668 child.job.Parallel = parent.job.Parallel
669 child.job.Matrix = parent.job.Matrix
670 })
671 copyField("services", func() { child.job.Services = parent.job.Services })
672 copyField("environment", func() { child.job.Environment = parent.job.Environment })
673 copyField("interruptible", func() { child.job.Interruptible = parent.job.Interruptible })
619 if len(parent.job.Variables) > 0 {
674 if len(parent.job.Variables) > 0 {
620 merged := map[string]string{}
675 merged := map[string]string{}
621 for k, v := range parent.job.Variables {
676 for k, v := range parent.job.Variables {
@@ -835,7 +890,7 @@ func parseRules(node *yaml.Node) ([]Rule, []Issue) {
835 continue
890 continue
836 }
891 }
837 r.If = strings.TrimSpace(val.Value)
892 r.If = strings.TrimSpace(val.Value)
838 if _, err := compileRule(r.If); err != nil {
893 if err := checkRule(r.If); err != nil {
839 if ie, ok := err.(i18n.Error); ok {
894 if ie, ok := err.(i18n.Error); ok {
840 issues = append(issues, Issue{Line: val.Line, Key: ie.Key, Args: ie.Args})
895 issues = append(issues, Issue{Line: val.Line, Key: ie.Key, Args: ie.Args})
841 } else {
896 } else {
internal/i18n/catalog.go +8 −0
@@ -207,12 +207,14 @@ var en = map[string]string{
207 207 "pipe.pending": "Pending",
208 208 "pipe.manual": "Manual",
209 209 "pipe.skipped": "Skipped",
210 "pipe.canceled": "Canceled",
210 211 "pipe.warning": "Warning",
211 212 "pipe.stage": "Stage",
212 213 "pipe.duration": "Duration",
213 214 "pipe.finished": "Finished",
214 215 "pipe.queued": "Queued",
215 216 "pipe.timeout": "Timeout",
217 "pipe.environment": "Environment",
216 218 "pipe.runner": "Runner",
217 219 "pipe.runnerHere": "bluegit #1",
218 220 "pipe.source": "Source",
@@ -562,6 +564,12 @@ var en = map[string]string{
562 564 "ci.err.ruleExpr": "rule %q is not understood",
563 565 "ci.err.ruleCombine": "rule %q cannot combine conditions with && or ||",
564 566 "ci.err.regex": "invalid regex in %q: %s",
567 "ci.err.parallel": "parallel creates %d jobs; the maximum is %d",
568 "ci.err.parallelForm": "parallel must be a number from 1 to %d, or a matrix",
569 "ci.err.retry": "retry must be a number from 0 to 2, or max and when",
570 "ci.err.service": "a service needs an image name",
571 "ci.err.environment": "environment needs a name",
572 "ci.warn.retryMax": "retry max %d is limited to %d",
565 573 "ci.err.when": "when %q is not supported",
566 574 "ci.err.timeout": "timeout %q is invalid",
567 575 "ci.err.extendsList": "extends must be a job name or a list",
@@ -207,12 +207,14 @@ var en = map[string]string{
207 "pipe.pending": "Pending",
207 "pipe.pending": "Pending",
208 "pipe.manual": "Manual",
208 "pipe.manual": "Manual",
209 "pipe.skipped": "Skipped",
209 "pipe.skipped": "Skipped",
210 "pipe.canceled": "Canceled",
210 "pipe.warning": "Warning",
211 "pipe.warning": "Warning",
211 "pipe.stage": "Stage",
212 "pipe.stage": "Stage",
212 "pipe.duration": "Duration",
213 "pipe.duration": "Duration",
213 "pipe.finished": "Finished",
214 "pipe.finished": "Finished",
214 "pipe.queued": "Queued",
215 "pipe.queued": "Queued",
215 "pipe.timeout": "Timeout",
216 "pipe.timeout": "Timeout",
217 "pipe.environment": "Environment",
216 "pipe.runner": "Runner",
218 "pipe.runner": "Runner",
217 "pipe.runnerHere": "bluegit #1",
219 "pipe.runnerHere": "bluegit #1",
218 "pipe.source": "Source",
220 "pipe.source": "Source",
@@ -562,6 +564,12 @@ var en = map[string]string{
562 "ci.err.ruleExpr": "rule %q is not understood",
564 "ci.err.ruleExpr": "rule %q is not understood",
563 "ci.err.ruleCombine": "rule %q cannot combine conditions with && or ||",
565 "ci.err.ruleCombine": "rule %q cannot combine conditions with && or ||",
564 "ci.err.regex": "invalid regex in %q: %s",
566 "ci.err.regex": "invalid regex in %q: %s",
567 "ci.err.parallel": "parallel creates %d jobs; the maximum is %d",
568 "ci.err.parallelForm": "parallel must be a number from 1 to %d, or a matrix",
569 "ci.err.retry": "retry must be a number from 0 to 2, or max and when",
570 "ci.err.service": "a service needs an image name",
571 "ci.err.environment": "environment needs a name",
572 "ci.warn.retryMax": "retry max %d is limited to %d",
565 "ci.err.when": "when %q is not supported",
573 "ci.err.when": "when %q is not supported",
566 "ci.err.timeout": "timeout %q is invalid",
574 "ci.err.timeout": "timeout %q is invalid",
567 "ci.err.extendsList": "extends must be a job name or a list",
575 "ci.err.extendsList": "extends must be a job name or a list",
internal/i18n/catalog_lang.go +24 −0
@@ -214,12 +214,14 @@ var nl = map[string]string{
214 214 "pipe.pending": "Wachtend",
215 215 "pipe.manual": "Handmatig",
216 216 "pipe.skipped": "Overgeslagen",
217 "pipe.canceled": "Geannuleerd",
217 218 "pipe.warning": "Waarschuwing",
218 219 "pipe.stage": "Stage",
219 220 "pipe.duration": "Duur",
220 221 "pipe.finished": "Klaar",
221 222 "pipe.queued": "Wachtrij",
222 223 "pipe.timeout": "Timeout",
224 "pipe.environment": "Omgeving",
223 225 "pipe.runner": "Runner",
224 226 "pipe.runnerHere": "bluegit #1",
225 227 "pipe.source": "Bron",
@@ -557,6 +559,12 @@ var nl = map[string]string{
557 559 "ci.err.ruleExpr": "rule %q wordt niet begrepen",
558 560 "ci.err.ruleCombine": "rule %q kan geen voorwaarden combineren met && of ||",
559 561 "ci.err.regex": "ongeldige regex in %q: %s",
562 "ci.err.parallel": "parallel maakt %d jobs; het maximum is %d",
563 "ci.err.parallelForm": "parallel is een getal van 1 tot %d, of een matrix",
564 "ci.err.retry": "retry is een getal van 0 tot 2, of max en when",
565 "ci.err.service": "een service heeft een imagenaam nodig",
566 "ci.err.environment": "environment heeft een naam nodig",
567 "ci.warn.retryMax": "retry-maximum %d is beperkt tot %d",
560 568 "ci.err.when": "when %q wordt niet ondersteund",
561 569 "ci.err.timeout": "timeout %q is ongeldig",
562 570 "ci.err.extendsList": "extends moet een jobnaam of een lijst zijn",
@@ -780,12 +788,14 @@ var de = map[string]string{
780 788 "pipe.pending": "Wartend",
781 789 "pipe.manual": "Manuell",
782 790 "pipe.skipped": "Übersprungen",
791 "pipe.canceled": "Abgebrochen",
783 792 "pipe.warning": "Warnung",
784 793 "pipe.stage": "Stage",
785 794 "pipe.duration": "Dauer",
786 795 "pipe.finished": "Beendet",
787 796 "pipe.queued": "Warteschlange",
788 797 "pipe.timeout": "Timeout",
798 "pipe.environment": "Umgebung",
789 799 "pipe.runner": "Runner",
790 800 "pipe.runnerHere": "bluegit #1",
791 801 "pipe.source": "Quelle",
@@ -1123,6 +1133,12 @@ var de = map[string]string{
1123 1133 "ci.err.ruleExpr": "Rule %q wird nicht verstanden",
1124 1134 "ci.err.ruleCombine": "Rule %q kann Bedingungen nicht mit && oder || verbinden",
1125 1135 "ci.err.regex": "ungültiger regulärer Ausdruck in %q: %s",
1136 "ci.err.parallel": "parallel erzeugt %d Jobs; das Maximum ist %d",
1137 "ci.err.parallelForm": "parallel ist eine Zahl von 1 bis %d, oder eine Matrix",
1138 "ci.err.retry": "retry ist eine Zahl von 0 bis 2, oder max und when",
1139 "ci.err.service": "ein Service braucht einen Image-Namen",
1140 "ci.err.environment": "environment braucht einen Namen",
1141 "ci.warn.retryMax": "retry-Maximum %d ist auf %d begrenzt",
1126 1142 "ci.err.when": "when %q wird nicht unterstützt",
1127 1143 "ci.err.timeout": "timeout %q ist ungültig",
1128 1144 "ci.err.extendsList": "extends muss ein Jobname oder eine Liste sein",
@@ -1346,12 +1362,14 @@ var fr = map[string]string{
1346 1362 "pipe.pending": "En attente",
1347 1363 "pipe.manual": "Manuel",
1348 1364 "pipe.skipped": "Ignoré",
1365 "pipe.canceled": "Annulé",
1349 1366 "pipe.warning": "Avertissement",
1350 1367 "pipe.stage": "Stage",
1351 1368 "pipe.duration": "Durée",
1352 1369 "pipe.finished": "Terminé",
1353 1370 "pipe.queued": "File",
1354 1371 "pipe.timeout": "Délai",
1372 "pipe.environment": "Environnement",
1355 1373 "pipe.runner": "Runner",
1356 1374 "pipe.runnerHere": "bluegit #1",
1357 1375 "pipe.source": "Source",
@@ -1689,6 +1707,12 @@ var fr = map[string]string{
1689 1707 "ci.err.ruleExpr": "la règle %q n'est pas comprise",
1690 1708 "ci.err.ruleCombine": "la règle %q ne peut pas combiner des conditions avec && ou ||",
1691 1709 "ci.err.regex": "expression régulière invalide dans %q : %s",
1710 "ci.err.parallel": "parallel crée %d jobs ; le maximum est %d",
1711 "ci.err.parallelForm": "parallel est un nombre de 1 à %d, ou une matrice",
1712 "ci.err.retry": "retry est un nombre de 0 à 2, ou max et when",
1713 "ci.err.service": "un service a besoin d'un nom d'image",
1714 "ci.err.environment": "environment a besoin d'un nom",
1715 "ci.warn.retryMax": "le maximum de retry %d est limité à %d",
1692 1716 "ci.err.when": "when %q n'est pas pris en charge",
1693 1717 "ci.err.timeout": "timeout %q est invalide",
1694 1718 "ci.err.extendsList": "extends doit être un nom de job ou une liste",
@@ -214,12 +214,14 @@ var nl = map[string]string{
214 "pipe.pending": "Wachtend",
214 "pipe.pending": "Wachtend",
215 "pipe.manual": "Handmatig",
215 "pipe.manual": "Handmatig",
216 "pipe.skipped": "Overgeslagen",
216 "pipe.skipped": "Overgeslagen",
217 "pipe.canceled": "Geannuleerd",
217 "pipe.warning": "Waarschuwing",
218 "pipe.warning": "Waarschuwing",
218 "pipe.stage": "Stage",
219 "pipe.stage": "Stage",
219 "pipe.duration": "Duur",
220 "pipe.duration": "Duur",
220 "pipe.finished": "Klaar",
221 "pipe.finished": "Klaar",
221 "pipe.queued": "Wachtrij",
222 "pipe.queued": "Wachtrij",
222 "pipe.timeout": "Timeout",
223 "pipe.timeout": "Timeout",
224 "pipe.environment": "Omgeving",
223 "pipe.runner": "Runner",
225 "pipe.runner": "Runner",
224 "pipe.runnerHere": "bluegit #1",
226 "pipe.runnerHere": "bluegit #1",
225 "pipe.source": "Bron",
227 "pipe.source": "Bron",
@@ -557,6 +559,12 @@ var nl = map[string]string{
557 "ci.err.ruleExpr": "rule %q wordt niet begrepen",
559 "ci.err.ruleExpr": "rule %q wordt niet begrepen",
558 "ci.err.ruleCombine": "rule %q kan geen voorwaarden combineren met && of ||",
560 "ci.err.ruleCombine": "rule %q kan geen voorwaarden combineren met && of ||",
559 "ci.err.regex": "ongeldige regex in %q: %s",
561 "ci.err.regex": "ongeldige regex in %q: %s",
562 "ci.err.parallel": "parallel maakt %d jobs; het maximum is %d",
563 "ci.err.parallelForm": "parallel is een getal van 1 tot %d, of een matrix",
564 "ci.err.retry": "retry is een getal van 0 tot 2, of max en when",
565 "ci.err.service": "een service heeft een imagenaam nodig",
566 "ci.err.environment": "environment heeft een naam nodig",
567 "ci.warn.retryMax": "retry-maximum %d is beperkt tot %d",
560 "ci.err.when": "when %q wordt niet ondersteund",
568 "ci.err.when": "when %q wordt niet ondersteund",
561 "ci.err.timeout": "timeout %q is ongeldig",
569 "ci.err.timeout": "timeout %q is ongeldig",
562 "ci.err.extendsList": "extends moet een jobnaam of een lijst zijn",
570 "ci.err.extendsList": "extends moet een jobnaam of een lijst zijn",
@@ -780,12 +788,14 @@ var de = map[string]string{
780 "pipe.pending": "Wartend",
788 "pipe.pending": "Wartend",
781 "pipe.manual": "Manuell",
789 "pipe.manual": "Manuell",
782 "pipe.skipped": "Übersprungen",
790 "pipe.skipped": "Übersprungen",
791 "pipe.canceled": "Abgebrochen",
783 "pipe.warning": "Warnung",
792 "pipe.warning": "Warnung",
784 "pipe.stage": "Stage",
793 "pipe.stage": "Stage",
785 "pipe.duration": "Dauer",
794 "pipe.duration": "Dauer",
786 "pipe.finished": "Beendet",
795 "pipe.finished": "Beendet",
787 "pipe.queued": "Warteschlange",
796 "pipe.queued": "Warteschlange",
788 "pipe.timeout": "Timeout",
797 "pipe.timeout": "Timeout",
798 "pipe.environment": "Umgebung",
789 "pipe.runner": "Runner",
799 "pipe.runner": "Runner",
790 "pipe.runnerHere": "bluegit #1",
800 "pipe.runnerHere": "bluegit #1",
791 "pipe.source": "Quelle",
801 "pipe.source": "Quelle",
@@ -1123,6 +1133,12 @@ var de = map[string]string{
1123 "ci.err.ruleExpr": "Rule %q wird nicht verstanden",
1133 "ci.err.ruleExpr": "Rule %q wird nicht verstanden",
1124 "ci.err.ruleCombine": "Rule %q kann Bedingungen nicht mit && oder || verbinden",
1134 "ci.err.ruleCombine": "Rule %q kann Bedingungen nicht mit && oder || verbinden",
1125 "ci.err.regex": "ungültiger regulärer Ausdruck in %q: %s",
1135 "ci.err.regex": "ungültiger regulärer Ausdruck in %q: %s",
1136 "ci.err.parallel": "parallel erzeugt %d Jobs; das Maximum ist %d",
1137 "ci.err.parallelForm": "parallel ist eine Zahl von 1 bis %d, oder eine Matrix",
1138 "ci.err.retry": "retry ist eine Zahl von 0 bis 2, oder max und when",
1139 "ci.err.service": "ein Service braucht einen Image-Namen",
1140 "ci.err.environment": "environment braucht einen Namen",
1141 "ci.warn.retryMax": "retry-Maximum %d ist auf %d begrenzt",
1126 "ci.err.when": "when %q wird nicht unterstützt",
1142 "ci.err.when": "when %q wird nicht unterstützt",
1127 "ci.err.timeout": "timeout %q ist ungültig",
1143 "ci.err.timeout": "timeout %q ist ungültig",
1128 "ci.err.extendsList": "extends muss ein Jobname oder eine Liste sein",
1144 "ci.err.extendsList": "extends muss ein Jobname oder eine Liste sein",
@@ -1346,12 +1362,14 @@ var fr = map[string]string{
1346 "pipe.pending": "En attente",
1362 "pipe.pending": "En attente",
1347 "pipe.manual": "Manuel",
1363 "pipe.manual": "Manuel",
1348 "pipe.skipped": "Ignoré",
1364 "pipe.skipped": "Ignoré",
1365 "pipe.canceled": "Annulé",
1349 "pipe.warning": "Avertissement",
1366 "pipe.warning": "Avertissement",
1350 "pipe.stage": "Stage",
1367 "pipe.stage": "Stage",
1351 "pipe.duration": "Durée",
1368 "pipe.duration": "Durée",
1352 "pipe.finished": "Terminé",
1369 "pipe.finished": "Terminé",
1353 "pipe.queued": "File",
1370 "pipe.queued": "File",
1354 "pipe.timeout": "Délai",
1371 "pipe.timeout": "Délai",
1372 "pipe.environment": "Environnement",
1355 "pipe.runner": "Runner",
1373 "pipe.runner": "Runner",
1356 "pipe.runnerHere": "bluegit #1",
1374 "pipe.runnerHere": "bluegit #1",
1357 "pipe.source": "Source",
1375 "pipe.source": "Source",
@@ -1689,6 +1707,12 @@ var fr = map[string]string{
1689 "ci.err.ruleExpr": "la règle %q n'est pas comprise",
1707 "ci.err.ruleExpr": "la règle %q n'est pas comprise",
1690 "ci.err.ruleCombine": "la règle %q ne peut pas combiner des conditions avec && ou ||",
1708 "ci.err.ruleCombine": "la règle %q ne peut pas combiner des conditions avec && ou ||",
1691 "ci.err.regex": "expression régulière invalide dans %q : %s",
1709 "ci.err.regex": "expression régulière invalide dans %q : %s",
1710 "ci.err.parallel": "parallel crée %d jobs ; le maximum est %d",
1711 "ci.err.parallelForm": "parallel est un nombre de 1 à %d, ou une matrice",
1712 "ci.err.retry": "retry est un nombre de 0 à 2, ou max et when",
1713 "ci.err.service": "un service a besoin d'un nom d'image",
1714 "ci.err.environment": "environment a besoin d'un nom",
1715 "ci.warn.retryMax": "le maximum de retry %d est limité à %d",
1692 "ci.err.when": "when %q n'est pas pris en charge",
1716 "ci.err.when": "when %q n'est pas pris en charge",
1693 "ci.err.timeout": "timeout %q est invalide",
1717 "ci.err.timeout": "timeout %q est invalide",
1694 "ci.err.extendsList": "extends doit être un nom de job ou une liste",
1718 "ci.err.extendsList": "extends doit être un nom de job ou une liste",
internal/runner/retry_test.go added +29 −0
@@ -0,0 +1,29 @@
1 package runner
2
3 import (
4 "testing"
5
6 "gitlab.com/Timfa/bluegit/internal/store"
7 )
8
9 func TestShouldRetry(t *testing.T) {
10 job := store.Job{RetryMax: 2}
11 if !shouldRetry(job, "script_failure") {
12 t.Fatal("lege when herhaalt")
13 }
14 job.Attempt = 2
15 if shouldRetry(job, "script_failure") {
16 t.Fatal("max bereikt")
17 }
18 job.Attempt = 0
19 job.RetryWhen = []string{"runner_system_failure", "stuck_or_timeout_failure"}
20 if shouldRetry(job, "script_failure") {
21 t.Fatal("script hoort niet")
22 }
23 if !shouldRetry(job, "runner_system_failure") || !shouldRetry(job, "job_execution_timeout") {
24 t.Fatal("systeem en timeout")
25 }
26 if safeSegment("build 1/3") != "build_1_3" {
27 t.Fatal(safeSegment("build 1/3"))
28 }
29 }
@@ -0,0 +1,29 @@
1package runner
2
3import (
4 "testing"
5
6 "gitlab.com/Timfa/bluegit/internal/store"
7)
8
9func TestShouldRetry(t *testing.T) {
10 job := store.Job{RetryMax: 2}
11 if !shouldRetry(job, "script_failure") {
12 t.Fatal("lege when herhaalt")
13 }
14 job.Attempt = 2
15 if shouldRetry(job, "script_failure") {
16 t.Fatal("max bereikt")
17 }
18 job.Attempt = 0
19 job.RetryWhen = []string{"runner_system_failure", "stuck_or_timeout_failure"}
20 if shouldRetry(job, "script_failure") {
21 t.Fatal("script hoort niet")
22 }
23 if !shouldRetry(job, "runner_system_failure") || !shouldRetry(job, "job_execution_timeout") {
24 t.Fatal("systeem en timeout")
25 }
26 if safeSegment("build 1/3") != "build_1_3" {
27 t.Fatal(safeSegment("build 1/3"))
28 }
29}
internal/runner/runner.go +220 −26
@@ -4,6 +4,7 @@ package runner
4 4 import (
5 5 "bytes"
6 6 "context"
7 "errors"
7 8 "fmt"
8 9 "io"
9 10 "log/slog"
@@ -23,17 +24,45 @@ import (
23 24
24 25 // Runner plant jobs en stuurt het resultaat door naar webhooks.
25 26 type Runner struct {
26 Store *store.Store
27 GitBin string
28 Docker string
29 Storage string
30 BaseURL string
31 Network string
32 Timeout time.Duration
33 Log *slog.Logger
34 OnEvent func(repoID int64, ev webhook.Event)
35 slots chan struct{}
36 mu sync.RWMutex
27 Store *store.Store
28 GitBin string
29 Docker string
30 Storage string
31 BaseURL string
32 Network string
33 Timeout time.Duration
34 Log *slog.Logger
35 OnEvent func(repoID int64, ev webhook.Event)
36 slots chan struct{}
37 mu sync.RWMutex
38 cancelMu sync.Mutex
39 cancel map[int64]context.CancelFunc
40 }
41
42 // Interrupt stopt lopende jobs waarvan een nieuwere pipeline de beurt overneemt.
43 func (r *Runner) Interrupt(ids []int64) {
44 r.cancelMu.Lock()
45 defer r.cancelMu.Unlock()
46 for _, id := range ids {
47 if c := r.cancel[id]; c != nil {
48 c()
49 }
50 }
51 }
52
53 func (r *Runner) track(id int64, cancel context.CancelFunc) {
54 r.cancelMu.Lock()
55 defer r.cancelMu.Unlock()
56 if r.cancel == nil {
57 r.cancel = map[int64]context.CancelFunc{}
58 }
59 r.cancel[id] = cancel
60 }
61
62 func (r *Runner) untrack(id int64) {
63 r.cancelMu.Lock()
64 defer r.cancelMu.Unlock()
65 delete(r.cancel, id)
37 66 }
38 67
39 68 // SetStorage wijzigt het pad waarop jobs hun clone en artifacts zetten.
@@ -186,13 +215,52 @@ func (r *Runner) execute(ctx context.Context, p store.Pipeline, job store.Job) {
186 215 defer func() { <-r.slots }()
187 216 }
188 217 before, _ := r.Store.PipelineByID(context.Background(), p.RepoID, p.ID)
189 mask, err := r.runOne(ctx, p, job)
218 base, stop := context.WithCancel(ctx)
219 r.track(job.ID, stop)
220 defer func() {
221 r.untrack(job.ID)
222 stop()
223 }()
224 runCtx := base
225 limit := r.Timeout
226 if job.Timeout > 0 {
227 limit = job.Timeout
228 }
229 if limit > 0 {
230 var cancel context.CancelFunc
231 runCtx, cancel = context.WithTimeout(base, limit)
232 defer cancel()
233 }
234 mask, err := r.runOne(runCtx, p, job)
190 235 status := "success"
191 236 extra := ""
192 if err != nil {
237 kind := ""
238 switch {
239 case err == nil:
240 case ctx.Err() == nil && base.Err() != nil:
241 status = "canceled"
242 extra = stamp("canceled because a newer pipeline started for the same ref")
243 case errors.Is(runCtx.Err(), context.DeadlineExceeded):
193 244 status = "failed"
245 kind = "job_execution_timeout"
246 extra = stamp(ci.Mask(err.Error(), mask))
247 default:
248 status = "failed"
249 kind = classifyFailure(err)
194 250 extra = stamp(ci.Mask(err.Error(), mask))
195 251 }
252 if status == "failed" && shouldRetry(job, kind) {
253 note := stamp(fmt.Sprintf("retrying (%d/%d) after %s", job.Attempt+1, job.RetryMax, kind))
254 if ferr := r.Store.RequeueJob(context.Background(), job.ID, job.Attempt+1, note+extra); ferr != nil && r.Log != nil {
255 r.Log.Error("job opnieuw", "err", ferr)
256 } else if ferr == nil {
257 r.settle(context.Background(), p.ID)
258 if _, rerr := r.Store.RefreshPipelineStatus(context.Background(), p.ID); rerr != nil && r.Log != nil {
259 r.Log.Error("pipelinestatus", "err", rerr)
260 }
261 return
262 }
263 }
196 264 if ferr := r.Store.FinishJob(context.Background(), job.ID, status, extra); ferr != nil {
197 265 r.Log.Error("job afronden", "err", ferr)
198 266 }
@@ -230,22 +298,13 @@ func (r *Runner) execute(ctx context.Context, p store.Pipeline, job store.Job) {
230 298 }
231 299
232 300 func (r *Runner) runOne(ctx context.Context, p store.Pipeline, job store.Job) ([]string, error) {
233 limit := r.Timeout
234 if job.Timeout > 0 {
235 limit = job.Timeout
236 }
237 if limit > 0 {
238 var cancel context.CancelFunc
239 ctx, cancel = context.WithTimeout(ctx, limit)
240 defer cancel()
241 }
242 301 repo, err := r.Store.RepoByID(ctx, p.RepoID)
243 302 if err != nil {
244 303 return nil, err
245 304 }
246 305 sink := &logSink{store: r.Store, jobID: job.ID}
247 306 sink.section("prepare", "Preparing the docker executor")
248 work := filepath.Join(r.storage(), "work", fmt.Sprint(p.ID), job.Name)
307 work := filepath.Join(r.storage(), "work", fmt.Sprint(p.ID), safeSegment(job.Name))
249 308 _ = os.RemoveAll(work)
250 309 if err := os.MkdirAll(work, 0o755); err != nil {
251 310 sink.closeSection()
@@ -286,13 +345,23 @@ func (r *Runner) runOne(ctx context.Context, p store.Pipeline, job store.Job) ([
286 345 sink.secrets = mask
287 346 if _, err := exec.LookPath(r.Docker); err != nil {
288 347 sink.closeSection()
289 return mask, fmt.Errorf("docker (%s) is not available; the job only runs in a container", r.Docker)
348 return mask, fmt.Errorf("runner_system_failure: docker (%s) is not available; the job only runs in a container", r.Docker)
349 }
350 network := r.Network
351 if len(job.Services) > 0 {
352 net, cleanup, serr := r.startServices(ctx, job)
353 if serr != nil {
354 sink.closeSection()
355 return mask, serr
356 }
357 defer cleanup()
358 network = net
290 359 }
291 360 sink.closeSection()
292 361 // -e NAAM zonder waarde laat Docker de variabele uit de omgeving van
293 362 // deze client overnemen. Zo blijft een meerregelige sleutel heel en
294 363 // staat de waarde niet in de proceslijst.
295 args := []string{"run", "--rm", "--network", r.Network, "-v", work + ":/workspace", "-w", "/workspace"}
364 args := []string{"run", "--rm", "--network", network, "-v", work + ":/workspace", "-w", "/workspace"}
296 365 cmdEnv := append([]string{}, os.Environ()...)
297 366 for k, v := range env {
298 367 args = append(args, "-e", k)
@@ -451,11 +520,27 @@ func (r *Runner) environment(ctx context.Context, repo store.Repo, p store.Pipel
451 520 env[k] = v
452 521 }
453 522 base := strings.TrimRight(r.BaseURL, "/")
523 var info *ci.MRInfo
524 if p.MRID != 0 {
525 if m, merr := r.Store.MergeRequest(ctx, repo.ID, p.MRID); merr == nil {
526 target := ""
527 dir := gitrepo.Dir(r.storage(), repo.OwnerPath, repo.Name)
528 if out, gerr := exec.CommandContext(ctx, r.GitBin, "-C", dir, "rev-parse", "--verify", "refs/heads/"+m.TargetBranch).Output(); gerr == nil {
529 target = strings.TrimSpace(string(out))
530 }
531 info = &ci.MRInfo{
532 ID: m.ID, IID: m.Number, Title: m.Title,
533 SourceBranch: m.SourceBranch, TargetBranch: m.TargetBranch,
534 ProjectPath: repo.FullPath(), ProjectURL: base + "/" + repo.FullPath(),
535 TargetSHA: target,
536 }
537 }
538 }
454 539 pre := ci.Variables(ci.Ref{
455 540 SHA: p.CommitSHA, FullRef: p.Ref, Title: p.CommitTitle,
456 541 DefaultBranch: repo.DefaultBranch, Source: p.Source,
457 542 ProjectPath: repo.FullPath(), ProjectURL: base + "/" + repo.FullPath(),
458 APIURL: base + "/api/v1", PipelineID: p.ID,
543 APIURL: base + "/api/v1", PipelineID: p.ID, MR: info,
459 544 })
460 545 for k, v := range pre {
461 546 env[k] = v
@@ -465,10 +550,119 @@ func (r *Runner) environment(ctx context.Context, repo store.Repo, p store.Pipel
465 550 env["CI_JOB_STAGE"] = job.Stage
466 551 env["CI_JOB_URL"] = base + "/" + repo.FullPath() + "/-/pipelines/" + fmt.Sprint(p.ID) + "/jobs/" + fmt.Sprint(job.ID)
467 552 env["CI_JOB_TOKEN"] = token
553 if job.Environment != "" {
554 env["CI_ENVIRONMENT_NAME"] = job.Environment
555 if job.EnvironmentURL != "" {
556 env["CI_ENVIRONMENT_URL"] = ci.Expand(job.EnvironmentURL, env)
557 }
558 }
559 for k, v := range job.Variables {
560 if strings.HasPrefix(k, "CI_NODE_") {
561 env[k] = v
562 }
563 }
468 564 mask = append(mask, token)
469 565 return env, mask, nil
470 566 }
471 567
568 func (r *Runner) startServices(ctx context.Context, job store.Job) (string, func(), error) {
569 netName := fmt.Sprintf("bluegit-%d", job.ID)
570 if _, err := execOut(ctx, r.Docker, "network", "create", netName); err != nil {
571 return "", func() {}, fmt.Errorf("runner_system_failure: docker network: %w", err)
572 }
573 var names []string
574 cleanup := func() {
575 cctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
576 defer cancel()
577 for _, name := range names {
578 _, _ = execOut(cctx, r.Docker, "rm", "-f", name)
579 }
580 _, _ = execOut(cctx, r.Docker, "network", "rm", netName)
581 }
582 for i, svc := range job.Services {
583 cname := fmt.Sprintf("bluegit-svc-%d-%d", job.ID, i)
584 args := []string{"run", "-d", "--name", cname, "--network", netName}
585 if svc.Alias != "" {
586 args = append(args, "--network-alias", svc.Alias)
587 }
588 cmdEnv := append([]string{}, os.Environ()...)
589 for k, v := range svc.Variables {
590 args = append(args, "-e", k)
591 cmdEnv = append(cmdEnv, k+"="+v)
592 }
593 if len(svc.Entrypoint) > 0 {
594 args = append(args, "--entrypoint", svc.Entrypoint[0])
595 }
596 args = append(args, svc.Name)
597 if len(svc.Entrypoint) > 1 {
598 args = append(args, svc.Entrypoint[1:]...)
599 }
600 args = append(args, svc.Command...)
601 cmd := exec.CommandContext(ctx, r.Docker, args...)
602 cmd.Env = cmdEnv
603 var stderr bytes.Buffer
604 cmd.Stderr = &stderr
605 if err := cmd.Run(); err != nil {
606 cleanup()
607 msg := strings.TrimSpace(stderr.String())
608 if msg == "" {
609 msg = err.Error()
610 }
611 return "", func() {}, fmt.Errorf("runner_system_failure: service %s: %s", svc.Name, msg)
612 }
613 names = append(names, cname)
614 }
615 return netName, cleanup, nil
616 }
617
618 func shouldRetry(job store.Job, kind string) bool {
619 if job.RetryMax <= 0 || job.Attempt >= job.RetryMax || kind == "" {
620 return false
621 }
622 if len(job.RetryWhen) == 0 || containsStr(job.RetryWhen, "always") {
623 return true
624 }
625 if containsStr(job.RetryWhen, kind) {
626 return true
627 }
628 return kind == "job_execution_timeout" && containsStr(job.RetryWhen, "stuck_or_timeout_failure")
629 }
630
631 func classifyFailure(err error) string {
632 msg := err.Error()
633 if strings.Contains(msg, "runner_system_failure") || strings.Contains(msg, "is not available") {
634 return "runner_system_failure"
635 }
636 if errors.Is(err, context.DeadlineExceeded) || strings.Contains(msg, "deadline exceeded") {
637 return "job_execution_timeout"
638 }
639 return "script_failure"
640 }
641
642 func containsStr(list []string, s string) bool {
643 for _, item := range list {
644 if item == s {
645 return true
646 }
647 }
648 return false
649 }
650
651 func safeSegment(name string) string {
652 var b strings.Builder
653 for _, r := range name {
654 if (r >= 'a' && r <= 'z') || (r >= 'A' && r <= 'Z') || (r >= '0' && r <= '9') || r == '.' || r == '_' || r == '-' {
655 b.WriteRune(r)
656 } else {
657 b.WriteByte('_')
658 }
659 }
660 if b.Len() == 0 {
661 return "job"
662 }
663 return b.String()
664 }
665
472 666 func (r *Runner) restoreCache(repoID int64, job store.Job, work string) error {
473 667 if len(job.CachePaths) == 0 {
474 668 return nil
@@ -4,6 +4,7 @@ package runner
4import (
4import (
5 "bytes"
5 "bytes"
6 "context"
6 "context"
7 "errors"
7 "fmt"
8 "fmt"
8 "io"
9 "io"
9 "log/slog"
10 "log/slog"
@@ -23,17 +24,45 @@ import (
23
24
24// Runner plant jobs en stuurt het resultaat door naar webhooks.
25// Runner plant jobs en stuurt het resultaat door naar webhooks.
25type Runner struct {
26type Runner struct {
26 Store *store.Store
27 Store *store.Store
27 GitBin string
28 GitBin string
28 Docker string
29 Docker string
29 Storage string
30 Storage string
30 BaseURL string
31 BaseURL string
31 Network string
32 Network string
32 Timeout time.Duration
33 Timeout time.Duration
33 Log *slog.Logger
34 Log *slog.Logger
34 OnEvent func(repoID int64, ev webhook.Event)
35 OnEvent func(repoID int64, ev webhook.Event)
35 slots chan struct{}
36 slots chan struct{}
36 mu sync.RWMutex
37 mu sync.RWMutex
38 cancelMu sync.Mutex
39 cancel map[int64]context.CancelFunc
40}
41
42// Interrupt stopt lopende jobs waarvan een nieuwere pipeline de beurt overneemt.
43func (r *Runner) Interrupt(ids []int64) {
44 r.cancelMu.Lock()
45 defer r.cancelMu.Unlock()
46 for _, id := range ids {
47 if c := r.cancel[id]; c != nil {
48 c()
49 }
50 }
51}
52
53func (r *Runner) track(id int64, cancel context.CancelFunc) {
54 r.cancelMu.Lock()
55 defer r.cancelMu.Unlock()
56 if r.cancel == nil {
57 r.cancel = map[int64]context.CancelFunc{}
58 }
59 r.cancel[id] = cancel
60}
61
62func (r *Runner) untrack(id int64) {
63 r.cancelMu.Lock()
64 defer r.cancelMu.Unlock()
65 delete(r.cancel, id)
37}
66}
38
67
39// SetStorage wijzigt het pad waarop jobs hun clone en artifacts zetten.
68// SetStorage wijzigt het pad waarop jobs hun clone en artifacts zetten.
@@ -186,13 +215,52 @@ func (r *Runner) execute(ctx context.Context, p store.Pipeline, job store.Job) {
186 defer func() { <-r.slots }()
215 defer func() { <-r.slots }()
187 }
216 }
188 before, _ := r.Store.PipelineByID(context.Background(), p.RepoID, p.ID)
217 before, _ := r.Store.PipelineByID(context.Background(), p.RepoID, p.ID)
189 mask, err := r.runOne(ctx, p, job)
218 base, stop := context.WithCancel(ctx)
219 r.track(job.ID, stop)
220 defer func() {
221 r.untrack(job.ID)
222 stop()
223 }()
224 runCtx := base
225 limit := r.Timeout
226 if job.Timeout > 0 {
227 limit = job.Timeout
228 }
229 if limit > 0 {
230 var cancel context.CancelFunc
231 runCtx, cancel = context.WithTimeout(base, limit)
232 defer cancel()
233 }
234 mask, err := r.runOne(runCtx, p, job)
190 status := "success"
235 status := "success"
191 extra := ""
236 extra := ""
192 if err != nil {
237 kind := ""
238 switch {
239 case err == nil:
240 case ctx.Err() == nil && base.Err() != nil:
241 status = "canceled"
242 extra = stamp("canceled because a newer pipeline started for the same ref")
243 case errors.Is(runCtx.Err(), context.DeadlineExceeded):
193 status = "failed"
244 status = "failed"
245 kind = "job_execution_timeout"
246 extra = stamp(ci.Mask(err.Error(), mask))
247 default:
248 status = "failed"
249 kind = classifyFailure(err)
194 extra = stamp(ci.Mask(err.Error(), mask))
250 extra = stamp(ci.Mask(err.Error(), mask))
195 }
251 }
252 if status == "failed" && shouldRetry(job, kind) {
253 note := stamp(fmt.Sprintf("retrying (%d/%d) after %s", job.Attempt+1, job.RetryMax, kind))
254 if ferr := r.Store.RequeueJob(context.Background(), job.ID, job.Attempt+1, note+extra); ferr != nil && r.Log != nil {
255 r.Log.Error("job opnieuw", "err", ferr)
256 } else if ferr == nil {
257 r.settle(context.Background(), p.ID)
258 if _, rerr := r.Store.RefreshPipelineStatus(context.Background(), p.ID); rerr != nil && r.Log != nil {
259 r.Log.Error("pipelinestatus", "err", rerr)
260 }
261 return
262 }
263 }
196 if ferr := r.Store.FinishJob(context.Background(), job.ID, status, extra); ferr != nil {
264 if ferr := r.Store.FinishJob(context.Background(), job.ID, status, extra); ferr != nil {
197 r.Log.Error("job afronden", "err", ferr)
265 r.Log.Error("job afronden", "err", ferr)
198 }
266 }
@@ -230,22 +298,13 @@ func (r *Runner) execute(ctx context.Context, p store.Pipeline, job store.Job) {
230}
298}
231
299
232func (r *Runner) runOne(ctx context.Context, p store.Pipeline, job store.Job) ([]string, error) {
300func (r *Runner) runOne(ctx context.Context, p store.Pipeline, job store.Job) ([]string, error) {
233 limit := r.Timeout
234 if job.Timeout > 0 {
235 limit = job.Timeout
236 }
237 if limit > 0 {
238 var cancel context.CancelFunc
239 ctx, cancel = context.WithTimeout(ctx, limit)
240 defer cancel()
241 }
242 repo, err := r.Store.RepoByID(ctx, p.RepoID)
301 repo, err := r.Store.RepoByID(ctx, p.RepoID)
243 if err != nil {
302 if err != nil {
244 return nil, err
303 return nil, err
245 }
304 }
246 sink := &logSink{store: r.Store, jobID: job.ID}
305 sink := &logSink{store: r.Store, jobID: job.ID}
247 sink.section("prepare", "Preparing the docker executor")
306 sink.section("prepare", "Preparing the docker executor")
248 work := filepath.Join(r.storage(), "work", fmt.Sprint(p.ID), job.Name)
307 work := filepath.Join(r.storage(), "work", fmt.Sprint(p.ID), safeSegment(job.Name))
249 _ = os.RemoveAll(work)
308 _ = os.RemoveAll(work)
250 if err := os.MkdirAll(work, 0o755); err != nil {
309 if err := os.MkdirAll(work, 0o755); err != nil {
251 sink.closeSection()
310 sink.closeSection()
@@ -286,13 +345,23 @@ func (r *Runner) runOne(ctx context.Context, p store.Pipeline, job store.Job) ([
286 sink.secrets = mask
345 sink.secrets = mask
287 if _, err := exec.LookPath(r.Docker); err != nil {
346 if _, err := exec.LookPath(r.Docker); err != nil {
288 sink.closeSection()
347 sink.closeSection()
289 return mask, fmt.Errorf("docker (%s) is not available; the job only runs in a container", r.Docker)
348 return mask, fmt.Errorf("runner_system_failure: docker (%s) is not available; the job only runs in a container", r.Docker)
349 }
350 network := r.Network
351 if len(job.Services) > 0 {
352 net, cleanup, serr := r.startServices(ctx, job)
353 if serr != nil {
354 sink.closeSection()
355 return mask, serr
356 }
357 defer cleanup()
358 network = net
290 }
359 }
291 sink.closeSection()
360 sink.closeSection()
292 // -e NAAM zonder waarde laat Docker de variabele uit de omgeving van
361 // -e NAAM zonder waarde laat Docker de variabele uit de omgeving van
293 // deze client overnemen. Zo blijft een meerregelige sleutel heel en
362 // deze client overnemen. Zo blijft een meerregelige sleutel heel en
294 // staat de waarde niet in de proceslijst.
363 // staat de waarde niet in de proceslijst.
295 args := []string{"run", "--rm", "--network", r.Network, "-v", work + ":/workspace", "-w", "/workspace"}
364 args := []string{"run", "--rm", "--network", network, "-v", work + ":/workspace", "-w", "/workspace"}
296 cmdEnv := append([]string{}, os.Environ()...)
365 cmdEnv := append([]string{}, os.Environ()...)
297 for k, v := range env {
366 for k, v := range env {
298 args = append(args, "-e", k)
367 args = append(args, "-e", k)
@@ -451,11 +520,27 @@ func (r *Runner) environment(ctx context.Context, repo store.Repo, p store.Pipel
451 env[k] = v
520 env[k] = v
452 }
521 }
453 base := strings.TrimRight(r.BaseURL, "/")
522 base := strings.TrimRight(r.BaseURL, "/")
523 var info *ci.MRInfo
524 if p.MRID != 0 {
525 if m, merr := r.Store.MergeRequest(ctx, repo.ID, p.MRID); merr == nil {
526 target := ""
527 dir := gitrepo.Dir(r.storage(), repo.OwnerPath, repo.Name)
528 if out, gerr := exec.CommandContext(ctx, r.GitBin, "-C", dir, "rev-parse", "--verify", "refs/heads/"+m.TargetBranch).Output(); gerr == nil {
529 target = strings.TrimSpace(string(out))
530 }
531 info = &ci.MRInfo{
532 ID: m.ID, IID: m.Number, Title: m.Title,
533 SourceBranch: m.SourceBranch, TargetBranch: m.TargetBranch,
534 ProjectPath: repo.FullPath(), ProjectURL: base + "/" + repo.FullPath(),
535 TargetSHA: target,
536 }
537 }
538 }
454 pre := ci.Variables(ci.Ref{
539 pre := ci.Variables(ci.Ref{
455 SHA: p.CommitSHA, FullRef: p.Ref, Title: p.CommitTitle,
540 SHA: p.CommitSHA, FullRef: p.Ref, Title: p.CommitTitle,
456 DefaultBranch: repo.DefaultBranch, Source: p.Source,
541 DefaultBranch: repo.DefaultBranch, Source: p.Source,
457 ProjectPath: repo.FullPath(), ProjectURL: base + "/" + repo.FullPath(),
542 ProjectPath: repo.FullPath(), ProjectURL: base + "/" + repo.FullPath(),
458 APIURL: base + "/api/v1", PipelineID: p.ID,
543 APIURL: base + "/api/v1", PipelineID: p.ID, MR: info,
459 })
544 })
460 for k, v := range pre {
545 for k, v := range pre {
461 env[k] = v
546 env[k] = v
@@ -465,10 +550,119 @@ func (r *Runner) environment(ctx context.Context, repo store.Repo, p store.Pipel
465 env["CI_JOB_STAGE"] = job.Stage
550 env["CI_JOB_STAGE"] = job.Stage
466 env["CI_JOB_URL"] = base + "/" + repo.FullPath() + "/-/pipelines/" + fmt.Sprint(p.ID) + "/jobs/" + fmt.Sprint(job.ID)
551 env["CI_JOB_URL"] = base + "/" + repo.FullPath() + "/-/pipelines/" + fmt.Sprint(p.ID) + "/jobs/" + fmt.Sprint(job.ID)
467 env["CI_JOB_TOKEN"] = token
552 env["CI_JOB_TOKEN"] = token
553 if job.Environment != "" {
554 env["CI_ENVIRONMENT_NAME"] = job.Environment
555 if job.EnvironmentURL != "" {
556 env["CI_ENVIRONMENT_URL"] = ci.Expand(job.EnvironmentURL, env)
557 }
558 }
559 for k, v := range job.Variables {
560 if strings.HasPrefix(k, "CI_NODE_") {
561 env[k] = v
562 }
563 }
468 mask = append(mask, token)
564 mask = append(mask, token)
469 return env, mask, nil
565 return env, mask, nil
470}
566}
471
567
568func (r *Runner) startServices(ctx context.Context, job store.Job) (string, func(), error) {
569 netName := fmt.Sprintf("bluegit-%d", job.ID)
570 if _, err := execOut(ctx, r.Docker, "network", "create", netName); err != nil {
571 return "", func() {}, fmt.Errorf("runner_system_failure: docker network: %w", err)
572 }
573 var names []string
574 cleanup := func() {
575 cctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
576 defer cancel()
577 for _, name := range names {
578 _, _ = execOut(cctx, r.Docker, "rm", "-f", name)
579 }
580 _, _ = execOut(cctx, r.Docker, "network", "rm", netName)
581 }
582 for i, svc := range job.Services {
583 cname := fmt.Sprintf("bluegit-svc-%d-%d", job.ID, i)
584 args := []string{"run", "-d", "--name", cname, "--network", netName}
585 if svc.Alias != "" {
586 args = append(args, "--network-alias", svc.Alias)
587 }
588 cmdEnv := append([]string{}, os.Environ()...)
589 for k, v := range svc.Variables {
590 args = append(args, "-e", k)
591 cmdEnv = append(cmdEnv, k+"="+v)
592 }
593 if len(svc.Entrypoint) > 0 {
594 args = append(args, "--entrypoint", svc.Entrypoint[0])
595 }
596 args = append(args, svc.Name)
597 if len(svc.Entrypoint) > 1 {
598 args = append(args, svc.Entrypoint[1:]...)
599 }
600 args = append(args, svc.Command...)
601 cmd := exec.CommandContext(ctx, r.Docker, args...)
602 cmd.Env = cmdEnv
603 var stderr bytes.Buffer
604 cmd.Stderr = &stderr
605 if err := cmd.Run(); err != nil {
606 cleanup()
607 msg := strings.TrimSpace(stderr.String())
608 if msg == "" {
609 msg = err.Error()
610 }
611 return "", func() {}, fmt.Errorf("runner_system_failure: service %s: %s", svc.Name, msg)
612 }
613 names = append(names, cname)
614 }
615 return netName, cleanup, nil
616}
617
618func shouldRetry(job store.Job, kind string) bool {
619 if job.RetryMax <= 0 || job.Attempt >= job.RetryMax || kind == "" {
620 return false
621 }
622 if len(job.RetryWhen) == 0 || containsStr(job.RetryWhen, "always") {
623 return true
624 }
625 if containsStr(job.RetryWhen, kind) {
626 return true
627 }
628 return kind == "job_execution_timeout" && containsStr(job.RetryWhen, "stuck_or_timeout_failure")
629}
630
631func classifyFailure(err error) string {
632 msg := err.Error()
633 if strings.Contains(msg, "runner_system_failure") || strings.Contains(msg, "is not available") {
634 return "runner_system_failure"
635 }
636 if errors.Is(err, context.DeadlineExceeded) || strings.Contains(msg, "deadline exceeded") {
637 return "job_execution_timeout"
638 }
639 return "script_failure"
640}
641
642func containsStr(list []string, s string) bool {
643 for _, item := range list {
644 if item == s {
645 return true
646 }
647 }
648 return false
649}
650
651func safeSegment(name string) string {
652 var b strings.Builder
653 for _, r := range name {
654 if (r >= 'a' && r <= 'z') || (r >= 'A' && r <= 'Z') || (r >= '0' && r <= '9') || r == '.' || r == '_' || r == '-' {
655 b.WriteRune(r)
656 } else {
657 b.WriteByte('_')
658 }
659 }
660 if b.Len() == 0 {
661 return "job"
662 }
663 return b.String()
664}
665
472func (r *Runner) restoreCache(repoID int64, job store.Job, work string) error {
666func (r *Runner) restoreCache(repoID int64, job store.Job, work string) error {
473 if len(job.CachePaths) == 0 {
667 if len(job.CachePaths) == 0 {
474 return nil
668 return nil
internal/server/events.go +23 −0
@@ -42,6 +42,7 @@ func (s *Server) AfterPush(repo store.Repo, user store.User, before, after map[s
42 42 s.recordPushContributions(repo, user, map[string]string{ref: before[ref]}, map[string]string{ref: sha})
43 43 s.startPipeline(context.Background(), repo, sha, ref, actor, "push", before[ref], nil)
44 44 s.startMergeRequestPipelines(context.Background(), repo, sha, branch, actor, before[ref])
45 s.rerunMergeRequestsOnTarget(context.Background(), repo, branch, actor)
45 46 web := strings.TrimRight(s.Cfg.BaseURL, "/") + "/" + repo.FullPath()
46 47 s.Dispatch(repo.ID, webhook.Event{
47 48 Type: webhook.EventPush,
@@ -72,6 +73,9 @@ func (s *Server) startPipeline(ctx context.Context, repo store.Repo, sha, ref, a
72 73 if mr != nil && id != 0 {
73 74 _ = s.Store.AttachPipelineMR(ctx, id, mr.ID)
74 75 }
76 if id != 0 {
77 s.cancelOlder(ctx, repo.ID, ref, id)
78 }
75 79 return id
76 80 }
77 81 if len(jobs) == 0 {
@@ -87,9 +91,28 @@ func (s *Server) startPipeline(ctx context.Context, repo store.Repo, sha, ref, a
87 91 if mr != nil {
88 92 _ = s.Store.AttachPipelineMR(ctx, id, mr.ID)
89 93 }
94 s.cancelOlder(ctx, repo.ID, ref, id)
90 95 return id
91 96 }
92 97
98 func (s *Server) cancelOlder(ctx context.Context, repoID int64, ref string, keep int64) {
99 running, pipes, err := s.Store.CancelInterruptible(ctx, repoID, ref, keep)
100 if err != nil {
101 if s.Log != nil {
102 s.Log.Error("interruptible", "err", err)
103 }
104 return
105 }
106 if s.Runner != nil {
107 s.Runner.Interrupt(running)
108 }
109 for _, id := range pipes {
110 if _, err := s.Store.RefreshPipelineStatus(ctx, id); err != nil && s.Log != nil {
111 s.Log.Error("pipelinestatus", "err", err)
112 }
113 }
114 }
115
93 116 func gitOne(ctx context.Context, gitBin, dir string, args ...string) string {
94 117 out, err := exec.CommandContext(ctx, gitBin, append([]string{"-C", dir}, args...)...).Output()
95 118 if err != nil {
@@ -42,6 +42,7 @@ func (s *Server) AfterPush(repo store.Repo, user store.User, before, after map[s
42 s.recordPushContributions(repo, user, map[string]string{ref: before[ref]}, map[string]string{ref: sha})
42 s.recordPushContributions(repo, user, map[string]string{ref: before[ref]}, map[string]string{ref: sha})
43 s.startPipeline(context.Background(), repo, sha, ref, actor, "push", before[ref], nil)
43 s.startPipeline(context.Background(), repo, sha, ref, actor, "push", before[ref], nil)
44 s.startMergeRequestPipelines(context.Background(), repo, sha, branch, actor, before[ref])
44 s.startMergeRequestPipelines(context.Background(), repo, sha, branch, actor, before[ref])
45 s.rerunMergeRequestsOnTarget(context.Background(), repo, branch, actor)
45 web := strings.TrimRight(s.Cfg.BaseURL, "/") + "/" + repo.FullPath()
46 web := strings.TrimRight(s.Cfg.BaseURL, "/") + "/" + repo.FullPath()
46 s.Dispatch(repo.ID, webhook.Event{
47 s.Dispatch(repo.ID, webhook.Event{
47 Type: webhook.EventPush,
48 Type: webhook.EventPush,
@@ -72,6 +73,9 @@ func (s *Server) startPipeline(ctx context.Context, repo store.Repo, sha, ref, a
72 if mr != nil && id != 0 {
73 if mr != nil && id != 0 {
73 _ = s.Store.AttachPipelineMR(ctx, id, mr.ID)
74 _ = s.Store.AttachPipelineMR(ctx, id, mr.ID)
74 }
75 }
76 if id != 0 {
77 s.cancelOlder(ctx, repo.ID, ref, id)
78 }
75 return id
79 return id
76 }
80 }
77 if len(jobs) == 0 {
81 if len(jobs) == 0 {
@@ -87,9 +91,28 @@ func (s *Server) startPipeline(ctx context.Context, repo store.Repo, sha, ref, a
87 if mr != nil {
91 if mr != nil {
88 _ = s.Store.AttachPipelineMR(ctx, id, mr.ID)
92 _ = s.Store.AttachPipelineMR(ctx, id, mr.ID)
89 }
93 }
94 s.cancelOlder(ctx, repo.ID, ref, id)
90 return id
95 return id
91}
96}
92
97
98func (s *Server) cancelOlder(ctx context.Context, repoID int64, ref string, keep int64) {
99 running, pipes, err := s.Store.CancelInterruptible(ctx, repoID, ref, keep)
100 if err != nil {
101 if s.Log != nil {
102 s.Log.Error("interruptible", "err", err)
103 }
104 return
105 }
106 if s.Runner != nil {
107 s.Runner.Interrupt(running)
108 }
109 for _, id := range pipes {
110 if _, err := s.Store.RefreshPipelineStatus(ctx, id); err != nil && s.Log != nil {
111 s.Log.Error("pipelinestatus", "err", err)
112 }
113 }
114}
115
93func gitOne(ctx context.Context, gitBin, dir string, args ...string) string {
116func gitOne(ctx context.Context, gitBin, dir string, args ...string) string {
94 out, err := exec.CommandContext(ctx, gitBin, append([]string{"-C", dir}, args...)...).Output()
117 out, err := exec.CommandContext(ctx, gitBin, append([]string{"-C", dir}, args...)...).Output()
95 if err != nil {
118 if err != nil {
internal/server/mr_pipeline.go +34 −1
@@ -38,7 +38,15 @@ func (s *Server) planPipeline(ctx context.Context, repo store.Repo, sha, ref, so
38 38 DefaultBranch: repo.DefaultBranch, Source: source, ProjectPath: repo.FullPath(),
39 39 ProjectURL: base + "/" + repo.FullPath(), APIURL: base + "/api/v1", MR: mr,
40 40 })
41 active, err := spec.Select(ci.Eval{Env: env, Changed: changed, Files: files})
41 ev := ci.Eval{Env: env, Changed: changed, Files: files}
42 allow, err := spec.Allow(ev)
43 if err != nil {
44 return "", nil, err
45 }
46 if !allow {
47 return title, nil, nil
48 }
49 active, err := spec.Select(ev)
42 50 if err != nil {
43 51 return "", nil, err
44 52 }
@@ -59,6 +67,12 @@ func (s *Server) planPipeline(ctx context.Context, repo store.Repo, sha, ref, so
59 67 if paths == nil {
60 68 paths = []string{}
61 69 }
70 var services []store.Service
71 for _, svc := range j.Services {
72 services = append(services, store.Service{
73 Name: svc.Name, Alias: svc.Alias, Command: svc.Command, Entrypoint: svc.Entrypoint, Variables: svc.Variables,
74 })
75 }
62 76 jobs = append(jobs, store.NewJob{
63 77 Name: j.Name, Stage: j.Stage, Position: i, Image: j.Image,
64 78 Before: strings.Join(j.Before, "\n"), Script: strings.Join(j.Script, "\n"), After: strings.Join(j.After, "\n"),
@@ -66,6 +80,8 @@ func (s *Server) planPipeline(ctx context.Context, repo store.Repo, sha, ref, so
66 80 Variables: vars, ArtifactPaths: paths, ArtifactExpire: j.Artifacts.ExpireIn,
67 81 AllowFailure: j.AllowFailure, When: j.When, Timeout: j.Timeout, Tags: j.Tags,
68 82 CachePaths: j.Cache.Paths, CacheKey: j.Cache.Key, ResourceGroup: j.ResourceGroup,
83 RetryMax: j.Retry.Max, RetryWhen: j.Retry.When, Services: services,
84 Environment: j.Environment.Name, EnvironmentURL: j.Environment.URL, Interruptible: j.Interruptible,
69 85 })
70 86 }
71 87 return title, jobs, nil
@@ -88,6 +104,23 @@ func (s *Server) startMergeRequestPipelines(ctx context.Context, repo store.Repo
88 104 }
89 105 }
90 106
107 // rerunMergeRequestsOnTarget start een nieuwe merge-request-pipeline op de
108 // broncommit wanneer de doelbranch zelf een push krijgt.
109 func (s *Server) rerunMergeRequestsOnTarget(ctx context.Context, repo store.Repo, branch, actor string) {
110 mrs, err := s.Store.OpenMergeRequestsByTarget(ctx, repo.ID, branch)
111 if err != nil {
112 return
113 }
114 dir := gitrepo.Dir(s.StoragePath(), repo.OwnerPath, repo.Name)
115 for _, mr := range mrs {
116 sha := gitOne(ctx, s.Git, dir, "rev-parse", "--verify", "refs/heads/"+mr.SourceBranch)
117 if sha == "" {
118 continue
119 }
120 s.startOneMergePipeline(ctx, repo, sha, actor, "", mr)
121 }
122 }
123
91 124 func (s *Server) startOneMergePipeline(ctx context.Context, repo store.Repo, sha, actor, before string, mr store.MergeRequest) {
92 125 dir := gitrepo.Dir(s.StoragePath(), repo.OwnerPath, repo.Name)
93 126 base := strings.TrimRight(s.Cfg.BaseURL, "/")
@@ -38,7 +38,15 @@ func (s *Server) planPipeline(ctx context.Context, repo store.Repo, sha, ref, so
38 DefaultBranch: repo.DefaultBranch, Source: source, ProjectPath: repo.FullPath(),
38 DefaultBranch: repo.DefaultBranch, Source: source, ProjectPath: repo.FullPath(),
39 ProjectURL: base + "/" + repo.FullPath(), APIURL: base + "/api/v1", MR: mr,
39 ProjectURL: base + "/" + repo.FullPath(), APIURL: base + "/api/v1", MR: mr,
40 })
40 })
41 active, err := spec.Select(ci.Eval{Env: env, Changed: changed, Files: files})
41 ev := ci.Eval{Env: env, Changed: changed, Files: files}
42 allow, err := spec.Allow(ev)
43 if err != nil {
44 return "", nil, err
45 }
46 if !allow {
47 return title, nil, nil
48 }
49 active, err := spec.Select(ev)
42 if err != nil {
50 if err != nil {
43 return "", nil, err
51 return "", nil, err
44 }
52 }
@@ -59,6 +67,12 @@ func (s *Server) planPipeline(ctx context.Context, repo store.Repo, sha, ref, so
59 if paths == nil {
67 if paths == nil {
60 paths = []string{}
68 paths = []string{}
61 }
69 }
70 var services []store.Service
71 for _, svc := range j.Services {
72 services = append(services, store.Service{
73 Name: svc.Name, Alias: svc.Alias, Command: svc.Command, Entrypoint: svc.Entrypoint, Variables: svc.Variables,
74 })
75 }
62 jobs = append(jobs, store.NewJob{
76 jobs = append(jobs, store.NewJob{
63 Name: j.Name, Stage: j.Stage, Position: i, Image: j.Image,
77 Name: j.Name, Stage: j.Stage, Position: i, Image: j.Image,
64 Before: strings.Join(j.Before, "\n"), Script: strings.Join(j.Script, "\n"), After: strings.Join(j.After, "\n"),
78 Before: strings.Join(j.Before, "\n"), Script: strings.Join(j.Script, "\n"), After: strings.Join(j.After, "\n"),
@@ -66,6 +80,8 @@ func (s *Server) planPipeline(ctx context.Context, repo store.Repo, sha, ref, so
66 Variables: vars, ArtifactPaths: paths, ArtifactExpire: j.Artifacts.ExpireIn,
80 Variables: vars, ArtifactPaths: paths, ArtifactExpire: j.Artifacts.ExpireIn,
67 AllowFailure: j.AllowFailure, When: j.When, Timeout: j.Timeout, Tags: j.Tags,
81 AllowFailure: j.AllowFailure, When: j.When, Timeout: j.Timeout, Tags: j.Tags,
68 CachePaths: j.Cache.Paths, CacheKey: j.Cache.Key, ResourceGroup: j.ResourceGroup,
82 CachePaths: j.Cache.Paths, CacheKey: j.Cache.Key, ResourceGroup: j.ResourceGroup,
83 RetryMax: j.Retry.Max, RetryWhen: j.Retry.When, Services: services,
84 Environment: j.Environment.Name, EnvironmentURL: j.Environment.URL, Interruptible: j.Interruptible,
69 })
85 })
70 }
86 }
71 return title, jobs, nil
87 return title, jobs, nil
@@ -88,6 +104,23 @@ func (s *Server) startMergeRequestPipelines(ctx context.Context, repo store.Repo
88 }
104 }
89}
105}
90
106
107// rerunMergeRequestsOnTarget start een nieuwe merge-request-pipeline op de
108// broncommit wanneer de doelbranch zelf een push krijgt.
109func (s *Server) rerunMergeRequestsOnTarget(ctx context.Context, repo store.Repo, branch, actor string) {
110 mrs, err := s.Store.OpenMergeRequestsByTarget(ctx, repo.ID, branch)
111 if err != nil {
112 return
113 }
114 dir := gitrepo.Dir(s.StoragePath(), repo.OwnerPath, repo.Name)
115 for _, mr := range mrs {
116 sha := gitOne(ctx, s.Git, dir, "rev-parse", "--verify", "refs/heads/"+mr.SourceBranch)
117 if sha == "" {
118 continue
119 }
120 s.startOneMergePipeline(ctx, repo, sha, actor, "", mr)
121 }
122}
123
91func (s *Server) startOneMergePipeline(ctx context.Context, repo store.Repo, sha, actor, before string, mr store.MergeRequest) {
124func (s *Server) startOneMergePipeline(ctx context.Context, repo store.Repo, sha, actor, before string, mr store.MergeRequest) {
92 dir := gitrepo.Dir(s.StoragePath(), repo.OwnerPath, repo.Name)
125 dir := gitrepo.Dir(s.StoragePath(), repo.OwnerPath, repo.Name)
93 base := strings.TrimRight(s.Cfg.BaseURL, "/")
126 base := strings.TrimRight(s.Cfg.BaseURL, "/")
internal/server/mr_pipeline_test.go +23 −0
@@ -187,4 +187,27 @@ soft:
187 187 if !strings.Contains(page, "Warning") && !strings.Contains(page, "Passed") {
188 188 t.Fatalf("status na allow_failure:\n%s", page)
189 189 }
190
191 git(t, work, "checkout", "master")
192 writeFile(t, filepath.Join(work, "README.md"), "target\n")
193 git(t, work, "add", "README.md")
194 git(t, work, "commit", "-m", "target")
195 git(t, work, "push", "origin", "master")
196 pipes, err = st.ListPipelines(ctx, repo.ID)
197 if err != nil {
198 t.Fatal(err)
199 }
200 var newest int64
201 for _, p := range pipes {
202 if p.Source == "merge_request_event" && p.ID > newest {
203 newest = p.ID
204 if p.CommitSHA == mrPipe.CommitSHA && p.MRID == mrPipe.MRID {
205 continue
206 }
207 t.Fatalf("nieuwe mr-pipeline hoort op de broncommit: %+v", p)
208 }
209 }
210 if newest <= mrPipe.ID {
211 t.Fatalf("push naar de doelbranch start geen nieuwe mr-pipeline: %d", newest)
212 }
190 213 }
@@ -187,4 +187,27 @@ soft:
187 if !strings.Contains(page, "Warning") && !strings.Contains(page, "Passed") {
187 if !strings.Contains(page, "Warning") && !strings.Contains(page, "Passed") {
188 t.Fatalf("status na allow_failure:\n%s", page)
188 t.Fatalf("status na allow_failure:\n%s", page)
189 }
189 }
190
191 git(t, work, "checkout", "master")
192 writeFile(t, filepath.Join(work, "README.md"), "target\n")
193 git(t, work, "add", "README.md")
194 git(t, work, "commit", "-m", "target")
195 git(t, work, "push", "origin", "master")
196 pipes, err = st.ListPipelines(ctx, repo.ID)
197 if err != nil {
198 t.Fatal(err)
199 }
200 var newest int64
201 for _, p := range pipes {
202 if p.Source == "merge_request_event" && p.ID > newest {
203 newest = p.ID
204 if p.CommitSHA == mrPipe.CommitSHA && p.MRID == mrPipe.MRID {
205 continue
206 }
207 t.Fatalf("nieuwe mr-pipeline hoort op de broncommit: %+v", p)
208 }
209 }
210 if newest <= mrPipe.ID {
211 t.Fatalf("push naar de doelbranch start geen nieuwe mr-pipeline: %d", newest)
212 }
190}
213}
internal/server/pipeline_page.go +49 −28
@@ -52,33 +52,36 @@ type JobLink struct {
52 52
53 53 // JobPage is de console van één job.
54 54 type JobPage struct {
55 Job store.Job
56 Pipe store.Pipeline
57 Name string
58 Status string
59 Label string
60 Actor string
61 Avatar string
62 Initial string
63 When string
64 Duration string
65 Finished string
66 Queued string
67 Timeout string
68 Source string
69 Ref string
70 RefKind string
71 Short string
72 Title string
73 Stage string
74 PipeStatus string
75 PipeLabel string
76 Stages []string
77 Related []JobLink
78 Blocks []LogBlock
79 Running bool
80 LogBytes int
81 CanPlay bool
55 Job store.Job
56 Pipe store.Pipeline
57 Name string
58 Status string
59 Label string
60 Actor string
61 Avatar string
62 Initial string
63 When string
64 Duration string
65 Finished string
66 Queued string
67 Timeout string
68 Source string
69 Ref string
70 RefKind string
71 Short string
72 Title string
73 Stage string
74 PipeStatus string
75 PipeLabel string
76 Stages []string
77 Related []JobLink
78 Blocks []LogBlock
79 Running bool
80 LogBytes int
81 CanPlay bool
82 Environment string
83 EnvironmentURL string
84 EnvironmentLink string
82 85 }
83 86
84 87 // LogBlock is een losse regel of een inklapbare sectie.
@@ -183,6 +186,8 @@ func displayStatus(pipe string, jobs []store.Job) string {
183 186 return "manual"
184 187 case "queued":
185 188 return "pending"
189 case "canceled":
190 return "canceled"
186 191 }
187 192 warn := false
188 193 for _, j := range jobs {
@@ -228,7 +233,7 @@ func stageViews(lang i18n.Lang, jobs []store.Job, tr func(string) string) []Stag
228 233 }
229 234
230 235 func stageStatus(jobs []store.Job) string {
231 pending, failed, warning, running := false, false, false, false
236 pending, failed, warning, running, canceled := false, false, false, false, false
232 237 seen := false
233 238 for _, j := range jobs {
234 239 seen = true
@@ -237,6 +242,8 @@ func stageStatus(jobs []store.Job) string {
237 242 running = true
238 243 case j.Status == "failed" && !j.AllowFailure:
239 244 failed = true
245 case j.Status == "canceled" && !j.AllowFailure:
246 canceled = true
240 247 case j.Status == "failed" && j.AllowFailure:
241 248 warning = true
242 249 case j.Status == "queued" || j.Status == "manual":
@@ -252,6 +259,9 @@ func stageStatus(jobs []store.Job) string {
252 259 if failed {
253 260 return "failed"
254 261 }
262 if canceled {
263 return "canceled"
264 }
255 265 if warning {
256 266 return "warning"
257 267 }
@@ -451,7 +461,18 @@ func (s *Server) jobPage(r *http.Request, repo store.Repo, p store.Pipeline, job
451 461 PipeStatus: pipeStatus, PipeLabel: s.tr(r, "pipe."+pipeStatus),
452 462 Stages: stages, Related: related, Blocks: parseLog(job.Log), Running: job.Status == "running" || job.Status == "queued",
453 463 LogBytes: len(job.Log), CanPlay: job.Status == "manual" && role.CanWriteCode(),
464 Environment: job.Environment, EnvironmentURL: job.EnvironmentURL, EnvironmentLink: safeEnvironmentURL(job.EnvironmentURL),
465 }
466 }
467
468 func safeEnvironmentURL(raw string) string {
469 if strings.Contains(raw, "$") {
470 return ""
471 }
472 if strings.HasPrefix(raw, "https://") || strings.HasPrefix(raw, "http://") {
473 return raw
454 474 }
475 return ""
455 476 }
456 477
457 478 func parseLog(raw string) []LogBlock {
@@ -52,33 +52,36 @@ type JobLink struct {
52
52
53// JobPage is de console van één job.
53// JobPage is de console van één job.
54type JobPage struct {
54type JobPage struct {
55 Job store.Job
55 Job store.Job
56 Pipe store.Pipeline
56 Pipe store.Pipeline
57 Name string
57 Name string
58 Status string
58 Status string
59 Label string
59 Label string
60 Actor string
60 Actor string
61 Avatar string
61 Avatar string
62 Initial string
62 Initial string
63 When string
63 When string
64 Duration string
64 Duration string
65 Finished string
65 Finished string
66 Queued string
66 Queued string
67 Timeout string
67 Timeout string
68 Source string
68 Source string
69 Ref string
69 Ref string
70 RefKind string
70 RefKind string
71 Short string
71 Short string
72 Title string
72 Title string
73 Stage string
73 Stage string
74 PipeStatus string
74 PipeStatus string
75 PipeLabel string
75 PipeLabel string
76 Stages []string
76 Stages []string
77 Related []JobLink
77 Related []JobLink
78 Blocks []LogBlock
78 Blocks []LogBlock
79 Running bool
79 Running bool
80 LogBytes int
80 LogBytes int
81 CanPlay bool
81 CanPlay bool
82 Environment string
83 EnvironmentURL string
84 EnvironmentLink string
82}
85}
83
86
84// LogBlock is een losse regel of een inklapbare sectie.
87// LogBlock is een losse regel of een inklapbare sectie.
@@ -183,6 +186,8 @@ func displayStatus(pipe string, jobs []store.Job) string {
183 return "manual"
186 return "manual"
184 case "queued":
187 case "queued":
185 return "pending"
188 return "pending"
189 case "canceled":
190 return "canceled"
186 }
191 }
187 warn := false
192 warn := false
188 for _, j := range jobs {
193 for _, j := range jobs {
@@ -228,7 +233,7 @@ func stageViews(lang i18n.Lang, jobs []store.Job, tr func(string) string) []Stag
228}
233}
229
234
230func stageStatus(jobs []store.Job) string {
235func stageStatus(jobs []store.Job) string {
231 pending, failed, warning, running := false, false, false, false
236 pending, failed, warning, running, canceled := false, false, false, false, false
232 seen := false
237 seen := false
233 for _, j := range jobs {
238 for _, j := range jobs {
234 seen = true
239 seen = true
@@ -237,6 +242,8 @@ func stageStatus(jobs []store.Job) string {
237 running = true
242 running = true
238 case j.Status == "failed" && !j.AllowFailure:
243 case j.Status == "failed" && !j.AllowFailure:
239 failed = true
244 failed = true
245 case j.Status == "canceled" && !j.AllowFailure:
246 canceled = true
240 case j.Status == "failed" && j.AllowFailure:
247 case j.Status == "failed" && j.AllowFailure:
241 warning = true
248 warning = true
242 case j.Status == "queued" || j.Status == "manual":
249 case j.Status == "queued" || j.Status == "manual":
@@ -252,6 +259,9 @@ func stageStatus(jobs []store.Job) string {
252 if failed {
259 if failed {
253 return "failed"
260 return "failed"
254 }
261 }
262 if canceled {
263 return "canceled"
264 }
255 if warning {
265 if warning {
256 return "warning"
266 return "warning"
257 }
267 }
@@ -451,7 +461,18 @@ func (s *Server) jobPage(r *http.Request, repo store.Repo, p store.Pipeline, job
451 PipeStatus: pipeStatus, PipeLabel: s.tr(r, "pipe."+pipeStatus),
461 PipeStatus: pipeStatus, PipeLabel: s.tr(r, "pipe."+pipeStatus),
452 Stages: stages, Related: related, Blocks: parseLog(job.Log), Running: job.Status == "running" || job.Status == "queued",
462 Stages: stages, Related: related, Blocks: parseLog(job.Log), Running: job.Status == "running" || job.Status == "queued",
453 LogBytes: len(job.Log), CanPlay: job.Status == "manual" && role.CanWriteCode(),
463 LogBytes: len(job.Log), CanPlay: job.Status == "manual" && role.CanWriteCode(),
464 Environment: job.Environment, EnvironmentURL: job.EnvironmentURL, EnvironmentLink: safeEnvironmentURL(job.EnvironmentURL),
465 }
466}
467
468func safeEnvironmentURL(raw string) string {
469 if strings.Contains(raw, "$") {
470 return ""
471 }
472 if strings.HasPrefix(raw, "https://") || strings.HasPrefix(raw, "http://") {
473 return raw
454 }
474 }
475 return ""
455}
476}
456
477
457func parseLog(raw string) []LogBlock {
478func parseLog(raw string) []LogBlock {
internal/server/static/style.css +3 −1
@@ -346,11 +346,13 @@ svg.ico { width: 14px; height: 14px; flex: none; }
346 346 .status.failed { background: #fde8e8; color: #9f2d2d; }
347 347 .status.warning { background: #fff4e0; color: #8a5a00; }
348 348 .status.running { background: #e7f0ff; color: #1f6feb; }
349 .status.canceled { background: #f3f0f0; color: #57606a; }
349 350 .mark { position: relative; width: 16px; height: 16px; border-radius: 50%; display: inline-block; flex: none; background: #8c959f; }
350 351 .status.passed .mark, .stage-dot.passed .mark { background: #1aaa55; }
351 352 .status.failed .mark, .stage-dot.failed .mark { background: #dd2b0e; }
352 353 .status.warning .mark, .stage-dot.warning .mark { background: #c17d10; }
353 354 .status.running .mark, .stage-dot.running .mark { background: #1f75cb; }
355 .status.canceled .mark, .stage-dot.canceled .mark { background: #8c959f; }
354 356 .passed .mark::after {
355 357 content: ""; position: absolute; left: 5px; top: 2px; width: 4px; height: 8px;
356 358 border: solid #fff; border-width: 0 2px 2px 0; transform: rotate(45deg);
@@ -366,7 +368,7 @@ svg.ico { width: 14px; height: 14px; flex: none; }
366 368 .warning .mark::after {
367 369 content: ""; position: absolute; left: 7px; top: 11px; width: 2px; height: 2px; background: #fff;
368 370 }
369 .skipped .mark::after {
371 .skipped .mark::after, .canceled .mark::after {
370 372 content: ""; position: absolute; left: 4px; top: 7px; width: 8px; height: 2px; background: #fff;
371 373 }
372 374 .running .mark::after {
@@ -346,11 +346,13 @@ svg.ico { width: 14px; height: 14px; flex: none; }
346.status.failed { background: #fde8e8; color: #9f2d2d; }
346.status.failed { background: #fde8e8; color: #9f2d2d; }
347.status.warning { background: #fff4e0; color: #8a5a00; }
347.status.warning { background: #fff4e0; color: #8a5a00; }
348.status.running { background: #e7f0ff; color: #1f6feb; }
348.status.running { background: #e7f0ff; color: #1f6feb; }
349.status.canceled { background: #f3f0f0; color: #57606a; }
349.mark { position: relative; width: 16px; height: 16px; border-radius: 50%; display: inline-block; flex: none; background: #8c959f; }
350.mark { position: relative; width: 16px; height: 16px; border-radius: 50%; display: inline-block; flex: none; background: #8c959f; }
350.status.passed .mark, .stage-dot.passed .mark { background: #1aaa55; }
351.status.passed .mark, .stage-dot.passed .mark { background: #1aaa55; }
351.status.failed .mark, .stage-dot.failed .mark { background: #dd2b0e; }
352.status.failed .mark, .stage-dot.failed .mark { background: #dd2b0e; }
352.status.warning .mark, .stage-dot.warning .mark { background: #c17d10; }
353.status.warning .mark, .stage-dot.warning .mark { background: #c17d10; }
353.status.running .mark, .stage-dot.running .mark { background: #1f75cb; }
354.status.running .mark, .stage-dot.running .mark { background: #1f75cb; }
355.status.canceled .mark, .stage-dot.canceled .mark { background: #8c959f; }
354.passed .mark::after {
356.passed .mark::after {
355 content: ""; position: absolute; left: 5px; top: 2px; width: 4px; height: 8px;
357 content: ""; position: absolute; left: 5px; top: 2px; width: 4px; height: 8px;
356 border: solid #fff; border-width: 0 2px 2px 0; transform: rotate(45deg);
358 border: solid #fff; border-width: 0 2px 2px 0; transform: rotate(45deg);
@@ -366,7 +368,7 @@ svg.ico { width: 14px; height: 14px; flex: none; }
366.warning .mark::after {
368.warning .mark::after {
367 content: ""; position: absolute; left: 7px; top: 11px; width: 2px; height: 2px; background: #fff;
369 content: ""; position: absolute; left: 7px; top: 11px; width: 2px; height: 2px; background: #fff;
368}
370}
369.skipped .mark::after {
371.skipped .mark::after, .canceled .mark::after {
370 content: ""; position: absolute; left: 4px; top: 7px; width: 8px; height: 2px; background: #fff;
372 content: ""; position: absolute; left: 4px; top: 7px; width: 8px; height: 2px; background: #fff;
371}
373}
372.running .mark::after {
374.running .mark::after {
internal/server/templates/collab.html +1 −0
@@ -291,6 +291,7 @@
291 291 <dt>{{t "pipe.finished"}}</dt><dd>{{if .Finished}}{{.Finished}}{{else}}—{{end}}</dd>
292 292 <dt>{{t "pipe.queued"}}</dt><dd>{{if .Queued}}{{.Queued}}{{else}}—{{end}}</dd>
293 293 <dt>{{t "pipe.timeout"}}</dt><dd>{{.Timeout}}</dd>
294 {{if .Environment}}<dt>{{t "pipe.environment"}}</dt><dd>{{.Environment}}{{if .EnvironmentLink}} <a href="{{.EnvironmentLink}}">{{.EnvironmentURL}}</a>{{else if .EnvironmentURL}} {{.EnvironmentURL}}{{end}}</dd>{{end}}
294 295 <dt>{{t "pipe.runner"}}</dt><dd>{{t "pipe.runnerHere"}}</dd>
295 296 <dt>{{t "pipe.source"}}</dt><dd>{{.Source}}</dd>
296 297 <dt>{{t "pipe.commit"}}</dt><dd><code>{{.Short}}</code> <button type="button" class="iconbtn" data-copy="{{.Pipe.CommitSHA}}" aria-label="{{t "action.copy"}}">{{t "action.copy"}}</button>{{if .Title}}<div>{{.Title}}</div>{{end}}</dd>
@@ -291,6 +291,7 @@
291 <dt>{{t "pipe.finished"}}</dt><dd>{{if .Finished}}{{.Finished}}{{else}}—{{end}}</dd>
291 <dt>{{t "pipe.finished"}}</dt><dd>{{if .Finished}}{{.Finished}}{{else}}—{{end}}</dd>
292 <dt>{{t "pipe.queued"}}</dt><dd>{{if .Queued}}{{.Queued}}{{else}}—{{end}}</dd>
292 <dt>{{t "pipe.queued"}}</dt><dd>{{if .Queued}}{{.Queued}}{{else}}—{{end}}</dd>
293 <dt>{{t "pipe.timeout"}}</dt><dd>{{.Timeout}}</dd>
293 <dt>{{t "pipe.timeout"}}</dt><dd>{{.Timeout}}</dd>
294 {{if .Environment}}<dt>{{t "pipe.environment"}}</dt><dd>{{.Environment}}{{if .EnvironmentLink}} <a href="{{.EnvironmentLink}}">{{.EnvironmentURL}}</a>{{else if .EnvironmentURL}} {{.EnvironmentURL}}{{end}}</dd>{{end}}
294 <dt>{{t "pipe.runner"}}</dt><dd>{{t "pipe.runnerHere"}}</dd>
295 <dt>{{t "pipe.runner"}}</dt><dd>{{t "pipe.runnerHere"}}</dd>
295 <dt>{{t "pipe.source"}}</dt><dd>{{.Source}}</dd>
296 <dt>{{t "pipe.source"}}</dt><dd>{{.Source}}</dd>
296 <dt>{{t "pipe.commit"}}</dt><dd><code>{{.Short}}</code> <button type="button" class="iconbtn" data-copy="{{.Pipe.CommitSHA}}" aria-label="{{t "action.copy"}}">{{t "action.copy"}}</button>{{if .Title}}<div>{{.Title}}</div>{{end}}</dd>
297 <dt>{{t "pipe.commit"}}</dt><dd><code>{{.Short}}</code> <button type="button" class="iconbtn" data-copy="{{.Pipe.CommitSHA}}" aria-label="{{t "action.copy"}}">{{t "action.copy"}}</button>{{if .Title}}<div>{{.Title}}</div>{{end}}</dd>
internal/store/flow.go +110 −9
@@ -231,7 +231,19 @@ func (s *Store) SetMergeRequestStatus(ctx context.Context, id int64, status stri
231 231
232 232 // OpenMergeRequestsBySource geeft open verzoeken waarvan de bronbranch deze is.
233 233 func (s *Store) OpenMergeRequestsBySource(ctx context.Context, repoID int64, branch string) ([]MergeRequest, error) {
234 rows, err := s.db.QueryContext(ctx, mrSelect+` WHERE m.repo_id = ? AND m.source_branch = ? AND m.status = 'open' ORDER BY m.number`, repoID, branch)
234 return s.openMergeRequests(ctx, repoID, "source_branch", branch)
235 }
236
237 // OpenMergeRequestsByTarget geeft open verzoeken waarvan de doelbranch deze is.
238 func (s *Store) OpenMergeRequestsByTarget(ctx context.Context, repoID int64, branch string) ([]MergeRequest, error) {
239 return s.openMergeRequests(ctx, repoID, "target_branch", branch)
240 }
241
242 func (s *Store) openMergeRequests(ctx context.Context, repoID int64, column, branch string) ([]MergeRequest, error) {
243 if column != "source_branch" && column != "target_branch" {
244 return nil, errNotFound
245 }
246 rows, err := s.db.QueryContext(ctx, mrSelect+` WHERE m.repo_id = ? AND m.`+column+` = ? AND m.status = 'open' ORDER BY m.number`, repoID, branch)
235 247 if err != nil {
236 248 return nil, err
237 249 }
@@ -280,7 +292,7 @@ func PipelineBlocksMerge(jobs []Job) string {
280 292 for _, j := range jobs {
281 293 switch j.Status {
282 294 case "success", "skipped":
283 case "failed":
295 case "failed", "canceled":
284 296 if !j.AllowFailure {
285 297 hard = true
286 298 }
@@ -319,6 +331,21 @@ type NewJob struct {
319 331 CachePaths []string
320 332 CacheKey string
321 333 ResourceGroup string
334 RetryMax int
335 RetryWhen []string
336 Services []Service
337 Environment string
338 EnvironmentURL string
339 Interruptible bool
340 }
341
342 // Service is een sidecar die de runner naast de job start.
343 type Service struct {
344 Name string `json:"name"`
345 Alias string `json:"alias"`
346 Command []string `json:"command,omitempty"`
347 Entrypoint []string `json:"entrypoint,omitempty"`
348 Variables map[string]string `json:"variables,omitempty"`
322 349 }
323 350
324 351 // Pipeline is een draai van blue-ci-cd.yml.
@@ -363,6 +390,13 @@ type Job struct {
363 390 CachePaths []string
364 391 CacheKey string
365 392 ResourceGroup string
393 RetryMax int
394 RetryWhen []string
395 Attempt int
396 Services []Service
397 Environment string
398 EnvironmentURL string
399 Interruptible bool
366 400 Log string
367 401 CommitSHA string
368 402 Ref string
@@ -428,14 +462,21 @@ func (s *Store) CreatePipeline(ctx context.Context, repoID int64, sha, ref, sour
428 462 if j.When == "" {
429 463 j.When = "on_success"
430 464 }
465 if j.RetryWhen == nil {
466 j.RetryWhen = []string{}
467 }
468 if j.Services == nil {
469 j.Services = []Service{}
470 }
431 471 status := "queued"
432 472 if j.When == "manual" {
433 473 status = "manual"
434 474 }
435 _, err := tx.ExecContext(ctx, `INSERT INTO jobs (pipeline_id, repo_id, name, stage, position, status, image, script, needs_json, needs_set, variables_json, artifact_paths, artifact_expire_seconds, allow_failure, log, before_script, after_script, when_mode, timeout_seconds, tags_json, cache_paths, cache_key, resource_group, optional_json)
436 VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, '', ?, ?, ?, ?, ?, ?, ?, ?, ?)`,
475 _, err := tx.ExecContext(ctx, `INSERT INTO jobs (pipeline_id, repo_id, name, stage, position, status, image, script, needs_json, needs_set, variables_json, artifact_paths, artifact_expire_seconds, allow_failure, log, before_script, after_script, when_mode, timeout_seconds, tags_json, cache_paths, cache_key, resource_group, optional_json, retry_max, retry_when, services_json, environment_name, environment_url, interruptible)
476 VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, '', ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`,
437 477 id, repoID, j.Name, j.Stage, j.Position, status, j.Image, j.Script, mustJSON(j.Needs), boolInt(j.NeedsSet), mustJSON(j.Variables), mustJSON(j.ArtifactPaths), int(j.ArtifactExpire/time.Second), boolInt(j.AllowFailure),
438 j.Before, j.After, j.When, int(j.Timeout/time.Second), mustJSON(j.Tags), mustJSON(j.CachePaths), j.CacheKey, j.ResourceGroup, mustJSON(j.Optional))
478 j.Before, j.After, j.When, int(j.Timeout/time.Second), mustJSON(j.Tags), mustJSON(j.CachePaths), j.CacheKey, j.ResourceGroup, mustJSON(j.Optional),
479 j.RetryMax, mustJSON(j.RetryWhen), mustJSON(j.Services), j.Environment, j.EnvironmentURL, boolInt(j.Interruptible))
439 480 if err != nil {
440 481 return 0, err
441 482 }
@@ -513,19 +554,20 @@ func (s *Store) JobsByPipeline(ctx context.Context, pipelineID int64) ([]Job, er
513 554 return out, rows.Err()
514 555 }
515 556
516 const jobSelect = `SELECT j.id, j.pipeline_id, j.repo_id, j.name, j.stage, j.position, j.status, j.image, j.script, j.needs_json, j.needs_set, j.variables_json, j.artifact_paths, j.artifact_expire_seconds, j.allow_failure, j.log, p.commit_sha, p.ref, j.before_script, j.after_script, j.when_mode, j.timeout_seconds, j.tags_json, j.cache_paths, j.cache_key, j.resource_group, j.optional_json, j.started_at, j.finished_at
557 const jobSelect = `SELECT j.id, j.pipeline_id, j.repo_id, j.name, j.stage, j.position, j.status, j.image, j.script, j.needs_json, j.needs_set, j.variables_json, j.artifact_paths, j.artifact_expire_seconds, j.allow_failure, j.log, p.commit_sha, p.ref, j.before_script, j.after_script, j.when_mode, j.timeout_seconds, j.tags_json, j.cache_paths, j.cache_key, j.resource_group, j.optional_json, j.started_at, j.finished_at, j.retry_max, j.retry_when, j.attempt, j.services_json, j.environment_name, j.environment_url, j.interruptible
517 558 FROM jobs j JOIN pipelines p ON p.id = j.pipeline_id`
518 559
519 560 func scanJob(rows *sql.Rows) (Job, error) {
520 561 var j Job
521 var needs, vars, paths, tags, cache, optional string
522 var needsSet, allow, expire, timeout int
562 var needs, vars, paths, tags, cache, optional, retryWhen, services string
563 var needsSet, allow, expire, timeout, interruptible int
523 564 var started, finished sql.NullString
524 if err := rows.Scan(&j.ID, &j.PipelineID, &j.RepoID, &j.Name, &j.Stage, &j.Position, &j.Status, &j.Image, &j.Script, &needs, &needsSet, &vars, &paths, &expire, &allow, &j.Log, &j.CommitSHA, &j.Ref, &j.Before, &j.After, &j.When, &timeout, &tags, &cache, &j.CacheKey, &j.ResourceGroup, &optional, &started, &finished); err != nil {
565 if err := rows.Scan(&j.ID, &j.PipelineID, &j.RepoID, &j.Name, &j.Stage, &j.Position, &j.Status, &j.Image, &j.Script, &needs, &needsSet, &vars, &paths, &expire, &allow, &j.Log, &j.CommitSHA, &j.Ref, &j.Before, &j.After, &j.When, &timeout, &tags, &cache, &j.CacheKey, &j.ResourceGroup, &optional, &started, &finished, &j.RetryMax, &retryWhen, &j.Attempt, &services, &j.Environment, &j.EnvironmentURL, &interruptible); err != nil {
525 566 return Job{}, err
526 567 }
527 568 j.NeedsSet = needsSet == 1
528 569 j.AllowFailure = allow == 1
570 j.Interruptible = interruptible == 1
529 571 j.ArtifactExpire = time.Duration(expire) * time.Second
530 572 j.Timeout = time.Duration(timeout) * time.Second
531 573 j.StartedAt = parseTime(started.String)
@@ -536,6 +578,8 @@ func scanJob(rows *sql.Rows) (Job, error) {
536 578 _ = json.Unmarshal([]byte(tags), &j.Tags)
537 579 _ = json.Unmarshal([]byte(cache), &j.CachePaths)
538 580 _ = json.Unmarshal([]byte(optional), &j.Optional)
581 _ = json.Unmarshal([]byte(retryWhen), &j.RetryWhen)
582 _ = json.Unmarshal([]byte(services), &j.Services)
539 583 return j, nil
540 584 }
541 585
@@ -582,6 +626,55 @@ func (s *Store) AppendJobLog(ctx context.Context, id int64, chunk string) error
582 626 return err
583 627 }
584 628
629 // RequeueJob zet een mislukte job terug in de wachtrij voor een nieuwe poging.
630 func (s *Store) RequeueJob(ctx context.Context, id int64, attempt int, note string) error {
631 _, err := s.db.ExecContext(ctx, `UPDATE jobs SET status = 'queued', attempt = ?, log = log || ?, finished_at = NULL, started_at = NULL WHERE id = ?`, attempt, note, id)
632 return err
633 }
634
635 // CancelInterruptible annuleert onderbreekbare jobs van oudere pipelines op dezelfde ref.
636 // Lopende jobs komen terug zodat de runner hun context kan stoppen.
637 func (s *Store) CancelInterruptible(ctx context.Context, repoID int64, ref string, keep int64) (running []int64, pipes []int64, err error) {
638 rows, err := s.db.QueryContext(ctx, `SELECT j.id, j.pipeline_id, j.status FROM jobs j
639 JOIN pipelines p ON p.id = j.pipeline_id
640 WHERE p.repo_id = ? AND p.ref = ? AND p.id != ? AND j.interruptible = 1 AND j.status IN ('queued', 'running', 'manual')`, repoID, ref, keep)
641 if err != nil {
642 return nil, nil, err
643 }
644 defer rows.Close()
645 type row struct {
646 id, pipe int64
647 status string
648 }
649 var found []row
650 seenPipe := map[int64]bool{}
651 for rows.Next() {
652 var item row
653 if err := rows.Scan(&item.id, &item.pipe, &item.status); err != nil {
654 return nil, nil, err
655 }
656 found = append(found, item)
657 if !seenPipe[item.pipe] {
658 seenPipe[item.pipe] = true
659 pipes = append(pipes, item.pipe)
660 }
661 }
662 if err := rows.Err(); err != nil {
663 return nil, nil, err
664 }
665 note := time.Now().UTC().Format(time.RFC3339) + "\tcanceled because a newer pipeline started for the same ref\n"
666 for _, item := range found {
667 if item.status == "running" {
668 running = append(running, item.id)
669 continue
670 }
671 if _, err := s.db.ExecContext(ctx, `UPDATE jobs SET status = 'canceled', log = log || ?, finished_at = ? WHERE id = ? AND status IN ('queued', 'manual')`, note, now(), item.id); err != nil {
672 return running, pipes, err
673 }
674 }
675 return running, pipes, nil
676 }
677
585 678 // RefreshPipelineStatus berekent de pipelinestatus uit de jobs.
586 679 func (s *Store) RefreshPipelineStatus(ctx context.Context, pipelineID int64) (string, error) {
587 680 jobs, err := s.JobsByPipeline(ctx, pipelineID)
@@ -591,12 +684,17 @@ func (s *Store) RefreshPipelineStatus(ctx context.Context, pipelineID int64) (st
591 684 status := "success"
592 685 open := false
593 686 manual := false
687 canceled := false
594 688 for _, j := range jobs {
595 689 switch j.Status {
596 690 case "failed":
597 691 if !j.AllowFailure {
598 692 status = "failed"
599 693 }
694 case "canceled":
695 if !j.AllowFailure {
696 canceled = true
697 }
600 698 case "queued", "running":
601 699 open = true
602 700 case "manual":
@@ -611,6 +709,9 @@ func (s *Store) RefreshPipelineStatus(ctx context.Context, pipelineID int64) (st
611 709 _, err = s.db.ExecContext(ctx, `UPDATE pipelines SET status = 'running', finished_at = NULL WHERE id = ?`, pipelineID)
612 710 case status == "failed":
613 711 _, err = s.db.ExecContext(ctx, `UPDATE pipelines SET status = 'failed', finished_at = ? WHERE id = ?`, now(), pipelineID)
712 case canceled:
713 status = "canceled"
714 _, err = s.db.ExecContext(ctx, `UPDATE pipelines SET status = 'canceled', finished_at = ? WHERE id = ?`, now(), pipelineID)
614 715 case manual:
615 716 status = "manual"
616 717 _, err = s.db.ExecContext(ctx, `UPDATE pipelines SET status = 'manual', finished_at = NULL WHERE id = ?`, pipelineID)
@@ -231,7 +231,19 @@ func (s *Store) SetMergeRequestStatus(ctx context.Context, id int64, status stri
231
231
232// OpenMergeRequestsBySource geeft open verzoeken waarvan de bronbranch deze is.
232// OpenMergeRequestsBySource geeft open verzoeken waarvan de bronbranch deze is.
233func (s *Store) OpenMergeRequestsBySource(ctx context.Context, repoID int64, branch string) ([]MergeRequest, error) {
233func (s *Store) OpenMergeRequestsBySource(ctx context.Context, repoID int64, branch string) ([]MergeRequest, error) {
234 rows, err := s.db.QueryContext(ctx, mrSelect+` WHERE m.repo_id = ? AND m.source_branch = ? AND m.status = 'open' ORDER BY m.number`, repoID, branch)
234 return s.openMergeRequests(ctx, repoID, "source_branch", branch)
235}
236
237// OpenMergeRequestsByTarget geeft open verzoeken waarvan de doelbranch deze is.
238func (s *Store) OpenMergeRequestsByTarget(ctx context.Context, repoID int64, branch string) ([]MergeRequest, error) {
239 return s.openMergeRequests(ctx, repoID, "target_branch", branch)
240}
241
242func (s *Store) openMergeRequests(ctx context.Context, repoID int64, column, branch string) ([]MergeRequest, error) {
243 if column != "source_branch" && column != "target_branch" {
244 return nil, errNotFound
245 }
246 rows, err := s.db.QueryContext(ctx, mrSelect+` WHERE m.repo_id = ? AND m.`+column+` = ? AND m.status = 'open' ORDER BY m.number`, repoID, branch)
235 if err != nil {
247 if err != nil {
236 return nil, err
248 return nil, err
237 }
249 }
@@ -280,7 +292,7 @@ func PipelineBlocksMerge(jobs []Job) string {
280 for _, j := range jobs {
292 for _, j := range jobs {
281 switch j.Status {
293 switch j.Status {
282 case "success", "skipped":
294 case "success", "skipped":
283 case "failed":
295 case "failed", "canceled":
284 if !j.AllowFailure {
296 if !j.AllowFailure {
285 hard = true
297 hard = true
286 }
298 }
@@ -319,6 +331,21 @@ type NewJob struct {
319 CachePaths []string
331 CachePaths []string
320 CacheKey string
332 CacheKey string
321 ResourceGroup string
333 ResourceGroup string
334 RetryMax int
335 RetryWhen []string
336 Services []Service
337 Environment string
338 EnvironmentURL string
339 Interruptible bool
340}
341
342// Service is een sidecar die de runner naast de job start.
343type Service struct {
344 Name string `json:"name"`
345 Alias string `json:"alias"`
346 Command []string `json:"command,omitempty"`
347 Entrypoint []string `json:"entrypoint,omitempty"`
348 Variables map[string]string `json:"variables,omitempty"`
322}
349}
323
350
324// Pipeline is een draai van blue-ci-cd.yml.
351// Pipeline is een draai van blue-ci-cd.yml.
@@ -363,6 +390,13 @@ type Job struct {
363 CachePaths []string
390 CachePaths []string
364 CacheKey string
391 CacheKey string
365 ResourceGroup string
392 ResourceGroup string
393 RetryMax int
394 RetryWhen []string
395 Attempt int
396 Services []Service
397 Environment string
398 EnvironmentURL string
399 Interruptible bool
366 Log string
400 Log string
367 CommitSHA string
401 CommitSHA string
368 Ref string
402 Ref string
@@ -428,14 +462,21 @@ func (s *Store) CreatePipeline(ctx context.Context, repoID int64, sha, ref, sour
428 if j.When == "" {
462 if j.When == "" {
429 j.When = "on_success"
463 j.When = "on_success"
430 }
464 }
465 if j.RetryWhen == nil {
466 j.RetryWhen = []string{}
467 }
468 if j.Services == nil {
469 j.Services = []Service{}
470 }
431 status := "queued"
471 status := "queued"
432 if j.When == "manual" {
472 if j.When == "manual" {
433 status = "manual"
473 status = "manual"
434 }
474 }
435 _, err := tx.ExecContext(ctx, `INSERT INTO jobs (pipeline_id, repo_id, name, stage, position, status, image, script, needs_json, needs_set, variables_json, artifact_paths, artifact_expire_seconds, allow_failure, log, before_script, after_script, when_mode, timeout_seconds, tags_json, cache_paths, cache_key, resource_group, optional_json)
475 _, err := tx.ExecContext(ctx, `INSERT INTO jobs (pipeline_id, repo_id, name, stage, position, status, image, script, needs_json, needs_set, variables_json, artifact_paths, artifact_expire_seconds, allow_failure, log, before_script, after_script, when_mode, timeout_seconds, tags_json, cache_paths, cache_key, resource_group, optional_json, retry_max, retry_when, services_json, environment_name, environment_url, interruptible)
436 VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, '', ?, ?, ?, ?, ?, ?, ?, ?, ?)`,
476 VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, '', ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`,
437 id, repoID, j.Name, j.Stage, j.Position, status, j.Image, j.Script, mustJSON(j.Needs), boolInt(j.NeedsSet), mustJSON(j.Variables), mustJSON(j.ArtifactPaths), int(j.ArtifactExpire/time.Second), boolInt(j.AllowFailure),
477 id, repoID, j.Name, j.Stage, j.Position, status, j.Image, j.Script, mustJSON(j.Needs), boolInt(j.NeedsSet), mustJSON(j.Variables), mustJSON(j.ArtifactPaths), int(j.ArtifactExpire/time.Second), boolInt(j.AllowFailure),
438 j.Before, j.After, j.When, int(j.Timeout/time.Second), mustJSON(j.Tags), mustJSON(j.CachePaths), j.CacheKey, j.ResourceGroup, mustJSON(j.Optional))
478 j.Before, j.After, j.When, int(j.Timeout/time.Second), mustJSON(j.Tags), mustJSON(j.CachePaths), j.CacheKey, j.ResourceGroup, mustJSON(j.Optional),
479 j.RetryMax, mustJSON(j.RetryWhen), mustJSON(j.Services), j.Environment, j.EnvironmentURL, boolInt(j.Interruptible))
439 if err != nil {
480 if err != nil {
440 return 0, err
481 return 0, err
441 }
482 }
@@ -513,19 +554,20 @@ func (s *Store) JobsByPipeline(ctx context.Context, pipelineID int64) ([]Job, er
513 return out, rows.Err()
554 return out, rows.Err()
514}
555}
515
556
516const jobSelect = `SELECT j.id, j.pipeline_id, j.repo_id, j.name, j.stage, j.position, j.status, j.image, j.script, j.needs_json, j.needs_set, j.variables_json, j.artifact_paths, j.artifact_expire_seconds, j.allow_failure, j.log, p.commit_sha, p.ref, j.before_script, j.after_script, j.when_mode, j.timeout_seconds, j.tags_json, j.cache_paths, j.cache_key, j.resource_group, j.optional_json, j.started_at, j.finished_at
557const jobSelect = `SELECT j.id, j.pipeline_id, j.repo_id, j.name, j.stage, j.position, j.status, j.image, j.script, j.needs_json, j.needs_set, j.variables_json, j.artifact_paths, j.artifact_expire_seconds, j.allow_failure, j.log, p.commit_sha, p.ref, j.before_script, j.after_script, j.when_mode, j.timeout_seconds, j.tags_json, j.cache_paths, j.cache_key, j.resource_group, j.optional_json, j.started_at, j.finished_at, j.retry_max, j.retry_when, j.attempt, j.services_json, j.environment_name, j.environment_url, j.interruptible
517FROM jobs j JOIN pipelines p ON p.id = j.pipeline_id`
558FROM jobs j JOIN pipelines p ON p.id = j.pipeline_id`
518
559
519func scanJob(rows *sql.Rows) (Job, error) {
560func scanJob(rows *sql.Rows) (Job, error) {
520 var j Job
561 var j Job
521 var needs, vars, paths, tags, cache, optional string
562 var needs, vars, paths, tags, cache, optional, retryWhen, services string
522 var needsSet, allow, expire, timeout int
563 var needsSet, allow, expire, timeout, interruptible int
523 var started, finished sql.NullString
564 var started, finished sql.NullString
524 if err := rows.Scan(&j.ID, &j.PipelineID, &j.RepoID, &j.Name, &j.Stage, &j.Position, &j.Status, &j.Image, &j.Script, &needs, &needsSet, &vars, &paths, &expire, &allow, &j.Log, &j.CommitSHA, &j.Ref, &j.Before, &j.After, &j.When, &timeout, &tags, &cache, &j.CacheKey, &j.ResourceGroup, &optional, &started, &finished); err != nil {
565 if err := rows.Scan(&j.ID, &j.PipelineID, &j.RepoID, &j.Name, &j.Stage, &j.Position, &j.Status, &j.Image, &j.Script, &needs, &needsSet, &vars, &paths, &expire, &allow, &j.Log, &j.CommitSHA, &j.Ref, &j.Before, &j.After, &j.When, &timeout, &tags, &cache, &j.CacheKey, &j.ResourceGroup, &optional, &started, &finished, &j.RetryMax, &retryWhen, &j.Attempt, &services, &j.Environment, &j.EnvironmentURL, &interruptible); err != nil {
525 return Job{}, err
566 return Job{}, err
526 }
567 }
527 j.NeedsSet = needsSet == 1
568 j.NeedsSet = needsSet == 1
528 j.AllowFailure = allow == 1
569 j.AllowFailure = allow == 1
570 j.Interruptible = interruptible == 1
529 j.ArtifactExpire = time.Duration(expire) * time.Second
571 j.ArtifactExpire = time.Duration(expire) * time.Second
530 j.Timeout = time.Duration(timeout) * time.Second
572 j.Timeout = time.Duration(timeout) * time.Second
531 j.StartedAt = parseTime(started.String)
573 j.StartedAt = parseTime(started.String)
@@ -536,6 +578,8 @@ func scanJob(rows *sql.Rows) (Job, error) {
536 _ = json.Unmarshal([]byte(tags), &j.Tags)
578 _ = json.Unmarshal([]byte(tags), &j.Tags)
537 _ = json.Unmarshal([]byte(cache), &j.CachePaths)
579 _ = json.Unmarshal([]byte(cache), &j.CachePaths)
538 _ = json.Unmarshal([]byte(optional), &j.Optional)
580 _ = json.Unmarshal([]byte(optional), &j.Optional)
581 _ = json.Unmarshal([]byte(retryWhen), &j.RetryWhen)
582 _ = json.Unmarshal([]byte(services), &j.Services)
539 return j, nil
583 return j, nil
540}
584}
541
585
@@ -582,6 +626,55 @@ func (s *Store) AppendJobLog(ctx context.Context, id int64, chunk string) error
582 return err
626 return err
583}
627}
584
628
629// RequeueJob zet een mislukte job terug in de wachtrij voor een nieuwe poging.
630func (s *Store) RequeueJob(ctx context.Context, id int64, attempt int, note string) error {
631 _, err := s.db.ExecContext(ctx, `UPDATE jobs SET status = 'queued', attempt = ?, log = log || ?, finished_at = NULL, started_at = NULL WHERE id = ?`, attempt, note, id)
632 return err
633}
634
635// CancelInterruptible annuleert onderbreekbare jobs van oudere pipelines op dezelfde ref.
636// Lopende jobs komen terug zodat de runner hun context kan stoppen.
637func (s *Store) CancelInterruptible(ctx context.Context, repoID int64, ref string, keep int64) (running []int64, pipes []int64, err error) {
638 rows, err := s.db.QueryContext(ctx, `SELECT j.id, j.pipeline_id, j.status FROM jobs j
639 JOIN pipelines p ON p.id = j.pipeline_id
640 WHERE p.repo_id = ? AND p.ref = ? AND p.id != ? AND j.interruptible = 1 AND j.status IN ('queued', 'running', 'manual')`, repoID, ref, keep)
641 if err != nil {
642 return nil, nil, err
643 }
644 defer rows.Close()
645 type row struct {
646 id, pipe int64
647 status string
648 }
649 var found []row
650 seenPipe := map[int64]bool{}
651 for rows.Next() {
652 var item row
653 if err := rows.Scan(&item.id, &item.pipe, &item.status); err != nil {
654 return nil, nil, err
655 }
656 found = append(found, item)
657 if !seenPipe[item.pipe] {
658 seenPipe[item.pipe] = true
659 pipes = append(pipes, item.pipe)
660 }
661 }
662 if err := rows.Err(); err != nil {
663 return nil, nil, err
664 }
665 note := time.Now().UTC().Format(time.RFC3339) + "\tcanceled because a newer pipeline started for the same ref\n"
666 for _, item := range found {
667 if item.status == "running" {
668 running = append(running, item.id)
669 continue
670 }
671 if _, err := s.db.ExecContext(ctx, `UPDATE jobs SET status = 'canceled', log = log || ?, finished_at = ? WHERE id = ? AND status IN ('queued', 'manual')`, note, now(), item.id); err != nil {
672 return running, pipes, err
673 }
674 }
675 return running, pipes, nil
676}
677
585// RefreshPipelineStatus berekent de pipelinestatus uit de jobs.
678// RefreshPipelineStatus berekent de pipelinestatus uit de jobs.
586func (s *Store) RefreshPipelineStatus(ctx context.Context, pipelineID int64) (string, error) {
679func (s *Store) RefreshPipelineStatus(ctx context.Context, pipelineID int64) (string, error) {
587 jobs, err := s.JobsByPipeline(ctx, pipelineID)
680 jobs, err := s.JobsByPipeline(ctx, pipelineID)
@@ -591,12 +684,17 @@ func (s *Store) RefreshPipelineStatus(ctx context.Context, pipelineID int64) (st
591 status := "success"
684 status := "success"
592 open := false
685 open := false
593 manual := false
686 manual := false
687 canceled := false
594 for _, j := range jobs {
688 for _, j := range jobs {
595 switch j.Status {
689 switch j.Status {
596 case "failed":
690 case "failed":
597 if !j.AllowFailure {
691 if !j.AllowFailure {
598 status = "failed"
692 status = "failed"
599 }
693 }
694 case "canceled":
695 if !j.AllowFailure {
696 canceled = true
697 }
600 case "queued", "running":
698 case "queued", "running":
601 open = true
699 open = true
602 case "manual":
700 case "manual":
@@ -611,6 +709,9 @@ func (s *Store) RefreshPipelineStatus(ctx context.Context, pipelineID int64) (st
611 _, err = s.db.ExecContext(ctx, `UPDATE pipelines SET status = 'running', finished_at = NULL WHERE id = ?`, pipelineID)
709 _, err = s.db.ExecContext(ctx, `UPDATE pipelines SET status = 'running', finished_at = NULL WHERE id = ?`, pipelineID)
612 case status == "failed":
710 case status == "failed":
613 _, err = s.db.ExecContext(ctx, `UPDATE pipelines SET status = 'failed', finished_at = ? WHERE id = ?`, now(), pipelineID)
711 _, err = s.db.ExecContext(ctx, `UPDATE pipelines SET status = 'failed', finished_at = ? WHERE id = ?`, now(), pipelineID)
712 case canceled:
713 status = "canceled"
714 _, err = s.db.ExecContext(ctx, `UPDATE pipelines SET status = 'canceled', finished_at = ? WHERE id = ?`, now(), pipelineID)
614 case manual:
715 case manual:
615 status = "manual"
716 status = "manual"
616 _, err = s.db.ExecContext(ctx, `UPDATE pipelines SET status = 'manual', finished_at = NULL WHERE id = ?`, pipelineID)
717 _, err = s.db.ExecContext(ctx, `UPDATE pipelines SET status = 'manual', finished_at = NULL WHERE id = ?`, pipelineID)
internal/store/pipeline_status_test.go +81 −0
@@ -21,6 +21,9 @@ func TestPipelineBlocksMerge(t *testing.T) {
21 21 if PipelineBlocksMerge([]Job{{Status: "failed"}}) != "failed" {
22 22 t.Fatal("harde fout")
23 23 }
24 if PipelineBlocksMerge([]Job{{Status: "canceled"}}) != "failed" {
25 t.Fatal("geannuleerd houdt de merge tegen")
26 }
24 27 }
25 28
26 29 func TestRefreshWaitsForQueuedJobs(t *testing.T) {
@@ -68,3 +71,81 @@ func TestRefreshWaitsForQueuedJobs(t *testing.T) {
68 71 t.Fatalf("na het overslaan: %s %v", status, err)
69 72 }
70 73 }
74
75 func TestCancelInterruptibleAndRequeue(t *testing.T) {
76 s := openTest(t)
77 ctx := context.Background()
78 tim, err := s.UpsertUser(ctx, 1, "tim", "", false)
79 if err != nil {
80 t.Fatal(err)
81 }
82 repo, err := s.CreateRepo(ctx, "user", tim.ID, "ci", "", "master", "private")
83 if err != nil {
84 t.Fatal(err)
85 }
86 first, err := s.CreatePipeline(ctx, repo.ID, "abc", "refs/heads/master", "push", "tim", "eerste", []NewJob{
87 {Name: "stop", Stage: "test", Script: "echo", Interruptible: true},
88 {Name: "keep", Stage: "test", Script: "echo"},
89 })
90 if err != nil {
91 t.Fatal(err)
92 }
93 second, err := s.CreatePipeline(ctx, repo.ID, "def", "refs/heads/master", "push", "tim", "tweede", []NewJob{
94 {Name: "next", Stage: "test", Script: "echo", Interruptible: true},
95 })
96 if err != nil {
97 t.Fatal(err)
98 }
99 running, pipes, err := s.CancelInterruptible(ctx, repo.ID, "refs/heads/master", second)
100 if err != nil {
101 t.Fatal(err)
102 }
103 if len(running) != 0 || len(pipes) != 1 || pipes[0] != first {
104 t.Fatalf("running %v pipes %v", running, pipes)
105 }
106 jobs, err := s.JobsByPipeline(ctx, first)
107 if err != nil {
108 t.Fatal(err)
109 }
110 for _, j := range jobs {
111 if j.Name == "stop" && j.Status != "canceled" {
112 t.Fatalf("stop: %s", j.Status)
113 }
114 if j.Name == "keep" && j.Status != "queued" {
115 t.Fatalf("keep: %s", j.Status)
116 }
117 }
118 if err := s.FinishJob(ctx, jobs[0].ID, "failed", "nope\n"); err != nil {
119 t.Fatal(err)
120 }
121 if err := s.RequeueJob(ctx, jobs[0].ID, 1, "retry\n"); err != nil {
122 t.Fatal(err)
123 }
124 again, err := s.JobsByPipeline(ctx, first)
125 if err != nil {
126 t.Fatal(err)
127 }
128 if again[0].Status != "queued" || again[0].Attempt != 1 {
129 t.Fatalf("%s attempt %d", again[0].Status, again[0].Attempt)
130 }
131 status, err := s.RefreshPipelineStatus(ctx, first)
132 if err != nil || status != "running" {
133 t.Fatalf("na retry: %s %v", status, err)
134 }
135 for _, j := range again {
136 if j.Name == "keep" {
137 if err := s.SkipJob(ctx, j.ID, "skip"); err != nil {
138 t.Fatal(err)
139 }
140 }
141 if j.Name == "stop" {
142 if err := s.FinishJob(ctx, j.ID, "canceled", ""); err != nil {
143 t.Fatal(err)
144 }
145 }
146 }
147 status, err = s.RefreshPipelineStatus(ctx, first)
148 if err != nil || status != "canceled" {
149 t.Fatalf("pipeline: %s %v", status, err)
150 }
151 }
@@ -21,6 +21,9 @@ func TestPipelineBlocksMerge(t *testing.T) {
21 if PipelineBlocksMerge([]Job{{Status: "failed"}}) != "failed" {
21 if PipelineBlocksMerge([]Job{{Status: "failed"}}) != "failed" {
22 t.Fatal("harde fout")
22 t.Fatal("harde fout")
23 }
23 }
24 if PipelineBlocksMerge([]Job{{Status: "canceled"}}) != "failed" {
25 t.Fatal("geannuleerd houdt de merge tegen")
26 }
24}
27}
25
28
26func TestRefreshWaitsForQueuedJobs(t *testing.T) {
29func TestRefreshWaitsForQueuedJobs(t *testing.T) {
@@ -68,3 +71,81 @@ func TestRefreshWaitsForQueuedJobs(t *testing.T) {
68 t.Fatalf("na het overslaan: %s %v", status, err)
71 t.Fatalf("na het overslaan: %s %v", status, err)
69 }
72 }
70}
73}
74
75func TestCancelInterruptibleAndRequeue(t *testing.T) {
76 s := openTest(t)
77 ctx := context.Background()
78 tim, err := s.UpsertUser(ctx, 1, "tim", "", false)
79 if err != nil {
80 t.Fatal(err)
81 }
82 repo, err := s.CreateRepo(ctx, "user", tim.ID, "ci", "", "master", "private")
83 if err != nil {
84 t.Fatal(err)
85 }
86 first, err := s.CreatePipeline(ctx, repo.ID, "abc", "refs/heads/master", "push", "tim", "eerste", []NewJob{
87 {Name: "stop", Stage: "test", Script: "echo", Interruptible: true},
88 {Name: "keep", Stage: "test", Script: "echo"},
89 })
90 if err != nil {
91 t.Fatal(err)
92 }
93 second, err := s.CreatePipeline(ctx, repo.ID, "def", "refs/heads/master", "push", "tim", "tweede", []NewJob{
94 {Name: "next", Stage: "test", Script: "echo", Interruptible: true},
95 })
96 if err != nil {
97 t.Fatal(err)
98 }
99 running, pipes, err := s.CancelInterruptible(ctx, repo.ID, "refs/heads/master", second)
100 if err != nil {
101 t.Fatal(err)
102 }
103 if len(running) != 0 || len(pipes) != 1 || pipes[0] != first {
104 t.Fatalf("running %v pipes %v", running, pipes)
105 }
106 jobs, err := s.JobsByPipeline(ctx, first)
107 if err != nil {
108 t.Fatal(err)
109 }
110 for _, j := range jobs {
111 if j.Name == "stop" && j.Status != "canceled" {
112 t.Fatalf("stop: %s", j.Status)
113 }
114 if j.Name == "keep" && j.Status != "queued" {
115 t.Fatalf("keep: %s", j.Status)
116 }
117 }
118 if err := s.FinishJob(ctx, jobs[0].ID, "failed", "nope\n"); err != nil {
119 t.Fatal(err)
120 }
121 if err := s.RequeueJob(ctx, jobs[0].ID, 1, "retry\n"); err != nil {
122 t.Fatal(err)
123 }
124 again, err := s.JobsByPipeline(ctx, first)
125 if err != nil {
126 t.Fatal(err)
127 }
128 if again[0].Status != "queued" || again[0].Attempt != 1 {
129 t.Fatalf("%s attempt %d", again[0].Status, again[0].Attempt)
130 }
131 status, err := s.RefreshPipelineStatus(ctx, first)
132 if err != nil || status != "running" {
133 t.Fatalf("na retry: %s %v", status, err)
134 }
135 for _, j := range again {
136 if j.Name == "keep" {
137 if err := s.SkipJob(ctx, j.ID, "skip"); err != nil {
138 t.Fatal(err)
139 }
140 }
141 if j.Name == "stop" {
142 if err := s.FinishJob(ctx, j.ID, "canceled", ""); err != nil {
143 t.Fatal(err)
144 }
145 }
146 }
147 status, err = s.RefreshPipelineStatus(ctx, first)
148 if err != nil || status != "canceled" {
149 t.Fatalf("pipeline: %s %v", status, err)
150 }
151}
internal/store/store.go +14 −0
@@ -81,6 +81,13 @@ func (s *Store) migrate() error {
81 81 `ALTER TABLE jobs ADD COLUMN cache_key TEXT NOT NULL DEFAULT ''`,
82 82 `ALTER TABLE jobs ADD COLUMN resource_group TEXT NOT NULL DEFAULT ''`,
83 83 `ALTER TABLE jobs ADD COLUMN optional_json TEXT NOT NULL DEFAULT '[]'`,
84 `ALTER TABLE jobs ADD COLUMN retry_max INTEGER NOT NULL DEFAULT 0`,
85 `ALTER TABLE jobs ADD COLUMN retry_when TEXT NOT NULL DEFAULT '[]'`,
86 `ALTER TABLE jobs ADD COLUMN attempt INTEGER NOT NULL DEFAULT 0`,
87 `ALTER TABLE jobs ADD COLUMN services_json TEXT NOT NULL DEFAULT '[]'`,
88 `ALTER TABLE jobs ADD COLUMN environment_name TEXT NOT NULL DEFAULT ''`,
89 `ALTER TABLE jobs ADD COLUMN environment_url TEXT NOT NULL DEFAULT ''`,
90 `ALTER TABLE jobs ADD COLUMN interruptible INTEGER NOT NULL DEFAULT 0`,
84 91 `ALTER TABLE pipelines ADD COLUMN mr_id INTEGER NOT NULL DEFAULT 0`,
85 92 `ALTER TABLE groups ADD COLUMN owner_id INTEGER NOT NULL DEFAULT 0`,
86 93 }
@@ -271,6 +278,13 @@ CREATE TABLE IF NOT EXISTS jobs (
271 278 cache_key TEXT NOT NULL DEFAULT '',
272 279 resource_group TEXT NOT NULL DEFAULT '',
273 280 optional_json TEXT NOT NULL DEFAULT '[]',
281 retry_max INTEGER NOT NULL DEFAULT 0,
282 retry_when TEXT NOT NULL DEFAULT '[]',
283 attempt INTEGER NOT NULL DEFAULT 0,
284 services_json TEXT NOT NULL DEFAULT '[]',
285 environment_name TEXT NOT NULL DEFAULT '',
286 environment_url TEXT NOT NULL DEFAULT '',
287 interruptible INTEGER NOT NULL DEFAULT 0,
274 288 UNIQUE(pipeline_id, name)
275 289 );
276 290 CREATE TABLE IF NOT EXISTS artifacts (
@@ -81,6 +81,13 @@ func (s *Store) migrate() error {
81 `ALTER TABLE jobs ADD COLUMN cache_key TEXT NOT NULL DEFAULT ''`,
81 `ALTER TABLE jobs ADD COLUMN cache_key TEXT NOT NULL DEFAULT ''`,
82 `ALTER TABLE jobs ADD COLUMN resource_group TEXT NOT NULL DEFAULT ''`,
82 `ALTER TABLE jobs ADD COLUMN resource_group TEXT NOT NULL DEFAULT ''`,
83 `ALTER TABLE jobs ADD COLUMN optional_json TEXT NOT NULL DEFAULT '[]'`,
83 `ALTER TABLE jobs ADD COLUMN optional_json TEXT NOT NULL DEFAULT '[]'`,
84 `ALTER TABLE jobs ADD COLUMN retry_max INTEGER NOT NULL DEFAULT 0`,
85 `ALTER TABLE jobs ADD COLUMN retry_when TEXT NOT NULL DEFAULT '[]'`,
86 `ALTER TABLE jobs ADD COLUMN attempt INTEGER NOT NULL DEFAULT 0`,
87 `ALTER TABLE jobs ADD COLUMN services_json TEXT NOT NULL DEFAULT '[]'`,
88 `ALTER TABLE jobs ADD COLUMN environment_name TEXT NOT NULL DEFAULT ''`,
89 `ALTER TABLE jobs ADD COLUMN environment_url TEXT NOT NULL DEFAULT ''`,
90 `ALTER TABLE jobs ADD COLUMN interruptible INTEGER NOT NULL DEFAULT 0`,
84 `ALTER TABLE pipelines ADD COLUMN mr_id INTEGER NOT NULL DEFAULT 0`,
91 `ALTER TABLE pipelines ADD COLUMN mr_id INTEGER NOT NULL DEFAULT 0`,
85 `ALTER TABLE groups ADD COLUMN owner_id INTEGER NOT NULL DEFAULT 0`,
92 `ALTER TABLE groups ADD COLUMN owner_id INTEGER NOT NULL DEFAULT 0`,
86 }
93 }
@@ -271,6 +278,13 @@ CREATE TABLE IF NOT EXISTS jobs (
271 cache_key TEXT NOT NULL DEFAULT '',
278 cache_key TEXT NOT NULL DEFAULT '',
272 resource_group TEXT NOT NULL DEFAULT '',
279 resource_group TEXT NOT NULL DEFAULT '',
273 optional_json TEXT NOT NULL DEFAULT '[]',
280 optional_json TEXT NOT NULL DEFAULT '[]',
281 retry_max INTEGER NOT NULL DEFAULT 0,
282 retry_when TEXT NOT NULL DEFAULT '[]',
283 attempt INTEGER NOT NULL DEFAULT 0,
284 services_json TEXT NOT NULL DEFAULT '[]',
285 environment_name TEXT NOT NULL DEFAULT '',
286 environment_url TEXT NOT NULL DEFAULT '',
287 interruptible INTEGER NOT NULL DEFAULT 0,
274 UNIQUE(pipeline_id, name)
288 UNIQUE(pipeline_id, name)
275);
289);
276CREATE TABLE IF NOT EXISTS artifacts (
290CREATE TABLE IF NOT EXISTS artifacts (