BlueGit

timfa bluegit

Merge branch 'cursor/ci-spec-65da' into 'master'

CI-specificatie dichter bij GitLab

See merge request Timfa/bluegit!9

timfa Tim Falken · pushed by timfa

Authored 2026-10-22 · 3 weeks ago

Hash
e66981c7aba6e4343d95d02eece0bd78416c2396
13 files changed +2444 −514
docs/CI.md +30 −13
@@ -1,6 +1,6 @@
1 1 # blue-ci-cd.yml
2 2
3 Het bestand ligt in de root van de commit die de pipeline start. De vorm lijkt op GitLab CI: stages, jobs, needs en rules. Elke job draait in het image dat erbij staat.
3 Het bestand ligt in de root van de commit die de pipeline start. De vorm volgt GitLab CI zo ver dat een bestaand `.gitlab-ci.yml` meestal hernoemd kan worden naar `blue-ci-cd.yml`. Jobs mogen los op het hoogste niveau staan. Een ouder bestand met een `jobs:`-map blijft geldig.
4 4
5 5 ```yaml
6 6 stages: [test, build]
@@ -32,28 +32,45 @@ jobs:
32 32 | Veld | Betekenis |
33 33 | --- | --- |
34 34 | `stages` | Volgorde. Weggelaten: de stages in de volgorde waarin de jobs ze noemen. Een job zonder stage valt in `test`. |
35 | `image` | Standaardimage. Een job zonder eigen image gebruikt die. |
36 | `variables` | Omgevingsvariabelen voor elke job. Een job kan ze overschrijven. |
37 | `jobs.<naam>.script` | Niet-lege lijst. Ze draaien na elkaar in `sh`, met `set -e`. |
38 | `needs` | Jobs die eerst geslaagd of overgeslagen moeten zijn. `needs: []` start meteen. Weggelaten: wacht op alle eerdere stages. |
39 | `rules` | De job draait als minstens één `if` klopt. Zonder rules draait ze altijd. |
40 | `artifacts.paths` | Relatieve paden, geen `..`. Na succes bewaard, en in een volgende job gezet die deze job nodig heeft. |
41 | `artifacts.expire_in` | `30m`, `12h`, `14d`, `2w` of `never`. |
42 | `allow_failure` | Een mislukking houdt de rest niet tegen. |
35 | `image` | Standaardimage, als tekst of als `name:`. `default.image` wint als die gezet is. |
36 | `default` | `image`, `before_script`, `after_script`, `tags`, `timeout`, `cache` voor jobs die het zelf niet zetten. |
37 | `include` | Alleen `local`. Een job met dezelfde naam wordt samengevoegd; het buitenste bestand wint per veld. |
38 | `variables` | Tekst, of `{value: ...}`. Een job overschrijft dezelfde naam. |
39 | `before_script`, `script`, `after_script` | `after_script` draait ook als `script` mislukt. |
40 | `needs` | Namen, of `{job, optional}`. `needs: []` start meteen. Weggelaten: wacht op eerdere stages. |
41 | `rules` | De eerste rule die klopt wint. `if`, `changes`, `exists` en `when`. Zonder rules gelden `only` en `except`. |
42 | `only`, `except` | Taknaam, `branches`, `tags`, of `/regex/`. |
43 | `when` | `on_success` (standaard), `on_failure`, `always`, `manual`. In een rule ook `never`. |
44 | `allow_failure` | Een mislukking houdt de rest niet tegen. De job zelf wordt een waarschuwing. |
45 | `artifacts` | `paths` en `expire_in` (`30s`, `30m`, `12h`, `14d`, `2w`, `never`). |
46 | `cache` | `paths` en een tekstuele `key`. Lokaal per repository, niet gedeeld met een andere machine. |
47 | `resource_group` | Jobs met dezelfde naam in één repository draaien niet tegelijk. |
48 | `timeout` | Overschrijft `ci.timeout_seconds` voor die job. |
49 | `tags` | Bewaard bij de job. De ingebouwde runner draait elke job; er is geen tweede runner om op tags te kiezen. |
50 | `extends` | Eén naam of een lijst. Een job waarvan de naam met `.` begint is een sjabloon en draait niet. |
43 51
44 Een `if` is één van:
52 Een `if` is één vergelijking, zonder `&&` of `||`:
45 53
46 54 ```text
47 55 $NAAM == "waarde"
56 $NAAM == $ANDERE
57 $NAAM == null
48 58 $NAAM != "waarde"
49 59 $NAAM =~ /reguliere-expressie/
60 $NAAM !~ /reguliere-expressie/
50 61 ```
51 62
52 Beschikbaar in rules en in de container: `CI_COMMIT_SHA`, `CI_COMMIT_BRANCH`, `CI_COMMIT_REF`, `CI_PROJECT_PATH`, `CI_PIPELINE_ID`, `CI_JOB_ID`, `CI_JOB_NAME`, `CI_JOB_TOKEN`, `CI_API_URL`.
63 `changes` kijkt naar de diff van de push. Is die diff niet te bepalen, dan geldt de voorwaarde. `exists` eist dat elk pad in de commit staat.
64
65 Voorgedefinieerd, in rules en in de container: `CI`, `CI_COMMIT_SHA`, `CI_COMMIT_SHORT_SHA`, `CI_COMMIT_BRANCH`, `CI_COMMIT_TAG`, `CI_COMMIT_REF`, `CI_COMMIT_REF_NAME`, `CI_COMMIT_REF_SLUG`, `CI_COMMIT_TITLE`, `CI_DEFAULT_BRANCH`, `CI_PIPELINE_ID`, `CI_PIPELINE_SOURCE`, `CI_PIPELINE_URL`, `CI_PROJECT_PATH`, `CI_PROJECT_NAME`, `CI_PROJECT_NAMESPACE`, `CI_PROJECT_URL`, `CI_JOB_ID`, `CI_JOB_NAME`, `CI_JOB_STAGE`, `CI_JOB_URL`, `CI_JOB_TOKEN`, `CI_API_URL`, `CI_CONFIG_PATH`, `CI_SERVER_NAME`. Op een tag is `CI_COMMIT_BRANCH` leeg. `CI_PIPELINE_SOURCE` is `push` of `web`.
66
67 ## Bewust niet
68
69 `services`, `pages`, `workflow`, `spec`, een include van een ander project of van een URL, `parallel`/`matrix`, trigger- en child-pipelines, `environment`, `retry`, `coverage`, `id_tokens`, `inherit` en `hooks`. Die sleutels geven een waarschuwing in de editor en houden de rest van het bestand niet tegen. Er is één runner, in het BlueGit-proces, die Docker op dezelfde host aanroept.
53 70
54 71 ## Geheimen
55 72
56 Repositorygeheimen komen als omgevingsvariabele mee. Bij een groepsrepository eerst de groep, daarna de repository; dezelfde naam laat de repository winnen. Een geheim dat als beschermd is gemarkeerd, komt alleen mee als de branch op een beschermd patroon past.
73 Repositorygeheimen komen als omgevingsvariabele mee, zoals CI/CD-variabelen. Bij een groepsrepository eerst de groep, daarna de repository; dezelfde naam laat de repository winnen. Een geheim dat als beschermd is gemarkeerd, komt alleen mee als de branch of tag op een beschermd patroon past. Waarden van acht tekens of meer worden in het joblog vervangen door `[MASKED]`. `CI_JOB_TOKEN` ook.
57 74
58 75 ## Release vanuit een job
59 76
@@ -68,6 +85,6 @@ Het commit moet in de repository bestaan. Het asset moet een artifact zijn van e
68 85
69 86 ## Wanneer een job start
70 87
71 `Gate` in `internal/ci` geeft `ready`, `wait` of `skip`. `skip` volgt op een mislukte voorganger zonder `allow_failure`. De runner verwerkt pipelines elke twee seconden, met `ci.concurrency` tegelijk (standaard 1, passend bij de mini-PC).
88 `Gate` in `internal/ci` geeft `ready`, `wait`, `skip` of `fail`. `skip` volgt op een harde mislukking bij `when: on_success`, of op succes bij `when: on_failure`. `when: always` wacht tot de voorgangers klaar zijn en draait dan toch. Een job met `allow_failure` telt niet als harde mislukking. De runner verwerkt pipelines elke twee seconden, met `ci.concurrency` tegelijk (standaard 1).
72 89
73 90 Ontbreekt Docker, dan faalt de job met een regel in het log. De time-out is `ci.timeout_seconds` (standaard 1800).
@@ -1,6 +1,6 @@
1# blue-ci-cd.yml
1# blue-ci-cd.yml
2
2
3Het bestand ligt in de root van de commit die de pipeline start. De vorm lijkt op GitLab CI: stages, jobs, needs en rules. Elke job draait in het image dat erbij staat.
3Het bestand ligt in de root van de commit die de pipeline start. De vorm volgt GitLab CI zo ver dat een bestaand `.gitlab-ci.yml` meestal hernoemd kan worden naar `blue-ci-cd.yml`. Jobs mogen los op het hoogste niveau staan. Een ouder bestand met een `jobs:`-map blijft geldig.
4
4
5```yaml
5```yaml
6stages: [test, build]
6stages: [test, build]
@@ -32,28 +32,45 @@ jobs:
32| Veld | Betekenis |
32| Veld | Betekenis |
33| --- | --- |
33| --- | --- |
34| `stages` | Volgorde. Weggelaten: de stages in de volgorde waarin de jobs ze noemen. Een job zonder stage valt in `test`. |
34| `stages` | Volgorde. Weggelaten: de stages in de volgorde waarin de jobs ze noemen. Een job zonder stage valt in `test`. |
35| `image` | Standaardimage. Een job zonder eigen image gebruikt die. |
35| `image` | Standaardimage, als tekst of als `name:`. `default.image` wint als die gezet is. |
36| `variables` | Omgevingsvariabelen voor elke job. Een job kan ze overschrijven. |
36| `default` | `image`, `before_script`, `after_script`, `tags`, `timeout`, `cache` voor jobs die het zelf niet zetten. |
37| `jobs.<naam>.script` | Niet-lege lijst. Ze draaien na elkaar in `sh`, met `set -e`. |
37| `include` | Alleen `local`. Een job met dezelfde naam wordt samengevoegd; het buitenste bestand wint per veld. |
38| `needs` | Jobs die eerst geslaagd of overgeslagen moeten zijn. `needs: []` start meteen. Weggelaten: wacht op alle eerdere stages. |
38| `variables` | Tekst, of `{value: ...}`. Een job overschrijft dezelfde naam. |
39| `rules` | De job draait als minstens één `if` klopt. Zonder rules draait ze altijd. |
39| `before_script`, `script`, `after_script` | `after_script` draait ook als `script` mislukt. |
40| `artifacts.paths` | Relatieve paden, geen `..`. Na succes bewaard, en in een volgende job gezet die deze job nodig heeft. |
40| `needs` | Namen, of `{job, optional}`. `needs: []` start meteen. Weggelaten: wacht op eerdere stages. |
41| `artifacts.expire_in` | `30m`, `12h`, `14d`, `2w` of `never`. |
41| `rules` | De eerste rule die klopt wint. `if`, `changes`, `exists` en `when`. Zonder rules gelden `only` en `except`. |
42| `allow_failure` | Een mislukking houdt de rest niet tegen. |
42| `only`, `except` | Taknaam, `branches`, `tags`, of `/regex/`. |
43| `when` | `on_success` (standaard), `on_failure`, `always`, `manual`. In een rule ook `never`. |
44| `allow_failure` | Een mislukking houdt de rest niet tegen. De job zelf wordt een waarschuwing. |
45| `artifacts` | `paths` en `expire_in` (`30s`, `30m`, `12h`, `14d`, `2w`, `never`). |
46| `cache` | `paths` en een tekstuele `key`. Lokaal per repository, niet gedeeld met een andere machine. |
47| `resource_group` | Jobs met dezelfde naam in één repository draaien niet tegelijk. |
48| `timeout` | Overschrijft `ci.timeout_seconds` voor die job. |
49| `tags` | Bewaard bij de job. De ingebouwde runner draait elke job; er is geen tweede runner om op tags te kiezen. |
50| `extends` | Eén naam of een lijst. Een job waarvan de naam met `.` begint is een sjabloon en draait niet. |
43
51
44Een `if` is één van:
52Een `if` is één vergelijking, zonder `&&` of `||`:
45
53
46```text
54```text
47$NAAM == "waarde"
55$NAAM == "waarde"
56$NAAM == $ANDERE
57$NAAM == null
48$NAAM != "waarde"
58$NAAM != "waarde"
49$NAAM =~ /reguliere-expressie/
59$NAAM =~ /reguliere-expressie/
60$NAAM !~ /reguliere-expressie/
50```
61```
51
62
52Beschikbaar in rules en in de container: `CI_COMMIT_SHA`, `CI_COMMIT_BRANCH`, `CI_COMMIT_REF`, `CI_PROJECT_PATH`, `CI_PIPELINE_ID`, `CI_JOB_ID`, `CI_JOB_NAME`, `CI_JOB_TOKEN`, `CI_API_URL`.
63`changes` kijkt naar de diff van de push. Is die diff niet te bepalen, dan geldt de voorwaarde. `exists` eist dat elk pad in de commit staat.
64
65Voorgedefinieerd, in rules en in de container: `CI`, `CI_COMMIT_SHA`, `CI_COMMIT_SHORT_SHA`, `CI_COMMIT_BRANCH`, `CI_COMMIT_TAG`, `CI_COMMIT_REF`, `CI_COMMIT_REF_NAME`, `CI_COMMIT_REF_SLUG`, `CI_COMMIT_TITLE`, `CI_DEFAULT_BRANCH`, `CI_PIPELINE_ID`, `CI_PIPELINE_SOURCE`, `CI_PIPELINE_URL`, `CI_PROJECT_PATH`, `CI_PROJECT_NAME`, `CI_PROJECT_NAMESPACE`, `CI_PROJECT_URL`, `CI_JOB_ID`, `CI_JOB_NAME`, `CI_JOB_STAGE`, `CI_JOB_URL`, `CI_JOB_TOKEN`, `CI_API_URL`, `CI_CONFIG_PATH`, `CI_SERVER_NAME`. Op een tag is `CI_COMMIT_BRANCH` leeg. `CI_PIPELINE_SOURCE` is `push` of `web`.
66
67## Bewust niet
68
69`services`, `pages`, `workflow`, `spec`, een include van een ander project of van een URL, `parallel`/`matrix`, trigger- en child-pipelines, `environment`, `retry`, `coverage`, `id_tokens`, `inherit` en `hooks`. Die sleutels geven een waarschuwing in de editor en houden de rest van het bestand niet tegen. Er is één runner, in het BlueGit-proces, die Docker op dezelfde host aanroept.
53
70
54## Geheimen
71## Geheimen
55
72
56Repositorygeheimen komen als omgevingsvariabele mee. Bij een groepsrepository eerst de groep, daarna de repository; dezelfde naam laat de repository winnen. Een geheim dat als beschermd is gemarkeerd, komt alleen mee als de branch op een beschermd patroon past.
73Repositorygeheimen komen als omgevingsvariabele mee, zoals CI/CD-variabelen. Bij een groepsrepository eerst de groep, daarna de repository; dezelfde naam laat de repository winnen. Een geheim dat als beschermd is gemarkeerd, komt alleen mee als de branch of tag op een beschermd patroon past. Waarden van acht tekens of meer worden in het joblog vervangen door `[MASKED]`. `CI_JOB_TOKEN` ook.
57
74
58## Release vanuit een job
75## Release vanuit een job
59
76
@@ -68,6 +85,6 @@ Het commit moet in de repository bestaan. Het asset moet een artifact zijn van e
68
85
69## Wanneer een job start
86## Wanneer een job start
70
87
71`Gate` in `internal/ci` geeft `ready`, `wait` of `skip`. `skip` volgt op een mislukte voorganger zonder `allow_failure`. De runner verwerkt pipelines elke twee seconden, met `ci.concurrency` tegelijk (standaard 1, passend bij de mini-PC).
88`Gate` in `internal/ci` geeft `ready`, `wait`, `skip` of `fail`. `skip` volgt op een harde mislukking bij `when: on_success`, of op succes bij `when: on_failure`. `when: always` wacht tot de voorgangers klaar zijn en draait dan toch. Een job met `allow_failure` telt niet als harde mislukking. De runner verwerkt pipelines elke twee seconden, met `ci.concurrency` tegelijk (standaard 1).
72
89
73Ontbreekt Docker, dan faalt de job met een regel in het log. De time-out is `ci.timeout_seconds` (standaard 1800).
90Ontbreekt Docker, dan faalt de job met een regel in het log. De time-out is `ci.timeout_seconds` (standaard 1800).
internal/ci/eval.go added +488 −0
@@ -0,0 +1,488 @@
1 package ci
2
3 import (
4 "fmt"
5 "path"
6 "regexp"
7 "strings"
8
9 "gitlab.com/Timfa/bluegit/internal/i18n"
10 )
11
12 // Variables zet de voorgedefinieerde CI_-variabelen. Job-specifieke namen
13 // vult de runner zelf aan.
14 func Variables(r Ref) map[string]string {
15 name := r.Name
16 kind := r.Kind
17 if kind == "" {
18 name, kind = SplitRef(r.FullRef)
19 }
20 short := r.SHA
21 if len(short) > 8 {
22 short = short[:8]
23 }
24 project := r.ProjectPath
25 ns, base := splitProject(project)
26 env := map[string]string{
27 "CI": "true",
28 "CI_CONFIG_PATH": "blue-ci-cd.yml",
29 "CI_SERVER_NAME": "BlueGit",
30 "CI_COMMIT_SHA": r.SHA,
31 "CI_COMMIT_SHORT_SHA": short,
32 "CI_COMMIT_REF": r.FullRef,
33 "CI_COMMIT_REF_NAME": name,
34 "CI_COMMIT_REF_SLUG": slug(name),
35 "CI_COMMIT_TITLE": r.Title,
36 "CI_COMMIT_BRANCH": "",
37 "CI_COMMIT_TAG": "",
38 "CI_DEFAULT_BRANCH": r.DefaultBranch,
39 "CI_PIPELINE_ID": fmt.Sprint(r.PipelineID),
40 "CI_PIPELINE_SOURCE": r.Source,
41 "CI_PIPELINE_URL": strings.TrimRight(r.ProjectURL, "/") + "/-/pipelines/" + fmt.Sprint(r.PipelineID),
42 "CI_PROJECT_PATH": project,
43 "CI_PROJECT_NAME": base,
44 "CI_PROJECT_NAMESPACE": ns,
45 "CI_PROJECT_URL": r.ProjectURL,
46 "CI_API_URL": r.APIURL,
47 }
48 if kind == "tag" {
49 env["CI_COMMIT_TAG"] = name
50 } else {
51 env["CI_COMMIT_BRANCH"] = name
52 }
53 if env["CI_PIPELINE_SOURCE"] == "" {
54 env["CI_PIPELINE_SOURCE"] = "push"
55 }
56 return env
57 }
58
59 // SplitRef haalt de korte naam en het soort uit refs/heads of refs/tags.
60 func SplitRef(ref string) (name, kind string) {
61 switch {
62 case strings.HasPrefix(ref, "refs/tags/"):
63 return strings.TrimPrefix(ref, "refs/tags/"), "tag"
64 case strings.HasPrefix(ref, "refs/heads/"):
65 return strings.TrimPrefix(ref, "refs/heads/"), "branch"
66 default:
67 return ref, "branch"
68 }
69 }
70
71 func splitProject(path string) (ns, name string) {
72 i := strings.LastIndex(path, "/")
73 if i < 0 {
74 return "", path
75 }
76 return path[:i], path[i+1:]
77 }
78
79 func slug(s string) string {
80 var b strings.Builder
81 prev := false
82 for _, r := range strings.ToLower(s) {
83 if (r >= 'a' && r <= 'z') || (r >= '0' && r <= '9') {
84 b.WriteRune(r)
85 prev = false
86 continue
87 }
88 if !prev && b.Len() > 0 {
89 b.WriteByte('-')
90 prev = true
91 }
92 }
93 return strings.Trim(b.String(), "-")
94 }
95
96 // Select houdt de jobs waarvan de rules, only en except kloppen.
97 // De When van de job wordt de when van de winnende rule.
98 func (p Pipeline) Select(ev Eval) ([]Job, error) {
99 if ev.Env == nil {
100 ev.Env = map[string]string{}
101 }
102 var out []Job
103 active := map[string]bool{}
104 var pending []Job
105 for _, j := range p.Jobs {
106 ok, when, err := j.matches(ev)
107 if err != nil {
108 return nil, fmt.Errorf("job %s: %w", j.Name, err)
109 }
110 if !ok {
111 continue
112 }
113 if when == "" {
114 when = "on_success"
115 }
116 j.When = when
117 pending = append(pending, j)
118 active[j.Name] = true
119 }
120 for i := range pending {
121 var needs []string
122 for _, n := range pending[i].Needs {
123 if active[n] {
124 needs = append(needs, n)
125 continue
126 }
127 if contains(pending[i].Optional, n) {
128 continue
129 }
130 needs = append(needs, n)
131 }
132 if needs == nil {
133 needs = []string{}
134 }
135 pending[i].Needs = needs
136 out = append(out, pending[i])
137 }
138 return out, nil
139 }
140
141 // ActiveJobs is Select met alleen variabelen. changes en exists gelden dan als waar.
142 func (p Pipeline) ActiveJobs(env map[string]string) ([]Job, error) {
143 return p.Select(Eval{Env: env})
144 }
145
146 func (j Job) matches(ev Eval) (bool, string, error) {
147 when := j.When
148 if when == "" {
149 when = "on_success"
150 }
151 if len(j.Rules) == 0 {
152 ok, err := onlyExcept(j, ev)
153 return ok, when, err
154 }
155 for _, r := range j.Rules {
156 ok, err := ruleMatch(r, ev)
157 if err != nil {
158 return false, "", err
159 }
160 if !ok {
161 continue
162 }
163 w := r.When
164 if w == "" {
165 w = when
166 }
167 if w == "never" {
168 return false, w, nil
169 }
170 return true, w, nil
171 }
172 return false, "", nil
173 }
174
175 func ruleMatch(r Rule, ev Eval) (bool, error) {
176 if r.If != "" {
177 ok, err := evalRule(r.If, ev.Env)
178 if err != nil || !ok {
179 return ok, err
180 }
181 }
182 if len(r.Changes) > 0 && ev.Changed != nil {
183 if !anyGlob(r.Changes, ev.Changed) {
184 return false, nil
185 }
186 }
187 if len(r.Exists) > 0 && ev.Files != nil {
188 if !allExist(r.Exists, ev.Files) {
189 return false, nil
190 }
191 }
192 if r.If == "" && len(r.Changes) == 0 && len(r.Exists) == 0 && r.When == "" {
193 return false, nil
194 }
195 return true, nil
196 }
197
198 func onlyExcept(j Job, ev Eval) (bool, error) {
199 if len(j.Only) == 0 && len(j.Except) == 0 {
200 return true, nil
201 }
202 name := ev.Env["CI_COMMIT_REF_NAME"]
203 if name == "" {
204 name = ev.Env["CI_COMMIT_BRANCH"]
205 }
206 tag := ev.Env["CI_COMMIT_TAG"] != ""
207 if len(j.Only) > 0 {
208 ok, err := refListed(j.Only, name, tag)
209 if err != nil || !ok {
210 return false, err
211 }
212 }
213 if len(j.Except) > 0 {
214 ok, err := refListed(j.Except, name, tag)
215 if err != nil {
216 return false, err
217 }
218 if ok {
219 return false, nil
220 }
221 }
222 return true, nil
223 }
224
225 func refListed(list []string, name string, tag bool) (bool, error) {
226 for _, item := range list {
227 switch item {
228 case "branches":
229 if !tag {
230 return true, nil
231 }
232 case "tags":
233 if tag {
234 return true, nil
235 }
236 default:
237 if strings.HasPrefix(item, "/") && strings.HasSuffix(item, "/") && len(item) > 2 {
238 re, err := regexp.Compile(item[1 : len(item)-1])
239 if err != nil {
240 return false, i18n.E("ci.err.regex", item, err.Error())
241 }
242 if re.MatchString(name) {
243 return true, nil
244 }
245 continue
246 }
247 if item == name {
248 return true, nil
249 }
250 }
251 }
252 return false, nil
253 }
254
255 func anyGlob(patterns, names []string) bool {
256 for _, name := range names {
257 for _, pattern := range patterns {
258 if Glob(pattern, name) {
259 return true
260 }
261 }
262 }
263 return false
264 }
265
266 func allExist(patterns, files []string) bool {
267 for _, pattern := range patterns {
268 found := false
269 for _, file := range files {
270 if file == pattern || Glob(pattern, file) {
271 found = true
272 break
273 }
274 }
275 if !found {
276 return false
277 }
278 }
279 return true
280 }
281
282 // Glob volgt GitLab-padpatronen: * blijft in één segment, ** loopt door mappen.
283 func Glob(pattern, name string) bool {
284 pattern = strings.TrimPrefix(pattern, "/")
285 name = strings.TrimPrefix(name, "/")
286 if pattern == name {
287 return true
288 }
289 re := globRegexp(pattern)
290 if re.MatchString(name) {
291 return true
292 }
293 if !strings.Contains(pattern, "/") && re.MatchString(path.Base(name)) {
294 return true
295 }
296 return false
297 }
298
299 func globRegexp(pattern string) *regexp.Regexp {
300 var b strings.Builder
301 b.WriteString("^")
302 for i := 0; i < len(pattern); i++ {
303 if pattern[i] == '*' && i+1 < len(pattern) && pattern[i+1] == '*' {
304 b.WriteString(".*")
305 i++
306 if i+1 < len(pattern) && pattern[i+1] == '/' {
307 i++
308 }
309 continue
310 }
311 switch pattern[i] {
312 case '*':
313 b.WriteString("[^/]*")
314 case '?':
315 b.WriteString("[^/]")
316 default:
317 b.WriteString(regexp.QuoteMeta(pattern[i : i+1]))
318 }
319 }
320 b.WriteString("$")
321 return regexp.MustCompile(b.String())
322 }
323
324 // Gate beslist of een job mag starten.
325 // "ready" betekent dat alle voorgangers klaar zijn.
326 // "wait" betekent dat er nog een voorganger loopt of in de wachtrij staat.
327 // "skip" betekent dat een voorganger mislukte zonder allow_failure,
328 // of dat when: on_failure geen mislukking zag.
329 // "fail" betekent dat een verplichte need niet in deze pipeline zit.
330 //
331 // NeedsSet volgt de genoemde jobs, ook als de lijst leeg is.
332 // Zonder needs wacht de job op alle jobs in eerdere stages.
333 func Gate(jobs []Job, status map[string]string, job Job) string {
334 byName := map[string]Job{}
335 for _, j := range jobs {
336 byName[j.Name] = j
337 }
338 optional := map[string]bool{}
339 for _, n := range job.Optional {
340 optional[n] = true
341 }
342 var deps []Job
343 if job.NeedsSet {
344 for _, n := range job.Needs {
345 d, ok := byName[n]
346 if !ok {
347 if optional[n] {
348 continue
349 }
350 return "fail"
351 }
352 deps = append(deps, d)
353 }
354 } else {
355 stagePos := map[string]int{}
356 order := 0
357 for _, j := range jobs {
358 if _, ok := stagePos[j.Stage]; !ok {
359 stagePos[j.Stage] = order
360 order++
361 }
362 }
363 my := stagePos[job.Stage]
364 for _, j := range jobs {
365 if stagePos[j.Stage] < my {
366 deps = append(deps, j)
367 }
368 }
369 }
370 failedAny := false
371 failedHard := false
372 for _, d := range deps {
373 switch status[d.Name] {
374 case "success", "skipped":
375 continue
376 case "failed":
377 failedAny = true
378 if !d.AllowFailure {
379 failedHard = true
380 }
381 case "manual":
382 if d.AllowFailure {
383 continue
384 }
385 return "wait"
386 default:
387 return "wait"
388 }
389 }
390 when := job.When
391 if when == "" {
392 when = "on_success"
393 }
394 switch when {
395 case "always":
396 return "ready"
397 case "on_failure":
398 if failedAny {
399 return "ready"
400 }
401 return "skip"
402 default:
403 if failedHard {
404 return "skip"
405 }
406 return "ready"
407 }
408 }
409
410 var (
411 reEq = regexp.MustCompile(`^\$([A-Za-z_][A-Za-z0-9_]*)\s*==\s*(?:"([^"]*)"|\$([A-Za-z_][A-Za-z0-9_]*)|null)$`)
412 reNe = regexp.MustCompile(`^\$([A-Za-z_][A-Za-z0-9_]*)\s*!=\s*(?:"([^"]*)"|\$([A-Za-z_][A-Za-z0-9_]*)|null)$`)
413 reRe = regexp.MustCompile(`^\$([A-Za-z_][A-Za-z0-9_]*)\s*=~\s*/(.*)/$`)
414 reNRe = regexp.MustCompile(`^\$([A-Za-z_][A-Za-z0-9_]*)\s*!~\s*/(.*)/$`)
415 )
416
417 type compiledRule struct {
418 kind string
419 name string
420 arg string
421 other string
422 null bool
423 re *regexp.Regexp
424 }
425
426 func compileRule(expr string) (compiledRule, error) {
427 expr = strings.TrimSpace(expr)
428 if strings.Contains(expr, "&&") || strings.Contains(expr, "||") {
429 return compiledRule{}, i18n.E("ci.err.ruleCombine", expr)
430 }
431 if m := reEq.FindStringSubmatch(expr); m != nil {
432 return compiledRule{kind: "eq", name: m[1], arg: m[2], other: m[3], null: m[2] == "" && m[3] == "" && strings.HasSuffix(strings.TrimSpace(expr), "null")}, nil
433 }
434 if m := reNe.FindStringSubmatch(expr); m != nil {
435 return compiledRule{kind: "ne", name: m[1], arg: m[2], other: m[3], null: m[2] == "" && m[3] == "" && strings.HasSuffix(strings.TrimSpace(expr), "null")}, nil
436 }
437 if m := reRe.FindStringSubmatch(expr); m != nil {
438 re, err := regexp.Compile(m[2])
439 if err != nil {
440 return compiledRule{}, i18n.E("ci.err.regex", expr, err.Error())
441 }
442 return compiledRule{kind: "re", name: m[1], re: re}, nil
443 }
444 if m := reNRe.FindStringSubmatch(expr); m != nil {
445 re, err := regexp.Compile(m[2])
446 if err != nil {
447 return compiledRule{}, i18n.E("ci.err.regex", expr, err.Error())
448 }
449 return compiledRule{kind: "nre", name: m[1], re: re}, nil
450 }
451 return compiledRule{}, i18n.E("ci.err.ruleExpr", expr)
452 }
453
454 func evalRule(expr string, env map[string]string) (bool, error) {
455 c, err := compileRule(expr)
456 if err != nil {
457 return false, err
458 }
459 val := env[c.name]
460 right := c.arg
461 if c.other != "" {
462 right = env[c.other]
463 }
464 if c.null {
465 right = ""
466 }
467 switch c.kind {
468 case "eq":
469 return val == right, nil
470 case "ne":
471 return val != right, nil
472 case "re":
473 return c.re.MatchString(val), nil
474 case "nre":
475 return !c.re.MatchString(val), nil
476 default:
477 return false, i18n.E("ci.err.ruleExpr", expr)
478 }
479 }
480
481 func contains(list []string, s string) bool {
482 for _, item := range list {
483 if item == s {
484 return true
485 }
486 }
487 return false
488 }
@@ -0,0 +1,488 @@
1package ci
2
3import (
4 "fmt"
5 "path"
6 "regexp"
7 "strings"
8
9 "gitlab.com/Timfa/bluegit/internal/i18n"
10)
11
12// Variables zet de voorgedefinieerde CI_-variabelen. Job-specifieke namen
13// vult de runner zelf aan.
14func Variables(r Ref) map[string]string {
15 name := r.Name
16 kind := r.Kind
17 if kind == "" {
18 name, kind = SplitRef(r.FullRef)
19 }
20 short := r.SHA
21 if len(short) > 8 {
22 short = short[:8]
23 }
24 project := r.ProjectPath
25 ns, base := splitProject(project)
26 env := map[string]string{
27 "CI": "true",
28 "CI_CONFIG_PATH": "blue-ci-cd.yml",
29 "CI_SERVER_NAME": "BlueGit",
30 "CI_COMMIT_SHA": r.SHA,
31 "CI_COMMIT_SHORT_SHA": short,
32 "CI_COMMIT_REF": r.FullRef,
33 "CI_COMMIT_REF_NAME": name,
34 "CI_COMMIT_REF_SLUG": slug(name),
35 "CI_COMMIT_TITLE": r.Title,
36 "CI_COMMIT_BRANCH": "",
37 "CI_COMMIT_TAG": "",
38 "CI_DEFAULT_BRANCH": r.DefaultBranch,
39 "CI_PIPELINE_ID": fmt.Sprint(r.PipelineID),
40 "CI_PIPELINE_SOURCE": r.Source,
41 "CI_PIPELINE_URL": strings.TrimRight(r.ProjectURL, "/") + "/-/pipelines/" + fmt.Sprint(r.PipelineID),
42 "CI_PROJECT_PATH": project,
43 "CI_PROJECT_NAME": base,
44 "CI_PROJECT_NAMESPACE": ns,
45 "CI_PROJECT_URL": r.ProjectURL,
46 "CI_API_URL": r.APIURL,
47 }
48 if kind == "tag" {
49 env["CI_COMMIT_TAG"] = name
50 } else {
51 env["CI_COMMIT_BRANCH"] = name
52 }
53 if env["CI_PIPELINE_SOURCE"] == "" {
54 env["CI_PIPELINE_SOURCE"] = "push"
55 }
56 return env
57}
58
59// SplitRef haalt de korte naam en het soort uit refs/heads of refs/tags.
60func SplitRef(ref string) (name, kind string) {
61 switch {
62 case strings.HasPrefix(ref, "refs/tags/"):
63 return strings.TrimPrefix(ref, "refs/tags/"), "tag"
64 case strings.HasPrefix(ref, "refs/heads/"):
65 return strings.TrimPrefix(ref, "refs/heads/"), "branch"
66 default:
67 return ref, "branch"
68 }
69}
70
71func splitProject(path string) (ns, name string) {
72 i := strings.LastIndex(path, "/")
73 if i < 0 {
74 return "", path
75 }
76 return path[:i], path[i+1:]
77}
78
79func slug(s string) string {
80 var b strings.Builder
81 prev := false
82 for _, r := range strings.ToLower(s) {
83 if (r >= 'a' && r <= 'z') || (r >= '0' && r <= '9') {
84 b.WriteRune(r)
85 prev = false
86 continue
87 }
88 if !prev && b.Len() > 0 {
89 b.WriteByte('-')
90 prev = true
91 }
92 }
93 return strings.Trim(b.String(), "-")
94}
95
96// Select houdt de jobs waarvan de rules, only en except kloppen.
97// De When van de job wordt de when van de winnende rule.
98func (p Pipeline) Select(ev Eval) ([]Job, error) {
99 if ev.Env == nil {
100 ev.Env = map[string]string{}
101 }
102 var out []Job
103 active := map[string]bool{}
104 var pending []Job
105 for _, j := range p.Jobs {
106 ok, when, err := j.matches(ev)
107 if err != nil {
108 return nil, fmt.Errorf("job %s: %w", j.Name, err)
109 }
110 if !ok {
111 continue
112 }
113 if when == "" {
114 when = "on_success"
115 }
116 j.When = when
117 pending = append(pending, j)
118 active[j.Name] = true
119 }
120 for i := range pending {
121 var needs []string
122 for _, n := range pending[i].Needs {
123 if active[n] {
124 needs = append(needs, n)
125 continue
126 }
127 if contains(pending[i].Optional, n) {
128 continue
129 }
130 needs = append(needs, n)
131 }
132 if needs == nil {
133 needs = []string{}
134 }
135 pending[i].Needs = needs
136 out = append(out, pending[i])
137 }
138 return out, nil
139}
140
141// ActiveJobs is Select met alleen variabelen. changes en exists gelden dan als waar.
142func (p Pipeline) ActiveJobs(env map[string]string) ([]Job, error) {
143 return p.Select(Eval{Env: env})
144}
145
146func (j Job) matches(ev Eval) (bool, string, error) {
147 when := j.When
148 if when == "" {
149 when = "on_success"
150 }
151 if len(j.Rules) == 0 {
152 ok, err := onlyExcept(j, ev)
153 return ok, when, err
154 }
155 for _, r := range j.Rules {
156 ok, err := ruleMatch(r, ev)
157 if err != nil {
158 return false, "", err
159 }
160 if !ok {
161 continue
162 }
163 w := r.When
164 if w == "" {
165 w = when
166 }
167 if w == "never" {
168 return false, w, nil
169 }
170 return true, w, nil
171 }
172 return false, "", nil
173}
174
175func ruleMatch(r Rule, ev Eval) (bool, error) {
176 if r.If != "" {
177 ok, err := evalRule(r.If, ev.Env)
178 if err != nil || !ok {
179 return ok, err
180 }
181 }
182 if len(r.Changes) > 0 && ev.Changed != nil {
183 if !anyGlob(r.Changes, ev.Changed) {
184 return false, nil
185 }
186 }
187 if len(r.Exists) > 0 && ev.Files != nil {
188 if !allExist(r.Exists, ev.Files) {
189 return false, nil
190 }
191 }
192 if r.If == "" && len(r.Changes) == 0 && len(r.Exists) == 0 && r.When == "" {
193 return false, nil
194 }
195 return true, nil
196}
197
198func onlyExcept(j Job, ev Eval) (bool, error) {
199 if len(j.Only) == 0 && len(j.Except) == 0 {
200 return true, nil
201 }
202 name := ev.Env["CI_COMMIT_REF_NAME"]
203 if name == "" {
204 name = ev.Env["CI_COMMIT_BRANCH"]
205 }
206 tag := ev.Env["CI_COMMIT_TAG"] != ""
207 if len(j.Only) > 0 {
208 ok, err := refListed(j.Only, name, tag)
209 if err != nil || !ok {
210 return false, err
211 }
212 }
213 if len(j.Except) > 0 {
214 ok, err := refListed(j.Except, name, tag)
215 if err != nil {
216 return false, err
217 }
218 if ok {
219 return false, nil
220 }
221 }
222 return true, nil
223}
224
225func refListed(list []string, name string, tag bool) (bool, error) {
226 for _, item := range list {
227 switch item {
228 case "branches":
229 if !tag {
230 return true, nil
231 }
232 case "tags":
233 if tag {
234 return true, nil
235 }
236 default:
237 if strings.HasPrefix(item, "/") && strings.HasSuffix(item, "/") && len(item) > 2 {
238 re, err := regexp.Compile(item[1 : len(item)-1])
239 if err != nil {
240 return false, i18n.E("ci.err.regex", item, err.Error())
241 }
242 if re.MatchString(name) {
243 return true, nil
244 }
245 continue
246 }
247 if item == name {
248 return true, nil
249 }
250 }
251 }
252 return false, nil
253}
254
255func anyGlob(patterns, names []string) bool {
256 for _, name := range names {
257 for _, pattern := range patterns {
258 if Glob(pattern, name) {
259 return true
260 }
261 }
262 }
263 return false
264}
265
266func allExist(patterns, files []string) bool {
267 for _, pattern := range patterns {
268 found := false
269 for _, file := range files {
270 if file == pattern || Glob(pattern, file) {
271 found = true
272 break
273 }
274 }
275 if !found {
276 return false
277 }
278 }
279 return true
280}
281
282// Glob volgt GitLab-padpatronen: * blijft in één segment, ** loopt door mappen.
283func Glob(pattern, name string) bool {
284 pattern = strings.TrimPrefix(pattern, "/")
285 name = strings.TrimPrefix(name, "/")
286 if pattern == name {
287 return true
288 }
289 re := globRegexp(pattern)
290 if re.MatchString(name) {
291 return true
292 }
293 if !strings.Contains(pattern, "/") && re.MatchString(path.Base(name)) {
294 return true
295 }
296 return false
297}
298
299func globRegexp(pattern string) *regexp.Regexp {
300 var b strings.Builder
301 b.WriteString("^")
302 for i := 0; i < len(pattern); i++ {
303 if pattern[i] == '*' && i+1 < len(pattern) && pattern[i+1] == '*' {
304 b.WriteString(".*")
305 i++
306 if i+1 < len(pattern) && pattern[i+1] == '/' {
307 i++
308 }
309 continue
310 }
311 switch pattern[i] {
312 case '*':
313 b.WriteString("[^/]*")
314 case '?':
315 b.WriteString("[^/]")
316 default:
317 b.WriteString(regexp.QuoteMeta(pattern[i : i+1]))
318 }
319 }
320 b.WriteString("$")
321 return regexp.MustCompile(b.String())
322}
323
324// Gate beslist of een job mag starten.
325// "ready" betekent dat alle voorgangers klaar zijn.
326// "wait" betekent dat er nog een voorganger loopt of in de wachtrij staat.
327// "skip" betekent dat een voorganger mislukte zonder allow_failure,
328// of dat when: on_failure geen mislukking zag.
329// "fail" betekent dat een verplichte need niet in deze pipeline zit.
330//
331// NeedsSet volgt de genoemde jobs, ook als de lijst leeg is.
332// Zonder needs wacht de job op alle jobs in eerdere stages.
333func Gate(jobs []Job, status map[string]string, job Job) string {
334 byName := map[string]Job{}
335 for _, j := range jobs {
336 byName[j.Name] = j
337 }
338 optional := map[string]bool{}
339 for _, n := range job.Optional {
340 optional[n] = true
341 }
342 var deps []Job
343 if job.NeedsSet {
344 for _, n := range job.Needs {
345 d, ok := byName[n]
346 if !ok {
347 if optional[n] {
348 continue
349 }
350 return "fail"
351 }
352 deps = append(deps, d)
353 }
354 } else {
355 stagePos := map[string]int{}
356 order := 0
357 for _, j := range jobs {
358 if _, ok := stagePos[j.Stage]; !ok {
359 stagePos[j.Stage] = order
360 order++
361 }
362 }
363 my := stagePos[job.Stage]
364 for _, j := range jobs {
365 if stagePos[j.Stage] < my {
366 deps = append(deps, j)
367 }
368 }
369 }
370 failedAny := false
371 failedHard := false
372 for _, d := range deps {
373 switch status[d.Name] {
374 case "success", "skipped":
375 continue
376 case "failed":
377 failedAny = true
378 if !d.AllowFailure {
379 failedHard = true
380 }
381 case "manual":
382 if d.AllowFailure {
383 continue
384 }
385 return "wait"
386 default:
387 return "wait"
388 }
389 }
390 when := job.When
391 if when == "" {
392 when = "on_success"
393 }
394 switch when {
395 case "always":
396 return "ready"
397 case "on_failure":
398 if failedAny {
399 return "ready"
400 }
401 return "skip"
402 default:
403 if failedHard {
404 return "skip"
405 }
406 return "ready"
407 }
408}
409
410var (
411 reEq = regexp.MustCompile(`^\$([A-Za-z_][A-Za-z0-9_]*)\s*==\s*(?:"([^"]*)"|\$([A-Za-z_][A-Za-z0-9_]*)|null)$`)
412 reNe = regexp.MustCompile(`^\$([A-Za-z_][A-Za-z0-9_]*)\s*!=\s*(?:"([^"]*)"|\$([A-Za-z_][A-Za-z0-9_]*)|null)$`)
413 reRe = regexp.MustCompile(`^\$([A-Za-z_][A-Za-z0-9_]*)\s*=~\s*/(.*)/$`)
414 reNRe = regexp.MustCompile(`^\$([A-Za-z_][A-Za-z0-9_]*)\s*!~\s*/(.*)/$`)
415)
416
417type compiledRule struct {
418 kind string
419 name string
420 arg string
421 other string
422 null bool
423 re *regexp.Regexp
424}
425
426func compileRule(expr string) (compiledRule, error) {
427 expr = strings.TrimSpace(expr)
428 if strings.Contains(expr, "&&") || strings.Contains(expr, "||") {
429 return compiledRule{}, i18n.E("ci.err.ruleCombine", expr)
430 }
431 if m := reEq.FindStringSubmatch(expr); m != nil {
432 return compiledRule{kind: "eq", name: m[1], arg: m[2], other: m[3], null: m[2] == "" && m[3] == "" && strings.HasSuffix(strings.TrimSpace(expr), "null")}, nil
433 }
434 if m := reNe.FindStringSubmatch(expr); m != nil {
435 return compiledRule{kind: "ne", name: m[1], arg: m[2], other: m[3], null: m[2] == "" && m[3] == "" && strings.HasSuffix(strings.TrimSpace(expr), "null")}, nil
436 }
437 if m := reRe.FindStringSubmatch(expr); m != nil {
438 re, err := regexp.Compile(m[2])
439 if err != nil {
440 return compiledRule{}, i18n.E("ci.err.regex", expr, err.Error())
441 }
442 return compiledRule{kind: "re", name: m[1], re: re}, nil
443 }
444 if m := reNRe.FindStringSubmatch(expr); m != nil {
445 re, err := regexp.Compile(m[2])
446 if err != nil {
447 return compiledRule{}, i18n.E("ci.err.regex", expr, err.Error())
448 }
449 return compiledRule{kind: "nre", name: m[1], re: re}, nil
450 }
451 return compiledRule{}, i18n.E("ci.err.ruleExpr", expr)
452}
453
454func evalRule(expr string, env map[string]string) (bool, error) {
455 c, err := compileRule(expr)
456 if err != nil {
457 return false, err
458 }
459 val := env[c.name]
460 right := c.arg
461 if c.other != "" {
462 right = env[c.other]
463 }
464 if c.null {
465 right = ""
466 }
467 switch c.kind {
468 case "eq":
469 return val == right, nil
470 case "ne":
471 return val != right, nil
472 case "re":
473 return c.re.MatchString(val), nil
474 case "nre":
475 return !c.re.MatchString(val), nil
476 default:
477 return false, i18n.E("ci.err.ruleExpr", expr)
478 }
479}
480
481func contains(list []string, s string) bool {
482 for _, item := range list {
483 if item == s {
484 return true
485 }
486 }
487 return false
488}
internal/ci/mask.go added +26 −0
@@ -0,0 +1,26 @@
1 package ci
2
3 import "strings"
4
5 // Mask vervangt geheime waarden door [MASKED].
6 // Alleen waarden van minstens 8 tekens, zodat korte variabelen de log niet uitwissen.
7 func Mask(text string, secrets []string) string {
8 if text == "" || len(secrets) == 0 {
9 return text
10 }
11 ordered := append([]string(nil), secrets...)
12 for i := 0; i < len(ordered); i++ {
13 for j := i + 1; j < len(ordered); j++ {
14 if len(ordered[j]) > len(ordered[i]) {
15 ordered[i], ordered[j] = ordered[j], ordered[i]
16 }
17 }
18 }
19 for _, secret := range ordered {
20 if len(secret) < 8 || strings.TrimSpace(secret) == "" {
21 continue
22 }
23 text = strings.ReplaceAll(text, secret, "[MASKED]")
24 }
25 return text
26 }
@@ -0,0 +1,26 @@
1package ci
2
3import "strings"
4
5// Mask vervangt geheime waarden door [MASKED].
6// Alleen waarden van minstens 8 tekens, zodat korte variabelen de log niet uitwissen.
7func Mask(text string, secrets []string) string {
8 if text == "" || len(secrets) == 0 {
9 return text
10 }
11 ordered := append([]string(nil), secrets...)
12 for i := 0; i < len(ordered); i++ {
13 for j := i + 1; j < len(ordered); j++ {
14 if len(ordered[j]) > len(ordered[i]) {
15 ordered[i], ordered[j] = ordered[j], ordered[i]
16 }
17 }
18 }
19 for _, secret := range ordered {
20 if len(secret) < 8 || strings.TrimSpace(secret) == "" {
21 continue
22 }
23 text = strings.ReplaceAll(text, secret, "[MASKED]")
24 }
25 return text
26}
internal/ci/model.go added +94 −0
@@ -0,0 +1,94 @@
1 package ci
2
3 import "time"
4
5 // Pipeline is een geparset blue-ci-cd.yml.
6 type Pipeline struct {
7 Stages []string
8 Variables map[string]string
9 Image string
10 Before []string
11 After []string
12 Cache Cache
13 Timeout time.Duration
14 Tags []string
15 Jobs []Job
16 Warnings []Issue
17 }
18
19 // Job is één containerstap.
20 type Job struct {
21 Name string
22 Stage string
23 Image string
24 Before []string
25 Script []string
26 After []string
27 Needs []string
28 Optional []string
29 NeedsSet bool
30 Variables map[string]string
31 Artifacts Artifacts
32 Rules []Rule
33 AllowFailure bool
34 When string
35 Timeout time.Duration
36 Tags []string
37 Cache Cache
38 ResourceGroup string
39 Only []string
40 Except []string
41 }
42
43 // Artifacts zijn paden die na een geslaagde job bewaard blijven.
44 type Artifacts struct {
45 Paths []string
46 ExpireIn time.Duration
47 }
48
49 // Cache is een map die tussen jobs van één repository gedeeld wordt.
50 type Cache struct {
51 Key string
52 Paths []string
53 }
54
55 // Rule is één voorwaarde. De eerste rule die klopt wint.
56 // Zonder rules draait de job, tenzij only of except dat tegenhoudt.
57 type Rule struct {
58 If string
59 Changes []string
60 Exists []string
61 When string
62 }
63
64 // Issue is een fout of een waarschuwing met een regelnummer.
65 type Issue struct {
66 Line int
67 Key string
68 Args []any
69 Warning bool
70 }
71
72 // Eval is de context waarin rules worden beoordeeld.
73 // Changed == nil betekent dat de diff onbekend is; changes klopt dan.
74 // Files == nil betekent dat de boom onbekend is; exists klopt dan.
75 type Eval struct {
76 Env map[string]string
77 Changed []string
78 Files []string
79 }
80
81 // Ref beschrijft de commit waarvoor voorgedefinieerde variabelen gelden.
82 type Ref struct {
83 SHA string
84 FullRef string
85 Name string
86 Kind string
87 Title string
88 DefaultBranch string
89 Source string
90 ProjectPath string
91 ProjectURL string
92 APIURL string
93 PipelineID int64
94 }
@@ -0,0 +1,94 @@
1package ci
2
3import "time"
4
5// Pipeline is een geparset blue-ci-cd.yml.
6type Pipeline struct {
7 Stages []string
8 Variables map[string]string
9 Image string
10 Before []string
11 After []string
12 Cache Cache
13 Timeout time.Duration
14 Tags []string
15 Jobs []Job
16 Warnings []Issue
17}
18
19// Job is één containerstap.
20type Job struct {
21 Name string
22 Stage string
23 Image string
24 Before []string
25 Script []string
26 After []string
27 Needs []string
28 Optional []string
29 NeedsSet bool
30 Variables map[string]string
31 Artifacts Artifacts
32 Rules []Rule
33 AllowFailure bool
34 When string
35 Timeout time.Duration
36 Tags []string
37 Cache Cache
38 ResourceGroup string
39 Only []string
40 Except []string
41}
42
43// Artifacts zijn paden die na een geslaagde job bewaard blijven.
44type Artifacts struct {
45 Paths []string
46 ExpireIn time.Duration
47}
48
49// Cache is een map die tussen jobs van één repository gedeeld wordt.
50type Cache struct {
51 Key string
52 Paths []string
53}
54
55// Rule is één voorwaarde. De eerste rule die klopt wint.
56// Zonder rules draait de job, tenzij only of except dat tegenhoudt.
57type Rule struct {
58 If string
59 Changes []string
60 Exists []string
61 When string
62}
63
64// Issue is een fout of een waarschuwing met een regelnummer.
65type Issue struct {
66 Line int
67 Key string
68 Args []any
69 Warning bool
70}
71
72// Eval is de context waarin rules worden beoordeeld.
73// Changed == nil betekent dat de diff onbekend is; changes klopt dan.
74// Files == nil betekent dat de boom onbekend is; exists klopt dan.
75type Eval struct {
76 Env map[string]string
77 Changed []string
78 Files []string
79}
80
81// Ref beschrijft de commit waarvoor voorgedefinieerde variabelen gelden.
82type Ref struct {
83 SHA string
84 FullRef string
85 Name string
86 Kind string
87 Title string
88 DefaultBranch string
89 Source string
90 ProjectPath string
91 ProjectURL string
92 APIURL string
93 PipelineID int64
94}
internal/ci/parse.go +865 −335
@@ -1,327 +1,712 @@
1 1 // Package ci leest blue-ci-cd.yml en bepaalt welke jobs draaien.
2 // De vorm ligt dicht bij GitLab CI: stages, jobs, needs, rules, artifacts.
2 // De vorm volgt GitLab CI waar dat in BlueGit past. Een oud bestand met
3 // een jobs:-map blijft geldig. Jobs mogen ook los op het hoogste niveau staan.
3 4 package ci
4 5
5 6 import (
6 7 "fmt"
7 "regexp"
8 8 "strconv"
9 9 "strings"
10 10 "time"
11 11
12 "gitlab.com/Timfa/bluegit/internal/i18n"
12 13 "gopkg.in/yaml.v3"
13 14 )
14 15
15 // Pipeline is een geparset configuratiebestand.
16 type Pipeline struct {
17 Stages []string
18 Variables map[string]string
19 Image string
20 Jobs []Job
16 // Parse leest YAML. Jobvolgorde blijft de volgorde uit het bestand.
17 func Parse(data []byte) (Pipeline, error) {
18 return ParseIn(data, nil)
19 }
20
21 // ParseIn leest YAML en haalt include: local via read op.
22 func ParseIn(data []byte, read func(string) ([]byte, error)) (Pipeline, error) {
23 p, issues, err := parseBytes(data, read, nil)
24 if err != nil {
25 return Pipeline{}, err
26 }
27 for _, is := range issues {
28 if !is.Warning {
29 return Pipeline{}, is
30 }
31 }
32 p.Warnings = warningsOnly(issues)
33 return p, nil
21 34 }
22 35
23 // Job is één containerstap.
24 type Job struct {
25 Name string
26 Stage string
27 Image string
28 Script []string
29 Needs []string
30 NeedsSet bool
31 Variables map[string]string
32 Artifacts Artifacts
33 Rules []Rule
34 AllowFailure bool
36 // Lint geeft elke fout en waarschuwing, met regelnummer, zonder te stoppen bij de eerste.
37 func Lint(data []byte, read func(string) ([]byte, error)) (Pipeline, []Issue) {
38 p, issues, err := parseBytes(data, read, nil)
39 if err != nil {
40 issues = append([]Issue{{Line: 1, Key: "ci.err.yaml", Args: []any{err.Error()}}}, issues...)
41 }
42 return p, issues
43 }
44
45 func (i Issue) Error() string { return i18n.T(i18n.EN, i.Key, i.Args...) }
46
47 func warningsOnly(in []Issue) []Issue {
48 var out []Issue
49 for _, is := range in {
50 if is.Warning {
51 out = append(out, is)
52 }
53 }
54 return out
35 55 }
36 56
37 // Artifacts zijn paden die na een geslaagde job bewaard blijven.
38 type Artifacts struct {
39 Paths []string
40 ExpireIn time.Duration
57 var reserved = map[string]bool{
58 "stages": true, "variables": true, "image": true, "default": true,
59 "include": true, "before_script": true, "after_script": true, "cache": true,
60 "workflow": true, "services": true, "spec": true, "pages": true, "jobs": true,
41 61 }
42 62
43 // Rule is één voorwaarde. De job draait als minstens één rule klopt.
44 // Zonder rules draait de job altijd.
45 type Rule struct {
46 If string
63 type draft struct {
64 job Job
65 present map[string]bool
66 extends []string
67 line int
47 68 }
48 69
49 // Parse leest YAML. Jobvolgorde blijft de volgorde uit het bestand.
50 func Parse(data []byte) (Pipeline, error) {
70 func parseBytes(data []byte, read func(string) ([]byte, error), stack []string) (Pipeline, []Issue, error) {
51 71 var root yaml.Node
52 72 if err := yaml.Unmarshal(data, &root); err != nil {
53 return Pipeline{}, fmt.Errorf("yaml: %w", err)
73 return Pipeline{}, nil, i18n.E("ci.err.yaml", err.Error())
54 74 }
55 75 if root.Kind != yaml.DocumentNode || len(root.Content) != 1 || root.Content[0].Kind != yaml.MappingNode {
56 return Pipeline{}, fmt.Errorf("blue-ci-cd.yml moet een mapping zijn")
76 return Pipeline{}, []Issue{{Line: 1, Key: "ci.err.mapping"}}, nil
57 77 }
58 78 body := root.Content[0]
59 var p Pipeline
60 p.Variables = map[string]string{}
79 var included Pipeline
80 var issues []Issue
81 for i := 0; i < len(body.Content); i += 2 {
82 if body.Content[i].Value != "include" {
83 continue
84 }
85 part, more, err := parseInclude(body.Content[i+1], read, stack)
86 if err != nil {
87 return Pipeline{}, nil, err
88 }
89 issues = append(issues, more...)
90 included = mergePipeline(included, part)
91 }
92 p := included
93 if p.Variables == nil {
94 p.Variables = map[string]string{}
95 }
96 var def draft
97 def.present = map[string]bool{}
98 var jobs []draft
99 seenLegacy := false
100 seenTop := false
61 101 for i := 0; i < len(body.Content); i += 2 {
62 key := body.Content[i].Value
102 keyNode := body.Content[i]
103 key := keyNode.Value
63 104 val := body.Content[i+1]
64 105 switch key {
106 case "include":
107 continue
65 108 case "stages":
66 stages, err := scalarList(val)
67 if err != nil {
68 return Pipeline{}, fmt.Errorf("stages: %w", err)
109 stages, iss := scalarList(val, "ci.err.list")
110 issues = append(issues, iss...)
111 if len(stages) > 0 {
112 p.Stages = stages
69 113 }
70 p.Stages = stages
71 114 case "variables":
72 vars, err := scalarMap(val)
73 if err != nil {
74 return Pipeline{}, fmt.Errorf("variables: %w", err)
115 vars, iss := scalarMap(val)
116 issues = append(issues, iss...)
117 for k, v := range vars {
118 p.Variables[k] = v
75 119 }
76 p.Variables = vars
77 120 case "image":
78 if val.Kind != yaml.ScalarNode {
79 return Pipeline{}, fmt.Errorf("image moet een tekst zijn")
121 img, iss := imageOf(val)
122 issues = append(issues, iss...)
123 if img != "" {
124 p.Image = img
80 125 }
81 p.Image = val.Value
126 case "before_script":
127 lines, iss := scriptList(val)
128 issues = append(issues, iss...)
129 p.Before = lines
130 def.present["before_script"] = true
131 def.job.Before = lines
132 case "after_script":
133 lines, iss := scriptList(val)
134 issues = append(issues, iss...)
135 p.After = lines
136 def.present["after_script"] = true
137 def.job.After = lines
138 case "cache":
139 c, iss := parseCache(val)
140 issues = append(issues, iss...)
141 p.Cache = c
142 def.present["cache"] = true
143 def.job.Cache = c
144 case "default":
145 d, iss := parseDefault(val)
146 issues = append(issues, iss...)
147 def = overlayDefault(def, d)
148 case "workflow", "services", "spec", "pages":
149 issues = append(issues, Issue{Line: keyNode.Line, Key: "ci.warn.ignored", Args: []any{key}, Warning: true})
82 150 case "jobs":
83 jobs, err := parseJobs(val)
84 if err != nil {
85 return Pipeline{}, err
86 }
87 p.Jobs = jobs
151 seenLegacy = true
152 parsed, iss := parseJobMap(val)
153 issues = append(issues, iss...)
154 jobs = append(jobs, parsed...)
88 155 default:
89 return Pipeline{}, fmt.Errorf("onbekend veld %q", key)
156 if val.Kind != yaml.MappingNode {
157 issues = append(issues, Issue{Line: keyNode.Line, Key: "ci.err.unknown", Args: []any{key}})
158 continue
159 }
160 seenTop = true
161 job, iss := parseJob(key, val)
162 issues = append(issues, iss...)
163 jobs = append(jobs, job)
90 164 }
91 165 }
92 if err := p.validate(); err != nil {
93 return Pipeline{}, err
166 if seenLegacy && seenTop {
167 issues = append(issues, Issue{Line: 1, Key: "ci.err.both"})
168 }
169 if p.Image != "" && !def.present["image"] {
170 def.present["image"] = true
171 def.job.Image = p.Image
172 }
173 resolved, iss := applyExtends(jobs, def)
174 issues = append(issues, iss...)
175 byName := map[string]int{}
176 var merged []Job
177 for _, j := range p.Jobs {
178 byName[j.Name] = len(merged)
179 merged = append(merged, j)
180 }
181 for _, d := range resolved {
182 if strings.HasPrefix(d.job.Name, ".") {
183 continue
184 }
185 if i, ok := byName[d.job.Name]; ok {
186 merged[i] = overlayJob(merged[i], d)
187 continue
188 }
189 byName[d.job.Name] = len(merged)
190 merged = append(merged, d.job)
191 }
192 p.Jobs = merged
193 issues = append(issues, validate(&p)...)
194 if hasError(issues) {
195 return p, issues, nil
196 }
197 return p, issues, nil
198 }
199
200 func parseInclude(node *yaml.Node, read func(string) ([]byte, error), stack []string) (Pipeline, []Issue, error) {
201 var items []*yaml.Node
202 switch node.Kind {
203 case yaml.SequenceNode:
204 items = node.Content
205 default:
206 items = []*yaml.Node{node}
207 }
208 var out Pipeline
209 var issues []Issue
210 for _, item := range items {
211 local, iss := includeLocal(item)
212 issues = append(issues, iss...)
213 if local == "" {
214 continue
215 }
216 if read == nil {
217 issues = append(issues, Issue{Line: item.Line, Key: "ci.err.includeRead", Args: []any{local}})
218 continue
219 }
220 for _, prev := range stack {
221 if prev == local {
222 issues = append(issues, Issue{Line: item.Line, Key: "ci.err.includeCycle", Args: []any{local}})
223 local = ""
224 break
225 }
226 }
227 if local == "" {
228 continue
229 }
230 raw, err := read(local)
231 if err != nil {
232 issues = append(issues, Issue{Line: item.Line, Key: "ci.err.includeRead", Args: []any{local}})
233 continue
234 }
235 part, more, err := parseBytes(raw, read, append(append([]string{}, stack...), local))
236 if err != nil {
237 return Pipeline{}, nil, err
238 }
239 issues = append(issues, more...)
240 out = mergePipeline(out, part)
241 }
242 return out, issues, nil
243 }
244
245 func includeLocal(node *yaml.Node) (string, []Issue) {
246 switch node.Kind {
247 case yaml.ScalarNode:
248 return cleanLocal(node.Value, node)
249 case yaml.MappingNode:
250 for i := 0; i < len(node.Content); i += 2 {
251 key := node.Content[i].Value
252 if key == "local" {
253 if node.Content[i+1].Kind != yaml.ScalarNode {
254 return "", []Issue{{Line: node.Line, Key: "ci.err.includeLocal"}}
255 }
256 return cleanLocal(node.Content[i+1].Value, node.Content[i+1])
257 }
258 }
259 return "", []Issue{{Line: node.Line, Key: "ci.err.includeRemote"}}
260 default:
261 return "", []Issue{{Line: node.Line, Key: "ci.err.includeLocal"}}
262 }
263 }
264
265 func cleanLocal(p string, node *yaml.Node) (string, []Issue) {
266 p = strings.TrimPrefix(strings.TrimSpace(p), "/")
267 if p == "" || strings.Contains(p, "\\") || strings.Contains(p, "..") {
268 return "", []Issue{{Line: node.Line, Key: "ci.err.rel", Args: []any{p}}}
94 269 }
95 270 return p, nil
96 271 }
97 272
98 func parseJobs(node *yaml.Node) ([]Job, error) {
273 func mergePipeline(base, extra Pipeline) Pipeline {
274 if len(extra.Stages) > 0 {
275 base.Stages = extra.Stages
276 }
277 if base.Variables == nil {
278 base.Variables = map[string]string{}
279 }
280 for k, v := range extra.Variables {
281 base.Variables[k] = v
282 }
283 if extra.Image != "" {
284 base.Image = extra.Image
285 }
286 if extra.Before != nil {
287 base.Before = extra.Before
288 }
289 if extra.After != nil {
290 base.After = extra.After
291 }
292 byName := map[string]int{}
293 for i, j := range base.Jobs {
294 byName[j.Name] = i
295 }
296 for _, j := range extra.Jobs {
297 if i, ok := byName[j.Name]; ok {
298 base.Jobs[i] = j
299 continue
300 }
301 byName[j.Name] = len(base.Jobs)
302 base.Jobs = append(base.Jobs, j)
303 }
304 return base
305 }
306
307 func parseDefault(node *yaml.Node) (draft, []Issue) {
308 d := draft{present: map[string]bool{}}
99 309 if node.Kind != yaml.MappingNode {
100 return nil, fmt.Errorf("jobs moet een mapping zijn")
310 return d, []Issue{{Line: node.Line, Key: "ci.err.map"}}
101 311 }
102 var jobs []Job
312 var issues []Issue
103 313 for i := 0; i < len(node.Content); i += 2 {
104 name := node.Content[i].Value
105 body := node.Content[i+1]
106 job, err := parseJob(name, body)
107 if err != nil {
108 return nil, fmt.Errorf("job %s: %w", name, err)
314 key := node.Content[i].Value
315 val := node.Content[i+1]
316 switch key {
317 case "image":
318 img, iss := imageOf(val)
319 issues = append(issues, iss...)
320 d.job.Image = img
321 d.present["image"] = true
322 case "before_script":
323 lines, iss := scriptList(val)
324 issues = append(issues, iss...)
325 d.job.Before = lines
326 d.present["before_script"] = true
327 case "after_script":
328 lines, iss := scriptList(val)
329 issues = append(issues, iss...)
330 d.job.After = lines
331 d.present["after_script"] = true
332 case "tags":
333 tags, iss := scalarList(val, "ci.err.list")
334 issues = append(issues, iss...)
335 d.job.Tags = tags
336 d.present["tags"] = true
337 case "timeout":
338 dur, iss := durationOf(val)
339 issues = append(issues, iss...)
340 d.job.Timeout = dur
341 d.present["timeout"] = true
342 case "cache":
343 c, iss := parseCache(val)
344 issues = append(issues, iss...)
345 d.job.Cache = c
346 d.present["cache"] = true
347 default:
348 issues = append(issues, Issue{Line: node.Content[i].Line, Key: "ci.warn.ignored", Args: []any{"default." + key}, Warning: true})
109 349 }
110 jobs = append(jobs, job)
111 350 }
112 return jobs, nil
351 return d, issues
113 352 }
114 353
115 func parseJob(name string, node *yaml.Node) (Job, error) {
354 func overlayJob(base Job, d draft) Job {
355 set := func(key string, apply func()) {
356 if d.present[key] {
357 apply()
358 }
359 }
360 set("stage", func() { base.Stage = d.job.Stage })
361 set("image", func() { base.Image = d.job.Image })
362 set("before_script", func() { base.Before = d.job.Before })
363 set("script", func() { base.Script = d.job.Script })
364 set("after_script", func() { base.After = d.job.After })
365 set("needs", func() {
366 base.Needs = d.job.Needs
367 base.Optional = d.job.Optional
368 base.NeedsSet = d.job.NeedsSet
369 })
370 set("artifacts", func() { base.Artifacts = d.job.Artifacts })
371 set("rules", func() { base.Rules = d.job.Rules })
372 set("allow_failure", func() { base.AllowFailure = d.job.AllowFailure })
373 set("when", func() { base.When = d.job.When })
374 set("timeout", func() { base.Timeout = d.job.Timeout })
375 set("tags", func() { base.Tags = d.job.Tags })
376 set("cache", func() { base.Cache = d.job.Cache })
377 set("resource_group", func() { base.ResourceGroup = d.job.ResourceGroup })
378 set("only", func() { base.Only = d.job.Only })
379 set("except", func() { base.Except = d.job.Except })
380 if d.present["variables"] {
381 mergedVars := map[string]string{}
382 for k, v := range base.Variables {
383 mergedVars[k] = v
384 }
385 for k, v := range d.job.Variables {
386 mergedVars[k] = v
387 }
388 base.Variables = mergedVars
389 }
390 return base
391 }
392
393 func overlayDefault(base, extra draft) draft {
394 if base.present == nil {
395 base.present = map[string]bool{}
396 }
397 for k, on := range extra.present {
398 if !on {
399 continue
400 }
401 base.present[k] = true
402 switch k {
403 case "image":
404 base.job.Image = extra.job.Image
405 case "before_script":
406 base.job.Before = extra.job.Before
407 case "after_script":
408 base.job.After = extra.job.After
409 case "tags":
410 base.job.Tags = extra.job.Tags
411 case "timeout":
412 base.job.Timeout = extra.job.Timeout
413 case "cache":
414 base.job.Cache = extra.job.Cache
415 }
416 }
417 return base
418 }
419
420 func parseJobMap(node *yaml.Node) ([]draft, []Issue) {
116 421 if node.Kind != yaml.MappingNode {
117 return Job{}, fmt.Errorf("moet een mapping zijn")
422 return nil, []Issue{{Line: node.Line, Key: "ci.err.jobsMap"}}
118 423 }
119 j := Job{Name: name, Variables: map[string]string{}}
424 var jobs []draft
425 var issues []Issue
426 for i := 0; i < len(node.Content); i += 2 {
427 job, iss := parseJob(node.Content[i].Value, node.Content[i+1])
428 issues = append(issues, iss...)
429 if job.job.Name != "" {
430 jobs = append(jobs, job)
431 }
432 }
433 return jobs, issues
434 }
435
436 func parseJob(name string, node *yaml.Node) (draft, []Issue) {
437 d := draft{present: map[string]bool{}, line: node.Line, job: Job{Name: name, Variables: map[string]string{}}}
438 if node.Kind != yaml.MappingNode {
439 return d, []Issue{{Line: node.Line, Key: "ci.err.jobMap", Args: []any{name}}}
440 }
441 var issues []Issue
120 442 for i := 0; i < len(node.Content); i += 2 {
121 443 key := node.Content[i].Value
122 444 val := node.Content[i+1]
445 d.present[key] = true
123 446 switch key {
124 447 case "stage":
125 448 if val.Kind != yaml.ScalarNode {
126 return Job{}, fmt.Errorf("stage moet tekst zijn")
449 issues = append(issues, Issue{Line: val.Line, Key: "ci.err.stageText"})
450 continue
127 451 }
128 j.Stage = val.Value
452 d.job.Stage = val.Value
129 453 case "image":
130 if val.Kind != yaml.ScalarNode {
131 return Job{}, fmt.Errorf("image moet tekst zijn")
132 }
133 j.Image = val.Value
454 img, iss := imageOf(val)
455 issues = append(issues, iss...)
456 d.job.Image = img
457 case "before_script":
458 lines, iss := scriptList(val)
459 issues = append(issues, iss...)
460 d.job.Before = lines
134 461 case "script":
135 lines, err := scalarList(val)
136 if err != nil {
137 return Job{}, fmt.Errorf("script: %w", err)
138 }
139 j.Script = lines
462 lines, iss := scriptList(val)
463 issues = append(issues, iss...)
464 d.job.Script = lines
465 case "after_script":
466 lines, iss := scriptList(val)
467 issues = append(issues, iss...)
468 d.job.After = lines
140 469 case "needs":
141 needs, err := scalarList(val)
142 if err != nil {
143 return Job{}, fmt.Errorf("needs: %w", err)
144 }
145 j.Needs = needs
146 j.NeedsSet = true
470 needs, optional, set, iss := parseNeeds(val)
471 issues = append(issues, iss...)
472 d.job.Needs = needs
473 d.job.Optional = optional
474 d.job.NeedsSet = set
147 475 case "variables":
148 vars, err := scalarMap(val)
149 if err != nil {
150 return Job{}, fmt.Errorf("variables: %w", err)
151 }
152 j.Variables = vars
476 vars, iss := scalarMap(val)
477 issues = append(issues, iss...)
478 d.job.Variables = vars
153 479 case "artifacts":
154 art, err := parseArtifacts(val)
155 if err != nil {
156 return Job{}, err
157 }
158 j.Artifacts = art
480 art, iss := parseArtifacts(val)
481 issues = append(issues, iss...)
482 d.job.Artifacts = art
159 483 case "rules":
160 rules, err := parseRules(val)
161 if err != nil {
162 return Job{}, err
163 }
164 j.Rules = rules
484 rules, iss := parseRules(val)
485 issues = append(issues, iss...)
486 d.job.Rules = rules
165 487 case "allow_failure":
166 b, err := parseBool(val)
167 if err != nil {
168 return Job{}, fmt.Errorf("allow_failure: %w", err)
488 b, iss := parseBool(val)
489 issues = append(issues, iss...)
490 d.job.AllowFailure = b
491 case "when":
492 w, iss := whenOf(val, false)
493 issues = append(issues, iss...)
494 d.job.When = w
495 case "timeout":
496 dur, iss := durationOf(val)
497 issues = append(issues, iss...)
498 d.job.Timeout = dur
499 case "tags":
500 tags, iss := scalarList(val, "ci.err.list")
501 issues = append(issues, iss...)
502 d.job.Tags = tags
503 case "cache":
504 c, iss := parseCache(val)
505 issues = append(issues, iss...)
506 d.job.Cache = c
507 case "resource_group":
508 if val.Kind != yaml.ScalarNode || strings.TrimSpace(val.Value) == "" {
509 issues = append(issues, Issue{Line: val.Line, Key: "ci.err.resource"})
510 continue
169 511 }
170 j.AllowFailure = b
512 d.job.ResourceGroup = strings.TrimSpace(val.Value)
513 case "only":
514 list, iss := scalarList(val, "ci.err.list")
515 issues = append(issues, iss...)
516 d.job.Only = list
517 case "except":
518 list, iss := scalarList(val, "ci.err.list")
519 issues = append(issues, iss...)
520 d.job.Except = list
521 case "extends":
522 list, iss := extendsOf(val)
523 issues = append(issues, iss...)
524 d.extends = list
171 525 default:
172 return Job{}, fmt.Errorf("onbekend veld %q", key)
526 if _, ok := reserved[key]; ok {
527 issues = append(issues, Issue{Line: node.Content[i].Line, Key: "ci.warn.ignored", Args: []any{key}, Warning: true})
528 continue
529 }
530 issues = append(issues, Issue{Line: node.Content[i].Line, Key: "ci.warn.ignored", Args: []any{name + "." + key}, Warning: true})
173 531 }
174 532 }
175 return j, nil
533 return d, issues
176 534 }
177 535
178 func parseArtifacts(node *yaml.Node) (Artifacts, error) {
179 if node.Kind != yaml.MappingNode {
180 return Artifacts{}, fmt.Errorf("artifacts moet een mapping zijn")
536 func applyExtends(jobs []draft, def draft) ([]draft, []Issue) {
537 by := map[string]*draft{}
538 var order []string
539 var issues []Issue
540 for i := range jobs {
541 name := jobs[i].job.Name
542 if by[name] != nil {
543 issues = append(issues, Issue{Line: jobs[i].line, Key: "ci.err.jobDup", Args: []any{name}})
544 continue
545 }
546 by[name] = &jobs[i]
547 order = append(order, name)
181 548 }
182 var a Artifacts
183 for i := 0; i < len(node.Content); i += 2 {
184 key := node.Content[i].Value
185 val := node.Content[i+1]
186 switch key {
187 case "paths":
188 paths, err := scalarList(val)
189 if err != nil {
190 return Artifacts{}, fmt.Errorf("artifacts.paths: %w", err)
191 }
192 a.Paths = paths
193 case "expire_in":
194 if val.Kind != yaml.ScalarNode {
195 return Artifacts{}, fmt.Errorf("expire_in moet tekst zijn")
196 }
197 d, err := parseExpire(val.Value)
198 if err != nil {
199 return Artifacts{}, err
549 visiting := map[string]bool{}
550 var walk func(string)
551 walk = func(name string) {
552 d := by[name]
553 if d == nil || d.present["__resolved"] {
554 return
555 }
556 if visiting[name] {
557 issues = append(issues, Issue{Line: d.line, Key: "ci.err.extendsCycle", Args: []any{name}})
558 return
559 }
560 visiting[name] = true
561 for _, parent := range d.extends {
562 base := by[parent]
563 if base == nil {
564 issues = append(issues, Issue{Line: d.line, Key: "ci.err.extends", Args: []any{name, parent}})
565 continue
200 566 }
201 a.ExpireIn = d
202 default:
203 return Artifacts{}, fmt.Errorf("onbekend artifacts-veld %q", key)
567 walk(parent)
568 inherit(d, base)
204 569 }
570 delete(visiting, name)
571 d.present["__resolved"] = true
572 inherit(d, &def)
573 }
574 for _, name := range order {
575 walk(name)
205 576 }
206 return a, nil
577 return jobs, issues
207 578 }
208 579
209 func parseRules(node *yaml.Node) ([]Rule, error) {
210 if node.Kind != yaml.SequenceNode {
211 return nil, fmt.Errorf("rules moet een lijst zijn")
580 func inherit(child, parent *draft) {
581 if parent == nil {
582 return
212 583 }
213 var rules []Rule
214 for _, item := range node.Content {
215 if item.Kind != yaml.MappingNode {
216 return nil, fmt.Errorf("rule moet een mapping zijn")
584 copyField := func(key string, apply func()) {
585 if child.present[key] || !parent.present[key] {
586 return
217 587 }
218 var r Rule
219 for i := 0; i < len(item.Content); i += 2 {
220 key := item.Content[i].Value
221 val := item.Content[i+1]
222 if key != "if" {
223 return nil, fmt.Errorf("rule ondersteunt alleen if, kreeg %q", key)
224 }
225 if val.Kind != yaml.ScalarNode || strings.TrimSpace(val.Value) == "" {
226 return nil, fmt.Errorf("rule.if moet tekst zijn")
227 }
228 r.If = strings.TrimSpace(val.Value)
229 }
230 if r.If == "" {
231 return nil, fmt.Errorf("rule mist if")
588 apply()
589 child.present[key] = true
590 }
591 copyField("stage", func() { child.job.Stage = parent.job.Stage })
592 copyField("image", func() { child.job.Image = parent.job.Image })
593 copyField("before_script", func() { child.job.Before = parent.job.Before })
594 copyField("script", func() { child.job.Script = parent.job.Script })
595 copyField("after_script", func() { child.job.After = parent.job.After })
596 copyField("needs", func() {
597 child.job.Needs = parent.job.Needs
598 child.job.Optional = parent.job.Optional
599 child.job.NeedsSet = parent.job.NeedsSet
600 })
601 copyField("artifacts", func() { child.job.Artifacts = parent.job.Artifacts })
602 copyField("rules", func() { child.job.Rules = parent.job.Rules })
603 copyField("allow_failure", func() { child.job.AllowFailure = parent.job.AllowFailure })
604 copyField("when", func() { child.job.When = parent.job.When })
605 copyField("timeout", func() { child.job.Timeout = parent.job.Timeout })
606 copyField("tags", func() { child.job.Tags = parent.job.Tags })
607 copyField("cache", func() { child.job.Cache = parent.job.Cache })
608 copyField("resource_group", func() { child.job.ResourceGroup = parent.job.ResourceGroup })
609 copyField("only", func() { child.job.Only = parent.job.Only })
610 copyField("except", func() { child.job.Except = parent.job.Except })
611 if len(parent.job.Variables) > 0 {
612 merged := map[string]string{}
613 for k, v := range parent.job.Variables {
614 merged[k] = v
232 615 }
233 if _, err := compileRule(r.If); err != nil {
234 return nil, err
616 for k, v := range child.job.Variables {
617 merged[k] = v
235 618 }
236 rules = append(rules, r)
619 child.job.Variables = merged
237 620 }
238 return rules, nil
239 621 }
240 622
241 func (p *Pipeline) validate() error {
623 func validate(p *Pipeline) []Issue {
624 var issues []Issue
242 625 if len(p.Jobs) == 0 {
243 return fmt.Errorf("geen jobs")
626 return []Issue{{Line: 1, Key: "ci.err.noJobs"}}
244 627 }
245 628 seenStage := map[string]bool{}
246 629 for _, s := range p.Stages {
247 630 if s == "" {
248 return fmt.Errorf("lege stage-naam")
631 issues = append(issues, Issue{Line: 1, Key: "ci.err.stageEmpty"})
632 continue
249 633 }
250 634 if seenStage[s] {
251 return fmt.Errorf("dubbele stage %q", s)
635 issues = append(issues, Issue{Line: 1, Key: "ci.err.stageDup", Args: []any{s}})
252 636 }
253 637 seenStage[s] = true
254 638 }
255 639 if len(p.Stages) == 0 {
256 640 var order []string
257 have := map[string]bool{}
258 641 for i := range p.Jobs {
259 642 if p.Jobs[i].Stage == "" {
260 643 p.Jobs[i].Stage = "test"
261 644 }
262 if !have[p.Jobs[i].Stage] {
263 have[p.Jobs[i].Stage] = true
645 if !seenStage[p.Jobs[i].Stage] {
646 seenStage[p.Jobs[i].Stage] = true
264 647 order = append(order, p.Jobs[i].Stage)
265 648 }
266 649 }
267 650 p.Stages = order
268 for _, s := range p.Stages {
269 seenStage[s] = true
270 }
271 651 }
272 652 names := map[string]bool{}
273 653 for i := range p.Jobs {
274 654 j := &p.Jobs[i]
275 655 if j.Name == "" || strings.ContainsAny(j.Name, " \t\n/\\") {
276 return fmt.Errorf("ongeldige jobnaam %q", j.Name)
277 }
278 if names[j.Name] {
279 return fmt.Errorf("dubbele job %q", j.Name)
656 issues = append(issues, Issue{Line: 1, Key: "ci.err.jobName", Args: []any{j.Name}})
657 continue
280 658 }
281 659 names[j.Name] = true
282 660 if j.Stage == "" {
283 return fmt.Errorf("job %s mist stage", j.Name)
284 }
285 if !seenStage[j.Stage] {
286 return fmt.Errorf("job %s gebruikt onbekende stage %q", j.Name, j.Stage)
661 issues = append(issues, Issue{Line: 1, Key: "ci.err.jobStage", Args: []any{j.Name}})
662 } else if !seenStage[j.Stage] {
663 issues = append(issues, Issue{Line: 1, Key: "ci.err.stageUnknown", Args: []any{j.Name, j.Stage}})
287 664 }
288 665 if j.Image == "" {
289 666 j.Image = p.Image
290 667 }
291 668 if strings.TrimSpace(j.Image) == "" || strings.ContainsAny(j.Image, " \t\n") {
292 return fmt.Errorf("job %s mist een image", j.Name)
669 issues = append(issues, Issue{Line: 1, Key: "ci.err.noImage", Args: []any{j.Name}})
293 670 }
294 671 if len(j.Script) == 0 {
295 return fmt.Errorf("job %s mist script", j.Name)
672 issues = append(issues, Issue{Line: 1, Key: "ci.err.noScript", Args: []any{j.Name}})
296 673 }
297 674 for _, line := range j.Script {
298 675 if strings.TrimSpace(line) == "" {
299 return fmt.Errorf("job %s heeft een lege scriptregel", j.Name)
676 issues = append(issues, Issue{Line: 1, Key: "ci.err.emptyLine", Args: []any{j.Name}})
300 677 }
301 678 }
302 679 for _, path := range j.Artifacts.Paths {
303 if err := safeRelPath(path); err != nil {
304 return fmt.Errorf("job %s artifact: %w", j.Name, err)
680 if iss := safeRel(path); iss.Key != "" {
681 issues = append(issues, iss)
305 682 }
306 683 }
684 for _, path := range j.Cache.Paths {
685 if iss := safeRel(path); iss.Key != "" {
686 issues = append(issues, iss)
687 }
688 }
689 if j.When == "" {
690 j.When = "on_success"
691 }
307 692 }
308 693 for _, j := range p.Jobs {
309 694 for _, need := range j.Needs {
310 if !names[need] {
311 return fmt.Errorf("job %s needs onbekende job %q", j.Name, need)
695 if !names[need] && !contains(j.Optional, need) {
696 issues = append(issues, Issue{Line: 1, Key: "ci.err.needUnknown", Args: []any{j.Name, need}})
312 697 }
313 698 if need == j.Name {
314 return fmt.Errorf("job %s kan niet van zichzelf afhangen", j.Name)
699 issues = append(issues, Issue{Line: 1, Key: "ci.err.needSelf", Args: []any{j.Name}})
315 700 }
316 701 }
317 702 }
318 if err := detectCycles(p.Jobs); err != nil {
319 return err
703 if iss := detectCycles(p.Jobs); iss.Key != "" {
704 issues = append(issues, iss)
320 705 }
321 return nil
706 return issues
322 707 }
323 708
324 func detectCycles(jobs []Job) error {
709 func detectCycles(jobs []Job) Issue {
325 710 idx := map[string]int{}
326 711 for i, j := range jobs {
327 712 idx[j.Name] = i
@@ -332,237 +717,382 @@ func detectCycles(jobs []Job) error {
332 717 black = 2
333 718 )
334 719 color := make([]int, len(jobs))
335 var visit func(int) error
336 visit = func(i int) error {
720 var visit func(int) Issue
721 visit = func(i int) Issue {
337 722 if color[i] == gray {
338 return fmt.Errorf("cyclus in needs bij job %s", jobs[i].Name)
723 return Issue{Line: 1, Key: "ci.err.cycle", Args: []any{jobs[i].Name}}
339 724 }
340 725 if color[i] == black {
341 return nil
726 return Issue{}
342 727 }
343 728 color[i] = gray
344 729 for _, n := range jobs[i].Needs {
345 if err := visit(idx[n]); err != nil {
346 return err
730 target, ok := idx[n]
731 if !ok {
732 continue
733 }
734 if iss := visit(target); iss.Key != "" {
735 return iss
347 736 }
348 737 }
349 738 color[i] = black
350 return nil
739 return Issue{}
351 740 }
352 741 for i := range jobs {
353 if err := visit(i); err != nil {
354 return err
742 if iss := visit(i); iss.Key != "" {
743 return iss
355 744 }
356 745 }
357 return nil
746 return Issue{}
358 747 }
359 748
360 // ActiveJobs filtert jobs waarvan de rules niet kloppen.
361 func (p Pipeline) ActiveJobs(env map[string]string) ([]Job, error) {
362 var out []Job
363 for _, j := range p.Jobs {
364 ok, err := j.matches(env)
365 if err != nil {
366 return nil, fmt.Errorf("job %s: %w", j.Name, err)
367 }
368 if ok {
369 out = append(out, j)
749 func parseArtifacts(node *yaml.Node) (Artifacts, []Issue) {
750 if node.Kind != yaml.MappingNode {
751 return Artifacts{}, []Issue{{Line: node.Line, Key: "ci.err.artifacts"}}
752 }
753 var a Artifacts
754 var issues []Issue
755 for i := 0; i < len(node.Content); i += 2 {
756 key := node.Content[i].Value
757 val := node.Content[i+1]
758 switch key {
759 case "paths":
760 paths, iss := scalarList(val, "ci.err.list")
761 issues = append(issues, iss...)
762 a.Paths = paths
763 case "expire_in":
764 if val.Kind != yaml.ScalarNode {
765 issues = append(issues, Issue{Line: val.Line, Key: "ci.err.expire"})
766 continue
767 }
768 d, err := parseExpire(val.Value)
769 if err != nil {
770 issues = append(issues, Issue{Line: val.Line, Key: "ci.err.expireBad", Args: []any{val.Value}})
771 continue
772 }
773 a.ExpireIn = d
774 default:
775 issues = append(issues, Issue{Line: node.Content[i].Line, Key: "ci.warn.ignored", Args: []any{"artifacts." + key}, Warning: true})
370 776 }
371 777 }
372 return out, nil
778 return a, issues
373 779 }
374 780
375 func (j Job) matches(env map[string]string) (bool, error) {
376 if len(j.Rules) == 0 {
377 return true, nil
781 func parseCache(node *yaml.Node) (Cache, []Issue) {
782 if node.Kind != yaml.MappingNode {
783 return Cache{}, []Issue{{Line: node.Line, Key: "ci.err.cache"}}
378 784 }
379 for _, r := range j.Rules {
380 ok, err := evalRule(r.If, env)
381 if err != nil {
382 return false, err
383 }
384 if ok {
385 return true, nil
386 }
387 }
388 return false, nil
389 }
390
391 // Gate beslist of een job mag starten.
392 // "ready" betekent dat alle voorgangers klaar zijn.
393 // "wait" betekent dat er nog een voorganger loopt of in de wachtrij staat.
394 // "skip" betekent dat een voorganger mislukte zonder allow_failure.
395 //
396 // NeedsSet volgt de genoemde jobs, ook als de lijst leeg is.
397 // Zonder needs wacht de job op alle jobs in eerdere stages.
398 func Gate(jobs []Job, status map[string]string, job Job) string {
399 byName := map[string]Job{}
400 for _, j := range jobs {
401 byName[j.Name] = j
402 }
403 var deps []Job
404 if job.NeedsSet {
405 for _, n := range job.Needs {
406 deps = append(deps, byName[n])
407 }
408 } else {
409 stagePos := map[string]int{}
410 order := 0
411 for _, j := range jobs {
412 if _, ok := stagePos[j.Stage]; !ok {
413 stagePos[j.Stage] = order
414 order++
785 var c Cache
786 var issues []Issue
787 for i := 0; i < len(node.Content); i += 2 {
788 key := node.Content[i].Value
789 val := node.Content[i+1]
790 switch key {
791 case "paths":
792 paths, iss := scalarList(val, "ci.err.list")
793 issues = append(issues, iss...)
794 c.Paths = paths
795 case "key":
796 if val.Kind != yaml.ScalarNode {
797 issues = append(issues, Issue{Line: val.Line, Key: "ci.err.cacheKey"})
798 continue
415 799 }
800 c.Key = val.Value
801 default:
802 issues = append(issues, Issue{Line: node.Content[i].Line, Key: "ci.warn.ignored", Args: []any{"cache." + key}, Warning: true})
416 803 }
417 my := stagePos[job.Stage]
418 for _, j := range jobs {
419 if stagePos[j.Stage] < my {
420 deps = append(deps, j)
804 }
805 return c, issues
806 }
807
808 func parseRules(node *yaml.Node) ([]Rule, []Issue) {
809 if node.Kind != yaml.SequenceNode {
810 return nil, []Issue{{Line: node.Line, Key: "ci.err.rulesList"}}
811 }
812 var rules []Rule
813 var issues []Issue
814 for _, item := range node.Content {
815 if item.Kind != yaml.MappingNode {
816 issues = append(issues, Issue{Line: item.Line, Key: "ci.err.ruleMap"})
817 continue
818 }
819 var r Rule
820 for i := 0; i < len(item.Content); i += 2 {
821 key := item.Content[i].Value
822 val := item.Content[i+1]
823 switch key {
824 case "if":
825 if val.Kind != yaml.ScalarNode || strings.TrimSpace(val.Value) == "" {
826 issues = append(issues, Issue{Line: val.Line, Key: "ci.err.ruleIf"})
827 continue
828 }
829 r.If = strings.TrimSpace(val.Value)
830 if _, err := compileRule(r.If); err != nil {
831 if ie, ok := err.(i18n.Error); ok {
832 issues = append(issues, Issue{Line: val.Line, Key: ie.Key, Args: ie.Args})
833 } else {
834 issues = append(issues, Issue{Line: val.Line, Key: "ci.err.ruleExpr", Args: []any{r.If}})
835 }
836 }
837 case "changes":
838 list, iss := scalarList(val, "ci.err.list")
839 issues = append(issues, iss...)
840 r.Changes = list
841 case "exists":
842 list, iss := scalarList(val, "ci.err.list")
843 issues = append(issues, iss...)
844 r.Exists = list
845 case "when":
846 w, iss := whenOf(val, true)
847 issues = append(issues, iss...)
848 r.When = w
849 default:
850 issues = append(issues, Issue{Line: item.Content[i].Line, Key: "ci.warn.ignored", Args: []any{"rules." + key}, Warning: true})
421 851 }
422 852 }
423 }
424 for _, d := range deps {
425 switch status[d.Name] {
426 case "success", "skipped":
853 if r.If == "" && len(r.Changes) == 0 && len(r.Exists) == 0 && r.When == "" {
854 issues = append(issues, Issue{Line: item.Line, Key: "ci.err.ruleEmpty"})
427 855 continue
428 case "failed":
429 if d.AllowFailure {
856 }
857 rules = append(rules, r)
858 }
859 return rules, issues
860 }
861
862 func parseNeeds(node *yaml.Node) (needs, optional []string, set bool, issues []Issue) {
863 set = true
864 if node.Kind != yaml.SequenceNode {
865 return nil, nil, true, []Issue{{Line: node.Line, Key: "ci.err.needs"}}
866 }
867 for _, item := range node.Content {
868 switch item.Kind {
869 case yaml.ScalarNode:
870 needs = append(needs, item.Value)
871 case yaml.MappingNode:
872 var name string
873 var opt bool
874 for i := 0; i < len(item.Content); i += 2 {
875 switch item.Content[i].Value {
876 case "job":
877 if item.Content[i+1].Kind == yaml.ScalarNode {
878 name = item.Content[i+1].Value
879 }
880 case "optional":
881 b, iss := parseBool(item.Content[i+1])
882 issues = append(issues, iss...)
883 opt = b
884 default:
885 issues = append(issues, Issue{Line: item.Content[i].Line, Key: "ci.warn.ignored", Args: []any{"needs." + item.Content[i].Value}, Warning: true})
886 }
887 }
888 if name == "" {
889 issues = append(issues, Issue{Line: item.Line, Key: "ci.err.needs"})
430 890 continue
431 891 }
432 return "skip"
892 needs = append(needs, name)
893 if opt {
894 optional = append(optional, name)
895 }
433 896 default:
434 return "wait"
897 issues = append(issues, Issue{Line: item.Line, Key: "ci.err.needs"})
435 898 }
436 899 }
437 return "ready"
900 if needs == nil {
901 needs = []string{}
902 }
903 return needs, optional, set, issues
438 904 }
439 905
440 var (
441 reEq = regexp.MustCompile(`^\$([A-Za-z_][A-Za-z0-9_]*)\s*==\s*"([^"]*)"$`)
442 reNe = regexp.MustCompile(`^\$([A-Za-z_][A-Za-z0-9_]*)\s*!=\s*"([^"]*)"$`)
443 reRe = regexp.MustCompile(`^\$([A-Za-z_][A-Za-z0-9_]*)\s*=~\s*/(.+)/$`)
444 )
445
446 type compiledRule struct {
447 kind string
448 name string
449 arg string
450 re *regexp.Regexp
906 func imageOf(node *yaml.Node) (string, []Issue) {
907 switch node.Kind {
908 case yaml.ScalarNode:
909 return node.Value, nil
910 case yaml.MappingNode:
911 for i := 0; i < len(node.Content); i += 2 {
912 if node.Content[i].Value == "name" && node.Content[i+1].Kind == yaml.ScalarNode {
913 return node.Content[i+1].Value, nil
914 }
915 }
916 return "", []Issue{{Line: node.Line, Key: "ci.err.image"}}
917 default:
918 return "", []Issue{{Line: node.Line, Key: "ci.err.image"}}
919 }
451 920 }
452 921
453 func compileRule(expr string) (compiledRule, error) {
454 if m := reEq.FindStringSubmatch(expr); m != nil {
455 return compiledRule{kind: "eq", name: m[1], arg: m[2]}, nil
456 }
457 if m := reNe.FindStringSubmatch(expr); m != nil {
458 return compiledRule{kind: "ne", name: m[1], arg: m[2]}, nil
922 func whenOf(node *yaml.Node, allowNever bool) (string, []Issue) {
923 if node.Kind != yaml.ScalarNode {
924 return "", []Issue{{Line: node.Line, Key: "ci.err.when", Args: []any{""}}}
459 925 }
460 if m := reRe.FindStringSubmatch(expr); m != nil {
461 re, err := regexp.Compile(m[2])
462 if err != nil {
463 return compiledRule{}, fmt.Errorf("ongeldige regex in rule %q: %w", expr, err)
926 switch node.Value {
927 case "on_success", "on_failure", "always", "manual":
928 return node.Value, nil
929 case "never":
930 if allowNever {
931 return node.Value, nil
464 932 }
465 return compiledRule{kind: "re", name: m[1], re: re}, nil
466 933 }
467 return compiledRule{}, fmt.Errorf("rule %q wordt niet begrepen", expr)
934 return "", []Issue{{Line: node.Line, Key: "ci.err.when", Args: []any{node.Value}}}
468 935 }
469 936
470 func evalRule(expr string, env map[string]string) (bool, error) {
471 c, err := compileRule(expr)
937 func durationOf(node *yaml.Node) (time.Duration, []Issue) {
938 if node.Kind != yaml.ScalarNode {
939 return 0, []Issue{{Line: node.Line, Key: "ci.err.timeout", Args: []any{""}}}
940 }
941 d, err := parseExpire(node.Value)
472 942 if err != nil {
473 return false, err
474 }
475 val := env[c.name]
476 switch c.kind {
477 case "eq":
478 return val == c.arg, nil
479 case "ne":
480 return val != c.arg, nil
481 case "re":
482 return c.re.MatchString(val), nil
483 default:
484 return false, fmt.Errorf("interne rulefout")
943 if strings.HasSuffix(node.Value, "s") {
944 n, nerr := strconv.Atoi(strings.TrimSuffix(node.Value, "s"))
945 if nerr == nil && n >= 0 {
946 return time.Duration(n) * time.Second, nil
947 }
948 }
949 return 0, []Issue{{Line: node.Line, Key: "ci.err.timeout", Args: []any{node.Value}}}
485 950 }
951 return d, nil
486 952 }
487 953
488 func parseExpire(s string) (time.Duration, error) {
489 s = strings.TrimSpace(s)
490 if s == "" || s == "never" {
491 return 0, nil
954 func extendsOf(node *yaml.Node) ([]string, []Issue) {
955 if node.Kind == yaml.ScalarNode && strings.TrimSpace(node.Value) != "" {
956 return []string{node.Value}, nil
492 957 }
493 if len(s) < 2 {
494 return 0, fmt.Errorf("expire_in %q is ongeldig", s)
495 }
496 n, err := strconv.Atoi(s[:len(s)-1])
497 if err != nil || n < 0 {
498 return 0, fmt.Errorf("expire_in %q is ongeldig", s)
958 return scalarList(node, "ci.err.extendsList")
959 }
960
961 func scriptList(node *yaml.Node) ([]string, []Issue) {
962 if node.Kind == yaml.ScalarNode {
963 if strings.TrimSpace(node.Value) == "" {
964 return nil, []Issue{{Line: node.Line, Key: "ci.err.script"}}
965 }
966 return splitLines(node.Value), nil
499 967 }
500 switch s[len(s)-1] {
501 case 'm':
502 return time.Duration(n) * time.Minute, nil
503 case 'h':
504 return time.Duration(n) * time.Hour, nil
505 case 'd':
506 return time.Duration(n) * 24 * time.Hour, nil
507 case 'w':
508 return time.Duration(n) * 7 * 24 * time.Hour, nil
509 default:
510 return 0, fmt.Errorf("expire_in %q is ongeldig", s)
968 return scalarList(node, "ci.err.script")
969 }
970
971 func splitLines(s string) []string {
972 parts := strings.Split(s, "\n")
973 var out []string
974 for _, p := range parts {
975 if strings.TrimSpace(p) == "" {
976 continue
977 }
978 out = append(out, p)
511 979 }
980 return out
512 981 }
513 982
514 func scalarList(node *yaml.Node) ([]string, error) {
983 func scalarList(node *yaml.Node, key string) ([]string, []Issue) {
515 984 if node.Kind != yaml.SequenceNode {
516 return nil, fmt.Errorf("verwacht een lijst")
985 return nil, []Issue{{Line: node.Line, Key: key}}
517 986 }
518 987 var out []string
988 var issues []Issue
519 989 for _, item := range node.Content {
520 990 if item.Kind != yaml.ScalarNode {
521 return nil, fmt.Errorf("lijst mag alleen tekst bevatten")
991 issues = append(issues, Issue{Line: item.Line, Key: "ci.err.listText"})
992 continue
522 993 }
523 994 out = append(out, item.Value)
524 995 }
525 return out, nil
996 return out, issues
526 997 }
527 998
528 func scalarMap(node *yaml.Node) (map[string]string, error) {
999 func scalarMap(node *yaml.Node) (map[string]string, []Issue) {
529 1000 if node.Kind != yaml.MappingNode {
530 return nil, fmt.Errorf("verwacht een mapping")
1001 return nil, []Issue{{Line: node.Line, Key: "ci.err.map"}}
531 1002 }
532 1003 out := map[string]string{}
1004 var issues []Issue
533 1005 for i := 0; i < len(node.Content); i += 2 {
534 1006 k := node.Content[i]
535 1007 v := node.Content[i+1]
536 if k.Kind != yaml.ScalarNode || v.Kind != yaml.ScalarNode {
537 return nil, fmt.Errorf("variabelen moeten tekst zijn")
1008 if k.Kind != yaml.ScalarNode {
1009 issues = append(issues, Issue{Line: k.Line, Key: "ci.err.varText"})
1010 continue
1011 }
1012 switch v.Kind {
1013 case yaml.ScalarNode:
1014 out[k.Value] = v.Value
1015 case yaml.MappingNode:
1016 val, iss := mappingValue(v)
1017 issues = append(issues, iss...)
1018 if val != "" || len(iss) == 0 {
1019 out[k.Value] = val
1020 }
1021 default:
1022 issues = append(issues, Issue{Line: v.Line, Key: "ci.err.varText"})
1023 }
1024 }
1025 return out, issues
1026 }
1027
1028 func mappingValue(node *yaml.Node) (string, []Issue) {
1029 for i := 0; i < len(node.Content); i += 2 {
1030 if node.Content[i].Value == "value" && node.Content[i+1].Kind == yaml.ScalarNode {
1031 return node.Content[i+1].Value, nil
538 1032 }
539 out[k.Value] = v.Value
540 1033 }
541 return out, nil
1034 return "", []Issue{{Line: node.Line, Key: "ci.err.varText"}}
542 1035 }
543 1036
544 func parseBool(node *yaml.Node) (bool, error) {
1037 func parseBool(node *yaml.Node) (bool, []Issue) {
545 1038 if node.Kind != yaml.ScalarNode {
546 return false, fmt.Errorf("verwacht ja of nee")
1039 return false, []Issue{{Line: node.Line, Key: "ci.err.bool"}}
547 1040 }
548 1041 switch strings.ToLower(node.Value) {
549 case "true", "yes", "ja":
1042 case "true", "yes":
550 1043 return true, nil
551 case "false", "no", "nee":
1044 case "false", "no":
552 1045 return false, nil
553 1046 default:
554 return false, fmt.Errorf("verwacht true of false")
1047 return false, []Issue{{Line: node.Line, Key: "ci.err.bool"}}
555 1048 }
556 1049 }
557 1050
558 func safeRelPath(p string) error {
1051 func parseExpire(s string) (time.Duration, error) {
1052 s = strings.TrimSpace(s)
1053 if s == "" || s == "never" {
1054 return 0, nil
1055 }
1056 if len(s) < 2 {
1057 return 0, fmt.Errorf("expire")
1058 }
1059 n, err := strconv.Atoi(s[:len(s)-1])
1060 if err != nil || n < 0 {
1061 return 0, fmt.Errorf("expire")
1062 }
1063 switch s[len(s)-1] {
1064 case 's':
1065 return time.Duration(n) * time.Second, nil
1066 case 'm':
1067 return time.Duration(n) * time.Minute, nil
1068 case 'h':
1069 return time.Duration(n) * time.Hour, nil
1070 case 'd':
1071 return time.Duration(n) * 24 * time.Hour, nil
1072 case 'w':
1073 return time.Duration(n) * 7 * 24 * time.Hour, nil
1074 default:
1075 return 0, fmt.Errorf("expire")
1076 }
1077 }
1078
1079 func safeRel(p string) Issue {
559 1080 if p == "" || strings.HasPrefix(p, "/") || strings.Contains(p, "\\") {
560 return fmt.Errorf("pad %q moet relatief zijn", p)
1081 return Issue{Line: 1, Key: "ci.err.rel", Args: []any{p}}
561 1082 }
562 1083 for _, part := range strings.Split(p, "/") {
563 1084 if part == ".." {
564 return fmt.Errorf("pad %q mag niet buiten de workspace", p)
1085 return Issue{Line: 1, Key: "ci.err.dotdot", Args: []any{p}}
1086 }
1087 }
1088 return Issue{}
1089 }
1090
1091 func hasError(issues []Issue) bool {
1092 for _, is := range issues {
1093 if !is.Warning {
1094 return true
565 1095 }
566 1096 }
567 return nil
1097 return false
568 1098 }
@@ -1,327 +1,712 @@
1// Package ci leest blue-ci-cd.yml en bepaalt welke jobs draaien.
1// Package ci leest blue-ci-cd.yml en bepaalt welke jobs draaien.
2// De vorm ligt dicht bij GitLab CI: stages, jobs, needs, rules, artifacts.
2// De vorm volgt GitLab CI waar dat in BlueGit past. Een oud bestand met
3// een jobs:-map blijft geldig. Jobs mogen ook los op het hoogste niveau staan.
3package ci
4package ci
4
5
5import (
6import (
6 "fmt"
7 "fmt"
7 "regexp"
8 "strconv"
8 "strconv"
9 "strings"
9 "strings"
10 "time"
10 "time"
11
11
12 "gitlab.com/Timfa/bluegit/internal/i18n"
12 "gopkg.in/yaml.v3"
13 "gopkg.in/yaml.v3"
13)
14)
14
15
15// Pipeline is een geparset configuratiebestand.
16// Parse leest YAML. Jobvolgorde blijft de volgorde uit het bestand.
16type Pipeline struct {
17func Parse(data []byte) (Pipeline, error) {
17 Stages []string
18 return ParseIn(data, nil)
18 Variables map[string]string
19}
19 Image string
20
20 Jobs []Job
21// ParseIn leest YAML en haalt include: local via read op.
22func ParseIn(data []byte, read func(string) ([]byte, error)) (Pipeline, error) {
23 p, issues, err := parseBytes(data, read, nil)
24 if err != nil {
25 return Pipeline{}, err
26 }
27 for _, is := range issues {
28 if !is.Warning {
29 return Pipeline{}, is
30 }
31 }
32 p.Warnings = warningsOnly(issues)
33 return p, nil
21}
34}
22
35
23// Job is één containerstap.
36// Lint geeft elke fout en waarschuwing, met regelnummer, zonder te stoppen bij de eerste.
24type Job struct {
37func Lint(data []byte, read func(string) ([]byte, error)) (Pipeline, []Issue) {
25 Name string
38 p, issues, err := parseBytes(data, read, nil)
26 Stage string
39 if err != nil {
27 Image string
40 issues = append([]Issue{{Line: 1, Key: "ci.err.yaml", Args: []any{err.Error()}}}, issues...)
28 Script []string
41 }
29 Needs []string
42 return p, issues
30 NeedsSet bool
43}
31 Variables map[string]string
44
32 Artifacts Artifacts
45func (i Issue) Error() string { return i18n.T(i18n.EN, i.Key, i.Args...) }
33 Rules []Rule
46
34 AllowFailure bool
47func warningsOnly(in []Issue) []Issue {
48 var out []Issue
49 for _, is := range in {
50 if is.Warning {
51 out = append(out, is)
52 }
53 }
54 return out
35}
55}
36
56
37// Artifacts zijn paden die na een geslaagde job bewaard blijven.
57var reserved = map[string]bool{
38type Artifacts struct {
58 "stages": true, "variables": true, "image": true, "default": true,
39 Paths []string
59 "include": true, "before_script": true, "after_script": true, "cache": true,
40 ExpireIn time.Duration
60 "workflow": true, "services": true, "spec": true, "pages": true, "jobs": true,
41}
61}
42
62
43// Rule is één voorwaarde. De job draait als minstens één rule klopt.
63type draft struct {
44// Zonder rules draait de job altijd.
64 job Job
45type Rule struct {
65 present map[string]bool
46 If string
66 extends []string
67 line int
47}
68}
48
69
49// Parse leest YAML. Jobvolgorde blijft de volgorde uit het bestand.
70func parseBytes(data []byte, read func(string) ([]byte, error), stack []string) (Pipeline, []Issue, error) {
50func Parse(data []byte) (Pipeline, error) {
51 var root yaml.Node
71 var root yaml.Node
52 if err := yaml.Unmarshal(data, &root); err != nil {
72 if err := yaml.Unmarshal(data, &root); err != nil {
53 return Pipeline{}, fmt.Errorf("yaml: %w", err)
73 return Pipeline{}, nil, i18n.E("ci.err.yaml", err.Error())
54 }
74 }
55 if root.Kind != yaml.DocumentNode || len(root.Content) != 1 || root.Content[0].Kind != yaml.MappingNode {
75 if root.Kind != yaml.DocumentNode || len(root.Content) != 1 || root.Content[0].Kind != yaml.MappingNode {
56 return Pipeline{}, fmt.Errorf("blue-ci-cd.yml moet een mapping zijn")
76 return Pipeline{}, []Issue{{Line: 1, Key: "ci.err.mapping"}}, nil
57 }
77 }
58 body := root.Content[0]
78 body := root.Content[0]
59 var p Pipeline
79 var included Pipeline
60 p.Variables = map[string]string{}
80 var issues []Issue
81 for i := 0; i < len(body.Content); i += 2 {
82 if body.Content[i].Value != "include" {
83 continue
84 }
85 part, more, err := parseInclude(body.Content[i+1], read, stack)
86 if err != nil {
87 return Pipeline{}, nil, err
88 }
89 issues = append(issues, more...)
90 included = mergePipeline(included, part)
91 }
92 p := included
93 if p.Variables == nil {
94 p.Variables = map[string]string{}
95 }
96 var def draft
97 def.present = map[string]bool{}
98 var jobs []draft
99 seenLegacy := false
100 seenTop := false
61 for i := 0; i < len(body.Content); i += 2 {
101 for i := 0; i < len(body.Content); i += 2 {
62 key := body.Content[i].Value
102 keyNode := body.Content[i]
103 key := keyNode.Value
63 val := body.Content[i+1]
104 val := body.Content[i+1]
64 switch key {
105 switch key {
106 case "include":
107 continue
65 case "stages":
108 case "stages":
66 stages, err := scalarList(val)
109 stages, iss := scalarList(val, "ci.err.list")
67 if err != nil {
110 issues = append(issues, iss...)
68 return Pipeline{}, fmt.Errorf("stages: %w", err)
111 if len(stages) > 0 {
112 p.Stages = stages
69 }
113 }
70 p.Stages = stages
71 case "variables":
114 case "variables":
72 vars, err := scalarMap(val)
115 vars, iss := scalarMap(val)
73 if err != nil {
116 issues = append(issues, iss...)
74 return Pipeline{}, fmt.Errorf("variables: %w", err)
117 for k, v := range vars {
118 p.Variables[k] = v
75 }
119 }
76 p.Variables = vars
77 case "image":
120 case "image":
78 if val.Kind != yaml.ScalarNode {
121 img, iss := imageOf(val)
79 return Pipeline{}, fmt.Errorf("image moet een tekst zijn")
122 issues = append(issues, iss...)
123 if img != "" {
124 p.Image = img
80 }
125 }
81 p.Image = val.Value
126 case "before_script":
127 lines, iss := scriptList(val)
128 issues = append(issues, iss...)
129 p.Before = lines
130 def.present["before_script"] = true
131 def.job.Before = lines
132 case "after_script":
133 lines, iss := scriptList(val)
134 issues = append(issues, iss...)
135 p.After = lines
136 def.present["after_script"] = true
137 def.job.After = lines
138 case "cache":
139 c, iss := parseCache(val)
140 issues = append(issues, iss...)
141 p.Cache = c
142 def.present["cache"] = true
143 def.job.Cache = c
144 case "default":
145 d, iss := parseDefault(val)
146 issues = append(issues, iss...)
147 def = overlayDefault(def, d)
148 case "workflow", "services", "spec", "pages":
149 issues = append(issues, Issue{Line: keyNode.Line, Key: "ci.warn.ignored", Args: []any{key}, Warning: true})
82 case "jobs":
150 case "jobs":
83 jobs, err := parseJobs(val)
151 seenLegacy = true
84 if err != nil {
152 parsed, iss := parseJobMap(val)
85 return Pipeline{}, err
153 issues = append(issues, iss...)
86 }
154 jobs = append(jobs, parsed...)
87 p.Jobs = jobs
88 default:
155 default:
89 return Pipeline{}, fmt.Errorf("onbekend veld %q", key)
156 if val.Kind != yaml.MappingNode {
157 issues = append(issues, Issue{Line: keyNode.Line, Key: "ci.err.unknown", Args: []any{key}})
158 continue
159 }
160 seenTop = true
161 job, iss := parseJob(key, val)
162 issues = append(issues, iss...)
163 jobs = append(jobs, job)
90 }
164 }
91 }
165 }
92 if err := p.validate(); err != nil {
166 if seenLegacy && seenTop {
93 return Pipeline{}, err
167 issues = append(issues, Issue{Line: 1, Key: "ci.err.both"})
168 }
169 if p.Image != "" && !def.present["image"] {
170 def.present["image"] = true
171 def.job.Image = p.Image
172 }
173 resolved, iss := applyExtends(jobs, def)
174 issues = append(issues, iss...)
175 byName := map[string]int{}
176 var merged []Job
177 for _, j := range p.Jobs {
178 byName[j.Name] = len(merged)
179 merged = append(merged, j)
180 }
181 for _, d := range resolved {
182 if strings.HasPrefix(d.job.Name, ".") {
183 continue
184 }
185 if i, ok := byName[d.job.Name]; ok {
186 merged[i] = overlayJob(merged[i], d)
187 continue
188 }
189 byName[d.job.Name] = len(merged)
190 merged = append(merged, d.job)
191 }
192 p.Jobs = merged
193 issues = append(issues, validate(&p)...)
194 if hasError(issues) {
195 return p, issues, nil
196 }
197 return p, issues, nil
198}
199
200func parseInclude(node *yaml.Node, read func(string) ([]byte, error), stack []string) (Pipeline, []Issue, error) {
201 var items []*yaml.Node
202 switch node.Kind {
203 case yaml.SequenceNode:
204 items = node.Content
205 default:
206 items = []*yaml.Node{node}
207 }
208 var out Pipeline
209 var issues []Issue
210 for _, item := range items {
211 local, iss := includeLocal(item)
212 issues = append(issues, iss...)
213 if local == "" {
214 continue
215 }
216 if read == nil {
217 issues = append(issues, Issue{Line: item.Line, Key: "ci.err.includeRead", Args: []any{local}})
218 continue
219 }
220 for _, prev := range stack {
221 if prev == local {
222 issues = append(issues, Issue{Line: item.Line, Key: "ci.err.includeCycle", Args: []any{local}})
223 local = ""
224 break
225 }
226 }
227 if local == "" {
228 continue
229 }
230 raw, err := read(local)
231 if err != nil {
232 issues = append(issues, Issue{Line: item.Line, Key: "ci.err.includeRead", Args: []any{local}})
233 continue
234 }
235 part, more, err := parseBytes(raw, read, append(append([]string{}, stack...), local))
236 if err != nil {
237 return Pipeline{}, nil, err
238 }
239 issues = append(issues, more...)
240 out = mergePipeline(out, part)
241 }
242 return out, issues, nil
243}
244
245func includeLocal(node *yaml.Node) (string, []Issue) {
246 switch node.Kind {
247 case yaml.ScalarNode:
248 return cleanLocal(node.Value, node)
249 case yaml.MappingNode:
250 for i := 0; i < len(node.Content); i += 2 {
251 key := node.Content[i].Value
252 if key == "local" {
253 if node.Content[i+1].Kind != yaml.ScalarNode {
254 return "", []Issue{{Line: node.Line, Key: "ci.err.includeLocal"}}
255 }
256 return cleanLocal(node.Content[i+1].Value, node.Content[i+1])
257 }
258 }
259 return "", []Issue{{Line: node.Line, Key: "ci.err.includeRemote"}}
260 default:
261 return "", []Issue{{Line: node.Line, Key: "ci.err.includeLocal"}}
262 }
263}
264
265func cleanLocal(p string, node *yaml.Node) (string, []Issue) {
266 p = strings.TrimPrefix(strings.TrimSpace(p), "/")
267 if p == "" || strings.Contains(p, "\\") || strings.Contains(p, "..") {
268 return "", []Issue{{Line: node.Line, Key: "ci.err.rel", Args: []any{p}}}
94 }
269 }
95 return p, nil
270 return p, nil
96}
271}
97
272
98func parseJobs(node *yaml.Node) ([]Job, error) {
273func mergePipeline(base, extra Pipeline) Pipeline {
274 if len(extra.Stages) > 0 {
275 base.Stages = extra.Stages
276 }
277 if base.Variables == nil {
278 base.Variables = map[string]string{}
279 }
280 for k, v := range extra.Variables {
281 base.Variables[k] = v
282 }
283 if extra.Image != "" {
284 base.Image = extra.Image
285 }
286 if extra.Before != nil {
287 base.Before = extra.Before
288 }
289 if extra.After != nil {
290 base.After = extra.After
291 }
292 byName := map[string]int{}
293 for i, j := range base.Jobs {
294 byName[j.Name] = i
295 }
296 for _, j := range extra.Jobs {
297 if i, ok := byName[j.Name]; ok {
298 base.Jobs[i] = j
299 continue
300 }
301 byName[j.Name] = len(base.Jobs)
302 base.Jobs = append(base.Jobs, j)
303 }
304 return base
305}
306
307func parseDefault(node *yaml.Node) (draft, []Issue) {
308 d := draft{present: map[string]bool{}}
99 if node.Kind != yaml.MappingNode {
309 if node.Kind != yaml.MappingNode {
100 return nil, fmt.Errorf("jobs moet een mapping zijn")
310 return d, []Issue{{Line: node.Line, Key: "ci.err.map"}}
101 }
311 }
102 var jobs []Job
312 var issues []Issue
103 for i := 0; i < len(node.Content); i += 2 {
313 for i := 0; i < len(node.Content); i += 2 {
104 name := node.Content[i].Value
314 key := node.Content[i].Value
105 body := node.Content[i+1]
315 val := node.Content[i+1]
106 job, err := parseJob(name, body)
316 switch key {
107 if err != nil {
317 case "image":
108 return nil, fmt.Errorf("job %s: %w", name, err)
318 img, iss := imageOf(val)
319 issues = append(issues, iss...)
320 d.job.Image = img
321 d.present["image"] = true
322 case "before_script":
323 lines, iss := scriptList(val)
324 issues = append(issues, iss...)
325 d.job.Before = lines
326 d.present["before_script"] = true
327 case "after_script":
328 lines, iss := scriptList(val)
329 issues = append(issues, iss...)
330 d.job.After = lines
331 d.present["after_script"] = true
332 case "tags":
333 tags, iss := scalarList(val, "ci.err.list")
334 issues = append(issues, iss...)
335 d.job.Tags = tags
336 d.present["tags"] = true
337 case "timeout":
338 dur, iss := durationOf(val)
339 issues = append(issues, iss...)
340 d.job.Timeout = dur
341 d.present["timeout"] = true
342 case "cache":
343 c, iss := parseCache(val)
344 issues = append(issues, iss...)
345 d.job.Cache = c
346 d.present["cache"] = true
347 default:
348 issues = append(issues, Issue{Line: node.Content[i].Line, Key: "ci.warn.ignored", Args: []any{"default." + key}, Warning: true})
109 }
349 }
110 jobs = append(jobs, job)
111 }
350 }
112 return jobs, nil
351 return d, issues
113}
352}
114
353
115func parseJob(name string, node *yaml.Node) (Job, error) {
354func overlayJob(base Job, d draft) Job {
355 set := func(key string, apply func()) {
356 if d.present[key] {
357 apply()
358 }
359 }
360 set("stage", func() { base.Stage = d.job.Stage })
361 set("image", func() { base.Image = d.job.Image })
362 set("before_script", func() { base.Before = d.job.Before })
363 set("script", func() { base.Script = d.job.Script })
364 set("after_script", func() { base.After = d.job.After })
365 set("needs", func() {
366 base.Needs = d.job.Needs
367 base.Optional = d.job.Optional
368 base.NeedsSet = d.job.NeedsSet
369 })
370 set("artifacts", func() { base.Artifacts = d.job.Artifacts })
371 set("rules", func() { base.Rules = d.job.Rules })
372 set("allow_failure", func() { base.AllowFailure = d.job.AllowFailure })
373 set("when", func() { base.When = d.job.When })
374 set("timeout", func() { base.Timeout = d.job.Timeout })
375 set("tags", func() { base.Tags = d.job.Tags })
376 set("cache", func() { base.Cache = d.job.Cache })
377 set("resource_group", func() { base.ResourceGroup = d.job.ResourceGroup })
378 set("only", func() { base.Only = d.job.Only })
379 set("except", func() { base.Except = d.job.Except })
380 if d.present["variables"] {
381 mergedVars := map[string]string{}
382 for k, v := range base.Variables {
383 mergedVars[k] = v
384 }
385 for k, v := range d.job.Variables {
386 mergedVars[k] = v
387 }
388 base.Variables = mergedVars
389 }
390 return base
391}
392
393func overlayDefault(base, extra draft) draft {
394 if base.present == nil {
395 base.present = map[string]bool{}
396 }
397 for k, on := range extra.present {
398 if !on {
399 continue
400 }
401 base.present[k] = true
402 switch k {
403 case "image":
404 base.job.Image = extra.job.Image
405 case "before_script":
406 base.job.Before = extra.job.Before
407 case "after_script":
408 base.job.After = extra.job.After
409 case "tags":
410 base.job.Tags = extra.job.Tags
411 case "timeout":
412 base.job.Timeout = extra.job.Timeout
413 case "cache":
414 base.job.Cache = extra.job.Cache
415 }
416 }
417 return base
418}
419
420func parseJobMap(node *yaml.Node) ([]draft, []Issue) {
116 if node.Kind != yaml.MappingNode {
421 if node.Kind != yaml.MappingNode {
117 return Job{}, fmt.Errorf("moet een mapping zijn")
422 return nil, []Issue{{Line: node.Line, Key: "ci.err.jobsMap"}}
118 }
423 }
119 j := Job{Name: name, Variables: map[string]string{}}
424 var jobs []draft
425 var issues []Issue
426 for i := 0; i < len(node.Content); i += 2 {
427 job, iss := parseJob(node.Content[i].Value, node.Content[i+1])
428 issues = append(issues, iss...)
429 if job.job.Name != "" {
430 jobs = append(jobs, job)
431 }
432 }
433 return jobs, issues
434}
435
436func parseJob(name string, node *yaml.Node) (draft, []Issue) {
437 d := draft{present: map[string]bool{}, line: node.Line, job: Job{Name: name, Variables: map[string]string{}}}
438 if node.Kind != yaml.MappingNode {
439 return d, []Issue{{Line: node.Line, Key: "ci.err.jobMap", Args: []any{name}}}
440 }
441 var issues []Issue
120 for i := 0; i < len(node.Content); i += 2 {
442 for i := 0; i < len(node.Content); i += 2 {
121 key := node.Content[i].Value
443 key := node.Content[i].Value
122 val := node.Content[i+1]
444 val := node.Content[i+1]
445 d.present[key] = true
123 switch key {
446 switch key {
124 case "stage":
447 case "stage":
125 if val.Kind != yaml.ScalarNode {
448 if val.Kind != yaml.ScalarNode {
126 return Job{}, fmt.Errorf("stage moet tekst zijn")
449 issues = append(issues, Issue{Line: val.Line, Key: "ci.err.stageText"})
450 continue
127 }
451 }
128 j.Stage = val.Value
452 d.job.Stage = val.Value
129 case "image":
453 case "image":
130 if val.Kind != yaml.ScalarNode {
454 img, iss := imageOf(val)
131 return Job{}, fmt.Errorf("image moet tekst zijn")
455 issues = append(issues, iss...)
132 }
456 d.job.Image = img
133 j.Image = val.Value
457 case "before_script":
458 lines, iss := scriptList(val)
459 issues = append(issues, iss...)
460 d.job.Before = lines
134 case "script":
461 case "script":
135 lines, err := scalarList(val)
462 lines, iss := scriptList(val)
136 if err != nil {
463 issues = append(issues, iss...)
137 return Job{}, fmt.Errorf("script: %w", err)
464 d.job.Script = lines
138 }
465 case "after_script":
139 j.Script = lines
466 lines, iss := scriptList(val)
467 issues = append(issues, iss...)
468 d.job.After = lines
140 case "needs":
469 case "needs":
141 needs, err := scalarList(val)
470 needs, optional, set, iss := parseNeeds(val)
142 if err != nil {
471 issues = append(issues, iss...)
143 return Job{}, fmt.Errorf("needs: %w", err)
472 d.job.Needs = needs
144 }
473 d.job.Optional = optional
145 j.Needs = needs
474 d.job.NeedsSet = set
146 j.NeedsSet = true
147 case "variables":
475 case "variables":
148 vars, err := scalarMap(val)
476 vars, iss := scalarMap(val)
149 if err != nil {
477 issues = append(issues, iss...)
150 return Job{}, fmt.Errorf("variables: %w", err)
478 d.job.Variables = vars
151 }
152 j.Variables = vars
153 case "artifacts":
479 case "artifacts":
154 art, err := parseArtifacts(val)
480 art, iss := parseArtifacts(val)
155 if err != nil {
481 issues = append(issues, iss...)
156 return Job{}, err
482 d.job.Artifacts = art
157 }
158 j.Artifacts = art
159 case "rules":
483 case "rules":
160 rules, err := parseRules(val)
484 rules, iss := parseRules(val)
161 if err != nil {
485 issues = append(issues, iss...)
162 return Job{}, err
486 d.job.Rules = rules
163 }
164 j.Rules = rules
165 case "allow_failure":
487 case "allow_failure":
166 b, err := parseBool(val)
488 b, iss := parseBool(val)
167 if err != nil {
489 issues = append(issues, iss...)
168 return Job{}, fmt.Errorf("allow_failure: %w", err)
490 d.job.AllowFailure = b
491 case "when":
492 w, iss := whenOf(val, false)
493 issues = append(issues, iss...)
494 d.job.When = w
495 case "timeout":
496 dur, iss := durationOf(val)
497 issues = append(issues, iss...)
498 d.job.Timeout = dur
499 case "tags":
500 tags, iss := scalarList(val, "ci.err.list")
501 issues = append(issues, iss...)
502 d.job.Tags = tags
503 case "cache":
504 c, iss := parseCache(val)
505 issues = append(issues, iss...)
506 d.job.Cache = c
507 case "resource_group":
508 if val.Kind != yaml.ScalarNode || strings.TrimSpace(val.Value) == "" {
509 issues = append(issues, Issue{Line: val.Line, Key: "ci.err.resource"})
510 continue
169 }
511 }
170 j.AllowFailure = b
512 d.job.ResourceGroup = strings.TrimSpace(val.Value)
513 case "only":
514 list, iss := scalarList(val, "ci.err.list")
515 issues = append(issues, iss...)
516 d.job.Only = list
517 case "except":
518 list, iss := scalarList(val, "ci.err.list")
519 issues = append(issues, iss...)
520 d.job.Except = list
521 case "extends":
522 list, iss := extendsOf(val)
523 issues = append(issues, iss...)
524 d.extends = list
171 default:
525 default:
172 return Job{}, fmt.Errorf("onbekend veld %q", key)
526 if _, ok := reserved[key]; ok {
527 issues = append(issues, Issue{Line: node.Content[i].Line, Key: "ci.warn.ignored", Args: []any{key}, Warning: true})
528 continue
529 }
530 issues = append(issues, Issue{Line: node.Content[i].Line, Key: "ci.warn.ignored", Args: []any{name + "." + key}, Warning: true})
173 }
531 }
174 }
532 }
175 return j, nil
533 return d, issues
176}
534}
177
535
178func parseArtifacts(node *yaml.Node) (Artifacts, error) {
536func applyExtends(jobs []draft, def draft) ([]draft, []Issue) {
179 if node.Kind != yaml.MappingNode {
537 by := map[string]*draft{}
180 return Artifacts{}, fmt.Errorf("artifacts moet een mapping zijn")
538 var order []string
539 var issues []Issue
540 for i := range jobs {
541 name := jobs[i].job.Name
542 if by[name] != nil {
543 issues = append(issues, Issue{Line: jobs[i].line, Key: "ci.err.jobDup", Args: []any{name}})
544 continue
545 }
546 by[name] = &jobs[i]
547 order = append(order, name)
181 }
548 }
182 var a Artifacts
549 visiting := map[string]bool{}
183 for i := 0; i < len(node.Content); i += 2 {
550 var walk func(string)
184 key := node.Content[i].Value
551 walk = func(name string) {
185 val := node.Content[i+1]
552 d := by[name]
186 switch key {
553 if d == nil || d.present["__resolved"] {
187 case "paths":
554 return
188 paths, err := scalarList(val)
555 }
189 if err != nil {
556 if visiting[name] {
190 return Artifacts{}, fmt.Errorf("artifacts.paths: %w", err)
557 issues = append(issues, Issue{Line: d.line, Key: "ci.err.extendsCycle", Args: []any{name}})
191 }
558 return
192 a.Paths = paths
559 }
193 case "expire_in":
560 visiting[name] = true
194 if val.Kind != yaml.ScalarNode {
561 for _, parent := range d.extends {
195 return Artifacts{}, fmt.Errorf("expire_in moet tekst zijn")
562 base := by[parent]
196 }
563 if base == nil {
197 d, err := parseExpire(val.Value)
564 issues = append(issues, Issue{Line: d.line, Key: "ci.err.extends", Args: []any{name, parent}})
198 if err != nil {
565 continue
199 return Artifacts{}, err
200 }
566 }
201 a.ExpireIn = d
567 walk(parent)
202 default:
568 inherit(d, base)
203 return Artifacts{}, fmt.Errorf("onbekend artifacts-veld %q", key)
204 }
569 }
570 delete(visiting, name)
571 d.present["__resolved"] = true
572 inherit(d, &def)
573 }
574 for _, name := range order {
575 walk(name)
205 }
576 }
206 return a, nil
577 return jobs, issues
207}
578}
208
579
209func parseRules(node *yaml.Node) ([]Rule, error) {
580func inherit(child, parent *draft) {
210 if node.Kind != yaml.SequenceNode {
581 if parent == nil {
211 return nil, fmt.Errorf("rules moet een lijst zijn")
582 return
212 }
583 }
213 var rules []Rule
584 copyField := func(key string, apply func()) {
214 for _, item := range node.Content {
585 if child.present[key] || !parent.present[key] {
215 if item.Kind != yaml.MappingNode {
586 return
216 return nil, fmt.Errorf("rule moet een mapping zijn")
217 }
587 }
218 var r Rule
588 apply()
219 for i := 0; i < len(item.Content); i += 2 {
589 child.present[key] = true
220 key := item.Content[i].Value
590 }
221 val := item.Content[i+1]
591 copyField("stage", func() { child.job.Stage = parent.job.Stage })
222 if key != "if" {
592 copyField("image", func() { child.job.Image = parent.job.Image })
223 return nil, fmt.Errorf("rule ondersteunt alleen if, kreeg %q", key)
593 copyField("before_script", func() { child.job.Before = parent.job.Before })
224 }
594 copyField("script", func() { child.job.Script = parent.job.Script })
225 if val.Kind != yaml.ScalarNode || strings.TrimSpace(val.Value) == "" {
595 copyField("after_script", func() { child.job.After = parent.job.After })
226 return nil, fmt.Errorf("rule.if moet tekst zijn")
596 copyField("needs", func() {
227 }
597 child.job.Needs = parent.job.Needs
228 r.If = strings.TrimSpace(val.Value)
598 child.job.Optional = parent.job.Optional
229 }
599 child.job.NeedsSet = parent.job.NeedsSet
230 if r.If == "" {
600 })
231 return nil, fmt.Errorf("rule mist if")
601 copyField("artifacts", func() { child.job.Artifacts = parent.job.Artifacts })
602 copyField("rules", func() { child.job.Rules = parent.job.Rules })
603 copyField("allow_failure", func() { child.job.AllowFailure = parent.job.AllowFailure })
604 copyField("when", func() { child.job.When = parent.job.When })
605 copyField("timeout", func() { child.job.Timeout = parent.job.Timeout })
606 copyField("tags", func() { child.job.Tags = parent.job.Tags })
607 copyField("cache", func() { child.job.Cache = parent.job.Cache })
608 copyField("resource_group", func() { child.job.ResourceGroup = parent.job.ResourceGroup })
609 copyField("only", func() { child.job.Only = parent.job.Only })
610 copyField("except", func() { child.job.Except = parent.job.Except })
611 if len(parent.job.Variables) > 0 {
612 merged := map[string]string{}
613 for k, v := range parent.job.Variables {
614 merged[k] = v
232 }
615 }
233 if _, err := compileRule(r.If); err != nil {
616 for k, v := range child.job.Variables {
234 return nil, err
617 merged[k] = v
235 }
618 }
236 rules = append(rules, r)
619 child.job.Variables = merged
237 }
620 }
238 return rules, nil
239}
621}
240
622
241func (p *Pipeline) validate() error {
623func validate(p *Pipeline) []Issue {
624 var issues []Issue
242 if len(p.Jobs) == 0 {
625 if len(p.Jobs) == 0 {
243 return fmt.Errorf("geen jobs")
626 return []Issue{{Line: 1, Key: "ci.err.noJobs"}}
244 }
627 }
245 seenStage := map[string]bool{}
628 seenStage := map[string]bool{}
246 for _, s := range p.Stages {
629 for _, s := range p.Stages {
247 if s == "" {
630 if s == "" {
248 return fmt.Errorf("lege stage-naam")
631 issues = append(issues, Issue{Line: 1, Key: "ci.err.stageEmpty"})
632 continue
249 }
633 }
250 if seenStage[s] {
634 if seenStage[s] {
251 return fmt.Errorf("dubbele stage %q", s)
635 issues = append(issues, Issue{Line: 1, Key: "ci.err.stageDup", Args: []any{s}})
252 }
636 }
253 seenStage[s] = true
637 seenStage[s] = true
254 }
638 }
255 if len(p.Stages) == 0 {
639 if len(p.Stages) == 0 {
256 var order []string
640 var order []string
257 have := map[string]bool{}
258 for i := range p.Jobs {
641 for i := range p.Jobs {
259 if p.Jobs[i].Stage == "" {
642 if p.Jobs[i].Stage == "" {
260 p.Jobs[i].Stage = "test"
643 p.Jobs[i].Stage = "test"
261 }
644 }
262 if !have[p.Jobs[i].Stage] {
645 if !seenStage[p.Jobs[i].Stage] {
263 have[p.Jobs[i].Stage] = true
646 seenStage[p.Jobs[i].Stage] = true
264 order = append(order, p.Jobs[i].Stage)
647 order = append(order, p.Jobs[i].Stage)
265 }
648 }
266 }
649 }
267 p.Stages = order
650 p.Stages = order
268 for _, s := range p.Stages {
269 seenStage[s] = true
270 }
271 }
651 }
272 names := map[string]bool{}
652 names := map[string]bool{}
273 for i := range p.Jobs {
653 for i := range p.Jobs {
274 j := &p.Jobs[i]
654 j := &p.Jobs[i]
275 if j.Name == "" || strings.ContainsAny(j.Name, " \t\n/\\") {
655 if j.Name == "" || strings.ContainsAny(j.Name, " \t\n/\\") {
276 return fmt.Errorf("ongeldige jobnaam %q", j.Name)
656 issues = append(issues, Issue{Line: 1, Key: "ci.err.jobName", Args: []any{j.Name}})
277 }
657 continue
278 if names[j.Name] {
279 return fmt.Errorf("dubbele job %q", j.Name)
280 }
658 }
281 names[j.Name] = true
659 names[j.Name] = true
282 if j.Stage == "" {
660 if j.Stage == "" {
283 return fmt.Errorf("job %s mist stage", j.Name)
661 issues = append(issues, Issue{Line: 1, Key: "ci.err.jobStage", Args: []any{j.Name}})
284 }
662 } else if !seenStage[j.Stage] {
285 if !seenStage[j.Stage] {
663 issues = append(issues, Issue{Line: 1, Key: "ci.err.stageUnknown", Args: []any{j.Name, j.Stage}})
286 return fmt.Errorf("job %s gebruikt onbekende stage %q", j.Name, j.Stage)
287 }
664 }
288 if j.Image == "" {
665 if j.Image == "" {
289 j.Image = p.Image
666 j.Image = p.Image
290 }
667 }
291 if strings.TrimSpace(j.Image) == "" || strings.ContainsAny(j.Image, " \t\n") {
668 if strings.TrimSpace(j.Image) == "" || strings.ContainsAny(j.Image, " \t\n") {
292 return fmt.Errorf("job %s mist een image", j.Name)
669 issues = append(issues, Issue{Line: 1, Key: "ci.err.noImage", Args: []any{j.Name}})
293 }
670 }
294 if len(j.Script) == 0 {
671 if len(j.Script) == 0 {
295 return fmt.Errorf("job %s mist script", j.Name)
672 issues = append(issues, Issue{Line: 1, Key: "ci.err.noScript", Args: []any{j.Name}})
296 }
673 }
297 for _, line := range j.Script {
674 for _, line := range j.Script {
298 if strings.TrimSpace(line) == "" {
675 if strings.TrimSpace(line) == "" {
299 return fmt.Errorf("job %s heeft een lege scriptregel", j.Name)
676 issues = append(issues, Issue{Line: 1, Key: "ci.err.emptyLine", Args: []any{j.Name}})
300 }
677 }
301 }
678 }
302 for _, path := range j.Artifacts.Paths {
679 for _, path := range j.Artifacts.Paths {
303 if err := safeRelPath(path); err != nil {
680 if iss := safeRel(path); iss.Key != "" {
304 return fmt.Errorf("job %s artifact: %w", j.Name, err)
681 issues = append(issues, iss)
305 }
682 }
306 }
683 }
684 for _, path := range j.Cache.Paths {
685 if iss := safeRel(path); iss.Key != "" {
686 issues = append(issues, iss)
687 }
688 }
689 if j.When == "" {
690 j.When = "on_success"
691 }
307 }
692 }
308 for _, j := range p.Jobs {
693 for _, j := range p.Jobs {
309 for _, need := range j.Needs {
694 for _, need := range j.Needs {
310 if !names[need] {
695 if !names[need] && !contains(j.Optional, need) {
311 return fmt.Errorf("job %s needs onbekende job %q", j.Name, need)
696 issues = append(issues, Issue{Line: 1, Key: "ci.err.needUnknown", Args: []any{j.Name, need}})
312 }
697 }
313 if need == j.Name {
698 if need == j.Name {
314 return fmt.Errorf("job %s kan niet van zichzelf afhangen", j.Name)
699 issues = append(issues, Issue{Line: 1, Key: "ci.err.needSelf", Args: []any{j.Name}})
315 }
700 }
316 }
701 }
317 }
702 }
318 if err := detectCycles(p.Jobs); err != nil {
703 if iss := detectCycles(p.Jobs); iss.Key != "" {
319 return err
704 issues = append(issues, iss)
320 }
705 }
321 return nil
706 return issues
322}
707}
323
708
324func detectCycles(jobs []Job) error {
709func detectCycles(jobs []Job) Issue {
325 idx := map[string]int{}
710 idx := map[string]int{}
326 for i, j := range jobs {
711 for i, j := range jobs {
327 idx[j.Name] = i
712 idx[j.Name] = i
@@ -332,237 +717,382 @@ func detectCycles(jobs []Job) error {
332 black = 2
717 black = 2
333 )
718 )
334 color := make([]int, len(jobs))
719 color := make([]int, len(jobs))
335 var visit func(int) error
720 var visit func(int) Issue
336 visit = func(i int) error {
721 visit = func(i int) Issue {
337 if color[i] == gray {
722 if color[i] == gray {
338 return fmt.Errorf("cyclus in needs bij job %s", jobs[i].Name)
723 return Issue{Line: 1, Key: "ci.err.cycle", Args: []any{jobs[i].Name}}
339 }
724 }
340 if color[i] == black {
725 if color[i] == black {
341 return nil
726 return Issue{}
342 }
727 }
343 color[i] = gray
728 color[i] = gray
344 for _, n := range jobs[i].Needs {
729 for _, n := range jobs[i].Needs {
345 if err := visit(idx[n]); err != nil {
730 target, ok := idx[n]
346 return err
731 if !ok {
732 continue
733 }
734 if iss := visit(target); iss.Key != "" {
735 return iss
347 }
736 }
348 }
737 }
349 color[i] = black
738 color[i] = black
350 return nil
739 return Issue{}
351 }
740 }
352 for i := range jobs {
741 for i := range jobs {
353 if err := visit(i); err != nil {
742 if iss := visit(i); iss.Key != "" {
354 return err
743 return iss
355 }
744 }
356 }
745 }
357 return nil
746 return Issue{}
358}
747}
359
748
360// ActiveJobs filtert jobs waarvan de rules niet kloppen.
749func parseArtifacts(node *yaml.Node) (Artifacts, []Issue) {
361func (p Pipeline) ActiveJobs(env map[string]string) ([]Job, error) {
750 if node.Kind != yaml.MappingNode {
362 var out []Job
751 return Artifacts{}, []Issue{{Line: node.Line, Key: "ci.err.artifacts"}}
363 for _, j := range p.Jobs {
752 }
364 ok, err := j.matches(env)
753 var a Artifacts
365 if err != nil {
754 var issues []Issue
366 return nil, fmt.Errorf("job %s: %w", j.Name, err)
755 for i := 0; i < len(node.Content); i += 2 {
367 }
756 key := node.Content[i].Value
368 if ok {
757 val := node.Content[i+1]
369 out = append(out, j)
758 switch key {
759 case "paths":
760 paths, iss := scalarList(val, "ci.err.list")
761 issues = append(issues, iss...)
762 a.Paths = paths
763 case "expire_in":
764 if val.Kind != yaml.ScalarNode {
765 issues = append(issues, Issue{Line: val.Line, Key: "ci.err.expire"})
766 continue
767 }
768 d, err := parseExpire(val.Value)
769 if err != nil {
770 issues = append(issues, Issue{Line: val.Line, Key: "ci.err.expireBad", Args: []any{val.Value}})
771 continue
772 }
773 a.ExpireIn = d
774 default:
775 issues = append(issues, Issue{Line: node.Content[i].Line, Key: "ci.warn.ignored", Args: []any{"artifacts." + key}, Warning: true})
370 }
776 }
371 }
777 }
372 return out, nil
778 return a, issues
373}
779}
374
780
375func (j Job) matches(env map[string]string) (bool, error) {
781func parseCache(node *yaml.Node) (Cache, []Issue) {
376 if len(j.Rules) == 0 {
782 if node.Kind != yaml.MappingNode {
377 return true, nil
783 return Cache{}, []Issue{{Line: node.Line, Key: "ci.err.cache"}}
378 }
784 }
379 for _, r := range j.Rules {
785 var c Cache
380 ok, err := evalRule(r.If, env)
786 var issues []Issue
381 if err != nil {
787 for i := 0; i < len(node.Content); i += 2 {
382 return false, err
788 key := node.Content[i].Value
383 }
789 val := node.Content[i+1]
384 if ok {
790 switch key {
385 return true, nil
791 case "paths":
386 }
792 paths, iss := scalarList(val, "ci.err.list")
387 }
793 issues = append(issues, iss...)
388 return false, nil
794 c.Paths = paths
389}
795 case "key":
390
796 if val.Kind != yaml.ScalarNode {
391// Gate beslist of een job mag starten.
797 issues = append(issues, Issue{Line: val.Line, Key: "ci.err.cacheKey"})
392// "ready" betekent dat alle voorgangers klaar zijn.
798 continue
393// "wait" betekent dat er nog een voorganger loopt of in de wachtrij staat.
394// "skip" betekent dat een voorganger mislukte zonder allow_failure.
395//
396// NeedsSet volgt de genoemde jobs, ook als de lijst leeg is.
397// Zonder needs wacht de job op alle jobs in eerdere stages.
398func Gate(jobs []Job, status map[string]string, job Job) string {
399 byName := map[string]Job{}
400 for _, j := range jobs {
401 byName[j.Name] = j
402 }
403 var deps []Job
404 if job.NeedsSet {
405 for _, n := range job.Needs {
406 deps = append(deps, byName[n])
407 }
408 } else {
409 stagePos := map[string]int{}
410 order := 0
411 for _, j := range jobs {
412 if _, ok := stagePos[j.Stage]; !ok {
413 stagePos[j.Stage] = order
414 order++
415 }
799 }
800 c.Key = val.Value
801 default:
802 issues = append(issues, Issue{Line: node.Content[i].Line, Key: "ci.warn.ignored", Args: []any{"cache." + key}, Warning: true})
416 }
803 }
417 my := stagePos[job.Stage]
804 }
418 for _, j := range jobs {
805 return c, issues
419 if stagePos[j.Stage] < my {
806}
420 deps = append(deps, j)
807
808func parseRules(node *yaml.Node) ([]Rule, []Issue) {
809 if node.Kind != yaml.SequenceNode {
810 return nil, []Issue{{Line: node.Line, Key: "ci.err.rulesList"}}
811 }
812 var rules []Rule
813 var issues []Issue
814 for _, item := range node.Content {
815 if item.Kind != yaml.MappingNode {
816 issues = append(issues, Issue{Line: item.Line, Key: "ci.err.ruleMap"})
817 continue
818 }
819 var r Rule
820 for i := 0; i < len(item.Content); i += 2 {
821 key := item.Content[i].Value
822 val := item.Content[i+1]
823 switch key {
824 case "if":
825 if val.Kind != yaml.ScalarNode || strings.TrimSpace(val.Value) == "" {
826 issues = append(issues, Issue{Line: val.Line, Key: "ci.err.ruleIf"})
827 continue
828 }
829 r.If = strings.TrimSpace(val.Value)
830 if _, err := compileRule(r.If); err != nil {
831 if ie, ok := err.(i18n.Error); ok {
832 issues = append(issues, Issue{Line: val.Line, Key: ie.Key, Args: ie.Args})
833 } else {
834 issues = append(issues, Issue{Line: val.Line, Key: "ci.err.ruleExpr", Args: []any{r.If}})
835 }
836 }
837 case "changes":
838 list, iss := scalarList(val, "ci.err.list")
839 issues = append(issues, iss...)
840 r.Changes = list
841 case "exists":
842 list, iss := scalarList(val, "ci.err.list")
843 issues = append(issues, iss...)
844 r.Exists = list
845 case "when":
846 w, iss := whenOf(val, true)
847 issues = append(issues, iss...)
848 r.When = w
849 default:
850 issues = append(issues, Issue{Line: item.Content[i].Line, Key: "ci.warn.ignored", Args: []any{"rules." + key}, Warning: true})
421 }
851 }
422 }
852 }
423 }
853 if r.If == "" && len(r.Changes) == 0 && len(r.Exists) == 0 && r.When == "" {
424 for _, d := range deps {
854 issues = append(issues, Issue{Line: item.Line, Key: "ci.err.ruleEmpty"})
425 switch status[d.Name] {
426 case "success", "skipped":
427 continue
855 continue
428 case "failed":
856 }
429 if d.AllowFailure {
857 rules = append(rules, r)
858 }
859 return rules, issues
860}
861
862func parseNeeds(node *yaml.Node) (needs, optional []string, set bool, issues []Issue) {
863 set = true
864 if node.Kind != yaml.SequenceNode {
865 return nil, nil, true, []Issue{{Line: node.Line, Key: "ci.err.needs"}}
866 }
867 for _, item := range node.Content {
868 switch item.Kind {
869 case yaml.ScalarNode:
870 needs = append(needs, item.Value)
871 case yaml.MappingNode:
872 var name string
873 var opt bool
874 for i := 0; i < len(item.Content); i += 2 {
875 switch item.Content[i].Value {
876 case "job":
877 if item.Content[i+1].Kind == yaml.ScalarNode {
878 name = item.Content[i+1].Value
879 }
880 case "optional":
881 b, iss := parseBool(item.Content[i+1])
882 issues = append(issues, iss...)
883 opt = b
884 default:
885 issues = append(issues, Issue{Line: item.Content[i].Line, Key: "ci.warn.ignored", Args: []any{"needs." + item.Content[i].Value}, Warning: true})
886 }
887 }
888 if name == "" {
889 issues = append(issues, Issue{Line: item.Line, Key: "ci.err.needs"})
430 continue
890 continue
431 }
891 }
432 return "skip"
892 needs = append(needs, name)
893 if opt {
894 optional = append(optional, name)
895 }
433 default:
896 default:
434 return "wait"
897 issues = append(issues, Issue{Line: item.Line, Key: "ci.err.needs"})
435 }
898 }
436 }
899 }
437 return "ready"
900 if needs == nil {
901 needs = []string{}
902 }
903 return needs, optional, set, issues
438}
904}
439
905
440var (
906func imageOf(node *yaml.Node) (string, []Issue) {
441 reEq = regexp.MustCompile(`^\$([A-Za-z_][A-Za-z0-9_]*)\s*==\s*"([^"]*)"$`)
907 switch node.Kind {
442 reNe = regexp.MustCompile(`^\$([A-Za-z_][A-Za-z0-9_]*)\s*!=\s*"([^"]*)"$`)
908 case yaml.ScalarNode:
443 reRe = regexp.MustCompile(`^\$([A-Za-z_][A-Za-z0-9_]*)\s*=~\s*/(.+)/$`)
909 return node.Value, nil
444)
910 case yaml.MappingNode:
445
911 for i := 0; i < len(node.Content); i += 2 {
446type compiledRule struct {
912 if node.Content[i].Value == "name" && node.Content[i+1].Kind == yaml.ScalarNode {
447 kind string
913 return node.Content[i+1].Value, nil
448 name string
914 }
449 arg string
915 }
450 re *regexp.Regexp
916 return "", []Issue{{Line: node.Line, Key: "ci.err.image"}}
917 default:
918 return "", []Issue{{Line: node.Line, Key: "ci.err.image"}}
919 }
451}
920}
452
921
453func compileRule(expr string) (compiledRule, error) {
922func whenOf(node *yaml.Node, allowNever bool) (string, []Issue) {
454 if m := reEq.FindStringSubmatch(expr); m != nil {
923 if node.Kind != yaml.ScalarNode {
455 return compiledRule{kind: "eq", name: m[1], arg: m[2]}, nil
924 return "", []Issue{{Line: node.Line, Key: "ci.err.when", Args: []any{""}}}
456 }
457 if m := reNe.FindStringSubmatch(expr); m != nil {
458 return compiledRule{kind: "ne", name: m[1], arg: m[2]}, nil
459 }
925 }
460 if m := reRe.FindStringSubmatch(expr); m != nil {
926 switch node.Value {
461 re, err := regexp.Compile(m[2])
927 case "on_success", "on_failure", "always", "manual":
462 if err != nil {
928 return node.Value, nil
463 return compiledRule{}, fmt.Errorf("ongeldige regex in rule %q: %w", expr, err)
929 case "never":
930 if allowNever {
931 return node.Value, nil
464 }
932 }
465 return compiledRule{kind: "re", name: m[1], re: re}, nil
466 }
933 }
467 return compiledRule{}, fmt.Errorf("rule %q wordt niet begrepen", expr)
934 return "", []Issue{{Line: node.Line, Key: "ci.err.when", Args: []any{node.Value}}}
468}
935}
469
936
470func evalRule(expr string, env map[string]string) (bool, error) {
937func durationOf(node *yaml.Node) (time.Duration, []Issue) {
471 c, err := compileRule(expr)
938 if node.Kind != yaml.ScalarNode {
939 return 0, []Issue{{Line: node.Line, Key: "ci.err.timeout", Args: []any{""}}}
940 }
941 d, err := parseExpire(node.Value)
472 if err != nil {
942 if err != nil {
473 return false, err
943 if strings.HasSuffix(node.Value, "s") {
474 }
944 n, nerr := strconv.Atoi(strings.TrimSuffix(node.Value, "s"))
475 val := env[c.name]
945 if nerr == nil && n >= 0 {
476 switch c.kind {
946 return time.Duration(n) * time.Second, nil
477 case "eq":
947 }
478 return val == c.arg, nil
948 }
479 case "ne":
949 return 0, []Issue{{Line: node.Line, Key: "ci.err.timeout", Args: []any{node.Value}}}
480 return val != c.arg, nil
481 case "re":
482 return c.re.MatchString(val), nil
483 default:
484 return false, fmt.Errorf("interne rulefout")
485 }
950 }
951 return d, nil
486}
952}
487
953
488func parseExpire(s string) (time.Duration, error) {
954func extendsOf(node *yaml.Node) ([]string, []Issue) {
489 s = strings.TrimSpace(s)
955 if node.Kind == yaml.ScalarNode && strings.TrimSpace(node.Value) != "" {
490 if s == "" || s == "never" {
956 return []string{node.Value}, nil
491 return 0, nil
492 }
957 }
493 if len(s) < 2 {
958 return scalarList(node, "ci.err.extendsList")
494 return 0, fmt.Errorf("expire_in %q is ongeldig", s)
959}
495 }
960
496 n, err := strconv.Atoi(s[:len(s)-1])
961func scriptList(node *yaml.Node) ([]string, []Issue) {
497 if err != nil || n < 0 {
962 if node.Kind == yaml.ScalarNode {
498 return 0, fmt.Errorf("expire_in %q is ongeldig", s)
963 if strings.TrimSpace(node.Value) == "" {
964 return nil, []Issue{{Line: node.Line, Key: "ci.err.script"}}
965 }
966 return splitLines(node.Value), nil
499 }
967 }
500 switch s[len(s)-1] {
968 return scalarList(node, "ci.err.script")
501 case 'm':
969}
502 return time.Duration(n) * time.Minute, nil
970
503 case 'h':
971func splitLines(s string) []string {
504 return time.Duration(n) * time.Hour, nil
972 parts := strings.Split(s, "\n")
505 case 'd':
973 var out []string
506 return time.Duration(n) * 24 * time.Hour, nil
974 for _, p := range parts {
507 case 'w':
975 if strings.TrimSpace(p) == "" {
508 return time.Duration(n) * 7 * 24 * time.Hour, nil
976 continue
509 default:
977 }
510 return 0, fmt.Errorf("expire_in %q is ongeldig", s)
978 out = append(out, p)
511 }
979 }
980 return out
512}
981}
513
982
514func scalarList(node *yaml.Node) ([]string, error) {
983func scalarList(node *yaml.Node, key string) ([]string, []Issue) {
515 if node.Kind != yaml.SequenceNode {
984 if node.Kind != yaml.SequenceNode {
516 return nil, fmt.Errorf("verwacht een lijst")
985 return nil, []Issue{{Line: node.Line, Key: key}}
517 }
986 }
518 var out []string
987 var out []string
988 var issues []Issue
519 for _, item := range node.Content {
989 for _, item := range node.Content {
520 if item.Kind != yaml.ScalarNode {
990 if item.Kind != yaml.ScalarNode {
521 return nil, fmt.Errorf("lijst mag alleen tekst bevatten")
991 issues = append(issues, Issue{Line: item.Line, Key: "ci.err.listText"})
992 continue
522 }
993 }
523 out = append(out, item.Value)
994 out = append(out, item.Value)
524 }
995 }
525 return out, nil
996 return out, issues
526}
997}
527
998
528func scalarMap(node *yaml.Node) (map[string]string, error) {
999func scalarMap(node *yaml.Node) (map[string]string, []Issue) {
529 if node.Kind != yaml.MappingNode {
1000 if node.Kind != yaml.MappingNode {
530 return nil, fmt.Errorf("verwacht een mapping")
1001 return nil, []Issue{{Line: node.Line, Key: "ci.err.map"}}
531 }
1002 }
532 out := map[string]string{}
1003 out := map[string]string{}
1004 var issues []Issue
533 for i := 0; i < len(node.Content); i += 2 {
1005 for i := 0; i < len(node.Content); i += 2 {
534 k := node.Content[i]
1006 k := node.Content[i]
535 v := node.Content[i+1]
1007 v := node.Content[i+1]
536 if k.Kind != yaml.ScalarNode || v.Kind != yaml.ScalarNode {
1008 if k.Kind != yaml.ScalarNode {
537 return nil, fmt.Errorf("variabelen moeten tekst zijn")
1009 issues = append(issues, Issue{Line: k.Line, Key: "ci.err.varText"})
1010 continue
1011 }
1012 switch v.Kind {
1013 case yaml.ScalarNode:
1014 out[k.Value] = v.Value
1015 case yaml.MappingNode:
1016 val, iss := mappingValue(v)
1017 issues = append(issues, iss...)
1018 if val != "" || len(iss) == 0 {
1019 out[k.Value] = val
1020 }
1021 default:
1022 issues = append(issues, Issue{Line: v.Line, Key: "ci.err.varText"})
1023 }
1024 }
1025 return out, issues
1026}
1027
1028func mappingValue(node *yaml.Node) (string, []Issue) {
1029 for i := 0; i < len(node.Content); i += 2 {
1030 if node.Content[i].Value == "value" && node.Content[i+1].Kind == yaml.ScalarNode {
1031 return node.Content[i+1].Value, nil
538 }
1032 }
539 out[k.Value] = v.Value
540 }
1033 }
541 return out, nil
1034 return "", []Issue{{Line: node.Line, Key: "ci.err.varText"}}
542}
1035}
543
1036
544func parseBool(node *yaml.Node) (bool, error) {
1037func parseBool(node *yaml.Node) (bool, []Issue) {
545 if node.Kind != yaml.ScalarNode {
1038 if node.Kind != yaml.ScalarNode {
546 return false, fmt.Errorf("verwacht ja of nee")
1039 return false, []Issue{{Line: node.Line, Key: "ci.err.bool"}}
547 }
1040 }
548 switch strings.ToLower(node.Value) {
1041 switch strings.ToLower(node.Value) {
549 case "true", "yes", "ja":
1042 case "true", "yes":
550 return true, nil
1043 return true, nil
551 case "false", "no", "nee":
1044 case "false", "no":
552 return false, nil
1045 return false, nil
553 default:
1046 default:
554 return false, fmt.Errorf("verwacht true of false")
1047 return false, []Issue{{Line: node.Line, Key: "ci.err.bool"}}
555 }
1048 }
556}
1049}
557
1050
558func safeRelPath(p string) error {
1051func parseExpire(s string) (time.Duration, error) {
1052 s = strings.TrimSpace(s)
1053 if s == "" || s == "never" {
1054 return 0, nil
1055 }
1056 if len(s) < 2 {
1057 return 0, fmt.Errorf("expire")
1058 }
1059 n, err := strconv.Atoi(s[:len(s)-1])
1060 if err != nil || n < 0 {
1061 return 0, fmt.Errorf("expire")
1062 }
1063 switch s[len(s)-1] {
1064 case 's':
1065 return time.Duration(n) * time.Second, nil
1066 case 'm':
1067 return time.Duration(n) * time.Minute, nil
1068 case 'h':
1069 return time.Duration(n) * time.Hour, nil
1070 case 'd':
1071 return time.Duration(n) * 24 * time.Hour, nil
1072 case 'w':
1073 return time.Duration(n) * 7 * 24 * time.Hour, nil
1074 default:
1075 return 0, fmt.Errorf("expire")
1076 }
1077}
1078
1079func safeRel(p string) Issue {
559 if p == "" || strings.HasPrefix(p, "/") || strings.Contains(p, "\\") {
1080 if p == "" || strings.HasPrefix(p, "/") || strings.Contains(p, "\\") {
560 return fmt.Errorf("pad %q moet relatief zijn", p)
1081 return Issue{Line: 1, Key: "ci.err.rel", Args: []any{p}}
561 }
1082 }
562 for _, part := range strings.Split(p, "/") {
1083 for _, part := range strings.Split(p, "/") {
563 if part == ".." {
1084 if part == ".." {
564 return fmt.Errorf("pad %q mag niet buiten de workspace", p)
1085 return Issue{Line: 1, Key: "ci.err.dotdot", Args: []any{p}}
1086 }
1087 }
1088 return Issue{}
1089}
1090
1091func hasError(issues []Issue) bool {
1092 for _, is := range issues {
1093 if !is.Warning {
1094 return true
565 }
1095 }
566 }
1096 }
567 return nil
1097 return false
568}
1098}
internal/ci/spec_test.go added +216 −0
@@ -0,0 +1,216 @@
1 package ci
2
3 import (
4 "strings"
5 "testing"
6 )
7
8 func TestGitLabDocument(t *testing.T) {
9 raw := []byte(`
10 stages: [test, deploy]
11 default:
12 image: alpine:3.20
13 before_script: ["echo prep"]
14 variables:
15 MODE: {value: "ci"}
16 .php:
17 image: php:8.3
18 stage: test
19 php_tests:
20 extends: .php
21 script: ["phpunit"]
22 rules:
23 - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
24 changes: ["**/*.php"]
25 - when: never
26 frontend_checks:
27 stage: test
28 script: ["node check"]
29 rules:
30 - if: $CI_COMMIT_TAG == null
31 exists: ["package.json"]
32 allow_failure: true
33 cache:
34 key: modules
35 paths: ["node_modules"]
36 deploy:
37 stage: deploy
38 when: manual
39 resource_group: production
40 timeout: 30m
41 tags: [deploy]
42 only: ["master", "tags"]
43 script: ["echo deploy"]
44 needs:
45 - job: php_tests
46 optional: true
47 `)
48 p, err := Parse(raw)
49 if err != nil {
50 t.Fatal(err)
51 }
52 if strings.Join(p.Stages, ",") != "test,deploy" {
53 t.Fatalf("stages %v", p.Stages)
54 }
55 if p.Variables["MODE"] != "ci" {
56 t.Fatal(p.Variables)
57 }
58 by := map[string]Job{}
59 for _, j := range p.Jobs {
60 by[j.Name] = j
61 }
62 if _, ok := by[".php"]; ok {
63 t.Fatal("verborgen job hoort niet te draaien")
64 }
65 php := by["php_tests"]
66 if php.Image != "php:8.3" || php.Stage != "test" || php.Before[0] != "echo prep" {
67 t.Fatalf("extends %+v", php)
68 }
69 front := by["frontend_checks"]
70 if !front.AllowFailure || front.Cache.Key != "modules" || front.Cache.Paths[0] != "node_modules" {
71 t.Fatalf("frontend %+v", front)
72 }
73 dep := by["deploy"]
74 if dep.When != "manual" || dep.ResourceGroup != "production" || dep.Timeout.Minutes() != 30 {
75 t.Fatalf("deploy %+v", dep)
76 }
77 if len(dep.Needs) != 1 || dep.Optional[0] != "php_tests" {
78 t.Fatalf("needs %+v opt %+v", dep.Needs, dep.Optional)
79 }
80
81 env := map[string]string{"CI_COMMIT_BRANCH": "master", "CI_DEFAULT_BRANCH": "master", "CI_COMMIT_REF_NAME": "master"}
82 active, err := p.Select(Eval{Env: env, Changed: []string{"src/App.php"}, Files: []string{"package.json"}})
83 if err != nil {
84 t.Fatal(err)
85 }
86 names := map[string]bool{}
87 for _, j := range active {
88 names[j.Name] = true
89 }
90 if !names["php_tests"] || !names["frontend_checks"] || !names["deploy"] {
91 t.Fatalf("actief %v", names)
92 }
93 active, err = p.Select(Eval{Env: map[string]string{"CI_COMMIT_BRANCH": "feature", "CI_DEFAULT_BRANCH": "master", "CI_COMMIT_REF_NAME": "feature"}, Changed: []string{"README"}, Files: []string{}})
94 if err != nil {
95 t.Fatal(err)
96 }
97 names = map[string]bool{}
98 for _, j := range active {
99 names[j.Name] = true
100 if j.Name == "deploy" && j.When != "manual" {
101 t.Fatal(j.When)
102 }
103 }
104 if names["php_tests"] || names["frontend_checks"] {
105 t.Fatalf("feature hoort die jobs over te slaan: %v", names)
106 }
107 if names["deploy"] {
108 t.Fatal("only master/tags")
109 }
110 }
111
112 func TestIncludeLocalAndWarnings(t *testing.T) {
113 files := map[string]string{
114 "ci/base.yml": "jobs:\n unit:\n image: alpine\n script: ['echo unit']\n",
115 }
116 raw := []byte("include:\n - local: ci/base.yml\nservices: [db]\njobs:\n unit:\n script: ['echo replaced']\n")
117 p, err := ParseIn(raw, func(path string) ([]byte, error) { return []byte(files[path]), nil })
118 if err != nil {
119 t.Fatal(err)
120 }
121 if len(p.Jobs) != 1 || p.Jobs[0].Script[0] != "echo replaced" || p.Jobs[0].Image != "alpine" {
122 t.Fatalf("%+v", p.Jobs)
123 }
124 if len(p.Warnings) != 1 || p.Warnings[0].Key != "ci.warn.ignored" {
125 t.Fatalf("warnings %+v", p.Warnings)
126 }
127 _, issues := Lint([]byte("include:\n - project: other\njobs:\n a:\n image: alpine\n script: ['x']\n"), nil)
128 found := false
129 for _, is := range issues {
130 if is.Key == "ci.err.includeRemote" {
131 found = true
132 }
133 }
134 if !found {
135 t.Fatalf("issues %+v", issues)
136 }
137 }
138
139 func TestWhenAndAllowFailure(t *testing.T) {
140 p, err := Parse([]byte(`
141 stages: [een, twee]
142 jobs:
143 a:
144 stage: een
145 image: alpine
146 script: ['echo a']
147 allow_failure: true
148 b:
149 stage: twee
150 image: alpine
151 script: ['echo b']
152 c:
153 stage: twee
154 image: alpine
155 when: always
156 script: ['echo c']
157 d:
158 stage: twee
159 image: alpine
160 when: on_failure
161 script: ['echo d']
162 `))
163 if err != nil {
164 t.Fatal(err)
165 }
166 st := map[string]string{"a": "failed", "b": "queued", "c": "queued", "d": "queued"}
167 if Gate(p.Jobs, st, p.Jobs[1]) != "ready" {
168 t.Fatal("allow_failure houdt de volgende stage niet tegen")
169 }
170 if Gate(p.Jobs, st, p.Jobs[2]) != "ready" || Gate(p.Jobs, st, p.Jobs[3]) != "ready" {
171 t.Fatal("always en on_failure na een mislukking")
172 }
173 ok := map[string]string{"a": "success", "b": "queued", "c": "queued", "d": "queued"}
174 if Gate(p.Jobs, ok, p.Jobs[3]) != "skip" {
175 t.Fatal("on_failure slaat over na succes")
176 }
177 hard, err := Parse([]byte(`
178 stages: [een, twee]
179 jobs:
180 a:
181 stage: een
182 image: alpine
183 script: ['echo a']
184 b:
185 stage: twee
186 image: alpine
187 script: ['echo b']
188 `))
189 if err != nil {
190 t.Fatal(err)
191 }
192 if Gate(hard.Jobs, map[string]string{"a": "failed", "b": "queued"}, hard.Jobs[1]) != "skip" {
193 t.Fatal("harde fout slaat de rest over")
194 }
195 }
196
197 func TestMaskAndVariables(t *testing.T) {
198 if got := Mask("token=supersecret value", []string{"supersecret", "ab"}); got != "token=[MASKED] value" {
199 t.Fatal(got)
200 }
201 env := Variables(Ref{SHA: "abcdef1234567890", FullRef: "refs/tags/v1.2.0", Title: "Uitgave", DefaultBranch: "master", Source: "push", ProjectPath: "tim/app", ProjectURL: "https://git.example/tim/app", APIURL: "https://git.example/api/v1", PipelineID: 9})
202 if env["CI_COMMIT_TAG"] != "v1.2.0" || env["CI_COMMIT_BRANCH"] != "" || env["CI_COMMIT_SHORT_SHA"] != "abcdef12" {
203 t.Fatalf("%v", env)
204 }
205 if env["CI_DEFAULT_BRANCH"] != "master" || env["CI_PROJECT_NAME"] != "app" || env["CI_PIPELINE_SOURCE"] != "push" {
206 t.Fatalf("%v", env)
207 }
208 ok, err := evalRule(`$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH`, map[string]string{"CI_COMMIT_BRANCH": "master", "CI_DEFAULT_BRANCH": "master"})
209 if err != nil || !ok {
210 t.Fatal(err, ok)
211 }
212 ok, err = evalRule(`$CI_COMMIT_TAG == null`, map[string]string{"CI_COMMIT_TAG": ""})
213 if err != nil || !ok {
214 t.Fatal("null", err, ok)
215 }
216 }
@@ -0,0 +1,216 @@
1package ci
2
3import (
4 "strings"
5 "testing"
6)
7
8func TestGitLabDocument(t *testing.T) {
9 raw := []byte(`
10stages: [test, deploy]
11default:
12 image: alpine:3.20
13 before_script: ["echo prep"]
14variables:
15 MODE: {value: "ci"}
16.php:
17 image: php:8.3
18 stage: test
19php_tests:
20 extends: .php
21 script: ["phpunit"]
22 rules:
23 - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
24 changes: ["**/*.php"]
25 - when: never
26frontend_checks:
27 stage: test
28 script: ["node check"]
29 rules:
30 - if: $CI_COMMIT_TAG == null
31 exists: ["package.json"]
32 allow_failure: true
33 cache:
34 key: modules
35 paths: ["node_modules"]
36deploy:
37 stage: deploy
38 when: manual
39 resource_group: production
40 timeout: 30m
41 tags: [deploy]
42 only: ["master", "tags"]
43 script: ["echo deploy"]
44 needs:
45 - job: php_tests
46 optional: true
47`)
48 p, err := Parse(raw)
49 if err != nil {
50 t.Fatal(err)
51 }
52 if strings.Join(p.Stages, ",") != "test,deploy" {
53 t.Fatalf("stages %v", p.Stages)
54 }
55 if p.Variables["MODE"] != "ci" {
56 t.Fatal(p.Variables)
57 }
58 by := map[string]Job{}
59 for _, j := range p.Jobs {
60 by[j.Name] = j
61 }
62 if _, ok := by[".php"]; ok {
63 t.Fatal("verborgen job hoort niet te draaien")
64 }
65 php := by["php_tests"]
66 if php.Image != "php:8.3" || php.Stage != "test" || php.Before[0] != "echo prep" {
67 t.Fatalf("extends %+v", php)
68 }
69 front := by["frontend_checks"]
70 if !front.AllowFailure || front.Cache.Key != "modules" || front.Cache.Paths[0] != "node_modules" {
71 t.Fatalf("frontend %+v", front)
72 }
73 dep := by["deploy"]
74 if dep.When != "manual" || dep.ResourceGroup != "production" || dep.Timeout.Minutes() != 30 {
75 t.Fatalf("deploy %+v", dep)
76 }
77 if len(dep.Needs) != 1 || dep.Optional[0] != "php_tests" {
78 t.Fatalf("needs %+v opt %+v", dep.Needs, dep.Optional)
79 }
80
81 env := map[string]string{"CI_COMMIT_BRANCH": "master", "CI_DEFAULT_BRANCH": "master", "CI_COMMIT_REF_NAME": "master"}
82 active, err := p.Select(Eval{Env: env, Changed: []string{"src/App.php"}, Files: []string{"package.json"}})
83 if err != nil {
84 t.Fatal(err)
85 }
86 names := map[string]bool{}
87 for _, j := range active {
88 names[j.Name] = true
89 }
90 if !names["php_tests"] || !names["frontend_checks"] || !names["deploy"] {
91 t.Fatalf("actief %v", names)
92 }
93 active, err = p.Select(Eval{Env: map[string]string{"CI_COMMIT_BRANCH": "feature", "CI_DEFAULT_BRANCH": "master", "CI_COMMIT_REF_NAME": "feature"}, Changed: []string{"README"}, Files: []string{}})
94 if err != nil {
95 t.Fatal(err)
96 }
97 names = map[string]bool{}
98 for _, j := range active {
99 names[j.Name] = true
100 if j.Name == "deploy" && j.When != "manual" {
101 t.Fatal(j.When)
102 }
103 }
104 if names["php_tests"] || names["frontend_checks"] {
105 t.Fatalf("feature hoort die jobs over te slaan: %v", names)
106 }
107 if names["deploy"] {
108 t.Fatal("only master/tags")
109 }
110}
111
112func TestIncludeLocalAndWarnings(t *testing.T) {
113 files := map[string]string{
114 "ci/base.yml": "jobs:\n unit:\n image: alpine\n script: ['echo unit']\n",
115 }
116 raw := []byte("include:\n - local: ci/base.yml\nservices: [db]\njobs:\n unit:\n script: ['echo replaced']\n")
117 p, err := ParseIn(raw, func(path string) ([]byte, error) { return []byte(files[path]), nil })
118 if err != nil {
119 t.Fatal(err)
120 }
121 if len(p.Jobs) != 1 || p.Jobs[0].Script[0] != "echo replaced" || p.Jobs[0].Image != "alpine" {
122 t.Fatalf("%+v", p.Jobs)
123 }
124 if len(p.Warnings) != 1 || p.Warnings[0].Key != "ci.warn.ignored" {
125 t.Fatalf("warnings %+v", p.Warnings)
126 }
127 _, issues := Lint([]byte("include:\n - project: other\njobs:\n a:\n image: alpine\n script: ['x']\n"), nil)
128 found := false
129 for _, is := range issues {
130 if is.Key == "ci.err.includeRemote" {
131 found = true
132 }
133 }
134 if !found {
135 t.Fatalf("issues %+v", issues)
136 }
137}
138
139func TestWhenAndAllowFailure(t *testing.T) {
140 p, err := Parse([]byte(`
141stages: [een, twee]
142jobs:
143 a:
144 stage: een
145 image: alpine
146 script: ['echo a']
147 allow_failure: true
148 b:
149 stage: twee
150 image: alpine
151 script: ['echo b']
152 c:
153 stage: twee
154 image: alpine
155 when: always
156 script: ['echo c']
157 d:
158 stage: twee
159 image: alpine
160 when: on_failure
161 script: ['echo d']
162`))
163 if err != nil {
164 t.Fatal(err)
165 }
166 st := map[string]string{"a": "failed", "b": "queued", "c": "queued", "d": "queued"}
167 if Gate(p.Jobs, st, p.Jobs[1]) != "ready" {
168 t.Fatal("allow_failure houdt de volgende stage niet tegen")
169 }
170 if Gate(p.Jobs, st, p.Jobs[2]) != "ready" || Gate(p.Jobs, st, p.Jobs[3]) != "ready" {
171 t.Fatal("always en on_failure na een mislukking")
172 }
173 ok := map[string]string{"a": "success", "b": "queued", "c": "queued", "d": "queued"}
174 if Gate(p.Jobs, ok, p.Jobs[3]) != "skip" {
175 t.Fatal("on_failure slaat over na succes")
176 }
177 hard, err := Parse([]byte(`
178stages: [een, twee]
179jobs:
180 a:
181 stage: een
182 image: alpine
183 script: ['echo a']
184 b:
185 stage: twee
186 image: alpine
187 script: ['echo b']
188`))
189 if err != nil {
190 t.Fatal(err)
191 }
192 if Gate(hard.Jobs, map[string]string{"a": "failed", "b": "queued"}, hard.Jobs[1]) != "skip" {
193 t.Fatal("harde fout slaat de rest over")
194 }
195}
196
197func TestMaskAndVariables(t *testing.T) {
198 if got := Mask("token=supersecret value", []string{"supersecret", "ab"}); got != "token=[MASKED] value" {
199 t.Fatal(got)
200 }
201 env := Variables(Ref{SHA: "abcdef1234567890", FullRef: "refs/tags/v1.2.0", Title: "Uitgave", DefaultBranch: "master", Source: "push", ProjectPath: "tim/app", ProjectURL: "https://git.example/tim/app", APIURL: "https://git.example/api/v1", PipelineID: 9})
202 if env["CI_COMMIT_TAG"] != "v1.2.0" || env["CI_COMMIT_BRANCH"] != "" || env["CI_COMMIT_SHORT_SHA"] != "abcdef12" {
203 t.Fatalf("%v", env)
204 }
205 if env["CI_DEFAULT_BRANCH"] != "master" || env["CI_PROJECT_NAME"] != "app" || env["CI_PIPELINE_SOURCE"] != "push" {
206 t.Fatalf("%v", env)
207 }
208 ok, err := evalRule(`$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH`, map[string]string{"CI_COMMIT_BRANCH": "master", "CI_DEFAULT_BRANCH": "master"})
209 if err != nil || !ok {
210 t.Fatal(err, ok)
211 }
212 ok, err = evalRule(`$CI_COMMIT_TAG == null`, map[string]string{"CI_COMMIT_TAG": ""})
213 if err != nil || !ok {
214 t.Fatal("null", err, ok)
215 }
216}
internal/i18n/catalog.go +53 −0
@@ -412,4 +412,57 @@ var en = map[string]string{
412 412 "err.sugSame": "the suggestions do not change the file",
413 413 "err.sugBranch": "you cannot push to %s",
414 414 "flash.suggestions": "Suggestions applied",
415
416 "ci.err.yaml": "invalid YAML: %s",
417 "ci.err.mapping": "blue-ci-cd.yml must be a mapping",
418 "ci.err.list": "expected a list",
419 "ci.err.unknown": "unknown field %q",
420 "ci.warn.ignored": "%s is not used",
421 "ci.err.both": "use either top-level jobs or a jobs map",
422 "ci.err.includeRead": "cannot read include %s",
423 "ci.err.includeCycle": "include cycle at %s",
424 "ci.err.includeLocal": "include local must be a path",
425 "ci.err.includeRemote": "only local includes are supported",
426 "ci.err.rel": "path %q must be relative",
427 "ci.err.map": "expected a mapping",
428 "ci.err.jobsMap": "jobs must be a mapping",
429 "ci.err.jobMap": "job %s must be a mapping",
430 "ci.err.stageText": "stage must be text",
431 "ci.err.script": "script must be a list of commands",
432 "ci.err.needs": "needs must name a job",
433 "ci.err.resource": "resource_group must be text",
434 "ci.err.jobDup": "duplicate job %q",
435 "ci.err.extendsCycle": "cycle in extends at %s",
436 "ci.err.extends": "job %s extends unknown job %s",
437 "ci.err.noJobs": "no jobs",
438 "ci.err.stageEmpty": "empty stage name",
439 "ci.err.stageDup": "duplicate stage %q",
440 "ci.err.jobName": "invalid job name %q",
441 "ci.err.jobStage": "job %s has no stage",
442 "ci.err.stageUnknown": "job %s uses unknown stage %q",
443 "ci.err.noImage": "job %s has no image",
444 "ci.err.noScript": "job %s has no script",
445 "ci.err.emptyLine": "job %s has an empty script line",
446 "ci.err.needUnknown": "job %s needs unknown job %q",
447 "ci.err.needSelf": "job %s cannot depend on itself",
448 "ci.err.cycle": "cycle in needs at job %s",
449 "ci.err.artifacts": "artifacts must be a mapping",
450 "ci.err.expire": "expire_in must be text",
451 "ci.err.expireBad": "expire_in %q is invalid",
452 "ci.err.cache": "cache must be a mapping",
453 "ci.err.cacheKey": "cache key must be text",
454 "ci.err.rulesList": "rules must be a list",
455 "ci.err.ruleMap": "a rule must be a mapping",
456 "ci.err.ruleIf": "rule.if must be text",
457 "ci.err.ruleEmpty": "rule has no condition",
458 "ci.err.ruleExpr": "rule %q is not understood",
459 "ci.err.ruleCombine": "rule %q cannot combine conditions with && or ||",
460 "ci.err.regex": "invalid regex in %q: %s",
461 "ci.err.when": "when %q is not supported",
462 "ci.err.timeout": "timeout %q is invalid",
463 "ci.err.extendsList": "extends must be a job name or a list",
464 "ci.err.listText": "a list may only contain text",
465 "ci.err.varText": "variables must be text",
466 "ci.err.bool": "expected true or false",
467 "ci.err.dotdot": "path %q must stay inside the workspace",
415 468 }
@@ -412,4 +412,57 @@ var en = map[string]string{
412 "err.sugSame": "the suggestions do not change the file",
412 "err.sugSame": "the suggestions do not change the file",
413 "err.sugBranch": "you cannot push to %s",
413 "err.sugBranch": "you cannot push to %s",
414 "flash.suggestions": "Suggestions applied",
414 "flash.suggestions": "Suggestions applied",
415
416 "ci.err.yaml": "invalid YAML: %s",
417 "ci.err.mapping": "blue-ci-cd.yml must be a mapping",
418 "ci.err.list": "expected a list",
419 "ci.err.unknown": "unknown field %q",
420 "ci.warn.ignored": "%s is not used",
421 "ci.err.both": "use either top-level jobs or a jobs map",
422 "ci.err.includeRead": "cannot read include %s",
423 "ci.err.includeCycle": "include cycle at %s",
424 "ci.err.includeLocal": "include local must be a path",
425 "ci.err.includeRemote": "only local includes are supported",
426 "ci.err.rel": "path %q must be relative",
427 "ci.err.map": "expected a mapping",
428 "ci.err.jobsMap": "jobs must be a mapping",
429 "ci.err.jobMap": "job %s must be a mapping",
430 "ci.err.stageText": "stage must be text",
431 "ci.err.script": "script must be a list of commands",
432 "ci.err.needs": "needs must name a job",
433 "ci.err.resource": "resource_group must be text",
434 "ci.err.jobDup": "duplicate job %q",
435 "ci.err.extendsCycle": "cycle in extends at %s",
436 "ci.err.extends": "job %s extends unknown job %s",
437 "ci.err.noJobs": "no jobs",
438 "ci.err.stageEmpty": "empty stage name",
439 "ci.err.stageDup": "duplicate stage %q",
440 "ci.err.jobName": "invalid job name %q",
441 "ci.err.jobStage": "job %s has no stage",
442 "ci.err.stageUnknown": "job %s uses unknown stage %q",
443 "ci.err.noImage": "job %s has no image",
444 "ci.err.noScript": "job %s has no script",
445 "ci.err.emptyLine": "job %s has an empty script line",
446 "ci.err.needUnknown": "job %s needs unknown job %q",
447 "ci.err.needSelf": "job %s cannot depend on itself",
448 "ci.err.cycle": "cycle in needs at job %s",
449 "ci.err.artifacts": "artifacts must be a mapping",
450 "ci.err.expire": "expire_in must be text",
451 "ci.err.expireBad": "expire_in %q is invalid",
452 "ci.err.cache": "cache must be a mapping",
453 "ci.err.cacheKey": "cache key must be text",
454 "ci.err.rulesList": "rules must be a list",
455 "ci.err.ruleMap": "a rule must be a mapping",
456 "ci.err.ruleIf": "rule.if must be text",
457 "ci.err.ruleEmpty": "rule has no condition",
458 "ci.err.ruleExpr": "rule %q is not understood",
459 "ci.err.ruleCombine": "rule %q cannot combine conditions with && or ||",
460 "ci.err.regex": "invalid regex in %q: %s",
461 "ci.err.when": "when %q is not supported",
462 "ci.err.timeout": "timeout %q is invalid",
463 "ci.err.extendsList": "extends must be a job name or a list",
464 "ci.err.listText": "a list may only contain text",
465 "ci.err.varText": "variables must be text",
466 "ci.err.bool": "expected true or false",
467 "ci.err.dotdot": "path %q must stay inside the workspace",
415}
468}
internal/i18n/catalog_lang.go +216 −60
@@ -389,26 +389,78 @@ var nl = map[string]string{
389 389 "err.whServer": "server-URL ontbreekt",
390 390 "err.whMethod": "methode moet GET, POST, PUT of PATCH zijn",
391 391
392 "mr.changes": "Wijzigingen",
393 "action.cancel": "Annuleren",
394 "sug.applyWord": "Suggesties toepassen",
395 "sug.include": "Deze suggestie meenemen",
396 "sug.title": "Titel",
397 "sug.description": "Beschrijving",
398 "sug.optional": "Optioneel.",
399 "sug.submit": "Commit maken",
400 "sug.hint": "Het suggestion-blok vervangt de geselecteerde regels.",
401 "sug.bubble": "Reageren op de geselecteerde regels",
402 "err.sugConflict": "%s:%d-%d is intussen veranderd",
403 "err.sugOverlap": "suggesties in %s overlappen",
404 "err.sugEmpty": "kies een suggestie",
405 "err.sugTitle": "een titel is verplicht",
406 "err.sugLines": "die regels staan niet in het bestand",
407 "err.sugFile": "dat bestand is niet beschikbaar",
408 "err.sugGone": "die suggestie hoort niet bij deze merge request",
409 "err.sugSame": "de suggesties veranderen het bestand niet",
410 "err.sugBranch": "je kunt niet pushen naar %s",
411 "flash.suggestions": "Suggesties toegepast",
392 "mr.changes": "Wijzigingen",
393 "action.cancel": "Annuleren",
394 "sug.applyWord": "Suggesties toepassen",
395 "sug.include": "Deze suggestie meenemen",
396 "sug.title": "Titel",
397 "sug.description": "Beschrijving",
398 "sug.optional": "Optioneel.",
399 "sug.submit": "Commit maken",
400 "sug.hint": "Het suggestion-blok vervangt de geselecteerde regels.",
401 "sug.bubble": "Reageren op de geselecteerde regels",
402 "err.sugConflict": "%s:%d-%d is intussen veranderd",
403 "err.sugOverlap": "suggesties in %s overlappen",
404 "err.sugEmpty": "kies een suggestie",
405 "err.sugTitle": "een titel is verplicht",
406 "err.sugLines": "die regels staan niet in het bestand",
407 "err.sugFile": "dat bestand is niet beschikbaar",
408 "err.sugGone": "die suggestie hoort niet bij deze merge request",
409 "err.sugSame": "de suggesties veranderen het bestand niet",
410 "err.sugBranch": "je kunt niet pushen naar %s",
411 "flash.suggestions": "Suggesties toegepast",
412 "ci.err.yaml": "ongeldige YAML: %s",
413 "ci.err.mapping": "blue-ci-cd.yml moet een mapping zijn",
414 "ci.err.list": "verwacht een lijst",
415 "ci.err.unknown": "onbekend veld %q",
416 "ci.warn.ignored": "%s wordt niet gebruikt",
417 "ci.err.both": "gebruik of losse jobs, of een jobs-map",
418 "ci.err.includeRead": "include %s is niet te lezen",
419 "ci.err.includeCycle": "include-cyclus bij %s",
420 "ci.err.includeLocal": "include local moet een pad zijn",
421 "ci.err.includeRemote": "alleen lokale includes worden ondersteund",
422 "ci.err.rel": "pad %q moet relatief zijn",
423 "ci.err.map": "verwacht een mapping",
424 "ci.err.jobsMap": "jobs moet een mapping zijn",
425 "ci.err.jobMap": "job %s moet een mapping zijn",
426 "ci.err.stageText": "stage moet tekst zijn",
427 "ci.err.script": "script moet een lijst commando's zijn",
428 "ci.err.needs": "needs moet een job noemen",
429 "ci.err.resource": "resource_group moet tekst zijn",
430 "ci.err.jobDup": "dubbele job %q",
431 "ci.err.extendsCycle": "cyclus in extends bij %s",
432 "ci.err.extends": "job %s erft van onbekende job %s",
433 "ci.err.noJobs": "geen jobs",
434 "ci.err.stageEmpty": "lege stage-naam",
435 "ci.err.stageDup": "dubbele stage %q",
436 "ci.err.jobName": "ongeldige jobnaam %q",
437 "ci.err.jobStage": "job %s heeft geen stage",
438 "ci.err.stageUnknown": "job %s gebruikt onbekende stage %q",
439 "ci.err.noImage": "job %s heeft geen image",
440 "ci.err.noScript": "job %s heeft geen script",
441 "ci.err.emptyLine": "job %s heeft een lege scriptregel",
442 "ci.err.needUnknown": "job %s heeft onbekende need %q",
443 "ci.err.needSelf": "job %s kan niet van zichzelf afhangen",
444 "ci.err.cycle": "cyclus in needs bij job %s",
445 "ci.err.artifacts": "artifacts moet een mapping zijn",
446 "ci.err.expire": "expire_in moet tekst zijn",
447 "ci.err.expireBad": "expire_in %q is ongeldig",
448 "ci.err.cache": "cache moet een mapping zijn",
449 "ci.err.cacheKey": "cache-sleutel moet tekst zijn",
450 "ci.err.rulesList": "rules moet een lijst zijn",
451 "ci.err.ruleMap": "een rule moet een mapping zijn",
452 "ci.err.ruleIf": "rule.if moet tekst zijn",
453 "ci.err.ruleEmpty": "rule heeft geen voorwaarde",
454 "ci.err.ruleExpr": "rule %q wordt niet begrepen",
455 "ci.err.ruleCombine": "rule %q kan geen voorwaarden combineren met && of ||",
456 "ci.err.regex": "ongeldige regex in %q: %s",
457 "ci.err.when": "when %q wordt niet ondersteund",
458 "ci.err.timeout": "timeout %q is ongeldig",
459 "ci.err.extendsList": "extends moet een jobnaam of een lijst zijn",
460 "ci.err.listText": "een lijst mag alleen tekst bevatten",
461 "ci.err.varText": "variabelen moeten tekst zijn",
462 "ci.err.bool": "verwacht true of false",
463 "ci.err.dotdot": "pad %q moet binnen de werkmap blijven",
412 464 }
413 465
414 466 var de = map[string]string{
@@ -800,26 +852,78 @@ var de = map[string]string{
800 852 "err.whServer": "Server-URL fehlt",
801 853 "err.whMethod": "Methode muss GET, POST, PUT oder PATCH sein",
802 854
803 "mr.changes": "Änderungen",
804 "action.cancel": "Abbrechen",
805 "sug.applyWord": "Vorschläge anwenden",
806 "sug.include": "Diesen Vorschlag übernehmen",
807 "sug.title": "Titel",
808 "sug.description": "Beschreibung",
809 "sug.optional": "Optional.",
810 "sug.submit": "Commit erstellen",
811 "sug.hint": "Der suggestion-Block ersetzt die ausgewählten Zeilen.",
812 "sug.bubble": "Die ausgewählten Zeilen kommentieren",
813 "err.sugConflict": "%s:%d-%d hat sich inzwischen geändert",
814 "err.sugOverlap": "Vorschläge in %s überlappen sich",
815 "err.sugEmpty": "wähle einen Vorschlag",
816 "err.sugTitle": "ein Titel ist erforderlich",
817 "err.sugLines": "diese Zeilen stehen nicht in der Datei",
818 "err.sugFile": "diese Datei ist nicht verfügbar",
819 "err.sugGone": "dieser Vorschlag gehört nicht zu diesem Merge-Request",
820 "err.sugSame": "die Vorschläge ändern die Datei nicht",
821 "err.sugBranch": "du kannst nicht nach %s pushen",
822 "flash.suggestions": "Vorschläge angewendet",
855 "mr.changes": "Änderungen",
856 "action.cancel": "Abbrechen",
857 "sug.applyWord": "Vorschläge anwenden",
858 "sug.include": "Diesen Vorschlag übernehmen",
859 "sug.title": "Titel",
860 "sug.description": "Beschreibung",
861 "sug.optional": "Optional.",
862 "sug.submit": "Commit erstellen",
863 "sug.hint": "Der suggestion-Block ersetzt die ausgewählten Zeilen.",
864 "sug.bubble": "Die ausgewählten Zeilen kommentieren",
865 "err.sugConflict": "%s:%d-%d hat sich inzwischen geändert",
866 "err.sugOverlap": "Vorschläge in %s überlappen sich",
867 "err.sugEmpty": "wähle einen Vorschlag",
868 "err.sugTitle": "ein Titel ist erforderlich",
869 "err.sugLines": "diese Zeilen stehen nicht in der Datei",
870 "err.sugFile": "diese Datei ist nicht verfügbar",
871 "err.sugGone": "dieser Vorschlag gehört nicht zu diesem Merge-Request",
872 "err.sugSame": "die Vorschläge ändern die Datei nicht",
873 "err.sugBranch": "du kannst nicht nach %s pushen",
874 "flash.suggestions": "Vorschläge angewendet",
875 "ci.err.yaml": "ungültiges YAML: %s",
876 "ci.err.mapping": "blue-ci-cd.yml muss ein Mapping sein",
877 "ci.err.list": "eine Liste wird erwartet",
878 "ci.err.unknown": "unbekanntes Feld %q",
879 "ci.warn.ignored": "%s wird nicht verwendet",
880 "ci.err.both": "entweder einzelne Jobs oder eine jobs-Map verwenden",
881 "ci.err.includeRead": "include %s ist nicht lesbar",
882 "ci.err.includeCycle": "Include-Zyklus bei %s",
883 "ci.err.includeLocal": "include local muss ein Pfad sein",
884 "ci.err.includeRemote": "nur lokale Includes werden unterstützt",
885 "ci.err.rel": "Pfad %q muss relativ sein",
886 "ci.err.map": "ein Mapping wird erwartet",
887 "ci.err.jobsMap": "jobs muss ein Mapping sein",
888 "ci.err.jobMap": "Job %s muss ein Mapping sein",
889 "ci.err.stageText": "stage muss Text sein",
890 "ci.err.script": "script muss eine Befehlsliste sein",
891 "ci.err.needs": "needs muss einen Job nennen",
892 "ci.err.resource": "resource_group muss Text sein",
893 "ci.err.jobDup": "doppelter Job %q",
894 "ci.err.extendsCycle": "Zyklus in extends bei %s",
895 "ci.err.extends": "Job %s erweitert den unbekannten Job %s",
896 "ci.err.noJobs": "keine Jobs",
897 "ci.err.stageEmpty": "leerer Stage-Name",
898 "ci.err.stageDup": "doppelte Stage %q",
899 "ci.err.jobName": "ungültiger Jobname %q",
900 "ci.err.jobStage": "Job %s hat keine Stage",
901 "ci.err.stageUnknown": "Job %s nutzt die unbekannte Stage %q",
902 "ci.err.noImage": "Job %s hat kein Image",
903 "ci.err.noScript": "Job %s hat kein script",
904 "ci.err.emptyLine": "Job %s hat eine leere Scriptzeile",
905 "ci.err.needUnknown": "Job %s braucht den unbekannten Job %q",
906 "ci.err.needSelf": "Job %s kann nicht von sich selbst abhängen",
907 "ci.err.cycle": "Zyklus in needs bei Job %s",
908 "ci.err.artifacts": "artifacts muss ein Mapping sein",
909 "ci.err.expire": "expire_in muss Text sein",
910 "ci.err.expireBad": "expire_in %q ist ungültig",
911 "ci.err.cache": "cache muss ein Mapping sein",
912 "ci.err.cacheKey": "Cache-Schlüssel muss Text sein",
913 "ci.err.rulesList": "rules muss eine Liste sein",
914 "ci.err.ruleMap": "eine Rule muss ein Mapping sein",
915 "ci.err.ruleIf": "rule.if muss Text sein",
916 "ci.err.ruleEmpty": "Rule hat keine Bedingung",
917 "ci.err.ruleExpr": "Rule %q wird nicht verstanden",
918 "ci.err.ruleCombine": "Rule %q kann Bedingungen nicht mit && oder || verbinden",
919 "ci.err.regex": "ungültiger regulärer Ausdruck in %q: %s",
920 "ci.err.when": "when %q wird nicht unterstützt",
921 "ci.err.timeout": "timeout %q ist ungültig",
922 "ci.err.extendsList": "extends muss ein Jobname oder eine Liste sein",
923 "ci.err.listText": "eine Liste darf nur Text enthalten",
924 "ci.err.varText": "Variablen müssen Text sein",
925 "ci.err.bool": "true oder false erwartet",
926 "ci.err.dotdot": "Pfad %q muss im Arbeitsverzeichnis bleiben",
823 927 }
824 928
825 929 var fr = map[string]string{
@@ -1211,24 +1315,76 @@ var fr = map[string]string{
1211 1315 "err.whServer": "l’URL du serveur manque",
1212 1316 "err.whMethod": "la méthode doit être GET, POST, PUT ou PATCH",
1213 1317
1214 "mr.changes": "Modifications",
1215 "action.cancel": "Annuler",
1216 "sug.applyWord": "Appliquer les suggestions",
1217 "sug.include": "Inclure cette suggestion",
1218 "sug.title": "Titre",
1219 "sug.description": "Description",
1220 "sug.optional": "Facultatif.",
1221 "sug.submit": "Créer le commit",
1222 "sug.hint": "Le bloc suggestion remplace les lignes sélectionnées.",
1223 "sug.bubble": "Commenter les lignes sélectionnées",
1224 "err.sugConflict": "%s:%d-%d a changé entre-temps",
1225 "err.sugOverlap": "les suggestions dans %s se chevauchent",
1226 "err.sugEmpty": "choisis une suggestion",
1227 "err.sugTitle": "un titre est obligatoire",
1228 "err.sugLines": "ces lignes ne sont pas dans le fichier",
1229 "err.sugFile": "ce fichier n’est pas disponible",
1230 "err.sugGone": "cette suggestion n’appartient pas à cette merge request",
1231 "err.sugSame": "les suggestions ne modifient pas le fichier",
1232 "err.sugBranch": "tu ne peux pas pousser vers %s",
1233 "flash.suggestions": "Suggestions appliquées",
1318 "mr.changes": "Modifications",
1319 "action.cancel": "Annuler",
1320 "sug.applyWord": "Appliquer les suggestions",
1321 "sug.include": "Inclure cette suggestion",
1322 "sug.title": "Titre",
1323 "sug.description": "Description",
1324 "sug.optional": "Facultatif.",
1325 "sug.submit": "Créer le commit",
1326 "sug.hint": "Le bloc suggestion remplace les lignes sélectionnées.",
1327 "sug.bubble": "Commenter les lignes sélectionnées",
1328 "err.sugConflict": "%s:%d-%d a changé entre-temps",
1329 "err.sugOverlap": "les suggestions dans %s se chevauchent",
1330 "err.sugEmpty": "choisis une suggestion",
1331 "err.sugTitle": "un titre est obligatoire",
1332 "err.sugLines": "ces lignes ne sont pas dans le fichier",
1333 "err.sugFile": "ce fichier n’est pas disponible",
1334 "err.sugGone": "cette suggestion n’appartient pas à cette merge request",
1335 "err.sugSame": "les suggestions ne modifient pas le fichier",
1336 "err.sugBranch": "tu ne peux pas pousser vers %s",
1337 "flash.suggestions": "Suggestions appliquées",
1338 "ci.err.yaml": "YAML invalide : %s",
1339 "ci.err.mapping": "blue-ci-cd.yml doit être un mapping",
1340 "ci.err.list": "une liste est attendue",
1341 "ci.err.unknown": "champ inconnu %q",
1342 "ci.warn.ignored": "%s n'est pas utilisé",
1343 "ci.err.both": "utilise soit des jobs au premier niveau, soit une map jobs",
1344 "ci.err.includeRead": "include %s est illisible",
1345 "ci.err.includeCycle": "cycle d'include à %s",
1346 "ci.err.includeLocal": "include local doit être un chemin",
1347 "ci.err.includeRemote": "seuls les includes locaux sont pris en charge",
1348 "ci.err.rel": "le chemin %q doit être relatif",
1349 "ci.err.map": "un mapping est attendu",
1350 "ci.err.jobsMap": "jobs doit être un mapping",
1351 "ci.err.jobMap": "le job %s doit être un mapping",
1352 "ci.err.stageText": "stage doit être du texte",
1353 "ci.err.script": "script doit être une liste de commandes",
1354 "ci.err.needs": "needs doit nommer un job",
1355 "ci.err.resource": "resource_group doit être du texte",
1356 "ci.err.jobDup": "job en double %q",
1357 "ci.err.extendsCycle": "cycle dans extends à %s",
1358 "ci.err.extends": "le job %s étend le job inconnu %s",
1359 "ci.err.noJobs": "aucun job",
1360 "ci.err.stageEmpty": "nom de stage vide",
1361 "ci.err.stageDup": "stage en double %q",
1362 "ci.err.jobName": "nom de job invalide %q",
1363 "ci.err.jobStage": "le job %s n'a pas de stage",
1364 "ci.err.stageUnknown": "le job %s utilise le stage inconnu %q",
1365 "ci.err.noImage": "le job %s n'a pas d'image",
1366 "ci.err.noScript": "le job %s n'a pas de script",
1367 "ci.err.emptyLine": "le job %s a une ligne de script vide",
1368 "ci.err.needUnknown": "le job %s dépend du job inconnu %q",
1369 "ci.err.needSelf": "le job %s ne peut pas dépendre de lui-même",
1370 "ci.err.cycle": "cycle dans needs au job %s",
1371 "ci.err.artifacts": "artifacts doit être un mapping",
1372 "ci.err.expire": "expire_in doit être du texte",
1373 "ci.err.expireBad": "expire_in %q est invalide",
1374 "ci.err.cache": "cache doit être un mapping",
1375 "ci.err.cacheKey": "la clé de cache doit être du texte",
1376 "ci.err.rulesList": "rules doit être une liste",
1377 "ci.err.ruleMap": "une règle doit être un mapping",
1378 "ci.err.ruleIf": "rule.if doit être du texte",
1379 "ci.err.ruleEmpty": "la règle n'a pas de condition",
1380 "ci.err.ruleExpr": "la règle %q n'est pas comprise",
1381 "ci.err.ruleCombine": "la règle %q ne peut pas combiner des conditions avec && ou ||",
1382 "ci.err.regex": "expression régulière invalide dans %q : %s",
1383 "ci.err.when": "when %q n'est pas pris en charge",
1384 "ci.err.timeout": "timeout %q est invalide",
1385 "ci.err.extendsList": "extends doit être un nom de job ou une liste",
1386 "ci.err.listText": "une liste ne peut contenir que du texte",
1387 "ci.err.varText": "les variables doivent être du texte",
1388 "ci.err.bool": "true ou false attendu",
1389 "ci.err.dotdot": "le chemin %q doit rester dans l'espace de travail",
1234 1390 }
@@ -389,26 +389,78 @@ var nl = map[string]string{
389 "err.whServer": "server-URL ontbreekt",
389 "err.whServer": "server-URL ontbreekt",
390 "err.whMethod": "methode moet GET, POST, PUT of PATCH zijn",
390 "err.whMethod": "methode moet GET, POST, PUT of PATCH zijn",
391
391
392 "mr.changes": "Wijzigingen",
392 "mr.changes": "Wijzigingen",
393 "action.cancel": "Annuleren",
393 "action.cancel": "Annuleren",
394 "sug.applyWord": "Suggesties toepassen",
394 "sug.applyWord": "Suggesties toepassen",
395 "sug.include": "Deze suggestie meenemen",
395 "sug.include": "Deze suggestie meenemen",
396 "sug.title": "Titel",
396 "sug.title": "Titel",
397 "sug.description": "Beschrijving",
397 "sug.description": "Beschrijving",
398 "sug.optional": "Optioneel.",
398 "sug.optional": "Optioneel.",
399 "sug.submit": "Commit maken",
399 "sug.submit": "Commit maken",
400 "sug.hint": "Het suggestion-blok vervangt de geselecteerde regels.",
400 "sug.hint": "Het suggestion-blok vervangt de geselecteerde regels.",
401 "sug.bubble": "Reageren op de geselecteerde regels",
401 "sug.bubble": "Reageren op de geselecteerde regels",
402 "err.sugConflict": "%s:%d-%d is intussen veranderd",
402 "err.sugConflict": "%s:%d-%d is intussen veranderd",
403 "err.sugOverlap": "suggesties in %s overlappen",
403 "err.sugOverlap": "suggesties in %s overlappen",
404 "err.sugEmpty": "kies een suggestie",
404 "err.sugEmpty": "kies een suggestie",
405 "err.sugTitle": "een titel is verplicht",
405 "err.sugTitle": "een titel is verplicht",
406 "err.sugLines": "die regels staan niet in het bestand",
406 "err.sugLines": "die regels staan niet in het bestand",
407 "err.sugFile": "dat bestand is niet beschikbaar",
407 "err.sugFile": "dat bestand is niet beschikbaar",
408 "err.sugGone": "die suggestie hoort niet bij deze merge request",
408 "err.sugGone": "die suggestie hoort niet bij deze merge request",
409 "err.sugSame": "de suggesties veranderen het bestand niet",
409 "err.sugSame": "de suggesties veranderen het bestand niet",
410 "err.sugBranch": "je kunt niet pushen naar %s",
410 "err.sugBranch": "je kunt niet pushen naar %s",
411 "flash.suggestions": "Suggesties toegepast",
411 "flash.suggestions": "Suggesties toegepast",
412 "ci.err.yaml": "ongeldige YAML: %s",
413 "ci.err.mapping": "blue-ci-cd.yml moet een mapping zijn",
414 "ci.err.list": "verwacht een lijst",
415 "ci.err.unknown": "onbekend veld %q",
416 "ci.warn.ignored": "%s wordt niet gebruikt",
417 "ci.err.both": "gebruik of losse jobs, of een jobs-map",
418 "ci.err.includeRead": "include %s is niet te lezen",
419 "ci.err.includeCycle": "include-cyclus bij %s",
420 "ci.err.includeLocal": "include local moet een pad zijn",
421 "ci.err.includeRemote": "alleen lokale includes worden ondersteund",
422 "ci.err.rel": "pad %q moet relatief zijn",
423 "ci.err.map": "verwacht een mapping",
424 "ci.err.jobsMap": "jobs moet een mapping zijn",
425 "ci.err.jobMap": "job %s moet een mapping zijn",
426 "ci.err.stageText": "stage moet tekst zijn",
427 "ci.err.script": "script moet een lijst commando's zijn",
428 "ci.err.needs": "needs moet een job noemen",
429 "ci.err.resource": "resource_group moet tekst zijn",
430 "ci.err.jobDup": "dubbele job %q",
431 "ci.err.extendsCycle": "cyclus in extends bij %s",
432 "ci.err.extends": "job %s erft van onbekende job %s",
433 "ci.err.noJobs": "geen jobs",
434 "ci.err.stageEmpty": "lege stage-naam",
435 "ci.err.stageDup": "dubbele stage %q",
436 "ci.err.jobName": "ongeldige jobnaam %q",
437 "ci.err.jobStage": "job %s heeft geen stage",
438 "ci.err.stageUnknown": "job %s gebruikt onbekende stage %q",
439 "ci.err.noImage": "job %s heeft geen image",
440 "ci.err.noScript": "job %s heeft geen script",
441 "ci.err.emptyLine": "job %s heeft een lege scriptregel",
442 "ci.err.needUnknown": "job %s heeft onbekende need %q",
443 "ci.err.needSelf": "job %s kan niet van zichzelf afhangen",
444 "ci.err.cycle": "cyclus in needs bij job %s",
445 "ci.err.artifacts": "artifacts moet een mapping zijn",
446 "ci.err.expire": "expire_in moet tekst zijn",
447 "ci.err.expireBad": "expire_in %q is ongeldig",
448 "ci.err.cache": "cache moet een mapping zijn",
449 "ci.err.cacheKey": "cache-sleutel moet tekst zijn",
450 "ci.err.rulesList": "rules moet een lijst zijn",
451 "ci.err.ruleMap": "een rule moet een mapping zijn",
452 "ci.err.ruleIf": "rule.if moet tekst zijn",
453 "ci.err.ruleEmpty": "rule heeft geen voorwaarde",
454 "ci.err.ruleExpr": "rule %q wordt niet begrepen",
455 "ci.err.ruleCombine": "rule %q kan geen voorwaarden combineren met && of ||",
456 "ci.err.regex": "ongeldige regex in %q: %s",
457 "ci.err.when": "when %q wordt niet ondersteund",
458 "ci.err.timeout": "timeout %q is ongeldig",
459 "ci.err.extendsList": "extends moet een jobnaam of een lijst zijn",
460 "ci.err.listText": "een lijst mag alleen tekst bevatten",
461 "ci.err.varText": "variabelen moeten tekst zijn",
462 "ci.err.bool": "verwacht true of false",
463 "ci.err.dotdot": "pad %q moet binnen de werkmap blijven",
412}
464}
413
465
414var de = map[string]string{
466var de = map[string]string{
@@ -800,26 +852,78 @@ var de = map[string]string{
800 "err.whServer": "Server-URL fehlt",
852 "err.whServer": "Server-URL fehlt",
801 "err.whMethod": "Methode muss GET, POST, PUT oder PATCH sein",
853 "err.whMethod": "Methode muss GET, POST, PUT oder PATCH sein",
802
854
803 "mr.changes": "Änderungen",
855 "mr.changes": "Änderungen",
804 "action.cancel": "Abbrechen",
856 "action.cancel": "Abbrechen",
805 "sug.applyWord": "Vorschläge anwenden",
857 "sug.applyWord": "Vorschläge anwenden",
806 "sug.include": "Diesen Vorschlag übernehmen",
858 "sug.include": "Diesen Vorschlag übernehmen",
807 "sug.title": "Titel",
859 "sug.title": "Titel",
808 "sug.description": "Beschreibung",
860 "sug.description": "Beschreibung",
809 "sug.optional": "Optional.",
861 "sug.optional": "Optional.",
810 "sug.submit": "Commit erstellen",
862 "sug.submit": "Commit erstellen",
811 "sug.hint": "Der suggestion-Block ersetzt die ausgewählten Zeilen.",
863 "sug.hint": "Der suggestion-Block ersetzt die ausgewählten Zeilen.",
812 "sug.bubble": "Die ausgewählten Zeilen kommentieren",
864 "sug.bubble": "Die ausgewählten Zeilen kommentieren",
813 "err.sugConflict": "%s:%d-%d hat sich inzwischen geändert",
865 "err.sugConflict": "%s:%d-%d hat sich inzwischen geändert",
814 "err.sugOverlap": "Vorschläge in %s überlappen sich",
866 "err.sugOverlap": "Vorschläge in %s überlappen sich",
815 "err.sugEmpty": "wähle einen Vorschlag",
867 "err.sugEmpty": "wähle einen Vorschlag",
816 "err.sugTitle": "ein Titel ist erforderlich",
868 "err.sugTitle": "ein Titel ist erforderlich",
817 "err.sugLines": "diese Zeilen stehen nicht in der Datei",
869 "err.sugLines": "diese Zeilen stehen nicht in der Datei",
818 "err.sugFile": "diese Datei ist nicht verfügbar",
870 "err.sugFile": "diese Datei ist nicht verfügbar",
819 "err.sugGone": "dieser Vorschlag gehört nicht zu diesem Merge-Request",
871 "err.sugGone": "dieser Vorschlag gehört nicht zu diesem Merge-Request",
820 "err.sugSame": "die Vorschläge ändern die Datei nicht",
872 "err.sugSame": "die Vorschläge ändern die Datei nicht",
821 "err.sugBranch": "du kannst nicht nach %s pushen",
873 "err.sugBranch": "du kannst nicht nach %s pushen",
822 "flash.suggestions": "Vorschläge angewendet",
874 "flash.suggestions": "Vorschläge angewendet",
875 "ci.err.yaml": "ungültiges YAML: %s",
876 "ci.err.mapping": "blue-ci-cd.yml muss ein Mapping sein",
877 "ci.err.list": "eine Liste wird erwartet",
878 "ci.err.unknown": "unbekanntes Feld %q",
879 "ci.warn.ignored": "%s wird nicht verwendet",
880 "ci.err.both": "entweder einzelne Jobs oder eine jobs-Map verwenden",
881 "ci.err.includeRead": "include %s ist nicht lesbar",
882 "ci.err.includeCycle": "Include-Zyklus bei %s",
883 "ci.err.includeLocal": "include local muss ein Pfad sein",
884 "ci.err.includeRemote": "nur lokale Includes werden unterstützt",
885 "ci.err.rel": "Pfad %q muss relativ sein",
886 "ci.err.map": "ein Mapping wird erwartet",
887 "ci.err.jobsMap": "jobs muss ein Mapping sein",
888 "ci.err.jobMap": "Job %s muss ein Mapping sein",
889 "ci.err.stageText": "stage muss Text sein",
890 "ci.err.script": "script muss eine Befehlsliste sein",
891 "ci.err.needs": "needs muss einen Job nennen",
892 "ci.err.resource": "resource_group muss Text sein",
893 "ci.err.jobDup": "doppelter Job %q",
894 "ci.err.extendsCycle": "Zyklus in extends bei %s",
895 "ci.err.extends": "Job %s erweitert den unbekannten Job %s",
896 "ci.err.noJobs": "keine Jobs",
897 "ci.err.stageEmpty": "leerer Stage-Name",
898 "ci.err.stageDup": "doppelte Stage %q",
899 "ci.err.jobName": "ungültiger Jobname %q",
900 "ci.err.jobStage": "Job %s hat keine Stage",
901 "ci.err.stageUnknown": "Job %s nutzt die unbekannte Stage %q",
902 "ci.err.noImage": "Job %s hat kein Image",
903 "ci.err.noScript": "Job %s hat kein script",
904 "ci.err.emptyLine": "Job %s hat eine leere Scriptzeile",
905 "ci.err.needUnknown": "Job %s braucht den unbekannten Job %q",
906 "ci.err.needSelf": "Job %s kann nicht von sich selbst abhängen",
907 "ci.err.cycle": "Zyklus in needs bei Job %s",
908 "ci.err.artifacts": "artifacts muss ein Mapping sein",
909 "ci.err.expire": "expire_in muss Text sein",
910 "ci.err.expireBad": "expire_in %q ist ungültig",
911 "ci.err.cache": "cache muss ein Mapping sein",
912 "ci.err.cacheKey": "Cache-Schlüssel muss Text sein",
913 "ci.err.rulesList": "rules muss eine Liste sein",
914 "ci.err.ruleMap": "eine Rule muss ein Mapping sein",
915 "ci.err.ruleIf": "rule.if muss Text sein",
916 "ci.err.ruleEmpty": "Rule hat keine Bedingung",
917 "ci.err.ruleExpr": "Rule %q wird nicht verstanden",
918 "ci.err.ruleCombine": "Rule %q kann Bedingungen nicht mit && oder || verbinden",
919 "ci.err.regex": "ungültiger regulärer Ausdruck in %q: %s",
920 "ci.err.when": "when %q wird nicht unterstützt",
921 "ci.err.timeout": "timeout %q ist ungültig",
922 "ci.err.extendsList": "extends muss ein Jobname oder eine Liste sein",
923 "ci.err.listText": "eine Liste darf nur Text enthalten",
924 "ci.err.varText": "Variablen müssen Text sein",
925 "ci.err.bool": "true oder false erwartet",
926 "ci.err.dotdot": "Pfad %q muss im Arbeitsverzeichnis bleiben",
823}
927}
824
928
825var fr = map[string]string{
929var fr = map[string]string{
@@ -1211,24 +1315,76 @@ var fr = map[string]string{
1211 "err.whServer": "l’URL du serveur manque",
1315 "err.whServer": "l’URL du serveur manque",
1212 "err.whMethod": "la méthode doit être GET, POST, PUT ou PATCH",
1316 "err.whMethod": "la méthode doit être GET, POST, PUT ou PATCH",
1213
1317
1214 "mr.changes": "Modifications",
1318 "mr.changes": "Modifications",
1215 "action.cancel": "Annuler",
1319 "action.cancel": "Annuler",
1216 "sug.applyWord": "Appliquer les suggestions",
1320 "sug.applyWord": "Appliquer les suggestions",
1217 "sug.include": "Inclure cette suggestion",
1321 "sug.include": "Inclure cette suggestion",
1218 "sug.title": "Titre",
1322 "sug.title": "Titre",
1219 "sug.description": "Description",
1323 "sug.description": "Description",
1220 "sug.optional": "Facultatif.",
1324 "sug.optional": "Facultatif.",
1221 "sug.submit": "Créer le commit",
1325 "sug.submit": "Créer le commit",
1222 "sug.hint": "Le bloc suggestion remplace les lignes sélectionnées.",
1326 "sug.hint": "Le bloc suggestion remplace les lignes sélectionnées.",
1223 "sug.bubble": "Commenter les lignes sélectionnées",
1327 "sug.bubble": "Commenter les lignes sélectionnées",
1224 "err.sugConflict": "%s:%d-%d a changé entre-temps",
1328 "err.sugConflict": "%s:%d-%d a changé entre-temps",
1225 "err.sugOverlap": "les suggestions dans %s se chevauchent",
1329 "err.sugOverlap": "les suggestions dans %s se chevauchent",
1226 "err.sugEmpty": "choisis une suggestion",
1330 "err.sugEmpty": "choisis une suggestion",
1227 "err.sugTitle": "un titre est obligatoire",
1331 "err.sugTitle": "un titre est obligatoire",
1228 "err.sugLines": "ces lignes ne sont pas dans le fichier",
1332 "err.sugLines": "ces lignes ne sont pas dans le fichier",
1229 "err.sugFile": "ce fichier n’est pas disponible",
1333 "err.sugFile": "ce fichier n’est pas disponible",
1230 "err.sugGone": "cette suggestion n’appartient pas à cette merge request",
1334 "err.sugGone": "cette suggestion n’appartient pas à cette merge request",
1231 "err.sugSame": "les suggestions ne modifient pas le fichier",
1335 "err.sugSame": "les suggestions ne modifient pas le fichier",
1232 "err.sugBranch": "tu ne peux pas pousser vers %s",
1336 "err.sugBranch": "tu ne peux pas pousser vers %s",
1233 "flash.suggestions": "Suggestions appliquées",
1337 "flash.suggestions": "Suggestions appliquées",
1338 "ci.err.yaml": "YAML invalide : %s",
1339 "ci.err.mapping": "blue-ci-cd.yml doit être un mapping",
1340 "ci.err.list": "une liste est attendue",
1341 "ci.err.unknown": "champ inconnu %q",
1342 "ci.warn.ignored": "%s n'est pas utilisé",
1343 "ci.err.both": "utilise soit des jobs au premier niveau, soit une map jobs",
1344 "ci.err.includeRead": "include %s est illisible",
1345 "ci.err.includeCycle": "cycle d'include à %s",
1346 "ci.err.includeLocal": "include local doit être un chemin",
1347 "ci.err.includeRemote": "seuls les includes locaux sont pris en charge",
1348 "ci.err.rel": "le chemin %q doit être relatif",
1349 "ci.err.map": "un mapping est attendu",
1350 "ci.err.jobsMap": "jobs doit être un mapping",
1351 "ci.err.jobMap": "le job %s doit être un mapping",
1352 "ci.err.stageText": "stage doit être du texte",
1353 "ci.err.script": "script doit être une liste de commandes",
1354 "ci.err.needs": "needs doit nommer un job",
1355 "ci.err.resource": "resource_group doit être du texte",
1356 "ci.err.jobDup": "job en double %q",
1357 "ci.err.extendsCycle": "cycle dans extends à %s",
1358 "ci.err.extends": "le job %s étend le job inconnu %s",
1359 "ci.err.noJobs": "aucun job",
1360 "ci.err.stageEmpty": "nom de stage vide",
1361 "ci.err.stageDup": "stage en double %q",
1362 "ci.err.jobName": "nom de job invalide %q",
1363 "ci.err.jobStage": "le job %s n'a pas de stage",
1364 "ci.err.stageUnknown": "le job %s utilise le stage inconnu %q",
1365 "ci.err.noImage": "le job %s n'a pas d'image",
1366 "ci.err.noScript": "le job %s n'a pas de script",
1367 "ci.err.emptyLine": "le job %s a une ligne de script vide",
1368 "ci.err.needUnknown": "le job %s dépend du job inconnu %q",
1369 "ci.err.needSelf": "le job %s ne peut pas dépendre de lui-même",
1370 "ci.err.cycle": "cycle dans needs au job %s",
1371 "ci.err.artifacts": "artifacts doit être un mapping",
1372 "ci.err.expire": "expire_in doit être du texte",
1373 "ci.err.expireBad": "expire_in %q est invalide",
1374 "ci.err.cache": "cache doit être un mapping",
1375 "ci.err.cacheKey": "la clé de cache doit être du texte",
1376 "ci.err.rulesList": "rules doit être une liste",
1377 "ci.err.ruleMap": "une règle doit être un mapping",
1378 "ci.err.ruleIf": "rule.if doit être du texte",
1379 "ci.err.ruleEmpty": "la règle n'a pas de condition",
1380 "ci.err.ruleExpr": "la règle %q n'est pas comprise",
1381 "ci.err.ruleCombine": "la règle %q ne peut pas combiner des conditions avec && ou ||",
1382 "ci.err.regex": "expression régulière invalide dans %q : %s",
1383 "ci.err.when": "when %q n'est pas pris en charge",
1384 "ci.err.timeout": "timeout %q est invalide",
1385 "ci.err.extendsList": "extends doit être un nom de job ou une liste",
1386 "ci.err.listText": "une liste ne peut contenir que du texte",
1387 "ci.err.varText": "les variables doivent être du texte",
1388 "ci.err.bool": "true ou false attendu",
1389 "ci.err.dotdot": "le chemin %q doit rester dans l'espace de travail",
1234}
1390}
internal/runner/runner.go +233 −48
@@ -92,18 +92,27 @@ func (r *Runner) schedule(ctx context.Context, p store.Pipeline) {
92 92 specs := make([]ci.Job, len(jobs))
93 93 status := map[string]string{}
94 94 for i, j := range jobs {
95 specs[i] = ci.Job{Name: j.Name, Stage: j.Stage, Needs: j.Needs, NeedsSet: j.NeedsSet, AllowFailure: j.AllowFailure}
95 specs[i] = ci.Job{Name: j.Name, Stage: j.Stage, Needs: j.Needs, Optional: j.Optional, NeedsSet: j.NeedsSet, AllowFailure: j.AllowFailure, When: j.When}
96 96 status[j.Name] = j.Status
97 97 }
98 98 for _, j := range jobs {
99 if j.Status != "queued" {
99 if j.Status != "queued" && j.Status != "manual" {
100 100 continue
101 101 }
102 102 switch ci.Gate(specs, status, specs[indexOf(jobs, j.Name)]) {
103 103 case "skip":
104 _ = r.Store.SkipJob(ctx, j.ID, "overgeslagen omdat een eerdere job mislukte")
104 _ = r.Store.SkipJob(ctx, j.ID, "skipped because an earlier job failed\n")
105 105 status[j.Name] = "skipped"
106 case "fail":
107 _ = r.Store.FinishJob(ctx, j.ID, "failed", "a required job is not part of this pipeline\n")
108 status[j.Name] = "failed"
106 109 case "ready":
110 if j.Status != "queued" {
111 continue
112 }
113 if busy, _ := r.Store.ResourceBusy(ctx, j.RepoID, j.ResourceGroup, j.ID); busy {
114 continue
115 }
107 116 ok, err := r.Store.MarkJobRunning(ctx, j.ID)
108 117 if err != nil || !ok {
109 118 continue
@@ -131,16 +140,14 @@ func (r *Runner) execute(ctx context.Context, p store.Pipeline, job store.Job) {
131 140 defer func() { <-r.slots }()
132 141 }
133 142 before, _ := r.Store.PipelineByID(context.Background(), p.RepoID, p.ID)
134 logText, err := r.runOne(ctx, p, job)
143 mask, err := r.runOne(ctx, p, job)
135 144 status := "success"
145 extra := ""
136 146 if err != nil {
137 147 status = "failed"
138 if logText != "" {
139 logText += "\n"
140 }
141 logText += err.Error() + "\n"
148 extra = stamp(ci.Mask(err.Error(), mask))
142 149 }
143 if ferr := r.Store.FinishJob(context.Background(), job.ID, status, logText); ferr != nil {
150 if ferr := r.Store.FinishJob(context.Background(), job.ID, status, extra); ferr != nil {
144 151 r.Log.Error("job afronden", "err", ferr)
145 152 }
146 153 final, rerr := r.Store.RefreshPipelineStatus(context.Background(), p.ID)
@@ -175,46 +182,66 @@ func (r *Runner) execute(ctx context.Context, p store.Pipeline, job store.Job) {
175 182 }
176 183 }
177 184
178 func (r *Runner) runOne(ctx context.Context, p store.Pipeline, job store.Job) (string, error) {
179 if r.Timeout > 0 {
185 func (r *Runner) runOne(ctx context.Context, p store.Pipeline, job store.Job) ([]string, error) {
186 limit := r.Timeout
187 if job.Timeout > 0 {
188 limit = job.Timeout
189 }
190 if limit > 0 {
180 191 var cancel context.CancelFunc
181 ctx, cancel = context.WithTimeout(ctx, r.Timeout)
192 ctx, cancel = context.WithTimeout(ctx, limit)
182 193 defer cancel()
183 194 }
184 195 repo, err := r.Store.RepoByID(ctx, p.RepoID)
185 196 if err != nil {
186 return "", err
197 return nil, err
187 198 }
199 sink := &logSink{store: r.Store, jobID: job.ID}
200 sink.section("prepare", "Preparing the docker executor")
188 201 work := filepath.Join(r.storage(), "work", fmt.Sprint(p.ID), job.Name)
189 202 _ = os.RemoveAll(work)
190 203 if err := os.MkdirAll(work, 0o755); err != nil {
191 return "", err
204 sink.closeSection()
205 return nil, err
192 206 }
193 207 bare := gitrepo.Dir(r.storage(), repo.OwnerPath, repo.Name)
194 208 if _, err := execOut(ctx, r.GitBin, "clone", "--local", bare, work); err != nil {
195 return "", fmt.Errorf("clone: %w", err)
209 sink.closeSection()
210 return nil, fmt.Errorf("clone: %w", err)
196 211 }
197 212 if _, err := execOut(ctx, r.GitBin, "-C", work, "checkout", "--detach", p.CommitSHA); err != nil {
198 return "", fmt.Errorf("checkout %s: %w", p.CommitSHA, err)
213 sink.closeSection()
214 return nil, fmt.Errorf("checkout %s: %w", p.CommitSHA, err)
215 }
216 if err := r.restoreCache(repo.ID, job, work); err != nil {
217 sink.closeSection()
218 return nil, err
199 219 }
200 220 if err := r.materialize(ctx, p, job, work); err != nil {
201 return "", err
221 sink.closeSection()
222 return nil, err
202 223 }
203 224 scriptPath := filepath.Join(work, ".bluegit-job.sh")
204 if err := os.WriteFile(scriptPath, []byte(scriptOf(job.Script)), 0o755); err != nil {
205 return "", err
225 if err := os.WriteFile(scriptPath, []byte(shellScript(job)), 0o755); err != nil {
226 sink.closeSection()
227 return nil, err
206 228 }
207 229 token, err := r.Store.CreateJobToken(ctx, job.ID)
208 230 if err != nil {
209 return "", err
231 sink.closeSection()
232 return nil, err
210 233 }
211 env, err := r.environment(ctx, repo, p, job, token)
234 env, mask, err := r.environment(ctx, repo, p, job, token)
212 235 if err != nil {
213 return "", err
236 sink.closeSection()
237 return nil, err
214 238 }
239 sink.secrets = mask
215 240 if _, err := exec.LookPath(r.Docker); err != nil {
216 return "", fmt.Errorf("docker (%s) is niet beschikbaar; de job draait alleen in een container", r.Docker)
241 sink.closeSection()
242 return mask, fmt.Errorf("docker (%s) is not available; the job only runs in a container", r.Docker)
217 243 }
244 sink.closeSection()
218 245 // -e NAAM zonder waarde laat Docker de variabele uit de omgeving van
219 246 // deze client overnemen. Zo blijft een meerregelige sleutel heel en
220 247 // staat de waarde niet in de proceslijst.
@@ -227,27 +254,55 @@ func (r *Runner) runOne(ctx context.Context, p store.Pipeline, job store.Job) (s
227 254 args = append(args, "--entrypoint", "sh", job.Image, "/workspace/.bluegit-job.sh")
228 255 cmd := exec.CommandContext(ctx, r.Docker, args...)
229 256 cmd.Env = cmdEnv
230 var buf bytes.Buffer
231 cmd.Stdout = &buf
232 cmd.Stderr = &buf
257 cmd.Stdout = sink
258 cmd.Stderr = sink
233 259 err = cmd.Run()
234 logText := buf.String()
260 sink.flush()
235 261 if err != nil {
236 return logText, fmt.Errorf("container eindigde met een fout: %w", err)
262 return mask, fmt.Errorf("container ended with an error: %w", err)
237 263 }
238 264 if err := r.collect(ctx, job, work); err != nil {
239 return logText, err
265 return mask, err
266 }
267 if err := r.saveCache(repo.ID, job, work); err != nil {
268 return mask, err
240 269 }
241 return logText, nil
270 return mask, nil
242 271 }
243 272
244 func scriptOf(lines string) string {
273 func shellScript(job store.Job) string {
245 274 var b strings.Builder
246 b.WriteString("#!/bin/sh\nset -e\ncd /workspace\n")
247 b.WriteString(lines)
248 if !strings.HasSuffix(lines, "\n") {
275 b.WriteString("#!/bin/sh\ncd /workspace\nrc=0\n")
276 fmt.Fprintf(&b, "echo \"section_start:before:Preparing environment\"\n")
277 if strings.TrimSpace(job.Before) != "" {
278 b.WriteString("(\nset -e\n")
279 b.WriteString(job.Before)
280 if !strings.HasSuffix(job.Before, "\n") {
281 b.WriteByte('\n')
282 }
283 b.WriteString(")\nbrc=$?\n")
284 } else {
285 b.WriteString("brc=0\n")
286 }
287 b.WriteString("echo \"section_end:before:Preparing environment\"\n")
288 b.WriteString("echo \"section_start:script:Running script\"\n")
289 b.WriteString("if [ \"$brc\" -eq 0 ]; then\n(\nset -e\n")
290 b.WriteString(job.Script)
291 if !strings.HasSuffix(job.Script, "\n") {
249 292 b.WriteByte('\n')
250 293 }
294 b.WriteString(")\nrc=$?\nelse\nrc=$brc\nfi\n")
295 b.WriteString("echo \"section_end:script:Running script\"\n")
296 b.WriteString("echo \"section_start:after:After script\"\n")
297 if strings.TrimSpace(job.After) != "" {
298 b.WriteString("(\nset -e\n")
299 b.WriteString(job.After)
300 if !strings.HasSuffix(job.After, "\n") {
301 b.WriteByte('\n')
302 }
303 b.WriteString(")\n")
304 }
305 b.WriteString("echo \"section_end:after:After script\"\nexit \"$rc\"\n")
251 306 return b.String()
252 307 }
253 308
@@ -318,44 +373,174 @@ func (r *Runner) collect(ctx context.Context, job store.Job, work string) error
318 373 return nil
319 374 }
320 375
321 func (r *Runner) environment(ctx context.Context, repo store.Repo, p store.Pipeline, job store.Job, token string) (map[string]string, error) {
322 branch := strings.TrimPrefix(p.Ref, "refs/heads/")
376 func (r *Runner) environment(ctx context.Context, repo store.Repo, p store.Pipeline, job store.Job, token string) (map[string]string, []string, error) {
377 name, _ := ci.SplitRef(p.Ref)
323 378 patterns, err := r.Store.ProtectedPatterns(ctx, repo.ID)
324 379 if err != nil {
325 return nil, err
380 return nil, nil, err
326 381 }
327 refProtected := access.BranchProtected(patterns, branch)
382 refProtected := access.BranchProtected(patterns, name)
328 383 env := map[string]string{}
384 var mask []string
329 385 if repo.OwnerKind == "group" {
330 386 groupSecrets, err := r.Store.SecretValues(ctx, "group", repo.OwnerID, refProtected)
331 387 if err != nil {
332 return nil, err
388 return nil, nil, err
333 389 }
334 390 for k, v := range groupSecrets {
335 391 env[k] = v
392 mask = append(mask, v)
336 393 }
337 394 }
338 395 repoSecrets, err := r.Store.SecretValues(ctx, "repo", repo.ID, refProtected)
339 396 if err != nil {
340 return nil, err
397 return nil, nil, err
341 398 }
342 399 for k, v := range repoSecrets {
343 400 env[k] = v
401 mask = append(mask, v)
344 402 }
345 403 for k, v := range job.Variables {
346 404 env[k] = v
347 405 }
348 env["CI"] = "true"
349 env["CI_COMMIT_SHA"] = p.CommitSHA
350 env["CI_COMMIT_BRANCH"] = branch
351 env["CI_COMMIT_REF"] = p.Ref
352 env["CI_PROJECT_PATH"] = repo.FullPath()
353 env["CI_PIPELINE_ID"] = fmt.Sprint(p.ID)
406 base := strings.TrimRight(r.BaseURL, "/")
407 pre := ci.Variables(ci.Ref{
408 SHA: p.CommitSHA, FullRef: p.Ref, Title: p.CommitTitle,
409 DefaultBranch: repo.DefaultBranch, Source: p.Source,
410 ProjectPath: repo.FullPath(), ProjectURL: base + "/" + repo.FullPath(),
411 APIURL: base + "/api/v1", PipelineID: p.ID,
412 })
413 for k, v := range pre {
414 env[k] = v
415 }
354 416 env["CI_JOB_ID"] = fmt.Sprint(job.ID)
355 417 env["CI_JOB_NAME"] = job.Name
418 env["CI_JOB_STAGE"] = job.Stage
419 env["CI_JOB_URL"] = base + "/" + repo.FullPath() + "/-/pipelines/" + fmt.Sprint(p.ID) + "/jobs/" + fmt.Sprint(job.ID)
356 420 env["CI_JOB_TOKEN"] = token
357 env["CI_API_URL"] = strings.TrimRight(r.BaseURL, "/") + "/api/v1"
358 return env, nil
421 mask = append(mask, token)
422 return env, mask, nil
423 }
424
425 func (r *Runner) restoreCache(repoID int64, job store.Job, work string) error {
426 if len(job.CachePaths) == 0 {
427 return nil
428 }
429 root := r.cacheDir(repoID, job.CacheKey)
430 for _, rel := range job.CachePaths {
431 src := filepath.Join(root, filepath.FromSlash(rel))
432 if _, err := os.Stat(src); err != nil {
433 continue
434 }
435 if err := copyFile(src, filepath.Join(work, filepath.FromSlash(rel))); err != nil {
436 return err
437 }
438 }
439 return nil
440 }
441
442 func (r *Runner) saveCache(repoID int64, job store.Job, work string) error {
443 if len(job.CachePaths) == 0 {
444 return nil
445 }
446 root := r.cacheDir(repoID, job.CacheKey)
447 for _, rel := range job.CachePaths {
448 src := filepath.Join(work, filepath.FromSlash(rel))
449 info, err := os.Stat(src)
450 if err != nil || info.IsDir() {
451 continue
452 }
453 if err := copyFile(src, filepath.Join(root, filepath.FromSlash(rel))); err != nil {
454 return err
455 }
456 }
457 return nil
458 }
459
460 func (r *Runner) cacheDir(repoID int64, key string) string {
461 if key == "" {
462 key = "default"
463 }
464 var b strings.Builder
465 for _, r := range key {
466 if (r >= 'a' && r <= 'z') || (r >= 'A' && r <= 'Z') || (r >= '0' && r <= '9') || r == '.' || r == '_' || r == '-' {
467 b.WriteRune(r)
468 }
469 }
470 safe := b.String()
471 if safe == "" {
472 safe = "default"
473 }
474 return filepath.Join(r.storage(), "cache", fmt.Sprint(repoID), safe)
475 }
476
477 func stamp(line string) string {
478 return time.Now().UTC().Format(time.RFC3339) + "\t" + line + "\n"
479 }
480
481 type logSink struct {
482 mu sync.Mutex
483 store *store.Store
484 jobID int64
485 secrets []string
486 partial string
487 current string
488 opened time.Time
489 }
490
491 func (l *logSink) Write(p []byte) (int, error) {
492 l.mu.Lock()
493 defer l.mu.Unlock()
494 l.partial += string(p)
495 for {
496 i := strings.IndexByte(l.partial, '\n')
497 if i < 0 {
498 break
499 }
500 l.emit(strings.TrimRight(l.partial[:i], "\r"))
501 l.partial = l.partial[i+1:]
502 }
503 return len(p), nil
504 }
505
506 func (l *logSink) flush() {
507 l.mu.Lock()
508 defer l.mu.Unlock()
509 if strings.TrimSpace(l.partial) != "" {
510 l.emit(l.partial)
511 }
512 l.partial = ""
513 }
514
515 func (l *logSink) emit(line string) {
516 line = ci.Mask(line, l.secrets)
517 if strings.HasPrefix(line, "section_start:") || strings.HasPrefix(line, "section_end:") {
518 if l.current != "" && strings.HasPrefix(line, "section_end:") {
519 dur := int(time.Since(l.opened).Seconds())
520 line = line + "\t" + fmt.Sprintf("%02d:%02d", dur/60, dur%60)
521 l.current = ""
522 }
523 if strings.HasPrefix(line, "section_start:") {
524 l.current = line
525 l.opened = time.Now()
526 }
527 }
528 _ = l.store.AppendJobLog(context.Background(), l.jobID, stamp(line))
529 }
530
531 func (l *logSink) section(name, title string) {
532 l.emit("section_start:" + name + ":" + title)
533 }
534
535 func (l *logSink) closeSection() {
536 if l.current == "" {
537 return
538 }
539 name := "prepare"
540 if parts := strings.SplitN(strings.TrimPrefix(l.current, "section_start:"), ":", 2); len(parts) > 0 && parts[0] != "" {
541 name = parts[0]
542 }
543 l.emit("section_end:" + name + ":done")
359 544 }
360 545
361 546 func copyFile(src, dst string) error {
@@ -92,18 +92,27 @@ func (r *Runner) schedule(ctx context.Context, p store.Pipeline) {
92 specs := make([]ci.Job, len(jobs))
92 specs := make([]ci.Job, len(jobs))
93 status := map[string]string{}
93 status := map[string]string{}
94 for i, j := range jobs {
94 for i, j := range jobs {
95 specs[i] = ci.Job{Name: j.Name, Stage: j.Stage, Needs: j.Needs, NeedsSet: j.NeedsSet, AllowFailure: j.AllowFailure}
95 specs[i] = ci.Job{Name: j.Name, Stage: j.Stage, Needs: j.Needs, Optional: j.Optional, NeedsSet: j.NeedsSet, AllowFailure: j.AllowFailure, When: j.When}
96 status[j.Name] = j.Status
96 status[j.Name] = j.Status
97 }
97 }
98 for _, j := range jobs {
98 for _, j := range jobs {
99 if j.Status != "queued" {
99 if j.Status != "queued" && j.Status != "manual" {
100 continue
100 continue
101 }
101 }
102 switch ci.Gate(specs, status, specs[indexOf(jobs, j.Name)]) {
102 switch ci.Gate(specs, status, specs[indexOf(jobs, j.Name)]) {
103 case "skip":
103 case "skip":
104 _ = r.Store.SkipJob(ctx, j.ID, "overgeslagen omdat een eerdere job mislukte")
104 _ = r.Store.SkipJob(ctx, j.ID, "skipped because an earlier job failed\n")
105 status[j.Name] = "skipped"
105 status[j.Name] = "skipped"
106 case "fail":
107 _ = r.Store.FinishJob(ctx, j.ID, "failed", "a required job is not part of this pipeline\n")
108 status[j.Name] = "failed"
106 case "ready":
109 case "ready":
110 if j.Status != "queued" {
111 continue
112 }
113 if busy, _ := r.Store.ResourceBusy(ctx, j.RepoID, j.ResourceGroup, j.ID); busy {
114 continue
115 }
107 ok, err := r.Store.MarkJobRunning(ctx, j.ID)
116 ok, err := r.Store.MarkJobRunning(ctx, j.ID)
108 if err != nil || !ok {
117 if err != nil || !ok {
109 continue
118 continue
@@ -131,16 +140,14 @@ func (r *Runner) execute(ctx context.Context, p store.Pipeline, job store.Job) {
131 defer func() { <-r.slots }()
140 defer func() { <-r.slots }()
132 }
141 }
133 before, _ := r.Store.PipelineByID(context.Background(), p.RepoID, p.ID)
142 before, _ := r.Store.PipelineByID(context.Background(), p.RepoID, p.ID)
134 logText, err := r.runOne(ctx, p, job)
143 mask, err := r.runOne(ctx, p, job)
135 status := "success"
144 status := "success"
145 extra := ""
136 if err != nil {
146 if err != nil {
137 status = "failed"
147 status = "failed"
138 if logText != "" {
148 extra = stamp(ci.Mask(err.Error(), mask))
139 logText += "\n"
140 }
141 logText += err.Error() + "\n"
142 }
149 }
143 if ferr := r.Store.FinishJob(context.Background(), job.ID, status, logText); ferr != nil {
150 if ferr := r.Store.FinishJob(context.Background(), job.ID, status, extra); ferr != nil {
144 r.Log.Error("job afronden", "err", ferr)
151 r.Log.Error("job afronden", "err", ferr)
145 }
152 }
146 final, rerr := r.Store.RefreshPipelineStatus(context.Background(), p.ID)
153 final, rerr := r.Store.RefreshPipelineStatus(context.Background(), p.ID)
@@ -175,46 +182,66 @@ func (r *Runner) execute(ctx context.Context, p store.Pipeline, job store.Job) {
175 }
182 }
176}
183}
177
184
178func (r *Runner) runOne(ctx context.Context, p store.Pipeline, job store.Job) (string, error) {
185func (r *Runner) runOne(ctx context.Context, p store.Pipeline, job store.Job) ([]string, error) {
179 if r.Timeout > 0 {
186 limit := r.Timeout
187 if job.Timeout > 0 {
188 limit = job.Timeout
189 }
190 if limit > 0 {
180 var cancel context.CancelFunc
191 var cancel context.CancelFunc
181 ctx, cancel = context.WithTimeout(ctx, r.Timeout)
192 ctx, cancel = context.WithTimeout(ctx, limit)
182 defer cancel()
193 defer cancel()
183 }
194 }
184 repo, err := r.Store.RepoByID(ctx, p.RepoID)
195 repo, err := r.Store.RepoByID(ctx, p.RepoID)
185 if err != nil {
196 if err != nil {
186 return "", err
197 return nil, err
187 }
198 }
199 sink := &logSink{store: r.Store, jobID: job.ID}
200 sink.section("prepare", "Preparing the docker executor")
188 work := filepath.Join(r.storage(), "work", fmt.Sprint(p.ID), job.Name)
201 work := filepath.Join(r.storage(), "work", fmt.Sprint(p.ID), job.Name)
189 _ = os.RemoveAll(work)
202 _ = os.RemoveAll(work)
190 if err := os.MkdirAll(work, 0o755); err != nil {
203 if err := os.MkdirAll(work, 0o755); err != nil {
191 return "", err
204 sink.closeSection()
205 return nil, err
192 }
206 }
193 bare := gitrepo.Dir(r.storage(), repo.OwnerPath, repo.Name)
207 bare := gitrepo.Dir(r.storage(), repo.OwnerPath, repo.Name)
194 if _, err := execOut(ctx, r.GitBin, "clone", "--local", bare, work); err != nil {
208 if _, err := execOut(ctx, r.GitBin, "clone", "--local", bare, work); err != nil {
195 return "", fmt.Errorf("clone: %w", err)
209 sink.closeSection()
210 return nil, fmt.Errorf("clone: %w", err)
196 }
211 }
197 if _, err := execOut(ctx, r.GitBin, "-C", work, "checkout", "--detach", p.CommitSHA); err != nil {
212 if _, err := execOut(ctx, r.GitBin, "-C", work, "checkout", "--detach", p.CommitSHA); err != nil {
198 return "", fmt.Errorf("checkout %s: %w", p.CommitSHA, err)
213 sink.closeSection()
214 return nil, fmt.Errorf("checkout %s: %w", p.CommitSHA, err)
215 }
216 if err := r.restoreCache(repo.ID, job, work); err != nil {
217 sink.closeSection()
218 return nil, err
199 }
219 }
200 if err := r.materialize(ctx, p, job, work); err != nil {
220 if err := r.materialize(ctx, p, job, work); err != nil {
201 return "", err
221 sink.closeSection()
222 return nil, err
202 }
223 }
203 scriptPath := filepath.Join(work, ".bluegit-job.sh")
224 scriptPath := filepath.Join(work, ".bluegit-job.sh")
204 if err := os.WriteFile(scriptPath, []byte(scriptOf(job.Script)), 0o755); err != nil {
225 if err := os.WriteFile(scriptPath, []byte(shellScript(job)), 0o755); err != nil {
205 return "", err
226 sink.closeSection()
227 return nil, err
206 }
228 }
207 token, err := r.Store.CreateJobToken(ctx, job.ID)
229 token, err := r.Store.CreateJobToken(ctx, job.ID)
208 if err != nil {
230 if err != nil {
209 return "", err
231 sink.closeSection()
232 return nil, err
210 }
233 }
211 env, err := r.environment(ctx, repo, p, job, token)
234 env, mask, err := r.environment(ctx, repo, p, job, token)
212 if err != nil {
235 if err != nil {
213 return "", err
236 sink.closeSection()
237 return nil, err
214 }
238 }
239 sink.secrets = mask
215 if _, err := exec.LookPath(r.Docker); err != nil {
240 if _, err := exec.LookPath(r.Docker); err != nil {
216 return "", fmt.Errorf("docker (%s) is niet beschikbaar; de job draait alleen in een container", r.Docker)
241 sink.closeSection()
242 return mask, fmt.Errorf("docker (%s) is not available; the job only runs in a container", r.Docker)
217 }
243 }
244 sink.closeSection()
218 // -e NAAM zonder waarde laat Docker de variabele uit de omgeving van
245 // -e NAAM zonder waarde laat Docker de variabele uit de omgeving van
219 // deze client overnemen. Zo blijft een meerregelige sleutel heel en
246 // deze client overnemen. Zo blijft een meerregelige sleutel heel en
220 // staat de waarde niet in de proceslijst.
247 // staat de waarde niet in de proceslijst.
@@ -227,27 +254,55 @@ func (r *Runner) runOne(ctx context.Context, p store.Pipeline, job store.Job) (s
227 args = append(args, "--entrypoint", "sh", job.Image, "/workspace/.bluegit-job.sh")
254 args = append(args, "--entrypoint", "sh", job.Image, "/workspace/.bluegit-job.sh")
228 cmd := exec.CommandContext(ctx, r.Docker, args...)
255 cmd := exec.CommandContext(ctx, r.Docker, args...)
229 cmd.Env = cmdEnv
256 cmd.Env = cmdEnv
230 var buf bytes.Buffer
257 cmd.Stdout = sink
231 cmd.Stdout = &buf
258 cmd.Stderr = sink
232 cmd.Stderr = &buf
233 err = cmd.Run()
259 err = cmd.Run()
234 logText := buf.String()
260 sink.flush()
235 if err != nil {
261 if err != nil {
236 return logText, fmt.Errorf("container eindigde met een fout: %w", err)
262 return mask, fmt.Errorf("container ended with an error: %w", err)
237 }
263 }
238 if err := r.collect(ctx, job, work); err != nil {
264 if err := r.collect(ctx, job, work); err != nil {
239 return logText, err
265 return mask, err
266 }
267 if err := r.saveCache(repo.ID, job, work); err != nil {
268 return mask, err
240 }
269 }
241 return logText, nil
270 return mask, nil
242}
271}
243
272
244func scriptOf(lines string) string {
273func shellScript(job store.Job) string {
245 var b strings.Builder
274 var b strings.Builder
246 b.WriteString("#!/bin/sh\nset -e\ncd /workspace\n")
275 b.WriteString("#!/bin/sh\ncd /workspace\nrc=0\n")
247 b.WriteString(lines)
276 fmt.Fprintf(&b, "echo \"section_start:before:Preparing environment\"\n")
248 if !strings.HasSuffix(lines, "\n") {
277 if strings.TrimSpace(job.Before) != "" {
278 b.WriteString("(\nset -e\n")
279 b.WriteString(job.Before)
280 if !strings.HasSuffix(job.Before, "\n") {
281 b.WriteByte('\n')
282 }
283 b.WriteString(")\nbrc=$?\n")
284 } else {
285 b.WriteString("brc=0\n")
286 }
287 b.WriteString("echo \"section_end:before:Preparing environment\"\n")
288 b.WriteString("echo \"section_start:script:Running script\"\n")
289 b.WriteString("if [ \"$brc\" -eq 0 ]; then\n(\nset -e\n")
290 b.WriteString(job.Script)
291 if !strings.HasSuffix(job.Script, "\n") {
249 b.WriteByte('\n')
292 b.WriteByte('\n')
250 }
293 }
294 b.WriteString(")\nrc=$?\nelse\nrc=$brc\nfi\n")
295 b.WriteString("echo \"section_end:script:Running script\"\n")
296 b.WriteString("echo \"section_start:after:After script\"\n")
297 if strings.TrimSpace(job.After) != "" {
298 b.WriteString("(\nset -e\n")
299 b.WriteString(job.After)
300 if !strings.HasSuffix(job.After, "\n") {
301 b.WriteByte('\n')
302 }
303 b.WriteString(")\n")
304 }
305 b.WriteString("echo \"section_end:after:After script\"\nexit \"$rc\"\n")
251 return b.String()
306 return b.String()
252}
307}
253
308
@@ -318,44 +373,174 @@ func (r *Runner) collect(ctx context.Context, job store.Job, work string) error
318 return nil
373 return nil
319}
374}
320
375
321func (r *Runner) environment(ctx context.Context, repo store.Repo, p store.Pipeline, job store.Job, token string) (map[string]string, error) {
376func (r *Runner) environment(ctx context.Context, repo store.Repo, p store.Pipeline, job store.Job, token string) (map[string]string, []string, error) {
322 branch := strings.TrimPrefix(p.Ref, "refs/heads/")
377 name, _ := ci.SplitRef(p.Ref)
323 patterns, err := r.Store.ProtectedPatterns(ctx, repo.ID)
378 patterns, err := r.Store.ProtectedPatterns(ctx, repo.ID)
324 if err != nil {
379 if err != nil {
325 return nil, err
380 return nil, nil, err
326 }
381 }
327 refProtected := access.BranchProtected(patterns, branch)
382 refProtected := access.BranchProtected(patterns, name)
328 env := map[string]string{}
383 env := map[string]string{}
384 var mask []string
329 if repo.OwnerKind == "group" {
385 if repo.OwnerKind == "group" {
330 groupSecrets, err := r.Store.SecretValues(ctx, "group", repo.OwnerID, refProtected)
386 groupSecrets, err := r.Store.SecretValues(ctx, "group", repo.OwnerID, refProtected)
331 if err != nil {
387 if err != nil {
332 return nil, err
388 return nil, nil, err
333 }
389 }
334 for k, v := range groupSecrets {
390 for k, v := range groupSecrets {
335 env[k] = v
391 env[k] = v
392 mask = append(mask, v)
336 }
393 }
337 }
394 }
338 repoSecrets, err := r.Store.SecretValues(ctx, "repo", repo.ID, refProtected)
395 repoSecrets, err := r.Store.SecretValues(ctx, "repo", repo.ID, refProtected)
339 if err != nil {
396 if err != nil {
340 return nil, err
397 return nil, nil, err
341 }
398 }
342 for k, v := range repoSecrets {
399 for k, v := range repoSecrets {
343 env[k] = v
400 env[k] = v
401 mask = append(mask, v)
344 }
402 }
345 for k, v := range job.Variables {
403 for k, v := range job.Variables {
346 env[k] = v
404 env[k] = v
347 }
405 }
348 env["CI"] = "true"
406 base := strings.TrimRight(r.BaseURL, "/")
349 env["CI_COMMIT_SHA"] = p.CommitSHA
407 pre := ci.Variables(ci.Ref{
350 env["CI_COMMIT_BRANCH"] = branch
408 SHA: p.CommitSHA, FullRef: p.Ref, Title: p.CommitTitle,
351 env["CI_COMMIT_REF"] = p.Ref
409 DefaultBranch: repo.DefaultBranch, Source: p.Source,
352 env["CI_PROJECT_PATH"] = repo.FullPath()
410 ProjectPath: repo.FullPath(), ProjectURL: base + "/" + repo.FullPath(),
353 env["CI_PIPELINE_ID"] = fmt.Sprint(p.ID)
411 APIURL: base + "/api/v1", PipelineID: p.ID,
412 })
413 for k, v := range pre {
414 env[k] = v
415 }
354 env["CI_JOB_ID"] = fmt.Sprint(job.ID)
416 env["CI_JOB_ID"] = fmt.Sprint(job.ID)
355 env["CI_JOB_NAME"] = job.Name
417 env["CI_JOB_NAME"] = job.Name
418 env["CI_JOB_STAGE"] = job.Stage
419 env["CI_JOB_URL"] = base + "/" + repo.FullPath() + "/-/pipelines/" + fmt.Sprint(p.ID) + "/jobs/" + fmt.Sprint(job.ID)
356 env["CI_JOB_TOKEN"] = token
420 env["CI_JOB_TOKEN"] = token
357 env["CI_API_URL"] = strings.TrimRight(r.BaseURL, "/") + "/api/v1"
421 mask = append(mask, token)
358 return env, nil
422 return env, mask, nil
423}
424
425func (r *Runner) restoreCache(repoID int64, job store.Job, work string) error {
426 if len(job.CachePaths) == 0 {
427 return nil
428 }
429 root := r.cacheDir(repoID, job.CacheKey)
430 for _, rel := range job.CachePaths {
431 src := filepath.Join(root, filepath.FromSlash(rel))
432 if _, err := os.Stat(src); err != nil {
433 continue
434 }
435 if err := copyFile(src, filepath.Join(work, filepath.FromSlash(rel))); err != nil {
436 return err
437 }
438 }
439 return nil
440}
441
442func (r *Runner) saveCache(repoID int64, job store.Job, work string) error {
443 if len(job.CachePaths) == 0 {
444 return nil
445 }
446 root := r.cacheDir(repoID, job.CacheKey)
447 for _, rel := range job.CachePaths {
448 src := filepath.Join(work, filepath.FromSlash(rel))
449 info, err := os.Stat(src)
450 if err != nil || info.IsDir() {
451 continue
452 }
453 if err := copyFile(src, filepath.Join(root, filepath.FromSlash(rel))); err != nil {
454 return err
455 }
456 }
457 return nil
458}
459
460func (r *Runner) cacheDir(repoID int64, key string) string {
461 if key == "" {
462 key = "default"
463 }
464 var b strings.Builder
465 for _, r := range key {
466 if (r >= 'a' && r <= 'z') || (r >= 'A' && r <= 'Z') || (r >= '0' && r <= '9') || r == '.' || r == '_' || r == '-' {
467 b.WriteRune(r)
468 }
469 }
470 safe := b.String()
471 if safe == "" {
472 safe = "default"
473 }
474 return filepath.Join(r.storage(), "cache", fmt.Sprint(repoID), safe)
475}
476
477func stamp(line string) string {
478 return time.Now().UTC().Format(time.RFC3339) + "\t" + line + "\n"
479}
480
481type logSink struct {
482 mu sync.Mutex
483 store *store.Store
484 jobID int64
485 secrets []string
486 partial string
487 current string
488 opened time.Time
489}
490
491func (l *logSink) Write(p []byte) (int, error) {
492 l.mu.Lock()
493 defer l.mu.Unlock()
494 l.partial += string(p)
495 for {
496 i := strings.IndexByte(l.partial, '\n')
497 if i < 0 {
498 break
499 }
500 l.emit(strings.TrimRight(l.partial[:i], "\r"))
501 l.partial = l.partial[i+1:]
502 }
503 return len(p), nil
504}
505
506func (l *logSink) flush() {
507 l.mu.Lock()
508 defer l.mu.Unlock()
509 if strings.TrimSpace(l.partial) != "" {
510 l.emit(l.partial)
511 }
512 l.partial = ""
513}
514
515func (l *logSink) emit(line string) {
516 line = ci.Mask(line, l.secrets)
517 if strings.HasPrefix(line, "section_start:") || strings.HasPrefix(line, "section_end:") {
518 if l.current != "" && strings.HasPrefix(line, "section_end:") {
519 dur := int(time.Since(l.opened).Seconds())
520 line = line + "\t" + fmt.Sprintf("%02d:%02d", dur/60, dur%60)
521 l.current = ""
522 }
523 if strings.HasPrefix(line, "section_start:") {
524 l.current = line
525 l.opened = time.Now()
526 }
527 }
528 _ = l.store.AppendJobLog(context.Background(), l.jobID, stamp(line))
529}
530
531func (l *logSink) section(name, title string) {
532 l.emit("section_start:" + name + ":" + title)
533}
534
535func (l *logSink) closeSection() {
536 if l.current == "" {
537 return
538 }
539 name := "prepare"
540 if parts := strings.SplitN(strings.TrimPrefix(l.current, "section_start:"), ":", 2); len(parts) > 0 && parts[0] != "" {
541 name = parts[0]
542 }
543 l.emit("section_end:" + name + ":done")
359}
544}
360
545
361func copyFile(src, dst string) error {
546func copyFile(src, dst string) error {
internal/server/events.go +66 −14
@@ -28,12 +28,19 @@ func (s *Server) AfterPush(repo store.Repo, user store.User, before, after map[s
28 28 if before[ref] == sha {
29 29 continue
30 30 }
31 if isZero(sha) {
32 continue
33 }
34 if strings.HasPrefix(ref, "refs/tags/") {
35 s.startPipeline(context.Background(), repo, sha, ref, actor, "push", before[ref])
36 continue
37 }
31 38 branch, ok := gitrepo.BranchOfRef(ref)
32 if !ok || isZero(sha) {
39 if !ok {
33 40 continue
34 41 }
35 42 s.recordPushContributions(repo, user, map[string]string{ref: before[ref]}, map[string]string{ref: sha})
36 s.startPipeline(context.Background(), repo, sha, ref, actor)
43 s.startPipeline(context.Background(), repo, sha, ref, actor, "push", before[ref])
37 44 web := strings.TrimRight(s.Cfg.BaseURL, "/") + "/" + repo.FullPath()
38 45 s.Dispatch(repo.ID, webhook.Event{
39 46 Type: webhook.EventPush,
@@ -51,7 +58,7 @@ func isZero(sha string) bool {
51 58 return strings.Trim(sha, "0") == ""
52 59 }
53 60
54 func (s *Server) startPipeline(ctx context.Context, repo store.Repo, sha, ref, actor string) {
61 func (s *Server) startPipeline(ctx context.Context, repo store.Repo, sha, ref, actor, source, before string) {
55 62 if !s.Cfg.CI.Enabled || s.StoragePath() == "" {
56 63 return
57 64 }
@@ -60,17 +67,26 @@ func (s *Server) startPipeline(ctx context.Context, repo store.Repo, sha, ref, a
60 67 if err != nil {
61 68 return
62 69 }
63 spec, err := ci.Parse(raw)
64 branch, _ := gitrepo.BranchOfRef(ref)
65 if branch == "" {
66 branch = ref
70 read := func(path string) ([]byte, error) {
71 if path == "" || strings.Contains(path, "..") || strings.HasPrefix(path, "/") {
72 return nil, i18n.E("ci.err.rel", path)
73 }
74 return exec.CommandContext(ctx, s.Git, "-C", dir, "show", sha+":"+path).Output()
67 75 }
76 spec, err := ci.ParseIn(raw, read)
68 77 if err != nil {
69 _ = s.Store.RecordConfigFailure(ctx, repo.ID, sha, ref, "blue-ci-cd.yml: "+err.Error())
78 _ = s.Store.RecordConfigFailure(ctx, repo.ID, sha, ref, err.Error())
70 79 return
71 80 }
72 env := map[string]string{"CI_COMMIT_BRANCH": branch, "CI_COMMIT_SHA": sha, "CI_COMMIT_REF": ref}
73 active, err := spec.ActiveJobs(env)
81 name, kind := ci.SplitRef(ref)
82 title := strings.TrimSpace(gitOne(ctx, s.Git, dir, "log", "-1", "--format=%s", sha))
83 files := gitLines(ctx, s.Git, dir, "ls-tree", "-r", "--name-only", sha)
84 changed := changedFiles(ctx, s.Git, dir, sha, before, files)
85 env := ci.Variables(ci.Ref{
86 SHA: sha, FullRef: ref, Name: name, Kind: kind, Title: title,
87 DefaultBranch: repo.DefaultBranch, Source: source, ProjectPath: repo.FullPath(),
88 })
89 active, err := spec.Select(ci.Eval{Env: env, Changed: changed, Files: files})
74 90 if err != nil {
75 91 _ = s.Store.RecordConfigFailure(ctx, repo.ID, sha, ref, err.Error())
76 92 return
@@ -97,15 +113,51 @@ func (s *Server) startPipeline(ctx context.Context, repo store.Repo, sha, ref, a
97 113 }
98 114 jobs = append(jobs, store.NewJob{
99 115 Name: j.Name, Stage: j.Stage, Position: i, Image: j.Image,
100 Script: strings.Join(j.Script, "\n"), Needs: needs, NeedsSet: j.NeedsSet,
116 Before: strings.Join(j.Before, "\n"), Script: strings.Join(j.Script, "\n"), After: strings.Join(j.After, "\n"),
117 Needs: needs, Optional: j.Optional, NeedsSet: j.NeedsSet,
101 118 Variables: vars, ArtifactPaths: paths, ArtifactExpire: j.Artifacts.ExpireIn,
102 AllowFailure: j.AllowFailure,
119 AllowFailure: j.AllowFailure, When: j.When, Timeout: j.Timeout, Tags: j.Tags,
120 CachePaths: j.Cache.Paths, CacheKey: j.Cache.Key, ResourceGroup: j.ResourceGroup,
103 121 })
104 122 }
105 if _, err := s.Store.CreatePipeline(ctx, repo.ID, sha, ref, jobs); err != nil && s.Log != nil {
123 if _, err := s.Store.CreatePipeline(ctx, repo.ID, sha, ref, source, actor, title, jobs); err != nil && s.Log != nil {
106 124 s.Log.Error("pipeline", "err", err)
107 125 }
108 _ = actor
126 }
127
128 func gitOne(ctx context.Context, gitBin, dir string, args ...string) string {
129 out, err := exec.CommandContext(ctx, gitBin, append([]string{"-C", dir}, args...)...).Output()
130 if err != nil {
131 return ""
132 }
133 return strings.TrimSpace(string(out))
134 }
135
136 func gitLines(ctx context.Context, gitBin, dir string, args ...string) []string {
137 out := gitOne(ctx, gitBin, dir, args...)
138 if out == "" {
139 return []string{}
140 }
141 return strings.Split(out, "\n")
142 }
143
144 func changedFiles(ctx context.Context, gitBin, dir, sha, before string, all []string) []string {
145 if before == "" || strings.Trim(before, "0") == "" {
146 parent := gitOne(ctx, gitBin, dir, "rev-parse", "--verify", sha+"^")
147 if parent == "" {
148 return all
149 }
150 before = parent
151 }
152 out, err := exec.CommandContext(ctx, gitBin, "-C", dir, "diff", "--name-only", before, sha).Output()
153 if err != nil {
154 return nil
155 }
156 text := strings.TrimSpace(string(out))
157 if text == "" {
158 return []string{}
159 }
160 return strings.Split(text, "\n")
109 161 }
110 162
111 163 func pushEvent(repo store.Repo, kind, branch, commit, actor, summary, path string) webhook.Event {
@@ -28,12 +28,19 @@ func (s *Server) AfterPush(repo store.Repo, user store.User, before, after map[s
28 if before[ref] == sha {
28 if before[ref] == sha {
29 continue
29 continue
30 }
30 }
31 if isZero(sha) {
32 continue
33 }
34 if strings.HasPrefix(ref, "refs/tags/") {
35 s.startPipeline(context.Background(), repo, sha, ref, actor, "push", before[ref])
36 continue
37 }
31 branch, ok := gitrepo.BranchOfRef(ref)
38 branch, ok := gitrepo.BranchOfRef(ref)
32 if !ok || isZero(sha) {
39 if !ok {
33 continue
40 continue
34 }
41 }
35 s.recordPushContributions(repo, user, map[string]string{ref: before[ref]}, map[string]string{ref: sha})
42 s.recordPushContributions(repo, user, map[string]string{ref: before[ref]}, map[string]string{ref: sha})
36 s.startPipeline(context.Background(), repo, sha, ref, actor)
43 s.startPipeline(context.Background(), repo, sha, ref, actor, "push", before[ref])
37 web := strings.TrimRight(s.Cfg.BaseURL, "/") + "/" + repo.FullPath()
44 web := strings.TrimRight(s.Cfg.BaseURL, "/") + "/" + repo.FullPath()
38 s.Dispatch(repo.ID, webhook.Event{
45 s.Dispatch(repo.ID, webhook.Event{
39 Type: webhook.EventPush,
46 Type: webhook.EventPush,
@@ -51,7 +58,7 @@ func isZero(sha string) bool {
51 return strings.Trim(sha, "0") == ""
58 return strings.Trim(sha, "0") == ""
52}
59}
53
60
54func (s *Server) startPipeline(ctx context.Context, repo store.Repo, sha, ref, actor string) {
61func (s *Server) startPipeline(ctx context.Context, repo store.Repo, sha, ref, actor, source, before string) {
55 if !s.Cfg.CI.Enabled || s.StoragePath() == "" {
62 if !s.Cfg.CI.Enabled || s.StoragePath() == "" {
56 return
63 return
57 }
64 }
@@ -60,17 +67,26 @@ func (s *Server) startPipeline(ctx context.Context, repo store.Repo, sha, ref, a
60 if err != nil {
67 if err != nil {
61 return
68 return
62 }
69 }
63 spec, err := ci.Parse(raw)
70 read := func(path string) ([]byte, error) {
64 branch, _ := gitrepo.BranchOfRef(ref)
71 if path == "" || strings.Contains(path, "..") || strings.HasPrefix(path, "/") {
65 if branch == "" {
72 return nil, i18n.E("ci.err.rel", path)
66 branch = ref
73 }
74 return exec.CommandContext(ctx, s.Git, "-C", dir, "show", sha+":"+path).Output()
67 }
75 }
76 spec, err := ci.ParseIn(raw, read)
68 if err != nil {
77 if err != nil {
69 _ = s.Store.RecordConfigFailure(ctx, repo.ID, sha, ref, "blue-ci-cd.yml: "+err.Error())
78 _ = s.Store.RecordConfigFailure(ctx, repo.ID, sha, ref, err.Error())
70 return
79 return
71 }
80 }
72 env := map[string]string{"CI_COMMIT_BRANCH": branch, "CI_COMMIT_SHA": sha, "CI_COMMIT_REF": ref}
81 name, kind := ci.SplitRef(ref)
73 active, err := spec.ActiveJobs(env)
82 title := strings.TrimSpace(gitOne(ctx, s.Git, dir, "log", "-1", "--format=%s", sha))
83 files := gitLines(ctx, s.Git, dir, "ls-tree", "-r", "--name-only", sha)
84 changed := changedFiles(ctx, s.Git, dir, sha, before, files)
85 env := ci.Variables(ci.Ref{
86 SHA: sha, FullRef: ref, Name: name, Kind: kind, Title: title,
87 DefaultBranch: repo.DefaultBranch, Source: source, ProjectPath: repo.FullPath(),
88 })
89 active, err := spec.Select(ci.Eval{Env: env, Changed: changed, Files: files})
74 if err != nil {
90 if err != nil {
75 _ = s.Store.RecordConfigFailure(ctx, repo.ID, sha, ref, err.Error())
91 _ = s.Store.RecordConfigFailure(ctx, repo.ID, sha, ref, err.Error())
76 return
92 return
@@ -97,15 +113,51 @@ func (s *Server) startPipeline(ctx context.Context, repo store.Repo, sha, ref, a
97 }
113 }
98 jobs = append(jobs, store.NewJob{
114 jobs = append(jobs, store.NewJob{
99 Name: j.Name, Stage: j.Stage, Position: i, Image: j.Image,
115 Name: j.Name, Stage: j.Stage, Position: i, Image: j.Image,
100 Script: strings.Join(j.Script, "\n"), Needs: needs, NeedsSet: j.NeedsSet,
116 Before: strings.Join(j.Before, "\n"), Script: strings.Join(j.Script, "\n"), After: strings.Join(j.After, "\n"),
117 Needs: needs, Optional: j.Optional, NeedsSet: j.NeedsSet,
101 Variables: vars, ArtifactPaths: paths, ArtifactExpire: j.Artifacts.ExpireIn,
118 Variables: vars, ArtifactPaths: paths, ArtifactExpire: j.Artifacts.ExpireIn,
102 AllowFailure: j.AllowFailure,
119 AllowFailure: j.AllowFailure, When: j.When, Timeout: j.Timeout, Tags: j.Tags,
120 CachePaths: j.Cache.Paths, CacheKey: j.Cache.Key, ResourceGroup: j.ResourceGroup,
103 })
121 })
104 }
122 }
105 if _, err := s.Store.CreatePipeline(ctx, repo.ID, sha, ref, jobs); err != nil && s.Log != nil {
123 if _, err := s.Store.CreatePipeline(ctx, repo.ID, sha, ref, source, actor, title, jobs); err != nil && s.Log != nil {
106 s.Log.Error("pipeline", "err", err)
124 s.Log.Error("pipeline", "err", err)
107 }
125 }
108 _ = actor
126}
127
128func gitOne(ctx context.Context, gitBin, dir string, args ...string) string {
129 out, err := exec.CommandContext(ctx, gitBin, append([]string{"-C", dir}, args...)...).Output()
130 if err != nil {
131 return ""
132 }
133 return strings.TrimSpace(string(out))
134}
135
136func gitLines(ctx context.Context, gitBin, dir string, args ...string) []string {
137 out := gitOne(ctx, gitBin, dir, args...)
138 if out == "" {
139 return []string{}
140 }
141 return strings.Split(out, "\n")
142}
143
144func changedFiles(ctx context.Context, gitBin, dir, sha, before string, all []string) []string {
145 if before == "" || strings.Trim(before, "0") == "" {
146 parent := gitOne(ctx, gitBin, dir, "rev-parse", "--verify", sha+"^")
147 if parent == "" {
148 return all
149 }
150 before = parent
151 }
152 out, err := exec.CommandContext(ctx, gitBin, "-C", dir, "diff", "--name-only", before, sha).Output()
153 if err != nil {
154 return nil
155 }
156 text := strings.TrimSpace(string(out))
157 if text == "" {
158 return []string{}
159 }
160 return strings.Split(text, "\n")
109}
161}
110
162
111func pushEvent(repo store.Repo, kind, branch, commit, actor, summary, path string) webhook.Event {
163func pushEvent(repo store.Repo, kind, branch, commit, actor, summary, path string) webhook.Event {
internal/server/repo_handlers.go +1 −1
@@ -391,7 +391,7 @@ func (s *Server) runPipeline(w http.ResponseWriter, r *http.Request) {
391 391 s.redirect(w, r, repoURL(repo)+"/-/pipelines", "", s.tr(r, "err.branchMissing"))
392 392 return
393 393 }
394 s.startPipeline(r.Context(), repo, shaOut, "refs/heads/"+ref, u.Username)
394 s.startPipeline(r.Context(), repo, shaOut, "refs/heads/"+ref, u.Username, "web", "")
395 395 s.redirect(w, r, repoURL(repo)+"/-/pipelines", s.tr(r, "flash.pipeline"), "")
396 396 }
397 397
@@ -391,7 +391,7 @@ func (s *Server) runPipeline(w http.ResponseWriter, r *http.Request) {
391 s.redirect(w, r, repoURL(repo)+"/-/pipelines", "", s.tr(r, "err.branchMissing"))
391 s.redirect(w, r, repoURL(repo)+"/-/pipelines", "", s.tr(r, "err.branchMissing"))
392 return
392 return
393 }
393 }
394 s.startPipeline(r.Context(), repo, shaOut, "refs/heads/"+ref, u.Username)
394 s.startPipeline(r.Context(), repo, shaOut, "refs/heads/"+ref, u.Username, "web", "")
395 s.redirect(w, r, repoURL(repo)+"/-/pipelines", s.tr(r, "flash.pipeline"), "")
395 s.redirect(w, r, repoURL(repo)+"/-/pipelines", s.tr(r, "flash.pipeline"), "")
396}
396}
397
397
internal/store/flow.go +129 −42
@@ -235,23 +235,37 @@ type NewJob struct {
235 235 Stage string
236 236 Position int
237 237 Image string
238 Before string
238 239 Script string
240 After string
239 241 Needs []string
242 Optional []string
240 243 NeedsSet bool
241 244 Variables map[string]string
242 245 ArtifactPaths []string
243 246 ArtifactExpire time.Duration
244 247 AllowFailure bool
248 When string
249 Timeout time.Duration
250 Tags []string
251 CachePaths []string
252 CacheKey string
253 ResourceGroup string
245 254 }
246 255
247 256 // Pipeline is een draai van blue-ci-cd.yml.
248 257 type Pipeline struct {
249 ID int64
250 RepoID int64
251 CommitSHA string
252 Ref string
253 Status string
254 CreatedAt time.Time
258 ID int64
259 RepoID int64
260 CommitSHA string
261 Ref string
262 Status string
263 Source string
264 Actor string
265 CommitTitle string
266 CreatedAt time.Time
267 StartedAt time.Time
268 FinishedAt time.Time
255 269 }
256 270
257 271 // Job is één container in een pipeline.
@@ -264,16 +278,27 @@ type Job struct {
264 278 Position int
265 279 Status string
266 280 Image string
281 Before string
267 282 Script string
283 After string
268 284 Needs []string
285 Optional []string
269 286 NeedsSet bool
270 287 Variables map[string]string
271 288 ArtifactPaths []string
272 289 ArtifactExpire time.Duration
273 290 AllowFailure bool
291 When string
292 Timeout time.Duration
293 Tags []string
294 CachePaths []string
295 CacheKey string
296 ResourceGroup string
274 297 Log string
275 298 CommitSHA string
276 299 Ref string
300 StartedAt time.Time
301 FinishedAt time.Time
277 302 }
278 303
279 304 // RecordConfigFailure bewaart een pipeline die niet kon starten.
@@ -298,13 +323,13 @@ func (s *Store) RecordConfigFailure(ctx context.Context, repoID int64, sha, ref,
298 323 }
299 324
300 325 // CreatePipeline maakt een pipeline met jobs in de wachtrij.
301 func (s *Store) CreatePipeline(ctx context.Context, repoID int64, sha, ref string, jobs []NewJob) (int64, error) {
326 func (s *Store) CreatePipeline(ctx context.Context, repoID int64, sha, ref, source, actor, title string, jobs []NewJob) (int64, error) {
302 327 tx, err := s.db.BeginTx(ctx, nil)
303 328 if err != nil {
304 329 return 0, err
305 330 }
306 331 defer func() { _ = tx.Rollback() }()
307 res, err := tx.ExecContext(ctx, `INSERT INTO pipelines (repo_id, commit_sha, ref, status, created_at) VALUES (?, ?, ?, 'queued', ?)`, repoID, sha, ref, now())
332 res, err := tx.ExecContext(ctx, `INSERT INTO pipelines (repo_id, commit_sha, ref, status, source, actor, commit_title, created_at) VALUES (?, ?, ?, 'queued', ?, ?, ?, ?)`, repoID, sha, ref, source, actor, title, now())
308 333 if err != nil {
309 334 return 0, err
310 335 }
@@ -319,9 +344,26 @@ func (s *Store) CreatePipeline(ctx context.Context, repoID int64, sha, ref strin
319 344 if j.ArtifactPaths == nil {
320 345 j.ArtifactPaths = []string{}
321 346 }
322 _, err := tx.ExecContext(ctx, `INSERT INTO jobs (pipeline_id, repo_id, name, stage, position, status, image, script, needs_json, needs_set, variables_json, artifact_paths, artifact_expire_seconds, allow_failure, log)
323 VALUES (?, ?, ?, ?, ?, 'queued', ?, ?, ?, ?, ?, ?, ?, ?, '')`,
324 id, repoID, j.Name, j.Stage, j.Position, j.Image, j.Script, mustJSON(j.Needs), boolInt(j.NeedsSet), mustJSON(j.Variables), mustJSON(j.ArtifactPaths), int(j.ArtifactExpire/time.Second), boolInt(j.AllowFailure))
347 if j.Optional == nil {
348 j.Optional = []string{}
349 }
350 if j.Tags == nil {
351 j.Tags = []string{}
352 }
353 if j.CachePaths == nil {
354 j.CachePaths = []string{}
355 }
356 if j.When == "" {
357 j.When = "on_success"
358 }
359 status := "queued"
360 if j.When == "manual" {
361 status = "manual"
362 }
363 _, err := tx.ExecContext(ctx, `INSERT INTO jobs (pipeline_id, repo_id, name, stage, position, status, image, script, needs_json, needs_set, variables_json, artifact_paths, artifact_expire_seconds, allow_failure, log, before_script, after_script, when_mode, timeout_seconds, tags_json, cache_paths, cache_key, resource_group, optional_json)
364 VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, '', ?, ?, ?, ?, ?, ?, ?, ?, ?)`,
365 id, repoID, j.Name, j.Stage, j.Position, status, j.Image, j.Script, mustJSON(j.Needs), boolInt(j.NeedsSet), mustJSON(j.Variables), mustJSON(j.ArtifactPaths), int(j.ArtifactExpire/time.Second), boolInt(j.AllowFailure),
366 j.Before, j.After, j.When, int(j.Timeout/time.Second), mustJSON(j.Tags), mustJSON(j.CachePaths), j.CacheKey, j.ResourceGroup, mustJSON(j.Optional))
325 367 if err != nil {
326 368 return 0, err
327 369 }
@@ -334,7 +376,7 @@ func (s *Store) CreatePipeline(ctx context.Context, repoID int64, sha, ref strin
334 376
335 377 // ListPipelines geeft de nieuwste pipelines.
336 378 func (s *Store) ListPipelines(ctx context.Context, repoID int64) ([]Pipeline, error) {
337 rows, err := s.db.QueryContext(ctx, `SELECT id, repo_id, commit_sha, ref, status, created_at FROM pipelines WHERE repo_id = ? ORDER BY id DESC LIMIT 50`, repoID)
379 rows, err := s.db.QueryContext(ctx, pipelineCols+` WHERE repo_id = ? ORDER BY id DESC LIMIT 50`, repoID)
338 380 if err != nil {
339 381 return nil, err
340 382 }
@@ -350,36 +392,40 @@ func (s *Store) ListPipelines(ctx context.Context, repoID int64) ([]Pipeline, er
350 392 return out, rows.Err()
351 393 }
352 394
395 const pipelineCols = `SELECT id, repo_id, commit_sha, ref, status, created_at, finished_at, source, actor, commit_title, started_at FROM pipelines`
396
353 397 func scanPipeline(rows *sql.Rows) (Pipeline, error) {
398 return scanPipelineRow(rows)
399 }
400
401 func scanPipelineRow(sc scanner) (Pipeline, error) {
354 402 var p Pipeline
355 var created string
356 if err := rows.Scan(&p.ID, &p.RepoID, &p.CommitSHA, &p.Ref, &p.Status, &created); err != nil {
403 var created, finished, started sql.NullString
404 if err := sc.Scan(&p.ID, &p.RepoID, &p.CommitSHA, &p.Ref, &p.Status, &created, &finished, &p.Source, &p.Actor, &p.CommitTitle, &started); err != nil {
357 405 return Pipeline{}, err
358 406 }
359 p.CreatedAt = parseTime(created)
407 p.CreatedAt = parseTime(created.String)
408 p.FinishedAt = parseTime(finished.String)
409 p.StartedAt = parseTime(started.String)
360 410 return p, nil
361 411 }
362 412
413 type scanner interface {
414 Scan(dest ...any) error
415 }
416
363 417 // PipelineByID leest één pipeline.
364 418 func (s *Store) PipelineByID(ctx context.Context, repoID, id int64) (Pipeline, error) {
365 var p Pipeline
366 var created string
367 err := s.db.QueryRowContext(ctx, `SELECT id, repo_id, commit_sha, ref, status, created_at FROM pipelines WHERE repo_id = ? AND id = ?`, repoID, id).
368 Scan(&p.ID, &p.RepoID, &p.CommitSHA, &p.Ref, &p.Status, &created)
419 p, err := scanPipelineRow(s.db.QueryRowContext(ctx, pipelineCols+` WHERE repo_id = ? AND id = ?`, repoID, id))
369 420 if errors.Is(err, sql.ErrNoRows) {
370 421 return Pipeline{}, errNotFound
371 422 }
372 if err != nil {
373 return Pipeline{}, err
374 }
375 p.CreatedAt = parseTime(created)
376 return p, nil
423 return p, err
377 424 }
378 425
379 426 // JobsByPipeline geeft de jobs in bestandsvolgorde.
380 427 func (s *Store) JobsByPipeline(ctx context.Context, pipelineID int64) ([]Job, error) {
381 rows, err := s.db.QueryContext(ctx, `SELECT j.id, j.pipeline_id, j.repo_id, j.name, j.stage, j.position, j.status, j.image, j.script, j.needs_json, j.needs_set, j.variables_json, j.artifact_paths, j.artifact_expire_seconds, j.allow_failure, j.log, p.commit_sha, p.ref
382 FROM jobs j JOIN pipelines p ON p.id = j.pipeline_id WHERE j.pipeline_id = ? ORDER BY j.position, j.id`, pipelineID)
428 rows, err := s.db.QueryContext(ctx, jobSelect+` WHERE j.pipeline_id = ? ORDER BY j.position, j.id`, pipelineID)
383 429 if err != nil {
384 430 return nil, err
385 431 }
@@ -395,25 +441,35 @@ func (s *Store) JobsByPipeline(ctx context.Context, pipelineID int64) ([]Job, er
395 441 return out, rows.Err()
396 442 }
397 443
444 const jobSelect = `SELECT j.id, j.pipeline_id, j.repo_id, j.name, j.stage, j.position, j.status, j.image, j.script, j.needs_json, j.needs_set, j.variables_json, j.artifact_paths, j.artifact_expire_seconds, j.allow_failure, j.log, p.commit_sha, p.ref, j.before_script, j.after_script, j.when_mode, j.timeout_seconds, j.tags_json, j.cache_paths, j.cache_key, j.resource_group, j.optional_json, j.started_at, j.finished_at
445 FROM jobs j JOIN pipelines p ON p.id = j.pipeline_id`
446
398 447 func scanJob(rows *sql.Rows) (Job, error) {
399 448 var j Job
400 var needs, vars, paths string
401 var needsSet, allow, expire int
402 if err := rows.Scan(&j.ID, &j.PipelineID, &j.RepoID, &j.Name, &j.Stage, &j.Position, &j.Status, &j.Image, &j.Script, &needs, &needsSet, &vars, &paths, &expire, &allow, &j.Log, &j.CommitSHA, &j.Ref); err != nil {
449 var needs, vars, paths, tags, cache, optional string
450 var needsSet, allow, expire, timeout int
451 var started, finished sql.NullString
452 if err := rows.Scan(&j.ID, &j.PipelineID, &j.RepoID, &j.Name, &j.Stage, &j.Position, &j.Status, &j.Image, &j.Script, &needs, &needsSet, &vars, &paths, &expire, &allow, &j.Log, &j.CommitSHA, &j.Ref, &j.Before, &j.After, &j.When, &timeout, &tags, &cache, &j.CacheKey, &j.ResourceGroup, &optional, &started, &finished); err != nil {
403 453 return Job{}, err
404 454 }
405 455 j.NeedsSet = needsSet == 1
406 456 j.AllowFailure = allow == 1
407 457 j.ArtifactExpire = time.Duration(expire) * time.Second
458 j.Timeout = time.Duration(timeout) * time.Second
459 j.StartedAt = parseTime(started.String)
460 j.FinishedAt = parseTime(finished.String)
408 461 _ = json.Unmarshal([]byte(needs), &j.Needs)
409 462 _ = json.Unmarshal([]byte(vars), &j.Variables)
410 463 _ = json.Unmarshal([]byte(paths), &j.ArtifactPaths)
464 _ = json.Unmarshal([]byte(tags), &j.Tags)
465 _ = json.Unmarshal([]byte(cache), &j.CachePaths)
466 _ = json.Unmarshal([]byte(optional), &j.Optional)
411 467 return j, nil
412 468 }
413 469
414 470 // ActivePipelines zijn nog niet af.
415 471 func (s *Store) ActivePipelines(ctx context.Context) ([]Pipeline, error) {
416 rows, err := s.db.QueryContext(ctx, `SELECT id, repo_id, commit_sha, ref, status, created_at FROM pipelines WHERE status IN ('queued', 'running') ORDER BY id`)
472 rows, err := s.db.QueryContext(ctx, pipelineCols+` WHERE status IN ('queued', 'running', 'manual') ORDER BY id`)
417 473 if err != nil {
418 474 return nil, err
419 475 }
@@ -437,14 +493,14 @@ func (s *Store) MarkJobRunning(ctx context.Context, id int64) (bool, error) {
437 493 }
438 494 n, _ := res.RowsAffected()
439 495 if n == 1 {
440 _, _ = s.db.ExecContext(ctx, `UPDATE pipelines SET status = 'running' WHERE id = (SELECT pipeline_id FROM jobs WHERE id = ?)`, id)
496 _, _ = s.db.ExecContext(ctx, `UPDATE pipelines SET status = 'running', started_at = COALESCE(started_at, ?) WHERE id = (SELECT pipeline_id FROM jobs WHERE id = ?)`, now(), id)
441 497 }
442 498 return n == 1, nil
443 499 }
444 500
445 501 // FinishJob schrijft het eindresultaat en het log.
446 func (s *Store) FinishJob(ctx context.Context, id int64, status, logText string) error {
447 _, err := s.db.ExecContext(ctx, `UPDATE jobs SET status = ?, log = ?, finished_at = ? WHERE id = ?`, status, logText, now(), id)
502 func (s *Store) FinishJob(ctx context.Context, id int64, status, extra string) error {
503 _, err := s.db.ExecContext(ctx, `UPDATE jobs SET status = ?, log = log || ?, finished_at = ? WHERE id = ?`, status, extra, now(), id)
448 504 return err
449 505 }
450 506
@@ -461,6 +517,8 @@ func (s *Store) RefreshPipelineStatus(ctx context.Context, pipelineID int64) (st
461 517 return "", err
462 518 }
463 519 status := "success"
520 open := false
521 manual := false
464 522 for _, j := range jobs {
465 523 switch j.Status {
466 524 case "failed":
@@ -468,22 +526,52 @@ func (s *Store) RefreshPipelineStatus(ctx context.Context, pipelineID int64) (st
468 526 status = "failed"
469 527 }
470 528 case "queued", "running":
471 if status != "failed" {
472 status = "running"
473 }
529 open = true
530 case "manual":
531 manual = true
474 532 }
475 533 }
476 if status == "success" || status == "failed" {
477 _, err = s.db.ExecContext(ctx, `UPDATE pipelines SET status = ?, finished_at = ? WHERE id = ?`, status, now(), pipelineID)
478 } else {
534 switch {
535 case status == "failed":
536 _, err = s.db.ExecContext(ctx, `UPDATE pipelines SET status = 'failed', finished_at = ? WHERE id = ?`, now(), pipelineID)
537 case open:
538 status = "running"
479 539 _, err = s.db.ExecContext(ctx, `UPDATE pipelines SET status = 'running' WHERE id = ?`, pipelineID)
540 case manual:
541 status = "manual"
542 _, err = s.db.ExecContext(ctx, `UPDATE pipelines SET status = 'manual', finished_at = NULL WHERE id = ?`, pipelineID)
543 default:
544 _, err = s.db.ExecContext(ctx, `UPDATE pipelines SET status = 'success', finished_at = ? WHERE id = ?`, now(), pipelineID)
480 545 }
481 546 return status, err
482 547 }
483 548
549 // ResourceBusy meldt of een andere job in deze repository die groep bezet.
550 func (s *Store) ResourceBusy(ctx context.Context, repoID int64, group string, except int64) (bool, error) {
551 if group == "" {
552 return false, nil
553 }
554 var n int
555 err := s.db.QueryRowContext(ctx, `SELECT COUNT(*) FROM jobs WHERE repo_id = ? AND resource_group = ? AND status = 'running' AND id != ?`, repoID, group, except).Scan(&n)
556 return n > 0, err
557 }
558
559 // QueueJob zet een handmatige job in de wachtrij.
560 func (s *Store) QueueJob(ctx context.Context, id int64) (bool, error) {
561 res, err := s.db.ExecContext(ctx, `UPDATE jobs SET status = 'queued' WHERE id = ? AND status = 'manual'`, id)
562 if err != nil {
563 return false, err
564 }
565 n, _ := res.RowsAffected()
566 if n == 1 {
567 _, _ = s.db.ExecContext(ctx, `UPDATE pipelines SET status = 'running', finished_at = NULL WHERE id = (SELECT pipeline_id FROM jobs WHERE id = ?)`, id)
568 }
569 return n == 1, nil
570 }
571
484 572 // SkipJob markeert een job die niet meer hoeft.
485 573 func (s *Store) SkipJob(ctx context.Context, id int64, reason string) error {
486 _, err := s.db.ExecContext(ctx, `UPDATE jobs SET status = 'skipped', log = ?, finished_at = ? WHERE id = ? AND status = 'queued'`, reason, now(), id)
574 _, err := s.db.ExecContext(ctx, `UPDATE jobs SET status = 'skipped', log = ?, finished_at = ? WHERE id = ? AND status IN ('queued', 'manual')`, reason, now(), id)
487 575 return err
488 576 }
489 577
@@ -515,8 +603,7 @@ func (s *Store) JobByToken(ctx context.Context, token string) (Job, error) {
515 603 if err != nil {
516 604 return Job{}, err
517 605 }
518 rows, err := s.db.QueryContext(ctx, `SELECT j.id, j.pipeline_id, j.repo_id, j.name, j.stage, j.position, j.status, j.image, j.script, j.needs_json, j.needs_set, j.variables_json, j.artifact_paths, j.artifact_expire_seconds, j.allow_failure, j.log, p.commit_sha, p.ref
519 FROM jobs j JOIN pipelines p ON p.id = j.pipeline_id WHERE j.id = ?`, jobID)
606 rows, err := s.db.QueryContext(ctx, jobSelect+` WHERE j.id = ?`, jobID)
520 607 if err != nil {
521 608 return Job{}, err
522 609 }
@@ -235,23 +235,37 @@ type NewJob struct {
235 Stage string
235 Stage string
236 Position int
236 Position int
237 Image string
237 Image string
238 Before string
238 Script string
239 Script string
240 After string
239 Needs []string
241 Needs []string
242 Optional []string
240 NeedsSet bool
243 NeedsSet bool
241 Variables map[string]string
244 Variables map[string]string
242 ArtifactPaths []string
245 ArtifactPaths []string
243 ArtifactExpire time.Duration
246 ArtifactExpire time.Duration
244 AllowFailure bool
247 AllowFailure bool
248 When string
249 Timeout time.Duration
250 Tags []string
251 CachePaths []string
252 CacheKey string
253 ResourceGroup string
245}
254}
246
255
247// Pipeline is een draai van blue-ci-cd.yml.
256// Pipeline is een draai van blue-ci-cd.yml.
248type Pipeline struct {
257type Pipeline struct {
249 ID int64
258 ID int64
250 RepoID int64
259 RepoID int64
251 CommitSHA string
260 CommitSHA string
252 Ref string
261 Ref string
253 Status string
262 Status string
254 CreatedAt time.Time
263 Source string
264 Actor string
265 CommitTitle string
266 CreatedAt time.Time
267 StartedAt time.Time
268 FinishedAt time.Time
255}
269}
256
270
257// Job is één container in een pipeline.
271// Job is één container in een pipeline.
@@ -264,16 +278,27 @@ type Job struct {
264 Position int
278 Position int
265 Status string
279 Status string
266 Image string
280 Image string
281 Before string
267 Script string
282 Script string
283 After string
268 Needs []string
284 Needs []string
285 Optional []string
269 NeedsSet bool
286 NeedsSet bool
270 Variables map[string]string
287 Variables map[string]string
271 ArtifactPaths []string
288 ArtifactPaths []string
272 ArtifactExpire time.Duration
289 ArtifactExpire time.Duration
273 AllowFailure bool
290 AllowFailure bool
291 When string
292 Timeout time.Duration
293 Tags []string
294 CachePaths []string
295 CacheKey string
296 ResourceGroup string
274 Log string
297 Log string
275 CommitSHA string
298 CommitSHA string
276 Ref string
299 Ref string
300 StartedAt time.Time
301 FinishedAt time.Time
277}
302}
278
303
279// RecordConfigFailure bewaart een pipeline die niet kon starten.
304// RecordConfigFailure bewaart een pipeline die niet kon starten.
@@ -298,13 +323,13 @@ func (s *Store) RecordConfigFailure(ctx context.Context, repoID int64, sha, ref,
298}
323}
299
324
300// CreatePipeline maakt een pipeline met jobs in de wachtrij.
325// CreatePipeline maakt een pipeline met jobs in de wachtrij.
301func (s *Store) CreatePipeline(ctx context.Context, repoID int64, sha, ref string, jobs []NewJob) (int64, error) {
326func (s *Store) CreatePipeline(ctx context.Context, repoID int64, sha, ref, source, actor, title string, jobs []NewJob) (int64, error) {
302 tx, err := s.db.BeginTx(ctx, nil)
327 tx, err := s.db.BeginTx(ctx, nil)
303 if err != nil {
328 if err != nil {
304 return 0, err
329 return 0, err
305 }
330 }
306 defer func() { _ = tx.Rollback() }()
331 defer func() { _ = tx.Rollback() }()
307 res, err := tx.ExecContext(ctx, `INSERT INTO pipelines (repo_id, commit_sha, ref, status, created_at) VALUES (?, ?, ?, 'queued', ?)`, repoID, sha, ref, now())
332 res, err := tx.ExecContext(ctx, `INSERT INTO pipelines (repo_id, commit_sha, ref, status, source, actor, commit_title, created_at) VALUES (?, ?, ?, 'queued', ?, ?, ?, ?)`, repoID, sha, ref, source, actor, title, now())
308 if err != nil {
333 if err != nil {
309 return 0, err
334 return 0, err
310 }
335 }
@@ -319,9 +344,26 @@ func (s *Store) CreatePipeline(ctx context.Context, repoID int64, sha, ref strin
319 if j.ArtifactPaths == nil {
344 if j.ArtifactPaths == nil {
320 j.ArtifactPaths = []string{}
345 j.ArtifactPaths = []string{}
321 }
346 }
322 _, err := tx.ExecContext(ctx, `INSERT INTO jobs (pipeline_id, repo_id, name, stage, position, status, image, script, needs_json, needs_set, variables_json, artifact_paths, artifact_expire_seconds, allow_failure, log)
347 if j.Optional == nil {
323 VALUES (?, ?, ?, ?, ?, 'queued', ?, ?, ?, ?, ?, ?, ?, ?, '')`,
348 j.Optional = []string{}
324 id, repoID, j.Name, j.Stage, j.Position, j.Image, j.Script, mustJSON(j.Needs), boolInt(j.NeedsSet), mustJSON(j.Variables), mustJSON(j.ArtifactPaths), int(j.ArtifactExpire/time.Second), boolInt(j.AllowFailure))
349 }
350 if j.Tags == nil {
351 j.Tags = []string{}
352 }
353 if j.CachePaths == nil {
354 j.CachePaths = []string{}
355 }
356 if j.When == "" {
357 j.When = "on_success"
358 }
359 status := "queued"
360 if j.When == "manual" {
361 status = "manual"
362 }
363 _, err := tx.ExecContext(ctx, `INSERT INTO jobs (pipeline_id, repo_id, name, stage, position, status, image, script, needs_json, needs_set, variables_json, artifact_paths, artifact_expire_seconds, allow_failure, log, before_script, after_script, when_mode, timeout_seconds, tags_json, cache_paths, cache_key, resource_group, optional_json)
364 VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, '', ?, ?, ?, ?, ?, ?, ?, ?, ?)`,
365 id, repoID, j.Name, j.Stage, j.Position, status, j.Image, j.Script, mustJSON(j.Needs), boolInt(j.NeedsSet), mustJSON(j.Variables), mustJSON(j.ArtifactPaths), int(j.ArtifactExpire/time.Second), boolInt(j.AllowFailure),
366 j.Before, j.After, j.When, int(j.Timeout/time.Second), mustJSON(j.Tags), mustJSON(j.CachePaths), j.CacheKey, j.ResourceGroup, mustJSON(j.Optional))
325 if err != nil {
367 if err != nil {
326 return 0, err
368 return 0, err
327 }
369 }
@@ -334,7 +376,7 @@ func (s *Store) CreatePipeline(ctx context.Context, repoID int64, sha, ref strin
334
376
335// ListPipelines geeft de nieuwste pipelines.
377// ListPipelines geeft de nieuwste pipelines.
336func (s *Store) ListPipelines(ctx context.Context, repoID int64) ([]Pipeline, error) {
378func (s *Store) ListPipelines(ctx context.Context, repoID int64) ([]Pipeline, error) {
337 rows, err := s.db.QueryContext(ctx, `SELECT id, repo_id, commit_sha, ref, status, created_at FROM pipelines WHERE repo_id = ? ORDER BY id DESC LIMIT 50`, repoID)
379 rows, err := s.db.QueryContext(ctx, pipelineCols+` WHERE repo_id = ? ORDER BY id DESC LIMIT 50`, repoID)
338 if err != nil {
380 if err != nil {
339 return nil, err
381 return nil, err
340 }
382 }
@@ -350,36 +392,40 @@ func (s *Store) ListPipelines(ctx context.Context, repoID int64) ([]Pipeline, er
350 return out, rows.Err()
392 return out, rows.Err()
351}
393}
352
394
395const pipelineCols = `SELECT id, repo_id, commit_sha, ref, status, created_at, finished_at, source, actor, commit_title, started_at FROM pipelines`
396
353func scanPipeline(rows *sql.Rows) (Pipeline, error) {
397func scanPipeline(rows *sql.Rows) (Pipeline, error) {
398 return scanPipelineRow(rows)
399}
400
401func scanPipelineRow(sc scanner) (Pipeline, error) {
354 var p Pipeline
402 var p Pipeline
355 var created string
403 var created, finished, started sql.NullString
356 if err := rows.Scan(&p.ID, &p.RepoID, &p.CommitSHA, &p.Ref, &p.Status, &created); err != nil {
404 if err := sc.Scan(&p.ID, &p.RepoID, &p.CommitSHA, &p.Ref, &p.Status, &created, &finished, &p.Source, &p.Actor, &p.CommitTitle, &started); err != nil {
357 return Pipeline{}, err
405 return Pipeline{}, err
358 }
406 }
359 p.CreatedAt = parseTime(created)
407 p.CreatedAt = parseTime(created.String)
408 p.FinishedAt = parseTime(finished.String)
409 p.StartedAt = parseTime(started.String)
360 return p, nil
410 return p, nil
361}
411}
362
412
413type scanner interface {
414 Scan(dest ...any) error
415}
416
363// PipelineByID leest één pipeline.
417// PipelineByID leest één pipeline.
364func (s *Store) PipelineByID(ctx context.Context, repoID, id int64) (Pipeline, error) {
418func (s *Store) PipelineByID(ctx context.Context, repoID, id int64) (Pipeline, error) {
365 var p Pipeline
419 p, err := scanPipelineRow(s.db.QueryRowContext(ctx, pipelineCols+` WHERE repo_id = ? AND id = ?`, repoID, id))
366 var created string
367 err := s.db.QueryRowContext(ctx, `SELECT id, repo_id, commit_sha, ref, status, created_at FROM pipelines WHERE repo_id = ? AND id = ?`, repoID, id).
368 Scan(&p.ID, &p.RepoID, &p.CommitSHA, &p.Ref, &p.Status, &created)
369 if errors.Is(err, sql.ErrNoRows) {
420 if errors.Is(err, sql.ErrNoRows) {
370 return Pipeline{}, errNotFound
421 return Pipeline{}, errNotFound
371 }
422 }
372 if err != nil {
423 return p, err
373 return Pipeline{}, err
374 }
375 p.CreatedAt = parseTime(created)
376 return p, nil
377}
424}
378
425
379// JobsByPipeline geeft de jobs in bestandsvolgorde.
426// JobsByPipeline geeft de jobs in bestandsvolgorde.
380func (s *Store) JobsByPipeline(ctx context.Context, pipelineID int64) ([]Job, error) {
427func (s *Store) JobsByPipeline(ctx context.Context, pipelineID int64) ([]Job, error) {
381 rows, err := s.db.QueryContext(ctx, `SELECT j.id, j.pipeline_id, j.repo_id, j.name, j.stage, j.position, j.status, j.image, j.script, j.needs_json, j.needs_set, j.variables_json, j.artifact_paths, j.artifact_expire_seconds, j.allow_failure, j.log, p.commit_sha, p.ref
428 rows, err := s.db.QueryContext(ctx, jobSelect+` WHERE j.pipeline_id = ? ORDER BY j.position, j.id`, pipelineID)
382 FROM jobs j JOIN pipelines p ON p.id = j.pipeline_id WHERE j.pipeline_id = ? ORDER BY j.position, j.id`, pipelineID)
383 if err != nil {
429 if err != nil {
384 return nil, err
430 return nil, err
385 }
431 }
@@ -395,25 +441,35 @@ func (s *Store) JobsByPipeline(ctx context.Context, pipelineID int64) ([]Job, er
395 return out, rows.Err()
441 return out, rows.Err()
396}
442}
397
443
444const jobSelect = `SELECT j.id, j.pipeline_id, j.repo_id, j.name, j.stage, j.position, j.status, j.image, j.script, j.needs_json, j.needs_set, j.variables_json, j.artifact_paths, j.artifact_expire_seconds, j.allow_failure, j.log, p.commit_sha, p.ref, j.before_script, j.after_script, j.when_mode, j.timeout_seconds, j.tags_json, j.cache_paths, j.cache_key, j.resource_group, j.optional_json, j.started_at, j.finished_at
445FROM jobs j JOIN pipelines p ON p.id = j.pipeline_id`
446
398func scanJob(rows *sql.Rows) (Job, error) {
447func scanJob(rows *sql.Rows) (Job, error) {
399 var j Job
448 var j Job
400 var needs, vars, paths string
449 var needs, vars, paths, tags, cache, optional string
401 var needsSet, allow, expire int
450 var needsSet, allow, expire, timeout int
402 if err := rows.Scan(&j.ID, &j.PipelineID, &j.RepoID, &j.Name, &j.Stage, &j.Position, &j.Status, &j.Image, &j.Script, &needs, &needsSet, &vars, &paths, &expire, &allow, &j.Log, &j.CommitSHA, &j.Ref); err != nil {
451 var started, finished sql.NullString
452 if err := rows.Scan(&j.ID, &j.PipelineID, &j.RepoID, &j.Name, &j.Stage, &j.Position, &j.Status, &j.Image, &j.Script, &needs, &needsSet, &vars, &paths, &expire, &allow, &j.Log, &j.CommitSHA, &j.Ref, &j.Before, &j.After, &j.When, &timeout, &tags, &cache, &j.CacheKey, &j.ResourceGroup, &optional, &started, &finished); err != nil {
403 return Job{}, err
453 return Job{}, err
404 }
454 }
405 j.NeedsSet = needsSet == 1
455 j.NeedsSet = needsSet == 1
406 j.AllowFailure = allow == 1
456 j.AllowFailure = allow == 1
407 j.ArtifactExpire = time.Duration(expire) * time.Second
457 j.ArtifactExpire = time.Duration(expire) * time.Second
458 j.Timeout = time.Duration(timeout) * time.Second
459 j.StartedAt = parseTime(started.String)
460 j.FinishedAt = parseTime(finished.String)
408 _ = json.Unmarshal([]byte(needs), &j.Needs)
461 _ = json.Unmarshal([]byte(needs), &j.Needs)
409 _ = json.Unmarshal([]byte(vars), &j.Variables)
462 _ = json.Unmarshal([]byte(vars), &j.Variables)
410 _ = json.Unmarshal([]byte(paths), &j.ArtifactPaths)
463 _ = json.Unmarshal([]byte(paths), &j.ArtifactPaths)
464 _ = json.Unmarshal([]byte(tags), &j.Tags)
465 _ = json.Unmarshal([]byte(cache), &j.CachePaths)
466 _ = json.Unmarshal([]byte(optional), &j.Optional)
411 return j, nil
467 return j, nil
412}
468}
413
469
414// ActivePipelines zijn nog niet af.
470// ActivePipelines zijn nog niet af.
415func (s *Store) ActivePipelines(ctx context.Context) ([]Pipeline, error) {
471func (s *Store) ActivePipelines(ctx context.Context) ([]Pipeline, error) {
416 rows, err := s.db.QueryContext(ctx, `SELECT id, repo_id, commit_sha, ref, status, created_at FROM pipelines WHERE status IN ('queued', 'running') ORDER BY id`)
472 rows, err := s.db.QueryContext(ctx, pipelineCols+` WHERE status IN ('queued', 'running', 'manual') ORDER BY id`)
417 if err != nil {
473 if err != nil {
418 return nil, err
474 return nil, err
419 }
475 }
@@ -437,14 +493,14 @@ func (s *Store) MarkJobRunning(ctx context.Context, id int64) (bool, error) {
437 }
493 }
438 n, _ := res.RowsAffected()
494 n, _ := res.RowsAffected()
439 if n == 1 {
495 if n == 1 {
440 _, _ = s.db.ExecContext(ctx, `UPDATE pipelines SET status = 'running' WHERE id = (SELECT pipeline_id FROM jobs WHERE id = ?)`, id)
496 _, _ = s.db.ExecContext(ctx, `UPDATE pipelines SET status = 'running', started_at = COALESCE(started_at, ?) WHERE id = (SELECT pipeline_id FROM jobs WHERE id = ?)`, now(), id)
441 }
497 }
442 return n == 1, nil
498 return n == 1, nil
443}
499}
444
500
445// FinishJob schrijft het eindresultaat en het log.
501// FinishJob schrijft het eindresultaat en het log.
446func (s *Store) FinishJob(ctx context.Context, id int64, status, logText string) error {
502func (s *Store) FinishJob(ctx context.Context, id int64, status, extra string) error {
447 _, err := s.db.ExecContext(ctx, `UPDATE jobs SET status = ?, log = ?, finished_at = ? WHERE id = ?`, status, logText, now(), id)
503 _, err := s.db.ExecContext(ctx, `UPDATE jobs SET status = ?, log = log || ?, finished_at = ? WHERE id = ?`, status, extra, now(), id)
448 return err
504 return err
449}
505}
450
506
@@ -461,6 +517,8 @@ func (s *Store) RefreshPipelineStatus(ctx context.Context, pipelineID int64) (st
461 return "", err
517 return "", err
462 }
518 }
463 status := "success"
519 status := "success"
520 open := false
521 manual := false
464 for _, j := range jobs {
522 for _, j := range jobs {
465 switch j.Status {
523 switch j.Status {
466 case "failed":
524 case "failed":
@@ -468,22 +526,52 @@ func (s *Store) RefreshPipelineStatus(ctx context.Context, pipelineID int64) (st
468 status = "failed"
526 status = "failed"
469 }
527 }
470 case "queued", "running":
528 case "queued", "running":
471 if status != "failed" {
529 open = true
472 status = "running"
530 case "manual":
473 }
531 manual = true
474 }
532 }
475 }
533 }
476 if status == "success" || status == "failed" {
534 switch {
477 _, err = s.db.ExecContext(ctx, `UPDATE pipelines SET status = ?, finished_at = ? WHERE id = ?`, status, now(), pipelineID)
535 case status == "failed":
478 } else {
536 _, err = s.db.ExecContext(ctx, `UPDATE pipelines SET status = 'failed', finished_at = ? WHERE id = ?`, now(), pipelineID)
537 case open:
538 status = "running"
479 _, err = s.db.ExecContext(ctx, `UPDATE pipelines SET status = 'running' WHERE id = ?`, pipelineID)
539 _, err = s.db.ExecContext(ctx, `UPDATE pipelines SET status = 'running' WHERE id = ?`, pipelineID)
540 case manual:
541 status = "manual"
542 _, err = s.db.ExecContext(ctx, `UPDATE pipelines SET status = 'manual', finished_at = NULL WHERE id = ?`, pipelineID)
543 default:
544 _, err = s.db.ExecContext(ctx, `UPDATE pipelines SET status = 'success', finished_at = ? WHERE id = ?`, now(), pipelineID)
480 }
545 }
481 return status, err
546 return status, err
482}
547}
483
548
549// ResourceBusy meldt of een andere job in deze repository die groep bezet.
550func (s *Store) ResourceBusy(ctx context.Context, repoID int64, group string, except int64) (bool, error) {
551 if group == "" {
552 return false, nil
553 }
554 var n int
555 err := s.db.QueryRowContext(ctx, `SELECT COUNT(*) FROM jobs WHERE repo_id = ? AND resource_group = ? AND status = 'running' AND id != ?`, repoID, group, except).Scan(&n)
556 return n > 0, err
557}
558
559// QueueJob zet een handmatige job in de wachtrij.
560func (s *Store) QueueJob(ctx context.Context, id int64) (bool, error) {
561 res, err := s.db.ExecContext(ctx, `UPDATE jobs SET status = 'queued' WHERE id = ? AND status = 'manual'`, id)
562 if err != nil {
563 return false, err
564 }
565 n, _ := res.RowsAffected()
566 if n == 1 {
567 _, _ = s.db.ExecContext(ctx, `UPDATE pipelines SET status = 'running', finished_at = NULL WHERE id = (SELECT pipeline_id FROM jobs WHERE id = ?)`, id)
568 }
569 return n == 1, nil
570}
571
484// SkipJob markeert een job die niet meer hoeft.
572// SkipJob markeert een job die niet meer hoeft.
485func (s *Store) SkipJob(ctx context.Context, id int64, reason string) error {
573func (s *Store) SkipJob(ctx context.Context, id int64, reason string) error {
486 _, err := s.db.ExecContext(ctx, `UPDATE jobs SET status = 'skipped', log = ?, finished_at = ? WHERE id = ? AND status = 'queued'`, reason, now(), id)
574 _, err := s.db.ExecContext(ctx, `UPDATE jobs SET status = 'skipped', log = ?, finished_at = ? WHERE id = ? AND status IN ('queued', 'manual')`, reason, now(), id)
487 return err
575 return err
488}
576}
489
577
@@ -515,8 +603,7 @@ func (s *Store) JobByToken(ctx context.Context, token string) (Job, error) {
515 if err != nil {
603 if err != nil {
516 return Job{}, err
604 return Job{}, err
517 }
605 }
518 rows, err := s.db.QueryContext(ctx, `SELECT j.id, j.pipeline_id, j.repo_id, j.name, j.stage, j.position, j.status, j.image, j.script, j.needs_json, j.needs_set, j.variables_json, j.artifact_paths, j.artifact_expire_seconds, j.allow_failure, j.log, p.commit_sha, p.ref
606 rows, err := s.db.QueryContext(ctx, jobSelect+` WHERE j.id = ?`, jobID)
519 FROM jobs j JOIN pipelines p ON p.id = j.pipeline_id WHERE j.id = ?`, jobID)
520 if err != nil {
607 if err != nil {
521 return Job{}, err
608 return Job{}, err
522 }
609 }
internal/store/store.go +27 −1
@@ -68,6 +68,19 @@ func (s *Store) migrate() error {
68 68 `ALTER TABLE notes ADD COLUMN line_end INTEGER NOT NULL DEFAULT 0`,
69 69 `ALTER TABLE notes ADD COLUMN suggestion TEXT NOT NULL DEFAULT ''`,
70 70 `ALTER TABLE notes ADD COLUMN base_text TEXT NOT NULL DEFAULT ''`,
71 `ALTER TABLE pipelines ADD COLUMN source TEXT NOT NULL DEFAULT 'push'`,
72 `ALTER TABLE pipelines ADD COLUMN actor TEXT NOT NULL DEFAULT ''`,
73 `ALTER TABLE pipelines ADD COLUMN commit_title TEXT NOT NULL DEFAULT ''`,
74 `ALTER TABLE pipelines ADD COLUMN started_at TEXT`,
75 `ALTER TABLE jobs ADD COLUMN before_script TEXT NOT NULL DEFAULT ''`,
76 `ALTER TABLE jobs ADD COLUMN after_script TEXT NOT NULL DEFAULT ''`,
77 `ALTER TABLE jobs ADD COLUMN when_mode TEXT NOT NULL DEFAULT 'on_success'`,
78 `ALTER TABLE jobs ADD COLUMN timeout_seconds INTEGER NOT NULL DEFAULT 0`,
79 `ALTER TABLE jobs ADD COLUMN tags_json TEXT NOT NULL DEFAULT '[]'`,
80 `ALTER TABLE jobs ADD COLUMN cache_paths TEXT NOT NULL DEFAULT '[]'`,
81 `ALTER TABLE jobs ADD COLUMN cache_key TEXT NOT NULL DEFAULT ''`,
82 `ALTER TABLE jobs ADD COLUMN resource_group TEXT NOT NULL DEFAULT ''`,
83 `ALTER TABLE jobs ADD COLUMN optional_json TEXT NOT NULL DEFAULT '[]'`,
71 84 }
72 85 for _, q := range alters {
73 86 if _, err := s.db.Exec(q); err != nil && !strings.Contains(strings.ToLower(err.Error()), "duplicate column") {
@@ -219,7 +232,11 @@ CREATE TABLE IF NOT EXISTS pipelines (
219 232 ref TEXT NOT NULL,
220 233 status TEXT NOT NULL,
221 234 created_at TEXT NOT NULL,
222 finished_at TEXT
235 finished_at TEXT,
236 source TEXT NOT NULL DEFAULT 'push',
237 actor TEXT NOT NULL DEFAULT '',
238 commit_title TEXT NOT NULL DEFAULT '',
239 started_at TEXT
223 240 );
224 241 CREATE TABLE IF NOT EXISTS jobs (
225 242 id INTEGER PRIMARY KEY,
@@ -240,6 +257,15 @@ CREATE TABLE IF NOT EXISTS jobs (
240 257 log TEXT NOT NULL DEFAULT '',
241 258 started_at TEXT,
242 259 finished_at TEXT,
260 before_script TEXT NOT NULL DEFAULT '',
261 after_script TEXT NOT NULL DEFAULT '',
262 when_mode TEXT NOT NULL DEFAULT 'on_success',
263 timeout_seconds INTEGER NOT NULL DEFAULT 0,
264 tags_json TEXT NOT NULL DEFAULT '[]',
265 cache_paths TEXT NOT NULL DEFAULT '[]',
266 cache_key TEXT NOT NULL DEFAULT '',
267 resource_group TEXT NOT NULL DEFAULT '',
268 optional_json TEXT NOT NULL DEFAULT '[]',
243 269 UNIQUE(pipeline_id, name)
244 270 );
245 271 CREATE TABLE IF NOT EXISTS artifacts (
@@ -68,6 +68,19 @@ func (s *Store) migrate() error {
68 `ALTER TABLE notes ADD COLUMN line_end INTEGER NOT NULL DEFAULT 0`,
68 `ALTER TABLE notes ADD COLUMN line_end INTEGER NOT NULL DEFAULT 0`,
69 `ALTER TABLE notes ADD COLUMN suggestion TEXT NOT NULL DEFAULT ''`,
69 `ALTER TABLE notes ADD COLUMN suggestion TEXT NOT NULL DEFAULT ''`,
70 `ALTER TABLE notes ADD COLUMN base_text TEXT NOT NULL DEFAULT ''`,
70 `ALTER TABLE notes ADD COLUMN base_text TEXT NOT NULL DEFAULT ''`,
71 `ALTER TABLE pipelines ADD COLUMN source TEXT NOT NULL DEFAULT 'push'`,
72 `ALTER TABLE pipelines ADD COLUMN actor TEXT NOT NULL DEFAULT ''`,
73 `ALTER TABLE pipelines ADD COLUMN commit_title TEXT NOT NULL DEFAULT ''`,
74 `ALTER TABLE pipelines ADD COLUMN started_at TEXT`,
75 `ALTER TABLE jobs ADD COLUMN before_script TEXT NOT NULL DEFAULT ''`,
76 `ALTER TABLE jobs ADD COLUMN after_script TEXT NOT NULL DEFAULT ''`,
77 `ALTER TABLE jobs ADD COLUMN when_mode TEXT NOT NULL DEFAULT 'on_success'`,
78 `ALTER TABLE jobs ADD COLUMN timeout_seconds INTEGER NOT NULL DEFAULT 0`,
79 `ALTER TABLE jobs ADD COLUMN tags_json TEXT NOT NULL DEFAULT '[]'`,
80 `ALTER TABLE jobs ADD COLUMN cache_paths TEXT NOT NULL DEFAULT '[]'`,
81 `ALTER TABLE jobs ADD COLUMN cache_key TEXT NOT NULL DEFAULT ''`,
82 `ALTER TABLE jobs ADD COLUMN resource_group TEXT NOT NULL DEFAULT ''`,
83 `ALTER TABLE jobs ADD COLUMN optional_json TEXT NOT NULL DEFAULT '[]'`,
71 }
84 }
72 for _, q := range alters {
85 for _, q := range alters {
73 if _, err := s.db.Exec(q); err != nil && !strings.Contains(strings.ToLower(err.Error()), "duplicate column") {
86 if _, err := s.db.Exec(q); err != nil && !strings.Contains(strings.ToLower(err.Error()), "duplicate column") {
@@ -219,7 +232,11 @@ CREATE TABLE IF NOT EXISTS pipelines (
219 ref TEXT NOT NULL,
232 ref TEXT NOT NULL,
220 status TEXT NOT NULL,
233 status TEXT NOT NULL,
221 created_at TEXT NOT NULL,
234 created_at TEXT NOT NULL,
222 finished_at TEXT
235 finished_at TEXT,
236 source TEXT NOT NULL DEFAULT 'push',
237 actor TEXT NOT NULL DEFAULT '',
238 commit_title TEXT NOT NULL DEFAULT '',
239 started_at TEXT
223);
240);
224CREATE TABLE IF NOT EXISTS jobs (
241CREATE TABLE IF NOT EXISTS jobs (
225 id INTEGER PRIMARY KEY,
242 id INTEGER PRIMARY KEY,
@@ -240,6 +257,15 @@ CREATE TABLE IF NOT EXISTS jobs (
240 log TEXT NOT NULL DEFAULT '',
257 log TEXT NOT NULL DEFAULT '',
241 started_at TEXT,
258 started_at TEXT,
242 finished_at TEXT,
259 finished_at TEXT,
260 before_script TEXT NOT NULL DEFAULT '',
261 after_script TEXT NOT NULL DEFAULT '',
262 when_mode TEXT NOT NULL DEFAULT 'on_success',
263 timeout_seconds INTEGER NOT NULL DEFAULT 0,
264 tags_json TEXT NOT NULL DEFAULT '[]',
265 cache_paths TEXT NOT NULL DEFAULT '[]',
266 cache_key TEXT NOT NULL DEFAULT '',
267 resource_group TEXT NOT NULL DEFAULT '',
268 optional_json TEXT NOT NULL DEFAULT '[]',
243 UNIQUE(pipeline_id, name)
269 UNIQUE(pipeline_id, name)
244);
270);
245CREATE TABLE IF NOT EXISTS artifacts (
271CREATE TABLE IF NOT EXISTS artifacts (