Merge branch 'cursor/ci-spec-65da' into 'master'
CI-specificatie dichter bij GitLab See merge request Timfa/bluegit!9
@@ -1,6 +1,6 @@
1
1
# blue-ci-cd.yml
2
2
3
Het bestand ligt in de root van de commit die de pipeline start. De vorm lijkt op GitLab CI: stages, jobs, needs en rules. Elke job draait in het image dat erbij staat.
3
Het bestand ligt in de root van de commit die de pipeline start. De vorm volgt GitLab CI zo ver dat een bestaand `.gitlab-ci.yml` meestal hernoemd kan worden naar `blue-ci-cd.yml`. Jobs mogen los op het hoogste niveau staan. Een ouder bestand met een `jobs:`-map blijft geldig.
4
4
5
5
```yaml
6
6
stages: [test, build]
@@ -32,28 +32,45 @@ jobs:
32
32
| Veld | Betekenis |
33
33
| --- | --- |
34
34
| `stages` | Volgorde. Weggelaten: de stages in de volgorde waarin de jobs ze noemen. Een job zonder stage valt in `test`. |
35
| `image` | Standaardimage. Een job zonder eigen image gebruikt die. |
36
| `variables` | Omgevingsvariabelen voor elke job. Een job kan ze overschrijven. |
37
| `jobs.<naam>.script` | Niet-lege lijst. Ze draaien na elkaar in `sh`, met `set -e`. |
38
| `needs` | Jobs die eerst geslaagd of overgeslagen moeten zijn. `needs: []` start meteen. Weggelaten: wacht op alle eerdere stages. |
39
| `rules` | De job draait als minstens één `if` klopt. Zonder rules draait ze altijd. |
40
| `artifacts.paths` | Relatieve paden, geen `..`. Na succes bewaard, en in een volgende job gezet die deze job nodig heeft. |
41
| `artifacts.expire_in` | `30m`, `12h`, `14d`, `2w` of `never`. |
42
| `allow_failure` | Een mislukking houdt de rest niet tegen. |
35
| `image` | Standaardimage, als tekst of als `name:`. `default.image` wint als die gezet is. |
36
| `default` | `image`, `before_script`, `after_script`, `tags`, `timeout`, `cache` voor jobs die het zelf niet zetten. |
37
| `include` | Alleen `local`. Een job met dezelfde naam wordt samengevoegd; het buitenste bestand wint per veld. |
38
| `variables` | Tekst, of `{value: ...}`. Een job overschrijft dezelfde naam. |
39
| `before_script`, `script`, `after_script` | `after_script` draait ook als `script` mislukt. |
40
| `needs` | Namen, of `{job, optional}`. `needs: []` start meteen. Weggelaten: wacht op eerdere stages. |
41
| `rules` | De eerste rule die klopt wint. `if`, `changes`, `exists` en `when`. Zonder rules gelden `only` en `except`. |
42
| `only`, `except` | Taknaam, `branches`, `tags`, of `/regex/`. |
43
| `when` | `on_success` (standaard), `on_failure`, `always`, `manual`. In een rule ook `never`. |
44
| `allow_failure` | Een mislukking houdt de rest niet tegen. De job zelf wordt een waarschuwing. |
45
| `artifacts` | `paths` en `expire_in` (`30s`, `30m`, `12h`, `14d`, `2w`, `never`). |
46
| `cache` | `paths` en een tekstuele `key`. Lokaal per repository, niet gedeeld met een andere machine. |
47
| `resource_group` | Jobs met dezelfde naam in één repository draaien niet tegelijk. |
48
| `timeout` | Overschrijft `ci.timeout_seconds` voor die job. |
49
| `tags` | Bewaard bij de job. De ingebouwde runner draait elke job; er is geen tweede runner om op tags te kiezen. |
50
| `extends` | Eén naam of een lijst. Een job waarvan de naam met `.` begint is een sjabloon en draait niet. |
43
51
44
Een `if` is één van:
52
Een `if` is één vergelijking, zonder `&&` of `||`:
45
53
46
54
```text
47
55
$NAAM == "waarde"
56
$NAAM == $ANDERE
57
$NAAM == null
48
58
$NAAM != "waarde"
49
59
$NAAM =~ /reguliere-expressie/
60
$NAAM !~ /reguliere-expressie/
50
61
```
51
62
52
Beschikbaar in rules en in de container: `CI_COMMIT_SHA`, `CI_COMMIT_BRANCH`, `CI_COMMIT_REF`, `CI_PROJECT_PATH`, `CI_PIPELINE_ID`, `CI_JOB_ID`, `CI_JOB_NAME`, `CI_JOB_TOKEN`, `CI_API_URL`.
63
`changes` kijkt naar de diff van de push. Is die diff niet te bepalen, dan geldt de voorwaarde. `exists` eist dat elk pad in de commit staat.
64
65
Voorgedefinieerd, in rules en in de container: `CI`, `CI_COMMIT_SHA`, `CI_COMMIT_SHORT_SHA`, `CI_COMMIT_BRANCH`, `CI_COMMIT_TAG`, `CI_COMMIT_REF`, `CI_COMMIT_REF_NAME`, `CI_COMMIT_REF_SLUG`, `CI_COMMIT_TITLE`, `CI_DEFAULT_BRANCH`, `CI_PIPELINE_ID`, `CI_PIPELINE_SOURCE`, `CI_PIPELINE_URL`, `CI_PROJECT_PATH`, `CI_PROJECT_NAME`, `CI_PROJECT_NAMESPACE`, `CI_PROJECT_URL`, `CI_JOB_ID`, `CI_JOB_NAME`, `CI_JOB_STAGE`, `CI_JOB_URL`, `CI_JOB_TOKEN`, `CI_API_URL`, `CI_CONFIG_PATH`, `CI_SERVER_NAME`. Op een tag is `CI_COMMIT_BRANCH` leeg. `CI_PIPELINE_SOURCE` is `push` of `web`.
66
67
## Bewust niet
68
69
`services`, `pages`, `workflow`, `spec`, een include van een ander project of van een URL, `parallel`/`matrix`, trigger- en child-pipelines, `environment`, `retry`, `coverage`, `id_tokens`, `inherit` en `hooks`. Die sleutels geven een waarschuwing in de editor en houden de rest van het bestand niet tegen. Er is één runner, in het BlueGit-proces, die Docker op dezelfde host aanroept.
53
70
54
71
## Geheimen
55
72
56
Repositorygeheimen komen als omgevingsvariabele mee. Bij een groepsrepository eerst de groep, daarna de repository; dezelfde naam laat de repository winnen. Een geheim dat als beschermd is gemarkeerd, komt alleen mee als de branch op een beschermd patroon past.
73
Repositorygeheimen komen als omgevingsvariabele mee, zoals CI/CD-variabelen. Bij een groepsrepository eerst de groep, daarna de repository; dezelfde naam laat de repository winnen. Een geheim dat als beschermd is gemarkeerd, komt alleen mee als de branch of tag op een beschermd patroon past. Waarden van acht tekens of meer worden in het joblog vervangen door `[MASKED]`. `CI_JOB_TOKEN` ook.
57
74
58
75
## Release vanuit een job
59
76
@@ -68,6 +85,6 @@ Het commit moet in de repository bestaan. Het asset moet een artifact zijn van e
68
85
69
86
## Wanneer een job start
70
87
71
`Gate` in `internal/ci` geeft `ready`, `wait` of `skip`. `skip` volgt op een mislukte voorganger zonder `allow_failure`. De runner verwerkt pipelines elke twee seconden, met `ci.concurrency` tegelijk (standaard 1, passend bij de mini-PC).
88
`Gate` in `internal/ci` geeft `ready`, `wait`, `skip` of `fail`. `skip` volgt op een harde mislukking bij `when: on_success`, of op succes bij `when: on_failure`. `when: always` wacht tot de voorgangers klaar zijn en draait dan toch. Een job met `allow_failure` telt niet als harde mislukking. De runner verwerkt pipelines elke twee seconden, met `ci.concurrency` tegelijk (standaard 1).
72
89
73
90
Ontbreekt Docker, dan faalt de job met een regel in het log. De time-out is `ci.timeout_seconds` (standaard 1800).
@@ -1,6 +1,6 @@1
# blue-ci-cd.yml1
# blue-ci-cd.yml2
2
3
Het bestand ligt in de root van de commit die de pipeline start. De vorm lijkt op GitLab CI: stages, jobs, needs en rules. Elke job draait in het image dat erbij staat.3
Het bestand ligt in de root van de commit die de pipeline start. De vorm volgt GitLab CI zo ver dat een bestaand `.gitlab-ci.yml` meestal hernoemd kan worden naar `blue-ci-cd.yml`. Jobs mogen los op het hoogste niveau staan. Een ouder bestand met een `jobs:`-map blijft geldig.4
4
5
```yaml5
```yaml6
stages: [test, build]6
stages: [test, build]@@ -32,28 +32,45 @@ jobs:32
| Veld | Betekenis |32
| Veld | Betekenis |33
| --- | --- |33
| --- | --- |34
| `stages` | Volgorde. Weggelaten: de stages in de volgorde waarin de jobs ze noemen. Een job zonder stage valt in `test`. |34
| `stages` | Volgorde. Weggelaten: de stages in de volgorde waarin de jobs ze noemen. Een job zonder stage valt in `test`. |35
| `image` | Standaardimage. Een job zonder eigen image gebruikt die. |35
| `image` | Standaardimage, als tekst of als `name:`. `default.image` wint als die gezet is. |36
| `variables` | Omgevingsvariabelen voor elke job. Een job kan ze overschrijven. |36
| `default` | `image`, `before_script`, `after_script`, `tags`, `timeout`, `cache` voor jobs die het zelf niet zetten. |37
| `jobs.<naam>.script` | Niet-lege lijst. Ze draaien na elkaar in `sh`, met `set -e`. |37
| `include` | Alleen `local`. Een job met dezelfde naam wordt samengevoegd; het buitenste bestand wint per veld. |38
| `needs` | Jobs die eerst geslaagd of overgeslagen moeten zijn. `needs: []` start meteen. Weggelaten: wacht op alle eerdere stages. |38
| `variables` | Tekst, of `{value: ...}`. Een job overschrijft dezelfde naam. |39
| `rules` | De job draait als minstens één `if` klopt. Zonder rules draait ze altijd. |39
| `before_script`, `script`, `after_script` | `after_script` draait ook als `script` mislukt. |40
| `artifacts.paths` | Relatieve paden, geen `..`. Na succes bewaard, en in een volgende job gezet die deze job nodig heeft. |40
| `needs` | Namen, of `{job, optional}`. `needs: []` start meteen. Weggelaten: wacht op eerdere stages. |41
| `artifacts.expire_in` | `30m`, `12h`, `14d`, `2w` of `never`. |41
| `rules` | De eerste rule die klopt wint. `if`, `changes`, `exists` en `when`. Zonder rules gelden `only` en `except`. |42
| `allow_failure` | Een mislukking houdt de rest niet tegen. |42
| `only`, `except` | Taknaam, `branches`, `tags`, of `/regex/`. |43
| `when` | `on_success` (standaard), `on_failure`, `always`, `manual`. In een rule ook `never`. |44
| `allow_failure` | Een mislukking houdt de rest niet tegen. De job zelf wordt een waarschuwing. |45
| `artifacts` | `paths` en `expire_in` (`30s`, `30m`, `12h`, `14d`, `2w`, `never`). |46
| `cache` | `paths` en een tekstuele `key`. Lokaal per repository, niet gedeeld met een andere machine. |47
| `resource_group` | Jobs met dezelfde naam in één repository draaien niet tegelijk. |48
| `timeout` | Overschrijft `ci.timeout_seconds` voor die job. |49
| `tags` | Bewaard bij de job. De ingebouwde runner draait elke job; er is geen tweede runner om op tags te kiezen. |50
| `extends` | Eén naam of een lijst. Een job waarvan de naam met `.` begint is een sjabloon en draait niet. |43
51
44
Een `if` is één van:52
Een `if` is één vergelijking, zonder `&&` of `||`:45
53
46
```text54
```text47
$NAAM == "waarde"55
$NAAM == "waarde"56
$NAAM == $ANDERE57
$NAAM == null48
$NAAM != "waarde"58
$NAAM != "waarde"49
$NAAM =~ /reguliere-expressie/59
$NAAM =~ /reguliere-expressie/60
$NAAM !~ /reguliere-expressie/50
```61
```51
62
52
Beschikbaar in rules en in de container: `CI_COMMIT_SHA`, `CI_COMMIT_BRANCH`, `CI_COMMIT_REF`, `CI_PROJECT_PATH`, `CI_PIPELINE_ID`, `CI_JOB_ID`, `CI_JOB_NAME`, `CI_JOB_TOKEN`, `CI_API_URL`.63
`changes` kijkt naar de diff van de push. Is die diff niet te bepalen, dan geldt de voorwaarde. `exists` eist dat elk pad in de commit staat.64
65
Voorgedefinieerd, in rules en in de container: `CI`, `CI_COMMIT_SHA`, `CI_COMMIT_SHORT_SHA`, `CI_COMMIT_BRANCH`, `CI_COMMIT_TAG`, `CI_COMMIT_REF`, `CI_COMMIT_REF_NAME`, `CI_COMMIT_REF_SLUG`, `CI_COMMIT_TITLE`, `CI_DEFAULT_BRANCH`, `CI_PIPELINE_ID`, `CI_PIPELINE_SOURCE`, `CI_PIPELINE_URL`, `CI_PROJECT_PATH`, `CI_PROJECT_NAME`, `CI_PROJECT_NAMESPACE`, `CI_PROJECT_URL`, `CI_JOB_ID`, `CI_JOB_NAME`, `CI_JOB_STAGE`, `CI_JOB_URL`, `CI_JOB_TOKEN`, `CI_API_URL`, `CI_CONFIG_PATH`, `CI_SERVER_NAME`. Op een tag is `CI_COMMIT_BRANCH` leeg. `CI_PIPELINE_SOURCE` is `push` of `web`.66
67
## Bewust niet68
69
`services`, `pages`, `workflow`, `spec`, een include van een ander project of van een URL, `parallel`/`matrix`, trigger- en child-pipelines, `environment`, `retry`, `coverage`, `id_tokens`, `inherit` en `hooks`. Die sleutels geven een waarschuwing in de editor en houden de rest van het bestand niet tegen. Er is één runner, in het BlueGit-proces, die Docker op dezelfde host aanroept.53
70
54
## Geheimen71
## Geheimen55
72
56
Repositorygeheimen komen als omgevingsvariabele mee. Bij een groepsrepository eerst de groep, daarna de repository; dezelfde naam laat de repository winnen. Een geheim dat als beschermd is gemarkeerd, komt alleen mee als de branch op een beschermd patroon past.73
Repositorygeheimen komen als omgevingsvariabele mee, zoals CI/CD-variabelen. Bij een groepsrepository eerst de groep, daarna de repository; dezelfde naam laat de repository winnen. Een geheim dat als beschermd is gemarkeerd, komt alleen mee als de branch of tag op een beschermd patroon past. Waarden van acht tekens of meer worden in het joblog vervangen door `[MASKED]`. `CI_JOB_TOKEN` ook.57
74
58
## Release vanuit een job75
## Release vanuit een job59
76
@@ -68,6 +85,6 @@ Het commit moet in de repository bestaan. Het asset moet een artifact zijn van e68
85
69
## Wanneer een job start86
## Wanneer een job start70
87
71
`Gate` in `internal/ci` geeft `ready`, `wait` of `skip`. `skip` volgt op een mislukte voorganger zonder `allow_failure`. De runner verwerkt pipelines elke twee seconden, met `ci.concurrency` tegelijk (standaard 1, passend bij de mini-PC).88
`Gate` in `internal/ci` geeft `ready`, `wait`, `skip` of `fail`. `skip` volgt op een harde mislukking bij `when: on_success`, of op succes bij `when: on_failure`. `when: always` wacht tot de voorgangers klaar zijn en draait dan toch. Een job met `allow_failure` telt niet als harde mislukking. De runner verwerkt pipelines elke twee seconden, met `ci.concurrency` tegelijk (standaard 1).72
89
73
Ontbreekt Docker, dan faalt de job met een regel in het log. De time-out is `ci.timeout_seconds` (standaard 1800).90
Ontbreekt Docker, dan faalt de job met een regel in het log. De time-out is `ci.timeout_seconds` (standaard 1800).@@ -0,0 +1,488 @@
1
package ci
2
3
import (
4
"fmt"
5
"path"
6
"regexp"
7
"strings"
8
9
"gitlab.com/Timfa/bluegit/internal/i18n"
10
)
11
12
// Variables zet de voorgedefinieerde CI_-variabelen. Job-specifieke namen
13
// vult de runner zelf aan.
14
func Variables(r Ref) map[string]string {
15
name := r.Name
16
kind := r.Kind
17
if kind == "" {
18
name, kind = SplitRef(r.FullRef)
19
}
20
short := r.SHA
21
if len(short) > 8 {
22
short = short[:8]
23
}
24
project := r.ProjectPath
25
ns, base := splitProject(project)
26
env := map[string]string{
27
"CI": "true",
28
"CI_CONFIG_PATH": "blue-ci-cd.yml",
29
"CI_SERVER_NAME": "BlueGit",
30
"CI_COMMIT_SHA": r.SHA,
31
"CI_COMMIT_SHORT_SHA": short,
32
"CI_COMMIT_REF": r.FullRef,
33
"CI_COMMIT_REF_NAME": name,
34
"CI_COMMIT_REF_SLUG": slug(name),
35
"CI_COMMIT_TITLE": r.Title,
36
"CI_COMMIT_BRANCH": "",
37
"CI_COMMIT_TAG": "",
38
"CI_DEFAULT_BRANCH": r.DefaultBranch,
39
"CI_PIPELINE_ID": fmt.Sprint(r.PipelineID),
40
"CI_PIPELINE_SOURCE": r.Source,
41
"CI_PIPELINE_URL": strings.TrimRight(r.ProjectURL, "/") + "/-/pipelines/" + fmt.Sprint(r.PipelineID),
42
"CI_PROJECT_PATH": project,
43
"CI_PROJECT_NAME": base,
44
"CI_PROJECT_NAMESPACE": ns,
45
"CI_PROJECT_URL": r.ProjectURL,
46
"CI_API_URL": r.APIURL,
47
}
48
if kind == "tag" {
49
env["CI_COMMIT_TAG"] = name
50
} else {
51
env["CI_COMMIT_BRANCH"] = name
52
}
53
if env["CI_PIPELINE_SOURCE"] == "" {
54
env["CI_PIPELINE_SOURCE"] = "push"
55
}
56
return env
57
}
58
59
// SplitRef haalt de korte naam en het soort uit refs/heads of refs/tags.
60
func SplitRef(ref string) (name, kind string) {
61
switch {
62
case strings.HasPrefix(ref, "refs/tags/"):
63
return strings.TrimPrefix(ref, "refs/tags/"), "tag"
64
case strings.HasPrefix(ref, "refs/heads/"):
65
return strings.TrimPrefix(ref, "refs/heads/"), "branch"
66
default:
67
return ref, "branch"
68
}
69
}
70
71
func splitProject(path string) (ns, name string) {
72
i := strings.LastIndex(path, "/")
73
if i < 0 {
74
return "", path
75
}
76
return path[:i], path[i+1:]
77
}
78
79
func slug(s string) string {
80
var b strings.Builder
81
prev := false
82
for _, r := range strings.ToLower(s) {
83
if (r >= 'a' && r <= 'z') || (r >= '0' && r <= '9') {
84
b.WriteRune(r)
85
prev = false
86
continue
87
}
88
if !prev && b.Len() > 0 {
89
b.WriteByte('-')
90
prev = true
91
}
92
}
93
return strings.Trim(b.String(), "-")
94
}
95
96
// Select houdt de jobs waarvan de rules, only en except kloppen.
97
// De When van de job wordt de when van de winnende rule.
98
func (p Pipeline) Select(ev Eval) ([]Job, error) {
99
if ev.Env == nil {
100
ev.Env = map[string]string{}
101
}
102
var out []Job
103
active := map[string]bool{}
104
var pending []Job
105
for _, j := range p.Jobs {
106
ok, when, err := j.matches(ev)
107
if err != nil {
108
return nil, fmt.Errorf("job %s: %w", j.Name, err)
109
}
110
if !ok {
111
continue
112
}
113
if when == "" {
114
when = "on_success"
115
}
116
j.When = when
117
pending = append(pending, j)
118
active[j.Name] = true
119
}
120
for i := range pending {
121
var needs []string
122
for _, n := range pending[i].Needs {
123
if active[n] {
124
needs = append(needs, n)
125
continue
126
}
127
if contains(pending[i].Optional, n) {
128
continue
129
}
130
needs = append(needs, n)
131
}
132
if needs == nil {
133
needs = []string{}
134
}
135
pending[i].Needs = needs
136
out = append(out, pending[i])
137
}
138
return out, nil
139
}
140
141
// ActiveJobs is Select met alleen variabelen. changes en exists gelden dan als waar.
142
func (p Pipeline) ActiveJobs(env map[string]string) ([]Job, error) {
143
return p.Select(Eval{Env: env})
144
}
145
146
func (j Job) matches(ev Eval) (bool, string, error) {
147
when := j.When
148
if when == "" {
149
when = "on_success"
150
}
151
if len(j.Rules) == 0 {
152
ok, err := onlyExcept(j, ev)
153
return ok, when, err
154
}
155
for _, r := range j.Rules {
156
ok, err := ruleMatch(r, ev)
157
if err != nil {
158
return false, "", err
159
}
160
if !ok {
161
continue
162
}
163
w := r.When
164
if w == "" {
165
w = when
166
}
167
if w == "never" {
168
return false, w, nil
169
}
170
return true, w, nil
171
}
172
return false, "", nil
173
}
174
175
func ruleMatch(r Rule, ev Eval) (bool, error) {
176
if r.If != "" {
177
ok, err := evalRule(r.If, ev.Env)
178
if err != nil || !ok {
179
return ok, err
180
}
181
}
182
if len(r.Changes) > 0 && ev.Changed != nil {
183
if !anyGlob(r.Changes, ev.Changed) {
184
return false, nil
185
}
186
}
187
if len(r.Exists) > 0 && ev.Files != nil {
188
if !allExist(r.Exists, ev.Files) {
189
return false, nil
190
}
191
}
192
if r.If == "" && len(r.Changes) == 0 && len(r.Exists) == 0 && r.When == "" {
193
return false, nil
194
}
195
return true, nil
196
}
197
198
func onlyExcept(j Job, ev Eval) (bool, error) {
199
if len(j.Only) == 0 && len(j.Except) == 0 {
200
return true, nil
201
}
202
name := ev.Env["CI_COMMIT_REF_NAME"]
203
if name == "" {
204
name = ev.Env["CI_COMMIT_BRANCH"]
205
}
206
tag := ev.Env["CI_COMMIT_TAG"] != ""
207
if len(j.Only) > 0 {
208
ok, err := refListed(j.Only, name, tag)
209
if err != nil || !ok {
210
return false, err
211
}
212
}
213
if len(j.Except) > 0 {
214
ok, err := refListed(j.Except, name, tag)
215
if err != nil {
216
return false, err
217
}
218
if ok {
219
return false, nil
220
}
221
}
222
return true, nil
223
}
224
225
func refListed(list []string, name string, tag bool) (bool, error) {
226
for _, item := range list {
227
switch item {
228
case "branches":
229
if !tag {
230
return true, nil
231
}
232
case "tags":
233
if tag {
234
return true, nil
235
}
236
default:
237
if strings.HasPrefix(item, "/") && strings.HasSuffix(item, "/") && len(item) > 2 {
238
re, err := regexp.Compile(item[1 : len(item)-1])
239
if err != nil {
240
return false, i18n.E("ci.err.regex", item, err.Error())
241
}
242
if re.MatchString(name) {
243
return true, nil
244
}
245
continue
246
}
247
if item == name {
248
return true, nil
249
}
250
}
251
}
252
return false, nil
253
}
254
255
func anyGlob(patterns, names []string) bool {
256
for _, name := range names {
257
for _, pattern := range patterns {
258
if Glob(pattern, name) {
259
return true
260
}
261
}
262
}
263
return false
264
}
265
266
func allExist(patterns, files []string) bool {
267
for _, pattern := range patterns {
268
found := false
269
for _, file := range files {
270
if file == pattern || Glob(pattern, file) {
271
found = true
272
break
273
}
274
}
275
if !found {
276
return false
277
}
278
}
279
return true
280
}
281
282
// Glob volgt GitLab-padpatronen: * blijft in één segment, ** loopt door mappen.
283
func Glob(pattern, name string) bool {
284
pattern = strings.TrimPrefix(pattern, "/")
285
name = strings.TrimPrefix(name, "/")
286
if pattern == name {
287
return true
288
}
289
re := globRegexp(pattern)
290
if re.MatchString(name) {
291
return true
292
}
293
if !strings.Contains(pattern, "/") && re.MatchString(path.Base(name)) {
294
return true
295
}
296
return false
297
}
298
299
func globRegexp(pattern string) *regexp.Regexp {
300
var b strings.Builder
301
b.WriteString("^")
302
for i := 0; i < len(pattern); i++ {
303
if pattern[i] == '*' && i+1 < len(pattern) && pattern[i+1] == '*' {
304
b.WriteString(".*")
305
i++
306
if i+1 < len(pattern) && pattern[i+1] == '/' {
307
i++
308
}
309
continue
310
}
311
switch pattern[i] {
312
case '*':
313
b.WriteString("[^/]*")
314
case '?':
315
b.WriteString("[^/]")
316
default:
317
b.WriteString(regexp.QuoteMeta(pattern[i : i+1]))
318
}
319
}
320
b.WriteString("$")
321
return regexp.MustCompile(b.String())
322
}
323
324
// Gate beslist of een job mag starten.
325
// "ready" betekent dat alle voorgangers klaar zijn.
326
// "wait" betekent dat er nog een voorganger loopt of in de wachtrij staat.
327
// "skip" betekent dat een voorganger mislukte zonder allow_failure,
328
// of dat when: on_failure geen mislukking zag.
329
// "fail" betekent dat een verplichte need niet in deze pipeline zit.
330
//
331
// NeedsSet volgt de genoemde jobs, ook als de lijst leeg is.
332
// Zonder needs wacht de job op alle jobs in eerdere stages.
333
func Gate(jobs []Job, status map[string]string, job Job) string {
334
byName := map[string]Job{}
335
for _, j := range jobs {
336
byName[j.Name] = j
337
}
338
optional := map[string]bool{}
339
for _, n := range job.Optional {
340
optional[n] = true
341
}
342
var deps []Job
343
if job.NeedsSet {
344
for _, n := range job.Needs {
345
d, ok := byName[n]
346
if !ok {
347
if optional[n] {
348
continue
349
}
350
return "fail"
351
}
352
deps = append(deps, d)
353
}
354
} else {
355
stagePos := map[string]int{}
356
order := 0
357
for _, j := range jobs {
358
if _, ok := stagePos[j.Stage]; !ok {
359
stagePos[j.Stage] = order
360
order++
361
}
362
}
363
my := stagePos[job.Stage]
364
for _, j := range jobs {
365
if stagePos[j.Stage] < my {
366
deps = append(deps, j)
367
}
368
}
369
}
370
failedAny := false
371
failedHard := false
372
for _, d := range deps {
373
switch status[d.Name] {
374
case "success", "skipped":
375
continue
376
case "failed":
377
failedAny = true
378
if !d.AllowFailure {
379
failedHard = true
380
}
381
case "manual":
382
if d.AllowFailure {
383
continue
384
}
385
return "wait"
386
default:
387
return "wait"
388
}
389
}
390
when := job.When
391
if when == "" {
392
when = "on_success"
393
}
394
switch when {
395
case "always":
396
return "ready"
397
case "on_failure":
398
if failedAny {
399
return "ready"
400
}
401
return "skip"
402
default:
403
if failedHard {
404
return "skip"
405
}
406
return "ready"
407
}
408
}
409
410
var (
411
reEq = regexp.MustCompile(`^\$([A-Za-z_][A-Za-z0-9_]*)\s*==\s*(?:"([^"]*)"|\$([A-Za-z_][A-Za-z0-9_]*)|null)$`)
412
reNe = regexp.MustCompile(`^\$([A-Za-z_][A-Za-z0-9_]*)\s*!=\s*(?:"([^"]*)"|\$([A-Za-z_][A-Za-z0-9_]*)|null)$`)
413
reRe = regexp.MustCompile(`^\$([A-Za-z_][A-Za-z0-9_]*)\s*=~\s*/(.*)/$`)
414
reNRe = regexp.MustCompile(`^\$([A-Za-z_][A-Za-z0-9_]*)\s*!~\s*/(.*)/$`)
415
)
416
417
type compiledRule struct {
418
kind string
419
name string
420
arg string
421
other string
422
null bool
423
re *regexp.Regexp
424
}
425
426
func compileRule(expr string) (compiledRule, error) {
427
expr = strings.TrimSpace(expr)
428
if strings.Contains(expr, "&&") || strings.Contains(expr, "||") {
429
return compiledRule{}, i18n.E("ci.err.ruleCombine", expr)
430
}
431
if m := reEq.FindStringSubmatch(expr); m != nil {
432
return compiledRule{kind: "eq", name: m[1], arg: m[2], other: m[3], null: m[2] == "" && m[3] == "" && strings.HasSuffix(strings.TrimSpace(expr), "null")}, nil
433
}
434
if m := reNe.FindStringSubmatch(expr); m != nil {
435
return compiledRule{kind: "ne", name: m[1], arg: m[2], other: m[3], null: m[2] == "" && m[3] == "" && strings.HasSuffix(strings.TrimSpace(expr), "null")}, nil
436
}
437
if m := reRe.FindStringSubmatch(expr); m != nil {
438
re, err := regexp.Compile(m[2])
439
if err != nil {
440
return compiledRule{}, i18n.E("ci.err.regex", expr, err.Error())
441
}
442
return compiledRule{kind: "re", name: m[1], re: re}, nil
443
}
444
if m := reNRe.FindStringSubmatch(expr); m != nil {
445
re, err := regexp.Compile(m[2])
446
if err != nil {
447
return compiledRule{}, i18n.E("ci.err.regex", expr, err.Error())
448
}
449
return compiledRule{kind: "nre", name: m[1], re: re}, nil
450
}
451
return compiledRule{}, i18n.E("ci.err.ruleExpr", expr)
452
}
453
454
func evalRule(expr string, env map[string]string) (bool, error) {
455
c, err := compileRule(expr)
456
if err != nil {
457
return false, err
458
}
459
val := env[c.name]
460
right := c.arg
461
if c.other != "" {
462
right = env[c.other]
463
}
464
if c.null {
465
right = ""
466
}
467
switch c.kind {
468
case "eq":
469
return val == right, nil
470
case "ne":
471
return val != right, nil
472
case "re":
473
return c.re.MatchString(val), nil
474
case "nre":
475
return !c.re.MatchString(val), nil
476
default:
477
return false, i18n.E("ci.err.ruleExpr", expr)
478
}
479
}
480
481
func contains(list []string, s string) bool {
482
for _, item := range list {
483
if item == s {
484
return true
485
}
486
}
487
return false
488
}
@@ -0,0 +1,488 @@1
package ci2
3
import (4
"fmt"5
"path"6
"regexp"7
"strings"8
9
"gitlab.com/Timfa/bluegit/internal/i18n"10
)11
12
// Variables zet de voorgedefinieerde CI_-variabelen. Job-specifieke namen13
// vult de runner zelf aan.14
func Variables(r Ref) map[string]string {15
name := r.Name16
kind := r.Kind17
if kind == "" {18
name, kind = SplitRef(r.FullRef)19
}20
short := r.SHA21
if len(short) > 8 {22
short = short[:8]23
}24
project := r.ProjectPath25
ns, base := splitProject(project)26
env := map[string]string{27
"CI": "true",28
"CI_CONFIG_PATH": "blue-ci-cd.yml",29
"CI_SERVER_NAME": "BlueGit",30
"CI_COMMIT_SHA": r.SHA,31
"CI_COMMIT_SHORT_SHA": short,32
"CI_COMMIT_REF": r.FullRef,33
"CI_COMMIT_REF_NAME": name,34
"CI_COMMIT_REF_SLUG": slug(name),35
"CI_COMMIT_TITLE": r.Title,36
"CI_COMMIT_BRANCH": "",37
"CI_COMMIT_TAG": "",38
"CI_DEFAULT_BRANCH": r.DefaultBranch,39
"CI_PIPELINE_ID": fmt.Sprint(r.PipelineID),40
"CI_PIPELINE_SOURCE": r.Source,41
"CI_PIPELINE_URL": strings.TrimRight(r.ProjectURL, "/") + "/-/pipelines/" + fmt.Sprint(r.PipelineID),42
"CI_PROJECT_PATH": project,43
"CI_PROJECT_NAME": base,44
"CI_PROJECT_NAMESPACE": ns,45
"CI_PROJECT_URL": r.ProjectURL,46
"CI_API_URL": r.APIURL,47
}48
if kind == "tag" {49
env["CI_COMMIT_TAG"] = name50
} else {51
env["CI_COMMIT_BRANCH"] = name52
}53
if env["CI_PIPELINE_SOURCE"] == "" {54
env["CI_PIPELINE_SOURCE"] = "push"55
}56
return env57
}58
59
// SplitRef haalt de korte naam en het soort uit refs/heads of refs/tags.60
func SplitRef(ref string) (name, kind string) {61
switch {62
case strings.HasPrefix(ref, "refs/tags/"):63
return strings.TrimPrefix(ref, "refs/tags/"), "tag"64
case strings.HasPrefix(ref, "refs/heads/"):65
return strings.TrimPrefix(ref, "refs/heads/"), "branch"66
default:67
return ref, "branch"68
}69
}70
71
func splitProject(path string) (ns, name string) {72
i := strings.LastIndex(path, "/")73
if i < 0 {74
return "", path75
}76
return path[:i], path[i+1:]77
}78
79
func slug(s string) string {80
var b strings.Builder81
prev := false82
for _, r := range strings.ToLower(s) {83
if (r >= 'a' && r <= 'z') || (r >= '0' && r <= '9') {84
b.WriteRune(r)85
prev = false86
continue87
}88
if !prev && b.Len() > 0 {89
b.WriteByte('-')90
prev = true91
}92
}93
return strings.Trim(b.String(), "-")94
}95
96
// Select houdt de jobs waarvan de rules, only en except kloppen.97
// De When van de job wordt de when van de winnende rule.98
func (p Pipeline) Select(ev Eval) ([]Job, error) {99
if ev.Env == nil {100
ev.Env = map[string]string{}101
}102
var out []Job103
active := map[string]bool{}104
var pending []Job105
for _, j := range p.Jobs {106
ok, when, err := j.matches(ev)107
if err != nil {108
return nil, fmt.Errorf("job %s: %w", j.Name, err)109
}110
if !ok {111
continue112
}113
if when == "" {114
when = "on_success"115
}116
j.When = when117
pending = append(pending, j)118
active[j.Name] = true119
}120
for i := range pending {121
var needs []string122
for _, n := range pending[i].Needs {123
if active[n] {124
needs = append(needs, n)125
continue126
}127
if contains(pending[i].Optional, n) {128
continue129
}130
needs = append(needs, n)131
}132
if needs == nil {133
needs = []string{}134
}135
pending[i].Needs = needs136
out = append(out, pending[i])137
}138
return out, nil139
}140
141
// ActiveJobs is Select met alleen variabelen. changes en exists gelden dan als waar.142
func (p Pipeline) ActiveJobs(env map[string]string) ([]Job, error) {143
return p.Select(Eval{Env: env})144
}145
146
func (j Job) matches(ev Eval) (bool, string, error) {147
when := j.When148
if when == "" {149
when = "on_success"150
}151
if len(j.Rules) == 0 {152
ok, err := onlyExcept(j, ev)153
return ok, when, err154
}155
for _, r := range j.Rules {156
ok, err := ruleMatch(r, ev)157
if err != nil {158
return false, "", err159
}160
if !ok {161
continue162
}163
w := r.When164
if w == "" {165
w = when166
}167
if w == "never" {168
return false, w, nil169
}170
return true, w, nil171
}172
return false, "", nil173
}174
175
func ruleMatch(r Rule, ev Eval) (bool, error) {176
if r.If != "" {177
ok, err := evalRule(r.If, ev.Env)178
if err != nil || !ok {179
return ok, err180
}181
}182
if len(r.Changes) > 0 && ev.Changed != nil {183
if !anyGlob(r.Changes, ev.Changed) {184
return false, nil185
}186
}187
if len(r.Exists) > 0 && ev.Files != nil {188
if !allExist(r.Exists, ev.Files) {189
return false, nil190
}191
}192
if r.If == "" && len(r.Changes) == 0 && len(r.Exists) == 0 && r.When == "" {193
return false, nil194
}195
return true, nil196
}197
198
func onlyExcept(j Job, ev Eval) (bool, error) {199
if len(j.Only) == 0 && len(j.Except) == 0 {200
return true, nil201
}202
name := ev.Env["CI_COMMIT_REF_NAME"]203
if name == "" {204
name = ev.Env["CI_COMMIT_BRANCH"]205
}206
tag := ev.Env["CI_COMMIT_TAG"] != ""207
if len(j.Only) > 0 {208
ok, err := refListed(j.Only, name, tag)209
if err != nil || !ok {210
return false, err211
}212
}213
if len(j.Except) > 0 {214
ok, err := refListed(j.Except, name, tag)215
if err != nil {216
return false, err217
}218
if ok {219
return false, nil220
}221
}222
return true, nil223
}224
225
func refListed(list []string, name string, tag bool) (bool, error) {226
for _, item := range list {227
switch item {228
case "branches":229
if !tag {230
return true, nil231
}232
case "tags":233
if tag {234
return true, nil235
}236
default:237
if strings.HasPrefix(item, "/") && strings.HasSuffix(item, "/") && len(item) > 2 {238
re, err := regexp.Compile(item[1 : len(item)-1])239
if err != nil {240
return false, i18n.E("ci.err.regex", item, err.Error())241
}242
if re.MatchString(name) {243
return true, nil244
}245
continue246
}247
if item == name {248
return true, nil249
}250
}251
}252
return false, nil253
}254
255
func anyGlob(patterns, names []string) bool {256
for _, name := range names {257
for _, pattern := range patterns {258
if Glob(pattern, name) {259
return true260
}261
}262
}263
return false264
}265
266
func allExist(patterns, files []string) bool {267
for _, pattern := range patterns {268
found := false269
for _, file := range files {270
if file == pattern || Glob(pattern, file) {271
found = true272
break273
}274
}275
if !found {276
return false277
}278
}279
return true280
}281
282
// Glob volgt GitLab-padpatronen: * blijft in één segment, ** loopt door mappen.283
func Glob(pattern, name string) bool {284
pattern = strings.TrimPrefix(pattern, "/")285
name = strings.TrimPrefix(name, "/")286
if pattern == name {287
return true288
}289
re := globRegexp(pattern)290
if re.MatchString(name) {291
return true292
}293
if !strings.Contains(pattern, "/") && re.MatchString(path.Base(name)) {294
return true295
}296
return false297
}298
299
func globRegexp(pattern string) *regexp.Regexp {300
var b strings.Builder301
b.WriteString("^")302
for i := 0; i < len(pattern); i++ {303
if pattern[i] == '*' && i+1 < len(pattern) && pattern[i+1] == '*' {304
b.WriteString(".*")305
i++306
if i+1 < len(pattern) && pattern[i+1] == '/' {307
i++308
}309
continue310
}311
switch pattern[i] {312
case '*':313
b.WriteString("[^/]*")314
case '?':315
b.WriteString("[^/]")316
default:317
b.WriteString(regexp.QuoteMeta(pattern[i : i+1]))318
}319
}320
b.WriteString("$")321
return regexp.MustCompile(b.String())322
}323
324
// Gate beslist of een job mag starten.325
// "ready" betekent dat alle voorgangers klaar zijn.326
// "wait" betekent dat er nog een voorganger loopt of in de wachtrij staat.327
// "skip" betekent dat een voorganger mislukte zonder allow_failure,328
// of dat when: on_failure geen mislukking zag.329
// "fail" betekent dat een verplichte need niet in deze pipeline zit.330
//331
// NeedsSet volgt de genoemde jobs, ook als de lijst leeg is.332
// Zonder needs wacht de job op alle jobs in eerdere stages.333
func Gate(jobs []Job, status map[string]string, job Job) string {334
byName := map[string]Job{}335
for _, j := range jobs {336
byName[j.Name] = j337
}338
optional := map[string]bool{}339
for _, n := range job.Optional {340
optional[n] = true341
}342
var deps []Job343
if job.NeedsSet {344
for _, n := range job.Needs {345
d, ok := byName[n]346
if !ok {347
if optional[n] {348
continue349
}350
return "fail"351
}352
deps = append(deps, d)353
}354
} else {355
stagePos := map[string]int{}356
order := 0357
for _, j := range jobs {358
if _, ok := stagePos[j.Stage]; !ok {359
stagePos[j.Stage] = order360
order++361
}362
}363
my := stagePos[job.Stage]364
for _, j := range jobs {365
if stagePos[j.Stage] < my {366
deps = append(deps, j)367
}368
}369
}370
failedAny := false371
failedHard := false372
for _, d := range deps {373
switch status[d.Name] {374
case "success", "skipped":375
continue376
case "failed":377
failedAny = true378
if !d.AllowFailure {379
failedHard = true380
}381
case "manual":382
if d.AllowFailure {383
continue384
}385
return "wait"386
default:387
return "wait"388
}389
}390
when := job.When391
if when == "" {392
when = "on_success"393
}394
switch when {395
case "always":396
return "ready"397
case "on_failure":398
if failedAny {399
return "ready"400
}401
return "skip"402
default:403
if failedHard {404
return "skip"405
}406
return "ready"407
}408
}409
410
var (411
reEq = regexp.MustCompile(`^\$([A-Za-z_][A-Za-z0-9_]*)\s*==\s*(?:"([^"]*)"|\$([A-Za-z_][A-Za-z0-9_]*)|null)$`)412
reNe = regexp.MustCompile(`^\$([A-Za-z_][A-Za-z0-9_]*)\s*!=\s*(?:"([^"]*)"|\$([A-Za-z_][A-Za-z0-9_]*)|null)$`)413
reRe = regexp.MustCompile(`^\$([A-Za-z_][A-Za-z0-9_]*)\s*=~\s*/(.*)/$`)414
reNRe = regexp.MustCompile(`^\$([A-Za-z_][A-Za-z0-9_]*)\s*!~\s*/(.*)/$`)415
)416
417
type compiledRule struct {418
kind string419
name string420
arg string421
other string422
null bool423
re *regexp.Regexp424
}425
426
func compileRule(expr string) (compiledRule, error) {427
expr = strings.TrimSpace(expr)428
if strings.Contains(expr, "&&") || strings.Contains(expr, "||") {429
return compiledRule{}, i18n.E("ci.err.ruleCombine", expr)430
}431
if m := reEq.FindStringSubmatch(expr); m != nil {432
return compiledRule{kind: "eq", name: m[1], arg: m[2], other: m[3], null: m[2] == "" && m[3] == "" && strings.HasSuffix(strings.TrimSpace(expr), "null")}, nil433
}434
if m := reNe.FindStringSubmatch(expr); m != nil {435
return compiledRule{kind: "ne", name: m[1], arg: m[2], other: m[3], null: m[2] == "" && m[3] == "" && strings.HasSuffix(strings.TrimSpace(expr), "null")}, nil436
}437
if m := reRe.FindStringSubmatch(expr); m != nil {438
re, err := regexp.Compile(m[2])439
if err != nil {440
return compiledRule{}, i18n.E("ci.err.regex", expr, err.Error())441
}442
return compiledRule{kind: "re", name: m[1], re: re}, nil443
}444
if m := reNRe.FindStringSubmatch(expr); m != nil {445
re, err := regexp.Compile(m[2])446
if err != nil {447
return compiledRule{}, i18n.E("ci.err.regex", expr, err.Error())448
}449
return compiledRule{kind: "nre", name: m[1], re: re}, nil450
}451
return compiledRule{}, i18n.E("ci.err.ruleExpr", expr)452
}453
454
func evalRule(expr string, env map[string]string) (bool, error) {455
c, err := compileRule(expr)456
if err != nil {457
return false, err458
}459
val := env[c.name]460
right := c.arg461
if c.other != "" {462
right = env[c.other]463
}464
if c.null {465
right = ""466
}467
switch c.kind {468
case "eq":469
return val == right, nil470
case "ne":471
return val != right, nil472
case "re":473
return c.re.MatchString(val), nil474
case "nre":475
return !c.re.MatchString(val), nil476
default:477
return false, i18n.E("ci.err.ruleExpr", expr)478
}479
}480
481
func contains(list []string, s string) bool {482
for _, item := range list {483
if item == s {484
return true485
}486
}487
return false488
}@@ -0,0 +1,26 @@
1
package ci
2
3
import "strings"
4
5
// Mask vervangt geheime waarden door [MASKED].
6
// Alleen waarden van minstens 8 tekens, zodat korte variabelen de log niet uitwissen.
7
func Mask(text string, secrets []string) string {
8
if text == "" || len(secrets) == 0 {
9
return text
10
}
11
ordered := append([]string(nil), secrets...)
12
for i := 0; i < len(ordered); i++ {
13
for j := i + 1; j < len(ordered); j++ {
14
if len(ordered[j]) > len(ordered[i]) {
15
ordered[i], ordered[j] = ordered[j], ordered[i]
16
}
17
}
18
}
19
for _, secret := range ordered {
20
if len(secret) < 8 || strings.TrimSpace(secret) == "" {
21
continue
22
}
23
text = strings.ReplaceAll(text, secret, "[MASKED]")
24
}
25
return text
26
}
@@ -0,0 +1,26 @@1
package ci2
3
import "strings"4
5
// Mask vervangt geheime waarden door [MASKED].6
// Alleen waarden van minstens 8 tekens, zodat korte variabelen de log niet uitwissen.7
func Mask(text string, secrets []string) string {8
if text == "" || len(secrets) == 0 {9
return text10
}11
ordered := append([]string(nil), secrets...)12
for i := 0; i < len(ordered); i++ {13
for j := i + 1; j < len(ordered); j++ {14
if len(ordered[j]) > len(ordered[i]) {15
ordered[i], ordered[j] = ordered[j], ordered[i]16
}17
}18
}19
for _, secret := range ordered {20
if len(secret) < 8 || strings.TrimSpace(secret) == "" {21
continue22
}23
text = strings.ReplaceAll(text, secret, "[MASKED]")24
}25
return text26
}@@ -0,0 +1,94 @@
1
package ci
2
3
import "time"
4
5
// Pipeline is een geparset blue-ci-cd.yml.
6
type Pipeline struct {
7
Stages []string
8
Variables map[string]string
9
Image string
10
Before []string
11
After []string
12
Cache Cache
13
Timeout time.Duration
14
Tags []string
15
Jobs []Job
16
Warnings []Issue
17
}
18
19
// Job is één containerstap.
20
type Job struct {
21
Name string
22
Stage string
23
Image string
24
Before []string
25
Script []string
26
After []string
27
Needs []string
28
Optional []string
29
NeedsSet bool
30
Variables map[string]string
31
Artifacts Artifacts
32
Rules []Rule
33
AllowFailure bool
34
When string
35
Timeout time.Duration
36
Tags []string
37
Cache Cache
38
ResourceGroup string
39
Only []string
40
Except []string
41
}
42
43
// Artifacts zijn paden die na een geslaagde job bewaard blijven.
44
type Artifacts struct {
45
Paths []string
46
ExpireIn time.Duration
47
}
48
49
// Cache is een map die tussen jobs van één repository gedeeld wordt.
50
type Cache struct {
51
Key string
52
Paths []string
53
}
54
55
// Rule is één voorwaarde. De eerste rule die klopt wint.
56
// Zonder rules draait de job, tenzij only of except dat tegenhoudt.
57
type Rule struct {
58
If string
59
Changes []string
60
Exists []string
61
When string
62
}
63
64
// Issue is een fout of een waarschuwing met een regelnummer.
65
type Issue struct {
66
Line int
67
Key string
68
Args []any
69
Warning bool
70
}
71
72
// Eval is de context waarin rules worden beoordeeld.
73
// Changed == nil betekent dat de diff onbekend is; changes klopt dan.
74
// Files == nil betekent dat de boom onbekend is; exists klopt dan.
75
type Eval struct {
76
Env map[string]string
77
Changed []string
78
Files []string
79
}
80
81
// Ref beschrijft de commit waarvoor voorgedefinieerde variabelen gelden.
82
type Ref struct {
83
SHA string
84
FullRef string
85
Name string
86
Kind string
87
Title string
88
DefaultBranch string
89
Source string
90
ProjectPath string
91
ProjectURL string
92
APIURL string
93
PipelineID int64
94
}
@@ -0,0 +1,94 @@1
package ci2
3
import "time"4
5
// Pipeline is een geparset blue-ci-cd.yml.6
type Pipeline struct {7
Stages []string8
Variables map[string]string9
Image string10
Before []string11
After []string12
Cache Cache13
Timeout time.Duration14
Tags []string15
Jobs []Job16
Warnings []Issue17
}18
19
// Job is één containerstap.20
type Job struct {21
Name string22
Stage string23
Image string24
Before []string25
Script []string26
After []string27
Needs []string28
Optional []string29
NeedsSet bool30
Variables map[string]string31
Artifacts Artifacts32
Rules []Rule33
AllowFailure bool34
When string35
Timeout time.Duration36
Tags []string37
Cache Cache38
ResourceGroup string39
Only []string40
Except []string41
}42
43
// Artifacts zijn paden die na een geslaagde job bewaard blijven.44
type Artifacts struct {45
Paths []string46
ExpireIn time.Duration47
}48
49
// Cache is een map die tussen jobs van één repository gedeeld wordt.50
type Cache struct {51
Key string52
Paths []string53
}54
55
// Rule is één voorwaarde. De eerste rule die klopt wint.56
// Zonder rules draait de job, tenzij only of except dat tegenhoudt.57
type Rule struct {58
If string59
Changes []string60
Exists []string61
When string62
}63
64
// Issue is een fout of een waarschuwing met een regelnummer.65
type Issue struct {66
Line int67
Key string68
Args []any69
Warning bool70
}71
72
// Eval is de context waarin rules worden beoordeeld.73
// Changed == nil betekent dat de diff onbekend is; changes klopt dan.74
// Files == nil betekent dat de boom onbekend is; exists klopt dan.75
type Eval struct {76
Env map[string]string77
Changed []string78
Files []string79
}80
81
// Ref beschrijft de commit waarvoor voorgedefinieerde variabelen gelden.82
type Ref struct {83
SHA string84
FullRef string85
Name string86
Kind string87
Title string88
DefaultBranch string89
Source string90
ProjectPath string91
ProjectURL string92
APIURL string93
PipelineID int6494
}@@ -1,327 +1,712 @@
1
1
// Package ci leest blue-ci-cd.yml en bepaalt welke jobs draaien.
2
// De vorm ligt dicht bij GitLab CI: stages, jobs, needs, rules, artifacts.
2
// De vorm volgt GitLab CI waar dat in BlueGit past. Een oud bestand met
3
// een jobs:-map blijft geldig. Jobs mogen ook los op het hoogste niveau staan.
3
4
package ci
4
5
5
6
import (
6
7
"fmt"
7
"regexp"
8
8
"strconv"
9
9
"strings"
10
10
"time"
11
11
12
"gitlab.com/Timfa/bluegit/internal/i18n"
12
13
"gopkg.in/yaml.v3"
13
14
)
14
15
15
// Pipeline is een geparset configuratiebestand.
16
type Pipeline struct {
17
Stages []string
18
Variables map[string]string
19
Image string
20
Jobs []Job
16
// Parse leest YAML. Jobvolgorde blijft de volgorde uit het bestand.
17
func Parse(data []byte) (Pipeline, error) {
18
return ParseIn(data, nil)
19
}
20
21
// ParseIn leest YAML en haalt include: local via read op.
22
func ParseIn(data []byte, read func(string) ([]byte, error)) (Pipeline, error) {
23
p, issues, err := parseBytes(data, read, nil)
24
if err != nil {
25
return Pipeline{}, err
26
}
27
for _, is := range issues {
28
if !is.Warning {
29
return Pipeline{}, is
30
}
31
}
32
p.Warnings = warningsOnly(issues)
33
return p, nil
21
34
}
22
35
23
// Job is één containerstap.
24
type Job struct {
25
Name string
26
Stage string
27
Image string
28
Script []string
29
Needs []string
30
NeedsSet bool
31
Variables map[string]string
32
Artifacts Artifacts
33
Rules []Rule
34
AllowFailure bool
36
// Lint geeft elke fout en waarschuwing, met regelnummer, zonder te stoppen bij de eerste.
37
func Lint(data []byte, read func(string) ([]byte, error)) (Pipeline, []Issue) {
38
p, issues, err := parseBytes(data, read, nil)
39
if err != nil {
40
issues = append([]Issue{{Line: 1, Key: "ci.err.yaml", Args: []any{err.Error()}}}, issues...)
41
}
42
return p, issues
43
}
44
45
func (i Issue) Error() string { return i18n.T(i18n.EN, i.Key, i.Args...) }
46
47
func warningsOnly(in []Issue) []Issue {
48
var out []Issue
49
for _, is := range in {
50
if is.Warning {
51
out = append(out, is)
52
}
53
}
54
return out
35
55
}
36
56
37
// Artifacts zijn paden die na een geslaagde job bewaard blijven.
38
type Artifacts struct {
39
Paths []string
40
ExpireIn time.Duration
57
var reserved = map[string]bool{
58
"stages": true, "variables": true, "image": true, "default": true,
59
"include": true, "before_script": true, "after_script": true, "cache": true,
60
"workflow": true, "services": true, "spec": true, "pages": true, "jobs": true,
41
61
}
42
62
43
// Rule is één voorwaarde. De job draait als minstens één rule klopt.
44
// Zonder rules draait de job altijd.
45
type Rule struct {
46
If string
63
type draft struct {
64
job Job
65
present map[string]bool
66
extends []string
67
line int
47
68
}
48
69
49
// Parse leest YAML. Jobvolgorde blijft de volgorde uit het bestand.
50
func Parse(data []byte) (Pipeline, error) {
70
func parseBytes(data []byte, read func(string) ([]byte, error), stack []string) (Pipeline, []Issue, error) {
51
71
var root yaml.Node
52
72
if err := yaml.Unmarshal(data, &root); err != nil {
53
return Pipeline{}, fmt.Errorf("yaml: %w", err)
73
return Pipeline{}, nil, i18n.E("ci.err.yaml", err.Error())
54
74
}
55
75
if root.Kind != yaml.DocumentNode || len(root.Content) != 1 || root.Content[0].Kind != yaml.MappingNode {
56
return Pipeline{}, fmt.Errorf("blue-ci-cd.yml moet een mapping zijn")
76
return Pipeline{}, []Issue{{Line: 1, Key: "ci.err.mapping"}}, nil
57
77
}
58
78
body := root.Content[0]
59
var p Pipeline
60
p.Variables = map[string]string{}
79
var included Pipeline
80
var issues []Issue
81
for i := 0; i < len(body.Content); i += 2 {
82
if body.Content[i].Value != "include" {
83
continue
84
}
85
part, more, err := parseInclude(body.Content[i+1], read, stack)
86
if err != nil {
87
return Pipeline{}, nil, err
88
}
89
issues = append(issues, more...)
90
included = mergePipeline(included, part)
91
}
92
p := included
93
if p.Variables == nil {
94
p.Variables = map[string]string{}
95
}
96
var def draft
97
def.present = map[string]bool{}
98
var jobs []draft
99
seenLegacy := false
100
seenTop := false
61
101
for i := 0; i < len(body.Content); i += 2 {
62
key := body.Content[i].Value
102
keyNode := body.Content[i]
103
key := keyNode.Value
63
104
val := body.Content[i+1]
64
105
switch key {
106
case "include":
107
continue
65
108
case "stages":
66
stages, err := scalarList(val)
67
if err != nil {
68
return Pipeline{}, fmt.Errorf("stages: %w", err)
109
stages, iss := scalarList(val, "ci.err.list")
110
issues = append(issues, iss...)
111
if len(stages) > 0 {
112
p.Stages = stages
69
113
}
70
p.Stages = stages
71
114
case "variables":
72
vars, err := scalarMap(val)
73
if err != nil {
74
return Pipeline{}, fmt.Errorf("variables: %w", err)
115
vars, iss := scalarMap(val)
116
issues = append(issues, iss...)
117
for k, v := range vars {
118
p.Variables[k] = v
75
119
}
76
p.Variables = vars
77
120
case "image":
78
if val.Kind != yaml.ScalarNode {
79
return Pipeline{}, fmt.Errorf("image moet een tekst zijn")
121
img, iss := imageOf(val)
122
issues = append(issues, iss...)
123
if img != "" {
124
p.Image = img
80
125
}
81
p.Image = val.Value
126
case "before_script":
127
lines, iss := scriptList(val)
128
issues = append(issues, iss...)
129
p.Before = lines
130
def.present["before_script"] = true
131
def.job.Before = lines
132
case "after_script":
133
lines, iss := scriptList(val)
134
issues = append(issues, iss...)
135
p.After = lines
136
def.present["after_script"] = true
137
def.job.After = lines
138
case "cache":
139
c, iss := parseCache(val)
140
issues = append(issues, iss...)
141
p.Cache = c
142
def.present["cache"] = true
143
def.job.Cache = c
144
case "default":
145
d, iss := parseDefault(val)
146
issues = append(issues, iss...)
147
def = overlayDefault(def, d)
148
case "workflow", "services", "spec", "pages":
149
issues = append(issues, Issue{Line: keyNode.Line, Key: "ci.warn.ignored", Args: []any{key}, Warning: true})
82
150
case "jobs":
83
jobs, err := parseJobs(val)
84
if err != nil {
85
return Pipeline{}, err
86
}
87
p.Jobs = jobs
151
seenLegacy = true
152
parsed, iss := parseJobMap(val)
153
issues = append(issues, iss...)
154
jobs = append(jobs, parsed...)
88
155
default:
89
return Pipeline{}, fmt.Errorf("onbekend veld %q", key)
156
if val.Kind != yaml.MappingNode {
157
issues = append(issues, Issue{Line: keyNode.Line, Key: "ci.err.unknown", Args: []any{key}})
158
continue
159
}
160
seenTop = true
161
job, iss := parseJob(key, val)
162
issues = append(issues, iss...)
163
jobs = append(jobs, job)
90
164
}
91
165
}
92
if err := p.validate(); err != nil {
93
return Pipeline{}, err
166
if seenLegacy && seenTop {
167
issues = append(issues, Issue{Line: 1, Key: "ci.err.both"})
168
}
169
if p.Image != "" && !def.present["image"] {
170
def.present["image"] = true
171
def.job.Image = p.Image
172
}
173
resolved, iss := applyExtends(jobs, def)
174
issues = append(issues, iss...)
175
byName := map[string]int{}
176
var merged []Job
177
for _, j := range p.Jobs {
178
byName[j.Name] = len(merged)
179
merged = append(merged, j)
180
}
181
for _, d := range resolved {
182
if strings.HasPrefix(d.job.Name, ".") {
183
continue
184
}
185
if i, ok := byName[d.job.Name]; ok {
186
merged[i] = overlayJob(merged[i], d)
187
continue
188
}
189
byName[d.job.Name] = len(merged)
190
merged = append(merged, d.job)
191
}
192
p.Jobs = merged
193
issues = append(issues, validate(&p)...)
194
if hasError(issues) {
195
return p, issues, nil
196
}
197
return p, issues, nil
198
}
199
200
func parseInclude(node *yaml.Node, read func(string) ([]byte, error), stack []string) (Pipeline, []Issue, error) {
201
var items []*yaml.Node
202
switch node.Kind {
203
case yaml.SequenceNode:
204
items = node.Content
205
default:
206
items = []*yaml.Node{node}
207
}
208
var out Pipeline
209
var issues []Issue
210
for _, item := range items {
211
local, iss := includeLocal(item)
212
issues = append(issues, iss...)
213
if local == "" {
214
continue
215
}
216
if read == nil {
217
issues = append(issues, Issue{Line: item.Line, Key: "ci.err.includeRead", Args: []any{local}})
218
continue
219
}
220
for _, prev := range stack {
221
if prev == local {
222
issues = append(issues, Issue{Line: item.Line, Key: "ci.err.includeCycle", Args: []any{local}})
223
local = ""
224
break
225
}
226
}
227
if local == "" {
228
continue
229
}
230
raw, err := read(local)
231
if err != nil {
232
issues = append(issues, Issue{Line: item.Line, Key: "ci.err.includeRead", Args: []any{local}})
233
continue
234
}
235
part, more, err := parseBytes(raw, read, append(append([]string{}, stack...), local))
236
if err != nil {
237
return Pipeline{}, nil, err
238
}
239
issues = append(issues, more...)
240
out = mergePipeline(out, part)
241
}
242
return out, issues, nil
243
}
244
245
func includeLocal(node *yaml.Node) (string, []Issue) {
246
switch node.Kind {
247
case yaml.ScalarNode:
248
return cleanLocal(node.Value, node)
249
case yaml.MappingNode:
250
for i := 0; i < len(node.Content); i += 2 {
251
key := node.Content[i].Value
252
if key == "local" {
253
if node.Content[i+1].Kind != yaml.ScalarNode {
254
return "", []Issue{{Line: node.Line, Key: "ci.err.includeLocal"}}
255
}
256
return cleanLocal(node.Content[i+1].Value, node.Content[i+1])
257
}
258
}
259
return "", []Issue{{Line: node.Line, Key: "ci.err.includeRemote"}}
260
default:
261
return "", []Issue{{Line: node.Line, Key: "ci.err.includeLocal"}}
262
}
263
}
264
265
func cleanLocal(p string, node *yaml.Node) (string, []Issue) {
266
p = strings.TrimPrefix(strings.TrimSpace(p), "/")
267
if p == "" || strings.Contains(p, "\\") || strings.Contains(p, "..") {
268
return "", []Issue{{Line: node.Line, Key: "ci.err.rel", Args: []any{p}}}
94
269
}
95
270
return p, nil
96
271
}
97
272
98
func parseJobs(node *yaml.Node) ([]Job, error) {
273
func mergePipeline(base, extra Pipeline) Pipeline {
274
if len(extra.Stages) > 0 {
275
base.Stages = extra.Stages
276
}
277
if base.Variables == nil {
278
base.Variables = map[string]string{}
279
}
280
for k, v := range extra.Variables {
281
base.Variables[k] = v
282
}
283
if extra.Image != "" {
284
base.Image = extra.Image
285
}
286
if extra.Before != nil {
287
base.Before = extra.Before
288
}
289
if extra.After != nil {
290
base.After = extra.After
291
}
292
byName := map[string]int{}
293
for i, j := range base.Jobs {
294
byName[j.Name] = i
295
}
296
for _, j := range extra.Jobs {
297
if i, ok := byName[j.Name]; ok {
298
base.Jobs[i] = j
299
continue
300
}
301
byName[j.Name] = len(base.Jobs)
302
base.Jobs = append(base.Jobs, j)
303
}
304
return base
305
}
306
307
func parseDefault(node *yaml.Node) (draft, []Issue) {
308
d := draft{present: map[string]bool{}}
99
309
if node.Kind != yaml.MappingNode {
100
return nil, fmt.Errorf("jobs moet een mapping zijn")
310
return d, []Issue{{Line: node.Line, Key: "ci.err.map"}}
101
311
}
102
var jobs []Job
312
var issues []Issue
103
313
for i := 0; i < len(node.Content); i += 2 {
104
name := node.Content[i].Value
105
body := node.Content[i+1]
106
job, err := parseJob(name, body)
107
if err != nil {
108
return nil, fmt.Errorf("job %s: %w", name, err)
314
key := node.Content[i].Value
315
val := node.Content[i+1]
316
switch key {
317
case "image":
318
img, iss := imageOf(val)
319
issues = append(issues, iss...)
320
d.job.Image = img
321
d.present["image"] = true
322
case "before_script":
323
lines, iss := scriptList(val)
324
issues = append(issues, iss...)
325
d.job.Before = lines
326
d.present["before_script"] = true
327
case "after_script":
328
lines, iss := scriptList(val)
329
issues = append(issues, iss...)
330
d.job.After = lines
331
d.present["after_script"] = true
332
case "tags":
333
tags, iss := scalarList(val, "ci.err.list")
334
issues = append(issues, iss...)
335
d.job.Tags = tags
336
d.present["tags"] = true
337
case "timeout":
338
dur, iss := durationOf(val)
339
issues = append(issues, iss...)
340
d.job.Timeout = dur
341
d.present["timeout"] = true
342
case "cache":
343
c, iss := parseCache(val)
344
issues = append(issues, iss...)
345
d.job.Cache = c
346
d.present["cache"] = true
347
default:
348
issues = append(issues, Issue{Line: node.Content[i].Line, Key: "ci.warn.ignored", Args: []any{"default." + key}, Warning: true})
109
349
}
110
jobs = append(jobs, job)
111
350
}
112
return jobs, nil
351
return d, issues
113
352
}
114
353
115
func parseJob(name string, node *yaml.Node) (Job, error) {
354
func overlayJob(base Job, d draft) Job {
355
set := func(key string, apply func()) {
356
if d.present[key] {
357
apply()
358
}
359
}
360
set("stage", func() { base.Stage = d.job.Stage })
361
set("image", func() { base.Image = d.job.Image })
362
set("before_script", func() { base.Before = d.job.Before })
363
set("script", func() { base.Script = d.job.Script })
364
set("after_script", func() { base.After = d.job.After })
365
set("needs", func() {
366
base.Needs = d.job.Needs
367
base.Optional = d.job.Optional
368
base.NeedsSet = d.job.NeedsSet
369
})
370
set("artifacts", func() { base.Artifacts = d.job.Artifacts })
371
set("rules", func() { base.Rules = d.job.Rules })
372
set("allow_failure", func() { base.AllowFailure = d.job.AllowFailure })
373
set("when", func() { base.When = d.job.When })
374
set("timeout", func() { base.Timeout = d.job.Timeout })
375
set("tags", func() { base.Tags = d.job.Tags })
376
set("cache", func() { base.Cache = d.job.Cache })
377
set("resource_group", func() { base.ResourceGroup = d.job.ResourceGroup })
378
set("only", func() { base.Only = d.job.Only })
379
set("except", func() { base.Except = d.job.Except })
380
if d.present["variables"] {
381
mergedVars := map[string]string{}
382
for k, v := range base.Variables {
383
mergedVars[k] = v
384
}
385
for k, v := range d.job.Variables {
386
mergedVars[k] = v
387
}
388
base.Variables = mergedVars
389
}
390
return base
391
}
392
393
func overlayDefault(base, extra draft) draft {
394
if base.present == nil {
395
base.present = map[string]bool{}
396
}
397
for k, on := range extra.present {
398
if !on {
399
continue
400
}
401
base.present[k] = true
402
switch k {
403
case "image":
404
base.job.Image = extra.job.Image
405
case "before_script":
406
base.job.Before = extra.job.Before
407
case "after_script":
408
base.job.After = extra.job.After
409
case "tags":
410
base.job.Tags = extra.job.Tags
411
case "timeout":
412
base.job.Timeout = extra.job.Timeout
413
case "cache":
414
base.job.Cache = extra.job.Cache
415
}
416
}
417
return base
418
}
419
420
func parseJobMap(node *yaml.Node) ([]draft, []Issue) {
116
421
if node.Kind != yaml.MappingNode {
117
return Job{}, fmt.Errorf("moet een mapping zijn")
422
return nil, []Issue{{Line: node.Line, Key: "ci.err.jobsMap"}}
118
423
}
119
j := Job{Name: name, Variables: map[string]string{}}
424
var jobs []draft
425
var issues []Issue
426
for i := 0; i < len(node.Content); i += 2 {
427
job, iss := parseJob(node.Content[i].Value, node.Content[i+1])
428
issues = append(issues, iss...)
429
if job.job.Name != "" {
430
jobs = append(jobs, job)
431
}
432
}
433
return jobs, issues
434
}
435
436
func parseJob(name string, node *yaml.Node) (draft, []Issue) {
437
d := draft{present: map[string]bool{}, line: node.Line, job: Job{Name: name, Variables: map[string]string{}}}
438
if node.Kind != yaml.MappingNode {
439
return d, []Issue{{Line: node.Line, Key: "ci.err.jobMap", Args: []any{name}}}
440
}
441
var issues []Issue
120
442
for i := 0; i < len(node.Content); i += 2 {
121
443
key := node.Content[i].Value
122
444
val := node.Content[i+1]
445
d.present[key] = true
123
446
switch key {
124
447
case "stage":
125
448
if val.Kind != yaml.ScalarNode {
126
return Job{}, fmt.Errorf("stage moet tekst zijn")
449
issues = append(issues, Issue{Line: val.Line, Key: "ci.err.stageText"})
450
continue
127
451
}
128
j.Stage = val.Value
452
d.job.Stage = val.Value
129
453
case "image":
130
if val.Kind != yaml.ScalarNode {
131
return Job{}, fmt.Errorf("image moet tekst zijn")
132
}
133
j.Image = val.Value
454
img, iss := imageOf(val)
455
issues = append(issues, iss...)
456
d.job.Image = img
457
case "before_script":
458
lines, iss := scriptList(val)
459
issues = append(issues, iss...)
460
d.job.Before = lines
134
461
case "script":
135
lines, err := scalarList(val)
136
if err != nil {
137
return Job{}, fmt.Errorf("script: %w", err)
138
}
139
j.Script = lines
462
lines, iss := scriptList(val)
463
issues = append(issues, iss...)
464
d.job.Script = lines
465
case "after_script":
466
lines, iss := scriptList(val)
467
issues = append(issues, iss...)
468
d.job.After = lines
140
469
case "needs":
141
needs, err := scalarList(val)
142
if err != nil {
143
return Job{}, fmt.Errorf("needs: %w", err)
144
}
145
j.Needs = needs
146
j.NeedsSet = true
470
needs, optional, set, iss := parseNeeds(val)
471
issues = append(issues, iss...)
472
d.job.Needs = needs
473
d.job.Optional = optional
474
d.job.NeedsSet = set
147
475
case "variables":
148
vars, err := scalarMap(val)
149
if err != nil {
150
return Job{}, fmt.Errorf("variables: %w", err)
151
}
152
j.Variables = vars
476
vars, iss := scalarMap(val)
477
issues = append(issues, iss...)
478
d.job.Variables = vars
153
479
case "artifacts":
154
art, err := parseArtifacts(val)
155
if err != nil {
156
return Job{}, err
157
}
158
j.Artifacts = art
480
art, iss := parseArtifacts(val)
481
issues = append(issues, iss...)
482
d.job.Artifacts = art
159
483
case "rules":
160
rules, err := parseRules(val)
161
if err != nil {
162
return Job{}, err
163
}
164
j.Rules = rules
484
rules, iss := parseRules(val)
485
issues = append(issues, iss...)
486
d.job.Rules = rules
165
487
case "allow_failure":
166
b, err := parseBool(val)
167
if err != nil {
168
return Job{}, fmt.Errorf("allow_failure: %w", err)
488
b, iss := parseBool(val)
489
issues = append(issues, iss...)
490
d.job.AllowFailure = b
491
case "when":
492
w, iss := whenOf(val, false)
493
issues = append(issues, iss...)
494
d.job.When = w
495
case "timeout":
496
dur, iss := durationOf(val)
497
issues = append(issues, iss...)
498
d.job.Timeout = dur
499
case "tags":
500
tags, iss := scalarList(val, "ci.err.list")
501
issues = append(issues, iss...)
502
d.job.Tags = tags
503
case "cache":
504
c, iss := parseCache(val)
505
issues = append(issues, iss...)
506
d.job.Cache = c
507
case "resource_group":
508
if val.Kind != yaml.ScalarNode || strings.TrimSpace(val.Value) == "" {
509
issues = append(issues, Issue{Line: val.Line, Key: "ci.err.resource"})
510
continue
169
511
}
170
j.AllowFailure = b
512
d.job.ResourceGroup = strings.TrimSpace(val.Value)
513
case "only":
514
list, iss := scalarList(val, "ci.err.list")
515
issues = append(issues, iss...)
516
d.job.Only = list
517
case "except":
518
list, iss := scalarList(val, "ci.err.list")
519
issues = append(issues, iss...)
520
d.job.Except = list
521
case "extends":
522
list, iss := extendsOf(val)
523
issues = append(issues, iss...)
524
d.extends = list
171
525
default:
172
return Job{}, fmt.Errorf("onbekend veld %q", key)
526
if _, ok := reserved[key]; ok {
527
issues = append(issues, Issue{Line: node.Content[i].Line, Key: "ci.warn.ignored", Args: []any{key}, Warning: true})
528
continue
529
}
530
issues = append(issues, Issue{Line: node.Content[i].Line, Key: "ci.warn.ignored", Args: []any{name + "." + key}, Warning: true})
173
531
}
174
532
}
175
return j, nil
533
return d, issues
176
534
}
177
535
178
func parseArtifacts(node *yaml.Node) (Artifacts, error) {
179
if node.Kind != yaml.MappingNode {
180
return Artifacts{}, fmt.Errorf("artifacts moet een mapping zijn")
536
func applyExtends(jobs []draft, def draft) ([]draft, []Issue) {
537
by := map[string]*draft{}
538
var order []string
539
var issues []Issue
540
for i := range jobs {
541
name := jobs[i].job.Name
542
if by[name] != nil {
543
issues = append(issues, Issue{Line: jobs[i].line, Key: "ci.err.jobDup", Args: []any{name}})
544
continue
545
}
546
by[name] = &jobs[i]
547
order = append(order, name)
181
548
}
182
var a Artifacts
183
for i := 0; i < len(node.Content); i += 2 {
184
key := node.Content[i].Value
185
val := node.Content[i+1]
186
switch key {
187
case "paths":
188
paths, err := scalarList(val)
189
if err != nil {
190
return Artifacts{}, fmt.Errorf("artifacts.paths: %w", err)
191
}
192
a.Paths = paths
193
case "expire_in":
194
if val.Kind != yaml.ScalarNode {
195
return Artifacts{}, fmt.Errorf("expire_in moet tekst zijn")
196
}
197
d, err := parseExpire(val.Value)
198
if err != nil {
199
return Artifacts{}, err
549
visiting := map[string]bool{}
550
var walk func(string)
551
walk = func(name string) {
552
d := by[name]
553
if d == nil || d.present["__resolved"] {
554
return
555
}
556
if visiting[name] {
557
issues = append(issues, Issue{Line: d.line, Key: "ci.err.extendsCycle", Args: []any{name}})
558
return
559
}
560
visiting[name] = true
561
for _, parent := range d.extends {
562
base := by[parent]
563
if base == nil {
564
issues = append(issues, Issue{Line: d.line, Key: "ci.err.extends", Args: []any{name, parent}})
565
continue
200
566
}
201
a.ExpireIn = d
202
default:
203
return Artifacts{}, fmt.Errorf("onbekend artifacts-veld %q", key)
567
walk(parent)
568
inherit(d, base)
204
569
}
570
delete(visiting, name)
571
d.present["__resolved"] = true
572
inherit(d, &def)
573
}
574
for _, name := range order {
575
walk(name)
205
576
}
206
return a, nil
577
return jobs, issues
207
578
}
208
579
209
func parseRules(node *yaml.Node) ([]Rule, error) {
210
if node.Kind != yaml.SequenceNode {
211
return nil, fmt.Errorf("rules moet een lijst zijn")
580
func inherit(child, parent *draft) {
581
if parent == nil {
582
return
212
583
}
213
var rules []Rule
214
for _, item := range node.Content {
215
if item.Kind != yaml.MappingNode {
216
return nil, fmt.Errorf("rule moet een mapping zijn")
584
copyField := func(key string, apply func()) {
585
if child.present[key] || !parent.present[key] {
586
return
217
587
}
218
var r Rule
219
for i := 0; i < len(item.Content); i += 2 {
220
key := item.Content[i].Value
221
val := item.Content[i+1]
222
if key != "if" {
223
return nil, fmt.Errorf("rule ondersteunt alleen if, kreeg %q", key)
224
}
225
if val.Kind != yaml.ScalarNode || strings.TrimSpace(val.Value) == "" {
226
return nil, fmt.Errorf("rule.if moet tekst zijn")
227
}
228
r.If = strings.TrimSpace(val.Value)
229
}
230
if r.If == "" {
231
return nil, fmt.Errorf("rule mist if")
588
apply()
589
child.present[key] = true
590
}
591
copyField("stage", func() { child.job.Stage = parent.job.Stage })
592
copyField("image", func() { child.job.Image = parent.job.Image })
593
copyField("before_script", func() { child.job.Before = parent.job.Before })
594
copyField("script", func() { child.job.Script = parent.job.Script })
595
copyField("after_script", func() { child.job.After = parent.job.After })
596
copyField("needs", func() {
597
child.job.Needs = parent.job.Needs
598
child.job.Optional = parent.job.Optional
599
child.job.NeedsSet = parent.job.NeedsSet
600
})
601
copyField("artifacts", func() { child.job.Artifacts = parent.job.Artifacts })
602
copyField("rules", func() { child.job.Rules = parent.job.Rules })
603
copyField("allow_failure", func() { child.job.AllowFailure = parent.job.AllowFailure })
604
copyField("when", func() { child.job.When = parent.job.When })
605
copyField("timeout", func() { child.job.Timeout = parent.job.Timeout })
606
copyField("tags", func() { child.job.Tags = parent.job.Tags })
607
copyField("cache", func() { child.job.Cache = parent.job.Cache })
608
copyField("resource_group", func() { child.job.ResourceGroup = parent.job.ResourceGroup })
609
copyField("only", func() { child.job.Only = parent.job.Only })
610
copyField("except", func() { child.job.Except = parent.job.Except })
611
if len(parent.job.Variables) > 0 {
612
merged := map[string]string{}
613
for k, v := range parent.job.Variables {
614
merged[k] = v
232
615
}
233
if _, err := compileRule(r.If); err != nil {
234
return nil, err
616
for k, v := range child.job.Variables {
617
merged[k] = v
235
618
}
236
rules = append(rules, r)
619
child.job.Variables = merged
237
620
}
238
return rules, nil
239
621
}
240
622
241
func (p *Pipeline) validate() error {
623
func validate(p *Pipeline) []Issue {
624
var issues []Issue
242
625
if len(p.Jobs) == 0 {
243
return fmt.Errorf("geen jobs")
626
return []Issue{{Line: 1, Key: "ci.err.noJobs"}}
244
627
}
245
628
seenStage := map[string]bool{}
246
629
for _, s := range p.Stages {
247
630
if s == "" {
248
return fmt.Errorf("lege stage-naam")
631
issues = append(issues, Issue{Line: 1, Key: "ci.err.stageEmpty"})
632
continue
249
633
}
250
634
if seenStage[s] {
251
return fmt.Errorf("dubbele stage %q", s)
635
issues = append(issues, Issue{Line: 1, Key: "ci.err.stageDup", Args: []any{s}})
252
636
}
253
637
seenStage[s] = true
254
638
}
255
639
if len(p.Stages) == 0 {
256
640
var order []string
257
have := map[string]bool{}
258
641
for i := range p.Jobs {
259
642
if p.Jobs[i].Stage == "" {
260
643
p.Jobs[i].Stage = "test"
261
644
}
262
if !have[p.Jobs[i].Stage] {
263
have[p.Jobs[i].Stage] = true
645
if !seenStage[p.Jobs[i].Stage] {
646
seenStage[p.Jobs[i].Stage] = true
264
647
order = append(order, p.Jobs[i].Stage)
265
648
}
266
649
}
267
650
p.Stages = order
268
for _, s := range p.Stages {
269
seenStage[s] = true
270
}
271
651
}
272
652
names := map[string]bool{}
273
653
for i := range p.Jobs {
274
654
j := &p.Jobs[i]
275
655
if j.Name == "" || strings.ContainsAny(j.Name, " \t\n/\\") {
276
return fmt.Errorf("ongeldige jobnaam %q", j.Name)
277
}
278
if names[j.Name] {
279
return fmt.Errorf("dubbele job %q", j.Name)
656
issues = append(issues, Issue{Line: 1, Key: "ci.err.jobName", Args: []any{j.Name}})
657
continue
280
658
}
281
659
names[j.Name] = true
282
660
if j.Stage == "" {
283
return fmt.Errorf("job %s mist stage", j.Name)
284
}
285
if !seenStage[j.Stage] {
286
return fmt.Errorf("job %s gebruikt onbekende stage %q", j.Name, j.Stage)
661
issues = append(issues, Issue{Line: 1, Key: "ci.err.jobStage", Args: []any{j.Name}})
662
} else if !seenStage[j.Stage] {
663
issues = append(issues, Issue{Line: 1, Key: "ci.err.stageUnknown", Args: []any{j.Name, j.Stage}})
287
664
}
288
665
if j.Image == "" {
289
666
j.Image = p.Image
290
667
}
291
668
if strings.TrimSpace(j.Image) == "" || strings.ContainsAny(j.Image, " \t\n") {
292
return fmt.Errorf("job %s mist een image", j.Name)
669
issues = append(issues, Issue{Line: 1, Key: "ci.err.noImage", Args: []any{j.Name}})
293
670
}
294
671
if len(j.Script) == 0 {
295
return fmt.Errorf("job %s mist script", j.Name)
672
issues = append(issues, Issue{Line: 1, Key: "ci.err.noScript", Args: []any{j.Name}})
296
673
}
297
674
for _, line := range j.Script {
298
675
if strings.TrimSpace(line) == "" {
299
return fmt.Errorf("job %s heeft een lege scriptregel", j.Name)
676
issues = append(issues, Issue{Line: 1, Key: "ci.err.emptyLine", Args: []any{j.Name}})
300
677
}
301
678
}
302
679
for _, path := range j.Artifacts.Paths {
303
if err := safeRelPath(path); err != nil {
304
return fmt.Errorf("job %s artifact: %w", j.Name, err)
680
if iss := safeRel(path); iss.Key != "" {
681
issues = append(issues, iss)
305
682
}
306
683
}
684
for _, path := range j.Cache.Paths {
685
if iss := safeRel(path); iss.Key != "" {
686
issues = append(issues, iss)
687
}
688
}
689
if j.When == "" {
690
j.When = "on_success"
691
}
307
692
}
308
693
for _, j := range p.Jobs {
309
694
for _, need := range j.Needs {
310
if !names[need] {
311
return fmt.Errorf("job %s needs onbekende job %q", j.Name, need)
695
if !names[need] && !contains(j.Optional, need) {
696
issues = append(issues, Issue{Line: 1, Key: "ci.err.needUnknown", Args: []any{j.Name, need}})
312
697
}
313
698
if need == j.Name {
314
return fmt.Errorf("job %s kan niet van zichzelf afhangen", j.Name)
699
issues = append(issues, Issue{Line: 1, Key: "ci.err.needSelf", Args: []any{j.Name}})
315
700
}
316
701
}
317
702
}
318
if err := detectCycles(p.Jobs); err != nil {
319
return err
703
if iss := detectCycles(p.Jobs); iss.Key != "" {
704
issues = append(issues, iss)
320
705
}
321
return nil
706
return issues
322
707
}
323
708
324
func detectCycles(jobs []Job) error {
709
func detectCycles(jobs []Job) Issue {
325
710
idx := map[string]int{}
326
711
for i, j := range jobs {
327
712
idx[j.Name] = i
@@ -332,237 +717,382 @@ func detectCycles(jobs []Job) error {
332
717
black = 2
333
718
)
334
719
color := make([]int, len(jobs))
335
var visit func(int) error
336
visit = func(i int) error {
720
var visit func(int) Issue
721
visit = func(i int) Issue {
337
722
if color[i] == gray {
338
return fmt.Errorf("cyclus in needs bij job %s", jobs[i].Name)
723
return Issue{Line: 1, Key: "ci.err.cycle", Args: []any{jobs[i].Name}}
339
724
}
340
725
if color[i] == black {
341
return nil
726
return Issue{}
342
727
}
343
728
color[i] = gray
344
729
for _, n := range jobs[i].Needs {
345
if err := visit(idx[n]); err != nil {
346
return err
730
target, ok := idx[n]
731
if !ok {
732
continue
733
}
734
if iss := visit(target); iss.Key != "" {
735
return iss
347
736
}
348
737
}
349
738
color[i] = black
350
return nil
739
return Issue{}
351
740
}
352
741
for i := range jobs {
353
if err := visit(i); err != nil {
354
return err
742
if iss := visit(i); iss.Key != "" {
743
return iss
355
744
}
356
745
}
357
return nil
746
return Issue{}
358
747
}
359
748
360
// ActiveJobs filtert jobs waarvan de rules niet kloppen.
361
func (p Pipeline) ActiveJobs(env map[string]string) ([]Job, error) {
362
var out []Job
363
for _, j := range p.Jobs {
364
ok, err := j.matches(env)
365
if err != nil {
366
return nil, fmt.Errorf("job %s: %w", j.Name, err)
367
}
368
if ok {
369
out = append(out, j)
749
func parseArtifacts(node *yaml.Node) (Artifacts, []Issue) {
750
if node.Kind != yaml.MappingNode {
751
return Artifacts{}, []Issue{{Line: node.Line, Key: "ci.err.artifacts"}}
752
}
753
var a Artifacts
754
var issues []Issue
755
for i := 0; i < len(node.Content); i += 2 {
756
key := node.Content[i].Value
757
val := node.Content[i+1]
758
switch key {
759
case "paths":
760
paths, iss := scalarList(val, "ci.err.list")
761
issues = append(issues, iss...)
762
a.Paths = paths
763
case "expire_in":
764
if val.Kind != yaml.ScalarNode {
765
issues = append(issues, Issue{Line: val.Line, Key: "ci.err.expire"})
766
continue
767
}
768
d, err := parseExpire(val.Value)
769
if err != nil {
770
issues = append(issues, Issue{Line: val.Line, Key: "ci.err.expireBad", Args: []any{val.Value}})
771
continue
772
}
773
a.ExpireIn = d
774
default:
775
issues = append(issues, Issue{Line: node.Content[i].Line, Key: "ci.warn.ignored", Args: []any{"artifacts." + key}, Warning: true})
370
776
}
371
777
}
372
return out, nil
778
return a, issues
373
779
}
374
780
375
func (j Job) matches(env map[string]string) (bool, error) {
376
if len(j.Rules) == 0 {
377
return true, nil
781
func parseCache(node *yaml.Node) (Cache, []Issue) {
782
if node.Kind != yaml.MappingNode {
783
return Cache{}, []Issue{{Line: node.Line, Key: "ci.err.cache"}}
378
784
}
379
for _, r := range j.Rules {
380
ok, err := evalRule(r.If, env)
381
if err != nil {
382
return false, err
383
}
384
if ok {
385
return true, nil
386
}
387
}
388
return false, nil
389
}
390
391
// Gate beslist of een job mag starten.
392
// "ready" betekent dat alle voorgangers klaar zijn.
393
// "wait" betekent dat er nog een voorganger loopt of in de wachtrij staat.
394
// "skip" betekent dat een voorganger mislukte zonder allow_failure.
395
//
396
// NeedsSet volgt de genoemde jobs, ook als de lijst leeg is.
397
// Zonder needs wacht de job op alle jobs in eerdere stages.
398
func Gate(jobs []Job, status map[string]string, job Job) string {
399
byName := map[string]Job{}
400
for _, j := range jobs {
401
byName[j.Name] = j
402
}
403
var deps []Job
404
if job.NeedsSet {
405
for _, n := range job.Needs {
406
deps = append(deps, byName[n])
407
}
408
} else {
409
stagePos := map[string]int{}
410
order := 0
411
for _, j := range jobs {
412
if _, ok := stagePos[j.Stage]; !ok {
413
stagePos[j.Stage] = order
414
order++
785
var c Cache
786
var issues []Issue
787
for i := 0; i < len(node.Content); i += 2 {
788
key := node.Content[i].Value
789
val := node.Content[i+1]
790
switch key {
791
case "paths":
792
paths, iss := scalarList(val, "ci.err.list")
793
issues = append(issues, iss...)
794
c.Paths = paths
795
case "key":
796
if val.Kind != yaml.ScalarNode {
797
issues = append(issues, Issue{Line: val.Line, Key: "ci.err.cacheKey"})
798
continue
415
799
}
800
c.Key = val.Value
801
default:
802
issues = append(issues, Issue{Line: node.Content[i].Line, Key: "ci.warn.ignored", Args: []any{"cache." + key}, Warning: true})
416
803
}
417
my := stagePos[job.Stage]
418
for _, j := range jobs {
419
if stagePos[j.Stage] < my {
420
deps = append(deps, j)
804
}
805
return c, issues
806
}
807
808
func parseRules(node *yaml.Node) ([]Rule, []Issue) {
809
if node.Kind != yaml.SequenceNode {
810
return nil, []Issue{{Line: node.Line, Key: "ci.err.rulesList"}}
811
}
812
var rules []Rule
813
var issues []Issue
814
for _, item := range node.Content {
815
if item.Kind != yaml.MappingNode {
816
issues = append(issues, Issue{Line: item.Line, Key: "ci.err.ruleMap"})
817
continue
818
}
819
var r Rule
820
for i := 0; i < len(item.Content); i += 2 {
821
key := item.Content[i].Value
822
val := item.Content[i+1]
823
switch key {
824
case "if":
825
if val.Kind != yaml.ScalarNode || strings.TrimSpace(val.Value) == "" {
826
issues = append(issues, Issue{Line: val.Line, Key: "ci.err.ruleIf"})
827
continue
828
}
829
r.If = strings.TrimSpace(val.Value)
830
if _, err := compileRule(r.If); err != nil {
831
if ie, ok := err.(i18n.Error); ok {
832
issues = append(issues, Issue{Line: val.Line, Key: ie.Key, Args: ie.Args})
833
} else {
834
issues = append(issues, Issue{Line: val.Line, Key: "ci.err.ruleExpr", Args: []any{r.If}})
835
}
836
}
837
case "changes":
838
list, iss := scalarList(val, "ci.err.list")
839
issues = append(issues, iss...)
840
r.Changes = list
841
case "exists":
842
list, iss := scalarList(val, "ci.err.list")
843
issues = append(issues, iss...)
844
r.Exists = list
845
case "when":
846
w, iss := whenOf(val, true)
847
issues = append(issues, iss...)
848
r.When = w
849
default:
850
issues = append(issues, Issue{Line: item.Content[i].Line, Key: "ci.warn.ignored", Args: []any{"rules." + key}, Warning: true})
421
851
}
422
852
}
423
}
424
for _, d := range deps {
425
switch status[d.Name] {
426
case "success", "skipped":
853
if r.If == "" && len(r.Changes) == 0 && len(r.Exists) == 0 && r.When == "" {
854
issues = append(issues, Issue{Line: item.Line, Key: "ci.err.ruleEmpty"})
427
855
continue
428
case "failed":
429
if d.AllowFailure {
856
}
857
rules = append(rules, r)
858
}
859
return rules, issues
860
}
861
862
func parseNeeds(node *yaml.Node) (needs, optional []string, set bool, issues []Issue) {
863
set = true
864
if node.Kind != yaml.SequenceNode {
865
return nil, nil, true, []Issue{{Line: node.Line, Key: "ci.err.needs"}}
866
}
867
for _, item := range node.Content {
868
switch item.Kind {
869
case yaml.ScalarNode:
870
needs = append(needs, item.Value)
871
case yaml.MappingNode:
872
var name string
873
var opt bool
874
for i := 0; i < len(item.Content); i += 2 {
875
switch item.Content[i].Value {
876
case "job":
877
if item.Content[i+1].Kind == yaml.ScalarNode {
878
name = item.Content[i+1].Value
879
}
880
case "optional":
881
b, iss := parseBool(item.Content[i+1])
882
issues = append(issues, iss...)
883
opt = b
884
default:
885
issues = append(issues, Issue{Line: item.Content[i].Line, Key: "ci.warn.ignored", Args: []any{"needs." + item.Content[i].Value}, Warning: true})
886
}
887
}
888
if name == "" {
889
issues = append(issues, Issue{Line: item.Line, Key: "ci.err.needs"})
430
890
continue
431
891
}
432
return "skip"
892
needs = append(needs, name)
893
if opt {
894
optional = append(optional, name)
895
}
433
896
default:
434
return "wait"
897
issues = append(issues, Issue{Line: item.Line, Key: "ci.err.needs"})
435
898
}
436
899
}
437
return "ready"
900
if needs == nil {
901
needs = []string{}
902
}
903
return needs, optional, set, issues
438
904
}
439
905
440
var (
441
reEq = regexp.MustCompile(`^\$([A-Za-z_][A-Za-z0-9_]*)\s*==\s*"([^"]*)"$`)
442
reNe = regexp.MustCompile(`^\$([A-Za-z_][A-Za-z0-9_]*)\s*!=\s*"([^"]*)"$`)
443
reRe = regexp.MustCompile(`^\$([A-Za-z_][A-Za-z0-9_]*)\s*=~\s*/(.+)/$`)
444
)
445
446
type compiledRule struct {
447
kind string
448
name string
449
arg string
450
re *regexp.Regexp
906
func imageOf(node *yaml.Node) (string, []Issue) {
907
switch node.Kind {
908
case yaml.ScalarNode:
909
return node.Value, nil
910
case yaml.MappingNode:
911
for i := 0; i < len(node.Content); i += 2 {
912
if node.Content[i].Value == "name" && node.Content[i+1].Kind == yaml.ScalarNode {
913
return node.Content[i+1].Value, nil
914
}
915
}
916
return "", []Issue{{Line: node.Line, Key: "ci.err.image"}}
917
default:
918
return "", []Issue{{Line: node.Line, Key: "ci.err.image"}}
919
}
451
920
}
452
921
453
func compileRule(expr string) (compiledRule, error) {
454
if m := reEq.FindStringSubmatch(expr); m != nil {
455
return compiledRule{kind: "eq", name: m[1], arg: m[2]}, nil
456
}
457
if m := reNe.FindStringSubmatch(expr); m != nil {
458
return compiledRule{kind: "ne", name: m[1], arg: m[2]}, nil
922
func whenOf(node *yaml.Node, allowNever bool) (string, []Issue) {
923
if node.Kind != yaml.ScalarNode {
924
return "", []Issue{{Line: node.Line, Key: "ci.err.when", Args: []any{""}}}
459
925
}
460
if m := reRe.FindStringSubmatch(expr); m != nil {
461
re, err := regexp.Compile(m[2])
462
if err != nil {
463
return compiledRule{}, fmt.Errorf("ongeldige regex in rule %q: %w", expr, err)
926
switch node.Value {
927
case "on_success", "on_failure", "always", "manual":
928
return node.Value, nil
929
case "never":
930
if allowNever {
931
return node.Value, nil
464
932
}
465
return compiledRule{kind: "re", name: m[1], re: re}, nil
466
933
}
467
return compiledRule{}, fmt.Errorf("rule %q wordt niet begrepen", expr)
934
return "", []Issue{{Line: node.Line, Key: "ci.err.when", Args: []any{node.Value}}}
468
935
}
469
936
470
func evalRule(expr string, env map[string]string) (bool, error) {
471
c, err := compileRule(expr)
937
func durationOf(node *yaml.Node) (time.Duration, []Issue) {
938
if node.Kind != yaml.ScalarNode {
939
return 0, []Issue{{Line: node.Line, Key: "ci.err.timeout", Args: []any{""}}}
940
}
941
d, err := parseExpire(node.Value)
472
942
if err != nil {
473
return false, err
474
}
475
val := env[c.name]
476
switch c.kind {
477
case "eq":
478
return val == c.arg, nil
479
case "ne":
480
return val != c.arg, nil
481
case "re":
482
return c.re.MatchString(val), nil
483
default:
484
return false, fmt.Errorf("interne rulefout")
943
if strings.HasSuffix(node.Value, "s") {
944
n, nerr := strconv.Atoi(strings.TrimSuffix(node.Value, "s"))
945
if nerr == nil && n >= 0 {
946
return time.Duration(n) * time.Second, nil
947
}
948
}
949
return 0, []Issue{{Line: node.Line, Key: "ci.err.timeout", Args: []any{node.Value}}}
485
950
}
951
return d, nil
486
952
}
487
953
488
func parseExpire(s string) (time.Duration, error) {
489
s = strings.TrimSpace(s)
490
if s == "" || s == "never" {
491
return 0, nil
954
func extendsOf(node *yaml.Node) ([]string, []Issue) {
955
if node.Kind == yaml.ScalarNode && strings.TrimSpace(node.Value) != "" {
956
return []string{node.Value}, nil
492
957
}
493
if len(s) < 2 {
494
return 0, fmt.Errorf("expire_in %q is ongeldig", s)
495
}
496
n, err := strconv.Atoi(s[:len(s)-1])
497
if err != nil || n < 0 {
498
return 0, fmt.Errorf("expire_in %q is ongeldig", s)
958
return scalarList(node, "ci.err.extendsList")
959
}
960
961
func scriptList(node *yaml.Node) ([]string, []Issue) {
962
if node.Kind == yaml.ScalarNode {
963
if strings.TrimSpace(node.Value) == "" {
964
return nil, []Issue{{Line: node.Line, Key: "ci.err.script"}}
965
}
966
return splitLines(node.Value), nil
499
967
}
500
switch s[len(s)-1] {
501
case 'm':
502
return time.Duration(n) * time.Minute, nil
503
case 'h':
504
return time.Duration(n) * time.Hour, nil
505
case 'd':
506
return time.Duration(n) * 24 * time.Hour, nil
507
case 'w':
508
return time.Duration(n) * 7 * 24 * time.Hour, nil
509
default:
510
return 0, fmt.Errorf("expire_in %q is ongeldig", s)
968
return scalarList(node, "ci.err.script")
969
}
970
971
func splitLines(s string) []string {
972
parts := strings.Split(s, "\n")
973
var out []string
974
for _, p := range parts {
975
if strings.TrimSpace(p) == "" {
976
continue
977
}
978
out = append(out, p)
511
979
}
980
return out
512
981
}
513
982
514
func scalarList(node *yaml.Node) ([]string, error) {
983
func scalarList(node *yaml.Node, key string) ([]string, []Issue) {
515
984
if node.Kind != yaml.SequenceNode {
516
return nil, fmt.Errorf("verwacht een lijst")
985
return nil, []Issue{{Line: node.Line, Key: key}}
517
986
}
518
987
var out []string
988
var issues []Issue
519
989
for _, item := range node.Content {
520
990
if item.Kind != yaml.ScalarNode {
521
return nil, fmt.Errorf("lijst mag alleen tekst bevatten")
991
issues = append(issues, Issue{Line: item.Line, Key: "ci.err.listText"})
992
continue
522
993
}
523
994
out = append(out, item.Value)
524
995
}
525
return out, nil
996
return out, issues
526
997
}
527
998
528
func scalarMap(node *yaml.Node) (map[string]string, error) {
999
func scalarMap(node *yaml.Node) (map[string]string, []Issue) {
529
1000
if node.Kind != yaml.MappingNode {
530
return nil, fmt.Errorf("verwacht een mapping")
1001
return nil, []Issue{{Line: node.Line, Key: "ci.err.map"}}
531
1002
}
532
1003
out := map[string]string{}
1004
var issues []Issue
533
1005
for i := 0; i < len(node.Content); i += 2 {
534
1006
k := node.Content[i]
535
1007
v := node.Content[i+1]
536
if k.Kind != yaml.ScalarNode || v.Kind != yaml.ScalarNode {
537
return nil, fmt.Errorf("variabelen moeten tekst zijn")
1008
if k.Kind != yaml.ScalarNode {
1009
issues = append(issues, Issue{Line: k.Line, Key: "ci.err.varText"})
1010
continue
1011
}
1012
switch v.Kind {
1013
case yaml.ScalarNode:
1014
out[k.Value] = v.Value
1015
case yaml.MappingNode:
1016
val, iss := mappingValue(v)
1017
issues = append(issues, iss...)
1018
if val != "" || len(iss) == 0 {
1019
out[k.Value] = val
1020
}
1021
default:
1022
issues = append(issues, Issue{Line: v.Line, Key: "ci.err.varText"})
1023
}
1024
}
1025
return out, issues
1026
}
1027
1028
func mappingValue(node *yaml.Node) (string, []Issue) {
1029
for i := 0; i < len(node.Content); i += 2 {
1030
if node.Content[i].Value == "value" && node.Content[i+1].Kind == yaml.ScalarNode {
1031
return node.Content[i+1].Value, nil
538
1032
}
539
out[k.Value] = v.Value
540
1033
}
541
return out, nil
1034
return "", []Issue{{Line: node.Line, Key: "ci.err.varText"}}
542
1035
}
543
1036
544
func parseBool(node *yaml.Node) (bool, error) {
1037
func parseBool(node *yaml.Node) (bool, []Issue) {
545
1038
if node.Kind != yaml.ScalarNode {
546
return false, fmt.Errorf("verwacht ja of nee")
1039
return false, []Issue{{Line: node.Line, Key: "ci.err.bool"}}
547
1040
}
548
1041
switch strings.ToLower(node.Value) {
549
case "true", "yes", "ja":
1042
case "true", "yes":
550
1043
return true, nil
551
case "false", "no", "nee":
1044
case "false", "no":
552
1045
return false, nil
553
1046
default:
554
return false, fmt.Errorf("verwacht true of false")
1047
return false, []Issue{{Line: node.Line, Key: "ci.err.bool"}}
555
1048
}
556
1049
}
557
1050
558
func safeRelPath(p string) error {
1051
func parseExpire(s string) (time.Duration, error) {
1052
s = strings.TrimSpace(s)
1053
if s == "" || s == "never" {
1054
return 0, nil
1055
}
1056
if len(s) < 2 {
1057
return 0, fmt.Errorf("expire")
1058
}
1059
n, err := strconv.Atoi(s[:len(s)-1])
1060
if err != nil || n < 0 {
1061
return 0, fmt.Errorf("expire")
1062
}
1063
switch s[len(s)-1] {
1064
case 's':
1065
return time.Duration(n) * time.Second, nil
1066
case 'm':
1067
return time.Duration(n) * time.Minute, nil
1068
case 'h':
1069
return time.Duration(n) * time.Hour, nil
1070
case 'd':
1071
return time.Duration(n) * 24 * time.Hour, nil
1072
case 'w':
1073
return time.Duration(n) * 7 * 24 * time.Hour, nil
1074
default:
1075
return 0, fmt.Errorf("expire")
1076
}
1077
}
1078
1079
func safeRel(p string) Issue {
559
1080
if p == "" || strings.HasPrefix(p, "/") || strings.Contains(p, "\\") {
560
return fmt.Errorf("pad %q moet relatief zijn", p)
1081
return Issue{Line: 1, Key: "ci.err.rel", Args: []any{p}}
561
1082
}
562
1083
for _, part := range strings.Split(p, "/") {
563
1084
if part == ".." {
564
return fmt.Errorf("pad %q mag niet buiten de workspace", p)
1085
return Issue{Line: 1, Key: "ci.err.dotdot", Args: []any{p}}
1086
}
1087
}
1088
return Issue{}
1089
}
1090
1091
func hasError(issues []Issue) bool {
1092
for _, is := range issues {
1093
if !is.Warning {
1094
return true
565
1095
}
566
1096
}
567
return nil
1097
return false
568
1098
}
@@ -1,327 +1,712 @@1
// Package ci leest blue-ci-cd.yml en bepaalt welke jobs draaien.1
// Package ci leest blue-ci-cd.yml en bepaalt welke jobs draaien.2
// De vorm ligt dicht bij GitLab CI: stages, jobs, needs, rules, artifacts.2
// De vorm volgt GitLab CI waar dat in BlueGit past. Een oud bestand met3
// een jobs:-map blijft geldig. Jobs mogen ook los op het hoogste niveau staan.3
package ci4
package ci4
5
5
import (6
import (6
"fmt"7
"fmt"7
"regexp"8
"strconv"8
"strconv"9
"strings"9
"strings"10
"time"10
"time"11
11
12
"gitlab.com/Timfa/bluegit/internal/i18n"12
"gopkg.in/yaml.v3"13
"gopkg.in/yaml.v3"13
)14
)14
15
15
// Pipeline is een geparset configuratiebestand.16
// Parse leest YAML. Jobvolgorde blijft de volgorde uit het bestand.16
type Pipeline struct {17
func Parse(data []byte) (Pipeline, error) {17
Stages []string18
return ParseIn(data, nil)18
Variables map[string]string19
}19
Image string20
20
Jobs []Job21
// ParseIn leest YAML en haalt include: local via read op.22
func ParseIn(data []byte, read func(string) ([]byte, error)) (Pipeline, error) {23
p, issues, err := parseBytes(data, read, nil)24
if err != nil {25
return Pipeline{}, err26
}27
for _, is := range issues {28
if !is.Warning {29
return Pipeline{}, is30
}31
}32
p.Warnings = warningsOnly(issues)33
return p, nil21
}34
}22
35
23
// Job is één containerstap.36
// Lint geeft elke fout en waarschuwing, met regelnummer, zonder te stoppen bij de eerste.24
type Job struct {37
func Lint(data []byte, read func(string) ([]byte, error)) (Pipeline, []Issue) {25
Name string38
p, issues, err := parseBytes(data, read, nil)26
Stage string39
if err != nil {27
Image string40
issues = append([]Issue{{Line: 1, Key: "ci.err.yaml", Args: []any{err.Error()}}}, issues...)28
Script []string41
}29
Needs []string42
return p, issues30
NeedsSet bool43
}31
Variables map[string]string44
32
Artifacts Artifacts45
func (i Issue) Error() string { return i18n.T(i18n.EN, i.Key, i.Args...) }33
Rules []Rule46
34
AllowFailure bool47
func warningsOnly(in []Issue) []Issue {48
var out []Issue49
for _, is := range in {50
if is.Warning {51
out = append(out, is)52
}53
}54
return out35
}55
}36
56
37
// Artifacts zijn paden die na een geslaagde job bewaard blijven.57
var reserved = map[string]bool{38
type Artifacts struct {58
"stages": true, "variables": true, "image": true, "default": true,39
Paths []string59
"include": true, "before_script": true, "after_script": true, "cache": true,40
ExpireIn time.Duration60
"workflow": true, "services": true, "spec": true, "pages": true, "jobs": true,41
}61
}42
62
43
// Rule is één voorwaarde. De job draait als minstens één rule klopt.63
type draft struct {44
// Zonder rules draait de job altijd.64
job Job45
type Rule struct {65
present map[string]bool46
If string66
extends []string67
line int47
}68
}48
69
49
// Parse leest YAML. Jobvolgorde blijft de volgorde uit het bestand.70
func parseBytes(data []byte, read func(string) ([]byte, error), stack []string) (Pipeline, []Issue, error) {50
func Parse(data []byte) (Pipeline, error) {51
var root yaml.Node71
var root yaml.Node52
if err := yaml.Unmarshal(data, &root); err != nil {72
if err := yaml.Unmarshal(data, &root); err != nil {53
return Pipeline{}, fmt.Errorf("yaml: %w", err)73
return Pipeline{}, nil, i18n.E("ci.err.yaml", err.Error())54
}74
}55
if root.Kind != yaml.DocumentNode || len(root.Content) != 1 || root.Content[0].Kind != yaml.MappingNode {75
if root.Kind != yaml.DocumentNode || len(root.Content) != 1 || root.Content[0].Kind != yaml.MappingNode {56
return Pipeline{}, fmt.Errorf("blue-ci-cd.yml moet een mapping zijn")76
return Pipeline{}, []Issue{{Line: 1, Key: "ci.err.mapping"}}, nil57
}77
}58
body := root.Content[0]78
body := root.Content[0]59
var p Pipeline79
var included Pipeline60
p.Variables = map[string]string{}80
var issues []Issue81
for i := 0; i < len(body.Content); i += 2 {82
if body.Content[i].Value != "include" {83
continue84
}85
part, more, err := parseInclude(body.Content[i+1], read, stack)86
if err != nil {87
return Pipeline{}, nil, err88
}89
issues = append(issues, more...)90
included = mergePipeline(included, part)91
}92
p := included93
if p.Variables == nil {94
p.Variables = map[string]string{}95
}96
var def draft97
def.present = map[string]bool{}98
var jobs []draft99
seenLegacy := false100
seenTop := false61
for i := 0; i < len(body.Content); i += 2 {101
for i := 0; i < len(body.Content); i += 2 {62
key := body.Content[i].Value102
keyNode := body.Content[i]103
key := keyNode.Value63
val := body.Content[i+1]104
val := body.Content[i+1]64
switch key {105
switch key {106
case "include":107
continue65
case "stages":108
case "stages":66
stages, err := scalarList(val)109
stages, iss := scalarList(val, "ci.err.list")67
if err != nil {110
issues = append(issues, iss...)68
return Pipeline{}, fmt.Errorf("stages: %w", err)111
if len(stages) > 0 {112
p.Stages = stages69
}113
}70
p.Stages = stages71
case "variables":114
case "variables":72
vars, err := scalarMap(val)115
vars, iss := scalarMap(val)73
if err != nil {116
issues = append(issues, iss...)74
return Pipeline{}, fmt.Errorf("variables: %w", err)117
for k, v := range vars {118
p.Variables[k] = v75
}119
}76
p.Variables = vars77
case "image":120
case "image":78
if val.Kind != yaml.ScalarNode {121
img, iss := imageOf(val)79
return Pipeline{}, fmt.Errorf("image moet een tekst zijn")122
issues = append(issues, iss...)123
if img != "" {124
p.Image = img80
}125
}81
p.Image = val.Value126
case "before_script":127
lines, iss := scriptList(val)128
issues = append(issues, iss...)129
p.Before = lines130
def.present["before_script"] = true131
def.job.Before = lines132
case "after_script":133
lines, iss := scriptList(val)134
issues = append(issues, iss...)135
p.After = lines136
def.present["after_script"] = true137
def.job.After = lines138
case "cache":139
c, iss := parseCache(val)140
issues = append(issues, iss...)141
p.Cache = c142
def.present["cache"] = true143
def.job.Cache = c144
case "default":145
d, iss := parseDefault(val)146
issues = append(issues, iss...)147
def = overlayDefault(def, d)148
case "workflow", "services", "spec", "pages":149
issues = append(issues, Issue{Line: keyNode.Line, Key: "ci.warn.ignored", Args: []any{key}, Warning: true})82
case "jobs":150
case "jobs":83
jobs, err := parseJobs(val)151
seenLegacy = true84
if err != nil {152
parsed, iss := parseJobMap(val)85
return Pipeline{}, err153
issues = append(issues, iss...)86
}154
jobs = append(jobs, parsed...)87
p.Jobs = jobs88
default:155
default:89
return Pipeline{}, fmt.Errorf("onbekend veld %q", key)156
if val.Kind != yaml.MappingNode {157
issues = append(issues, Issue{Line: keyNode.Line, Key: "ci.err.unknown", Args: []any{key}})158
continue159
}160
seenTop = true161
job, iss := parseJob(key, val)162
issues = append(issues, iss...)163
jobs = append(jobs, job)90
}164
}91
}165
}92
if err := p.validate(); err != nil {166
if seenLegacy && seenTop {93
return Pipeline{}, err167
issues = append(issues, Issue{Line: 1, Key: "ci.err.both"})168
}169
if p.Image != "" && !def.present["image"] {170
def.present["image"] = true171
def.job.Image = p.Image172
}173
resolved, iss := applyExtends(jobs, def)174
issues = append(issues, iss...)175
byName := map[string]int{}176
var merged []Job177
for _, j := range p.Jobs {178
byName[j.Name] = len(merged)179
merged = append(merged, j)180
}181
for _, d := range resolved {182
if strings.HasPrefix(d.job.Name, ".") {183
continue184
}185
if i, ok := byName[d.job.Name]; ok {186
merged[i] = overlayJob(merged[i], d)187
continue188
}189
byName[d.job.Name] = len(merged)190
merged = append(merged, d.job)191
}192
p.Jobs = merged193
issues = append(issues, validate(&p)...)194
if hasError(issues) {195
return p, issues, nil196
}197
return p, issues, nil198
}199
200
func parseInclude(node *yaml.Node, read func(string) ([]byte, error), stack []string) (Pipeline, []Issue, error) {201
var items []*yaml.Node202
switch node.Kind {203
case yaml.SequenceNode:204
items = node.Content205
default:206
items = []*yaml.Node{node}207
}208
var out Pipeline209
var issues []Issue210
for _, item := range items {211
local, iss := includeLocal(item)212
issues = append(issues, iss...)213
if local == "" {214
continue215
}216
if read == nil {217
issues = append(issues, Issue{Line: item.Line, Key: "ci.err.includeRead", Args: []any{local}})218
continue219
}220
for _, prev := range stack {221
if prev == local {222
issues = append(issues, Issue{Line: item.Line, Key: "ci.err.includeCycle", Args: []any{local}})223
local = ""224
break225
}226
}227
if local == "" {228
continue229
}230
raw, err := read(local)231
if err != nil {232
issues = append(issues, Issue{Line: item.Line, Key: "ci.err.includeRead", Args: []any{local}})233
continue234
}235
part, more, err := parseBytes(raw, read, append(append([]string{}, stack...), local))236
if err != nil {237
return Pipeline{}, nil, err238
}239
issues = append(issues, more...)240
out = mergePipeline(out, part)241
}242
return out, issues, nil243
}244
245
func includeLocal(node *yaml.Node) (string, []Issue) {246
switch node.Kind {247
case yaml.ScalarNode:248
return cleanLocal(node.Value, node)249
case yaml.MappingNode:250
for i := 0; i < len(node.Content); i += 2 {251
key := node.Content[i].Value252
if key == "local" {253
if node.Content[i+1].Kind != yaml.ScalarNode {254
return "", []Issue{{Line: node.Line, Key: "ci.err.includeLocal"}}255
}256
return cleanLocal(node.Content[i+1].Value, node.Content[i+1])257
}258
}259
return "", []Issue{{Line: node.Line, Key: "ci.err.includeRemote"}}260
default:261
return "", []Issue{{Line: node.Line, Key: "ci.err.includeLocal"}}262
}263
}264
265
func cleanLocal(p string, node *yaml.Node) (string, []Issue) {266
p = strings.TrimPrefix(strings.TrimSpace(p), "/")267
if p == "" || strings.Contains(p, "\\") || strings.Contains(p, "..") {268
return "", []Issue{{Line: node.Line, Key: "ci.err.rel", Args: []any{p}}}94
}269
}95
return p, nil270
return p, nil96
}271
}97
272
98
func parseJobs(node *yaml.Node) ([]Job, error) {273
func mergePipeline(base, extra Pipeline) Pipeline {274
if len(extra.Stages) > 0 {275
base.Stages = extra.Stages276
}277
if base.Variables == nil {278
base.Variables = map[string]string{}279
}280
for k, v := range extra.Variables {281
base.Variables[k] = v282
}283
if extra.Image != "" {284
base.Image = extra.Image285
}286
if extra.Before != nil {287
base.Before = extra.Before288
}289
if extra.After != nil {290
base.After = extra.After291
}292
byName := map[string]int{}293
for i, j := range base.Jobs {294
byName[j.Name] = i295
}296
for _, j := range extra.Jobs {297
if i, ok := byName[j.Name]; ok {298
base.Jobs[i] = j299
continue300
}301
byName[j.Name] = len(base.Jobs)302
base.Jobs = append(base.Jobs, j)303
}304
return base305
}306
307
func parseDefault(node *yaml.Node) (draft, []Issue) {308
d := draft{present: map[string]bool{}}99
if node.Kind != yaml.MappingNode {309
if node.Kind != yaml.MappingNode {100
return nil, fmt.Errorf("jobs moet een mapping zijn")310
return d, []Issue{{Line: node.Line, Key: "ci.err.map"}}101
}311
}102
var jobs []Job312
var issues []Issue103
for i := 0; i < len(node.Content); i += 2 {313
for i := 0; i < len(node.Content); i += 2 {104
name := node.Content[i].Value314
key := node.Content[i].Value105
body := node.Content[i+1]315
val := node.Content[i+1]106
job, err := parseJob(name, body)316
switch key {107
if err != nil {317
case "image":108
return nil, fmt.Errorf("job %s: %w", name, err)318
img, iss := imageOf(val)319
issues = append(issues, iss...)320
d.job.Image = img321
d.present["image"] = true322
case "before_script":323
lines, iss := scriptList(val)324
issues = append(issues, iss...)325
d.job.Before = lines326
d.present["before_script"] = true327
case "after_script":328
lines, iss := scriptList(val)329
issues = append(issues, iss...)330
d.job.After = lines331
d.present["after_script"] = true332
case "tags":333
tags, iss := scalarList(val, "ci.err.list")334
issues = append(issues, iss...)335
d.job.Tags = tags336
d.present["tags"] = true337
case "timeout":338
dur, iss := durationOf(val)339
issues = append(issues, iss...)340
d.job.Timeout = dur341
d.present["timeout"] = true342
case "cache":343
c, iss := parseCache(val)344
issues = append(issues, iss...)345
d.job.Cache = c346
d.present["cache"] = true347
default:348
issues = append(issues, Issue{Line: node.Content[i].Line, Key: "ci.warn.ignored", Args: []any{"default." + key}, Warning: true})109
}349
}110
jobs = append(jobs, job)111
}350
}112
return jobs, nil351
return d, issues113
}352
}114
353
115
func parseJob(name string, node *yaml.Node) (Job, error) {354
func overlayJob(base Job, d draft) Job {355
set := func(key string, apply func()) {356
if d.present[key] {357
apply()358
}359
}360
set("stage", func() { base.Stage = d.job.Stage })361
set("image", func() { base.Image = d.job.Image })362
set("before_script", func() { base.Before = d.job.Before })363
set("script", func() { base.Script = d.job.Script })364
set("after_script", func() { base.After = d.job.After })365
set("needs", func() {366
base.Needs = d.job.Needs367
base.Optional = d.job.Optional368
base.NeedsSet = d.job.NeedsSet369
})370
set("artifacts", func() { base.Artifacts = d.job.Artifacts })371
set("rules", func() { base.Rules = d.job.Rules })372
set("allow_failure", func() { base.AllowFailure = d.job.AllowFailure })373
set("when", func() { base.When = d.job.When })374
set("timeout", func() { base.Timeout = d.job.Timeout })375
set("tags", func() { base.Tags = d.job.Tags })376
set("cache", func() { base.Cache = d.job.Cache })377
set("resource_group", func() { base.ResourceGroup = d.job.ResourceGroup })378
set("only", func() { base.Only = d.job.Only })379
set("except", func() { base.Except = d.job.Except })380
if d.present["variables"] {381
mergedVars := map[string]string{}382
for k, v := range base.Variables {383
mergedVars[k] = v384
}385
for k, v := range d.job.Variables {386
mergedVars[k] = v387
}388
base.Variables = mergedVars389
}390
return base391
}392
393
func overlayDefault(base, extra draft) draft {394
if base.present == nil {395
base.present = map[string]bool{}396
}397
for k, on := range extra.present {398
if !on {399
continue400
}401
base.present[k] = true402
switch k {403
case "image":404
base.job.Image = extra.job.Image405
case "before_script":406
base.job.Before = extra.job.Before407
case "after_script":408
base.job.After = extra.job.After409
case "tags":410
base.job.Tags = extra.job.Tags411
case "timeout":412
base.job.Timeout = extra.job.Timeout413
case "cache":414
base.job.Cache = extra.job.Cache415
}416
}417
return base418
}419
420
func parseJobMap(node *yaml.Node) ([]draft, []Issue) {116
if node.Kind != yaml.MappingNode {421
if node.Kind != yaml.MappingNode {117
return Job{}, fmt.Errorf("moet een mapping zijn")422
return nil, []Issue{{Line: node.Line, Key: "ci.err.jobsMap"}}118
}423
}119
j := Job{Name: name, Variables: map[string]string{}}424
var jobs []draft425
var issues []Issue426
for i := 0; i < len(node.Content); i += 2 {427
job, iss := parseJob(node.Content[i].Value, node.Content[i+1])428
issues = append(issues, iss...)429
if job.job.Name != "" {430
jobs = append(jobs, job)431
}432
}433
return jobs, issues434
}435
436
func parseJob(name string, node *yaml.Node) (draft, []Issue) {437
d := draft{present: map[string]bool{}, line: node.Line, job: Job{Name: name, Variables: map[string]string{}}}438
if node.Kind != yaml.MappingNode {439
return d, []Issue{{Line: node.Line, Key: "ci.err.jobMap", Args: []any{name}}}440
}441
var issues []Issue120
for i := 0; i < len(node.Content); i += 2 {442
for i := 0; i < len(node.Content); i += 2 {121
key := node.Content[i].Value443
key := node.Content[i].Value122
val := node.Content[i+1]444
val := node.Content[i+1]445
d.present[key] = true123
switch key {446
switch key {124
case "stage":447
case "stage":125
if val.Kind != yaml.ScalarNode {448
if val.Kind != yaml.ScalarNode {126
return Job{}, fmt.Errorf("stage moet tekst zijn")449
issues = append(issues, Issue{Line: val.Line, Key: "ci.err.stageText"})450
continue127
}451
}128
j.Stage = val.Value452
d.job.Stage = val.Value129
case "image":453
case "image":130
if val.Kind != yaml.ScalarNode {454
img, iss := imageOf(val)131
return Job{}, fmt.Errorf("image moet tekst zijn")455
issues = append(issues, iss...)132
}456
d.job.Image = img133
j.Image = val.Value457
case "before_script":458
lines, iss := scriptList(val)459
issues = append(issues, iss...)460
d.job.Before = lines134
case "script":461
case "script":135
lines, err := scalarList(val)462
lines, iss := scriptList(val)136
if err != nil {463
issues = append(issues, iss...)137
return Job{}, fmt.Errorf("script: %w", err)464
d.job.Script = lines138
}465
case "after_script":139
j.Script = lines466
lines, iss := scriptList(val)467
issues = append(issues, iss...)468
d.job.After = lines140
case "needs":469
case "needs":141
needs, err := scalarList(val)470
needs, optional, set, iss := parseNeeds(val)142
if err != nil {471
issues = append(issues, iss...)143
return Job{}, fmt.Errorf("needs: %w", err)472
d.job.Needs = needs144
}473
d.job.Optional = optional145
j.Needs = needs474
d.job.NeedsSet = set146
j.NeedsSet = true147
case "variables":475
case "variables":148
vars, err := scalarMap(val)476
vars, iss := scalarMap(val)149
if err != nil {477
issues = append(issues, iss...)150
return Job{}, fmt.Errorf("variables: %w", err)478
d.job.Variables = vars151
}152
j.Variables = vars153
case "artifacts":479
case "artifacts":154
art, err := parseArtifacts(val)480
art, iss := parseArtifacts(val)155
if err != nil {481
issues = append(issues, iss...)156
return Job{}, err482
d.job.Artifacts = art157
}158
j.Artifacts = art159
case "rules":483
case "rules":160
rules, err := parseRules(val)484
rules, iss := parseRules(val)161
if err != nil {485
issues = append(issues, iss...)162
return Job{}, err486
d.job.Rules = rules163
}164
j.Rules = rules165
case "allow_failure":487
case "allow_failure":166
b, err := parseBool(val)488
b, iss := parseBool(val)167
if err != nil {489
issues = append(issues, iss...)168
return Job{}, fmt.Errorf("allow_failure: %w", err)490
d.job.AllowFailure = b491
case "when":492
w, iss := whenOf(val, false)493
issues = append(issues, iss...)494
d.job.When = w495
case "timeout":496
dur, iss := durationOf(val)497
issues = append(issues, iss...)498
d.job.Timeout = dur499
case "tags":500
tags, iss := scalarList(val, "ci.err.list")501
issues = append(issues, iss...)502
d.job.Tags = tags503
case "cache":504
c, iss := parseCache(val)505
issues = append(issues, iss...)506
d.job.Cache = c507
case "resource_group":508
if val.Kind != yaml.ScalarNode || strings.TrimSpace(val.Value) == "" {509
issues = append(issues, Issue{Line: val.Line, Key: "ci.err.resource"})510
continue169
}511
}170
j.AllowFailure = b512
d.job.ResourceGroup = strings.TrimSpace(val.Value)513
case "only":514
list, iss := scalarList(val, "ci.err.list")515
issues = append(issues, iss...)516
d.job.Only = list517
case "except":518
list, iss := scalarList(val, "ci.err.list")519
issues = append(issues, iss...)520
d.job.Except = list521
case "extends":522
list, iss := extendsOf(val)523
issues = append(issues, iss...)524
d.extends = list171
default:525
default:172
return Job{}, fmt.Errorf("onbekend veld %q", key)526
if _, ok := reserved[key]; ok {527
issues = append(issues, Issue{Line: node.Content[i].Line, Key: "ci.warn.ignored", Args: []any{key}, Warning: true})528
continue529
}530
issues = append(issues, Issue{Line: node.Content[i].Line, Key: "ci.warn.ignored", Args: []any{name + "." + key}, Warning: true})173
}531
}174
}532
}175
return j, nil533
return d, issues176
}534
}177
535
178
func parseArtifacts(node *yaml.Node) (Artifacts, error) {536
func applyExtends(jobs []draft, def draft) ([]draft, []Issue) {179
if node.Kind != yaml.MappingNode {537
by := map[string]*draft{}180
return Artifacts{}, fmt.Errorf("artifacts moet een mapping zijn")538
var order []string539
var issues []Issue540
for i := range jobs {541
name := jobs[i].job.Name542
if by[name] != nil {543
issues = append(issues, Issue{Line: jobs[i].line, Key: "ci.err.jobDup", Args: []any{name}})544
continue545
}546
by[name] = &jobs[i]547
order = append(order, name)181
}548
}182
var a Artifacts549
visiting := map[string]bool{}183
for i := 0; i < len(node.Content); i += 2 {550
var walk func(string)184
key := node.Content[i].Value551
walk = func(name string) {185
val := node.Content[i+1]552
d := by[name]186
switch key {553
if d == nil || d.present["__resolved"] {187
case "paths":554
return188
paths, err := scalarList(val)555
}189
if err != nil {556
if visiting[name] {190
return Artifacts{}, fmt.Errorf("artifacts.paths: %w", err)557
issues = append(issues, Issue{Line: d.line, Key: "ci.err.extendsCycle", Args: []any{name}})191
}558
return192
a.Paths = paths559
}193
case "expire_in":560
visiting[name] = true194
if val.Kind != yaml.ScalarNode {561
for _, parent := range d.extends {195
return Artifacts{}, fmt.Errorf("expire_in moet tekst zijn")562
base := by[parent]196
}563
if base == nil {197
d, err := parseExpire(val.Value)564
issues = append(issues, Issue{Line: d.line, Key: "ci.err.extends", Args: []any{name, parent}})198
if err != nil {565
continue199
return Artifacts{}, err200
}566
}201
a.ExpireIn = d567
walk(parent)202
default:568
inherit(d, base)203
return Artifacts{}, fmt.Errorf("onbekend artifacts-veld %q", key)204
}569
}570
delete(visiting, name)571
d.present["__resolved"] = true572
inherit(d, &def)573
}574
for _, name := range order {575
walk(name)205
}576
}206
return a, nil577
return jobs, issues207
}578
}208
579
209
func parseRules(node *yaml.Node) ([]Rule, error) {580
func inherit(child, parent *draft) {210
if node.Kind != yaml.SequenceNode {581
if parent == nil {211
return nil, fmt.Errorf("rules moet een lijst zijn")582
return212
}583
}213
var rules []Rule584
copyField := func(key string, apply func()) {214
for _, item := range node.Content {585
if child.present[key] || !parent.present[key] {215
if item.Kind != yaml.MappingNode {586
return216
return nil, fmt.Errorf("rule moet een mapping zijn")217
}587
}218
var r Rule588
apply()219
for i := 0; i < len(item.Content); i += 2 {589
child.present[key] = true220
key := item.Content[i].Value590
}221
val := item.Content[i+1]591
copyField("stage", func() { child.job.Stage = parent.job.Stage })222
if key != "if" {592
copyField("image", func() { child.job.Image = parent.job.Image })223
return nil, fmt.Errorf("rule ondersteunt alleen if, kreeg %q", key)593
copyField("before_script", func() { child.job.Before = parent.job.Before })224
}594
copyField("script", func() { child.job.Script = parent.job.Script })225
if val.Kind != yaml.ScalarNode || strings.TrimSpace(val.Value) == "" {595
copyField("after_script", func() { child.job.After = parent.job.After })226
return nil, fmt.Errorf("rule.if moet tekst zijn")596
copyField("needs", func() {227
}597
child.job.Needs = parent.job.Needs228
r.If = strings.TrimSpace(val.Value)598
child.job.Optional = parent.job.Optional229
}599
child.job.NeedsSet = parent.job.NeedsSet230
if r.If == "" {600
})231
return nil, fmt.Errorf("rule mist if")601
copyField("artifacts", func() { child.job.Artifacts = parent.job.Artifacts })602
copyField("rules", func() { child.job.Rules = parent.job.Rules })603
copyField("allow_failure", func() { child.job.AllowFailure = parent.job.AllowFailure })604
copyField("when", func() { child.job.When = parent.job.When })605
copyField("timeout", func() { child.job.Timeout = parent.job.Timeout })606
copyField("tags", func() { child.job.Tags = parent.job.Tags })607
copyField("cache", func() { child.job.Cache = parent.job.Cache })608
copyField("resource_group", func() { child.job.ResourceGroup = parent.job.ResourceGroup })609
copyField("only", func() { child.job.Only = parent.job.Only })610
copyField("except", func() { child.job.Except = parent.job.Except })611
if len(parent.job.Variables) > 0 {612
merged := map[string]string{}613
for k, v := range parent.job.Variables {614
merged[k] = v232
}615
}233
if _, err := compileRule(r.If); err != nil {616
for k, v := range child.job.Variables {234
return nil, err617
merged[k] = v235
}618
}236
rules = append(rules, r)619
child.job.Variables = merged237
}620
}238
return rules, nil239
}621
}240
622
241
func (p *Pipeline) validate() error {623
func validate(p *Pipeline) []Issue {624
var issues []Issue242
if len(p.Jobs) == 0 {625
if len(p.Jobs) == 0 {243
return fmt.Errorf("geen jobs")626
return []Issue{{Line: 1, Key: "ci.err.noJobs"}}244
}627
}245
seenStage := map[string]bool{}628
seenStage := map[string]bool{}246
for _, s := range p.Stages {629
for _, s := range p.Stages {247
if s == "" {630
if s == "" {248
return fmt.Errorf("lege stage-naam")631
issues = append(issues, Issue{Line: 1, Key: "ci.err.stageEmpty"})632
continue249
}633
}250
if seenStage[s] {634
if seenStage[s] {251
return fmt.Errorf("dubbele stage %q", s)635
issues = append(issues, Issue{Line: 1, Key: "ci.err.stageDup", Args: []any{s}})252
}636
}253
seenStage[s] = true637
seenStage[s] = true254
}638
}255
if len(p.Stages) == 0 {639
if len(p.Stages) == 0 {256
var order []string640
var order []string257
have := map[string]bool{}258
for i := range p.Jobs {641
for i := range p.Jobs {259
if p.Jobs[i].Stage == "" {642
if p.Jobs[i].Stage == "" {260
p.Jobs[i].Stage = "test"643
p.Jobs[i].Stage = "test"261
}644
}262
if !have[p.Jobs[i].Stage] {645
if !seenStage[p.Jobs[i].Stage] {263
have[p.Jobs[i].Stage] = true646
seenStage[p.Jobs[i].Stage] = true264
order = append(order, p.Jobs[i].Stage)647
order = append(order, p.Jobs[i].Stage)265
}648
}266
}649
}267
p.Stages = order650
p.Stages = order268
for _, s := range p.Stages {269
seenStage[s] = true270
}271
}651
}272
names := map[string]bool{}652
names := map[string]bool{}273
for i := range p.Jobs {653
for i := range p.Jobs {274
j := &p.Jobs[i]654
j := &p.Jobs[i]275
if j.Name == "" || strings.ContainsAny(j.Name, " \t\n/\\") {655
if j.Name == "" || strings.ContainsAny(j.Name, " \t\n/\\") {276
return fmt.Errorf("ongeldige jobnaam %q", j.Name)656
issues = append(issues, Issue{Line: 1, Key: "ci.err.jobName", Args: []any{j.Name}})277
}657
continue278
if names[j.Name] {279
return fmt.Errorf("dubbele job %q", j.Name)280
}658
}281
names[j.Name] = true659
names[j.Name] = true282
if j.Stage == "" {660
if j.Stage == "" {283
return fmt.Errorf("job %s mist stage", j.Name)661
issues = append(issues, Issue{Line: 1, Key: "ci.err.jobStage", Args: []any{j.Name}})284
}662
} else if !seenStage[j.Stage] {285
if !seenStage[j.Stage] {663
issues = append(issues, Issue{Line: 1, Key: "ci.err.stageUnknown", Args: []any{j.Name, j.Stage}})286
return fmt.Errorf("job %s gebruikt onbekende stage %q", j.Name, j.Stage)287
}664
}288
if j.Image == "" {665
if j.Image == "" {289
j.Image = p.Image666
j.Image = p.Image290
}667
}291
if strings.TrimSpace(j.Image) == "" || strings.ContainsAny(j.Image, " \t\n") {668
if strings.TrimSpace(j.Image) == "" || strings.ContainsAny(j.Image, " \t\n") {292
return fmt.Errorf("job %s mist een image", j.Name)669
issues = append(issues, Issue{Line: 1, Key: "ci.err.noImage", Args: []any{j.Name}})293
}670
}294
if len(j.Script) == 0 {671
if len(j.Script) == 0 {295
return fmt.Errorf("job %s mist script", j.Name)672
issues = append(issues, Issue{Line: 1, Key: "ci.err.noScript", Args: []any{j.Name}})296
}673
}297
for _, line := range j.Script {674
for _, line := range j.Script {298
if strings.TrimSpace(line) == "" {675
if strings.TrimSpace(line) == "" {299
return fmt.Errorf("job %s heeft een lege scriptregel", j.Name)676
issues = append(issues, Issue{Line: 1, Key: "ci.err.emptyLine", Args: []any{j.Name}})300
}677
}301
}678
}302
for _, path := range j.Artifacts.Paths {679
for _, path := range j.Artifacts.Paths {303
if err := safeRelPath(path); err != nil {680
if iss := safeRel(path); iss.Key != "" {304
return fmt.Errorf("job %s artifact: %w", j.Name, err)681
issues = append(issues, iss)305
}682
}306
}683
}684
for _, path := range j.Cache.Paths {685
if iss := safeRel(path); iss.Key != "" {686
issues = append(issues, iss)687
}688
}689
if j.When == "" {690
j.When = "on_success"691
}307
}692
}308
for _, j := range p.Jobs {693
for _, j := range p.Jobs {309
for _, need := range j.Needs {694
for _, need := range j.Needs {310
if !names[need] {695
if !names[need] && !contains(j.Optional, need) {311
return fmt.Errorf("job %s needs onbekende job %q", j.Name, need)696
issues = append(issues, Issue{Line: 1, Key: "ci.err.needUnknown", Args: []any{j.Name, need}})312
}697
}313
if need == j.Name {698
if need == j.Name {314
return fmt.Errorf("job %s kan niet van zichzelf afhangen", j.Name)699
issues = append(issues, Issue{Line: 1, Key: "ci.err.needSelf", Args: []any{j.Name}})315
}700
}316
}701
}317
}702
}318
if err := detectCycles(p.Jobs); err != nil {703
if iss := detectCycles(p.Jobs); iss.Key != "" {319
return err704
issues = append(issues, iss)320
}705
}321
return nil706
return issues322
}707
}323
708
324
func detectCycles(jobs []Job) error {709
func detectCycles(jobs []Job) Issue {325
idx := map[string]int{}710
idx := map[string]int{}326
for i, j := range jobs {711
for i, j := range jobs {327
idx[j.Name] = i712
idx[j.Name] = i@@ -332,237 +717,382 @@ func detectCycles(jobs []Job) error {332
black = 2717
black = 2333
)718
)334
color := make([]int, len(jobs))719
color := make([]int, len(jobs))335
var visit func(int) error720
var visit func(int) Issue336
visit = func(i int) error {721
visit = func(i int) Issue {337
if color[i] == gray {722
if color[i] == gray {338
return fmt.Errorf("cyclus in needs bij job %s", jobs[i].Name)723
return Issue{Line: 1, Key: "ci.err.cycle", Args: []any{jobs[i].Name}}339
}724
}340
if color[i] == black {725
if color[i] == black {341
return nil726
return Issue{}342
}727
}343
color[i] = gray728
color[i] = gray344
for _, n := range jobs[i].Needs {729
for _, n := range jobs[i].Needs {345
if err := visit(idx[n]); err != nil {730
target, ok := idx[n]346
return err731
if !ok {732
continue733
}734
if iss := visit(target); iss.Key != "" {735
return iss347
}736
}348
}737
}349
color[i] = black738
color[i] = black350
return nil739
return Issue{}351
}740
}352
for i := range jobs {741
for i := range jobs {353
if err := visit(i); err != nil {742
if iss := visit(i); iss.Key != "" {354
return err743
return iss355
}744
}356
}745
}357
return nil746
return Issue{}358
}747
}359
748
360
// ActiveJobs filtert jobs waarvan de rules niet kloppen.749
func parseArtifacts(node *yaml.Node) (Artifacts, []Issue) {361
func (p Pipeline) ActiveJobs(env map[string]string) ([]Job, error) {750
if node.Kind != yaml.MappingNode {362
var out []Job751
return Artifacts{}, []Issue{{Line: node.Line, Key: "ci.err.artifacts"}}363
for _, j := range p.Jobs {752
}364
ok, err := j.matches(env)753
var a Artifacts365
if err != nil {754
var issues []Issue366
return nil, fmt.Errorf("job %s: %w", j.Name, err)755
for i := 0; i < len(node.Content); i += 2 {367
}756
key := node.Content[i].Value368
if ok {757
val := node.Content[i+1]369
out = append(out, j)758
switch key {759
case "paths":760
paths, iss := scalarList(val, "ci.err.list")761
issues = append(issues, iss...)762
a.Paths = paths763
case "expire_in":764
if val.Kind != yaml.ScalarNode {765
issues = append(issues, Issue{Line: val.Line, Key: "ci.err.expire"})766
continue767
}768
d, err := parseExpire(val.Value)769
if err != nil {770
issues = append(issues, Issue{Line: val.Line, Key: "ci.err.expireBad", Args: []any{val.Value}})771
continue772
}773
a.ExpireIn = d774
default:775
issues = append(issues, Issue{Line: node.Content[i].Line, Key: "ci.warn.ignored", Args: []any{"artifacts." + key}, Warning: true})370
}776
}371
}777
}372
return out, nil778
return a, issues373
}779
}374
780
375
func (j Job) matches(env map[string]string) (bool, error) {781
func parseCache(node *yaml.Node) (Cache, []Issue) {376
if len(j.Rules) == 0 {782
if node.Kind != yaml.MappingNode {377
return true, nil783
return Cache{}, []Issue{{Line: node.Line, Key: "ci.err.cache"}}378
}784
}379
for _, r := range j.Rules {785
var c Cache380
ok, err := evalRule(r.If, env)786
var issues []Issue381
if err != nil {787
for i := 0; i < len(node.Content); i += 2 {382
return false, err788
key := node.Content[i].Value383
}789
val := node.Content[i+1]384
if ok {790
switch key {385
return true, nil791
case "paths":386
}792
paths, iss := scalarList(val, "ci.err.list")387
}793
issues = append(issues, iss...)388
return false, nil794
c.Paths = paths389
}795
case "key":390
796
if val.Kind != yaml.ScalarNode {391
// Gate beslist of een job mag starten.797
issues = append(issues, Issue{Line: val.Line, Key: "ci.err.cacheKey"})392
// "ready" betekent dat alle voorgangers klaar zijn.798
continue393
// "wait" betekent dat er nog een voorganger loopt of in de wachtrij staat.394
// "skip" betekent dat een voorganger mislukte zonder allow_failure.395
//396
// NeedsSet volgt de genoemde jobs, ook als de lijst leeg is.397
// Zonder needs wacht de job op alle jobs in eerdere stages.398
func Gate(jobs []Job, status map[string]string, job Job) string {399
byName := map[string]Job{}400
for _, j := range jobs {401
byName[j.Name] = j402
}403
var deps []Job404
if job.NeedsSet {405
for _, n := range job.Needs {406
deps = append(deps, byName[n])407
}408
} else {409
stagePos := map[string]int{}410
order := 0411
for _, j := range jobs {412
if _, ok := stagePos[j.Stage]; !ok {413
stagePos[j.Stage] = order414
order++415
}799
}800
c.Key = val.Value801
default:802
issues = append(issues, Issue{Line: node.Content[i].Line, Key: "ci.warn.ignored", Args: []any{"cache." + key}, Warning: true})416
}803
}417
my := stagePos[job.Stage]804
}418
for _, j := range jobs {805
return c, issues419
if stagePos[j.Stage] < my {806
}420
deps = append(deps, j)807
808
func parseRules(node *yaml.Node) ([]Rule, []Issue) {809
if node.Kind != yaml.SequenceNode {810
return nil, []Issue{{Line: node.Line, Key: "ci.err.rulesList"}}811
}812
var rules []Rule813
var issues []Issue814
for _, item := range node.Content {815
if item.Kind != yaml.MappingNode {816
issues = append(issues, Issue{Line: item.Line, Key: "ci.err.ruleMap"})817
continue818
}819
var r Rule820
for i := 0; i < len(item.Content); i += 2 {821
key := item.Content[i].Value822
val := item.Content[i+1]823
switch key {824
case "if":825
if val.Kind != yaml.ScalarNode || strings.TrimSpace(val.Value) == "" {826
issues = append(issues, Issue{Line: val.Line, Key: "ci.err.ruleIf"})827
continue828
}829
r.If = strings.TrimSpace(val.Value)830
if _, err := compileRule(r.If); err != nil {831
if ie, ok := err.(i18n.Error); ok {832
issues = append(issues, Issue{Line: val.Line, Key: ie.Key, Args: ie.Args})833
} else {834
issues = append(issues, Issue{Line: val.Line, Key: "ci.err.ruleExpr", Args: []any{r.If}})835
}836
}837
case "changes":838
list, iss := scalarList(val, "ci.err.list")839
issues = append(issues, iss...)840
r.Changes = list841
case "exists":842
list, iss := scalarList(val, "ci.err.list")843
issues = append(issues, iss...)844
r.Exists = list845
case "when":846
w, iss := whenOf(val, true)847
issues = append(issues, iss...)848
r.When = w849
default:850
issues = append(issues, Issue{Line: item.Content[i].Line, Key: "ci.warn.ignored", Args: []any{"rules." + key}, Warning: true})421
}851
}422
}852
}423
}853
if r.If == "" && len(r.Changes) == 0 && len(r.Exists) == 0 && r.When == "" {424
for _, d := range deps {854
issues = append(issues, Issue{Line: item.Line, Key: "ci.err.ruleEmpty"})425
switch status[d.Name] {426
case "success", "skipped":427
continue855
continue428
case "failed":856
}429
if d.AllowFailure {857
rules = append(rules, r)858
}859
return rules, issues860
}861
862
func parseNeeds(node *yaml.Node) (needs, optional []string, set bool, issues []Issue) {863
set = true864
if node.Kind != yaml.SequenceNode {865
return nil, nil, true, []Issue{{Line: node.Line, Key: "ci.err.needs"}}866
}867
for _, item := range node.Content {868
switch item.Kind {869
case yaml.ScalarNode:870
needs = append(needs, item.Value)871
case yaml.MappingNode:872
var name string873
var opt bool874
for i := 0; i < len(item.Content); i += 2 {875
switch item.Content[i].Value {876
case "job":877
if item.Content[i+1].Kind == yaml.ScalarNode {878
name = item.Content[i+1].Value879
}880
case "optional":881
b, iss := parseBool(item.Content[i+1])882
issues = append(issues, iss...)883
opt = b884
default:885
issues = append(issues, Issue{Line: item.Content[i].Line, Key: "ci.warn.ignored", Args: []any{"needs." + item.Content[i].Value}, Warning: true})886
}887
}888
if name == "" {889
issues = append(issues, Issue{Line: item.Line, Key: "ci.err.needs"})430
continue890
continue431
}891
}432
return "skip"892
needs = append(needs, name)893
if opt {894
optional = append(optional, name)895
}433
default:896
default:434
return "wait"897
issues = append(issues, Issue{Line: item.Line, Key: "ci.err.needs"})435
}898
}436
}899
}437
return "ready"900
if needs == nil {901
needs = []string{}902
}903
return needs, optional, set, issues438
}904
}439
905
440
var (906
func imageOf(node *yaml.Node) (string, []Issue) {441
reEq = regexp.MustCompile(`^\$([A-Za-z_][A-Za-z0-9_]*)\s*==\s*"([^"]*)"$`)907
switch node.Kind {442
reNe = regexp.MustCompile(`^\$([A-Za-z_][A-Za-z0-9_]*)\s*!=\s*"([^"]*)"$`)908
case yaml.ScalarNode:443
reRe = regexp.MustCompile(`^\$([A-Za-z_][A-Za-z0-9_]*)\s*=~\s*/(.+)/$`)909
return node.Value, nil444
)910
case yaml.MappingNode:445
911
for i := 0; i < len(node.Content); i += 2 {446
type compiledRule struct {912
if node.Content[i].Value == "name" && node.Content[i+1].Kind == yaml.ScalarNode {447
kind string913
return node.Content[i+1].Value, nil448
name string914
}449
arg string915
}450
re *regexp.Regexp916
return "", []Issue{{Line: node.Line, Key: "ci.err.image"}}917
default:918
return "", []Issue{{Line: node.Line, Key: "ci.err.image"}}919
}451
}920
}452
921
453
func compileRule(expr string) (compiledRule, error) {922
func whenOf(node *yaml.Node, allowNever bool) (string, []Issue) {454
if m := reEq.FindStringSubmatch(expr); m != nil {923
if node.Kind != yaml.ScalarNode {455
return compiledRule{kind: "eq", name: m[1], arg: m[2]}, nil924
return "", []Issue{{Line: node.Line, Key: "ci.err.when", Args: []any{""}}}456
}457
if m := reNe.FindStringSubmatch(expr); m != nil {458
return compiledRule{kind: "ne", name: m[1], arg: m[2]}, nil459
}925
}460
if m := reRe.FindStringSubmatch(expr); m != nil {926
switch node.Value {461
re, err := regexp.Compile(m[2])927
case "on_success", "on_failure", "always", "manual":462
if err != nil {928
return node.Value, nil463
return compiledRule{}, fmt.Errorf("ongeldige regex in rule %q: %w", expr, err)929
case "never":930
if allowNever {931
return node.Value, nil464
}932
}465
return compiledRule{kind: "re", name: m[1], re: re}, nil466
}933
}467
return compiledRule{}, fmt.Errorf("rule %q wordt niet begrepen", expr)934
return "", []Issue{{Line: node.Line, Key: "ci.err.when", Args: []any{node.Value}}}468
}935
}469
936
470
func evalRule(expr string, env map[string]string) (bool, error) {937
func durationOf(node *yaml.Node) (time.Duration, []Issue) {471
c, err := compileRule(expr)938
if node.Kind != yaml.ScalarNode {939
return 0, []Issue{{Line: node.Line, Key: "ci.err.timeout", Args: []any{""}}}940
}941
d, err := parseExpire(node.Value)472
if err != nil {942
if err != nil {473
return false, err943
if strings.HasSuffix(node.Value, "s") {474
}944
n, nerr := strconv.Atoi(strings.TrimSuffix(node.Value, "s"))475
val := env[c.name]945
if nerr == nil && n >= 0 {476
switch c.kind {946
return time.Duration(n) * time.Second, nil477
case "eq":947
}478
return val == c.arg, nil948
}479
case "ne":949
return 0, []Issue{{Line: node.Line, Key: "ci.err.timeout", Args: []any{node.Value}}}480
return val != c.arg, nil481
case "re":482
return c.re.MatchString(val), nil483
default:484
return false, fmt.Errorf("interne rulefout")485
}950
}951
return d, nil486
}952
}487
953
488
func parseExpire(s string) (time.Duration, error) {954
func extendsOf(node *yaml.Node) ([]string, []Issue) {489
s = strings.TrimSpace(s)955
if node.Kind == yaml.ScalarNode && strings.TrimSpace(node.Value) != "" {490
if s == "" || s == "never" {956
return []string{node.Value}, nil491
return 0, nil492
}957
}493
if len(s) < 2 {958
return scalarList(node, "ci.err.extendsList")494
return 0, fmt.Errorf("expire_in %q is ongeldig", s)959
}495
}960
496
n, err := strconv.Atoi(s[:len(s)-1])961
func scriptList(node *yaml.Node) ([]string, []Issue) {497
if err != nil || n < 0 {962
if node.Kind == yaml.ScalarNode {498
return 0, fmt.Errorf("expire_in %q is ongeldig", s)963
if strings.TrimSpace(node.Value) == "" {964
return nil, []Issue{{Line: node.Line, Key: "ci.err.script"}}965
}966
return splitLines(node.Value), nil499
}967
}500
switch s[len(s)-1] {968
return scalarList(node, "ci.err.script")501
case 'm':969
}502
return time.Duration(n) * time.Minute, nil970
503
case 'h':971
func splitLines(s string) []string {504
return time.Duration(n) * time.Hour, nil972
parts := strings.Split(s, "\n")505
case 'd':973
var out []string506
return time.Duration(n) * 24 * time.Hour, nil974
for _, p := range parts {507
case 'w':975
if strings.TrimSpace(p) == "" {508
return time.Duration(n) * 7 * 24 * time.Hour, nil976
continue509
default:977
}510
return 0, fmt.Errorf("expire_in %q is ongeldig", s)978
out = append(out, p)511
}979
}980
return out512
}981
}513
982
514
func scalarList(node *yaml.Node) ([]string, error) {983
func scalarList(node *yaml.Node, key string) ([]string, []Issue) {515
if node.Kind != yaml.SequenceNode {984
if node.Kind != yaml.SequenceNode {516
return nil, fmt.Errorf("verwacht een lijst")985
return nil, []Issue{{Line: node.Line, Key: key}}517
}986
}518
var out []string987
var out []string988
var issues []Issue519
for _, item := range node.Content {989
for _, item := range node.Content {520
if item.Kind != yaml.ScalarNode {990
if item.Kind != yaml.ScalarNode {521
return nil, fmt.Errorf("lijst mag alleen tekst bevatten")991
issues = append(issues, Issue{Line: item.Line, Key: "ci.err.listText"})992
continue522
}993
}523
out = append(out, item.Value)994
out = append(out, item.Value)524
}995
}525
return out, nil996
return out, issues526
}997
}527
998
528
func scalarMap(node *yaml.Node) (map[string]string, error) {999
func scalarMap(node *yaml.Node) (map[string]string, []Issue) {529
if node.Kind != yaml.MappingNode {1000
if node.Kind != yaml.MappingNode {530
return nil, fmt.Errorf("verwacht een mapping")1001
return nil, []Issue{{Line: node.Line, Key: "ci.err.map"}}531
}1002
}532
out := map[string]string{}1003
out := map[string]string{}1004
var issues []Issue533
for i := 0; i < len(node.Content); i += 2 {1005
for i := 0; i < len(node.Content); i += 2 {534
k := node.Content[i]1006
k := node.Content[i]535
v := node.Content[i+1]1007
v := node.Content[i+1]536
if k.Kind != yaml.ScalarNode || v.Kind != yaml.ScalarNode {1008
if k.Kind != yaml.ScalarNode {537
return nil, fmt.Errorf("variabelen moeten tekst zijn")1009
issues = append(issues, Issue{Line: k.Line, Key: "ci.err.varText"})1010
continue1011
}1012
switch v.Kind {1013
case yaml.ScalarNode:1014
out[k.Value] = v.Value1015
case yaml.MappingNode:1016
val, iss := mappingValue(v)1017
issues = append(issues, iss...)1018
if val != "" || len(iss) == 0 {1019
out[k.Value] = val1020
}1021
default:1022
issues = append(issues, Issue{Line: v.Line, Key: "ci.err.varText"})1023
}1024
}1025
return out, issues1026
}1027
1028
func mappingValue(node *yaml.Node) (string, []Issue) {1029
for i := 0; i < len(node.Content); i += 2 {1030
if node.Content[i].Value == "value" && node.Content[i+1].Kind == yaml.ScalarNode {1031
return node.Content[i+1].Value, nil538
}1032
}539
out[k.Value] = v.Value540
}1033
}541
return out, nil1034
return "", []Issue{{Line: node.Line, Key: "ci.err.varText"}}542
}1035
}543
1036
544
func parseBool(node *yaml.Node) (bool, error) {1037
func parseBool(node *yaml.Node) (bool, []Issue) {545
if node.Kind != yaml.ScalarNode {1038
if node.Kind != yaml.ScalarNode {546
return false, fmt.Errorf("verwacht ja of nee")1039
return false, []Issue{{Line: node.Line, Key: "ci.err.bool"}}547
}1040
}548
switch strings.ToLower(node.Value) {1041
switch strings.ToLower(node.Value) {549
case "true", "yes", "ja":1042
case "true", "yes":550
return true, nil1043
return true, nil551
case "false", "no", "nee":1044
case "false", "no":552
return false, nil1045
return false, nil553
default:1046
default:554
return false, fmt.Errorf("verwacht true of false")1047
return false, []Issue{{Line: node.Line, Key: "ci.err.bool"}}555
}1048
}556
}1049
}557
1050
558
func safeRelPath(p string) error {1051
func parseExpire(s string) (time.Duration, error) {1052
s = strings.TrimSpace(s)1053
if s == "" || s == "never" {1054
return 0, nil1055
}1056
if len(s) < 2 {1057
return 0, fmt.Errorf("expire")1058
}1059
n, err := strconv.Atoi(s[:len(s)-1])1060
if err != nil || n < 0 {1061
return 0, fmt.Errorf("expire")1062
}1063
switch s[len(s)-1] {1064
case 's':1065
return time.Duration(n) * time.Second, nil1066
case 'm':1067
return time.Duration(n) * time.Minute, nil1068
case 'h':1069
return time.Duration(n) * time.Hour, nil1070
case 'd':1071
return time.Duration(n) * 24 * time.Hour, nil1072
case 'w':1073
return time.Duration(n) * 7 * 24 * time.Hour, nil1074
default:1075
return 0, fmt.Errorf("expire")1076
}1077
}1078
1079
func safeRel(p string) Issue {559
if p == "" || strings.HasPrefix(p, "/") || strings.Contains(p, "\\") {1080
if p == "" || strings.HasPrefix(p, "/") || strings.Contains(p, "\\") {560
return fmt.Errorf("pad %q moet relatief zijn", p)1081
return Issue{Line: 1, Key: "ci.err.rel", Args: []any{p}}561
}1082
}562
for _, part := range strings.Split(p, "/") {1083
for _, part := range strings.Split(p, "/") {563
if part == ".." {1084
if part == ".." {564
return fmt.Errorf("pad %q mag niet buiten de workspace", p)1085
return Issue{Line: 1, Key: "ci.err.dotdot", Args: []any{p}}1086
}1087
}1088
return Issue{}1089
}1090
1091
func hasError(issues []Issue) bool {1092
for _, is := range issues {1093
if !is.Warning {1094
return true565
}1095
}566
}1096
}567
return nil1097
return false568
}1098
}@@ -0,0 +1,216 @@
1
package ci
2
3
import (
4
"strings"
5
"testing"
6
)
7
8
func TestGitLabDocument(t *testing.T) {
9
raw := []byte(`
10
stages: [test, deploy]
11
default:
12
image: alpine:3.20
13
before_script: ["echo prep"]
14
variables:
15
MODE: {value: "ci"}
16
.php:
17
image: php:8.3
18
stage: test
19
php_tests:
20
extends: .php
21
script: ["phpunit"]
22
rules:
23
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
24
changes: ["**/*.php"]
25
- when: never
26
frontend_checks:
27
stage: test
28
script: ["node check"]
29
rules:
30
- if: $CI_COMMIT_TAG == null
31
exists: ["package.json"]
32
allow_failure: true
33
cache:
34
key: modules
35
paths: ["node_modules"]
36
deploy:
37
stage: deploy
38
when: manual
39
resource_group: production
40
timeout: 30m
41
tags: [deploy]
42
only: ["master", "tags"]
43
script: ["echo deploy"]
44
needs:
45
- job: php_tests
46
optional: true
47
`)
48
p, err := Parse(raw)
49
if err != nil {
50
t.Fatal(err)
51
}
52
if strings.Join(p.Stages, ",") != "test,deploy" {
53
t.Fatalf("stages %v", p.Stages)
54
}
55
if p.Variables["MODE"] != "ci" {
56
t.Fatal(p.Variables)
57
}
58
by := map[string]Job{}
59
for _, j := range p.Jobs {
60
by[j.Name] = j
61
}
62
if _, ok := by[".php"]; ok {
63
t.Fatal("verborgen job hoort niet te draaien")
64
}
65
php := by["php_tests"]
66
if php.Image != "php:8.3" || php.Stage != "test" || php.Before[0] != "echo prep" {
67
t.Fatalf("extends %+v", php)
68
}
69
front := by["frontend_checks"]
70
if !front.AllowFailure || front.Cache.Key != "modules" || front.Cache.Paths[0] != "node_modules" {
71
t.Fatalf("frontend %+v", front)
72
}
73
dep := by["deploy"]
74
if dep.When != "manual" || dep.ResourceGroup != "production" || dep.Timeout.Minutes() != 30 {
75
t.Fatalf("deploy %+v", dep)
76
}
77
if len(dep.Needs) != 1 || dep.Optional[0] != "php_tests" {
78
t.Fatalf("needs %+v opt %+v", dep.Needs, dep.Optional)
79
}
80
81
env := map[string]string{"CI_COMMIT_BRANCH": "master", "CI_DEFAULT_BRANCH": "master", "CI_COMMIT_REF_NAME": "master"}
82
active, err := p.Select(Eval{Env: env, Changed: []string{"src/App.php"}, Files: []string{"package.json"}})
83
if err != nil {
84
t.Fatal(err)
85
}
86
names := map[string]bool{}
87
for _, j := range active {
88
names[j.Name] = true
89
}
90
if !names["php_tests"] || !names["frontend_checks"] || !names["deploy"] {
91
t.Fatalf("actief %v", names)
92
}
93
active, err = p.Select(Eval{Env: map[string]string{"CI_COMMIT_BRANCH": "feature", "CI_DEFAULT_BRANCH": "master", "CI_COMMIT_REF_NAME": "feature"}, Changed: []string{"README"}, Files: []string{}})
94
if err != nil {
95
t.Fatal(err)
96
}
97
names = map[string]bool{}
98
for _, j := range active {
99
names[j.Name] = true
100
if j.Name == "deploy" && j.When != "manual" {
101
t.Fatal(j.When)
102
}
103
}
104
if names["php_tests"] || names["frontend_checks"] {
105
t.Fatalf("feature hoort die jobs over te slaan: %v", names)
106
}
107
if names["deploy"] {
108
t.Fatal("only master/tags")
109
}
110
}
111
112
func TestIncludeLocalAndWarnings(t *testing.T) {
113
files := map[string]string{
114
"ci/base.yml": "jobs:\n unit:\n image: alpine\n script: ['echo unit']\n",
115
}
116
raw := []byte("include:\n - local: ci/base.yml\nservices: [db]\njobs:\n unit:\n script: ['echo replaced']\n")
117
p, err := ParseIn(raw, func(path string) ([]byte, error) { return []byte(files[path]), nil })
118
if err != nil {
119
t.Fatal(err)
120
}
121
if len(p.Jobs) != 1 || p.Jobs[0].Script[0] != "echo replaced" || p.Jobs[0].Image != "alpine" {
122
t.Fatalf("%+v", p.Jobs)
123
}
124
if len(p.Warnings) != 1 || p.Warnings[0].Key != "ci.warn.ignored" {
125
t.Fatalf("warnings %+v", p.Warnings)
126
}
127
_, issues := Lint([]byte("include:\n - project: other\njobs:\n a:\n image: alpine\n script: ['x']\n"), nil)
128
found := false
129
for _, is := range issues {
130
if is.Key == "ci.err.includeRemote" {
131
found = true
132
}
133
}
134
if !found {
135
t.Fatalf("issues %+v", issues)
136
}
137
}
138
139
func TestWhenAndAllowFailure(t *testing.T) {
140
p, err := Parse([]byte(`
141
stages: [een, twee]
142
jobs:
143
a:
144
stage: een
145
image: alpine
146
script: ['echo a']
147
allow_failure: true
148
b:
149
stage: twee
150
image: alpine
151
script: ['echo b']
152
c:
153
stage: twee
154
image: alpine
155
when: always
156
script: ['echo c']
157
d:
158
stage: twee
159
image: alpine
160
when: on_failure
161
script: ['echo d']
162
`))
163
if err != nil {
164
t.Fatal(err)
165
}
166
st := map[string]string{"a": "failed", "b": "queued", "c": "queued", "d": "queued"}
167
if Gate(p.Jobs, st, p.Jobs[1]) != "ready" {
168
t.Fatal("allow_failure houdt de volgende stage niet tegen")
169
}
170
if Gate(p.Jobs, st, p.Jobs[2]) != "ready" || Gate(p.Jobs, st, p.Jobs[3]) != "ready" {
171
t.Fatal("always en on_failure na een mislukking")
172
}
173
ok := map[string]string{"a": "success", "b": "queued", "c": "queued", "d": "queued"}
174
if Gate(p.Jobs, ok, p.Jobs[3]) != "skip" {
175
t.Fatal("on_failure slaat over na succes")
176
}
177
hard, err := Parse([]byte(`
178
stages: [een, twee]
179
jobs:
180
a:
181
stage: een
182
image: alpine
183
script: ['echo a']
184
b:
185
stage: twee
186
image: alpine
187
script: ['echo b']
188
`))
189
if err != nil {
190
t.Fatal(err)
191
}
192
if Gate(hard.Jobs, map[string]string{"a": "failed", "b": "queued"}, hard.Jobs[1]) != "skip" {
193
t.Fatal("harde fout slaat de rest over")
194
}
195
}
196
197
func TestMaskAndVariables(t *testing.T) {
198
if got := Mask("token=supersecret value", []string{"supersecret", "ab"}); got != "token=[MASKED] value" {
199
t.Fatal(got)
200
}
201
env := Variables(Ref{SHA: "abcdef1234567890", FullRef: "refs/tags/v1.2.0", Title: "Uitgave", DefaultBranch: "master", Source: "push", ProjectPath: "tim/app", ProjectURL: "https://git.example/tim/app", APIURL: "https://git.example/api/v1", PipelineID: 9})
202
if env["CI_COMMIT_TAG"] != "v1.2.0" || env["CI_COMMIT_BRANCH"] != "" || env["CI_COMMIT_SHORT_SHA"] != "abcdef12" {
203
t.Fatalf("%v", env)
204
}
205
if env["CI_DEFAULT_BRANCH"] != "master" || env["CI_PROJECT_NAME"] != "app" || env["CI_PIPELINE_SOURCE"] != "push" {
206
t.Fatalf("%v", env)
207
}
208
ok, err := evalRule(`$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH`, map[string]string{"CI_COMMIT_BRANCH": "master", "CI_DEFAULT_BRANCH": "master"})
209
if err != nil || !ok {
210
t.Fatal(err, ok)
211
}
212
ok, err = evalRule(`$CI_COMMIT_TAG == null`, map[string]string{"CI_COMMIT_TAG": ""})
213
if err != nil || !ok {
214
t.Fatal("null", err, ok)
215
}
216
}
@@ -0,0 +1,216 @@1
package ci2
3
import (4
"strings"5
"testing"6
)7
8
func TestGitLabDocument(t *testing.T) {9
raw := []byte(`10
stages: [test, deploy]11
default:12
image: alpine:3.2013
before_script: ["echo prep"]14
variables:15
MODE: {value: "ci"}16
.php:17
image: php:8.318
stage: test19
php_tests:20
extends: .php21
script: ["phpunit"]22
rules:23
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH24
changes: ["**/*.php"]25
- when: never26
frontend_checks:27
stage: test28
script: ["node check"]29
rules:30
- if: $CI_COMMIT_TAG == null31
exists: ["package.json"]32
allow_failure: true33
cache:34
key: modules35
paths: ["node_modules"]36
deploy:37
stage: deploy38
when: manual39
resource_group: production40
timeout: 30m41
tags: [deploy]42
only: ["master", "tags"]43
script: ["echo deploy"]44
needs:45
- job: php_tests46
optional: true47
`)48
p, err := Parse(raw)49
if err != nil {50
t.Fatal(err)51
}52
if strings.Join(p.Stages, ",") != "test,deploy" {53
t.Fatalf("stages %v", p.Stages)54
}55
if p.Variables["MODE"] != "ci" {56
t.Fatal(p.Variables)57
}58
by := map[string]Job{}59
for _, j := range p.Jobs {60
by[j.Name] = j61
}62
if _, ok := by[".php"]; ok {63
t.Fatal("verborgen job hoort niet te draaien")64
}65
php := by["php_tests"]66
if php.Image != "php:8.3" || php.Stage != "test" || php.Before[0] != "echo prep" {67
t.Fatalf("extends %+v", php)68
}69
front := by["frontend_checks"]70
if !front.AllowFailure || front.Cache.Key != "modules" || front.Cache.Paths[0] != "node_modules" {71
t.Fatalf("frontend %+v", front)72
}73
dep := by["deploy"]74
if dep.When != "manual" || dep.ResourceGroup != "production" || dep.Timeout.Minutes() != 30 {75
t.Fatalf("deploy %+v", dep)76
}77
if len(dep.Needs) != 1 || dep.Optional[0] != "php_tests" {78
t.Fatalf("needs %+v opt %+v", dep.Needs, dep.Optional)79
}80
81
env := map[string]string{"CI_COMMIT_BRANCH": "master", "CI_DEFAULT_BRANCH": "master", "CI_COMMIT_REF_NAME": "master"}82
active, err := p.Select(Eval{Env: env, Changed: []string{"src/App.php"}, Files: []string{"package.json"}})83
if err != nil {84
t.Fatal(err)85
}86
names := map[string]bool{}87
for _, j := range active {88
names[j.Name] = true89
}90
if !names["php_tests"] || !names["frontend_checks"] || !names["deploy"] {91
t.Fatalf("actief %v", names)92
}93
active, err = p.Select(Eval{Env: map[string]string{"CI_COMMIT_BRANCH": "feature", "CI_DEFAULT_BRANCH": "master", "CI_COMMIT_REF_NAME": "feature"}, Changed: []string{"README"}, Files: []string{}})94
if err != nil {95
t.Fatal(err)96
}97
names = map[string]bool{}98
for _, j := range active {99
names[j.Name] = true100
if j.Name == "deploy" && j.When != "manual" {101
t.Fatal(j.When)102
}103
}104
if names["php_tests"] || names["frontend_checks"] {105
t.Fatalf("feature hoort die jobs over te slaan: %v", names)106
}107
if names["deploy"] {108
t.Fatal("only master/tags")109
}110
}111
112
func TestIncludeLocalAndWarnings(t *testing.T) {113
files := map[string]string{114
"ci/base.yml": "jobs:\n unit:\n image: alpine\n script: ['echo unit']\n",115
}116
raw := []byte("include:\n - local: ci/base.yml\nservices: [db]\njobs:\n unit:\n script: ['echo replaced']\n")117
p, err := ParseIn(raw, func(path string) ([]byte, error) { return []byte(files[path]), nil })118
if err != nil {119
t.Fatal(err)120
}121
if len(p.Jobs) != 1 || p.Jobs[0].Script[0] != "echo replaced" || p.Jobs[0].Image != "alpine" {122
t.Fatalf("%+v", p.Jobs)123
}124
if len(p.Warnings) != 1 || p.Warnings[0].Key != "ci.warn.ignored" {125
t.Fatalf("warnings %+v", p.Warnings)126
}127
_, issues := Lint([]byte("include:\n - project: other\njobs:\n a:\n image: alpine\n script: ['x']\n"), nil)128
found := false129
for _, is := range issues {130
if is.Key == "ci.err.includeRemote" {131
found = true132
}133
}134
if !found {135
t.Fatalf("issues %+v", issues)136
}137
}138
139
func TestWhenAndAllowFailure(t *testing.T) {140
p, err := Parse([]byte(`141
stages: [een, twee]142
jobs:143
a:144
stage: een145
image: alpine146
script: ['echo a']147
allow_failure: true148
b:149
stage: twee150
image: alpine151
script: ['echo b']152
c:153
stage: twee154
image: alpine155
when: always156
script: ['echo c']157
d:158
stage: twee159
image: alpine160
when: on_failure161
script: ['echo d']162
`))163
if err != nil {164
t.Fatal(err)165
}166
st := map[string]string{"a": "failed", "b": "queued", "c": "queued", "d": "queued"}167
if Gate(p.Jobs, st, p.Jobs[1]) != "ready" {168
t.Fatal("allow_failure houdt de volgende stage niet tegen")169
}170
if Gate(p.Jobs, st, p.Jobs[2]) != "ready" || Gate(p.Jobs, st, p.Jobs[3]) != "ready" {171
t.Fatal("always en on_failure na een mislukking")172
}173
ok := map[string]string{"a": "success", "b": "queued", "c": "queued", "d": "queued"}174
if Gate(p.Jobs, ok, p.Jobs[3]) != "skip" {175
t.Fatal("on_failure slaat over na succes")176
}177
hard, err := Parse([]byte(`178
stages: [een, twee]179
jobs:180
a:181
stage: een182
image: alpine183
script: ['echo a']184
b:185
stage: twee186
image: alpine187
script: ['echo b']188
`))189
if err != nil {190
t.Fatal(err)191
}192
if Gate(hard.Jobs, map[string]string{"a": "failed", "b": "queued"}, hard.Jobs[1]) != "skip" {193
t.Fatal("harde fout slaat de rest over")194
}195
}196
197
func TestMaskAndVariables(t *testing.T) {198
if got := Mask("token=supersecret value", []string{"supersecret", "ab"}); got != "token=[MASKED] value" {199
t.Fatal(got)200
}201
env := Variables(Ref{SHA: "abcdef1234567890", FullRef: "refs/tags/v1.2.0", Title: "Uitgave", DefaultBranch: "master", Source: "push", ProjectPath: "tim/app", ProjectURL: "https://git.example/tim/app", APIURL: "https://git.example/api/v1", PipelineID: 9})202
if env["CI_COMMIT_TAG"] != "v1.2.0" || env["CI_COMMIT_BRANCH"] != "" || env["CI_COMMIT_SHORT_SHA"] != "abcdef12" {203
t.Fatalf("%v", env)204
}205
if env["CI_DEFAULT_BRANCH"] != "master" || env["CI_PROJECT_NAME"] != "app" || env["CI_PIPELINE_SOURCE"] != "push" {206
t.Fatalf("%v", env)207
}208
ok, err := evalRule(`$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH`, map[string]string{"CI_COMMIT_BRANCH": "master", "CI_DEFAULT_BRANCH": "master"})209
if err != nil || !ok {210
t.Fatal(err, ok)211
}212
ok, err = evalRule(`$CI_COMMIT_TAG == null`, map[string]string{"CI_COMMIT_TAG": ""})213
if err != nil || !ok {214
t.Fatal("null", err, ok)215
}216
}@@ -412,4 +412,57 @@ var en = map[string]string{
412
412
"err.sugSame": "the suggestions do not change the file",
413
413
"err.sugBranch": "you cannot push to %s",
414
414
"flash.suggestions": "Suggestions applied",
415
416
"ci.err.yaml": "invalid YAML: %s",
417
"ci.err.mapping": "blue-ci-cd.yml must be a mapping",
418
"ci.err.list": "expected a list",
419
"ci.err.unknown": "unknown field %q",
420
"ci.warn.ignored": "%s is not used",
421
"ci.err.both": "use either top-level jobs or a jobs map",
422
"ci.err.includeRead": "cannot read include %s",
423
"ci.err.includeCycle": "include cycle at %s",
424
"ci.err.includeLocal": "include local must be a path",
425
"ci.err.includeRemote": "only local includes are supported",
426
"ci.err.rel": "path %q must be relative",
427
"ci.err.map": "expected a mapping",
428
"ci.err.jobsMap": "jobs must be a mapping",
429
"ci.err.jobMap": "job %s must be a mapping",
430
"ci.err.stageText": "stage must be text",
431
"ci.err.script": "script must be a list of commands",
432
"ci.err.needs": "needs must name a job",
433
"ci.err.resource": "resource_group must be text",
434
"ci.err.jobDup": "duplicate job %q",
435
"ci.err.extendsCycle": "cycle in extends at %s",
436
"ci.err.extends": "job %s extends unknown job %s",
437
"ci.err.noJobs": "no jobs",
438
"ci.err.stageEmpty": "empty stage name",
439
"ci.err.stageDup": "duplicate stage %q",
440
"ci.err.jobName": "invalid job name %q",
441
"ci.err.jobStage": "job %s has no stage",
442
"ci.err.stageUnknown": "job %s uses unknown stage %q",
443
"ci.err.noImage": "job %s has no image",
444
"ci.err.noScript": "job %s has no script",
445
"ci.err.emptyLine": "job %s has an empty script line",
446
"ci.err.needUnknown": "job %s needs unknown job %q",
447
"ci.err.needSelf": "job %s cannot depend on itself",
448
"ci.err.cycle": "cycle in needs at job %s",
449
"ci.err.artifacts": "artifacts must be a mapping",
450
"ci.err.expire": "expire_in must be text",
451
"ci.err.expireBad": "expire_in %q is invalid",
452
"ci.err.cache": "cache must be a mapping",
453
"ci.err.cacheKey": "cache key must be text",
454
"ci.err.rulesList": "rules must be a list",
455
"ci.err.ruleMap": "a rule must be a mapping",
456
"ci.err.ruleIf": "rule.if must be text",
457
"ci.err.ruleEmpty": "rule has no condition",
458
"ci.err.ruleExpr": "rule %q is not understood",
459
"ci.err.ruleCombine": "rule %q cannot combine conditions with && or ||",
460
"ci.err.regex": "invalid regex in %q: %s",
461
"ci.err.when": "when %q is not supported",
462
"ci.err.timeout": "timeout %q is invalid",
463
"ci.err.extendsList": "extends must be a job name or a list",
464
"ci.err.listText": "a list may only contain text",
465
"ci.err.varText": "variables must be text",
466
"ci.err.bool": "expected true or false",
467
"ci.err.dotdot": "path %q must stay inside the workspace",
415
468
}
@@ -412,4 +412,57 @@ var en = map[string]string{412
"err.sugSame": "the suggestions do not change the file",412
"err.sugSame": "the suggestions do not change the file",413
"err.sugBranch": "you cannot push to %s",413
"err.sugBranch": "you cannot push to %s",414
"flash.suggestions": "Suggestions applied",414
"flash.suggestions": "Suggestions applied",415
416
"ci.err.yaml": "invalid YAML: %s",417
"ci.err.mapping": "blue-ci-cd.yml must be a mapping",418
"ci.err.list": "expected a list",419
"ci.err.unknown": "unknown field %q",420
"ci.warn.ignored": "%s is not used",421
"ci.err.both": "use either top-level jobs or a jobs map",422
"ci.err.includeRead": "cannot read include %s",423
"ci.err.includeCycle": "include cycle at %s",424
"ci.err.includeLocal": "include local must be a path",425
"ci.err.includeRemote": "only local includes are supported",426
"ci.err.rel": "path %q must be relative",427
"ci.err.map": "expected a mapping",428
"ci.err.jobsMap": "jobs must be a mapping",429
"ci.err.jobMap": "job %s must be a mapping",430
"ci.err.stageText": "stage must be text",431
"ci.err.script": "script must be a list of commands",432
"ci.err.needs": "needs must name a job",433
"ci.err.resource": "resource_group must be text",434
"ci.err.jobDup": "duplicate job %q",435
"ci.err.extendsCycle": "cycle in extends at %s",436
"ci.err.extends": "job %s extends unknown job %s",437
"ci.err.noJobs": "no jobs",438
"ci.err.stageEmpty": "empty stage name",439
"ci.err.stageDup": "duplicate stage %q",440
"ci.err.jobName": "invalid job name %q",441
"ci.err.jobStage": "job %s has no stage",442
"ci.err.stageUnknown": "job %s uses unknown stage %q",443
"ci.err.noImage": "job %s has no image",444
"ci.err.noScript": "job %s has no script",445
"ci.err.emptyLine": "job %s has an empty script line",446
"ci.err.needUnknown": "job %s needs unknown job %q",447
"ci.err.needSelf": "job %s cannot depend on itself",448
"ci.err.cycle": "cycle in needs at job %s",449
"ci.err.artifacts": "artifacts must be a mapping",450
"ci.err.expire": "expire_in must be text",451
"ci.err.expireBad": "expire_in %q is invalid",452
"ci.err.cache": "cache must be a mapping",453
"ci.err.cacheKey": "cache key must be text",454
"ci.err.rulesList": "rules must be a list",455
"ci.err.ruleMap": "a rule must be a mapping",456
"ci.err.ruleIf": "rule.if must be text",457
"ci.err.ruleEmpty": "rule has no condition",458
"ci.err.ruleExpr": "rule %q is not understood",459
"ci.err.ruleCombine": "rule %q cannot combine conditions with && or ||",460
"ci.err.regex": "invalid regex in %q: %s",461
"ci.err.when": "when %q is not supported",462
"ci.err.timeout": "timeout %q is invalid",463
"ci.err.extendsList": "extends must be a job name or a list",464
"ci.err.listText": "a list may only contain text",465
"ci.err.varText": "variables must be text",466
"ci.err.bool": "expected true or false",467
"ci.err.dotdot": "path %q must stay inside the workspace",415
}468
}@@ -389,26 +389,78 @@ var nl = map[string]string{
389
389
"err.whServer": "server-URL ontbreekt",
390
390
"err.whMethod": "methode moet GET, POST, PUT of PATCH zijn",
391
391
392
"mr.changes": "Wijzigingen",
393
"action.cancel": "Annuleren",
394
"sug.applyWord": "Suggesties toepassen",
395
"sug.include": "Deze suggestie meenemen",
396
"sug.title": "Titel",
397
"sug.description": "Beschrijving",
398
"sug.optional": "Optioneel.",
399
"sug.submit": "Commit maken",
400
"sug.hint": "Het suggestion-blok vervangt de geselecteerde regels.",
401
"sug.bubble": "Reageren op de geselecteerde regels",
402
"err.sugConflict": "%s:%d-%d is intussen veranderd",
403
"err.sugOverlap": "suggesties in %s overlappen",
404
"err.sugEmpty": "kies een suggestie",
405
"err.sugTitle": "een titel is verplicht",
406
"err.sugLines": "die regels staan niet in het bestand",
407
"err.sugFile": "dat bestand is niet beschikbaar",
408
"err.sugGone": "die suggestie hoort niet bij deze merge request",
409
"err.sugSame": "de suggesties veranderen het bestand niet",
410
"err.sugBranch": "je kunt niet pushen naar %s",
411
"flash.suggestions": "Suggesties toegepast",
392
"mr.changes": "Wijzigingen",
393
"action.cancel": "Annuleren",
394
"sug.applyWord": "Suggesties toepassen",
395
"sug.include": "Deze suggestie meenemen",
396
"sug.title": "Titel",
397
"sug.description": "Beschrijving",
398
"sug.optional": "Optioneel.",
399
"sug.submit": "Commit maken",
400
"sug.hint": "Het suggestion-blok vervangt de geselecteerde regels.",
401
"sug.bubble": "Reageren op de geselecteerde regels",
402
"err.sugConflict": "%s:%d-%d is intussen veranderd",
403
"err.sugOverlap": "suggesties in %s overlappen",
404
"err.sugEmpty": "kies een suggestie",
405
"err.sugTitle": "een titel is verplicht",
406
"err.sugLines": "die regels staan niet in het bestand",
407
"err.sugFile": "dat bestand is niet beschikbaar",
408
"err.sugGone": "die suggestie hoort niet bij deze merge request",
409
"err.sugSame": "de suggesties veranderen het bestand niet",
410
"err.sugBranch": "je kunt niet pushen naar %s",
411
"flash.suggestions": "Suggesties toegepast",
412
"ci.err.yaml": "ongeldige YAML: %s",
413
"ci.err.mapping": "blue-ci-cd.yml moet een mapping zijn",
414
"ci.err.list": "verwacht een lijst",
415
"ci.err.unknown": "onbekend veld %q",
416
"ci.warn.ignored": "%s wordt niet gebruikt",
417
"ci.err.both": "gebruik of losse jobs, of een jobs-map",
418
"ci.err.includeRead": "include %s is niet te lezen",
419
"ci.err.includeCycle": "include-cyclus bij %s",
420
"ci.err.includeLocal": "include local moet een pad zijn",
421
"ci.err.includeRemote": "alleen lokale includes worden ondersteund",
422
"ci.err.rel": "pad %q moet relatief zijn",
423
"ci.err.map": "verwacht een mapping",
424
"ci.err.jobsMap": "jobs moet een mapping zijn",
425
"ci.err.jobMap": "job %s moet een mapping zijn",
426
"ci.err.stageText": "stage moet tekst zijn",
427
"ci.err.script": "script moet een lijst commando's zijn",
428
"ci.err.needs": "needs moet een job noemen",
429
"ci.err.resource": "resource_group moet tekst zijn",
430
"ci.err.jobDup": "dubbele job %q",
431
"ci.err.extendsCycle": "cyclus in extends bij %s",
432
"ci.err.extends": "job %s erft van onbekende job %s",
433
"ci.err.noJobs": "geen jobs",
434
"ci.err.stageEmpty": "lege stage-naam",
435
"ci.err.stageDup": "dubbele stage %q",
436
"ci.err.jobName": "ongeldige jobnaam %q",
437
"ci.err.jobStage": "job %s heeft geen stage",
438
"ci.err.stageUnknown": "job %s gebruikt onbekende stage %q",
439
"ci.err.noImage": "job %s heeft geen image",
440
"ci.err.noScript": "job %s heeft geen script",
441
"ci.err.emptyLine": "job %s heeft een lege scriptregel",
442
"ci.err.needUnknown": "job %s heeft onbekende need %q",
443
"ci.err.needSelf": "job %s kan niet van zichzelf afhangen",
444
"ci.err.cycle": "cyclus in needs bij job %s",
445
"ci.err.artifacts": "artifacts moet een mapping zijn",
446
"ci.err.expire": "expire_in moet tekst zijn",
447
"ci.err.expireBad": "expire_in %q is ongeldig",
448
"ci.err.cache": "cache moet een mapping zijn",
449
"ci.err.cacheKey": "cache-sleutel moet tekst zijn",
450
"ci.err.rulesList": "rules moet een lijst zijn",
451
"ci.err.ruleMap": "een rule moet een mapping zijn",
452
"ci.err.ruleIf": "rule.if moet tekst zijn",
453
"ci.err.ruleEmpty": "rule heeft geen voorwaarde",
454
"ci.err.ruleExpr": "rule %q wordt niet begrepen",
455
"ci.err.ruleCombine": "rule %q kan geen voorwaarden combineren met && of ||",
456
"ci.err.regex": "ongeldige regex in %q: %s",
457
"ci.err.when": "when %q wordt niet ondersteund",
458
"ci.err.timeout": "timeout %q is ongeldig",
459
"ci.err.extendsList": "extends moet een jobnaam of een lijst zijn",
460
"ci.err.listText": "een lijst mag alleen tekst bevatten",
461
"ci.err.varText": "variabelen moeten tekst zijn",
462
"ci.err.bool": "verwacht true of false",
463
"ci.err.dotdot": "pad %q moet binnen de werkmap blijven",
412
464
}
413
465
414
466
var de = map[string]string{
@@ -800,26 +852,78 @@ var de = map[string]string{
800
852
"err.whServer": "Server-URL fehlt",
801
853
"err.whMethod": "Methode muss GET, POST, PUT oder PATCH sein",
802
854
803
"mr.changes": "Änderungen",
804
"action.cancel": "Abbrechen",
805
"sug.applyWord": "Vorschläge anwenden",
806
"sug.include": "Diesen Vorschlag übernehmen",
807
"sug.title": "Titel",
808
"sug.description": "Beschreibung",
809
"sug.optional": "Optional.",
810
"sug.submit": "Commit erstellen",
811
"sug.hint": "Der suggestion-Block ersetzt die ausgewählten Zeilen.",
812
"sug.bubble": "Die ausgewählten Zeilen kommentieren",
813
"err.sugConflict": "%s:%d-%d hat sich inzwischen geändert",
814
"err.sugOverlap": "Vorschläge in %s überlappen sich",
815
"err.sugEmpty": "wähle einen Vorschlag",
816
"err.sugTitle": "ein Titel ist erforderlich",
817
"err.sugLines": "diese Zeilen stehen nicht in der Datei",
818
"err.sugFile": "diese Datei ist nicht verfügbar",
819
"err.sugGone": "dieser Vorschlag gehört nicht zu diesem Merge-Request",
820
"err.sugSame": "die Vorschläge ändern die Datei nicht",
821
"err.sugBranch": "du kannst nicht nach %s pushen",
822
"flash.suggestions": "Vorschläge angewendet",
855
"mr.changes": "Änderungen",
856
"action.cancel": "Abbrechen",
857
"sug.applyWord": "Vorschläge anwenden",
858
"sug.include": "Diesen Vorschlag übernehmen",
859
"sug.title": "Titel",
860
"sug.description": "Beschreibung",
861
"sug.optional": "Optional.",
862
"sug.submit": "Commit erstellen",
863
"sug.hint": "Der suggestion-Block ersetzt die ausgewählten Zeilen.",
864
"sug.bubble": "Die ausgewählten Zeilen kommentieren",
865
"err.sugConflict": "%s:%d-%d hat sich inzwischen geändert",
866
"err.sugOverlap": "Vorschläge in %s überlappen sich",
867
"err.sugEmpty": "wähle einen Vorschlag",
868
"err.sugTitle": "ein Titel ist erforderlich",
869
"err.sugLines": "diese Zeilen stehen nicht in der Datei",
870
"err.sugFile": "diese Datei ist nicht verfügbar",
871
"err.sugGone": "dieser Vorschlag gehört nicht zu diesem Merge-Request",
872
"err.sugSame": "die Vorschläge ändern die Datei nicht",
873
"err.sugBranch": "du kannst nicht nach %s pushen",
874
"flash.suggestions": "Vorschläge angewendet",
875
"ci.err.yaml": "ungültiges YAML: %s",
876
"ci.err.mapping": "blue-ci-cd.yml muss ein Mapping sein",
877
"ci.err.list": "eine Liste wird erwartet",
878
"ci.err.unknown": "unbekanntes Feld %q",
879
"ci.warn.ignored": "%s wird nicht verwendet",
880
"ci.err.both": "entweder einzelne Jobs oder eine jobs-Map verwenden",
881
"ci.err.includeRead": "include %s ist nicht lesbar",
882
"ci.err.includeCycle": "Include-Zyklus bei %s",
883
"ci.err.includeLocal": "include local muss ein Pfad sein",
884
"ci.err.includeRemote": "nur lokale Includes werden unterstützt",
885
"ci.err.rel": "Pfad %q muss relativ sein",
886
"ci.err.map": "ein Mapping wird erwartet",
887
"ci.err.jobsMap": "jobs muss ein Mapping sein",
888
"ci.err.jobMap": "Job %s muss ein Mapping sein",
889
"ci.err.stageText": "stage muss Text sein",
890
"ci.err.script": "script muss eine Befehlsliste sein",
891
"ci.err.needs": "needs muss einen Job nennen",
892
"ci.err.resource": "resource_group muss Text sein",
893
"ci.err.jobDup": "doppelter Job %q",
894
"ci.err.extendsCycle": "Zyklus in extends bei %s",
895
"ci.err.extends": "Job %s erweitert den unbekannten Job %s",
896
"ci.err.noJobs": "keine Jobs",
897
"ci.err.stageEmpty": "leerer Stage-Name",
898
"ci.err.stageDup": "doppelte Stage %q",
899
"ci.err.jobName": "ungültiger Jobname %q",
900
"ci.err.jobStage": "Job %s hat keine Stage",
901
"ci.err.stageUnknown": "Job %s nutzt die unbekannte Stage %q",
902
"ci.err.noImage": "Job %s hat kein Image",
903
"ci.err.noScript": "Job %s hat kein script",
904
"ci.err.emptyLine": "Job %s hat eine leere Scriptzeile",
905
"ci.err.needUnknown": "Job %s braucht den unbekannten Job %q",
906
"ci.err.needSelf": "Job %s kann nicht von sich selbst abhängen",
907
"ci.err.cycle": "Zyklus in needs bei Job %s",
908
"ci.err.artifacts": "artifacts muss ein Mapping sein",
909
"ci.err.expire": "expire_in muss Text sein",
910
"ci.err.expireBad": "expire_in %q ist ungültig",
911
"ci.err.cache": "cache muss ein Mapping sein",
912
"ci.err.cacheKey": "Cache-Schlüssel muss Text sein",
913
"ci.err.rulesList": "rules muss eine Liste sein",
914
"ci.err.ruleMap": "eine Rule muss ein Mapping sein",
915
"ci.err.ruleIf": "rule.if muss Text sein",
916
"ci.err.ruleEmpty": "Rule hat keine Bedingung",
917
"ci.err.ruleExpr": "Rule %q wird nicht verstanden",
918
"ci.err.ruleCombine": "Rule %q kann Bedingungen nicht mit && oder || verbinden",
919
"ci.err.regex": "ungültiger regulärer Ausdruck in %q: %s",
920
"ci.err.when": "when %q wird nicht unterstützt",
921
"ci.err.timeout": "timeout %q ist ungültig",
922
"ci.err.extendsList": "extends muss ein Jobname oder eine Liste sein",
923
"ci.err.listText": "eine Liste darf nur Text enthalten",
924
"ci.err.varText": "Variablen müssen Text sein",
925
"ci.err.bool": "true oder false erwartet",
926
"ci.err.dotdot": "Pfad %q muss im Arbeitsverzeichnis bleiben",
823
927
}
824
928
825
929
var fr = map[string]string{
@@ -1211,24 +1315,76 @@ var fr = map[string]string{
1211
1315
"err.whServer": "l’URL du serveur manque",
1212
1316
"err.whMethod": "la méthode doit être GET, POST, PUT ou PATCH",
1213
1317
1214
"mr.changes": "Modifications",
1215
"action.cancel": "Annuler",
1216
"sug.applyWord": "Appliquer les suggestions",
1217
"sug.include": "Inclure cette suggestion",
1218
"sug.title": "Titre",
1219
"sug.description": "Description",
1220
"sug.optional": "Facultatif.",
1221
"sug.submit": "Créer le commit",
1222
"sug.hint": "Le bloc suggestion remplace les lignes sélectionnées.",
1223
"sug.bubble": "Commenter les lignes sélectionnées",
1224
"err.sugConflict": "%s:%d-%d a changé entre-temps",
1225
"err.sugOverlap": "les suggestions dans %s se chevauchent",
1226
"err.sugEmpty": "choisis une suggestion",
1227
"err.sugTitle": "un titre est obligatoire",
1228
"err.sugLines": "ces lignes ne sont pas dans le fichier",
1229
"err.sugFile": "ce fichier n’est pas disponible",
1230
"err.sugGone": "cette suggestion n’appartient pas à cette merge request",
1231
"err.sugSame": "les suggestions ne modifient pas le fichier",
1232
"err.sugBranch": "tu ne peux pas pousser vers %s",
1233
"flash.suggestions": "Suggestions appliquées",
1318
"mr.changes": "Modifications",
1319
"action.cancel": "Annuler",
1320
"sug.applyWord": "Appliquer les suggestions",
1321
"sug.include": "Inclure cette suggestion",
1322
"sug.title": "Titre",
1323
"sug.description": "Description",
1324
"sug.optional": "Facultatif.",
1325
"sug.submit": "Créer le commit",
1326
"sug.hint": "Le bloc suggestion remplace les lignes sélectionnées.",
1327
"sug.bubble": "Commenter les lignes sélectionnées",
1328
"err.sugConflict": "%s:%d-%d a changé entre-temps",
1329
"err.sugOverlap": "les suggestions dans %s se chevauchent",
1330
"err.sugEmpty": "choisis une suggestion",
1331
"err.sugTitle": "un titre est obligatoire",
1332
"err.sugLines": "ces lignes ne sont pas dans le fichier",
1333
"err.sugFile": "ce fichier n’est pas disponible",
1334
"err.sugGone": "cette suggestion n’appartient pas à cette merge request",
1335
"err.sugSame": "les suggestions ne modifient pas le fichier",
1336
"err.sugBranch": "tu ne peux pas pousser vers %s",
1337
"flash.suggestions": "Suggestions appliquées",
1338
"ci.err.yaml": "YAML invalide : %s",
1339
"ci.err.mapping": "blue-ci-cd.yml doit être un mapping",
1340
"ci.err.list": "une liste est attendue",
1341
"ci.err.unknown": "champ inconnu %q",
1342
"ci.warn.ignored": "%s n'est pas utilisé",
1343
"ci.err.both": "utilise soit des jobs au premier niveau, soit une map jobs",
1344
"ci.err.includeRead": "include %s est illisible",
1345
"ci.err.includeCycle": "cycle d'include à %s",
1346
"ci.err.includeLocal": "include local doit être un chemin",
1347
"ci.err.includeRemote": "seuls les includes locaux sont pris en charge",
1348
"ci.err.rel": "le chemin %q doit être relatif",
1349
"ci.err.map": "un mapping est attendu",
1350
"ci.err.jobsMap": "jobs doit être un mapping",
1351
"ci.err.jobMap": "le job %s doit être un mapping",
1352
"ci.err.stageText": "stage doit être du texte",
1353
"ci.err.script": "script doit être une liste de commandes",
1354
"ci.err.needs": "needs doit nommer un job",
1355
"ci.err.resource": "resource_group doit être du texte",
1356
"ci.err.jobDup": "job en double %q",
1357
"ci.err.extendsCycle": "cycle dans extends à %s",
1358
"ci.err.extends": "le job %s étend le job inconnu %s",
1359
"ci.err.noJobs": "aucun job",
1360
"ci.err.stageEmpty": "nom de stage vide",
1361
"ci.err.stageDup": "stage en double %q",
1362
"ci.err.jobName": "nom de job invalide %q",
1363
"ci.err.jobStage": "le job %s n'a pas de stage",
1364
"ci.err.stageUnknown": "le job %s utilise le stage inconnu %q",
1365
"ci.err.noImage": "le job %s n'a pas d'image",
1366
"ci.err.noScript": "le job %s n'a pas de script",
1367
"ci.err.emptyLine": "le job %s a une ligne de script vide",
1368
"ci.err.needUnknown": "le job %s dépend du job inconnu %q",
1369
"ci.err.needSelf": "le job %s ne peut pas dépendre de lui-même",
1370
"ci.err.cycle": "cycle dans needs au job %s",
1371
"ci.err.artifacts": "artifacts doit être un mapping",
1372
"ci.err.expire": "expire_in doit être du texte",
1373
"ci.err.expireBad": "expire_in %q est invalide",
1374
"ci.err.cache": "cache doit être un mapping",
1375
"ci.err.cacheKey": "la clé de cache doit être du texte",
1376
"ci.err.rulesList": "rules doit être une liste",
1377
"ci.err.ruleMap": "une règle doit être un mapping",
1378
"ci.err.ruleIf": "rule.if doit être du texte",
1379
"ci.err.ruleEmpty": "la règle n'a pas de condition",
1380
"ci.err.ruleExpr": "la règle %q n'est pas comprise",
1381
"ci.err.ruleCombine": "la règle %q ne peut pas combiner des conditions avec && ou ||",
1382
"ci.err.regex": "expression régulière invalide dans %q : %s",
1383
"ci.err.when": "when %q n'est pas pris en charge",
1384
"ci.err.timeout": "timeout %q est invalide",
1385
"ci.err.extendsList": "extends doit être un nom de job ou une liste",
1386
"ci.err.listText": "une liste ne peut contenir que du texte",
1387
"ci.err.varText": "les variables doivent être du texte",
1388
"ci.err.bool": "true ou false attendu",
1389
"ci.err.dotdot": "le chemin %q doit rester dans l'espace de travail",
1234
1390
}
@@ -389,26 +389,78 @@ var nl = map[string]string{389
"err.whServer": "server-URL ontbreekt",389
"err.whServer": "server-URL ontbreekt",390
"err.whMethod": "methode moet GET, POST, PUT of PATCH zijn",390
"err.whMethod": "methode moet GET, POST, PUT of PATCH zijn",391
391
392
"mr.changes": "Wijzigingen",392
"mr.changes": "Wijzigingen",393
"action.cancel": "Annuleren",393
"action.cancel": "Annuleren",394
"sug.applyWord": "Suggesties toepassen",394
"sug.applyWord": "Suggesties toepassen",395
"sug.include": "Deze suggestie meenemen",395
"sug.include": "Deze suggestie meenemen",396
"sug.title": "Titel",396
"sug.title": "Titel",397
"sug.description": "Beschrijving",397
"sug.description": "Beschrijving",398
"sug.optional": "Optioneel.",398
"sug.optional": "Optioneel.",399
"sug.submit": "Commit maken",399
"sug.submit": "Commit maken",400
"sug.hint": "Het suggestion-blok vervangt de geselecteerde regels.",400
"sug.hint": "Het suggestion-blok vervangt de geselecteerde regels.",401
"sug.bubble": "Reageren op de geselecteerde regels",401
"sug.bubble": "Reageren op de geselecteerde regels",402
"err.sugConflict": "%s:%d-%d is intussen veranderd",402
"err.sugConflict": "%s:%d-%d is intussen veranderd",403
"err.sugOverlap": "suggesties in %s overlappen",403
"err.sugOverlap": "suggesties in %s overlappen",404
"err.sugEmpty": "kies een suggestie",404
"err.sugEmpty": "kies een suggestie",405
"err.sugTitle": "een titel is verplicht",405
"err.sugTitle": "een titel is verplicht",406
"err.sugLines": "die regels staan niet in het bestand",406
"err.sugLines": "die regels staan niet in het bestand",407
"err.sugFile": "dat bestand is niet beschikbaar",407
"err.sugFile": "dat bestand is niet beschikbaar",408
"err.sugGone": "die suggestie hoort niet bij deze merge request",408
"err.sugGone": "die suggestie hoort niet bij deze merge request",409
"err.sugSame": "de suggesties veranderen het bestand niet",409
"err.sugSame": "de suggesties veranderen het bestand niet",410
"err.sugBranch": "je kunt niet pushen naar %s",410
"err.sugBranch": "je kunt niet pushen naar %s",411
"flash.suggestions": "Suggesties toegepast",411
"flash.suggestions": "Suggesties toegepast",412
"ci.err.yaml": "ongeldige YAML: %s",413
"ci.err.mapping": "blue-ci-cd.yml moet een mapping zijn",414
"ci.err.list": "verwacht een lijst",415
"ci.err.unknown": "onbekend veld %q",416
"ci.warn.ignored": "%s wordt niet gebruikt",417
"ci.err.both": "gebruik of losse jobs, of een jobs-map",418
"ci.err.includeRead": "include %s is niet te lezen",419
"ci.err.includeCycle": "include-cyclus bij %s",420
"ci.err.includeLocal": "include local moet een pad zijn",421
"ci.err.includeRemote": "alleen lokale includes worden ondersteund",422
"ci.err.rel": "pad %q moet relatief zijn",423
"ci.err.map": "verwacht een mapping",424
"ci.err.jobsMap": "jobs moet een mapping zijn",425
"ci.err.jobMap": "job %s moet een mapping zijn",426
"ci.err.stageText": "stage moet tekst zijn",427
"ci.err.script": "script moet een lijst commando's zijn",428
"ci.err.needs": "needs moet een job noemen",429
"ci.err.resource": "resource_group moet tekst zijn",430
"ci.err.jobDup": "dubbele job %q",431
"ci.err.extendsCycle": "cyclus in extends bij %s",432
"ci.err.extends": "job %s erft van onbekende job %s",433
"ci.err.noJobs": "geen jobs",434
"ci.err.stageEmpty": "lege stage-naam",435
"ci.err.stageDup": "dubbele stage %q",436
"ci.err.jobName": "ongeldige jobnaam %q",437
"ci.err.jobStage": "job %s heeft geen stage",438
"ci.err.stageUnknown": "job %s gebruikt onbekende stage %q",439
"ci.err.noImage": "job %s heeft geen image",440
"ci.err.noScript": "job %s heeft geen script",441
"ci.err.emptyLine": "job %s heeft een lege scriptregel",442
"ci.err.needUnknown": "job %s heeft onbekende need %q",443
"ci.err.needSelf": "job %s kan niet van zichzelf afhangen",444
"ci.err.cycle": "cyclus in needs bij job %s",445
"ci.err.artifacts": "artifacts moet een mapping zijn",446
"ci.err.expire": "expire_in moet tekst zijn",447
"ci.err.expireBad": "expire_in %q is ongeldig",448
"ci.err.cache": "cache moet een mapping zijn",449
"ci.err.cacheKey": "cache-sleutel moet tekst zijn",450
"ci.err.rulesList": "rules moet een lijst zijn",451
"ci.err.ruleMap": "een rule moet een mapping zijn",452
"ci.err.ruleIf": "rule.if moet tekst zijn",453
"ci.err.ruleEmpty": "rule heeft geen voorwaarde",454
"ci.err.ruleExpr": "rule %q wordt niet begrepen",455
"ci.err.ruleCombine": "rule %q kan geen voorwaarden combineren met && of ||",456
"ci.err.regex": "ongeldige regex in %q: %s",457
"ci.err.when": "when %q wordt niet ondersteund",458
"ci.err.timeout": "timeout %q is ongeldig",459
"ci.err.extendsList": "extends moet een jobnaam of een lijst zijn",460
"ci.err.listText": "een lijst mag alleen tekst bevatten",461
"ci.err.varText": "variabelen moeten tekst zijn",462
"ci.err.bool": "verwacht true of false",463
"ci.err.dotdot": "pad %q moet binnen de werkmap blijven",412
}464
}413
465
414
var de = map[string]string{466
var de = map[string]string{@@ -800,26 +852,78 @@ var de = map[string]string{800
"err.whServer": "Server-URL fehlt",852
"err.whServer": "Server-URL fehlt",801
"err.whMethod": "Methode muss GET, POST, PUT oder PATCH sein",853
"err.whMethod": "Methode muss GET, POST, PUT oder PATCH sein",802
854
803
"mr.changes": "Änderungen",855
"mr.changes": "Änderungen",804
"action.cancel": "Abbrechen",856
"action.cancel": "Abbrechen",805
"sug.applyWord": "Vorschläge anwenden",857
"sug.applyWord": "Vorschläge anwenden",806
"sug.include": "Diesen Vorschlag übernehmen",858
"sug.include": "Diesen Vorschlag übernehmen",807
"sug.title": "Titel",859
"sug.title": "Titel",808
"sug.description": "Beschreibung",860
"sug.description": "Beschreibung",809
"sug.optional": "Optional.",861
"sug.optional": "Optional.",810
"sug.submit": "Commit erstellen",862
"sug.submit": "Commit erstellen",811
"sug.hint": "Der suggestion-Block ersetzt die ausgewählten Zeilen.",863
"sug.hint": "Der suggestion-Block ersetzt die ausgewählten Zeilen.",812
"sug.bubble": "Die ausgewählten Zeilen kommentieren",864
"sug.bubble": "Die ausgewählten Zeilen kommentieren",813
"err.sugConflict": "%s:%d-%d hat sich inzwischen geändert",865
"err.sugConflict": "%s:%d-%d hat sich inzwischen geändert",814
"err.sugOverlap": "Vorschläge in %s überlappen sich",866
"err.sugOverlap": "Vorschläge in %s überlappen sich",815
"err.sugEmpty": "wähle einen Vorschlag",867
"err.sugEmpty": "wähle einen Vorschlag",816
"err.sugTitle": "ein Titel ist erforderlich",868
"err.sugTitle": "ein Titel ist erforderlich",817
"err.sugLines": "diese Zeilen stehen nicht in der Datei",869
"err.sugLines": "diese Zeilen stehen nicht in der Datei",818
"err.sugFile": "diese Datei ist nicht verfügbar",870
"err.sugFile": "diese Datei ist nicht verfügbar",819
"err.sugGone": "dieser Vorschlag gehört nicht zu diesem Merge-Request",871
"err.sugGone": "dieser Vorschlag gehört nicht zu diesem Merge-Request",820
"err.sugSame": "die Vorschläge ändern die Datei nicht",872
"err.sugSame": "die Vorschläge ändern die Datei nicht",821
"err.sugBranch": "du kannst nicht nach %s pushen",873
"err.sugBranch": "du kannst nicht nach %s pushen",822
"flash.suggestions": "Vorschläge angewendet",874
"flash.suggestions": "Vorschläge angewendet",875
"ci.err.yaml": "ungültiges YAML: %s",876
"ci.err.mapping": "blue-ci-cd.yml muss ein Mapping sein",877
"ci.err.list": "eine Liste wird erwartet",878
"ci.err.unknown": "unbekanntes Feld %q",879
"ci.warn.ignored": "%s wird nicht verwendet",880
"ci.err.both": "entweder einzelne Jobs oder eine jobs-Map verwenden",881
"ci.err.includeRead": "include %s ist nicht lesbar",882
"ci.err.includeCycle": "Include-Zyklus bei %s",883
"ci.err.includeLocal": "include local muss ein Pfad sein",884
"ci.err.includeRemote": "nur lokale Includes werden unterstützt",885
"ci.err.rel": "Pfad %q muss relativ sein",886
"ci.err.map": "ein Mapping wird erwartet",887
"ci.err.jobsMap": "jobs muss ein Mapping sein",888
"ci.err.jobMap": "Job %s muss ein Mapping sein",889
"ci.err.stageText": "stage muss Text sein",890
"ci.err.script": "script muss eine Befehlsliste sein",891
"ci.err.needs": "needs muss einen Job nennen",892
"ci.err.resource": "resource_group muss Text sein",893
"ci.err.jobDup": "doppelter Job %q",894
"ci.err.extendsCycle": "Zyklus in extends bei %s",895
"ci.err.extends": "Job %s erweitert den unbekannten Job %s",896
"ci.err.noJobs": "keine Jobs",897
"ci.err.stageEmpty": "leerer Stage-Name",898
"ci.err.stageDup": "doppelte Stage %q",899
"ci.err.jobName": "ungültiger Jobname %q",900
"ci.err.jobStage": "Job %s hat keine Stage",901
"ci.err.stageUnknown": "Job %s nutzt die unbekannte Stage %q",902
"ci.err.noImage": "Job %s hat kein Image",903
"ci.err.noScript": "Job %s hat kein script",904
"ci.err.emptyLine": "Job %s hat eine leere Scriptzeile",905
"ci.err.needUnknown": "Job %s braucht den unbekannten Job %q",906
"ci.err.needSelf": "Job %s kann nicht von sich selbst abhängen",907
"ci.err.cycle": "Zyklus in needs bei Job %s",908
"ci.err.artifacts": "artifacts muss ein Mapping sein",909
"ci.err.expire": "expire_in muss Text sein",910
"ci.err.expireBad": "expire_in %q ist ungültig",911
"ci.err.cache": "cache muss ein Mapping sein",912
"ci.err.cacheKey": "Cache-Schlüssel muss Text sein",913
"ci.err.rulesList": "rules muss eine Liste sein",914
"ci.err.ruleMap": "eine Rule muss ein Mapping sein",915
"ci.err.ruleIf": "rule.if muss Text sein",916
"ci.err.ruleEmpty": "Rule hat keine Bedingung",917
"ci.err.ruleExpr": "Rule %q wird nicht verstanden",918
"ci.err.ruleCombine": "Rule %q kann Bedingungen nicht mit && oder || verbinden",919
"ci.err.regex": "ungültiger regulärer Ausdruck in %q: %s",920
"ci.err.when": "when %q wird nicht unterstützt",921
"ci.err.timeout": "timeout %q ist ungültig",922
"ci.err.extendsList": "extends muss ein Jobname oder eine Liste sein",923
"ci.err.listText": "eine Liste darf nur Text enthalten",924
"ci.err.varText": "Variablen müssen Text sein",925
"ci.err.bool": "true oder false erwartet",926
"ci.err.dotdot": "Pfad %q muss im Arbeitsverzeichnis bleiben",823
}927
}824
928
825
var fr = map[string]string{929
var fr = map[string]string{@@ -1211,24 +1315,76 @@ var fr = map[string]string{1211
"err.whServer": "l’URL du serveur manque",1315
"err.whServer": "l’URL du serveur manque",1212
"err.whMethod": "la méthode doit être GET, POST, PUT ou PATCH",1316
"err.whMethod": "la méthode doit être GET, POST, PUT ou PATCH",1213
1317
1214
"mr.changes": "Modifications",1318
"mr.changes": "Modifications",1215
"action.cancel": "Annuler",1319
"action.cancel": "Annuler",1216
"sug.applyWord": "Appliquer les suggestions",1320
"sug.applyWord": "Appliquer les suggestions",1217
"sug.include": "Inclure cette suggestion",1321
"sug.include": "Inclure cette suggestion",1218
"sug.title": "Titre",1322
"sug.title": "Titre",1219
"sug.description": "Description",1323
"sug.description": "Description",1220
"sug.optional": "Facultatif.",1324
"sug.optional": "Facultatif.",1221
"sug.submit": "Créer le commit",1325
"sug.submit": "Créer le commit",1222
"sug.hint": "Le bloc suggestion remplace les lignes sélectionnées.",1326
"sug.hint": "Le bloc suggestion remplace les lignes sélectionnées.",1223
"sug.bubble": "Commenter les lignes sélectionnées",1327
"sug.bubble": "Commenter les lignes sélectionnées",1224
"err.sugConflict": "%s:%d-%d a changé entre-temps",1328
"err.sugConflict": "%s:%d-%d a changé entre-temps",1225
"err.sugOverlap": "les suggestions dans %s se chevauchent",1329
"err.sugOverlap": "les suggestions dans %s se chevauchent",1226
"err.sugEmpty": "choisis une suggestion",1330
"err.sugEmpty": "choisis une suggestion",1227
"err.sugTitle": "un titre est obligatoire",1331
"err.sugTitle": "un titre est obligatoire",1228
"err.sugLines": "ces lignes ne sont pas dans le fichier",1332
"err.sugLines": "ces lignes ne sont pas dans le fichier",1229
"err.sugFile": "ce fichier n’est pas disponible",1333
"err.sugFile": "ce fichier n’est pas disponible",1230
"err.sugGone": "cette suggestion n’appartient pas à cette merge request",1334
"err.sugGone": "cette suggestion n’appartient pas à cette merge request",1231
"err.sugSame": "les suggestions ne modifient pas le fichier",1335
"err.sugSame": "les suggestions ne modifient pas le fichier",1232
"err.sugBranch": "tu ne peux pas pousser vers %s",1336
"err.sugBranch": "tu ne peux pas pousser vers %s",1233
"flash.suggestions": "Suggestions appliquées",1337
"flash.suggestions": "Suggestions appliquées",1338
"ci.err.yaml": "YAML invalide : %s",1339
"ci.err.mapping": "blue-ci-cd.yml doit être un mapping",1340
"ci.err.list": "une liste est attendue",1341
"ci.err.unknown": "champ inconnu %q",1342
"ci.warn.ignored": "%s n'est pas utilisé",1343
"ci.err.both": "utilise soit des jobs au premier niveau, soit une map jobs",1344
"ci.err.includeRead": "include %s est illisible",1345
"ci.err.includeCycle": "cycle d'include à %s",1346
"ci.err.includeLocal": "include local doit être un chemin",1347
"ci.err.includeRemote": "seuls les includes locaux sont pris en charge",1348
"ci.err.rel": "le chemin %q doit être relatif",1349
"ci.err.map": "un mapping est attendu",1350
"ci.err.jobsMap": "jobs doit être un mapping",1351
"ci.err.jobMap": "le job %s doit être un mapping",1352
"ci.err.stageText": "stage doit être du texte",1353
"ci.err.script": "script doit être une liste de commandes",1354
"ci.err.needs": "needs doit nommer un job",1355
"ci.err.resource": "resource_group doit être du texte",1356
"ci.err.jobDup": "job en double %q",1357
"ci.err.extendsCycle": "cycle dans extends à %s",1358
"ci.err.extends": "le job %s étend le job inconnu %s",1359
"ci.err.noJobs": "aucun job",1360
"ci.err.stageEmpty": "nom de stage vide",1361
"ci.err.stageDup": "stage en double %q",1362
"ci.err.jobName": "nom de job invalide %q",1363
"ci.err.jobStage": "le job %s n'a pas de stage",1364
"ci.err.stageUnknown": "le job %s utilise le stage inconnu %q",1365
"ci.err.noImage": "le job %s n'a pas d'image",1366
"ci.err.noScript": "le job %s n'a pas de script",1367
"ci.err.emptyLine": "le job %s a une ligne de script vide",1368
"ci.err.needUnknown": "le job %s dépend du job inconnu %q",1369
"ci.err.needSelf": "le job %s ne peut pas dépendre de lui-même",1370
"ci.err.cycle": "cycle dans needs au job %s",1371
"ci.err.artifacts": "artifacts doit être un mapping",1372
"ci.err.expire": "expire_in doit être du texte",1373
"ci.err.expireBad": "expire_in %q est invalide",1374
"ci.err.cache": "cache doit être un mapping",1375
"ci.err.cacheKey": "la clé de cache doit être du texte",1376
"ci.err.rulesList": "rules doit être une liste",1377
"ci.err.ruleMap": "une règle doit être un mapping",1378
"ci.err.ruleIf": "rule.if doit être du texte",1379
"ci.err.ruleEmpty": "la règle n'a pas de condition",1380
"ci.err.ruleExpr": "la règle %q n'est pas comprise",1381
"ci.err.ruleCombine": "la règle %q ne peut pas combiner des conditions avec && ou ||",1382
"ci.err.regex": "expression régulière invalide dans %q : %s",1383
"ci.err.when": "when %q n'est pas pris en charge",1384
"ci.err.timeout": "timeout %q est invalide",1385
"ci.err.extendsList": "extends doit être un nom de job ou une liste",1386
"ci.err.listText": "une liste ne peut contenir que du texte",1387
"ci.err.varText": "les variables doivent être du texte",1388
"ci.err.bool": "true ou false attendu",1389
"ci.err.dotdot": "le chemin %q doit rester dans l'espace de travail",1234
}1390
}@@ -92,18 +92,27 @@ func (r *Runner) schedule(ctx context.Context, p store.Pipeline) {
92
92
specs := make([]ci.Job, len(jobs))
93
93
status := map[string]string{}
94
94
for i, j := range jobs {
95
specs[i] = ci.Job{Name: j.Name, Stage: j.Stage, Needs: j.Needs, NeedsSet: j.NeedsSet, AllowFailure: j.AllowFailure}
95
specs[i] = ci.Job{Name: j.Name, Stage: j.Stage, Needs: j.Needs, Optional: j.Optional, NeedsSet: j.NeedsSet, AllowFailure: j.AllowFailure, When: j.When}
96
96
status[j.Name] = j.Status
97
97
}
98
98
for _, j := range jobs {
99
if j.Status != "queued" {
99
if j.Status != "queued" && j.Status != "manual" {
100
100
continue
101
101
}
102
102
switch ci.Gate(specs, status, specs[indexOf(jobs, j.Name)]) {
103
103
case "skip":
104
_ = r.Store.SkipJob(ctx, j.ID, "overgeslagen omdat een eerdere job mislukte")
104
_ = r.Store.SkipJob(ctx, j.ID, "skipped because an earlier job failed\n")
105
105
status[j.Name] = "skipped"
106
case "fail":
107
_ = r.Store.FinishJob(ctx, j.ID, "failed", "a required job is not part of this pipeline\n")
108
status[j.Name] = "failed"
106
109
case "ready":
110
if j.Status != "queued" {
111
continue
112
}
113
if busy, _ := r.Store.ResourceBusy(ctx, j.RepoID, j.ResourceGroup, j.ID); busy {
114
continue
115
}
107
116
ok, err := r.Store.MarkJobRunning(ctx, j.ID)
108
117
if err != nil || !ok {
109
118
continue
@@ -131,16 +140,14 @@ func (r *Runner) execute(ctx context.Context, p store.Pipeline, job store.Job) {
131
140
defer func() { <-r.slots }()
132
141
}
133
142
before, _ := r.Store.PipelineByID(context.Background(), p.RepoID, p.ID)
134
logText, err := r.runOne(ctx, p, job)
143
mask, err := r.runOne(ctx, p, job)
135
144
status := "success"
145
extra := ""
136
146
if err != nil {
137
147
status = "failed"
138
if logText != "" {
139
logText += "\n"
140
}
141
logText += err.Error() + "\n"
148
extra = stamp(ci.Mask(err.Error(), mask))
142
149
}
143
if ferr := r.Store.FinishJob(context.Background(), job.ID, status, logText); ferr != nil {
150
if ferr := r.Store.FinishJob(context.Background(), job.ID, status, extra); ferr != nil {
144
151
r.Log.Error("job afronden", "err", ferr)
145
152
}
146
153
final, rerr := r.Store.RefreshPipelineStatus(context.Background(), p.ID)
@@ -175,46 +182,66 @@ func (r *Runner) execute(ctx context.Context, p store.Pipeline, job store.Job) {
175
182
}
176
183
}
177
184
178
func (r *Runner) runOne(ctx context.Context, p store.Pipeline, job store.Job) (string, error) {
179
if r.Timeout > 0 {
185
func (r *Runner) runOne(ctx context.Context, p store.Pipeline, job store.Job) ([]string, error) {
186
limit := r.Timeout
187
if job.Timeout > 0 {
188
limit = job.Timeout
189
}
190
if limit > 0 {
180
191
var cancel context.CancelFunc
181
ctx, cancel = context.WithTimeout(ctx, r.Timeout)
192
ctx, cancel = context.WithTimeout(ctx, limit)
182
193
defer cancel()
183
194
}
184
195
repo, err := r.Store.RepoByID(ctx, p.RepoID)
185
196
if err != nil {
186
return "", err
197
return nil, err
187
198
}
199
sink := &logSink{store: r.Store, jobID: job.ID}
200
sink.section("prepare", "Preparing the docker executor")
188
201
work := filepath.Join(r.storage(), "work", fmt.Sprint(p.ID), job.Name)
189
202
_ = os.RemoveAll(work)
190
203
if err := os.MkdirAll(work, 0o755); err != nil {
191
return "", err
204
sink.closeSection()
205
return nil, err
192
206
}
193
207
bare := gitrepo.Dir(r.storage(), repo.OwnerPath, repo.Name)
194
208
if _, err := execOut(ctx, r.GitBin, "clone", "--local", bare, work); err != nil {
195
return "", fmt.Errorf("clone: %w", err)
209
sink.closeSection()
210
return nil, fmt.Errorf("clone: %w", err)
196
211
}
197
212
if _, err := execOut(ctx, r.GitBin, "-C", work, "checkout", "--detach", p.CommitSHA); err != nil {
198
return "", fmt.Errorf("checkout %s: %w", p.CommitSHA, err)
213
sink.closeSection()
214
return nil, fmt.Errorf("checkout %s: %w", p.CommitSHA, err)
215
}
216
if err := r.restoreCache(repo.ID, job, work); err != nil {
217
sink.closeSection()
218
return nil, err
199
219
}
200
220
if err := r.materialize(ctx, p, job, work); err != nil {
201
return "", err
221
sink.closeSection()
222
return nil, err
202
223
}
203
224
scriptPath := filepath.Join(work, ".bluegit-job.sh")
204
if err := os.WriteFile(scriptPath, []byte(scriptOf(job.Script)), 0o755); err != nil {
205
return "", err
225
if err := os.WriteFile(scriptPath, []byte(shellScript(job)), 0o755); err != nil {
226
sink.closeSection()
227
return nil, err
206
228
}
207
229
token, err := r.Store.CreateJobToken(ctx, job.ID)
208
230
if err != nil {
209
return "", err
231
sink.closeSection()
232
return nil, err
210
233
}
211
env, err := r.environment(ctx, repo, p, job, token)
234
env, mask, err := r.environment(ctx, repo, p, job, token)
212
235
if err != nil {
213
return "", err
236
sink.closeSection()
237
return nil, err
214
238
}
239
sink.secrets = mask
215
240
if _, err := exec.LookPath(r.Docker); err != nil {
216
return "", fmt.Errorf("docker (%s) is niet beschikbaar; de job draait alleen in een container", r.Docker)
241
sink.closeSection()
242
return mask, fmt.Errorf("docker (%s) is not available; the job only runs in a container", r.Docker)
217
243
}
244
sink.closeSection()
218
245
// -e NAAM zonder waarde laat Docker de variabele uit de omgeving van
219
246
// deze client overnemen. Zo blijft een meerregelige sleutel heel en
220
247
// staat de waarde niet in de proceslijst.
@@ -227,27 +254,55 @@ func (r *Runner) runOne(ctx context.Context, p store.Pipeline, job store.Job) (s
227
254
args = append(args, "--entrypoint", "sh", job.Image, "/workspace/.bluegit-job.sh")
228
255
cmd := exec.CommandContext(ctx, r.Docker, args...)
229
256
cmd.Env = cmdEnv
230
var buf bytes.Buffer
231
cmd.Stdout = &buf
232
cmd.Stderr = &buf
257
cmd.Stdout = sink
258
cmd.Stderr = sink
233
259
err = cmd.Run()
234
logText := buf.String()
260
sink.flush()
235
261
if err != nil {
236
return logText, fmt.Errorf("container eindigde met een fout: %w", err)
262
return mask, fmt.Errorf("container ended with an error: %w", err)
237
263
}
238
264
if err := r.collect(ctx, job, work); err != nil {
239
return logText, err
265
return mask, err
266
}
267
if err := r.saveCache(repo.ID, job, work); err != nil {
268
return mask, err
240
269
}
241
return logText, nil
270
return mask, nil
242
271
}
243
272
244
func scriptOf(lines string) string {
273
func shellScript(job store.Job) string {
245
274
var b strings.Builder
246
b.WriteString("#!/bin/sh\nset -e\ncd /workspace\n")
247
b.WriteString(lines)
248
if !strings.HasSuffix(lines, "\n") {
275
b.WriteString("#!/bin/sh\ncd /workspace\nrc=0\n")
276
fmt.Fprintf(&b, "echo \"section_start:before:Preparing environment\"\n")
277
if strings.TrimSpace(job.Before) != "" {
278
b.WriteString("(\nset -e\n")
279
b.WriteString(job.Before)
280
if !strings.HasSuffix(job.Before, "\n") {
281
b.WriteByte('\n')
282
}
283
b.WriteString(")\nbrc=$?\n")
284
} else {
285
b.WriteString("brc=0\n")
286
}
287
b.WriteString("echo \"section_end:before:Preparing environment\"\n")
288
b.WriteString("echo \"section_start:script:Running script\"\n")
289
b.WriteString("if [ \"$brc\" -eq 0 ]; then\n(\nset -e\n")
290
b.WriteString(job.Script)
291
if !strings.HasSuffix(job.Script, "\n") {
249
292
b.WriteByte('\n')
250
293
}
294
b.WriteString(")\nrc=$?\nelse\nrc=$brc\nfi\n")
295
b.WriteString("echo \"section_end:script:Running script\"\n")
296
b.WriteString("echo \"section_start:after:After script\"\n")
297
if strings.TrimSpace(job.After) != "" {
298
b.WriteString("(\nset -e\n")
299
b.WriteString(job.After)
300
if !strings.HasSuffix(job.After, "\n") {
301
b.WriteByte('\n')
302
}
303
b.WriteString(")\n")
304
}
305
b.WriteString("echo \"section_end:after:After script\"\nexit \"$rc\"\n")
251
306
return b.String()
252
307
}
253
308
@@ -318,44 +373,174 @@ func (r *Runner) collect(ctx context.Context, job store.Job, work string) error
318
373
return nil
319
374
}
320
375
321
func (r *Runner) environment(ctx context.Context, repo store.Repo, p store.Pipeline, job store.Job, token string) (map[string]string, error) {
322
branch := strings.TrimPrefix(p.Ref, "refs/heads/")
376
func (r *Runner) environment(ctx context.Context, repo store.Repo, p store.Pipeline, job store.Job, token string) (map[string]string, []string, error) {
377
name, _ := ci.SplitRef(p.Ref)
323
378
patterns, err := r.Store.ProtectedPatterns(ctx, repo.ID)
324
379
if err != nil {
325
return nil, err
380
return nil, nil, err
326
381
}
327
refProtected := access.BranchProtected(patterns, branch)
382
refProtected := access.BranchProtected(patterns, name)
328
383
env := map[string]string{}
384
var mask []string
329
385
if repo.OwnerKind == "group" {
330
386
groupSecrets, err := r.Store.SecretValues(ctx, "group", repo.OwnerID, refProtected)
331
387
if err != nil {
332
return nil, err
388
return nil, nil, err
333
389
}
334
390
for k, v := range groupSecrets {
335
391
env[k] = v
392
mask = append(mask, v)
336
393
}
337
394
}
338
395
repoSecrets, err := r.Store.SecretValues(ctx, "repo", repo.ID, refProtected)
339
396
if err != nil {
340
return nil, err
397
return nil, nil, err
341
398
}
342
399
for k, v := range repoSecrets {
343
400
env[k] = v
401
mask = append(mask, v)
344
402
}
345
403
for k, v := range job.Variables {
346
404
env[k] = v
347
405
}
348
env["CI"] = "true"
349
env["CI_COMMIT_SHA"] = p.CommitSHA
350
env["CI_COMMIT_BRANCH"] = branch
351
env["CI_COMMIT_REF"] = p.Ref
352
env["CI_PROJECT_PATH"] = repo.FullPath()
353
env["CI_PIPELINE_ID"] = fmt.Sprint(p.ID)
406
base := strings.TrimRight(r.BaseURL, "/")
407
pre := ci.Variables(ci.Ref{
408
SHA: p.CommitSHA, FullRef: p.Ref, Title: p.CommitTitle,
409
DefaultBranch: repo.DefaultBranch, Source: p.Source,
410
ProjectPath: repo.FullPath(), ProjectURL: base + "/" + repo.FullPath(),
411
APIURL: base + "/api/v1", PipelineID: p.ID,
412
})
413
for k, v := range pre {
414
env[k] = v
415
}
354
416
env["CI_JOB_ID"] = fmt.Sprint(job.ID)
355
417
env["CI_JOB_NAME"] = job.Name
418
env["CI_JOB_STAGE"] = job.Stage
419
env["CI_JOB_URL"] = base + "/" + repo.FullPath() + "/-/pipelines/" + fmt.Sprint(p.ID) + "/jobs/" + fmt.Sprint(job.ID)
356
420
env["CI_JOB_TOKEN"] = token
357
env["CI_API_URL"] = strings.TrimRight(r.BaseURL, "/") + "/api/v1"
358
return env, nil
421
mask = append(mask, token)
422
return env, mask, nil
423
}
424
425
func (r *Runner) restoreCache(repoID int64, job store.Job, work string) error {
426
if len(job.CachePaths) == 0 {
427
return nil
428
}
429
root := r.cacheDir(repoID, job.CacheKey)
430
for _, rel := range job.CachePaths {
431
src := filepath.Join(root, filepath.FromSlash(rel))
432
if _, err := os.Stat(src); err != nil {
433
continue
434
}
435
if err := copyFile(src, filepath.Join(work, filepath.FromSlash(rel))); err != nil {
436
return err
437
}
438
}
439
return nil
440
}
441
442
func (r *Runner) saveCache(repoID int64, job store.Job, work string) error {
443
if len(job.CachePaths) == 0 {
444
return nil
445
}
446
root := r.cacheDir(repoID, job.CacheKey)
447
for _, rel := range job.CachePaths {
448
src := filepath.Join(work, filepath.FromSlash(rel))
449
info, err := os.Stat(src)
450
if err != nil || info.IsDir() {
451
continue
452
}
453
if err := copyFile(src, filepath.Join(root, filepath.FromSlash(rel))); err != nil {
454
return err
455
}
456
}
457
return nil
458
}
459
460
func (r *Runner) cacheDir(repoID int64, key string) string {
461
if key == "" {
462
key = "default"
463
}
464
var b strings.Builder
465
for _, r := range key {
466
if (r >= 'a' && r <= 'z') || (r >= 'A' && r <= 'Z') || (r >= '0' && r <= '9') || r == '.' || r == '_' || r == '-' {
467
b.WriteRune(r)
468
}
469
}
470
safe := b.String()
471
if safe == "" {
472
safe = "default"
473
}
474
return filepath.Join(r.storage(), "cache", fmt.Sprint(repoID), safe)
475
}
476
477
func stamp(line string) string {
478
return time.Now().UTC().Format(time.RFC3339) + "\t" + line + "\n"
479
}
480
481
type logSink struct {
482
mu sync.Mutex
483
store *store.Store
484
jobID int64
485
secrets []string
486
partial string
487
current string
488
opened time.Time
489
}
490
491
func (l *logSink) Write(p []byte) (int, error) {
492
l.mu.Lock()
493
defer l.mu.Unlock()
494
l.partial += string(p)
495
for {
496
i := strings.IndexByte(l.partial, '\n')
497
if i < 0 {
498
break
499
}
500
l.emit(strings.TrimRight(l.partial[:i], "\r"))
501
l.partial = l.partial[i+1:]
502
}
503
return len(p), nil
504
}
505
506
func (l *logSink) flush() {
507
l.mu.Lock()
508
defer l.mu.Unlock()
509
if strings.TrimSpace(l.partial) != "" {
510
l.emit(l.partial)
511
}
512
l.partial = ""
513
}
514
515
func (l *logSink) emit(line string) {
516
line = ci.Mask(line, l.secrets)
517
if strings.HasPrefix(line, "section_start:") || strings.HasPrefix(line, "section_end:") {
518
if l.current != "" && strings.HasPrefix(line, "section_end:") {
519
dur := int(time.Since(l.opened).Seconds())
520
line = line + "\t" + fmt.Sprintf("%02d:%02d", dur/60, dur%60)
521
l.current = ""
522
}
523
if strings.HasPrefix(line, "section_start:") {
524
l.current = line
525
l.opened = time.Now()
526
}
527
}
528
_ = l.store.AppendJobLog(context.Background(), l.jobID, stamp(line))
529
}
530
531
func (l *logSink) section(name, title string) {
532
l.emit("section_start:" + name + ":" + title)
533
}
534
535
func (l *logSink) closeSection() {
536
if l.current == "" {
537
return
538
}
539
name := "prepare"
540
if parts := strings.SplitN(strings.TrimPrefix(l.current, "section_start:"), ":", 2); len(parts) > 0 && parts[0] != "" {
541
name = parts[0]
542
}
543
l.emit("section_end:" + name + ":done")
359
544
}
360
545
361
546
func copyFile(src, dst string) error {
@@ -92,18 +92,27 @@ func (r *Runner) schedule(ctx context.Context, p store.Pipeline) {92
specs := make([]ci.Job, len(jobs))92
specs := make([]ci.Job, len(jobs))93
status := map[string]string{}93
status := map[string]string{}94
for i, j := range jobs {94
for i, j := range jobs {95
specs[i] = ci.Job{Name: j.Name, Stage: j.Stage, Needs: j.Needs, NeedsSet: j.NeedsSet, AllowFailure: j.AllowFailure}95
specs[i] = ci.Job{Name: j.Name, Stage: j.Stage, Needs: j.Needs, Optional: j.Optional, NeedsSet: j.NeedsSet, AllowFailure: j.AllowFailure, When: j.When}96
status[j.Name] = j.Status96
status[j.Name] = j.Status97
}97
}98
for _, j := range jobs {98
for _, j := range jobs {99
if j.Status != "queued" {99
if j.Status != "queued" && j.Status != "manual" {100
continue100
continue101
}101
}102
switch ci.Gate(specs, status, specs[indexOf(jobs, j.Name)]) {102
switch ci.Gate(specs, status, specs[indexOf(jobs, j.Name)]) {103
case "skip":103
case "skip":104
_ = r.Store.SkipJob(ctx, j.ID, "overgeslagen omdat een eerdere job mislukte")104
_ = r.Store.SkipJob(ctx, j.ID, "skipped because an earlier job failed\n")105
status[j.Name] = "skipped"105
status[j.Name] = "skipped"106
case "fail":107
_ = r.Store.FinishJob(ctx, j.ID, "failed", "a required job is not part of this pipeline\n")108
status[j.Name] = "failed"106
case "ready":109
case "ready":110
if j.Status != "queued" {111
continue112
}113
if busy, _ := r.Store.ResourceBusy(ctx, j.RepoID, j.ResourceGroup, j.ID); busy {114
continue115
}107
ok, err := r.Store.MarkJobRunning(ctx, j.ID)116
ok, err := r.Store.MarkJobRunning(ctx, j.ID)108
if err != nil || !ok {117
if err != nil || !ok {109
continue118
continue@@ -131,16 +140,14 @@ func (r *Runner) execute(ctx context.Context, p store.Pipeline, job store.Job) {131
defer func() { <-r.slots }()140
defer func() { <-r.slots }()132
}141
}133
before, _ := r.Store.PipelineByID(context.Background(), p.RepoID, p.ID)142
before, _ := r.Store.PipelineByID(context.Background(), p.RepoID, p.ID)134
logText, err := r.runOne(ctx, p, job)143
mask, err := r.runOne(ctx, p, job)135
status := "success"144
status := "success"145
extra := ""136
if err != nil {146
if err != nil {137
status = "failed"147
status = "failed"138
if logText != "" {148
extra = stamp(ci.Mask(err.Error(), mask))139
logText += "\n"140
}141
logText += err.Error() + "\n"142
}149
}143
if ferr := r.Store.FinishJob(context.Background(), job.ID, status, logText); ferr != nil {150
if ferr := r.Store.FinishJob(context.Background(), job.ID, status, extra); ferr != nil {144
r.Log.Error("job afronden", "err", ferr)151
r.Log.Error("job afronden", "err", ferr)145
}152
}146
final, rerr := r.Store.RefreshPipelineStatus(context.Background(), p.ID)153
final, rerr := r.Store.RefreshPipelineStatus(context.Background(), p.ID)@@ -175,46 +182,66 @@ func (r *Runner) execute(ctx context.Context, p store.Pipeline, job store.Job) {175
}182
}176
}183
}177
184
178
func (r *Runner) runOne(ctx context.Context, p store.Pipeline, job store.Job) (string, error) {185
func (r *Runner) runOne(ctx context.Context, p store.Pipeline, job store.Job) ([]string, error) {179
if r.Timeout > 0 {186
limit := r.Timeout187
if job.Timeout > 0 {188
limit = job.Timeout189
}190
if limit > 0 {180
var cancel context.CancelFunc191
var cancel context.CancelFunc181
ctx, cancel = context.WithTimeout(ctx, r.Timeout)192
ctx, cancel = context.WithTimeout(ctx, limit)182
defer cancel()193
defer cancel()183
}194
}184
repo, err := r.Store.RepoByID(ctx, p.RepoID)195
repo, err := r.Store.RepoByID(ctx, p.RepoID)185
if err != nil {196
if err != nil {186
return "", err197
return nil, err187
}198
}199
sink := &logSink{store: r.Store, jobID: job.ID}200
sink.section("prepare", "Preparing the docker executor")188
work := filepath.Join(r.storage(), "work", fmt.Sprint(p.ID), job.Name)201
work := filepath.Join(r.storage(), "work", fmt.Sprint(p.ID), job.Name)189
_ = os.RemoveAll(work)202
_ = os.RemoveAll(work)190
if err := os.MkdirAll(work, 0o755); err != nil {203
if err := os.MkdirAll(work, 0o755); err != nil {191
return "", err204
sink.closeSection()205
return nil, err192
}206
}193
bare := gitrepo.Dir(r.storage(), repo.OwnerPath, repo.Name)207
bare := gitrepo.Dir(r.storage(), repo.OwnerPath, repo.Name)194
if _, err := execOut(ctx, r.GitBin, "clone", "--local", bare, work); err != nil {208
if _, err := execOut(ctx, r.GitBin, "clone", "--local", bare, work); err != nil {195
return "", fmt.Errorf("clone: %w", err)209
sink.closeSection()210
return nil, fmt.Errorf("clone: %w", err)196
}211
}197
if _, err := execOut(ctx, r.GitBin, "-C", work, "checkout", "--detach", p.CommitSHA); err != nil {212
if _, err := execOut(ctx, r.GitBin, "-C", work, "checkout", "--detach", p.CommitSHA); err != nil {198
return "", fmt.Errorf("checkout %s: %w", p.CommitSHA, err)213
sink.closeSection()214
return nil, fmt.Errorf("checkout %s: %w", p.CommitSHA, err)215
}216
if err := r.restoreCache(repo.ID, job, work); err != nil {217
sink.closeSection()218
return nil, err199
}219
}200
if err := r.materialize(ctx, p, job, work); err != nil {220
if err := r.materialize(ctx, p, job, work); err != nil {201
return "", err221
sink.closeSection()222
return nil, err202
}223
}203
scriptPath := filepath.Join(work, ".bluegit-job.sh")224
scriptPath := filepath.Join(work, ".bluegit-job.sh")204
if err := os.WriteFile(scriptPath, []byte(scriptOf(job.Script)), 0o755); err != nil {225
if err := os.WriteFile(scriptPath, []byte(shellScript(job)), 0o755); err != nil {205
return "", err226
sink.closeSection()227
return nil, err206
}228
}207
token, err := r.Store.CreateJobToken(ctx, job.ID)229
token, err := r.Store.CreateJobToken(ctx, job.ID)208
if err != nil {230
if err != nil {209
return "", err231
sink.closeSection()232
return nil, err210
}233
}211
env, err := r.environment(ctx, repo, p, job, token)234
env, mask, err := r.environment(ctx, repo, p, job, token)212
if err != nil {235
if err != nil {213
return "", err236
sink.closeSection()237
return nil, err214
}238
}239
sink.secrets = mask215
if _, err := exec.LookPath(r.Docker); err != nil {240
if _, err := exec.LookPath(r.Docker); err != nil {216
return "", fmt.Errorf("docker (%s) is niet beschikbaar; de job draait alleen in een container", r.Docker)241
sink.closeSection()242
return mask, fmt.Errorf("docker (%s) is not available; the job only runs in a container", r.Docker)217
}243
}244
sink.closeSection()218
// -e NAAM zonder waarde laat Docker de variabele uit de omgeving van245
// -e NAAM zonder waarde laat Docker de variabele uit de omgeving van219
// deze client overnemen. Zo blijft een meerregelige sleutel heel en246
// deze client overnemen. Zo blijft een meerregelige sleutel heel en220
// staat de waarde niet in de proceslijst.247
// staat de waarde niet in de proceslijst.@@ -227,27 +254,55 @@ func (r *Runner) runOne(ctx context.Context, p store.Pipeline, job store.Job) (s227
args = append(args, "--entrypoint", "sh", job.Image, "/workspace/.bluegit-job.sh")254
args = append(args, "--entrypoint", "sh", job.Image, "/workspace/.bluegit-job.sh")228
cmd := exec.CommandContext(ctx, r.Docker, args...)255
cmd := exec.CommandContext(ctx, r.Docker, args...)229
cmd.Env = cmdEnv256
cmd.Env = cmdEnv230
var buf bytes.Buffer257
cmd.Stdout = sink231
cmd.Stdout = &buf258
cmd.Stderr = sink232
cmd.Stderr = &buf233
err = cmd.Run()259
err = cmd.Run()234
logText := buf.String()260
sink.flush()235
if err != nil {261
if err != nil {236
return logText, fmt.Errorf("container eindigde met een fout: %w", err)262
return mask, fmt.Errorf("container ended with an error: %w", err)237
}263
}238
if err := r.collect(ctx, job, work); err != nil {264
if err := r.collect(ctx, job, work); err != nil {239
return logText, err265
return mask, err266
}267
if err := r.saveCache(repo.ID, job, work); err != nil {268
return mask, err240
}269
}241
return logText, nil270
return mask, nil242
}271
}243
272
244
func scriptOf(lines string) string {273
func shellScript(job store.Job) string {245
var b strings.Builder274
var b strings.Builder246
b.WriteString("#!/bin/sh\nset -e\ncd /workspace\n")275
b.WriteString("#!/bin/sh\ncd /workspace\nrc=0\n")247
b.WriteString(lines)276
fmt.Fprintf(&b, "echo \"section_start:before:Preparing environment\"\n")248
if !strings.HasSuffix(lines, "\n") {277
if strings.TrimSpace(job.Before) != "" {278
b.WriteString("(\nset -e\n")279
b.WriteString(job.Before)280
if !strings.HasSuffix(job.Before, "\n") {281
b.WriteByte('\n')282
}283
b.WriteString(")\nbrc=$?\n")284
} else {285
b.WriteString("brc=0\n")286
}287
b.WriteString("echo \"section_end:before:Preparing environment\"\n")288
b.WriteString("echo \"section_start:script:Running script\"\n")289
b.WriteString("if [ \"$brc\" -eq 0 ]; then\n(\nset -e\n")290
b.WriteString(job.Script)291
if !strings.HasSuffix(job.Script, "\n") {249
b.WriteByte('\n')292
b.WriteByte('\n')250
}293
}294
b.WriteString(")\nrc=$?\nelse\nrc=$brc\nfi\n")295
b.WriteString("echo \"section_end:script:Running script\"\n")296
b.WriteString("echo \"section_start:after:After script\"\n")297
if strings.TrimSpace(job.After) != "" {298
b.WriteString("(\nset -e\n")299
b.WriteString(job.After)300
if !strings.HasSuffix(job.After, "\n") {301
b.WriteByte('\n')302
}303
b.WriteString(")\n")304
}305
b.WriteString("echo \"section_end:after:After script\"\nexit \"$rc\"\n")251
return b.String()306
return b.String()252
}307
}253
308
@@ -318,44 +373,174 @@ func (r *Runner) collect(ctx context.Context, job store.Job, work string) error318
return nil373
return nil319
}374
}320
375
321
func (r *Runner) environment(ctx context.Context, repo store.Repo, p store.Pipeline, job store.Job, token string) (map[string]string, error) {376
func (r *Runner) environment(ctx context.Context, repo store.Repo, p store.Pipeline, job store.Job, token string) (map[string]string, []string, error) {322
branch := strings.TrimPrefix(p.Ref, "refs/heads/")377
name, _ := ci.SplitRef(p.Ref)323
patterns, err := r.Store.ProtectedPatterns(ctx, repo.ID)378
patterns, err := r.Store.ProtectedPatterns(ctx, repo.ID)324
if err != nil {379
if err != nil {325
return nil, err380
return nil, nil, err326
}381
}327
refProtected := access.BranchProtected(patterns, branch)382
refProtected := access.BranchProtected(patterns, name)328
env := map[string]string{}383
env := map[string]string{}384
var mask []string329
if repo.OwnerKind == "group" {385
if repo.OwnerKind == "group" {330
groupSecrets, err := r.Store.SecretValues(ctx, "group", repo.OwnerID, refProtected)386
groupSecrets, err := r.Store.SecretValues(ctx, "group", repo.OwnerID, refProtected)331
if err != nil {387
if err != nil {332
return nil, err388
return nil, nil, err333
}389
}334
for k, v := range groupSecrets {390
for k, v := range groupSecrets {335
env[k] = v391
env[k] = v392
mask = append(mask, v)336
}393
}337
}394
}338
repoSecrets, err := r.Store.SecretValues(ctx, "repo", repo.ID, refProtected)395
repoSecrets, err := r.Store.SecretValues(ctx, "repo", repo.ID, refProtected)339
if err != nil {396
if err != nil {340
return nil, err397
return nil, nil, err341
}398
}342
for k, v := range repoSecrets {399
for k, v := range repoSecrets {343
env[k] = v400
env[k] = v401
mask = append(mask, v)344
}402
}345
for k, v := range job.Variables {403
for k, v := range job.Variables {346
env[k] = v404
env[k] = v347
}405
}348
env["CI"] = "true"406
base := strings.TrimRight(r.BaseURL, "/")349
env["CI_COMMIT_SHA"] = p.CommitSHA407
pre := ci.Variables(ci.Ref{350
env["CI_COMMIT_BRANCH"] = branch408
SHA: p.CommitSHA, FullRef: p.Ref, Title: p.CommitTitle,351
env["CI_COMMIT_REF"] = p.Ref409
DefaultBranch: repo.DefaultBranch, Source: p.Source,352
env["CI_PROJECT_PATH"] = repo.FullPath()410
ProjectPath: repo.FullPath(), ProjectURL: base + "/" + repo.FullPath(),353
env["CI_PIPELINE_ID"] = fmt.Sprint(p.ID)411
APIURL: base + "/api/v1", PipelineID: p.ID,412
})413
for k, v := range pre {414
env[k] = v415
}354
env["CI_JOB_ID"] = fmt.Sprint(job.ID)416
env["CI_JOB_ID"] = fmt.Sprint(job.ID)355
env["CI_JOB_NAME"] = job.Name417
env["CI_JOB_NAME"] = job.Name418
env["CI_JOB_STAGE"] = job.Stage419
env["CI_JOB_URL"] = base + "/" + repo.FullPath() + "/-/pipelines/" + fmt.Sprint(p.ID) + "/jobs/" + fmt.Sprint(job.ID)356
env["CI_JOB_TOKEN"] = token420
env["CI_JOB_TOKEN"] = token357
env["CI_API_URL"] = strings.TrimRight(r.BaseURL, "/") + "/api/v1"421
mask = append(mask, token)358
return env, nil422
return env, mask, nil423
}424
425
func (r *Runner) restoreCache(repoID int64, job store.Job, work string) error {426
if len(job.CachePaths) == 0 {427
return nil428
}429
root := r.cacheDir(repoID, job.CacheKey)430
for _, rel := range job.CachePaths {431
src := filepath.Join(root, filepath.FromSlash(rel))432
if _, err := os.Stat(src); err != nil {433
continue434
}435
if err := copyFile(src, filepath.Join(work, filepath.FromSlash(rel))); err != nil {436
return err437
}438
}439
return nil440
}441
442
func (r *Runner) saveCache(repoID int64, job store.Job, work string) error {443
if len(job.CachePaths) == 0 {444
return nil445
}446
root := r.cacheDir(repoID, job.CacheKey)447
for _, rel := range job.CachePaths {448
src := filepath.Join(work, filepath.FromSlash(rel))449
info, err := os.Stat(src)450
if err != nil || info.IsDir() {451
continue452
}453
if err := copyFile(src, filepath.Join(root, filepath.FromSlash(rel))); err != nil {454
return err455
}456
}457
return nil458
}459
460
func (r *Runner) cacheDir(repoID int64, key string) string {461
if key == "" {462
key = "default"463
}464
var b strings.Builder465
for _, r := range key {466
if (r >= 'a' && r <= 'z') || (r >= 'A' && r <= 'Z') || (r >= '0' && r <= '9') || r == '.' || r == '_' || r == '-' {467
b.WriteRune(r)468
}469
}470
safe := b.String()471
if safe == "" {472
safe = "default"473
}474
return filepath.Join(r.storage(), "cache", fmt.Sprint(repoID), safe)475
}476
477
func stamp(line string) string {478
return time.Now().UTC().Format(time.RFC3339) + "\t" + line + "\n"479
}480
481
type logSink struct {482
mu sync.Mutex483
store *store.Store484
jobID int64485
secrets []string486
partial string487
current string488
opened time.Time489
}490
491
func (l *logSink) Write(p []byte) (int, error) {492
l.mu.Lock()493
defer l.mu.Unlock()494
l.partial += string(p)495
for {496
i := strings.IndexByte(l.partial, '\n')497
if i < 0 {498
break499
}500
l.emit(strings.TrimRight(l.partial[:i], "\r"))501
l.partial = l.partial[i+1:]502
}503
return len(p), nil504
}505
506
func (l *logSink) flush() {507
l.mu.Lock()508
defer l.mu.Unlock()509
if strings.TrimSpace(l.partial) != "" {510
l.emit(l.partial)511
}512
l.partial = ""513
}514
515
func (l *logSink) emit(line string) {516
line = ci.Mask(line, l.secrets)517
if strings.HasPrefix(line, "section_start:") || strings.HasPrefix(line, "section_end:") {518
if l.current != "" && strings.HasPrefix(line, "section_end:") {519
dur := int(time.Since(l.opened).Seconds())520
line = line + "\t" + fmt.Sprintf("%02d:%02d", dur/60, dur%60)521
l.current = ""522
}523
if strings.HasPrefix(line, "section_start:") {524
l.current = line525
l.opened = time.Now()526
}527
}528
_ = l.store.AppendJobLog(context.Background(), l.jobID, stamp(line))529
}530
531
func (l *logSink) section(name, title string) {532
l.emit("section_start:" + name + ":" + title)533
}534
535
func (l *logSink) closeSection() {536
if l.current == "" {537
return538
}539
name := "prepare"540
if parts := strings.SplitN(strings.TrimPrefix(l.current, "section_start:"), ":", 2); len(parts) > 0 && parts[0] != "" {541
name = parts[0]542
}543
l.emit("section_end:" + name + ":done")359
}544
}360
545
361
func copyFile(src, dst string) error {546
func copyFile(src, dst string) error {@@ -28,12 +28,19 @@ func (s *Server) AfterPush(repo store.Repo, user store.User, before, after map[s
28
28
if before[ref] == sha {
29
29
continue
30
30
}
31
if isZero(sha) {
32
continue
33
}
34
if strings.HasPrefix(ref, "refs/tags/") {
35
s.startPipeline(context.Background(), repo, sha, ref, actor, "push", before[ref])
36
continue
37
}
31
38
branch, ok := gitrepo.BranchOfRef(ref)
32
if !ok || isZero(sha) {
39
if !ok {
33
40
continue
34
41
}
35
42
s.recordPushContributions(repo, user, map[string]string{ref: before[ref]}, map[string]string{ref: sha})
36
s.startPipeline(context.Background(), repo, sha, ref, actor)
43
s.startPipeline(context.Background(), repo, sha, ref, actor, "push", before[ref])
37
44
web := strings.TrimRight(s.Cfg.BaseURL, "/") + "/" + repo.FullPath()
38
45
s.Dispatch(repo.ID, webhook.Event{
39
46
Type: webhook.EventPush,
@@ -51,7 +58,7 @@ func isZero(sha string) bool {
51
58
return strings.Trim(sha, "0") == ""
52
59
}
53
60
54
func (s *Server) startPipeline(ctx context.Context, repo store.Repo, sha, ref, actor string) {
61
func (s *Server) startPipeline(ctx context.Context, repo store.Repo, sha, ref, actor, source, before string) {
55
62
if !s.Cfg.CI.Enabled || s.StoragePath() == "" {
56
63
return
57
64
}
@@ -60,17 +67,26 @@ func (s *Server) startPipeline(ctx context.Context, repo store.Repo, sha, ref, a
60
67
if err != nil {
61
68
return
62
69
}
63
spec, err := ci.Parse(raw)
64
branch, _ := gitrepo.BranchOfRef(ref)
65
if branch == "" {
66
branch = ref
70
read := func(path string) ([]byte, error) {
71
if path == "" || strings.Contains(path, "..") || strings.HasPrefix(path, "/") {
72
return nil, i18n.E("ci.err.rel", path)
73
}
74
return exec.CommandContext(ctx, s.Git, "-C", dir, "show", sha+":"+path).Output()
67
75
}
76
spec, err := ci.ParseIn(raw, read)
68
77
if err != nil {
69
_ = s.Store.RecordConfigFailure(ctx, repo.ID, sha, ref, "blue-ci-cd.yml: "+err.Error())
78
_ = s.Store.RecordConfigFailure(ctx, repo.ID, sha, ref, err.Error())
70
79
return
71
80
}
72
env := map[string]string{"CI_COMMIT_BRANCH": branch, "CI_COMMIT_SHA": sha, "CI_COMMIT_REF": ref}
73
active, err := spec.ActiveJobs(env)
81
name, kind := ci.SplitRef(ref)
82
title := strings.TrimSpace(gitOne(ctx, s.Git, dir, "log", "-1", "--format=%s", sha))
83
files := gitLines(ctx, s.Git, dir, "ls-tree", "-r", "--name-only", sha)
84
changed := changedFiles(ctx, s.Git, dir, sha, before, files)
85
env := ci.Variables(ci.Ref{
86
SHA: sha, FullRef: ref, Name: name, Kind: kind, Title: title,
87
DefaultBranch: repo.DefaultBranch, Source: source, ProjectPath: repo.FullPath(),
88
})
89
active, err := spec.Select(ci.Eval{Env: env, Changed: changed, Files: files})
74
90
if err != nil {
75
91
_ = s.Store.RecordConfigFailure(ctx, repo.ID, sha, ref, err.Error())
76
92
return
@@ -97,15 +113,51 @@ func (s *Server) startPipeline(ctx context.Context, repo store.Repo, sha, ref, a
97
113
}
98
114
jobs = append(jobs, store.NewJob{
99
115
Name: j.Name, Stage: j.Stage, Position: i, Image: j.Image,
100
Script: strings.Join(j.Script, "\n"), Needs: needs, NeedsSet: j.NeedsSet,
116
Before: strings.Join(j.Before, "\n"), Script: strings.Join(j.Script, "\n"), After: strings.Join(j.After, "\n"),
117
Needs: needs, Optional: j.Optional, NeedsSet: j.NeedsSet,
101
118
Variables: vars, ArtifactPaths: paths, ArtifactExpire: j.Artifacts.ExpireIn,
102
AllowFailure: j.AllowFailure,
119
AllowFailure: j.AllowFailure, When: j.When, Timeout: j.Timeout, Tags: j.Tags,
120
CachePaths: j.Cache.Paths, CacheKey: j.Cache.Key, ResourceGroup: j.ResourceGroup,
103
121
})
104
122
}
105
if _, err := s.Store.CreatePipeline(ctx, repo.ID, sha, ref, jobs); err != nil && s.Log != nil {
123
if _, err := s.Store.CreatePipeline(ctx, repo.ID, sha, ref, source, actor, title, jobs); err != nil && s.Log != nil {
106
124
s.Log.Error("pipeline", "err", err)
107
125
}
108
_ = actor
126
}
127
128
func gitOne(ctx context.Context, gitBin, dir string, args ...string) string {
129
out, err := exec.CommandContext(ctx, gitBin, append([]string{"-C", dir}, args...)...).Output()
130
if err != nil {
131
return ""
132
}
133
return strings.TrimSpace(string(out))
134
}
135
136
func gitLines(ctx context.Context, gitBin, dir string, args ...string) []string {
137
out := gitOne(ctx, gitBin, dir, args...)
138
if out == "" {
139
return []string{}
140
}
141
return strings.Split(out, "\n")
142
}
143
144
func changedFiles(ctx context.Context, gitBin, dir, sha, before string, all []string) []string {
145
if before == "" || strings.Trim(before, "0") == "" {
146
parent := gitOne(ctx, gitBin, dir, "rev-parse", "--verify", sha+"^")
147
if parent == "" {
148
return all
149
}
150
before = parent
151
}
152
out, err := exec.CommandContext(ctx, gitBin, "-C", dir, "diff", "--name-only", before, sha).Output()
153
if err != nil {
154
return nil
155
}
156
text := strings.TrimSpace(string(out))
157
if text == "" {
158
return []string{}
159
}
160
return strings.Split(text, "\n")
109
161
}
110
162
111
163
func pushEvent(repo store.Repo, kind, branch, commit, actor, summary, path string) webhook.Event {
@@ -28,12 +28,19 @@ func (s *Server) AfterPush(repo store.Repo, user store.User, before, after map[s28
if before[ref] == sha {28
if before[ref] == sha {29
continue29
continue30
}30
}31
if isZero(sha) {32
continue33
}34
if strings.HasPrefix(ref, "refs/tags/") {35
s.startPipeline(context.Background(), repo, sha, ref, actor, "push", before[ref])36
continue37
}31
branch, ok := gitrepo.BranchOfRef(ref)38
branch, ok := gitrepo.BranchOfRef(ref)32
if !ok || isZero(sha) {39
if !ok {33
continue40
continue34
}41
}35
s.recordPushContributions(repo, user, map[string]string{ref: before[ref]}, map[string]string{ref: sha})42
s.recordPushContributions(repo, user, map[string]string{ref: before[ref]}, map[string]string{ref: sha})36
s.startPipeline(context.Background(), repo, sha, ref, actor)43
s.startPipeline(context.Background(), repo, sha, ref, actor, "push", before[ref])37
web := strings.TrimRight(s.Cfg.BaseURL, "/") + "/" + repo.FullPath()44
web := strings.TrimRight(s.Cfg.BaseURL, "/") + "/" + repo.FullPath()38
s.Dispatch(repo.ID, webhook.Event{45
s.Dispatch(repo.ID, webhook.Event{39
Type: webhook.EventPush,46
Type: webhook.EventPush,@@ -51,7 +58,7 @@ func isZero(sha string) bool {51
return strings.Trim(sha, "0") == ""58
return strings.Trim(sha, "0") == ""52
}59
}53
60
54
func (s *Server) startPipeline(ctx context.Context, repo store.Repo, sha, ref, actor string) {61
func (s *Server) startPipeline(ctx context.Context, repo store.Repo, sha, ref, actor, source, before string) {55
if !s.Cfg.CI.Enabled || s.StoragePath() == "" {62
if !s.Cfg.CI.Enabled || s.StoragePath() == "" {56
return63
return57
}64
}@@ -60,17 +67,26 @@ func (s *Server) startPipeline(ctx context.Context, repo store.Repo, sha, ref, a60
if err != nil {67
if err != nil {61
return68
return62
}69
}63
spec, err := ci.Parse(raw)70
read := func(path string) ([]byte, error) {64
branch, _ := gitrepo.BranchOfRef(ref)71
if path == "" || strings.Contains(path, "..") || strings.HasPrefix(path, "/") {65
if branch == "" {72
return nil, i18n.E("ci.err.rel", path)66
branch = ref73
}74
return exec.CommandContext(ctx, s.Git, "-C", dir, "show", sha+":"+path).Output()67
}75
}76
spec, err := ci.ParseIn(raw, read)68
if err != nil {77
if err != nil {69
_ = s.Store.RecordConfigFailure(ctx, repo.ID, sha, ref, "blue-ci-cd.yml: "+err.Error())78
_ = s.Store.RecordConfigFailure(ctx, repo.ID, sha, ref, err.Error())70
return79
return71
}80
}72
env := map[string]string{"CI_COMMIT_BRANCH": branch, "CI_COMMIT_SHA": sha, "CI_COMMIT_REF": ref}81
name, kind := ci.SplitRef(ref)73
active, err := spec.ActiveJobs(env)82
title := strings.TrimSpace(gitOne(ctx, s.Git, dir, "log", "-1", "--format=%s", sha))83
files := gitLines(ctx, s.Git, dir, "ls-tree", "-r", "--name-only", sha)84
changed := changedFiles(ctx, s.Git, dir, sha, before, files)85
env := ci.Variables(ci.Ref{86
SHA: sha, FullRef: ref, Name: name, Kind: kind, Title: title,87
DefaultBranch: repo.DefaultBranch, Source: source, ProjectPath: repo.FullPath(),88
})89
active, err := spec.Select(ci.Eval{Env: env, Changed: changed, Files: files})74
if err != nil {90
if err != nil {75
_ = s.Store.RecordConfigFailure(ctx, repo.ID, sha, ref, err.Error())91
_ = s.Store.RecordConfigFailure(ctx, repo.ID, sha, ref, err.Error())76
return92
return@@ -97,15 +113,51 @@ func (s *Server) startPipeline(ctx context.Context, repo store.Repo, sha, ref, a97
}113
}98
jobs = append(jobs, store.NewJob{114
jobs = append(jobs, store.NewJob{99
Name: j.Name, Stage: j.Stage, Position: i, Image: j.Image,115
Name: j.Name, Stage: j.Stage, Position: i, Image: j.Image,100
Script: strings.Join(j.Script, "\n"), Needs: needs, NeedsSet: j.NeedsSet,116
Before: strings.Join(j.Before, "\n"), Script: strings.Join(j.Script, "\n"), After: strings.Join(j.After, "\n"),117
Needs: needs, Optional: j.Optional, NeedsSet: j.NeedsSet,101
Variables: vars, ArtifactPaths: paths, ArtifactExpire: j.Artifacts.ExpireIn,118
Variables: vars, ArtifactPaths: paths, ArtifactExpire: j.Artifacts.ExpireIn,102
AllowFailure: j.AllowFailure,119
AllowFailure: j.AllowFailure, When: j.When, Timeout: j.Timeout, Tags: j.Tags,120
CachePaths: j.Cache.Paths, CacheKey: j.Cache.Key, ResourceGroup: j.ResourceGroup,103
})121
})104
}122
}105
if _, err := s.Store.CreatePipeline(ctx, repo.ID, sha, ref, jobs); err != nil && s.Log != nil {123
if _, err := s.Store.CreatePipeline(ctx, repo.ID, sha, ref, source, actor, title, jobs); err != nil && s.Log != nil {106
s.Log.Error("pipeline", "err", err)124
s.Log.Error("pipeline", "err", err)107
}125
}108
_ = actor126
}127
128
func gitOne(ctx context.Context, gitBin, dir string, args ...string) string {129
out, err := exec.CommandContext(ctx, gitBin, append([]string{"-C", dir}, args...)...).Output()130
if err != nil {131
return ""132
}133
return strings.TrimSpace(string(out))134
}135
136
func gitLines(ctx context.Context, gitBin, dir string, args ...string) []string {137
out := gitOne(ctx, gitBin, dir, args...)138
if out == "" {139
return []string{}140
}141
return strings.Split(out, "\n")142
}143
144
func changedFiles(ctx context.Context, gitBin, dir, sha, before string, all []string) []string {145
if before == "" || strings.Trim(before, "0") == "" {146
parent := gitOne(ctx, gitBin, dir, "rev-parse", "--verify", sha+"^")147
if parent == "" {148
return all149
}150
before = parent151
}152
out, err := exec.CommandContext(ctx, gitBin, "-C", dir, "diff", "--name-only", before, sha).Output()153
if err != nil {154
return nil155
}156
text := strings.TrimSpace(string(out))157
if text == "" {158
return []string{}159
}160
return strings.Split(text, "\n")109
}161
}110
162
111
func pushEvent(repo store.Repo, kind, branch, commit, actor, summary, path string) webhook.Event {163
func pushEvent(repo store.Repo, kind, branch, commit, actor, summary, path string) webhook.Event {@@ -391,7 +391,7 @@ func (s *Server) runPipeline(w http.ResponseWriter, r *http.Request) {
391
391
s.redirect(w, r, repoURL(repo)+"/-/pipelines", "", s.tr(r, "err.branchMissing"))
392
392
return
393
393
}
394
s.startPipeline(r.Context(), repo, shaOut, "refs/heads/"+ref, u.Username)
394
s.startPipeline(r.Context(), repo, shaOut, "refs/heads/"+ref, u.Username, "web", "")
395
395
s.redirect(w, r, repoURL(repo)+"/-/pipelines", s.tr(r, "flash.pipeline"), "")
396
396
}
397
397
@@ -391,7 +391,7 @@ func (s *Server) runPipeline(w http.ResponseWriter, r *http.Request) {391
s.redirect(w, r, repoURL(repo)+"/-/pipelines", "", s.tr(r, "err.branchMissing"))391
s.redirect(w, r, repoURL(repo)+"/-/pipelines", "", s.tr(r, "err.branchMissing"))392
return392
return393
}393
}394
s.startPipeline(r.Context(), repo, shaOut, "refs/heads/"+ref, u.Username)394
s.startPipeline(r.Context(), repo, shaOut, "refs/heads/"+ref, u.Username, "web", "")395
s.redirect(w, r, repoURL(repo)+"/-/pipelines", s.tr(r, "flash.pipeline"), "")395
s.redirect(w, r, repoURL(repo)+"/-/pipelines", s.tr(r, "flash.pipeline"), "")396
}396
}397
397
@@ -235,23 +235,37 @@ type NewJob struct {
235
235
Stage string
236
236
Position int
237
237
Image string
238
Before string
238
239
Script string
240
After string
239
241
Needs []string
242
Optional []string
240
243
NeedsSet bool
241
244
Variables map[string]string
242
245
ArtifactPaths []string
243
246
ArtifactExpire time.Duration
244
247
AllowFailure bool
248
When string
249
Timeout time.Duration
250
Tags []string
251
CachePaths []string
252
CacheKey string
253
ResourceGroup string
245
254
}
246
255
247
256
// Pipeline is een draai van blue-ci-cd.yml.
248
257
type Pipeline struct {
249
ID int64
250
RepoID int64
251
CommitSHA string
252
Ref string
253
Status string
254
CreatedAt time.Time
258
ID int64
259
RepoID int64
260
CommitSHA string
261
Ref string
262
Status string
263
Source string
264
Actor string
265
CommitTitle string
266
CreatedAt time.Time
267
StartedAt time.Time
268
FinishedAt time.Time
255
269
}
256
270
257
271
// Job is één container in een pipeline.
@@ -264,16 +278,27 @@ type Job struct {
264
278
Position int
265
279
Status string
266
280
Image string
281
Before string
267
282
Script string
283
After string
268
284
Needs []string
285
Optional []string
269
286
NeedsSet bool
270
287
Variables map[string]string
271
288
ArtifactPaths []string
272
289
ArtifactExpire time.Duration
273
290
AllowFailure bool
291
When string
292
Timeout time.Duration
293
Tags []string
294
CachePaths []string
295
CacheKey string
296
ResourceGroup string
274
297
Log string
275
298
CommitSHA string
276
299
Ref string
300
StartedAt time.Time
301
FinishedAt time.Time
277
302
}
278
303
279
304
// RecordConfigFailure bewaart een pipeline die niet kon starten.
@@ -298,13 +323,13 @@ func (s *Store) RecordConfigFailure(ctx context.Context, repoID int64, sha, ref,
298
323
}
299
324
300
325
// CreatePipeline maakt een pipeline met jobs in de wachtrij.
301
func (s *Store) CreatePipeline(ctx context.Context, repoID int64, sha, ref string, jobs []NewJob) (int64, error) {
326
func (s *Store) CreatePipeline(ctx context.Context, repoID int64, sha, ref, source, actor, title string, jobs []NewJob) (int64, error) {
302
327
tx, err := s.db.BeginTx(ctx, nil)
303
328
if err != nil {
304
329
return 0, err
305
330
}
306
331
defer func() { _ = tx.Rollback() }()
307
res, err := tx.ExecContext(ctx, `INSERT INTO pipelines (repo_id, commit_sha, ref, status, created_at) VALUES (?, ?, ?, 'queued', ?)`, repoID, sha, ref, now())
332
res, err := tx.ExecContext(ctx, `INSERT INTO pipelines (repo_id, commit_sha, ref, status, source, actor, commit_title, created_at) VALUES (?, ?, ?, 'queued', ?, ?, ?, ?)`, repoID, sha, ref, source, actor, title, now())
308
333
if err != nil {
309
334
return 0, err
310
335
}
@@ -319,9 +344,26 @@ func (s *Store) CreatePipeline(ctx context.Context, repoID int64, sha, ref strin
319
344
if j.ArtifactPaths == nil {
320
345
j.ArtifactPaths = []string{}
321
346
}
322
_, err := tx.ExecContext(ctx, `INSERT INTO jobs (pipeline_id, repo_id, name, stage, position, status, image, script, needs_json, needs_set, variables_json, artifact_paths, artifact_expire_seconds, allow_failure, log)
323
VALUES (?, ?, ?, ?, ?, 'queued', ?, ?, ?, ?, ?, ?, ?, ?, '')`,
324
id, repoID, j.Name, j.Stage, j.Position, j.Image, j.Script, mustJSON(j.Needs), boolInt(j.NeedsSet), mustJSON(j.Variables), mustJSON(j.ArtifactPaths), int(j.ArtifactExpire/time.Second), boolInt(j.AllowFailure))
347
if j.Optional == nil {
348
j.Optional = []string{}
349
}
350
if j.Tags == nil {
351
j.Tags = []string{}
352
}
353
if j.CachePaths == nil {
354
j.CachePaths = []string{}
355
}
356
if j.When == "" {
357
j.When = "on_success"
358
}
359
status := "queued"
360
if j.When == "manual" {
361
status = "manual"
362
}
363
_, err := tx.ExecContext(ctx, `INSERT INTO jobs (pipeline_id, repo_id, name, stage, position, status, image, script, needs_json, needs_set, variables_json, artifact_paths, artifact_expire_seconds, allow_failure, log, before_script, after_script, when_mode, timeout_seconds, tags_json, cache_paths, cache_key, resource_group, optional_json)
364
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, '', ?, ?, ?, ?, ?, ?, ?, ?, ?)`,
365
id, repoID, j.Name, j.Stage, j.Position, status, j.Image, j.Script, mustJSON(j.Needs), boolInt(j.NeedsSet), mustJSON(j.Variables), mustJSON(j.ArtifactPaths), int(j.ArtifactExpire/time.Second), boolInt(j.AllowFailure),
366
j.Before, j.After, j.When, int(j.Timeout/time.Second), mustJSON(j.Tags), mustJSON(j.CachePaths), j.CacheKey, j.ResourceGroup, mustJSON(j.Optional))
325
367
if err != nil {
326
368
return 0, err
327
369
}
@@ -334,7 +376,7 @@ func (s *Store) CreatePipeline(ctx context.Context, repoID int64, sha, ref strin
334
376
335
377
// ListPipelines geeft de nieuwste pipelines.
336
378
func (s *Store) ListPipelines(ctx context.Context, repoID int64) ([]Pipeline, error) {
337
rows, err := s.db.QueryContext(ctx, `SELECT id, repo_id, commit_sha, ref, status, created_at FROM pipelines WHERE repo_id = ? ORDER BY id DESC LIMIT 50`, repoID)
379
rows, err := s.db.QueryContext(ctx, pipelineCols+` WHERE repo_id = ? ORDER BY id DESC LIMIT 50`, repoID)
338
380
if err != nil {
339
381
return nil, err
340
382
}
@@ -350,36 +392,40 @@ func (s *Store) ListPipelines(ctx context.Context, repoID int64) ([]Pipeline, er
350
392
return out, rows.Err()
351
393
}
352
394
395
const pipelineCols = `SELECT id, repo_id, commit_sha, ref, status, created_at, finished_at, source, actor, commit_title, started_at FROM pipelines`
396
353
397
func scanPipeline(rows *sql.Rows) (Pipeline, error) {
398
return scanPipelineRow(rows)
399
}
400
401
func scanPipelineRow(sc scanner) (Pipeline, error) {
354
402
var p Pipeline
355
var created string
356
if err := rows.Scan(&p.ID, &p.RepoID, &p.CommitSHA, &p.Ref, &p.Status, &created); err != nil {
403
var created, finished, started sql.NullString
404
if err := sc.Scan(&p.ID, &p.RepoID, &p.CommitSHA, &p.Ref, &p.Status, &created, &finished, &p.Source, &p.Actor, &p.CommitTitle, &started); err != nil {
357
405
return Pipeline{}, err
358
406
}
359
p.CreatedAt = parseTime(created)
407
p.CreatedAt = parseTime(created.String)
408
p.FinishedAt = parseTime(finished.String)
409
p.StartedAt = parseTime(started.String)
360
410
return p, nil
361
411
}
362
412
413
type scanner interface {
414
Scan(dest ...any) error
415
}
416
363
417
// PipelineByID leest één pipeline.
364
418
func (s *Store) PipelineByID(ctx context.Context, repoID, id int64) (Pipeline, error) {
365
var p Pipeline
366
var created string
367
err := s.db.QueryRowContext(ctx, `SELECT id, repo_id, commit_sha, ref, status, created_at FROM pipelines WHERE repo_id = ? AND id = ?`, repoID, id).
368
Scan(&p.ID, &p.RepoID, &p.CommitSHA, &p.Ref, &p.Status, &created)
419
p, err := scanPipelineRow(s.db.QueryRowContext(ctx, pipelineCols+` WHERE repo_id = ? AND id = ?`, repoID, id))
369
420
if errors.Is(err, sql.ErrNoRows) {
370
421
return Pipeline{}, errNotFound
371
422
}
372
if err != nil {
373
return Pipeline{}, err
374
}
375
p.CreatedAt = parseTime(created)
376
return p, nil
423
return p, err
377
424
}
378
425
379
426
// JobsByPipeline geeft de jobs in bestandsvolgorde.
380
427
func (s *Store) JobsByPipeline(ctx context.Context, pipelineID int64) ([]Job, error) {
381
rows, err := s.db.QueryContext(ctx, `SELECT j.id, j.pipeline_id, j.repo_id, j.name, j.stage, j.position, j.status, j.image, j.script, j.needs_json, j.needs_set, j.variables_json, j.artifact_paths, j.artifact_expire_seconds, j.allow_failure, j.log, p.commit_sha, p.ref
382
FROM jobs j JOIN pipelines p ON p.id = j.pipeline_id WHERE j.pipeline_id = ? ORDER BY j.position, j.id`, pipelineID)
428
rows, err := s.db.QueryContext(ctx, jobSelect+` WHERE j.pipeline_id = ? ORDER BY j.position, j.id`, pipelineID)
383
429
if err != nil {
384
430
return nil, err
385
431
}
@@ -395,25 +441,35 @@ func (s *Store) JobsByPipeline(ctx context.Context, pipelineID int64) ([]Job, er
395
441
return out, rows.Err()
396
442
}
397
443
444
const jobSelect = `SELECT j.id, j.pipeline_id, j.repo_id, j.name, j.stage, j.position, j.status, j.image, j.script, j.needs_json, j.needs_set, j.variables_json, j.artifact_paths, j.artifact_expire_seconds, j.allow_failure, j.log, p.commit_sha, p.ref, j.before_script, j.after_script, j.when_mode, j.timeout_seconds, j.tags_json, j.cache_paths, j.cache_key, j.resource_group, j.optional_json, j.started_at, j.finished_at
445
FROM jobs j JOIN pipelines p ON p.id = j.pipeline_id`
446
398
447
func scanJob(rows *sql.Rows) (Job, error) {
399
448
var j Job
400
var needs, vars, paths string
401
var needsSet, allow, expire int
402
if err := rows.Scan(&j.ID, &j.PipelineID, &j.RepoID, &j.Name, &j.Stage, &j.Position, &j.Status, &j.Image, &j.Script, &needs, &needsSet, &vars, &paths, &expire, &allow, &j.Log, &j.CommitSHA, &j.Ref); err != nil {
449
var needs, vars, paths, tags, cache, optional string
450
var needsSet, allow, expire, timeout int
451
var started, finished sql.NullString
452
if err := rows.Scan(&j.ID, &j.PipelineID, &j.RepoID, &j.Name, &j.Stage, &j.Position, &j.Status, &j.Image, &j.Script, &needs, &needsSet, &vars, &paths, &expire, &allow, &j.Log, &j.CommitSHA, &j.Ref, &j.Before, &j.After, &j.When, &timeout, &tags, &cache, &j.CacheKey, &j.ResourceGroup, &optional, &started, &finished); err != nil {
403
453
return Job{}, err
404
454
}
405
455
j.NeedsSet = needsSet == 1
406
456
j.AllowFailure = allow == 1
407
457
j.ArtifactExpire = time.Duration(expire) * time.Second
458
j.Timeout = time.Duration(timeout) * time.Second
459
j.StartedAt = parseTime(started.String)
460
j.FinishedAt = parseTime(finished.String)
408
461
_ = json.Unmarshal([]byte(needs), &j.Needs)
409
462
_ = json.Unmarshal([]byte(vars), &j.Variables)
410
463
_ = json.Unmarshal([]byte(paths), &j.ArtifactPaths)
464
_ = json.Unmarshal([]byte(tags), &j.Tags)
465
_ = json.Unmarshal([]byte(cache), &j.CachePaths)
466
_ = json.Unmarshal([]byte(optional), &j.Optional)
411
467
return j, nil
412
468
}
413
469
414
470
// ActivePipelines zijn nog niet af.
415
471
func (s *Store) ActivePipelines(ctx context.Context) ([]Pipeline, error) {
416
rows, err := s.db.QueryContext(ctx, `SELECT id, repo_id, commit_sha, ref, status, created_at FROM pipelines WHERE status IN ('queued', 'running') ORDER BY id`)
472
rows, err := s.db.QueryContext(ctx, pipelineCols+` WHERE status IN ('queued', 'running', 'manual') ORDER BY id`)
417
473
if err != nil {
418
474
return nil, err
419
475
}
@@ -437,14 +493,14 @@ func (s *Store) MarkJobRunning(ctx context.Context, id int64) (bool, error) {
437
493
}
438
494
n, _ := res.RowsAffected()
439
495
if n == 1 {
440
_, _ = s.db.ExecContext(ctx, `UPDATE pipelines SET status = 'running' WHERE id = (SELECT pipeline_id FROM jobs WHERE id = ?)`, id)
496
_, _ = s.db.ExecContext(ctx, `UPDATE pipelines SET status = 'running', started_at = COALESCE(started_at, ?) WHERE id = (SELECT pipeline_id FROM jobs WHERE id = ?)`, now(), id)
441
497
}
442
498
return n == 1, nil
443
499
}
444
500
445
501
// FinishJob schrijft het eindresultaat en het log.
446
func (s *Store) FinishJob(ctx context.Context, id int64, status, logText string) error {
447
_, err := s.db.ExecContext(ctx, `UPDATE jobs SET status = ?, log = ?, finished_at = ? WHERE id = ?`, status, logText, now(), id)
502
func (s *Store) FinishJob(ctx context.Context, id int64, status, extra string) error {
503
_, err := s.db.ExecContext(ctx, `UPDATE jobs SET status = ?, log = log || ?, finished_at = ? WHERE id = ?`, status, extra, now(), id)
448
504
return err
449
505
}
450
506
@@ -461,6 +517,8 @@ func (s *Store) RefreshPipelineStatus(ctx context.Context, pipelineID int64) (st
461
517
return "", err
462
518
}
463
519
status := "success"
520
open := false
521
manual := false
464
522
for _, j := range jobs {
465
523
switch j.Status {
466
524
case "failed":
@@ -468,22 +526,52 @@ func (s *Store) RefreshPipelineStatus(ctx context.Context, pipelineID int64) (st
468
526
status = "failed"
469
527
}
470
528
case "queued", "running":
471
if status != "failed" {
472
status = "running"
473
}
529
open = true
530
case "manual":
531
manual = true
474
532
}
475
533
}
476
if status == "success" || status == "failed" {
477
_, err = s.db.ExecContext(ctx, `UPDATE pipelines SET status = ?, finished_at = ? WHERE id = ?`, status, now(), pipelineID)
478
} else {
534
switch {
535
case status == "failed":
536
_, err = s.db.ExecContext(ctx, `UPDATE pipelines SET status = 'failed', finished_at = ? WHERE id = ?`, now(), pipelineID)
537
case open:
538
status = "running"
479
539
_, err = s.db.ExecContext(ctx, `UPDATE pipelines SET status = 'running' WHERE id = ?`, pipelineID)
540
case manual:
541
status = "manual"
542
_, err = s.db.ExecContext(ctx, `UPDATE pipelines SET status = 'manual', finished_at = NULL WHERE id = ?`, pipelineID)
543
default:
544
_, err = s.db.ExecContext(ctx, `UPDATE pipelines SET status = 'success', finished_at = ? WHERE id = ?`, now(), pipelineID)
480
545
}
481
546
return status, err
482
547
}
483
548
549
// ResourceBusy meldt of een andere job in deze repository die groep bezet.
550
func (s *Store) ResourceBusy(ctx context.Context, repoID int64, group string, except int64) (bool, error) {
551
if group == "" {
552
return false, nil
553
}
554
var n int
555
err := s.db.QueryRowContext(ctx, `SELECT COUNT(*) FROM jobs WHERE repo_id = ? AND resource_group = ? AND status = 'running' AND id != ?`, repoID, group, except).Scan(&n)
556
return n > 0, err
557
}
558
559
// QueueJob zet een handmatige job in de wachtrij.
560
func (s *Store) QueueJob(ctx context.Context, id int64) (bool, error) {
561
res, err := s.db.ExecContext(ctx, `UPDATE jobs SET status = 'queued' WHERE id = ? AND status = 'manual'`, id)
562
if err != nil {
563
return false, err
564
}
565
n, _ := res.RowsAffected()
566
if n == 1 {
567
_, _ = s.db.ExecContext(ctx, `UPDATE pipelines SET status = 'running', finished_at = NULL WHERE id = (SELECT pipeline_id FROM jobs WHERE id = ?)`, id)
568
}
569
return n == 1, nil
570
}
571
484
572
// SkipJob markeert een job die niet meer hoeft.
485
573
func (s *Store) SkipJob(ctx context.Context, id int64, reason string) error {
486
_, err := s.db.ExecContext(ctx, `UPDATE jobs SET status = 'skipped', log = ?, finished_at = ? WHERE id = ? AND status = 'queued'`, reason, now(), id)
574
_, err := s.db.ExecContext(ctx, `UPDATE jobs SET status = 'skipped', log = ?, finished_at = ? WHERE id = ? AND status IN ('queued', 'manual')`, reason, now(), id)
487
575
return err
488
576
}
489
577
@@ -515,8 +603,7 @@ func (s *Store) JobByToken(ctx context.Context, token string) (Job, error) {
515
603
if err != nil {
516
604
return Job{}, err
517
605
}
518
rows, err := s.db.QueryContext(ctx, `SELECT j.id, j.pipeline_id, j.repo_id, j.name, j.stage, j.position, j.status, j.image, j.script, j.needs_json, j.needs_set, j.variables_json, j.artifact_paths, j.artifact_expire_seconds, j.allow_failure, j.log, p.commit_sha, p.ref
519
FROM jobs j JOIN pipelines p ON p.id = j.pipeline_id WHERE j.id = ?`, jobID)
606
rows, err := s.db.QueryContext(ctx, jobSelect+` WHERE j.id = ?`, jobID)
520
607
if err != nil {
521
608
return Job{}, err
522
609
}
@@ -235,23 +235,37 @@ type NewJob struct {235
Stage string235
Stage string236
Position int236
Position int237
Image string237
Image string238
Before string238
Script string239
Script string240
After string239
Needs []string241
Needs []string242
Optional []string240
NeedsSet bool243
NeedsSet bool241
Variables map[string]string244
Variables map[string]string242
ArtifactPaths []string245
ArtifactPaths []string243
ArtifactExpire time.Duration246
ArtifactExpire time.Duration244
AllowFailure bool247
AllowFailure bool248
When string249
Timeout time.Duration250
Tags []string251
CachePaths []string252
CacheKey string253
ResourceGroup string245
}254
}246
255
247
// Pipeline is een draai van blue-ci-cd.yml.256
// Pipeline is een draai van blue-ci-cd.yml.248
type Pipeline struct {257
type Pipeline struct {249
ID int64258
ID int64250
RepoID int64259
RepoID int64251
CommitSHA string260
CommitSHA string252
Ref string261
Ref string253
Status string262
Status string254
CreatedAt time.Time263
Source string264
Actor string265
CommitTitle string266
CreatedAt time.Time267
StartedAt time.Time268
FinishedAt time.Time255
}269
}256
270
257
// Job is één container in een pipeline.271
// Job is één container in een pipeline.@@ -264,16 +278,27 @@ type Job struct {264
Position int278
Position int265
Status string279
Status string266
Image string280
Image string281
Before string267
Script string282
Script string283
After string268
Needs []string284
Needs []string285
Optional []string269
NeedsSet bool286
NeedsSet bool270
Variables map[string]string287
Variables map[string]string271
ArtifactPaths []string288
ArtifactPaths []string272
ArtifactExpire time.Duration289
ArtifactExpire time.Duration273
AllowFailure bool290
AllowFailure bool291
When string292
Timeout time.Duration293
Tags []string294
CachePaths []string295
CacheKey string296
ResourceGroup string274
Log string297
Log string275
CommitSHA string298
CommitSHA string276
Ref string299
Ref string300
StartedAt time.Time301
FinishedAt time.Time277
}302
}278
303
279
// RecordConfigFailure bewaart een pipeline die niet kon starten.304
// RecordConfigFailure bewaart een pipeline die niet kon starten.@@ -298,13 +323,13 @@ func (s *Store) RecordConfigFailure(ctx context.Context, repoID int64, sha, ref,298
}323
}299
324
300
// CreatePipeline maakt een pipeline met jobs in de wachtrij.325
// CreatePipeline maakt een pipeline met jobs in de wachtrij.301
func (s *Store) CreatePipeline(ctx context.Context, repoID int64, sha, ref string, jobs []NewJob) (int64, error) {326
func (s *Store) CreatePipeline(ctx context.Context, repoID int64, sha, ref, source, actor, title string, jobs []NewJob) (int64, error) {302
tx, err := s.db.BeginTx(ctx, nil)327
tx, err := s.db.BeginTx(ctx, nil)303
if err != nil {328
if err != nil {304
return 0, err329
return 0, err305
}330
}306
defer func() { _ = tx.Rollback() }()331
defer func() { _ = tx.Rollback() }()307
res, err := tx.ExecContext(ctx, `INSERT INTO pipelines (repo_id, commit_sha, ref, status, created_at) VALUES (?, ?, ?, 'queued', ?)`, repoID, sha, ref, now())332
res, err := tx.ExecContext(ctx, `INSERT INTO pipelines (repo_id, commit_sha, ref, status, source, actor, commit_title, created_at) VALUES (?, ?, ?, 'queued', ?, ?, ?, ?)`, repoID, sha, ref, source, actor, title, now())308
if err != nil {333
if err != nil {309
return 0, err334
return 0, err310
}335
}@@ -319,9 +344,26 @@ func (s *Store) CreatePipeline(ctx context.Context, repoID int64, sha, ref strin319
if j.ArtifactPaths == nil {344
if j.ArtifactPaths == nil {320
j.ArtifactPaths = []string{}345
j.ArtifactPaths = []string{}321
}346
}322
_, err := tx.ExecContext(ctx, `INSERT INTO jobs (pipeline_id, repo_id, name, stage, position, status, image, script, needs_json, needs_set, variables_json, artifact_paths, artifact_expire_seconds, allow_failure, log)347
if j.Optional == nil {323
VALUES (?, ?, ?, ?, ?, 'queued', ?, ?, ?, ?, ?, ?, ?, ?, '')`,348
j.Optional = []string{}324
id, repoID, j.Name, j.Stage, j.Position, j.Image, j.Script, mustJSON(j.Needs), boolInt(j.NeedsSet), mustJSON(j.Variables), mustJSON(j.ArtifactPaths), int(j.ArtifactExpire/time.Second), boolInt(j.AllowFailure))349
}350
if j.Tags == nil {351
j.Tags = []string{}352
}353
if j.CachePaths == nil {354
j.CachePaths = []string{}355
}356
if j.When == "" {357
j.When = "on_success"358
}359
status := "queued"360
if j.When == "manual" {361
status = "manual"362
}363
_, err := tx.ExecContext(ctx, `INSERT INTO jobs (pipeline_id, repo_id, name, stage, position, status, image, script, needs_json, needs_set, variables_json, artifact_paths, artifact_expire_seconds, allow_failure, log, before_script, after_script, when_mode, timeout_seconds, tags_json, cache_paths, cache_key, resource_group, optional_json)364
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, '', ?, ?, ?, ?, ?, ?, ?, ?, ?)`,365
id, repoID, j.Name, j.Stage, j.Position, status, j.Image, j.Script, mustJSON(j.Needs), boolInt(j.NeedsSet), mustJSON(j.Variables), mustJSON(j.ArtifactPaths), int(j.ArtifactExpire/time.Second), boolInt(j.AllowFailure),366
j.Before, j.After, j.When, int(j.Timeout/time.Second), mustJSON(j.Tags), mustJSON(j.CachePaths), j.CacheKey, j.ResourceGroup, mustJSON(j.Optional))325
if err != nil {367
if err != nil {326
return 0, err368
return 0, err327
}369
}@@ -334,7 +376,7 @@ func (s *Store) CreatePipeline(ctx context.Context, repoID int64, sha, ref strin334
376
335
// ListPipelines geeft de nieuwste pipelines.377
// ListPipelines geeft de nieuwste pipelines.336
func (s *Store) ListPipelines(ctx context.Context, repoID int64) ([]Pipeline, error) {378
func (s *Store) ListPipelines(ctx context.Context, repoID int64) ([]Pipeline, error) {337
rows, err := s.db.QueryContext(ctx, `SELECT id, repo_id, commit_sha, ref, status, created_at FROM pipelines WHERE repo_id = ? ORDER BY id DESC LIMIT 50`, repoID)379
rows, err := s.db.QueryContext(ctx, pipelineCols+` WHERE repo_id = ? ORDER BY id DESC LIMIT 50`, repoID)338
if err != nil {380
if err != nil {339
return nil, err381
return nil, err340
}382
}@@ -350,36 +392,40 @@ func (s *Store) ListPipelines(ctx context.Context, repoID int64) ([]Pipeline, er350
return out, rows.Err()392
return out, rows.Err()351
}393
}352
394
395
const pipelineCols = `SELECT id, repo_id, commit_sha, ref, status, created_at, finished_at, source, actor, commit_title, started_at FROM pipelines`396
353
func scanPipeline(rows *sql.Rows) (Pipeline, error) {397
func scanPipeline(rows *sql.Rows) (Pipeline, error) {398
return scanPipelineRow(rows)399
}400
401
func scanPipelineRow(sc scanner) (Pipeline, error) {354
var p Pipeline402
var p Pipeline355
var created string403
var created, finished, started sql.NullString356
if err := rows.Scan(&p.ID, &p.RepoID, &p.CommitSHA, &p.Ref, &p.Status, &created); err != nil {404
if err := sc.Scan(&p.ID, &p.RepoID, &p.CommitSHA, &p.Ref, &p.Status, &created, &finished, &p.Source, &p.Actor, &p.CommitTitle, &started); err != nil {357
return Pipeline{}, err405
return Pipeline{}, err358
}406
}359
p.CreatedAt = parseTime(created)407
p.CreatedAt = parseTime(created.String)408
p.FinishedAt = parseTime(finished.String)409
p.StartedAt = parseTime(started.String)360
return p, nil410
return p, nil361
}411
}362
412
413
type scanner interface {414
Scan(dest ...any) error415
}416
363
// PipelineByID leest één pipeline.417
// PipelineByID leest één pipeline.364
func (s *Store) PipelineByID(ctx context.Context, repoID, id int64) (Pipeline, error) {418
func (s *Store) PipelineByID(ctx context.Context, repoID, id int64) (Pipeline, error) {365
var p Pipeline419
p, err := scanPipelineRow(s.db.QueryRowContext(ctx, pipelineCols+` WHERE repo_id = ? AND id = ?`, repoID, id))366
var created string367
err := s.db.QueryRowContext(ctx, `SELECT id, repo_id, commit_sha, ref, status, created_at FROM pipelines WHERE repo_id = ? AND id = ?`, repoID, id).368
Scan(&p.ID, &p.RepoID, &p.CommitSHA, &p.Ref, &p.Status, &created)369
if errors.Is(err, sql.ErrNoRows) {420
if errors.Is(err, sql.ErrNoRows) {370
return Pipeline{}, errNotFound421
return Pipeline{}, errNotFound371
}422
}372
if err != nil {423
return p, err373
return Pipeline{}, err374
}375
p.CreatedAt = parseTime(created)376
return p, nil377
}424
}378
425
379
// JobsByPipeline geeft de jobs in bestandsvolgorde.426
// JobsByPipeline geeft de jobs in bestandsvolgorde.380
func (s *Store) JobsByPipeline(ctx context.Context, pipelineID int64) ([]Job, error) {427
func (s *Store) JobsByPipeline(ctx context.Context, pipelineID int64) ([]Job, error) {381
rows, err := s.db.QueryContext(ctx, `SELECT j.id, j.pipeline_id, j.repo_id, j.name, j.stage, j.position, j.status, j.image, j.script, j.needs_json, j.needs_set, j.variables_json, j.artifact_paths, j.artifact_expire_seconds, j.allow_failure, j.log, p.commit_sha, p.ref428
rows, err := s.db.QueryContext(ctx, jobSelect+` WHERE j.pipeline_id = ? ORDER BY j.position, j.id`, pipelineID)382
FROM jobs j JOIN pipelines p ON p.id = j.pipeline_id WHERE j.pipeline_id = ? ORDER BY j.position, j.id`, pipelineID)383
if err != nil {429
if err != nil {384
return nil, err430
return nil, err385
}431
}@@ -395,25 +441,35 @@ func (s *Store) JobsByPipeline(ctx context.Context, pipelineID int64) ([]Job, er395
return out, rows.Err()441
return out, rows.Err()396
}442
}397
443
444
const jobSelect = `SELECT j.id, j.pipeline_id, j.repo_id, j.name, j.stage, j.position, j.status, j.image, j.script, j.needs_json, j.needs_set, j.variables_json, j.artifact_paths, j.artifact_expire_seconds, j.allow_failure, j.log, p.commit_sha, p.ref, j.before_script, j.after_script, j.when_mode, j.timeout_seconds, j.tags_json, j.cache_paths, j.cache_key, j.resource_group, j.optional_json, j.started_at, j.finished_at445
FROM jobs j JOIN pipelines p ON p.id = j.pipeline_id`446
398
func scanJob(rows *sql.Rows) (Job, error) {447
func scanJob(rows *sql.Rows) (Job, error) {399
var j Job448
var j Job400
var needs, vars, paths string449
var needs, vars, paths, tags, cache, optional string401
var needsSet, allow, expire int450
var needsSet, allow, expire, timeout int402
if err := rows.Scan(&j.ID, &j.PipelineID, &j.RepoID, &j.Name, &j.Stage, &j.Position, &j.Status, &j.Image, &j.Script, &needs, &needsSet, &vars, &paths, &expire, &allow, &j.Log, &j.CommitSHA, &j.Ref); err != nil {451
var started, finished sql.NullString452
if err := rows.Scan(&j.ID, &j.PipelineID, &j.RepoID, &j.Name, &j.Stage, &j.Position, &j.Status, &j.Image, &j.Script, &needs, &needsSet, &vars, &paths, &expire, &allow, &j.Log, &j.CommitSHA, &j.Ref, &j.Before, &j.After, &j.When, &timeout, &tags, &cache, &j.CacheKey, &j.ResourceGroup, &optional, &started, &finished); err != nil {403
return Job{}, err453
return Job{}, err404
}454
}405
j.NeedsSet = needsSet == 1455
j.NeedsSet = needsSet == 1406
j.AllowFailure = allow == 1456
j.AllowFailure = allow == 1407
j.ArtifactExpire = time.Duration(expire) * time.Second457
j.ArtifactExpire = time.Duration(expire) * time.Second458
j.Timeout = time.Duration(timeout) * time.Second459
j.StartedAt = parseTime(started.String)460
j.FinishedAt = parseTime(finished.String)408
_ = json.Unmarshal([]byte(needs), &j.Needs)461
_ = json.Unmarshal([]byte(needs), &j.Needs)409
_ = json.Unmarshal([]byte(vars), &j.Variables)462
_ = json.Unmarshal([]byte(vars), &j.Variables)410
_ = json.Unmarshal([]byte(paths), &j.ArtifactPaths)463
_ = json.Unmarshal([]byte(paths), &j.ArtifactPaths)464
_ = json.Unmarshal([]byte(tags), &j.Tags)465
_ = json.Unmarshal([]byte(cache), &j.CachePaths)466
_ = json.Unmarshal([]byte(optional), &j.Optional)411
return j, nil467
return j, nil412
}468
}413
469
414
// ActivePipelines zijn nog niet af.470
// ActivePipelines zijn nog niet af.415
func (s *Store) ActivePipelines(ctx context.Context) ([]Pipeline, error) {471
func (s *Store) ActivePipelines(ctx context.Context) ([]Pipeline, error) {416
rows, err := s.db.QueryContext(ctx, `SELECT id, repo_id, commit_sha, ref, status, created_at FROM pipelines WHERE status IN ('queued', 'running') ORDER BY id`)472
rows, err := s.db.QueryContext(ctx, pipelineCols+` WHERE status IN ('queued', 'running', 'manual') ORDER BY id`)417
if err != nil {473
if err != nil {418
return nil, err474
return nil, err419
}475
}@@ -437,14 +493,14 @@ func (s *Store) MarkJobRunning(ctx context.Context, id int64) (bool, error) {437
}493
}438
n, _ := res.RowsAffected()494
n, _ := res.RowsAffected()439
if n == 1 {495
if n == 1 {440
_, _ = s.db.ExecContext(ctx, `UPDATE pipelines SET status = 'running' WHERE id = (SELECT pipeline_id FROM jobs WHERE id = ?)`, id)496
_, _ = s.db.ExecContext(ctx, `UPDATE pipelines SET status = 'running', started_at = COALESCE(started_at, ?) WHERE id = (SELECT pipeline_id FROM jobs WHERE id = ?)`, now(), id)441
}497
}442
return n == 1, nil498
return n == 1, nil443
}499
}444
500
445
// FinishJob schrijft het eindresultaat en het log.501
// FinishJob schrijft het eindresultaat en het log.446
func (s *Store) FinishJob(ctx context.Context, id int64, status, logText string) error {502
func (s *Store) FinishJob(ctx context.Context, id int64, status, extra string) error {447
_, err := s.db.ExecContext(ctx, `UPDATE jobs SET status = ?, log = ?, finished_at = ? WHERE id = ?`, status, logText, now(), id)503
_, err := s.db.ExecContext(ctx, `UPDATE jobs SET status = ?, log = log || ?, finished_at = ? WHERE id = ?`, status, extra, now(), id)448
return err504
return err449
}505
}450
506
@@ -461,6 +517,8 @@ func (s *Store) RefreshPipelineStatus(ctx context.Context, pipelineID int64) (st461
return "", err517
return "", err462
}518
}463
status := "success"519
status := "success"520
open := false521
manual := false464
for _, j := range jobs {522
for _, j := range jobs {465
switch j.Status {523
switch j.Status {466
case "failed":524
case "failed":@@ -468,22 +526,52 @@ func (s *Store) RefreshPipelineStatus(ctx context.Context, pipelineID int64) (st468
status = "failed"526
status = "failed"469
}527
}470
case "queued", "running":528
case "queued", "running":471
if status != "failed" {529
open = true472
status = "running"530
case "manual":473
}531
manual = true474
}532
}475
}533
}476
if status == "success" || status == "failed" {534
switch {477
_, err = s.db.ExecContext(ctx, `UPDATE pipelines SET status = ?, finished_at = ? WHERE id = ?`, status, now(), pipelineID)535
case status == "failed":478
} else {536
_, err = s.db.ExecContext(ctx, `UPDATE pipelines SET status = 'failed', finished_at = ? WHERE id = ?`, now(), pipelineID)537
case open:538
status = "running"479
_, err = s.db.ExecContext(ctx, `UPDATE pipelines SET status = 'running' WHERE id = ?`, pipelineID)539
_, err = s.db.ExecContext(ctx, `UPDATE pipelines SET status = 'running' WHERE id = ?`, pipelineID)540
case manual:541
status = "manual"542
_, err = s.db.ExecContext(ctx, `UPDATE pipelines SET status = 'manual', finished_at = NULL WHERE id = ?`, pipelineID)543
default:544
_, err = s.db.ExecContext(ctx, `UPDATE pipelines SET status = 'success', finished_at = ? WHERE id = ?`, now(), pipelineID)480
}545
}481
return status, err546
return status, err482
}547
}483
548
549
// ResourceBusy meldt of een andere job in deze repository die groep bezet.550
func (s *Store) ResourceBusy(ctx context.Context, repoID int64, group string, except int64) (bool, error) {551
if group == "" {552
return false, nil553
}554
var n int555
err := s.db.QueryRowContext(ctx, `SELECT COUNT(*) FROM jobs WHERE repo_id = ? AND resource_group = ? AND status = 'running' AND id != ?`, repoID, group, except).Scan(&n)556
return n > 0, err557
}558
559
// QueueJob zet een handmatige job in de wachtrij.560
func (s *Store) QueueJob(ctx context.Context, id int64) (bool, error) {561
res, err := s.db.ExecContext(ctx, `UPDATE jobs SET status = 'queued' WHERE id = ? AND status = 'manual'`, id)562
if err != nil {563
return false, err564
}565
n, _ := res.RowsAffected()566
if n == 1 {567
_, _ = s.db.ExecContext(ctx, `UPDATE pipelines SET status = 'running', finished_at = NULL WHERE id = (SELECT pipeline_id FROM jobs WHERE id = ?)`, id)568
}569
return n == 1, nil570
}571
484
// SkipJob markeert een job die niet meer hoeft.572
// SkipJob markeert een job die niet meer hoeft.485
func (s *Store) SkipJob(ctx context.Context, id int64, reason string) error {573
func (s *Store) SkipJob(ctx context.Context, id int64, reason string) error {486
_, err := s.db.ExecContext(ctx, `UPDATE jobs SET status = 'skipped', log = ?, finished_at = ? WHERE id = ? AND status = 'queued'`, reason, now(), id)574
_, err := s.db.ExecContext(ctx, `UPDATE jobs SET status = 'skipped', log = ?, finished_at = ? WHERE id = ? AND status IN ('queued', 'manual')`, reason, now(), id)487
return err575
return err488
}576
}489
577
@@ -515,8 +603,7 @@ func (s *Store) JobByToken(ctx context.Context, token string) (Job, error) {515
if err != nil {603
if err != nil {516
return Job{}, err604
return Job{}, err517
}605
}518
rows, err := s.db.QueryContext(ctx, `SELECT j.id, j.pipeline_id, j.repo_id, j.name, j.stage, j.position, j.status, j.image, j.script, j.needs_json, j.needs_set, j.variables_json, j.artifact_paths, j.artifact_expire_seconds, j.allow_failure, j.log, p.commit_sha, p.ref606
rows, err := s.db.QueryContext(ctx, jobSelect+` WHERE j.id = ?`, jobID)519
FROM jobs j JOIN pipelines p ON p.id = j.pipeline_id WHERE j.id = ?`, jobID)520
if err != nil {607
if err != nil {521
return Job{}, err608
return Job{}, err522
}609
}@@ -68,6 +68,19 @@ func (s *Store) migrate() error {
68
68
`ALTER TABLE notes ADD COLUMN line_end INTEGER NOT NULL DEFAULT 0`,
69
69
`ALTER TABLE notes ADD COLUMN suggestion TEXT NOT NULL DEFAULT ''`,
70
70
`ALTER TABLE notes ADD COLUMN base_text TEXT NOT NULL DEFAULT ''`,
71
`ALTER TABLE pipelines ADD COLUMN source TEXT NOT NULL DEFAULT 'push'`,
72
`ALTER TABLE pipelines ADD COLUMN actor TEXT NOT NULL DEFAULT ''`,
73
`ALTER TABLE pipelines ADD COLUMN commit_title TEXT NOT NULL DEFAULT ''`,
74
`ALTER TABLE pipelines ADD COLUMN started_at TEXT`,
75
`ALTER TABLE jobs ADD COLUMN before_script TEXT NOT NULL DEFAULT ''`,
76
`ALTER TABLE jobs ADD COLUMN after_script TEXT NOT NULL DEFAULT ''`,
77
`ALTER TABLE jobs ADD COLUMN when_mode TEXT NOT NULL DEFAULT 'on_success'`,
78
`ALTER TABLE jobs ADD COLUMN timeout_seconds INTEGER NOT NULL DEFAULT 0`,
79
`ALTER TABLE jobs ADD COLUMN tags_json TEXT NOT NULL DEFAULT '[]'`,
80
`ALTER TABLE jobs ADD COLUMN cache_paths TEXT NOT NULL DEFAULT '[]'`,
81
`ALTER TABLE jobs ADD COLUMN cache_key TEXT NOT NULL DEFAULT ''`,
82
`ALTER TABLE jobs ADD COLUMN resource_group TEXT NOT NULL DEFAULT ''`,
83
`ALTER TABLE jobs ADD COLUMN optional_json TEXT NOT NULL DEFAULT '[]'`,
71
84
}
72
85
for _, q := range alters {
73
86
if _, err := s.db.Exec(q); err != nil && !strings.Contains(strings.ToLower(err.Error()), "duplicate column") {
@@ -219,7 +232,11 @@ CREATE TABLE IF NOT EXISTS pipelines (
219
232
ref TEXT NOT NULL,
220
233
status TEXT NOT NULL,
221
234
created_at TEXT NOT NULL,
222
finished_at TEXT
235
finished_at TEXT,
236
source TEXT NOT NULL DEFAULT 'push',
237
actor TEXT NOT NULL DEFAULT '',
238
commit_title TEXT NOT NULL DEFAULT '',
239
started_at TEXT
223
240
);
224
241
CREATE TABLE IF NOT EXISTS jobs (
225
242
id INTEGER PRIMARY KEY,
@@ -240,6 +257,15 @@ CREATE TABLE IF NOT EXISTS jobs (
240
257
log TEXT NOT NULL DEFAULT '',
241
258
started_at TEXT,
242
259
finished_at TEXT,
260
before_script TEXT NOT NULL DEFAULT '',
261
after_script TEXT NOT NULL DEFAULT '',
262
when_mode TEXT NOT NULL DEFAULT 'on_success',
263
timeout_seconds INTEGER NOT NULL DEFAULT 0,
264
tags_json TEXT NOT NULL DEFAULT '[]',
265
cache_paths TEXT NOT NULL DEFAULT '[]',
266
cache_key TEXT NOT NULL DEFAULT '',
267
resource_group TEXT NOT NULL DEFAULT '',
268
optional_json TEXT NOT NULL DEFAULT '[]',
243
269
UNIQUE(pipeline_id, name)
244
270
);
245
271
CREATE TABLE IF NOT EXISTS artifacts (
@@ -68,6 +68,19 @@ func (s *Store) migrate() error {68
`ALTER TABLE notes ADD COLUMN line_end INTEGER NOT NULL DEFAULT 0`,68
`ALTER TABLE notes ADD COLUMN line_end INTEGER NOT NULL DEFAULT 0`,69
`ALTER TABLE notes ADD COLUMN suggestion TEXT NOT NULL DEFAULT ''`,69
`ALTER TABLE notes ADD COLUMN suggestion TEXT NOT NULL DEFAULT ''`,70
`ALTER TABLE notes ADD COLUMN base_text TEXT NOT NULL DEFAULT ''`,70
`ALTER TABLE notes ADD COLUMN base_text TEXT NOT NULL DEFAULT ''`,71
`ALTER TABLE pipelines ADD COLUMN source TEXT NOT NULL DEFAULT 'push'`,72
`ALTER TABLE pipelines ADD COLUMN actor TEXT NOT NULL DEFAULT ''`,73
`ALTER TABLE pipelines ADD COLUMN commit_title TEXT NOT NULL DEFAULT ''`,74
`ALTER TABLE pipelines ADD COLUMN started_at TEXT`,75
`ALTER TABLE jobs ADD COLUMN before_script TEXT NOT NULL DEFAULT ''`,76
`ALTER TABLE jobs ADD COLUMN after_script TEXT NOT NULL DEFAULT ''`,77
`ALTER TABLE jobs ADD COLUMN when_mode TEXT NOT NULL DEFAULT 'on_success'`,78
`ALTER TABLE jobs ADD COLUMN timeout_seconds INTEGER NOT NULL DEFAULT 0`,79
`ALTER TABLE jobs ADD COLUMN tags_json TEXT NOT NULL DEFAULT '[]'`,80
`ALTER TABLE jobs ADD COLUMN cache_paths TEXT NOT NULL DEFAULT '[]'`,81
`ALTER TABLE jobs ADD COLUMN cache_key TEXT NOT NULL DEFAULT ''`,82
`ALTER TABLE jobs ADD COLUMN resource_group TEXT NOT NULL DEFAULT ''`,83
`ALTER TABLE jobs ADD COLUMN optional_json TEXT NOT NULL DEFAULT '[]'`,71
}84
}72
for _, q := range alters {85
for _, q := range alters {73
if _, err := s.db.Exec(q); err != nil && !strings.Contains(strings.ToLower(err.Error()), "duplicate column") {86
if _, err := s.db.Exec(q); err != nil && !strings.Contains(strings.ToLower(err.Error()), "duplicate column") {@@ -219,7 +232,11 @@ CREATE TABLE IF NOT EXISTS pipelines (219
ref TEXT NOT NULL,232
ref TEXT NOT NULL,220
status TEXT NOT NULL,233
status TEXT NOT NULL,221
created_at TEXT NOT NULL,234
created_at TEXT NOT NULL,222
finished_at TEXT235
finished_at TEXT,236
source TEXT NOT NULL DEFAULT 'push',237
actor TEXT NOT NULL DEFAULT '',238
commit_title TEXT NOT NULL DEFAULT '',239
started_at TEXT223
);240
);224
CREATE TABLE IF NOT EXISTS jobs (241
CREATE TABLE IF NOT EXISTS jobs (225
id INTEGER PRIMARY KEY,242
id INTEGER PRIMARY KEY,@@ -240,6 +257,15 @@ CREATE TABLE IF NOT EXISTS jobs (240
log TEXT NOT NULL DEFAULT '',257
log TEXT NOT NULL DEFAULT '',241
started_at TEXT,258
started_at TEXT,242
finished_at TEXT,259
finished_at TEXT,260
before_script TEXT NOT NULL DEFAULT '',261
after_script TEXT NOT NULL DEFAULT '',262
when_mode TEXT NOT NULL DEFAULT 'on_success',263
timeout_seconds INTEGER NOT NULL DEFAULT 0,264
tags_json TEXT NOT NULL DEFAULT '[]',265
cache_paths TEXT NOT NULL DEFAULT '[]',266
cache_key TEXT NOT NULL DEFAULT '',267
resource_group TEXT NOT NULL DEFAULT '',268
optional_json TEXT NOT NULL DEFAULT '[]',243
UNIQUE(pipeline_id, name)269
UNIQUE(pipeline_id, name)244
);270
);245
CREATE TABLE IF NOT EXISTS artifacts (271
CREATE TABLE IF NOT EXISTS artifacts (