-
-
Notifications
You must be signed in to change notification settings - Fork 1
Expand file tree
/
Copy pathci-cd.html
More file actions
496 lines (436 loc) · 28 KB
/
Copy pathci-cd.html
File metadata and controls
496 lines (436 loc) · 28 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>CI/CD recipes — SpecTracer</title>
<meta name="description" content="Copy-paste SpecTracer pipelines for GitHub Actions, GitLab CI, and Jenkins — plus PR comments, build gating with fail_on, and posting coverage trends to your own dashboard.">
<link rel="icon" href="logo.png">
<link rel="stylesheet" href="css/style.css">
<script>(function(){var t;try{t=localStorage.getItem('st-theme')}catch(e){}document.documentElement.setAttribute('data-theme',t||(window.matchMedia('(prefers-color-scheme:dark)').matches?'dark':'light'))})();</script>
</head>
<body>
<a class="skip-link" href="#main">Skip to content</a>
<nav class="nav">
<div class="nav-inner">
<a href="index.html" class="nav-logo"><img src="logo.png" alt="">SpecTracer</a>
<ul class="nav-links">
<li><a href="getting-started.html">Docs</a></li>
<li><a href="report.html">Report tour</a></li>
<li><a href="configuration.html">Configuration</a></li>
<li><a href="ci-cd.html" class="active">CI/CD</a></li>
</ul>
<button class="icon-btn theme-toggle" aria-label="Toggle colour theme">
<svg class="icon-sun" width="17" height="17" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round"><circle cx="12" cy="12" r="4"/><path d="M12 2v2M12 20v2M4.9 4.9l1.4 1.4M17.7 17.7l1.4 1.4M2 12h2M20 12h2M4.9 19.1l1.4-1.4M17.7 6.3l1.4-1.4"/></svg>
<svg class="icon-moon" width="17" height="17" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M21 12.79A9 9 0 1 1 11.21 3 7 7 0 0 0 21 12.79z"/></svg>
</button>
<button class="icon-btn nav-toggle" aria-label="Toggle navigation" aria-expanded="false">
<svg width="17" height="17" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round"><path d="M3 6h18M3 12h18M3 18h18"/></svg>
</button>
<a href="https://github.com/ampyard/spec-tracer" class="nav-cta">GitHub</a>
</div>
</nav>
<div class="docs">
<aside class="sidebar">
<nav class="sidebar-nav" aria-label="Documentation">
<div class="sidebar-group">
<span class="sidebar-label">Start here</span>
<a href="getting-started.html">Getting started</a>
<a href="workflow.html">Declare → Tag → Run</a>
</div>
<div class="sidebar-group">
<span class="sidebar-label">Core concepts</span>
<a href="tagging.html">Tagging model</a>
<a href="configuration.html">Configuration</a>
</div>
<div class="sidebar-group">
<span class="sidebar-label">Using the output</span>
<a href="report.html">Report tour</a>
<a href="ci-cd.html" class="active">CI/CD recipes</a>
<div class="sub">
<a href="#shape">The shape of it</a>
<a href="#github">GitHub Actions</a>
<a href="#pr">PR comments</a>
<a href="#gating">Gating the build</a>
<a href="#trends">Historical trends</a>
<a href="#gitlab">GitLab CI</a>
<a href="#jenkins">Jenkins</a>
<a href="#rollout">Rollout playbook</a>
</div>
</div>
<div class="sidebar-group">
<span class="sidebar-label">Reference</span>
<a href="architecture.html">How it works</a>
<a href="troubleshooting.html">Troubleshooting & FAQ</a>
</div>
</nav>
</aside>
<main class="doc" id="main">
<span class="eyebrow">Using the output</span>
<h1>CI/CD recipes</h1>
<p class="lede">SpecTracer is a CLI that reads files and writes files. There's no agent, no service,
and no state — which makes CI integration almost boring. These are working pipelines you can paste
and adjust.</p>
<h2 id="shape">The shape of it</h2>
<p>Every integration, on every platform, is the same four moves:</p>
<div class="steps" style="margin:1.5rem 0">
<div class="step">
<div class="step-num">1</div>
<div class="step-body"><h3>Run your suites with machine-readable output</h3>
<p>JUnit XML for unit and integration, Cucumber JSON for E2E. Most runners already do this, or need one flag.</p></div>
</div>
<div class="step">
<div class="step-num">2</div>
<div class="step-body"><h3>Make sure all the files land in one workspace</h3>
<p>If your suites run in parallel jobs, that means artifacts up and artifacts down. This is the only genuinely fiddly part.</p></div>
</div>
<div class="step">
<div class="step-num">3</div>
<div class="step-body"><h3>Run <code>spec-tracer</code></h3>
<p>One command. It exits <code>1</code> if a gate you configured is breached.</p></div>
</div>
<div class="step">
<div class="step-num">4</div>
<div class="step-body"><h3>Publish the report</h3>
<p>Upload the HTML as an artifact; optionally read the JSON to comment on the PR or post metrics.</p></div>
</div>
</div>
<div class="note">
<strong>Don't skip step 2's implications</strong>
Tests that fail to run produce no results file, and a missing results file is silently treated as
"zero tests for that layer" — not as an error. A crashed E2E job can therefore make coverage
<em>drop</em> without failing the report step. Make the test jobs themselves fail loudly.
</div>
<h2 id="github">GitHub Actions</h2>
<p>Single-job version — simplest, and correct for most repos.</p>
<div class="code">
<div class="code-head"><span>.github/workflows/ci.yml</span><button class="copy-btn" aria-label="Copy to clipboard"><svg class="clip" width="14" height="14" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><rect x="9" y="9" width="12" height="12" rx="2"/><path d="M5 15H4a2 2 0 0 1-2-2V4a2 2 0 0 1 2-2h9a2 2 0 0 1 2 2v1"/></svg><svg class="check" width="14" height="14" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2.5" stroke-linecap="round" stroke-linejoin="round"><path d="M20 6L9 17l-5-5"/></svg></button></div>
<pre><code><span class="c-fn">name:</span> CI
<span class="c-fn">on:</span> [push, pull_request]
<span class="c-fn">jobs:</span>
<span class="c-fn">test-and-report:</span>
<span class="c-fn">runs-on:</span> ubuntu-latest
<span class="c-fn">steps:</span>
- <span class="c-fn">uses:</span> actions/checkout@v4
- <span class="c-fn">uses:</span> astral-sh/setup-uv@v5
<span class="c-fn">with:</span>
<span class="c-fn">python-version:</span> <span class="c-str">"3.12"</span>
- <span class="c-fn">name:</span> Install dependencies
<span class="c-fn">run:</span> uv sync
- <span class="c-fn">name:</span> Unit tests
<span class="c-fn">run:</span> uv run pytest tests/unit --junitxml=reports/unit.xml
- <span class="c-fn">name:</span> Integration tests
<span class="c-fn">run:</span> uv run pytest tests/integration --junitxml=reports/int.xml
- <span class="c-fn">name:</span> E2E tests
<span class="c-fn">run:</span> uv run behave features/ --format json -o reports/e2e.json
- <span class="c-fn">name:</span> Generate SpecTracer report
<span class="c-fn">run:</span> uv run spec-tracer
- <span class="c-fn">name:</span> Upload report
<span class="c-fn">if:</span> always()
<span class="c-fn">uses:</span> actions/upload-artifact@v4
<span class="c-fn">with:</span>
<span class="c-fn">name:</span> spectracer-report
<span class="c-fn">path:</span> |
reports/spectracer-report.html
reports/spectracer-report.json</code></pre>
</div>
<div class="note good">
<strong>Note the <code>if: always()</code> on the upload</strong>
Without it, a gated <code>spec-tracer</code> run that exits <code>1</code> skips the upload — and
you lose the report exactly when you most need to read it.
</div>
<div class="code">
<div class="code-head"><span>spectracer.config.json to go with it</span><button class="copy-btn" aria-label="Copy to clipboard"><svg class="clip" width="14" height="14" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><rect x="9" y="9" width="12" height="12" rx="2"/><path d="M5 15H4a2 2 0 0 1-2-2V4a2 2 0 0 1 2-2h9a2 2 0 0 1 2 2v1"/></svg><svg class="check" width="14" height="14" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2.5" stroke-linecap="round" stroke-linejoin="round"><path d="M20 6L9 17l-5-5"/></svg></button></div>
<pre><code>{
<span class="c-fn">"features"</span>: [<span class="c-str">"./features"</span>],
<span class="c-fn">"unit"</span>: { <span class="c-str">""</span>: [<span class="c-str">"./reports/unit.xml"</span>] },
<span class="c-fn">"integration"</span>: { <span class="c-str">""</span>: [<span class="c-str">"./reports/int.xml"</span>] },
<span class="c-fn">"e2e"</span>: { <span class="c-str">""</span>: [<span class="c-str">"./reports/e2e.json"</span>] },
<span class="c-fn">"output"</span>: <span class="c-str">"./reports/spectracer-report.html"</span>,
<span class="c-fn">"output_json"</span>: <span class="c-str">"./reports/spectracer-report.json"</span>
}</code></pre>
</div>
<h3>Parallel test jobs</h3>
<p>If unit, integration, and E2E run as separate jobs, each uploads its results and a final job
downloads them all into one workspace before running SpecTracer.</p>
<div class="code">
<div class="code-head"><span>.github/workflows/ci.yml — report job</span><button class="copy-btn" aria-label="Copy to clipboard"><svg class="clip" width="14" height="14" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><rect x="9" y="9" width="12" height="12" rx="2"/><path d="M5 15H4a2 2 0 0 1-2-2V4a2 2 0 0 1 2-2h9a2 2 0 0 1 2 2v1"/></svg><svg class="check" width="14" height="14" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2.5" stroke-linecap="round" stroke-linejoin="round"><path d="M20 6L9 17l-5-5"/></svg></button></div>
<pre><code> <span class="c-fn">report:</span>
<span class="c-fn">needs:</span> [unit, integration, e2e]
<span class="c-fn">if:</span> always() <span class="c-com"># still report when a suite failed</span>
<span class="c-fn">runs-on:</span> ubuntu-latest
<span class="c-fn">steps:</span>
- <span class="c-fn">uses:</span> actions/checkout@v4
- <span class="c-fn">uses:</span> astral-sh/setup-uv@v5
- <span class="c-fn">name:</span> Collect every suite's results
<span class="c-fn">uses:</span> actions/download-artifact@v4
<span class="c-fn">with:</span>
<span class="c-fn">path:</span> reports
<span class="c-fn">merge-multiple:</span> <span class="c-num">true</span>
- <span class="c-fn">run:</span> uv sync && uv run spec-tracer</code></pre>
</div>
<p>Point the config at the <em>directories</em> rather than individual files so it doesn't need to
know how many artifacts arrived:</p>
<div class="code">
<div class="code-head"><span>spectracer.config.json</span></div>
<pre><code> <span class="c-fn">"unit"</span>: { <span class="c-str">""</span>: [<span class="c-str">"./reports"</span>] }, <span class="c-com">// every .xml found, recursively</span>
<span class="c-fn">"e2e"</span>: { <span class="c-str">""</span>: [<span class="c-str">"./reports"</span>] }, <span class="c-com">// every .json found, recursively</span></code></pre>
</div>
<div class="note warn">
<strong>Careful with a shared directory</strong>
Scanning the same directory for both <code>unit</code> and <code>integration</code> registers every
XML file under <em>both</em> layers and double-counts your pyramid. Give each layer its own
subdirectory (<code>reports/unit</code>, <code>reports/integration</code>) whenever more than one
layer emits the same file extension.
</div>
<h2 id="pr">PR comments</h2>
<p>Two levels. Link to the artifact, or read the JSON and post the actual numbers.</p>
<h3>Link to the report</h3>
<div class="code">
<div class="code-head"><span>.github/workflows/ci.yml</span><button class="copy-btn" aria-label="Copy to clipboard"><svg class="clip" width="14" height="14" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><rect x="9" y="9" width="12" height="12" rx="2"/><path d="M5 15H4a2 2 0 0 1-2-2V4a2 2 0 0 1 2-2h9a2 2 0 0 1 2 2v1"/></svg><svg class="check" width="14" height="14" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2.5" stroke-linecap="round" stroke-linejoin="round"><path d="M20 6L9 17l-5-5"/></svg></button></div>
<pre><code> - <span class="c-fn">name:</span> Comment PR
<span class="c-fn">if:</span> github.event_name == 'pull_request'
<span class="c-fn">uses:</span> actions/github-script@v7
<span class="c-fn">with:</span>
<span class="c-fn">script:</span> |
github.rest.issues.createComment({
issue_number: context.issue.number,
owner: context.repo.owner,
repo: context.repo.repo,
body: `## SpecTracer coverage report\n\n[Download the report](${context.serverUrl}/${context.repo.owner}/${context.repo.repo}/actions/runs/${context.runId})`
})</code></pre>
</div>
<h3>Post the numbers</h3>
<p>Better, because it shows up in the timeline without anyone downloading anything. Requires
<code>output_json</code>.</p>
<div class="code">
<div class="code-head"><span>.github/workflows/ci.yml</span><button class="copy-btn" aria-label="Copy to clipboard"><svg class="clip" width="14" height="14" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><rect x="9" y="9" width="12" height="12" rx="2"/><path d="M5 15H4a2 2 0 0 1-2-2V4a2 2 0 0 1 2-2h9a2 2 0 0 1 2 2v1"/></svg><svg class="check" width="14" height="14" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2.5" stroke-linecap="round" stroke-linejoin="round"><path d="M20 6L9 17l-5-5"/></svg></button></div>
<pre><code> - <span class="c-fn">name:</span> Comment PR with completion
<span class="c-fn">if:</span> always() && github.event_name == 'pull_request'
<span class="c-fn">env:</span>
<span class="c-fn">GH_TOKEN:</span> ${{ secrets.GITHUB_TOKEN }}
<span class="c-fn">run:</span> |
REPORT=reports/spectracer-report.json
PERCENT=$(jq -r '.summary.completion.percent' "$REPORT")
HEALTH=$(jq -r '.summary.health.status' "$REPORT")
REASONS=$(jq -r '.summary.health.reasons[]? | "- " + .' "$REPORT")
gh pr comment ${{ github.event.pull_request.number }} --body "$(cat <<EOF
**SpecTracer** · declared tests matched: **${PERCENT}%** · health: **${HEALTH}**
${REASONS}
EOF
)"</code></pre>
</div>
<h2 id="gating">Gating the build</h2>
<p>SpecTracer ships two gates, and you can build any third one you need on top of the JSON.</p>
<div class="table-scroll">
<table>
<thead><tr><th>Gate</th><th>Config</th><th>Fails when</th></tr></thead>
<tbody>
<tr><td>Test failures</td><td><code>"error_on_failure": true</code></td><td>Any collected result is a failure.</td></tr>
<tr><td>Coverage floor</td><td><code>"fail_on": ["progress"]</code></td><td>Declared-tests-matched drops below <code>progress_threshold_amber</code>.</td></tr>
<tr><td>Inverted pyramid</td><td><code>"fail_on": ["pyramid"]</code></td><td>Unit count is below integration + E2E combined.</td></tr>
<tr><td>Slow E2E suite</td><td><code>"fail_on": ["e2e_runtime"]</code></td><td>Total E2E time exceeds <code>e2e_duration_red_seconds</code>.</td></tr>
</tbody>
</table>
</div>
<p>Amber never gates — only red does. All gates are additive: any one of them exiting non-zero fails
the step.</p>
<h3>A custom gate</h3>
<p>For anything the built-ins don't cover — an absolute coverage floor, a no-regression rule — read
the JSON:</p>
<div class="code">
<div class="code-head"><span>fail if completion is under 75%</span><button class="copy-btn" aria-label="Copy to clipboard"><svg class="clip" width="14" height="14" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><rect x="9" y="9" width="12" height="12" rx="2"/><path d="M5 15H4a2 2 0 0 1-2-2V4a2 2 0 0 1 2-2h9a2 2 0 0 1 2 2v1"/></svg><svg class="check" width="14" height="14" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2.5" stroke-linecap="round" stroke-linejoin="round"><path d="M20 6L9 17l-5-5"/></svg></button></div>
<pre><code> - <span class="c-fn">name:</span> Enforce coverage floor
<span class="c-fn">run:</span> |
PERCENT=$(jq -r '.summary.completion.percent' reports/spectracer-report.json)
<span class="c-key">if</span> (( $(echo "$PERCENT < 75" | bc -l) )); <span class="c-key">then</span>
echo "::error::Scenario coverage ${PERCENT}% is below the 75% floor"
exit 1
<span class="c-key">fi</span></code></pre>
</div>
<div class="note bad">
<strong>Don't scrape the HTML</strong>
The HTML report's markup is not a stable interface and will change between versions. The JSON is
governed by a published JSON Schema; script against that.
</div>
<h2 id="trends">Historical trends</h2>
<p>SpecTracer stores nothing between runs — deliberately. Every run emits a
<code>summary.completion</code> / <code>summary.pyramid</code> / <code>summary.health</code>
snapshot, and whatever metrics system you already run owns the history, retention, and charting.</p>
<div class="code">
<div class="code-head"><span>.github/workflows/ci.yml</span><button class="copy-btn" aria-label="Copy to clipboard"><svg class="clip" width="14" height="14" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><rect x="9" y="9" width="12" height="12" rx="2"/><path d="M5 15H4a2 2 0 0 1-2-2V4a2 2 0 0 1 2-2h9a2 2 0 0 1 2 2v1"/></svg><svg class="check" width="14" height="14" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2.5" stroke-linecap="round" stroke-linejoin="round"><path d="M20 6L9 17l-5-5"/></svg></button></div>
<pre><code> - <span class="c-fn">name:</span> Generate report
<span class="c-fn">run:</span> uv run spec-tracer
- <span class="c-fn">name:</span> Post coverage metrics
<span class="c-fn">env:</span>
<span class="c-fn">METRICS_ENDPOINT:</span> ${{ secrets.METRICS_ENDPOINT }}
<span class="c-fn">run:</span> |
curl -X POST "$METRICS_ENDPOINT" \
-H "Content-Type: application/json" \
-d @reports/spectracer-report.json</code></pre>
</div>
<p>That's the whole integration: one request reading a file that already exists. Swap the
<code>curl</code> for your dashboard's SDK if it has one — Datadog, Grafana Cloud, an internal
service, even a spreadsheet webhook. The only requirement on the far end is something that can store
a time series and draw it.</p>
<h2 id="gitlab">GitLab CI</h2>
<div class="code">
<div class="code-head"><span>.gitlab-ci.yml</span><button class="copy-btn" aria-label="Copy to clipboard"><svg class="clip" width="14" height="14" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><rect x="9" y="9" width="12" height="12" rx="2"/><path d="M5 15H4a2 2 0 0 1-2-2V4a2 2 0 0 1 2-2h9a2 2 0 0 1 2 2v1"/></svg><svg class="check" width="14" height="14" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2.5" stroke-linecap="round" stroke-linejoin="round"><path d="M20 6L9 17l-5-5"/></svg></button></div>
<pre><code><span class="c-fn">stages:</span>
- test
- report
<span class="c-fn">variables:</span>
<span class="c-fn">UV_PYTHON:</span> <span class="c-str">"3.12"</span>
<span class="c-fn">unit-tests:</span>
<span class="c-fn">stage:</span> test
<span class="c-fn">script:</span>
- uv run pytest tests/unit --junitxml=reports/unit.xml
<span class="c-fn">artifacts:</span>
<span class="c-fn">when:</span> always
<span class="c-fn">paths:</span> [reports/unit.xml]
<span class="c-fn">integration-tests:</span>
<span class="c-fn">stage:</span> test
<span class="c-fn">script:</span>
- uv run pytest tests/integration --junitxml=reports/int.xml
<span class="c-fn">artifacts:</span>
<span class="c-fn">when:</span> always
<span class="c-fn">paths:</span> [reports/int.xml]
<span class="c-fn">e2e-tests:</span>
<span class="c-fn">stage:</span> test
<span class="c-fn">script:</span>
- uv run behave features/ --format json -o reports/e2e.json
<span class="c-fn">artifacts:</span>
<span class="c-fn">when:</span> always
<span class="c-fn">paths:</span> [reports/e2e.json]
<span class="c-fn">spectracer-report:</span>
<span class="c-fn">stage:</span> report
<span class="c-fn">when:</span> always
<span class="c-fn">needs:</span> [unit-tests, integration-tests, e2e-tests]
<span class="c-fn">script:</span>
- uv sync
- uv run spec-tracer
<span class="c-fn">artifacts:</span>
<span class="c-fn">when:</span> always
<span class="c-fn">paths:</span> [reports/spectracer-report.html]
<span class="c-fn">expose_as:</span> Coverage Report</code></pre>
</div>
<p><code>expose_as</code> puts a direct link to the report on the merge request page.</p>
<h2 id="jenkins">Jenkins</h2>
<div class="code">
<div class="code-head"><span>Jenkinsfile (declarative)</span><button class="copy-btn" aria-label="Copy to clipboard"><svg class="clip" width="14" height="14" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><rect x="9" y="9" width="12" height="12" rx="2"/><path d="M5 15H4a2 2 0 0 1-2-2V4a2 2 0 0 1 2-2h9a2 2 0 0 1 2 2v1"/></svg><svg class="check" width="14" height="14" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2.5" stroke-linecap="round" stroke-linejoin="round"><path d="M20 6L9 17l-5-5"/></svg></button></div>
<pre><code><span class="c-fn">pipeline</span> {
<span class="c-fn">agent</span> any
<span class="c-fn">stages</span> {
<span class="c-fn">stage</span>(<span class="c-str">'Install'</span>) {
<span class="c-fn">steps</span> { sh <span class="c-str">'uv sync'</span> }
}
<span class="c-fn">stage</span>(<span class="c-str">'Run tests'</span>) {
<span class="c-fn">parallel</span> {
<span class="c-fn">stage</span>(<span class="c-str">'Unit'</span>) {
<span class="c-fn">steps</span> { sh <span class="c-str">'uv run pytest tests/unit --junitxml=reports/unit.xml'</span> }
}
<span class="c-fn">stage</span>(<span class="c-str">'Integration'</span>) {
<span class="c-fn">steps</span> { sh <span class="c-str">'uv run pytest tests/integration --junitxml=reports/int.xml'</span> }
}
<span class="c-fn">stage</span>(<span class="c-str">'E2E'</span>) {
<span class="c-fn">steps</span> { sh <span class="c-str">'uv run behave features/ --format json -o reports/e2e.json'</span> }
}
}
}
<span class="c-fn">stage</span>(<span class="c-str">'SpecTracer report'</span>) {
<span class="c-fn">steps</span> { sh <span class="c-str">'uv run spec-tracer'</span> }
}
}
<span class="c-fn">post</span> {
<span class="c-fn">always</span> {
<span class="c-fn">archiveArtifacts</span> artifacts: <span class="c-str">'reports/spectracer-report.*'</span>,
allowEmptyArchive: <span class="c-num">true</span>,
fingerprint: <span class="c-num">true</span>
}
}
}</code></pre>
</div>
<h2 id="rollout">Rollout playbook</h2>
<p>Getting the pipeline right is the easy half. Getting a team to care about the number is the other
half.</p>
<div class="grid-2">
<div class="card">
<span class="card-mark unit">1</span>
<h3>Write the feature files first</h3>
<p>Whoever owns requirements — QA, product, tech lead — writes the <code>.feature</code> files
and assigns each scenario an <code>@id:</code>. This is the scope definition, and it should
happen before any tagging of tests.</p>
</div>
<div class="card">
<span class="card-mark unit">2</span>
<h3>Tag tests opportunistically</h3>
<p>Don't schedule a tagging sprint. Add <code>@scenario:</code> tags as people touch tests
anyway, and require them on new tests in code review. The number climbs on its own.</p>
</div>
<div class="card">
<span class="card-mark unit">3</span>
<h3>Publish the report before gating it</h3>
<p>Upload the artifact on every build for a sprint or two. Let people see their own number move
before it can block them.</p>
</div>
<div class="card">
<span class="card-mark unit">4</span>
<h3>Gate, then raise the bar</h3>
<p>Turn on <code>error_on_failure</code> and the health checks you care about. Ratchet
<code>progress_threshold_amber</code> upward as the real number climbs past it.</p>
</div>
</div>
<div class="note good">
<strong>Commit the config, ignore the report</strong>
<code>spectracer.config.json</code> is your coverage contract and belongs in version control. The
generated HTML and JSON are build artifacts — add them to <code>.gitignore</code>.
</div>
<div class="note">
<strong>Microservices in one repo</strong>
Give each service a module key and have its scenarios declare
<code>@require-unit:<service></code>. Only that service's results will satisfy the
requirement, so one team's coverage can't paper over another's. See
<a href="tagging.html#module-scope">module scope</a>.
</div>
<nav class="doc-nextprev">
<a href="report.html"><span class="dir">Previous</span><span class="ttl">← Report tour</span></a>
<a href="architecture.html" class="next"><span class="dir">Next</span><span class="ttl">How it works →</span></a>
</nav>
</main>
</div>
<footer class="footer">
<div class="footer-grid">
<div>
<a href="index.html" class="footer-brand"><img src="logo.png" alt="">SpecTracer</a>
<p class="footer-blurb">Gherkin-driven test traceability across unit, integration, and E2E. Built by Ampyard.</p>
</div>
<div class="footer-col">
<h5>Docs</h5>
<a href="getting-started.html">Getting started</a>
<a href="workflow.html">Declare → Tag → Run</a>
<a href="tagging.html">Tagging model</a>
<a href="configuration.html">Configuration</a>
<a href="report.html">Report tour</a>
</div>
<div class="footer-col">
<h5>Reference</h5>
<a href="ci-cd.html">CI/CD recipes</a>
<a href="architecture.html">How it works</a>
<a href="troubleshooting.html">Troubleshooting</a>
</div>
<div class="footer-col">
<h5>Project</h5>
<a href="https://github.com/ampyard/spec-tracer">GitHub</a>
<a href="https://pypi.org/project/spec-tracer/">PyPI</a>
<a href="https://github.com/ampyard/spec-tracer/issues">Issues</a>
<a href="https://github.com/ampyard/spec-tracer/blob/main/LICENSE">MIT License</a>
</div>
</div>
<div class="footer-bottom">
<span>© 2026 Ampyard · MIT Licensed</span>
<span>Not a test runner. Not a coverage tool. A traceability report.</span>
</div>
</footer>
<script src="js/site.js" defer></script>
</body>
</html>