fix(dtrack-init): surface tool-install failures + generate API key via PUT#117
fix(dtrack-init): surface tool-install failures + generate API key via PUT#117dragonpaw wants to merge 2 commits into
Conversation
…a PUT
Three failure modes the current bootstrap silently buries:
1. **`apk add` errors are suppressed**. The Job wrapper runs
`apk add --no-cache curl jq >/dev/null 2>&1`. When the pod has no
egress to the alpine package mirrors (cluster egress firewall,
private node pool, etc.), the install fails, the script then runs
without `curl` installed, and every `curl -sf "$DT_URL/api/version"`
call is silently a no-op (sh "command not found" goes to stderr,
which `-sf` doesn't surface anyway). The wait loop burns its
5-minute budget, all 3 backoffs run, the Job goes Failed, and the
only signal is `[dtrack-init] ERROR: timeout`. Operator has no
idea the actual failure was at apk add.
Fix: drop the output redirect (so apk's own errors land in the pod
log) and add a `command -v curl jq` preflight in the script so the
downstream "curl never runs" symptom is also surfaced loudly.
2. **API key extraction breaks on DependencyTrack 4.12+**. The current
script reads `.apiKeys[0].key` from the team listing
(`GET /api/v1/team`). DT 4.12 redacts `.key` in the listing
response (only `.publicId` / `.maskedKey` remain) for security, so
the script gets an empty value, errors out with
`[dtrack-init] ERROR: no API key`, and the bootstrap is stuck on
anything past 4.11.
Fix: stop trying to read an existing key. Resolve the team's UUID
and generate a fresh one via `PUT /api/v1/team/{uuid}/key`. The
response body returns the unmasked secret -- this is the only way
to retrieve it on 4.12+, and works on 4.11 too (4.11 also accepted
POST without a body; 4.12+ returns 405 for POST and requires PUT +
Content-Type: application/json). The idempotency guard at the top
(`[ -f "$API_KEY_FILE" ] && [ -s "$API_KEY_FILE" ]`) prevents
minting a new key on every Job restart.
3. **NVD legacy feed mirror 404s on DependencyTrack 4.10+**. NIST
retired the file-based JSON feeds in late 2023:
$ curl -I https://services.nvd.nist.gov/rest/json/cves/2.0/json/cve/2.0/nvdcve-2.0-2019.meta
HTTP/2 404
DT 4.10+ replaced the legacy source with the NVD REST API 2.0;
`nvd.api.enabled=true` opts in. The chart was previously silent on
NVD config, so deployments inherited whatever the dtrack image
defaulted to (legacy feeds on 4.10+, → 404s → empty vulnerability
DB → empty findings on every scan).
Fix: POST `nvd.api.enabled=true` to `/api/v1/configProperty` during
bootstrap. Safe to set unconditionally -- DT ignores unknown
property names, so pre-4.10 images aren't affected.
Bonus, minor: switch the form-urlencoded body construction to
`curl --data-urlencode` per field. The `&`-joined string form worked
in 4.11 by coincidence (passwords in our deploys had no `&` / `=` /
unicode), but breaks the moment an operator sets a password containing
those. `--data-urlencode` per field is the boring-correct shape.
Validated against a self-hosted DT 4.11.4 deploy on dev-shared-gke
(my org's AK install): force-changed admin/admin -> the configured
password, generated a fresh Automation API key, and the backend
picked it up at next start. Integration `/api/v1/dependency-track/status`
moved from `healthy: false` to `healthy: true`.
|
Thanks for this, the diagnosis and the three fixes (un-silencing Holding on the API-key change before merge: switching retrieval to The other two dtrack PRs (#113, #114) look mergeable; this one just needs that validation. (The red SonarCloud check is a known non-blocking fork limitation and can be ignored.) |
Validation against live Dependency-Track 4.11.4 / 4.12.7 / 4.13.0 /
4.14.2 (fresh installs) corrected two claims in the bootstrap comments:
- POST /api/v1/team/{uuid}/key returns 405 on *all* tested versions,
not just 4.12+. DT only accepts POST .../key/{key} to regenerate an
existing key; PUT is the create/generate verb on every 4.x release.
So PUT is the correct version-agnostic path, not a 4.12 workaround.
- The team-listing `.apiKeys[*].key` redaction starts at 4.13.0, not
4.12 — 4.11.4 and 4.12.7 still return the full key. Generating a
fresh key via PUT remains the right approach on every version.
Code unchanged; comments only.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Validated the PUT path against live Dependency-Track 4.11.4, 4.12.7, 4.13.0, and 4.14.2 (embedded H2, fresh installs):
So Two corrections I just pushed to the branch (comments only, no code change):
Good to go on your end? |
Summary
Three failure modes in the
dtrack-initbootstrap Job, all silent or only visible as[dtrack-init] ERROR: timeout. All three bit me on a stock dev-shared-gke deploy of1.1.9and required manual recovery to get the backend ↔ DependencyTrack integration working.1.
apk adderrors are silently droppedThe Job's wrapper runs:
When the pod has no egress to the alpine package mirrors (cluster egress firewall, private node pool, IP allowlist on the node, etc.),
apk addfails. The redirect swallows its error. The script then runs withoutcurlinstalled. Everycurl -sf "$DT_URL/api/version"is silently a no-op —sh: curl: command not foundgoes to stderr, which-sfdoesn't surface anyway. The 60×5s wait loop burns its full budget, the Job marks Failed after 3 backoffs, and the only signal is[dtrack-init] ERROR: timeout— pointing at the wrong layer.Fix: drop
>/dev/null 2>&1so apk's own error message lands in the pod log, and add acommand -v curl jqpreflight at the top of the script so the downstream "binaries missing" symptom is also surfaced loudly. Either of the two now produces a useful diagnostic; both together make this debuggable without source-diving.2. API key extraction breaks on DependencyTrack 4.12+
The current script reads
.apiKeys[0].keyfrom the team listing:DT 4.12 redacts
.keyin the listing response (only.publicId/.maskedKeyremain — security hardening so anyone with read access on/api/v1/teamcan't enumerate keys). The script's jq selector returns empty, errors out with[dtrack-init] ERROR: no API key, and the bootstrap is permanently stuck on anything past 4.11.Fix: resolve the team's UUID from the listing, then
PUT /api/v1/team/{uuid}/keyto generate a fresh key. The response body returns the unmasked secret — this is the only way to retrieve it on 4.12+. PUT also works on 4.11 (which previously accepted POST without a body); POST on 4.12+ returns 405 Method Not Allowed, so PUT is the strict superset. Same fix shape as #1231 just applied to the K8s chart instead of docker-compose.The idempotency guard at the top of the script (
[ -f \"$API_KEY_FILE\" ] && [ -s \"$API_KEY_FILE\" ]) prevents this from minting a new key on every Job restart.3. NVD legacy feed mirror 404s on DependencyTrack 4.10+
NIST retired the file-based JSON feeds in late 2023:
DT 4.10+ replaced the legacy source with the NVD REST API 2.0;
nvd.api.enabled=trueopts in. The chart was previously silent on NVD config, so deployments inherited whatever the dtrack image defaulted to — which on 4.10+ is the now-dead legacy feeds → 404s on every scheduled poll → empty vulnerability DB → empty findings on every scan, even when the integration is otherwise healthy.Fix: POST
nvd.api.enabled=trueto/api/v1/configPropertyduring bootstrap. Safe to set unconditionally — DT silently ignores unknown property names, so pre-4.10 images aren't affected. Same fix as the one #1231 added to the docker-composeinit-dtrack.sh, just in the chart's templated copy.Bonus (minor): form-urlencoded body construction
Switch the password-bearing curl calls to
--data-urlencodeper field. The&-joined string form worked in 4.11 only because passwords in test deploys happened to contain no&/=/ unicode.--data-urlencodeper field is the boring-correct shape and avoids a foot-gun whenever an operator sets a password containing those characters.Tested
Validated against
1.1.9(DT 4.11.4) on dev-shared-gke:/api/v1/dependency-track/statushealthy: false.helm templateand the script logic re-confirmed against the same dtrack 4.11.4 instance.I do not yet have a 4.12+ install to validate the PUT path against the live API, but the change is conservatively the upstream PR #1231 fix applied to the chart side; community feedback welcome if anyone has a 4.12+ deploy to confirm.
Related
artifact-keeper/artifact-keeper#1231— sameapiKeys[*].key+ NVD config fixes, applied to docker-compose'sdocker/init-dtrack.sh. This PR ports the equivalent fixes to the helm chart.artifact-keeper/artifact-keeper#1308— root-cause writeup for the dtrack-4.12 API drift and NVD feed deprecation that motivated #1231.Out of scope
The dtrack-init Job needs
ak-shared-config(RWO) to write the API key, which the backend Deployment also mounts. On clusters with cross-node pod scheduling, the Job pod can't attach the PVC until backend releases it (Multi-Attach error), and there's no podAffinity to keep them co-located. Workaround for now is to either scale backend to 0 during first init, or set adependencyTrack.bootstrap.affinitythat pins the Job to the backend's node. Worth a follow-up: either inject podAffinity automatically (same pattern as #7), or write the key to a Secret the backend reads viavalueFrom.secretKeyRef(removes the shared-volume requirement entirely).Closes #154