Skip to content

fix(dtrack-init): surface tool-install failures + generate API key via PUT#117

Open
dragonpaw wants to merge 2 commits into
artifact-keeper:mainfrom
dragonpaw:fix/dtrack-init-script-robustness
Open

fix(dtrack-init): surface tool-install failures + generate API key via PUT#117
dragonpaw wants to merge 2 commits into
artifact-keeper:mainfrom
dragonpaw:fix/dtrack-init-script-robustness

Conversation

@dragonpaw

@dragonpaw dragonpaw commented May 21, 2026

Copy link
Copy Markdown
Contributor

Summary

Three failure modes in the dtrack-init bootstrap Job, all silent or only visible as [dtrack-init] ERROR: timeout. All three bit me on a stock dev-shared-gke deploy of 1.1.9 and required manual recovery to get the backend ↔ DependencyTrack integration working.

1. apk add errors are silently dropped

The Job's wrapper runs:

args:
  - |
    apk add --no-cache curl jq >/dev/null 2>&1
    /bin/sh /scripts/init-dtrack.sh

When the pod has no egress to the alpine package mirrors (cluster egress firewall, private node pool, IP allowlist on the node, etc.), apk add fails. The redirect swallows its error. The script then runs without curl installed. Every curl -sf "$DT_URL/api/version" is silently a no-op — sh: curl: command not found goes to stderr, which -sf doesn't surface anyway. The 60×5s wait loop burns its full budget, the Job marks Failed after 3 backoffs, and the only signal is [dtrack-init] ERROR: timeout — pointing at the wrong layer.

Fix: drop >/dev/null 2>&1 so apk's own error message lands in the pod log, and add a command -v curl jq preflight at the top of the script so the downstream "binaries missing" symptom is also surfaced loudly. Either of the two now produces a useful diagnostic; both together make this debuggable without source-diving.

2. API key extraction breaks on DependencyTrack 4.12+

The current script reads .apiKeys[0].key from the team listing:

API_KEY=$(curl -sf "$DT_URL/api/v1/team" \
  -H "Authorization: Bearer $TOKEN" | \
  jq -r '.[] | select(.name == "Automation") | .apiKeys[0].key // empty')

DT 4.12 redacts .key in the listing response (only .publicId / .maskedKey remain — security hardening so anyone with read access on /api/v1/team can't enumerate keys). The script's jq selector returns empty, errors out with [dtrack-init] ERROR: no API key, and the bootstrap is permanently stuck on anything past 4.11.

Fix: resolve the team's UUID from the listing, then PUT /api/v1/team/{uuid}/key to generate a fresh key. The response body returns the unmasked secret — this is the only way to retrieve it on 4.12+. PUT also works on 4.11 (which previously accepted POST without a body); POST on 4.12+ returns 405 Method Not Allowed, so PUT is the strict superset. Same fix shape as #1231 just applied to the K8s chart instead of docker-compose.

The idempotency guard at the top of the script ([ -f \"$API_KEY_FILE\" ] && [ -s \"$API_KEY_FILE\" ]) prevents this from minting a new key on every Job restart.

3. NVD legacy feed mirror 404s on DependencyTrack 4.10+

NIST retired the file-based JSON feeds in late 2023:

$ curl -I https://services.nvd.nist.gov/rest/json/cves/2.0/json/cve/2.0/nvdcve-2.0-2019.meta
HTTP/2 404

DT 4.10+ replaced the legacy source with the NVD REST API 2.0; nvd.api.enabled=true opts in. The chart was previously silent on NVD config, so deployments inherited whatever the dtrack image defaulted to — which on 4.10+ is the now-dead legacy feeds → 404s on every scheduled poll → empty vulnerability DB → empty findings on every scan, even when the integration is otherwise healthy.

Fix: POST nvd.api.enabled=true to /api/v1/configProperty during bootstrap. Safe to set unconditionally — DT silently ignores unknown property names, so pre-4.10 images aren't affected. Same fix as the one #1231 added to the docker-compose init-dtrack.sh, just in the chart's templated copy.

Bonus (minor): form-urlencoded body construction

Switch the password-bearing curl calls to --data-urlencode per field. The &-joined string form worked in 4.11 only because passwords in test deploys happened to contain no & / = / unicode. --data-urlencode per field is the boring-correct shape and avoids a foot-gun whenever an operator sets a password containing those characters.

Tested

Validated against 1.1.9 (DT 4.11.4) on dev-shared-gke:

  • Pre-fix: Job timed out (Increase Meilisearch memory limit and cap indexing threads #1 path) every 3 backoffs over 2.5+ days; backend reported /api/v1/dependency-track/status healthy: false.
  • Manual recovery (the bootstrap done by hand via port-forward to dtrack) confirmed the rest of the flow works end-to-end and the backend picks up the key on next start.
  • Post-fix script renders cleanly via helm template and the script logic re-confirmed against the same dtrack 4.11.4 instance.

I do not yet have a 4.12+ install to validate the PUT path against the live API, but the change is conservatively the upstream PR #1231 fix applied to the chart side; community feedback welcome if anyone has a 4.12+ deploy to confirm.

Related

  • artifact-keeper/artifact-keeper#1231 — same apiKeys[*].key + NVD config fixes, applied to docker-compose's docker/init-dtrack.sh. This PR ports the equivalent fixes to the helm chart.
  • artifact-keeper/artifact-keeper#1308 — root-cause writeup for the dtrack-4.12 API drift and NVD feed deprecation that motivated #1231.

Out of scope

The dtrack-init Job needs ak-shared-config (RWO) to write the API key, which the backend Deployment also mounts. On clusters with cross-node pod scheduling, the Job pod can't attach the PVC until backend releases it (Multi-Attach error), and there's no podAffinity to keep them co-located. Workaround for now is to either scale backend to 0 during first init, or set a dependencyTrack.bootstrap.affinity that pins the Job to the backend's node. Worth a follow-up: either inject podAffinity automatically (same pattern as #7), or write the key to a Secret the backend reads via valueFrom.secretKeyRef (removes the shared-volume requirement entirely).


Closes #154

…a PUT

Three failure modes the current bootstrap silently buries:

1. **`apk add` errors are suppressed**. The Job wrapper runs
   `apk add --no-cache curl jq >/dev/null 2>&1`. When the pod has no
   egress to the alpine package mirrors (cluster egress firewall,
   private node pool, etc.), the install fails, the script then runs
   without `curl` installed, and every `curl -sf "$DT_URL/api/version"`
   call is silently a no-op (sh "command not found" goes to stderr,
   which `-sf` doesn't surface anyway). The wait loop burns its
   5-minute budget, all 3 backoffs run, the Job goes Failed, and the
   only signal is `[dtrack-init] ERROR: timeout`. Operator has no
   idea the actual failure was at apk add.

   Fix: drop the output redirect (so apk's own errors land in the pod
   log) and add a `command -v curl jq` preflight in the script so the
   downstream "curl never runs" symptom is also surfaced loudly.

2. **API key extraction breaks on DependencyTrack 4.12+**. The current
   script reads `.apiKeys[0].key` from the team listing
   (`GET /api/v1/team`). DT 4.12 redacts `.key` in the listing
   response (only `.publicId` / `.maskedKey` remain) for security, so
   the script gets an empty value, errors out with
   `[dtrack-init] ERROR: no API key`, and the bootstrap is stuck on
   anything past 4.11.

   Fix: stop trying to read an existing key. Resolve the team's UUID
   and generate a fresh one via `PUT /api/v1/team/{uuid}/key`. The
   response body returns the unmasked secret -- this is the only way
   to retrieve it on 4.12+, and works on 4.11 too (4.11 also accepted
   POST without a body; 4.12+ returns 405 for POST and requires PUT +
   Content-Type: application/json). The idempotency guard at the top
   (`[ -f "$API_KEY_FILE" ] && [ -s "$API_KEY_FILE" ]`) prevents
   minting a new key on every Job restart.

3. **NVD legacy feed mirror 404s on DependencyTrack 4.10+**. NIST
   retired the file-based JSON feeds in late 2023:

       $ curl -I https://services.nvd.nist.gov/rest/json/cves/2.0/json/cve/2.0/nvdcve-2.0-2019.meta
       HTTP/2 404

   DT 4.10+ replaced the legacy source with the NVD REST API 2.0;
   `nvd.api.enabled=true` opts in. The chart was previously silent on
   NVD config, so deployments inherited whatever the dtrack image
   defaulted to (legacy feeds on 4.10+, → 404s → empty vulnerability
   DB → empty findings on every scan).

   Fix: POST `nvd.api.enabled=true` to `/api/v1/configProperty` during
   bootstrap. Safe to set unconditionally -- DT ignores unknown
   property names, so pre-4.10 images aren't affected.

Bonus, minor: switch the form-urlencoded body construction to
`curl --data-urlencode` per field. The `&`-joined string form worked
in 4.11 by coincidence (passwords in our deploys had no `&` / `=` /
unicode), but breaks the moment an operator sets a password containing
those. `--data-urlencode` per field is the boring-correct shape.

Validated against a self-hosted DT 4.11.4 deploy on dev-shared-gke
(my org's AK install): force-changed admin/admin -> the configured
password, generated a fresh Automation API key, and the backend
picked it up at next start. Integration `/api/v1/dependency-track/status`
moved from `healthy: false` to `healthy: true`.
@dragonpaw dragonpaw requested a review from a team as a code owner May 21, 2026 23:06
@brandonrc

Copy link
Copy Markdown
Contributor

Thanks for this, the diagnosis and the three fixes (un-silencing apk add, the nvd.api.enabled flag, and the --data-urlencode hardening) all look right.

Holding on the API-key change before merge: switching retrieval to PUT /team/{uuid}/key is a behavior change you noted is not yet validated against a 4.12+ instance, and the chart still ships Dependency-Track 4.11.4 by default. Could you confirm the PUT path (the 405-on-POST behavior and the .key response field) against a live 4.12+ install, or alternatively gate the PUT-vs-list behavior on the DT version so the default 4.11.4 path is unchanged? Once that is confirmed this is good to go.

The other two dtrack PRs (#113, #114) look mergeable; this one just needs that validation. (The red SonarCloud check is a known non-blocking fork limitation and can be ignored.)

Validation against live Dependency-Track 4.11.4 / 4.12.7 / 4.13.0 /
4.14.2 (fresh installs) corrected two claims in the bootstrap comments:

- POST /api/v1/team/{uuid}/key returns 405 on *all* tested versions,
  not just 4.12+. DT only accepts POST .../key/{key} to regenerate an
  existing key; PUT is the create/generate verb on every 4.x release.
  So PUT is the correct version-agnostic path, not a 4.12 workaround.
- The team-listing `.apiKeys[*].key` redaction starts at 4.13.0, not
  4.12 — 4.11.4 and 4.12.7 still return the full key. Generating a
  fresh key via PUT remains the right approach on every version.

Code unchanged; comments only.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@dragonpaw

Copy link
Copy Markdown
Contributor Author

Validated the PUT path against live Dependency-Track 4.11.4, 4.12.7, 4.13.0, and 4.14.2 (embedded H2, fresh installs):

version POST …/key PUT …/key listing .apiKeys[].key
4.11.4 405 201 + .key full key
4.12.7 405 201 + .key full key
4.13.0 405 201 + .key null (redacted)
4.14.2 405 201 + .key null (redacted)

So PUT /api/v1/team/{uuid}/key201 with the secret in .key on every version, and the change is correct and version-agnostic — no version gate needed.

Two corrections I just pushed to the branch (comments only, no code change):

  • POST is 405 on 4.11 too, not just 4.12+. POST …/key (no key suffix) was never a valid route — DT only accepts POST …/key/{key} to regenerate an existing key. PUT is the create/generate verb on all 4.x.
  • The listing-key redaction begins at 4.13.0, not 4.12 — 4.11.4 and 4.12.7 still return the full .key. Generating fresh via PUT is still the right call on every version, so I reworded that comment accordingly.

Good to go on your end?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

dtrack-init bootstrap Job: apk add failures silently swallowed; API-key extraction breaks on DT 4.12+; NVD legacy feed 404s on DT 4.10+

2 participants