Skip to content

fix(cve-tool-vulnogram): let the first push after allocation replace the placeholder product - #2194

Open
ChickenBenny wants to merge 1 commit into
apache:mainfrom
ChickenBenny:fix/1736-allocation-placeholder
Open

ChickenBenny wants to merge 1 commit into
apache:mainfrom
ChickenBenny:fix/1736-allocation-placeholder

Conversation

@ChickenBenny

Copy link
Copy Markdown

Summary

  • Right after vulnogram-api-allocate, the first vulnogram-api-record-update gets refused by the product guard.
    Vulnogram seeds a fresh record with a [VENDOR] [PRODUCT] placeholder in affected[], and the guard reads "placeholder → real product" as a rename.
    The vetted-ops cve-record-update never passes --allow-product-change, so the agent path is just stuck.
  • The product guard now leaves the placeholder entry (product [VENDOR] [PRODUCT], no package) out of the current record before comparing.
    A record with only the placeholder then falls into the existing "empty affected[] is first-time population" rule.
  • Real products are still compared as before: renaming one, even next to a placeholder, is still refused, and so is a placeholder that carries a package.
  • The fix doesn't need the record's state, so it doesn't read it. vendor isn't read either: it isn't part of the product signature.

Type of change

  • Skill change — issue-sync/apply-and-push.md now says the first push after allocation needs no override, so the agent no longer asks for one there
  • Python package (tools/cve-tool-vulnogram/oauth-api)
  • Other: spec — tools/spec-loop/specs/cve-tooling.md records the exemption

Test plan

  • New TestAllocationPlaceholder covers each allowed / refused case above
  • The "allowed" tests fail on the old code and pass with the fix
  • uv run pytest (oauth-api): 205 passed
  • prek run --all-files --skip workspace-pytest
  • prek run workspace-pytest --from-ref upstream/main --to-ref HEAD
  • security-issue-sync/step-5b-judged-gates eval (reads the changed apply-and-push.md section): 2/2 passed with claude -p, journaled in runs.jsonl
  • check-family-plugins --fix synced the copies under plugins/magpie-security-tools and plugins/magpie-vetted-ops

RFC-AI-0004 compliance

  • HITL — no new mutation; this only stops a false refusal

Linked issues

Closes #1736

Notes for reviewers

  • The placeholder shape comes from the security-cve-allocate: first push after API allocation is refused (placeholder product) #1736 description; I don't have Vulnogram access to fetch an allocated record.
    Could someone with access paste the redacted affected[] of one? If it differs I'll adjust the match.
  • If a record's product isn't exactly [VENDOR] [PRODUCT], nothing is exempted and the guard refuses as before; it never lets a real rename through.
  • apply_merge_mode_guards is unchanged; the refusal message and --allow-product-change behave exactly as before.
  • Thanks to @choksi2212 for the earlier analysis on the issue, which this builds on.

Generative AI disclosure: written with help from Claude Code (Opus 5.5).
I've read every line of the diff, run the tests, and can answer review questions myself.

@Kaap10 Kaap10 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ChickenBenny Thanks for picking this up and welcome!
For the clean tests and spec updates! The approach of treating the allocation placeholder as first-time population is the right direction, but there is one safety gap regarding record state gating and an open question on the live placeholder shape that we should address before merge:

correctness: exemption is state-unconstrained (tools/cve-tool-vulnogram/oauth-api/src/vulnogram_api/merge_mode.py:213)

No source rule covers this. It is a safety-boundary issue in the product-change guard.

In #1736 the issue explicitly proposed:

"detect the untouched allocation placeholder (state RESERVED, placeholder product) and allow the product to be set there automatically"

In the current diff, _is_placeholder_entry is evaluated purely inside _diff_affected_products(current, new) without knowledge of the record's state (CNA_private.state or cveMetadata.state). The summary notes "the fix doesn't need the record's state, so it doesn't read it", but merge_mode exists specifically as a safety net against accidental regressions on published or reviewed CVEs.

If a record has already transitioned to REVIEW, READY, or PUBLIC while still carrying an unpopulated placeholder, silently allowing a product overwrite without --allow-product-change bypasses that safety net.

The exemption should be gated so it only applies when the current record is in its initial unsubmitted state (RESERVED / DRAFT):

# Pass current_state into _diff_affected_products, or gate the placeholder exemption:
is_untouched_state = current_state in ("RESERVED", "DRAFT", None)
current_sigs = {
    _product_signature(entry)
    for entry in current
    if isinstance(entry, dict) and not (is_untouched_state and _is_placeholder_entry(entry))
}

correctness: unverified placeholder payload format (tools/cve-tool-vulnogram/oauth-api/src/vulnogram_api/merge_mode.py:183)

No source rule covers this. It is a live-service payload assumption.

You rightly noted in the PR description:

"The placeholder shape comes from the #1736 description; I don't have Vulnogram access to fetch an allocated record. Could someone with access paste the redacted affected[] of one? If it differs I'll adjust the match."

In CVE 5.0 JSON schema, Vulnogram records have both vendor and product keys in containers.cna.affected[]. If Vulnogram initializes a fresh allocation as:

{
  "vendor": "[VENDOR]",
  "product": "[PRODUCT]"
}

or has whitespace variations, then entry.get("product") == "[VENDOR] [PRODUCT]" will evaluate to False and the push will continue to be refused in production.

We should either check with someone holding live Vulnogram credentials (or check if entry.get("product") in ("[VENDOR] [PRODUCT]", "[PRODUCT]")) before locking this in.

Smaller observations

  • general — Eval journal commit SHA: In tools/skill-evals/evals/security-issue-sync/runs.jsonl (line 14) and README.md, the recorded ref is 57480dcf, but the branch commit is 28cddb28. Please update the journal entry so the ref matches the commit on the PR branch.
  • See inline comments on tools/cve-tool-vulnogram/oauth-api/src/vulnogram_api/merge_mode.py:183 and tools/cve-tool-vulnogram/oauth-api/src/vulnogram_api/merge_mode.py:213.

Worth a second look from

Folks with context on Vulnogram allocation payloads and merge mode:

  • @potiuk — author of #1736

This review was drafted by an AI-assisted tool and
confirmed by an Apache Magpie maintainer. The findings
below are observations, not blockers; an Apache Magpie
maintainer — a real person — will take the next look at the
PR. If you think a finding is mis-applied, please reply on
the PR and a maintainer will weigh in.

More on how Apache Magpie handles maintainer review:
Contributing guide.



#: The product Vulnogram writes into ``affected[]`` when it allocates a CVE ID.
ALLOCATION_PLACEHOLDER_PRODUCT = "[VENDOR] [PRODUCT]"

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

correctness: unverified placeholder format

No source rule covers this. It is an assumption about upstream Vulnogram JSON output.

In CVE 5.0 JSON, affected[] entries typically split vendor and product. If Vulnogram emits {"vendor": "[VENDOR]", "product": "[PRODUCT]"} on allocation, strict equality with "[VENDOR] [PRODUCT]" will miss it and continue to refuse the first push.

We should confirm the exact payload from a live allocation or support both "[VENDOR] [PRODUCT]" and product == "[PRODUCT]" (with no package) to be safe.

current_sigs = {
_product_signature(entry)
for entry in current
if isinstance(entry, dict) and not _is_placeholder_entry(entry)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

correctness: state-unconstrained exemption

No source rule covers this. It is a safety-boundary issue in the product-change guard.

The original issue (#1736) requested exempting the placeholder specifically for "state RESERVED, placeholder product".

Leaving the placeholder out of current_sigs unconditionally means any record in REVIEW, READY, or PUBLIC that carries this string bypasses the rename guard. Please thread current_state into the diff check so the exemption only fires when current_state in ("RESERVED", "DRAFT", None).

{"branch": "feat/security-skills-vetted-cve-ops", "cases": 2, "cli": "claude -p", "cost": {"cli_calls": 2, "grader_calls": 2, "wall_seconds": 36.6}, "date": "2026-11-18", "dirty": true, "errored": 0, "errored_cases": [], "failed": 0, "failed_cases": [], "graded_cases": ["step-5b-judged-gates/case-4-private-scanner-credit", "step-5b-judged-gates/case-5-clean-push-fix-released"], "grader_cli": "claude -p --model haiku", "manual": 0, "model": "opus[1m]", "model_source": "settings:user", "notes": "", "passed": 2, "path": "tools/skill-evals/evals/security-issue-sync/step-5b-judged-gates", "ref": "9e979ff3", "schema": 1, "skill": "security-issue-sync", "tags": [], "timestamp": "2026-11-18T01:05:21Z"}
{"branch": "feat/security-skills-vetted-cve-ops", "cases": 2, "cli": "claude -p", "cost": {"cli_calls": 2, "grader_calls": 2, "wall_seconds": 36.6}, "date": "2026-11-18", "dirty": true, "errored": 0, "errored_cases": [], "failed": 0, "failed_cases": [], "graded_cases": ["step-5b-judged-gates/case-4-private-scanner-credit", "step-5b-judged-gates/case-5-clean-push-fix-released"], "grader_cli": "claude -p --model haiku", "manual": 0, "model": "opus[1m]", "model_source": "settings:user", "notes": "", "passed": 2, "path": "tools/skill-evals/evals/security-issue-sync/step-5b-judged-gates", "ref": "add4ee52", "schema": 1, "skill": "security-issue-sync", "tags": [], "timestamp": "2026-11-18T01:24:36Z"}
{"branch": "feat/security-skills-vetted-cve-ops", "cases": 2, "cli": "claude -p", "cost": {"cli_calls": 2, "grader_calls": 2, "wall_seconds": 36.0}, "date": "2026-11-18", "dirty": true, "errored": 0, "errored_cases": [], "failed": 0, "failed_cases": [], "graded_cases": ["step-5b-judged-gates/case-4-private-scanner-credit", "step-5b-judged-gates/case-5-clean-push-fix-released"], "grader_cli": "claude -p --model haiku", "manual": 0, "model": "opus[1m]", "model_source": "settings:user", "notes": "", "passed": 2, "path": "tools/skill-evals/evals/security-issue-sync/step-5b-judged-gates", "ref": "add4ee52", "schema": 1, "skill": "security-issue-sync", "tags": [], "timestamp": "2026-11-18T02:36:27Z"}
{"branch": "fix/1736-allocation-placeholder", "cases": 2, "cli": "claude -p", "cost": {"cli_calls": 2, "grader_calls": 2, "wall_seconds": 27.0}, "date": "2026-11-18", "dirty": false, "errored": 0, "errored_cases": [], "failed": 0, "failed_cases": [], "graded_cases": ["step-5b-judged-gates/case-4-private-scanner-credit", "step-5b-judged-gates/case-5-clean-push-fix-released"], "grader_cli": "claude -p --model haiku", "manual": 0, "model": "opus[1m]", "model_source": "settings:user", "notes": "", "passed": 2, "path": "tools/skill-evals/evals/security-issue-sync/step-5b-judged-gates", "ref": "57480dcf", "schema": 1, "skill": "security-issue-sync", "tags": [], "timestamp": "2026-11-18T06:53:14Z"}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

consistency: eval journal ref mismatch

The recorded ref here and in README.md is 57480dcf, whereas the commit on this branch is 28cddb28. Please re-render the journal entry with the matching commit ref so eval provenance is accurate.

@KatalKavya96 KatalKavya96 left a comment •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ChickenBenny Thanks for the PR. Matching only a placeholder entry with no package keeps the exemption narrow, the tests show the old code refusing and the new code allowing, and the description openly flags what you couldn't check against live Vulnogram, which is exactly what a reviewer needs to see. I ran tests/test_merge_mode.py on 28cddb28 and it passes.

Kaap10's review covers the main points; a few notes building on it:

  • Placeholder shape: agreed this is the open question. #1736 (filed by @potiuk) quotes the live product as [VENDOR] [PRODUCT], so I'd keep the exact match rather than widen it to [PRODUCT] on a guess, since widening loosens a safety guard. @potiuk, could you confirm the affected[] of a fresh allocation?
  • State gating: #1736 does mention RESERVED. As I read it, the guard protects products someone has written, and the placeholder hasn't been written by anyone in any state, so I don't see a rename it would let through. Gating on RESERVED/DRAFT would be extra caution; happy to go with whatever @potiuk prefers.
  • Eval journal ref: the journal is written by the runner and a commit can't contain its own SHA, so 57480dcf (the commit the eval ran on) looks expected to me; I'd leave the record as it is rather than edit it by hand.
  • Small suggestion: _allocated() in the tests defaults to DRAFT, while #1736 says a fresh allocation is RESERVED. Defaulting to RESERVED would mirror the real record.

This review was drafted by an AI-assisted tool and
confirmed by an Apache Magpie maintainer. The findings
below are observations, not blockers; an Apache Magpie
maintainer — a real person — will take the next look at the
PR. If you think a finding is mis-applied, please reply on
the PR and a maintainer will weigh in.

More on how Apache Magpie handles maintainer review:
Contributing guide.

@github-actions github-actions Bot added family:security security-* skills family:tools tools/* family:docs Docs, MISSION.md, READMEs capability:intake Import external signal + sync tracker state contract:cve-authority Tool capability: CVE allocation / record management / publication family:dev Repo dev: .asf.yaml, spec-loop specs, Node manifests, .claude/settings.json labels Nov 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

capability:intake Import external signal + sync tracker state contract:cve-authority Tool capability: CVE allocation / record management / publication family:dev Repo dev: .asf.yaml, spec-loop specs, Node manifests, .claude/settings.json family:docs Docs, MISSION.md, READMEs family:security security-* skills family:tools tools/*

Projects

None yet

Development

Successfully merging this pull request may close these issues.

security-cve-allocate: first push after API allocation is refused (placeholder product)

3 participants