Repository navigation
fix(cve-tool-vulnogram): let the first push after allocation replace the placeholder product - #2194
ChickenBenny wants to merge 1 commit into
Conversation
Kaap10
left a comment
There was a problem hiding this comment.
@ChickenBenny Thanks for picking this up and welcome!
For the clean tests and spec updates! The approach of treating the allocation placeholder as first-time population is the right direction, but there is one safety gap regarding record state gating and an open question on the live placeholder shape that we should address before merge:
correctness: exemption is state-unconstrained (tools/cve-tool-vulnogram/oauth-api/src/vulnogram_api/merge_mode.py:213)
No source rule covers this. It is a safety-boundary issue in the product-change guard.
In #1736 the issue explicitly proposed:
"detect the untouched allocation placeholder (state RESERVED, placeholder product) and allow the product to be set there automatically"
In the current diff, _is_placeholder_entry is evaluated purely inside _diff_affected_products(current, new) without knowledge of the record's state (CNA_private.state or cveMetadata.state). The summary notes "the fix doesn't need the record's state, so it doesn't read it", but merge_mode exists specifically as a safety net against accidental regressions on published or reviewed CVEs.
If a record has already transitioned to REVIEW, READY, or PUBLIC while still carrying an unpopulated placeholder, silently allowing a product overwrite without --allow-product-change bypasses that safety net.
The exemption should be gated so it only applies when the current record is in its initial unsubmitted state (RESERVED / DRAFT):
# Pass current_state into _diff_affected_products, or gate the placeholder exemption:
is_untouched_state = current_state in ("RESERVED", "DRAFT", None)
current_sigs = {
_product_signature(entry)
for entry in current
if isinstance(entry, dict) and not (is_untouched_state and _is_placeholder_entry(entry))
}correctness: unverified placeholder payload format (tools/cve-tool-vulnogram/oauth-api/src/vulnogram_api/merge_mode.py:183)
No source rule covers this. It is a live-service payload assumption.
You rightly noted in the PR description:
"The placeholder shape comes from the #1736 description; I don't have Vulnogram access to fetch an allocated record. Could someone with access paste the redacted affected[] of one? If it differs I'll adjust the match."
In CVE 5.0 JSON schema, Vulnogram records have both vendor and product keys in containers.cna.affected[]. If Vulnogram initializes a fresh allocation as:
{
"vendor": "[VENDOR]",
"product": "[PRODUCT]"
}or has whitespace variations, then entry.get("product") == "[VENDOR] [PRODUCT]" will evaluate to False and the push will continue to be refused in production.
We should either check with someone holding live Vulnogram credentials (or check if entry.get("product") in ("[VENDOR] [PRODUCT]", "[PRODUCT]")) before locking this in.
Smaller observations
general— Eval journal commit SHA: Intools/skill-evals/evals/security-issue-sync/runs.jsonl(line 14) andREADME.md, the recorded ref is57480dcf, but the branch commit is28cddb28. Please update the journal entry so the ref matches the commit on the PR branch.- See inline comments on
tools/cve-tool-vulnogram/oauth-api/src/vulnogram_api/merge_mode.py:183andtools/cve-tool-vulnogram/oauth-api/src/vulnogram_api/merge_mode.py:213.
Worth a second look from
Folks with context on Vulnogram allocation payloads and merge mode:
@potiuk— author of #1736
This review was drafted by an AI-assisted tool and
confirmed by an Apache Magpie maintainer. The findings
below are observations, not blockers; an Apache Magpie
maintainer — a real person — will take the next look at the
PR. If you think a finding is mis-applied, please reply on
the PR and a maintainer will weigh in.More on how Apache Magpie handles maintainer review:
Contributing guide.
|
|
||
|
|
||
| #: The product Vulnogram writes into ``affected[]`` when it allocates a CVE ID. | ||
| ALLOCATION_PLACEHOLDER_PRODUCT = "[VENDOR] [PRODUCT]" |
There was a problem hiding this comment.
correctness: unverified placeholder format
No source rule covers this. It is an assumption about upstream Vulnogram JSON output.
In CVE 5.0 JSON, affected[] entries typically split vendor and product. If Vulnogram emits {"vendor": "[VENDOR]", "product": "[PRODUCT]"} on allocation, strict equality with "[VENDOR] [PRODUCT]" will miss it and continue to refuse the first push.
We should confirm the exact payload from a live allocation or support both "[VENDOR] [PRODUCT]" and product == "[PRODUCT]" (with no package) to be safe.
| current_sigs = { | ||
| _product_signature(entry) | ||
| for entry in current | ||
| if isinstance(entry, dict) and not _is_placeholder_entry(entry) |
There was a problem hiding this comment.
correctness: state-unconstrained exemption
No source rule covers this. It is a safety-boundary issue in the product-change guard.
The original issue (#1736) requested exempting the placeholder specifically for "state RESERVED, placeholder product".
Leaving the placeholder out of current_sigs unconditionally means any record in REVIEW, READY, or PUBLIC that carries this string bypasses the rename guard. Please thread current_state into the diff check so the exemption only fires when current_state in ("RESERVED", "DRAFT", None).
| {"branch": "feat/security-skills-vetted-cve-ops", "cases": 2, "cli": "claude -p", "cost": {"cli_calls": 2, "grader_calls": 2, "wall_seconds": 36.6}, "date": "2026-11-18", "dirty": true, "errored": 0, "errored_cases": [], "failed": 0, "failed_cases": [], "graded_cases": ["step-5b-judged-gates/case-4-private-scanner-credit", "step-5b-judged-gates/case-5-clean-push-fix-released"], "grader_cli": "claude -p --model haiku", "manual": 0, "model": "opus[1m]", "model_source": "settings:user", "notes": "", "passed": 2, "path": "tools/skill-evals/evals/security-issue-sync/step-5b-judged-gates", "ref": "9e979ff3", "schema": 1, "skill": "security-issue-sync", "tags": [], "timestamp": "2026-11-18T01:05:21Z"} | ||
| {"branch": "feat/security-skills-vetted-cve-ops", "cases": 2, "cli": "claude -p", "cost": {"cli_calls": 2, "grader_calls": 2, "wall_seconds": 36.6}, "date": "2026-11-18", "dirty": true, "errored": 0, "errored_cases": [], "failed": 0, "failed_cases": [], "graded_cases": ["step-5b-judged-gates/case-4-private-scanner-credit", "step-5b-judged-gates/case-5-clean-push-fix-released"], "grader_cli": "claude -p --model haiku", "manual": 0, "model": "opus[1m]", "model_source": "settings:user", "notes": "", "passed": 2, "path": "tools/skill-evals/evals/security-issue-sync/step-5b-judged-gates", "ref": "add4ee52", "schema": 1, "skill": "security-issue-sync", "tags": [], "timestamp": "2026-11-18T01:24:36Z"} | ||
| {"branch": "feat/security-skills-vetted-cve-ops", "cases": 2, "cli": "claude -p", "cost": {"cli_calls": 2, "grader_calls": 2, "wall_seconds": 36.0}, "date": "2026-11-18", "dirty": true, "errored": 0, "errored_cases": [], "failed": 0, "failed_cases": [], "graded_cases": ["step-5b-judged-gates/case-4-private-scanner-credit", "step-5b-judged-gates/case-5-clean-push-fix-released"], "grader_cli": "claude -p --model haiku", "manual": 0, "model": "opus[1m]", "model_source": "settings:user", "notes": "", "passed": 2, "path": "tools/skill-evals/evals/security-issue-sync/step-5b-judged-gates", "ref": "add4ee52", "schema": 1, "skill": "security-issue-sync", "tags": [], "timestamp": "2026-11-18T02:36:27Z"} | ||
| {"branch": "fix/1736-allocation-placeholder", "cases": 2, "cli": "claude -p", "cost": {"cli_calls": 2, "grader_calls": 2, "wall_seconds": 27.0}, "date": "2026-11-18", "dirty": false, "errored": 0, "errored_cases": [], "failed": 0, "failed_cases": [], "graded_cases": ["step-5b-judged-gates/case-4-private-scanner-credit", "step-5b-judged-gates/case-5-clean-push-fix-released"], "grader_cli": "claude -p --model haiku", "manual": 0, "model": "opus[1m]", "model_source": "settings:user", "notes": "", "passed": 2, "path": "tools/skill-evals/evals/security-issue-sync/step-5b-judged-gates", "ref": "57480dcf", "schema": 1, "skill": "security-issue-sync", "tags": [], "timestamp": "2026-11-18T06:53:14Z"} |
There was a problem hiding this comment.
consistency: eval journal ref mismatch
The recorded ref here and in README.md is 57480dcf, whereas the commit on this branch is 28cddb28. Please re-render the journal entry with the matching commit ref so eval provenance is accurate.
There was a problem hiding this comment.
@ChickenBenny Thanks for the PR. Matching only a placeholder entry with no package keeps the exemption narrow, the tests show the old code refusing and the new code allowing, and the description openly flags what you couldn't check against live Vulnogram, which is exactly what a reviewer needs to see. I ran tests/test_merge_mode.py on 28cddb28 and it passes.
Kaap10's review covers the main points; a few notes building on it:
- Placeholder shape: agreed this is the open question. #1736 (filed by @potiuk) quotes the live product as
[VENDOR] [PRODUCT], so I'd keep the exact match rather than widen it to[PRODUCT]on a guess, since widening loosens a safety guard. @potiuk, could you confirm theaffected[]of a fresh allocation? - State gating: #1736 does mention RESERVED. As I read it, the guard protects products someone has written, and the placeholder hasn't been written by anyone in any state, so I don't see a rename it would let through. Gating on RESERVED/DRAFT would be extra caution; happy to go with whatever @potiuk prefers.
- Eval journal ref: the journal is written by the runner and a commit can't contain its own SHA, so
57480dcf(the commit the eval ran on) looks expected to me; I'd leave the record as it is rather than edit it by hand. - Small suggestion:
_allocated()in the tests defaults toDRAFT, while #1736 says a fresh allocation isRESERVED. Defaulting toRESERVEDwould mirror the real record.
This review was drafted by an AI-assisted tool and
confirmed by an Apache Magpie maintainer. The findings
below are observations, not blockers; an Apache Magpie
maintainer — a real person — will take the next look at the
PR. If you think a finding is mis-applied, please reply on
the PR and a maintainer will weigh in.More on how Apache Magpie handles maintainer review:
Contributing guide.
Summary
vulnogram-api-allocate, the firstvulnogram-api-record-updategets refused by the product guard.Vulnogram seeds a fresh record with a
[VENDOR] [PRODUCT]placeholder inaffected[], and the guard reads "placeholder → real product" as a rename.The vetted-ops
cve-record-updatenever passes--allow-product-change, so the agent path is just stuck.[VENDOR] [PRODUCT], no package) out of the current record before comparing.A record with only the placeholder then falls into the existing "empty
affected[]is first-time population" rule.vendorisn't read either: it isn't part of the product signature.Type of change
issue-sync/apply-and-push.mdnow says the first push after allocation needs no override, so the agent no longer asks for one theretools/cve-tool-vulnogram/oauth-api)tools/spec-loop/specs/cve-tooling.mdrecords the exemptionTest plan
TestAllocationPlaceholdercovers each allowed / refused case aboveuv run pytest(oauth-api): 205 passedprek run --all-files --skip workspace-pytestprek run workspace-pytest --from-ref upstream/main --to-ref HEADsecurity-issue-sync/step-5b-judged-gateseval (reads the changedapply-and-push.mdsection): 2/2 passed withclaude -p, journaled inruns.jsonlcheck-family-plugins --fixsynced the copies underplugins/magpie-security-toolsandplugins/magpie-vetted-opsRFC-AI-0004 compliance
Linked issues
Closes #1736
Notes for reviewers
Could someone with access paste the redacted
affected[]of one? If it differs I'll adjust the match.[VENDOR] [PRODUCT], nothing is exempted and the guard refuses as before; it never lets a real rename through.apply_merge_mode_guardsis unchanged; the refusal message and--allow-product-changebehave exactly as before.Generative AI disclosure: written with help from Claude Code (Opus 5.5).
I've read every line of the diff, run the tests, and can answer review questions myself.