feat: add guided generation benchmark tooling - #32
Conversation
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8aff2b967e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if (parsedFindings.length > 0) { | ||
| findings.push(...parsedFindings); | ||
| evaluation = parseGuardEvaluation(payload); | ||
| } else { |
There was a problem hiding this comment.
Parse guard evaluation even when findings are empty
When the generation guard returns JSON with no findings (the common pass path), this branch skips parseGuardEvaluation(payload) and leaves evaluation at its default values, so adapter responses can incorrectly report colorPolicyEvaluated=false / iconPolicyEvaluated=false in descriptor-parity runs. tools/check-generation-boundaries.mjs emits an evaluation object even on zero-findings responses, so this loses real coverage data and can skew benchmark/report gating that relies on coverage fields.
Useful? React with 👍 / 👎.
| case "restrict-icon-sources": | ||
| return "icons.allowedSources"; |
There was a problem hiding this comment.
Point icon-source suggestions at surface-scoped contract path
This maps restrict-icon-sources to icons.allowedSources, but icon policy is defined per surface (surfaces[].icons). As a result, generated contract-delta suggestions for icon.source-disallowed direct reviewers to a non-existent/wrong contract path, which makes accepted suggestions hard to apply correctly and can cause edits in the wrong place.
Useful? React with 👍 / 👎.
| case "restore-marketing-typography-profile": | ||
| return `surfaces[id=${surfaceId}].allowedFonts`; |
There was a problem hiding this comment.
Map typography-profile repair to typography profile fields
restore-marketing-typography-profile currently resolves to surfaces[id=...].allowedFonts, but the underlying repair and finding are about marketingTypographyProfile/policy, not the font allowlist. This produces misleading suggestion paths for marketing.typography.profile-missing, so reviewers are directed to edit unrelated fields instead of the actual contract knobs that control the violation.
Useful? React with 👍 / 👎.
Summary
Testing