Projects v2 board + Codex/Gemma SAST with cross-validation#5
Merged
Conversation
Replaces the parent-issue / sub-issue model with a flat GitHub Projects v2
board. Removes the 100-sub-issue cap that was blocking real runs, and uses
the board's `Severity` + `Category` single-select fields so triage can
group/sort/filter inside the project UI.
Config schema change — `parent_issue: 451` is gone. Required instead:
project:
owner: leverj
number: 5
The PAT now needs the `project` scope in addition to `repo`.
What changed in code:
- `secscan/config.py`: new `ProjectConfig` (owner, number); loader validates
both fields.
- `secscan/github.py`: replaces `list_subissues` + `link_subissue` with a
GraphQL Projects v2 surface — `resolve_project` (idempotently provisions
Severity + Category fields on first run), `list_project_items` (paginated
for dedup), `add_to_project`, `set_project_field`. New `ProjectContext` /
`ProjectField` dataclasses. Org-vs-user lookup is two sequential queries
to avoid combined-query error-handling fragility.
- `secscan/sync.py`: walks project items for dedup, then adds each new
issue to the project and sets the two custom fields. Fingerprint scheme
is unchanged (still marker in issue body).
- `secscan/main.py`: resolves the project once at startup. Dry-run path
returns a synthetic ProjectContext so no HTTP fires.
- `secscan/notify.py` + `secscan/triage.py`: digest/intro headers now
link to the project URL instead of citing a parent issue number.
- Tests rewritten for the new surface; 191 passing. Smoke-tested against
the real Projects v2 board (`leverj/projects/5`, 100 items present after
the one-time migration).
Migration of the existing 100 sub-issues into the board was done out-of-band
(one-shot script, deleted) so this commit doesn't carry any deployment-
specific code. New runs will dedup cleanly against the existing items.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two opt-in LLM-driven SAST scanners and a cross-validation pass between
them. Both default OFF; flip in `scanners:` to enable.
What each is good for:
- Codex (cloud, via local `codex` CLI on user's ChatGPT subscription):
depth detective — multi-file reasoning, framework idioms, subtle
auth/business-logic bugs. No API key — auth via `codex login`.
- Gemma (local, via Ollama): fast & free peer reviewer — high-volume
pattern detection and bidirectional validation.
When BOTH are enabled, `secscan/cross_validate.py` does a second pass:
each tool reviews the other's findings ("real / false_positive /
uncertain + reason"). A `false_positive` verdict downgrades severity
one notch (high→medium, medium→low, low→info); critical is asymmetric
and never auto-downgrades (cost of missing a real critical > one
extra board item). Findings are NEVER suppressed — disagreement is
surfaced in `finding.extra.cross_validation` so the project board is
the single source of triage truth.
Both runners emit synthetic SARIF so the existing `normalize_sarif`
+ fingerprint + sync pipeline handles them identically to semgrep/osv.
Rule IDs are namespaced (`codex.<id>`, `gemma.<id>`) so no collisions
with other scanners' rule IDs in fingerprints/labels.
Safety guardrails on both:
- Codex invoked with `-s read-only` + `--ephemeral`; cannot mutate repo.
- Gemma prompt is capped by file count, per-file bytes, and total bytes.
- Both contribute zero findings on any failure mode (timeout, parse
error, binary missing, auth fail) — never reads as "all clear".
- Cross-validation falls back to "uncertain" on validator unreachable;
never blocks the run.
Config additions: `scanners.codex` / `scanners.gemma` flags, plus
`codex:` / `gemma:` / `cross_validate:` blocks for tunables. Gemma
falls back to the existing `triage:` Ollama URL/model if unset, so
most users configure Ollama once.
221 tests pass (30 new across codex/gemma/cross-validate).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This PR refactors secscan’s GitHub filing/dedup model from sub-issues under a parent issue to a flat GitHub Projects v2 board (with Severity + Category fields), and introduces two opt-in LLM-driven SAST scanners (Codex CLI + Gemma via Ollama) with an optional bidirectional cross-validation step.
Changes:
- Replace sub-issue listing/linking with Projects v2 GraphQL flows: resolve project + paginate items + add issue to project + set single-select fields.
- Add new Codex and Gemma SAST runners and wire cross-validation to annotate/downgrade severities (without suppressing findings).
- Update config schema, README/config example, and tests to reflect the new project-based workflow and new scanners.
Reviewed changes
Copilot reviewed 24 out of 24 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/test_triage.py | Updates triage Slack intro/digest call signatures to include project owner/number. |
| tests/test_sync.py | Refactors sync tests for Projects v2 items + field setting, adds field-option assertions. |
| tests/test_resolve_rules.py | Updates Config fixture to use project instead of parent_issue. |
| tests/test_notify.py | Updates Slack digest tests for project link/header parameters. |
| tests/test_main.py | Updates E2E-style main tests to mock project resolution and project mutations. |
| tests/test_github.py | Replaces sub-issue tests with Projects v2 GraphQL tests (resolve/list/add/set), adjusts create_issue expectations. |
| tests/test_gemma_runner.py | Adds coverage for Gemma runner behavior, caps, and Ollama failure modes. |
| tests/test_e2e_dryrun.py | Updates dry-run pipeline tests for Projects v2 (and verifies zero HTTP calls in dry-run). |
| tests/test_cross_validate.py | Adds coverage for cross-validation verdict mapping and failure handling. |
| tests/test_config.py | Updates config loading tests for required project block and new scanner/cross-validate config. |
| tests/test_codex_runner.py | Adds coverage for Codex runner subprocess/schema handling and failure modes. |
| secscan/triage.py | Updates Slack intro payload to reference project instead of parent issue. |
| secscan/sync.py | Dedups against project items and files new issues into a project with severity/category fields. |
| secscan/runners/gemma.py | Introduces Ollama-backed Gemma SAST runner with prompt/file-size caps and SARIF output. |
| secscan/runners/codex.py | Introduces Codex CLI-backed SAST runner with structured output schema and SARIF output. |
| secscan/notify.py | Updates Slack digest header to link to the Projects v2 board (owner/number). |
| secscan/normalize.py | Maps new scanners (codex/gemma) into the sast category for downstream handling. |
| secscan/main.py | Adds cross-validation hook and resolves Projects v2 context before syncing findings. |
| secscan/github.py | Adds Projects v2 GraphQL support (resolve/list/add/set) and returns node_id in dry-run create_issue. |
| secscan/detect.py | Adds Codex/Gemma targets when enabled and source code is present. |
| secscan/cross_validate.py | Adds bidirectional validation between Codex and Gemma findings with severity downgrades. |
| secscan/config.py | Replaces parent_issue with project, adds codex/gemma/cross_validate config blocks. |
| README.md | Updates workflow/docs from parent sub-issues to Projects v2 + PAT scope changes. |
| config.example.yaml | Updates example config for Projects v2 and adds optional codex/gemma/cross_validate blocks. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+81
to
+86
| # Cross-validation: if both codex AND gemma ran, each tool reviews the other's | ||
| # findings. Strictly additive — bad reviews downgrade severity but never drop. | ||
| if (cfg.cross_validate.enabled | ||
| and "codex" in completed_scanners | ||
| and "gemma" in completed_scanners): | ||
| from secscan.cross_validate import cross_validate |
Comment on lines
+121
to
+123
| item_id = gh.add_to_project(project.id, issue["node_id"]) | ||
| gh.set_project_field(project.id, item_id, project.severity, f.severity) | ||
| gh.set_project_field(project.id, item_id, project.category, f.category) |
| @@ -117,7 +118,9 @@ def sync( | |||
|
|
|||
| body = inject_marker(body, fp, f) | |||
Comment on lines
+133
to
+137
| lines: list[str] = [ | ||
| f":lock: *secscan* — `{repo}@{ref}` — " | ||
| f"<https://github.com/orgs/{project_owner}/projects/{project_number}|" | ||
| f"{project_owner}/projects/{project_number}>" | ||
| ] |
Comment on lines
+151
to
+156
| snippet = _read_snippet(repo_dir, f.file_path, f.line) or (f.extra or {}).get("snippet", "") | ||
| prompt = _REVIEW_PROMPT.format( | ||
| finding_json=json.dumps(_finding_summary(f), indent=2), | ||
| snippet=(str(snippet)[:1200] or "(unavailable)"), | ||
| ) | ||
| try: |
Comment on lines
+28
to
+45
| # Extensions worth feeding to the model. Mirrors secscan/detect._SEMGREP_EXTS with | ||
| # a few SQL/HCL/TF additions since LLM reading isn't limited to semgrep's parsers. | ||
| _SOURCE_EXTS = { | ||
| ".js", ".jsx", ".ts", ".tsx", ".mjs", ".cjs", | ||
| ".py", ".pyw", | ||
| ".rb", | ||
| ".go", | ||
| ".java", ".kt", ".kts", ".scala", | ||
| ".swift", | ||
| ".c", ".h", ".cc", ".cpp", ".cxx", ".hpp", ".hh", | ||
| ".rs", | ||
| ".php", | ||
| ".sql", | ||
| ".sh", ".bash", | ||
| ".tf", ".hcl", | ||
| ".yaml", ".yml", | ||
| ".env", ".envrc", | ||
| } |
CI lint job flagged the typing.Iterable import; switch to the modern collections.abc.Iterable. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two related changes that together unlock a much larger scanning surface and remove the 100-sub-issue cap that was blocking real runs:
c55292c) — drop the parent-issue / sub-issue model in favor of a flat GitHub Projects v2 board withSeverity+Categorysingle-select fields. Removes the 100-cap blocker.1a9b32e) — two opt-in LLM-driven SAST scanners, with bidirectional review when both are enabled.What changed
Projects v2 (
c55292c)parent_issue: 451→project: { owner, number }.github.py: replaceslist_subissues/link_subissuewith GraphQL —resolve_project(auto-provisions Severity + Category fields on first run),list_project_items(paginated),add_to_project,set_project_field. NewProjectContext/ProjectFielddataclasses.sync.py: walks project items for dedup, then adds each new issue to the project and sets both custom fields. Fingerprint scheme unchanged.main.py+notify.py+triage.py: resolve project once at startup; digest header links to the project URL instead of citing a parent issue.projectscope in addition torepo. README updated./tmp) so this PR carries no deployment-specific code.Codex + Gemma + cross-validation (
1a9b32e)secscan/runners/codex.py— shells out to the localcodexCLI (codex exec -s read-only --output-schema ... -o ...). Auth viacodex login/ ChatGPT subscription; no API key.secscan/runners/gemma.py— uses Ollama; reuses the existingtriage:config by default. Hard caps on file count + per-file bytes + total prompt bytes.secscan/cross_validate.py— when BOTH scanners are enabled, each tool reviews the other's findings ("real / false_positive / uncertain + reason").false_positivedowngrades severity one notch; critical is asymmetric and never auto-downgrades (cost of missing a real critical >> one extra board item). Findings are never suppressed — humans triage on the project board.scanners:. Tunable via newcodex:/gemma:/cross_validate:config blocks.Test plan
python -m pytest -q).leverj/projects/5— finds all 100 migrated items, both custom fields resolve correctly, no error.returntocorp/semgrep:1.97.0for the bundled XSS/SQLi/Supabase rule packs landed earlier onmain.scanners.codex+scanners.gemmatotrueagainst a small repo first to calibrate signal/noise before turning loose onezel.https://github.com/orgs/leverj/projects/3/workflows→Auto-add to project, set filteris:issue,pr is:open -label:security. Stops new secscan issues from auto-flowing into project Redact high-entropy substrings from SAST snippets before sending to remote LLM #3 (the workflow-filter mutation isn't exposed via GraphQL).Migration / rollout notes
config.yamlis gitignored; the local deployment config is already updated to the new schema.config.example.yamldocuments both the newproject:block and the optionalcodex:/gemma:/cross_validate:blocks.🤖 Generated with Claude Code