feat(eval): sink-first analysis for deep hunt - #101
Open
whatever wants to merge 28 commits into
Open
Conversation
whatever
marked this pull request as ready for review
July 17, 2026 11:57
ropoctl
approved these changes
Jul 17, 2026
ropoctl
left a comment
ropoctl
left a comment
Contributor
There was a problem hiding this comment.
delete inspect_ai, keep the rest
whatever
changed the title
whatever
changed the title
whatever
force-pushed
the
feat/cross-file-chatter
branch
from
July 22, 2026 02:08
b1f69c7 to
67f90f8
Compare
Contributor
There was a problem hiding this comment.
this might be broken?
Root cause identified. The bundled rule signed-int-pointer-arithmetic has a pattern semgrep's C parser rejects — the bare trailing expression $PTR[$IDX] (no semicolon) isn't a valid C statement. When any one rule fails to parse, semgrep returns rc=2, and the sidecar (semgrep_sidecar.py:119) discards the entire scan — including all the valid p/security-audit findings.
whatever
force-pushed
the
feat/cross-file-chatter
branch
from
August 3, 2026 22:00
5660b5a to
f8c5d9f
Compare
Introduces a structured evaluation harness for sourcehunt using inspect_ai: - Single parameterized `cve_hunt` task replacing a dynamic globals factory - `dynamic_runner_solver` runs SourceHuntRunner per CVE with hardcoded eval flags matching the bash scripts: no-per-file-hunt, no-verify, no-exploit, no-variant-loop, no-mechanism-memory, max_parallel=4, budget_usd=18.00 - All SourceHuntResult stats forwarded to state.metadata: cost_usd, tokens_used, duration_seconds, files_ranked, files_hunted, subsystems_hunted, subsystem_spent_usd, spent_per_tier, status - LLM judge defaults to openai/gpt-5.4 - cves.yaml: initial CVE corpus with difficulty tags and subsystem hints - METRICS.md: design doc for adding steps_taken metric end-to-end
- cves.yaml: add directory-level sourcehunt.subsystem paths to 12 CVEs that previously used single vulnerable files as subsystem roots - preprocessor.py: index .pl and .pm files (Perl/asm generators) so subsystem hunts no longer skip CVE-2026-28386 (aes-cfb-avx512.pl) - telemetry.py: add gpt-5.4 and gpt-5.4-mini pricing ($2.50/$15 and $0.75/$4.50 per 1M tokens)
The strict Anthropic-direct provider check was firing for direct table matches (e.g. claude-opus-4-6 routed via LiteLLM), which is a valid gateway configuration. Restrict the check to alias-inferred matches only. Also drop redundant per-file subsystem targets from evaluate.py — when a sourcehunt.subsystem directory is set, files: no longer appended as extra subsystem paths.
… scoping - Add func_calls_out to CallGraph for per-function edge tracking - Add callers_of() / callees_of() methods to CallGraph - Add lookup_callers / lookup_callees tools to deep agent (sink-first traversal) - Wire callgraph into HunterContext - Remove broken digit-normalized loop detection (was blocking file pagination) - Simplify sitrep to just facts (step/findings/files/cost), no advice text - Scope CVE-2026-40033 subsystem to single file gfx.c
The repeated-tool-call dedup guard strips all digits from a tool call's JSON args before hashing, to catch a model reissuing the same call with a slowly-widening numeric literal (e.g. `grep -B10` creeping to `-B1750`). But read_file(path, offset, limit) and read_source_file(path, start_line, end_line) use offset/limit as the ONLY fields that legitimately differ between successive, non-redundant paginated reads of the same file. Digit-stripping collapsed every read on a given path to one key after 3 calls, so the 4th+ legitimately-different read was falsely rejected. Confirmed live against crAPI: hunting shop/views.py (433 lines) and mechanic/views.py (459 lines), every pass had reads falsely throttled after the 3rd call regardless of the offset requested, preventing the hunter from reading far enough to find a real SQL injection later in shop/views.py's ApplyCouponView. Keep read_file/read_source_file's arguments literal (not digit-normalized) for the dedup key so distinct offset/limit pairs get distinct keys, while an exact repeat of the same call is still throttled. All other tools keep the existing digit-normalization behavior. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
whatever
force-pushed
the
feat/cross-file-chatter
branch
from
August 6, 2026 19:57
e8bd2be to
c8d5e11
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR fixes some bugs with
semgrep, prioritizes sinks, and works backwards from them withcallee_lookupto pass potential findings to verify.NOTABLY:
func_calls_out) andlookup_callers/lookup_calleestools for sink-first traversal in deep agent modegfx.c(1 file vs 29); confirmed pass in solo evalResults
CVE Breakdown
Test plan
uv run inspect eval evaluations/evaluate.py --model none/none --display plain