Make the daily scan's "no new items" verifiable

[xdotli]

Why

The scan has opened no PR for 31 consecutive days (last one: #33, 2026-06-29) while reporting green in Slack every morning. It was not out of material — a manual pass over the same seams the prompt names found 8 in-window candidates for July, 3 of them direct sequels to URLs already in the list, including OpenAI's 2026-07-08 audit retracting its own SWE-Bench Pro recommendation (README already carries the predecessor post). Runs average 12 turns and 109s against a 60-minute budget and a 200-turn cap.

What changed

1. The prompt contradicted itself. Step 2 said concluding "nothing new" without fetching the company-blog indexes is a FAILED run; step 3 said "no new items" is the normal successful outcome. Only false positives had a stated cost, so emitting zero was both the rational answer and the cheap one (~3 turns to read the dedup files vs. per-item subagents + verification for the PR path). Misses now carry an explicit cost, and every run must end with a search report: sources fetched, candidates rejected with reasons, and counts.

2. Runs are now auditable. Uploads the execution log reduced to tool name + input, with every tool_result dropped, so you can see which URLs the scan actually fetched.

show_full_output is deliberately left off: the execution file is already complete without it (the SDK writes the unfiltered message array regardless), so enabling it would add no audit value and would print tool results into this public repo's Actions log. Tool results are excluded because artifact bytes are not secret-masked by the runner, and the scan settings allow Bash(cat:*) and unrestricted Read. Filter tested against a synthetic log with planted CLAUDE_CODE_OAUTH_TOKEN / SLACK_BOT_TOKEN / GITHUB_TOKEN values: zero secrets survive, all tool calls and distinct-fetch counts do.

3. Two Slack bugs. The result step matched any open scan/* PR and tested that before the job status. While #33 sat open, the 06-30 and 07-01 runs re-announced it as that day's finds — and a failed run with a stale PR open would have posted green. Now scoped to PRs created during this run (failing closed if the timestamp is missing, since an empty --arg matches everything), status checked first, and the summary no longer suppressed entirely when the start post fails. The counts ride along, so "no new items" next to "0 distinct fetches" is visibly wrong at a glance.

Verification

• execution_file confirmed as a real action output; YAML parses; all three shell blocks pass bash -n.
• Redaction filter and the run-scoped PR selector both tested against fixtures, including a reproduction of the 06-30 misfire now falling through correctly.

Not included

SHA-pinning the actions plus a dependabot.yml, and a fix for CONTRIBUTING.md's format spec (it matches 0 of 468 entries). Both are worth doing separately.

[github-actions]

This is an infrastructure / workflow PR, not a resource submission — reviewing it as such.

The diagnosis is right. Prompt step 3 calling "no new items" the "normal successful outcome" directly contradicted step 2's "no fetch = failed run," and because only false positives carried an explicit cost, emitting zero was both the rational answer and the cheap path (~3 turns to read dedup files vs. per-item subagents + fetch overhead for the PR path). The observable effect — 31 consecutive green runs on a 12-turn, ~109s average against a 60-minute budget — is exactly what you'd expect. The fix (making misses carry an explicit cost and requiring a falsifiable search report every run) addresses the root cause.

Audit trail design is sound. Stripping tool_result while keeping name + input is the right call: it gives you the one thing that answers "did the scan actually look?" (which URLs it fetched) without risking secrets leaking into artifact bytes that the runner won't mask. The bash -e guard with ::warning:: on parse failure is the correct degradation — a missing audit trail should never swallow the Slack summary. The execution_file output confirmed as real, and the fallback to $RUNNER_TEMP/claude-execution-output.json covers the edge case gracefully.

Both Slack bugs are real and the fixes are correct.

• Stale-PR re-announcement: scoping to createdAt >= started_at (ISO-8601 UTC on both sides → lexicographic comparison works) and failing closed when the timestamp is empty (pr='' stays empty rather than matching everything) is exactly right.
• Status-before-PR ordering: the old logic could post green when a run failed with a stale PR open — the new check is status != success first, always.
• Thread fallback: conditional thread_ts via jq (if $ts == "" then {} else {thread_ts:$ts} end) is clean, and removing the steps.slack_start.outputs.ts != '' gate from the if: means the message lands in-channel rather than being silently dropped when the start post failed.

One minor note: actions/upload-artifact@v4 and anthropics/claude-code-action@v1 are floating tags. The PR explicitly calls out SHA-pinning as out of scope, which is a reasonable call for a focused fix — just worth doing in a follow-up.

The search-report requirement in step 7 is the addition I'd most want to see in action: "0 distinct fetches" next to "no new items" is immediately auditable, which is the whole point.

Looks mergeable. The fix is scoped, the reasoning is documented, and the security posture (secrets exclusion from artifacts) was tested. The "Not included" callouts (SHA-pinning, CONTRIBUTING.md format spec) are the right things to punt on separately.

---

awesome-evals assistant