Skip to content

Tier 3: Independent Evidence Runner (CI adjudicator) - #1

Merged
kirder24-code merged 11 commits into
mainfrom
tier3-plan
Jun 28, 2026
Merged

kirder24-code merged 11 commits into
mainfrom
tier3-plan

Conversation

@kirder24-code

@kirder24-code kirder24-code commented Jun 28, 2026 •

Copy link
Copy Markdown
Owner

Tier 3: Independent CI adjudicator

Runcap can now recompute a pull request's merge decision in a clean CI job from
the PR's base commit, never trusting the agent or the agent's receipt. This
is the layer that turns Runcap from a single developer's terminal tool into a
required red/green PR check: an AI-generated change has to earn merge
eligibility instead of asserting it.

What this PR adds

  • runcap ci --mode adjudicate - the CI-side judge. It reads the mission policy
    and the verification command (plus the files it names) from the base
    commit
    , applies only the diff's allowed, in-scope, regular UTF-8 text edits,
    and replays the base-pinned verifier in a throwaway worktree: the verifier
    must fail at base and pass after the change, or the verdict is not PASS.
  • Three verdicts: PASS (exit 0), BLOCKED (exit 1), and
    HUMAN_APPROVAL_REQUIRED (exit 0) when the change touches the policy, a
    workflow, a verifier file, a dependency manifest/lockfile, or a protected
    path.
  • examples/runcap-adjudicate.yml - a hardened consumer reference workflow
    (on: pull_request, permissions: contents: read, persist-credentials: false, every action pinned by full commit SHA, capped runtime, no needs:).
  • Docs: a "CI Adjudication (v0.6)" section in docs/trust-model.md, a reframed
    README, and a CHANGELOG.md 0.6.0 entry. Version bumped 0.5.0 -> 0.6.0.

The receipt does not decide the verdict

The required gate never reads the agent's self-reported receipt. It is
neither graded nor displayed by the gate, so a forged VERIFIED_STRONG receipt
has no effect on the merge decision. The verdict is recomputed from the base
commit every time.

There is no self-adjudicating workflow in this repo

An earlier revision shipped .github/workflows/runcap-adjudicate.yml that ran
the adjudicator against this repo's own PRs. That has been removed. The
reference is now the consumer template at examples/runcap-adjudicate.yml,
which is intentionally not runnable until you insert a release SHA (the
judge action is pinned to an all-zero placeholder).

Bootstrap rule (why this PR is reviewed by a human, not by Runcap)

The judge must come from an immutable source outside the candidate PR.
This implementation PR therefore cannot self-adjudicate:

  • there is no base mission policy (.runcap/mission.{yaml,yml,json}) yet, and
  • base code can't be trusted to judge head code that changes the trust surface.

So the first merge of the Runcap trust surface (adjudicator, policy, verifier,
workflow) is approved by a human CODEOWNER. After a release, a separate
consumer/demo repo
runs the published action pinned by a full 40-char commit
SHA - because that judge is immutable and lives outside the PR, a malicious PR
cannot rewrite its own judge to force a fake PASS.

Honest framing

The verdict is a CI-attested replay under a documented hardened GitHub
profile
. It is not "unspoofable" and not "fully independent": its integrity
rests on the required GitHub setup (protected branch, required check, CODEOWNERS
for the trust surface, no merge bypass, action pinned by immutable SHA) being in
place. It does not meter LLM spend, prove network isolation, or guard against
repo admins. See docs/trust-model.md#ci-adjudication-v06 for the full
"proves / does not prove" breakdown.

Commits in this PR's security + release-prep pass

  • 5fe91aa, 0823d7a, c90191a - Tier 3 security-review fixes (receipt
    excluded from the gate, comment-stripped safety regexes, removed the
    self-adjudicating workflow).
  • 2076993 - release-prep docs + version bump to 0.6.0.
  • 5465da6 - release-safety: adjudicate-by-default Action, truthful README, real check script.

Tests

npm test (delta, loop, loop-e2e, outcome, guard, policy, mission, adjudicate)
and npm run test:tier3 both pass locally, including adversarial cases: a head
PR that rewrites bin/runcap.mjs or src/adjudicate.mjs to fake a PASS is
still BLOCKED, because the trusted adjudicator never imports the head copy.

Not in this PR: npm publish, release tag, Marketplace publication, and the
Tier 2 demo repo - those wait for the next review.

kirder24-code and others added 5 commits June 28, 2026 12:59
Plan-only design for the CI-side evidence runner. Verdict is computed
only by an adjudicator job that re-runs verification from base-SHA
inputs; everything from the agent environment is untrusted telemetry.
Includes threat model, repository-hardening prerequisite (protected
branch + required check + CODEOWNERS), exact base/head handling, the
HUMAN_APPROVAL_REQUIRED lifecycle, and 17 acceptance tests. Not
implemented; for review on this branch only.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…d hardening)

Apply 13 review corrections to the proposal (still no code):
- Required Proof Gate is one self-sufficient adjudicator job; no `needs:`
  on any untrusted executor; agent telemetry is optional/advisory only.
- Three distinct non-PASS states with exact exit semantics:
  BLOCKED (non-zero), HUMAN_APPROVAL_REQUIRED (success/neutral; CODEOWNER
  gates merge), INSTALLATION_NOT_HARDENED (fail closed, never PASS).
- Minimal permissions: contents:read only; no PR comments in v1.
- Base-freshness (branch up to date; no merge queue in v1), stale-approval
  dismissal, no-bypass; separate hardening configuration vs detection
  (HARDENING_UNVERIFIED, never infer HARDENED).
- Rename claim to "protected CI-attested required check" (drop "unspoofable").
- Expanded diff rejection (A/M text only; reject D/R/C/T/symlink/mode/
  submodule/binary/LFS); corrected exfiltration claim + hard timeout;
  base-pinned dependency-install contract (npm ci --ignore-scripts);
  honest budget scope; 9 new acceptance tests.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add src/adjudicate.mjs: an independent evidence runner that never trusts
the agent's receipt. It resolves base/head only from the trusted
pull_request event (or explicit --base/--head), sources the policy,
verifier and lockfile from the base SHA, classifies the diff
(text-only/in-scope -> candidate; structural or out-of-scope -> BLOCKED;
policy/workflow/verifier/dependency edits -> HUMAN_APPROVAL_REQUIRED), and
replays the base-pinned verifier in a clean worktree. Verdict truth is the
adjudicator's own replay; receipt/budget telemetry is advisory only.

Wire `runcap ci --mode adjudicate` in bin/runcap.mjs (exit 0 on
PASS/HUMAN_APPROVAL_REQUIRED, 1 on BLOCKED) without disturbing the existing
receipt-grading path.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add .github/workflows/runcap-adjudicate.yml: a single self-sufficient
required check on the pull_request event, contents:read only, GitHub-hosted,
timeout-minutes 10, no needs/secrets/id-token, deps via
`npm ci --ignore-scripts`. Extend action.yml with a `mode` input so the
composite action can run the adjudicator as well as the receipt grader.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add scripts/adjudicate-test.mjs covering the three verdict states and the
threat scenarios: honest pass, out-of-scope edit, baseline-already-green,
clean-replay fail, protected/verifier/policy/workflow/dependency human gates,
unresolved SHA, untrusted pull_request_target, diff smuggling
(delete/symlink/binary), forged VERIFIED_STRONG and forged budget telemetry,
the honest hardening-provenance check, the pinned script-free install check,
the real bin exit codes, and the reference workflow least-privilege checks.
Wire it into the `test` chain and a `test:tier3` alias.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@kirder24-code kirder24-code changed the title Tier 3: Independent Evidence Runner (plan only, no code) Tier 3: Independent Evidence Runner (CI adjudicator) Jun 28, 2026
kirder24-code and others added 3 commits June 28, 2026 14:01
The agent's receipt is agent-controlled input - a forged VERIFIED_STRONG
receipt is exactly the attack the gate exists to defeat. The required job
now neither grades nor displays it: parsing attacker-controlled JSON in the
mandatory check is needless attack surface (a malformed or enormous receipt
could crash or stall the only gate guarding the merge). The verdict reports
a frozen constant stating no receipt was consulted.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Runcap's own repo cannot self-adjudicate the bootstrap PR (no base policy
exists at the base commit, and head code can't be trusted to judge itself),
so the self-running workflow is removed and replaced with a consumer-facing
template under examples/. The judge is the released Runcap action pinned by a
full 40-char commit SHA - never PR-workspace code, never `uses: ./`, never an
`npm ci` of the PR manifest. Checkout uses persist-credentials: false and
every action is pinned by full SHA.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds security coverage: a forged receipt is neither graded nor read
(present=false); adversarial receipts (malformed JSON, 5MB blob, bare array,
path-traversal latest pointer) cannot crash or stall the gate; the reference
template never executes PR-workspace code, pins every action by full SHA, and
sets persist-credentials: false; and a head PR rewriting bin/runcap.mjs or
src/adjudicate.mjs to force PASS is still BLOCKED because the judge is the
trusted released-action code, never the head copy. Safety regexes strip YAML
comments so they assert on effective directives, not header prose.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@kirder24-code

Copy link
Copy Markdown
Owner Author

Security review applied (3 commits) - green CI

Addressed the proof-gate threat model before any docs/version/merge.

Why the self-running Runcap adjudicate workflow was removed (the live CI failure):
The prior run (28333774203) returned BLOCKED / exit 1 with the exact reason:
no policy (.runcap/mission.{yaml,yml,json}) found at the base commit. This is the
bootstrap paradox: Runcap's own implementation PR has no base policy to adjudicate
against, and - more importantly - a candidate PR must never judge itself. The first
merge of Tier 3 therefore requires human review. Dogfooding via the released action
(pinned by SHA) happens in a separate PR after release.

Fixes:

  1. Required gate never reads the agent receipt (src/adjudicate.mjs). A forged
    VERIFIED_STRONG receipt is the very attack this gate defeats. The job neither grades
    nor displays it - parsing attacker-controlled JSON in the mandatory check is needless
    attack surface (a malformed/enormous receipt could crash the only merge gate). The
    verdict reports a frozen agent_telemetry: {present:false, influence_on_verdict:none}.
  2. Consumer reference is a SHA-pinned template (examples/runcap-adjudicate.yml),
    not a self-running workflow. The judge is the released Runcap action pinned by a full
    40-char commit SHA - never node ./bin/runcap.mjs from the PR workspace, never
    uses: ./, never npm ci of the PR manifest. Checkout uses persist-credentials: false
    and every action is pinned by full SHA.
  3. Security tests (scripts/adjudicate-test.mjs): adversarial receipts (malformed,
    5MB blob, bare array, path-traversal latest) cannot crash/stall the gate; the
    template never contains workspace-execution patterns and pins every action by SHA; a
    head PR rewriting bin/runcap.mjs or src/adjudicate.mjs to force PASS is still
    BLOCKED because the judge is the trusted released-action code, never the head copy.

npm test (full chain) and npm run test:tier3 both pass. Latest CI run on this branch
(28334291609) is green - only ci.yml runs now.

Still NOT done pending independent review: README/trust-model docs, version bump, merge,
npm publish, release tag, Tier 2.

kirder24-code and others added 3 commits June 28, 2026 14:11
… CHANGELOG

Bump 0.5.0 -> 0.6.0 (new runcap ci --mode adjudicate + public integration
model). Reframe README around earning merge eligibility: base-pinned policy /
verifier, clean-room replay, PASS / BLOCKED / HUMAN_APPROVAL_REQUIRED, and the
honest "CI-attested replay under a documented hardened GitHub profile" framing
(not unspoofable, not fully independent, not independent budget enforcement).
Document the consumer install path against examples/runcap-adjudicate.yml.
Append the "CI Adjudication (v0.6)" section to the trust model (what it proves,
required GitHub setup, what it does NOT prove, bootstrap rule). Add CHANGELOG.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…, real check script

- action.yml: rename to "Runcap Proof Gate"; default mode adjudicate (grade is an
  explicit legacy/local-receipt compat mode, flagged as not merge-proof); drop the
  "trusted outcome evidence" claim from the description.
- package.json: `check` now also `node --check`s src/adjudicate.mjs, matching what
  the release report claims it validates.
- README: drop "100% local / never touch a server" (untrue with optional CI
  adjudication and remote model calls); soften "the layer no other proxy has" and
  "does the thing they don't"; replace the thesis with "AI can propose a change.
  It should not certify its own success."; replace the hosted pricing table
  (Founding Pro / Pro / Team) with an Availability section - those plans do not
  exist yet.
- trust-model: clarify that local grading and `--mode grade` carry agent-environment
  evidence; only `--mode adjudicate` under the hardened profile is the merge gate.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@kirder24-code
kirder24-code marked this pull request as ready for review June 28, 2026 20:23
@kirder24-code
kirder24-code merged commit 1eb8745 into main Jun 28, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant