Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -371,6 +371,20 @@ See a public consumer-repository demo with three live pull requests: an in-scope

Each verdict was produced by the pinned Runcap action running in GitHub Actions on a real pull request.

### Watch the live proof

[![Watch the Runcap Proof Gate live demo](docs/assets/runcap-proof-gate-demo-poster.png)](docs/assets/runcap-proof-gate-demo-v0.6.0.mp4)

Watch a 70-second screen recording made from the public consumer demo and its real GitHub Actions runs. It shows one problem and three verdicts:

> An AI-generated PR can make CI green by changing the test that proves it succeeded.

- a scoped source fix → `PASS`
- a correct fix plus an unrelated file → `BLOCKED`
- a verifier edit → `HUMAN_APPROVAL_REQUIRED`

This is a CI-attested replay under a documented hardened GitHub profile - not an "unspoofable" or "fully independent" guarantee. The recording is an edited screen recording assembled from real public GitHub browser captures (no synthetic UI). See the live evidence on the [demo repo](https://github.com/kirder24-code/runcap-proof-gate-demo#live-evidence) and pull requests [#1](https://github.com/kirder24-code/runcap-proof-gate-demo/pull/1), [#2](https://github.com/kirder24-code/runcap-proof-gate-demo/pull/2), [#3](https://github.com/kirder24-code/runcap-proof-gate-demo/pull/3). The recording plan, captions, shot list, and evidence manifest live in [`docs/media/`](docs/media/).

## Pricing table

Costs are calculated from a sourced multi-provider table - Anthropic (Opus / Sonnet / Haiku), OpenAI (GPT-5 family + legacy GPT-4), and DeepSeek (V4 Flash / V4 Pro) - with cache-read and batch discounts handled, labeled with source and verification date. When a model is unknown, Runcap says `unknown_price` rather than guessing.
Expand Down
Binary file added docs/assets/runcap-proof-gate-demo-poster.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/assets/runcap-proof-gate-demo-v0.6.0.mp4
Binary file not shown.
39 changes: 39 additions & 0 deletions docs/media/proof-gate-demo-captions.srt
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
1
00:00:00,000 --> 00:00:08,000
An AI-generated PR can make CI green by changing the test that proves it succeeded.

2
00:00:08,000 --> 00:00:16,000
This is a public demo repo. The verifier requires both admin and member, but the code only allows admin, so the base branch fails on purpose.

3
00:00:16,000 --> 00:00:25,000
The gate is the Runcap action, pinned to release commit 1eb8745, running as a pull-request check.

4
00:00:25,000 --> 00:00:34,000
PR #1 edits only the source file to allow member too. The check is green. The log says Verdict: PASS.

5
00:00:34,000 --> 00:00:42,000
PR #2 makes the same correct fix, but adds an unrelated file outside the allowed scope.

6
00:00:42,000 --> 00:00:50,000
The check is red. Verdict: BLOCKED, naming docs/unrelated-change.md as out of scope.

7
00:00:50,000 --> 00:00:58,000
PR #3 doesn't fix the code. It edits the verifier - the proof itself.

8
00:00:58,000 --> 00:01:05,000
Verdict: HUMAN_APPROVAL_REQUIRED. Runcap declines to auto-certify and hands the decision to a human CODEOWNER.

9
00:01:05,000 --> 00:01:12,000
CI-attested replay under a documented hardened GitHub profile.

10
00:01:12,000 --> 00:01:15,000
AI can propose a change. It should not certify its own success.
82 changes: 82 additions & 0 deletions docs/media/proof-gate-demo-evidence.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
# Runcap Proof Gate demo - evidence manifest

This records exactly how the screen recording in `docs/assets/` was produced and
what it shows. It exists so a reviewer can confirm the video is real and not a
synthetic product UI.

## Source URLs captured

All pages are public and were captured logged-out (no account, no token, no
session). The capture browser showed the GitHub "Sign in / Sign up" header
throughout.

- Demo repo: https://github.com/kirder24-code/runcap-proof-gate-demo
- Demo repo "Live evidence" section: https://github.com/kirder24-code/runcap-proof-gate-demo#live-evidence
- Source fixture: https://github.com/kirder24-code/runcap-proof-gate-demo/blob/main/src/access.mjs
- Verifier fixture: https://github.com/kirder24-code/runcap-proof-gate-demo/blob/main/scripts/verify.mjs
- PR #1 (PASS): https://github.com/kirder24-code/runcap-proof-gate-demo/pull/1
- PR #2 (BLOCKED): https://github.com/kirder24-code/runcap-proof-gate-demo/pull/2
- PR #3 (HUMAN_APPROVAL_REQUIRED): https://github.com/kirder24-code/runcap-proof-gate-demo/pull/3
- Run #1: https://github.com/kirder24-code/runcap-proof-gate-demo/actions/runs/28336067039
- Run #2: https://github.com/kirder24-code/runcap-proof-gate-demo/actions/runs/28336110446
- Run #3: https://github.com/kirder24-code/runcap-proof-gate-demo/actions/runs/28336160932

Judge: the Runcap action pinned to the v0.6.0 release commit
`1eb87456333093c9fb8da6e9c21eef8d850891bc`.

## Expected verdicts (the three the video shows)

| PR | Change | Verdict | Run status |
| --- | --- | --- | --- |
| #1 | `src/access.mjs` only (allow member) | `PASS` | success |
| #2 | correct fix plus `docs/unrelated-change.md` | `BLOCKED` | failure |
| #3 | edits `scripts/verify.mjs` | `HUMAN_APPROVAL_REQUIRED` | success / neutral |

## What is shown, and an honest note on log visibility

Logged-out, GitHub hides the raw Actions step logs ("Sign in to view logs"), so
the literal `Verdict:` lines are not visible to an anonymous viewer on the run
pages themselves. The recording therefore shows two real public surfaces per
scenario:

1. the live Actions run page - the real GitHub-rendered **Status** (Success /
Failure) and exit-code annotation; and
2. the demo repo's committed **"Live evidence"** section, which publicly renders
the exact `Verdict: PASS` / `Verdict: BLOCKED` / `Verdict:
HUMAN_APPROVAL_REQUIRED` lines and their reasons, transcribed from those same
runs.

No step logs were reconstructed, retyped into a fake terminal, or invented. The
narration overlays are subtitles drawn on top of the real captured browser
screen.

## Capture method

- Tool: Playwright (Chromium) video recording at 1280x720, run from a temporary
directory outside this repository. No Playwright, ffmpeg, or video dependency
was added to `package.json`.
- The raw capture was a continuous browser navigation across the public pages
above; it was then speed-adjusted and re-encoded with ffmpeg to land within the
60-75 second target. It is therefore not a single real-time take.

Edited screen recording assembled from real public GitHub browser captures.
No synthetic product UI or invented terminal output was used.

## Artifacts

- Video: `docs/assets/runcap-proof-gate-demo-v0.6.0.mp4`
- Duration: 70.0 seconds
- Resolution: 1280x720, H.264 (yuv420p) MP4
- Size: 1,889,139 bytes (~1.8 MB)
- SHA-256: `1a7189e2479df40bb706b0fd96eaec15d1f749db330d451f4568624393181cd2`
- Poster: `docs/assets/runcap-proof-gate-demo-poster.png`
- Captured from the real demo repo landing page (frame from the video)
- Resolution: 1280x720
- Size: 325,515 bytes (~0.32 MB)
- SHA-256: `db513fa8fcdb812a2b32ca7d369010bf045e5bc1990d924f036d94982c20942f`

## What this does not claim

The video and this manifest do not claim the gate is "unspoofable", "fully
independent", a "cryptographic proof", or that it "guarantees safe merges". It is
a CI-attested replay under a documented hardened GitHub profile.
93 changes: 93 additions & 0 deletions docs/media/proof-gate-demo-recording-plan.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
# Runcap Proof Gate - demo recording plan (45-75 seconds)

A factual, reproducible terminal-and-browser recording. Every screen below is a
real page or a real command output. No synthetic UI, no fabricated terminal text,
no staged screenshots, no invented numbers.

Suggested output file: `runcap-proof-gate-demo-v0.6.0.mp4`

## Source of truth (only these)

- Demo repo: https://github.com/kirder24-code/runcap-proof-gate-demo
- PASS PR: https://github.com/kirder24-code/runcap-proof-gate-demo/pull/1
- BLOCKED PR: https://github.com/kirder24-code/runcap-proof-gate-demo/pull/2
- HUMAN_APPROVAL_REQUIRED PR: https://github.com/kirder24-code/runcap-proof-gate-demo/pull/3
- Runcap v0.6.0 release commit: `1eb87456333093c9fb8da6e9c21eef8d850891bc`

The exact `Verdict:` and reason lines for each PR are quoted in the demo repo
README's "Live evidence" section. Logged-out GitHub viewers cannot access raw
Actions step logs. For each scenario, the recording shows the real public
Actions run page for its Success / Failure status and the demo repo's public
"Live evidence" section for the exact `Verdict:` and reason lines transcribed
from that same run.

## Timeline

**[0:00-0:08] The problem (title card or narration over the demo repo home)**

On screen, the opening line, verbatim:

```
An AI-generated PR can make CI green by changing the test that proves it succeeded.
```

Show the demo repo landing page so the viewer sees this is a real public repo.

**[0:08-0:16] The setup (browse the fixture)**

Open `scripts/verify.mjs` and `src/access.mjs` on the repo's default branch.
Narrate: the verifier requires both `admin` and `member`, the code only allows
`admin`, so the base branch fails on purpose. The gate is the Runcap action,
pinned to release commit `1eb8745`, running as a pull-request check.

**[0:16-0:34] Scenario 1 - PASS (PR #1)**

Open PR #1. Show the green Actions status, then show the demo repo's public
"Live evidence" section with `Verdict: PASS`. Narrate: the fix edits only
`src/access.mjs`; the verifier failed at the base commit and passed after the
in-scope change, replayed in a clean checkout.

**[0:34-0:50] Scenario 2 - BLOCKED (PR #2)**

Open PR #2. Show the red Actions status, then show the demo repo's public
"Live evidence" section with `Verdict: BLOCKED` and the reason naming
`docs/unrelated-change.md` as outside the allowed scope. Narrate: same correct
fix, but an unrelated out-of-scope file rides along, so the gate blocks it.

**[0:50-1:05] Scenario 3 - HUMAN_APPROVAL_REQUIRED (PR #3)**

Open PR #3. Show the successful / neutral Actions status, then show the demo
repo's public "Live evidence" section with `Verdict: HUMAN_APPROVAL_REQUIRED`
and the reason naming `scripts/verify.mjs` as a verifier/evidence change.
Narrate: this PR edits the verifier - the proof itself - so Runcap declines to
auto-certify and hands the decision to a human CODEOWNER.

**[1:05-1:12] The honesty line (on-screen, required)**

```
CI-attested replay under a documented hardened GitHub profile.
```

**[1:12-1:15] Close (title card)**

```
AI can propose a change.
It should not certify its own success.
```

## What the recorder must NOT do

- Do not claim "unspoofable", "fully independent", "cryptographic proof", or
"guaranteed secure merges".
- Do not fabricate terminal output, GitHub screenshots, workflow conclusions,
customer logos, adoption numbers, or benchmark results.
- Do not build a synthetic or mock UI. Use the real PR pages and real Actions
run pages.
- The three demo PRs are intentionally open and unmerged - leave them that way
so the runs stay inspectable.

## Reproducing the runs (optional, for the recorder)

The three PRs already exist with their live runs. To re-derive a verdict
locally against the same released judge, a viewer can read the demo repo
README's "Live evidence" and "Why the base verifier fails on main" sections.
28 changes: 28 additions & 0 deletions docs/media/proof-gate-demo-shot-list.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
# Runcap Proof Gate - demo shot list

Each shot is a real page. No synthetic UI. Capture verdict lines by reading the
live Actions run log, not by retyping.

| # | Duration | Screen / URL | What must be visible |
| --- | --- | --- | --- |
| 1 | 0:00-0:08 | Title card over https://github.com/kirder24-code/runcap-proof-gate-demo | Opening line: "An AI-generated PR can make CI green by changing the test that proves it succeeded." Real repo landing page behind it. |
| 2 | 0:08-0:13 | https://github.com/kirder24-code/runcap-proof-gate-demo/blob/main/scripts/verify.mjs | Verifier asserts both `admin` and `member` have access. |
| 3 | 0:13-0:16 | https://github.com/kirder24-code/runcap-proof-gate-demo/blob/main/src/access.mjs | `canAccess` returns true only for `admin` (so base fails). |
| 4 | 0:16-0:20 | https://github.com/kirder24-code/runcap-proof-gate-demo/blob/main/.github/workflows/runcap-adjudicate.yml | Runcap action pinned to commit `1eb87456333093c9fb8da6e9c21eef8d850891bc`; `on: pull_request`; `permissions: contents: read`. |
| 5 | 0:20-0:34 | https://github.com/kirder24-code/runcap-proof-gate-demo/pull/1 then its run https://github.com/kirder24-code/runcap-proof-gate-demo/actions/runs/28336067039 | Green check on PR #1; run log line `Verdict: PASS` and the "failed at base and passed after ... in-scope text change(s), recomputed in a clean base checkout" reason. |
| 6 | 0:34-0:50 | https://github.com/kirder24-code/runcap-proof-gate-demo/pull/2 then its run https://github.com/kirder24-code/runcap-proof-gate-demo/actions/runs/28336110446 | Red check on PR #2; run log line `Verdict: BLOCKED` and "docs/unrelated-change.md: outside the policy's allowed scope". |
| 7 | 0:50-1:05 | https://github.com/kirder24-code/runcap-proof-gate-demo/pull/3 then its run https://github.com/kirder24-code/runcap-proof-gate-demo/actions/runs/28336160932 | PR #3; run log line `Verdict: HUMAN_APPROVAL_REQUIRED` and "scripts/verify.mjs: edits a verifier file (the evidence) - human CODEOWNER must approve". |
| 8 | 1:05-1:12 | On-screen honesty line | "CI-attested replay under a documented hardened GitHub profile." |
| 9 | 1:12-1:15 | Closing title card | "AI can propose a change." / "It should not certify its own success." |

## Notes for the recorder

- Show the Actions tab and each PR's checks live. Do not fabricate screenshots or
workflow conclusions.
- The exact `Verdict:` and reason lines are also quoted in the demo repo README's
"Live evidence" section - cross-check against the live run, do not paraphrase.
- The three PRs are intentionally open and unmerged. Do not merge them while
recording.
- Do not claim "unspoofable", "fully independent", "cryptographic proof", or
"guaranteed secure merges".
- Output file: `runcap-proof-gate-demo-v0.6.0.mp4`.
Loading