diff --git a/README.md b/README.md index 69f816b..b13a9d3 100644 --- a/README.md +++ b/README.md @@ -371,6 +371,20 @@ See a public consumer-repository demo with three live pull requests: an in-scope Each verdict was produced by the pinned Runcap action running in GitHub Actions on a real pull request. +### Watch the live proof + +[![Watch the Runcap Proof Gate live demo](docs/assets/runcap-proof-gate-demo-poster.png)](docs/assets/runcap-proof-gate-demo-v0.6.0.mp4) + +Watch a 70-second screen recording made from the public consumer demo and its real GitHub Actions runs. It shows one problem and three verdicts: + +> An AI-generated PR can make CI green by changing the test that proves it succeeded. + +- a scoped source fix → `PASS` +- a correct fix plus an unrelated file → `BLOCKED` +- a verifier edit → `HUMAN_APPROVAL_REQUIRED` + +This is a CI-attested replay under a documented hardened GitHub profile - not an "unspoofable" or "fully independent" guarantee. The recording is an edited screen recording assembled from real public GitHub browser captures (no synthetic UI). See the live evidence on the [demo repo](https://github.com/kirder24-code/runcap-proof-gate-demo#live-evidence) and pull requests [#1](https://github.com/kirder24-code/runcap-proof-gate-demo/pull/1), [#2](https://github.com/kirder24-code/runcap-proof-gate-demo/pull/2), [#3](https://github.com/kirder24-code/runcap-proof-gate-demo/pull/3). The recording plan, captions, shot list, and evidence manifest live in [`docs/media/`](docs/media/). + ## Pricing table Costs are calculated from a sourced multi-provider table - Anthropic (Opus / Sonnet / Haiku), OpenAI (GPT-5 family + legacy GPT-4), and DeepSeek (V4 Flash / V4 Pro) - with cache-read and batch discounts handled, labeled with source and verification date. When a model is unknown, Runcap says `unknown_price` rather than guessing. diff --git a/docs/assets/runcap-proof-gate-demo-poster.png b/docs/assets/runcap-proof-gate-demo-poster.png new file mode 100644 index 0000000..e86a8e2 Binary files /dev/null and b/docs/assets/runcap-proof-gate-demo-poster.png differ diff --git a/docs/assets/runcap-proof-gate-demo-v0.6.0.mp4 b/docs/assets/runcap-proof-gate-demo-v0.6.0.mp4 new file mode 100644 index 0000000..1387075 Binary files /dev/null and b/docs/assets/runcap-proof-gate-demo-v0.6.0.mp4 differ diff --git a/docs/media/proof-gate-demo-captions.srt b/docs/media/proof-gate-demo-captions.srt new file mode 100644 index 0000000..29bead6 --- /dev/null +++ b/docs/media/proof-gate-demo-captions.srt @@ -0,0 +1,39 @@ +1 +00:00:00,000 --> 00:00:08,000 +An AI-generated PR can make CI green by changing the test that proves it succeeded. + +2 +00:00:08,000 --> 00:00:16,000 +This is a public demo repo. The verifier requires both admin and member, but the code only allows admin, so the base branch fails on purpose. + +3 +00:00:16,000 --> 00:00:25,000 +The gate is the Runcap action, pinned to release commit 1eb8745, running as a pull-request check. + +4 +00:00:25,000 --> 00:00:34,000 +PR #1 edits only the source file to allow member too. The check is green. The log says Verdict: PASS. + +5 +00:00:34,000 --> 00:00:42,000 +PR #2 makes the same correct fix, but adds an unrelated file outside the allowed scope. + +6 +00:00:42,000 --> 00:00:50,000 +The check is red. Verdict: BLOCKED, naming docs/unrelated-change.md as out of scope. + +7 +00:00:50,000 --> 00:00:58,000 +PR #3 doesn't fix the code. It edits the verifier - the proof itself. + +8 +00:00:58,000 --> 00:01:05,000 +Verdict: HUMAN_APPROVAL_REQUIRED. Runcap declines to auto-certify and hands the decision to a human CODEOWNER. + +9 +00:01:05,000 --> 00:01:12,000 +CI-attested replay under a documented hardened GitHub profile. + +10 +00:01:12,000 --> 00:01:15,000 +AI can propose a change. It should not certify its own success. diff --git a/docs/media/proof-gate-demo-evidence.md b/docs/media/proof-gate-demo-evidence.md new file mode 100644 index 0000000..244d2d6 --- /dev/null +++ b/docs/media/proof-gate-demo-evidence.md @@ -0,0 +1,82 @@ +# Runcap Proof Gate demo - evidence manifest + +This records exactly how the screen recording in `docs/assets/` was produced and +what it shows. It exists so a reviewer can confirm the video is real and not a +synthetic product UI. + +## Source URLs captured + +All pages are public and were captured logged-out (no account, no token, no +session). The capture browser showed the GitHub "Sign in / Sign up" header +throughout. + +- Demo repo: https://github.com/kirder24-code/runcap-proof-gate-demo +- Demo repo "Live evidence" section: https://github.com/kirder24-code/runcap-proof-gate-demo#live-evidence +- Source fixture: https://github.com/kirder24-code/runcap-proof-gate-demo/blob/main/src/access.mjs +- Verifier fixture: https://github.com/kirder24-code/runcap-proof-gate-demo/blob/main/scripts/verify.mjs +- PR #1 (PASS): https://github.com/kirder24-code/runcap-proof-gate-demo/pull/1 +- PR #2 (BLOCKED): https://github.com/kirder24-code/runcap-proof-gate-demo/pull/2 +- PR #3 (HUMAN_APPROVAL_REQUIRED): https://github.com/kirder24-code/runcap-proof-gate-demo/pull/3 +- Run #1: https://github.com/kirder24-code/runcap-proof-gate-demo/actions/runs/28336067039 +- Run #2: https://github.com/kirder24-code/runcap-proof-gate-demo/actions/runs/28336110446 +- Run #3: https://github.com/kirder24-code/runcap-proof-gate-demo/actions/runs/28336160932 + +Judge: the Runcap action pinned to the v0.6.0 release commit +`1eb87456333093c9fb8da6e9c21eef8d850891bc`. + +## Expected verdicts (the three the video shows) + +| PR | Change | Verdict | Run status | +| --- | --- | --- | --- | +| #1 | `src/access.mjs` only (allow member) | `PASS` | success | +| #2 | correct fix plus `docs/unrelated-change.md` | `BLOCKED` | failure | +| #3 | edits `scripts/verify.mjs` | `HUMAN_APPROVAL_REQUIRED` | success / neutral | + +## What is shown, and an honest note on log visibility + +Logged-out, GitHub hides the raw Actions step logs ("Sign in to view logs"), so +the literal `Verdict:` lines are not visible to an anonymous viewer on the run +pages themselves. The recording therefore shows two real public surfaces per +scenario: + +1. the live Actions run page - the real GitHub-rendered **Status** (Success / + Failure) and exit-code annotation; and +2. the demo repo's committed **"Live evidence"** section, which publicly renders + the exact `Verdict: PASS` / `Verdict: BLOCKED` / `Verdict: + HUMAN_APPROVAL_REQUIRED` lines and their reasons, transcribed from those same + runs. + +No step logs were reconstructed, retyped into a fake terminal, or invented. The +narration overlays are subtitles drawn on top of the real captured browser +screen. + +## Capture method + +- Tool: Playwright (Chromium) video recording at 1280x720, run from a temporary + directory outside this repository. No Playwright, ffmpeg, or video dependency + was added to `package.json`. +- The raw capture was a continuous browser navigation across the public pages + above; it was then speed-adjusted and re-encoded with ffmpeg to land within the + 60-75 second target. It is therefore not a single real-time take. + +Edited screen recording assembled from real public GitHub browser captures. +No synthetic product UI or invented terminal output was used. + +## Artifacts + +- Video: `docs/assets/runcap-proof-gate-demo-v0.6.0.mp4` + - Duration: 70.0 seconds + - Resolution: 1280x720, H.264 (yuv420p) MP4 + - Size: 1,889,139 bytes (~1.8 MB) + - SHA-256: `1a7189e2479df40bb706b0fd96eaec15d1f749db330d451f4568624393181cd2` +- Poster: `docs/assets/runcap-proof-gate-demo-poster.png` + - Captured from the real demo repo landing page (frame from the video) + - Resolution: 1280x720 + - Size: 325,515 bytes (~0.32 MB) + - SHA-256: `db513fa8fcdb812a2b32ca7d369010bf045e5bc1990d924f036d94982c20942f` + +## What this does not claim + +The video and this manifest do not claim the gate is "unspoofable", "fully +independent", a "cryptographic proof", or that it "guarantees safe merges". It is +a CI-attested replay under a documented hardened GitHub profile. diff --git a/docs/media/proof-gate-demo-recording-plan.md b/docs/media/proof-gate-demo-recording-plan.md new file mode 100644 index 0000000..f8a1e30 --- /dev/null +++ b/docs/media/proof-gate-demo-recording-plan.md @@ -0,0 +1,93 @@ +# Runcap Proof Gate - demo recording plan (45-75 seconds) + +A factual, reproducible terminal-and-browser recording. Every screen below is a +real page or a real command output. No synthetic UI, no fabricated terminal text, +no staged screenshots, no invented numbers. + +Suggested output file: `runcap-proof-gate-demo-v0.6.0.mp4` + +## Source of truth (only these) + +- Demo repo: https://github.com/kirder24-code/runcap-proof-gate-demo +- PASS PR: https://github.com/kirder24-code/runcap-proof-gate-demo/pull/1 +- BLOCKED PR: https://github.com/kirder24-code/runcap-proof-gate-demo/pull/2 +- HUMAN_APPROVAL_REQUIRED PR: https://github.com/kirder24-code/runcap-proof-gate-demo/pull/3 +- Runcap v0.6.0 release commit: `1eb87456333093c9fb8da6e9c21eef8d850891bc` + +The exact `Verdict:` and reason lines for each PR are quoted in the demo repo +README's "Live evidence" section. Logged-out GitHub viewers cannot access raw +Actions step logs. For each scenario, the recording shows the real public +Actions run page for its Success / Failure status and the demo repo's public +"Live evidence" section for the exact `Verdict:` and reason lines transcribed +from that same run. + +## Timeline + +**[0:00-0:08] The problem (title card or narration over the demo repo home)** + +On screen, the opening line, verbatim: + +``` +An AI-generated PR can make CI green by changing the test that proves it succeeded. +``` + +Show the demo repo landing page so the viewer sees this is a real public repo. + +**[0:08-0:16] The setup (browse the fixture)** + +Open `scripts/verify.mjs` and `src/access.mjs` on the repo's default branch. +Narrate: the verifier requires both `admin` and `member`, the code only allows +`admin`, so the base branch fails on purpose. The gate is the Runcap action, +pinned to release commit `1eb8745`, running as a pull-request check. + +**[0:16-0:34] Scenario 1 - PASS (PR #1)** + +Open PR #1. Show the green Actions status, then show the demo repo's public +"Live evidence" section with `Verdict: PASS`. Narrate: the fix edits only +`src/access.mjs`; the verifier failed at the base commit and passed after the +in-scope change, replayed in a clean checkout. + +**[0:34-0:50] Scenario 2 - BLOCKED (PR #2)** + +Open PR #2. Show the red Actions status, then show the demo repo's public +"Live evidence" section with `Verdict: BLOCKED` and the reason naming +`docs/unrelated-change.md` as outside the allowed scope. Narrate: same correct +fix, but an unrelated out-of-scope file rides along, so the gate blocks it. + +**[0:50-1:05] Scenario 3 - HUMAN_APPROVAL_REQUIRED (PR #3)** + +Open PR #3. Show the successful / neutral Actions status, then show the demo +repo's public "Live evidence" section with `Verdict: HUMAN_APPROVAL_REQUIRED` +and the reason naming `scripts/verify.mjs` as a verifier/evidence change. +Narrate: this PR edits the verifier - the proof itself - so Runcap declines to +auto-certify and hands the decision to a human CODEOWNER. + +**[1:05-1:12] The honesty line (on-screen, required)** + +``` +CI-attested replay under a documented hardened GitHub profile. +``` + +**[1:12-1:15] Close (title card)** + +``` +AI can propose a change. +It should not certify its own success. +``` + +## What the recorder must NOT do + +- Do not claim "unspoofable", "fully independent", "cryptographic proof", or + "guaranteed secure merges". +- Do not fabricate terminal output, GitHub screenshots, workflow conclusions, + customer logos, adoption numbers, or benchmark results. +- Do not build a synthetic or mock UI. Use the real PR pages and real Actions + run pages. +- The three demo PRs are intentionally open and unmerged - leave them that way + so the runs stay inspectable. + +## Reproducing the runs (optional, for the recorder) + +The three PRs already exist with their live runs. To re-derive a verdict +locally against the same released judge, a viewer can read the demo repo +README's "Live evidence" and "Why the base verifier fails on main" sections. diff --git a/docs/media/proof-gate-demo-shot-list.md b/docs/media/proof-gate-demo-shot-list.md new file mode 100644 index 0000000..95b2d6e --- /dev/null +++ b/docs/media/proof-gate-demo-shot-list.md @@ -0,0 +1,28 @@ +# Runcap Proof Gate - demo shot list + +Each shot is a real page. No synthetic UI. Capture verdict lines by reading the +live Actions run log, not by retyping. + +| # | Duration | Screen / URL | What must be visible | +| --- | --- | --- | --- | +| 1 | 0:00-0:08 | Title card over https://github.com/kirder24-code/runcap-proof-gate-demo | Opening line: "An AI-generated PR can make CI green by changing the test that proves it succeeded." Real repo landing page behind it. | +| 2 | 0:08-0:13 | https://github.com/kirder24-code/runcap-proof-gate-demo/blob/main/scripts/verify.mjs | Verifier asserts both `admin` and `member` have access. | +| 3 | 0:13-0:16 | https://github.com/kirder24-code/runcap-proof-gate-demo/blob/main/src/access.mjs | `canAccess` returns true only for `admin` (so base fails). | +| 4 | 0:16-0:20 | https://github.com/kirder24-code/runcap-proof-gate-demo/blob/main/.github/workflows/runcap-adjudicate.yml | Runcap action pinned to commit `1eb87456333093c9fb8da6e9c21eef8d850891bc`; `on: pull_request`; `permissions: contents: read`. | +| 5 | 0:20-0:34 | https://github.com/kirder24-code/runcap-proof-gate-demo/pull/1 then its run https://github.com/kirder24-code/runcap-proof-gate-demo/actions/runs/28336067039 | Green check on PR #1; run log line `Verdict: PASS` and the "failed at base and passed after ... in-scope text change(s), recomputed in a clean base checkout" reason. | +| 6 | 0:34-0:50 | https://github.com/kirder24-code/runcap-proof-gate-demo/pull/2 then its run https://github.com/kirder24-code/runcap-proof-gate-demo/actions/runs/28336110446 | Red check on PR #2; run log line `Verdict: BLOCKED` and "docs/unrelated-change.md: outside the policy's allowed scope". | +| 7 | 0:50-1:05 | https://github.com/kirder24-code/runcap-proof-gate-demo/pull/3 then its run https://github.com/kirder24-code/runcap-proof-gate-demo/actions/runs/28336160932 | PR #3; run log line `Verdict: HUMAN_APPROVAL_REQUIRED` and "scripts/verify.mjs: edits a verifier file (the evidence) - human CODEOWNER must approve". | +| 8 | 1:05-1:12 | On-screen honesty line | "CI-attested replay under a documented hardened GitHub profile." | +| 9 | 1:12-1:15 | Closing title card | "AI can propose a change." / "It should not certify its own success." | + +## Notes for the recorder + +- Show the Actions tab and each PR's checks live. Do not fabricate screenshots or + workflow conclusions. +- The exact `Verdict:` and reason lines are also quoted in the demo repo README's + "Live evidence" section - cross-check against the live run, do not paraphrase. +- The three PRs are intentionally open and unmerged. Do not merge them while + recording. +- Do not claim "unspoofable", "fully independent", "cryptographic proof", or + "guaranteed secure merges". +- Output file: `runcap-proof-gate-demo-v0.6.0.mp4`.