Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions docs/build/sprints.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@ Status: **Done** (merged and signed off by Jim), **In progress** (work merged or
| C4 | Full eval run | paticolcol | **Done** (PR #21; signed off by Jim 2026-11-12) | `73a0e0d` (seven presets, call counts, cost); `docs/results/results.json` (`78e3fcb`, total 300); `classifier_only` not run (classifier model download failed); MELON not built |
| C5 | Conformal hold threshold + unseen-family rotation | paticolcol | **Done** (PR #21; signed off by Jim 2026-11-12) | q̂ frozen at 0.84 (`61e9f52`), 150 benign, ε = 5%; realised false-hold 4.67% (7/150); `calibration.py` strict `>` fix `28a33e9` |
| C6 | README results and limits; freeze thresholds | — | **Done** (PR #21; signed off by Jim 2026-11-12) | `README.md` Results and Limits from `docs/results/results.json` (`78e3fcb`); `FUSION_HOLD_THRESHOLD` 0.84 |
| D1 | Landing page skeleton | Helena | **In progress** | Merged in PR #14 (`site/`); not deployed yet (done when a deployed URL exists) |
| D1 | Landing page skeleton | Helena | **Done** | Merged in PR #14 (`site/`); deployed 2026-11-13 to https://eclipse-20v.pages.dev/ (Cloudflare Pages project `eclipse`, from main `38aeaf0`) |
| D4 | Results section on the real `results.json` | Helena | **In progress** | PR #24: E3's `ResultsSection` reads `docs/results/results.json` (P6, classifier on) and `redteam-results.json`; per-family from `by_family`; `cd site && npm test` 28/28, `npm run build` pass |
| E1 | Live receipts panel | Ace | **In progress** | PR #11 review changes: panel integrated into `site/` using the shared React 18.3.1 / Vite 5.4.21 stack and brand; standalone app removed. 8/8 unit tests, production build and 7 Edge browser checks pass (390px, no horizontal scroll); Helena's re-review pending; see [site/README.md](../../site/README.md) |
| P1 | Classifier fails loudly; `firewall-eval --preset`, scenario sets `external` and `paysim` | Jim | **Done** (signed off by Jim 2026-11-13) | `805a573`, follow-ups `73cb901`, `457d727`; signed off by Jim 2026-11-13 |
Expand Down Expand Up @@ -102,7 +102,7 @@ Sprints 2–3: planned (owners and due dates below), not started. Sprints 4–5:
| B2 | B | Backend | S | Nightly end-to-end script: one benign and one attack payment on testnet (human runs it). **Done, merged in PR #9 (`agent/src/firewall_agent/nightly_e2e.py` (`firewall-nightly-e2e`), `scripts/nightly-e2e.*`; unit tests in `agent/tests/test_nightly_e2e.py`; runbook & expected output in [nightly-e2e.md](nightly-e2e.md)); signed off by Jim 2026-11-11** | Script plus expected output | — | Rc | Done |
| C1 | C | Backend | S | Freeze `results.json` (schema + example under `schemas/`) and add fixture DecisionRecords with filled signals for fusion tests. **Done, merged in PR #7; signed off by Jim 2026-11-11** | Schema and example committed; E3 and A3 can use them | — | paticolcol | Done |
| C2 | C | Backend | M | Injection classifier: host `protectai/deberta-v3-base-prompt-injection-v2`, checker client scores each context item into `signals.injection`, measure latency. **Done, merged in PR #7; signed off by Jim 2026-11-11** | Score per item in the record; latency recorded | frozen `signals.injection` | paticolcol | Done |
| D1 | D | Frontend | M | Landing page skeleton: choose a static stack and host (must stay up through 2026-11-18, ideally 2026-11-25); sections: problem, how it works, NEAR's role, Jev's role, safety layers, demo, results, links | Deployed URL with draft copy | — | Helena | Not started |
| D1 | D | Frontend | M | Landing page skeleton: choose a static stack and host (must stay up through 2026-11-18, ideally 2026-11-25); sections: problem, how it works, NEAR's role, Jev's role, safety layers, demo, results, links | Deployed URL with draft copy | — | Helena | Done (deployed to https://eclipse-20v.pages.dev/ on 2026-11-13) |
| E1 | E | Frontend | M | Live receipts panel: read-only calls to Guardian views over a public testnet RPC, explorer links, no wallet. **2026-11-11: integrated into `site/`, with filtering, pagination, refresh and exact token amounts; Helena's re-review pending.** | Shows the existing testnet receipts | live testnet Guardian | Ace | In progress |

**Decision at the end of Sprint 1:** if A1 is not working, lane A replaces A4 with **A0: HTTPS + signed-verdict fallback mode** ([idea.md Risks](idea.md#risks-and-fallbacks)) in Sprint 2 and A4 moves to Sprint 3.
Expand Down
4 changes: 3 additions & 1 deletion site/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,9 +56,11 @@ The section shows three headline numbers for the conformal row against no defens

## Deploy (human, needs a Cloudflare account)

Live at https://eclipse-20v.pages.dev/ (Cloudflare Pages project `eclipse`, first deployed 2026-11-13 from main `38aeaf0`). The project is not connected to GitHub: redeploy from `site/` on an up-to-date `main` after each merge that touches the site.

```
npm run build
npx wrangler pages deploy dist --project-name <name>
npx wrangler pages deploy dist --project-name eclipse --branch main
```

The first run asks for a Cloudflare login and creates the project. Any static host works: `dist/` uses relative paths.
Expand Down
5 changes: 3 additions & 2 deletions site/src/components/ResultsSection.jsx
Original file line number Diff line number Diff line change
Expand Up @@ -135,7 +135,7 @@ export function ResultsSection({ rows = resultRows }) {
<td className="num">n/a</td>
<td className="num">{formatResultNumber(redteamResult.missedPct)} <small>({redteamResult.held} held, {redteamResult.blocked} blocked)</small></td>
<td className="num">{formatResultNumber(redteamResult.medianMs)}</td>
<td className="num">1 Jev call + 1 classifier call</td>
<td className="num">{formatCalls(redteamResult)}</td>
</tr>
<tr className="results-extra">
<th scope="row"><span className="results-config">melon</span><span className="results-description">Masked re-run (MELON) only</span></th>
Expand Down Expand Up @@ -165,7 +165,8 @@ export function ResultsSection({ rows = resultRows }) {
</li>
<li>
Detection covers the six evaluated attack families. Against new or adaptive attacks, the loss is bounded by the
hard caps in contract code: {PER_PAYMENT_CAP} tUSDC per payment, {DAILY_CAP} tUSDC per day.
hard caps in contract code: {PER_PAYMENT_CAP} tUSDC per payment, and {DAILY_CAP} tUSDC committed per UTC request
day. Payouts on one day can also include holds committed the day before.
</li>
<li>
Latency includes the injection classifier running on a local CPU. Cost is unpriced, so the table shows calls per
Expand Down
3 changes: 3 additions & 0 deletions site/src/results.js
Original file line number Diff line number Diff line change
Expand Up @@ -68,13 +68,16 @@ export function headline(results) {
export function redteamRow(redteam) {
validateResultsFormat(redteam);
const s = redteam.summary;
const [config] = redteam.configurations ?? [];
return {
total: redteam.attack_total,
missedPct: s.missed_attacks_pct,
missed: s.missed_attacks,
held: s.held_attacks,
blocked: s.blocked_attacks,
medianMs: redteam.latency_ms.p50_ms,
jevCalls: config?.jev_calls_per_payment ?? 0,
classifierCalls: config?.classifier_calls_per_payment ?? 0,
};
}

Expand Down
6 changes: 4 additions & 2 deletions site/src/results.test.js
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,9 @@ test("maps every real configuration in source order with a README label", () =>
test("headline, red-team and family numbers match the README Results section", () => {
const { base, best } = headline(results);
assert.deepEqual([base.missed_attacks_pct, best.missed_attacks_pct, best.false_blocks_pct, best.false_holds_pct], [100, 19.33, 0, 5.33]);
assert.deepEqual(redteamRow(redteam), { total: 17, missedPct: 41.18, missed: 7, held: 7, blocked: 3, medianMs: 1137.9 });
assert.deepEqual(redteamRow(redteam), { total: 17, missedPct: 41.18, missed: 7, held: 7, blocked: 3, medianMs: 1137.9,
jevCalls: redteam.configurations[0].jev_calls_per_payment, classifierCalls: redteam.configurations[0].classifier_calls_per_payment });
assert.equal(formatCalls(redteamRow(redteam)), "1 Jev call + 1 classifier call");
const missed = Object.fromEntries(familyRows(results).map((f) => [f.family, f.total - f.caught]));
assert.deepEqual(missed, { direct_injection: 1, encoded_injection: 2, indirect_injection: 1, lookalike_receiver: 2, fake_invoice: 12, memory_poisoning: 12 });
});
Expand All @@ -57,7 +59,7 @@ test("renders the real results with all rows, units and caveats, and no sample l
const html = renderToStaticMarkup(React.createElement(ResultsSection));
for (const entry of results.configurations) assert.ok(html.includes(entry.configuration), entry.configuration);
for (const label of ["False holds (%)", "False blocks (%)", "Missed attacks (%)", "Median latency (ms)", "Per payment",
"19.33%", "exchangeability", "in-sample", "six evaluated attack families", "Not built (cut list)", "docs/results/results.json"]) {
"19.33%", "exchangeability", "in-sample", "six evaluated attack families", "committed per UTC request", "Not built (cut list)", "docs/results/results.json"]) {
assert.ok(html.includes(label), `Missing ${label}`);
}
assert.doesNotMatch(html, /Sample data|Cost per payment|Numbers land after/);
Expand Down
Loading