Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
255 changes: 80 additions & 175 deletions README.md

Large diffs are not rendered by default.

2 changes: 2 additions & 0 deletions docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,8 @@ Frontend D1/E1: [site/README.md](../site/README.md) — shared landing page and
| [results/ledger.md](results/ledger.md) | Ledger of every recorded test and eval run: what ran, keys, result, Jev calls, link to the record; entry template | Adding a run; finding the record of a run |
| [results/results-detail.md](results/results-detail.md) | Full results: current and baseline tables, repeat run, sweep, testnet runs, benchmarks, complete limits | Checking any number behind the README summary |
| [reference.md](reference.md) | Evaluation and benchmark steps 7–12, configuration switches, local run and testnet instructions, function-call key | Running evals, changing config, running locally |
| [try-it.md](try-it.md) | Try-it steps 1–6 in full: keyless dry-run, Jev-key dry-run, decision record, nightly dry-run, with expected output | Running the demo for the first time |
| [third-party.md](third-party.md) | Every third-party library, tool and dataset with link, license and use | Checking attribution or licenses |
| [build/demo-storyboard.md](build/demo-storyboard.md) | D3 demo storyboard: shot list for the 2-minute demo, numbers slide, 20-second GIF plan, open questions | Recording the demo (A8), making the GIF and slide (E4) |
| [build/e4-demo-assets.md](build/e4-demo-assets.md) | E4 numbers-slide sources, regeneration and verification; actual shot 4 footage needed for the GIF | Regenerating the slide or completing the blocked-attack GIF |
| [hackathon-context/HACKATHON.md](hackathon-context/HACKATHON.md) | Official agent instructions and rules snapshot; the live https://nearhackathon.com/HACKATHON.md wins | Before any rules question or submission step |
Expand Down
4 changes: 2 additions & 2 deletions docs/reference.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Reference

Moved from the README on 2026-11-13: evaluation and benchmark steps 7–12, configuration switches, and local run instructions. Steps 1–6 stay in [README Try it](../README.md#try-it).
Moved from the README on 2026-11-13: evaluation and benchmark steps 7–12, configuration switches, and local run instructions. Steps 1–6: [try-it.md](try-it.md) (short form: [README Quick start](../README.md#quick-start)).

## Evaluation and benchmarks

Expand Down Expand Up @@ -52,7 +52,7 @@ cargo build -p checker --target wasm32-wasip2 --release
cd agent && uv sync && uv run pytest
```

Contract wasm files: `target/near/guardian/guardian.wasm` and `target/near/tusdc/tusdc.wasm`. `--no-abi` skips ABI generation, which fails to link on Windows MSVC. The dry-run needs no keys, but without `TYPESAFE_API_KEY` every payment is held (step 3 of Try it in the [README](../README.md#try-it)).
Contract wasm files: `target/near/guardian/guardian.wasm` and `target/near/tusdc/tusdc.wasm`. `--no-abi` skips ABI generation, which fails to link on Windows MSVC. The dry-run needs no keys, but without `TYPESAFE_API_KEY` every payment is held (step 3 in [try-it.md](try-it.md#offline-no-keys)).

### Testnet

Expand Down
13 changes: 11 additions & 2 deletions docs/results/results-detail.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,18 @@
# Results in detail

Moved from the README on 2026-11-13; every number is read from the files in this folder. Short version: [README](../../README.md#results-at-a-glance).
Moved from the README on 2026-11-13; every number is read from the files in this folder. Short version: [README](../../README.md#results).

Two versions of the checker are reported. [Current checker](#current-checker-evaluation-of-2026-11-13) is what the code in this repository does now. [Baseline](#baseline-before-the-provenance-rules-and-jev-v2) is the earlier checker, kept unchanged for comparison. Every number is read from a committed file under [docs/results/](./); the run-by-run log is [ledger.md](ledger.md) (runs R15–R23).

### How it was evaluated

- Three scenario sets: main (150 benign, 150 attacks; six attack families, 25 each; written by the team), external (100 and 100, rewritten from public benchmarks with attribution) and red team (17 attacks).
- Presets switch layers on one at a time, so each layer's effect is measured.
- Split-conformal calibration sets the two hold thresholds.
- Recorded decisions are re-scored offline (`checker --refuse`, no Jev calls).
- A Jev call ledger: 3,441 of 4,100 calls used ([jev-budget.md](jev-budget.md)). Every run is logged in [ledger.md](ledger.md).
- A repeat run for stability, and a rigor review of every number in the results detail ([rigor-review.md](rigor-review.md), grade Weak Accept).

### Current checker: evaluation of 2026-11-13

**Under test.** The checker at commit `5d57145`, with three changes against the baseline:
Expand Down Expand Up @@ -106,7 +115,7 @@ These runs used 595 Jev calls; the count per run is in [jev-budget.md](jev-budge
3. Run `cargo test --workspace` → every suite ends with `test result: ok`.
4. In `agent/`, run `uv run pytest` → `517 passed`.
5. In `agent/`, run `uv run firewall-calibrate ../docs/results/flagging-main-calibration.json --epsilon 0.04` → `q_hat : 0.81`; add `--score injection_driven --epsilon 0.01` → `q_hat : 0.81`.
6. With a Jev key ([step 4](../../README.md#with-a-jev-key)) and the classifier ([step 9](../reference.md#evaluation-and-benchmarks)): in `agent/`, run `uv run firewall-eval --scenario-set external --preset jev_fusion_conformal --output ../data/eval/external.json` → `scenarios: 200 (benign=100, attack=100)` and a `jev_fusion_conformal:` row. The committed run printed `false_holds=16 (16.0%) missed_attacks=0 (0.0%)`; Jev scores vary between runs. This spends 200 Jev calls.
6. With a Jev key ([step 4](../try-it.md#with-a-jev-key)) and the classifier ([step 9](../reference.md#evaluation-and-benchmarks)): in `agent/`, run `uv run firewall-eval --scenario-set external --preset jev_fusion_conformal --output ../data/eval/external.json` → `scenarios: 200 (benign=100, attack=100)` and a `jev_fusion_conformal:` row. The committed run printed `false_holds=16 (16.0%) missed_attacks=0 (0.0%)`; Jev scores vary between runs. This spends 200 Jev calls.

### Baseline: before the provenance rules and Jev v2

Expand Down
46 changes: 46 additions & 0 deletions docs/third-party.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
# Third-party libraries, tools and datasets

Moved from the README on 2026-11-13. The README keeps a names-only table: [README Built with](../README.md#built-with).

## Decision and platform

- [TypeSafe Jev](https://typesafe.ai): calibrated typed decisions, raw HTTPS; the decision brain
- [NEAR AI Cloud](https://cloud.near.ai): LLM for agent tool calls and decision fallback; optional Jev route (`JEV_PROVIDER=nearai`)
- [OutLayer](https://app.outlayer.ai): TEE execution of the checker (wasm32-wasip2)
- [protectai/deberta-v3-base-prompt-injection-v2](https://huggingface.co/protectai/deberta-v3-base-prompt-injection-v2): injection classifier, default; Apache-2.0 (verify on model card)
- [meta-llama/Llama-Prompt-Guard-2-86M](https://huggingface.co/meta-llama/Llama-Prompt-Guard-2-86M): injection classifier, when HF access is granted; Llama 4 Community License (verify on model card)
- [transformers](https://github.com/huggingface/transformers) and [PyTorch](https://pytorch.org) (optional, only for `firewall-classifier`; not in `pyproject.toml`, installed with `uv run --with`)

## Datasets (used by the benchmarks and the external set; full attribution in [agent/scenarios/external/ATTRIBUTION.md](../agent/scenarios/external/ATTRIBUTION.md))

- [deepset/prompt-injections](https://huggingface.co/datasets/deepset/prompt-injections): Apache-2.0; read by `bench_classifier.py` only, not committed
- [3nesdeniz/agentic-prompt-injection-boundary-pairs](https://huggingface.co/datasets/3nesdeniz/agentic-prompt-injection-boundary-pairs): CC-BY-4.0; rewritten rows in `agent/scenarios/external/`; also read by the classifier benchmark
- [nvidia/Nemotron-RL-Agentic-Indirect-Prompt-Injection-v1](https://huggingface.co/datasets/nvidia/Nemotron-RL-Agentic-Indirect-Prompt-Injection-v1): CC-BY-4.0; rewritten rows in `agent/scenarios/external/`; also read by the classifier benchmark
- [microsoft/llmail-inject-challenge](https://huggingface.co/datasets/microsoft/llmail-inject-challenge): MIT; benign emails in `agent/scenarios/external/`; also read by the classifier benchmark
- [InjecAgent](https://github.com/uiuc-kang-lab/InjecAgent): MIT; rewritten rows in `agent/scenarios/external/`
- [AgentDojo](https://github.com/ethz-spylab/agentdojo): MIT; rewritten rows in `agent/scenarios/external/`
- [Kaggle homoglyph domain attacks](https://www.kaggle.com/datasets/akshat4112/homoglyph-domain-attacks): CC0; benchmark input only (`bench_lookalike.py`), not committed
- [Kaggle adversarial BEC email dataset](https://www.kaggle.com/datasets/yoadjei/adversarial-bec-email-dataset): CC BY-SA 4.0; read by `build_bec_examples.py`, not committed
- [Kaggle PaySim](https://www.kaggle.com/datasets/ealaxi/paysim1): CC BY-SA 4.0; read by `build_paysim_set.py`, not committed
- [Jim10/attested-payment-firewall-scenarios-eclipse](https://huggingface.co/datasets/Jim10/attested-payment-firewall-scenarios-eclipse): this project's own scenarios (main, redteam, external splits) as exported by `agent/tools/export_hf_dataset.py`; published copy

## Rust (contracts and checker)

- [near-sdk-rs](https://github.com/near/near-sdk-rs) and near-contract-standards (`=5.29.1`): contract SDK, NEP-141
- [cargo-near](https://github.com/near/cargo-near): contract build
- serde, serde_json, sha2, hex: serialization and hashing (checker; hex also in the Guardian)
- ureq (native checker) and wasi-http-client (wasm32-wasip2 checker): HTTP client

## Python (agent, relayer, eval)

- [uv](https://docs.astral.sh/uv/) and hatchling: environment and build
- [py-near](https://github.com/pvolnov/py-near): NEAR Python client (testnet extra)
- [openai](https://github.com/openai/openai-python): NEAR AI Cloud client
- httpx, jsonschema, python-dotenv, pytest
- [matplotlib](https://matplotlib.org): README figures (`agent/tools/plot_results.py`; not in `pyproject.toml`, installed with `uv run --with`)

## Frontend (`site/`)

- React, React DOM, Vite, `@vitejs/plugin-react` and vite-plugin-singlefile: shared landing page and read-only receipts frontend
- three, three-stdlib, `@react-three/fiber`, `@react-three/drei`, GSAP, Lenis and lucide-react: shared frontend stack ([site/README.md](../site/README.md))
- Google Fonts (Geist, Geist Mono), NEARBlocks testnet explorer, FastNEAR public testnet RPC: frontend fonts and public data links
40 changes: 40 additions & 0 deletions docs/try-it.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
# Try it

Moved from the README on 2026-11-13. Short form: [README Quick start](../README.md#quick-start). Steps 7–12 (evaluation, calibration, classifier, external set, benchmarks, dataset export) and configuration switches: [reference.md](reference.md#evaluation-and-benchmarks).

## Quick start (no keys)

Three commands, no keys and no funds:

```
cargo build -p checker --release
cd agent && uv sync
uv run firewall-demo --scenario benign-hosting --reset-spend
```

Expected: `verdict: hold reasons=check_unavailable…` and `action: held`. With no Jev key the checker fails closed and never allows a payment. With a Jev key the same command allows the payment; see step 4.

Numbered steps, each “do X → see Y”, run from the repo root unless a step says otherwise; steps 1–3 and 5 need no key, steps 4 and 6 need a TypeSafe Jev key.

## Offline, no keys

1. `cargo build -p checker --release` → `target/release/checker` (`.exe` on Windows) exists.
2. `cd agent && uv sync` → the Python environment is ready. Optional: `uv run pytest` runs the Python tests.
3. Offline dry-run, no keys. In `agent/`, run `uv run firewall-demo --scenario benign-hosting --reset-spend` → the lines below (`--reset-spend` clears the local dry-run daily spend, which is kept across runs like the contract):
```
scenario: benign-hosting (benign)
proposed: pay 18000000 units (18 tUSDC) to hostco.demo.testnet memo="INV-2026-11"
verdict: hold reasons=check_unavailable… record=<64 hex>
action: held
held until <UTC time>
record_file: data/decisions/dry-run/<request id>.json
```
This is the fail-closed design, not a bug: with no `TYPESAFE_API_KEY` Jev cannot be called, and the checker never allows a payment without a Jev verdict. It returns `hold` with reason `check_unavailable` (more reasons may follow it) and the payment is not paid. The dry-run never waits for or releases the hold. The demo substitutes `OWNER_ACCOUNT_ID` (from `.env`) for the scenario namespace when it is set, otherwise `demo.testnet`, so `hostco.demo.testnet` above becomes `hostco.<OWNER_ACCOUNT_ID>`.

## With a Jev key

4. Dry-run with Jev. Put your key in `.env` at the repo root (`cp .env.example .env`, then set `TYPESAFE_API_KEY=`). Optional: `cd agent && uv run firewall-jev-ping` checks the key. Then in `agent/`:
- `uv run firewall-demo --scenario benign-hosting --reset-spend` → `verdict: allow …` and `action: would_pay`.
- `uv run firewall-demo --scenario attack-morse-lookalike --reset-spend` → `verdict: hold …` (`action: held`) or `verdict: block …` (`action: refused`); the receiver `hostc0.demo.testnet` is a lookalike of `hostco.demo.testnet`.
5. Open the file named on the `record_file:` line (e.g. `data/decisions/dry-run/2.json`) → the full decision record; `sha256sum <that file>` equals the `record=` value. The path is relative to the repository root, even though the command ran in `agent/`. Each dry-run gets the next request id, so earlier records are kept; the checker refuses to overwrite a record with different content.
6. Nightly end-to-end check, offline. In `agent/`, run `uv run firewall-nightly-e2e --mode dry-run -y` → one benign and one attack payment, then `Overall Status: SUCCESS` and exit code 0. It needs the Jev key from step 4. Without a key it prints `[FAIL] benign-hosting (benign)` with `Verdict: outcome=hold reasons=check_unavailable`, the attack payment still `[PASS]`, then `Overall Status: FAILURE` and exit code 1. Its scenario namespace follows `OWNER_ACCOUNT_ID` as in step 3. Details: [docs/build/nightly-e2e.md](build/nightly-e2e.md).
25 changes: 19 additions & 6 deletions site/src/App.jsx
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import { lazy, Suspense, useEffect, useRef, useState } from "react";
import { Component, lazy, Suspense, useCallback, useEffect, useRef, useState } from "react";
import {
ArrowRight,
Box,
Expand Down Expand Up @@ -40,6 +40,15 @@ import { startMotion } from "./motion.js";
// three.js loads after first paint, in its own chunk; the prerender shows the placeholder.
const Eclipse = lazy(() => import("./components/Eclipse.jsx"));

// A missing WebGL context (graphics acceleration off, blocklisted GPU) or a failed chunk load
// must cost only the globe, not unmount the whole page.
class SceneBoundary extends Component {
state = { failed: false };
static getDerivedStateFromError() { return { failed: true }; }
componentDidCatch(error) { console.warn("3D scene disabled:", error); }
render() { return this.state.failed ? this.props.fallback : this.props.children; }
}

const receiptsReader = createReader({ guardian: GUARDIAN_ID, rpc: NEAR_RPC_URL });

const NAV = [
Expand Down Expand Up @@ -226,9 +235,11 @@ export default function App() {
const refs = {
heroGlow: useRef({ v: 1 }),
ctaGlow: useRef({ v: 0.4 }),
globeProgress: useRef(0),
morseProgress: useRef(0),
};
// The protocol scene reports its phase; the matching step card lights up (CSS on data-step).
const stepsRef = useRef(null);
const onStep = useCallback((i) => { if (stepsRef.current) stepsRef.current.dataset.step = i; }, []);
const [client, setClient] = useState(false);
const [menuOpen, setMenuOpen] = useState(false);
useEffect(() => {
Expand Down Expand Up @@ -351,11 +362,13 @@ export default function App() {
<div className="protocol">
<Frame className="stage3" id="protocol">
{client ? (
<Suspense fallback={<div className="globe globe-wait" aria-hidden="true" />}>
<Eclipse progress={refs.globeProgress} />
</Suspense>
<SceneBoundary fallback={<div className="globe globe-wait" aria-hidden="true" />}>
<Suspense fallback={<div className="globe globe-wait" aria-hidden="true" />}>
<Eclipse onStep={onStep} />
</Suspense>
</SceneBoundary>
) : <div className="globe globe-wait" aria-hidden="true" />}
<ol className="psteps psteps-overlay">
<ol className="psteps psteps-overlay" ref={stepsRef}>
{STEPS.map(({ Icon, title, text }, i) => (
<li key={title}>
<span className="pstep-n">{String(i + 1).padStart(2, "0")}</span>
Expand Down
3 changes: 2 additions & 1 deletion site/src/components/Console.jsx
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,8 @@ import { DAILY_CAP } from "../config";
import { prefersReducedMotion } from "../motion.js";

// Static sample rows: a preview of the receipts view, not live data (the live panel is in #receipts).
const ROWS = [
// The protocol scene (Eclipse.jsx) acts out the same four payments.
export const ROWS = [
["block", "hostc0.testnet", 180, "injection + lookalike payee"],
["allow", "hostco.testnet", 18, "known payee · usual amount"],
["hold", "supplier.testnet", 90, "above usual range · 1 h escrow"],
Expand Down
Loading
Loading