diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml new file mode 100644 index 0000000..c68eadd --- /dev/null +++ b/.github/workflows/ci.yml @@ -0,0 +1,23 @@ +name: ci + +on: + push: + branches: [main] + pull_request: + +jobs: + sim-tests: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-python@v5 + with: + python-version: "3.12" + cache: pip + - name: Install CPU torch and test dependencies + run: | + pip install torch --index-url https://download.pytorch.org/whl/cpu + pip install numpy matplotlib pytest + - name: Sim test suite + working-directory: sim + run: python -m pytest tests -q diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..e3000b6 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,53 @@ +# Contributing + +This repository holds the Phase 0 proof: the simulation and the documents. The network +substrate (Anchor programs, trainer, verifier daemon) lives in wienerlabs/leviathan-net; the +site lives in wienerlabs/leviathan-web. + +## Setup + +``` +cd sim +uv venv && uv pip install torch numpy matplotlib pytest # CPU torch is fine +``` + +The full sim needs the tinyshakespeare corpus at `sim/data/shakespeare.txt` (gitignored): + +``` +curl -sSL -o data/shakespeare.txt \ + https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt +``` + +## Tests + +``` +cd sim && python -m pytest tests -q +``` + +The suite runs on CPU with a synthetic corpus and no downloads; CI runs exactly this. Every +qualitative claim the README makes should have a test that would fail if the sim stopped +backing it — if you change aggregation, attacks, the verifier or the economy, extend the tests +with the claim your change adds. + +## Experiments + +``` +python -m leviathan_sim.run --rounds 30 # the seven headline scenarios +python -m leviathan_sim.run --verify # tolerance-band replay experiment +python -m leviathan_sim.run --sparse # dense vs SparseLoCo transport +python -m leviathan_sim.run --band-sweep # within-band adversary budget +``` + +Add `--smoke` for a 3-round sanity pass. Outputs land in `sim/out/` (gitignored); curated +results are promoted to `docs/assets/` by hand, together with the numbers quoted in README and +docs. If you regenerate an asset, update every number that quotes it in the same commit. + +## Docs discipline + +The honest-claims policy (docs/DECISIONS.md, D6) binds prose, not just marketing: no claim in +README, the whitepaper or ARCHITECTURE.md should outrun what the sim or the fork actually +demonstrates. If you find one that does, that is a bug — file it or fix it via docs/GAPS.md. + +## Security findings + +See SECURITY.md — exploitable findings go through private reporting, not public issues. diff --git a/README.md b/README.md index 1103906..87cd296 100644 --- a/README.md +++ b/README.md @@ -109,6 +109,10 @@ The break-even bond law held on a real transformer run: at audit probability p = ![security economics](docs/assets/security_economics.png) +The published tolerance band is, by construction, the adversary's undetectable budget — so the sim prices it. A 5/16 coalition biasing against the honest mean at 0.9x the band passes every replay audit (measured distance exactly 0.9x band, zero fraud verdicts) and is fully accepted by the aggregator; what it buys is the damage curve below. At the operating band of 0.05 the whole coalition purchases +0.019 final loss against the 2.173 honest reference, about 0.9%, and the cost grows roughly linearly with band width. That number is why the band is a published dashboard metric and why calibrating it tightly per hardware class is a Genesis-gating task. + +![band sweep](docs/assets/band_sweep.png) + ## Phase 1 progress The network substrate is a private fork of PsycheFoundation/nousnet (Apache-2.0) at wienerlabs/leviathan-net, carrying the layer upstream designed but left unimplemented: bonds, replay audits, slashing. @@ -137,6 +141,7 @@ Phase 0: the proof that the security economics survive contact with real transfo | `docs/DECISIONS.md` | prior-art sweep and the six locked decisions, sources inline | | `docs/ARCHITECTURE.md` | on-chain programs, daemons, round lifecycle, determinism strategy | | `docs/PRD.md`, `docs/TASKS.md` | phase acceptance criteria and the task ladder | +| `docs/GAPS.md` | self-audit: where the docs promise more than the code delivers, ordered by risk | Reproduce the sim: @@ -150,4 +155,4 @@ PYTORCH_ENABLE_MPS_FALLBACK=1 uv run python -m leviathan_sim.run --rounds 30 Built on the shoulders of, and differentiated from: Psyche/nousnet (Apache-2.0 fork substrate: Solana coordinator, iroh P2P, Rust DisTrO), SparseLoCo (MIT compression recipe), TOPLOC (MIT inference verification), OVIG (the audit loop's academic validation), Condorcet (wienerlabs: the aggregation and bond-economics research core), DanteGPU (supply), Wienerpad (futarchy governance), zk-lokomotive (Arweave rail). -Private under wienerlabs while Phase 1 lands. +Private under wienerlabs while the Genesis Run is prepared; going public is gated on the Phase 2 repo-hygiene checklist in docs/TASKS.md (the LICENSE decision is the open item — CI, tests, SECURITY.md and CONTRIBUTING.md are in place). diff --git a/SECURITY.md b/SECURITY.md new file mode 100644 index 0000000..7d200ee --- /dev/null +++ b/SECURITY.md @@ -0,0 +1,34 @@ +# Security policy + +Leviathan's product is a trust machine; a vulnerability report is a contribution to the core +product, not an embarrassment. The honest-claims policy (docs/DECISIONS.md, D6) applies here +first: we would rather publish a weakness than pretend it does not exist. + +## Reporting a vulnerability + +Use GitHub's private vulnerability reporting on this repository (Security tab → "Report a +vulnerability"). Reports go to the wienerlabs maintainers privately; do not open a public issue +for anything exploitable. + +In scope, in rough order of how much we care: + +1. Economic-security breaks: any way to cheat the bond/audit/slash loop with positive expected + value — grinding the round seed, dodging the audit lottery, submitting work that survives + replay without doing it, draining or freezing bonds in the treasurer. +2. Aggregation breaks: attacks that get more damage past centered-clip + excision than the + published within-band budget implies, at coalition sizes below the excision threshold. +3. Sim errors that would make a published number wrong (loss tables, catch times, bond curve). +4. Classic vulnerabilities in the Anchor programs or daemons (leviathan-net) once public. + +Out of scope: volume/DoS against public devnet endpoints, findings that require the run +authority's own keys, and anything already listed as a known limitation in docs/GAPS.md or the +whitepaper's honest-limitations section — though sharpening a known limitation into a concrete +exploit is very much in scope. + +## What to expect + +Acknowledgement within 72 hours, a severity assessment within a week, credit in the fix's +release notes unless you prefer otherwise. There is no paid bounty yet; the red-team bounty +program is a Phase 2 deliverable (docs/TASKS.md) and this policy will be updated when it ships. + +Good-faith research within this policy will not be met with legal action. diff --git a/docs/GAPS.md b/docs/GAPS.md new file mode 100644 index 0000000..5fff052 --- /dev/null +++ b/docs/GAPS.md @@ -0,0 +1,105 @@ +# Gap analysis + +Snapshot 2026-07-20, after Phase 1 closed on devnet. A self-audit of what the docs promise +against what the code and task ladder actually deliver. Ordered by how much each gap can hurt. +Items marked (T) were folded into docs/TASKS.md. + +Status update, same day: the gaps closable in this repo were closed — the within-band damage +scenario (`--band-sweep`: +0.019 loss at the 0.05 operating band for a 5/16 coalition, ~0.9%, +roughly linear in band width, with explicit replay evidence that audits pass by construction), the +zero-fraud verifier burn projection and Genesis operating point in economy.py, a 28-test sim +suite with CI, SECURITY.md, CONTRIBUTING.md, and the doc-drift fixes (PRD as-built note, README +privacy line). Still open: the LICENSE decision (owner call), and everything that lives in +leviathan-net or requires hardware — band calibration across GPU classes, two-GPU replay, the +production verifier daemon, multi-epoch re-join, randomness hardening, the dispute committee. + +## 1. Security-model gaps + +### Tolerance band is uncalibrated +The band is the core security parameter — its width is, by our own definition, the adversary's +safe cheating margin — and it has only ever been exercised at 0.05 on one machine against a +synthetic 1% drift. The cross-hardware calibration harness (4090, 3090, A100, H100) is a Phase 2 +task that nothing else may ship before: a band set too tight slashes honest nodes and violates +the PRD's "zero honest-node slashes" metric; too loose and it hands stealth coalitions a budget. +(T: already in Phase 2.) + +### No within-band damage scenario in the sim +ARCHITECTURE.md claims "sim/ tracks how much loss damage fits inside a given band so the +parameter is chosen with eyes open." No scenario in run.py does this: the seven scenarios cover +sign-flip, ALIE and non-IID FPR, but none measures the maximum training bias an adversary can +inflict while staying inside the published band. The whitepaper's honest-limitations section +admits the weakness; the sim does not yet quantify it. Docs and code disagree. (T: Phase 2.) + +### Dispute resolution is a single authority +`slash_client` is main_authority gated. The bonded multi-verifier vote with reporter bounty is +deferred to Phase 3, which means the entire Genesis Run — 50+ external nodes with real bonded +value at (testnet) stake — runs with a centralized slash decision. That may be acceptable, but +it must be published as a limitation on the Genesis dashboard, with the scope and the timeline +for decentralizing it. (T: Phase 2 disclosure, Phase 3 mechanism.) + +### Weak round randomness is not on the task ladder +`get_random_seed(clock)` = sha256(unix_timestamp, slot) is grindable by a leader. CODEMAP.md +acknowledges this and says hardening "goes on the mainnet checklist" — but TASKS.md Phase 3 has +no such entry. A checklist item that is not on the checklist is a lost item. (T: Phase 3.) + +### Two-GPU replay reproduction was skipped, not done +PRD Phase 1 acceptance requires the replay verifier to reproduce a sampled contribution within +the band on two different GPUs. It was folded into 1.9's swarm bring-up and then the swarm ran +on a single MPS machine. Phase 1 is declared complete with this criterion unmet. Carry it +forward explicitly rather than letting the fold hide it. (T: Phase 2.) + +### No production verifier daemon +The decision core exists twice — Python sim (verifier.py) and the 1.8 replay experiment — but +the daemon that subscribes as Committee::Verifier, replays real gradients and calls run_slash +does not. It shares the libtorch prerequisite that 1.9 already solved, so the path is open. +(T: Phase 2.) + +### Multi-epoch re-join resilience +Deferred out of 1.9. A 7-day churn run cannot pass without it. (T: already implied by Phase 2 +rehearsal; now explicit.) + +## 2. Economics gaps + +### Verifier income has no zero-fraud sustainability story +The whitepaper says slashed stake pays the verifiers hunting it. In the equilibrium we actually +want — no fraud — verifier income is purely the audit_fee drawn from the treasury. Who funds +sustained audit pressure, at what burn rate, and for how long, is unmodeled. The economy sim has +the ledger; add the projection. (T: Phase 2.) + +### Genesis bond parameters are undefined +Real-bond calibration is Phase 3, but Phase 2 Genesis Run needs concrete (testnet) bond values, +and the choice communicates the economics to the first external cohort. Decide and publish them +with the same (1-p)/p discipline. (T: Phase 2.) + +## 3. Repository hygiene + +- **No LICENSE.** A project whose thesis is "the model belongs to the network" currently + reserves all rights by default. The fork substrate is Apache-2.0; this repo needs an explicit + choice before anything is made public. Owner decision, not picked unilaterally here. +- **No tests, no CI.** The README's headline numbers (2.175 honest loss, 9.8-round mean catch + time) are reproducible only by hand. A CI job that runs a short-horizon sim and asserts the + qualitative outcomes (clip neutralizes sign-flip, audit catches ALIE, zero honest FP) would + keep the claims honest as the sim evolves. (T: Phase 2.) +- **No SECURITY.md.** The red-team bounty is a Phase 2 design task, but a responsible-disclosure + channel should exist from the first day the repo is public. (T: Phase 2.) +- **No CONTRIBUTING.md.** Low priority while private; required before Genesis opens the doors. + +## 4. Doc drift + +- README still says "Private under wienerlabs while Phase 1 lands" — Phase 1 has landed; the + sentence and the publicity decision need refreshing. +- PRD Phase 1 acceptance said "4-process local swarm trains the 826k-parameter sim model through + the real chain path"; what actually ran was a single-client Nano-Llama epoch on devnet. The + substitution is defensible (live chain beats local processes) but the PRD should record it + instead of silently diverging. +- CODEMAP inherits the upstream `comptue_hash` misspelling; cheap to fix in the fork, and the + note in CODEMAP should flip from "grep accordingly" to "fixed at " when it is. +- Devnet program keypairs live in the private repo history. CODEMAP already rules them + devnet-only; the mainnet key regeneration must be a task, not a footnote. (T: Phase 3.) + +## 5. What is explicitly not a gap + +Scope discipline the docs already get right, listed so nobody re-litigates it: no on-chain +tensors, no Windows nodes, no generic compute marketplace, no frontier-quality claims, no +cryptographic-proof claims. The honest-claims policy (D6) is the standard this file holds the +rest of the project to. diff --git a/docs/PRD.md b/docs/PRD.md index 91457ad..86f9715 100644 --- a/docs/PRD.md +++ b/docs/PRD.md @@ -23,6 +23,8 @@ - End-to-end conviction demo: a node submitting fabricated deltas is audited, disputed, slashed and ejected, recorded in one continuous capture. - All state transitions covered by anchor tests; replay verifier reproduces a sampled contribution within the tolerance band on two different GPUs. +As-built note, 2026-07-20: the swarm criterion was met by substitution — a single-client Nano-Llama epoch on live devnet (17 rounds, DisTrO over iroh, real Join/Witness/Tick transactions) instead of a 4-process local swarm on the 826k sim model. Live chain beats local processes, so the substitution strengthens the criterion it replaced. The two-GPU replay reproduction was NOT met: it was folded into the swarm bring-up, which ran on one machine. It is carried forward as an explicit Phase 2 task in docs/TASKS.md. + ## Phase 2 acceptance (Genesis Run) - 50+ external nodes sustain a 350M to 1B run for 7+ days with churn. diff --git a/docs/TASKS.md b/docs/TASKS.md index 75635cf..c610d38 100644 --- a/docs/TASKS.md +++ b/docs/TASKS.md @@ -29,15 +29,28 @@ Exit criteria: recorded end-to-end conviction demo, anchor test suite green, two ## Phase 2, Genesis Run - [ ] Tolerance band calibration harness across hardware classes (4090, 3090, A100, H100) +- [ ] Two-GPU tolerance-band replay reproduction (Phase 1 exit criterion carried forward; folded into 1.9 but never run on two distinct GPUs) +- [x] Sim: within-band adversary budget scenario — `--band-sweep` runs a 5/16 coalition that biases against the honest mean at 0.9x the published band; explicit replay evidence shows the audit passes by construction (distance = 0.9 x band, fraud=false) at every band width, so aggregation is the only layer that pushes back. Measured over 30 rounds vs the 2.173 honest reference: +0.012 loss at band 0.02, +0.019 at 0.05 (the operating point, ~0.9%), +0.032 at 0.1, +0.063 at 0.2 — roughly linear in band width, coalition accepted 100% throughout (docs/assets/band_sweep.png) +- [ ] Production verifier daemon: subscribe as Committee::Verifier, replay real gradients, call run_slash (decision core done in 1.8; shares the 1.9 libtorch toolchain) +- [ ] Multi-epoch re-join resilience (deferred out of 1.9; prerequisite for any churn run) +- [x] Verifier economics: economy.py gains the zero-fraud projection (audit fee = 1.1x per-contribution H100 cost; treasury burn ~9.2% of rewards at p=0.1, preset-independent) and genesis_parameters() publishing the Genesis operating point (1B preset, p=0.1, band 0.05, bond = 9 rounds of reward); both emitted into results.json +- [ ] Publish the centralized-dispute limitation on the Genesis dashboard (slash_client is main_authority gated until the Phase 3 committee lands) - [ ] One-line installer + wallet flow + bond funding UX - [ ] Public site: live loss curve, node map, leaderboard, audit/slash feed - [ ] Relay infrastructure for non-hole-punchable nodes +- [x] SECURITY.md responsible-disclosure channel (GitHub private vulnerability reporting; economic-security breaks explicitly in scope, no paid bounty until the program below ships) - [ ] Red-team bounty program design +- [x] CI + tests: .github/workflows/ci.yml runs the 28-test sim suite on CPU torch (aggregation excises the outlier, within-band stays inside the band, replay catches sign-flip/lazy and passes honest, economy math); CONTRIBUTING.md documents setup, experiments and the docs discipline +- [ ] LICENSE decision (owner call; blocks going public) - [ ] 350M run rehearsal with internal nodes, then 1B public run ## Phase 3, mainnet - [ ] Legal review of token design +- [ ] Independent security audit of the Anchor programs (coordinator, treasurer, authorizer) before real bonds +- [ ] Round randomness hardening: VRF or recent-blockhash commitment replacing sha256(unix_timestamp, slot) (grindable, flagged in CODEMAP.md) +- [ ] Bonded multi-verifier dispute committee with reporter bounty and treasury remainder, replacing the main_authority resolver (evolution locked in 1.6) +- [ ] Regenerate program keypairs for mainnet (devnet keypairs live in private repo history, devnet-only per CODEMAP.md) - [ ] TGE via Wiener Launchpad rails; distributor program reuse for airdrop and vesting - [ ] Real-bond parameters from sim calibration; treasury and endowment wiring - [ ] Inference v0: vLLM workers + TOPLOC validation + settlement program diff --git a/docs/WHITEPAPER.md b/docs/WHITEPAPER.md index 174022c..7fa746f 100644 --- a/docs/WHITEPAPER.md +++ b/docs/WHITEPAPER.md @@ -67,7 +67,7 @@ Flywheel: inference fees (vLLM workers verified by TOPLOC activation commitments ## 8. Honest limitations -- Economic security has a floor: coalitions below the excision threshold that also stay inside the tolerance band can bias training slowly. The published band is the adversary's budget; we choose it with the sim, and we would rather publish that number than pretend it is zero. +- Economic security has a floor: coalitions below the excision threshold that also stay inside the tolerance band can bias training slowly. The published band is the adversary's budget; we choose it with the sim, and we would rather publish that number than pretend it is zero. The number, measured: a 5/16 within-band coalition passes every replay audit by construction and buys +0.019 final loss (~0.9% of the 2.173 honest reference) at the 0.05 operating band, scaling roughly linearly to +0.063 at band 0.2. This is what tight per-hardware band calibration is worth. - A 1B volunteer-trained model is not a frontier model. The claim is a proven path, not an arrived destination: each run scales the trust machine that a frontier-scale run requires. - Solana congestion or program bugs stall rounds; the coordinator inherits Psyche's pause and resume machinery, and the fork keeps its centralized backend for disaster recovery. - Token design must survive regulation; utility-first, no yield promises, legal review before TGE. diff --git a/docs/assets/band_sweep.png b/docs/assets/band_sweep.png new file mode 100644 index 0000000..bc37245 Binary files /dev/null and b/docs/assets/band_sweep.png differ diff --git a/docs/assets/band_sweep_results.json b/docs/assets/band_sweep_results.json new file mode 100644 index 0000000..4d8b303 --- /dev/null +++ b/docs/assets/band_sweep_results.json @@ -0,0 +1,212 @@ +{ + "config": { + "rounds": 30, + "workers": 16, + "n_malicious": 5, + "device": "cpu", + "model_parameters": 826368, + "bands": [ + 0.02, + 0.05, + 0.1, + 0.2 + ] + }, + "honest_reference": { + "final_val_loss": 2.173410952091217, + "val_losses": [ + 2.770378962159157, + 2.60888409614563, + 2.543832153081894, + 2.508126735687256, + 2.486658349633217, + 2.474105015397072, + 2.4595586955547333, + 2.450014144182205, + 2.442861393094063, + 2.4389298856258392, + 2.43013197183609, + 2.421166941523552, + 2.411952391266823, + 2.4090156108140945, + 2.4044787734746933, + 2.3957075625658035, + 2.3918637335300446, + 2.381677985191345, + 2.376203015446663, + 2.371914267539978, + 2.3553656339645386, + 2.343187779188156, + 2.3320121318101883, + 2.318408355116844, + 2.313457652926445, + 2.28567735850811, + 2.2693462818861008, + 2.2291798889636993, + 2.2051504105329514, + 2.173410952091217 + ] + }, + "sweep": [ + { + "band": 0.02, + "final_val_loss": 2.1857606917619705, + "damage_vs_honest": 0.012349739670753479, + "malicious_selection_rate": 1.0, + "replay_distance": 0.018000036478042603, + "replay_fraud": false, + "val_losses": [ + 2.7701356112957, + 2.6086870282888412, + 2.5437359660863876, + 2.5076621919870377, + 2.4862434715032578, + 2.4740501940250397, + 2.460038959980011, + 2.450979769229889, + 2.4425385743379593, + 2.44069904088974, + 2.429084986448288, + 2.4198197722434998, + 2.412846550345421, + 2.408899664878845, + 2.4049845784902573, + 2.3951989114284515, + 2.390958398580551, + 2.381100833415985, + 2.375213637948036, + 2.37044657766819, + 2.3560407906770706, + 2.346657931804657, + 2.3309069871902466, + 2.324841544032097, + 2.306785002350807, + 2.2894617468118668, + 2.286434456706047, + 2.2406703531742096, + 2.2160698622465134, + 2.1857606917619705 + ] + }, + { + "band": 0.05, + "final_val_loss": 2.192902535200119, + "damage_vs_honest": 0.019491583108901978, + "malicious_selection_rate": 1.0, + "replay_distance": 0.045000094920396805, + "replay_fraud": false, + "val_losses": [ + 2.769803687930107, + 2.6083097606897354, + 2.543393239378929, + 2.5068593621253967, + 2.4855589121580124, + 2.4738713800907135, + 2.4603098779916763, + 2.451279506087303, + 2.4427553862333298, + 2.440789893269539, + 2.4283525943756104, + 2.4201632887125015, + 2.4137205481529236, + 2.4100505858659744, + 2.4057970196008682, + 2.3954578042030334, + 2.3913018107414246, + 2.3814733922481537, + 2.375205308198929, + 2.369316577911377, + 2.3576837927103043, + 2.3469138890504837, + 2.3355731517076492, + 2.3230367302894592, + 2.313744932413101, + 2.292399510741234, + 2.281036853790283, + 2.242164269089699, + 2.2211836725473404, + 2.192902535200119 + ] + }, + { + "band": 0.1, + "final_val_loss": 2.20535109937191, + "damage_vs_honest": 0.031940147280693054, + "malicious_selection_rate": 1.0, + "replay_distance": 0.09000018984079361, + "replay_fraud": false, + "val_losses": [ + 2.7693415135145187, + 2.60792475938797, + 2.542679116129875, + 2.505945608019829, + 2.484595239162445, + 2.4737804532051086, + 2.460836261510849, + 2.451386868953705, + 2.442452296614647, + 2.440273940563202, + 2.4281808733940125, + 2.421149030327797, + 2.415631502866745, + 2.4107213020324707, + 2.4053375869989395, + 2.3949068039655685, + 2.390969842672348, + 2.3814787417650223, + 2.37664195895195, + 2.369709238409996, + 2.3622162342071533, + 2.3499815315008163, + 2.340071141719818, + 2.327909767627716, + 2.314935952425003, + 2.3002878576517105, + 2.2984621673822403, + 2.255413994193077, + 2.234349101781845, + 2.20535109937191 + ] + }, + { + "band": 0.2, + "final_val_loss": 2.2362954765558243, + "damage_vs_honest": 0.06288452446460724, + "malicious_selection_rate": 1.0, + "replay_distance": 0.18000037968158722, + "replay_fraud": false, + "val_losses": [ + 2.768751382827759, + 2.6073982417583466, + 2.5410755276679993, + 2.505242884159088, + 2.4836170226335526, + 2.473900690674782, + 2.4607941061258316, + 2.4521148204803467, + 2.442139372229576, + 2.4397433400154114, + 2.4293213188648224, + 2.4214515388011932, + 2.4157699197530746, + 2.412311851978302, + 2.406392380595207, + 2.3969900757074356, + 2.391791358590126, + 2.3835081458091736, + 2.3786582201719284, + 2.3734836876392365, + 2.365206152200699, + 2.3547249138355255, + 2.35318922996521, + 2.340775266289711, + 2.333874449133873, + 2.3210405111312866, + 2.311697021126747, + 2.277094006538391, + 2.256584584712982, + 2.2362954765558243 + ] + } + ] +} \ No newline at end of file diff --git a/sim/leviathan_sim/attacks.py b/sim/leviathan_sim/attacks.py index eac5006..72cb582 100644 --- a/sim/leviathan_sim/attacks.py +++ b/sim/leviathan_sim/attacks.py @@ -4,7 +4,7 @@ from leviathan_sim.swarm import LocalUpdate -VALID_ATTACKS = frozenset({"none", "sign_flip", "gaussian", "lazy", "alie"}) +VALID_ATTACKS = frozenset({"none", "sign_flip", "gaussian", "lazy", "alie", "within_band"}) @dataclass(frozen=True) @@ -14,6 +14,8 @@ class InjectionConfig: sign_flip_scale: float = 5.0 gaussian_std: float = 0.02 alie_z: float = 1.5 + band: float = 0.05 + band_margin: float = 0.9 @dataclass(frozen=True) @@ -37,6 +39,24 @@ def _attack(self, updates: dict[int, LocalUpdate]) -> dict[int, LocalUpdate]: if config.attack == "none" or not self.malicious_ids: return updates out = dict(updates) + if config.attack == "within_band": + # The adversary's whole budget is the published tolerance band: each + # malicious worker submits its honest delta plus a coordinated bias + # scaled so the relative replay distance stays at band_margin * band. + # Replay audits pass by construction; only aggregation limits this. + honest = torch.stack( + [u.delta for wid, u in updates.items() if wid not in self.malicious_ids] + ) + direction = honest.mean(dim=0) + direction = direction / torch.linalg.vector_norm(direction).clamp(min=1e-12) + budget = config.band * config.band_margin + for wid in self.malicious_ids: + if wid not in out: + continue + update = out[wid] + bias = -direction * budget * torch.linalg.vector_norm(update.delta) + out[wid] = replace(update, delta=update.delta + bias) + return out if config.attack == "alie": honest = torch.stack( [u.delta for wid, u in updates.items() if wid not in self.malicious_ids] diff --git a/sim/leviathan_sim/economy.py b/sim/leviathan_sim/economy.py index 2e11343..16feee4 100644 --- a/sim/leviathan_sim/economy.py +++ b/sim/leviathan_sim/economy.py @@ -99,3 +99,53 @@ def calibration_table(audit_probabilities: list[float]) -> list[dict]: } ) return rows + + +def audit_burn_projection( + audit_probabilities: list[float], + n_workers: int = 100, + fee_multiplier: float = 1.1, +) -> list[dict]: + """Zero-fraud equilibrium: with nobody to slash, verifier income is pure + treasury burn. A replay audit redoes one contribution's compute, so the fee + floor is the per-contribution H100 cost times a verifier margin.""" + rows = [] + for preset in PRESETS: + cost = h100_round_cost_usd(preset) + reward = 1.2 * cost + for p in audit_probabilities: + fee = fee_multiplier * cost + burn = p * n_workers * fee + rewards_paid = n_workers * reward + rows.append( + { + "preset": preset.label, + "audit_probability": p, + "n_workers": n_workers, + "audit_fee_usd": fee, + "expected_audits_per_round": p * n_workers, + "treasury_burn_per_round_usd": burn, + "burn_share_of_rewards": burn / rewards_paid, + } + ) + return rows + + +def genesis_parameters(audit_probability: float = 0.1, band: float = 0.05) -> dict: + """The published operating point for the Genesis Run testnet, derived from + the same (1-p)/p discipline as the calibration table. Values are per worker + per the 1B preset; testnet bonds use these numbers denominated in the + testnet collateral mint.""" + preset = next(p for p in PRESETS if p.params_billion == 1.0) + cost = h100_round_cost_usd(preset) + reward = 1.2 * cost + return { + "preset": preset.label, + "audit_probability": audit_probability, + "tolerance_band": band, + "round_reward_usd": reward, + "bond_usd": break_even_bond(audit_probability, reward), + "bond_rounds_of_reward": (1.0 - audit_probability) / audit_probability, + "expected_rounds_to_catch": 1.0 / audit_probability, + "audit_burn_share_of_rewards": audit_probability * 1.1 / 1.2, + } diff --git a/sim/leviathan_sim/run.py b/sim/leviathan_sim/run.py index 8b9a2f0..2b22617 100644 --- a/sim/leviathan_sim/run.py +++ b/sim/leviathan_sim/run.py @@ -14,7 +14,13 @@ from leviathan_sim.aggregators import CenteredClipAggregator, MeanAggregator from leviathan_sim.attacks import InjectionConfig, Injector -from leviathan_sim.economy import EconomyConfig, StakeLedger, calibration_table +from leviathan_sim.economy import ( + EconomyConfig, + StakeLedger, + audit_burn_projection, + calibration_table, + genesis_parameters, +) from leviathan_sim.model import build_model, parameter_count from leviathan_sim.sparse import chunked_topk_sign from leviathan_sim.swarm import NesterovOuter, SwarmWorker, evaluate, load_corpus @@ -78,6 +84,7 @@ def run_scenario( device: torch.device, sparse_density: float | None = None, sparse_chunk: int = 64, + attack_band: float = 0.05, ) -> dict: model = build_model(corpus.vocab_size, block_size, seed=7, device=device) theta = parameters_to_vector(model.parameters()).detach().clone() @@ -98,7 +105,7 @@ def run_scenario( for wid in worker_ids ] injector = Injector( - InjectionConfig(n_malicious=scenario.n_malicious, attack=scenario.attack), + InjectionConfig(n_malicious=scenario.n_malicious, attack=scenario.attack, band=attack_band), worker_ids, seed=11, ) @@ -299,6 +306,7 @@ def main(): parser.add_argument("--sparse", action="store_true") parser.add_argument("--sparse-density", type=float, default=0.02) parser.add_argument("--verify", action="store_true") + parser.add_argument("--band-sweep", action="store_true") parser.add_argument("--drift", type=float, default=0.01) parser.add_argument("--band", type=float, default=0.05) args = parser.parse_args() @@ -311,6 +319,9 @@ def main(): if args.verify: run_verify_experiment(args, corpus, device, base) return + if args.band_sweep: + run_band_sweep(args, corpus, rounds, device, base, probe) + return if args.sparse: run_sparse_comparison(args, corpus, rounds, device, base, probe) return @@ -352,6 +363,8 @@ def main(): }, "scenarios": results, "economy_calibration": calibration, + "audit_burn_projection": audit_burn_projection([0.02, 0.05, 0.1, 0.2, 0.3]), + "genesis_parameters": genesis_parameters(), } (out_dir / f"{prefix}results.json").write_text(json.dumps(payload, indent=2)) plot_loss_curves(results, out_dir / f"{prefix}loss_curves.png") @@ -437,6 +450,122 @@ def plot_verify(rows, band, drift, out_path: Path): plt.close(fig) +BAND_SWEEP = [0.02, 0.05, 0.1, 0.2] + + +def within_band_replay_evidence(args, corpus, device, band: float) -> dict: + """One round of explicit replay: a within-band submission must pass the + audit at exactly band_margin * band relative distance. This is the proof + that layer 2 is blind to this adversary, not an assumption.""" + n_workers = args.workers + model = build_model(corpus.vocab_size, args.block_size, seed=7, device=device) + theta = parameters_to_vector(model.parameters()).detach().clone() + workers = [ + SwarmWorker( + wid, corpus, n_workers, True, args.inner_steps, 2e-3, + args.batch_size, args.block_size, 100 + wid, device, + ) + for wid in range(n_workers) + ] + updates = {w.wid: w.local_round(theta, model, 0) for w in workers} + injector = Injector( + InjectionConfig(n_malicious=5, attack="within_band", band=band), + list(range(n_workers)), + seed=11, + ) + attacked, report = injector.apply(updates) + target_wid = min(report.malicious_ids) + verdict = replay_and_verify( + workers[target_wid], model, theta, 0, attacked[target_wid].delta, band + ) + return {"wid": target_wid, "distance": verdict.distance, "fraud": verdict.fraud} + + +def run_band_sweep(args, corpus, rounds, device, base, probe): + honest = run_scenario( + Scenario("withinband-honest", "Honest, clip", True, "none", 0, "clip", 0.0), + corpus, rounds, args.workers, args.inner_steps, + args.batch_size, args.block_size, device, + ) + print(f"withinband-honest: final={honest['final_val_loss']:.3f}") + rows = [] + for band in BAND_SWEEP: + scenario = Scenario( + f"withinband-{band}", f"Within-band 5/16, band {band}", True, + "within_band", 5, "clip", 0.0, + ) + result = run_scenario( + scenario, corpus, rounds, args.workers, args.inner_steps, + args.batch_size, args.block_size, device, attack_band=band, + ) + evidence = within_band_replay_evidence(args, corpus, device, band) + row = { + "band": band, + "final_val_loss": result["final_val_loss"], + "damage_vs_honest": result["final_val_loss"] - honest["final_val_loss"], + "malicious_selection_rate": result["malicious_selection_rate"], + "replay_distance": evidence["distance"], + "replay_fraud": evidence["fraud"], + "val_losses": result["val_losses"], + } + rows.append(row) + print( + f"withinband band={band}: final={row['final_val_loss']:.3f} " + f"damage={row['damage_vs_honest']:+.3f} " + f"mal_sel={fmt(row['malicious_selection_rate'])} " + f"replay_distance={row['replay_distance']:.4f} " + f"replay_fraud={row['replay_fraud']}" + ) + out_dir = base / "out" + out_dir.mkdir(exist_ok=True) + prefix = "smoke_band_" if args.smoke else "band_" + payload = { + "config": { + "rounds": rounds, "workers": args.workers, "n_malicious": 5, + "device": device.type, "model_parameters": parameter_count(probe), + "bands": BAND_SWEEP, + }, + "honest_reference": { + "final_val_loss": honest["final_val_loss"], + "val_losses": honest["val_losses"], + }, + "sweep": rows, + } + (out_dir / f"{prefix}sweep_results.json").write_text(json.dumps(payload, indent=2)) + plot_band_sweep(rows, honest["final_val_loss"], out_dir / f"{prefix}sweep.png") + print(f"wrote {out_dir / (prefix + 'sweep_results.json')}") + + +def plot_band_sweep(rows, honest_final, out_path: Path): + fig, (left, right) = plt.subplots(1, 2, figsize=(12.5, 4.6), facecolor=SURFACE) + bands = [r["band"] for r in rows] + style_axes(left) + left.plot(bands, [r["final_val_loss"] for r in rows], color=PALETTE[5], + linewidth=2.0, marker="o", markersize=5, label="within-band 5/16, clip") + left.axhline(honest_final, color=PALETTE[1], linewidth=1.6, linestyle="--", + label="honest reference") + left.set_xlabel("published tolerance band", color=INK_SECONDARY, fontsize=9) + left.set_ylabel("final validation loss", color=INK_SECONDARY, fontsize=9) + left.set_title("Loss damage that fits inside the band", color=INK, + fontsize=11, loc="left", pad=10) + left.legend(frameon=False, fontsize=8, labelcolor=INK_SECONDARY) + style_axes(right) + right.bar([str(b) for b in bands], [r["malicious_selection_rate"] or 0.0 for r in rows], + color=PALETTE[7], width=0.6) + right.set_ylim(0, 1.05) + right.set_xlabel("published tolerance band", color=INK_SECONDARY, fontsize=9) + right.set_ylabel("malicious selection rate", color=INK_SECONDARY, fontsize=9) + right.set_title("Aggregation is the only layer that pushes back", color=INK, + fontsize=11, loc="left", pad=10) + fig.suptitle( + "The band is the adversary's budget: replay audits pass by construction", + color=INK, fontsize=13, x=0.01, ha="left", + ) + fig.tight_layout(rect=(0, 0, 1, 0.94)) + fig.savefig(out_path, dpi=180, facecolor=SURFACE) + plt.close(fig) + + SPARSE_KEYS = [ "honest-mean-iid", "signflip-clip-iid", diff --git a/sim/pyproject.toml b/sim/pyproject.toml index 22be38f..44dcd07 100644 --- a/sim/pyproject.toml +++ b/sim/pyproject.toml @@ -9,5 +9,13 @@ dependencies = [ "matplotlib>=3.9", ] +[dependency-groups] +dev = [ + "pytest>=8.0", +] + [tool.uv] package = false + +[tool.pytest.ini_options] +testpaths = ["tests"] diff --git a/sim/tests/conftest.py b/sim/tests/conftest.py new file mode 100644 index 0000000..c36a7ef --- /dev/null +++ b/sim/tests/conftest.py @@ -0,0 +1,12 @@ +import torch +import pytest + +from leviathan_sim.swarm import CharCorpus + + +@pytest.fixture(scope="session") +def corpus() -> CharCorpus: + generator = torch.Generator().manual_seed(3) + vocab_size = 16 + tokens = torch.randint(0, vocab_size, (4096,), generator=generator) + return CharCorpus(tokens[:3600], tokens[3600:], vocab_size) diff --git a/sim/tests/test_aggregators.py b/sim/tests/test_aggregators.py new file mode 100644 index 0000000..e6fa827 --- /dev/null +++ b/sim/tests/test_aggregators.py @@ -0,0 +1,49 @@ +import torch + +from leviathan_sim.aggregators import CenteredClipAggregator, MeanAggregator + + +def make_deltas(n_honest: int = 7, n_malicious: int = 1, scale: float = 40.0): + generator = torch.Generator().manual_seed(5) + base = torch.randn(64, generator=generator) + deltas = {} + for wid in range(n_honest): + noise = torch.randn(64, generator=generator) * 0.05 + deltas[wid] = base + noise + for wid in range(n_honest, n_honest + n_malicious): + deltas[wid] = -scale * base + honest_mean = torch.stack([deltas[w] for w in range(n_honest)]).mean(dim=0) + return deltas, honest_mean, set(range(n_honest, n_honest + n_malicious)) + + +def test_mean_is_corrupted_by_one_outlier(): + deltas, honest_mean, _ = make_deltas() + agg, mask = MeanAggregator().aggregate(deltas) + assert all(mask.values()) + assert torch.linalg.vector_norm(agg - honest_mean) > torch.linalg.vector_norm(honest_mean) + + +def test_clip_excises_far_outlier_and_tracks_honest_mean(): + deltas, honest_mean, malicious = make_deltas() + agg, mask = CenteredClipAggregator().aggregate(deltas) + for wid in malicious: + assert mask[wid] is False + for wid in range(7): + assert mask[wid] is True + error = torch.linalg.vector_norm(agg - honest_mean) / torch.linalg.vector_norm(honest_mean) + assert error < 0.1 + + +def test_clip_center_persists_across_rounds(): + deltas, _, _ = make_deltas(n_malicious=0) + aggregator = CenteredClipAggregator() + first, _ = aggregator.aggregate(deltas) + assert aggregator.center is not None + second, _ = aggregator.aggregate(deltas) + assert torch.linalg.vector_norm(second - first) < torch.linalg.vector_norm(first) + + +def test_excision_falls_back_to_keep_all_when_everything_is_far(): + deltas = {0: torch.full((8,), 100.0), 1: torch.full((8,), -100.0)} + _, mask = CenteredClipAggregator(excision_multiplier=0.0).aggregate(deltas) + assert all(mask.values()) diff --git a/sim/tests/test_attacks.py b/sim/tests/test_attacks.py new file mode 100644 index 0000000..4965387 --- /dev/null +++ b/sim/tests/test_attacks.py @@ -0,0 +1,89 @@ +import torch + +from leviathan_sim.attacks import InjectionConfig, Injector +from leviathan_sim.swarm import LocalUpdate + + +def make_updates(n_workers: int = 6, dim: int = 128, seed: int = 9): + generator = torch.Generator().manual_seed(seed) + return { + wid: LocalUpdate(wid, torch.randn(dim, generator=generator), 8) + for wid in range(n_workers) + } + + +def relative_distance(a: torch.Tensor, b: torch.Tensor) -> float: + return float(torch.linalg.vector_norm(a - b) / torch.linalg.vector_norm(b)) + + +def test_within_band_stays_inside_the_published_band(): + band = 0.05 + updates = make_updates() + injector = Injector( + InjectionConfig(n_malicious=2, attack="within_band", band=band), list(range(6)), seed=1 + ) + attacked, report = injector.apply(updates) + assert report.malicious_ids == frozenset({0, 1}) + for wid in report.malicious_ids: + distance = relative_distance(attacked[wid].delta, updates[wid].delta) + assert distance <= band + assert abs(distance - band * 0.9) < 1e-5 + for wid in range(2, 6): + assert torch.equal(attacked[wid].delta, updates[wid].delta) + + +def test_within_band_scales_with_the_band(): + updates = make_updates() + for band in (0.02, 0.2): + injector = Injector( + InjectionConfig(n_malicious=1, attack="within_band", band=band), list(range(6)), seed=1 + ) + attacked, _ = injector.apply(updates) + assert abs(relative_distance(attacked[0].delta, updates[0].delta) - band * 0.9) < 1e-5 + + +def test_within_band_bias_opposes_honest_mean(): + updates = make_updates() + injector = Injector( + InjectionConfig(n_malicious=1, attack="within_band", band=0.1), list(range(6)), seed=1 + ) + attacked, _ = injector.apply(updates) + honest_mean = torch.stack([updates[w].delta for w in range(1, 6)]).mean(dim=0) + bias = attacked[0].delta - updates[0].delta + cosine = torch.dot(bias, honest_mean) / ( + torch.linalg.vector_norm(bias) * torch.linalg.vector_norm(honest_mean) + ) + assert cosine < -0.99 + + +def test_sign_flip_scales_and_negates(): + updates = make_updates() + injector = Injector(InjectionConfig(n_malicious=1, attack="sign_flip"), list(range(6)), seed=1) + attacked, _ = injector.apply(updates) + assert torch.allclose(attacked[0].delta, -5.0 * updates[0].delta) + + +def test_lazy_zeroes_delta_and_work(): + updates = make_updates() + injector = Injector(InjectionConfig(n_malicious=1, attack="lazy"), list(range(6)), seed=1) + attacked, _ = injector.apply(updates) + assert torch.count_nonzero(attacked[0].delta) == 0 + assert attacked[0].work_done == 0 + + +def test_alie_coalition_submits_identical_crafted_delta(): + updates = make_updates() + injector = Injector(InjectionConfig(n_malicious=2, attack="alie"), list(range(6)), seed=1) + attacked, _ = injector.apply(updates) + assert torch.equal(attacked[0].delta, attacked[1].delta) + honest = torch.stack([updates[w].delta for w in range(2, 6)]) + expected = honest.mean(dim=0) + 1.5 * honest.std(dim=0) + assert torch.allclose(attacked[0].delta, expected) + + +def test_unknown_attack_rejected(): + try: + Injector(InjectionConfig(attack="nope"), [0], seed=1) + except ValueError: + return + raise AssertionError("expected ValueError") diff --git a/sim/tests/test_economy.py b/sim/tests/test_economy.py new file mode 100644 index 0000000..d55e90c --- /dev/null +++ b/sim/tests/test_economy.py @@ -0,0 +1,64 @@ +from leviathan_sim.economy import ( + EconomyConfig, + StakeLedger, + audit_burn_projection, + break_even_bond, + calibration_table, + genesis_parameters, +) + + +def test_break_even_bond_is_nine_rewards_at_p_10(): + assert break_even_bond(0.1, 1.0) == 9.0 + + +def test_certain_audit_slashes_malicious_on_first_round(): + ledger = StakeLedger([0, 1], EconomyConfig(audit_probability=1.0), seed=17) + caught = ledger.settle_round(0, {0: True, 1: True}, frozenset({1})) + assert caught == [1] + assert ledger.balances[1] == 0.0 + assert 1 not in ledger.active_ids + assert ledger.caught[1] == 0 + + +def test_slash_splits_bond_between_bounty_and_treasury(): + config = EconomyConfig(bond=10.0, slash_bounty_fraction=0.5) + ledger = StakeLedger([0], config, seed=17) + ledger.slash(0) + assert ledger.verifier_income == 5.0 + assert ledger.treasury == 5.0 + assert ledger.pnl()[0] == -10.0 + + +def test_honest_selected_worker_earns_reward(): + ledger = StakeLedger([0], EconomyConfig(audit_probability=0.0), seed=17) + ledger.settle_round(0, {0: True}, frozenset()) + assert ledger.pnl()[0] == ledger.config.reward_selected + + +def test_calibration_bond_scales_with_reward(): + rows = calibration_table([0.1]) + by_preset = {row["preset"]: row for row in rows} + for row in rows: + assert abs(row["break_even_bond_usd"] - 9.0 * row["round_reward_usd"]) < 1e-9 + assert ( + by_preset["7B scale run"]["break_even_bond_usd"] + > by_preset["125M proof run"]["break_even_bond_usd"] + ) + + +def test_zero_fraud_burn_share_is_preset_independent(): + rows = audit_burn_projection([0.1]) + expected = 0.1 * 1.1 / 1.2 + for row in rows: + assert abs(row["burn_share_of_rewards"] - expected) < 1e-9 + assert row["treasury_burn_per_round_usd"] > 0.0 + + +def test_genesis_parameters_follow_the_published_discipline(): + params = genesis_parameters(audit_probability=0.1, band=0.05) + assert params["preset"] == "1B genesis run" + assert params["bond_rounds_of_reward"] == 9.0 + assert abs(params["bond_usd"] - 9.0 * params["round_reward_usd"]) < 1e-9 + assert params["expected_rounds_to_catch"] == 10.0 + assert params["tolerance_band"] == 0.05 diff --git a/sim/tests/test_sparse.py b/sim/tests/test_sparse.py new file mode 100644 index 0000000..72f24b7 --- /dev/null +++ b/sim/tests/test_sparse.py @@ -0,0 +1,35 @@ +import torch + +from leviathan_sim.sparse import chunked_topk_sign + + +def test_density_close_to_target(): + vector = torch.randn(64 * 100, generator=torch.Generator().manual_seed(4)) + compressed = chunked_topk_sign(vector, density=0.02, chunk_size=64, quantize_sign=True) + density = float((compressed != 0).float().mean()) + assert 0.005 <= density <= 0.05 + + +def test_shape_preserved_with_padding(): + vector = torch.randn(1000, generator=torch.Generator().manual_seed(4)) + compressed = chunked_topk_sign(vector, density=0.05, chunk_size=64, quantize_sign=True) + assert compressed.shape == vector.shape + + +def test_signs_match_original(): + vector = torch.randn(256, generator=torch.Generator().manual_seed(4)) + compressed = chunked_topk_sign(vector, density=0.1, chunk_size=64, quantize_sign=True) + kept = compressed != 0 + assert torch.equal(torch.sign(compressed[kept]), torch.sign(vector[kept])) + + +def test_full_density_is_identity(): + vector = torch.randn(128, generator=torch.Generator().manual_seed(4)) + assert torch.equal(chunked_topk_sign(vector, 1.0, 64, True), vector) + + +def test_kept_entries_are_the_largest_magnitudes(): + vector = torch.arange(1.0, 65.0) + compressed = chunked_topk_sign(vector, density=0.05, chunk_size=64, quantize_sign=False) + kept_indices = torch.nonzero(compressed).flatten().tolist() + assert kept_indices == [61, 62, 63] diff --git a/sim/tests/test_verifier.py b/sim/tests/test_verifier.py new file mode 100644 index 0000000..469d90e --- /dev/null +++ b/sim/tests/test_verifier.py @@ -0,0 +1,70 @@ +import torch +from torch.nn.utils import parameters_to_vector + +from leviathan_sim.attacks import InjectionConfig, Injector +from leviathan_sim.model import build_model +from leviathan_sim.swarm import SwarmWorker +from leviathan_sim.verifier import relative_distance, replay_and_verify + +BAND = 0.05 +BLOCK_SIZE = 16 +DEVICE = torch.device("cpu") + + +def make_swarm(corpus, n_workers: int = 4): + model = build_model(corpus.vocab_size, BLOCK_SIZE, seed=7, device=DEVICE) + theta = parameters_to_vector(model.parameters()).detach().clone() + workers = [ + SwarmWorker( + wid, corpus, n_workers, True, inner_steps=2, inner_lr=2e-3, + batch_size=4, block_size=BLOCK_SIZE, seed=100 + wid, device=DEVICE, + ) + for wid in range(n_workers) + ] + return model, theta, workers + + +def test_honest_replay_is_exact(corpus): + model, theta, workers = make_swarm(corpus) + submitted = workers[0].local_round(theta, model, 0).delta + verdict = replay_and_verify(workers[0], model, theta, 0, submitted, BAND) + assert verdict.distance < 1e-6 + assert not verdict.fraud + + +def test_sign_flip_lands_far_above_the_band(corpus): + model, theta, workers = make_swarm(corpus) + honest = workers[1].local_round(theta, model, 0).delta + verdict = replay_and_verify(workers[1], model, theta, 0, -5.0 * honest, BAND) + assert verdict.distance > 1.0 + assert verdict.fraud + + +def test_lazy_zero_delta_is_caught(corpus): + model, theta, workers = make_swarm(corpus) + verdict = replay_and_verify( + workers[2], model, theta, 0, torch.zeros_like(theta), BAND + ) + assert abs(verdict.distance - 1.0) < 1e-5 + assert verdict.fraud + + +def test_within_band_submission_passes_replay(corpus): + model, theta, workers = make_swarm(corpus) + updates = {w.wid: w.local_round(theta, model, 0) for w in workers} + injector = Injector( + InjectionConfig(n_malicious=1, attack="within_band", band=BAND), + [w.wid for w in workers], + seed=11, + ) + attacked, report = injector.apply(updates) + wid = min(report.malicious_ids) + verdict = replay_and_verify(workers[wid], model, theta, 0, attacked[wid].delta, BAND) + assert not verdict.fraud + assert abs(verdict.distance - BAND * 0.9) < 1e-4 + + +def test_relative_distance_is_scale_invariant(): + a = torch.randn(32, generator=torch.Generator().manual_seed(2)) + assert relative_distance(3.0 * a, 3.0 * a) == 0.0 + assert abs(relative_distance(1.1 * a, a) - 0.1) < 1e-6