Skip to content

Latest commit

 

History

History
323 lines (233 loc) · 14 KB

File metadata and controls

323 lines (233 loc) · 14 KB

VERIFICATION.md — what was run, and what it said

The defect register is audit.md. The plan is upgrade.md. This is the third document: what was actually executed against the remediated tree, and what came back.

It exists because the estate has been burned twice by the gap between "the code changed" and "the thing works". docker compose build returns exit 0 while cancelling every target, and a rejected CORS preflight 204s so the handler never runs and nothing appears in any log. Both defects passed a reading of the diff. Neither would have survived being run.

Every number below is either something executed here, or something reported by the agent that did the work — and the two are labelled separately throughout, because they are not the same kind of evidence. Where a figure was re-derived independently, both readings are given.


0. The results, in one table

what result
Continuous integration, 6 repositories all checks green, including every Docker image build
Typechecks, 7 workspaces 7/7 clean
Hearth full suite 31 suites, 86,094 checks, exit 0
Hearth EVM conformance (GeneralStateTests) 20,077/20,077, 60,231 checks, 0 failed
Beacon monitoring suite 23/23
Hearth PoW parameter harness 7/7
Explorer API — fixtures / in-process / external node 177/177 · 27/27 · 21/21
Faucet 66 passing
Full-block execution 35.3 s → 5.2 s, 212.1 MiB → 9.2 MiB
Tickets put through adversarial review 29, of which 19 confirmed clean
Security regressions found by that review and closed 3, including one account-takeover

Two results are worth stating in words rather than numbers. The Docker image builds pass in CI — in the previous round every image in the estate silently failed to build while the build command reported success. And the 2 GiB PoW target was measured for the first time and found to cost 185.7 seconds per evaluation, which is a genesis-blocking finding rather than a passing test; §3.

§6 lists what was not tested. It is part of the result, not an appendix to it.


1. Scope

51 items: the 49 register tickets, plus two that are not defects — outbound mail delivery for Nimbus, and the missing JSON-RPC methods.

count
Tickets with a recorded outcome 40
— resolved 23
— already fixed by earlier work, verified and closed 9
— partial, with the remainder written down 8
Tickets put through adversarial review 29
— confirmed: defect gone, nothing else broken 19
— incomplete: partially closed 7
— regression: defect closed, something else broken 3

The reviewer was told to assume each fix was not real and to walk the ticket's own How it fails scenario through the new code until it either stopped or didn't. All three regressions were sent back and closed; see §4.


2. Executed here

Run directly, not reported by an agent.

Typechecks — 7/7 clean

pnpm typecheck at the root of each workspace: crucible, forge-keyvault, forge-mint, forge-pay, ninety-days-after, platform, shared-libs.

forge-mint initially failed with ERR_PNPM_VERIFY_DEPS_BEFORE_RUN — a test script had been added without a matching install. pnpm install resolved it; the lockfile did not change and the supply-chain policy check passed on all 363 entries.

Beacon — 23/23

node --test in infra/beacon. Covers the new JSON-RPC monitoring surface: two nodes naming the same block with different state roots is down; a node whose JSON-RPC height lags its own REST height is down; one block of skew is not an incident; a JSON-RPC height that stops moving eventually goes down.

Hearth PoW parameters — 7/7

node test/pow-params.js. The harness parameterises the real homefireHash and asserts it reproduces the configured digest before it measures anything, so a passing run is a measurement of the shipped function rather than of a copy of it.

Independent extrapolation from this run: 183.6 s per evaluation at 2 GiB. The agent's measured full run: 185.7 s. 1% apart — see §3.

CI — all green

Six pull requests, every check passing:

repo checks
crucible Docker image, Secret hygiene, "The settlement pass bills a gain once", Typecheck + build
forge-keyvault Docker image, Secret hygiene, "Signing refusals and the treasury pin", Typecheck
forge-mint Committed bytecode matches the source, Docker image, Secret hygiene, Typecheck + build
forge-pay Docker image, Secret hygiene, Typecheck
ninety-days-after Docker images, Secret hygiene, Typecheck + build
platform Docker images, Secret hygiene, Typecheck + build

The Docker image jobs passing is the load-bearing result here. In the previous round every image in the estate silently failed to build while the build command reported success, and a "healthy" stack was running stale images. These are real builds, in CI, with a real exit code.


3. The PoW measurement

The single most consequential result, and the reason this document exists at all.

Every Hearth block ever produced — in tests, in CI, on the compose testnet — used a 64 KiB scratchpad and a 256-step walk. The documented mainnet intent is 2 GiB and 2,048+ steps: a 32,768× increase in per-attempt memory that no code had ever executed at.

Measured, one machine, per evaluation:

pad per evaluation peak RSS
64 KiB — what everything has always run 6.57 ms
256 MiB 24.0 s
2 GiB — the intent 185.7 s 2,165 MiB

Against a 15-second target block time, one attempt is 12× the whole block interval. A 200-block getblocks page costs roughly 10.3 core-hours to verify. The 2,048-step walk turns out to cost 1.31×, not 8× — so the pad is the cost, and it is CPU-bound, not memory-bound.

node/src/params.js now carries POW_MAX_SCRATCH_KIB: 4096 and refuses to start above it. A node configured with the launch checklist's own 2 GiB value was demonstrated declining to boot, which is the correct failure: better than mining a chain nothing can validate.

Two things were deliberately not measured, and both are recorded as unmeasured rather than estimated: the 3-node measurement network (at 185.7 s per attempt the first block is ~13 core-hours, so the measurement that justified the network is the one proving it cannot run), and P2P_BLOCK_VERIFY_BURST, because no value works at that cost.

This is a genesis-blocking product decision, not a bug. A mineable pad size has to be chosen, and that choice changes Hearth's security argument.


4. The three regressions

Each was found by adversarial review, each was sent back, and each was re-read in the tree here rather than accepted on report.

Password-reset delivery — account takeover. Wiring the mailer made a latent weakness live: the reset link was built from the request Host header, so anyone who knew an address could have the deployment mail them a correctly-branded link pointing at a host of their choosing. It reproduced against a running service with an SMTP sink. Now minted from NIMBUS_PUBLIC_URL regardless of how the request was addressed, with an audit line when the two disagree — deliberately not a refusal, because refusing tells a prober which hosts are real.

The same review found the route had become an enumeration oracle by timing: awaiting the SMTP send split response time 10 ms for an unknown address against 6,015 ms for a known one. It now answers before the send.

CF-36 — orphaned refresh tokens. The grace window minted a sibling token, but sign-out revoked only the one presented, leaving a live server-side token nobody held for 30 days. revokeRefreshToken now revokes by family.

CF-19 — a paused bot could be billed for a gain it never realised. The sweep called full assessment on non-running bots. It now carries a scope: running bots are assessed, paused and stopped bots have arrears collected and nothing more.


5. Reported by the agents that did the work

Not re-run here. Recorded with attribution so a later reader knows which claims still want independent confirmation.

suite result
Hearth pnpm test 31 suites, 86,094 checks, exit 0
Hearth GeneralStateTests conformance 20,077/20,077, 60,231 checks, 0 failed (~32 min)
explorer-api fixtures 177/177
explorer-api live chain, in process 27/27
explorer-api against an external node 21/21
faucet 66

Full-block execution (CF-10), same machine, same 30M-gas SSTORE transaction: 35.3 s / 212.1 MiB → 5.2 s / 9.2 MiB. Five times the state-trie width now costs 1.00× rather than 1.31×. The gate fails on pre-fix code (3/5) and passes now (5/5), with wall-clock expressed as a ratio to a calibration block so a slow runner cannot flake it. One dimension is explicitly partial: setStorage still materialises the storage root per write, which is about a third of the remaining 5.2 s. Deferring it needs slot-level journalling and a rework of how revert works, and is written up rather than half-done.


5a. End-to-end, against the assembled stack

The gap this document opened with, now closed. All 18 services rebuilt from the remediated source and recreated; 16 images built, zero cancelled targets. Every service reported healthy.

The pre-existing containers had been "healthy" for 20 hours on pre-remediation images — none of the resolved tickets were in them. Nothing below would have meant anything without the rebuild.

Password reset over SMTP — the whole loop

Driven against a local SMTP sink, through the real service, with a throwaway account:

step result
register 201
request reset 202
message delivered captured at the relay
redeem token, set new password 204
sign in with the new password 200
sign in with the old password 401
replay the same token 401 — single-use holds

The forged-Host attack, closed

A request carrying Host: evil.attacker.test was answered 202 like any other, and the link in the delivered mail read https://account.cloudsforge.online/reset#token=… — the configured origin, not the forged one. Nimbus wrote the audit line naming both, exactly as designed:

"audit":"password_reset_host_ignored",
"addressedAs":"http://evil.attacker.test",
"minted":"https://account.cloudsforge.online"

The token leaks nowhere

Both 64-character tokens were decoded out of the quoted-printable message bodies and grepped against all 18 container logs. Zero occurrences. CF-17 holds in the assembled stack, not just in the diff.

The timing oracle is gone

Known address 12.3 ms, unknown address 8.3 ms — no separation to read. Before the fix the same measurement split 10 ms against 6,015 ms.

CF-02 — the browser preflight

OPTIONS /cosmetics with Origin and Access-Control-Request-Method: PUT now answers:

access-control-allow-methods: GET, HEAD, OPTIONS, POST, PUT

This is the faithful test: a preflight is exactly what a browser sends, so this is not a curl standing in for a browser.

The chain

node block peers
seed 0x11bc 2
miner1 0x11bc 1
miner2 0x11bc 1

eth_chainId0x1cf4 (7412) and net_version7412 on the JSON-RPC port. All three nodes at one height, so CF-27's genesis-hash handshake is admitting legitimate peers rather than partitioning the network — the specific regression risk that change carried. net_peerCount and eth_newBlockFilter both answer, confirming the new method families are live.

Beacon refuses unauthenticated reads with 401, which is correct: the status page is not public.


6. What was NOT verified

The honest list. Everything here is a real gap, not a formality.

Outbound mail has never reached a real inbox. The loop in §5a ran against a local SMTP sink, which proves the code path, the link origin and the single-use redemption — but not that a real relay accepts the message, nor that it survives SPF/DKIM/DMARC and lands somewhere other than a spam folder. The first send through Brevo or any real provider is still ahead.

Most of the estate was not driven. §5a covers password reset, the game CORS preflight and the chain. It does not cover the money paths — no deposit was credited, no withdrawal reached a terminal state, no manual sweep was refused for an overstated amount (CF-08), and no ForgeMint order was deployed with a customer-supplied ownerAddress (CF-01). Those are the highest-value remaining checks.

Nothing was driven through an actual browser. The CF-02 preflight is faithful because a preflight is a plain HTTP request, but no page was loaded, no SPA rendered, and no user journey clicked through.

Seven tickets remain INCOMPLETE after adversarial review: CF-08, CF-15, CF-22, CF-37, CF-38, CF-42, CF-43. Repair passes ran afterwards and may have closed some of them; that has not been confirmed ticket by ticket.

Eight tickets are partial, with the remainder written down in their commit bodies rather than in this file.

No load, soak or adversarial network testing of any kind.

The PoW figures are one machine. Consumer Apple silicon, one process, no thermal headroom analysis. They establish an order of magnitude and a boot-refusing ceiling. They are not a parameter-selection study, and choosing the real pad size needs one.


7. How to re-run any of this

# per repo
cd repos/<name> && pnpm install && pnpm typecheck

# beacon
cd infra/beacon && node --test

# hearth: everything
cd repos/hearth/node && pnpm test

# hearth: PoW parameters (add --full to measure 2 GiB for real — minutes, and 2+ GiB of RSS)
cd repos/hearth/node && node test/pow-params.js --sweep

# hearth: EVM conformance
cd repos/hearth/node && node test/conformance/runner.js --suite=GeneralStateTests

--full is not in the default path on purpose: it allocates 2 GiB and takes minutes per evaluation, which is the finding, not a defect in the harness.