Skip to content

Latest commit

 

History

History
354 lines (275 loc) · 21.5 KB

File metadata and controls

354 lines (275 loc) · 21.5 KB

CloudsForge — testing and monitoring

This describes Beacon (infra/beacon, port 4011): the thing that proves the estate works, and keeps proving it.

MAP.md describes what is between the services. This describes what watches them.

Verified against source. Where a claim here contradicts a comment in a repository, the source was read and the comment was wrong.


1. Why Lantern is not enough

Lantern is a log aggregator. It reads container output, parses it, groups errors into issues, and serves them. It is good at its job and Beacon does not replace it.

But a log aggregator can only report a failure that has already happened to someone. It is reactive by construction: no user, no request; no request, no log line; no log line, no signal. A service that has been quietly refusing every request for six hours produces a great deal of evidence in Lantern and none at all if nobody tried.

Three specific gaps, all of which are now closed:

Every /health route in this estate is a literal. Not "shallow" — literal:

app.get('/health', async () => ({ ok: true, service: 'nimbus' }))

That is platform/services/nimbus/src/index.ts:128, and pay, game, forge-mint, crucible and keyvault are the same closure with a different string. None of them touch Postgres. None check an upstream. A service whose database vanished an hour ago returns 200 {"ok":true} from all of them — and docker compose ps calls the container healthy, because the compose healthcheck asks that same route the same question. Seventeen healthchecks, all agreeing, all uninformative.

Eight of nine product repositories have no tests at all. No test runner, no test files, no test dependency. What passes for verification is inline shell in each repository's CI: typecheck, build the image, import() a package inside it to prove the dependency resolves. Those are real checks and they catch real things, but none of them execute a line of product logic. Only hearth has tests — sixteen files, a hand-rolled zero-dependency harness — and its own CI runs seven of them.

There is no time series anywhere. No metrics, no uptime history, no latency distribution, no alerting, no synthetic checks. Lantern computes error counts per service on demand and has a /api/timeline endpoint that nothing calls. So "is this slower than last week" and "how long was it down" were unanswerable, and "how long has it been down" was answerable only by watching.

Beacon is the other half. Lantern reads what happened; Beacon provokes what should happen.


2. Beacon

One container, built from infra/ in this repository rather than pulled — the same decision Lantern makes and for the same reason: a server clones only stack, so the tools you reach for when a deploy is broken must be the ones a broken deploy cannot remove. The GHCR overlay does not override either of them.

Plain ES-module JavaScript, no build step, three dependencies (postgres, jose, and playwright-core for the browser). It does three things:

Cadence Question it answers
Probes 30 s Is it up, and is the thing behind it up?
Journeys 5 min – 6 h Does the product actually work?
Conformance 1 h Is the chain's arithmetic still correct?

The one rule everything else follows

Not-run is not passed.

This is borrowed verbatim from repos/hearth/docs/evm-spec.md §0, where it exists because the vectors that assert failure are the easiest ones to fake. It applies identically here. A journey with no credentials reports skip with the reason, and a skip is never green. A conformance suite that cannot run says so. A probe that throws inside Beacon is down, not up.

The spec's companion rule is also borrowed: a failed assertion and a thrown error are different outcomes. There, "an implementation signals EVM failure by returning {exception}, never by throwing — a thrown JS error is a harness-level ERROR, not a pass." Here, an assertion failure is fail (the product is broken) and anything else thrown is error (Beacon is broken). Collapse them and a TypeError in a journey reads as an outage, which is an evening spent debugging a service that was fine.

Live state is in memory; history is in Postgres

Beacon's history lives in the same Postgres instance the monitored services use. So a status page that read its current state from the database would fail on the single question it exists to answer, at the exact moment it was asked.

Everything the status page renders is held in memory and served from there. Postgres holds the time series. When it is unreachable the grid keeps working, the graphs go blank and say why, and history resumes when it returns. For the same reason Beacon has no depends_on — the one place it departs from Lantern's shape. condition: service_healthy on postgres would mean that a database which never comes up is a monitor which never starts.


3. Probes

Thirty-three of them, every thirty seconds, all concurrent. Sequentially, thirty-three targets against a five-second timeout is a worst case of a hundred and sixty-five seconds, so the monitor's resolution would collapse precisely when the network went bad.

Each service gets the liveness probe everyone already has and a deep probe that cannot answer correctly unless the thing behind it works. The deep ones are free — reads that were already public or already cached. Each names the dependency it proves.

Probe What it proves that /health does not
nimbus.jwks The signing key is read from Postgres, so this is the one Nimbus route that cannot succeed with the database down. It is also the route every other service calls to verify a token — the highest-leverage single check in the estate.
pay.rates The oracle is fail-closed: a quote stale past PAY_ORACLE_MAX_AGE_SECONDS stops being usable and every conversion starts returning 503. Invisible from outside until a user tries to convert.
game.auth Unauthenticated on purpose. 401 proves the routes are registered and the auth hook ran. 503 means the game cannot reach Nimbus — a different incident with a different owner.
crucible.catalog Strategies loaded, and surfaces whether CRUCIBLE_LIVE_ENABLED is on.
lantern The one health route in the estate that tells the truth: it reports its Postgres connection, its write queue and its drop counter.
hearth.work A node can answer /info from state while being unable to build a block. Every miner needs /mining/template and nothing else reports on it.
hearth.rpc.<node> A Hearth node runs two listeners: REST on 8645 and Ethereum JSON-RPC on 8545. Everything above asks the first one; forge-pay, the explorer, MetaMask and Hardhat all use the second. It fails soft — listenJsonRpc logs json-rpc listen failed and the node serves REST happily with no eth_* surface at all — so this is the only check that can tell "the node is up" from "the RPC is up". It also asserts the chain id in both encodings: eth_chainId is a hex quantity and net_version is a decimal string, both derived from the same integer by different code, and getting the pair backwards is what makes a wallet refuse the network outright.
hearth-web.eth-rpc The explorer's and the wallet's first call, made the way a visitor's browser makes it: eth_chainId through the bundle's own origin at /eth-rpc/, which nginx proxies to HEARTH_ETH_RPC_UPSTREAM. hearth-web proves the HTML is served and hearth.rpc.seed proves the node answers eth_*; neither proves the wire between them, and that wire is what was broken (CF-13): the proxy named the REST port, every call came back HTTP 404 with {"err":"this is the REST API …"}, and both probes either side stayed green while the public explorer showed nothing but errors. A wrong chain id here is amber rather than red — the per-node probe owns that verdict — and means the proxy reaches a different node.
hearth.rpc.state Picks a settled height every node has and makes each of them name the block and the state root it holds there. hearth.consensus below compares tips, which can only speak about nodes that happen to be level right now — a fork behind the tip reads as lag for as long as the heights keep differing. Same height with a different block is that fork; the same block with a different state root is worse, because the header commits to the root, so one node is serving balances no other node agrees with.

Five checks are derived — they read the other results rather than making a request, because the question cannot be put to any single node:

  • hearth.consensus — two nodes at the same height with different tips is a fork, and a fork nobody notices is how a chain quietly becomes two. This is the single most valuable check on the network and no node can perform it about itself.
  • hearth.liveness — the chain is producing blocks, not merely answering questions. Block time is 15 s; ninety seconds of silence is amber, five minutes is red. On a deployment that holds the chain but does not mine it, set BEACON_CHAIN_EXPECT_BLOCKS=false: a seed node with no local hashrate is doing its job, and a check that reddens because nobody else has shown up to mine is a check you learn to ignore — which discredits every check beside it. Height is still recorded and still graphed. hearth.mempool follows the same switch, because a queue with nothing mining is the expected state rather than a fault. An unreachable node is still an outage either way.
  • hearth.rpc.liveness — the same question asked of the other listener, plus one only a pair of listeners can answer. eth_blockNumber is the number forge-pay's deposit watcher polls, and a watcher whose height never moves credits nothing and says nothing; and because both listeners are the same process reading the same chain object, a JSON-RPC height sitting more than a block or two behind that node's own REST height means one of them is serving a chain that has moved on. It follows BEACON_CHAIN_EXPECT_BLOCKS for exactly the reasons hearth.liveness does.
  • hearth.peering — an isolated miner is not down. It is happily mining a chain nobody will ever accept, which is worse and looks fine from inside.
  • hearth.mempool — transactions are being mined rather than accumulating.

Hysteresis, in both directions. A single failed probe is a dropped packet; declaring an incident on one is how a monitor teaches its operators to ignore it. Three consecutive failures is an outage — ninety seconds to detection — and two consecutive successes closes it, so a flapping target does not produce a stream of paired notifications.

Edge probes (BEACON_EDGE_ENABLED) hit the public hostnames through the tunnel. Everything else runs on the compose network, where cloudflared does not exist — and a stack that is perfectly healthy behind a broken tunnel is, to everyone not on the box, completely down.


4. Journeys

Twenty-four working scenarios, driven against the live stack. Probes tell you a service answers. Journeys tell you the product works.

They live centrally in infra/beacon/src/journeys/ rather than in each repository, because the interesting ones span five repositories and belong to none of them — and because eight of the nine have no harness to host them.

Journey Every What breaking looks like
identity.signin 5 m Nobody can sign into anything.
identity.signup 1 h Registration is broken — the one path no existing user would notice.
identity.handoff 5 m A session cannot cross between products.
pay.wallet 5 m A user cannot see their Shards or be given an address to fund them.
pay.ledger 5 m Nothing can be paid for and nothing refunded.
pay.convert 15 m The exchange is shut and every price in the product is unverified.
pay.entitlements 10 m The shop renders empty; the game stops honouring what people bought.
mint.order 6 h An order takes a user's Shards and cannot then be funded.
crucible.catalog 5 m The workbench has nothing to press.
crucible.backtest 15 m Nobody can test a rule before risking money on it.
game.worlds 5 m A player signs in and is shown nothing to join.
game.join 10 m The game is browsable but unplayable.
game.cosmetics 10 m Everything a player paid for stops being wearable.
chain.explorer 5 m The explorer shows a chain that does not agree with itself.
chain.accounting 15 m The reported coin supply cannot be trusted.
chain.propagation 10 m The network has forked or stopped.
web.site web.play web.admin web.hearth 15 m The bundle 404s or throws, and nginx says 200 anyway.
web.explorer 15 m The browser cannot reach the node RPC.
web.signin 30 m The login form no longer submits.
platform.lantern 10 m The observability pipeline is broken, and the silence reads as health.
platform.postgres 5 m The one dependency every service shares.

Four decisions worth defending

One long-lived synthetic account, not a registration per run. Nimbus has no account deletion — no DELETE route, and its CORS does not permit the method at all. A journey registering every five minutes leaves 105,000 permanent rows a year, removable only by hand-written SQL against production. Registration is still tested hourly by identity.signup, which namespaces its addresses so they can be found and removed:

DELETE FROM users WHERE email LIKE 'beacon+%';

Journeys are staggered and queued, never concurrent. Twelve scenarios starting together would each sign in at once, trip Nimbus's five-per-minute registration limit and ten-per-minute login limit, and report an identity outage Beacon itself caused. That is the classic monitoring failure — the observer becoming the incident — and it is entirely avoidable by spacing the first run of each. Serialising also keeps the step-duration graphs meaningful and stops two journeys moving the synthetic wallet underneath each other.

Money journeys mint and burn their own value. pay.ledger and pay.convert credit the synthetic account through Forge Pay's /internal surface, move the value, and charge it back. Nothing real is spent, and BEACON_SYNTHETIC_SHARDS caps what a bug in a journey can cost. Withdrawals, sweeps, key reveals and world rentals are never touched.

Browser journeys load the public hostname, never the compose address. Every front-end resolves its API hosts at runtime from window.location.hostname (cloudsforgeHosts() in @cloudsforge/ui). Load the site as http://site:80 and the bundle concludes the apex is site and starts calling https://nimbus.site. So a browser journey has to use a hostname the stack is really served on — which is the right test anyway, and covers the tunnel no other check can see. Without one configured they skip and say exactly this.


5. The chain

Hearth gets its own page, because it is the only component where being approximately right is indistinguishable from being wrong until somebody loses money.

The chain is mid-rewrite from UTXO to an EVM account model against repos/hearth/docs/evm-spec.md. That document's central claim is that writing your own EVM is only tractable because the reference vectors exist, and that no component is done until its vectors pass.

Its CI does not run them.

repos/hearth/.github/workflows/ci.yml runs seven of sixteen test entrypoints and skips every one of the nine vector suites plus the entire conformance runner — including the --selftest that pnpm test invokes. So the phases the spec calls "provably correct before they touch consensus" were, in CI, not being proved at all.

Beacon runs all nine, hourly, and reports them as a first-class panel:

SUITE                STATUS  PASS   TIME
vectors.keccak       pass     52     82ms   Keccak-256 padding is Ethereum's, not NIST SHA3's
vectors.rlp          pass    149    121ms
vectors.secp256k1    pass    179   1231ms   RFC 6979 signing and public-key recovery
vectors.uint256      pass    162     35ms   256-bit wrapping and two's-complement signed arithmetic
vectors.trie         pass    302    157ms   the secure MPT reproduces published roots
vectors.statedb      pass    151    284ms
vectors.opcodes      pass     81     36ms
vectors.gas          pass    205     35ms   a wrong cost here is a chain split
vectors.precompiles  pass     83     49ms
harness.selftest     pass     85     80ms   the harness itself is not the thing that is broken

1,364 assertions across the nine suites, plus 85 self-test checks, attributed to the commit they ran against. Alongside it, a phase tracker computed from which modules exist under node/src/:

1  Primitives  done     keccak, rlp, secp256k1, uint256
2  State       done     trie, statedb
3  Execution   partial  stack, memory, interpreter missing
4  Transition  todo
5–8                     todo

RLPTests, TrieTests, VMTests and GeneralStateTests currently report skip, with the reason: src/evm/interpreter.js does not exist, so there is nothing to run them against. That is the honest answer and it is deliberately not a green — the spec's own instruction is that if a vector cannot be made to pass, the correct response is to say so.

When the interpreter lands, those four go live with no change to Beacon.


6. The pages

Five views, one page, plain ES modules — the same construction as Lantern and for the same reason.

  • Status — the verdict, then every target grouped, with what it proves, current latency and a 90-day uptime strip. Open incidents banner above everything.
  • Journeys — every scenario, its last result, its pass rate, and a Run now button that executes it against the live stack and renders the step waterfall inline. A failed run shows the failing step and the full error, which is the entire point of the page.
  • Graphs — p95 latency and availability per target over 1 h to 30 d, and per-journey duration over time. Step names are stable because renaming one abandons its history.
  • Chain — height, tip, the three nodes side by side with disagreement flagged, mempool, supply, and the conformance and phase panels above.
  • Incidents — reverse-chronological, open ones pinned.

GET /metrics serves Prometheus exposition from memory. That is the deliberate hedge on the build-versus-buy decision: if this estate outgrows a bespoke monitor, Grafana bolts on without redesigning anything. It reads no database, so a scrape cannot load Postgres.

The public status view is a much smaller thing than the operator's. With BEACON_PUBLIC_STATUS=true, /api/status answers without a credential — carrying names, states and uptime and nothing else. No error strings, no proves text, no latency, no journey step names, no chain internals. Every one of those describes the shape of an outage to whoever is causing it.


7. Alerting

One outbound webhook, fired on incident open and close and on nothing else.

Thirty-three targets probed twice a minute is about ninety-five thousand check results a day. A rule that fires on any of them is a rule whose channel is muted in week one, after which the monitor is decorative. Journeys need two consecutive failures — ten minutes at the default cadence — because a scenario touches six services and one flake in any of them is not news.

The payload is plain JSON with a text field, so Slack, Discord and a shell script all work. Hard-coding one vendor's block-kit schema into the monitor is how you end up unable to change chat tools. Failure to deliver is logged and swallowed: the incident is already recorded, and a broken webhook must not take the monitor with it.


8. What this does not do

  • No load or performance testing. Latency is recorded, not driven.
  • No unit tests for the eight repositories that have none. Beacon tests the assembled system from outside. It does not make crucible's strategy maths correct; it makes it observable that a backtest completes. Those repositories still need their own tests, and Beacon passing is not an argument that they do not.
  • No POST /tx or /mining/submit. Beacon reads the chain and asks for work; it does not broadcast.
  • Nothing destructive, ever. Withdrawals, sweeps, abandonments, key reveals, world rentals and /deploy are all deliberately untouched, and named as such at each call site.
  • mint.order leaves a row per run. That is why it runs every six hours rather than every five minutes, and it is the only journey that accumulates state on purpose.

9. Running it

docker compose up -d --build beacon      # it is part of the stack
open http://localhost:4011

It needs no configuration to be useful: probes, chain checks and conformance run out of the box. BEACON_SYNTHETIC_EMAIL / _PASSWORD unlock the journeys that need an identity, PAY_SERVICE_TOKEN the ones that move value, BEACON_EDGE_ENABLED the browser and tunnel ones. Each reports skip with its reason when unset, because a monitor that shows red for missing configuration is a monitor you learn to ignore.

By hand, in the container or on the host:

node src/cli.js probe          # one cycle, exit 1 if a critical target is down
node src/cli.js journey [name] # run one or all, print the step waterfall
node src/cli.js conformance    # the nine vector suites and the phase tracker
node src/cli.js targets        # the probe catalogue, no network
node src/cli.js journeys       # the journey catalogue, no network

conformance is worth wiring into hearth's own CI, which is where it should have been.