This describes Beacon (infra/beacon, port 4011): the thing that proves the estate works,
and keeps proving it.
MAP.md describes what is between the services. This describes what watches them.
Verified against source. Where a claim here contradicts a comment in a repository, the source was read and the comment was wrong.
Lantern is a log aggregator. It reads container output, parses it, groups errors into issues, and serves them. It is good at its job and Beacon does not replace it.
But a log aggregator can only report a failure that has already happened to someone. It is reactive by construction: no user, no request; no request, no log line; no log line, no signal. A service that has been quietly refusing every request for six hours produces a great deal of evidence in Lantern and none at all if nobody tried.
Three specific gaps, all of which are now closed:
Every /health route in this estate is a literal. Not "shallow" — literal:
app.get('/health', async () => ({ ok: true, service: 'nimbus' }))That is platform/services/nimbus/src/index.ts:128, and pay, game, forge-mint, crucible and
keyvault are the same closure with a different string. None of them touch Postgres. None check an
upstream. A service whose database vanished an hour ago returns 200 {"ok":true} from all of them
— and docker compose ps calls the container healthy, because the compose healthcheck asks that
same route the same question. Seventeen healthchecks, all agreeing, all uninformative.
Eight of nine product repositories have no tests at all. No test runner, no test files, no test
dependency. What passes for verification is inline shell in each repository's CI: typecheck, build
the image, import() a package inside it to prove the dependency resolves. Those are real checks
and they catch real things, but none of them execute a line of product logic. Only hearth has
tests — sixteen files, a hand-rolled zero-dependency harness — and its own CI runs seven of them.
There is no time series anywhere. No metrics, no uptime history, no latency distribution, no
alerting, no synthetic checks. Lantern computes error counts per service on demand and has a
/api/timeline endpoint that nothing calls. So "is this slower than last week" and "how long was
it down" were unanswerable, and "how long has it been down" was answerable only by watching.
Beacon is the other half. Lantern reads what happened; Beacon provokes what should happen.
One container, built from infra/ in this repository rather than pulled — the same decision
Lantern makes and for the same reason: a server clones only stack, so the tools you reach for
when a deploy is broken must be the ones a broken deploy cannot remove. The GHCR overlay does not
override either of them.
Plain ES-module JavaScript, no build step, three dependencies (postgres, jose, and
playwright-core for the browser). It does three things:
| Cadence | Question it answers | |
|---|---|---|
| Probes | 30 s | Is it up, and is the thing behind it up? |
| Journeys | 5 min – 6 h | Does the product actually work? |
| Conformance | 1 h | Is the chain's arithmetic still correct? |
Not-run is not passed.
This is borrowed verbatim from repos/hearth/docs/evm-spec.md §0, where it exists because the
vectors that assert failure are the easiest ones to fake. It applies identically here. A journey
with no credentials reports skip with the reason, and a skip is never green. A conformance suite
that cannot run says so. A probe that throws inside Beacon is down, not up.
The spec's companion rule is also borrowed: a failed assertion and a thrown error are different
outcomes. There, "an implementation signals EVM failure by returning {exception}, never by
throwing — a thrown JS error is a harness-level ERROR, not a pass." Here, an assertion failure is
fail (the product is broken) and anything else thrown is error (Beacon is broken). Collapse
them and a TypeError in a journey reads as an outage, which is an evening spent debugging a
service that was fine.
Beacon's history lives in the same Postgres instance the monitored services use. So a status page that read its current state from the database would fail on the single question it exists to answer, at the exact moment it was asked.
Everything the status page renders is held in memory and served from there. Postgres holds the time
series. When it is unreachable the grid keeps working, the graphs go blank and say why, and history
resumes when it returns. For the same reason Beacon has no depends_on — the one place it
departs from Lantern's shape. condition: service_healthy on postgres would mean that a database
which never comes up is a monitor which never starts.
Thirty-three of them, every thirty seconds, all concurrent. Sequentially, thirty-three targets against a five-second timeout is a worst case of a hundred and sixty-five seconds, so the monitor's resolution would collapse precisely when the network went bad.
Each service gets the liveness probe everyone already has and a deep probe that cannot answer correctly unless the thing behind it works. The deep ones are free — reads that were already public or already cached. Each names the dependency it proves.
| Probe | What it proves that /health does not |
|---|---|
nimbus.jwks |
The signing key is read from Postgres, so this is the one Nimbus route that cannot succeed with the database down. It is also the route every other service calls to verify a token — the highest-leverage single check in the estate. |
pay.rates |
The oracle is fail-closed: a quote stale past PAY_ORACLE_MAX_AGE_SECONDS stops being usable and every conversion starts returning 503. Invisible from outside until a user tries to convert. |
game.auth |
Unauthenticated on purpose. 401 proves the routes are registered and the auth hook ran. 503 means the game cannot reach Nimbus — a different incident with a different owner. |
crucible.catalog |
Strategies loaded, and surfaces whether CRUCIBLE_LIVE_ENABLED is on. |
lantern |
The one health route in the estate that tells the truth: it reports its Postgres connection, its write queue and its drop counter. |
hearth.work |
A node can answer /info from state while being unable to build a block. Every miner needs /mining/template and nothing else reports on it. |
hearth.rpc.<node> |
A Hearth node runs two listeners: REST on 8645 and Ethereum JSON-RPC on 8545. Everything above asks the first one; forge-pay, the explorer, MetaMask and Hardhat all use the second. It fails soft — listenJsonRpc logs json-rpc listen failed and the node serves REST happily with no eth_* surface at all — so this is the only check that can tell "the node is up" from "the RPC is up". It also asserts the chain id in both encodings: eth_chainId is a hex quantity and net_version is a decimal string, both derived from the same integer by different code, and getting the pair backwards is what makes a wallet refuse the network outright. |
hearth-web.eth-rpc |
The explorer's and the wallet's first call, made the way a visitor's browser makes it: eth_chainId through the bundle's own origin at /eth-rpc/, which nginx proxies to HEARTH_ETH_RPC_UPSTREAM. hearth-web proves the HTML is served and hearth.rpc.seed proves the node answers eth_*; neither proves the wire between them, and that wire is what was broken (CF-13): the proxy named the REST port, every call came back HTTP 404 with {"err":"this is the REST API …"}, and both probes either side stayed green while the public explorer showed nothing but errors. A wrong chain id here is amber rather than red — the per-node probe owns that verdict — and means the proxy reaches a different node. |
hearth.rpc.state |
Picks a settled height every node has and makes each of them name the block and the state root it holds there. hearth.consensus below compares tips, which can only speak about nodes that happen to be level right now — a fork behind the tip reads as lag for as long as the heights keep differing. Same height with a different block is that fork; the same block with a different state root is worse, because the header commits to the root, so one node is serving balances no other node agrees with. |
Five checks are derived — they read the other results rather than making a request, because the question cannot be put to any single node:
hearth.consensus— two nodes at the same height with different tips is a fork, and a fork nobody notices is how a chain quietly becomes two. This is the single most valuable check on the network and no node can perform it about itself.hearth.liveness— the chain is producing blocks, not merely answering questions. Block time is 15 s; ninety seconds of silence is amber, five minutes is red. On a deployment that holds the chain but does not mine it, setBEACON_CHAIN_EXPECT_BLOCKS=false: a seed node with no local hashrate is doing its job, and a check that reddens because nobody else has shown up to mine is a check you learn to ignore — which discredits every check beside it. Height is still recorded and still graphed.hearth.mempoolfollows the same switch, because a queue with nothing mining is the expected state rather than a fault. An unreachable node is still an outage either way.hearth.rpc.liveness— the same question asked of the other listener, plus one only a pair of listeners can answer.eth_blockNumberis the number forge-pay's deposit watcher polls, and a watcher whose height never moves credits nothing and says nothing; and because both listeners are the same process reading the same chain object, a JSON-RPC height sitting more than a block or two behind that node's own REST height means one of them is serving a chain that has moved on. It followsBEACON_CHAIN_EXPECT_BLOCKSfor exactly the reasonshearth.livenessdoes.hearth.peering— an isolated miner is not down. It is happily mining a chain nobody will ever accept, which is worse and looks fine from inside.hearth.mempool— transactions are being mined rather than accumulating.
Hysteresis, in both directions. A single failed probe is a dropped packet; declaring an incident on one is how a monitor teaches its operators to ignore it. Three consecutive failures is an outage — ninety seconds to detection — and two consecutive successes closes it, so a flapping target does not produce a stream of paired notifications.
Edge probes (BEACON_EDGE_ENABLED) hit the public hostnames through the tunnel. Everything else
runs on the compose network, where cloudflared does not exist — and a stack that is perfectly
healthy behind a broken tunnel is, to everyone not on the box, completely down.
Twenty-four working scenarios, driven against the live stack. Probes tell you a service answers. Journeys tell you the product works.
They live centrally in infra/beacon/src/journeys/ rather than in each repository, because the
interesting ones span five repositories and belong to none of them — and because eight of the nine
have no harness to host them.
| Journey | Every | What breaking looks like |
|---|---|---|
identity.signin |
5 m | Nobody can sign into anything. |
identity.signup |
1 h | Registration is broken — the one path no existing user would notice. |
identity.handoff |
5 m | A session cannot cross between products. |
pay.wallet |
5 m | A user cannot see their Shards or be given an address to fund them. |
pay.ledger |
5 m | Nothing can be paid for and nothing refunded. |
pay.convert |
15 m | The exchange is shut and every price in the product is unverified. |
pay.entitlements |
10 m | The shop renders empty; the game stops honouring what people bought. |
mint.order |
6 h | An order takes a user's Shards and cannot then be funded. |
crucible.catalog |
5 m | The workbench has nothing to press. |
crucible.backtest |
15 m | Nobody can test a rule before risking money on it. |
game.worlds |
5 m | A player signs in and is shown nothing to join. |
game.join |
10 m | The game is browsable but unplayable. |
game.cosmetics |
10 m | Everything a player paid for stops being wearable. |
chain.explorer |
5 m | The explorer shows a chain that does not agree with itself. |
chain.accounting |
15 m | The reported coin supply cannot be trusted. |
chain.propagation |
10 m | The network has forked or stopped. |
web.site web.play web.admin web.hearth |
15 m | The bundle 404s or throws, and nginx says 200 anyway. |
web.explorer |
15 m | The browser cannot reach the node RPC. |
web.signin |
30 m | The login form no longer submits. |
platform.lantern |
10 m | The observability pipeline is broken, and the silence reads as health. |
platform.postgres |
5 m | The one dependency every service shares. |
One long-lived synthetic account, not a registration per run. Nimbus has no account deletion —
no DELETE route, and its CORS does not permit the method at all. A journey registering every five
minutes leaves 105,000 permanent rows a year, removable only by hand-written SQL against
production. Registration is still tested hourly by identity.signup, which namespaces its
addresses so they can be found and removed:
DELETE FROM users WHERE email LIKE 'beacon+%';Journeys are staggered and queued, never concurrent. Twelve scenarios starting together would each sign in at once, trip Nimbus's five-per-minute registration limit and ten-per-minute login limit, and report an identity outage Beacon itself caused. That is the classic monitoring failure — the observer becoming the incident — and it is entirely avoidable by spacing the first run of each. Serialising also keeps the step-duration graphs meaningful and stops two journeys moving the synthetic wallet underneath each other.
Money journeys mint and burn their own value. pay.ledger and pay.convert credit the
synthetic account through Forge Pay's /internal surface, move the value, and charge it back.
Nothing real is spent, and BEACON_SYNTHETIC_SHARDS caps what a bug in a journey can cost.
Withdrawals, sweeps, key reveals and world rentals are never touched.
Browser journeys load the public hostname, never the compose address. Every front-end resolves
its API hosts at runtime from window.location.hostname (cloudsforgeHosts() in
@cloudsforge/ui). Load the site as http://site:80 and the bundle concludes the apex is site
and starts calling https://nimbus.site. So a browser journey has to use a hostname the stack is
really served on — which is the right test anyway, and covers the tunnel no other check can see.
Without one configured they skip and say exactly this.
Hearth gets its own page, because it is the only component where being approximately right is indistinguishable from being wrong until somebody loses money.
The chain is mid-rewrite from UTXO to an EVM account model against
repos/hearth/docs/evm-spec.md. That document's central claim is that writing your own EVM is only
tractable because the reference vectors exist, and that no component is done until its vectors
pass.
Its CI does not run them.
repos/hearth/.github/workflows/ci.yml runs seven of sixteen test entrypoints and skips every one
of the nine vector suites plus the entire conformance runner — including the --selftest that
pnpm test invokes. So the phases the spec calls "provably correct before they touch consensus"
were, in CI, not being proved at all.
Beacon runs all nine, hourly, and reports them as a first-class panel:
SUITE STATUS PASS TIME
vectors.keccak pass 52 82ms Keccak-256 padding is Ethereum's, not NIST SHA3's
vectors.rlp pass 149 121ms
vectors.secp256k1 pass 179 1231ms RFC 6979 signing and public-key recovery
vectors.uint256 pass 162 35ms 256-bit wrapping and two's-complement signed arithmetic
vectors.trie pass 302 157ms the secure MPT reproduces published roots
vectors.statedb pass 151 284ms
vectors.opcodes pass 81 36ms
vectors.gas pass 205 35ms a wrong cost here is a chain split
vectors.precompiles pass 83 49ms
harness.selftest pass 85 80ms the harness itself is not the thing that is broken
1,364 assertions across the nine suites, plus 85 self-test checks, attributed to the commit they ran
against. Alongside it, a phase tracker computed from which modules exist under node/src/:
1 Primitives done keccak, rlp, secp256k1, uint256
2 State done trie, statedb
3 Execution partial stack, memory, interpreter missing
4 Transition todo
5–8 todo
RLPTests, TrieTests, VMTests and GeneralStateTests currently report skip, with the
reason: src/evm/interpreter.js does not exist, so there is nothing to run them against. That is
the honest answer and it is deliberately not a green — the spec's own instruction is that if a
vector cannot be made to pass, the correct response is to say so.
When the interpreter lands, those four go live with no change to Beacon.
Five views, one page, plain ES modules — the same construction as Lantern and for the same reason.
- Status — the verdict, then every target grouped, with what it proves, current latency and a 90-day uptime strip. Open incidents banner above everything.
- Journeys — every scenario, its last result, its pass rate, and a Run now button that executes it against the live stack and renders the step waterfall inline. A failed run shows the failing step and the full error, which is the entire point of the page.
- Graphs — p95 latency and availability per target over 1 h to 30 d, and per-journey duration over time. Step names are stable because renaming one abandons its history.
- Chain — height, tip, the three nodes side by side with disagreement flagged, mempool, supply, and the conformance and phase panels above.
- Incidents — reverse-chronological, open ones pinned.
GET /metrics serves Prometheus exposition from memory. That is the deliberate hedge on the
build-versus-buy decision: if this estate outgrows a bespoke monitor, Grafana bolts on without
redesigning anything. It reads no database, so a scrape cannot load Postgres.
The public status view is a much smaller thing than the operator's. With
BEACON_PUBLIC_STATUS=true, /api/status answers without a credential — carrying names, states and
uptime and nothing else. No error strings, no proves text, no latency, no journey step names, no
chain internals. Every one of those describes the shape of an outage to whoever is causing it.
One outbound webhook, fired on incident open and close and on nothing else.
Thirty-three targets probed twice a minute is about ninety-five thousand check results a day. A rule that fires on any of them is a rule whose channel is muted in week one, after which the monitor is decorative. Journeys need two consecutive failures — ten minutes at the default cadence — because a scenario touches six services and one flake in any of them is not news.
The payload is plain JSON with a text field, so Slack, Discord and a shell script all work.
Hard-coding one vendor's block-kit schema into the monitor is how you end up unable to change chat
tools. Failure to deliver is logged and swallowed: the incident is already recorded, and a broken
webhook must not take the monitor with it.
- No load or performance testing. Latency is recorded, not driven.
- No unit tests for the eight repositories that have none. Beacon tests the assembled system
from outside. It does not make
crucible's strategy maths correct; it makes it observable that a backtest completes. Those repositories still need their own tests, and Beacon passing is not an argument that they do not. - No
POST /txor/mining/submit. Beacon reads the chain and asks for work; it does not broadcast. - Nothing destructive, ever. Withdrawals, sweeps, abandonments, key reveals, world rentals and
/deployare all deliberately untouched, and named as such at each call site. mint.orderleaves a row per run. That is why it runs every six hours rather than every five minutes, and it is the only journey that accumulates state on purpose.
docker compose up -d --build beacon # it is part of the stack
open http://localhost:4011
It needs no configuration to be useful: probes, chain checks and conformance run out of the box.
BEACON_SYNTHETIC_EMAIL / _PASSWORD unlock the journeys that need an identity, PAY_SERVICE_TOKEN
the ones that move value, BEACON_EDGE_ENABLED the browser and tunnel ones. Each reports skip
with its reason when unset, because a monitor that shows red for missing configuration is a monitor
you learn to ignore.
By hand, in the container or on the host:
node src/cli.js probe # one cycle, exit 1 if a critical target is down
node src/cli.js journey [name] # run one or all, print the step waterfall
node src/cli.js conformance # the nine vector suites and the phase tracker
node src/cli.js targets # the probe catalogue, no network
node src/cli.js journeys # the journey catalogue, no network
conformance is worth wiring into hearth's own CI, which is where it should have been.