Skip to content

feat: status.hellonear.xyz, a status page for HelloNear and its providers - #302

Merged
Lexirieru merged 7 commits into
mainfrom
feat/status-page
Nov 12, 2026
Merged

Lexirieru merged 7 commits into
mainfrom
feat/status-page

Conversation

@Lexirieru

Copy link
Copy Markdown
Contributor

@Lin1er @AncungAulia @Nabil-Aufa On 13 Nov, two outages came from outside our code in one night: NEAR AI's embeddings hung, and 1Click quotes timed out. We only found them with curl. This PR adds a public status page so the team and the judges can see that at a glance.

What

  • Backend (apps/be/src/status.ts, statusMath.ts, migration 016_service_checks.sql):
    • Probes 9 services in parallel every 5 minutes, in this order:
      • HelloNear API and engine;
      • NEAR AI chat and embeddings, through the engine's new internal /internal/v1/probes, because only the engine has the key;
      • 1Click dry quote;
      • OutLayer;
      • FastNear RPC;
      • NEAR AI Agent Market;
      • Phala TEE Explorer.
    • Keeps 8 days of checks.
    • Serves a public GET /v1/status (CORS *, cached 30 s) with, per service:
      • status, latency and the impact on users;
      • 24 h and 7 d uptime;
      • 48 half-hour buckets with checks / bad / detail;
      • incidents (runs of failed checks) from the last 7 days.
    • Thresholds are in docs/api-contract.md.
  • apps/status (new Next.js app, deployed on Vercel), modelled on status.claude.com:
    • an overall banner, and what is down with what users see because of it;
    • per service, green / yellow / red bars (all checks fine / some failed / all failed);
    • a hover card on every bar;
    • past incidents by day;
    • light and dark themes.
    • Server-rendered, revalidated every 30 s. If the status API can't be read, the page says so.
  • New CI workflow status. Docs: apps/status/CLAUDE.md, root CLAUDE.md, both API docs, README (Live now table, 63 operations).

Tests (local)

  • be:
    • typecheck; test:unit 38/38;
    • status.test.ts 6/6 against local Postgres with fake upstreams (some up, one slow, one 500, engine down). It checks the shape, CORS, order, statuses, incidents, the cache and pruning, and now cleans up its rows afterwards;
    • full bun run test: 156 pass, 0 fail, 1 skipped.
  • engine: typecheck; test:unit 191/191 (probes 3/3, 401 without the key).
  • status: typecheck; node:test 5/5; next build passes.
  • Load, GET /v1/status with 7 days of rows:
    • cached: 500 requests at 10 concurrent, 0 errors, p95 6.2 ms;
    • uncached worst case: p95 970 ms.
  • Live on a local stack (Axel checked localhost:3200):
    • 1Click quote down: HTTP 500 after 25.9 s;
    • NEAR AI embeddings back (1.1 s), chat 1.5 s;
    • everything else operational.

After merge (needs Axel / James)

  1. Axel: run vercel and then vercel --prod from apps/status. STATUS_API_URL defaults to https://api.hellonear.xyz/v1/status.
  2. James: add a CNAME status → cname.vercel-dns.com in the hellonear.xyz Cloudflare zone (DNS only, not proxied); Axel adds the domain in Vercel. This is a DNS change, so it needs Axel's approval (given).
  3. On the hosted box the table fills from the first probe round after the deploy, about 20 s after the backend starts.

Approved by Axel (PR and merge).

🤖 Generated with Claude Code

https://claude.ai/code/session_01Q5ZrzK8une3S2YyhDG746e

Lexirieru and others added 6 commits November 13, 2026 03:18
…ts providers

Two outages from outside our code in one night (13 Nov: NEAR AI embeddings
hung, 1Click quotes timed out) were found only by curl. apps/status is a
small Next.js page for Vercel that reads the backend's public
GET /v1/status: an overall banner, what is down and what users see because
of it, and per service 48 half-hour bars for the last 24 h with uptime.
Server-rendered and revalidated every 30 s, refreshed every minute while
visible; if the API can't be read it says so instead of showing all down.
Web app tokens, light and dark. STATUS_FIXTURE=1 shows the 13 Nov outages
for local work.

Tests: typecheck clean; node:test 3/3 (lib); next build passes. Checked in
Chrome on localhost with the fixture. CI workflow `status` added.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q5ZrzK8une3S2YyhDG746e
Axel asked for status.claude.com's hover card and Past Incidents. Each bar
now opens a small card: the time range (UTC), its state, about how long it
was bad (bad checks x 5 min), the last problem's detail and the number of
checks (optional bucket fields from the backend). Below the services, the
last 7 UTC days list the backend's incidents (runs of failed checks of one
service), Ongoing or Resolved with the duration; quiet days say so. The
card no longer clips the tooltip. All times are UTC and labelled, since the
page is rendered on the server.

Tests: typecheck clean; node:test 4/4. Checked in Chrome on localhost with
the fixture (hover card near the right edge, past incidents).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q5ZrzK8une3S2YyhDG746e
…e GET /v1/status

For status.hellonear.xyz. The backend probes 9 services in parallel every
5 minutes (HelloNear API and engine, NEAR AI chat and embeddings through
the engine's new /internal/v1/probes, 1Click dry quote, OutLayer, FastNear
RPC, NEAR AI Agent Market, Phala TEE Explorer), keeps 8 days in a new
service_checks table, and answers a public GET /v1/status (CORS *, cached
30 s): per service status, latency, impact, 24 h / 7 d uptime, 48 half-hour
buckets with checks/bad/detail, and incidents (runs of failed checks, last
7 days). A failing probe never stops the others; without the engine NEAR AI
reads unknown, not a guess. Thresholds per service in docs/api-contract.md.

Tests: be typecheck; test:unit 38/38; status.test.ts 6/6 (local Postgres,
fake upstreams: shape, CORS, order, statuses, incidents, cache, pruning);
full be suite 153 pass / 0 fail / 1 skipped before the last two contract
additions. engine typecheck; test:unit 191/191 (probes 3/3); 401 without
the key. Load: cached 500 req, 10 concurrent, 0 errors, p95 6.2 ms;
uncached worst case p95 970 ms. Live probe run 13 Nov: NEAR AI chat and
embeddings up again, 1Click quote down (500 after 25 s), the rest up.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q5ZrzK8une3S2YyhDG746e
A single failed check painted a whole half hour red. Now a bar is green
when every check was fine ("No downtime recorded in this half hour"),
yellow for a partial outage (some checks failed, or it was only slow) and
red for a major outage (every check failed), from the bucket's checks/bad.

Tests: typecheck clean; node:test 5/5.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q5ZrzK8une3S2YyhDG746e
… up its rows

The backend's "degraded" also covers a provider being down while HelloNear
answers, so "Some services are slow" was wrong (1Click was down, not slow):
it now reads "Partial outage: some services are having problems", and
"down" (HelloNear's own API or engine) reads "Major outage". The status
test now deletes its fake-upstream rows afterwards: they stayed in the dev
database and a local backend showed 4 ms 1Click answers.

Tests: status typecheck clean. The be status test runs before the PR (it
clears the table the local check reads).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q5ZrzK8une3S2YyhDG746e
Live now table and API reference (63 operations). The status row in Live
now points at the page that goes up on Vercel after this merge.

Docs only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q5ZrzK8une3S2YyhDG746e
@Lin1er

Lin1er commented Nov 12, 2026

Copy link
Copy Markdown
Contributor

@Lexirieru the status CI failure is the lockfile, not the code:

  • apps/status/bun.lock is "lockfileVersion": 2. The other four apps are on version 1, and every workflow (incl. the new status.yml) pins bun-version: 1.3.9, which can't read version 2: error: Unknown lockfile version → lockfile had changes, but lockfile is frozen (run).
  • Fix: regenerate it with the pinned version, e.g. cd apps/status && rm bun.lock && bunx bun@1.3.9 install, and commit the new bun.lock. Bumping only status.yml to a newer bun would also work, but then this app is built with a different bun than the others.

Backend part (status.ts, statusMath.ts, migration 016, server.ts, engine probes.ts), read in the diff, tests not run here: no blockers.

  • /v1/status is handled before the app's CORS and same-origin check, GET/HEAD only, and the details are fixed strings plus clipped upstream messages, so it leaks no key or URL.
  • The NEAR AI chat probe costs about 288 short calls a day (max_tokens: 32), small against the credit.
  • One small thing, not blocking: when the 30 s cache expires, several requests at once each run the 7-day query. That's fine at our traffic.

DNS after the merge: I'll tell James the exact record (CNAME status → cname.vercel-dns.com, DNS only).

Bun 1.4 on the laptop wrote lockfileVersion 2, which CI's Bun 1.3.9 can't
read ("Unknown lockfile version", frozen install failed). Regenerated with
bun@1.3.9; a frozen install passes with it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q5ZrzK8une3S2YyhDG746e
@Lexirieru
Lexirieru merged commit ed4a598 into main Nov 12, 2026
3 checks passed
@Lexirieru

Copy link
Copy Markdown
Contributor Author

Merged (ed4a598). All CI passed on the last commit (be, engine, status); the first status run failed on a Bun lockfile format, fixed in 73d78ec. @Lin1er the backend and engine deploy automatically. After that, the probe table on the box fills about 20 s after the backend starts.

Next:

  1. Axel: deploy apps/status to Vercel.
  2. @Lin1er: once Axel has the Vercel project, add a CNAME status → cname.vercel-dns.com in the hellonear.xyz Cloudflare zone (DNS only, not proxied). Axel approved it.

@AncungAulia @Nabil-Aufa FYI: status.hellonear.xyz will show whether HelloNear and its providers (NEAR AI, 1Click, OutLayer, RPC, Agent Market, TEE Explorer) are up, handy while you test #298.

@Lin1er

Lin1er commented Nov 12, 2026

Copy link
Copy Markdown
Contributor

Live on the hosted demo: main ed4a598 (automatic deploy after the merge).

  • Before: 039ccb3, and the box matched git at that revision. CI: passed: test,build,test.
  • GET /v1/health → {"ok":true,"engine":true,"revision":"ed4a598"}. No rollback.
  • Rollback target: 039ccb3 (/opt/hellonear/release-old).

@Lexirieru

Copy link
Copy Markdown
Contributor Author

@Lin1er the status page is on Vercel now (project hellonear-status, live at https://hellonear-status.vercel.app). status.hellonear.xyz is added to the project and verified there. Please add this one DNS record in the hellonear.xyz Cloudflare zone (Axel approved):

Type Name Target Proxy
CNAME status e658d8fd4c40ebb3.vercel-dns-017.com DNS only (grey cloud)

This is the exact record vercel domains verify asks for. It is project-specific, so use it instead of the generic cname.vercel-dns.com I wrote earlier. Keep the nameservers on Cloudflare; Vercel's suggestion to switch them is not needed. Once it resolves, Vercel issues the certificate by itself. Ping me and I'll check https://status.hellonear.xyz.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants