[Aycode01]

Ops dashboard reads the same counters the metrics route exports

Closes #227

Summary

The ops dashboard and the metrics route each counted decisions independently, so an operator could look at /ops and see a different total from the one paging alerts use. This PR makes one in-process metrics snapshot feed both views.

• getMetricsSnapshot() now exposes a counters block (allow, deny, rateLimit, submitFailures).
• The same four numbers are exported in the Prometheus body, zero-initialised.
• The Ops tab loader (src/app/settings/page.tsx, reached via /ops) reads getMetricsSnapshot() on the server and hands it to src/components/ops-dashboard.tsx; the dashboard renders those counters.
• The four totals are never recomputed by scanning the audit table, and an empty snapshot renders zeroes instead of an endless spinner.

How it works

1. One snapshot, two consumers — src/lib/observability/metrics.ts

getOpsCounters() rolls the existing low-cardinality counters up into the four numbers operators act on:

Counter	Derivation	Rationale
allow	APPROVE + WARN	The two outcomes POST /api/decision lets through (it consumes usage for both).
deny	REQUIRE_APPROVAL + BLOCK	The two outcomes that stop execution.
rateLimit	rate-limit rejections	Requests rejected before any handler work ran.
submitFailures	horizon_failure, validation_failure, idempotency_conflict, source_wallet_mismatch	Every signed-submission result that did not settle on-chain. success and idempotency_replay are excluded — a replay is a successful no-op.

The grouping lives in exported constants (ALLOW_OUTCOMES, DENY_OUTCOMES, SUBMIT_FAILURE_RESULTS) so the roll-up is defined in exactly one place and is asserted against the existing allowlists in tests.

snapshot.counters is now part of the MetricsSnapshot type, which is what GET /api/metrics serialises and what the dashboard renders — the two cannot drift because they are the same object.

2. The scrape exports the same numbers — toPrometheusText()

# HELP fortexa_decisions_allowed_total Total decisions that were allowed to proceed
# TYPE fortexa_decisions_allowed_total counter
fortexa_decisions_allowed_total 2
# HELP fortexa_decisions_denied_total Total decisions that stopped the action
# TYPE fortexa_decisions_denied_total counter
fortexa_decisions_denied_total 1
# HELP fortexa_rate_limit_rejections_total Total requests rejected by the rate limiter
# TYPE fortexa_rate_limit_rejections_total counter
fortexa_rate_limit_rejections_total 0
# HELP fortexa_stellar_submit_failures_total Total Stellar submissions that did not settle on-chain
# TYPE fortexa_stellar_submit_failures_total counter
fortexa_stellar_submit_failures_total 0

These are always emitted (even at zero) so alert rules and Grafana panels do not break on a freshly restarted process. The fortexa_decision_outcomes_total{outcome=…} and fortexa_stellar_submit_results_total{result=…} families remain as the per-label source data; the four counters are derived roll-ups of them.

3. Rate-limit rejections are counted at the choke point — src/lib/security/rate-limit.ts

consumeRateLimit() records the rejection in the branch that returns ok: false, i.e. exactly where the 429 is produced. No route handler needed changing, and the counter cannot drift from the number of 429s a caller actually receives. It is reset by resetMetrics() alongside the other counters.

4. The dashboard renders the snapshot — src/components/ops-dashboard.tsx

• Accepts initialMetrics?: MetricsSnapshot and seeds its metrics state from it, so the counters (and totals/routes) are correct on first paint, before any client fetch.
• New counter row: Allowed, Denied, Rate limited, Submit failures, each with a hint describing exactly what it counts, plus a footer naming the source (GET /api/metrics).
• readCounters() falls back to a zeroed EMPTY_COUNTERS when there is no snapshot, or when a payload predates the counters block (rolling deploys), so a tile can never render Loading and never throws.
• The existing 8s poll of /api/metrics is unchanged: it keeps route/traffic data moving, and because the route returns the same snapshot shape the counters simply refresh.

The signed-TX card still reads the audit export — that is a different number and is unrelated to these four totals.

Tests

File	Coverage
src/components/ops-dashboard.snapshot.test.tsx (new)	Renders from a fixture snapshot and asserts zero fetch calls; asserts the rendered numbers equal snapshot.counters and equal the body GET /api/metrics exports (called in-process, no HTTP); empty snapshot → zeroes with no spinner in the counter section; pre-snapshot render → zeroes; mutating the in-process counters changes both the dashboard and the route body; counters are not attributable to the audit export.
src/app/settings/page.test.tsx (new)	The dashboard loader hands the in-process snapshot to the dashboard (ops tab), and renders zeroes when nothing is recorded.
src/lib/observability/metrics.test.ts	Roll-ups per group, rate-limit counter, replays/successes are not failures, counters are not derived from request buckets, Prometheus export matches the JSON snapshot, zero-initialised output, allow/deny groups stay inside the exported allowlists, resetMetrics() clears the new counter.
src/app/api/metrics/route.test.ts	JSON shape includes counters; Prometheus export equals the JSON body; zeroes before any activity.
src/lib/security/rate-limit.test.ts	A rejected bucket increments the metric (and only when capacity is actually exhausted).
src/components/ops-dashboard.test.ts	Existing refresh-behaviour tests updated for the new props signature and icon mock; assertions unchanged.

Documentation

• docs/observability.md: new metrics rows, counters in the JSON shape, PromQL for decision throughput / denied share / rate-limit rejections / submit failures, a "One snapshot, two views" subsection explaining the loader and the zero-rendering rule, and a caveat that the in-app dashboard is per-instance.
• README.md (§13), ARCHITECTURE.md (§8), docs/REVIEWER_EVIDENCE_MATRIX.md: ops dashboard counter list and the loader/evidence pointers.
• npm run check:docs passes (88 links, 0 broken).

Acceptance criteria

• A fixture snapshot appears on the dashboard with the same numbers the metrics body encodes — ops-dashboard.snapshot.test.tsx ("renders the counters carried by the snapshot…", "renders the same counters the metrics route exports").
• An empty snapshot renders zeroes — "renders zeroes for an empty snapshot instead of a spinner".
• The dashboard test does not scrape HTTP — the test asserts fetch was never called; /api/metrics is only invoked in-process to compare payloads.
• Changing the snapshot changes both views — "changes both views when the snapshot changes" records new decisions and asserts the dashboard markup and the route body both move from 0 to 1.

CI pre-checks (run locally, from a clean .next)

Check	Command	Result
Lint	npm run lint	0 errors (4 pre-existing warnings, none from this work)
Tests	npm run test	70 files, 597 tests passed
Build	npm run build	exit 0, compiled successfully, no type errors
Type check	npx tsc --noEmit	clean
Docs	npm run check:docs	0 broken links

⚠️ Also in this PR: pre-existing breakage that made main red

This branch would never have gone green on its own: main currently fails all three CI jobs. These defects come from earlier bad merges and are unrelated to the ops dashboard, so they are isolated in the first commit (fix(ci): repair pre-existing breakage blocking build, tests and lint) and can be reviewed/cherry-picked separately.

Syntax errors (nothing compiled):

• src/lib/auth/use-auth-session.ts:101 — the refresh async helper ended with }, []); instead of };. Neither tsc nor the bundler could parse the file; npm run build failed outright.
• src/lib/security/analyzer.ts — fetchBlocklistWithTimeout had an unclosed catch with a duplicated isTimeout binding and referenced an out-of-scope timeoutMs. Restored a catch that returns a blocked status (health-reported error first, then the thrown error) and sourced the timeout from getAnalyzerConfig().

Type errors (surfaced once the files parsed):

• src/lib/observability/logger.ts — export LogContext (validation/errors.ts imported a non-exported type) and widen its index signature to unknown so a flattened Zod error can be logged.
• src/lib/validation/errors.ts — normalise flattened field errors with Array.isArray (?? [] is a type error on Zod v4's flatten()).
• src/app/api/policy/rollback/preview/route.ts — log rawBody on validation failure; it referenced an undefined bodyResult.data.
• src/lib/readiness/production.ts, src/lib/stellar/network.ts — accept a partial env (ReadinessEnv / StellarEnv); Next 16 declares NodeJS.ProcessEnv with a required NODE_ENV, which made every partial-env call site a type error.
• src/app/api/audit/export/route.test.ts — removed a duplicated response declaration left by a bad merge.
• src/app/api/stellar/submit-signed/route.test.ts — typed importOriginal() so the partial mock can be spread.
• tsconfig.json — target ES2017 → ES2020, required by the BigInt literals in src/lib/stellar/stroops.ts (issue [SCF high][easy] Parse policy limits without floating-point drift #178).

Tests/lint correctness:

• analyzer.test.ts asserted blocklistStatus === "timeout", but commit ea9da4f deliberately changed timeouts to report "error" with the blocklistTimedOut flag. The test now asserts the current contract (it had never run, because the file did not parse).
• metrics.test.ts — @ts-expect-error directives now carry descriptions, as ban-ts-comment requires.

One existing test assertion was deliberately narrowed: route.test.ts asserted that every fortexa_* line carries route/method labels. That is only true while every exported series is a route bucket; it now checks the three request-bucket families explicitly (the new roll-up counters are intentionally label-free).

Reviewer notes

• No new dependencies, no API contract removal. /api/metrics gains a counters object — additive.
• Cardinality is unaffected: no new labels are introduced anywhere.
• Security posture is unchanged; the new counter is a plain integer incremented at a single point.
• Known limitation, documented in docs/observability.md: the dashboard reads the snapshot of the instance that served the page, so in a multi-replica deployment it shows one replica. Prometheus remains the fleet-wide source of truth — this is pre-existing behaviour for the totals/routes too, not something this PR changes.

[drips-wave]

@Aycode01 Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits