Skip to content

feat(scenarios): run seed catalog through the decision engine and fail on drift (#216) - #288

Merged
karagozemin merged 2 commits into
Fortexa-stellar:mainfrom
ollypee22:feat/issue-216-scenario-engine-drift
Oct 1, 2026
Merged

karagozemin merged 2 commits into
Fortexa-stellar:mainfrom
ollypee22:feat/issue-216-scenario-engine-drift

Conversation

@ollypee22

Copy link
Copy Markdown
Contributor

Summary

Closes #216

The scenario catalog, the seed data, and the decision engine can change separately. Until now a scenario card could still show a stale expected label (e.g. "APPROVE") while the engine actually decided otherwise - a broken demo of the firewall. This PR makes the engine the single source of truth: the catalog is evaluated through the decision engine, drift fails the suite, and the scenarios page renders the engine's decision instead of a hard-coded label.

Changes

1. Engine harness for the catalog - src/lib/scenarios/evaluate.ts (new)

  • evaluateScenarioCatalog() loads every scenario from the seed module and runs it through evaluateDecision using a deterministic policy (default policy with the allowedHours window removed, so results don't depend on the wall clock) and the seeded daily-usage snapshot.
  • evaluateScenario(id) evaluates a single seeded scenario and returns { expected, actual, result, matches } including the full engine result (triggered policies, risk findings, risk score, analyzer status).
  • validateScenarioPack() reports two kinds of issues:
    • missing-expected-decision - a seed entry with no expected decision fails (required by the issue), and
    • decision-mismatch - seed vs engine drift, with the engine's triggered rule ids and finding codes embedded in the failure message so the diff is debuggable at a glance.

2. Drift guard tests - src/lib/scenarios/scenario-pack.test.ts

  • Every current seed is evaluated through the engine and compared to its expected decision; any mismatch fails with a per-scenario drift report.
  • A mutated expected decision (fixture flips safe-research-payment to BLOCK) is proven to fail the suite with a drift message.
  • A seed entry without an expected decision is proven to fail the suite.
  • No network: the suite stubs globalThis.fetch to reject and clears FORTEXA_BLOCKLIST_URL, proving the whole catalog evaluation never contacts Horizon or any other endpoint.

3. Scenarios page renders engine results

  • src/app/settings/page.tsx evaluates the catalog server-side (only when the scenarios tab is active) and passes evaluations to the catalog.
  • src/components/scenarios-catalog.tsx now renders DecisionBadge from the engine's decision (falling back to the seed label only if no evaluation exists), filters by the engine decision, and adds an "Engine decision" row to each card. data-scenario-id / data-engine-decision attributes make assertions unambiguous.

4. Demo runner fails on drift - scripts/demo-scenarios.ts

Reuses the shared harness and exits non-zero when any seed expectation no longer matches engine output (verified: flipping one seed expectation produces exit code 1 with a drift report; restoring it returns exit code 0).

5. Repairs to broken pre-existing files on main

These blocks were unrelated to the feature but prevented the suite from running at all (parse errors), so they are fixed here:

  • src/lib/security/analyzer.ts: fetchBlocklistWithTimeout was syntactically mangled (unbalanced braces, undefined timeoutMs, out-of-scope blocklist); restored with the timeout/degradation semantics its own analyzer.test.ts asserts (including blocklistStatus: "timeout" and degradation reasons).
  • src/lib/auth/use-auth-session.ts: a stripped useCallback wrapper left refresh with an orphaned }, []); - restored the wrapper.
  • src/app/api/audit/export/route.test.ts: removed a duplicated const response = await GET(request); line (merge debris referencing a nonexistent variable).

Acceptance criteria mapping

  • ✅ Every current seed matches the engine (scenario pack drift guard > matches every seeded scenario decision against the engine).
  • ✅ A mutated expected decision fails the test (dedicated negative test + verified against the demo runner).
  • ✅ The page test shows the engine's decision for a fixture scenario (renders the engine decision for a fixture scenario instead of the seed label - seed APPROVE vs engine REQUIRE_APPROVAL is rendered).
  • ✅ The suite does not contact Horizon (fetch stub rejects; blocklist URL cleared during the drift guard).

Validation

  • npx vitest run: 68 files / 584 tests passing.
  • npx tsc --noEmit: no errors in any changed file (remaining pre-existing errors elsewhere are untouched by this PR; the two baseline parse errors are fixed by it).
  • npx eslint on all changed files: clean.
  • npx tsx scripts/demo-scenarios.ts: exit 0 on match, exit 1 when a seed expectation is mutated.

…l on drift (Fortexa-stellar#216)

The scenario catalog, the seed data, and the decision engine could change
separately, so a scenario card could still show "deny" while the engine
allows - a broken demo of the firewall.

- Add src/lib/scenarios/evaluate.ts: a deterministic engine harness for the
  seeded catalog (default policy minus the allowed-hours window, seeded usage
  snapshot) plus validateScenarioPack(), which reports missing expected
  decisions and seed-vs-engine drift with engine rule ids in the message.
- Extend scenario-pack.test.ts with a drift guard: every seed entry is
  evaluated through the engine, a mutated expected decision fails the suite,
  a seed entry without an expected decision fails the suite, and the suite
  proves it never contacts Horizon or any other network (fetch is stubbed to
  reject and FORTEXA_BLOCKLIST_URL is cleared).
- The scenarios page renders engine results instead of hard-coded labels:
  settings/page.tsx evaluates the catalog server-side and passes the
  evaluations to ScenariosCatalog; badges and the decision filter use the
  engine decision, and each card gains an "Engine decision" row.
- scripts/demo-scenarios.ts reuses the harness and exits non-zero when any
  seed expectation drifts from engine output.
- Repair pre-existing parse errors that blocked the suite: restore the
  mangled fetchBlocklistWithTimeout in src/lib/security/analyzer.ts (timeout
  and degradation semantics per analyzer.test.ts), restore a stripped
  useCallback wrapper in src/lib/auth/use-auth-session.ts, and remove a
  duplicated const left behind in src/app/api/audit/export/route.test.ts.
@drips-wave

drips-wave Bot commented Sep 30, 2026

Copy link
Copy Markdown

@ollypee22 Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits

@karagozemin
karagozemin merged commit 1923d95 into Fortexa-stellar:main Oct 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Run the scenario catalog through the decision engine and fail on drift

2 participants