Repository navigation
feat(scenarios): run seed catalog through the decision engine and fail on drift (#216) - #288
Merged
karagozemin merged 2 commits intoOct 1, 2026
Conversation
…l on drift (Fortexa-stellar#216) The scenario catalog, the seed data, and the decision engine could change separately, so a scenario card could still show "deny" while the engine allows - a broken demo of the firewall. - Add src/lib/scenarios/evaluate.ts: a deterministic engine harness for the seeded catalog (default policy minus the allowed-hours window, seeded usage snapshot) plus validateScenarioPack(), which reports missing expected decisions and seed-vs-engine drift with engine rule ids in the message. - Extend scenario-pack.test.ts with a drift guard: every seed entry is evaluated through the engine, a mutated expected decision fails the suite, a seed entry without an expected decision fails the suite, and the suite proves it never contacts Horizon or any other network (fetch is stubbed to reject and FORTEXA_BLOCKLIST_URL is cleared). - The scenarios page renders engine results instead of hard-coded labels: settings/page.tsx evaluates the catalog server-side and passes the evaluations to ScenariosCatalog; badges and the decision filter use the engine decision, and each card gains an "Engine decision" row. - scripts/demo-scenarios.ts reuses the harness and exits non-zero when any seed expectation drifts from engine output. - Repair pre-existing parse errors that blocked the suite: restore the mangled fetchBlocklistWithTimeout in src/lib/security/analyzer.ts (timeout and degradation semantics per analyzer.test.ts), restore a stripped useCallback wrapper in src/lib/auth/use-auth-session.ts, and remove a duplicated const left behind in src/app/api/audit/export/route.test.ts.
|
@ollypee22 Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits. You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #216
The scenario catalog, the seed data, and the decision engine can change separately. Until now a scenario card could still show a stale expected label (e.g. "APPROVE") while the engine actually decided otherwise - a broken demo of the firewall. This PR makes the engine the single source of truth: the catalog is evaluated through the decision engine, drift fails the suite, and the scenarios page renders the engine's decision instead of a hard-coded label.
Changes
1. Engine harness for the catalog -
src/lib/scenarios/evaluate.ts(new)evaluateScenarioCatalog()loads every scenario from the seed module and runs it throughevaluateDecisionusing a deterministic policy (default policy with theallowedHourswindow removed, so results don't depend on the wall clock) and the seeded daily-usage snapshot.evaluateScenario(id)evaluates a single seeded scenario and returns{ expected, actual, result, matches }including the full engine result (triggered policies, risk findings, risk score, analyzer status).validateScenarioPack()reports two kinds of issues:missing-expected-decision- a seed entry with no expected decision fails (required by the issue), anddecision-mismatch- seed vs engine drift, with the engine's triggered rule ids and finding codes embedded in the failure message so the diff is debuggable at a glance.2. Drift guard tests -
src/lib/scenarios/scenario-pack.test.tssafe-research-paymenttoBLOCK) is proven to fail the suite with a drift message.globalThis.fetchto reject and clearsFORTEXA_BLOCKLIST_URL, proving the whole catalog evaluation never contacts Horizon or any other endpoint.3. Scenarios page renders engine results
src/app/settings/page.tsxevaluates the catalog server-side (only when the scenarios tab is active) and passes evaluations to the catalog.src/components/scenarios-catalog.tsxnow rendersDecisionBadgefrom the engine's decision (falling back to the seed label only if no evaluation exists), filters by the engine decision, and adds an "Engine decision" row to each card.data-scenario-id/data-engine-decisionattributes make assertions unambiguous.4. Demo runner fails on drift -
scripts/demo-scenarios.tsReuses the shared harness and exits non-zero when any seed expectation no longer matches engine output (verified: flipping one seed expectation produces exit code 1 with a drift report; restoring it returns exit code 0).
5. Repairs to broken pre-existing files on
mainThese blocks were unrelated to the feature but prevented the suite from running at all (parse errors), so they are fixed here:
src/lib/security/analyzer.ts:fetchBlocklistWithTimeoutwas syntactically mangled (unbalanced braces, undefinedtimeoutMs, out-of-scopeblocklist); restored with the timeout/degradation semantics its ownanalyzer.test.tsasserts (includingblocklistStatus: "timeout"and degradation reasons).src/lib/auth/use-auth-session.ts: a strippeduseCallbackwrapper leftrefreshwith an orphaned}, []);- restored the wrapper.src/app/api/audit/export/route.test.ts: removed a duplicatedconst response = await GET(request);line (merge debris referencing a nonexistent variable).Acceptance criteria mapping
scenario pack drift guard > matches every seeded scenario decision against the engine).renders the engine decision for a fixture scenario instead of the seed label- seed APPROVE vs engine REQUIRE_APPROVAL is rendered).Validation
npx vitest run: 68 files / 584 tests passing.npx tsc --noEmit: no errors in any changed file (remaining pre-existing errors elsewhere are untouched by this PR; the two baseline parse errors are fixed by it).npx eslinton all changed files: clean.npx tsx scripts/demo-scenarios.ts: exit 0 on match, exit 1 when a seed expectation is mutated.