fix: deterministic demo, LLM fallback, engine provenance, pricing/ROI footer - #4
Merged
Merged
Conversation
Replace Date.now() PRNG seeding in demoAnthropic/demoOpenAI with a fixed default seed (DEMO_SEED = 20260717) accepted as a parameter, so one demo run generates one coherent org, a real 6-month trend, and an identical hero savings number on every run. Add a determinism test asserting two full 6-month demo runs produce byte-identical reports.
findIssuesLLM now returns findings plus the NIM call's own token usage
(with an estimated cost for the ROI footer). New analyzeWithFallback wrapper
runs the LLM path and, on any failure, falls back to the deterministic rule
engine and stamps a visible notice on the report. Reports gain optional
engine ('rules' | 'llm'), notice, and llmUsage fields; the home page stamps
them everywhere it builds a report and threads DEMO_SEED through demo calls.
Add a test forcing the LLM call to throw, asserting the rules ran and the
notice is set.
Recommendations page: engine badge in the report header ('Rule engine' /
'LLM analysis'), a banner when the report carries a fallback notice, an
always-visible 'Prices as of <date>' footer line (the 90-day staleness
warning stays), and an ROI footer line on LLM-engine reports comparing the
analysis token cost to the identified monthly savings. Month-fetch paths on
the recommendations and analytics pages stamp engine: 'rules'.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Demo-hardening fixes from the v2 plan (#3): the demo never depends on a live key, the rules keep computing the money, and every report says which engine produced it.
DEMO_SEED = 20260717) threaded through alldemoAnthropic/demoOpenAIcalls instead ofDate.now(), so one demo run is one coherent org with a stable 6-month trend and hero savings number. Vitest test asserts two full 6-month runs produce byte-identical reports.analyzeWithFallbackwrapsfindIssuesLLM; any failure falls back to the deterministic rule engine and stamps a visible "LLM unavailable, showing deterministic analysis" notice on the report. Test forces the LLM call to throw and asserts the rules ran and the notice is set.engine: "rules" | "llm"added to the stored report, set at every report-producing site (home analysis, demo, recommendations month-fetch, analytics year-fetch, and the Step 2 fallback), and shown as a badge in the recommendations report header. Toggle behavior unchanged (LLM still replaces rules when on).usagefield is now parsed, costed, and stored on the report; LLM-engine reports render a footer line comparing the analysis token cost to the identified monthly savings. Omitted on rules-engine reports.Verification:
npm run type-check,npm run lint,npm run format:check,npm testall pass — 52 tests (46 existing + 6 new).