feat: rework analysis into a rules + LLM consensus engine - #5
Merged
Conversation
…ings merge with provenance
The AI toggle no longer swaps engines: rules always run, the LLM augments.
The LLM emits only {rowId, category, severity, confidence, reason, action}
from a fixed category enum; every proposal is priced by the shared costing
module (uncostable categories are dropped, org/quality ones keep savings 0).
Findings merge by (row, category) into source rules/llm/both, reports gain
engine "hybrid", and LLM failure degrades to rules-only with a notice.
…indings Rule signals now carry human-readable labels stored on each finding; the report row detail renders a "Based on:" trail (unmet signals dimmed), consensus findings get a "Confirmed by AI" tag, and AI-spotted findings show the row metrics the deterministic pricing used.
Replaces the 2-workspace random-scenario generator with 8 named workspaces plus an overloaded default, each running a distinct workload (caching miss, Haiku downgrade, RAG bloat, Monday-spike batch, Opus overkill, legacy model, cache-write waste, quiet staging) so every rule category fires in a demo run. DEMO_SEED mechanism unchanged — one seed, one coherent org, byte-identical runs. Demo hero numbers change as accepted.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Reworks the analysis architecture so that code prices everything and the LLM only detects and explains. Previously the NIM toggle replaced the rule engine with an LLM that emitted
savingsMonthly— the thing it's worst at, requiring guardrail clamps — while its real strengths (fuzzy detection, workload characterization) added nothing because the rules were discarded.New architecture
Shared costing modules (
src/lib/anthropic/costing.ts,src/lib/openai/costing.ts): every rule's optimized-cost formula extracted into per-vendor functions plus a category dispatcher(category, row metrics) → optimized cost | null. The rule engines call these functions; output is numerically identical to before.LLM proposes, never prices: the NIM prompt now asks for
{rowId, category, severity, confidence, reason, action}only — no savings field of any kind.categorymust come from the fixed enum (listed verbatim in the prompt). Each proposal is priced by the costing dispatcher; uncostable categories are dropped, except zero-savings org/quality categories which survive at $0. Tier-sanity and one-downgrade-per-row guardrails kept; the savings clamps are deleted — there is no LLM number to clamp.Consensus merge: with the AI toggle on, BOTH engines run — rules first (synchronous), then the LLM. Findings merge by (row id, category):
source: "both",conf = min(0.95, max(ruleConf, llmConf) + 0.1)source: "rules", unchangedsource: "llm", LLM's text, deterministic priceReports gain
engine: "hybrid";"rules"and"llm"remain valid so stored reports still parse. LLM failure degrades to rules-only findings with the existing fallback notice — the report is never empty. Toggle relabeled to AI-augmentation language; each Row shows a per-finding source badge, and AI-spotted findings are badged "priced deterministically from your data".Explainability: rule signals are now
{weight, met, label}with short human labels, stored on findings, rendered as a "Based on:" trail (unmet signals dimmed). Consensus findings get a "Confirmed by AI" tag; LLM-only findings show the row metrics the pricing used.Demo org: grown from 2 to 8 workspaces with a messy, realistic distribution — an overloaded default workspace (triggers the org-structure finding) plus per-workspace patterns (caching miss, Haiku downgrade, RAG bloat, Monday-spike batch, Opus overkill, legacy model, cache-write waste, quiet staging) so every rule category fires in a demo run.
DEMO_SEEDmechanism unchanged: one seed, one coherent org, byte-identical runs.Steps
source;engine: "hybrid"; graceful degradation; UI relabel + badges)Tests
84 tests pass (was 52). New coverage: each costing function verified against the rule engine's own output plus dispatcher null-cases; consensus merge (both / rules-only / llm-only, confidence-boost cap, uncostable-category drop, org-finding merge across id shapes); LLM failure degrading to rules with the notice; assertions that neither the system prompt nor the request payload contains any savings field; demo determinism, 8-workspace shape, and full category coverage.
Verification
npm ci&&npm run type-check&&npm run lint&&npm run format:check&&npm test— all pass. No new dependencies; no live API calls (LLM path tested with mocked responses only).