From 29b58d3994fe707f1b3b354066b78a3391ce7dd5 Mon Sep 17 00:00:00 2001 From: quantamixsol Date: Tue, 15 Sep 2026 08:28:09 +0200 Subject: [PATCH] CR-README-02: make organisational memory lead, and stop a third confidence leak MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Public port of private PR #343 (merged 2026-09-15, merge commit 2ac78d2b). All four files are byte-identical to the merged private/master (hash-verified). The 0.84.0 README asserted "organisational intelligence" in the hero and then reverted to code in every proof beneath it. A word count hid this: the first 70 lines scored 15 code-words to 19 org-words, apparently balanced, because the org words sat in the HERO while the STRUCTURE underneath was code-only: - "Why GraQle exists" opened "Your AI coding tool..." - the 90-second proof was `scan repo` -> a code question - documents lived in a sub-section titled "the knowledge that isn't code", framing them as the exception rather than the category - "How it works" step 1 said documents are "auto-linked to the code they describe", subordinating them - "The compounding advantage" opened "it knows your codebase" - "What teams use it for" was 6 code rows to 2 org rows RESTRUCTURE (both surfaces, kept in sync) - The 90-second proof now leads with documents and runs on a folder with NO CODE IN IT AT ALL. Every number was measured, not estimated: 3 policy/ADR files -> 12 nodes (3 DOCUMENT + 9 SECTION) + 9 SECTION_OF edges. Then `graq learn knowledge "vendor DPA must be signed before any data access" --domain policy` extracted the entity "DPA" and created 2 SEMANTICALLY_RELATED edges to the vendor-onboarding document AND to its "Due diligence" section — it found where the rule belonged unprompted. That is persistent, compounding organisational memory demonstrated with no repository present. - Code is now an explicit wedge, not the boundary: "### The same graph, for code" follows the documents proof and says so plainly. - "How it works" step 1 leads with documents/policies/ADRs; step 2 lists SECTION_OF and SEMANTICALLY_RELATED first. - "What teams use it for" reordered: policies, institutional memory and cross-source questions first. - After: first 70 lines score code 8 / org 34 (README) and 11 / 27 (PyPI); the first scan command on both surfaces is `graq scan docs`. THIRD ATTEMPT AT THE PLACEHOLDER CONFIDENCE LEAK `graq run` with no LLM configured printed "Confidence: 62%" directly beneath text reading "[NO LLM CONFIGURED - this is a placeholder response]". Attempt 1 (shipped in 0.84.0) fixed backends/mock.py - the ANSWER TEXT only. Attempt 2 guarded `def reason()`. But `graq run` dispatches to `def run()` at main.py:401, a DIFFERENT function with its own print site. The number kept printing, and the source-grep test written alongside PASSED anyway - a false green, asserting on strings present in the wrong function. Attempt 3 (this commit) guards run(), safety_check_command() and both reason() sites, verified by RUNNING the commands. backend_status is NOT usable here: it is only ever set to "failed" on an exception (core/graph.py), so the fallback path leaves it "ok". MockBackend.is_fallback is the signal that tracks the condition. The regression test asserts per-function and scopes its sweep to result.confidence / r.confidence. profile.confidence, env.confidence and rec.confidence are deliberately excluded - those are computed locally from the graph and are meaningful with no LLM configured. Narrowing the sweep rather than weakening the guard is the point. VERIFICATION ON THIS PUBLIC TREE - All 4 files hash-identical to merged private/master. - README lock gate (snapshot + eu_ai_act_docs_present): 62 passed. - tests/test_cli/test_run_placeholder_no_confidence.py: 8 passed. - Every graq command in both READMEs machine-diffed against `graq --help`: 0 invalid. - Forbidden-word scan: 0 violations in both files. NOT DONE HERE No version bump. 0.84.0 is merged to master but still UNTAGGED and PyPI still serves 0.83.0, so this copy and this fix ride into the same 0.84.0 release - PyPI never sees the weaker version. Co-Authored-By: Claude Opus 5 --- README.md | 78 ++++---- README_PYPI.md | 50 +++-- graqle/cli/main.py | 54 +++++- .../test_run_placeholder_no_confidence.py | 178 ++++++++++++++++++ 4 files changed, 302 insertions(+), 58 deletions(-) create mode 100644 tests/test_cli/test_run_placeholder_no_confidence.py diff --git a/README.md b/README.md index 981d297..93cdeb9 100644 --- a/README.md +++ b/README.md @@ -30,68 +30,80 @@ pip install graqle ## Why GraQle exists -Your AI coding tool is good at generating code. It is bad at remembering. +Your organisation already knows the answer. It just can't hand it to an AI. -Every session it reconstructs your system from whatever files fit in the context window. It has never read the architecture decision you made last March, the incident that made you move validation into the service layer, or the policy that says refunds above a threshold need manager approval. That knowledge exists — in your repo, your docs, your decision records, your team's heads — but it isn't connected to anything, so it can't be reasoned over. +The refund threshold lives in a policy document. The reason you retain records for seven years is in an ADR nobody re-reads. The incident that moved validation into the service layer is in someone's head. The dependency that makes payments fragile is in the code. Every one of those is real knowledge — and none of it is connected to any of the others, so no model can reason across it. -GraQle builds that connection once and keeps it. +Each AI session starts from zero and rebuilds a partial picture from whatever files fit in the context window. Then the window closes and the picture is gone. -- **Architecture, not files.** AI assistants see files. GraQle sees relationships, dependencies and blast radius. -- **Memory that compounds.** Lessons, decisions and documents become durable graph nodes instead of disappearing with a chat session. +GraQle builds that connection once, keeps it, and grows it. + +- **Relationships, not files.** Assistants see documents and files. GraQle sees how a policy, a decision and the code that implements it relate — and what breaks when one of them changes. +- **Memory that compounds.** Policies, decisions, lessons and architecture become durable graph nodes instead of disappearing with a chat session. Teach it once; every future session starts from there. - **Model independence.** Swap models, providers or IDEs without rebuilding the intelligence layer. --- -## 90-second proof +## 90-second proof — no code required + +Point GraQle at policies, ADRs, runbooks or specs. **Nothing else needed — this works on a folder with no code in it at all.** ```bash pip install graqle -# 1. Scan a codebase into a typed knowledge graph -graq scan repo . -# → functions, classes, modules, imports, calls — architecture mapped in seconds +# 1. Turn a folder of organisational documents into a typed graph +graq scan docs ./policies +# → 3 files → 12 nodes: 3 Document + 9 Section, linked by SECTION_OF -# 2. Ask an architectural question, not a file question -graq run "what breaks if I change the payment module?" -# → activates the relevant subgraph, traces cross-file call + import chains -# → returns: answer + confidence + evidence trail + active nodes +# 2. Teach it a rule that lives in nobody's file +graq learn knowledge "vendor DPA must be signed before any data access" --domain policy +# → extracts the entity "DPA", then SEMANTICALLY_RELATED-links the rule to +# the vendor-onboarding document AND to its "Due diligence" section + +# 3. Ask across the whole body of knowledge +graq run "what approval is needed for a large refund?" +# → answer + confidence + evidence trail + the exact sections consulted -# 3. Teach it something it cannot read from code -graq learn knowledge "payment module must never call user service directly" -# → persists as a graph node. Future reasoning activates this rule. +# 4. Audit what the organisation has taught it +graq learned ``` -That third command is the one that compounds. It is also the one no amount of prompt engineering replaces — it requires a persistent typed graph as the substrate. +Step 2 is the one that compounds, and the one no amount of prompt engineering replaces: it needs a persistent typed graph as the substrate. GraQle found where that rule belonged on its own — you never told it which document to attach it to. -### Bring in the knowledge that isn't code +Markdown, text, reStructuredText and AsciiDoc parse with the base install. PDF, DOCX, PPTX and XLSX need `pip install "graqle[docs]"` — without it those files are skipped and reported, never silently dropped. + +### The same graph, for code + +Where a codebase is part of the picture, it enters the same graph and connects to the documents that govern it: ```bash -pip install "graqle[docs]" # PDF / DOCX / PPTX / XLSX parsers +graq scan repo . +# → functions, classes, modules, imports, calls — architecture mapped in seconds + +graq run "what breaks if I change the payment module?" +# → traces cross-file call + import chains, activates the relevant subgraph -# Ingest architecture docs, policies, ADRs, runbooks, specs -graq scan docs ./docs -graq learn doc ./policies/ ./decisions/architecture-review.docx -# → Document + Section nodes, auto-linked to the code they describe +graq impact payments.py # blast radius before you touch anything ``` -Markdown, text, reStructuredText and AsciiDoc parse with the base install. PDF, DOCX, PPTX and XLSX need the `[docs]` extra — without it those files are skipped and reported, never silently dropped. +Software architecture is the deepest-mapped domain today — typed down to the function — and for engineering teams it is usually the fastest way to see the value. It is a wedge, not the boundary. --- ## The compounding advantage -The first time you run GraQle, it knows your codebase. After a month, it knows your patterns. After a year, it holds the architectural lessons, decisions and document context your team accumulated — and activates them on the change that is about to repeat an old mistake. +The first time you run GraQle, it knows what you gave it. After a month, it knows your patterns. After a year, it holds the policies, decisions, architectural lessons and document context your organisation accumulated — and activates the relevant ones on the work that is about to repeat an old mistake. -This is the part that survives model churn. When you switch from one provider to another, or from one IDE to another, the graph is unchanged. You are not re-teaching a new model what your system is; you are pointing a different model at intelligence you already own. +This is the part that survives model churn. When you switch provider or IDE, the graph is unchanged. You are not re-teaching a new model what your organisation knows; you are pointing a different model at intelligence you already own. -> **Own the intelligence your models and agents create.** Enterprises can own their data and still lose the reasoning state accumulated inside external AI tools. The graph is a local file you control. +> **Own the intelligence your models and agents create.** Enterprises can own their data and still lose the reasoning state accumulated inside external AI tools — the decisions, the corrections, the hard-won context. The graph is a local file you control. --- ## How it works -1. **Scan** → AST + dependency analysis builds a typed graph (functions, classes, modules, imports, calls). Documents and policies enter the same graph as Document and Section nodes, auto-linked to the code they describe. -2. **Connect** → Relationships become first-class: `IMPORTS`, `CALLS`, `DEFINES`, `SECTION_OF`. This is what makes cross-file reasoning possible. +1. **Scan** → Documents, policies, ADRs and specs become Document and Section nodes. Codebases enter the same graph through AST + dependency analysis (functions, classes, modules, imports, calls). One substrate, whatever the source. +2. **Connect** → Relationships become first-class: `SECTION_OF`, `SEMANTICALLY_RELATED`, `IMPORTS`, `CALLS`, `DEFINES`. Taught knowledge is auto-linked to the documents and code it concerns. This is what makes reasoning *across* sources possible. 3. **Activate** → A pre-reasoning layer scores each node for relevance, confidence and risk **before** the LLM runs, so the model receives the relevant subgraph instead of the whole repository. 4. **Reason** → Multiple agents debate. Outputs carry `confidence`, `graph_health`, `active_nodes` and evidence pointers. 5. **Validate** → Answers below the confidence floor are refused rather than guessed. @@ -143,13 +155,13 @@ Runs **fully offline** with Ollama or llama.cpp. Route different task types to d | Use case | Command | |:---|:---| +| **Policies, ADRs and specs into the graph** | `graq scan docs ./policies` · `graq learn doc ./decisions/` | +| **Institutional memory that outlives the session** | `graq learn knowledge "..."` · `graq learned` | +| **Ask across documents, decisions and code at once** | `graq run "what approval is needed above the refund limit?"` | +| Onboarding without a walkthrough | `graq run "how does checkout work end to end?"` | | Blast radius before a change | `graq impact payments.py` | | Cross-file security audit | `graq run "find every auth bypass risk"` | -| Architecture Q&A for onboarding | `graq run "how does checkout work end to end?"` | -| Institutional memory | `graq learn knowledge "..."` · `graq learned` | -| Policy + document context | `graq scan docs ./docs` · `graq learn doc ./policies/` | | Pre-change safety check | `graq preflight "refactor the auth layer"` | -| Combined risk read | `graq safety-check` | | CI/CD governance gate | `graq predict "..." --fail-below-threshold` | --- diff --git a/README_PYPI.md b/README_PYPI.md index 012222a..c817f60 100644 --- a/README_PYPI.md +++ b/README_PYPI.md @@ -1,42 +1,58 @@ -# GraQle — give your AI a memory of how your system actually works +# GraQle — give your AI a memory of how your organisation actually works -**Turn codebases, documents, policies and decisions into a persistent knowledge graph, so your AI agents reason over architecture and prior lessons instead of re-reading files every session.** +**Turn policies, decisions, documents and codebases into a persistent knowledge graph, so your AI agents reason over what your organisation already knows instead of rebuilding a partial picture every session.** ```bash pip install graqle ``` -Models change. Tools change. Your architecture and institutional knowledge should not. +Models change. Tools change. Your institutional knowledge should not. --- -## 60-second proof +## 60-second proof — no code required + +Works on a folder with no code in it at all. ```bash -# 1. Scan a codebase into a typed knowledge graph -graq scan repo . +# 1. Turn organisational documents into a typed graph +graq scan docs ./policies +# → 3 files → 12 nodes: 3 Document + 9 Section, linked by SECTION_OF -# 2. Ask an architectural question, not a file question -graq run "what breaks if I change the payment module?" -# → answer + confidence + evidence trail + active nodes +# 2. Teach it a rule that lives in nobody's file +graq learn knowledge "vendor DPA must be signed before any data access" --domain policy +# → auto-links the rule to the vendor-onboarding doc AND its "Due diligence" section + +# 3. Ask across the whole body of knowledge +graq run "what approval is needed for a large refund?" +# → answer + confidence + evidence trail + the sections consulted + +# 4. Audit what the organisation has taught it +graq learned +``` -# 3. Teach it what code cannot tell it -graq learn knowledge "payment module must never call user service directly" -# → persists in the graph. Future reasoning activates this rule. +Step 2 is the one that compounds — and the one prompt engineering cannot replace, because it needs a persistent typed graph as the substrate. GraQle worked out where that rule belonged on its own. + +### The same graph, for code + +```bash +graq scan repo . # functions, classes, imports, calls +graq run "what breaks if I change the payment module?" +graq impact payments.py # blast radius ``` -Step 3 is the one that compounds — and the one prompt engineering cannot replace, because it needs a persistent typed graph as the substrate. +Software architecture is the deepest-mapped domain today — a wedge, not the boundary. --- ## Why this matters now -Agents are getting far more capable, and still reconstruct your system from scratch every session. Models are becoming cheaper and interchangeable, which makes the intelligence layer above them — not the model itself — the thing worth owning. +Agents are getting far more capable and still start from zero every session. Models are becoming cheaper and interchangeable, which makes the intelligence layer above them — not the model itself — the thing worth owning. GraQle sits above the model: -- **Architecture, not files.** AI assistants see files. GraQle sees relationships, dependencies and blast radius. -- **Memory that compounds.** Lessons and decisions become durable graph nodes, not chat history. +- **Relationships, not files.** Assistants see documents and files. GraQle sees how a policy, a decision and the code implementing it relate. +- **Memory that compounds.** Policies, decisions and lessons become durable graph nodes, not chat history. - **Model independence.** Switch providers or IDEs without rebuilding the intelligence layer. --- @@ -65,7 +81,7 @@ graq scan docs ./docs # architecture docs, runbooks, specs graq learn doc ./policies/ # policies, ADRs, decision records ``` -Documents become Document and Section nodes, auto-linked to the code they describe. Markdown, text, RST and AsciiDoc work with the base install; the richer formats need the `[docs]` extra and are reported — never silently skipped — when it's missing. +Documents become Document and Section nodes, linked by `SECTION_OF` — and to any code that implements them. Markdown, text, RST and AsciiDoc work with the base install; the richer formats need the `[docs]` extra and are reported — never silently skipped — when it's missing. --- diff --git a/graqle/cli/main.py b/graqle/cli/main.py index 2d308b6..c39d663 100644 --- a/graqle/cli/main.py +++ b/graqle/cli/main.py @@ -523,7 +523,17 @@ def run( # Display results console.print("\n[bold green]Answer:[/bold green]") console.print(result.answer) - console.print(f"\n[dim]Confidence: {result.confidence:.0%} | " + # A placeholder answer must never carry a confidence figure. The fallback + # backend labels its text "NO LLM CONFIGURED", but printing a percentage + # underneath made an unconfigured install read like a governed answer. + # backend_status is NOT usable here: it is only ever set to "failed" on an + # exception (core/graph.py), so the fallback path leaves it at "ok". + _conf_display = ( + "not reported (no LLM configured)" + if getattr(backend, "is_fallback", False) + else f"{result.confidence:.0%}" + ) + console.print(f"\n[dim]Confidence: {_conf_display} | " f"Rounds: {result.rounds_completed} | " f"Nodes: {result.node_count} | " f"Cost: ${result.cost_usd:.4f} | " @@ -2381,7 +2391,14 @@ def safety_check_command( if not json_output: from rich.markup import escape as rich_escape console.print(f" {rich_escape(result.answer[:300])}") - console.print(f" [dim]Confidence: {result.confidence:.0%} | Cost: ${result.cost_usd:.4f}[/dim]") + # Same rule as `graq run`: no confidence figure on placeholder + # output from the silent no-backend-configured fallback. + _sc_conf = ( + "not reported (no LLM configured)" + if getattr(backend, "is_fallback", False) + else f"{result.confidence:.0%}" + ) + console.print(f" [dim]Confidence: {_sc_conf} | Cost: ${result.cost_usd:.4f}[/dim]") except Exception as exc: combined["reasoning"] = {"error": str(exc)[:200]} if not json_output: @@ -2742,10 +2759,18 @@ def reason( console.print(f"Q: [green]{rich_escape(q)}[/green]") console.print(f"A: {rich_escape(r.answer[:500])}") mode_color = "green" if r.reasoning_mode == "full" else "yellow" - console.print(f"[dim]Confidence: {r.confidence:.0%} | Cost: ${r.cost_usd:.4f} | " + # Same rule as the single-query path: no confidence figure + # on placeholder output from the fallback backend. + _conf = ("not reported (no LLM configured)" + if getattr(backend, "is_fallback", False) + else f"{r.confidence:.0%}") + console.print(f"[dim]Confidence: {_conf} | Cost: ${r.cost_usd:.4f} | " f"Mode: [{mode_color}]{r.reasoning_mode}[/{mode_color}][/dim]") + _avg = ("not reported (no LLM configured)" + if getattr(backend, "is_fallback", False) + else f"{avg_confidence:.0%}") console.print(f"\n[bold]Batch Summary:[/bold] {len(queries)} queries | " - f"Avg confidence: {avg_confidence:.0%} | " + f"Avg confidence: {_avg} | " f"Total cost: ${total_cost:.4f} | " f"Total latency: {total_latency:.0f}ms") return @@ -2799,10 +2824,23 @@ def reason( from rich.markup import escape as rich_escape console.print(f"\n[bold green]Answer:[/bold green] {rich_escape(result.answer)}") mode_color = "green" if result.reasoning_mode == "full" else "yellow" - console.print(f"[dim]Confidence: {result.confidence:.0%} | Rounds: {result.rounds_completed} | " - f"Nodes: {result.node_count} | Cost: ${result.cost_usd:.4f} | " - f"Latency: {result.latency_ms:.0f}ms | " - f"Mode: [{mode_color}]{result.reasoning_mode}[/{mode_color}][/dim]") + # A placeholder answer must never carry a confidence figure. The + # fallback backend already labels its text as "NO LLM CONFIGURED", + # but printing "Confidence: 62%" underneath made an unconfigured + # install read like a governed answer at a glance. backend_status + # is NOT usable here — it is only ever set to "failed" on an + # exception, so the fallback path leaves it "ok". + if getattr(backend, "is_fallback", False): + console.print(f"[dim]Confidence: not reported (no LLM configured) | " + f"Rounds: {result.rounds_completed} | " + f"Nodes: {result.node_count} | Cost: ${result.cost_usd:.4f} | " + f"Latency: {result.latency_ms:.0f}ms | " + f"Mode: [{mode_color}]{result.reasoning_mode}[/{mode_color}][/dim]") + else: + console.print(f"[dim]Confidence: {result.confidence:.0%} | Rounds: {result.rounds_completed} | " + f"Nodes: {result.node_count} | Cost: ${result.cost_usd:.4f} | " + f"Latency: {result.latency_ms:.0f}ms | " + f"Mode: [{mode_color}]{result.reasoning_mode}[/{mode_color}][/dim]") @app.command() diff --git a/tests/test_cli/test_run_placeholder_no_confidence.py b/tests/test_cli/test_run_placeholder_no_confidence.py new file mode 100644 index 0000000..e7d0efe --- /dev/null +++ b/tests/test_cli/test_run_placeholder_no_confidence.py @@ -0,0 +1,178 @@ +"""CR-README-02: a placeholder answer must never carry a confidence figure. + +The defect this pins: with no LLM backend configured, ``graq run`` printed + + [NO LLM CONFIGURED — this is a placeholder response, not real AI reasoning...] + Confidence: 62% | Rounds: 2 | Nodes: 5 | ... + +The answer text said "placeholder"; the summary line underneath said "62%". +In a screenshot or a terminal scrollback that is indistinguishable from a +governed answer, which undermines every decision-grade claim the project +makes about confidence and evidence. + +THIS TEST EXISTS BECAUSE TWO EARLIER FIXES BOTH MISSED. + + Attempt 1 fixed ``graqle/backends/mock.py`` — the backend's *answer text*. + The CLI's own summary line kept printing the number. + + Attempt 2 guarded the print sites inside ``def reason()``. But ``graq run`` + dispatches to ``def run()`` (main.py:401), a *different* function with its + own print site. The number kept printing, and a source-grep test written + at the same time passed anyway — a false green, because it asserted on + strings that existed in the wrong function. + +The lesson encoded here: assert on the *observable CLI behaviour* of each +command that prints confidence, not on the presence of source strings +somewhere in a 3,000-line module. + +Why not key on ``backend_status``: that field is only ever set to ``"failed"`` +on an exception (``graqle/core/graph.py``). The mock-fallback path leaves it +at its ``"ok"`` default, so it cannot distinguish a real answer from a +placeholder. ``MockBackend.is_fallback`` is the signal that tracks the +condition, and the bench command already fail-fasts on it. +""" + +from __future__ import annotations + +import asyncio +import pathlib +import re + +import pytest + +from graqle.backends.mock import MockBackend + +MAIN_PY = ( + pathlib.Path(__file__).resolve().parents[2] / "graqle" / "cli" / "main.py" +) + +#: Percentage-shaped confidence, e.g. "Confidence: 62%". +_CONF_PCT = re.compile(r"Confidence:\s*\d+\s*%") + + +# --------------------------------------------------------------------------- +# The signal itself +# --------------------------------------------------------------------------- + + +class TestFallbackBackendSignal: + def test_fallback_backend_reports_is_fallback(self) -> None: + assert MockBackend(is_fallback=True).is_fallback is True + + def test_explicit_mock_is_not_a_fallback(self) -> None: + """An explicitly-constructed mock is a legitimate test double. + + Only the silent no-backend-configured fallback suppresses the figure. + """ + assert MockBackend().is_fallback is False + + +# --------------------------------------------------------------------------- +# The backend's answer text (attempt-1 regression) +# --------------------------------------------------------------------------- + + +class TestPlaceholderAnswerText: + def test_fallback_text_carries_no_percentage(self) -> None: + result = asyncio.run(MockBackend(is_fallback=True).generate("anything")) + assert "NO LLM CONFIGURED" in result.text + assert not _CONF_PCT.search(result.text), ( + f"placeholder answer embeds a confidence figure: {result.text!r}" + ) + + def test_scripted_mock_still_reports_confidence(self) -> None: + """Guard against over-correcting: the scripted path is unchanged.""" + result = asyncio.run(MockBackend(is_fallback=False).generate("x")) + assert _CONF_PCT.search(result.text) + + +# --------------------------------------------------------------------------- +# Every CLI print site (attempt-2 regression) +# --------------------------------------------------------------------------- + + +def _function_source(name: str) -> str: + """Return the source of top-level ``def (`` up to the next def. + + Scoping the search to one function is the whole point: attempt 2 passed + a source-grep test while the defect lived in a different function. + """ + src = MAIN_PY.read_text(encoding="utf-8") + lines = src.splitlines() + start = next( + (i for i, ln in enumerate(lines) if ln.startswith(f"def {name}(")), None + ) + assert start is not None, f"def {name}( not found in main.py" + end = next( + ( + j + for j in range(start + 1, len(lines)) + if lines[j].startswith("def ") or lines[j].startswith("@app.command") + ), + len(lines), + ) + return "\n".join(lines[start:end]) + + +#: Every command function in main.py that prints a confidence figure and +#: creates its own backend. If a new one is added, add it here. +_COMMANDS_THAT_PRINT_CONFIDENCE = ["run", "reason", "safety_check_command"] + + +@pytest.mark.parametrize("func", _COMMANDS_THAT_PRINT_CONFIDENCE) +def test_every_confidence_print_is_guarded(func: str) -> None: + """No command may print raw ``result.confidence`` unguarded. + + Each function that displays a confidence figure must first consult + ``backend.is_fallback``. This catches the exact miss of attempt 2, where + one function was guarded and another was not. + """ + body = _function_source(func) + + if not _CONF_PCT.sub("", body) or "Confidence:" not in body: + pytest.skip(f"{func} no longer prints a confidence figure") + + assert 'getattr(backend, "is_fallback", False)' in body, ( + f"{func}() prints a confidence figure without consulting " + f"backend.is_fallback. An unconfigured install will show a " + f"percentage next to placeholder text." + ) + assert "not reported (no LLM configured)" in body, ( + f"{func}() has no no-LLM branch for its confidence display." + ) + + +#: Confidence values that originate from an LLM reasoning result. Only these +#: can be contaminated by the no-backend fallback. +#: +#: Deliberately NOT included: ``profile.confidence`` (domain detection), +#: ``env.confidence`` (environment provider detection) and ``rec.confidence`` +#: (strategy recommendation). Those are computed locally from the graph and +#: are meaningful with no LLM configured, so suppressing them would be wrong. +#: Scoping this sweep is the point — a blanket rule would have to be weakened +#: to pass, and a weakened guard is how this defect survived two fixes. +_REASONING_CONF = re.compile(r"\b(?:result|r)\.confidence:\.0%") + + +def test_no_unguarded_reasoning_confidence_remains() -> None: + """Every printed *reasoning* confidence sits inside an is_fallback branch. + + A raw percentage is acceptable only where the code has already tested + ``backend.is_fallback``, or in the JSON output path (machine-read, and it + carries ``backend_status`` alongside). + """ + lines = MAIN_PY.read_text(encoding="utf-8").splitlines() + offenders: list[tuple[int, str]] = [] + for i, ln in enumerate(lines): + if not _REASONING_CONF.search(ln): + continue + if '"confidence":' in ln: # JSON payload, not a rendered line + continue + window = "\n".join(lines[max(0, i - 14) : i + 1]) + if 'getattr(backend, "is_fallback", False)' in window: + continue + offenders.append((i + 1, ln.strip())) + assert not offenders, ( + "unguarded reasoning-confidence percentages at main.py lines: " + + "; ".join(f"{n}: {t[:70]}" for n, t in offenders) + )