fix(inspection): stage 3 content classification was dead when AGT is installed - #477
fix(inspection): stage 3 content classification was dead when AGT is installed#477imran-siddique wants to merge 2 commits into
Conversation
…installed `_classify_sensitivity` called `_agt_redactor.find_credentials(response_text)`. That method has never existed on agt-core's `CredentialRedactor`. Verified against the pinned 4.1.0 wheel and against every commit in AGT's history: the real API is `find_matches` (secrets) and `find_pii_matches` (PII spans). The call raised `AttributeError` straight into a bare `except Exception: pass`, and because the local `_PII_PATTERNS` sweep sat in the `else` branch of the `_AGT_AVAILABLE and _agt_redactor is not None` check, it never ran either. So whenever agent-os was importable, source 3 of stage 3 contributed nothing: no response ever received a content-derived sensitivity tag, and classification silently degraded to catalog annotations alone. With agent-os absent the fallback worked, which is why the existing tests, all written on the no-AGT path, stayed green. Changes: - Call `find_matches` and `find_pii_matches`, resolved with `getattr` so a future upstream rename degrades to a logged warning rather than a silent no-op. - Run the local patterns as a second pass rather than an either/or. Besides covering an AGT-side failure, this is the defence against microsoft/agent-governance-toolkit#3494, where a secret with a suffix glued to it (`AKIA..._old`) is not redacted at all. That issue's fix PR was closed unmerged on 2026-08-05, so it is unfixed upstream with no release pending. - Replace the blanket `except Exception: pass` with a narrowed handler that logs. Swallowing everything is what hid this. Three regression tests, each failing on the previous implementation and passing here: AGT missing the method, AGT raising, and the #3494 suffix case. Note: cmcp is not exposed to microsoft/agent-governance-toolkit#3496. That defect is in `MCPResponseScanner.sanitize_response`, reached only under `ResponsePolicy.SANITIZE`. `proxy.py` leaves the default `BLOCK`, and the pipeline calls `scan_response`, not `sanitize_response`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Imran Siddique <imran.siddique@opaque.co>
Follow-on from the dead `find_credentials` call in the previous commit. Audited every AGT symbol and method cmcp calls against the pinned agt-core 4.1.0 wheel. `find_credentials` was the only phantom API; the rest resolve, including the `GovernancePolicy` re-export in `proxy.py`, which is a real module-level import from `agent_os.integrations.base` and works at runtime despite the type: ignore. What the audit did find is the handling around those calls. 1. All three AGT components were constructed in one try block, so a failure building the first left the other two unbuilt and all three silently None. One upstream API change would disable three security components at once with no log line. Each is now constructed independently and logs on failure. 2. `PromptInjectionDetector.detect()` failing fell through to the local patterns, which is correct, but silently. A broken detector was indistinguishable from a working one while the weaker starter set was what actually ran. Now logged with the pattern-set version that took over. 3. `MCPResponseScanner.scan_response()` failing was swallowed entirely. It correctly does not deny, since an errored scanner has produced no verdict and stage 4's own detection still runs, but losing MCP-specific threat coverage for a response should not be invisible. Now logged with the tool name. `catalog/scanner.py` already did this correctly and is unchanged: it logs, sets an explicit `_available = False`, and distinguishes not-installed from failed. Two tests, both failing on the previous commit: one pinning that a failing component no longer takes the others down, one asserting the warning is emitted. Not changed, flagged for review instead: when AGT's detector returns a clean verdict, stage 4 returns allow immediately and the local pattern set never runs. Making both run would be more conservative, but it changes deny behaviour and risks false positives in a gateway, so it wants a deliberate decision rather than being folded into a robustness fix. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Imran Siddique <imran.siddique@opaque.co>
Second commit: the handlers that hid the first bugAudited every AGT symbol and method cmcp calls against the pinned agt-core 4.1.0 wheel, since a phantom method surviving in Good news on the API surface.
On What the audit did find is the error handling around those calls, all in
Two more tests, both verified failing on the previous commit. 58 pass across One thing deliberately not changedWhen AGT's detector returns a clean verdict, stage 4 returns Running both and denying if either denies would be more conservative and consistent with the stage 3 fix in the first commit. But it changes deny behaviour and risks false positives in a gateway, and |
What this fixes
_classify_sensitivitycalled_agt_redactor.find_credentials(response_text). That method has never existed on agt-core'sCredentialRedactor. I checked the pinned 4.1.0 wheel and every commit in AGT history; the real API isfind_matches(secrets) andfind_pii_matches(PII spans).The
AttributeErrorwent into a bareexcept Exception: pass, and the local pattern sweep sat in theelsebranch, so it did not run either. Whenever agent-os was importable, source 3 of stage 3 contributed nothing. No response received a content-derived sensitivity tag; classification silently fell back to catalog annotations alone.The existing tests all patch
_AGT_AVAILABLEtoFalse, so they exercised the fallback and stayed green throughout.Changes
find_matchesandfind_pii_matches, resolved viagetattrso a future upstream rename degrades to a logged warning instead of a silent no-op._PII_PATTERNSas a second pass rather than an either/or.except Exception: passwith a narrowed, logging handler. Swallowing everything is what hid this.Why the second pass, and not just the corrected method name
microsoft/agent-governance-toolkit#3494: AGT's redactor does not redact a secret at all when a suffix is glued to it, so
AKIAIOSFODNN7EXAMPLE_oldpasses through whole. That issue's fix PR was closed unmerged on 2026-08-05 and there is no release pending, so relying on AGT alone leaves a live key untagged.Tests
Three regression tests, each verified failing on the previous implementation and passing here:
test_local_patterns_still_run_when_agt_lacks_the_methodtest_local_patterns_still_run_when_agt_raisestest_suffixed_secret_still_tagged_despite_agt_3494_oldsuffix casetests/unit/test_stage3_sensitivity.pyandtests/unit/test_inspection.py: 56 passed.ruffclean,mypyclean.Not affected
cmcp is not exposed to #3496. That defect is in
MCPResponseScanner.sanitize_response, reached only underResponsePolicy.SANITIZE.proxy.pyleaves the defaultBLOCK, and the pipeline callsscan_response.Note on the local test environment
pytest tests/unitshows 21 collection errors on this branch and on pristinemainalike. They come from a dev checkout of AGT shadowing the pinned agt-core 4.1.0, soagent_os.policies.backendsandmcp_gateway.GovernancePolicyare absent. That is #472 reproduced locally, and worth noting there: on 5.x,proxy.pylosesGovernancePolicyas well as the evaluator losingCedarBackend.Refs #471, #472.