From 102901bbf2c848a1005f6de0a3e0f8b7e2292d64 Mon Sep 17 00:00:00 2001 From: ericksondelacruz Date: Mon, 19 Oct 2026 23:02:39 +0800 Subject: [PATCH 1/2] feat(rules): detect missing EBTE tool-call claims and mediation (schema 27) New claim_use discovery fact on Python decorated tools that declare a claim parameter (unused / passthrough / checked / wrapped; an OpenAI tool input guardrail counts as a possible mediator) and a tool_claim_use predicate, with a loader check on its values. SupportedSchemaVersion 26 -> 27. Fixture mirror of nine new rules, a ladder per SDK (OpenAI Agents, MCP, LangChain; Python): OAI-034/MCP-033/LC-028 side-effecting tool with no claim, OAI-035/MCP-034/LC-029 claim never read, OAI-036/MCP-035/LC-030 claim only printed, logged, serialized or returned. --- ARCHITECTURE.md | 21 +- CHANGELOG.md | 19 ++ CLAUDE.md | 8 + COVERAGE.md | 7 +- internal/analysis/claim_use.go | 180 ++++++++++++++++ internal/analysis/claim_use_test.go | 198 ++++++++++++++++++ internal/analysis/discovery.go | 7 + internal/models/models.go | 24 +++ internal/rules/evaluator.go | 22 ++ internal/rules/loader.go | 7 + internal/rules/policies_test.go | 186 ++++++++++++++++ internal/rules/predicates.go | 43 ++++ internal/rules/predicates_claim_test.go | 75 +++++++ internal/rules/schema.go | 4 + internal/rules/schema.yaml | 15 ++ internal/rules/schema_version.go | 2 +- internal/scanner/claims_scan_test.go | 98 +++++++++ .../langchain/tool_call_claims.yaml | 138 ++++++++++++ testdata/rules-fixture/manifest.yaml | 2 +- .../rules-fixture/mcp/tool_call_claims.yaml | 141 +++++++++++++ .../openai_sdk/tool_call_claims.yaml | 141 +++++++++++++ 21 files changed, 1332 insertions(+), 6 deletions(-) create mode 100644 internal/analysis/claim_use.go create mode 100644 internal/analysis/claim_use_test.go create mode 100644 internal/rules/predicates_claim_test.go create mode 100644 internal/scanner/claims_scan_test.go create mode 100644 testdata/rules-fixture/langchain/tool_call_claims.yaml create mode 100644 testdata/rules-fixture/mcp/tool_call_claims.yaml create mode 100644 testdata/rules-fixture/openai_sdk/tool_call_claims.yaml diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 3176b447..4e062f51 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -495,6 +495,17 @@ For each language recon cleared, do the AST work and produce a `RepoInventory`: - **DiscoverTools** (`discovery.go`) — two-pass Python discovery (decorated functions and bare shell-invoking functions). Also captures decorator kwargs into `ToolDef.Config` for rules that inspect `@function_tool(strict_mode=...)`. + For a decorated tool that declares a parameter named `claim` (an EBTE-style + `ToolInvocationExplanation`), `pythonClaimUse` (`claim_use.go`) stamps the + `claim_use` fact (`models.FactClaimUse`): `unused` (the body never reads it), + `passthrough` (every read is an argument to print / a logger method / + `json.dumps` / `str` / `repr`, or part of a return value), `checked` (some read + feeds another call, a comparison or a condition), or `wrapped` (a second + decorator, or for an OpenAI tool an attached tool input guardrail, may + mediate). A tool with no `claim` parameter carries no fact, so scan output for + repos that do not use claims is unchanged. The `tool_claim_use` predicate + (`schema_version` 27) reads it; its `absent` value is decided from + `ParamNames`. Read by OAI-034..036, MCP-033..035 and LC-028..030. - **DiscoverAgents** (`agents.go`) — finds `Agent(...)` / `SandboxAgent(...)` / `AgentDefinition(...)` constructor calls; captures all constructor kwargs into a typed `KwargTree`; sets `Opaque=true` for `Agent(**config)` or @@ -2068,6 +2079,10 @@ internal/ │ │ stamps a structural shells_out fact via │ │ pythonBodyShellsOut so agent rules can see a │ │ decorated tool that shells out). +│ ├── claim_use.go pythonClaimUse: the claim_use fact on a decorated +│ │ tool with a `claim` parameter (unused / +│ │ passthrough / checked / wrapped), plus +│ │ openAIToolHasInputGuardrail. │ ├── agents.go Python agent / guardrail / session discovery and │ │ edge resolution (DiscoverAgents, DiscoverGuardrails, │ │ DiscoverSessions, ResolveEdges, SessionClasses). @@ -2111,7 +2126,7 @@ internal/ ├── rules/ YAML-driven detection engine. Authoritative. │ ├── schema.go PolicyFile / RuleDef / MatchExpr types. │ ├── schema_version.go SupportedSchemaVersion const (engine ↔ pack gate). -│ ├── loader.go Validating YAML loader (recursive walk; skips manifest.yaml). Rejects repo_has_sdk_in_code values that are not SDK-enum tokens (catches the claude_sdk-vs-claude_agent_sdk silent never-fire). +│ ├── loader.go Validating YAML loader (recursive walk; skips manifest.yaml). Rejects repo_has_sdk_in_code values that are not SDK-enum tokens (catches the claude_sdk-vs-claude_agent_sdk silent never-fire), and tool_claim_use values outside absent / unused / passthrough / checked / wrapped. │ ├── predicates.go One Pred* per detection primitive. TS-aware: PredHasShellCall/PredHasWriteCall/PredHasCodeExecCall/PredHasDynamicURLCall / PredHasUnpinnedSchemeURLCall read the shells_out/writes_fs/code_exec/dynamic_url/url_scheme_unpinned facts for TypeScript and walk the AST for Python; PredHasBodyText uses a [Line, EndLine] span substring fallback (bodyTextFromSpan), kept for textual-absence checks only. │ ├── evaluator.go MatchExpr.Evaluate — recursive walker. │ └── rule_detector.go RuleDetector adapter + LoadRegistry. @@ -2288,6 +2303,10 @@ negation. - Required-field validation, enum validation (severity / category), duplicate-rule-ID detection across files. Every error is collected via `errors.Join` so a contributor sees every problem in one run. +- **Closed-value predicates**: a `repo_has_sdk_in_code` value that is not an + SDK-enum token, or a `tool_claim_use` value outside `absent` / `unused` / + `passthrough` / `checked` / `wrapped`, is rejected, since either would + silently never match. - **`applies_to`-vs-scope** check (`validAppliesToForScope`): the declared `applies_to` kinds must be legal for the rule's `scope`. - **`match`-vs-scope** check (`MatchExpr.outOfScopePredicates`): every predicate diff --git a/CHANGELOG.md b/CHANGELOG.md index 0222fc9d..0a1a7200 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,25 @@ to follow Semantic Versioning once it reaches 1.0. ## [Unreleased] +Rule schema 26 → 27 (`rules.SupportedSchemaVersion`). The rules fixture goes +from 343 to 352 rules. **Overall scores can fall on repositories with +side-effecting Python OpenAI Agents, MCP or LangChain tools** (OAI-034 / MCP-033 / +LC-028 fire on every such tool that takes no `claim` argument). + +### Added + +- **Tool-call claims (EBTE, schema 27).** Nine tool-scope rules for Python + OpenAI Agents, MCP and LangChain tools, over the new `tool_claim_use` + predicate and `claim_use` discovery fact: OAI-034 / MCP-033 / LC-028 (low) + flag a side-effecting tool with no `claim` argument, OAI-035 / MCP-034 / + LC-029 (medium) a `claim` nothing reads, and OAI-036 / MCP-035 / LC-030 + (medium) a `claim` that is only printed, logged, serialized or returned. A + second decorator, or an OpenAI tool input guardrail, counts as a possible + mediator. The fact is stamped only on tools that declare `claim`, so scan + output for repos that do not use claims is unchanged apart from the new + OAI-034 / MCP-033 / LC-028 findings. **Release order:** this engine first, + then the rules pack. + ### Fixed - The `Harness files:` line in the scan summary no longer says that no rule judges diff --git a/CLAUDE.md b/CLAUDE.md index fdafb071..bb78abfb 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -350,6 +350,14 @@ When changing a rule (add / remove / edit severity, confidence, match, text): 6. Commit and push the rules repo **and** the rulebook (the user pushes engine commits manually; confirm before pushing any of the three). +> **Tool-call claims, EBTE (2026-10-19, `schema_version` 26 → 27; +9 rules):** new tool-scope predicate `tool_claim_use` over the +> `claim_use` fact that Python decorated-tool discovery stamps on a tool declaring a `claim` parameter (`internal/analysis/claim_use.go`; +> `absent` is decided from `ParamNames`). A three-rung ladder per SDK: **OAI-034 / MCP-033 / LC-028** (low, 0.5) side-effecting tool with +> no claim, **OAI-035 / MCP-034 / LC-029** (medium, 0.6) claim never read, **OAI-036 / MCP-035 / LC-030** (medium, 0.55) claim only +> printed / logged / serialized / returned. A second decorator, or an OpenAI tool input guardrail, makes the fact `wrapped` (silent). Rules +> live in `/tool_call_claims.yaml`; rulebook docs in `docs/Policy//tool_call_claims.md`. Python only; TypeScript, Claude SDK, +> ADK, CrewAI, Pydantic AI, AutoGen and Vercel AI are follow-ups. The fixture holds **352** rules (343 + 9; re-derive). +> > **Pre-trust agent config (2026-10-12, `schema_version` 22 → 23; +5 rules):** new unconditionally-loaded category `agent_config` > (**AGCFG-004..007**, **MCPCFG-002**, repo scope) over five new repo predicates backed by `RepoInventory.ProjectConfig` > (`analysis.DiscoverProjectConfig`). The walk now enters `.gemini/`, `.vscode/`, `.cursor/`, `.codex/`. Env files are read for **key diff --git a/COVERAGE.md b/COVERAGE.md index febb17c6..b66ef08d 100644 --- a/COVERAGE.md +++ b/COVERAGE.md @@ -4,6 +4,7 @@ Coverage matrix for Trustabl's static analysis: which agent SDKs (and which languages) we currently scan, analyse, and detect against. This file is the at-a-glance reference; `ARCHITECTURE.md` has the implementation detail. +_Last reviewed: 2026-10-19 (branch `feat/analyzer-ebte-claims`, rebased on `main` c047646) — **Tool-call claims (EBTE; `schema_version` 26 → 27; +9 rules, 352 total):** new `claim_use` fact on Python decorated tools that declare a `claim` parameter (`internal/analysis/claim_use.go`) and the `tool_claim_use` predicate, read by **OAI-034..036** (OpenAI Agents), **MCP-033..035** (MCP) and **LC-028..030** (LangChain): no claim on a side-effecting tool, a claim nothing reads, a claim only written out or returned. Not covered: TypeScript, Claude Agent SDK, Google ADK, CrewAI, Pydantic AI, AutoGen, Vercel AI, and LangChain `StructuredTool.from_function` tools (no `claim_use` fact). On the 25-repo corpus the new rules add 4 findings, all on the no-claim rung._ _Last reviewed: 2026-10-13 (branch `feat/agcfg-preauth-config`, merged with `main` 32d845e) — **Pre-trust agent config (`schema_version` 22 → 23; +5 rules, 339 total):** new unconditionally-loaded category `agent_config` with **AGCFG-004** (`enableAllProjectMcpServers` / `enabledMcpjsonServers` in `.claude/settings.json`), **AGCFG-005** (project hooks running repo scripts / curl / `bash -c` / credential paths; Claude and Gemini settings), **AGCFG-006** (sandbox / proxy / endpoint / loader override keys in `.gemini/.env`, `.codex/config.toml`, narrowly root `.env`; key names only), **AGCFG-007** (`*autoApprove*` / `*yolo*` = true in `.vscode/` and `.cursor/` settings) and **MCPCFG-002** (MCP client config launching a local command). The walk now enters `.gemini/`, `.vscode/`, `.cursor/`, `.codex/`; `.mcp.json` is an `mcp_config`. The Gemini CLI hooks shape and Codex project-config trust behavior are not verified against vendor docs. Prior review: 2026-10-05 (HEAD d8a4f57 + uncommitted working tree, branch `feat/server-entrypoint-detection`) — **Server-entrypoint fact (Python; no schema change, stays 20; no rule, no predicate):** `RepoInventory.Entrypoints` (also `ScanResult.entrypoints`) records FastAPI / Flask route handlers (`http`) and Celery tasks / Dramatiq actors (`worker`), import-gated, skipping `examples/`, `demo/`, `docs/`, `samples/`, `cookbook/`, `tutorial(s)/`; `AgentRunCallDef.ServerReachable` is stamped in memory when a run call is in the entrypoint, a same-file helper, or an imported helper (one hop each; relative imports resolved within the scan set). Nothing consumes it yet; ranks 4 and 6 build on it in follow-up PRs. Deferred: Django, Starlette `Route()`, RQ, Temporal, serverless handlers, `add_api_route` / `add_url_rule`, class-based views, TypeScript. Corpus baseline: 2 entrypoints (Flask), 40 run calls, 0 stamped — all run calls are standalone scripts. Prior review: 2026-10-05 (HEAD 05c44c9 + uncommitted working tree, branch `fix/obs-global-instrumentor-credit`) — **Observability false-positive fix: process-wide instrumentor calls are credited (no schema change; `schema_version` stays 20, no new rules, no fixture change).** PYD-107's fix text recommends `logfire.instrument_pydantic_ai()`, but its match (`repo_has_observability` + `agent_kwarg_missing: [instrument]`) never read that call, so following the fix still produced one finding per agent; LC-112's "configure a global handler" advice had the same gap. Crediting the call as repo-level init evidence alone cannot fix this (the kwarg check does not read init evidence) and would make it worse for `Agent.instrument_all()`, so the credit is agent-scoped: new `AgentDef.InstrumentedBy` (`instrumented_by` in the JSON report, omitted when unset), stamped by `analysis.ApplyGlobalInstrumentation` from the init signals, and `PredAgentKwargMissing` treats the exact kwarg it names as set. Covered: `logfire.instrument_pydantic_ai` and `Agent.instrument_all()` (→ `instrument`, Pydantic AI, Python), `LangChainInstrumentor().instrument()` (→ `callbacks`, LangChain, Python). Also recognized as init evidence but crediting no agent: `logfire.instrument_openai_agents`, `logfire.instrument_mcp`, any other `Instrumentor(...).instrument(...)`. `.instrument_all()` counts only in a file importing `pydantic_ai` or `logfire`; the instrumentor shape only in a file importing `openinference`, `opentelemetry` or `traceloop` (the two-step `i = XInstrumentor(); i.instrument()` is not matched). Behavior changes beyond PYD-107/LC-112: a repo whose only signal is one of these calls now counts as observed (the repo absence rules go quiet, OBS-001 stays silent). VAI-101 is unchanged because the Vercel AI SDK has no process-wide telemetry switch — `experimental_telemetry` is enabled per call. **Opik** added as a vendor (Python `opik` import, `opik.configure` / `OpikTracer` / `track_openai`; TS `opik` import, `new Opik()` / `OpikExporter` / `OpikCallbackHandler`; recon dep needle `opik`). Known risk: a repo that uses Opik only through the `@track` decorator plus env configuration imports `opik` but makes no init call, so it reads as imported-never-initialized and OBS-001 fires; Langfuse `@observe` already behaves the same way, and decorator detection (`@observe`, `@track`, `@traceable`) is deliberately deferred until a rule consumes it. **Observability backlog items investigated and deliberately not shipped.** **Tool result not recorded on a span (P0 rank 3): no rule.** Frameworks span every tool call once tracing is enabled (OpenAI Agents by default, Google ADK, LangChain `BaseTool`, Pydantic AI `instrument`, Vercel `experimental_telemetry`, CrewAI instrumentors), and enablement is often env- or CLI-only (LangSmith, Claude Code, ADK), so a repo-level "tracing present" gate cannot say whether a given tool is spanned. Unlike the HTTPS-only rules it is not same-body evidence, and a rule would fire on correctly traced tools. **Required span attributes (P0 rank 2): no rule.** In every audited SDK the attributes are written by the SDK or by the OTel / OpenInference instrumentor, so a well-instrumented repo contains no attribute literals and a static rule cannot tell "the instrumentor sets them" from "nobody sets them". The OTel GenAI semantic conventions are still Development stability and have renamed keys (`gen_ai.usage.prompt_tokens` → `gen_ai.usage.input_tokens`, `gen_ai.system` → `gen_ai.provider.name`). The backlog's `thread.id` is the OS-thread attribute, not a conversation id; GenAI's conversation attribute is `gen_ai.conversation.id`, but the registry page now marks it deprecated / moved to the GenAI conventions repo — verify the current name before citing it. The session / `group_id` correlation idea (OpenAI `Runner.run` without `RunConfig(group_id=)`, Langfuse / LangChain without `session_id`) belongs to rank 4 (run-level correlation id), which needs a run-call kwarg predicate and is tracked with the HTTP-entrypoint design ticket. Prior review 2026-10-02 (HEAD 2acaf88 + uncommitted working tree, branch `feat/p0-gap-10-fetch-allowlist`) — **P0 rank #10, fetch tool HTTPS-only (`schema_version` 19 → 20): 14 new rules, one per SSRF rule.** The host allow-list half of the gap already shipped as the allow-list credit on the 14 SSRF rules; what was missing was the scheme. New tool-scope predicate **`has_unpinned_scheme_url_call`** (a recognized HTTP call whose URL is dynamic and does not begin with a literal `https://`; Python via `analysis.URLArgSchemePinned`, TS via the new `url_scheme_unpinned` handler fact next to `dynamic_url`; f-string / template / `+` / `%` / `.format` forms with a leading `https://` are pinned). New rules, medium / 0.6: **CSDK-024/025, OAI-032/033, MCP-031/032, ADK-022/023, LC-026/LC-027, CREW-015, PYD-015, AG2-022, VAI-021** — each fires only when its SSRF sibling has been silenced by an allow-list and no scheme check (`.scheme ==`, `.protocol ===`, `startswith("https://")`) is present, so SSRF and HTTPS never fire on the same tool. **PYD-103** fix text now notes `WebFetchTool(allowed_domains=)` is Anthropic-only and not a substitute for egress controls. Not built: LangGraph nodes (state is not a model-chosen argument, no SSRF base rule), `raw_llm_sdk` (repo scope), a Pydantic `WebFetchTool` `allowed_domains` rule (provider-dependent), scheme validation in another function/module, fully literal `http://` URLs (not dynamic). Rule count **330** (316 + 14; re-derive). Prior review: 2026-10-01 (HEAD 1d24b98 + uncommitted working tree, branch `feat/p0-partial-gaps`) — **P0 partial-gaps batch (no new schema field; `schema_version` stays 19): wall-clock timeout extended to Google ADK and AutoGen; cost cap and general tool-output trust investigated and deliberately not shipped.** New rules **ADK-113** and **AG2-021** (run call with no wall-clock timeout) ride on the existing `agent_run_call_wall_clock_timeout_missing` predicate, widened to ADK and AutoGen: `DiscoverAgentRunCalls` now also discovers ADK `Runner.run` / `run_async` (runner variable resolved to its agent through a same-file `Runner(agent=)` / `InMemoryRunner(agent)` pre-pass; `run_live` and `app=`-built runners are not run calls here) and AutoGen `initiate_chat` / `a_initiate_chat` (receiver **and** recipient) / `run` / `a_run` / `run_stream`; the predicate also credits ADK `abort_signal` and AutoGen `cancellation_token` (non-None). **VAI-020** now also requires `timeout` to be absent (the Vercel AI SDK's native `timeout` option, number or `{ totalMs, stepMs, ... }`, was a false positive); Vercel needed no new TS discovery. **Per-run token/cost cap (P0 rank 8): no further rule.** Checked against upstream: OpenAI Agents (`Runner.run*`/`RunConfig`), LangChain/LangGraph (`max_iterations`, `ModelCallLimitMiddleware` counts calls not tokens), Google ADK (`RunConfig.max_llm_calls` counts calls) and Vercel AI (`maxOutputTokens` is per step) have no run-level token/cost parameter; CrewAI's `Agent.max_tokens` is declared but never read by the agent or executor (usage reaches only `TokenCalcHandler`, which tracks and does not enforce), so an absence rule would point at a no-op. AutoGen v0.4's `TokenUsageTermination` is a real primitive but is a composable termination object on team constructors, which are not discovered — not built. **Untrusted tool output (P0 rank 11): no general rule.** Framework loops return tool output through the structured tool-result channel (no user-code concatenation site), user-code `f"...{result}"` needs inter-procedural taint to call `result` a tool result, and a "delimiter present" check would reward cosmetic tags; RAW-001/002 stay the only statically defensible slice. Rule count **316** (314 + 2; re-derive). Prior review: **P0 remaining-gaps, engine batch (`schema_version` 18 → 19): six rules that needed new predicates or discovery.** **LC-114** (raw `StateGraph` compiled with no `checkpointer`; new `agent_kwargs_observed` + dual-scope `repo_langgraph_platform_config_present`, so it stays silent on unlinked/chained/opaque/subgraph compiles and on repos with a `langgraph.json`), **LC-115** (a function registered via `add_node` that shells out / executes code / writes / fetches a dynamic URL with no `interrupt(` — new `langgraph_node` ToolKind via `DiscoverLangGraphNodes`; same-file undecorated top-level functions only, confidence 0.45), **OAI-120** / **PYD-108** (run call with no wall-clock timeout — new `AgentRunCallDef.WallClockTimeoutWrapped` from a structural ancestor walk of `asyncio.wait_for` / `asyncio.timeout` / `anyio.move_on_after` / `anyio.fail_after`, new predicate `agent_run_call_wall_clock_timeout_missing`), and **RAW-001** / **RAW-002** (new `raw_llm_sdk` category, loaded unconditionally: the bare `anthropic` / `openai` client in a manual tool loop that builds a dynamic system prompt/message from tool-call-derived values — a deliberately narrow same-function heuristic via `DetectRawLLMToolOutputInSystem`). Rule count **314** (308 + 6; re-derive). Prior review 2026-09-30 (branch `feat/p0-remaining-gaps-batch1`, merged with Batch 2 on `main`) — **P0 remaining-gaps batch, rules-layer only (no discovery, predicate or `schema_version` change; stays 18).** New rules **CSDK-114** (TS `query()` main agent with inline `options` and no `options.maxBudgetUsd`), **CREW-111** (`Agent` with no `max_execution_time`, exempting `config=`), **LC-113** (Python `AgentExecutor` with no `max_execution_time`) and **VAI-020** (`generateText`/`streamText` with a `stopWhen`/`maxSteps` loop and no `abortSignal`; `ToolLoopAgent` excluded). The 14 SSRF rules (CSDK-009/013, OAI-018/024, LC-005/013, CREW-005, PYD-005, ADK-012/016, MCP-008/013, AG2-011, VAI-003) now **credit a host allow-list** in the tool body (`not: has_body_text`) instead of firing on any non-literal URL; IDs/severity/confidence unchanged. Deferred because they need new predicates or discovery, which this batch was barred from adding: OAI/PYD whole-run wall-clock deadline (needs a run-call deadline-wrapper fact), Python `ClaudeAgentOptions(max_budget_usd)`, LangGraph no-checkpointer and mutating-node-without-HITL (need a Platform/`langgraph.json` exemption, opaque-`compile()` handling and an `interrupt()` fact), LangChain.js `maxExecutionTime` (option not verified to exist), and untrusted-tool-output concat (no raw-API-loop discovery). Rule count **308** (302 + Batch 2's 6); re-derive with `grep -rhoE '^\s*-\s*id:\s*\S+' testdata/rules-fixture/*/*.yaml | wc -l`. Prior review 2026-09-29 (branch `feat/p0-side-effect-bounds-batch2`, off `main` e5cef47) — Batch 2 of the side-effect-bounds rule: **CREW-014** (CrewAI), **ADK-020** (Python) / **ADK-021** (TypeScript), **AG2-020** (AutoGen / AG2), **VAI-019** (Vercel AI SDK), and **CSDK-023** (Claude Agent SDK TypeScript) — same detection as Batch 1, adapted to each SDK's tool shape (one new `side_effect_bounds.yaml` per SDK; severity high, confidence 0.5). Approval gates differ per SDK and are honest about it: Vercel keys on `needsApproval` (as VAI-018); ADK Python on `require_confirmation` or a `request_confirmation(` call; ADK TypeScript has **no** declarative confirmation option on `FunctionTool`, so ADK-021 treats a `requestConfirmation(` call in the handler as the gate; CrewAI, AutoGen and Claude TS have no tool-level gate (`Task(human_input=True)`, `human_input_mode`, and `canUseTool`/`permissionMode` live on the task/agent/`query()` options, which a tool-scope rule cannot see), so those rules can fire on a tool gated elsewhere. One small discovery change: `DiscoverADKTools` now copies `FunctionTool(...)` keyword arguments into `ToolDef.Config` (previously dropped), which is what lets ADK-020 see `require_confirmation`. Known discovery limits carried into the rules: Vercel tools are matched only through a `const x = tool(...)` binding (inline `tools: { x: tool(...) }` has no name); Claude TS reads parameter names only from an inline object-literal schema; CrewAI `BaseTool` subclasses, autogen_core `FunctionTool`, and ADK bare functions in `tools=[...]` are not discovered. No schema/predicate/`schema_version` change. Rule count **304** (298 + 6; re-derive rather than trust this figure). Prior review 2026-09-29 (pending merge commit; `origin/main` merged into `feat/p0-side-effect-bounds-batch1`) — Batch 1 of the _Last reviewed: 2026-10-12 (branch `feat/langgraph-inmemory-checkpointer-server`, merged with `main` 44b2062) — **LC-116: in-memory LangGraph checkpointer run from a server entrypoint (`schema_version` 21 → 22; 1 rule, 2 predicates).** High / 0.6, agent scope, Python, `langchain_state_graph` + `langchain_agent`: `checkpointer=` resolves to `MemorySaver` / `InMemorySaver` (call, `MemorySaver(serde=...)`, aliased, or a same-file variable bound to one), a run call of that graph is reachable from a FastAPI / Flask / Celery / Dramatiq entrypoint, and the repo has no `langgraph.json`. New predicates `agent_checkpointer_in_memory`, `agent_server_reachable`; new in-memory fields `AgentDef.CheckpointerClass`, `AgentRunCallDef.AgentFilePath`. Run-call discovery now anchors LangGraph `invoke`/`ainvoke`/`stream`/`astream`/`astream_events`/`batch`/`abatch` on compiled-graph variables, the direct `builder.compile(...).invoke(...)` chain and prebuilt-agent variables only (never on an arbitrary `.invoke`), cross-file for module-level graphs. Test paths (`tests/`, `test_*.py`, `conftest.py`) never satisfy `agent_server_reachable`. **Not covered:** a helper of a helper; a graph built/compiled in a factory function; a saver chosen at runtime; `import graph; graph.app.invoke()`; Django/Starlette/RQ/serverless entrypoints; TypeScript. **Recall on real server repos is unmeasured:** the corpus has 40 run calls, 0 LangGraph, 0 server-reachable, and its full findings output (185) is identical before and after. Rule count **334** (333 + 1; re-derive). Prior review: 2026-10-11 (branch `feat/otel-labels-rules`, rebased onto 9fc2c39) — **OBS-006..008 credit fixes (no schema change, stays 21; no rule change).** Each file's own imports of agent-reliability-otel-labels are followed, so a renamed import (`mark_tool_span as mts`, `markToolSpan as label`) still counts. `trustabl.*` label keys written to a span by hand (`set_attribute` / `setAttribute` first argument, or a key of the literal passed to `set_attributes` / `setAttributes`) are recorded as `attr_literal` signals and credit the ladder rung by rung, so a repo that writes the labels without the library no longer draws OBS-006..008. Every signal carries `TestOrigin`; test-file signals still credit OBS-006..008. Rule counts re-derived: **333** fixture. Prior review 2026-10-07 (HEAD 8d0fb76 + uncommitted working tree, branch `feat/forge-runtime-tracing`) — **agent-reliability-otel-labels rules renumbered to `schema_version` 21.** The OBS-006..008 batch below was built in parallel with the HTTPS-only batch, and both took 19 → 20; after merging main this batch takes 20 → 21 (engine constant and both manifests). Rule counts re-derived: **333** fixture / **333** production (330 + OBS-006..008); `scripts/check-rules-sync.sh` passes. No coverage change. Prior review 2026-10-05 (HEAD d8a4f57 + uncommitted working tree, branch `feat/server-entrypoint-detection`) — **Server-entrypoint fact (Python; no schema change, stays 20; no rule, no predicate):** `RepoInventory.Entrypoints` (also `ScanResult.entrypoints`) records FastAPI / Flask route handlers (`http`) and Celery tasks / Dramatiq actors (`worker`), import-gated, skipping `examples/`, `demo/`, `docs/`, `samples/`, `cookbook/`, `tutorial(s)/`; `AgentRunCallDef.ServerReachable` is stamped in memory when a run call is in the entrypoint, a same-file helper, or an imported helper (one hop each; relative imports resolved within the scan set). Nothing consumes it yet; ranks 4 and 6 build on it in follow-up PRs. Deferred: Django, Starlette `Route()`, RQ, Temporal, serverless handlers, `add_api_route` / `add_url_rule`, class-based views, TypeScript. Corpus baseline: 2 entrypoints (Flask), 40 run calls, 0 stamped — all run calls are standalone scripts. Prior review 2026-10-05 (HEAD 05c44c9 + uncommitted working tree, branch `fix/obs-global-instrumentor-credit`) — **Observability false-positive fix: process-wide instrumentor calls are credited (no schema change; `schema_version` stays 20, no new rules, no fixture change).** PYD-107's fix text recommends `logfire.instrument_pydantic_ai()`, but its match (`repo_has_observability` + `agent_kwarg_missing: [instrument]`) never read that call, so following the fix still produced one finding per agent; LC-112's "configure a global handler" advice had the same gap. Crediting the call as repo-level init evidence alone cannot fix this (the kwarg check does not read init evidence) and would make it worse for `Agent.instrument_all()`, so the credit is agent-scoped: new `AgentDef.InstrumentedBy` (`instrumented_by` in the JSON report, omitted when unset), stamped by `analysis.ApplyGlobalInstrumentation` from the init signals, and `PredAgentKwargMissing` treats the exact kwarg it names as set. Covered: `logfire.instrument_pydantic_ai` and `Agent.instrument_all()` (→ `instrument`, Pydantic AI, Python), `LangChainInstrumentor().instrument()` (→ `callbacks`, LangChain, Python). Also recognized as init evidence but crediting no agent: `logfire.instrument_openai_agents`, `logfire.instrument_mcp`, any other `Instrumentor(...).instrument(...)`. `.instrument_all()` counts only in a file importing `pydantic_ai` or `logfire`; the instrumentor shape only in a file importing `openinference`, `opentelemetry` or `traceloop` (the two-step `i = XInstrumentor(); i.instrument()` is not matched). Behavior changes beyond PYD-107/LC-112: a repo whose only signal is one of these calls now counts as observed (the repo absence rules go quiet, OBS-001 stays silent). VAI-101 is unchanged because the Vercel AI SDK has no process-wide telemetry switch — `experimental_telemetry` is enabled per call. **Opik** added as a vendor (Python `opik` import, `opik.configure` / `OpikTracer` / `track_openai`; TS `opik` import, `new Opik()` / `OpikExporter` / `OpikCallbackHandler`; recon dep needle `opik`). Known risk: a repo that uses Opik only through the `@track` decorator plus env configuration imports `opik` but makes no init call, so it reads as imported-never-initialized and OBS-001 fires; Langfuse `@observe` already behaves the same way, and decorator detection (`@observe`, `@track`, `@traceable`) is deliberately deferred until a rule consumes it. **Observability backlog items investigated and deliberately not shipped.** **Tool result not recorded on a span (P0 rank 3): no rule.** Frameworks span every tool call once tracing is enabled (OpenAI Agents by default, Google ADK, LangChain `BaseTool`, Pydantic AI `instrument`, Vercel `experimental_telemetry`, CrewAI instrumentors), and enablement is often env- or CLI-only (LangSmith, Claude Code, ADK), so a repo-level "tracing present" gate cannot say whether a given tool is spanned. Unlike the HTTPS-only rules it is not same-body evidence, and a rule would fire on correctly traced tools. **Required span attributes (P0 rank 2): no rule.** In every audited SDK the attributes are written by the SDK or by the OTel / OpenInference instrumentor, so a well-instrumented repo contains no attribute literals and a static rule cannot tell "the instrumentor sets them" from "nobody sets them". The OTel GenAI semantic conventions are still Development stability and have renamed keys (`gen_ai.usage.prompt_tokens` → `gen_ai.usage.input_tokens`, `gen_ai.system` → `gen_ai.provider.name`). The backlog's `thread.id` is the OS-thread attribute, not a conversation id; GenAI's conversation attribute is `gen_ai.conversation.id`, but the registry page now marks it deprecated / moved to the GenAI conventions repo — verify the current name before citing it. The session / `group_id` correlation idea (OpenAI `Runner.run` without `RunConfig(group_id=)`, Langfuse / LangChain without `session_id`) belongs to rank 4 (run-level correlation id), which needs a run-call kwarg predicate and is tracked with the HTTP-entrypoint design ticket. Prior review 2026-10-04 (HEAD 4f8f6bd + uncommitted working tree, branch `feat/forge-runtime-tracing`) — **agent-reliability-otel-labels process-label rules (`schema_version` 20 → 21; recorded as 19 → 20 before main's HTTPS-only batch was merged).** New inventory pass `analysis.DiscoverOtelLabels` emits typed `OtelLabelsSignal` records (Kind `import` / `run_start` / `tool_span` / `run_end`) for Python and TypeScript/JavaScript; the Go binding is not inspected. Recon gains `RepoProfile.OtelLabelsDeclaredIn` (`agent-reliability-otel-labels` / `agent_reliability_otel_labels` in the Python root manifests, `@trustabl/agent-reliability-otel-labels` in `package.json`). Both are kept out of `ObservabilitySignals` / `ObsDeps` because the library never creates a span and must not satisfy its own prerequisite. Three new repo-scope predicates (`repo_otel_labels_installed`, `repo_otel_labels_initialized`, `repo_otel_labels_implemented`) back three new rules in `observability/quality.yaml`, all gated on `repo_observability_initialized: true` and mutually exclusive: **OBS-006** (instrumented, library neither declared nor imported; low), **OBS-007** (installed, no `mark_start`; medium), **OBS-008** (run started, no `mark_run_end`, or tools exist and no `mark_tool_span`; medium). They check that the calls exist, not that labels land or are correct: a missing current span makes every call a silent no-op, and a shared `Run`, a constant `attempt`, labelling outside `finally`, a `side_effect` that contradicts the tool body, and a missing `mark_handoff` are deferred. Rule counts at the time: 319 fixture / 319 production, before main's 14 HTTPS-only rules were merged (333 after). Prior review 2026-10-04 (HEAD 5ba476d, branch `feat/forge-runtime-tracing`) — **Recon now detects Google ADK for Go (`go.mod`: `google.golang.org/adk` → recon-only dep name `google-adk-go`, not mapped to an SDK for `scan`; used by `trustabl forge` to select the Go runtime-tracing recipe).** Prior review 2026-10-02 (HEAD 2acaf88 + uncommitted working tree, branch `feat/p0-gap-10-fetch-allowlist`) — **P0 rank #10, fetch tool HTTPS-only (`schema_version` 19 → 20): 14 new rules, one per SSRF rule.** The host allow-list half of the gap already shipped as the allow-list credit on the 14 SSRF rules; what was missing was the scheme. New tool-scope predicate **`has_unpinned_scheme_url_call`** (a recognized HTTP call whose URL is dynamic and does not begin with a literal `https://`; Python via `analysis.URLArgSchemePinned`, TS via the new `url_scheme_unpinned` handler fact next to `dynamic_url`; f-string / template / `+` / `%` / `.format` forms with a leading `https://` are pinned). New rules, medium / 0.6: **CSDK-024/025, OAI-032/033, MCP-031/032, ADK-022/023, LC-026/LC-027, CREW-015, PYD-015, AG2-022, VAI-021** — each fires only when its SSRF sibling has been silenced by an allow-list and no scheme check (`.scheme ==`, `.protocol ===`, `startswith("https://")`) is present, so SSRF and HTTPS never fire on the same tool. **PYD-103** fix text now notes `WebFetchTool(allowed_domains=)` is Anthropic-only and not a substitute for egress controls. Not built: LangGraph nodes (state is not a model-chosen argument, no SSRF base rule), `raw_llm_sdk` (repo scope), a Pydantic `WebFetchTool` `allowed_domains` rule (provider-dependent), scheme validation in another function/module, fully literal `http://` URLs (not dynamic). Rule count **330** (316 + 14; re-derive). Prior review: 2026-10-01 (HEAD 1d24b98 + uncommitted working tree, branch `feat/p0-partial-gaps`) — **P0 partial-gaps batch (no new schema field; `schema_version` stays 19): wall-clock timeout extended to Google ADK and AutoGen; cost cap and general tool-output trust investigated and deliberately not shipped.** New rules **ADK-113** and **AG2-021** (run call with no wall-clock timeout) ride on the existing `agent_run_call_wall_clock_timeout_missing` predicate, widened to ADK and AutoGen: `DiscoverAgentRunCalls` now also discovers ADK `Runner.run` / `run_async` (runner variable resolved to its agent through a same-file `Runner(agent=)` / `InMemoryRunner(agent)` pre-pass; `run_live` and `app=`-built runners are not run calls here) and AutoGen `initiate_chat` / `a_initiate_chat` (receiver **and** recipient) / `run` / `a_run` / `run_stream`; the predicate also credits ADK `abort_signal` and AutoGen `cancellation_token` (non-None). **VAI-020** now also requires `timeout` to be absent (the Vercel AI SDK's native `timeout` option, number or `{ totalMs, stepMs, ... }`, was a false positive); Vercel needed no new TS discovery. **Per-run token/cost cap (P0 rank 8): no further rule.** Checked against upstream: OpenAI Agents (`Runner.run*`/`RunConfig`), LangChain/LangGraph (`max_iterations`, `ModelCallLimitMiddleware` counts calls not tokens), Google ADK (`RunConfig.max_llm_calls` counts calls) and Vercel AI (`maxOutputTokens` is per step) have no run-level token/cost parameter; CrewAI's `Agent.max_tokens` is declared but never read by the agent or executor (usage reaches only `TokenCalcHandler`, which tracks and does not enforce), so an absence rule would point at a no-op. AutoGen v0.4's `TokenUsageTermination` is a real primitive but is a composable termination object on team constructors, which are not discovered — not built. **Untrusted tool output (P0 rank 11): no general rule.** Framework loops return tool output through the structured tool-result channel (no user-code concatenation site), user-code `f"...{result}"` needs inter-procedural taint to call `result` a tool result, and a "delimiter present" check would reward cosmetic tags; RAW-001/002 stay the only statically defensible slice. Rule count **316** (314 + 2; re-derive). Prior review: **P0 remaining-gaps, engine batch (`schema_version` 18 → 19): six rules that needed new predicates or discovery.** **LC-114** (raw `StateGraph` compiled with no `checkpointer`; new `agent_kwargs_observed` + dual-scope `repo_langgraph_platform_config_present`, so it stays silent on unlinked/chained/opaque/subgraph compiles and on repos with a `langgraph.json`), **LC-115** (a function registered via `add_node` that shells out / executes code / writes / fetches a dynamic URL with no `interrupt(` — new `langgraph_node` ToolKind via `DiscoverLangGraphNodes`; same-file undecorated top-level functions only, confidence 0.45), **OAI-120** / **PYD-108** (run call with no wall-clock timeout — new `AgentRunCallDef.WallClockTimeoutWrapped` from a structural ancestor walk of `asyncio.wait_for` / `asyncio.timeout` / `anyio.move_on_after` / `anyio.fail_after`, new predicate `agent_run_call_wall_clock_timeout_missing`), and **RAW-001** / **RAW-002** (new `raw_llm_sdk` category, loaded unconditionally: the bare `anthropic` / `openai` client in a manual tool loop that builds a dynamic system prompt/message from tool-call-derived values — a deliberately narrow same-function heuristic via `DetectRawLLMToolOutputInSystem`). Rule count **314** (308 + 6; re-derive). Prior review 2026-09-30 (branch `feat/p0-remaining-gaps-batch1`, merged with Batch 2 on `main`) — **P0 remaining-gaps batch, rules-layer only (no discovery, predicate or `schema_version` change; stays 18).** New rules **CSDK-114** (TS `query()` main agent with inline `options` and no `options.maxBudgetUsd`), **CREW-111** (`Agent` with no `max_execution_time`, exempting `config=`), **LC-113** (Python `AgentExecutor` with no `max_execution_time`) and **VAI-020** (`generateText`/`streamText` with a `stopWhen`/`maxSteps` loop and no `abortSignal`; `ToolLoopAgent` excluded). The 14 SSRF rules (CSDK-009/013, OAI-018/024, LC-005/013, CREW-005, PYD-005, ADK-012/016, MCP-008/013, AG2-011, VAI-003) now **credit a host allow-list** in the tool body (`not: has_body_text`) instead of firing on any non-literal URL; IDs/severity/confidence unchanged. Deferred because they need new predicates or discovery, which this batch was barred from adding: OAI/PYD whole-run wall-clock deadline (needs a run-call deadline-wrapper fact), Python `ClaudeAgentOptions(max_budget_usd)`, LangGraph no-checkpointer and mutating-node-without-HITL (need a Platform/`langgraph.json` exemption, opaque-`compile()` handling and an `interrupt()` fact), LangChain.js `maxExecutionTime` (option not verified to exist), and untrusted-tool-output concat (no raw-API-loop discovery). Rule count **308** (302 + Batch 2's 6); re-derive with `grep -rhoE '^\s*-\s*id:\s*\S+' testdata/rules-fixture/*/*.yaml | wc -l`. Prior review 2026-09-29 (branch `feat/p0-side-effect-bounds-batch2`, off `main` e5cef47) — Batch 2 of the side-effect-bounds rule: **CREW-014** (CrewAI), **ADK-020** (Python) / **ADK-021** (TypeScript), **AG2-020** (AutoGen / AG2), **VAI-019** (Vercel AI SDK), and **CSDK-023** (Claude Agent SDK TypeScript) — same detection as Batch 1, adapted to each SDK's tool shape (one new `side_effect_bounds.yaml` per SDK; severity high, confidence 0.5). Approval gates differ per SDK and are honest about it: Vercel keys on `needsApproval` (as VAI-018); ADK Python on `require_confirmation` or a `request_confirmation(` call; ADK TypeScript has **no** declarative confirmation option on `FunctionTool`, so ADK-021 treats a `requestConfirmation(` call in the handler as the gate; CrewAI, AutoGen and Claude TS have no tool-level gate (`Task(human_input=True)`, `human_input_mode`, and `canUseTool`/`permissionMode` live on the task/agent/`query()` options, which a tool-scope rule cannot see), so those rules can fire on a tool gated elsewhere. One small discovery change: `DiscoverADKTools` now copies `FunctionTool(...)` keyword arguments into `ToolDef.Config` (previously dropped), which is what lets ADK-020 see `require_confirmation`. Known discovery limits carried into the rules: Vercel tools are matched only through a `const x = tool(...)` binding (inline `tools: { x: tool(...) }` has no name); Claude TS reads parameter names only from an inline object-literal schema; CrewAI `BaseTool` subclasses, autogen_core `FunctionTool`, and ADK bare functions in `tools=[...]` are not discovered. No schema/predicate/`schema_version` change. Rule count **304** (298 + 6; re-derive rather than trust this figure). Prior review 2026-09-29 (pending merge commit; `origin/main` merged into `feat/p0-side-effect-bounds-batch1`) — Batch 1 of the side-effect-bounds rule (competitive-analysis backlog rank 12, P0): new rules @@ -86,9 +87,9 @@ Legend: ✅ full · ◐ partial · ❌ none · — N/A |---|---|---|---|---| | **Claude Agent SDK** | Python | ✅ dep-scan + file inventory + `.claude/` & `.claude-plugin/` components | ✅ tools, agents, subagents (canonical + flat-collection shape fallback), skills (`SKILL.md`), slash commands, plugin manifests, settings, `ClaudeAgentOptions` session config | ✅ tool CSDK-001..009, 107, 108 (008 = `**kwargs` without input_schema, 009 = SSRF, 024 = allow-listed host without pinned https://); agent CSDK-101..105; subagent CSDK-110, 111 (fire on pure-markdown collections); repo CSDK-201..205 (`defaultMode` / `permission_mode` bypass; 203 = SDK code without an agent-guidance doc: AGENTS.md/CLAUDE.md; 204 = no explicit `max_turns` on `ClaudeAgentOptions`; 205 = `acceptEdits` with no `disallowed_tools` deny-list); skill CSKILL-001 (unrestricted shell), 002/003 (dynamic-context exec + egress/secret), 010/011 (bundled-script egress + credential-read), 030 (hardcoded secret in bundled file), 020 (external-URL fetch), 040 (injection markers incl. hidden-Unicode), 050 (model-invocable + side-effecting), 060 (description claims read-only but grants side-effecting tools) | | **Claude Agent SDK** | TypeScript | ✅ dep-scan (`@anthropic-ai/claude-agent-sdk`) + file inventory + `.claude/` components | ✅ tools (`tool()` factory), agents (main thread `QueryMainAgent` per `query()` call + sub-agents inline-in-query + typed-const `AgentDefinition`), MCP servers (createSdkMcpServer + 4 config literals) | ◐ tool CSDK-010 (shell), 011 (eval/new Function), 012 (fs-write), 013 (SSRF / dynamic URL), 025 (allow-listed host, https:// not pinned), 014 (no description), 016 (mutating tool no idempotency key), CSDK-023 (side-effecting tool with a free-form recipient/amount and no visible bound; no tool-level gate exists to check); agent CSDK-120 (permissionMode bypass), 121..124 (`AgentDefinition` grants Bash / WebSearch / write built-ins / WebFetch), 130 (`query()` main agent grants Bash), 131 (`query()` main agent grants write/fetch built-ins), 114 (`query()` main agent with no `options.maxBudgetUsd`); META-004 no longer fires | -| **OpenAI Agents SDK** | Python | ✅ dep-scan + file inventory | ✅ tools, hosted tools (11 classes), agents, MCP servers (3 transports + alias, kwargs captured incl. `tool_filter`), guardrails, sessions, `Runner.run`/`run_sync`/`run_streamed` call sites (`max_turns`) | ✅ tool OAI-001..015, 018 (018 = SSRF / caller-controlled URL), 032 (allow-listed host, https:// not pinned), 030 (side-effecting tool with a free-form recipient/amount and no visible bound); agent OAI-101..104, 106, 109, 110, 111, 112 (`Runner.run` family with no `max_turns`), 120 (run call with no wall-clock timeout), 115 (`HostedMCPTool` with no `allowed_tools`), 118 (MCP server with no `tool_filter`); repo OAI-201, 202 (202 = SDK code without an agent-guidance doc: AGENTS.md/CLAUDE.md) | +| **OpenAI Agents SDK** | Python | ✅ dep-scan + file inventory | ✅ tools, hosted tools (11 classes), agents, MCP servers (3 transports + alias, kwargs captured incl. `tool_filter`), guardrails, sessions, `Runner.run`/`run_sync`/`run_streamed` call sites (`max_turns`) | ✅ tool OAI-001..015, 018 (018 = SSRF / caller-controlled URL), 032 (allow-listed host, https:// not pinned), 030 (side-effecting tool with a free-form recipient/amount and no visible bound); agent OAI-101..104, 106, 109, 110, 111, 112 (`Runner.run` family with no `max_turns`), 120 (run call with no wall-clock timeout), 115 (`HostedMCPTool` with no `allowed_tools`), 118 (MCP server with no `tool_filter`); repo OAI-201, 202 (202 = SDK code without an agent-guidance doc: AGENTS.md/CLAUDE.md); tool-call claims (EBTE) OAI-034..036 — Python decorated tools only | | **OpenAI Agents SDK** | TypeScript | ✅ dep-scan (`@openai/agents` substring catches `-core` / `-openai`) + file inventory | ✅ tools (`tool({...})` factory), agents (`new Agent({...})` + `Agent.create(...)`), hosted tools (9 factories across `@openai/agents-core` and `@openai/agents-openai`), MCP servers (3 transports + `MCPServers` wrapper, kwargs captured incl. `toolFilter`), guardrails (4 `defineX` factories), sessions (`MemorySession` / `OpenAIConversationsSession` / `OpenAIResponsesCompactionSession`) | ✅ tool OAI-016 (fetch without AbortSignal timeout), OAI-017 (eval / new Function), OAI-019 (mutating tool without idempotency), OAI-022 (no description), OAI-024 (dynamic URL / SSRF), OAI-033 (allow-listed host, https:// not pinned), OAI-031 (side-effecting tool with a free-form recipient/amount and no visible bound — the TS sibling of the Python row's OAI-030); agent OAI-105 (content hosted-tool without inputGuardrails), OAI-116 (`hostedMcpTool` with no `allowedTools` allow-list), OAI-117 (`hostedMcpTool` with no `requireApproval`), OAI-119 (MCP server with no `toolFilter`); all with fire/silent cases in the per-rule harness | -| **MCP** | Python | ✅ tool registrations + config files | ◐ tool registrations only (no server-side resource/prompt discovery) | ✅ dedicated `mcp/` pack: tool MCP-001..010 (001 no description, 002 untyped params, 003 ambiguous name, 004 network timeout, 005 path safety, 006 error contract, 007 idempotency, 008 SSRF, 031 allow-listed host without pinned https://, 009 code-exec, 010 shell), MCP-030 (side-effecting tool with a free-form recipient/amount and no visible bound). `mcp_tool` coverage now lives ONLY in this pack — stripped from the CSDK rules' `applies_to` so a pure-MCP repo is covered and a mixed Claude+MCP repo does not double-fire | +| **MCP** | Python | ✅ tool registrations + config files | ◐ tool registrations only (no server-side resource/prompt discovery) | ✅ dedicated `mcp/` pack: tool MCP-001..010 (001 no description, 002 untyped params, 003 ambiguous name, 004 network timeout, 005 path safety, 006 error contract, 007 idempotency, 008 SSRF, 031 allow-listed host without pinned https://, 009 code-exec, 010 shell), MCP-030 (side-effecting tool with a free-form recipient/amount and no visible bound). `mcp_tool` coverage now lives ONLY in this pack — stripped from the CSDK rules' `applies_to` so a pure-MCP repo is covered and a mixed Claude+MCP repo does not double-fire; tool-call claims (EBTE) MCP-033..035 — Python decorated tools only | | **MCP** | TypeScript | ✅ dep-scan (`@modelcontextprotocol/sdk`) + file inventory | ✅ server authoring: `new McpServer(...)` receiver tracked, `registerTool` / legacy `tool` registrations → `KindMCPTool` (reuses `tsZodParamNames` / `tsHandlerFacts`). Distinct from the Claude client-config `createSdkMcpServer` discovery. Low-level `Server` + `setRequestHandler` not extracted (gap) | ✅ tool MCP-011..014 (011 no description, 012 shell, 013 SSRF, MCP-032 allow-listed host without pinned https://, 014 eval / new Function); shared `mcp/` pack, `language: typescript` | | **MCP** | Go | ✅ dep-scan (`go.mod`: mark3labs/mcp-go, official go-sdk, metoro-io/mcp-golang) + file inventory | ◐ tools (`mcp.NewTool("n", mcp.WithDescription(...), mcp.WithString(...))` — mark3labs, full name/desc/params; `mcp.AddTool(server, &mcp.Tool{Name, Description}, fn)` — official go-sdk, name/desc). metoro `RegisterTool`, the official handler-struct param schema, and the `s.AddTool` registration edge are v1 gaps | ◐ field-based language:go: MCP-015 (no description), MCP-016 (ambiguous name). Body-fact rules (shell/SSRF/timeout) need Go AST predicate branches (fast-follow); untyped-params is N/A (Go is statically typed) | | **MCP** | C#/.NET | ✅ dep-scan (`ModelContextProtocol` in `Directory.Packages.props` / `packages.config`; variable-named `.csproj` is a best-effort gap) + file inventory | ◐ tools (`[McpServerTool]`-attributed methods, official ModelContextProtocol SDK; name = method name, description from a co-located `[Description(...)]`, typed params). `[McpServerTool(Name=...)]` override, Semantic Kernel `[KernelFunction]`, and AutoGen `[Function]` are gaps | ◐ field-based language:csharp: MCP-017 (no description), MCP-018 (ambiguous name). Body-fact rules need C# AST predicate branches (fast-follow); untyped-params N/A (C# is statically typed) | @@ -98,7 +99,7 @@ Legend: ✅ full · ◐ partial · ❌ none · — N/A | **Google ADK** | TypeScript | ✅ dep-scan (`@google/adk`) + file inventory | ✅ tools (`new FunctionTool({...})`), agents (5 constructors: LlmAgent + SequentialAgent + ParallelAgent + LoopAgent + RoutedAgent), hosted tools (13 classes), subAgents edges | ◐ tool ADK-013 (no description), 015 (eval / new Function), 016 (SSRF / dynamic URL), 023 (allow-listed host, https:// not pinned), 021 (side-effecting tool with a free-form recipient/amount and no visible bound, no `requestConfirmation(` in the handler); agent ADK-109 (LlmAgent no description); first ADK TS pack; META-004 no longer fires | | **Google ADK** | Go | ◐ dep-scan only (`go.mod`: `google.golang.org/adk`). Recon-only name `google-adk-go`: not mapped to an SDK for `scan`, used by `trustabl forge` to select the Go runtime-tracing recipe; forge then lists only the language-independent `google_adk` rules, with a note that no Go rules exist | ❌ none | ❌ none | | **Google ADK** | Java / Kotlin | ❌ | ❌ | ❌ | -| **LangChain / LangGraph** | Python | ✅ dep-scan (`langchain` / `langgraph` needles, all manifests) + file inventory | ✅ tools (`@tool` decorator — import-gated to disambiguate from the Claude SDK's `@tool`; `StructuredTool` / `Tool` factories + `.from_function`), agents (`create_react_agent`, `create_agent`, `AgentExecutor` + `AgentExecutor.from_agent_and_tools`; positional `tools` captured), raw `StateGraph` graphs (`StateGraph(...)` → `AgentDef` Class `StateGraph`, import-bound; `compile()` kwargs captured; `ToolNode` / `bind_tools` tools resolved), dangerous built-ins (`PythonREPLTool` / `PythonAstREPLTool` / `ShellTool` / `Requests*` → `HostedToolDef` edges) | ✅ tool LC-001 (no description), LC-002 (untyped params), LC-003 (shell), LC-004 (code-exec), LC-005 (SSRF), LC-006 (`return_direct`), LC-025 (side-effecting tool with a free-form recipient/amount and no visible bound), LC-026 (allow-listed host, https:// not pinned), LC-115 (`add_node` function — Kind `langgraph_node` — with a side effect and no `interrupt(` gate); agent LC-101 (code-exec/shell built-in — incl. raw `StateGraph`), LC-102 (AgentExecutor no `max_iterations`), LC-113 (AgentExecutor no `max_execution_time`), LC-114 (`StateGraph` compiled with no `checkpointer`, silent on `langgraph.json` repos); repo LC-201 (no agent-guidance doc) | +| **LangChain / LangGraph** | Python | ✅ dep-scan (`langchain` / `langgraph` needles, all manifests) + file inventory | ✅ tools (`@tool` decorator — import-gated to disambiguate from the Claude SDK's `@tool`; `StructuredTool` / `Tool` factories + `.from_function`), agents (`create_react_agent`, `create_agent`, `AgentExecutor` + `AgentExecutor.from_agent_and_tools`; positional `tools` captured), raw `StateGraph` graphs (`StateGraph(...)` → `AgentDef` Class `StateGraph`, import-bound; `compile()` kwargs captured; `ToolNode` / `bind_tools` tools resolved), dangerous built-ins (`PythonREPLTool` / `PythonAstREPLTool` / `ShellTool` / `Requests*` → `HostedToolDef` edges) | ✅ tool LC-001 (no description), LC-002 (untyped params), LC-003 (shell), LC-004 (code-exec), LC-005 (SSRF), LC-006 (`return_direct`), LC-025 (side-effecting tool with a free-form recipient/amount and no visible bound), LC-026 (allow-listed host, https:// not pinned), LC-115 (`add_node` function — Kind `langgraph_node` — with a side effect and no `interrupt(` gate); agent LC-101 (code-exec/shell built-in — incl. raw `StateGraph`), LC-102 (AgentExecutor no `max_iterations`), LC-113 (AgentExecutor no `max_execution_time`), LC-114 (`StateGraph` compiled with no `checkpointer`, silent on `langgraph.json` repos); repo LC-201 (no agent-guidance doc); tool-call claims (EBTE) LC-028..030 — Python decorated tools only | | **LangChain / LangGraph** | TypeScript | ✅ dep-scan (`@langchain/*` / `langchain` / `langgraph`) + file inventory | ✅ tools (`tool(fn, {...})` factory — import-gated, config from arg 1; `DynamicStructuredTool` / `DynamicTool`), agents (`createReactAgent`, `createAgent`, `new AgentExecutor`) | ◐ tool LC-010 (no description), LC-011 (shell), LC-012 (code-exec), LC-013 (SSRF), LC-014 (`returnDirect`), LC-027 (allow-listed host, https:// not pinned); agent LC-111 (AgentExecutor no `maxIterations`). Provider hosted tools (`shell()` / `bash_*` / `applyPatch`) and the raw `StateGraph` graph agent are documented gaps | | **CrewAI** | Python | ✅ dep-scan (`crewai` / `crewai-tools`) + file inventory | ✅ agents (`Agent(...)`, import-gated to `crewai` so it doesn't collide with OpenAI/ADK `Agent`), tools (`@tool` decorator routed in `kindFromDecorators` by import binding; `Tool(fn)` factory), dangerous built-ins (`CodeInterpreterTool` / `FileReadTool` / scrape+RAG tools → `HostedToolDef`) | ✅ tool CREW-001..006 (no-desc, untyped, code-exec, shell, SSRF, idempotency), CREW-108 (`result_as_answer`), CREW-014 (side-effecting tool with a free-form recipient/amount and no visible bound; no tool-level gate exists), CREW-015 (allow-listed host, https:// not pinned); agent CREW-101..104 (`allow_code_execution`, `code_execution_mode=unsafe`, wired `CodeInterpreterTool`, `allow_delegation`), CREW-106 (unconstrained `FileReadTool`), CREW-107 (URL-fetching tools), CREW-110 (no explicit `max_iter`), CREW-111 (no `max_execution_time`); repo CREW-201. `class X(BaseTool)` and `Crew(...)` orchestration are v1 gaps | | **AutoGen / AG2** | Python | ✅ dep-scan (`pyautogen` / `ag2` / `autogen-agentchat`) + file inventory | ✅ two import gates (AG2/0.2 `autogen`; Microsoft v0.4 `autogen_agentchat`/`_core`/`_ext`): agents `ConversableAgent` / `UserProxyAgent` / `AssistantAgent` / `GroupChat` / `GroupChatManager` / `CodeExecutorAgent`; tools `register_function(fn,...)` + stacked `@x.register_for_llm` / `@x.register_for_execution` attribute decorators; nested `code_execution_config` dict captured | ✅ agent AG2-001 (`use_docker=False`), 002 (`human_input_mode=NEVER` + code exec), 004 (`GroupChat` no `max_round`), 005 (`AssistantAgent` code exec), 006 (no `max_consecutive_auto_reply`), AG2-021 (no wall-clock timeout around `initiate_chat` / `a_initiate_chat` / `run` / `a_run` / `run_stream`; credits `cancellation_token`); tool AG2-007..012 (no-desc, untyped, shell, code-exec, SSRF, timeout), AG2-020 (side-effecting tool with a free-form recipient/amount and no visible bound; no tool-level gate exists), AG2-022 (allow-listed host, https:// not pinned); repo AG2-201. AG2-003 (v0.4 executor-class), the `register_function` caller/executor edge, and AG2 `@tool` are v1 gaps | diff --git a/internal/analysis/claim_use.go b/internal/analysis/claim_use.go new file mode 100644 index 00000000..97d0afde --- /dev/null +++ b/internal/analysis/claim_use.go @@ -0,0 +1,180 @@ +package analysis + +import ( + "strings" + + sitter "github.com/smacker/go-tree-sitter" + + "github.com/trustabl/trustabl/internal/analysis/astutil" + "github.com/trustabl/trustabl/internal/models" +) + +// claimParam is the reserved parameter an EBTE-style tool takes its +// ToolInvocationExplanation in. +const claimParam = "claim" + +// claimSinkCallees are the bare callees whose arguments are output, not a +// decision: a claim that only reaches one of them is passed through. +var claimSinkCallees = map[string]bool{ + "print": true, "str": true, "repr": true, "pprint": true, + "json.dumps": true, "json.dump": true, "pprint.pprint": true, +} + +// claimSinkMethods are logger method names. Matched on the method alone +// (logger.info, self.log.debug, logging.warning), since the receiver's type is +// not resolved. +var claimSinkMethods = map[string]bool{ + "debug": true, "info": true, "warning": true, "warn": true, + "error": true, "exception": true, "critical": true, "log": true, +} + +// pythonClaimUse classifies how a Python tool function uses its `claim` +// parameter, returning one of the models.ClaimUse* values, or "" when the +// function declares no such parameter. +// +// dec is the function's decorated_definition, or nil. A tool with more than +// one decorator is ClaimUseWrapped: the extra decorator may strip and check the +// claim before the body runs, and that cannot be seen from here. +// +// Each read of `claim` in the body is followed outward to the first thing that +// consumes it. The innermost call it is an argument of decides: an output call +// (print, a logger method, json.dumps, str, repr) makes it a passthrough read, +// any other call makes it checked. A read that reaches a return statement +// without passing through a call is a passthrough read; one that reaches any +// other statement (a condition, a comparison, an assignment) is checked. The +// fact is checked if any read is, passthrough if every read is, unused if +// there are none. Anything not recognized as output counts as checked, so an +// unusual shape is silent rather than a false finding. +func pythonClaimUse(fn, dec *sitter.Node, src []byte) string { + declared := false + for _, p := range astutil.FunctionParams(fn, src) { + if p == claimParam { + declared = true + break + } + } + if !declared { + return "" + } + if dec != nil && len(astutil.Decorators(dec)) > 1 { + return models.ClaimUseWrapped + } + body := fn.ChildByFieldName("body") + if body == nil { + return models.ClaimUseUnused + } + + reads, passthrough := 0, 0 + astutil.Walk(body, func(n *sitter.Node) bool { + if n.Type() != "identifier" || astutil.NodeText(n, src) != claimParam || !isClaimRead(n) { + return true + } + reads++ + if claimReadIsPassthrough(n, fn, src) { + passthrough++ + } + return true + }) + switch { + case reads == 0: + return models.ClaimUseUnused + case passthrough == reads: + return models.ClaimUsePassthrough + default: + return models.ClaimUseChecked + } +} + +// isClaimRead rejects an identifier spelled `claim` that is not a read of the +// variable: the attribute name in `x.claim`, or the keyword in `f(claim=...)`. +func isClaimRead(n *sitter.Node) bool { + p := n.Parent() + if p == nil { + return true + } + switch p.Type() { + case "attribute": + if a := p.ChildByFieldName("attribute"); a != nil && a.Equal(n) { + return false + } + case "keyword_argument": + if k := p.ChildByFieldName("name"); k != nil && k.Equal(n) { + return false + } + } + return true +} + +// claimReadIsPassthrough follows one read of `claim` outward, stopping at fn. +func claimReadIsPassthrough(n, fn *sitter.Node, src []byte) bool { + cur := n + for p := n.Parent(); p != nil && !p.Equal(fn); cur, p = p, p.Parent() { + switch p.Type() { + case "call": + args := p.ChildByFieldName("arguments") + if args == nil || !args.Equal(cur) { + // claim is the callee's receiver (claim.get(...)): the call + // yields a value read from the claim, so keep following it. + continue + } + return isClaimSinkCallee(p.ChildByFieldName("function"), src) + case "return_statement": + return true + case "expression_statement", "assignment", "augmented_assignment", + "if_statement", "elif_clause", "while_statement", "assert_statement", + "raise_statement", "for_statement", "with_statement", "match_statement": + return false + } + } + return false +} + +func isClaimSinkCallee(callee *sitter.Node, src []byte) bool { + if callee == nil { + return false + } + text := astutil.NodeText(callee, src) + if claimSinkCallees[text] { + return true + } + if callee.Type() == "attribute" { + if i := strings.LastIndex(text, "."); i >= 0 { + return claimSinkMethods[text[i+1:]] + } + } + return false +} + +// openAIToolHasInputGuardrail reports whether an OpenAI Agents SDK tool has a +// tool input guardrail attached, either as the decorator kwarg +// (@function_tool(tool_input_guardrails=[...])) or by assignment anywhere in the +// file (send_email.tool_input_guardrails = [...], the shape the SDK's own +// examples use). An input guardrail runs before the tool with the call's +// arguments, so it is where a mediation step can read the claim. Whether it +// actually checks the claim is not verified. +func openAIToolHasInputGuardrail(t models.ToolDef, root *sitter.Node, src []byte) bool { + if _, ok := t.Config["tool_input_guardrails"]; ok { + return true + } + found := false + astutil.Walk(root, func(n *sitter.Node) bool { + if found { + return false + } + if n.Type() != "assignment" { + return true + } + left := n.ChildByFieldName("left") + if left == nil || left.Type() != "attribute" { + return true + } + obj, attr := left.ChildByFieldName("object"), left.ChildByFieldName("attribute") + if obj != nil && attr != nil && obj.Type() == "identifier" && + astutil.NodeText(obj, src) == t.Name && astutil.NodeText(attr, src) == "tool_input_guardrails" { + found = true + return false + } + return true + }) + return found +} diff --git a/internal/analysis/claim_use_test.go b/internal/analysis/claim_use_test.go new file mode 100644 index 00000000..39e559e7 --- /dev/null +++ b/internal/analysis/claim_use_test.go @@ -0,0 +1,198 @@ +package analysis_test + +import ( + "testing" + + "github.com/trustabl/trustabl/internal/analysis" + "github.com/trustabl/trustabl/internal/models" +) + +// claimUseOf discovers the single tool in src and returns its claim_use fact +// ("" when the fact is not stamped). +func claimUseOf(t *testing.T, src string) string { + t.Helper() + tools := analysis.DiscoverToolsFromParsed([]analysis.ParsedFile{parsePyFile(t, "tools.py", src)}) + if len(tools) != 1 { + t.Fatalf("expected 1 tool, got %d: %+v", len(tools), tools) + } + return tools[0].Facts[models.FactClaimUse] +} + +func TestClaimUse(t *testing.T) { + cases := []struct { + name string + src string + want string + }{ + { + name: "no claim param: fact not stamped", + src: ` +from agents import function_tool + +@function_tool +def send_email(to: str, body: str) -> str: + return "ok" +`, + want: "", + }, + { + name: "claim declared, never referenced", + src: ` +from agents import function_tool + +@function_tool +def send_email(to: str, body: str, claim: dict) -> str: + return "ok" +`, + want: models.ClaimUseUnused, + }, + { + name: "claim only printed", + src: ` +from agents import function_tool + +@function_tool +def send_email(to: str, body: str, claim: dict) -> str: + print(claim) + return "ok" +`, + want: models.ClaimUsePassthrough, + }, + { + name: "claim only logged and returned", + src: ` +import json, logging +from agents import function_tool +logger = logging.getLogger(__name__) + +@function_tool +def send_email(to: str, body: str, claim: dict) -> str: + logger.info("claim %s", json.dumps(claim)) + logging.debug(f"claim={claim}") + return f"sent with {claim}" +`, + want: models.ClaimUsePassthrough, + }, + { + name: "claim passed to a verifier", + src: ` +from agents import function_tool + +@function_tool +def send_email(to: str, body: str, claim: dict) -> str: + decision = verify_claim(claim, tool="send_email", args={"to": to}) + if decision != "allow": + raise PermissionError("claim does not match route") + return "ok" +`, + want: models.ClaimUseChecked, + }, + { + name: "claim field gates the call", + src: ` +from agents import function_tool + +@function_tool +def send_email(to: str, body: str, claim: dict) -> str: + if claim["selected_tool"] != "send_email": + raise PermissionError("claim does not match route") + return "ok" +`, + want: models.ClaimUseChecked, + }, + { + name: "verifier result logged: the inner call decides", + src: ` +import logging +from agents import function_tool + +@function_tool +def send_email(to: str, body: str, claim: dict) -> str: + logging.info("decision %s", mediate(claim)) + return "ok" +`, + want: models.ClaimUseChecked, + }, + { + name: "extra decorator: a wrapper may mediate", + src: ` +from agents import function_tool + +@function_tool +@mediated +def send_email(to: str, body: str, claim: dict) -> str: + return "ok" +`, + want: models.ClaimUseWrapped, + }, + { + name: "OpenAI tool input guardrail attached by assignment", + src: ` +from agents import function_tool, tool_input_guardrail + +@function_tool +def send_email(to: str, claim: dict) -> str: + return "ok" + +send_email.tool_input_guardrails = [check_claim] +`, + want: models.ClaimUseWrapped, + }, + { + name: "OpenAI tool input guardrail attached by decorator kwarg", + src: ` +from agents import function_tool + +@function_tool(tool_input_guardrails=[check_claim]) +def send_email(to: str, claim: dict) -> str: + print(claim) + return "ok" +`, + want: models.ClaimUseWrapped, + }, + { + name: "an output guardrail does not mediate the call", + src: ` +from agents import function_tool + +@function_tool +def send_email(to: str, claim: dict) -> str: + return "ok" + +send_email.tool_output_guardrails = [check_output] +`, + want: models.ClaimUseUnused, + }, + { + name: "MCP tool, claim unused", + src: ` +from mcp.server.fastmcp import FastMCP +mcp = FastMCP("x") + +@mcp.tool() +def delete_file(path: str, claim: dict) -> str: + return "ok" +`, + want: models.ClaimUseUnused, + }, + { + name: "a local named claim_id is not a reference", + src: ` +from agents import function_tool + +@function_tool +def send_email(to: str, claim: dict) -> str: + claim_id = "x" + return claim_id +`, + want: models.ClaimUseUnused, + }, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + if got := claimUseOf(t, tc.src); got != tc.want { + t.Errorf("claim_use = %q, want %q", got, tc.want) + } + }) + } +} diff --git a/internal/analysis/discovery.go b/internal/analysis/discovery.go index bd6823ef..a801b093 100644 --- a/internal/analysis/discovery.go +++ b/internal/analysis/discovery.go @@ -117,6 +117,13 @@ func toolsInFile(pf ParsedFile) []models.ToolDef { } tool := buildTool(fn, pf, kind) tool.Config = extractDecoratorKwargs(dec, pf.Source) + if use := pythonClaimUse(fn, dec, pf.Source); use != "" { + if kind == models.KindOpenAITool && use != models.ClaimUseChecked && + openAIToolHasInputGuardrail(tool, root, pf.Source) { + use = models.ClaimUseWrapped + } + tool.Facts[models.FactClaimUse] = use + } out = append(out, tool) } diff --git a/internal/models/models.go b/internal/models/models.go index f4a0e6e1..670a6aec 100644 --- a/internal/models/models.go +++ b/internal/models/models.go @@ -212,6 +212,30 @@ type ToolDef struct { Config map[string]string `json:"config,omitempty"` // decorator kwargs } +// FactClaimUse is the ToolDef.Facts key for how a tool uses its EBTE-style +// `claim` argument (a typed ToolInvocationExplanation the model fills in with +// each call). It is stamped only on a Python decorated tool that declares a +// parameter named `claim`; a tool with no such parameter carries no fact, so +// scan output for repos that do not use claims is unchanged. +const FactClaimUse = "claim_use" + +// ClaimUse* are the values of FactClaimUse. +const ( + // ClaimUseUnused: the body never reads `claim`. + ClaimUseUnused = "unused" + // ClaimUsePassthrough: every read of `claim` is an argument to an output + // call (print, a logger method, json.dumps, str, repr) or part of a return + // value, so nothing decides on it. + ClaimUsePassthrough = "passthrough" + // ClaimUseChecked: at least one read of `claim` feeds something other than + // an output call — a verifier call, a comparison, a condition. + ClaimUseChecked = "checked" + // ClaimUseWrapped: something outside the body may mediate the claim — a + // decorator besides the one that registers the tool, or (OpenAI Agents SDK) + // an attached tool input guardrail. Not verified. + ClaimUseWrapped = "wrapped" +) + // ComponentKind labels the type of an agent component the normalizer found // outside of the tool function itself. Components are surfaced for context // and for future detection passes; today's rule engine only fires against diff --git a/internal/rules/evaluator.go b/internal/rules/evaluator.go index f2eb6f71..d20a0545 100644 --- a/internal/rules/evaluator.go +++ b/internal/rules/evaluator.go @@ -455,6 +455,9 @@ func (e MatchExpr) EvaluateTool(t models.ToolDef, pf analysis.ParsedFile) bool { if len(e.ToolDecoratorKwargPresent) > 0 && !PredToolDecoratorKwargPresent(e.ToolDecoratorKwargPresent, t) { return false } + if len(e.ToolClaimUse) > 0 && !PredToolClaimUse(e.ToolClaimUse, t) { + return false + } return true } @@ -477,6 +480,7 @@ var predicatesByScope = map[models.Scope]map[string]bool{ "param_name_matches": true, "call_without_kwarg": true, "call_uses_unnormalized_path_param": true, "tool_decorator_kwarg_value": true, "tool_decorator_kwarg_present": true, + "tool_claim_use": true, }, models.ScopeAgent: { "agent_class": true, "agent_kwarg_present": true, "agent_kwarg_missing": true, @@ -588,6 +592,7 @@ func (e MatchExpr) setPredicateNames() []string { add(e.CallUsesUnnormalizedPathParam != nil, "call_uses_unnormalized_path_param") add(e.ToolDecoratorKwargValue != nil, "tool_decorator_kwarg_value") add(len(e.ToolDecoratorKwargPresent) > 0, "tool_decorator_kwarg_present") + add(len(e.ToolClaimUse) > 0, "tool_claim_use") // Agent scope add(len(e.AgentClass) > 0, "agent_class") add(len(e.AgentKwargPresent) > 0, "agent_kwarg_present") @@ -689,6 +694,23 @@ func (e MatchExpr) outOfScopePredicates(scope models.Scope) []string { return bad } +// toolClaimUseValues collects every tool_claim_use value in the tree, for the +// loader's value check. +func (e MatchExpr) toolClaimUseValues() []string { + var vals []string + vals = append(vals, e.ToolClaimUse...) + for _, sub := range e.All { + vals = append(vals, sub.toolClaimUseValues()...) + } + for _, sub := range e.Any { + vals = append(vals, sub.toolClaimUseValues()...) + } + if e.Not != nil { + vals = append(vals, e.Not.toolClaimUseValues()...) + } + return vals +} + // repoSDKInCodeValues collects every value passed to a repo_has_sdk_in_code // predicate anywhere in the match tree (recursing through all/any/not). The // loader validates these against the real SDK-enum token set so a category diff --git a/internal/rules/loader.go b/internal/rules/loader.go index cdcf814a..91af9383 100644 --- a/internal/rules/loader.go +++ b/internal/rules/loader.go @@ -270,6 +270,13 @@ func loadPolicies(fsys fs.FS, lenient bool) ([]PolicyFile, []string, error) { errs = append(errs, fmt.Errorf("%s: repo_has_sdk_in_code value %q is not a known SDK token (want one of: claude_agent_sdk, openai_agents, google_adk, mcp, langchain, openshell)", tag, sdk)) } } + // tool_claim_use matches a closed value set; a typo silently never + // matches, so the rule never fires. + for _, v := range rule.Match.toolClaimUseValues() { + if !validToolClaimUse(v) { + errs = append(errs, fmt.Errorf("%s: tool_claim_use value %q is not known (want one of: absent, unused, passthrough, checked, wrapped)", tag, v)) + } + } // A degenerate combinator (empty any/all list, or not: over an empty // expression) is an authoring mistake, not intent. if degen := rule.Match.degenerateCombinators(); len(degen) > 0 { diff --git a/internal/rules/policies_test.go b/internal/rules/policies_test.go index d9371abc..86a111d5 100644 --- a/internal/rules/policies_test.go +++ b/internal/rules/policies_test.go @@ -120,6 +120,7 @@ type policyRuleCase struct { kind models.ToolKind // ToolKind for the synthetic tool src string // Python OR TypeScript snippet, per lang toolConfig map[string]string // optional Config override (for decorator-kwarg rules) + toolFacts map[string]string // optional Facts override (for discovery-computed facts) wantFires bool // expected: rule fires for this snippet lang models.Language // "" defaults to python (existing cases) } @@ -3563,6 +3564,188 @@ def deploy(state): def summarize(state): return {"summary": state["text"][:100]} `, wantFires: false}, + + // ─── Tool-call claims (EBTE): OAI-034..036, MCP-033..035, LC-028..030 ── + // claim_use is a discovery fact and parsePy does not run discovery, so + // these cases set it through toolFacts. TestScan_ToolCallClaims runs the + // same shapes through real discovery. + {name: "OAI-034 fires on a side-effecting tool with no claim", ruleID: "OAI-034", kind: models.KindOpenAITool, src: ` +def send_email(to: str, body: str) -> str: + """Send an email.""" + return mailer.send(to, body) +`, wantFires: true}, + {name: "OAI-034 fires on a shell tool with no claim", ruleID: "OAI-034", kind: models.KindOpenAITool, src: ` +def run_report(name: str) -> str: + """Build a report.""" + import subprocess + return subprocess.run(["make", name]).stdout +`, wantFires: true}, + {name: "OAI-034 silent on a read-only tool with no claim", ruleID: "OAI-034", kind: models.KindOpenAITool, src: ` +def get_weather(city: str) -> str: + """Get the weather.""" + return city +`, wantFires: false}, + {name: "OAI-034 silent when the tool takes a claim", ruleID: "OAI-034", kind: models.KindOpenAITool, src: ` +def send_email(to: str, body: str, claim: dict) -> str: + """Send an email.""" + return mailer.send(to, body) +`, toolFacts: map[string]string{models.FactClaimUse: models.ClaimUseChecked}, wantFires: false}, + {name: "OAI-035 fires on an unused claim", ruleID: "OAI-035", kind: models.KindOpenAITool, src: ` +def send_email(to: str, body: str, claim: dict) -> str: + """Send an email.""" + return mailer.send(to, body) +`, toolFacts: map[string]string{models.FactClaimUse: models.ClaimUseUnused}, wantFires: true}, + {name: "OAI-035 silent on a checked claim", ruleID: "OAI-035", kind: models.KindOpenAITool, src: ` +def send_email(to: str, body: str, claim: dict) -> str: + """Send an email.""" + verify_claim(claim) + return mailer.send(to, body) +`, toolFacts: map[string]string{models.FactClaimUse: models.ClaimUseChecked}, wantFires: false}, + {name: "OAI-035 silent on a wrapped claim", ruleID: "OAI-035", kind: models.KindOpenAITool, src: ` +def send_email(to: str, body: str, claim: dict) -> str: + """Send an email.""" + return mailer.send(to, body) +`, toolFacts: map[string]string{models.FactClaimUse: models.ClaimUseWrapped}, wantFires: false}, + {name: "OAI-035 silent with no claim at all", ruleID: "OAI-035", kind: models.KindOpenAITool, src: ` +def send_email(to: str, body: str) -> str: + """Send an email.""" + return mailer.send(to, body) +`, wantFires: false}, + {name: "OAI-036 fires on a claim that is only logged", ruleID: "OAI-036", kind: models.KindOpenAITool, src: ` +def send_email(to: str, body: str, claim: dict) -> str: + """Send an email.""" + logger.info("claim %s", claim) + return mailer.send(to, body) +`, toolFacts: map[string]string{models.FactClaimUse: models.ClaimUsePassthrough}, wantFires: true}, + {name: "OAI-036 silent on a checked claim", ruleID: "OAI-036", kind: models.KindOpenAITool, src: ` +def send_email(to: str, body: str, claim: dict) -> str: + """Send an email.""" + verify_claim(claim) + return mailer.send(to, body) +`, toolFacts: map[string]string{models.FactClaimUse: models.ClaimUseChecked}, wantFires: false}, + {name: "OAI-036 silent on an unused claim", ruleID: "OAI-036", kind: models.KindOpenAITool, src: ` +def send_email(to: str, body: str, claim: dict) -> str: + """Send an email.""" + return mailer.send(to, body) +`, toolFacts: map[string]string{models.FactClaimUse: models.ClaimUseUnused}, wantFires: false}, + {name: "MCP-033 fires on a side-effecting tool with no claim", ruleID: "MCP-033", kind: models.KindMCPTool, src: ` +def send_email(to: str, body: str) -> str: + """Send an email.""" + return mailer.send(to, body) +`, wantFires: true}, + {name: "MCP-033 fires on a shell tool with no claim", ruleID: "MCP-033", kind: models.KindMCPTool, src: ` +def run_report(name: str) -> str: + """Build a report.""" + import subprocess + return subprocess.run(["make", name]).stdout +`, wantFires: true}, + {name: "MCP-033 silent on a read-only tool with no claim", ruleID: "MCP-033", kind: models.KindMCPTool, src: ` +def get_weather(city: str) -> str: + """Get the weather.""" + return city +`, wantFires: false}, + {name: "MCP-033 silent when the tool takes a claim", ruleID: "MCP-033", kind: models.KindMCPTool, src: ` +def send_email(to: str, body: str, claim: dict) -> str: + """Send an email.""" + return mailer.send(to, body) +`, toolFacts: map[string]string{models.FactClaimUse: models.ClaimUseChecked}, wantFires: false}, + {name: "MCP-034 fires on an unused claim", ruleID: "MCP-034", kind: models.KindMCPTool, src: ` +def send_email(to: str, body: str, claim: dict) -> str: + """Send an email.""" + return mailer.send(to, body) +`, toolFacts: map[string]string{models.FactClaimUse: models.ClaimUseUnused}, wantFires: true}, + {name: "MCP-034 silent on a checked claim", ruleID: "MCP-034", kind: models.KindMCPTool, src: ` +def send_email(to: str, body: str, claim: dict) -> str: + """Send an email.""" + verify_claim(claim) + return mailer.send(to, body) +`, toolFacts: map[string]string{models.FactClaimUse: models.ClaimUseChecked}, wantFires: false}, + {name: "MCP-034 silent on a wrapped claim", ruleID: "MCP-034", kind: models.KindMCPTool, src: ` +def send_email(to: str, body: str, claim: dict) -> str: + """Send an email.""" + return mailer.send(to, body) +`, toolFacts: map[string]string{models.FactClaimUse: models.ClaimUseWrapped}, wantFires: false}, + {name: "MCP-034 silent with no claim at all", ruleID: "MCP-034", kind: models.KindMCPTool, src: ` +def send_email(to: str, body: str) -> str: + """Send an email.""" + return mailer.send(to, body) +`, wantFires: false}, + {name: "MCP-035 fires on a claim that is only logged", ruleID: "MCP-035", kind: models.KindMCPTool, src: ` +def send_email(to: str, body: str, claim: dict) -> str: + """Send an email.""" + logger.info("claim %s", claim) + return mailer.send(to, body) +`, toolFacts: map[string]string{models.FactClaimUse: models.ClaimUsePassthrough}, wantFires: true}, + {name: "MCP-035 silent on a checked claim", ruleID: "MCP-035", kind: models.KindMCPTool, src: ` +def send_email(to: str, body: str, claim: dict) -> str: + """Send an email.""" + verify_claim(claim) + return mailer.send(to, body) +`, toolFacts: map[string]string{models.FactClaimUse: models.ClaimUseChecked}, wantFires: false}, + {name: "MCP-035 silent on an unused claim", ruleID: "MCP-035", kind: models.KindMCPTool, src: ` +def send_email(to: str, body: str, claim: dict) -> str: + """Send an email.""" + return mailer.send(to, body) +`, toolFacts: map[string]string{models.FactClaimUse: models.ClaimUseUnused}, wantFires: false}, + {name: "LC-028 fires on a side-effecting tool with no claim", ruleID: "LC-028", kind: models.KindLangChainTool, src: ` +def send_email(to: str, body: str) -> str: + """Send an email.""" + return mailer.send(to, body) +`, wantFires: true}, + {name: "LC-028 fires on a shell tool with no claim", ruleID: "LC-028", kind: models.KindLangChainTool, src: ` +def run_report(name: str) -> str: + """Build a report.""" + import subprocess + return subprocess.run(["make", name]).stdout +`, wantFires: true}, + {name: "LC-028 silent on a read-only tool with no claim", ruleID: "LC-028", kind: models.KindLangChainTool, src: ` +def get_weather(city: str) -> str: + """Get the weather.""" + return city +`, wantFires: false}, + {name: "LC-028 silent when the tool takes a claim", ruleID: "LC-028", kind: models.KindLangChainTool, src: ` +def send_email(to: str, body: str, claim: dict) -> str: + """Send an email.""" + return mailer.send(to, body) +`, toolFacts: map[string]string{models.FactClaimUse: models.ClaimUseChecked}, wantFires: false}, + {name: "LC-029 fires on an unused claim", ruleID: "LC-029", kind: models.KindLangChainTool, src: ` +def send_email(to: str, body: str, claim: dict) -> str: + """Send an email.""" + return mailer.send(to, body) +`, toolFacts: map[string]string{models.FactClaimUse: models.ClaimUseUnused}, wantFires: true}, + {name: "LC-029 silent on a checked claim", ruleID: "LC-029", kind: models.KindLangChainTool, src: ` +def send_email(to: str, body: str, claim: dict) -> str: + """Send an email.""" + verify_claim(claim) + return mailer.send(to, body) +`, toolFacts: map[string]string{models.FactClaimUse: models.ClaimUseChecked}, wantFires: false}, + {name: "LC-029 silent on a wrapped claim", ruleID: "LC-029", kind: models.KindLangChainTool, src: ` +def send_email(to: str, body: str, claim: dict) -> str: + """Send an email.""" + return mailer.send(to, body) +`, toolFacts: map[string]string{models.FactClaimUse: models.ClaimUseWrapped}, wantFires: false}, + {name: "LC-029 silent with no claim at all", ruleID: "LC-029", kind: models.KindLangChainTool, src: ` +def send_email(to: str, body: str) -> str: + """Send an email.""" + return mailer.send(to, body) +`, wantFires: false}, + {name: "LC-030 fires on a claim that is only logged", ruleID: "LC-030", kind: models.KindLangChainTool, src: ` +def send_email(to: str, body: str, claim: dict) -> str: + """Send an email.""" + logger.info("claim %s", claim) + return mailer.send(to, body) +`, toolFacts: map[string]string{models.FactClaimUse: models.ClaimUsePassthrough}, wantFires: true}, + {name: "LC-030 silent on a checked claim", ruleID: "LC-030", kind: models.KindLangChainTool, src: ` +def send_email(to: str, body: str, claim: dict) -> str: + """Send an email.""" + verify_claim(claim) + return mailer.send(to, body) +`, toolFacts: map[string]string{models.FactClaimUse: models.ClaimUseChecked}, wantFires: false}, + {name: "LC-030 silent on an unused claim", ruleID: "LC-030", kind: models.KindLangChainTool, src: ` +def send_email(to: str, body: str, claim: dict) -> str: + """Send an email.""" + return mailer.send(to, body) +`, toolFacts: map[string]string{models.FactClaimUse: models.ClaimUseUnused}, wantFires: false}, } // policyRepoRuleCases covers repo-scoped rules. @@ -7349,6 +7532,9 @@ func TestPolicyRules(t *testing.T) { if tc.toolConfig != nil { tool.Config = tc.toolConfig } + for k, v := range tc.toolFacts { + tool.Facts[k] = v + } inv := models.RepoInventory{} if !d.Applies(tool) { if tc.wantFires { diff --git a/internal/rules/predicates.go b/internal/rules/predicates.go index 40d9dd57..290142e4 100644 --- a/internal/rules/predicates.go +++ b/internal/rules/predicates.go @@ -359,6 +359,49 @@ func PredNameHasPrefix(prefixes []string, t models.ToolDef) bool { return false } +// ClaimUseAbsent is the tool_claim_use value for a tool that declares no +// `claim` parameter. It is decided from ToolDef.ParamNames, so it holds for +// every discovery path; the other values come from the FactClaimUse fact, +// which only Python decorated-tool discovery stamps. +const ClaimUseAbsent = "absent" + +// PredToolClaimUse reports whether the tool's use of its `claim` argument is +// one of uses. A tool that declares `claim` but carries no FactClaimUse fact +// (a discovery path that does not classify it) matches no value: unknown is +// never a match, so the rule stays silent rather than guessing. +func PredToolClaimUse(uses []string, t models.ToolDef) bool { + got := t.Facts[models.FactClaimUse] + declared := false + for _, p := range t.ParamNames { + if p == "claim" { + declared = true + break + } + } + for _, u := range uses { + if u == ClaimUseAbsent { + if !declared { + return true + } + continue + } + if declared && got != "" && u == got { + return true + } + } + return false +} + +// validToolClaimUse reports whether v is a value tool_claim_use can match. +func validToolClaimUse(v string) bool { + switch v { + case ClaimUseAbsent, models.ClaimUseUnused, models.ClaimUsePassthrough, + models.ClaimUseChecked, models.ClaimUseWrapped: + return true + } + return false +} + func PredHasBodyText(needles []string, t models.ToolDef, pf analysis.ParsedFile) bool { var body string if root := analysis.FindFunctionNode(t, pf); root != nil { diff --git a/internal/rules/predicates_claim_test.go b/internal/rules/predicates_claim_test.go new file mode 100644 index 00000000..9a8eecde --- /dev/null +++ b/internal/rules/predicates_claim_test.go @@ -0,0 +1,75 @@ +package rules_test + +import ( + "strings" + "testing" + + "github.com/trustabl/trustabl/internal/models" + "github.com/trustabl/trustabl/internal/rules" +) + +func TestPredToolClaimUse(t *testing.T) { + withClaim := func(use string) models.ToolDef { + td := models.ToolDef{ParamNames: []string{"to", "claim"}, Facts: map[string]string{}} + if use != "" { + td.Facts[models.FactClaimUse] = use + } + return td + } + noClaim := models.ToolDef{ParamNames: []string{"to", "body"}} + + cases := []struct { + name string + uses []string + tool models.ToolDef + want bool + }{ + {"absent matches a tool with no claim param", []string{"absent"}, noClaim, true}, + {"absent does not match a declared claim", []string{"absent"}, withClaim(models.ClaimUseUnused), false}, + {"unused matches its fact", []string{"unused"}, withClaim(models.ClaimUseUnused), true}, + {"passthrough matches its fact", []string{"passthrough"}, withClaim(models.ClaimUsePassthrough), true}, + {"checked is not unused", []string{"unused", "passthrough"}, withClaim(models.ClaimUseChecked), false}, + {"wrapped is not unused", []string{"unused"}, withClaim(models.ClaimUseWrapped), false}, + // A discovery path that does not classify the claim leaves no fact: + // unknown is never a match, for any value. + {"declared claim with no fact matches nothing", []string{"absent", "unused", "passthrough", "checked", "wrapped"}, withClaim(""), false}, + // A stale fact on a tool with no claim param cannot match. + {"fact without param is ignored", []string{"unused"}, models.ToolDef{Facts: map[string]string{models.FactClaimUse: models.ClaimUseUnused}}, false}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + if got := rules.PredToolClaimUse(tc.uses, tc.tool); got != tc.want { + t.Errorf("PredToolClaimUse(%v) = %v, want %v", tc.uses, got, tc.want) + } + }) + } +} + +func TestLoader_RejectsUnknownToolClaimUse(t *testing.T) { + const pack = `policy: + id: test_policy + name: Test + category: openai_sdk + description: Test. +rules: + - id: TEST-001 + title: Test rule + scope: tool + severity: medium + confidence: 0.5 + applies_to: + - openai_tool + match: + not: + tool_claim_use: [chekced] + explanation: Some explanation. + fix: Some fix. +` + _, err := rules.Load(makeFS(map[string]string{"test.yaml": pack})) + if err == nil { + t.Fatal("expected an error for an unknown tool_claim_use value, got nil") + } + if !strings.Contains(err.Error(), `tool_claim_use value "chekced"`) { + t.Errorf("error does not name the bad value: %v", err) + } +} diff --git a/internal/rules/schema.go b/internal/rules/schema.go index 47bc5844..e05fcad2 100644 --- a/internal/rules/schema.go +++ b/internal/rules/schema.go @@ -80,6 +80,10 @@ type MatchExpr struct { ToolDecoratorKwargValue *ToolDecoratorKwargValueExpr `yaml:"tool_decorator_kwarg_value,omitempty"` ToolDecoratorKwargPresent []string `yaml:"tool_decorator_kwarg_present,omitempty"` + // ToolClaimUse: how the tool uses its EBTE-style `claim` argument; see + // PredToolClaimUse and validToolClaimUse. + ToolClaimUse []string `yaml:"tool_claim_use,omitempty"` + // Agent-scope predicates AgentClass []string `yaml:"agent_class,omitempty"` AgentKwargPresent []string `yaml:"agent_kwarg_present,omitempty"` diff --git a/internal/rules/schema.yaml b/internal/rules/schema.yaml index 917582ee..b3840183 100644 --- a/internal/rules/schema.yaml +++ b/internal/rules/schema.yaml @@ -218,6 +218,21 @@ rules: # Reserve new uses for a known constant that has no structural # signature. # +# tool_claim_use: [absent | unused | passthrough | checked | wrapped, ...] +# How the tool uses its EBTE-style `claim` argument (a typed +# ToolInvocationExplanation the model fills in with each call). True when +# the tool's use is any listed value. `absent`: no parameter named +# `claim` (read from param_names, so it holds for every discovery path). +# The other values come from the claim_use fact, which only Python +# decorated-tool discovery stamps: `unused` (never read), `passthrough` +# (every read is an argument to print / a logger method / json.dumps / +# str / repr, or part of a return value), `checked` (some read feeds a +# verifier call, a comparison or a condition), `wrapped` (an extra +# decorator besides the registering one, which may mediate; not +# verified; for an OpenAI Agents SDK tool an attached tool input guardrail +# counts too). A tool that declares `claim` without the fact matches no +# value. The loader rejects any other value. +# # has_description_text: [substring, substring, ...] # The tool's description (docstring for Python, the explicit # description argument for TypeScript) contains any listed substring. diff --git a/internal/rules/schema_version.go b/internal/rules/schema_version.go index 449cee8c..e7e44dba 100644 --- a/internal/rules/schema_version.go +++ b/internal/rules/schema_version.go @@ -27,4 +27,4 @@ package rules // that rule (safe) instead of mis-evaluating a predicate whose meaning changed. // That rename discipline is what keeps lenient loading safe without a hard // version gate. -const SupportedSchemaVersion = 26 +const SupportedSchemaVersion = 27 diff --git a/internal/scanner/claims_scan_test.go b/internal/scanner/claims_scan_test.go new file mode 100644 index 00000000..e7bc3827 --- /dev/null +++ b/internal/scanner/claims_scan_test.go @@ -0,0 +1,98 @@ +package scanner_test + +import "testing" + +// TestScan_ToolCallClaims runs the tool-call-claims ladder (OAI-034..036, +// MCP-033..035, LC-028..030) through real discovery, so a change that stops +// discovery from stamping claim_use fails here rather than silently killing +// the rules. The per-rule table in policies_test.go sets the fact by hand. +func TestScan_ToolCallClaims(t *testing.T) { + type sdk struct { + manifest, header, decorator string + absent, unused, passthrough string + } + sdks := map[string]sdk{ + "openai": { + manifest: "[project]\nname = \"f\"\ndependencies = [\"openai-agents\"]\n", + header: "from agents import function_tool\n", + decorator: "@function_tool", + absent: "OAI-034", unused: "OAI-035", passthrough: "OAI-036", + }, + "mcp": { + manifest: "[project]\nname = \"f\"\ndependencies = [\"mcp\"]\n", + header: "from mcp.server.fastmcp import FastMCP\nmcp = FastMCP(\"x\")\n", + decorator: "@mcp.tool()", + absent: "MCP-033", unused: "MCP-034", passthrough: "MCP-035", + }, + "langchain": { + manifest: "[project]\nname = \"f\"\ndependencies = [\"langchain-core\"]\n", + header: "from langchain_core.tools import tool\n", + decorator: "@tool", + absent: "LC-028", unused: "LC-029", passthrough: "LC-030", + }, + } + bodies := map[string]string{ + "absent": ` +def send_email(to: str, body: str) -> str: + """Send an email.""" + return "ok" +`, + "unused": ` +def send_email(to: str, body: str, claim: dict) -> str: + """Send an email.""" + return "ok" +`, + "passthrough": ` +def send_email(to: str, body: str, claim: dict) -> str: + """Send an email.""" + print(claim) + return "ok" +`, + "checked": ` +def send_email(to: str, body: str, claim: dict) -> str: + """Send an email.""" + if verify_claim(claim, tool="send_email") != "allow": + raise PermissionError("claim does not match route") + return "ok" +`, + } + for name, s := range sdks { + rungs := map[string]string{"absent": s.absent, "unused": s.unused, "passthrough": s.passthrough} + for shape, body := range bodies { + files := map[string]string{ + "pyproject.toml": s.manifest, + "tools.py": s.header + "\n" + s.decorator + body, + } + for rung, ruleID := range rungs { + want := rung == shape // the ladder is exclusive: one rung per shape, none for checked + t.Run(name+"/"+shape+"/"+ruleID, func(t *testing.T) { + if got := scanFires(t, files, ruleID); got != want { + t.Errorf("%s on a %s claim: fired=%v, want %v", ruleID, shape, got, want) + } + }) + } + } + } +} + +// An OpenAI tool whose claim is read by an attached tool input guardrail, the +// SDK's own pre-tool hook, is not an unchecked claim. +func TestScan_ToolCallClaims_OpenAIInputGuardrailSilencesUnused(t *testing.T) { + files := map[string]string{ + "pyproject.toml": "[project]\nname = \"f\"\ndependencies = [\"openai-agents\"]\n", + "tools.py": `from agents import function_tool, tool_input_guardrail + +@function_tool +def send_email(to: str, body: str, claim: dict) -> str: + """Send an email.""" + return "ok" + +send_email.tool_input_guardrails = [check_claim] +`, + } + for _, id := range []string{"OAI-034", "OAI-035", "OAI-036"} { + if scanFires(t, files, id) { + t.Errorf("%s fired on a tool whose claim is mediated by a tool input guardrail", id) + } + } +} diff --git a/testdata/rules-fixture/langchain/tool_call_claims.yaml b/testdata/rules-fixture/langchain/tool_call_claims.yaml new file mode 100644 index 00000000..a68b8201 --- /dev/null +++ b/testdata/rules-fixture/langchain/tool_call_claims.yaml @@ -0,0 +1,138 @@ +policy: + id: langchain_tool_call_claims + name: LangChain tool-call claims (EBTE) + category: langchain + description: > + Rules that flag a LangChain tool whose calls are not bound to a typed action + claim checked outside the model, after Explanation-Bound Tool Execution + (EBTE, arXiv 2607.25364). The claim is a reserved `claim` argument (a + ToolInvocationExplanation: intent class, selected tool, expected effect, + risk tier, uncertainty, untrusted-context dependencies, evidence digests) + that a mediation step compares with server-held facts before the call has + any effect. The three rules form a ladder over the tool's claim_use fact: no + claim on a side-effecting tool, a claim nothing reads, and a claim that is + only written out or returned. A checked claim, or a tool with an extra + decorator that may mediate, is silent. + +rules: + - id: LC-028 + title: Side-effecting tool takes no structured claim for its calls + severity: low + confidence: 0.5 + language: python + applies_to: + - langchain_tool + scope: tool + match: + all: + - tool_claim_use: [absent] + - any: + - has_shell_call: true + - has_code_exec_call: true + - has_write_call: true + - name_has_prefix: + - send + - notify + - email + - post_ + - publish + - delete + - remove + - drop + - purge + - refund + - charge + - payout + - pay_ + - transfer + - deploy + - execute + - write + - update + - create + - insert + - upsert + explanation: > + This LangChain `@tool` function has a side effect — it shells out, + executes code, writes a file, or its name says it sends, deletes, pays, + deploys, creates or updates — and it takes no `claim` argument, so nothing + about why the model chose this call or what it expects the call to do + reaches the code that runs it. The model's own reasoning is free text: it + can be wrong, written after the fact, or steered by a poisoned tool result + or document, and nothing can compare it with the task, the policy or the + arguments before the effect happens. This is a defense-in-depth gap, not a + bug on its own: the tool's existing controls still apply, and a claim + never grants authority. The rule only reads the parameter list, so a claim + carried some other way (a separate intent tool, a context object) still + fires. + fix: > + Add a reserved `claim` parameter typed as a ToolInvocationExplanation + (schema_version, intent_class, selected_tool, expected_effect, risk_tier, + uncertainty, untrusted_context, evidence_refs; an optional summary that no + check reads) and check it in a mediation step before the effect, in one + shared decorator stacked under `@tool`, or a graph node that runs before + the tool node. The step denies a missing or mismatched claim, sends an + unresolved fact or a high-risk tool to review, and combines with the + existing controls so the most restrictive outcome wins. Map a review + outcome to `interrupt(` so the graph pauses for a person. + + - id: LC-029 + title: Tool declares a claim argument that nothing checks + severity: medium + confidence: 0.6 + language: python + applies_to: + - langchain_tool + scope: tool + match: + tool_claim_use: [unused] + explanation: > + This LangChain `@tool` function takes a `claim` argument, so the model + fills in a typed claim about the call, but the tool body never reads it + and the tool has no decorator besides the one that registers it, so no + mediation step visible here can check it. A claim nobody checks is worse + than none: reviewers and auditors see a claim-carrying tool and assume the + call was verified against the task and the policy, while a prompt-injected + or mistaken call runs exactly as it would have without one. The rule + cannot see a mediation step that lives in a dispatch layer outside the + tool (custom runner code, server middleware), so that shape also fires. + fix: > + Check the claim before the effect, in one shared decorator stacked under + `@tool`, or a graph node that runs before the tool node: remove it from + the arguments, validate it, resolve the facts server-side (tool registry, + the task's authorized intent, policy, route), and compare. Deny a + conflict, review anything unresolved, and run the tool only on allow. Map + a review outcome to `interrupt(` so the graph pauses for a person. If the + claim is checked in a dispatch layer this rule cannot see, move the check + into a decorator on the tool so the binding is visible where the tool is + defined. + + - id: LC-030 + title: Tool's claim argument is only written out or returned, never checked + severity: medium + confidence: 0.55 + language: python + applies_to: + - langchain_tool + scope: tool + match: + tool_claim_use: [passthrough] + explanation: > + This LangChain `@tool` function reads its `claim` argument, but every read + only writes it out — to print, a logger, json.dumps, str or repr — or puts + it in the return value. Nothing compares the claim with the arguments, the + task or the policy before the side effect runs, so a claim that + contradicts what the call actually does is recorded rather than stopped. + An audit trail of unchecked claims is evidence of what the model said, not + of what was allowed. The rule follows each read to the first call it is an + argument of within the tool body; a check done inside a helper the claim + is first copied into (`c = claim`) is counted as checked and stays silent. + fix: > + Pass the claim to a verifier before the effect and act on its outcome: + deny on a conflict between the claim and the arguments, task or policy, + review on anything unresolved or high risk, and run the tool only on + allow. Keep writing the outcome and the per-check results to the audit + trail, but not the raw claim summary or untrusted content. Do it in one + shared decorator stacked under `@tool`, or a graph node that runs before + the tool node. Map a review outcome to `interrupt(` so the graph pauses + for a person. diff --git a/testdata/rules-fixture/manifest.yaml b/testdata/rules-fixture/manifest.yaml index 92f97d39..232ca8d9 100644 --- a/testdata/rules-fixture/manifest.yaml +++ b/testdata/rules-fixture/manifest.yaml @@ -11,4 +11,4 @@ # # This file is metadata, not a rule: the engine's loader skips manifest.yaml # when walking the pack for policy files. -schema_version: 26 +schema_version: 27 diff --git a/testdata/rules-fixture/mcp/tool_call_claims.yaml b/testdata/rules-fixture/mcp/tool_call_claims.yaml new file mode 100644 index 00000000..3219f166 --- /dev/null +++ b/testdata/rules-fixture/mcp/tool_call_claims.yaml @@ -0,0 +1,141 @@ +policy: + id: mcp_tool_call_claims + name: MCP server tool-call claims (EBTE) + category: mcp + description: > + Rules that flag a MCP server tool whose calls are not bound to a typed + action claim checked outside the model, after Explanation-Bound Tool + Execution (EBTE, arXiv 2607.25364). The claim is a reserved `claim` argument + (a ToolInvocationExplanation: intent class, selected tool, expected effect, + risk tier, uncertainty, untrusted-context dependencies, evidence digests) + that a mediation step compares with server-held facts before the call has + any effect. The three rules form a ladder over the tool's claim_use fact: no + claim on a side-effecting tool, a claim nothing reads, and a claim that is + only written out or returned. A checked claim, or a tool with an extra + decorator that may mediate, is silent. + +rules: + - id: MCP-033 + title: Side-effecting tool takes no structured claim for its calls + severity: low + confidence: 0.5 + language: python + applies_to: + - mcp_tool + scope: tool + match: + all: + - tool_claim_use: [absent] + - any: + - has_shell_call: true + - has_code_exec_call: true + - has_write_call: true + - name_has_prefix: + - send + - notify + - email + - post_ + - publish + - delete + - remove + - drop + - purge + - refund + - charge + - payout + - pay_ + - transfer + - deploy + - execute + - write + - update + - create + - insert + - upsert + explanation: > + This MCP tool handler (`@mcp.tool()` / `@server.tool()`) has a side effect + — it shells out, executes code, writes a file, or its name says it sends, + deletes, pays, deploys, creates or updates — and it takes no `claim` + argument, so nothing about why the model chose this call or what it + expects the call to do reaches the code that runs it. The model's own + reasoning is free text: it can be wrong, written after the fact, or + steered by a poisoned tool result or document, and nothing can compare it + with the task, the policy or the arguments before the effect happens. This + is a defense-in-depth gap, not a bug on its own: the tool's existing + controls still apply, and a claim never grants authority. The rule only + reads the parameter list, so a claim carried some other way (a separate + intent tool, a context object) still fires. + fix: > + Add a reserved `claim` parameter typed as a ToolInvocationExplanation + (schema_version, intent_class, selected_tool, expected_effect, risk_tier, + uncertainty, untrusted_context, evidence_refs; an optional summary that no + check reads) and check it in a mediation step before the effect, in the + tool handler, before any side effect, or one shared decorator every + handler goes through. The step denies a missing or mismatched claim, sends + an unresolved fact or a high-risk tool to review, and combines with the + existing controls so the most restrictive outcome wins. A claim from the + client is as untrusted as any other argument; the facts it is compared + with must come from the server. + + - id: MCP-034 + title: Tool declares a claim argument that nothing checks + severity: medium + confidence: 0.6 + language: python + applies_to: + - mcp_tool + scope: tool + match: + tool_claim_use: [unused] + explanation: > + This MCP tool handler (`@mcp.tool()` / `@server.tool()`) takes a `claim` + argument, so the model fills in a typed claim about the call, but the tool + body never reads it and the tool has no decorator besides the one that + registers it, so no mediation step visible here can check it. A claim + nobody checks is worse than none: reviewers and auditors see a + claim-carrying tool and assume the call was verified against the task and + the policy, while a prompt-injected or mistaken call runs exactly as it + would have without one. The rule cannot see a mediation step that lives in + a dispatch layer outside the tool (custom runner code, server middleware), + so that shape also fires. + fix: > + Check the claim before the effect, in the tool handler, before any side + effect, or one shared decorator every handler goes through: remove it from + the arguments, validate it, resolve the facts server-side (tool registry, + the task's authorized intent, policy, route), and compare. Deny a + conflict, review anything unresolved, and run the tool only on allow. A + claim from the client is as untrusted as any other argument; the facts it + is compared with must come from the server. If the claim is checked in a + dispatch layer this rule cannot see, move the check into a decorator on + the tool so the binding is visible where the tool is defined. + + - id: MCP-035 + title: Tool's claim argument is only written out or returned, never checked + severity: medium + confidence: 0.55 + language: python + applies_to: + - mcp_tool + scope: tool + match: + tool_claim_use: [passthrough] + explanation: > + This MCP tool handler (`@mcp.tool()` / `@server.tool()`) reads its `claim` + argument, but every read only writes it out — to print, a logger, + json.dumps, str or repr — or puts it in the return value. Nothing compares + the claim with the arguments, the task or the policy before the side + effect runs, so a claim that contradicts what the call actually does is + recorded rather than stopped. An audit trail of unchecked claims is + evidence of what the model said, not of what was allowed. The rule follows + each read to the first call it is an argument of within the tool body; a + check done inside a helper the claim is first copied into (`c = claim`) is + counted as checked and stays silent. + fix: > + Pass the claim to a verifier before the effect and act on its outcome: + deny on a conflict between the claim and the arguments, task or policy, + review on anything unresolved or high risk, and run the tool only on + allow. Keep writing the outcome and the per-check results to the audit + trail, but not the raw claim summary or untrusted content. Do it in the + tool handler, before any side effect, or one shared decorator every + handler goes through. A claim from the client is as untrusted as any other + argument; the facts it is compared with must come from the server. diff --git a/testdata/rules-fixture/openai_sdk/tool_call_claims.yaml b/testdata/rules-fixture/openai_sdk/tool_call_claims.yaml new file mode 100644 index 00000000..5a318d46 --- /dev/null +++ b/testdata/rules-fixture/openai_sdk/tool_call_claims.yaml @@ -0,0 +1,141 @@ +policy: + id: openai_sdk_tool_call_claims + name: OpenAI Agents SDK tool-call claims (EBTE) + category: openai_sdk + description: > + Rules that flag a OpenAI Agents SDK tool whose calls are not bound to a + typed action claim checked outside the model, after Explanation-Bound Tool + Execution (EBTE, arXiv 2607.25364). The claim is a reserved `claim` argument + (a ToolInvocationExplanation: intent class, selected tool, expected effect, + risk tier, uncertainty, untrusted-context dependencies, evidence digests) + that a mediation step compares with server-held facts before the call has + any effect. The three rules form a ladder over the tool's claim_use fact: no + claim on a side-effecting tool, a claim nothing reads, and a claim that is + only written out or returned. A checked claim, or a tool with an extra + decorator that may mediate, is silent. + +rules: + - id: OAI-034 + title: Side-effecting tool takes no structured claim for its calls + severity: low + confidence: 0.5 + language: python + applies_to: + - openai_tool + scope: tool + match: + all: + - tool_claim_use: [absent] + - any: + - has_shell_call: true + - has_code_exec_call: true + - has_write_call: true + - name_has_prefix: + - send + - notify + - email + - post_ + - publish + - delete + - remove + - drop + - purge + - refund + - charge + - payout + - pay_ + - transfer + - deploy + - execute + - write + - update + - create + - insert + - upsert + explanation: > + This `@function_tool` function has a side effect — it shells out, executes + code, writes a file, or its name says it sends, deletes, pays, deploys, + creates or updates — and it takes no `claim` argument, so nothing about + why the model chose this call or what it expects the call to do reaches + the code that runs it. The model's own reasoning is free text: it can be + wrong, written after the fact, or steered by a poisoned tool result or + document, and nothing can compare it with the task, the policy or the + arguments before the effect happens. This is a defense-in-depth gap, not a + bug on its own: the tool's existing controls still apply, and a claim + never grants authority. The rule only reads the parameter list, so a claim + carried some other way (a separate intent tool, a context object) still + fires. + fix: > + Add a reserved `claim` parameter typed as a ToolInvocationExplanation + (schema_version, intent_class, selected_tool, expected_effect, risk_tier, + uncertainty, untrusted_context, evidence_refs; an optional summary that no + check reads) and check it in a mediation step before the effect, in a tool + input guardrail attached through `tool_input_guardrails` (it runs before + the tool with the call's arguments), or one shared decorator stacked under + `@function_tool`. The step denies a missing or mismatched claim, sends an + unresolved fact or a high-risk tool to review, and combines with the + existing controls so the most restrictive outcome wins. Map a review + outcome to the tool's `needs_approval` gate. + + - id: OAI-035 + title: Tool declares a claim argument that nothing checks + severity: medium + confidence: 0.6 + language: python + applies_to: + - openai_tool + scope: tool + match: + tool_claim_use: [unused] + explanation: > + This `@function_tool` function takes a `claim` argument, so the model + fills in a typed claim about the call, but the tool body never reads it + and the tool has no decorator besides the one that registers it and no + tool input guardrail, so no mediation step visible here can check it. A + claim nobody checks is worse than none: reviewers and auditors see a + claim-carrying tool and assume the call was verified against the task and + the policy, while a prompt-injected or mistaken call runs exactly as it + would have without one. The rule cannot see a mediation step that lives in + a dispatch layer outside the tool (custom runner code, server middleware), + so that shape also fires. + fix: > + Check the claim before the effect, in a tool input guardrail attached + through `tool_input_guardrails` (it runs before the tool with the call's + arguments), or one shared decorator stacked under `@function_tool`: remove + it from the arguments, validate it, resolve the facts server-side (tool + registry, the task's authorized intent, policy, route), and compare. Deny + a conflict, review anything unresolved, and run the tool only on allow. + Map a review outcome to the tool's `needs_approval` gate. If the claim is + checked in a dispatch layer this rule cannot see, move the check into a + decorator on the tool so the binding is visible where the tool is defined. + + - id: OAI-036 + title: Tool's claim argument is only written out or returned, never checked + severity: medium + confidence: 0.55 + language: python + applies_to: + - openai_tool + scope: tool + match: + tool_claim_use: [passthrough] + explanation: > + This `@function_tool` function reads its `claim` argument, but every read + only writes it out — to print, a logger, json.dumps, str or repr — or puts + it in the return value. Nothing compares the claim with the arguments, the + task or the policy before the side effect runs, so a claim that + contradicts what the call actually does is recorded rather than stopped. + An audit trail of unchecked claims is evidence of what the model said, not + of what was allowed. The rule follows each read to the first call it is an + argument of within the tool body; a check done inside a helper the claim + is first copied into (`c = claim`) is counted as checked and stays silent. + fix: > + Pass the claim to a verifier before the effect and act on its outcome: + deny on a conflict between the claim and the arguments, task or policy, + review on anything unresolved or high risk, and run the tool only on + allow. Keep writing the outcome and the per-check results to the audit + trail, but not the raw claim summary or untrusted content. Do it in a tool + input guardrail attached through `tool_input_guardrails` (it runs before + the tool with the call's arguments), or one shared decorator stacked under + `@function_tool`. Map a review outcome to the tool's `needs_approval` + gate. From 0622fa8dc3db7bdfed387f218790c8116810b6ec Mon Sep 17 00:00:00 2001 From: ericksondelacruz Date: Thu, 22 Oct 2026 10:29:10 +0800 Subject: [PATCH 2/2] feat(rules): ship the no-claim rung at info and settle the open questions OAI-034 / MCP-033 / LC-028 move from low to info. Nobody uses a claim argument yet, so at low they would fail every --strict run on a repo with side-effecting tools. Info shows the finding and its fix in every report, never fails --strict, and weighs 0.05 in the score instead of 0.15. The unused and passthrough rungs stay medium: they fire only on code that adopted claims. The other open questions are settled without code changes. The passthrough reading and the exact `claim` name stay. The topic file stays. Claude Agent SDK is the next SDK. Signing and the badge are left to the rules repo's workflows. --- CHANGELOG.md | 8 ++++---- CLAUDE.md | 2 +- testdata/rules-fixture/langchain/tool_call_claims.yaml | 2 +- testdata/rules-fixture/mcp/tool_call_claims.yaml | 2 +- testdata/rules-fixture/openai_sdk/tool_call_claims.yaml | 2 +- 5 files changed, 8 insertions(+), 8 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 0a1a7200..dc1b02be 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,15 +7,15 @@ to follow Semantic Versioning once it reaches 1.0. ## [Unreleased] Rule schema 26 → 27 (`rules.SupportedSchemaVersion`). The rules fixture goes -from 343 to 352 rules. **Overall scores can fall on repositories with -side-effecting Python OpenAI Agents, MCP or LangChain tools** (OAI-034 / MCP-033 / -LC-028 fire on every such tool that takes no `claim` argument). +from 343 to 352 rules. OAI-034 / MCP-033 / LC-028 fire at **info** on every +side-effecting Python OpenAI Agents, MCP or LangChain tool that takes no `claim` +argument. They never fail `--strict`, and they lower the score only slightly. ### Added - **Tool-call claims (EBTE, schema 27).** Nine tool-scope rules for Python OpenAI Agents, MCP and LangChain tools, over the new `tool_claim_use` - predicate and `claim_use` discovery fact: OAI-034 / MCP-033 / LC-028 (low) + predicate and `claim_use` discovery fact: OAI-034 / MCP-033 / LC-028 (info) flag a side-effecting tool with no `claim` argument, OAI-035 / MCP-034 / LC-029 (medium) a `claim` nothing reads, and OAI-036 / MCP-035 / LC-030 (medium) a `claim` that is only printed, logged, serialized or returned. A diff --git a/CLAUDE.md b/CLAUDE.md index bb78abfb..f08f4711 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -352,7 +352,7 @@ When changing a rule (add / remove / edit severity, confidence, match, text): > **Tool-call claims, EBTE (2026-10-19, `schema_version` 26 → 27; +9 rules):** new tool-scope predicate `tool_claim_use` over the > `claim_use` fact that Python decorated-tool discovery stamps on a tool declaring a `claim` parameter (`internal/analysis/claim_use.go`; -> `absent` is decided from `ParamNames`). A three-rung ladder per SDK: **OAI-034 / MCP-033 / LC-028** (low, 0.5) side-effecting tool with +> `absent` is decided from `ParamNames`). A three-rung ladder per SDK: **OAI-034 / MCP-033 / LC-028** (info, 0.5; the pack's first info rules) side-effecting tool with > no claim, **OAI-035 / MCP-034 / LC-029** (medium, 0.6) claim never read, **OAI-036 / MCP-035 / LC-030** (medium, 0.55) claim only > printed / logged / serialized / returned. A second decorator, or an OpenAI tool input guardrail, makes the fact `wrapped` (silent). Rules > live in `/tool_call_claims.yaml`; rulebook docs in `docs/Policy//tool_call_claims.md`. Python only; TypeScript, Claude SDK, diff --git a/testdata/rules-fixture/langchain/tool_call_claims.yaml b/testdata/rules-fixture/langchain/tool_call_claims.yaml index a68b8201..2a741d94 100644 --- a/testdata/rules-fixture/langchain/tool_call_claims.yaml +++ b/testdata/rules-fixture/langchain/tool_call_claims.yaml @@ -17,7 +17,7 @@ policy: rules: - id: LC-028 title: Side-effecting tool takes no structured claim for its calls - severity: low + severity: info confidence: 0.5 language: python applies_to: diff --git a/testdata/rules-fixture/mcp/tool_call_claims.yaml b/testdata/rules-fixture/mcp/tool_call_claims.yaml index 3219f166..e3666b97 100644 --- a/testdata/rules-fixture/mcp/tool_call_claims.yaml +++ b/testdata/rules-fixture/mcp/tool_call_claims.yaml @@ -17,7 +17,7 @@ policy: rules: - id: MCP-033 title: Side-effecting tool takes no structured claim for its calls - severity: low + severity: info confidence: 0.5 language: python applies_to: diff --git a/testdata/rules-fixture/openai_sdk/tool_call_claims.yaml b/testdata/rules-fixture/openai_sdk/tool_call_claims.yaml index 5a318d46..7f01aacb 100644 --- a/testdata/rules-fixture/openai_sdk/tool_call_claims.yaml +++ b/testdata/rules-fixture/openai_sdk/tool_call_claims.yaml @@ -17,7 +17,7 @@ policy: rules: - id: OAI-034 title: Side-effecting tool takes no structured claim for its calls - severity: low + severity: info confidence: 0.5 language: python applies_to: