Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 20 additions & 1 deletion ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -540,6 +540,17 @@ For each language recon cleared, do the AST work and produce a `RepoInventory`:
- **DiscoverTools** (`discovery.go`) — two-pass Python discovery (decorated
functions and bare shell-invoking functions). Also captures decorator kwargs
into `ToolDef.Config` for rules that inspect `@function_tool(strict_mode=...)`.
For a decorated tool that declares a parameter named `claim` (an EBTE-style
`ToolInvocationExplanation`), `pythonClaimUse` (`claim_use.go`) stamps the
`claim_use` fact (`models.FactClaimUse`): `unused` (the body never reads it),
`passthrough` (every read is an argument to print / a logger method /
`json.dumps` / `str` / `repr`, or part of a return value), `checked` (some read
feeds another call, a comparison or a condition), or `wrapped` (a second
decorator, or for an OpenAI tool an attached tool input guardrail, may
mediate). A tool with no `claim` parameter carries no fact, so scan output for
repos that do not use claims is unchanged. The `tool_claim_use` predicate
(`schema_version` 32) reads it; its `absent` value is decided from
`ParamNames`. Read by OAI-034..036, MCP-033..035 and LC-028..030.
- **DiscoverAgents** (`agents.go`) — finds `Agent(...)` / `SandboxAgent(...)` /
`AgentDefinition(...)` constructor calls; captures all constructor kwargs into
a typed `KwargTree`; sets `Opaque=true` for `Agent(**config)` or
Expand Down Expand Up @@ -2143,6 +2154,10 @@ internal/
│ │ stamps a structural shells_out fact via
│ │ pythonBodyShellsOut so agent rules can see a
│ │ decorated tool that shells out).
│ ├── claim_use.go pythonClaimUse: the claim_use fact on a decorated
│ │ tool with a `claim` parameter (unused /
│ │ passthrough / checked / wrapped), plus
│ │ openAIToolHasInputGuardrail.
│ ├── agents.go Python agent / guardrail / session discovery and
│ │ edge resolution (DiscoverAgents, DiscoverGuardrails,
│ │ DiscoverSessions, ResolveEdges, SessionClasses).
Expand Down Expand Up @@ -2186,7 +2201,7 @@ internal/
├── rules/ YAML-driven detection engine. Authoritative.
│ ├── schema.go PolicyFile / RuleDef / MatchExpr types.
│ ├── schema_version.go SupportedSchemaVersion const (engine ↔ pack gate).
│ ├── loader.go Validating YAML loader (recursive walk; skips manifest.yaml). Rejects repo_has_sdk_in_code values that are not SDK-enum tokens (catches the claude_sdk-vs-claude_agent_sdk silent never-fire).
│ ├── loader.go Validating YAML loader (recursive walk; skips manifest.yaml). Rejects repo_has_sdk_in_code values that are not SDK-enum tokens (catches the claude_sdk-vs-claude_agent_sdk silent never-fire), and tool_claim_use values outside absent / unused / passthrough / checked / wrapped.
│ ├── predicates.go One Pred* per detection primitive. TS-aware: PredHasShellCall/PredHasWriteCall/PredHasCodeExecCall/PredHasDynamicURLCall / PredHasUnpinnedSchemeURLCall read the shells_out/writes_fs/code_exec/dynamic_url/url_scheme_unpinned facts for TypeScript and walk the AST for Python; PredHasBodyText uses a [Line, EndLine] span substring fallback (bodyTextFromSpan), kept for textual-absence checks only.
│ ├── evaluator.go MatchExpr.Evaluate — recursive walker.
│ └── rule_detector.go RuleDetector adapter + LoadRegistry.
Expand Down Expand Up @@ -2363,6 +2378,10 @@ negation.
- Required-field validation, enum validation (severity / category),
duplicate-rule-ID detection across files. Every error is collected via
`errors.Join` so a contributor sees every problem in one run.
- **Closed-value predicates**: a `repo_has_sdk_in_code` value that is not an
SDK-enum token, or a `tool_claim_use` value outside `absent` / `unused` /
`passthrough` / `checked` / `wrapped`, is rejected, since either would
silently never match.
- **`applies_to`-vs-scope** check (`validAppliesToForScope`): the declared
`applies_to` kinds must be legal for the rule's `scope`.
- **`match`-vs-scope** check (`MatchExpr.outOfScopePredicates`): every predicate
Expand Down
16 changes: 16 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,24 @@ to follow Semantic Versioning once it reaches 1.0.

## [Unreleased]

Rule schema 31 → 32 (`rules.SupportedSchemaVersion`). The rules fixture goes
from 355 to 364 rules. OAI-034 / MCP-033 / LC-028 fire at **info** on every
side-effecting Python OpenAI Agents, MCP or LangChain tool that takes no `claim`
argument. They never fail `--strict`, and they lower the score only slightly.

### Added

- **Tool-call claims (EBTE, schema 32).** Nine tool-scope rules for Python
OpenAI Agents, MCP and LangChain tools, over the new `tool_claim_use`
predicate and `claim_use` discovery fact: OAI-034 / MCP-033 / LC-028 (info)
flag a side-effecting tool with no `claim` argument, OAI-035 / MCP-034 /
LC-029 (medium) a `claim` nothing reads, and OAI-036 / MCP-035 / LC-030
(medium) a `claim` that is only printed, logged, serialized or returned. A
second decorator, or an OpenAI tool input guardrail, counts as a possible
mediator. The fact is stamped only on tools that declare `claim`, so scan
output for repos that do not use claims is unchanged apart from the new
OAI-034 / MCP-033 / LC-028 findings. **Release order:** this engine first,
then the rules pack.
- **A one-time star line after the first interactive scan.** `trustabl scan`
prints "⭐ Found this useful? Star Agent Reliability Analyzer" with the
repository link to stderr once, after the first successful scan with human
Expand Down
8 changes: 8 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -350,6 +350,14 @@ When changing a rule (add / remove / edit severity, confidence, match, text):
6. Commit and push the rules repo **and** the rulebook (the user pushes engine
commits manually; confirm before pushing any of the three).

> **Tool-call claims, EBTE (2026-10-19, `schema_version` 31 → 32; +9 rules; main took 27 for OBS-009..012, 28 for CI-001..005, 29 for CSDK-115, 30 for MCPCFG-003, 31 for AGCFG-011):** new tool-scope predicate `tool_claim_use` over the
> `claim_use` fact that Python decorated-tool discovery stamps on a tool declaring a `claim` parameter (`internal/analysis/claim_use.go`;
> `absent` is decided from `ParamNames`). A three-rung ladder per SDK: **OAI-034 / MCP-033 / LC-028** (info, 0.5; the pack's first info rules) side-effecting tool with
> no claim, **OAI-035 / MCP-034 / LC-029** (medium, 0.6) claim never read, **OAI-036 / MCP-035 / LC-030** (medium, 0.55) claim only
> printed / logged / serialized / returned. A second decorator, or an OpenAI tool input guardrail, makes the fact `wrapped` (silent). Rules
> live in `<sdk>/tool_call_claims.yaml`; rulebook docs in `docs/Policy/<sdk>/tool_call_claims.md`. Python only; TypeScript, Claude SDK,
> ADK, CrewAI, Pydantic AI, AutoGen and Vercel AI are follow-ups. The fixture holds **364** rules (355 + 9; re-derive).
>
> **Pre-approved fetch from a user-content host (2026-10-21, `schema_version` 30 -> 31, provisional; +1 rule):** **AGCFG-011** (high, 0.7, repo
> scope, category `agent_config`, new pack file `agent_config/fetch_permissions.yaml`) over the new predicate
> `repo_settings_preapprove_user_content_fetch`, backed by `ProjectConfigFacts.UserContentFetchAllow`. It flags a WebFetch rule in
Expand Down
7 changes: 4 additions & 3 deletions COVERAGE.md

Large diffs are not rendered by default.

180 changes: 180 additions & 0 deletions internal/analysis/claim_use.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,180 @@
package analysis

import (
"strings"

sitter "github.com/smacker/go-tree-sitter"

"github.com/trustabl/trustabl/internal/analysis/astutil"
"github.com/trustabl/trustabl/internal/models"
)

// claimParam is the reserved parameter an EBTE-style tool takes its
// ToolInvocationExplanation in.
const claimParam = "claim"

// claimSinkCallees are the bare callees whose arguments are output, not a
// decision: a claim that only reaches one of them is passed through.
var claimSinkCallees = map[string]bool{
"print": true, "str": true, "repr": true, "pprint": true,
"json.dumps": true, "json.dump": true, "pprint.pprint": true,
}

// claimSinkMethods are logger method names. Matched on the method alone
// (logger.info, self.log.debug, logging.warning), since the receiver's type is
// not resolved.
var claimSinkMethods = map[string]bool{
"debug": true, "info": true, "warning": true, "warn": true,
"error": true, "exception": true, "critical": true, "log": true,
}

// pythonClaimUse classifies how a Python tool function uses its `claim`
// parameter, returning one of the models.ClaimUse* values, or "" when the
// function declares no such parameter.
//
// dec is the function's decorated_definition, or nil. A tool with more than
// one decorator is ClaimUseWrapped: the extra decorator may strip and check the
// claim before the body runs, and that cannot be seen from here.
//
// Each read of `claim` in the body is followed outward to the first thing that
// consumes it. The innermost call it is an argument of decides: an output call
// (print, a logger method, json.dumps, str, repr) makes it a passthrough read,
// any other call makes it checked. A read that reaches a return statement
// without passing through a call is a passthrough read; one that reaches any
// other statement (a condition, a comparison, an assignment) is checked. The
// fact is checked if any read is, passthrough if every read is, unused if
// there are none. Anything not recognized as output counts as checked, so an
// unusual shape is silent rather than a false finding.
func pythonClaimUse(fn, dec *sitter.Node, src []byte) string {
declared := false
for _, p := range astutil.FunctionParams(fn, src) {
if p == claimParam {
declared = true
break
}
}
if !declared {
return ""
}
if dec != nil && len(astutil.Decorators(dec)) > 1 {
return models.ClaimUseWrapped
}
body := fn.ChildByFieldName("body")
if body == nil {
return models.ClaimUseUnused
}

reads, passthrough := 0, 0
astutil.Walk(body, func(n *sitter.Node) bool {
if n.Type() != "identifier" || astutil.NodeText(n, src) != claimParam || !isClaimRead(n) {
return true
}
reads++
if claimReadIsPassthrough(n, fn, src) {
passthrough++
}
return true
})
switch {
case reads == 0:
return models.ClaimUseUnused
case passthrough == reads:
return models.ClaimUsePassthrough
default:
return models.ClaimUseChecked
}
}

// isClaimRead rejects an identifier spelled `claim` that is not a read of the
// variable: the attribute name in `x.claim`, or the keyword in `f(claim=...)`.
func isClaimRead(n *sitter.Node) bool {
p := n.Parent()
if p == nil {
return true
}
switch p.Type() {
case "attribute":
if a := p.ChildByFieldName("attribute"); a != nil && a.Equal(n) {
return false
}
case "keyword_argument":
if k := p.ChildByFieldName("name"); k != nil && k.Equal(n) {
return false
}
}
return true
}

// claimReadIsPassthrough follows one read of `claim` outward, stopping at fn.
func claimReadIsPassthrough(n, fn *sitter.Node, src []byte) bool {
cur := n
for p := n.Parent(); p != nil && !p.Equal(fn); cur, p = p, p.Parent() {
switch p.Type() {
case "call":
args := p.ChildByFieldName("arguments")
if args == nil || !args.Equal(cur) {
// claim is the callee's receiver (claim.get(...)): the call
// yields a value read from the claim, so keep following it.
continue
}
return isClaimSinkCallee(p.ChildByFieldName("function"), src)
case "return_statement":
return true
case "expression_statement", "assignment", "augmented_assignment",
"if_statement", "elif_clause", "while_statement", "assert_statement",
"raise_statement", "for_statement", "with_statement", "match_statement":
return false
}
}
return false
}

func isClaimSinkCallee(callee *sitter.Node, src []byte) bool {
if callee == nil {
return false
}
text := astutil.NodeText(callee, src)
if claimSinkCallees[text] {
return true
}
if callee.Type() == "attribute" {
if i := strings.LastIndex(text, "."); i >= 0 {
return claimSinkMethods[text[i+1:]]
}
}
return false
}

// openAIToolHasInputGuardrail reports whether an OpenAI Agents SDK tool has a
// tool input guardrail attached, either as the decorator kwarg
// (@function_tool(tool_input_guardrails=[...])) or by assignment anywhere in the
// file (send_email.tool_input_guardrails = [...], the shape the SDK's own
// examples use). An input guardrail runs before the tool with the call's
// arguments, so it is where a mediation step can read the claim. Whether it
// actually checks the claim is not verified.
func openAIToolHasInputGuardrail(t models.ToolDef, root *sitter.Node, src []byte) bool {
if _, ok := t.Config["tool_input_guardrails"]; ok {
return true
}
found := false
astutil.Walk(root, func(n *sitter.Node) bool {
if found {
return false
}
if n.Type() != "assignment" {
return true
}
left := n.ChildByFieldName("left")
if left == nil || left.Type() != "attribute" {
return true
}
obj, attr := left.ChildByFieldName("object"), left.ChildByFieldName("attribute")
if obj != nil && attr != nil && obj.Type() == "identifier" &&
astutil.NodeText(obj, src) == t.Name && astutil.NodeText(attr, src) == "tool_input_guardrails" {
found = true
return false
}
return true
})
return found
}
Loading
Loading