Skip to content

docs(rulebook): new-SDK rationale + VAI-011 / OAI-016 structural #7

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
jhumel-code wants to merge 1 commit into from
Closed

docs(rulebook): new-SDK rationale + VAI-011 / OAI-016 structural #7

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 16 additions & 15 deletions POLICY_INDEX.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,9 +47,9 @@ Users can contribute their own policies by:
| LangChain / LangGraph | 11 | 3 | 0 | 1 | 15 | [langchain/POLICY_INDEX.md](langchain/POLICY_INDEX.md) |
| CrewAI | 7 | 6 | 0 | 1 | 14 | [crewai/POLICY_INDEX.md](crewai/POLICY_INDEX.md) |
| AutoGen / AG2 | 6 | 5 | 0 | 1 | 12 | [autogen/POLICY_INDEX.md](autogen/POLICY_INDEX.md) |
| Vercel AI SDK | 5 | 3 | 0 | 1 | 9 | [vercel_ai/POLICY_INDEX.md](vercel_ai/POLICY_INDEX.md) |
| Vercel AI SDK | 6 | 3 | 0 | 1 | 10 | [vercel_ai/POLICY_INDEX.md](vercel_ai/POLICY_INDEX.md) |
| Pydantic AI | 7 | 4 | 0 | 1 | 12 | [pydantic_ai/POLICY_INDEX.md](pydantic_ai/POLICY_INDEX.md) |
| **All** | **102** | **49** | **2** | **11** | **164** | |
| **All** | **103** | **49** | **2** | **11** | **165** | |

## All rules

Expand Down Expand Up @@ -206,16 +206,17 @@ Users can contribute their own policies by:
| 149 | VAI-006 | Vercel AI | agent | vercel_ai_agent | Vercel AI agent wires a provider shell / computer / code-execution tool | high | 0.85 | 59.5 | [vercel_ai/agent_safety.yaml](https://github.com/trustabl/trustabl-rules/blob/main/vercel_ai/agent_safety.yaml) |
| 150 | VAI-007 | Vercel AI | agent | vercel_ai_agent | Vercel AI agent tool loop has no step bound | medium | 0.60 | 24.0 | [vercel_ai/agent_safety.yaml](https://github.com/trustabl/trustabl-rules/blob/main/vercel_ai/agent_safety.yaml) |
| 151 | VAI-008 | Vercel AI | agent | vercel_ai_agent | Vercel AI agent forces a provider execution tool every step | medium | 0.65 | 26.0 | [vercel_ai/agent_safety.yaml](https://github.com/trustabl/trustabl-rules/blob/main/vercel_ai/agent_safety.yaml) |
| 152 | VAI-012 | Vercel AI | repo | vercel_ai | Vercel AI project ships no agent-guidance doc (AGENTS.md/CLAUDE.md) | low | 0.90 | 13.5 | [vercel_ai/repo_hygiene.yaml](https://github.com/trustabl/trustabl-rules/blob/main/vercel_ai/repo_hygiene.yaml) |
| 153 | PYD-001 | Pydantic AI | tool | pydantic_ai_tool | Pydantic AI tool has no description | low | 0.90 | 13.5 | [pydantic_ai/tool_definition.yaml](https://github.com/trustabl/trustabl-rules/blob/main/pydantic_ai/tool_definition.yaml) |
| 154 | PYD-002 | Pydantic AI | tool | pydantic_ai_tool | Pydantic AI tool parameters are not type-annotated | medium | 0.85 | 34.0 | [pydantic_ai/tool_definition.yaml](https://github.com/trustabl/trustabl-rules/blob/main/pydantic_ai/tool_definition.yaml) |
| 155 | PYD-003 | Pydantic AI | tool | pydantic_ai_tool | Pydantic AI tool body spawns a subprocess | high | 0.85 | 59.5 | [pydantic_ai/shell_safety.yaml](https://github.com/trustabl/trustabl-rules/blob/main/pydantic_ai/shell_safety.yaml) |
| 156 | PYD-004 | Pydantic AI | tool | pydantic_ai_tool | Pydantic AI tool body evaluates dynamic code | high | 0.85 | 59.5 | [pydantic_ai/code_execution.yaml](https://github.com/trustabl/trustabl-rules/blob/main/pydantic_ai/code_execution.yaml) |
| 157 | PYD-005 | Pydantic AI | tool | pydantic_ai_tool | Pydantic AI tool fetches a caller-controlled URL (SSRF) | high | 0.80 | 56.0 | [pydantic_ai/ssrf.yaml](https://github.com/trustabl/trustabl-rules/blob/main/pydantic_ai/ssrf.yaml) |
| 158 | PYD-006 | Pydantic AI | tool | pydantic_ai_tool | Pydantic AI tool network call has no timeout | high | 0.85 | 59.5 | [pydantic_ai/network.yaml](https://github.com/trustabl/trustabl-rules/blob/main/pydantic_ai/network.yaml) |
| 159 | PYD-007 | Pydantic AI | tool | pydantic_ai_tool | Mutating Pydantic AI tool has no idempotency key | medium | 0.55 | 22.0 | [pydantic_ai/idempotency.yaml](https://github.com/trustabl/trustabl-rules/blob/main/pydantic_ai/idempotency.yaml) |
| 160 | PYD-101 | Pydantic AI | agent | pydantic_ai_agent | Pydantic AI agent has no structured output validation | low | 0.70 | 10.5 | [pydantic_ai/agent_safety.yaml](https://github.com/trustabl/trustabl-rules/blob/main/pydantic_ai/agent_safety.yaml) |
| 161 | PYD-102 | Pydantic AI | agent | pydantic_ai_agent | Pydantic AI agent wires the code-execution native tool | high | 0.85 | 59.5 | [pydantic_ai/agent_safety.yaml](https://github.com/trustabl/trustabl-rules/blob/main/pydantic_ai/agent_safety.yaml) |
| 162 | PYD-103 | Pydantic AI | agent | pydantic_ai_agent | Pydantic AI agent wires a model-driven URL-fetching native tool | medium | 0.75 | 30.0 | [pydantic_ai/agent_safety.yaml](https://github.com/trustabl/trustabl-rules/blob/main/pydantic_ai/agent_safety.yaml) |
| 163 | PYD-105 | Pydantic AI | agent | pydantic_ai_agent | Pydantic AI agent retries with the exhaustive end strategy | low | 0.70 | 10.5 | [pydantic_ai/agent_safety.yaml](https://github.com/trustabl/trustabl-rules/blob/main/pydantic_ai/agent_safety.yaml) |
| 164 | PYD-201 | Pydantic AI | repo | pydantic_ai | Pydantic AI project ships no agent-guidance doc (AGENTS.md/CLAUDE.md) | low | 0.90 | 13.5 | [pydantic_ai/repo_hygiene.yaml](https://github.com/trustabl/trustabl-rules/blob/main/pydantic_ai/repo_hygiene.yaml) |
| 152 | VAI-011 | Vercel AI | tool | vercel_ai_tool | Vercel AI tool HTTP call has no timeout | high | 0.60 | 42.0 | [vercel_ai/network.yaml](https://github.com/trustabl/trustabl-rules/blob/main/vercel_ai/network.yaml) |
| 153 | VAI-012 | Vercel AI | repo | vercel_ai | Vercel AI project ships no agent-guidance doc (AGENTS.md/CLAUDE.md) | low | 0.90 | 13.5 | [vercel_ai/repo_hygiene.yaml](https://github.com/trustabl/trustabl-rules/blob/main/vercel_ai/repo_hygiene.yaml) |
| 154 | PYD-001 | Pydantic AI | tool | pydantic_ai_tool | Pydantic AI tool has no description | low | 0.90 | 13.5 | [pydantic_ai/tool_definition.yaml](https://github.com/trustabl/trustabl-rules/blob/main/pydantic_ai/tool_definition.yaml) |
| 155 | PYD-002 | Pydantic AI | tool | pydantic_ai_tool | Pydantic AI tool parameters are not type-annotated | medium | 0.85 | 34.0 | [pydantic_ai/tool_definition.yaml](https://github.com/trustabl/trustabl-rules/blob/main/pydantic_ai/tool_definition.yaml) |
| 156 | PYD-003 | Pydantic AI | tool | pydantic_ai_tool | Pydantic AI tool body spawns a subprocess | high | 0.85 | 59.5 | [pydantic_ai/shell_safety.yaml](https://github.com/trustabl/trustabl-rules/blob/main/pydantic_ai/shell_safety.yaml) |
| 157 | PYD-004 | Pydantic AI | tool | pydantic_ai_tool | Pydantic AI tool body evaluates dynamic code | high | 0.85 | 59.5 | [pydantic_ai/code_execution.yaml](https://github.com/trustabl/trustabl-rules/blob/main/pydantic_ai/code_execution.yaml) |
| 158 | PYD-005 | Pydantic AI | tool | pydantic_ai_tool | Pydantic AI tool fetches a caller-controlled URL (SSRF) | high | 0.80 | 56.0 | [pydantic_ai/ssrf.yaml](https://github.com/trustabl/trustabl-rules/blob/main/pydantic_ai/ssrf.yaml) |
| 159 | PYD-006 | Pydantic AI | tool | pydantic_ai_tool | Pydantic AI tool network call has no timeout | high | 0.85 | 59.5 | [pydantic_ai/network.yaml](https://github.com/trustabl/trustabl-rules/blob/main/pydantic_ai/network.yaml) |
| 160 | PYD-007 | Pydantic AI | tool | pydantic_ai_tool | Mutating Pydantic AI tool has no idempotency key | medium | 0.55 | 22.0 | [pydantic_ai/idempotency.yaml](https://github.com/trustabl/trustabl-rules/blob/main/pydantic_ai/idempotency.yaml) |
| 161 | PYD-101 | Pydantic AI | agent | pydantic_ai_agent | Pydantic AI agent has no structured output validation | low | 0.70 | 10.5 | [pydantic_ai/agent_safety.yaml](https://github.com/trustabl/trustabl-rules/blob/main/pydantic_ai/agent_safety.yaml) |
| 162 | PYD-102 | Pydantic AI | agent | pydantic_ai_agent | Pydantic AI agent wires the code-execution native tool | high | 0.85 | 59.5 | [pydantic_ai/agent_safety.yaml](https://github.com/trustabl/trustabl-rules/blob/main/pydantic_ai/agent_safety.yaml) |
| 163 | PYD-103 | Pydantic AI | agent | pydantic_ai_agent | Pydantic AI agent wires a model-driven URL-fetching native tool | medium | 0.75 | 30.0 | [pydantic_ai/agent_safety.yaml](https://github.com/trustabl/trustabl-rules/blob/main/pydantic_ai/agent_safety.yaml) |
| 164 | PYD-105 | Pydantic AI | agent | pydantic_ai_agent | Pydantic AI agent retries with the exhaustive end strategy | low | 0.70 | 10.5 | [pydantic_ai/agent_safety.yaml](https://github.com/trustabl/trustabl-rules/blob/main/pydantic_ai/agent_safety.yaml) |
| 165 | PYD-201 | Pydantic AI | repo | pydantic_ai | Pydantic AI project ships no agent-guidance doc (AGENTS.md/CLAUDE.md) | low | 0.90 | 13.5 | [pydantic_ai/repo_hygiene.yaml](https://github.com/trustabl/trustabl-rules/blob/main/pydantic_ai/repo_hygiene.yaml) |
37 changes: 24 additions & 13 deletions docs/Policy/openai_sdk/network.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -105,30 +105,40 @@ False positives: code that wraps `urlopen` behind a helper that injects a timeou

### OAI-016 — TypeScript tool fetch call has no AbortSignal timeout (Severity: high, Confidence: 0.6, Fix type: code)

**What we detect:** a TypeScript tool body that calls `fetch(` with no
`AbortSignal` / `AbortController` / `signal:` / `AbortSignal.timeout` present
(`has_body_text` for `fetch(` AND `not has_body_text` for the abort markers).
**What we detect:** a TypeScript tool `execute` body that makes an HTTP call
(`fetch` / `axios` / `got` / `undici`) with no timeout bound — the call carries no
options object with a `signal`, `timeout`, or `abortSignal` key
(`has_http_call_without_timeout: true`). The fact is **structural**:
`tsHandlerFacts` walks the handler, recognizes a recognized HTTP-client
`call_expression`, and inspects its argument objects for a timeout-bearing key, so
a bare `fetch(url)` or `fetch(url, { method })` fires while `fetch(url, { signal:
AbortSignal.timeout(ms) })` does not. This replaced the original brittle
`has_body_text` substring check.

**Why it is flaggable:** Node's and the browser's `fetch` have no implicit
timeout; a slow host blocks the tool's `execute` callback — and the agent run
loop — indefinitely, burning the turn's wall-clock budget and tying up the worker.
Because these templates often interpolate the URL from tool arguments, the
un-cancellable call also amplifies SSRF/exfiltration impact.
When the URL is interpolated from tool arguments, the un-cancellable call also
amplifies SSRF/exfiltration impact.

**Real-world consequence:** a TS `web_fetch` tool hangs the whole agent turn when
the model supplies a slow or hostile URL.

**Why severity is high and not medium:** the failure denies the agent loop, with
no in-band mitigation short of an explicit signal.
no in-band mitigation short of an explicit signal/timeout.

**Fix type — code:** pass `signal: AbortSignal.timeout(ms)` (or an
`AbortController`) to `fetch`.

**Confidence 0.6:** `has_body_text` is a brittle substring check — it can miss a
timeout wired through a wrapper, or fire when the abort lives in a helper.

**Provisional (TypeScript):** this rule loads and validates today but will not
fire until the engine's TypeScript tool parser ships; it is load-validated only.
`AbortController`) to `fetch`; `axios`/`got` take a `timeout` option directly.

**Confidence 0.6:** the structural check is precise about the call shape (no
substring guesswork), but it cannot see a timeout reached indirectly. **False
positives** (fires though bounded): an options object passed by identifier
(`fetch(url, opts)`), a `signal` / `AbortController` defined on a separate line, a
`Promise.race([fetch(url), timeout])`, or an `axios.create({ timeout })` instance.
**False negatives** (silent though unbounded): `signal: req.signal` with no
deadline behind it, or `timeout: 0` (axios "no timeout"). These indirection blind
spots — not substring brittleness — are why it sits with the TS dynamic-URL rule
OAI-024 at 0.6 rather than higher.

### OAI-018 — Tool builds outbound URL from non-literal value (Severity: medium, Confidence: 0.55, Fix type: code)

Expand Down Expand Up @@ -218,6 +228,7 @@ the call all escape the first-argument check on a known callee.
- Network calls made transitively — the tool calls a helper that performs the request without a timeout. The predicate inspects the tool body directly and does not follow calls into other modules.
- Retries without backoff. A tool that times out cleanly but retries in a tight loop is still a denial-of-budget hazard; that is OAI-009 / idempotency territory, not this policy.
- For OAI-024: HTTP clients outside the recognized `fetch`/`axios`/`got`/`undici` set (`node:http`/`https` `request`, `superagent`, `ky`, a wrapped client), a URL constructed via `new URL(base, modelValue)` before the call, and a model-supplied value passed positionally into a helper that performs the fetch — all escape the first-argument check on a known callee.
- For OAI-016: a timeout reached indirectly — an options object passed by identifier, a `signal`/`AbortController` bound on a separate line, a `Promise.race` deadline, or an `axios.create({ timeout })` instance — is not seen, so the rule fires on some already-bounded calls; conversely a non-deadline `signal: req.signal` or an `axios` `timeout: 0` ("no timeout") is treated as bounded and does not fire.

---

Expand Down
Loading
Loading