Skip to content

Fail the tests when an agent tool's inputs drift from its checks - #20

Merged
ericchernuka merged 1 commit into
masterfrom
ec/agent-tool-schemas
Nov 10, 2026
Merged

ericchernuka merged 1 commit into
masterfrom
ec/agent-tool-schemas

Conversation

@ericchernuka

@ericchernuka ericchernuka commented Nov 10, 2026 •

Copy link
Copy Markdown
Owner

Summary

Each agent tool's inputs are written twice: the valibot schema that parseInput checks, and the JSON Schema the agent reads. This keeps both copies, and a test now fails when they disagree.

for each of choose_colour, choose_format, list_squares
  shown   = the inputSchema the agent reads
  checked = toJsonSchema(valibot schema)
  expect rules(shown) == rules(checked)     # rules = no descriptions, no $schema
 requests.ts
-  HEX = /^#?(?:[\da-f]{3}|[\da-f]{6})$/i
+  HEX = /^#?(?:[\dA-Fa-f]{3}|[\dA-Fa-f]{6})$/     # JSON Schema can't carry the i flag
 tools.ts
-  pattern: '^#?([0-9a-fA-F]{3}|[0-9a-fA-F]{6})$'
+  pattern: String.raw`^#?(?:[\dA-Fa-f]{3}|[\dA-Fa-f]{6})$`
 package.json
+  devDependencies: @valibot/to-json-schema       # test only, ships nothing

The fake agent gains agent.inputs(name), so the test reads the schema the way an agent does.

Evidence

  • Before: I set maximum: 99 in tools.ts: × when an agent reads choose_colour, then its inputs match what the tool checks. I changed picklist(FORMATS.slice(1)) in requests.ts: × when an agent reads choose_format, ….
    After: with both files restored, pnpm check passes: 134 tests, types, lint, format, fallow and build.
  • Main chunk: master 491.92 kB, this branch 491.94 kB.

Merge Danger

Door: two-way

Blast Radius: tests

Agents see the same inputs as before. The only change they can see is how the hex pattern is spelled, and it matches the same strings.

@vercel

vercel Bot commented Nov 10, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
huebo Ready Ready Preview Nov 10, 2026 4:23am UTC

Each tool's inputs are written twice: the valibot schema that
parseInput checks, and the JSON Schema the agent reads. Nothing kept
them in step, and the two hex patterns were already spelled
differently.

A test now converts each valibot schema with @valibot/to-json-schema
and compares it with what the tool shows an agent, ignoring
descriptions. The converter is a dev dependency, so the page ships
nothing new. The hex pattern is now spelled the same way on both sides,
and without the i flag, which JSON Schema can't carry.
@ericchernuka ericchernuka changed the title Generate the agent tools' input schemas from valibot Fail the tests when an agent tool's inputs drift from its checks Nov 10, 2026
@ericchernuka
ericchernuka merged commit fe7b32f into master Nov 10, 2026
3 checks passed
@ericchernuka
ericchernuka deleted the ec/agent-tool-schemas branch November 10, 2026 04:26

This branch was successfully deployed

1 active deployment
Preview — 58197992 Deployed Nov 10, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant