Skip to content

fix: describe tool schema fields and show one request payload - #7

Merged
DevMortimer merged 2 commits into
DevMortimer:mainfrom
horizonzzzz:fix/tool-schema-descriptions
Sep 22, 2026
Merged

DevMortimer merged 2 commits into
DevMortimer:mainfrom
horizonzzzz:fix/tool-schema-descriptions

Conversation

@horizonzzzz

Copy link
Copy Markdown
Contributor

What and why

Fixes #6.

Models inspecting typesafe_evaluate consistently fail their first call because neither the JSON Schema nor the prompt guidelines showed what a valid payload looks like:

  1. evaluationSchema in src/schema.ts had 0 property-level description fields on state, questions, type, instructions, and criteria. LLMs had to infer structural constraints from raw anyOf and patternProperties unions.
  2. The canonical sample payload was only exposed to /typesafe test and the playground, while prompt guidelines contained usage policies but no syntax example.
  3. Because the syntax reminder was only appended to validation error messages, a first-call failure was the only way for models to discover the expected object-map shape of questions.

This PR implements the suggested scope discussed in #6:

  • Added description annotations to agent-authored schema fields: state, questions, type literals, instructions, criteria (per question type), and model.
  • Exported a single-line request example in promptGuidelines reusing the existing module-level sample.
  • Left array-to-map normalisation out of this PR to keep scope focused. Accepted input is unchanged.

Verification

  • npm ci and npm run check pass: build (tsc -p tsconfig.build.json) and typecheck (tsc --noEmit) succeed cleanly.
  • Changed behavior has offline tests, or this PR does not change behavior:
    • Added structural test in tests/schema.test.ts verifying all authored schema fields and question variants carry descriptions.
    • Added assertion in tests/extension.test.ts verifying promptGuidelines contains the single-line payload example without newlines.
    • All 82 cross-platform offline unit tests pass. (Note: On native Windows, the 8 pre-existing permission assertion failures related to POSIX 0o600 vs NTFS 0o666 occur identically before and after this change; CI runs on Linux and macOS where they pass).
  • Documentation and changelog notes are updated where needed: accepted API and runtime behavior are unchanged.

Live testing: Not applicable. Transport and response validation are untouched.

Compatibility and release notes

  • Fully backward-compatible. Runtime validation logic and accepted payloads are unchanged.
  • Serialized schema length increased from 1968 to 2998 bytes (~260 tokens), adding self-documenting metadata without altering schema validation.
  • No version bump: maintainer-owned release process respected.

yqf and others added 2 commits September 22, 2026 11:42
Models read `parameters` and `promptGuidelines` before their first call, and
neither said what the payload must look like: the emitted schema carried no
`description` on `state`, `questions`, `type`, `instructions`, or `criteria`,
and the canonical sample only reached `/typesafe test` and the playground. The
syntax guide lived in the validation error, so a first-call failure was the
only way to learn the shape — `questions` authored as an array failed every
time.

Descriptions on the fields the agent authors, plus the existing sample inlined
as a single-line guideline. Accepted input is unchanged.

Refs DevMortimer#6
The guideline test only checked that some line contained `"state":`; it now
parses the payload out of the guideline and runs it through
parseEvaluationRequest, so the example the model copies is one the schema
accepts. The sample's score rubric said in sentences what three words say,
and every session pays for that line on every tool listing, so it is short
now and the test caps its length.

The schema test walks the serialized schema instead of pinning TypeBox's
patternProperties/anyOf layout. `entry` takes the description directly, and
the questions description drops the em-dash the rest of the file avoids.

Version 0.6.2 with a changelog line. Refs DevMortimer#6, DevMortimer#7.
@DevMortimer
DevMortimer merged commit 75a68b1 into DevMortimer:main Sep 22, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

improve(tool): LLMs consistently fail first tool call to typesafe_evaluate due to lack of schema descriptions and prompt examples

2 participants