tool_calls grader's expect[].args reads stale session_digest data, not the new tool_events[].args

[ShaharPotash1]

Summary

The tool_calls grader's new structured argument matchers (expect: [{tool, args}], added in #388 / v0.38.0) fail even when the recorded tool call's arguments exactly match the declared expectation. Root cause: the grader's argument-reading path appears to use session_digest.tool_calls[].arguments, not the new tool_events[].args field also added in #388 — and session_digest.tool_calls[].arguments reports an unrelated placeholder shape (fields from other built-in tools like edit/bash), not the actual arguments of the call being graded.

#388's own description says the expect grader "checks every entry against the recorded tool calls for the run (the same data that is also serialized into tool_events[] in results.json)" — in practice this doesn't hold for the argument values.

Reproduction

eval.yaml (schemaVersion 1.1, native mcp_mocks):

name: repro
skill: repro-stub
schemaVersion: "1.1"
version: "1.0"
config:
  trials_per_task: 1
  executor: copilot-sdk
  model: claude-sonnet-5
  inject_skill_body: false
  disabled_skills: ["*"]
mcp_mocks:
  - name: nessie-candidate
    tools:
      entity_search_clients:
        input_schema:
          type: object
          required: [query]
        responses:
          - match:
              query: "Bissell"
            return:
              results:
                matches:
                  - name: "Bissell"
                    client_id: "c-4471"
tasks:
  - "tasks/*.yaml"

tasks/entity-search-clients.yaml:

id: repro-001
name: Call entity_search_clients and report the client_id
inputs:
  prompt: >-
    Call the entity_search_clients tool with exactly {"query": "Bissell"} and report the
    exact client_id it returns for the match named "Bissell".
graders:
  - name: tool_was_called
    type: tool_calls
    config:
      expect:
        - tool: nessie-candidate-entity_search_clients
  - name: called_with_expected_args
    type: tool_calls
    config:
      expect:
        - tool: nessie-candidate-entity_search_clients
          args:
            query: { equals: "Bissell" }

Run:

waza run eval.yaml --model claude-sonnet-5 --no-cache -o result.json

Expected

Both graders pass — the model calls entity_search_clients with exactly {"query": "Bissell"} (confirmed in the raw transcript's tool.execution_start.arguments and in results.json's own tool_events[].args, both of which correctly show {"query": "Bissell"}).

Actual

• tool_was_called (no args:) — passes.
• called_with_expected_args (with args:) — fails, with feedback:

  expectation tool="nessie-candidate-entity_search_clients" not satisfied: argument "query": not present on tool call
  

Inspecting the same result.json's session_digest.tool_calls[].arguments for this call shows:

{
  "path": "",
  "file_text": "",
  "command": "",
  "description": "",
  "skill": ""
}

— an unrelated placeholder shape (looks like a merged/zeroed superset of several built-in tools' argument schemas), not the actual {"query": "Bissell"} sent. The correctly-populated data is present a few keys over, in the same result file's tool_events[] array:

{
  "tool_call_id": "...",
  "tool_name": "nessie-candidate-entity_search_clients",
  "args": {"query": "Bissell"},
  ...
}

Environment

• waza version 0.38.4
• executor: copilot-sdk
• Confirmed via both a live run and waza grade <eval.yaml> --results result.json (offline re-grading against the same captured result), isolating the failure to the args-matching path specifically (tool-name-only matching passes on the identical data).

Notes

• This may be the same underlying session_digest.tool_calls[].arguments placeholder-shape issue independently observed with mcp_servers (not just the new mcp_mocks) prior to v0.38 — happy to share more detail if useful, but wanted to file this against the new expect[].args grader specifically since that's the feature this affects most directly.
• Workaround in the meantime: read tool_events[].args (or the raw transcript) directly rather than relying on the tool_calls grader's args: matching.

[github-actions]

🤖 Routed to @copilot (Coding Agent)

Issue: #474 — tool_calls grader's expect[].args reads stale session_digest data, not the new tool_events[].args

@copilot has been assigned and will pick this up automatically.

The coding agent will create a copilot/* branch and open a draft PR.
Review the PR as you would any team member's work.

[github-actions]

📋 Marked as release:backlog — assign a release target when ready.

[spboyer]

Correction to prior comment (formatting):

Validation update:

• Confirmed the tool_calls grader matches against SessionDigest.ToolCalls (internal/graders/tool_calls_grader.go), while canonical per-call MCP args are also present in tool_events[] (internal/models/outcome_schema_test.go, schema v1.1).
• ToolCallArgs.Extra is intentionally excluded from JSON (json:"-" in internal/models/events.go), which can cause argument keys like query to be absent after round-trip through results.json when grading from digest-backed data.
• This aligns with the reported symptom (argument "query": not present on tool call).

Repro status:

• I validated the code path and ran targeted grader/model tests locally.
• I could not execute the exact end-to-end Copilot SDK run path from this environment (external model/runtime credentials needed), so this is validated as a high-confidence code-path repro.

Additional info that would fully close validation for the live path:

• A minimal redacted result.json containing one failing case with both session_digest.tool_calls[] and tool_events[] for the same tool_call_id from the same run.