Skip to content

tool-request-tracing.md

Latest commit

 

History

History
105 lines (80 loc) · 4.59 KB

tool-request-tracing.md

File metadata and controls

105 lines (80 loc) · 4.59 KB
Raw

Tool Request Tracing — Maintainer Forensics

This page is the maintainer-level contract for WebCodex tool-request tracing. Ordinary operators should start with Troubleshooting, reproduce the problem once, and only use the details below when field-by-field correlation is actually required.

Enable a bounded capture

Tracing is disabled by default. Metadata mode records lightweight lifecycle information; full additionally captures request/response payloads for forensic debugging.

WEBCODEX_TOOL_REQUEST_TRACE=full
WEBCODEX_TOOL_REQUEST_TRACE_DIR=/var/lib/webcodex/tool-request-traces
WEBCODEX_TOOL_REQUEST_TRACE_RETENTION_HOURS=168
WEBCODEX_TOOL_REQUEST_TRACE_MAX_TOTAL_BYTES=2147483648

Apply the Server environment change using the normal deployment lifecycle, then reproduce the target call once. Disable full tracing again when the diagnostic capture is complete.

Server trace layout

Each captured request has a Server-generated server_trace_id. The default full store keeps one directory per trace under the configured trace root. The trace directory contains lifecycle events plus compressed JSON payloads, for example:

TRACE_ROOT=/var/lib/webcodex/tool-request-traces
find "$TRACE_ROOT/<server_trace_id>" -maxdepth 2 -type f -print
cat "$TRACE_ROOT/<server_trace_id>/events.jsonl"
zstd -dc "$TRACE_ROOT/<server_trace_id>/payloads/<payload>.json.zst" | jq .

For a model tool call, the useful payload layers are:

  • raw_request_body for the HTTP API, or raw_arguments for MCP: what reached the Server before wrapper/session normalization;
  • effective_arguments: the semantic argument object passed into the runtime/Connector dispatch;
  • runner_request: the exact typed request enqueued to the selected Runner when dispatch reached that boundary;
  • final_response: the bounded JSON response generated by the Server when one exists.

These layers let a maintainer determine where a field disappeared or changed. For example, a caller-visible wrapper field such as context_request can be compared between the raw and effective argument layers without making that field part of ordinary user troubleshooting.

Runner correlation

When a request reaches a Runner, events.jsonl records the existing runner_request_id together with bounded Runner/transport/build metadata. If Runner-side tool tracing is enabled, search the Runner journal for that same request id. Later Runner results and Job updates are correlated back to the originating Server trace through the bounded in-memory correlation index.

The correlation identifiers are diagnostics only. They are not authority, execution identity, Session continuity, or retry permission.

Missing or omitted payloads

Absence of a payload file does not prove that the payload was empty or truncated. Check the Server journal for trace-capture diagnostics. In particular:

  • tool_trace_capture_omitted + trace_writer_queue_full means the bounded background writer queue was saturated;
  • tool_trace_capture_omitted + trace_disk_budget_exceeded means the configured retained-byte budget could not admit the capture after pruning;
  • tool_trace_capture_failed means trace persistence/correlation failed for a diagnostic reason.

Trace failures are fail-open for the underlying tool call: they reduce forensic evidence but do not redefine the tool result.

Delivery boundary

tool_handler_returned proves only that WebCodex handed a response to the HTTP framework. It is not proof that the remote client received it. When the Server trace ends cleanly but the client reports no response, correlate the request time with the reverse-proxy/access logs before treating the call as a runtime failure.

Sensitive data

Full tracing is raw forensic capture. The trace path does not read the WebCodex ingress HTTP Authorization header, but a token, key, source fragment, patch, script/stdin value, command output, user message, or other secret that is itself present inside a captured tool/Runner payload can appear in the trace.

Treat the trace directory as sensitive diagnostic data. Do not attach raw traces to public issues or chat without reviewing/redacting them first. Keep retention and total-byte budgets bounded.

Documentation boundary

Public troubleshooting should document only the operator action: enable tracing, reproduce once, locate the capture, protect it, and escalate when needed. Exact payload names, request ids, correlation rules, queue/disk omission codes, and wire-normalization comparisons belong on this maintainer page, in tests, or in the implementation that validates them.