Skip to content

telemetry: opt-in OpenTelemetry export (refs #26) - #46

Merged
blockmandev merged 1 commit into
mainfrom
feat/otel
Oct 22, 2026
Merged

blockmandev merged 1 commit into
mainfrom
feat/otel

Conversation

@blockmandev

Copy link
Copy Markdown
Member

#26 item 1.

Decisions

  • No new crates. OTLP/HTTP with JSON, built by hand and posted with the reqwest client incode already has, through runtime::http, so proxies, loopback and INCODE_CA_CERTS all apply. No opentelemetry crates, no gRPC.
  • Off by default. It needs INCODE_TELEMETRY=1, or "telemetry": true in admin.json, and OTEL_EXPORTER_OTLP_ENDPOINT (or the per-signal endpoints).
  • It refuses rather than guesses. In each of these cases it is reported once on stderr and stays off:
    • a protocol other than http/json;
    • a collector that is neither https nor this machine (the headers usually carry a token);
    • malformed OTEL_EXPORTER_OTLP_HEADERS;
    • turned on with no endpoint.
  • Never sent: prompt, file or tool contents. A prompt's length is always sent; its text only with OTEL_LOG_USER_PROMPTS=1.

What

metric (delta sum) attributes
incode.session.count mode: repl / acp / task
incode.token.usage type, model
incode.cost.usage (USD) model, priced models only
incode.tool.calls tool, outcome
incode.turn.count, incode.turn.duration (s) model

Tool outcomes. allowed means every gate passed and the tool ran, however it then ended. denied means a gate refused it. It's counted by a thin wrapper around Toolbox::run_call, whose about 15 return points stay in decide_and_run. A thread-local flag set at the dispatch line tells the two apart, and it's saved and restored so a sub-agent's nested calls can't change the outer call's count.

Events (log records): incode.permission_prompt (from confirm_with_person and ask_protected), incode.compaction (manual and automatic, API route), incode.user_prompt.

When. After every turn, on its own thread, and at the end of -t, before exit. Each export is bounded at 5 s; a failed one is traced and dropped.

Where configured. incode --admin shows whether it's on and where it sends. There's a new telemetry field in admin.json.

Tests

  • telemetry::tests::recorded_usage_reaches_a_collector_as_otlp_json: a fake collector on loopback receives /v1/metrics and /v1/logs with the operator's header, delta temporality, and the right sums and events. No prompt text arrives, and nothing is sent twice.
  • telemetry::tests::telemetry_is_off_unless_turned_on_and_refuses_what_it_cannot_do: every refusal case, plus header decoding.
  • toolbox::tests::a_tool_call_is_counted_allowed_when_it_ran_and_denied_when_refused: a refused Write counts as denied; a read and a failed read count as allowed. Control: never setting the flag fails it.
  • cli::agent_turn::tests::a_finished_turn_records_its_tokens_and_one_turn. Control: not recording fails it.

Local Windows gate: fmt, clippy -D warnings, cargo test (2229 lib passing; the one failure was the dead-capability check, now fixed and passing).

Not covered

  • A tool name the model invents, which is refused as unknown, is still sent as the tool attribute.
  • The memory engine's and the agent CLIs' own requests are theirs to export.

Item 1 of #26. Off unless INCODE_TELEMETRY=1 (or admin.json telemetry)
and an OTLP endpoint are set. Exported by hand over the reqwest incode
already has, so no OpenTelemetry crates and no gRPC stack: http/json
only (another protocol is reported and nothing is sent), and only to
https or this machine, since OTEL_EXPORTER_OTLP_HEADERS carries tokens.
Metrics as delta sums: sessions, tokens by type and model, cost, tool
calls by tool and outcome (allowed when the tool ran, denied when a gate
refused it, counted around Toolbox::run_call), turns and their seconds.
Events as log records: permission prompts, compactions, user prompts
(length only; text only with OTEL_LOG_USER_PROMPTS=1). Exported after
each turn on a thread of its own, and at the end of -t.
@blockmandev
blockmandev merged commit 9f9e3c9 into main Oct 22, 2026
7 checks passed
@blockmandev
blockmandev deleted the feat/otel branch November 15, 2026 20:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant