observability: auto-wire Phoenix from env + fix OpenInference span rendering - #135
Open
whatever wants to merge 4 commits into
Open
observability: auto-wire Phoenix from env + fix OpenInference span rendering#135whatever wants to merge 4 commits into
whatever wants to merge 4 commits into
Conversation
The ObservabilityIntegration existed but was inert — nothing ever called `.connect()`, so PHOENIX_ENDPOINT / PHOENIX_PROJECT had no effect and no LLM spans ever reached the collector. Add `ObservabilityIntegration.bootstrap_from_env()` as a singleton factory that idempotently instantiates + connects when both env vars are set (returns None otherwise). Wire it into every process entry point: - `clearwing.main()` covers all CLI subcommands, including `clearwing operate --machine-fd 3` used by embedding hosts. - `create_app()` covers the webui/FastAPI server and registers a shutdown hook so the OTel batch processor flushes on graceful exit. Safe to call from multiple entry points concurrently; the classvar `_singleton` guard makes the call idempotent. `disconnect()` releases the singleton so a fresh bootstrap can rebuild — primarily useful for tests.
The synthetic ``llm_call`` span was rendered as a generic "internal" span in the Arize Phoenix UI because the attribute names didn't match Phoenix's OpenInference-based classifier. Switch to the documented OpenInference conventions: - openinference.span.kind = "LLM" (was: span.kind = "llm") - llm.model_name (was: llm.model) - llm.token_count.prompt (was: llm.token_count.input) - llm.token_count.completion (was: llm.token_count.output) - llm.token_count.total (new — sum) - llm.token_count.cached (unchanged) - llm.provider, llm.cost_usd (unchanged) Reference: https://github.com/Arize-ai/openinference/blob/main/spec/semantic_conventions.md Also coerce int-typed token counts up front so downstream span serializers (OTel expects concrete ints/floats) don't have to re-parse.
``_to_readable_span`` hard-coded ``SpanKind.INTERNAL`` on every export, so even after the LLM span carried the correct OpenInference ``openinference.span.kind = "LLM"`` attribute, the OTel envelope Phoenix received still classified it as an internal generic span. Introduce ``_span_kind_from_openinference`` that maps the attribute to the most fitting OTel kind: - LLM / TOOL → CLIENT (outbound call semantics) - CHAIN / AGENT / * → INTERNAL (orchestration) The attribute itself is preserved on the exported span — Phoenix keys off both the string and the OTel enum.
Two related fixes so Phoenix LLM spans surface every real LLM call with useful attributes: 1. Enrich the ``COST_UPDATE`` payload emitted from ``CostTracker.record_llm_call`` with ``cached_tokens``, ``elapsed_ms``, and ``provider``. Adds keyword-only ``elapsed_ms`` and ``provider`` args (backward-compatible — legacy callers still work and get safe defaults). 2. Fix a latent bug: ``EventBus.emit`` was called on the class, not an instance, so ``event_type`` was silently binding to ``self`` and the ``COST_UPDATE`` handler never fired. Route through ``EventBus()`` (the singleton) instead. 3. Call ``CostTracker().record_llm_call`` from the sourcehunt hunter's per-turn LLM loop as well. It previously called ``_estimate_cost_usd`` (a local wrapper around ``CostTracker.estimate_cost``) that never touched the singleton, so Phoenix saw operator LLM spans but no hunter ones. The local ``total_cost_usd`` tally is preserved unchanged — the additional call only surfaces telemetry. 4. Measure wall-clock latency around both LLM call sites so the backdated Phoenix span duration reflects real API latency. Tests: cover the new payload fields, the OpenInference attribute names on the emitted span, the ``_span_kind_from_openinference`` mapping, and ``bootstrap_from_env`` singleton semantics.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes the Phoenix OTLP tracing that shipped in #131. The observability module
existed but was inert end-to-end — nothing called `connect()`, the OpenInference
attribute names didn't match Phoenix's classifier, the OTel bridge hard-coded
`SpanKind.INTERNAL`, and `EventBus.emit` was called on the class rather than
the singleton so `COST_UPDATE` handlers never fired.
After this PR: exporting `PHOENIX_ENDPOINT` and `PHOENIX_PROJECT` and running
any `clearwing` subcommand (CLI, webui, machine-fd, or sourcehunt) sends real
LLM spans to Phoenix with the correct model / tokens / provider / latency.
Four reviewable commits, one logical change each:
`ObservabilityIntegration.bootstrap_from_env()` singleton factory; wired
into `clearwing.main()` and `create_app()` (with shutdown flush hook).
spans` — rename attributes to `openinference.span.kind`, `llm.model_name`,
`llm.token_count.prompt|completion|total|cached` per the OpenInference spec.
openinference.span.kind` — map LLM/TOOL → CLIENT, CHAIN/AGENT → INTERNAL
instead of hard-coding INTERNAL.
sourcehunt` — add `elapsed_ms` / `provider` / `cached_tokens` to the
payload; fix `EventBus.emit` to use the singleton; wall-clock timing
around both LLM call sites; sourcehunt hunter now emits COST_UPDATE too.
Test plan
`PHOENIX_ENDPOINT=http://127.0.0.1:6006 PHOENIX_PROJECT=clearwing-dev`,
run a small `clearwing operate` invocation, confirm `llm_call` spans
show up in Phoenix UI with correct model / tokens / provider / duration.
Follow-ups (deferred)
`EventType` values and touches busy code paths; deferred.
(async concurrency bug — same-thread asyncio tasks clobber each other's
parent). Only matters once CHAIN spans land and concurrent hunts share a
Tracer.