Repository navigation
feat(agents): fan out intent-tier traces to Phoenix via AOS - #21
Merged
Merged
Conversation
Traces from the four intent-tier agents were previously raw ClickHouse only. Add a third trace sink to the tier-owned collector's fan-out (research.md Decision 8, one emission): - deploy/agents/telemetry.yaml: otlphttp/phoenix exporter (OTLP/HTTP, gzip — the otlp gRPC exporter cannot reach Phoenix) into the traces pipeline; endpoint from AOS_OTLP_HTTP_ENDPOINT (lab default http://172.17.0.1:4318, docker bridge gateway from inside kind). Metrics pipelines unchanged. - deploy/agents/namespace-rbac.yaml: allow-egress-aos-otel (T051) — collector pod only, ipBlock 172.17.0.1/32 TCP 4318; without it the deny-all baseline drops the dial and the sink fails silently. - agents/common/llm.py: stamp agentic-netops.tokens.{prompt,completion, total} on the active span next to the T333-T335 model id / redacted prompt / response. NOTE: the SDK's context scoping currently keeps agentic-netops.* off exported spans (verified live: zero spans carry them); the OpenInference instrumentation records usage natively as gen_ai.usage.* / llm.usage.total_tokens, so this is redundant-but-harmless until the SDK context is fixed. - deploy/agents/README.md: Telemetry (traces) section — three sinks, T051 policy, rollout-restart caveat for the file-mounted collector config, kill switch, verified live span shape (OpenInference-native; A2A round-trips file under the supervisor's Phoenix project). - deploy/agents/telemetry.yaml header: note that the T034 qualified shape is superseded by the verified live shape (see prompts/PHOENIX_TRACES_PROMPT.md, local-only per .gitignore). Verified live: supervisor LLM spans render as LLM with model (openai/gpt-5), token counts and full prompt/completion content; agent decorator + graph-workflow spans render as AGENT; A2A round-trips arrive as single traces. Unit: 268 passed, 4 skipped.
Contributor
Merge Queue Status
This pull request spent 9 seconds in the queue, including 1 second running CI. Required conditions to merge
|
mairp
added a commit
that referenced
this pull request
Oct 19, 2026
Resolve conflicts with #19 (sonic-vm conformance profile) in favour of the feature branch, whose versions are the measured successors of the same code: node_sudo privilege prefix, bounded ssh with -n, seg6 probe on Loopback0 with SKIP 77, spine resolved by name, running-AS peering, one-line image@sha256 pin (FR-009), sonic-vm default profile (T247/T248), and docs that describe the delivered SRv6Service. Brings in #21 (Phoenix trace fan-out) unchanged. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: mairp <oec.valle.art@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The four intent-tier agents (supervisor / mapper / allocator / deployer) emitted OTLP once to the tier-owned collector, which fanned out to ClickHouse (
agent_analytics) and feature 001 (metrics only). Traces had no queryable UI — raw ClickHouse. This adds a third trace sink: the host-side agent-observability-stack collector (Arize Phoenix, UI :3006), per research.md Decision 8 (one emission, one fan-out point).Changes
otlphttp/phoenixexporter (OTLP/HTTP, gzip; the gRPCotlpexporter cannot reach Phoenix) added to the traces pipeline; endpoint fromAOS_OTLP_HTTP_ENDPOINT(lab defaulthttp://172.17.0.1:4318— the docker bridge gateway from inside kind; must always resolve or the collector fails to start). Metrics pipelines untouched.allow-egress-aos-otel(T051): collector pod only, ipBlock172.17.0.1/32TCP 4318. Without it the deny-all baseline drops the dial and the sink fails silently (observed: labelled probe podcode=000, unlabelledcode=004).agentic-netops.tokens.{prompt,completion,total}on the active span next to the T333–T335 model id / redacted prompt / response. Known caveat (verified live): the SDK's context scoping currently keeps allagentic-netops.*attributes off exported spans (zero spans carry them; the T333–T335 redaction stamping is inert too), so this is redundant-but-harmless — the OpenInference instrumentation records usage natively asgen_ai.usage.*/llm.usage.total_tokens.gen_ai.*, resolved model inioa_observe.association.properties.ls_model_name, A2A round-trips as single traces filed under the supervisor's Phoenix project).The grounded prompt driving this work lives at
prompts/PHOENIX_TRACES_PROMPT.md— local-only, as/prompts/is gitignored by repo convention.Verification (live, this lab)
openai/gpt-5(fixed from the literalunknown), token counts, full prompt/completion content.Pairs with
mairp/agent-observability-stack— PR adding thetransform/phoenix_agentic_netopsper-producer transform on the AOS side.Out-of-scope notes (pre-existing, observed)
otlp/feature001gRPC export from the tier collector fails (i/o timeout to 10.101.111.153:4317); zeroagentic_netops_*metrics in Prometheus.otlphttp/phoenixexporter shows intermittent pre-existing EOF / connection-reset errors though most batches succeed.