diff --git a/.agents/product-boundary.md b/.agents/product-boundary.md index 03ff3a52f..50b39c07a 100644 --- a/.agents/product-boundary.md +++ b/.agents/product-boundary.md @@ -25,9 +25,9 @@ After the 2026-06-20 product decision, Phoenix is not an AgentV artifact owner o AgentV must not export or project completed AgentV runs, traces, transcripts, datasets, experiments, or indexes into Phoenix. AgentV-owned Git/GitHub run artifacts and the local Dashboard remain the supported zero-infra inspection path for AgentV run, trace, session, transcript, and comparison data. -Phoenix may be referenced as UI inspiration and as optional external trace infrastructure when Codex, Arize, or another hook already emitted spans independently. The supported integration shape is read-only correlation/read-through through Phoenix GraphQL/API using safe `external_trace` metadata. That metadata can point from an AgentV artifact to an external Phoenix trace or session, but Phoenix does not become the storage, transcript, index, or run system of record. +Phoenix may be referenced as UI inspiration and as optional external trace infrastructure when Codex, Arize, or another hook already emitted spans independently. The supported integration shape is link-out correlation through safe `external_trace` metadata. That metadata can point from an AgentV artifact to an external Phoenix trace or session and provide an `Open in Phoenix` URL, but Phoenix does not become the storage, transcript, index, or run system of record. -Dashboard must not require the `px` CLI at runtime, must not query Phoenix database tables directly, and must not introduce a Phoenix runtime dependency for the zero-infra local path. If Phoenix data is surfaced later, it should be through the public Phoenix GraphQL/API boundary and should degrade cleanly when Phoenix is unavailable. +Dashboard must not require the `px` CLI at runtime, must not query Phoenix database tables directly, must not proxy Phoenix GraphQL/REST, and must not introduce a Phoenix runtime dependency for the zero-infra local path. If Phoenix data is needed, Dashboard should direct users to Phoenix through a safe external link instead of duplicating Phoenix sessions, traces, or spans inside AgentV. AgentV transcript artifacts are not Phoenix-native conversation inputs. Model-call spans may contain cumulative input messages, so treating Phoenix span input as a linear transcript can duplicate or distort the conversation. Keep AgentV transcript/index/storage semantics in AgentV artifacts. diff --git a/AGENTS.md b/AGENTS.md index 3810c54ef..8964b6181 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -17,7 +17,7 @@ Phoenix boundary after the 2026-06-20 product decision: - AgentV-owned run bundles, traces, transcripts, datasets, experiments, indexes, and Git-backed artifacts are not exported or projected into Phoenix. - The local Dashboard is the supported zero-infra inspection path for AgentV run, trace, and session artifacts. - Phoenix may be referenced as UI inspiration and optional external trace infrastructure only when Codex, Arize, or another hook already emitted spans independently. -- Optional Phoenix integration is read-only correlation/read-through through Phoenix GraphQL/API using safe `external_trace` metadata. +- Optional Phoenix integration is link-out correlation only through safe `external_trace` metadata and an `Open in Phoenix` URL when available. - Dashboard must not require the `px` CLI at runtime or query Phoenix database tables directly. Design guardrails: diff --git a/ROADMAP.md b/ROADMAP.md index 1cd6a28ad..fb69a1c16 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -11,7 +11,7 @@ This roadmap translates [STRATEGY.md](STRATEGY.md) into the next few product pha - Phoenix is optional external trace infrastructure only when Codex, Arize, or another hook already emitted spans independently; AgentV may correlate with those sessions but does not write AgentV artifacts into Phoenix. - Harbor stays an optional benchmark-grade runner boundary, not AgentV core. - AgentV YAML remains the authoring surface even when execution moves behind another runner; prefer a lightweight translation layer over duplicated specs. -- Adapters, workers, and artifact projections are preferred over rebuilding adjacent platforms inside AgentV, except Phoenix: Phoenix integration is read-only correlation/read-through and not an AgentV-to-Phoenix projection path. +- Adapters, workers, and artifact projections are preferred over rebuilding adjacent platforms inside AgentV, except Phoenix: Phoenix integration is link-out correlation only and not an AgentV-to-Phoenix projection path. ## Phase 1: Finish the artifact and local inspection foundation @@ -19,12 +19,12 @@ This roadmap translates [STRATEGY.md](STRATEGY.md) into the next few product pha - Finish the vendor-neutral local export seams that let completed runs be re-read, compared, exported, and attached to non-Phoenix adapters without vendor-specific logic in core. - Keep OTLP/OpenInference mapping generic and reusable before building backend-specific upload or import paths. -## Phase 2: Keep Phoenix read-only and externally owned +## Phase 2: Keep Phoenix link-only and externally owned - Document the Phoenix boundary across public docs, CLI help, Dashboard copy, and AI-facing guides. -- Allow optional read-only Phoenix session links or views through Phoenix GraphQL/API when safe `external_trace` metadata points at spans emitted independently by Codex, Arize, or another hook. +- Allow optional Phoenix links when safe `external_trace` metadata points at spans emitted independently by Codex, Arize, or another hook. - Keep Phoenix from owning AgentV transcript, index, run, dataset, experiment, or storage state. -- Keep Dashboard free of a Phoenix runtime dependency: no `px` CLI requirement and no direct reads from Phoenix database tables. +- Keep Dashboard free of a Phoenix runtime dependency: no `px` CLI requirement, no Phoenix GraphQL/REST proxy, and no direct reads from Phoenix database tables. ## Phase 3: Strengthen the local cockpit and the product boundary @@ -38,7 +38,7 @@ This roadmap translates [STRATEGY.md](STRATEGY.md) into the next few product pha - Harbor: move toward Harbor becoming the benchmark-grade runner behind a lightweight translation layer from AgentV YAML, while AgentV stays the authoring, gating, import, and comparison surface. - Harbor: in the near term, launch or import benchmark-grade runs through a runner boundary; over time, converge on Harbor as the execution layer for the suites it already owns. - Opik and similar systems: consume completed AgentV projection bundles as post-run adapters rather than as runtime owners. -- Phoenix: remain excluded from AgentV post-run export/projection; use read-only correlation/read-through only. +- Phoenix: remain excluded from AgentV post-run export/projection; use link-out correlation only. - Additional observability backends should reuse the same projection and export seams instead of adding new core product models. ## Not On This Roadmap diff --git a/STRATEGY.md b/STRATEGY.md index b56e0a90a..a021b0101 100644 --- a/STRATEGY.md +++ b/STRATEGY.md @@ -40,7 +40,7 @@ _Why it serves the approach:_ Portable artifacts let local runs, CI, static repo ### Adapter-led integrations -Add Phoenix, Harbor, Opik, Langfuse, and similar systems through narrow correlation, runner, adapter, or export boundaries rather than copying their product models into core. For Phoenix specifically, the supported boundary is read-only correlation/read-through from safe `external_trace` metadata; AgentV does not export or project completed runs, traces, transcripts, datasets, experiments, or indexes into Phoenix. +Add Phoenix, Harbor, Opik, Langfuse, and similar systems through narrow correlation, runner, adapter, or export boundaries rather than copying their product models into core. For Phoenix specifically, the supported boundary is link-out correlation from safe `external_trace` metadata; AgentV does not read through, export, or project completed runs, traces, transcripts, datasets, experiments, or indexes into Phoenix. _Why it serves the approach:_ This expands AgentV's reach without turning it into a hosted observability stack, benchmark platform, or integration kitchen sink. diff --git a/apps/cli/src/commands/results/phoenix-readthrough.ts b/apps/cli/src/commands/results/phoenix-readthrough.ts deleted file mode 100644 index 6538d30dc..000000000 --- a/apps/cli/src/commands/results/phoenix-readthrough.ts +++ /dev/null @@ -1,1555 +0,0 @@ -import type { ExternalTraceMetadata, ExternalTraceMetadataWire } from '@agentv/core'; - -const RESPONSE_SCHEMA_VERSION = 'agentv.dashboard.phoenix_session.v1'; -const DEFAULT_PAGE_LIMIT = 1000; -const REDACTED_VALUE = '[redacted]'; - -export type PhoenixReadStatus = - | 'ok' - | 'missing_external_trace' - | 'not_configured' - | 'unresolved' - | 'unreachable' - | 'schema_mismatch'; - -export interface PhoenixTokenUsage { - readonly input?: number; - readonly output?: number; - readonly reasoning?: number; - readonly cached?: number; - readonly total?: number; -} - -export interface PhoenixAnnotation { - readonly id?: string; - readonly name?: string; - readonly annotator_kind?: string; - readonly label?: string; - readonly score?: number; - readonly explanation?: string; - readonly target?: 'session' | 'trace' | 'span'; - readonly target_id?: string; - readonly result?: unknown; -} - -export interface PhoenixSpanDetail { - readonly span_id: string; - readonly trace_id?: string; - readonly parent_span_id?: string; - readonly name?: string; - readonly span_kind?: string; - readonly status?: string; - readonly start_time?: string; - readonly end_time?: string; - readonly duration_ms?: number; - readonly input?: unknown; - readonly output?: unknown; - readonly token_usage?: PhoenixTokenUsage; - readonly cost_usd?: number; - readonly attributes?: Record; - readonly annotations?: readonly PhoenixAnnotation[]; -} - -export interface PhoenixTraceTreeNode extends PhoenixSpanDetail { - readonly depth: number; - readonly child_count: number; -} - -export interface PhoenixSessionTurn { - readonly index: number; - readonly trace_id?: string; - readonly start_time?: string; - readonly end_time?: string; - readonly duration_ms?: number; - readonly status?: string; - readonly root_span_id?: string; - readonly input?: unknown; - readonly output?: unknown; - readonly token_usage?: PhoenixTokenUsage; - readonly cost_usd?: number; - readonly annotations?: readonly PhoenixAnnotation[]; -} - -export interface PhoenixSessionSummary { - readonly id?: string; - readonly session_id?: string; - readonly project_id?: string; - readonly project?: string; - readonly start_time?: string; - readonly end_time?: string; - readonly duration_ms?: number; - readonly trace_count: number; - readonly token_usage?: PhoenixTokenUsage; - readonly cost_usd?: number; - readonly annotations?: readonly PhoenixAnnotation[]; -} - -export interface PhoenixLinkedSessionResponse { - readonly schema_version: typeof RESPONSE_SCHEMA_VERSION; - readonly status: PhoenixReadStatus; - readonly message?: string; - readonly external_trace?: ExternalTraceMetadataWire; - readonly open_in_phoenix_url?: string; - readonly session?: PhoenixSessionSummary; - readonly turns?: readonly PhoenixSessionTurn[]; - readonly spans?: readonly PhoenixSpanDetail[]; - readonly trace_tree?: readonly PhoenixTraceTreeNode[]; - readonly annotations?: readonly PhoenixAnnotation[]; -} - -interface PhoenixConfig { - readonly baseUrl: string; - readonly graphqlUrl: string; - readonly project?: string; - readonly headers: Record; -} - -interface ReadPhoenixLinkedSessionOptions { - readonly env?: Readonly>; - readonly fetch?: typeof fetch; -} - -interface PhoenixRestSession { - readonly id?: string; - readonly sessionId?: string; - readonly projectId?: string; - readonly startTime?: string; - readonly endTime?: string; - readonly traces: readonly PhoenixRestTrace[]; - readonly raw?: Record; -} - -interface PhoenixRestTrace { - readonly traceId?: string; - readonly startTime?: string; - readonly endTime?: string; - readonly raw?: Record; -} - -class PhoenixReadError extends Error { - constructor( - readonly code: Exclude, - message: string, - ) { - super(message); - this.name = 'PhoenixReadError'; - } -} - -const PHOENIX_LINKED_SESSION_GRAPHQL_FIELDS = ` - fragment AgentVPhoenixSpanFields on Span { - id - spanId - parentId - name - spanKind - statusCode: propagatedStatusCode - startTime - endTime - latencyMs - input { - value - truncatedValue - mimeType - } - output { - value - truncatedValue - mimeType - } - attributes - tokenCountTotal - tokenCountPrompt - tokenCountCompletion - cumulativeTokenCountTotal - cumulativeTokenCountPrompt - cumulativeTokenCountCompletion - costSummary { - prompt { - cost - tokens - } - completion { - cost - tokens - } - total { - cost - tokens - } - } - spanAnnotations { - id - name - annotatorKind - label - score - explanation - metadata - identifier - spanId - createdAt - updatedAt - } - trace { - id - traceId - costSummary { - total { - cost - tokens - } - } - } - project { - id - name - } - } - - fragment AgentVPhoenixTraceFields on Trace { - id - traceId - startTime - endTime - latencyMs - projectId - project { - id - name - } - projectSessionId - costSummary { - total { - cost - tokens - } - } - traceAnnotations { - id - name - annotatorKind - label - score - explanation - metadata - identifier - createdAt - updatedAt - } - rootSpan { - ...AgentVPhoenixSpanFields - } - spans(first: $spanFirst) { - edges { - node { - ...AgentVPhoenixSpanFields - } - } - } - } - - fragment AgentVPhoenixSessionFields on ProjectSession { - id - sessionId - startTime - endTime - project { - id - name - } - numTraces - tokenUsage { - prompt - completion - total - } - costSummary { - prompt { - cost - tokens - } - completion { - cost - tokens - } - total { - cost - tokens - } - } - sessionAnnotations { - id - name - annotatorKind - label - score - explanation - metadata - identifier - projectSessionId - createdAt - updatedAt - } - traces(first: $traceFirst) { - edges { - node { - ...AgentVPhoenixTraceFields - } - } - } - } -`; - -function isRecord(value: unknown): value is Record { - return typeof value === 'object' && value !== null && !Array.isArray(value); -} - -function asArray(value: unknown): unknown[] { - return Array.isArray(value) ? value : []; -} - -function stringValue(value: unknown): string | undefined { - return typeof value === 'string' && value.trim().length > 0 ? value.trim() : undefined; -} - -function isCredentialLikeKey(key: string): boolean { - const normalized = key.toLowerCase(); - if (normalized === 'traceparent' || normalized === 'tracestate') { - return false; - } - return /(^|[._-])(api[._-]?key|authorization|bearer|password|secret|private[._-]?key|access[._-]?token|auth[._-]?token|client[._-]?secret|cookie|id[._-]?token|refresh[._-]?token|session[._-]?token|token)($|[._-])/.test( - normalized, - ); -} - -function scrubCredentialFields(value: unknown, depth = 0): unknown { - if (depth > 20) { - return value; - } - if (Array.isArray(value)) { - return value.map((entry) => scrubCredentialFields(entry, depth + 1)); - } - if (!isRecord(value)) { - return value; - } - return Object.fromEntries( - Object.entries(value).map(([key, entry]) => [ - key, - isCredentialLikeKey(key) ? REDACTED_VALUE : scrubCredentialFields(entry, depth + 1), - ]), - ); -} - -function scrubCredentialRecord( - value: Record | undefined, -): Record | undefined { - const scrubbed = scrubCredentialFields(value); - return isRecord(scrubbed) ? scrubbed : undefined; -} - -function finiteNumber(value: unknown): number | undefined { - return typeof value === 'number' && Number.isFinite(value) ? value : undefined; -} - -function firstNumber(...values: unknown[]): number | undefined { - for (const value of values) { - const numberValue = finiteNumber(value); - if (numberValue !== undefined) { - return numberValue; - } - } - return undefined; -} - -function compactRecord>(value: T): T | undefined { - const entries = Object.entries(value).filter(([, entry]) => entry !== undefined); - return entries.length > 0 ? (Object.fromEntries(entries) as T) : undefined; -} - -function durationMs( - startTime: string | undefined, - endTime: string | undefined, -): number | undefined { - if (!startTime || !endTime) { - return undefined; - } - const start = Date.parse(startTime); - const end = Date.parse(endTime); - if (Number.isNaN(start) || Number.isNaN(end)) { - return undefined; - } - return Math.max(0, end - start); -} - -function normalizeBaseUrl(value: string | undefined): string | undefined { - if (!value) { - return undefined; - } - try { - const url = new URL(value); - if (!['http:', 'https:'].includes(url.protocol) || url.username || url.password) { - return undefined; - } - url.search = ''; - url.hash = ''; - url.pathname = url.pathname - .replace(/\/graphql\/?$/i, '') - .replace(/\/v1\/traces\/?$/i, '') - .replace(/\/v1\/?$/i, '') - .replace(/\/+$/g, ''); - if (!url.pathname) { - url.pathname = '/'; - } - return url.toString().replace(/\/+$/, ''); - } catch { - return undefined; - } -} - -function normalizeGraphqlUrl(value: string | undefined): string | undefined { - const baseUrl = normalizeBaseUrl(value); - return baseUrl ? `${baseUrl}/graphql` : undefined; -} - -function envString( - env: Readonly>, - keys: readonly string[], -): string | undefined { - for (const key of keys) { - const value = stringValue(env[key]); - if (value) { - return value; - } - } - return undefined; -} - -function parseHeaderEntries(value: string | undefined): Record { - const raw = stringValue(value); - if (!raw) { - return {}; - } - - try { - const parsed = JSON.parse(raw) as unknown; - if (isRecord(parsed)) { - return Object.fromEntries( - Object.entries(parsed).flatMap(([name, headerValue]) => { - const normalizedName = stringValue(name); - const normalizedValue = stringValue(headerValue); - return normalizedName && normalizedValue ? [[normalizedName, normalizedValue]] : []; - }), - ); - } - } catch { - // Fall through to the PHOENIX_CLIENT_HEADERS comma-separated form. - } - - const headers: Record = {}; - for (const segment of raw.split(',')) { - const separator = segment.indexOf('='); - if (separator <= 0) { - continue; - } - const name = decodeURIComponent(segment.slice(0, separator).trim()); - const headerValue = decodeURIComponent(segment.slice(separator + 1).trim()); - if (name && headerValue) { - headers[name] = headerValue; - } - } - return headers; -} - -function hasHeader(headers: Record, name: string): boolean { - const normalized = name.toLowerCase(); - return Object.keys(headers).some((header) => header.toLowerCase() === normalized); -} - -function resolvePhoenixConfig( - externalTrace: ExternalTraceMetadata, - env: Readonly>, -): PhoenixConfig | undefined { - const baseUrl = normalizeBaseUrl( - envString(env, [ - 'AGENTV_PHOENIX_ENDPOINT', - 'AGENTV_PHOENIX_BASE_URL', - 'AGENTV_PHOENIX_HOST', - 'PHOENIX_HOST', - 'PHOENIX_BASE_URL', - 'PHOENIX_ENDPOINT', - 'PHOENIX_COLLECTOR_ENDPOINT', - 'OTEL_EXPORTER_OTLP_TRACES_ENDPOINT', - 'OTEL_EXPORTER_OTLP_ENDPOINT', - ]), - ); - const graphqlUrl = - normalizeGraphqlUrl(envString(env, ['AGENTV_PHOENIX_GRAPHQL_URL', 'PHOENIX_GRAPHQL_URL'])) ?? - normalizeGraphqlUrl(baseUrl); - - if (!baseUrl || !graphqlUrl) { - return undefined; - } - - const headers = { - ...parseHeaderEntries(envString(env, ['AGENTV_PHOENIX_HEADERS', 'PHOENIX_CLIENT_HEADERS'])), - }; - const authorization = envString(env, ['AGENTV_PHOENIX_AUTHORIZATION', 'PHOENIX_AUTHORIZATION']); - const apiKey = envString(env, ['AGENTV_PHOENIX_API_KEY', 'PHOENIX_API_KEY']); - const cookie = envString(env, ['AGENTV_PHOENIX_COOKIE', 'PHOENIX_COOKIE']); - if (authorization && !hasHeader(headers, 'authorization')) { - headers.Authorization = authorization; - } - if (apiKey && !hasHeader(headers, 'authorization')) { - headers.Authorization = `Bearer ${apiKey}`; - } - if (cookie && !hasHeader(headers, 'cookie')) { - headers.Cookie = cookie; - } - - return { - baseUrl, - graphqlUrl, - project: - externalTrace.projectId ?? - externalTrace.project ?? - envString(env, [ - 'AGENTV_PHOENIX_PROJECT_ID', - 'AGENTV_PHOENIX_PROJECT', - 'AGENTV_PHOENIX_PROJECT_NAME', - 'PHOENIX_PROJECT', - 'PHOENIX_PROJECT_ID', - 'PHOENIX_PROJECT_NAME', - 'OPENINFERENCE_PROJECT_NAME', - ]), - headers, - }; -} - -function response( - status: PhoenixReadStatus, - message: string, - externalTrace: ExternalTraceMetadataWire | undefined, - details: Omit = {}, -): PhoenixLinkedSessionResponse { - return { - schema_version: RESPONSE_SCHEMA_VERSION, - status, - message, - ...(externalTrace && { external_trace: externalTrace }), - ...details, - }; -} - -function encodePathSegment(value: string): string { - return encodeURIComponent(value).replace(/%2F/g, '%252F'); -} - -function appendQuery(url: URL, key: string, value: string | readonly string[] | undefined): void { - if (value === undefined) { - return; - } - if (Array.isArray(value)) { - for (const entry of value) { - url.searchParams.append(key, entry); - } - return; - } - if (typeof value === 'string') { - url.searchParams.set(key, value); - } -} - -async function fetchJson( - requestUrl: string, - init: RequestInit, - fetchImpl: typeof fetch, -): Promise { - let responseValue: Response; - try { - responseValue = await fetchImpl(requestUrl, init); - } catch (error) { - throw new PhoenixReadError( - 'unreachable', - `Phoenix request failed: ${error instanceof Error ? error.message : String(error)}`, - ); - } - - if (responseValue.status === 404) { - throw new PhoenixReadError('unresolved', 'No matching Phoenix session or trace was found.'); - } - if (!responseValue.ok) { - throw new PhoenixReadError( - responseValue.status >= 500 ? 'unreachable' : 'schema_mismatch', - `Phoenix returned HTTP ${responseValue.status}.`, - ); - } - - try { - return await responseValue.json(); - } catch { - throw new PhoenixReadError('schema_mismatch', 'Phoenix returned a non-JSON response.'); - } -} - -async function phoenixRestGet( - config: PhoenixConfig, - pathName: string, - query: Record, - fetchImpl: typeof fetch, -): Promise { - const url = new URL( - pathName, - config.baseUrl.endsWith('/') ? config.baseUrl : `${config.baseUrl}/`, - ); - for (const [key, value] of Object.entries(query)) { - appendQuery(url, key, value); - } - return fetchJson( - url.toString(), - { - method: 'GET', - headers: { - Accept: 'application/json', - ...config.headers, - }, - }, - fetchImpl, - ); -} - -async function phoenixGraphql( - config: PhoenixConfig, - body: { query: string; variables: Record }, - fetchImpl: typeof fetch, -): Promise { - const payload = await fetchJson( - config.graphqlUrl, - { - method: 'POST', - headers: { - Accept: 'application/json', - 'Content-Type': 'application/json', - ...config.headers, - }, - body: JSON.stringify(body), - }, - fetchImpl, - ); - const record = isRecord(payload) ? payload : {}; - if (Array.isArray(record.errors) && record.errors.length > 0) { - throw new PhoenixReadError('schema_mismatch', 'Phoenix GraphQL returned schema errors.'); - } - return record.data; -} - -function normalizeTrace(raw: unknown): PhoenixRestTrace | undefined { - const record = isRecord(raw) ? raw : undefined; - if (!record) { - return undefined; - } - const traceId = stringValue(record.trace_id) ?? stringValue(record.traceId); - if (!traceId) { - return undefined; - } - return { - traceId, - startTime: stringValue(record.start_time) ?? stringValue(record.startTime), - endTime: stringValue(record.end_time) ?? stringValue(record.endTime), - raw: record, - }; -} - -function normalizeSession(raw: unknown): PhoenixRestSession | undefined { - const record = isRecord(raw) ? raw : undefined; - if (!record) { - return undefined; - } - const data = isRecord(record.data) ? record.data : record; - const traces = asArray(data.traces) - .map(normalizeTrace) - .filter((trace): trace is PhoenixRestTrace => trace !== undefined); - const sessionId = - stringValue(data.session_id) ?? stringValue(data.sessionId) ?? stringValue(data.id); - if (!sessionId && traces.length === 0) { - return undefined; - } - return { - id: stringValue(data.id), - sessionId, - projectId: stringValue(data.project_id) ?? stringValue(data.projectId), - startTime: stringValue(data.start_time) ?? stringValue(data.startTime), - endTime: stringValue(data.end_time) ?? stringValue(data.endTime), - traces, - raw: data, - }; -} - -function normalizeGraphqlSession(raw: unknown): PhoenixRestSession | undefined { - const node = sessionNodeFromGraphqlData(raw); - if (!node) { - return undefined; - } - - const traceNodes = connectionNodes(node.traces) - .map(normalizeTrace) - .filter((trace): trace is PhoenixRestTrace => trace !== undefined); - const sessionId = - stringValue(node.session_id) ?? stringValue(node.sessionId) ?? stringValue(node.id); - if (!sessionId && traceNodes.length === 0) { - return undefined; - } - - return { - id: stringValue(node.id), - sessionId, - projectId: projectIdFromGraphqlNode(node), - startTime: stringValue(node.start_time) ?? stringValue(node.startTime), - endTime: stringValue(node.end_time) ?? stringValue(node.endTime), - traces: traceNodes, - raw: node, - }; -} - -async function fetchSessionByNodeId( - config: PhoenixConfig, - sessionNodeId: string, - fetchImpl: typeof fetch, -): Promise { - const data = await phoenixGraphql( - config, - { - query: ` - query AgentVPhoenixLinkedSessionByNode($id: ID!, $traceFirst: Int!, $spanFirst: Int!) { - node(id: $id) { - __typename - ... on ProjectSession { - ...AgentVPhoenixSessionFields - } - } - } - ${PHOENIX_LINKED_SESSION_GRAPHQL_FIELDS} - `, - variables: { id: sessionNodeId, traceFirst: 100, spanFirst: DEFAULT_PAGE_LIMIT }, - }, - fetchImpl, - ); - return normalizeGraphqlSession(data); -} - -async function fetchSessionByIdGraphql( - config: PhoenixConfig, - projectIdentifier: string, - sessionId: string, - fetchImpl: typeof fetch, -): Promise { - const data = await phoenixGraphql( - config, - { - query: ` - query AgentVPhoenixLinkedSessionBySessionId( - $projectId: ID! - $sessionId: String! - $traceFirst: Int! - $spanFirst: Int! - ) { - project: node(id: $projectId) { - __typename - ... on Project { - sessions(first: 1, sessionId: $sessionId) { - edges { - node { - ...AgentVPhoenixSessionFields - } - } - } - } - } - } - ${PHOENIX_LINKED_SESSION_GRAPHQL_FIELDS} - `, - variables: { - projectId: projectIdentifier, - sessionId, - traceFirst: 100, - spanFirst: DEFAULT_PAGE_LIMIT, - }, - }, - fetchImpl, - ); - return normalizeGraphqlSession(data); -} - -async function fetchSessionByTraceIdGraphql( - config: PhoenixConfig, - projectIdentifier: string, - traceId: string, - fetchImpl: typeof fetch, -): Promise { - const data = await phoenixGraphql( - config, - { - query: ` - query AgentVPhoenixLinkedSessionByTraceId( - $projectId: ID! - $traceId: ID! - $traceFirst: Int! - $spanFirst: Int! - ) { - project: node(id: $projectId) { - __typename - ... on Project { - trace(traceId: $traceId) { - session { - ...AgentVPhoenixSessionFields - } - } - } - } - } - ${PHOENIX_LINKED_SESSION_GRAPHQL_FIELDS} - `, - variables: { - projectId: projectIdentifier, - traceId, - traceFirst: 100, - spanFirst: DEFAULT_PAGE_LIMIT, - }, - }, - fetchImpl, - ); - return normalizeGraphqlSession(data); -} - -async function fetchSessionByIdRest( - config: PhoenixConfig, - sessionId: string, - fetchImpl: typeof fetch, -): Promise { - return normalizeSession( - await phoenixRestGet(config, `v1/sessions/${encodePathSegment(sessionId)}`, {}, fetchImpl), - ); -} - -function extractRecords(payload: unknown): Record[] { - const record = isRecord(payload) ? payload : {}; - const data = isRecord(record.data) ? record.data.data : record.data; - return asArray(data) - .map((entry) => (isRecord(entry) ? entry : undefined)) - .filter((entry): entry is Record => entry !== undefined); -} - -function connectionNodes(value: unknown): Record[] { - if (Array.isArray(value)) { - return value.filter((entry): entry is Record => isRecord(entry)); - } - const record = isRecord(value) ? value : undefined; - return asArray(record?.edges) - .map((edge) => (isRecord(edge) && isRecord(edge.node) ? edge.node : edge)) - .filter((entry): entry is Record => isRecord(entry)); -} - -function projectIdFromGraphqlNode(node: Record): string | undefined { - const project = isRecord(node.project) ? node.project : undefined; - return stringValue(node.projectId) ?? stringValue(project?.id); -} - -function sessionNodeFromGraphqlData(raw: unknown): Record | undefined { - const data = isRecord(raw) ? raw : undefined; - if (!data) { - return undefined; - } - if (isRecord(data.node) && stringValue(data.node.sessionId)) { - return data.node; - } - if (isRecord(data.projectSession)) { - return data.projectSession; - } - const project = isRecord(data.project) ? data.project : undefined; - const sessionFromProject = connectionNodes(project?.sessions)[0]; - if (sessionFromProject) { - return sessionFromProject; - } - const trace = isRecord(project?.trace) ? project.trace : undefined; - return isRecord(trace?.session) ? trace.session : undefined; -} - -async function fetchTraceSpans( - config: PhoenixConfig, - projectIdentifier: string, - traceId: string, - fetchImpl: typeof fetch, -): Promise[]> { - return extractRecords( - await phoenixRestGet( - config, - `v1/projects/${encodePathSegment(projectIdentifier)}/spans`, - { limit: String(DEFAULT_PAGE_LIMIT), trace_id: traceId }, - fetchImpl, - ), - ); -} - -async function fetchAnnotations( - config: PhoenixConfig, - projectIdentifier: string, - kind: 'session' | 'trace' | 'span', - ids: readonly string[], - fetchImpl: typeof fetch, -): Promise { - if (ids.length === 0) { - return []; - } - const queryKey = kind === 'session' ? 'session_ids' : kind === 'trace' ? 'trace_ids' : 'span_ids'; - const pathName = `v1/projects/${encodePathSegment(projectIdentifier)}/${kind}_annotations`; - try { - return extractRecords( - await phoenixRestGet( - config, - pathName, - { [queryKey]: ids, limit: String(DEFAULT_PAGE_LIMIT) }, - fetchImpl, - ), - ).map((annotation) => normalizeAnnotation(annotation, kind)); - } catch (error) { - if (error instanceof PhoenixReadError && error.code === 'unresolved') { - return []; - } - throw error; - } -} - -function normalizeAnnotation( - annotation: Record, - target: 'session' | 'trace' | 'span', - targetId?: string, -): PhoenixAnnotation { - const result = isRecord(annotation.result) ? annotation.result : undefined; - return { - id: stringValue(annotation.id), - name: stringValue(annotation.name), - annotator_kind: stringValue(annotation.annotator_kind) ?? stringValue(annotation.annotatorKind), - label: stringValue(annotation.label) ?? stringValue(result?.label), - score: finiteNumber(annotation.score) ?? finiteNumber(result?.score), - explanation: stringValue(annotation.explanation) ?? stringValue(result?.explanation), - target, - target_id: - stringValue(annotation.session_id) ?? - stringValue(annotation.projectSessionId) ?? - stringValue(annotation.trace_id) ?? - stringValue(annotation.span_id) ?? - stringValue(annotation.spanId) ?? - targetId, - result: scrubCredentialFields(annotation.result ?? annotation.metadata), - }; -} - -function spanContext(span: Record): Record { - return isRecord(span.context) ? span.context : span; -} - -function spanAttributes(span: Record): Record | undefined { - if (isRecord(span.attributes)) { - return span.attributes; - } - const raw = stringValue(span.attributes); - if (!raw) { - return undefined; - } - try { - const parsed = JSON.parse(raw) as unknown; - return isRecord(parsed) ? parsed : undefined; - } catch { - return undefined; - } -} - -function attributeString( - attributes: Record | undefined, - keys: readonly string[], -): string | undefined { - if (!attributes) { - return undefined; - } - for (const key of keys) { - const value = stringValue(attributes[key]); - if (value) { - return value; - } - } - return undefined; -} - -function attributeNumber( - attributes: Record | undefined, - keys: readonly string[], -): number | undefined { - if (!attributes) { - return undefined; - } - for (const key of keys) { - const value = finiteNumber(attributes[key]); - if (value !== undefined) { - return value; - } - } - return undefined; -} - -function tokenUsageFromSpan( - span: Record, - attributes: Record | undefined, -): PhoenixTokenUsage | undefined { - const nested = isRecord(attributes?.token_usage) ? attributes.token_usage : {}; - return compactRecord({ - input: firstNumber( - nested.input, - nested.prompt, - span.input_tokens, - span.tokenCountPrompt, - span.cumulativeTokenCountPrompt, - attributeNumber(attributes, ['gen_ai.usage.input_tokens', 'llm.token_count.prompt']), - ), - output: firstNumber( - nested.output, - nested.completion, - span.output_tokens, - span.tokenCountCompletion, - span.cumulativeTokenCountCompletion, - attributeNumber(attributes, ['gen_ai.usage.output_tokens', 'llm.token_count.completion']), - ), - reasoning: firstNumber(nested.reasoning, attributes?.['gen_ai.usage.reasoning.output_tokens']), - cached: firstNumber(nested.cached, attributes?.['gen_ai.usage.cache_read.input_tokens']), - total: firstNumber( - nested.total, - span.total_tokens, - span.tokenCountTotal, - span.cumulativeTokenCountTotal, - attributes?.total_tokens, - ), - }) as PhoenixTokenUsage | undefined; -} - -function spanInput(span: Record, attributes: Record | undefined) { - const input = isRecord(span.input) ? span.input.value : span.input; - return ( - input ?? - attributes?.['input.value'] ?? - attributes?.input ?? - attributes?.['llm.input_messages'] ?? - attributes?.['gen_ai.prompt'] - ); -} - -function spanOutput( - span: Record, - attributes: Record | undefined, -) { - const output = isRecord(span.output) ? span.output.value : span.output; - return ( - output ?? - attributes?.['output.value'] ?? - attributes?.output ?? - attributes?.['llm.output_messages'] ?? - attributes?.['gen_ai.completion'] - ); -} - -function normalizeSpan( - span: Record, - annotationsBySpanId: Map, -): PhoenixSpanDetail | undefined { - const context = spanContext(span); - const attributes = spanAttributes(span); - const spanId = - stringValue(context.span_id) ?? - stringValue(span.span_id) ?? - stringValue(span.spanId) ?? - stringValue(span.id); - if (!spanId) { - return undefined; - } - const trace = isRecord(span.trace) ? span.trace : undefined; - const costSummary = isRecord(span.costSummary) ? span.costSummary : undefined; - const totalCostSummary = isRecord(costSummary?.total) ? costSummary.total : undefined; - const traceId = - stringValue(context.trace_id) ?? stringValue(span.trace_id) ?? stringValue(trace?.traceId); - const startTime = stringValue(span.start_time) ?? stringValue(span.startTime); - const endTime = stringValue(span.end_time) ?? stringValue(span.endTime); - const annotations = annotationsBySpanId.get(spanId); - - return compactRecord({ - span_id: spanId, - trace_id: traceId, - parent_span_id: - stringValue(span.parent_id) ?? stringValue(span.parent_span_id) ?? stringValue(span.parentId), - name: stringValue(span.name), - span_kind: - stringValue(span.span_kind) ?? - stringValue(span.spanKind) ?? - attributeString(attributes, ['openinference.span.kind', 'span.kind']), - status: - stringValue(span.status_code) ?? - stringValue(span.statusCode) ?? - stringValue(isRecord(span.status) ? span.status.code : undefined), - start_time: startTime, - end_time: endTime, - duration_ms: finiteNumber(span.latency_ms) ?? durationMs(startTime, endTime), - input: scrubCredentialFields(spanInput(span, attributes)), - output: scrubCredentialFields(spanOutput(span, attributes)), - token_usage: tokenUsageFromSpan(span, attributes), - cost_usd: firstNumber( - span.cost_usd, - span.cost, - totalCostSummary?.cost, - attributes?.cost, - attributes?.['llm.cost.usd'], - ), - attributes: scrubCredentialRecord(attributes), - ...(annotations && annotations.length > 0 ? { annotations } : {}), - }) as PhoenixSpanDetail | undefined; -} - -function buildSpanTree(spans: readonly PhoenixSpanDetail[]): PhoenixTraceTreeNode[] { - const childrenByParent = new Map(); - for (const span of spans) { - const parentId = span.parent_span_id; - const children = childrenByParent.get(parentId) ?? []; - children.push(span); - childrenByParent.set(parentId, children); - } - - const visited = new Set(); - const nodes: PhoenixTraceTreeNode[] = []; - const roots = spans.filter( - (span) => !span.parent_span_id || !spans.some((s) => s.span_id === span.parent_span_id), - ); - - function visit(span: PhoenixSpanDetail, depth: number): void { - if (visited.has(span.span_id)) { - return; - } - visited.add(span.span_id); - const children = childrenByParent.get(span.span_id) ?? []; - nodes.push({ ...span, depth, child_count: children.length }); - for (const child of children) { - visit(child, depth + 1); - } - } - - for (const root of roots) { - visit(root, 0); - } - for (const span of spans) { - visit(span, 0); - } - return nodes; -} - -function sumTokenUsage( - items: readonly { token_usage?: PhoenixTokenUsage }[], -): PhoenixTokenUsage | undefined { - const total: { - input?: number; - output?: number; - reasoning?: number; - cached?: number; - total?: number; - } = {}; - function add(key: keyof typeof total, value: number | undefined): void { - if (value === undefined) { - return; - } - total[key] = (total[key] ?? 0) + value; - } - for (const item of items) { - const usage = item.token_usage; - if (!usage) { - continue; - } - add('input', usage.input); - add('output', usage.output); - add('reasoning', usage.reasoning); - add('cached', usage.cached); - add('total', usage.total); - } - return compactRecord(total) as PhoenixTokenUsage | undefined; -} - -function tokenUsageFromGraphql(value: unknown): PhoenixTokenUsage | undefined { - const record = isRecord(value) ? value : undefined; - if (!record) { - return undefined; - } - return compactRecord({ - input: firstNumber(record.input, record.prompt), - output: firstNumber(record.output, record.completion), - reasoning: finiteNumber(record.reasoning), - cached: finiteNumber(record.cached), - total: finiteNumber(record.total), - }) as PhoenixTokenUsage | undefined; -} - -function costFromGraphqlSummary(value: unknown): number | undefined { - const record = isRecord(value) ? value : undefined; - const total = isRecord(record?.total) ? record.total : undefined; - return finiteNumber(total?.cost); -} - -function annotationsByTarget( - annotations: readonly PhoenixAnnotation[], - target: 'trace' | 'span', -): Map { - const byTarget = new Map(); - for (const annotation of annotations) { - if (annotation.target !== target || !annotation.target_id) { - continue; - } - const entries = byTarget.get(annotation.target_id) ?? []; - entries.push(annotation); - byTarget.set(annotation.target_id, entries); - } - return byTarget; -} - -function graphqlSpanId(span: Record): string | undefined { - const context = spanContext(span); - return stringValue(context.span_id) ?? stringValue(span.span_id) ?? stringValue(span.spanId); -} - -function graphqlTraceRows(session: PhoenixRestSession): Record[] { - const rows: Record[] = []; - const seen = new Set(); - for (const trace of session.traces) { - const rawTrace = trace.raw; - if (!rawTrace) { - continue; - } - const candidates = [ - ...(isRecord(rawTrace.rootSpan) ? [rawTrace.rootSpan] : []), - ...connectionNodes(rawTrace.spans), - ]; - for (const span of candidates) { - const spanId = graphqlSpanId(span); - const dedupeKey = `${trace.traceId ?? ''}:${spanId ?? JSON.stringify(span)}`; - if (seen.has(dedupeKey)) { - continue; - } - seen.add(dedupeKey); - rows.push(span); - } - } - return rows; -} - -function graphqlAnnotations( - entries: readonly unknown[], - target: 'session' | 'trace' | 'span', - targetId?: string, -): PhoenixAnnotation[] { - return entries - .filter((entry): entry is Record => isRecord(entry)) - .map((annotation) => normalizeAnnotation(annotation, target, targetId)); -} - -function graphqlSessionAnnotations(session: PhoenixRestSession): PhoenixAnnotation[] { - return graphqlAnnotations( - asArray(session.raw?.sessionAnnotations), - 'session', - session.sessionId ?? session.id, - ); -} - -function graphqlTraceAnnotations(session: PhoenixRestSession): PhoenixAnnotation[] { - return session.traces.flatMap((trace) => - graphqlAnnotations(asArray(trace.raw?.traceAnnotations), 'trace', trace.traceId), - ); -} - -function graphqlSpanAnnotations(spans: readonly Record[]): PhoenixAnnotation[] { - return spans.flatMap((span) => - graphqlAnnotations(asArray(span.spanAnnotations), 'span', graphqlSpanId(span)), - ); -} - -function sessionIdFromSpans(spans: readonly Record[]): string | undefined { - for (const span of spans) { - const attributes = spanAttributes(span); - const sessionId = - attributeString(attributes, [ - 'session.id', - 'session_id', - 'metadata.session_id', - 'openinference.session.id', - ]) ?? stringValue(span.session_id); - if (sessionId) { - return sessionId; - } - } - return undefined; -} - -function buildOpenInPhoenixUrl( - config: PhoenixConfig, - externalTrace: ExternalTraceMetadata, - session: PhoenixRestSession | undefined, -): string | undefined { - if (externalTrace.uiUrl) { - return externalTrace.uiUrl; - } - const sessionId = session?.sessionId ?? externalTrace.sessionId ?? externalTrace.sessionNodeId; - if (sessionId) { - return `${config.baseUrl}/sessions/${encodeURIComponent(sessionId)}`; - } - if (externalTrace.traceId) { - return `${config.baseUrl}/projects/${encodeURIComponent( - externalTrace.projectId ?? externalTrace.project ?? config.project ?? '', - )}/traces/${encodeURIComponent(externalTrace.traceId)}`; - } - return config.baseUrl; -} - -async function buildLinkedSessionResponse( - externalTrace: ExternalTraceMetadata, - externalTraceWire: ExternalTraceMetadataWire, - config: PhoenixConfig, - session: PhoenixRestSession, - fetchImpl: typeof fetch, -): Promise { - const projectIdentifier = - session.projectId ?? externalTrace.projectId ?? externalTrace.project ?? config.project; - const traceIds = session.traces.map((trace) => trace.traceId).filter((id): id is string => !!id); - const graphqlSpanRows = graphqlTraceRows(session); - const spanRows = - graphqlSpanRows.length > 0 - ? graphqlSpanRows - : projectIdentifier - ? ( - await Promise.all( - traceIds.map((traceId) => - fetchTraceSpans(config, projectIdentifier, traceId, fetchImpl), - ), - ) - ).flat() - : []; - - const graphqlSessionAnnotationRows = graphqlSessionAnnotations(session); - const graphqlTraceAnnotationRows = graphqlTraceAnnotations(session); - const graphqlSpanAnnotationRows = graphqlSpanAnnotations(spanRows); - const shouldFetchRestAnnotations = - projectIdentifier && - graphqlSessionAnnotationRows.length === 0 && - graphqlTraceAnnotationRows.length === 0 && - graphqlSpanAnnotationRows.length === 0; - - const [sessionAnnotations, traceAnnotations, spanAnnotations] = shouldFetchRestAnnotations - ? await Promise.all([ - fetchAnnotations( - config, - projectIdentifier, - 'session', - [session.sessionId, session.id].filter((id): id is string => !!id), - fetchImpl, - ), - fetchAnnotations(config, projectIdentifier, 'trace', traceIds, fetchImpl), - fetchAnnotations( - config, - projectIdentifier, - 'span', - spanRows.map(graphqlSpanId).filter((id): id is string => !!id), - fetchImpl, - ), - ]) - : [graphqlSessionAnnotationRows, graphqlTraceAnnotationRows, graphqlSpanAnnotationRows]; - - const spanAnnotationsById = annotationsByTarget(spanAnnotations, 'span'); - const spans = spanRows - .map((span) => normalizeSpan(span, spanAnnotationsById)) - .filter((span): span is PhoenixSpanDetail => span !== undefined); - const spansByTraceId = new Map(); - for (const span of spans) { - if (!span.trace_id) { - continue; - } - const traceSpans = spansByTraceId.get(span.trace_id) ?? []; - traceSpans.push(span); - spansByTraceId.set(span.trace_id, traceSpans); - } - const traceAnnotationsById = annotationsByTarget(traceAnnotations, 'trace'); - const turns = session.traces.map((trace, index) => { - const traceSpans = trace.traceId ? (spansByTraceId.get(trace.traceId) ?? []) : []; - const rootSpan = traceSpans.find((span) => !span.parent_span_id) ?? traceSpans[0]; - const startTime = trace.startTime ?? rootSpan?.start_time; - const endTime = trace.endTime ?? rootSpan?.end_time; - const annotations = trace.traceId ? traceAnnotationsById.get(trace.traceId) : undefined; - return compactRecord({ - index: index + 1, - trace_id: trace.traceId, - start_time: startTime, - end_time: endTime, - duration_ms: durationMs(startTime, endTime) ?? rootSpan?.duration_ms, - status: rootSpan?.status, - root_span_id: rootSpan?.span_id, - input: rootSpan?.input, - output: rootSpan?.output, - token_usage: sumTokenUsage(traceSpans), - cost_usd: traceSpans.reduce((sum, span) => sum + (span.cost_usd ?? 0), 0), - ...(annotations && annotations.length > 0 ? { annotations } : {}), - }) as PhoenixSessionTurn; - }); - - const summary: PhoenixSessionSummary = { - id: session.id, - session_id: session.sessionId, - project_id: projectIdentifier, - project: externalTrace.project, - start_time: session.startTime, - end_time: session.endTime, - duration_ms: durationMs(session.startTime, session.endTime), - trace_count: finiteNumber(session.raw?.numTraces) ?? traceIds.length, - token_usage: tokenUsageFromGraphql(session.raw?.tokenUsage) ?? sumTokenUsage(turns), - cost_usd: - costFromGraphqlSummary(session.raw?.costSummary) ?? - turns.reduce((sum, turn) => sum + (turn.cost_usd ?? 0), 0), - ...(sessionAnnotations.length > 0 ? { annotations: sessionAnnotations } : {}), - }; - const annotations = [...sessionAnnotations, ...traceAnnotations, ...spanAnnotations]; - - return response('ok', 'Linked Phoenix session loaded.', externalTraceWire, { - open_in_phoenix_url: buildOpenInPhoenixUrl(config, externalTrace, session), - session: summary, - turns, - spans, - trace_tree: buildSpanTree(spans), - ...(annotations.length > 0 ? { annotations } : {}), - }); -} - -async function resolveSession( - externalTrace: ExternalTraceMetadata, - config: PhoenixConfig, - fetchImpl: typeof fetch, -): Promise { - const projectIdentifier = externalTrace.projectId ?? externalTrace.project ?? config.project; - - if (externalTrace.sessionNodeId) { - try { - const session = await fetchSessionByNodeId(config, externalTrace.sessionNodeId, fetchImpl); - if (session) { - return session; - } - } catch (error) { - if (!(error instanceof PhoenixReadError && error.code === 'schema_mismatch')) { - throw error; - } - // REST is only a compatibility fallback when the configured Phoenix - // GraphQL schema lacks the web-equivalent fields AgentV needs. - } - return fetchSessionByIdRest(config, externalTrace.sessionNodeId, fetchImpl); - } - - if (externalTrace.sessionId) { - if (projectIdentifier) { - try { - const session = await fetchSessionByIdGraphql( - config, - projectIdentifier, - externalTrace.sessionId, - fetchImpl, - ); - if (session) { - return session; - } - } catch (error) { - if (!(error instanceof PhoenixReadError && error.code === 'schema_mismatch')) { - throw error; - } - // Project-scoped GraphQL lookup requires a Phoenix GraphQL project ID. - // REST remains a narrow fallback for configured project names or older - // Phoenix schemas without the session read fields used by the web UI. - } - } - return fetchSessionByIdRest(config, externalTrace.sessionId, fetchImpl); - } - - const traceId = externalTrace.traceId; - if (!traceId || !projectIdentifier) { - return undefined; - } - - try { - const session = await fetchSessionByTraceIdGraphql( - config, - projectIdentifier, - traceId, - fetchImpl, - ); - if (session) { - return session; - } - } catch (error) { - if (!(error instanceof PhoenixReadError && error.code === 'schema_mismatch')) { - throw error; - } - // Fallback path for project-name configs or older Phoenix GraphQL schemas: - // use the public spans API only to recover the session ID from trace attrs. - } - - const spans = await fetchTraceSpans(config, projectIdentifier, traceId, fetchImpl); - const resolvedSessionId = sessionIdFromSpans(spans); - if (!resolvedSessionId) { - return undefined; - } - return fetchSessionByIdRest(config, resolvedSessionId, fetchImpl); -} - -export async function readPhoenixLinkedSession( - externalTrace: ExternalTraceMetadata | undefined, - externalTraceWire: ExternalTraceMetadataWire | undefined, - options: ReadPhoenixLinkedSessionOptions = {}, -): Promise { - if (!externalTrace || !externalTraceWire) { - return response( - 'missing_external_trace', - 'This AgentV run does not include external_trace metadata.', - undefined, - ); - } - - const env = options.env ?? process.env; - const fetchImpl = options.fetch ?? fetch; - const config = resolvePhoenixConfig(externalTrace, env); - if (!config) { - return response( - 'not_configured', - 'Phoenix read-through is not configured. Set AGENTV_PHOENIX_ENDPOINT or PHOENIX_HOST on the Dashboard server.', - externalTraceWire, - { open_in_phoenix_url: externalTrace.uiUrl }, - ); - } - - try { - const session = await resolveSession(externalTrace, config, fetchImpl); - if (!session) { - return response( - 'unresolved', - 'No Phoenix session matched the external_trace correlation metadata.', - externalTraceWire, - { open_in_phoenix_url: buildOpenInPhoenixUrl(config, externalTrace, undefined) }, - ); - } - return buildLinkedSessionResponse(externalTrace, externalTraceWire, config, session, fetchImpl); - } catch (error) { - if (error instanceof PhoenixReadError) { - return response(error.code, error.message, externalTraceWire, { - open_in_phoenix_url: buildOpenInPhoenixUrl(config, externalTrace, undefined), - }); - } - return response( - 'unreachable', - `Phoenix read-through failed: ${error instanceof Error ? error.message : String(error)}`, - externalTraceWire, - { open_in_phoenix_url: buildOpenInPhoenixUrl(config, externalTrace, undefined) }, - ); - } -} diff --git a/apps/cli/src/commands/results/serve.ts b/apps/cli/src/commands/results/serve.ts index 72674e8ac..bb5409107 100644 --- a/apps/cli/src/commands/results/serve.ts +++ b/apps/cli/src/commands/results/serve.ts @@ -10,7 +10,6 @@ * - GET /api/runs — list available run workspaces with metadata * - GET /api/runs/:filename — load results from a specific run workspace * - GET /api/runs/:filename/log — stream the captured console.log for a run - * - GET /api/runs/:filename/phoenix-session — read linked Phoenix session data * - GET /api/runs/:filename/evals/:evalId/files/* — read artifact files as JSON, * or as raw/downloadable text with ?raw=1 / ?download=1 * - GET /api/runs/:filename/evals/:evalId/trace-session — read an AgentV @@ -64,6 +63,7 @@ import { loadConfig, loadProjectRegistry, normalizeTraceArtifactToTraceSessionResponse, + omitExternalTraceMetadataKeys, readGitResultArtifact, removeProject, syncProjects, @@ -92,7 +92,6 @@ import { loadManifestResults, parseResultManifest, } from './manifest.js'; -import { readPhoenixLinkedSession } from './phoenix-readthrough.js'; import { type SourcedResultFileMeta, clearRemoteRunTags, @@ -663,12 +662,19 @@ function parseTraceArtifactJson(content: string): { value?: unknown; message?: s function stripHeavyFields(results: readonly EvaluationResult[]) { return results.map((r) => { - const { requests, trace, ...rest } = r as EvaluationResult & Record; + const { + requests, + trace, + metadata: rawMetadata, + ...rest + } = r as EvaluationResult & Record; + const metadata = omitExternalTraceMetadataKeys(rawMetadata as Record); const toolCalls = trace?.toolCalls && Object.keys(trace.toolCalls).length > 0 ? trace.toolCalls : undefined; const graderDurationMs = (r.scores ?? []).reduce((sum, s) => sum + (s.durationMs ?? 0), 0); return { ...rest, + ...(metadata && { metadata }), ...(toolCalls && { _toolCalls: toolCalls }), ...(graderDurationMs > 0 && { _graderDurationMs: graderDurationMs }), }; @@ -1050,6 +1056,7 @@ async function handleRunDetail(c: C, { searchDir, projectId }: DataContext) { if (!meta) return c.json({ error: 'Run not found' }, 404); try { const loaded = await loadManifestResultsForMeta(searchDir, meta, projectId); + const records = await parseManifestForMeta(searchDir, meta, projectId); // Surface run_dir + suite_filter for local runs so the UI can launch a // Dashboard-side resume against this exact run. Remote runs live in the // results-repo cache and cannot be resumed in place, so omit both fields. @@ -1057,7 +1064,7 @@ async function handleRunDetail(c: C, { searchDir, projectId }: DataContext) { const liveStatus = meta.source === 'local' ? getActiveRunStatus(meta.path) : undefined; const tagFields = await readRunTagFields(searchDir, meta, projectId); return c.json({ - results: stripHeavyFields(loaded), + results: attachExternalTraceFields(stripHeavyFields(loaded), records), source: meta.source, source_label: meta.displayName, ...tagFields, @@ -1069,67 +1076,38 @@ async function handleRunDetail(c: C, { searchDir, projectId }: DataContext) { } } -interface ExternalTraceSelection { - readonly metadata: ExternalTraceMetadata; - readonly wire: ExternalTraceMetadataWire; -} - function isPhoenixExternalTrace(metadata: ExternalTraceMetadata): boolean { const provider = metadata.provider?.toLowerCase(); return provider === undefined || provider === 'phoenix'; } -function externalTraceFromManifestRecord( - record: ResultManifestRecord, -): ExternalTraceSelection | undefined { - const topLevel = record.external_trace - ? externalTraceMetadataFromRecord({ external_trace: record.external_trace }) - : undefined; - const fromMetadata = externalTraceMetadataFromRecord(record.metadata); - const metadata = topLevel ?? fromMetadata; +function externalTraceWireFromManifestRecord( + record: ResultManifestRecord | undefined, +): ExternalTraceMetadataWire | undefined { + if (!record) { + return undefined; + } + const metadata = + externalTraceMetadataFromRecord(record as unknown as Record) ?? + externalTraceMetadataFromRecord(record.metadata); if (!metadata || !isPhoenixExternalTrace(metadata)) { return undefined; } try { - return { - metadata, - wire: toExternalTraceMetadataWire(metadata), - }; + return toExternalTraceMetadataWire(metadata); } catch { return undefined; } } -function findExternalTrace( +function attachExternalTraceFields>( + results: readonly T[], records: readonly ResultManifestRecord[], -): ExternalTraceSelection | undefined { - for (const record of records) { - const externalTrace = externalTraceFromManifestRecord(record); - if (externalTrace) { - return externalTrace; - } - } - return undefined; -} - -async function handlePhoenixSession(c: C, { searchDir, projectId }: DataContext) { - const filename = c.req.param('filename') ?? ''; - const meta = await findRunById(searchDir, filename, projectId); - if (!meta) return c.json({ error: 'Run not found' }, 404); - - try { - const records = await parseManifestForMeta(searchDir, meta, projectId); - const externalTrace = findExternalTrace(records); - return c.json(await readPhoenixLinkedSession(externalTrace?.metadata, externalTrace?.wire)); - } catch (error) { - return c.json({ - schema_version: 'agentv.dashboard.phoenix_session.v1', - status: 'schema_mismatch', - message: `Failed to inspect AgentV run metadata: ${ - error instanceof Error ? error.message : String(error) - }`, - }); - } +): T[] { + return results.map((result, index) => { + const externalTrace = externalTraceWireFromManifestRecord(records[index]); + return externalTrace ? { ...result, external_trace: externalTrace } : result; + }); } /** @@ -2606,7 +2584,6 @@ export function createApp( return handleRunDelete(c, defaultCtx); }); app.get('/api/runs/:filename', (c) => handleRunDetail(c, defaultCtx)); - app.get('/api/runs/:filename/phoenix-session', (c) => handlePhoenixSession(c, defaultCtx)); app.get('/api/runs/:filename/log', (c) => handleRunLog(c, defaultCtx)); app.get('/api/runs/:filename/suites', (c) => handleRunSuites(c, defaultCtx)); app.get('/api/runs/:filename/categories', (c) => handleRunCategories(c, defaultCtx)); @@ -2727,9 +2704,6 @@ export function createApp( return withProject(c, handleRunDelete); }); app.get('/api/projects/:projectId/runs/:filename', (c) => withProject(c, handleRunDetail)); - app.get('/api/projects/:projectId/runs/:filename/phoenix-session', (c) => - withProject(c, handlePhoenixSession), - ); app.get('/api/projects/:projectId/runs/:filename/log', (c) => withProject(c, handleRunLog)); app.get('/api/projects/:projectId/runs/:filename/suites', (c) => withProject(c, handleRunSuites)); app.get('/api/projects/:projectId/runs/:filename/categories', (c) => diff --git a/apps/cli/test/commands/results/serve.test.ts b/apps/cli/test/commands/results/serve.test.ts index fc4c65940..95702419b 100644 --- a/apps/cli/test/commands/results/serve.test.ts +++ b/apps/cli/test/commands/results/serve.test.ts @@ -517,40 +517,15 @@ function createMockStudioDir(baseDir: string): string { // ── Hono app (Dashboard SPA + API) ───────────────────────────────────────── describe('serve app', () => { - const PHOENIX_ENV_KEYS = [ - 'AGENTV_PHOENIX_ENDPOINT', - 'AGENTV_PHOENIX_API_KEY', - 'AGENTV_PHOENIX_AUTHORIZATION', - 'AGENTV_PHOENIX_HEADERS', - 'AGENTV_PHOENIX_GRAPHQL_URL', - 'PHOENIX_HOST', - 'PHOENIX_API_KEY', - 'PHOENIX_CLIENT_HEADERS', - ] as const; let tempDir: string; let studioDir: string; - let originalFetch: typeof fetch; - let originalPhoenixEnv: Record<(typeof PHOENIX_ENV_KEYS)[number], string | undefined>; beforeEach(() => { tempDir = mkdtempSync(path.join(tmpdir(), 'agentv-serve-test-')); studioDir = createMockStudioDir(tempDir); - originalFetch = globalThis.fetch; - originalPhoenixEnv = Object.fromEntries( - PHOENIX_ENV_KEYS.map((key) => [key, process.env[key]]), - ) as Record<(typeof PHOENIX_ENV_KEYS)[number], string | undefined>; }); afterEach(() => { - globalThis.fetch = originalFetch; - for (const key of PHOENIX_ENV_KEYS) { - const value = originalPhoenixEnv[key]; - if (value === undefined) { - delete process.env[key]; - } else { - process.env[key] = value; - } - } rmSync(tempDir, { recursive: true, force: true }); }); @@ -1055,381 +1030,65 @@ describe('serve app', () => { }); }); - it('reads a linked Phoenix session through narrow server-side GraphQL/API requests', async () => { + it('exposes sanitized Phoenix external_trace metadata through run detail only', async () => { const filename = '2026-03-25T10-15-00-000Z'; - createLocalRun(tempDir, filename, { - ...RESULT_A, - external_trace: { - provider: 'phoenix', - source: 'codex', - endpoint: 'https://phoenix.example/v1/traces?api_key=artifact-secret', - project: 'agentv-dogfood', - session_node_id: 'UHJvamVjdFNlc3Npb246MQ==', - ui_url: 'https://phoenix.example/sessions/codex-session-1?token=artifact-secret', + createLocalRun( + tempDir, + filename, + { + ...RESULT_A, + external_trace: { + provider: 'phoenix', + source: 'codex', + endpoint: 'https://phoenix.example/v1/traces?api_key=artifact-secret', + project: 'agentv-dogfood', + session_id: 'codex-session-1', + ui_url: 'https://phoenix.example/sessions/codex-session-1?token=artifact-secret', + }, }, - }); - process.env.AGENTV_PHOENIX_ENDPOINT = 'https://phoenix.example'; - process.env.AGENTV_PHOENIX_API_KEY = 'server-secret'; - - const requests: Request[] = []; - globalThis.fetch = (async (input: RequestInfo | URL, init?: RequestInit) => { - const request = new Request(input, init); - requests.push(request); - const url = new URL(request.url); - if (url.pathname === '/graphql') { - const body = (await request.json()) as { - query: string; - variables: Record; - }; - expect(body.query).toContain('AgentVPhoenixLinkedSessionByNode'); - expect(body.query).toContain('AgentVPhoenixSessionFields'); - expect(body.query).toContain('AgentVPhoenixTraceFields'); - expect(body.query).toContain('AgentVPhoenixSpanFields'); - expect(body.variables.id).toBe('UHJvamVjdFNlc3Npb246MQ=='); - expect(request.headers.get('authorization')).toBe('Bearer server-secret'); - return Response.json({ - data: { - node: { - __typename: 'ProjectSession', - id: 'UHJvamVjdFNlc3Npb246MQ==', - sessionId: 'codex-session-1', - project: { id: 'project-1', name: 'agentv-dogfood' }, - numTraces: 1, - tokenUsage: { prompt: 12, completion: 8, total: 20 }, - costSummary: { - prompt: { cost: 0.01, tokens: 12 }, - completion: { cost: 0.01, tokens: 8 }, - total: { cost: 0.02, tokens: 20 }, - }, - startTime: '2026-03-25T10:00:00.000Z', - endTime: '2026-03-25T10:00:04.000Z', - sessionAnnotations: [ - { - id: 'ann-session', - projectSessionId: 'codex-session-1', - name: 'review', - annotatorKind: 'HUMAN', - label: 'pass', - score: 1, - explanation: 'Looks good', - metadata: { cookie: 'annotation-cookie-secret' }, - identifier: 'session-review', - createdAt: '2026-03-25T10:00:05.000Z', - updatedAt: '2026-03-25T10:00:05.000Z', - }, - ], - traces: { - edges: [ - { - node: { - id: 'trace-node-1', - traceId: 'trace-1', - startTime: '2026-03-25T10:00:00.000Z', - endTime: '2026-03-25T10:00:04.000Z', - latencyMs: 4000, - projectId: 'project-1', - project: { id: 'project-1', name: 'agentv-dogfood' }, - projectSessionId: 'UHJvamVjdFNlc3Npb246MQ==', - costSummary: { - total: { cost: 0.02, tokens: 20 }, - }, - traceAnnotations: [], - rootSpan: { - id: 'span-node-root', - spanId: 'span-root', - parentId: null, - name: 'agent turn', - spanKind: 'agent', - statusCode: 'OK', - startTime: '2026-03-25T10:00:00.000Z', - endTime: '2026-03-25T10:00:04.000Z', - latencyMs: 4000, - input: { - value: 'summarize the repo', - truncatedValue: 'summarize the repo', - mimeType: 'text/plain', - }, - output: { - value: 'repo summary', - truncatedValue: 'repo summary', - mimeType: 'text/plain', - }, - attributes: JSON.stringify({ - 'input.value': 'summarize the repo', - 'output.value': 'repo summary', - authorization: 'span-authorization-secret', - nested: { api_key: 'span-api-key-secret', visible: 'safe' }, - }), - tokenCountTotal: 20, - tokenCountPrompt: 12, - tokenCountCompletion: 8, - cumulativeTokenCountTotal: 20, - cumulativeTokenCountPrompt: 12, - cumulativeTokenCountCompletion: 8, - costSummary: { - total: { cost: 0.02, tokens: 20 }, - }, - spanAnnotations: [ - { - id: 'ann-span', - spanId: 'span-root', - name: 'latency', - annotatorKind: 'CODE', - label: 'ok', - score: null, - explanation: null, - metadata: { - headers: { authorization: 'span-annotation-secret' }, - visible: 'safe', - }, - identifier: 'span-latency', - createdAt: '2026-03-25T10:00:05.000Z', - updatedAt: '2026-03-25T10:00:05.000Z', - }, - ], - trace: { - id: 'trace-node-1', - traceId: 'trace-1', - costSummary: { total: { cost: 0.02, tokens: 20 } }, - }, - project: { id: 'project-1', name: 'agentv-dogfood' }, - }, - spans: { - edges: [ - { - node: { - id: 'span-node-root', - spanId: 'span-root', - parentId: null, - name: 'agent turn', - spanKind: 'agent', - statusCode: 'OK', - startTime: '2026-03-25T10:00:00.000Z', - endTime: '2026-03-25T10:00:04.000Z', - latencyMs: 4000, - input: { - value: 'summarize the repo', - truncatedValue: 'summarize the repo', - mimeType: 'text/plain', - }, - output: { - value: 'repo summary', - truncatedValue: 'repo summary', - mimeType: 'text/plain', - }, - attributes: '{}', - tokenCountTotal: 20, - tokenCountPrompt: 12, - tokenCountCompletion: 8, - cumulativeTokenCountTotal: 20, - cumulativeTokenCountPrompt: 12, - cumulativeTokenCountCompletion: 8, - costSummary: { total: { cost: 0.02, tokens: 20 } }, - spanAnnotations: [ - { - id: 'ann-span', - spanId: 'span-root', - name: 'latency', - annotatorKind: 'CODE', - label: 'ok', - score: null, - explanation: null, - metadata: {}, - identifier: 'span-latency', - createdAt: '2026-03-25T10:00:05.000Z', - updatedAt: '2026-03-25T10:00:05.000Z', - }, - ], - trace: { - id: 'trace-node-1', - traceId: 'trace-1', - costSummary: { total: { cost: 0.02, tokens: 20 } }, - }, - project: { id: 'project-1', name: 'agentv-dogfood' }, - }, - }, - { - node: { - id: 'span-node-child', - spanId: 'span-child', - parentId: 'span-root', - name: 'tool call', - spanKind: 'tool', - statusCode: 'OK', - startTime: '2026-03-25T10:00:01.000Z', - endTime: '2026-03-25T10:00:02.000Z', - latencyMs: 1000, - input: null, - output: null, - attributes: JSON.stringify({ 'openinference.span.kind': 'TOOL' }), - tokenCountTotal: null, - tokenCountPrompt: null, - tokenCountCompletion: null, - cumulativeTokenCountTotal: null, - cumulativeTokenCountPrompt: null, - cumulativeTokenCountCompletion: null, - costSummary: null, - spanAnnotations: [], - trace: { - id: 'trace-node-1', - traceId: 'trace-1', - costSummary: { total: { cost: 0.02, tokens: 20 } }, - }, - project: { id: 'project-1', name: 'agentv-dogfood' }, - }, - }, - ], - }, - }, - }, - ], - }, - }, + { + ...RESULT_B, + metadata: { + external_trace: { + provider: 'phoenix', + endpoint: 'https://phoenix.example/v1/traces?api_key=artifact-secret', + session_id: 'codex-session-2', + ui_url: 'https://phoenix.example/sessions/codex-session-2?token=artifact-secret', }, - }); - } - return Response.json({ error: 'unexpected request' }, { status: 500 }); - }) as typeof fetch; - - const app = createApp([], tempDir, tempDir, undefined, { studioDir }); - const res = await app.request(`/api/runs/${encodeURIComponent(filename)}/phoenix-session`); - expect(res.status).toBe(200); - const data = (await res.json()) as { - status: string; - external_trace: { endpoint: string; ui_url: string }; - session: { session_id: string; trace_count: number; token_usage?: { input?: number } }; - turns: Array<{ root_span_id?: string; input?: string; output?: string; cost_usd?: number }>; - trace_tree: Array<{ span_id: string; depth: number }>; - annotations: Array<{ name?: string; target?: string }>; - }; - - expect(data.status).toBe('ok'); - expect(data.external_trace).toMatchObject({ - endpoint: 'https://phoenix.example/', - ui_url: 'https://phoenix.example/sessions/codex-session-1', - }); - expect(data.session).toMatchObject({ - session_id: 'codex-session-1', - trace_count: 1, - token_usage: { input: 12 }, - }); - expect(data.turns[0]).toMatchObject({ - root_span_id: 'span-root', - input: 'summarize the repo', - output: 'repo summary', - cost_usd: 0.02, - }); - expect(data.trace_tree.map(({ span_id, depth }) => ({ span_id, depth }))).toEqual([ - { span_id: 'span-root', depth: 0 }, - { span_id: 'span-child', depth: 1 }, - ]); - expect(data.annotations).toEqual( - expect.arrayContaining([ - expect.objectContaining({ name: 'review', target: 'session' }), - expect.objectContaining({ name: 'latency', target: 'span' }), - ]), - ); - expect(requests.map((request) => new URL(request.url).pathname)).toEqual(['/graphql']); - const serialized = JSON.stringify(data); - expect(serialized).not.toContain('server-secret'); - expect(serialized).not.toContain('artifact-secret'); - expect(serialized).not.toContain('annotation-cookie-secret'); - expect(serialized).not.toContain('span-authorization-secret'); - expect(serialized).not.toContain('span-api-key-secret'); - expect(serialized).not.toContain('span-annotation-secret'); - expect(serialized).not.toContain('Authorization'); - expect(serialized).toContain('safe'); - }); - - it('does not use artifact Phoenix endpoints as server network config', async () => { - const filename = '2026-03-25T10-17-00-000Z'; - createLocalRun(tempDir, filename, { - ...RESULT_A, - external_trace: { - provider: 'phoenix', - endpoint: 'https://artifact-controlled.example', - session_node_id: 'UHJvamVjdFNlc3Npb246MQ==', - ui_url: 'https://artifact-controlled.example/sessions/codex-session-1', + }, }, - }); - process.env.AGENTV_PHOENIX_API_KEY = 'server-secret'; - const requests: Request[] = []; - globalThis.fetch = (async (input: RequestInfo | URL, init?: RequestInit) => { - const request = new Request(input, init); - requests.push(request); - return Response.json({ error: 'unexpected request' }, { status: 500 }); - }) as typeof fetch; + ); const app = createApp([], tempDir, tempDir, undefined, { studioDir }); - const res = await app.request(`/api/runs/${encodeURIComponent(filename)}/phoenix-session`); - expect(res.status).toBe(200); - const data = (await res.json()) as { - status: string; - message: string; - open_in_phoenix_url?: string; - }; - expect(data.status).toBe('not_configured'); - expect(data.message).toContain('AGENTV_PHOENIX_ENDPOINT'); - expect(data.open_in_phoenix_url).toBe( - 'https://artifact-controlled.example/sessions/codex-session-1', - ); - expect(requests).toHaveLength(0); - }); + const detailRes = await app.request(`/api/runs/${encodeURIComponent(filename)}`); - it('reports missing Phoenix configuration without contacting Phoenix', async () => { - const filename = '2026-03-25T10-20-00-000Z'; - createLocalRun(tempDir, filename, { - ...RESULT_A, + expect(detailRes.status).toBe(200); + const detailData = (await detailRes.json()) as { + results: Array<{ + testId: string; + external_trace?: { endpoint?: string; ui_url?: string; session_id?: string }; + }>; + }; + expect(detailData.results[0]).toMatchObject({ + testId: 'test-greeting', external_trace: { - provider: 'phoenix', + endpoint: 'https://phoenix.example/', + ui_url: 'https://phoenix.example/sessions/codex-session-1', session_id: 'codex-session-1', }, }); - const requests: Request[] = []; - globalThis.fetch = (async (input: RequestInfo | URL, init?: RequestInit) => { - requests.push(new Request(input, init)); - return Response.json({ error: 'unexpected request' }, { status: 500 }); - }) as typeof fetch; - - const app = createApp([], tempDir, tempDir, undefined, { studioDir }); - const res = await app.request(`/api/runs/${encodeURIComponent(filename)}/phoenix-session`); - expect(res.status).toBe(200); - const data = (await res.json()) as { status: string; message: string }; - - expect(data.status).toBe('not_configured'); - expect(data.message).toContain('AGENTV_PHOENIX_ENDPOINT'); - expect(requests).toEqual([]); - }); - - it('keeps run detail readable when Phoenix is unreachable', async () => { - const filename = '2026-03-25T10-25-00-000Z'; - createLocalRun(tempDir, filename, { - ...RESULT_A, + expect(detailData.results[1]).toMatchObject({ + testId: 'test-math', external_trace: { - provider: 'phoenix', - endpoint: 'https://phoenix.example', - trace_id: 'trace-1', - project: 'agentv-dogfood', + endpoint: 'https://phoenix.example/', + ui_url: 'https://phoenix.example/sessions/codex-session-2', + session_id: 'codex-session-2', }, }); - process.env.AGENTV_PHOENIX_ENDPOINT = 'https://phoenix.example'; - globalThis.fetch = (async () => { - throw new Error('connect ECONNREFUSED'); - }) as typeof fetch; - - const app = createApp([], tempDir, tempDir, undefined, { studioDir }); - const phoenixRes = await app.request( - `/api/runs/${encodeURIComponent(filename)}/phoenix-session`, - ); - expect(phoenixRes.status).toBe(200); - const phoenixData = (await phoenixRes.json()) as { status: string; message: string }; - expect(phoenixData.status).toBe('unreachable'); - expect(phoenixData.message).toContain('connect ECONNREFUSED'); - const detailRes = await app.request(`/api/runs/${encodeURIComponent(filename)}`); - expect(detailRes.status).toBe(200); - const detailData = (await detailRes.json()) as { results: Array<{ testId: string }> }; - expect(detailData.results[0].testId).toBe('test-greeting'); + const serialized = JSON.stringify(detailData); + expect(serialized).not.toContain('artifact-secret'); + expect(serialized).not.toContain('token='); }); it('exposes materialized final state and oplog watermark for local run tags', async () => { diff --git a/apps/dashboard/src/components/PhoenixLinkedSessionPanel.test.tsx b/apps/dashboard/src/components/PhoenixLinkedSessionPanel.test.tsx deleted file mode 100644 index 8cb84b66f..000000000 --- a/apps/dashboard/src/components/PhoenixLinkedSessionPanel.test.tsx +++ /dev/null @@ -1,102 +0,0 @@ -import { describe, expect, it } from 'bun:test'; -import { renderToStaticMarkup } from 'react-dom/server'; - -import type { PhoenixLinkedSessionResponse } from '~/lib/types'; - -import { PhoenixLinkedSessionPanelContent } from './PhoenixLinkedSessionPanel'; - -describe('PhoenixLinkedSessionPanelContent', () => { - it('renders linked session summary, turns, trace tree, and selected span detail', () => { - const response: PhoenixLinkedSessionResponse = { - schema_version: 'agentv.dashboard.phoenix_session.v1', - status: 'ok', - open_in_phoenix_url: 'https://phoenix.example/sessions/codex-session-1', - external_trace: { - provider: 'phoenix', - session_id: 'codex-session-1', - }, - session: { - session_id: 'codex-session-1', - project_id: 'project-1', - trace_count: 1, - duration_ms: 4200, - token_usage: { input: 12, output: 8, total: 20 }, - cost_usd: 0.02, - start_time: '2026-03-25T10:00:00.000Z', - }, - turns: [ - { - index: 1, - trace_id: 'trace-1', - duration_ms: 4200, - root_span_id: 'span-root', - input: 'summarize the repo', - output: 'repo summary', - }, - ], - spans: [ - { - span_id: 'span-root', - trace_id: 'trace-1', - name: 'agent turn', - status: 'OK', - input: 'summarize the repo', - output: 'repo summary', - token_usage: { total: 20 }, - attributes: { 'openinference.span.kind': 'LLM' }, - }, - ], - trace_tree: [ - { - span_id: 'span-root', - trace_id: 'trace-1', - name: 'agent turn', - depth: 0, - child_count: 1, - duration_ms: 4200, - }, - { - span_id: 'span-child', - trace_id: 'trace-1', - parent_span_id: 'span-root', - name: 'tool call', - depth: 1, - child_count: 0, - duration_ms: 900, - }, - ], - annotations: [{ name: 'review', target: 'session', label: 'pass' }], - }; - - const html = renderToStaticMarkup( - , - ); - - expect(html).toContain('Phoenix Session'); - expect(html).toContain('codex-session-1'); - expect(html).toContain('Open in Phoenix'); - expect(html).toContain('summarize the repo'); - expect(html).toContain('repo summary'); - expect(html).toContain('agent turn'); - expect(html).toContain('tool call'); - expect(html).toContain('openinference.span.kind'); - }); - - it('renders recoverable unresolved state', () => { - const html = renderToStaticMarkup( - , - ); - - expect(html).toContain('Phoenix Session'); - expect(html).toContain('unresolved'); - expect(html).toContain('No Phoenix session matched'); - expect(html).toContain('Open in Phoenix'); - }); -}); diff --git a/apps/dashboard/src/components/PhoenixLinkedSessionPanel.tsx b/apps/dashboard/src/components/PhoenixLinkedSessionPanel.tsx deleted file mode 100644 index 32ac3123e..000000000 --- a/apps/dashboard/src/components/PhoenixLinkedSessionPanel.tsx +++ /dev/null @@ -1,322 +0,0 @@ -import { useState } from 'react'; - -import { usePhoenixLinkedSession } from '~/lib/api'; -import type { - PhoenixLinkedSessionResponse, - PhoenixLinkedSessionSpan, - PhoenixLinkedSessionTokenUsage, - PhoenixLinkedSessionTraceNode, -} from '~/lib/types'; - -interface PhoenixLinkedSessionPanelProps { - runId: string; - projectId?: string; -} - -interface PhoenixLinkedSessionPanelContentProps { - response: PhoenixLinkedSessionResponse; - initialSpanId?: string; -} - -function formatDuration(ms: number | undefined): string { - if (ms === undefined) return 'n/a'; - if (ms < 1000) return `${Math.round(ms)} ms`; - const seconds = ms / 1000; - if (seconds < 60) return `${seconds.toFixed(seconds >= 10 ? 0 : 1)} s`; - const minutes = Math.floor(seconds / 60); - const remainder = Math.round(seconds % 60); - return `${minutes}m ${remainder}s`; -} - -function formatCost(value: number | undefined): string { - if (value === undefined || value <= 0) return 'n/a'; - return `$${value.toFixed(value < 0.01 ? 4 : 2)}`; -} - -function formatTokenUsage(usage: PhoenixLinkedSessionTokenUsage | undefined): string { - if (!usage) return 'n/a'; - const total = - usage.total ?? - (usage.input ?? 0) + (usage.output ?? 0) + (usage.reasoning ?? 0) + (usage.cached ?? 0); - return total > 0 ? total.toLocaleString() : 'n/a'; -} - -function formatDateTime(value: string | undefined): string { - if (!value) return 'n/a'; - const date = new Date(value); - return Number.isNaN(date.getTime()) ? value : date.toLocaleString(); -} - -function valuePreview(value: unknown): string { - if (value === undefined || value === null) return ''; - if (typeof value === 'string') return value; - try { - return JSON.stringify(value, null, 2); - } catch { - return String(value); - } -} - -function compactPreview(value: unknown): string { - return valuePreview(value).replace(/\s+/g, ' ').trim(); -} - -function statusClass(status: PhoenixLinkedSessionResponse['status']): string { - if (status === 'ok') return 'border-emerald-800 bg-emerald-950/40 text-emerald-300'; - if (status === 'not_configured' || status === 'unresolved') { - return 'border-amber-800 bg-amber-950/40 text-amber-300'; - } - return 'border-red-800 bg-red-950/40 text-red-300'; -} - -function Metric({ label, value }: { label: string; value: string }) { - return ( -
-
{label}
-
- {value} -
-
- ); -} - -function OpenInPhoenixLink({ href }: { href?: string }) { - if (!href) return null; - return ( - - Open in Phoenix - - ); -} - -function ErrorState({ response }: { response: PhoenixLinkedSessionResponse }) { - return ( -
-
-
-

Phoenix Session

-

{response.message}

-
-
- - {response.status} - - -
-
-
- ); -} - -function SpanTree({ - nodes, - selectedSpanId, - onSelect, -}: { - nodes: readonly PhoenixLinkedSessionTraceNode[]; - selectedSpanId?: string; - onSelect: (spanId: string) => void; -}) { - if (nodes.length === 0) { - return
No span tree returned.
; - } - - return ( -
- {nodes.map((node) => { - const active = node.span_id === selectedSpanId; - return ( - - ); - })} -
- ); -} - -function SpanDetail({ span }: { span?: PhoenixLinkedSessionSpan }) { - if (!span) { - return
No span selected.
; - } - - return ( -
-
- - - -
-
-
-
Input
-
-            {valuePreview(span.input)}
-          
-
-
-
Output
-
-            {valuePreview(span.output)}
-          
-
-
- {span.attributes ? ( -
-
Attributes
-
-            {valuePreview(span.attributes)}
-          
-
- ) : null} -
- ); -} - -export function PhoenixLinkedSessionPanelContent({ - response, - initialSpanId, -}: PhoenixLinkedSessionPanelContentProps) { - const tree = response.trace_tree ?? []; - const spans = response.spans ?? []; - const [selectedSpanId, setSelectedSpanId] = useState( - initialSpanId ?? tree[0]?.span_id ?? spans[0]?.span_id, - ); - - if (response.status === 'missing_external_trace') { - return null; - } - - if (response.status !== 'ok') { - return ; - } - - const selectedSpan = - spans.find((span) => span.span_id === selectedSpanId) ?? - tree.find((span) => span.span_id === selectedSpanId) ?? - spans[0] ?? - tree[0]; - const session = response.session; - - return ( -
-
-
-
-

Phoenix Session

- - linked - -
-

- {session?.session_id ?? response.external_trace?.session_id ?? 'session'} -

-
- -
- -
- - - - - -
- -
-
-
-

Turns

-
- {(response.turns ?? []).map((turn) => ( -
- #{turn.index} - - - {compactPreview(turn.input) || turn.trace_id || 'trace'} - - - {compactPreview(turn.output)} - - - - {formatDuration(turn.duration_ms)} - -
- ))} -
-
- -
-

Trace Tree

- -
-
- -
-

Span Details

- -
-
-
- ); -} - -export function PhoenixLinkedSessionPanel({ runId, projectId }: PhoenixLinkedSessionPanelProps) { - const { data, isLoading, error } = usePhoenixLinkedSession(runId, projectId); - - if (data?.status === 'missing_external_trace') { - return null; - } - - if (isLoading) { - return ( -
- Loading Phoenix session... -
- ); - } - - if (error) { - return ( - - ); - } - - return data ? : null; -} diff --git a/apps/dashboard/src/components/RunDetail.tsx b/apps/dashboard/src/components/RunDetail.tsx index 08693f36a..1f20c2950 100644 --- a/apps/dashboard/src/components/RunDetail.tsx +++ b/apps/dashboard/src/components/RunDetail.tsx @@ -22,11 +22,11 @@ import { Link } from '@tanstack/react-router'; import type { EvalResult } from '~/lib/types'; import { useRunLog, useStudioConfig } from '~/lib/api'; +import { findPhoenixExternalTraceUrl } from '~/lib/external-trace-link'; import { summarizeQuality } from '~/lib/result-summary'; import { formatCategoryDisplay } from '~/lib/run-detail-context'; import { PassRatePill } from './PassRatePill'; -import { PhoenixLinkedSessionPanel } from './PhoenixLinkedSessionPanel'; import { ResultTable } from './ResultTable'; import { StatsCards } from './StatsCards'; @@ -114,6 +114,7 @@ function buildCategoryGroups(results: EvalResult[], passThreshold: number): Cate export function RunDetail({ results, runId, projectId }: RunDetailProps) { const { data: config } = useStudioConfig(projectId); const passThreshold = config?.threshold ?? config?.pass_threshold ?? 0.8; + const phoenixUrl = findPhoenixExternalTraceUrl(results); const total = results.length; const summary = summarizeQuality(results, passThreshold); @@ -125,7 +126,7 @@ export function RunDetail({ results, runId, projectId }: RunDetailProps) { return (
- +

No evaluations found

This run has no results yet.

@@ -145,7 +146,7 @@ export function RunDetail({ results, runId, projectId }: RunDetailProps) { totalCost={totalCost > 0 ? totalCost : undefined} /> - + {/* Category Breakdown */}
@@ -249,6 +250,29 @@ export function RunDetail({ results, runId, projectId }: RunDetailProps) { ); } +function ExternalTraceLink({ href }: { href?: string }) { + if (!href) return null; + + return ( +
+
+
+

External trace

+

Phoenix

+
+ + Open in Phoenix + +
+
+ ); +} + function ConsoleLogSection({ runId, projectId }: { runId: string; projectId?: string }) { const [open, setOpen] = useState(false); const { data: log, isLoading, error } = useRunLog(runId, projectId); diff --git a/apps/dashboard/src/lib/api.ts b/apps/dashboard/src/lib/api.ts index 4d5ed06ce..869e1e81d 100644 --- a/apps/dashboard/src/lib/api.ts +++ b/apps/dashboard/src/lib/api.ts @@ -32,7 +32,6 @@ import type { FilesystemBrowseResponse, FilesystemBrowseResponseWire, IndexResponse, - PhoenixLinkedSessionResponse, ProjectEntry, ProjectEntryWire, ProjectListResponse, @@ -153,26 +152,10 @@ export function runLogOptions(filename: string, projectId?: string) { }); } -export function phoenixLinkedSessionOptions(filename: string, projectId?: string) { - const url = projectId - ? `${projectApiBase(projectId)}/runs/${encodeURIComponent(filename)}/phoenix-session` - : `/api/runs/${encodeURIComponent(filename)}/phoenix-session`; - return queryOptions({ - queryKey: ['runs', filename, 'phoenix-session', projectId ?? ''], - queryFn: () => fetchJson(url), - enabled: !!filename, - staleTime: 30_000, - }); -} - export function useRunLog(filename: string, projectId?: string) { return useQuery(runLogOptions(filename, projectId)); } -export function usePhoenixLinkedSession(filename: string, projectId?: string) { - return useQuery(phoenixLinkedSessionOptions(filename, projectId)); -} - export function runSuitesOptions(runId: string) { return queryOptions({ queryKey: ['runs', runId, 'suites'], diff --git a/apps/dashboard/src/lib/external-trace-link.test.ts b/apps/dashboard/src/lib/external-trace-link.test.ts new file mode 100644 index 000000000..d4810413e --- /dev/null +++ b/apps/dashboard/src/lib/external-trace-link.test.ts @@ -0,0 +1,57 @@ +import { describe, expect, it } from 'bun:test'; + +import type { EvalResult } from './types'; + +import { findPhoenixExternalTraceUrl, sanitizeExternalTraceUiUrl } from './external-trace-link'; + +function resultWith(fields: Partial): EvalResult { + return { + testId: 'test-greeting', + score: 1, + output: '', + ...fields, + }; +} + +describe('external trace links', () => { + it('sanitizes ordinary external trace UI URLs before rendering', () => { + expect( + sanitizeExternalTraceUiUrl('https://phoenix.example/sessions/codex-session-1?token=secret#x'), + ).toBe('https://phoenix.example/sessions/codex-session-1'); + expect(sanitizeExternalTraceUiUrl('javascript:alert(1)')).toBeUndefined(); + expect(sanitizeExternalTraceUiUrl('https://user:pass@phoenix.example/session')).toBeUndefined(); + }); + + it('finds the first Phoenix UI URL from run detail results', () => { + const url = findPhoenixExternalTraceUrl([ + resultWith({ + external_trace: { + provider: 'phoenix', + session_id: 'codex-session-1', + ui_url: 'https://phoenix.example/sessions/codex-session-1?token=secret', + }, + }), + ]); + + expect(url).toBe('https://phoenix.example/sessions/codex-session-1'); + }); + + it('supports legacy camelCase and flat metadata without linking other providers', () => { + const results = [ + resultWith({ + externalTrace: { + provider: 'langfuse', + uiUrl: 'https://langfuse.example/traces/trace-1', + }, + }), + resultWith({ + metadata: { + external_trace_provider: 'phoenix', + external_trace_ui_url: 'https://phoenix.example/traces/trace-2', + }, + }), + ]; + + expect(findPhoenixExternalTraceUrl(results)).toBe('https://phoenix.example/traces/trace-2'); + }); +}); diff --git a/apps/dashboard/src/lib/external-trace-link.ts b/apps/dashboard/src/lib/external-trace-link.ts new file mode 100644 index 000000000..62e8f93df --- /dev/null +++ b/apps/dashboard/src/lib/external-trace-link.ts @@ -0,0 +1,74 @@ +import type { EvalResult } from './types'; + +function isRecord(value: unknown): value is Record { + return typeof value === 'object' && value !== null && !Array.isArray(value); +} + +function nonEmptyString(value: unknown): string | undefined { + return typeof value === 'string' && value.trim().length > 0 ? value.trim() : undefined; +} + +export function sanitizeExternalTraceUiUrl(value: unknown): string | undefined { + const raw = nonEmptyString(value); + if (!raw) return undefined; + + try { + const url = new URL(raw); + if (!['http:', 'https:'].includes(url.protocol) || url.username || url.password) { + return undefined; + } + url.search = ''; + url.hash = ''; + return url.toString(); + } catch { + return undefined; + } +} + +function isPhoenixTrace(value: Record): boolean { + const provider = nonEmptyString(value.provider); + return provider === undefined || provider.toLowerCase() === 'phoenix'; +} + +function flatExternalTraceFromMetadata( + metadata: Record | undefined, +): Record | undefined { + if (!metadata) return undefined; + return { + provider: metadata.external_trace_provider ?? metadata['external_trace.provider'], + ui_url: + metadata.external_trace_ui_url ?? + metadata.external_trace_url ?? + metadata['external_trace.ui_url'] ?? + metadata['external_trace.url'], + }; +} + +function externalTraceCandidates(result: EvalResult): unknown[] { + const raw = result as EvalResult & { + readonly externalTrace?: unknown; + readonly external_trace?: unknown; + }; + const metadata = isRecord(result.metadata) ? result.metadata : undefined; + return [ + raw.external_trace, + raw.externalTrace, + metadata?.external_trace, + metadata?.externalTrace, + flatExternalTraceFromMetadata(metadata), + ]; +} + +export function findPhoenixExternalTraceUrl(results: readonly EvalResult[]): string | undefined { + for (const result of results) { + for (const candidate of externalTraceCandidates(result)) { + if (!isRecord(candidate) || !isPhoenixTrace(candidate)) continue; + + const url = sanitizeExternalTraceUiUrl( + candidate.ui_url ?? candidate.uiUrl ?? candidate.url ?? candidate.href, + ); + if (url) return url; + } + } + return undefined; +} diff --git a/apps/dashboard/src/lib/types.ts b/apps/dashboard/src/lib/types.ts index ad69a3549..ff78ed288 100644 --- a/apps/dashboard/src/lib/types.ts +++ b/apps/dashboard/src/lib/types.ts @@ -152,95 +152,25 @@ export interface SourceTraceability { export type ExternalTraceMetadata = CoreExternalTraceMetadata; -export interface PhoenixLinkedSessionTokenUsage { - input?: number; - output?: number; - reasoning?: number; - cached?: number; - total?: number; -} - -export interface PhoenixLinkedSessionAnnotation { - id?: string; - name?: string; - annotator_kind?: string; - label?: string; - score?: number; - explanation?: string; - target?: 'session' | 'trace' | 'span'; - target_id?: string; - result?: unknown; -} - -export interface PhoenixLinkedSessionSpan { - span_id: string; - trace_id?: string; - parent_span_id?: string; - name?: string; - span_kind?: string; - status?: string; - start_time?: string; - end_time?: string; - duration_ms?: number; - input?: unknown; - output?: unknown; - token_usage?: PhoenixLinkedSessionTokenUsage; - cost_usd?: number; - attributes?: Record; - annotations?: PhoenixLinkedSessionAnnotation[]; -} - -export interface PhoenixLinkedSessionTraceNode extends PhoenixLinkedSessionSpan { - depth: number; - child_count: number; -} - -export interface PhoenixLinkedSessionTurn { - index: number; - trace_id?: string; - start_time?: string; - end_time?: string; - duration_ms?: number; - status?: string; - root_span_id?: string; - input?: unknown; - output?: unknown; - token_usage?: PhoenixLinkedSessionTokenUsage; - cost_usd?: number; - annotations?: PhoenixLinkedSessionAnnotation[]; -} - -export interface PhoenixLinkedSessionSummary { - id?: string; - session_id?: string; - project_id?: string; +export interface CamelExternalTraceMetadata { + provider?: string; + source?: string; + endpoint?: string; + profile?: string; project?: string; - start_time?: string; - end_time?: string; - duration_ms?: number; - trace_count: number; - token_usage?: PhoenixLinkedSessionTokenUsage; - cost_usd?: number; - annotations?: PhoenixLinkedSessionAnnotation[]; -} - -export interface PhoenixLinkedSessionResponse { - schema_version: 'agentv.dashboard.phoenix_session.v1'; - status: - | 'ok' - | 'missing_external_trace' - | 'not_configured' - | 'unresolved' - | 'unreachable' - | 'schema_mismatch'; - message?: string; - external_trace?: ExternalTraceMetadata; - open_in_phoenix_url?: string; - session?: PhoenixLinkedSessionSummary; - turns?: PhoenixLinkedSessionTurn[]; - spans?: PhoenixLinkedSessionSpan[]; - trace_tree?: PhoenixLinkedSessionTraceNode[]; - annotations?: PhoenixLinkedSessionAnnotation[]; + projectId?: string; + sessionId?: string; + sessionNodeId?: string; + traceId?: string; + traceNodeId?: string; + spanId?: string; + spanNodeId?: string; + traceparent?: string; + tracestate?: string; + uiUrl?: string; + runId?: string; + testId?: string; + target?: string; } export type TraceSessionTokenUsage = CoreTraceSessionTokenUsage; @@ -276,6 +206,8 @@ export interface EvalResult { output?: string; _toolCalls?: Record; _graderDurationMs?: number; + external_trace?: ExternalTraceMetadata; + externalTrace?: CamelExternalTraceMetadata; metadata?: Record; source_traceability?: SourceTraceability; } diff --git a/apps/web/src/content/docs/docs/integrations/phoenix.mdx b/apps/web/src/content/docs/docs/integrations/phoenix.mdx index da25d0816..3fec46b37 100644 --- a/apps/web/src/content/docs/docs/integrations/phoenix.mdx +++ b/apps/web/src/content/docs/docs/integrations/phoenix.mdx @@ -23,10 +23,9 @@ Phoenix can still appear in AgentV workflows in two narrow ways: already emitted spans independently. When an AgentV run artifact includes safe `external_trace` metadata, AgentV may -link to that external Phoenix session. Dashboard can also read the linked -session through Phoenix GraphQL/API from the AgentV server process and show a -compact session panel. AgentV must not write AgentV artifacts into Phoenix as -part of that flow. +link to that external Phoenix session or trace. Dashboard does not read Phoenix +sessions, traces, or spans through a server-side proxy; it opens Phoenix as the +external viewer when a safe UI URL is present. ## Local Inspection @@ -41,31 +40,9 @@ workspaces, remote results repositories, trace sidecars, transcripts, and artifact manifests. It does not require Phoenix, the `px` CLI, Phoenix database tables, or any Phoenix runtime process. -If a run has `external_trace` metadata and the Dashboard server has a Phoenix -endpoint configured, the run detail page can show the linked Phoenix session -summary, turns, trace IDs, root span input/output, timing, token usage, costs, -annotations, a trace tree, and selected span details. Missing configuration, -unreachable Phoenix, schema mismatches, or no matching session render as -recoverable empty/error states; AgentV run detail still loads from AgentV -artifacts. - -Configure the Dashboard server with one of: - -```bash -AGENTV_PHOENIX_ENDPOINT=https://phoenix.example -PHOENIX_HOST=https://phoenix.example -``` - -Optional server-side auth can be supplied with: - -```bash -AGENTV_PHOENIX_API_KEY=... -AGENTV_PHOENIX_AUTHORIZATION="Bearer ..." -PHOENIX_CLIENT_HEADERS='{"Authorization":"Bearer ..."}' -``` - -These credentials stay server-side and are not written to AgentV artifacts or -sent to the browser. +If a run has safe `external_trace.ui_url` metadata, the run detail page can show +an **Open in Phoenix** link. Missing Phoenix metadata does not affect AgentV run +detail because Dashboard reads AgentV artifacts as the canonical source. ## External Trace Metadata @@ -106,6 +83,7 @@ already-existing session. transcripts, datasets, experiments, or indexes. - No Phoenix-owned AgentV transcript, index, or storage model. - No Dashboard runtime dependency on Phoenix or `px`. +- No Dashboard Phoenix GraphQL/REST proxy or embedded Phoenix session/span UI. - No direct Dashboard access to Phoenix database tables. - No Phoenix dataset or experiment creation as part of the zero-infra local path. - No browser-side exposure of Phoenix API keys, authorization headers, cookies, diff --git a/apps/web/src/content/docs/docs/tools/dashboard.mdx b/apps/web/src/content/docs/docs/tools/dashboard.mdx index 409a3fdbc..72a72c359 100644 --- a/apps/web/src/content/docs/docs/tools/dashboard.mdx +++ b/apps/web/src/content/docs/docs/tools/dashboard.mdx @@ -50,19 +50,11 @@ service. If a trace artifact includes safe `external_trace` metadata for spans that were already emitted to Phoenix by Codex, Arize, or another hook, Dashboard may show -that external reference and, when the Dashboard server has a Phoenix endpoint -configured, read the linked Phoenix session through Phoenix GraphQL/API. The -run detail page renders this as optional external context with session summary, -turns, trace IDs, root span input/output, timing, token usage, costs, -annotations, trace tree, selected span details, and an **Open in Phoenix** link. - -Phoenix endpoint and auth configuration stay on the server process -(`AGENTV_PHOENIX_ENDPOINT`, `PHOENIX_HOST`, `AGENTV_PHOENIX_API_KEY`, -`AGENTV_PHOENIX_AUTHORIZATION`, or `PHOENIX_CLIENT_HEADERS`). API keys, -authorization headers, cookies, and tokens are not written to AgentV artifacts -or sent to the browser. AgentV still treats the local/Git-backed run artifacts -as canonical and does not export or project completed runs, transcripts, -datasets, experiments, or indexes into Phoenix. +that external reference as an **Open in Phoenix** link. Dashboard does not +proxy Phoenix GraphQL/REST or embed Phoenix session, trace, or span views. +AgentV still treats the local/Git-backed run artifacts as canonical and does +not export or project completed runs, transcripts, datasets, experiments, or +indexes into Phoenix. ## Options diff --git a/apps/web/src/content/docs/docs/tools/prepare.mdx b/apps/web/src/content/docs/docs/tools/prepare.mdx index 2a45848e6..975ae20b2 100644 --- a/apps/web/src/content/docs/docs/tools/prepare.mdx +++ b/apps/web/src/content/docs/docs/tools/prepare.mdx @@ -75,7 +75,7 @@ Use `--response` when the final answer text should be graded independently of th - Use provider-native settings, target hooks, or environment variables to enable session logs. - Use AgentV's OTLP options during normal eval runs, such as `--otel-file` or `--export-otel`, when AgentV is the runner. - For Opik, Langfuse, or another export-capable backend, treat their traces as external artifacts that can be imported or projected back into AgentV later. -- For Phoenix, use only optional read-only GraphQL/API correlation when safe `external_trace` metadata points to spans already emitted independently by Codex, Arize, or another hook. +- For Phoenix, use only optional link-out correlation when safe `external_trace` metadata points to spans already emitted independently by Codex, Arize, or another hook. AgentV remains responsible for eval definitions, workspace setup, grading, result bundles, and CI gates. Live trace storage, dashboards, and provider-specific run monitoring belong in the observability backend or the external harness. diff --git a/docs/adr/2026-06-11-phoenix-observability-adapter.md b/docs/adr/2026-06-11-phoenix-observability-adapter.md index a58741068..49012940b 100644 --- a/docs/adr/2026-06-11-phoenix-observability-adapter.md +++ b/docs/adr/2026-06-11-phoenix-observability-adapter.md @@ -6,9 +6,9 @@ Status: Superseded in part by [2026-06-21 Phoenix read-only correlation boundary This ADR remains useful for the narrower point that Phoenix-specific behavior does not belong in `packages/core`. Its earlier allowance for Phoenix dataset, -experiment, trace import/export helpers is superseded: Phoenix is now read-only -correlation/read-through for externally emitted traces, not an AgentV artifact -projection target. +experiment, trace import/export helpers is superseded: Phoenix is now link-out +correlation for externally emitted traces, not an AgentV artifact projection +target or Dashboard Phoenix-runtime dependency. ## Context @@ -39,7 +39,7 @@ Phoenix integration should live outside core behind an adapter boundary, current - a Phoenix OTel backend resolver; - Phoenix/OpenInference span-kind mapping; -- read-only Phoenix GraphQL/API helpers for externally emitted trace/session correlation; +- link-out helpers for externally emitted trace/session correlation; - explicit unsupported/lossy mapping reports. ## Minimal extension seam @@ -47,7 +47,7 @@ Phoenix integration should live outside core behind an adapter boundary, current Historical note: this ADR originally considered a first-class `--otel-backend phoenix` ergonomics path. That must not be used to make Phoenix a Dashboard dependency or an AgentV-owned artifact destination. Any future Phoenix work should be framed as -read-only correlation/read-through for externally emitted spans. +link-out correlation for externally emitted spans. A resolver should be approximately: @@ -82,7 +82,7 @@ The earlier prototype exposed a resolver, for example `phoenixOtelBackend`, so u - `--otel-file` for offline OTLP JSON export 2. Add a tiny backend resolver seam only if ergonomic backend names are needed. 3. Implement Phoenix endpoint/header/project routing in the Phoenix adapter boundary, not in core. -4. Keep any Phoenix read-through behavior outside core and behind Phoenix GraphQL/API. +4. Keep Phoenix out of Dashboard runtime fetch paths; use safe external links instead. 5. Consider moving existing vendor-specific core presets to the same resolver model later, but do not couple that cleanup to the Phoenix decision unless the implementation already touches the preset registry. ## Consequences @@ -91,7 +91,7 @@ Positive: - Keeps core aligned with AgentV's lightweight-core and composition principles. - Prevents Phoenix concepts from leaking into the generic trace model. -- Gives Phoenix users a read-only correlation path without blocking generic OTLP users. +- Gives Phoenix users a link-out correlation path without blocking generic OTLP users. - Reuses AgentV's existing pattern of narrow registries and convention-based local discovery. Negative: @@ -102,7 +102,7 @@ Negative: ## Tracker impact -- `av-vwa.6` remains valid only for generic trace artifacts and OTLP/OpenInference shapes. Phoenix-specific read-through stays outside core and must not become AgentV-to-Phoenix artifact projection. +- `av-vwa.6` remains valid only for generic trace artifacts and OTLP/OpenInference shapes. Phoenix-specific link-out metadata must not become AgentV-to-Phoenix artifact projection or Dashboard runtime fetching. - `av-vwa.6.1` is superseded as a Phoenix preset/resolver task unless it is reframed under the read-only external-trace correlation boundary. ## Open questions diff --git a/docs/adr/2026-06-21-phoenix-read-only-correlation-boundary.md b/docs/adr/2026-06-21-phoenix-read-only-correlation-boundary.md index ebcc70a80..542bebe6c 100644 --- a/docs/adr/2026-06-21-phoenix-read-only-correlation-boundary.md +++ b/docs/adr/2026-06-21-phoenix-read-only-correlation-boundary.md @@ -27,15 +27,17 @@ AgentV-owned run artifacts and the local Dashboard remain the supported zero-infra inspection path for AgentV run, trace, session, transcript, and comparison data. -Phoenix integration, when present, is read-only correlation/read-through: +Phoenix integration, when present, is link-out correlation: - AgentV artifacts may carry safe `external_trace` metadata that identifies an external Phoenix trace or session. -- Dashboard or CLI features may use Phoenix GraphQL/API to read that external - session in a future follow-up. +- Dashboard may expose an `Open in Phoenix` link when that metadata includes a + safe UI URL. - Phoenix does not own AgentV transcript, index, run, dataset, experiment, or storage state. - Dashboard must not require the `px` CLI at runtime. +- Dashboard must not proxy Phoenix GraphQL/REST or duplicate Phoenix sessions, + traces, or spans in AgentV UI. - Dashboard must not query Phoenix database tables directly. - The local Dashboard path must work without any Phoenix runtime dependency. @@ -50,8 +52,8 @@ Positive: - Keeps AgentV's source of truth portable across local development and CI. - Avoids a second write path for run, transcript, trace, and index state. -- Leaves room for optional Phoenix read-through without coupling Dashboard to a - Phoenix runtime, database schema, or CLI. +- Keeps Phoenix useful as an external viewer without coupling Dashboard to a + Phoenix runtime, database schema, API, or CLI. Negative: @@ -61,8 +63,9 @@ Negative: ## Follow-up -`av-kve.7` may add read-only Phoenix GraphQL/API read-through for externally -emitted sessions that are already referenced by `external_trace` metadata. That -work must not add AgentV-to-Phoenix export/projection, direct Phoenix database -access, `px` runtime requirements, Phoenix-owned indexes, or Phoenix transcript -ingestion. +`av-2il.5` removes the Dashboard Phoenix runtime fetch path and keeps only a +link-out affordance for externally emitted sessions that are already referenced +by safe `external_trace` metadata. Future work must not add AgentV-to-Phoenix +export/projection, direct Phoenix database access, `px` runtime requirements, +Phoenix-owned indexes, Phoenix transcript ingestion, or embedded Phoenix +session/span UI. diff --git a/docs/plans/trace-evaluation-architecture.md b/docs/plans/trace-evaluation-architecture.md index 4d557da1b..cf9e373ad 100644 --- a/docs/plans/trace-evaluation-architecture.md +++ b/docs/plans/trace-evaluation-architecture.md @@ -22,8 +22,8 @@ the read-only Phoenix correlation boundary in [docs/adr/2026-06-21-phoenix-read-only-correlation-boundary.md](../adr/2026-06-21-phoenix-read-only-correlation-boundary.md). Do not use this plan as current Phoenix product scope. AgentV does not export, import, or project completed runs, traces, transcripts, datasets, experiments, -or indexes into Phoenix. Phoenix is optional read-only GraphQL/API -correlation/read-through only when safe `external_trace` metadata points to +or indexes into Phoenix. Phoenix is optional link-out correlation only when +safe `external_trace` metadata points to spans already emitted independently by Codex, Arize, or another hook. --- @@ -50,7 +50,7 @@ The best-practice direction is clear: larger players own trace stores, dashboard - R6. AgentV OTLP export must continue to emit standards-aligned GenAI spans, especially `invoke_agent`, `chat`, and `execute_tool` operations. - R7. AgentV must map trace artifacts to and from OTLP/OpenInference-style traces without making Phoenix-specific assumptions in core. -- R8. Phoenix integration, if present, must be read-only correlation/read-through for externally emitted traces; Phoenix must not become the AgentV trace, dataset, experiment, transcript, or index backend. +- R8. Phoenix integration, if present, must be link-out correlation for externally emitted traces; Phoenix must not become the AgentV trace, dataset, experiment, transcript, or index backend. - R9. Unsupported or lossy mappings must be explicit in conversion reports instead of silently approximated. **Post-Hoc Trace And Transcript Evaluation** @@ -212,9 +212,9 @@ The exact schema belongs in implementation, but these concepts should be stable: ### U4. Phoenix Read-Only Correlation Path - **Goal:** Superseded for this plan by the 2026-06-21 Phoenix boundary ADR. Phoenix can be an external trace reference when spans were emitted independently, but it is not an AgentV trace, dataset, experiment, transcript, or index backend. -- **Files:** Future read-through belongs outside core and should use Phoenix GraphQL/API plus safe `external_trace` metadata. +- **Files:** Phoenix links belong outside core and should use safe `external_trace` metadata. - **Patterns:** Keep AgentV artifacts canonical. Do not add AgentV-to-Phoenix export/projection, direct Phoenix database access, `px` runtime requirements, or Phoenix-owned indexes. -- **Test Scenarios:** Future read-through should prove missing Phoenix state degrades cleanly and that secrets are not surfaced from `external_trace` metadata. +- **Test Scenarios:** Phoenix link-out should prove missing metadata degrades cleanly and that secrets are not surfaced from `external_trace` metadata. - **Verification:** Dashboard and AgentV result inspection must continue to work without Phoenix installed or running. ### U5. Pi Session Importer @@ -346,7 +346,7 @@ Outside this product's identity: - **Lossy imports:** Some sources lack tool timing, outputs, status, or branch data. Mitigation: record provenance and warning metadata instead of fabricating precision. - **Artifact size:** Full trace artifacts can be large. Mitigation: keep compact summaries as default result metadata and store full trace artifacts as optional sidecar artifacts. - **Privacy mistakes:** Tool args/results may contain secrets or PII. Mitigation: default to redacted content and centralize capture policy. -- **Phoenix scope creep:** It is tempting to mirror Phoenix concepts in core. Mitigation: keep Phoenix read-through outside core, read-only, and driven by safe `external_trace` metadata; do not add Phoenix dataset or experiment projection. +- **Phoenix scope creep:** It is tempting to mirror Phoenix concepts in core. Mitigation: keep Phoenix link-out outside core and driven by safe `external_trace` metadata; do not add Phoenix dataset or experiment projection. --- diff --git a/packages/phoenix-adapter/docs/support-matrix.md b/packages/phoenix-adapter/docs/support-matrix.md index 9596dcd9a..64cbc7066 100644 --- a/packages/phoenix-adapter/docs/support-matrix.md +++ b/packages/phoenix-adapter/docs/support-matrix.md @@ -5,9 +5,9 @@ YAML-to-Phoenix adapter fixture. It is not the supported AgentV product path: AgentV does not export or project completed runs, traces, transcripts, datasets, experiments, or indexes into Phoenix. -The current supported Phoenix boundary is read-only correlation/read-through -from safe `external_trace` metadata when Codex, Arize, or another hook already -emitted spans independently. +The current supported Phoenix boundary is link-out correlation from safe +`external_trace` metadata when Codex, Arize, or another hook already emitted +spans independently. If the legacy fixture is run for internal parity checks, its deterministic coverage is: