Skip to content
This repository was archived by the owner on Oct 16, 2026. It is now read-only.
This repository was archived by the owner on Oct 16, 2026. It is now read-only.

Plan AgentV -> Opik export integration #1388

Closed
Closed
Plan AgentV -> Opik export integration#1388

Description

@christso
christso
opened on Jun 15, 2026

Context

We evaluated whether Opik can replace AgentV for CI gating. Decision: do not replace AgentV. AgentV should remain the CI authority and portable coding-agent harness; Opik should be the observability/eval/repair platform AgentV exports to.

Durable handoff findings are in the research wiki:

https://github.com/tsoyang-org/ai-research-wiki/blob/ai-research-wiki-coordinator/concepts/agentv-opik-integration-findings.md

Decision

Build AgentV -> Opik first, not opik run agentv.

Expected flow:

agentv run evals/*.yaml
  -> AgentV computes pass/fail and exits nonzero on gates
  -> AgentV exports traces/scores/run metadata to Opik
  -> Opik stores experiments, traces, feedback scores, dashboards
  -> selected Opik failures can become future AgentV regression cases

Braintrust is out of scope for this workstream. For the current need, AgentV + Opik covers CI gating, observability, experiment history, dashboards, trace-to-test-suite growth, and repair-loop workflows.

Proposed workstreams

  1. Document the boundary

    • AgentV is the runner/gate.
    • Opik is an export target and repair/observability platform.
    • Keep AgentV's existing CI exit semantics authoritative.

  2. Add Opik OTLP export recipe

    • Use AgentV's OpenTelemetry direction as the first trace path.
    • Document Opik endpoint config for local self-host and cloud/enterprise.
    • Add required span attributes: agentv.run_id, agentv.eval_id, agentv.test_id, agentv.target, agentv.provider, agentv.workspace_mode, agentv.gate_status.

  3. Add agentv export opik for completed runs

    • Input: .agentv/results/runs/<run-id>/index.jsonl.
    • Output: Opik experiment with per-test items, feedback scores, and trace links.
    • Keep SDK/API usage isolated and version-pinned.

  4. Add explicit Opik failure importer

    • Proposed shape: agentv import opik --project <name> --filter <query> --out cases/opik-failures.jsonl.
    • Convert selected failed Opik traces/test-suite items into AgentV regression cases.
    • Require review before commit because production traces may contain secrets/customer data.

  5. Publish CI recipe

    • CI still fails on AgentV gates.
    • Opik upload runs best-effort/always() after the AgentV run so failures are visible.
    • Missing Opik credentials should not block local OSS users unless export is explicitly required.

Acceptance criteria

  • agentv run ... --export-opik or agentv export opik <run-id> uploads one completed run to Opik.
  • Opik shows one experiment for the AgentV run with per-case scores.
  • Each case links back to AgentV run/test IDs and local artifact paths.
  • Required evaluator failures are visible as feedback scores and metadata.
  • AgentV CI behavior is unchanged: gates still fail the job via AgentV's own exit code.
  • Export can be disabled, best-effort, or required through explicit config.

Request for coordinator/agents

Please brainstorm the implementation shape, then break this down into beads. Suggested initial beads:

  • docs: AgentV/Opik boundary and CI authority
  • otel: Opik OTLP export recipe
  • exporter: agentv export opik adapter
  • schema: export metadata mapping for run/test/score/gate fields
  • importer: Opik failure -> AgentV regression cases
  • ci: GitHub Actions example with best-effort export
  • security: redaction and metadata-only export mode

Source anchors

Activity

  1. github-project-automation
    added this to AgentV OSS Boardon Jun 15, 2026
  2. github-project-automation
    moved this to Backlog in AgentV OSS Boardon Jun 15, 2026

  3. christso commented on Jun 15, 2026

    @christso
    christso
    on Jun 15, 2026
    CollaboratorAuthor

    Coordination is now active in Beads for the WTG Opik workstream.

    Active Beads:

    • Worker: av-vwa.15 (P2 Opik: install Opik and port WTG PR 679 E2E evals)
    • Coordinator: av-vwa.15.1 (Coordinate WTG Opik E2E eval workstream)

    Worker WorkMux session: wtg-opik-e2e-evals
    Worker branch: private/av-vwa-15-opik-e2e
    Worker path: /home/entity/projects/EntityProcess/wtg-ai-prompts-experiment__worktrees/wtg-opik-e2e-evals

    Current status: worker is active, has read the AgentV/Opik handoff via authenticated GitHub access after anonymous wiki access 404ed, and is pursuing a dry-run-first Opik adapter path because the local opik package is not installed. AgentV remains the CI authority; Opik remains the trace/experiment/test-suite/feedback/dashboard target. Braintrust remains out of scope.

  4. christso commented on Jun 15, 2026

    @christso
    christso
    on Jun 15, 2026 – with ChatGPT Codex Connector
    CollaboratorAuthor

    Coordination update: a dedicated WTG AI Prompts coordinator has been spawned for this workstream.

    Beads:

    • Coordinator bead: av-vwa.15.1 — Coordinate WTG Opik E2E eval workstream
    • Worker bead: av-vwa.15 — P2 Opik: install Opik and port WTG PR 679 E2E evals

    WorkMux sessions:

    • Coordinator: wtg-ai-prompts-coordinator
      • branch: private/av-vwa-15-1-opik-coordinator
      • path: /home/entity/projects/EntityProcess/wtg-ai-prompts-experiment__worktrees/wtg-ai-prompts-coordinator
    • Worker: wtg-opik-e2e-evals
      • branch: private/av-vwa-15-opik-e2e
      • path: /home/entity/projects/EntityProcess/wtg-ai-prompts-experiment__worktrees/wtg-opik-e2e-evals

    The coordinator is responsible for monitoring the worker, resolving safe blockers, breaking follow-up findings into beads, and posting only material milestones here. Braintrust remains out of scope for this workstream; the target stack is AgentV + Opik.

  5. christso commented on Jun 15, 2026

    @christso
    christso
    on Jun 15, 2026
    CollaboratorAuthor

    WTG Opik PR 679 worker milestone for Bead av-vwa.15.

    Branch/worktree:

    • branch: private/av-vwa-15-opik-e2e
    • path: /home/entity/projects/EntityProcess/wtg-ai-prompts-experiment__worktrees/wtg-opik-e2e-evals

    Artifacts added in EntityProcess/wtg-ai-prompts-experiment:

    • framework-parity/pr-679/opik/pr50857_opik_adapter.py
    • framework-parity/pr-679/opik/README.md
    • framework-parity/tests/test_pr679_opik_adapter.py

    Result:

    • Installed and inspected opik==2.0.64 in ignored .agentv/opik-venv only; no repo-level dependency was added.
    • Created a dry-runable Opik test-suite seed for both PR 679 E2E cases.
    • Created a completed-run dry-run export path against local AgentV run pr679-pr50857-clean-2026-06-08T05-42-55Z, preserving AgentV case IDs, scores, grader scores, assertion verdicts, artifact paths, source PR provenance, and redacted tool trace summaries.
    • Live upload remains blocked in this shell because OPIK_API_KEY, OPIK_WORKSPACE, OPIK_PROJECT_NAME, and OPIK_URL_OVERRIDE were unset; no raw secrets were printed or persisted.

    Mapping findings useful for agentv export opik:

    • Eval run -> Opik experiment plus root trace metadata; AgentV remains CI gate authority.
    • Test/case -> Opik test-suite item/dataset item, preserving AgentV eval path, case ID, source PR refs, workspace template, and type:file source paths.
    • Final score, per-grader scores, and assertion pass/fail -> Opik feedback scores.
    • Skill-trigger -> Opik text surrogate/global assertion plus feedback score from completed runs; the AgentV skill-trigger remains authoritative for tool trajectory.
    • Transcript tool calls -> Opik tool-span plan with tool name, call ID, timing, counts, and SHA-256 hashes of redacted inputs/outputs.

  6. christso commented on Jun 15, 2026

    @christso
    christso
    on Jun 15, 2026 – with ChatGPT Codex Connector
    CollaboratorAuthor

    Coordinator close-out for WTG Opik workstream.

    Worker av-vwa.15 has a final result comment and WorkMux session wtg-opik-e2e-evals is done. Coordinator av-vwa.15.1 is closed.

    Follow-up Beads under av-vwa.15:

    • av-vwa.15.2 — first-class agentv export opik completed-run exporter
    • av-vwa.15.3 — redaction and metadata-only export policy
    • av-vwa.15.4 — safe credentialed live-upload smoke path
    • av-vwa.15.5 — PR 679 adapter raw-output edge hardening, resolved by the worker follow-up

    Post-review verification update: the worker added the raw-output missing-path guard and regression test; focused Opik tests now pass 4 cases and combined parity tests pass 8 cases.

    Main milestone remains: #1388 (comment)

  7. christso commented on Jun 21, 2026

    @christso
    christso
    on Jun 21, 2026
    CollaboratorAuthor

    Thanks, this older Opik planning thread is now folded into the public adapter-led integrations epic #1467. The public Opik post-run boundary is also documented in PR #1406. Closing this issue as consolidated; use #1467 for roadmap discussion.

  8. christso
    closed this as completedon Jun 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions