docs: arbiter agent design document

[Hyperkid123]

Summary

• Design doc for the arbiter meta-agent — audits, analyzes, and improves all Rehor instances
• 7 capabilities: idle cycle audit, transcript pattern mining, PR/MR review gap analysis, cost anomaly detection, config drift detection, cross-instance learning, preflight effectiveness audit
• Architecture: NetworkPolicy for fleet port isolation, SA identity via auth proxy, dual-port pattern (instance-scoped vs fleet-scoped)
• Daily rotation cadence — cheap capabilities every cycle, one expensive capability per weekday
• Task-based state tracking (watermarks), memory-based learning (patterns, corrections)
• All open questions resolved into decisions

Jira

• Epic: REHOR-54 — Arbiter Agent
• Story: REHOR-55 — Review design doc

Test plan

• Review all 7 capability definitions for completeness
• Validate architecture decisions (NetworkPolicy, SA identity, dual-port)
• Confirm cadence rotation covers all capabilities fairly
• Check security considerations for public repo output sanitization
• Verify dependencies list is current
• Approve or file follow-up issues for any gaps

🤖 Generated with Claude Code

[petrsimon]

I've just learned a bit more about LangFuse and it seem like it's worthwhile to consider it before duplicating functionality we could get for free. Or has it been already ruled out? Let me know what you think, I can look into more details. Below are two summaries from agents.

The arbiter is building a custom, bespoke version of several things Langfuse provides out of the box. Specifically:

Arbiter Capability	Langfuse Equivalent
§1 Idle Cycle Auditor — detect token waste	Cost & latency dashboard + monitors/alerts
§2 Transcript Pattern Miner — find repetitive tool call sequences	Trace hierarchy + session grouping (query traces for repeated span patterns)
§3 PR Review Gap Analyzer — mine review comments for instruction gaps	Evaluation/scoring — LLM-as-judge on traces, user feedback collection
§4 Cost Anomaly Detector — spot unusual cost patterns	Built-in cost analytics + monitors with Slack/webhook alerts
§7 Preflight Effectiveness Audit — correlate preflight with outcomes	Trace filtering + custom metrics

Not replaceable by Langfuse:

• §5 Config Drift Detector — semantic analysis of config repo divergence (domain-specific)
• §6 Cross-Instance Learning — propagating skills across fleet (domain-specific)
• Output side (PRs to config repos, Jira tickets, instruction fixes) — Langfuse is read-only observability, not an actuator

The interesting question: Should the arbiter consume Langfuse data instead of building its own telemetry pipeline? If every bot cycle was
traced in Langfuse with hierarchical spans (preflight → agent loop → tool calls), the arbiter could query Langfuse's API for:

• Idle cycles (traces with high token count but no meaningful tool calls)
• Repeated patterns (similar trace structures across instances)
• Cost anomalies (Langfuse already computes per-trace cost)
• Quality scores (attach evals to traces, query for degradation)

That would let you skip building the watermark/transcript-store/dashboard-query plumbing in §2, §3, and §4, and instead have the arbiter
query a Langfuse instance as its primary data source. The dashboard cycle records and transcript store become Langfuse traces.

Also found this https://source.redhat.com/departments/strategy_and_operations/it/itx/document_management_and_collaboration/dmc_wiki/langfuse__the_universal_guide_to_ai_observability

Red Hat already runs a Langfuse instance with onboarding, CMDB integration, and an internal support channel
(#forum--langfuse).

A few things stand out for REHOR specifically:

Good fit:

• Trace hierarchy maps directly to REHOR's cycle structure: trace = cycle, spans = preflight → agent loop → tool calls → cost push
• Cost tracking per trace would complement/replace the custom Prometheus counters in REHOR-45 — Langfuse already does per-call USD
  attribution
• Eval/scoring is exactly what the arbiter's §3 (PR Review Gap Analyzer) needs — attach quality scores to traces, trend them over time
• Environment tagging (env:stage) solves the "13 instances across 9 teams" separation without needing separate projects

Friction points for REHOR:

• §7.1 VPN issue — REHOR runs on OSD (OpenShift Dedicated), so the "Run Experiment" button won't work. All experiment triggers must be programmatic from inside
  the cluster. Fine for the arbiter (it's code anyway), but limits ad-hoc UI experimentation
• §7.3 Multi-turn — REHOR cycles are single-turn (one agent invocation per cycle), so this isn't a problem. But if you ever want
  session-level views across cycles for the same Jira ticket, you'd hit this limitation
• §7.2 Filtering sub-traces — the arbiter's transcript pattern mining (§2) needs to filter on specific tool call spans inside traces. The
  workaround (use LLM-as-a-Judge page instead of Evaluator page) works but is clunky for programmatic access

What this means for the arbiter (REHOR-54):

• §1 (Idle Cycle Auditor) and §4 (Cost Anomaly) could query Langfuse traces instead of building custom dashboard API queries + watermarks
• §2 (Transcript Pattern Miner) gets the trace hierarchy for free — no need to build a separate transcript store
• §3 (Review Gap Analyzer) uses Langfuse evaluations/scores natively
• The arbiter's watermark system (task-based last_processed_cycle_id) could be replaced by Langfuse's time-based trace filtering

Bottom line: The instrumentation side (tracing every cycle, cost tracking, eval scores) should just use the existing Langfuse instance — no
reason to rebuild that plumbing. The arbiter then becomes a consumer of Langfuse data via its API, not a standalone telemetry pipeline. The
arbiter's unique value (PRs, cross-instance learning, config drift) stays custom.

[Hyperkid123]

@petrsimon this is a good find. I don't think this agent needs another REHOR instance. We should try their local docker compose setup and see the output. I'd love to use something that already exists and give we already have RH instance the onboarding could be quick.