[Feature]: Agent Performance Regression Testing & Drift Detection

[VasuBansal7576]

Problem Statement

Agent behavior can degrade over time due to:

• Model upgrades

• Prompt changes

• Tooling or runtime modifications

• Environment or dependency changes

Currently, there is no automated way to detect performance regressions or confirm that an agent still behaves as expected after changes. This makes it risky to:

• Upgrade models

• Refactor agents

• Deploy new runtime versions

Issue #2156 highlights the challenge of agent drift, but there is no concrete mechanism to measure or enforce performance stability.

Proposed Solution

Introduce an agent regression testing framework that enables automated performance validation over time.

Core capabilities:

• Define “golden” test cases with expected outputs or evaluation criteria

• Run agents against these test cases on demand or in CI

• Compare results against a stored baseline

• Detect statistically significant regressions

• Fail CI or raise alerts if performance drops beyond a threshold (e.g., >5%)

This brings software-style regression testing to autonomous agents.

Alternatives Considered

• Manual spot checks

❌ Not scalable

❌ Highly subjective

• Logging and observability only

❌ Detects failures, not regressions

❌ No baseline comparison

Neither approach provides confidence that agent quality is preserved across changes.

Additional Context

This feature is particularly valuable for:

• Production agents with SLAs

• Teams iterating quickly on prompts or tools

• Multi-agent systems where small regressions compound

• Regression testing is a foundational requirement for safe, long-term agent evolution.

Implementation Ideas

High-level approach:

• Store baseline results for a set of “golden” inputs

• Re-run agents against the same inputs after changes

• Compare outputs using pluggable evaluation strategies

• Aggregate results into a regression report

• Example sketch:

class AgentRegressionTest:
def test_agent(self, agent_path: str, test_cases: list):
results = []
for case in test_cases:
output = run_agent(agent_path, case.input)
score = compare_to_baseline(case.expected, output)
results.append(score)
return RegressionReport(success_rate=avg(results))

Possible extensions:

• Multiple evaluation strategies (exact match, semantic similarity, task success)

• Model-aware baselines

• Trend tracking across releases

• Native CI/CD integration

Why This Matters

This directly addresses the agent drift problem discussed in #2156 and provides a concrete, enforceable solution.

It enables:

• Safer upgrades

• Faster iteration

• Higher confidence in production deployments

This is a hard, high-leverage problem that significantly improves agent reliability over time.

[github-actions]

Found a possible duplicate of #2463: Both issues propose agent performance monitoring and regression detection systems. Issue #2463 focuses on LLM model performance monitoring and drift detection for agent calls, while #2897 focuses on agent behavior regression testing. They address the same core need: detecting performance degradation over time through baseline comparison and automated testing.

[VasuBansal7576]

Thanks for flagging the similarity to #2463. These are complementary features addressing different aspects:
#2463: Monitors LLM model performance degradation (latency, tokens, API health)
This issue (#2897): Tests agent behavior/output consistency against golden test cases
Example difference:
#2463 catches "GPT-4 got slower"
#2897 catches "Sales agent stopped asking for phone number"
Both are needed for production reliability. I'd like to implement #2897 (behavior regression testing) as it pairs well with the telemetry work in #2156. Happy to collaborate with whoever is working on #2463!

[austin931114]

@VasuBansal7576 Thank you for your contribution. Could you elaborate more details on how you would like to integrate this method with the current structure?

[Samir-atra]

Hello,

I am interested in working on this. Please assign me. I have created a pull request.

[nanookclaw]

Interesting distinction here: regression-after-a-change is not quite the same thing as longitudinal drift when nothing obvious changed.

We just published a small longitudinal reliability paper (PDR, DOI: https://doi.org/10.5281/zenodo.19028012), and two design details from that work might matter for Hive's eval system:

• A single golden baseline can overcall regressions. In one repeated-run case, 10 byte-identical executions split into two stable reasoning attractors (6A / 4B) with the same prompt/model/tooling. If the task spec is underspecified, "different from baseline" is not automatically drift.
• Windowed history mattered more than single failures. On a separate production fleet case, drift signals only became stable once the comparison window got large enough (roughly >= 5 observations). One-off failures were noisy; rolling patterns were actionable.

That makes me think the eval/history layer should store more than "expected output":

1. specification_clarity / ambiguity metadata so variance from underspecified tasks doesn't get mislabeled as regression
2. acceptable outcome sets or trajectory families rather than one canonical baseline
3. rolling temporal aggregates (consistency / adaptation / reliability over windows), not just point-in-time pass/fail

Concrete question: do you see Hive's upcoming eval work (#6520 / #5650) handling longitudinal history + windowed drift detection inside the eval framework itself, or would that live better as a separate analysis layer over run history?

[nanookclaw]

Following up from my earlier comment — that question about where longitudinal history belongs has an answer now.

We shipped drift.temporal as a standalone contrib evaluator (PR #140 in agentcontrol/agent-control), following the Galileo contrib pattern. Implementation detail that might be relevant here:

We ended up putting the rolling window math inside the evaluator rather than as a post-hoc analysis layer, because the signal needs access to the same run context (prompt, model, config) to separate spec-ambiguity variance from genuine drift. Keeping it inside the eval keeps that context in scope.

Two things we learned during implementation that connect to this issue directly:

1. Window size matters more than metric choice. Below ~5 observations, consistency/adaptation/reliability scores are too noisy to act on. Hive would likely want to gate alerts on minimum window depth, not just deviation from baseline.
2. The spec-ambiguity dimension is load-bearing. In our validation data, ~30% of apparent 'regressions' were bifurcation from underspecified tasks — same prompt producing two stable reasoning attractors. Without flagging that separately, regression testing has false positive noise that erodes trust in the system.

If Hive's eval system ends up tracking longitudinal history, happy to share the windowed aggregation logic or discuss integration patterns.