AGENT_EVALUATION.md

CodeWell Agent Evaluation

This document defines a lightweight agent-level evaluation protocol for CodeWell.

The goal is not to benchmark the raw model alone. The goal is to measure whether CodeWell improves an agent's task execution by providing better local retrieval and context assembly.

Evaluation Goal

Measure the delta between:

• baseline: an agent working without CodeWell retrieval
• codewell: the same agent working with CodeWell search, trace, and context tools

The key question is:

Does CodeWell help a coding agent finish repository tasks with fewer search steps, less wasted context, and higher task success?

Scope

This protocol is intentionally lightweight.

It does not require:

• full SWE-bench scale
• remote infrastructure
• required LLM APIs inside CodeWell
• a new heavyweight benchmark runner before the task design is stable

It is designed for:

• Codex
• Claude Code
• similar local or semi-local coding agents

What To Compare

For each task, run two conditions:

1. baseline
2. codewell

Keep these factors fixed between the two runs:

• same repository snapshot
• same task statement
• same model or agent configuration
• same time budget or step budget

The main difference should be whether the agent is allowed to use CodeWell retrieval.

Recommended Task Types

Start with 10-20 tasks before expanding further.

Recommended TS/JS-heavy categories:

• route or controller to service flow
• service to repository or data layer flow
• auth, guard, or middleware flow
• test to implementation discovery
• bootstrap, config, or wiring discovery
• frontend route to feature to API flow
• form submission to mutation to client call flow
• protected route or redirect logic

Per-Task Inputs

Store tasks as JSON using evaluations/agent_eval_task_template.json.

Each task should define:

• task id and name
• repository label and optional subpath
• task prompt
• language and framework tags
• expected files
• expected symbols
• optional expected trace relationships
• success criteria

Ground truth should be established before running the agent.

Per-Run Outputs

Store results using evaluations/agent_eval_result_template.json.

Each run should capture:

• condition: baseline or codewell
• agent name and model
• task success
• final patch status
• files touched
• first relevant file hit
• tool call counts
• manual search steps
• approximate token usage if available
• token usage details when the local Claude session log exposes them
• elapsed time
• Claude session id and local log path for auditability when available
• notes about failure or confusion

Core Metrics

Track both outcome and effort.

Outcome metrics:

• task_success_rate
• correct_file_hit_rate
• correct_symbol_hit_rate
• final_patch_in_expected_area

Effort metrics:

• time_to_first_relevant_file_seconds
• tool_call_count
• manual_search_steps
• elapsed_seconds
• token_usage when available

Quality and failure metrics:

• wrong_context_events
• irrelevant_file_open_count
• abandoned_runs

Suggested A/B Report

Aggregate results by:

• repository
• task category
• framework
• condition
• task bucket

Recommended summary fields:

• tasks total
• tasks succeeded
• success rate
• median elapsed seconds
• median time to first relevant file
• median tool call count
• median manual search steps
• median token usage
• wrong context event rate

Task-Bucket View

The repo now includes a lightweight bucket-summary helper:

• scripts/agent_eval_bucket_summary.py
• evaluations/agent_eval_task_buckets.json
• scripts/agent_eval_repeated_run_summary.py
• evaluations/agent_eval_prompt_goal_labels.json
• scripts/agent_eval_repeat_plan.py
• evaluations/agent_eval_repeat_protocol_strong_multi_goal.json

Use it after one or more A/B result files have been produced:

python scripts/agent_eval_bucket_summary.py `
  .agent-eval-wiring-temp\results\agent_eval_results.json `
  .agent-eval-wiring-temp\results\agent_eval_results_phase2.json `
  --bucket-file evaluations\agent_eval_task_buckets.json

This reports:

• complete A/B pair count
• incomplete task ids
• per-bucket success / elapsed / tool / token deltas
• overall summary across all completed pairs

Current buckets:

• navigation-heavy
• obvious-near-obvious
• middle-bucket

The current internal wiring batch shows a clear pattern:

• navigation-heavy: stable positive delta for CodeWell
• obvious-near-obvious: weak or negative delta, useful as a boundary bucket

Treat this bucketed view as the default lens for deciding whether a retrieval policy change is actually helping the right task type.

Repeated-Run View

For prompts with high run-to-run variance, especially multi-goal prompts, use the repeated-run summary helper:

python scripts/agent_eval_repeated_run_summary.py `
  .agent-eval-wiring-temp\results\agent_eval_results.json `
  .agent-eval-wiring-temp\results\agent_eval_results_phase2.json `
  .agent-eval-wiring-temp\results\agent_eval_results_post_nav_support_pass1.json `
  --goal-label-file evaluations\agent_eval_prompt_goal_labels.json

This reports:

• sample count per task and condition
• median / mean / range for elapsed time, tool calls, and token usage
• first relevant file stability
• goal-shape labels such as weak_multi_goal and strong_multi_goal

Recommended interpretation rule:

• single-run results are acceptable for clearly stable tasks
• repeated-run median should be preferred for strong_multi_goal tasks
• high variance should be treated as a prompt-goal signal, not automatically as a retrieval failure

The repeated-run summary now also supports:

• --goal-shape strong_multi_goal to filter by prompt goal shape
• --min-samples 3 to focus on tasks with enough repeated evidence

Example:

python scripts/agent_eval_repeated_run_summary.py `
  .agent-eval-wiring-temp\results\agent_eval_results.json `
  .agent-eval-wiring-temp\results\agent_eval_results_phase2.json `
  .agent-eval-wiring-temp\results\agent_eval_results_post_nav_support_pass1.json `
  .agent-eval-wiring-temp\results\agent_eval_results_post_classifier_fix.json `
  .agent-eval-wiring-temp\results\agent_eval_results_post_kernel_companion_fix.json `
  --goal-label-file evaluations\agent_eval_prompt_goal_labels.json `
  --goal-shape strong_multi_goal `
  --min-samples 2

Strong-Multi-Goal Repeat Protocol

strong_multi_goal tasks should now be treated as a separate protocol class rather than as normal single-run A/B tasks.

Default rule:

• run 3 A/B pairs per task
• keep task text fixed across all repeats
• report median by condition
• inspect range and first-relevant-file stability before declaring a win or loss

Generate a repeat plan from the current goal labels:

python scripts/agent_eval_repeat_plan.py `
  --goal-label-file evaluations\agent_eval_prompt_goal_labels.json `
  --output evaluations\agent_eval_repeat_protocol_strong_multi_goal.generated.json

The repository also includes a checked-in current manifest for the present strong-multi-goal batch:

• evaluations/agent_eval_repeat_protocol_strong_multi_goal.json
• .agent-eval-wiring-temp/results/agent_eval_repeats_strong_multi_goal.json

This makes the repeated-run protocol operational instead of ad hoc:

• selected task ids are explicit
• repeat count is explicit
• result-file naming is explicit
• per-run session labels are explicit

Current completed read for the checked-in strong-multi-goal batch:

• all 3 tasks have now reached 3 samples per condition
• adonis_kernel_auth_middleware_wiring is the clearest stable-first-file case
• nestjs_users_admin_guard_service_wiring is the clearest unstable-first-file case
• adonis_profile_route_auth_middleware_wiring demonstrates why a partial repeated-run read can still flip after the third sample lands

Current interpretation rule should therefore be:

• treat strong_multi_goal as a different evaluation slice, not a noisy version of normal tasks
• require the full repeated-run protocol before using such tasks in a product-facing win/loss claim

Recommended First Pass

Use the current maintained TS/JS real-project set first.

Suggested first batch:

• Express + Prisma
• Bulletproof React
• NestJS clean architecture
• NestJS boilerplate
• AdonisJS backend

Start with 2 tasks per project.

That gives a small but useful 10-task baseline for agent A/B comparison.

The repository includes a ready-to-run first batch at:

• evaluations/agent_eval_tsjs_first_batch.json
• evaluations/agent_eval_tsjs_first_batch_results.empty.json

Source Hygiene

Keep evaluation source repositories separate from writable run workspaces.

Recommended split:

• eval-repos/source-readonly/ contains clean baseline source trees
• CodeWell_test/runs/ contains per-run writable copies

Do not start Claude Code inside eval-repos/source-readonly/. Only let scripts/agent_eval_session.py copy from that source root into a fresh run workspace.

The current automation defaults to eval-repos/source-readonly/ as --source-root.

Do not use eval-repos/github/ as a formal A/B source root. That older tree has already produced polluted copies for at least one task and should be treated as a staging/import area, not as a clean baseline.

Simplified Local Workflow

Use the session helper to reduce the manual setup work around each run.

Fastest path for repeatable experiments:

python scripts/agent_eval_session.py run-once `
  --task-id express_login_flow_patch_area `
  --condition codewell `
  --claude-model deepseek-v4-pro[1m]

This fully automates one run:

• creates a fresh copied workspace
• configures MCP automatically for codewell
• runs claude -p non-interactively
• saves the raw Claude output to claude_output.txt
• extracts the final JSON result
• writes the run into the aggregate results file

To run both baseline and codewell back-to-back for one task:

python scripts/agent_eval_session.py run-pair `
  --task-id express_login_flow_patch_area `
  --claude-model deepseek-v4-pro[1m]

To run a whole task file in pair mode:

python scripts/agent_eval_session.py run-batch `
  --claude-model deepseek-v4-pro[1m]

Optional batch controls:

• --task-ids task_a task_b runs only a subset
• --mode baseline or --mode codewell runs only one condition per task
• --continue-on-error keeps going after one failed task
• --first-condition codewell on run-pair flips the order for that task

For a manual step-by-step run, use prepare-run plus record-run as described below.

Prepare a run:

python scripts/agent_eval_session.py prepare-run `
  --task-id express_login_flow_patch_area `
  --condition codewell `
  --configure-mcp

This does four things:

• creates a fresh copied workspace under CodeWell_test/runs/
• initializes the aggregate result file if it does not exist yet
• writes the exact Claude prompt into prompt.txt
• optionally toggles Claude Code MCP for the prepared workspace

The script prints a JSON payload with:

• working_directory
• prompt_file
• results_file
• next_steps

After that, the human loop is short:

1. cd into the printed working_directory
2. start claude
3. paste the prompt from prompt_file
4. copy Claude's final output

Then record the result:

python scripts/agent_eval_session.py record-run `
  --task-id express_login_flow_patch_area `
  --condition codewell `
  --results-file <agent-eval-root>\results\agent_eval_results.json `
  --agent-name claude-code `
  --model deepseek-v4-pro `
  --result-json "<paste Claude final output here>"

record-run extracts the JSON object from Claude's final answer and merges it into the aggregate results file automatically.

Recommended Practice

For formal A/B runs:

• use a fresh copied workspace for every condition
• keep the same model, skills, and time budget across both runs
• use the same prompt text for baseline and codewell
• prefer narrow prompts that name one concrete bug or behavior, not a broad "patch a bug in this area"
• only change whether CodeWell MCP is available
• if the agent drifts to a different bug, mark that run as failed or note prompt drift explicitly

Relation To SWE-bench

This protocol comes before SWE-bench-style evaluation.

Use it first because it is:

• cheaper
• easier to reproduce locally
• easier to attribute to CodeWell itself
• more sensitive to retrieval and context quality

After this protocol is stable, a smaller SWE-bench-Lite-style comparison can be added as an external validation layer.