Agent Evaluation Framework Template

Overview

This is a template framework for creating and evaluating AI agent tasks. It provides a structured approach to:

• Define coding tasks with clear specifications
• Grade agent solutions automatically using test-based validation
• Manage multiple task difficulties (easy, medium, hard)
• Run tasks in isolated environments with proper grading

Project Structure

.
├── src/hud_controller/          # Main framework code
│   ├── env.py                   # HUD v6 environment (ssh + two-yield template)
│   ├── app.py                   # validate_problem CLI for imagectl3 -v
│   ├── prompts.py               # Problem prompt helpers
│   ├── spec.py                  # Core specifications (Problem, Grade)
│   ├── grading_runner.py        # Test execution and grading logic
│   ├── utils.py                 # Utility functions
│   ├── setup.py                 # Environment setup
│   ├── problems/                # Task definitions by difficulty
│   │   ├── basic.py             # Easy difficulty tasks
├── run_eval.py                  # Local Docker eval driver (HUD v6)
├── tasks.py                     # Generated task rows (imagectl3 -j)
├── pyproject.toml               # Python package configuration
├── Dockerfile                   # Container setup
└── README.md                    # This file

Core Concepts

1. Problem Definition

Problems are defined using the ProblemSpec data class with these key fields:

    ProblemSpec(
        id="simple_counter", # the unique ID of the problem
        description="""Please implement a simple synchronous counter that with reset, enable, and load functionality.
Inputs:
clk - Clock signal (triggers on rising edge)
rst - Synchronous reset signal
ena - Enable signal (allows counting)
set - Load signal (sets counter to a specific value)
din - 8-bit data input (value to load when set is high)
Output:
counter - 8-bit counter value        
        
""", # What you want the agent to do
        difficulty="easy", # how difficult the problem is
        # the branch names
        base="simple_counter_baseline", 
        test="simple_counter_test",
        golden="simple_counter_golden",
        test_files=["tests/test_simple_counter_hidden.py"]
    )

2. Test-Based Validation

Tasks are graded by:

1. Copying the repository (including whatever changes the agent made) to a clean workspace
2. Applying the agent's solution patch
3. Applying a test patch on top of what the agent did (adds tests that would fail in an unmodified repo)
4. Running pytest <test files> to test the build

Creating New Tasks

Step 1: Prepare Git Branches

You need three branches in your target repository (the one that we clone in the dockerfile):

1. baseline - Starting state with the bug/missing feature
2. test - Adds tests that should fail on baseline, and pass in golden branch
3. golden - Contains the correct solution (for reference). Notably, this should not contain the tests.

Step 2: Define the Task

We currently only have src/hud_controller/problems/basic.py, but feel free to make more files in the subdirectory. Once you do that, you can add a problem to the registry as follows:

PROBLEM_REGISTRY.append(
    ProblemSpec(
        id="simple_counter",
        description="""Please implement a simple synchronous counter that with reset, enable, and load functionality.
Inputs:
clk - Clock signal (triggers on rising edge)
rst - Synchronous reset signal
ena - Enable signal (allows counting)
set - Load signal (sets counter to a specific value)
din - 8-bit data input (value to load when set is high)
Output:
counter - 8-bit counter value        
        
""",
        difficulty="easy",
        base="simple_counter_baseline",
        test="simple_counter_test",
        golden="simple_counter_golden",
        test_files=["tests/test_simple_counter_hidden.py"],
    )
)

The base, test, and golden branches must correspond to the branches you created in the first step.

Step 3: Validate your problem

It's important to ensure that your problems pass a basic sanity check:

• All tests at the baseline branch should pass
• When we apply the hidden test set, the hidden tests should fail
• When we apply the golden patch and then apply the hidden test set, all tests should pass

To help you with this, we have a script called utils/imagectl3.py.

To run and build the images you can do:

uv run utils/imagectl3.py --build --validate

You can specify the exact image you want to test with the --ids flag. You can also make this easier to type by using the shorform -b flag for --build and the shortform -v flag for --validate.

uv run utils/imagectl3.py -bv --ids simple_counter

Note: ensure your image is built before you try to validate it.

Running Tasks

Setup Environment

uv sync

Build, validate, and generate tasks.py

uv run utils/imagectl3.py verilog_ -bvj

This builds Docker images with prefix verilog_, validates patch logic, and writes tasks.py. For many problems, use parallel jobs:

uv run utils/imagectl3.py verilog_ -bvj --jobs 4

Run agent eval locally (HUD v6)

Each problem runs in its own Docker image via run_eval.py:

uv run python run_eval.py --ids simple_counter --agent claude \
  --model claude-sonnet-4-5 --max-steps 150 --group-size 10

Use --full to run all tasks in tasks.py with max-steps 100.

Remote eval (push images)

uv run utils/imagectl3.py govindhud/verilog_ -bvjp --jobs 4

Hosted HUD v6 eval (hud deploy + hud sync tasks) is not wired up in this template yet.

Configuration

Target Repository Dependency

The framework clones a target Verilog repository that contains the problems to be solved. This is configured in two places:

1. Dockerfile (line ~109): Specifies which repo to clone

   RUN git clone https://github.com/hud-evals/example-verilog-codebase /home/ubuntu/example-codebase

2. grading_runner.py (line 46): Path where the cloned repo lives

   self.original_repo_path = "/home/ubuntu/example-codebase"

Important: When making changes to the remote repository, ALWAYS increment the random variable in the Dockerfile (line ~108) to force Docker to re-clone:

ENV random=random6  # Increment this number!

Environment Variables

Key environment variables:

• PROBLEM_ID - The specific problem baked into each Docker image
• HINTS - Hint level for the problem (none or all)

Docker Configuration

The included Dockerfile sets up the complete environment:

• Base system with required tools
• Verilog toolchain (iverilog, verilator)
• Python test framework (pytest, cocotb)
• VNC for GUI testing (if needed)

Best Practices

Task Design

1. Clear Descriptions: Provide detailed, unambiguous task descriptions
2. Focused Scope: Each task should test one concept or skill
3. Realistic Scenarios: Base tasks on real-world debugging/development scenarios
4. Fair Hints: If providing hints, ensure they guide without giving away the solution

Test Design

1. Comprehensive Coverage: Tests should fully validate the requirement
2. Clear Failures: Test failures should clearly indicate what's wrong
3. Minimal Changes: Test patches should only add tests, not modify existing code
4. Isolation: Tests should not depend on external state

Branch Management

1. Clean Baseline: Baseline should be stable and buildable
2. Minimal Test Patch: Only add tests that verify the specific requirement
3. Correct Golden: Golden solution should be minimal and idiomatic