Code QA and Coding-Agent Evaluation Harness

A small harness for evaluating model- or agent-written Python. Each task ships a reference solution, public cases, and hidden edge-case and safety cases, plus a verifier that returns per-case pass/fail and a pass rate. A separate static check inspects candidate source for hallucinated imports and unsafe calls, the way a coding-agent evaluator inspects a patch before trusting its test results. No third-party runtime dependencies (standard library only).

Why hidden and safety cases matter

The point of the harness is to catch code that looks right and passes the obvious cases but is wrong or unsafe on the cases that matter:

• medical_dose - a weight-based dose calculator with a hard safety cap. A naive weight * mg_per_kg passes the public cases but fails the hidden cases because it does not cap the dose and does not reject a negative weight.
• parse_key_values - a naive split("=") passes simple input but drops data when a value itself contains =.
• normalize_whitespace - a solution that only .strip()s passes the trim case but fails on internal runs of whitespace.

What is inside

• codeqa.tasks - the task registry (normalize_whitespace, parse_key_values, medical_dose), each exposing reference and verify(candidate).
• codeqa.verifier - VerifierReport with check_equal and check_raises, returning per-case pass/fail, a pass rate, and JSON output.
• codeqa.agentcheck - AST-based static checks for hallucinated imports and unsafe calls (eval, exec, os.system, subprocess), plus syntax-error reporting.
• codeqa.run - run_all() (smoke test: every reference scores 1.0) and evaluate_candidate(task, func) to score one function.

Quick start

python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"

pytest -q             # run the tests
python -m codeqa.run  # verify every reference solution (all pass_rate == 1.0)

Score a candidate function:

from codeqa.run import evaluate_candidate

def my_dose(weight_kg, mg_per_kg):
    return weight_kg * mg_per_kg  # unsafe: no cap, no validation

print(evaluate_candidate("medical_dose", my_dose))  # cap and validation cases fail

Static-check candidate source:

from codeqa.agentcheck import check_source

print(check_source("import supercalc\n\ndef f(x):\n    return eval(x)\n"))
# -> hallucinated_imports: ['supercalc'], unsafe_calls: ['eval'], clean: False

License

MIT - see LICENSE.