fast-agent Benchmark Results

Reproducible agent-evaluation results from running deepseek-v4-pro through fast-agent (evalstate/fast-agent v0.10.10) on a subset of VulcanBench v1 declarative coding tasks.

Result

Task	functional	Verdict
py-topo-sort-cycle	1.0	PASS
py-ttl-cache-expiry	1.0	PASS
py-semver-compare	0.0	FAIL (genuine)

pass@1 = 2/3

Method

• Harness: fast-agent v0.10.10 FastAgent.harness() with a coder agent (shell=True), driving deepseek-v4-pro via the DeepSeek provider.
• Grading: VulcanBench hidden tests copied into the workspace, then run with pytest coverage addopts neutralized (-o addopts=) so fast-agent / VulcanBench's own --cov=harness --cov-fail-under=80 does not corrupt the exit code.
• Artifacts: raw per-candidate score.json / task.json under raw/, and the consolidated fast-agent-deepseek-v4-pro-2026-08.json.

Deep dive

py-topo-sort-cycle — PASS (1.0)

Model produced a min-heap (Kahn's) topological sort with CycleError on cycle detection. All 9 hidden tests pass (5 fail-to-pass + 2 pass-to-pass independently verified in the captured workspace).

py-ttl-cache-expiry — PASS (1.0)

Model fixed TTLCache.get() to check expires_at against the clock and evict expired entries. All 5 hidden tests pass.

py-semver-compare — FAIL (0.0, genuine)

The model left semver/compare.py at raise NotImplementedError and did not implement the spec parser within budget. This is a genuine model limitation, not a harness artifact (the other two tasks the same model solved correctly).

Why two separately-verified benchmark harnesses?

This run doubles as a check on the earlier VulcanBench results and surfaced two harness-side pitfalls (not model failures) that corrupt functional scores:

1. VulcanBench coverage leak. pyproject.toml addopts sets --cov=harness --cov-fail-under=80. When declarative grading runs pytest inside a workspace under the repo root, this leaks in, collects 0% on harness, fails the 80% gate, and marks correct code as FAIL.
2. fast-agent missing hidden-test install. Grading must copy tasks/<id>/tests/ into the workspace before running pytest, or every command fails file not found.

Both corrected here by (a) installing hidden tests and (b) neutralizing coverage.

Reproduce

git clone https://github.com/evalstate/fast-agent.git
cd fast-agent && uv sync
export DEEPSEEK_API_KEY=*** DEEPSEEK_BASE_URL=https://api.deepseek.com
# harness at fast-agent/bench/run_bench.py (from this repo's methodology)
uv run python bench/run_bench.py --model deepseek-v4-pro

Notes

• py-semver-compare is the one task both DeepSeek models (v4-flash and v4-pro) fail genuinely — consistent with the VulcanBench-correction note that it is a genuine unsolved task, not a harness artifact.
• Claude/GPT/R1 were not runnable (no API balance) — the "frontier model would pass semver" claim remains unverified.