面向 Qwen3.8-27B 及其他代码/工具 Agent 的公开测试集，重点评估代码检索、长程轨迹、上下文管理、提示注入韧性、运维诊断、跨领域工具调用和多轮角色扮演。仓库同时提供原创可物化代码树，以及按固定上游提交获取真实项目缺陷的注册表。
| Directory | Public tasks | Description |
|---|---|---|
main-benchmark |
152 | 30 simple code, 50 complex code, 20 long-horizon code, 20 math, 20 philosophy, 12 context/tool tasks |
domain-supplement |
100 | 50% coding, 20% operations/SRE, 30% other domains |
roleplay-benchmark |
40 | Persona fidelity, memory, hierarchy/injection, professional boundaries and tool-driven roleplay |
real-project-benchmarks |
24 + 16 sources | Public calibration tasks and a pinned, license-aware registry of real/cold upstream benchmarks |
reasoning-frontier |
60 | 20 math, 20 philosophy and 20 cross-domain auditable-reasoning tasks |
The public release contains 150 materialized synthetic coding repositories and their public tests, plus 24 on-demand real-project task specifications. Private semantic evaluators, hidden contracts, private spreadsheets, gold patches and answer-bearing fixture manifests are deliberately excluded to reduce benchmark leakage.
- Select a task from a
tasks_public.jsonlfile. - Copy only the matching
source_trees/<task-id>/repointo the agent workspace. - Deliver only the public prompt and allowed tools.
- Run
node run_public.mjsinside the repository for the public seed contract. - Keep your own hidden evaluator outside the model context.
The source trees under main-benchmark and domain-supplement are synthetic benchmark repositories. real-project-benchmarks only stores metadata and materializers; upstream source remains governed by its own license and is not mirrored here. Report model, quantization, serving backend, chat template, tool parser, context size, sampling settings, tool versions and random seeds with every result. Do not train on holdout tasks or publish hidden evaluators.