Install with python -m pip install -e ".[dev]", then run pytest -q,
ruff check ., and ruff format --check .. Install Node.js to exercise the
independent JavaScript support policy; that test is skipped if Node is absent.
CI explicitly provisions Node. See task authoring for
the current built-in registry and evidence contract.
The workflow guide covers readiness checks and comparisons. Use new output paths for manual runs so earlier evidence remains available.
A useful contribution makes an evaluation claim more testable. For a new task or grader change, include:
- The observable contract and task version.
- A known-good implementation independent of the oracle's implementation.
- Plausible defective implementations and the specific cases that detect them.
- Reproduction commands, seeds, runtime limits, and data provenance.
- An explicit description of behaviors not tested.
Changing the semantics of requests, scoring weights, or case generation requires a task version change. Do not import evaluation-only benchmark data into training tasks. Cite upstream sources and respect their licenses. Public randomized seeds are not sufficient evidence of decontamination.
The current tasks are purpose-built and small. Contributions toward harder tasks should include human calibration and evidence that frontier agents do not trivially saturate them. New cloud integrations should be validated separately; do not label an adapter as supported solely because it emits a similar JSON shape.