silentbug

An agent receives a small, runnable training repository that contains exactly one injected defect, or a clean control. The defective code runs to completion, the loss decreases, and the reported metric is plausible. The agent names the defect from a fixed taxonomy, points to a file and line, and returns a unified diff. A program applies the patch out of process, retrains at a fixed seed, and scores six components. No model grades another model.

There are 18 tasks on four CPU-only PyTorch substrates: a tabular MLP, a tiny CNN, a char-level GRU, and a contrastive encoder. Four of the 18 are controls.

Task format

Each agent-visible bundle is data/tasks/<task_id>/:

• prompt.md — the instruction
• schema.json — the submission schema
• repo/ — README.md, train.py, model.py, data.py, metrics.py

Ground truth (labels, bands, reference patches, hidden tests) lives under data/gold/ and is not copied into the agent tree.

task id	substrate	tier
tb-01-zero-grad	tabular	easy
tb-02-overlap	tabular	easy
tb-03-no-eval	vision	easy
tb-04-sched	sequence	easy
tb-05-softmax	sequence	medium
tb-06-sort	tabular	medium
tb-07-bn-pollute	vision	medium
tb-08-loss-sum	tabular	medium
tb-09-clip-before	vision	medium
tb-10-copy-shift	sequence	hard
tb-11-val-aug	vision	hard
tb-12-detach	contrastive	hard
tb-13-test-norm	tabular	hard
tb-14-early-stop	contrastive	hard
tb-15-ctrl-tabular	tabular	control
tb-16-ctrl-vision	vision	control
tb-17-ctrl-sequence	sequence	control
tb-18-ctrl-contrast	contrastive	control

Submission schema

Return one JSON object:

{
  "task_id": "tb-01-zero-grad",
  "defect_label": "missing_zero_grad",
  "localization_file": "train.py",
  "localization_line": 14,
  "patch": "--- a/train.py\n+++ b/train.py\n...",
  "notes": ""
}

defect_label must be one of: missing_zero_grad, train_test_overlap, missing_eval_mode, scheduler_step_frequency, softmax_wrong_axis, feature_label_misalignment, batchnorm_train_mode_leak, loss_reduction_mismatch, clip_before_backward, target_shift_off_by_one, augmentation_on_eval, detach_severed_branch, test_set_normalisation, early_stop_sign_flip, none.

On a control, set defect_label to "none", localisation fields to null, and patch to the empty string.

Scoring

Six independently logged components. total is a weighted sum over non-null components and is never the only number written to disk.

Identity short-circuit. On a defective task, an empty patch scores 0 on every component even though the code would run. On a control, an empty patch with label none scores 1 on every component.

component	defective weight	control weight
patch_applies	0.15	0.10
metric_recovered	0.25	0.20
unit_test_passes	0.20	0.10
defect_localized	0.20	0.20
diff_minimality	0.20	0.10
false_positive	(null)	0.30

false_positive is empty on defective rows, never 0.0. Averaging it over the four controls gives false_positive_rate = 1 - mean(false_positive).

Claim vs demonstration (logged, not extra weights):

• claim_score = defect_localized
• defective demo_score = (0.15 patch + 0.25 metric + 0.20 unit + 0.20 mini) / 0.80
• control demo_score = (0.10 patch + 0.20 metric + 0.10 unit + 0.10 mini) / 0.50

diff_minimality

Normalise a diff line by dropping a trailing newline and rstrip of spaces and tabs. Count only +/- lines, not headers or context.

EXTRA_TOL = 2
MISSING_WEIGHT = 0.5
CONTROL_SCALE = 8.0

extra, missing from the change multisets
if n_ref == 0:
    score = 1 if n_agent == 0 else max(0, 1 - n_agent / CONTROL_SCALE)
else:
    score = max(0, 1 - (max(0, extra - EXTRA_TOL) + MISSING_WEIGHT * missing) / n_ref)

A behaviourally equivalent one-token rewrite (dim=2 vs dim=-1) keeps 0.75 of this component under the specified parser (both - and + lines count). A 120-line file rewrite scores 0.

Inversion note

Some tasks make the reported metric look better than the clean run. A too-good number is outside the hidden band. Do not assume a high score means a clean trainer.

Runtime and hardware

CPU only. CUDA_VISIBLE_DEVICES is cleared by the verifier. torch.set_num_threads(4). Every task must verify in under 10 seconds; the hard train timeout is 8 seconds. After lock, clean walls are about 0.05 s (tabular), 0.33 s (vision), 0.15 s (sequence), 0.22 s (contrastive) on the calibrating machine.

Baselines

• null — always {defect_label: none, patch: ""}. Floor on the 14 defective tasks is 0; suite mean if all 18 are averaged equally is 4/18.
• naive — deterministic AST/grep scanner. Hits a subset of one-token defects; no overlap, sort, scheduler, val-aug, BN, copy-shift, or early-stop rules.
• model_backed — prompt + schema + repo files into an Adapter; garbage completions become the null submission.

python -m harness.runner --baseline null --out results/null.csv
python -m harness.runner --baseline naive --out results/naive.csv
python -m harness.runner --baseline model_backed --adapter echo --out results/echo.csv
python -m harness.runner --summarise results/null.csv results/naive.csv --summarise-out results/RESULTS.md

Results

Committed numbers from python -m harness.runner on the locked task set. The naive scanner has no overlap, sort, scheduler, val-aug, BN, copy-shift, or early-stop rules, and it does not fire on tb-03 because a leftover evaluate() helper still mentions model.eval.

baseline	easy	medium	hard	control	all	false_positive_rate
null	0.000	0.000	0.000	1.000	0.222	0.000
naive	0.245	0.588	0.196	1.000	0.494	0.000

Claim vs demonstration (suite headline):

baseline	tier	claim	demo
null	easy / medium / hard	0.000	0.000
null	control	1.000	1.000
naive	easy	0.225	0.250
naive	medium	0.540	0.600
naive	hard	0.180	0.200
naive	control	1.000	1.000

Inversion subfamily mean (tb-02, tb-10, tb-13): null 0.000, naive 0.000. Other eleven defective tasks: null 0.000, naive 0.445.

Full per-task component tables are in results/RESULTS.md.

Reproducing

python -m pip install -e '.[dev]'
python -m silentbug.generate
python -m pytest -q
python -m silentbug.calibrate --write docs/CALIBRATION.md
python -m harness.runner --baseline null --out results/null.csv

Known limitations

• Bands are locked on a single CPU class; a slower machine may need the 8 s timeout more than the 2–3 s budget.
• Feature-only test-set StandardScaler did not invert accuracy in prototypes; the shipped tb-13 inversion uses the {mean, std, class_prior} bundle, so the leaked statistic includes the test class prior.
• Contrastive Recall@1 has a wide seed range; the lock used 7 seeds, 400 steps, view noise 0.20, and a std cap via the band formula.
• diff_minimality is line-multiset equality after rstrip, not semantic equivalence. A one-token equivalent fix scores 0.75 on that component.
• Controls still contain leftover comments that look like bugs. That is intended, and it will inflate some models' false-positive rate.
• The benchmark measures diagnosis-plus-minimal-patch on four tiny synthetic trainers. It does not measure debugging of real-scale runs.
• Vision 7-seed std is large (~0.04–0.09). Locked bands for vision tasks are correspondingly wide; that is a property of the 12×12 6-class bars, not a hidden second defect.

License

MIT. Cite via CITATION.cff.