NetlistBench

A structure-verified benchmark for evaluating whether LLMs can reliably recognize and manipulate SPICE netlists as structured circuit representations. Model outputs are scored by a deterministic, structure-aware oracle that compares circuits through a canonical intermediate representation (IR) rather than by surface text.

This is the public benchmark release. It contains the complete evaluation artifacts needed to score the NetlistBench task suite locally.

---

What is included

benchmark/Cases/
  netbench_v2/MANIFEST.json        # canonical benchmark definition (24 families, 2342 cases)
  v2_edit_raw/                     # manipulation + equivalence-judgment cases
    <family>/cases/<case_id>/      #   case.json, source.sp, expected.sp
  v2_recognition_raw/              # recognition cases
    {flat,subckt}/cases/<case_id>/ #   case.json (with inline `expected`), spice.sp, ir.json, pyspice.py
scripts/                           # oracle, runners, and the v2 case-generation pipeline
prompts/prompts_v2_inspection.md   # the exact system+user prompt shown to models, per family
example_run/deepseek_demo/         # pre-generated sample outputs + oracle verdicts (no API key needed to inspect)
requirements.txt
LICENSE                         # code/data license map
DATA_LICENSE.md                 # CC-BY-4.0 notice for data and prompts

Want to see actual model outputs without running anything? example_run/deepseek_demo/ contains a pre-generated representative slice (all 17 edit families + all 7 recognition tasks, 2 cases each) produced with deepseek-chat, alongside the oracle's verdict for every output. See example_run/deepseek_demo/README.md for the layout and the exact commands that produced it.

Case totals (match the paper exactly):

Modality	Cases	Families
Manipulation (edit)	1542	16
Recognition	800	8 (7 JSON extraction + equivalence judgment)
Total	2342	24

The equivalence-judgment family (equivalence_judgment_mixed, 100 cases) is stored physically under v2_edit_raw/ but belongs to the recognition modality in the reported results; MANIFEST.json records the authoritative grouping and per-family case counts.

---

Install

python3 -m pip install -r requirements.txt    # Python 3.10+

The oracle and runners use only the Python standard library plus networkx (equivalence-label audit) and numpy (recognition-case generation).

---

Quick start: score an output with the oracle

Run all commands from the repository root. The oracle reduces every output to a binary pass/fail by comparing canonical IRs.

Manipulation (SPICE):

export PYTHONPATH="$PWD/scripts"
C=benchmark/Cases/v2_edit_raw/device_replace/cases/medium_device_replace_077

# The reference target passes:
python3 scripts/eval_ir_match.py "$C/case.json" "$C/expected.sp" --output-format spice
# -> "pass": true

# The unedited source fails (the requested edit was not applied):
python3 scripts/eval_ir_match.py "$C/case.json" "$C/source.sp" --output-format spice
# -> "pass": false

Recognition — verify that every gold answer is accepted by the oracle:

export PYTHONPATH="$PWD/scripts"
python3 scripts/eval_recognition.py --self-check benchmark/Cases/v2_recognition_raw/flat/cases
python3 scripts/eval_recognition.py --self-check benchmark/Cases/v2_recognition_raw/subckt/cases
# -> [self-check] 500/500 pass   /   200/200 pass

---

Reproducing the leaderboard

1. Query a model. Runners read each case.json, build the prompt, and write one output file per case. Providers are auto-detected from the model name and API keys are read from the environment (e.g. ANTHROPIC_API_KEY, OPENAI_API_KEY, GEMINI_API_KEY, DEEPSEEK_API_KEY, DASHSCOPE_API_KEY). Edit/judgment families use scripts/run_all_families.py; recognition uses scripts/run_recognition_on_cases.py.
2. Score. Edit/judgment outputs are scored with scripts/run_family_reports.py (which dispatches to eval_ir_match.py / eval_equivalence_label.py); recognition with scripts/eval_recognition.py.

Decoding is deterministic (temperature 0, single shot). Because cases are generated from a fixed seed and scored by a fixed oracle version, the pipeline is reproducible.

Regenerating cases (optional)

This release ships the generators for the 24 benchmark families only; generators for unrelated or experimental families are not included. The original source corpora (flat netlists from AnalogGenie and hierarchical netlists from ALIGN) are not redistributed, so a user cannot reconstruct every case from scratch using this repository alone. The shipped benchmark is complete for evaluation: scoring uses the included source.sp, expected.sp, spice.sp, and case.json files. The source_path and source_artifact.path fields are provenance metadata.

License and citation

Code in scripts/ is licensed under Apache-2.0. Benchmark cases, prompts, and pre-generated example outputs are licensed under CC-BY-4.0; see LICENSE and DATA_LICENSE.md for the precise scope. Citation metadata is provided in CITATION.cff.

Security

Never commit API keys. The runners read provider credentials exclusively from environment variables. See SECURITY.md for reporting guidance.

---

The oracle in one paragraph

Each model output and the reference target are parsed into a canonical IR that lists every device by instance name with its kind, ordered terminal nodes, and parameters, plus top-level directives and (for hierarchical circuits) each subcircuit's port interface and internal devices. An output passes only if its IR matches the reference under semantics-preserving normalizations: identical device sets, identical terminal–node bindings, parameters equal up to numeric normalization (1k == 1000), identical directives; symmetric two-terminal passives (R/C/L) are compared with unordered terminals. The same comparator scores SPICE, PySpice, and IR outputs through one interface.