Topic
Packaging the cases as Harbor tasks — the format
Terminal-Bench moved to.
What I'm asking / claiming
The cases already look almost exactly like Harbor tasks, so converting them wouldn't be
a rewrite — the pieces are just in slightly different places. The SDK Docker images
(Zephyr / ESP-IDF / STM32) are already the environment, which is the part that's
usually painful to port. A Harbor verifier is just a tests/test.sh that writes out a
reward, so the L0–L4 checks in checks/*.py can go in as-is, and since the reward is a
float the weighted scoring keeps working. The prompt is the instruction, the reference
solution is the solution, and everything runs in QEMU/native_sim inside the container,
so there's no hardware dependency — which is what Harbor assumes anyway. The usual hard
parts of porting a benchmark are already done here, which is why I think this repo is a
good fit. Happy to put up a PR converting one case (probably a Zephyr native_sim one)
as a template if you're open to it.
Pointers
What would change my mind
If per-layer L0–L4 scores are a headline metric you rely on: Harbor only aggregates a
single reward out of the box, so keeping per-layer scores would need a small custom
metric.py. If that's not worth the maintenance, the case for switching is weaker.
Happy to contribute
I can put up a PR converting one case (probably a Zephyr native_sim one) as a working
template, and port the rest if it looks right.
Topic
Packaging the cases as Harbor tasks — the format
Terminal-Bench moved to.
What I'm asking / claiming
The cases already look almost exactly like Harbor tasks, so converting them wouldn't be
a rewrite — the pieces are just in slightly different places. The SDK Docker images
(Zephyr / ESP-IDF / STM32) are already the environment, which is the part that's
usually painful to port. A Harbor verifier is just a
tests/test.shthat writes out areward, so the L0–L4 checks in
checks/*.pycan go in as-is, and since the reward is afloat the weighted scoring keeps working. The prompt is the instruction, the reference
solution is the solution, and everything runs in QEMU/native_sim inside the container,
so there's no hardware dependency — which is what Harbor assumes anyway. The usual hard
parts of porting a benchmark are already done here, which is why I think this repo is a
good fit. Happy to put up a PR converting one case (probably a Zephyr native_sim one)
as a template if you're open to it.
Pointers
src/embedeval/checks/{static,behavior,negatives}.py— the checks that'd move intotest.shdocs/METHODOLOGY.md— L0–L4 definitionsWhat would change my mind
If per-layer L0–L4 scores are a headline metric you rely on: Harbor only aggregates a
single
rewardout of the box, so keeping per-layer scores would need a small custommetric.py. If that's not worth the maintenance, the case for switching is weaker.Happy to contribute
I can put up a PR converting one case (probably a Zephyr native_sim one) as a working
template, and port the rest if it looks right.