Skip to content

[methodology] Adopt the Harbor task format #2

Description

@Touka20

Topic

Packaging the cases as Harbor tasks — the format
Terminal-Bench moved to.

What I'm asking / claiming

The cases already look almost exactly like Harbor tasks, so converting them wouldn't be
a rewrite — the pieces are just in slightly different places. The SDK Docker images
(Zephyr / ESP-IDF / STM32) are already the environment, which is the part that's
usually painful to port. A Harbor verifier is just a tests/test.sh that writes out a
reward, so the L0–L4 checks in checks/*.py can go in as-is, and since the reward is a
float the weighted scoring keeps working. The prompt is the instruction, the reference
solution is the solution, and everything runs in QEMU/native_sim inside the container,
so there's no hardware dependency — which is what Harbor assumes anyway. The usual hard
parts of porting a benchmark are already done here, which is why I think this repo is a
good fit. Happy to put up a PR converting one case (probably a Zephyr native_sim one)
as a template if you're open to it.

Pointers

What would change my mind

If per-layer L0–L4 scores are a headline metric you rely on: Harbor only aggregates a
single reward out of the box, so keeping per-layer scores would need a small custom
metric.py. If that's not worth the maintenance, the case for switching is weaker.

Happy to contribute

I can put up a PR converting one case (probably a Zephyr native_sim one) as a working
template, and port the rest if it looks right.

Metadata

Metadata

Assignees

No one assigned

    Labels

    questionFurther information is requested

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions