[BUG]: create-eval-dataset --refine cannot match Harbor trial folders to case ids

[mimran-khan]

What happened?

After a live evaluate run, --refine is supposed to load trajectory.json from the latest results tree and attach each trajectory to the matching eval case.

Discovery keys trajectories by the trial directory name:

case_id = trial_dir.name
...
trajectories[case_id] = traj

in generate_dataset.py. Refine then does trajectories.get(case["id"]) in _refine_with_llm.

Collector persists trials under Harbor's physical name, not the eval case id:

return (f"{trial_root_name}__{step_name}" if step_name else trial_root_name), trial_root_name

in collector.py. Harbor trial folders look like tech-writing-001__Lmi47iy (the shape from #70). Case ids look like tech-writing-001.

The collector already has _canonical_case_id / a split on __. Refine does not use it.

Expected: after evaluate, --refine matches tech-writing-001__Lmi47iy to case tech-writing-001 and updates expected_behavior from the trajectory.

Actual: discovery returns { "tech-writing-001__Lmi47iy": traj } and trajectories.get("tech-writing-001") is None. Refine is a no-op even when trajectory.json is sitting on disk.

The unit test plants a folder literally named case-001 (test_generate_dataset_results.py), which is not what collection writes, so the mismatch is not covered.

Reproduction steps

No Docker required. This is the lookup, not the agent run:

import json
from pathlib import Path
from skillevaluator.tier3.generate_dataset import _discover_trajectories
from skillevaluator.tier3.harbor.collector import _canonical_case_id, _persisted_trial_name

root = Path("/tmp/refine-repro")
skill = root / "demo"
skill.mkdir(parents=True, exist_ok=True)
(skill / "SKILL.md").write_text("---\nname: demo\ndescription: Refine lookup check.\n---\n# x\n")

run = root / "results" / "demo" / "20260709_120000"
harbor_folder = "demo-001__Lmi47iy"
trial = run / "claude-code" / "with-skill" / "trials" / harbor_folder
trial.mkdir(parents=True)
traj = {"steps": [{"tool_calls": [{"tool": "Read"}]}]}
(trial / "trajectory.json").write_text(json.dumps(traj))
(run / "run_config.json").write_text("{}")
(run / "result.json").write_text(json.dumps({"run_id": "20260709_120000"}))
(root / "results" / "demo" / "latest").symlink_to("20260709_120000")

found = _discover_trajectories(skill, results_dir=root / "results")
print(list(found))                          # ['demo-001__Lmi47iy']
print(found.get("demo-001"))                # None
print(_persisted_trial_name({"_trial_root_name": harbor_folder, "_step_name": None}))
print(_canonical_case_id(harbor_folder.split("__", 1)[0], {"demo-001"}))  # demo-001

On a real evaluate tree the folder name has the same __ suffix. --refine then prints that it found trajectories, and every case still gets no trajectory available.

SkillEvaluator version or commit

009aa300be7925c7ba75760592baeb941cc29ba8 (0.2.1)

Environment

• macOS 15, arm64
• Python 3.12.2
• uv sync --python 3.12 --all-extras
• Repro above does not start Harbor. Live evaluate folders use the same case-id__suffix naming.

Before submitting

• I agree to follow this project's Code of Conduct
• I have searched the open issues and found no duplicate of this report

Happy to send a PR that keys refine through _canonical_case_id (and updates the unit test to use a Harbor-style folder name) if that would help.