What happened?
create-eval-dataset --full --no-llm (and the LLM-fallback template) builds bucket 4 as a negative control with expected_skill: null, then asks:
What does the {name} skill do and what are its capabilities?
That is generate_dataset.py. expected_behavior also requires "The agent's response accurately describes what {name} does".
The LLM prompt for the same bucket says the opposite:
4. Negative - should NOT trigger this skill (unrelated question)
in generate_dataset.py.
Tier 3 scores expected_skill: null as "do not read or apply this skill". A competent agent answering "what does this skill do?" will open SKILL.md, then fail the negative-control checks.
Expected: the template negative case is an unrelated question that should not load the skill, matching the LLM prompt and the scorer.
Actual: --no-llm writes a prompt that names the skill and asks for its capabilities, with expected_skill: null.
This is not #57. That issue was about LLM insights misclassifying a correctly handled negative control. This is the dataset generator writing a self-contradictory negative case.
Reproduction steps
tmp=$(mktemp -d)
mkdir -p "$tmp/demo"
cat > "$tmp/demo/SKILL.md" <<'EOF'
---
name: demo
description: A small skill used only to show dataset generation.
---
# Demo
EOF
skillevaluator create-eval-dataset "$tmp/demo" --full --no-llm --force
cat "$tmp/demo/evals/evals.json"
Observed last case:
id: demo-neg-001prompt: What does the demo skill do and what are its capabilities?expected_skill: null
SkillEvaluator version or commit
009aa300be7925c7ba75760592baeb941cc29ba8 (0.2.1)
Environment
- macOS 15, arm64
- Python 3.12.2
uv sync --python 3.12 --all-extras- No API key (
--no-llm)
Before submitting
Happy to send a PR that makes the template negative case an unrelated question (and keeps expected_skill null) so it matches the LLM prompt and the scorer.
What happened?
create-eval-dataset --full --no-llm(and the LLM-fallback template) builds bucket 4 as a negative control withexpected_skill: null, then asks:That is
generate_dataset.py.expected_behavioralso requires "The agent's response accurately describes what {name} does".The LLM prompt for the same bucket says the opposite:
in
generate_dataset.py.Tier 3 scores
expected_skill: nullas "do not read or apply this skill". A competent agent answering "what does this skill do?" will openSKILL.md, then fail the negative-control checks.Expected: the template negative case is an unrelated question that should not load the skill, matching the LLM prompt and the scorer.
Actual:
--no-llmwrites a prompt that names the skill and asks for its capabilities, withexpected_skill: null.This is not #57. That issue was about LLM insights misclassifying a correctly handled negative control. This is the dataset generator writing a self-contradictory negative case.
Reproduction steps
Observed last case:
id:demo-neg-001prompt:What does the demo skill do and what are its capabilities?expected_skill:nullSkillEvaluator version or commit
009aa300be7925c7ba75760592baeb941cc29ba8(0.2.1)Environment
uv sync --python 3.12 --all-extras--no-llm)Before submitting
Happy to send a PR that makes the template negative case an unrelated question (and keeps
expected_skillnull) so it matches the LLM prompt and the scorer.