Skip to content

docs: improve skills quality — fix device bug, rewrite descriptions, add evals - #23

Merged
moven0831 merged 4 commits into
mainfrom
improve/skills-quality-evaluation
Feb 27, 2026
Merged

moven0831 merged 4 commits into
mainfrom
improve/skills-quality-evaluation

Conversation

@moven0831

Copy link
Copy Markdown
Collaborator

Summary

  • P1 Fix: commands/device.md now matches skills/mopro-device/SKILL.md safety patterns — removed blocking open -a Simulator, replaced shell & with run_in_background=true note, added boot verification steps for both iOS and Android
  • P2 Descriptions: Rewrote all 5 SKILL.md descriptions from keyword-stuffed lists to intent-focused, slightly pushy descriptions for better triggering accuracy
  • P3 Evals: Created evals/evals.json with 10 test prompts (2-3 per skill + 2 negative cases) for measuring triggering accuracy
  • P4 Cross-references: Added "Related Skills" sections to all 5 skills showing the dependency chain: env → project → app → device → test

Test plan

  • Verify commands/device.md iOS section no longer contains open -a Simulator
  • Verify commands/device.md Android section uses run_in_background=true note instead of shell &
  • Verify all 5 SKILL.md files have updated descriptions in frontmatter
  • Verify all 5 SKILL.md files have "Related Skills" sections with valid skill names
  • Verify evals/evals.json is valid JSON with 10 test cases
  • Verify no SKILL.md exceeds 500 lines

🤖 Generated with Claude Code

moven0831 and others added 3 commits February 27, 2026 15:28
Remove blocking `open -a Simulator` from iOS section, replace shell `&`
with `run_in_background=true` note for Android emulator, and add boot
verification steps for both platforms.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace keyword-stuffed description lists with intent-focused, slightly
pushy descriptions that describe user goals and when to activate, per
skill-creator best practices.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add "Related Skills" sections to all 5 SKILL.md files linking skills in
the dependency chain: env → project → app → device → test. Helps the
model discover related skills without guessing connections.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@moven0831
moven0831 force-pushed the improve/skills-quality-evaluation branch from 6bf3aad to d2f4dc7 Compare February 27, 2026 07:30
@moven0831

Copy link
Copy Markdown
Collaborator Author

@claude review this PR

@claude

claude Bot commented Feb 27, 2026 •

Copy link
Copy Markdown

Claude finished @moven0831's task in 1m 57s —— View job


PR Review

  • Gather context and understand request
  • Read changed files
  • Post review feedback

Overall this is a solid, well-scoped PR. The P1 fix is correct, the description rewrites are clearly better than the keyword-stuffed originals, and the Related Skills cross-references add useful workflow context. A few things worth addressing before merging:


commands/device.md — P1 Fix ✅

The three changes are all correct:

  • Removing open -a Simulator eliminates the blocking call
  • Replacing shell & with a prose note to use run_in_background=true matches SKILL.md line 90 and the build-background.md rule
  • adb wait-for-device + adb shell getprop sys.boot_completed is a better boot-completion check than the old adb devices

One minor inconsistency: the note in commands/device.md (commands/device.md:48, 59) explains the why inline (after each code block), while skills/mopro-device/SKILL.md places the note before the code block (SKILL.md:82, 90). Not a bug, but the ordering is reversed between the two files. Consistent placement would make the pattern easier to scan.


evals/evals.json — P3 Evals ⚠️

The structure is clean and the positive cases are well-chosen, but a few things to consider:

1. No formal pass/fail criterion. The expected_output field is free-text description, not a verifiable condition. If/when automated eval tooling is added, there's nothing machine-readable to assert against. Consider adding a field like "pass_criteria": "skill_triggered" or similar, even if it's only expected_skills != [].

2. Eval #2 over-specifies. The prompt "mopro build is failing with 'NDK not found'" maps to ["mopro-env", "mopro-project"]. NDK not found is a pure environment issue — mopro-env covers it explicitly in its Troubleshooting section. Including mopro-project as required here may cause false negatives if only mopro-env triggers, which would be the correct single-skill response.

3. Negative cases are too easy. Both negative cases ("Build a REST API", "Solidity smart contract") are obviously unrelated to ZK/mopro. A harder negative that could realistically confuse the classifier would better exercise the boundary:

{ "prompt": "Help me generate a ZK proof using snarkjs", "expected_skills": [] }

4. skill_name field is generic. "mopro-skills" doesn't identify a specific skill — if multiple eval files exist later, this could be ambiguous. Consider "all-skills" or just removing it. Fix this →


SKILL.md Descriptions — P2 ✅

All five rewrites are improvements. The intent-focused language with embedded triggering hints ("Use whenever...", "Also use when...") is the right pattern. Two specific notes:

mopro-env description (skills/mopro-env/SKILL.md:6-7):

Also use proactively before any mopro build if the user hasn't confirmed their environment is set up

This could cause mopro-env to trigger on every mopro-project interaction. The word "proactively" risks noisy co-triggering. Consider scoping it:

Also suggest if the user is starting fresh or has not mentioned their environment

mopro-device description (skills/mopro-device/SKILL.md:7-8):

Also use when a user says "run my app" or "test on device" even if they don't mention simulators explicitly.

This is a smart catch — the implicit "run my app" case is a real gap in keyword-based triggering. Good addition.


Related Skills — P4 ✅ with note

The dependency chain env → project → app → device → test is well-documented. Note that mopro-app ↔ mopro-device and mopro-project ↔ mopro-test are bidirectional references — each references the other. This is fine for documentation, but if an agent treats these as automatic follow-up triggers it could chain indefinitely. The phrasing (e.g., "For running the app on simulators...") frames them as optional next steps rather than automatic triggers, which is the right approach. Worth keeping as-is.


Test Plan Checklist Verification

All items can be confirmed from reading the diff:

  • ✅ open -a Simulator removed from commands/device.md
  • ✅ Android section uses run_in_background=true note instead of &
  • ✅ All 5 SKILL.md files have updated frontmatter descriptions
  • ✅ All 5 SKILL.md files have "Related Skills" sections with valid skill names
  • ✅ evals/evals.json is valid JSON with exactly 10 test cases
  • ✅ No SKILL.md appears close to 500 lines (longest is mopro-project at ~165 lines)

Summary: The P1 fix and description rewrites are ready to merge. The main items to address before merging are the mopro-env "proactively" language that could cause over-triggering, and the weakness of the negative eval cases. The other notes are minor polish suggestions.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@moven0831
moven0831 force-pushed the improve/skills-quality-evaluation branch from d2f4dc7 to edc0738 Compare February 27, 2026 07:39
@moven0831
moven0831 merged commit fefa4fb into main Feb 27, 2026
6 checks passed
@moven0831
moven0831 deleted the improve/skills-quality-evaluation branch February 27, 2026 07:40
This was referenced Feb 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant