Add a two-tier regression harness for SKILL.md and curricula - #1
Merged
Conversation
Every fix in 1.2.0 was validated by a one-off simulated session with no way to re-check it after a future spec edit. Adds tests/ with a deterministic structural checker (check_progress_schema.py, validates progress-file artifacts against SKILL.md's Progress Rules, no LLM needed) and five fixed behavioral scenarios targeting the exact bug classes found and fixed in 1.2.0 (onboarding/profile location, topic-key derivation and reuse, toolchain-vs-learner-bug detection, hint-streak phase-boundary scoping, last-exercise track completion), plus the reusable agent prompt to run them. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019FNUmdkni7KChUL6s7hFvB
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
There was a problem hiding this comment.
Hi @luxsolari! 👋
Your private repo does not have access to Sourcery.
Please upgrade to continue using Sourcery ✨
Progress Rule 1 read as an absolute gate on every progress-file write, contradicting Rules 5/6/8 which already mandate immediate writes for completed_exercises, topic_confidence/review_due, and hint streaks. Reworded to distinguish the narrated full-file checkpoint from the field-level writes other rules already require. check_progress_schema.py's next_up:null advisory was implemented as warn(label, True, detail) -- warn() only emits WARN on a falsy condition, so this could never print anything but PASS, silently suppressing its own reminder. Added a dedicated note() helper for unconditional advisories. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019FNUmdkni7KChUL6s7hFvB
- tests/fixtures/ + tests/test_check_progress_schema.py: unit-tests the checker itself against 7 fixtures (valid, valid-track-complete, and 5 invalid variants each isolating one violation), since it's plain deterministic code and shouldn't need an expensive Tier 2 run to catch a bug in it -- confirmed by reverting the earlier next_up/warn() fix and watching the new test fail. - .github/workflows/tier1-checks.yml runs those tests plus a plugin manifest sanity check on every push/PR to main. - .gitignore: carve out tests/fixtures/**/.sage-progress.json and .sage-profile.md from the blanket learner-data ignore rule -- those are committed synthetic fixtures, not real learner state. - Pre-release checklist added to tests/README.md and CONTRIBUTING.md: version bumps require running all five Tier 2 scenarios and recording the result in CHANGELOG.md. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019FNUmdkni7KChUL6s7hFvB
Scenario 04 asserted a fresh recalibration offer would fire after P1-comprehension-refactor, but that's Phase 1's second of three exercises, not its last -- no phase transition happens there, so under SKILL.md's actual phase-transition-gated mechanism the assertion could never pass as scripted. Extended the scenario one step further to P1-json-roundtrip (Phase 1's real last exercise) and added an explicit assertion that no offer fires at the mistimed point either, turning the false negative into a check of the phase-transition gating itself. Same live run flagged a secondary wording asymmetry in Axis Re-Calibration: only the low_hint_streak bullet said "at the next phase transition," leaving high_hint_streak's trigger ambiguous. Made both signals explicitly phase-transition-gated. Full Tier 2 results so far (see CHANGELOG): scenarios 01/02/03 passed clean; 05's one non-pass was the check_progress_schema.py bug fixed separately; 04's re-run against this fix is in progress. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019FNUmdkni7KChUL6s7hFvB
Re-ran the corrected scenario end-to-end (fresh scratch project, all 7 steps replayed with real exercises and verify runs): 6/6 PASS, including the two new assertions -- the recalibration offer correctly stays silent at both non-phase-transition threshold-met points and re-fires as a genuinely fresh signal at the actual Phase 1->2 transition. All five Tier 2 scenarios now pass cleanly against the current SKILL.md and curricula/python-basics.md. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019FNUmdkni7KChUL6s7hFvB
Ran the plugin through Claude Code's actual plugin-loading mechanism for the first time (claude -p --plugin-dir, not an agent role-playing the spec): triggered from unscripted natural language, completed Profile Setup, offered the bundled Python Foundations track, and produced a real Phase 0 lesson bridging to the stated Java/JS background. The resulting .sage-profile.md/.sage-progress.json -- written by the real mechanism -- pass check_progress_schema.py clean. claude plugin validate . also passes. Bumped .claude-plugin/plugin.json to 1.2.1 and closed out the CHANGELOG's Unreleased section under that version, since this session's changes (Progress Rule 1 clarification, Axis Re-Calibration wording fix, the tests/ regression harness and CI gate) are ready to ship. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019FNUmdkni7KChUL6s7hFvB
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Every bug fixed in 1.2.0 (execution-based verification, topic-key naming, hint-streak scoping, track-completion handling, learner-profile location) was found and validated by a one-off simulated session, with no way to check it hadn't regressed after a later edit to
SKILL.mdor a curriculum. This adds a repeatable regression harness undertests/:tests/check_progress_schema.py: a deterministic, no-LLM checker that validates a produced.sage-progress.json/.sage-profile.mdagainstSKILL.md's Progress Rules (kebab-case topic keys, mutually-exclusive hint streaks,review_duestaying a subset oftopic_confidence, exercise slug format, profile file location). Fast, cheap, CI-friendly.tests/scenarios/*.md: five fixed, repeatable session scripts (not open-ended exploration), one per bug class closed in 1.2.0 — onboarding/profile location, topic-key derivation and reuse, toolchain-failure-vs-learner-bug detection, hint-streak phase-boundary scoping + decline behavior, and last-exercise track completion.tests/run_scenario_prompt.mdis the reusable agent prompt that plays both Sage and the scripted learner, executes real commands (per Step 6b), and grades against each scenario's assertion checklist.tests/README.mdexplains the two tiers and why Tier 2 is a pre-release gate rather than per-commit CI (each run is a real multi-turn LLM conversation).CONTRIBUTING.mdandCHANGELOG.mdupdated to point maintainers at the harness when editing spec-level behavior.Test plan
check_progress_schema.pysmoke-tested against both a deliberately broken fixture (non-kebab-case key, danglingreview_duereference, non-exclusive streaks — correctly exits 1 with specific findings) and a well-formed one (exits 0).SKILL.md/curricula/python-basics.md, to confirm the harness produces real signal rather than existing only on paper.🤖 Generated with Claude Code
Generated by Claude Code