Add a two-tier regression harness for SKILL.md and curricula

.claude-plugin/plugin.json

@@ -1,6 +1,6 @@
 {
   "name": "sage-instructor",
-  "version": "1.2.0",
+  "version": "1.2.1",
   "description": "Adaptive programming instructor — structured courses with discovery-first teaching, AskUserQuestion interactions, progress tracking, and pluggable curricula. Powered by the Three Axes Framework.",
   "author": {
     "name": "Lux Solari",

.github/workflows/tier1-checks.yml

@@ -0,0 +1,24 @@
+name: Tier 1 regression checks
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+    branches: [main]
+
+jobs:
+  check-progress-schema:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.x"
+      - name: Compile check
+        run: python3 -m py_compile tests/check_progress_schema.py tests/test_check_progress_schema.py
+      - name: Run checker unit tests against tests/fixtures/
+        run: python3 tests/test_check_progress_schema.py -v
+      - name: Validate plugin structure
+        run: |
+          test -f .claude-plugin/plugin.json
+          python3 -c "import json; json.load(open('.claude-plugin/plugin.json'))"

.gitignore

@@ -24,3 +24,8 @@ CLAUDE.local.md
 # Never commit someone's actual progress/profile into the plugin repo.
 .sage-progress.json
 .sage-profile.md
+# ...except the checked-in, synthetic fixtures the Tier 1 harness tests
+# against (tests/fixtures/*/.sage-progress.json, .sage-profile.md) — those
+# are committed on purpose, no real learner data.
+!tests/fixtures/**/.sage-progress.json
+!tests/fixtures/**/.sage-profile.md

CHANGELOG.md

@@ -1,5 +1,103 @@
 # Changelog
 
+## [1.2.1] — 2026-07-02
+
+### Added
+- **`tests/` regression harness.** Closes the gap where every fix in 1.2.0
+  was validated by a one-off simulated session with no way to re-check it
+  after a future edit. Two tiers:
+  - `tests/check_progress_schema.py` — deterministic, no-LLM structural
+    checker for `.sage-progress.json`/`.sage-profile.md` against the Progress
+    Rules in `SKILL.md` (kebab-case topic keys, mutually-exclusive hint
+    streaks, `review_due` staying a subset of `topic_confidence`, exercise
+    slug format, profile file location). Fast enough for CI on every push.
+  - `tests/scenarios/*.md` — fixed, repeatable session scripts (not
+    open-ended exploration), one per bug class found and fixed in 1.2.0:
+    onboarding/profile location, topic-key derivation and reuse, toolchain
+    failure vs. learner bug, hint-streak scoping across a phase boundary
+    plus decline-resets-streak, and last-exercise track completion.
+    `tests/run_scenario_prompt.md` is the reusable agent prompt that plays
+    both Sage and the scripted learner, executes real commands, and grades
+    against each scenario's assertion checklist. See `tests/README.md`.
+- **`tests/test_check_progress_schema.py` + `tests/fixtures/`.** The
+  checker itself is plain deterministic code, so it now has unit tests
+  against seven fixtures (a valid progress file, a valid track-completion
+  state, and five invalid variants each isolating one violation) instead of
+  relying on an expensive Tier 2 run to notice a bug in it — which is
+  exactly how the `next_up`/`warn()` bug below shipped in the first place.
+  Verified the new test would have caught that exact bug by reverting the
+  fix and confirming it fails.
+- **`.github/workflows/tier1-checks.yml`.** Runs the Tier 1 checker's unit
+  tests and a plugin-manifest sanity check on every push/PR to `main` — Tier
+  1 is now an actual automated gate, not a script someone has to remember to
+  run.
+- **Pre-release checklist in `tests/README.md`/`CONTRIBUTING.md`.** Bumping
+  the plugin version now requires running all five Tier 2 scenarios (not
+  just the ones nearest the change) and recording the result in the
+  CHANGELOG entry, so "the harness was run" is checkable later.
+
+### Fixed
+- **Found by actually running the harness against the live spec (scenarios
+  02, 03, 05 — see `tests/scenarios/`):**
+  - Progress Rule 1 ("Checkpoints are explicit. Write ONLY on `/checkpoint`
+    or learner confirmation") read as an absolute gate on every field write,
+    contradicting Rules 5/6/8 and Step 6b, which already mandate immediate,
+    unconditional writes for `completed_exercises`, `topic_confidence`/
+    `review_due`, and the hint streaks. Reworded to make explicit that Rule 1
+    governs the narrated full-file save, not the field-level writes other
+    rules already require — an interrupted session shouldn't lose those.
+  - `tests/check_progress_schema.py`'s `next_up: null` advisory was
+    implemented as `warn(label, True, detail)` — since `warn()` only emits
+    `WARN` on a falsy condition, this line could never print anything but
+    `PASS`, silently suppressing the "verify track completion manually"
+    reminder in exactly the case it exists to flag. Added a dedicated
+    `note()` helper for unconditional advisories and switched this check to
+    use it.
+  - The Axis Re-Calibration section's two threshold bullets were worded
+    asymmetrically — only the `low_hint_streak` bullet stated "at the next
+    phase transition," leaving it ambiguous whether `high_hint_streak`'s
+    offer could also surface at a plain exercise-complete menu. Made
+    explicit that both signals are checked only at phase-transition points.
+
+### Verified
+- **Full Tier 2 pre-release run (all five scenarios) against this
+  `SKILL.md`/`curricula/python-basics.md`, live via an agent playing both
+  Sage and each scripted learner:**
+  - `01-onboarding-and-profile-location`: 6/6 PASS.
+  - `02-topic-key-consistency`: 5/5 PASS.
+  - `03-toolchain-vs-learner-bug`: 6/6 PASS.
+  - `04-hint-streak-scoping-and-decline`: first run surfaced 4/5 PASS, 1
+    FAIL — but the FAIL was a bug in the *scenario itself* (it asserted a
+    fresh recalibration offer after `P1-comprehension-refactor`, which isn't
+    Phase 1's last exercise, so per SKILL.md's phase-transition-gated
+    mechanism no offer was ever going to fire there). Fixed the scenario to
+    assert at `P1-json-roundtrip` (Phase 1's actual last exercise) instead;
+    also tightened the Axis Re-Calibration wording asymmetry noted above.
+    Re-ran end-to-end (fresh scratch project, all 7 steps replayed with real
+    exercises/verify runs): 6/6 PASS, including the two new assertions that
+    the offer correctly stays silent at both non-transition points and
+    re-fires as a genuinely fresh signal at the real Phase 1→2 transition.
+  - `05-track-completion-handling`: 5/6 PASS, 1 "FAIL" that was the
+    `check_progress_schema.py` bug documented above, not a spec issue.
+  - **Net result: all five Tier 2 scenarios pass cleanly against the current
+    `SKILL.md` and `curricula/python-basics.md`.**
+- **Real-install smoke test.** Every scenario above (and the original 1.2.0
+  bug hunt) validated the spec via an agent reading `SKILL.md` directly and
+  role-playing Sage — never through Claude Code's actual plugin-loading and
+  skill-triggering machinery. Ran `claude -p --plugin-dir` against this repo
+  from a fresh scratch project with no prior profile/progress files: the
+  skill triggered correctly from unscripted natural language ("teach me
+  Python, let's start," no slash command), ran Profile Setup, offered the
+  bundled "Python Foundations" track via Round 0, skipped straight to Phase
+  0 on selection, and produced a real lesson (Steps 1-5, correctly bridging
+  to the stated Java/JS background including the `[]` truthiness
+  JS-vs-Python gotcha). The resulting `.sage-profile.md` and
+  `.sage-progress.json` — written by the real mechanism, not staged — pass
+  `check_progress_schema.py` with 0 failures. (`AskUserQuestion` isn't
+  available in headless `-p` mode; Sage correctly degraded to plain
+  numbered questions instead of erroring.) `claude plugin validate .` also
+  passes.
+
 ## [1.2.0] — 2026-07-01
 
 ### Added

CONTRIBUTING.md

@@ -12,7 +12,16 @@ Open an issue describing: what you expected, what happened, and your Claude Code
 1. Clone the repo
 2. `claude plugin validate .` to verify structure
 3. Test locally: install from the local directory
-4. Submit a PR with a clear description
+4. If you're editing `SKILL.md` or a curriculum's spec-level behavior (progress
+   fields, verification, streaks, onboarding), run the relevant scenario(s) in
+   `tests/scenarios/` — see `tests/README.md`. These are fixed regression
+   scripts targeting bugs that were previously found and fixed; a spec edit
+   that silently breaks one of them is exactly what they're for catching.
+   `python3 tests/test_check_progress_schema.py` (also run in CI) covers the
+   deterministic Tier 1 checks and needs no agent.
+5. Bumping the plugin version? Run `tests/README.md`'s full pre-release
+   checklist (all five Tier 2 scenarios) first.
+6. Submit a PR with a clear description
 
 ## Code of Conduct
 Be kind. Be constructive. We're all here to learn.

skills/sage-instructor/SKILL.md

@@ -177,7 +177,7 @@ Stored as `.sage-progress.json` in the project root.
 ```
 
 ### Progress Rules
-1. **Checkpoints are explicit.** Write ONLY on `/checkpoint` or learner confirmation.
+1. **Checkpoints are explicit — but that governs the narrated full save, not every field.** Sage doesn't announce "checkpoint saved" or write the whole file speculatively except on `/checkpoint`, a phase-transition option that includes it, or explicit learner confirmation. That's distinct from the field-level writes other rules already mandate unconditionally — `completed_exercises` (Rule 5), `topic_confidence`/`review_due` (Rule 6), the hint streaks (Rule 8), and the Step 6b exercise-pointer promotion. Those persist immediately when their triggering event fires; they are not held back waiting for a narrated checkpoint, so an interrupted session doesn't silently lose them.
 2. **Always read before writing.** Load, merge, write.
 3. **hint_count resets** per exercise.
 4. **observations** — pedagogical notes, under 200 chars.
@@ -301,10 +301,10 @@ Before asking, check the recalibration signal below — if it fired, add a 5th o
 
 ### Axis Re-Calibration
 
-Axis levels are declared once at track creation and go stale. `low_hint_streak` / `high_hint_streak` (see Progress File Format) are the signal for when they no longer match reality:
+Axis levels are declared once at track creation and go stale. `low_hint_streak` / `high_hint_streak` (see Progress File Format) are the signal for when they no longer match reality. Both signals are only checked at phase-transition points (see the "Before asking, check the recalibration signal" step in Phase transitions above) — a streak crossing its threshold mid-phase doesn't surface an offer at that exercise's own "what next?" menu, only at the next phase transition, whether or not that transition happens to land in a different phase than where the streak started:
 
 - **`low_hint_streak >= 3`** (three exercises in a row, zero hints) → the declared Mastery is probably too low. Offer a bump at the next phase transition.
-- **`high_hint_streak >= 2`** (two exercises in a row needing 3+ hints) → the declared Mastery or pace is probably too high. Offer to dial back — either Mastery down a level, or just slower pacing within the same level.
+- **`high_hint_streak >= 2`** (two exercises in a row needing 3+ hints) → the declared Mastery or pace is probably too high. Offer to dial back — either Mastery down a level, or just slower pacing within the same level — at the next phase transition.
 - If the learner accepts, write the new level to `axis_overrides` in the progress file (never overwrite the curriculum file itself — the override layers on top, see Progress Rules). Confirm what changed in one sentence.
 - If the learner declines, reset the streak that triggered the offer to 0. Otherwise the same offer resurfaces at every subsequent phase transition until it's acted on — a fresh streak accumulating from here is a new signal and earns a fresh offer, but a stale one shouldn't nag.
 - This only ever surfaces as an offer, never a silent change. The learner decides.

tests/README.md

@@ -0,0 +1,106 @@
+# Regression harness
+
+Sage's "source code" is natural-language instructions (`SKILL.md`, `curricula/*.md`)
+interpreted by an LLM at runtime, not a program you can unit-test in the usual
+sense. This harness makes regression testing repeatable anyway, split into two
+tiers:
+
+## Tier 1 — deterministic structural checks
+
+`check_progress_schema.py` validates a `.sage-progress.json` (and the presence
+of `.sage-profile.md`) against the rules in `SKILL.md`'s Progress Rules
+section — kebab-case topic keys (Rule 10), mutually-exclusive hint streaks
+(Rule 8), `review_due` staying a subset of `topic_confidence`, observations
+length, exercise slug format, profile file location. No LLM involved; runs in
+milliseconds; wired into CI on every push and PR
+(`.github/workflows/tier1-checks.yml`).
+
+```
+python3 tests/check_progress_schema.py <path-to-a-project-dir-with-.sage-progress.json>
+```
+
+This catches schema drift (an artifact malformed relative to the spec) but
+says nothing about whether Sage's *behavior* was pedagogically correct — that
+needs Tier 2.
+
+### Testing the checker itself
+
+`check_progress_schema.py` is plain deterministic code, so it gets its own
+unit tests rather than relying on an expensive Tier 2 run to notice a bug in
+it (that's exactly how the `next_up`/`warn()` bug shipped originally — see
+`CHANGELOG.md`). `test_check_progress_schema.py` runs the checker against the
+fixtures in `fixtures/` (`valid/`, `valid-track-complete/`, and several
+`invalid-*/` directories each isolating one violation) and pins the expected
+exit code and failure message per fixture:
+
+```
+python3 tests/test_check_progress_schema.py -v
+```
+
+This is what `.github/workflows/tier1-checks.yml` runs on every push and PR
+against `main` — Tier 1 is the actual automated gate; Tier 2 remains manual
+(see below).
+
+## Tier 2 — scripted behavioral scenarios
+
+`scenarios/*.md` are fixed, repeatable session scripts — not open-ended
+"go explore" prompts. Each one targets a specific bug class that was
+previously found and fixed (see `CHANGELOG.md` 1.2.0), so re-running them
+after a `SKILL.md` or curriculum edit tells you whether that fix still holds.
+
+Each scenario file has:
+- **Regression target** — the specific bug this guards against, with a
+  `SKILL.md` line reference.
+- **Setup** — the scratch project state to start from.
+- **Script** — a fixed sequence of learner turns/commands. Not improvised.
+- **Assertions** — a checklist split into `[mechanical]` (checkable by
+  `check_progress_schema.py` or by grepping the transcript for a literal) and
+  `[behavioral]` (requires reading the transcript and judging intent).
+
+### Running a scenario
+
+There's no separate test-runner binary — the "test runner" is an agent that
+plays both Sage (per the current `SKILL.md`/`curricula/`) and the scripted
+learner persona, executing real commands where the spec calls for real
+execution (Step 6b `verify`), then grades itself against the scenario's
+assertion checklist. `run_scenario_prompt.md` is the reusable prompt template
+for that agent.
+
+Use it via Claude Code's `Agent` tool (or paste it into a fresh session):
+
+1. Give the agent `run_scenario_prompt.md`'s contents with `{SCENARIO}`
+   substituted for a path under `tests/scenarios/`.
+2. It works in an isolated scratch directory (never inside this repo), reads
+   the relevant `SKILL.md`/curriculum sections directly (not from memory),
+   executes the script turn by turn, and produces a transcript plus real
+   `.sage-progress.json` / `.sage-profile.md` artifacts.
+3. It runs `check_progress_schema.py` against those artifacts for the
+   mechanical assertions and self-grades the behavioral ones, citing the
+   specific transcript line and `SKILL.md` rule for each.
+4. It reports PASS/FAIL per assertion — not a vibe summary.
+
+### Why Tier 2 isn't wired into CI
+
+Each scenario run is a multi-turn LLM conversation that costs real tokens and
+minutes, and the "system under test" and the "test runner" are both LLMs
+interpreting a spec, so results aren't bit-for-bit deterministic the way a
+unit test is. Tier 2 is a **pre-release gate**, run manually, not a
+per-commit CI check.
+
+### Pre-release checklist
+
+Before bumping the plugin version (`.claude-plugin/plugin.json` +
+`CHANGELOG.md`), run **all five** scenarios in `scenarios/` — not just the
+ones nearest whatever you changed. Record the result (pass/fail per
+assertion) in the version's CHANGELOG entry the way 1.2.0 and the harness's
+own rollout did, so "we ran the harness" is checkable later instead of
+trusted on faith. If a scenario fails, that's either a spec bug (fix
+`SKILL.md`/the curriculum) or a scenario gone stale (the curriculum changed
+underneath it — update the scenario's setup/script to match).
+
+### Adding a scenario
+
+When a live-session bug hunt (like the ones behind 1.2.0) finds something new
+and it gets fixed in `SKILL.md`, add a scenario file here targeting it, so the
+next spec edit gets checked against it instead of relying on someone
+remembering to re-discover the same bug.