Cmf submission - #4
Open
MargoBenedichuk wants to merge 15 commits into
Open
MargoBenedichuk wants to merge 15 commits into
MargoBenedichuk wants to merge 15 commits into
Conversation
Repo structure, role assignments, and TODO-stub files for all 6 roles (course design, mentor/student prompts, orchestration, docs).
# Conflicts: # .gitignore
All 10 lessons written out (explanation -> verification question -> transfer scenario), each anchored on a non-standard canary term. answer_key.json backs each lesson with the canary fact, its definition, and checkable expected-application points; bluff lessons (3, 6, 9) get an extra note on what a bluffed answer tends to look like.
Turn the scaffold into a runnable simulation on top of the existing course design: - Mentor and student system prompts, with rationale and versioned snapshots under prompts/*/versions. - Relay loop with two mirrored contexts, driven per lesson through fixed phases (open -> apply -> probe -> forced gate). Deterministic seeded practice injection; bluff lessons leave the practice log empty so the student has nothing real to report. - Ledger, practice log, seeded practice simulator, advance-decision gate, and a deterministic precision/recall judge. - Backend is env-configurable (OPENAI_BASE_URL / MODEL) for any OpenAI-compatible endpoint. - Offline test suite: a fake client drives the whole loop with no API key and enforces OpenAI tool-response ordering. RETRY reopens a lesson for a bounded extra probe; the forced gate answers every tool call so the reused context stays valid. Each run writes a transcript, meta, and memory snapshot under courses/<course>/logs/.
…gs. Added: Per-role models, empty-response guard + reasoning-off student, richer practice log, adaptive probing, per-lesson context isolation + course canon, prompt calibration. See CHANGELOG.md. Result: precision 1.0, recall 0.667.
Review of mvp-2-add-more (main...31b8ddf) surfaced two correctness bugs that undercut the branch's own precision claim: the forced-gate prose fallback matched BLUFF_SUSPECTED before PASS regardless of negation ("not a BLUFF_SUSPECTED case, clearly a PASS" misread as a bluff), and an unresolvable prose response silently defaulted to BLUFF_SUSPECTED instead of failing loudly. Both now require an unambiguous whole-word match and raise instead of guessing. Also: STUDENT_EXTRA_BODY's OpenRouter-only `reasoning` field was sent unconditionally, which 400s against the default (plain OpenAI) endpoint from .env.example's own quickstart — now gated on OPENAI_BASE_URL being set. mentor_probe() no longer keeps looping for a text message after advance_decision already fired mid-turn. Dropped the stale RETRY value from the ledger_write tool schema (removed everywhere else). Added a real .gitignore pattern for future run_* log dirs. 17 pre-existing tests still pass; added 2 covering the fallback fix. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ole 4 code Memory: mentor_ledger.json now persists across separate relay.run() invocations for the same student by default (matches what README/PLAN.md already claimed), instead of being wiped every run — a student retaking the course should have prior weak_spots recalled. `--keep-memory` is replaced by `--reset` (opt-in wipe, e.g. for QA comparing seeds cleanly). Added an optional `reflection` field to advance_decision/ledger_write — a one-sentence self-check separate from `reason`. Documented the episodic (per-lesson ledger entries) vs semantic (weak_spots_summary distillation) split that was already implicit in the code. Also fixes the weak_spots="" vs None ambiguity: the terminal ledger write after a verdict is finalized now always writes the current weak_spots list (even empty), instead of silently leaving a stale value from an earlier retry in place. Cleanup (deferred from the earlier review pass): extracted load_json/ save_json into config.py and pointed mentor_ledger.py, student_ practice_log.py, and relay.py's meta/snapshot writes at them instead of three hand-rolled copies; merged mentor()/mentor_probe() into one helper; collapsed _forced_gate's duplicated client.chat.completions. create() call; AdvanceDecision.__init__ now calls reset(); MENTOR_ PHASE_TOOLS filters by tool name instead of a positional slice that would silently drift if MENTOR_TOOLS is reordered; transcript.txt is now appended incrementally per lesson instead of rewritten in full each time; dropped a couple of comments that only restated the next line; renamed the `ak` parameter to `answer_key_entry`. 23 tests passing (19 previous + 4 new: reflection roundtrip, explicit empty weak_spots, MENTOR_PHASE_TOOLS filtering, cross-run persistence + --reset). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Previous version ran relay.run() once and checked verdicts/seed — never compared two runs, so it couldn't catch nondeterminism at all. Now runs twice with reset_memory=True (needed since the ledger persists across runs by default now) and diffs the transcripts byte-for-byte. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
QA: checklist run against all 12 committed logs. run_012 is the only clean 10-lesson run (precision 1.0, recall 0.667) and passes every README checklist item; runs 003/005/007/008 chart the mentor's calibration arc (over-flagging before the evidence bar was tightened), 002/009/011 crashed pre-meta.json (predate the forced-gate fallback fix on mvp-2-fixes, left as-is). Editor: all 5 sections filled — mentor/student prompts, tools + why, run_012's full transcript, and a reflection covering the one confirmed miss (lesson 3: student fabricated consistent-enough practice detail to pass), prompt version history, and the CHANGELOG's live-validation pass. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ti-agent-simulation template
Standalone submission document (Markdown source + rendered PDF, 26 pages): assignment recap, architecture (mirrored contexts, fixed-phase gate, bluff-detection levers), mentor/student prompts, tools table, run_012's full transcript (tool-call JSON dumps stripped for readability), and the reflection. Built with pandoc (md -> html) + headless Chrome (html -> pdf); no LaTeX toolchain available in this environment. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Previous version (26 pages) inlined the full prompts and full best-run transcript, which is redundant with docs/final_submission.md and the repo files themselves. This version is a summary — architecture, tools, result numbers, reflection — with pointers to the full-detail files (prompts, transcript, QA report, final_submission.md) instead of inlining them. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.