feat(quiz): confidence rating — pre-reveal prompt, calibration, and agent visibility

[jonasbrami]

What

Roadmap #3 — the confidence slice of the feedback loop, end to end.

In the browser (practice mode): committing a deterministic question (mcq/tf/mermaid) asks "How sure are you? 1–5" before revealing. The revealed card surfaces calibration:

• confident (≥4) but wrong → "Confident but wrong — worth a closer look." (red)
• right but unsure (≤2) → "Right — but you weren't sure. Lock in why." (orange)

Exposed to the host agent (so it can follow up — re-probe confident-but-wrong, drill weak concepts):

• POST /confidence {question_id, value:1-5} — fired on pick (validated; rejects out-of-range / non-int).
• QuizState.confidences: dict[str,int] — persisted in the snapshot beside answers; reset on set_quiz, dropped per-question on replace_question.
• QuestionResult.confidence: int | None — grade_state attaches each rating to the grade tool result.
• get_answers tool returns confidences too (agent can inspect before grading), with a docstring nudge.

Scope / invariants

• Confidence is collected only in practice mode; exam/batch flow untouched.
• Backward compatible: optional throughout — old snapshots load with an empty map, results with confidence=None.
• Honor-system (the answer already ships in /state); the server copy just mirrors the local rating for the agent.

Teach-back (#6) is the remaining feedback-loop slice, as its own PR.

Testing

• Unit: state persistence/reset/drop; QuestionResult.confidence validation (1–5); /confidence record + 422 paths; get_answers + grade carry confidence. Full mcp/engine suite green; ruff + ruff format + mypy --strict clean.
• Real browser (Chrome DevTools): picker gates the reveal; under/over-confidence flags render; and end-to-end rating → POST /confidence → /state.confidences → grade result carries confidence. Screenshots shared.

🤖 Generated with Claude Code

[jonasbrami]

Quiz results

Total: 60%

Question 1 — ❌ 0%

Prompt: Which diagram correctly shows what renderQuestion does with a deterministic, answered question in practice mode — from answer committed to result card visible?

Your answer: C

Question 2 — ✅ 100%

Prompt: Before this PR, shouldReveal(q) returned true for any answered deterministic question in practice mode. After this PR, what is the earliest moment it can return true?

Your answer: After the user clicks one of the 1–5 confidence buttons

Question 3 — ✅ 100%

Prompt: A user rates confidence 4 ('Fairly sure') and their answer is correct. What does calibration(q) return?

Your answer: null

Question 4 — ✅ 100%

Prompt: In exam mode, a user who answers a question will see the confidence prompt before the result is revealed.

Your answer: false

Question 5 — ❌ 0%

Prompt: The comment on line 415 says confidence is an 'honor system (the answer's already in the page)'. What does this mean for the integrity of the calibration feedback, and why did the author still consider it worth shipping?

Your answer: testing

The answer ("testing") does not address any rubric item — it makes no claim about the answer being accessible in the page before the confidence prompt, about calibration data relying on honest self-reporting, or about why the feature is still worth shipping despite that limitation.

---

Results state (used by the CLI)

{
  "version": "1",
  "pr_number": 31,
  "total_score": 60,
  "per_question": [
    {
      "question_id": "q1",
      "correct": false,
      "score": 0,
      "feedback": "",
      "confidence": 4
    },
    {
      "question_id": "q2",
      "correct": true,
      "score": 100,
      "feedback": "",
      "confidence": 2
    },
    {
      "question_id": "q3",
      "correct": true,
      "score": 100,
      "feedback": "",
      "confidence": 3
    },
    {
      "question_id": "q4",
      "correct": true,
      "score": 100,
      "feedback": "",
      "confidence": 2
    },
    {
      "question_id": "q5",
      "correct": false,
      "score": 0,
      "feedback": "The answer (\"testing\") does not address any rubric item — it makes no claim about the answer being accessible in the page before the confidence prompt, about calibration data relying on honest self-reporting, or about why the feature is still worth shipping despite that limitation.",
      "confidence": null
    }
  ]
}