diff --git a/CHANGELOG.md b/CHANGELOG.md index e1d1d81..a2ab9c5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,13 +12,12 @@ All notable changes to lectern are documented here. - **Sanitization lint (`lectern.feedback_sanitize`).** Deterministic guard that withholds any student-facing comment leaking internal jargon (triage verdicts, honor-gate, advisory framing) or another student's name. Deliberately excludes crypto-colliding words (`oracle`, `digest`) so legitimate feedback isn't censored. ### Changed +- **`reg-exam-build` now emits the rich per-question GRADING_NOTE.md** that `reg-gradescope-stats` consumes — closing the "Known gap" below. The build previously wrote a summary table (`| Q | Name | Pts | Type | Answer | Rubric |`) that only carried the answer *letter*, so it couldn't drive per-option ISA grading or round-trip to per-distractor item analysis. `exam_pack.parse_outline_from_tex` now parses each option's text + correctness from the `.tex` (`\correctchoice`/`\wrongchoice`, balanced-brace aware, with a `_detex` LaTeX→text pass), and `emit_grading_note` renders the canonical form: `####
·Q · · · ` headings, a per-option `| Pts | Key | Rubric item |` table keyed `·Q·` (a/b/c/d · T/F · true/false · b1.. ), a "No answer / multiple marks" row, `*Scoring:*` captions on fib/code, and a `` Post-exam-statistics section. Legacy inline `\textsc{T~/~F.}` **code** items (no `\correctchoice` enumerate) synthesize their True/False rows from the answer so keys still round-trip. Regression-tested (`test_emit_grading_note_structure`, `test_build_options_legacy_inline_code`); CECS 378 Su26 Exam 2 + Final grading notes regenerated and answer-cross-checked against `_outline.csv`. +- **TF/code outline answers normalized to the verdict word.** `parse_outline_from_tex` now emits `True`/`False` (never a letter) in `_outline.csv` for `tf`/`code` items regardless of whether the source reveal reads the house-standard `\textbf{Answer:} True` or a deviating `\textbf{Answer:} a (True)` / `b (False) --- …`. Keeps the outline consistent across exams (surfaced by the Su26 Final, which authored the letter-prefixed form). Docs reconciled: `docs/design/exam-tex-format.md` and the vault `exam-tex-doctrine` now both state the bare-word house standard + the defensive normalization. Tested (`test_parse_outline_normalizes_letter_prefixed_tf_reveal`). - **`reg-lab-digest` results now carry a `student_comment`** — a sanitized, student-facing comment alongside the internal terse `comment`. One grading pass yields both; `merge` runs the sanitize lint and withholds the student comment on low-confidence/abstain or any lint hit. Additive, backward-compatible schema change to the Layer-2 contract (the grader-prompt doc documents the new field). - **True/False exam questions now use stacked `(a) True / (b) False` choices** — the house standard documented in `references/reference_exam.tex` and `docs/design/exam-tex-format.md`. The previous inline `\textsc{T~/~F.}` form listed no answer options on their own lines, so Gradescope's region detection could not find them. The `\textsc{T~/~F.}` label and inline `Answer:` reveal are retained, so questions stay typed `tf` and `parse_outline_from_tex` still emits `True`/`False` in `_outline.csv` — no code change, purely an authoring-convention fix. - **Refreshed the `examples/cecs-378-demo` worked example** to cover the current command surface. The demo exam gains a stacked-T/F section and a `gradescope: region` build (emitting `gradescope/` + `GRADING_NOTE.md`), and the README adds runnable stages for `reg-syllabus` (stamp + build), `reg-qbank` (validate + emit), `reg-exam-readinglist` (Lectern→Scriptorium seam), and `reg-gradescope-stats` (item analysis), plus a documented "requires live infrastructure" section for the Classroom / ISA-publish / triage / term-finalize verbs. -### Known gaps -- **`reg-exam-build` GRADING_NOTE.md ≠ `reg-gradescope-stats` input format.** The build emits a summary-table grading note (`| Q | Name | Pts | Type | Answer | Rubric |`); `reg-gradescope-stats` parses the richer per-question form (`#### ·Q · … · ` + `| Pts | Key | Rubric item |`). The demo's gradescope-stats stage uses a hand-authored stats-compatible note; bridging the two is a tracked follow-up. - ### Security - **Vendored `slugify`, dropped the `vaultkit` dependency** — the name `vaultkit` on PyPI is an unrelated third-party SDK, so depending on it was a dependency-confusion risk. The one helper used (`slugify`) is now inlined in `lectern/_text.py`, making the public distribution self-contained with no private/ambiguous dependency. diff --git a/docs/design/exam-tex-format.md b/docs/design/exam-tex-format.md index 47cfbf7..a7531db 100644 --- a/docs/design/exam-tex-format.md +++ b/docs/design/exam-tex-format.md @@ -221,10 +221,15 @@ region detection only finds answer choices that sit on their own lines; an inlin ``` Wrap the correct option in `\correctchoice` (green in the key) and the other in -`\wrongchoice`. Keep the inline `\textbf{Answer:} True`/`False` reveal — that line, -not the choice list, is what `parse_outline_from_tex` reads for the `_outline.csv` -answer column, so it stays the word `True`/`False` (not a letter). Directions -should tell students to "circle the correct option, (a) True or (b) False." +`\wrongchoice`. Keep the inline `\textbf{Answer:} True`/`False` reveal — the bare +verdict **word**, not a letter — as the house standard. That line, not the choice +list, is what `parse_outline_from_tex` reads for the `_outline.csv` answer column. +The parser **normalizes defensively**: a deviating source that writes the +letter-prefixed `\textbf{Answer:} a (True)` (or `b (False) --- explanation`) still +yields the word `True`/`False` in `_outline.csv`, never the letter — so the outline +stays consistent across exams regardless of reveal style. The same holds for `code` +verdict items. Directions should tell students to "circle the correct option, +(a) True or (b) False." --- diff --git a/lectern/exam_pack.py b/lectern/exam_pack.py index fa55c08..9af725e 100644 --- a/lectern/exam_pack.py +++ b/lectern/exam_pack.py @@ -14,7 +14,7 @@ import shutil import subprocess import sys -from dataclasses import dataclass +from dataclasses import dataclass, field from pathlib import Path import yaml @@ -60,6 +60,10 @@ class OutlineRow: answer: str name: str = "" rubric: str = "" + # Per-option breakdown for the Gradescope grading note. Each entry is + # (slot, item_text, is_correct): slot is the rubric key suffix + # (a/b/c/d for mc, T/F for tf, true/false for code, b1.. for fib). + options: list = field(default_factory=list) @dataclass @@ -201,6 +205,71 @@ def _parse_annotations(tex_content: str) -> dict[int, tuple[str, str]]: return out +_CHOICE_RE = re.compile(r"\\(correct|wrong)choice\{") +_DETEX_WRAP = re.compile(r"\\(?:texttt|textbf|textit|emph|textsc|textrm|text|mathrm)\{") + + +def _balanced(s: str, start: int) -> tuple[str, int]: + """Given `start` just past an opening brace, return (inner_text, index_past_close).""" + depth, j = 1, start + while j < len(s) and depth: + if s[j] == "{": + depth += 1 + elif s[j] == "}": + depth -= 1 + j += 1 + return s[start:j - 1], j + + +def _detex(s: str) -> str: + """Best-effort LaTeX -> plain text for grading-note rubric-item display.""" + prev = None + while prev != s: # unwrap font/markup macros, balanced + prev = s + m = _DETEX_WRAP.search(s) + if m: + inner, after = _balanced(s, m.end()) + s = s[:m.start()] + inner + s[after:] + s = s.replace("$^\\wedge$", "^").replace("\\^{}", "^") + s = re.sub(r"\$([^$]*)\$", r"\1", s) # drop inline-math delimiters + for a, b in (("\\textasciitilde", "~"), ("\\textbackslash", "\\"), + ("\\%", "%"), ("\\_", "_"), ("\\&", "&"), ("\\#", "#"), + ("\\$", "$"), ("\\{", "{"), ("\\}", "}"), ("\\,", " "), + ("\\ ", " "), ("\\textperiodcentered", "·"), + ("---", "—"), ("--", "–"), ("~", " ")): + s = s.replace(a, b) + s = re.sub(r"\\[a-zA-Z]+", "", s) # strip leftover control words + return re.sub(r"\s+", " ", s).strip() + + +def _build_options(block: str, qtype: str, answer: str) -> list: + """[(slot, item_text, is_correct)] feeding the per-option grading-note table.""" + if qtype == "fib": + blanks = [b.strip() for b in answer.split(";") if b.strip()] + return [(f"b{i + 1}", f'blank {i + 1} = "{_detex(v)}"', True) + for i, v in enumerate(blanks)] + choices = [] # mc / tf / code — choices in doc order + for m in _CHOICE_RE.finditer(block): + text, _ = _balanced(block, m.end()) + choices.append((m.group(1) == "correct", _detex(text))) + if not choices and qtype in ("tf", "code"): + # Legacy inline "T / F." items (no \correctchoice enumerate) — synthesize + # the two verdict rows from the answer so the key still round-trips. + ans_true = answer.strip().lower().startswith("t") + choices = [(ans_true, "True"), (not ans_true, "False")] + opts = [] + for i, (is_correct, text) in enumerate(choices): + if qtype == "tf": + slot, item = (text[:1].upper() or "?"), text # True->T False->F + elif qtype == "code": + slot, item = text.lower(), text # true / false + else: + slot = chr(ord("a") + i) # mc -> a/b/c/d + item = f"({slot}) {text}" + opts.append((slot, item, is_correct)) + return opts + + def parse_outline_from_tex(tex_content: str) -> list[OutlineRow]: body = tex_content.split("\\begin{document}", 1)[-1] # Split at every \item, then REGROUP: a question starts at a fragment that @@ -227,8 +296,21 @@ def parse_outline_from_tex(tex_content: str) -> list[OutlineRow]: qtype = _classify(block) ans_m = _ANS_RE.search(block) raw = ans_m.group(1).strip() if ans_m else "" - # type-aware: FIB keeps all ;-joined blanks; others take first token - answer = raw if qtype == "fib" else (raw.split()[0].rstrip(".") if raw else "") + # type-aware normalization of the outline answer: + # fib -> keep all ;-joined blanks + # tf/code -> the verdict WORD True/False, never a letter, regardless of + # whether the reveal reads `True` or the letter-prefixed + # `a (True)` form (house standard is the bare word; this keeps + # `_outline.csv` consistent if a source deviates) + # mc -> the choice letter (first token) + if qtype == "fib": + answer = raw + elif qtype in ("tf", "code"): + low = raw.lower() + answer = "True" if "true" in low else "False" if "false" in low else ( + raw.split()[0].rstrip(".") if raw else "") + else: + answer = raw.split()[0].rstrip(".") if raw else "" name, rubric = annotations.get(q, ("", "")) if not name: raise SystemExit(f"exam_pack: question {q} is missing a '% name:' annotation") @@ -238,8 +320,10 @@ def parse_outline_from_tex(tex_content: str) -> list[OutlineRow]: f"exam_pack: missing rubric annotation on {qtype} question {q}" ) rubric = f"Correct = {answer} ({int(m_pts.group(1))} pts, all-or-nothing)." + options = _build_options(block, qtype, answer) rows.append(OutlineRow(q_num=q, points=int(m_pts.group(1)), type=qtype, - answer=answer, name=name, rubric=rubric)) + answer=answer, name=name, rubric=rubric, + options=options)) return rows @@ -314,28 +398,46 @@ def _course_tag(course: str) -> str: return course.strip().lower().replace(" ", "-") -def _grading_note_table(rows) -> str: - head = "| Q | Name | Pts | Type | Answer | Rubric |\n| -: | --- | -: | --- | --- | --- |" - body = "\n".join( - f"| {r.q_num} | {r.name} | {r.points} | {r.type} | {r.answer} | " - f"{r.rubric.replace(chr(10), ' ')} |" - for r in rows - ) - return head + "\n" + body +def _grading_note_question(form_id: str, r: OutlineRow) -> str: + """One rich per-question block: heading + per-option Gradescope rubric table.""" + qkey = f"{form_id}·Q{r.q_num}" + head = f"#### {qkey} · {r.name} · {r.points} pts · {r.type.upper()}\n" + caption = "" + if r.type in ("fib", "code") and r.rubric: + caption = f"*Scoring:* {r.rubric.replace(chr(10), ' ')}\n\n" + lines = ["| Pts | Key | Rubric item (paste into Gradescope) |", + "| --: | --- | --- |"] + if r.type == "fib": + n = len(r.options) + per = r.points // n if n and r.points % n == 0 else None + for slot, item, _ in r.options: + pts = f"+{per}" if per is not None else "·" + lines.append(f"| {pts} | `{qkey}·{slot}` | {item} |") + if n > 1: + lines.append(f"| 0 | `{qkey}·order` | Incorrect order / wrong mapping " + "(right values, wrong blanks) |") + else: + for slot, item, is_correct in r.options: + lines.append(f"| {'+' + str(r.points) if is_correct else '0'} | " + f"`{qkey}·{slot}` | {item} |") + lines.append(f"| 0 | `{qkey}·none` | No answer / multiple marks |") + return head + "\n" + caption + "\n".join(lines) + "\n" def emit_grading_note(manifest, per_form_outline, per_form_pages, exam_root) -> Path: exam_root = Path(exam_root) out = exam_root / "GRADING_NOTE.md" forms = [f.id for f in manifest.forms] - total_pts = sum(r.points for r in next(iter(per_form_outline.values()))) - nq = sum(len(v) for v in per_form_outline.values()) + first = per_form_outline[forms[0]] + total_pts = sum(r.points for r in first) + nq = len(first) # per-form question count pages = per_form_pages.get(forms[0], 0) tag = _course_tag(manifest.course) + forms_label = f"{len(forms)} form{'s' if len(forms) != 1 else ''} ({'/'.join(forms)})" fm = ( "---\n" - f"type: grading-note\n" + "type: grading-note\n" f"tags: [teaching, {tag}, exam, gradescope, answer-key, internal]\n" "visibility: private\n" "icon: LiClipboardCheck\n" @@ -345,22 +447,36 @@ def emit_grading_note(manifest, per_form_outline, per_form_pages, exam_root) -> parts = [ fm, f"# {manifest.exam} — Gradescope Grading Note ({manifest.course} {manifest.term})\n", - "> [!warning] Internal — answer key\n" - "> Grader/ISA only. Never student-facing. Serials/register are grader " - "infra (see [[project_exam_serial_internal_only]]).\n", - f"{total_pts} pts · {nq} questions · {len(forms)} forms · {pages}/exam\n", + "> [!warning] Internal — ISA work order\n" + "> Grader/ISA only, never student-facing. Enter each question's rubric items " + "into Gradescope **verbatim**; grade by matching — your only decision per paper " + "is *which item applies*. Keys (`form·Qn·slot`) round-trip to per-distractor " + "statistics — keep them intact. Serials/register are grader infra " + "(see [[project_exam_serial_internal_only]]).\n", + f"{total_pts} pts · {nq} questions · {forms_label} · {pages} pages/exam\n", "## Gradescope setup\n" - "1. Two assignments — one per form — linked as a **Version Set**.\n" + "1. Two assignments (one per form) linked as a **Version Set**.\n" "2. Upload `gradescope/_template.pdf` (the BLANK form) as each template.\n" - "3. Build the outline: one region per question, points from the table.\n" - "4. Enter the key in-UI (Gradescope imports nothing); keep " - "`gradescope/_answer_key.pdf` open.\n" - f"5. Upload each form's scanned stack to its own version; length = {pages} pages.\n", + "3. Per question: draw the region, add **one rubric item per row** (`Pts` + item " + "text). Credit is the `+N` rows; `0` rows are no-credit (apply for feedback/stats).\n" + f"4. Upload each form's scanned stack to its own version; length = {pages} pages.\n", + "> [!note] FIB/code scoring\n" + "> `fib`/`code` rows carry a **Scoring** caption — the authoritative rule. For " + "multi-blank fills that don't divide evenly, apply the caption (the per-blank rows " + "are reference, marked `·`). Add verbose “why-wrong” prose to any Gradescope " + "item if you like; keep keys intact.\n", ] for fid in forms: - parts.append(f"## Form {fid}\n" + _grading_note_table(per_form_outline[fid]) + "\n") - # verify dir is always build/ — register.csv's output_pdf carries the - # .parts/ prefix in single layout, so it resolves relative to build/. + blocks = "\n".join(_grading_note_question(fid, r) for r in per_form_outline[fid]) + parts.append(f"## Form {fid}\n\n" + blocks) + parts.append( + "\n" + "## Post-exam statistics\n" + "Per-distractor item analysis from this exam's Gradescope evaluations " + "(difficulty, dead distractors, miskey alarm):\n\n" + "- [[ITEM_ANALYSIS|Item analysis (Markdown)]]\n" + "- `item_analysis.html` — Item Analysis broadsheet (open in a browser)\n" + ) parts.append( "## Appeals\n" "Each paper's footer `Serial · ID` resolves to one student + form via " diff --git a/tests/test_exam_pack.py b/tests/test_exam_pack.py index 779013f..ea567d1 100644 --- a/tests/test_exam_pack.py +++ b/tests/test_exam_pack.py @@ -168,6 +168,35 @@ def test_parse_outline_captures_name_rubric_and_full_fib_answer(): assert rows[3].type == "fib" and rows[3].answer == "confusion; diffusion" +def test_parse_outline_normalizes_letter_prefixed_tf_reveal(): + # House standard is `Answer: True`, but a source may deviate to `Answer: a (True)`. + # The outline answer must still be the verdict WORD (never the letter), and the + # per-option breakdown must credit the right verdict slot. + tex = ( + "\\begin{document}\\begin{enumerate}\n" + "% name: claim one\n" + "\\item \\textit{(2 pts)}~\\textsc{T~/~F.}~Claim one.\n" + " \\begin{enumerate}[label=(\\alph*)]\n" + " \\item \\correctchoice{True}\n" + " \\item \\wrongchoice{False}\n" + " \\end{enumerate}\n" + " \\ifanswers \\textbf{Answer:} a (True) \\fi\n" + "% name: claim two\n" + "\\item \\textit{(2 pts)}~\\textsc{T~/~F.}~Claim two.\n" + " \\begin{enumerate}[label=(\\alph*)]\n" + " \\item \\wrongchoice{True}\n" + " \\item \\correctchoice{False}\n" + " \\end{enumerate}\n" + " \\ifanswers \\textbf{Answer:} b (False) --- because reasons \\fi\n" + "\\end{enumerate}\\end{document}" + ) + rows = parse_outline_from_tex(tex) + assert rows[0].type == "tf" and rows[0].answer == "True" # not "a" + assert rows[1].type == "tf" and rows[1].answer == "False" # not "b" + assert ("T", "True", True) in rows[0].options + assert ("F", "False", True) in rows[1].options + + def test_missing_name_errors_with_question_number(tmp_path): tex = ( "\\begin{document}\\begin{enumerate}\n" @@ -233,12 +262,35 @@ def test_emit_gradescope_roster_columns(tmp_path): assert rows[0]["Email"] == "" # email not carried — left blank +def test_build_options_legacy_inline_code(): + # Legacy code items (pre-2026-06-09) have no \correctchoice enumerate; the + # True/False rows must be synthesized from the answer so the key round-trips. + from lectern.exam_pack import _build_options + block = (r"\textsc{Code.}~Circle \textbf{True} or \textbf{False}: ..." + "\n\\begin{lstlisting}\nx = 1\n\\end{lstlisting}\n" + r"\ifanswers \textbf{Answer:} False --- amplification multiplies \fi") + assert _build_options(block, "code", "False") == [ + ("true", "True", False), ("false", "False", True)] + # Modern items with the enumerate parse their own choices. + modern = (r"\begin{enumerate}[label=(\alph*)]\item \correctchoice{True}" + r"\item \wrongchoice{False}\end{enumerate}") + assert _build_options(modern, "code", "True") == [ + ("true", "True", True), ("false", "False", False)] + + def test_emit_grading_note_structure(tmp_path): + # Rows carry per-option breakdowns, as parse_outline_from_tex now populates. a = [ - OutlineRow(1, 2, "mc", "a", "CIA — confidentiality", "Correct = a (2 pts)."), - OutlineRow(2, 2, "fib", "confusion; diffusion", "FIB — Shannon", "1 pt/blank."), + OutlineRow(1, 2, "mc", "a", "CIA — confidentiality", "Correct = a (2 pts).", + options=[("a", "(a) Confidentiality", True), + ("b", "(b) Availability", False)]), + OutlineRow(2, 2, "fib", "confusion; diffusion", "FIB — Shannon", "1 pt per blank.", + options=[("b1", 'blank 1 = "confusion"', True), + ("b2", 'blank 2 = "diffusion"', True)]), ] - b = [OutlineRow(1, 2, "mc", "c", "CIA — availability", "Correct = c (2 pts).")] + b = [OutlineRow(1, 2, "mc", "c", "CIA — availability", "Correct = c (2 pts).", + options=[("a", "(a) Confidentiality", False), + ("c", "(c) Availability", True)])] man = ExamManifest(course="CECS 378", term="su26", exam="Exam 1", forms=[FormSpec("A", tmp_path / "A.tex"), FormSpec("B", tmp_path / "B.tex")], @@ -251,11 +303,20 @@ def test_emit_grading_note_structure(tmp_path): assert "type: grading-note" in text assert "tags: [teaching, cecs-378, exam, gradescope, answer-key, internal]" in text assert "## Form A" in text and "## Form B" in text - assert "CIA — confidentiality" in text - assert "confusion; diffusion" in text # FIB full answer in table - assert "4 pts · 3 questions · 2 forms · 4/exam" in text + # rich per-question block: heading + per-option rows with form·Qn·slot keys + assert "#### A·Q1 · CIA — confidentiality · 2 pts · MC" in text + assert "| +2 | `A·Q1·a` | (a) Confidentiality |" in text # correct option scored + assert "| 0 | `A·Q1·b` | (b) Availability |" in text # distractor zero-credit + assert "`A·Q1·none`" in text # no-answer row present + # FIB renders one row per blank (not a joined cell) + a Scoring caption + assert '| +1 | `A·Q2·b1` | blank 1 = "confusion" |' in text + assert '| +1 | `A·Q2·b2` | blank 2 = "diffusion" |' in text + assert "*Scoring:* 1 pt per blank." in text + # summary uses the PER-FORM question count + form ids + assert "4 pts · 2 questions · 2 forms (A/B) · 4 pages/exam" in text assert "length = 4 pages" in text assert "[!warning] Internal" in text + assert "## Post-exam statistics" in text def test_emit_region_products_copies_and_outlines(tmp_path):