diff --git a/SKILL.md b/SKILL.md index 11e83a6..aff1907 100644 --- a/SKILL.md +++ b/SKILL.md @@ -92,6 +92,19 @@ coverage = edit the manifest's `topics` and re-run. (First built for CECS 326, per-topic lecture reading lists are still produced by the lecture-materials `reading-list` artifact.) +## Authoring assignments + +Author every coding assignment with its **graders' contract** included — a student-facing spec +(README + fixed deliverable paths + optional `solution.yaml`), an ISA rubric and/or grading +skill, and an automated grading path wherever a deliverable is machine-checkable. Choose the +grading mechanism per deliverable: **gradebox** (sandbox / runnable code / deterministic +artifacts), the **oracle** (verify-by-proof; document the `/verify` receiving-end contract + +course-token CI wiring), or **manual** (subjective). Keep one point split across README, course +`CLAUDE.md`, Classroom issues, and the rubric. Bake integrity in via forcing-functions/canaries +and flag-don't-deduct. New to this? Start with [`docs/grading-types.md`](docs/grading-types.md) +— a plain-language, use-case guide to the grading types and which to pick (no security/OS +background assumed). Full procedure: [`docs/assignment-authoring.md`](docs/assignment-authoring.md). + ## Teaching workflow See the project README for term-end/mid-term rituals. diff --git a/docs/assignment-authoring.md b/docs/assignment-authoring.md new file mode 100644 index 0000000..0320b9c --- /dev/null +++ b/docs/assignment-authoring.md @@ -0,0 +1,212 @@ +# Authoring a Coding Assignment + +How to author a CECS coding assignment so it is **gradeable the same disciplined way every +time**: a clear student-facing spec, an ISA rubric (and/or grading skill), and an automated +grading path wherever a deliverable is machine-checkable — with the student-facing point split +and the grader-facing rubric kept in agreement. + +This guide is **generic** (any coding assignment). The CECS 378 Pokémon-malware lab is used as a +worked example at the end. + +> **Doctrine:** every assignment ships with its **graders' contract** — the rubric/skill *and* +> the run instructions for whatever automated grader applies. Don't bolt grading on after the +> students have already submitted. + +--- + +## The two halves of every assignment + +| Half | Artifacts | Owner | +|---|---|---| +| **Student-facing** | README (deliverables, fixed paths/names), starter repo, optional machine-readable solution manifest, **student self-check CI** (non-binding) | the student sees these | +| **Grader-facing** | ISA rubric and/or grading skill, the automated grader (gradebox spec / oracle provisioning), the run-doc, the integrity checklist | ISAs + the operator | + +The two halves must agree on **one point split**. A README that says 33/8/17/42 while the rubric +says 20/20/15/20/25 is a defect — reconcile at authoring time, across README, the course +`CLAUDE.md`, the GitHub Classroom deliverable issues, and the rubric. + +--- + +## Step 0 — Choose the grading mechanism + +> **New to the grading types, or want the use-cases first?** Read [`docs/grading-types.md`](grading-types.md) +> — a plain-language decision guide (no security/OS background assumed) that walks the same choice +> below with worked examples and "use when / don't use when" for each type. The table here is the +> quick version. + +| Mechanism | Use when | What it does | +|---|---|---| +| **gradebox** | The deliverable is **runnable code, an exploit, or a deterministic artifact** (a patch, a binary, a parseable file) | Runs untrusted student work in a hardened, ephemeral container → binary tests → score → `gradebook.csv`. See `oracle` repo `docs/gradebox-authoring.md`. | +| **oracle** | The lab reduces to **"prove you obtained a result"** and you must **not** run student code server-side (e.g. crypto: ciphertext + a candidate proof) | Verify-by-proof network service. The student runs the work their side; `/verify` is the sole arbiter; secrets never leave the host. See "oracle — receiving end" below. | +| **manual** | The deliverable is **inherently subjective** (a writeup, a design, a reflection) | ISA rubric / grading skill only. | +| **hybrid** | Most real labs | Deterministic slice → gradebox/oracle; subjective slice → manual rubric. The automated grader also emits **evidence** that accelerates the manual pass. | + +Pick the smallest mechanism that covers each deliverable. Don't try to autograde an inherently +visual or subjective deliverable — that is false confidence. Autograde what is *decidable*; emit +*evidence* for the rest. + +--- + +## Step 1 — Author the student-facing assignment + +- **Fixed deliverable paths/names.** State them exactly (e.g. `WRITEUP.md` at repo root, + `patch.ips`, `solution.yaml`). Graders match case-sensitively; naming variance becomes a false + "missing deliverable." (See `feedback_lab_deliverable_naming_contract`.) +- **A machine-readable solution manifest where it helps.** Having the student *declare* their + answer (the species, the config, the chosen target) in a small `solution.yaml` turns an + ambiguous "did they do it" into a deterministic "does it match the declared answer." +- **Deliverables map 1:1 to rubric sections**, so the grade breakdown is legible. + +--- + +## Step 2 — Author the rubric / grading skill + +A detailed ISA rubric (see the existing `*_lab_grading_rubric.md` notes for the house style): + +- Per-section point grids; **four-anchor language** (Full / Solid / Weak / Absent). +- An explicit **grading workflow order** (what to check first; the authenticity check is + non-negotiable and comes first). +- A **mandatory academic-integrity checklist** ("flag, don't deduct" — escalate to the + instructor). +- Point split **identical** to the student-facing breakdown. + +Optionally encode the same rubric as a Claude Code grading skill so an agent can ISA-grade +consistently. The rubric and the skill are two renderings of one contract — keep them in sync. + +--- + +## Step 3a — gradebox path (sandbox / runnable / deterministic) + +Authoring a gradebox lab is **zero-Python**: a Docker image + one YAML spec. Any helper logic is +baked **into the image** as lab content, not added to the gradebox engine. + +1. **Image** (`images//`): a minimal base + the tools the checks need + any baked assets. + - **Bake licensed/copyright assets into a PRIVATE image** and **do not publish it** (e.g. a + base ROM, a dataset). Keep keys/secrets out of images entirely. +2. **Spec** (`.yaml`): `build.cmd` + `tests[]` (each a **binary** pass/fail check with + `exit_code`/`stdout_regex`/`stdout_contains` predicates) + `limits` + optional `sandbox` + profile. Use the **`dynamic_flag`** test type for exploit labs — a fresh random secret per + round defeats a hardcoded-output cheat. +3. **Evidence tests (0-pt)**: print analysis (diffs, region maps, cross-checks) to the + transcript so the manual pass is faster. Hard facts (a diff, a git history) are evidence; + heuristic cross-checks are **advisory** — never auto-penalize on a heuristic + (`feedback_triage_facts_vs_advisory`). +4. **Run-doc** (ships with the lab): + ```sh + python -m gradebox doctor # verify isolation controls + python -m gradebox run --spec .yaml \ + --submissions ./repos --out ./out --jobs 4 # → out/gradebook.csv + reg-gradebook import … # lectern boundary + ``` + +Authoritative reference: `oracle` repo `docs/gradebox-authoring.md` + `docs/gradebox.md`. + +--- + +## Step 3b — oracle path (verify-by-proof) — receiving end + +Use the oracle when the lab is a **proof of a result** and you must not execute student code on +the server. The oracle never runs student code: students run the exploit/computation their side +and submit ciphertext + a candidate proof; `/verify` returns `{pass: bool}`; per-student keys are +derived `HMAC(root_key, course|repo|assignment|challenge|label)` so the key never leaves the host. + +What the **authoring** side does (operator, off-HTTP admin CLI): + +- Provision the course, enable the assignment, issue a **per-course service token** + (`python -m oracle_server.admin add-course / enable-assignment / issue-token`). + +What ships in the **assignment** (the receiving end — this is what to document for students/CI): + +- The **`/verify` contract**: the exact route (`/{course}/{assignment}/verify`), what the + student POSTs (e.g. ciphertext + candidate proof), and the `{pass}` response. +- **Course token wiring**: the service token as a GitHub Classroom **org secret** + a + `$_COURSE` variable; the student CI calls the course-scoped route. The token is *not* in + the repo or image. +- **Fail-closed** behavior: no proof / wrong proof → no pass; the oracle is the sole arbiter. + +Authoritative reference: `oracle` repo `docs/DEPLOYMENT.md` / `docs/COMPLIANCE.md`; +`project_grading_oracle_licensing`. + +--- + +## Step 4 — Academic-integrity authoring patterns + +Bake integrity into the assignment's **construction**, not just the grading. + +- **Forcing functions / canaries** (`feedback_brown_mm_assignment_canaries`): a **class-specific + element students must touch**. Borrowed work (a peer's patch, a public solution) never touches + your unique element → it fails by construction and is detectable. Rotate a **per-term token** so + last term's work also fails. +- **Hash / input forcing**: distribute a **custom base artifact** (a different hash than any + stock/public version) so a solution built against the public version won't apply/validate — + students are forced to engage *your* specific input. Going further, **significantly diverge** + the base from any well-documented original (the "Super Star Trek" technique) so public + documentation/solutions don't transfer and students must reverse *your* version. +- **Per-student individualized artifacts** (the exam per-student-serial pattern applied to labs): + stamp each student's starter artifact with a token derived from their identity + (`HMAC(class_key, student_id)`) so every student works against a provably **unique** input. The + grader **regenerates** the artifact from the bound id and trusts nothing committed in the repo — + borrowed work fails structurally (wrong base → no round-trip; embedded identity mismatch). +- **`dynamic_flag`** (gradebox): per-round random secret defeats hardcoded-output exploits. +- **Flag, don't deduct.** Hard facts (a diff that contradicts the writeup, a git history) are + audit-grade evidence; heuristic scores are advisory; **no student is penalized without human + review**. `reg-triage` provides git-history authenticity triage with this two-tier structure. + +--- + +## Step 5 — Academic use & IP posture + +- **Third-party / copyrighted assets** (ROMs, datasets, sample malware): keep them in **private, + non-distributed** grading images for **educational binary-analysis use**. Be honest about + derivative status — a build of a third-party disassembly is a *flavored build of that project*, + not original IP; don't claim ownership. Match the posture the asset already has in the course. +- **Never bake secrets/keys** into images or repos. Oracle keys are derived server-side; gradebox + injects per-round secrets at grade time. +- **Tooling licensing.** The oracle (and gradebox, which lives in the oracle repo) is + **source-available** — PolyForm Strict + an Educational Institution Grant: free for academia, + distribution-restricted. Public lectern tooling is MIT. (`project_grading_oracle_licensing`.) + +--- + +## Step 6 — Reconcile & propagate + +- **One point split** across README, course `CLAUDE.md`, Classroom deliverable issues, and the + rubric. +- **Propagate through the template chain** (static copies, not live regen): dev template → + student-facing template → per-class static copy → student forks (student work is never touched). +- **Student self-check CI**: a `.github/workflows` job running the **public** subset of the + authoritative grader's checks, **non-binding** — students see green/red before submitting; the + operator-run grader remains authoritative (mirrors the gradebox-vs-`autograde.yml` authority + model). + +--- + +## Worked example — CECS 378 Pokémon Malware lab (hybrid) + +A binary-analysis lab (modify a Pokémon Yellow ROM) being redesigned to this standard: + +- **Custom base artifact + canary:** a flavored ROM built from a pret/pokeyellow fork with a + CECS-378 element students must change — different hash than the commercial dump (forcing + function) and a per-section plagiarism canary in one. +- **gradebox (hybrid):** the `rom-lab` private image bakes the base ROM + a `.sym`-derived offset + map + a pure-Python `romlab` helper; the spec scores the deterministic slice (IPS round-trip, + canary-changed, each data structure changed / matching the declared species in `solution.yaml`) + and emits evidence (writeup-offset cross-check, region map) for the manual slice (sprite + aesthetics, malware reflection). +- **Reconciled split + run-doc + self-check CI** ship with the lab. + +Full program design: `oracle` repo +`docs/superpowers/specs/2026-06-28-pokemon-malware-gradebox-program-design.md`. + +--- + +## See also + +- [`docs/grading-types.md`](grading-types.md) — plain-language guide to the grading types and which to pick (read first if you're new) +- `oracle` repo: `docs/grading-model.md` (the engine reference behind the grading types), + `docs/gradebox-authoring.md`, `docs/gradebox.md`, `docs/exploit-verification.md`, + `docs/DEPLOYMENT.md` +- lectern: `docs/recon-report-workflow.md` (cohort recon), `docs/gradescope-workflow.md` +- Memories: `feedback_assignment_authoring_grading_contract`, `project_gradebox_sandbox_runner`, + `feedback_brown_mm_assignment_canaries`, `feedback_lab_deliverable_naming_contract`, + `feedback_triage_facts_vs_advisory`, `project_grading_oracle_licensing` diff --git a/docs/grading-types.md b/docs/grading-types.md new file mode 100644 index 0000000..fb3a3b7 --- /dev/null +++ b/docs/grading-types.md @@ -0,0 +1,223 @@ +# Lab Grading Types — An Instructor's Guide to Choosing One + +**Who this is for.** Instructors and ISAs deciding *how a lab will be graded* — before writing it. +You don't need a security or operating-systems background; specialized terms are explained the first +time they appear. This is the **which-do-I-pick** guide. For *how each type works under the hood*, +see the engine reference it links to (oracle `docs/grading-model.md`); for the **deep technical +internals** that an OS or computer-security specialist would want, follow the "dig deeper" links at +the end of each section — those docs are kept precisely so the specialist can go as far down as they +like. + +> **Companion:** once you've picked a type here, `docs/assignment-authoring.md` (Step 0 onward) walks +> you through *authoring* the assignment so its graders have everything they need. + +--- + +## Why a course needs more than one grader + +A normal programming assignment grades itself: run the student's code, diff the output against a key. +Hands-on security and systems labs break that in two ways, and each break needs a different answer: + +- Some labs **can't** be graded by running student code — the "answer" is a secret you must keep, and + running their code on your machine would be unsafe or beside the point (think: *break this cipher*). +- Some labs **must** run student code that is *designed* to misbehave — a lab about crashing a + program, or a deliberately buggy concurrent program, can take your grading machine down with it. + +So instead of one grader there's a small menu. Pick by answering one question about your lab. + +--- + +## The one question — then a decision tree + +> **Does grading require *running* the student's code — and if so, is the grade just *how the +> program behaves*?** + +``` +Is the deliverable subjective (a writeup, design, analysis)? +│ +├── YES ───────────────────────────────────► MANUAL (rubric / grading skill) +│ +└── NO: can you grade it WITHOUT running their code? (check a single submitted answer) + │ + ├── YES ───────────────────────────► ORACLE (verify-by-proof service) + │ + └── NO: you must run it. What did the student submit? + ├── a program, judged by behavior ──► GRADEBOX · code-running + ├── a working exploit of a target ──► GRADEBOX · exploit-verification + └── a modified binary file + patch ─► GRADEBOX · binary-artifact / ROM +``` + +Almost every real lab ends up **hybrid** — one machine-graded part plus one human-graded part. That's +expected; see the last section. + +--- + +## Manual · rubric or grading skill + +**Use it when** the deliverable is *inherently subjective*: a malware-analysis writeup, a design +rationale, a reflection, the *quality* of an explanation. A human grades it against a rubric (the +four-anchor Full / Solid / Weak / Absent language) and/or a Claude Code grading skill. + +**Don't use it for** anything a machine can decide objectively — autograding the objective part frees +your ISAs to spend their judgment where it actually matters. + +**The payoff with the other types:** every automated type below *emits evidence* (what changed, what +passed, integrity flags) straight into the human grader's hands, so the manual pass is faster and more +consistent. + +**Maturity: in production.** (Pokémon-malware and buffer-overflow rubrics are in active use.) + +**Author it:** `docs/assignment-authoring.md` Step 2. + +--- + +## Oracle · verify-by-proof + +**Plain version.** A small web service holds the lab's secret and **never runs student code.** The +student does the work on their own machine and submits only the *result* — a value only a real +solution could produce. The service checks it and returns a trustworthy pass/fail. (Picture a +combination lock you mailed out: you never watch them work, but the right combination back proves they +opened it.) + +**Use it when** the lab reduces to *"prove you obtained this result"* **and** you must not run student +code on the grading host: +- cryptography — recover a key, forge a message authentication code, break an encryption mode +- web auth — mint a login token the server is fooled into accepting +- any "submit the one value only the real solution produces" challenge + +**Don't use it when** the thing you're grading is a program's *behavior* — there's no single answer to +submit. That's gradebox. + +**Why it's hard to cheat:** each student gets their own secret, derived from their identity, so answers +can't be shared and the secret never leaves the server. A "pass" means the work was genuinely done. + +**Ships with:** ready-made `spellbreaker` (symmetric crypto) and `webauth` (token forgery) modules. + +**Maturity: in production** — it has graded a live CECS 378 cryptography lab. + +**Author it:** oracle `docs/ADDING_AN_ASSIGNMENT.md` (a small Python module). **Dig deeper (specialist):** +oracle `docs/grading-model.md` §"verify-by-proof", and the per-course derived-key / audit design in the +oracle README + `docs/COMPLIANCE.md`. + +--- + +## gradebox · running student code safely + +When you *must* run student code, gradebox runs **each submission inside its own sandbox** — a +disposable, locked-down container with no network, no access to the host or to other students' work, +and hard caps on CPU, memory, and process count. Hostile code runs safely; the grade is *what it did*, +not what it claimed. A preflight check refuses to grade at all if the machine can't isolate properly, +and results roll up into a gradebook spreadsheet. Authoring takes **no programming** — a container +recipe plus one short configuration file. + +gradebox has three flavors. They differ only in *what the student submits*. + +### code-running — "did they build a working program?" + +**Use it when** the deliverable is *a working program* whose correctness you can see by running it: +operating-systems and systems labs — processes, threads, synchronization, producer/consumer, +"does it finish without deadlocking." + +**Don't use it when** there's no objective behavioral check (use manual), or the student must *attack* +a target rather than build to spec (use exploit-verification). + +**The hazard it tames:** a **fork bomb** — a program that endlessly spawns copies of itself until the +machine runs out of process slots and freezes. (A real one took down a grading host; that's why +gradebox exists.) The sandbox contains it and *labels* it instead of crashing. + +**Maturity: built and stress-tested** against fork bombs, memory hogs, infinite loops, and escape +attempts. **Author it / dig deeper:** oracle `docs/gradebox-authoring.md` (worked synchronization-lab +example); internals in oracle `docs/gradebox.md`. + +### exploit-verification — "did they *actually* break the target?" + +**Use it when** the student must *demonstrate a capability* against a target program — buffer overflow, +privilege escalation, "make this program do what it shouldn't." The obvious grader — search the output +for a secret "flag" — is defeated by a student who simply *prints the flag*. This flavor plants a +**fresh random secret each round** and checks the exploit actually retrieves *that* secret, several +rounds in a row. Hardcoding the flag scores zero. + +**Don't use it for** build-to-spec programs (that's code-running) or anything with no target to break. + +**The cheat it closes:** a real submission once commented out its exploit and just printed a +precomputed flag — and a naive text-search grader passed it. Not anymore. + +**Maturity: validated end-to-end** on a real CECS 378 buffer-overflow submission (real exploit full +marks; print-the-flag zero). **Author it / dig deeper:** oracle `docs/exploit-verification.md`. + +### binary-artifact / ROM — "did they correctly modify this file?" + +**Use it when** the deliverable is *a modified binary file plus a patch* — e.g. a retro game-cartridge +image (a "ROM") the student hand-edited, submitted with the small patch describing their changes. The +grade is **deterministic checks on the bytes**: does applying their patch to the official starting file +reproduce their binary; would the file actually boot on real hardware (not just a lenient emulator); +did the right data structures change; and — because each student starts from an *individualized* base +with a hidden, tamper-evident identity stamp — is this genuinely *their* copy. Borrow a classmate's +patch and it fails by construction. + +**Don't use it for** behavior you'd need to run (that's the other two flavors); this is for truth that +lives in the file itself. + +**Maturity (precise):** the **byte-level foundation is built and merged** (patch applier, boot-validity +checks, byte-diff, structure reader, identity stamp — tested and reviewed); the **full lab grader that +wraps it is still in design**, first target the CECS 378 Pokémon-malware lab. Real tested plumbing; the +finished end-to-end grader is forthcoming. **Dig deeper (specialist):** oracle `docs/grading-model.md` +§2c, and the design write-up `docs/superpowers/specs/2026-06-28-pokemon-malware-gradebox-program-design.md` +(kept in the private `-dev` lab repo for the operator-only details). + +--- + +## Hybrid · what most labs actually are + +Real labs mix an objective slice with a subjective one — so you mix types. Author the deliverables so +each slice goes to the smallest tool that covers it: + +> **CECS 378 Pokémon-malware lab.** The patched ROM, its structure changes, and the hidden identity → +> gradebox binary-artifact (objective). The malware-analysis writeup and the creative work → manual +> rubric (subjective). The machine's evidence accelerates the human pass. + +**The rule of thumb:** autograde what is *decidable*; emit *evidence* for everything that isn't. Don't +try to autograde an inherently visual or subjective deliverable — that's false confidence. + +--- + +## Cheat-resistance you can add to *any* type + +These are authoring techniques, not separate graders — sprinkle them in. None needs a security +background: + +- **A required fingerprint (the "brown M&M").** A small, course-specific element a correct solution + *must* touch; copied work never touches it, so it stands out. (Named for the band that hid a + "no brown M&Ms" clause to spot venues that hadn't read the contract.) +- **Per-student individualization.** Give each student a slightly different starting point/input + stamped from their identity; the grader regenerates it and trusts nothing in the files — so one + student's work can't pass as another's. (Same idea as per-student exam serials.) +- **Make the public solution not fit.** Hand out a custom starting point so online walkthroughs don't + transfer. +- **Fresh-secret checking** (the exploit-verification trick) — defeats "hardcode the expected output." +- **Authenticity triage** (`reg-triage`) — scans submission *history* for tell-tale patterns. Golden + rule: **flag for a human, never auto-penalize.** No student is sanctioned by a script alone. + +--- + +## Pick-the-type cheat sheet + +| Your lab's deliverable | Grading type | Author it with | Maturity | +|---|---|---|---| +| A writeup, design, or analysis | **Manual rubric** | `assignment-authoring.md` §2 | production | +| "Prove you recovered this secret / forged this token" | **Oracle verify-by-proof** | oracle `ADDING_AN_ASSIGNMENT.md` | production | +| A program judged by how it behaves | **gradebox code-running** | oracle `gradebox-authoring.md` | built & stress-tested | +| A working exploit of a target | **gradebox exploit-verification** | oracle `exploit-verification.md` | validated e2e | +| A patched / modified binary file | **gradebox binary-artifact/ROM** | program spec (in design) | foundation merged; grader forthcoming | +| Some of each (most labs) | **Hybrid** | `assignment-authoring.md` | — | + +--- + +## See also + +- `docs/assignment-authoring.md` — the full authoring procedure (Step 0 picks the mechanism per deliverable) +- oracle `docs/grading-model.md` — the engine reference: what each type *is* (the layer this guide chooses among) +- **Dig-deeper, specialist docs** (kept for OS / computer-security depth): oracle `docs/gradebox.md` + (sandbox security model), `docs/gradebox-authoring.md`, `docs/exploit-verification.md`, + `docs/ADDING_AN_ASSIGNMENT.md`, `docs/COMPLIANCE.md` +- `docs/recon-report-workflow.md` (cohort recon) · `docs/gradescope-workflow.md`