Signs in to NTULearn once and keeps a copy of your courses on disk: pages and announcements as Markdown, attachments as the files they already are. Authentication and sync state stay in this repository; course files go to whichever folder you point each course at.
MIT licensed and public — docs/adr/0002.
In use. Course pages, announcements and attachments sync today, and the command line and the
shape of config/courses.json are settled — a change to either would be a breaking change rather
than a Tuesday. The local media runtime, Kaltura/YouTube/direct content-tree tracers, and
fixture-driven Kaltura Media Gallery discovery, controlled browser-playback fallback, and
one-at-a-time queue worker are explicit and Owner-started. Durable media completeness is reported
separately from sync and verify.
npm run login # refresh the NTU SSO/MFA session
npm run discover # list the NTULearn courses you can see
npm run watchdog # run sync and verify under the daily-run watchdog
npm run sync -- MH2100 # sync one configured course
npm run sync -- all # sync every configured course
npm run verify -- all # check what is on disk against NTULearn, writing nothing
npm run renumber -- MH2500 # rename what is on disk back into the course's order today
npm run media:setup # Owner-started: prepare and verify the local media runtime
npm run media:discover -- all # Owner-started: discover and queue recording appearances
npm run media:worker -- manual # Owner-started: process every enabled queue now
npm run media:worker # Scheduled semantics: work only from 00:00 through 03:59
npm run media:withdraw -- MH1101 media-gallery:_9_1:gallery-entry confirm # confirm one withdrawalmedia:setup is the only command that prepares media dependencies or models. Sync, verify,
watchdog and future scheduled media runs never install anything. A successful media discovery
writes its per-course queue under .data/media-queue/; a red discovery writes no jobs.
The production entrypoint is npm run media:worker -- <scheduled|manual>. One invocation covers
every enabled course and provider in its aggregate digest. Unsupported appearances become terminal
red failures. A red, queued, checkpointed, locked, or otherwise incomplete aggregate exits
non-zero. scheduled is the default and runs only from 00:00 through
03:59 local time, checkpoints the active appearance at 04:00, and writes the independent
.data/media-latest.json digest plus .data/media-logs/. mode: "manual" ignores that time
boundary. Queue entries are queued, active, checkpointed, complete, or red; successful
entries are skipped on later runs, while failures remain retryable. Runs share
.data/media-queue.lock, so a manual run cannot overlap a scheduled one. The worker never calls
media:setup. Every enabled course has Media Gallery/media-status.md, and every discovered
appearance has a sibling .media-status.md with provider, source, stage, video/audio availability,
transcript provenance, retryability, and limitations. A queued appearance is yellow until its next
eligible worker window; an attempted incomplete source or derivative is red and remains retryable.
The status documents and queue remain independent from sync and verify verdicts.
Copy the example and edit it. config/courses.json is ignored by Git — it holds your own
destination paths, and it is meant to stay on your machine.
cp config/courses.example.json config/courses.json{
"profilePath": ".data/chrome-profile",
"statePath": ".data/state.json",
"driveMountPath": "/absolute/path/to/Google Drive",
"watchdogTimeoutMs": 900000,
"media": {
"mediaRoot": "/Volumes/RAID0/Media",
"freeSpaceReserveBytes": 107374182400,
"tools": {
"ffprobe": "ffprobe",
"ytDlp": "yt-dlp"
}
},
"courses": [
{
"key": "AB1234",
"courseId": "_0000000_1",
"destination": "/absolute/path/to/Google Drive/My Drive/Modules/Y1S1/AB1234/NTULearn",
"mediaMode": "off"
}
]
}| Field | Required | What it is |
|---|---|---|
courses[].key |
yes | What you type at npm run sync -- <key>. Matched case-insensitively, and must be unique across the file under that same matching. The course code is the obvious choice. |
courses[].courseId |
yes | NTULearn's own identifier for the course, of the form _1234567_1. Run npm run discover to list the ones you can see. |
courses[].destination |
yes | Where the files land. Absolute, or relative to the repository root. No two courses may share one, or nest one inside another. |
profilePath |
no | The saved browser session. Defaults to .data/chrome-profile. |
statePath |
no | What has already been downloaded. Defaults to .data/state.json. |
driveMountPath |
no (watchdog yes) | The Google Drive mount that contains the destinations. Before writing, the watchdog requires both this directory and each destination's first Drive root below it to be present. |
watchdogTimeoutMs |
no | The watchdog's initial timeout in milliseconds. The Owner pins the placeholder 900000 from the first week's logged durations. |
media.mediaRoot |
required for active/pilot |
The explicit Media store. It must be a directory below /Volumes/RAID0; there is no system-disk fallback. |
media.freeSpaceReserveBytes |
no | Free space retained on the Media store before setup or acquisition. Defaults to 100 GiB. |
media.tools.ffprobe |
no | Portable command name or explicit path for ffprobe; defaults to ffprobe on PATH. Worker preflight verifies it can execute. |
media.tools.ytDlp |
no | Portable command name or explicit path for yt-dlp; defaults to yt-dlp on PATH. Worker preflight verifies it can execute. |
courses[].mediaMode |
no | Exactly active, pilot, or off; omitted means off for legacy configurations. No semester is inferred. |
An active or pilot course also requires media.setup. It names five pinned artifacts:
mediaTool (FFmpeg), asr.runtime and asr.model (whisper.cpp), and formatter.runtime and
formatter.model (llama.cpp plus the selected local formatter model). Each artifact records a
name, destination filename, local file or HTTPS source, revision, SHA-256 sha256, and
license; runtime artifacts may also set verifyArgs, defaulting to --version.
The Owner runs npm run media:setup after filling those entries. It checks that RAID0 is mounted,
the Media store is a real directory, the reserve is available, and existing runtime paths are not
symlinks before creating Media/.runtime/{bin,models,cache,tmp,work,metadata}. It copies or
downloads the pinned artifacts, verifies their checksums and runtime commands, and writes only
Media/.runtime/metadata/runtime.json. The manifest records identity, revision, checksum,
licence, path and size. Every worker run refuses a missing, replaced or misconfigured runtime or
tool and points back to media:setup; model weights, caches and working files remain outside this
repository.
The selected runtime and model licences are documented in
docs/research/media-runtime.md. Setup is intentionally not run by CI or by an ordinary sync.
An active or pilot course may have recording appearances alongside its ordinary sync. Recording
completeness is a separate concern: the sync remains additive and does not claim that media work is
complete. Each appearance keeps its own placement and status, while source evidence and working
artifacts stay on the configured Media store and readable derivatives stay beside the numbered item.
Kaltura, YouTube, and direct video/audio links are classified without retaining expiring query strings;
opaque embedded or launch players are reported as unsupported rather than silently omitted. Known
FeedbackFruits, Cengage, Blackboard placement, Padlet, and Turnitin shapes keep their provider name,
stable reference, retryability, and limitation in the status; provider-specific acquisition is an
injected adapter seam. NTULearn file-shaped non-media references remain visible as retryable
non-recordings. Session material and expiring provider addresses are never persisted. A transcript
is complete only when both
a validated source and formatted Markdown derivative exist. Recording completeness remains
independent of sync and follows ADR-0014;
source provenance and status remain visible with the course artifacts. The routine job never
replaces a formatted derivative. An agent-led caller must opt into runMediaJob({ regenerate: true });
the storage proof still requires current workflow ownership and a matching source digest, so a
manually owned file or preserved source cannot be overwritten.
Browser playback remains a last resort because it is more intrusive and less reproducible than provider retrieval. The fallback is limited to controls and media already visible to the signed-in student, checks for meaningful audio before capture, restores temporary routing on every exit, and keeps 2x disabled until Owner evidence proves it safe for one provider.
Enabled pilot and active courses can use the separate Media Gallery workflow. It opens the
signed-in course surface, exhausts its visible Load More/pagination controls, and refuses to
queue any appearance until the discovered visible count matches the Gallery's displayed total.
Gallery order is retained; repeated provider entries remain separate appearances, and sanitized
creation-time/title names receive a collision number only when necessary. Kaltura provider media,
provider transcripts, normalized sources, and working artifacts stay under the RAID0 Media store.
Formatted transcripts and per-recording status stay under the course destination's Media Gallery/
folder. Courses with mediaMode: "off" are never opened. The media:discover command emits the
reconciled appearance queue but does not change sync or verify's completeness verdict. The
complete queue is the handoff to the existing media-job seam; the queue worker consumes it one
job at a time and supplies the execution context (course, signal, mode, and
requestCheckpoint) to the provider-backed runner. The queue artifact is the durable handoff; its
independent worker digest never changes sync or verify's completeness verdict.
media:withdraw is the explicit confirmation route for one queued appearance. It writes a
withdrawn tombstone into the queue, leaves every existing artifact alone, and never withdraws a
completed appearance.
The checked-in example is config/com.jerome-group.ntulearn.media-worker.example.plist. Copy it
to ~/Library/LaunchAgents/, replace its Node and repository placeholders, set explicit
media.tools paths when launchd's PATH cannot find them, then bootstrap it. It starts at 00:00;
the worker itself enforces the 04:00 checkpoint. Its process exit is non-zero for every red or
incomplete aggregate, so launchd and the two log paths retain the unsuccessful run signal.
The checked-in example is config/com.jerome-group.ntulearn.watchdog.example.plist. Copy it to
~/Library/LaunchAgents/com.jerome-group.ntulearn.watchdog.plist, replace every placeholder with
the real Node executable and repository paths, then load it for the logged-in user:
cp config/com.jerome-group.ntulearn.watchdog.example.plist \
~/Library/LaunchAgents/com.jerome-group.ntulearn.watchdog.plist
# edit the copied plist
launchctl bootstrap "gui/$(id -u)" \
~/Library/LaunchAgents/com.jerome-group.ntulearn.watchdog.plistIt fires at about 05:00 through StartCalendarInterval. If the Mac is asleep then, launchd catches
up when it wakes; that catch-up-on-wake behaviour is why this schedule uses launchd rather than
cron. Installing or rehearsing the real LaunchAgent is the Owner's task because it changes this
machine and spends the saved NTULearn session.
The watchdog writes durable evidence in the state directory, the parent directory of the
configured statePath:
logs/<timestamp>-<uuid>.jsoncontains the captured run and attempt evidence.latest.jsonis the stable input for a future delivery channel. It containsverdict(green,yellow, orred),message,timestamp(the UTC finish time), andrunLog(a path relative to the state directory):
{
"verdict": "green",
"message": "synced, 0 new files",
"timestamp": "2026-08-15T05:00:01.234Z",
"runLog": "logs/2026-08-15T05-00-01-234Z-00000000-0000-0000-0000-000000000000.json"
}Delivery reads this digest; it does not rerun the watchdog or re-derive the verdict from a log.
Point each destination at a dedicated NTULearn subfolder, so your own files in that course's
folder are never touched.
One folder per NTULearn site, not per course. A course often has more than one site — a lecture
site and a tutorial site are separate courses to NTULearn and each needs its own entry. Give the
main site NTULearn and each other site a sibling beside it, NTULearn_Tutorial and so on. Two
entries pointing at one folder, or at a folder inside another's, is refused at startup: they would
interleave their numbered trees, and a sync never deletes (docs/adr/0003), so untangling them
afterwards is hand work.
Incremental and additive: unchanged downloads are skipped, and nothing is ever deleted, so a
run that sees less than the last one leaves the earlier files where they are. Your own files in a
destination are safe for the same reason — docs/adr/0003 argues it. Page text and
announcements become Markdown; attachments keep their original file type. Each course gets a
Course.md overview and an Announcements/ folder, and the content tree is reproduced as
numbered folders in NTULearn's own order.
A file already in the destination under an earlier number is left where it is rather than written a
second time. A name carries its item's position in the course, so one item inserted upstream moves
every later name by one while nothing on disk moves — and a run that wrote to the new number would
leave the destination holding two of each, for good. The run counts those files as renumbered. It
compares the bytes before leaving anything in place, so a file whose contents differ is written at
today's number beside the older one and nothing is ever written over. A folder works the same way
and is where its children go, so a course that reorders keeps growing in the folder it already has
rather than starting a second one beside it; docs/adr/0009 argues it, and ls keeps showing the
order the files arrived in.
A Last synced.md beside the overview records when the sync last ran. It is the only file in a
destination rewritten on every run — everything else is written only when the course moved, so a
run over a course with nothing new writes nothing at all; docs/adr/0008 argues both halves.
An item a sync cannot copy — a quiz, a test, a submission point, anything holding no text, no link
and no attachment — still gets a Markdown file at its own numbered place, naming it and saying
there was nothing to bring across. The numbering stays continuous, and the copy never leaves out
something the course tells you to do; docs/adr/0006 argues it.
A download that fails says where it was and where it would have gone, so the file can be found in NTULearn without walking the course by hand:
{
"file": "Career_Platform_User_Guide.pdf",
"trail": "(For EEE Students Only) Career Pathways Platform › Instruction Manual",
"path": "09 (For EEE Students Only) Career Pathways Platform/03 Instruction Manual/01 Career_Platform_User_Guide.pdf",
"error": "Download failed: HTTP 404"
}A run prints one JSON object: courses, a row per course saying what that run did to it, and
refused beside it when there was one.
A course NTULearn will not hand over — closed at the end of its semester, or one the student is no
longer enrolled in — is named under refused and the run carries on to the next course. Both
sync -- all and verify -- all work this way: a closed course is permanent and there is nothing
to do about it, so it is reported rather than treated as the end of the run. A session that has
lapsed is the other case and still stops everything, because every course after it would fail the
same way and npm run login fixes them all at once.
{
"key": "SLAF01",
"courseId": "_2694562_1",
"reason": "NTULearn refused course _2694562_1 for this student (HTTP 403). …"
}The saved Chrome profile in .data/chrome-profile is the reusable secret. It is
permission-restricted and ignored by Git. The university expires it periodically; run
npm run login again when that happens. Do not copy cookies into configuration files.
A sync says what that run did. npm run verify -- all says what the destination holds: it walks
each configured course in NTULearn, works out the path of every file a sync would write there — the
attachments and the Markdown documents both — and reports which of those paths hold a file. It
downloads nothing and writes nothing on either side, and it exits 1 when anything is absent —
docs/adr/0005.
{
"files": 246,
"attachments": 128,
"documents": 118,
"present": 236,
"renumbered": 1,
"complete": false,
"courses": [
{
"key": "CC0006",
"course": "Sustainability: Seeing Through the Haze",
"destination": "/…/CC0006/NTULearn",
"files": 24,
"attachments": 10,
"documents": 14,
"present": 22,
"missing": [
{
"file": "Career_Platform_User_Guide.pdf",
"trail": "Career Pathways Platform › Instruction Manual",
"path": "09 Career Pathways Platform/03 Instruction Manual/01 Career_Platform_User_Guide.pdf"
},
{
"file": "Video Lecture: Topic 1 - Introduction.md",
"trail": "Week 1",
"path": "01 Week 1/04 Video Lecture_ Topic 1 - Introduction.md"
}
],
"renumbered": [
{
"file": "Cengage WebAssign.md",
"trail": "",
"path": "02 Cengage WebAssign.md",
"onDisk": "01 Cengage WebAssign.md"
}
]
}
],
"refused": [
{
"key": "SLAF01",
"courseId": "_2694562_1",
"reason": "NTULearn refused course _2694562_1 for this student (HTTP 403). …"
}
],
"notCovered": ["A content item this walk did not return expects nothing, …"]
}The gaps it names are fixed by running the sync again; it never repairs anything itself.
A file's name carries its item's position in the course, so one item inserted upstream moves every
later name by one — and nothing on disk moves with it, because a sync never renames
(docs/adr/0003). Those files are on disk under the number they were written with, so verify
counts them present and names them under renumbered, with path where a sync would write the
file today and onDisk where it actually is. Reporting them as missing would be a red that is
almost all noise.
What it does not do is repair the numbering, so ls shows the course in the order it had when each
file was written. A sync does not repair it either — it leaves those files where they are, so a
reordered course stops duplicating itself and stays in the order it arrived in (docs/adr/0009).
This list is what makes that drift legible, and npm run renumber is what answers it.
Two limits. It will not answer at all where two items in the same folder share a title: the name
inside the number identifies neither, so the file is reported missing rather than guessed at. And a
file left behind for an item NTULearn has stopped returning may carry the title of one that moved —
nothing but the bytes separates them, and verify never opens a file — so it is counted present.
npm run renumber -- <course|all> renames what the destination already holds so its numbers carry
the order NTULearn gives the course today. It is deliberately its own command: a rename is a delete
and a create to Google Drive and to anything holding a path to the file, which is not something to
do in a run nobody is watching. A sync never renames and never will — docs/adr/0010 argues both
halves.
It renames only what it can prove the sync wrote and nothing has touched since — the sha256 a
download recorded, or, for a Markdown document, the text the walk is holding. A file you have
annotated fails that check, is left exactly where it is, and is named under kept with the reason.
Nothing is deleted, nothing is written over, nothing moves between folders, and a name that already
holds something is reported under blocked rather than taken.
{
"renamed": 9,
"kept": 1,
"courses": [
{
"key": "MH2500",
"course": "26S1-MH2500-PROBABILITY",
"destination": "/…/MH2500/NTULearn",
"renamed": [
{
"file": "Hand00_MH2500-2026.pdf",
"trail": "",
"from": "09 Hand00_MH2500-2026.pdf",
"to": "10 Hand00_MH2500-2026.pdf"
}
],
"kept": [
{
"file": "Hand01_Part_1_MH2500-2026.pdf",
"trail": "",
"path": "09 Hand01_Part_1_MH2500-2026.pdf",
"onDisk": "08 Hand01_Part_1_MH2500-2026.pdf",
"why": "it has changed since the sync wrote it"
}
]
}
]
}It exits 1 only when something was blocked. A kept file is the command working as intended,
and the report names it on every run so a destination that has gone permanently mixed says so.
A rename breaks anything holding the old path as text — a link from your own notes, a symlink, a script. What survives is anything tracking the file rather than its name: a macOS alias, and a Google Drive share link, since Drive carries a rename across and the file keeps its id. The digest proves nobody edited the file; it proves nothing about who linked to it. That is the cost, and it is why you run this rather than the sync doing it for you.
The number counts the files a sync would write, present at a path, and it is worth reading as
narrowly as that says. The report carries a notCovered list saying so on every run.
It is also relative to one reading of the course, and that reading is named: the walk down
NTULearn's content-item tree, which is what a sync takes as its input and what verify counts
against (docs/adr/0011). Three other readings were tried and refused, so what follows is what the
walk does not cover rather than what nobody has got round to yet.
- A content item the walk did not return is in neither number: nothing expects what nothing saw, so the count it is missing from is a count it was never in. This is the blind spot the four gaps found so far all came out of, and every one was found by opening NTULearn in a browser rather than by the tool disagreeing with itself.
- A category NTULearn would not return — a course whose announcements the student may not read
— expects nothing for the same reason, so the count passes over it. The course says
unreadwhen that has happened. - A course NTULearn would not hand over is in neither number at all — it was never read, so
nothing of it is counted as present or as missing. It is named under
refused, and it does not make the run red: the course is closed,npm run loginopens nothing, and a red that can never go green is one nobody reads. Readcomplete: truealongside that list, never instead of it. - Recorded lecture videos and their transcripts are not read at all. The page naming the lecture is counted; whatever is on the other side of the link is absent from both sides of the number rather than counted as missing.
- External tools — anything reached through LTI — are recorded as a link, on the same terms.
- Presence is not content.
verifyasks the filesystem whether a file is at the path and nothing more, so a truncated, corrupt or since-replaced file counts as present (docs/adr/0005). - An embed this tool does not recognise is one a sync never downloads and
verifynever expects, so both are silent about it together. What that has actually been measured to be is a video player's own output — the streams, thumbnails and caption tracks a Kaltura or YouTube player writes into the page after it starts — which is on the far side of the recorded-lecture limit above rather than a separate one (docs/adr/0011). What a page carries is not silent any more: an embedded<iframe>,<object>or<embed>leaves a> **Not copied**line where it sat, naming what was there. It is not counted, and it is said. - What the destination holds beyond the course is never looked at.
verifyreads only at the paths NTULearn named, so a file kept for an item NTULearn has stopped returning is correct rather than reported — a destination only ever grows (docs/adr/0003). It reads a folder's listing for one question only, and about a name NTULearn did give it: whether the file is there under a number that has since moved.
So complete: true says that every file this tool knows to look for arrived. It does not say the
copy is the course.
Only content visible to the signed-in student can be read. Release-rule-hidden content, instructor-only material, live grades and submissions, and third-party LTI data are not copied; external tools are recorded as links. What is not copied is written down where it sat, so a limit shows up in the destination rather than only here.
npm ci
npm test # node --test
npm run lint # eslint
npm run format:check # prettierAGENTS.md is the instruction file for agents and contributors both; CONTRIBUTING.md is how
work flows here, and MAP.md says where everything lives.