|
| 1 | +# AutoNote Architecture Design |
| 2 | + |
| 3 | +## System Overview |
| 4 | + |
| 5 | +AutoNote is a desktop application that generates comprehensive study notes from Canvas LMS lecture materials and videos. It uses an Electron frontend, a set of Python pipeline scripts executed as subprocesses, and an isolated ML virtual environment for GPU-accelerated processing. |
| 6 | + |
| 7 | +``` |
| 8 | +User Interface (Electron) |
| 9 | + │ |
| 10 | + ├── main.js IPC bridge, subprocess management, file operations |
| 11 | + ├── renderer/app.js Single-page app (7 pages), terminal, state management |
| 12 | + └── preload.js Secure IPC bridge between renderer and main process |
| 13 | + │ |
| 14 | + ▼ |
| 15 | +Pipeline Scripts (Python, executed as subprocesses) |
| 16 | + │ |
| 17 | + ├── downloader.py Canvas + Panopto download |
| 18 | + ├── extract_caption.py Whisper transcription |
| 19 | + ├── frame_extractor.py Screen-share frame extraction + dedup |
| 20 | + ├── semantic_alignment.py Transcript ↔ slide alignment (FAISS + Viterbi) |
| 21 | + ├── alignment_parser.py Compact alignment JSON for LLM prompts |
| 22 | + └── note_generation.py LLM-based note writing + image filtering |
| 23 | + │ |
| 24 | + ▼ |
| 25 | +ML Environment (~/.auto_note/venv/) |
| 26 | + PyTorch, faster-whisper, sentence-transformers, FAISS, BGE-M3 |
| 27 | +``` |
| 28 | + |
| 29 | +## Data Flow |
| 30 | + |
| 31 | +``` |
| 32 | +Canvas API Panopto API |
| 33 | + │ │ |
| 34 | + ▼ ▼ |
| 35 | +materials/ (PDF, PPTX) videos/ (MP4) |
| 36 | + │ │ |
| 37 | + │ ┌─────┴──────┐ |
| 38 | + │ │ │ |
| 39 | + │ Camera? Screen share? |
| 40 | + │ │ │ |
| 41 | + │ ▼ ▼ |
| 42 | + │ captions/ frames/ + captions/ |
| 43 | + │ (Whisper) (ffmpeg scene detect |
| 44 | + │ + perceptual hash dedup |
| 45 | + │ + info-score selection) |
| 46 | + │ │ │ |
| 47 | + └────────┬───────────┘ │ |
| 48 | + │ │ |
| 49 | + ▼ ▼ |
| 50 | + alignment/ alignment/ |
| 51 | + (BGE-M3 matching → (timestamp-based |
| 52 | + FAISS + Viterbi) frame assignment) |
| 53 | + │ │ |
| 54 | + └────────┬───────────────┘ |
| 55 | + │ |
| 56 | + ▼ |
| 57 | + alignment/*.compact.json |
| 58 | + (10x smaller for LLM prompts) |
| 59 | + │ |
| 60 | + ▼ |
| 61 | + notes/sections/L{N}_S{ci}.md |
| 62 | + (per-chunk LLM generation, cached) |
| 63 | + │ |
| 64 | + ▼ |
| 65 | + notes/CourseName_notes.md |
| 66 | + (merged + image-filtered final output) |
| 67 | +``` |
| 68 | + |
| 69 | +## Component Details |
| 70 | + |
| 71 | +### Electron Frontend |
| 72 | + |
| 73 | +**Architecture**: Single-page application with 7 pages and a persistent terminal panel. |
| 74 | + |
| 75 | +**State management**: Global `State` object in `app.js` persists form values across page navigation. Key fields include `pipeline.courseId`, `pipeline.language`, `pipeline.detail`, `pipeline.force`, and step checkboxes. |
| 76 | + |
| 77 | +**Subprocess execution**: `main.js` spawns Python scripts via `child_process.spawn()` (or `node-pty` when available for tty support). Stdout/stderr are streamed to the renderer via IPC events (`process:data`, `process:done`). |
| 78 | + |
| 79 | +**Dashboard detail modal**: Clicking a course card opens a modal overlay that lists all transcribed videos with their processing status (caption/alignment/notes). Each video has a delete button that removes the transcript, alignment, note sections, and per-video note files via the `course:deleteVideo` IPC handler. |
| 80 | + |
| 81 | +### Pipeline Scripts |
| 82 | + |
| 83 | +#### downloader.py |
| 84 | +- Downloads videos from Panopto and materials from Canvas |
| 85 | +- Tracks download state in `manifest.json` and `download_log.json` |
| 86 | +- Slack mode adds random delays to avoid rate-limiting |
| 87 | +- Smart size filter uses LLM to select relevant files when > 1 GB |
| 88 | + |
| 89 | +#### extract_caption.py |
| 90 | +- Selects backend automatically: faster-whisper (GPU) or OpenAI Whisper API |
| 91 | +- Produces timestamped segment-level JSON |
| 92 | +- `--force` flag re-transcribes even if captions already exist |
| 93 | +- Language detection probes from mid-audio for accuracy |
| 94 | + |
| 95 | +#### frame_extractor.py |
| 96 | +- Classifies videos as screen-share or camera using edge/uniformity heuristics |
| 97 | +- Scene detection via ffmpeg scene filter + periodic sampling fallback |
| 98 | +- **Same-page deduplication**: Groups consecutive frames by perceptual hash similarity (dHash, Hamming distance < 45 bits). From each group, selects the frame with the highest visual information score (edge density on 160x120 grayscale). This ensures incremental bullet reveals keep only the most complete version. |
| 99 | +- Builds timestamp-based alignment JSON compatible with the rest of the pipeline |
| 100 | + |
| 101 | +#### semantic_alignment.py |
| 102 | +- Extracts text from slides (PDF/PPTX/DOCX) with image enrichment for sparse slides |
| 103 | +- **Matching priority** (in `process_course`): |
| 104 | + 1. User-supplied mapping JSON |
| 105 | + 2. Automatic name/number matching |
| 106 | + 3. BGE-M3 embedding match (pre-computed via `suggest_matches`) |
| 107 | + 4. mpnet content embedding fallback |
| 108 | +- Embeds slides and transcript windows with sentence-transformers |
| 109 | +- FAISS IndexFlatIP for fast cosine-similarity K-NN lookup |
| 110 | +- Viterbi temporal smoothing with forward bias and temporal position prior |
| 111 | +- Off-slide detection for Q&A/demo segments (cosine < threshold) |
| 112 | +- `--force` flag re-aligns even if alignment files already exist |
| 113 | + |
| 114 | +#### alignment_parser.py |
| 115 | +- Compresses full alignment JSON (300 KB) into compact per-slide format (30 KB) |
| 116 | +- Cleans filler words from transcripts |
| 117 | +- Used by note_generation to build token-efficient LLM prompts |
| 118 | + |
| 119 | +#### note_generation.py |
| 120 | +- Multi-provider LLM support: OpenAI, Anthropic, Google Gemini, DeepSeek, xAI, Mistral |
| 121 | +- **Language system**: `--language en|zh` CLI flag overrides the `NOTE_LANGUAGE` constant. The `_P(key)` function selects from `_PROMPTS["en"]` or `_PROMPTS["zh"]` dictionaries containing complete prompt sets (system, chunk, slide_only, verify, exam, detail_instructions). Language is selectable per-run from the Pipeline and Generate page dropdowns. |
| 122 | +- Per-lecture chunking: CHAPTER_SIZE slides per LLM call |
| 123 | +- Section caching: each chunk saved as `L{N}_S{ci}.md` for resume support |
| 124 | +- `--force` flag re-generates all sections from scratch |
| 125 | +- Image filtering: multi-step decision pipeline (cache description keywords → title pattern → vision API). Includes all slides with visual elements; only excludes administrative/non-course elements. |
| 126 | +- Self-scoring: coverage, terminology, callouts, code blocks (weighted average) |
| 127 | +- Per-video mode (`--per-video`): one note file per lecture instead of merged |
| 128 | +- Iterative mode: raises detail level until quality target is reached |
| 129 | +- All terminal output (print/tqdm.write) is in English regardless of note language |
| 130 | + |
| 131 | +### Force Regenerate Behavior |
| 132 | + |
| 133 | +The "Force regenerate" toggle applies to whichever pipeline steps are selected: |
| 134 | + |
| 135 | +| Step selected | Without force | With force | |
| 136 | +|---|---|---| |
| 137 | +| Transcribe | Skips videos with existing captions | Re-transcribes all videos | |
| 138 | +| Align | Skips captions with existing alignment | Re-aligns all captions | |
| 139 | +| Generate notes | Uses cached section .md files | Re-calls LLM for all sections | |
| 140 | + |
| 141 | +Without force, the pipeline is incremental: only missing files are processed. |
| 142 | + |
| 143 | +### Image Inclusion Pipeline |
| 144 | + |
| 145 | +Images pass through multiple filtering layers before appearing in the final notes: |
| 146 | + |
| 147 | +1. **Image hints generation**: Slides with word_count < 80, cached descriptions, or code are offered to the LLM as available images |
| 148 | +2. **LLM prompt instructions**: System prompt instructs to insert all slides with visual elements (diagrams, charts, code, math, etc.) and skip pure text or administrative slides |
| 149 | +3. **Post-generation filter** (`filter_images_pass`): |
| 150 | + - Screen-share frames: always kept |
| 151 | + - Cache-verified visual description: kept |
| 152 | + - Title/divider pattern: removed |
| 153 | + - Vision API (GPT-4o-mini): decides uncertain cases; defaults to keep |
| 154 | + |
| 155 | +## Data Storage |
| 156 | + |
| 157 | +**No database** — all data is file-based JSON/Markdown. |
| 158 | + |
| 159 | +| File | Location | Purpose | |
| 160 | +|------|----------|---------| |
| 161 | +| `config.json` | `~/.auto_note/` | Canvas URL, Panopto host, output dir | |
| 162 | +| `*_api.txt` / `*_token.txt` | `~/.auto_note/` | API keys and tokens | |
| 163 | +| `manifest.json` | Output dir root | Video download state tracking | |
| 164 | +| `download_log.json` | Per-course | Material download tracking | |
| 165 | +| `captions/*.json` | Per-course | Whisper transcript (timestamped segments) | |
| 166 | +| `alignment/*.json` | Per-course | Full segment-level alignment | |
| 167 | +| `alignment/*.compact.json` | Per-course | Token-efficient alignment for LLM | |
| 168 | +| `notes/sections/L*_S*.md` | Per-course | Cached per-chunk note sections | |
| 169 | +| `notes/*_notes.md` | Per-course | Final merged/per-video notes | |
| 170 | +| `notes/*.score.json` | Per-course | Self-score breakdown | |
| 171 | +| `notes/images/L*/` | Per-course | Rendered slide PNGs | |
| 172 | + |
| 173 | +## Testing |
| 174 | + |
| 175 | +Tests are in `test/` and organized by scope: |
| 176 | + |
| 177 | +| File | Scope | |
| 178 | +|------|-------| |
| 179 | +| `test_unit.py` | Offline unit tests: Viterbi, timeline, hashing, alignment parsing, slide discovery | |
| 180 | +| `test_pipeline.py` | Integration tests: script execution, manifest schema, CLI flags (some require network) | |
| 181 | +| `test_language_and_skip.py` | Language selection, terminal output (no CJK in prints), skip logic with/without --force | |
| 182 | +| `test_note_generation.py` | Note generation specific tests | |
| 183 | +| `test_gui.py` | GUI-specific tests | |
| 184 | +| `electron/test/main.test.js` | Electron main process tests | |
| 185 | + |
| 186 | +Run all offline tests: `python -m pytest test/ -v -k "not integration"` |
0 commit comments