feat: add external API runtime and request control plane

.env.example

@@ -78,6 +78,10 @@ YOUTUBE_OUTPUT_DIR=./downloads
 # Default: true (authentication enabled)
 AUTH_ENABLED=true
 
+# Versioned external API. This API always requires an external API key even
+# when AUTH_ENABLED=false. It is disabled by default; set to true to enable it.
+EXTERNAL_API_ENABLED=false
+
 # Initial Admin Credentials (for headless/automated deployments)
 # If not set, you'll create credentials through the web UI on first access
 # And they will be saved to config/config.json (password hashed)

.github/workflows/ci.yml

@@ -121,6 +121,9 @@ jobs:
       - name: Install dependencies
         run: npm ci --ignore-scripts
 
+      - name: Validate external API fixture and checksum
+        run: npm run test:external-fixture
+
       - name: Run backend tests with coverage
         run: npm run test:backend -- --coverage --coverageReporters=json-summary --coverageReporters=lcov --coverageReporters=text
         env:
@@ -161,7 +164,7 @@ jobs:
             coverage/lcov.info
 
   test-external-api-database:
-    name: External API DB migrations (${{ matrix.engine }})
+    name: External API DB runtime and migrations (${{ matrix.engine }})
     runs-on: ubuntu-latest
     timeout-minutes: 10
     strategy:
@@ -197,7 +200,7 @@ jobs:
         run: npm install -g npm@11.15.0 --ignore-scripts
       - name: Install dependencies
         run: npm ci --ignore-scripts
-      - name: Run external API migration lifecycle
+      - name: Run external API runtime and migration lifecycle
         run: |
           npm run test:external-api-database -- --json --outputFile=external-api-database-results.json
           node -e "const result = require('./external-api-database-results.json'); if (result.numPassedTests < 1 || result.numPendingTestSuites !== 0 || result.numPendingTests !== 0 || result.numTodoTests !== 0) process.exit(1);"
@@ -208,6 +211,19 @@ jobs:
           DB_PASSWORD: rootpass
           DB_ADMIN_USER: root
           DB_ADMIN_PASSWORD: rootpass
+      - name: Verify runtime query plans on synthetic data
+        run: npm run test:external-index-plans
+        env:
+          DB_HOST: 127.0.0.1
+          DB_PORT: 3321
+          DB_USER: root
+          DB_PASSWORD: rootpass
+          EXTERNAL_API_INDEX_REPORT: external-api-index-plans.json
+      - name: Upload runtime query plans
+        uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
+        with:
+          name: external-api-index-plans-${{ matrix.engine }}
+          path: external-api-index-plans.json
 
   test-frontend:
     name: Frontend Tests

CLAUDE.md

@@ -29,6 +29,9 @@ For multi-part requests (e.g., "review this PR AND explain WebSocket handling"),
 ## Architecture
 
 ### Backend (server/)
+- `routes/externalApi.js` and `middleware/externalApiAuth.js`: opt-in `/external-api/v1` routes with mandatory external-key authentication, bounded UTF-8 JSON bodies, private responses, and per-key limits. All External API business logic lives in `modules/externalApi/`; `externalApiConfig.js` (feature availability and `isExternalApiPath`), `externalApiResponse.js`, `externalPermissions.js`, and `externalEligibility.js` (media/rating policy; maps TV-Y7 to level 2 locally since `ratingMapper` has no numeric TV-Y7) centralize the shared rules, and `apiKeyChannelGrantModule.js` manages per-key channel grants.
+- `modules/externalApi/externalCatalogService.js`: cached, grant-filtered catalog and asset lookup. Reads never spawn yt-dlp or write: video detail uses `videoMetadataModule.getCachedVideoMetadata` (cached `.info.json` only). Raw SQL stays portable (snake_case aliases, `POSITION(... IN ...)`), keyset cursors (v3) tie-break on `cv.id`, and single-channel lookups go through `Channel.findOne` with the `apiKeyGrants` include. `externalCatalogRequests.js` batches current-page request status, `externalPagination.js` shares bounded page cursors, and `externalThumbnailProxy.js` enforces upstream image URL, size, and timeout limits.
+- `modules/externalApi/externalRequestService.js`: request creation (auto-approved requests are stored `approved` with `approval_source='auto'`) and owner history; reads never change request state. `externalRequestExecutor.js` is the background executor (started in `server.js` only when the External API is enabled; sweeps every minute, wakes on create/approve, and on `jobModule.onJobEnded`): it rechecks authority, claims `approved` -> `processing`, queues video downloads through `downloadModule.doSpecificDownloads` with the request id as the job id, provisions/enables/grants channels, and deletes videos through `videoDeletionModule`, holding requests with a message while downloads are paused or the video is busy, and recovering interrupted rows. `externalRequestTargets.js` validates grant/channel/video/policy targets, `externalRequestReconciliation.js` closes processing video requests whose job ended, `externalRequestReviewService.js` owns idempotent administrator approve/reject (approve only marks `approved` and wakes the executor), `externalRequestPrimitives.js` shares validation/DTOs, and `externalQuotaService.js` counts `approved` + `processing` as active jobs. Key-before-request locks serialize decisions; `externalWorkLimiter.js` bounds expensive work. `routes/externalRequests.js` exposes admin management endpoints that reject every API key; their UI is a separate follow-up.
 - `modules/externalCookies.js`: validates external cookie snapshots through the installed yt-dlp parser in `utils/validate-cookies.py`, memoizes unchanged results, and reports unusable files for omission; `modules/uploadedCookies.js` gives the uploaded `config/cookies.user.txt` a disposable private copy per run (yt-dlp rewrites its cookie file in place on exit, so concurrent runs sharing it could read a half-written jar or undo a Settings upload; nothing is written back, and Settings uploads are written to a temp file and renamed into place; if the copy fails, the run goes without cookies (error-level log; info when the upload was just deleted) and yt-dlp is never handed the uploaded file itself, whose exit-time jar save could overwrite a newer upload or recreate deleted cookies); `modules/ytdlpProcess.js` owns both kinds of copy's lifetime around yt-dlp processes. No persistent cookie fallback.
 - `modules/rescanRowUpdate.js` + `modules/videoRowGuard.js`: the filesystem rescan (`videosModule.backfillVideoMetadata`) walks the disk once and reads rows in chunks, so downloads and deletions in between make the walk stale. The walk only nominates rows (`isRescanCandidate`); `resolveRescanUpdate` recomputes each nominated row's write from a fresh per-format stat (present only for a regular file / missing on `ENOENT` or `ENOTDIR` or a non-file / unknown on any other error; both formats missing marks the row missing and keeps its stored paths; a missing format is cleared only when the other is present; any present format means `removed: false` even if the other is unknown; an unknown format is never cleared, and with nothing present an unknown writes nothing), and the write only lands if the row is unchanged since read (`videoRowGuard.unchangedSinceRead`: removed, both paths and sizes, `video_resolution`, `last_downloaded_at`; zero affected rows means a conflict only when the write includes a column that really differs, because Sequelize's MySQL dialect connects with -FOUND_ROWS and mysql2 then counts changed rows, not matched rows). Nominated rows left as recorded (files still at a stored path, e.g. outside the downloads folder, or files that could not be checked; re-checked every run) are counted as `kept` (`videosKept` in the run details, not the message). The walk's count of folders and files it could not read is `unreadableOnDisk` in the run details and log (not the message: those rows are re-checked by path). Rows another writer changed are counted as `skippedChanged` (`videosSkipped` in the run record) and failed writes as `failed` (`videosFailed`); both appear in the run message, and a finished scan with failed writes records as `error`/`partial` (a timed-out scan stays `timed-out` so Maintenance can continue it); the completion `rescanStatus` broadcast is built through the same `rescanRunSummary` record so the live Maintenance view matches the run history. Library repair (`jobModule.backfillFromCompleteList`) uses the same guard: it reads the row before searching the disk, writes file details and `removed: false` only for a file it actually found, and looks up a lost row's files by YouTube id with one `videosModule.scanForVideoFiles` walk per run (only when a lost row exists; video and audio, any folder layout), recreating it as missing when the walk finds nothing. The walk skips and counts (`unreadable`) folders and files it cannot read rather than stopping, logging unreadable files once per folder; repair still recreates unmatched rows as missing after a partial walk, because deferring would block recovery wherever a folder is never readable (lost+found, NAS recycle bins), and the next rescan restores a row whose file it finds. Such a run records as `error`/`partial`, naming the count in its message (`recreatedUnverified` and `unreadableOnDisk` in the details); a partial walk that affected no row stays `completed`.
 - `modules/cookieDetails.js`: summarizes the YouTube login cookies (names, counts, earliest expiry; never values) in the active cookie file as `details` on `GET /api/cookies/status` and the upload/delete responses, reading external files through `externalCookies.readExternalCookies`. `modules/cookieTest.js`: `POST /api/cookies/test`, one yt-dlp request to the subscriptions feed with `buildCommonArgs` (same proxy/IP family/cookies/cache as downloads) through `ytDlpRunner`, against a private temp copy of uploaded cookies (yt-dlp writes its jar back on exit, which would otherwise undo a replace/delete made mid-test; external files get spawnYtDlp's snapshot) and never logs stderr (yt-dlp echoes malformed cookie lines with their values); any exit 0 with JSON is signed in (even zero subscriptions), and the feed's "Incomplete yt initial data" warnings mean signed out (EXPIRED_COOKIES), a meaning specific to this feed, so it is not in the shared subscription-import classifier. One test at a time (409), rate-limited by `cookieTestRateLimiter`.
@@ -232,6 +235,8 @@ npm run test:frontend                                        # all frontend test
 npm run test:frontend -- client/src/hooks/__tests__/foo.ts   # one frontend test
 npm run test:backend                                         # all backend tests
 npm run test:backend -- server/modules/__tests__/foo.test.js # one backend test
+npm run test:external-api-database                           # real database migration/runtime regressions
+npm run test:external-index-plans                            # local synthetic query/EXPLAIN report
 npm run lint                                                  # lint front + back
 npm run lint:ts                                               # TypeScript typecheck
 ```