diff --git a/.github/workflows/deploy-cms.yml b/.github/workflows/deploy-cms.yml index a8bad4bce..7a7bc86d7 100644 --- a/.github/workflows/deploy-cms.yml +++ b/.github/workflows/deploy-cms.yml @@ -165,6 +165,11 @@ jobs: BREVO_API_KEY: ${{ secrets.BREVO_API_KEY }} GOOGLE_APPLICATION_CREDENTIALS_JSON: ${{ secrets.GOOGLE_APPLICATION_CREDENTIALS_JSON }} CRUX_API_KEY: ${{ secrets.CRUX_API_KEY }} + YOUTUBE_API_KEY: ${{ secrets.YOUTUBE_API_KEY }} + YOUTUBE_CHANNEL_ID: ${{ vars.YOUTUBE_CHANNEL_ID }} + YOUTUBE_EXPLAINER_PLAYLIST_ID: ${{ vars.YOUTUBE_EXPLAINER_PLAYLIST_ID }} + YOUTUBE_DEMO_PLAYLIST_ID: ${{ vars.YOUTUBE_DEMO_PLAYLIST_ID }} + VIDEOS_AUTO_PUBLISH: ${{ vars.VIDEOS_AUTO_PUBLISH }} CLARITY_API_TOKEN: ${{ secrets.CLARITY_API_TOKEN }} CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }} CALCOM_SIGNING_SECRET: ${{ secrets.CALCOM_SIGNING_SECRET }} @@ -256,6 +261,11 @@ jobs: echo "PARTNERS_SENDER_NAME=${PARTNERS_SENDER_NAME}" echo "GOOGLE_APPLICATION_CREDENTIALS_JSON=${GOOGLE_APPLICATION_CREDENTIALS_JSON}" echo "CRUX_API_KEY=${CRUX_API_KEY}" + echo "YOUTUBE_API_KEY=${YOUTUBE_API_KEY}" + echo "YOUTUBE_CHANNEL_ID=${YOUTUBE_CHANNEL_ID}" + echo "YOUTUBE_EXPLAINER_PLAYLIST_ID=${YOUTUBE_EXPLAINER_PLAYLIST_ID}" + echo "YOUTUBE_DEMO_PLAYLIST_ID=${YOUTUBE_DEMO_PLAYLIST_ID}" + echo "VIDEOS_AUTO_PUBLISH=${VIDEOS_AUTO_PUBLISH}" echo "CLARITY_API_TOKEN=${CLARITY_API_TOKEN}" echo "CLOUDFLARE_API_TOKEN=${CLOUDFLARE_API_TOKEN}" echo "CLOUDFLARE_ACCOUNT_TAG=${CLOUDFLARE_ACCOUNT_TAG}" diff --git a/AGENTS.md b/AGENTS.md index e918c02a7..a34e4b36c 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -320,7 +320,7 @@ that triggers each and the file that sends it. ## Background jobs -Twelve cron tasks in `apps/cms/src/payload/jobs/`. All gated by `PAYLOAD_AUTO_RUN=true` — set this in `.env` to enable; omitting it (e.g. in test runs) prevents spurious fires. +Thirteen cron tasks in `apps/cms/src/payload/jobs/`. All gated by `PAYLOAD_AUTO_RUN=true` — set this in `.env` to enable; omitting it (e.g. in test runs) prevents spurious fires. | Job | Schedule (UTC) | File | |-----|---------------|------| @@ -336,6 +336,7 @@ Twelve cron tasks in `apps/cms/src/payload/jobs/`. All gated by `PAYLOAD_AUTO_RU | Meilisearch reindex | daily 05:00 | `reindex-meili.ts` | | Content-insights snapshot rebuild | daily 06:30 | `refresh-content-insights.ts` | | Core Web Vitals (CrUX) refresh | daily 06:45 | `refresh-crux.ts` (no-op without `CRUX_API_KEY`) | +| YouTube video sync | hourly :07 | `sync-youtube-videos.ts` (RSS without `YOUTUBE_API_KEY`, Data API with it; new uploads become needs-review drafts) | Update this table when adding/changing jobs. Every job needs a test file and the `PAYLOAD_AUTO_RUN` gate. diff --git a/CLAUDE.md b/CLAUDE.md index d7709d937..1f6079f24 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -311,7 +311,7 @@ that triggers each and the file that sends it. ## Background jobs -Twelve cron tasks in `apps/cms/src/payload/jobs/`. All gated by `PAYLOAD_AUTO_RUN=true` — set this in `.env` to enable; omitting it (e.g. in test runs) prevents spurious fires. +Thirteen cron tasks in `apps/cms/src/payload/jobs/`. All gated by `PAYLOAD_AUTO_RUN=true` — set this in `.env` to enable; omitting it (e.g. in test runs) prevents spurious fires. | Job | Schedule (UTC) | File | |-----|---------------|------| @@ -327,6 +327,7 @@ Twelve cron tasks in `apps/cms/src/payload/jobs/`. All gated by `PAYLOAD_AUTO_RU | Meilisearch reindex | daily 05:00 | `reindex-meili.ts` | | Content-insights snapshot rebuild | daily 06:30 | `refresh-content-insights.ts` | | Core Web Vitals (CrUX) refresh | daily 06:45 | `refresh-crux.ts` (no-op without `CRUX_API_KEY`) | +| YouTube video sync | hourly :07 | `sync-youtube-videos.ts` (RSS without `YOUTUBE_API_KEY`, Data API with it; new uploads become needs-review drafts) | Update this table when adding/changing jobs. Every job needs a test file and the `PAYLOAD_AUTO_RUN` gate. diff --git a/apps/cms/.env.example b/apps/cms/.env.example index 6b3756477..0609fefdb 100644 --- a/apps/cms/.env.example +++ b/apps/cms/.env.example @@ -325,6 +325,24 @@ CONSENT_INGEST_SECRET= CRUX_API_KEY= # CRUX_ORIGIN=https://www.cleanstart.com +# YouTube video sync (hourly task, apps/cms/src/payload/jobs/sync-youtube-videos.ts). +# YOUTUBE_API_KEY is OPTIONAL. Unset = the sync reads the channel's RSS feed (title, date, description, +# hashtags; duration is typed in by editors). Set = YouTube Data API v3 adds duration, privacy and +# embeddable checks and "missing on YouTube" detection. Google Cloud -> enable "YouTube Data API v3" +# -> API key, restricted to that API. Read only and free within quota (about 3 units per run). +YOUTUBE_API_KEY= +# Defaults to the CleanStart channel (UC13c-XRmRlkWpp6WMG9SDjw). +YOUTUBE_CHANNEL_ID= +# Optional. A new upload gets its Category suggested from which of these two YouTube playlists it is in. +# Each takes a bare playlist id or a full youtube.com/playlist?list=... URL. A video in both gets no +# suggestion. The suggestion is written once, when the draft is created, and never changes it later; the +# draft is still "Needs review" and an editor confirms it. An invalid value is ignored with a log warning. +YOUTUBE_EXPLAINER_PLAYLIST_ID= +YOUTUBE_DEMO_PLAYLIST_ID= +# Reserved: new uploads always land as needs-review drafts today, because the publish gate needs a +# format that nothing guesses. Setting this to true does not bypass the gate. +VIDEOS_AUTO_PUBLISH= + # Edge caching for anonymous reads of public content collections (seconds). # Unset or 0 = disabled, and src/proxy.ts changes no response. # diff --git a/apps/cms/scripts/data/videos-catalog.json b/apps/cms/scripts/data/videos-catalog.json new file mode 100644 index 000000000..0483fae4f --- /dev/null +++ b/apps/cms/scripts/data/videos-catalog.json @@ -0,0 +1,283 @@ +{ + "topics": [ + { + "slug": "base-images", + "name": "Base images", + "aliases": [ + "baseimage", + "containerbaseimages" + ] + }, + { + "slug": "attack-surface", + "name": "Attack surface", + "aliases": [ + "attacksurfacereduction", + "containerattacksurface" + ] + }, + { + "slug": "cves", + "name": "CVEs", + "aliases": [ + "cve", + "vulnerabilitymanagement", + "vulnerabilities" + ] + }, + { + "slug": "supply-chain", + "name": "Supply chain", + "aliases": [ + "softwaresupplychain", + "supplychainsecurity", + "softwaresupplychainsecurity" + ] + }, + { + "slug": "sbom", + "name": "SBOM", + "aliases": [ + "sboms", + "software bill of materials" + ] + }, + { + "slug": "vex", + "name": "VEX", + "aliases": [] + }, + { + "slug": "provenance", + "name": "Provenance", + "aliases": [ + "containerprovenance" + ] + }, + { + "slug": "dependencies", + "name": "Dependencies", + "aliases": [ + "dependencysecurity", + "cleanlibraries" + ] + }, + { + "slug": "posture", + "name": "Security posture", + "aliases": [ + "securityposture", + "sspm" + ] + }, + { + "slug": "container-sprawl", + "name": "Container sprawl", + "aliases": [] + }, + { + "slug": "ai", + "name": "AI", + "aliases": [] + } + ], + "videos": [ + { + "externalId": "uBItoX-x7Do", + "slug": "what-is-vex", + "title": "What Is VEX? How It Tells Which Vulnerabilities Matter", + "format": "explainer", + "topics": [ + "vex", + "sbom", + "cves" + ], + "summary": "A vulnerability scanner can flag hundreds of CVEs in a single container image, and some of them can’t be exploited where you run it. VEX (Vulnerability Exploitability eXchange) is how a software supplier tells you which ones apply.", + "durationSeconds": 101, + "publishedAt": "2026-10-13T14:17:14+00:00", + "thumb": "maxres" + }, + { + "externalId": "m9olMTM72tI", + "slug": "sboms-explained", + "title": "SBOMs Explained: What They Tell You and What They Don’t", + "format": "explainer", + "topics": [ + "sbom", + "provenance" + ], + "summary": "A critical CVE drops. Is that component in your container image at all? An SBOM answers that in seconds. Whether it puts you at risk is a separate question, and this 2-minute explainer covers both.", + "durationSeconds": 113, + "publishedAt": "2026-10-12T10:58:15+00:00", + "thumb": "maxres" + }, + { + "externalId": "UViY8slB4G8", + "slug": "clean-libraries-ide-demo", + "title": "Clean Libraries Demo: Scan and Fix Vulnerabilities in Your IDE", + "format": "demo", + "topics": [ + "dependencies", + "cves" + ], + "summary": "See Clean Libraries in action inside VS Code. As you code, it scans your project dependencies for known vulnerabilities and flags risky libraries directly in the IDE.", + "durationSeconds": 188, + "publishedAt": "2026-09-25T05:19:56+00:00", + "thumb": "maxres" + }, + { + "externalId": "dQpZX5vMgqk", + "slug": "what-is-container-sprawl", + "title": "What Is Container Sprawl?", + "format": "explainer", + "topics": [ + "container-sprawl", + "posture" + ], + "summary": "Container sprawl occurs when container images grow faster than teams can track, secure, and manage them. This video explains the causes of container sprawl, its impact on container security, and why visibility across registries, clusters, and running workloads matters.", + "durationSeconds": 98, + "publishedAt": "2026-09-11T03:47:39+00:00", + "thumb": "maxres" + }, + { + "externalId": "nh4n-o_ulvk", + "slug": "meet-cubr", + "title": "Meet CUBR, Your Guide to Trusted Software", + "format": "demo", + "topics": [ + "supply-chain", + "provenance" + ], + "summary": "Forged from precision, powered by trust, and built on the principles of secure-by-design, CUBR represents CleanStart's vision for a safer software supply chain.", + "durationSeconds": 105, + "publishedAt": "2026-07-30T10:27:29+00:00", + "thumb": "maxres" + }, + { + "externalId": "Il4uk_RPD5M", + "slug": "clean-libraries-secure-dependencies", + "title": "Clean Libraries: Secure Software Dependencies at the Source", + "format": "demo", + "topics": [ + "dependencies", + "supply-chain" + ], + "summary": "Every release introduces hundreds of dependencies, including transitive packages and AI-generated libraries. Without verification, untrusted components become part of your software supply chain.", + "durationSeconds": 159, + "publishedAt": "2026-07-24T09:56:59+00:00", + "thumb": "hq" + }, + { + "externalId": "6T3PGeattMI", + "slug": "what-is-provenance-in-containers", + "title": "What Is Provenance in Containers?", + "format": "explainer", + "topics": [ + "provenance", + "supply-chain", + "sbom" + ], + "summary": "Container provenance is a verifiable record of where a container image came from and how it was built. This video explains what provenance captures, why most images don't include it, and how a signed record lets you catch tampering before an image reaches production.", + "durationSeconds": 89, + "publishedAt": "2026-07-09T04:16:00+00:00", + "thumb": "maxres" + }, + { + "externalId": "81vUYFBS-dY", + "slug": "public-vs-hardened-vs-cleanstart-images", + "title": "Public vs Hardened vs CleanStart Images", + "format": "explainer", + "topics": [ + "base-images", + "attack-surface" + ], + "summary": "Every container starts with a base image, but not all base images are built the same. This video compares Public Images, Hardened Images, and CleanStart Source-Built Images to show how image composition impacts vulnerabilities, attack surface, and trust.", + "durationSeconds": 153, + "publishedAt": "2026-07-02T18:47:00+00:00", + "thumb": "maxres" + }, + { + "externalId": "92d9bzMUoI4", + "slug": "base-images-explained", + "title": "Base Images Explained in 60 Seconds", + "format": "explainer", + "topics": [ + "base-images" + ], + "summary": "A base image is the foundation that everything else is built on, from dependencies to application code. In this 60-second explainer, discover what a base image is and why choosing the right one matters.", + "durationSeconds": 59, + "publishedAt": "2026-06-24T20:23:34+00:00", + "thumb": "maxres" + }, + { + "externalId": "1pPSjboIzoU", + "slug": "what-is-a-shell-less-container", + "title": "What Is a Shell-Less Container? Container Security Decoded", + "format": "explainer", + "topics": [ + "attack-surface", + "base-images" + ], + "summary": "Most production containers ship with a shell they never use. It sits inside the runtime as convenience tooling, and it quietly becomes one of the easiest paths for an attacker to turn a single vulnerability into full operational control.", + "durationSeconds": 131, + "publishedAt": "2026-06-10T12:14:14+00:00", + "thumb": "maxres" + }, + { + "externalId": "GqFPH4KKqpA", + "slug": "cleansight-software-security-posture-management", + "title": "CleanSight: Complete Software Security Posture Management", + "format": "demo", + "topics": [ + "posture", + "cves" + ], + "summary": "Introducing CleanSight, a complete Software Security Posture Management platform designed to give you unified visibility across your infrastructure, clusters, and containers.", + "durationSeconds": 73, + "publishedAt": "2026-05-27T13:36:30+00:00", + "thumb": "maxres" + }, + { + "externalId": "juh9IYHXXZA", + "slug": "cve-fatigue-explained", + "title": "CVE Fatigue Explained: Why Your Vulnerability Backlog Never Gets Shorter", + "format": "explainer", + "topics": [ + "cves", + "base-images" + ], + "summary": "2,000 CVEs flagged. Your team patches, rebuilds, redeploys. Next week: 500 more. The cycle repeats. This video breaks down what CVE fatigue is, where it actually comes from, and how teams escape it.", + "durationSeconds": 154, + "publishedAt": "2026-05-22T04:46:17+00:00", + "thumb": "maxres" + }, + { + "externalId": "neWpaEOf3XM", + "slug": "busybox-in-containers", + "title": "BusyBox in Containers: The Risk Nobody Talks About", + "format": "explainer", + "topics": [ + "attack-surface", + "base-images" + ], + "summary": "BusyBox ships in 85% of lightweight container images. A single C binary from 1996, bundling 300+ utilities into one executable. 47 CVEs in five years. Nobody on your team installed it. Nobody reviewed it. It just showed up.", + "durationSeconds": 117, + "publishedAt": "2026-05-13T12:55:14+00:00", + "thumb": "maxres" + }, + { + "externalId": "WvxSvEzXbiQ", + "slug": "what-is-attack-surface-in-containers", + "title": "What Is Attack Surface in Containers?", + "format": "explainer", + "topics": [ + "attack-surface", + "base-images" + ], + "summary": "Your container image has hundreds of packages. How many does your application actually need? Probably fewer than you think, and every extra one is a potential vulnerability.", + "durationSeconds": 160, + "publishedAt": "2026-04-30T06:56:26+00:00", + "thumb": "maxres" + } + ] +} diff --git a/apps/cms/scripts/seed-videos-from-catalog.ts b/apps/cms/scripts/seed-videos-from-catalog.ts new file mode 100644 index 000000000..87823813f --- /dev/null +++ b/apps/cms/scripts/seed-videos-from-catalog.ts @@ -0,0 +1,228 @@ +#!/usr/bin/env -S node --no-warnings --experimental-strip-types +/** + * Moves the code catalog behind /videos (scripts/data/videos-catalog.json, a + * snapshot of apps/web/src/lib/videos-catalog.ts) into the `videos` and + * `videoTopics` collections, keeping every live slug. + * + * Order: topics, then videos (create what is missing, apply the catalog's + * editorial fields and publish), then one YouTube sync pass to fill in the + * synced snapshot (description, tags, thumbnails) from the channel's RSS feed. + * + * SAFETY: idempotent; a second run plans nothing. A slug already on a document + * is a URL commitment, so a different slug (or a slug held by another video) + * stops the run before anything is written. Dry run is the default. + * Publish webhooks are skipped for the bulk write and web revalidation is + * suppressed per document, then sent once at the end. + * + * Run from apps/cms with the env file loaded: + * pnpm exec tsx --env-file=.env scripts/seed-videos-from-catalog.ts + * pnpm exec tsx --env-file=.env scripts/seed-videos-from-catalog.ts --apply + */ +import { readFileSync } from 'node:fs'; +import { fileURLToPath } from 'node:url'; +import { getPayload } from 'payload'; + +import config from '../src/payload.config.ts'; +import { flushPendingAfterCommitEffects } from '../src/payload/lib/after-commit.ts'; +import { resolveYoutubeCredentials } from '../src/payload/lib/integrations/credentials.ts'; +import { + type CatalogTopicSeed, + type CatalogVideoSeed, + type ExistingBackfillVideo, + type ExistingTopic, + planBackfill, +} from '../src/payload/lib/videos/plan-backfill.ts'; +import { type RunSyncPayload, runYoutubeSync } from '../src/payload/lib/videos/run-sync.ts'; +import { revalidateWeb } from '../src/payload/lib/web-revalidate.ts'; +import { createYoutubeClient } from '../src/payload/lib/videos/youtube-client.ts'; +import { createYoutubeRssClient } from '../src/payload/lib/videos/youtube-rss-client.ts'; + +type DocId = number | string; +type Doc = Record; + +const catalog = JSON.parse( + readFileSync(fileURLToPath(new URL('./data/videos-catalog.json', import.meta.url)), 'utf8'), +) as { topics: CatalogTopicSeed[]; videos: CatalogVideoSeed[] }; + +const aliasRows = (aliases: readonly string[]): { value: string }[] => aliases.map((value) => ({ value })); +const refId = (ref: unknown): DocId | null => { + if (ref && typeof ref === 'object') return (ref as { id?: DocId }).id ?? null; + return typeof ref === 'string' || typeof ref === 'number' ? ref : null; +}; + +async function run(): Promise { + const apply = process.argv.includes('--apply'); + if (apply) process.env.WEB_REVALIDATE_SUPPRESS = 'true'; + const payload = await getPayload({ config }); + + const readAll = async (collection: 'videos' | 'videoTopics'): Promise => { + const docs: Doc[] = []; + for (let page = 1; ; page += 1) { + const res = await payload.find({ collection, limit: 200, page, depth: 0, draft: true, overrideAccess: true }); + docs.push(...(res.docs as unknown as Doc[])); + if (!res.hasNextPage) return docs; + } + }; + + const loadExisting = async (): Promise<{ + topics: ExistingTopic[]; + topicIdBySlug: Map; + videos: ExistingBackfillVideo[]; + }> => { + const [topicDocs, videoDocs] = await Promise.all([readAll('videoTopics'), readAll('videos')]); + const topicSlugById = new Map(topicDocs.map((d) => [d.id as DocId, String(d.slug)])); + return { + topics: topicDocs.map((d) => ({ + id: d.id as DocId, + slug: String(d.slug), + name: String(d.name), + aliases: ((d.aliases as { value?: string }[] | undefined) ?? []).flatMap((a) => (a.value ? [a.value] : [])), + })), + topicIdBySlug: new Map(topicDocs.map((d) => [String(d.slug), d.id as DocId])), + videos: videoDocs.map((d) => ({ + id: d.id as DocId, + externalId: String(d.externalId ?? ''), + slug: typeof d.slug === 'string' && d.slug ? d.slug : null, + title: typeof d.title === 'string' ? d.title : null, + format: typeof d.format === 'string' ? d.format : null, + topicSlugs: ((d.topics as unknown[] | undefined) ?? []).flatMap((t) => { + const id = refId(t); + const slug = id === null ? undefined : topicSlugById.get(id); + return slug ? [slug] : []; + }), + summary: typeof d.summary === 'string' ? d.summary : null, + status: d._status === 'published' ? ('published' as const) : ('draft' as const), + syncState: typeof d.syncState === 'string' ? d.syncState : null, + })), + }; + }; + + const describe = (plan: ReturnType): void => { + payload.logger.info( + { + topicsCreate: plan.topics.create.length, + topicsUpdate: plan.topics.update.length, + topicsUnchanged: plan.topics.unchanged, + videosCreate: plan.videos.create.length, + videosUpdate: plan.videos.update.map((u) => `${u.video.slug}: ${u.reasons.join('+')}`), + videosUnchanged: plan.videos.unchanged, + conflicts: plan.videos.conflicts, + }, + apply ? 'videos backfill plan' : 'videos backfill plan (dry run, pass --apply to write)', + ); + }; + + let existing = await loadExisting(); + let plan = planBackfill({ + catalogTopics: catalog.topics, + catalogVideos: catalog.videos, + existingTopics: existing.topics, + existingVideos: existing.videos, + }); + describe(plan); + if (plan.videos.conflicts.length > 0) { + payload.logger.error({ conflicts: plan.videos.conflicts }, 'slug conflicts, nothing written'); + process.exit(1); + } + if (!apply) process.exit(0); + + const write = { overrideAccess: true, depth: 0, context: { skipPublishWebhooks: true } } as const; + + for (const topic of plan.topics.create) { + await payload.create({ + collection: 'videoTopics', + data: { slug: topic.slug, name: topic.name, aliases: aliasRows(topic.aliases) }, + ...write, + }); + } + for (const { id, topic } of plan.topics.update) { + await payload.update({ + collection: 'videoTopics', + id, + data: { name: topic.name, aliases: aliasRows(topic.aliases) }, + ...write, + }); + } + + existing = await loadExisting(); + const topicIds = (slugs: readonly string[]): DocId[] => + slugs.map((slug) => { + const id = existing.topicIdBySlug.get(slug); + if (id === undefined) throw new Error(`Catalog topic "${slug}" has no videoTopics document`); + return id; + }); + const editorial = (v: CatalogVideoSeed): Doc => ({ + slug: v.slug, + title: v.title, + format: v.format, + topics: topicIds(v.topics), + summary: v.summary, + syncState: 'synced', + _status: 'published', + }); + + plan = planBackfill({ + catalogTopics: catalog.topics, + catalogVideos: catalog.videos, + existingTopics: existing.topics, + existingVideos: existing.videos, + }); + const fetchedAt = new Date().toISOString(); + for (const v of plan.videos.create) { + await payload.create({ + collection: 'videos', + data: { + provider: 'youtube', + externalId: v.externalId, + ...editorial(v), + source: { + title: v.title, + description: '', + durationSeconds: v.durationSeconds, + publishedAt: v.publishedAt, + tags: [], + privacyStatus: 'public', + embeddable: true, + hasMaxres: v.thumb === 'maxres', + thumbnailUrl: `https://i.ytimg.com/vi/${v.externalId}/${v.thumb === 'maxres' ? 'maxresdefault' : 'hqdefault'}.jpg`, + fetchedAt, + }, + }, + ...write, + }); + } + for (const { id, video } of plan.videos.update) { + await payload.update({ collection: 'videos', id, data: editorial(video), ...write }); + } + + // Fills description, tags and thumbnails; keeps the catalog's duration when the RSS feed has none. + const creds = resolveYoutubeCredentials(); + const synced = await runYoutubeSync({ + payload: payload as unknown as RunSyncPayload, + client: creds.apiKey ? createYoutubeClient({ apiKey: creds.apiKey }) : createYoutubeRssClient(), + channelId: creds.channelId, + autoPublish: false, + }); + payload.logger.info(synced, 'videos backfill: sync pass done'); + + await flushPendingAfterCommitEffects(); + process.env.WEB_REVALIDATE_SUPPRESS = 'false'; + const revalidated = await revalidateWeb(payload, { + tags: ['videos'], + paths: ['/videos', '/sitemap.xml', ...catalog.videos.map((v) => `/videos/${v.slug}`)], + }); + payload.logger.info(revalidated, 'videos backfill: web revalidation'); + + const after = planBackfill({ + catalogTopics: catalog.topics, + catalogVideos: catalog.videos, + ...(await loadExisting().then((e) => ({ existingTopics: e.topics, existingVideos: e.videos }))), + }); + describe(after); + process.exit(0); +} + +run().catch((error: unknown) => { + console.error(error); + process.exit(1); +}); diff --git a/apps/cms/src/app/(payload)/admin/importMap.js b/apps/cms/src/app/(payload)/admin/importMap.js index 3526fc30a..3a76a6b31 100644 --- a/apps/cms/src/app/(payload)/admin/importMap.js +++ b/apps/cms/src/app/(payload)/admin/importMap.js @@ -150,6 +150,12 @@ import { PointField as PointField_dc800e7ca3d2eb7ea39fd4bc6138c687 } from '@/pay import { RadioField as RadioField_31aae9b520654e49d40e444571bafe41 } from '@/payload/admin/components/fields/RadioField.tsx' import { TabsField as TabsField_c014faddfba6da88b9efedda256f4990 } from '@/payload/admin/components/fields/TabsField.tsx' import { RowField as RowField_1ef7316c3d3f9450222b0eb086a8e243 } from '@/payload/admin/components/fields/RowField.tsx' +import { VideoSourceSummary as VideoSourceSummary_c96fbcdf17e1ea6b0ec94f543434cd27 } from '@/payload/admin/components/VideoSourceSummary.tsx' +import { VideoThumbCell as VideoThumbCell_a6fc69bad4fd684c1000e3675bb5a77a } from '@/payload/admin/components/VideoThumbCell.tsx' +import { VideoLengthCell as VideoLengthCell_4ae28695c16ecdf12c15a176eb6e0774 } from '@/payload/admin/components/VideoLengthCell.tsx' +import { VideoStateCell as VideoStateCell_70d255ecab4ca7e768a3cff708566031 } from '@/payload/admin/components/VideoStateCell.tsx' +import { VideoTopicSuggestions as VideoTopicSuggestions_354abe4af848e2bbd6afc4620ae30591 } from '@/payload/admin/components/VideoTopicSuggestions.tsx' +import { UseDescriptionSummary as UseDescriptionSummary_19de247a3719b8cb4c2371ecb84ed232 } from '@/payload/admin/components/UseDescriptionSummary.tsx' /** @type import('payload').ImportMap */ export const importMap = { @@ -304,5 +310,11 @@ export const importMap = { "@/payload/admin/components/fields/PointField.tsx#PointField": PointField_dc800e7ca3d2eb7ea39fd4bc6138c687, "@/payload/admin/components/fields/RadioField.tsx#RadioField": RadioField_31aae9b520654e49d40e444571bafe41, "@/payload/admin/components/fields/TabsField.tsx#TabsField": TabsField_c014faddfba6da88b9efedda256f4990, - "@/payload/admin/components/fields/RowField.tsx#RowField": RowField_1ef7316c3d3f9450222b0eb086a8e243 + "@/payload/admin/components/fields/RowField.tsx#RowField": RowField_1ef7316c3d3f9450222b0eb086a8e243, + "@/payload/admin/components/VideoSourceSummary.tsx#VideoSourceSummary": VideoSourceSummary_c96fbcdf17e1ea6b0ec94f543434cd27, + "@/payload/admin/components/VideoThumbCell.tsx#VideoThumbCell": VideoThumbCell_a6fc69bad4fd684c1000e3675bb5a77a, + "@/payload/admin/components/VideoLengthCell.tsx#VideoLengthCell": VideoLengthCell_4ae28695c16ecdf12c15a176eb6e0774, + "@/payload/admin/components/VideoStateCell.tsx#VideoStateCell": VideoStateCell_70d255ecab4ca7e768a3cff708566031, + "@/payload/admin/components/VideoTopicSuggestions.tsx#VideoTopicSuggestions": VideoTopicSuggestions_354abe4af848e2bbd6afc4620ae30591, + "@/payload/admin/components/UseDescriptionSummary.tsx#UseDescriptionSummary": UseDescriptionSummary_19de247a3719b8cb4c2371ecb84ed232, } diff --git a/apps/cms/src/migrations/20261031_100000_add_videos.ts b/apps/cms/src/migrations/20261031_100000_add_videos.ts new file mode 100644 index 000000000..4d35b452c --- /dev/null +++ b/apps/cms/src/migrations/20261031_100000_add_videos.ts @@ -0,0 +1,232 @@ +import { type MigrateDownArgs, type MigrateUpArgs, sql } from '@payloadcms/db-postgres' + +/** + * Adds the `videos` collection (editorial videos listed at /videos, fed by the + * YouTube sync) and the `videoTopics` taxonomy. `videos` is versioned with + * drafts, so it has a `_videos_v` mirror. Videos play inline on the listing, so + * there is no SEO group, no `schemaAddons` field and no `blocks_*` tables, and + * the slug column exists but is hidden in the admin. + * + * Outside the new tables it touches three things: + * - `payload_locked_documents_rels.videos_id` / `.video_topics_id`: Payload + * loads that whole row on any document-lock read, so leaving them out breaks + * every collection's list and edit view (the 2026-06-24 taxonomy incident). + * - `enum_integrations_routing_events`, `enum_integrations_teams_config_mentions_trigger_on` + * and `enum_webhooks_dead_letter_event` gain 'video.needs-review' (the webhook + * event the sync fires when a new upload arrives as a draft; the dead-letter + * row for a failed delivery stores it). + * - `enum_payload_jobs_task_slug` and `enum_payload_jobs_log_task_slug` gain + * 'syncYoutubeVideos' (the hourly sync task; a scheduled job whose slug is + * not in the enum fails to enqueue). + * + * Hand-reviewed rather than from `payload migrate:create` (it diffs against a + * June snapshot and would emit hundreds of unrelated statements). The DDL was + * extracted from the schema Payload pushed onto an empty scratch database for + * the final collection definitions, and made idempotent by mechanical + * rewriting: IF NOT EXISTS on tables and indexes, duplicate_object guards on + * types and foreign keys. It was then applied twice to a clone of a pre-videos + * database and the catalogs (columns, enums, indexes, constraints) compared + * with the push: they match. + * + * This migration has not shipped, so it was edited in place when the SEO + * group, the 'talk' format and the link-field and routing-collection 'videos' + * targets were dropped from the collection. + * + * Idempotent throughout: local databases run `push: true` and already have + * these objects. + */ + +const UP = [ + sql`DO $$ BEGIN CREATE TYPE "public"."enum__videos_v_version_format" AS ENUM('explainer', 'demo'); EXCEPTION WHEN duplicate_object THEN null; END $$;`, + sql`DO $$ BEGIN CREATE TYPE "public"."enum__videos_v_version_provider" AS ENUM('youtube', 'vimeo'); EXCEPTION WHEN duplicate_object THEN null; END $$;`, + sql`DO $$ BEGIN CREATE TYPE "public"."enum__videos_v_version_status" AS ENUM('draft', 'published'); EXCEPTION WHEN duplicate_object THEN null; END $$;`, + sql`DO $$ BEGIN CREATE TYPE "public"."enum__videos_v_version_sync_state" AS ENUM('synced', 'needs-review', 'missing', 'manual'); EXCEPTION WHEN duplicate_object THEN null; END $$;`, + sql`DO $$ BEGIN CREATE TYPE "public"."enum_videos_format" AS ENUM('explainer', 'demo'); EXCEPTION WHEN duplicate_object THEN null; END $$;`, + sql`DO $$ BEGIN CREATE TYPE "public"."enum_videos_provider" AS ENUM('youtube', 'vimeo'); EXCEPTION WHEN duplicate_object THEN null; END $$;`, + sql`DO $$ BEGIN CREATE TYPE "public"."enum_videos_status" AS ENUM('draft', 'published'); EXCEPTION WHEN duplicate_object THEN null; END $$;`, + sql`DO $$ BEGIN CREATE TYPE "public"."enum_videos_sync_state" AS ENUM('synced', 'needs-review', 'missing', 'manual'); EXCEPTION WHEN duplicate_object THEN null; END $$;`, + sql`ALTER TYPE "public"."enum_integrations_routing_events" ADD VALUE IF NOT EXISTS 'video.needs-review';`, + sql`ALTER TYPE "public"."enum_integrations_teams_config_mentions_trigger_on" ADD VALUE IF NOT EXISTS 'video.needs-review';`, + sql`ALTER TYPE "public"."enum_webhooks_dead_letter_event" ADD VALUE IF NOT EXISTS 'video.needs-review';`, + sql`ALTER TYPE "public"."enum_payload_jobs_task_slug" ADD VALUE IF NOT EXISTS 'syncYoutubeVideos' BEFORE 'schedulePublish';`, + sql`ALTER TYPE "public"."enum_payload_jobs_log_task_slug" ADD VALUE IF NOT EXISTS 'syncYoutubeVideos' BEFORE 'schedulePublish';`, + sql`CREATE TABLE IF NOT EXISTS "_videos_v" ( + "id" serial PRIMARY KEY NOT NULL, + "parent_id" integer, + "version_title" varchar, + "version_slug" varchar, + "version_provider" "public"."enum__videos_v_version_provider" DEFAULT 'youtube'::"public"."enum__videos_v_version_provider", + "version_external_id" varchar, + "version_format" "public"."enum__videos_v_version_format", + "version_summary" varchar, + "version_duration_label" varchar, + "version_featured" boolean DEFAULT false, + "version_sync_state" "public"."enum__videos_v_version_sync_state" DEFAULT 'manual'::"public"."enum__videos_v_version_sync_state", + "version_source_title" varchar, + "version_source_description" varchar, + "version_source_duration_seconds" numeric, + "version_source_published_at" timestamp(3) with time zone, + "version_source_privacy_status" varchar, + "version_source_embeddable" boolean, + "version_source_has_maxres" boolean, + "version_source_thumbnail_url" varchar, + "version_source_content_hash" varchar, + "version_source_fetched_at" timestamp(3) with time zone, + "version_published_at" timestamp(3) with time zone, + "version_display_published_at" timestamp(3) with time zone, + "version_content_updated_at" timestamp(3) with time zone, + "version_updated_at" timestamp(3) with time zone, + "version_created_at" timestamp(3) with time zone, + "version__status" "public"."enum__videos_v_version_status" DEFAULT 'draft'::"public"."enum__videos_v_version_status", + "created_at" timestamp(3) with time zone DEFAULT now() NOT NULL, + "updated_at" timestamp(3) with time zone DEFAULT now() NOT NULL, + "latest" boolean +);`, + sql`CREATE TABLE IF NOT EXISTS "_videos_v_rels" ( + "id" serial PRIMARY KEY NOT NULL, + "order" integer, + "parent_id" integer NOT NULL, + "path" varchar NOT NULL, + "video_topics_id" integer +);`, + sql`CREATE TABLE IF NOT EXISTS "_videos_v_version_source_tags" ( + "_order" integer NOT NULL, + "_parent_id" integer NOT NULL, + "id" serial PRIMARY KEY NOT NULL, + "value" varchar, + "_uuid" varchar +);`, + sql`CREATE TABLE IF NOT EXISTS "video_topics" ( + "id" serial PRIMARY KEY NOT NULL, + "name" varchar NOT NULL, + "slug" varchar NOT NULL, + "description" varchar, + "updated_at" timestamp(3) with time zone DEFAULT now() NOT NULL, + "created_at" timestamp(3) with time zone DEFAULT now() NOT NULL +);`, + sql`CREATE TABLE IF NOT EXISTS "video_topics_aliases" ( + "_order" integer NOT NULL, + "_parent_id" integer NOT NULL, + "id" varchar PRIMARY KEY NOT NULL, + "value" varchar NOT NULL +);`, + sql`CREATE TABLE IF NOT EXISTS "videos" ( + "id" serial PRIMARY KEY NOT NULL, + "title" varchar, + "slug" varchar, + "provider" "public"."enum_videos_provider" DEFAULT 'youtube'::"public"."enum_videos_provider", + "external_id" varchar, + "format" "public"."enum_videos_format", + "summary" varchar, + "duration_label" varchar, + "featured" boolean DEFAULT false, + "sync_state" "public"."enum_videos_sync_state" DEFAULT 'manual'::"public"."enum_videos_sync_state", + "source_title" varchar, + "source_description" varchar, + "source_duration_seconds" numeric, + "source_published_at" timestamp(3) with time zone, + "source_privacy_status" varchar, + "source_embeddable" boolean, + "source_has_maxres" boolean, + "source_thumbnail_url" varchar, + "source_content_hash" varchar, + "source_fetched_at" timestamp(3) with time zone, + "published_at" timestamp(3) with time zone, + "display_published_at" timestamp(3) with time zone, + "content_updated_at" timestamp(3) with time zone, + "updated_at" timestamp(3) with time zone DEFAULT now() NOT NULL, + "created_at" timestamp(3) with time zone DEFAULT now() NOT NULL, + "_status" "public"."enum_videos_status" DEFAULT 'draft'::"public"."enum_videos_status" +);`, + sql`CREATE TABLE IF NOT EXISTS "videos_rels" ( + "id" serial PRIMARY KEY NOT NULL, + "order" integer, + "parent_id" integer NOT NULL, + "path" varchar NOT NULL, + "video_topics_id" integer +);`, + sql`CREATE TABLE IF NOT EXISTS "videos_source_tags" ( + "_order" integer NOT NULL, + "_parent_id" integer NOT NULL, + "id" varchar PRIMARY KEY NOT NULL, + "value" varchar +);`, + sql`ALTER TABLE "payload_locked_documents_rels" ADD COLUMN IF NOT EXISTS "videos_id" integer;`, + sql`ALTER TABLE "payload_locked_documents_rels" ADD COLUMN IF NOT EXISTS "video_topics_id" integer;`, + sql`CREATE INDEX IF NOT EXISTS "_videos_v_created_at_idx" ON "_videos_v" USING btree ("created_at");`, + sql`CREATE INDEX IF NOT EXISTS "_videos_v_latest_idx" ON "_videos_v" USING btree ("latest");`, + sql`CREATE INDEX IF NOT EXISTS "_videos_v_parent_idx" ON "_videos_v" USING btree ("parent_id");`, + sql`CREATE INDEX IF NOT EXISTS "_videos_v_rels_order_idx" ON "_videos_v_rels" USING btree ("order");`, + sql`CREATE INDEX IF NOT EXISTS "_videos_v_rels_parent_idx" ON "_videos_v_rels" USING btree ("parent_id");`, + sql`CREATE INDEX IF NOT EXISTS "_videos_v_rels_path_idx" ON "_videos_v_rels" USING btree ("path");`, + sql`CREATE INDEX IF NOT EXISTS "_videos_v_rels_video_topics_id_idx" ON "_videos_v_rels" USING btree ("video_topics_id");`, + sql`CREATE INDEX IF NOT EXISTS "_videos_v_updated_at_idx" ON "_videos_v" USING btree ("updated_at");`, + sql`CREATE INDEX IF NOT EXISTS "_videos_v_version_source_tags_order_idx" ON "_videos_v_version_source_tags" USING btree ("_order");`, + sql`CREATE INDEX IF NOT EXISTS "_videos_v_version_source_tags_parent_id_idx" ON "_videos_v_version_source_tags" USING btree ("_parent_id");`, + sql`CREATE INDEX IF NOT EXISTS "_videos_v_version_version__status_idx" ON "_videos_v" USING btree ("version__status");`, + sql`CREATE INDEX IF NOT EXISTS "_videos_v_version_version_created_at_idx" ON "_videos_v" USING btree ("version_created_at");`, + sql`CREATE INDEX IF NOT EXISTS "_videos_v_version_version_display_published_at_idx" ON "_videos_v" USING btree ("version_display_published_at");`, + sql`CREATE INDEX IF NOT EXISTS "_videos_v_version_version_external_id_idx" ON "_videos_v" USING btree ("version_external_id");`, + sql`CREATE INDEX IF NOT EXISTS "_videos_v_version_version_slug_idx" ON "_videos_v" USING btree ("version_slug");`, + sql`CREATE INDEX IF NOT EXISTS "_videos_v_version_version_updated_at_idx" ON "_videos_v" USING btree ("version_updated_at");`, + sql`CREATE INDEX IF NOT EXISTS "payload_locked_documents_rels_video_topics_id_idx" ON "payload_locked_documents_rels" USING btree ("video_topics_id");`, + sql`CREATE INDEX IF NOT EXISTS "payload_locked_documents_rels_videos_id_idx" ON "payload_locked_documents_rels" USING btree ("videos_id");`, + sql`CREATE INDEX IF NOT EXISTS "video_topics_aliases_order_idx" ON "video_topics_aliases" USING btree ("_order");`, + sql`CREATE INDEX IF NOT EXISTS "video_topics_aliases_parent_id_idx" ON "video_topics_aliases" USING btree ("_parent_id");`, + sql`CREATE INDEX IF NOT EXISTS "video_topics_created_at_idx" ON "video_topics" USING btree ("created_at");`, + sql`CREATE UNIQUE INDEX IF NOT EXISTS "video_topics_slug_idx" ON "video_topics" USING btree ("slug");`, + sql`CREATE INDEX IF NOT EXISTS "video_topics_updated_at_idx" ON "video_topics" USING btree ("updated_at");`, + sql`CREATE INDEX IF NOT EXISTS "videos__status_idx" ON "videos" USING btree ("_status");`, + sql`CREATE INDEX IF NOT EXISTS "videos_created_at_idx" ON "videos" USING btree ("created_at");`, + sql`CREATE INDEX IF NOT EXISTS "videos_display_published_at_idx" ON "videos" USING btree ("display_published_at");`, + sql`CREATE UNIQUE INDEX IF NOT EXISTS "videos_external_id_idx" ON "videos" USING btree ("external_id");`, + sql`CREATE INDEX IF NOT EXISTS "videos_rels_order_idx" ON "videos_rels" USING btree ("order");`, + sql`CREATE INDEX IF NOT EXISTS "videos_rels_parent_idx" ON "videos_rels" USING btree ("parent_id");`, + sql`CREATE INDEX IF NOT EXISTS "videos_rels_path_idx" ON "videos_rels" USING btree ("path");`, + sql`CREATE INDEX IF NOT EXISTS "videos_rels_video_topics_id_idx" ON "videos_rels" USING btree ("video_topics_id");`, + sql`CREATE UNIQUE INDEX IF NOT EXISTS "videos_slug_idx" ON "videos" USING btree ("slug");`, + sql`CREATE INDEX IF NOT EXISTS "videos_source_tags_order_idx" ON "videos_source_tags" USING btree ("_order");`, + sql`CREATE INDEX IF NOT EXISTS "videos_source_tags_parent_id_idx" ON "videos_source_tags" USING btree ("_parent_id");`, + sql`CREATE INDEX IF NOT EXISTS "videos_updated_at_idx" ON "videos" USING btree ("updated_at");`, + sql`DO $$ BEGIN ALTER TABLE "_videos_v" ADD CONSTRAINT "_videos_v_parent_id_videos_id_fk" FOREIGN KEY ("parent_id") REFERENCES "public"."videos"("id") ON DELETE set null ON UPDATE no action; EXCEPTION WHEN duplicate_object THEN null; END $$;`, + sql`DO $$ BEGIN ALTER TABLE "_videos_v_rels" ADD CONSTRAINT "_videos_v_rels_parent_fk" FOREIGN KEY ("parent_id") REFERENCES "public"."_videos_v"("id") ON DELETE cascade ON UPDATE no action; EXCEPTION WHEN duplicate_object THEN null; END $$;`, + sql`DO $$ BEGIN ALTER TABLE "_videos_v_rels" ADD CONSTRAINT "_videos_v_rels_video_topics_fk" FOREIGN KEY ("video_topics_id") REFERENCES "public"."video_topics"("id") ON DELETE cascade ON UPDATE no action; EXCEPTION WHEN duplicate_object THEN null; END $$;`, + sql`DO $$ BEGIN ALTER TABLE "_videos_v_version_source_tags" ADD CONSTRAINT "_videos_v_version_source_tags_parent_id_fk" FOREIGN KEY ("_parent_id") REFERENCES "public"."_videos_v"("id") ON DELETE cascade ON UPDATE no action; EXCEPTION WHEN duplicate_object THEN null; END $$;`, + sql`DO $$ BEGIN ALTER TABLE "payload_locked_documents_rels" ADD CONSTRAINT "payload_locked_documents_rels_video_topics_fk" FOREIGN KEY ("video_topics_id") REFERENCES "public"."video_topics"("id") ON DELETE cascade ON UPDATE no action; EXCEPTION WHEN duplicate_object THEN null; END $$;`, + sql`DO $$ BEGIN ALTER TABLE "payload_locked_documents_rels" ADD CONSTRAINT "payload_locked_documents_rels_videos_fk" FOREIGN KEY ("videos_id") REFERENCES "public"."videos"("id") ON DELETE cascade ON UPDATE no action; EXCEPTION WHEN duplicate_object THEN null; END $$;`, + sql`DO $$ BEGIN ALTER TABLE "video_topics_aliases" ADD CONSTRAINT "video_topics_aliases_parent_id_fk" FOREIGN KEY ("_parent_id") REFERENCES "public"."video_topics"("id") ON DELETE cascade ON UPDATE no action; EXCEPTION WHEN duplicate_object THEN null; END $$;`, + sql`DO $$ BEGIN ALTER TABLE "videos_rels" ADD CONSTRAINT "videos_rels_parent_fk" FOREIGN KEY ("parent_id") REFERENCES "public"."videos"("id") ON DELETE cascade ON UPDATE no action; EXCEPTION WHEN duplicate_object THEN null; END $$;`, + sql`DO $$ BEGIN ALTER TABLE "videos_rels" ADD CONSTRAINT "videos_rels_video_topics_fk" FOREIGN KEY ("video_topics_id") REFERENCES "public"."video_topics"("id") ON DELETE cascade ON UPDATE no action; EXCEPTION WHEN duplicate_object THEN null; END $$;`, + sql`DO $$ BEGIN ALTER TABLE "videos_source_tags" ADD CONSTRAINT "videos_source_tags_parent_id_fk" FOREIGN KEY ("_parent_id") REFERENCES "public"."videos"("id") ON DELETE cascade ON UPDATE no action; EXCEPTION WHEN duplicate_object THEN null; END $$;`, +] + +export async function up({ db }: MigrateUpArgs): Promise { + for (const statement of UP) { + await db.execute(statement) + } +} + +export async function down({ db }: MigrateDownArgs): Promise { + await db.execute(sql`ALTER TABLE "payload_locked_documents_rels" DROP COLUMN IF EXISTS "videos_id";`) + await db.execute(sql`ALTER TABLE "payload_locked_documents_rels" DROP COLUMN IF EXISTS "video_topics_id";`) + await db.execute(sql`DROP TABLE IF EXISTS "videos_source_tags" CASCADE;`) + await db.execute(sql`DROP TABLE IF EXISTS "videos_rels" CASCADE;`) + await db.execute(sql`DROP TABLE IF EXISTS "videos" CASCADE;`) + await db.execute(sql`DROP TABLE IF EXISTS "video_topics_aliases" CASCADE;`) + await db.execute(sql`DROP TABLE IF EXISTS "video_topics" CASCADE;`) + await db.execute(sql`DROP TABLE IF EXISTS "_videos_v_version_source_tags" CASCADE;`) + await db.execute(sql`DROP TABLE IF EXISTS "_videos_v_rels" CASCADE;`) + await db.execute(sql`DROP TABLE IF EXISTS "_videos_v" CASCADE;`) + await db.execute(sql`DROP TYPE IF EXISTS "public"."enum__videos_v_version_format";`) + await db.execute(sql`DROP TYPE IF EXISTS "public"."enum__videos_v_version_provider";`) + await db.execute(sql`DROP TYPE IF EXISTS "public"."enum__videos_v_version_status";`) + await db.execute(sql`DROP TYPE IF EXISTS "public"."enum__videos_v_version_sync_state";`) + await db.execute(sql`DROP TYPE IF EXISTS "public"."enum_videos_format";`) + await db.execute(sql`DROP TYPE IF EXISTS "public"."enum_videos_provider";`) + await db.execute(sql`DROP TYPE IF EXISTS "public"."enum_videos_status";`) + await db.execute(sql`DROP TYPE IF EXISTS "public"."enum_videos_sync_state";`) + // Postgres cannot drop a single enum label, so 'video.needs-review' stays on + // enum_integrations_routing_events, enum_integrations_teams_config_mentions_trigger_on and + // enum_webhooks_dead_letter_event, and 'syncYoutubeVideos' on the two payload_jobs + // task-slug enums. Unused labels are harmless. +} diff --git a/apps/cms/src/migrations/index.ts b/apps/cms/src/migrations/index.ts index b50faa3a5..9befeccc6 100644 --- a/apps/cms/src/migrations/index.ts +++ b/apps/cms/src/migrations/index.ts @@ -66,6 +66,7 @@ import * as migration_20261023_120000_add_advisories from './20261023_120000_add import * as migration_20261027_120000_advisories_structured_fields from './20261027_120000_advisories_structured_fields'; import * as migration_20261028_120000_advisories_cover_title from './20261028_120000_advisories_cover_title'; import * as migration_20261026_120000_add_missing_job_log_task_slugs from './20261026_120000_add_missing_job_log_task_slugs'; +import * as migration_20261031_100000_add_videos from './20261031_100000_add_videos'; export const migrations = [ { @@ -383,4 +384,9 @@ export const migrations = [ down: migration_20261028_120000_advisories_cover_title.down, name: '20261028_120000_advisories_cover_title', }, + { + up: migration_20261031_100000_add_videos.up, + down: migration_20261031_100000_add_videos.down, + name: '20261031_100000_add_videos', + }, ]; diff --git a/apps/cms/src/payload-types.ts b/apps/cms/src/payload-types.ts index 01c829066..b38248caa 100644 --- a/apps/cms/src/payload-types.ts +++ b/apps/cms/src/payload-types.ts @@ -69,6 +69,8 @@ export interface Config { collections: { blogs: Blog; advisories: Advisory; + videos: Video; + videoTopics: VideoTopic; news: News; guides: Guide; 'case-studies': CaseStudy; @@ -127,6 +129,8 @@ export interface Config { collectionsSelect: { blogs: BlogsSelect | BlogsSelect; advisories: AdvisoriesSelect | AdvisoriesSelect; + videos: VideosSelect | VideosSelect; + videoTopics: VideoTopicsSelect | VideoTopicsSelect; news: NewsSelect | NewsSelect; guides: GuidesSelect | GuidesSelect; 'case-studies': CaseStudiesSelect | CaseStudiesSelect; @@ -220,6 +224,7 @@ export interface Config { purgeDealRegistrations: TaskPurgeDealRegistrations; refreshContentInsights: TaskRefreshContentInsights; refreshCrux: TaskRefreshCrux; + syncYoutubeVideos: TaskSyncYoutubeVideos; schedulePublish: TaskSchedulePublish; inline: { input: unknown; @@ -2280,6 +2285,106 @@ export interface Advisory { createdAt: string; _status?: ('draft' | 'published') | null; } +/** + * Videos shown on /videos. New YouTube uploads arrive as drafts marked "Needs review": choose a category and topics, write the summary, then publish. + * + * This interface was referenced by `Config`'s JSON-Schema + * via the `definition` "videos". + */ +export interface Video { + id: number; + title: string; + slug: string; + /** + * Required to publish. Explainer is "what is" and fundamentals; product demo is how to use the product. Drives the category tabs on /videos. + */ + format?: ('explainer' | 'demo') | null; + /** + * At least one is required to publish. Drives the topic list on /videos. A topic that does not exist yet can be added from the search box. + */ + topics?: (number | VideoTopic)[] | null; + /** + * Shown on the card. Required to publish, up to 320 characters. + */ + summary?: string | null; + /** + * Plays at the top of /videos. Only one video can be featured: publishing this one clears the other. When none is featured, the newest video is. + */ + featured?: boolean | null; + /** + * Set by the YouTube sync. "Missing" means the video is gone or private on YouTube; it is not unpublished automatically. + */ + syncState?: ('synced' | 'needs-review' | 'missing' | 'manual') | null; + /** + * The length is read from YouTube. Fill this only when the video shows no length, for example a live stream (1:41). + */ + durationLabel?: string | null; + provider: 'youtube' | 'vimeo'; + /** + * The video id on the host (the 11 characters in a YouTube link). + */ + externalId: string; + /** + * Copied from the host by the sync. Read only. + */ + source?: { + title?: string | null; + description?: string | null; + durationSeconds?: number | null; + publishedAt?: string | null; + tags?: + | { + value?: string | null; + id?: string | null; + }[] + | null; + privacyStatus?: string | null; + embeddable?: boolean | null; + hasMaxres?: boolean | null; + thumbnailUrl?: string | null; + contentHash?: string | null; + fetchedAt?: string | null; + }; + /** + * Auto-set on first publish. Read-only — backdating is intentionally locked. Use the Payload Local API with overrideAccess for legacy imports. + */ + publishedAt?: string | null; + /** + * The date Google sees as the original publish date. Defaults to publish time. Backdating beyond 30 days can trigger spam-policy flags. + */ + displayPublishedAt?: string | null; + /** + * Set automatically when the title, body, summary or FAQs change. Shown publicly as the "Updated" date. + */ + contentUpdatedAt?: string | null; + updatedAt: string; + createdAt: string; + _status?: ('draft' | 'published') | null; +} +/** + * This interface was referenced by `Config`'s JSON-Schema + * via the `definition` "videoTopics". + */ +export interface VideoTopic { + id: number; + name: string; + /** + * URL-safe slug. Auto-generated from "name" on first save; safe to edit later (a redirect row is created automatically when you do). Cap 120 characters. + */ + slug: string; + description?: string | null; + /** + * YouTube tags and hashtags that suggest this topic on import, e.g. "sbom" or "software bill of materials". Case and spacing are ignored. + */ + aliases?: + | { + value: string; + id?: string | null; + }[] + | null; + updatedAt: string; + createdAt: string; +} /** * This interface was referenced by `Config`'s JSON-Schema * via the `definition` "news". @@ -9307,7 +9412,7 @@ export interface Integration { * Choose which activity on the site triggers a notification. Leave all filters empty to receive everything. */ routing?: { - events?: ('document.published' | 'lead.submitted')[] | null; + events?: ('document.published' | 'lead.submitted' | 'video.needs-review')[] | null; /** * Filter document.published events by collection. Empty = all collections. */ @@ -9358,7 +9463,7 @@ export interface Integration { /** * Optional — only mention this person for specific event types. Leave empty to mention them on all notifications. */ - triggerOn?: ('document.published' | 'lead.submitted')[] | null; + triggerOn?: ('document.published' | 'lead.submitted' | 'video.needs-review')[] | null; id?: string | null; }[] | null; @@ -9683,7 +9788,7 @@ export interface WebhooksDeadLetter { * Unique ID for this delivery attempt (UUID). */ webhookId: string; - event: 'document.published' | 'lead.submitted'; + event: 'document.published' | 'lead.submitted' | 'video.needs-review'; /** * Full event data blob. Used to re-run the delivery on retry. */ @@ -9815,6 +9920,7 @@ export interface PayloadJob { | 'purgeDealRegistrations' | 'refreshContentInsights' | 'refreshCrux' + | 'syncYoutubeVideos' | 'schedulePublish'; taskID: string; input?: @@ -9867,6 +9973,7 @@ export interface PayloadJob { | 'purgeDealRegistrations' | 'refreshContentInsights' | 'refreshCrux' + | 'syncYoutubeVideos' | 'schedulePublish' ) | null; @@ -9900,6 +10007,14 @@ export interface PayloadLockedDocument { relationTo: 'advisories'; value: number | Advisory; } | null) + | ({ + relationTo: 'videos'; + value: number | Video; + } | null) + | ({ + relationTo: 'videoTopics'; + value: number | VideoTopic; + } | null) | ({ relationTo: 'news'; value: number | News; @@ -10466,6 +10581,65 @@ export interface AdvisoriesSelect { createdAt?: T; _status?: T; } +/** + * This interface was referenced by `Config`'s JSON-Schema + * via the `definition` "videos_select". + */ +export interface VideosSelect { + title?: T; + slug?: T; + format?: T; + topics?: T; + summary?: T; + featured?: T; + syncState?: T; + durationLabel?: T; + provider?: T; + externalId?: T; + source?: + | T + | { + title?: T; + description?: T; + durationSeconds?: T; + publishedAt?: T; + tags?: + | T + | { + value?: T; + id?: T; + }; + privacyStatus?: T; + embeddable?: T; + hasMaxres?: T; + thumbnailUrl?: T; + contentHash?: T; + fetchedAt?: T; + }; + publishedAt?: T; + displayPublishedAt?: T; + contentUpdatedAt?: T; + updatedAt?: T; + createdAt?: T; + _status?: T; +} +/** + * This interface was referenced by `Config`'s JSON-Schema + * via the `definition` "videoTopics_select". + */ +export interface VideoTopicsSelect { + name?: T; + slug?: T; + description?: T; + aliases?: + | T + | { + value?: T; + id?: T; + }; + updatedAt?: T; + createdAt?: T; +} /** * This interface was referenced by `Config`'s JSON-Schema * via the `definition` "news_select". @@ -14880,6 +15054,14 @@ export interface TaskRefreshCrux { input?: unknown; output?: unknown; } +/** + * This interface was referenced by `Config`'s JSON-Schema + * via the `definition` "TaskSyncYoutubeVideos". + */ +export interface TaskSyncYoutubeVideos { + input?: unknown; + output?: unknown; +} /** * This interface was referenced by `Config`'s JSON-Schema * via the `definition` "TaskSchedulePublish". @@ -14897,6 +15079,10 @@ export interface TaskSchedulePublish { relationTo: 'advisories'; value: number | Advisory; } | null) + | ({ + relationTo: 'videos'; + value: number | Video; + } | null) | ({ relationTo: 'news'; value: number | News; diff --git a/apps/cms/src/payload.config.ts b/apps/cms/src/payload.config.ts index b0a59f831..a6b2837ac 100644 --- a/apps/cms/src/payload.config.ts +++ b/apps/cms/src/payload.config.ts @@ -16,6 +16,8 @@ import { AuditLog } from './payload/collections/audit-log'; import { WebhookDeadLetter } from './payload/collections/WebhookDeadLetter'; import { Authors } from './payload/collections/Authors'; import { Advisories } from './payload/collections/Advisories'; +import { VideoTopics } from './payload/collections/VideoTopics'; +import { Videos } from './payload/collections/Videos'; import { Blogs } from './payload/collections/Blogs'; import { BrokenLinks } from './payload/collections/BrokenLinks'; import { CaseStudies } from './payload/collections/CaseStudies'; @@ -109,6 +111,7 @@ import { retryWebhookTask } from './payload/jobs/retry-webhook'; import { analyticsCachePruneTask } from './payload/jobs/analytics-cache-prune'; import { refreshContentInsightsTask } from './payload/jobs/refresh-content-insights'; import { refreshCruxTask } from './payload/jobs/refresh-crux'; +import { syncYoutubeVideosTask } from './payload/jobs/sync-youtube-videos'; import { dashboardRefreshDailyTask } from './payload/jobs/dashboard-refresh-daily'; import { dashboardRefreshFrequentTask } from './payload/jobs/dashboard-refresh-frequent'; import { registerLeadHandlers } from './payload/lib/lead-handlers'; @@ -391,6 +394,8 @@ export default buildConfig({ // Authors) trail. Blogs, Advisories, + Videos, + VideoTopics, News, Guides, CaseStudies, @@ -535,6 +540,7 @@ export default buildConfig({ purgeDealRegistrationsTask, refreshContentInsightsTask, refreshCruxTask, + syncYoutubeVideosTask, ], autoRun: [ { @@ -601,6 +607,10 @@ export default buildConfig({ cron: '45 6 * * *', // daily at 06:45 UTC — Core Web Vitals (CrUX) origin-level cache queue: 'cruxRefresh', }, + { + cron: '7 * * * *', // hourly at :07 UTC — YouTube channel sync (new uploads become needs-review drafts) + queue: 'youtubeVideosSync', + }, { cron: '* * * * *', // every minute — Payload built-in schedulePublish jobs land in the `default` queue; wait_until gates actual execution queue: 'default', diff --git a/apps/cms/src/payload/admin/components/CommandPalette.tsx b/apps/cms/src/payload/admin/components/CommandPalette.tsx index 79f43fb9e..0b0239624 100644 --- a/apps/cms/src/payload/admin/components/CommandPalette.tsx +++ b/apps/cms/src/payload/admin/components/CommandPalette.tsx @@ -81,6 +81,7 @@ const CREATE_TARGETS: Array> = [ const SEARCHABLE_COLLECTIONS: ReadonlyArray<{ slug: string; label: string }> = [ { slug: 'blogs', label: 'Blog' }, { slug: 'advisories', label: 'Advisory' }, + { slug: 'videos', label: 'Video' }, { slug: 'news', label: 'News' }, { slug: 'guides', label: 'Guide' }, { slug: 'resources', label: 'Resource' }, diff --git a/apps/cms/src/payload/admin/components/Dashboard/Dashboard.tsx b/apps/cms/src/payload/admin/components/Dashboard/Dashboard.tsx index 9a8857174..129d288a5 100644 --- a/apps/cms/src/payload/admin/components/Dashboard/Dashboard.tsx +++ b/apps/cms/src/payload/admin/components/Dashboard/Dashboard.tsx @@ -14,6 +14,7 @@ const PROJECT_NAME = 'CleanStart'; const VERSIONED_CONTENT = [ 'blogs', 'advisories', + 'videos', 'news', 'guides', 'resources', @@ -28,6 +29,7 @@ type VersionedSlug = (typeof VERSIONED_CONTENT)[number]; const COLLECTION_LABELS: Record = { blogs: 'Blog', advisories: 'Advisory', + videos: 'Video', news: 'News', guides: 'Guide', resources: 'Resource', @@ -52,6 +54,8 @@ type RecentEdit = { type Pulse = { draftsPending: number; publishedThisWeek: number; + // `null` when the user cannot see videos: the card is omitted, not shown as zero. + videosToReview: number | null; // `null` for scoped (departmental) users, who don't own leads/redirects — // the corresponding cards are omitted rather than shown as zero. leadsToday: number | null; @@ -89,7 +93,7 @@ const fetchPulse = async ( const dayAgo = new Date(Date.now() - ONE_DAY).toISOString(); const weekAgo = new Date(Date.now() - SEVEN_DAYS).toISOString(); - const [draftCounts, publishedCounts, leadsToday, redirectsActive] = + const [draftCounts, publishedCounts, leadsToday, redirectsActive, videosToReview] = await Promise.all([ Promise.all( contentSlugs.map((slug) => @@ -112,11 +116,15 @@ const fetchPulse = async ( includeGlobalStats ? safeFind(payload, 'redirects', undefined) : Promise.resolve(null), + contentSlugs.includes('videos') + ? safeFind(payload, 'videos', { syncState: { equals: 'needs-review' } }, true) + : Promise.resolve(null), ]); return { draftsPending: draftCounts.reduce((acc, n) => acc + n, 0), publishedThisWeek: publishedCounts.reduce((acc, n) => acc + n, 0), + videosToReview, leadsToday, redirectsActive, }; @@ -549,6 +557,15 @@ export const Dashboard = async ( href={draftsHref} tone={pulse.draftsPending > 0 ? 'amber' : 'neutral'} /> + {pulse.videosToReview !== null ? ( + 0 ? 'amber' : 'neutral'} + /> + ) : null} = { Blogs: 'blogs', Advisories: 'advisories', + Videos: 'videos', News: 'news', Guides: 'guides', Resources: 'resources', diff --git a/apps/cms/src/payload/admin/components/UseDescriptionSummary.tsx b/apps/cms/src/payload/admin/components/UseDescriptionSummary.tsx new file mode 100644 index 000000000..e4a5b5309 --- /dev/null +++ b/apps/cms/src/payload/admin/components/UseDescriptionSummary.tsx @@ -0,0 +1,30 @@ +'use client'; + +import { useField, useFormFields } from '@payloadcms/ui'; +import type { ReactElement } from 'react'; + +import { firstParagraph } from '../../lib/videos/description-text'; + +/** Under the Summary field: fills it from the first real paragraph of the YouTube description. */ +export const UseDescriptionSummary = (): ReactElement | null => { + const { value, setValue } = useField({ path: 'summary' }); + const description = useFormFields(([fields]) => { + const raw = fields['source.description']?.value; + return typeof raw === 'string' ? raw : ''; + }); + const suggestion = firstParagraph(description); + if (!suggestion || suggestion === value) return null; + return ( +
+ +
+ ); +}; + +export default UseDescriptionSummary; diff --git a/apps/cms/src/payload/admin/components/VideoLengthCell.tsx b/apps/cms/src/payload/admin/components/VideoLengthCell.tsx new file mode 100644 index 000000000..e4738bc5e --- /dev/null +++ b/apps/cms/src/payload/admin/components/VideoLengthCell.tsx @@ -0,0 +1,12 @@ +'use client'; + +import type { ReactElement } from 'react'; + +import { type VideoStateRow, videoLength } from '../../lib/videos/video-state'; + +export const VideoLengthCell = ({ rowData }: { rowData?: VideoStateRow }): ReactElement => { + const length = rowData ? videoLength(rowData) : null; + return {length ?? '—'}; +}; + +export default VideoLengthCell; diff --git a/apps/cms/src/payload/admin/components/VideoSourceSummary.tsx b/apps/cms/src/payload/admin/components/VideoSourceSummary.tsx new file mode 100644 index 000000000..48c3b1ab8 --- /dev/null +++ b/apps/cms/src/payload/admin/components/VideoSourceSummary.tsx @@ -0,0 +1,76 @@ +'use client'; + +import { useFormFields } from '@payloadcms/ui'; +import type { ReactElement } from 'react'; + +const chip = (tone: 'ok' | 'warn' | 'plain'): React.CSSProperties => ({ + display: 'inline-block', + padding: '2px 8px', + borderRadius: 999, + fontSize: 12, + background: + tone === 'ok' ? 'var(--theme-success-100)' : tone === 'warn' ? 'var(--theme-warning-100)' : 'var(--theme-elevation-100)', + color: + tone === 'ok' ? 'var(--theme-success-800)' : tone === 'warn' ? 'var(--theme-warning-800)' : 'var(--theme-elevation-800)', +}); + +const ago = (iso: string): string => { + const minutes = Math.max(0, Math.round((Date.now() - new Date(iso).getTime()) / 60_000)); + if (minutes < 60) return `${minutes} min ago`; + const hours = Math.round(minutes / 60); + return hours < 48 ? `${hours} h ago` : `${Math.round(hours / 24)} days ago`; +}; + +/** + * Top of the video edit screen: the video itself and the facts about it on YouTube, so an editor + * sees what they are filing before they scroll to the fields. Shows nothing on a video that has not + * been saved yet, because there is no YouTube id to show. + */ +export const VideoSourceSummary = (): ReactElement | null => { + const [externalId, youtubeTitle, privacy, embeddable, fetchedAt] = useFormFields(([fields]) => [ + fields.externalId?.value, + fields['source.title']?.value, + fields['source.privacyStatus']?.value, + fields['source.embeddable']?.value, + fields['source.fetchedAt']?.value, + ]); + if (typeof externalId !== 'string' || externalId.length === 0) return null; + + const watchUrl = `https://www.youtube.com/watch?v=${externalId}`; + return ( +
+ + {/* eslint-disable-next-line @next/next/no-img-element */} + Open the video on YouTube + +
+
Title on YouTube
+
+ {typeof youtubeTitle === 'string' && youtubeTitle ? youtubeTitle : 'Not synced yet'} +
+
+ {typeof privacy === 'string' ? {privacy === 'public' ? 'Public' : privacy === 'unlisted' ? 'Unlisted' : 'Private'} : null} + {embeddable === true ? Can embed : null} + {embeddable === false ? Can't be embedded : null} + {typeof fetchedAt === 'string' ? ( + // The age depends on the clock, which differs between the server render and the browser. + + Synced {ago(fetchedAt)} + + ) : null} + + Open on YouTube + +
+
+
+ ); +}; + +export default VideoSourceSummary; diff --git a/apps/cms/src/payload/admin/components/VideoStateCell.tsx b/apps/cms/src/payload/admin/components/VideoStateCell.tsx new file mode 100644 index 000000000..0cff9b7ae --- /dev/null +++ b/apps/cms/src/payload/admin/components/VideoStateCell.tsx @@ -0,0 +1,33 @@ +'use client'; + +import type { ReactElement } from 'react'; + +import { type VideoStateRow, type VideoStateTone, videoState } from '../../lib/videos/video-state'; + +const TONES: Record = { + live: { background: 'var(--theme-success-100)', color: 'var(--theme-success-800)' }, + review: { background: 'var(--theme-warning-100)', color: 'var(--theme-warning-800)' }, + draft: { background: 'var(--theme-elevation-100)', color: 'var(--theme-elevation-800)' }, + problem: { background: 'var(--theme-error-100)', color: 'var(--theme-error-800)' }, +}; + +export const VideoStateCell = ({ rowData }: { rowData?: VideoStateRow }): ReactElement | null => { + if (!rowData) return null; + const { label, tone } = videoState(rowData); + return ( + + {label} + + ); +}; + +export default VideoStateCell; diff --git a/apps/cms/src/payload/admin/components/VideoThumbCell.tsx b/apps/cms/src/payload/admin/components/VideoThumbCell.tsx new file mode 100644 index 000000000..bf89b1cd8 --- /dev/null +++ b/apps/cms/src/payload/admin/components/VideoThumbCell.tsx @@ -0,0 +1,26 @@ +'use client'; + +import type { ReactElement } from 'react'; + +type Row = { externalId?: string | null; source?: { thumbnailUrl?: string | null } | null }; + +export const VideoThumbCell = ({ rowData }: { rowData?: Row }): ReactElement => { + const src = + rowData?.source?.thumbnailUrl || + (rowData?.externalId ? `https://i.ytimg.com/vi/${rowData.externalId}/mqdefault.jpg` : null); + return src ? ( + // eslint-disable-next-line @next/next/no-img-element + + ) : ( + + ); +}; + +export default VideoThumbCell; diff --git a/apps/cms/src/payload/admin/components/VideoTopicSuggestions.tsx b/apps/cms/src/payload/admin/components/VideoTopicSuggestions.tsx new file mode 100644 index 000000000..607a03523 --- /dev/null +++ b/apps/cms/src/payload/admin/components/VideoTopicSuggestions.tsx @@ -0,0 +1,88 @@ +'use client'; + +import { useField, useFormFields } from '@payloadcms/ui'; +import type { ReactElement } from 'react'; +import { useEffect, useState } from 'react'; + +import { type TopicRef, suggestTopicIds } from '../../lib/videos/topic-match'; + +type TopicDoc = { id: number | string; name?: string; aliases?: { value?: string }[] | null }; +type Id = number | string; + +const idOf = (entry: unknown): Id | null => { + if (typeof entry === 'number' || typeof entry === 'string') return entry; + if (entry && typeof entry === 'object' && 'id' in entry) return (entry as { id: Id }).id; + return null; +}; + +/** + * Under the Topics field: topics whose name or alias appears in the video's YouTube tags or + * hashtags, one click to add. Uses the same matching rule as the sync, so what the editor is + * offered here is what an import would have pre-filled. + */ +export const VideoTopicSuggestions = (): ReactElement | null => { + const { value, setValue } = useField({ path: 'topics' }); + const tags = useFormFields(([fields]) => + Object.entries(fields) + .filter(([key]) => key.startsWith('source.tags.') && key.endsWith('.value')) + .map(([, field]) => field.value) + .filter((tag): tag is string => typeof tag === 'string'), + ); + const description = useFormFields(([fields]) => { + const raw = fields['source.description']?.value; + return typeof raw === 'string' ? raw : ''; + }); + const [topics, setTopics] = useState<(TopicRef & { name: string })[]>([]); + + useEffect(() => { + let cancelled = false; + fetch('/api/videoTopics?limit=200&depth=0', { credentials: 'include' }) + .then((res) => (res.ok ? (res.json() as Promise<{ docs?: TopicDoc[] }>) : { docs: [] })) + .then(({ docs = [] }) => { + if (cancelled) return; + setTopics( + docs.map((d) => ({ + id: d.id, + name: d.name ?? '', + aliases: (d.aliases ?? []).flatMap((a) => (a.value ? [a.value] : [])), + })), + ); + }) + .catch(() => undefined); + return () => { + cancelled = true; + }; + }, []); + + const selected = new Set((Array.isArray(value) ? value : []).map(idOf).filter((id): id is Id => id !== null)); + const suggested = suggestTopicIds({ tags, description }, topics) + .filter((id) => !selected.has(id)) + .map((id) => topics.find((t) => t.id === id)) + .filter((t): t is TopicRef & { name: string } => t !== undefined); + if (suggested.length === 0) return null; + + return ( +
+ Suggested from the YouTube hashtags: + {suggested.map((topic) => ( + + ))} +
+ ); +}; + +export default VideoTopicSuggestions; diff --git a/apps/cms/src/payload/admin/components/fields/RelationshipField.tsx b/apps/cms/src/payload/admin/components/fields/RelationshipField.tsx index 0b8fd590b..f43a753f6 100644 --- a/apps/cms/src/payload/admin/components/fields/RelationshipField.tsx +++ b/apps/cms/src/payload/admin/components/fields/RelationshipField.tsx @@ -73,6 +73,8 @@ const ROW_HINTS: Record = { // Content collections: stacked chips with status badge, no thumbs. blogs: { primary: ['title'], secondary: ['slug'], showThumb: false, showCreate: false, chipLayout: 'stacked', typeLabel: 'Blog' }, advisories: { primary: ['title'], secondary: ['slug'], showThumb: false, showCreate: false, chipLayout: 'stacked', typeLabel: 'Advisory' }, + videos: { primary: ['title'], secondary: ['slug'], showThumb: false, showCreate: false, chipLayout: 'stacked', typeLabel: 'Video' }, + videoTopics: { primary: ['name'], secondary: ['slug'], showThumb: false, showCreate: true, chipLayout: 'inline' }, news: { primary: ['title'], secondary: ['slug'], showThumb: false, showCreate: false, chipLayout: 'stacked', typeLabel: 'News' }, guides: { primary: ['title'], secondary: ['slug'], showThumb: false, showCreate: false, chipLayout: 'stacked', typeLabel: 'Guide' }, resources: { primary: ['title'], secondary: ['slug'], showThumb: false, showCreate: false, chipLayout: 'stacked', typeLabel: 'Resource' }, @@ -857,7 +859,7 @@ export const RelationshipField = (props: RelationshipFieldClientProps): ReactEle ref={inputRef} type="search" className="cs-relationship-field__input" - placeholder={pills.length === 0 ? `Search ${slugs.join(' / ')}…` : 'Add more…'} + placeholder={pills.length === 0 ? emptyPlaceholder : 'Add more…'} value={query} onChange={onInputChange} onFocus={() => setOpen(true)} diff --git a/apps/cms/src/payload/admin/components/integrations/AuditTrail.tsx b/apps/cms/src/payload/admin/components/integrations/AuditTrail.tsx index e25f0d257..dc191ea51 100644 --- a/apps/cms/src/payload/admin/components/integrations/AuditTrail.tsx +++ b/apps/cms/src/payload/admin/components/integrations/AuditTrail.tsx @@ -7,7 +7,7 @@ import { useEffect, useState } from 'react'; interface AuditRow { id: number | string; webhookId: string; - event: 'document.published' | 'lead.submitted'; + event: 'document.published' | 'lead.submitted' | 'video.needs-review'; attemptCount: number; lastError: string | null; nextRetryAt: string | null; diff --git a/apps/cms/src/payload/admin/components/integrations/EventsMultiSelect.tsx b/apps/cms/src/payload/admin/components/integrations/EventsMultiSelect.tsx index b9dc8bd36..7b74e154c 100644 --- a/apps/cms/src/payload/admin/components/integrations/EventsMultiSelect.tsx +++ b/apps/cms/src/payload/admin/components/integrations/EventsMultiSelect.tsx @@ -15,6 +15,11 @@ const OPTIONS = [ sublabel: 'on every public form submission', value: 'lead.submitted', }, + { + label: 'video.needs-review', + sublabel: 'when a sync run imports new videos', + value: 'video.needs-review', + }, ] as const; interface Props { diff --git a/apps/cms/src/payload/admin/components/views/list/filters/list-filter-config.ts b/apps/cms/src/payload/admin/components/views/list/filters/list-filter-config.ts index 223cc1f9a..49530903a 100644 --- a/apps/cms/src/payload/admin/components/views/list/filters/list-filter-config.ts +++ b/apps/cms/src/payload/admin/components/views/list/filters/list-filter-config.ts @@ -41,6 +41,30 @@ export const LIST_FILTER_FIELDS: Record { name: 'slug', type: 'slug', label: 'Slug', source: 'name' }, ], }, + videoTopics: { + title: 'New video topic', + supportsDrafts: false, + fields: [ + { + name: 'name', + type: 'text', + label: 'Name', + required: true, + maxLength: 120, + }, + { name: 'slug', type: 'slug', label: 'Slug', source: 'name' }, + ], + }, knowledgeCategories: { title: 'New knowledge category', supportsDrafts: true, diff --git a/apps/cms/src/payload/collections/Integrations.ts b/apps/cms/src/payload/collections/Integrations.ts index 499a0c336..2e8879bee 100644 --- a/apps/cms/src/payload/collections/Integrations.ts +++ b/apps/cms/src/payload/collections/Integrations.ts @@ -256,6 +256,7 @@ const teamsConfigGroup: Field = { options: [ { label: 'When content is published', value: 'document.published' }, { label: 'When a form is submitted', value: 'lead.submitted' }, + { label: 'When new videos need review', value: 'video.needs-review' }, ], admin: { description: 'Optional — only mention this person for specific event types. Leave empty to mention them on all notifications.', @@ -580,6 +581,7 @@ export const Integrations: CollectionConfig = { options: [ { label: 'document.published — on first publish of a content doc', value: 'document.published' }, { label: 'lead.submitted — on every public form submission', value: 'lead.submitted' }, + { label: 'video.needs-review: when a sync run imports new videos', value: 'video.needs-review' }, ], admin: { components: { diff --git a/apps/cms/src/payload/collections/VideoTopics.ts b/apps/cms/src/payload/collections/VideoTopics.ts new file mode 100644 index 000000000..1ce9f46da --- /dev/null +++ b/apps/cms/src/payload/collections/VideoTopics.ts @@ -0,0 +1,49 @@ +import type { CollectionConfig } from 'payload'; + +import { isAdminOrEditor } from '../access'; +import { slugField } from '../fields/slug'; +import { + revalidateWebAfterDeleteHook, + revalidateWebPublishAfterChangeHook, +} from '../hooks/revalidate-web-publish'; + +/** + * Topics for /videos. Topics are filters on the listing, not pages, so this + * taxonomy has no SEO fields, no drafts and no URL of its own. `aliases` are the + * YouTube tags and hashtags that should suggest the topic on import. + */ +export const VideoTopics: CollectionConfig = { + slug: 'videoTopics', + labels: { singular: 'Video topic', plural: 'Video topics' }, + admin: { + useAsTitle: 'name', + defaultColumns: ['name', 'slug', 'updatedAt'], + group: 'Taxonomies', + }, + access: { + read: () => true, + create: isAdminOrEditor, + update: isAdminOrEditor, + delete: isAdminOrEditor, + }, + fields: [ + { name: 'name', type: 'text', required: true }, + slugField({ source: 'name' }), + { name: 'description', type: 'textarea' }, + { + name: 'aliases', + type: 'array', + labels: { singular: 'Alias', plural: 'Aliases' }, + admin: { + description: + 'YouTube tags and hashtags that suggest this topic on import, e.g. "sbom" or "software bill of materials". Case and spacing are ignored.', + }, + fields: [{ name: 'value', type: 'text', required: true }], + }, + ], + hooks: { + afterChange: [revalidateWebPublishAfterChangeHook('videoTopics')], + afterDelete: [revalidateWebAfterDeleteHook('videoTopics')], + }, + timestamps: true, +}; diff --git a/apps/cms/src/payload/collections/Videos.test.ts b/apps/cms/src/payload/collections/Videos.test.ts new file mode 100644 index 000000000..ec449304f --- /dev/null +++ b/apps/cms/src/payload/collections/Videos.test.ts @@ -0,0 +1,61 @@ +import { describe, expect, it, vi } from 'vitest'; + +import { Videos, singleFeaturedHook } from './Videos'; + +type HookArgs = Parameters[0]; + +const run = async (doc: Record, others: { id: number }[], context: Record = {}) => { + const find = vi.fn().mockResolvedValue({ docs: others }); + const update = vi.fn().mockResolvedValue({}); + const req = { payload: { find, update } }; + await singleFeaturedHook({ doc, req, context } as unknown as HookArgs); + return { find, update }; +}; + +describe('singleFeaturedHook', () => { + it('clears the flag on the previous hero when a featured video is published', async () => { + const { find, update } = await run({ id: 5, featured: true, _status: 'published' }, [{ id: 2 }, { id: 3 }]); + expect(find).toHaveBeenCalledOnce(); + expect(update.mock.calls.map(([args]) => [args.id, args.data])).toEqual([ + [2, { featured: false }], + [3, { featured: false }], + ]); + }); + + it('leaves the live hero alone while the featured video is still a draft', async () => { + const { find, update } = await run({ id: 5, featured: true, _status: 'draft' }, [{ id: 2 }]); + expect(find).not.toHaveBeenCalled(); + expect(update).not.toHaveBeenCalled(); + }); + + it('does nothing for a video that is not featured', async () => { + const { find } = await run({ id: 5, featured: false, _status: 'published' }, [{ id: 2 }]); + expect(find).not.toHaveBeenCalled(); + }); + + it('does not loop on the updates it makes itself', async () => { + const { find } = await run({ id: 5, featured: true, _status: 'published' }, [{ id: 2 }], { singleFeatured: true }); + expect(find).not.toHaveBeenCalled(); + }); +}); + +describe('Videos collection shape', () => { + const names = Videos.fields.map((f) => ('name' in f ? f.name : undefined)); + + it('has no per-video page parts: no SEO block, no permalink', () => { + expect(names).not.toContain('seo'); + expect(names).not.toContain('permalink'); + }); + + it('keeps the slug but hides it from editors', () => { + const slug = Videos.fields.find((f) => 'name' in f && f.name === 'slug'); + const admin = (slug && 'admin' in slug ? slug.admin : undefined) as { hidden?: boolean } | undefined; + expect(admin?.hidden).toBe(true); + }); + + it('offers exactly two categories', () => { + const format = Videos.fields.find((f) => 'name' in f && f.name === 'format'); + const values = format && 'options' in format ? format.options.map((o) => (typeof o === 'string' ? o : o.value)) : []; + expect(values).toEqual(['explainer', 'demo']); + }); +}); diff --git a/apps/cms/src/payload/collections/Videos.ts b/apps/cms/src/payload/collections/Videos.ts new file mode 100644 index 000000000..ef115aa97 --- /dev/null +++ b/apps/cms/src/payload/collections/Videos.ts @@ -0,0 +1,297 @@ +import type { CollectionAfterChangeHook, CollectionBeforeChangeHook, CollectionConfig, Field, TextField } from 'payload'; +import { ValidationError } from 'payload'; + +import { isAdminOrEditor, publishedOrAuthenticated } from '../access'; +import { docStatusBarEditConfig } from '../admin/doc-status-bar-mount'; +import { contentUpdatedAtField } from '../fields/content-updated-at'; +import { displayPublishedAtField } from '../fields/display-published-at'; +import { publishedAtField } from '../fields/published-at'; +import { slugField } from '../fields/slug'; +import { contentTitleField } from '../fields/title'; +import { contentUpdatedAtHook } from '../hooks/content-updated-at'; +import { displayPublishedAtAuditHook } from '../hooks/display-published-at-audit'; +import { displayPublishedAtBackfillHook } from '../hooks/display-published-at-backfill'; +import { firstPublishHook } from '../hooks/first-publish'; +import { + revalidateWebAfterDeleteHook, + revalidateWebPublishAfterChangeHook, +} from '../hooks/revalidate-web-publish'; + +/** Plan 2 adds 'vimeo'. The enum value exists now so that is not an enum migration. */ +export const SUPPORTED_PROVIDERS: readonly string[] = ['youtube']; + +const SUMMARY_MAX = 320; +const DURATION_LABEL = /^\d{1,2}(:\d{2}){1,2}$/; + +const readOnly = { update: () => false } as const; + +export const enforceSupportedProviderHook: CollectionBeforeChangeHook = ({ data }) => { + const provider = (data as { provider?: string }).provider; + if (provider && !SUPPORTED_PROVIDERS.includes(provider)) { + throw new ValidationError({ + errors: [{ path: 'provider', message: `Provider "${provider}" is not enabled yet.` }], + }); + } + return data; +}; + +/** Old videos keep their real date: publishedAt follows the upload date until an editor sets one. */ +export const publishedAtFromSourceHook: CollectionBeforeChangeHook = ({ data, operation }) => { + const source = (data as { source?: { publishedAt?: string } }).source; + const current = (data as { publishedAt?: string | null }).publishedAt; + if (operation === 'create' && !current && source?.publishedAt) { + return { ...data, publishedAt: source.publishedAt }; + } + return data; +}; + +/** + * Only one video plays in the /videos hero. Publishing a featured video clears the flag on the + * others, so an editor never has to untick the old one by hand. A draft is ignored: it is not live, + * so it must not take the hero away from the video that is. + */ +export const singleFeaturedHook: CollectionAfterChangeHook = async ({ doc, req, context }) => { + const current = doc as { id: number | string; featured?: boolean | null; _status?: string }; + if (context?.singleFeatured || current.featured !== true || current._status !== 'published') return doc; + const others = await req.payload.find({ + collection: 'videos', + where: { and: [{ featured: { equals: true } }, { id: { not_equals: current.id } }] }, + limit: 100, + depth: 0, + pagination: false, + overrideAccess: true, + req, + }); + for (const other of others.docs) { + await req.payload.update({ + collection: 'videos', + id: other.id, + data: { featured: false }, + overrideAccess: true, + depth: 0, + context: { singleFeatured: true }, + req, + }); + } + return doc; +}; + +/** The slug has no public meaning (videos play inline), so it is generated from the title and not shown. */ +const hiddenSlugField = (): Field => { + const field = slugField({ source: 'title' }) as TextField; + const { components: _components, position: _position, description: _description, ...admin } = field.admin ?? {}; + return { ...field, admin: { ...admin, hidden: true } }; +}; + +/** + * Editorial videos for /videos. Imported from YouTube by the hourly sync as + * needs-review drafts (see `lib/videos`). The `source` group is owned by the + * sync, everything else by editors; neither overwrites the other. Videos play + * inline on the listing, so there is no per-video page, URL or SEO block here. + */ +export const Videos: CollectionConfig = { + slug: 'videos', + labels: { singular: 'Video', plural: 'Videos' }, + admin: { + useAsTitle: 'title', + defaultColumns: ['thumb', 'title', 'state', 'format', 'topics', 'length'], + description: + 'Videos shown on /videos. New YouTube uploads arrive as drafts marked "Needs review": choose a category and topics, write the summary, then publish.', + group: 'Content', + components: { + edit: docStatusBarEditConfig({ showStats: false, showPublishedAt: true, showPurge: false }), + }, + }, + defaultSort: '-publishedAt', + access: { + read: publishedOrAuthenticated, + create: isAdminOrEditor, + update: isAdminOrEditor, + delete: isAdminOrEditor, + }, + fields: [ + { + name: 'youtubePreview', + type: 'ui', + admin: { + components: { Field: '@/payload/admin/components/VideoSourceSummary.tsx#VideoSourceSummary' }, + }, + }, + { ...contentTitleField, label: 'Title on the website' }, + hiddenSlugField(), + { + name: 'format', + type: 'radio', + label: 'Category', + options: [ + { label: 'Explainer', value: 'explainer' }, + { label: 'Product demo', value: 'demo' }, + ], + admin: { + layout: 'horizontal', + description: + 'Required to publish. Explainer is "what is" and fundamentals; product demo is how to use the product. Drives the category tabs on /videos.', + }, + }, + { + name: 'topics', + type: 'relationship', + relationTo: 'videoTopics', + hasMany: true, + admin: { + description: + 'At least one is required to publish. Drives the topic list on /videos. A topic that does not exist yet can be added from the search box.', + custom: { emptyPlaceholder: 'Search topics…' }, + components: { + Cell: { + path: '@/payload/admin/components/RelationshipCell.tsx#RelationshipCell', + clientProps: { collectionSlug: 'videoTopics' }, + }, + }, + }, + }, + { + name: 'topicSuggestions', + type: 'ui', + admin: { + components: { Field: '@/payload/admin/components/VideoTopicSuggestions.tsx#VideoTopicSuggestions' }, + }, + }, + { + name: 'summary', + type: 'textarea', + maxLength: SUMMARY_MAX, + admin: { + description: `Shown on the card. Required to publish, up to ${SUMMARY_MAX} characters.`, + }, + }, + { + name: 'summaryFromYoutube', + type: 'ui', + admin: { + components: { Field: '@/payload/admin/components/UseDescriptionSummary.tsx#UseDescriptionSummary' }, + }, + }, + { + name: 'featured', + type: 'checkbox', + label: 'Show in the hero', + defaultValue: false, + admin: { + position: 'sidebar', + description: + 'Plays at the top of /videos. Only one video can be featured: publishing this one clears the other. When none is featured, the newest video is.', + }, + }, + { + name: 'syncState', + type: 'select', + label: 'Review state', + defaultValue: 'manual', + options: [ + { label: 'Synced', value: 'synced' }, + { label: 'Needs review', value: 'needs-review' }, + { label: 'Missing on YouTube', value: 'missing' }, + { label: 'Manual', value: 'manual' }, + ], + access: readOnly, + admin: { + position: 'sidebar', + readOnly: true, + description: + 'Set by the YouTube sync. "Missing" means the video is gone or private on YouTube; it is not unpublished automatically.', + }, + }, + { + name: 'durationLabel', + type: 'text', + label: 'Length (only if YouTube has none)', + validate: (value: string | null | undefined) => + value == null || value === '' || DURATION_LABEL.test(value) + ? true + : 'Use m:ss or h:mm:ss, for example 1:41 or 1:02:10.', + admin: { + position: 'sidebar', + description: + 'The length is read from YouTube. Fill this only when the video shows no length, for example a live stream (1:41).', + }, + }, + { + type: 'collapsible', + label: 'From YouTube (read only)', + admin: { initCollapsed: true }, + fields: [ + { + name: 'provider', + type: 'select', + required: true, + defaultValue: 'youtube', + options: [ + { label: 'YouTube', value: 'youtube' }, + { label: 'Vimeo', value: 'vimeo' }, + ], + access: readOnly, + admin: { readOnly: true }, + }, + { + name: 'externalId', + type: 'text', + required: true, + unique: true, + index: true, + access: readOnly, + admin: { + readOnly: true, + description: 'The video id on the host (the 11 characters in a YouTube link).', + }, + }, + { + name: 'source', + type: 'group', + admin: { description: 'Copied from the host by the sync. Read only.' }, + fields: [ + { name: 'title', type: 'text', access: readOnly }, + { name: 'description', type: 'textarea', access: readOnly }, + { name: 'durationSeconds', type: 'number', access: readOnly }, + { name: 'publishedAt', type: 'date', access: readOnly }, + { + name: 'tags', + type: 'array', + access: readOnly, + fields: [{ name: 'value', type: 'text' }], + }, + { name: 'privacyStatus', type: 'text', access: readOnly }, + { name: 'embeddable', type: 'checkbox', access: readOnly }, + { name: 'hasMaxres', type: 'checkbox', access: readOnly }, + { name: 'thumbnailUrl', type: 'text', access: readOnly }, + { name: 'contentHash', type: 'text', access: readOnly, admin: { hidden: true } }, + { name: 'fetchedAt', type: 'date', access: readOnly }, + ], + }, + ], + }, + { name: 'thumb', type: 'ui', admin: { components: { Cell: '@/payload/admin/components/VideoThumbCell.tsx#VideoThumbCell' } } }, + { name: 'length', type: 'ui', label: 'Length', admin: { components: { Cell: '@/payload/admin/components/VideoLengthCell.tsx#VideoLengthCell' } } }, + { name: 'state', type: 'ui', label: 'State', admin: { components: { Cell: '@/payload/admin/components/VideoStateCell.tsx#VideoStateCell' } } }, + publishedAtField, + displayPublishedAtField, + contentUpdatedAtField, + ], + hooks: { + beforeChange: [ + enforceSupportedProviderHook, + publishedAtFromSourceHook, + contentUpdatedAtHook(), + firstPublishHook(), + displayPublishedAtBackfillHook, + ], + afterChange: [ + displayPublishedAtAuditHook('videos'), + singleFeaturedHook, + revalidateWebPublishAfterChangeHook('videos'), + ], + afterDelete: [revalidateWebAfterDeleteHook('videos')], + }, + versions: { drafts: { schedulePublish: true }, maxPerDoc: 25 }, + timestamps: true, +}; diff --git a/apps/cms/src/payload/collections/WebhookDeadLetter.ts b/apps/cms/src/payload/collections/WebhookDeadLetter.ts index e0f5a6ab1..5875403c2 100644 --- a/apps/cms/src/payload/collections/WebhookDeadLetter.ts +++ b/apps/cms/src/payload/collections/WebhookDeadLetter.ts @@ -57,6 +57,7 @@ export const WebhookDeadLetter: CollectionConfig = { options: [ { label: 'document.published', value: 'document.published' }, { label: 'lead.submitted', value: 'lead.submitted' }, + { label: 'video.needs-review', value: 'video.needs-review' }, ], }, { diff --git a/apps/cms/src/payload/collections/__snapshots__/VideoTopics.snap.json b/apps/cms/src/payload/collections/__snapshots__/VideoTopics.snap.json new file mode 100644 index 000000000..6316f7020 --- /dev/null +++ b/apps/cms/src/payload/collections/__snapshots__/VideoTopics.snap.json @@ -0,0 +1,38 @@ +{ + "slug": "videoTopics", + "timestamps": true, + "fields": [ + { + "type": "text", + "name": "name", + "required": true + }, + { + "type": "text", + "name": "slug", + "required": true, + "unique": true + }, + { + "type": "textarea", + "name": "description" + }, + { + "type": "array", + "name": "aliases", + "fields": [ + { + "type": "text", + "name": "value", + "required": true + } + ] + } + ], + "hooks": { + "beforeChange": 0, + "afterChange": 1, + "beforeValidate": 0, + "afterDelete": 1 + } +} \ No newline at end of file diff --git a/apps/cms/src/payload/collections/__snapshots__/Videos.snap.json b/apps/cms/src/payload/collections/__snapshots__/Videos.snap.json new file mode 100644 index 000000000..10d2e83bf --- /dev/null +++ b/apps/cms/src/payload/collections/__snapshots__/Videos.snap.json @@ -0,0 +1,190 @@ +{ + "slug": "videos", + "versions": { + "drafts": true + }, + "timestamps": true, + "fields": [ + { + "type": "ui", + "name": "youtubePreview" + }, + { + "type": "text", + "name": "title", + "required": true + }, + { + "type": "text", + "name": "slug", + "required": true, + "unique": true + }, + { + "type": "radio", + "name": "format", + "options": [ + { + "value": "explainer" + }, + { + "value": "demo" + } + ] + }, + { + "type": "relationship", + "name": "topics", + "hasMany": true, + "relationTo": "videoTopics" + }, + { + "type": "ui", + "name": "topicSuggestions" + }, + { + "type": "textarea", + "name": "summary" + }, + { + "type": "ui", + "name": "summaryFromYoutube" + }, + { + "type": "checkbox", + "name": "featured" + }, + { + "type": "select", + "name": "syncState", + "options": [ + { + "value": "synced" + }, + { + "value": "needs-review" + }, + { + "value": "missing" + }, + { + "value": "manual" + } + ] + }, + { + "type": "text", + "name": "durationLabel" + }, + { + "type": "collapsible", + "fields": [ + { + "type": "select", + "name": "provider", + "required": true, + "options": [ + { + "value": "youtube" + }, + { + "value": "vimeo" + } + ] + }, + { + "type": "text", + "name": "externalId", + "required": true, + "unique": true + }, + { + "type": "group", + "name": "source", + "fields": [ + { + "type": "text", + "name": "title" + }, + { + "type": "textarea", + "name": "description" + }, + { + "type": "number", + "name": "durationSeconds" + }, + { + "type": "date", + "name": "publishedAt" + }, + { + "type": "array", + "name": "tags", + "fields": [ + { + "type": "text", + "name": "value" + } + ] + }, + { + "type": "text", + "name": "privacyStatus" + }, + { + "type": "checkbox", + "name": "embeddable" + }, + { + "type": "checkbox", + "name": "hasMaxres" + }, + { + "type": "text", + "name": "thumbnailUrl" + }, + { + "type": "text", + "name": "contentHash" + }, + { + "type": "date", + "name": "fetchedAt" + } + ] + } + ] + }, + { + "type": "ui", + "name": "thumb" + }, + { + "type": "ui", + "name": "length" + }, + { + "type": "ui", + "name": "state" + }, + { + "type": "date", + "name": "publishedAt" + }, + { + "type": "date", + "name": "displayPublishedAt" + }, + { + "type": "date", + "name": "contentUpdatedAt" + } + ], + "hooks": { + "beforeChange": 5, + "afterChange": 3, + "beforeValidate": 0, + "afterDelete": 1 + } +} \ No newline at end of file diff --git a/apps/cms/src/payload/collections/schema-surface.test.ts b/apps/cms/src/payload/collections/schema-surface.test.ts index b370d7488..346dbd79a 100644 --- a/apps/cms/src/payload/collections/schema-surface.test.ts +++ b/apps/cms/src/payload/collections/schema-surface.test.ts @@ -3,6 +3,8 @@ import type { CollectionConfig, Field } from 'payload'; import { AboutGalleries } from './AboutGalleries'; import { Advisories } from './Advisories'; +import { Videos } from './Videos'; +import { VideoTopics } from './VideoTopics'; import { AuditLog } from './audit-log'; import { Authors } from './Authors'; import { Blogs } from './Blogs'; @@ -95,6 +97,8 @@ const collections: { name: string; config: CollectionConfig }[] = [ { name: 'AboutGalleries', config: AboutGalleries }, { name: 'AuditLog', config: AuditLog }, { name: 'Advisories', config: Advisories }, + { name: 'Videos', config: Videos }, + { name: 'VideoTopics', config: VideoTopics }, { name: 'Authors', config: Authors }, { name: 'Blogs', config: Blogs }, { name: 'BrokenLinks', config: BrokenLinks }, diff --git a/apps/cms/src/payload/endpoints/integrations-actions.ts b/apps/cms/src/payload/endpoints/integrations-actions.ts index c1e3a1ca1..cf6e6f576 100644 --- a/apps/cms/src/payload/endpoints/integrations-actions.ts +++ b/apps/cms/src/payload/endpoints/integrations-actions.ts @@ -8,7 +8,7 @@ import { } from '../lib/integrations/credentials'; import { dispatchEvent } from '../lib/webhooks/dispatch'; import type { TeamsMention } from '../lib/webhooks/teams'; -import type { WebhookEventName } from '../lib/webhooks/dispatch'; +import type { WebhookEvent, WebhookEventName } from '../lib/webhooks/dispatch'; const json = (data: unknown, init?: ResponseInit): Response => new Response(JSON.stringify(data), { @@ -37,24 +37,33 @@ interface IntegrationRow { } | null; } -const fixtureEvent = (event: WebhookEventName) => - event === 'lead.submitted' - ? { - event: 'lead.submitted' as const, +const fixtureEvent = (event: WebhookEventName): WebhookEvent => { + switch (event) { + case 'lead.submitted': + return { + event, data: { formSlug: '_integration_test', duplicate: false, source: 'cms' as const, }, - } - : { - event: 'document.published' as const, + }; + case 'video.needs-review': + return { + event, + data: { count: 1, titles: ['Integration test video'] }, + }; + case 'document.published': + return { + event, data: { collection: '_integration_test', slug: 'integration-test', title: 'Integration test', }, }; + } +}; /** * POST /api/integrations/:id/test diff --git a/apps/cms/src/payload/hooks/publish-gate.test.ts b/apps/cms/src/payload/hooks/publish-gate.test.ts index 63f82adbd..cf3fd7911 100644 --- a/apps/cms/src/payload/hooks/publish-gate.test.ts +++ b/apps/cms/src/payload/hooks/publish-gate.test.ts @@ -88,3 +88,62 @@ describe('publishGateHook — allows SEO char-length issues through', () => { expect(() => runHook(data)).not.toThrow(); }); }); + +describe('runPublishChecks: videos', () => { + const readyVideo = { + ...validBase, + format: 'explainer', + topics: [4], + summary: 'What the scanner reports and how to read it.', + durationLabel: '4:12', + }; + const find = (data: Record, id: string) => + runPublishChecks(data, 'videos').find((c) => c.id === id); + + it('passes every video check for a complete video', () => { + const failing = runPublishChecks(readyVideo, 'videos').filter( + (c) => !c.pass && c.id.startsWith('video-'), + ); + expect(failing).toEqual([]); + }); + + it.each([ + ['video-format', { format: undefined }], + ['video-topics', { topics: [] }], + ['video-summary', { summary: ' ' }], + ])('blocks publishing without %s', (id, patch) => { + const check = find({ ...readyVideo, ...patch }, id); + expect(check?.severity).toBe('blocker'); + expect(check?.pass).toBe(false); + }); + + it('only warns when the duration is unknown', () => { + const check = find({ ...readyVideo, durationLabel: undefined }, 'video-duration'); + expect(check?.severity).toBe('warn'); + expect(check?.pass).toBe(false); + }); + + it('accepts a duration that came from the source', () => { + const check = find( + { ...readyVideo, durationLabel: undefined, source: { durationSeconds: 252 } }, + 'video-duration', + ); + expect(check?.pass).toBe(true); + }); + + it('runs no page checks (SEO title, slug, meta description) on a video', () => { + const ids = runPublishChecks(readyVideo, 'videos').map((c) => c.id); + expect(ids.every((id) => id.startsWith('video-'))).toBe(true); + }); + + it('adds no video checks to other collections', () => { + const ids = runPublishChecks(validBase, 'blogs').map((c) => c.id); + expect(ids.some((id) => id.startsWith('video-'))).toBe(false); + }); + + it('rejects a publish that is missing a format', () => { + expect(() => + runHook({ ...readyVideo, format: undefined, _status: 'published' }, 'videos'), + ).toThrow(ValidationError); + }); +}); diff --git a/apps/cms/src/payload/hooks/publish-gate.ts b/apps/cms/src/payload/hooks/publish-gate.ts index ae5cad067..c08f24084 100644 --- a/apps/cms/src/payload/hooks/publish-gate.ts +++ b/apps/cms/src/payload/hooks/publish-gate.ts @@ -53,6 +53,9 @@ export const runPublishChecks = ( data: Record, collectionSlug: string, ): CheckResult[] => { + // Videos play inline on /videos: no page, slug or SEO block, so none of the page checks apply. + if (collectionSlug === 'videos') return videoChecks(data); + const seo = (data.seo ?? {}) as Record; const checks: CheckResult[] = []; @@ -151,6 +154,48 @@ export const runPublishChecks = ( return checks; }; +/** + * The sync imports a video as a draft with no format, because a format cannot + * be guessed from YouTube. These blockers keep an unreviewed import from going + * live; the sync itself only ever saves drafts, so it is never stopped by them. + */ +const videoChecks = (data: Record): CheckResult[] => { + const topics = Array.isArray(data.topics) ? data.topics : []; + const source = (data.source ?? {}) as Record; + const hasDuration = + resolveString(data.durationLabel) != null || typeof source.durationSeconds === 'number'; + const blocker = (id: string, label: string, pass: boolean, message: string): CheckResult => ({ + id, + label, + severity: 'blocker', + pass, + message: pass ? null : message, + }); + + return [ + blocker( + 'video-format', + 'Category is chosen', + resolveString(data.format) != null, + 'Choose a category.', + ), + blocker('video-topics', 'At least one topic', topics.length > 0, 'Add at least one topic.'), + blocker( + 'video-summary', + 'Summary is written', + resolveString(data.summary) != null, + 'Write the summary shown on the card and in search results.', + ), + { + id: 'video-duration', + label: 'Duration is known', + severity: 'warn', + pass: hasDuration, + message: hasDuration ? null : 'No length yet: the card will show none. YouTube reports it once the sync has run with an API key.', + }, + ]; +}; + /** * beforeChange hook factory for the publishing checklist gate. * diff --git a/apps/cms/src/payload/hooks/webhooks-publish.test.ts b/apps/cms/src/payload/hooks/webhooks-publish.test.ts index cd676ab48..9f1eb7358 100644 --- a/apps/cms/src/payload/hooks/webhooks-publish.test.ts +++ b/apps/cms/src/payload/hooks/webhooks-publish.test.ts @@ -47,6 +47,19 @@ describe('webhooksPublishAfterChangeHook', () => { ); }); + it('stays quiet when the write opts out with skipPublishWebhooks (bulk backfills)', async () => { + const hook = webhooksPublishAfterChangeHook('videos'); + const req = { ...makeReq(), context: { skipPublishWebhooks: true } } as unknown as Parameters< + ReturnType + >[0]['req']; + await hook({ + doc: { id: 1, slug: 'x', title: 'X', _status: 'published', updatedAt: '2026-05-05' }, + previousDoc: { _status: 'draft' }, + req, + } as unknown as Parameters>[0]); + expect(dispatchSpy).not.toHaveBeenCalled(); + }); + it('includes updatedAt only when re-publishing older content (publishedAt much earlier)', async () => { dispatchSpy.mockResolvedValueOnce([]); const hook = webhooksPublishAfterChangeHook('blogs'); diff --git a/apps/cms/src/payload/hooks/webhooks-publish.ts b/apps/cms/src/payload/hooks/webhooks-publish.ts index 7f8e18478..028a94b55 100644 --- a/apps/cms/src/payload/hooks/webhooks-publish.ts +++ b/apps/cms/src/payload/hooks/webhooks-publish.ts @@ -35,6 +35,9 @@ export const webhooksPublishAfterChangeHook = const wasPublished = previous === 'published'; const isPublished = current === 'published'; if (!isPublished || wasPublished) return doc; + // Bulk scripts (e.g. the videos backfill, which publishes a whole catalog at once) + // set this in the write's context so a backfill does not post one card per doc. + if ((req.context as Record | undefined)?.skipPublishWebhooks === true) return doc; const requestId = getRequestId(req.headers as { get(name: string): string | null }); const typed = doc as Record; diff --git a/apps/cms/src/payload/jobs/sync-youtube-videos.ts b/apps/cms/src/payload/jobs/sync-youtube-videos.ts new file mode 100644 index 000000000..087d40fb7 --- /dev/null +++ b/apps/cms/src/payload/jobs/sync-youtube-videos.ts @@ -0,0 +1,44 @@ +import type { TaskConfig } from 'payload'; + +import { resolveYoutubeCredentials } from '../lib/integrations/credentials'; +import { notifyNeedsReview } from '../lib/videos/notify-needs-review'; +import { type RunSyncPayload, runYoutubeSync } from '../lib/videos/run-sync'; +import { createYoutubeClient } from '../lib/videos/youtube-client'; +import { createYoutubeRssClient } from '../lib/videos/youtube-rss-client'; + +/** + * Hourly at minute 7 UTC: imports new public YouTube uploads as needs-review + * drafts and refreshes the synced snapshot of existing ones. Uses the Data API + * when YOUTUBE_API_KEY is set (duration, privacy, missing-video detection) and + * the RSS feed otherwise. New videos get a suggested category from the optional + * explainer and demo playlists, and one Teams message announces a run that created + * any. Gated by PAYLOAD_AUTO_RUN. + */ +export const syncYoutubeVideosTask: TaskConfig<'syncYoutubeVideos'> = { + slug: 'syncYoutubeVideos', + retries: 1, + schedule: [{ cron: '7 * * * *', queue: 'youtubeVideosSync' }], + handler: async ({ req }) => { + const creds = resolveYoutubeCredentials(); + const mode = creds.apiKey ? 'api' : 'rss'; + // Logged every run so a missing key is visible, never silent. + req.payload.logger.info({ mode }, 'syncYoutubeVideos starting'); + for (const name of creds.invalidPlaylistVars) { + req.payload.logger.warn({ name }, 'syncYoutubeVideos: ignoring a playlist setting that is not a playlist id or youtube.com URL'); + } + const { createdTitles, ...result } = await runYoutubeSync({ + payload: req.payload as unknown as RunSyncPayload, + client: creds.apiKey ? createYoutubeClient({ apiKey: creds.apiKey }) : createYoutubeRssClient(), + channelId: creds.channelId, + autoPublish: process.env.VIDEOS_AUTO_PUBLISH === 'true', + playlists: { explainer: creds.explainerPlaylistId, demo: creds.demoPlaylistId }, + }); + await notifyNeedsReview({ + payload: req.payload, + count: result.created, + titles: createdTitles, + skip: req.context?.skipPublishWebhooks === true, + }); + return { output: { mode, ...result } }; + }, +}; diff --git a/apps/cms/src/payload/lib/export/exportable-collections.ts b/apps/cms/src/payload/lib/export/exportable-collections.ts index 50417b620..bba0c4ae4 100644 --- a/apps/cms/src/payload/lib/export/exportable-collections.ts +++ b/apps/cms/src/payload/lib/export/exportable-collections.ts @@ -20,6 +20,7 @@ export const EXPORTABLE_COLLECTION_SLUGS = [ 'blogs', 'advisories', + 'videos', 'news', 'guides', 'case-studies', diff --git a/apps/cms/src/payload/lib/integrations/credentials.test.ts b/apps/cms/src/payload/lib/integrations/credentials.test.ts index 01b07b624..a85c9e16d 100644 --- a/apps/cms/src/payload/lib/integrations/credentials.test.ts +++ b/apps/cms/src/payload/lib/integrations/credentials.test.ts @@ -9,6 +9,7 @@ import { resolveGscCredentials, resolveHubspotCredentials, resolveTeamsCredentials, + resolveYoutubeCredentials, } from './credentials'; import { encrypt } from './secrets'; @@ -23,6 +24,10 @@ const ENV_KEYS = [ 'CLOUDFLARE_ACCOUNT_TAG', 'CLOUDFLARE_API_TOKEN', 'CALCOM_SIGNING_SECRET', + 'YOUTUBE_API_KEY', + 'YOUTUBE_CHANNEL_ID', + 'YOUTUBE_EXPLAINER_PLAYLIST_ID', + 'YOUTUBE_DEMO_PLAYLIST_ID', ] as const; const snapshot: Record = {}; @@ -232,3 +237,36 @@ describe('resolveCalcomCredentials', () => { expect(resolveCalcomCredentials({ calcomConfig: { fallbackFormId: 9 } })?.fallbackFormId).toBe(9); }); }); + +describe('resolveYoutubeCredentials', () => { + it('has no playlists by default and falls back to the CleanStart channel', () => { + expect(resolveYoutubeCredentials()).toEqual({ + apiKey: null, + channelId: 'UC13c-XRmRlkWpp6WMG9SDjw', + explainerPlaylistId: null, + demoPlaylistId: null, + invalidPlaylistVars: [], + }); + }); + + it('accepts a bare id for one playlist and a full URL for the other', () => { + process.env.YOUTUBE_EXPLAINER_PLAYLIST_ID = 'PLexplainer00001'; + process.env.YOUTUBE_DEMO_PLAYLIST_ID = 'https://www.youtube.com/playlist?list=PLdemo000000001'; + const creds = resolveYoutubeCredentials(); + expect(creds).toMatchObject({ + explainerPlaylistId: 'PLexplainer00001', + demoPlaylistId: 'PLdemo000000001', + invalidPlaylistVars: [], + }); + }); + + it('ignores an invalid value and names the variable so the caller can warn', () => { + process.env.YOUTUBE_EXPLAINER_PLAYLIST_ID = 'https://example.com/playlist?list=PLexplainer00001'; + process.env.YOUTUBE_DEMO_PLAYLIST_ID = 'PLdemo000000001'; + expect(resolveYoutubeCredentials()).toMatchObject({ + explainerPlaylistId: null, + demoPlaylistId: 'PLdemo000000001', + invalidPlaylistVars: ['YOUTUBE_EXPLAINER_PLAYLIST_ID'], + }); + }); +}); diff --git a/apps/cms/src/payload/lib/integrations/credentials.ts b/apps/cms/src/payload/lib/integrations/credentials.ts index 190ae2068..4352fe9f6 100644 --- a/apps/cms/src/payload/lib/integrations/credentials.ts +++ b/apps/cms/src/payload/lib/integrations/credentials.ts @@ -1,3 +1,5 @@ +import { parsePlaylistId } from '../videos/playlist-id'; +import type { WebhookEventName } from '../webhooks/dispatch'; import { decrypt, isEncrypted } from './secrets'; /** @@ -54,7 +56,7 @@ export interface TeamsCredentials { displayName: string; aadObjectId: string; upn: string; - triggerOn?: ReadonlyArray<'document.published' | 'lead.submitted'>; + triggerOn?: ReadonlyArray; }>; } @@ -194,6 +196,44 @@ export const resolveCruxCredentials = (): CruxCredentials | null => { return apiKey ? { apiKey } : null; }; +// ─── YouTube (video sync) ──────────────────────────────────────── + +export interface YoutubeCredentials { + /** null selects the no-key RSS client: a missing key is a supported mode, not an error. */ + readonly apiKey: string | null; + readonly channelId: string; + /** Playlist whose members are suggested the Explainer category; null when unset or invalid. */ + readonly explainerPlaylistId: string | null; + readonly demoPlaylistId: string | null; + /** Names of playlist env vars that were set but held neither a playlist id nor a youtube.com URL. */ + readonly invalidPlaylistVars: readonly string[]; +} + +const PLAYLIST_VARS = { + explainer: 'YOUTUBE_EXPLAINER_PLAYLIST_ID', + demo: 'YOUTUBE_DEMO_PLAYLIST_ID', +} as const; + +export const resolveYoutubeCredentials = (): YoutubeCredentials => { + const invalidPlaylistVars: string[] = []; + const playlist = (name: string): string | null => { + const raw = readEnv(name); + if (!raw) return null; + const id = parsePlaylistId(raw); + if (!id) invalidPlaylistVars.push(name); + return id; + }; + const explainerPlaylistId = playlist(PLAYLIST_VARS.explainer); + const demoPlaylistId = playlist(PLAYLIST_VARS.demo); + return { + apiKey: readEnv('YOUTUBE_API_KEY'), + channelId: readEnv('YOUTUBE_CHANNEL_ID') ?? 'UC13c-XRmRlkWpp6WMG9SDjw', + explainerPlaylistId, + demoPlaylistId, + invalidPlaylistVars, + }; +}; + // ─── MS Clarity ────────────────────────────────────────────────── export interface ClarityCredentials { diff --git a/apps/cms/src/payload/lib/integrations/router.test.ts b/apps/cms/src/payload/lib/integrations/router.test.ts index ba994a092..b518e22c6 100644 --- a/apps/cms/src/payload/lib/integrations/router.test.ts +++ b/apps/cms/src/payload/lib/integrations/router.test.ts @@ -13,6 +13,11 @@ const leadEvent = (formSlug: string): WebhookEvent => ({ data: { formSlug, email: 'a@b.com' }, }); +const videoEvent: WebhookEvent = { + event: 'video.needs-review', + data: { count: 2, titles: ['A', 'B'] }, +}; + describe('integrations/router', () => { it('returns false when the event is not subscribed', () => { const routing: IntegrationRouting = { events: ['lead.submitted'] }; @@ -72,4 +77,25 @@ describe('integrations/router', () => { // lead events are unaffected by the collections filter. expect(routingMatches(routing, leadEvent('contact'))).toBe(true); }); + + it('matches video.needs-review for an empty events list and for a row subscribed to it', () => { + expect(routingMatches({ events: [] }, videoEvent)).toBe(true); + expect(routingMatches({ events: ['video.needs-review'] }, videoEvent)).toBe(true); + }); + + it('does not deliver video.needs-review to rows subscribed to other events only', () => { + expect(routingMatches({ events: ['document.published', 'lead.submitted'] }, videoEvent)).toBe(false); + }); + + it('ignores the collection and form filters for video.needs-review', () => { + expect( + routingMatches({ events: ['video.needs-review'], collections: ['blogs'], formSlugs: ['contact'] }, videoEvent), + ).toBe(true); + }); + + it('does not deliver content or lead events to a row subscribed only to video.needs-review', () => { + const routing: IntegrationRouting = { events: ['video.needs-review'] }; + expect(routingMatches(routing, publishEvent('blogs'))).toBe(false); + expect(routingMatches(routing, leadEvent('contact'))).toBe(false); + }); }); diff --git a/apps/cms/src/payload/lib/integrations/router.ts b/apps/cms/src/payload/lib/integrations/router.ts index 934407563..e2b5b4abc 100644 --- a/apps/cms/src/payload/lib/integrations/router.ts +++ b/apps/cms/src/payload/lib/integrations/router.ts @@ -7,6 +7,8 @@ import type { WebhookEvent, WebhookEventName } from '../webhooks/dispatch'; * collections — limit `document.published` to specific collection slugs * formSlugs — limit `lead.submitted` to specific form slugs * + * `video.needs-review` has no filters beyond `events`. + * * Empty arrays mean "no filter" for all three. An empty `events` list is * "subscribe to everything" — matching the admin UI, which shows an "All * events" chip and the guidance "Leave all filters empty to receive diff --git a/apps/cms/src/payload/lib/route-prefixes.test.ts b/apps/cms/src/payload/lib/route-prefixes.test.ts index cfd2d5e1b..17d07f1ac 100644 --- a/apps/cms/src/payload/lib/route-prefixes.test.ts +++ b/apps/cms/src/payload/lib/route-prefixes.test.ts @@ -27,6 +27,13 @@ describe('collectionUrlFromSlug / collectionUrlFromDoc (detail URLs)', () => { }); }); +describe('videos have a listing but no detail URL', () => { + it('composes no detail URL, because videos play inline', () => { + expect(collectionUrlFromSlug('videos', 'x')).toBeNull(); + expect(collectionUrlFromDoc('videos', { slug: 'x' })).toBeNull(); + }); +}); + describe('listingPathForCollection (index URLs for revalidation)', () => { it('overrides collections whose listing route differs from the detail prefix', () => { // These differ from ROUTE_PREFIX; revalidating the prefix would purge a diff --git a/apps/cms/src/payload/lib/videos/apply-sync.test.ts b/apps/cms/src/payload/lib/videos/apply-sync.test.ts new file mode 100644 index 000000000..a1e14dcff --- /dev/null +++ b/apps/cms/src/payload/lib/videos/apply-sync.test.ts @@ -0,0 +1,149 @@ +import { describe, expect, it, vi } from 'vitest'; +import { applyPlan, firstParagraph } from './apply-sync'; +import { type SyncPlan, toSource } from './plan-sync'; + +const source = toSource({ + id: 'AAAAAAAAAAA', + title: 'T', + description: 'First paragraph.\n\nmore', + publishedAt: '2026-10-13T14:17:14Z', + durationSeconds: 101, + tags: ['sbom'], + privacyStatus: 'public', + embeddable: true, + liveBroadcastContent: 'none', + hasMaxres: true, +}); +const emptyPlan: SyncPlan = { creates: [], updates: [], unchanged: 0, missing: [], skipped: [] }; +const now = (): Date => new Date('2026-10-23T12:00:00Z'); + +const fakePayload = () => ({ + create: vi.fn().mockResolvedValue({ id: 1 }), + update: vi.fn().mockResolvedValue({ id: 1 }), + logger: { info: vi.fn(), warn: vi.fn(), error: vi.fn() }, +}); + +const callData = (mock: ReturnType, n = 0): Record => + (mock.mock.calls[n]?.[0] as { data: Record }).data; + +describe('firstParagraph', () => { + it('takes the first real paragraph, collapsing whitespace', () => { + expect(firstParagraph('Hello there\nfriend.\n\nMore')).toBe('Hello there friend.'); + }); + + it('skips a leading hashtag-only paragraph', () => { + expect(firstParagraph('#SBOM #VEX\n\nThe real text.')).toBe('The real text.'); + }); + + it('never exceeds the 320 character summary limit', () => { + const out = firstParagraph('word '.repeat(200)); + expect(out.length).toBeLessThanOrEqual(320); + expect(out.endsWith('word.')).toBe(true); + }); +}); + +describe('applyPlan', () => { + it('creates a needs-review draft with title, summary, topics and the source snapshot', async () => { + const payload = fakePayload(); + const plan = { ...emptyPlan, creates: [{ externalId: 'AAAAAAAAAAA', source, suggestedTopicIds: [1, 2], suggestedFormat: null }] }; + const result = await applyPlan(payload, plan, { autoPublish: false }, now); + expect(payload.create).toHaveBeenCalledWith( + expect.objectContaining({ collection: 'videos', draft: true, overrideAccess: true, depth: 0 }), + ); + expect(callData(payload.create)).toMatchObject({ + provider: 'youtube', + externalId: 'AAAAAAAAAAA', + title: 'T', + summary: 'First paragraph.', + topics: [1, 2], + syncState: 'needs-review', + _status: 'draft', + source: { + title: 'T', + durationSeconds: 101, + tags: [{ value: 'sbom' }], + fetchedAt: '2026-10-23T12:00:00.000Z', + }, + }); + expect(result.created).toBe(1); + }); + + it('writes the suggested category on create, still as a needs-review draft', async () => { + const payload = fakePayload(); + const plan = { + ...emptyPlan, + creates: [{ externalId: 'AAAAAAAAAAA', source, suggestedTopicIds: [], suggestedFormat: 'demo' as const }], + }; + await applyPlan(payload, plan, { autoPublish: true }, now); + expect(callData(payload.create)).toMatchObject({ format: 'demo', syncState: 'needs-review', _status: 'draft' }); + expect(payload.create).toHaveBeenCalledWith(expect.objectContaining({ draft: true })); + }); + + it('omits format entirely when nothing is suggested', async () => { + const payload = fakePayload(); + const plan = { + ...emptyPlan, + creates: [{ externalId: 'AAAAAAAAAAA', source, suggestedTopicIds: [], suggestedFormat: null }], + }; + await applyPlan(payload, plan, { autoPublish: false }, now); + expect(callData(payload.create)).not.toHaveProperty('format'); + }); + + it('reports the titles of the videos it created, skipping failures', async () => { + const payload = fakePayload(); + payload.create.mockRejectedValueOnce(new Error('boom')).mockResolvedValueOnce({ id: 2 }); + const other = { ...source, title: 'Second' }; + const plan = { + ...emptyPlan, + creates: [ + { externalId: 'AAAAAAAAAAA', source, suggestedTopicIds: [], suggestedFormat: null }, + { externalId: 'BBBBBBBBBBB', source: other, suggestedTopicIds: [], suggestedFormat: null }, + ], + }; + const result = await applyPlan(payload, plan, { autoPublish: false }, now); + expect(result.createdTitles).toEqual(['Second']); + }); + + it('writes only the source group when updating', async () => { + const payload = fakePayload(); + await applyPlan(payload, { ...emptyPlan, updates: [{ id: 9, source, restoreFromMissing: false }] }, { autoPublish: false }, now); + expect(Object.keys(callData(payload.update))).toEqual(['source']); + }); + + it('restores syncState when a missing video returns', async () => { + const payload = fakePayload(); + await applyPlan(payload, { ...emptyPlan, updates: [{ id: 9, source, restoreFromMissing: true }] }, { autoPublish: false }, now); + expect(callData(payload.update).syncState).toBe('synced'); + }); + + it('marks vanished videos missing without unpublishing them', async () => { + const payload = fakePayload(); + await applyPlan(payload, { ...emptyPlan, missing: [4] }, { autoPublish: false }, now); + expect(payload.update).toHaveBeenCalledWith(expect.objectContaining({ id: 4, data: { syncState: 'missing' } })); + }); + + it('keeps going when one item fails and reports it without leaking details beyond the message', async () => { + const payload = fakePayload(); + payload.create.mockRejectedValueOnce(new Error('boom')).mockResolvedValueOnce({ id: 2 }); + const plan = { + ...emptyPlan, + creates: [ + { externalId: 'AAAAAAAAAAA', source, suggestedTopicIds: [], suggestedFormat: null }, + { externalId: 'BBBBBBBBBBB', source, suggestedTopicIds: [], suggestedFormat: null }, + ], + }; + const result = await applyPlan(payload, plan, { autoPublish: false }, now); + expect(result).toMatchObject({ created: 1, failed: 1 }); + expect(payload.logger.error).toHaveBeenCalledWith( + expect.objectContaining({ externalId: 'AAAAAAAAAAA', error: 'boom' }), + expect.any(String), + ); + }); + + it('never publishes on import, even with autoPublish on: no format can be guessed', async () => { + const payload = fakePayload(); + const plan = { ...emptyPlan, creates: [{ externalId: 'AAAAAAAAAAA', source, suggestedTopicIds: [1], suggestedFormat: null }] }; + await applyPlan(payload, plan, { autoPublish: true }, now); + expect(callData(payload.create)._status).toBe('draft'); + }); +}); diff --git a/apps/cms/src/payload/lib/videos/apply-sync.ts b/apps/cms/src/payload/lib/videos/apply-sync.ts new file mode 100644 index 000000000..6a7a3b389 --- /dev/null +++ b/apps/cms/src/payload/lib/videos/apply-sync.ts @@ -0,0 +1,122 @@ +import { firstParagraph } from './description-text'; +import { type DocId, type SourceSnapshot, type SyncPlan, contentHash } from './plan-sync'; + +export { firstParagraph }; + +export interface SyncPayload { + create(args: { + collection: 'videos'; + data: Record; + draft: boolean; + overrideAccess: boolean; + depth: number; + }): Promise<{ id: DocId }>; + update(args: { + collection: 'videos'; + id: DocId; + data: Record; + overrideAccess: boolean; + depth: number; + }): Promise; + logger: { + info(meta: Record, message: string): void; + warn(meta: Record, message: string): void; + error(meta: Record, message: string): void; + }; +} + +export interface ApplyResult { + created: number; + updated: number; + markedMissing: number; + failed: number; + /** Titles of the videos created this run, in plan order. */ + createdTitles: string[]; +} + +const sourceData = (source: SourceSnapshot, fetchedAt: string): Record => ({ + title: source.title, + description: source.description, + durationSeconds: source.durationSeconds, + publishedAt: source.publishedAt, + tags: source.tags.map((value) => ({ value })), + privacyStatus: source.privacyStatus, + embeddable: source.embeddable, + hasMaxres: source.hasMaxres, + thumbnailUrl: source.thumbnailUrl, + contentHash: contentHash(source), + fetchedAt, +}); + +export const applyPlan = async ( + payload: SyncPayload, + plan: SyncPlan, + // `autoPublish` is accepted so the env switch has one home, but a new video is never published + // here: the publish gate needs a format, which nothing guesses. + _options: { autoPublish: boolean }, + now: () => Date = () => new Date(), +): Promise => { + const result: ApplyResult = { created: 0, updated: 0, markedMissing: 0, failed: 0, createdTitles: [] }; + const fetchedAt = now().toISOString(); + + // One bad document must never abort the batch. + const attempt = async (what: string, ref: Record, run: () => Promise): Promise => { + try { + await run(); + return true; + } catch (error) { + result.failed += 1; + payload.logger.error( + { ...ref, error: error instanceof Error ? error.message : String(error) }, + `videos sync: ${what} failed`, + ); + return false; + } + }; + + for (const item of plan.creates) { + const ok = await attempt('create', { externalId: item.externalId }, () => + payload.create({ + collection: 'videos', + draft: true, + overrideAccess: true, + depth: 0, + data: { + provider: 'youtube', + externalId: item.externalId, + title: item.source.title, + summary: firstParagraph(item.source.description), + topics: item.suggestedTopicIds, + // Create only: an editor's later choice must survive every re-sync. + ...(item.suggestedFormat ? { format: item.suggestedFormat } : {}), + syncState: 'needs-review', + source: sourceData(item.source, fetchedAt), + _status: 'draft', + }, + }), + ); + if (ok) { + result.created += 1; + result.createdTitles.push(item.source.title); + } + } + + for (const item of plan.updates) { + const data: Record = { source: sourceData(item.source, fetchedAt) }; + if (item.restoreFromMissing) data.syncState = 'synced'; + const ok = await attempt('update', { id: item.id }, () => + payload.update({ collection: 'videos', id: item.id, data, overrideAccess: true, depth: 0 }), + ); + if (ok) result.updated += 1; + } + + for (const id of plan.missing) { + const ok = await attempt('mark missing', { id }, () => + payload.update({ collection: 'videos', id, data: { syncState: 'missing' }, overrideAccess: true, depth: 0 }), + ); + if (ok) result.markedMissing += 1; + } + + payload.logger.info({ ...result, unchanged: plan.unchanged, skipped: plan.skipped.length }, 'videos sync applied'); + return result; +}; diff --git a/apps/cms/src/payload/lib/videos/description-text.ts b/apps/cms/src/payload/lib/videos/description-text.ts new file mode 100644 index 000000000..e699bd8cc --- /dev/null +++ b/apps/cms/src/payload/lib/videos/description-text.ts @@ -0,0 +1,18 @@ +// Pure and free of Node imports: used by the sync and by the "use the first paragraph" button in the admin. + +// One below the `summary` field's 320 maxLength, so the closing "." still fits. +const SUMMARY_CUT = 319; +const isHashtagLine = (text: string): boolean => text.split(/\s+/).every((t) => t.startsWith('#')); + +/** First real paragraph of a description, whitespace collapsed, cut at a word boundary. */ +export const firstParagraph = (text: string): string => { + const paragraph = + text + .split(/\n\s*\n/) + .map((p) => p.replace(/\s+/g, ' ').trim()) + .find((p) => p.length > 0 && !isHashtagLine(p)) ?? ''; + if (paragraph.length <= SUMMARY_CUT) return paragraph; + const cut = paragraph.slice(0, SUMMARY_CUT); + const lastSpace = cut.lastIndexOf(' '); + return `${cut.slice(0, lastSpace > 0 ? lastSpace : SUMMARY_CUT).replace(/[,;:\s]+$/, '')}.`; +}; diff --git a/apps/cms/src/payload/lib/videos/notify-needs-review.test.ts b/apps/cms/src/payload/lib/videos/notify-needs-review.test.ts new file mode 100644 index 000000000..f6ee6cc07 --- /dev/null +++ b/apps/cms/src/payload/lib/videos/notify-needs-review.test.ts @@ -0,0 +1,68 @@ +import type { BasePayload } from 'payload'; +import { afterEach, describe, expect, it, vi } from 'vitest'; + +import { notifyNeedsReview, needsReviewListUrl } from './notify-needs-review'; + +const fakePayload = (): BasePayload => ({ logger: { warn: vi.fn() } }) as unknown as BasePayload; +const SERVER = 'https://cms.cleanstart.com/'; + +afterEach(() => vi.unstubAllEnvs()); + +describe('needsReviewListUrl', () => { + it('points at the videos list filtered to Needs review, without a double slash', () => { + expect(needsReviewListUrl(SERVER)).toBe( + 'https://cms.cleanstart.com/admin/collections/videos?where[syncState][equals]=needs-review', + ); + }); + + it('is null without a base URL', () => { + expect(needsReviewListUrl(undefined)).toBeNull(); + expect(needsReviewListUrl('')).toBeNull(); + }); +}); + +describe('notifyNeedsReview', () => { + it('sends nothing when no video was created', async () => { + const dispatch = vi.fn(); + await notifyNeedsReview({ payload: fakePayload(), count: 0, titles: [], skip: false, serverUrl: SERVER, dispatch }); + expect(dispatch).not.toHaveBeenCalled(); + }); + + it('sends nothing in a bulk script context', async () => { + const dispatch = vi.fn(); + await notifyNeedsReview({ payload: fakePayload(), count: 4, titles: ['A'], skip: true, serverUrl: SERVER, dispatch }); + expect(dispatch).not.toHaveBeenCalled(); + }); + + it('sends one event with count, at most five titles and the admin URL', async () => { + const dispatch = vi.fn().mockResolvedValue([]); + const titles = ['A', 'B', 'C', 'D', 'E', 'F', 'G']; + await notifyNeedsReview({ payload: fakePayload(), count: 7, titles, skip: false, serverUrl: SERVER, dispatch }); + expect(dispatch).toHaveBeenCalledTimes(1); + expect(dispatch.mock.calls[0]?.[0]).toEqual({ + event: 'video.needs-review', + data: { + count: 7, + titles: ['A', 'B', 'C', 'D', 'E'], + adminUrl: 'https://cms.cleanstart.com/admin/collections/videos?where[syncState][equals]=needs-review', + }, + }); + }); + + it('omits adminUrl when the CMS base URL is not configured', async () => { + // An undefined argument falls back to the environment, so blank the variable the CI job sets. + vi.stubEnv('PAYLOAD_PUBLIC_SERVER_URL', ''); + const dispatch = vi.fn().mockResolvedValue([]); + await notifyNeedsReview({ payload: fakePayload(), count: 1, titles: ['A'], skip: false, serverUrl: undefined, dispatch }); + expect(dispatch.mock.calls[0]?.[0]).toEqual({ event: 'video.needs-review', data: { count: 1, titles: ['A'] } }); + }); + + it('never throws when dispatch does, and logs a warning', async () => { + const payload = fakePayload(); + const dispatch = vi.fn().mockRejectedValue(new Error('teams down')); + await expect( + notifyNeedsReview({ payload, count: 1, titles: ['A'], skip: false, serverUrl: SERVER, dispatch }), + ).resolves.toBeUndefined(); + expect(payload.logger.warn).toHaveBeenCalledWith({ error: 'teams down' }, expect.any(String)); + }); +}); diff --git a/apps/cms/src/payload/lib/videos/notify-needs-review.ts b/apps/cms/src/payload/lib/videos/notify-needs-review.ts new file mode 100644 index 000000000..b24dea882 --- /dev/null +++ b/apps/cms/src/payload/lib/videos/notify-needs-review.ts @@ -0,0 +1,48 @@ +import type { BasePayload } from 'payload'; + +import { type WebhookEvent, dispatchEvent } from '../webhooks/dispatch'; + +const MAX_TITLES = 5; + +export const needsReviewListUrl = (serverUrl: string | undefined): string | null => + serverUrl + ? `${serverUrl.replace(/\/+$/, '')}/admin/collections/videos?where[syncState][equals]=needs-review` + : null; + +/** + * One `video.needs-review` event per sync run. Never throws: a Teams outage must not fail the sync, + * and `dispatchEvent` already parks failed deliveries in the dead-letter queue for retry. + */ +export const notifyNeedsReview = async ({ + payload, + count, + titles, + skip, + serverUrl = process.env.PAYLOAD_PUBLIC_SERVER_URL, + dispatch = dispatchEvent, +}: { + payload: BasePayload; + count: number; + titles: readonly string[]; + /** True in a bulk-script write context (`context.skipPublishWebhooks`). */ + skip: boolean; + serverUrl?: string | undefined; + dispatch?: (event: WebhookEvent, options: { logger: BasePayload['logger']; payload: BasePayload }) => Promise; +}): Promise => { + if (count <= 0 || skip) return; + const adminUrl = needsReviewListUrl(serverUrl); + try { + await dispatch( + { + event: 'video.needs-review', + data: { count, titles: titles.slice(0, MAX_TITLES), ...(adminUrl ? { adminUrl } : {}) }, + }, + { logger: payload.logger, payload }, + ); + } catch (error) { + payload.logger.warn( + { error: error instanceof Error ? error.message : String(error) }, + 'videos sync: needs-review notification failed', + ); + } +}; diff --git a/apps/cms/src/payload/lib/videos/plan-backfill.test.ts b/apps/cms/src/payload/lib/videos/plan-backfill.test.ts new file mode 100644 index 000000000..4efc9a0ff --- /dev/null +++ b/apps/cms/src/payload/lib/videos/plan-backfill.test.ts @@ -0,0 +1,139 @@ +import { describe, expect, it } from 'vitest'; + +import { + type CatalogTopicSeed, + type CatalogVideoSeed, + type ExistingBackfillVideo, + type ExistingTopic, + planBackfill, +} from './plan-backfill'; + +const topics: CatalogTopicSeed[] = [ + { slug: 'sbom', name: 'SBOM', aliases: ['sboms'] }, + { slug: 'vex', name: 'VEX', aliases: [] }, +]; + +const seed = (over: Partial = {}): CatalogVideoSeed => ({ + externalId: 'AAAAAAAAAAA', + slug: 'what-is-vex', + title: 'What Is VEX?', + format: 'explainer', + topics: ['vex', 'sbom'], + summary: 'A summary.', + durationSeconds: 101, + publishedAt: '2026-10-13T14:17:14+00:00', + thumb: 'maxres', + ...over, +}); + +const doc = (over: Partial = {}): ExistingBackfillVideo => ({ + id: 7, + externalId: 'AAAAAAAAAAA', + slug: 'what-is-vex', + title: 'What Is VEX?', + format: 'explainer', + topicSlugs: ['vex', 'sbom'], + summary: 'A summary.', + status: 'published', + syncState: 'synced', + ...over, +}); + +const existingTopics: ExistingTopic[] = [ + { id: 1, slug: 'sbom', name: 'SBOM', aliases: ['sboms'] }, + { id: 2, slug: 'vex', name: 'VEX', aliases: [] }, +]; + +const run = (over: { + videos?: CatalogVideoSeed[]; + existing?: ExistingBackfillVideo[]; + topics?: ExistingTopic[]; +} = {}) => + planBackfill({ + catalogTopics: topics, + catalogVideos: over.videos ?? [seed()], + existingTopics: over.topics ?? existingTopics, + existingVideos: over.existing ?? [], + }); + +describe('planBackfill topics', () => { + it('creates topics that do not exist yet', () => { + expect(run({ topics: [] }).topics.create.map((t) => t.slug)).toEqual(['sbom', 'vex']); + }); + it('updates a topic whose name or aliases differ, ignoring alias order', () => { + const changed = [ + { id: 1, slug: 'sbom', name: 'SBOMs', aliases: ['sboms'] }, + { id: 2, slug: 'vex', name: 'VEX', aliases: [] }, + ]; + const plan = run({ topics: changed }); + expect(plan.topics.update).toEqual([{ id: 1, topic: topics[0] }]); + expect(plan.topics.unchanged).toBe(1); + const reordered = [{ id: 1, slug: 'sbom', name: 'SBOM', aliases: ['sboms'] }, { id: 2, slug: 'vex', name: 'VEX', aliases: [] }]; + expect(run({ topics: reordered }).topics.update).toEqual([]); + }); +}); + +describe('planBackfill videos', () => { + it('creates a video that is not in the CMS', () => { + expect(run().videos.create.map((v) => v.externalId)).toEqual(['AAAAAAAAAAA']); + }); + + it('leaves a fully matching, published, synced video alone', () => { + const plan = run({ existing: [doc()] }); + expect(plan.videos).toMatchObject({ create: [], update: [], unchanged: 1, conflicts: [] }); + }); + + it('updates and names every reason when the editorial layer differs', () => { + const plan = run({ + existing: [doc({ title: 'Old', format: null, topicSlugs: ['vex'], summary: '', status: 'draft', syncState: 'needs-review' })], + }); + expect(plan.videos.update).toHaveLength(1); + expect(plan.videos.update[0]?.reasons.sort()).toEqual(['format', 'publish', 'summary', 'syncState', 'title', 'topics']); + }); + + it('treats topic order as irrelevant', () => { + const plan = run({ existing: [doc({ topicSlugs: ['sbom', 'vex'] })] }); + expect(plan.videos.unchanged).toBe(1); + }); + + it('refuses to change a live slug (URLs are commitments)', () => { + const plan = run({ existing: [doc({ slug: 'different-slug' })] }); + expect(plan.videos.conflicts).toEqual([ + { externalId: 'AAAAAAAAAAA', kind: 'slug-differs', wanted: 'what-is-vex', existing: 'different-slug' }, + ]); + expect(plan.videos.update).toEqual([]); + }); + + it('refuses a slug already held by a different video', () => { + const plan = run({ existing: [doc({ id: 9, externalId: 'ZZZZZZZZZZZ', slug: 'what-is-vex' })] }); + expect(plan.videos.conflicts).toEqual([ + { externalId: 'AAAAAAAAAAA', kind: 'slug-taken', wanted: 'what-is-vex', existing: 'ZZZZZZZZZZZ' }, + ]); + expect(plan.videos.create).toEqual([]); + }); + + it('refuses to change the slug of a draft that was edited or published before', () => { + const plan = run({ existing: [doc({ slug: 'edited-slug', status: 'draft', syncState: 'synced' })] }); + expect(plan.videos.conflicts).toHaveLength(1); + }); + + it('adopts the catalog slug on an untouched import that got an auto slug from its title', () => { + const plan = run({ + existing: [doc({ slug: 'what-is-vex-how-it-tells-which-vulnerabilities-matter', status: 'draft', syncState: 'needs-review' })], + }); + expect(plan.videos.conflicts).toEqual([]); + expect(plan.videos.update[0]?.reasons).toContain('slug'); + }); + + it('adopts the catalog slug when the synced draft has none yet', () => { + const plan = run({ existing: [doc({ slug: null, status: 'draft', syncState: 'needs-review' })] }); + expect(plan.videos.conflicts).toEqual([]); + expect(plan.videos.update[0]?.reasons).toContain('slug'); + }); + + it('is idempotent: applying the plan and planning again changes nothing', () => { + const first = run({ existing: [doc({ status: 'draft', syncState: 'needs-review', slug: null, title: 'T' })] }); + expect(first.videos.update).toHaveLength(1); + expect(run({ existing: [doc()] }).videos.update).toEqual([]); + }); +}); diff --git a/apps/cms/src/payload/lib/videos/plan-backfill.ts b/apps/cms/src/payload/lib/videos/plan-backfill.ts new file mode 100644 index 000000000..b659271e4 --- /dev/null +++ b/apps/cms/src/payload/lib/videos/plan-backfill.ts @@ -0,0 +1,140 @@ +type DocId = number | string; + +export type VideoFormat = 'explainer' | 'demo' | 'talk'; + +export interface CatalogTopicSeed { + readonly slug: string; + readonly name: string; + readonly aliases: readonly string[]; +} + +export interface CatalogVideoSeed { + readonly externalId: string; + readonly slug: string; + readonly title: string; + readonly format: VideoFormat; + readonly topics: readonly string[]; + readonly summary: string; + readonly durationSeconds: number; + readonly publishedAt: string; + readonly thumb: 'maxres' | 'hq'; +} + +export interface ExistingTopic { + readonly id: DocId; + readonly slug: string; + readonly name: string; + readonly aliases: readonly string[]; +} + +export interface ExistingBackfillVideo { + readonly id: DocId; + readonly externalId: string; + readonly slug: string | null; + readonly title: string | null; + readonly format: string | null; + readonly topicSlugs: readonly string[]; + readonly summary: string | null; + readonly status: 'draft' | 'published'; + readonly syncState: string | null; +} + +export type UpdateReason = 'slug' | 'title' | 'format' | 'topics' | 'summary' | 'publish' | 'syncState'; + +export interface BackfillConflict { + readonly externalId: string; + readonly kind: 'slug-differs' | 'slug-taken'; + readonly wanted: string; + /** The slug already on the document, or the externalId of the document holding the wanted slug. */ + readonly existing: string; +} + +export interface BackfillPlan { + readonly topics: { + readonly create: CatalogTopicSeed[]; + readonly update: { id: DocId; topic: CatalogTopicSeed }[]; + readonly unchanged: number; + }; + readonly videos: { + readonly create: CatalogVideoSeed[]; + readonly update: { id: DocId; video: CatalogVideoSeed; reasons: UpdateReason[] }[]; + readonly unchanged: number; + readonly conflicts: BackfillConflict[]; + }; +} + +const sameSet = (a: readonly string[], b: readonly string[]): boolean => + a.length === b.length && [...a].sort().every((value, i) => value === [...b].sort()[i]); + +/** + * Decides what moving the code catalog into the CMS would change, without + * touching anything. Pure, so a dry run and the real run share one decision and + * a second run always plans nothing. A live slug is a URL commitment, so a + * difference there is a conflict that blocks the run, never an update. + */ +export const planBackfill = ({ + catalogTopics, + catalogVideos, + existingTopics, + existingVideos, +}: { + catalogTopics: readonly CatalogTopicSeed[]; + catalogVideos: readonly CatalogVideoSeed[]; + existingTopics: readonly ExistingTopic[]; + existingVideos: readonly ExistingBackfillVideo[]; +}): BackfillPlan => { + const topicBySlug = new Map(existingTopics.map((t) => [t.slug, t])); + const topicsCreate: CatalogTopicSeed[] = []; + const topicsUpdate: BackfillPlan['topics']['update'] = []; + let topicsUnchanged = 0; + for (const seed of catalogTopics) { + const current = topicBySlug.get(seed.slug); + if (!current) topicsCreate.push(seed); + else if (current.name !== seed.name || !sameSet(current.aliases, seed.aliases)) { + topicsUpdate.push({ id: current.id, topic: seed }); + } else topicsUnchanged += 1; + } + + const byExternalId = new Map(existingVideos.map((v) => [v.externalId, v])); + const bySlug = new Map(existingVideos.filter((v) => v.slug).map((v) => [v.slug as string, v])); + const videosCreate: CatalogVideoSeed[] = []; + const videosUpdate: BackfillPlan['videos']['update'] = []; + const conflicts: BackfillConflict[] = []; + let videosUnchanged = 0; + + for (const seed of catalogVideos) { + const current = byExternalId.get(seed.externalId); + const holder = bySlug.get(seed.slug); + if (holder && holder.externalId !== seed.externalId) { + conflicts.push({ externalId: seed.externalId, kind: 'slug-taken', wanted: seed.slug, existing: holder.externalId }); + continue; + } + if (!current) { + videosCreate.push(seed); + continue; + } + // The sync gives every import an auto slug from its title, so a differing slug on an + // untouched import (never published, still needs-review) is adopted, not a conflict. + // Anything live or edited keeps its URL: that is a commitment, so it blocks the run. + const slugIsAdoptable = !current.slug || (current.status === 'draft' && current.syncState === 'needs-review'); + if (current.slug && current.slug !== seed.slug && !slugIsAdoptable) { + conflicts.push({ externalId: seed.externalId, kind: 'slug-differs', wanted: seed.slug, existing: current.slug }); + continue; + } + const reasons: UpdateReason[] = []; + if (current.slug !== seed.slug) reasons.push('slug'); + if (current.title !== seed.title) reasons.push('title'); + if (current.format !== seed.format) reasons.push('format'); + if (!sameSet(current.topicSlugs, seed.topics)) reasons.push('topics'); + if ((current.summary ?? '') !== seed.summary) reasons.push('summary'); + if (current.status !== 'published') reasons.push('publish'); + if (current.syncState !== 'synced') reasons.push('syncState'); + if (reasons.length > 0) videosUpdate.push({ id: current.id, video: seed, reasons }); + else videosUnchanged += 1; + } + + return { + topics: { create: topicsCreate, update: topicsUpdate, unchanged: topicsUnchanged }, + videos: { create: videosCreate, update: videosUpdate, unchanged: videosUnchanged, conflicts }, + }; +}; diff --git a/apps/cms/src/payload/lib/videos/plan-sync.test.ts b/apps/cms/src/payload/lib/videos/plan-sync.test.ts new file mode 100644 index 000000000..6f6ca7aea --- /dev/null +++ b/apps/cms/src/payload/lib/videos/plan-sync.test.ts @@ -0,0 +1,178 @@ +import { describe, expect, it } from 'vitest'; +import type { YoutubeVideo } from './youtube-client'; +import { + type ExistingVideo, + contentHash, + normaliseTag, + planSync, + suggestTopicIds, + toSource, + withKnownDuration, +} from './plan-sync'; + +const video = (over: Partial = {}): YoutubeVideo => ({ + id: 'AAAAAAAAAAA', + title: 'T', + description: 'First paragraph.\n\n#SBOM #VEX', + publishedAt: '2026-10-13T14:17:14Z', + durationSeconds: 101, + tags: [], + privacyStatus: 'public', + embeddable: true, + liveBroadcastContent: 'none', + hasMaxres: true, + ...over, +}); + +const existing = (over: Partial = {}): ExistingVideo => ({ + id: 9, + externalId: 'AAAAAAAAAAA', + syncState: 'synced', + contentHash: null, + status: 'published', + durationSeconds: 101, + ...over, +}); + +const topics = [ + { id: 1, name: 'SBOM', aliases: ['software bill of materials'] }, + { id: 2, name: 'VEX', aliases: [] }, + { id: 3, name: 'Base images', aliases: [] }, +]; + +describe('normaliseTag', () => { + it('ignores case, hashes, hyphens and spacing', () => { + expect(normaliseTag('#Base Images')).toBe('baseimages'); + expect(normaliseTag('base-images')).toBe('baseimages'); + }); +}); + +describe('suggestTopicIds', () => { + it('matches hashtags in the description and YouTube tags, once each', () => { + const ids = suggestTopicIds( + { tags: ['Software Bill of Materials'], description: 'x\n\n#VEX #ContainerSecurity #sbom' }, + topics, + ); + expect(ids).toEqual([1, 2]); + }); + + it('suggests nothing when nothing matches', () => { + expect(suggestTopicIds({ tags: ['kubernetes'], description: '#devops' }, topics)).toEqual([]); + }); + + it('only reads hashtags that start a word, not a "#" inside a URL fragment', () => { + expect(suggestTopicIds({ tags: [], description: 'see https://x.test/page#vex' }, topics)).toEqual([]); + }); +}); + +describe('planSync', () => { + const base = { existing: [], complete: true, topics }; + + it('creates a draft candidate for a new public video, with suggested topics', () => { + const plan = planSync({ ...base, fetched: [video()] }); + expect(plan.creates).toHaveLength(1); + expect(plan.creates[0]).toMatchObject({ externalId: 'AAAAAAAAAAA', suggestedTopicIds: [1, 2] }); + }); + + it('carries the playlist-suggested category on a create, and null when the video is in no mapped playlist', () => { + const formatByVideoId = new Map([['AAAAAAAAAAA', 'demo' as const]]); + const plan = planSync({ ...base, fetched: [video(), video({ id: 'BBBBBBBBBBB' })], formatByVideoId }); + expect(plan.creates.map((c) => c.suggestedFormat)).toEqual(['demo', null]); + }); + + it('defaults to no suggestion when no map is passed', () => { + expect(planSync({ ...base, fetched: [video()] }).creates[0]?.suggestedFormat).toBeNull(); + }); + + it('does not put a suggestion on an update', () => { + const formatByVideoId = new Map([['AAAAAAAAAAA', 'demo' as const]]); + const plan = planSync({ ...base, existing: [existing({ contentHash: 'old' })], fetched: [video()], formatByVideoId }); + expect(plan.updates).toHaveLength(1); + expect(plan.updates[0]).not.toHaveProperty('suggestedFormat'); + }); + + it.each([ + ['unlisted', video({ privacyStatus: 'unlisted' }), 'not-public'], + ['private', video({ privacyStatus: 'private' }), 'not-public'], + ['not embeddable', video({ embeddable: false }), 'not-embeddable'], + ['upcoming', video({ liveBroadcastContent: 'upcoming' }), 'not-finished'], + ['live', video({ liveBroadcastContent: 'live' }), 'not-finished'], + ])('skips a %s video', (_label, v, reason) => { + const plan = planSync({ ...base, fetched: [v] }); + expect(plan.creates).toEqual([]); + expect(plan.skipped).toEqual([{ externalId: 'AAAAAAAAAAA', reason }]); + }); + + it('leaves an unchanged video alone', () => { + const v = video(); + const plan = planSync({ + ...base, + fetched: [v], + existing: [existing({ contentHash: contentHash(toSource(v)) })], + }); + expect(plan).toMatchObject({ creates: [], updates: [], unchanged: 1 }); + }); + + it('updates only the source when YouTube content changed', () => { + const v = video({ description: 'edited' }); + const plan = planSync({ ...base, fetched: [v], existing: [existing({ contentHash: 'old' })] }); + expect(plan.updates).toEqual([{ id: 9, source: toSource(v), restoreFromMissing: false }]); + }); + + it('flags a vanished video as missing, but only when the listing was complete', () => { + const gone = existing({ externalId: 'GONEGONEGON', contentHash: 'h' }); + expect(planSync({ ...base, fetched: [], existing: [gone] }).missing).toEqual([9]); + expect(planSync({ ...base, fetched: [], existing: [gone], complete: false }).missing).toEqual([]); + }); + + it('never touches manual documents', () => { + const manual = existing({ id: 5, externalId: 'MANUALMANUA', syncState: 'manual', durationSeconds: null }); + const plan = planSync({ ...base, fetched: [video({ id: 'MANUALMANUA' })], existing: [manual] }); + expect(plan).toMatchObject({ creates: [], updates: [], missing: [], unchanged: 1 }); + expect(planSync({ ...base, fetched: [], existing: [manual] }).missing).toEqual([]); + }); + + it('brings a missing video back when it reappears', () => { + const v = video(); + const plan = planSync({ + ...base, + fetched: [v], + existing: [existing({ syncState: 'missing', contentHash: contentHash(toSource(v)) })], + }); + expect(plan.updates).toEqual([{ id: 9, source: toSource(v), restoreFromMissing: true }]); + }); + + it('does not treat an RSS run (no duration) as a change, and never erases a known duration', () => { + const stored = toSource(video()); + const rssView = video({ durationSeconds: null }); + const plan = planSync({ + ...base, + fetched: [rssView], + existing: [existing({ contentHash: contentHash(stored) })], + }); + expect(plan).toMatchObject({ updates: [], unchanged: 1 }); + }); + + it('keeps the stored duration inside an update that changes something else', () => { + const rssView = video({ durationSeconds: null, description: 'edited' }); + const plan = planSync({ ...base, fetched: [rssView], existing: [existing({ contentHash: 'old' })] }); + expect(plan.updates[0]?.source.durationSeconds).toBe(101); + }); +}); + +describe('withKnownDuration', () => { + it('only fills in a missing duration', () => { + const unknown = toSource(video({ durationSeconds: null })); + expect(withKnownDuration(unknown, 55).durationSeconds).toBe(55); + expect(withKnownDuration(unknown, null).durationSeconds).toBeNull(); + expect(withKnownDuration(toSource(video()), 55).durationSeconds).toBe(101); + }); +}); + +describe('contentHash', () => { + it('is stable and sensitive to every snapshot field it covers', () => { + const a = toSource(video()); + expect(contentHash(a)).toBe(contentHash(toSource(video()))); + expect(contentHash(a)).not.toBe(contentHash(toSource(video({ title: 'other' })))); + }); +}); diff --git a/apps/cms/src/payload/lib/videos/plan-sync.ts b/apps/cms/src/payload/lib/videos/plan-sync.ts new file mode 100644 index 000000000..8b8c7c933 --- /dev/null +++ b/apps/cms/src/payload/lib/videos/plan-sync.ts @@ -0,0 +1,127 @@ +import { createHash } from 'node:crypto'; + +import type { YoutubeVideo } from './youtube-client'; +import { type TopicRef, normaliseTag, suggestTopicIds } from './topic-match'; + +export { normaliseTag, suggestTopicIds }; +export type { TopicRef }; + +export type SyncState = 'synced' | 'needs-review' | 'missing' | 'manual'; +export type SkipReason = 'not-public' | 'not-embeddable' | 'not-finished'; +export type DocId = number | string; +export type VideoFormat = 'explainer' | 'demo'; + +export interface SourceSnapshot { + readonly title: string; + readonly description: string; + readonly durationSeconds: number | null; + readonly publishedAt: string; + readonly tags: readonly string[]; + readonly privacyStatus: string; + readonly embeddable: boolean; + readonly hasMaxres: boolean; + readonly thumbnailUrl: string; +} + +export interface ExistingVideo { + readonly id: DocId; + readonly externalId: string; + readonly syncState: SyncState; + readonly contentHash: string | null; + readonly status: 'draft' | 'published'; + /** What the document already stores, so an RSS run (which knows no duration) never erases it. */ + readonly durationSeconds: number | null; +} + +export interface SyncPlan { + readonly creates: { + externalId: string; + source: SourceSnapshot; + suggestedTopicIds: DocId[]; + suggestedFormat: VideoFormat | null; + }[]; + readonly updates: { id: DocId; source: SourceSnapshot; restoreFromMissing: boolean }[]; + readonly unchanged: number; + readonly missing: DocId[]; + readonly skipped: { externalId: string; reason: SkipReason }[]; +} + +export const toSource = (v: YoutubeVideo): SourceSnapshot => ({ + title: v.title, + description: v.description, + durationSeconds: v.durationSeconds, + publishedAt: v.publishedAt, + tags: v.tags, + privacyStatus: v.privacyStatus, + embeddable: v.embeddable, + hasMaxres: v.hasMaxres, + thumbnailUrl: `https://i.ytimg.com/vi/${v.id}/${v.hasMaxres ? 'maxresdefault' : 'hqdefault'}.jpg`, +}); + +/** Unknown (null) never overwrites a duration we already have. */ +export const withKnownDuration = (source: SourceSnapshot, existing: number | null): SourceSnapshot => + source.durationSeconds === null && existing !== null ? { ...source, durationSeconds: existing } : source; + +export const contentHash = (source: SourceSnapshot): string => + createHash('sha256').update(JSON.stringify(source)).digest('hex'); + +const skipReason = (v: YoutubeVideo): SkipReason | null => { + if (v.privacyStatus !== 'public') return 'not-public'; + if (!v.embeddable) return 'not-embeddable'; + if (v.liveBroadcastContent !== 'none') return 'not-finished'; + return null; +}; + +export const planSync = ({ + existing, + fetched, + complete, + topics, + formatByVideoId = new Map(), +}: { + existing: readonly ExistingVideo[]; + fetched: readonly YoutubeVideo[]; + complete: boolean; + topics: readonly TopicRef[]; + formatByVideoId?: ReadonlyMap; +}): SyncPlan => { + const byExternalId = new Map(existing.map((e) => [e.externalId, e])); + const seen = new Set(); + const creates: SyncPlan['creates'] = []; + const updates: SyncPlan['updates'] = []; + const skipped: SyncPlan['skipped'] = []; + let unchanged = 0; + + for (const v of fetched) { + seen.add(v.id); + const reason = skipReason(v); + if (reason) { + skipped.push({ externalId: v.id, reason }); + continue; + } + const current = byExternalId.get(v.id); + const source = withKnownDuration(toSource(v), current?.durationSeconds ?? null); + if (!current) { + creates.push({ + externalId: v.id, + source, + suggestedTopicIds: suggestTopicIds({ tags: v.tags, description: v.description }, topics), + suggestedFormat: formatByVideoId.get(v.id) ?? null, + }); + } else if (current.syncState === 'manual') { + unchanged += 1; + } else if (current.contentHash === contentHash(source) && current.syncState !== 'missing') { + unchanged += 1; + } else { + updates.push({ id: current.id, source, restoreFromMissing: current.syncState === 'missing' }); + } + } + + const missing = complete + ? existing + .filter((e) => e.syncState !== 'manual' && e.syncState !== 'missing' && !seen.has(e.externalId)) + .map((e) => e.id) + : []; + + return { creates, updates, unchanged, missing, skipped }; +}; diff --git a/apps/cms/src/payload/lib/videos/playlist-id.test.ts b/apps/cms/src/payload/lib/videos/playlist-id.test.ts new file mode 100644 index 000000000..e94d404d2 --- /dev/null +++ b/apps/cms/src/payload/lib/videos/playlist-id.test.ts @@ -0,0 +1,30 @@ +import { describe, expect, it } from 'vitest'; +import { parsePlaylistId } from './playlist-id'; + +const ID = 'PLabcdEFGH1234567890_-xyz'; + +describe('parsePlaylistId', () => { + it.each([ + ['a bare id', ID], + ['a bare id with surrounding whitespace', ` ${ID}\n`], + ['a playlist URL', `https://www.youtube.com/playlist?list=${ID}`], + ['a watch URL inside a playlist', `https://www.youtube.com/watch?v=AAAAAAAAAAA&list=${ID}&index=2`], + ['a mobile URL', `https://m.youtube.com/playlist?list=${ID}`], + ['a bare-domain URL', `https://youtube.com/playlist?list=${ID}`], + ])('reads %s', (_label, raw) => { + expect(parsePlaylistId(raw)).toBe(ID); + }); + + it.each([ + ['empty', ''], + ['too short', 'PL123'], + ['spaces inside', 'PL abc def ghi jkl'], + ['a URL on another host', `https://evil.example/playlist?list=${ID}`], + ['a lookalike host', `https://youtube.com.evil.example/playlist?list=${ID}`], + ['a URL with no list', 'https://www.youtube.com/watch?v=AAAAAAAAAAA'], + ['a URL with a malformed list', 'https://www.youtube.com/playlist?list=a%20b'], + ['a non-http scheme', `javascript://www.youtube.com/playlist?list=${ID}`], + ])('rejects %s', (_label, raw) => { + expect(parsePlaylistId(raw)).toBeNull(); + }); +}); diff --git a/apps/cms/src/payload/lib/videos/playlist-id.ts b/apps/cms/src/payload/lib/videos/playlist-id.ts new file mode 100644 index 000000000..2e2e8f19f --- /dev/null +++ b/apps/cms/src/payload/lib/videos/playlist-id.ts @@ -0,0 +1,18 @@ +const BARE_ID = /^[A-Za-z0-9_-]{10,64}$/; +const YOUTUBE_HOSTS = new Set(['youtube.com', 'www.youtube.com', 'm.youtube.com', 'music.youtube.com']); + +/** Accepts a bare playlist id or a youtube.com URL carrying `?list=`. Anything else is null. */ +export const parsePlaylistId = (raw: string): string | null => { + const value = raw.trim(); + if (BARE_ID.test(value)) return value; + let url: URL; + try { + url = new URL(value); + } catch { + return null; + } + if (url.protocol !== 'https:' && url.protocol !== 'http:') return null; + if (!YOUTUBE_HOSTS.has(url.hostname)) return null; + const list = url.searchParams.get('list'); + return list !== null && BARE_ID.test(list) ? list : null; +}; diff --git a/apps/cms/src/payload/lib/videos/run-sync.test.ts b/apps/cms/src/payload/lib/videos/run-sync.test.ts new file mode 100644 index 000000000..0b3bec7ec --- /dev/null +++ b/apps/cms/src/payload/lib/videos/run-sync.test.ts @@ -0,0 +1,196 @@ +import { describe, expect, it, vi } from 'vitest'; +import { runYoutubeSync } from './run-sync'; +import { contentHash, toSource } from './plan-sync'; +import type { YoutubeClient, YoutubeVideo } from './youtube-client'; + +const video = (id: string, over: Partial = {}): YoutubeVideo => ({ + id, + title: `Title ${id}`, + description: 'First paragraph.\n\n#SBOM', + publishedAt: '2026-10-13T14:17:14Z', + durationSeconds: 101, + tags: [], + privacyStatus: 'public', + embeddable: true, + liveBroadcastContent: 'none', + hasMaxres: true, + ...over, +}); + +const fakeClient = ( + videos: YoutubeVideo[], + complete: boolean, + playlists: Record = {}, +): YoutubeClient => ({ + listUploadIds: vi.fn(async () => ({ ids: videos.map((v) => v.id), complete })), + getVideos: vi.fn(async () => videos), + listPlaylistVideoIds: vi.fn(async (playlistId: string) => { + const list = playlists[playlistId] ?? []; + if (list instanceof Error) throw list; + return list; + }), +}); + +const fakePayload = (existingDocs: Record[]) => ({ + find: vi.fn(async ({ collection }: { collection: string; page?: number }) => + collection === 'videoTopics' + ? { docs: [{ id: 1, name: 'SBOM', aliases: [{ value: 'software bill of materials' }] }], hasNextPage: false } + : { docs: existingDocs, hasNextPage: false }, + ), + create: vi.fn().mockResolvedValue({ id: 100 }), + update: vi.fn().mockResolvedValue({ id: 1 }), + logger: { info: vi.fn(), warn: vi.fn(), error: vi.fn() }, +}); + +describe('runYoutubeSync', () => { + it('lists the uploads before it asks for their details (the RSS client depends on it)', async () => { + const client = fakeClient([video('AAAAAAAAAAA')], false); + await runYoutubeSync({ payload: fakePayload([]), client, channelId: 'UCabc', autoPublish: false }); + const list = vi.mocked(client.listUploadIds).mock.invocationCallOrder[0] ?? 0; + const get = vi.mocked(client.getVideos).mock.invocationCallOrder[0] ?? 0; + expect(list).toBeGreaterThan(0); + expect(list).toBeLessThan(get); + expect(client.listUploadIds).toHaveBeenCalledWith('UCabc'); + }); + + it('creates new videos with topics suggested from the loaded taxonomy', async () => { + const payload = fakePayload([]); + const result = await runYoutubeSync({ + payload, + client: fakeClient([video('AAAAAAAAAAA')], true), + channelId: 'UCabc', + autoPublish: false, + }); + expect(result).toMatchObject({ created: 1, updated: 0, markedMissing: 0, failed: 0 }); + expect((payload.create.mock.calls[0]?.[0] as { data: { topics: number[] } }).data.topics).toEqual([1]); + }); + + it('leaves an unchanged video alone and reads the stored duration and hash from the document', async () => { + const v = video('AAAAAAAAAAA'); + const stored = { + id: 9, + externalId: 'AAAAAAAAAAA', + syncState: 'synced', + _status: 'published', + source: { contentHash: contentHash(toSource(v)), durationSeconds: 101 }, + }; + const payload = fakePayload([stored]); + const result = await runYoutubeSync({ + payload, + client: fakeClient([v], true), + channelId: 'UCabc', + autoPublish: false, + }); + expect(result).toMatchObject({ created: 0, updated: 0, unchanged: 1 }); + expect(payload.update).not.toHaveBeenCalled(); + }); + + it('flags a vanished video as missing only when the listing was complete (API mode)', async () => { + const stored = { id: 9, externalId: 'GONEGONEGON', syncState: 'synced', _status: 'published', source: {} }; + const api = await runYoutubeSync({ + payload: fakePayload([stored]), + client: fakeClient([], true), + channelId: 'UCabc', + autoPublish: false, + }); + expect(api.markedMissing).toBe(1); + const rss = await runYoutubeSync({ + payload: fakePayload([stored]), + client: fakeClient([], false), + channelId: 'UCabc', + autoPublish: false, + }); + expect(rss.markedMissing).toBe(0); + }); + + it('reads every page of existing videos', async () => { + const doc = (id: number, externalId: string) => ({ id, externalId, syncState: 'synced', _status: 'published', source: {} }); + const payload = fakePayload([]); + payload.find.mockImplementation(async ({ collection, page }: { collection: string; page?: number }) => { + if (collection === 'videoTopics') return { docs: [], hasNextPage: false }; + return page === 1 + ? { docs: [doc(1, 'AAAAAAAAAAA')], hasNextPage: true } + : { docs: [doc(2, 'BBBBBBBBBBB')], hasNextPage: false }; + }); + const result = await runYoutubeSync({ + payload, + client: fakeClient([], true), + channelId: 'UCabc', + autoPublish: false, + }); + expect(result.markedMissing).toBe(2); + }); + describe('category suggestions from playlists', () => { + const playlists = { explainer: 'PLexplainer0001', demo: 'PLdemo000000001' }; + const run = (client: YoutubeClient, payload = fakePayload([])) => + runYoutubeSync({ payload, client, channelId: 'UCabc', autoPublish: false, playlists }).then((result) => ({ + result, + payload, + })); + const created = (payload: ReturnType, n = 0): Record => + (payload.create.mock.calls[n]?.[0] as { data: Record }).data; + + it('suggests the category of the playlist a new video is in, and keeps it a needs-review draft', async () => { + const client = fakeClient([video('AAAAAAAAAAA'), video('BBBBBBBBBBB'), video('CCCCCCCCCCC')], true, { + PLexplainer0001: ['AAAAAAAAAAA'], + PLdemo000000001: ['BBBBBBBBBBB'], + }); + const { payload } = await run(client); + expect(created(payload, 0)).toMatchObject({ format: 'explainer', syncState: 'needs-review', _status: 'draft' }); + expect(created(payload, 1)).toMatchObject({ format: 'demo', syncState: 'needs-review', _status: 'draft' }); + expect(created(payload, 2)).not.toHaveProperty('format'); + }); + + it('gives no suggestion to a video that is in both playlists', async () => { + const client = fakeClient([video('AAAAAAAAAAA')], true, { + PLexplainer0001: ['AAAAAAAAAAA'], + PLdemo000000001: ['AAAAAAAAAAA'], + }); + const { payload } = await run(client); + expect(created(payload)).not.toHaveProperty('format'); + }); + + it('survives a failing playlist read: warns, keeps the other list, still creates the video', async () => { + const client = fakeClient([video('AAAAAAAAAAA'), video('BBBBBBBBBBB')], true, { + PLexplainer0001: new Error('quota'), + PLdemo000000001: ['BBBBBBBBBBB'], + }); + const { result, payload } = await run(client); + expect(result).toMatchObject({ created: 2, failed: 0 }); + expect(created(payload, 0)).not.toHaveProperty('format'); + expect(created(payload, 1)).toMatchObject({ format: 'demo' }); + expect(payload.logger.warn).toHaveBeenCalledTimes(1); + }); + + it('never reads a playlist that is not configured', async () => { + const client = fakeClient([video('AAAAAAAAAAA')], true); + await runYoutubeSync({ payload: fakePayload([]), client, channelId: 'UCabc', autoPublish: false }); + expect(client.listPlaylistVideoIds).not.toHaveBeenCalled(); + }); + + it('does not touch the category of a video that already exists', async () => { + const v = video('AAAAAAAAAAA', { description: 'edited upstream' }); + const stored = { id: 9, externalId: 'AAAAAAAAAAA', syncState: 'synced', _status: 'published', source: { contentHash: 'old' } }; + const payload = fakePayload([stored]); + const client = fakeClient([v], true, { PLexplainer0001: ['AAAAAAAAAAA'] }); + await run(client, payload); + expect(payload.update).toHaveBeenCalledTimes(1); + expect((payload.update.mock.calls[0]?.[0] as { data: Record }).data).not.toHaveProperty('format'); + }); + }); + + it('never erases a known duration when a later run (RSS) cannot report one', async () => { + const rssView = video('AAAAAAAAAAA', { durationSeconds: null, description: 'edited upstream' }); + const stored = { + id: 9, + externalId: 'AAAAAAAAAAA', + syncState: 'synced', + _status: 'published', + source: { contentHash: 'old', durationSeconds: 101 }, + }; + const payload = fakePayload([stored]); + await runYoutubeSync({ payload, client: fakeClient([rssView], false), channelId: 'UCabc', autoPublish: false }); + const data = (payload.update.mock.calls[0]?.[0] as { data: { source: { durationSeconds: number | null } } }).data; + expect(data.source.durationSeconds).toBe(101); + }); +}); diff --git a/apps/cms/src/payload/lib/videos/run-sync.ts b/apps/cms/src/payload/lib/videos/run-sync.ts new file mode 100644 index 000000000..b13cff468 --- /dev/null +++ b/apps/cms/src/payload/lib/videos/run-sync.ts @@ -0,0 +1,133 @@ +import { type ApplyResult, type SyncPayload, applyPlan } from './apply-sync'; +import { + type DocId, + type ExistingVideo, + type SyncState, + type TopicRef, + type VideoFormat, + planSync, +} from './plan-sync'; +import type { YoutubeClient } from './youtube-client'; + +export interface RunSyncPayload extends SyncPayload { + find(args: { + collection: 'videos' | 'videoTopics'; + limit: number; + page: number; + depth: number; + draft?: boolean; + overrideAccess: boolean; + }): Promise<{ docs: Record[]; hasNextPage?: boolean }>; +} + +export interface RunSyncResult extends ApplyResult { + unchanged: number; + skipped: number; + fetched: number; + /** False in RSS mode: the feed cannot prove a video is gone, so nothing is flagged missing. */ + complete: boolean; +} + +const PAGE_SIZE = 200; +const MAX_PAGES = 50; + +const readAll = async ( + payload: RunSyncPayload, + collection: 'videos' | 'videoTopics', + draft: boolean, +): Promise[]> => { + const out: Record[] = []; + for (let page = 1; page <= MAX_PAGES; page += 1) { + const res = await payload.find({ collection, limit: PAGE_SIZE, page, depth: 0, draft, overrideAccess: true }); + out.push(...res.docs); + if (!res.hasNextPage) return out; + } + return out; +}; + +const asNumber = (value: unknown): number | null => (typeof value === 'number' ? value : null); + +const toExisting = (doc: Record): ExistingVideo | null => { + const externalId = doc.externalId; + if (typeof externalId !== 'string' || externalId.length === 0) return null; + const source = (doc.source ?? {}) as { contentHash?: unknown; durationSeconds?: unknown }; + return { + id: doc.id as DocId, + externalId, + syncState: (doc.syncState as SyncState | undefined) ?? 'manual', + contentHash: typeof source.contentHash === 'string' ? source.contentHash : null, + status: doc._status === 'published' ? 'published' : 'draft', + durationSeconds: asNumber(source.durationSeconds), + }; +}; + +const toTopic = (doc: Record): TopicRef => ({ + id: doc.id as DocId, + name: String(doc.name ?? ''), + aliases: ((doc.aliases as { value?: string }[] | undefined) ?? []).flatMap((a) => (a.value ? [a.value] : [])), +}); + +export interface FormatPlaylists { + readonly explainer: string | null; + readonly demo: string | null; +} + +/** videoId to the category its playlist suggests. A video in both playlists is ambiguous, so it gets none. */ +const loadFormatSuggestions = async ( + client: YoutubeClient, + playlists: FormatPlaylists, + logger: SyncPayload['logger'], +): Promise> => { + const read = async (format: VideoFormat, playlistId: string | null): Promise => { + if (!playlistId) return []; + try { + return await client.listPlaylistVideoIds(playlistId); + } catch (error) { + logger.warn( + { format, error: error instanceof Error ? error.message : String(error) }, + 'videos sync: playlist read failed, no category suggestion from it', + ); + return []; + } + }; + const [explainer, demo] = await Promise.all([read('explainer', playlists.explainer), read('demo', playlists.demo)]); + const inDemo = new Set(demo); + const inExplainer = new Set(explainer); + const out = new Map(); + for (const id of explainer) if (!inDemo.has(id)) out.set(id, 'explainer'); + for (const id of demo) if (!inExplainer.has(id)) out.set(id, 'demo'); + return out; +}; + +export const runYoutubeSync = async ({ + payload, + client, + channelId, + autoPublish, + playlists = { explainer: null, demo: null }, +}: { + payload: RunSyncPayload; + client: YoutubeClient; + channelId: string; + autoPublish: boolean; + playlists?: FormatPlaylists; +}): Promise => { + // Order matters: the RSS client reads its whole feed in listUploadIds. + const { ids, complete } = await client.listUploadIds(channelId); + const fetched = await client.getVideos(ids); + const formatByVideoId = await loadFormatSuggestions(client, playlists, payload.logger); + + // draft: true reads the latest version, so a draft-only import is seen as existing. + const [existingDocs, topicDocs] = await Promise.all([ + readAll(payload, 'videos', true), + readAll(payload, 'videoTopics', false), + ]); + const existing = existingDocs.flatMap((d) => { + const e = toExisting(d); + return e ? [e] : []; + }); + + const plan = planSync({ existing, fetched, complete, topics: topicDocs.map(toTopic), formatByVideoId }); + const applied = await applyPlan(payload, plan, { autoPublish }); + return { ...applied, unchanged: plan.unchanged, skipped: plan.skipped.length, fetched: fetched.length, complete }; +}; diff --git a/apps/cms/src/payload/lib/videos/topic-match.ts b/apps/cms/src/payload/lib/videos/topic-match.ts new file mode 100644 index 000000000..3ed99acdb --- /dev/null +++ b/apps/cms/src/payload/lib/videos/topic-match.ts @@ -0,0 +1,26 @@ +// Pure and free of Node imports, so the admin (browser) and the sync (server) share one rule. + +export interface TopicRef { + readonly id: number | string; + readonly name: string; + readonly aliases: readonly string[]; +} + +export const normaliseTag = (raw: string): string => raw.toLowerCase().replace(/[^a-z0-9]/g, ''); + +// A hashtag must start a word, so "page#vex" in a URL fragment is not one. +const HASHTAG = /(?:^|\s)#([\p{L}\p{N}_]+)/gu; + +/** Topics whose name or alias matches a YouTube tag or a hashtag in the description. */ +export const suggestTopicIds = ( + input: { tags: readonly string[]; description: string }, + topics: readonly TopicRef[], +): (number | string)[] => { + const candidates = new Set([ + ...input.tags.map(normaliseTag), + ...[...input.description.matchAll(HASHTAG)].map((m) => normaliseTag(m[1] ?? '')), + ]); + return topics + .filter((t) => [t.name, ...t.aliases].some((label) => candidates.has(normaliseTag(label)))) + .map((t) => t.id); +}; diff --git a/apps/cms/src/payload/lib/videos/video-state.test.ts b/apps/cms/src/payload/lib/videos/video-state.test.ts new file mode 100644 index 000000000..ce883b80e --- /dev/null +++ b/apps/cms/src/payload/lib/videos/video-state.test.ts @@ -0,0 +1,46 @@ +import { describe, expect, it } from 'vitest'; + +import { formatSeconds, videoLength, videoState } from './video-state'; + +describe('videoState', () => { + it('calls a published video live', () => { + expect(videoState({ _status: 'published', syncState: 'synced' })).toEqual({ label: 'Live', tone: 'live' }); + }); + + it('marks a fresh import as needing review', () => { + expect(videoState({ _status: 'draft', syncState: 'needs-review' }).label).toBe('Needs review'); + }); + + it('calls any other draft a draft', () => { + expect(videoState({ _status: 'draft', syncState: 'manual' })).toEqual({ label: 'Draft', tone: 'draft' }); + }); + + it.each([ + [{ syncState: 'missing' }, 'Missing on YouTube'], + [{ source: { privacyStatus: 'private' } }, 'Private on YouTube'], + [{ source: { embeddable: false } }, "Can't be embedded"], + ])('flags %j as a problem even when the video is live', (extra, label) => { + expect(videoState({ _status: 'published', ...extra })).toEqual({ label, tone: 'problem' }); + }); + + it('treats an unlisted video as fine, because it still embeds', () => { + expect(videoState({ _status: 'published', source: { privacyStatus: 'unlisted', embeddable: true } }).tone).toBe('live'); + }); +}); + +describe('videoLength', () => { + it('formats the length YouTube reported', () => { + expect(formatSeconds(101)).toBe('1:41'); + expect(formatSeconds(3730)).toBe('1:02:10'); + expect(videoLength({ source: { durationSeconds: 138 } })).toBe('2:18'); + }); + + it('prefers the YouTube length over a typed one', () => { + expect(videoLength({ source: { durationSeconds: 101 }, durationLabel: '9:99' })).toBe('1:41'); + }); + + it('falls back to the typed length, then to nothing', () => { + expect(videoLength({ source: { durationSeconds: null }, durationLabel: ' 2:08 ' })).toBe('2:08'); + expect(videoLength({})).toBeNull(); + }); +}); diff --git a/apps/cms/src/payload/lib/videos/video-state.ts b/apps/cms/src/payload/lib/videos/video-state.ts new file mode 100644 index 000000000..188bbe330 --- /dev/null +++ b/apps/cms/src/payload/lib/videos/video-state.ts @@ -0,0 +1,46 @@ +export type VideoStateTone = 'live' | 'review' | 'draft' | 'problem'; + +export interface VideoStateRow { + _status?: string | null; + syncState?: string | null; + source?: { + privacyStatus?: string | null; + embeddable?: boolean | null; + durationSeconds?: number | null; + } | null; + durationLabel?: string | null; +} + +export interface VideoState { + label: string; + tone: VideoStateTone; +} + +/** + * What an editor needs to know about a video at a glance. A problem outranks everything else, + * because a live video that went private on YouTube shows a broken player on the site. + */ +export const videoState = (row: VideoStateRow): VideoState => { + if (row.syncState === 'missing') return { label: 'Missing on YouTube', tone: 'problem' }; + if (row.source?.privacyStatus === 'private') return { label: 'Private on YouTube', tone: 'problem' }; + if (row.source?.embeddable === false) return { label: "Can't be embedded", tone: 'problem' }; + if (row._status === 'published') return { label: 'Live', tone: 'live' }; + if (row.syncState === 'needs-review') return { label: 'Needs review', tone: 'review' }; + return { label: 'Draft', tone: 'draft' }; +}; + +/** 101 becomes "1:41", 3730 becomes "1:02:10". */ +export const formatSeconds = (total: number): string => { + const hours = Math.floor(total / 3600); + const minutes = Math.floor((total % 3600) / 60); + const seconds = Math.floor(total % 60); + const pad = (n: number): string => String(n).padStart(2, '0'); + return hours > 0 ? `${hours}:${pad(minutes)}:${pad(seconds)}` : `${minutes}:${pad(seconds)}`; +}; + +/** The length YouTube reported, or the one an editor typed when YouTube had none. */ +export const videoLength = (row: VideoStateRow): string | null => { + const seconds = row.source?.durationSeconds; + if (typeof seconds === 'number' && seconds > 0) return formatSeconds(seconds); + return row.durationLabel?.trim() || null; +}; diff --git a/apps/cms/src/payload/lib/videos/youtube-client.test.ts b/apps/cms/src/payload/lib/videos/youtube-client.test.ts new file mode 100644 index 000000000..74625a06a --- /dev/null +++ b/apps/cms/src/payload/lib/videos/youtube-client.test.ts @@ -0,0 +1,143 @@ +import { describe, expect, it, vi } from 'vitest'; +import { YoutubeApiError, createYoutubeClient, parseIsoDuration } from './youtube-client'; + +const json = (body: unknown, status = 200): Response => + ({ ok: status < 400, status, json: async () => body }) as Response; + +describe('parseIsoDuration', () => { + it.each([ + ['PT1M41S', 101], + ['PT59S', 59], + ['PT27M46S', 1666], + ['PT1H2M3S', 3723], + ['PT0S', 0], + ['P1DT1H', 90000], + ])('%s -> %i seconds', (iso, seconds) => { + expect(parseIsoDuration(iso)).toBe(seconds); + }); + + it('rejects garbage', () => { + expect(() => parseIsoDuration('1:41')).toThrow(YoutubeApiError); + }); +}); + +describe('createYoutubeClient', () => { + it('walks the uploads playlist and reports whether it saw everything', async () => { + const fetchImpl = vi + .fn() + .mockResolvedValueOnce(json({ items: [{ contentDetails: { relatedPlaylists: { uploads: 'UUabc' } } }] })) + .mockResolvedValueOnce(json({ items: [{ contentDetails: { videoId: 'AAAAAAAAAAA' } }], nextPageToken: 'p2' })) + .mockResolvedValueOnce(json({ items: [{ contentDetails: { videoId: 'BBBBBBBBBBB' } }] })); + const client = createYoutubeClient({ apiKey: 'k', fetchImpl }); + expect(await client.listUploadIds('UCabc')).toEqual({ ids: ['AAAAAAAAAAA', 'BBBBBBBBBBB'], complete: true }); + }); + + it('marks the listing incomplete when it hits the page cap', async () => { + const fetchImpl = vi + .fn() + .mockResolvedValueOnce(json({ items: [{ contentDetails: { relatedPlaylists: { uploads: 'UUabc' } } }] })) + .mockResolvedValue(json({ items: [{ contentDetails: { videoId: 'AAAAAAAAAAA' } }], nextPageToken: 'more' })); + const client = createYoutubeClient({ apiKey: 'k', fetchImpl }); + const result = await client.listUploadIds('UCabc', { maxPages: 2 }); + expect(result.complete).toBe(false); + }); + + it('lists a playlist page by page without resolving a channel first', async () => { + const fetchImpl = vi + .fn() + .mockResolvedValueOnce(json({ items: [{ contentDetails: { videoId: 'AAAAAAAAAAA' } }], nextPageToken: 'p2' })) + .mockResolvedValueOnce(json({ items: [{ contentDetails: { videoId: 'BBBBBBBBBBB' } }] })); + const client = createYoutubeClient({ apiKey: 'k', fetchImpl }); + expect(await client.listPlaylistVideoIds('PLexplainer0001')).toEqual(['AAAAAAAAAAA', 'BBBBBBBBBBB']); + const first = new URL(String(fetchImpl.mock.calls[0]?.[0])); + const second = new URL(String(fetchImpl.mock.calls[1]?.[0])); + expect(first.pathname).toBe('/youtube/v3/playlistItems'); + expect(Object.fromEntries(first.searchParams)).toMatchObject({ + part: 'contentDetails', + playlistId: 'PLexplainer0001', + maxResults: '50', + }); + expect(second.searchParams.get('pageToken')).toBe('p2'); + }); + + it('stops paging a playlist at the page cap instead of looping forever', async () => { + const fetchImpl = vi + .fn() + .mockResolvedValue(json({ items: [{ contentDetails: { videoId: 'AAAAAAAAAAA' } }], nextPageToken: 'more' })); + const ids = await createYoutubeClient({ apiKey: 'k', fetchImpl }).listPlaylistVideoIds('PLexplainer0001'); + expect(fetchImpl).toHaveBeenCalledTimes(20); + expect(ids).toHaveLength(20); + }); + + it('rejects a playlist response that does not match the expected shape', async () => { + const fetchImpl = vi.fn().mockResolvedValue(json({ items: [{ contentDetails: { videoId: 'short' } }] })); + await expect(createYoutubeClient({ apiKey: 'k', fetchImpl }).listPlaylistVideoIds('PLx0000000001')).rejects.toThrow( + /expected shape/, + ); + }); + + it('surfaces a missing playlist (404) as a typed error without the key', async () => { + const fetchImpl = vi.fn().mockResolvedValue(json({ error: {} }, 404)); + const error = await createYoutubeClient({ apiKey: 'secret-key', fetchImpl }) + .listPlaylistVideoIds('PLx0000000001') + .catch((e: unknown) => e); + expect(error).toBeInstanceOf(YoutubeApiError); + expect((error as YoutubeApiError).status).toBe(404); + expect((error as YoutubeApiError).message).not.toContain('secret-key'); + }); + + it('maps a video, reading maxres from the thumbnails', async () => { + const fetchImpl = vi.fn().mockResolvedValueOnce( + json({ + items: [ + { + id: 'AAAAAAAAAAA', + snippet: { + title: 'T', + description: 'D', + publishedAt: '2026-10-13T14:17:14Z', + tags: ['sbom'], + liveBroadcastContent: 'none', + thumbnails: { default: { url: 'u' }, maxres: { url: 'https://i.ytimg.com/vi/AAAAAAAAAAA/maxresdefault.jpg' } }, + }, + contentDetails: { duration: 'PT1M41S' }, + status: { privacyStatus: 'public', embeddable: true }, + }, + ], + }), + ); + const [video] = await createYoutubeClient({ apiKey: 'k', fetchImpl }).getVideos(['AAAAAAAAAAA']); + expect(video).toMatchObject({ id: 'AAAAAAAAAAA', durationSeconds: 101, hasMaxres: true, tags: ['sbom'] }); + }); + + it('throws a typed error that never contains the API key', async () => { + const fetchImpl = vi.fn().mockResolvedValue(json({ error: { message: 'quota' } }, 403)); + const client = createYoutubeClient({ apiKey: 'SECRET-KEY', fetchImpl }); + const error = await client.getVideos(['AAAAAAAAAAA']).catch((e: unknown) => e); + expect(error).toBeInstanceOf(YoutubeApiError); + expect((error as Error).message).not.toContain('SECRET-KEY'); + }); + + it('hides the key when the request itself fails', async () => { + const fetchImpl = vi.fn().mockRejectedValue(new Error('connect failed for https://x?key=SECRET-KEY')); + const error = await createYoutubeClient({ apiKey: 'SECRET-KEY', fetchImpl }) + .getVideos(['AAAAAAAAAAA']) + .catch((e: unknown) => e); + expect(error).toBeInstanceOf(YoutubeApiError); + expect((error as Error).message).not.toContain('SECRET-KEY'); + }); + + it('rejects a response that does not match the schema', async () => { + const fetchImpl = vi.fn().mockResolvedValue(json({ items: [{ id: 'short' }] })); + await expect(createYoutubeClient({ apiKey: 'k', fetchImpl }).getVideos(['AAAAAAAAAAA'])).rejects.toBeInstanceOf( + YoutubeApiError, + ); + }); + + it('batches video ids in groups of 50', async () => { + const fetchImpl = vi.fn().mockResolvedValue(json({ items: [] })); + const ids = Array.from({ length: 120 }, (_, i) => String(i).padStart(11, 'x')); + await createYoutubeClient({ apiKey: 'k', fetchImpl }).getVideos(ids); + expect(fetchImpl).toHaveBeenCalledTimes(3); + }); +}); diff --git a/apps/cms/src/payload/lib/videos/youtube-client.ts b/apps/cms/src/payload/lib/videos/youtube-client.ts new file mode 100644 index 000000000..4c639b3ec --- /dev/null +++ b/apps/cms/src/payload/lib/videos/youtube-client.ts @@ -0,0 +1,174 @@ +import { z } from 'zod'; + +export class YoutubeApiError extends Error { + constructor( + message: string, + readonly status: number | null = null, + ) { + super(message); + this.name = 'YoutubeApiError'; + } +} + +const ISO_DURATION = /^P(?:(\d+)D)?T?(?:(\d+)H)?(?:(\d+)M)?(?:(\d+)S)?$/; + +export const parseIsoDuration = (iso: string): number => { + const match = ISO_DURATION.exec(iso); + if (!match) throw new YoutubeApiError(`Unparseable duration: ${iso}`); + const [, d = '0', h = '0', m = '0', s = '0'] = match; + return Number(d) * 86_400 + Number(h) * 3_600 + Number(m) * 60 + Number(s); +}; + +const Id = z.string().length(11); + +const ChannelsResponse = z.object({ + items: z + .array(z.object({ contentDetails: z.object({ relatedPlaylists: z.object({ uploads: z.string() }) }) })) + .default([]), +}); + +const PlaylistItemsResponse = z.object({ + items: z.array(z.object({ contentDetails: z.object({ videoId: Id }) })).default([]), + nextPageToken: z.string().optional(), +}); + +const VideosResponse = z.object({ + items: z + .array( + z.object({ + id: Id, + snippet: z.object({ + title: z.string(), + description: z.string().default(''), + publishedAt: z.string(), + tags: z.array(z.string()).default([]), + liveBroadcastContent: z.enum(['none', 'live', 'upcoming']).default('none'), + thumbnails: z.record(z.object({ url: z.string() })).default({}), + }), + contentDetails: z.object({ duration: z.string() }), + status: z.object({ + privacyStatus: z.enum(['public', 'unlisted', 'private']), + embeddable: z.boolean().default(true), + }), + }), + ) + .default([]), +}); + +export interface YoutubeVideo { + readonly id: string; + readonly title: string; + readonly description: string; + readonly publishedAt: string; + /** `null` when the source cannot say (the RSS feed carries no duration). */ + readonly durationSeconds: number | null; + readonly tags: readonly string[]; + readonly privacyStatus: 'public' | 'unlisted' | 'private'; + readonly embeddable: boolean; + readonly liveBroadcastContent: 'none' | 'live' | 'upcoming'; + readonly hasMaxres: boolean; +} + +export interface YoutubeClient { + /** Must run before `getVideos`: the RSS client reads the whole feed here. */ + listUploadIds( + channelId: string, + options?: { maxPages?: number }, + ): Promise<{ ids: string[]; complete: boolean }>; + getVideos(ids: readonly string[]): Promise; + /** Ids of the videos in a playlist. Unlike `listUploadIds` it never primes state `getVideos` relies on. */ + listPlaylistVideoIds(playlistId: string): Promise; +} + +interface ClientOptions { + apiKey: string; + fetchImpl?: typeof fetch; + timeoutMs?: number; +} + +const BASE = 'https://www.googleapis.com/youtube/v3'; +const BATCH = 50; +const DEFAULT_MAX_PAGES = 20; + +export const createYoutubeClient = ({ + apiKey, + fetchImpl = fetch, + timeoutMs = 10_000, +}: ClientOptions): YoutubeClient => { + const get = async ( + path: string, + params: Record, + schema: z.ZodType, + ): Promise => { + const url = `${BASE}/${path}?${new URLSearchParams({ ...params, key: apiKey }).toString()}`; + let res: Response; + try { + res = await fetchImpl(url, { signal: AbortSignal.timeout(timeoutMs) }); + } catch { + // The URL carries the key, so never forward the underlying error text. + throw new YoutubeApiError(`YouTube request failed: ${path}`); + } + if (!res.ok) throw new YoutubeApiError(`YouTube ${path} returned ${res.status}`, res.status); + const parsed = schema.safeParse(await res.json()); + if (!parsed.success) throw new YoutubeApiError(`YouTube ${path} response did not match the expected shape`); + return parsed.data; + }; + + const walkPlaylist = async ( + playlistId: string, + maxPages: number, + ): Promise<{ ids: string[]; complete: boolean }> => { + const ids: string[] = []; + let pageToken: string | undefined; + for (let page = 0; page < maxPages; page += 1) { + const res = await get( + 'playlistItems', + { part: 'contentDetails', playlistId, maxResults: String(BATCH), ...(pageToken ? { pageToken } : {}) }, + PlaylistItemsResponse, + ); + ids.push(...res.items.map((i) => i.contentDetails.videoId)); + pageToken = res.nextPageToken; + if (!pageToken) return { ids, complete: true }; + } + return { ids, complete: false }; + }; + + return { + async listUploadIds(channelId, { maxPages = DEFAULT_MAX_PAGES } = {}) { + const channels = await get('channels', { part: 'contentDetails', id: channelId }, ChannelsResponse); + const uploads = channels.items[0]?.contentDetails.relatedPlaylists.uploads; + if (!uploads) throw new YoutubeApiError(`No uploads playlist for channel ${channelId}`); + return walkPlaylist(uploads, maxPages); + }, + + async listPlaylistVideoIds(playlistId) { + return (await walkPlaylist(playlistId, DEFAULT_MAX_PAGES)).ids; + }, + + async getVideos(ids) { + const out: YoutubeVideo[] = []; + for (let i = 0; i < ids.length; i += BATCH) { + const res = await get( + 'videos', + { part: 'snippet,contentDetails,status', id: ids.slice(i, i + BATCH).join(',') }, + VideosResponse, + ); + for (const v of res.items) { + out.push({ + id: v.id, + title: v.snippet.title, + description: v.snippet.description, + publishedAt: v.snippet.publishedAt, + durationSeconds: parseIsoDuration(v.contentDetails.duration), + tags: v.snippet.tags, + privacyStatus: v.status.privacyStatus, + embeddable: v.status.embeddable, + liveBroadcastContent: v.snippet.liveBroadcastContent, + hasMaxres: 'maxres' in v.snippet.thumbnails, + }); + } + } + return out; + }, + }; +}; diff --git a/apps/cms/src/payload/lib/videos/youtube-rss-client.test.ts b/apps/cms/src/payload/lib/videos/youtube-rss-client.test.ts new file mode 100644 index 000000000..13edcb680 --- /dev/null +++ b/apps/cms/src/payload/lib/videos/youtube-rss-client.test.ts @@ -0,0 +1,97 @@ +import { describe, expect, it, vi } from 'vitest'; +import { createYoutubeRssClient, parseRssFeed } from './youtube-rss-client'; + +const FEED = ` + + AAAAAAAAAAA + What Is VEX? & why + 2026-10-13T14:17:14+00:00 + First paragraph. + +#VEX #SBOM + + BBBBBBBBBBBSecond2026-10-12T10:58:15+00:00 +`; + +const respond = (url: string): Response => { + if (url.includes('feeds/videos.xml')) return { ok: true, status: 200, text: async () => FEED } as Response; + // HEAD for the maxres poster: present for A, missing for B. + const present = url.includes('AAAAAAAAAAA'); + return { ok: present, status: present ? 200 : 404 } as Response; +}; + +describe('parseRssFeed', () => { + it('reads id, decoded title, date and the multi-line description', () => { + const [first, second] = parseRssFeed(FEED); + expect(first).toMatchObject({ id: 'AAAAAAAAAAA', title: 'What Is VEX? & why', publishedAt: '2026-10-13T14:17:14+00:00' }); + expect(first?.description).toBe('First paragraph.\n\n#VEX #SBOM'); + expect(second?.description).toBe(''); + }); + + it('decodes ampersand-escaped entities once, not twice', () => { + const [entry] = parseRssFeed('AAAAAAAAAAAa &lt; b'); + expect(entry?.title).toBe('a < b'); + }); + + it('returns [] for empty or malformed input', () => { + expect(parseRssFeed('')).toEqual([]); + expect(parseRssFeed('')).toEqual([]); + }); +}); + +describe('createYoutubeRssClient', () => { + it('never claims a complete listing (the feed only holds the newest 15)', async () => { + const fetchImpl = vi.fn(async (input: RequestInfo | URL, _init?: RequestInit) => respond(String(input))); + const client = createYoutubeRssClient({ fetchImpl }); + expect((await client.listUploadIds('UCabc')).complete).toBe(false); + }); + + it('returns unknown duration as null, assumes public and embeddable, and checks maxres with HEAD', async () => { + const fetchImpl = vi.fn(async (input: RequestInfo | URL, _init?: RequestInit) => respond(String(input))); + const client = createYoutubeRssClient({ fetchImpl }); + const { ids } = await client.listUploadIds('UCabc'); + const videos = await client.getVideos(ids); + expect(videos).toHaveLength(2); + expect(videos[0]).toMatchObject({ + id: 'AAAAAAAAAAA', + durationSeconds: null, + privacyStatus: 'public', + embeddable: true, + hasMaxres: true, + tags: [], + }); + expect(videos[1]?.hasMaxres).toBe(false); + expect(fetchImpl).toHaveBeenCalledWith( + expect.stringContaining('maxresdefault.jpg'), + expect.objectContaining({ method: 'HEAD' }), + ); + }); + + it('treats a failed HEAD as no maxres instead of failing the run', async () => { + const fetchImpl = vi.fn(async (input: RequestInfo | URL, _init?: RequestInit) => { + if (String(input).includes('feeds/videos.xml')) return respond(String(input)); + throw new Error('network down'); + }); + const client = createYoutubeRssClient({ fetchImpl }); + const { ids } = await client.listUploadIds('UCabc'); + expect((await client.getVideos(ids)).every((v) => v.hasMaxres === false)).toBe(true); + }); + + it('reads a playlist feed by playlist_id and leaves the channel listing untouched', async () => { + const fetchImpl = vi.fn(async (input: RequestInfo | URL, _init?: RequestInit) => respond(String(input))); + const client = createYoutubeRssClient({ fetchImpl }); + expect(await client.listPlaylistVideoIds('PLexplainer0001')).toEqual(['AAAAAAAAAAA', 'BBBBBBBBBBB']); + expect(String(fetchImpl.mock.calls[0]?.[0])).toBe('https://www.youtube.com/feeds/videos.xml?playlist_id=PLexplainer0001'); + expect(await client.getVideos(['AAAAAAAAAAA'])).toEqual([]); + }); + + it('throws a typed error when the playlist feed is unreachable', async () => { + const fetchImpl = vi.fn(async () => ({ ok: false, status: 404, text: async () => '' }) as Response); + await expect(createYoutubeRssClient({ fetchImpl }).listPlaylistVideoIds('PLx0000000001')).rejects.toThrow(/feed/i); + }); + + it('throws a typed error when the feed itself is unreachable', async () => { + const fetchImpl = vi.fn(async () => ({ ok: false, status: 404, text: async () => '' }) as Response); + await expect(createYoutubeRssClient({ fetchImpl }).listUploadIds('UCabc')).rejects.toThrow(/feed/i); + }); +}); diff --git a/apps/cms/src/payload/lib/videos/youtube-rss-client.ts b/apps/cms/src/payload/lib/videos/youtube-rss-client.ts new file mode 100644 index 000000000..26401c941 --- /dev/null +++ b/apps/cms/src/payload/lib/videos/youtube-rss-client.ts @@ -0,0 +1,104 @@ +import { YoutubeApiError, type YoutubeClient, type YoutubeVideo } from './youtube-client'; + +// Deliberate duplicate of apps/web/src/lib/youtube-feed.ts: the CMS cannot import from apps/web. +const ENTRY = /([\s\S]*?)<\/entry>/g; +const ID = /([A-Za-z0-9_-]{11})<\/yt:videoId>/; +const TITLE = /([\s\S]*?)<\/title>/; +const PUBLISHED = /<published>([\s\S]*?)<\/published>/; +const DESCRIPTION = /<media:description>([\s\S]*?)<\/media:description>/; + +// & last, so "&lt;" decodes to "<" and not all the way to "<". +const decode = (value: string): string => + value + .replace(/</g, '<') + .replace(/>/g, '>') + .replace(/"/g, '"') + .replace(/'/g, "'") + .replace(/&#(\d+);/g, (_, code: string) => String.fromCodePoint(Number(code))) + .replace(/&/g, '&') + .trim(); + +export interface RssEntry { + readonly id: string; + readonly title: string; + readonly publishedAt: string; + readonly description: string; +} + +export const parseRssFeed = (xml: string): RssEntry[] => { + const out: RssEntry[] = []; + for (const match of xml.matchAll(ENTRY)) { + const block = match[1] ?? ''; + const id = ID.exec(block)?.[1]; + if (!id) continue; + out.push({ + id, + title: decode(TITLE.exec(block)?.[1] ?? ''), + publishedAt: PUBLISHED.exec(block)?.[1]?.trim() ?? '', + description: decode(DESCRIPTION.exec(block)?.[1] ?? ''), + }); + } + return out; +}; + +interface RssClientOptions { + fetchImpl?: typeof fetch; + timeoutMs?: number; +} + +const FEEDS = 'https://www.youtube.com/feeds/videos.xml'; + +export const createYoutubeRssClient = ({ + fetchImpl = fetch, + timeoutMs = 10_000, +}: RssClientOptions = {}): YoutubeClient => { + let entries: readonly RssEntry[] = []; + + const readFeed = async (query: string): Promise<RssEntry[]> => { + let res: Response; + try { + res = await fetchImpl(`${FEEDS}?${query}`, { signal: AbortSignal.timeout(timeoutMs) }); + } catch { + throw new YoutubeApiError('YouTube RSS feed request failed'); + } + if (!res.ok) throw new YoutubeApiError(`YouTube RSS feed returned ${res.status}`, res.status); + return parseRssFeed(await res.text()); + }; + + return { + async listUploadIds(channelId) { + entries = await readFeed(`channel_id=${encodeURIComponent(channelId)}`); + // The feed holds only the newest 15, so absence from it proves nothing. + return { ids: entries.map((e) => e.id), complete: false }; + }, + + async listPlaylistVideoIds(playlistId) { + return (await readFeed(`playlist_id=${encodeURIComponent(playlistId)}`)).map((e) => e.id); + }, + + async getVideos(ids) { + const wanted = new Set(ids); + const out: YoutubeVideo[] = []; + for (const entry of entries.filter((e) => wanted.has(e.id))) { + const poster = await fetchImpl(`https://i.ytimg.com/vi/${entry.id}/maxresdefault.jpg`, { + method: 'HEAD', + signal: AbortSignal.timeout(timeoutMs), + }).catch(() => null); + out.push({ + id: entry.id, + title: entry.title, + description: entry.description, + publishedAt: entry.publishedAt, + durationSeconds: null, + tags: [], + // The feed lists public videos only and cannot report embeddability; the draft review covers it. + privacyStatus: 'public', + embeddable: true, + liveBroadcastContent: 'none', + hasMaxres: poster?.ok === true, + }); + } + return out; + }, + }; +}; diff --git a/apps/cms/src/payload/lib/webhooks/dispatch.test.ts b/apps/cms/src/payload/lib/webhooks/dispatch.test.ts index 8a51b8ce9..354400c22 100644 --- a/apps/cms/src/payload/lib/webhooks/dispatch.test.ts +++ b/apps/cms/src/payload/lib/webhooks/dispatch.test.ts @@ -1,6 +1,6 @@ import { describe, expect, it, vi } from 'vitest'; -import { destinationsFromEnv, dispatchEvent } from './dispatch'; +import { type WebhookEventName, destinationsFromEnv, dispatchEvent } from './dispatch'; import { verifyWebhook } from './sign'; describe('destinationsFromEnv', () => { @@ -43,11 +43,11 @@ describe('destinationsFromEnv', () => { WEBHOOK_TEAMS_URL: 'https://x', WEBHOOK_TEAMS_EVENTS: 'bogus.event', } as unknown as NodeJS.ProcessEnv); - expect(dests[0]?.routing.events).toEqual(['document.published', 'lead.submitted']); + expect(dests[0]?.routing.events).toEqual(['document.published', 'lead.submitted', 'video.needs-review']); }); }); -const teamsDest = (id: string, events: ('document.published' | 'lead.submitted')[]) => ({ +const teamsDest = (id: string, events: WebhookEventName[]) => ({ id, source: 'env' as const, kind: 'teams' as const, @@ -95,6 +95,18 @@ describe('dispatchEvent', () => { expect(fetchMock).toHaveBeenCalledTimes(1); }); + it('posts a video.needs-review card to a Teams row subscribed to it, and skips one that is not', async () => { + const fetchMock = vi.fn(async (_url: string, _init?: RequestInit) => new Response(null, { status: 202 })); + const event = { event: 'video.needs-review' as const, data: { count: 2, titles: ['A', 'B'] } }; + await dispatchEvent(event, { + destinations: [teamsDest('other', ['document.published']), teamsDest('videos', ['video.needs-review'])], + fetch: fetchMock as unknown as typeof fetch, + }); + expect(fetchMock).toHaveBeenCalledTimes(1); + expect(fetchMock.mock.calls[0]?.[0]).toBe('https://videos'); + expect(String(fetchMock.mock.calls[0]?.[1]?.body)).toContain('2 new videos need review'); + }); + it('applies routing.formSlugs filter to lead.submitted events', async () => { const fetchMock = vi.fn(async () => new Response(null, { status: 202 })); const dest = { diff --git a/apps/cms/src/payload/lib/webhooks/dispatch.ts b/apps/cms/src/payload/lib/webhooks/dispatch.ts index 87b2a0e07..8ce669e35 100644 --- a/apps/cms/src/payload/lib/webhooks/dispatch.ts +++ b/apps/cms/src/payload/lib/webhooks/dispatch.ts @@ -13,7 +13,13 @@ import { buildTeamsPayload, postTeamsWebhook, type TeamsMention } from './teams' * change — keeping it locked here means TypeScript catches typos * at the call site. */ -export type WebhookEventName = 'document.published' | 'lead.submitted'; +export type WebhookEventName = 'document.published' | 'lead.submitted' | 'video.needs-review'; + +export const WEBHOOK_EVENT_NAMES: readonly WebhookEventName[] = [ + 'document.published', + 'lead.submitted', + 'video.needs-review', +]; export interface WebhookEvent { readonly event: WebhookEventName; @@ -43,15 +49,13 @@ interface GenericDestination extends WebhookDestination { type AnyDestination = TeamsDestination | GenericDestination; -const ALL_EVENTS: readonly WebhookEventName[] = ['document.published', 'lead.submitted']; - const parseEvents = (raw: string | undefined): readonly WebhookEventName[] => { - if (!raw) return ALL_EVENTS; + if (!raw) return WEBHOOK_EVENT_NAMES; const parsed = raw .split(',') .map((s) => s.trim()) - .filter((s): s is WebhookEventName => s === 'document.published' || s === 'lead.submitted'); - return parsed.length > 0 ? parsed : ALL_EVENTS; + .filter((s): s is WebhookEventName => (WEBHOOK_EVENT_NAMES as readonly string[]).includes(s)); + return parsed.length > 0 ? parsed : WEBHOOK_EVENT_NAMES; }; /** diff --git a/apps/cms/src/payload/lib/webhooks/teams.test.ts b/apps/cms/src/payload/lib/webhooks/teams.test.ts index 024bacd02..edb1937aa 100644 --- a/apps/cms/src/payload/lib/webhooks/teams.test.ts +++ b/apps/cms/src/payload/lib/webhooks/teams.test.ts @@ -209,6 +209,65 @@ describe('buildTeamsPayload', () => { }); }); +describe('buildTeamsPayload for video.needs-review', () => { + const card = (data: Record<string, unknown>) => + ( + JSON.parse(buildTeamsPayload({ event: 'video.needs-review', data })) as { + attachments: { + content: { + body: { type: string; text?: string; facts?: unknown[] }[]; + actions?: { type: string; title: string; url: string }[]; + }; + }[]; + } + ).attachments[0]?.content; + + const adminUrl = 'https://cms.cleanstart.com/admin/collections/videos?where[syncState][equals]=needs-review'; + + it('headlines the count, with the event eyebrow below', () => { + const body = card({ count: 3, titles: ['A', 'B', 'C'], adminUrl })?.body ?? []; + expect(body[0]?.text).toBe('3 new videos need review'); + expect(body[1]?.text).toBe('CleanStart · video.needs-review'); + }); + + it('uses the singular for one video', () => { + expect(card({ count: 1, titles: ['A'] })?.body[0]?.text).toBe('1 new video needs review'); + }); + + it('lists each title as a bullet and says how many more were not listed', () => { + const body = card({ count: 8, titles: ['A', 'B', 'C', 'D', 'E'], adminUrl })?.body ?? []; + expect(body.slice(2).map((b) => b.text)).toEqual(['• A', '• B', '• C', '• D', '• E', 'and 3 more']); + }); + + it('adds no "and N more" line when every title is listed', () => { + const body = card({ count: 2, titles: ['A', 'B'] })?.body ?? []; + expect(body.map((b) => b.text)).not.toContain('and 0 more'); + expect(body).toHaveLength(4); + }); + + it('renders no FactSet (count and titles are already on the card)', () => { + expect(card({ count: 2, titles: ['A', 'B'], adminUrl })?.body.some((b) => b.type === 'FactSet')).toBe(false); + }); + + it('opens the Needs review list from a button', () => { + expect(card({ count: 1, titles: ['A'], adminUrl })?.actions).toEqual([ + { type: 'Action.OpenUrl', title: 'Review in CMS', url: adminUrl }, + ]); + }); + + it('has no button when the CMS base URL is unknown, and ignores a non-http adminUrl', () => { + expect(card({ count: 1, titles: ['A'] })?.actions).toBeUndefined(); + expect(card({ count: 1, titles: ['A'], adminUrl: 'javascript:alert(1)' })?.actions).toBeUndefined(); + }); + + it('cuts a very long title and tolerates malformed data', () => { + const long = 'x'.repeat(500); + const body = card({ count: 1, titles: [long] })?.body ?? []; + expect((body[2]?.text ?? '').length).toBeLessThanOrEqual(122); + expect(card({ count: 'many', titles: 'nope' })?.body[0]?.text).toBe('0 new videos need review'); + }); +}); + describe('postTeamsWebhook', () => { it('POSTs the rendered card body to the URL', async () => { const fetchMock = vi.fn(async () => new Response(null, { status: 202 })); diff --git a/apps/cms/src/payload/lib/webhooks/teams.ts b/apps/cms/src/payload/lib/webhooks/teams.ts index fe1227290..cdb54be7e 100644 --- a/apps/cms/src/payload/lib/webhooks/teams.ts +++ b/apps/cms/src/payload/lib/webhooks/teams.ts @@ -132,7 +132,8 @@ interface AdaptiveOpenUrlAction { const isHttpUrl = (raw: unknown): raw is string => typeof raw === 'string' && /^https?:\/\//i.test(raw); -const actionsFromData = (data: Record<string, unknown>): AdaptiveOpenUrlAction[] => { +const actionsFromData = (event: WebhookEvent): AdaptiveOpenUrlAction[] => { + const data = event.data; const actions: AdaptiveOpenUrlAction[] = []; if (isHttpUrl(data.url)) { actions.push({ type: 'Action.OpenUrl', title: 'View live page', url: data.url }); @@ -142,6 +143,10 @@ const actionsFromData = (data: Record<string, unknown>): AdaptiveOpenUrlAction[] // everyone else to a login wall, plus it surfaces an internal URL into a // potentially wide channel audience. It stays in the event payload for // generic-webhook/automation subscribers that want it. + // The review list is the one exception: its whole audience is the editors who can open it. + if (event.event === 'video.needs-review' && isHttpUrl(data.adminUrl)) { + actions.push({ type: 'Action.OpenUrl', title: 'Review in CMS', url: data.adminUrl }); + } return actions; }; @@ -152,8 +157,29 @@ const actionsFromData = (data: Record<string, unknown>): AdaptiveOpenUrlAction[] * slug is intentionally NOT appended (it lives in the FactSet and the * "View live page" button) so the title reads cleanly at the top. */ +const videoReviewCount = (data: Record<string, unknown>): number => + typeof data.count === 'number' && Number.isFinite(data.count) ? Math.max(0, Math.trunc(data.count)) : 0; + +const videoReviewTitles = (data: Record<string, unknown>): string[] => + Array.isArray(data.titles) ? data.titles.filter((t): t is string => typeof t === 'string') : []; + +const TITLE_LINE_MAX = 120; + +// One subtle bullet per title, then "and N more" when the event carried fewer titles than videos. +const videoReviewBlocks = (data: Record<string, unknown>): object[] => { + const titles = videoReviewTitles(data); + const more = videoReviewCount(data) - titles.length; + const lines = titles.map((t) => `• ${truncateValue(t, TITLE_LINE_MAX)}`); + if (more > 0) lines.push(`and ${more} more`); + return lines.map((text, i) => ({ type: 'TextBlock', wrap: true, spacing: i === 0 ? 'Small' : 'None', text })); +}; + const titleFromEvent = (event: WebhookEvent): string => { const data = event.data as Record<string, unknown>; + if (event.event === 'video.needs-review') { + const count = videoReviewCount(data); + return count === 1 ? '1 new video needs review' : `${count} new videos need review`; + } const title = typeof data.title === 'string' && data.title.length > 0 ? data.title : null; if (title) return title; if (typeof data.email === 'string' && data.email.length > 0) return data.email; @@ -183,8 +209,8 @@ export const buildTeamsPayload = ( event: WebhookEvent, options: BuildTeamsPayloadOptions = {}, ): string => { - const facts = factsFromData(event.data); - const actions = actionsFromData(event.data); + const facts = event.event === 'video.needs-review' ? [] : factsFromData(event.data); + const actions = actionsFromData(event); const eligible = (options.mentions ?? []).filter((m) => isMentionEligible(m, event)); const entities: TeamsEntity[] = eligible.map((m) => ({ type: 'mention', @@ -231,6 +257,7 @@ export const buildTeamsPayload = ( ...(mentionLine ? [{ type: 'TextBlock', wrap: true, text: mentionLine }] : []), + ...(event.event === 'video.needs-review' ? videoReviewBlocks(event.data) : []), ...(facts.length > 0 ? [ { diff --git a/apps/cms/src/payload/lib/wire-publish-gate.ts b/apps/cms/src/payload/lib/wire-publish-gate.ts index 0e050e93d..f3f7281b7 100644 --- a/apps/cms/src/payload/lib/wire-publish-gate.ts +++ b/apps/cms/src/payload/lib/wire-publish-gate.ts @@ -17,6 +17,7 @@ const SLUG_GUARD_PATH = const GATED_SLUGS = new Set([ 'blogs', 'advisories', + 'videos', 'news', 'guides', 'resources', diff --git a/apps/cms/src/proxy.ts b/apps/cms/src/proxy.ts index da69aa892..0182aba0e 100644 --- a/apps/cms/src/proxy.ts +++ b/apps/cms/src/proxy.ts @@ -29,6 +29,7 @@ const CACHEABLE_COLLECTIONS = new Set([ 'authors', 'blogs', 'advisories', + 'videos', 'case-studies', 'events', 'guides', diff --git a/docs/operations/PRODUCTION-ROLLOUT.md b/docs/operations/PRODUCTION-ROLLOUT.md index 4bc5c33ba..0da6ea7fc 100644 --- a/docs/operations/PRODUCTION-ROLLOUT.md +++ b/docs/operations/PRODUCTION-ROLLOUT.md @@ -161,3 +161,23 @@ No migration. The advisory body is now one rich-text box (`body`, "Advisory cont ### 25. Dropped two Knowledge Hub pages (2026-10-28) Done on prod already; nothing to run. `knowledge-hub/no-shell-read-only` (a stub pointing at two other guides) and `knowledge-hub/rekor-publishing` were deleted with `scripts/drop-knowledge-hub-pages.ts`, which first wrote a 301 for each: to `/knowledge-hub/combined-security-model` and `/knowledge-hub/image-signing-sigstore`. The script purges the web cache, but a script that ends with `process.exit` kills the deferred search-index delete, so the two Meilisearch documents (`knowledgeBase_3`, `knowledgeBase_84`) had to be removed by hand. After any scripted delete, check the index. The sitemap needed its `sitemap:knowledgeBase` tag purged as well. + +### 26. Videos CMS cutover + +Until this runs, `/videos` reads `apps/web/src/lib/videos-catalog.ts` plus the YouTube RSS feed. The `videos` and `videoTopics` collections, the hourly `syncYoutubeVideos` task and the web CMS reader are already in the code; `VIDEOS_SOURCE` decides which one the site uses. + +0. **Create the YouTube API key** (the length of each video is read from YouTube with it, and videos that go private or are removed are hidden from the page). In Google Cloud: create or pick a project, enable "YouTube Data API v3", create an API key under Credentials and restrict it to that API. Add it as the GitHub Actions secret `YOUTUBE_API_KEY`. The key is free for this volume (one small request per hour). +1. **Deploy the CMS** (migration `20261031_100000_add_videos` runs on boot). Check `/api/health`, then that the `videos` collection opens in the admin. +2. **Backfill.** On the droplet, in the cms container: + ```bash + /app/node_modules/.bin/tsx scripts/seed-videos-from-catalog.ts # dry run, writes nothing + /app/node_modules/.bin/tsx scripts/seed-videos-from-catalog.ts --apply + ``` + Creates 11 topics and 14 published videos (explainers and product demos only) with the slugs from the code catalog. A slug conflict stops the run before any write. Re-running plans nothing. Publish webhooks are skipped and web revalidation is sent once at the end. +3. **Compare.** Open `/videos` with `VIDEOS_SOURCE=cms` on a preview deploy and check it matches production. There are no per-video pages: videos play inline. +4. **Flip.** Set `VIDEOS_SOURCE=cms` in Vercel production env and redeploy. Rollback is unsetting it and redeploying. +5. **Keep for a week**, then delete `videos-catalog.ts` and the feed merge from `apps/web`. + +**Suggested category and the Teams message.** Create two public YouTube playlists on the channel, one for explainers and one for product demos, and set the repository variables `YOUTUBE_EXPLAINER_PLAYLIST_ID` and `YOUTUBE_DEMO_PLAYLIST_ID` (a playlist id or its full `youtube.com/playlist?list=` URL), then redeploy the CMS. Adding a new upload to one of them before the next hourly run pre-fills its Category; a video in both playlists, or in neither, gets no suggestion. The value is written only when the draft is created and is never touched by later syncs. For the "new videos need review" Teams message, add a Teams integration row with the event `video.needs-review` (or leave its events empty for all events); one message goes out per sync run that imports at least one video, with a button to the Needs review list, which is built from `PAYLOAD_PUBLIC_SERVER_URL`. + +New uploads land as `needs-review` drafts. An editor chooses the category, at least one topic and a summary, then publishes (the publish gate blocks otherwise). The admin dashboard shows a "Videos to review" card that opens the filtered list. Without `YOUTUBE_API_KEY` the sync falls back to the YouTube RSS feed, which carries no length and cannot tell that a video was removed, so editors would type the length in the "Length" field; YouTube's feed endpoint has also been seen returning 404, which is another reason to set the key. diff --git a/docs/superpowers/plans/2026-10-23-videos-cms-sync.md b/docs/superpowers/plans/2026-10-23-videos-cms-sync.md new file mode 100644 index 000000000..9fb258cee --- /dev/null +++ b/docs/superpowers/plans/2026-10-23-videos-cms-sync.md @@ -0,0 +1,1954 @@ +# Videos CMS and YouTube Sync Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Marketing manages videos, formats and topics in the Payload admin, new YouTube uploads arrive on their own as reviewed drafts with description, thumbnail and suggested topics (and duration once a YouTube API key is added), and `/videos` reads from the CMS with zero URL changes for the 15 videos already live. + +**Architecture:** A new routed collection `videos` (drafts, scheduled publish) plus a lean taxonomy `videoTopics`. An hourly Payload task pulls the channel through the YouTube Data API, plans the changes as a pure function, and applies them: new videos become `needs-review` drafts, existing videos only get their sync-owned `source` snapshot refreshed. The web app reads the CMS through `fetchCMS` with cache tags and on-demand revalidation, behind a `VIDEOS_SOURCE` switch so cutover and rollback are one environment variable. + +**Tech Stack:** Payload 3.81 (Postgres, hand-written migrations), Next.js 16 ISR with cache tags, Zod 3 at the API boundary, Vitest, YouTube Data API v3 (API key, read only). + +**This is Plan 1 of 2.** Plan 2 (Vimeo or other uploaded video, transcripts, chapters, video sitemap) is outlined in section 12 and gets its own document, because it needs decisions and credentials that do not exist yet. + +--- + +## 0. Decisions + +**Made (2026-10-23)** + +| # | Decision | Outcome | Consequence for this plan | +|---|---|---|---| +| 1 | New YouTube uploads: auto-publish or review? | **Draft for review.** An env switch (`VIDEOS_AUTO_PUBLISH`) is kept for later, but it cannot bypass the publish gate (see Task 7 note). | The review step is also the safety net for the things the RSS feed cannot tell us (see D4). | +| 2 | How does the sync read YouTube? Needs an API key? | **RSS now, API key later.** No key is needed to build, ship or run the sync. | The sync client is pluggable (D4). With no key it reads the RSS feed and editors enter duration by hand. Adding `YOUTUBE_API_KEY` later switches on automatic duration, privacy and embeddable checks and "missing video" detection with no code change. | + +**Still open (not blocking PR 1 or PR 2)** + +| # | Decision | Recommendation | Why it matters | +|---|---|---|---| +| 3 | Which host for non-YouTube uploads (Plan 2)? | Vimeo if the client already pays for it, otherwise Cloudflare Stream | The `provider` field is designed now so Plan 2 is additive. | +| 4 | One channel or several? | One channel (`UC13c-XRmRlkWpp6WMG9SDjw`) | Several channels are a one-line change later. | +| 5 | Keep the "Videos" link in the Resources menu? | Yes, but merge it only after the CMS cutover | Its own commit on `feat/videos-page`. | +| 6 | Create the YouTube API key now or later? | Later, when someone has 10 minutes. It restricts to "YouTube Data API v3", read only, free within quota, stored as the GitHub secret `YOUTUBE_API_KEY`. | Only improves data quality (D4). | + +--- + +## 1. Where we are and why this changes + +`/videos` today is `apps/web/src/lib/videos-catalog.ts` (15 hand-written entries: slug, title, format, topics, duration, summary) merged with the YouTube RSS feed (`youtube-feed.ts`). + +What that costs: + +- Editors cannot change anything. Every category or topic edit is a code change and a deploy. +- The RSS feed returns only the newest 15 uploads and carries no duration, tags or format. A video with no catalog entry disappears once 15 newer ones exist. +- An unfiled upload gets its URL from its title, so renaming it on YouTube changes the URL. +- Vimeo or any other host has no home. + +What stays good and must survive: the page design, URL-state filters, the watch page, the pure helpers in `videos-utils.ts` (tested), and all 15 existing URLs. + +--- + +## 2. Target architecture + +``` + hourly (PAYLOAD_AUTO_RUN=true) +YouTube Data API ------------------------------> syncYoutubeVideos task + channels.list (1 unit, once) | + playlistItems.list (1 unit per 50) | planSync() pure, unit tested + videos.list (1 unit per 50) | applyPlan() per-item try/catch + v + Payload `videos` (draft, needs-review) + Payload `videoTopics` (editor managed) + | + editor classifies + publishes (publish gate enforces format, topic, summary) + | + afterChange hooks: search sync, IndexNow, revalidateWeb(tag "videos") + v +apps/web getVideos() --fetchCMS(tags:["videos"], revalidate 3600 backstop)--> /videos, /videos/[slug], sitemap +``` + +Principles: + +1. **One source of truth per field.** Sync owns the `source` group. Editors own everything else. Neither overwrites the other. +2. **The network is a boundary.** Every YouTube response goes through Zod before it touches a Payload write. +3. **Pure core, thin shell.** Planning the sync is a pure function over plain data, so the risky logic is tested without a database or the network. +4. **Never lose the last good page.** If the CMS is down during an ISR revalidation, the web app throws so Next keeps serving the previous static page. It must not cache an empty listing. +5. **Reversible rollout.** The web app can read the old catalog or the CMS by flipping one env var. + +--- + +## 3. Design decisions (with the alternatives that were rejected) + +| # | Decision | Rejected alternative | Reason | +|---|---|---|---| +| D1 | New Payload collection `videos` is the source of truth | Keep the code catalog and add a webhook | Editorial fields (format, topics, slug, featured) do not exist on YouTube, a review workflow needs draft state, and the CMS already carries ten routed collections with the same machinery. | +| D2 | `provider` select with `youtube` and `vimeo` now, `vimeo` rejected by a hook until Plan 2 | Add the field in Plan 2 | Adding an enum value later is a Postgres enum migration. One unused value costs nothing today. | +| D3 | `format` is a fixed `select` (explainer, demo, talk). Topics are a taxonomy collection. | Taxonomy for both | Formats are a page design decision (three tabs), not content. Topics are content and must be editable without a deploy. The repo is still unwinding enum+Ref pairs (see memory `taxonomy-enum-migration-status`), so no new pair is created: `format` has no twin. | +| D4 | **A pluggable YouTube source: RSS feed when there is no key, Data API v3 when `YOUTUBE_API_KEY` is set.** Both implement one `YoutubeClient` interface, so nothing downstream knows which ran. | API key as a hard requirement, or RSS only forever | RSS needs no setup and already carries title, date, thumbnail and the full description with its hashtags, which is what topic suggestion needs. It lacks duration, tags and a full channel history, and cannot tell us a video is non-embeddable. The API adds all of that for about 3 quota units per run. Keeping both means the project never waits on Google Cloud access, and the sync keeps working if a key is revoked or the quota is hit. | +| D4b | **What RSS mode gives up, and what covers it.** (a) Duration is unknown: editors type it (`durationLabel`), and cards simply show no badge until then. (b) Non-embeddable videos cannot be detected: the draft-review step catches this. (c) "Missing on YouTube" cannot be detected, because the feed only holds the newest 15: RSS mode never flags missing. (d) Maxres poster availability is checked with a HEAD request to `i.ytimg.com`, no key needed. | Guessing | Each gap has a named owner, and none can put a broken video in front of visitors because every import is a reviewed draft (decision 1). | +| D5 | New uploads become drafts with `syncState = needs-review` (decided) | Auto-publish | See decision 1. Also guards against unlisted or private videos going public by accident (only `privacyStatus = public` is ever imported) and is what makes RSS mode safe (D4b). | +| D6 | `source` group is read-only and sync-owned. `title` is copied from it once, at creation, then belongs to the editor. | Overwrite title on every sync | Editors fix capitalisation and wording. A sync that undoes their edits would be re-introducing the bug the catalog's `title` override solved. | +| D7 | Topics are suggested from YouTube `tags` and description hashtags by exact normalised match against `videoTopics.name` plus `aliases`. Format is never guessed. | Fuzzy or LLM classification | Deterministic and testable. A wrong auto-category is worse than a missing one, and the publish gate forces a human to confirm. | +| D8 | No `schemaAddons` field and no CMS JSON-LD branch | Copy Advisories fully | The CMS "Schema Manager" JSON-LD engine is not what production serves; apps/web composes its own (memory `schema-two-disconnected-systems`). Skipping it removes about 20 `blocks_*` tables from the migration. Verified in Task 2 Step 5. | +| D9 | Sync settings are env vars (`YOUTUBE_API_KEY`, `YOUTUBE_CHANNEL_ID`, `VIDEOS_AUTO_PUBLISH`) | A new Payload global | A global needs its own migration (memory `global-without-migration-crashes-admin`). Add one only if marketing asks to flip auto-publish themselves. | +| D10 | Cutover behind `VIDEOS_SOURCE=catalog|cms` | Big-bang switch | Gives an instant rollback and lets the CMS be populated and checked before any visitor sees it. | +| D11 | Existing slugs are preserved by a backfill, not regenerated | Let the CMS generate slugs | URLs are commitments. A changed slug is a lost ranking. | + +--- + +## 4. Data model + +### `videoTopics` (taxonomy, no versions, no SEO fields) + +| Field | Type | Notes | +|---|---|---| +| `name` | text, required | "Base images" | +| `slug` | slugField(source: name) | used in `?topic=` URLs, so equal to today's ids (`base-images`, `attack-surface`, ...) | +| `description` | textarea | optional | +| `aliases` | array of `{ value: text }` | matched against YouTube tags and hashtags, normalised | + +It has no URL of its own (topics are filters on `/videos`), so it skips `buildTaxonomyFields`' SEO and media fields and drafts. Fewer tables, simpler migration. + +### `videos` (routed collection, drafts, schedulePublish, maxPerDoc 25) + +| Field | Type | Owner | Notes | +|---|---|---|---| +| `title` | contentTitleField | editor | initialised from `source.title` | +| `slug` | slugField(source: title) | editor | permanent after publish (slug change redirect hook already exists) | +| `provider` | select youtube / vimeo, required, default youtube | system | see D2 | +| `externalId` | text, required, unique, indexed | system | YouTube video id | +| `format` | select explainer / demo / talk | editor | required to publish | +| `topics` | relationship to `videoTopics`, hasMany | editor, seeded by sync | at least one required to publish | +| `summary` | textarea, max 320 | editor | card blurb and meta description, required to publish. Pre-filled from the first paragraph of `source.description`. | +| `durationLabel` | text, optional, validated `m:ss` or `h:mm:ss` | editor | only needed in RSS mode (or to override). The web shows `source.durationSeconds` when the sync knows it, else this. Not required to publish; the publish gate warns, it does not block. | +| `featured` | checkbox, sidebar | editor | hero pick, newest when none | +| `transcript` | textarea | editor | stored now, rendered in Plan 2 | +| `source` | group, every field read-only | sync | `title`, `description`, `durationSeconds` (nullable), `publishedAt`, `tags[]`, `privacyStatus`, `embeddable`, `hasMaxres`, `thumbnailUrl`, `contentHash`, `fetchedAt` | +| `syncState` | select synced / needs-review / missing / manual | sync | `manual` = created by hand, never touched by sync | +| `publishedAt`, `displayPublishedAt`, `contentUpdatedAt` | shared fields | hooks | `publishedAt` is set from `source.publishedAt` so old videos keep their real date | +| `permalink`, SEO sidebar | shared | editor | `descriptionSource: 'summary'` | + +--- + +## 5. Touchpoint map: how to wire a new routed collection + +The Advisories commit `15fbe79f` touched about 45 files to add one collection. That commit is the checklist, and the repo gives you a way to find every spot mechanically: + +```bash +cd /Users/a12345/Desktop/AI/cleanstart/cleanstart-website +rg -l "'advisories'|\"advisories\"|advisories:" apps/cms/src apps/web/src +git show --stat 15fbe79f +``` + +For each hit, add `videos` the same way, with these deliberate exceptions: + +| Skip or change | Why | +|---|---| +| `lib/jsonld/dispatch.ts`, `endpoints/jsonld.ts`, `SchemaPreviewField.tsx`, `schemaAddonsField` | D8: production JSON-LD is built in apps/web. | +| `authors`, `reviewedBy`, `AuthorCredibilityField`, `user-offboard.ts` author logic | Videos have no authors. | +| `sitemap/image.ts` | No hero image field. Use the poster only if the sitemap helper accepts a URL field; otherwise skip. | +| `page-registry-seed.ts` | Change the existing `/videos` row from `static` to `cms-listing` (backingCollection `videos`), add `/videos/[slug]` as `cms-template`, and remove `/videos/[slug]` from `INTENTIONALLY_UNREGISTERED` in `page-registry-coverage.ts`. | +| `ROUTE_PREFIX`, `LISTING_PATH_OVERRIDE` in `route-prefixes.ts` | `videos: '/videos'`. `videoTopics` has no prefix, so add `videoTopics: '/videos'` to `LISTING_PATH_OVERRIDE` (same trick `resourceTypes` uses) so a topic edit purges the listing. | +| `detail-route-not-found-guard.ts` (web) | Add `{ prefix: "/videos/", collection: "videos", extraFilter: PUBLISHED_FILTER }`. This gives real 404s for unpublished or unknown slugs. | +| `publish-gate.ts`, `wire-publish-gate.ts` | Add `videos` to `GATED_SLUGS` and the video-specific blockers in Task 9. | +| Tests that count collections | `web-pages.test.ts` (PURGEABLE_COLLECTIONS 14 to 15), `schema-surface.test.ts`, `cache-purge-field.test.ts`, `Pages.snap.json`, new `Videos.snap.json`. | + +--- + +## 6. File map + +**Create (CMS)** + +- `apps/cms/src/payload/collections/Videos.ts` +- `apps/cms/src/payload/collections/VideoTopics.ts` +- `apps/cms/src/migrations/20261024_100000_add_videos.ts` +- `apps/cms/src/payload/lib/videos/youtube-client.ts` and `.test.ts` +- `apps/cms/src/payload/lib/videos/plan-sync.ts` and `.test.ts` +- `apps/cms/src/payload/lib/videos/apply-sync.ts` and `.test.ts` +- `apps/cms/src/payload/lib/videos/run-sync.ts` +- `apps/cms/src/payload/jobs/sync-youtube-videos.ts` +- `apps/cms/scripts/seed-videos-from-catalog.ts` +- `apps/cms/scripts/data/videos-catalog.json` + +**Modify (CMS):** every file the section 5 search finds, plus `payload.config.ts` (collections, task, cron), `lib/integrations/credentials.ts`, `.env.example`, `.github/workflows/deploy-cms.yml`, `payload-types.ts` (generated). + +**Create (web)** + +- `apps/web/src/lib/videos-cms.ts` and `.test.ts` (CMS read and mapping) + +**Modify (web):** `lib/videos.ts` (source switch), `lib/videos-utils.ts` (taxonomy as data), `lib/videos-catalog.ts` (becomes seed-only, deleted in PR 4), `app/sitemap.ts`, `app/preview/[collection]/[slug]/page.tsx`, `components/search/search-ui.tsx`, `lib/detail-route-not-found-guard.ts`, `VideosBrowser.tsx`, `VideoSelect` callers, `app/videos/**`. + +**Docs:** `CLAUDE.md` jobs table, `docs/web/WEB-PAGES.md`, `docs/operations/PRODUCTION-ROLLOUT.md`, `docs/operations/CMS-DEPLOYMENT.md`. + +--- + +## 7. Delivery plan + +| PR | Content | Visible to visitors? | Estimate | +|---|---|---|---| +| 1 | Collections, migration, all wiring, generated types, schema tests | No | 2.5 days (the migration is the long pole) | +| 2 | YouTube API and RSS clients, planSync, applySync, task, env and deploy plumbing, publish gate, optional Teams event | No (creates drafts only). Works from day one with no key. | 2.5 days | +| 3 | Backfill script, web CMS reader, source switch, sitemap, guard, preview, search, docs. Merge with `VIDEOS_SOURCE=catalog`. | No until the flip | 2.5 days | +| flip | Run backfill on prod, verify, set `VIDEOS_SOURCE=cms` in Vercel, redeploy | Yes | 0.5 day | +| 4 | Delete the catalog and the switch, remove dual-read code | No | 0.5 day, after one week of soak | + +Total about 8 engineer days plus the one-week soak. + +**Rollout order (prod)** + +1. Deploy PR 1 (CMS image, migration runs). Verify `/admin` loads and `/api/health` is 200 (a missing migration 500s the admin, see memory `global-without-migration-crashes-admin`). +2. Deploy PR 2 (no key needed). Run the task once from the admin jobs view and confirm the log says `mode: rss`. Expect the 15 existing videos to be created as drafts (the sync does not know about the catalog yet). +3. Deploy PR 3. Run the backfill in dry-run, read the report, run `--apply`. It merges onto the synced drafts by `externalId`, sets slugs and editorial fields, publishes. Web still reads the catalog. +4. Compare: `curl` both sources, assert identical slugs, titles, topics and order (Task 12 provides the script). +5. Flip `VIDEOS_SOURCE=cms`, redeploy web, run the verification checklist in Task 14. + +**Rollback:** set `VIDEOS_SOURCE=catalog` and redeploy web (about 2 minutes). The CMS data stays. If the migration itself must be reverted, restore from the droplet backup (`docs/operations/RESTORE-LOG.md`); the migration is additive only. + +--- + +## 8. Tasks + +### Task 1: `videoTopics` taxonomy + +**Files:** +- Create: `apps/cms/src/payload/collections/VideoTopics.ts` +- Modify: `apps/cms/src/payload.config.ts`, `apps/cms/src/payload/lib/route-prefixes.ts` + +- [ ] **Step 1: Create the collection** + +```ts +import type { CollectionConfig } from 'payload'; + +import { isAdminOrEditor } from '../access'; +import { slugField } from '../fields/slug'; +import { + revalidateWebAfterDeleteHook, + revalidateWebPublishAfterChangeHook, +} from '../hooks/revalidate-web-publish'; + +/** + * Topics for /videos. Topics are filters on the listing, not pages, so this + * taxonomy has no SEO fields, no drafts and no URL of its own. `aliases` are the + * YouTube tags and hashtags that should suggest the topic on import. + */ +export const VideoTopics: CollectionConfig = { + slug: 'videoTopics', + labels: { singular: 'Video topic', plural: 'Video topics' }, + admin: { + useAsTitle: 'name', + defaultColumns: ['name', 'slug', 'updatedAt'], + group: 'Taxonomies', + }, + access: { + read: () => true, + create: isAdminOrEditor, + update: isAdminOrEditor, + delete: isAdminOrEditor, + }, + fields: [ + { name: 'name', type: 'text', required: true }, + slugField({ source: 'name' }), + { name: 'description', type: 'textarea' }, + { + name: 'aliases', + type: 'array', + labels: { singular: 'Alias', plural: 'Aliases' }, + admin: { + description: + 'YouTube tags and hashtags that suggest this topic on import, e.g. "sbom" or "software bill of materials". Case and spacing are ignored.', + }, + fields: [{ name: 'value', type: 'text', required: true }], + }, + ], + hooks: { + afterChange: [revalidateWebPublishAfterChangeHook('videoTopics')], + afterDelete: [revalidateWebAfterDeleteHook('videoTopics')], + }, + timestamps: true, +}; +``` + +- [ ] **Step 2: Register it** + +In `payload.config.ts` add `import { VideoTopics } from './payload/collections/VideoTopics';` and `VideoTopics,` in the `collections` array, next to `Advisories`. + +In `route-prefixes.ts`, inside `LISTING_PATH_OVERRIDE`, add: + +```ts + // Topics render as the /videos filter, so a topic edit must purge that listing. + videoTopics: '/videos', +``` + +- [ ] **Step 3: Commit** + +```bash +git add apps/cms/src/payload/collections/VideoTopics.ts apps/cms/src/payload.config.ts apps/cms/src/payload/lib/route-prefixes.ts +git commit -m "feat(cms): videoTopics taxonomy for /videos" +``` + +### Task 2: `videos` collection + +**Files:** +- Create: `apps/cms/src/payload/collections/Videos.ts` +- Modify: `apps/cms/src/payload.config.ts`, `apps/cms/src/payload/lib/route-prefixes.ts` + +- [ ] **Step 1: Add the route prefix** + +In `route-prefixes.ts`, add `videos: '/videos',` to `ROUTE_PREFIX`. No `LISTING_PATH_OVERRIDE` entry is needed because the listing and detail prefix are the same. + +- [ ] **Step 2: Create the collection** + +```ts +import type { CollectionBeforeChangeHook, CollectionConfig } from 'payload'; +import { ValidationError } from 'payload'; + +import { isAdminOrEditor, publishedOrAuthenticated } from '../access'; +import { docStatusBarEditConfig } from '../admin/doc-status-bar-mount'; +import { contentUpdatedAtField } from '../fields/content-updated-at'; +import { displayPublishedAtField } from '../fields/display-published-at'; +import { publishedAtField } from '../fields/published-at'; +import { seoFieldsForSidebar, seoSidebarFields } from '../fields/seo'; +import { slugField } from '../fields/slug'; +import { contentTitleField } from '../fields/title'; +import { contentUpdatedAtHook } from '../hooks/content-updated-at'; +import { displayPublishedAtAuditHook } from '../hooks/display-published-at-audit'; +import { displayPublishedAtBackfillHook } from '../hooks/display-published-at-backfill'; +import { firstPublishHook } from '../hooks/first-publish'; +import { indexNowPublishAfterChangeHook } from '../hooks/indexnow-publish'; +import { + revalidateWebAfterDeleteHook, + revalidateWebPublishAfterChangeHook, +} from '../hooks/revalidate-web-publish'; +import { searchSyncAfterChangeHook, searchSyncAfterDeleteHook } from '../hooks/search-sync'; +import { slugChangeRedirectHook } from '../hooks/slug-change-redirect'; +import { webhooksPublishAfterChangeHook } from '../hooks/webhooks-publish'; +import { ROUTE_PREFIX } from '../lib/route-prefixes'; + +/** Plan 2 adds 'vimeo'. The enum value exists now so that is not an enum migration. */ +export const SUPPORTED_PROVIDERS: readonly string[] = ['youtube']; + +const SUMMARY_MAX = 320; + +const readOnly = { update: () => false } as const; + +const enforceSupportedProviderHook: CollectionBeforeChangeHook = ({ data }) => { + const provider = (data as { provider?: string }).provider; + if (provider && !SUPPORTED_PROVIDERS.includes(provider)) { + throw new ValidationError({ + errors: [{ path: 'provider', message: `Provider "${provider}" is not enabled yet.` }], + }); + } + return data; +}; + +/** Old videos keep their real date: publishedAt follows the upload date until an editor sets one. */ +const publishedAtFromSourceHook: CollectionBeforeChangeHook = ({ data, operation }) => { + const source = (data as { source?: { publishedAt?: string } }).source; + const current = (data as { publishedAt?: string | null }).publishedAt; + if (operation === 'create' && !current && source?.publishedAt) { + return { ...data, publishedAt: source.publishedAt }; + } + return data; +}; + +export const Videos: CollectionConfig = { + slug: 'videos', + labels: { singular: 'Video', plural: 'Videos' }, + admin: { + useAsTitle: 'title', + defaultColumns: ['title', 'format', 'syncState', '_status', 'publishedAt'], + group: 'Content', + components: { + edit: docStatusBarEditConfig({ showStats: false, showPublishedAt: true, showPurge: true }), + }, + }, + access: { + read: publishedOrAuthenticated, + create: isAdminOrEditor, + update: isAdminOrEditor, + delete: isAdminOrEditor, + }, + fields: [ + contentTitleField, + slugField({ source: 'title' }), + { + name: 'provider', + type: 'select', + required: true, + defaultValue: 'youtube', + options: [ + { label: 'YouTube', value: 'youtube' }, + { label: 'Vimeo', value: 'vimeo' }, + ], + access: readOnly, + admin: { position: 'sidebar' }, + }, + { + name: 'externalId', + type: 'text', + required: true, + unique: true, + index: true, + access: readOnly, + admin: { position: 'sidebar', description: 'The video id on the host (the 11 characters in a YouTube link).' }, + }, + { + name: 'format', + type: 'select', + options: [ + { label: 'Explainer', value: 'explainer' }, + { label: 'Product demo', value: 'demo' }, + { label: 'Talk or webinar', value: 'talk' }, + ], + admin: { position: 'sidebar', description: 'Required to publish. Drives the format tabs on /videos.' }, + }, + { + name: 'topics', + type: 'relationship', + relationTo: 'videoTopics', + hasMany: true, + admin: { description: 'At least one is required to publish. Suggested from YouTube tags on import.' }, + }, + { + name: 'summary', + type: 'textarea', + maxLength: SUMMARY_MAX, + admin: { description: `Card text and meta description. Required to publish, up to ${SUMMARY_MAX} characters.` }, + }, + { + name: 'featured', + type: 'checkbox', + defaultValue: false, + admin: { position: 'sidebar', description: 'Plays in the /videos hero. When none is featured, the newest video is.' }, + }, + { + name: 'durationLabel', + type: 'text', + validate: (value: unknown) => + value == null || value === '' || (typeof value === 'string' && /^\d{1,2}(:\d{2}){1,2}$/.test(value)) + ? true + : 'Use m:ss or h:mm:ss, for example 1:41 or 1:02:10.', + admin: { + position: 'sidebar', + description: + 'Length shown on cards. Filled automatically when the YouTube API key is set; type it by hand otherwise (1:41).', + }, + }, + { name: 'transcript', type: 'textarea', admin: { description: 'Optional. Shown on the watch page once Plan 2 ships.' } }, + { + name: 'syncState', + type: 'select', + defaultValue: 'manual', + options: [ + { label: 'Synced', value: 'synced' }, + { label: 'Needs review', value: 'needs-review' }, + { label: 'Missing on YouTube', value: 'missing' }, + { label: 'Manual', value: 'manual' }, + ], + access: readOnly, + admin: { position: 'sidebar', description: 'Set by the YouTube sync. "Missing" means the video is gone or private on YouTube; it is not unpublished automatically.' }, + }, + { + name: 'source', + type: 'group', + admin: { description: 'Copied from the host by the sync. Read only.' }, + fields: [ + { name: 'title', type: 'text', access: readOnly }, + { name: 'description', type: 'textarea', access: readOnly }, + { name: 'durationSeconds', type: 'number', access: readOnly }, + { name: 'publishedAt', type: 'date', access: readOnly }, + { name: 'tags', type: 'array', access: readOnly, fields: [{ name: 'value', type: 'text' }] }, + { name: 'privacyStatus', type: 'text', access: readOnly }, + { name: 'embeddable', type: 'checkbox', access: readOnly }, + { name: 'hasMaxres', type: 'checkbox', access: readOnly }, + { name: 'thumbnailUrl', type: 'text', access: readOnly }, + { name: 'contentHash', type: 'text', access: readOnly, admin: { hidden: true } }, + { name: 'fetchedAt', type: 'date', access: readOnly }, + ], + }, + { + name: 'permalink', + type: 'ui', + admin: { + position: 'sidebar', + components: { + Field: { + path: '@/payload/admin/components/PermalinkField.tsx#PermalinkField', + clientProps: { pathPrefix: ROUTE_PREFIX.videos }, + }, + }, + }, + }, + publishedAtField, + displayPublishedAtField, + contentUpdatedAtField, + ...seoSidebarFields({ pathPrefix: ROUTE_PREFIX.videos, descriptionSource: 'summary' }), + ...seoFieldsForSidebar('videos'), + ], + hooks: { + beforeChange: [ + enforceSupportedProviderHook, + publishedAtFromSourceHook, + contentUpdatedAtHook(), + firstPublishHook(), + displayPublishedAtBackfillHook, + ], + afterChange: [ + slugChangeRedirectHook('videos'), + displayPublishedAtAuditHook('videos'), + searchSyncAfterChangeHook('videos'), + webhooksPublishAfterChangeHook('videos'), + indexNowPublishAfterChangeHook('videos'), + revalidateWebPublishAfterChangeHook('videos'), + ], + afterDelete: [searchSyncAfterDeleteHook('videos'), revalidateWebAfterDeleteHook('videos')], + }, + versions: { drafts: { schedulePublish: true }, maxPerDoc: 25 }, + timestamps: true, +}; +``` + +- [ ] **Step 3: Register it** next to `Advisories` in `payload.config.ts`. + +- [ ] **Step 4: Typecheck** + +Run: `pnpm --filter @cleanstart/cms typecheck` +Expected: errors only about `'videos'` / `'videoTopics'` missing from generated union types (fixed in Task 3). + +- [ ] **Step 5: Check D8** + +Run: `rg -n "schemaAddons|validateOverrideForCollection" apps/cms/src/payload/hooks apps/cms/src/payload/lib/jsonld/override-validator.ts | head` +Expected: no code path requires a `schemaAddons` field on every collection. If one does, add `schemaAddonsField` to `Videos.ts` and budget the extra `blocks_*` tables in Task 4. + +- [ ] **Step 6: Commit** + +```bash +git add apps/cms/src/payload/collections/Videos.ts apps/cms/src/payload.config.ts apps/cms/src/payload/lib/route-prefixes.ts +git commit -m "feat(cms): videos collection" +``` + +### Task 3: Wire `videos` into every registry, regenerate types + +**Files:** every hit from the section 5 search. + +- [ ] **Step 1: List the places** + +Run: `rg -l "'advisories'|\"advisories\"|advisories:" apps/cms/src apps/web/src > /tmp/advisory-touchpoints.txt && cat /tmp/advisory-touchpoints.txt` + +- [ ] **Step 2: Add `videos` at each hit**, applying the exceptions table in section 5. Examples of the exact edits: + +```ts +// apps/cms/src/payload/lib/sitemap/collect.ts +{ collection: 'videos', defaultPriority: 0.6, defaultChangeFreq: 'weekly', listingPath: '/videos' }, + +// apps/cms/src/payload/lib/web-pages.ts +videos: { detailPrefix: '/videos', listingPath: '/videos' }, + +// apps/cms/src/payload/lib/search/index-schema.ts (weight lower than advisories: 95) +videos: 70, + +// apps/cms/src/payload/admin/components/internal-routes.ts +videos: '/videos', + +// apps/cms/src/payload/admin/components/views/list/filters/list-filter-config.ts +videos: [ + { kind: 'select', name: 'format', label: 'Format' }, + { kind: 'select', name: 'syncState', label: 'Sync state' }, + STATUS_FILTER, +], +``` + +Open each neighbouring entry first and copy its exact shape. If `kind: 'select'` is not a supported filter kind in `list-filter-config.ts`, use the kind its other select filters use. + +- [ ] **Step 3: Page registry** + +In `page-registry-seed.ts` replace the `/videos` row and add the template row: + +```ts + { path: '/videos', title: 'Videos', kind: 'cms-listing', order: 61, backingCollection: 'videos' }, + { path: '/videos/[slug]', title: 'Video (template)', kind: 'cms-template', order: 62, backingCollection: 'videos' }, +``` + +In `page-registry-coverage.ts` remove the `'/videos/[slug]'` entry and its comment from `INTENTIONALLY_UNREGISTERED`. + +- [ ] **Step 4: Update the count tests** + +`apps/cms/src/payload/lib/web-pages.test.ts`: add `'videos'` to the expected list and change `toHaveLength(14)` to `15`. + +- [ ] **Step 5: Regenerate types and snapshots** + +```bash +pnpm --filter @cleanstart/cms generate:types +pnpm --filter @cleanstart/cms test -- schema-surface cache-purge-field web-pages +``` +Expected: schema-surface fails once with a new snapshot to approve. Run with `-u` only for `Videos.snap.json` and `Pages.snap.json`, then read the diff and confirm it contains only `videos` additions. + +- [ ] **Step 6: Gate and commit** + +```bash +pnpm --filter @cleanstart/cms lint && pnpm --filter @cleanstart/cms typecheck && pnpm --filter @cleanstart/cms verify:page-registry +git add -A apps/cms apps/web/src +git commit -m "feat(cms): wire videos into the routed-collection registries" +``` +Review `git status` before `git add -A`: stage only the files in the touchpoint list (the repo forbids `git add -A` at the root, and sibling sessions may have unrelated edits). + +### Task 4: Migration (the long pole) + +**Files:** +- Create: `apps/cms/src/migrations/20261024_100000_add_videos.ts` +- Modify: `apps/cms/src/migrations/index.ts` + +Do **not** trust `payload migrate:create` (CLAUDE.md, mandatory check 6): it diffs against a stale June snapshot and emits hundreds of unrelated statements. Use it only to read the SQL for the new tables. + +- [ ] **Step 1: Build two scratch databases from a dump of local dev** + +```bash +pg_dump -Fc cleanstart > /tmp/cs-base.dump +createdb cs_vid_a && pg_restore -d cs_vid_a /tmp/cs-base.dump +createdb cs_vid_b && pg_restore -d cs_vid_b /tmp/cs-base.dump +``` + +- [ ] **Step 2: Let Payload push the true schema onto clone A** + +Local dev runs in push mode. Start the CMS once against `DATABASE_URI=postgres://postgres:1234@localhost:5432/cs_vid_a`, wait for it to boot, stop it. Then: + +```bash +pg_dump --schema-only cs_vid_a > /tmp/schema-a.sql +pg_dump --schema-only cs_vid_b > /tmp/schema-base.sql +diff /tmp/schema-base.sql /tmp/schema-a.sql > /tmp/videos-ddl.diff +``` +`/tmp/videos-ddl.diff` is the truth to transcribe. + +- [ ] **Step 3: Write the migration, guarded and idempotent** + +Follow `20261023_120000_add_advisories.ts` exactly in structure. Every statement must be re-runnable because local databases already have these objects: + +```ts +import { type MigrateDownArgs, type MigrateUpArgs, sql } from '@payloadcms/db-postgres' + +const UP = [ + sql`DO $$ BEGIN CREATE TYPE "public"."enum_videos_provider" AS ENUM('youtube', 'vimeo'); EXCEPTION WHEN duplicate_object THEN null; END $$;`, + sql`DO $$ BEGIN CREATE TYPE "public"."enum_videos_format" AS ENUM('explainer', 'demo', 'talk'); EXCEPTION WHEN duplicate_object THEN null; END $$;`, + sql`DO $$ BEGIN CREATE TYPE "public"."enum_videos_sync_state" AS ENUM('synced', 'needs-review', 'missing', 'manual'); EXCEPTION WHEN duplicate_object THEN null; END $$;`, + // ... CREATE TABLE IF NOT EXISTS for: video_topics, video_topics_aliases, + // videos, videos_topics (rel), videos_source_tags, _videos_v, _videos_v_version_source_tags, + // _videos_v_rels, plus indexes, copied from /tmp/videos-ddl.diff. + // The three non-obvious touches, copied from the advisories migration: + sql`ALTER TABLE "payload_locked_documents_rels" ADD COLUMN IF NOT EXISTS "videos_id" integer;`, + sql`ALTER TABLE "payload_locked_documents_rels" ADD COLUMN IF NOT EXISTS "video_topics_id" integer;`, + sql`ALTER TABLE "pages_rels" ADD COLUMN IF NOT EXISTS "videos_id" integer;`, + sql`ALTER TABLE "_pages_v_rels" ADD COLUMN IF NOT EXISTS "videos_id" integer;`, + sql`ALTER TYPE "public"."enum_integrations_routing_collections" ADD VALUE IF NOT EXISTS 'videos';`, +] + +export async function up({ db }: MigrateUpArgs): Promise<void> { + for (const statement of UP) await db.execute(statement) +} + +export async function down(_args: MigrateDownArgs): Promise<void> { + // Additive only. Restore from the droplet backup if this must be reverted. +} +``` +The `payload_locked_documents_rels` columns are the ones that took the whole admin down in the 2026-06-24 taxonomy incident when omitted. Do not skip them. + +- [ ] **Step 4: Register it** in `apps/cms/src/migrations/index.ts` (import plus array entry, same shape as the advisories lines). + +- [ ] **Step 5: Prove it against the real thing** + +```bash +DATABASE_URI=postgres://postgres:1234@localhost:5432/cs_vid_b pnpm --filter @cleanstart/cms migrate +pg_dump --schema-only cs_vid_b > /tmp/schema-b.sql +diff /tmp/schema-a.sql /tmp/schema-b.sql && echo "SCHEMA MATCH" +DATABASE_URI=postgres://postgres:1234@localhost:5432/cs_vid_b pnpm --filter @cleanstart/cms migrate # second run must be a no-op +``` +Expected: `SCHEMA MATCH`, and the second run reports nothing to run. Then start the CMS against `cs_vid_b`, open `/admin`, open Videos and Video topics lists and create one of each. + +- [ ] **Step 6: Commit** + +```bash +git add apps/cms/src/migrations/20261024_100000_add_videos.ts apps/cms/src/migrations/index.ts +git commit -m "feat(cms): migration for videos and videoTopics" +``` + +### Task 5: YouTube clients (API with Zod, and RSS) behind one interface + +**Files:** +- Create: `apps/cms/src/payload/lib/videos/youtube-client.ts`, `youtube-client.test.ts` (the API client, steps 1 to 4) +- Create: `apps/cms/src/payload/lib/videos/youtube-rss-client.ts`, `youtube-rss-client.test.ts` (the no-key client, step 5) + +- [ ] **Step 1: Write the failing tests** + +```ts +import { describe, expect, it, vi } from 'vitest'; +import { YoutubeApiError, createYoutubeClient, parseIsoDuration } from './youtube-client'; + +const json = (body: unknown, status = 200): Response => + ({ ok: status < 400, status, json: async () => body }) as Response; + +describe('parseIsoDuration', () => { + it.each([ + ['PT1M41S', 101], + ['PT59S', 59], + ['PT27M46S', 1666], + ['PT1H2M3S', 3723], + ['PT0S', 0], + ['P1DT1H', 90000], + ])('%s -> %i seconds', (iso, seconds) => { + expect(parseIsoDuration(iso)).toBe(seconds); + }); + it('rejects garbage', () => { + expect(() => parseIsoDuration('1:41')).toThrow(YoutubeApiError); + }); +}); + +describe('createYoutubeClient', () => { + it('walks the uploads playlist and reports whether it saw everything', async () => { + const fetchImpl = vi + .fn() + .mockResolvedValueOnce(json({ items: [{ contentDetails: { relatedPlaylists: { uploads: 'UUabc' } } }] })) + .mockResolvedValueOnce(json({ items: [{ contentDetails: { videoId: 'AAAAAAAAAAA' } }], nextPageToken: 'p2' })) + .mockResolvedValueOnce(json({ items: [{ contentDetails: { videoId: 'BBBBBBBBBBB' } }] })); + const client = createYoutubeClient({ apiKey: 'k', fetchImpl }); + expect(await client.listUploadIds('UCabc')).toEqual({ ids: ['AAAAAAAAAAA', 'BBBBBBBBBBB'], complete: true }); + }); + + it('marks the listing incomplete when it hits the page cap', async () => { + const fetchImpl = vi + .fn() + .mockResolvedValueOnce(json({ items: [{ contentDetails: { relatedPlaylists: { uploads: 'UUabc' } } }] })) + .mockResolvedValue(json({ items: [{ contentDetails: { videoId: 'AAAAAAAAAAA' } }], nextPageToken: 'more' })); + const client = createYoutubeClient({ apiKey: 'k', fetchImpl }); + const result = await client.listUploadIds('UCabc', { maxPages: 2 }); + expect(result.complete).toBe(false); + }); + + it('maps a video, reading maxres from the thumbnails', async () => { + const fetchImpl = vi.fn().mockResolvedValueOnce( + json({ + items: [ + { + id: 'AAAAAAAAAAA', + snippet: { + title: 'T', description: 'D', publishedAt: '2026-10-13T14:17:14Z', tags: ['sbom'], + liveBroadcastContent: 'none', + thumbnails: { default: { url: 'u' }, maxres: { url: 'https://i.ytimg.com/vi/AAAAAAAAAAA/maxresdefault.jpg' } }, + }, + contentDetails: { duration: 'PT1M41S' }, + status: { privacyStatus: 'public', embeddable: true }, + }, + ], + }), + ); + const [video] = await createYoutubeClient({ apiKey: 'k', fetchImpl }).getVideos(['AAAAAAAAAAA']); + expect(video).toMatchObject({ id: 'AAAAAAAAAAA', durationSeconds: 101, hasMaxres: true, tags: ['sbom'] }); + }); + + it('throws a typed error that never contains the API key', async () => { + const fetchImpl = vi.fn().mockResolvedValue(json({ error: { message: 'quota' } }, 403)); + const client = createYoutubeClient({ apiKey: 'SECRET-KEY', fetchImpl }); + const error = await client.getVideos(['AAAAAAAAAAA']).catch((e: unknown) => e); + expect(error).toBeInstanceOf(YoutubeApiError); + expect(String((error as Error).message)).not.toContain('SECRET-KEY'); + }); + + it('rejects a response that does not match the schema', async () => { + const fetchImpl = vi.fn().mockResolvedValue(json({ items: [{ id: 'short' }] })); + await expect(createYoutubeClient({ apiKey: 'k', fetchImpl }).getVideos(['AAAAAAAAAAA'])).rejects.toBeInstanceOf(YoutubeApiError); + }); + + it('batches video ids in groups of 50', async () => { + const fetchImpl = vi.fn().mockResolvedValue(json({ items: [] })); + const ids = Array.from({ length: 120 }, (_, i) => String(i).padStart(11, 'x')); + await createYoutubeClient({ apiKey: 'k', fetchImpl }).getVideos(ids); + expect(fetchImpl).toHaveBeenCalledTimes(3); + }); +}); +``` + +- [ ] **Step 2: Run, expect failure** + +Run: `pnpm --filter @cleanstart/cms exec vitest run src/payload/lib/videos/youtube-client.test.ts` +Expected: FAIL, cannot find module `./youtube-client`. + +- [ ] **Step 3: Implement** + +```ts +import { z } from 'zod'; + +export class YoutubeApiError extends Error { + constructor( + message: string, + readonly status: number | null = null, + ) { + super(message); + this.name = 'YoutubeApiError'; + } +} + +const ISO_DURATION = /^P(?:(\d+)D)?T?(?:(\d+)H)?(?:(\d+)M)?(?:(\d+)S)?$/; + +export const parseIsoDuration = (iso: string): number => { + const match = ISO_DURATION.exec(iso); + if (!match) throw new YoutubeApiError(`Unparseable duration: ${iso}`); + const [, d = '0', h = '0', m = '0', s = '0'] = match; + return Number(d) * 86_400 + Number(h) * 3_600 + Number(m) * 60 + Number(s); +}; + +const Id = z.string().length(11); + +const ChannelsResponse = z.object({ + items: z + .array(z.object({ contentDetails: z.object({ relatedPlaylists: z.object({ uploads: z.string() }) }) })) + .default([]), +}); + +const PlaylistItemsResponse = z.object({ + items: z.array(z.object({ contentDetails: z.object({ videoId: Id }) })).default([]), + nextPageToken: z.string().optional(), +}); + +const VideosResponse = z.object({ + items: z + .array( + z.object({ + id: Id, + snippet: z.object({ + title: z.string(), + description: z.string().default(''), + publishedAt: z.string(), + tags: z.array(z.string()).default([]), + liveBroadcastContent: z.enum(['none', 'live', 'upcoming']).default('none'), + thumbnails: z.record(z.object({ url: z.string() })).default({}), + }), + contentDetails: z.object({ duration: z.string() }), + status: z.object({ + privacyStatus: z.enum(['public', 'unlisted', 'private']), + embeddable: z.boolean().default(true), + }), + }), + ) + .default([]), +}); + +export interface YoutubeVideo { + readonly id: string; + readonly title: string; + readonly description: string; + readonly publishedAt: string; + /** `null` when the source cannot say (the RSS feed carries no duration). */ + readonly durationSeconds: number | null; + readonly tags: readonly string[]; + readonly privacyStatus: 'public' | 'unlisted' | 'private'; + readonly embeddable: boolean; + readonly liveBroadcastContent: 'none' | 'live' | 'upcoming'; + readonly hasMaxres: boolean; +} + +export interface YoutubeClient { + listUploadIds( + channelId: string, + options?: { maxPages?: number }, + ): Promise<{ ids: string[]; complete: boolean }>; + getVideos(ids: readonly string[]): Promise<YoutubeVideo[]>; +} + +interface ClientOptions { + apiKey: string; + fetchImpl?: typeof fetch; + timeoutMs?: number; +} + +const BASE = 'https://www.googleapis.com/youtube/v3'; +const BATCH = 50; + +export const createYoutubeClient = ({ + apiKey, + fetchImpl = fetch, + timeoutMs = 10_000, +}: ClientOptions): YoutubeClient => { + const get = async <T>( + path: string, + params: Record<string, string>, + schema: z.ZodType<T>, + ): Promise<T> => { + const url = `${BASE}/${path}?${new URLSearchParams({ ...params, key: apiKey }).toString()}`; + let res: Response; + try { + res = await fetchImpl(url, { signal: AbortSignal.timeout(timeoutMs) }); + } catch { + // The URL carries the key, so never forward the underlying error text. + throw new YoutubeApiError(`YouTube request failed: ${path}`); + } + if (!res.ok) throw new YoutubeApiError(`YouTube ${path} returned ${res.status}`, res.status); + const parsed = schema.safeParse(await res.json()); + if (!parsed.success) throw new YoutubeApiError(`YouTube ${path} response did not match the expected shape`); + return parsed.data; + }; + + return { + async listUploadIds(channelId, { maxPages = 20 } = {}) { + const channels = await get('channels', { part: 'contentDetails', id: channelId }, ChannelsResponse); + const uploads = channels.items[0]?.contentDetails.relatedPlaylists.uploads; + if (!uploads) throw new YoutubeApiError(`No uploads playlist for channel ${channelId}`); + const ids: string[] = []; + let pageToken: string | undefined; + for (let page = 0; page < maxPages; page += 1) { + const res = await get( + 'playlistItems', + { part: 'contentDetails', playlistId: uploads, maxResults: String(BATCH), ...(pageToken ? { pageToken } : {}) }, + PlaylistItemsResponse, + ); + ids.push(...res.items.map((i) => i.contentDetails.videoId)); + pageToken = res.nextPageToken; + if (!pageToken) return { ids, complete: true }; + } + return { ids, complete: false }; + }, + + async getVideos(ids) { + const out: YoutubeVideo[] = []; + for (let i = 0; i < ids.length; i += BATCH) { + const res = await get( + 'videos', + { part: 'snippet,contentDetails,status', id: ids.slice(i, i + BATCH).join(',') }, + VideosResponse, + ); + for (const v of res.items) { + out.push({ + id: v.id, + title: v.snippet.title, + description: v.snippet.description, + publishedAt: v.snippet.publishedAt, + durationSeconds: parseIsoDuration(v.contentDetails.duration), + tags: v.snippet.tags, + privacyStatus: v.status.privacyStatus, + embeddable: v.status.embeddable, + liveBroadcastContent: v.snippet.liveBroadcastContent, + hasMaxres: 'maxres' in v.snippet.thumbnails, + }); + } + } + return out; + }, + }; +}; +``` + +- [ ] **Step 4: Run, expect pass**, then commit. + +```bash +pnpm --filter @cleanstart/cms exec vitest run src/payload/lib/videos/youtube-client.test.ts +git add apps/cms/src/payload/lib/videos/youtube-client.ts apps/cms/src/payload/lib/videos/youtube-client.test.ts +git commit -m "feat(cms): YouTube Data API client with schema-checked responses" +``` + +- [ ] **Step 5: The no-key RSS client (same interface)** + +The web app has a regex parser for this feed (`apps/web/src/lib/youtube-feed.ts`). The CMS cannot import from `apps/web`, so this is a deliberate small duplicate, covered by its own tests. + +Tests first (`youtube-rss-client.test.ts`): + +```ts +import { describe, expect, it, vi } from 'vitest'; +import { createYoutubeRssClient, parseRssFeed } from './youtube-rss-client'; + +const FEED = `<feed xmlns:yt="http://www.youtube.com/xml/schemas/2015" xmlns:media="http://search.yahoo.com/mrss/"> + <entry> + <yt:videoId>AAAAAAAAAAA</yt:videoId> + <title>What Is VEX? & why + 2026-10-13T14:17:14+00:00 + First paragraph. + +#VEX #SBOM + + BBBBBBBBBBBSecond2026-10-12T10:58:15+00:00 +`; + +describe('parseRssFeed', () => { + it('reads id, decoded title, date and the multi-line description', () => { + const [first, second] = parseRssFeed(FEED); + expect(first).toMatchObject({ id: 'AAAAAAAAAAA', title: 'What Is VEX? & why', publishedAt: '2026-10-13T14:17:14+00:00' }); + expect(first?.description).toBe('First paragraph.\n\n#VEX #SBOM'); + expect(second?.description).toBe(''); + }); + it('returns [] for empty or malformed input', () => { + expect(parseRssFeed('')).toEqual([]); + expect(parseRssFeed('')).toEqual([]); + }); +}); + +describe('createYoutubeRssClient', () => { + const respond = (url: string, init?: RequestInit): Response => { + if (url.includes('feeds/videos.xml')) return { ok: true, status: 200, text: async () => FEED } as Response; + // HEAD for the maxres poster: present for A, missing for B. + const present = url.includes('AAAAAAAAAAA'); + return { ok: present, status: present ? 200 : 404 } as Response; + }; + + it('never claims a complete listing (the feed only holds the newest 15)', async () => { + const fetchImpl = vi.fn(async (url: string, init?: RequestInit) => respond(url, init)); + const client = createYoutubeRssClient({ fetchImpl }); + expect((await client.listUploadIds('UCabc')).complete).toBe(false); + }); + + it('returns unknown duration as null, assumes public and embeddable, and checks maxres with HEAD', async () => { + const fetchImpl = vi.fn(async (url: string, init?: RequestInit) => respond(url, init)); + const client = createYoutubeRssClient({ fetchImpl }); + const { ids } = await client.listUploadIds('UCabc'); + const videos = await client.getVideos(ids); + expect(videos).toHaveLength(2); + expect(videos[0]).toMatchObject({ id: 'AAAAAAAAAAA', durationSeconds: null, privacyStatus: 'public', embeddable: true, hasMaxres: true, tags: [] }); + expect(videos[1]?.hasMaxres).toBe(false); + expect(fetchImpl).toHaveBeenCalledWith(expect.stringContaining('maxresdefault.jpg'), expect.objectContaining({ method: 'HEAD' })); + }); + + it('throws a typed error when the feed itself is unreachable', async () => { + const fetchImpl = vi.fn(async () => ({ ok: false, status: 404, text: async () => '' }) as Response); + await expect(createYoutubeRssClient({ fetchImpl }).listUploadIds('UCabc')).rejects.toThrow(/feed/i); + }); +}); +``` + +Implementation: + +```ts +import { YoutubeApiError, type YoutubeClient, type YoutubeVideo } from './youtube-client'; + +const ENTRY = /([\s\S]*?)<\/entry>/g; +const ID = /([A-Za-z0-9_-]{11})<\/yt:videoId>/; +const TITLE = /([\s\S]*?)<\/title>/; +const PUBLISHED = /<published>([\s\S]*?)<\/published>/; +const DESCRIPTION = /<media:description>([\s\S]*?)<\/media:description>/; + +const decode = (value: string): string => + value + .replace(/</g, '<') + .replace(/>/g, '>') + .replace(/"/g, '"') + .replace(/'/g, "'") + .replace(/&#(\d+);/g, (_, code: string) => String.fromCodePoint(Number(code))) + .replace(/&/g, '&') + .trim(); + +export interface RssEntry { + id: string; + title: string; + publishedAt: string; + description: string; +} + +export const parseRssFeed = (xml: string): RssEntry[] => { + const out: RssEntry[] = []; + for (const match of xml.matchAll(ENTRY)) { + const block = match[1] ?? ''; + const id = ID.exec(block)?.[1]; + if (!id) continue; + out.push({ + id, + title: decode(TITLE.exec(block)?.[1] ?? ''), + publishedAt: PUBLISHED.exec(block)?.[1]?.trim() ?? '', + description: decode(DESCRIPTION.exec(block)?.[1] ?? ''), + }); + } + return out; +}; + +interface RssClientOptions { + fetchImpl?: typeof fetch; + timeoutMs?: number; +} + +export const createYoutubeRssClient = ({ + fetchImpl = fetch, + timeoutMs = 10_000, +}: RssClientOptions = {}): YoutubeClient => { + let cache: RssEntry[] = []; + return { + async listUploadIds(channelId) { + let res: Response; + try { + res = await fetchImpl(`https://www.youtube.com/feeds/videos.xml?channel_id=${channelId}`, { + signal: AbortSignal.timeout(timeoutMs), + }); + } catch { + throw new YoutubeApiError('YouTube RSS feed request failed'); + } + if (!res.ok) throw new YoutubeApiError(`YouTube RSS feed returned ${res.status}`, res.status); + cache = parseRssFeed(await res.text()); + // The feed holds only the newest 15, so absence from it proves nothing. + return { ids: cache.map((e) => e.id), complete: false }; + }, + + async getVideos(ids) { + const wanted = new Set(ids); + const out: YoutubeVideo[] = []; + for (const entry of cache.filter((e) => wanted.has(e.id))) { + const poster = await fetchImpl(`https://i.ytimg.com/vi/${entry.id}/maxresdefault.jpg`, { + method: 'HEAD', + signal: AbortSignal.timeout(timeoutMs), + }).catch(() => null); + out.push({ + id: entry.id, + title: entry.title, + description: entry.description, + publishedAt: entry.publishedAt, + durationSeconds: null, + tags: [], + // The feed lists public videos only and cannot report embeddability; the draft review covers it. + privacyStatus: 'public', + embeddable: true, + liveBroadcastContent: 'none', + hasMaxres: poster?.ok === true, + }); + } + return out; + }, + }; +}; +``` + +`getVideos` depends on `listUploadIds` having run first (it reads the feed once and keeps it); `runYoutubeSync` always calls them in that order, and Task 8's test asserts it. + +- [ ] **Step 6: Run both test files, expect pass, commit** + +```bash +pnpm --filter @cleanstart/cms exec vitest run src/payload/lib/videos +git add apps/cms/src/payload/lib/videos/youtube-rss-client.ts apps/cms/src/payload/lib/videos/youtube-rss-client.test.ts +git commit -m "feat(cms): no-key RSS client for the YouTube video sync" +``` + +### Task 6: `planSync`, the pure core + +**Files:** +- Create: `apps/cms/src/payload/lib/videos/plan-sync.ts`, `plan-sync.test.ts` + +- [ ] **Step 1: Write the failing tests** + +```ts +import { describe, expect, it } from 'vitest'; +import type { YoutubeVideo } from './youtube-client'; +import { contentHash, normaliseTag, planSync, suggestTopicIds, toSource } from './plan-sync'; + +const video = (over: Partial<YoutubeVideo> = {}): YoutubeVideo => ({ + id: 'AAAAAAAAAAA', title: 'T', description: 'First paragraph.\n\n#SBOM #VEX', publishedAt: '2026-10-13T14:17:14Z', + durationSeconds: 101, tags: [], privacyStatus: 'public', embeddable: true, liveBroadcastContent: 'none', hasMaxres: true, + ...over, +}); +const topics = [ + { id: 1, name: 'SBOM', aliases: ['software bill of materials'] }, + { id: 2, name: 'VEX', aliases: [] }, + { id: 3, name: 'Base images', aliases: [] }, +]; + +describe('normaliseTag', () => { + it('ignores case, hashes and spacing', () => { + expect(normaliseTag('#Base Images')).toBe('baseimages'); + expect(normaliseTag('base-images')).toBe('baseimages'); + }); +}); + +describe('suggestTopicIds', () => { + it('matches hashtags in the description and YouTube tags, once each', () => { + const ids = suggestTopicIds({ tags: ['Software Bill of Materials'], description: 'x\n\n#VEX #ContainerSecurity #sbom' }, topics); + expect(ids).toEqual([1, 2]); + }); + it('suggests nothing when nothing matches', () => { + expect(suggestTopicIds({ tags: ['kubernetes'], description: '#devops' }, topics)).toEqual([]); + }); +}); + +describe('planSync', () => { + const base = { existing: [], complete: true, topics }; + + it('creates a needs-review draft for a new public video, with suggested topics', () => { + const plan = planSync({ ...base, fetched: [video()] }); + expect(plan.creates).toHaveLength(1); + expect(plan.creates[0]).toMatchObject({ externalId: 'AAAAAAAAAAA', suggestedTopicIds: [1, 2] }); + }); + + it.each([ + ['unlisted', video({ privacyStatus: 'unlisted' }), 'not-public'], + ['private', video({ privacyStatus: 'private' }), 'not-public'], + ['not embeddable', video({ embeddable: false }), 'not-embeddable'], + ['upcoming', video({ liveBroadcastContent: 'upcoming' }), 'not-finished'], + ['live', video({ liveBroadcastContent: 'live' }), 'not-finished'], + ])('skips a %s video', (_label, v, reason) => { + const plan = planSync({ ...base, fetched: [v] }); + expect(plan.creates).toEqual([]); + expect(plan.skipped).toEqual([{ externalId: 'AAAAAAAAAAA', reason }]); + }); + + it('leaves an unchanged video alone', () => { + const v = video(); + const plan = planSync({ + ...base, fetched: [v], + existing: [{ id: 9, externalId: v.id, syncState: 'synced', contentHash: contentHash(toSource(v)), status: 'published', durationSeconds: 101 }], + }); + expect(plan).toMatchObject({ creates: [], updates: [], unchanged: 1 }); + }); + + it('updates only the source when YouTube content changed', () => { + const v = video({ description: 'edited' }); + const plan = planSync({ + ...base, fetched: [v], + existing: [{ id: 9, externalId: v.id, syncState: 'synced', contentHash: 'old', status: 'published', durationSeconds: 101 }], + }); + expect(plan.updates).toEqual([{ id: 9, source: toSource(v), restoreFromMissing: false }]); + }); + + it('flags a vanished video as missing, but only when the listing was complete', () => { + const existing = [{ id: 9, externalId: 'GONEGONEGON', syncState: 'synced' as const, contentHash: 'h', status: 'published' as const, durationSeconds: null }]; + expect(planSync({ ...base, fetched: [], existing }).missing).toEqual([9]); + expect(planSync({ ...base, fetched: [], existing, complete: false }).missing).toEqual([]); + }); + + it('does not treat an RSS run (no duration) as a change, and never erases a known duration', () => { + const withDuration = video(); + const stored = toSource(withDuration); + const rssView = video({ durationSeconds: null }); + const plan = planSync({ + ...base, fetched: [rssView], + existing: [{ id: 9, externalId: rssView.id, syncState: 'synced', contentHash: contentHash(stored), status: 'published', durationSeconds: 101 }], + }); + expect(plan).toMatchObject({ updates: [], unchanged: 1 }); + }); + + it('never touches manual documents', () => { + const existing = [{ id: 5, externalId: 'MANUALMANUA', syncState: 'manual' as const, contentHash: null, status: 'published' as const, durationSeconds: null }]; + const plan = planSync({ ...base, fetched: [], existing }); + expect(plan.missing).toEqual([]); + }); + + it('brings a missing video back when it reappears', () => { + const v = video(); + const plan = planSync({ + ...base, fetched: [v], + existing: [{ id: 9, externalId: v.id, syncState: 'missing', contentHash: contentHash(toSource(v)), status: 'published', durationSeconds: 101 }], + }); + expect(plan.updates).toEqual([{ id: 9, source: toSource(v), restoreFromMissing: true }]); + }); +}); +``` + +- [ ] **Step 2: Run, expect failure** (module not found). + +- [ ] **Step 3: Implement** + +```ts +import { createHash } from 'node:crypto'; + +import type { YoutubeVideo } from './youtube-client'; + +export type SyncState = 'synced' | 'needs-review' | 'missing' | 'manual'; +export type SkipReason = 'not-public' | 'not-embeddable' | 'not-finished'; +type DocId = number | string; + +export interface SourceSnapshot { + readonly title: string; + readonly description: string; + readonly durationSeconds: number | null; + readonly publishedAt: string; + readonly tags: readonly string[]; + readonly privacyStatus: string; + readonly embeddable: boolean; + readonly hasMaxres: boolean; + readonly thumbnailUrl: string; +} + +export interface ExistingVideo { + readonly id: DocId; + readonly externalId: string; + readonly syncState: SyncState; + readonly contentHash: string | null; + readonly status: 'draft' | 'published'; + /** What the document already stores, so an RSS run (which knows no duration) never erases it. */ + readonly durationSeconds: number | null; +} + +export interface TopicRef { + readonly id: DocId; + readonly name: string; + readonly aliases: readonly string[]; +} + +export interface SyncPlan { + readonly creates: { externalId: string; source: SourceSnapshot; suggestedTopicIds: DocId[] }[]; + readonly updates: { id: DocId; source: SourceSnapshot; restoreFromMissing: boolean }[]; + readonly unchanged: number; + readonly missing: DocId[]; + readonly skipped: { externalId: string; reason: SkipReason }[]; +} + +export const normaliseTag = (raw: string): string => raw.toLowerCase().replace(/[^a-z0-9]/g, ''); + +export const toSource = (v: YoutubeVideo): SourceSnapshot => ({ + title: v.title, + description: v.description, + durationSeconds: v.durationSeconds, + publishedAt: v.publishedAt, + tags: v.tags, + privacyStatus: v.privacyStatus, + embeddable: v.embeddable, + hasMaxres: v.hasMaxres, + thumbnailUrl: `https://i.ytimg.com/vi/${v.id}/${v.hasMaxres ? 'maxresdefault' : 'hqdefault'}.jpg`, +}); + +/** Unknown (null) never overwrites a duration we already have. */ +export const withKnownDuration = (source: SourceSnapshot, existing: number | null): SourceSnapshot => + source.durationSeconds === null && existing !== null ? { ...source, durationSeconds: existing } : source; + +export const contentHash = (source: SourceSnapshot): string => + createHash('sha256').update(JSON.stringify(source)).digest('hex'); + +const HASHTAG = /(?:^|\s)#([\p{L}\p{N}_]+)/gu; + +export const suggestTopicIds = ( + input: { tags: readonly string[]; description: string }, + topics: readonly TopicRef[], +): DocId[] => { + const candidates = new Set<string>([ + ...input.tags.map(normaliseTag), + ...[...input.description.matchAll(HASHTAG)].map((m) => normaliseTag(m[1] ?? '')), + ]); + return topics + .filter((t) => [t.name, ...t.aliases].some((label) => candidates.has(normaliseTag(label)))) + .map((t) => t.id); +}; + +const skipReason = (v: YoutubeVideo): SkipReason | null => { + if (v.privacyStatus !== 'public') return 'not-public'; + if (!v.embeddable) return 'not-embeddable'; + if (v.liveBroadcastContent !== 'none') return 'not-finished'; + return null; +}; + +export const planSync = ({ + existing, + fetched, + complete, + topics, +}: { + existing: readonly ExistingVideo[]; + fetched: readonly YoutubeVideo[]; + complete: boolean; + topics: readonly TopicRef[]; +}): SyncPlan => { + const byExternalId = new Map(existing.map((e) => [e.externalId, e])); + const seen = new Set<string>(); + const creates: SyncPlan['creates'] = []; + const updates: SyncPlan['updates'] = []; + const skipped: SyncPlan['skipped'] = []; + let unchanged = 0; + + for (const v of fetched) { + seen.add(v.id); + const reason = skipReason(v); + if (reason) { + skipped.push({ externalId: v.id, reason }); + continue; + } + const current = byExternalId.get(v.id); + const source = withKnownDuration(toSource(v), current?.durationSeconds ?? null); + if (!current) { + creates.push({ + externalId: v.id, + source, + suggestedTopicIds: suggestTopicIds({ tags: v.tags, description: v.description }, topics), + }); + } else if (current.syncState === 'manual') { + unchanged += 1; + } else if (current.contentHash === contentHash(source) && current.syncState !== 'missing') { + unchanged += 1; + } else { + updates.push({ id: current.id, source, restoreFromMissing: current.syncState === 'missing' }); + } + } + + const missing = complete + ? existing + .filter((e) => e.syncState !== 'manual' && e.syncState !== 'missing' && !seen.has(e.externalId)) + .map((e) => e.id) + : []; + + return { creates, updates, unchanged, missing, skipped }; +}; +``` + +- [ ] **Step 4: Run, expect pass, commit** + +```bash +pnpm --filter @cleanstart/cms exec vitest run src/payload/lib/videos/plan-sync.test.ts +git add apps/cms/src/payload/lib/videos/plan-sync.ts apps/cms/src/payload/lib/videos/plan-sync.test.ts +git commit -m "feat(cms): pure planner for the YouTube video sync" +``` + +### Task 7: `applyPlan`, the Payload shell + +**Files:** +- Create: `apps/cms/src/payload/lib/videos/apply-sync.ts`, `apply-sync.test.ts` + +The shell takes a structurally typed `payload` (the pattern in `purge-preview-audit.ts`) so tests use a fake. One failing document must never abort the batch. + +- [ ] **Step 1: Write the failing tests** (fake payload recording calls) + +```ts +import { describe, expect, it, vi } from 'vitest'; +import { applyPlan } from './apply-sync'; +import { toSource } from './plan-sync'; + +const source = toSource({ + id: 'AAAAAAAAAAA', title: 'T', description: 'First paragraph.\n\nmore', publishedAt: '2026-10-13T14:17:14Z', + durationSeconds: 101, tags: [], privacyStatus: 'public', embeddable: true, liveBroadcastContent: 'none', hasMaxres: true, +}); +const emptyPlan = { creates: [], updates: [], unchanged: 0, missing: [], skipped: [] }; + +const fakePayload = () => ({ + create: vi.fn().mockResolvedValue({ id: 1 }), + update: vi.fn().mockResolvedValue({ id: 1 }), + logger: { info: vi.fn(), error: vi.fn() }, +}); + +describe('applyPlan', () => { + it('creates drafts in needs-review and pre-fills title, slug input, summary and topics', async () => { + const payload = fakePayload(); + const result = await applyPlan(payload, { ...emptyPlan, creates: [{ externalId: 'AAAAAAAAAAA', source, suggestedTopicIds: [1, 2] }] }, { autoPublish: false }); + expect(payload.create).toHaveBeenCalledWith( + expect.objectContaining({ + collection: 'videos', + draft: true, + overrideAccess: true, + data: expect.objectContaining({ + provider: 'youtube', externalId: 'AAAAAAAAAAA', title: 'T', summary: 'First paragraph.', + topics: [1, 2], syncState: 'needs-review', _status: 'draft', + }), + }), + ); + expect(result.created).toBe(1); + }); + + it('writes only the source group when updating', async () => { + const payload = fakePayload(); + await applyPlan(payload, { ...emptyPlan, updates: [{ id: 9, source, restoreFromMissing: false }] }, { autoPublish: false }); + const args = payload.update.mock.calls[0]?.[0] as { data: Record<string, unknown> }; + expect(Object.keys(args.data)).toEqual(['source']); + }); + + it('restores syncState when a missing video returns', async () => { + const payload = fakePayload(); + await applyPlan(payload, { ...emptyPlan, updates: [{ id: 9, source, restoreFromMissing: true }] }, { autoPublish: false }); + expect((payload.update.mock.calls[0]?.[0] as { data: Record<string, unknown> }).data.syncState).toBe('synced'); + }); + + it('marks vanished videos missing without unpublishing them', async () => { + const payload = fakePayload(); + await applyPlan(payload, { ...emptyPlan, missing: [4] }, { autoPublish: false }); + expect(payload.update).toHaveBeenCalledWith(expect.objectContaining({ id: 4, data: { syncState: 'missing' } })); + }); + + it('keeps going when one item fails and reports it', async () => { + const payload = fakePayload(); + payload.create.mockRejectedValueOnce(new Error('boom')).mockResolvedValueOnce({ id: 2 }); + const plan = { ...emptyPlan, creates: [ + { externalId: 'AAAAAAAAAAA', source, suggestedTopicIds: [] }, + { externalId: 'BBBBBBBBBBB', source, suggestedTopicIds: [] }, + ] }; + const result = await applyPlan(payload, plan, { autoPublish: false }); + expect(result).toMatchObject({ created: 1, failed: 1 }); + expect(payload.logger.error).toHaveBeenCalled(); + }); + + it('auto-publishes only when asked and only when classification is complete', async () => { + const payload = fakePayload(); + await applyPlan(payload, { ...emptyPlan, creates: [{ externalId: 'AAAAAAAAAAA', source, suggestedTopicIds: [1] }] }, { autoPublish: true }); + // No format can be guessed, so the publish gate would reject it: it stays a draft. + expect((payload.create.mock.calls[0]?.[0] as { data: { _status: string } }).data._status).toBe('draft'); + }); +}); +``` + +- [ ] **Step 2: Run, expect failure.** + +- [ ] **Step 3: Implement** + +```ts +import { contentHash, type SourceSnapshot, type SyncPlan } from './plan-sync'; + +type DocId = number | string; + +export interface SyncPayload { + create(args: { + collection: 'videos'; + data: Record<string, unknown>; + draft: boolean; + overrideAccess: boolean; + depth: number; + }): Promise<{ id: DocId }>; + update(args: { + collection: 'videos'; + id: DocId; + data: Record<string, unknown>; + overrideAccess: boolean; + depth: number; + }): Promise<unknown>; + logger: { + info(meta: Record<string, unknown>, message: string): void; + error(meta: Record<string, unknown>, message: string): void; + }; +} + +export interface ApplyResult { + created: number; + updated: number; + markedMissing: number; + failed: number; +} + +const FIRST_PARAGRAPH_MAX = 320; + +const firstParagraph = (text: string): string => { + const paragraph = text.split(/\n\s*\n/).map((p) => p.replace(/\s+/g, ' ').trim()).find((p) => p.length > 0) ?? ''; + if (paragraph.length <= FIRST_PARAGRAPH_MAX) return paragraph; + const cut = paragraph.slice(0, FIRST_PARAGRAPH_MAX); + return `${cut.slice(0, Math.max(cut.lastIndexOf(' '), 0) || FIRST_PARAGRAPH_MAX).replace(/[,;:\s]+$/, '')}.`; +}; + +const sourceData = (source: SourceSnapshot, fetchedAt: string): Record<string, unknown> => ({ + title: source.title, + description: source.description, + durationSeconds: source.durationSeconds, + publishedAt: source.publishedAt, + tags: source.tags.map((value) => ({ value })), + privacyStatus: source.privacyStatus, + embeddable: source.embeddable, + hasMaxres: source.hasMaxres, + thumbnailUrl: source.thumbnailUrl, + contentHash: contentHash(source), + fetchedAt, +}); + +export const applyPlan = async ( + payload: SyncPayload, + plan: SyncPlan, + // `autoPublish` is accepted so the env switch has one place to be honoured, but a new + // video is never published here: the publish gate needs a format, which nothing guesses. + _options: { autoPublish: boolean }, + now: () => Date = () => new Date(), +): Promise<ApplyResult> => { + const result: ApplyResult = { created: 0, updated: 0, markedMissing: 0, failed: 0 }; + const fetchedAt = now().toISOString(); + + const attempt = async (what: string, ref: Record<string, unknown>, run: () => Promise<unknown>): Promise<boolean> => { + try { + await run(); + return true; + } catch (error) { + result.failed += 1; + payload.logger.error({ ...ref, error: error instanceof Error ? error.message : String(error) }, `videos sync: ${what} failed`); + return false; + } + }; + + for (const item of plan.creates) { + const ok = await attempt('create', { externalId: item.externalId }, () => + payload.create({ + collection: 'videos', + draft: true, + overrideAccess: true, + depth: 0, + data: { + provider: 'youtube', + externalId: item.externalId, + title: item.source.title, + summary: firstParagraph(item.source.description), + topics: item.suggestedTopicIds, + syncState: 'needs-review', + source: sourceData(item.source, fetchedAt), + _status: 'draft', + }, + }), + ); + if (ok) result.created += 1; + } + + for (const item of plan.updates) { + const data: Record<string, unknown> = { source: sourceData(item.source, fetchedAt) }; + if (item.restoreFromMissing) data.syncState = 'synced'; + const ok = await attempt('update', { id: item.id }, () => + payload.update({ collection: 'videos', id: item.id, data, overrideAccess: true, depth: 0 }), + ); + if (ok) result.updated += 1; + } + + for (const id of plan.missing) { + const ok = await attempt('mark missing', { id }, () => + payload.update({ collection: 'videos', id, data: { syncState: 'missing' }, overrideAccess: true, depth: 0 }), + ); + if (ok) result.markedMissing += 1; + } + + payload.logger.info({ ...result, unchanged: plan.unchanged, skipped: plan.skipped.length }, 'videos sync applied'); + return result; +}; +``` + +- [ ] **Step 4: Run, expect pass, commit.** + +```bash +pnpm --filter @cleanstart/cms exec vitest run src/payload/lib/videos/apply-sync.test.ts +git add apps/cms/src/payload/lib/videos/apply-sync.ts apps/cms/src/payload/lib/videos/apply-sync.test.ts +git commit -m "feat(cms): apply a video sync plan with per-item failure isolation" +``` + +Note on `autoPublish`: the env switch exists for decision 1, but publishing requires a `format`, which nothing guesses (D7). When marketing wants hands-off, the follow-up is a small rule such as "duration over 10 minutes means talk", agreed explicitly. Until then, the switch is accepted and documented but never bypasses the gate. + +### Task 8: The scheduled task, credentials and env plumbing + +**Files:** +- Create: `apps/cms/src/payload/lib/videos/run-sync.ts`, `apps/cms/src/payload/jobs/sync-youtube-videos.ts` +- Modify: `apps/cms/src/payload.config.ts`, `apps/cms/src/payload/lib/integrations/credentials.ts`, `apps/cms/.env.example`, `.github/workflows/deploy-cms.yml`, `CLAUDE.md` (jobs table) + +- [ ] **Step 1: Credentials resolver** in `credentials.ts`, next to `resolveCruxCredentials`. Unlike CrUX it never returns null: no key is a supported mode. + +```ts +export interface YoutubeCredentials { + /** null selects the no-key RSS client. */ + readonly apiKey: string | null; + readonly channelId: string; +} + +export const resolveYoutubeCredentials = (): YoutubeCredentials => ({ + apiKey: readEnv('YOUTUBE_API_KEY'), + channelId: readEnv('YOUTUBE_CHANNEL_ID') ?? 'UC13c-XRmRlkWpp6WMG9SDjw', +}); +``` + +- [ ] **Step 2: Orchestrator** `run-sync.ts` loads existing videos and topics (paginated `find`, `overrideAccess: true`, `depth: 0`, `select` only the fields it needs), calls `client.listUploadIds` then `client.getVideos`, `planSync`, `applyPlan`, and returns the counts. Constructor-inject `client` and `payload` so it is testable with the fakes from Tasks 5 and 7. It must call `listUploadIds` before `getVideos` (the RSS client relies on that order). Add a test that runs it end to end with a fake client and fake payload, asserts the counts line up with the plan, and asserts the call order. Also load each existing document's `source.durationSeconds` into `ExistingVideo.durationSeconds`. + +- [ ] **Step 3: Task** (mirror `refresh-crux.ts`, including the loud log when the key is missing): + +```ts +import type { TaskConfig } from 'payload'; + +import { resolveYoutubeCredentials } from '../lib/integrations/credentials'; +import { runYoutubeSync } from '../lib/videos/run-sync'; +import { createYoutubeClient } from '../lib/videos/youtube-client'; +import { createYoutubeRssClient } from '../lib/videos/youtube-rss-client'; + +/** + * Hourly at minute 7 UTC: imports new public YouTube uploads as needs-review + * drafts and refreshes the synced snapshot of existing ones. Uses the Data API + * when YOUTUBE_API_KEY is set (duration, privacy, missing-video detection) and + * the RSS feed otherwise. Gated by PAYLOAD_AUTO_RUN. + */ +export const syncYoutubeVideosTask: TaskConfig<'syncYoutubeVideos'> = { + slug: 'syncYoutubeVideos', + retries: 1, + schedule: [{ cron: '7 * * * *', queue: 'youtubeVideosSync' }], + handler: async ({ req }) => { + const creds = resolveYoutubeCredentials(); + const mode = creds.apiKey ? 'api' : 'rss'; + // Logged on every run so a missing key is visible, never silent. + req.payload.logger.info({ mode }, 'syncYoutubeVideos starting'); + const result = await runYoutubeSync({ + payload: req.payload as unknown as Parameters<typeof runYoutubeSync>[0]['payload'], + client: creds.apiKey ? createYoutubeClient({ apiKey: creds.apiKey }) : createYoutubeRssClient(), + channelId: creds.channelId, + autoPublish: process.env.VIDEOS_AUTO_PUBLISH === 'true', + }); + return { output: { mode, ...result } }; + }, +}; +``` + +- [ ] **Step 4: Register** in `payload.config.ts`: import, add `syncYoutubeVideosTask` to `jobs.tasks`, and add to `autoRun`: + +```ts + { + cron: '7 * * * *', // hourly at :07 UTC, YouTube channel sync (new uploads become needs-review drafts) + queue: 'youtubeVideosSync', + }, +``` + +- [ ] **Step 5: Env plumbing.** Add `YOUTUBE_API_KEY=` (optional: leave empty to use the RSS feed), `YOUTUBE_CHANNEL_ID=`, `VIDEOS_AUTO_PUBLISH=` (with comments) to `apps/cms/.env.example`. In `.github/workflows/deploy-cms.yml` add `YOUTUBE_API_KEY: ${{ secrets.YOUTUBE_API_KEY }}` beside `CRUX_API_KEY` (line ~167) and `echo "YOUTUBE_API_KEY=${YOUTUBE_API_KEY}"` beside line ~258. The deploy re-renders `/opt/cleanstart/.env` from these on every run, so editing the file on the droplet does not survive (see the Email section of `CLAUDE.md`, which documents the same re-render behaviour). The workflow lines are added now so nothing needs a code change later; someone with repo admin adds the GitHub secret whenever decision 6 is acted on. An empty secret is fine. + +- [ ] **Step 6: Regenerate types, update docs, gate** + +```bash +pnpm --filter @cleanstart/cms generate:types +pnpm --filter @cleanstart/cms lint && pnpm --filter @cleanstart/cms typecheck && pnpm --filter @cleanstart/cms test +``` +Add a row to the Background jobs table in `CLAUDE.md` (thirteen tasks, `syncYoutubeVideos`, hourly :07, `jobs/sync-youtube-videos.ts`). + +- [ ] **Step 7: Commit** + +```bash +git add apps/cms/src/payload/lib/videos apps/cms/src/payload/jobs/sync-youtube-videos.ts apps/cms/src/payload.config.ts apps/cms/src/payload/lib/integrations/credentials.ts apps/cms/.env.example .github/workflows/deploy-cms.yml CLAUDE.md apps/cms/src/payload-types.ts +git commit -m "feat(cms): hourly YouTube video sync task" +``` + +### Task 9: Publish gate for videos + +**Files:** +- Modify: `apps/cms/src/payload/hooks/publish-gate.ts`, `apps/cms/src/payload/lib/wire-publish-gate.ts`, the publish-gate test (find it with `rg -l runPublishChecks apps/cms/src`) + +- [ ] **Step 1: Failing test** asserting that `runPublishChecks({ title: 'X', summary: '' }, 'videos')` returns blockers `video-format`, `video-topics`, `video-summary`, a non-blocking warning `video-duration`, and that a complete doc returns none of them failing. + +- [ ] **Step 2: Add `'videos'` to `GATED_SLUGS`** in `wire-publish-gate.ts`. + +- [ ] **Step 3: Add the checks** inside `runPublishChecks`, after the existing checks: + +```ts + if (collectionSlug === 'videos') { + const hasDuration = + resolveString(data.durationLabel) != null || + typeof (data.source as { durationSeconds?: unknown } | undefined)?.durationSeconds === 'number'; + const topics = Array.isArray(data.topics) ? data.topics : []; + const summary = resolveString(data.summary); + checks.push( + { + id: 'video-format', + label: 'Format chosen', + severity: 'blocker', + pass: resolveString(data.format) != null, + message: resolveString(data.format) != null ? null : 'Choose Explainer, Product demo or Talk.', + }, + { + id: 'video-topics', + label: 'At least one topic', + severity: 'blocker', + pass: topics.length > 0, + message: topics.length > 0 ? null : 'Add at least one topic so the video shows up in the topic filter.', + }, + { + id: 'video-summary', + label: 'Summary written', + severity: 'blocker', + pass: summary != null, + message: summary != null ? null : 'Write the one or two sentence summary shown on cards and in search results.', + }, + { + id: 'video-duration', + label: 'Duration known', + severity: 'warn', + pass: hasDuration, + message: hasDuration ? null : 'Type the length (for example 1:41) so the card can show a time badge.', + }, + ); + } +``` + +- [ ] **Step 4: Run the gate tests, commit** (`feat(cms): publish checks for videos`). + +### Task 10: Optional, a Teams notification for new imports + +Small and independently shippable. Skip if Teams alerts are not wanted. + +- [ ] Add `'videos.imported'` to `WebhookEventName` in `lib/webhooks/dispatch.ts`, add a card builder in `teams.ts` (title "N new videos need review", link to `/admin/collections/videos?where[syncState][equals]=needs-review`), and call `dispatchEvent` at the end of `runYoutubeSync` when `created > 0`. Wrap in try/catch so a Teams outage never fails the sync. Test the card builder in `teams.test.ts`. + +### Task 11: Backfill script and seed data + +**Files:** +- Create: `apps/cms/scripts/seed-videos-from-catalog.ts`, `apps/cms/scripts/data/videos-catalog.json` + +The CMS app cannot import from `apps/web`, so the catalog is exported once to JSON (the one-time conversion is a throwaway `tsx` one-liner over `apps/web/src/lib/videos-catalog.ts`, committed as data, not as a dependency). + +- [ ] **Step 1: Export the catalog to JSON** (`id`, `slug`, `title`, `format`, `topics`, `summary`) plus the 11 topics with `aliases` seeded from the real hashtags already used in the video descriptions (`sbom`, `vex`, `cves`, `containersecurity` maps to no topic on purpose). + +- [ ] **Step 2: Script behaviour** (follow `backfill-resource-type.ts` conventions: dry run by default, `--apply` to write, idempotent, no `.env` read, runs with `/app/node_modules/.bin/tsx` in the container): + 1. Upsert `videoTopics` by slug (slug equals the old id so `?topic=` URLs keep working). + 2. For each catalog entry, find the `videos` doc by `externalId`. If the sync already created it (drafts exist after rollout step 2), update it with the catalog's `slug`, `title`, `format`, `topics`, `summary`, set `syncState: 'synced'`, and publish. If it does not exist, create it with `source` built from the same client the sync uses (RSS or API) and `durationSeconds` taken from the catalog JSON (all 15 are known). Compute `contentHash` with the same `withKnownDuration` + `contentHash` functions as the sync, otherwise the next sync would see a spurious change. + 3. Print a table of created, updated, unchanged, and any slug conflict. A slug conflict (the doc already has a different slug) aborts the run; never silently renumber. + 4. After writing, call `revalidateWeb` itself and wait for it. A script that ends in `process.exit` kills the hook's deferred purge timer (memory `prod-script-skips-deferred-purge`), which is why prod once served stale pages after a backfill. + +- [ ] **Step 3: Test the planning half** with a fixture (pure function that decides create versus update versus conflict), `seed-videos-from-catalog.test.ts`. + +- [ ] **Step 4: Dry run locally against the scratch DB**, read the report, then `--apply`, then open the admin and spot check five videos. + +- [ ] **Step 5: Commit** (`feat(cms): backfill script that moves the video catalog into the CMS`). + +### Task 12: Web reads the CMS, behind a switch + +**Files:** +- Create: `apps/web/src/lib/videos-cms.ts`, `videos-cms.test.ts` +- Modify: `apps/web/src/lib/videos.ts`, `videos-utils.ts`, `videos-catalog.ts`, `VideosBrowser.tsx`, `app/videos/page.tsx`, `app/videos/[slug]/page.tsx` + +- [ ] **Step 1: Make topics data, not constants.** `videos-utils.ts` currently imports `VIDEO_TOPICS` as a constant. Change `Video.topics` to `readonly { slug: string; label: string }[]`, make `topicLabel` read from the video's own topic objects, and make `parseVideoFilters(read, knownTopicSlugs)` validate against the list passed in. Update `videos-utils.test.ts` first (the existing 26 tests are the safety net) so it fails, then fix the code. `VideosBrowser` receives `topics` (the taxonomy, with counts) as a prop from the server page. + +- [ ] **Step 2: CMS reader with a mapping test.** + +```ts +// apps/web/src/lib/videos-cms.ts +import { fetchCMS } from "./cms-fetch"; +import type { Video } from "./videos-utils"; + +type CmsTopic = { slug: string; name: string }; +type CmsVideo = { + id: number | string; + slug: string; + title: string; + externalId: string; + format?: "explainer" | "demo" | "talk" | null; + topics?: (CmsTopic | number | string)[] | null; + summary?: string | null; + source?: { + description?: string | null; + durationSeconds?: number | null; + publishedAt?: string | null; + hasMaxres?: boolean | null; + } | null; + publishedAt?: string | null; + durationLabel?: string | null; +}; +type CmsList = { docs: CmsVideo[] }; + +/** "1:41" -> 101, "1:02:10" -> 3730, anything else -> null. */ +export const parseDurationLabel = (label: string | null | undefined): number | null => { + const match = /^(?:(\d{1,2}):)?(\d{1,2}):(\d{2})$/.exec(label ?? ""); + if (!match) return null; + const [, h = "0", m = "0", s = "0"] = match; + return Number(h) * 3600 + Number(m) * 60 + Number(s); +}; + +export const mapCmsVideo = (doc: CmsVideo): Video => ({ + id: doc.externalId, + slug: doc.slug, + title: doc.title, + format: doc.format ?? null, + topics: (doc.topics ?? []).filter((t): t is CmsTopic => typeof t === "object").map((t) => ({ slug: t.slug, label: t.name })), + publishedAt: doc.source?.publishedAt ?? doc.publishedAt ?? "", + durationSeconds: doc.source?.durationSeconds ?? parseDurationLabel(doc.durationLabel) ?? undefined, + summary: doc.summary ?? "", + description: doc.source?.description ?? doc.summary ?? "", + thumb: doc.source?.hasMaxres === false ? "hq" : "maxres", +}); + +export async function getCmsVideos(): Promise<Video[]> { + const params = new URLSearchParams({ + "where[_status][equals]": "published", + sort: "-source.publishedAt", + depth: "1", + limit: "200", + }); + // Throws on failure on purpose: during ISR revalidation Next then keeps serving the last good page + // instead of caching an empty listing. + const data = await fetchCMS<CmsList>(`/api/videos?${params.toString()}`, { tags: ["videos"] }); + return data.docs.filter((d) => d.slug && d.externalId).map(mapCmsVideo); +} +``` +Tests: mapping with populated topics, with unpopulated ids (dropped), with `hasMaxres: false` giving `"hq"`, with `source.durationSeconds` absent and `durationLabel: "1:41"` giving 101 (and `"nonsense"` giving no duration), `parseDurationLabel` cases (`1:41`, `1:02:10`, `0:59`, empty, `1:2`), and an assertion that a `fetchCMS` rejection propagates (not swallowed). + +- [ ] **Step 3: The switch** in `lib/videos.ts`: + +```ts +import { getCmsVideos } from "./videos-cms"; + +const usesCms = (): boolean => process.env.VIDEOS_SOURCE === "cms"; + +export async function getVideos(): Promise<Video[]> { + if (usesCms()) return getCmsVideos(); + const feed = await getChannelVideos(FEED_LIMIT); + return mergeVideos(VIDEO_CATALOG, feed); +} +``` +Everything else (cards, filters, watch page, JSON-LD) keeps working because `Video` keeps its shape. That is the point of mapping at the edge. + +- [ ] **Step 4: Run the whole web gate** with `VIDEOS_SOURCE` unset (catalog) and again with it set to `cms` against a local CMS containing the backfilled data. Both must pass the 26-check browser suite from this session (kept at `scratchpad/live/interact.cjs`; copy it into `apps/web/tests/e2e/videos.spec.ts` as part of this task so it lives in the repo). + +- [ ] **Step 5: Commit** (`feat(web): read /videos from the CMS behind VIDEOS_SOURCE`). + +### Task 13: Web wiring + +**Files:** `apps/web/src/app/sitemap.ts`, `lib/detail-route-not-found-guard.ts`, `app/preview/[collection]/[slug]/page.tsx`, `components/search/search-ui.tsx` + +- [ ] **Sitemap:** with the CMS as source, read published videos through the existing `fetchDocs('videos', BLOG_FILTER)` helper (like `advisories`) instead of `getVideos()`, so noindex flags and `contentUpdatedAt` lastmod apply. Extend `sitemap.test.ts`: indexable video listed with its real `lastmod`, a `seo.indexable = noindex` video dropped. Keep the catalog branch while the switch exists. +- [ ] **404 guard:** add the `/videos/` row to `DETAIL_ROUTES` (section 5). Add a test in the guard's test file for an unknown video slug. +- [ ] **Preview:** register a `renderVideoDetail` in the preview route map, exactly like `advisories: renderAdvisoryDetail`, so editors can preview a draft at `/preview/videos/<slug>`. +- [ ] **Search UI:** add an icon and accent for `videos` in `search-ui.tsx`. +- [ ] **Gate and commit.** + +### Task 14: Cutover, verification and rollback drill + +- [ ] Add a "Videos CMS cutover" entry to `docs/operations/PRODUCTION-ROLLOUT.md` containing the rollout order from section 7 verbatim, the backfill command with `--apply`, and the rollback line. +- [ ] Add the three env vars to `docs/operations/CMS-DEPLOYMENT.md`. +- [ ] **Parity check before the flip** (script committed under `apps/web/scripts/compare-videos-sources.mjs`): fetch the list from both sources and assert same count, same slugs, same titles, same topic slugs, same order. Any difference blocks the flip. +- [ ] **After the flip, verify on production:** `/videos` returns 200 with 15 cards in the static HTML (`curl | grep -c 'href="/videos/'`), every existing URL still returns 200, `sitemap.xml` lists all 15, one watch page has `VideoObject` JSON-LD, an unknown slug is a real 404. +- [ ] **Drill the rollback once:** set `VIDEOS_SOURCE=catalog`, redeploy, confirm the site is unchanged, flip back. Do this before declaring done, not during an incident. +- [ ] **Editor acceptance:** a marketing editor, with the checklist only, publishes one imported draft end to end (set format, confirm topics, write summary, publish) and sees it live within five minutes without a developer. + +### Task 15 (PR 4, after one week): Remove the dual path + +- [ ] Delete `videos-catalog.ts`, the `VIDEOS_SOURCE` switch, `mergeVideos` and its tests, the RSS dependency in `videos.ts` (keep `youtube-feed.ts` only if `/podcast` still uses it, which it does), and the catalog JSON seed after confirming prod is populated. Update `WEB-PAGES.md` and the project memory note. + +--- + +## 9. Testing strategy + +| Layer | What | How | +|---|---|---| +| Pure logic | duration parsing, topic suggestion, `planSync`, mapping | Vitest, no I/O. Every branch in section 3's D5 to D7 has a test. | +| Boundary | YouTube client | fake `fetch`, schema failure, error text never contains the key, batching, page cap | +| Shell | `applyPlan`, `runYoutubeSync` | fake payload, per-item failure isolation | +| Schema | collections | snapshot tests (`schema-surface`), migration applied to two clones and `pg_dump` diffed, run twice | +| Web | mapping, source switch, sitemap, guard | Vitest, plus a regression that a CMS failure throws | +| End to end | the 26 browser checks and axe | Playwright against a local CMS with the backfill applied, in both source modes | +| Real feed and API | live smokes | run the sync once without a key (RSS) against the real channel in a non-prod DB and eyeball the drafts; repeat with a key when one exists | + +The sync is never tested against production data until the dry-run output has been read by a human. + +--- + +## 10. Risks and mitigations + +| Risk | Likelihood | Impact | Mitigation | +|---|---|---|---| +| Hand-written migration drifts from what Payload expects (2026-06-24 incident class) | Medium | High (admin 500) | Task 4 method: push to clone A, transcribe, apply to clone B, `pg_dump` diff empty, second run no-op. Include `payload_locked_documents_rels`. | +| API key leaks into logs or errors | Low | High | Client never forwards underlying error text, test asserts it, key lives only in env and GitHub secrets, key restricted to the YouTube Data API. | +| Quota exhaustion (API mode only) | Very low | Medium | About 3 units per hourly run against 10,000 per day. If it ever trips, set the key empty and the sync falls back to RSS. | +| RSS mode cannot see duration, embeddability or deletions | Certain until a key is added | Low | Editors type the duration, the draft review catches a non-embeddable video, and RSS mode never flags "missing". Adding `YOUTUBE_API_KEY` removes all three gaps with no code change. | +| Sync overwrites editor work | Medium if designed carelessly | High | D6: sync writes only `source` and `syncState`. Tested by `applyPlan` "writes only the source group". | +| A video is deleted or made private on YouTube | Medium | Medium | Flagged `missing`, shown in the admin, not auto-unpublished (editor decides). Only flagged when the listing was complete. | +| Slug changes after publish | Medium | High (lost rankings) | Slug is not synced, backfill preserves all 15, `slugChangeRedirectHook` writes a redirect on edits. | +| CMS down during an ISR revalidation caches an empty page | Medium | High | Task 12: `getCmsVideos` throws; Next keeps the last good page. Tested. | +| Cutover regression visible to visitors | Medium | High | `VIDEOS_SOURCE` switch, parity script, rollback drill, one-week soak before deleting the old path. | +| Custom Studio admin (feat/studio-build) does not know the new collection | Medium | Medium | Before PR 1 merges, check `docs/studio/` for any per-collection registration the Studio build requires, and coordinate order with whoever owns that branch. | +| Sibling sessions edit the same registry files (this checkout is shared) | High | Low | Stage explicit paths only, commit with `-- <pathspec>`, never `git add -A` at the root (memory `concurrent-sessions-share-checkout`). | +| Production build flakiness from the live CMS (520s seen this session) | Medium | Low | Retry the build; do not treat a CMS timeout on unrelated routes as a failure of this change. | + +--- + +## 11. Definition of done + +- [ ] Marketing can add, edit, recategorise and unpublish a video and a topic in the admin, with no developer. +- [ ] A new YouTube upload appears as a `needs-review` draft within an hour, with description, thumbnail frame, summary and suggested topics filled in (and duration too once `YOUTUBE_API_KEY` is set; until then an editor types it). +- [ ] Publishing requires format, topic and summary; the checklist says what is missing. +- [ ] All 15 existing URLs return 200, with identical slugs, titles and topics. +- [ ] `/videos` and `/videos/[slug]` update within seconds of a publish (on-demand revalidation), not after a deploy. +- [ ] Sitemap, search index and IndexNow include videos; unpublished videos 404. +- [ ] `VIDEOS_SOURCE` rollback drilled once. +- [ ] Lint, typecheck, tests, build, `verify:copy`, `verify:page-registry` green; migration proven on two clones. +- [ ] `CLAUDE.md` jobs table, `WEB-PAGES.md`, rollout and deployment docs updated; memory note updated. + +--- + +## 12. Plan 2 outline (separate document, not started) + +| Item | Approach | Needs | +|---|---|---| +| Vimeo or uploaded video | `provider` already exists. Add a `getEmbed(video)` adapter used by `VideoPlayer`, `videoObjectSchema` and the CSP (`frame-src player.vimeo.com`, `img-src i.vimeocdn.com`). Import by pasting a URL (oEmbed, no token) first; API sync only if they upload at volume. For true uploads without Vimeo, Cloudflare Stream is the natural fit. | Decision 3, an account, a token stored via the existing Integrations secrets pattern (`lib/integrations/secrets.ts`) | +| Transcripts | Field exists. Render on the watch page, emit `VideoObject.transcript`. Source: pasted by editors, or generated from the audio later. YouTube's caption download needs the channel owner's OAuth, so it is not part of this. | Product decision | +| Chapters and key moments | Parse timestamp lines (the parser already groups them) into `hasPart` `Clip` entries for Google key moments. | none | +| Video sitemap | `/video-sitemap.xml` with `<video:video>` entries, added to `robots.txt`. | none | +| Like button | Not planned. YouTube offers no embed for it; a real in-page like needs a Google OAuth client and app verification. Decided against on 2026-10-23. | n/a | + +--- + +## Self-review + +**Spec coverage.** Editors change videos and categories without a deploy: Tasks 1 to 3, 9. Automatic import of new uploads with useful metadata: Tasks 5 to 8. "How does it work and is it automatic": section 2 and the Task 8 cadence. Existing URLs preserved: D11, Task 11, Task 14 parity. Vimeo and direct upload: designed for (D2), delivered in Plan 2 with the decision called out. Safety and rollback: D10, section 7, Task 14. + +**Placeholder scan.** Tasks 3, 4 and 8 name the exact files and commands but do not reproduce the roughly 45 one-line registry edits or the roughly 400 lines of generated DDL. That is deliberate: the registry edits are found mechanically with the `rg` command in section 5 against the Advisories commit, and the DDL must be transcribed from a real Payload push (Task 4 Step 2), because inventing it here would be wrong in the way that took the admin down in June. Task 12 Step 1 describes the `videos-utils` refactor rather than printing the diff because the existing 26 tests define it; write them red first. + +**Type consistency.** `SourceSnapshot`, `SyncPlan`, `ExistingVideo`, `TopicRef`, `YoutubeVideo` are defined in Tasks 5 and 6 and used unchanged in Tasks 7 and 8. The web `Video` shape is unchanged except `topics` becoming `{ slug, label }[]` (Task 12 Step 1), and `mapCmsVideo` returns exactly that. `syncState` values match between the collection (Task 2), the planner (Task 6) and the shell (Task 7). The task slug `syncYoutubeVideos` and queue `youtubeVideosSync` are used identically in Task 8's three places. + +**Decisions applied.** Draft-for-review (decision 1) is D5 and Task 7. RSS-now, key-later (decision 2) is D4, D4b, Task 5 Step 5, Task 6's `withKnownDuration`, Task 8's client choice, and the `durationLabel` field. + +**Known limits stated, not hidden.** `autoPublish` cannot actually publish until a format rule is agreed (Task 7 note). Whether `schemaAddons` is required is verified rather than assumed (Task 2 Step 5). The Studio admin interaction is a flagged risk, not a solved one.