diff --git a/.codemap/codemap.html b/.codemap/codemap.html new file mode 100644 index 0000000..23a81a2 --- /dev/null +++ b/.codemap/codemap.html @@ -0,0 +1,634 @@ + + + + + + +Functional Architecture Map + + + +
+
+
+
+

Functional Architecture Map

+
+
+
+ + + + + + + + +
+
+
+ + + + + + +
+
click a module · scroll to pan
+
+ +
+ + + + + diff --git a/.codemap/codemap.md b/.codemap/codemap.md new file mode 100644 index 0000000..9a26823 --- /dev/null +++ b/.codemap/codemap.md @@ -0,0 +1,182 @@ + + +# NiuMa Studio — Functional Module Quality Audit + +> **Interactive view:** [`.codemap/codemap.html`](codemap.html) — per-module scores, findings, LoC, and the dependency graph. This file is the written report. + +**Generated:** 2026-08-24 · **Modules:** 13 · **Size:** 55768 tracked LoC across 132 files + +## Health by layer + +| Layer | Modules | Avg score | +|---|--:|--:| +| 界面 · API | 2 | 62 | +| 业务编排 | 3 | 66 | +| 媒体与 AI 处理 | 4 | 69 | +| 外部执行边界 | 2 | 77 | +| 持久化与运维 | 2 | 70 | + +## Per-module lines of code & score + +_LoC is the representative file/folder per module; folder-level modules overlap and are not additive._ + +### 界面 · API + +| Module | LoC | Score | Tags | +|---|--:|:--|:--| +| Frontend UI | 13,764 | 62 C | bloat, glue, fallback, silent-except, legacy, over-fit | +| API & Runtime | 2,948 | 63 C | fallback, legacy, dual-format, bloat, glue | + +### 业务编排 + +| Module | LoC | Score | Tags | +|---|--:|:--|:--| +| Publish Center | 5,654 | 65 C | god-component, bloat, legacy, dual-format, fallback, silent-except, placeholder, duplication | +| Pipeline & Job Queue | 4,095 | 66 C | fallback, silent-except, legacy, stub, bloat, god-component, glue | +| Task Review & Cut | 3,094 | 68 C | god-component, glue, duplication, fallback, dual-format, legacy | + +### 媒体与 AI 处理 + +| Module | LoC | Score | Tags | +|---|--:|:--|:--| +| AI Selection | 6,031 | 70 C | fallback, legacy, dual-format, placeholder, fake-output, bloat, god-component | +| Subtitle | 3,067 | 64 C | fallback, silent-except, legacy, duplication, bloat, god-component | +| Transcription | 2,276 | 64 C | fallback, silent-except, duplication, god-component | +| Media & Storage | 1,635 | 78 B | fallback, legacy, dual-format, silent-except | + +### 外部执行边界 + +| Module | LoC | Score | Tags | +|---|--:|:--|:--| +| Publishers & Worker | 2,738 | 80 B | legacy, glue, silent-except, fallback | +| Publish Scheduler | 2,362 | 74 C | god-component, bloat, legacy, silent-except | + +### 持久化与运维 + +| Module | LoC | Score | Tags | +|---|--:|:--|:--| +| Ops & Delivery | 4,348 | 68 C | fallback, legacy, dual-format, duplication, bloat, glue, silent-except, monkeypatch | +| SQLite Persistence | 3,756 | 72 C | legacy, dual-format, bloat, god-component, glue | + +## Worst offenders + +- **Frontend UI (62/C)** — app/templates/clip_review.html:154; app/static/js/app.js:103-132: 片段审核空状态的“运行 Codex CLI 分析”仍绑定通用 handleProcessAction;/process/ai 现在返回持久化 job_id 后,该处理器没有等待或轮询 Job,而是在收到排队响应后直接 window.location.reload()。用户会看到仍无候选片段的旧页面,并可能在后台任务未完成时再次点击,造成重复请求或重复计费边界。 +- **API & Runtime (63/C)** — app/routers/publish.py:59: OAuth URL 与 callback 仍捕获 broad Exception,并把 str(exc) 直接放入 HTTP detail 或重定向查询参数;tasks.py、subtitles.py 也存在同类映射。异常文本可能包含本地路径、数据库、FFmpeg 或第三方接口诊断,形成浏览器可见的信息泄漏。 +- **Transcription (64/C)** — app/services/transcript_service.py:1099-1129,1159-1196: faster-whisper 的 model.transcribe 与模型加载没有单次调用或绝对时限;模型/解码器卡住时只能依赖外层 Worker,直接转写路径可能长期占用。 +- **Subtitle (64/C)** — app/services/subtitle_auto_workflow_service.py:42-50,51-108: prepare_task_subtitle_review 先执行 ensure_source_track/ensure_clip_track 并提交 revision/track 副作用,之后才检查活动 Job 与当前 lease;旧 Worker 失租或并发冲突时仍可能留下字幕数据库变更。 +- **Publish Center (65/C)** — app/services/publish_service.py:1: publish_service.py 仍约4776行,集中承担配置、账号、OAuth、文案、封面、队列、历史、OpenCLI兼容和API发布职责,属于真实 God Component,修改影响面较大。 +- **Pipeline & Job Queue (66/C)** — app/services/pipeline_engine.py:1600: _write_json_atomic 只在写临时文件前后调用 require_active_job_lease,最终 temporary_path.replace(path) 以及 _create_schedule 的直接 write_text(1614-1616)不带 owner/lease 条件;租约在最后检查后被接管时,旧 Worker 仍可能覆盖新 Worker 的 metadata/schedule 派生文件。数据库 Job 状态虽已 fencing,但文件产物仍存在跨进程陈旧写入窗口。 +- **Task Review & Cut (68/C)** — app/services/video_cut_workflow_service.py:516-582; app/services/video_cut_workflow_service.py:87-105: process_task_video_cuts 创建切片批次前仍未校验任务必须处于允许切片的状态,也未检查同一任务是否已有其他 queued/running workflow job;_create_cut_run 只排除 CANCELLED,能够把任意非取消任务写成 cutting。同步切片、AI 候选替换和异步切片并发时仍存在过期候选和状态互相覆盖风险。 +- **Ops & Delivery (68/C)** — scripts/migrate_task_dirs_to_project_names.py:211: 任务目录迁移用非 WAL-aware 主库 copy,先移动目录再统一更新提交,无文件补偿,异常会让路径/DB 不一致。 +- **AI Selection (70/C)** — app/services/ai_analysis_workflow_service.py:1273-1320,1322-1360: active run 的 payload clips、clip_candidates 和 analysis.json 分开读取;恢复/返回路径未校验三者的数量、ID、内容及代际一致性,候选账本漂移时仍可能把 active run 当作可信结果继续展示或恢复。 +- **SQLite Persistence (72/C)** — app/db/database.py:614: init_db 仍先执行历史列探测迁移 helper,再执行账本迁移;helper 内部使用 executescript(如 :1074),会隐式提交此前 DML,后续 helper 失败时可能留下部分 Schema/数据变更而无对应 schema_migrations 记录。 + +## All findings + +### HIGH (1) + +- **Ops & Delivery** · `scripts/migrate_task_dirs_to_project_names.py:211` — 任务目录迁移用非 WAL-aware 主库 copy,先移动目录再统一更新提交,无文件补偿,异常会让路径/DB 不一致。 + +### MED (60) + +- **Frontend UI** · `app/templates/clip_review.html:154; app/static/js/app.js:103-132` — 片段审核空状态的“运行 Codex CLI 分析”仍绑定通用 handleProcessAction;/process/ai 现在返回持久化 job_id 后,该处理器没有等待或轮询 Job,而是在收到排队响应后直接 window.location.reload()。用户会看到仍无候选片段的旧页面,并可能在后台任务未完成时再次点击,造成重复请求或重复计费边界。 +- **Frontend UI** · `app/static/js/app.js:437-490; app/templates/clip_review.html:194-198` — AI 结果卡片始终按“AI 分析完成”渲染并显示“去检查并生成切片”,片段审核页只按是否存在 clips 启用生成切片和同步发送中心按钮;前端没有读取或展示 analysis_incomplete、quality_degraded、partial 等结果标记。后端虽可拒绝不合格切片,但页面会把降级/不完整结果呈现为正常结果,用户仍会尝试后续操作。 +- **Frontend UI** · `app/static/js/app.js:1927-1945` — AI 状态轮询只在成功读取后安排下一次定时器;fetch、JSON 解析或临时网络异常会被调用方 catch(() => {}) 静默吞掉,且不会重新调度,导致页面永久停止刷新而无提示。轮询恢复没有退避、重试上限或明确的读取失败状态。 +- **Frontend UI** · `app/static/js/app.js:1805-1822` — waitForAiAnalysisJob 是无限 while 循环,每秒直接 fetch,没有 AbortController、总超时或网络错误重试;一次临时读取异常就会跳到外层“AI 分析失败”,但服务端 Job 仍可能继续执行,前端重新启用按钮,用户可能再次发起请求并误判任务已失败。 +- **Frontend UI** · `app/static/js/app.js:711-718; app/static/js/app.js:1927-1946` — 页面重新打开时只轮询 AI 状态和运行日志;当后台 Job 在该页面生命周期内完成后,轮询停止,但不会自动重新读取 ai-analysis-runs 或渲染最新结果。跨页面/跨进程恢复场景可能显示 100%/已完成,却仍保留旧候选片段和旧历史,必须人工刷新历史或整页刷新。 +- **API & Runtime** · `app/routers/publish.py:59` — OAuth URL 与 callback 仍捕获 broad Exception,并把 str(exc) 直接放入 HTTP detail 或重定向查询参数;tasks.py、subtitles.py 也存在同类映射。异常文本可能包含本地路径、数据库、FFmpeg 或第三方接口诊断,形成浏览器可见的信息泄漏。 +- **API & Runtime** · `app/routers/tasks.py:212` — async process_audio 直接调用同步的 process_task_audio,服务层会执行 FFmpeg/文件与数据库操作;process_video_cuts:392-400 同样直接调用同步切片流程。长请求会阻塞 FastAPI event loop,使轮询、健康检查和其他 API 排队,并增加超时后的重复请求风险。 +- **API & Runtime** · `app/models/task.py:38` — TaskStatus 同时保留 uppercase 自动流程状态与 lowercase legacy 状态(pending_processing、ai_analyzing、completed 等),同一业务状态存在双协议;API、数据库查询和状态转换需要维护两套值,容易出现状态筛选和迁移口径不一致。 +- **API & Runtime** · `app/models/task.py:243` — 批量输入列表缺少统一上限:PublishBatchScheduleUpdate.job_ids/confirmed_schedule、PublishBatchJobCreate.output_clip_ids:287、ClipCandidateBatchUpdate.clips:406 与 PublishBatchTargetUpdate.job_ids 均可任意增长。对应路由会逐项执行数据库、文件或发布操作,缺少请求级资源上限,异常大请求可造成内存和处理时间消耗。 +- **Media & Storage** · `app/services/storage_service.py:693` — 任务媒体删除已写入 manifest 并分阶段移动,但没有跨进程扫描、加载或恢复入口。进程在暂存完成、数据库提交前崩溃会使数据库仍可见任务的原目录已被移走;数据库提交后、finalize 前崩溃则 cleanup_pending 只存在于本次响应,重启后不会自动清理或恢复。 +- **Media & Storage** · `app/services/storage_service.py:362` — resolve_video_file_path 对已存在路径直接返回,不验证路径是否位于允许根目录;get_source_video_path 及多个非 HTTP 调用方复用该入口,数据库中的异常视频/产物路径仍可能被发布、字幕或任务查询流程读取为本地任意文件。HTTP 媒体路由已有更严格的任务目录校验,但共享原语本身仍不安全。 +- **Transcription** · `app/services/transcript_service.py:1099-1129,1159-1196` — faster-whisper 的 model.transcribe 与模型加载没有单次调用或绝对时限;模型/解码器卡住时只能依赖外层 Worker,直接转写路径可能长期占用。 +- **Transcription** · `app/services/transcript_service.py:164-191` — FFmpeg 音频提取只有无进展 watchdog,没有绝对 wall-clock deadline;进程持续输出进度但实际不结束时仍可能无限运行。 +- **Transcription** · `app/services/transcript_service.py:981-989` — 进度文件损坏、读取失败或顶层非 dict 时统一返回空字典,静默丢失故障证据,恢复逻辑无法区分初始状态与持久化损坏。 +- **Transcription** · `app/services/transcript_service.py:67-77,1259-1270` — 活动 provider/model/device/compute_type 仍使用模块级可变全局;同一进程并发转写会互相覆盖运行元数据和进度展示。 +- **Transcription** · `app/services/transcript_service.py:347-549` — 本地与火山远程分片分别复制 chunk、checkpoint、进度和异常收口逻辑,两个实现继续存在维护分叉。 +- **Transcription** · `app/services/transcript_service.py:83-1395` — 单文件仍混合 FFmpeg、faster-whisper、远程 HTTP、幂等 checkpoint、进度文件和 Markdown 生成,约 1400 行,职责边界和回归影响面偏大。 +- **AI Selection** · `app/services/ai_analysis_workflow_service.py:1273-1320,1322-1360` — active run 的 payload clips、clip_candidates 和 analysis.json 分开读取;恢复/返回路径未校验三者的数量、ID、内容及代际一致性,候选账本漂移时仍可能把 active run 当作可信结果继续展示或恢复。 +- **AI Selection** · `app/services/ai/ai_clip_analyzer.py:414-490,597-615` — AI JSON 解析通过别名映射、literal_eval 和默认值补齐缺失标题、摘要、理由、置信度等字段;模型 schema 退化时仍可能被合成为看似完整的候选结果。 +- **AI Selection** · `app/services/ai_analysis_workflow_service.py:1-1470` — 单文件约 1400 行,同时承担 Provider 编排、任务状态、租约、候选替换、文件落盘、历史 run、恢复和 API 返回;职责与测试边界过宽,修改 blast radius 较大。 +- **Task Review & Cut** · `app/services/video_cut_workflow_service.py:516-582; app/services/video_cut_workflow_service.py:87-105` — process_task_video_cuts 创建切片批次前仍未校验任务必须处于允许切片的状态,也未检查同一任务是否已有其他 queued/running workflow job;_create_cut_run 只排除 CANCELLED,能够把任意非取消任务写成 cutting。同步切片、AI 候选替换和异步切片并发时仍存在过期候选和状态互相覆盖风险。 +- **Task Review & Cut** · `app/services/task_lifecycle_service.py:373-406` — 公共状态转换的完成前置条件仍只查询 output_clip 数据库记录,不验证 output_file_path 对应文件真实存在,也不核对当前 active cut_run。残留或损坏的 output_clip 索引可以让任务被标记为 completed,主状态与实际产物不一致。 +- **Task Review & Cut** · `app/services/task_query_service.py:234-272` — 片段总览的 can_cut 仍只依据 enabled_count 和 source_exists,不读取已统一校验的 analysis_incomplete/quality_degraded,也不排除 AI_ANALYZING 等运行状态。后端已阻止质量降级或不完整分析切片时,页面仍会显示可生成切片。 +- **Task Review & Cut** · `app/services/task_query_service.py:204-208; app/services/task_query_service.py:245-304; app/services/task_query_service.py:417-428` — Dashboard、片段总览和系统状态统计只识别 lowercase completed/completed_with_errors,而全自动流水线使用 uppercase COMPLETED 状态,自动完成任务可能漏计或显示为待前置/待检查,继续保留两套状态格式的展示不一致。 +- **Task Review & Cut** · `app/services/task_service.py:687-744` — list_clip_candidates 对每条历史候选直接解析 start_time/end_time 及多个评分字段,单条坏时间或坏数值格式会让整个审核列表抛 ValueError,阻断人工检查和后续修复。 +- **Task Review & Cut** · `app/services/task_service.py:926-960` — _active_outputs_match_enabled_candidates 只比较启用候选 ID、数量和 output_file_path 是否存在,不验证文件内容、大小、可播放性或 source_fingerprint。路径存在但文件损坏或被外部替换时,审核同步可能错误跳过重新切片。 +- **Task Review & Cut** · `app/services/task_service.py:1-101; app/services/task_service.py:1246-1299; app/services/task_query_service.py:89-172` — TaskService 仍约 1,300 行,同时承担任务状态、AI、字幕、转写、切片、数据库写入和页面查询;底部保留多组只转发到 task_query_service 的包装函数,而 task_query_service 又复制 list_output_clips 的路径解析、字幕状态和发布就绪逻辑,核心修改的 blast radius 仍较大。 +- **Subtitle** · `app/services/subtitle_auto_workflow_service.py:42-50,51-108` — prepare_task_subtitle_review 先执行 ensure_source_track/ensure_clip_track 并提交 revision/track 副作用,之后才检查活动 Job 与当前 lease;旧 Worker 失租或并发冲突时仍可能留下字幕数据库变更。 +- **Subtitle** · `app/services/subtitle_data_service.py:1017-1074` — _load_source_cues 对 checksum、JSON 或 segment 错误直接跳过;只要其他 chunk 仍有 cue 就生成部分字幕 revision,损坏转写会被静默降级为缺句结果。 +- **Subtitle** · `app/services/subtitle_data_service.py:1222-1230,236-330` — _sync_dependent_clip_tracks 逐条调用各自事务的 sync_clip_track;源轨已提交后任一切片同步失败,会留下部分 up_to_date、部分 pending_sync 状态。 +- **Subtitle** · `app/services/subtitle_auto_workflow_service.py:513-569` — checkpoint 与 DB 恢复证据只检查 completed/verified、路径 exists 和 is_file,不检查非空、size/fingerprint 或可被 FFprobe 验证;截断或零字节最终文件仍可能被恢复为成功。 +- **Subtitle** · `app/services/subtitle_data_service.py:1222-1230` — 字幕数据层继续集中承担 revision/cue 持久化、导入导出、ASS、波形和 FFprobe,单文件约 1434 行,变更影响面和重复事务路径较大。 +- **Pipeline & Job Queue** · `app/services/pipeline_engine.py:1600` — _write_json_atomic 只在写临时文件前后调用 require_active_job_lease,最终 temporary_path.replace(path) 以及 _create_schedule 的直接 write_text(1614-1616)不带 owner/lease 条件;租约在最后检查后被接管时,旧 Worker 仍可能覆盖新 Worker 的 metadata/schedule 派生文件。数据库 Job 状态虽已 fencing,但文件产物仍存在跨进程陈旧写入窗口。 +- **Pipeline & Job Queue** · `app/services/job_worker.py:234` — WorkflowJobRunner.stop 只等待 worker 线程最多 25 秒;若 terminate_process_tree 持续失败,_run_job_subprocess 会保留 lease 并继续重试,但 stop 返回后应用生命周期仍会继续关闭,线程/子进程可能未结束、lease 也未立即释放,下一进程只能等待租约过期恢复。 +- **Pipeline & Job Queue** · `app/services/job_worker.py:272` — 父 Worker 启动子进程时将 stdout/stderr 都重定向到 DEVNULL(272-276),job_worker_process 仅把异常文本打印到 stderr;子进程实际堆栈、Provider/FFmpeg 错误完全丢失,父进程最终只能记录退出码,重启恢复和人工诊断证据不足。 +- **Pipeline & Job Queue** · `app/services/job_service.py:163` — _row_to_dict 对 payload_json/result_json/checkpoint_json 的 JSONDecodeError 仅 pass,字段保留为原始字符串;job_worker 随后按 dict 调用 payload.get 等操作时会产生不清晰的 AttributeError/失败,损坏队列数据没有明确的不可恢复数据错误状态。 +- **Pipeline & Job Queue** · `app/services/job_worker.py:70` — JOB_TYPE_PUBLISH 仍在 job_service 中声明并可由 create_job 创建,但 execute_job 只有未支持分支(70-71)会失败;这是可入队、可查询却没有执行器的遗留/stub 入口,队列状态语义对调用方不完整。 +- **Pipeline & Job Queue** · `app/services/pipeline_engine.py:88` — PipelineEngine 文件约 1,926 行,仍集中承担步骤编排、租约/取消检查、AutoPipelineCheckpoint 恢复、文件证据、AI/切片/字幕、文案、排期和发布任务创建;与 job_service(约1,171行)共同形成核心 God Component,修改任一步骤的回归半径很大。 +- **Publish Center** · `app/services/publish_service.py:1` — publish_service.py 仍约4776行,集中承担配置、账号、OAuth、文案、封面、队列、历史、OpenCLI兼容和API发布职责,属于真实 God Component,修改影响面较大。 +- **Publish Center** · `app/services/publish_service.py:2968` — 历史查询 SQL 仍先按大写 PUBLISH_HISTORY_STATUSES 过滤,再做 legacy 状态归一化;数据库中的 lowercase/旧状态记录会在归一化前被排除,历史页与统计可能漏数据。 +- **Publish Center** · `app/services/publish_service.py:2173` — sync_task_publish_jobs 中封面生成异常仅写入 cover_error 后继续创建 WAITING 发布任务,未计入 errors;同步接口可能返回 ok,但任务实际缺少封面,后续发送就绪检查才失败。 +- **Publish Center** · `app/services/publish_service.py:2818` — create_batch_publish_jobs 对不存在的 output_clip_id 直接 continue,最后无论缺失多少输入都返回 status=ok,调用方无法区分全部无效、部分成功和完整成功。 +- **Publish Center** · `app/services/publish_readiness.py:167` — PUBLISH_MODES 仍允许 api_publish,但 build_send_readiness 的 resolved_mode 只接受 local_browser/manual_export;API 模式可以创建,却在发送前统一判定 unsupported_publish_mode。 +- **Publish Center** · `app/services/publish_providers.py:141` — BilibiliPublishProvider 完成配置和 access_token 校验后固定抛出 bilibili_provider_pending,属于可创建、可排队但无法执行的真实发布入口,仍是 placeholder/stub。 +- **Publish Center** · `app/services/publish_providers.py:81` — DouyinPublishProvider 将上传和创建作品拆成两个非幂等 HTTP 调用;上传已被平台接受但客户端读取响应时超时或断网,重试可能产生重复作品,未转入人工复核或使用幂等键。 +- **Publish Scheduler** · `app/services/publish_scheduler.py:756-785` — recover_interrupted_jobs 每轮加载全部 PUBLISHING 任务,并逐个同步调用 Worker execution;没有批量上限、并发控制或退避,Worker 超时或不可用时恢复耗时随任务数线性增长并阻塞后续排期。 +- **Publish Scheduler** · `app/services/publish_scheduler.py:283-304` — list_due_jobs 每轮 SELECT 全部 SCHEDULED 记录,再逐条在 Python 中解析并过滤时间,没有 SQL due 条件、分页或批量上限,排期规模增长会增加扫描内存和调度延迟。 +- **Publish Scheduler** · `app/services/publish_scheduler.py:133-1789` — PublishScheduler 约 1,700 行并集中承担排期计算、任务领取、结果写回、Worker 恢复、重试修复、人工复核、批量排期及旧版兼容,状态路径修改的回归半径较大。 +- **Publishers & Worker** · `scripts/publish_host_worker.py:297` — _prior_job_execution_requires_review 会跳过 corrupt 或缺失/无效 identity 的旧 journal;若旧执行已开始上传但身份未持久化,重试可能无法关联并再次提交。browser_opened 仍属于 safe_retry_phases,存在浏览器动作已发生而 journal 尚未推进的重复发布窗口。 +- **SQLite Persistence** · `app/db/database.py:614` — init_db 仍先执行历史列探测迁移 helper,再执行账本迁移;helper 内部使用 executescript(如 :1074),会隐式提交此前 DML,后续 helper 失败时可能留下部分 Schema/数据变更而无对应 schema_migrations 记录。 +- **SQLite Persistence** · `scripts/backup_restore.py:362` — 备份验证仅执行 quick_check、必要表/行数和 manifest 文件哈希校验,未执行 PRAGMA foreign_key_check,也未核对 schema_migrations 版本/checksum;外键损坏或结构账本漂移可能被当作可恢复备份。 +- **SQLite Persistence** · `scripts/backup_restore.py:602` — restore_backup_bundle 在替换数据库及删除 -wal/-shm 前没有活动应用/Worker 互斥或停止检查;运行中的 SQLite 连接可能继续写入旧 inode/WAL,造成恢复后丢写或双数据库视图。 +- **Ops & Delivery** · `.github/workflows/ci.yml:3` — CI 仅监听 master push/PR;feature/docs 直接 push 不即时验证,问题延迟到开 PR。 +- **Ops & Delivery** · `scripts/start_docker_opencli.ps1:24` — 健康检查异常只警告不返回非零,服务失败也可能被调用方视为启动成功。 +- **Ops & Delivery** · `scripts/start.ps1:30` — Demo 覆盖生效前先对正式 E 盘配置运行 doctor,新机/迁移机的隔离 Demo 可能被旧路径阻断。 +- **Ops & Delivery** · `scripts/start.ps1:71` — Worker 启动异常被捕获后继续并报告工作台成功,发布能力不可用时易被误判完整健康。 +- **Ops & Delivery** · `scripts/restore.ps1:27` — 恢复只以健康响应判断 App 运行,StopServices 只停 Compose;native/异常健康进程可能仍持有 DB 时执行替换。 +- **Ops & Delivery** · `scripts/backup.ps1:33` — 备份默认包含 .env 且不加密,仅显式 ExcludeEnv 才排除,包被共享时泄漏本机秘密。 +- **Ops & Delivery** · `tests/test_native_scripts.py:11` — 启停测试主要是源码字符串断言,Windows smoke 不做实际 restore/冲突/清理失败与恢复后健康验证。 +- **Ops & Delivery** · `scripts/seed_demo_data.py:32` — 缺 FFmpeg 时仍插入无媒体路径 Demo 数据并成功退出,数量检查通过但媒体 smoke 不可信。 + +### LOW (24) + +- **Frontend UI** · `app/static/js/app.js:374-386; app/static/js/app.js:721-776` — AI 分析期间禁用的控件集合不包含动态生成的历史结果恢复按钮,也不包含显示/刷新历史按钮;用户仍可在新分析运行时恢复旧结果或刷新历史,与后台新 Job 的最终写回形成前端竞态。后端若拒绝该操作,页面仍会额外发出冲突请求。 +- **API & Runtime** · `app/core/config.py:93` — 新分域 AI_ANALYSIS_REMOTE_* / AI_PUBLISH_REMOTE_* 配置仍通过 _env_first 回退到 AI_REMOTE_*,同时 Settings 继续保留完整旧字段组;兼容来源有文档说明但长期存在来源漂移和配置行为不一致风险。 +- **API & Runtime** · `app/routers/tasks.py:1` — tasks.py 约 537 行、publish.py 约 514 行,包含大量重复的 try/except 与一行式 service 转发;路由、错误映射、同步/异步边界和兼容规则集中在大型 glue 文件中,修改入口时 blast radius 较大。 +- **Media & Storage** · `app/services/storage_service.py:256` — allocate_task_dir_name(reserve=True) 先独占创建目录,再由调用方写入 tasks 记录;进程在两步之间退出会留下没有数据库记录的孤儿任务目录,当前没有启动扫描或清理机制。 +- **Media & Storage** · `app/services/storage_service.py:503` — 上传直接写入最终 source 文件;上传或后续建档失败时的 unlink 异常被静默忽略,进程在上传中途退出也没有跨进程清理清单,可能留下未被任务引用的半成品媒体。 +- **Media & Storage** · `app/services/storage_service.py:842` — 遗留 move_task_directory_to_trash 使用 reserve=False 的存在性检查后分配回收站目录,没有独占预占;并发删除同名任务可能选择相同目标目录并发生移动冲突。当前未发现运行时调用,但兼容入口仍可被恢复使用。 +- **Media & Storage** · `app/services/video_cut_service.py:185` — cut_single_clip 在 FFmpeg 成功前先删除 plan.output_path;若重复执行同一 CutPlan 后发生超时、启动失败或输出切换失败,之前已经有效的成片会先被删除,临时文件原子替换无法保护旧结果。 +- **AI Selection** · `app/services/ai_analysis_workflow_service.py:878-925` — 数据库没有历史 run 时会从 analysis.json 兼容合成 active run,当前只检查 JSON 对象和基础 meta 类型,不验证 checksum、完整 schema、selection_profile、候选字段或文件与数据库代际一致性。 +- **AI Selection** · `app/services/ai/ai_clip_service.py:4-39` — generate_candidate_clips_placeholder 仍返回固定 fake output;当前未发现主流程调用,但误调用会产生伪造候选,属于 legacy/placeholder dead code。 +- **Task Review & Cut** · `app/services/task_lifecycle_service.py:159-164; app/services/task_lifecycle_service.py:182-229` — create_task_record 在数据库 INSERT 前预创建任务目录;若随后数据库写入失败,服务层本身没有补偿删除空目录的路径,直接调用服务函数时仍可能留下孤儿目录。 +- **Subtitle** · `app/services/subtitle_workflow_service.py:240-257` — 遗留 _activate_subtitle_job 仍可无 lease、revision、status 或 task/output_clip 约束直接激活指定记录;当前主渲染路径未调用,但 task_service 仍导入并保留兼容入口。 +- **Pipeline & Job Queue** · `app/services/pipeline_engine.py:1804` — _read_json 在 OSError/JSONDecodeError 时静默返回 fallback;损坏的 metadata、schedule 或配置会被当作空结构继续处理,降低故障可见性和人工恢复依据。 +- **Publish Center** · `app/services/publish_service.py:1610` — 批量查询每一行都调用 _normalize_job;该函数随后构建 readiness 并触发账号/运行态解析,批量页面可能产生逐行重复查询,数据量增长后会放大响应延迟。 +- **Publish Center** · `app/services/publish_providers.py:204` — _post_multipart 使用 read_bytes 将整个视频读入内存,再与 multipart 头部 join;大文件发送峰值内存约为文件大小的两倍,存在资源受限环境下的稳定性风险。 +- **Publish Scheduler** · `app/services/publish_scheduler.py:276-281` — shutdown 会等待后台 Task 完整结束,但没有超时或取消兜底;run_once 被 Worker 长超时、数据库锁或文件操作阻塞时,应用停机可能长时间挂起。 +- **Publish Scheduler** · `app/services/publish_scheduler.py:1840-1845` — queue_snapshot 对非法 scheduled_at 静默 pass,损坏排期会从今日任务视图消失,未产生告警或异常计数。 +- **Publishers & Worker** · `app/services/publishers/browser_runtime.py:315` — screenshot 捕获 Exception 后直接返回空字符串,不记录错误或阶段;发布失败诊断可能静默丢失截图证据。 +- **Publishers & Worker** · `app/services/publishers/page_scripts.py:14` — 通过延迟导入 app.services.publish_service 并调用其私有 DOM 脚本维持旧发布协议,形成 Worker/发布服务间的 legacy glue 耦合。 +- **SQLite Persistence** · `app/db/database.py:811` — workflow_jobs claim 索引仅覆盖 status、next_attempt_at、created_at,未包含 lease_expires_at;租约过期接管/恢复扫描仍需过滤大量记录,规模增长后会退化。 +- **SQLite Persistence** · `app/db/database.py:169` — tasks.status 无 CHECK 约束且默认仍为 pending_video,历史迁移 :1091-1100 继续归一化 lowercase legacy 状态,而 app/models/task.py:38 同时接受新旧两套枚举,数据库层无法保证唯一状态协议。 +- **SQLite Persistence** · `scripts/backup_restore_runtime.py:142` — runtime 入口通过 monkeypatch 替换 backup_restore 的全局函数并重新导出 API;core 与 runtime 两套导入路径的行为可能随新增校验漂移,形成 legacy/dual-format glue。 +- **Ops & Delivery** · `scripts/acceptance.ps1:115` — 验收递归扫描正式任务目录,媒体量大/锁文件会显著拖慢或阻断 release gate。 +- **Ops & Delivery** · `scripts/backup_restore_runtime.py:142` — import 时 monkeypatch backup core,全局行为依赖导入顺序。 +- **Ops & Delivery** · `docs/PORTABLE_SETUP.md:127` — 文档描述与当前 start.ps1 实现不符,旧 Next Steps 又保留兼容入口,启动排障认知漂移。 + +## Cross-cutting themes + +- **本地单体架构是当前可运行的主要原因.** FastAPI + SQLite WAL + 文件产物 + 持久化 Job + Windows Worker 与个人本机规模匹配;不需要微服务化,现有恢复骨架应保留。 +- **P0 数据一致性与迁移入口已经 fail closed.** 测试误删、活动库外键、媒体删除回滚、切片/字幕批次和迁移账本已处理;仍需在受控停机窗口首次正式迁移,并逐步把旧 helper 收编到账本。 +- **Job、发布执行和字幕恢复代际已封口.** Workflow lease token、Publish execution fencing、任务状态转换、取消恢复、切片与字幕原子提交已经完成;旧 Worker/旧 execution 不能覆盖新结果。 +- **P0/P1 运行时稳定线已经完成.** AI 候选/run/任务终态、持久 Job、单元计费账本、不完整结果门禁与本地安全边界已收口;当前可按本机单用户、人工复核范围判定为稳定 V1。 +- **本地安全边界已从隐式信任变为显式 fail closed.** AI/Publish Secret DTO、旧 Provider 结果脱敏、loopback client 校验、Docker 回环绑定、Origin 白名单、作品 URL 和动态 DOM XSS 已收紧;不扩大为多用户权限系统。 +- **复杂度集中而非全项目平均恶化.** publish_service.py、publish-center.js、app.js、publish_scheduler.py、subtitle_data_service.py 和 transcript_service.py 是主要 God Component;应按业务边界渐进拆分。 +- **测试已与活动数据隔离,但 Coverage 与真实故障闭环仍不足.** Pytest 已强制使用进程级 sandbox,P0/P1 失败矩阵持续扩充;Coverage 尚未采集,Playwright 可选跳过和真实平台/进程故障闭环仍需在 P2 补齐。 +- **遗留目录迁移脚本不得用于活动库.** scripts/migrate_task_dirs_to_project_names.py 仍是唯一 Codemap HIGH:它不属于当前运行链路,但缺少 WAL-aware 备份与文件补偿;在 P2 专项替换前必须保持停用。 diff --git a/.codemap/config.json b/.codemap/config.json new file mode 100644 index 0000000..84a8873 --- /dev/null +++ b/.codemap/config.json @@ -0,0 +1,8 @@ +{ + "lang": "zh", + "project": "NiuMa Studio", + "subtitle": "牛马片场 · 本地 AI 高光生产后台工程体检", + "outputDir": ".codemap", + "htmlFile": "codemap.html", + "mdFile": "codemap.md" +} diff --git a/.codemap/modules.json b/.codemap/modules.json new file mode 100644 index 0000000..11f1755 --- /dev/null +++ b/.codemap/modules.json @@ -0,0 +1,1086 @@ +{ + "meta": { + "project": "NiuMa Studio", + "lang": "zh", + "subtitle": "牛马片场 · 本地 AI 高光生产后台工程体检", + "generatedAt": "2026-08-24", + "htmlPath": ".codemap/codemap.html", + "mdPath": ".codemap/codemap.md", + "spineDesc": "用户通过页面/API 创建素材任务,SQLite Job 驱动转写、AI 选片、审核切片和字幕,再由发送中心排期并交给 Windows Chrome Worker,所有状态与审计证据回写 SQLite。", + "tracked_loc": 55768, + "tracked_files": 132, + "rev": "56f4d501087860145412fa6ecc411252ae8e877e" + }, + "excludes": [ + ".venv/", + "__pycache__/", + ".pytest_cache/", + ".ruff_cache/", + ".scannerwork/", + "app/static/vendor/", + "app/static/img/", + "data/", + "tasks/", + "workspace/", + "outputs/", + "backups/", + "acceptance-results/" + ], + "bands": [ + { + "id": "presentation", + "tier": "fe", + "t": "界面 · API", + "d": "浏览器页面、前端交互、FastAPI 路由与运行配置" + }, + { + "id": "wire1", + "wire": true, + "t": "HTTP / JSON / 文件上传" + }, + { + "id": "workflow", + "tier": "be", + "t": "业务编排", + "d": "任务、流水线、发送中心和状态机" + }, + { + "id": "processing", + "tier": "be", + "t": "媒体与 AI 处理", + "d": "转写、AI 选片、切片和字幕" + }, + { + "id": "execution", + "tier": "be", + "t": "外部执行边界", + "d": "调度、Publisher 与 Windows Chrome Worker" + }, + { + "id": "persistence", + "tier": "be", + "t": "持久化与运维", + "d": "SQLite Schema/迁移/备份与本地交付脚本" + } + ], + "spine": [ + "frontend_ui", + "api_runtime", + "pipeline_queue", + "transcription", + "ai_selection", + "task_review_cut", + "subtitle", + "publish_center", + "publish_scheduler", + "publisher_worker", + "persistence" + ], + "reportThemes": [ + [ + "本地单体架构是当前可运行的主要原因", + "FastAPI + SQLite WAL + 文件产物 + 持久化 Job + Windows Worker 与个人本机规模匹配;不需要微服务化,现有恢复骨架应保留。" + ], + [ + "P0 数据一致性与迁移入口已经 fail closed", + "测试误删、活动库外键、媒体删除回滚、切片/字幕批次和迁移账本已处理;仍需在受控停机窗口首次正式迁移,并逐步把旧 helper 收编到账本。" + ], + [ + "Job、发布执行和字幕恢复代际已封口", + "Workflow lease token、Publish execution fencing、任务状态转换、取消恢复、切片与字幕原子提交已经完成;旧 Worker/旧 execution 不能覆盖新结果。" + ], + [ + "P0/P1 运行时稳定线已经完成", + "AI 候选/run/任务终态、持久 Job、单元计费账本、不完整结果门禁与本地安全边界已收口;当前可按本机单用户、人工复核范围判定为稳定 V1。" + ], + [ + "本地安全边界已从隐式信任变为显式 fail closed", + "AI/Publish Secret DTO、旧 Provider 结果脱敏、loopback client 校验、Docker 回环绑定、Origin 白名单、作品 URL 和动态 DOM XSS 已收紧;不扩大为多用户权限系统。" + ], + [ + "复杂度集中而非全项目平均恶化", + "publish_service.py、publish-center.js、app.js、publish_scheduler.py、subtitle_data_service.py 和 transcript_service.py 是主要 God Component;应按业务边界渐进拆分。" + ], + [ + "测试已与活动数据隔离,但 Coverage 与真实故障闭环仍不足", + "Pytest 已强制使用进程级 sandbox,P0/P1 失败矩阵持续扩充;Coverage 尚未采集,Playwright 可选跳过和真实平台/进程故障闭环仍需在 P2 补齐。" + ], + [ + "遗留目录迁移脚本不得用于活动库", + "scripts/migrate_task_dirs_to_project_names.py 仍是唯一 Codemap HIGH:它不属于当前运行链路,但缺少 WAL-aware 备份与文件补偿;在 P2 专项替换前必须保持停用。" + ] + ], + "modules": [ + { + "id": "frontend_ui", + "label": "Frontend UI", + "band": "presentation", + "path": "app/templates + app/static", + "paths": [ + "app/templates/**/*.html", + "app/static/css/styles.css", + "app/static/js/*.js" + ], + "coupling": "high", + "deps": [ + "api_runtime" + ], + "desc": "用 Jinja2、原生 JavaScript 和 CSS 提供任务、审核、字幕与发送中心界面。", + "tests": [ + "tests/test_*browser.py", + "tests/test_group_select_all.py", + "tests/test_subtitle_editor.py" + ], + "loc": 13764, + "contentHash": "2d2166297bd28a802517fdea820e36153f83cdf9799a3e3593a86d7d3b649519", + "score": 62, + "grade": "C", + "tags": [ + "bloat", + "glue", + "fallback", + "silent-except", + "legacy", + "over-fit" + ], + "findings": [ + { + "sev": "MED", + "loc": "app/templates/clip_review.html:154; app/static/js/app.js:103-132", + "text": "片段审核空状态的“运行 Codex CLI 分析”仍绑定通用 handleProcessAction;/process/ai 现在返回持久化 job_id 后,该处理器没有等待或轮询 Job,而是在收到排队响应后直接 window.location.reload()。用户会看到仍无候选片段的旧页面,并可能在后台任务未完成时再次点击,造成重复请求或重复计费边界。" + }, + { + "sev": "MED", + "loc": "app/static/js/app.js:437-490; app/templates/clip_review.html:194-198", + "text": "AI 结果卡片始终按“AI 分析完成”渲染并显示“去检查并生成切片”,片段审核页只按是否存在 clips 启用生成切片和同步发送中心按钮;前端没有读取或展示 analysis_incomplete、quality_degraded、partial 等结果标记。后端虽可拒绝不合格切片,但页面会把降级/不完整结果呈现为正常结果,用户仍会尝试后续操作。" + }, + { + "sev": "MED", + "loc": "app/static/js/app.js:1927-1945", + "text": "AI 状态轮询只在成功读取后安排下一次定时器;fetch、JSON 解析或临时网络异常会被调用方 catch(() => {}) 静默吞掉,且不会重新调度,导致页面永久停止刷新而无提示。轮询恢复没有退避、重试上限或明确的读取失败状态。" + }, + { + "sev": "MED", + "loc": "app/static/js/app.js:1805-1822", + "text": "waitForAiAnalysisJob 是无限 while 循环,每秒直接 fetch,没有 AbortController、总超时或网络错误重试;一次临时读取异常就会跳到外层“AI 分析失败”,但服务端 Job 仍可能继续执行,前端重新启用按钮,用户可能再次发起请求并误判任务已失败。" + }, + { + "sev": "MED", + "loc": "app/static/js/app.js:711-718; app/static/js/app.js:1927-1946", + "text": "页面重新打开时只轮询 AI 状态和运行日志;当后台 Job 在该页面生命周期内完成后,轮询停止,但不会自动重新读取 ai-analysis-runs 或渲染最新结果。跨页面/跨进程恢复场景可能显示 100%/已完成,却仍保留旧候选片段和旧历史,必须人工刷新历史或整页刷新。" + }, + { + "sev": "LOW", + "loc": "app/static/js/app.js:374-386; app/static/js/app.js:721-776", + "text": "AI 分析期间禁用的控件集合不包含动态生成的历史结果恢复按钮,也不包含显示/刷新历史按钮;用户仍可在新分析运行时恢复旧结果或刷新历史,与后台新 Job 的最终写回形成前端竞态。后端若拒绝该操作,页面仍会额外发出冲突请求。" + } + ], + "auditedHash": "2d2166297bd28a802517fdea820e36153f83cdf9799a3e3593a86d7d3b649519", + "auditedAt": "2026-08-25", + "auditedRev": "56f4d501087860145412fa6ecc411252ae8e877e" + }, + { + "id": "api_runtime", + "label": "API & Runtime", + "band": "presentation", + "path": "app/main.py + core/models/routers", + "paths": [ + "app/main.py", + "app/core/**/*.py", + "app/models/**/*.py", + "app/routers/**/*.py" + ], + "coupling": "core", + "deps": [ + "media_storage", + "transcription", + "ai_selection", + "task_review_cut", + "subtitle", + "pipeline_queue", + "publish_center", + "publish_scheduler", + "persistence" + ], + "desc": "加载配置、初始化应用与安全中间件,并把页面/API 请求路由到业务服务。", + "tests": [ + "tests/test_p0_security.py", + "tests/test_publish_api_flow.py", + "tests/test_task_defaults.py" + ], + "loc": 2948, + "contentHash": "be58ee5d147bab5fd5da3046e53a33a2f05b899fbad204785a71e9b12114c622", + "score": 63, + "grade": "C", + "tags": [ + "fallback", + "legacy", + "dual-format", + "bloat", + "glue" + ], + "findings": [ + { + "sev": "MED", + "loc": "app/routers/publish.py:59", + "text": "OAuth URL 与 callback 仍捕获 broad Exception,并把 str(exc) 直接放入 HTTP detail 或重定向查询参数;tasks.py、subtitles.py 也存在同类映射。异常文本可能包含本地路径、数据库、FFmpeg 或第三方接口诊断,形成浏览器可见的信息泄漏。" + }, + { + "sev": "MED", + "loc": "app/routers/tasks.py:212", + "text": "async process_audio 直接调用同步的 process_task_audio,服务层会执行 FFmpeg/文件与数据库操作;process_video_cuts:392-400 同样直接调用同步切片流程。长请求会阻塞 FastAPI event loop,使轮询、健康检查和其他 API 排队,并增加超时后的重复请求风险。" + }, + { + "sev": "MED", + "loc": "app/models/task.py:38", + "text": "TaskStatus 同时保留 uppercase 自动流程状态与 lowercase legacy 状态(pending_processing、ai_analyzing、completed 等),同一业务状态存在双协议;API、数据库查询和状态转换需要维护两套值,容易出现状态筛选和迁移口径不一致。" + }, + { + "sev": "MED", + "loc": "app/models/task.py:243", + "text": "批量输入列表缺少统一上限:PublishBatchScheduleUpdate.job_ids/confirmed_schedule、PublishBatchJobCreate.output_clip_ids:287、ClipCandidateBatchUpdate.clips:406 与 PublishBatchTargetUpdate.job_ids 均可任意增长。对应路由会逐项执行数据库、文件或发布操作,缺少请求级资源上限,异常大请求可造成内存和处理时间消耗。" + }, + { + "sev": "LOW", + "loc": "app/core/config.py:93", + "text": "新分域 AI_ANALYSIS_REMOTE_* / AI_PUBLISH_REMOTE_* 配置仍通过 _env_first 回退到 AI_REMOTE_*,同时 Settings 继续保留完整旧字段组;兼容来源有文档说明但长期存在来源漂移和配置行为不一致风险。" + }, + { + "sev": "LOW", + "loc": "app/routers/tasks.py:1", + "text": "tasks.py 约 537 行、publish.py 约 514 行,包含大量重复的 try/except 与一行式 service 转发;路由、错误映射、同步/异步边界和兼容规则集中在大型 glue 文件中,修改入口时 blast radius 较大。" + } + ], + "auditedHash": "be58ee5d147bab5fd5da3046e53a33a2f05b899fbad204785a71e9b12114c622", + "auditedAt": "2026-08-25", + "auditedRev": "56f4d501087860145412fa6ecc411252ae8e877e" + }, + { + "id": "media_storage", + "label": "Media & Storage", + "band": "processing", + "path": "storage/media/FFmpeg services", + "paths": [ + "app/services/storage_service.py", + "app/services/media_preflight_service.py", + "app/services/managed_process_service.py", + "app/services/video_cut_service.py", + "app/services/audio_reaction_service.py" + ], + "coupling": "high", + "deps": [ + "persistence" + ], + "desc": "管理 E 盘任务目录、路径安全、素材预检、FFmpeg/FFprobe 与受控子进程。", + "tests": [ + "tests/test_media_storage_lifecycle.py", + "tests/test_path_resolution.py", + "tests/test_long_live_foundation.py" + ], + "loc": 1635, + "contentHash": "ec7d7c1c77275b12ac6cad06027256c44aa6a9e9a75e21c277cb6734001772e0", + "score": 78, + "grade": "B", + "tags": [ + "fallback", + "legacy", + "dual-format", + "silent-except" + ], + "findings": [ + { + "sev": "MED", + "loc": "app/services/storage_service.py:693", + "text": "任务媒体删除已写入 manifest 并分阶段移动,但没有跨进程扫描、加载或恢复入口。进程在暂存完成、数据库提交前崩溃会使数据库仍可见任务的原目录已被移走;数据库提交后、finalize 前崩溃则 cleanup_pending 只存在于本次响应,重启后不会自动清理或恢复。" + }, + { + "sev": "MED", + "loc": "app/services/storage_service.py:362", + "text": "resolve_video_file_path 对已存在路径直接返回,不验证路径是否位于允许根目录;get_source_video_path 及多个非 HTTP 调用方复用该入口,数据库中的异常视频/产物路径仍可能被发布、字幕或任务查询流程读取为本地任意文件。HTTP 媒体路由已有更严格的任务目录校验,但共享原语本身仍不安全。" + }, + { + "sev": "LOW", + "loc": "app/services/storage_service.py:256", + "text": "allocate_task_dir_name(reserve=True) 先独占创建目录,再由调用方写入 tasks 记录;进程在两步之间退出会留下没有数据库记录的孤儿任务目录,当前没有启动扫描或清理机制。" + }, + { + "sev": "LOW", + "loc": "app/services/storage_service.py:503", + "text": "上传直接写入最终 source 文件;上传或后续建档失败时的 unlink 异常被静默忽略,进程在上传中途退出也没有跨进程清理清单,可能留下未被任务引用的半成品媒体。" + }, + { + "sev": "LOW", + "loc": "app/services/storage_service.py:842", + "text": "遗留 move_task_directory_to_trash 使用 reserve=False 的存在性检查后分配回收站目录,没有独占预占;并发删除同名任务可能选择相同目标目录并发生移动冲突。当前未发现运行时调用,但兼容入口仍可被恢复使用。" + }, + { + "sev": "LOW", + "loc": "app/services/video_cut_service.py:185", + "text": "cut_single_clip 在 FFmpeg 成功前先删除 plan.output_path;若重复执行同一 CutPlan 后发生超时、启动失败或输出切换失败,之前已经有效的成片会先被删除,临时文件原子替换无法保护旧结果。" + } + ], + "auditedHash": "ec7d7c1c77275b12ac6cad06027256c44aa6a9e9a75e21c277cb6734001772e0", + "auditedAt": "2026-08-25", + "auditedRev": "56f4d501087860145412fa6ecc411252ae8e877e" + }, + { + "id": "transcription", + "label": "Transcription", + "band": "processing", + "path": "transcript services", + "paths": [ + "app/services/transcript_service.py", + "app/services/transcript_workflow_service.py", + "app/services/transcription_checkpoint_service.py" + ], + "coupling": "high", + "deps": [ + "media_storage", + "persistence" + ], + "desc": "提取音频,调用火山引擎或 faster-whisper,并以分块 checkpoint 生成时间戳转写。", + "tests": [ + "tests/test_job_queue.py", + "tests/test_auto_pipeline.py", + "tests/test_long_live_foundation.py" + ], + "loc": 2276, + "contentHash": "0e7b974a289de7b62a22333d90b6ac8e9db2a8118cac505279546d75ea2fe74a", + "score": 64, + "grade": "C", + "tags": [ + "fallback", + "silent-except", + "duplication", + "god-component" + ], + "findings": [ + { + "sev": "MED", + "loc": "app/services/transcript_service.py:1099-1129,1159-1196", + "text": "faster-whisper 的 model.transcribe 与模型加载没有单次调用或绝对时限;模型/解码器卡住时只能依赖外层 Worker,直接转写路径可能长期占用。" + }, + { + "sev": "MED", + "loc": "app/services/transcript_service.py:164-191", + "text": "FFmpeg 音频提取只有无进展 watchdog,没有绝对 wall-clock deadline;进程持续输出进度但实际不结束时仍可能无限运行。" + }, + { + "sev": "MED", + "loc": "app/services/transcript_service.py:981-989", + "text": "进度文件损坏、读取失败或顶层非 dict 时统一返回空字典,静默丢失故障证据,恢复逻辑无法区分初始状态与持久化损坏。" + }, + { + "sev": "MED", + "loc": "app/services/transcript_service.py:67-77,1259-1270", + "text": "活动 provider/model/device/compute_type 仍使用模块级可变全局;同一进程并发转写会互相覆盖运行元数据和进度展示。" + }, + { + "sev": "MED", + "loc": "app/services/transcript_service.py:347-549", + "text": "本地与火山远程分片分别复制 chunk、checkpoint、进度和异常收口逻辑,两个实现继续存在维护分叉。" + }, + { + "sev": "MED", + "loc": "app/services/transcript_service.py:83-1395", + "text": "单文件仍混合 FFmpeg、faster-whisper、远程 HTTP、幂等 checkpoint、进度文件和 Markdown 生成,约 1400 行,职责边界和回归影响面偏大。" + } + ], + "auditedHash": "0e7b974a289de7b62a22333d90b6ac8e9db2a8118cac505279546d75ea2fe74a", + "auditedAt": "2026-08-25", + "auditedRev": "56f4d501087860145412fa6ecc411252ae8e877e" + }, + { + "id": "ai_selection", + "label": "AI Selection", + "band": "processing", + "path": "AI providers and analyzers", + "paths": [ + "app/services/ai/**/*.py", + "app/services/ai_analysis_workflow_service.py", + "app/services/ai_config_service.py", + "app/services/ai_prompt_preset_service.py", + "app/services/ai_clip_service.py", + "app/services/clip_feedback_service.py" + ], + "coupling": "core", + "deps": [ + "transcription", + "persistence" + ], + "desc": "选择 Codex/远程/本地 Provider,解析不稳定 AI 输出并生成通用、综艺或长直播候选片段。", + "tests": [ + "tests/test_ai_json_parsing.py", + "tests/test_codex_cli_provider.py", + "tests/test_long_live_selection.py", + "tests/test_variety_comedy_selection.py" + ], + "loc": 6031, + "contentHash": "60f1d8ab002c31dbec0d955d62707d06502b95f72ee9d12776155772fc01735f", + "score": 70, + "grade": "C", + "tags": [ + "fallback", + "legacy", + "dual-format", + "placeholder", + "fake-output", + "bloat", + "god-component" + ], + "findings": [ + { + "sev": "MED", + "loc": "app/services/ai_analysis_workflow_service.py:1273-1320,1322-1360", + "text": "active run 的 payload clips、clip_candidates 和 analysis.json 分开读取;恢复/返回路径未校验三者的数量、ID、内容及代际一致性,候选账本漂移时仍可能把 active run 当作可信结果继续展示或恢复。" + }, + { + "sev": "LOW", + "loc": "app/services/ai_analysis_workflow_service.py:878-925", + "text": "数据库没有历史 run 时会从 analysis.json 兼容合成 active run,当前只检查 JSON 对象和基础 meta 类型,不验证 checksum、完整 schema、selection_profile、候选字段或文件与数据库代际一致性。" + }, + { + "sev": "MED", + "loc": "app/services/ai/ai_clip_analyzer.py:414-490,597-615", + "text": "AI JSON 解析通过别名映射、literal_eval 和默认值补齐缺失标题、摘要、理由、置信度等字段;模型 schema 退化时仍可能被合成为看似完整的候选结果。" + }, + { + "sev": "MED", + "loc": "app/services/ai_analysis_workflow_service.py:1-1470", + "text": "单文件约 1400 行,同时承担 Provider 编排、任务状态、租约、候选替换、文件落盘、历史 run、恢复和 API 返回;职责与测试边界过宽,修改 blast radius 较大。" + }, + { + "sev": "LOW", + "loc": "app/services/ai/ai_clip_service.py:4-39", + "text": "generate_candidate_clips_placeholder 仍返回固定 fake output;当前未发现主流程调用,但误调用会产生伪造候选,属于 legacy/placeholder dead code。" + } + ], + "auditedHash": "60f1d8ab002c31dbec0d955d62707d06502b95f72ee9d12776155772fc01735f", + "auditedAt": "2026-08-25", + "auditedRev": "56f4d501087860145412fa6ecc411252ae8e877e" + }, + { + "id": "task_review_cut", + "label": "Task Review & Cut", + "band": "workflow", + "path": "task and cut workflow services", + "paths": [ + "app/services/task_lifecycle_service.py", + "app/services/task_service.py", + "app/services/task_query_service.py", + "app/services/task_log_service.py", + "app/services/video_cut_workflow_service.py" + ], + "coupling": "core", + "deps": [ + "media_storage", + "ai_selection", + "publish_center", + "persistence" + ], + "desc": "维护任务状态与视图,保存候选审核选择,生成切片版本并同步发送中心。", + "tests": [ + "tests/test_split_services.py", + "tests/test_task_query_service.py", + "tests/test_clip_review_publish_sync.py", + "tests/test_versioning_rollback.py" + ], + "loc": 3094, + "contentHash": "b61dc1321ba5e1e1b14052163be7ca8bf64850b6f493dc996594dbe89977b657", + "score": 68, + "grade": "C", + "tags": [ + "god-component", + "glue", + "duplication", + "fallback", + "dual-format", + "legacy" + ], + "findings": [ + { + "sev": "MED", + "loc": "app/services/video_cut_workflow_service.py:516-582; app/services/video_cut_workflow_service.py:87-105", + "text": "process_task_video_cuts 创建切片批次前仍未校验任务必须处于允许切片的状态,也未检查同一任务是否已有其他 queued/running workflow job;_create_cut_run 只排除 CANCELLED,能够把任意非取消任务写成 cutting。同步切片、AI 候选替换和异步切片并发时仍存在过期候选和状态互相覆盖风险。" + }, + { + "sev": "MED", + "loc": "app/services/task_lifecycle_service.py:373-406", + "text": "公共状态转换的完成前置条件仍只查询 output_clip 数据库记录,不验证 output_file_path 对应文件真实存在,也不核对当前 active cut_run。残留或损坏的 output_clip 索引可以让任务被标记为 completed,主状态与实际产物不一致。" + }, + { + "sev": "MED", + "loc": "app/services/task_query_service.py:234-272", + "text": "片段总览的 can_cut 仍只依据 enabled_count 和 source_exists,不读取已统一校验的 analysis_incomplete/quality_degraded,也不排除 AI_ANALYZING 等运行状态。后端已阻止质量降级或不完整分析切片时,页面仍会显示可生成切片。" + }, + { + "sev": "MED", + "loc": "app/services/task_query_service.py:204-208; app/services/task_query_service.py:245-304; app/services/task_query_service.py:417-428", + "text": "Dashboard、片段总览和系统状态统计只识别 lowercase completed/completed_with_errors,而全自动流水线使用 uppercase COMPLETED 状态,自动完成任务可能漏计或显示为待前置/待检查,继续保留两套状态格式的展示不一致。" + }, + { + "sev": "MED", + "loc": "app/services/task_service.py:687-744", + "text": "list_clip_candidates 对每条历史候选直接解析 start_time/end_time 及多个评分字段,单条坏时间或坏数值格式会让整个审核列表抛 ValueError,阻断人工检查和后续修复。" + }, + { + "sev": "MED", + "loc": "app/services/task_service.py:926-960", + "text": "_active_outputs_match_enabled_candidates 只比较启用候选 ID、数量和 output_file_path 是否存在,不验证文件内容、大小、可播放性或 source_fingerprint。路径存在但文件损坏或被外部替换时,审核同步可能错误跳过重新切片。" + }, + { + "sev": "MED", + "loc": "app/services/task_service.py:1-101; app/services/task_service.py:1246-1299; app/services/task_query_service.py:89-172", + "text": "TaskService 仍约 1,300 行,同时承担任务状态、AI、字幕、转写、切片、数据库写入和页面查询;底部保留多组只转发到 task_query_service 的包装函数,而 task_query_service 又复制 list_output_clips 的路径解析、字幕状态和发布就绪逻辑,核心修改的 blast radius 仍较大。" + }, + { + "sev": "LOW", + "loc": "app/services/task_lifecycle_service.py:159-164; app/services/task_lifecycle_service.py:182-229", + "text": "create_task_record 在数据库 INSERT 前预创建任务目录;若随后数据库写入失败,服务层本身没有补偿删除空目录的路径,直接调用服务函数时仍可能留下孤儿目录。" + } + ], + "auditedHash": "b61dc1321ba5e1e1b14052163be7ca8bf64850b6f493dc996594dbe89977b657", + "auditedAt": "2026-08-25", + "auditedRev": "56f4d501087860145412fa6ecc411252ae8e877e" + }, + { + "id": "subtitle", + "label": "Subtitle", + "band": "processing", + "path": "subtitle services", + "paths": [ + "app/services/subtitle_*.py" + ], + "coupling": "high", + "deps": [ + "media_storage", + "task_review_cut", + "ai_selection", + "persistence" + ], + "desc": "维护字幕轨、不可变 revision、cue、AI 建议、渲染验证与自动流水线交付决定。", + "tests": [ + "tests/test_subtitle_editor.py", + "tests/test_subtitle_auto_workflow.py" + ], + "loc": 3067, + "contentHash": "bdbb1edf69efb3c42f96fd9cff31dfa9e173ba56a2345d191b73531c5ad8f37f", + "score": 64, + "grade": "C", + "tags": [ + "fallback", + "silent-except", + "legacy", + "duplication", + "bloat", + "god-component" + ], + "findings": [ + { + "sev": "MED", + "loc": "app/services/subtitle_auto_workflow_service.py:42-50,51-108", + "text": "prepare_task_subtitle_review 先执行 ensure_source_track/ensure_clip_track 并提交 revision/track 副作用,之后才检查活动 Job 与当前 lease;旧 Worker 失租或并发冲突时仍可能留下字幕数据库变更。" + }, + { + "sev": "MED", + "loc": "app/services/subtitle_data_service.py:1017-1074", + "text": "_load_source_cues 对 checksum、JSON 或 segment 错误直接跳过;只要其他 chunk 仍有 cue 就生成部分字幕 revision,损坏转写会被静默降级为缺句结果。" + }, + { + "sev": "MED", + "loc": "app/services/subtitle_data_service.py:1222-1230,236-330", + "text": "_sync_dependent_clip_tracks 逐条调用各自事务的 sync_clip_track;源轨已提交后任一切片同步失败,会留下部分 up_to_date、部分 pending_sync 状态。" + }, + { + "sev": "MED", + "loc": "app/services/subtitle_auto_workflow_service.py:513-569", + "text": "checkpoint 与 DB 恢复证据只检查 completed/verified、路径 exists 和 is_file,不检查非空、size/fingerprint 或可被 FFprobe 验证;截断或零字节最终文件仍可能被恢复为成功。" + }, + { + "sev": "LOW", + "loc": "app/services/subtitle_workflow_service.py:240-257", + "text": "遗留 _activate_subtitle_job 仍可无 lease、revision、status 或 task/output_clip 约束直接激活指定记录;当前主渲染路径未调用,但 task_service 仍导入并保留兼容入口。" + }, + { + "sev": "MED", + "loc": "app/services/subtitle_data_service.py:1222-1230", + "text": "字幕数据层继续集中承担 revision/cue 持久化、导入导出、ASS、波形和 FFprobe,单文件约 1434 行,变更影响面和重复事务路径较大。" + } + ], + "auditedHash": "bdbb1edf69efb3c42f96fd9cff31dfa9e173ba56a2345d191b73531c5ad8f37f", + "auditedAt": "2026-08-25", + "auditedRev": "56f4d501087860145412fa6ecc411252ae8e877e" + }, + { + "id": "pipeline_queue", + "label": "Pipeline & Job Queue", + "band": "workflow", + "path": "pipeline and workflow job services", + "paths": [ + "app/services/pipeline_engine.py", + "app/services/job_service.py", + "app/services/job_worker.py", + "app/services/job_worker_process.py", + "app/services/auto_publish_service.py", + "app/services/metadata_generator.py" + ], + "coupling": "core", + "deps": [ + "transcription", + "ai_selection", + "task_review_cut", + "subtitle", + "publish_center", + "persistence" + ], + "desc": "用 SQLite Job、Lease、Heartbeat、Checkpoint 和子进程串联长任务与自动流水线。", + "tests": [ + "tests/test_auto_pipeline.py", + "tests/test_job_queue.py", + "tests/test_subtitle_auto_workflow.py" + ], + "loc": 4095, + "contentHash": "bd2e33520b473b9dff951c8facb733411fa879987643082a891b8bf2d5694a2e", + "score": 66, + "grade": "C", + "tags": [ + "fallback", + "silent-except", + "legacy", + "stub", + "bloat", + "god-component", + "glue" + ], + "findings": [ + { + "sev": "MED", + "loc": "app/services/pipeline_engine.py:1600", + "text": "_write_json_atomic 只在写临时文件前后调用 require_active_job_lease,最终 temporary_path.replace(path) 以及 _create_schedule 的直接 write_text(1614-1616)不带 owner/lease 条件;租约在最后检查后被接管时,旧 Worker 仍可能覆盖新 Worker 的 metadata/schedule 派生文件。数据库 Job 状态虽已 fencing,但文件产物仍存在跨进程陈旧写入窗口。" + }, + { + "sev": "MED", + "loc": "app/services/job_worker.py:234", + "text": "WorkflowJobRunner.stop 只等待 worker 线程最多 25 秒;若 terminate_process_tree 持续失败,_run_job_subprocess 会保留 lease 并继续重试,但 stop 返回后应用生命周期仍会继续关闭,线程/子进程可能未结束、lease 也未立即释放,下一进程只能等待租约过期恢复。" + }, + { + "sev": "MED", + "loc": "app/services/job_worker.py:272", + "text": "父 Worker 启动子进程时将 stdout/stderr 都重定向到 DEVNULL(272-276),job_worker_process 仅把异常文本打印到 stderr;子进程实际堆栈、Provider/FFmpeg 错误完全丢失,父进程最终只能记录退出码,重启恢复和人工诊断证据不足。" + }, + { + "sev": "MED", + "loc": "app/services/job_service.py:163", + "text": "_row_to_dict 对 payload_json/result_json/checkpoint_json 的 JSONDecodeError 仅 pass,字段保留为原始字符串;job_worker 随后按 dict 调用 payload.get 等操作时会产生不清晰的 AttributeError/失败,损坏队列数据没有明确的不可恢复数据错误状态。" + }, + { + "sev": "MED", + "loc": "app/services/job_worker.py:70", + "text": "JOB_TYPE_PUBLISH 仍在 job_service 中声明并可由 create_job 创建,但 execute_job 只有未支持分支(70-71)会失败;这是可入队、可查询却没有执行器的遗留/stub 入口,队列状态语义对调用方不完整。" + }, + { + "sev": "MED", + "loc": "app/services/pipeline_engine.py:88", + "text": "PipelineEngine 文件约 1,926 行,仍集中承担步骤编排、租约/取消检查、AutoPipelineCheckpoint 恢复、文件证据、AI/切片/字幕、文案、排期和发布任务创建;与 job_service(约1,171行)共同形成核心 God Component,修改任一步骤的回归半径很大。" + }, + { + "sev": "LOW", + "loc": "app/services/pipeline_engine.py:1804", + "text": "_read_json 在 OSError/JSONDecodeError 时静默返回 fallback;损坏的 metadata、schedule 或配置会被当作空结构继续处理,降低故障可见性和人工恢复依据。" + } + ], + "auditedHash": "bd2e33520b473b9dff951c8facb733411fa879987643082a891b8bf2d5694a2e", + "auditedAt": "2026-08-25", + "auditedRev": "56f4d501087860145412fa6ecc411252ae8e877e" + }, + { + "id": "publish_center", + "label": "Publish Center", + "band": "workflow", + "path": "publish content preparation services", + "paths": [ + "app/services/publish_service.py", + "app/services/publish_copy_rules.py", + "app/services/publish_domain.py", + "app/services/publish_adapters.py", + "app/services/publish_readiness.py", + "app/services/publish_providers.py" + ], + "coupling": "core", + "deps": [ + "task_review_cut", + "subtitle", + "publish_scheduler", + "publisher_worker", + "persistence" + ], + "desc": "准备标题、简介、话题、封面、账号和发布草稿,并保留历史兼容执行入口。", + "tests": [ + "tests/test_publish_copy_rules.py", + "tests/test_publish_readiness.py", + "tests/test_publish_task_linkage.py", + "tests/test_publish_history.py" + ], + "loc": 5654, + "contentHash": "7a13cff1d83735a2396e98709b58d9a9ec55b3aef5c32b91373a3a5de4cac13f", + "score": 65, + "grade": "C", + "tags": [ + "god-component", + "bloat", + "legacy", + "dual-format", + "fallback", + "silent-except", + "placeholder", + "duplication" + ], + "findings": [ + { + "sev": "MED", + "loc": "app/services/publish_service.py:1", + "text": "publish_service.py 仍约4776行,集中承担配置、账号、OAuth、文案、封面、队列、历史、OpenCLI兼容和API发布职责,属于真实 God Component,修改影响面较大。" + }, + { + "sev": "MED", + "loc": "app/services/publish_service.py:2968", + "text": "历史查询 SQL 仍先按大写 PUBLISH_HISTORY_STATUSES 过滤,再做 legacy 状态归一化;数据库中的 lowercase/旧状态记录会在归一化前被排除,历史页与统计可能漏数据。" + }, + { + "sev": "MED", + "loc": "app/services/publish_service.py:2173", + "text": "sync_task_publish_jobs 中封面生成异常仅写入 cover_error 后继续创建 WAITING 发布任务,未计入 errors;同步接口可能返回 ok,但任务实际缺少封面,后续发送就绪检查才失败。" + }, + { + "sev": "MED", + "loc": "app/services/publish_service.py:2818", + "text": "create_batch_publish_jobs 对不存在的 output_clip_id 直接 continue,最后无论缺失多少输入都返回 status=ok,调用方无法区分全部无效、部分成功和完整成功。" + }, + { + "sev": "MED", + "loc": "app/services/publish_readiness.py:167", + "text": "PUBLISH_MODES 仍允许 api_publish,但 build_send_readiness 的 resolved_mode 只接受 local_browser/manual_export;API 模式可以创建,却在发送前统一判定 unsupported_publish_mode。" + }, + { + "sev": "MED", + "loc": "app/services/publish_providers.py:141", + "text": "BilibiliPublishProvider 完成配置和 access_token 校验后固定抛出 bilibili_provider_pending,属于可创建、可排队但无法执行的真实发布入口,仍是 placeholder/stub。" + }, + { + "sev": "MED", + "loc": "app/services/publish_providers.py:81", + "text": "DouyinPublishProvider 将上传和创建作品拆成两个非幂等 HTTP 调用;上传已被平台接受但客户端读取响应时超时或断网,重试可能产生重复作品,未转入人工复核或使用幂等键。" + }, + { + "sev": "LOW", + "loc": "app/services/publish_service.py:1610", + "text": "批量查询每一行都调用 _normalize_job;该函数随后构建 readiness 并触发账号/运行态解析,批量页面可能产生逐行重复查询,数据量增长后会放大响应延迟。" + }, + { + "sev": "LOW", + "loc": "app/services/publish_providers.py:204", + "text": "_post_multipart 使用 read_bytes 将整个视频读入内存,再与 multipart 头部 join;大文件发送峰值内存约为文件大小的两倍,存在资源受限环境下的稳定性风险。" + } + ], + "auditedHash": "7a13cff1d83735a2396e98709b58d9a9ec55b3aef5c32b91373a3a5de4cac13f", + "auditedAt": "2026-08-25", + "auditedRev": "56f4d501087860145412fa6ecc411252ae8e877e" + }, + { + "id": "publish_scheduler", + "label": "Publish Scheduler", + "band": "execution", + "path": "scheduler/repository/executor", + "paths": [ + "app/publish_scheduler.py", + "app/services/publish_scheduler.py", + "app/services/publish_repository.py", + "app/services/publish_executor.py", + "app/services/publish_time.py" + ], + "coupling": "core", + "deps": [ + "publish_center", + "publisher_worker", + "persistence" + ], + "desc": "计算排期、原子领取发布任务、重试、恢复不确定执行并维护发布状态机。", + "tests": [ + "tests/test_publish_scheduler.py", + "tests/test_publish_scheduler_state_machine.py", + "tests/test_publish_fencing.py", + "tests/test_publish_timezones.py" + ], + "loc": 2362, + "contentHash": "7e0f1ec4494c65899b8f29f572c60be557c92140e767f74419bf81b361f10d05", + "score": 74, + "grade": "C", + "tags": [ + "god-component", + "bloat", + "legacy", + "silent-except" + ], + "findings": [ + { + "sev": "MED", + "loc": "app/services/publish_scheduler.py:756-785", + "text": "recover_interrupted_jobs 每轮加载全部 PUBLISHING 任务,并逐个同步调用 Worker execution;没有批量上限、并发控制或退避,Worker 超时或不可用时恢复耗时随任务数线性增长并阻塞后续排期。" + }, + { + "sev": "MED", + "loc": "app/services/publish_scheduler.py:283-304", + "text": "list_due_jobs 每轮 SELECT 全部 SCHEDULED 记录,再逐条在 Python 中解析并过滤时间,没有 SQL due 条件、分页或批量上限,排期规模增长会增加扫描内存和调度延迟。" + }, + { + "sev": "MED", + "loc": "app/services/publish_scheduler.py:133-1789", + "text": "PublishScheduler 约 1,700 行并集中承担排期计算、任务领取、结果写回、Worker 恢复、重试修复、人工复核、批量排期及旧版兼容,状态路径修改的回归半径较大。" + }, + { + "sev": "LOW", + "loc": "app/services/publish_scheduler.py:276-281", + "text": "shutdown 会等待后台 Task 完整结束,但没有超时或取消兜底;run_once 被 Worker 长超时、数据库锁或文件操作阻塞时,应用停机可能长时间挂起。" + }, + { + "sev": "LOW", + "loc": "app/services/publish_scheduler.py:1840-1845", + "text": "queue_snapshot 对非法 scheduled_at 静默 pass,损坏排期会从今日任务视图消失,未产生告警或异常计数。" + } + ], + "auditedHash": "7e0f1ec4494c65899b8f29f572c60be557c92140e767f74419bf81b361f10d05", + "auditedAt": "2026-08-25", + "auditedRev": "56f4d501087860145412fa6ecc411252ae8e877e" + }, + { + "id": "publisher_worker", + "label": "Publishers & Worker", + "band": "execution", + "path": "publisher registry and Windows worker", + "paths": [ + "app/services/publishers/**/*.py", + "scripts/publish_host_worker.py", + "scripts/opencli_host_bridge.py" + ], + "coupling": "high", + "deps": [ + "media_storage", + "persistence" + ], + "desc": "通过 Publisher Registry 和 Windows Chrome Worker 执行抖音/B站投稿、导出与兼容模式。", + "tests": [ + "tests/test_publisher_registry.py", + "tests/test_local_browser_publishers.py", + "tests/test_page_script_publishers.py", + "tests/test_publish_worker_client.py", + "tests/test_publish_fencing.py" + ], + "loc": 2738, + "contentHash": "13f332520f2fc5d872bd52b2dc645804524e005dbedd0990f787c72ee8fbafb4", + "score": 80, + "grade": "B", + "tags": [ + "legacy", + "glue", + "silent-except", + "fallback" + ], + "findings": [ + { + "sev": "MED", + "loc": "scripts/publish_host_worker.py:297", + "text": "_prior_job_execution_requires_review 会跳过 corrupt 或缺失/无效 identity 的旧 journal;若旧执行已开始上传但身份未持久化,重试可能无法关联并再次提交。browser_opened 仍属于 safe_retry_phases,存在浏览器动作已发生而 journal 尚未推进的重复发布窗口。" + }, + { + "sev": "LOW", + "loc": "app/services/publishers/browser_runtime.py:315", + "text": "screenshot 捕获 Exception 后直接返回空字符串,不记录错误或阶段;发布失败诊断可能静默丢失截图证据。" + }, + { + "sev": "LOW", + "loc": "app/services/publishers/page_scripts.py:14", + "text": "通过延迟导入 app.services.publish_service 并调用其私有 DOM 脚本维持旧发布协议,形成 Worker/发布服务间的 legacy glue 耦合。" + } + ], + "auditedHash": "13f332520f2fc5d872bd52b2dc645804524e005dbedd0990f787c72ee8fbafb4", + "auditedAt": "2026-08-25", + "auditedRev": "56f4d501087860145412fa6ecc411252ae8e877e" + }, + { + "id": "persistence", + "label": "SQLite Persistence", + "band": "persistence", + "path": "app/db + backup services", + "paths": [ + "app/db/**/*.py", + "app/services/database_backup_service.py", + "scripts/backup_restore.py", + "scripts/backup_restore_runtime.py", + "scripts/cleanup_database_backups.py", + "scripts/repair_foreign_key_integrity.py" + ], + "coupling": "core", + "deps": [], + "desc": "创建 SQLite Schema、执行启动时兼容迁移和索引,并提供 WAL 安全备份恢复。", + "tests": [ + "tests/test_backup_restore.py", + "tests/test_backup_restore_wal.py", + "tests/test_database_backup_service.py", + "tests/test_p1_1_db_performance.py" + ], + "loc": 3756, + "contentHash": "9ad2d3c8874bf07b1938c8f9784fcce5b45e686ccddf0a4e056b2744299e213c", + "score": 72, + "grade": "C", + "tags": [ + "legacy", + "dual-format", + "bloat", + "god-component", + "glue" + ], + "findings": [ + { + "sev": "MED", + "loc": "app/db/database.py:614", + "text": "init_db 仍先执行历史列探测迁移 helper,再执行账本迁移;helper 内部使用 executescript(如 :1074),会隐式提交此前 DML,后续 helper 失败时可能留下部分 Schema/数据变更而无对应 schema_migrations 记录。" + }, + { + "sev": "MED", + "loc": "scripts/backup_restore.py:362", + "text": "备份验证仅执行 quick_check、必要表/行数和 manifest 文件哈希校验,未执行 PRAGMA foreign_key_check,也未核对 schema_migrations 版本/checksum;外键损坏或结构账本漂移可能被当作可恢复备份。" + }, + { + "sev": "MED", + "loc": "scripts/backup_restore.py:602", + "text": "restore_backup_bundle 在替换数据库及删除 -wal/-shm 前没有活动应用/Worker 互斥或停止检查;运行中的 SQLite 连接可能继续写入旧 inode/WAL,造成恢复后丢写或双数据库视图。" + }, + { + "sev": "LOW", + "loc": "app/db/database.py:811", + "text": "workflow_jobs claim 索引仅覆盖 status、next_attempt_at、created_at,未包含 lease_expires_at;租约过期接管/恢复扫描仍需过滤大量记录,规模增长后会退化。" + }, + { + "sev": "LOW", + "loc": "app/db/database.py:169", + "text": "tasks.status 无 CHECK 约束且默认仍为 pending_video,历史迁移 :1091-1100 继续归一化 lowercase legacy 状态,而 app/models/task.py:38 同时接受新旧两套枚举,数据库层无法保证唯一状态协议。" + }, + { + "sev": "LOW", + "loc": "scripts/backup_restore_runtime.py:142", + "text": "runtime 入口通过 monkeypatch 替换 backup_restore 的全局函数并重新导出 API;core 与 runtime 两套导入路径的行为可能随新增校验漂移,形成 legacy/dual-format glue。" + } + ], + "auditedHash": "9ad2d3c8874bf07b1938c8f9784fcce5b45e686ccddf0a4e056b2744299e213c", + "auditedAt": "2026-08-25", + "auditedRev": "4d30e346b2f75ca8c55a3fdb2fb836eaa0328e78" + }, + { + "id": "ops_delivery", + "label": "Ops & Delivery", + "band": "persistence", + "path": "Docker/PowerShell/CI", + "paths": [ + "Dockerfile", + "docker-compose*.yml", + ".github/**/*.yml", + "scripts/**/*.ps1", + "scripts/doctor.ps1", + "scripts/seed_demo_data.py", + "scripts/purge_deleted_task_media.py", + "scripts/migrate_task_dirs_to_project_names.py" + ], + "coupling": "med", + "deps": [ + "api_runtime", + "publisher_worker", + "persistence" + ], + "desc": "提供 Windows/Docker 启停、诊断、CI、发布门禁、Demo 与维护脚本。", + "tests": [ + "tests/test_native_scripts.py", + "tests/test_release_version.py", + "tests/test_publish_worker_autostart.py" + ], + "loc": 4348, + "contentHash": "cedfdb41e1d9fb89d361d1f4071b696bc0d1584e9e35db556c5a9f51c513ab2a", + "score": 68, + "grade": "C", + "tags": [ + "fallback", + "legacy", + "dual-format", + "duplication", + "bloat", + "glue", + "silent-except", + "monkeypatch" + ], + "findings": [ + { + "sev": "MED", + "loc": ".github/workflows/ci.yml:3", + "text": "CI 仅监听 master push/PR;feature/docs 直接 push 不即时验证,问题延迟到开 PR。" + }, + { + "sev": "MED", + "loc": "scripts/start_docker_opencli.ps1:24", + "text": "健康检查异常只警告不返回非零,服务失败也可能被调用方视为启动成功。" + }, + { + "sev": "MED", + "loc": "scripts/start.ps1:30", + "text": "Demo 覆盖生效前先对正式 E 盘配置运行 doctor,新机/迁移机的隔离 Demo 可能被旧路径阻断。" + }, + { + "sev": "MED", + "loc": "scripts/start.ps1:71", + "text": "Worker 启动异常被捕获后继续并报告工作台成功,发布能力不可用时易被误判完整健康。" + }, + { + "sev": "HIGH", + "loc": "scripts/migrate_task_dirs_to_project_names.py:211", + "text": "任务目录迁移用非 WAL-aware 主库 copy,先移动目录再统一更新提交,无文件补偿,异常会让路径/DB 不一致。" + }, + { + "sev": "MED", + "loc": "scripts/restore.ps1:27", + "text": "恢复只以健康响应判断 App 运行,StopServices 只停 Compose;native/异常健康进程可能仍持有 DB 时执行替换。" + }, + { + "sev": "MED", + "loc": "scripts/backup.ps1:33", + "text": "备份默认包含 .env 且不加密,仅显式 ExcludeEnv 才排除,包被共享时泄漏本机秘密。" + }, + { + "sev": "MED", + "loc": "tests/test_native_scripts.py:11", + "text": "启停测试主要是源码字符串断言,Windows smoke 不做实际 restore/冲突/清理失败与恢复后健康验证。" + }, + { + "sev": "MED", + "loc": "scripts/seed_demo_data.py:32", + "text": "缺 FFmpeg 时仍插入无媒体路径 Demo 数据并成功退出,数量检查通过但媒体 smoke 不可信。" + }, + { + "sev": "LOW", + "loc": "scripts/acceptance.ps1:115", + "text": "验收递归扫描正式任务目录,媒体量大/锁文件会显著拖慢或阻断 release gate。" + }, + { + "sev": "LOW", + "loc": "scripts/backup_restore_runtime.py:142", + "text": "import 时 monkeypatch backup core,全局行为依赖导入顺序。" + }, + { + "sev": "LOW", + "loc": "docs/PORTABLE_SETUP.md:127", + "text": "文档描述与当前 start.ps1 实现不符,旧 Next Steps 又保留兼容入口,启动排障认知漂移。" + } + ], + "auditedHash": "cedfdb41e1d9fb89d361d1f4071b696bc0d1584e9e35db556c5a9f51c513ab2a", + "auditedAt": "2026-08-25", + "auditedRev": "56f4d501087860145412fa6ecc411252ae8e877e" + } + ] +} \ No newline at end of file diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index fad157f..6a3d596 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -151,13 +151,14 @@ jobs: run: | docker run --detach \ --name niuma-studio-ci \ - --publish 8001:8001 \ + --publish 127.0.0.1:8001:8001 \ --env DEMO_MODE=true \ --env DATA_DIR=/app/ci-data \ --env DATABASE_PATH=/app/ci-data/workflow.sqlite3 \ --env STORAGE_ROOT=/app/ci-tasks \ --env TASKS_DIR=/app/ci-tasks \ --env UPLOAD_TEMP_DIR=/app/ci-tasks/_temp \ + --env NIUMA_TRUST_DOCKER_LOOPBACK_PROXY=true \ --env PUBLISH_SCHEDULER_ENABLED=false \ niuma-studio:ci diff --git a/.gitignore b/.gitignore index 15d015f..458ac2f 100644 --- a/.gitignore +++ b/.gitignore @@ -48,3 +48,9 @@ tasks/* .DS_Store Thumbs.db + +# 本地工程审计产物 +.scannerwork/ +.coverage +coverage.xml +test-results.xml diff --git a/AUDIT_TASK.md b/AUDIT_TASK.md new file mode 100644 index 0000000..8c7323e --- /dev/null +++ b/AUDIT_TASK.md @@ -0,0 +1,53 @@ +# NiuMa Studio 全面工程体检任务书 + +## 背景 + +当前项目为 Windows 本地 AI 高光生产后台 V2.1.0。本轮联合使用 Codemap、Code Overhaul 与本地 SonarQube,回答项目为什么能运行、哪些部分可靠、哪些部分存在高风险技术债,以及应按什么顺序低风险整改。 + +## 目标 + +- 建立当前项目的功能模块图、依赖图、核心数据流与状态流。 +- 审计架构、业务、代码、数据、稳定性、测试、安全、性能、可观测性和维护性。 +- 运行现有安全测试、Lint、覆盖率与 SonarQube 静态扫描。 +- 交叉验证三方结果并生成 `PROJECT_AUDIT.md`。 + +## 允许修改范围 + +- `.codemap/` 下的 Codemap 状态、配置和生成报告。 +- `sonar-project.properties` 等仅用于本轮静态扫描的工程配置。 +- `AUDIT_TASK.md`、`PROJECT_AUDIT.md`。 +- 按项目规则追加 `DEVELOPMENT_LOG.md`、`NEXT_STEPS.md` 的审计记录。 + +## 禁止修改范围 + +- `app/`、`scripts/`、`tests/`、`prompts/` 中的生产逻辑和测试逻辑。 +- 数据库 Schema、Migration、真实数据、浏览器登录态、发布队列和任务文件。 +- `.env`、Cookie、Token、API Key、浏览器数据及任何秘密内容。 +- 依赖版本、运行时行为、外部平台状态与远端系统。 + +## 已确定实现要求 + +- Codemap 以功能模块为单位,核心/高耦合模块独立评分,小型叶子模块可同一子任务内分别评分。 +- Code Overhaul 使用 FULL AUDIT 模式,不在各章节暂停整改。 +- SonarQube 优先复用现有本地容器;无法获得的指标必须明确写为“未取得”,不得估算成 Sonar 指标。 +- Dead/Legacy/Mock/兼容代码只列出,不删除。 +- 自动化测试不得连接真实 AI Provider 或触发真实投稿。 + +## 验收标准 + +- `PROJECT_AUDIT.md` 包含用户要求的全部章节、100 分健康度、Sonar 指标、P0-P3、Top 10、删除候选、暂不修改区和可独立回滚的整改路线图。 +- 关键发现有 `file:line` 证据;Sonar 事实与人工审查结论明确区分。 +- 记录实际测试命令、退出码、通过/失败/跳过数量和覆盖率。 +- 最终 `git diff` 不包含生产代码、数据库、日志、缓存或敏感信息。 + +## 测试与扫描命令 + +- `.venv\Scripts\python.exe -m pytest` +- `.venv\Scripts\python.exe -m ruff check app tests scripts/seed_demo_data.py scripts/backup_restore.py scripts/backup_restore_runtime.py` +- `.venv\Scripts\python.exe -m coverage run -m pytest`(仅在 Coverage 可用时) +- Sonar Scanner(使用本地 SonarQube 与隔离的扫描配置) + +## 返回格式 + +- 子代理返回精确命令、退出码、耗时与 `file:line` 证据。 +- 主代理统一输出 `PROJECT_AUDIT.md`,并说明审计产物、测试结果、Git 提交、推送与 PR 状态。 diff --git a/DEVELOPMENT_LOG.md b/DEVELOPMENT_LOG.md index 4028a04..ef3799e 100644 --- a/DEVELOPMENT_LOG.md +++ b/DEVELOPMENT_LOG.md @@ -1,5 +1,222 @@ # Development Log +## 2026-08-28 2.1 集成 PR 与 Docker 冒烟修复 + +- 将当前线性领先 `master` 的 26 个提交完整保留到 `codex/integrate-v2.1-stable`,新增独立空白清理提交并创建顶层集成 PR #60;不 rebase、不 squash,也不自动合并。 +- 本地门禁通过:Python Compileall、Ruff、`801 passed`、4 个 JavaScript 文件、20 个 PowerShell 脚本和三套 Docker Compose 配置均正常。 +- 首轮 GitHub CI 的 Linux 全量测试和 Windows 主机冒烟通过;Docker 镜像构建、启动、健康检查与 Demo 数据写入通过,但主要页面因 CI 端口参数未满足 P1.5 本机访问门禁而返回 403。 +- Docker smoke 改为仅绑定 `127.0.0.1:8001:8001`,并显式设置 `NIUMA_TRUST_DOCKER_LOOPBACK_PROXY=true`;只修正 CI 宿主机到容器的回环识别,不放宽正式环境的远程访问保护。 + +## 2026-08-26 工作台每日任务柱状图精简 + +- 移除工作台“本周每日新增任务”柱状图,不用其他装饰模块填补空间;首页直接从四项核心指标进入最近任务列表。 +- 本周统计上下文由 `weekly_chart` 精简为 `weekly_summary`,只保留本周新增总数和日期范围;上海时区、周一边界和历史时间格式兼容逻辑保持不变。 +- 同步删除每日计数、柱高、当天高亮、柱状图响应式样式和专用动效;统计条补充底部留白,桌面与窄屏继续保持完整卡片间距。 +- 本轮不修改数据库、任务状态机、发布口径、AI Provider 或真实抖音/B站发送链路。 +- 定向 Dashboard / 性能 / 动效回归 `40 passed`,完整回归 `799 passed`;Python Compileall、JavaScript 语法、Ruff 和 `git diff --check` 均通过,10 条提示均为既有弃用警告。 +- 8001 Web/Scheduler 进程使用 `-SkipWorker` 安全重启,8765 Windows 发布 Worker 保持原 PID;`/health`、Scheduler 和首页均正常。浏览器桌面与 390px 窄屏验收通过,无横向溢出、无控制台 warning/error,未触发真实投稿。 + +## 2026-08-26 全站克制动效与减少动态效果支持 + +- 调研 Motion、AutoAnimate、Animate.css、AOS、GSAP 和 View Transitions 后,按当前原生 HTML/CSS/JavaScript + Jinja 架构实现本地轻量动效层;没有引入 npm、CDN、React/Vue 或第三方运行时依赖。 +- 工作台四项核心统计首次出现时播放一次短动效;新建任务只在长直播设置展开和文件选择确认时反馈;任务详情只在进度、阶段或数值真实变化时反馈;片段审核、字幕和发送中心复用进度、弹窗、抽屉、标签和主动新增内容动效。 +- 任务列表、完整转写、系统静态状态、运行日志、字幕虚拟行、波形、视频、42 格日历和 5 秒发布记录刷新不添加批量入场或循环动画,避免为凑数量造成闪烁、错位和阅读干扰。 +- `prefers-reduced-motion` 同时关闭非必要 CSS 动画/过渡和 JavaScript 平滑滚动;脚本失败或禁用时不隐藏任何业务内容,动画不改变按钮状态、请求或页面语义。 +- 实时文本观察器只在文本值确实变化时高亮,不会因 3/5 秒轮询重复闪烁,也不覆盖状态标签原有背景色。 +- 窄屏顺带补齐片段审核网格和卡片的 `min-width: 0` 约束,390px 验收横向溢出由 49px 收口为 0;其他检查页面也无整页横向溢出。 +- 本轮仅修改呈现与可访问性,不触发 AI、转写、切片、字幕烧录、封面生成、排期、数据库写入或真实平台发布。 +- 动效与关联前端定向回归 `62 passed`、完整回归 `799 passed`;Python Compileall、三份 JavaScript 语法检查、Ruff 和 `git diff --check` 均通过,10 条提示均为既有弃用告警。 + +## 2026-08-26 发送中心 AI 文案操作分层 + +- 明确区分“同步遗漏切片”和“AI 重写文案”:前者继续固定使用 `use_ai=false`,只为遗漏成片补建发布草稿与默认封面;后者只修改已存在草稿的标题、话题和简介。 +- 原页面顶栏“补充缺失任务”移入内容准备的维护区并改名为“同步遗漏切片”,维护区明确标注“不调用 AI、不修改已有文案”;账号管理继续留在页面顶栏。 +- 单条 AI 操作移动到发布文案字段上方并命名为“AI 重写本条文案”;勾选后的批量入口改为“AI 重写已选文案”,切换到排期计划或执行记录后隐藏。 +- 移除 `/publish` 页面加载时静默调用旧草稿全局 AI 升级的行为;后端兼容接口继续保留,但 AI 文案只在用户明确点击单条或已选批量入口后执行。 +- 发布任务、调度器、Publisher、账号、排期及执行记录状态机均未修改。 +- 定向发送中心回归 `52 passed`、完整回归 `794 passed`;Python Compileall、JavaScript 语法、Ruff 和 `git diff --check` 均通过,测试使用隔离数据与 Mock,未调用真实 AI、封面生成、排期或平台发送。 + +## 2026-08-26 片段审核累计统计精简 + +- `/clips` 顶部统计从“待 AI 分析、待检查、可生成切片、已完成、异常任务”五项过程指标,精简为“累计审核任务、已通过视频、已完成任务”三项累计结果。 +- “累计审核任务”按当前未删除任务中已有未删除候选片段的任务数计算,表示已经进入候选片段审核流程;不把自动流程写入的 `reviewed` 字段误称为纯人工审核。 +- “已通过视频”按当前未删除任务下启用且未删除的候选片段数计算,卡片说明明确为“当前启用的视频片段”。 +- “已完成任务”兼容 `completed`、`completed_with_errors` 和历史大写 `COMPLETED`,并排除已删除任务;审核队列的状态徽标使用同一套规范化口径,不再把历史大写完成状态误显示为“待检查”。 +- 三项统计使用独立三列布局,不改变字幕工作台继续复用的五列统计样式;窄屏仍按现有响应式规则切换为单列。 +- 当前活动库按新口径显示 `20 / 163 / 20`;定向统计与性能回归 `35 passed`、完整回归 `793 passed`,Python Compileall、Ruff 和 `git diff --check` 均通过,未触发真实 AI、切片或平台发布。 + +## 2026-08-26 任务详情实时进度与日志统一 + +- `/api/tasks/{task_id}/live-status` 扩展为任务详情唯一实时快照:同一次响应返回主任务状态、活动 Workflow Job、转写细分进度、最新运行日志、候选/输出数量和发布任务汇总。 +- 手动任务不再被 `auto_mode` 条件排除;手动转写、手动 AI、后台切片及全自动流程都能依据活动 Job 或运行状态持续每 3 秒局部刷新,切回页面时立即补取一次。 +- 状态卡把“总流程进度”和“当前操作进度”分开显示;转写的分段百分比、后台 Job 的真实进度和说明不再与固定任务阶段百分比混为一谈。 +- 补齐全自动流程内部小写 `transcribing / ai_analyzing / cutting` 的步骤映射,避免时间线短暂错误退回“任务创建”。 +- 已完成处理但仍有排期/发送任务时,状态概览继续读取每条有效切片的最新发布状态;当前任务 20 显示托管发布 `9/12`、当前发布进度 `75%` 和总流程 `97%`,而不是笼统的“已完成 100%”。失败或需复核会把最后一步显示为警告。 +- AI 专用轮询只更新 AI 区域自身进度,运行日志固定由统一快照刷新;新增请求去重,避免多个轮询响应互相覆盖。 +- 定向回归 `40 passed`、完整回归 `793 passed`;Ruff、Python Compileall、JavaScript 语法和 `git diff --check` 均通过。未调用真实 AI、火山转写、FFmpeg 或抖音/B站发布。 +- 8001 Web 进程已在保留 8765 Windows Worker 的情况下安全重启;`/health`、Scheduler 和 Worker 均健康,活动库 `quick_check=ok`、`foreign_key_check=0`。浏览器实测快照时间从 `14:58:11` 自动更新到 `14:58:14`,桌面与 390px 窄屏状态卡无横向溢出。 + +## 2026-08-26 工作台周统计与发布口径修正 + +- 工作台顶部改为单一“本周任务概览”面板,核心指标精简为本周新增、已切片、待推送和失败任务;原“待处理、待检查、待加字幕、工作流程、今日处理焦点”已移除。 +- “本周新增任务”的周边界显式按 `Asia/Shanghai` 计算,兼容 UTC ISO、`Z` 结尾和旧的无时区时间记录。 +- “已切片任务”改为按有效完成切片反推任务数,同时展示完成切片总数,不再漏算自动流程的 `COMPLETED` 任务。 +- “待推送任务”不再复用全部完成切片数;系统按每个有效切片的最新 `publish_jobs` 状态统计,`PUBLISHED / EXPORTED / CANCELLED` 排除,没有记录或仍处于等待、排期、发送中、失败、需复核的切片继续计入。 +- 当前活动页从错误的“待推送 163”修正为“待推送任务 4,涉及 11 条待发送或复核切片”;本周新增 1、已切片任务 20、有效切片 163、失败任务 0。 +- 8001 Web 进程已在保留 8765 Windows Worker 的情况下安全重启;重启前后 `/health=ok`、`worker_available=true`。浏览器桌面与 390px 窄屏验收通过,无整页横向溢出、无控制台 warning/error,未点击或触发真实发布操作。 +- 完整验收通过:`790 passed`;Ruff、Python Compileall、`node --check app/static/js/app.js` 与 `git diff --check` 均通过。现有 10 条第三方 / Pydantic 弃用警告不属于本次回归。 + +## 2026-08-25 任务 20 状态恢复与新建流程精简 + +- 已确认任务 `20 - E1829 110128 舒淇,安以轩P29-29` 的第 2 次切片批次、12 条 active 成片和发送中心 12/12 关联完整,原“失败”来自完成后重复触发 AI 分析并尝试删除仍被切片引用的候选记录。 +- 修复前 SQLite Online Backup 为 `backups/workflow-before-task20-repair-20260824-233721.sqlite3`;只把任务 `3210d91ee1fb` 恢复为 `completed / 100%` 并清空误报错误,候选、切片、Workflow Job、AI run 和发布任务 ID 前后完全一致,修复依据已写入任务日志。 +- 手动 AI 入口新增原子重入保护:活跃全自动 Job、已有 active 切片或已有发送中心记录时返回 HTTP 409;候选替换在同一事务内先检查 `output_clip` 引用,改为可理解的业务冲突,不再暴露 SQLite 外键错误,也不会把已完成任务降级为失败。 +- 新建任务只保留本机视频上传;任务名称历史候选最多显示最近 5 个未删除且去重后的名称,JSON 已有文件创建入口、目录浏览 API 和 NAS 页面能力已移除。 +- `source_type`、`nas_file_path` 物理列继续保留兼容;账本迁移 `20260824_02_task_upload_only` 在写入前创建 SQLite Online Backup,将旧来源归一到 `original_video_path + upload`,不删除外部视频。 +- “每小时高光密度”和候选总上限仅在长直播高光模式显示和提交;通用与康熙模式不读取页面参数,服务端固定使用兼容默认值。 +- 新增 `POST /api/subtitles/tasks/{task_id}/skip-to-review`;明确跳过字幕后进入片段审核,不再自动续跑文案和发送中心。审核保存并完整关联后才标记完成;关联不完整时保留审核状态与错误。 +- 本轮未修改 AI Provider、Prompt、模型、候选算法或真实发布逻辑,也没有重跑任务 20 的 AI、切片或发布任务。 +- 活动库迁移备份为 `data/backups/workflow-before-task-upload-only-20260825-000702-586803-d14c09b5.sqlite3`;迁移后旧来源记录为 0,`integrity_check=ok`、`foreign_key_check=0`。完整测试 `672 passed`,Ruff、Python Compileall、JavaScript 语法和浏览器烟测通过;8001、Scheduler 与 8765 Windows Worker 健康。 + +## 2026-08-24 稳定 V1 P1.3c 数据库迁移账本与唯一索引 Fail-Closed + +- 新增 `schema_migrations` 账本;每条正式新迁移记录稳定 version、name、checksum 和完成时间,同版本定义漂移时拒绝启动。 +- 迁移执行使用 `BEGIN IMMEDIATE`,Schema 修改、不变量验证和账本写入同事务提交;失败会回滚且不会伪造成功记录,两个初始化进程共享一条迁移记录。 +- 发布活动任务唯一约束迁移到版本化 `uq_publish_jobs_active_clip_platform_mode_v2`;先检查 `DRAFT/WAITING/SCHEDULED/PUBLISHING/NEED_REVIEW` 重复组,再创建并验证新版索引,最后才删除旧索引。 +- 唯一索引缺失、定义漂移、旧索引残留、重复活动记录或 SQLite 索引错误均明确阻止启动;不再执行 `except sqlite3.Error: pass` 后继续无约束运行。 +- 历史库执行 P1.3c 写入前先使用 SQLite Online Backup 创建可移植快照;异常/缺列账本也先进入备份路径,再由账本结构验证拒绝迁移。 +- 新增 12 个隔离回归,覆盖重复初始化、并发迁移、checksum/Schema 漂移、异常账本、备份、失败历史兼容、活动重复数据、旧索引回滚保护和普通索引错误传播。 +- 正式 `workflow.sqlite3` 本轮只读预检:尚无 ledger、旧唯一索引仍存在、活动重复组为 0;未重启服务、未写入正式库,也未调用 AI、FFmpeg、Chrome、抖音或 B站。 +- 最终完整隔离测试 `663 passed`,Ruff、Python Compileall 与 `git diff --check` 全部通过。Codemap 独立复评将 `SQLite Persistence` 从 `64/C` 提升到 `73/C`,原 HIGH fail-open 与无账本 finding 已移除。 +- 诚实边界:启动期旧版列探测/`executescript` 仍是 pre-ledger compatibility,不能把全部历史迁移宣称为原子;备份恢复包的 FK/ledger/关键索引校验和运行中恢复闸门留待独立后续轮次。 + +## 2026-08-24 稳定 V1 P1.3b 自动流水线断点恢复 + +- 新增 `auto_pipeline_step_v1` 版本化 checkpoint;每一步记录 `running/succeeded/failed`、连续完成前缀、输入基线和紧凑产物证据,不保存 Secret、Prompt、完整 AI payload 或大对象。 +- checkpoint 写入继续使用 Workflow Job owner + lease token fencing;传入 `job_id` 却没有当前有效 lease 时,在状态更新、handler 和文件副作用之前直接拒绝执行。 +- Worker 重启/租约接管后会验证转写 Markdown、AI active run 与候选集合、选片 JSON、active cut run、字幕审核状态、文案/排期 JSON 和发布草稿,再从第一个未确认步骤继续。 +- AI、转写、切片、准备素材、字幕、文案和排期的 running 步骤支持产物 reconciliation;证据完整时补记成功,输入变化或证据不足时重做,不把旧产物当作新结果。 +- 切片恢复绑定 `cut_run_id`、启用候选、当前任务受控目录、非空文件、size 与首尾 fingerprint;空文件、错 run、路径越界或内容变化均 fail closed。 +- AI 恢复同时核对分析文件、数据库 `analysis_payload_json`、active run 和候选 `clip_key`;转写或候选输入变化时拒绝复用,避免恢复过期结果。 +- 发布草稿恢复绑定 schedule 的切片/平台、草稿字段、允许状态、视频路径/指纹和封面 hash;旧 Worker 取消本轮草稿时还必须满足当前 Job owner/token 的同事务 `EXISTS` 条件。 +- `/auto-retry` 现在在 `BEGIN IMMEDIATE` 中原位重排最新失败/取消 Job,保留其 checkpoint;不会另建空 Job 后重复调用 AI、FFmpeg 或发布草稿。直接 Job retry 仍保持同一语义。 +- 自动流水线 checkpoint 与字幕渲染 Job 的旧 `{completed: ...}` checkpoint 按 Job 类型隔离;没有修改数据库 Schema、活动库、AI/FFmpeg 参数或真实发布逻辑。 +- 定向回归覆盖崩溃后重领、失败 Job 原位重试、旧 token、无 lease、输入变化、损坏/未知 checkpoint、空切片/错 run、发布草稿篡改、取消竞态和测试清理隔离;最终完整测试 `651 passed`,Ruff 与 Python Compileall 全部通过。Codemap 独立复评将 `Pipeline & Job Queue` 从 `72/C` 提升到 `88/B`,剩余低风险项是发布创建步骤依赖幂等去重而非专用 reconcile,以及尚无真实进程重启级测试。 +- 诚实边界:Provider 返回前进程崩溃且没有落下可验证 run/文件时仍可能再次计费;SQLite 与外部 AI/FFmpeg 无法构成严格 exactly-once,后续 P1.4 继续处理超时与调用幂等。 + +## 2026-08-24 稳定 V1 P1.3a 任务状态与切片原子性 + +- `PATCH /api/tasks/{task_id}/status` 改为显式允许表 + `BEGIN IMMEDIATE` 条件更新;空任务不能直接标记完成,并发状态变化不会被旧请求覆盖。所有内部状态写入同时拒绝已永久删除任务。 +- 自动流水线在每个步骤开始前、处理完成后和最终 READY 写入前检查取消;取消后任务稳定进入 `CANCELLED`,Job 进入 `cancelled`,重试仍可从流水线起点恢复。 +- 自动流水线成功后稳定停在 `READY_TO_PUBLISH`,不再在同一次 run 内立刻覆盖为 `COMPLETED`;这表示内容已准备好,但仍等待用户在发送中心人工确认。 +- Task 步骤状态、切片批次和最终 READY 写回全部绑定当前 Workflow Job 的 owner、token、未过期 lease 与取消标记;旧 Worker 或锁等待期间过期的执行不能覆盖新状态。 +- 公开 Task 取消会在同一 SQLite 事务中请求活跃自动 Job 停止;READY 后取消只清理 `provider_response.workflow_job_id` 明确属于本轮且尚未发布的记录,不会误伤其他排期。 +- `running + cancel_requested` 在子进程退出或 lease 过期时会收敛为 `cancelled`,不再永久卡在运行态。 +- 任务汇总文件写入改为非遮蔽降级:磁盘或源文件异常只补充日志,不会覆盖原始业务失败或取消原因。 +- `cut_run` 编号分配放入 `BEGIN IMMEDIATE`;每个批次使用 `run_序号_id` 独立输出目录,避免并发/重试覆盖同名视频。 +- 一个批次的全部 `output_clip`、旧版本停用和新版本激活在同一个 SQLite 事务内提交;中途失败整批回滚并保留旧 active,较老批次即使最后完成也不能覆盖已完成的较新批次。 +- `cut_run` 创建与 Task=`cutting` 同事务,Task 终态只允许从当前 `cutting` CAS 写入;用户或其他流程刚更新的状态不会被晚到的切片结果覆盖。 +- Pytest 增加 session 级隔离数据库初始化,测试文件单独或换顺序执行时不再依赖其他文件先调用 `init_db()`。 +- 本轮不修改数据库 Schema、活动数据、AI Provider、FFmpeg 参数或真实发布流程;没有触发 AI、Chrome、抖音或 B站。 +- 最终隔离验收通过:完整测试 `637 passed`;Ruff 和 Python Compileall 全部通过,业务代码与文档的 whitespace 检查通过。Codemap 复评为 `Task Review & Cut 78/B`、`Pipeline & Job Queue 72/C`、`API & Runtime 48/D`、`SQLite Persistence 64/C`。 +- API/Persistence 分数下降不是本轮状态修复回归:独立复评删除了已修的状态跳跃/deleted 发现,同时把原始 Secret 读取、可选鉴权、同步重任务阻塞 async 路由、唯一索引重建失败静默吞掉和无版本迁移重新按 HIGH/MED 计分;这些保留到 P1.3b~P1.5。 + +## 2026-08-24 稳定 V1 P1B.2 发布执行代际与 Worker 幂等 + +- Scheduler 每次领取发布任务都会保留唯一 `execution_id`;平台结果、阶段、成功、导出、失败、人工复核和安全重排队全部要求命中当前 `PUBLISHING + execution_id`。旧执行写回会返回 `skipped`,同一事务中的结果和事件一并回滚。 +- `retry_failed` 与旧任务安全修复把源状态复核、活跃替代任务检查、克隆和事件写入放入同一个 `BEGIN IMMEDIATE`;并发点击只会产生一个替代任务。排期修改同时校验原状态和 `updated_at`,不能把已领取或被并发编辑的任务覆盖回 `SCHEDULED`。 +- 发送内容、平台目标和 AI 文案重新生成使用微秒级 `updated_at` 乐观并发版本;AI 慢调用返回时若任务已被人工修改或进入 `PUBLISHING`,旧结果会被拒绝,不覆盖真实投稿采用的文案。 +- Worker 查询失败不再根据数据库旧阶段自动重排队;HTTP 超时后若 Worker 仍持有 execution 锁,任务保持 `PUBLISHING` 等待明确终态。只有 Worker 可查询、锁已释放且确认仍停在上传前阶段时,才允许安全重试。 +- Windows Worker 对 job、execution 和账号同时使用进程内互斥与操作系统级跨进程文件锁;进程崩溃后锁由操作系统自动释放,不再依赖“读 PID 后删除锁文件”的竞态回收。同 execution 的终态直接重放既有结果,身份冲突、损坏/不一致 journal、上传后中断和并行 execution 全部 fail closed。 +- `job_id`、`execution_id`、`account_id`、浏览器 profile/artifact 路径和本地发布包目录增加 Windows/路径安全校验;手工发布 URL 改为解析 hostname,损坏风险 JSON 会进入人工复核而不是当作无风险。 +- Provider 结果、journal 和发布事件统一脱敏 `accessToken`、`refresh-token`、`session_token`、`csrfToken`、`clientSecret` 等常见变体;发布日志写入失败会留下受控错误日志,不再静默消失。 +- Scheduler 后台 Task 由应用保存引用;关闭时停止新扫描并等待当前轮结束,避免停机时丢弃进行中的恢复/发布状态处理。 +- 独立验收通过:发布定向 `144 passed`,完整测试 `610 passed`;Ruff、Python Compileall 和 `git diff --check` 全部通过。测试未修改工作区,未调用真实 AI、Chrome、远程 Worker、抖音或 B站。 +- Codemap 独立复评:`Publish Scheduler 62→71(C)`、`Publishers & Worker 62→76(B)`;`Publish Center` 的本轮并发旧写已修,但原始 Secret 响应、God Service 与旧 API Provider 风险未处理,仍为 `52(D)`。 +- 本轮没有修改数据库 Schema 或活动数据。P1B.1 的 `workflow_jobs.lease_token` 活动库迁移与正式服务重启仍需在单独的备份/无活动 Job 窗口执行。 + +## 2026-08-24 稳定 V1 P1B.1 Workflow Job 执行代际隔离 + +- `workflow_jobs` 新增可空 `lease_token`;每次领取任务都生成新的随机 token,并与 `lease_owner` 一起构成当前执行代际。旧数据库迁移前会先创建 SQLite Online Backup。 +- 进度、checkpoint、心跳、完成、失败、取消和释放租约全部增加 owner + token 条件;旧 Worker 在租约过期或任务被接管后写回会抛出 `JobLeaseLostError`,不能覆盖新执行。 +- 独立子进程启动前验证任务仍为 `running`、owner/token 一致且租约未过期;父 Worker 发现任务已换代时会终止旧子进程,旧子进程退出也不能把新执行标成失败。 +- `claim_next_job` 的选取和 claim 写入合并进同一个 `BEGIN IMMEDIATE` 事务;达到最大尝试次数但租约仍有效的任务不再被其他 Worker提前判为失败。 +- Pipeline、转写和字幕深层写回通过执行上下文继承领取时捕获的 token,不再重新从数据库读取“当前 owner”冒充旧执行身份。 +- 转写分块 checkpoint 的复用、成功、失败和完成写入,以及最终 `transcript.md` 原子替换,也会重新验证当前 Job token;远端请求返回时若任务已被接管,旧执行不能覆盖新 checkpoint 或 Markdown。 +- 字幕父进程清理临时文件时,在同一 `BEGIN IMMEDIATE` 内验证 owner/token、更新从属记录并删除精确 `.part` 文件;旧 token 不会改字幕记录或删除新执行文件。 +- 新增 `tests/test_job_fencing.py`,覆盖旧代 progress/checkpoint/heartbeat/终态/release 全部拒绝、过期子进程禁止启动、有效 max-attempt Worker 保持运行、过期上限失败和 retry token 失效。 +- 隔离验收通过:P1B.1 定向 `92 passed`,完整测试 `546 passed`;全项目 Ruff、Python Compileall 和差异检查通过。未迁移活动数据库、未调用真实 AI 或真实投稿;正式库应用迁移与真实重启烟测将在 P1B.2 完成后一并执行。 + +## 2026-08-24 稳定 V1 P0 数据安全整改 + +- Pytest 启动时改用每进程独立的系统临时 sandbox,并无条件隔离数据库、任务目录、上传临时目录和发布包目录;即使外部 `DATABASE_PATH` 指向活动库,测试也不会连接或清理真实数据。 +- `test_task_query_service` 的整表清理增加第二道 fail-closed 路径校验:数据库不在本次 pytest sandbox 或文件名不是 `test_workflow.sqlite3` 时立即中止。 +- 任务永久删除改为两阶段:托管媒体先原子移动到同卷隔离区并写 manifest,数据库提交失败时逆序恢复;数据库成功后才最终清除,清除失败返回 `cleanup_pending` 并保留恢复证据。外部唯一原片继续不移动、不删除。 +- 新增默认 dry-run 的 `scripts/repair_foreign_key_integrity.py`。活动库预演确认 17 条异常来自 8 个历史缺失的 `output_clip` 父记录;修复只新增 8 个 `is_active=0`、无媒体路径的 tombstone,保留 589 条发布任务、4 条字幕任务、28 条 `NEED_REVIEW` 及发布事件历史。 +- 修复前备份为 `data/backups/workflow-before-foreign-key-repair-20260824-130321-535723-35a0f972.sqlite3`;备份 `quick_check=ok` 并保留修复前 17 条外键异常,活动库修复后 `quick_check=ok`、`foreign_key_check=0`。 +- 三类 SQLite Online Backup 统一转换为 `journal_mode=DELETE` 的便携单文件快照,避免迁移、媒体清理和外键修复备份依赖 WAL/SHM sidecar。 +- 新增测试覆盖恶意活动库环境变量、清理 guard、第二个目录移动失败、数据库提交失败、最终清理失败、外键 dry-run/备份/tombstone/拒绝路径和单文件备份。 +- Windows Web 由 Alter 托管并会自动重启,未停止 Alter;数据库修复在 `BEGIN IMMEDIATE` 锁内完成备份和写入。修复后 `8001 /health=ok`,Scheduler `running=true`、`consecutive_failures=0`,Windows Worker `8765 /health=ok`;未触发真实投稿。 +- 最终验证通过:P0 定向测试 `42 passed`、完整测试 `512 passed`、Ruff、Python 编译与 Git 空白检查全部成功。全量测试故意继承活动库路径后仍使用临时 sandbox;正式 Scheduler 同期持续运行,因此活动库 mtime 会变化,但数据库大小稳定且 `foreign_key_check` 始终为 0。 + +## 2026-08-24 全项目工程体检:Codemap + Code Overhaul + SonarQube + +- 按“只审计、不整改”边界完成 13 个功能模块的 Codemap 独立评分,并生成 `.codemap/modules.json`、`.codemap/codemap.html` 和 `.codemap/codemap.md`。 +- 完成架构、业务状态、数据库、并发恢复、AI/转写、发布安全、性能、测试和 Dead/Legacy 的 Code Overhaul 全量 Review。 +- 使用本机 SonarQube Community Build 26.8 和官方 Scanner 镜像实扫 `app/`、`scripts/`:38,963 ncloc,29 Bugs、2 Vulnerabilities、0 Hotspots、479 Code Smells、0.2% Duplication,维护性 A、可靠性 C、安全性 D,技术债估算约 58 小时。 +- Sonar Coverage 为 0% 是因为项目没有生成 `coverage.xml`;JUnit 已导入 500 项测试且成功率 100%,报告没有把“无覆盖率输入”误写成“代码没有测试”。 +- 隔离测试通过:`500 passed, 9 warnings`;Ruff、Python 编译、前端 JS 语法、PowerShell Parser、三组 Docker Compose 配置和 `pip check` 均通过。Ruff format check 显示 105 个文件会被重排,本轮没有格式化。 +- 只读核验当前原生服务与 Windows 发布 Worker 均在 `127.0.0.1` 健康;没有调用真实 AI、没有计费生成、没有真实投稿。 +- 发现并记录 P0:测试 fixture 在错误数据库环境下可整表删除、活动 SQLite 已有 17 条外键违规、永久删除的文件系统操作无法随数据库事务回滚。 +- 项目综合健康度定为 `59/100`,成熟度为“可用 V1”;完整证据、技术债 Top 10、删除候选、暂时不要动的区域和独立可回滚路线图见 `PROJECT_AUDIT.md`。 +- 本轮仅新增审计文档、Codemap 状态/可视化和 Sonar 本地扫描配置;没有修改生产业务代码、Schema、真实数据或发布流程。 + +## 2026-08-24 字幕审核、异步渲染与自动流水线整合(PR 4) + +- 全自动流水线在切片后创建原片/切片字幕草稿,并停在 `pending_subtitle_review`;不会继续生成文案或发送任务。 +- 用户必须选择“审核并批量烧录”或“跳过字幕,使用原片继续”。决定写入任务配置,恢复流水线后不会静默切换视频来源。 +- 字幕批量烧录接入 `workflow_jobs` 持久化队列,固定引用已审核 revision,逐条保存 checkpoint;重启只复用文件存在且验证通过的结果。 +- FFmpeg 渲染支持真实进度、取消、无进展超时、临时文件和原子替换;优先尝试可用的 NVENC,失败回退 `libx264`,输出必须通过 H.264、`yuv420p`、音轨和时长校验。 +- 旧 active 字幕成片只在新结果验证成功后切换;失败、取消或进程异常不会覆盖旧文件,精确临时文件会被清理。 +- AI 纠错按所选 cue 分批创建 `ai_suggestion` revision,仅接受文字和断句建议;差异需人工选择,接受后再生成新的人工 revision,不自动覆盖 active 版本。 +- 发送中心新增字幕来源证据门禁:自动使用字幕时必须同时满足 revision 已审核、渲染已验证、文件存在;明确跳过时只使用原片。 +- 新增第四阶段专项测试,覆盖暂停/恢复、显式跳过、审核门禁、checkpoint、取消清理、NVENC 回退、验证失败、AI 建议与发布就绪证据。 +- 验证通过:Ruff、Python 编译、两个 JavaScript 语法检查、定向测试 `143 passed`、完整测试 `500 passed`;本机 9:16、16:9、1:1 三种画幅真实 FFmpeg 烧录与 FFprobe 校验通过。 + +## 2026-08-23 字幕数据层与专业编辑器重构(PR 3) + +- 新增 `subtitle_tracks / subtitle_revisions / subtitle_cues`,以原片主时间轴为事实源;每次人工编辑、导入或同步都创建不可变 revision,渲染任务固定引用 revision。 +- 输出切片在切割提交时保存原片起止毫秒、时长和源指纹快照;切片字幕按快照截取并换算本地时间,人工编辑后只标记待同步,不自动覆盖。 +- 结构化转写 checkpoint 成为字幕首选数据源,保留置信度和毫秒时间;`transcript.md` 只作为旧任务兼容输入,不再限制 120 行。 +- 锁定 `pysubs2==1.9.0`,支持 SRT、VTT、ASS 导入导出和动态 ASS 序列化;ASS 受格式规范限制为 10ms 精度,内部 revision 与 SRT/VTT 继续保持 1ms。 +- 本地固定 `wavesurfer.js@7.12.11` 及 Regions/Timeline;原片波形由服务端 FFmpeg 以 100Hz 预计算 peaks 并缓存,浏览器不解码完整长视频音频。 +- 字幕工作台新增视频联动、当前行高亮、虚拟列表、搜索替换、毫秒编辑、区间拖动、拆分合并、增删、批量位移、撤销重做、说话人、自动保存、审核与导入导出。 +- 中文质量规则只提示不改字;ASS 样式按 9:16、16:9、1:1 实际分辨率计算,支持安全区、描边、阴影和说话人颜色。 +- GPL-3.0 的 VideoCaptioner 仅研究流程,没有复制源码;依赖版本及 MIT/BSD 许可证已记录在 `THIRD_PARTY_NOTICES.md`。 +- 新增 20 项字幕专项测试,覆盖幂等迁移、毫秒精度、150 行不截断、切片边界、人工版本保护、原片继承、多步编辑、三格式往返、三种画幅、说话人样式、固定渲染 revision、服务端 peaks 与 API 范围查询。 + +## 2026-08-23 长直播分层高光选片(PR 2) + +- `long_live_talk` 不再进入通用选片,改用固定约 300 秒、重叠 60 秒的语言高光窗口。 +- 新增窗口级 SQLite checkpoint:状态、累计尝试次数、结果校验和、错误和下次重试时间均独立保存;同一转写指纹下成功窗口跨进程复用。 +- 远程窗口单轮最多尝试 3 次并指数退避;失败窗口不会抹掉其他窗口结果,再次分析只请求缺失窗口。 +- 高光按金句观点、故事经历、情绪峰值、冲突反转、实用知识、互动幽默六类召回,跨窗口按时间与文本语义合并。 +- 最终结果先执行每小时密度上限,再轮询各小时执行总量上限,避免前半场提前占满全部名额。 +- 时间轴覆盖不足 90% 时保存为“分析不完整”,自动流水线和手动切片均会在生成文件、同步发送中心之前停止。 +- 新增六小时结构化时间轴、跨窗口去重、三次重试、断点复用、覆盖率门禁和幂等数据库测试。 + +## 2026-08-23 长直播基础设施(PR 1) + +- 新建任务移除综艺隐藏默认值,页面、multipart 上传与 JSON API 都要求显式选择三种模式;历史任务保留原模式。 +- 新建页接入受根目录保护的本地/NAS已有文件浏览,4 GB 以上素材引导使用引用模式,不复制外部唯一原片。 +- 增加媒体/磁盘预检、6 小时验收范围提示,以及长直播每小时密度和总量参数。 +- 工作流 Job 增加 lease、heartbeat、取消、重试和 checkpoint;应用启动单重型 worker,每个任务由独立 Python 子进程执行,全自动流水线和转写进入持久化队列。 +- 转写新增 run/chunk、源指纹、逐块原子提交和校验恢复;本地/火山结果统一支持置信度和词级毫秒时间戳。 +- 音频提取改为临时输出原子替换,并支持真实进度、无进展超时和终止进程树;显式 CPU 配置保持优先。 +- 本阶段不实现 `long_live_talk` 分层 AI 选片和新字幕编辑器;分别留给 PR 2 与 PR 3。 +- 验证通过:Ruff、Python 编译、`app.js` 语法、PR 1 定向测试 `85 passed`、完整测试 `455 passed`。 +- 六小时低码率实测通过:H.264/AAC 原片为 `21600s`,媒体/空间预检正常;实际提取 `16kHz` 单声道 PCM WAV 仍为 `21600s`(约 691 MB),验收临时媒体已清理。 + ## 2026-08-23 v2.1.0 主线整合与片段审核稳定性增强 - 审计全部本地/远端分支、开放 PR 和 5 个 worktree:图片文档、续接排期、发送中心文案规则等近期分支已进入 `origin/master`;当前 PR #38 是唯一需要整体合入的有效新主线。存在冲突、失败 CI 或旧 opencli 架构的历史分支不做盲目整分支合并。 @@ -1064,3 +1281,90 @@ - 系统状态页新增 Codex 路径、模型、超时和版本检查;分析与发布文案分别选择 Provider,默认均为 Codex,远程 DeepSeek 与本地 Ollama 继续作为人工回退。 - 任务详情页把 Codex CLI 分析作为主按钮,发起前明确提示会消耗 Codex 套餐额度;转写 Provider 仍只允许火山/本地,不与文本 AI Provider 混用。 - 本机 `.env` 仅把非敏感 Provider 配置切换到 Codex,未读取、输出或改动已有 API Key;未执行真实 AI、发布或投递。 + +## 2026-08-24 稳定 V1 P1A 路径与进程边界 + +- 任务目录解析统一限定在 `TASKS_DIR`:拒绝绝对路径、盘符、`..` 和符号链接逃逸,同时保留普通任务名中的 `~` 与受控旧目录兼容。 +- 新任务目录分配改为 `mkdir(exist_ok=False)` 原子预占;并发同名任务会得到不同目录,遗留回收站移动入口显式保持“目标尚不存在”语义。 +- 数据库读取任务目录失败不再静默回退为 task id 或空集合,避免数据库锁定/损坏被伪装成正常兼容路径。 +- 对外媒体响应增加任务级目录约束:切片只能来自当前任务的 `05_clips`/旧 `clips`,字幕成片只能来自 `06_subtitled`,异常数据库路径返回 404;白名单 NAS/本地原片继续可用。 +- Windows 子进程树终止现在校验 `taskkill` 退出码并等待进程退出;超时、启动失败、非零退出和无法确认退出都会显式报错,不再让上层误以为已经安全停止。 +- 移除路径测试中的全局模块 reload 泄漏,并让边界测试始终引用当前运行时 `storage_service`,消除全量测试顺序依赖。 +- 独立验收:边界专项 `52 passed`,全量 `533 passed`;Ruff、Python Compileall 和 `git diff --check` 全部通过。未访问真实 NAS 内容,未触发 AI 或真实投稿。 + +## 2026-08-25 稳定 V1 P1.3d 字幕批次原子性与恢复 + +- 字幕“审核并批量烧录”改为一个 `BEGIN IMMEDIATE` 事务:所有当前 active revision 在事务内重新校验,批准、创建/复用 Workflow Job 和 `subtitle_delivery_mode` 同时提交;任一 revision、配置或 Job 插入失败时整批回滚。 +- 单条批准也在写锁内重新核对 active revision,过期页面请求不能把字幕轨回退到旧版本;损坏的 `auto_config_json` 改为明确失败,不再静默覆盖为 `{}`。 +- 字幕成片的临时文件切换、completed/verified 写回和 active 版本切换合并为一次短事务,并绑定当前 Workflow Job 的 owner、lease token、未过期时间和取消状态;旧 Worker 或旧 revision 不能激活迟到结果。 +- 新 Worker 接管时会收口同一 Workflow Job 遗留的 processing 子任务,只清理本执行标记的 `.part.mp4` 和没有数据库引用的中断最终文件;其他 Job、已验证 active 文件、外部文件和历史 revision 均保留。 +- 若进程在“字幕 DB 已提交、checkpoint 尚未写入”的窄窗口退出,重启会从同一 Workflow Job 的 active + verified 结果恢复 checkpoint,不重复运行 FFmpeg。 +- 字幕完成与后续自动流水线 Job 改为同事务提交;取消、过期 lease 或不兼容的续跑 payload 会整笔回滚。发布草稿批次在提交前再次核验 lease,并可从同一 Workflow Job 的持久化草稿证据恢复 checkpoint。 +- 源轨生成、切片轨同步、手工保存和字幕导入均在写锁内重读 active revision 并使用条件更新;迟到的旧读取不能覆盖较新的人工版本。“跳过字幕”也会拒绝已经排队的后续自动流水线。 +- Worker 子进程启动失败会明确写入 failed 而不是让队列线程退出;重启接管同时收口 processing/queued 字幕子记录,按 revision 复用的 ASS 缓存不会被误删。 +- 最终独立验收 Ruff、Compileall 通过,字幕/Job fencing/队列/自动流水线/checkpoint/状态机/发布关联/版本回滚共 `167 passed`。Pytest 明确使用临时 `test_workflow.sqlite3`;活动库前后 size 与 SHA256 完全一致。未调用 AI、Chrome 或真实平台投稿。 + +## 2026-08-25 稳定 V1 P1.4 AI、转写与媒体失败边界 + +- AI Provider 新增统一错误分类和计费不确定标记;429/确认未连接只做最多 3 次有限重试,5xx、408、超时、空响应、坏 JSON、坏 Schema 和空模型结果都不自动重发。 +- 普通选片与综艺选片删除“解析失败后再次调用模型”的旧逻辑;Markdown/代码围栏 JSON 只在本地解析层兼容。Codex CLI 超时/坏输出按计费不确定收口,本地模型只在 404/405 时切换协议。 +- 长直播窗口 checkpoint 的 running/completed/failed 写回绑定当前 Workflow lease,只有安全错误才在本次执行内重试;成功窗口在恢复时继续复用。 +- 火山引擎远程转写在请求前保存稳定 request id 和 `requesting` 状态;超时、5xx、坏响应或成功响应字段异常进入 `uncertain`。普通 Job 重试不再创建新请求,只有用户明确“重新生成转写”才创建新 run。 +- 修复取消异常被 Provider 通用异常包装的问题;用户取消现在由转写工作流记录为 cancelled,不会误显示为 Provider failed。 +- 自动流水线文案按切片与平台逐项写入带 request fingerprint 的原子 checkpoint;恢复必须完整覆盖 output×platform。AI 失败规则文案带风险标记并进入 `NEED_REVIEW`,不会作为 AI 成功静默发布。 +- 任务详情和媒体预检的 FFprobe 增加超时/OSError 边界;视频切片、音频提取和字幕烧录增加超时/停滞终止与 `.part` 原子切换/失败清理。独立 ASS 导出在无媒体时使用 1080×1920 画布,有媒体但探测失败时明确报错。 +- 新增 Provider 错误矩阵、重复调用、远程转写 uncertain、取消传播、媒体超时、部分文件清理和文案 checkpoint 恢复测试;重点回归 `67 passed`。最终独立验收 Ruff、Compileall 通过;全量收集 714 项,排除 3 个真实 FFmpeg 参数实例后 `711 passed`、0 失败、10 个弃用告警。 +- 未读取或修改 `.env`,未访问真实 AI、火山引擎、FFmpeg、Chrome 或平台,也未写入活动 SQLite。下一轮 P1.5 处理密钥响应、输入校验、XSS 与本地管理员接口门禁。 + +## 2026-08-25 稳定 V1 P1.5 本地安全边界 + +- AI 配置读取 DTO 不再返回火山、远程分析、发布文案或本地 Provider 的原始 Key,也不再返回 `.env` 绝对路径;页面改为“已配置,留空保持”的密码框,空值保存继续保留已有密钥。 +- AI HTTP 错误与坏 JSON 不再把 Provider 原始正文写入任务日志;日志只保留 HTTP 状态、错误类别、重试和计费不确定语义,避免上游回显 Prompt、Token 或内部诊断。 +- 发布平台配置和账号读取 DTO 删除原始 `client_secret`、`access_token`、`refresh_token`,只保留脱敏摘要;Provider 内部调用通过私有读取路径取得凭据,避免把展示 DTO 误用于真实发布。 +- OAuth callback helper 保存 Token 后只返回脱敏账号 DTO,不再把平台原始授权响应重新挂回返回值。 +- 发布执行结果 DTO 会递归脱敏 Provider 响应中的 Token、Cookie、Authorization 和 Secret;数据库事件与立即执行响应共用同一脱敏边界。 +- 脱敏规则覆盖 `Set-Cookie`、`X-Auth-Token`、`provider_access_token` 等前后缀键,并清理 message/异常字符串中的 Bearer、Authorization、Cookie 与常见 Token 赋值,不只依赖精确字段名。 +- 发布任务列表、详情、历史与 Scheduler 快照不再返回旧库中的原始 `provider_response`/`publish_result` 字符串,只返回递归脱敏后的结构;这也覆盖新脱敏逻辑上线前的历史记录。 +- 损坏或非对象的历史 Provider JSON 只返回 `invalid_payload` 标志,不回显 `raw/data` 原文,避免半截 Token JSON 绕过按字段名脱敏。 +- 除健康检查、favicon 和静态资源外,页面、API 与媒体默认同时校验回环 Host 和客户端 IP;伪造 `Host: localhost` 不能绕过。非本机请求必须携带配置的 Bearer Token,写请求还会拒绝跨站 Origin。Token 不再注入 HTML 或前端 JavaScript。 +- Docker Web 端口改为只绑定 `127.0.0.1:8001`;容器内显式启用 loopback 代理兼容,必须与该 Host 绑定成对保留。保持当前个人本地使用方式,不引入多用户权限系统,也不改变 Windows Worker、平台验证码和人工复核边界。 +- 媒体 CORS 取消“任意 localhost 端口”通配,只允许明确列出的本地前端端口和抖音/B站创作者中心;未知本地网页不能读取源视频或产物。 +- 设置模型统一拒绝控制字符、异常长度、带凭据/查询/片段的服务地址、非 HTTPS 远程端点和越界响应路径;本地 AI URL 只允许明确的本机主机名。 +- 发布平台 OAuth/API 配置同步拒绝控制字符、危险 scheme、URL 内嵌凭据和非本机明文 HTTP,避免持久化后形成 OAuth 跳转或 Provider SSRF 入口。 +- 发布结果 URL 在最终落库和返回前按协议与真实 hostname 校验;伪造域名或危险 scheme 会进入 `NEED_REVIEW`,不会被记录为可信平台链接。 +- `risk_flags` JSON 损坏或结构不是数组时一律 fail closed 进入 `NEED_REVIEW`,不再把合法 JSON 的错误结构当作“无风险”。 +- 发送中心动态排期预览和任务发布链接改为 DOM `textContent` 构造,不再把标题、描述或 URL 拼进 `innerHTML`;Jinja 与字幕工作台既有转义保持不变。 +- 新增密钥不回显、空值保留、远程门禁、跨站写入、验证错误不回显原输入、发布 Token DTO、恶意 URL 和动态 XSS 回归。所有测试只使用临时 SQLite 与 mock,不读取 `.env`,不调用真实 Provider、Chrome 或平台。 +- 修复安全门禁误拒随机本机端口同源 POST 的回归:现在只有 Origin 与请求自身的 scheme/host/port 完全一致才按同源放行,不重新开放任意 localhost 跨域。 +- 最终隔离验收收集 751 项,排除 3 个真实 FFmpeg 参数实例后 `748 passed`、0 失败、10 个依赖告警;Ruff、Compileall、两个 JavaScript 语法检查、三套 Docker Compose 配置和 `git diff --check` 全部通过。活动库 `data/workflow.sqlite3` 前后大小均为 `7041024` bytes,SHA256 均为 `7AA6BE7955B46CA66A4A255A712E945832BA91E453BE73013699AEE1D4413E7F`。 + +## 2026-08-25 稳定 V1 P1.4b AI 结果一致性与跨进程恢复 + +- 人工 `/process/ai` 不再在 Web 请求线程直接调用 Provider,改为原子创建/复用持久化 `ai_analysis` Workflow Job;页面轮询 Job 状态,完成后重新读取分析摘要和历史。 +- AI Worker 复用现有子进程、heartbeat、owner/token 和过期 lease 接管;候选、active run 与任务 `pending_review` 在最终 SQLite 事务内重新验证 lease,旧 Worker 不能提交结果或覆盖新执行。 +- `clip_candidates` 替换、旧 run 取消激活、新 run 插入和任务终态改为一个 `BEGIN IMMEDIATE` 事务;任一步失败都会回滚到旧候选和旧 active run。 +- `candidate_clips.json` 改为由 `ai_analysis_runs.analysis_payload_json` 重建的派生缓存;同一 Workflow Job 在数据库已提交后中断,会复用已提交 run 并重建文件,不再次调用 Provider。 +- general 分段和 variety recall/expansion 新增结构化 `analysis_meta`:记录期望/完成/失败单元、无结果单元、无效条目、覆盖率、失败阶段和质量降级。 +- 合法空窗口继续视为成功;超时、HTTP/坏 JSON、坏结构或无效条目会标记分析不完整。全部单元失败继续 hard fail;部分成功保留候选供人工检查。 +- 自动流水线和手动“生成切片”都对三种 profile 统一检查 `analysis_incomplete`;长直播继续保留 90% 覆盖率门槛,普通/综艺局部失败也不能进入切片或发送中心。 +- Codemap 首轮复核新增的重复计费边界也已封口:普通分段、综艺召回/扩展/全局评审在调用前写入 Job 内单元 checkpoint,成功结果带 checksum;接管只复用已确认结果,发现上次请求已开始但未落账时进入 `uncertain`,不自动再次计费。自动流水线步骤 checkpoint 会保留这组恢复证据。 +- 长直播不再把 `clips/candidates` 当作 `moments`,显式空数组仍是合法空窗口;坏条目、缺字段或前次请求结果不确定都会标记失败/不完整。综艺全局评审必须覆盖全部候选,`quality_degraded` 与 `analysis_incomplete` 均锁定自动切片。 +- AI Job 取消会把仍在 AI 阶段的 Task 收口为可重试的 `pending_ai`;父 Worker 失败会写入明确失败态。停机 release 原子区分 queued/cancelled,不能再产生 `queued + cancel_requested=1`;进程树终止失败会保留 lease 并重试,不再杀死 Worker 线程。 +- 失败或取消后的人工重试会原位重新排队同一个 AI Job,保留已确认结果与 `billing_uncertain` 单元账本;切换按钮参数也不会绕过旧账本创建新 Job。Provider 的实际 model、endpoint、protocol 和实现类型进入输入指纹,避免配置变化后误把旧模型结果标成新模型。 +- active run JSON、`analysis_meta`、Schema、选片模式、布尔质量标记、有限覆盖率及 ratio/percent 对应关系统一 fail closed;缺失、NaN/Inf、bool 冒充数字或模式漂移都不能进入手动/自动切片。 +- 最终安全交叉复审把静态媒体 CORS 与管理 API 写入 Origin 拆开;抖音/B站创作者中心仍可按白名单读取受控媒体,但不能据此跨站写本地管理 API。 +- 新增人工 Job 去重、Worker 完成、事务回滚、旧 lease fencing、DB 已提交后接管、单元 checkpoint 复用/计费不确定、自动 checkpoint 共存、坏长直播结构、质量门禁和父进程收尾测试。最终独立全量验收 `785 passed, 3 deselected`;Ruff、Compileall、`app.js`/`publish-center.js` 语法、三套 Compose 合并配置和 `git diff --check` 全部通过。 +- Pytest 使用进程级 `niuma-pytest-*\data\test_workflow.sqlite3`,未触碰活动库。活动服务继续由 `127.0.0.1:8001` 的 Uvicorn PID `56576` 持有;只读数据库检查为 `integrity_check=ok`、`foreign_key_check=0`。活动库哈希随常驻服务 WAL 写入发生变化,未为取得静态哈希而停止正式服务。 + +## 2026-08-26 修复中文路径视频预检误判 + +- 修复 Windows 默认 GBK 解码 FFprobe/FFmpeg UTF-8 输出时,中文文件路径触发解码异常并被误报为“源文件没有视频轨”的问题。 +- 媒体创建预检与首尾解码抽样统一显式使用 UTF-8,并以替换非法字节的方式保留可诊断输出;真正缺少视频轨或音轨的素材仍继续拒绝创建。 +- 增加中文文件名编码回归和真正无视频轨回归测试;不修改上传目录、任务数据库、页面结构或原始视频文件。 + +## 2026-08-27 抖音优先的康熙综艺选片 Prompt + +- 1 号默认 Prompt 收敛为抖音优先、连续原片可直接发布的康熙综艺选片标准,不再要求模型生成综艺 V2 不消费的 JSON 字段。 +- 强制用原始 `start_time` 后前 3 秒评价钩子,要求 15 秒内出现第一次有效刺激,并把评论动机、单一话题和完整反应闭环设为硬门槛。 +- 片段以 60–90 秒为主,普通可看内容不得虚高到 78 分;本次不修改三阶段分析代码、A/B/C 门槛、候选池或历史分析结果。 +- 相关选片与 Prompt 测试 `43 passed`,Ruff 和 `git diff --check` 通过;活动 SQLite 更新前已生成一致性备份,更新后 `integrity_check=ok`、外键异常为 0,其他 Prompt 与任务记录未变化。 diff --git a/NEXT_STEPS.md b/NEXT_STEPS.md index 1183418..1fc8210 100644 --- a/NEXT_STEPS.md +++ b/NEXT_STEPS.md @@ -1,5 +1,121 @@ # Next Steps +## 2026-08-28 2.1 集成收口 + +1. 等待 PR #60 的 Linux、Windows 与 Docker 三组 CI 全部通过;Docker 页面冒烟必须确认 `/`、`/tasks`、`/clips`、`/publish` 均返回成功。 +2. CI 全绿后由用户在 GitHub 使用普通 merge commit 手动合并到 `master`,不要 squash、rebase 或自动合并,以保留原 26 个提交历史。 +3. 合并后确认 `master` 包含提交 `c3e52be`,再关闭被顶层 PR 覆盖的堆叠 PR;不要删除分支,也不要处理无关的 Dependabot #44/#45。 +4. 只有 2.1 基线合并完成,才从最新 `master` 创建 `codex/douyin-content-review` 开始 2.2 内容复盘开发。 + +## 2026-08-26 页面动效检查 + +1. 打开工作台,刷新后只会看到四项核心统计播放一次轻微渐显;最近任务表、导航和静态内容不会反复晃动。 +2. 新建任务页选择“长直播高光(语言类)”,两项长直播设置应平滑出现;任务详情只有进度或状态真的变化时才短暂提示,停留轮询不会每 3 秒闪一次。 +3. 发送中心切换“内容准备 / 排期计划 / 执行记录”,以及打开账号或排期抽屉时应有轻微过渡;系统不会给每 5 秒刷新的记录和 42 格日历重复套动画。 +4. Windows 11 可在“设置 → 辅助功能 → 视觉效果 → 动画效果”关闭动画后刷新页面,所有内容应立即显示,抽屉和定位改为即时反馈。 +5. 本轮动效检查不需要点击 AI、切片、字幕烧录、封面、排期确认或发送按钮,也不会触发真实抖音/B站投稿。 + +## 2026-08-26 发送中心 AI 文案操作检查 + +1. 打开 `http://127.0.0.1:8001/publish`:页面顶栏只保留“账号管理”,“同步遗漏切片”和“补齐抖音缺失封面”位于内容准备的维护区。 +2. “同步遗漏切片”只为尚未进入发送中心的成片补建草稿和默认封面,不调用 AI,也不会改写已有文案;平时自动同步正常时无需点击。 +3. 需要改一条文案时,使用内容卡上方的“AI 重写本条文案”;需要改多条时先勾选,再使用底部“AI 重写已选文案”。AI 结果会立即保存到对应草稿。 +4. 排期计划只管理时间,执行记录只核对发送结果,这两个区域不再提供 AI 文案按钮。刷新页面不会自动重写旧草稿。 +5. 本轮验收不需要点击任何 AI、封面、排期或发送按钮,也不会触发真实抖音/B站投稿。 + +## 2026-08-26 片段审核累计统计检查 + +1. 打开 `http://127.0.0.1:8001/clips`,顶部应只显示“累计审核任务、已通过视频、已完成任务”三张卡片,不再出现待 AI、待检查、可生成切片和异常任务统计。 +2. 当前活动库预期显示累计审核任务 `20`、已通过视频 `163`、已完成任务 `20`;以后新增、启用、停用或删除任务/候选片段后,数值会按当前未删除数据重新计算。 +3. “已通过视频”指当前处于启用状态的视频片段,可能来自人工审核或全自动选片,不应理解为全部都经过人工逐条点击确认。 +4. 本轮只调整查询统计与展示,不需要运行 AI、重新切片或点击发送中心,也不会触发抖音/B站投稿。 + +## 2026-08-26 任务详情实时进度检查 + +1. 打开 `http://127.0.0.1:8001/tasks/3210d91ee1fb`,状态概览应显示“托管发布中 · 9/12”,并分别显示总流程 `97%`、当前操作“托管发布” `75%`;实际数值会随后续排期发送继续变化。 +2. 停留在当前页面即可观察:有运行任务或待执行排期时,状态概览、当前操作和右侧运行日志每 3 秒使用同一份快照更新,不再需要 F5 或切换任务页。 +3. 以后创建低风险手动测试任务时,可观察转写或 AI 的“当前操作”百分比;不要为了验证界面重复运行重要任务,也不要使用会产生额度或平台投稿的真实流程做测试。 +4. 发布失败或结果不确定时会显示黄色警告并保留 `NEED_REVIEW` 人工核对边界;本轮没有触发真实 AI、切片或投稿。 + +## 2026-08-26 工作台改版后检查 + +1. 继续通过 `http://127.0.0.1:8001/` 使用工作台;顶部应显示本周一至周日范围和四项核心指标,随后直接进入最近任务列表,不再显示每日新增任务柱状图。 +2. “待推送任务”表示至少还有一个有效切片未进入发布终态的任务;`NEED_REVIEW` 会继续计入,必须人工核对,不能自动当作已发布。 +3. 已发布、已导出或已取消的最新记录不会计入待推送;若用户新建了更晚的排期 / 重发记录,对应任务会再次进入待推送,这是预期行为。 +4. 本轮不需要点击“立即发送”,没有执行真实抖音 / B站投稿;后续真实发送仍需遵守账号登录、验证码、风控和人工确认边界。 + +## 2026-08-25 任务 20 修复后检查 + +1. 打开 `http://127.0.0.1:8001/`,确认任务 20 显示“已完成”、无外键错误,并仍为 12 条 active 切片和发送中心已关联 12/12。 +2. 打开“新建任务”:任务名候选最多 5 个,只能上传本机文件;通用/康熙模式不显示密度设置,长直播模式才显示。 +3. 对低风险全自动测试任务选择“跳过字幕,进入片段审核”,确认不会立即生成发送中心任务;在片段审核保存后才同步。 +4. 不需要点击“立即发送”;本轮不会执行真实投稿,也不改变 `NEED_REVIEW` 的人工确认边界。 + +## 2026-08-24 稳定 V1 整改路线 + +- [x] P0.1:Pytest 与活动数据库、媒体目录彻底隔离,危险整表清理增加 fail-closed 校验。 +- [x] P0.2:修复活动库 17 条孤儿外键;以 8 个不可见 tombstone 保留发布、字幕和人工复核历史,最终 `foreign_key_check=0`。 +- [x] P0.3:永久删除改为“同卷隔离暂存 → 数据库提交 → 最终清理”,提交失败可恢复,最终清理失败可重试。 +- [x] P0.4:SQLite Online Backup 固定为不依赖 WAL/SHM 的单文件快照。 +- [x] P1.1:收紧媒体读取与任务目录边界,补齐核心读写路径的 traversal / arbitrary-file 回归测试;并补充同名目录原子预占和进程树退出确认。 +- [x] P1.2a:Workflow Job 增加每次 claim 唯一的 `lease_token`,旧 Worker 的心跳、进度、checkpoint、终态和 release 均不能覆盖新执行。 +- [x] P1.2b:Publish Job 使用现有 `execution_id` 完成所有写回 fencing,并让 Windows Publish Worker 对重复 execution 幂等;跨进程锁、恢复 fail-closed 和 Scheduler 优雅停机已覆盖。 +- [x] P1.3a:任务状态接口受控转换;Task/Job lease 代际一致;自动流水线取消恢复、READY 终态和关联发布清理明确;切片批次编号、输出目录、Task 终态和数据库激活原子化。 +- [x] P1.3b:自动流水线已增加版本化步骤 checkpoint、输入/产物证据、旧 Job 原位重试和重领恢复;已覆盖 AI、转写、切片、文案、排期与发布草稿的失效边界。 +- [x] P1.3c:已增加 `schema_migrations` 版本/checksum 账本、迁移前 Online Backup 和 fail-closed v2 活动发布唯一索引;重复数据、索引漂移或账本异常都会拒绝启动。本轮未直接迁移活动库。 +- [x] 任务 20 专项:完成状态恢复、重复 AI 409 保护、上传单入口、最近 5 个任务名、长直播参数显隐和“跳过字幕后进入片段审核”闭环。 +- [x] P1.3d:字幕批准、Job 创建和交付模式已原子提交;成片激活绑定 lease/revision;接管会清理本执行残留并恢复“DB 已提交但 checkpoint 未写”的结果。 +- [x] P1.4:已统一 AI/远程转写/FFmpeg/FFprobe 超时、错误 JSON、429/5xx 与重试幂等边界;计费不确定请求默认禁止自动重发。 +- [x] P1.4b:AI 候选/run/任务终态已原子提交;人工 AI 已使用持久 Job/lease;普通/综艺/长直播具备单元恢复、计费不确定和不完整/质量降级门禁,父 Worker 取消与异常收尾已明确。 +- [x] P1.5:配置读取 DTO 已脱敏,非本机请求受 Bearer 门禁保护,写请求校验同源;设置、平台 URL 与动态 HTML 输入边界已收紧。 +- [ ] P2:先替换并继续停用非 WAL-aware 的旧任务目录迁移脚本,再拆分 God Service、去除查询重复、补核心集成/故障测试和可观测性;不做全面重构。 + +### 本轮人工检查 + +1. 正常打开 `http://127.0.0.1:8001/` 和发送中心,确认页面可用、Scheduler 正常、Windows Worker 正常。 +2. 不需要点击“立即发送”;本轮没有执行真实投稿,也没有改变 28 条 `NEED_REVIEW` 的人工确认边界。 +3. 若未来永久删除返回 `cleanup_pending`,不要手工移动隔离目录;保留返回信息和 manifest,使用后续安全清理入口重试。 +4. 修复前数据库备份位于 `data/backups/workflow-before-foreign-key-repair-20260824-130321-535723-35a0f972.sqlite3`,只有活动库无法通过完整性检查时才考虑恢复,不要直接覆盖当前数据库。 +5. P1B.1/P1B.2、P1.3a 与 P1.3b 代码及隔离测试均已完成;活动库尚未增加 `workflow_jobs.lease_token`,本轮也未重启正式服务。下一独立轮先确认没有活动 Workflow Job,再做在线备份、幂等迁移和正式重启烟测。 +6. P1.3b 不增加数据库列。合并并完成正式迁移/重启后,可用一条低风险测试任务在 AI 或切片完成后重启 Worker,确认同一 Job 从 checkpoint 续跑;不要点击真实发布,也不要用重要待发布素材做故障演练。 +7. P1.3c 对正式库只做了只读预检:当前无活动发布重复组,旧唯一索引仍在,尚未写入 `schema_migrations`。合并后的首次正式重启会先创建迁移备份再切换 v2 索引;应与尚未部署的 Workflow fencing Schema 一起安排受控迁移和重启验收,不要在服务运行中手工改索引。 +8. P1.3d 不增加数据库列,也不需要手工清理 `06_subtitled`。合并并正式重启后,可用一条低风险测试任务执行“审核并批量烧录”;不要通过强杀正式服务制造故障,跨进程接管已由隔离 lease 回归验证。 +9. P1.4 不增加数据库列,也未调用真实 AI、火山引擎、FFmpeg 或平台。429/确认未连接可在同一 request id 内有限重试;5xx、超时、空响应、错误 JSON 和字段异常均视为可能已计费,普通重试不会再次调用。只有用户明确“重新生成转写”才允许从 uncertain 状态创建新批次。 +10. P1.5 不改变个人本地使用方式:继续通过 `http://127.0.0.1:8001` 使用。Docker 端口只绑定回环地址;如果未来确实需要局域网访问,必须先设置本地管理员 Token,并由客户端使用 Bearer Token,不能直接暴露端口。 +11. P0/P1 稳定线已完成,下一轮进入 P2。先补可观测性与核心故障集成测试,再渐进拆分 God Service;保持一轮一处、独立测试和可回滚,不做全面重构。 + +## 2026-08-24 工程体检确认的原始整改顺序 + +本轮审计已经完成,用户已确认按以下顺序逐轮整改;不要一次性全面重构: + +1. **P0.1 测试数据库保护**:让 pytest 只能使用唯一临时库,发现活动库路径立即拒绝启动。 +2. **P0.2 数据一致性**:先做 WAL-aware 备份和 dry-run,再逐条处理活动库 17 条外键违规;修复前不要手工删记录。 +3. **P0.3 可恢复删除**:把“直接删文件后提交数据库”改成带隔离区和 manifest 的两阶段删除。 +4. **P1.1 Secret 与本地鉴权**:已完成。读取接口不再返回原始 API Key/OAuth Token;非 loopback 请求必须通过 Bearer Token,Docker 默认只绑定 `127.0.0.1`。 +5. **P1.2 路径边界**:任务目录、媒体响应、Worker execution/account id、浏览器 profile 和手工发布包边界均已完成。 +6. **P1.3 Job/Publish fencing**:Workflow lease token 与 Publish execution fencing 已完成;旧 Worker/旧 execution 不能覆盖新 attempt,重复 execution id 不会再次进入 Publisher。 +7. **P1.4 状态与部分成功**:逐步封住任务状态跳跃、切片/字幕批次半提交、转写旧结果复用和 AI partial 成本边界。 +8. P0/P1 稳定后,再补 Coverage、真实故障测试、readiness/日志,然后渐进拆分 `publish_service.py` 与前端大脚本。 + +整改期间继续遵守:每轮范围有限、独立测试、独立提交、可回滚;不更换 SQLite、不引入微服务/React/Vue、不删除历史 Publisher、不改掉 `NEED_REVIEW` 人工复核。 + +## 2026-08-23 长直播四阶段进度 + +- [x] PR 1:模式必选、已有文件入口、媒体/磁盘预检、持久化重型 Job、转写断点与词级时间戳。 +- [x] PR 2:`long_live_talk` 5 分钟重叠窗口、每小时覆盖、全局配额、去重、90% 覆盖门禁和窗口级恢复。 +- [x] PR 3:统一字幕 track/revision/cue、pysubs2 导入导出、wavesurfer 波形与专业编辑器。 +- [x] PR 4:字幕审核暂停、AI 建议 revision、异步批量烧录、NVENC 回退和发送中心门禁。 + +当前验证重点:四个开发阶段均已落地。合并时必须按 PR 1 → PR 2 → PR 3 → PR 4 的顺序;每个 PR 只合并到它标明的前序分支,最终再进入 `master`,不要跳过中间数据库与接口变更。 + +### PR 4 人工检查 + +1. 新建一个低风险全自动测试任务并完成切片;任务应停在“字幕待审核”,发送中心不能提前出现该任务。 +2. 打开字幕工作台,修改并审核字幕后点击“审核并批量烧录”;观察队列进度,完成后确认带字幕视频可播放,任务随后恢复生成文案。 +3. 另建测试任务,在暂停页明确点击“跳过字幕,使用原片继续”;发送中心应显示原片来源,不应误用旧字幕文件。 +4. 选中少量 cue 请求 AI 建议时会调用已配置的文本 AI 并产生额度;先检查 Provider,再确认差异只包含文字,人工接受前 active revision 不变。 +5. 取消一条正在烧录的测试任务,确认状态进入已取消、旧 active 字幕仍可播放、刷新后可以重试。不要用真实待发布内容做这一项。 + ## 2026-08-23 v2.1.0 主线同步后检查 1. 最终提交进入 `master` 后先查看 GitHub CI;只有 Python、Windows host smoke 和 Docker image smoke 全部通过,才把该提交作为 2.1.0 候选版本。 @@ -393,7 +509,7 @@ ## 2026-06-06 三类 AI 接口配置怎么测试 1. 重启或刷新本地后台服务,打开 `http://127.0.0.1:8001/system`。 2. 在页面中间找到“三类 AI 接口配置”,确认能直接看到 `1. 音频转写`、`2. 分析文字稿,生成候选切片`、`3. 发送中心生成发布文案`。 -3. 检查三个 API Key 输入框是不是普通文本框,并且会完整显示你自己填写过的 Key。 +3. 检查三个 API Key 输入框是否为密码框且默认留空;已配置时应显示“已配置,留空保持”的占位提示,页面源码和接口响应中都不应出现原始 Key。 4. 修改“发送中心生成发布文案”的模型,例如改成 `deepseek-chat`,点击“保存三类 AI 接口配置”。 5. 保存后刷新页面,确认发布文案模型仍是你刚才填写的值;同时 `.env` 里的 `VOLCENGINE_ASR_API_KEY` 不应被删除。 6. 新建或打开一个未完成转写的任务,先保证“音频转写”里火山引擎 Key 已填写,再点击“开始处理 / 继续处理”。 @@ -995,3 +1111,17 @@ 2. 先用一条短测试文字稿执行 Codex 分析,核对候选片段 JSON、时间范围和封面时间;这一步会消耗 Codex 额度,需人工发起。 3. 在发送中心只对一条未发布草稿点击 AI 补齐,确认标题、话题、简介通过现有内容安全规则;不要点击真实发布。 4. 若 Codex 不可用,可在系统状态页手动选择远程 AI 或本地模型;旧 API 配置和历史任务保持不变。 + +## 2026-08-26 中文路径视频创建验收 + +1. 打开 `http://127.0.0.1:8001/tasks/new`,选择包含中文目录或中文文件名的正常 MP4 视频。 +2. 创建任务时不应再出现“源文件没有视频轨”;系统应继续完成视频轨、音轨、时长、解码能力和磁盘空间预检。 +3. 纯音频文件、真正没有视频轨或音轨的素材仍应显示对应拦截原因,不会被当作有效视频任务创建。 +4. 本次验收只创建处理任务,不会自动绕过登录、验证码、平台风控或执行真实投稿。 + +## 2026-08-27 抖音优先选片验收 + +1. 下一期康熙综艺任务继续使用 1 号默认 Prompt,不需要额外选择方案。 +2. 人工检查候选片段原始开头前 3 秒是否独立成立、15 秒内是否出现有效刺激,并优先保留 60–90 秒的完整互动回合。 +3. 不要为了凑数开启普通片段;记录“不好笑、节奏拖沓、铺垫不足、内容重复”等反馈,供后续全局复审参考。 +4. 连续发布 10 条后,在抖音后台与此前 10 条比较前段留存、平均观看、完播和评论表现;若无改善,再排查评分门槛、标题封面、账号分发或素材重复。 diff --git a/PROJECT_AUDIT.md b/PROJECT_AUDIT.md new file mode 100644 index 0000000..51fe93e --- /dev/null +++ b/PROJECT_AUDIT.md @@ -0,0 +1,936 @@ +# NiuMa Studio 全项目工程体检报告 + +> 审计日期:2026-08-24 +> +> 审计基线:`38fd60386a609b6ea85a3eb511b83cb296f4d2a1` +> +> 审计分支:`docs/project-engineering-audit` +> +> 方法:Codemap 独立模块审计 + Code Overhaul 全量工程 Review + SonarQube Community Build 26.8 实扫 +> +> 边界:本轮只审计、运行隔离测试并生成文档/审计配置;没有修改生产业务代码、数据库 Schema 或真实发布流程。 + +## 0. P0 整改后增量状态(2026-08-24) + +> 本节记录审计后的已验证整改,不改写下方原始审计快照。P1 尚未完成,因此项目成熟度暂不提前改判为“稳定 V1”。 + +| P0 项目 | 结果 | 验证证据 | +| --- | --- | --- | +| 测试误连活动库 | 已封口 | Pytest 无条件使用进程级临时 sandbox;整表清理二次校验路径。故意传入活动 `DATABASE_PATH` 后全量 `512 passed`,测试路径仍在 sandbox。 | +| 17 条 SQLite 外键异常 | 已修复 | 补充 8 个 `is_active=0`、无媒体路径的 tombstone;发布、字幕和 28 条 `NEED_REVIEW` 未减少;活动库 `quick_check=ok`、`foreign_key_check=0`。 | +| 文件删除无法随数据库回滚 | 已封口 | 同卷隔离暂存 + manifest;第二个目录移动失败和数据库提交失败均恢复原目录;最终清理失败返回 `cleanup_pending` 并保留恢复证据。 | +| SQLite 备份依赖 WAL/SHM | 已封口 | 三类 Online Backup 均转换为 `journal_mode=DELETE`,回归测试确认不生成配套 sidecar。 | + +Codemap 独立增量复评只上调受本轮实际影响的模块:`Media & Storage 69→73`、`Task Review & Cut 61→65`、`SQLite Persistence 63→71`;其余模块保持原分。仍未解决的高风险包括媒体读取/任务目录路径边界、任务状态跳跃、切片半提交、Job/Publish fencing、Secret 返回和恢复门禁,因此下方 `59/100 · 可用 V1` 仍作为完整审计基线,待 P1 全部验收后再统一重算总健康度。 + +## 0.1 P1A 边界整改状态(2026-08-24) + +| P1A 项目 | 结果 | 验证证据 | +| --- | --- | --- | +| 任务目录越界 | 已封口 | 统一拒绝绝对路径、盘符、`..` 和符号链接逃逸;核心创建/读取/产物路径均受 `TASKS_DIR` 约束。 | +| 同名任务目录竞态 | 已封口 | 新任务使用原子 `mkdir(exist_ok=False)` 预占;并发回归验证同名请求得到不同目录。 | +| 媒体任意文件响应 | 已封口 | 切片、字幕成片、封面分别绑定当前任务受控子目录;任务外路径和 symlink 越界均返回 404;白名单外部原片保持兼容。 | +| 进程树假终止 | 已封口 | Windows `taskkill` 检查退出码并等待退出;失败、超时或无法确认时显式报错。 | +| 全量回归 | 通过 | 边界专项 `52 passed`,完整测试 `533 passed`,Ruff、Compileall、差异检查通过。 | + +Codemap 复评后 `Media & Storage` 为 **84/B**。`API & Runtime` 仍为 **58/D**、`Task Review & Cut` 为 **60/C**;后两者的低分来自尚未整改的 Secret 读取、状态跳跃、切片半提交和 FFprobe 超时,并非本轮路径改动回归。因此项目仍暂定“可用 V1”,待 Job/Publish fencing、状态原子性和本地安全门禁完成后再统一改判。 + +## 0.2 P1B.1 Workflow Job fencing 状态(2026-08-24) + +| P1B.1 项目 | 结果 | 验证证据 | +| --- | --- | --- | +| Claim 代际 | 已封口 | `workflow_jobs.lease_token` 每次领取重新生成,不能用复用的 worker 名称冒充新 attempt。 | +| 旧 Worker 写回 | 已封口 | progress、checkpoint、heartbeat、completed、failed、cancelled、release 均要求当前 `running + owner + token`;旧 token 回归测试全部被拒。 | +| 子进程启动/退出竞态 | 已封口 | `already_claimed` 在业务副作用前验证未过期租约;父进程只处理自己捕获的 token,换代后终止旧子进程且不写失败终态。 | +| 转写/字幕从属副作用 | 已封口 | 转写分块 checkpoint 与最终 Markdown 写入前复核 token;字幕 cleanup 在同一写锁内核对 token、更新从属记录并删除精确临时文件。 | +| 最大尝试次数 | 已封口 | 有效租约即使达到上限也保持运行;只有 queued 或租约已过期的 running job 才会失败。 | +| 回归验证 | 通过 | P1B.1 定向 `92 passed`,完整测试 `546 passed`,Ruff、Compileall、差异检查通过。 | + +本轮尚未在活动库执行加列迁移,也未进行正式服务重启;这是为了避免 P1B.2 尚未完成时反复触碰运行环境。Publish Scheduler/Windows Worker 仍缺 `execution_id` 写回 fencing 与重复 execution 幂等,所以项目成熟度继续保持“可用 V1”,不能只凭 Workflow Job 一侧完成就提前改判为“稳定 V1”。 + +## 0.3 P1B.2 Publish execution fencing 状态(2026-08-24) + +| P1B.2 项目 | 结果 | 验证证据 | +| --- | --- | --- | +| 发布写回代际 | 已封口 | Provider 结果、阶段、成功、导出、失败、人工复核和安全重排队全部要求当前 `PUBLISHING + execution_id`;旧执行写回及同事务事件会回滚。 | +| Worker 重复执行 | 已封口 | 同一 execution 只有身份匹配的完整终态才重放;执行与账号使用进程内互斥和 Windows 跨进程独占锁。同一 job 的旧 execution 进入上传后,新 execution 会被持久化 journal 证据阻断。 | +| 网络不确定性 | 已封口 | 超时/连接重置后先查询 execution;锁仍活跃时保持 `PUBLISHING`,只有确认停在上传前且锁已释放才安全重试,避免盲目重复投稿。 | +| 并发修复、编辑与排期 | 已封口 | retry/repair 在 `BEGIN IMMEDIATE` 中复核源状态和活跃替代任务;发送内容、目标、AI 文案回写和排期更新均校验原状态与 `updated_at`,不能覆盖已领取或并发编辑的任务。 | +| 输入、日志与停机 | 已封口 | Worker 标识、浏览器/导出路径、人工发布 URL、风险 JSON 与敏感字段均收紧;本地发布包完整暂存后原子切换;应用保存 Scheduler Task 并在关闭时停止新扫描、等待当前轮结束。 | +| 回归验证 | 通过 | 发布定向 `144 passed`,完整测试 `610 passed`;Ruff、Compileall、差异检查全部通过,未调用真实平台。 | + +P1B.1 与 P1B.2 已把“旧 Worker/旧发布 execution 覆盖新执行”和“网络不确定时盲目重试”这两类重复副作用边界封住。不过任务主状态仍允许跳跃、切片记录仍可能部分提交,Secret 读取与本地管理员接口也尚未完成收口;因此当前仍维持“可用 V1”,待 P1.3~P1.5 验收后再统一重算健康度并判断是否达到“稳定 V1”。 + +Codemap 独立复评结果:`Publish Scheduler 62→71(C)`、`Publishers & Worker 62→76(B)`;`Publish Center` 因 4750 行 God Service、原始 Secret 响应和旧 API Provider 风险仍维持 `52(D)`。复评未发现 Scheduler/Worker 新的 HIGH 问题;Worker 尚保留“升级前无 identity/损坏 journal 无法归属 job”的兼容边界,不能据此对历史不确定任务自动重投。 + +## 0.4 P1.3a 任务状态与切片原子性状态(2026-08-24) + +| P1.3a 项目 | 结果 | 验证证据 | +| --- | --- | --- | +| 对外任务状态跳跃 | 已封口 | `PATCH /status` 使用显式允许转换表和同事务条件更新;空任务直接标记完成返回 409,已删除任务拒绝任何后续状态写入。 | +| 取消后的明确终态 | 已封口 | 自动 Job 请求取消时任务进入 `CANCELLED`;流水线在步骤前、步骤后和 READY 前复核取消,不再把取消映射为失败或继续完成。 | +| Task/Job 执行代际 | 已封口 | Task 步骤、切片提交和 READY 写回均校验当前 owner、token、未过期 lease 与取消标记;锁等待跨过租约时旧执行写回被拒。 | +| 取消恢复与发布清理 | 已封口 | 公开取消会同步停止活跃自动 Job;子进程退出或租约过期可收敛为 cancelled;只清理本 workflow_job_id 关联且尚未发布的任务。 | +| `READY_TO_PUBLISH` 瞬态 | 已封口 | 流水线成功后稳定保留 `READY_TO_PUBLISH`,Job 可以完成,但任务继续表示“等待人工确认发布”。 | +| 切片半提交 | 已封口 | 一个批次的全部 output clip、旧 active 停用和新 active 激活在同一 `BEGIN IMMEDIATE` 事务提交;第二条插入失败时整批回滚。 | +| 切片并发覆盖 | 已封口 | run number 在写锁内分配;每个 run 使用独立输出目录;较老 run 最后完成时不能覆盖任意已完成的较新 run。 | +| Task 终态 CAS | 已封口 | cut_run 创建与 `cutting` 同事务;切片终态只从当前 `cutting` 写入,其他流程刚更新的状态不会被晚到批次覆盖。 | +| 失败证据遮蔽 | 已封口 | summary 写入异常降级为日志和空路径,原始失败/取消状态与错误保持不变。 | +| 隔离回归 | 通过 | 完整测试 `637 passed`,Ruff 与 Python Compileall 通过;未调用真实 AI、FFmpeg、Chrome 或平台投稿。 | + +本轮没有增加或迁移数据库列,也没有触碰活动数据、真实 AI 或真实投稿。Codemap 增量复评为 `Task Review & Cut 78/B`、`Pipeline & Job Queue 72/C`、`API & Runtime 48/D`、`SQLite Persistence 64/C`。后两项分数下降不是本轮状态修复回归,而是独立复评把原始 Secret 读取、可选鉴权、同步重任务阻塞 async 路由、唯一索引重建失败静默吞掉和无版本迁移重新按实际风险计分。P1.3a 解决的是“状态与切片提交能否可信”这一层;自动流水线的跨进程 step checkpoint、数据库迁移 fail-open、FFprobe/第三方错误边界和 Secret/本地管理员门禁仍按后续独立轮次处理。因此项目继续维持“可用 V1”,不能在剩余 P1 门禁完成前提前宣称“稳定 V1”。 + +## 0.5 P1.3b 自动流水线断点恢复状态(2026-08-24) + +| P1.3b 项目 | 结果 | 验证证据 | +| --- | --- | --- | +| 步骤持久化 | 已封口 | `auto_pipeline_step_v1` 保存连续步骤状态、输入基线和紧凑证据;未知版本、损坏 JSON、越序记录或输入配置变化均 fail closed。 | +| 重启与重领 | 已封口 | 已完成步骤逐项复核;running 的转写、AI、选片、切片、字幕、文案和排期可从持久化产物 reconcile,证据不足才重做。 | +| 失败后重试 | 已封口 | 自动重试在写锁中原位重排旧 Workflow Job 并保留 checkpoint;新 token 接管后可复用已验证副作用,不再创建丢失上下文的新 Job。 | +| AI/切片证据 | 已封口 | AI 同时核对 transcript、分析文件、run payload 和候选集合;切片绑定 active run、候选、受控目录、非空文件、size 与 fingerprint。 | +| 文案/排期/草稿 | 已封口 | 下游从 JSON 恢复内存 context;发布草稿绑定 schedule、草稿字段、允许状态、视频指纹和封面 hash,篡改或文件变化会停止恢复。 | +| Lease 与取消 | 已封口 | `job_id` 无有效 lease 时不执行;旧 Worker 不能写 checkpoint,也不能在新 Worker 接管后取消本轮发布草稿。 | +| 人工门禁 | 保持 | 字幕完成后仍稳定停在 `PENDING_SUBTITLE_REVIEW`;最终任务仍停在 `READY_TO_PUBLISH`,不会自动真实投稿。 | + +完整隔离回归为 `651 passed`,Ruff 与 Python Compileall 全部通过。Codemap 独立复评将 `Pipeline & Job Queue` 从 **72/C 提升到 88/B**;未再发现本轮新增 HIGH,保留的 LOW 是发布创建步骤主要依赖数据库幂等去重而非专用 reconcile,以及尚未进行真实进程重启级故障测试。 + +本轮没有新增数据库列、没有迁移活动库,也没有调用真实 AI、FFmpeg、Chrome、抖音或 B站。恢复语义是“有可验证证据时尽量避免重复”,不是跨 SQLite 与外部服务的严格 exactly-once:若 Provider 已计费但进程在任何 run/文件落盘前崩溃,系统无法证明该次结果,重试仍可能再次调用。数据库迁移 fail-open、第三方超时/429/错误 JSON、字幕批准批次和 Secret/本地管理接口仍属于 P1.3c~P1.5,因此项目成熟度继续维持 **可用 V1**,尚不提前改判为“稳定 V1”。 + +## 0.6 P1.3c 数据库迁移可靠性状态(2026-08-24) + +| P1.3c 项目 | 结果 | 验证证据 | +| --- | --- | --- | +| 迁移账本 | 已封口 | `schema_migrations` 保存 version/name/checksum/applied_at;同版本 checksum 漂移、缺列或 version 非主键均拒绝继续。 | +| 迁移事务 | 已封口(新迁移) | P1.3c 起每条注册迁移使用 `BEGIN IMMEDIATE`,Schema、验证和账本同事务提交;并发 runner 最终只有一条记录。 | +| 唯一索引切换 | 已封口 | 先创建并验证 `_v2` 索引,再删除旧索引;失败回滚后旧索引仍在,不再出现无约束继续启动。 | +| 重复与漂移 | 已封口 | 活动状态含 `NEED_REVIEW`;重复组不自动改写而是明确阻止迁移;索引缺失、错误定义和旧索引残留同样 fail closed。 | +| 迁移前备份 | 已封口 | 已有数据库缺 ledger/v2 索引时,任何写入前先用 SQLite Online Backup 生成可移植快照;异常 ledger 也先备份再拒绝。 | +| 活动库边界 | 未执行写入 | 只读预检显示 ledger 尚未落地、旧索引存在、活动重复组为 0;本轮没有重启正式服务或迁移正式库。 | + +Codemap 独立复评将 `SQLite Persistence` 从 **64/C 提升到 73/C**,原 HIGH“先删索引、重建失败静默吞掉”和 MED“没有迁移账本”均已移除。模块仍为 C:历史列探测和多处 `executescript` 继续属于 pre-ledger compatibility,不能保证整个旧库升级原子;备份恢复工具还缺 FK/ledger/关键索引验证与运行中服务闸门。P1.3c 已完成有限范围目标,但项目整体仍维持 **可用 V1**,下一轮按路线处理 P1.3d,而不是借机全面重写数据库层。 + +## 0.7 P1.3d 字幕批次与跨进程恢复状态(2026-08-25) + +| P1.3d 项目 | 结果 | 验证证据 | +| --- | --- | --- | +| 批量批准原子性 | 已封口 | active revision、cue/质量、批准、Workflow Job 创建/复用和字幕交付模式在同一个 `BEGIN IMMEDIATE` 中提交;任一条失败整批回滚。 | +| Revision 并发覆盖 | 已封口 | 手工保存、单条批准、原片轨生成、切片轨同步和字幕导入均在写锁内重读 current active revision 并使用条件更新;旧页面或迟到执行不能把 active 指针回退。 | +| 成片激活与旧 Worker | 已封口 | 临时文件切换、completed/verified 和 active 版本切换绑定当前 workflow owner/token/expiry/cancel 与当前已批准 revision;迟到结果不能覆盖新版本。 | +| 中断目录与子记录 | 已封口 | 接管会收口同一 Workflow Job 的 `processing/queued` 子记录,只清理本次标记的 `.part.mp4` 和无数据库引用的孤儿最终视频;其他 Job、active 文件、历史结果和可复用 ASS 不删除。 | +| 跨进程 checkpoint | 已封口 | “字幕 DB 已提交但 checkpoint 未写”可恢复;字幕完成与后续 auto-pipeline Job 同事务提交;发布草稿已提交但 checkpoint 未写时按 schedule、任务目录和 workflow 证据恢复。 | +| Lease 与进程启动 | 已封口 | 所有通用 Job 写回拒绝过期 lease;发布批次提交前二次验 lease;子进程启动失败明确标记 failed,不再让 Worker 线程无声退出。 | +| 路径和证据 | 已封口(自动流水线) | Workflow 创建发布草稿前,视频/封面必须位于当前任务受控目录、扩展名合法且非空,并持久化 size/fingerprint,坏草稿不会先提交再被去重掩盖。 | +| 隔离回归 | 通过 | Ruff、Compileall 通过;10 组字幕/Job/自动流水线/checkpoint/状态机/发布关联/版本回滚测试 `167 passed`。活动库前后 size 与 SHA256 一致。 | + +Codemap 最终独立复评未再发现 P1.3d 新的 HIGH。`Subtitle` 为 **69/C**:本轮的批准、激活、接管和 revision 覆盖风险已封口,剩余中风险主要是源轨向多个切片逐条同步的部分完成、损坏 ASR chunk 静默跳过、FFprobe 画布 fallback 以及 1400 行 God Service;前两项中的输入/探测故障归入 P1.4,结构拆分归入 P2。`Pipeline & Job Queue` 为 **87/B**,核心 lease/follow-up/发布恢复与创建时媒体/文案证据边界已经收口;剩余问题主要是恢复计数语义和 God Component 拆分,均不值得在本轮扩大重构。 + +本轮没有增加数据库列、没有迁移活动库、没有调用 AI Provider、Chrome 或真实投稿。项目成熟度继续维持 **可用 V1**:P1.3d 已解决“字幕正常时能跑、进程切换时可能重复或覆盖”的问题,但 P1.4 的第三方超时/错误响应/计费幂等和 P1.5 的密钥/输入/XSS/管理员门禁仍是改判“稳定 V1”的必要条件。 + +## 0.8 P1.4 第三方调用与媒体失败边界(2026-08-25) + +| P1.4 项目 | 结果 | 验证证据 | +| --- | --- | --- | +| AI 错误分类 | 已封口 | 429、连接前失败、5xx、408、超时、空响应、错误 JSON、错误 Schema 和空模型输出均转换为带 `category/safe_to_retry/billing_uncertain` 的统一错误;嵌套 `message` 异常不再泄漏原始 `AttributeError`。 | +| 重试与重复计费 | 已封口(可证明范围) | 只对 429 和确认未建立连接的错误有限重试;5xx、超时、坏 JSON、空响应和已收到但字段异常的结果都不自动重发。普通/综艺解析失败不再二次调用模型,本地协议只在 404/405 时切换。 | +| 远程转写账本 | 已封口 | 火山引擎请求前持久化稳定 request id 和 `requesting`;响应/持久化不确定时写入 `uncertain`。普通 Job 重试 fail closed,只有用户明确“重新生成转写”才允许创建新 run;成功响应字段解析异常同样标记计费不确定。 | +| AI/文案恢复 | 已封口(自动流水线) | 长直播成功窗口继续复用且写回绑定 Workflow lease;自动文案按切片、平台、Provider、模型和输入 fingerprint 逐条原子 checkpoint,多平台恢复必须完整覆盖每个 output×platform。 | +| 失败 fallback | 已收紧 | AI 文案失败后的规则文案携带风险标记并进入 `NEED_REVIEW`,不会伪装为 AI 成功;计费不确定 fallback 被保留为人工复核证据,不在自动恢复中盲目重试。 | +| FFmpeg / FFprobe | 已封口(当前入口) | 任务详情、媒体预检、切片、音频提取和字幕烧录都有超时/停滞边界;切片和音频先写 `.part` 再原子切换,失败清理部分文件;无媒体的独立 ASS 导出使用固定画布,有媒体但探测失败时明确报错。 | +| 取消语义 | 已封口 | 转写进度回调的取消异常穿透 Provider 包装,工作流记录 cancelled 而非 failed;超时或子进程异常会终止进程树并保留可诊断错误。 | +| 隔离回归 | 通过 | P1.4 重点错误矩阵与 fencing 回归 `67 passed`;全量收集 714 项,排除 3 个真实 FFmpeg 参数实例后 `711 passed`、0 失败。所有 HTTP、AI、转写和媒体外部边界均使用 mock/本地逻辑。 | + +Codemap 最终独立复核真实保留了未完成项:`AI Selection 55/D` 中的最高风险是 P1.5 尚未处理的原始 API Key 配置响应;`Pipeline & Job Queue 76/B` 仍是约 1800 行 God Component,文件 checkpoint 与 SQLite lease 也无法提供跨两种介质的严格原子提交;`Transcription` 在二次发现并修复“成功后落账失败”和“读取响应连接重置”两条重复计费边界后,由 **52/D 提升到 64/C**,最终没有 HIGH。保留的中风险包括普通/综艺分析的 partial 语义、faster-whisper 缺少进程级绝对时限、模块级转写运行元数据和大服务拆分,分别进入 P2 的一致性/结构改进,不在本轮扩成全面重构。 + +本轮没有新增数据库列、没有迁移或写入活动库,也没有调用真实 AI、火山引擎、FFmpeg、Chrome、抖音或 B站。当前保证是 **at-most-once 自动重试 + 明确不确定态**,不是跨第三方服务与 SQLite 的数学意义 exactly-once;进程可能在第三方已受理、请求前账本尚未提交这一不可消除的极窄边界退出。由于 P1.5 的密钥、输入、XSS 和本地管理员门禁尚未完成,项目成熟度继续维持 **可用 V1**,不提前改判“稳定 V1”。 + +## 0.9 P1.5 密钥、输入、XSS 与本地管理员边界(2026-08-25) + +| P1.5 项目 | 结果 | 验证证据 | +| --- | --- | --- | +| AI Secret DTO | 已封口 | AI 配置读取不返回火山、分析、发布或本地 Provider Key,也不返回 `.env` 绝对路径;密码框始终留空,空值保存保留原 Secret。 | +| Publish Secret DTO | 已封口 | 配置、账号、OAuth callback、PublishResult、任务详情/历史与 Scheduler 快照均不返回原始 Secret;旧库、嵌套键、异常键前后缀、字符串 Bearer 和损坏 JSON 都经过 fail-closed 脱敏。 | +| 本地管理员门禁 | 已封口(当前部署) | 原生请求同时校验 loopback Host 与 client IP,伪造 `Host: localhost` 不能绕过;非本机请求要求 Bearer Token。Docker 只绑定 `127.0.0.1:8001`,容器代理兼容必须与该绑定成对保留。 | +| 浏览器写入与媒体读取 | 已封口 | `/api` 写请求允许严格同源或明确白名单 Origin,拒绝其他来源;媒体 CORS 不再放行任意 localhost 端口,仅允许明确本地前端与抖音/B站创作者中心。管理员 Token 不再注入 HTML/JavaScript。 | +| 输入与 URL | 已收紧 | AI/ASR/OAuth/发布 API 配置拒绝控制字符、危险 scheme、URL 内嵌凭据、远程明文 HTTP 和异常 Responses 路径;任务错误文本有长度上限,作品 URL 按真实 hostname 校验。 | +| 前端 XSS | 已封口(已确认入口) | 发送关联摘要和排期预览不再把动态内容拼入 `innerHTML`,统一使用 DOM + `textContent`;Jinja 与字幕既有转义保持。 | +| 发布 fail-closed | 已封口 | 不可信作品 URL 和损坏/错误结构 `risk_flags` 进入 `NEED_REVIEW`,不会生成可信链接或继续自动投稿。 | + +独立 Codemap 复评中,P1.5 直接触碰模块已不再出现 Secret/XSS/Host 绕过 HIGH:`Frontend UI 71/C`、`API Runtime 65/C`、`Persistence 72/C`、`Publish Center 65/C`、`Publish Scheduler 74/C`、`Publisher Worker 80/B`。保留项主要是 God Component、旧状态/旧发布模式、同步路由阻塞、错误详情统一脱敏和真实浏览器故障测试,进入 P2 渐进处理。 + +最终隔离验收收集 751 项,排除 3 个真实 FFmpeg 参数实例后 `748 passed`;Ruff、Compileall、两个 JavaScript 语法检查、三套 Docker Compose 配置和差异检查全部通过。活动 SQLite 的大小和 SHA256 在验收前后完全一致。 + +本轮不读取或修改 `.env`,不迁移活动库,也不调用真实 AI、FFmpeg、Chrome 或平台。Codemap 在交叉复评 P1.4 时新增确认三条 AI 一致性 HIGH:候选/run/终态半提交、人工 AI 无持久 lease、普通/综艺局部失败仍可进入切片。因此此节点仍诚实维持 **可用 V1**,先完成独立 P1.4b 后再统一重算“稳定 V1”,不以安全项通过掩盖 AI 恢复缺口。 + +## 0.10 P1.4b AI 结果一致性与稳定 V1 收口(2026-08-25) + +| P1.4b 项目 | 结果 | 验证证据 | +| --- | --- | --- | +| 人工 AI 持久执行 | 已封口 | `/process/ai` 只创建或复用持久 `ai_analysis` Job;Provider 在 Worker 子进程执行,候选、active run 与任务终态绑定当前 lease。 | +| 数据库原子提交 | 已封口 | 候选替换、旧 run 失活、新 run 插入和 `pending_review` 在同一 `BEGIN IMMEDIATE` 中提交;派生 JSON 可从 active run 重建。 | +| 单元计费账本 | 已封口(可证明范围) | 普通、综艺、长直播在请求前持久化单元状态;完成结果带 checksum。running、uncertain、未知/损坏状态、输入指纹变化或 completed payload 校验失败均不自动重发。 | +| 重试与 Provider 切换 | 已封口 | failed/cancelled Job 原位重排队并保留 checkpoint;同一账本不能切换 Provider,避免借新 Job 丢弃旧不确定证据。 | +| 不完整结果门禁 | 已封口 | 三种 profile 统一校验 Schema、模式、布尔质量标记、有限覆盖率、单元/窗口计数和失败清单;缺失、NaN/Inf、矛盾覆盖率或坏数组均 fail closed。 | +| Worker/Task 收尾 | 已封口 | 取消、父进程失败、停机 release 和进程树终止失败均进入明确可恢复状态;不会留下 `queued + cancel_requested=1`。 | +| 安全交叉回归 | 已封口 | 静态媒体 CORS 与管理 API 写入 Origin 分离;抖音/B站创作者中心不能跨站写本地管理接口。 | +| 最终隔离验收 | 通过 | 收集 `788` 项,`785 passed, 3 deselected`;Ruff、Compileall、两个 JavaScript 语法检查、三套 Compose 和 `git diff --check` 全部通过。 | + +Codemap 已刷新全部 13 个模块,`needs_audit=0`。本轮直接相关模块最终为:`AI Selection 70/C`、`Pipeline & Job Queue 66/C`、`Task Review & Cut 68/C`、`API Runtime 63/C`、`Frontend UI 62/C`、`Media & Storage 78/B`;这些模块已无 HIGH。分数没有被人为拉高,因为 God Service、旧状态双协议、前端轮询/提示、历史兼容恢复和可观测性问题仍真实保留。 + +仓库仍有一条 HIGH:`scripts/migrate_task_dirs_to_project_names.py` 使用非 WAL-aware 复制并先移动目录、后统一写库。它不在当前应用、Worker 或发布运行链路中,也没有本轮调用;在 P2 用安全迁移器替换前必须保持停用,绝不能用于活动库。原始 SonarQube 指标继续保留为审计快照,本轮没有为了 Quality Gate 或评分机械改代码。 + +### 当前成熟度重算 + +当前项目为 **78 / 100 · 稳定 V1**,限定范围是:**Windows 本机、单用户、SQLite、人工审核与真实发布人工风控边界**。它不是生产级 SaaS,也不等于真实 Provider、FFmpeg、Chrome 或平台故障演练已经完成。 + +| 维度 | 当前分数 | 依据 | +| --- | ---: | --- | +| 架构合理性 | 7/10 | 本地单体与项目规模匹配;核心 God Service 仍大。 | +| 业务逻辑 | 9/10 | 素材到待发布闭环完整,人工门禁、失败状态与恢复顺序明确。 | +| 代码质量 | 6/10 | 高风险路径已收口,但大文件、兼容层、重复查询与 fallback 仍多。 | +| 数据设计 | 8/10 | 原子批次、fencing、迁移账本与备份边界已建立;旧 helper 仍需渐进收编。 | +| 稳定性 | 8/10 | 超时、取消、接管、部分成功和不确定计费已可恢复;真实外部故障演练仍不足。 | +| 测试 | 9/10 | `785` 项隔离测试通过且活动库强制隔离;3 个真实 FFmpeg 参数实例未在本轮运行。 | +| 安全性 | 8/10 | Secret、loopback/Bearer、Origin、URL 和 XSS 已收紧;仍是本地单用户模型。 | +| 性能与资源 | 7/10 | 已处理明确超时和重复调用;God Service、全量扫描和大文件路径仍有证据化 P2 项。 | +| 可观测性 | 7/10 | Job/Task 日志与不确定状态可诊断;子进程 stderr、静默 fallback 和健康指标仍不足。 | +| 文档与可维护性 | 9/10 | Code Map、工程审计、开发日志、UI 参考和分轮任务文件同步;结构债仍需按 P2 小步处理。 | +| **总分** | **78/100** | **已达到当前产品边界内的稳定 V1;尚未达到生产级。** | + +## 1. Executive Summary(2026-08-24 原始审计快照) + +### 结论 + +原始审计时项目健康度为 **59 / 100**,成熟度属于 **可用 V1**;完成 P0/P1 后的当前结论以 0.10 节 **78/100 · 稳定 V1** 为准。 + +它已经明显超过 Demo:真实的素材接入、转写、AI 选片、人工审核、切片、字幕审核、内容准备、排期和 Windows Chrome Worker 发布链路都存在;当前本机的 FastAPI 服务和发布 Worker 也都在 `127.0.0.1` 正常监听并返回健康状态。500 项隔离测试全部通过,Ruff、Python 编译、前端 JavaScript 语法、PowerShell 脚本解析和 Docker Compose 配置检查也通过。 + +但它还不能称为“稳定 V1”。原因不是页面不好看或代码格式不统一,而是几个关键失败边界尚未封口:测试数据库隔离可被外部环境变量绕过,活动 SQLite 库已经存在 17 条外键不一致,长任务租约缺少 fencing,文件删除与数据库事务无法原子回滚,发布/AI 配置读取接口可能返回原始密钥,路径边界、状态转移和部分成功恢复仍较依赖调用顺序。 + +### 这个项目为什么现在能够运行 + +1. **运行模型简单且适合个人本地工具。** FastAPI、SQLite、文件系统、FFmpeg 和后台 Job 均在一台 Windows 主机上,只有真实发布被明确隔离到 Chrome Worker;没有不必要的微服务、Kafka、Kubernetes、CQRS 或 Event Sourcing。 +2. **有一条真实闭环。** 系统不是展示型壳子;任务可从素材一路走到可发布视频和排期记录。 +3. **已经有可靠性骨架。** SQLite 启用了 WAL、外键和 busy timeout;长任务有持久化 Job、lease、heartbeat、checkpoint;切片/字幕有版本记录;发布不确定时会进入 `NEED_REVIEW`,不会把“可能成功”伪装成确定成功。 +4. **文件产物帮助恢复。** 音频、转写、分析、切片、字幕、封面和发布包分目录保存,许多步骤可以从已有产物继续。 +5. **测试数量和工程脚本已经形成保护网。** 500 项测试本轮全过,CI 还覆盖 Windows/Docker smoke 的入口。 + +### 哪些部分可靠,哪些部分只是尚未暴露问题 + +| 判断 | 代表区域 | 依据 | +| --- | --- | --- | +| 相对可靠 | SQLite WAL、基础 Job 去重、发布任务条件抢占、上传失败清理、发布后不确定状态 | 有明确事务/状态、现有测试和代码证据 | +| 有条件可靠 | 字幕 revision、视频版本、AI 分析、转写 checkpoint、自动流水线 | 正常顺序可用,但并发、重启、旧 Worker、部分提交时边界不完整 | +| 高风险侥幸 | 测试库隔离、现有外键一致性、原始密钥读取、路径解析、删除事务、租约 fencing | 已发现可复现的代码路径或真实数据证据,不是代码风格问题 | + +### 审计限制 + +- 本轮没有调用真实 AI Provider、没有产生计费请求、没有真实投稿抖音/B站,也没有绕过登录、验证码或平台风控。 +- 没有执行 Docker 镜像重建;已验证三组 Compose 配置合法。 +- 没有运行完整真实素材 E2E,因此“500 tests passed”不能替代一次真实长视频和真实账号验收。 +- 项目未安装 Coverage 工具,也没有生成 `coverage.xml`。SonarQube 的 0% 是“没有覆盖率输入”,不是测试实际覆盖率为零。 +- 依赖过期查询遇到 PyPI SSL EOF,未得到可靠的最新版本清单;`pip check` 已通过。 + +## 2. 项目架构图 + +### 2.1 运行拓扑 + +```mermaid +flowchart LR + U[本地用户 / 浏览器] --> UI[Jinja2 + 原生 JS/CSS] + UI --> API[FastAPI Routers] + API --> TS[Task / Review / Cut] + API --> SUB[Subtitle Workbench] + API --> PUB[Publish Center] + + TS --> FS[任务目录 / E 盘素材] + TS --> TR[FFmpeg + Transcription] + TR --> AI[AI Provider + Clip Analyzer] + AI --> CUT[审核 + 视频切片] + CUT --> SUB + SUB --> PUB + + TS --> JOB[Workflow Jobs / Lease / Heartbeat] + TR --> JOB + AI --> JOB + CUT --> JOB + SUB --> JOB + + PUB --> SCH[SQLite Publish Scheduler] + SCH --> REG[Publisher Registry] + REG --> WK[Windows Chrome Worker :8765] + WK --> PLAT[抖音 / B站 / 本地导出] + + DB[(SQLite WAL)] --- API + DB --- JOB + DB --- SCH + FS --- TR + FS --- CUT + FS --- SUB + FS --- WK +``` + +### 2.2 核心模块与职责 + +| 模块 | 职责 | 关键依赖 | Codemap 判断 | +| --- | --- | --- | --- | +| Frontend UI | 任务、审核、字幕、发送中心界面 | API Runtime | 大脚本/God Component,存在鉴权绕行和 `innerHTML` 风险 | +| API Runtime | 启动、配置、路由、健康检查 | 几乎全部业务模块 | 本地单机适配合理,但鉴权、状态边界和同步长任务混杂 | +| Media & Storage | 任务目录、上传、路径、FFmpeg 进程 | Persistence | 正常路径完整,路径安全、非原子命名和删除回滚有缺口 | +| Transcription | 音频提取、本地/火山转写、分块、checkpoint | Media、AI config | 职责过宽;旧转写复用、取消传播和并发元数据存在风险 | +| AI Selection | Provider、Prompt、解析、三种选片 profile | Transcription、Persistence | 宽松解析提升可用性,但会掩盖部分窗口失败和重复调用成本 | +| Task / Review / Cut | 状态、候选审核、切片版本、发送中心同步 | AI、Media、Publish | 核心 God Service;状态跳跃、半提交和查询口径重复 | +| Subtitle | track/revision/cue、AI 建议、渲染、交付门禁 | Cut、FFmpeg、Publish | 设计方向正确,但批量批准和并发激活不是整体原子 | +| Pipeline & Queue | 自动流程、持久化 Job、子进程、lease | 所有处理模块 | 有恢复骨架,缺 owner fencing 与 step 级副作用 checkpoint | +| Publish Center | 文案、封面、账号、草稿、历史、兼容发布 | AI、Cut、Scheduler | 后端最大热点,密钥 DTO 和历史兼容边界是主要风险 | +| Publish Scheduler | 排期、抢占、重试、恢复、人工复核 | Publisher、Worker、DB | 状态策略成熟,但后台 task 引用、全量扫描和幂等仍需加强 | +| Publisher Worker | Chrome、账号、执行日志、真实平台边界 | Scheduler、文件系统 | 正确保留人工确认,但 execution id 幂等和路径字符校验不足 | +| Persistence | Schema、启动迁移、索引、备份恢复 | 全系统 | WAL/备份是优点;无版本账本、索引异常吞掉、真实库已有孤儿关系 | +| Ops & Delivery | CI、启动、备份、诊断、Docker | Runtime、Persistence | 脚本丰富,但文档历史层叠、工具版本与环境存在漂移 | + +### 2.3 Codemap 模块健康评分 + +13 个功能模块均由独立只读子任务按同一固定量表评分:结构 20、正确性 20、可维护性 20、测试 15、性能 15、安全 10。平均分为 **62.6 / 100**;10 个 C、3 个 D,没有 A/B。 + +| 模块 | LoC | 耦合 | 分数 | 等级 | 核心判断 | +| --- | --: | --- | --: | :--: | --- | +| Frontend UI | 13,840 | High | 65 | C | 能支撑当前 UI,但大脚本、XSS 和鉴权请求口径需收敛 | +| API Runtime | 2,713 | Core | 63 | C | 启动简单;鉴权、状态 API 和同步长任务边界不足 | +| Media Storage | 1,300 | High | 69 | C | 正常路径完整;允许根、命名和删除原子性有风险 | +| Transcription | 1,930 | High | 58 | D | 旧结果复用、取消、并发元数据和职责过宽 | +| AI Selection | 4,420 | Core | 63 | C | 兼容强,但 partial/fallback/Secret context 风险明显 | +| Task / Review / Cut | 2,436 | Core | 61 | C | 状态跳跃、半提交、重复查询和测试隔离风险 | +| Subtitle | 2,391 | High | 70 | C | revision 设计较好;批次、并发激活和真实媒体测试不足 | +| Pipeline / Queue | 1,745 | Core | 58 | D | 缺 lease fencing、step checkpoint 和稳定 READY 状态 | +| Publish Center | 5,561 | Core | 52 | D | Secret 泄漏 + 最大 God Service + 多种隐式 partial | +| Publish Scheduler | 1,754 | Core | 62 | C | 发布安全思想正确;execution fencing、后台 Task 和恢复竞态不足 | +| Publisher Worker | 2,116 | High | 62 | C | 真实边界清楚;execution id 幂等/路径和 journal 脱敏不足 | +| Persistence | 3,678 | Core | 73 | C | WAL/备份、迁移账本与关键索引 fail-closed 已具备;旧兼容迁移和恢复闸门仍是债务 | +| Ops / Delivery | 4,347 | Medium | 68 | C | CI/脚本较完整;恢复、Demo fail-open、默认含 `.env` 和文档漂移 | + +Codemap 分数高于总健康度 59 分,是因为总健康度额外纳入了“活动数据库已经发生 FK 违规”和“测试误指真实库可整表删除”这两项运行态 P0 证据,而模块评分主要评价对应代码边界。 + +### 2.4 核心业务和状态流 + +```text +素材进入 + → 媒体预检 / 任务目录 + → 音频提取 + → 带时间戳转写 + → AI 分段选片 + → 人工审核候选 + → 生成并验证切片 + → 字幕草稿 + → 人工审核并烧录,或明确跳过 + → 标题 / 简介 / 话题 / 封面 + → 创建排期 + → Scheduler 抢占任务 + → Publisher Registry + → Windows Chrome Worker + → PUBLISHED / FAILED / NEED_REVIEW / CANCELLED +``` + +关键状态并不是一个统一状态机,而是四组并行状态: + +- `tasks.status`:大写自动流水线状态 + 小写手动流程状态。 +- `workflow_jobs.status`:`queued/running/completed/failed/cancelled`。 +- `subtitle_jobs.status` 与 revision 激活状态。 +- `publish_jobs.status`:`DRAFT/WAITING/SCHEDULED/PUBLISHING/PUBLISHED/EXPORTED/FAILED/NEED_REVIEW/CANCELLED`。 + +这解释了系统为什么能承载多步流程,也解释了恢复困难的来源:一个用户动作可能同时更新任务主状态、Job 状态、文件产物、字幕 revision 和发布草稿。 + +### 2.5 修改影响半径 + +| 修改点 | 可能被影响的下游 | +| --- | --- | +| `task_service.py` | 任务详情、候选审核、切片、字幕、发送中心同步、Dashboard | +| `publish_service.py` | 文案、封面、账号、草稿、历史、兼容 Publisher、页面上下文 | +| `database.py` | 启动、所有 Service、迁移、索引、Prompt 种子、备份恢复 | +| `job_service.py` / `job_worker.py` | 转写、切片、自动流水线、字幕以及未来 Job 类型 | +| `storage_service.py` | 上传、媒体预览、任务删除、转写、切片、发布 Worker | +| `app.js` / `publish-center.js` | 多个页面的写请求、轮询、状态呈现和本地鉴权 | + +## 3. 项目健康度 + +| 维度 | 分数 | 依据 | +| --- | --: | --- | +| 架构合理性 | 7/10 | 单体 + SQLite + Worker 符合个人本地项目规模;没有过度设计。扣分来自 Service 直接访问 DB、动态导入和几个 God Service。 | +| 业务逻辑 | 7/10 | 真实闭环完整,发布不确定状态处理正确;扣分来自任意状态跳跃、部分成功和四套状态并行。 | +| 代码质量 | 5/10 | Ruff/编译通过;但 Sonar 479 个 Smell、聚合 Cognitive Complexity 7307,多个 1k-4k 行热点。 | +| 数据设计 | 4/10 | WAL、FK、备份存在;但真实库 17 条 FK 违规、迁移无版本账本、索引失败被吞、部分操作非原子。 | +| 稳定性 | 5/10 | Job、checkpoint、NEED_REVIEW 提供恢复骨架;lease fencing、取消、旧产物复用和文件/DB 原子性不足。 | +| 测试 | 7/10 | 500 项全过,覆盖多种核心服务;但无 Coverage 数据、真实 E2E 不完整,且存在可清空真实库的隔离风险。 | +| 安全性 | 4/10 | `.env` 被忽略、路径删除有部分保护;但原始密钥读取、默认无写鉴权、DOM XSS、路径穿越/本地文件响应构成高价值问题。 | +| 性能与资源 | 7/10 | 当前单机规模没有明确系统性瓶颈;扣分来自高频轮询、全量排期扫描、完整 PCM/大文件内存和重复 AI fallback。 | +| 可观测性 | 6/10 | 有任务日志、进度、checkpoint、Scheduler/Worker health 和 NEED_REVIEW;但 `/health` 过浅,子进程 stderr 被丢弃,静默 fallback 多。 | +| 文档与可维护性 | 7/10 | README、架构、流程、开发日志和 CI 文档丰富;但历史段落与当前行为混排,少量模块说明已过时。 | +| **总分** | **59/100** | **可用 V1:正常路径可用,异常和并发边界尚不足以称为稳定 V1。** | + +## 4. SonarQube 客观指标 + +### 4.1 扫描信息 + +- Server:SonarQube Community Build `26.8.0.126808` +- Scanner:SonarScanner CLI `8.0.1.6346`(官方 Docker 镜像) +- Project key:`niuma-studio-local-audit` +- 扫描范围:`app/`、`scripts/`,测试目录为 `tests/`;vendor、图片、手工真实 Provider 脚本和缓存已排除。 +- 有效代码:`38,963` ncloc。 +- 分析任务:成功;服务器处理耗时约 17 秒,首次完整 Scanner 运行约 15 分 55 秒。 +- 本地 Dashboard:`http://127.0.0.1:9000/dashboard?id=niuma-studio-local-audit` +- 审计环境提示:Sonar 端口只映射到 `127.0.0.1`,但默认管理员凭据仍有效;扫描用临时 Token 已在查询结束后撤销。应单独修改 Sonar 管理员密码,这不属于 NiuMa 生产代码整改。 + +### 4.2 指标 + +| 指标 | 实测 | 解读 | +| --- | ---: | --- | +| Bugs | 29 | 其中大量是 HTML 语义容器/label 规则;真正优先核对的是 Scheduler 后台 Task 引用和少数 Python 规则。 | +| Vulnerabilities | 2 | 一项是自定义临时目录安全,一项是允许人工填 `http://` 平台链接;都需上下文审查,不等于已被攻击。 | +| Security Hotspots | 0 | Sonar 未报 Hotspot,但源码 Review 仍发现密钥响应、DOM XSS、路径和可选鉴权问题。 | +| Code Smells | 479 | 主要由 FastAPI 未声明响应码 128 项、CSS 属性/兼容规则、复杂度和重复字符串构成。 | +| Duplication | 0.2% | 总体很低;局部最高是 `transcript_service.py` 3.6%、`task_query_service.py` 3.3%。 | +| Coverage | 0.0% | **无 `coverage.xml` 输入**;不能作为真实覆盖率。JUnit 已导入 500 项测试且 100% 成功。 | +| Cognitive Complexity | 7,307 | 聚合值;说明复杂度集中明显,不代表每个文件都差。 | +| Cyclomatic Complexity | 7,608 | 聚合值;需结合文件/函数热点使用。 | +| Maintainability Rating | A | 技术债比率 0.3%;规则估算对本项目偏乐观。 | +| Reliability Rating | C | 与 29 个 Bug 指标对应。 | +| Security Rating | D | 与 2 个 Vulnerability 指标对应。 | +| Technical Debt | 3,499 分钟 | 约 58 小时 19 分;这是规则修复估算,不是整改项目工期。 | +| Quality Gate | OK | 当前 Gate 返回 `conditions=[]`,因此“OK”没有实质门禁含义。 | + +问题严重度总数:4 Blocker、134 Critical、316 Major、56 Minor,共 510 项。这里的严重度是 Sonar 规则严重度,不直接等同本报告的 P0/P1。 + +### 4.3 复杂度热点 + +| 文件 | ncloc | Cognitive | Cyclomatic | 判断 | +| --- | ---: | ---: | ---: | --- | +| `app/services/publish_service.py` | 4,245 | 965 | 1,001 | 全项目最大后端热点;同时承担多种领域职责。 | +| `app/static/js/publish-center.js` | 2,104 | 931 | 796 | 页面状态、轮询和事件高度集中。 | +| `app/static/js/app.js` | 2,298 | 585 | 935 | 多页面全局脚本,鉴权和错误处理口径不一。 | +| `app/services/publish_scheduler.py` | 1,340 | 295 | 281 | 调度、claim、恢复和终态集中。 | +| `app/services/subtitle_data_service.py` | 1,176 | 288 | 270 | track/revision/cue/导入导出/波形职责过宽。 | +| `app/services/transcript_service.py` | 1,187 | 280 | 251 | FFmpeg、Provider、分块、进度、解析集中。 | +| `app/services/ai/ai_clip_analyzer.py` | 760 | 267 | 211 | Prompt、解析、修复、归一化集中。 | +| `app/db/database.py` | 1,563 | 141 | 147 | 行数极大但复杂度不是首位;主要风险是迁移/一致性而非分支数。 | + +Sonar 单函数最高确认项包括: + +- `publish_service.py:2067`:Cognitive Complexity 68。 +- `subtitle_data_service.py:950`:47。 +- `auto_publish_service.py:22`:41。 +- `subtitle_data_service.py:871`:41。 +- `long_live_talk_analyzer.py:83`:39。 +- `job_worker.py:173`:32。 + +### 4.4 值得修与低价值规则 + +值得修: + +- `publish_scheduler.py:1403` 创建后台 Task 后不保存引用:属于真实生命周期风险,应和运行架构一起修。 +- Cognitive Complexity 超过 30 且位于发布、Job、字幕、AI 主路径的函数:先用 characterization test 锁定行为,再小步拆分。 +- 密钥、路径、状态、并发和数据一致性问题:即使 Sonar 没报,也应优先于格式/响应文档规则。 +- CSS 对比度和缺少 label:影响可访问性,可在 P2/P3 独立处理。 + +低价值或需人工确认: + +- 128 个 `python:S8415` 要求在 FastAPI `responses` 中声明每个异常码,主要改善 OpenAPI 文档,不能排在状态一致性之前。 +- 大量重复“任务不存在”“BEGIN IMMEDIATE”字符串,不应为了规则分数强行抽象成全局常量。 +- `publish_scheduler.py:961` 允许人工确认链接使用 HTTP:建议收紧为 HTTPS,但它不是当前最主要的系统安全风险。 +- `storage_service.py:79` 的临时目录规则需要结合 Windows ACL 和实际 E 盘目录判断,不能仅凭规则就改动存储布局。 +- HTML `dt/li` 容器和 label 问题应修,但不应把 29 个 Bugs 机械解释为 29 个核心业务缺陷。 + +Sonar 官方说明:Python Coverage 需要先由外部测试工具生成报告,并通过 `sonar.python.coverage.reportPaths` 导入;Scanner 本身不会计算覆盖率。参考 [SonarScanner CLI](https://docs.sonarsource.com/sonarqube-server/analyzing-source-code/scanners/sonarscanner) 与 [Python test coverage](https://docs.sonarsource.com/sonarqube-server/analyzing-source-code/test-coverage/python-test-coverage)。 + +## 5. 问题优先级 + +### P0:先消除数据损坏和不可逆风险 + +#### P0-1 测试可在错误环境下清空真实数据库 + +| 字段 | 内容 | +| --- | --- | +| 文件 / 位置 | `tests/conftest.py:11-14`;`tests/test_task_query_service.py:175-195` | +| 模块 | 测试体系 / Persistence | +| 来源 | Codemap + Code Overhaul 共同发现 | +| 原因 | `conftest.py` 使用 `os.environ.setdefault`,外部已有 `DATABASE_PATH` 时不会强制切换到测试库;该测试的 autouse fixture 每个用例前后无条件删除 8 张核心表。 | +| 实际影响 | 如果 pytest 继承活动库路径,真实任务、候选、Job、切片、字幕和发布索引可能被直接清空。 | +| 发生概率 | 日常 CI 低;开发者终端已设置生产 `DATABASE_PATH` 后手工跑测试时中等。 | +| 修改收益 / 成本 / 风险 | 收益极高;成本低;修改测试基础设施风险低。 | +| Blast Radius | 整个活动 SQLite 数据库。 | +| 推荐方案 | 测试启动时强制创建唯一临时目录和数据库;若路径不是 pytest 临时目录或文件名命中活动库则 fail closed;给清理 fixture 加防误删断言。 | + +本轮 500 项测试使用了显式临时 `test_workflow.sqlite3`,所以此次测试没有以活动库为目标;这不降低代码本身的风险等级。 + +#### P0-2 活动 SQLite 已存在 17 条外键不一致 + +| 字段 | 内容 | +| --- | --- | +| 文件 / 位置 | 运行态 `data/workflow.sqlite3` 的只读 `PRAGMA foreign_key_check`;Schema 位于 `app/db/database.py` | +| 模块 | Persistence / Publish / Subtitle | +| 来源 | Code Overhaul + 真实数据库只读核验 | +| 原因 | 历史删除/迁移/兼容流程允许发布和字幕记录继续引用已不存在的 `output_clip`。 | +| 实际影响 | 16 条 `publish_jobs → output_clip` 和 1 条 `subtitle_jobs → output_clip` 违规;其中发布记录含 14 条 `CANCELLED`、2 条 `NEED_REVIEW`。复核、详情联查和未来约束迁移可能失败。 | +| 发生概率 | 已经发生,不是推测。 | +| 修改收益 / 成本 / 风险 | 收益极高;成本中;直接修数据风险高,必须先确定保留历史还是恢复父记录。 | +| Blast Radius | 发布历史、人工复核、字幕记录和后续 Schema 迁移。 | +| 推荐方案 | 先做 WAL-aware 备份和 dry-run 报表;逐条判定“恢复父记录 / 将引用置空 / 归档历史”;修复后要求 `foreign_key_check` 为 0,再加回归检查。 | + +补充:`PRAGMA quick_check` 返回 `ok`,说明数据库文件页结构正常;“文件没坏”与“关系一致”是两件事。 + +#### P0-3 永久删除把文件系统操作放在数据库提交之前 + +| 字段 | 内容 | +| --- | --- | +| 文件 / 位置 | `app/services/task_lifecycle_service.py:299-372`;`app/services/storage_service.py:533-554` | +| 模块 | Media Storage / Task Lifecycle / Persistence | +| 来源 | Codemap + Code Overhaul 共同发现 | +| 原因 | `BEGIN IMMEDIATE` 后先执行 `shutil.rmtree()`,随后才更新数据库并 commit;SQLite rollback 无法恢复已删除目录。 | +| 实际影响 | 文件删除成功但数据库更新/提交失败时,数据库记录回滚,托管切片、字幕和发布包却已经消失。外部唯一原片通常会保留,但派生产物仍会丢失。 | +| 发生概率 | 低,但磁盘权限、SQLite 锁、异常退出时成立。 | +| 修改收益 / 成本 / 风险 | 收益高;成本中;直接改变删除顺序有兼容风险。 | +| Blast Radius | 单个任务的全部项目托管产物。 | +| 推荐方案 | 改为“两阶段可恢复删除”:先把目标移动到项目内隔离区并记录 manifest,数据库提交后再异步清理;失败时可移回。 | + +### P1:核心安全、并发和恢复边界 + +#### P1-1 配置读取接口返回原始 Secret + +| 字段 | 内容 | +| --- | --- | +| 文件 / 位置 | `app/services/publish_service.py:564-599`、`:846-956`;`app/routers/publish.py:34-76`;`app/services/ai_config_service.py:318-335`;`app/routers/settings.py:10-16` | +| 模块 | Publish Center / AI Config / API Runtime | +| 来源 | Codemap + Code Overhaul 共同发现;Sonar 未发现 | +| 原因 | `_normalize_config/_normalize_account` 从完整 DB row 开始,仅新增 masked 字段,没有移除 `client_secret/access_token/refresh_token`;AI context 也返回完整 `values`。 | +| 实际影响 | 浏览器 DOM、开发者工具、本地调用方或代理日志可读取 OAuth/API Secret。 | +| 发生概率 | 调用配置/账号接口即可触发,中高。 | +| 修改收益 / 成本 / 风险 | 收益极高;成本低;风险低,但要保证保存页面不依赖回显原值。 | +| Blast Radius | 所有 AI/ASR 配置、平台 Client Secret、全部发布账号 Token。 | +| 推荐方案 | 建立只读响应 DTO,只返回 `configured` 和 masked 值;写接口使用“空值表示不变”的专用字段;增加接口级“原始值绝不出现”测试。 | + +#### P1-2 默认写鉴权关闭,部署边界依赖启动方式 + +| 字段 | 内容 | +| --- | --- | +| 文件 / 位置 | `app/main.py:91-106`;`app/core/config.py:80,247`;`docker-compose.yml:14-16,34` | +| 模块 | API Runtime / Ops | +| 来源 | Codemap + Code Overhaul 共同发现 | +| 原因 | 只有 `LOCAL_ADMIN_TOKEN` 非空才验证写请求;默认值为空。Docker 内 Uvicorn 监听 `0.0.0.0`,Worker Token 也可为空。 | +| 实际影响 | 如果端口被映射到局域网,其他访问者可改任务、排期、账号和配置。个人本机 `127.0.0.1` 风险较低。 | +| 发生概率 | 当前原生服务低;Docker/LAN 场景中等。 | +| 修改收益 / 成本 / 风险 | 收益高;成本低到中;风险是启动体验改变。 | +| Blast Radius | 全部写 API 与发布操作。 | +| 推荐方案 | 原生模式继续 loopback;Docker 非 loopback 时要求 Token 或明确开发开关,启动日志给出强提醒;不要引入多用户 RBAC。 | + +#### P1-3 任务目录和媒体响应的允许根边界不闭合 + +| 字段 | 内容 | +| --- | --- | +| 文件 / 位置 | `app/services/storage_service.py:174-179,218-230,298-315,557-565`;`app/routers/media.py:58-77` | +| 模块 | Media Storage / API Runtime | +| 来源 | Codemap 独立审计 + Code Review | +| 原因 | `task_dir_name` 的 parts 未统一走安全相对路径校验;已存在媒体路径直接返回;媒体响应只检查 exists/is_file;目录名分配是查询后返回而非原子占位。 | +| 实际影响 | 构造异常目录名可能越出任务根;异常 DB 路径可能让媒体接口返回本地其他文件;并发任务可能获得同名目录并覆盖。 | +| 发生概率 | 正常 UI 低;导入旧数据、恶意本地请求或并发重试时中等。 | +| 修改收益 / 成本 / 风险 | 收益高;成本中;风险是旧路径兼容需盘点。 | +| Blast Radius | 任务素材、切片、字幕、发布 Worker 可见文件。 | +| 推荐方案 | 所有路径入口统一 `resolve + relative_to(allowed_root)`;目录名只允许受控片段;使用原子目录占位或 DB unique;为旧 `/workspace/tasks` 映射保留显式迁移层。 | + +#### P1-4 Workflow Job 租约没有 fencing + +| 字段 | 内容 | +| --- | --- | +| 文件 / 位置 | `app/services/job_service.py:246-279,329,400-474`;`app/services/job_worker.py:38,173-237` | +| 模块 | Pipeline & Queue | +| 来源 | Codemap + Code Overhaul 共同发现 | +| 原因 | claim 时记录 `lease_owner` 和 attempt,但 progress/checkpoint/completed/failed/cancelled 更新只按 job id;旧子进程也不核对当前 owner。 | +| 实际影响 | 租约过期并被新 Worker 接管后,旧 Worker 恢复可覆盖新 attempt 的进度、checkpoint 和终态,造成 split-brain、重复 AI/FFmpeg 或错误完成。 | +| 发生概率 | 正常短任务低;休眠、长任务、卡顿、超时和进程恢复时中等。 | +| 修改收益 / 成本 / 风险 | 收益极高;成本中;风险中,需要迁移兼容现有 Job。 | +| Blast Radius | 单个 Job 及其任务、切片、字幕、AI 调用和下游发布草稿。 | +| 推荐方案 | 使用单调递增 `attempt_token`;所有 heartbeat、checkpoint、终态更新带 `WHERE id=? AND lease_owner=? AND attempt_token=? AND status='running'`;旧 Worker 更新失败后立即停止。 | + +#### P1-5 Scheduler 后台 Task 未保存引用,Worker 请求也缺少执行幂等 + +| 字段 | 内容 | +| --- | --- | +| 文件 / 位置 | `app/services/publish_scheduler.py:1399-1404`(Sonar `python:S7502`);`scripts/publish_host_worker.py:262-294` | +| 模块 | Publish Scheduler / Publisher Worker | +| 来源 | SonarQube + Code Overhaul 交叉发现 | +| 原因 | `asyncio.create_task()` 返回值不保存;Worker 收到相同 `execution_id` 时不先返回已完成 journal,而是再次执行发布。 | +| 实际影响 | Scheduler task 理论上可能被过早回收;网络超时或调用重试可能重复投稿,增加账号风控风险。 | +| 发生概率 | Scheduler 回收低到中;重复请求在网络异常时中等。 | +| 修改收益 / 成本 / 风险 | 收益高;成本低到中;风险低。 | +| Blast Radius | 全部排期扫描;重复执行影响单条平台内容和账号。 | +| 推荐方案 | App lifespan 显式保存/取消 scheduler task;Worker 以 execution id 建立幂等状态机,已完成直接返回,进行中返回冲突/同一结果。 | + +#### P1-6 状态可任意跳跃,切片结果可部分提交 + +| 字段 | 内容 | +| --- | --- | +| 文件 / 位置 | `app/services/task_lifecycle_service.py:152`;`app/routers/tasks.py:174-179,390-399`;`app/services/video_cut_workflow_service.py:18-45,248-301` | +| 模块 | Task / Review / Cut | +| 来源 | Codemap + Code Overhaul 共同发现 | +| 原因 | 状态 API 只验证枚举,不验证合法转移/前置产物;同步切片无统一 Job 去重;run number 的 MAX 与 INSERT 分离,每条 output clip 独立提交。 | +| 实际影响 | 空任务可被标记完成;重复点击可能产生并发 run;中途异常会留下部分 output clip、processing run 和已完成主状态。 | +| 发生概率 | 正常单用户低;重试、重复请求、异常写入时中等。 | +| 修改收益 / 成本 / 风险 | 收益高;成本中到高;风险中,需兼容历史状态。 | +| Blast Radius | 单任务状态、切片版本、字幕来源和发布草稿。 | +| 推荐方案 | 先建立合法转移表和 characterization tests;同步接口统一进入持久化 Job;run number 与 run 创建同事务并加 unique;整批 output clip 与 active 切换采用阶段提交。 | + +#### P1-7 转写会误复用旧结果,取消也可能变成失败 + +| 字段 | 内容 | +| --- | --- | +| 文件 / 位置 | `app/services/transcription_checkpoint_service.py:22-70`;`app/services/transcript_service.py:260-284`;`app/services/transcript_workflow_service.py:318-396` | +| 模块 | Transcription | +| 来源 | Codemap 独立审计 + Sonar 复杂度证据 | +| 原因 | 大文件指纹只覆盖大小和首尾各 1 MiB;`transcript.md` 存在即可判完成;宽泛异常包装会吞掉取消异常类型。 | +| 实际影响 | 中部被替换的视频可能复用旧转写;强制重转失败后旧文件仍被当当前结果;用户取消显示为 failed。 | +| 发生概率 | 文件替换/重试低到中;取消路径中等。 | +| 修改收益 / 成本 / 风险 | 收益高;成本中;风险中,重新计算全量 hash 会有大文件 IO 成本。 | +| Blast Radius | 单任务转写、AI 选片、字幕和后续切片。 | +| 推荐方案 | 使用可解释的源版本标识(文件元数据 + 分块 hash/显式导入版本);完成标记必须绑定 run;取消异常单独透传;旧结果作为历史而非隐式 current。 | + +#### P1-8 AI 部分窗口失败和协议 fallback 可能产生不完整结果及重复成本 + +| 字段 | 内容 | +| --- | --- | +| 文件 / 位置 | `app/services/ai/ai_clip_analyzer.py:90-158`;`app/services/ai/local_model_provider.py:19-35`;`app/services/ai/variety_comedy_analyzer.py:513-528` | +| 模块 | AI Selection | +| 来源 | Codemap + Code Overhaul 共同发现 | +| 原因 | 通用 profile 只要部分窗口产生候选就返回;全局评审失败可降级;任意 `AIProviderError` 都可能触发第二协议调用。 | +| 实际影响 | 看似成功的候选只覆盖部分素材;429/认证/超时也可能重复调用,增加 Token/费用并掩盖故障。 | +| 发生概率 | 模型格式或网络波动时中等。 | +| 修改收益 / 成本 / 风险 | 收益高;成本中;风险中,需要明确不同 profile 的产品门槛。 | +| Blast Radius | 单次 AI 任务的候选质量、费用和发布时间。 | +| 推荐方案 | 为每个 profile 定义最低窗口覆盖率和 partial 状态;只对幂等且可重试错误退避;协议 fallback 仅用于明确协议不匹配;UI 展示失败窗口与费用次数。 | + +#### P1-9 前端存在 DOM XSS,且部分写请求绕过统一鉴权封装 + +| 字段 | 内容 | +| --- | --- | +| 文件 / 位置 | `app/static/js/app.js:1-12,211,1729`;`app/static/js/publish-center.js:2152` | +| 模块 | Frontend UI / API Runtime | +| 来源 | Codemap + 手工源码 Review;Sonar 未发现 | +| 原因 | 外部消息/任务标题进入 `innerHTML`;许多写请求直接 `fetch`,没有经过注入 Bearer Token 的 `apiFetch`。 | +| 实际影响 | 用户输入或 API 消息可破坏页面/执行脚本;启用 `LOCAL_ADMIN_TOKEN` 后部分正常操作反而 401。 | +| 发生概率 | XSS 低到中;开启 Token 后鉴权失败高。 | +| 修改收益 / 成本 / 风险 | 收益高;成本低到中;风险低。 | +| Blast Radius | 任务详情、审核、字幕和发送中心页面。 | +| 推荐方案 | 动态文本统一 `textContent/createElement`;需要 HTML 的内容只接受白名单模板;所有 API 请求走同一封装并加超时/错误规范化。 | + +#### P1-10 启动迁移核心 fail-open(P1.3c 已完成,保留历史兼容边界) + +| 字段 | 内容 | +| --- | --- | +| 文件 / 位置 | `app/db/database.py`;`scripts/backup_restore.py:305-383,529-634` | +| 模块 | Persistence | +| 来源 | Code Overhaul + Codemap | +| 原因 | P1.3c 已建立 ledger、备份和 v2 索引 fail-closed;剩余问题是早期列探测/`executescript` 仍先于账本迁移执行,恢复包也未验证 FK/ledger/关键索引或阻止运行中替换。 | +| 实际影响 | 新迁移不会再静默丢唯一约束;但非常旧的 Schema 在兼容 helper 中途失败仍可能留下可重试的部分结构,运行中恢复还可能形成新旧连接视图分离。 | +| 发生概率 | 正常新库低;跨多个历史版本升级或运行中恢复时低到中。 | +| 修改收益 / 成本 / 风险 | 核心收益已取得;剩余改造收益中高、成本高、风险高,应拆成独立恢复/legacy migration 轮次。 | +| Blast Radius | 历史数据库升级与整库恢复,不影响正常业务请求路径。 | +| 推荐方案 | 后续新 Schema 全部走现有 ledger;独立增加 restore 离线闸门和 FK/ledger/index 校验,再按版本逐步收编旧 helper,不做一次性全面重写。 | + +### P2:有限范围逐步偿还 + +| 问题 | 文件 / 模块 / 来源 | 原因与实际影响 | 概率 / Blast Radius | 收益 / 成本 / 风险 | 推荐方案 | +| --- | --- | --- | --- | --- | --- | +| 发布中心 God Service | `publish_service.py`;Codemap + Overhaul + Sonar | 文案、封面、账号、历史、兼容 Publisher 和页面上下文共处,Cognitive 965 | 修改发布功能时高;影响全部发布路径 | 高 / 高 / 中高 | 先按只读查询、Secret DTO、内容准备、Job 创建、Legacy adapter 分 4-5 轮抽取,不改外部 API。 | +| 前端 God Scripts | `app.js`、`publish-center.js`;Codemap + Sonar | 全局状态、轮询、事件和请求口径集中 | UI 改动时中;多个页面 | 中高 / 中 / 中 | 先抽统一 request/polling/store,再按页面拆文件;不引入 React/Vue。 | +| 多 Service 直接 DB | 约 20 个文件;Codemap + Overhaul | Repository 边界只在发布域局部存在,事务跨 Service 难组合 | 新功能时高;跨域 | 高 / 高 / 高 | 只为需要原子操作的聚合增加小型 Repository/Unit of Work,不做全量 ORM 重写。 | +| 字幕批量部分成功 | `subtitle_auto_workflow_service.py:87-117`;Codemap | clip 逐项批准后才建 Job,中途失败留混合状态 | 中;单任务所有字幕 | 高 / 中 / 中 | 预检全量、单事务记录批次,再由 Job 执行;保留 revision 历史。 | +| 发布排期全量扫描与 5 秒轮询 | `publish_scheduler.py:260-281`、`publish-center.js:2227`;Overhaul | 每轮读取全部 SCHEDULED,页面重复请求 jobs/accounts/health | 当前低,规模增长后确定;发布中心 | 中 / 低 / 低 | SQL 直接筛选到期 + 合适索引;前端合并状态端点或退避。 | +| 超长媒体内存峰值 | `transcript_service.py:675`、`subtitle_data_service.py:761`;Overhaul/Codemap | Base64/PCM 完整载入内存后处理 | 长视频时中;单请求/进程 | 中 / 中 / 低 | 流式上传/分块编码;波形边读边降采样。 | +| 文件输出并非统一原子替换 | 上传、切片、analysis/metadata 写入;Overhaul | 崩溃/磁盘满可留下截断文件 | 低到中;单产物 | 高 / 中 / 中 | 统一 `.part` + fsync + replace + manifest;先覆盖最关键 source/clip/analysis。 | +| 可观测性不足 | `/health`、`job_worker.py:175`、多处 silent fallback;Codemap/Overhaul | 健康只看进程,子进程 stderr 丢弃,损坏数据常被当默认值 | 故障时高;诊断全链路 | 高 / 中 / 低 | 分离 liveness/readiness;保留受控 stderr 摘要和错误码;fallback 必须记录来源。 | +| 无覆盖率和类型检查 | `requirements-dev.txt`、CI;Sonar + Operator | 有 500 测试但无 line/branch coverage,mypy/pyright/tsc 不存在 | 持续;全项目 | 中高 / 中 / 低 | 先加 coverage 数据,仅设核心模块“不得下降”;Python 类型检查从新/高风险模块渐进启用。 | +| 文档当前态与历史态混排 | `docs/TASK_FLOW.md:3-14,114`、`NEXT_STEPS.md:288-309`;Overhaul | 顶部已是字幕审核恢复,旧段仍说自动模式跳过字幕 | 新人阅读时高;运维/验收 | 中 / 低 / 低 | 保留历史但显式标“历史行为”,新增唯一 Current State 页面。 | +| 环境版本漂移 | `.venv` 与 `requirements-dev.txt`;Operator | 实际 pytest 9.0.3 / Ruff 0.15.16,声明 9.1.1 / 0.16.1 | 本机与 CI 差异时中 | 中 / 低 / 低 | 重建受控 dev venv 或在 CI/本机打印并校验工具版本。 | + +### P3:低优先级与代码洁癖 + +| 问题 | 依据 | 处理建议 | +| --- | --- | --- | +| 128 个 FastAPI 响应码未写入 OpenAPI | Sonar `python:S8415` | API 对外稳定后按 Router 批量补,不影响 P0/P1。 | +| 重复字符串常量 | Sonar `python:S1192` | 只抽真正表达领域含义的常量;“任务不存在”不必全局设计模式化。 | +| HTML 语义容器、label、对比度 | Sonar Bugs/Web/CSS | 作为独立可访问性轮次修,不和状态/事务改动混合。 | +| 105 个文件不符合 Ruff format | `ruff format app tests --check` | 不在整改前批量格式化,避免淹没真实 diff;按触碰文件渐进格式化。 | +| Pydantic V1 `@validator` 警告 | 7 条测试警告 | 等 Pydantic 升级轮次集中处理。 | +| CSS 未定义别名与旧属性 | Codemap + Sonar CSS | 先修真实视觉失效的变量,忽略纯兼容规则。 | + +## 6. 专项检查 + +### 6.1 数据与数据库 + +#### 真实库只读结果 + +- 文件:`data/workflow.sqlite3`,处于 WAL 模式,审计时存在活跃 `-wal/-shm`,所以没有复制、迁移或写入。 +- `PRAGMA quick_check`:`ok`。 +- `PRAGMA foreign_key_check`:17 条违规,详见 P0-2。 +- `PRAGMA user_version`:`0`,没有可追踪的 Schema 版本号。 +- 关键规模:35 tasks、232 candidates、335 output clips、589 publish jobs、837 publish events。 +- `tasks.status` 同时存在大写自动状态和小写手动状态;这是代码定义的一部分,不是简单大小写脏数据,但会增加查询、迁移和恢复复杂度。 + +#### Schema 判断 + +| 检查项 | 判断 | 风险 | +| --- | --- | --- | +| Primary Key | 主要使用文本 UUID,适合本地分布式生成 | 低 | +| Foreign Key | 连接时启用 FK,但历史数据已有违规,部分 FK 无明确删除策略 | 高 | +| Unique Constraint | 有发布活动任务等局部唯一索引;重建失败会被吞 | 高 | +| Index | 关键状态/时间有部分索引;失败不可见,全量排期扫描仍存在 | 中 | +| 状态字段 | 字符串为主,无 DB CHECK;四组状态并行 | 高 | +| 时间字段 | 多为 ISO 字符串;跨午夜排期有专门逻辑 | 中 | +| 删除策略 | 任务行保留历史、托管文件删除;语义接近“隐藏并清理媒体” | 高(命名和原子性) | +| 更新策略 | 多 Service 各自连接和 commit;跨文件/跨表事务难组合 | 高 | +| 回滚策略 | 数据库可 rollback,文件系统和外部平台不可自动 rollback | 高 | +| Migration | 启动探测式迁移,部分有备份,但无统一版本账本 | 高 | + +### 6.2 稳定性故障矩阵 + +| 场景 | 当前行为 | 恢复判断 | 主要证据 / 缺口 | +| --- | --- | --- | --- | +| API 超时 / 网络断开 | AI/发布通常转异常或失败;旧 API 发布可能无法判断平台是否已接收 | 部分可恢复 | 超时后不总是进入 `NEED_REVIEW`;重试可能重复请求 | +| 第三方 500 / 429 | 转成 Provider error,缺统一指数退避和状态码分类 | 不稳定 | 短暂故障可能直接失败;fallback 可能重复费用 | +| AI 空内容 | 多数路径判无候选/失败 | 较明确 | 某些 Provider/归一化会用默认值继续 | +| AI 错误 JSON / Markdown | 支持 fence、修复、trailing comma、`literal_eval` | 可用但过宽 | 提高兼容性,也可能把坏结果包装成合法结果 | +| 字段缺失 / null | 候选和文案大量默认值 | 部分成功 | 数据可继续流动,但质量退化可能不醒目 | +| 上传失败 | 普通异常路径会清理任务目录 | 较明确 | 进程突然退出时直接写最终路径可留截断文件 | +| 文件损坏 | 多处有 FFprobe;部分读取静默 fallback | 部分可恢复 | transcript/progress/analysis 损坏可能被当“不存在” | +| 数据库异常 | busy timeout/WAL 有帮助 | 不稳定 | 多处静默 `except sqlite3.Error`,可能走错误 fallback | +| Worker 异常退出 | lease/heartbeat 可重新领取 | 部分可恢复 | 无 fencing,旧 Worker 恢复后仍可写 | +| Task 中途失败 | 有失败状态和日志 | 部分可恢复 | step 副作用 checkpoint 不完整,summary 二次失败可遮蔽原错 | +| 重复执行 | Job 创建有基础去重,Publish claim 有条件更新 | 部分可防 | 同步切片、execution id、旧 Worker 缺幂等 | +| 程序重启 | 持久化 Job/文件产物帮助续跑 | 有条件恢复 | 进程内转写锁、取消集合和 Pipeline context 丢失 | +| 并发请求 | SQLite `BEGIN IMMEDIATE` 覆盖部分关键动作 | 不完整 | 目录命名、cut run、字幕 revision/track 存在 check-then-act 竞态 | + +总体回答:**系统失败后通常能留下一个状态,但不总能保证该状态唯一、最新且与文件/外部副作用一致。** 这正是从“可用 V1”到“稳定 V1”的主要差距。 + +### 6.3 测试体系 + +| 类型 | 当前情况 | 本轮结果 | 关键缺口 | +| --- | --- | --- | --- | +| Unit Test | Provider 解析、状态 helper、规则、路径等较多 | 纳入 500 项,全过 | monkeypatch 较多,真实资源边界不足 | +| Integration Test | SQLite、WAL/FK、Job、字幕、发布调度较丰富 | 全过 | 文件 + DB + 进程崩溃的一致性不足 | +| API Test | Tasks、Publish、Settings 等有覆盖 | 全过 | Secret 不回显、Token 开启、路径越界和并发写缺口 | +| Browser / E2E | 发送中心有 Playwright,字幕多为页面/静态检查 | 环境存在时纳入;可被 `importorskip` | 无真实“上传→转写→AI→切片→字幕→发布”闭环 | +| Smoke | CI 有 Windows/Docker 入口;当前真实本机服务/Worker 健康 | 两个 health 均 `ok` | `/health` 不检查 DB/Job/Scheduler/FFmpeg;未重建 Docker 镜像 | +| Coverage | 未配置 | Sonar 显示 0%,仅代表无报告 | 无法知道核心行/分支实际覆盖 | +| Typecheck | 未配置 mypy/pyright/tsc | 工具不存在 | `Any` 和动态 dict 边界只能靠运行时测试 | + +本轮实际命令结果: + +- `pytest --collect-only -q`:500 项。 +- 完整 pytest:`500 passed, 9 warnings`,约 90 秒;JUnit 再跑 `500 passed`。 +- Ruff lint:通过。 +- Ruff format check:失败,105 个文件会被重排;没有执行格式化。 +- `python -m compileall -q app tests scripts`:通过。 +- 导入 `app.main`:通过。 +- 全部前端 JS `node --check`:通过。 +- 20 个 PowerShell 脚本 Parser:通过。 +- 基础/dev/demo 三种 Docker Compose 配置:通过。 +- `pip check`:通过。 + +目前仍完全或主要依赖人工点击的核心验证: + +1. 真实长视频端到端质量和耗时。 +2. 火山/本地转写真实失败恢复。 +3. 真实 AI Provider 的 429、超时、坏 JSON、计费重试。 +4. 字幕批量烧录的真实 FFmpeg/NVENC 失败恢复。 +5. Chrome 登录态、平台 UI 变化、投稿后不确定结果。 +6. 程序休眠/重启、旧 Worker 复活、同 execution id 重放。 + +### 6.4 安全 + +#### 部署语境 + +当前设计适合 **个人本机、loopback、受信任操作者**。它不是内部多用户系统,也不具备对外 SaaS 所需的身份、授权、租户隔离、审计和限流边界。审计不建议为当前阶段引入企业级 RBAC;建议把“只允许本机”变成明确且可验证的默认条件。 + +| 检查项 | 结果 | 优先级 | +| --- | --- | --- | +| API Key / Secret / `.env` | 当前只跟踪 `.env.example`;`.env`、浏览器数据、DB 被忽略;但读取接口会回显原始 Secret | P1 | +| Git 历史 Secret | 文件名扫描未发现历史 `.env`/私钥/DB;内容模式命中经脱敏核验为 `data-task-*` 属性误报 | 已核验,无确认泄漏 | +| Authentication | 可选 Bearer Token,默认空 | P1(非 loopback) | +| Authorization / 用户隔离 | 无,多用户不在当前范围 | 本地可接受,SaaS 不可接受 | +| Input Validation | Pydantic/扩展名/部分路径有校验;状态、目录片段和 Worker ID 边界不足 | P1 | +| SQL Injection | 未发现高置信度用户输入直接拼 SQL;大多数参数化 | 低 | +| XSS | 两处动态 `innerHTML` 高置信度 | P1 | +| CSRF | 不使用 Cookie 会降低经典 CSRF;Token 为空时写 API 本身无认证 | 被鉴权问题覆盖 | +| 任意文件上传 | 有扩展名/预检/允许根;突然退出的原子性不足 | P2 | +| Path Traversal / 本地文件读取 | `task_dir_name` 和媒体已存在路径边界不闭合 | P1 | +| 敏感日志 / Token 日志 | Worker/Provider 整体有隐藏意图;原始响应 DTO 和 URL 错误消息仍可能泄漏 | P1/P2 | +| CORS | 没有发现对所有来源开放写请求;本地静态/媒体 GET 有专门处理 | 低 | +| Rate Limit | 无 | 本地低;对外不可接受 | + +Sonar 的 2 个 Vulnerability: + +1. `storage_service.py:79`:把自定义上传目录设为系统临时目录。需检查实际 Windows ACL;当前不能仅凭该规则判定目录“publicly writable”。 +2. `publish_scheduler.py:961`:人工确认作品链接允许 HTTP。建议改 HTTPS,但其影响低于 Secret DTO、路径和鉴权。 + +Sonar 没有发现原始 Token 响应、DOM XSS 和目录拼接,说明静态扫描必须和业务 Review 组合使用。 + +### 6.5 性能与资源 + +有明确证据、值得逐步处理的性能问题: + +- 发送中心打开时每 5 秒重复刷新 jobs、accounts 和 Scheduler health;后端 jobs 查询还会重复读账号/Worker 健康。 +- Scheduler 每轮读取全部 `SCHEDULED` 后在 Python 解析/筛选到期时间;当前 589 条 publish jobs 尚小,未来规模增长会线性放大。 +- 旧 API multipart 发布用 `read_bytes()` + `b''.join()`,大视频产生约双份内存。 +- 字幕 waveform 把完整 PCM 读入内存后再降采样。 +- 火山转写 chunk 完整 Base64 入内存。 +- 本地模型协议 fallback 对所有 ProviderError 尝试第二协议,可能重复模型调用和 Token 消耗。 +- 通用 AI 分段失败后仍可能返回部分候选;如果人工重跑整任务,会重复已成功窗口的成本。 +- `publish_service.py` 批量上下文存在账号查询 N+1;本地小规模影响暂低。 + +没有证据支持现在引入缓存服务、分布式队列或复杂前端构建系统。先减少重复请求、修 SQL 筛选、流式处理大文件和精确 Provider 重试即可。 + +### 6.6 Dead Code / Legacy / Mock 清单 + +本轮仅列出,没有删除。 + +| 候选 | 证据 | 初步判断 | +| --- | --- | --- | +| `app/services/ai_clip_service.py` | 只含硬编码三条候选的 `generate_candidate_clips_placeholder`,全仓未发现调用 | fake output,优先删除候选 | +| `publish_repository.update_execution_phase()` | 全仓未发现生产调用;测试还断言 Worker 不含该旧行为 | 可能安全删除,但需再做动态调用确认 | +| `execute_opencli_send_job()` / `execute_api_publish_job()` | 仍被 compat Publisher 调用 | Legacy,不能直接删 | +| `publishers/opencli_compat.py`、`api_compat.py` | Registry 仍注册;旧任务模式可能引用 | 需数据盘点后删除 | +| `publishers/page_scripts.py` | 通过 wrapper 回调 `publish_service` 私有函数 | 丑但仍是 Worker 兼容胶水,暂不删 | +| `job_service` 的 `ai_analysis` / `publish` Job 类型 | execute_job 没有对应分支 | stub/未来预留,需确认产品方向 | +| `transcript_service.allow_fallback` / `TRANSCRIPTION_FALLBACK_PROVIDER` | 未发现 `allow_fallback=True` 生产调用 | 需要确认后删除 | +| `/workspace/tasks` 旧路径映射 | `storage_service.py:298-340` | Docker/历史数据兼容,暂不删 | +| 旧 AI_REMOTE 配置别名 | 运行时仍同步新旧字段 | 需配置迁移后删除 | +| 旧小写/新大写发布状态与字段别名 | DB 和服务并存 | 必须先迁移真实历史数据 | +| 历史文档段落 | 与当前字幕行为并存 | 应归档/标历史,不是直接删除证据 | + +### 6.7 当前已有且应保留的工程资产 + +- SQLite WAL、busy timeout、连接级 FK。 +- `create_or_get_active_job()` 的事务性基础去重。 +- 发布 Scheduler 的条件 claim 和上传后 `NEED_REVIEW` 原则。 +- Windows Worker 的 execution journal 和账号锁。 +- 长直播 checkpoint 和 90% 覆盖率门槛。 +- 字幕 track/revision/cue 历史和发布前验证门禁。 +- FFmpeg/FFprobe、备份恢复、启动健康检查脚本。 +- CI 的 Windows + Docker smoke 入口、敏感文件检查和独立 demo DB。 + +## 7. 技术债 Top 10 + +排序依据不是“代码难看”,而是 **发生风险 × 业务影响 × 未来维护成本 × 修复收益**。 + +| 排名 | 技术债 | 风险 / 影响 | 未来成本 | 修复收益 | 为什么排在这里 | +| --: | --- | --- | --- | --- | --- | +| 1 | 测试数据库 fail-open,可执行整表删除 | 极高 / 全库 | 每次手工测试都可能触发 | 极高 | 修改成本最低,却能消除最不可接受的数据损坏入口。 | +| 2 | 活动库已有 17 条 FK 违规,迁移无版本账本 | 已发生 / 发布与字幕历史 | 每次迁移都会放大 | 极高 | 先恢复数据可信度,其他重构才有可靠基线。 | +| 3 | 原始 API/OAuth Secret 经读取接口回显 | 高 / 全部凭据 | 接口越多越难清理 | 极高 | 本地项目也不应把原始 Token 发到浏览器。 | +| 4 | Workflow/Publish execution 缺 lease fencing | 高 / 重复执行与错终态 | 新 Job 类型会复制缺陷 | 极高 | 是“平时能跑、异常顺序就错”的典型核心债。 | +| 5 | 存储路径允许根、目录名和媒体响应边界不统一 | 高 / 本地文件与任务产物 | 旧路径越多越难迁移 | 高 | 安全和数据覆盖同时受影响。 | +| 6 | 永久删除的文件系统与 DB 事务不可回滚 | 低概率高损失 / 单任务全部产物 | 删除类型增加会放大 | 高 | 必须在继续扩展清理功能前改成可恢复两阶段。 | +| 7 | 状态任意跳跃、切片/字幕批量结果可部分提交 | 中高 / 下游全链路 | 状态越多越难收敛 | 高 | 当前四套状态的核心一致性问题。 | +| 8 | 转写旧产物复用、取消传播和并发元数据不可靠 | 中高 / AI 与字幕输入 | 长视频越多越贵 | 高 | 错转写会悄悄污染所有下游结果。 | +| 9 | AI 部分窗口/fallback 缺明确成本和质量门槛 | 中 / 质量与费用 | Provider/profile 增加会放大 | 高 | 需要产品语义,不是简单加 retry。 | +| 10 | `publish_service.py` 与前端大脚本成为发布域 God Components | 中 / 发布全域 | 每个新功能持续提高回归成本 | 高 | 应在安全和状态边界稳定后渐进拆分,不能先大重构。 | + +## 8. 删除候选清单 + +### 可以安全删除(仍应放在独立清理 PR 中验证) + +| 候选 | 依据 | 删除前最低验证 | +| --- | --- | --- | +| `app/services/ai_clip_service.py` | 仅返回三条硬编码 fake candidates,全仓静态引用为 0;架构文档对它的描述已过时 | 再做动态导入搜索;跑 AI/自动流水线测试;同步修 `docs/ARCHITECTURE.md` | +| `publish_repository.update_execution_phase()` | 全仓未发现生产调用,测试明确要求 Worker 不依赖它 | 跑 Publisher/Scheduler 全套测试 | +| `publish_service.py` 中被后续定义覆盖的首组 `STATUS_LABELS/STATUS_TONES` | 同文件后续重新定义,前值不会生效 | 用导入后对象断言 + 发布历史/页面测试确认 | + +“可以安全删除”是静态审计结论,不代表本轮已删除;本轮没有改动这些文件。 + +### 需要确认后删除 + +| 候选 | 需要确认什么 | +| --- | --- | +| `opencli_compat.py`、`api_compat.py` 及两个 legacy execute 函数 | 真实 DB 是否仍有对应 `publish_mode/provider` 记录;是否还需要历史重试 | +| `job_service` 的 `ai_analysis` / `publish` 类型 | 是未来计划还是未完成 stub;若保留必须接线,若无计划应删定义和 UI | +| `TRANSCRIPTION_FALLBACK_PROVIDER` / `allow_fallback` | 是否有项目外脚本或用户工作流依赖 | +| AI_REMOTE_* 旧配置别名 | 活动 `.env` 和持久化设置是否仍使用旧名;不得读取/输出 Secret 值,只盘点键名 | +| `/workspace/tasks` 旧路径映射 | Docker 历史记录是否仍保存该路径 | +| task_service 末尾查询兼容 wrappers | 所有 Router/模板是否已迁到 task_query_service | +| 历史文档中旧流程段落 | 先决定迁入 `docs/history/` 还是加“历史行为”标记 | + +### 暂时不要删除 + +| 代码 / 数据 | 原因 | +| --- | --- | +| B站后端 Publisher、历史记录和字段 | 当前前端主推抖音不等于后端历史可删除;删除前必须迁移真实记录。 | +| `manual_export` | 是真实发布不可用时的安全、可验证降级,不是 fake success。 | +| `page_scripts.py` 与浏览器兼容胶水 | 虽然耦合丑,但真实 Worker 当前仍依赖。 | +| `NEED_REVIEW`、execution journal、Worker artifacts | 它们是防止不确定结果被误判成功的核心安全证据。 | +| `database_backup_service.py`、WAL-aware backup/runtime wrapper | 这是数据库整改前的安全基础;应整理而非先删。 | +| 旧数据库列/状态别名 | 真实库已有历史数据,必须迁移并验收后才能删除。 | +| 活动 `.db-wal/.db-shm` | 表示存在写连接,不能当缓存随意删除。 | + +## 9. 暂时不要动的地方 + +1. **不要把 SQLite 改成 PostgreSQL,也不要引入 ORM 大迁移。** 当前规模下 SQLite 合理,问题是约束、迁移账本和事务边界,而不是数据库品牌。 +2. **不要把单体拆微服务。** 当前故障多来自跨状态/跨文件原子性;拆服务只会增加网络和分布式一致性问题。 +3. **不要为了 Sonar A 评级先拆 `database.py`。** 先修真实数据和 migration ledger,再按版本迁移小步拆分。 +4. **不要改掉 `NEED_REVIEW` 和上传后不自动重试策略。** 这是平台发布最重要的防重复边界之一。 +5. **不要重写已经工作的字幕 revision 模型。** 它保留历史、可审核、可回滚;只需加强批量事务和 owner/version 条件。 +6. **不要直接统一删除大写/小写状态。** 真实数据、查询和 UI 均依赖;先建立映射、数据统计和迁移测试。 +7. **不要先批量格式化 105 个文件。** 大 diff 会掩盖 P0/P1 修复,也增加合并冲突。 +8. **不要先清理所有 Legacy Publisher。** 必须先盘点 DB 模式和历史任务,确保没有可恢复任务被切断。 +9. **不要大改 E 盘任务目录布局。** 文件路径已被数据库、字幕、发布包和 Worker 引用;路径安全可以在保持布局的前提下修。 +10. **不要因为 `styles.css/app.js` 很大就先重写前端。** 先修 `innerHTML`、统一 API 请求和轮询,再按页面渐进拆分;不引入 React/Vue。 + +## 10. Impact × Effort 矩阵 + +| | 低成本 | 中成本 | 高成本 | +| --- | --- | --- | --- | +| **高影响** | 测试库 fail-closed;Secret DTO;保存 scheduler task;URL hostname 校验;统一前端鉴权 | 路径允许根;lease fencing;AI partial 门槛;转写 run 绑定 | FK 历史修复;状态/切片原子化;migration ledger | +| **中影响** | CSS 变量;文档当前态;工具版本校验 | 覆盖率/核心 E2E;日志和 readiness;轮询/SQL due | `publish_service` 渐进拆分;DB 访问边界收敛 | +| **低影响** | OpenAPI responses;重复字符串;小型命名 | 可访问性批次;格式化触碰文件 | 全前端重写、ORM、微服务——当前不做 | + +## 11. 整改路线图 + +每一轮都限制范围、可单独测试、可单独回滚。**不要合并为“全面重构”。** + +| 轮次 | 有限范围 | 独立验收 | 回滚方式 | +| --- | --- | --- | --- | +| P0.1 测试数据库保护 | 仅 `tests/conftest.py` 和危险 fixture;强制临时库、危险路径拒绝 | 在故意设置活动库路径时 pytest 应在收集前拒绝;正常临时库 500 项全过 | 回滚测试基础设施提交,无生产数据迁移 | +| P0.2 数据一致性盘点与修复 | WAL-aware 备份、17 条孤儿 dry-run、逐类处理策略 | backup hash/quick_check/FK check;修复后 0 违规;应用只读页面回归 | 整库备份恢复;保留修复 manifest | +| P0.3 可恢复删除 | 单个任务删除改为隔离区 + manifest + DB commit + 延迟清理 | 模拟 DB commit 失败、文件移动失败、进程中断;均能恢复 | 把隔离目录移回;回滚代码 | +| P1.1 Secret DTO 与本地鉴权 | AI/Publish 读取 DTO;非 loopback 启动门禁;不做多用户系统 | API 响应扫描不含原值;Token on/off、原生/Docker 组合测试 | 兼容写入字段保留;回滚 API DTO 提交 | +| P1.2 路径和 Worker 标识边界 | `task_dir_name`、media FileResponse、execution/account id;不改目录布局 | traversal、绝对路径、symlink、异常 DB 路径、重复 execution 测试 | 兼容旧合法路径白名单;回滚验证层 | +| P1.3 Workflow lease fencing | 增 attempt token;所有更新带 owner/token/status;旧 worker fail closed | 两 Worker、租约过期、休眠恢复、旧终态覆盖测试 | Schema 先兼容可空列;回滚代码保留列 | +| P1.4 Publish execution 幂等与 Scheduler 生命周期 | 保存/await task;execution journal 幂等;repair 条件锁 | 同 execution 重放、旧 execution 写回、并发 repair、优雅关停 | 回滚应用层状态机,journal 数据仍兼容 | +| P1.5 Task/Cut/Subtitle 状态与批次原子性 | 先合法转移表,再切片 run 和字幕批准批次;不改 UI 主流程 | 非法跳跃拒绝、并发 run、批次中途失败、旧 active 保留 | 每项独立提交;特性开关保留旧入口短期回退 | +| P1.6 Transcription 正确性 | run/source 绑定、取消透传、损坏产物显式状态 | 大文件中部变化、旧文件、取消、进度 JSON 损坏、重启测试 | 旧 transcript 只降为历史,不删除 | +| P1.7 AI 质量与成本门禁 | profile 覆盖率、partial 状态、错误分类、有限退避 | 429/500/超时/坏 JSON/部分窗口/重复调用次数测试 | 配置化门槛,可回滚到旧行为但保留日志 | +| P1.8 Migration ledger 与索引不变量(P1.3c 核心已完成) | 已为未来迁移增加版本/checksum 账本、备份和 v2 索引 preflight;旧 helper 暂留 | 重复数据、checksum/索引漂移、并发 runner 和失败回滚已覆盖 | 每版前备份;一个 migration 一次提交;旧 helper 后续逐条收编 | +| P2.1 覆盖率与真实故障测试 | 加 Coverage;先锁 P0/P1 核心模块,不追漂亮总百分比 | CI 产 `coverage.xml`,Sonar 能导入;核心门槛不得下降 | 移除阈值不影响生产代码 | +| P2.2 可观测性 | readiness、Job/Worker 错误码、受控 stderr、fallback 日志 | DB/Worker/FFmpeg 故障时状态明确且不泄密 | 回滚观测层,无数据迁移 | +| P2.3 Publish Service 渐进拆分 | 先 Secret DTO/query,再内容准备,再 job,最后 legacy adapter | 每轮 API contract、发布状态机和浏览器测试全过 | 一轮一个提交,旧 facade 保持接口 | +| P2.4 原生前端渐进拆分 | 先 request + escape,再 polling,再按页面模块化 | Token 开启、XSS、轮询、字幕/发送中心 Playwright | 保留原函数 facade,不换框架 | +| P2.5 Dead/Legacy 清理 | 每次只处理一个已确认无引用候选 | `rg`/动态导入盘点 + 全测试 + 历史 DB 模式统计 | 单候选单提交,直接 revert | +| P3.1 文档和格式 | Current State 文档、历史标记、触碰文件格式化、可访问性 | 文档链接、Ruff、页面 smoke | 不与业务改动混合,单独回滚 | + +## 12. 最终判断 + +### 当前为什么能跑 + +因为项目选择了合适的本地单体架构,并且已经建立了 SQLite WAL、持久化 Job、文件产物、版本化字幕/切片和发布人工复核这些实用保护;500 项测试也覆盖了大量正常路径和若干失败路径。 + +### 哪些是可靠的 + +单机正常顺序下的核心业务闭环、基础 Job 去重、发布 claim、上传失败清理、字幕 revision 历史、发布不确定状态和 Windows Worker 隔离是可信资产。 + +### 哪些只是侥幸没出问题 + +测试是否永远指向测试库、旧 Worker 是否永远不会复活、目录名是否永远来自正常 UI、文件删除后数据库是否永远能提交、AI 是否总有足够窗口成功、配置接口是否只被可信浏览器访问——这些都依赖环境或顺序,并未被代码强制保证。 + +### 应该先改什么 + +先保护测试和真实数据,再封 Secret/路径/lease/删除/状态边界;随后补覆盖率和故障可观测性;最后才拆 God Service、清 Legacy 和整理格式。 + +### 哪些不要碰 + +不要换架构、不要改掉人工复核、不要先删除历史兼容、不要直接迁移状态和目录、不要用大规模格式化或前端重写掩盖真正风险。 + +本报告完成后应停止审计阶段。下一步应由用户先确认 P0/P1 顺序,再逐轮整改。 diff --git a/STABILITY_P1_BOUNDARIES_TASK.md b/STABILITY_P1_BOUNDARIES_TASK.md new file mode 100644 index 0000000..c3ccc45 --- /dev/null +++ b/STABILITY_P1_BOUNDARIES_TASK.md @@ -0,0 +1,63 @@ +# 稳定 V1 P1A 边界加固任务书 + +## 背景 + +P0 已封住测试误删、外键异常和永久删除半成功等数据安全风险。Codemap 与源码交叉审查仍确认三类高风险边界:任务目录可被恶意 `task_dir_name` 越界、并发任务可能获得同一目录、媒体接口可能把数据库中的任意本地路径作为文件响应;Windows 子进程终止也没有确认进程树真实退出。 + +## 目标 + +1. 所有任务目录解析都必须限定在 `TASKS_DIR`,拒绝绝对路径、盘符、`..` 与符号链接逃逸。 +2. 新任务目录名使用原子目录预占,避免并发上传或重试复用同一目录。 +3. 源视频继续支持 `STORAGE_ROOT` / `TASKS_DIR` / `ALLOWED_MEDIA_ROOTS`,但切片、字幕成片和封面响应必须绑定到当前任务及对应产物子目录。 +4. 子进程终止必须确认退出;无法确认时显式失败,调用方不得继续按“已终止”处理。 + +## 允许修改范围 + +- `app/services/storage_service.py` +- `app/routers/media.py` +- `app/services/managed_process_service.py` +- 与本轮边界直接相关的测试 +- `DEVELOPMENT_LOG.md`、`NEXT_STEPS.md`、Codemap 和本任务书 + +## 禁止修改范围 + +- 不改变 AI Provider、模型、登录或认证配置。 +- 不改变真实投稿、字幕、切片业务语义。 +- 不访问真实 NAS 文件,不触发真实第三方请求或投稿。 +- 不修改数据库 Schema,不删除历史数据。 +- 不删除遗留 `move_task_directory_to_trash`,且不改变其目标必须尚不存在的语义。 + +## 已确定实现要求 + +- 路径判断使用解析后的父子关系,不使用容易出现前缀碰撞的字符串判断。 +- 任务目录名允许普通文件名中的 `~`,但拒绝绝对路径、盘符、空路径、根目录和任何 `..` 跳转。 +- 原子分配通过 `mkdir(exist_ok=False)` 完成;只有任务创建链显式启用预占。 +- 数据库读取失败不得静默伪装为“无任务目录”。 +- 切片只允许 `05_clips`(兼容旧 `clips`),字幕成片只允许 `06_subtitled`;封面只允许 `07_covers`。 +- 对旧 C 盘任务目录只保留受控兼容读取,不扩大到任意路径。 +- Windows `taskkill` 的超时、启动失败、非零退出码和退出确认失败均需有明确行为;若进程已并发退出可视为成功。 + +## 验收标准 + +- `..\\outside`、绝对路径、盘符和符号链接逃逸不能创建或读取任务目录。 +- 两个并发同名目录预占获得不同名称,不发生文件覆盖。 +- 数据库中指向任务目录外的切片/字幕路径返回 404;合法任务产物仍可读取。 +- 外部白名单源视频读取保持兼容。 +- 模拟 `taskkill` 成功、并发退出、非零失败、超时均有确定测试。 +- 定向测试、全量测试、Ruff 和 Compileall 通过。 + +## 测试命令 + +```powershell +pytest -q tests/test_storage_boundaries.py tests/test_managed_process_service.py tests/test_media_storage_lifecycle.py +pytest -q +ruff check app tests scripts +python -m compileall -q app scripts +``` + +## 返回格式 + +- 修改文件与行为边界。 +- 测试命令、通过数量与失败证据。 +- 对外部原片、遗留目录和现有调用方的兼容结论。 +- Commit、分支、Push、PR 状态和下一轮 P1 风险。 diff --git a/STABILITY_P1_DATABASE_MIGRATION_TASK.md b/STABILITY_P1_DATABASE_MIGRATION_TASK.md new file mode 100644 index 0000000..c74d339 --- /dev/null +++ b/STABILITY_P1_DATABASE_MIGRATION_TASK.md @@ -0,0 +1,87 @@ +# P1.3c 数据库迁移账本与唯一索引 Fail-Closed + +## 背景 + +当前 `init_db()` 通过表结构探测、`ALTER TABLE` 和 `executescript()` 兼容历史数据库,但没有可查询的迁移版本账本。更高风险的是 `_create_indexes()` 会先删除发布任务的活动唯一索引,再静默吞掉所有索引创建异常;一旦重建失败,应用仍会启动并允许重复活动发布任务进入数据库。 + +## 本轮目标 + +1. 建立最小、可扩展的 `schema_migrations` 迁移账本。 +2. 每条新迁移记录稳定版本、名称、校验和与完成时间。 +3. 版本相同但校验和变化时拒绝启动,避免迁移定义被静默改写。 +4. 发布活动任务唯一索引采用“先建立并验证新版索引,再删除旧索引”的顺序。 +5. 索引缺失、定义漂移、重复数据或 SQLite 异常时明确失败,不再静默继续运行。 +6. 迁移执行与账本写入位于同一 SQLite 事务中;失败不写成功记录。 +7. 已有数据库在 P1.3c 首次写入前创建 SQLite Online Backup;备份失败则不开始迁移。 + +## 允许修改 + +- `app/db/database.py` +- 新增与本轮直接相关的隔离数据库测试 +- `DEVELOPMENT_LOG.md` +- `NEXT_STEPS.md` +- `PROJECT_AUDIT.md` +- `.codemap/*` +- 本任务说明 + +## 禁止修改 + +- 不直接运行或迁移正式 `workflow.sqlite3`。 +- 不删除、合并或自动修复正式库中的历史发布记录。 +- 不修改业务状态机、发布调度、Provider、Worker 或页面逻辑。 +- 不引入 Alembic 或其他新依赖,不重写全部历史兼容迁移。 +- 不处理 P1.4 Provider 超时或 P1.5 Secret/Auth。 + +## 已确定实现要求 + +### 迁移账本 + +- `schema_migrations.version` 为主键,并保存 `name`、`checksum`、`applied_at`。 +- P1.3c 及后续正式 Schema 变更必须通过有序迁移注册表执行。 +- 已应用迁移必须校验 checksum 和数据库不变量;不能只看到版本号就假定成功。 +- 账本只记录完整成功的迁移,不把失败尝试伪装成已应用。 +- 旧的列探测兼容逻辑本轮保留,并明确视为 pre-ledger compatibility,不宣称已经完成全面 Alembic 化。 + +### 唯一索引 + +- 新版索引使用独立版本化名称,约束同一 `output_clip_id + platform + publish_mode` 只能存在一条活动发布任务。 +- 活动状态继续包括 `DRAFT`、`WAITING`、`SCHEDULED`、`PUBLISHING`、`NEED_REVIEW`。 +- 新索引创建和定义验证成功前,不删除旧版保护索引。 +- 发现活动重复数据时拒绝迁移,并返回不包含 Secret 的诊断信息;不擅自决定保留或取消哪条发布记录。 +- 已应用迁移启动时仍验证索引存在、唯一性和定义,防止手工删索引或 Schema 漂移后继续运行。 + +### 事务与并发 + +- 每条账本迁移通过 `BEGIN IMMEDIATE` 串行化。 +- Schema 修改、验证和账本写入同事务提交;任一步失败全部回滚。 +- 两个本地进程同时初始化同一库时,后到进程应读取已完成账本,而不是重复写入或吞掉锁错误。 + +## 验收标准 + +1. 新建隔离库初始化后恰有一条 P1.3c 迁移记录,重复初始化不新增记录。 +2. 新版唯一索引存在且旧版索引已在成功后移除。 +3. 迁移 checksum 漂移时明确拒绝启动。 +4. 已应用迁移的索引缺失或定义错误时明确拒绝启动。 +5. 活动重复数据导致索引无法建立时,启动失败、迁移不入账、数据不被自动改写。 +6. 模拟新版索引验证失败时,旧版索引仍存在,证明切换顺序和回滚有效。 +7. 索引列表中的其他 SQLite 异常不再被静默吞掉。 +8. 定向测试、完整 Pytest、Ruff、Compileall 通过;测试只使用临时数据库。 +9. 正式数据库文件的大小、mtime、hash 均不因本轮验证改变。 + +## 测试命令 + +```powershell +.\.venv\Scripts\python.exe -m pytest -q tests/test_schema_migration_ledger.py tests/test_database_backup_service.py tests/test_publish_scheduler_state_machine.py tests/test_job_fencing.py tests/test_long_live_foundation.py +.\.venv\Scripts\python.exe -m pytest -q +.\.venv\Scripts\python.exe -m ruff check app tests +.\.venv\Scripts\python.exe -m compileall -q app tests scripts +``` + +## 返回格式 + +- 迁移账本与唯一索引切换方式。 +- 失败/漂移/重复数据的 fail-closed 证据。 +- 正式数据库未被修改的证据。 +- 定向与全量验证结果。 +- 仍保留的 pre-ledger 历史迁移边界。 +- Commit、分支、PR 与下一独立轮次。 diff --git a/STABILITY_P1_FENCING_TASK.md b/STABILITY_P1_FENCING_TASK.md new file mode 100644 index 0000000..a39792e --- /dev/null +++ b/STABILITY_P1_FENCING_TASK.md @@ -0,0 +1,77 @@ +# 稳定 V1 P1B 执行代际 Fencing 任务书 + +## 背景 + +P0 与 P1A 已封住真实数据、路径和进程终止边界。剩余最高风险集中在异步执行代际:旧 Workflow Worker 在 lease 过期并被接管后仍能写进度或终态;旧 Publish execution 也能覆盖新 claim;Windows Publish Worker 对同一 execution id 会再次调用真实 Publisher。 + +## 目标 + +1. Workflow Job 每次 claim 生成唯一 `lease_token`,所有 Worker 写回必须同时匹配 owner + token。 +2. 旧子进程在执行副作用前验证租约;失去租约后不能更新 checkpoint、进度或终态。 +3. 存活但达到最大尝试次数的 Worker 不得被其他 Worker 提前判为失败。 +4. Publish Job 使用现有 `execution_id` 作为代际 token,旧执行不能覆盖新 claim。 +5. Publish Worker 对同一 execution id 幂等;路径标识只能是安全的单段 ID。 + +## 分阶段范围 + +### P1B.1 Workflow Job + +- `app/db/database.py` +- `app/services/job_service.py` +- `app/services/job_worker.py` +- `app/services/job_worker_process.py` +- 必要的 Pipeline/转写/字幕 heartbeat 调用 +- `tests/test_job_fencing.py` 及相关既有测试 + +### P1B.2 Publish Scheduler / Worker + +- `app/services/publish_repository.py` +- `app/services/publish_scheduler.py` +- `app/services/publish_executor.py` +- `scripts/publish_host_worker.py` +- `app/services/publishers/browser_runtime.py` +- `app/services/publishers/worker_client.py` +- 发布 fencing/幂等测试 + +## 禁止范围 + +- 不改变 AI Provider、模型、账号、Token、Cookie 或 Chrome Profile 内容。 +- 不触发真实 AI、真实投稿、登录或平台验证。 +- 不删除任务、发布记录、execution journal 或历史证据。 +- 不引入消息队列、微服务或新数据库。 +- 不自动合并 PR,不强推。 + +## 数据库变化 + +- 只为 `workflow_jobs` 新增可空 `lease_token TEXT`。 +- 不重建表、不删除列、不改外键。 +- 每次 claim 写入新随机 token;release/retry/终态清空 token。 +- 正式库应用迁移前必须创建 SQLite Online Backup 并确认 `quick_check=ok`;存在旧 running 且 token 为空时必须停止并人工处理,不能静默视为有效租约。 + +## 验收标准 + +- Worker A 过期、Worker B 接管后,A 的 heartbeat/progress/checkpoint/completed/failed/cancelled/release 全部被拒。 +- `already_claimed` owner/token 不匹配、lease 过期或状态不正确时,不调用任何业务执行器。 +- running 且 lease 有效的任务达到 max_attempts 后仍保持 running;过期后才失败。 +- execution A 不能覆盖 execution B 的 Provider 结果、phase 或终态。 +- 并发 `repair_and_publish` 只创建一个替代任务。 +- 同 execution id 重试不再次调用 Publisher;不同身份复用同 id fail-closed。 +- 定向、全量、Ruff、Compileall 通过;真实发布调用次数为 0。 + +## 测试命令 + +```powershell +pytest -q tests/test_job_fencing.py tests/test_job_queue.py tests/test_long_live_foundation.py +pytest -q tests/test_publish_fencing.py tests/test_publish_scheduler_state_machine.py tests/test_publish_worker_client.py +pytest -q +ruff check app tests scripts +python -m compileall -q app scripts +``` + +## 返回格式 + +- Schema 变化、备份和兼容处理。 +- 旧执行被拒的 SQL/测试证据。 +- 全量测试与静态检查结果。 +- Commit、分支、Push、PR 状态。 +- 剩余状态机、半提交和外部调用风险。 diff --git a/STABILITY_P1_PIPELINE_CHECKPOINT_TASK.md b/STABILITY_P1_PIPELINE_CHECKPOINT_TASK.md new file mode 100644 index 0000000..621bcb8 --- /dev/null +++ b/STABILITY_P1_PIPELINE_CHECKPOINT_TASK.md @@ -0,0 +1,85 @@ +# P1.3b 自动流水线持久化步骤 Checkpoint + +## 背景 + +P1.3a 已封住 Workflow Job 代际覆盖、Task 状态跳跃、取消竞态和切片批次半提交,但 `PipelineEngine` 的步骤结果仍只保存在进程内 `context`。Web/Worker 重启、子进程异常退出或租约接管后,同一个 Job 可能从头执行,造成重复 AI 调用、重复 FFmpeg、重复封面和上下文缺失。 + +## 本轮目标 + +只复用现有 `workflow_jobs.checkpoint_json`,为 `auto_pipeline` 增加版本化、带 lease fencing 的步骤 checkpoint 和重启恢复: + +1. 每个步骤开始前持久化 `running`、输入基线和开始时间。 +2. 每个步骤完成后只保存路径、ID、hash、计数等紧凑证据,并持久化 `succeeded`。 +3. 新 Worker 接管时逐步验证数据库/文件证据,跳过已确认完成的步骤。 +4. `running` 步骤先尝试 reconcile;证据完整则补记成功,证据不足才重做。 +5. 旧 owner/token 不能写 checkpoint、Task 状态或终态。 +6. `METADATA_GENERATING`、`SCHEDULE_CREATING` 和 `PUBLISH_JOB_CREATING` 不再依赖上一进程的内存对象。 +7. 失败/取消后的自动重试必须复用同一 Workflow Job,不能把 checkpoint 留在旧 Job 后另建空 Job。 + +## 允许修改 + +- `app/services/pipeline_engine.py` +- `app/services/job_service.py`(仅必要的 checkpoint 读取错误边界) +- 新增独立的 auto pipeline checkpoint 服务 +- 与本轮直接相关的隔离测试 +- `DEVELOPMENT_LOG.md`、`NEXT_STEPS.md`、`PROJECT_AUDIT.md`、`.codemap/*` + +## 禁止修改 + +- 不新增或迁移数据库列。 +- 不修改字幕 Job 现有 `{completed: ...}` checkpoint 格式。 +- 不调用真实 AI、FFmpeg、Chrome、抖音或 B站。 +- 不处理 P1.3c 数据库迁移账本、P1.4 Provider 超时、P1.5 Secret/Auth。 +- 不删除历史 cut run、AI run、发布任务或用户数据。 + +## 已确定实现要求 + +### Checkpoint envelope + +- `kind = auto_pipeline_step_v1`,并记录 `task_id`、`start_step`、`current_step`、`completed_steps`、`steps`。 +- completed steps 必须构成从 `start_step` 开始的连续前缀;非连续、跨 Task、未知 step、未知 kind 或非字典 checkpoint 均拒绝自动执行。 +- checkpoint 不保存 Secret、完整 Prompt、完整 AI payload 或完整 output clip 对象。 + +### 恢复与复用 + +- 已完成步骤必须重新验证证据;证据缺失时 fail-closed,不自动覆盖人工或历史结果。 +- 中断步骤允许用本轮开始前基线识别新产生的 AI run、cut run 或 artifact;能确认完整才补记成功。 +- 切片证据必须绑定 active `cut_run_id`、候选 ID、任务受控目录、非空文件、size 与 fingerprint;发布草稿同时绑定 schedule、视频、封面和草稿字段。 +- 普通转写继续复用现有 transcript/分块 checkpoint;发布草稿继续使用现有 active 去重边界。 +- metadata/schedule 从持久化 JSON 恢复下游输入,不能依赖旧进程内存。 +- 字幕草稿完成后的 `PENDING_SUBTITLE_REVIEW` 仍是稳定人工门禁,重启不得自动越过。 + +### 诚实边界 + +- 外部模型/FFmpeg 与 SQLite checkpoint 无法做到同一事务;本轮目标是通过 reconciliation 显著减少重复,不宣称严格 exactly-once。 +- Provider 在返回前进程崩溃且没有可验证产物时,重试仍可能再次计费;该边界保留到 P1.4 的 Provider 幂等策略。 + +## 验收标准 + +1. Job 重领后从第一个未确认完成步骤继续,已完成 handler 不再调用。 +2. AI/cut 在“副作用已完成、checkpoint 未落盘”场景能根据新 run 证据恢复。 +3. metadata/schedule 能从文件恢复完整下游输入,且 checkpoint 只保存紧凑证据。 +4. 全部步骤 checkpoint 写入继续受 owner/token fencing 保护。 +5. malformed/unknown auto checkpoint 给出明确失败,不静默从头执行。 +6. NULL/空 checkpoint 和字幕旧 checkpoint 行为保持兼容。 +7. 字幕审核暂停在重启后仍返回 `pending_subtitle_review`。 +8. 定向测试、完整 Pytest、Ruff、Compileall 通过;工作区没有测试产物或活动数据改动。 +9. `job_id` 没有当前有效 lease 时,在任何 handler、Task 状态或文件副作用前 fail-closed。 +10. `/auto-retry` 复用旧 Job 的 running/failed checkpoint,并由新 token 继续恢复。 + +## 测试命令 + +```powershell +.\.venv\Scripts\python.exe -m pytest -q tests/test_pipeline_checkpoint.py tests/test_pipeline_state_stability.py tests/test_job_fencing.py tests/test_auto_pipeline.py tests/test_subtitle_auto_workflow.py +.\.venv\Scripts\python.exe -m pytest -q +.\.venv\Scripts\python.exe -m ruff check app tests +.\.venv\Scripts\python.exe -m compileall -q app tests scripts +``` + +## 返回格式 + +- 实际修改与非目标。 +- 重启/重领/损坏 checkpoint 的验证证据。 +- 全量测试结果。 +- 仍不能保证 exactly-once 的边界。 +- Commit、分支、PR 与下一独立轮次。 diff --git a/STABILITY_P1_PUBLISH_FENCING_TASK.md b/STABILITY_P1_PUBLISH_FENCING_TASK.md new file mode 100644 index 0000000..caba696 --- /dev/null +++ b/STABILITY_P1_PUBLISH_FENCING_TASK.md @@ -0,0 +1,77 @@ +# P1B.2 发布执行代际与 Worker 幂等整改任务 + +## 背景 + +工程审计确认 `publish_jobs.execution_id` 已能标识一次发布执行,但数据库结果写回、重启恢复、旧任务修复和 Windows Worker journal 尚未完整使用该代际。旧执行在任务重新排队并被新执行领取后,仍可能覆盖新状态;重复 HTTP 请求也可能再次进入真实 Publisher。 + +## 目标 + +1. 所有发布中的阶段、平台结果和终态写回必须同时匹配 `job_id + PUBLISHING + execution_id`。 +2. 重启恢复和 Worker 断连处理只允许改变读取时捕获的 execution;代际已变化时返回 skipped,不记录伪事件。 +3. `repair_and_publish` 对源状态复核、既有替代任务检查、克隆和事件写入在同一 `BEGIN IMMEDIATE` 事务中完成。 +4. Windows Worker 对同一 execution 串行执行;终态请求直接返回既有结果,危险中间阶段不重复投稿。 +5. Worker 使用的 `job_id`、`execution_id`、`account_id` 必须经过 Windows 安全标识校验,不能影响 journal、浏览器 profile 或截图目录边界。 + +## 允许修改范围 + +- `app/services/publish_repository.py` +- `app/services/publish_scheduler.py` +- `app/services/publish_service.py`(仅统一 publish_jobs 乐观并发时间版本) +- `app/services/publish_time.py`(仅生成微秒级并发版本时间) +- `app/services/publish_executor.py` +- `app/services/publishers/worker_client.py` +- `app/services/publishers/browser_runtime.py` +- `app/services/publishers/local_browser.py`(仅在 Worker 投稿前触发 dispatch CAS) +- `app/services/publishers/base.py` +- `app/services/publishers/manual_export.py` +- `scripts/publish_host_worker.py` +- `app/main.py`(仅 Scheduler 后台 Task 的优雅停机) +- 发布调度、Worker、幂等与安全标识相关测试 +- `PROJECT_AUDIT.md`、`DEVELOPMENT_LOG.md`、`NEXT_STEPS.md`、`.codemap/*` + +## 禁止修改范围 + +- 不改变平台页面脚本、真实投稿步骤、账号认证方式或平台风控边界。 +- 不触发真实抖音/B站投稿、真实账号登录或真实 AI 调用。 +- 不修改用户活动数据库,不删除 journal、媒体、发布包或浏览器 profile。 +- 不引入消息队列、分布式锁、微服务或新数据库。 + +## 已确定实现要求 + +- Scheduler claim 返回并保留本次 `execution_id`;生产 executor 在真正调用 Publisher 前以 `execution_id + updated_at` 原子保留 dispatch 权,恢复扫描只能写回自己读取的旧快照。 +- `record_provider_result` 与 `update_execution_phase` 支持 expected execution 条件并返回是否写入。 +- PUBLISHING 终态函数必须显式接收 expected execution;人工/上传前状态变化必须显式限定来源状态。 +- 数据库 UPDATE 未命中时必须回滚同事务 provider result,不写事件、不追加成功/失败日志。 +- Worker journal 保存不可变执行身份;相同 execution 的终态只有在 `job_id/platform/account_id` 全部匹配时才可重放,身份冲突或缺失 fail closed。 +- 同一 publish job 的旧 execution 一旦进入上传、提交或不确定终态,后续不同 execution 会被 Worker 的持久化 journal 证据阻断,不能再次进入 Publisher。 +- job、execution 与账号互斥同时覆盖进程内和操作系统级跨进程文件锁;进程崩溃后由操作系统释放,不能通过竞态删除新锁并重复进入真实 Publisher。 +- `received/browser_opening/browser_opened/rejected` 才允许同 execution 在进程重启后安全续试;上传开始后的非终态一律转人工复核,禁止重复 Publisher 调用。 +- 标识仅允许 ASCII 字母、数字、点、下划线和连字符;禁止路径分隔符、`..`、控制字符、盘符和 Windows 保留名。 +- Scheduler 保存后台 Task 引用;应用关闭时先停止新扫描,再等待当前一轮安全结束。 +- 本地发布包先在同卷 staging 目录完整生成,再原子切换;中途失败保留上一份完整包。 + +## 验收标准 + +- 旧 execution 的 provider result、PUBLISHED/FAILED/NEED_REVIEW/重排队写回全部被拒绝。 +- 代际不匹配时事件数量不增加,最新任务字段不被污染。 +- 并发两次旧任务修复只产生一个替代任务。 +- 同一 Worker execution 的并发/重复请求只调用一次 Mock Publisher;终态结果可重复读取。 +- execution 身份冲突、路径穿越、Windows 保留名均在创建路径前被拒绝。 +- 相关测试、全量 Pytest、Ruff、compileall 与 `git diff --check` 通过。 + +## 测试命令 + +```powershell +.\.venv\Scripts\python.exe -m pytest tests/test_publish_fencing.py tests/test_publish_scheduler_state_machine.py tests/test_publish_scheduler.py tests/test_publish_worker_client.py tests/test_local_browser_publishers.py tests/test_publish_readiness.py tests/test_publish_api_flow.py -q +.\.venv\Scripts\python.exe -m pytest -q +.\.venv\Scripts\python.exe -m ruff check app scripts tests +.\.venv\Scripts\python.exe -m compileall -q app scripts +git diff --check +``` + +## 返回格式 + +- 修改文件和关键行为 +- 定向/全量测试准确数量与结果 +- 未触发真实发布的证据 +- 剩余风险和下一轮范围 diff --git a/STABILITY_P1_STATE_ATOMICITY_TASK.md b/STABILITY_P1_STATE_ATOMICITY_TASK.md new file mode 100644 index 0000000..44d6d93 --- /dev/null +++ b/STABILITY_P1_STATE_ATOMICITY_TASK.md @@ -0,0 +1,110 @@ +# P1.3 任务状态与切片原子性 + +## 背景 + +工程审计确认了四个相互关联的稳定性缺口:任务状态接口可以任意跳转;全自动流水线的取消会被记录成失败或继续完成;`READY_TO_PUBLISH` 只短暂存在;切片结果逐条提交,失败时可能留下半成功数据。 + +基线证据:隔离测试数据库初始化后,相关 8 个测试文件共 `112 passed`,Ruff 检查通过。 + +## 目标 + +1. 对外状态更新只能执行明确允许的相邻或恢复转换,禁止把空任务直接标记完成。 +2. 自动流水线在步骤前后都检查取消,取消后任务进入明确、可重试的 `CANCELLED` 状态。 +3. 自动流水线成功终态稳定保留为 `READY_TO_PUBLISH`,不再立即覆盖为 `COMPLETED`。 +4. 同一切片批次的结果写入、旧版本停用和新版本激活在一个 SQLite 事务内完成。 +5. 每个切片批次写入独立目录,避免并发/重试覆盖同名媒体文件。 +6. 汇总文件写入失败不得遮蔽原始业务错误或取消原因。 + +## 影响与成本 + +| 项目 | 影响 | 成本 | 本轮决策 | +| --- | --- | --- | --- | +| 测试数据库统一初始化 | 高 | 低 | 实施 | +| 状态接口受控转换 | 高 | 中 | 实施 | +| 取消终态与 READY 稳定化 | 高 | 中 | 实施 | +| 切片事务与批次目录隔离 | 高 | 中 | 实施 | +| 汇总写入降级 | 中 | 低 | 实施 | +| 流水线跨进程持久化步骤 checkpoint | 高 | 高 | 留到 P1.3b | +| FFprobe、AI Provider、Secret/Auth | 高 | 中至高 | 按后续独立轮次处理 | + +## 状态图 + +修改前: + +```text +任意状态 --PATCH /status--> 任意状态 +PUBLISH_JOB_CREATING -> READY_TO_PUBLISH -> COMPLETED +运行中 --取消/异常--> FAILED_ 或残留运行态 +``` + +修改后: + +```text +对外 PATCH: 当前状态 --允许表--> 相邻状态/明确恢复状态 +PUBLISH_JOB_CREATING -> READY_TO_PUBLISH(稳定终态,等待人工发布) +运行中 --取消--> CANCELLED --重试--> 对应流水线起点 +``` + +## 允许修改范围 + +- `app/models/task.py` +- `app/routers/tasks.py` +- `app/services/task_lifecycle_service.py` +- `app/services/task_service.py` +- `app/services/pipeline_engine.py` +- `app/services/auto_publish_service.py` +- `app/services/job_service.py` +- `app/services/job_worker.py` +- `app/services/video_cut_workflow_service.py` +- 与上述行为直接相关的测试 +- `PROJECT_AUDIT.md`、`DEVELOPMENT_LOG.md`、`NEXT_STEPS.md` +- Codemap 生成数据与产物(必须通过 skill 脚本生成) + +## 禁止修改范围 + +- 数据库 Schema 和生产数据 +- AI Provider、Prompt、FFmpeg 参数和发布平台逻辑 +- UI 结构与样式 +- 依赖版本 +- Dead Code 删除和大规模模块拆分 + +## 已确定实现要求 + +- 保留手动流程小写状态与自动流程大写状态的兼容性。 +- 内部工作流仍可写步骤状态;对外 `PATCH /status` 必须走独立的合法转换检查和数据库条件更新。 +- 所有任务状态写入拒绝已永久删除任务。 +- 切片数据库提交失败时,新批次标记失败,旧 active 版本保持不变。 +- 较旧并发批次不得覆盖已经完成的较新 active 批次。 +- 发布中心同步只能在当前批次成功激活后执行。 +- 取消检查放在步骤开始前和处理完成后,最终 READY 写入前再检查一次。 +- Task 步骤状态、切片提交和 READY 写回必须绑定当前 Workflow Job 的 owner、token 与未过期 lease。 +- 公开 Task 取消必须同步请求活跃自动 Job 停止;Job 取消只清理 provider 证据明确关联到本执行代际的未发布任务。 +- `running + cancel_requested` 在子进程退出或 lease 过期后必须收敛到 `cancelled`,不能永久残留运行态。 + +## 验收标准 + +- 空任务不能通过 API 直接变成 `completed`/`COMPLETED`,返回 HTTP 409。 +- 合法相邻状态转换成功;并发状态变化时条件更新失败,不覆盖新状态。 +- 取消的自动任务最终为 `CANCELLED`,job 为 `cancelled`,不会写 READY/COMPLETED。 +- 旧 lease Worker 不能写 Task 步骤、切片结果或 READY;READY 后取消会清理本轮关联且尚未发布的排期记录。 +- 成功流水线最终任务状态和返回状态均为 `READY_TO_PUBLISH`/`ready_to_publish`。 +- 人为制造第二条 output clip 插入失败时,数据库中不保留该批次的部分 clip,旧 active 结果不变。 +- 并发创建 cut run 时 `run_number` 不重复;批次输出目录不同。 +- 汇总写入失败时仍返回原始失败/取消结果。 +- 定向测试、全量测试和 Ruff 全部通过。 + +## 建议测试命令 + +```powershell +python -m pytest tests/test_task_state_machine.py tests/test_cut_atomicity.py tests/test_auto_pipeline.py tests/test_job_queue.py tests/test_job_fencing.py -q +python -m pytest -q +python -m ruff check app tests +``` + +## 回滚方式 + +本轮不迁移 Schema;回滚代码提交即可。新生成的 `clips/run_*` 目录由现有数据库路径引用,旧目录和旧记录不会删除。`provider_response.workflow_job_id` 只是本地关联证据,不改变发布表结构。 + +## 返回格式 + +报告修改文件、状态行为变化、测试证据、Codemap 复审结果、提交哈希、分支、Push 与 PR 链接。 diff --git a/STABILITY_REMEDIATION_TASK.md b/STABILITY_REMEDIATION_TASK.md new file mode 100644 index 0000000..fb277ba --- /dev/null +++ b/STABILITY_REMEDIATION_TASK.md @@ -0,0 +1,84 @@ +# 稳定 V1 整改任务书 + +## 背景 + +工程审计确认当前项目属于“可用 V1”,但存在会伤害真实数据或让失败状态不可恢复的 P0 风险。本轮目标是按小步、可测试、可回滚的方式把项目提升到“稳定 V1”,不做全面重构,也不扩大产品范围。 + +## 本轮目标 + +1. 隔离 Pytest 数据库和媒体目录,任何外部 `DATABASE_PATH` 都不能让测试连接活动库;危险清理夹具必须在删除前再次 fail-closed 校验。 +2. 为 SQLite 外键异常提供默认只读预演、应用前强制备份、事务内修复、修复后完整性复查的工具;只在验证备份后处理已确认的孤儿引用。 +3. 把永久删除改为“托管目录暂存隔离 -> 数据库提交 -> 延迟清除”;数据库失败时可把文件恢复原位,外部唯一原片始终不动。 +4. 完成独立测试与验收,并记录剩余 P1/P2 风险和下一轮顺序。 + +## 允许修改范围 + +- `tests/conftest.py` 及与本轮 P0 直接相关的测试。 +- `app/services/storage_service.py` +- `app/services/task_lifecycle_service.py` +- `app/services/database_backup_service.py`(仅复用或补充安全备份能力)。 +- `scripts/` 下新增或调整本轮修复、验证脚本。 +- `DEVELOPMENT_LOG.md`、`NEXT_STEPS.md`、本任务书和必要审计文档。 + +## 禁止修改范围 + +- 不改变 AI Provider、投稿平台、字幕和切片的正常业务语义。 +- 不更改生产 Schema,不删除任务、发布或字幕历史。 +- 不读取、输出或提交 `.env`、Token、Cookie、账号凭据。 +- 不绕过平台登录、验证码、风控或人工确认。 +- 不自动合并 PR,不强制推送,不删除分支。 + +## 已确定实现要求 + +### P0.1 测试隔离 + +- Pytest 启动时无条件使用进程级临时根目录,不继承调用者传入的活动库路径。 +- 临时数据库和媒体目录必须位于同一隔离根目录。 +- 对整表清理增加第二道路径校验;路径不在 Pytest 隔离根目录时立即中止。 +- 验证从命令行故意传入活动库路径时,测试仍不会连接或改写活动库。 + +### P0.2 外键修复 + +- 工具默认 dry-run;只有显式 `--apply` 才写入。 +- 应用前使用 SQLite Online Backup API 创建唯一备份并执行 `quick_check`。 +- 只处理当前检测到且策略明确的孤儿 `publish_jobs.output_clip_id` 和 `subtitle_jobs.output_clip_id`。 +- 修复必须单事务提交;提交前后执行 `foreign_key_check`,不允许产生新异常。 +- 尽量保留历史证据;若表约束不允许安全置空,则先归档必要字段再做最小删除,并在报告中逐条列出。 + +### P0.3 两阶段永久删除 + +- 只处理经过现有托管根目录校验的目录。 +- 文件先原子移动到同盘隔离区并写清单;任一步失败要恢复已经移动的目录。 +- 数据库提交失败时必须恢复目录;不得留下“文件没了、任务仍可见”的半成功状态。 +- 数据库提交成功后再清除隔离区;清除失败要返回明确的 `cleanup_pending`,不得把逻辑删除回滚成可见状态。 +- 重复执行必须幂等;外部原片保持不变。 + +## 验收标准 + +- 相关 P0 回归测试全部通过。 +- 全量测试、Lint/语法检查、前端语法检查通过,或对既有失败给出可复现证据。 +- 活动数据库在运行普通测试前后文件哈希、大小和外键异常计数不发生变化。 +- 外键修复应用前生成可读备份;修复后 `PRAGMA quick_check = ok` 且 `PRAGMA foreign_key_check` 为空。 +- 模拟数据库提交失败时,暂存文件恢复到原路径,任务仍可见。 +- 模拟最终清理失败时,任务保持已删除并返回可恢复的待清理状态。 +- `git diff` 只包含本轮范围,且无敏感信息、调试残留或临时产物。 + +## 测试命令 + +具体临时目录由执行者生成,不得使用 `data/workflow.sqlite3`: + +```powershell +pytest -q tests/test_task_query_service.py tests/test_media_storage_lifecycle.py tests/test_database_backup_service.py +pytest -q +ruff check app tests scripts +python -m compileall -q app scripts +node --check app/static/js/task-detail.js +``` + +## 返回格式 + +- 修改文件与关键行为。 +- 测试命令、退出码、通过/失败数量。 +- 活动数据库备份路径、修复前后外键计数和完整性结果(不含业务内容)。 +- Commit、分支、Push 和 PR 状态。 +- 未完成的 P1/P2 风险与下一轮建议。 diff --git a/THIRD_PARTY_NOTICES.md b/THIRD_PARTY_NOTICES.md index 38f9fca..367c8df 100644 --- a/THIRD_PARTY_NOTICES.md +++ b/THIRD_PARTY_NOTICES.md @@ -22,3 +22,30 @@ - 许可证:Apache License 2.0(以 Playwright 官方仓库许可证为准)。 第三方网站和平台名称、商标及页面属于各自权利人。使用本项目投稿时,用户仍需遵守抖音、哔哩哔哩及浏览器相关服务条款,不得使用本项目绕过验证或平台风控。 + +## pysubs2 1.9.0 + +- 项目:https://github.com/tkarabela/pysubs2 +- 用途:SRT、WebVTT、ASS 的解析、导入、导出与 ASS 样式序列化。 +- 许可证:MIT License。 +- 本项目锁定运行时版本 `pysubs2==1.9.0`;完整许可证保存在 `third_party_licenses/pysubs2-LICENSE.txt`。 + +## wavesurfer.js 7.12.11 + +- 项目:https://github.com/katspaugh/wavesurfer.js +- 用途:字幕编辑器波形、Regions 区间拖动与 Timeline 时间刻度。 +- 许可证:BSD 3-Clause License。 +- 本项目固定并本地托管 `wavesurfer.min.js`、`regions.min.js` 与 `timeline.min.js`,避免运行时依赖 CDN;完整许可证保存在 `app/static/vendor/wavesurfer/LICENSE`。 + +## Subtitle Edit(设计参考) + +- 项目:https://github.com/SubtitleEdit/subtitleedit +- 参考范围:波形联动、字幕行编辑、拆分合并、撤销重做与质量检查交互。 +- 本项目没有复制 Subtitle Edit 源码,仅借鉴成熟交互设计。 + +## VideoCaptioner(仅流程研究) + +- 项目:https://github.com/WEIFENG2333/VideoCaptioner +- 许可证:GPL-3.0。 +- 参考范围:字幕识别、校对、导出和视频合成的产品流程。 +- GPL 源码没有复制、改写或打包进入本项目。 diff --git a/app/core/config.py b/app/core/config.py index 09ebbab..bac3df5 100644 --- a/app/core/config.py +++ b/app/core/config.py @@ -243,9 +243,6 @@ class Settings: # === P0 安全与稳定性配置 === - # 本地 API 访问保护:对 /api 下写接口校验 token - local_admin_token: str = _env("LOCAL_ADMIN_TOKEN", "") - # 允许浏览的媒体根目录(逗号分隔的绝对路径列表) allowed_media_roots: str = _env("ALLOWED_MEDIA_ROOTS", "") diff --git a/app/db/database.py b/app/db/database.py index aa9f677..cfd57c7 100644 --- a/app/db/database.py +++ b/app/db/database.py @@ -1,11 +1,13 @@ +import hashlib import json import sqlite3 from datetime import datetime -from collections.abc import Iterator +from collections.abc import Callable, Iterator from contextlib import contextmanager +from dataclasses import dataclass from app.core.config import settings -from app.services.database_backup_service import create_publish_migration_backup +from app.services.database_backup_service import create_publish_migration_backup, create_schema_migration_backup DEFAULT_AI_PROMPT_PRESET_ID = "preset_001" @@ -13,6 +15,67 @@ VARIETY_AI_PROMPT_PATH = settings.project_root / "prompts" / "variety_interview_prompt_preset_002.txt" COMEDY_V2_AI_PROMPT_PATH = settings.project_root / "prompts" / "variety_comedy_v2_prompt.txt" +PUBLISH_ACTIVE_UNIQUE_INDEX_LEGACY_NAME = "uq_publish_jobs_active_clip_platform_mode" +PUBLISH_ACTIVE_UNIQUE_INDEX_NAME = "uq_publish_jobs_active_clip_platform_mode_v2" +PUBLISH_UNIQUE_ACTIVE_STATUSES = ( + "DRAFT", + "WAITING", + "SCHEDULED", + "PUBLISHING", + "NEED_REVIEW", +) +PUBLISH_UNIQUE_ACTIVE_STATUS_SQL = ", ".join(f"'{status}'" for status in PUBLISH_UNIQUE_ACTIVE_STATUSES) +PUBLISH_ACTIVE_UNIQUE_INDEX_SQL = f""" +CREATE UNIQUE INDEX IF NOT EXISTS {PUBLISH_ACTIVE_UNIQUE_INDEX_NAME} +ON publish_jobs(output_clip_id, platform, publish_mode) +WHERE status IN ({PUBLISH_UNIQUE_ACTIVE_STATUS_SQL}) + AND output_clip_id IS NOT NULL AND output_clip_id <> '' +""".strip() +PUBLISH_ACTIVE_INDEX_MIGRATION_VERSION = "20260824_01_publish_active_unique_index_v2" +PUBLISH_ACTIVE_INDEX_MIGRATION_NAME = "发布活动任务唯一索引安全切换" +PUBLISH_ACTIVE_INDEX_MIGRATION_CHECKSUM = hashlib.sha256( + ( + f"{PUBLISH_ACTIVE_INDEX_MIGRATION_VERSION}\n" + f"{PUBLISH_ACTIVE_INDEX_MIGRATION_NAME}\n" + f"{PUBLISH_ACTIVE_UNIQUE_INDEX_SQL}\n" + f"DROP INDEX {PUBLISH_ACTIVE_UNIQUE_INDEX_LEGACY_NAME}" + ).encode("utf-8") +).hexdigest() +TASK_UPLOAD_ONLY_MIGRATION_VERSION = "20260824_02_task_upload_only" +TASK_UPLOAD_ONLY_MIGRATION_NAME = "任务视频来源归一为本机上传" +TASK_UPLOAD_ONLY_MIGRATION_SQL = """ +UPDATE tasks +SET original_video_path = CASE + WHEN original_video_path IS NULL OR TRIM(original_video_path) = '' THEN nas_file_path + ELSE original_video_path + END, + source_type = 'upload', + nas_file_path = NULL +WHERE source_type != 'upload' + OR source_type IS NULL + OR (nas_file_path IS NOT NULL AND TRIM(nas_file_path) != '') +""".strip() +TASK_UPLOAD_ONLY_MIGRATION_CHECKSUM = hashlib.sha256( + ( + f"{TASK_UPLOAD_ONLY_MIGRATION_VERSION}\n" + f"{TASK_UPLOAD_ONLY_MIGRATION_NAME}\n" + f"{TASK_UPLOAD_ONLY_MIGRATION_SQL}" + ).encode("utf-8") +).hexdigest() + + +class SchemaMigrationError(RuntimeError): + """数据库结构迁移或不变量验证失败。""" + + +@dataclass(frozen=True) +class SchemaMigration: + version: str + name: str + checksum: str + apply: Callable[[sqlite3.Connection], None] + verify: Callable[[sqlite3.Connection], None] + @contextmanager def get_connection() -> Iterator[sqlite3.Connection]: @@ -32,6 +95,56 @@ def init_db() -> None: settings.data_dir.mkdir(parents=True, exist_ok=True) settings.tasks_dir.mkdir(parents=True, exist_ok=True) + needs_long_live_backup = _requires_long_live_schema_migration(settings.database_path) + needs_workflow_fencing_backup = _requires_workflow_job_fencing_migration(settings.database_path) + needs_subtitle_editor_backup = _requires_subtitle_editor_schema_migration(settings.database_path) + needs_subtitle_auto_backup = _requires_subtitle_auto_schema_migration(settings.database_path) + needs_publish_index_backup = _requires_publish_active_index_migration(settings.database_path) + needs_task_upload_only_backup = _requires_task_upload_only_migration(settings.database_path) + if needs_long_live_backup: + create_schema_migration_backup( + settings.database_path, + settings.data_dir / "backups", + "long-live-foundation", + ) + if needs_workflow_fencing_backup and not needs_long_live_backup: + create_schema_migration_backup( + settings.database_path, + settings.data_dir / "backups", + "workflow-job-fencing", + ) + if needs_subtitle_editor_backup: + create_schema_migration_backup( + settings.database_path, + settings.data_dir / "backups", + "subtitle-editor-rebuild", + ) + if needs_subtitle_auto_backup: + create_schema_migration_backup( + settings.database_path, + settings.data_dir / "backups", + "subtitle-auto-workflow", + ) + if needs_publish_index_backup and not any( + ( + needs_long_live_backup, + needs_workflow_fencing_backup, + needs_subtitle_editor_backup, + needs_subtitle_auto_backup, + ) + ): + create_schema_migration_backup( + settings.database_path, + settings.data_dir / "backups", + "publish-active-unique-index-v2", + ) + if needs_task_upload_only_backup: + create_schema_migration_backup( + settings.database_path, + settings.data_dir / "backups", + "task-upload-only", + ) + with get_connection() as connection: connection.executescript( """ @@ -47,6 +160,8 @@ def init_db() -> None: candidate_clip_count INTEGER NOT NULL DEFAULT 12, selection_profile TEXT NOT NULL DEFAULT 'general', final_clip_target INTEGER NOT NULL DEFAULT 5, + highlight_density_per_hour INTEGER NOT NULL DEFAULT 4, + highlight_total_limit INTEGER NOT NULL DEFAULT 30, ai_preference TEXT, ai_prompt_preset_id TEXT NOT NULL DEFAULT 'preset_001', auto_mode INTEGER NOT NULL DEFAULT 0, @@ -61,6 +176,13 @@ def init_db() -> None: updated_at TEXT NOT NULL ); + CREATE TABLE IF NOT EXISTS schema_migrations ( + version TEXT PRIMARY KEY, + name TEXT NOT NULL, + checksum TEXT NOT NULL, + applied_at TEXT NOT NULL + ); + CREATE TABLE IF NOT EXISTS clip_candidates ( id TEXT PRIMARY KEY, task_id TEXT NOT NULL, @@ -104,6 +226,11 @@ def init_db() -> None: output_file_name TEXT, status TEXT NOT NULL DEFAULT 'pending', error_message TEXT, + source_start_ms INTEGER, + source_end_ms INTEGER, + source_duration_ms INTEGER, + source_fingerprint TEXT, + snapshot_source TEXT NOT NULL DEFAULT 'legacy_inferred', created_at TEXT NOT NULL, updated_at TEXT NOT NULL, FOREIGN KEY(task_id) REFERENCES tasks(id), @@ -147,6 +274,10 @@ def init_db() -> None: font_color TEXT NOT NULL DEFAULT '#ffffff', stroke_color TEXT NOT NULL DEFAULT '#111827', shadow_enabled INTEGER NOT NULL DEFAULT 1, + outline_width REAL NOT NULL DEFAULT 3, + shadow_depth REAL NOT NULL DEFAULT 1, + safe_area_percent REAL NOT NULL DEFAULT 5, + speaker_styles_json TEXT NOT NULL DEFAULT '{}', is_default INTEGER NOT NULL DEFAULT 1, created_at TEXT NOT NULL, updated_at TEXT NOT NULL @@ -156,18 +287,81 @@ def init_db() -> None: id TEXT PRIMARY KEY, task_id TEXT NOT NULL, output_clip_id TEXT NOT NULL, + revision_id TEXT, + workflow_job_id TEXT, style_preset_id TEXT, status TEXT NOT NULL DEFAULT 'pending', subtitle_file_path TEXT, output_file_path TEXT, error_message TEXT, + validation_status TEXT NOT NULL DEFAULT 'legacy_unverified', + validation_json TEXT NOT NULL DEFAULT '{}', + encoder TEXT NOT NULL DEFAULT '', + verified_at TEXT, + is_active INTEGER NOT NULL DEFAULT 1, created_at TEXT NOT NULL, updated_at TEXT NOT NULL, FOREIGN KEY(task_id) REFERENCES tasks(id), FOREIGN KEY(output_clip_id) REFERENCES output_clip(id), + FOREIGN KEY(revision_id) REFERENCES subtitle_revisions(id), + FOREIGN KEY(workflow_job_id) REFERENCES workflow_jobs(id), FOREIGN KEY(style_preset_id) REFERENCES subtitle_style_presets(id) ); + CREATE TABLE IF NOT EXISTS subtitle_tracks ( + id TEXT PRIMARY KEY, + task_id TEXT NOT NULL, + track_type TEXT NOT NULL, + output_clip_id TEXT, + name TEXT NOT NULL, + language TEXT NOT NULL DEFAULT 'zh-CN', + source_track_id TEXT, + source_revision_id TEXT, + source_fingerprint TEXT NOT NULL DEFAULT '', + active_revision_id TEXT, + sync_status TEXT NOT NULL DEFAULT 'up_to_date', + has_manual_edits INTEGER NOT NULL DEFAULT 0, + is_active INTEGER NOT NULL DEFAULT 1, + created_at TEXT NOT NULL, + updated_at TEXT NOT NULL, + UNIQUE(task_id, track_type, output_clip_id), + FOREIGN KEY(task_id) REFERENCES tasks(id), + FOREIGN KEY(output_clip_id) REFERENCES output_clip(id), + FOREIGN KEY(source_track_id) REFERENCES subtitle_tracks(id) + ); + + CREATE TABLE IF NOT EXISTS subtitle_revisions ( + id TEXT PRIMARY KEY, + track_id TEXT NOT NULL, + revision_number INTEGER NOT NULL, + origin TEXT NOT NULL, + parent_revision_id TEXT, + status TEXT NOT NULL DEFAULT 'draft', + note TEXT, + cue_count INTEGER NOT NULL DEFAULT 0, + checksum TEXT NOT NULL, + created_at TEXT NOT NULL, + approved_at TEXT, + UNIQUE(track_id, revision_number), + FOREIGN KEY(track_id) REFERENCES subtitle_tracks(id) ON DELETE CASCADE, + FOREIGN KEY(parent_revision_id) REFERENCES subtitle_revisions(id) + ); + + CREATE TABLE IF NOT EXISTS subtitle_cues ( + id TEXT PRIMARY KEY, + revision_id TEXT NOT NULL, + cue_index INTEGER NOT NULL, + start_ms INTEGER NOT NULL, + end_ms INTEGER NOT NULL, + text TEXT NOT NULL, + confidence REAL, + speaker TEXT NOT NULL DEFAULT '', + source_cue_id TEXT, + created_at TEXT NOT NULL, + UNIQUE(revision_id, cue_index), + FOREIGN KEY(revision_id) REFERENCES subtitle_revisions(id) ON DELETE CASCADE + ); + CREATE TABLE IF NOT EXISTS publish_platform_configs ( platform TEXT PRIMARY KEY, app_name TEXT NOT NULL DEFAULT '', @@ -308,6 +502,81 @@ def init_db() -> None: updated_at TEXT NOT NULL, started_at TEXT, finished_at TEXT, + attempt_count INTEGER NOT NULL DEFAULT 0, + max_attempts INTEGER NOT NULL DEFAULT 3, + next_attempt_at TEXT, + lease_owner TEXT, + lease_token TEXT, + lease_expires_at TEXT, + heartbeat_at TEXT, + cancel_requested INTEGER NOT NULL DEFAULT 0, + checkpoint_json TEXT, + checkpoint_updated_at TEXT, + FOREIGN KEY(task_id) REFERENCES tasks(id) + ); + + CREATE TABLE IF NOT EXISTS transcription_runs ( + id TEXT PRIMARY KEY, + task_id TEXT NOT NULL, + source_fingerprint TEXT NOT NULL, + provider TEXT NOT NULL, + model TEXT NOT NULL DEFAULT '', + device TEXT NOT NULL DEFAULT '', + compute_type TEXT NOT NULL DEFAULT '', + chunk_seconds INTEGER NOT NULL, + overlap_seconds INTEGER NOT NULL, + status TEXT NOT NULL DEFAULT 'processing', + total_chunks INTEGER NOT NULL DEFAULT 0, + completed_chunks INTEGER NOT NULL DEFAULT 0, + is_active INTEGER NOT NULL DEFAULT 0, + error_message TEXT, + created_at TEXT NOT NULL, + updated_at TEXT NOT NULL, + completed_at TEXT, + FOREIGN KEY(task_id) REFERENCES tasks(id) + ); + + CREATE TABLE IF NOT EXISTS transcription_chunks ( + id TEXT PRIMARY KEY, + run_id TEXT NOT NULL, + task_id TEXT NOT NULL, + chunk_index INTEGER NOT NULL, + start_ms INTEGER NOT NULL, + end_ms INTEGER NOT NULL, + status TEXT NOT NULL DEFAULT 'queued', + attempt_count INTEGER NOT NULL DEFAULT 0, + result_json TEXT, + result_checksum TEXT, + error_message TEXT, + created_at TEXT NOT NULL, + updated_at TEXT NOT NULL, + UNIQUE(run_id, chunk_index), + FOREIGN KEY(run_id) REFERENCES transcription_runs(id) ON DELETE CASCADE, + FOREIGN KEY(task_id) REFERENCES tasks(id) + ); + + CREATE TABLE IF NOT EXISTS ai_analysis_windows ( + id TEXT PRIMARY KEY, + task_id TEXT NOT NULL, + transcript_fingerprint TEXT NOT NULL, + provider TEXT NOT NULL, + model TEXT NOT NULL DEFAULT '', + window_index INTEGER NOT NULL, + start_seconds INTEGER NOT NULL, + end_seconds INTEGER NOT NULL, + status TEXT NOT NULL DEFAULT 'queued', + attempt_count INTEGER NOT NULL DEFAULT 0, + result_json TEXT, + result_checksum TEXT, + error_message TEXT, + next_retry_at TEXT, + created_at TEXT NOT NULL, + updated_at TEXT NOT NULL, + completed_at TEXT, + UNIQUE( + task_id, transcript_fingerprint, provider, model, + window_index, start_seconds, end_seconds + ), FOREIGN KEY(task_id) REFERENCES tasks(id) ); @@ -348,18 +617,161 @@ def init_db() -> None: _migrate_ai_analysis_runs_table(connection) _migrate_subtitle_style_presets_table(connection) _migrate_subtitle_jobs_table(connection) + _migrate_subtitle_editor_tables(connection) _migrate_publish_platform_configs_table(connection) _migrate_publish_accounts_table(connection) _migrate_publish_jobs_table(connection) _migrate_publish_job_events_table(connection) _restore_legacy_user_cancelled_publish_jobs(connection) _migrate_workflow_jobs_table(connection) + _guard_unfenced_running_workflow_jobs(connection) + _migrate_transcription_tables(connection) + _migrate_ai_analysis_windows_table(connection) _migrate_cut_runs_table(connection) _seed_ai_prompt_presets(connection) _seed_subtitle_style_preset(connection) _seed_publish_platform_configs(connection) _create_indexes(connection) connection.commit() + _run_schema_migrations(connection) + + +def _requires_long_live_schema_migration(database_path) -> bool: + """只对已经存在且确实缺少新结构的数据库创建一次迁移前备份。""" + if not database_path.exists() or database_path.stat().st_size == 0: + return False + try: + connection = sqlite3.connect(f"{database_path.resolve().as_uri()}?mode=ro", uri=True, timeout=10) + task_columns = {row[1] for row in connection.execute("PRAGMA table_info(tasks)").fetchall()} + job_columns = {row[1] for row in connection.execute("PRAGMA table_info(workflow_jobs)").fetchall()} + table_names = {row[0] for row in connection.execute("SELECT name FROM sqlite_master WHERE type='table'").fetchall()} + finally: + if "connection" in locals(): + connection.close() + return ( + "highlight_density_per_hour" not in task_columns + or "lease_owner" not in job_columns + or "transcription_runs" not in table_names + or "transcription_chunks" not in table_names + or "ai_analysis_windows" not in table_names + ) + + +def _requires_workflow_job_fencing_migration(database_path) -> bool: + """已有 Workflow Job 表缺少 claim 代际 token 时,先创建可移植备份。""" + if not database_path.exists() or database_path.stat().st_size == 0: + return False + connection = None + try: + connection = sqlite3.connect(f"{database_path.resolve().as_uri()}?mode=ro", uri=True, timeout=10) + columns = {row[1] for row in connection.execute("PRAGMA table_info(workflow_jobs)").fetchall()} + finally: + if connection is not None: + connection.close() + return bool(columns) and "lease_token" not in columns + + +def _requires_subtitle_editor_schema_migration(database_path) -> bool: + """已有数据库缺少字幕 revision 结构时,先做在线备份。""" + if not database_path.exists() or database_path.stat().st_size == 0: + return False + connection = None + try: + connection = sqlite3.connect(f"{database_path.resolve().as_uri()}?mode=ro", uri=True, timeout=10) + output_columns = {row[1] for row in connection.execute("PRAGMA table_info(output_clip)").fetchall()} + job_columns = {row[1] for row in connection.execute("PRAGMA table_info(subtitle_jobs)").fetchall()} + table_names = {row[0] for row in connection.execute("SELECT name FROM sqlite_master WHERE type='table'").fetchall()} + finally: + if connection: + connection.close() + return ( + "source_start_ms" not in output_columns + or "revision_id" not in job_columns + or not {"subtitle_tracks", "subtitle_revisions", "subtitle_cues"} <= table_names + ) + + +def _requires_subtitle_auto_schema_migration(database_path) -> bool: + """已有字幕任务缺少异步渲染验证字段时,迁移前先备份。""" + if not database_path.exists() or database_path.stat().st_size == 0: + return False + connection = None + try: + connection = sqlite3.connect(f"{database_path.resolve().as_uri()}?mode=ro", uri=True, timeout=10) + columns = {row[1] for row in connection.execute("PRAGMA table_info(subtitle_jobs)").fetchall()} + finally: + if connection: + connection.close() + return bool(columns) and not { + "workflow_job_id", + "validation_status", + "validation_json", + "encoder", + "verified_at", + } <= columns + + +def _requires_publish_active_index_migration(database_path) -> bool: + """已有发布表尚未完成 P1.3c 迁移时,启动写入前先做可移植备份。""" + if not database_path.exists() or database_path.stat().st_size == 0: + return False + connection = None + try: + connection = sqlite3.connect(f"{database_path.resolve().as_uri()}?mode=ro", uri=True, timeout=10) + publish_table = connection.execute( + "SELECT 1 FROM sqlite_master WHERE type = 'table' AND name = 'publish_jobs'" + ).fetchone() + if publish_table is None: + return False + index_row = connection.execute( + "SELECT 1 FROM sqlite_master WHERE type = 'index' AND name = ?", + (PUBLISH_ACTIVE_UNIQUE_INDEX_NAME,), + ).fetchone() + ledger_table = connection.execute( + "SELECT 1 FROM sqlite_master WHERE type = 'table' AND name = 'schema_migrations'" + ).fetchone() + ledger_row = None + if ledger_table is not None: + try: + ledger_row = connection.execute( + "SELECT 1 FROM schema_migrations WHERE version = ? AND checksum = ?", + ( + PUBLISH_ACTIVE_INDEX_MIGRATION_VERSION, + PUBLISH_ACTIVE_INDEX_MIGRATION_CHECKSUM, + ), + ).fetchone() + except sqlite3.Error: + # 异常账本同样属于迁移风险:先要求备份,后续由结构验证给出明确错误。 + return True + return index_row is None or ledger_row is None + finally: + if connection is not None: + connection.close() + + +def _requires_task_upload_only_migration(database_path) -> bool: + """只有确实存在旧 NAS 来源数据时才在归一化前备份。""" + if not database_path.exists() or database_path.stat().st_size == 0: + return False + connection = None + try: + connection = sqlite3.connect(f"{database_path.resolve().as_uri()}?mode=ro", uri=True, timeout=10) + columns = {row[1] for row in connection.execute("PRAGMA table_info(tasks)").fetchall()} + if not {"source_type", "nas_file_path"} <= columns: + return False + row = connection.execute( + """ + SELECT 1 FROM tasks + WHERE source_type != 'upload' + OR source_type IS NULL + OR (nas_file_path IS NOT NULL AND TRIM(nas_file_path) != '') + LIMIT 1 + """ + ).fetchone() + return row is not None + finally: + if connection is not None: + connection.close() def _get_table_columns(connection: sqlite3.Connection, table_name: str) -> set[str]: @@ -369,8 +781,6 @@ def _get_table_columns(connection: sqlite3.Connection, table_name: str) -> set[s def _create_indexes(connection: sqlite3.Connection) -> None: """创建常用查询索引(IF NOT EXISTS 语法兼容 SQLite 3.27+)。""" - # v1.4 的索引把 FAILED 也视为活动任务,导致无法为失败记录创建新的重试任务。 - connection.execute("DROP INDEX IF EXISTS uq_publish_jobs_active_clip_platform_mode") indexes = [ # 任务列表与状态筛选 "CREATE INDEX IF NOT EXISTS idx_tasks_status_created ON tasks(status, created_at)", @@ -385,6 +795,11 @@ def _create_indexes(connection: sqlite3.Connection) -> None: "CREATE INDEX IF NOT EXISTS idx_clip_feedback_task_clip ON clip_feedback(task_id, clip_candidate_id)", # 字幕任务(按任务、输出切片、状态) "CREATE INDEX IF NOT EXISTS idx_subtitle_jobs_task_output_status ON subtitle_jobs(task_id, output_clip_id, status)", + "CREATE INDEX IF NOT EXISTS idx_subtitle_jobs_workflow_job ON subtitle_jobs(workflow_job_id)", + "CREATE INDEX IF NOT EXISTS idx_subtitle_tracks_task_type ON subtitle_tracks(task_id, track_type, is_active)", + "CREATE UNIQUE INDEX IF NOT EXISTS uq_subtitle_tracks_active_source ON subtitle_tracks(task_id) WHERE track_type = 'source' AND is_active = 1", + "CREATE INDEX IF NOT EXISTS idx_subtitle_revisions_track_created ON subtitle_revisions(track_id, created_at)", + "CREATE INDEX IF NOT EXISTS idx_subtitle_cues_revision_time ON subtitle_cues(revision_id, start_ms, cue_index)", # 发布任务(按状态、平台、时间;按任务、输出切片) "CREATE INDEX IF NOT EXISTS idx_publish_jobs_status_platform_created ON publish_jobs(status, platform, created_at)", "CREATE INDEX IF NOT EXISTS idx_publish_jobs_task_output ON publish_jobs(task_id, output_clip_id)", @@ -393,18 +808,188 @@ def _create_indexes(connection: sqlite3.Connection) -> None: "CREATE INDEX IF NOT EXISTS idx_publish_jobs_execution ON publish_jobs(execution_id)", "CREATE INDEX IF NOT EXISTS idx_publish_jobs_history_visibility ON publish_jobs(history_hidden, platform, status, created_at)", "CREATE INDEX IF NOT EXISTS idx_publish_job_events_job_time ON publish_job_events(job_id, occurred_at)", - """CREATE UNIQUE INDEX IF NOT EXISTS uq_publish_jobs_active_clip_platform_mode - ON publish_jobs(output_clip_id, platform, publish_mode) - WHERE status IN ('DRAFT', 'WAITING', 'SCHEDULED', 'PUBLISHING', 'NEED_REVIEW') - AND output_clip_id IS NOT NULL AND output_clip_id <> ''""", + "CREATE INDEX IF NOT EXISTS idx_workflow_jobs_claim ON workflow_jobs(status, next_attempt_at, created_at)", + "CREATE INDEX IF NOT EXISTS idx_workflow_jobs_task_type_status ON workflow_jobs(task_id, job_type, status)", + "CREATE INDEX IF NOT EXISTS idx_transcription_runs_task_active ON transcription_runs(task_id, is_active, updated_at)", + "CREATE INDEX IF NOT EXISTS idx_transcription_chunks_run_status ON transcription_chunks(run_id, status, chunk_index)", + "CREATE INDEX IF NOT EXISTS idx_ai_analysis_windows_resume ON ai_analysis_windows(task_id, transcript_fingerprint, provider, model, status, window_index)", # OAuth state 过期清理 "CREATE INDEX IF NOT EXISTS idx_oauth_states_expires ON oauth_states(expires_at)", ] for sql in indexes: + connection.execute(sql) + + +def _normalize_schema_sql(sql: str | None) -> str: + normalized = " ".join(str(sql or "").strip().rstrip(";").lower().split()) + return normalized.replace(" if not exists ", " ") + + +def _ensure_schema_migrations_table(connection: sqlite3.Connection) -> None: + connection.execute( + """ + CREATE TABLE IF NOT EXISTS schema_migrations ( + version TEXT PRIMARY KEY, + name TEXT NOT NULL, + checksum TEXT NOT NULL, + applied_at TEXT NOT NULL + ) + """ + ) + table_info = connection.execute("PRAGMA table_info(schema_migrations)").fetchall() + columns = {row[1] for row in table_info} + required_columns = {"version", "name", "checksum", "applied_at"} + missing_columns = sorted(required_columns - columns) + if missing_columns: + raise SchemaMigrationError( + "schema_migrations 账本结构不完整,缺少字段:" + ", ".join(missing_columns) + ) + version_column = next(row for row in table_info if row[1] == "version") + if version_column[5] != 1: + raise SchemaMigrationError("schema_migrations.version 不是主键,已拒绝继续迁移") + + +def _assert_no_duplicate_active_publish_jobs(connection: sqlite3.Connection) -> None: + rows = connection.execute( + f""" + SELECT output_clip_id, platform, publish_mode, COUNT(*) AS duplicate_count + FROM publish_jobs + WHERE status IN ({PUBLISH_UNIQUE_ACTIVE_STATUS_SQL}) + AND output_clip_id IS NOT NULL AND output_clip_id <> '' + GROUP BY output_clip_id, platform, publish_mode + HAVING COUNT(*) > 1 + ORDER BY output_clip_id, platform, publish_mode + LIMIT 5 + """ + ).fetchall() + if not rows: + return + samples = "; ".join( + f"{row['output_clip_id']}/{row['platform']}/{row['publish_mode']} x{row['duplicate_count']}" + for row in rows + ) + raise SchemaMigrationError( + "检测到重复的活动发布任务,拒绝建立唯一索引;请先人工核对后再迁移。示例:" + samples + ) + + +def _verify_publish_active_unique_index(connection: sqlite3.Connection) -> None: + row = connection.execute( + "SELECT sql FROM sqlite_master WHERE type = 'index' AND name = ?", + (PUBLISH_ACTIVE_UNIQUE_INDEX_NAME,), + ).fetchone() + if row is None: + raise SchemaMigrationError( + f"关键唯一索引 {PUBLISH_ACTIVE_UNIQUE_INDEX_NAME} 不存在,已拒绝启动" + ) + actual_sql = row["sql"] if isinstance(row, sqlite3.Row) else row[0] + if _normalize_schema_sql(actual_sql) != _normalize_schema_sql(PUBLISH_ACTIVE_UNIQUE_INDEX_SQL): + raise SchemaMigrationError( + f"关键唯一索引 {PUBLISH_ACTIVE_UNIQUE_INDEX_NAME} 定义发生漂移,已拒绝启动" + ) + + +def _verify_publish_active_unique_index_migration(connection: sqlite3.Connection) -> None: + _verify_publish_active_unique_index(connection) + legacy_row = connection.execute( + "SELECT 1 FROM sqlite_master WHERE type = 'index' AND name = ?", + (PUBLISH_ACTIVE_UNIQUE_INDEX_LEGACY_NAME,), + ).fetchone() + if legacy_row is not None: + raise SchemaMigrationError( + f"旧版唯一索引 {PUBLISH_ACTIVE_UNIQUE_INDEX_LEGACY_NAME} 仍存在,迁移未完整完成" + ) + + +def _apply_publish_active_unique_index_migration(connection: sqlite3.Connection) -> None: + # 先建立和验证新版索引;任一步失败时旧版索引仍保留在同一事务中。 + _assert_no_duplicate_active_publish_jobs(connection) + connection.execute(PUBLISH_ACTIVE_UNIQUE_INDEX_SQL) + _verify_publish_active_unique_index(connection) + connection.execute(f"DROP INDEX IF EXISTS {PUBLISH_ACTIVE_UNIQUE_INDEX_LEGACY_NAME}") + + +def _apply_task_upload_only_migration(connection: sqlite3.Connection) -> None: + connection.execute(TASK_UPLOAD_ONLY_MIGRATION_SQL) + + +def _verify_task_upload_only_migration(connection: sqlite3.Connection) -> None: + row = connection.execute( + """ + SELECT 1 FROM tasks + WHERE source_type != 'upload' + OR source_type IS NULL + OR (nas_file_path IS NOT NULL AND TRIM(nas_file_path) != '') + LIMIT 1 + """ + ).fetchone() + if row is not None: + raise SchemaMigrationError("仍存在未归一化的 NAS 视频来源记录") + + +def _registered_schema_migrations() -> tuple[SchemaMigration, ...]: + return ( + SchemaMigration( + version=PUBLISH_ACTIVE_INDEX_MIGRATION_VERSION, + name=PUBLISH_ACTIVE_INDEX_MIGRATION_NAME, + checksum=PUBLISH_ACTIVE_INDEX_MIGRATION_CHECKSUM, + apply=_apply_publish_active_unique_index_migration, + verify=_verify_publish_active_unique_index_migration, + ), + SchemaMigration( + version=TASK_UPLOAD_ONLY_MIGRATION_VERSION, + name=TASK_UPLOAD_ONLY_MIGRATION_NAME, + checksum=TASK_UPLOAD_ONLY_MIGRATION_CHECKSUM, + apply=_apply_task_upload_only_migration, + verify=_verify_task_upload_only_migration, + ), + ) + + +def _run_schema_migrations(connection: sqlite3.Connection) -> None: + """串行执行有账本的新迁移;历史列探测迁移继续作为兼容层保留。""" + if connection.in_transaction: + raise SchemaMigrationError("执行账本迁移前存在未提交事务,已拒绝继续") + + for migration in _registered_schema_migrations(): + version = migration.version try: - connection.execute(sql) - except sqlite3.Error: - pass + connection.execute("BEGIN IMMEDIATE") + _ensure_schema_migrations_table(connection) + applied = connection.execute( + "SELECT name, checksum FROM schema_migrations WHERE version = ?", + (version,), + ).fetchone() + if applied is not None: + applied_checksum = applied["checksum"] if isinstance(applied, sqlite3.Row) else applied[1] + if applied_checksum != migration.checksum: + raise SchemaMigrationError( + f"迁移 {version} 的 checksum 与账本不一致,已拒绝启动" + ) + migration.verify(connection) + connection.commit() + continue + + migration.apply(connection) + migration.verify(connection) + connection.execute( + """ + INSERT INTO schema_migrations (version, name, checksum, applied_at) + VALUES (?, ?, ?, ?) + """, + ( + version, + migration.name, + migration.checksum, + datetime.now().astimezone().isoformat(timespec="seconds"), + ), + ) + connection.commit() + except Exception as exc: + connection.rollback() + if isinstance(exc, SchemaMigrationError): + raise + raise SchemaMigrationError(f"数据库迁移 {version} 执行失败:{exc}") from exc def _migrate_tasks_table(connection: sqlite3.Connection) -> None: @@ -420,6 +1005,8 @@ def _migrate_tasks_table(connection: sqlite3.Connection) -> None: "candidate_clip_count": "ALTER TABLE tasks ADD COLUMN candidate_clip_count INTEGER NOT NULL DEFAULT 12", "selection_profile": "ALTER TABLE tasks ADD COLUMN selection_profile TEXT NOT NULL DEFAULT 'general'", "final_clip_target": "ALTER TABLE tasks ADD COLUMN final_clip_target INTEGER NOT NULL DEFAULT 5", + "highlight_density_per_hour": "ALTER TABLE tasks ADD COLUMN highlight_density_per_hour INTEGER NOT NULL DEFAULT 4", + "highlight_total_limit": "ALTER TABLE tasks ADD COLUMN highlight_total_limit INTEGER NOT NULL DEFAULT 30", "ai_preference": "ALTER TABLE tasks ADD COLUMN ai_preference TEXT", "ai_prompt_preset_id": "ALTER TABLE tasks ADD COLUMN ai_prompt_preset_id TEXT NOT NULL DEFAULT 'preset_001'", "auto_mode": "ALTER TABLE tasks ADD COLUMN auto_mode INTEGER NOT NULL DEFAULT 0", @@ -490,8 +1077,10 @@ def _migrate_tasks_table(connection: sqlite3.Connection) -> None: UPDATE tasks SET task_dir_name = id WHERE task_dir_name IS NULL OR task_dir_name = ''; UPDATE tasks SET is_deleted = 0 WHERE is_deleted IS NULL; UPDATE tasks SET ai_prompt_preset_id = 'preset_001' WHERE ai_prompt_preset_id IS NULL OR ai_prompt_preset_id = ''; - UPDATE tasks SET selection_profile = 'general' WHERE selection_profile NOT IN ('general', 'variety_comedy') OR selection_profile IS NULL OR selection_profile = ''; + UPDATE tasks SET selection_profile = 'general' WHERE selection_profile NOT IN ('general', 'variety_comedy', 'long_live_talk') OR selection_profile IS NULL OR selection_profile = ''; UPDATE tasks SET final_clip_target = 5 WHERE final_clip_target IS NULL OR final_clip_target < 1 OR final_clip_target > 12; + UPDATE tasks SET highlight_density_per_hour = 4 WHERE highlight_density_per_hour IS NULL OR highlight_density_per_hour < 1 OR highlight_density_per_hour > 10; + UPDATE tasks SET highlight_total_limit = 30 WHERE highlight_total_limit IS NULL OR highlight_total_limit < 1 OR highlight_total_limit > 50; UPDATE tasks SET source_type = 'upload' WHERE source_type NOT IN ('upload', 'nas') OR source_type IS NULL OR source_type = ''; UPDATE tasks SET platform = 'douyin' WHERE platform IN ('抖音', 'douyin'); @@ -512,10 +1101,12 @@ def _migrate_tasks_table(connection: sqlite3.Connection) -> None: UPDATE tasks SET status = 'failed' WHERE status IN ('失败', 'failed'); UPDATE tasks SET status = 'pending_video' WHERE status NOT IN ( 'CREATED', 'PREPARING_SOURCE', 'TRANSCRIBING', 'AI_ANALYZING', - 'CLIP_SELECTING', 'VIDEO_CUTTING', 'METADATA_GENERATING', + 'CLIP_SELECTING', 'VIDEO_CUTTING', 'SUBTITLE_DRAFTING', + 'PENDING_SUBTITLE_REVIEW', 'METADATA_GENERATING', 'SCHEDULE_CREATING', 'PUBLISH_JOB_CREATING', 'READY_TO_PUBLISH', - 'COMPLETED', 'FAILED_PREPARING_SOURCE', 'FAILED_TRANSCRIBING', + 'COMPLETED', 'CANCELLED', 'FAILED_PREPARING_SOURCE', 'FAILED_TRANSCRIBING', 'FAILED_AI_ANALYZING', 'FAILED_CLIP_SELECTING', 'FAILED_VIDEO_CUTTING', + 'FAILED_SUBTITLE_DRAFTING', 'FAILED_METADATA_GENERATING', 'FAILED_SCHEDULE_CREATING', 'FAILED_PUBLISH_JOB_CREATING', 'pending_video', 'pending_processing', 'audio_extracting', 'transcribing', @@ -590,6 +1181,11 @@ def _migrate_output_clip_table(connection: sqlite3.Connection) -> None: "updated_at": "ALTER TABLE output_clip ADD COLUMN updated_at TEXT NOT NULL DEFAULT ''", "cut_run_id": "ALTER TABLE output_clip ADD COLUMN cut_run_id TEXT", "is_active": "ALTER TABLE output_clip ADD COLUMN is_active INTEGER NOT NULL DEFAULT 1", + "source_start_ms": "ALTER TABLE output_clip ADD COLUMN source_start_ms INTEGER", + "source_end_ms": "ALTER TABLE output_clip ADD COLUMN source_end_ms INTEGER", + "source_duration_ms": "ALTER TABLE output_clip ADD COLUMN source_duration_ms INTEGER", + "source_fingerprint": "ALTER TABLE output_clip ADD COLUMN source_fingerprint TEXT", + "snapshot_source": "ALTER TABLE output_clip ADD COLUMN snapshot_source TEXT NOT NULL DEFAULT 'legacy_inferred'", } for column, statement in migrations.items(): @@ -662,6 +1258,10 @@ def _migrate_subtitle_style_presets_table(connection: sqlite3.Connection) -> Non "font_color": "ALTER TABLE subtitle_style_presets ADD COLUMN font_color TEXT NOT NULL DEFAULT '#ffffff'", "stroke_color": "ALTER TABLE subtitle_style_presets ADD COLUMN stroke_color TEXT NOT NULL DEFAULT '#111827'", "shadow_enabled": "ALTER TABLE subtitle_style_presets ADD COLUMN shadow_enabled INTEGER NOT NULL DEFAULT 1", + "outline_width": "ALTER TABLE subtitle_style_presets ADD COLUMN outline_width REAL NOT NULL DEFAULT 3", + "shadow_depth": "ALTER TABLE subtitle_style_presets ADD COLUMN shadow_depth REAL NOT NULL DEFAULT 1", + "safe_area_percent": "ALTER TABLE subtitle_style_presets ADD COLUMN safe_area_percent REAL NOT NULL DEFAULT 5", + "speaker_styles_json": "ALTER TABLE subtitle_style_presets ADD COLUMN speaker_styles_json TEXT NOT NULL DEFAULT '{}'", "is_default": "ALTER TABLE subtitle_style_presets ADD COLUMN is_default INTEGER NOT NULL DEFAULT 1", "created_at": "ALTER TABLE subtitle_style_presets ADD COLUMN created_at TEXT NOT NULL DEFAULT ''", "updated_at": "ALTER TABLE subtitle_style_presets ADD COLUMN updated_at TEXT NOT NULL DEFAULT ''", @@ -679,11 +1279,17 @@ def _migrate_subtitle_jobs_table(connection: sqlite3.Connection) -> None: migrations = { "task_id": "ALTER TABLE subtitle_jobs ADD COLUMN task_id TEXT", "output_clip_id": "ALTER TABLE subtitle_jobs ADD COLUMN output_clip_id TEXT", + "revision_id": "ALTER TABLE subtitle_jobs ADD COLUMN revision_id TEXT", + "workflow_job_id": "ALTER TABLE subtitle_jobs ADD COLUMN workflow_job_id TEXT", "style_preset_id": "ALTER TABLE subtitle_jobs ADD COLUMN style_preset_id TEXT", "status": "ALTER TABLE subtitle_jobs ADD COLUMN status TEXT NOT NULL DEFAULT 'pending'", "subtitle_file_path": "ALTER TABLE subtitle_jobs ADD COLUMN subtitle_file_path TEXT", "output_file_path": "ALTER TABLE subtitle_jobs ADD COLUMN output_file_path TEXT", "error_message": "ALTER TABLE subtitle_jobs ADD COLUMN error_message TEXT", + "validation_status": "ALTER TABLE subtitle_jobs ADD COLUMN validation_status TEXT NOT NULL DEFAULT 'legacy_unverified'", + "validation_json": "ALTER TABLE subtitle_jobs ADD COLUMN validation_json TEXT NOT NULL DEFAULT '{}'", + "encoder": "ALTER TABLE subtitle_jobs ADD COLUMN encoder TEXT NOT NULL DEFAULT ''", + "verified_at": "ALTER TABLE subtitle_jobs ADD COLUMN verified_at TEXT", "created_at": "ALTER TABLE subtitle_jobs ADD COLUMN created_at TEXT NOT NULL DEFAULT ''", "updated_at": "ALTER TABLE subtitle_jobs ADD COLUMN updated_at TEXT NOT NULL DEFAULT ''", "is_active": "ALTER TABLE subtitle_jobs ADD COLUMN is_active INTEGER NOT NULL DEFAULT 1", @@ -698,6 +1304,48 @@ def _migrate_subtitle_jobs_table(connection: sqlite3.Connection) -> None: connection.execute("UPDATE subtitle_jobs SET is_active = 1 WHERE is_active IS NULL") +def _migrate_subtitle_editor_tables(connection: sqlite3.Connection) -> None: + """创建不可变字幕轨、revision 与 cue 数据层。""" + connection.executescript( + """ + CREATE TABLE IF NOT EXISTS subtitle_tracks ( + id TEXT PRIMARY KEY, task_id TEXT NOT NULL, track_type TEXT NOT NULL, + output_clip_id TEXT, name TEXT NOT NULL, language TEXT NOT NULL DEFAULT 'zh-CN', + source_track_id TEXT, source_revision_id TEXT, source_fingerprint TEXT NOT NULL DEFAULT '', + active_revision_id TEXT, + sync_status TEXT NOT NULL DEFAULT 'up_to_date', + has_manual_edits INTEGER NOT NULL DEFAULT 0, is_active INTEGER NOT NULL DEFAULT 1, + created_at TEXT NOT NULL, updated_at TEXT NOT NULL, + UNIQUE(task_id, track_type, output_clip_id), + FOREIGN KEY(task_id) REFERENCES tasks(id), + FOREIGN KEY(output_clip_id) REFERENCES output_clip(id), + FOREIGN KEY(source_track_id) REFERENCES subtitle_tracks(id) + ); + CREATE TABLE IF NOT EXISTS subtitle_revisions ( + id TEXT PRIMARY KEY, track_id TEXT NOT NULL, revision_number INTEGER NOT NULL, + origin TEXT NOT NULL, parent_revision_id TEXT, status TEXT NOT NULL DEFAULT 'draft', + note TEXT, cue_count INTEGER NOT NULL DEFAULT 0, checksum TEXT NOT NULL, + created_at TEXT NOT NULL, approved_at TEXT, + UNIQUE(track_id, revision_number), + FOREIGN KEY(track_id) REFERENCES subtitle_tracks(id) ON DELETE CASCADE, + FOREIGN KEY(parent_revision_id) REFERENCES subtitle_revisions(id) + ); + CREATE TABLE IF NOT EXISTS subtitle_cues ( + id TEXT PRIMARY KEY, revision_id TEXT NOT NULL, cue_index INTEGER NOT NULL, + start_ms INTEGER NOT NULL, end_ms INTEGER NOT NULL, text TEXT NOT NULL, + confidence REAL, speaker TEXT NOT NULL DEFAULT '', source_cue_id TEXT, + created_at TEXT NOT NULL, UNIQUE(revision_id, cue_index), + FOREIGN KEY(revision_id) REFERENCES subtitle_revisions(id) ON DELETE CASCADE + ); + """ + ) + track_columns = _get_table_columns(connection, "subtitle_tracks") + if "source_fingerprint" not in track_columns: + connection.execute( + "ALTER TABLE subtitle_tracks ADD COLUMN source_fingerprint TEXT NOT NULL DEFAULT ''" + ) + + def _migrate_publish_platform_configs_table(connection: sqlite3.Connection) -> None: columns = _get_table_columns(connection, "publish_platform_configs") if not columns: @@ -1135,10 +1783,96 @@ def _migrate_workflow_jobs_table(connection: sqlite3.Connection) -> None: "updated_at": "ALTER TABLE workflow_jobs ADD COLUMN updated_at TEXT NOT NULL DEFAULT ''", "started_at": "ALTER TABLE workflow_jobs ADD COLUMN started_at TEXT", "finished_at": "ALTER TABLE workflow_jobs ADD COLUMN finished_at TEXT", + "attempt_count": "ALTER TABLE workflow_jobs ADD COLUMN attempt_count INTEGER NOT NULL DEFAULT 0", + "max_attempts": "ALTER TABLE workflow_jobs ADD COLUMN max_attempts INTEGER NOT NULL DEFAULT 3", + "next_attempt_at": "ALTER TABLE workflow_jobs ADD COLUMN next_attempt_at TEXT", + "lease_owner": "ALTER TABLE workflow_jobs ADD COLUMN lease_owner TEXT", + "lease_token": "ALTER TABLE workflow_jobs ADD COLUMN lease_token TEXT", + "lease_expires_at": "ALTER TABLE workflow_jobs ADD COLUMN lease_expires_at TEXT", + "heartbeat_at": "ALTER TABLE workflow_jobs ADD COLUMN heartbeat_at TEXT", + "cancel_requested": "ALTER TABLE workflow_jobs ADD COLUMN cancel_requested INTEGER NOT NULL DEFAULT 0", + "checkpoint_json": "ALTER TABLE workflow_jobs ADD COLUMN checkpoint_json TEXT", + "checkpoint_updated_at": "ALTER TABLE workflow_jobs ADD COLUMN checkpoint_updated_at TEXT", } for column, statement in migrations.items(): - if column not in columns: + if column in columns: + continue + try: connection.execute(statement) + except sqlite3.OperationalError: + # 两个本地进程可能同时初始化同一个旧库。另一个进程若已完成 + # 同一 ADD COLUMN,本进程把它视为幂等成功;其他错误继续抛出。 + if column not in _get_table_columns(connection, "workflow_jobs"): + raise + columns = _get_table_columns(connection, "workflow_jobs") + + +def _guard_unfenced_running_workflow_jobs(connection: sqlite3.Connection) -> None: + """部署切换不完整时拒绝启动,避免旧 Worker 与新 Worker 重叠写入。""" + rows = connection.execute( + """ + SELECT id + FROM workflow_jobs + WHERE status = 'running' AND (lease_token IS NULL OR lease_token = '') + ORDER BY created_at + LIMIT 10 + """ + ).fetchall() + if not rows: + return + job_ids = ", ".join(str(row["id"]) for row in rows) + raise RuntimeError( + "检测到未带 lease_token 的运行中 Workflow Job,已拒绝启动以防旧 Worker 覆盖新执行。" + f"请先停止旧版本服务并处理这些任务:{job_ids}" + ) + + +def _migrate_transcription_tables(connection: sqlite3.Connection) -> None: + """创建转写分块 checkpoint 表;重复启动不会覆盖现有结果。""" + connection.executescript( + """ + CREATE TABLE IF NOT EXISTS transcription_runs ( + id TEXT PRIMARY KEY, task_id TEXT NOT NULL, source_fingerprint TEXT NOT NULL, + provider TEXT NOT NULL, model TEXT NOT NULL DEFAULT '', device TEXT NOT NULL DEFAULT '', + compute_type TEXT NOT NULL DEFAULT '', chunk_seconds INTEGER NOT NULL, + overlap_seconds INTEGER NOT NULL, status TEXT NOT NULL DEFAULT 'processing', + total_chunks INTEGER NOT NULL DEFAULT 0, completed_chunks INTEGER NOT NULL DEFAULT 0, + is_active INTEGER NOT NULL DEFAULT 0, error_message TEXT, created_at TEXT NOT NULL, + updated_at TEXT NOT NULL, completed_at TEXT, FOREIGN KEY(task_id) REFERENCES tasks(id) + ); + CREATE TABLE IF NOT EXISTS transcription_chunks ( + id TEXT PRIMARY KEY, run_id TEXT NOT NULL, task_id TEXT NOT NULL, + chunk_index INTEGER NOT NULL, start_ms INTEGER NOT NULL, end_ms INTEGER NOT NULL, + status TEXT NOT NULL DEFAULT 'queued', attempt_count INTEGER NOT NULL DEFAULT 0, + result_json TEXT, result_checksum TEXT, error_message TEXT, created_at TEXT NOT NULL, + updated_at TEXT NOT NULL, UNIQUE(run_id, chunk_index), + FOREIGN KEY(run_id) REFERENCES transcription_runs(id) ON DELETE CASCADE, + FOREIGN KEY(task_id) REFERENCES tasks(id) + ); + """ + ) + + +def _migrate_ai_analysis_windows_table(connection: sqlite3.Connection) -> None: + """创建长直播 AI 窗口 checkpoint 表;成功窗口可跨进程复用。""" + connection.executescript( + """ + CREATE TABLE IF NOT EXISTS ai_analysis_windows ( + id TEXT PRIMARY KEY, task_id TEXT NOT NULL, + transcript_fingerprint TEXT NOT NULL, provider TEXT NOT NULL, + model TEXT NOT NULL DEFAULT '', window_index INTEGER NOT NULL, + start_seconds INTEGER NOT NULL, end_seconds INTEGER NOT NULL, + status TEXT NOT NULL DEFAULT 'queued', attempt_count INTEGER NOT NULL DEFAULT 0, + result_json TEXT, result_checksum TEXT, error_message TEXT, next_retry_at TEXT, + created_at TEXT NOT NULL, updated_at TEXT NOT NULL, completed_at TEXT, + UNIQUE( + task_id, transcript_fingerprint, provider, model, + window_index, start_seconds, end_seconds + ), + FOREIGN KEY(task_id) REFERENCES tasks(id) + ); + """ + ) def _migrate_cut_runs_table(connection: sqlite3.Connection) -> None: diff --git a/app/main.py b/app/main.py index c06b7df..f952b75 100644 --- a/app/main.py +++ b/app/main.py @@ -1,16 +1,21 @@ from contextlib import asynccontextmanager +import hmac +import ipaddress import os import tempfile +from urllib.parse import urlsplit from fastapi import FastAPI, Request, Response from fastapi.responses import FileResponse, JSONResponse +from fastapi.exceptions import RequestValidationError from fastapi.staticfiles import StaticFiles from app.core.config import settings from app.db.database import init_db -from app.routers import ai_prompts, files, media, pages, publish, settings as settings_router, tasks +from app.routers import ai_prompts, media, pages, publish, settings as settings_router, subtitles, tasks from app.services.publish_scheduler import start_scheduler_background from app.services.storage_service import configure_runtime_media_storage +from app.services.job_worker import WorkflowJobRunner # /media 和 /static 的 Origin 白名单 @@ -25,8 +30,20 @@ "http://127.0.0.1:5173", } -# /api 写方法(需要 token 校验) +# 静态媒体可读 Origin 与管理 API 可写 Origin 必须分离。平台创作者中心 +# 只需要读取受控媒体,不能因此获得本地管理写权限。 +_TRUSTED_WRITE_ORIGINS = { + "http://localhost:8001", + "http://127.0.0.1:8001", + "http://localhost:8000", + "http://127.0.0.1:8000", + "http://localhost:5173", + "http://127.0.0.1:5173", +} + _WRITE_METHODS = {"POST", "PUT", "PATCH", "DELETE"} +_LOOPBACK_HOSTS = {"localhost", "127.0.0.1", "::1"} +_PUBLIC_PATHS = {"/health", "/favicon.ico"} def _is_origin_allowed(origin: str) -> bool: @@ -38,31 +55,91 @@ def _is_origin_allowed(origin: str) -> bool: def _build_allow_origin_header(origin: str) -> str: if _is_origin_allowed(origin): return origin - # 对于 localhost / 127.0.0.1 的任意端口也放行(本地开发) - if origin.startswith("http://localhost:") or origin.startswith("http://127.0.0.1:"): - return origin return "null" +def _is_local_origin(origin: str) -> bool: + return origin in _TRUSTED_WRITE_ORIGINS + + +def _is_same_origin(request: Request, origin: str) -> bool: + try: + parsed = urlsplit(origin) + return ( + parsed.scheme.lower() == request.url.scheme.lower() + and (parsed.hostname or "").lower().rstrip(".") + == str(request.url.hostname or "").lower().rstrip(".") + and parsed.port == request.url.port + ) + except ValueError: + return False + + +def _is_loopback_address(value: str) -> bool: + try: + return ipaddress.ip_address(value).is_loopback + except ValueError: + return False + + +def _is_local_request(request: Request) -> bool: + hostname = str(request.url.hostname or "").lower().rstrip(".") + client_host = str(request.client.host if request.client else "").lower() + if hostname == "testserver" and client_host == "testclient": + return True + if hostname not in _LOOPBACK_HOSTS: + return False + if _is_loopback_address(client_host): + return True + # Docker NAT 无法把宿主机回环连接保留成 loopback client IP。该兼容仅在 + # Compose 显式开启,并与 127.0.0.1 host 端口绑定共同构成本地边界。 + return os.environ.get("NIUMA_TRUST_DOCKER_LOOPBACK_PROXY", "").strip().lower() == "true" + + +def _has_valid_admin_token(request: Request) -> bool: + configured = str(settings.local_admin_token or "") + if not configured: + return False + scheme, separator, credential = request.headers.get("Authorization", "").partition(" ") + if separator != " " or scheme.lower() != "bearer" or not credential: + return False + return hmac.compare_digest(credential.encode("utf-8"), configured.encode("utf-8")) + + +def _is_public_path(path: str) -> bool: + return path in _PUBLIC_PATHS or path.startswith("/static/") + + @asynccontextmanager async def lifespan(app: FastAPI): previous_temp = tempfile.tempdir previous_temp_env = {name: os.environ.get(name) for name in ("TEMP", "TMP")} - app.state.media_storage = configure_runtime_media_storage() - init_db() - scheduler = await start_scheduler_background() - app.state.publish_scheduler = scheduler + workflow_job_runner = None + scheduler = None try: + app.state.media_storage = configure_runtime_media_storage() + init_db() + workflow_job_runner = WorkflowJobRunner() + workflow_job_runner.start() + app.state.workflow_job_runner = workflow_job_runner + scheduler = await start_scheduler_background() + app.state.publish_scheduler = scheduler yield finally: - if scheduler: - scheduler.stop() - tempfile.tempdir = previous_temp - for name, value in previous_temp_env.items(): - if value is None: - os.environ.pop(name, None) - else: - os.environ[name] = value + try: + if workflow_job_runner: + workflow_job_runner.stop() + finally: + try: + if scheduler: + await scheduler.shutdown() + finally: + tempfile.tempdir = previous_temp + for name, value in previous_temp_env.items(): + if value is None: + os.environ.pop(name, None) + else: + os.environ[name] = value app = FastAPI( @@ -73,46 +150,83 @@ async def lifespan(app: FastAPI): ) +@app.exception_handler(RequestValidationError) +async def request_validation_error_handler(_request: Request, exc: RequestValidationError) -> JSONResponse: + for error in exc.errors(): + location = tuple(error.get("loc") or ()) + if "selection_profile" in location: + return JSONResponse(status_code=422, content={"detail": "请选择有效的选片模式"}) + safe_errors = [ + { + "type": str(error.get("type") or "validation_error"), + "loc": list(error.get("loc") or ()), + "msg": str(error.get("msg") or "输入校验失败"), + } + for error in exc.errors() + ] + return JSONResponse(status_code=422, content={"detail": safe_errors}) + + @app.middleware("http") async def security_middleware(request: Request, call_next): request_path = request.url.path request_origin = request.headers.get("Origin", "") - # ── 1. API 写保护 ── - if request_path.startswith("/api/") and request.method in _WRITE_METHODS: - token = settings.local_admin_token - if token: - auth_header = request.headers.get("Authorization", "") - expected = f"Bearer {token}" - if auth_header != expected: - return JSONResponse( - status_code=401, - content={"detail": "API 写操作需要有效的 LOCAL_ADMIN_TOKEN"}, - ) - - # ── 2. OPTIONS 预检请求处理 ── + # ── 1. 本地单用户门禁 ── + # 正式启动和 Docker 端口均只绑定 loopback;若用户主动改为远程暴露, + # 页面、API 和媒体必须携带独立 Bearer Token,且 Token 永不下发到浏览器。 + if not _is_public_path(request_path) and not _is_local_request(request): + if not settings.local_admin_token: + return JSONResponse( + status_code=403, + content={"detail": "NiuMa Studio 管理界面仅允许从本机访问"}, + ) + if not _has_valid_admin_token(request): + return JSONResponse( + status_code=401, + content={"detail": "远程访问需要有效的 LOCAL_ADMIN_TOKEN"}, + ) + + # ── 2. 浏览器跨站写保护 ── + if ( + request_path.startswith("/api/") + and request.method in _WRITE_METHODS + and request_origin + and not _is_same_origin(request, request_origin) + and not _is_local_origin(request_origin) + ): + return JSONResponse( + status_code=403, + content={"detail": "已拒绝非本机网页发起的写操作"}, + ) + + # ── 3. OPTIONS 预检请求处理 ── is_static_asset = request_path.startswith(("/media/", "/static/")) if is_static_asset and request.method == "OPTIONS": response = Response() else: response = await call_next(request) - # ── 3. 静态资源缓存 ── + # ── 4. 静态资源缓存 ── if request_path.startswith("/static/"): response.headers["Cache-Control"] = "no-store, max-age=0" response.headers["Pragma"] = "no-cache" response.headers["Expires"] = "0" - # ── 4. CORS / Private Network Access ── + # ── 5. CORS / Private Network Access ── if is_static_asset: allow_origin = _build_allow_origin_header(request_origin) response.headers["Access-Control-Allow-Origin"] = allow_origin response.headers["Access-Control-Allow-Methods"] = "GET, OPTIONS" response.headers["Access-Control-Allow-Headers"] = "*" # 仅白名单 Origin 返回 Private Network 预检头 - if _is_origin_allowed(request_origin) or request_origin.startswith("http://localhost:") or request_origin.startswith("http://127.0.0.1:"): + if _is_origin_allowed(request_origin) or _is_local_origin(request_origin): response.headers["Access-Control-Allow-Private-Network"] = "true" + response.headers["X-Content-Type-Options"] = "nosniff" + response.headers["Referrer-Policy"] = "same-origin" + response.headers["X-Frame-Options"] = "DENY" + return response @@ -120,7 +234,7 @@ async def security_middleware(request: Request, call_next): app.include_router(pages.router) app.include_router(ai_prompts.router) app.include_router(tasks.router) -app.include_router(files.router) +app.include_router(subtitles.router) app.include_router(media.router) app.include_router(publish.router) app.include_router(settings_router.router) diff --git a/app/models/settings.py b/app/models/settings.py index c3730e5..f7b8137 100644 --- a/app/models/settings.py +++ b/app/models/settings.py @@ -1,60 +1,132 @@ -from pydantic import BaseModel, Field +from urllib.parse import urlsplit + +from pydantic import BaseModel, Field, field_validator + + +_LOCAL_AI_HOSTS = {"localhost", "127.0.0.1", "::1", "host.docker.internal", "host.containers.internal"} +_REMOTE_URL_FIELDS = ( + "ai_analysis_remote_base_url", + "ai_publish_remote_base_url", + "ai_remote_base_url", +) +_RESPONSE_PATH_FIELDS = ( + "ai_analysis_remote_responses_path", + "ai_publish_remote_responses_path", + "ai_remote_responses_path", +) + + +def _validate_http_url(value: str, *, local_only: bool = False, https_only: bool = False) -> str: + text = value.strip() + if not text: + return text + parsed = urlsplit(text) + hostname = str(parsed.hostname or "").lower().rstrip(".") + try: + _ = parsed.port + except ValueError as exc: + raise ValueError("URL 端口无效") from exc + if parsed.scheme not in {"http", "https"} or not hostname: + raise ValueError("URL 必须是完整的 http:// 或 https:// 地址") + if parsed.username or parsed.password or parsed.query or parsed.fragment: + raise ValueError("URL 不能包含账号、密码、查询参数或片段") + if https_only and parsed.scheme != "https": + raise ValueError("该接口地址必须使用 https://") + if local_only and hostname not in _LOCAL_AI_HOSTS: + raise ValueError("本地 AI 地址只能使用 localhost、127.0.0.1 或容器宿主机地址") + if not local_only and parsed.scheme == "http" and hostname not in _LOCAL_AI_HOSTS: + raise ValueError("非本机接口地址必须使用 https://") + return text class AIConfigUpdate(BaseModel): - ai_default_provider: str = Field(default="codex", pattern="^(codex|remote|local)$") - ai_publish_provider: str = Field(default="codex", pattern="^(codex|remote|local)$") + ai_default_provider: str = Field(default="codex", pattern="^(codex|remote|local)$", max_length=20) + ai_publish_provider: str = Field(default="codex", pattern="^(codex|remote|local)$", max_length=20) ai_request_timeout_seconds: int = Field(default=120, ge=10, le=600) - ai_codex_path: str = Field(default="codex") - ai_codex_home: str = Field(default="") - ai_codex_model: str = Field(default="gpt-5.6-sol") + ai_codex_path: str = Field(default="codex", max_length=1000) + ai_codex_home: str = Field(default="", max_length=1000) + ai_codex_model: str = Field(default="gpt-5.6-sol", max_length=200) ai_codex_timeout_seconds: int = Field(default=300, ge=10, le=1800) - transcription_provider: str = Field(default="volcengine", pattern="^(volcengine|local)$") - transcription_fallback_provider: str = Field(default="") - volcengine_asr_api_url: str = Field(default="") - volcengine_asr_api_key: str = Field(default="") - volcengine_asr_app_key: str = Field(default="") - volcengine_asr_access_key: str = Field(default="") - volcengine_asr_resource_id: str = Field(default="volc.bigasr.auc_turbo") + transcription_provider: str = Field(default="volcengine", pattern="^(volcengine|local)$", max_length=20) + transcription_fallback_provider: str = Field(default="", pattern="^(|volcengine|local)$", max_length=20) + volcengine_asr_api_url: str = Field(default="", max_length=2000) + volcengine_asr_api_key: str = Field(default="", max_length=4096) + volcengine_asr_app_key: str = Field(default="", max_length=4096) + volcengine_asr_access_key: str = Field(default="", max_length=4096) + volcengine_asr_resource_id: str = Field(default="volc.bigasr.auc_turbo", max_length=200) volcengine_asr_timeout_seconds: int = Field(default=300, ge=30, le=1800) - volcengine_asr_audio_format: str = Field(default="mp3") - - ai_analysis_remote_base_url: str = Field(default="") - ai_analysis_remote_api_key: str = Field(default="") - ai_analysis_remote_model: str = Field(default="deepseek-v4-flash") - ai_analysis_remote_protocol: str = Field(default="chat_completions") - ai_analysis_remote_reasoning_effort: str = Field(default="") - ai_analysis_remote_responses_path: str = Field(default="/v1/responses") + volcengine_asr_audio_format: str = Field(default="mp3", pattern="^(mp3|ogg)$", max_length=10) + + ai_analysis_remote_base_url: str = Field(default="", max_length=2000) + ai_analysis_remote_api_key: str = Field(default="", max_length=4096) + ai_analysis_remote_model: str = Field(default="deepseek-v4-flash", max_length=200) + ai_analysis_remote_protocol: str = Field( + default="chat_completions", pattern="^(chat_completions|responses)$", max_length=30 + ) + ai_analysis_remote_reasoning_effort: str = Field(default="", max_length=80) + ai_analysis_remote_responses_path: str = Field(default="/v1/responses", max_length=300) ai_analysis_remote_disable_response_storage: bool = Field(default=True) ai_analysis_request_timeout_seconds: int = Field(default=120, ge=10, le=600) - ai_publish_remote_base_url: str = Field(default="") - ai_publish_remote_api_key: str = Field(default="") - ai_publish_remote_model: str = Field(default="deepseek-v4-flash") - ai_publish_remote_protocol: str = Field(default="chat_completions") - ai_publish_remote_reasoning_effort: str = Field(default="") - ai_publish_remote_responses_path: str = Field(default="/v1/responses") + ai_publish_remote_base_url: str = Field(default="", max_length=2000) + ai_publish_remote_api_key: str = Field(default="", max_length=4096) + ai_publish_remote_model: str = Field(default="deepseek-v4-flash", max_length=200) + ai_publish_remote_protocol: str = Field( + default="chat_completions", pattern="^(chat_completions|responses)$", max_length=30 + ) + ai_publish_remote_reasoning_effort: str = Field(default="", max_length=80) + ai_publish_remote_responses_path: str = Field(default="/v1/responses", max_length=300) ai_publish_remote_disable_response_storage: bool = Field(default=True) ai_publish_request_timeout_seconds: int = Field(default=120, ge=10, le=600) # Legacy fields are accepted so older pages or scripts do not break. - ai_remote_base_url: str = Field(default="") - ai_remote_api_key: str = Field(default="") - ai_remote_model: str = Field(default="deepseek-v4-flash") - ai_remote_review_model: str = Field(default="deepseek-v4-flash") - ai_remote_publish_model: str = Field(default="deepseek-v4-flash") - ai_remote_protocol: str = Field(default="chat_completions") - ai_remote_reasoning_effort: str = Field(default="") - ai_remote_responses_path: str = Field(default="/v1/responses") + ai_remote_base_url: str = Field(default="", max_length=2000) + ai_remote_api_key: str = Field(default="", max_length=4096) + ai_remote_model: str = Field(default="deepseek-v4-flash", max_length=200) + ai_remote_review_model: str = Field(default="deepseek-v4-flash", max_length=200) + ai_remote_publish_model: str = Field(default="deepseek-v4-flash", max_length=200) + ai_remote_protocol: str = Field(default="chat_completions", pattern="^(chat_completions|responses)$", max_length=30) + ai_remote_reasoning_effort: str = Field(default="", max_length=80) + ai_remote_responses_path: str = Field(default="/v1/responses", max_length=300) ai_remote_disable_response_storage: bool = Field(default=True) - ai_local_base_url: str = Field(default="") - ai_local_api_key: str = Field(default="") - ai_local_model: str = Field(default="") - ai_local_protocol: str = Field(default="chat_completions") - ai_local_fallback_protocol: str = Field(default="") + ai_local_base_url: str = Field(default="", max_length=2000) + ai_local_api_key: str = Field(default="", max_length=4096) + ai_local_model: str = Field(default="", max_length=200) + ai_local_protocol: str = Field(default="chat_completions", pattern="^(chat_completions|responses)$", max_length=30) + ai_local_fallback_protocol: str = Field(default="", pattern="^(|chat_completions|responses)$", max_length=30) ai_local_health_timeout_seconds: int = Field(default=30, ge=3, le=120) - ai_network_access: str = Field(default="enabled") + ai_network_access: str = Field(default="enabled", pattern="^(enabled|disabled)$", max_length=20) ai_windows_wsl_setup_acknowledged: bool = Field(default=True) - ai_model_context_window: int = Field(default=1000000, ge=1) - ai_model_auto_compact_token_limit: int = Field(default=900000, ge=1) + ai_model_context_window: int = Field(default=1000000, ge=1, le=10000000) + ai_model_auto_compact_token_limit: int = Field(default=900000, ge=1, le=10000000) + + @field_validator("*", mode="before") + @classmethod + def reject_control_characters(cls, value): + if isinstance(value, str) and any(ord(character) < 32 or ord(character) == 127 for character in value): + raise ValueError("配置值不能包含换行或控制字符") + return value + + @field_validator("volcengine_asr_api_url") + @classmethod + def validate_volcengine_url(cls, value: str) -> str: + return _validate_http_url(value, https_only=True) + + @field_validator(*_REMOTE_URL_FIELDS) + @classmethod + def validate_remote_url(cls, value: str) -> str: + return _validate_http_url(value) + + @field_validator("ai_local_base_url") + @classmethod + def validate_local_url(cls, value: str) -> str: + return _validate_http_url(value, local_only=True) + + @field_validator(*_RESPONSE_PATH_FIELDS) + @classmethod + def validate_response_path(cls, value: str) -> str: + text = value.strip() + if not text.startswith("/") or text.startswith("//") or "\\" in text or "://" in text: + raise ValueError("Responses 路径必须是站内绝对路径,例如 /v1/responses") + return text diff --git a/app/models/subtitle.py b/app/models/subtitle.py new file mode 100644 index 0000000..25867ec --- /dev/null +++ b/app/models/subtitle.py @@ -0,0 +1,86 @@ +from __future__ import annotations + +from typing import Any, Literal + +from pydantic import BaseModel, Field, model_validator + + +class SubtitleCueInput(BaseModel): + id: str | None = Field(default=None, max_length=64) + start_ms: int = Field(ge=0) + end_ms: int = Field(gt=0) + text: str = Field(min_length=1, max_length=4000) + confidence: float | None = Field(default=None, ge=0, le=1) + speaker: str = Field(default="", max_length=80) + source_cue_id: str | None = Field(default=None, max_length=64) + + @model_validator(mode="after") + def validate_range(self): + if self.end_ms <= self.start_ms: + raise ValueError("字幕结束时间必须晚于开始时间") + return self + + +class SubtitleRevisionCreate(BaseModel): + base_revision_id: str | None = Field(default=None, max_length=64) + cues: list[SubtitleCueInput] = Field(default_factory=list, max_length=20000) + note: str = Field(default="", max_length=500) + + +class SubtitleOperation(BaseModel): + type: Literal["update", "split", "merge", "add", "delete", "shift", "replace"] + cue_id: str | None = Field(default=None, max_length=64) + cue_ids: list[str] = Field(default_factory=list, max_length=20000) + start_ms: int | None = Field(default=None, ge=0) + end_ms: int | None = Field(default=None, gt=0) + split_ms: int | None = Field(default=None, gt=0) + text: str | None = Field(default=None, max_length=4000) + second_text: str | None = Field(default=None, max_length=4000) + speaker: str | None = Field(default=None, max_length=80) + confidence: float | None = Field(default=None, ge=0, le=1) + delta_ms: int | None = Field(default=None, ge=-86_400_000, le=86_400_000) + search: str | None = Field(default=None, max_length=500) + replacement: str | None = Field(default=None, max_length=500) + cue: SubtitleCueInput | None = None + + +class SubtitleOperationsRequest(BaseModel): + base_revision_id: str = Field(min_length=1, max_length=64) + operations: list[SubtitleOperation] = Field(min_length=1, max_length=1000) + note: str = Field(default="", max_length=500) + + +class SubtitleApproveRequest(BaseModel): + revision_id: str = Field(min_length=1, max_length=64) + + +class SubtitleSyncRequest(BaseModel): + force: bool = False + + +class SubtitleAIRevisionRequest(BaseModel): + revision_id: str = Field(min_length=1, max_length=64) + cue_ids: list[str] = Field(default_factory=list, max_length=500) + instructions: str = Field(default="", max_length=2000) + + +class SubtitleAISuggestionAcceptRequest(BaseModel): + base_revision_id: str = Field(min_length=1, max_length=64) + cue_ids: list[str] = Field(min_length=1, max_length=500) + + +class SubtitleTaskRenderRequest(BaseModel): + approve_active_revisions: bool = True + + +class SubtitleStyleExtendedUpdate(BaseModel): + font_family: str = Field(default="Microsoft YaHei", min_length=1, max_length=120) + font_size: int = Field(default=42, ge=12, le=160) + position: Literal["bottom_center", "middle_lower", "top_center"] = "bottom_center" + font_color: str = Field(default="#ffffff", pattern=r"^#[0-9A-Fa-f]{6}$") + stroke_color: str = Field(default="#111827", pattern=r"^#[0-9A-Fa-f]{6}$") + shadow_enabled: bool = True + outline_width: float = Field(default=3, ge=0, le=20) + shadow_depth: float = Field(default=1, ge=0, le=20) + safe_area_percent: float = Field(default=5, ge=0, le=25) + speaker_styles: dict[str, dict[str, Any]] = Field(default_factory=dict) diff --git a/app/models/task.py b/app/models/task.py index acc1050..7a2c197 100644 --- a/app/models/task.py +++ b/app/models/task.py @@ -1,7 +1,38 @@ from enum import Enum from typing import Any, Literal, Optional +from urllib.parse import urlsplit -from pydantic import BaseModel, Field, validator +from pydantic import BaseModel, Field, field_validator, validator + + +_PUBLISH_CONFIG_URL_FIELDS = ( + "redirect_uri", + "api_base_url", + "auth_url", + "token_url", + "refresh_url", + "upload_url", + "create_url", +) + + +def _validate_publish_config_url(value: str) -> str: + text = value.strip() + if not text: + return text + parsed = urlsplit(text) + hostname = str(parsed.hostname or "").lower().rstrip(".") + try: + _ = parsed.port + except ValueError as exc: + raise ValueError("发布接口 URL 端口无效") from exc + if parsed.scheme not in {"http", "https"} or not hostname: + raise ValueError("发布接口必须是完整的 http:// 或 https:// URL") + if parsed.username or parsed.password or parsed.query or parsed.fragment: + raise ValueError("发布接口 URL 不能包含账号、密码、查询参数或片段") + if parsed.scheme != "https" and hostname not in {"localhost", "127.0.0.1", "::1"}: + raise ValueError("非本机发布接口必须使用 https://") + return text class TaskStatus(str, Enum): @@ -11,16 +42,20 @@ class TaskStatus(str, Enum): AI_ANALYZING = "AI_ANALYZING" CLIP_SELECTING = "CLIP_SELECTING" VIDEO_CUTTING = "VIDEO_CUTTING" + SUBTITLE_DRAFTING = "SUBTITLE_DRAFTING" + PENDING_SUBTITLE_REVIEW = "PENDING_SUBTITLE_REVIEW" METADATA_GENERATING = "METADATA_GENERATING" SCHEDULE_CREATING = "SCHEDULE_CREATING" PUBLISH_JOB_CREATING = "PUBLISH_JOB_CREATING" READY_TO_PUBLISH = "READY_TO_PUBLISH" COMPLETED = "COMPLETED" + CANCELLED = "CANCELLED" FAILED_PREPARING_SOURCE = "FAILED_PREPARING_SOURCE" FAILED_TRANSCRIBING = "FAILED_TRANSCRIBING" FAILED_AI_ANALYZING = "FAILED_AI_ANALYZING" FAILED_CLIP_SELECTING = "FAILED_CLIP_SELECTING" FAILED_VIDEO_CUTTING = "FAILED_VIDEO_CUTTING" + FAILED_SUBTITLE_DRAFTING = "FAILED_SUBTITLE_DRAFTING" FAILED_METADATA_GENERATING = "FAILED_METADATA_GENERATING" FAILED_SCHEDULE_CREATING = "FAILED_SCHEDULE_CREATING" FAILED_PUBLISH_JOB_CREATING = "FAILED_PUBLISH_JOB_CREATING" @@ -39,14 +74,14 @@ class TaskStatus(str, Enum): class TaskCreate(BaseModel): task_name: str = Field(..., min_length=1, max_length=120) - source_type: Literal["upload", "nas"] = "upload" platform: Literal["douyin", "bilibili", "general"] = "general" original_video_path: Optional[str] = None - nas_file_path: Optional[str] = None max_clip_duration: int = Field(default=10, ge=1, le=60) candidate_clip_count: int = Field(default=12, ge=1, le=50) - selection_profile: Literal["general", "variety_comedy"] = "variety_comedy" + selection_profile: Literal["general", "variety_comedy", "long_live_talk"] final_clip_target: int = Field(default=5, ge=1, le=12) + highlight_density_per_hour: int = Field(default=4, ge=1, le=10) + highlight_total_limit: int = Field(default=30, ge=1, le=50) ai_preference: Optional[str] = None auto_mode: bool = False auto_clip_count: str = Field(default="auto", max_length=10) @@ -86,7 +121,7 @@ class TaskSummary(BaseModel): class TaskStatusUpdate(BaseModel): status: TaskStatus - error_message: Optional[str] = None + error_message: Optional[str] = Field(default=None, max_length=2000) class TaskAIPreferenceUpdate(BaseModel): @@ -107,8 +142,10 @@ class TaskCandidateClipCountUpdate(BaseModel): class TaskSelectionSettingsUpdate(BaseModel): - selection_profile: Literal["general", "variety_comedy"] = "variety_comedy" + selection_profile: Literal["general", "variety_comedy", "long_live_talk"] final_clip_target: int = Field(default=5, ge=1, le=12) + highlight_density_per_hour: int = Field(default=4, ge=1, le=10) + highlight_total_limit: int = Field(default=30, ge=1, le=50) class ClipFeedbackCreate(BaseModel): @@ -132,6 +169,10 @@ class SubtitleStyleUpdate(BaseModel): font_color: str = Field(default="#ffffff", pattern=r"^#[0-9a-fA-F]{6}$") stroke_color: str = Field(default="#111827", pattern=r"^#[0-9a-fA-F]{6}$") shadow_enabled: bool = True + outline_width: float = Field(default=3, ge=0, le=20) + shadow_depth: float = Field(default=1, ge=0, le=20) + safe_area_percent: float = Field(default=5, ge=0, le=25) + speaker_styles: dict[str, dict[str, Any]] = Field(default_factory=dict) class PublishPlatformConfigUpdate(BaseModel): @@ -148,6 +189,18 @@ class PublishPlatformConfigUpdate(BaseModel): create_url: str = Field(default="", max_length=1000) extra_config: Optional[str] = Field(default="", max_length=4000) + @field_validator("app_name", "client_key", "client_secret", "scope", mode="before") + @classmethod + def reject_config_control_characters(cls, value): + if isinstance(value, str) and any(ord(character) < 32 or ord(character) == 127 for character in value): + raise ValueError("发布配置不能包含换行或控制字符") + return value + + @field_validator(*_PUBLISH_CONFIG_URL_FIELDS) + @classmethod + def validate_config_urls(cls, value: str) -> str: + return _validate_publish_config_url(value) + class PublishAccountCreate(BaseModel): platform: Literal["douyin", "bilibili"] @@ -391,3 +444,4 @@ class AIClipAnalysisResult(BaseModel): task_id: str | int = "" analysis_summary: str = "" clips: list[AIClipItem] = Field(default_factory=list) + analysis_meta: dict[str, Any] = Field(default_factory=dict) diff --git a/app/routers/files.py b/app/routers/files.py deleted file mode 100644 index 4a00a61..0000000 --- a/app/routers/files.py +++ /dev/null @@ -1,11 +0,0 @@ -from fastapi import APIRouter - -from app.services.storage_service import browse_video_directory - - -router = APIRouter(prefix="/api/files", tags=["files"]) - - -@router.get("/browse") -async def browse_files(path: str | None = None) -> dict: - return browse_video_directory(path) diff --git a/app/routers/media.py b/app/routers/media.py index a709cc6..fdc1ad7 100644 --- a/app/routers/media.py +++ b/app/routers/media.py @@ -4,8 +4,14 @@ from fastapi import APIRouter, HTTPException from fastapi.responses import FileResponse -from app.services import task_service -from app.services.storage_service import get_artifact_paths, get_source_video_path, resolve_video_file_path, validate_source_video_path +from app.services import storage_service, task_service +from app.services.storage_service import ( + IMAGE_EXTENSIONS, + get_artifact_paths, + get_source_video_path, + resolve_task_media_file_path, + validate_source_video_path, +) router = APIRouter(prefix="/media/tasks", tags=["media"]) @@ -37,7 +43,10 @@ async def get_task_source_video(task_id: str) -> FileResponse: if not task: raise HTTPException(status_code=404, detail="任务不存在") - source_path = get_source_video_path(task) + try: + source_path = get_source_video_path(task) + except storage_service.StorageSafetyError as exc: + raise HTTPException(status_code=404, detail="源视频不存在") from exc valid, error_message = validate_source_video_path(str(source_path) if source_path else None) if not valid or source_path is None: raise HTTPException(status_code=404, detail=error_message or "源视频不存在") @@ -55,8 +64,13 @@ async def get_task_output_clip(task_id: str, output_clip_id: str) -> FileRespons if not output_clip: raise HTTPException(status_code=404, detail="切片记录不存在") - output_path = resolve_video_file_path(output_clip.get("output_file_path")) or Path(output_clip.get("output_file_path") or "") - if not output_path.exists() or not output_path.is_file(): + output_path = resolve_task_media_file_path( + output_clip.get("output_file_path"), + task_id=task_id, + task_dir_name=task.get("task_dir_name"), + allowed_subdirectories=("05_clips", "clips"), + ) + if output_path is None or not output_path.exists() or not output_path.is_file(): raise HTTPException(status_code=404, detail="切片视频文件不存在") return _video_response(output_path) @@ -72,8 +86,13 @@ async def get_task_subtitled_clip(task_id: str, output_clip_id: str) -> FileResp if not output_clip: raise HTTPException(status_code=404, detail="切片记录不存在") - output_path = resolve_video_file_path(output_clip.get("subtitled_output_file_path")) or Path(output_clip.get("subtitled_output_file_path") or "") - if not output_path.exists() or not output_path.is_file(): + output_path = resolve_task_media_file_path( + output_clip.get("subtitled_output_file_path"), + task_id=task_id, + task_dir_name=task.get("task_dir_name"), + allowed_subdirectories=("06_subtitled",), + ) + if output_path is None or not output_path.exists() or not output_path.is_file(): raise HTTPException(status_code=404, detail="带字幕视频文件不存在") return _video_response(output_path) @@ -86,8 +105,18 @@ async def get_task_cover(task_id: str, file_name: str) -> FileResponse: raise HTTPException(status_code=404, detail="任务不存在") safe_name = Path(file_name).name - cover_path = get_artifact_paths(task_id, task.get("task_dir_name"))["covers_dir"] / safe_name - if not cover_path.exists() or not cover_path.is_file(): + try: + candidate_path = get_artifact_paths(task_id, task.get("task_dir_name"))["covers_dir"] / safe_name + cover_path = resolve_task_media_file_path( + str(candidate_path), + task_id=task_id, + task_dir_name=task.get("task_dir_name"), + allowed_subdirectories=("07_covers",), + allowed_extensions=IMAGE_EXTENSIONS, + ) + except storage_service.StorageSafetyError as exc: + raise HTTPException(status_code=404, detail="封面文件不存在") from exc + if cover_path is None or not cover_path.exists() or not cover_path.is_file(): raise HTTPException(status_code=404, detail="封面文件不存在") return _image_response(cover_path) diff --git a/app/routers/subtitles.py b/app/routers/subtitles.py new file mode 100644 index 0000000..9bf91e6 --- /dev/null +++ b/app/routers/subtitles.py @@ -0,0 +1,256 @@ +from __future__ import annotations + +from urllib.parse import quote + +from fastapi import APIRouter, File, Form, HTTPException, Query, Response, UploadFile + +from app.models.subtitle import ( + SubtitleApproveRequest, + SubtitleAISuggestionAcceptRequest, + SubtitleAIRevisionRequest, + SubtitleOperationsRequest, + SubtitleRevisionCreate, + SubtitleTaskRenderRequest, + SubtitleSyncRequest, +) +from app.services import job_service +from app.services.subtitle_ai_service import generate_subtitle_suggestions +from app.services.subtitle_auto_workflow_service import ( + enqueue_task_subtitle_render, + prepare_task_subtitle_review, + skip_task_subtitles_and_resume, + skip_task_subtitles_to_review, +) +from app.services.subtitle_data_service import ( + SubtitleRevisionConflict, + accept_suggestion_revision, + apply_revision_operations, + approve_revision, + create_manual_revision, + ensure_source_track, + export_subtitle_text, + get_revision_cues, + get_track, + get_waveform_peaks, + import_subtitle_text, + list_revisions, + list_task_tracks, + sync_clip_track, +) + + +router = APIRouter(prefix="/api/subtitles", tags=["subtitles"]) + + +@router.get("/tasks/{task_id}/tracks") +def list_tracks(task_id: str, ensure: bool = Query(default=True)) -> dict: + return _call(lambda: {"tracks": list_task_tracks(task_id, ensure=ensure)}) + + +@router.post("/tasks/{task_id}/source-track") +def generate_source_track(task_id: str, payload: SubtitleSyncRequest) -> dict: + return _call(lambda: {"track": ensure_source_track(task_id, force=payload.force)}) + + +@router.post("/tasks/{task_id}/prepare-review") +def prepare_review(task_id: str) -> dict: + return _call(lambda: prepare_task_subtitle_review(task_id)) + + +@router.post("/tasks/{task_id}/approve-and-render") +def approve_and_render(task_id: str, payload: SubtitleTaskRenderRequest) -> dict: + return _call( + lambda: enqueue_task_subtitle_render( + task_id, + approve_active_revisions=payload.approve_active_revisions, + continue_pipeline=True, + ) + ) + + +@router.post("/tasks/{task_id}/skip-and-resume") +def skip_and_resume(task_id: str) -> dict: + return _call(lambda: skip_task_subtitles_and_resume(task_id)) + + +@router.post("/tasks/{task_id}/skip-to-review") +def skip_to_review(task_id: str) -> dict: + return _call(lambda: skip_task_subtitles_to_review(task_id)) + + +@router.get("/tasks/{task_id}/jobs") +def list_subtitle_jobs(task_id: str) -> dict: + jobs = [ + job + for job in job_service.list_jobs(task_id=task_id) + if job.get("job_type") == job_service.JOB_TYPE_SUBTITLE + ] + return {"task_id": task_id, "jobs": jobs, "count": len(jobs)} + + +@router.get("/tracks/{track_id}") +def read_track(track_id: str) -> dict: + return _call(lambda: {"track": get_track(track_id)}) + + +@router.get("/tracks/{track_id}/revisions") +def read_revisions(track_id: str) -> dict: + return _call(lambda: {"revisions": list_revisions(track_id)}) + + +@router.get("/tracks/{track_id}/cues") +def read_cues( + track_id: str, + revision_id: str | None = Query(default=None), + start_ms: int | None = Query(default=None, ge=0), + end_ms: int | None = Query(default=None, ge=0), + offset: int = Query(default=0, ge=0), + limit: int = Query(default=500, ge=1, le=2000), +) -> dict: + return _call( + lambda: get_revision_cues( + track_id, + revision_id=revision_id, + start_ms=start_ms, + end_ms=end_ms, + offset=offset, + limit=limit, + ) + ) + + +@router.post("/tracks/{track_id}/revisions") +def save_revision(track_id: str, payload: SubtitleRevisionCreate) -> dict: + return _call( + lambda: { + "revision": create_manual_revision( + track_id, + base_revision_id=payload.base_revision_id, + cues=payload.cues, + note=payload.note, + ) + } + ) + + +@router.post("/tracks/{track_id}/operations") +def apply_operations(track_id: str, payload: SubtitleOperationsRequest) -> dict: + return _call( + lambda: { + "revision": apply_revision_operations( + track_id, + base_revision_id=payload.base_revision_id, + operations=payload.operations, + note=payload.note, + ) + } + ) + + +@router.post("/tracks/{track_id}/approve") +def approve(track_id: str, payload: SubtitleApproveRequest) -> dict: + return _call(lambda: {"revision": approve_revision(track_id, payload.revision_id)}) + + +@router.post("/tracks/{track_id}/sync-source") +def sync_source(track_id: str, payload: SubtitleSyncRequest) -> dict: + return _call(lambda: {"track": sync_clip_track(track_id, force=payload.force)}) + + +@router.post("/tracks/{track_id}/ai-suggestions") +def ai_suggestions(track_id: str, payload: SubtitleAIRevisionRequest) -> dict: + return _call( + lambda: generate_subtitle_suggestions( + track_id, + revision_id=payload.revision_id, + cue_ids=payload.cue_ids, + instructions=payload.instructions, + ) + ) + + +@router.post("/tracks/{track_id}/ai-suggestions/{suggestion_revision_id}/accept") +def accept_ai_suggestions( + track_id: str, + suggestion_revision_id: str, + payload: SubtitleAISuggestionAcceptRequest, +) -> dict: + return _call( + lambda: { + "revision": accept_suggestion_revision( + track_id, + suggestion_revision_id=suggestion_revision_id, + base_revision_id=payload.base_revision_id, + cue_ids=payload.cue_ids, + ) + } + ) + + +@router.post("/tracks/{track_id}/import") +async def import_subtitle( + track_id: str, + file: UploadFile = File(...), + format_name: str | None = Form(default=None), +) -> dict: + raw = await file.read(10 * 1024 * 1024 + 1) + if len(raw) > 10 * 1024 * 1024: + raise HTTPException(status_code=413, detail="字幕文件不能超过 10 MB") + try: + content = raw.decode("utf-8-sig") + except UnicodeDecodeError: + try: + content = raw.decode("gb18030") + except UnicodeDecodeError as exc: + raise HTTPException(status_code=422, detail="字幕文件必须是 UTF-8 或 GB18030 文本") from exc + resolved_format = format_name or (file.filename or "").rsplit(".", 1)[-1] + return _call( + lambda: { + "revision": import_subtitle_text( + track_id, + content=content, + format_name=resolved_format, + note=f"导入文件:{file.filename or 'subtitle'}", + ) + } + ) + + +@router.get("/tracks/{track_id}/export") +def export_subtitle( + track_id: str, + format_name: str = Query(pattern=r"^(srt|vtt|ass)$"), + revision_id: str | None = Query(default=None), +) -> Response: + try: + content, media_type, filename = export_subtitle_text( + track_id, + revision_id=revision_id, + format_name=format_name, + ) + except ValueError as exc: + raise HTTPException(status_code=422, detail=str(exc)) from exc + return Response( + content=content.encode("utf-8-sig"), + media_type=f"{media_type}; charset=utf-8", + headers={"Content-Disposition": f"attachment; filename*=UTF-8''{quote(filename)}"}, + ) + + +@router.get("/tracks/{track_id}/peaks") +def waveform_peaks( + track_id: str, + max_points: int = Query(default=12000, ge=1000, le=50000), +) -> dict: + return _call(lambda: get_waveform_peaks(track_id, max_points=max_points)) + + +def _call(callback): + try: + return callback() + except SubtitleRevisionConflict as exc: + raise HTTPException(status_code=409, detail=str(exc)) from exc + except ValueError as exc: + raise HTTPException(status_code=422, detail=str(exc)) from exc + except RuntimeError as exc: + raise HTTPException(status_code=503, detail=str(exc)) from exc diff --git a/app/routers/tasks.py b/app/routers/tasks.py index eabffb5..a788890 100644 --- a/app/routers/tasks.py +++ b/app/routers/tasks.py @@ -27,7 +27,6 @@ ) from app.services.task_lifecycle_service import TaskDeletionConflictError from app.services import job_service -from app.services import job_worker router = APIRouter(prefix="/api/tasks", tags=["tasks"]) @@ -38,17 +37,6 @@ async def list_tasks() -> list[dict]: return task_service.list_tasks() -@router.post("") -async def create_task(payload: TaskCreate, background_tasks: BackgroundTasks) -> dict: - try: - result = task_service.create_task_record(payload) - if payload.auto_mode: - result["auto_pipeline"] = start_auto_pipeline(result["id"], background_tasks=background_tasks) - return result - except ValueError as exc: - raise HTTPException(status_code=400, detail=str(exc)) from exc - - @router.post("/upload") async def create_upload_task( background_tasks: BackgroundTasks, @@ -56,8 +44,10 @@ async def create_upload_task( platform: str = Form("general"), max_clip_duration: int = Form(10), candidate_clip_count: int = Form(12), - selection_profile: str = Form("variety_comedy"), + selection_profile: str | None = Form(None), final_clip_target: int = Form(5), + highlight_density_per_hour: int = Form(4), + highlight_total_limit: int = Form(30), ai_preference: str | None = Form(None), auto_mode: bool = Form(False), auto_clip_count: str = Form("auto"), @@ -71,6 +61,8 @@ async def create_upload_task( auto_metadata_use_ai: bool = Form(False), video_file: UploadFile = File(...), ) -> dict: + if not selection_profile: + raise HTTPException(status_code=422, detail="请选择选片模式") task_id = uuid4().hex[:12] task_dir_name = allocate_task_dir_name(task_name, exclude_task_id=task_id) task_record_created = False @@ -84,13 +76,14 @@ async def create_upload_task( ) payload = TaskCreate( task_name=task_name, - source_type="upload", platform=platform, original_video_path=str(saved_path), max_clip_duration=max_clip_duration, candidate_clip_count=candidate_clip_count, selection_profile=selection_profile, final_clip_target=final_clip_target, + highlight_density_per_hour=(highlight_density_per_hour if selection_profile == "long_live_talk" else 4), + highlight_total_limit=(highlight_total_limit if selection_profile == "long_live_talk" else 30), ai_preference=ai_preference, auto_mode=auto_mode, auto_clip_count=auto_clip_count, @@ -168,7 +161,10 @@ async def delete_task(task_id: str) -> dict: @router.patch("/{task_id}/status") async def patch_task_status(task_id: str, payload: TaskStatusUpdate) -> dict: - task = task_service.update_task_status(task_id, payload.status, payload.error_message) + try: + task = task_service.transition_task_status(task_id, payload.status, payload.error_message) + except task_service.TaskStatusConflictError as exc: + raise HTTPException(status_code=409, detail=str(exc)) from exc if not task: raise HTTPException(status_code=404, detail="任务不存在") return task @@ -205,6 +201,8 @@ async def patch_task_selection_settings(task_id: str, payload: TaskSelectionSett task_id, payload.selection_profile, payload.final_clip_target, + payload.highlight_density_per_hour, + payload.highlight_total_limit, ) except ValueError as exc: raise HTTPException(status_code=400, detail=str(exc)) from exc @@ -242,12 +240,23 @@ async def process_transcript_workflow( provider: str | None = Query(default=None, pattern="^(remote|local)$"), ) -> dict: try: - return task_service.process_task_transcript_workflow( - task_id, - background_tasks=background_tasks, - force=force, - provider=provider, + task = task_service.get_task(task_id, include_video_probe=False) + if not task: + raise ValueError("任务不存在") + if task.get("transcript_exists") and not force: + return {"status": "completed", "message": "转写已经生成,无需重复处理。", "task": task} + job, created = job_service.create_or_get_active_job( + task_id=task_id, + job_type=job_service.JOB_TYPE_TRANSCRIPT, + payload={"force": force, "provider": provider}, ) + return { + "status": job["status"], + "message": "转写任务已加入持久化队列" if created else "已有转写任务正在排队或运行", + "job_id": job["id"], + "job": job, + "task": task, + } except ValueError as exc: raise HTTPException(status_code=400, detail=str(exc)) from exc except RuntimeError as exc: @@ -257,6 +266,13 @@ async def process_transcript_workflow( @router.post("/{task_id}/process/transcript-cancel") async def cancel_transcript(task_id: str) -> dict: try: + active_jobs = [ + job for job in job_service.list_jobs(task_id=task_id) + if job.get("job_type") == job_service.JOB_TYPE_TRANSCRIPT + and job.get("status") in {job_service.JOB_STATUS_QUEUED, job_service.JOB_STATUS_RUNNING} + ] + if active_jobs: + job_service.request_job_cancel(active_jobs[0]["id"]) return task_service.cancel_task_transcript(task_id) except ValueError as exc: raise HTTPException(status_code=404, detail=str(exc)) from exc @@ -268,7 +284,15 @@ async def process_ai_analysis( provider: str | None = Query(default=None, pattern="^(codex|remote|local)$"), ) -> dict: try: - return await run_in_threadpool(task_service.process_task_ai_analysis, task_id, provider=provider) + job, created = task_service.queue_task_ai_analysis(task_id, provider=provider) + return { + "status": job["status"], + "message": "AI 分析任务已加入持久化队列" if created else "已有 AI 分析任务正在排队或运行", + "job_id": job["id"], + "job": job, + } + except task_service.AIAnalysisConflictError as exc: + raise HTTPException(status_code=409, detail=str(exc)) from exc except ValueError as exc: raise HTTPException(status_code=400, detail=str(exc)) from exc @@ -289,6 +313,8 @@ async def get_ai_analysis_runs(task_id: str) -> dict: async def restore_ai_analysis_run(task_id: str, run_id: str) -> dict: try: return task_service.restore_ai_analysis_run(task_id, run_id) + except task_service.AIAnalysisConflictError as exc: + raise HTTPException(status_code=409, detail=str(exc)) from exc except ValueError as exc: raise HTTPException(status_code=400, detail=str(exc)) from exc @@ -416,7 +442,14 @@ async def resume_auto_pipeline(task_id: str, background_tasks: BackgroundTasks) @router.post("/{task_id}/output-clips/{output_clip_id}/subtitles") async def render_output_clip_subtitles(task_id: str, output_clip_id: str) -> dict: try: - return task_service.render_subtitles_for_output_clip(task_id, output_clip_id) + from app.services.subtitle_auto_workflow_service import enqueue_task_subtitle_render + + return enqueue_task_subtitle_render( + task_id, + output_clip_ids=[output_clip_id], + approve_active_revisions=False, + continue_pipeline=False, + ) except ValueError as exc: raise HTTPException(status_code=400, detail=str(exc)) from exc except RuntimeError as exc: @@ -436,7 +469,6 @@ async def save_subtitle_style(payload: SubtitleStyleUpdate) -> dict: @router.post("/{task_id}/process/cuts-async") async def process_video_cuts_async( task_id: str, - background_tasks: BackgroundTasks, ) -> dict: """自动切片异步版:创建 job 后立即返回,后台执行切割""" # 先验证任务存在 @@ -453,9 +485,6 @@ async def process_video_cuts_async( except Exception as exc: raise HTTPException(status_code=500, detail=f"创建切片任务失败:{exc}") from exc - if created: - background_tasks.add_task(_run_job_in_background, job["id"]) - return { "status": job["status"], "message": ( @@ -478,6 +507,22 @@ async def get_job_status(job_id: str) -> dict: return job +@router.post("/jobs/{job_id}/cancel") +async def cancel_job(job_id: str) -> dict: + job = job_service.request_job_cancel(job_id) + if not job: + raise HTTPException(status_code=404, detail="job 不存在") + return {"status": job["status"], "message": job.get("message"), "job": job} + + +@router.post("/jobs/{job_id}/retry") +async def retry_job(job_id: str) -> dict: + job = job_service.retry_job(job_id) + if not job: + raise HTTPException(status_code=409, detail="只有失败或已取消的 job 可以重试") + return {"status": job["status"], "message": job.get("message"), "job": job} + + @router.get("/{task_id}/jobs") async def list_task_jobs(task_id: str, status: str | None = Query(default=None)) -> dict: """查看某个任务下的所有 job 记录""" @@ -490,12 +535,3 @@ async def list_task_jobs(task_id: str, status: str | None = Query(default=None)) "jobs": jobs, "count": len(jobs), } - - -def _run_job_in_background(job_id: str) -> None: - """后台线程执行 job""" - try: - job_worker.execute_job(job_id) - except Exception: - # 错误已由 worker 记录到 job 的 error_message 字段 - pass diff --git a/app/services/ai/ai_clip_analyzer.py b/app/services/ai/ai_clip_analyzer.py index 1e37621..1e25126 100644 --- a/app/services/ai/ai_clip_analyzer.py +++ b/app/services/ai/ai_clip_analyzer.py @@ -2,6 +2,7 @@ import ast from dataclasses import dataclass +import hashlib import math import json import re @@ -12,10 +13,20 @@ from app.core.config import settings from app.models.task import AIClipAnalysisResult -from app.services.ai.base import AIProvider, AIProviderError, ProviderConfig +from app.services.ai.base import ( + AIProvider, + AIProviderError, + ProviderConfig, + generate_json_with_safe_retry, +) from app.services.ai.codex_cli_provider import CodexCliConfig, CodexCliProvider from app.services.ai.local_model_provider import LocalModelProvider from app.services.ai.remote_responses_provider import RemoteResponsesProvider +from app.services.ai.unit_checkpoint import ( + build_unit_fingerprint, + execute_checkpointed_ai_unit, + provider_fingerprint_fields, +) PROMPT_PATH = settings.project_root / "prompts" / "clip_analysis_prompt.txt" @@ -98,7 +109,20 @@ def _analyze_task_transcript_in_chunks( raise AIAnalysisError("AI 分段分析失败:没有从转写文本中解析到可分析的时间戳正文") clips = [] - failures: list[str] = [] + failures: list[dict[str, Any]] = [] + empty_units = 0 + unit_fingerprint = build_unit_fingerprint( + { + "profile": "general", + "transcript_sha256": hashlib.sha256(transcript_text.encode("utf-8")).hexdigest(), + "provider": request.provider_name, + "provider_identity": provider_fingerprint_fields(provider), + "prompt_template": request.prompt_template or "", + "ai_preference": request.ai_preference, + "target_clip_count": request.target_clip_count, + "max_clip_duration_minutes": request.max_clip_duration_minutes, + } + ) per_chunk_target = max(1, min(3, math.ceil(request.target_clip_count / len(chunks)) + 1)) for chunk in chunks: prompt = _render_prompt( @@ -108,36 +132,53 @@ def _analyze_task_transcript_in_chunks( transcript_text=chunk.text, prompt_template=request.prompt_template, ) + def analyze_chunk() -> dict[str, Any]: + raw_text = generate_json_with_safe_retry(provider, prompt) + chunk_result = _parse_and_validate(raw_text, task_id=request.task_id) + _validate_clip_constraints(chunk_result, request, transcript_bounds) + return result_to_jsonable(chunk_result) + + execution = execute_checkpointed_ai_unit( + task_id=request.task_id or "", + namespace="general_chunks", + input_fingerprint=unit_fingerprint, + unit_id=f"chunk_{chunk.index:03d}", + operation=analyze_chunk, + ) + if execution.status != "completed" or not isinstance(execution.payload, dict): + failures.append( + { + "stage": "chunk", + "unit_id": f"chunk_{chunk.index:03d}", + "start_time": _seconds_to_time(chunk.start_seconds), + "end_time": _seconds_to_time(chunk.end_seconds), + "outcome": execution.status, + "message": " ".join(str(execution.error or "AI 分段失败").split())[:500], + } + ) + continue try: - raw_text = provider.generate_json(prompt) - try: - chunk_result = _parse_and_validate(raw_text, task_id=request.task_id) - except AIAnalysisError as first_error: - retry_instruction = ( - "上一次输出无法被程序解析或校验。请重新输出严格 JSON," - "不要 Markdown,不要解释文字。每个 clips 项必须包含:" - "clip_id、title、start_time、end_time、duration_seconds、cover_time_seconds、summary、" - "highlight_reason、spread_value、suggested_editing、confidence_score、selected_by_default。" - "cover_time_seconds 是相对于短视频开头的秒数,必须大于或等于 0 且小于 duration_seconds。" - "spread_value 只能是“高”“中”“低”。片段时长不能超限。" - ) - raw_text = provider.generate_json(prompt, retry_instruction=retry_instruction) - try: - chunk_result = _parse_and_validate(raw_text, task_id=request.task_id) - except AIAnalysisError as second_error: - raise AIAnalysisError(f"AI 返回非法 JSON,安全重试后仍失败:{second_error}") from first_error + chunk_result = AIClipAnalysisResult(**execution.payload) _validate_clip_constraints(chunk_result, request, transcript_bounds) - clips.extend(chunk_result.clips) except Exception as exc: failures.append( - f"第 {chunk.index}/{chunk.total} 段失败," - f"时间范围 {_seconds_to_time(chunk.start_seconds)}-{_seconds_to_time(chunk.end_seconds)}," - f"prompt 约 {len(prompt)} 字:{exc}" + { + "stage": "chunk", + "unit_id": f"chunk_{chunk.index:03d}", + "start_time": _seconds_to_time(chunk.start_seconds), + "end_time": _seconds_to_time(chunk.end_seconds), + "outcome": "invalid_checkpoint", + "message": " ".join(str(exc or "AI checkpoint 结构无效").split())[:500], + } ) + continue + if not chunk_result.clips: + empty_units += 1 + clips.extend(chunk_result.clips) merged_clips = _dedupe_and_rank_clips(clips, request.target_clip_count) if not merged_clips: - failure_text = ";".join(failures[:5]) if failures else "没有候选片段" + failure_text = ";".join(item["message"] for item in failures[:5]) if failures else "没有候选片段" raise AIAnalysisError(f"AI 分段分析没有生成可用候选片段:{failure_text}") for index, clip in enumerate(merged_clips, start=1): @@ -153,7 +194,28 @@ def _analyze_task_transcript_in_chunks( summary = f"{provider_label} 已按 {len(chunks)} 个小段完成分段分析,并合并为 {len(merged_clips)} 条候选片段。" if failures: summary += f" 有 {len(failures)} 个小段失败,已跳过失败小段。" - result = AIClipAnalysisResult(task_id=request.task_id, analysis_summary=summary, clips=merged_clips) + expected_units = len(chunks) + completed_units = expected_units - len(failures) + coverage_ratio = completed_units / expected_units if expected_units else 0.0 + result = AIClipAnalysisResult( + task_id=request.task_id, + analysis_summary=summary, + clips=merged_clips, + analysis_meta={ + "schema_version": 2, + "coverage_basis": "chunk_count", + "expected_units": expected_units, + "completed_units": completed_units, + "failed_units": len(failures), + "empty_unit_count": empty_units, + "invalid_item_count": 0, + "coverage_ratio": round(coverage_ratio, 6), + "coverage_percent": round(coverage_ratio * 100, 2), + "analysis_incomplete": bool(failures), + "quality_degraded": False, + "failed_stages": failures, + }, + ) _validate_clip_constraints(result, request, transcript_bounds) return result @@ -624,6 +686,11 @@ def _loads_ai_json(raw_text: str) -> Any: raise AIAnalysisError("JSON 解析失败:AI 没有返回可识别的 JSON 内容") +def loads_ai_json(raw_text: str) -> Any: + """供不同 AI 工作流复用同一套容错 JSON 解析。""" + return _loads_ai_json(raw_text) + + def _iter_json_candidates(raw_text: str) -> list[str]: cleaned = (raw_text or "").strip().lstrip("\ufeff") candidates: list[str] = [] @@ -886,5 +953,6 @@ def result_to_jsonable(result: AIClipAnalysisResult) -> dict[str, Any]: "build_provider", "build_remote_provider", "inspect_local_analysis_plan", + "loads_ai_json", "result_to_jsonable", ] diff --git a/app/services/ai/base.py b/app/services/ai/base.py index 811fe16..428f022 100644 --- a/app/services/ai/base.py +++ b/app/services/ai/base.py @@ -3,6 +3,8 @@ from dataclasses import dataclass import json import socket +import time +from collections.abc import Callable from typing import Any, Protocol from urllib.error import HTTPError, URLError from urllib.parse import urlparse @@ -10,7 +12,32 @@ class AIProviderError(RuntimeError): - pass + """携带重试与计费语义的 AI Provider 错误。 + + `safe_to_retry` 只在能够确认请求未被模型正常接受时为真。调用方不得根据 + “网络错误”或“5xx”几个字自行重试,因为这两类错误可能发生在模型已经执行之后。 + """ + + def __init__( + self, + message: str, + *, + category: str = "provider_error", + http_status: int | None = None, + safe_to_retry: bool = False, + billing_uncertain: bool = False, + retry_after_seconds: float | None = None, + ) -> None: + super().__init__(message) + self.category = category + self.http_status = http_status + self.safe_to_retry = safe_to_retry + self.billing_uncertain = billing_uncertain + self.retry_after_seconds = retry_after_seconds + + def checkpoint_message(self) -> str: + suffix = ";本次是否计费不确定,未自动重试" if self.billing_uncertain else "" + return f"[{self.category}] {self}{suffix}" class AIProvider(Protocol): @@ -34,6 +61,29 @@ class ProviderConfig: api_key_name: str = "远程接口 API Key" +def generate_json_with_safe_retry( + provider: AIProvider, + prompt: str, + retry_instruction: str | None = None, + *, + max_attempts: int = 3, + sleep_fn: Callable[[float], None] = time.sleep, +) -> str: + """只重试 Provider 明确标记为未执行/被拒绝的调用。""" + attempts = max(1, min(int(max_attempts), 3)) + for attempt in range(1, attempts + 1): + try: + if retry_instruction is None: + return provider.generate_json(prompt) + return provider.generate_json(prompt, retry_instruction=retry_instruction) + except AIProviderError as exc: + if not exc.safe_to_retry or attempt >= attempts: + raise + delay = exc.retry_after_seconds if exc.retry_after_seconds is not None else 2 ** (attempt - 1) + sleep_fn(max(0.0, min(delay, 60.0))) + raise AssertionError("AI Provider 安全重试循环异常结束") + + def build_url(base_url: str, path: str) -> str: normalized_base = base_url.rstrip("/") normalized_path = path.strip("/") @@ -68,19 +118,79 @@ def post_json(url: str, payload: dict[str, Any], api_key: str, timeout_seconds: with urlopen(request, timeout=timeout_seconds) as response: body = response.read().decode("utf-8", errors="replace") except HTTPError as exc: - detail = exc.read().decode("utf-8", errors="replace") + # Provider 错误正文可能回显 Authorization、Prompt、内部地址或账号诊断。 + # 读取并丢弃以释放连接,但只向任务日志暴露状态类别。 + exc.read() if exc.code in {401, 403}: - detail = f"{detail}。请检查远程 AI Key 是否缺失、无效,或中转站账号是否有当前模型权限。" - raise AIProviderError(f"AI 接口返回 HTTP {exc.code}:{detail}") from exc + detail = "请检查远程 AI Key 是否缺失、无效,或中转站账号是否有当前模型权限" + elif exc.code == 429: + detail = "请求过于频繁,Provider 已限流" + elif exc.code >= 500: + detail = "Provider 服务暂时异常" + else: + detail = "Provider 拒绝了本次请求" + retry_after = _parse_retry_after((exc.headers or {}).get("Retry-After")) if exc.code == 429 else None + raise AIProviderError( + f"AI 接口返回 HTTP {exc.code}:{detail}", + category="rate_limited" if exc.code == 429 else "http_error", + http_status=exc.code, + safe_to_retry=exc.code == 429, + billing_uncertain=exc.code >= 500 or exc.code == 408, + retry_after_seconds=retry_after, + ) from exc except URLError as exc: - raise AIProviderError(f"无法连接 AI 接口:{exc.reason}") from exc + reason = exc.reason + is_timeout = isinstance(reason, (TimeoutError, socket.timeout)) + is_preconnect_failure = isinstance(reason, (ConnectionRefusedError, socket.gaierror)) + raise AIProviderError( + "AI 接口连接超时" if is_timeout else "无法连接 AI 接口", + category="timeout" if is_timeout else "network_error", + safe_to_retry=is_preconnect_failure, + billing_uncertain=not is_preconnect_failure, + ) from exc except (TimeoutError, socket.timeout) as exc: - raise AIProviderError("AI 接口连接超时") from exc + raise AIProviderError( + "AI 接口连接或读取超时", + category="timeout", + billing_uncertain=True, + ) from exc + except OSError as exc: + raise AIProviderError( + "AI 接口请求失败", + category="network_error", + billing_uncertain=True, + ) from exc + + if not body.strip(): + raise AIProviderError( + "AI 接口返回空响应", + category="empty_response", + billing_uncertain=True, + ) try: - return json.loads(body) + parsed = json.loads(body) except json.JSONDecodeError as exc: - raise AIProviderError(f"AI 接口响应不是 JSON:{body[:300]}") from exc + raise AIProviderError( + "AI 接口响应不是有效 JSON", + category="invalid_response_json", + billing_uncertain=True, + ) from exc + if not isinstance(parsed, dict): + raise AIProviderError( + "AI 接口响应 JSON 顶层不是对象", + category="invalid_response_schema", + billing_uncertain=True, + ) + return parsed + + +def _parse_retry_after(value: str | None) -> float | None: + try: + seconds = float(str(value or "").strip()) + except ValueError: + return None + return max(0.0, min(seconds, 60.0)) def extract_responses_text(response: dict[str, Any]) -> str: @@ -90,21 +200,45 @@ def extract_responses_text(response: dict[str, Any]) -> str: chunks: list[str] = [] for output_item in response.get("output", []) or []: + if not isinstance(output_item, dict): + continue for content_item in output_item.get("content", []) or []: + if not isinstance(content_item, dict): + continue text = content_item.get("text") if isinstance(text, str): chunks.append(text) if chunks: return "\n".join(chunks).strip() - raise AIProviderError("AI Responses 响应中没有找到文本内容") + raise AIProviderError( + "AI Responses 响应中没有找到文本内容", + category="empty_model_output", + billing_uncertain=True, + ) def extract_chat_completion_text(response: dict[str, Any]) -> str: choices = response.get("choices") or [] if not choices: - raise AIProviderError("AI Chat Completions 响应中没有 choices") + raise AIProviderError( + "AI Chat Completions 响应中没有 choices", + category="empty_model_output", + billing_uncertain=True, + ) first_choice = choices[0] + if not isinstance(first_choice, dict): + raise AIProviderError( + "AI Chat Completions choices 条目格式无效", + category="invalid_response_schema", + billing_uncertain=True, + ) message = first_choice.get("message") or {} + if not isinstance(message, dict): + raise AIProviderError( + "AI Chat Completions message 结构无效", + category="invalid_response_schema", + billing_uncertain=True, + ) content = message.get("content") if isinstance(content, str) and content.strip(): return content.strip() @@ -115,5 +249,7 @@ def extract_chat_completion_text(response: dict[str, Any]) -> str: reasoning_chars = len(reasoning_content) if isinstance(reasoning_content, str) else 0 raise AIProviderError( "AI Chat Completions 响应中没有文本内容" - f"(finish_reason={finish_reason},message_keys={message_keys},reasoning_chars={reasoning_chars})" + f"(finish_reason={finish_reason},message_keys={message_keys},reasoning_chars={reasoning_chars})", + category="empty_model_output", + billing_uncertain=True, ) diff --git a/app/services/ai/codex_cli_provider.py b/app/services/ai/codex_cli_provider.py index 1624a7a..cbb4a9b 100644 --- a/app/services/ai/codex_cli_provider.py +++ b/app/services/ai/codex_cli_provider.py @@ -75,24 +75,52 @@ def generate_json(self, prompt: str, retry_instruction: str | None = None) -> st check=False, ) if completed.returncode != 0: - raise AIProviderError(f"Codex CLI 执行失败(退出码 {completed.returncode})") + raise AIProviderError( + f"Codex CLI 执行失败(退出码 {completed.returncode})", + category="cli_exit_error", + billing_uncertain=True, + ) if not output_path.is_file(): - raise AIProviderError("Codex CLI 未生成最终结果文件") + raise AIProviderError( + "Codex CLI 未生成最终结果文件", + category="empty_model_output", + billing_uncertain=True, + ) result = _strip_json_fence(output_path.read_text(encoding="utf-8").strip()) if not result: - raise AIProviderError("Codex CLI 返回空结果") + raise AIProviderError( + "Codex CLI 返回空结果", + category="empty_model_output", + billing_uncertain=True, + ) try: parsed = json.loads(result) except json.JSONDecodeError as exc: - raise AIProviderError("Codex CLI 返回内容不是合法 JSON") from exc + raise AIProviderError( + "Codex CLI 返回内容不是合法 JSON", + category="invalid_response_json", + billing_uncertain=True, + ) from exc if not isinstance(parsed, (dict, list)): - raise AIProviderError("Codex CLI JSON 顶层必须是对象或数组") + raise AIProviderError( + "Codex CLI JSON 顶层必须是对象或数组", + category="invalid_response_schema", + billing_uncertain=True, + ) return result except subprocess.TimeoutExpired as exc: - raise AIProviderError(f"Codex CLI 执行超时(>{self.config.timeout_seconds} 秒)") from exc + raise AIProviderError( + f"Codex CLI 执行超时(>{self.config.timeout_seconds} 秒)", + category="timeout", + billing_uncertain=True, + ) from exc except OSError as exc: - raise AIProviderError("Codex CLI 无法启动,请检查可执行文件路径") from exc + raise AIProviderError( + "Codex CLI 无法启动,请检查可执行文件路径", + category="cli_start_error", + safe_to_retry=False, + ) from exc def version_status(self) -> dict[str, str | bool]: executable = self._resolve_executable() diff --git a/app/services/ai/local_model_provider.py b/app/services/ai/local_model_provider.py index 7fca8a1..032bc89 100644 --- a/app/services/ai/local_model_provider.py +++ b/app/services/ai/local_model_provider.py @@ -31,6 +31,8 @@ def generate_json(self, prompt: str, retry_instruction: str | None = None) -> st raise AIProviderError(f"暂不支持本地 AI 协议:{protocol}") except AIProviderError as exc: last_error = exc + if exc.http_status not in {404, 405}: + break raise last_error or AIProviderError("本地 AI 调用失败") diff --git a/app/services/ai/long_live_talk_analyzer.py b/app/services/ai/long_live_talk_analyzer.py new file mode 100644 index 0000000..50216e2 --- /dev/null +++ b/app/services/ai/long_live_talk_analyzer.py @@ -0,0 +1,911 @@ +"""语言类长直播的可恢复分层高光选片。""" + +from __future__ import annotations + +from collections import defaultdict +from dataclasses import dataclass +from datetime import datetime, timedelta +import hashlib +import json +import math +from pathlib import Path +import re +import time +from typing import Any, Callable +from uuid import uuid4 + +from app.db.database import get_connection +from app.models.task import AIClipAnalysisResult +from app.services import job_service +from app.services.ai.ai_clip_analyzer import ( + AIAnalysisError, + TranscriptRow, + _extract_transcript_rows, + _loads_ai_json, + _read_transcript, + _seconds_to_time, + _time_to_seconds, + build_provider, +) +from app.services.ai.base import AIProvider, AIProviderError +from app.services.ai.unit_checkpoint import provider_fingerprint_fields + + +WINDOW_SECONDS = 300 +WINDOW_OVERLAP_SECONDS = 60 +WINDOW_CHAR_BUDGET = 12_000 +WINDOW_RECALL_LIMIT = 5 +MIN_COMPLETE_COVERAGE = 0.90 +ALLOWED_CATEGORIES = ( + "quote_opinion", + "story_experience", + "emotional_peak", + "conflict_reversal", + "practical_knowledge", + "interactive_humor", +) +CATEGORY_LABELS = { + "quote_opinion": "金句观点", + "story_experience": "故事经历", + "emotional_peak": "情绪峰值", + "conflict_reversal": "冲突反转", + "practical_knowledge": "实用知识", + "interactive_humor": "互动幽默", +} + + +@dataclass(frozen=True) +class LongLiveAnalysisRequest: + task_id: str + transcript_path: Path + provider_name: str + model_name: str + density_per_hour: int = 4 + total_limit: int = 30 + ai_preference: str = "" + prompt_template: str | None = None + + +@dataclass(frozen=True) +class LongLiveWindow: + index: int + total: int + start_seconds: int + end_seconds: int + rows: tuple[TranscriptRow, ...] + text: str + + +@dataclass(frozen=True) +class LongLiveAnalysisOutcome: + result: AIClipAnalysisResult + meta: dict[str, Any] + + +def analyze_long_live_talk( + request: LongLiveAnalysisRequest, + *, + provider: AIProvider | None = None, + sleep_fn: Callable[[float], None] = time.sleep, + progress_callback: Callable[[dict[str, Any]], None] | None = None, +) -> LongLiveAnalysisOutcome: + transcript_text = _read_transcript(request.transcript_path) + rows = _extract_transcript_rows(transcript_text) + if not rows: + raise AIAnalysisError("长直播分析失败:转写中没有可识别的逐句时间戳") + + windows = build_long_live_windows(rows) + if not windows: + raise AIAnalysisError("长直播分析失败:没有生成可分析窗口") + + transcript_fingerprint = hashlib.sha256(transcript_text.encode("utf-8")).hexdigest() + provider = provider or build_provider(request.provider_name) + preference = _preference_summary(request.prompt_template or "", request.ai_preference) + checkpoint_fingerprint = hashlib.sha256( + json.dumps( + { + "transcript_sha256": transcript_fingerprint, + "provider": request.provider_name, + "model": request.model_name, + "provider_identity": provider_fingerprint_fields(provider), + "preference": preference, + "density_per_hour": request.density_per_hour, + "total_limit": request.total_limit, + }, + ensure_ascii=False, + sort_keys=True, + ).encode("utf-8") + ).hexdigest() + successful_payloads: list[dict[str, Any]] = [] + completed_windows: list[LongLiveWindow] = [] + failed_windows: list[dict[str, Any]] = [] + invalid_item_count = 0 + reused_count = 0 + + for window in windows: + checkpoint = _get_or_create_checkpoint(request, checkpoint_fingerprint, window) + if checkpoint.get("status") == "completed": + payload = _load_verified_checkpoint_payload(checkpoint, window=window) + if isinstance(payload, dict): + successful_payloads.append(payload) + completed_windows.append(window) + invalid_item_count += int(payload.get("invalid_item_count") or 0) + reused_count += 1 + _report_progress(progress_callback, window, "reused", len(completed_windows)) + continue + error = "已完成的长直播窗口 checkpoint 校验失败,未自动重复请求" + _mark_checkpoint_failed(checkpoint["id"], error, 0, uncertain=True) + failed_windows.append( + { + "window_index": window.index, + "start_seconds": window.start_seconds, + "end_seconds": window.end_seconds, + "error": error, + "outcome": "uncertain", + } + ) + _report_progress(progress_callback, window, "failed", len(completed_windows)) + continue + if checkpoint.get("status") in {"running", "uncertain"}: + error = str( + checkpoint.get("error_message") + or "上一次窗口请求已开始但结果未确认,未自动重复请求" + ) + if checkpoint.get("status") == "running": + _mark_checkpoint_failed(checkpoint["id"], error, 0, uncertain=True) + failed_windows.append( + { + "window_index": window.index, + "start_seconds": window.start_seconds, + "end_seconds": window.end_seconds, + "error": error, + "outcome": "uncertain", + } + ) + _report_progress(progress_callback, window, "failed", len(completed_windows)) + continue + + max_attempts = 3 if request.provider_name == "remote" else 1 + last_error = "" + payload = None + for attempt in range(1, max_attempts + 1): + _mark_checkpoint_running(checkpoint["id"]) + try: + raw = provider.generate_json(_window_prompt(window, preference)) + payload = _parse_window_payload(raw, window) + if not payload.get("moments"): + payload = {"moments": []} + _mark_checkpoint_completed(checkpoint["id"], payload) + successful_payloads.append(payload) + completed_windows.append(window) + invalid_item_count += int(payload.get("invalid_item_count") or 0) + _report_progress(progress_callback, window, "completed", len(completed_windows)) + break + except Exception as exc: # 每个窗口必须独立记录,不能丢失前面成功结果 + provider_error = exc if isinstance(exc, AIProviderError) else None + error_text = provider_error.checkpoint_message() if provider_error else str(exc) + last_error = " ".join(error_text.split())[:1000] or "未知错误" + should_retry = bool( + provider_error + and provider_error.safe_to_retry + and attempt < max_attempts + ) + configured_delay = provider_error.retry_after_seconds if provider_error else None + delay_seconds = configured_delay if configured_delay is not None else 2 ** (attempt - 1) + delay_seconds = delay_seconds if should_retry else 0 + uncertain = not bool( + provider_error + and provider_error.safe_to_retry + and not provider_error.billing_uncertain + ) + _mark_checkpoint_failed( + checkpoint["id"], + last_error, + delay_seconds, + uncertain=uncertain and not should_retry, + ) + if should_retry: + sleep_fn(delay_seconds) + else: + break + if payload is None: + failed_windows.append( + { + "window_index": window.index, + "start_seconds": window.start_seconds, + "end_seconds": window.end_seconds, + "error": last_error, + } + ) + _report_progress(progress_callback, window, "failed", len(completed_windows)) + + if not completed_windows: + detail = ";".join(item["error"] for item in failed_windows[:3]) or "全部窗口均失败" + raise AIAnalysisError(f"长直播分析没有完成任何窗口:{detail}") + + moments: list[dict[str, Any]] = [] + for payload in successful_payloads: + moments.extend(payload.get("moments") or []) + deduplicated = deduplicate_long_live_moments(moments) + density = max(1, min(10, int(request.density_per_hour or 4))) + total_limit = max(1, min(50, int(request.total_limit or 30))) + selected = select_temporally_balanced_highlights(deduplicated, density, total_limit) + clips = [_moment_to_clip(moment, index) for index, moment in enumerate(selected, start=1)] + + transcript_start = rows[0].start_seconds + transcript_end = rows[-1].end_seconds + coverage_ratio = calculate_window_coverage( + [(window.start_seconds, window.end_seconds) for window in completed_windows], + transcript_start, + transcript_end, + ) + incomplete = coverage_ratio < MIN_COMPLETE_COVERAGE or bool(invalid_item_count or failed_windows) + coverage_percent = round(coverage_ratio * 100, 2) + summary = ( + f"长直播高光已完成 {len(completed_windows)}/{len(windows)} 个重叠窗口," + f"时间轴覆盖 {coverage_percent:.2f}%,去重后保留 {len(clips)} 条候选。" + ) + if incomplete: + summary += " 当前分析不完整,必须补齐失败窗口后才能进入自动切片。" + + meta = { + "transcript_fingerprint": transcript_fingerprint, + "window_seconds": WINDOW_SECONDS, + "window_overlap_seconds": WINDOW_OVERLAP_SECONDS, + "window_count": len(windows), + "completed_window_count": len(completed_windows), + "failed_window_count": len(failed_windows), + "failed_windows": failed_windows, + "invalid_item_count": invalid_item_count, + "reused_window_count": reused_count, + "coverage_ratio": round(coverage_ratio, 6), + "coverage_percent": coverage_percent, + "analysis_incomplete": incomplete, + "minimum_complete_coverage": MIN_COMPLETE_COVERAGE, + "highlight_density_per_hour": density, + "highlight_total_limit": total_limit, + "deduplicated_moment_count": len(deduplicated), + "selected_highlight_count": len(clips), + } + return LongLiveAnalysisOutcome( + result=AIClipAnalysisResult(task_id=request.task_id, analysis_summary=summary, clips=clips), + meta=meta, + ) + + +def build_long_live_windows(rows: list[TranscriptRow]) -> list[LongLiveWindow]: + """按约 5 分钟、60 秒重叠构造窗口,并兼顾 prompt 字符预算。""" + if not rows: + return [] + raw_windows: list[tuple[TranscriptRow, ...]] = [] + start_index = 0 + while start_index < len(rows): + start_seconds = rows[start_index].start_seconds + current: list[TranscriptRow] = [] + current_chars = 0 + end_index = start_index + while end_index < len(rows): + row = rows[end_index] + line = _format_row(row) + exceeds_time = bool(current) and row.end_seconds - start_seconds > WINDOW_SECONDS + exceeds_chars = bool(current) and current_chars + len(line) + 1 > WINDOW_CHAR_BUDGET + if exceeds_time or exceeds_chars: + break + current.append(row) + current_chars += len(line) + 1 + end_index += 1 + if not current: + current = [rows[start_index]] + end_index = start_index + 1 + raw_windows.append(tuple(current)) + if end_index >= len(rows): + break + next_time = max(current[0].start_seconds + 1, current[-1].end_seconds - WINDOW_OVERLAP_SECONDS) + next_index = start_index + 1 + while next_index < end_index and rows[next_index].start_seconds < next_time: + next_index += 1 + start_index = max(start_index + 1, next_index) + + total = len(raw_windows) + return [ + LongLiveWindow( + index=index, + total=total, + start_seconds=window_rows[0].start_seconds, + end_seconds=window_rows[-1].end_seconds, + rows=window_rows, + text="\n".join(_format_row(row) for row in window_rows), + ) + for index, window_rows in enumerate(raw_windows, start=1) + ] + + +def calculate_window_coverage( + intervals: list[tuple[int, int]], + timeline_start: int, + timeline_end: int, +) -> float: + if timeline_end <= timeline_start: + return 1.0 if intervals else 0.0 + clipped = sorted( + (max(timeline_start, start), min(timeline_end, end)) + for start, end in intervals + if end > timeline_start and start < timeline_end and end > start + ) + if not clipped: + return 0.0 + merged: list[list[int]] = [] + for start, end in clipped: + if not merged or start > merged[-1][1]: + merged.append([start, end]) + else: + merged[-1][1] = max(merged[-1][1], end) + covered = sum(end - start for start, end in merged) + return min(1.0, covered / (timeline_end - timeline_start)) + + +def deduplicate_long_live_moments(moments: list[dict[str, Any]]) -> list[dict[str, Any]]: + ranked = sorted(moments, key=lambda item: float(item.get("score") or 0), reverse=True) + selected: list[dict[str, Any]] = [] + for raw in ranked: + moment = _normalize_moment(raw) + if not moment: + continue + duplicate_index = None + for index, existing in enumerate(selected): + if _moments_are_duplicate(moment, existing): + duplicate_index = index + break + if duplicate_index is None: + selected.append(moment) + else: + selected[duplicate_index] = _merge_moments(selected[duplicate_index], moment) + return sorted(selected, key=lambda item: int(item["start_seconds"])) + + +def select_temporally_balanced_highlights( + moments: list[dict[str, Any]], + density_per_hour: int, + total_limit: int, +) -> list[dict[str, Any]]: + density = max(1, min(10, int(density_per_hour or 4))) + limit = max(1, min(50, int(total_limit or 30))) + buckets: dict[int, list[dict[str, Any]]] = defaultdict(list) + for moment in moments: + midpoint = (int(moment["start_seconds"]) + int(moment["end_seconds"])) / 2 + buckets[int(midpoint // 3600)].append(moment) + for bucket in buckets.values(): + bucket.sort(key=lambda item: float(item.get("score") or 0), reverse=True) + del bucket[density:] + + hours = sorted(buckets) + if len(hours) > limit: + indexes = _evenly_spaced_indexes(len(hours), limit) + hours = [hours[index] for index in indexes] + + selected: list[dict[str, Any]] = [] + for rank in range(density): + for hour in hours: + if rank < len(buckets[hour]): + selected.append(buckets[hour][rank]) + if len(selected) >= limit: + return sorted(selected, key=lambda item: int(item["start_seconds"])) + return sorted(selected, key=lambda item: int(item["start_seconds"])) + + +def list_long_live_window_checkpoints(task_id: str) -> list[dict[str, Any]]: + with get_connection() as connection: + rows = connection.execute( + """ + SELECT * FROM ai_analysis_windows + WHERE task_id = ? + ORDER BY updated_at DESC, window_index ASC + """, + (task_id,), + ).fetchall() + return [dict(row) for row in rows] + + +def get_latest_long_live_window_status(task_id: str) -> dict[str, Any]: + """返回最新一组窗口的轻量统计,不把历史转写指纹混入当前进度。""" + with get_connection() as connection: + latest = connection.execute( + """ + SELECT transcript_fingerprint, provider, model + FROM ai_analysis_windows + WHERE task_id = ? + ORDER BY updated_at DESC + LIMIT 1 + """, + (task_id,), + ).fetchone() + if not latest: + return {} + rows = connection.execute( + """ + SELECT window_index, start_seconds, end_seconds, status, + attempt_count, error_message, updated_at + FROM ai_analysis_windows + WHERE task_id = ? AND transcript_fingerprint = ? AND provider = ? AND model = ? + ORDER BY window_index ASC + """, + (task_id, latest["transcript_fingerprint"], latest["provider"], latest["model"]), + ).fetchall() + items = [dict(row) for row in rows] + completed = sum(1 for item in items if item["status"] == "completed") + failed = [item for item in items if item["status"] in {"failed", "uncertain"}] + return { + "provider": latest["provider"], + "model": latest["model"], + "window_count": len(items), + "completed_window_count": completed, + "failed_window_count": len(failed), + "failed_windows": failed, + "percent": round(completed / len(items) * 100) if items else 0, + } + + +def _get_or_create_checkpoint( + request: LongLiveAnalysisRequest, + fingerprint: str, + window: LongLiveWindow, +) -> dict[str, Any]: + now = _now_iso() + checkpoint_id = uuid4().hex + with get_connection() as connection: + connection.execute("BEGIN IMMEDIATE") + _assert_current_job_lease(connection) + connection.execute( + """ + INSERT OR IGNORE INTO ai_analysis_windows ( + id, task_id, transcript_fingerprint, provider, model, + window_index, start_seconds, end_seconds, status, + attempt_count, created_at, updated_at + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, 'queued', 0, ?, ?) + """, + ( + checkpoint_id, + request.task_id, + fingerprint, + request.provider_name, + request.model_name, + window.index, + window.start_seconds, + window.end_seconds, + now, + now, + ), + ) + row = connection.execute( + """ + SELECT * FROM ai_analysis_windows + WHERE task_id = ? AND transcript_fingerprint = ? AND provider = ? AND model = ? + AND window_index = ? AND start_seconds = ? AND end_seconds = ? + """, + ( + request.task_id, + fingerprint, + request.provider_name, + request.model_name, + window.index, + window.start_seconds, + window.end_seconds, + ), + ).fetchone() + connection.commit() + if not row: + raise AIAnalysisError(f"无法创建长直播窗口 checkpoint:{window.index}") + return dict(row) + + +def _mark_checkpoint_running(checkpoint_id: str) -> None: + with get_connection() as connection: + connection.execute("BEGIN IMMEDIATE") + _assert_current_job_lease(connection) + connection.execute( + """ + UPDATE ai_analysis_windows + SET status = 'running', attempt_count = attempt_count + 1, + error_message = NULL, next_retry_at = NULL, updated_at = ? + WHERE id = ? + """, + (_now_iso(), checkpoint_id), + ) + connection.commit() + + +def _mark_checkpoint_completed(checkpoint_id: str, payload: dict[str, Any]) -> None: + now = _now_iso() + result_json = json.dumps(payload, ensure_ascii=False, sort_keys=True) + checksum = hashlib.sha256(result_json.encode("utf-8")).hexdigest() + with get_connection() as connection: + connection.execute("BEGIN IMMEDIATE") + _assert_current_job_lease(connection) + connection.execute( + """ + UPDATE ai_analysis_windows + SET status = 'completed', result_json = ?, result_checksum = ?, + error_message = NULL, next_retry_at = NULL, + updated_at = ?, completed_at = ? + WHERE id = ? + """, + (result_json, checksum, now, now, checkpoint_id), + ) + connection.commit() + + +def _mark_checkpoint_failed( + checkpoint_id: str, + error: str, + delay_seconds: float, + *, + uncertain: bool = False, +) -> None: + now = datetime.now().astimezone() + next_retry_at = (now + timedelta(seconds=delay_seconds)).isoformat(timespec="seconds") if delay_seconds else None + with get_connection() as connection: + connection.execute("BEGIN IMMEDIATE") + _assert_current_job_lease(connection) + connection.execute( + """ + UPDATE ai_analysis_windows + SET status = ?, error_message = ?, next_retry_at = ?, updated_at = ? + WHERE id = ? + """, + ( + "uncertain" if uncertain else "failed", + error, + next_retry_at, + now.isoformat(timespec="seconds"), + checkpoint_id, + ), + ) + connection.commit() + + +def _assert_current_job_lease(connection) -> None: + active = job_service.current_job_lease() + if active is None: + return + job_id, lease_owner, lease_token = active + row = connection.execute( + """ + SELECT 1 FROM workflow_jobs + WHERE id = ? AND status = 'running' AND lease_owner = ? AND lease_token = ? + AND lease_expires_at > ? AND cancel_requested = 0 + """, + (job_id, lease_owner, lease_token, _now_iso()), + ).fetchone() + if not row: + raise job_service.JobLeaseLostError(f"Workflow Job 租约已失效:{job_id}") + + +def _load_verified_checkpoint_payload( + checkpoint: dict[str, Any], + *, + window: LongLiveWindow | None = None, +) -> dict[str, Any] | None: + result_json = str(checkpoint.get("result_json") or "") + expected_checksum = str(checkpoint.get("result_checksum") or "") + if not result_json or not expected_checksum: + return None + actual_checksum = hashlib.sha256(result_json.encode("utf-8")).hexdigest() + if actual_checksum != expected_checksum: + return None + try: + payload = json.loads(result_json) + except json.JSONDecodeError: + return None + if not isinstance(payload, dict): + return None + moments = payload.get("moments") + if not isinstance(moments, list) or type(payload.get("invalid_item_count")) is not int: + return None + if payload["invalid_item_count"] != 0: + return None + required = { + "title", + "start_seconds", + "end_seconds", + "key_seconds", + "summary", + "highlight_reason", + "suggested_editing", + "category", + "topic_key", + "score", + "source_window_indexes", + } + for item in moments: + if not isinstance(item, dict) or not required.issubset(item): + return None + if any(type(item[name]) not in (int, float) for name in ("start_seconds", "end_seconds", "key_seconds", "score")): + return None + start_seconds = float(item["start_seconds"]) + end_seconds = float(item["end_seconds"]) + key_seconds = float(item["key_seconds"]) + score = float(item["score"]) + if not all(math.isfinite(value) for value in (start_seconds, end_seconds, key_seconds, score)): + return None + if end_seconds <= start_seconds or not (15 <= end_seconds - start_seconds <= 300): + return None + if not (start_seconds <= key_seconds <= end_seconds) or not (0 <= score <= 100): + return None + if window and (start_seconds < window.start_seconds - 5 or end_seconds > window.end_seconds + 5): + return None + if item.get("category") not in ALLOWED_CATEGORIES: + return None + if not all(isinstance(item.get(name), str) and item.get(name) for name in ( + "title", "summary", "highlight_reason", "suggested_editing", "topic_key" + )): + return None + indexes = item.get("source_window_indexes") + if not isinstance(indexes, list) or not all(type(index) is int for index in indexes): + return None + if window and window.index not in indexes: + return None + return payload + + +def _parse_window_payload(raw: str, window: LongLiveWindow) -> dict[str, Any]: + payload = _loads_ai_json(raw) + if isinstance(payload, list): + payload = {"moments": payload} + if not isinstance(payload, dict): + raise AIAnalysisError("长直播窗口输出必须是 JSON 对象") + moments = payload.get("moments") + if not isinstance(moments, list): + raise AIAnalysisError("长直播窗口输出缺少 moments 数组") + normalized: list[dict[str, Any]] = [] + invalid_item_count = 0 + for index, item in enumerate(moments): + if not isinstance(item, dict): + invalid_item_count += 1 + continue + moment = _normalize_moment(item, window=window) + if moment: + if index < WINDOW_RECALL_LIMIT: + normalized.append(moment) + else: + invalid_item_count += 1 + if invalid_item_count: + raise AIAnalysisError(f"长直播窗口 moments 含 {invalid_item_count} 条无效条目") + return {"moments": normalized, "invalid_item_count": invalid_item_count} + + +def _normalize_moment( + item: dict[str, Any], + *, + window: LongLiveWindow | None = None, +) -> dict[str, Any] | None: + try: + start_seconds = _coerce_seconds(item.get("start_seconds"), item.get("start_time")) + end_seconds = _coerce_seconds(item.get("end_seconds"), item.get("end_time")) + except (TypeError, ValueError): + return None + if end_seconds <= start_seconds: + return None + if window and (start_seconds < window.start_seconds - 5 or end_seconds > window.end_seconds + 5): + return None + duration = end_seconds - start_seconds + if duration < 15 or duration > 300: + return None + category = str(item.get("category") or "quote_opinion").strip().lower() + if category not in ALLOWED_CATEGORIES: + category = "quote_opinion" + score = _bounded_float(item.get("score") or item.get("confidence_score") or 70, 0, 100) + title = _clean_text(item.get("title"), "长直播高光", 160) + summary = _clean_text(item.get("summary"), title, 1000) + reason = _clean_text(item.get("highlight_reason") or item.get("reason"), CATEGORY_LABELS[category], 1000) + topic_key = _clean_text(item.get("topic_key"), title, 120) + key_seconds = _coerce_seconds(item.get("key_seconds"), item.get("key_time"), fallback=(start_seconds + end_seconds) // 2) + key_seconds = max(start_seconds, min(end_seconds, key_seconds)) + return { + "title": title, + "start_seconds": start_seconds, + "end_seconds": end_seconds, + "key_seconds": key_seconds, + "summary": summary, + "highlight_reason": reason, + "suggested_editing": _clean_text( + item.get("suggested_editing"), + "保留观点或故事闭环,剪掉明显停顿和重复表达。", + 1000, + ), + "category": category, + "topic_key": topic_key, + "score": score, + "source_window_indexes": [window.index] if window else list(item.get("source_window_indexes") or []), + } + + +def _moments_are_duplicate(first: dict[str, Any], second: dict[str, Any]) -> bool: + overlap = max( + 0, + min(int(first["end_seconds"]), int(second["end_seconds"])) + - max(int(first["start_seconds"]), int(second["start_seconds"])), + ) + shorter = max( + 1, + min( + int(first["end_seconds"]) - int(first["start_seconds"]), + int(second["end_seconds"]) - int(second["start_seconds"]), + ), + ) + if overlap / shorter >= 0.35: + return True + key_distance = abs(int(first["key_seconds"]) - int(second["key_seconds"])) + if key_distance <= 30: + return True + semantic = _semantic_similarity( + f"{first.get('topic_key', '')}{first.get('title', '')}{first.get('summary', '')}", + f"{second.get('topic_key', '')}{second.get('title', '')}{second.get('summary', '')}", + ) + return key_distance <= 120 and semantic >= 0.34 + + +def _merge_moments(first: dict[str, Any], second: dict[str, Any]) -> dict[str, Any]: + stronger, other = (first, second) if float(first["score"]) >= float(second["score"]) else (second, first) + merged = dict(stronger) + merged["start_seconds"] = min(int(first["start_seconds"]), int(second["start_seconds"])) + merged["end_seconds"] = max(int(first["end_seconds"]), int(second["end_seconds"])) + merged["source_window_indexes"] = sorted( + set(first.get("source_window_indexes") or []) | set(second.get("source_window_indexes") or []) + ) + if len(str(other.get("summary") or "")) > len(str(merged.get("summary") or "")): + merged["summary"] = other["summary"] + return merged + + +def _moment_to_clip(moment: dict[str, Any], index: int) -> dict[str, Any]: + start = int(moment["start_seconds"]) + end = int(moment["end_seconds"]) + duration = max(1, end - start) + score = float(moment.get("score") or 0) + quality_tier = "A" if score >= 80 else "B" if score >= 65 else "C" + return { + "clip_id": f"long_live_{index:03d}", + "title": moment["title"], + "start_time": _seconds_to_time(start), + "end_time": _seconds_to_time(end), + "duration_seconds": duration, + "cover_time_seconds": max(0.0, min(duration - 0.001, float(moment["key_seconds"] - start))), + "summary": moment["summary"], + "highlight_reason": f"{CATEGORY_LABELS[moment['category']]}:{moment['highlight_reason']}", + "spread_value": "高" if score >= 80 else "中", + "suggested_editing": moment["suggested_editing"], + "confidence_score": round(score / 100, 4), + "selected_by_default": True, + "quality_tier": quality_tier, + "quality_score": score, + "text_quality_score": score, + "humor_score": score if moment["category"] == "interactive_humor" else 0, + "completeness_score": score, + "audio_reaction_score": 0, + "topic_key": moment["topic_key"], + "key_moment_time": _seconds_to_time(int(moment["key_seconds"])), + "quality_evidence": { + "highlight_category": moment["category"], + "highlight_category_label": CATEGORY_LABELS[moment["category"]], + "source_window_indexes": moment.get("source_window_indexes") or [], + }, + "rejection_reason": "", + } + + +def _window_prompt(window: LongLiveWindow, preference: str) -> str: + return f"""你是语言内容型长直播的高光召回编辑。只分析当前约 5 分钟窗口,不评价整场直播。 +从下列六类中找出 0-{WINDOW_RECALL_LIMIT} 个可独立理解的高光: +1. quote_opinion 金句观点;2. story_experience 故事经历;3. emotional_peak 情绪峰值; +4. conflict_reversal 冲突反转;5. practical_knowledge 实用知识;6. interactive_humor 互动幽默。 +不要为凑数输出寒暄、重复表达、纯过场。片段建议 45-180 秒,必要时可为 15-300 秒。 +{preference} +只返回严格 JSON:{{"moments":[{{"title":"标题","category":"六类英文值之一","start_time":"HH:MM:SS","end_time":"HH:MM:SS","key_time":"HH:MM:SS","topic_key":"稳定话题标识","summary":"完整内容闭环","highlight_reason":"具体价值","suggested_editing":"剪辑建议","score":0}}]}} +score 为 0-100;时间必须来自窗口转写。没有高光就返回空数组。 + +窗口 {window.index}/{window.total},范围 {_seconds_to_time(window.start_seconds)}-{_seconds_to_time(window.end_seconds)}: +{window.text}""" + + +def _preference_summary(prompt_template: str, ai_preference: str) -> str: + prompt = (prompt_template or "").replace("{{AI_PREFERENCE}}", ai_preference or "") + for marker in ("# Output Format", "【输出格式】", "输出 JSON", "转写文本:", "# Transcript", "{{TRANSCRIPT_TEXT}}"): + if marker in prompt: + prompt = prompt.split(marker, 1)[0] + prompt = " ".join(prompt.split())[:2000] + extra = " ".join((ai_preference or "").split())[:500] + parts = [] + if prompt: + parts.append(f"沿用本任务内容偏好:{prompt}") + if extra and extra not in prompt: + parts.append(f"用户补充偏好:{extra}") + return "\n".join(parts) + + +def _semantic_similarity(first: str, second: str) -> float: + first_tokens = _semantic_tokens(first) + second_tokens = _semantic_tokens(second) + if not first_tokens or not second_tokens: + return 0.0 + return len(first_tokens & second_tokens) / len(first_tokens | second_tokens) + + +def _semantic_tokens(value: str) -> set[str]: + normalized = re.sub(r"\s+", "", str(value or "").lower()) + chinese = "".join(re.findall(r"[\u4e00-\u9fff]", normalized)) + tokens = {chinese[index : index + 2] for index in range(max(0, len(chinese) - 1))} + tokens.update(re.findall(r"[a-z0-9]{2,}", normalized)) + return tokens + + +def _coerce_seconds(value: Any, time_text: Any, fallback: int | None = None) -> int: + if value not in (None, ""): + return int(float(value)) + if time_text not in (None, ""): + return _time_to_seconds(str(time_text)) + if fallback is not None: + return fallback + raise ValueError("缺少时间") + + +def _bounded_float(value: Any, minimum: float, maximum: float) -> float: + try: + number = float(value) + except (TypeError, ValueError): + number = minimum + return round(max(minimum, min(maximum, number)), 2) + + +def _clean_text(value: Any, fallback: str, limit: int) -> str: + text = " ".join(str(value or "").split()) or fallback + return text[:limit] + + +def _format_row(row: TranscriptRow) -> str: + return f"{row.start_time} - {row.end_time} {row.text}" + + +def _evenly_spaced_indexes(length: int, count: int) -> list[int]: + if count >= length: + return list(range(length)) + if count <= 1: + return [length // 2] + return sorted({round(index * (length - 1) / (count - 1)) for index in range(count)}) + + +def _report_progress( + callback: Callable[[dict[str, Any]], None] | None, + window: LongLiveWindow, + status: str, + completed_count: int, +) -> None: + if callback: + callback( + { + "window_index": window.index, + "window_count": window.total, + "status": status, + "completed_count": completed_count, + "percent": min(99, math.floor(window.index / max(1, window.total) * 100)), + } + ) + + +def _now_iso() -> str: + return datetime.now().astimezone().isoformat(timespec="seconds") + + +__all__ = [ + "ALLOWED_CATEGORIES", + "LongLiveAnalysisOutcome", + "LongLiveAnalysisRequest", + "LongLiveWindow", + "MIN_COMPLETE_COVERAGE", + "analyze_long_live_talk", + "build_long_live_windows", + "calculate_window_coverage", + "deduplicate_long_live_moments", + "get_latest_long_live_window_status", + "list_long_live_window_checkpoints", + "select_temporally_balanced_highlights", +] diff --git a/app/services/ai/unit_checkpoint.py b/app/services/ai/unit_checkpoint.py new file mode 100644 index 0000000..0d4b560 --- /dev/null +++ b/app/services/ai/unit_checkpoint.py @@ -0,0 +1,341 @@ +"""Workflow Job 内的 AI 单元调用 checkpoint。 + +在调用 Provider 前先记录 ``running``,成功解析后再记录经过校验的结果。 +如果进程在两者之间退出,下一代 lease 会把该单元视为计费结果不确定, +不会自动再次请求。这样不能消除第三方已经受理但本地尚未落账的物理窗口, +但可以把这个窗口显式化并阻止无证据的重复计费。 +""" + +from __future__ import annotations + +from dataclasses import dataclass +import hashlib +import json +from typing import Any, Callable + +from app.db.database import get_connection +from app.services import job_service +from app.services.ai.base import AIProviderError + + +_CHECKPOINT_KEY = "_ai_analysis_units_v1" +_ALLOWED_JOB_TYPES = { + job_service.JOB_TYPE_AI_ANALYSIS, + job_service.JOB_TYPE_AUTO_PIPELINE, +} + + +@dataclass(frozen=True) +class AIUnitExecution: + status: str + payload: dict[str, Any] | None = None + error: str = "" + reused: bool = False + + +def build_unit_fingerprint(payload: dict[str, Any]) -> str: + serialized = json.dumps(payload, ensure_ascii=False, sort_keys=True, separators=(",", ":")) + return hashlib.sha256(serialized.encode("utf-8")).hexdigest() + + +def provider_fingerprint_fields(provider: Any) -> dict[str, str]: + """把真正影响结果来源的 Provider 配置纳入不可逆输入指纹。""" + config = getattr(provider, "config", None) + return { + "provider_class": f"{type(provider).__module__}.{type(provider).__qualname__}", + "provider_name": str(getattr(provider, "name", "") or ""), + "model": str(getattr(config, "model", getattr(provider, "model", "")) or ""), + "base_url": str(getattr(config, "base_url", "") or "").rstrip("/"), + "protocol": str(getattr(config, "protocol", "") or ""), + "fallback_protocol": str(getattr(config, "fallback_protocol", "") or ""), + "responses_path": str(getattr(config, "responses_path", "") or ""), + "reasoning_effort": str(getattr(config, "reasoning_effort", "") or ""), + "executable": str(getattr(config, "executable", "") or ""), + "codex_home": str(getattr(config, "codex_home", "") or ""), + } + + +def execute_checkpointed_ai_unit( + *, + task_id: str, + namespace: str, + input_fingerprint: str, + unit_id: str, + operation: Callable[[], dict[str, Any]], +) -> AIUnitExecution: + """执行一个可恢复 AI 单元;无 Job context 时保持旧的库内调用兼容。""" + active = job_service.current_job_lease() + if active is None: + return _execute_untracked(operation) + + state = _begin_unit( + task_id=task_id, + namespace=namespace, + input_fingerprint=input_fingerprint, + unit_id=unit_id, + ) + if state.status != "call_provider": + return state + + try: + payload = operation() + if not isinstance(payload, dict): + raise ValueError("AI 单元结果不是 JSON 对象") + except job_service.JobLeaseLostError: + raise + except Exception as exc: + retryable = bool( + isinstance(exc, AIProviderError) + and exc.safe_to_retry + and not exc.billing_uncertain + ) + error = _safe_error(exc) + _finish_unit_failure( + task_id=task_id, + namespace=namespace, + input_fingerprint=input_fingerprint, + unit_id=unit_id, + error=error, + retryable=retryable, + ) + return AIUnitExecution( + status="retryable_failed" if retryable else "uncertain", + error=error, + ) + + _finish_unit_success( + task_id=task_id, + namespace=namespace, + input_fingerprint=input_fingerprint, + unit_id=unit_id, + payload=payload, + ) + return AIUnitExecution(status="completed", payload=payload) + + +def _execute_untracked(operation: Callable[[], dict[str, Any]]) -> AIUnitExecution: + try: + payload = operation() + if not isinstance(payload, dict): + raise ValueError("AI 单元结果不是 JSON 对象") + return AIUnitExecution(status="completed", payload=payload) + except Exception as exc: + retryable = bool( + isinstance(exc, AIProviderError) + and exc.safe_to_retry + and not exc.billing_uncertain + ) + return AIUnitExecution( + status="retryable_failed" if retryable else "uncertain", + error=_safe_error(exc), + ) + + +def _begin_unit( + *, + task_id: str, + namespace: str, + input_fingerprint: str, + unit_id: str, +) -> AIUnitExecution: + with get_connection() as connection: + connection.execute("BEGIN IMMEDIATE") + job = job_service.require_job_lease_with_connection( + connection, + task_id=task_id, + allowed_job_types=_ALLOWED_JOB_TYPES, + ) + checkpoint = _checkpoint_dict(job.get("checkpoint_json")) + namespace_state = _namespace_state(checkpoint, namespace, input_fingerprint) + units = namespace_state.setdefault("units", {}) + previous = units.get(unit_id) + if isinstance(previous, dict) and previous.get("status") == "completed": + payload = _verified_payload(previous) + if payload is not None: + connection.commit() + return AIUnitExecution(status="completed", payload=payload, reused=True) + previous = { + "status": "uncertain", + "error": "AI 单元 checkpoint 校验失败,未自动重复请求", + } + units[unit_id] = previous + _write_checkpoint(connection, str(job["id"]), checkpoint) + connection.commit() + return AIUnitExecution(status="uncertain", error=str(previous["error"]), reused=True) + if isinstance(previous, dict) and previous.get("status") in {"running", "uncertain"}: + error = str(previous.get("error") or "上一次 AI 请求已开始但结果未确认,未自动重复请求") + units[unit_id] = { + **previous, + "status": "uncertain", + "error": error, + } + _write_checkpoint(connection, str(job["id"]), checkpoint) + connection.commit() + return AIUnitExecution(status="uncertain", error=error, reused=True) + + if previous is not None and not ( + isinstance(previous, dict) and previous.get("status") == "retryable_failed" + ): + error = "AI 单元 checkpoint 状态未知或结构损坏,未自动重复请求" + units[unit_id] = {"status": "uncertain", "error": error} + _write_checkpoint(connection, str(job["id"]), checkpoint) + connection.commit() + return AIUnitExecution(status="uncertain", error=error, reused=True) + + active = job_service.current_job_lease() + if active is None: + connection.rollback() + raise job_service.JobLeaseLostError("AI 单元缺少 Workflow Job 租约") + units[unit_id] = { + "status": "running", + "workflow_job_id": str(job["id"]), + "lease_generation": hashlib.sha256(active[2].encode("utf-8")).hexdigest(), + } + _write_checkpoint(connection, str(job["id"]), checkpoint) + connection.commit() + return AIUnitExecution(status="call_provider") + + +def _finish_unit_success( + *, + task_id: str, + namespace: str, + input_fingerprint: str, + unit_id: str, + payload: dict[str, Any], +) -> None: + result_json = json.dumps(payload, ensure_ascii=False, sort_keys=True, separators=(",", ":")) + result_checksum = hashlib.sha256(result_json.encode("utf-8")).hexdigest() + _finish_unit( + task_id=task_id, + namespace=namespace, + input_fingerprint=input_fingerprint, + unit_id=unit_id, + state={ + "status": "completed", + "result_json": result_json, + "result_checksum": result_checksum, + }, + ) + + +def _finish_unit_failure( + *, + task_id: str, + namespace: str, + input_fingerprint: str, + unit_id: str, + error: str, + retryable: bool, +) -> None: + _finish_unit( + task_id=task_id, + namespace=namespace, + input_fingerprint=input_fingerprint, + unit_id=unit_id, + state={ + "status": "retryable_failed" if retryable else "uncertain", + "error": error, + }, + ) + + +def _finish_unit( + *, + task_id: str, + namespace: str, + input_fingerprint: str, + unit_id: str, + state: dict[str, Any], +) -> None: + with get_connection() as connection: + connection.execute("BEGIN IMMEDIATE") + job = job_service.require_job_lease_with_connection( + connection, + task_id=task_id, + allowed_job_types=_ALLOWED_JOB_TYPES, + ) + checkpoint = _checkpoint_dict(job.get("checkpoint_json")) + namespace_state = _namespace_state(checkpoint, namespace, input_fingerprint) + units = namespace_state.setdefault("units", {}) + previous = units.get(unit_id) + if not isinstance(previous, dict) or previous.get("status") != "running": + connection.rollback() + raise job_service.JobLeaseLostError(f"AI 单元执行代际已改变:{unit_id}") + active = job_service.current_job_lease() + generation = hashlib.sha256(active[2].encode("utf-8")).hexdigest() if active else "" + if str(previous.get("lease_generation") or "") != generation: + connection.rollback() + raise job_service.JobLeaseLostError(f"AI 单元 lease 代际已改变:{unit_id}") + units[unit_id] = state + _write_checkpoint(connection, str(job["id"]), checkpoint) + connection.commit() + + +def _checkpoint_dict(value: Any) -> dict[str, Any]: + if value in (None, "", {}): + return {} + if not isinstance(value, dict): + raise ValueError("Workflow Job checkpoint_json 已损坏,拒绝重复调用 AI") + return value + + +def _namespace_state( + checkpoint: dict[str, Any], + namespace: str, + input_fingerprint: str, +) -> dict[str, Any]: + root = checkpoint.setdefault(_CHECKPOINT_KEY, {}) + if not isinstance(root, dict): + raise ValueError("AI 单元 checkpoint 根节点已损坏,拒绝重复调用 AI") + namespaces = root.setdefault("namespaces", {}) + if not isinstance(namespaces, dict): + raise ValueError("AI 单元 checkpoint namespaces 已损坏,拒绝重复调用 AI") + state = namespaces.get(namespace) + if state is None: + state = {"input_fingerprint": input_fingerprint, "units": {}} + namespaces[namespace] = state + elif not isinstance(state, dict): + raise ValueError(f"AI 单元 checkpoint namespace 已损坏:{namespace}") + elif not isinstance(state.get("input_fingerprint"), str): + raise ValueError(f"AI 单元 checkpoint 缺少输入指纹:{namespace}") + elif state.get("input_fingerprint") != input_fingerprint: + raise ValueError( + f"AI 单元输入指纹已变化,旧恢复证据不会被覆盖:{namespace};" + "请保留当前 Job 并人工确认后再创建新的分析批次" + ) + if not isinstance(state.get("units"), dict): + raise ValueError(f"AI 单元 checkpoint 已损坏:{namespace}") + return state + + +def _write_checkpoint(connection, job_id: str, checkpoint: dict[str, Any]) -> None: + cursor = connection.execute( + "UPDATE workflow_jobs SET checkpoint_json = ?, checkpoint_updated_at = CURRENT_TIMESTAMP, " + "updated_at = CURRENT_TIMESTAMP WHERE id = ?", + (json.dumps(checkpoint, ensure_ascii=False), job_id), + ) + if cursor.rowcount != 1: + raise job_service.JobLeaseLostError(f"Workflow Job checkpoint 写入失败:{job_id}") + + +def _verified_payload(state: dict[str, Any]) -> dict[str, Any] | None: + result_json = str(state.get("result_json") or "") + checksum = str(state.get("result_checksum") or "") + if not result_json or not checksum: + return None + if hashlib.sha256(result_json.encode("utf-8")).hexdigest() != checksum: + return None + try: + payload = json.loads(result_json) + except json.JSONDecodeError: + return None + return payload if isinstance(payload, dict) else None + + +def _safe_error(exc: Exception) -> str: + if isinstance(exc, AIProviderError): + text = exc.checkpoint_message() + else: + text = str(exc) + return " ".join(str(text or "AI 单元执行失败").split())[:1000] diff --git a/app/services/ai/variety_comedy_analyzer.py b/app/services/ai/variety_comedy_analyzer.py index e8cb720..92aff65 100644 --- a/app/services/ai/variety_comedy_analyzer.py +++ b/app/services/ai/variety_comedy_analyzer.py @@ -3,13 +3,15 @@ from __future__ import annotations from dataclasses import dataclass +import hashlib import json +import math from pathlib import Path import re from typing import Any from app.models.task import AIClipAnalysisResult -from app.services.ai.base import AIProvider +from app.services.ai.base import AIProvider, generate_json_with_safe_retry from app.services.ai.ai_clip_analyzer import ( AIAnalysisError, TranscriptRow, @@ -22,6 +24,11 @@ ) from app.services.audio_reaction_service import analyze_audio_reaction from app.services.clip_feedback_service import list_recent_feedback_context +from app.services.ai.unit_checkpoint import ( + build_unit_fingerprint, + execute_checkpointed_ai_unit, + provider_fingerprint_fields, +) REMOTE_WINDOW_SECONDS = 300 @@ -73,18 +80,38 @@ def analyze_variety_comedy(request: ComedyAnalysisRequest) -> AIClipAnalysisResu provider = build_provider(request.provider_name) windows = build_comedy_windows(rows, provider_name=request.provider_name) preference = _preference_summary(request.prompt_template or "", request.ai_preference) - moments, recall_failures = _recall_moments(provider, windows, preference) + unit_fingerprint = build_unit_fingerprint( + { + "profile": "variety_comedy", + "transcript_sha256": hashlib.sha256(transcript_text.encode("utf-8")).hexdigest(), + "provider": request.provider_name, + "provider_identity": provider_fingerprint_fields(provider), + "prompt_template": request.prompt_template or "", + "ai_preference": request.ai_preference, + "candidate_pool_limit": request.candidate_pool_limit, + "final_clip_target": request.final_clip_target, + } + ) + moments, recall_failures, recall_stats = _recall_moments( + provider, + windows, + preference, + task_id=request.task_id, + input_fingerprint=unit_fingerprint, + ) moments = dedupe_recall_moments(moments)[:MAX_PRELIMINARY_MOMENTS] if not moments: detail = ";".join(recall_failures[:3]) or "没有召回达到条件的笑点时刻" raise AIAnalysisError(f"综艺笑点分析没有召回可用内容:{detail}") - expanded, expansion_failures = _expand_moments( + expanded, expansion_failures, expansion_stats = _expand_moments( provider, rows, moments, preference, provider_name=request.provider_name, + task_id=request.task_id, + input_fingerprint=unit_fingerprint, ) expanded = dedupe_expanded_candidates(expanded)[:MAX_PRELIMINARY_MOMENTS] if not expanded: @@ -105,7 +132,14 @@ def analyze_variety_comedy(request: ComedyAnalysisRequest) -> AIClipAnalysisResu ) feedback = list_recent_feedback_context("variety_comedy", limit=20) - judge_payload, judge_warning = _global_judge(provider, expanded, preference, feedback) + judge_payload, judge_warning = _global_judge( + provider, + expanded, + preference, + feedback, + task_id=request.task_id, + input_fingerprint=unit_fingerprint, + ) scored = [ score_comedy_candidate(candidate, judge_payload.get(candidate["source_id"]) or {}) for candidate in expanded @@ -139,7 +173,50 @@ def analyze_variety_comedy(request: ComedyAnalysisRequest) -> AIClipAnalysisResu warnings.append(judge_warning) if warnings: summary += f" 有 {len(warnings)} 个局部步骤已降级或跳过。" - return AIClipAnalysisResult(task_id=request.task_id, analysis_summary=summary, clips=clips) + expected_units = int(recall_stats["expected_units"]) + int(expansion_stats["expected_units"]) + 1 + completed_units = ( + int(recall_stats["completed_units"]) + + int(expansion_stats["completed_units"]) + + (0 if judge_warning else 1) + ) + failed_units = ( + int(recall_stats["failed_units"]) + + int(expansion_stats["failed_units"]) + + (1 if judge_warning else 0) + ) + invalid_item_count = int(recall_stats["invalid_item_count"]) + int( + expansion_stats["invalid_item_count"] + ) + coverage_ratio = completed_units / expected_units if expected_units else 0.0 + failed_stages = [ + {"stage": "recall", "message": " ".join(message.split())[:500]} + for message in recall_failures + ] + [ + {"stage": "expansion", "message": " ".join(message.split())[:500]} + for message in expansion_failures + ] + if judge_warning: + failed_stages.append({"stage": "global_judge", "message": " ".join(judge_warning.split())[:500]}) + return AIClipAnalysisResult( + task_id=request.task_id, + analysis_summary=summary, + clips=clips, + analysis_meta={ + "schema_version": 2, + "coverage_basis": "recall_and_expansion_units", + "expected_units": expected_units, + "completed_units": completed_units, + "failed_units": failed_units, + "empty_unit_count": int(recall_stats["empty_unit_count"]) + + int(expansion_stats["empty_unit_count"]), + "invalid_item_count": invalid_item_count, + "coverage_ratio": round(coverage_ratio, 6), + "coverage_percent": round(coverage_ratio * 100, 2), + "analysis_incomplete": bool(failed_units or invalid_item_count or judge_warning), + "quality_degraded": bool(judge_warning), + "failed_stages": failed_stages, + }, + ) def build_comedy_windows( @@ -369,24 +446,54 @@ def _recall_moments( provider: AIProvider, windows: list[ComedyTranscriptWindow], preference: str, -) -> tuple[list[dict], list[str]]: + *, + task_id: str, + input_fingerprint: str, +) -> tuple[list[dict], list[str], dict[str, int]]: moments: list[dict] = [] failures = [] + completed_units = 0 + failed_units = 0 + empty_units = 0 + invalid_item_count = 0 for window in windows: prompt = _recall_prompt(window, preference) + execution = execute_checkpointed_ai_unit( + task_id=task_id, + namespace="variety_recall", + input_fingerprint=input_fingerprint, + unit_id=f"window_{window.index:03d}", + operation=lambda prompt=prompt: _generate_payload(provider, prompt, expected_key="moments"), + ) + if execution.status != "completed" or not isinstance(execution.payload, dict): + failed_units += 1 + failures.append( + f"召回窗口 {window.index}/{window.total} 跳过:" + f"{execution.error or execution.status}" + ) + continue try: - payload = _generate_payload(provider, prompt, expected_key="moments") - raw_moments = payload.get("moments") or [] + payload = execution.payload + raw_moments = payload.get("moments") if not isinstance(raw_moments, list): raise AIAnalysisError("moments 不是数组") + completed_units += 1 + if not raw_moments: + empty_units += 1 for index, item in enumerate(raw_moments[:RECALL_LIMIT_PER_WINDOW], start=1): if not isinstance(item, dict): + invalid_item_count += 1 + failures.append(f"召回窗口 {window.index}/{window.total} 第 {index} 条不是对象") continue key_text = _first_time(item, ("key_time", "key_moment_time", "moment_time", "start_time")) if not key_text: + invalid_item_count += 1 + failures.append(f"召回窗口 {window.index}/{window.total} 第 {index} 条缺少关键时间") continue key_seconds = _time_to_seconds(key_text) if key_seconds < window.start_seconds or key_seconds > window.end_seconds: + invalid_item_count += 1 + failures.append(f"召回窗口 {window.index}/{window.total} 第 {index} 条关键时间越界") continue moments.append( { @@ -400,8 +507,15 @@ def _recall_moments( } ) except Exception as exc: + failed_units += 1 failures.append(f"召回窗口 {window.index}/{window.total} 跳过:{exc}") - return moments, failures + return moments, failures, { + "expected_units": len(windows), + "completed_units": completed_units, + "failed_units": failed_units, + "empty_unit_count": empty_units, + "invalid_item_count": invalid_item_count, + } def _expand_moments( @@ -411,10 +525,17 @@ def _expand_moments( preference: str, *, provider_name: str, -) -> tuple[list[dict], list[str]]: + task_id: str, + input_fingerprint: str, +) -> tuple[list[dict], list[str], dict[str, int]]: expanded = [] failures = [] batch_size = 1 if provider_name == "local" else EXPANSION_BATCH_SIZE_REMOTE + expected_units = math.ceil(len(moments) / batch_size) if moments else 0 + completed_units = 0 + failed_units = 0 + empty_units = 0 + invalid_item_count = 0 for offset in range(0, len(moments), batch_size): batch = moments[offset : offset + batch_size] contexts = [] @@ -435,23 +556,47 @@ def _expand_moments( "transcript": "\n".join(_format_row(row) for row in context_rows), } ) + prompt = _expansion_prompt(contexts, preference) + batch_number = offset // batch_size + 1 + execution = execute_checkpointed_ai_unit( + task_id=task_id, + namespace="variety_expansion", + input_fingerprint=input_fingerprint, + unit_id=f"batch_{batch_number:03d}", + operation=lambda prompt=prompt: _generate_payload(provider, prompt, expected_key="clips"), + ) + if execution.status != "completed" or not isinstance(execution.payload, dict): + failed_units += 1 + failures.append( + f"上下文扩展批次 {batch_number} 跳过:{execution.error or execution.status}" + ) + continue try: - payload = _generate_payload(provider, _expansion_prompt(contexts, preference), expected_key="clips") - raw_clips = payload.get("clips") or [] + payload = execution.payload + raw_clips = payload.get("clips") if not isinstance(raw_clips, list): raise AIAnalysisError("clips 不是数组") - for item in raw_clips: + completed_units += 1 + if not raw_clips: + empty_units += 1 + for item_index, item in enumerate(raw_clips, start=1): if not isinstance(item, dict): + invalid_item_count += 1 + failures.append(f"上下文扩展批次 {offset // batch_size + 1} 第 {item_index} 条不是对象") continue source_id = str(item.get("source_id") or "") moment = next((value for value in batch if value["source_id"] == source_id), None) context_rows = context_rows_by_id.get(source_id) or [] if not moment or not context_rows: + invalid_item_count += 1 + failures.append(f"上下文扩展批次 {offset // batch_size + 1} 第 {item_index} 条 source_id 无效") continue start_text = _first_time(item, ("start_time",)) end_text = _first_time(item, ("end_time",)) key_text = _first_time(item, ("key_moment_time", "key_time")) or moment["key_time"] if not start_text or not end_text: + invalid_item_count += 1 + failures.append(f"上下文扩展批次 {offset // batch_size + 1} 第 {item_index} 条缺少起止时间") continue bounds = normalize_clip_bounds( _time_to_seconds(start_text), @@ -460,6 +605,8 @@ def _expand_moments( context_rows, ) if not bounds: + invalid_item_count += 1 + failures.append(f"上下文扩展批次 {offset // batch_size + 1} 第 {item_index} 条时间范围无效") continue start_seconds, end_seconds = bounds key_seconds = min(end_seconds - 1, max(start_seconds, _time_to_seconds(key_text))) @@ -485,8 +632,15 @@ def _expand_moments( } ) except Exception as exc: + failed_units += 1 failures.append(f"上下文扩展批次 {offset // batch_size + 1} 跳过:{exc}") - return expanded, failures + return expanded, failures, { + "expected_units": expected_units, + "completed_units": completed_units, + "failed_units": failed_units, + "empty_unit_count": empty_units, + "invalid_item_count": invalid_item_count, + } def _global_judge( @@ -494,6 +648,9 @@ def _global_judge( candidates: list[dict], preference: str, feedback: list[dict], + *, + task_id: str, + input_fingerprint: str, ) -> tuple[dict[str, dict], str]: prompt_candidates = [] for item in candidates: @@ -510,20 +667,42 @@ def _global_judge( "audio_reaction": item.get("audio_evidence") or {}, } ) + prompt = _judge_prompt(prompt_candidates, preference, feedback) + execution = execute_checkpointed_ai_unit( + task_id=task_id, + namespace="variety_global_judge", + input_fingerprint=input_fingerprint, + unit_id="judge_001", + operation=lambda: _generate_payload(provider, prompt, expected_key="ranked_clips"), + ) + if execution.status != "completed" or not isinstance(execution.payload, dict): + return {}, f"全局评审结果不确定,已锁定自动切片:{execution.error or execution.status}" try: - payload = _generate_payload( - provider, - _judge_prompt(prompt_candidates, preference, feedback), - expected_key="ranked_clips", - ) - raw_items = payload.get("ranked_clips") or payload.get("clips") or [] + payload = execution.payload + raw_items = payload.get("ranked_clips") if not isinstance(raw_items, list): raise AIAnalysisError("ranked_clips 不是数组") - return { - str(item.get("source_id")): item + known_ids = {str(item["source_id"]) for item in candidates} + valid_items = [ + item for item in raw_items - if isinstance(item, dict) and item.get("source_id") - }, "" + if isinstance(item, dict) and str(item.get("source_id") or "") in known_ids + ] + judged = {str(item["source_id"]): item for item in valid_items} + invalid_count = len(raw_items) - len(valid_items) + duplicate_count = len(valid_items) - len(judged) + missing_ids = sorted(known_ids - set(judged)) + if not judged: + raise AIAnalysisError("全局评审没有返回任何可对应的 source_id") + issues = [] + if invalid_count: + issues.append(f"{invalid_count} 条无效条目") + if duplicate_count: + issues.append(f"{duplicate_count} 条重复 source_id") + if missing_ids: + issues.append(f"缺少 {len(missing_ids)} 个候选") + warning = f"全局评审{'、'.join(issues)},已锁定自动切片" if issues else "" + return judged, warning except Exception as exc: return {}, f"全局评审调用失败,已使用扩展阶段评分降级:{exc}" @@ -560,20 +739,12 @@ def _to_clip_payload(item: dict, index: int) -> dict: def _generate_payload(provider: AIProvider, prompt: str, *, expected_key: str) -> dict: - raw = provider.generate_json(prompt) - try: - payload = _loads_ai_json(raw) - except AIAnalysisError as first_error: - raw = provider.generate_json( - prompt, - retry_instruction=f"上一次输出无法解析。只返回严格 JSON,并确保包含 {expected_key} 数组。", - ) - try: - payload = _loads_ai_json(raw) - except AIAnalysisError as second_error: - raise AIAnalysisError(str(second_error)) from first_error + raw = generate_json_with_safe_retry(provider, prompt) + payload = _loads_ai_json(raw) if not isinstance(payload, dict): raise AIAnalysisError("AI 输出必须是 JSON 对象") + if not isinstance(payload.get(expected_key), list): + raise AIAnalysisError(f"AI 输出缺少 {expected_key} 数组") return payload diff --git a/app/services/ai_analysis_workflow_service.py b/app/services/ai_analysis_workflow_service.py index 6f0e487..d135b87 100644 --- a/app/services/ai_analysis_workflow_service.py +++ b/app/services/ai_analysis_workflow_service.py @@ -4,6 +4,8 @@ """ import json +import math +import os from pathlib import Path from sqlite3 import Row from uuid import uuid4 @@ -19,12 +21,42 @@ result_to_jsonable, ) from app.services.ai.variety_comedy_analyzer import ComedyAnalysisRequest, analyze_variety_comedy +from app.services.ai.long_live_talk_analyzer import ( + LongLiveAnalysisOutcome, + LongLiveAnalysisRequest, + analyze_long_live_talk, + get_latest_long_live_window_status, +) from app.services.ai.diagnostics import ensure_local_ai_ready +from app.services.ai.base import AIProviderError from app.services.ai_prompt_preset_service import get_task_ai_prompt_preset from app.services.storage_service import get_artifact_paths from app.services.task_log_service import append_task_log, read_task_log_tail AI_CLIP_MIN_RECOMMENDED_SECONDS = 45 +_ANALYSIS_INTEGRITY_MARKER = "_analysis_integrity_error" + + +class AIAnalysisConflictError(ValueError): + """当前任务已有执行中或下游结果,拒绝破坏性 AI 重入。""" + + +def _invalid_analysis_payload(task_id: str, reason: str) -> dict: + """把损坏或缺失的质量证据转换为明确的 fail-closed 元数据。""" + return { + "task_id": task_id, + "analysis_summary": "AI 分析完整性证据已损坏,请重新分析或恢复可信历史。", + "clips": [], + "analysis_meta": { + "schema_version": 2, + "analysis_incomplete": True, + "quality_degraded": True, + "coverage_ratio": 0.0, + "coverage_percent": 0.0, + "integrity_error": reason, + }, + _ANALYSIS_INTEGRITY_MARKER: reason, + } # ---------- AI Provider 辅助 ---------- @@ -63,14 +95,111 @@ def _summarize_ai_error(error: str) -> str: def _read_analysis_meta(task_id: str) -> dict: + active_payload = _load_active_analysis_payload(task_id) + if active_payload: + meta = active_payload.get("analysis_meta") + return dict(meta) if isinstance(meta, dict) else {} paths = get_artifact_paths(task_id) if not paths["analysis_path"].exists(): return {} try: payload = json.loads(paths["analysis_path"].read_text(encoding="utf-8", errors="replace")) except (OSError, json.JSONDecodeError): - return {} - return payload.get("analysis_meta") or {} + return dict(_invalid_analysis_payload(task_id, "analysis_file_invalid")["analysis_meta"]) + meta = payload.get("analysis_meta") if isinstance(payload, dict) else None + if not isinstance(meta, dict): + return dict(_invalid_analysis_payload(task_id, "analysis_meta_missing")["analysis_meta"]) + return dict(meta) + + +def get_task_ai_analysis_meta(task_id: str) -> dict: + """读取当前生效分析的元数据,供长直播覆盖率门禁复用。""" + return dict(_read_analysis_meta(task_id)) + + +def validate_ai_analysis_meta_for_cut(meta: dict, expected_profile: str) -> dict: + """验证进入切片所需的 AI 质量证据;异常证据统一 fail closed。""" + resolved = dict(meta) if isinstance(meta, dict) else {} + reason = "" + schema_version = resolved.get("schema_version") + if type(schema_version) is not int or schema_version != 2: + reason = "analysis_schema_invalid" + elif str(resolved.get("selection_profile") or "") != str(expected_profile or "general"): + reason = "selection_profile_mismatch" + elif not isinstance(resolved.get("analysis_incomplete"), bool): + reason = "analysis_incomplete_missing" + elif not isinstance(resolved.get("quality_degraded"), bool): + reason = "quality_degraded_missing" + else: + coverage_ratio_value = resolved.get("coverage_ratio") + coverage_percent_value = resolved.get("coverage_percent") + if ( + type(coverage_ratio_value) not in (int, float) + or type(coverage_percent_value) not in (int, float) + ): + reason = "analysis_coverage_invalid" + else: + coverage_ratio = float(coverage_ratio_value) + coverage_percent = float(coverage_percent_value) + if ( + not math.isfinite(coverage_ratio) + or not math.isfinite(coverage_percent) + or not (0.0 <= coverage_ratio <= 1.0) + or not (0.0 <= coverage_percent <= 100.0) + or abs(coverage_percent - coverage_ratio * 100.0) > 0.01 + ): + reason = "analysis_coverage_invalid" + if not reason and not resolved["analysis_incomplete"] and not resolved["quality_degraded"]: + profile = str(expected_profile or "general") + invalid_item_count = resolved.get("invalid_item_count") + if type(invalid_item_count) is not int or invalid_item_count != 0: + reason = "analysis_unit_evidence_invalid" + elif profile == "long_live_talk": + window_count = resolved.get("window_count") + completed_count = resolved.get("completed_window_count") + failed_count = resolved.get("failed_window_count") + failed_windows = resolved.get("failed_windows") + if ( + type(window_count) is not int + or type(completed_count) is not int + or type(failed_count) is not int + or window_count <= 0 + or completed_count != window_count + or failed_count != 0 + or not isinstance(failed_windows, list) + or failed_windows + ): + reason = "analysis_window_evidence_invalid" + else: + expected_units = resolved.get("expected_units") + completed_units = resolved.get("completed_units") + failed_units = resolved.get("failed_units") + failed_stages = resolved.get("failed_stages") + if ( + type(expected_units) is not int + or type(completed_units) is not int + or type(failed_units) is not int + or expected_units <= 0 + or completed_units != expected_units + or failed_units != 0 + or not isinstance(failed_stages, list) + or failed_stages + or abs(float(resolved["coverage_ratio"]) - completed_units / expected_units) > 0.000001 + or abs(float(resolved["coverage_percent"]) - completed_units / expected_units * 100.0) > 0.01 + ): + reason = "analysis_unit_evidence_invalid" + if not reason: + return resolved + return { + **resolved, + "schema_version": 2, + "selection_profile": str(expected_profile or "general"), + "analysis_incomplete": True, + "quality_degraded": True, + "coverage_ratio": 0.0, + "coverage_percent": 0.0, + "integrity_error": reason, + } def _read_latest_ai_provider_from_log(task_id: str) -> str: @@ -113,15 +242,31 @@ def get_task_ai_analysis_status(task_id: str) -> dict: paths = get_artifact_paths(task_id) log_lines = read_task_log_tail(task_id) is_running = task.get("status") == TaskStatus.ai_analyzing.value - has_analysis = paths["analysis_path"].exists() + active_payload = _load_active_analysis_payload(task_id) + has_analysis = bool(active_payload) or paths["analysis_path"].exists() + if active_payload: + try: + _materialize_analysis_payload(task_id, active_payload) + except OSError: + pass + window_status = ( + get_latest_long_live_window_status(task_id) + if task.get("selection_profile") == "long_live_talk" + else {} + ) percent = 0 message = "等待开始 AI 分析" status = "idle" if is_running: status = "running" - percent = 48 - message = "AI 正在分析转写文本,请保持页面打开。" + percent = int(window_status.get("percent") or 0) or 1 + message = ( + f"长直播 AI 正在处理窗口:已完成 {window_status.get('completed_window_count', 0)}" + f"/{window_status.get('window_count', 0)}。" + if window_status + else "AI 正在分析转写文本,请保持页面打开。" + ) if any("将使用分段分析" in line for line in log_lines): percent = 62 message = "AI 已读取 Prompt 和转写文本,正在分段生成候选片段。" @@ -129,9 +274,16 @@ def get_task_ai_analysis_status(task_id: str) -> dict: percent = 72 message = "远程 AI 分析接口暂不可用,已暂停等待你确认下一步。" elif task.get("status") == TaskStatus.pending_review.value and has_analysis: - status = "completed" - percent = 100 - message = "AI 分析完成,候选片段已生成,可检查后直接生成切片。" + meta = _read_analysis_meta(task_id) + status = "incomplete" if meta.get("analysis_incomplete") else "completed" + percent = int(float(meta.get("coverage_percent") or 100)) + message = ( + f"{task.get('selection_profile') or 'general'} 分析覆盖 " + f"{float(meta.get('coverage_percent') or 0):.2f}%," + "仍有处理单元失败;请重试 AI 分析补齐结果。" + if meta.get("analysis_incomplete") + else "AI 分析完成,候选片段已生成,可检查后直接生成切片。" + ) elif task.get("status") == TaskStatus.failed.value and any("AI 分析失败" in line for line in log_lines): status = "failed" percent = 100 @@ -153,6 +305,7 @@ def get_task_ai_analysis_status(task_id: str) -> dict: "log_path": str(paths["log_path"]), "log_lines": log_lines, "error_message": task.get("error_message") or "", + "window_status": window_status, } @@ -232,8 +385,279 @@ def _replace_clip_candidates(task_id: str, clips: list[dict]) -> None: now = _now_iso() with get_connection() as connection: - connection.execute("DELETE FROM clip_candidates WHERE task_id = ?", (task_id,)) - _insert_clip_candidates_with_connection(connection, task_id, clips, now) + connection.execute("BEGIN IMMEDIATE") + _replace_clip_candidates_with_connection(connection, task_id, clips, now) + connection.commit() + + +def _replace_clip_candidates_with_connection( + connection, + task_id: str, + clips: list[dict], + now: str, +) -> None: + referenced = connection.execute( + """ + SELECT 1 + FROM output_clip + WHERE task_id = ? AND clip_candidate_id IS NOT NULL + LIMIT 1 + """, + (task_id,), + ).fetchone() + if referenced: + raise AIAnalysisConflictError( + "任务已有切片引用当前候选结果,不能直接覆盖 AI 分析;" + "请在片段审核页修改现有候选并重新切片。" + ) + connection.execute("DELETE FROM clip_candidates WHERE task_id = ?", (task_id,)) + _insert_clip_candidates_with_connection(connection, task_id, clips, now) + + +def _assert_ai_task_can_start(connection, task_id: str, current_status: str, *, current_job_id: str = "") -> None: + from app.services import job_service + active_pipeline = connection.execute( + """ + SELECT 1 + FROM workflow_jobs + WHERE task_id = ? AND job_type = ? AND status IN (?, ?) AND id <> ? + LIMIT 1 + """, + ( + task_id, + job_service.JOB_TYPE_AUTO_PIPELINE, + job_service.JOB_STATUS_QUEUED, + job_service.JOB_STATUS_RUNNING, + current_job_id, + ), + ).fetchone() + if active_pipeline: + raise AIAnalysisConflictError("全自动流水线正在处理此任务,请等待当前流程完成,不要重复启动 AI 分析。") + materialized = connection.execute( + """ + SELECT 1 + WHERE EXISTS ( + SELECT 1 FROM output_clip + WHERE task_id = ? AND COALESCE(is_active, 1) = 1 + ) OR EXISTS ( + SELECT 1 FROM publish_jobs WHERE task_id = ? + ) + """, + (task_id, task_id), + ).fetchone() + if materialized: + raise AIAnalysisConflictError( + "任务已经生成切片或进入发送中心,不能覆盖 AI 候选;请在片段审核页修改后重新切片。" + ) + if current_status not in { + TaskStatus.pending_ai.value, + TaskStatus.pending_review.value, + TaskStatus.failed.value, + TaskStatus.ai_analyzing.value, + TaskStatus.AI_ANALYZING.value, + }: + raise AIAnalysisConflictError("当前任务阶段不允许启动 AI 分析,请从任务详情继续正确流程。") + + +def queue_task_ai_analysis(task_id: str, provider: str | None = None) -> tuple[dict, bool]: + """原子验证人工请求并创建/复用持久 AI Job。""" + from app.services import job_service + from app.services.task_service import _now_iso + + with get_connection() as connection: + connection.execute("BEGIN IMMEDIATE") + current = connection.execute( + "SELECT status FROM tasks WHERE id = ? AND COALESCE(is_deleted, 0) = 0", + (task_id,), + ).fetchone() + if not current: + connection.rollback() + raise ValueError("任务不存在") + existing = connection.execute( + """ + SELECT id, payload_json FROM workflow_jobs + WHERE task_id = ? AND job_type = ? AND status IN (?, ?) + ORDER BY created_at DESC LIMIT 1 + """, + ( + task_id, + job_service.JOB_TYPE_AI_ANALYSIS, + job_service.JOB_STATUS_QUEUED, + job_service.JOB_STATUS_RUNNING, + ), + ).fetchone() + if existing: + connection.commit() + return job_service.get_job(str(existing["id"])), False + _assert_ai_task_can_start(connection, task_id, str(current["status"] or "")) + retryable = connection.execute( + """ + SELECT id, payload_json FROM workflow_jobs + WHERE task_id = ? AND job_type = ? AND status IN (?, ?) + ORDER BY created_at DESC LIMIT 1 + """, + ( + task_id, + job_service.JOB_TYPE_AI_ANALYSIS, + job_service.JOB_STATUS_FAILED, + job_service.JOB_STATUS_CANCELLED, + ), + ).fetchone() + if retryable: + try: + retry_payload = json.loads(str(retryable["payload_json"] or "{}")) + except json.JSONDecodeError as exc: + connection.rollback() + raise AIAnalysisConflictError("旧 AI Job 的 Provider 账本已损坏,拒绝自动重试。") from exc + retry_provider = str( + retry_payload.get("provider") if isinstance(retry_payload, dict) else "" + ).lower() + requested_provider = (provider or settings.ai_default_provider).lower() + if retry_provider and retry_provider != requested_provider: + connection.rollback() + raise AIAnalysisConflictError( + f"旧 AI Job 的恢复账本属于 {retry_provider},不能在同一账本中切换为 " + f"{requested_provider};请先保留不确定证据并新建独立分析批次。" + ) + now = _now_iso() + cursor = connection.execute( + """ + UPDATE workflow_jobs + SET status = ?, progress = 0, message = 'AI 分析已使用原恢复账本重新加入队列', + result_json = '{}', error_message = NULL, finished_at = NULL, + next_attempt_at = ?, cancel_requested = 0, + lease_owner = NULL, lease_token = NULL, lease_expires_at = NULL, + heartbeat_at = NULL, attempt_count = 0, updated_at = ? + WHERE id = ? AND status IN (?, ?) + """, + ( + job_service.JOB_STATUS_QUEUED, + now, + now, + retryable["id"], + job_service.JOB_STATUS_FAILED, + job_service.JOB_STATUS_CANCELLED, + ), + ) + if cursor.rowcount != 1: + connection.rollback() + raise AIAnalysisConflictError("AI 分析任务状态刚刚变化,请刷新后重试。") + connection.commit() + return job_service.get_job(str(retryable["id"])), True + job_id, created = job_service.create_or_get_active_job_with_connection( + connection, + task_id=task_id, + job_type=job_service.JOB_TYPE_AI_ANALYSIS, + payload={"provider": (provider or settings.ai_default_provider).lower()}, + ) + connection.commit() + return job_service.get_job(job_id), created + + +def _begin_ai_analysis(task_id: str, task: dict) -> dict: + """在同一事务验证执行租约并进入 ai_analyzing。""" + from app.services import job_service + from app.services.task_service import STATUS_PROGRESS, _now_iso + + with get_connection() as connection: + connection.execute("BEGIN IMMEDIATE") + job = job_service.require_job_lease_with_connection( + connection, + task_id=task_id, + allowed_job_types={job_service.JOB_TYPE_AI_ANALYSIS, job_service.JOB_TYPE_AUTO_PIPELINE}, + ) + current = connection.execute( + "SELECT status FROM tasks WHERE id = ? AND COALESCE(is_deleted, 0) = 0", + (task_id,), + ).fetchone() + if not current: + connection.rollback() + raise AIAnalysisConflictError("任务不存在或已被删除,不能启动 AI 分析。") + current_status = str(current["status"] or "") + _assert_ai_task_can_start(connection, task_id, current_status, current_job_id=str(job["id"])) + if current_status != str(task.get("status") or ""): + connection.rollback() + raise AIAnalysisConflictError("任务状态刚刚发生变化,请刷新页面后再决定是否重新分析。") + cursor = connection.execute( + """ + UPDATE tasks + SET status = ?, progress = ?, error_message = NULL, last_error = NULL, updated_at = ? + WHERE id = ? AND status = ? AND COALESCE(is_deleted, 0) = 0 + """, + ( + TaskStatus.ai_analyzing.value, + STATUS_PROGRESS[TaskStatus.ai_analyzing.value], + _now_iso(), + task_id, + current_status, + ), + ) + if cursor.rowcount != 1: + connection.rollback() + raise AIAnalysisConflictError("任务状态刚刚发生变化,请刷新后重试。") + connection.commit() + return job + + +def _restore_task_after_ai_conflict(task_id: str, previous_task: dict) -> None: + """只回滚仍停留在本次 ai_analyzing 的状态,避免覆盖其他流程的新状态。""" + from app.services import job_service + from app.services.task_service import _now_iso + + with get_connection() as connection: + connection.execute("BEGIN IMMEDIATE") + job_service.require_job_lease_with_connection( + connection, + task_id=task_id, + allowed_job_types={job_service.JOB_TYPE_AI_ANALYSIS, job_service.JOB_TYPE_AUTO_PIPELINE}, + ) + connection.execute( + """ + UPDATE tasks + SET status = ?, progress = ?, error_message = ?, last_error = ?, updated_at = ? + WHERE id = ? AND status = ? AND COALESCE(is_deleted, 0) = 0 + """, + ( + previous_task.get("status") or TaskStatus.pending_review.value, + int(previous_task.get("progress") or 0), + previous_task.get("error_message") or None, + previous_task.get("last_error") or None, + _now_iso(), + task_id, + TaskStatus.ai_analyzing.value, + ), + ) + connection.commit() + + +def _mark_ai_failed_if_still_running(task_id: str, error_message: str) -> None: + """AI 失败只能结束自己仍持有的可见状态,不能降级已完成任务。""" + from app.services import job_service + from app.services.task_service import STATUS_PROGRESS, _now_iso + + with get_connection() as connection: + connection.execute("BEGIN IMMEDIATE") + job_service.require_job_lease_with_connection( + connection, + task_id=task_id, + allowed_job_types={job_service.JOB_TYPE_AI_ANALYSIS, job_service.JOB_TYPE_AUTO_PIPELINE}, + ) + connection.execute( + """ + UPDATE tasks + SET status = ?, progress = ?, error_message = ?, last_error = ?, updated_at = ? + WHERE id = ? AND status = ? AND COALESCE(is_deleted, 0) = 0 + """, + ( + TaskStatus.failed.value, + STATUS_PROGRESS[TaskStatus.failed.value], + error_message, + error_message, + _now_iso(), + task_id, + TaskStatus.ai_analyzing.value, + ), + ) connection.commit() @@ -266,6 +690,8 @@ def _analysis_run_row_to_dict(row: Row, include_payload: bool = False) -> dict: payload = {} clips = payload.get("clips") or [] + analysis_meta = payload.get("analysis_meta") or {} + analysis_meta = analysis_meta if isinstance(analysis_meta, dict) else {} return { "id": run.get("id"), "task_id": run.get("task_id"), @@ -285,6 +711,9 @@ def _analysis_run_row_to_dict(row: Row, include_payload: bool = False) -> dict: "review_url": f"/tasks/{run.get('task_id')}/clips/review", "clips": clips if include_payload else [], "clip_summaries": _summarize_analysis_clips(clips) if include_payload else [], + "analysis_meta": analysis_meta if include_payload else {}, + "analysis_incomplete": bool(analysis_meta.get("analysis_incomplete")), + "coverage_ratio": float(analysis_meta.get("coverage_ratio") or 0), } @@ -293,6 +722,7 @@ def _analysis_payload_to_preview(task_id: str, payload: dict, fallback: dict | N fallback = fallback or {} meta = payload.get("analysis_meta") or {} + meta = meta if isinstance(meta, dict) else {} clips = payload.get("clips") or [] provider = meta.get("provider") or fallback.get("provider") or settings.ai_default_provider model = meta.get("model") or fallback.get("model") or _ai_model_name(provider) @@ -382,48 +812,72 @@ def _insert_ai_analysis_run( from app.services.task_service import _now_iso now = _now_iso() - run_id = uuid4().hex[:12] - clips = analysis_payload.get("clips") or [] with get_connection() as connection: - run_number = _next_ai_analysis_run_number(connection, task_id) - # 将同 task 下所有旧 run 标记为非活跃 - connection.execute( - "UPDATE ai_analysis_runs SET is_active = 0 WHERE task_id = ?", - (task_id,), - ) - connection.execute( - """ - INSERT INTO ai_analysis_runs ( - id, task_id, run_number, provider, provider_label, model, - ai_prompt_preset_id, ai_prompt_preset_name, requested_clip_count, - clip_count, analysis_summary, fallback_notice, analysis_payload_json, - is_active, created_at - ) - VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) - """, - ( - run_id, - task_id, - run_number, - provider, - provider_label, - model, - prompt_preset.get("id") or "", - prompt_preset.get("name") or "", - requested_clip_count, - len(clips), - analysis_payload.get("analysis_summary") or "", - fallback_notice, - json.dumps(analysis_payload, ensure_ascii=False), - 1, # is_active - now, - ), + run_id = _insert_ai_analysis_run_with_connection( + connection, + task_id=task_id, + analysis_payload=analysis_payload, + provider=provider, + provider_label=provider_label, + model=model, + fallback_notice=fallback_notice, + prompt_preset=prompt_preset, + requested_clip_count=requested_clip_count, + now=now, ) connection.commit() return get_ai_analysis_run(task_id, run_id) +def _insert_ai_analysis_run_with_connection( + connection, + *, + task_id: str, + analysis_payload: dict, + provider: str, + provider_label: str, + model: str, + fallback_notice: str, + prompt_preset: dict, + requested_clip_count: int, + now: str, +) -> str: + run_id = uuid4().hex[:12] + clips = analysis_payload.get("clips") or [] + run_number = _next_ai_analysis_run_number(connection, task_id) + connection.execute("UPDATE ai_analysis_runs SET is_active = 0 WHERE task_id = ?", (task_id,)) + connection.execute( + """ + INSERT INTO ai_analysis_runs ( + id, task_id, run_number, provider, provider_label, model, + ai_prompt_preset_id, ai_prompt_preset_name, requested_clip_count, + clip_count, analysis_summary, fallback_notice, analysis_payload_json, + is_active, created_at + ) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + """, + ( + run_id, + task_id, + run_number, + provider, + provider_label, + model, + prompt_preset.get("id") or "", + prompt_preset.get("name") or "", + requested_clip_count, + len(clips), + analysis_payload.get("analysis_summary") or "", + fallback_notice, + json.dumps(analysis_payload, ensure_ascii=False), + 1, + now, + ), + ) + return run_id + + def _ensure_ai_analysis_history_from_current_file(task_id: str) -> None: with get_connection() as connection: row = connection.execute( @@ -440,12 +894,15 @@ def _ensure_ai_analysis_history_from_current_file(task_id: str) -> None: payload = json.loads(paths["analysis_path"].read_text(encoding="utf-8", errors="replace")) except (OSError, json.JSONDecodeError): return + if not isinstance(payload, dict): + return from app.services.task_service import get_task # noqa: F811 task = get_task(task_id, include_video_probe=False) clips = payload.get("clips") or [] meta = payload.get("analysis_meta") or {} + meta = meta if isinstance(meta, dict) else {} provider = str(meta.get("provider") or settings.ai_default_provider).lower() prompt_preset = get_task_ai_prompt_preset(task_id) _insert_ai_analysis_run( @@ -478,17 +935,74 @@ def get_ai_analysis_run(task_id: str, run_id: str) -> dict: def _write_analysis_payload(task_id: str, payload: dict) -> None: paths = get_artifact_paths(task_id) paths["analysis_path"].parent.mkdir(parents=True, exist_ok=True) - paths["analysis_path"].write_text( - json.dumps(payload, ensure_ascii=False, indent=2), - encoding="utf-8", + temp_path = paths["analysis_path"].with_name( + f".{paths['analysis_path'].name}.{os.getpid()}.{uuid4().hex}.part" ) + try: + temp_path.write_text(json.dumps(payload, ensure_ascii=False, indent=2), encoding="utf-8") + os.replace(temp_path, paths["analysis_path"]) + finally: + try: + temp_path.unlink(missing_ok=True) + except OSError: + pass + + +def _load_active_analysis_payload(task_id: str) -> dict: + with get_connection() as connection: + row = connection.execute( + """ + SELECT analysis_payload_json + FROM ai_analysis_runs + WHERE task_id = ? AND is_active = 1 + ORDER BY run_number DESC, created_at DESC + LIMIT 1 + """, + (task_id,), + ).fetchone() + if not row: + return {} + try: + payload = json.loads(str(row["analysis_payload_json"] or "{}")) + except (TypeError, json.JSONDecodeError): + return _invalid_analysis_payload(task_id, "active_run_json_invalid") + if not isinstance(payload, dict): + return _invalid_analysis_payload(task_id, "active_run_payload_invalid") + if not isinstance(payload.get("analysis_meta"), dict): + invalid = _invalid_analysis_payload(task_id, "active_run_meta_missing") + invalid["clips"] = payload.get("clips") if isinstance(payload.get("clips"), list) else [] + return invalid + return payload + + +def _materialize_analysis_payload(task_id: str, payload: dict | None = None) -> bool: + resolved_payload = payload or _load_active_analysis_payload(task_id) + if not resolved_payload: + return False + if resolved_payload.get(_ANALYSIS_INTEGRITY_MARKER): + return False + paths = get_artifact_paths(task_id) + try: + existing = json.loads(paths["analysis_path"].read_text(encoding="utf-8", errors="replace")) + except (OSError, json.JSONDecodeError): + existing = None + if existing == resolved_payload: + return True + _write_analysis_payload(task_id, resolved_payload) + return True + + +def ensure_task_ai_analysis_artifact(task_id: str) -> bool: + """从 active run 恢复派生 JSON;不调用 AI Provider。""" + return _materialize_analysis_payload(task_id) def restore_ai_analysis_run(task_id: str, run_id: str) -> dict: from app.services.task_service import ( + STATUS_PROGRESS, + _now_iso, get_task, # noqa: F811 list_clip_candidates, - update_task_status, ) task = get_task(task_id, include_video_probe=False) @@ -511,20 +1025,52 @@ def restore_ai_analysis_run(task_id: str, run_id: str) -> dict: except json.JSONDecodeError as exc: raise ValueError("这条历史记录已损坏,无法恢复") from exc - _replace_clip_candidates(task_id, payload.get("clips") or []) - _write_analysis_payload(task_id, payload) - # 切换 active 到被恢复的 run with get_connection() as connection: - connection.execute( - "UPDATE ai_analysis_runs SET is_active = 0 WHERE task_id = ?", - (task_id,), - ) - connection.execute( - "UPDATE ai_analysis_runs SET is_active = 1 WHERE id = ?", - (run_id,), - ) - connection.commit() - update_task_status(task_id, TaskStatus.pending_review) + connection.execute("BEGIN IMMEDIATE") + try: + now = _now_iso() + active_job = connection.execute( + """ + SELECT 1 FROM workflow_jobs + WHERE task_id = ? AND job_type IN ('ai_analysis', 'auto_pipeline') + AND status IN ('queued', 'running') + LIMIT 1 + """, + (task_id,), + ).fetchone() + if active_job: + raise AIAnalysisConflictError("AI 分析或全自动流水线正在运行,不能同时恢复历史结果。") + _replace_clip_candidates_with_connection( + connection, + task_id, + payload.get("clips") or [], + now, + ) + connection.execute("UPDATE ai_analysis_runs SET is_active = 0 WHERE task_id = ?", (task_id,)) + cursor = connection.execute( + "UPDATE ai_analysis_runs SET is_active = 1 WHERE id = ? AND task_id = ?", + (run_id, task_id), + ) + if cursor.rowcount != 1: + raise ValueError("没有找到这条 AI 分析历史") + connection.execute( + """ + UPDATE tasks + SET status = ?, progress = ?, error_message = NULL, last_error = NULL, updated_at = ? + WHERE id = ? AND COALESCE(is_deleted, 0) = 0 + """, + ( + TaskStatus.pending_review.value, + STATUS_PROGRESS[TaskStatus.pending_review.value], + now, + task_id, + ), + ) + connection.commit() + except Exception: + connection.rollback() + raise + _materialize_analysis_payload(task_id, payload) append_task_log(task_id, f"已恢复 AI 分析历史:第 {run['run_number']} 次分析") return { @@ -573,6 +1119,37 @@ def _analyze_with_provider(task_id: str, task: dict, paths: dict[str, Path], pro ) ) + if task.get("selection_profile") == "long_live_talk": + density = max(1, min(10, int(task.get("highlight_density_per_hour") or 4))) + total_limit = max(1, min(50, int(task.get("highlight_total_limit") or 30))) + append_task_log( + task_id, + "长直播高光:使用约 5 分钟、重叠 60 秒的可恢复窗口;" + f"每小时最多 {density} 条,总计最多 {total_limit} 条。", + ) + + def report_progress(progress: dict) -> None: + window_index = int(progress.get("window_index") or 0) + window_count = int(progress.get("window_count") or 0) + status = str(progress.get("status") or "") + if status in {"failed", "reused"} or window_index in {1, window_count} or window_index % 10 == 0: + label = {"failed": "失败", "reused": "复用", "completed": "完成"}.get(status, status) + append_task_log(task_id, f"长直播 AI 窗口 {window_index}/{window_count}:{label}") + + return analyze_long_live_talk( + LongLiveAnalysisRequest( + task_id=task_id, + transcript_path=paths["transcript_path"], + provider_name=provider_name, + model_name=_ai_model_name(provider_name), + density_per_hour=density, + total_limit=total_limit, + ai_preference=task.get("ai_preference") or "", + prompt_template=prompt_template, + ), + progress_callback=report_progress, + ) + request = AnalysisRequest( task_id=task_id, transcript_path=paths["transcript_path"], @@ -619,58 +1196,238 @@ def _append_ai_clip_quality_warnings(task_id: str, clips: list[dict]) -> None: ) +def _commit_ai_analysis_result( + *, + task_id: str, + analysis_payload: dict, + provider: str, + provider_label: str, + model: str, + fallback_notice: str, + prompt_preset: dict, + requested_clip_count: int, +) -> dict: + """把候选、active run 与 Task 终态绑定到当前 lease 的同一事务。""" + from app.services import job_service + from app.services.task_service import STATUS_PROGRESS, _now_iso + + now = _now_iso() + with get_connection() as connection: + connection.execute("BEGIN IMMEDIATE") + try: + job = job_service.require_job_lease_with_connection( + connection, + task_id=task_id, + allowed_job_types={job_service.JOB_TYPE_AI_ANALYSIS, job_service.JOB_TYPE_AUTO_PIPELINE}, + ) + current = connection.execute( + "SELECT status FROM tasks WHERE id = ? AND COALESCE(is_deleted, 0) = 0", + (task_id,), + ).fetchone() + if not current or str(current["status"] or "") != TaskStatus.ai_analyzing.value: + raise AIAnalysisConflictError("AI 分析提交前任务状态已改变,本次结果不会覆盖新状态。") + _assert_ai_task_can_start( + connection, + task_id, + str(current["status"] or ""), + current_job_id=str(job["id"]), + ) + _replace_clip_candidates_with_connection( + connection, + task_id, + analysis_payload.get("clips") or [], + now, + ) + run_id = _insert_ai_analysis_run_with_connection( + connection, + task_id=task_id, + analysis_payload=analysis_payload, + provider=provider, + provider_label=provider_label, + model=model, + fallback_notice=fallback_notice, + prompt_preset=prompt_preset, + requested_clip_count=requested_clip_count, + now=now, + ) + cursor = connection.execute( + """ + UPDATE tasks + SET status = ?, progress = ?, error_message = NULL, last_error = NULL, updated_at = ? + WHERE id = ? AND status = ? AND COALESCE(is_deleted, 0) = 0 + """, + ( + TaskStatus.pending_review.value, + STATUS_PROGRESS[TaskStatus.pending_review.value], + now, + task_id, + TaskStatus.ai_analyzing.value, + ), + ) + if cursor.rowcount != 1: + raise AIAnalysisConflictError("AI 分析提交时任务状态已改变,本次结果已回滚。") + connection.commit() + except Exception: + connection.rollback() + raise + return get_ai_analysis_run(task_id, run_id) + + +def _resume_committed_ai_analysis(task_id: str) -> tuple[dict, dict] | None: + """同一 Job 在 DB 已提交后重启时复用 active run,避免再次调用 Provider。""" + from app.services import job_service + + with get_connection() as connection: + job = job_service.require_job_lease_with_connection( + connection, + task_id=task_id, + allowed_job_types={job_service.JOB_TYPE_AI_ANALYSIS, job_service.JOB_TYPE_AUTO_PIPELINE}, + ) + row = connection.execute( + """ + SELECT * FROM ai_analysis_runs + WHERE task_id = ? AND is_active = 1 + ORDER BY run_number DESC, created_at DESC LIMIT 1 + """, + (task_id,), + ).fetchone() + task = connection.execute( + "SELECT status FROM tasks WHERE id = ? AND COALESCE(is_deleted, 0) = 0", + (task_id,), + ).fetchone() + if not row or not task or str(task["status"] or "") != TaskStatus.pending_review.value: + return None + run = _analysis_run_row_to_dict(row, include_payload=True) + meta = run.get("analysis_meta") if isinstance(run.get("analysis_meta"), dict) else {} + if str(meta.get("workflow_job_id") or "") != str(job["id"]): + return None + payload = { + "task_id": task_id, + "analysis_summary": run.get("analysis_summary") or "", + "clips": run.get("clips") or [], + "analysis_meta": meta, + } + try: + _materialize_analysis_payload(task_id, payload) + except OSError as exc: + append_task_log(task_id, f"AI 结果缓存暂未重建,数据库结果仍有效:{exc}") + append_task_log(task_id, f"已从 Workflow Job {job['id']} 的已提交 AI run 恢复,未再次调用 Provider") + return run, payload + + +def _build_ai_process_result( + task_id: str, + *, + analysis_run: dict, + analysis_payload: dict, + fallback_notice: str = "", +) -> dict: + from app.services.task_service import get_task, list_clip_candidates + + meta = analysis_payload.get("analysis_meta") if isinstance(analysis_payload, dict) else {} + meta = meta if isinstance(meta, dict) else {} + incomplete = bool(meta.get("analysis_incomplete")) + coverage = float(meta.get("coverage_percent") or 0) + profile = str(meta.get("selection_profile") or "general") + if incomplete: + message = ( + f"{profile} 分析覆盖率为 {coverage:.2f}%,存在未完成单元。" + "成功结果已保存供人工检查;补齐前不会进入自动切片或发布。" + ) + else: + message = ( + f"AI 分析完成,已生成 {len(analysis_payload.get('clips') or [])} 条可直接切片的候选片段," + "可进入片段审核检查或直接生成切片。" + ) + if fallback_notice: + message = f"{fallback_notice} {message}" + paths = get_artifact_paths(task_id) + return { + "status": "ok", + "message": message, + "provider": analysis_run.get("provider") or meta.get("provider") or "", + "provider_label": analysis_run.get("provider_label") or meta.get("provider_label") or "", + "model": analysis_run.get("model") or meta.get("model") or "", + "fallback_notice": fallback_notice, + "analysis_summary": analysis_payload.get("analysis_summary") or "", + "clip_summaries": _summarize_analysis_clips(analysis_payload.get("clips") or []), + "analysis_run_id": analysis_run["id"], + "analysis_run": analysis_run, + "runs": list_ai_analysis_runs(task_id), + "analysis_path": str(paths["analysis_path"]), + "review_url": f"/tasks/{task_id}/clips/review", + "task": get_task(task_id, include_video_probe=False), + "clips": list_clip_candidates(task_id), + } + + def process_task_ai_analysis(task_id: str, provider: str | None = None) -> dict: + from app.services import job_service from app.services.task_service import ( _now_iso, get_task, - list_clip_candidates, - update_task_status, ) task = get_task(task_id, include_video_probe=False) if not task: raise ValueError("任务不存在") + resumed = _resume_committed_ai_analysis(task_id) + if resumed: + resumed_run, resumed_payload = resumed + return _build_ai_process_result( + task_id, + analysis_run=resumed_run, + analysis_payload=resumed_payload, + fallback_notice=str(resumed_run.get("fallback_notice") or ""), + ) + job = _begin_ai_analysis(task_id, task) paths = get_artifact_paths(task_id) - if not paths["transcript_path"].exists(): - error = "请先生成带时间戳的转写 Markdown,再开始 AI 分析" - update_task_status(task_id, TaskStatus.failed, error) - append_task_log(task_id, f"AI 分析失败:{error}") - raise ValueError(error) - provider_name = (provider or settings.ai_default_provider).lower() - update_task_status(task_id, TaskStatus.ai_analyzing) append_task_log(task_id, f"开始 AI 片段分析,Provider:{provider_name}") used_provider = provider_name fallback_notice = "" try: + if not paths["transcript_path"].exists(): + raise AIAnalysisError("请先生成带时间戳的转写 Markdown,再开始 AI 分析") try: analysis = _analyze_with_provider(task_id, task, paths, provider_name) except Exception as provider_exc: - provider_error = str(provider_exc) + provider_error = ( + provider_exc.checkpoint_message() + if isinstance(provider_exc, AIProviderError) + else str(provider_exc) + ) if provider_name == "remote": raise AIAnalysisError( "远程 AI 分析接口不可用,已暂停 AI 分析:" f"{provider_error}。如需使用本地模型,请点击\"本地 AI 分析\"。" ) from provider_exc raise + long_live_meta = {} + if isinstance(analysis, LongLiveAnalysisOutcome): + long_live_meta = analysis.meta + analysis = analysis.result analysis_payload = result_to_jsonable(analysis) + analyzer_meta = analysis_payload.get("analysis_meta") + analyzer_meta = analyzer_meta if isinstance(analyzer_meta, dict) else {} analysis_payload["analysis_meta"] = { + **analyzer_meta, + "schema_version": int(analyzer_meta.get("schema_version") or 2), "provider": used_provider, "provider_label": _ai_provider_label(used_provider), "model": _ai_model_name(used_provider), "selection_profile": task.get("selection_profile") or "general", "final_clip_target": int(task.get("final_clip_target") or 5), "generated_at": _now_iso(), + "workflow_job_id": str(job["id"]), + **long_live_meta, } prompt_preset = get_task_ai_prompt_preset(task_id) provider_label = _ai_provider_label(used_provider) model_name = _ai_model_name(used_provider) - _replace_clip_candidates(task_id, analysis_payload["clips"]) - _write_analysis_payload(task_id, analysis_payload) - # 插入新的 AI 分析历史 run(自动标记 is_active=1,旧 run 取消激活) - analysis_run = _insert_ai_analysis_run( + analysis_run = _commit_ai_analysis_result( task_id=task_id, analysis_payload=analysis_payload, provider=used_provider, @@ -678,35 +1435,43 @@ def process_task_ai_analysis(task_id: str, provider: str | None = None) -> dict: model=model_name, fallback_notice=fallback_notice, prompt_preset=prompt_preset, - requested_clip_count=int(task["candidate_clip_count"]), + requested_clip_count=( + int(task.get("highlight_total_limit") or 30) + if task.get("selection_profile") == "long_live_talk" + else int(task["candidate_clip_count"]) + ), ) + try: + _materialize_analysis_payload(task_id, analysis_payload) + except OSError as exc: + append_task_log(task_id, f"AI 结果已原子落库,但派生缓存暂未写入:{exc}") _append_ai_clip_quality_warnings(task_id, analysis_payload["clips"]) - except (AIAnalysisError, Exception) as exc: + except job_service.JobLeaseLostError: + raise + except AIAnalysisConflictError: + _restore_task_after_ai_conflict(task_id, task) + append_task_log(task_id, "AI 分析已停止:任务已有下游切片引用,原任务状态和数据保持不变") + raise + except Exception as exc: error = str(exc) user_error = _summarize_ai_error(error) - update_task_status(task_id, TaskStatus.failed, user_error) + _mark_ai_failed_if_still_running(task_id, user_error) append_task_log(task_id, f"AI 分析失败:{error}") raise ValueError(user_error) from exc - update_task_status(task_id, TaskStatus.pending_review) - append_task_log(task_id, f"AI 分析完成,Provider:{used_provider},生成候选片段:{len(analysis_payload['clips'])} 条") - message = f"AI 分析完成,已生成 {len(analysis_payload['clips'])} 条可直接切片的候选片段,可进入片段审核检查或直接生成切片。" - if fallback_notice: - message = f"{fallback_notice} {message}" - return { - "status": "ok", - "message": message, - "provider": used_provider, - "provider_label": provider_label, - "model": model_name, - "fallback_notice": fallback_notice, - "analysis_summary": analysis_payload.get("analysis_summary") or "", - "clip_summaries": _summarize_analysis_clips(analysis_payload["clips"]), - "analysis_run_id": analysis_run["id"], - "analysis_run": analysis_run, - "runs": list_ai_analysis_runs(task_id), - "analysis_path": str(paths["analysis_path"]), - "review_url": f"/tasks/{task_id}/clips/review", - "task": get_task(task_id, include_video_probe=False), - "clips": list_clip_candidates(task_id), - } + incomplete = bool(analysis_payload.get("analysis_meta", {}).get("analysis_incomplete")) + if incomplete: + coverage = float(analysis_payload["analysis_meta"].get("coverage_percent") or 0) + profile = str(analysis_payload["analysis_meta"].get("selection_profile") or "general") + append_task_log( + task_id, + f"{profile} AI 分析不完整:覆盖率 {coverage:.2f}%,已保留成功结果,自动切片已锁定。", + ) + else: + append_task_log(task_id, f"AI 分析完成,Provider:{used_provider},生成候选片段:{len(analysis_payload['clips'])} 条") + return _build_ai_process_result( + task_id, + analysis_run=analysis_run, + analysis_payload=analysis_payload, + fallback_notice=fallback_notice, + ) diff --git a/app/services/ai_config_service.py b/app/services/ai_config_service.py index a9d0caf..fb4c464 100644 --- a/app/services/ai_config_service.py +++ b/app/services/ai_config_service.py @@ -87,6 +87,17 @@ REMOTE_PROTOCOL_OPTIONS = ["chat_completions", "responses"] TRANSCRIPTION_AUDIO_FORMAT_OPTIONS = ["mp3", "ogg"] +SECRET_SETTING_KEYS = frozenset( + { + "VOLCENGINE_ASR_API_KEY", + "VOLCENGINE_ASR_APP_KEY", + "VOLCENGINE_ASR_ACCESS_KEY", + "AI_ANALYSIS_REMOTE_API_KEY", + "AI_PUBLISH_REMOTE_API_KEY", + "AI_LOCAL_API_KEY", + } +) + ENV_APPEND_GROUPS = [ ( "# AI common switch", @@ -280,8 +291,21 @@ def _remote_ready(values: dict[str, str], prefix: str) -> bool: ) +def _public_config_values(values: dict[str, str]) -> dict[str, str]: + """只把非敏感配置送到浏览器;Secret 用单独布尔状态表达。""" + return { + key: "" if key in SECRET_SETTING_KEYS else value + for key, value in values.items() + } + + def get_ai_config_context() -> dict: values = _current_config_values() + public_values = _public_config_values(values) + secret_configured = { + key: bool(str(values.get(key) or "").strip()) + for key in SECRET_SETTING_KEYS + } transcription_ready = bool( values["TRANSCRIPTION_PROVIDER"] == "local" or ( @@ -316,9 +340,9 @@ def get_ai_config_context() -> dict: if values["AI_PUBLISH_PROVIDER"] == "local": publish_ready = local_ready return { - "env_path": str(_env_path()), "env_exists": _env_path().exists(), - "values": values, + "values": public_values, + "secret_configured": secret_configured, "transcription_ready": transcription_ready, "analysis_ready": analysis_ready, "analysis_key_valid": _key_valid(values["AI_ANALYSIS_REMOTE_API_KEY"]), diff --git a/app/services/auto_publish_service.py b/app/services/auto_publish_service.py index f7868ec..44d10f3 100644 --- a/app/services/auto_publish_service.py +++ b/app/services/auto_publish_service.py @@ -3,14 +3,22 @@ from __future__ import annotations import json +from pathlib import Path from uuid import uuid4 from app.core.config import settings from app.db.database import get_connection +from app.services import job_service from app.services.publish_copy_rules import PUBLISH_COPY_RULE_VERSION from app.services.publish_service import DEFAULT_BILIBILI_TID, USER_REMOVED_ERROR_CODE, get_publish_job from app.services.publish_domain import AUTO_PUBLISH_PLATFORMS, validate_publish_mode, validate_target_platform +from app.services.storage_service import ( + IMAGE_EXTENSIONS, + resolve_task_media_file_path, + resolve_video_file_path, +) from app.services.task_service import _now_iso +from app.services.transcription_checkpoint_service import fingerprint_file def platforms_for_task(task: dict) -> list[str]: @@ -18,22 +26,82 @@ def platforms_for_task(task: dict) -> list[str]: return list(AUTO_PUBLISH_PLATFORMS) -def create_auto_publish_jobs(task: dict, scheduled_items: list[dict]) -> dict: +def create_auto_publish_jobs( + task: dict, + scheduled_items: list[dict], + *, + subtitle_delivery_mode: str, + workflow_job_id: str | None = None, +) -> dict: """为全自动流水线生成发布任务。 本轮只创建任务记录,不调用平台 API,也不启动 opencli 发送。 """ + if subtitle_delivery_mode not in {"subtitled", "original"}: + raise ValueError("字幕交付模式必须是 subtitled 或 original") created_ids: list[str] = [] skipped_ids: list[str] = [] - now = _now_iso() + lease = job_service.current_job_lease() if workflow_job_id else None + if workflow_job_id and (not lease or lease[0] != workflow_job_id): + raise job_service.JobLeaseLostError(f"发布草稿创建缺少当前 Workflow Job 租约:{workflow_job_id}") with get_connection() as connection: + connection.execute("BEGIN IMMEDIATE") + now = _now_iso() + if workflow_job_id: + active_lease = connection.execute( + """ + SELECT 1 FROM workflow_jobs + WHERE id = ? AND status = 'running' AND lease_owner = ? AND lease_token = ? + AND lease_expires_at > strftime('%Y-%m-%dT%H:%M:%S+00:00', 'now') + AND cancel_requested = 0 + """, + (workflow_job_id, lease[1], lease[2]), + ).fetchone() + if not active_lease: + connection.rollback() + raise job_service.JobLeaseLostError( + f"发布草稿创建前 Workflow Job 租约已失效:{workflow_job_id}" + ) for item in scheduled_items: output_clip = item["output_clip"] + video_source, video_file_path, resolved_video, subtitle_evidence = _resolve_auto_video_source( + task, + output_clip, + subtitle_delivery_mode, + require_managed_path=bool(workflow_job_id), + ) metadata = item["metadata"] cover = item.get("cover") or {} platform = validate_target_platform(metadata["platform"]) publish_mode = validate_publish_mode(settings.publish_default_mode) + cover_file_path = str(cover.get("cover_file_path") or "").strip() + if not cover_file_path: + raise ValueError(f"{output_clip.get('id') or '未知切片'} 没有生成封面,已停止创建不完整的发布任务") + resolved_cover = ( + resolve_task_media_file_path( + cover_file_path, + task_id=str(task["id"]), + task_dir_name=str(task.get("task_dir_name") or "") or None, + allowed_subdirectories=("07_covers",), + allowed_extensions=IMAGE_EXTENSIONS, + ) + if workflow_job_id + else resolve_video_file_path(cover_file_path) + ) + if ( + resolved_cover is None + or not resolved_cover.exists() + or not resolved_cover.is_file() + or resolved_cover.stat().st_size <= 0 + ): + raise ValueError(f"{output_clip.get('id') or '未知切片'} 的封面文件无效或不在受控目录") + cover_file_path = str(resolved_cover) + scheduled_at = str(item.get("scheduled_at") or "").strip() + if scheduled_at: + from app.services.publish_time import to_utc_iso + + scheduled_at = to_utc_iso(scheduled_at, settings.app_timezone) latest = connection.execute( """ SELECT id, status, error_code @@ -51,9 +119,14 @@ def create_auto_publish_jobs(task: dict, scheduled_items: list[dict]) -> dict: ): skipped_ids.append(latest["id"]) continue + if latest and str(latest["status"] or "").upper() in {"PUBLISHED", "EXPORTED"}: + skipped_ids.append(latest["id"]) + continue existing = connection.execute( """ - SELECT id + SELECT id, video_file_path, cover_file_path, provider_response, + title, description, caption, tags, hashtags, cover_text, + scheduled_at, cover_time_seconds FROM publish_jobs WHERE output_clip_id = ? AND platform = ? AND publish_mode = ? AND status IN ('DRAFT', 'WAITING', 'SCHEDULED', 'PUBLISHING', 'NEED_REVIEW') @@ -63,12 +136,63 @@ def create_auto_publish_jobs(task: dict, scheduled_items: list[dict]) -> dict: (output_clip["id"], platform, publish_mode), ).fetchone() if existing: + if workflow_job_id: + try: + evidence = json.loads(str(existing["provider_response"] or "{}")) + except json.JSONDecodeError as exc: + raise ValueError("已有发布草稿证据已损坏,拒绝复用") from exc + expected_caption = str(metadata.get("caption") or "") + expected_hashtags = ", ".join(metadata.get("hashtags") or []) + existing_video = resolve_task_media_file_path( + str(existing["video_file_path"] or ""), + task_id=str(task["id"]), + task_dir_name=str(task.get("task_dir_name") or "") or None, + allowed_subdirectories=( + ("06_subtitled",) + if video_source == "subtitled" + else ("05_clips", "clips") + ), + ) + existing_cover = resolve_task_media_file_path( + str(existing["cover_file_path"] or ""), + task_id=str(task["id"]), + task_dir_name=str(task.get("task_dir_name") or "") or None, + allowed_subdirectories=("07_covers",), + allowed_extensions=IMAGE_EXTENSIONS, + ) + if ( + not isinstance(evidence, dict) + or existing_video is None + or existing_cover is None + or not existing_video.is_file() + or not existing_cover.is_file() + or existing_video.resolve() != resolved_video.resolve() + or existing_cover.resolve() != resolved_cover.resolve() + or int(evidence.get("video_file_size") or -1) != existing_video.stat().st_size + or str(evidence.get("video_file_fingerprint") or "") + != fingerprint_file(existing_video) + or int(evidence.get("cover_file_size") or -1) != existing_cover.stat().st_size + or str(evidence.get("cover_file_fingerprint") or "") + != fingerprint_file(existing_cover) + or str(evidence.get("metadata_policy_version") or "") + != PUBLISH_COPY_RULE_VERSION + or str(evidence.get("subtitle_delivery_mode") or "") + != subtitle_delivery_mode + or str(existing["title"] or "") + != str(metadata.get("title") or "精彩片段") + or str(existing["description"] or "") != expected_caption + or str(existing["caption"] or "") != expected_caption + or str(existing["tags"] or "") != expected_hashtags + or str(existing["hashtags"] or "") != expected_hashtags + or str(existing["cover_text"] or "") + != str(metadata.get("cover_text") or "") + or str(existing["scheduled_at"] or "") != scheduled_at + or float(existing["cover_time_seconds"] or 0) + != float(cover.get("cover_time_seconds") or 0) + ): + raise ValueError("已有发布草稿的媒体或文案证据已失效,拒绝复用或重复创建") skipped_ids.append(existing["id"]) continue - - cover_file_path = str(cover.get("cover_file_path") or "").strip() - if not cover_file_path: - raise ValueError(f"{output_clip.get('id') or '未知切片'} 没有生成封面,已停止创建不完整的发布任务") cover_time_seconds = float(cover.get("cover_time_seconds") or 0) account_rows = connection.execute( """ @@ -83,17 +207,13 @@ def create_auto_publish_jobs(task: dict, scheduled_items: list[dict]) -> dict: if len(account_rows) == 1 and str(account_rows[0]["login_status"] or "") == "normal" else None ) - scheduled_at = str(item.get("scheduled_at") or "").strip() - if scheduled_at: - from app.services.publish_time import to_utc_iso - - scheduled_at = to_utc_iso(scheduled_at, settings.app_timezone) status = "NEED_REVIEW" if metadata.get("risk_flags") else ( "SCHEDULED" if scheduled_at and (publish_mode != "local_browser" or account_id) else "WAITING" ) job_id = uuid4().hex[:12] provider_response = { "source": "auto_pipeline", + "workflow_job_id": workflow_job_id or "", "target_platform": platform, "metadata_source": metadata.get("source") or "", "metadata_error": metadata.get("error") or "", @@ -104,6 +224,12 @@ def create_auto_publish_jobs(task: dict, scheduled_items: list[dict]) -> dict: "cover_time_seconds": cover_time_seconds, "risk_flags": metadata.get("risk_flags") or [], "publish_mode": publish_mode, + "subtitle_delivery_mode": subtitle_delivery_mode, + "video_file_size": int(resolved_video.stat().st_size), + "video_file_fingerprint": fingerprint_file(resolved_video), + "cover_file_size": int(resolved_cover.stat().st_size), + "cover_file_fingerprint": fingerprint_file(resolved_cover), + **subtitle_evidence, "note": "全自动流水线已直接创建最终发布任务,可在发送中心设置排期。", } connection.execute( @@ -117,7 +243,7 @@ def create_auto_publish_jobs(task: dict, scheduled_items: list[dict]) -> dict: schedule_timezone, timezone, status, audit_status, error_message, last_error, provider_response, publish_result, max_attempts, created_at, updated_at ) - VALUES (?, ?, ?, ?, ?, ?, ?, 'original', ?, ?, ?, ?, ?, ?, ?, ?, ?, 'public', + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, 'public', 'time', ?, 1, ?, 'original', '', ?, ?, ?, ?, ?, 'not_submitted', '', '', ?, '', ?, ?, ?) """, ( @@ -128,8 +254,9 @@ def create_auto_publish_jobs(task: dict, scheduled_items: list[dict]) -> dict: account_id, platform, publish_mode, - output_clip.get("output_file_path") or "", - output_clip.get("output_file_path") or "", + video_source, + video_file_path, + video_file_path, metadata.get("title") or "精彩片段", metadata.get("caption") or "", metadata.get("caption") or "", @@ -151,6 +278,21 @@ def create_auto_publish_jobs(task: dict, scheduled_items: list[dict]) -> dict: ), ) created_ids.append(job_id) + if workflow_job_id: + active_lease = connection.execute( + """ + SELECT 1 FROM workflow_jobs + WHERE id = ? AND status = 'running' AND lease_owner = ? AND lease_token = ? + AND lease_expires_at > strftime('%Y-%m-%dT%H:%M:%S+00:00', 'now') + AND cancel_requested = 0 + """, + (workflow_job_id, lease[1], lease[2]), + ).fetchone() + if not active_lease: + connection.rollback() + raise job_service.JobLeaseLostError( + f"发布草稿提交前 Workflow Job 租约已失效:{workflow_job_id}" + ) connection.commit() return { @@ -159,3 +301,60 @@ def create_auto_publish_jobs(task: dict, scheduled_items: list[dict]) -> dict: "created_count": len(created_ids), "skipped_count": len(skipped_ids), } + + +def _resolve_auto_video_source( + task: dict, + output_clip: dict, + delivery_mode: str, + *, + require_managed_path: bool, +) -> tuple[str, str, Path, dict]: + if delivery_mode == "original": + raw_path = str(output_clip.get("output_file_path") or "") + path = ( + resolve_task_media_file_path( + raw_path, + task_id=str(task["id"]), + task_dir_name=str(task.get("task_dir_name") or "") or None, + allowed_subdirectories=("05_clips", "clips"), + ) + if require_managed_path + else (resolve_video_file_path(raw_path) if raw_path else None) + ) + if not path or not path.exists() or not path.is_file() or path.stat().st_size <= 0: + raise ValueError("原片切片文件不存在,不能创建发布任务") + return "original", str(path), path, {"subtitle_skip_confirmed": True} + + raw_path = str(output_clip.get("subtitled_output_file_path") or "") + path = ( + resolve_task_media_file_path( + raw_path, + task_id=str(task["id"]), + task_dir_name=str(task.get("task_dir_name") or "") or None, + allowed_subdirectories=("06_subtitled",), + ) + if require_managed_path + else (resolve_video_file_path(raw_path) if raw_path else None) + ) + if ( + output_clip.get("subtitle_status") != "completed" + or output_clip.get("subtitle_validation_status") != "verified" + or output_clip.get("subtitle_revision_status") != "approved" + or not path + or not path.exists() + or not path.is_file() + or path.stat().st_size <= 0 + ): + raise ValueError("字幕成片尚未同时通过 revision 审核和 FFprobe 验证,不能进入发送中心") + return ( + "subtitled", + str(path), + path, + { + "subtitle_revision_id": output_clip.get("subtitle_revision_id") or "", + "subtitle_revision_status": output_clip.get("subtitle_revision_status") or "", + "subtitle_validation_status": output_clip.get("subtitle_validation_status") or "", + "subtitle_verified_at": output_clip.get("subtitle_verified_at") or "", + }, + ) diff --git a/app/services/database_backup_service.py b/app/services/database_backup_service.py index f868c51..68d1c99 100644 --- a/app/services/database_backup_service.py +++ b/app/services/database_backup_service.py @@ -46,6 +46,16 @@ class BackupCleanupResult: released_bytes: int +def _finalize_portable_backup(connection: sqlite3.Connection) -> None: + """把 Online Backup 结果固定为无需 WAL/SHM sidecar 的单文件快照。""" + connection.commit() + row = connection.execute("PRAGMA journal_mode = DELETE").fetchone() + journal_mode = str(row[0]).lower() if row else "" + if journal_mode != "delete": + raise BackupSafetyError(f"备份无法切换为单文件 journal_mode:{journal_mode or 'unknown'}") + connection.commit() + + def sqlite_quick_check(database_path: Path) -> str: path = database_path.resolve() if not path.is_file(): @@ -235,6 +245,7 @@ def create_publish_migration_backup( ) backup_connection = sqlite3.connect(str(temporary_path), timeout=10) source_connection.backup(backup_connection) + _finalize_portable_backup(backup_connection) backup_connection.close() backup_connection = None source_connection.close() @@ -265,6 +276,44 @@ def create_publish_migration_backup( source_connection.close() +def create_schema_migration_backup(database_path: Path, backup_dir: Path, label: str) -> Path: + """使用 SQLite Online Backup API 创建一次带完整性检查的结构迁移备份。""" + database_path = database_path.resolve() + backup_dir = backup_dir.resolve() + backup_dir.mkdir(parents=True, exist_ok=True) + safe_label = "".join(character for character in label.lower() if character.isalnum() or character == "-") + timestamp = datetime.now(BACKUP_TIMEZONE).strftime("%Y%m%d-%H%M%S-%f") + final_path = backup_dir / f"workflow-before-{safe_label}-{timestamp}-{uuid4().hex[:8]}.sqlite3" + temporary_path = final_path.with_suffix(final_path.suffix + ".tmp") + source_connection: sqlite3.Connection | None = None + backup_connection: sqlite3.Connection | None = None + try: + source_connection = sqlite3.connect(f"{database_path.as_uri()}?mode=ro", uri=True, timeout=10) + backup_connection = sqlite3.connect(str(temporary_path), timeout=10) + source_connection.backup(backup_connection) + _finalize_portable_backup(backup_connection) + backup_connection.close() + backup_connection = None + source_connection.close() + source_connection = None + integrity = sqlite_quick_check(temporary_path) + if integrity != "ok": + raise BackupSafetyError(f"新备份完整性检查失败:{integrity}") + os.replace(temporary_path, final_path) + return final_path + except Exception as exc: + if temporary_path.exists(): + temporary_path.unlink() + if isinstance(exc, BackupSafetyError): + raise + raise BackupSafetyError(f"创建结构迁移前备份失败:{exc}") from exc + finally: + if backup_connection is not None: + backup_connection.close() + if source_connection is not None: + source_connection.close() + + def create_media_cleanup_backup( database_path: Path, backup_dir: Path, @@ -292,6 +341,7 @@ def create_media_cleanup_backup( ) backup_connection = sqlite3.connect(str(temporary_path), timeout=10) source_connection.backup(backup_connection) + _finalize_portable_backup(backup_connection) backup_connection.close() backup_connection = None source_connection.close() diff --git a/app/services/job_service.py b/app/services/job_service.py index 763c7ca..b0ecb4b 100644 --- a/app/services/job_service.py +++ b/app/services/job_service.py @@ -5,7 +5,10 @@ """ import json -from datetime import datetime, timezone +from collections.abc import Iterator +from contextlib import contextmanager +from contextvars import ContextVar +from datetime import datetime, timedelta, timezone from typing import Optional from uuid import uuid4 @@ -17,6 +20,7 @@ JOB_TYPE_TRANSCRIPT = "transcript" JOB_TYPE_SUBTITLE = "subtitle" JOB_TYPE_PUBLISH = "publish" +JOB_TYPE_AUTO_PIPELINE = "auto_pipeline" # ── job 状态枚举 ───────────────────────────────────────────────── JOB_STATUS_QUEUED = "queued" @@ -39,9 +43,119 @@ JOB_TYPE_TRANSCRIPT: "转写", JOB_TYPE_SUBTITLE: "字幕", JOB_TYPE_PUBLISH: "发布", + JOB_TYPE_AUTO_PIPELINE: "全自动流水线", } +class JobLeaseLostError(RuntimeError): + """当前执行已失去 Workflow Job 的 claim 代际。""" + + +_active_job_lease: ContextVar[tuple[str, str, str] | None] = ContextVar( + "active_workflow_job_lease", + default=None, +) + + +@contextmanager +def job_lease_context(job_id: str, lease_owner: str, lease_token: str) -> Iterator[None]: + """让同一执行链中的深层进度/checkpoint 写回自动携带租约代际。""" + if not job_id or not lease_owner or not lease_token: + raise ValueError("Workflow Job 租约缺少 job_id、owner 或 token") + token = _active_job_lease.set((job_id, lease_owner, lease_token)) + try: + yield + finally: + _active_job_lease.reset(token) + + +def _resolve_job_lease( + lease_owner: str | None = None, + lease_token: str | None = None, +) -> tuple[str, str] | None: + if lease_owner is not None or lease_token is not None: + if not lease_owner or not lease_token: + raise ValueError("Workflow Job 租约必须同时提供 owner 和 token") + return lease_owner, lease_token + active = _active_job_lease.get() + return (active[1], active[2]) if active else None + + +def require_active_job_lease() -> tuple[str, str, str] | None: + """深层持久化副作用执行前确认当前 ContextVar 仍属于有效 claim。""" + active = _active_job_lease.get() + if active is None: + return None + job_id, lease_owner, lease_token = active + if not validate_job_lease(job_id, lease_owner, lease_token): + raise JobLeaseLostError(f"Workflow Job 租约已失效:{job_id}") + return active + + +def current_job_lease() -> tuple[str, str, str] | None: + """供同一 SQLite 事务把业务写入与 Job claim 条件绑定。""" + return _active_job_lease.get() + + +def require_job_lease_with_connection( + connection, + *, + task_id: str, + allowed_job_types: set[str] | frozenset[str], +) -> dict: + """在业务事务使用的同一连接内验证当前 Job claim,防止 TOCTOU。""" + active = _active_job_lease.get() + if active is None: + raise JobLeaseLostError("AI 处理必须由持久化 Workflow Job 执行") + job_id, lease_owner, lease_token = active + placeholders = ", ".join("?" for _ in allowed_job_types) + row = connection.execute( + f""" + SELECT * + FROM workflow_jobs + WHERE id = ? AND task_id = ? AND job_type IN ({placeholders}) + AND status = ? AND lease_owner = ? AND lease_token = ? + AND cancel_requested = 0 + AND lease_expires_at > strftime('%Y-%m-%dT%H:%M:%S+00:00', 'now') + """, + ( + job_id, + task_id, + *sorted(allowed_job_types), + JOB_STATUS_RUNNING, + lease_owner, + lease_token, + ), + ).fetchone() + if not row: + raise JobLeaseLostError(f"Workflow Job 租约已失效:{job_id}") + return _row_to_dict(row) + + +def _lease_write_condition( + lease_owner: str | None = None, + lease_token: str | None = None, +) -> tuple[str, tuple[str, ...], bool]: + lease = _resolve_job_lease(lease_owner, lease_token) + if lease: + return ( + "id = ? AND status = ? AND lease_owner = ? AND lease_token = ? " + "AND lease_expires_at > strftime('%Y-%m-%dT%H:%M:%S+00:00', 'now')", + (JOB_STATUS_RUNNING, lease[0], lease[1]), + True, + ) + return ( + "id = ? AND status = ? AND lease_owner IS NULL AND lease_token IS NULL", + (JOB_STATUS_RUNNING,), + False, + ) + + +def _raise_if_lease_lost(job_id: str, rowcount: int, fenced: bool) -> None: + if fenced and rowcount == 0: + raise JobLeaseLostError(f"Workflow Job 租约已失效:{job_id}") + + def _now_iso() -> str: return datetime.now(timezone.utc).isoformat(timespec="seconds") @@ -49,7 +163,7 @@ def _now_iso() -> str: def _row_to_dict(row) -> dict: """将 sqlite3.Row 转为普通字典,并解析 JSON 字段""" job = dict(row) - for field in ("payload_json", "result_json"): + for field in ("payload_json", "result_json", "checkpoint_json"): raw = job.get(field) if isinstance(raw, str) and raw: try: @@ -110,56 +224,69 @@ def create_or_get_active_job( 返回值中的布尔值表示是否新建。`BEGIN IMMEDIATE` 将“查询 + 新建” 串行化,避免连续点击或并发请求为同一任务创建重复的切片作业。 """ - resolved_job_id = uuid4().hex[:12] - now = _now_iso() - payload_json = json.dumps(payload or {}, ensure_ascii=False) - with get_connection() as connection: connection.execute("BEGIN IMMEDIATE") - existing = connection.execute( - """ - SELECT id - FROM workflow_jobs - WHERE task_id = ? - AND job_type = ? - AND status IN (?, ?) - ORDER BY created_at DESC - LIMIT 1 - """, - ( - task_id, - job_type, - JOB_STATUS_QUEUED, - JOB_STATUS_RUNNING, - ), - ).fetchone() - if existing: - connection.commit() - return get_job(existing["id"]), False - - connection.execute( - """ - INSERT INTO workflow_jobs ( - id, task_id, job_type, status, progress, message, - payload_json, result_json, error_message, - created_at, updated_at, started_at, finished_at - ) - VALUES (?, ?, ?, ?, 0, ?, ?, '{}', NULL, ?, ?, NULL, NULL) - """, - ( - resolved_job_id, - task_id, - job_type, - JOB_STATUS_QUEUED, - f"{JOB_TYPE_LABELS.get(job_type, job_type)}任务已加入队列", - payload_json, - now, - now, - ), + resolved_job_id, created = create_or_get_active_job_with_connection( + connection, + task_id=task_id, + job_type=job_type, + payload=payload, ) connection.commit() + return get_job(resolved_job_id), created + + +def create_or_get_active_job_with_connection( + connection, + *, + task_id: str, + job_type: str, + payload: Optional[dict] = None, +) -> tuple[str, bool]: + """在调用方事务中原子复用或创建活动 Job,不自行 commit。""" + existing = connection.execute( + """ + SELECT id + FROM workflow_jobs + WHERE task_id = ? + AND job_type = ? + AND status IN (?, ?) + ORDER BY created_at DESC + LIMIT 1 + """, + ( + task_id, + job_type, + JOB_STATUS_QUEUED, + JOB_STATUS_RUNNING, + ), + ).fetchone() + if existing: + return str(existing["id"]), False - return get_job(resolved_job_id), True + resolved_job_id = uuid4().hex[:12] + now = _now_iso() + connection.execute( + """ + INSERT INTO workflow_jobs ( + id, task_id, job_type, status, progress, message, + payload_json, result_json, error_message, + created_at, updated_at, started_at, finished_at + ) + VALUES (?, ?, ?, ?, 0, ?, ?, '{}', NULL, ?, ?, NULL, NULL) + """, + ( + resolved_job_id, + task_id, + job_type, + JOB_STATUS_QUEUED, + f"{JOB_TYPE_LABELS.get(job_type, job_type)}任务已加入队列", + json.dumps(payload or {}, ensure_ascii=False), + now, + now, + ), + ) + return resolved_job_id, True # ── 查询 job ───────────────────────────────────────────────────── @@ -222,82 +349,823 @@ def list_jobs(task_id: Optional[str] = None, status: Optional[str] = None) -> li # ── 状态流转 ───────────────────────────────────────────────────── def mark_job_running(job_id: str) -> dict | None: - """将 job 标记为 running""" - now = _now_iso() + """兼容入口:仍通过正式 claim 生成 token,不再制造无代际 running job。""" + return claim_job(job_id, f"legacy:{uuid4().hex}") + + +def claim_job(job_id: str, lease_owner: str, lease_seconds: int = 120) -> dict | None: + """原子领取一个排队任务,或接管 lease 已过期的运行任务。""" + lease_token = uuid4().hex with get_connection() as connection: - cursor = connection.execute( + connection.execute("BEGIN IMMEDIATE") + now = datetime.now(timezone.utc) + now_iso = now.isoformat(timespec="seconds") + lease_expires_at = (now + timedelta(seconds=max(30, lease_seconds))).isoformat(timespec="seconds") + row = connection.execute( + "SELECT status, lease_expires_at, cancel_requested, attempt_count, max_attempts FROM workflow_jobs WHERE id = ?", + (job_id,), + ).fetchone() + if not row: + connection.commit() + return None + expired = row["status"] == JOB_STATUS_RUNNING and ( + not row["lease_expires_at"] or str(row["lease_expires_at"]) <= now_iso + ) + claimable = row["status"] == JOB_STATUS_QUEUED or expired + if not claimable or int(row["cancel_requested"] or 0) or int(row["attempt_count"] or 0) >= int(row["max_attempts"] or 3): + connection.commit() + return None + connection.execute( """ UPDATE workflow_jobs - SET status = ?, progress = 10, message = '任务已开始执行', - started_at = ?, updated_at = ? + SET status = ?, progress = CASE WHEN progress < 10 THEN 10 ELSE progress END, + message = '任务已开始执行', started_at = COALESCE(started_at, ?), + updated_at = ?, heartbeat_at = ?, lease_owner = ?, lease_token = ?, lease_expires_at = ?, + attempt_count = attempt_count + 1 WHERE id = ? """, - (JOB_STATUS_RUNNING, now, now, job_id), + ( + JOB_STATUS_RUNNING, + now_iso, + now_iso, + now_iso, + lease_owner, + lease_token, + lease_expires_at, + job_id, + ), + ) + connection.commit() + return get_job(job_id) + + +def claim_next_job(lease_owner: str, lease_seconds: int = 120) -> dict | None: + """按创建时间领取一个重型任务,保证本地默认串行。""" + lease_token = uuid4().hex + with get_connection() as connection: + connection.execute("BEGIN IMMEDIATE") + now_value = datetime.now(timezone.utc) + now = now_value.isoformat(timespec="seconds") + lease_expires_at = (now_value + timedelta(seconds=max(30, lease_seconds))).isoformat(timespec="seconds") + connection.execute( + """ + UPDATE workflow_jobs + SET status = ?, message = '任务已取消,过期执行已回收', + finished_at = ?, updated_at = ?, lease_owner = NULL, + lease_token = NULL, lease_expires_at = NULL, heartbeat_at = NULL + WHERE status = ? AND cancel_requested = 1 + AND (lease_expires_at IS NULL OR lease_expires_at <= ?) + """, + (JOB_STATUS_CANCELLED, now, now, JOB_STATUS_RUNNING, now), ) + connection.execute( + """ + UPDATE workflow_jobs SET status = ?, error_message = '已达到最大尝试次数', + message = '任务失败:已达到最大尝试次数', finished_at = ?, updated_at = ?, + lease_owner = NULL, lease_token = NULL, lease_expires_at = NULL + WHERE attempt_count >= max_attempts + AND ( + status = ? + OR (status = ? AND (lease_expires_at IS NULL OR lease_expires_at <= ?)) + ) + """, + (JOB_STATUS_FAILED, now, now, JOB_STATUS_QUEUED, JOB_STATUS_RUNNING, now), + ) + row = connection.execute( + """ + SELECT id FROM workflow_jobs + WHERE cancel_requested = 0 + AND attempt_count < max_attempts + AND ( + (status = ? AND (next_attempt_at IS NULL OR next_attempt_at <= ?)) + OR (status = ? AND (lease_expires_at IS NULL OR lease_expires_at <= ?)) + ) + ORDER BY created_at ASC + LIMIT 1 + """, + (JOB_STATUS_QUEUED, now, JOB_STATUS_RUNNING, now), + ).fetchone() + if row: + cursor = connection.execute( + """ + UPDATE workflow_jobs + SET status = ?, progress = CASE WHEN progress < 10 THEN 10 ELSE progress END, + message = '任务已开始执行', started_at = COALESCE(started_at, ?), + updated_at = ?, heartbeat_at = ?, lease_owner = ?, lease_token = ?, + lease_expires_at = ?, attempt_count = attempt_count + 1 + WHERE id = ? AND attempt_count < max_attempts AND cancel_requested = 0 + AND ( + status = ? + OR (status = ? AND (lease_expires_at IS NULL OR lease_expires_at <= ?)) + ) + """, + ( + JOB_STATUS_RUNNING, + now, + now, + now, + lease_owner, + lease_token, + lease_expires_at, + row["id"], + JOB_STATUS_QUEUED, + JOB_STATUS_RUNNING, + now, + ), + ) + if cursor.rowcount != 1: + row = None connection.commit() + return get_job(row["id"]) if row else None + +def validate_job_lease( + job_id: str, + lease_owner: str, + lease_token: str, + *, + require_unexpired: bool = True, +) -> dict | None: + expiry_clause = ( + "AND lease_expires_at > strftime('%Y-%m-%dT%H:%M:%S+00:00', 'now')" + if require_unexpired + else "" + ) + params: tuple[str, ...] = ( + job_id, + JOB_STATUS_RUNNING, + lease_owner, + lease_token, + ) + with get_connection() as connection: + row = connection.execute( + f""" + SELECT * FROM workflow_jobs + WHERE id = ? AND status = ? AND lease_owner = ? AND lease_token = ? + {expiry_clause} + """, + params, + ).fetchone() + return _row_to_dict(row) if row else None + + +def heartbeat_job( + job_id: str, + lease_owner: str | None = None, + lease_token: str | None = None, + lease_seconds: int = 120, +) -> bool: + now = datetime.now(timezone.utc) + now_iso = now.isoformat(timespec="seconds") + expires = (now + timedelta(seconds=max(30, lease_seconds))).isoformat(timespec="seconds") + lease = _resolve_job_lease(lease_owner, lease_token) + if lease is None: + raise ValueError("heartbeat_job 需要有效的 Workflow Job 租约") + with get_connection() as connection: + cursor = connection.execute( + """ + UPDATE workflow_jobs SET heartbeat_at = ?, lease_expires_at = ?, updated_at = ? + WHERE id = ? AND status = ? AND lease_owner = ? AND lease_token = ? + AND lease_expires_at > strftime('%Y-%m-%dT%H:%M:%S+00:00', 'now') + """, + (now_iso, expires, now_iso, job_id, JOB_STATUS_RUNNING, lease[0], lease[1]), + ) + connection.commit() if cursor.rowcount == 0: - return None + raise JobLeaseLostError(f"Workflow Job 租约已失效:{job_id}") + return True + + +def update_job_checkpoint( + job_id: str, + checkpoint: dict, + *, + lease_owner: str | None = None, + lease_token: str | None = None, +) -> dict | None: + now = _now_iso() + condition, lease_params, fenced = _lease_write_condition(lease_owner, lease_token) + with get_connection() as connection: + connection.execute("BEGIN IMMEDIATE") + current = connection.execute( + "SELECT checkpoint_json FROM workflow_jobs WHERE id = ?", + (job_id,), + ).fetchone() + merged_checkpoint = dict(checkpoint) + if current: + try: + current_checkpoint = json.loads(str(current["checkpoint_json"] or "{}")) + except json.JSONDecodeError as exc: + connection.rollback() + raise ValueError("Workflow Job checkpoint_json 已损坏,拒绝覆盖 AI 恢复证据") from exc + if isinstance(current_checkpoint, dict) and "_ai_analysis_units_v1" in current_checkpoint: + merged_checkpoint.setdefault( + "_ai_analysis_units_v1", + current_checkpoint["_ai_analysis_units_v1"], + ) + cursor = connection.execute( + f"UPDATE workflow_jobs SET checkpoint_json = ?, checkpoint_updated_at = ?, updated_at = ? WHERE {condition}", + (json.dumps(merged_checkpoint, ensure_ascii=False), now, now, job_id, *lease_params), + ) + connection.commit() + _raise_if_lease_lost(job_id, cursor.rowcount, fenced) + return get_job(job_id) if cursor.rowcount else None + + +def request_job_cancel(job_id: str) -> dict | None: + now = _now_iso() + with get_connection() as connection: + connection.execute("BEGIN IMMEDIATE") + row = connection.execute( + "SELECT status, job_type, task_id FROM workflow_jobs WHERE id = ?", + (job_id,), + ).fetchone() + if not row: + connection.rollback() + return None + if row["status"] == JOB_STATUS_QUEUED: + cursor = connection.execute( + """UPDATE workflow_jobs SET status = ?, cancel_requested = 1, message = '任务已取消', + finished_at = ?, updated_at = ?, lease_owner = NULL, lease_token = NULL, + lease_expires_at = NULL WHERE id = ? AND status = ?""", + (JOB_STATUS_CANCELLED, now, now, job_id, JOB_STATUS_QUEUED), + ) + elif row["status"] == JOB_STATUS_RUNNING: + cursor = connection.execute( + """ + UPDATE workflow_jobs + SET cancel_requested = 1, message = '正在停止任务', updated_at = ? + WHERE id = ? AND status = ? + """, + (now, job_id, JOB_STATUS_RUNNING), + ) + else: + cursor = None + if cursor and cursor.rowcount == 1 and row["job_type"] == JOB_TYPE_AUTO_PIPELINE: + _cancel_linked_auto_publish_jobs( + connection, + task_id=str(row["task_id"] or ""), + workflow_job_ids=[job_id], + now=now, + ) + connection.execute( + """ + UPDATE tasks + SET status = 'CANCELLED', progress = 0, + error_message = '用户已取消全自动流水线', + last_error = '用户已取消全自动流水线', updated_at = ? + WHERE id = ? AND COALESCE(is_deleted, 0) = 0 + """, + (now, str(row["task_id"] or "")), + ) + elif cursor and cursor.rowcount == 1 and row["job_type"] == JOB_TYPE_AI_ANALYSIS: + _sync_ai_task_state( + connection, + task_id=str(row["task_id"] or ""), + outcome="cancelled", + message="用户已取消 AI 分析", + now=now, + ) + connection.commit() return get_job(job_id) -def update_job_progress(job_id: str, progress: int, message: Optional[str] = None) -> dict | None: +def cancel_active_auto_pipeline_jobs_for_task( + connection, + task_id: str, + *, + now: str, +) -> int: + """在调用方事务中取消一个 Task 的活跃自动流水线及其未发布结果。""" + rows = connection.execute( + """ + SELECT id, status + FROM workflow_jobs + WHERE task_id = ? AND job_type = ? AND status IN (?, ?) + """, + (task_id, JOB_TYPE_AUTO_PIPELINE, JOB_STATUS_QUEUED, JOB_STATUS_RUNNING), + ).fetchall() + if not rows: + return 0 + + queued_ids = [str(row["id"]) for row in rows if row["status"] == JOB_STATUS_QUEUED] + running_ids = [str(row["id"]) for row in rows if row["status"] == JOB_STATUS_RUNNING] + if queued_ids: + placeholders = ", ".join("?" for _ in queued_ids) + connection.execute( + f""" + UPDATE workflow_jobs + SET status = ?, cancel_requested = 1, message = '任务已取消', + finished_at = ?, updated_at = ?, lease_owner = NULL, + lease_token = NULL, lease_expires_at = NULL, heartbeat_at = NULL + WHERE id IN ({placeholders}) AND status = ? + """, + (JOB_STATUS_CANCELLED, now, now, *queued_ids, JOB_STATUS_QUEUED), + ) + if running_ids: + placeholders = ", ".join("?" for _ in running_ids) + connection.execute( + f""" + UPDATE workflow_jobs + SET cancel_requested = 1, message = '正在停止任务', updated_at = ? + WHERE id IN ({placeholders}) AND status = ? + """, + (now, *running_ids, JOB_STATUS_RUNNING), + ) + _cancel_linked_auto_publish_jobs( + connection, + task_id=task_id, + workflow_job_ids=[str(row["id"]) for row in rows], + now=now, + ) + return len(rows) + + +def _cancel_linked_auto_publish_jobs( + connection, + *, + task_id: str, + workflow_job_ids: list[str], + now: str, +) -> int: + """只取消 provider_response 明确归属于指定流水线的未发布任务。""" + resolved_ids = [job_id for job_id in workflow_job_ids if job_id] + if not task_id or not resolved_ids: + return 0 + candidates = connection.execute( + """ + SELECT id, provider_response + FROM publish_jobs + WHERE task_id = ? AND status IN ('DRAFT', 'WAITING', 'SCHEDULED', 'NEED_REVIEW') + """, + (task_id,), + ).fetchall() + linked_ids: list[str] = [] + allowed_workflow_ids = set(resolved_ids) + for row in candidates: + try: + provider_response = json.loads(str(row["provider_response"] or "{}")) + except (TypeError, json.JSONDecodeError): + continue + if not isinstance(provider_response, dict): + continue + if ( + provider_response.get("source") == "auto_pipeline" + and str(provider_response.get("workflow_job_id") or "") in allowed_workflow_ids + ): + linked_ids.append(str(row["id"])) + if not linked_ids: + return 0 + placeholders = ", ".join("?" for _ in linked_ids) + cursor = connection.execute( + f""" + UPDATE publish_jobs + SET status = 'CANCELLED', error_code = 'pipeline_cancelled', + error_message = '全自动流水线已取消', last_error = '全自动流水线已取消', + finished_at = ?, updated_at = ? + WHERE task_id = ? + AND status IN ('DRAFT', 'WAITING', 'SCHEDULED', 'NEED_REVIEW') + AND id IN ({placeholders}) + """, + (now, now, task_id, *linked_ids), + ) + return cursor.rowcount + + +def is_cancel_requested(job_id: str) -> bool: + job = get_job(job_id) + return bool(job and int(job.get("cancel_requested") or 0)) + + +def retry_job(job_id: str) -> dict | None: + now = _now_iso() + with get_connection() as connection: + cursor = connection.execute( + """ + UPDATE workflow_jobs + SET status = ?, progress = 0, message = '任务已重新加入队列', error_message = NULL, + finished_at = NULL, next_attempt_at = ?, cancel_requested = 0, + lease_owner = NULL, lease_token = NULL, lease_expires_at = NULL, heartbeat_at = NULL, + attempt_count = 0, updated_at = ? + WHERE id = ? AND status IN (?, ?) + """, + (JOB_STATUS_QUEUED, now, now, job_id, JOB_STATUS_FAILED, JOB_STATUS_CANCELLED), + ) + connection.commit() + return get_job(job_id) if cursor.rowcount else None + + +def retry_latest_or_get_active_job(task_id: str, job_type: str) -> tuple[dict | None, bool]: + """Return the active job, or atomically requeue the latest retryable job. + + Reusing the same row is important for auto-pipeline jobs because their + fenced step checkpoint belongs to that workflow_jobs.id. + """ + + now = _now_iso() + with get_connection() as connection: + connection.execute("BEGIN IMMEDIATE") + active = connection.execute( + """ + SELECT id FROM workflow_jobs + WHERE task_id = ? AND job_type = ? AND status IN (?, ?) + ORDER BY created_at DESC LIMIT 1 + """, + (task_id, job_type, JOB_STATUS_QUEUED, JOB_STATUS_RUNNING), + ).fetchone() + if active: + connection.commit() + return get_job(str(active["id"])), False + + retryable = connection.execute( + """ + SELECT id FROM workflow_jobs + WHERE task_id = ? AND job_type = ? AND status IN (?, ?) + ORDER BY created_at DESC LIMIT 1 + """, + (task_id, job_type, JOB_STATUS_FAILED, JOB_STATUS_CANCELLED), + ).fetchone() + if not retryable: + connection.commit() + return None, False + + cursor = connection.execute( + """ + UPDATE workflow_jobs + SET status = ?, progress = 0, message = '任务已重新加入队列', + result_json = '{}', error_message = NULL, finished_at = NULL, + next_attempt_at = ?, cancel_requested = 0, + lease_owner = NULL, lease_token = NULL, lease_expires_at = NULL, + heartbeat_at = NULL, attempt_count = 0, updated_at = ? + WHERE id = ? AND status IN (?, ?) + """, + ( + JOB_STATUS_QUEUED, + now, + now, + retryable["id"], + JOB_STATUS_FAILED, + JOB_STATUS_CANCELLED, + ), + ) + connection.commit() + if cursor.rowcount != 1: + return None, False + return get_job(str(retryable["id"])), True + + +def release_job_lease(job_id: str, lease_owner: str, lease_token: str) -> bool: + """Web 正常停止时释放子进程 Job,供新进程立即恢复。""" + now = _now_iso() + with get_connection() as connection: + connection.execute("BEGIN IMMEDIATE") + row = connection.execute( + """ + SELECT task_id, job_type, cancel_requested + FROM workflow_jobs + WHERE id = ? AND status = ? AND lease_owner = ? AND lease_token = ? + """, + (job_id, JOB_STATUS_RUNNING, lease_owner, lease_token), + ).fetchone() + if not row: + connection.rollback() + return False + cancelled = bool(int(row["cancel_requested"] or 0)) + cursor = connection.execute( + """ + UPDATE workflow_jobs + SET status = ?, message = ?, finished_at = CASE WHEN ? THEN ? ELSE finished_at END, + cancel_requested = CASE WHEN ? THEN 1 ELSE 0 END, + lease_owner = NULL, lease_token = NULL, lease_expires_at = NULL, + heartbeat_at = NULL, updated_at = ? + WHERE id = ? AND status = ? AND lease_owner = ? AND lease_token = ? + """, + ( + JOB_STATUS_CANCELLED if cancelled else JOB_STATUS_QUEUED, + "任务已取消" if cancelled else "应用停止,等待新 worker 恢复", + 1 if cancelled else 0, + now, + 1 if cancelled else 0, + now, + job_id, + JOB_STATUS_RUNNING, + lease_owner, + lease_token, + ), + ) + if cursor.rowcount == 1 and row["job_type"] == JOB_TYPE_AI_ANALYSIS: + _sync_ai_task_state( + connection, + task_id=str(row["task_id"] or ""), + outcome="cancelled" if cancelled else "queued", + message="用户已取消 AI 分析" if cancelled else "应用停止,AI 分析等待恢复", + now=now, + ) + connection.commit() + return cursor.rowcount == 1 + + +def update_job_progress( + job_id: str, + progress: int, + message: Optional[str] = None, + *, + lease_owner: str | None = None, + lease_token: str | None = None, +) -> dict | None: """更新 job 进度(0-100)和消息""" now = _now_iso() clamped_progress = max(0, min(100, progress)) + condition, lease_params, fenced = _lease_write_condition(lease_owner, lease_token) with get_connection() as connection: cursor = connection.execute( - """ + f""" UPDATE workflow_jobs SET progress = ?, message = ?, updated_at = ? - WHERE id = ? + WHERE {condition} """, - (clamped_progress, message or "", now, job_id), + (clamped_progress, message or "", now, job_id, *lease_params), ) connection.commit() + _raise_if_lease_lost(job_id, cursor.rowcount, fenced) if cursor.rowcount == 0: return None return get_job(job_id) -def mark_job_completed(job_id: str, result: Optional[dict] = None) -> dict | None: +def mark_job_completed( + job_id: str, + result: Optional[dict] = None, + *, + lease_owner: str | None = None, + lease_token: str | None = None, +) -> dict | None: """将 job 标记为 completed,附带结果""" now = _now_iso() result_json = json.dumps(result or {}, ensure_ascii=False) + condition, lease_params, fenced = _lease_write_condition(lease_owner, lease_token) with get_connection() as connection: cursor = connection.execute( - """ + f""" UPDATE workflow_jobs SET status = ?, progress = 100, message = '任务已完成', result_json = ?, - finished_at = ?, updated_at = ? - WHERE id = ? + finished_at = ?, updated_at = ?, lease_owner = NULL, + lease_token = NULL, lease_expires_at = NULL, heartbeat_at = NULL + WHERE {condition} AND cancel_requested = 0 """, - (JOB_STATUS_COMPLETED, result_json, now, now, job_id), + (JOB_STATUS_COMPLETED, result_json, now, now, job_id, *lease_params), ) connection.commit() + _raise_if_lease_lost(job_id, cursor.rowcount, fenced) if cursor.rowcount == 0: return None return get_job(job_id) -def mark_job_failed(job_id: str, error_message: str) -> dict | None: +def mark_job_completed_with_followup( + job_id: str, + result: Optional[dict], + *, + followup_task_id: str, + followup_job_type: str, + followup_payload: Optional[dict] = None, + result_followup_key: str = "followup_job_id", + lease_owner: str | None = None, + lease_token: str | None = None, +) -> tuple[dict, dict, bool]: + """在同一事务内完成当前 Job 并创建/复用后续 Job。""" + lease = _resolve_job_lease(lease_owner, lease_token) + if lease is None: + raise ValueError("完成并创建后续 Job 需要有效的 Workflow Job 租约") + with get_connection() as connection: + connection.execute("BEGIN IMMEDIATE") + now = _now_iso() + active = connection.execute( + """ + SELECT 1 FROM workflow_jobs + WHERE id = ? AND status = ? AND lease_owner = ? AND lease_token = ? + AND lease_expires_at > strftime('%Y-%m-%dT%H:%M:%S+00:00', 'now') + AND cancel_requested = 0 + """, + (job_id, JOB_STATUS_RUNNING, lease[0], lease[1]), + ).fetchone() + if not active: + connection.rollback() + raise JobLeaseLostError(f"Workflow Job 完成前租约已失效或已取消:{job_id}") + followup_job_id, created = create_or_get_active_job_with_connection( + connection, + task_id=followup_task_id, + job_type=followup_job_type, + payload=followup_payload, + ) + followup_row = connection.execute( + "SELECT payload_json FROM workflow_jobs WHERE id = ?", + (followup_job_id,), + ).fetchone() + try: + existing_followup_payload = json.loads(followup_row["payload_json"] or "{}") + except (TypeError, json.JSONDecodeError) as exc: + connection.rollback() + raise ValueError("后续 Workflow Job payload 已损坏,拒绝复用") from exc + if existing_followup_payload != (followup_payload or {}): + connection.rollback() + raise ValueError("已有后续 Workflow Job 的执行参数不同,拒绝错误复用") + result_payload = {**(result or {}), result_followup_key: followup_job_id} + cursor = connection.execute( + """ + UPDATE workflow_jobs + SET status = ?, progress = 100, message = '任务已完成', result_json = ?, + finished_at = ?, updated_at = ?, lease_owner = NULL, + lease_token = NULL, lease_expires_at = NULL, heartbeat_at = NULL + WHERE id = ? AND status = ? AND lease_owner = ? AND lease_token = ? + AND lease_expires_at > strftime('%Y-%m-%dT%H:%M:%S+00:00', 'now') + AND cancel_requested = 0 + """, + ( + JOB_STATUS_COMPLETED, + json.dumps(result_payload, ensure_ascii=False), + now, + now, + job_id, + JOB_STATUS_RUNNING, + lease[0], + lease[1], + ), + ) + if cursor.rowcount != 1: + connection.rollback() + raise JobLeaseLostError(f"Workflow Job 完成提交冲突:{job_id}") + connection.commit() + completed_job = get_job(job_id) + followup_job = get_job(followup_job_id) + if not completed_job or not followup_job: + raise RuntimeError("Workflow Job 原子提交后无法读取结果") + return completed_job, followup_job, created + + +def mark_job_failed( + job_id: str, + error_message: str, + *, + lease_owner: str | None = None, + lease_token: str | None = None, +) -> dict | None: """将 job 标记为 failed,记录错误信息""" now = _now_iso() + condition, lease_params, fenced = _lease_write_condition(lease_owner, lease_token) with get_connection() as connection: + connection.execute("BEGIN IMMEDIATE") + row = connection.execute( + "SELECT task_id, job_type FROM workflow_jobs WHERE id = ?", + (job_id,), + ).fetchone() cursor = connection.execute( - """ + f""" UPDATE workflow_jobs SET status = ?, error_message = ?, message = ?, - finished_at = ?, updated_at = ? - WHERE id = ? + finished_at = ?, updated_at = ?, lease_owner = NULL, + lease_token = NULL, lease_expires_at = NULL, heartbeat_at = NULL + WHERE {condition} AND cancel_requested = 0 """, - (JOB_STATUS_FAILED, error_message, f"任务失败:{error_message}", now, now, job_id), + ( + JOB_STATUS_FAILED, + error_message, + f"任务失败:{error_message}", + now, + now, + job_id, + *lease_params, + ), ) + if cursor.rowcount == 1 and row and row["job_type"] == JOB_TYPE_AI_ANALYSIS: + _sync_ai_task_state( + connection, + task_id=str(row["task_id"] or ""), + outcome="failed", + message=error_message, + now=now, + ) + elif cursor.rowcount == 1 and row and row["job_type"] == JOB_TYPE_AUTO_PIPELINE: + _sync_auto_pipeline_task_failed( + connection, + task_id=str(row["task_id"] or ""), + message=error_message, + now=now, + ) connection.commit() + _raise_if_lease_lost(job_id, cursor.rowcount, fenced) if cursor.rowcount == 0: return None return get_job(job_id) + + +def mark_job_cancelled( + job_id: str, + message: str = "任务已取消", + *, + lease_owner: str | None = None, + lease_token: str | None = None, +) -> dict | None: + now = _now_iso() + condition, lease_params, fenced = _lease_write_condition(lease_owner, lease_token) + with get_connection() as connection: + connection.execute("BEGIN IMMEDIATE") + row = connection.execute( + "SELECT task_id, job_type FROM workflow_jobs WHERE id = ?", + (job_id,), + ).fetchone() + cursor = connection.execute( + f""" + UPDATE workflow_jobs SET status = ?, message = ?, finished_at = ?, updated_at = ?, + lease_owner = NULL, lease_token = NULL, lease_expires_at = NULL, heartbeat_at = NULL + WHERE {condition} + """, + (JOB_STATUS_CANCELLED, message, now, now, job_id, *lease_params), + ) + if cursor.rowcount == 1 and row and row["job_type"] == JOB_TYPE_AI_ANALYSIS: + _sync_ai_task_state( + connection, + task_id=str(row["task_id"] or ""), + outcome="cancelled", + message=message, + now=now, + ) + connection.commit() + _raise_if_lease_lost(job_id, cursor.rowcount, fenced) + return get_job(job_id) if cursor.rowcount else None + + +def _sync_ai_task_state( + connection, + *, + task_id: str, + outcome: str, + message: str, + now: str, +) -> None: + """AI Job 结束/释放时只收口仍属于 AI 阶段的 Task,不覆盖已提交结果。""" + if not task_id: + return + if outcome == "failed": + status = "failed" + progress = 0 + error_message = " ".join(str(message or "AI 分析失败").split())[:1000] + else: + status = "pending_ai" + progress = 55 + error_message = None + connection.execute( + """ + UPDATE tasks + SET status = ?, progress = ?, error_message = ?, last_error = ?, updated_at = ? + WHERE id = ? AND status IN ('ai_analyzing', 'AI_ANALYZING') + AND COALESCE(is_deleted, 0) = 0 + """, + (status, progress, error_message, error_message, now, task_id), + ) + + +def _sync_auto_pipeline_task_failed( + connection, + *, + task_id: str, + message: str, + now: str, +) -> None: + """父 Worker 异常终止时,把仍在运行步骤的自动任务落到对应失败态。""" + if not task_id: + return + row = connection.execute( + "SELECT status FROM tasks WHERE id = ? AND COALESCE(is_deleted, 0) = 0", + (task_id,), + ).fetchone() + if not row: + return + failed_by_running = { + "CREATED": "FAILED_PREPARING_SOURCE", + "PREPARING_SOURCE": "FAILED_PREPARING_SOURCE", + "TRANSCRIBING": "FAILED_TRANSCRIBING", + "AI_ANALYZING": "FAILED_AI_ANALYZING", + "CLIP_SELECTING": "FAILED_CLIP_SELECTING", + "VIDEO_CUTTING": "FAILED_VIDEO_CUTTING", + "SUBTITLE_DRAFTING": "FAILED_SUBTITLE_DRAFTING", + "METADATA_GENERATING": "FAILED_METADATA_GENERATING", + "SCHEDULE_CREATING": "FAILED_SCHEDULE_CREATING", + "PUBLISH_JOB_CREATING": "FAILED_PUBLISH_JOB_CREATING", + } + failed_status = failed_by_running.get(str(row["status"] or "")) + if not failed_status: + return + error = " ".join(str(message or "自动流水线子进程异常退出").split())[:1000] + connection.execute( + """ + UPDATE tasks + SET status = ?, error_message = ?, last_error = ?, updated_at = ? + WHERE id = ? AND status = ? AND COALESCE(is_deleted, 0) = 0 + """, + (failed_status, error, error, now, task_id, str(row["status"])), + ) diff --git a/app/services/job_worker.py b/app/services/job_worker.py index 2bd8e83..263078a 100644 --- a/app/services/job_worker.py +++ b/app/services/job_worker.py @@ -1,42 +1,89 @@ -"""轻量本地工作流任务队列 —— Job Worker +"""单进程持久化工作流 Worker。 -负责在后台执行 job 记录对应的长任务。 -第一轮仅接入自动切片(video_cut),后续再逐步迁移其他流程。 +数据库 lease 让应用重启后可接管过期任务;默认只运行一个本地重型任务。 """ +from __future__ import annotations + +import os +import socket +import subprocess +import sys +import threading +import time +from uuid import uuid4 + from app.services import job_service from app.services import task_service +from app.core.config import settings +from app.services.managed_process_service import popen_process_group, terminate_process_tree -def execute_job(job_id: str) -> dict: +def execute_job( + job_id: str, + *, + lease_owner: str | None = None, + lease_token: str | None = None, + already_claimed: bool = False, +) -> dict: """根据 job 记录执行对应的后台任务 这是一个同步函数,调用方应自行决定放在线程池还是 BackgroundTasks 中运行。 返回最终的 job 记录。 """ + owner = lease_owner or f"{socket.gethostname()}:{os.getpid()}:{uuid4().hex[:8]}" job = job_service.get_job(job_id) if not job: raise ValueError(f"job 不存在:{job_id}") if job.get("status") == job_service.JOB_STATUS_CANCELLED: return job + if already_claimed: + if not lease_owner or not lease_token: + raise job_service.JobLeaseLostError("已领取的 Workflow Job 缺少 owner/token") + job = job_service.validate_job_lease(job_id, lease_owner, lease_token) + if not job: + raise job_service.JobLeaseLostError(f"Workflow Job 启动前租约已失效:{job_id}") + else: + job = job_service.claim_job(job_id, owner) + if not job: + return job_service.get_job(job_id) + lease_token = str(job.get("lease_token") or "") + if not lease_token: + raise job_service.JobLeaseLostError(f"Workflow Job claim 未生成 token:{job_id}") + job_type = job.get("job_type") task_id = job.get("task_id") - # 标记开始执行 - job = job_service.mark_job_running(job_id) - if not job: - raise RuntimeError(f"无法更新 job 状态:{job_id}") + with job_service.job_lease_context(job_id, owner, lease_token): + try: + if job_type == job_service.JOB_TYPE_VIDEO_CUT: + _execute_video_cut(job_id, task_id) + elif job_type == job_service.JOB_TYPE_TRANSCRIPT: + _execute_transcript(job_id, task_id, job.get("payload_json") or {}) + elif job_type == job_service.JOB_TYPE_AI_ANALYSIS: + _execute_ai_analysis(job_id, task_id, job.get("payload_json") or {}) + elif job_type == job_service.JOB_TYPE_AUTO_PIPELINE: + _execute_auto_pipeline(job_id, task_id, job.get("payload_json") or {}) + elif job_type == job_service.JOB_TYPE_SUBTITLE: + _execute_subtitle(job_id, task_id, job.get("payload_json") or {}) + else: + raise ValueError(f"暂不支持的 job 类型:{job_type}") + except job_service.JobLeaseLostError: + raise + except Exception as exc: + if job_type == job_service.JOB_TYPE_SUBTITLE: + from app.services.subtitle_auto_workflow_service import cleanup_interrupted_subtitle_job - try: - if job_type == job_service.JOB_TYPE_VIDEO_CUT: - _execute_video_cut(job_id, task_id) - else: - raise ValueError(f"暂不支持的 job 类型:{job_type}") - except Exception as exc: - error = str(exc) - job_service.mark_job_failed(job_id, error) - raise + cleanup_interrupted_subtitle_job( + job_id, + lease_owner=owner, + lease_token=lease_token, + status="failed", + message=str(exc), + ) + job_service.mark_job_failed(job_id, str(exc)) + raise return job_service.get_job(job_id) @@ -64,3 +111,373 @@ def _execute_video_cut(job_id: str, task_id: str) -> None: } job_service.mark_job_completed(job_id, result_summary) + + +def _execute_transcript(job_id: str, task_id: str, payload: dict) -> None: + from app.services import transcript_workflow_service + from app.services.storage_service import get_artifact_paths + from app.services.transcript_service import read_transcript_progress + + job_service.update_job_progress(job_id, 5, "正在准备音频和转写 checkpoint") + transcript_workflow_service.process_task_transcript_workflow( + task_id, + background_tasks=None, + force=bool(payload.get("force")), + provider=payload.get("provider"), + job_id=job_id, + ) + if job_service.is_cancel_requested(job_id): + job_service.mark_job_cancelled(job_id, "转写已取消,已完成分块 checkpoint 会保留") + return + progress = read_transcript_progress(get_artifact_paths(task_id)["transcript_path"]) + if progress.get("status") == "failed": + raise RuntimeError(str(progress.get("message") or "转写失败")) + job_service.mark_job_completed( + job_id, + {"task_id": task_id, "transcript_status": progress.get("status"), "checkpoint_retained": True}, + ) + + +def _execute_ai_analysis(job_id: str, task_id: str, payload: dict) -> None: + job_service.update_job_progress(job_id, 5, "正在准备 AI 分析与恢复检查") + result = task_service.process_task_ai_analysis(task_id, provider=payload.get("provider")) + if job_service.is_cancel_requested(job_id): + job_service.mark_job_cancelled(job_id, "AI 分析已取消") + return + job_service.update_job_progress(job_id, 95, "AI 结果已原子落库,正在完成 Job") + meta = (result.get("analysis_run") or {}).get("analysis_meta") or {} + job_service.mark_job_completed( + job_id, + { + "task_id": task_id, + "message": result.get("message") or "AI 分析完成", + "analysis_run_id": result.get("analysis_run_id") or "", + "clip_count": len(result.get("clips") or []), + "review_url": result.get("review_url") or f"/tasks/{task_id}/clips/review", + "analysis_incomplete": bool(meta.get("analysis_incomplete")), + "coverage_percent": float(meta.get("coverage_percent") or 0), + }, + ) + + +def _execute_auto_pipeline(job_id: str, task_id: str, payload: dict) -> None: + from app.services.pipeline_engine import run_auto_pipeline + + result = run_auto_pipeline( + task_id, + retry=bool(payload.get("retry")), + start_step=payload.get("start_step"), + job_id=job_id, + ) + if job_service.is_cancel_requested(job_id): + job_service.mark_job_cancelled(job_id, "全自动流水线已取消") + return + if result.get("status") == "cancelled": + job_service.mark_job_cancelled(job_id, "全自动流水线已取消") + return + if result.get("status") == "failed": + raise RuntimeError(str(result.get("last_error") or "全自动流水线失败")) + job_service.mark_job_completed(job_id, result) + + +def _execute_subtitle(job_id: str, task_id: str, payload: dict) -> None: + from app.services.subtitle_auto_workflow_service import execute_subtitle_render_job + from app.services.subtitle_workflow_service import SubtitleRenderCancelled + + job_service.update_job_progress(job_id, 3, "正在核对已审核的字幕 revision") + try: + result = execute_subtitle_render_job(job_id, task_id, payload) + except SubtitleRenderCancelled: + job_service.mark_job_cancelled(job_id, "字幕批量烧录已取消") + return + if job_service.is_cancel_requested(job_id): + job_service.mark_job_cancelled(job_id, "字幕批量烧录已取消") + return + if result.get("resume_requested"): + from app.models.task import TaskStatus + + _completed_job, resume_job, created = job_service.mark_job_completed_with_followup( + job_id, + result, + followup_task_id=task_id, + followup_job_type=job_service.JOB_TYPE_AUTO_PIPELINE, + followup_payload={"retry": False, "start_step": TaskStatus.METADATA_GENERATING.value}, + result_followup_key="resume_job_id", + ) + from app.services.task_log_service import append_task_log + + append_task_log( + task_id, + "字幕成片全部验证通过,已排队恢复自动文案与发送中心流程" + if created + else f"字幕成片全部验证通过,已复用自动流水线恢复 Job:{resume_job['id']}", + ) + return + job_service.mark_job_completed(job_id, result) + + +class WorkflowJobRunner: + """应用生命周期内的单 worker 线程。""" + + def __init__(self, poll_seconds: float = 1.0) -> None: + self.poll_seconds = poll_seconds + self.owner = f"{socket.gethostname()}:{os.getpid()}:{uuid4().hex[:8]}" + self._stop_event = threading.Event() + self._thread: threading.Thread | None = None + + def start(self) -> None: + if self._thread and self._thread.is_alive(): + return + self._thread = threading.Thread(target=self._run, name="workflow-job-worker", daemon=True) + self._thread.start() + + def stop(self) -> None: + self._stop_event.set() + if self._thread: + self._thread.join(timeout=25) + + def _run(self) -> None: + while not self._stop_event.is_set(): + job = job_service.claim_next_job(self.owner) + if not job: + self._stop_event.wait(self.poll_seconds) + continue + try: + self._run_job_subprocess(job["id"]) + except Exception as exc: + current = job_service.get_job(job["id"]) + if ( + current + and current.get("status") == job_service.JOB_STATUS_RUNNING + and current.get("lease_owner") == self.owner + and current.get("lease_token") + ): + try: + job_service.mark_job_failed( + job["id"], + f"Worker 收尾异常:{exc}", + lease_owner=self.owner, + lease_token=str(current["lease_token"]), + ) + except job_service.JobLeaseLostError: + pass + self._stop_event.wait(self.poll_seconds) + + def _run_job_subprocess(self, job_id: str) -> None: + job_before_start = job_service.get_job(job_id) or {} + lease_token = str(job_before_start.get("lease_token") or "") + if job_before_start.get("lease_owner") != self.owner or not lease_token: + return + try: + process = popen_process_group( + [sys.executable, "-m", "app.services.job_worker_process", job_id, self.owner, lease_token], + cwd=str(settings.project_root), + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, + ) + except Exception as exc: + try: + job_service.mark_job_failed( + job_id, + f"无法启动 Job 子进程:{exc}", + lease_owner=self.owner, + lease_token=lease_token, + ) + except job_service.JobLeaseLostError: + pass + return + last_heartbeat = 0.0 + last_progress_at = time.monotonic() + previous_progress: tuple[int, str] | None = None + no_progress_timeout = ( + settings.ffmpeg_subtitle_timeout + if job_before_start.get("job_type") == job_service.JOB_TYPE_SUBTITLE + else max( + 900, + settings.ffmpeg_audio_extract_timeout, + settings.ffmpeg_cut_timeout, + settings.volcengine_asr_timeout_seconds, + ) + ) + while process.poll() is None: + if self._stop_event.wait(1): + if not self._terminate_process(process, job_id, lease_token, "应用停止"): + time.sleep(1) + continue + if job_before_start.get("job_type") == job_service.JOB_TYPE_SUBTITLE: + from app.services.subtitle_auto_workflow_service import cleanup_interrupted_subtitle_job + + try: + job_service.heartbeat_job(job_id, self.owner, lease_token) + except job_service.JobLeaseLostError: + return + cleaned = cleanup_interrupted_subtitle_job( + job_id, + lease_owner=self.owner, + lease_token=lease_token, + status="queued", + message="应用停止,等待恢复字幕烧录", + ) + if not cleaned: + return + job_service.release_job_lease(job_id, self.owner, lease_token) + return + job = job_service.get_job(job_id) + if not job: + self._terminate_process(process, job_id, lease_token, "Job 记录已消失") + return + if ( + job.get("status") != job_service.JOB_STATUS_RUNNING + or job.get("lease_owner") != self.owner + or job.get("lease_token") != lease_token + ): + if self._terminate_process(process, job_id, lease_token, "Workflow Job 租约已改变"): + return + time.sleep(1) + continue + if job_service.is_cancel_requested(job_id): + if not self._terminate_process(process, job_id, lease_token, "取消任务"): + time.sleep(1) + continue + if job.get("job_type") == job_service.JOB_TYPE_SUBTITLE: + from app.services.subtitle_auto_workflow_service import cleanup_interrupted_subtitle_job + + try: + job_service.heartbeat_job(job_id, self.owner, lease_token) + except job_service.JobLeaseLostError: + return + cleaned = cleanup_interrupted_subtitle_job( + job_id, + lease_owner=self.owner, + lease_token=lease_token, + status="cancelled", + message="用户已取消字幕烧录", + ) + if not cleaned: + return + job_service.mark_job_cancelled( + job_id, + "任务已取消,子进程树已终止", + lease_owner=self.owner, + lease_token=lease_token, + ) + return + progress_state = (int(job.get("progress") or 0), str(job.get("message") or "")) + if previous_progress != progress_state: + previous_progress = progress_state + last_progress_at = time.monotonic() + if time.monotonic() - last_progress_at > no_progress_timeout: + if not self._terminate_process(process, job_id, lease_token, "任务无进展超时"): + time.sleep(1) + continue + if job.get("job_type") == job_service.JOB_TYPE_SUBTITLE: + from app.services.subtitle_auto_workflow_service import cleanup_interrupted_subtitle_job + + try: + job_service.heartbeat_job(job_id, self.owner, lease_token) + except job_service.JobLeaseLostError: + return + cleaned = cleanup_interrupted_subtitle_job( + job_id, + lease_owner=self.owner, + lease_token=lease_token, + status="failed", + message=f"字幕烧录连续 {no_progress_timeout} 秒没有进展", + ) + if not cleaned: + return + job_service.mark_job_failed( + job_id, + f"任务连续 {no_progress_timeout} 秒没有进展,已终止子进程树", + lease_owner=self.owner, + lease_token=lease_token, + ) + return + if time.monotonic() - last_heartbeat >= 20: + try: + job_service.heartbeat_job(job_id, self.owner, lease_token) + except job_service.JobLeaseLostError: + if self._terminate_process(process, job_id, lease_token, "Workflow Job 心跳租约已失效"): + return + time.sleep(1) + continue + last_heartbeat = time.monotonic() + final_job = job_service.validate_job_lease(job_id, self.owner, lease_token) + owns_final_lease = bool(final_job) + if owns_final_lease and int(final_job.get("cancel_requested") or 0): + job_service.mark_job_cancelled( + job_id, + "任务已取消,子进程已退出", + lease_owner=self.owner, + lease_token=lease_token, + ) + elif process.returncode != 0 and owns_final_lease: + if final_job.get("job_type") == job_service.JOB_TYPE_SUBTITLE: + from app.services.subtitle_auto_workflow_service import cleanup_interrupted_subtitle_job + + try: + job_service.heartbeat_job(job_id, self.owner, lease_token) + except job_service.JobLeaseLostError: + return + cleaned = cleanup_interrupted_subtitle_job( + job_id, + lease_owner=self.owner, + lease_token=lease_token, + status="failed", + message=f"字幕 Job 子进程异常退出,退出码:{process.returncode}", + ) + if not cleaned: + return + job_service.mark_job_failed( + job_id, + f"Job 子进程异常退出,退出码:{process.returncode}", + lease_owner=self.owner, + lease_token=lease_token, + ) + elif process.returncode == 0 and owns_final_lease: + if final_job.get("job_type") == job_service.JOB_TYPE_SUBTITLE: + from app.services.subtitle_auto_workflow_service import cleanup_interrupted_subtitle_job + + try: + job_service.heartbeat_job(job_id, self.owner, lease_token) + except job_service.JobLeaseLostError: + return + cleaned = cleanup_interrupted_subtitle_job( + job_id, + lease_owner=self.owner, + lease_token=lease_token, + status="failed", + message="字幕 Job 子进程正常退出但没有写入终态", + ) + if not cleaned: + return + job_service.mark_job_failed( + job_id, + "Job 子进程已退出但没有写入终态", + lease_owner=self.owner, + lease_token=lease_token, + ) + + def _terminate_process( + self, + process: subprocess.Popen, + job_id: str, + lease_token: str, + reason: str, + ) -> bool: + """终止失败时保留 Job/lease 并继续重试,不能让 Worker 主线程退出。""" + try: + terminate_process_tree(process) + return True + except Exception as exc: + try: + job_service.update_job_progress( + job_id, + int((job_service.get_job(job_id) or {}).get("progress") or 0), + f"{reason}时无法确认子进程已停止,将继续重试:{exc}", + lease_owner=self.owner, + lease_token=lease_token, + ) + except (job_service.JobLeaseLostError, ValueError): + pass + return False diff --git a/app/services/job_worker_process.py b/app/services/job_worker_process.py new file mode 100644 index 0000000..53bf18f --- /dev/null +++ b/app/services/job_worker_process.py @@ -0,0 +1,32 @@ +"""持久化 Job 的独立子进程入口。""" + +from __future__ import annotations + +import sys + +from app.services.job_worker import execute_job + + +def main() -> int: + if len(sys.argv) != 4: + print( + "用法:python -m app.services.job_worker_process ", + file=sys.stderr, + ) + return 2 + job_id, lease_owner, lease_token = sys.argv[1], sys.argv[2], sys.argv[3] + try: + execute_job( + job_id, + lease_owner=lease_owner, + lease_token=lease_token, + already_claimed=True, + ) + except Exception as exc: + print(str(exc), file=sys.stderr) + return 1 + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/app/services/managed_process_service.py b/app/services/managed_process_service.py new file mode 100644 index 0000000..e3c0a9f --- /dev/null +++ b/app/services/managed_process_service.py @@ -0,0 +1,62 @@ +"""可取消、可检测无进展超时的本地子进程工具。""" + +from __future__ import annotations + +import os +import signal +import subprocess + + +class ProcessTerminationError(RuntimeError): + """无法确认本地子进程树已经停止。""" + + +def popen_process_group(command: list[str], **kwargs) -> subprocess.Popen: + if os.name == "nt": + kwargs["creationflags"] = kwargs.get("creationflags", 0) | subprocess.CREATE_NEW_PROCESS_GROUP + else: + kwargs["start_new_session"] = True + return subprocess.Popen(command, **kwargs) + + +def terminate_process_tree(process: subprocess.Popen) -> None: + if process.poll() is not None: + return + if os.name == "nt": + try: + result = subprocess.run( + ["taskkill", "/PID", str(process.pid), "/T", "/F"], + capture_output=True, + check=False, + timeout=15, + ) + except (OSError, subprocess.TimeoutExpired) as exc: + if process.poll() is not None: + return + raise ProcessTerminationError(f"无法执行 taskkill 终止进程树 PID={process.pid}:{exc}") from exc + if result.returncode != 0 and process.poll() is None: + stderr = (result.stderr or b"").decode("utf-8", errors="replace").strip() + raise ProcessTerminationError( + f"taskkill 未能终止进程树 PID={process.pid},退出码 {result.returncode}" + + (f":{stderr}" if stderr else "") + ) + try: + process.wait(timeout=5) + except subprocess.TimeoutExpired as exc: + raise ProcessTerminationError(f"进程树 PID={process.pid} 在 taskkill 后仍未退出") from exc + else: + try: + os.killpg(os.getpgid(process.pid), signal.SIGTERM) + process.wait(timeout=5) + except ProcessLookupError as exc: + if process.poll() is None: + raise ProcessTerminationError(f"无法确认进程树 PID={process.pid} 已退出") from exc + except subprocess.TimeoutExpired: + try: + os.killpg(os.getpgid(process.pid), signal.SIGKILL) + process.wait(timeout=5) + except ProcessLookupError as exc: + if process.poll() is None: + raise ProcessTerminationError(f"无法确认进程树 PID={process.pid} 已退出") from exc + except subprocess.TimeoutExpired as exc: + raise ProcessTerminationError(f"进程树 PID={process.pid} 在 SIGKILL 后仍未退出") from exc diff --git a/app/services/media_preflight_service.py b/app/services/media_preflight_service.py new file mode 100644 index 0000000..82bdc5d --- /dev/null +++ b/app/services/media_preflight_service.py @@ -0,0 +1,190 @@ +"""长素材媒体与磁盘预检。 + +创建任务前只做只读探测,不改写原片。超过 6 小时是提示,不是拒绝条件。 +""" + +from __future__ import annotations + +import json +import shutil +import subprocess +from dataclasses import asdict, dataclass +from pathlib import Path + +from app.core.config import settings + + +MAX_TESTED_DURATION_SECONDS = 6 * 60 * 60 +PCM_BYTES_PER_SECOND = 16_000 * 2 +MIN_SAFETY_MARGIN_BYTES = 1024 * 1024 * 1024 + + +@dataclass(frozen=True) +class MediaPreflight: + path: str + duration_seconds: float + file_size_bytes: int + video_codec: str + audio_codec: str + width: int + height: int + frame_rate: float + audio_channels: int + audio_sample_rate: int + required_free_bytes: int + available_free_bytes: int + warnings: list[str] + + def to_dict(self) -> dict: + return asdict(self) + + +def _parse_rate(value: str | None) -> float: + text = str(value or "0/1") + try: + numerator, denominator = text.split("/", 1) + return float(numerator) / max(float(denominator), 1.0) + except (TypeError, ValueError, ZeroDivisionError): + return 0.0 + + +def _run_decode_sample(path: Path, start_seconds: float) -> None: + ffmpeg = shutil.which("ffmpeg") + if not ffmpeg: + raise ValueError("未找到 FFmpeg,无法验证视频可解码性") + command = [ffmpeg, "-v", "error", "-xerror"] + if start_seconds > 0: + command.extend(["-ss", f"{start_seconds:.3f}"]) + command.extend( + ["-i", str(path), "-t", "3", "-map", "0:v:0", "-map", "0:a:0", "-f", "null", "-"] + ) + try: + completed = subprocess.run( + command, + capture_output=True, + text=True, + encoding="utf-8", + errors="replace", + timeout=settings.ffprobe_timeout, + check=False, + ) + except subprocess.TimeoutExpired as exc: + raise ValueError(f"视频解码抽样超过 {settings.ffprobe_timeout} 秒,文件可能损坏") from exc + except OSError as exc: + raise ValueError(f"FFmpeg 无法执行视频解码抽样:{exc}") from exc + if completed.returncode != 0: + message = (completed.stderr or completed.stdout or "未知解码错误").strip() + raise ValueError(f"视频无法正常解码:{message[-500:]}") + + +def probe_media(path_value: str | Path) -> dict: + path = Path(path_value).resolve() + ffprobe = shutil.which("ffprobe") + if not ffprobe: + raise ValueError("未找到 FFprobe,无法创建视频任务") + try: + completed = subprocess.run( + [ + ffprobe, + "-v", + "error", + "-show_streams", + "-show_format", + "-of", + "json", + str(path), + ], + capture_output=True, + text=True, + encoding="utf-8", + errors="replace", + timeout=settings.ffprobe_timeout, + check=False, + ) + except subprocess.TimeoutExpired as exc: + raise ValueError(f"媒体探测超过 {settings.ffprobe_timeout} 秒,文件可能损坏") from exc + except OSError as exc: + raise ValueError(f"FFprobe 无法执行媒体探测:{exc}") from exc + if completed.returncode != 0: + message = (completed.stderr or "FFprobe 无法读取文件").strip() + raise ValueError(f"媒体探测失败:{message[-500:]}") + try: + payload = json.loads(completed.stdout or "{}") + except json.JSONDecodeError as exc: + raise ValueError("FFprobe 返回了无法解析的媒体信息") from exc + + streams = payload.get("streams") or [] + video_stream = next((item for item in streams if item.get("codec_type") == "video"), None) + audio_stream = next((item for item in streams if item.get("codec_type") == "audio"), None) + if not video_stream: + raise ValueError("源文件没有视频轨,不能创建视频处理任务") + if not audio_stream: + raise ValueError("源文件没有音轨,无法进行语言转写和高光选片") + try: + duration = float((payload.get("format") or {}).get("duration") or video_stream.get("duration") or 0) + except (TypeError, ValueError): + duration = 0.0 + if duration <= 0: + raise ValueError("源文件时长无效,无法创建任务") + + _run_decode_sample(path, 0) + if duration > 10: + _run_decode_sample(path, max(0.0, duration - 5.0)) + return { + "duration_seconds": duration, + "file_size_bytes": path.stat().st_size, + "video_codec": str(video_stream.get("codec_name") or "unknown"), + "audio_codec": str(audio_stream.get("codec_name") or "unknown"), + "width": int(video_stream.get("width") or 0), + "height": int(video_stream.get("height") or 0), + "frame_rate": _parse_rate(video_stream.get("avg_frame_rate") or video_stream.get("r_frame_rate")), + "audio_channels": int(audio_stream.get("channels") or 0), + "audio_sample_rate": int(audio_stream.get("sample_rate") or 0), + } + + +def estimate_required_bytes( + *, + duration_seconds: float, + source_size_bytes: int, + total_output_limit: int, +) -> int: + pcm_bytes = int(duration_seconds * PCM_BYTES_PER_SECOND) + # 默认按每条最多 5 分钟估算切片;同时预留一份字幕成片。 + output_fraction = min(1.0, total_output_limit * 300 / max(duration_seconds, 1.0)) + clip_and_subtitle_bytes = int(source_size_bytes * output_fraction * 2.2) + working_margin = max(MIN_SAFETY_MARGIN_BYTES, int((pcm_bytes + clip_and_subtitle_bytes) * 0.25)) + return pcm_bytes + clip_and_subtitle_bytes + working_margin + + +def preflight_media( + path_value: str | Path, + *, + total_output_limit: int, +) -> MediaPreflight: + path = Path(path_value).resolve() + probe = probe_media(path) + required = estimate_required_bytes( + duration_seconds=probe["duration_seconds"], + source_size_bytes=probe["file_size_bytes"], + total_output_limit=total_output_limit, + ) + storage_anchor = settings.tasks_dir + storage_anchor.mkdir(parents=True, exist_ok=True) + available = shutil.disk_usage(storage_anchor).free + if available < required: + required_gib = required / (1024 ** 3) + available_gib = available / (1024 ** 3) + raise ValueError( + f"任务存储空间不足:预计至少需要 {required_gib:.1f} GiB,当前可用 {available_gib:.1f} GiB" + ) + warnings: list[str] = [] + if probe["duration_seconds"] > MAX_TESTED_DURATION_SECONDS: + warnings.append("素材超过当前 6 小时验收范围,可以创建,但请重点关注耗时和磁盘空间") + return MediaPreflight( + path=str(path), + required_free_bytes=required, + available_free_bytes=available, + warnings=warnings, + **probe, + ) diff --git a/app/services/metadata_generator.py b/app/services/metadata_generator.py index 284e6b8..ecc5778 100644 --- a/app/services/metadata_generator.py +++ b/app/services/metadata_generator.py @@ -40,6 +40,9 @@ def generate(self, item: dict, platform: str) -> dict: hashtags = normalize_douyin_tags(normalized["tags"]) cover_text = self._cover_text(title) risk_flags = self._risk_flags(item, title, caption, hashtags) + metadata_error = str(metadata.get("error") or "") + if metadata_error: + risk_flags.append("AI 文案生成失败,已使用规则文案") return { "clip_id": item.get("output_clip_id") or item.get("id") or "", "clip_candidate_id": item.get("clip_candidate_id") or "", @@ -51,7 +54,7 @@ def generate(self, item: dict, platform: str) -> dict: "risk_flags": risk_flags, "status": "NEED_REVIEW" if risk_flags else "READY", "source": metadata.get("source") or "rule", - "error": metadata.get("error") or "", + "error": metadata_error, "recommend_reason": item.get("highlight_reason") or item.get("clip_summary") or "", } diff --git a/app/services/pipeline_checkpoint_service.py b/app/services/pipeline_checkpoint_service.py new file mode 100644 index 0000000..cb0b8fc --- /dev/null +++ b/app/services/pipeline_checkpoint_service.py @@ -0,0 +1,259 @@ +"""Versioned step checkpoints for the persistent auto pipeline.""" + +from __future__ import annotations + +from copy import deepcopy +from datetime import datetime, timezone +from typing import Any, Iterable + +from app.services import job_service + + +AUTO_PIPELINE_CHECKPOINT_KIND = "auto_pipeline_step_v1" +_STEP_STATES = {"running", "succeeded", "failed"} + + +class PipelineCheckpointError(RuntimeError): + """The persisted auto-pipeline checkpoint cannot be trusted.""" + + +def _now_iso() -> str: + return datetime.now(timezone.utc).isoformat(timespec="seconds") + + +class AutoPipelineCheckpoint: + """Small, fenced state machine stored in workflow_jobs.checkpoint_json.""" + + def __init__( + self, + *, + job_id: str, + task_id: str, + start_step: str, + run_key: str, + ordered_steps: Iterable[str], + state: dict[str, Any], + ) -> None: + self.job_id = job_id + self.task_id = task_id + self.start_step = start_step + self.run_key = run_key + self.ordered_steps = tuple(ordered_steps) + self.scoped_steps = self.ordered_steps[self.ordered_steps.index(start_step) :] + self.state = state + self._validate() + + @classmethod + def load( + cls, + *, + job_id: str, + task_id: str, + start_step: str, + run_key: str, + ordered_steps: Iterable[str], + ) -> AutoPipelineCheckpoint: + active = job_service.require_active_job_lease() + if not active or active[0] != job_id: + raise job_service.JobLeaseLostError( + f"自动流水线没有当前 Workflow Job 租约:{job_id}" + ) + job = job_service.get_job(job_id) + if not job or job.get("job_type") != job_service.JOB_TYPE_AUTO_PIPELINE: + raise PipelineCheckpointError("自动流水线 checkpoint 对应的 Job 不存在或类型不正确") + if str(job.get("task_id") or "") != task_id: + raise PipelineCheckpointError("自动流水线 checkpoint 的 Task 与 Job 不一致") + + raw = job.get("checkpoint_json") + if raw in (None, "", {}): + state = { + "kind": AUTO_PIPELINE_CHECKPOINT_KIND, + "task_id": task_id, + "run_key": run_key, + "start_step": start_step, + "current_step": "", + "completed_steps": [], + "steps": {}, + "last_error": "", + "updated_at": _now_iso(), + } + elif not isinstance(raw, dict): + raise PipelineCheckpointError("自动流水线 checkpoint JSON 已损坏,拒绝从头重复执行") + else: + state = deepcopy(raw) + + return cls( + job_id=job_id, + task_id=task_id, + start_step=start_step, + run_key=run_key, + ordered_steps=ordered_steps, + state=state, + ) + + @property + def has_history(self) -> bool: + return bool(self.state.get("steps") or self.state.get("completed_steps")) + + @property + def current_step(self) -> str: + return str(self.state.get("current_step") or "") + + @property + def completed_steps(self) -> tuple[str, ...]: + return tuple(str(item) for item in self.state.get("completed_steps") or []) + + def is_completed(self, step: str) -> bool: + return step in self.completed_steps + + def step_record(self, step: str) -> dict[str, Any]: + record = (self.state.get("steps") or {}).get(step) or {} + return deepcopy(record) if isinstance(record, dict) else {} + + def begin_step(self, step: str, *, baseline: dict[str, Any] | None = None) -> None: + self._require_next_step(step) + now = _now_iso() + previous = self.step_record(step) + attempts = int(previous.get("attempts") or 0) + 1 + self.state["current_step"] = step + self.state.setdefault("steps", {})[step] = { + "state": "running", + "attempts": attempts, + "started_at": now, + "baseline": deepcopy(baseline or {}), + "outputs": {}, + "error": "", + } + self.state["last_error"] = "" + self._persist() + + def complete_step( + self, + step: str, + *, + outputs: dict[str, Any], + recovered: bool = False, + ) -> None: + self._require_next_step(step) + previous = self.step_record(step) + now = _now_iso() + self.state.setdefault("steps", {})[step] = { + "state": "succeeded", + "attempts": max(1, int(previous.get("attempts") or 0)), + "started_at": str(previous.get("started_at") or now), + "completed_at": now, + "recovered": bool(recovered), + "baseline": deepcopy(previous.get("baseline") or {}), + "outputs": deepcopy(outputs), + "error": "", + } + completed = list(self.completed_steps) + completed.append(step) + self.state["completed_steps"] = completed + self.state["current_step"] = "" + self.state["last_error"] = "" + self._persist() + + def fail_step(self, step: str, error: str) -> None: + self._require_next_step(step) + previous = self.step_record(step) + now = _now_iso() + self.state.setdefault("steps", {})[step] = { + "state": "failed", + "attempts": max(1, int(previous.get("attempts") or 0)), + "started_at": str(previous.get("started_at") or now), + "failed_at": now, + "baseline": deepcopy(previous.get("baseline") or {}), + "outputs": deepcopy(previous.get("outputs") or {}), + "error": str(error or "自动流水线步骤失败"), + } + self.state["current_step"] = step + self.state["last_error"] = str(error or "自动流水线步骤失败") + self._persist() + + def invalidate_from(self, step: str, error: str) -> None: + if step not in self.scoped_steps: + raise PipelineCheckpointError(f"未知的自动流水线步骤:{step}") + index = self.scoped_steps.index(step) + retained = list(self.scoped_steps[:index]) + if tuple(retained) != self.completed_steps[:index]: + raise PipelineCheckpointError("自动流水线 checkpoint 的完成顺序已损坏") + previous = self.step_record(step) + self.state["completed_steps"] = retained + steps = self.state.setdefault("steps", {}) + for later in self.scoped_steps[index + 1 :]: + steps.pop(later, None) + steps[step] = { + "state": "failed", + "attempts": max(1, int(previous.get("attempts") or 0)), + "started_at": str(previous.get("started_at") or _now_iso()), + "failed_at": _now_iso(), + "baseline": deepcopy(previous.get("baseline") or {}), + "outputs": deepcopy(previous.get("outputs") or {}), + "error": str(error), + } + self.state["current_step"] = step + self.state["last_error"] = str(error) + self._persist() + + def _require_next_step(self, step: str) -> None: + if step not in self.scoped_steps: + raise PipelineCheckpointError(f"未知的自动流水线步骤:{step}") + completed = self.completed_steps + expected = self.scoped_steps[len(completed)] if len(completed) < len(self.scoped_steps) else "" + if expected != step: + raise PipelineCheckpointError( + f"自动流水线 checkpoint 顺序冲突:期望 {expected or '全部完成'},收到 {step}" + ) + + def _persist(self) -> None: + self.state["updated_at"] = _now_iso() + self._validate() + updated = job_service.update_job_checkpoint(self.job_id, self.state) + if not updated: + raise job_service.JobLeaseLostError( + f"自动流水线 checkpoint 写入失败,Workflow Job 租约已失效:{self.job_id}" + ) + + def _validate(self) -> None: + if self.start_step not in self.ordered_steps: + raise PipelineCheckpointError(f"自动流水线起始步骤无效:{self.start_step}") + if self.state.get("kind") != AUTO_PIPELINE_CHECKPOINT_KIND: + raise PipelineCheckpointError("自动流水线 checkpoint 版本未知,拒绝静默重跑") + if str(self.state.get("task_id") or "") != self.task_id: + raise PipelineCheckpointError("自动流水线 checkpoint 属于其他 Task") + if str(self.state.get("run_key") or "") != self.run_key: + raise PipelineCheckpointError("任务输入或自动配置已变化,请创建新的流水线 Job") + if str(self.state.get("start_step") or "") != self.start_step: + raise PipelineCheckpointError("自动流水线 checkpoint 的起始步骤与 Job payload 不一致") + + completed = list(self.completed_steps) + if completed != list(self.scoped_steps[: len(completed)]): + raise PipelineCheckpointError("自动流水线 checkpoint 的完成步骤不是连续前缀") + steps = self.state.get("steps") + if not isinstance(steps, dict): + raise PipelineCheckpointError("自动流水线 checkpoint.steps 不是对象") + unknown = set(steps) - set(self.scoped_steps) + if unknown: + raise PipelineCheckpointError(f"自动流水线 checkpoint 包含未知步骤:{sorted(unknown)}") + for step, record in steps.items(): + if not isinstance(record, dict) or record.get("state") not in _STEP_STATES: + raise PipelineCheckpointError(f"自动流水线 checkpoint 步骤状态无效:{step}") + if step in completed and record.get("state") != "succeeded": + raise PipelineCheckpointError(f"自动流水线已完成步骤缺少 succeeded 证据:{step}") + if not isinstance(record.get("outputs") or {}, dict): + raise PipelineCheckpointError(f"自动流水线步骤输出证据无效:{step}") + current = self.current_step + if current and current not in self.scoped_steps: + raise PipelineCheckpointError(f"自动流水线 current_step 无效:{current}") + if current and current in completed: + raise PipelineCheckpointError("自动流水线 current_step 与 completed_steps 冲突") + if current: + expected = self.scoped_steps[len(completed)] if len(completed) < len(self.scoped_steps) else "" + if current != expected: + raise PipelineCheckpointError("自动流水线 current_step 不是首个未完成步骤") + expected_records = set(completed) + if current: + expected_records.add(current) + if set(steps) != expected_records: + raise PipelineCheckpointError("自动流水线 checkpoint 包含缺失或越序的步骤记录") diff --git a/app/services/pipeline_engine.py b/app/services/pipeline_engine.py index 79ae240..2d6c6c4 100644 --- a/app/services/pipeline_engine.py +++ b/app/services/pipeline_engine.py @@ -2,22 +2,30 @@ from __future__ import annotations +import hashlib import json from datetime import datetime, time, timedelta from pathlib import Path from typing import Any +from app.core.config import settings from app.db.database import get_connection from app.models.task import TaskStatus -from app.services import task_service +from app.services import job_service, task_service from app.services.auto_publish_service import create_auto_publish_jobs, platforms_for_task from app.services.metadata_generator import MetadataGenerator +from app.services.pipeline_checkpoint_service import ( + AutoPipelineCheckpoint, + PipelineCheckpointError, +) from app.services.publish_service import generate_publish_cover_for_item from app.services.publish_time import next_allowed_schedule_time from app.services.storage_service import ( + IMAGE_EXTENSIONS, create_task_directory, get_artifact_paths, get_source_video_path, + resolve_task_media_file_path, validate_source_video_path, ) from app.services.task_log_service import append_task_log @@ -30,6 +38,7 @@ TaskStatus.AI_ANALYZING, TaskStatus.CLIP_SELECTING, TaskStatus.VIDEO_CUTTING, + TaskStatus.SUBTITLE_DRAFTING, TaskStatus.METADATA_GENERATING, TaskStatus.SCHEDULE_CREATING, TaskStatus.PUBLISH_JOB_CREATING, @@ -41,6 +50,7 @@ TaskStatus.AI_ANALYZING: TaskStatus.FAILED_AI_ANALYZING, TaskStatus.CLIP_SELECTING: TaskStatus.FAILED_CLIP_SELECTING, TaskStatus.VIDEO_CUTTING: TaskStatus.FAILED_VIDEO_CUTTING, + TaskStatus.SUBTITLE_DRAFTING: TaskStatus.FAILED_SUBTITLE_DRAFTING, TaskStatus.METADATA_GENERATING: TaskStatus.FAILED_METADATA_GENERATING, TaskStatus.SCHEDULE_CREATING: TaskStatus.FAILED_SCHEDULE_CREATING, TaskStatus.PUBLISH_JOB_CREATING: TaskStatus.FAILED_PUBLISH_JOB_CREATING, @@ -52,6 +62,7 @@ TaskStatus.FAILED_AI_ANALYZING.value: TaskStatus.AI_ANALYZING, TaskStatus.FAILED_CLIP_SELECTING.value: TaskStatus.CLIP_SELECTING, TaskStatus.FAILED_VIDEO_CUTTING.value: TaskStatus.VIDEO_CUTTING, + TaskStatus.FAILED_SUBTITLE_DRAFTING.value: TaskStatus.SUBTITLE_DRAFTING, TaskStatus.FAILED_METADATA_GENERATING.value: TaskStatus.METADATA_GENERATING, TaskStatus.FAILED_SCHEDULE_CREATING.value: TaskStatus.SCHEDULE_CREATING, TaskStatus.FAILED_PUBLISH_JOB_CREATING.value: TaskStatus.PUBLISH_JOB_CREATING, @@ -70,6 +81,10 @@ } +class PipelineCancelledError(RuntimeError): + """用户请求取消当前全自动流水线。""" + + class PipelineEngine: """只做流程调度,复用现有转写、AI、切片和发布任务服务。""" @@ -78,18 +93,39 @@ def run( task_id: str, retry: bool = False, start_step: TaskStatus | str | None = None, + job_id: str | None = None, ) -> dict: task = self._get_task(task_id) if not task.get("auto_mode"): raise ValueError("该任务未开启 auto_mode,手动流程不会被自动流水线接管。") - config = self._load_auto_config(task) resolved_start_step = ( start_step if isinstance(start_step, TaskStatus) else TaskStatus(start_step) if start_step else self._resolve_start_step(task, retry=retry) ) - if not retry and start_step is None: + config = self._load_auto_config(task) + checkpoint: AutoPipelineCheckpoint | None = None + if job_id: + active_lease = job_service.current_job_lease() + if not active_lease or active_lease[0] != job_id: + raise job_service.JobLeaseLostError( + f"自动流水线缺少当前 Workflow Job 租约:{job_id}" + ) + checkpoint = AutoPipelineCheckpoint.load( + job_id=job_id, + task_id=task_id, + start_step=resolved_start_step.value, + run_key=self._pipeline_run_key(task, config, resolved_start_step), + ordered_steps=[step.value for step in STEP_STATUSES], + ) + + if checkpoint and checkpoint.has_history: + append_task_log( + task_id, + f"从持久化 checkpoint 恢复全自动流水线:{resolved_start_step.value}", + ) + elif not retry and start_step is None: task_service.update_task_status(task_id, TaskStatus.CREATED) append_task_log(task_id, "全自动流水线启动") elif start_step is not None: @@ -97,7 +133,7 @@ def run( else: append_task_log(task_id, f"从失败步骤重试全自动流水线:{resolved_start_step.value}") - context: dict[str, Any] = {"config": config} + context: dict[str, Any] = {"config": config, "workflow_job_id": job_id or ""} steps = STEP_STATUSES[STEP_STATUSES.index(resolved_start_step) :] handlers = { TaskStatus.PREPARING_SOURCE: self._prepare_source, @@ -105,6 +141,7 @@ def run( TaskStatus.AI_ANALYZING: self._run_ai_analysis, TaskStatus.CLIP_SELECTING: self._select_clips, TaskStatus.VIDEO_CUTTING: self._cut_video, + TaskStatus.SUBTITLE_DRAFTING: self._prepare_subtitle_drafts, TaskStatus.METADATA_GENERATING: self._generate_metadata, TaskStatus.SCHEDULE_CREATING: self._create_schedule, TaskStatus.PUBLISH_JOB_CREATING: self._create_publish_jobs, @@ -112,34 +149,1159 @@ def run( for step in steps: try: - task_service.update_task_status(task_id, step) - context[step.value] = handlers[step](task_id, context) + if job_id: + self._raise_if_cancelled(job_id) + step_index = STEP_STATUSES.index(step) + job_service.update_job_progress( + job_id, + 5 + round(step_index / max(1, len(STEP_STATUSES)) * 90), + f"正在执行:{step.value}", + ) + job_service.heartbeat_job(job_id) + + if checkpoint and checkpoint.is_completed(step.value): + context[step.value] = self._restore_checkpoint_step( + task_id, + step, + checkpoint.step_record(step.value).get("outputs") or {}, + ) + append_task_log(task_id, f"checkpoint 证据有效,跳过已完成步骤:{step.value}") + else: + recovered = None + record = checkpoint.step_record(step.value) if checkpoint else {} + if checkpoint and checkpoint.current_step == step.value and record.get("state") == "running": + recovered = self._reconcile_interrupted_step(task_id, step, record) + if recovered is not None: + outputs = self._checkpoint_outputs(task_id, step, recovered) + checkpoint.complete_step(step.value, outputs=outputs, recovered=True) + context[step.value] = recovered + append_task_log(task_id, f"已根据持久化产物恢复中断步骤:{step.value}") + else: + if checkpoint: + checkpoint.begin_step( + step.value, + baseline=self._checkpoint_baseline(task_id, step), + ) + task_service.update_task_status(task_id, step) + context[step.value] = handlers[step](task_id, context) + if job_id: + job_service.heartbeat_job(job_id) + self._raise_if_cancelled(job_id) + if checkpoint: + checkpoint.complete_step( + step.value, + outputs=self._checkpoint_outputs(task_id, step, context[step.value]), + ) + if step == TaskStatus.SUBTITLE_DRAFTING: + return self._pending_subtitle_review_result(task_id) + except PipelineCancelledError as exc: + return self._cancel_pipeline(task_id, str(exc), context) + except job_service.JobLeaseLostError: + raise + except PipelineCheckpointError as exc: + error = str(exc) + if checkpoint: + if checkpoint.is_completed(step.value): + checkpoint.invalidate_from(step.value, error) + else: + checkpoint.fail_step(step.value, error) + return self._pipeline_failure(task_id, step, error) except Exception as exc: - failed_status = FAILED_BY_STEP[step] error = str(exc) or f"{step.value} 失败" - task_service.update_task_status(task_id, failed_status, error) - append_task_log(task_id, f"全自动流水线失败:{step.value},原因:{error}") - summary = self._write_task_summary(task_id, "failed", error) - return { - "status": "failed", - "failed_step": step.value, - "failed_status": failed_status.value, - "last_error": error, - "summary_path": summary["summary_path"], - "task": task_service.get_task(task_id, include_video_probe=False), - } + if checkpoint: + checkpoint.fail_step(step.value, error) + return self._pipeline_failure(task_id, step, error) - task_service.update_task_status(task_id, TaskStatus.READY_TO_PUBLISH) + if job_id: + try: + self._raise_if_cancelled(job_id) + except PipelineCancelledError as exc: + return self._cancel_pipeline(task_id, str(exc), context) + if not self._mark_ready_to_publish(task_id, job_id): + return self._cancel_pipeline(task_id, "用户已取消全自动流水线", context) append_task_log(task_id, "发布任务已创建,进入待人工确认发布状态") - summary = self._write_task_summary(task_id, "ready_to_publish", "") - task_service.update_task_status(task_id, TaskStatus.COMPLETED) - append_task_log(task_id, "全自动流水线完成。本轮已跳过字幕烧录,只保留原视频裁切片段。") + summary = self._safe_write_task_summary(task_id, "ready_to_publish", "") + append_task_log(task_id, "全自动流水线准备完成,发布内容等待人工确认。") + return { + "status": "ready_to_publish", + "summary_path": summary["summary_path"], + "task": task_service.get_task(task_id, include_video_probe=False), + } + + def _raise_if_cancelled(self, job_id: str) -> None: + if job_service.is_cancel_requested(job_id): + raise PipelineCancelledError("用户已取消全自动流水线") + + def _mark_ready_to_publish(self, task_id: str, job_id: str | None) -> bool: + if not job_id: + task_service.update_task_status(task_id, TaskStatus.READY_TO_PUBLISH) + return True + + active_lease = job_service.current_job_lease() + if not active_lease or active_lease[0] != job_id: + raise job_service.JobLeaseLostError( + f"Workflow Job 没有当前执行代际,不能写入 READY:{job_id}" + ) + _, lease_owner, lease_token = active_lease + with get_connection() as connection: + connection.execute("BEGIN IMMEDIATE") + now = task_service._now_iso() + job = connection.execute( + """ + SELECT status, cancel_requested, lease_owner, lease_token, lease_expires_at + FROM workflow_jobs WHERE id = ? + """, + (job_id,), + ).fetchone() + if not job: + connection.rollback() + raise ValueError("全自动 Workflow Job 不存在") + if int(job["cancel_requested"] or 0) or job["status"] == job_service.JOB_STATUS_CANCELLED: + connection.rollback() + return False + if ( + job["status"] != job_service.JOB_STATUS_RUNNING + or job["lease_owner"] != lease_owner + or job["lease_token"] != lease_token + or not job["lease_expires_at"] + or str(job["lease_expires_at"]) <= now + ): + connection.rollback() + raise job_service.JobLeaseLostError(f"Workflow Job 租约已失效,不能写入 READY:{job_id}") + final_now = task_service._now_iso() + cursor = connection.execute( + """ + UPDATE tasks + SET status = ?, progress = 100, error_message = NULL, + last_error = NULL, updated_at = ? + WHERE id = ? AND COALESCE(is_deleted, 0) = 0 + AND EXISTS ( + SELECT 1 FROM workflow_jobs + WHERE id = ? AND task_id = ? AND status = ? + AND lease_owner = ? AND lease_token = ? + AND lease_expires_at > ? AND cancel_requested = 0 + ) + """, + ( + TaskStatus.READY_TO_PUBLISH.value, + final_now, + task_id, + job_id, + task_id, + job_service.JOB_STATUS_RUNNING, + lease_owner, + lease_token, + final_now, + ), + ) + if cursor.rowcount != 1: + connection.rollback() + raise job_service.JobLeaseLostError( + f"Workflow Job 租约已失效或任务不可写,不能写入 READY:{job_id}" + ) + connection.commit() + return True + + def _cancel_pipeline(self, task_id: str, message: str, context: dict | None = None) -> dict: + cancelled_publish_jobs = self._cancel_unpublished_auto_jobs(task_id, context or {}) + task_service.update_task_status(task_id, TaskStatus.CANCELLED, message) + append_task_log(task_id, f"全自动流水线已取消:{message}") + summary = self._safe_write_task_summary(task_id, "cancelled", message) return { - "status": "completed", + "status": "cancelled", + "last_error": message, + "cancelled_publish_jobs": cancelled_publish_jobs, "summary_path": summary["summary_path"], "task": task_service.get_task(task_id, include_video_probe=False), } + def _cancel_unpublished_auto_jobs(self, task_id: str, context: dict) -> int: + publish_result = context.get(TaskStatus.PUBLISH_JOB_CREATING.value) or {} + created_ids = [ + str(item.get("id") or "") + for item in publish_result.get("created") or [] + if isinstance(item, dict) and item.get("id") + ] + if not created_ids: + return 0 + placeholders = ", ".join("?" for _ in created_ids) + now = task_service._now_iso() + lease_clause = "" + lease_params: tuple[str, ...] = () + workflow_job_id = str(context.get("workflow_job_id") or "") + if workflow_job_id: + active_lease = job_service.require_active_job_lease() + if not active_lease or active_lease[0] != workflow_job_id: + raise job_service.JobLeaseLostError( + f"取消发布草稿时 Workflow Job 租约已失效:{workflow_job_id}" + ) + _, lease_owner, lease_token = active_lease + lease_clause = """ + AND EXISTS ( + SELECT 1 FROM workflow_jobs + WHERE id = ? AND task_id = ? AND status = ? + AND lease_owner = ? AND lease_token = ? + AND julianday(lease_expires_at) > julianday('now') + ) + """ + lease_params = ( + workflow_job_id, + task_id, + job_service.JOB_STATUS_RUNNING, + lease_owner, + lease_token, + ) + with get_connection() as connection: + cursor = connection.execute( + f""" + UPDATE publish_jobs + SET status = 'CANCELLED', error_code = 'pipeline_cancelled', + error_message = '全自动流水线已取消', last_error = '全自动流水线已取消', + finished_at = ?, updated_at = ? + WHERE task_id = ? AND id IN ({placeholders}) + AND status IN ('DRAFT', 'WAITING', 'SCHEDULED', 'NEED_REVIEW') + {lease_clause} + """, + (now, now, task_id, *created_ids, *lease_params), + ) + connection.commit() + if cursor.rowcount: + append_task_log(task_id, f"已取消本轮新建但尚未发布的发送任务:{cursor.rowcount} 条") + return cursor.rowcount + + def _pending_subtitle_review_result(self, task_id: str) -> dict: + summary = self._safe_write_task_summary( + task_id, + TaskStatus.PENDING_SUBTITLE_REVIEW.value, + "", + ) + return { + "status": "pending_subtitle_review", + "message": "字幕草稿已生成,等待人工审核后再继续。", + "summary_path": summary["summary_path"], + "task": task_service.get_task(task_id, include_video_probe=False), + } + + def _pipeline_failure(self, task_id: str, step: TaskStatus, error: str) -> dict: + failed_status = FAILED_BY_STEP[step] + task_service.update_task_status(task_id, failed_status, error) + append_task_log(task_id, f"全自动流水线失败:{step.value},原因:{error}") + summary = self._safe_write_task_summary(task_id, "failed", error) + return { + "status": "failed", + "failed_step": step.value, + "failed_status": failed_status.value, + "last_error": error, + "summary_path": summary["summary_path"], + "task": task_service.get_task(task_id, include_video_probe=False), + } + + def _pipeline_run_key(self, task: dict, config: dict, start_step: TaskStatus) -> str: + from app.services.transcription_checkpoint_service import fingerprint_file + + source = get_source_video_path(task) + source_path = str(Path(source).resolve()) if source else "" + source_fingerprint = "" + if source and Path(source).exists() and Path(source).is_file(): + source_fingerprint = fingerprint_file(source) + payload = { + "task_id": task.get("id") or "", + "source_path": source_path, + "source_fingerprint": source_fingerprint, + "start_step": start_step.value, + "selection_profile": task.get("selection_profile") or "", + "candidate_clip_count": int(task.get("candidate_clip_count") or 0), + "final_clip_target": int(task.get("final_clip_target") or 0), + "max_clip_duration": int(task.get("max_clip_duration") or 0), + "config": config, + } + encoded = json.dumps(payload, ensure_ascii=False, sort_keys=True, separators=(",", ":"), default=str) + return hashlib.sha256(encoded.encode("utf-8")).hexdigest() + + def _enabled_candidate_ids(self, task_id: str) -> list[str]: + with get_connection() as connection: + rows = connection.execute( + """ + SELECT id FROM clip_candidates + WHERE task_id = ? AND enabled = 1 AND COALESCE(is_deleted, 0) = 0 + ORDER BY id + """, + (task_id,), + ).fetchall() + return [str(row["id"]) for row in rows] + + def _active_output_ids(self, task_id: str) -> list[str]: + return sorted( + str(item.get("id") or "") + for item in task_service.list_output_clips(task_id) + if item.get("status") == "completed" and item.get("file_exists") and item.get("id") + ) + + def _cut_output_evidence(self, task_id: str, cut_run_id: str) -> list[dict]: + from app.services.transcription_checkpoint_service import fingerprint_file + + task = self._get_task(task_id) + with get_connection() as connection: + rows = connection.execute( + """ + SELECT id, clip_candidate_id, cut_run_id, output_file_path + FROM output_clip + WHERE task_id = ? AND status = 'completed' AND is_active = 1 + ORDER BY id + """, + (task_id,), + ).fetchall() + if not rows: + raise PipelineCheckpointError("切片步骤没有 active output") + + evidence: list[dict] = [] + for row in rows: + row_run_id = str(row["cut_run_id"] or "") + if not row_run_id or row_run_id != cut_run_id: + raise PipelineCheckpointError("active output 不属于当前 cut run") + path = resolve_task_media_file_path( + str(row["output_file_path"] or ""), + task_id=task_id, + task_dir_name=task.get("task_dir_name"), + allowed_subdirectories=("05_clips", "clips"), + ) + if path is None or not path.exists() or not path.is_file(): + raise PipelineCheckpointError("切片输出文件缺失或不在当前任务目录") + size_bytes = int(path.stat().st_size) + if size_bytes <= 0: + raise PipelineCheckpointError("切片输出文件为空") + evidence.append( + { + "id": str(row["id"]), + "clip_candidate_id": str(row["clip_candidate_id"] or ""), + "cut_run_id": row_run_id, + "file": { + "path": str(path.resolve()), + "size_bytes": size_bytes, + "fingerprint": fingerprint_file(path), + }, + } + ) + return evidence + + def _publish_job_evidence(self, task_id: str, job_ids: list[str]) -> list[dict]: + normalized_ids = sorted({str(job_id) for job_id in job_ids if job_id}) + if not normalized_ids: + raise PipelineCheckpointError("发布任务 checkpoint 没有记录任何草稿") + placeholders = ", ".join("?" for _ in normalized_ids) + with get_connection() as connection: + rows = connection.execute( + f""" + SELECT pj.id, pj.output_clip_id, pj.platform, pj.publish_mode, pj.video_source, + pj.video_file_path, pj.cover_file_path, pj.scheduled_at, pj.status, + pj.title, pj.description, pj.caption, pj.tags, pj.hashtags, pj.cover_text, + pj.error_code, pj.provider_response, + oc.status AS output_status, oc.is_active AS output_is_active, + oc.cut_run_id, cr.status AS cut_run_status, cr.is_active AS cut_run_is_active + FROM publish_jobs pj + LEFT JOIN output_clip oc ON oc.id = pj.output_clip_id AND oc.task_id = pj.task_id + LEFT JOIN cut_runs cr ON cr.id = oc.cut_run_id AND cr.task_id = oc.task_id + WHERE pj.task_id = ? AND pj.id IN ({placeholders}) + ORDER BY pj.id + """, + (task_id, *normalized_ids), + ).fetchall() + if len(rows) != len(normalized_ids): + raise PipelineCheckpointError("发布任务 checkpoint 对应的草稿已缺失") + + evidence: list[dict] = [] + task = self._get_task(task_id) + allowed_statuses = {"DRAFT", "WAITING", "SCHEDULED", "NEED_REVIEW"} + for row in rows: + try: + provider_response = json.loads(str(row["provider_response"] or "{}")) + except json.JSONDecodeError as exc: + raise PipelineCheckpointError("发布任务 provider_response 已损坏") from exc + if not isinstance(provider_response, dict): + raise PipelineCheckpointError("发布任务 provider_response 不是对象") + if str(row["output_status"] or "") != "completed" or not bool(row["output_is_active"]): + raise PipelineCheckpointError("发布任务对应的切片已失活或未完成") + if row["cut_run_id"] and ( + str(row["cut_run_status"] or "") not in {"completed", "completed_with_errors"} + or not bool(row["cut_run_is_active"]) + ): + raise PipelineCheckpointError("发布任务对应的切片批次已失活或未完成") + status = str(row["status"] or "").upper() + if status not in allowed_statuses and not ( + status == "CANCELLED" and str(row["error_code"] or "") == "user_removed" + ): + raise PipelineCheckpointError(f"发布任务不在可恢复草稿状态:{status or 'UNKNOWN'}") + video_source = str(row["video_source"] or "") + video_subdirectories = ( + ("06_subtitled",) + if video_source == "subtitled" + else ("05_clips", "clips") + ) + video_path = resolve_task_media_file_path( + str(row["video_file_path"] or ""), + task_id=task_id, + task_dir_name=task.get("task_dir_name"), + allowed_subdirectories=video_subdirectories, + ) + cover_path = resolve_task_media_file_path( + str(row["cover_file_path"] or ""), + task_id=task_id, + task_dir_name=task.get("task_dir_name"), + allowed_subdirectories=("07_covers",), + allowed_extensions=IMAGE_EXTENSIONS, + ) + if video_path is None or not video_path.is_file() or video_path.stat().st_size <= 0: + raise PipelineCheckpointError("发布草稿的视频文件无效") + if cover_path is None or not cover_path.is_file() or cover_path.stat().st_size <= 0: + raise PipelineCheckpointError("发布草稿的封面文件无效") + from app.services.transcription_checkpoint_service import fingerprint_file + + video_fingerprint = fingerprint_file(video_path) + cover_fingerprint = fingerprint_file(cover_path) + if ( + int(provider_response.get("video_file_size") or -1) != video_path.stat().st_size + or str(provider_response.get("video_file_fingerprint") or "") != video_fingerprint + or int(provider_response.get("cover_file_size") or -1) != cover_path.stat().st_size + or str(provider_response.get("cover_file_fingerprint") or "") != cover_fingerprint + ): + raise PipelineCheckpointError("发布草稿媒体与创建时指纹不一致") + + evidence.append( + { + "id": str(row["id"]), + "output_clip_id": str(row["output_clip_id"] or ""), + "platform": str(row["platform"] or ""), + "publish_mode": str(row["publish_mode"] or ""), + "video_source": video_source, + "video_file_path": str(video_path.resolve()), + "video_file_size": int(video_path.stat().st_size), + "video_file_fingerprint": video_fingerprint, + "cover_file_path": str(cover_path.resolve()), + "cover_file": self._file_evidence(cover_path), + "subtitle_delivery_mode": str( + provider_response.get("subtitle_delivery_mode") or "" + ), + "workflow_job_id": str(provider_response.get("workflow_job_id") or ""), + "scheduled_at": str(row["scheduled_at"] or ""), + "status": status, + "title": str(row["title"] or ""), + "description": str(row["description"] or ""), + "caption": str(row["caption"] or ""), + "tags": str(row["tags"] or ""), + "hashtags": str(row["hashtags"] or ""), + "cover_text": str(row["cover_text"] or ""), + } + ) + return evidence + + def _verify_publish_schedule_pairs(self, schedule_path: Path, job_evidence: list[dict]) -> None: + scheduled_items = self._read_json_list(schedule_path) + self._validate_metadata_items(scheduled_items) + expected_pairs = sorted( + ( + str((item.get("output_clip") or {}).get("id") or ""), + str((item.get("metadata") or {}).get("platform") or ""), + ) + for item in scheduled_items + ) + actual_pairs = sorted( + (str(item.get("output_clip_id") or ""), str(item.get("platform") or "")) + for item in job_evidence + ) + if not expected_pairs or expected_pairs != actual_pairs: + raise PipelineCheckpointError("发布草稿与当前 schedule 的切片/平台不一致") + + def _checkpoint_baseline(self, task_id: str, step: TaskStatus) -> dict: + paths = get_artifact_paths(task_id) + if step == TaskStatus.AI_ANALYZING: + with get_connection() as connection: + row = connection.execute( + """ + SELECT id FROM ai_analysis_runs + WHERE task_id = ? AND COALESCE(is_active, 0) = 1 + ORDER BY run_number DESC LIMIT 1 + """, + (task_id,), + ).fetchone() + return { + "active_ai_run_id": str(row["id"] or "") if row else "", + "artifact": self._safe_file_evidence(paths["analysis_path"]), + "input_transcript": self._safe_file_evidence(paths["transcript_path"]), + } + if step == TaskStatus.VIDEO_CUTTING: + with get_connection() as connection: + row = connection.execute( + "SELECT COALESCE(MAX(run_number), 0) AS value FROM cut_runs WHERE task_id = ?", + (task_id,), + ).fetchone() + return { + "max_cut_run_number": int(row["value"] or 0), + "selected_candidate_ids": self._enabled_candidate_ids(task_id), + } + if step == TaskStatus.METADATA_GENERATING: + return { + "artifact": self._safe_file_evidence( + paths["analysis_path"].parent / "auto_publish_metadata.json" + ), + "output_clip_ids": self._active_output_ids(task_id), + } + if step == TaskStatus.SCHEDULE_CREATING: + return { + "artifact": self._safe_file_evidence( + paths["analysis_path"].parent / "auto_publish_schedule.json" + ), + "metadata_input": self._safe_file_evidence( + paths["analysis_path"].parent / "auto_publish_metadata.json" + ), + } + if step == TaskStatus.PUBLISH_JOB_CREATING: + return { + "schedule": self._safe_file_evidence( + paths["analysis_path"].parent / "auto_publish_schedule.json" + ) + } + artifact_by_step = { + TaskStatus.PREPARING_SOURCE: paths["task_dir"] / "source" / "source_reference.json", + TaskStatus.TRANSCRIBING: paths["transcript_path"], + TaskStatus.CLIP_SELECTING: paths["analysis_path"].parent / "auto_selected_clips.json", + } + artifact = artifact_by_step.get(step) + if artifact: + baseline = {"artifact": self._safe_file_evidence(artifact)} + if step == TaskStatus.CLIP_SELECTING: + baseline["input_analysis"] = self._safe_file_evidence(paths["analysis_path"]) + return baseline + if step == TaskStatus.SUBTITLE_DRAFTING: + from app.services.subtitle_data_service import list_task_tracks + + return { + "track_ids": sorted( + str(item.get("id") or "") + for item in list_task_tracks(task_id, ensure=False) + if item.get("id") + ), + "output_clip_ids": self._active_output_ids(task_id), + } + return {} + + def _checkpoint_outputs(self, task_id: str, step: TaskStatus, result: dict) -> dict: + paths = get_artifact_paths(task_id) + if step == TaskStatus.PREPARING_SOURCE: + return { + "source_reference": self._file_evidence( + paths["task_dir"] / "source" / "source_reference.json" + ), + "source_fingerprint": str(result.get("source_fingerprint") or ""), + } + if step == TaskStatus.TRANSCRIBING: + return { + "transcript": self._file_evidence(paths["transcript_path"]), + "source": str(result.get("source") or ""), + } + if step == TaskStatus.AI_ANALYZING: + run_id = str(result.get("analysis_run_id") or "") + if not run_id: + raise PipelineCheckpointError("AI 分析完成但没有 analysis_run_id") + return { + "analysis_run_id": run_id, + "analysis": self._file_evidence(paths["analysis_path"]), + "input_transcript": self._file_evidence(paths["transcript_path"]), + "clip_count": int(result.get("clip_count") or len(result.get("clips") or [])), + } + if step == TaskStatus.CLIP_SELECTING: + selected_ids = [ + str(item.get("clip_id") or "") + for item in result.get("selected") or [] + if isinstance(item, dict) and item.get("clip_id") + ] + return { + "selection": self._file_evidence( + paths["analysis_path"].parent / "auto_selected_clips.json" + ), + "input_analysis": self._file_evidence(paths["analysis_path"]), + "selected_ids": selected_ids, + } + if step == TaskStatus.VIDEO_CUTTING: + run_id = str(result.get("cut_run_id") or "") + if not run_id: + raise PipelineCheckpointError("切片步骤完成但缺少有效 cut_run 或输出文件") + output_files = self._cut_output_evidence(task_id, run_id) + return { + "cut_run_id": run_id, + "cut_run_number": int(result.get("cut_run_number") or 0), + "output_files": output_files, + "output_clip_ids": [item["id"] for item in output_files], + "selected_candidate_ids": self._enabled_candidate_ids(task_id), + } + if step == TaskStatus.SUBTITLE_DRAFTING: + tracks = result.get("tracks") or [] + return { + "source_track_id": str(result.get("source_track_id") or ""), + "track_ids": sorted( + str(item.get("id") or "") + for item in tracks + if isinstance(item, dict) and item.get("id") + ), + "track_count": int(result.get("track_count") or len(tracks)), + "output_clip_ids": self._active_output_ids(task_id), + } + if step == TaskStatus.METADATA_GENERATING: + return { + "metadata": self._file_evidence( + paths["analysis_path"].parent / "auto_publish_metadata.json" + ), + "metadata_count": int(result.get("metadata_count") or 0), + "need_review_count": int(result.get("need_review_count") or 0), + "output_clip_ids": self._active_output_ids(task_id), + } + if step == TaskStatus.SCHEDULE_CREATING: + scheduled_items = result.get("scheduled_items") or [] + return { + "schedule": self._file_evidence( + paths["analysis_path"].parent / "auto_publish_schedule.json" + ), + "metadata_input": self._file_evidence( + paths["analysis_path"].parent / "auto_publish_metadata.json" + ), + "scheduled_count": len(scheduled_items), + } + if step == TaskStatus.PUBLISH_JOB_CREATING: + created_ids = [ + str(item.get("id") or "") + for item in result.get("created") or [] + if isinstance(item, dict) and item.get("id") + ] + skipped_ids = [ + str(item.get("id") or "") + for item in result.get("skipped") or [] + if isinstance(item, dict) and item.get("id") + ] + schedule_path = paths["analysis_path"].parent / "auto_publish_schedule.json" + job_evidence = self._publish_job_evidence(task_id, [*created_ids, *skipped_ids]) + self._verify_publish_schedule_pairs(schedule_path, job_evidence) + return { + "created_ids": created_ids, + "skipped_ids": skipped_ids, + "created_count": int(result.get("created_count") or len(created_ids)), + "skipped_count": int(result.get("skipped_count") or len(skipped_ids)), + "schedule_input": self._file_evidence(schedule_path), + "job_evidence": job_evidence, + } + raise PipelineCheckpointError(f"没有定义 checkpoint 证据的步骤:{step.value}") + + def _restore_checkpoint_step(self, task_id: str, step: TaskStatus, outputs: dict) -> dict: + paths = get_artifact_paths(task_id) + if step == TaskStatus.PREPARING_SOURCE: + reference_path = paths["task_dir"] / "source" / "source_reference.json" + self._verify_file_evidence(reference_path, outputs.get("source_reference") or {}) + payload = self._read_json_object(reference_path) + source_path = str(payload.get("source_path") or "") + valid, error = validate_source_video_path(source_path) + if not valid: + raise PipelineCheckpointError(f"准备素材 checkpoint 已失效:{error}") + return {**payload, "source_fingerprint": outputs.get("source_fingerprint") or ""} + if step == TaskStatus.TRANSCRIBING: + transcript_path = paths["transcript_path"] + self._verify_file_evidence(transcript_path, outputs.get("transcript") or {}) + if not self._has_text_file(transcript_path): + raise PipelineCheckpointError("转写 checkpoint 对应的 transcript.md 为空") + return { + "source": str(outputs.get("source") or "checkpoint"), + "transcript_path": str(transcript_path), + } + if step == TaskStatus.AI_ANALYZING: + analysis_path = paths["analysis_path"] + task_service.ensure_task_ai_analysis_artifact(task_id) + self._verify_file_evidence( + paths["transcript_path"], + outputs.get("input_transcript") or {}, + ) + self._verify_file_evidence(analysis_path, outputs.get("analysis") or {}) + analysis_payload = self._read_json_object(analysis_path) + run_id = str(outputs.get("analysis_run_id") or "") + with get_connection() as connection: + row = connection.execute( + """ + SELECT id, clip_count, is_active, analysis_payload_json + FROM ai_analysis_runs + WHERE id = ? AND task_id = ? + """, + (run_id, task_id), + ).fetchone() + candidate_rows = connection.execute( + """ + SELECT clip_key FROM clip_candidates + WHERE task_id = ? AND COALESCE(is_deleted, 0) = 0 + ORDER BY clip_key + """, + (task_id,), + ).fetchall() + try: + run_payload = json.loads(str(row["analysis_payload_json"] or "{}")) if row else {} + except json.JSONDecodeError as exc: + raise PipelineCheckpointError("AI run 的分析 JSON 已损坏") from exc + clips = analysis_payload.get("clips") + if not isinstance(clips, list) or not all(isinstance(item, dict) for item in clips): + raise PipelineCheckpointError("AI checkpoint 的 clips 结构无效") + analysis_clip_ids = sorted(str(item.get("clip_id") or "") for item in clips) + candidate_clip_ids = sorted(str(item["clip_key"] or "") for item in candidate_rows) + if ( + not row + or not int(row["is_active"] or 0) + or int(row["clip_count"] or 0) <= 0 + or run_payload != analysis_payload + or not analysis_clip_ids + or "" in analysis_clip_ids + or candidate_clip_ids != analysis_clip_ids + or len(candidate_clip_ids) != int(row["clip_count"] or 0) + ): + raise PipelineCheckpointError("AI checkpoint 与 active run/候选片段不一致") + return { + "analysis_run_id": run_id, + "analysis_path": str(analysis_path), + "clip_count": int(row["clip_count"] or 0), + } + if step == TaskStatus.CLIP_SELECTING: + selection_path = paths["analysis_path"].parent / "auto_selected_clips.json" + self._verify_file_evidence( + paths["analysis_path"], + outputs.get("input_analysis") or {}, + ) + self._verify_file_evidence(selection_path, outputs.get("selection") or {}) + payload = self._read_json_object(selection_path) + selected_ids = {str(item) for item in outputs.get("selected_ids") or [] if item} + with get_connection() as connection: + enabled_ids = { + str(row["id"]) + for row in connection.execute( + """ + SELECT id FROM clip_candidates + WHERE task_id = ? AND enabled = 1 AND COALESCE(is_deleted, 0) = 0 + """, + (task_id,), + ).fetchall() + } + if not selected_ids or enabled_ids != selected_ids: + raise PipelineCheckpointError("选片 checkpoint 与当前启用候选不一致") + return {**payload, "selection_path": str(selection_path)} + if step == TaskStatus.VIDEO_CUTTING: + run_id = str(outputs.get("cut_run_id") or "") + with get_connection() as connection: + run = connection.execute( + """ + SELECT run_number, status, is_active FROM cut_runs + WHERE id = ? AND task_id = ? + """, + (run_id, task_id), + ).fetchone() + if ( + not run + or run["status"] not in {"completed", "completed_with_errors"} + or not int(run["is_active"] or 0) + ): + raise PipelineCheckpointError("切片 checkpoint 对应的 active cut run 不存在") + current_output_files = self._cut_output_evidence(task_id, run_id) + expected_output_files = outputs.get("output_files") + if not isinstance(expected_output_files, list) or current_output_files != expected_output_files: + raise PipelineCheckpointError("切片 checkpoint 与当前输出文件证据不一致") + output_ids = [item["id"] for item in current_output_files] + if output_ids != sorted(str(item) for item in outputs.get("output_clip_ids") or []): + raise PipelineCheckpointError("切片 checkpoint 与当前有效输出 ID 不一致") + selected_ids = self._enabled_candidate_ids(task_id) + output_candidate_ids = sorted( + item["clip_candidate_id"] for item in current_output_files if item["clip_candidate_id"] + ) + if ( + selected_ids != sorted(str(item) for item in outputs.get("selected_candidate_ids") or []) + or output_candidate_ids != selected_ids + ): + raise PipelineCheckpointError("切片 checkpoint 与当前启用候选边界不一致") + return { + "cut_run_id": run_id, + "cut_run_number": int(run["run_number"] or 0), + "success_count": len(current_output_files), + "failed_count": 0, + "failed": [], + } + if step == TaskStatus.SUBTITLE_DRAFTING: + from app.services.subtitle_data_service import list_task_tracks + + tracks = list_task_tracks(task_id, ensure=False) + track_ids = sorted(str(item.get("id") or "") for item in tracks if item.get("id")) + expected = sorted( + { + str(outputs.get("source_track_id") or ""), + *(str(item) for item in outputs.get("track_ids") or [] if item), + } + - {""} + ) + task = self._get_task(task_id) + if ( + not expected + or track_ids != expected + or self._active_output_ids(task_id) + != sorted(str(item) for item in outputs.get("output_clip_ids") or []) + or task.get("status") != TaskStatus.PENDING_SUBTITLE_REVIEW.value + ): + raise PipelineCheckpointError("字幕草稿 checkpoint 与当前审核状态不一致") + clip_tracks = [item for item in tracks if item.get("track_type") == "clip"] + return { + "status": "pending_subtitle_review", + "source_track_id": str(outputs.get("source_track_id") or ""), + "track_count": len(clip_tracks), + "tracks": clip_tracks, + } + if step == TaskStatus.METADATA_GENERATING: + metadata_path = paths["analysis_path"].parent / "auto_publish_metadata.json" + self._verify_file_evidence(metadata_path, outputs.get("metadata") or {}) + if self._active_output_ids(task_id) != sorted( + str(item) for item in outputs.get("output_clip_ids") or [] + ): + raise PipelineCheckpointError("文案 checkpoint 与当前 active output 不一致") + metadata_items = self._read_json_list(metadata_path) + self._validate_metadata_items(metadata_items) + task = self._get_task(task_id) + metadata_pairs = { + ( + str((item.get("output_clip") or {}).get("id") or ""), + str((item.get("metadata") or {}).get("platform") or ""), + ) + for item in metadata_items + } + expected_pairs = { + (output_id, platform) + for output_id in self._active_output_ids(task_id) + for platform in platforms_for_task(task) + } + if metadata_pairs != expected_pairs: + raise PipelineCheckpointError("文案 checkpoint 未完整覆盖当前 active output 与平台") + return { + "metadata_path": str(metadata_path), + "metadata_count": len(metadata_items), + "need_review_count": sum( + 1 for item in metadata_items if (item.get("metadata") or {}).get("risk_flags") + ), + "metadata_items": metadata_items, + } + if step == TaskStatus.SCHEDULE_CREATING: + schedule_path = paths["analysis_path"].parent / "auto_publish_schedule.json" + self._verify_file_evidence( + paths["analysis_path"].parent / "auto_publish_metadata.json", + outputs.get("metadata_input") or {}, + ) + self._verify_file_evidence(schedule_path, outputs.get("schedule") or {}) + scheduled_items = self._read_json_list(schedule_path) + self._validate_metadata_items(scheduled_items) + return {"schedule_path": str(schedule_path), "scheduled_items": scheduled_items} + if step == TaskStatus.PUBLISH_JOB_CREATING: + self._verify_file_evidence( + paths["analysis_path"].parent / "auto_publish_schedule.json", + outputs.get("schedule_input") or {}, + ) + created_ids = [str(item) for item in outputs.get("created_ids") or [] if item] + skipped_ids = [str(item) for item in outputs.get("skipped_ids") or [] if item] + all_ids = [*created_ids, *skipped_ids] + current_evidence = self._publish_job_evidence(task_id, all_ids) + expected_evidence = outputs.get("job_evidence") + if not isinstance(expected_evidence, list) or current_evidence != expected_evidence: + raise PipelineCheckpointError("发布任务 checkpoint 与当前草稿字段不一致") + self._verify_publish_schedule_pairs( + paths["analysis_path"].parent / "auto_publish_schedule.json", + current_evidence, + ) + from app.services.publish_service import get_publish_job + + return { + "created": [get_publish_job(job_id) for job_id in created_ids], + "skipped": [get_publish_job(job_id) for job_id in skipped_ids], + "created_count": len(created_ids), + "skipped_count": len(skipped_ids), + } + raise PipelineCheckpointError(f"无法恢复未知步骤:{step.value}") + + def _reconcile_interrupted_step( + self, + task_id: str, + step: TaskStatus, + record: dict, + ) -> dict | None: + baseline = record.get("baseline") if isinstance(record.get("baseline"), dict) else {} + paths = get_artifact_paths(task_id) + if step == TaskStatus.AI_ANALYZING: + task_service.ensure_task_ai_analysis_artifact(task_id) + baseline_transcript = ( + baseline.get("input_transcript") + if isinstance(baseline.get("input_transcript"), dict) + else {} + ) + if ( + not baseline_transcript + or self._safe_file_evidence(paths["transcript_path"]) != baseline_transcript + ): + return None + with get_connection() as connection: + row = connection.execute( + """ + SELECT id, clip_count FROM ai_analysis_runs + WHERE task_id = ? AND COALESCE(is_active, 0) = 1 + ORDER BY run_number DESC LIMIT 1 + """, + (task_id,), + ).fetchone() + if row and str(row["id"] or "") != str(baseline.get("active_ai_run_id") or ""): + provisional = { + "analysis_run_id": str(row["id"]), + "analysis_path": str(paths["analysis_path"]), + "clip_count": int(row["clip_count"] or 0), + } + outputs = self._checkpoint_outputs(task_id, step, provisional) + return self._restore_checkpoint_step(task_id, step, outputs) + return None + if step == TaskStatus.VIDEO_CUTTING: + baseline_candidates = sorted( + str(item) for item in baseline.get("selected_candidate_ids") or [] if item + ) + if not baseline_candidates or self._enabled_candidate_ids(task_id) != baseline_candidates: + return None + with get_connection() as connection: + row = connection.execute( + """ + SELECT id, run_number FROM cut_runs + WHERE task_id = ? AND COALESCE(is_active, 0) = 1 + AND status IN ('completed', 'completed_with_errors') + ORDER BY run_number DESC LIMIT 1 + """, + (task_id,), + ).fetchone() + if row and int(row["run_number"] or 0) > int(baseline.get("max_cut_run_number") or 0): + provisional = { + "cut_run_id": str(row["id"]), + "cut_run_number": int(row["run_number"] or 0), + } + outputs = self._checkpoint_outputs(task_id, step, provisional) + return self._restore_checkpoint_step(task_id, step, outputs) + return None + if step == TaskStatus.SUBTITLE_DRAFTING: + from app.services.subtitle_data_service import list_task_tracks + + tracks = list_task_tracks(task_id, ensure=False) + source_tracks = [item for item in tracks if item.get("track_type") == "source"] + clip_tracks = [item for item in tracks if item.get("track_type") == "clip"] + current_track_ids = sorted(str(item.get("id") or "") for item in tracks if item.get("id")) + baseline_output_ids = sorted( + str(item) for item in baseline.get("output_clip_ids") or [] if item + ) + task = self._get_task(task_id) + if ( + source_tracks + and clip_tracks + and current_track_ids + and self._active_output_ids(task_id) == baseline_output_ids + and task.get("status") == TaskStatus.PENDING_SUBTITLE_REVIEW.value + ): + provisional = { + "status": "pending_subtitle_review", + "source_track_id": str(source_tracks[0].get("id") or ""), + "track_count": len(clip_tracks), + "tracks": clip_tracks, + } + outputs = self._checkpoint_outputs(task_id, step, provisional) + return self._restore_checkpoint_step(task_id, step, outputs) + return None + if step == TaskStatus.PUBLISH_JOB_CREATING: + schedule_path = paths["analysis_path"].parent / "auto_publish_schedule.json" + baseline_schedule = ( + baseline.get("schedule") if isinstance(baseline.get("schedule"), dict) else {} + ) + if not baseline_schedule or self._safe_file_evidence(schedule_path) != baseline_schedule: + return None + lease = job_service.current_job_lease() + if not lease: + raise job_service.JobLeaseLostError("发布草稿恢复缺少当前 Workflow Job 租约") + with get_connection() as connection: + rows = connection.execute( + "SELECT id, provider_response FROM publish_jobs WHERE task_id = ?", + (task_id,), + ).fetchall() + recovered_ids: list[str] = [] + for row in rows: + try: + provider_response = json.loads(str(row["provider_response"] or "{}")) + except json.JSONDecodeError: + continue + if ( + isinstance(provider_response, dict) + and str(provider_response.get("source") or "") == "auto_pipeline" + and str(provider_response.get("workflow_job_id") or "") == lease[0] + ): + recovered_ids.append(str(row["id"])) + if not recovered_ids: + return None + try: + job_evidence = self._publish_job_evidence(task_id, recovered_ids) + self._verify_publish_schedule_pairs(schedule_path, job_evidence) + except PipelineCheckpointError: + return None + from app.services.publish_service import get_publish_job + + provisional = { + "created": [get_publish_job(job_id) for job_id in recovered_ids], + "skipped": [], + "created_count": len(recovered_ids), + "skipped_count": 0, + } + outputs = self._checkpoint_outputs(task_id, step, provisional) + return self._restore_checkpoint_step(task_id, step, outputs) + + artifact_by_step = { + TaskStatus.PREPARING_SOURCE: paths["task_dir"] / "source" / "source_reference.json", + TaskStatus.TRANSCRIBING: paths["transcript_path"], + TaskStatus.CLIP_SELECTING: paths["analysis_path"].parent / "auto_selected_clips.json", + TaskStatus.METADATA_GENERATING: paths["analysis_path"].parent / "auto_publish_metadata.json", + TaskStatus.SCHEDULE_CREATING: paths["analysis_path"].parent / "auto_publish_schedule.json", + } + artifact = artifact_by_step.get(step) + if artifact: + if step == TaskStatus.CLIP_SELECTING: + baseline_analysis = ( + baseline.get("input_analysis") + if isinstance(baseline.get("input_analysis"), dict) + else {} + ) + if ( + not baseline_analysis + or self._safe_file_evidence(paths["analysis_path"]) != baseline_analysis + ): + return None + elif step == TaskStatus.METADATA_GENERATING: + baseline_outputs = sorted( + str(item) for item in baseline.get("output_clip_ids") or [] if item + ) + if not baseline_outputs or self._active_output_ids(task_id) != baseline_outputs: + return None + elif step == TaskStatus.SCHEDULE_CREATING: + baseline_metadata = ( + baseline.get("metadata_input") + if isinstance(baseline.get("metadata_input"), dict) + else {} + ) + if ( + not baseline_metadata + or self._safe_file_evidence( + paths["analysis_path"].parent / "auto_publish_metadata.json" + ) + != baseline_metadata + ): + return None + current = self._safe_file_evidence(artifact) + previous = baseline.get("artifact") if isinstance(baseline.get("artifact"), dict) else {} + if current and current != previous: + if step == TaskStatus.PREPARING_SOURCE: + provisional = self._read_json_object(artifact) + elif step == TaskStatus.TRANSCRIBING: + if not self._has_text_file(artifact): + return None + provisional = { + "source": "checkpoint", + "transcript_path": str(artifact), + } + elif step == TaskStatus.CLIP_SELECTING: + payload = self._read_json_object(artifact) + provisional = {**payload, "selection_path": str(artifact)} + elif step == TaskStatus.METADATA_GENERATING: + items = self._read_json_list(artifact) + self._validate_metadata_items(items) + provisional = { + "metadata_path": str(artifact), + "metadata_count": len(items), + "need_review_count": sum( + 1 for item in items if (item.get("metadata") or {}).get("risk_flags") + ), + "metadata_items": items, + } + else: + items = self._read_json_list(artifact) + self._validate_metadata_items(items) + provisional = {"schedule_path": str(artifact), "scheduled_items": items} + outputs = self._checkpoint_outputs(task_id, step, provisional) + return self._restore_checkpoint_step(task_id, step, outputs) + return None + + def _safe_file_evidence(self, path: Path) -> dict: + try: + return self._file_evidence(path) + except (OSError, PipelineCheckpointError): + return {} + + def _file_evidence(self, path: Path) -> dict: + resolved = path.resolve() + if not resolved.exists() or not resolved.is_file(): + raise PipelineCheckpointError(f"checkpoint 产物不存在:{resolved}") + digest = hashlib.sha256() + with resolved.open("rb") as source: + for chunk in iter(lambda: source.read(1024 * 1024), b""): + digest.update(chunk) + stat = resolved.stat() + return { + "path": str(resolved), + "size_bytes": int(stat.st_size), + "mtime_ns": int(stat.st_mtime_ns), + "sha256": digest.hexdigest(), + } + + def _verify_file_evidence(self, path: Path, evidence: dict) -> None: + if not isinstance(evidence, dict) or not evidence.get("sha256"): + raise PipelineCheckpointError(f"checkpoint 缺少文件校验值:{path}") + current = self._file_evidence(path) + if ( + current["path"] != str(evidence.get("path") or "") + or current["size_bytes"] != int(evidence.get("size_bytes") or -1) + or current["sha256"] != str(evidence.get("sha256") or "") + ): + raise PipelineCheckpointError(f"checkpoint 文件证据已变化:{path}") + + def _read_json_object(self, path: Path) -> dict: + try: + payload = json.loads(path.read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError) as exc: + raise PipelineCheckpointError(f"checkpoint JSON 无法读取:{path}:{exc}") from exc + if not isinstance(payload, dict): + raise PipelineCheckpointError(f"checkpoint JSON 不是对象:{path}") + return payload + + def _read_json_list(self, path: Path) -> list[dict]: + try: + payload = json.loads(path.read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError) as exc: + raise PipelineCheckpointError(f"checkpoint JSON 无法读取:{path}:{exc}") from exc + if not isinstance(payload, list) or not all(isinstance(item, dict) for item in payload): + raise PipelineCheckpointError(f"checkpoint JSON 不是对象列表:{path}") + return payload + + def _validate_metadata_items(self, items: list[dict]) -> None: + if not items: + raise PipelineCheckpointError("checkpoint 文案/排期列表为空") + for item in items: + output_clip = item.get("output_clip") + metadata = item.get("metadata") + if ( + not isinstance(output_clip, dict) + or not output_clip.get("id") + or not isinstance(metadata, dict) + or not metadata.get("platform") + ): + raise PipelineCheckpointError("checkpoint 文案/排期条目缺少切片或平台") + + def _safe_write_task_summary(self, task_id: str, status: str, error: str) -> dict: + try: + return self._write_task_summary(task_id, status, error) + except Exception as exc: + try: + append_task_log(task_id, f"任务汇总写入失败,但主流程状态已保留:{exc}") + except Exception: + pass + return {"summary_path": ""} + def _get_task(self, task_id: str) -> dict: task = task_service.get_task(task_id, include_video_probe=False) if not task: @@ -166,6 +1328,8 @@ def _resolve_start_step(self, task: dict, retry: bool) -> TaskStatus: return TaskStatus.PREPARING_SOURCE def _prepare_source(self, task_id: str, context: dict) -> dict: + from app.services.transcription_checkpoint_service import fingerprint_file + task = self._get_task(task_id) create_task_directory(task_id, task.get("task_dir_name")) source_path = get_source_video_path(task) @@ -182,6 +1346,7 @@ def _prepare_source(self, task_id: str, context: dict) -> dict: "source_path": str(source), "exists": source.exists(), "size_bytes": source.stat().st_size if source.exists() else 0, + "source_fingerprint": fingerprint_file(source), } reference_path.write_text(json.dumps(reference_payload, ensure_ascii=False, indent=2), encoding="utf-8") append_task_log(task_id, f"全自动准备视频完成:{source}") @@ -211,11 +1376,42 @@ def _run_ai_analysis(self, task_id: str, context: dict) -> dict: result = task_service.process_task_ai_analysis(task_id) clip_count = len(result.get("clips") or []) append_task_log(task_id, f"全自动 AI 分析完成,候选片段:{clip_count} 条") - return {"clip_count": clip_count, "analysis_path": result.get("analysis_path") or ""} + return { + "clip_count": clip_count, + "analysis_run_id": result.get("analysis_run_id") or "", + "analysis_path": result.get("analysis_path") or "", + "analysis_meta": (result.get("analysis_run") or {}).get("analysis_meta") or {}, + } def _select_clips(self, task_id: str, context: dict) -> dict: task = self._get_task(task_id) config = context["config"] + meta = task_service.get_task_ai_analysis_meta(task_id) + profile = str(task.get("selection_profile") or "general") + if not meta: + raise ValueError( + "AI 分析缺少可信的完整性元数据;请重新分析或恢复可信历史," + "当前不会进入自动切片或发送中心。" + ) + meta = task_service.validate_ai_analysis_meta_for_cut(meta, profile) + coverage = float(meta.get("coverage_percent") or 0) + if profile == "long_live_talk" and ( + meta.get("analysis_incomplete") or float(meta.get("coverage_ratio") or 0) < 0.90 + ): + raise ValueError( + f"长直播分析覆盖率仅 {coverage:.2f}%,低于 90%;" + "请重试 AI 分析补齐缺失窗口,当前不会进入自动切片或发送中心。" + ) + if meta.get("analysis_incomplete"): + raise ValueError( + f"{profile} AI 分析存在未完成单元,当前覆盖率 {coverage:.2f}%;" + "请重试 AI 分析补齐失败单元,当前不会进入自动切片或发送中心。" + ) + if meta.get("quality_degraded"): + raise ValueError( + f"{profile} AI 分析质量评审未完整通过;" + "候选可供人工检查,但当前不会进入自动切片或发送中心。" + ) candidates = self._list_raw_candidates(task_id) if not candidates: latest_run = task_service.get_latest_ai_analysis_run(task_id) @@ -280,11 +1476,12 @@ def _select_clips(self, task_id: str, context: dict) -> dict: json.dumps(payload, ensure_ascii=False, indent=2), encoding="utf-8", ) + payload["selection_path"] = str(paths["analysis_path"].parent / "auto_selected_clips.json") append_task_log(task_id, f"全自动选片完成:选中 {len(selected)} 条,跳过 {len(skipped)} 条") return payload def _cut_video(self, task_id: str, context: dict) -> dict: - append_task_log(task_id, "全自动模式:开始原视频裁切,本轮明确跳过字幕烧录") + append_task_log(task_id, "全自动模式:开始原视频裁切,完成后生成字幕草稿") result = task_service.process_task_video_cuts(task_id, sync_publish_jobs=False) output_clips = task_service.list_output_clips(task_id) success = [clip for clip in output_clips if clip.get("status") == "completed" and clip.get("file_exists")] @@ -293,7 +1490,19 @@ def _cut_video(self, task_id: str, context: dict) -> dict: raise ValueError("全部切片失败,未生成可发布的原视频裁切片段") self._write_clip_metadata(task_id, output_clips, metadata_items=[]) append_task_log(task_id, f"全自动原片切割完成:成功 {len(success)} 条,失败 {len(failed)} 条") - return {"success_count": len(success), "failed_count": len(failed), "failed": failed} + return { + "success_count": len(success), + "failed_count": len(failed), + "failed": failed, + "cut_run_id": result.get("cut_run_id") or "", + "cut_run_number": int(result.get("cut_run_number") or 0), + } + + def _prepare_subtitle_drafts(self, task_id: str, context: dict) -> dict: + from app.services.subtitle_auto_workflow_service import prepare_task_subtitle_review + + del context + return prepare_task_subtitle_review(task_id) def _generate_metadata(self, task_id: str, context: dict) -> dict: task = self._get_task(task_id) @@ -305,23 +1514,53 @@ def _generate_metadata(self, task_id: str, context: dict) -> dict: if not output_clips: raise ValueError("没有可生成文案的成功切片") generator = MetadataGenerator(use_ai=config["auto_metadata_use_ai"]) + paths = get_artifact_paths(task_id) + metadata_path = paths["analysis_path"].parent / "auto_publish_metadata.json" + persisted_items = self._read_json(metadata_path, []) + persisted_items = persisted_items if isinstance(persisted_items, list) else [] + reusable = { + (str((item.get("output_clip") or {}).get("id") or ""), str((item.get("metadata") or {}).get("platform") or ""), str(item.get("request_fingerprint") or "")): item + for item in persisted_items + if isinstance(item, dict) + } metadata_items = [] for output_clip in output_clips: - cover = generate_publish_cover_for_item( + platforms = platforms_for_task(task) + existing_cover = next( + ( + item.get("cover") + for item in persisted_items + if str((item.get("output_clip") or {}).get("id") or "") == str(output_clip.get("id") or "") + and isinstance(item.get("cover"), dict) + and Path(str((item.get("cover") or {}).get("cover_file_path") or "")).is_file() + ), + None, + ) + cover = existing_cover or generate_publish_cover_for_item( output_clip, preferred_time_seconds=output_clip.get("cover_time_seconds"), ) - for platform in platforms_for_task(task): + for platform in platforms: + request_fingerprint = self._metadata_request_fingerprint( + output_clip, + platform, + use_ai=bool(config["auto_metadata_use_ai"]), + ) + cached = reusable.get((str(output_clip.get("id") or ""), platform, request_fingerprint)) + if cached: + metadata_items.append(cached) + continue + job_service.require_active_job_lease() metadata_items.append( { "output_clip": output_clip, "metadata": generator.generate(output_clip, platform), "cover": cover, + "request_fingerprint": request_fingerprint, } ) - paths = get_artifact_paths(task_id) - metadata_path = paths["analysis_path"].parent / "auto_publish_metadata.json" - metadata_path.write_text(json.dumps(metadata_items, ensure_ascii=False, indent=2), encoding="utf-8") + self._write_json_atomic(metadata_path, metadata_items) + self._write_json_atomic(metadata_path, metadata_items) self._write_clip_metadata(task_id, output_clips, metadata_items) need_review = [item for item in metadata_items if item["metadata"].get("risk_flags")] append_task_log(task_id, f"全自动标题文案生成完成:{len(metadata_items)} 条,需复核 {len(need_review)} 条") @@ -332,6 +1571,46 @@ def _generate_metadata(self, task_id: str, context: dict) -> dict: "metadata_items": metadata_items, } + @staticmethod + def _metadata_request_fingerprint(output_clip: dict, platform: str, *, use_ai: bool) -> str: + provider = str(settings.ai_publish_provider or "").lower() if use_ai else "rule" + model = ( + settings.ai_codex_model + if provider == "codex" + else settings.ai_local_model + if provider == "local" + else settings.ai_publish_remote_model + if use_ai + else "" + ) + payload = { + "fingerprint_version": 1, + "output_clip_id": output_clip.get("id") or "", + "clip_candidate_id": output_clip.get("clip_candidate_id") or "", + "task_name": output_clip.get("task_name") or "", + "clip_title": output_clip.get("clip_title") or "", + "clip_summary": output_clip.get("clip_summary") or "", + "highlight_reason": output_clip.get("highlight_reason") or "", + "spread_value": output_clip.get("spread_value") or "", + "suggested_editing": output_clip.get("suggested_editing") or "", + "platform": platform, + "use_ai": use_ai, + "provider": provider, + "model": model, + "protocol": settings.ai_publish_remote_protocol if provider == "remote" else "", + } + raw = json.dumps(payload, ensure_ascii=False, sort_keys=True, separators=(",", ":")) + return hashlib.sha256(raw.encode("utf-8")).hexdigest() + + @staticmethod + def _write_json_atomic(path: Path, payload: Any) -> None: + job_service.require_active_job_lease() + path.parent.mkdir(parents=True, exist_ok=True) + temporary_path = path.with_name(f"{path.name}.tmp") + temporary_path.write_text(json.dumps(payload, ensure_ascii=False, indent=2), encoding="utf-8") + job_service.require_active_job_lease() + temporary_path.replace(path) + def _create_schedule(self, task_id: str, context: dict) -> dict: metadata_result = context.get(TaskStatus.METADATA_GENERATING.value) or {} metadata_items = metadata_result.get("metadata_items") or [] @@ -342,7 +1621,11 @@ def _create_schedule(self, task_id: str, context: dict) -> dict: schedule_path = paths["analysis_path"].parent / "auto_publish_schedule.json" schedule_path.write_text(json.dumps(scheduled_items, ensure_ascii=False, indent=2), encoding="utf-8") append_task_log(task_id, f"全自动发送队列准备完成:{len(scheduled_items)} 条,发布时间待发送中心设置") - return {"schedule_path": str(schedule_path), "scheduled_items": scheduled_items} + return { + "schedule_path": str(schedule_path), + "scheduled_count": len(scheduled_items), + "scheduled_items": scheduled_items, + } def _create_publish_jobs(self, task_id: str, context: dict) -> dict: task = self._get_task(task_id) @@ -350,7 +1633,15 @@ def _create_publish_jobs(self, task_id: str, context: dict) -> dict: scheduled_items = schedule_result.get("scheduled_items") or [] if not scheduled_items: raise ValueError("没有可创建发布任务的排期记录") - result = create_auto_publish_jobs(task, scheduled_items) + delivery_mode = str(context["config"].get("subtitle_delivery_mode") or "") + if delivery_mode not in {"subtitled", "original"}: + raise ValueError("尚未确认字幕交付方式,不能创建发布任务") + result = create_auto_publish_jobs( + task, + scheduled_items, + subtitle_delivery_mode=delivery_mode, + workflow_job_id=str(context.get("workflow_job_id") or "") or None, + ) append_task_log( task_id, f"全自动发布任务创建完成:新增 {result['created_count']} 条,跳过已有 {result['skipped_count']} 条", @@ -390,6 +1681,8 @@ def _list_raw_candidates(self, task_id: str) -> list[dict]: def _resolve_target_count(self, task: dict, config: dict) -> int: if task.get("selection_profile") == "variety_comedy": return max(1, min(12, int(task.get("final_clip_target") or 5))) + if task.get("selection_profile") == "long_live_talk": + return max(1, min(50, int(task.get("highlight_total_limit") or 30))) return max(1, min(50, int(task.get("candidate_clip_count") or 12))) def _update_selected_clips(self, task_id: str, selected_ids: set[str]) -> None: @@ -456,6 +1749,11 @@ def _write_task_summary(self, task_id: str, status: str, error: str) -> dict: if error: failures.append(error) failures.extend(output_counts["errors"]) + next_step = ( + "打开字幕工作台审核草稿;确认后批量烧录,或明确跳过字幕继续使用原片。" + if status == TaskStatus.PENDING_SUBTITLE_REVIEW.value + else "打开发送中心核对内容、账号与北京时间;确认后可立即发送或设置排期。" + ) summary = { "task_id": task_id, "task_name": task.get("task_name") or "", @@ -473,10 +1771,11 @@ def _write_task_summary(self, task_id: str, status: str, error: str) -> dict: "publish_job_count": publish_job_count, "need_review_clips": need_review, "failures": failures, - "next_step": "打开发送中心核对内容、账号与北京时间;确认后可立即发送或设置排期。", - "subtitle_note": "v2.1 全自动模式继续跳过加字幕、字幕样式渲染和字幕烧录。", + "next_step": next_step, + "subtitle_note": "切片后必须人工确认字幕交付方式;只有已审核并验证的字幕成片才会自动进入发送中心。", } summary_path = paths["analysis_path"].parent / "task_summary.json" + summary_path.parent.mkdir(parents=True, exist_ok=True) summary_path.write_text(json.dumps(summary, ensure_ascii=False, indent=2), encoding="utf-8") summary["summary_path"] = str(summary_path) return summary @@ -586,8 +1885,9 @@ def run_auto_pipeline( task_id: str, retry: bool = False, start_step: TaskStatus | str | None = None, + job_id: str | None = None, ) -> dict: - return PipelineEngine().run(task_id, retry=retry, start_step=start_step) + return PipelineEngine().run(task_id, retry=retry, start_step=start_step, job_id=job_id) def start_auto_pipeline( @@ -597,6 +1897,36 @@ def start_auto_pipeline( start_step: TaskStatus | str | None = None, ) -> dict: if background_tasks is not None: - background_tasks.add_task(run_auto_pipeline, task_id, retry, start_step) - return {"status": "started", "message": "全自动流水线已在后台启动。", "task_id": task_id} + from app.services import job_service + + if retry: + job, requeued = job_service.retry_latest_or_get_active_job( + task_id, + job_service.JOB_TYPE_AUTO_PIPELINE, + ) + if job: + return { + "status": job["status"], + "message": ( + "已使用原 Workflow Job 的 checkpoint 重新加入队列。" + if requeued + else "全自动流水线已经在排队或运行。" + ), + "task_id": task_id, + "job_id": job["id"], + } + job, created = job_service.create_or_get_active_job( + task_id=task_id, + job_type=job_service.JOB_TYPE_AUTO_PIPELINE, + payload={ + "retry": retry, + "start_step": start_step.value if isinstance(start_step, TaskStatus) else start_step, + }, + ) + return { + "status": job["status"], + "message": "全自动流水线已加入持久化队列。" if created else "全自动流水线已经在排队或运行。", + "task_id": task_id, + "job_id": job["id"], + } return run_auto_pipeline(task_id, retry=retry, start_step=start_step) diff --git a/app/services/publish_domain.py b/app/services/publish_domain.py index 3551a58..b7541ef 100644 --- a/app/services/publish_domain.py +++ b/app/services/publish_domain.py @@ -1,5 +1,7 @@ """发布领域常量:目标平台、执行方式与状态。""" +from urllib.parse import urlsplit + TARGET_PLATFORMS = { "douyin": "抖音", "bilibili": "B站", @@ -30,6 +32,11 @@ TERMINAL_PUBLISH_STATUSES = {"PUBLISHED", "EXPORTED", "CANCELLED"} ACTIVE_PUBLISH_STATUSES = PUBLISH_STATUSES - TERMINAL_PUBLISH_STATUSES +_PLATFORM_URL_DOMAINS = { + "douyin": "douyin.com", + "bilibili": "bilibili.com", +} + def validate_target_platform(platform: str) -> str: value = (platform or "").strip().lower() @@ -43,3 +50,34 @@ def validate_publish_mode(publish_mode: str) -> str: if value not in PUBLISH_MODES: raise ValueError("不支持的发布执行方式") return value + + +def validate_platform_url(platform: str, platform_url: str, *, allow_empty: bool = True) -> str: + """校验可点击作品链接,避免 Provider 把危险 scheme 或外域链接带入页面。""" + target = validate_target_platform(platform) + value = str(platform_url or "").strip() + if not value and allow_empty: + return "" + expected_domain = _PLATFORM_URL_DOMAINS[target] + parsed = urlsplit(value) + hostname = str(parsed.hostname or "").lower().rstrip(".") + try: + _ = parsed.port + except ValueError as exc: + raise ValueError(f"请填写有效的 {expected_domain} 作品链接") from exc + if ( + parsed.scheme not in {"http", "https"} + or not hostname + or parsed.username + or parsed.password + or (hostname != expected_domain and not hostname.endswith(f".{expected_domain}")) + ): + raise ValueError(f"请填写有效的 {expected_domain} 作品链接") + return value + + +def safe_platform_url(platform: str, platform_url: str) -> str: + try: + return validate_platform_url(platform, platform_url) + except ValueError: + return "" diff --git a/app/services/publish_executor.py b/app/services/publish_executor.py index df139e3..18ce455 100644 --- a/app/services/publish_executor.py +++ b/app/services/publish_executor.py @@ -2,16 +2,27 @@ from __future__ import annotations +import threading from typing import Any, Callable from app.services.publish_repository import PublishRepository from app.services.publishers.registry import get_publisher +_ACTIVE_DISPATCHES: set[str] = set() +_ACTIVE_DISPATCHES_LOCK = threading.Lock() + + +def is_publish_dispatch_active(execution_id: str) -> bool: + with _ACTIVE_DISPATCHES_LOCK: + return execution_id in _ACTIVE_DISPATCHES + + def execute_publish_job( job_id: str, force: bool = False, *, + expected_execution_id: str | None = None, runner: Callable[[list[str]], Any] | None = None, repository: PublishRepository | None = None, worker_client=None, @@ -23,16 +34,68 @@ def execute_publish_job( from app.services.publishers.base import PublishValidationError raise PublishValidationError("发布任务不存在", "publish_job_not_found") + if expected_execution_id and ( + str(job.get("status") or "").upper() != "PUBLISHING" + or str(job.get("execution_id") or "") != expected_execution_id + ): + from app.services.publishers.base import PublishValidationError + + raise PublishValidationError("发布任务执行代际已变化", "publish_execution_stale") dependencies: dict[str, Any] = {"repository": repo, "runner": runner} if worker_client is not None: dependencies["worker_client"] = worker_client + publish_mode = str(job.get("publish_mode") or "").strip().lower() + dispatch_reserved = False + + def reserve_dispatch() -> None: + nonlocal dispatch_reserved + if dispatch_reserved: + return + if not expected_execution_id: + from app.services.publishers.base import PublishValidationError + + raise PublishValidationError("发布任务执行代际或状态已变化", "publish_execution_stale") + with _ACTIVE_DISPATCHES_LOCK: + if expected_execution_id in _ACTIVE_DISPATCHES: + from app.services.publishers.base import PublishValidationError + + raise PublishValidationError("发布任务正在提交,请勿重复执行", "publish_execution_active") + _ACTIVE_DISPATCHES.add(expected_execution_id) + try: + reserved = repo.begin_execution_dispatch( + job_id, + expected_execution_id, + str(job.get("updated_at") or ""), + ) + except Exception: + with _ACTIVE_DISPATCHES_LOCK: + _ACTIVE_DISPATCHES.discard(expected_execution_id) + raise + if not reserved: + with _ACTIVE_DISPATCHES_LOCK: + _ACTIVE_DISPATCHES.discard(expected_execution_id) + from app.services.publishers.base import PublishValidationError + + raise PublishValidationError("发布任务执行代际或状态已变化", "publish_execution_stale") + dispatch_reserved = True + + if publish_mode == "local_browser": + # 登录态检查不产生投稿副作用;把 CAS 推迟到 Worker /publish 前一刻, + # 避免长时间账号检查期间过早占用 dispatching 状态。 + dependencies["before_dispatch"] = reserve_dispatch publisher = get_publisher( str(job.get("platform") or ""), - str(job.get("publish_mode") or ""), + publish_mode, **dependencies, ) - result = publisher.publish(job) - # LocalBrowserPublisher 已即时记录 Worker 原始结果;其他模式在这里统一补写。 - if str(job.get("publish_mode") or "") != "local_browser": - repo.record_provider_result(job_id, result) + if publish_mode != "local_browser": + reserve_dispatch() + try: + result = publisher.publish(job) + finally: + if dispatch_reserved and expected_execution_id: + with _ACTIVE_DISPATCHES_LOCK: + _ACTIVE_DISPATCHES.discard(expected_execution_id) + # 平台结果与最终状态由 Scheduler 在同一 fenced 事务中落库,避免旧执行 + # 先写 provider_result、再被新 execution 接管后留下半成功数据。 return result.as_dict() diff --git a/app/services/publish_readiness.py b/app/services/publish_readiness.py index f47a8f1..c95387e 100644 --- a/app/services/publish_readiness.py +++ b/app/services/publish_readiness.py @@ -94,6 +94,54 @@ def _content_issues(job: dict[str, Any], platform: str, publish_mode: str) -> li return issues +def _subtitle_delivery_issues(job: dict[str, Any]) -> list[dict[str, Any]]: + provider_payload = job.get("provider_payload") if isinstance(job.get("provider_payload"), dict) else {} + delivery_mode = str( + job.get("subtitle_delivery_mode") + or provider_payload.get("subtitle_delivery_mode") + or "" + ) + video_source = str(job.get("video_source") or "original") + if delivery_mode == "original" and video_source != "original": + return [ + _issue( + "subtitle_delivery_mismatch", + "该任务已明确选择原片继续,当前视频源不一致", + "complete_content", + ) + ] + if delivery_mode == "subtitled" and video_source != "subtitled": + return [ + _issue( + "subtitle_review_required", + "该任务要求使用审核后的字幕成片,当前仍是原片", + "complete_content", + ) + ] + if video_source != "subtitled": + return [] + revision_id = str(job.get("subtitle_revision_id") or provider_payload.get("subtitle_revision_id") or "") + revision_status = str( + job.get("subtitle_revision_status") + or provider_payload.get("subtitle_revision_status") + or "" + ) + validation_status = str( + job.get("subtitle_validation_status") + or provider_payload.get("subtitle_validation_status") + or "" + ) + if not revision_id or revision_status != "approved" or validation_status != "verified": + return [ + _issue( + "subtitle_not_verified", + "带字幕成片缺少已审核 revision 或 FFprobe 验证证据", + "complete_content", + ) + ] + return [] + + def build_send_readiness( job: dict[str, Any], *, @@ -175,6 +223,7 @@ def build_send_readiness( ) ) + issues.extend(_subtitle_delivery_issues(job)) issues.extend(_content_issues(job, platform, resolved_mode)) if resolved_mode == "local_browser" and worker_available is False: diff --git a/app/services/publish_repository.py b/app/services/publish_repository.py index 51eeed2..626565a 100644 --- a/app/services/publish_repository.py +++ b/app/services/publish_repository.py @@ -21,16 +21,17 @@ def record_provider_result( job_id: str, result: PublishResult, *, + expected_execution_id: str | None = None, connection=None, updated_at: str | None = None, - ) -> None: - """保存脱敏平台结果;复用连接时由调用方统一提交事务。""" + ) -> bool: + """保存脱敏平台结果;可用 execution_id 拒绝旧执行写回。""" now = updated_at or utc_now_iso() provider_json = json.dumps( sanitize_provider_response(result.provider_response), ensure_ascii=False ) - result_json = json.dumps(result.as_dict(), ensure_ascii=False) + result_json = json.dumps(sanitize_provider_response(result.as_dict()), ensure_ascii=False) values = ( result.remote_video_id, result.remote_video_id, @@ -44,46 +45,89 @@ def record_provider_result( int(result.needs_manual_review), now, job_id, + *((expected_execution_id,) if expected_execution_id else ()), ) - sql = """ + execution_condition = ( + " AND status = 'PUBLISHING' AND execution_id = ?" + if expected_execution_id + else "" + ) + sql = f""" UPDATE publish_jobs SET remote_video_id = ?, platform_item_id = ?, platform_url = ?, provider_response = ?, publish_result = ?, published_at = ?, error_code = ?, last_error = ?, error_message = ?, needs_manual_review = ?, updated_at = ? - WHERE id = ? + WHERE id = ?{execution_condition} """ if connection is not None: - connection.execute(sql, values) - return + cursor = connection.execute(sql, values) + return int(cursor.rowcount or 0) == 1 with get_connection() as owned: - owned.execute(sql, values) + cursor = owned.execute(sql, values) owned.commit() + return int(cursor.rowcount or 0) == 1 def update_execution_phase( self, job_id: str, phase: str, details: dict[str, Any] | None = None, - ) -> None: - """同步 Worker 的实时阶段;不会在这里改变任务最终状态。""" + *, + expected_execution_id: str | None = None, + ) -> bool: + """同步 Worker 实时阶段;提供 execution 时拒绝旧执行写回。""" values = sanitize_provider_response(details or {}) message = str(values.get("message") or "") if isinstance(values, dict) else "" now = utc_now_iso() + execution_condition = " AND execution_id = ?" if expected_execution_id else "" + params = ( + phase, + message, + message, + message, + message, + now, + job_id, + *((expected_execution_id,) if expected_execution_id else ()), + ) with get_connection() as connection: - connection.execute( - """ + cursor = connection.execute( + f""" UPDATE publish_jobs SET execution_phase = ?, last_error = CASE WHEN ? <> '' THEN ? ELSE last_error END, error_message = CASE WHEN ? <> '' THEN ? ELSE error_message END, updated_at = ? - WHERE id = ? AND status = 'PUBLISHING' + WHERE id = ? AND status = 'PUBLISHING'{execution_condition} + """, + params, + ) + connection.commit() + return int(cursor.rowcount or 0) == 1 + + def begin_execution_dispatch( + self, + job_id: str, + expected_execution_id: str, + expected_updated_at: str, + ) -> bool: + """在外部投稿前原子保留 dispatch 权;与恢复扫描的快照写回互斥。""" + + now = utc_now_iso() + with get_connection() as connection: + cursor = connection.execute( + """ + UPDATE publish_jobs + SET execution_phase = 'dispatching', updated_at = ? + WHERE id = ? AND status = 'PUBLISHING' AND execution_id = ? + AND updated_at = ? """, - (phase, message, message, message, message, now, job_id), + (now, job_id, expected_execution_id, expected_updated_at), ) connection.commit() + return int(cursor.rowcount or 0) == 1 def add_event( self, diff --git a/app/services/publish_scheduler.py b/app/services/publish_scheduler.py index 6e02529..7675c3f 100644 --- a/app/services/publish_scheduler.py +++ b/app/services/publish_scheduler.py @@ -15,7 +15,8 @@ from app.core.config import settings from app.db.database import get_connection, init_db -from app.services.publish_executor import execute_publish_job +from app.services.publish_executor import execute_publish_job, is_publish_dispatch_active +from app.services.publish_domain import safe_platform_url, validate_platform_url from app.services.publish_readiness import ( PublishPlatformIsolationBlocked, SendReadinessBlocked, @@ -36,6 +37,7 @@ utc_now_iso, ) from app.services.publishers.base import ( + parse_public_json_dict, PublishError, PublishOutcome, PublishResult, @@ -152,6 +154,8 @@ def __init__( self._stop_event: asyncio.Event | None = None self._wake_event: asyncio.Event | None = None self._loop: asyncio.AbstractEventLoop | None = None + self._background_task: asyncio.Task[None] | None = None + self._shutdown_requested = False def run_once(self) -> dict[str, Any]: init_db() @@ -168,13 +172,21 @@ def run_once(self) -> dict[str, Any]: raise except Exception: logger.exception("发布任务执行出现未预期异常:%s", job.get("id")) - results.append( - self._mark_need_review( + current = self.repository.get_job(str(job["id"])) or {} + if str(current.get("status") or "").upper() == "SCHEDULED": + transition = self._mark_need_review( str(job["id"]), "unexpected_scheduler_error", "调度任务出现未预期异常,为避免重复投稿已转入人工复核", + expected_statuses=("SCHEDULED",), ) - ) + else: + transition = { + "status": "skipped", + "job_id": str(job["id"]), + "message": "任务已进入发布执行,保留给代际恢复流程处理", + } + results.append(transition) checked_at = utc_now() self._record_scan_success(checked_at) return { @@ -198,6 +210,8 @@ async def run_forever(self) -> None: self._loop = asyncio.get_running_loop() self._stop_event = asyncio.Event() self._wake_event = asyncio.Event() + if self._shutdown_requested: + self._stop_event.set() _ACTIVE_SCHEDULER = self _SCHEDULER_HEALTH["running"] = True try: @@ -252,11 +266,19 @@ def wake(self) -> None: self._loop.call_soon_threadsafe(self._wake_event.set) def stop(self) -> None: + self._shutdown_requested = True if self._loop and self._stop_event: self._loop.call_soon_threadsafe(self._stop_event.set) if self._wake_event: self._loop.call_soon_threadsafe(self._wake_event.set) + async def shutdown(self) -> None: + """停止扫描并等待当前一轮安全结束,避免应用退出时丢弃后台 Task。""" + self.stop() + task = self._background_task + if task and task is not asyncio.current_task(): + await task + def list_due_jobs(self) -> list[dict[str, Any]]: now = utc_now() due: list[dict[str, Any]] = [] @@ -297,6 +319,7 @@ def execute_job(self, job_id: str, *, force: bool = False, runner=None) -> dict[ job_id, "legacy_schedule_requires_confirmation", "旧版排期已暂停,未执行上传;请选择对应平台账号后逐条转换并发送", + expected_statuses=("SCHEDULED",), ) readiness = build_send_readiness( @@ -327,63 +350,111 @@ def execute_job(self, job_id: str, *, force: bool = False, runner=None) -> dict[ risk_flags = self._risk_flags(job) if risk_flags and not settings.publish_scheduler_allow_publish_without_review: - return self._mark_need_review(job_id, "risk_flags_require_review", f"内容风险标记需要人工复核:{risk_flags}") + return self._mark_need_review( + job_id, + "risk_flags_require_review", + f"内容风险标记需要人工复核:{risk_flags}", + expected_statuses=("SCHEDULED",), + ) try: due_at = parse_datetime(job.get("next_attempt_at") or job.get("scheduled_at")) except ValueError as exc: - return self._mark_failed(job_id, "invalid_scheduled_at", str(exc)) + return self._mark_failed( + job_id, "invalid_scheduled_at", str(exc), expected_statuses=("SCHEDULED",) + ) if not force and due_at > utc_now(): return {"status": "skipped", "job_id": job_id, "message": "尚未到计划发布时间"} max_attempts = max(1, int(job.get("max_attempts") or self.max_retry_count)) if not force and int(job.get("attempt_count") or 0) >= max_attempts: - return self._mark_failed(job_id, "max_retry_exceeded", "上传前安全重试次数已用完") - if not self._claim_scheduled_job(job_id): + return self._mark_failed( + job_id, + "max_retry_exceeded", + "上传前安全重试次数已用完", + expected_statuses=("SCHEDULED",), + ) + execution_id = self._claim_scheduled_job(job_id) + if not execution_id: return {"status": "skipped", "job_id": job_id, "message": "任务已被另一个调度器领取"} claimed = self.repository.get_job(job_id) or job try: raw_result = self.executor( job_id, force=force, + expected_execution_id=execution_id, runner=runner, repository=self.repository, worker_client=self.worker_client, ) result = PublishResult.from_dict(raw_result) + if ( + result.outcome == PublishOutcome.PUBLISHED + and (not result.published_at or result.needs_manual_review) + ): + invalid_result = result + result = PublishResult( + outcome=PublishOutcome.NEED_REVIEW, + message="Publisher 成功结果缺少时间证据或仍要求人工复核", + error_code="publish_result_inconsistent", + needs_manual_review=True, + provider_response={"invalid_result": invalid_result.as_dict()}, + ) except PublishWorkerUnavailable as exc: return self._handle_worker_unavailable(claimed, exc) except PublishError as exc: if exc.needs_manual_review: - return self._mark_need_review(job_id, exc.error_code, exc.message) - return self._mark_failed(job_id, exc.error_code, exc.message) + return self._mark_need_review( + job_id, exc.error_code, exc.message, expected_execution_id=execution_id + ) + return self._mark_failed( + job_id, exc.error_code, exc.message, expected_execution_id=execution_id + ) except Exception as exc: return self._mark_need_review( job_id, "publish_result_uncertain", f"执行器异常且无法确定是否已上传,请人工核对:{exc}", + expected_execution_id=execution_id, ) if result.outcome == PublishOutcome.PUBLISHED: - return self._mark_published(job_id, result) + return self._mark_published(job_id, result, expected_execution_id=execution_id) if result.outcome == PublishOutcome.EXPORTED: - return self._mark_exported(job_id, result) + return self._mark_exported(job_id, result, expected_execution_id=execution_id) if result.outcome == PublishOutcome.NEED_REVIEW or result.needs_manual_review: - return self._mark_need_review(job_id, result.error_code or "manual_review_required", result.message, result) - return self._mark_failed(job_id, result.error_code or "publish_failed", result.message, result) + return self._mark_need_review( + job_id, + result.error_code or "manual_review_required", + result.message, + result, + expected_execution_id=execution_id, + ) + return self._mark_failed( + job_id, + result.error_code or "publish_failed", + result.message, + result, + expected_execution_id=execution_id, + ) def publish_now(self, job_id: str) -> dict[str, Any]: job = self.repository.get_job(job_id) if not job: raise ValueError("发布任务不存在") - if str(job.get("status") or "").upper() not in {"DRAFT", "WAITING", "SCHEDULED"}: + status = str(job.get("status") or "").upper() + if status not in {"DRAFT", "WAITING", "SCHEDULED"}: raise ValueError("只有草稿、等待或已排期任务可以立即发送") readiness = self._require_ready_jobs([job_id], resolve_legacy=True, check_worker=True)[job_id] if str(job.get("publish_mode") or "") == "opencli_publish": - self._mark_need_review( + transition = self._mark_need_review( job_id, "legacy_schedule_requires_confirmation", "旧版任务已暂停;正在保留原记录并创建新的 Windows Chrome 投稿任务", + expected_statuses=(status,), + expected_updated_at=str(job.get("updated_at") or ""), ) + if transition.get("status") == "skipped": + raise ValueError("任务内容或状态已变化,请刷新页面后重试") return self.repair_and_publish( job_id, account_id=str(readiness.get("resolved_account_id") or ""), @@ -397,7 +468,7 @@ def publish_now(self, job_id: str) -> dict[str, Any]: timezone = ?, schedule_timezone = ?, status = 'SCHEDULED', error_code = '', error_message = '', last_error = '', needs_manual_review = 0, updated_at = ? - WHERE id = ? AND status IN ('DRAFT', 'WAITING', 'SCHEDULED') + WHERE id = ? AND status = ? AND updated_at = ? """, ( readiness["resolved_account_id"] or job.get("account_id") or None, @@ -407,6 +478,8 @@ def publish_now(self, job_id: str) -> dict[str, Any]: settings.app_timezone, now, job_id, + str(job.get("status") or ""), + str(job.get("updated_at") or ""), ), ) if cursor.rowcount: @@ -446,6 +519,17 @@ def retry_failed( )[job_id] resolved_mode = str(readiness.get("resolved_publish_mode") or source.get("publish_mode") or "") with get_connection() as connection: + connection.execute("BEGIN IMMEDIATE") + current_row = connection.execute( + "SELECT * FROM publish_jobs WHERE id = ?", (job_id,) + ).fetchone() + if not current_row or str(current_row["status"] or "").upper() != "FAILED": + connection.rollback() + raise ValueError("源任务状态已变化,请刷新发送中心后重试") + if str(current_row["updated_at"] or "") != str(source.get("updated_at") or ""): + connection.rollback() + raise ValueError("源任务内容已变化,请刷新发送中心后重试") + current = dict(current_row) active = connection.execute( """ SELECT id, status FROM publish_jobs @@ -455,28 +539,36 @@ def retry_failed( """, ( job_id, - source.get("output_clip_id"), - source.get("platform"), + current.get("output_clip_id"), + current.get("platform"), resolved_mode, ), ).fetchone() - if active: - raise ValueError( - f"同一视频已有任务 {active['id']} 处于 {active['status']};" - "如为人工复核,请先确认平台未发布并标记失败" + if active: + connection.rollback() + raise ValueError( + f"同一视频已有任务 {active['id']} 处于 {active['status']};" + "如为人工复核,请先确认平台未发布并标记失败" + ) + result = self._clone_job_for_retry( + current, + scheduled_at=schedule, + event_type="manual_retry_created", + event_message=f"由失败任务 {job_id} 创建", + event_from_status="FAILED", + overrides={ + "visibility": resolved_visibility, + "publish_mode": resolved_mode, + "account_id": readiness.get("resolved_account_id") + or current.get("account_id") + or None, + }, + connection=connection, ) - return self._clone_job_for_retry( - source, - scheduled_at=schedule, - event_type="manual_retry_created", - event_message=f"由失败任务 {job_id} 创建", - event_from_status="FAILED", - overrides={ - "visibility": resolved_visibility, - "publish_mode": resolved_mode, - "account_id": readiness.get("resolved_account_id") or source.get("account_id") or None, - }, - ) + connection.commit() + wake_scheduler() + result["job"] = self._public_job(str(result["job_id"])) + return result def repair_and_publish( self, @@ -494,6 +586,8 @@ def repair_and_publish( ) if str(source.get("status") or "").upper() != "NEED_REVIEW" or not source_readiness["repairable"]: raise ValueError("该任务不是明确发生在上传前的旧任务,不能自动修复;请先人工核对平台结果") + # 幂等快路径:已经成功创建替代任务时,不再重复做文件/Worker readiness。 + # 最终创建前仍会在 BEGIN IMMEDIATE 中再次检查,不能只依赖这里。 with get_connection() as connection: existing = connection.execute( """ @@ -530,29 +624,62 @@ def repair_and_publish( raise SendReadinessBlocked(readiness) if readiness["requires_worker"]: require_worker_available(self.worker_client) - result = self._clone_job_for_retry( - source, - scheduled_at=utc_now_iso(), - event_type="safe_repair_created", - event_message=f"由上传前失败任务 {job_id} 安全修复", - event_from_status="NEED_REVIEW", - overrides={ - "publish_mode": readiness["resolved_publish_mode"], - "account_id": readiness["resolved_account_id"] or None, - "visibility": resolved_visibility, - }, - ) + existing_id = "" with get_connection() as connection: - self.repository.add_event( - job_id, - "safe_repair_replacement_created", - from_status="NEED_REVIEW", - to_status="NEED_REVIEW", - message=f"已保留原记录并创建替代任务 {result['job_id']}", - payload={"replacement_job_id": result["job_id"]}, - connection=connection, - ) - connection.commit() + connection.execute("BEGIN IMMEDIATE") + current = connection.execute("SELECT * FROM publish_jobs WHERE id = ?", (job_id,)).fetchone() + existing = connection.execute( + """ + SELECT id FROM publish_jobs + WHERE retry_of_job_id = ? + ORDER BY created_at DESC LIMIT 1 + """, + (job_id,), + ).fetchone() + if existing: + existing_id = str(existing["id"]) + connection.commit() + elif not current or str(current["status"] or "").upper() != "NEED_REVIEW": + connection.rollback() + raise ValueError("源任务状态已变化,请刷新发送中心后重新确认") + elif str(current["updated_at"] or "") != str(source.get("updated_at") or ""): + connection.rollback() + raise ValueError("源任务内容已变化,请刷新发送中心后重新确认") + else: + result = self._clone_job_for_retry( + dict(current), + scheduled_at=utc_now_iso(), + event_type="safe_repair_created", + event_message=f"由上传前失败任务 {job_id} 安全修复", + event_from_status="NEED_REVIEW", + overrides={ + "publish_mode": readiness["resolved_publish_mode"], + "account_id": readiness["resolved_account_id"] or None, + "visibility": resolved_visibility, + }, + connection=connection, + ) + replacement_id = str(result["job_id"]) + self.repository.add_event( + job_id, + "safe_repair_replacement_created", + from_status="NEED_REVIEW", + to_status="NEED_REVIEW", + message=f"已保留原记录并创建替代任务 {replacement_id}", + payload={"replacement_job_id": replacement_id}, + connection=connection, + ) + connection.commit() + if existing_id: + return { + "status": "already_created", + "job_id": existing_id, + "source_job_id": job_id, + "job": self._public_job(existing_id), + "message": "该旧任务已经创建过替代任务,本次没有重复创建", + } + wake_scheduler() + result["job"] = self._public_job(str(result["job_id"])) result["message"] = "旧任务已保留,新的 Windows Chrome 投稿任务已进入调度器" result["source_job_id"] = job_id return result @@ -566,7 +693,25 @@ def _clone_job_for_retry( event_message: str, event_from_status: str, overrides: dict[str, Any] | None = None, + connection: sqlite3.Connection | None = None, ) -> dict[str, Any]: + """克隆重试任务;传入连接时由调用方统一提交和唤醒。""" + if connection is None: + with get_connection() as owned: + result = self._clone_job_for_retry( + source, + scheduled_at=scheduled_at, + event_type=event_type, + event_message=event_message, + event_from_status=event_from_status, + overrides=overrides, + connection=owned, + ) + owned.commit() + wake_scheduler() + result["job"] = self._public_job(str(result["job_id"])) + return result + source_id = str(source.get("id") or "") new_id = f"pub_{uuid4().hex}" columns_to_clear = { @@ -576,41 +721,38 @@ def _clone_job_for_retry( "error_message", "last_error", "provider_response", "publish_result", "published_at", "needs_manual_review", "created_at", "updated_at", "retry_of_job_id", } - with get_connection() as connection: - available = {row["name"] for row in connection.execute("PRAGMA table_info(publish_jobs)").fetchall()} - values = {key: value for key, value in source.items() if key in available and key not in columns_to_clear} - values.update({ - "id": new_id, - "status": "SCHEDULED", - "scheduled_at": scheduled_at, - "timezone": source.get("timezone") or settings.app_timezone, - "schedule_timezone": source.get("schedule_timezone") or settings.app_timezone, - "attempt_count": 0, - "retry_count": 0, - "max_attempts": source.get("max_attempts") or self.max_retry_count, - "needs_manual_review": 0, - "retry_of_job_id": source_id, - "created_at": utc_now_iso(), - "updated_at": utc_now_iso(), - }) - values.update(overrides or {}) - columns = list(values) - try: - connection.execute( - f"INSERT INTO publish_jobs ({', '.join(columns)}) VALUES ({', '.join('?' for _ in columns)})", - [values[column] for column in columns], - ) - except sqlite3.IntegrityError as exc: - raise ValueError( - "同一视频已经存在等待、排期、执行中或人工复核任务;请刷新发送中心后核对" - ) from exc - self.repository.add_event( - new_id, event_type, from_status=event_from_status, to_status="SCHEDULED", - message=event_message, payload={"retry_of_job_id": source_id}, connection=connection, + available = {row["name"] for row in connection.execute("PRAGMA table_info(publish_jobs)").fetchall()} + values = {key: value for key, value in source.items() if key in available and key not in columns_to_clear} + values.update({ + "id": new_id, + "status": "SCHEDULED", + "scheduled_at": scheduled_at, + "timezone": source.get("timezone") or settings.app_timezone, + "schedule_timezone": source.get("schedule_timezone") or settings.app_timezone, + "attempt_count": 0, + "retry_count": 0, + "max_attempts": source.get("max_attempts") or self.max_retry_count, + "needs_manual_review": 0, + "retry_of_job_id": source_id, + "created_at": utc_now_iso(), + "updated_at": utc_now_iso(), + }) + values.update(overrides or {}) + columns = list(values) + try: + connection.execute( + f"INSERT INTO publish_jobs ({', '.join(columns)}) VALUES ({', '.join('?' for _ in columns)})", + [values[column] for column in columns], ) - connection.commit() - wake_scheduler() - return {"status": "scheduled", "job_id": new_id, "retry_of_job_id": source_id, "job": self._public_job(new_id)} + except sqlite3.IntegrityError as exc: + raise ValueError( + "同一视频已经存在等待、排期、执行中或人工复核任务;请刷新发送中心后核对" + ) from exc + self.repository.add_event( + new_id, event_type, from_status=event_from_status, to_status="SCHEDULED", + message=event_message, payload={"retry_of_job_id": source_id}, connection=connection, + ) + return {"status": "scheduled", "job_id": new_id, "retry_of_job_id": source_id} def recover_interrupted_jobs(self) -> int: stale_before = utc_now() - timedelta(minutes=max(1, int(settings.publish_job_stale_minutes))) @@ -624,52 +766,104 @@ def recover_interrupted_jobs(self) -> int: except ValueError: updated_at = datetime.min.replace(tzinfo=timezone.utc) execution_id = str(job.get("execution_id") or "") + if execution_id and is_publish_dispatch_active(execution_id): + # 同一应用进程仍位于已原子保留的真实 dispatch 临界区;终态由执行线程写回。 + continue phase = str(job.get("execution_phase") or "unknown") details: dict[str, Any] = {} + worker_query_confirmed = False + execution_in_progress = False + execution_identity_valid = False # Worker 执行日志是跨进程恢复的唯一依据。只要有 execution_id 就主动查询, # 不依赖宿主 Worker 回写 SQLite,也不会因此重复调用投稿接口。 if execution_id: try: execution = self.worker_client.execution(execution_id) + worker_query_confirmed = True phase = str(execution.get("phase") or phase) details = execution.get("details") if isinstance(execution.get("details"), dict) else {} + execution_in_progress = bool(execution.get("in_progress")) + execution_identity_valid = self._worker_execution_matches_job(execution, job) except PublishError: - pass - if phase == "confirmed_success" and details: - try: - self._mark_published(job["id"], PublishResult.from_dict(details)) - except Exception: - self._mark_need_review(job["id"], "recovery_result_uncertain", "Worker 记录成功但结果数据不完整,请人工确认") - recovered += 1 - elif phase == "manual_review" and details: - result = PublishResult.from_dict(details) - self._mark_need_review( + worker_query_confirmed = False + transition: dict[str, Any] | None = None + if execution_id and phase in {"confirmed_success", "exported", "failed", "manual_review"}: + if not execution_identity_valid: + transition = self._mark_need_review( + job["id"], + "recovery_identity_mismatch", + "Worker 执行日志身份与当前发布任务不一致,请人工确认", + expected_execution_id=execution_id, + expected_updated_at=str(job.get("updated_at") or ""), + ) + else: + try: + result = self._parse_worker_terminal_result(phase, details) + if phase == "confirmed_success": + transition = self._mark_published( + job["id"], result, expected_execution_id=execution_id + ) + elif phase == "exported": + transition = self._mark_exported( + job["id"], result, expected_execution_id=execution_id + ) + elif phase == "failed": + transition = self._mark_failed( + job["id"], + result.error_code or "publish_failed", + result.message or "Worker 已确认发送失败", + result, + expected_execution_id=execution_id, + ) + else: + transition = self._mark_need_review( + job["id"], + result.error_code or "manual_review_required", + result.message or "Worker 已停止自动发送,请人工确认平台结果", + result, + expected_execution_id=execution_id, + ) + except (PublishError, ValueError, KeyError, TypeError): + transition = self._mark_need_review( + job["id"], + "recovery_result_uncertain", + "Worker 记录成功但结果数据不完整,请人工确认", + expected_execution_id=execution_id, + ) + elif updated_at > stale_before: + continue + elif worker_query_confirmed and not execution_identity_valid: + transition = self._mark_need_review( job["id"], - result.error_code or "manual_review_required", - result.message or "Worker 已停止自动发送,请人工确认平台结果", - result, + "recovery_identity_mismatch", + "Worker 执行日志缺少身份或与当前发布任务不一致,请人工确认", + expected_execution_id=execution_id, + expected_updated_at=str(job.get("updated_at") or ""), ) - recovered += 1 - elif phase == "failed" and details: - result = PublishResult.from_dict(details) - self._mark_failed( + elif worker_query_confirmed and execution_in_progress: + # Worker 仍持有该 execution 的跨进程锁;HTTP 超时不等于投稿未执行。 + # 保持 PUBLISHING,等待 journal 进入明确终态,绝不并行创建新 execution。 + continue + elif ( + worker_query_confirmed + and phase in {"received", "browser_opening", "browser_opened", "rejected"} + and execution_id + ): + transition = self._reschedule_before_upload( job["id"], - result.error_code or "publish_failed", - result.message or "Worker 已确认发送失败", - result, + "应用重启后确认尚未开始上传,已安全重新排队", + expected_execution_id=execution_id, + expected_updated_at=str(job.get("updated_at") or ""), ) - recovered += 1 - elif updated_at > stale_before: - continue - elif phase in {"received", "browser_opening", "browser_opened", "rejected"} and execution_id: - self._reschedule_before_upload(job["id"], "应用重启后确认尚未开始上传,已安全重新排队") - recovered += 1 else: - self._mark_need_review( + transition = self._mark_need_review( job["id"], "interrupted_publish_uncertain", "应用重启前的发布结果不确定,为避免重复投稿已停止自动重试", + expected_execution_id=execution_id, + expected_updated_at=str(job.get("updated_at") or ""), ) + if transition and transition.get("status") != "skipped": recovered += 1 return recovered @@ -683,15 +877,16 @@ def update_schedule(self, job_id: str, scheduled_at: str) -> dict[str, Any]: if str(job.get("publish_mode") or "") == "opencli_publish": raise ValueError("旧版任务不能直接改排期;请逐条使用“转换并发送”创建新的 Windows Chrome 任务") readiness = self._require_ready_jobs([job_id], resolve_legacy=True, check_worker=True)[job_id] + source_status = str(job.get("status") or "").upper() stored = to_utc_iso(parsed) now = utc_now_iso() with get_connection() as connection: - connection.execute( + cursor = connection.execute( """ UPDATE publish_jobs SET account_id = ?, publish_mode = ?, scheduled_at = ?, next_attempt_at = NULL, timezone = ?, schedule_timezone = ?, status = 'SCHEDULED', updated_at = ? - WHERE id = ? + WHERE id = ? AND status = ? AND updated_at = ? """, ( readiness["resolved_account_id"] or job.get("account_id") or None, @@ -701,14 +896,21 @@ def update_schedule(self, job_id: str, scheduled_at: str) -> dict[str, Any]: settings.app_timezone, now, job_id, + source_status, + job.get("updated_at"), ), ) - self._record_auto_target_resolution(job, readiness, connection=connection) - self.repository.add_event( - job_id, "schedule_updated", from_status=str(job.get("status") or ""), - to_status="SCHEDULED", payload={"scheduled_at": stored}, connection=connection, - ) - connection.commit() + if cursor.rowcount: + self._record_auto_target_resolution(job, readiness, connection=connection) + self.repository.add_event( + job_id, "schedule_updated", from_status=source_status, + to_status="SCHEDULED", payload={"scheduled_at": stored}, connection=connection, + ) + connection.commit() + else: + connection.rollback() + if not cursor.rowcount: + raise ValueError("任务状态已变化,请刷新页面后重试") wake_scheduler() return {"status": "ok", "job": self._public_job(job_id)} @@ -818,6 +1020,17 @@ def update_batch_schedule( confirmed_schedule: list[dict[str, str]] | None = None, ) -> dict[str, Any]: ids = self._validate_batch_jobs(job_ids, platform) + with get_connection() as connection: + expected_rows = connection.execute( + f"SELECT id, status, updated_at FROM publish_jobs WHERE id IN ({','.join('?' for _ in ids)})", + ids, + ).fetchall() + if len(expected_rows) != len(ids): + raise ValueError("部分发布任务已不存在,请刷新后重试") + expected_state = { + str(row["id"]): (str(row["status"] or "").upper(), str(row["updated_at"] or "")) + for row in expected_rows + } if action not in {"apply", "clear"}: raise ValueError("不支持的排期操作") if action == "apply": @@ -858,31 +1071,44 @@ def update_batch_schedule( } for job_id in ids] now = utc_now_iso() with get_connection() as connection: + connection.execute("BEGIN IMMEDIATE") rows = connection.execute( - f"SELECT id, status, account_id, publish_mode FROM publish_jobs WHERE id IN ({','.join('?' for _ in ids)})", ids + f"SELECT id, status, account_id, publish_mode, updated_at FROM publish_jobs WHERE id IN ({','.join('?' for _ in ids)})", + ids, ).fetchall() + if len(rows) != len(ids): + connection.rollback() + raise ValueError("部分发布任务已不存在,请刷新后重试") row_map = {row["id"]: dict(row) for row in rows} for item in schedule: job_id = item["job_id"] current = row_map[job_id] status = str(current["status"] or "").upper() if status not in {"DRAFT", "WAITING", "SCHEDULED"}: + connection.rollback() raise ValueError(f"任务 {job_id} 当前状态不能修改排期") + if expected_state.get(job_id) != (status, str(current.get("updated_at") or "")): + connection.rollback() + raise ValueError(f"任务 {job_id} 已被其他操作修改,请刷新后重试") next_status = "SCHEDULED" if action == "apply" else "WAITING" readiness = readiness_map.get(job_id) or {} - connection.execute( + cursor = connection.execute( """ UPDATE publish_jobs SET account_id = ?, publish_mode = ?, scheduled_at = ?, next_attempt_at = NULL, timezone = ?, schedule_timezone = ?, status = ?, updated_at = ? - WHERE id = ? + WHERE id = ? AND status = ? AND updated_at = ? """, ( readiness.get("resolved_account_id") or current.get("account_id") or None, readiness.get("resolved_publish_mode") or current.get("publish_mode"), - item["scheduled_at_utc"], timezone_name, timezone_name, next_status, now, job_id, + item["scheduled_at_utc"], timezone_name, timezone_name, next_status, now, + job_id, status, current.get("updated_at"), ), ) + if cursor.rowcount != 1: + connection.rollback() + raise ValueError(f"任务 {job_id} 状态已变化,请刷新后重试") if action == "apply": self._record_auto_target_resolution(current, readiness, connection=connection) self.repository.add_event( @@ -946,9 +1172,15 @@ def mark_failed_manually(self, job_id: str, message: str = "人工确认平台 job = self.repository.get_job(job_id) if not job: raise ValueError("发布任务不存在") - if str(job.get("status") or "").upper() not in {"NEED_REVIEW", "SCHEDULED", "WAITING"}: + status = str(job.get("status") or "").upper() + if status not in {"NEED_REVIEW", "SCHEDULED", "WAITING"}: raise ValueError("当前任务不能人工标记失败") - return self._mark_failed(job_id, "manually_marked_failed", message) + return self._mark_failed( + job_id, + "manually_marked_failed", + message, + expected_statuses=(status,), + ) def mark_published_manually(self, job_id: str, platform_url: str) -> dict[str, Any]: job = self.repository.get_job(job_id) @@ -956,10 +1188,7 @@ def mark_published_manually(self, job_id: str, platform_url: str) -> dict[str, A raise ValueError("发布任务不存在") if str(job.get("status") or "").upper() != "NEED_REVIEW": raise ValueError("只有需复核任务可以人工标记已发布") - url = str(platform_url or "").strip() - expected_domain = "douyin.com" if job.get("platform") == "douyin" else "bilibili.com" - if not url.startswith(("http://", "https://")) or expected_domain not in url.lower(): - raise ValueError(f"请填写有效的 {expected_domain} 作品链接") + url = validate_platform_url(str(job.get("platform") or ""), platform_url, allow_empty=False) result = PublishResult( outcome=PublishOutcome.PUBLISHED, message="人工核对平台后标记为已发布", @@ -1059,7 +1288,7 @@ def _public_job(self, job_id: str) -> dict[str, Any] | None: return publish_service.get_publish_job(job_id) or self.repository.get_job(job_id) - def _claim_scheduled_job(self, job_id: str) -> bool: + def _claim_scheduled_job(self, job_id: str) -> str | None: now = utc_now_iso() execution_id = uuid4().hex with get_connection() as connection: @@ -1083,77 +1312,232 @@ def _claim_scheduled_job(self, job_id: str) -> bool: worker_id=self.worker_id, payload={"execution_id": execution_id}, connection=connection, ) connection.commit() - return int(cursor.rowcount or 0) == 1 + return execution_id if int(cursor.rowcount or 0) == 1 else None def _handle_worker_unavailable(self, job: dict[str, Any], exc: PublishWorkerUnavailable) -> dict[str, Any]: execution_id = str(job.get("execution_id") or "") if exc.request_may_have_been_received and execution_id: try: execution = self.worker_client.execution(execution_id) + if not self._worker_execution_matches_job(execution, job): + return self._mark_need_review( + str(job["id"]), + "publish_worker_identity_mismatch", + "Worker 执行日志缺少身份或与当前发布任务不一致,请人工确认", + expected_execution_id=execution_id, + ) phase = str(execution.get("phase") or "unknown") details = execution.get("details") if isinstance(execution.get("details"), dict) else {} if phase == "confirmed_success" and details: - return self._mark_published(str(job["id"]), PublishResult.from_dict(details)) - if phase not in {"received", "browser_opening", "browser_opened", "rejected"}: + result = self._parse_worker_terminal_result(phase, details) + return self._mark_published( + str(job["id"]), + result, + expected_execution_id=execution_id, + ) + if phase == "manual_review" and details: + result = self._parse_worker_terminal_result(phase, details) + return self._mark_need_review( + str(job["id"]), + result.error_code or "manual_review_required", + result.message or "Worker 已停止自动发送,请人工确认平台结果", + result, + expected_execution_id=execution_id, + ) + if phase == "failed" and details: + result = self._parse_worker_terminal_result(phase, details) + return self._mark_failed( + str(job["id"]), + result.error_code or "publish_failed", + result.message or "Worker 已确认发送失败", + result, + expected_execution_id=execution_id, + ) + if phase in { + "received", "browser_opening", "browser_opened", "upload_started", + "upload_completed", "submit_clicked", + }: + return { + "status": "pending", + "job_id": job["id"], + "message": "Worker 已接收本次执行,保持 PUBLISHING 并等待恢复扫描确认结果", + } + if phase != "rejected": return self._mark_need_review( str(job["id"]), "publish_worker_result_uncertain", "Worker 连接中断且任务可能已经上传,请人工确认平台结果", + expected_execution_id=execution_id, ) - except PublishError: + except (PublishError, ValueError): return self._mark_need_review( str(job["id"]), "publish_worker_result_uncertain", "Worker 超时后无法读取执行阶段,为避免重复投稿已停止自动重试", + expected_execution_id=execution_id, ) - attempts = int((self.repository.get_job(str(job["id"])) or job).get("attempt_count") or 0) + attempts = int(job.get("attempt_count") or 0) max_attempts = max(1, int(job.get("max_attempts") or self.max_retry_count)) if attempts >= max_attempts: - return self._mark_failed(str(job["id"]), exc.error_code, f"{exc.message};3 次上传前安全重试已用完") + return self._mark_failed( + str(job["id"]), + exc.error_code, + f"{exc.message};3 次上传前安全重试已用完", + expected_execution_id=execution_id, + ) delays = (30, 120, 300) delay = delays[min(max(0, attempts - 1), len(delays) - 1)] next_attempt = (utc_now() + timedelta(seconds=delay)).isoformat(timespec="seconds") now = utc_now_iso() with get_connection() as connection: - connection.execute( + cursor = connection.execute( """ UPDATE publish_jobs SET status = 'SCHEDULED', next_attempt_at = ?, error_code = ?, error_message = ?, last_error = ?, execution_phase = 'worker_unavailable_before_upload', updated_at = ? - WHERE id = ? AND status = 'PUBLISHING' + WHERE id = ? AND status = 'PUBLISHING' AND execution_id = ? """, - (next_attempt, exc.error_code, exc.message, exc.message, now, job["id"]), - ) - self.repository.add_event( - str(job["id"]), "safe_retry_scheduled", from_status="PUBLISHING", to_status="SCHEDULED", - worker_id=self.worker_id, error_code=exc.error_code, message=exc.message, - payload={"next_attempt_at": next_attempt, "attempt_count": attempts}, connection=connection, + (next_attempt, exc.error_code, exc.message, exc.message, now, job["id"], execution_id), ) - connection.commit() + if cursor.rowcount: + self.repository.add_event( + str(job["id"]), "safe_retry_scheduled", from_status="PUBLISHING", to_status="SCHEDULED", + worker_id=self.worker_id, error_code=exc.error_code, message=exc.message, + payload={"next_attempt_at": next_attempt, "attempt_count": attempts}, connection=connection, + ) + connection.commit() + else: + connection.rollback() + if not cursor.rowcount: + return {"status": "skipped", "job_id": job["id"], "message": "发布执行代际已变化,未覆盖最新状态"} return {"status": "rescheduled", "job_id": job["id"], "next_attempt_at": next_attempt, "message": exc.message} - def _reschedule_before_upload(self, job_id: str, message: str) -> None: + @staticmethod + def _worker_execution_matches_job( + execution: dict[str, Any], + job: dict[str, Any], + ) -> bool: + identity = execution.get("identity") + if not isinstance(identity, dict): + return False + return identity == { + "job_id": str(job.get("id") or ""), + "platform": str(job.get("platform") or ""), + "account_id": str(job.get("account_id") or ""), + } + + @staticmethod + def _parse_worker_terminal_result(phase: str, details: dict[str, Any]) -> PublishResult: + expected_outcomes = { + "confirmed_success": PublishOutcome.PUBLISHED, + "exported": PublishOutcome.EXPORTED, + "failed": PublishOutcome.FAILED, + "manual_review": PublishOutcome.NEED_REVIEW, + } + expected = expected_outcomes.get(phase) + if expected is None: + raise ValueError("Worker execution phase 不是可接受终态") + required_fields = { + "outcome", "message", "remote_video_id", "platform_url", "published_at", + "provider_response", "error_code", "needs_manual_review", + } + if ( + not required_fields.issubset(details) + or not isinstance(details.get("provider_response"), dict) + or not isinstance(details.get("needs_manual_review"), bool) + or (phase == "confirmed_success" and not str(details.get("published_at") or "")) + or (phase == "manual_review" and details.get("needs_manual_review") is not True) + or (phase != "manual_review" and details.get("needs_manual_review") is not False) + ): + raise ValueError("Worker execution 终态结果不完整") + result = PublishResult.from_dict(details) + if result.outcome != expected: + raise ValueError("Worker execution phase 与终态结果不一致") + return result + + def _reschedule_before_upload( + self, + job_id: str, + message: str, + *, + expected_execution_id: str, + expected_updated_at: str, + ) -> dict[str, Any]: now = utc_now_iso() with get_connection() as connection: - connection.execute( + cursor = connection.execute( """ UPDATE publish_jobs SET status = 'SCHEDULED', next_attempt_at = ?, execution_phase = 'recovered_before_upload', last_error = ?, error_message = ?, updated_at = ? - WHERE id = ? AND status = 'PUBLISHING' + WHERE id = ? AND status = 'PUBLISHING' AND execution_id = ? + AND updated_at = ? """, - (now, message, message, now, job_id), + (now, message, message, now, job_id, expected_execution_id, expected_updated_at), ) - self.repository.add_event( - job_id, "recovered_before_upload", from_status="PUBLISHING", to_status="SCHEDULED", - message=message, connection=connection, - ) - connection.commit() + if cursor.rowcount: + self.repository.add_event( + job_id, "recovered_before_upload", from_status="PUBLISHING", to_status="SCHEDULED", + message=message, connection=connection, + ) + connection.commit() + else: + connection.rollback() + if not cursor.rowcount: + return {"status": "skipped", "job_id": job_id, "message": "发布执行代际已变化,未重新排队"} + return {"status": "rescheduled", "job_id": job_id, "message": message} - def _mark_published(self, job_id: str, result: PublishResult, *, require_publishing: bool = True) -> dict[str, Any]: + def _mark_published( + self, + job_id: str, + result: PublishResult, + *, + require_publishing: bool = True, + expected_execution_id: str | None = None, + ) -> dict[str, Any]: + job = self.repository.get_job(job_id) + if not job: + return {"status": "skipped", "job_id": job_id, "message": "发布任务不存在"} + try: + validate_platform_url(str(job.get("platform") or ""), result.platform_url) + except ValueError as exc: + if require_publishing: + review_result = PublishResult( + outcome=PublishOutcome.NEED_REVIEW, + message="Publisher 返回的作品链接不可信,已停止生成可点击链接", + published_at=result.published_at, + provider_response={"invalid_result": result.as_dict()}, + error_code="invalid_platform_url", + needs_manual_review=True, + ) + return self._mark_need_review( + job_id, + "invalid_platform_url", + f"Publisher 返回的作品链接不可信,已停止生成可点击链接:{exc}", + review_result, + expected_execution_id=expected_execution_id, + ) + raise now = utc_now_iso() - condition = "AND status = 'PUBLISHING'" if require_publishing else "AND status = 'NEED_REVIEW'" + if require_publishing: + if expected_execution_id is None: + condition = "AND status = 'PUBLISHING' AND COALESCE(execution_id, '') = ''" + condition_params: tuple[str, ...] = () + else: + condition = "AND status = 'PUBLISHING' AND COALESCE(execution_id, '') = ?" + condition_params = (expected_execution_id,) + else: + condition = "AND status = 'NEED_REVIEW'" + condition_params = () with get_connection() as connection: - self.repository.record_provider_result( - job_id, result, connection=connection, updated_at=now + connection.execute("BEGIN IMMEDIATE") + recorded = self.repository.record_provider_result( + job_id, + result, + expected_execution_id=expected_execution_id if require_publishing and expected_execution_id else None, + connection=connection, + updated_at=now, ) + if not recorded: + connection.rollback() + return {"status": "skipped", "job_id": job_id, "message": "发布执行代际已变化,未覆盖最新状态"} cursor = connection.execute( f""" UPDATE publish_jobs SET status = 'PUBLISHED', published_at = ?, finished_at = ?, @@ -1163,8 +1547,16 @@ def _mark_published(self, job_id: str, result: PublishResult, *, require_publish execution_phase = 'confirmed_success', updated_at = ? WHERE id = ? {condition} """, - (result.published_at or now, now, result.platform_url, result.remote_video_id, - result.remote_video_id, now, job_id), + ( + result.published_at or now, + now, + result.platform_url, + result.remote_video_id, + result.remote_video_id, + now, + job_id, + *condition_params, + ), ) if cursor.rowcount: self.repository.add_event( @@ -1179,19 +1571,33 @@ def _mark_published(self, job_id: str, result: PublishResult, *, require_publish self._append_log(job_id, "平台已确认投稿成功") return {"status": "published", "job_id": job_id, "publish_result": result.as_dict()} - def _mark_exported(self, job_id: str, result: PublishResult) -> dict[str, Any]: + def _mark_exported( + self, + job_id: str, + result: PublishResult, + *, + expected_execution_id: str, + ) -> dict[str, Any]: now = utc_now_iso() with get_connection() as connection: - self.repository.record_provider_result( - job_id, result, connection=connection, updated_at=now + connection.execute("BEGIN IMMEDIATE") + recorded = self.repository.record_provider_result( + job_id, + result, + expected_execution_id=expected_execution_id, + connection=connection, + updated_at=now, ) + if not recorded: + connection.rollback() + return {"status": "skipped", "job_id": job_id, "message": "发布执行代际已变化,未覆盖最新状态"} cursor = connection.execute( """ UPDATE publish_jobs SET status = 'EXPORTED', finished_at = ?, published_at = NULL, audit_status = 'not_submitted', execution_phase = 'exported', updated_at = ? - WHERE id = ? AND status = 'PUBLISHING' + WHERE id = ? AND status = 'PUBLISHING' AND execution_id = ? """, - (now, now, job_id), + (now, now, job_id, expected_execution_id), ) if cursor.rowcount: self.repository.add_event( @@ -1205,25 +1611,74 @@ def _mark_exported(self, job_id: str, result: PublishResult) -> dict[str, Any]: return {"status": "skipped", "job_id": job_id, "message": "任务状态已变化,未覆盖最新状态"} return {"status": "exported", "job_id": job_id, "publish_result": result.as_dict()} + @staticmethod + def _expected_publish_transition( + expected_execution_id: str | None, + expected_statuses: tuple[str, ...] | None, + expected_updated_at: str | None = None, + ) -> tuple[str, tuple[str, ...]]: + if expected_execution_id is not None: + condition = "status = 'PUBLISHING' AND COALESCE(execution_id, '') = ?" + params = (expected_execution_id,) + if expected_updated_at is not None: + condition += " AND updated_at = ?" + params = (*params, expected_updated_at) + return condition, params + statuses = tuple(str(status or "").upper() for status in (expected_statuses or ()) if status) + if not statuses: + raise ValueError("发布状态迁移必须指定 execution_id 或明确的来源状态") + placeholders = ", ".join("?" for _ in statuses) + condition = f"status IN ({placeholders})" + params = statuses + if expected_updated_at is not None: + condition += " AND updated_at = ?" + params = (*params, expected_updated_at) + return condition, params + def _mark_failed( - self, job_id: str, error_code: str, message: str, result: PublishResult | None = None + self, + job_id: str, + error_code: str, + message: str, + result: PublishResult | None = None, + *, + expected_execution_id: str | None = None, + expected_statuses: tuple[str, ...] | None = None, + expected_updated_at: str | None = None, ) -> dict[str, Any]: now = utc_now_iso() + condition, condition_params = self._expected_publish_transition( + expected_execution_id, expected_statuses, expected_updated_at + ) with get_connection() as connection: - row = connection.execute("SELECT status FROM publish_jobs WHERE id = ?", (job_id,)).fetchone() - from_status = str(row["status"] if row else "") + connection.execute("BEGIN IMMEDIATE") + row = connection.execute( + f"SELECT status FROM publish_jobs WHERE id = ? AND {condition}", + (job_id, *condition_params), + ).fetchone() + if not row: + connection.rollback() + return {"status": "skipped", "job_id": job_id, "message": "发布执行代际或状态已变化,未覆盖最新状态"} + from_status = str(row["status"]) if result: - self.repository.record_provider_result( - job_id, result, connection=connection, updated_at=now + recorded = self.repository.record_provider_result( + job_id, + result, + expected_execution_id=expected_execution_id or None, + connection=connection, + updated_at=now, ) + if not recorded: + connection.rollback() + return {"status": "skipped", "job_id": job_id, "message": "发布执行代际已变化,未覆盖最新状态"} cursor = connection.execute( - """ + f""" UPDATE publish_jobs SET status = 'FAILED', finished_at = ?, error_code = ?, error_message = ?, last_error = ?, needs_manual_review = 0, execution_phase = 'failed', updated_at = ? - WHERE id = ? AND status NOT IN ('PUBLISHED', 'EXPORTED', 'CANCELLED') + WHERE id = ? AND {condition} """, - (now, error_code, message, message, now, job_id), + (now, error_code, message, message, now, job_id, *condition_params), ) if cursor.rowcount: self.repository.add_event( @@ -1239,24 +1694,49 @@ def _mark_failed( return {"status": "failed", "job_id": job_id, "error_code": error_code, "message": message} def _mark_need_review( - self, job_id: str, error_code: str, message: str, result: PublishResult | None = None + self, + job_id: str, + error_code: str, + message: str, + result: PublishResult | None = None, + *, + expected_execution_id: str | None = None, + expected_statuses: tuple[str, ...] | None = None, + expected_updated_at: str | None = None, ) -> dict[str, Any]: now = utc_now_iso() + condition, condition_params = self._expected_publish_transition( + expected_execution_id, expected_statuses, expected_updated_at + ) with get_connection() as connection: - row = connection.execute("SELECT status FROM publish_jobs WHERE id = ?", (job_id,)).fetchone() - from_status = str(row["status"] if row else "") + connection.execute("BEGIN IMMEDIATE") + row = connection.execute( + f"SELECT status FROM publish_jobs WHERE id = ? AND {condition}", + (job_id, *condition_params), + ).fetchone() + if not row: + connection.rollback() + return {"status": "skipped", "job_id": job_id, "message": "发布执行代际或状态已变化,未覆盖最新状态"} + from_status = str(row["status"]) if result: - self.repository.record_provider_result( - job_id, result, connection=connection, updated_at=now + recorded = self.repository.record_provider_result( + job_id, + result, + expected_execution_id=expected_execution_id or None, + connection=connection, + updated_at=now, ) + if not recorded: + connection.rollback() + return {"status": "skipped", "job_id": job_id, "message": "发布执行代际已变化,未覆盖最新状态"} cursor = connection.execute( - """ + f""" UPDATE publish_jobs SET status = 'NEED_REVIEW', finished_at = ?, error_code = ?, error_message = ?, last_error = ?, needs_manual_review = 1, execution_phase = 'manual_review', updated_at = ? - WHERE id = ? AND status NOT IN ('PUBLISHED', 'EXPORTED', 'CANCELLED') + WHERE id = ? AND {condition} """, - (now, error_code, message, message, now, job_id), + (now, error_code, message, message, now, job_id, *condition_params), ) if cursor.rowcount: self.repository.add_event( @@ -1295,21 +1775,29 @@ def _transition_user_status(self, job_id: str, target: str, message: str) -> dic connection.commit() if not cursor.rowcount: raise ValueError("任务状态已变化,请刷新后重试") - return {"status": "ok", "job": self.repository.get_job(job_id)} + return {"status": "ok", "job": self._public_job(job_id)} @staticmethod def _risk_flags(job: dict[str, Any]) -> list[Any]: - for value in (job.get("risk_flags"), job.get("provider_response")): + for field_name, value in ( + ("risk_flags", job.get("risk_flags")), + ("provider_response", job.get("provider_response")), + ): if not value: continue try: parsed = json.loads(value) if isinstance(value, str) else value - except json.JSONDecodeError: - continue + except (json.JSONDecodeError, TypeError): + return ["risk_flags_invalid"] if isinstance(parsed, list): return [item for item in parsed if item] - if isinstance(parsed, dict) and isinstance(parsed.get("risk_flags"), list): - return [item for item in parsed["risk_flags"] if item] + if isinstance(parsed, dict): + if "risk_flags" not in parsed and field_name == "provider_response": + continue + nested = parsed.get("risk_flags") + if isinstance(nested, list): + return [item for item in nested if item] + return ["risk_flags_invalid"] return [] def _append_log(self, job_id: str, message: str) -> None: @@ -1320,7 +1808,7 @@ def _append_log(self, job_id: str, message: str) -> None: try: append_task_log(task_id, f"Publish job {job_id}: {message}") except Exception: - pass + logger.exception("写入发布任务日志失败:job_id=%s", job_id) def queue_snapshot(task_id: str | None = None) -> dict[str, Any]: @@ -1334,7 +1822,17 @@ def queue_snapshot(task_id: str | None = None) -> dict[str, Any]: f"SELECT * FROM publish_jobs {where} ORDER BY COALESCE(scheduled_at, created_at), created_at DESC", params, ).fetchall() - jobs = [dict(row) for row in rows] + jobs = [] + for row in rows: + job = dict(row) + job["provider_payload"] = parse_public_json_dict(job.pop("provider_response", None)) + job["publish_result_payload"] = parse_public_json_dict(job.pop("publish_result", None)) + jobs.append(job) + for job in jobs: + job["platform_url"] = safe_platform_url( + str(job.get("platform") or ""), + str(job.get("platform_url") or ""), + ) statuses = ("DRAFT", "WAITING", "SCHEDULED", "PUBLISHING", "PUBLISHED", "EXPORTED", "FAILED", "NEED_REVIEW", "CANCELLED") by_status = {status: [job for job in jobs if str(job.get("status") or "").upper() == status] for status in statuses} today = utc_now().astimezone(app_zone()).date() @@ -1400,7 +1898,9 @@ async def start_scheduler_background() -> PublishScheduler | None: if not settings.publish_scheduler_enabled: return None scheduler = PublishScheduler() - asyncio.create_task(scheduler.run_forever()) + scheduler._background_task = asyncio.create_task( + scheduler.run_forever(), name="niuma-publish-scheduler" + ) return scheduler diff --git a/app/services/publish_service.py b/app/services/publish_service.py index 7660f7c..f20a3f9 100644 --- a/app/services/publish_service.py +++ b/app/services/publish_service.py @@ -28,9 +28,11 @@ PublishJobTargetUpdate, PublishPlatformConfigUpdate, PublishSendJobUpdate, + TaskStatus, ) -from app.services.ai.ai_clip_analyzer import build_provider -from app.services.ai.base import AIProviderError +from app.services.ai.ai_clip_analyzer import build_provider, loads_ai_json +from app.services.ai.base import AIProviderError, generate_json_with_safe_retry +from app.services.publishers.base import parse_public_json_dict from app.services.database_backup_service import create_publish_migration_backup from app.services.publish_copy_rules import ( BILIBILI_TITLE_MAX, @@ -48,9 +50,14 @@ DouyinPublishProvider, PublishProviderError, ) -from app.services.publish_domain import AUTO_PUBLISH_PLATFORMS, PUBLISH_MODES, TARGET_PLATFORMS +from app.services.publish_domain import ( + AUTO_PUBLISH_PLATFORMS, + PUBLISH_MODES, + TARGET_PLATFORMS, + safe_platform_url, +) from app.services.publish_readiness import PublishPlatformIsolationBlocked -from app.services.publish_time import app_zone, local_display, parse_datetime +from app.services.publish_time import app_zone, local_display, parse_datetime, utc_now_iso from app.services.storage_service import get_artifact_paths, resolve_video_file_path from app.services.video_cut_service import ensure_ffmpeg_available, sanitize_filename_part, summarize_stderr @@ -266,6 +273,12 @@ def _now_iso() -> str: return datetime.now().isoformat(timespec="seconds") +def _version_iso() -> str: + """publish_jobs.updated_at 的乐观并发版本;创建时间继续保持旧显示顺序。""" + + return utc_now_iso() + + def _unique_paths(paths: list[Path]) -> list[Path]: seen: set[str] = set() unique: list[Path] = [] @@ -564,6 +577,7 @@ def _normalize_config(row) -> dict: config = dict(row) client_key = config.get("client_key") or "" client_secret = config.get("client_secret") or "" + config.pop("client_secret", None) config.update( { "platform_label": PLATFORM_LABELS.get(config.get("platform"), config.get("platform")), @@ -578,11 +592,15 @@ def _normalize_config(row) -> dict: def _normalize_account(row) -> dict: account = dict(row) login_status = account.get("login_status") or "login_required" + access_token = account.get("access_token") + refresh_token = account.get("refresh_token") + account.pop("access_token", None) + account.pop("refresh_token", None) account.update( { "platform_label": PLATFORM_LABELS.get(account.get("platform"), account.get("platform")), - "access_token_masked": _mask_secret(account.get("access_token")), - "refresh_token_masked": _mask_secret(account.get("refresh_token")), + "access_token_masked": _mask_secret(access_token), + "refresh_token_masked": _mask_secret(refresh_token), "is_authorized": account.get("authorization_status") == "authorized", "auth_type": account.get("auth_type") or "browser_profile", "login_status": login_status, @@ -638,11 +656,7 @@ def _format_task_created_at(value: str | None) -> str: def _task_source_file_name(job: dict) -> str: - source_path = ( - job.get("task_nas_file_path") - if job.get("task_source_type") == "nas" - else job.get("task_original_video_path") - ) + source_path = job.get("task_original_video_path") source_text = str(source_path or "").strip() if not source_text: return "未记录原视频文件名" @@ -657,8 +671,11 @@ def _normalize_job( ) -> dict: job = dict(row) status = _normalize_publish_status(job.get("status")) - provider_payload = _parse_json_text(job.get("provider_response")) - publish_result_payload = _parse_json_text(job.get("publish_result")) if "publish_result" in job else {} + provider_payload = parse_public_json_dict(job.get("provider_response")) + publish_result_payload = parse_public_json_dict(job.get("publish_result")) if "publish_result" in job else {} + # 旧记录和兼容写入可能包含未脱敏 JSON;公共 DTO 只返回清洗后的结构。 + job.pop("provider_response", None) + job.pop("publish_result", None) caption = job.get("caption") or job.get("description") or "" hashtags = job.get("hashtags") or job.get("tags") or "" clip_id = job.get("clip_id") or job.get("output_clip_id") or "" @@ -728,7 +745,10 @@ def _normalize_job( ), "provider_payload": provider_payload, "publish_result_payload": publish_result_payload, - "platform_url": job.get("platform_url") or provider_payload.get("url") or provider_payload.get("platform_url") or "", + "platform_url": safe_platform_url( + str(job.get("platform") or ""), + job.get("platform_url") or provider_payload.get("url") or provider_payload.get("platform_url") or "", + ), "trace_path": provider_payload.get("trace_path") or "", "content_complete": not missing_fields, "missing_fields": missing_fields, @@ -850,17 +870,22 @@ def list_platform_configs() -> list[dict]: return [_normalize_config(row) for row in rows] -def get_platform_config(platform: str) -> dict | None: +def _get_platform_config_record(platform: str) -> dict | None: with get_connection() as connection: row = connection.execute( "SELECT * FROM publish_platform_configs WHERE platform = ?", (platform,), ).fetchone() + return dict(row) if row else None + + +def get_platform_config(platform: str) -> dict | None: + row = _get_platform_config_record(platform) return _normalize_config(row) if row else None def update_platform_config(platform: str, payload: PublishPlatformConfigUpdate) -> dict: - existing = get_platform_config(platform) + existing = _get_platform_config_record(platform) if not existing: raise ValueError("发布平台不存在。") @@ -900,7 +925,7 @@ def update_platform_config(platform: str, payload: PublishPlatformConfigUpdate) def test_platform_config(platform: str) -> dict: - config = get_platform_config(platform) + config = _get_platform_config_record(platform) if not config: raise ValueError("发布平台不存在。") provider = _get_provider(platform, config) @@ -947,11 +972,16 @@ def _unique_normal_account_id(connection, platform: str) -> str: return str(rows[0]["id"] or "") -def get_account(account_id: str) -> dict | None: +def _get_account_record(account_id: str) -> dict | None: if not account_id: return None with get_connection() as connection: row = connection.execute("SELECT * FROM publish_accounts WHERE id = ?", (account_id,)).fetchone() + return dict(row) if row else None + + +def get_account(account_id: str) -> dict | None: + row = _get_account_record(account_id) return _normalize_account(row) if row else None @@ -1043,7 +1073,7 @@ def open_browser_creator_center(account_id: str) -> dict: def build_douyin_oauth_url() -> dict: - config = get_platform_config("douyin") + config = _get_platform_config_record("douyin") if not config: raise ValueError("抖音配置不存在。") state = uuid4().hex @@ -1088,7 +1118,7 @@ def save_douyin_oauth_account(code: str, state: str = "") -> dict: if not _validate_and_consume_oauth_state(state, "douyin"): raise ValueError("OAuth state 无效或已过期,请重新发起授权") - config = get_platform_config("douyin") + config = _get_platform_config_record("douyin") if not config: raise ValueError("抖音配置不存在。") response = DouyinPublishProvider(config).exchange_code(code) @@ -1110,9 +1140,7 @@ def save_douyin_oauth_account(code: str, state: str = "") -> dict: scopes=data.get("scope") or config.get("scope") or "", remark="OAuth 授权创建", ) - result = create_account(payload) - result["provider_response"] = response - return result + return create_account(payload) def _get_output_clip_for_publish(task_id: str, output_clip_id: str) -> dict | None: @@ -1129,11 +1157,16 @@ def _get_output_clip_for_publish(task_id: str, output_clip_id: str) -> dict | No clip_candidates.title AS clip_title, clip_candidates.cover_time_seconds AS ai_cover_time_seconds, subtitle_jobs.status AS subtitle_status, - subtitle_jobs.output_file_path AS subtitled_output_file_path + subtitle_jobs.output_file_path AS subtitled_output_file_path, + subtitle_jobs.revision_id AS subtitle_revision_id, + subtitle_jobs.validation_status AS subtitle_validation_status, + subtitle_jobs.verified_at AS subtitle_verified_at, + subtitle_revisions.status AS subtitle_revision_status FROM output_clip JOIN tasks ON tasks.id = output_clip.task_id LEFT JOIN clip_candidates ON clip_candidates.id = output_clip.clip_candidate_id LEFT JOIN subtitle_jobs ON subtitle_jobs.output_clip_id = output_clip.id AND subtitle_jobs.is_active = 1 + LEFT JOIN subtitle_revisions ON subtitle_revisions.id = subtitle_jobs.revision_id WHERE output_clip.task_id = ? AND output_clip.id = ? AND output_clip.is_active = 1 """, (task_id, output_clip_id), @@ -1155,11 +1188,16 @@ def _get_output_clip_by_id(output_clip_id: str) -> dict | None: clip_candidates.title AS clip_title, clip_candidates.cover_time_seconds AS ai_cover_time_seconds, subtitle_jobs.status AS subtitle_status, - subtitle_jobs.output_file_path AS subtitled_output_file_path + subtitle_jobs.output_file_path AS subtitled_output_file_path, + subtitle_jobs.revision_id AS subtitle_revision_id, + subtitle_jobs.validation_status AS subtitle_validation_status, + subtitle_jobs.verified_at AS subtitle_verified_at, + subtitle_revisions.status AS subtitle_revision_status FROM output_clip JOIN tasks ON tasks.id = output_clip.task_id LEFT JOIN clip_candidates ON clip_candidates.id = output_clip.clip_candidate_id LEFT JOIN subtitle_jobs ON subtitle_jobs.output_clip_id = output_clip.id AND subtitle_jobs.is_active = 1 + LEFT JOIN subtitle_revisions ON subtitle_revisions.id = subtitle_jobs.revision_id WHERE output_clip.id = ? AND output_clip.is_active = 1 """, (output_clip_id,), @@ -1193,11 +1231,16 @@ def _list_completed_publish_clips(task_id: str | None = None) -> list[dict]: clip_candidates.duration_seconds, clip_candidates.cover_time_seconds AS ai_cover_time_seconds, subtitle_jobs.status AS subtitle_status, - subtitle_jobs.output_file_path AS subtitled_output_file_path + subtitle_jobs.output_file_path AS subtitled_output_file_path, + subtitle_jobs.revision_id AS subtitle_revision_id, + subtitle_jobs.validation_status AS subtitle_validation_status, + subtitle_jobs.verified_at AS subtitle_verified_at, + subtitle_revisions.status AS subtitle_revision_status FROM output_clip JOIN tasks ON tasks.id = output_clip.task_id LEFT JOIN clip_candidates ON clip_candidates.id = output_clip.clip_candidate_id LEFT JOIN subtitle_jobs ON subtitle_jobs.output_clip_id = output_clip.id AND subtitle_jobs.is_active = 1 + LEFT JOIN subtitle_revisions ON subtitle_revisions.id = subtitle_jobs.revision_id WHERE tasks.is_deleted = 0 AND output_clip.status = 'completed' AND output_clip.is_active = 1 {where_task} ORDER BY output_clip.created_at DESC @@ -1217,8 +1260,8 @@ def _get_completed_publish_clip_by_output(output_clip_id: str) -> dict | None: def _resolve_publish_video_path(output_clip: dict, video_source: str) -> tuple[str, Path]: if video_source == "subtitled": raw_path = (output_clip.get("subtitled_output_file_path") or "").strip() - if output_clip.get("subtitle_status") != "completed" or not raw_path: - raise ValueError("这条切片还没有生成带字幕成片,不能选择“带字幕成片”。") + if not _subtitle_publish_ready(output_clip) or not raw_path: + raise ValueError("带字幕成片尚未同时通过 revision 审核和 FFprobe 验证,不能选择。") else: raw_path = (output_clip.get("output_file_path") or "").strip() if output_clip.get("output_status") != "completed" or not raw_path: @@ -1369,7 +1412,9 @@ def generate_publish_metadata(item: dict, use_ai: bool = False, *, platform: str try: provider = build_provider(settings.ai_publish_provider, purpose="publish") - parsed = json.loads(provider.generate_json(_metadata_prompt(item, platform))) + parsed = loads_ai_json(generate_json_with_safe_retry(provider, _metadata_prompt(item, platform))) + if not isinstance(parsed, dict): + raise ValueError("AI 文案响应必须是 JSON 对象") provider_name = getattr(provider, "name", settings.ai_publish_provider) publish_model = ( settings.ai_codex_model @@ -1397,7 +1442,7 @@ def generate_publish_metadata(item: dict, use_ai: bool = False, *, platform: str "policy_version": PUBLISH_COPY_RULE_VERSION if platform == "douyin" else 0, } except (AIProviderError, json.JSONDecodeError, TypeError, ValueError) as exc: - metadata["error"] = str(exc) + metadata["error"] = exc.checkpoint_message() if isinstance(exc, AIProviderError) else str(exc) return metadata @@ -1611,6 +1656,10 @@ def _insert_opencli_job( "output_status": item.get("output_status") or "completed", "subtitle_status": item.get("subtitle_status"), "subtitled_output_file_path": item.get("subtitled_output_file_path"), + "subtitle_revision_id": item.get("subtitle_revision_id"), + "subtitle_revision_status": item.get("subtitle_revision_status"), + "subtitle_validation_status": item.get("subtitle_validation_status"), + "subtitle_verified_at": item.get("subtitle_verified_at"), }, video_source, ) @@ -1684,7 +1733,11 @@ def _insert_opencli_job( cover_file_path, settings.app_timezone, settings.app_timezone, - _publish_provider_payload(metadata, cover), + _publish_provider_payload( + metadata, + cover, + existing=_subtitle_publish_evidence(item, video_source), + ), settings.publish_scheduler_max_retry_count, now, now, @@ -1901,11 +1954,34 @@ def _find_inheritable_publish_job(item: dict, platform: str) -> dict: def _preferred_video_source(item: dict, prefer_subtitled: bool) -> str: raw_path = str(item.get("subtitled_output_file_path") or "").strip() path = resolve_video_file_path(raw_path) if raw_path else None - if prefer_subtitled and item.get("subtitle_status") == "completed" and path and path.exists(): + if prefer_subtitled and _subtitle_publish_ready(item) and path and path.exists(): return "subtitled" return "original" +def _subtitle_publish_ready(item: dict) -> bool: + return bool( + item.get("subtitle_status") == "completed" + and item.get("subtitle_validation_status") == "verified" + and item.get("subtitle_revision_status") == "approved" + and item.get("subtitle_revision_id") + ) + + +def _subtitle_publish_evidence(item: dict, video_source: str) -> dict: + if video_source != "subtitled": + return {} + if not _subtitle_publish_ready(item): + raise ValueError("字幕成片缺少审核或验证证据") + return { + "subtitle_delivery_mode": "subtitled", + "subtitle_revision_id": item.get("subtitle_revision_id") or "", + "subtitle_revision_status": item.get("subtitle_revision_status") or "", + "subtitle_validation_status": item.get("subtitle_validation_status") or "", + "subtitle_verified_at": item.get("subtitle_verified_at") or "", + } + + def _update_preparation_video_source(job: dict, item: dict, video_source: str) -> dict: from app.services.publish_repository import PublishRepository @@ -1938,7 +2014,14 @@ def _update_preparation_video_source(job: dict, item: dict, video_source: str) - cover_mode, cover_time_seconds, cover_file_path, - _publish_provider_payload({}, cover, existing=job.get("provider_payload") or {}), + _publish_provider_payload( + {}, + cover, + existing={ + **(job.get("provider_payload") or {}), + **_subtitle_publish_evidence(item, video_source), + }, + ), now, job["id"], ), @@ -2018,11 +2101,13 @@ def sync_task_publish_jobs( ) -> dict: with get_connection() as connection: task = connection.execute( - "SELECT id, platform FROM tasks WHERE id = ? AND COALESCE(is_deleted, 0) = 0", + "SELECT id, platform, auto_mode, status FROM tasks WHERE id = ? AND COALESCE(is_deleted, 0) = 0", (task_id,), ).fetchone() if not task: raise ValueError("任务不存在") + if bool(task["auto_mode"]) and task["status"] == TaskStatus.PENDING_SUBTITLE_REVIEW.value: + raise ValueError("自动流水线正在等待字幕审核,请先批量烧录,或明确跳过字幕并完成片段审核") items = _list_completed_publish_clips(task_id) if not items: raise ValueError("当前任务还没有可同步的激活切片") @@ -2370,7 +2455,11 @@ def _list_missing_publish_cover_jobs(platform: str | None = None) -> list[dict]: output_clip.status AS output_status, clip_candidates.cover_time_seconds AS ai_cover_time_seconds, subtitle_jobs.status AS subtitle_status, - subtitle_jobs.output_file_path AS subtitled_output_file_path + subtitle_jobs.output_file_path AS subtitled_output_file_path, + subtitle_jobs.revision_id AS subtitle_revision_id, + subtitle_jobs.validation_status AS subtitle_validation_status, + subtitle_jobs.verified_at AS subtitle_verified_at, + subtitle_revisions.status AS subtitle_revision_status FROM publish_jobs JOIN output_clip ON output_clip.id = publish_jobs.output_clip_id JOIN tasks ON tasks.id = publish_jobs.task_id @@ -2378,6 +2467,7 @@ def _list_missing_publish_cover_jobs(platform: str | None = None) -> list[dict]: LEFT JOIN subtitle_jobs ON subtitle_jobs.output_clip_id = output_clip.id AND subtitle_jobs.is_active = 1 + LEFT JOIN subtitle_revisions ON subtitle_revisions.id = subtitle_jobs.revision_id WHERE publish_jobs.status IN ('DRAFT', 'WAITING', 'SCHEDULED') AND TRIM(COALESCE(publish_jobs.cover_file_path, '')) = '' AND output_clip.is_active = 1 @@ -2589,12 +2679,12 @@ def generate_publish_job_cover(job_id: str, payload: PublishCoverCreate) -> dict def _validate_api_publish_ready(payload: PublishJobCreate) -> tuple[dict, dict]: - account = get_account(payload.account_id or "") + account = _get_account_record(payload.account_id or "") if not account: raise ValueError("真实发布必须先选择一个已配置的发布账号。") if account.get("platform") != payload.platform: raise ValueError("账号平台和发布平台不一致。") - config = get_platform_config(payload.platform) + config = _get_platform_config_record(payload.platform) if not config: raise ValueError("发布平台配置不存在。") _get_provider(payload.platform, config).validate_config() @@ -3217,14 +3307,14 @@ def update_send_job(job_id: str, payload: PublishSendJobUpdate) -> dict: existing=job.get("provider_payload") or {}, upgrade_status="manual_saved", ) - connection.execute( + cursor = connection.execute( """ UPDATE publish_jobs SET title = ?, description = ?, caption = ?, tags = ?, hashtags = ?, visibility = ?, cover_file_path = ?, cover_time_seconds = ?, allow_download = ?, bilibili_tid = ?, bilibili_copyright = ?, bilibili_source = ?, account_id = ?, provider_response = ?, updated_at = ? - WHERE id = ? + WHERE id = ? AND status = ? AND updated_at = ? """, ( safe_content["title"], @@ -3241,10 +3331,15 @@ def update_send_job(job_id: str, payload: PublishSendJobUpdate) -> dict: (payload.bilibili_source or "").strip(), account_id or None, provider_response, - _now_iso(), + _version_iso(), job_id, + job.get("status"), + job.get("updated_at"), ), ) + if not cursor.rowcount: + connection.rollback() + raise ValueError("发送任务内容或状态已经变化,请刷新后重试。") connection.commit() return {"status": "ok", "message": "发送内容已保存。", "job": get_publish_job(job_id)} @@ -3274,13 +3369,24 @@ def update_publish_job_target(job_id: str, payload: PublishJobTargetUpdate) -> d raise ValueError("账号与目标平台不一致") try: with get_connection() as connection: - connection.execute( + cursor = connection.execute( """ UPDATE publish_jobs SET platform = ?, account_id = ?, publish_mode = ?, updated_at = ? - WHERE id = ? + WHERE id = ? AND status = ? AND updated_at = ? """, - (payload.platform, account_id or None, payload.publish_mode, _now_iso(), job_id), + ( + payload.platform, + account_id or None, + payload.publish_mode, + _version_iso(), + job_id, + job.get("status"), + job.get("updated_at"), + ), ) + if not cursor.rowcount: + connection.rollback() + raise ValueError("发布任务内容或状态已经变化,请刷新后重试") connection.commit() except Exception as exc: if "UNIQUE constraint" in str(exc): @@ -3330,7 +3436,7 @@ def update_publish_job_content(job_id: str, payload: PublishJobContentUpdate) -> ensure_future(payload.scheduled_at, settings.app_timezone) scheduled_at = to_utc_iso(payload.scheduled_at, settings.app_timezone) status = PUBLISH_STATUS_SCHEDULED if scheduled_at else PUBLISH_STATUS_WAITING - now = _now_iso() + now = _version_iso() with get_connection() as connection: account_id = str(job.get("account_id") or "") if not account_id and str(job.get("publish_mode") or "") == "local_browser": @@ -3340,14 +3446,14 @@ def update_publish_job_content(job_id: str, payload: PublishJobContentUpdate) -> existing=job.get("provider_payload") or {}, upgrade_status="manual_saved", ) - connection.execute( + cursor = connection.execute( """ UPDATE publish_jobs SET title = ?, description = ?, caption = ?, tags = ?, hashtags = ?, cover_text = ?, scheduled_at = ?, status = ?, risk_flags = '', account_id = ?, provider_response = ?, error_code = '', error_message = '', last_error = '', updated_at = ? - WHERE id = ? + WHERE id = ? AND status = ? AND updated_at = ? """, ( safe_content["title"], @@ -3362,8 +3468,13 @@ def update_publish_job_content(job_id: str, payload: PublishJobContentUpdate) -> provider_response, now, job_id, + job.get("status"), + job.get("updated_at"), ), ) + if not cursor.rowcount: + connection.rollback() + raise ValueError("发布任务内容或状态已经变化,请刷新后重试") connection.commit() return {"status": "ok", "message": "publish content saved", "job": get_publish_job(job_id)} @@ -3386,12 +3497,12 @@ def regenerate_send_job_metadata(job_id: str, use_ai: bool = True) -> dict: account_id = str(job.get("account_id") or "") if not account_id and str(job.get("publish_mode") or "") == "local_browser": account_id = _unique_normal_account_id(connection, str(job.get("platform") or "")) - connection.execute( + cursor = connection.execute( """ UPDATE publish_jobs SET title = ?, description = ?, caption = ?, tags = ?, hashtags = ?, account_id = ?, provider_response = ?, updated_at = ? - WHERE id = ? + WHERE id = ? AND status = ? AND updated_at = ? """, ( metadata["title"], @@ -3409,10 +3520,15 @@ def regenerate_send_job_metadata(job_id: str, use_ai: bool = True) -> dict: existing=job.get("provider_payload") or {}, upgrade_status="manual_retry" if use_ai else "rule_regenerated", ), - _now_iso(), + _version_iso(), job_id, + job.get("status"), + job.get("updated_at"), ), ) + if not cursor.rowcount: + connection.rollback() + raise ValueError("发送任务内容或状态已经变化,请刷新后重新生成文案。") connection.commit() return { "status": "ok", @@ -4455,8 +4571,8 @@ def execute_api_publish_job(job_id: str) -> dict: if not output_clip: raise ValueError("切片记录不存在。") _, video_path = _resolve_publish_video_path(output_clip, job["video_source"]) - config = get_platform_config(job["platform"]) - account = get_account(job.get("account_id") or "") + config = _get_platform_config_record(job["platform"]) + account = _get_account_record(job.get("account_id") or "") if not config or not account: raise ValueError("平台配置或账号不存在。") return _execute_publish_job(job_id, config=config, account=account, video_path=video_path) @@ -4513,14 +4629,19 @@ def get_publish_center_context(*, focus_task_id: str = "") -> dict: generated=True, ) original_available = bool(original_path and original_path.exists() and original_path.is_file()) - subtitled_available = bool(subtitled_path and subtitled_path.exists() and subtitled_path.is_file()) + subtitled_available = bool( + _subtitle_publish_ready(item) + and subtitled_path + and subtitled_path.exists() + and subtitled_path.is_file() + ) normalized_item = { **item, "default_title": default_title, "default_tags": format_douyin_tags(_fallback_tags(item), generated=True), "original_available": original_available, "subtitled_available": subtitled_available, - "subtitle_status_label": "已加字幕" if item.get("subtitle_status") == "completed" else "未加字幕", + "subtitle_status_label": "已审核并验证" if subtitled_available else "字幕未就绪", "video_media_url": _video_media_url(item["task_id"], item["output_clip_id"], "original"), } publish_items.append(normalized_item) diff --git a/app/services/publish_time.py b/app/services/publish_time.py index 3da9274..5e47f1b 100644 --- a/app/services/publish_time.py +++ b/app/services/publish_time.py @@ -25,7 +25,8 @@ def utc_now() -> datetime: def utc_now_iso() -> str: - return utc_now().isoformat(timespec="seconds") + # updated_at 同时承担轻量乐观并发版本;秒级精度无法区分同一秒内的两次写入。 + return utc_now().isoformat(timespec="microseconds") def parse_datetime(value: str | None, timezone_name: str | None = None) -> datetime: diff --git a/app/services/publishers/base.py b/app/services/publishers/base.py index e9dd511..3b5a595 100644 --- a/app/services/publishers/base.py +++ b/app/services/publishers/base.py @@ -78,7 +78,7 @@ def payload(self) -> dict[str, Any]: return self.provider_response def as_dict(self) -> dict[str, Any]: - payload = asdict(self) + payload = sanitize_provider_response(asdict(self)) payload["outcome"] = self.outcome.value return payload @@ -99,22 +99,51 @@ def from_dict(cls, payload: dict[str, Any]) -> "PublishResult": ) -_SENSITIVE_KEYS = { - "access_token", "authorization", "cookie", "cookies", "password", "refresh_token", - "secret", "storage_state", "token", "client_secret", +_SENSITIVE_KEY_PARTS = { + "accesskey", "accesstoken", "apikey", "apisecret", "authorization", "authtoken", + "bearer", "clientsecret", + "cookie", "cookies", "credential", "credentials", "csrftoken", "password", "privatekey", + "idtoken", "jwt", "refreshtoken", "secret", "secretkey", "sessiontoken", "storagestate", + "token", "tokenvalue", } +def _is_sensitive_key(key: Any) -> bool: + normalized = re.sub(r"[^a-z0-9]", "", str(key).lower()) + return normalized in _SENSITIVE_KEY_PARTS or any( + normalized.endswith(part) for part in _SENSITIVE_KEY_PARTS + ) + + +def _sanitize_sensitive_text(value: str) -> str: + cleaned = re.sub( + r"(?i)\bbearer\s+[A-Za-z0-9._~+/=-]{8,}", + "Bearer [REDACTED]", + value, + ) + return re.sub( + r"(?i)\b(authorization|access[_-]?token|refresh[_-]?token|api[_-]?key|client[_-]?secret|cookie)" + r"\s*[:=]\s*['\"]?[^'\"\s,;}]+'?", + r"\1=[REDACTED]", + cleaned, + ) + + def sanitize_provider_response(value: Any) -> Any: if isinstance(value, dict): return { - str(key): "[REDACTED]" if str(key).lower() in _SENSITIVE_KEYS else sanitize_provider_response(item) + str(key): "[REDACTED]" + if _is_sensitive_key(key) + else sanitize_provider_response(item) for key, item in value.items() } if isinstance(value, list): return [sanitize_provider_response(item) for item in value] - if isinstance(value, str) and len(value) > 20000: - return value[:20000] + "…" + if isinstance(value, str): + cleaned = _sanitize_sensitive_text(value) + if len(cleaned) > 20000: + return cleaned[:20000] + "…" + return cleaned return value @@ -130,6 +159,21 @@ def parse_json_dict(value: Any) -> dict[str, Any]: return parsed if isinstance(parsed, dict) else {"data": parsed} +def parse_public_json_dict(value: Any) -> dict[str, Any]: + """解析供 API/UI 返回的 Provider JSON;损坏正文不回显。""" + if isinstance(value, dict): + return sanitize_provider_response(value) + if not value: + return {} + try: + parsed = json.loads(str(value)) + except (json.JSONDecodeError, TypeError): + return {"invalid_payload": True} + if not isinstance(parsed, dict): + return {"invalid_payload": True} + return sanitize_provider_response(parsed) + + def job_video_path(job: dict[str, Any]) -> Path: raw_path = str(job.get("video_path") or job.get("video_file_path") or "").strip() if not raw_path: diff --git a/app/services/publishers/browser_runtime.py b/app/services/publishers/browser_runtime.py index 2fcca52..930d626 100644 --- a/app/services/publishers/browser_runtime.py +++ b/app/services/publishers/browser_runtime.py @@ -13,6 +13,7 @@ from app.core.config import settings from app.services.publishers.base import PublishError, PublishNeedsReview +from app.services.publishers.worker_client import validate_worker_identifier PhaseCallback = Callable[[str, dict[str, Any] | None], None] @@ -26,11 +27,11 @@ def __init__( *, phase_callback: PhaseCallback | None = None, ) -> None: - self.platform = platform - self.account_id = account_id + self.platform = validate_worker_identifier(platform, "platform", max_length=20) + self.account_id = validate_worker_identifier(account_id, "account_id", max_length=120) self.phase_callback = phase_callback or (lambda _phase, _details=None: None) - self.profile_dir = Path(settings.publish_browser_profile_dir) / platform / account_id - self.artifact_dir = Path(settings.publish_browser_artifact_dir) / platform / account_id + self.profile_dir = Path(settings.publish_browser_profile_dir) / self.platform / self.account_id + self.artifact_dir = Path(settings.publish_browser_artifact_dir) / self.platform / self.account_id def phase(self, phase: str, details: dict[str, Any] | None = None) -> None: self.phase_callback(phase, details) diff --git a/app/services/publishers/local_browser.py b/app/services/publishers/local_browser.py index 549507a..f6658d8 100644 --- a/app/services/publishers/local_browser.py +++ b/app/services/publishers/local_browser.py @@ -2,6 +2,7 @@ from __future__ import annotations +from collections.abc import Callable from typing import Any from app.services.publishers.base import ( @@ -24,11 +25,13 @@ def __init__( platform: str, worker_client: PublishWorkerClient | None = None, repository: Any | None = None, + before_dispatch: Callable[[], None] | None = None, **_: Any, ) -> None: self.platform = str(platform or "").lower() self.worker_client = worker_client or PublishWorkerClient() self.repository = repository + self.before_dispatch = before_dispatch or (lambda: None) def validate(self, job: dict[str, Any]) -> None: super().validate(job) @@ -65,4 +68,5 @@ def publish(self, job: dict[str, Any]) -> PublishResult: needs_manual_review=True, provider_response={"login_status": login_status}, ) + self.before_dispatch() return self.worker_client.publish(self.build_payload(job)) diff --git a/app/services/publishers/manual_export.py b/app/services/publishers/manual_export.py index d1ed2ce..74ea008 100644 --- a/app/services/publishers/manual_export.py +++ b/app/services/publishers/manual_export.py @@ -3,13 +3,24 @@ from __future__ import annotations import json +import logging import shutil from pathlib import Path from typing import Any +from uuid import uuid4 from app.core.config import settings from app.services.publish_time import utc_now_iso -from app.services.publishers.base import BasePublisher, PublishOutcome, PublishResult, job_video_path +from app.services.publishers.base import ( + BasePublisher, + PublishOutcome, + PublishResult, + PublishValidationError, + job_video_path, +) + + +logger = logging.getLogger(__name__) def _write_text(path: Path, value: str) -> None: @@ -20,6 +31,29 @@ def _write_json(path: Path, value: dict[str, Any]) -> None: path.write_text(json.dumps(value, ensure_ascii=False, indent=2), encoding="utf-8") +def _safe_package_component(value: Any, fallback: str) -> str: + component = str(value or fallback) + windows_reserved = { + "CON", "PRN", "AUX", "NUL", + *(f"COM{index}" for index in range(1, 10)), + *(f"LPT{index}" for index in range(1, 10)), + } + if ( + not component + or component != component.strip() + or component in {".", ".."} + or Path(component).is_absolute() + or "/" in component + or "\\" in component + or any(character in component for character in '<>:"|?*') + or any(ord(character) < 32 for character in component) + or component.endswith((".", " ")) + or component.split(".", 1)[0].upper() in windows_reserved + ): + raise PublishValidationError("发布包标识包含不安全路径", "unsafe_export_identifier") + return component + + class ManualExportPublisher(BasePublisher): name = "manual_export" @@ -27,33 +61,60 @@ def __init__(self, export_dir: Path | None = None, **_: Any) -> None: self.export_dir = Path(export_dir or settings.publish_scheduler_export_dir) def build_package_dir(self, job: dict[str, Any]) -> Path: - clip_id = str(job.get("clip_id") or job.get("output_clip_id") or "unknown_clip") - return self.export_dir / str(job.get("task_id") or "unknown_task") / clip_id + task_id = _safe_package_component(job.get("task_id"), "unknown_task") + clip_id = _safe_package_component( + job.get("clip_id") or job.get("output_clip_id"), "unknown_clip" + ) + root = self.export_dir.resolve() + package_dir = (root / task_id / clip_id).resolve() + if root not in package_dir.parents: + raise PublishValidationError("发布包路径超出导出目录", "unsafe_export_path") + return package_dir def publish(self, job: dict[str, Any]) -> PublishResult: self.validate(job) video_path = job_video_path(job) package_dir = self.build_package_dir(job) - package_dir.mkdir(parents=True, exist_ok=True) - clip_path = package_dir / f"clip{video_path.suffix.lower()}" - shutil.copy2(video_path, clip_path) - + package_dir.parent.mkdir(parents=True, exist_ok=True) + nonce = uuid4().hex + staging_dir = package_dir.with_name(f".{package_dir.name}.staging-{nonce}") + backup_dir = package_dir.with_name(f".{package_dir.name}.backup-{nonce}") + final_clip_path = package_dir / f"clip{video_path.suffix.lower()}" payload = self.build_payload(job) payload.update({ "package_dir": str(package_dir), - "clip_file": str(clip_path), + "clip_file": str(final_clip_path), "exported_at": utc_now_iso(), "notice": "本地发布包已生成,尚未向平台投稿。", }) - _write_text(package_dir / "title.txt", payload["title"]) - _write_text(package_dir / "caption.txt", payload["caption"]) - _write_text(package_dir / "hashtags.txt", payload["hashtags"]) - _write_json(package_dir / "publish_plan.json", payload) - _write_json(package_dir / "metadata.json", { - **payload, - "source_video_name": video_path.name, - "source_video_size_bytes": video_path.stat().st_size, - }) + try: + staging_dir.mkdir() + shutil.copy2(video_path, staging_dir / final_clip_path.name) + _write_text(staging_dir / "title.txt", payload["title"]) + _write_text(staging_dir / "caption.txt", payload["caption"]) + _write_text(staging_dir / "hashtags.txt", payload["hashtags"]) + _write_json(staging_dir / "publish_plan.json", payload) + _write_json(staging_dir / "metadata.json", { + **payload, + "source_video_name": video_path.name, + "source_video_size_bytes": video_path.stat().st_size, + }) + if package_dir.exists(): + package_dir.replace(backup_dir) + try: + staging_dir.replace(package_dir) + except Exception: + if backup_dir.exists() and not package_dir.exists(): + backup_dir.replace(package_dir) + raise + finally: + if staging_dir.exists(): + shutil.rmtree(staging_dir, ignore_errors=True) + if backup_dir.exists(): + try: + shutil.rmtree(backup_dir) + except OSError: + logger.warning("旧发布包备份清理失败,保留供人工检查:%s", backup_dir) return PublishResult( outcome=PublishOutcome.EXPORTED, message="本地发布包已生成,未向平台投稿", diff --git a/app/services/publishers/worker_client.py b/app/services/publishers/worker_client.py index 74c1ab0..56db7d2 100644 --- a/app/services/publishers/worker_client.py +++ b/app/services/publishers/worker_client.py @@ -3,13 +3,45 @@ from __future__ import annotations import json +import re import socket from typing import Any from urllib.error import HTTPError, URLError from urllib.request import Request, urlopen from app.core.config import settings -from app.services.publishers.base import PublishError, PublishResult, PublishWorkerUnavailable +from app.services.publishers.base import ( + PublishError, + PublishResult, + PublishValidationError, + PublishWorkerUnavailable, +) + + +_WORKER_IDENTIFIER = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._-]*$") +_WINDOWS_RESERVED_NAMES = { + "CON", "PRN", "AUX", "NUL", + *(f"COM{index}" for index in range(1, 10)), + *(f"LPT{index}" for index in range(1, 10)), +} + + +def validate_worker_identifier(value: str, field_name: str, *, max_length: int) -> str: + """验证会进入 URL、journal 或 Windows 目录名的稳定标识。""" + text = str(value or "") + if not text or len(text) > max_length: + raise PublishValidationError(f"{field_name} 长度不合法", "invalid_worker_identifier") + if ( + not _WORKER_IDENTIFIER.fullmatch(text) + or ".." in text + or text.endswith((".", " ")) + or text.split(".", 1)[0].upper() in _WINDOWS_RESERVED_NAMES + ): + raise PublishValidationError( + f"{field_name} 含有不安全字符或 Windows 保留名称", + "invalid_worker_identifier", + ) + return text class PublishWorkerClient: @@ -53,10 +85,16 @@ def _request(self, method: str, path: str, payload: dict[str, Any] | None = None except (URLError, OSError) as exc: reason = getattr(exc, "reason", exc) timed_out = isinstance(reason, (TimeoutError, socket.timeout)) or "timed out" in str(reason).lower() + definitely_not_connected = isinstance(reason, (ConnectionRefusedError, socket.gaierror)) + publish_result_uncertain = ( + method.upper() == "POST" + and path == "/v1/publish" + and not definitely_not_connected + ) raise PublishWorkerUnavailable( "发送服务正在随 Docker 中的牛马片场项目自动启动。" "如果刚刚运行项目,请稍候并重新检测;持续未连接时,请在 Docker Desktop 中停止后重新运行本项目。", - request_may_have_been_received=timed_out, + request_may_have_been_received=timed_out or publish_result_uncertain, ) from exc if not raw: return {} @@ -72,17 +110,31 @@ def health(self) -> dict[str, Any]: return self._request("GET", "/v1/health") def check_account(self, platform: str, account_id: str) -> dict[str, Any]: + account_id = validate_worker_identifier(account_id, "account_id", max_length=120) return self._request("POST", "/v1/accounts/check", {"platform": platform, "account_id": account_id}) def start_login(self, platform: str, account_id: str) -> dict[str, Any]: + account_id = validate_worker_identifier(account_id, "account_id", max_length=120) return self._request("POST", "/v1/accounts/login", {"platform": platform, "account_id": account_id}) def open_creator_center(self, platform: str, account_id: str) -> dict[str, Any]: + account_id = validate_worker_identifier(account_id, "account_id", max_length=120) return self._request("POST", "/v1/accounts/open-center", {"platform": platform, "account_id": account_id}) def publish(self, payload: dict[str, Any]) -> PublishResult: - response = self._request("POST", "/v1/publish", payload) + safe_payload = dict(payload) + safe_payload["job_id"] = validate_worker_identifier( + str(safe_payload.get("job_id") or ""), "job_id", max_length=160 + ) + safe_payload["execution_id"] = validate_worker_identifier( + str(safe_payload.get("execution_id") or ""), "execution_id", max_length=160 + ) + safe_payload["account_id"] = validate_worker_identifier( + str(safe_payload.get("account_id") or ""), "account_id", max_length=120 + ) + response = self._request("POST", "/v1/publish", safe_payload) return PublishResult.from_dict(response) def execution(self, execution_id: str) -> dict[str, Any]: + execution_id = validate_worker_identifier(execution_id, "execution_id", max_length=160) return self._request("GET", f"/v1/executions/{execution_id}") diff --git a/app/services/storage_service.py b/app/services/storage_service.py index 56065aa..29a72e1 100644 --- a/app/services/storage_service.py +++ b/app/services/storage_service.py @@ -1,5 +1,7 @@ from dataclasses import dataclass +from datetime import datetime, timezone from pathlib import Path, PureWindowsPath +import json import os import re import sqlite3 @@ -14,7 +16,9 @@ TASK_SUBDIRECTORIES = ("source", "audio", "transcripts", "analysis", "clips", "05_clips", "06_subtitled", "07_covers", "logs") VIDEO_EXTENSIONS = {".mp4", ".mov", ".mkv", ".avi", ".flv", ".webm", ".m4v", ".ts"} AUDIO_EXTENSIONS = {".wav", ".mp3", ".aac", ".flac", ".ogg", ".wma", ".m4a"} +IMAGE_EXTENSIONS = {".jpg", ".jpeg", ".png", ".webp"} TRASH_DIR_NAME = "_回收站" +DELETE_STAGING_DIR_NAME = ".niuma-delete-staging" _WINDOWS_FORBIDDEN_CHARS = re.compile(r'[<>:"/\\|?*\x00-\x1f]') _WINDOWS_RESERVED_NAMES = { "CON", @@ -25,8 +29,9 @@ *(f"LPT{index}" for index in range(1, 10)), } -# 路径遍历攻击的标记 -_PATH_TRAVERSAL_MARKERS = ("..", "~") +# 路径遍历攻击的标记。普通文件名中的 ``~`` 不会被 pathlib 自动展开, +# 因此只拒绝真正会改变父目录的 ``..``。 +_PATH_TRAVERSAL_MARKERS = ("..",) class StorageSafetyError(RuntimeError): @@ -55,6 +60,29 @@ class TaskMediaCleanupResult: deleted_paths: tuple[str, ...] freed_bytes: int external_source_preserved: bool + cleanup_pending: bool = False + staged_paths: tuple[str, ...] = () + + +@dataclass(frozen=True) +class StagedMediaTarget: + label: str + original_path: Path + staged_path: Path + size_bytes: int + + +@dataclass(frozen=True) +class StagedTaskMediaCleanup: + task_id: str + stage_id: str + targets: tuple[StagedMediaTarget, ...] + manifest_roots: tuple[Path, ...] + external_source_preserved: bool + + @property + def freed_bytes(self) -> int: + return sum(target.size_bytes for target in self.targets) def _ensure_writable_directory(path: Path, label: str) -> Path: @@ -171,12 +199,22 @@ def ensure_tasks_root() -> Path: return settings.tasks_dir -def _storage_relative_parts(task_dir_name: str) -> tuple[str, ...]: - return tuple(part for part in PureWindowsPath(task_dir_name).parts if part not in {"", "."}) +def _storage_relative_parts(task_dir_name: str, label: str = "任务目录名") -> tuple[str, ...]: + windows_path = PureWindowsPath(str(task_dir_name or "").strip()) + parts = tuple(part for part in windows_path.parts if part not in {"", "."}) + if ( + not parts + or windows_path.is_absolute() + or bool(windows_path.drive) + or any(part in _PATH_TRAVERSAL_MARKERS for part in parts) + ): + raise StorageSafetyError(f"{label}包含不安全路径:{task_dir_name}") + return parts def _storage_path_from_dir_name(task_dir_name: str) -> Path: - return settings.tasks_dir.joinpath(*_storage_relative_parts(task_dir_name)) + parts = _storage_relative_parts(task_dir_name) + return _safe_managed_child(settings.tasks_dir, parts, "任务目录") def sanitize_task_dir_name(task_name: str | None, fallback: str = "untitled") -> str: @@ -210,8 +248,8 @@ def _get_existing_task_dir_names(include_deleted: bool = True, exclude_task_id: f"SELECT task_dir_name FROM tasks WHERE {' AND '.join(where_parts)}", params, ).fetchall() - except sqlite3.Error: - return set() + except sqlite3.Error as exc: + raise RuntimeError(f"读取已有任务目录失败:{exc}") from exc return {str(row["task_dir_name"]).lower() for row in rows} @@ -219,22 +257,48 @@ def allocate_task_dir_name( task_name: str | None, parent_dir_name: str | None = None, exclude_task_id: str | None = None, + *, + reserve: bool = True, ) -> str: base_name = sanitize_task_dir_name(task_name, fallback=exclude_task_id or "untitled") - parent_parts = _storage_relative_parts(parent_dir_name or "") + parent_parts = _storage_relative_parts(parent_dir_name, "父目录名") if parent_dir_name else () existing_names = _get_existing_task_dir_names(exclude_task_id=exclude_task_id) - root = ensure_tasks_root().joinpath(*parent_parts) + tasks_root = ensure_tasks_root() + root = _safe_managed_child(tasks_root, parent_parts, "父目录") if parent_parts else tasks_root root.mkdir(parents=True, exist_ok=True) for index in range(1, 1000): candidate_name = base_name if index == 1 else f"{base_name} ({index})" candidate_parts = (*parent_parts, candidate_name) relative_name = str(PureWindowsPath(*candidate_parts)) - candidate_path = ensure_tasks_root().joinpath(*candidate_parts) - if relative_name.lower() not in existing_names and not candidate_path.exists(): + if relative_name.lower() in existing_names: + continue + candidate_path = _safe_managed_child(tasks_root, candidate_parts, "任务目录") + if reserve: + try: + candidate_path.mkdir(parents=False, exist_ok=False) + except FileExistsError: + continue + return relative_name + if not candidate_path.exists(): return relative_name - return str(PureWindowsPath(*parent_parts, f"{base_name}-{uuid4().hex[:6]}")) + for _attempt in range(100): + relative_name = str(PureWindowsPath(*parent_parts, f"{base_name}-{uuid4().hex[:8]}")) + candidate_path = _safe_managed_child( + tasks_root, + _storage_relative_parts(relative_name), + "任务目录", + ) + if reserve: + try: + candidate_path.mkdir(parents=False, exist_ok=False) + except FileExistsError: + continue + elif candidate_path.exists(): + continue + return relative_name + raise StorageSafetyError("无法分配唯一任务目录,请稍后重试") def _fetch_task_dir_name(task_id: str) -> str | None: @@ -247,8 +311,8 @@ def _fetch_task_dir_name(task_id: str) -> str | None: if "task_dir_name" not in columns: return None row = connection.execute("SELECT task_dir_name FROM tasks WHERE id = ?", (task_id,)).fetchone() - except sqlite3.Error: - return None + except sqlite3.Error as exc: + raise RuntimeError(f"读取任务目录映射失败:{exc}") from exc if not row: return None return row["task_dir_name"] or None @@ -312,7 +376,11 @@ def resolve_video_file_path(path_value: str | None) -> Path | None: if task_dir_name and task_dir_name != first_part: task_dir_posix = task_dir_name.replace("\\", "/") relative_value = f"{task_dir_posix}{separator}{rest}" if rest else task_dir_name - return settings.tasks_dir.joinpath(*PureWindowsPath(relative_value).parts) + return _safe_managed_child( + settings.tasks_dir, + _storage_relative_parts(relative_value, "容器媒体路径"), + "容器媒体路径", + ) windows_storage_root = str(settings.storage_root) raw_value = path_value.replace("/", "\\") @@ -324,7 +392,11 @@ def resolve_video_file_path(path_value: str | None) -> Path | None: if task_dir_name and task_dir_name != first_part: rest_parts = PureWindowsPath(relative_value).parts[1:] relative_value = str(PureWindowsPath(task_dir_name, *rest_parts)) - return settings.storage_root.joinpath(*PureWindowsPath(relative_value).parts) + return _safe_managed_child( + settings.storage_root, + _storage_relative_parts(relative_value, "旧版媒体路径"), + "旧版媒体路径", + ) if windows_storage_root: normalized_root = windows_storage_root.replace("/", "\\") @@ -335,15 +407,60 @@ def resolve_video_file_path(path_value: str | None) -> Path | None: if task_dir_name and task_dir_name != first_part: rest_parts = PureWindowsPath(relative_value).parts[1:] relative_value = str(PureWindowsPath(task_dir_name, *rest_parts)) - return settings.storage_root.joinpath(*PureWindowsPath(relative_value).parts) + return _safe_managed_child( + settings.storage_root, + _storage_relative_parts(relative_value, "媒体路径"), + "媒体路径", + ) return path +def resolve_task_media_file_path( + path_value: str | None, + *, + task_id: str, + task_dir_name: str | None, + allowed_subdirectories: tuple[str, ...], + allowed_extensions: set[str] | frozenset[str] = VIDEO_EXTENSIONS, +) -> Path | None: + """解析任务产物,并确认它只能落在当前任务的受控子目录内。""" + if not path_value or not allowed_subdirectories: + return None + try: + resolved_path = resolve_video_file_path(path_value) + if resolved_path is None or resolved_path.suffix.lower() not in allowed_extensions: + return None + task_root = get_task_directory(task_id, task_dir_name) + roots = [task_root] + + legacy_root = settings.project_root / "tasks" + task_id_parts = _safe_relative_parts(task_id, "任务 ID") + if len(task_id_parts) == 1: + roots.append(_safe_managed_child(legacy_root, task_id_parts, "旧版任务目录")) + dir_parts = _safe_relative_parts(task_dir_name or task_id, "任务目录名") + if len(dir_parts) == 1 and (task_dir_name or task_id).lower() != task_id.lower(): + roots.append(_safe_managed_child(legacy_root, dir_parts, "旧版任务目录")) + except (OSError, ValueError, StorageSafetyError): + return None + + for root in roots: + for subdirectory in allowed_subdirectories: + allowed_root = root / subdirectory + if resolved_path.resolve(strict=False) == allowed_root.resolve(strict=False): + continue + if _path_is_within(resolved_path, allowed_root): + return resolved_path + return None + + def validate_source_video_path(path_value: str | None) -> tuple[bool, str]: if not path_value: return False, "尚未选择视频文件" - path = resolve_video_file_path(path_value) + try: + path = resolve_video_file_path(path_value) + except StorageSafetyError: + return False, "视频路径包含不安全的路径跳转字符" if path is None: return False, "尚未选择视频文件" @@ -367,8 +484,7 @@ def validate_source_video_path(path_value: str | None) -> tuple[bool, str]: def get_source_video_path(task: dict) -> Path | None: - source_path = task.get("nas_file_path") if task.get("source_type") == "nas" else task.get("original_video_path") - return resolve_video_file_path(source_path) + return resolve_video_file_path(task.get("original_video_path")) def save_uploaded_video(task_id: str, filename: str, file_object: BinaryIO, task_dir_name: str | None = None) -> Path: @@ -423,16 +539,7 @@ def remove_failed_task_directory(task_id: str, task_dir_name: str) -> None: def _safe_relative_parts(value: str, label: str) -> tuple[str, ...]: - windows_path = PureWindowsPath(str(value or "").strip()) - parts = tuple(part for part in windows_path.parts if part not in {"", "."}) - if ( - not parts - or windows_path.is_absolute() - or windows_path.drive - or any(part in _PATH_TRAVERSAL_MARKERS for part in parts) - ): - raise StorageSafetyError(f"{label}包含不安全路径:{value}") - return parts + return _storage_relative_parts(value, label) def _safe_managed_child(root: Path, parts: tuple[str, ...], label: str) -> Path: @@ -444,7 +551,7 @@ def _safe_managed_child(root: Path, parts: tuple[str, ...], label: str) -> Path: except AttributeError: # pragma: no cover - Python 3.8 兼容 within_root = str(resolved_candidate).lower().startswith(str(resolved_root).lower() + os.sep) if resolved_candidate == resolved_root or not within_root or candidate.is_symlink(): - raise StorageSafetyError(f"拒绝删除不安全的{label}:{candidate}") + raise StorageSafetyError(f"拒绝访问不安全的{label}:{candidate}") return candidate @@ -530,82 +637,217 @@ def task_media_cleanup_plan_size(plan: TaskMediaCleanupPlan) -> int: ) -def apply_task_media_cleanup_plan(plan: TaskMediaCleanupPlan) -> TaskMediaCleanupResult: - deleted_paths: list[str] = [] - freed_bytes = 0 - for target in plan.targets: +def _cleanup_manifest_payload( + staged: StagedTaskMediaCleanup, + *, + status: str, + moved_paths: tuple[str, ...] = (), +) -> dict: + return { + "version": 1, + "task_id": staged.task_id, + "stage_id": staged.stage_id, + "status": status, + "updated_at": datetime.now(timezone.utc).isoformat(timespec="seconds"), + "moved_paths": list(moved_paths), + "targets": [ + { + "label": target.label, + "original_path": str(target.original_path), + "staged_path": str(target.staged_path), + "size_bytes": target.size_bytes, + } + for target in staged.targets + ], + } + + +def _write_cleanup_manifests( + staged: StagedTaskMediaCleanup, + *, + status: str, + moved_paths: tuple[str, ...] = (), +) -> None: + payload = _cleanup_manifest_payload(staged, status=status, moved_paths=moved_paths) + serialized = json.dumps(payload, ensure_ascii=False, indent=2) + for root in staged.manifest_roots: + root.mkdir(parents=True, exist_ok=True) + manifest_path = root / "manifest.json" + temporary_path = root / f"manifest.json.tmp-{uuid4().hex}" + temporary_path.write_text(serialized, encoding="utf-8") + os.replace(temporary_path, manifest_path) + + +def _cleanup_stage_roots(staged: StagedTaskMediaCleanup) -> tuple[str, ...]: + pending: list[str] = [] + for root in staged.manifest_roots: + if not root.exists(): + continue + try: + shutil.rmtree(root) + except OSError: + pending.append(str(root)) + return tuple(pending) + + +def stage_task_media_cleanup_plan(plan: TaskMediaCleanupPlan) -> StagedTaskMediaCleanup: + """把托管媒体原子移动到同卷隔离区,尚不执行永久删除。""" + stage_id = f"{plan.task_id}-{uuid4().hex}" + staged_targets: list[StagedMediaTarget] = [] + manifest_roots: list[Path] = [] + for index, target in enumerate(plan.targets): path = target.path if not path.exists(): continue if path.is_symlink() or not path.is_dir(): - raise StorageSafetyError(f"拒绝删除异常的{target.label}:{path}") - size = _directory_size_bytes(path) + raise StorageSafetyError(f"拒绝暂存异常的{target.label}:{path}") + + stage_root = _safe_managed_child( + path.parent, + (DELETE_STAGING_DIR_NAME, stage_id), + f"{target.label}删除隔离区", + ) + staged_path = _safe_managed_child( + stage_root, + (f"{index:02d}-{path.name}",), + f"{target.label}删除暂存目录", + ) + staged_targets.append( + StagedMediaTarget( + label=target.label, + original_path=path, + staged_path=staged_path, + size_bytes=_directory_size_bytes(path), + ) + ) + if stage_root not in manifest_roots: + manifest_roots.append(stage_root) + + staged = StagedTaskMediaCleanup( + task_id=plan.task_id, + stage_id=stage_id, + targets=tuple(staged_targets), + manifest_roots=tuple(manifest_roots), + external_source_preserved=plan.external_source_path is not None, + ) + if not staged.targets: + return staged + + _write_cleanup_manifests(staged, status="prepared") + moved_paths: list[str] = [] + try: + for target in staged.targets: + target.staged_path.parent.mkdir(parents=True, exist_ok=True) + shutil.move(str(target.original_path), str(target.staged_path)) + if target.original_path.exists() or not target.staged_path.exists(): + raise RuntimeError(f"暂存{target.label}后路径状态异常:{target.original_path}") + moved_paths.append(str(target.original_path)) + _write_cleanup_manifests( + staged, + status="staging", + moved_paths=tuple(moved_paths), + ) + _write_cleanup_manifests( + staged, + status="staged", + moved_paths=tuple(moved_paths), + ) + return staged + except Exception as exc: + try: + rollback_staged_task_media_cleanup(staged) + except Exception as rollback_exc: + raise RuntimeError( + f"暂存任务媒体失败且自动恢复未完成:{exc};恢复错误:{rollback_exc}" + ) from exc + raise RuntimeError(f"暂存任务媒体失败,已恢复原目录:{exc}") from exc + + +def rollback_staged_task_media_cleanup(staged: StagedTaskMediaCleanup) -> None: + """数据库提交前失败时,把已经暂存的目录恢复到原路径。""" + if not staged.targets: + return + try: + _write_cleanup_manifests(staged, status="rolling_back") + except OSError: + pass + + errors: list[str] = [] + for target in reversed(staged.targets): + source = target.staged_path + destination = target.original_path + if source.exists(): + if destination.exists(): + errors.append(f"原路径已被占用:{destination}") + continue + try: + destination.parent.mkdir(parents=True, exist_ok=True) + shutil.move(str(source), str(destination)) + except OSError as exc: + errors.append(f"恢复{target.label}失败:{exc}") + elif not destination.exists(): + errors.append(f"原路径与暂存路径都不存在:{destination}") + + if errors: try: - shutil.rmtree(path) - except OSError as exc: - raise RuntimeError(f"删除{target.label}失败:{path};原因:{exc}") from exc - freed_bytes += size - deleted_paths.append(str(path)) + _write_cleanup_manifests(staged, status="recovery_required") + except OSError: + pass + raise RuntimeError(";".join(errors)) + try: + _write_cleanup_manifests(staged, status="rolled_back") + except OSError: + pass + _cleanup_stage_roots(staged) + + +def finalize_staged_task_media_cleanup(staged: StagedTaskMediaCleanup) -> TaskMediaCleanupResult: + """数据库已提交后清除隔离目录;失败时保留清单供安全重试。""" + if staged.targets: + try: + _write_cleanup_manifests( + staged, + status="committed", + moved_paths=tuple(str(target.original_path) for target in staged.targets), + ) + except OSError: + return TaskMediaCleanupResult( + deleted_paths=tuple(str(target.original_path) for target in staged.targets), + freed_bytes=staged.freed_bytes, + external_source_preserved=staged.external_source_preserved, + cleanup_pending=True, + staged_paths=tuple(str(root) for root in staged.manifest_roots if root.exists()), + ) + + pending = _cleanup_stage_roots(staged) return TaskMediaCleanupResult( - deleted_paths=tuple(deleted_paths), - freed_bytes=freed_bytes, - external_source_preserved=plan.external_source_path is not None, + deleted_paths=tuple(str(target.original_path) for target in staged.targets), + freed_bytes=staged.freed_bytes, + external_source_preserved=staged.external_source_preserved, + cleanup_pending=bool(pending), + staged_paths=pending, ) +def apply_task_media_cleanup_plan(plan: TaskMediaCleanupPlan) -> TaskMediaCleanupResult: + """兼容入口:先暂存,再完成清理;需要数据库原子性的调用方应分阶段调用。""" + staged = stage_task_media_cleanup_plan(plan) + return finalize_staged_task_media_cleanup(staged) + + def move_task_directory_to_trash(task_id: str, task_name: str, task_dir_name: str | None = None) -> tuple[str, Path]: current_dir_name = resolve_task_dir_name(task_id, task_dir_name) source_dir = get_task_directory(task_id, current_dir_name) - trash_dir_name = allocate_task_dir_name(task_name, parent_dir_name=TRASH_DIR_NAME, exclude_task_id=task_id) + trash_dir_name = allocate_task_dir_name( + task_name, + parent_dir_name=TRASH_DIR_NAME, + exclude_task_id=task_id, + reserve=False, + ) target_dir = _storage_path_from_dir_name(trash_dir_name) target_dir.parent.mkdir(parents=True, exist_ok=True) if source_dir.exists() and source_dir.resolve() != target_dir.resolve(): shutil.move(str(source_dir), str(target_dir)) return trash_dir_name, target_dir - - -def browse_video_directory(path_value: str | None) -> dict: - allowed_roots = _collect_allowed_roots() - - if path_value: - base_path = Path(path_value) - # 阻止路径遍历 - path_str = str(base_path).replace("\\", "/") - if ".." in Path(path_str).parts: - return {"path": str(base_path), "exists": False, "directories": [], "files": [], - "error": "路径包含不安全的跳转字符"} - if not _is_path_within_roots(base_path, roots=allowed_roots): - # 回退到 STORAGE_ROOT - base_path = settings.storage_root - else: - base_path = settings.storage_root - - if not base_path.exists(): - return {"path": str(base_path), "exists": False, "directories": [], "files": []} - if base_path.is_file(): - base_path = base_path.parent - - # 再次确认父目录在允许范围内 - if not _is_path_within_roots(base_path, roots=allowed_roots): - base_path = settings.storage_root - - directories = [] - files = [] - try: - for item in sorted(base_path.iterdir(), key=lambda value: (value.is_file(), value.name.lower())): - if item.is_dir(): - directories.append({"name": item.name, "path": str(item)}) - elif is_video_file(item): - files.append({"name": item.name, "path": str(item), "size": item.stat().st_size}) - except PermissionError: - return {"path": str(base_path), "exists": True, "directories": [], "files": [], - "error": "没有权限浏览此目录"} - - return { - "path": str(base_path), - "exists": True, - "directories": directories, - "files": files, - } diff --git a/app/services/subtitle_ai_service.py b/app/services/subtitle_ai_service.py new file mode 100644 index 0000000..13b3eed --- /dev/null +++ b/app/services/subtitle_ai_service.py @@ -0,0 +1,105 @@ +"""字幕 AI 纠错建议:只生成非 active 文字建议 revision。""" + +from __future__ import annotations + +import json +from typing import Any + +from app.services.ai.ai_clip_analyzer import build_provider, loads_ai_json +from app.services.ai.base import generate_json_with_safe_retry +from app.services.subtitle_data_service import ( + create_suggestion_revision, + get_revision, + get_suggestion_diff, + get_track, +) + + +AI_BATCH_SIZE = 50 +ALLOWED_SUGGESTION_KEYS = {"cue_id", "suggested_text", "reason"} + + +def generate_subtitle_suggestions( + track_id: str, + *, + revision_id: str, + cue_ids: list[str], + instructions: str = "", + provider: Any | None = None, +) -> dict[str, Any]: + track = get_track(track_id) + if track.get("active_revision_id") != revision_id: + raise ValueError("当前字幕版本已经变化,请刷新后重新生成 AI 建议") + revision = get_revision(revision_id, include_cues=True) + if revision["track_id"] != track_id: + raise ValueError("revision 不属于当前字幕轨") + selected_ids = list(dict.fromkeys(str(value) for value in cue_ids if str(value))) + if not selected_ids: + raise ValueError("请先勾选需要 AI 纠错的字幕") + if len(selected_ids) > 500: + raise ValueError("单次最多选择 500 条字幕") + cue_map = {str(cue["id"]): cue for cue in revision["cues"]} + if not set(selected_ids) <= cue_map.keys(): + raise ValueError("选中的 cue 不属于当前 revision") + + resolved_provider = provider or build_provider() + suggestions: dict[str, str] = {} + for offset in range(0, len(selected_ids), AI_BATCH_SIZE): + batch_ids = selected_ids[offset : offset + AI_BATCH_SIZE] + prompt = _build_prompt([cue_map[cue_id] for cue_id in batch_ids], instructions) + payload = loads_ai_json(generate_json_with_safe_retry(resolved_provider, prompt)) + suggestions.update(_validate_suggestion_payload(payload, allowed_ids=set(batch_ids))) + + suggestion_revision = create_suggestion_revision( + track_id, + base_revision_id=revision_id, + suggested_text_by_cue_id=suggestions, + note=f"AI 字幕纠错建议 · {getattr(resolved_provider, 'name', 'AI')}", + ) + return { + "revision": suggestion_revision, + "base_revision_id": revision_id, + "provider": getattr(resolved_provider, "name", "AI"), + "diff": get_suggestion_diff(track_id, suggestion_revision["id"]), + } + + +def _build_prompt(cues: list[dict[str, Any]], instructions: str) -> str: + compact = [ + { + "cue_id": str(cue["id"]), + "text": str(cue.get("text") or ""), + } + for cue in cues + ] + extra = instructions.strip() or "修正明显错别字、同音误识别、标点和便于阅读的 cue 内换行" + return ( + "你是中文视频字幕校对员。只输出严格 JSON,不要解释。\n" + "你只能修改 suggested_text 的文字、标点和 cue 内换行;禁止增删 cue,禁止修改时间、说话人或 cue_id。\n" + "没有必要修改的 cue 也必须原样返回。换行请直接使用 JSON 字符串中的 \\n。\n" + f"额外要求:{extra}\n" + "输出结构:{\"suggestions\":[{\"cue_id\":\"...\",\"suggested_text\":\"...\",\"reason\":\"简短原因\"}]}\n" + f"字幕:{json.dumps(compact, ensure_ascii=False, separators=(',', ':'))}" + ) + + +def _validate_suggestion_payload(payload: Any, *, allowed_ids: set[str]) -> dict[str, str]: + if not isinstance(payload, dict) or not isinstance(payload.get("suggestions"), list): + raise ValueError("AI 字幕建议缺少 suggestions 数组") + resolved: dict[str, str] = {} + for item in payload["suggestions"]: + if not isinstance(item, dict): + raise ValueError("AI 字幕建议条目格式无效") + forbidden = set(item) - ALLOWED_SUGGESTION_KEYS + if forbidden: + raise ValueError(f"AI 字幕建议包含禁止字段:{', '.join(sorted(forbidden))}") + cue_id = str(item.get("cue_id") or "") + if cue_id not in allowed_ids or cue_id in resolved: + raise ValueError("AI 字幕建议包含未知或重复 cue_id") + text = str(item.get("suggested_text") or "").strip() + if not text or len(text) > 4000: + raise ValueError("AI 字幕建议文字为空或过长") + resolved[cue_id] = text + if set(resolved) != allowed_ids: + raise ValueError("AI 字幕建议没有完整返回所选 cue") + return resolved diff --git a/app/services/subtitle_auto_workflow_service.py b/app/services/subtitle_auto_workflow_service.py new file mode 100644 index 0000000..adad7d1 --- /dev/null +++ b/app/services/subtitle_auto_workflow_service.py @@ -0,0 +1,616 @@ +"""字幕审核暂停、批量烧录与自动流水线恢复。""" + +from __future__ import annotations + +import hashlib +import json +from pathlib import Path +from typing import Any + +from app.db.database import get_connection +from app.models.task import TaskStatus +from app.services import job_service +from app.services.storage_service import get_artifact_paths, resolve_video_file_path +from app.services.subtitle_data_service import ( + SubtitleRevisionConflict, + approve_revisions_with_connection, + ensure_clip_track, + ensure_source_track, + get_revision, +) +from app.services.subtitle_workflow_service import ( + SubtitleRenderCancelled, + render_subtitles_for_output_clip, +) +from app.services.task_log_service import append_task_log + + +def prepare_task_subtitle_review(task_id: str) -> dict[str, Any]: + """从原片主时间轴生成所有成功切片的字幕草稿。""" + from app.services import task_service + + task = task_service.get_task(task_id, include_video_probe=False) + if not task: + raise ValueError("任务不存在") + allowed_statuses = { + TaskStatus.VIDEO_CUTTING.value, + TaskStatus.SUBTITLE_DRAFTING.value, + TaskStatus.PENDING_SUBTITLE_REVIEW.value, + } + if task.get("status") not in allowed_statuses: + raise ValueError("当前任务状态不能进入字幕审核,请刷新后重试") + source_track = ensure_source_track(task_id) + output_clips = [ + item + for item in task_service.list_output_clips(task_id) + if item.get("status") == "completed" and item.get("file_exists") + ] + if not output_clips: + raise ValueError("没有可生成字幕草稿的成功切片") + tracks = [ensure_clip_track(task_id, item["id"]) for item in output_clips] + lease = job_service.current_job_lease() + with get_connection() as connection: + connection.execute("BEGIN IMMEDIATE") + active_job = connection.execute( + """ + SELECT id FROM workflow_jobs + WHERE task_id = ? AND status IN (?, ?) + AND (? = '' OR id != ?) + LIMIT 1 + """, + ( + task_id, + job_service.JOB_STATUS_QUEUED, + job_service.JOB_STATUS_RUNNING, + lease[0] if lease else "", + lease[0] if lease else "", + ), + ).fetchone() + if active_job: + connection.rollback() + raise ValueError("任务已有其他后台 Job,不能重复进入字幕审核") + lease_condition = "" + lease_params: tuple[str, ...] = () + if lease: + lease_condition = """ + AND EXISTS ( + SELECT 1 FROM workflow_jobs + WHERE id = ? AND task_id = ? AND status = 'running' + AND lease_owner = ? AND lease_token = ? + AND lease_expires_at > strftime('%Y-%m-%dT%H:%M:%S+00:00', 'now') + AND cancel_requested = 0 + ) + """ + lease_params = (lease[0], task_id, lease[1], lease[2]) + cursor = connection.execute( + f""" + UPDATE tasks + SET status = ?, progress = ?, error_message = NULL, last_error = NULL, updated_at = ? + WHERE id = ? AND COALESCE(is_deleted, 0) = 0 + AND status IN (?, ?, ?) {lease_condition} + """, + ( + TaskStatus.PENDING_SUBTITLE_REVIEW.value, + task_service.STATUS_PROGRESS[TaskStatus.PENDING_SUBTITLE_REVIEW.value], + task_service._now_iso(), + task_id, + TaskStatus.VIDEO_CUTTING.value, + TaskStatus.SUBTITLE_DRAFTING.value, + TaskStatus.PENDING_SUBTITLE_REVIEW.value, + *lease_params, + ), + ) + if cursor.rowcount != 1: + connection.rollback() + if lease: + raise job_service.JobLeaseLostError("字幕草稿提交前 Workflow Job 租约已失效") + raise ValueError("任务状态已变化,不能进入字幕审核") + connection.commit() + append_task_log(task_id, f"字幕草稿已生成:{len(tracks)} 条,流水线暂停等待人工审核") + return { + "status": "pending_subtitle_review", + "source_track_id": source_track["id"], + "track_count": len(tracks), + "tracks": tracks, + } + + +def enqueue_task_subtitle_render( + task_id: str, + *, + output_clip_ids: list[str] | None = None, + approve_active_revisions: bool = False, + continue_pipeline: bool = False, +) -> dict[str, Any]: + from app.services import task_service + + task = task_service.get_task(task_id, include_video_probe=False) + if not task: + raise ValueError("任务不存在") + if continue_pipeline and task.get("status") != TaskStatus.PENDING_SUBTITLE_REVIEW.value: + raise ValueError("当前任务不在字幕审核暂停状态") + requested = set(output_clip_ids or []) + output_clips = [ + item + for item in task_service.list_output_clips(task_id) + if item.get("status") == "completed" + and item.get("file_exists") + and (not requested or item["id"] in requested) + ] + if requested and {item["id"] for item in output_clips} != requested: + raise ValueError("部分切片不存在或尚未生成完成") + if not output_clips: + raise ValueError("没有可烧录字幕的成功切片") + + items = [] + approvals: list[tuple[str, str]] = [] + for output_clip in output_clips: + track = ensure_clip_track(task_id, output_clip["id"]) + revision_id = str(track.get("active_revision_id") or "") + if not revision_id: + raise ValueError(f"{output_clip.get('output_file_name') or output_clip['id']} 没有字幕 revision") + revision = get_revision(revision_id) + if int(revision.get("cue_count") or 0) <= 0: + raise ValueError(f"{output_clip.get('output_file_name') or output_clip['id']} 没有可烧录的字幕内容") + approvals.append((str(track["id"]), revision_id)) + items.append( + { + "output_clip_id": output_clip["id"], + "revision_id": revision_id, + "output_file_name": output_clip.get("output_file_name") or output_clip["id"], + } + ) + + job_payload = { + "items": items, + "continue_pipeline": continue_pipeline, + "subtitle_delivery_mode": "subtitled", + } + with get_connection() as connection: + connection.execute("BEGIN IMMEDIATE") + if continue_pipeline: + current_task = connection.execute( + "SELECT status FROM tasks WHERE id = ? AND COALESCE(is_deleted, 0) = 0", + (task_id,), + ).fetchone() + if not current_task or current_task["status"] != TaskStatus.PENDING_SUBTITLE_REVIEW.value: + connection.rollback() + raise ValueError("当前任务已离开字幕审核暂停状态,请刷新后重试") + active_resume = connection.execute( + """ + SELECT 1 FROM workflow_jobs + WHERE task_id = ? AND job_type = ? AND status IN (?, ?) + LIMIT 1 + """, + ( + task_id, + job_service.JOB_TYPE_AUTO_PIPELINE, + job_service.JOB_STATUS_QUEUED, + job_service.JOB_STATUS_RUNNING, + ), + ).fetchone() + if active_resume: + connection.rollback() + raise ValueError("后续自动流水线已排队或运行,不能重复执行字幕烧录") + if approve_active_revisions: + approve_revisions_with_connection(connection, approvals) + else: + _validate_approved_revisions_with_connection(connection, approvals) + job_id, created = job_service.create_or_get_active_job_with_connection( + connection, + task_id=task_id, + job_type=job_service.JOB_TYPE_SUBTITLE, + payload=job_payload, + ) + existing_payload_row = connection.execute( + "SELECT payload_json FROM workflow_jobs WHERE id = ?", + (job_id,), + ).fetchone() + try: + existing_payload = json.loads(existing_payload_row["payload_json"] or "{}") + except (TypeError, json.JSONDecodeError) as exc: + raise ValueError("已有字幕 Job 的 payload 已损坏,请先处理该任务") from exc + if existing_payload != job_payload: + raise ValueError("已有不同内容的字幕任务正在排队或运行,请等待完成或取消后重试") + if continue_pipeline: + _set_subtitle_delivery_mode_with_connection(connection, task_id, "subtitled") + connection.commit() + job = job_service.get_job(job_id) + if not job: + raise RuntimeError("字幕 Job 创建后无法读取") + append_task_log( + task_id, + f"字幕批量烧录{'已加入队列' if created else '已在队列中'}:{len(items)} 条", + ) + return { + "status": job["status"], + "message": "字幕批量烧录已加入持久化队列" if created else "已有字幕烧录任务正在排队或运行", + "job": job, + "job_id": job["id"], + "created": created, + "item_count": len(items), + } + + +def skip_task_subtitles_to_review(task_id: str) -> dict[str, Any]: + from app.services import task_service + + now = task_service._now_iso() + with get_connection() as connection: + connection.execute("BEGIN IMMEDIATE") + task = connection.execute( + """ + SELECT auto_mode, status, auto_config_json + FROM tasks + WHERE id = ? AND COALESCE(is_deleted, 0) = 0 + """, + (task_id,), + ).fetchone() + if not task: + connection.rollback() + raise ValueError("任务不存在") + if not bool(task["auto_mode"]): + connection.rollback() + raise ValueError("只有全自动任务需要执行字幕跳过决策") + if task["status"] != TaskStatus.PENDING_SUBTITLE_REVIEW.value: + connection.rollback() + raise ValueError("当前任务不在字幕审核暂停状态") + active_subtitle_job = connection.execute( + """ + SELECT 1 FROM workflow_jobs + WHERE task_id = ? AND job_type IN (?, ?) AND status IN (?, ?) + LIMIT 1 + """, + ( + task_id, + job_service.JOB_TYPE_SUBTITLE, + job_service.JOB_TYPE_AUTO_PIPELINE, + job_service.JOB_STATUS_QUEUED, + job_service.JOB_STATUS_RUNNING, + ), + ).fetchone() + if active_subtitle_job: + connection.rollback() + raise ValueError("字幕烧录或后续自动流水线仍在运行,请先取消并等待停止后再选择跳过字幕") + try: + config = json.loads(task["auto_config_json"] or "{}") + except (TypeError, json.JSONDecodeError) as exc: + connection.rollback() + raise ValueError("任务字幕配置已损坏,请先修复配置后重试") from exc + if not isinstance(config, dict): + connection.rollback() + raise ValueError("任务字幕配置格式无效,请先修复配置后重试") + config["subtitle_delivery_mode"] = "original" + config["subtitle_decided_at"] = now + connection.execute( + """ + UPDATE tasks + SET auto_config_json = ?, status = ?, progress = ?, + error_message = NULL, last_error = NULL, updated_at = ? + WHERE id = ? AND status = ? + """, + ( + json.dumps(config, ensure_ascii=False), + TaskStatus.pending_review.value, + task_service.STATUS_PROGRESS[TaskStatus.pending_review.value], + now, + task_id, + TaskStatus.PENDING_SUBTITLE_REVIEW.value, + ), + ) + connection.commit() + review_url = f"/tasks/{task_id}/clips/review" + append_task_log(task_id, "用户已明确跳过字幕,已进入片段审核;后续发送中心使用原片切片") + return { + "status": TaskStatus.pending_review.value, + "message": "已跳过字幕,请审核片段后再同步发送中心。", + "review_url": review_url, + } + + +def skip_task_subtitles_and_resume(task_id: str) -> dict[str, Any]: + """兼容旧页面缓存;行为已调整为跳过字幕后进入片段审核。""" + return skip_task_subtitles_to_review(task_id) + + +def execute_subtitle_render_job(job_id: str, task_id: str, payload: dict[str, Any]) -> dict[str, Any]: + items = payload.get("items") or [] + if not isinstance(items, list) or not items: + raise ValueError("字幕 Job 没有待渲染条目") + reconcile_interrupted_subtitle_job(job_id) + job = job_service.get_job(job_id) or {} + checkpoint = job.get("checkpoint_json") if isinstance(job.get("checkpoint_json"), dict) else {} + completed = dict(checkpoint.get("completed") or {}) + total = len(items) + for index, item in enumerate(items): + output_clip_id = str(item.get("output_clip_id") or "") + revision_id = str(item.get("revision_id") or "") + if not output_clip_id or not revision_id: + raise ValueError("字幕 Job 条目缺少 output_clip_id 或 revision_id") + if _checkpoint_result_is_valid(task_id, output_clip_id, revision_id, completed.get(output_clip_id)): + continue + recovered = _find_recoverable_subtitle_result( + task_id, + output_clip_id, + revision_id, + workflow_job_id=job_id, + ) + if recovered: + completed[output_clip_id] = recovered + job_service.update_job_checkpoint( + job_id, + {"completed": completed, "completed_count": len(completed), "total_count": total}, + ) + continue + if job_service.is_cancel_requested(job_id): + raise SubtitleRenderCancelled("用户已取消字幕批量烧录") + item_start = 5 + round(index / total * 88) + item_end = 5 + round((index + 1) / total * 88) + job_service.update_job_progress( + job_id, + item_start, + f"正在烧录第 {index + 1}/{total} 条:{item.get('output_file_name') or output_clip_id}", + ) + result = render_subtitles_for_output_clip( + task_id, + output_clip_id, + revision_id=revision_id, + workflow_job_id=job_id, + progress_start=item_start, + progress_end=item_end, + ) + completed[output_clip_id] = { + "revision_id": revision_id, + "subtitle_job_id": (result.get("job") or {}).get("id") or "", + "output_file_path": (result.get("job") or {}).get("output_file_path") or "", + } + job_service.update_job_checkpoint( + job_id, + {"completed": completed, "completed_count": len(completed), "total_count": total}, + ) + + return { + "task_id": task_id, + "completed_count": len(completed), + "total_count": total, + "completed": completed, + "resume_requested": bool(payload.get("continue_pipeline")), + "resume_job_id": "", + } + + +def cleanup_interrupted_subtitle_job( + workflow_job_id: str, + *, + lease_owner: str, + lease_token: str, + status: str, + message: str, +) -> bool: + """父 Worker 强制终止子进程后,修正从属字幕记录并清理精确临时文件。""" + from app.services.task_service import _now_iso + + with get_connection() as connection: + connection.execute("BEGIN IMMEDIATE") + now = _now_iso() + row = connection.execute( + """ + SELECT task_id FROM workflow_jobs + WHERE id = ? AND status = 'running' AND lease_owner = ? AND lease_token = ? + AND lease_expires_at > ? + """, + (workflow_job_id, lease_owner, lease_token, now), + ).fetchone() + if not row: + connection.rollback() + return False + connection.execute( + """ + UPDATE subtitle_jobs + SET status = ?, error_message = ?, updated_at = ? + WHERE workflow_job_id = ? AND status = 'processing' AND is_active = 0 + """, + (status, message, now, workflow_job_id), + ) + connection.commit() + _, failures = _cleanup_subtitle_attempt_files(str(row["task_id"]), workflow_job_id) + if failures: + append_task_log( + str(row["task_id"]), + f"字幕中断产物有 {len(failures)} 个暂时无法删除,将在下次接管时重试:{'; '.join(failures)}", + ) + return True + + +def reconcile_interrupted_subtitle_job(workflow_job_id: str) -> dict[str, Any]: + """新执行接管后收口旧 processing 记录,并再次清理本 Job 的临时文件。""" + from app.services.task_service import _now_iso + + lease = job_service.current_job_lease() + if not lease or lease[0] != workflow_job_id: + raise job_service.JobLeaseLostError(f"字幕 Job 缺少当前执行租约:{workflow_job_id}") + with get_connection() as connection: + connection.execute("BEGIN IMMEDIATE") + now = _now_iso() + row = connection.execute( + """ + SELECT task_id FROM workflow_jobs + WHERE id = ? AND status = 'running' AND lease_owner = ? AND lease_token = ? + AND lease_expires_at > ? AND cancel_requested = 0 + """, + (workflow_job_id, lease[1], lease[2], now), + ).fetchone() + if not row: + connection.rollback() + raise job_service.JobLeaseLostError(f"字幕 Job 接管前租约已失效:{workflow_job_id}") + cursor = connection.execute( + """ + UPDATE subtitle_jobs + SET status = 'failed', error_message = '上一次执行中断,已由当前 Worker 接管', updated_at = ? + WHERE workflow_job_id = ? AND status IN ('processing', 'queued') AND is_active = 0 + """, + (now, workflow_job_id), + ) + interrupted_count = cursor.rowcount + connection.commit() + deleted_count, failures = _cleanup_subtitle_attempt_files(str(row["task_id"]), workflow_job_id) + if failures: + append_task_log( + str(row["task_id"]), + f"字幕接管时有 {len(failures)} 个中断产物仍被占用,将保留为非活跃残留:{'; '.join(failures)}", + ) + return { + "interrupted_count": interrupted_count, + "deleted_artifact_count": deleted_count, + "cleanup_failures": failures, + } + + +def _cleanup_subtitle_attempt_files(task_id: str, workflow_job_id: str) -> tuple[int, list[str]]: + directory = get_artifact_paths(task_id)["subtitled_dir"] + if not directory.exists() or not directory.is_dir(): + return 0, [] + workflow_marker = hashlib.sha256(workflow_job_id.encode("utf-8")).hexdigest()[:12] + expected_temp_suffixes = { + f".{workflow_job_id}.part.mp4", + f".{workflow_marker}.part.mp4", + } + expected_final_marker = f"_subtitled_{workflow_marker}_" + with get_connection() as connection: + referenced_names = { + Path(str(row["output_file_path"])).name + for row in connection.execute( + """ + SELECT output_file_path FROM subtitle_jobs + WHERE task_id = ? AND COALESCE(output_file_path, '') != '' + """, + (task_id,), + ).fetchall() + } + deleted = 0 + failures: list[str] = [] + try: + candidates = list(directory.iterdir()) + except OSError as exc: + return 0, [f"{directory.name}: {exc}"] + for path in candidates: + try: + is_owned_temp = any(path.name.endswith(suffix) for suffix in expected_temp_suffixes) + is_owned_orphan_final = ( + expected_final_marker in path.name + and path.name.endswith(".mp4") + and path.name not in referenced_names + ) + if (not is_owned_temp and not is_owned_orphan_final) or not path.is_file(): + continue + path.unlink(missing_ok=True) + deleted += 1 + except OSError as exc: + failures.append(f"{path.name}: {exc}") + return deleted, failures + + +def _checkpoint_result_is_valid( + task_id: str, + output_clip_id: str, + revision_id: str, + checkpoint: Any, +) -> bool: + if not isinstance(checkpoint, dict) or checkpoint.get("revision_id") != revision_id: + return False + subtitle_job_id = str(checkpoint.get("subtitle_job_id") or "") + if not subtitle_job_id: + return False + with get_connection() as connection: + row = connection.execute( + """ + SELECT output_file_path FROM subtitle_jobs + WHERE id = ? AND task_id = ? AND output_clip_id = ? AND revision_id = ? + AND status = 'completed' AND validation_status = 'verified' AND is_active = 1 + """, + (subtitle_job_id, task_id, output_clip_id, revision_id), + ).fetchone() + path = resolve_video_file_path(row["output_file_path"]) if row else None + return bool(path and path.exists() and path.is_file()) + + +def _find_recoverable_subtitle_result( + task_id: str, + output_clip_id: str, + revision_id: str, + *, + workflow_job_id: str, +) -> dict[str, str] | None: + """恢复“DB 已提交但 checkpoint 尚未写入”的同一执行结果。""" + with get_connection() as connection: + row = connection.execute( + """ + SELECT sj.id, sj.output_file_path + FROM subtitle_jobs sj + JOIN subtitle_tracks st + ON st.task_id = sj.task_id AND st.output_clip_id = sj.output_clip_id + AND st.track_type = 'clip' AND st.is_active = 1 + JOIN subtitle_revisions sr ON sr.id = sj.revision_id AND sr.track_id = st.id + WHERE sj.task_id = ? AND sj.output_clip_id = ? AND sj.revision_id = ? + AND sj.workflow_job_id = ? AND sj.status = 'completed' + AND sj.validation_status = 'verified' AND sj.is_active = 1 + AND st.active_revision_id = sj.revision_id AND sr.status = 'approved' + ORDER BY sj.updated_at DESC LIMIT 1 + """, + (task_id, output_clip_id, revision_id, workflow_job_id), + ).fetchone() + path = resolve_video_file_path(row["output_file_path"]) if row else None + if not path or not path.exists() or not path.is_file(): + return None + return { + "revision_id": revision_id, + "subtitle_job_id": str(row["id"]), + "output_file_path": str(row["output_file_path"]), + } + + +def _validate_approved_revisions_with_connection( + connection, + approvals: list[tuple[str, str]], +) -> None: + for track_id, revision_id in approvals: + row = connection.execute( + """ + SELECT st.active_revision_id, sr.track_id, sr.status, sr.cue_count + FROM subtitle_tracks st + JOIN subtitle_revisions sr ON sr.id = ? + WHERE st.id = ? AND st.is_active = 1 + """, + (revision_id, track_id), + ).fetchone() + if not row or row["track_id"] != track_id: + raise ValueError("字幕 revision 不属于当前字幕轨") + if str(row["active_revision_id"] or "") != revision_id: + raise SubtitleRevisionConflict("字幕已产生新版本,请刷新后重新烧录") + if row["status"] != "approved": + raise ValueError("存在尚未审核的字幕 revision") + if int(row["cue_count"] or 0) <= 0: + raise ValueError("存在没有可烧录内容的字幕 revision") + + +def _set_subtitle_delivery_mode_with_connection(connection, task_id: str, mode: str) -> None: + if mode not in {"subtitled", "original"}: + raise ValueError("字幕交付模式无效") + from app.services.task_service import _now_iso + + now = _now_iso() + row = connection.execute("SELECT auto_config_json FROM tasks WHERE id = ?", (task_id,)).fetchone() + if not row: + raise ValueError("任务不存在") + try: + config = json.loads(row["auto_config_json"] or "{}") + except (TypeError, json.JSONDecodeError) as exc: + raise ValueError("任务字幕配置已损坏,请先修复配置后重试") from exc + if not isinstance(config, dict): + raise ValueError("任务字幕配置格式无效,请先修复配置后重试") + config["subtitle_delivery_mode"] = mode + config["subtitle_decided_at"] = now + connection.execute( + "UPDATE tasks SET auto_config_json = ?, updated_at = ? WHERE id = ?", + (json.dumps(config, ensure_ascii=False), now, task_id), + ) diff --git a/app/services/subtitle_data_service.py b/app/services/subtitle_data_service.py new file mode 100644 index 0000000..5b9383a --- /dev/null +++ b/app/services/subtitle_data_service.py @@ -0,0 +1,1443 @@ +"""不可变字幕轨、revision、cue、导入导出与长音频波形服务。""" + +from __future__ import annotations + +from array import array +from datetime import datetime, timezone +import hashlib +import json +import math +from pathlib import Path +import re +import shutil +import subprocess +import sys +from typing import Any, Iterable +from uuid import uuid4 + +import pysubs2 + +from app.core.config import settings +from app.db.database import get_connection +from app.services.ai.ai_clip_analyzer import _extract_transcript_rows, _read_transcript +from app.services.storage_service import ( + get_artifact_paths, + get_source_video_path, + resolve_video_file_path, +) +from app.services.transcription_checkpoint_service import fingerprint_file + + +MAX_CUES_PER_REVISION = 20_000 +MAX_RANGE_LIMIT = 2_000 +DEFAULT_RANGE_LIMIT = 500 +QUALITY_MAX_LINES = 2 +QUALITY_MAX_CHINESE_CHARS_PER_LINE = 18 +QUALITY_MIN_DURATION_MS = 800 +QUALITY_MAX_DURATION_MS = 7_000 +QUALITY_MIN_GAP_MS = 80 +QUALITY_MAX_CHINESE_CHARS_PER_SECOND = 12 + + +class SubtitleRevisionConflict(ValueError): + pass + + +def _now_iso() -> str: + return datetime.now(timezone.utc).isoformat(timespec="seconds") + + +def list_task_tracks(task_id: str, *, ensure: bool = True) -> list[dict[str, Any]]: + if ensure: + ensure_source_track(task_id) + with get_connection() as connection: + output_rows = connection.execute( + """ + SELECT id FROM output_clip + WHERE task_id = ? AND is_active = 1 AND status = 'completed' + ORDER BY created_at ASC + """, + (task_id,), + ).fetchall() + for row in output_rows: + ensure_clip_track(task_id, row["id"]) + + with get_connection() as connection: + rows = connection.execute( + """ + SELECT st.*, sr.revision_number, sr.origin AS revision_origin, + sr.status AS revision_status, sr.cue_count, + oc.output_file_name, oc.source_start_ms, oc.source_end_ms, + oc.source_duration_ms, oc.snapshot_source + FROM subtitle_tracks st + LEFT JOIN subtitle_revisions sr ON sr.id = st.active_revision_id + LEFT JOIN output_clip oc ON oc.id = st.output_clip_id + WHERE st.task_id = ? AND st.is_active = 1 + ORDER BY CASE st.track_type WHEN 'source' THEN 0 ELSE 1 END, + oc.source_start_ms, st.created_at + """, + (task_id,), + ).fetchall() + return [_track_to_dict(dict(row)) for row in rows] + + +def get_track(track_id: str) -> dict[str, Any]: + with get_connection() as connection: + row = connection.execute( + """ + SELECT st.*, sr.revision_number, sr.origin AS revision_origin, + sr.status AS revision_status, sr.cue_count, + oc.output_file_name, oc.source_start_ms, oc.source_end_ms, + oc.source_duration_ms, oc.snapshot_source + FROM subtitle_tracks st + LEFT JOIN subtitle_revisions sr ON sr.id = st.active_revision_id + LEFT JOIN output_clip oc ON oc.id = st.output_clip_id + WHERE st.id = ? + """, + (track_id,), + ).fetchone() + if not row: + raise ValueError("字幕轨不存在") + return _track_to_dict(dict(row)) + + +def ensure_source_track(task_id: str, *, force: bool = False) -> dict[str, Any]: + cues, source_fingerprint, origin = _load_source_cues(task_id) + if not cues: + raise ValueError("当前任务没有可用的结构化转写或时间戳 Markdown") + with get_connection() as connection: + connection.execute("BEGIN IMMEDIATE") + if not connection.execute("SELECT id FROM tasks WHERE id = ?", (task_id,)).fetchone(): + connection.rollback() + raise ValueError("任务不存在") + existing = connection.execute( + """ + SELECT * FROM subtitle_tracks + WHERE task_id = ? AND track_type = 'source' AND is_active = 1 + ORDER BY created_at DESC LIMIT 1 + """, + (task_id,), + ).fetchone() + if ( + existing + and existing["source_fingerprint"] == source_fingerprint + and existing["active_revision_id"] + ): + track_id = str(existing["id"]) + connection.commit() + return get_track(track_id) + if existing and existing["has_manual_edits"] and not force: + connection.execute( + "UPDATE subtitle_tracks SET sync_status = 'pending_source_refresh', updated_at = ? WHERE id = ?", + (_now_iso(), existing["id"]), + ) + connection.commit() + return get_track(str(existing["id"])) + + now = _now_iso() + track_id = str(existing["id"]) if existing else uuid4().hex + base_revision_id = str(existing["active_revision_id"] or "") if existing else "" + if not existing: + connection.execute( + """ + INSERT INTO subtitle_tracks ( + id, task_id, track_type, output_clip_id, name, language, + source_fingerprint, sync_status, has_manual_edits, + is_active, created_at, updated_at + ) VALUES (?, ?, 'source', NULL, '原片主字幕', 'zh-CN', ?, + 'up_to_date', 0, 1, ?, ?) + """, + (track_id, task_id, source_fingerprint, now, now), + ) + revision = _insert_revision_with_connection( + connection, + track_id, + cues, + origin=origin, + parent_revision_id=base_revision_id or None, + status="draft", + note="从结构化转写生成原片主时间轴" if origin == "asr" else "从旧版 Markdown 兼容生成", + activate=False, + ) + cursor = connection.execute( + """ + UPDATE subtitle_tracks + SET source_fingerprint = ?, active_revision_id = ?, sync_status = 'up_to_date', + has_manual_edits = 0, updated_at = ? + WHERE id = ? AND is_active = 1 + AND ((active_revision_id = ?) OR (active_revision_id IS NULL AND ? = '')) + """, + ( + source_fingerprint, + revision["id"], + now, + track_id, + base_revision_id, + base_revision_id, + ), + ) + if cursor.rowcount != 1: + connection.rollback() + raise SubtitleRevisionConflict("原片字幕已产生新版本,请刷新后重试") + connection.commit() + + with get_connection() as connection: + clip_tracks = connection.execute( + "SELECT id FROM subtitle_tracks WHERE task_id = ? AND track_type = 'clip' AND is_active = 1", + (task_id,), + ).fetchall() + for clip_track in clip_tracks: + sync_clip_track(clip_track["id"], force=False) + return get_track(track_id) + + +def ensure_clip_track(task_id: str, output_clip_id: str) -> dict[str, Any]: + source_track = ensure_source_track(task_id) + output = ensure_output_clip_snapshot(task_id, output_clip_id) + with get_connection() as connection: + connection.execute("BEGIN IMMEDIATE") + existing = connection.execute( + """ + SELECT * FROM subtitle_tracks + WHERE task_id = ? AND track_type = 'clip' AND output_clip_id = ? AND is_active = 1 + """, + (task_id, output_clip_id), + ).fetchone() + if not existing: + now = _now_iso() + track_id = uuid4().hex + connection.execute( + """ + INSERT INTO subtitle_tracks ( + id, task_id, track_type, output_clip_id, name, language, + source_track_id, source_revision_id, source_fingerprint, + sync_status, has_manual_edits, is_active, created_at, updated_at + ) VALUES (?, ?, 'clip', ?, ?, 'zh-CN', ?, NULL, ?, + 'pending_sync', 0, 1, ?, ?) + """, + ( + track_id, + task_id, + output_clip_id, + output.get("output_file_name") or "切片字幕", + source_track["id"], + source_track.get("source_fingerprint") or "", + now, + now, + ), + ) + else: + track_id = existing["id"] + connection.commit() + sync_clip_track(track_id, force=False) + return get_track(track_id) + + +def sync_clip_track(track_id: str, *, force: bool = False) -> dict[str, Any]: + track_hint = get_track(track_id) + if track_hint["track_type"] != "clip": + raise ValueError("只有切片字幕轨可以从原片同步") + ensure_output_clip_snapshot(track_hint["task_id"], track_hint["output_clip_id"]) + with get_connection() as connection: + connection.execute("BEGIN IMMEDIATE") + track_row = connection.execute( + "SELECT * FROM subtitle_tracks WHERE id = ? AND is_active = 1", + (track_id,), + ).fetchone() + if not track_row: + connection.rollback() + raise ValueError("字幕轨不存在或已停用") + track = dict(track_row) + if track["track_type"] != "clip": + connection.rollback() + raise ValueError("只有切片字幕轨可以从原片同步") + source_track_row = connection.execute( + "SELECT * FROM subtitle_tracks WHERE id = ? AND track_type = 'source' AND is_active = 1", + (track["source_track_id"],), + ).fetchone() + if not source_track_row or not source_track_row["active_revision_id"]: + connection.rollback() + raise ValueError("原片字幕还没有可同步 revision") + source_track = dict(source_track_row) + source_revision_id = str(source_track["active_revision_id"]) + if track.get("has_manual_edits") and not force: + if track.get("source_revision_id") != source_revision_id: + connection.execute( + "UPDATE subtitle_tracks SET sync_status = 'pending_sync', updated_at = ? WHERE id = ?", + (_now_iso(), track_id), + ) + connection.commit() + return get_track(track_id) + if track.get("source_revision_id") == source_revision_id and track.get("active_revision_id"): + connection.commit() + return get_track(track_id) + + output = connection.execute( + """ + SELECT source_start_ms, source_end_ms + FROM output_clip WHERE id = ? AND task_id = ? + """, + (track["output_clip_id"], track["task_id"]), + ).fetchone() + if not output or output["source_start_ms"] is None or output["source_end_ms"] is None: + connection.rollback() + raise ValueError("切片记录缺少原片时间范围") + source_cues = [ + dict(row) + for row in connection.execute( + "SELECT * FROM subtitle_cues WHERE revision_id = ? ORDER BY cue_index ASC", + (source_revision_id,), + ).fetchall() + ] + local_cues = inherit_cues_for_clip( + source_cues, + int(output["source_start_ms"]), + int(output["source_end_ms"]), + ) + base_revision_id = str(track.get("active_revision_id") or "") + revision = _insert_revision_with_connection( + connection, + track_id, + local_cues, + origin="source_sync", + parent_revision_id=base_revision_id or None, + status="draft", + note=f"继承原片 revision {source_revision_id}", + activate=False, + ) + cursor = connection.execute( + """ + UPDATE subtitle_tracks + SET source_revision_id = ?, source_fingerprint = ?, active_revision_id = ?, + sync_status = 'up_to_date', has_manual_edits = 0, updated_at = ? + WHERE id = ? AND is_active = 1 + AND ((active_revision_id = ?) OR (active_revision_id IS NULL AND ? = '')) + """, + ( + source_revision_id, + source_track.get("source_fingerprint") or "", + revision["id"], + _now_iso(), + track_id, + base_revision_id, + base_revision_id, + ), + ) + if cursor.rowcount != 1: + connection.rollback() + raise SubtitleRevisionConflict("切片字幕已产生新版本,请刷新后重试同步") + connection.commit() + return get_track(track_id) + + +def ensure_output_clip_snapshot(task_id: str, output_clip_id: str) -> dict[str, Any]: + with get_connection() as connection: + row = connection.execute( + """ + SELECT oc.*, cc.start_time, cc.end_time + FROM output_clip oc + LEFT JOIN clip_candidates cc ON cc.id = oc.clip_candidate_id + WHERE oc.id = ? AND oc.task_id = ? + """, + (output_clip_id, task_id), + ).fetchone() + if not row: + raise ValueError("切片记录不存在") + output = dict(row) + if output.get("source_start_ms") is not None and output.get("source_end_ms") is not None: + return output + if not output.get("start_time") or not output.get("end_time"): + raise ValueError("旧切片缺少可推断的原片边界,请重新生成切片") + from app.services.task_service import _parse_time_to_seconds + + start_ms = round(_parse_time_to_seconds(output["start_time"]) * 1000) + end_ms = round(_parse_time_to_seconds(output["end_time"]) * 1000) + if end_ms <= start_ms: + raise ValueError("旧切片的原片边界无效,请重新生成切片") + task = _get_task_row(task_id) + source_path = get_source_video_path(task) + source_fingerprint = fingerprint_file(source_path) if source_path and source_path.exists() else "" + with get_connection() as connection: + connection.execute( + """ + UPDATE output_clip + SET source_start_ms = ?, source_end_ms = ?, source_duration_ms = ?, + source_fingerprint = ?, snapshot_source = 'legacy_inferred', updated_at = ? + WHERE id = ? + """, + (start_ms, end_ms, end_ms - start_ms, source_fingerprint, _now_iso(), output_clip_id), + ) + connection.commit() + return ensure_output_clip_snapshot(task_id, output_clip_id) + + +def inherit_cues_for_clip( + source_cues: Iterable[dict[str, Any]], + source_start_ms: int, + source_end_ms: int, +) -> list[dict[str, Any]]: + inherited = [] + for cue in source_cues: + cue_start = int(cue["start_ms"]) + cue_end = int(cue["end_ms"]) + if cue_end <= source_start_ms or cue_start >= source_end_ms: + continue + local_start = max(0, cue_start - source_start_ms) + local_end = min(source_end_ms, cue_end) - source_start_ms + if local_end <= local_start: + continue + inherited.append( + { + "start_ms": local_start, + "end_ms": local_end, + "text": cue["text"], + "confidence": cue.get("confidence"), + "speaker": cue.get("speaker") or "", + "source_cue_id": cue.get("id") or cue.get("source_cue_id"), + } + ) + return inherited + + +def create_manual_revision( + track_id: str, + *, + base_revision_id: str | None, + cues: Iterable[Any], + note: str = "", +) -> dict[str, Any]: + normalized = [_cue_input_to_dict(cue) for cue in cues] + with get_connection() as connection: + connection.execute("BEGIN IMMEDIATE") + track_row = connection.execute( + "SELECT * FROM subtitle_tracks WHERE id = ? AND is_active = 1", + (track_id,), + ).fetchone() + if not track_row: + connection.rollback() + raise ValueError("字幕轨不存在或已停用") + track = dict(track_row) + if (track.get("active_revision_id") or None) != (base_revision_id or None): + connection.rollback() + raise SubtitleRevisionConflict("字幕已产生新版本,请刷新后再保存,当前编辑没有覆盖新版本") + revision = _insert_revision_with_connection( + connection, + track_id, + normalized, + origin="manual", + parent_revision_id=base_revision_id, + status="draft", + note=note or "字幕编辑器自动保存", + activate=False, + ) + cursor = connection.execute( + """ + UPDATE subtitle_tracks + SET active_revision_id = ?, has_manual_edits = 1, + sync_status = CASE WHEN track_type = 'clip' THEN 'manual' ELSE sync_status END, + updated_at = ? + WHERE id = ? AND is_active = 1 + AND ((active_revision_id = ?) OR (active_revision_id IS NULL AND ? IS NULL)) + """, + (revision["id"], _now_iso(), track_id, base_revision_id, base_revision_id), + ) + if cursor.rowcount != 1: + connection.rollback() + raise SubtitleRevisionConflict("字幕已产生新版本,请刷新后再保存,当前编辑没有覆盖新版本") + connection.commit() + if track["track_type"] == "source": + _sync_dependent_clip_tracks(track["task_id"]) + return get_revision(revision["id"], include_cues=True) + + +def apply_revision_operations( + track_id: str, + *, + base_revision_id: str, + operations: Iterable[Any], + note: str = "", +) -> dict[str, Any]: + cues = _fetch_all_revision_cues(base_revision_id) + for operation_value in operations: + operation = operation_value.model_dump() if hasattr(operation_value, "model_dump") else dict(operation_value) + cues = _apply_operation(cues, operation) + return create_manual_revision( + track_id, + base_revision_id=base_revision_id, + cues=cues, + note=note or "字幕批量编辑", + ) + + +def get_revision(revision_id: str, *, include_cues: bool = False) -> dict[str, Any]: + with get_connection() as connection: + row = connection.execute( + "SELECT * FROM subtitle_revisions WHERE id = ?", + (revision_id,), + ).fetchone() + if not row: + raise ValueError("字幕 revision 不存在") + revision = dict(row) + if include_cues: + revision["cues"] = _fetch_all_revision_cues(revision_id) + revision["quality"] = evaluate_subtitle_quality(revision["cues"]) + return revision + + +def list_revisions(track_id: str) -> list[dict[str, Any]]: + get_track(track_id) + with get_connection() as connection: + rows = connection.execute( + "SELECT * FROM subtitle_revisions WHERE track_id = ? ORDER BY revision_number DESC", + (track_id,), + ).fetchall() + return [dict(row) for row in rows] + + +def get_revision_cues( + track_id: str, + *, + revision_id: str | None = None, + start_ms: int | None = None, + end_ms: int | None = None, + offset: int = 0, + limit: int = DEFAULT_RANGE_LIMIT, +) -> dict[str, Any]: + track = get_track(track_id) + revision_id = revision_id or track.get("active_revision_id") + if not revision_id: + return {"track": track, "revision": None, "cues": [], "total": 0, "quality": {"issues": []}} + revision = get_revision(revision_id) + if revision["track_id"] != track_id: + raise ValueError("revision 不属于当前字幕轨") + clauses = ["revision_id = ?"] + params: list[Any] = [revision_id] + if start_ms is not None: + clauses.append("end_ms > ?") + params.append(max(0, start_ms)) + if end_ms is not None: + clauses.append("start_ms < ?") + params.append(max(0, end_ms)) + safe_limit = max(1, min(MAX_RANGE_LIMIT, int(limit))) + safe_offset = max(0, int(offset)) + where = " AND ".join(clauses) + with get_connection() as connection: + total = connection.execute( + f"SELECT COUNT(*) FROM subtitle_cues WHERE {where}", + params, + ).fetchone()[0] + rows = connection.execute( + f""" + SELECT * FROM subtitle_cues WHERE {where} + ORDER BY cue_index ASC LIMIT ? OFFSET ? + """, + [*params, safe_limit, safe_offset], + ).fetchall() + cues = [dict(row) for row in rows] + return { + "track": track, + "revision": revision, + "cues": cues, + "total": total, + "offset": safe_offset, + "limit": safe_limit, + "quality": evaluate_subtitle_quality(cues), + } + + +def approve_revision(track_id: str, revision_id: str) -> dict[str, Any]: + with get_connection() as connection: + connection.execute("BEGIN IMMEDIATE") + approve_revisions_with_connection( + connection, + [(track_id, revision_id)], + require_non_empty=False, + ) + connection.commit() + return get_revision(revision_id, include_cues=True) + + +def approve_revisions_with_connection( + connection, + approvals: Iterable[tuple[str, str]], + *, + require_non_empty: bool = True, +) -> list[dict[str, Any]]: + """在调用方事务内重新校验并批准一组当前 active revision。""" + now = _now_iso() + approved: list[dict[str, Any]] = [] + for track_id, revision_id in approvals: + track = connection.execute( + "SELECT id, active_revision_id FROM subtitle_tracks WHERE id = ? AND is_active = 1", + (track_id,), + ).fetchone() + if not track: + raise ValueError("字幕轨不存在或已停用") + if str(track["active_revision_id"] or "") != revision_id: + raise SubtitleRevisionConflict("字幕已产生新版本,请刷新后重新审核") + + revision = connection.execute( + "SELECT id, track_id, status, cue_count FROM subtitle_revisions WHERE id = ?", + (revision_id,), + ).fetchone() + if not revision: + raise ValueError("字幕 revision 不存在") + if revision["track_id"] != track_id: + raise ValueError("revision 不属于当前字幕轨") + if require_non_empty and int(revision["cue_count"] or 0) <= 0: + raise ValueError("当前字幕没有可审核内容") + + cues = [ + dict(row) + for row in connection.execute( + "SELECT * FROM subtitle_cues WHERE revision_id = ? ORDER BY cue_index ASC", + (revision_id,), + ).fetchall() + ] + if int(evaluate_subtitle_quality(cues).get("error_count") or 0) > 0: + raise ValueError("当前字幕仍有时间重叠错误,请修正后再审核") + + cursor = connection.execute( + """ + UPDATE subtitle_revisions + SET status = 'approved', approved_at = COALESCE(approved_at, ?) + WHERE id = ? AND track_id = ? + """, + (now, revision_id, track_id), + ) + if cursor.rowcount != 1: + raise SubtitleRevisionConflict("字幕 revision 已变化,请刷新后重新审核") + cursor = connection.execute( + """ + UPDATE subtitle_tracks SET active_revision_id = ?, updated_at = ? + WHERE id = ? AND active_revision_id = ? AND is_active = 1 + """, + (revision_id, now, track_id, revision_id), + ) + if cursor.rowcount != 1: + raise SubtitleRevisionConflict("字幕已产生新版本,请刷新后重新审核") + approved.append( + { + "id": revision_id, + "track_id": track_id, + "status": "approved", + "cue_count": int(revision["cue_count"] or 0), + } + ) + return approved + + +def create_suggestion_revision( + track_id: str, + *, + base_revision_id: str, + suggested_text_by_cue_id: dict[str, str], + note: str = "AI 字幕纠错建议", +) -> dict[str, Any]: + """保存非 active 建议版本;服务端只接受文字变化。""" + track = get_track(track_id) + if track.get("active_revision_id") != base_revision_id: + raise SubtitleRevisionConflict("字幕已产生新版本,请刷新后重新生成 AI 建议") + base_revision = get_revision(base_revision_id) + if base_revision["track_id"] != track_id: + raise ValueError("基础 revision 不属于当前字幕轨") + base_cues = _fetch_all_revision_cues(base_revision_id) + known_ids = {str(cue["id"]) for cue in base_cues} + unknown = set(suggested_text_by_cue_id) - known_ids + if unknown: + raise ValueError("AI 建议包含不属于当前 revision 的 cue") + suggestion_cues = [] + for cue in base_cues: + cue_id = str(cue["id"]) + text = str(suggested_text_by_cue_id.get(cue_id, cue["text"]) or "").strip() + if not text: + raise ValueError("AI 建议不能把字幕文字清空") + suggestion_cues.append( + { + "start_ms": int(cue["start_ms"]), + "end_ms": int(cue["end_ms"]), + "text": text, + "confidence": cue.get("confidence"), + "speaker": cue.get("speaker") or "", + "source_cue_id": cue_id, + } + ) + with get_connection() as connection: + connection.execute("BEGIN IMMEDIATE") + revision = _insert_revision_with_connection( + connection, + track_id, + suggestion_cues, + origin="ai_suggestion", + parent_revision_id=base_revision_id, + status="suggested", + note=note, + activate=False, + ) + connection.commit() + return get_revision(revision["id"], include_cues=True) + + +def get_suggestion_diff(track_id: str, suggestion_revision_id: str) -> list[dict[str, Any]]: + suggestion = get_revision(suggestion_revision_id, include_cues=True) + if suggestion["track_id"] != track_id or suggestion.get("origin") != "ai_suggestion": + raise ValueError("AI 建议 revision 不属于当前字幕轨") + base_revision_id = str(suggestion.get("parent_revision_id") or "") + if not base_revision_id: + raise ValueError("AI 建议缺少基础 revision") + base_cues = {str(cue["id"]): cue for cue in _fetch_all_revision_cues(base_revision_id)} + changes = [] + for cue in suggestion["cues"]: + source_id = str(cue.get("source_cue_id") or "") + base = base_cues.get(source_id) + if not base or str(base.get("text") or "") == str(cue.get("text") or ""): + continue + changes.append( + { + "cue_id": source_id, + "start_ms": int(base["start_ms"]), + "end_ms": int(base["end_ms"]), + "original_text": str(base.get("text") or ""), + "suggested_text": str(cue.get("text") or ""), + } + ) + return changes + + +def accept_suggestion_revision( + track_id: str, + *, + suggestion_revision_id: str, + base_revision_id: str, + cue_ids: Iterable[str], +) -> dict[str, Any]: + """选择接受 AI 文字建议,并生成新的人工草稿 revision。""" + track = get_track(track_id) + if track.get("active_revision_id") != base_revision_id: + raise SubtitleRevisionConflict("字幕已产生新版本,请刷新后再接受 AI 建议") + suggestion = get_revision(suggestion_revision_id, include_cues=True) + if ( + suggestion["track_id"] != track_id + or suggestion.get("origin") != "ai_suggestion" + or suggestion.get("parent_revision_id") != base_revision_id + ): + raise ValueError("AI 建议与当前字幕版本不匹配") + selected = {str(value) for value in cue_ids if str(value)} + if not selected: + raise ValueError("请至少选择一条 AI 建议") + suggested_text = { + str(cue.get("source_cue_id") or ""): str(cue.get("text") or "") + for cue in suggestion["cues"] + } + base_cues = _fetch_all_revision_cues(base_revision_id) + known_ids = {str(cue["id"]) for cue in base_cues} + if not selected <= known_ids: + raise ValueError("选中的 AI 建议不属于当前字幕版本") + merged = [] + for cue in base_cues: + cue_id = str(cue["id"]) + merged.append( + { + **cue, + "text": suggested_text.get(cue_id, cue["text"]) if cue_id in selected else cue["text"], + } + ) + return create_manual_revision( + track_id, + base_revision_id=base_revision_id, + cues=merged, + note=f"接受 {len(selected)} 条 AI 字幕建议", + ) + + +def import_subtitle_text( + track_id: str, + *, + content: str, + format_name: str, + note: str = "", +) -> dict[str, Any]: + format_name = _validate_format(format_name) + try: + document = pysubs2.SSAFile.from_string(content, format_=format_name) + except Exception as exc: + raise ValueError(f"字幕文件解析失败:{exc}") from exc + cues = [ + { + "start_ms": int(event.start), + "end_ms": int(event.end), + "text": event.plaintext.strip(), + "speaker": event.name or "", + "confidence": None, + "source_cue_id": None, + } + for event in document.events + if event.type == "Dialogue" and event.end > event.start and event.plaintext.strip() + ] + with get_connection() as connection: + connection.execute("BEGIN IMMEDIATE") + track_row = connection.execute( + "SELECT * FROM subtitle_tracks WHERE id = ? AND is_active = 1", + (track_id,), + ).fetchone() + if not track_row: + connection.rollback() + raise ValueError("字幕轨不存在或已停用") + track = dict(track_row) + base_revision_id = str(track.get("active_revision_id") or "") + revision = _insert_revision_with_connection( + connection, + track_id, + cues, + origin="import", + parent_revision_id=base_revision_id or None, + status="draft", + note=note or f"导入 {format_name.upper()} 字幕", + activate=False, + ) + cursor = connection.execute( + """ + UPDATE subtitle_tracks + SET active_revision_id = ?, has_manual_edits = 1, + sync_status = 'manual', updated_at = ? + WHERE id = ? AND is_active = 1 + AND ((active_revision_id = ?) OR (active_revision_id IS NULL AND ? = '')) + """, + (revision["id"], _now_iso(), track_id, base_revision_id, base_revision_id), + ) + if cursor.rowcount != 1: + connection.rollback() + raise SubtitleRevisionConflict("字幕已产生新版本,请刷新后重新导入") + connection.commit() + if track["track_type"] == "source": + _sync_dependent_clip_tracks(track["task_id"]) + return get_revision(revision["id"], include_cues=True) + + +def export_subtitle_text( + track_id: str, + *, + revision_id: str | None = None, + format_name: str, +) -> tuple[str, str, str]: + format_name = _validate_format(format_name) + track = get_track(track_id) + revision_id = revision_id or track.get("active_revision_id") + if not revision_id: + raise ValueError("当前字幕轨没有可导出的 revision") + revision = get_revision(revision_id) + if revision["track_id"] != track_id: + raise ValueError("revision 不属于当前字幕轨") + document = _build_pysubs2_document(track, revision_id, dynamic_ass=format_name == "ass") + content = document.to_string(format_name) + media_type = { + "srt": "application/x-subrip", + "vtt": "text/vtt", + "ass": "text/x-ssa", + }[format_name] + return content, media_type, f"{_safe_file_stem(track['name'])}.{format_name}" + + +def serialize_revision_to_ass(track_id: str, revision_id: str) -> str: + track = get_track(track_id) + revision = get_revision(revision_id) + if revision["track_id"] != track_id: + raise ValueError("revision 不属于当前字幕轨") + return _build_pysubs2_document(track, revision_id, dynamic_ass=True).to_string("ass") + + +def evaluate_subtitle_quality(cues: Iterable[dict[str, Any]]) -> dict[str, Any]: + ordered = sorted((dict(cue) for cue in cues), key=lambda cue: (int(cue["start_ms"]), int(cue["end_ms"]))) + issues: list[dict[str, Any]] = [] + previous = None + for cue in ordered: + cue_id = cue.get("id") or "" + text = str(cue.get("text") or "") + duration = int(cue["end_ms"]) - int(cue["start_ms"]) + lines = text.splitlines() or [text] + if len(lines) > QUALITY_MAX_LINES: + issues.append(_issue(cue_id, "too_many_lines", "warning", "建议最多 2 行")) + if any(_chinese_char_count(line) > QUALITY_MAX_CHINESE_CHARS_PER_LINE for line in lines): + issues.append(_issue(cue_id, "line_too_long", "warning", "单行建议不超过 18 个中文字符")) + if duration < QUALITY_MIN_DURATION_MS: + issues.append(_issue(cue_id, "too_short", "warning", "字幕显示时间短于 800ms")) + if duration > QUALITY_MAX_DURATION_MS: + issues.append(_issue(cue_id, "too_long", "warning", "字幕显示时间长于 7 秒")) + chars_per_second = _chinese_char_count(text) / max(0.001, duration / 1000) + if chars_per_second > QUALITY_MAX_CHINESE_CHARS_PER_SECOND: + issues.append(_issue(cue_id, "reading_speed", "warning", "中文阅读速度过快")) + if previous: + gap = int(cue["start_ms"]) - int(previous["end_ms"]) + if gap < 0: + issues.append(_issue(cue_id, "overlap", "error", "字幕时间与上一条重叠")) + elif gap < QUALITY_MIN_GAP_MS: + issues.append(_issue(cue_id, "small_gap", "warning", "与上一条间隔小于 80ms")) + previous = cue + return { + "issues": issues, + "error_count": sum(1 for item in issues if item["severity"] == "error"), + "warning_count": sum(1 for item in issues if item["severity"] == "warning"), + } + + +def get_waveform_peaks(track_id: str, *, max_points: int = 12_000) -> dict[str, Any]: + track = get_track(track_id) + max_points = max(1_000, min(50_000, int(max_points))) + media_path = _track_media_path(track) + if not media_path or not media_path.exists(): + raise ValueError("字幕轨对应的媒体文件不存在") + fingerprint = fingerprint_file(media_path) + cache_dir = get_artifact_paths(track["task_id"])["transcript_path"].parent + cache_dir.mkdir(parents=True, exist_ok=True) + cache_path = cache_dir / f"waveform_{track['track_type']}_{fingerprint[:12]}_{max_points}.json" + if cache_path.exists(): + try: + cached = json.loads(cache_path.read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError): + cached = None + if isinstance(cached, dict) and cached.get("fingerprint") == fingerprint: + return {**cached, "cached": True} + if not shutil.which("ffmpeg"): + raise RuntimeError("FFmpeg 不可用,无法预计算波形 peaks") + command = [ + "ffmpeg", "-v", "error", "-i", str(media_path), "-map", "0:a:0", + "-ac", "1", "-ar", "100", "-f", "s16le", "pipe:1", + ] + try: + result = subprocess.run( + command, + capture_output=True, + timeout=max(900, settings.ffmpeg_audio_extract_timeout), + check=False, + ) + except subprocess.TimeoutExpired as exc: + raise RuntimeError("波形预计算超时") from exc + if result.returncode != 0: + error = result.stderr.decode("utf-8", errors="replace").strip() + raise RuntimeError(error or "波形预计算失败") + samples = array("h") + samples.frombytes(result.stdout) + if sys.byteorder != "little": + samples.byteswap() + peaks = _downsample_peaks(samples, max_points) + duration_ms = round(len(samples) / 100 * 1000) + payload = { + "track_id": track_id, + "fingerprint": fingerprint, + "duration_ms": duration_ms, + "sample_rate": 100, + "point_count": len(peaks), + "peaks": peaks, + "cached": False, + } + temp_path = cache_path.with_suffix(".tmp") + temp_path.write_text(json.dumps(payload, ensure_ascii=False, separators=(",", ":")), encoding="utf-8") + temp_path.replace(cache_path) + return payload + + +def _insert_revision_with_connection( + connection, + track_id: str, + cues: Iterable[dict[str, Any]], + *, + origin: str, + parent_revision_id: str | None, + status: str, + note: str, + activate: bool, +) -> dict[str, Any]: + normalized = _normalize_cues(cues) + if len(normalized) > MAX_CUES_PER_REVISION: + raise ValueError(f"单个字幕 revision 最多 {MAX_CUES_PER_REVISION} 条") + revision_number = connection.execute( + "SELECT COALESCE(MAX(revision_number), 0) + 1 FROM subtitle_revisions WHERE track_id = ?", + (track_id,), + ).fetchone()[0] + revision_id = uuid4().hex + now = _now_iso() + checksum = _cue_checksum(normalized) + connection.execute( + """ + INSERT INTO subtitle_revisions ( + id, track_id, revision_number, origin, parent_revision_id, + status, note, cue_count, checksum, created_at, approved_at + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, NULL) + """, + ( + revision_id, + track_id, + revision_number, + origin, + parent_revision_id, + status, + note, + len(normalized), + checksum, + now, + ), + ) + for index, cue in enumerate(normalized): + connection.execute( + """ + INSERT INTO subtitle_cues ( + id, revision_id, cue_index, start_ms, end_ms, text, + confidence, speaker, source_cue_id, created_at + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + """, + ( + uuid4().hex, + revision_id, + index, + cue["start_ms"], + cue["end_ms"], + cue["text"], + cue.get("confidence"), + cue.get("speaker") or "", + cue.get("source_cue_id"), + now, + ), + ) + if activate: + connection.execute( + "UPDATE subtitle_tracks SET active_revision_id = ?, updated_at = ? WHERE id = ?", + (revision_id, now, track_id), + ) + return { + "id": revision_id, + "track_id": track_id, + "revision_number": revision_number, + "origin": origin, + "status": status, + "cue_count": len(normalized), + "checksum": checksum, + } + + +def _load_source_cues(task_id: str) -> tuple[list[dict[str, Any]], str, str]: + with get_connection() as connection: + run = connection.execute( + """ + SELECT * FROM transcription_runs + WHERE task_id = ? AND is_active = 1 + ORDER BY updated_at DESC LIMIT 1 + """, + (task_id,), + ).fetchone() + chunks = [] + if run: + chunks = connection.execute( + """ + SELECT * FROM transcription_chunks + WHERE run_id = ? AND status = 'completed' + ORDER BY chunk_index ASC + """, + (run["id"],), + ).fetchall() + if run and chunks: + cues: list[dict[str, Any]] = [] + checksum_parts = [] + overlap_ms = int(run["overlap_seconds"] or 0) * 1000 + for chunk in chunks: + raw = str(chunk["result_json"] or "") + checksum = hashlib.sha256(raw.encode("utf-8")).hexdigest() + if not raw or checksum != str(chunk["result_checksum"] or ""): + continue + checksum_parts.append(checksum) + try: + segments = json.loads(raw) + except json.JSONDecodeError: + continue + for segment in segments if isinstance(segments, list) else []: + if not isinstance(segment, dict): + continue + start_ms = int(chunk["start_ms"]) + round(float(segment.get("start_seconds") or 0) * 1000) + end_ms = int(chunk["start_ms"]) + round(float(segment.get("end_seconds") or 0) * 1000) + if int(chunk["chunk_index"]) > 1 and start_ms < int(chunk["start_ms"]) + overlap_ms: + continue + text = " ".join(str(segment.get("text") or "").split()) + if text and end_ms > start_ms: + cues.append( + { + "start_ms": start_ms, + "end_ms": end_ms, + "text": text, + "confidence": segment.get("confidence"), + "speaker": "", + "source_cue_id": None, + } + ) + if cues: + fingerprint = hashlib.sha256( + f"{run['source_fingerprint']}|{'|'.join(checksum_parts)}".encode("utf-8") + ).hexdigest() + return cues, fingerprint, "asr" + + transcript_path = get_artifact_paths(task_id)["transcript_path"] + if not transcript_path.exists(): + return [], "", "markdown" + transcript_text = _read_transcript(transcript_path) + rows = _extract_transcript_rows(transcript_text) + cues = [ + { + "start_ms": row.start_seconds * 1000, + "end_ms": row.end_seconds * 1000, + "text": row.text, + "confidence": None, + "speaker": "", + "source_cue_id": None, + } + for row in rows + if row.end_seconds > row.start_seconds + ] + return cues, hashlib.sha256(transcript_text.encode("utf-8")).hexdigest(), "markdown" + + +def _apply_operation(cues: list[dict[str, Any]], operation: dict[str, Any]) -> list[dict[str, Any]]: + operation_type = operation["type"] + items = [dict(cue) for cue in cues] + by_id = {str(cue.get("id") or ""): index for index, cue in enumerate(items)} + cue_id = str(operation.get("cue_id") or "") + if operation_type == "update": + if cue_id not in by_id: + raise ValueError("要更新的字幕行不存在") + cue = items[by_id[cue_id]] + for field in ("start_ms", "end_ms", "text", "speaker", "confidence"): + if operation.get(field) is not None: + cue[field] = operation[field] + elif operation_type == "split": + if cue_id not in by_id: + raise ValueError("要拆分的字幕行不存在") + index = by_id[cue_id] + cue = items[index] + split_ms = int(operation.get("split_ms") or 0) + if not int(cue["start_ms"]) < split_ms < int(cue["end_ms"]): + raise ValueError("拆分点必须位于字幕时间范围内") + first_text = str(operation.get("text") or cue["text"]).strip() + second_text = str(operation.get("second_text") or cue["text"]).strip() + first = {**cue, "end_ms": split_ms, "text": first_text} + second = {**cue, "id": None, "start_ms": split_ms, "text": second_text} + items[index : index + 1] = [first, second] + elif operation_type == "merge": + selected_ids = set(operation.get("cue_ids") or []) + selected = [cue for cue in items if cue.get("id") in selected_ids] + if len(selected) < 2: + raise ValueError("合并至少需要两条字幕") + selected.sort(key=lambda cue: int(cue["start_ms"])) + merged = { + **selected[0], + "start_ms": min(int(cue["start_ms"]) for cue in selected), + "end_ms": max(int(cue["end_ms"]) for cue in selected), + "text": str(operation.get("text") or " ".join(str(cue["text"]) for cue in selected)).strip(), + } + first_index = min(items.index(cue) for cue in selected) + items = [cue for cue in items if cue.get("id") not in selected_ids] + items.insert(first_index, merged) + elif operation_type == "add": + cue = operation.get("cue") + if not cue: + raise ValueError("新增字幕缺少 cue") + items.append(_cue_input_to_dict(cue)) + elif operation_type == "delete": + selected_ids = set(operation.get("cue_ids") or ([cue_id] if cue_id else [])) + items = [cue for cue in items if cue.get("id") not in selected_ids] + elif operation_type == "shift": + selected_ids = set(operation.get("cue_ids") or []) + delta = int(operation.get("delta_ms") or 0) + for cue in items: + if not selected_ids or cue.get("id") in selected_ids: + duration = int(cue["end_ms"]) - int(cue["start_ms"]) + cue["start_ms"] = max(0, int(cue["start_ms"]) + delta) + cue["end_ms"] = cue["start_ms"] + duration + elif operation_type == "replace": + search = str(operation.get("search") or "") + if not search: + raise ValueError("搜索文字不能为空") + replacement = str(operation.get("replacement") or "") + for cue in items: + cue["text"] = str(cue["text"]).replace(search, replacement) + else: + raise ValueError("不支持的字幕编辑操作") + return _normalize_cues(items) + + +def _normalize_cues(cues: Iterable[dict[str, Any]]) -> list[dict[str, Any]]: + normalized = [] + for raw in cues: + cue = _cue_input_to_dict(raw) + start_ms = int(cue["start_ms"]) + end_ms = int(cue["end_ms"]) + text = str(cue.get("text") or "").strip() + if start_ms < 0 or end_ms <= start_ms: + raise ValueError("字幕时间范围无效") + if not text: + raise ValueError("字幕文字不能为空") + normalized.append( + { + # 编辑操作在同一次请求内会连续执行,必须保留当前 revision 的 cue id。 + # 写入新 revision 时会生成新 id;checksum 也不会把旧 id 算进去。 + "id": cue.get("id"), + "start_ms": start_ms, + "end_ms": end_ms, + "text": text, + "confidence": cue.get("confidence"), + "speaker": str(cue.get("speaker") or "").strip()[:80], + "source_cue_id": cue.get("source_cue_id"), + } + ) + return sorted(normalized, key=lambda cue: (cue["start_ms"], cue["end_ms"])) + + +def _cue_input_to_dict(cue: Any) -> dict[str, Any]: + if hasattr(cue, "model_dump"): + return cue.model_dump() + return dict(cue) + + +def _cue_checksum(cues: list[dict[str, Any]]) -> str: + canonical = [ + { + "start_ms": int(cue["start_ms"]), + "end_ms": int(cue["end_ms"]), + "text": str(cue["text"]), + "confidence": cue.get("confidence"), + "speaker": str(cue.get("speaker") or ""), + "source_cue_id": cue.get("source_cue_id"), + } + for cue in cues + ] + raw = json.dumps(canonical, ensure_ascii=False, sort_keys=True, separators=(",", ":")) + return hashlib.sha256(raw.encode("utf-8")).hexdigest() + + +def _fetch_all_revision_cues(revision_id: str) -> list[dict[str, Any]]: + with get_connection() as connection: + rows = connection.execute( + "SELECT * FROM subtitle_cues WHERE revision_id = ? ORDER BY cue_index ASC", + (revision_id,), + ).fetchall() + return [dict(row) for row in rows] + + +def _sync_dependent_clip_tracks(task_id: str) -> None: + with get_connection() as connection: + rows = connection.execute( + "SELECT id FROM subtitle_tracks WHERE task_id = ? AND track_type = 'clip' AND is_active = 1", + (task_id,), + ).fetchall() + for row in rows: + sync_clip_track(row["id"], force=False) + + +def _track_to_dict(track: dict[str, Any]) -> dict[str, Any]: + track["has_manual_edits"] = bool(track.get("has_manual_edits")) + track["is_active"] = bool(track.get("is_active")) + if track.get("track_type") == "source": + track["media_url"] = f"/media/tasks/{track['task_id']}/source-video" + else: + track["media_url"] = f"/media/tasks/{track['task_id']}/output-clips/{track['output_clip_id']}" + track["peaks_url"] = f"/api/subtitles/tracks/{track['id']}/peaks" + return track + + +def _get_task_row(task_id: str) -> dict[str, Any]: + with get_connection() as connection: + row = connection.execute("SELECT * FROM tasks WHERE id = ?", (task_id,)).fetchone() + if not row: + raise ValueError("任务不存在") + return dict(row) + + +def _track_media_path(track: dict[str, Any]) -> Path | None: + if track["track_type"] == "source": + return get_source_video_path(_get_task_row(track["task_id"])) + with get_connection() as connection: + row = connection.execute( + "SELECT output_file_path FROM output_clip WHERE id = ? AND task_id = ?", + (track["output_clip_id"], track["task_id"]), + ).fetchone() + return resolve_video_file_path(row["output_file_path"]) if row and row["output_file_path"] else None + + +def _build_pysubs2_document( + track: dict[str, Any], + revision_id: str, + *, + dynamic_ass: bool, +) -> pysubs2.SSAFile: + document = pysubs2.SSAFile() + cues = _fetch_all_revision_cues(revision_id) + style_names: dict[str, str] = {} + if dynamic_ass: + style = _get_default_style() + width, height = _probe_media_dimensions(_track_media_path(track)) + document.info["PlayResX"] = str(width) + document.info["PlayResY"] = str(height) + document.info["ScaledBorderAndShadow"] = "yes" + alignment = { + "top_center": pysubs2.Alignment.TOP_CENTER, + "middle_lower": pysubs2.Alignment.MIDDLE_CENTER, + }.get(style.get("position"), pysubs2.Alignment.BOTTOM_CENTER) + base_height = 1920 if height > width else 1080 + scale = height / base_height + font_size = max(18, float(style.get("font_size") or 42) * scale) + margin_v = round(height * float(style.get("safe_area_percent") or 5) / 100) + default_style = pysubs2.SSAStyle( + fontname=_resolve_font(str(style.get("font_family") or "Microsoft YaHei")), + fontsize=font_size, + primarycolor=_hex_color(style.get("font_color") or "#ffffff"), + outlinecolor=_hex_color(style.get("stroke_color") or "#111827"), + backcolor=pysubs2.Color(0, 0, 0, 127), + bold=True, + outline=float(style.get("outline_width") or 3) * scale, + shadow=float(style.get("shadow_depth") or 1) * scale if style.get("shadow_enabled") else 0, + alignment=alignment, + marginl=round(width * 0.055), + marginr=round(width * 0.055), + marginv=margin_v, + ) + document.styles["Default"] = default_style + speaker_styles = style.get("speaker_styles") or {} + for speaker, overrides in speaker_styles.items(): + style_name = f"Speaker_{len(style_names) + 1}" + speaker_style = default_style.copy() + if isinstance(overrides, dict) and overrides.get("font_color"): + speaker_style.primarycolor = _hex_color(overrides["font_color"]) + document.styles[style_name] = speaker_style + style_names[str(speaker)] = style_name + for cue in cues: + speaker = str(cue.get("speaker") or "") + document.events.append( + pysubs2.SSAEvent( + start=int(cue["start_ms"]), + end=int(cue["end_ms"]), + text=str(cue["text"]).replace("\n", r"\N"), + name=speaker, + style=style_names.get(speaker, "Default"), + ) + ) + return document + + +def _get_default_style() -> dict[str, Any]: + with get_connection() as connection: + row = connection.execute( + "SELECT * FROM subtitle_style_presets WHERE is_default = 1 ORDER BY updated_at DESC LIMIT 1" + ).fetchone() + style = dict(row) if row else {} + try: + style["speaker_styles"] = json.loads(style.get("speaker_styles_json") or "{}") + except json.JSONDecodeError: + style["speaker_styles"] = {} + if not style["speaker_styles"]: + style["speaker_styles"] = { + "主播": {"font_color": "#ffffff"}, + "嘉宾": {"font_color": "#ffd60a"}, + } + style["shadow_enabled"] = bool(style.get("shadow_enabled", True)) + return style + + +def _probe_media_dimensions(media_path: Path | None) -> tuple[int, int]: + if not media_path or not media_path.is_file(): + # 独立导出 ASS 时可能没有绑定媒体;此时保留可移植的竖屏画布。 + # 一旦媒体存在,后续任何探测错误都必须显式失败,不能静默套用错误尺寸。 + return 1080, 1920 + ffprobe = shutil.which("ffprobe") + if not ffprobe: + raise RuntimeError("FFprobe 不可用,无法确定 ASS 画布尺寸") + command = [ + ffprobe, "-v", "error", "-select_streams", "v:0", + "-show_entries", "stream=width,height", "-of", "json", str(media_path), + ] + try: + result = subprocess.run( + command, + capture_output=True, + text=True, + encoding="utf-8", + errors="replace", + timeout=settings.ffprobe_timeout, + check=False, + ) + if result.returncode != 0: + raise RuntimeError((result.stderr or "FFprobe 无法读取视频尺寸").strip()[-500:]) + payload = json.loads(result.stdout or "{}") + stream = (payload.get("streams") or [{}])[0] + width = int(stream.get("width") or 0) + height = int(stream.get("height") or 0) + if width > 0 and height > 0: + return width, height + except subprocess.TimeoutExpired as exc: + raise RuntimeError(f"FFprobe 读取视频尺寸超过 {settings.ffprobe_timeout} 秒") from exc + except (OSError, json.JSONDecodeError, ValueError, IndexError) as exc: + raise RuntimeError(f"FFprobe 返回的视频尺寸无效:{exc}") from exc + raise RuntimeError("FFprobe 没有返回有效的视频尺寸") + + +def _resolve_font(font_family: str) -> str: + from app.services.subtitle_workflow_service import _resolve_subtitle_font_family + + return _resolve_subtitle_font_family(font_family) + + +def _hex_color(value: str) -> pysubs2.Color: + match = re.fullmatch(r"#?([0-9A-Fa-f]{6})", str(value or "")) + cleaned = match.group(1) if match else "ffffff" + return pysubs2.Color(int(cleaned[0:2], 16), int(cleaned[2:4], 16), int(cleaned[4:6], 16)) + + +def _downsample_peaks(samples: array, max_points: int) -> list[float]: + if not samples: + return [] + bucket_size = max(1, math.ceil(len(samples) / max_points)) + peaks = [] + for index in range(0, len(samples), bucket_size): + bucket = samples[index : index + bucket_size] + peak = max(bucket, key=lambda value: abs(value)) + peaks.append(round(float(peak) / 32768, 5)) + return peaks + + +def _validate_format(format_name: str) -> str: + normalized = str(format_name or "").lower().lstrip(".") + if normalized not in {"srt", "vtt", "ass"}: + raise ValueError("字幕格式只支持 SRT、VTT 或 ASS") + return normalized + + +def _safe_file_stem(value: str) -> str: + cleaned = re.sub(r"[^\w\-\u4e00-\u9fff]+", "_", value, flags=re.UNICODE).strip("_") + return cleaned[:80] or "subtitles" + + +def _issue(cue_id: str, code: str, severity: str, message: str) -> dict[str, str]: + return {"cue_id": cue_id, "code": code, "severity": severity, "message": message} + + +def _chinese_char_count(value: str) -> int: + return len(re.findall(r"[\u4e00-\u9fff]", value or "")) + + +__all__ = [ + "SubtitleRevisionConflict", + "apply_revision_operations", + "approve_revision", + "approve_revisions_with_connection", + "create_manual_revision", + "ensure_clip_track", + "ensure_output_clip_snapshot", + "ensure_source_track", + "evaluate_subtitle_quality", + "export_subtitle_text", + "get_revision", + "get_revision_cues", + "get_track", + "get_waveform_peaks", + "import_subtitle_text", + "inherit_cues_for_clip", + "list_revisions", + "list_task_tracks", + "serialize_revision_to_ass", + "sync_clip_track", +] diff --git a/app/services/subtitle_workflow_service.py b/app/services/subtitle_workflow_service.py index 60a6544..631e87b 100644 --- a/app/services/subtitle_workflow_service.py +++ b/app/services/subtitle_workflow_service.py @@ -3,14 +3,22 @@ 从 task_service 中拆分出来的字幕样式、ASS 渲染和字幕烧录函数。 """ +import hashlib +import json +import queue import shutil import subprocess +import threading +import time +from functools import lru_cache from pathlib import Path from typing import Any from uuid import uuid4 +from app.core.config import settings +from app.services import job_service +from app.services.managed_process_service import popen_process_group, terminate_process_tree from app.services.storage_service import get_artifact_paths, resolve_video_file_path -from app.services.transcript_service import read_transcript_range # ---------- 字幕字体常量 ---------- @@ -27,9 +35,15 @@ SUBTITLE_STATUS_LABELS = { "pending": "待加字幕", + "queued": "字幕排队中", "processing": "字幕生成中", "completed": "已加字幕", "failed": "字幕失败", + "cancelled": "字幕已取消", +} +DEFAULT_SPEAKER_STYLES = { + "主播": {"font_color": "#ffffff"}, + "嘉宾": {"font_color": "#ffd60a"}, } @@ -58,9 +72,19 @@ def get_default_subtitle_style() -> dict: "font_color": "#ffffff", "stroke_color": "#111827", "shadow_enabled": True, + "outline_width": 3, + "shadow_depth": 1, + "safe_area_percent": 5, + "speaker_styles": DEFAULT_SPEAKER_STYLES, } style = dict(row) style["shadow_enabled"] = bool(style.get("shadow_enabled")) + try: + style["speaker_styles"] = json.loads(style.get("speaker_styles_json") or "{}") + except json.JSONDecodeError: + style["speaker_styles"] = {} + if not style["speaker_styles"]: + style["speaker_styles"] = DEFAULT_SPEAKER_STYLES return style @@ -79,7 +103,9 @@ def update_default_subtitle_style(payload) -> dict: """ UPDATE subtitle_style_presets SET font_family = ?, font_size = ?, position = ?, font_color = ?, - stroke_color = ?, shadow_enabled = ?, updated_at = ? + stroke_color = ?, shadow_enabled = ?, outline_width = ?, + shadow_depth = ?, safe_area_percent = ?, speaker_styles_json = ?, + updated_at = ? WHERE id = ? """, ( @@ -89,6 +115,10 @@ def update_default_subtitle_style(payload) -> dict: payload.font_color, payload.stroke_color, 1 if payload.shadow_enabled else 0, + payload.outline_width, + payload.shadow_depth, + payload.safe_area_percent, + json.dumps(payload.speaker_styles, ensure_ascii=False, separators=(",", ":")), now, "default", ), @@ -98,9 +128,11 @@ def update_default_subtitle_style(payload) -> dict: """ INSERT INTO subtitle_style_presets ( id, name, font_family, font_size, position, font_color, - stroke_color, shadow_enabled, is_default, created_at, updated_at + stroke_color, shadow_enabled, outline_width, shadow_depth, + safe_area_percent, speaker_styles_json, + is_default, created_at, updated_at ) - VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) """, ( "default", @@ -111,6 +143,10 @@ def update_default_subtitle_style(payload) -> dict: payload.font_color, payload.stroke_color, 1 if payload.shadow_enabled else 0, + payload.outline_width, + payload.shadow_depth, + payload.safe_area_percent, + json.dumps(payload.speaker_styles, ensure_ascii=False, separators=(",", ":")), 1, now, now, @@ -159,6 +195,8 @@ def _create_subtitle_job( output_file_path: str = "", error_message: str = "", is_active: int = 0, + revision_id: str | None = None, + workflow_job_id: str | None = None, ) -> dict: """创建新的字幕任务记录(不再 upsert,每次生成都创建新记录)""" from app.db.database import get_connection @@ -170,16 +208,18 @@ def _create_subtitle_job( connection.execute( """ INSERT INTO subtitle_jobs ( - id, task_id, output_clip_id, style_preset_id, status, + id, task_id, output_clip_id, revision_id, workflow_job_id, style_preset_id, status, subtitle_file_path, output_file_path, error_message, is_active, created_at, updated_at ) - VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) """, ( job_id, task_id, output_clip_id, + revision_id, + workflow_job_id, "default", status, subtitle_file_path, @@ -191,7 +231,8 @@ def _create_subtitle_job( ), ) connection.commit() - return {"id": job_id, "task_id": task_id, "output_clip_id": output_clip_id, "status": status, + return {"id": job_id, "task_id": task_id, "output_clip_id": output_clip_id, + "revision_id": revision_id, "workflow_job_id": workflow_job_id, "status": status, "subtitle_file_path": subtitle_file_path, "output_file_path": output_file_path, "error_message": error_message, "is_active": is_active} @@ -216,23 +257,54 @@ def _activate_subtitle_job(task_id: str, output_clip_id: str, job_id: str) -> No connection.commit() -def _update_subtitle_job_status(job_id: str, status: str, error_message: str = "") -> None: +def _update_subtitle_job_status( + job_id: str, + status: str, + error_message: str = "", + *, + workflow_job_id: str | None = None, +) -> None: """更新字幕任务状态(不改变 is_active)""" from app.db.database import get_connection from app.services.task_service import _now_iso - now = _now_iso() with get_connection() as connection: + connection.execute("BEGIN IMMEDIATE") + now = _now_iso() + lease = job_service.current_job_lease() if workflow_job_id else None + if workflow_job_id: + if not lease or lease[0] != workflow_job_id: + connection.rollback() + raise job_service.JobLeaseLostError(f"字幕 Job 缺少当前执行租约:{workflow_job_id}") + active = connection.execute( + """ + SELECT 1 FROM workflow_jobs + WHERE id = ? AND status = 'running' AND lease_owner = ? AND lease_token = ? + AND lease_expires_at > ? + """, + (workflow_job_id, lease[1], lease[2], now), + ).fetchone() + if not active: + connection.rollback() + raise job_service.JobLeaseLostError(f"字幕 Job 租约已失效:{workflow_job_id}") + condition = "id = ?" + params: tuple[str, ...] = (job_id,) + if workflow_job_id: + condition += " AND workflow_job_id = ? AND is_active = 0" + params = (job_id, workflow_job_id) if error_message: - connection.execute( - "UPDATE subtitle_jobs SET status = ?, error_message = ?, updated_at = ? WHERE id = ?", - (status, error_message, now, job_id), + cursor = connection.execute( + f"UPDATE subtitle_jobs SET status = ?, error_message = ?, updated_at = ? WHERE {condition}", + (status, error_message, now, *params), ) else: - connection.execute( - "UPDATE subtitle_jobs SET status = ?, updated_at = ? WHERE id = ?", - (status, now, job_id), + cursor = connection.execute( + f"UPDATE subtitle_jobs SET status = ?, updated_at = ? WHERE {condition}", + (status, now, *params), ) + if workflow_job_id and cursor.rowcount != 1: + connection.rollback() + raise job_service.JobLeaseLostError(f"字幕子任务已被其他执行收口:{job_id}") connection.commit() @@ -274,63 +346,43 @@ def _resolve_subtitle_font_family(requested_font_family: str | None) -> str: def _build_subtitle_rows(task_id: str, output_clip: dict) -> tuple[int, list[dict[str, Any]]]: - from app.services.task_service import _parse_time_to_seconds, get_clip_candidate # noqa: F811 - - clip = get_clip_candidate(task_id, output_clip["clip_candidate_id"]) if output_clip.get("clip_candidate_id") else None - if not clip: - return 0, [{"start_seconds": 0, "end_seconds": 3, "text": output_clip.get("output_file_name") or "精彩片段"}] - - clip_start = int(clip["start_seconds"]) - clip_end = int(clip["end_seconds"]) - rows = read_transcript_range(get_artifact_paths(task_id)["transcript_path"], clip_start, clip_end, max_rows=120) - subtitle_rows = [] - for row in rows: - row_start = _parse_time_to_seconds(row["start_time"]) - row_end = _parse_time_to_seconds(row["end_time"]) - start_seconds = max(0, row_start - clip_start) - end_seconds = max(start_seconds + 1, min(clip_end, row_end) - clip_start) - subtitle_rows.append({"start_seconds": start_seconds, "end_seconds": end_seconds, "text": row["text"]}) - if subtitle_rows: - return clip_start, subtitle_rows - - fallback_text = clip.get("summary") or clip.get("title") or "精彩片段" - return clip_start, [{"start_seconds": 0, "end_seconds": min(5, max(3, clip_end - clip_start)), "text": fallback_text}] - - -def _write_ass_file(task_id: str, output_clip: dict, style: dict) -> Path: + """旧调用方兼容导出;数据来自统一 revision,不再读取并截断 Markdown。""" + from app.services.subtitle_data_service import ensure_clip_track, get_revision + + track = ensure_clip_track(task_id, output_clip["id"]) + revision = get_revision(track["active_revision_id"], include_cues=True) + source_start_ms = int(track.get("source_start_ms") or 0) + rows = [ + { + "start_seconds": int(cue["start_ms"]) / 1000, + "end_seconds": int(cue["end_ms"]) / 1000, + "text": cue["text"], + } + for cue in revision["cues"] + ] + return round(source_start_ms / 1000), rows + + +def _write_ass_file( + task_id: str, + output_clip: dict, + style: dict, + *, + revision_id: str | None = None, +) -> Path: + from app.services.subtitle_data_service import ensure_clip_track, serialize_revision_to_ass + paths = get_artifact_paths(task_id) paths["subtitled_dir"].mkdir(parents=True, exist_ok=True) - subtitle_path = paths["subtitled_dir"] / f"{Path(output_clip.get('output_file_name') or output_clip['id']).stem}.ass" - _, rows = _build_subtitle_rows(task_id, output_clip) - - alignment = "8" if style.get("position") == "top_center" else "2" - margin_v = "92" if style.get("position") == "bottom_center" else "190" - if style.get("position") == "top_center": - margin_v = "70" - outline = "3" if style.get("shadow_enabled") else "1" - shadow = "1" if style.get("shadow_enabled") else "0" - font_family = _resolve_subtitle_font_family(style.get("font_family")) - font_size = int(style.get("font_size") or 42) - primary_color = _hex_to_ass_color(style.get("font_color") or "#ffffff") - outline_color = _hex_to_ass_color(style.get("stroke_color") or "#111827") - events = "\n".join( - f"Dialogue: 0,{_ass_time(row['start_seconds'])},{_ass_time(row['end_seconds'])},Default,,0,0,0,,{_escape_ass_text(row['text'])}" - for row in rows + track = ensure_clip_track(task_id, output_clip["id"]) + selected_revision_id = revision_id or track.get("active_revision_id") + if not selected_revision_id: + raise ValueError("切片字幕轨没有可渲染的 revision") + subtitle_path = paths["subtitled_dir"] / ( + f"{Path(output_clip.get('output_file_name') or output_clip['id']).stem}" + f"_{selected_revision_id[:10]}.ass" ) - content = f"""[Script Info] -ScriptType: v4.00+ -PlayResX: 1080 -PlayResY: 1920 -ScaledBorderAndShadow: yes - -[V4+ Styles] -Format: Name, Fontname, Fontsize, PrimaryColour, SecondaryColour, OutlineColour, BackColour, Bold, Italic, Underline, StrikeOut, ScaleX, ScaleY, Spacing, Angle, BorderStyle, Outline, Shadow, Alignment, MarginL, MarginR, MarginV, Encoding -Style: Default,{font_family},{font_size},{primary_color},&H000000FF,{outline_color},&H7F000000,-1,0,0,0,100,100,0,0,1,{outline},{shadow},{alignment},60,60,{margin_v},1 - -[Events] -Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text -{events} -""" + content = serialize_revision_to_ass(track["id"], selected_revision_id) subtitle_path.write_text(content, encoding="utf-8") return subtitle_path @@ -349,7 +401,24 @@ def _ffmpeg_subtitles_filter(subtitle_path: Path) -> str: # ---------- 字幕烧录入口 ---------- -def render_subtitles_for_output_clip(task_id: str, output_clip_id: str) -> dict: + +class SubtitleRenderCancelled(RuntimeError): + pass + + +def _workflow_file_marker(workflow_job_id: str) -> str: + return hashlib.sha256(workflow_job_id.encode("utf-8")).hexdigest()[:12] + + +def render_subtitles_for_output_clip( + task_id: str, + output_clip_id: str, + *, + revision_id: str | None = None, + workflow_job_id: str | None = None, + progress_start: int = 5, + progress_end: int = 95, +) -> dict: from app.services.task_log_service import append_task_log from app.services.task_service import get_output_clip, get_task # noqa: F811 @@ -365,54 +434,100 @@ def render_subtitles_for_output_clip(task_id: str, output_clip_id: str) -> dict: if not shutil.which("ffmpeg"): raise RuntimeError("FFmpeg 不可用,无法生成字幕视频") + from app.services.subtitle_data_service import ensure_clip_track, get_revision + style = get_default_subtitle_style() + track = ensure_clip_track(task_id, output_clip_id) + selected_revision_id = revision_id or track.get("active_revision_id") + if not selected_revision_id: + raise ValueError("切片字幕轨没有可渲染的 revision") + revision = get_revision(selected_revision_id) + if revision["track_id"] != track["id"]: + raise ValueError("待渲染 revision 不属于当前切片字幕轨") + if revision.get("status") != "approved": + raise ValueError("只有已审核的字幕 revision 才能烧录") paths = get_artifact_paths(task_id) paths["subtitled_dir"].mkdir(parents=True, exist_ok=True) - output_path = paths["subtitled_dir"] / f"{input_path.stem}_subtitled.mp4" + workflow_marker = _workflow_file_marker(workflow_job_id) if workflow_job_id else "" + render_token = f"{workflow_marker}_{uuid4().hex[:10]}" if workflow_marker else uuid4().hex[:10] + output_path = paths["subtitled_dir"] / f"{input_path.stem}_subtitled_{render_token}.mp4" + temp_owner = workflow_marker or render_token + temporary_path = output_path.with_name(f".{output_path.stem}.{temp_owner}.part.mp4") # === 版本化:创建新的字幕 job,不覆盖旧的 === - job = _create_subtitle_job(task_id, output_clip_id, "processing", is_active=0) + job = _create_subtitle_job( + task_id, + output_clip_id, + "processing", + is_active=0, + revision_id=selected_revision_id, + workflow_job_id=workflow_job_id, + ) append_task_log(task_id, f"开始自动加字幕:{input_path.name}") try: - subtitle_path = _write_ass_file(task_id, output_clip, style) - command = [ - "ffmpeg", - "-y", - "-i", - str(input_path), - "-vf", - _ffmpeg_subtitles_filter(subtitle_path), - "-c:a", - "copy", - str(output_path), - ] - result = subprocess.run(command, capture_output=True, text=True, encoding="utf-8", errors="replace") - if result.returncode != 0: - raise RuntimeError(result.stderr.strip() or "FFmpeg 字幕生成失败") + subtitle_path = _write_ass_file( + task_id, + output_clip, + style, + revision_id=selected_revision_id, + ) + source_probe = _probe_media(input_path) + encoder, audio_mode = _render_with_fallback( + input_path, + subtitle_path, + temporary_path, + workflow_job_id=workflow_job_id, + duration_seconds=float(source_probe.get("duration") or 0), + has_audio=bool(source_probe.get("has_audio")), + source_audio_codec=str(source_probe.get("audio_codec") or ""), + progress_start=progress_start, + progress_end=progress_end, + ) + validation = _validate_rendered_media( + temporary_path, + source_duration=float(source_probe.get("duration") or 0), + source_has_audio=bool(source_probe.get("has_audio")), + ) + _finalize_subtitle_job( + task_id=task_id, + output_clip_id=output_clip_id, + revision_id=selected_revision_id, + subtitle_job_id=job["id"], + workflow_job_id=workflow_job_id, + subtitle_path=subtitle_path, + temporary_path=temporary_path, + output_path=output_path, + validation=validation, + encoder=encoder, + audio_mode=audio_mode, + ) + except job_service.JobLeaseLostError: + temporary_path.unlink(missing_ok=True) + raise + except SubtitleRenderCancelled as exc: + temporary_path.unlink(missing_ok=True) + _update_subtitle_job_status( + job["id"], + "cancelled", + error_message=str(exc), + workflow_job_id=workflow_job_id, + ) + append_task_log(task_id, f"字幕烧录已取消:{input_path.name}") + raise except Exception as exc: + temporary_path.unlink(missing_ok=True) error = str(exc) # 失败时:标记当前 job 为 failed,不激活,旧字幕保持 active - _update_subtitle_job_status(job["id"], "failed", error_message=error) + _update_subtitle_job_status( + job["id"], + "failed", + error_message=error, + workflow_job_id=workflow_job_id, + ) append_task_log(task_id, f"自动加字幕失败:{input_path.name},原因:{error}") raise - # 成功:更新 job 信息并切换为 active - _update_subtitle_job_status(job["id"], "completed") - # 用 subtitle_file_path 和 output_file_path 更新记录 - from app.db.database import get_connection - from app.services.task_service import _now_iso - - now = _now_iso() - with get_connection() as connection: - connection.execute( - "UPDATE subtitle_jobs SET subtitle_file_path = ?, output_file_path = ?, updated_at = ? WHERE id = ?", - (str(subtitle_path), str(output_path), now, job["id"]), - ) - connection.commit() - - # 激活当前字幕 job,旧字幕 job 标记为非活跃 - _activate_subtitle_job(task_id, output_clip_id, job["id"]) append_task_log(task_id, f"自动加字幕完成:{output_path.name}") job = _subtitle_job_for_output(task_id, output_clip_id, active_only=False) or job @@ -423,3 +538,366 @@ def render_subtitles_for_output_clip(task_id: str, output_clip_id: str) -> dict: "output_clip": get_output_clip(task_id, output_clip_id), "media_url": f"/media/tasks/{task_id}/subtitled-clips/{output_clip_id}", } + + +def _finalize_subtitle_job( + *, + task_id: str, + output_clip_id: str, + revision_id: str, + subtitle_job_id: str, + workflow_job_id: str | None, + subtitle_path: Path, + temporary_path: Path, + output_path: Path, + validation: dict[str, Any], + encoder: str, + audio_mode: str, +) -> None: + """在 lease/当前 revision 保护下原子切换最终文件与 active 字幕记录。""" + from app.db.database import get_connection + from app.services.task_service import _now_iso + + lease = job_service.current_job_lease() if workflow_job_id else None + if workflow_job_id and (not lease or lease[0] != workflow_job_id): + raise job_service.JobLeaseLostError(f"字幕 Job 缺少当前执行租约:{workflow_job_id}") + final_file_created = False + try: + with get_connection() as connection: + connection.execute("BEGIN IMMEDIATE") + now = _now_iso() + if workflow_job_id: + active_lease = connection.execute( + """ + SELECT 1 FROM workflow_jobs + WHERE id = ? AND status = 'running' AND lease_owner = ? AND lease_token = ? + AND lease_expires_at > ? AND cancel_requested = 0 + """, + (workflow_job_id, lease[1], lease[2], now), + ).fetchone() + if not active_lease: + raise job_service.JobLeaseLostError(f"字幕 Job 最终提交前租约已失效:{workflow_job_id}") + current_revision = connection.execute( + """ + SELECT sr.status + FROM subtitle_tracks st + JOIN subtitle_revisions sr ON sr.id = st.active_revision_id + WHERE st.task_id = ? AND st.output_clip_id = ? AND st.track_type = 'clip' + AND st.is_active = 1 AND st.active_revision_id = ? AND sr.track_id = st.id + """, + (task_id, output_clip_id, revision_id), + ).fetchone() + if not current_revision or current_revision["status"] != "approved": + raise ValueError("字幕 revision 已变化,旧渲染结果不会被激活") + subtitle_job = connection.execute( + """ + SELECT 1 FROM subtitle_jobs + WHERE id = ? AND task_id = ? AND output_clip_id = ? AND revision_id = ? + AND status = 'processing' AND is_active = 0 + AND ((? IS NULL AND workflow_job_id IS NULL) OR workflow_job_id = ?) + """, + ( + subtitle_job_id, + task_id, + output_clip_id, + revision_id, + workflow_job_id, + workflow_job_id, + ), + ).fetchone() + if not subtitle_job: + raise RuntimeError("字幕子任务已被其他执行收口,拒绝激活旧结果") + if output_path.exists(): + raise RuntimeError("字幕最终输出路径已存在,拒绝覆盖") + temporary_path.replace(output_path) + final_file_created = True + connection.execute( + """ + UPDATE subtitle_jobs SET is_active = 0, updated_at = ? + WHERE task_id = ? AND output_clip_id = ? AND id != ? + """, + (now, task_id, output_clip_id, subtitle_job_id), + ) + cursor = connection.execute( + """ + UPDATE subtitle_jobs + SET status = 'completed', subtitle_file_path = ?, output_file_path = ?, + error_message = '', validation_status = 'verified', validation_json = ?, + encoder = ?, verified_at = ?, updated_at = ?, is_active = 1 + WHERE id = ? AND task_id = ? AND output_clip_id = ? AND revision_id = ? + AND status = 'processing' AND is_active = 0 + """, + ( + str(subtitle_path), + str(output_path), + json.dumps({**validation, "audio_mode": audio_mode}, ensure_ascii=False), + encoder, + now, + now, + subtitle_job_id, + task_id, + output_clip_id, + revision_id, + ), + ) + if cursor.rowcount != 1: + raise RuntimeError("字幕最终状态提交冲突,拒绝激活旧结果") + connection.commit() + except Exception as exc: + if final_file_created: + try: + with get_connection() as verification_connection: + persisted = verification_connection.execute( + """ + SELECT 1 FROM subtitle_jobs + WHERE id = ? AND status = 'completed' AND validation_status = 'verified' + AND is_active = 1 AND output_file_path = ? + """, + (subtitle_job_id, str(output_path)), + ).fetchone() + except Exception as verification_exc: + exc.add_note(f"无法确认字幕最终提交是否持久化,已保留文件供恢复:{verification_exc}") + else: + if not persisted: + try: + output_path.unlink(missing_ok=True) + except OSError as cleanup_exc: + exc.add_note(f"回滚字幕最终文件失败:{cleanup_exc}") + raise + + +def _probe_media(path: Path) -> dict[str, Any]: + ffprobe = shutil.which("ffprobe") + if not ffprobe: + raise RuntimeError("FFprobe 不可用,无法验证字幕成片") + try: + result = subprocess.run( + [ + ffprobe, + "-v", + "error", + "-show_streams", + "-show_format", + "-of", + "json", + str(path), + ], + capture_output=True, + text=True, + encoding="utf-8", + errors="replace", + timeout=settings.ffprobe_timeout, + check=False, + ) + except subprocess.TimeoutExpired as exc: + raise RuntimeError("FFprobe 验证字幕视频超时") from exc + if result.returncode != 0: + raise RuntimeError(result.stderr.strip() or "FFprobe 无法读取字幕视频") + try: + payload = json.loads(result.stdout) + except json.JSONDecodeError as exc: + raise RuntimeError("FFprobe 返回了无效 JSON") from exc + streams = payload.get("streams") or [] + video = next((item for item in streams if item.get("codec_type") == "video"), {}) + audio = next((item for item in streams if item.get("codec_type") == "audio"), {}) + duration = (payload.get("format") or {}).get("duration") or video.get("duration") or 0 + return { + "duration": float(duration or 0), + "video_codec": str(video.get("codec_name") or ""), + "pixel_format": str(video.get("pix_fmt") or ""), + "has_audio": bool(audio), + "audio_codec": str(audio.get("codec_name") or ""), + } + + +@lru_cache(maxsize=8) +def _ffmpeg_has_encoder(name: str) -> bool: + try: + result = subprocess.run( + ["ffmpeg", "-hide_banner", "-encoders"], + capture_output=True, + text=True, + encoding="utf-8", + errors="replace", + timeout=20, + check=False, + ) + except (OSError, subprocess.TimeoutExpired): + return False + return result.returncode == 0 and name in result.stdout + + +def _render_with_fallback( + input_path: Path, + subtitle_path: Path, + temporary_path: Path, + *, + workflow_job_id: str | None, + duration_seconds: float, + has_audio: bool, + source_audio_codec: str, + progress_start: int, + progress_end: int, +) -> tuple[str, str]: + copy_safe_codecs = {"aac", "mp3", "ac3", "eac3", "alac"} + preferred_audio = "copy" if not has_audio or source_audio_codec in copy_safe_codecs else "aac" + attempts: list[tuple[str, str]] = [] + if _ffmpeg_has_encoder("h264_nvenc"): + attempts.append(("h264_nvenc", preferred_audio)) + attempts.append(("libx264", preferred_audio)) + if preferred_audio == "copy" and has_audio: + attempts.append(("libx264", "aac")) + errors = [] + for encoder, audio_mode in attempts: + temporary_path.unlink(missing_ok=True) + command = _build_ffmpeg_render_command( + input_path, + subtitle_path, + temporary_path, + encoder=encoder, + audio_mode=audio_mode, + ) + try: + _run_ffmpeg_progress( + command, + workflow_job_id=workflow_job_id, + duration_seconds=duration_seconds, + progress_start=progress_start, + progress_end=progress_end, + ) + return encoder, audio_mode + except (SubtitleRenderCancelled, job_service.JobLeaseLostError): + raise + except RuntimeError as exc: + errors.append(f"{encoder}/{audio_mode}:{exc}") + raise RuntimeError(";".join(errors) or "FFmpeg 字幕烧录失败") + + +def _build_ffmpeg_render_command( + input_path: Path, + subtitle_path: Path, + output_path: Path, + *, + encoder: str, + audio_mode: str, +) -> list[str]: + video_args = ( + ["-c:v", "h264_nvenc", "-preset", "p4", "-cq", "23"] + if encoder == "h264_nvenc" + else ["-c:v", "libx264", "-preset", "medium", "-crf", "20"] + ) + audio_args = ["-c:a", "copy"] if audio_mode == "copy" else ["-c:a", "aac", "-b:a", "192k"] + return [ + "ffmpeg", "-y", "-hide_banner", "-loglevel", "error", + "-i", str(input_path), + "-map", "0:v:0", "-map", "0:a:0?", + "-vf", _ffmpeg_subtitles_filter(subtitle_path), + *video_args, "-pix_fmt", "yuv420p", *audio_args, + "-movflags", "+faststart", "-progress", "pipe:1", "-nostats", + str(output_path), + ] + + +def _run_ffmpeg_progress( + command: list[str], + *, + workflow_job_id: str | None, + duration_seconds: float, + progress_start: int, + progress_end: int, +) -> None: + from app.services import job_service + + process = popen_process_group( + command, + stdout=subprocess.PIPE, + stderr=subprocess.STDOUT, + text=True, + encoding="utf-8", + errors="replace", + bufsize=1, + ) + output_tail: list[str] = [] + assert process.stdout is not None + output_queue: queue.Queue[str] = queue.Queue() + + def read_output() -> None: + assert process.stdout is not None + for raw_line in process.stdout: + output_queue.put(raw_line) + + output_thread = threading.Thread(target=read_output, daemon=True) + output_thread.start() + started_at = time.monotonic() + last_progress_at = started_at + absolute_timeout = max( + float(settings.ffmpeg_subtitle_timeout), + float(duration_seconds) * 4 + 300, + ) + try: + while process.poll() is None or not output_queue.empty(): + try: + raw_line = output_queue.get(timeout=0.5) + except queue.Empty: + raw_line = "" + line = raw_line.strip() + if line: + last_progress_at = time.monotonic() + output_tail.append(line) + output_tail = output_tail[-40:] + if workflow_job_id and job_service.is_cancel_requested(workflow_job_id): + terminate_process_tree(process) + raise SubtitleRenderCancelled("用户已取消字幕烧录") + if workflow_job_id and line.startswith("out_time_ms=") and duration_seconds > 0: + try: + processed_seconds = int(line.split("=", 1)[1]) / 1_000_000 + except ValueError: + continue + ratio = max(0.0, min(1.0, processed_seconds / duration_seconds)) + progress = round(progress_start + (progress_end - progress_start) * ratio) + job_service.update_job_progress(workflow_job_id, progress, "正在烧录并验证字幕成片") + elapsed = time.monotonic() - started_at + stalled = time.monotonic() - last_progress_at + if stalled > settings.ffmpeg_subtitle_timeout: + terminate_process_tree(process) + raise RuntimeError( + f"FFmpeg 字幕烧录连续 {settings.ffmpeg_subtitle_timeout} 秒没有进展,已终止进程树" + ) + if elapsed > absolute_timeout: + terminate_process_tree(process) + raise RuntimeError(f"FFmpeg 字幕烧录超过 {round(absolute_timeout)} 秒,已终止进程树") + output_thread.join(timeout=2) + return_code = process.wait(timeout=10) + finally: + if process.poll() is None: + terminate_process_tree(process) + if return_code != 0: + raise RuntimeError("\n".join(output_tail[-8:]) or f"FFmpeg 退出码 {return_code}") + + +def _validate_rendered_media( + path: Path, + *, + source_duration: float, + source_has_audio: bool, +) -> dict[str, Any]: + if not path.exists() or path.stat().st_size <= 0: + raise RuntimeError("字幕临时输出为空") + probe = _probe_media(path) + if probe["video_codec"] != "h264": + raise RuntimeError(f"字幕成片视频编码不是 H.264:{probe['video_codec'] or '未知'}") + if probe["pixel_format"] != "yuv420p": + raise RuntimeError(f"字幕成片像素格式不是 yuv420p:{probe['pixel_format'] or '未知'}") + if source_has_audio and not probe["has_audio"]: + raise RuntimeError("原切片包含音频,但字幕成片缺少音轨") + tolerance = max(1.5, source_duration * 0.03) + if source_duration > 0 and abs(probe["duration"] - source_duration) > tolerance: + raise RuntimeError("字幕成片时长与原切片不一致") + return { + "video_codec": probe["video_codec"], + "pixel_format": probe["pixel_format"], + "has_audio": probe["has_audio"], + "audio_codec": probe["audio_codec"], + "duration_seconds": probe["duration"], + "size_bytes": path.stat().st_size, + } diff --git a/app/services/task_lifecycle_service.py b/app/services/task_lifecycle_service.py index ca1f9de..d206da5 100644 --- a/app/services/task_lifecycle_service.py +++ b/app/services/task_lifecycle_service.py @@ -4,24 +4,120 @@ """ import json +from sqlite3 import Connection from uuid import uuid4 from app.db.database import get_connection from app.models.task import TaskCreate, TaskStatus from app.services.storage_service import ( - apply_task_media_cleanup_plan, allocate_task_dir_name, build_task_media_cleanup_plan, create_task_directory, + finalize_staged_task_media_cleanup, + rollback_staged_task_media_cleanup, + stage_task_media_cleanup_plan, validate_source_video_path, ) from app.services.task_log_service import append_task_log +from app.services.media_preflight_service import preflight_media class TaskDeletionConflictError(RuntimeError): """任务仍在执行,暂时不能删除其媒体文件。""" +class TaskStatusConflictError(RuntimeError): + """任务状态已变化、已删除,或请求的状态转换不被允许。""" + + +PUBLIC_TASK_STATUS_TRANSITIONS: dict[str, set[str]] = { + TaskStatus.pending_video.value: {TaskStatus.pending_processing.value}, + TaskStatus.pending_processing.value: {TaskStatus.audio_extracting.value, TaskStatus.failed.value}, + TaskStatus.audio_extracting.value: {TaskStatus.transcribing.value, TaskStatus.failed.value}, + TaskStatus.transcribing.value: {TaskStatus.pending_ai.value, TaskStatus.failed.value}, + TaskStatus.pending_ai.value: {TaskStatus.ai_analyzing.value, TaskStatus.failed.value}, + TaskStatus.ai_analyzing.value: {TaskStatus.pending_review.value, TaskStatus.failed.value}, + TaskStatus.pending_review.value: { + TaskStatus.ai_analyzing.value, + TaskStatus.cutting.value, + TaskStatus.completed.value, + TaskStatus.completed_with_errors.value, + }, + TaskStatus.cutting.value: { + TaskStatus.completed.value, + TaskStatus.completed_with_errors.value, + TaskStatus.failed.value, + }, + TaskStatus.completed.value: {TaskStatus.cutting.value}, + TaskStatus.completed_with_errors.value: {TaskStatus.cutting.value}, + TaskStatus.failed.value: { + TaskStatus.pending_processing.value, + TaskStatus.audio_extracting.value, + TaskStatus.transcribing.value, + TaskStatus.pending_ai.value, + TaskStatus.ai_analyzing.value, + TaskStatus.cutting.value, + }, + TaskStatus.CREATED.value: {TaskStatus.PREPARING_SOURCE.value}, + TaskStatus.PREPARING_SOURCE.value: { + TaskStatus.TRANSCRIBING.value, + TaskStatus.FAILED_PREPARING_SOURCE.value, + TaskStatus.CANCELLED.value, + }, + TaskStatus.TRANSCRIBING.value: { + TaskStatus.AI_ANALYZING.value, + TaskStatus.FAILED_TRANSCRIBING.value, + TaskStatus.CANCELLED.value, + }, + TaskStatus.AI_ANALYZING.value: { + TaskStatus.CLIP_SELECTING.value, + TaskStatus.FAILED_AI_ANALYZING.value, + TaskStatus.CANCELLED.value, + }, + TaskStatus.CLIP_SELECTING.value: { + TaskStatus.VIDEO_CUTTING.value, + TaskStatus.FAILED_CLIP_SELECTING.value, + TaskStatus.CANCELLED.value, + }, + TaskStatus.VIDEO_CUTTING.value: { + TaskStatus.SUBTITLE_DRAFTING.value, + TaskStatus.FAILED_VIDEO_CUTTING.value, + TaskStatus.CANCELLED.value, + }, + TaskStatus.SUBTITLE_DRAFTING.value: { + TaskStatus.PENDING_SUBTITLE_REVIEW.value, + TaskStatus.FAILED_SUBTITLE_DRAFTING.value, + TaskStatus.CANCELLED.value, + }, + TaskStatus.PENDING_SUBTITLE_REVIEW.value: {TaskStatus.METADATA_GENERATING.value}, + TaskStatus.METADATA_GENERATING.value: { + TaskStatus.SCHEDULE_CREATING.value, + TaskStatus.FAILED_METADATA_GENERATING.value, + TaskStatus.CANCELLED.value, + }, + TaskStatus.SCHEDULE_CREATING.value: { + TaskStatus.PUBLISH_JOB_CREATING.value, + TaskStatus.FAILED_SCHEDULE_CREATING.value, + TaskStatus.CANCELLED.value, + }, + TaskStatus.PUBLISH_JOB_CREATING.value: { + TaskStatus.READY_TO_PUBLISH.value, + TaskStatus.FAILED_PUBLISH_JOB_CREATING.value, + TaskStatus.CANCELLED.value, + }, + TaskStatus.FAILED_PREPARING_SOURCE.value: {TaskStatus.PREPARING_SOURCE.value}, + TaskStatus.FAILED_TRANSCRIBING.value: {TaskStatus.TRANSCRIBING.value}, + TaskStatus.FAILED_AI_ANALYZING.value: {TaskStatus.AI_ANALYZING.value}, + TaskStatus.FAILED_CLIP_SELECTING.value: {TaskStatus.CLIP_SELECTING.value}, + TaskStatus.FAILED_VIDEO_CUTTING.value: {TaskStatus.VIDEO_CUTTING.value}, + TaskStatus.FAILED_SUBTITLE_DRAFTING.value: {TaskStatus.SUBTITLE_DRAFTING.value}, + TaskStatus.FAILED_METADATA_GENERATING.value: {TaskStatus.METADATA_GENERATING.value}, + TaskStatus.FAILED_SCHEDULE_CREATING.value: {TaskStatus.SCHEDULE_CREATING.value}, + TaskStatus.FAILED_PUBLISH_JOB_CREATING.value: {TaskStatus.PUBLISH_JOB_CREATING.value}, + TaskStatus.CANCELLED.value: {TaskStatus.PREPARING_SOURCE.value}, +} + + ACTIVE_TASK_STATUSES = { TaskStatus.CREATED.value, TaskStatus.PREPARING_SOURCE.value, @@ -29,6 +125,7 @@ class TaskDeletionConflictError(RuntimeError): TaskStatus.AI_ANALYZING.value, TaskStatus.CLIP_SELECTING.value, TaskStatus.VIDEO_CUTTING.value, + TaskStatus.SUBTITLE_DRAFTING.value, TaskStatus.METADATA_GENERATING.value, TaskStatus.SCHEDULE_CREATING.value, TaskStatus.PUBLISH_JOB_CREATING.value, @@ -43,19 +140,28 @@ def create_task_record(payload: TaskCreate, task_id: str | None = None, task_dir from app.services.task_service import _now_iso, get_status_label, STATUS_PROGRESS # noqa: F811 resolved_task_id = task_id or uuid4().hex[:12] - resolved_task_dir_name = task_dir_name or allocate_task_dir_name( - payload.task_name, - exclude_task_id=resolved_task_id, - ) + resolved_task_dir_name = task_dir_name now = _now_iso() - create_task_directory(resolved_task_id, resolved_task_dir_name) - - source_path = payload.nas_file_path if payload.source_type == "nas" else payload.original_video_path + source_path = payload.original_video_path has_source_file = bool(source_path) + media_preflight = None if source_path: valid, error_message = validate_source_video_path(source_path) if not valid: raise ValueError(error_message) + output_limit = ( + payload.highlight_total_limit + if payload.selection_profile == "long_live_talk" + else payload.candidate_clip_count + ) + media_preflight = preflight_media(source_path, total_output_limit=output_limit) + + if not resolved_task_dir_name: + resolved_task_dir_name = allocate_task_dir_name( + payload.task_name, + exclude_task_id=resolved_task_id, + ) + create_task_directory(resolved_task_id, resolved_task_dir_name) initial_status = TaskStatus.CREATED.value if payload.auto_mode else ( TaskStatus.pending_processing.value if has_source_file else TaskStatus.pending_video.value @@ -81,14 +187,16 @@ def create_task_record(payload: TaskCreate, task_id: str | None = None, task_dir "id": resolved_task_id, "task_name": payload.task_name, "task_dir_name": resolved_task_dir_name, - "source_type": payload.source_type, + "source_type": "upload", "platform": payload.platform, "original_video_path": payload.original_video_path, - "nas_file_path": payload.nas_file_path, + "nas_file_path": None, "max_clip_duration": payload.max_clip_duration, "candidate_clip_count": payload.candidate_clip_count, "selection_profile": payload.selection_profile, "final_clip_target": payload.final_clip_target, + "highlight_density_per_hour": payload.highlight_density_per_hour, + "highlight_total_limit": payload.highlight_total_limit, "ai_preference": payload.ai_preference, "ai_prompt_preset_id": "preset_001", "auto_mode": 1 if payload.auto_mode else 0, @@ -106,7 +214,7 @@ def create_task_record(payload: TaskCreate, task_id: str | None = None, task_dir if "title" in existing_columns: insert_data["title"] = payload.task_name if "source_path" in existing_columns: - insert_data["source_path"] = payload.nas_file_path or payload.original_video_path + insert_data["source_path"] = payload.original_video_path if "max_clip_minutes" in existing_columns: insert_data["max_clip_minutes"] = payload.max_clip_duration if "target_clip_count" in existing_columns: @@ -123,7 +231,7 @@ def create_task_record(payload: TaskCreate, task_id: str | None = None, task_dir append_task_log(resolved_task_id, "任务已创建") if payload.auto_mode: append_task_log(resolved_task_id, "已开启全自动模式,等待流水线启动") - return { + result = { "id": resolved_task_id, "task_name": payload.task_name, "task_dir_name": resolved_task_dir_name, @@ -133,6 +241,9 @@ def create_task_record(payload: TaskCreate, task_id: str | None = None, task_dir "detail_url": f"/tasks/{resolved_task_id}", "message": "任务已创建并写入数据库。", } + if media_preflight: + result["media_preflight"] = media_preflight.to_dict() + return result def update_task_status( @@ -141,25 +252,158 @@ def update_task_status( error_message: str | None = None, ) -> dict | None: from app.services.task_service import _now_iso, get_task, STATUS_PROGRESS # noqa: F811 + from app.services import job_service - now = _now_iso() status_value = new_status.value progress = STATUS_PROGRESS.get(status_value, 0) + active_lease = job_service.current_job_lease() + + with get_connection() as connection: + connection.execute("BEGIN IMMEDIATE") + now = _now_iso() + current = connection.execute( + "SELECT COALESCE(is_deleted, 0) AS is_deleted FROM tasks WHERE id = ?", + (task_id,), + ).fetchone() + if not current: + connection.rollback() + return None + if int(current["is_deleted"] or 0): + connection.rollback() + raise TaskStatusConflictError("任务已永久删除,不能再更新处理状态") + + lease_condition = "" + lease_params: tuple[str, ...] = () + if active_lease: + active_job_id, lease_owner, lease_token = active_lease + cancel_condition = "" if new_status == TaskStatus.CANCELLED else "AND cancel_requested = 0" + lease_condition = f""" + AND EXISTS ( + SELECT 1 FROM workflow_jobs + WHERE id = ? AND task_id = ? AND status = ? + AND lease_owner = ? AND lease_token = ? + AND lease_expires_at > ? {cancel_condition} + ) + """ + lease_params = ( + active_job_id, + task_id, + job_service.JOB_STATUS_RUNNING, + lease_owner, + lease_token, + now, + ) + cursor = connection.execute( + f""" + UPDATE tasks + SET status = ?, progress = ?, error_message = ?, last_error = ?, updated_at = ? + WHERE id = ? AND COALESCE(is_deleted, 0) = 0 {lease_condition} + """, + (status_value, progress, error_message, error_message, now, task_id, *lease_params), + ) + if cursor.rowcount == 0: + connection.rollback() + if active_lease: + raise job_service.JobLeaseLostError( + f"Workflow Job 租约已失效,拒绝覆盖 Task 状态:{active_lease[0]}" + ) + raise TaskStatusConflictError("任务状态更新失败,请刷新后重试") + connection.commit() + + return get_task(task_id, include_video_probe=False) + +def transition_task_status( + task_id: str, + new_status: TaskStatus, + error_message: str | None = None, +) -> dict | None: + """执行对外可见的受控状态转换,并避免覆盖并发产生的新状态。""" + from app.services.task_service import _now_iso, get_task, STATUS_PROGRESS # noqa: F811 + + status_value = new_status.value + progress = STATUS_PROGRESS.get(status_value, 0) with get_connection() as connection: + connection.execute("BEGIN IMMEDIATE") + now = _now_iso() + current = connection.execute( + "SELECT status, COALESCE(is_deleted, 0) AS is_deleted FROM tasks WHERE id = ?", + (task_id,), + ).fetchone() + if not current: + connection.rollback() + return None + if int(current["is_deleted"] or 0): + connection.rollback() + raise TaskStatusConflictError("任务已永久删除,不能再更新处理状态") + + current_status = str(current["status"] or "") + allowed = PUBLIC_TASK_STATUS_TRANSITIONS.get(current_status, set()) + if status_value != current_status and status_value not in allowed: + connection.rollback() + raise TaskStatusConflictError( + f"不允许从 {current_status or '未知状态'} 跳转到 {status_value}" + ) + _validate_public_transition_preconditions(connection, task_id, status_value) + if status_value == TaskStatus.CANCELLED.value: + from app.services import job_service + + job_service.cancel_active_auto_pipeline_jobs_for_task( + connection, + task_id, + now=now, + ) + error_message = error_message or "用户已取消全自动流水线" + cursor = connection.execute( """ UPDATE tasks SET status = ?, progress = ?, error_message = ?, last_error = ?, updated_at = ? - WHERE id = ? + WHERE id = ? AND status = ? AND COALESCE(is_deleted, 0) = 0 """, - (status_value, progress, error_message, error_message, now, task_id), + (status_value, progress, error_message, error_message, now, task_id, current_status), ) + if cursor.rowcount != 1: + connection.rollback() + raise TaskStatusConflictError("任务状态已被其他流程更新,请刷新后重试") connection.commit() + return get_task(task_id, include_video_probe=False) - if cursor.rowcount == 0: - return None - return get_task(task_id) + +def _validate_public_transition_preconditions( + connection: Connection, + task_id: str, + status_value: str, +) -> None: + if status_value == TaskStatus.pending_processing.value: + source = connection.execute( + "SELECT original_video_path FROM tasks WHERE id = ?", + (task_id,), + ).fetchone() + if not source or not str(source["original_video_path"] or "").strip(): + raise TaskStatusConflictError("任务尚未绑定源视频,不能进入待处理状态") + elif status_value == TaskStatus.cutting.value: + enabled = connection.execute( + """ + SELECT 1 FROM clip_candidates + WHERE task_id = ? AND enabled = 1 AND COALESCE(is_deleted, 0) = 0 + LIMIT 1 + """, + (task_id,), + ).fetchone() + if not enabled: + raise TaskStatusConflictError("任务没有已启用的候选片段,不能进入切割状态") + elif status_value in {TaskStatus.completed.value, TaskStatus.completed_with_errors.value}: + output = connection.execute( + """ + SELECT 1 FROM output_clip + WHERE task_id = ? AND status = 'completed' AND COALESCE(is_active, 1) = 1 + LIMIT 1 + """, + (task_id,), + ).fetchone() + if not output: + raise TaskStatusConflictError("任务没有成功的活跃切片,不能标记完成") def update_task_ai_preference(task_id: str, ai_preference: str | None) -> dict: @@ -223,30 +467,51 @@ def update_task_selection_settings( task_id: str, selection_profile: str, final_clip_target: int, + highlight_density_per_hour: int = 4, + highlight_total_limit: int = 30, ) -> dict: from app.services.task_service import _now_iso, get_task # noqa: F811 task = get_task(task_id, include_video_probe=False) if not task: raise ValueError("任务不存在") - if selection_profile not in {"general", "variety_comedy"}: - raise ValueError("选片模式只能是康熙笑点选片模式或通用模式(历史任务)") + if selection_profile not in {"general", "variety_comedy", "long_live_talk"}: + raise ValueError("选片模式只能是通用内容价值、康熙笑点选片模式或长直播高光") if final_clip_target < 1 or final_clip_target > 12: raise ValueError("最终启用目标必须在 1 到 12 条之间") + if selection_profile != "long_live_talk": + highlight_density_per_hour = 4 + highlight_total_limit = 30 + if not 1 <= highlight_density_per_hour <= 10: + raise ValueError("每小时高光密度必须在 1 到 10 条之间") + if not 1 <= highlight_total_limit <= 50: + raise ValueError("高光总上限必须在 1 到 50 条之间") now = _now_iso() with get_connection() as connection: connection.execute( """ UPDATE tasks - SET selection_profile = ?, final_clip_target = ?, updated_at = ? + SET selection_profile = ?, final_clip_target = ?, + highlight_density_per_hour = ?, highlight_total_limit = ?, updated_at = ? WHERE id = ? """, - (selection_profile, final_clip_target, now, task_id), + ( + selection_profile, + final_clip_target, + highlight_density_per_hour, + highlight_total_limit, + now, + task_id, + ), ) connection.commit() - profile_label = "康熙笑点选片模式" if selection_profile == "variety_comedy" else "通用模式(历史任务)" + profile_label = { + "general": "通用内容价值", + "variety_comedy": "康熙笑点选片模式", + "long_live_talk": "长直播高光(语言类)", + }[selection_profile] append_task_log(task_id, f"已更新选片模式:{profile_label},最终启用目标:{final_clip_target} 条") return { "status": "ok", @@ -258,12 +523,13 @@ def update_task_selection_settings( def delete_task_permanently(task_id: str) -> dict: from app.services.task_service import _now_iso, get_task # noqa: F811 - task = get_task(task_id, include_video_probe=False) + task = get_task(task_id, include_video_probe=False, include_deleted=True) if not task: raise ValueError("任务不存在") cleanup_plan = build_task_media_cleanup_plan(task) existing_target_count = len(cleanup_plan.existing_targets) now = _now_iso() + staged_cleanup = None with get_connection() as connection: try: connection.execute("BEGIN IMMEDIATE") @@ -306,7 +572,7 @@ def delete_task_permanently(task_id: str) -> dict: if publishing_job: raise TaskDeletionConflictError("任务正在向平台发送视频,请等待发送结束后再删除。") - cleanup_result = apply_task_media_cleanup_plan(cleanup_plan) + staged_cleanup = stage_task_media_cleanup_plan(cleanup_plan) connection.execute( """ UPDATE workflow_jobs @@ -338,13 +604,32 @@ def delete_task_permanently(task_id: str) -> dict: (now, now, task_id), ) connection.commit() - except Exception: - connection.rollback() + except Exception as exc: + try: + connection.rollback() + finally: + if staged_cleanup is not None: + try: + rollback_staged_task_media_cleanup(staged_cleanup) + except Exception as rollback_exc: + raise RuntimeError( + "数据库删除状态提交失败,且媒体自动恢复未完成;" + f"请保留隔离清单并人工恢复。原错误:{exc};恢复错误:{rollback_exc}" + ) from exc raise - status = "already_deleted" if task.get("is_deleted") and existing_target_count == 0 else "deleted" + cleanup_result = finalize_staged_task_media_cleanup(staged_cleanup) + if cleanup_result.cleanup_pending: + status = "cleanup_pending" + else: + status = "already_deleted" if task.get("is_deleted") and existing_target_count == 0 else "deleted" freed_mb = cleanup_result.freed_bytes / (1024 * 1024) - if status == "already_deleted": + if status == "cleanup_pending": + message = ( + "任务已从系统中永久隐藏,但隔离区文件暂时无法清除;" + "清单已保留,可安全重试清理。" + ) + elif status == "already_deleted": message = "任务已经永久删除,当前没有残留的任务视频文件。" else: message = f"任务已永久删除,共释放约 {freed_mb:.1f} MB;数据库历史记录已隐藏保留。" @@ -354,6 +639,8 @@ def delete_task_permanently(task_id: str) -> dict: "freed_bytes": cleanup_result.freed_bytes, "external_source_preserved": cleanup_result.external_source_preserved, "deleted_paths": list(cleanup_result.deleted_paths), + "cleanup_pending": cleanup_result.cleanup_pending, + "staged_paths": list(cleanup_result.staged_paths), "message": message, } diff --git a/app/services/task_query_service.py b/app/services/task_query_service.py index 9cb420d..6583b5f 100644 --- a/app/services/task_query_service.py +++ b/app/services/task_query_service.py @@ -4,18 +4,19 @@ 不参与任务生命周期、状态变更等核心逻辑。 """ -from datetime import datetime +from datetime import datetime, timedelta import shutil from app.core.config import settings from app.db.database import get_connection from app.models.task import TaskStatus from app.services.ai_config_service import get_ai_config_context +from app.services.publish_domain import TERMINAL_PUBLISH_STATUSES +from app.services.publish_time import app_zone, parse_datetime from app.services.storage_service import resolve_video_file_path from app.services.subtitle_workflow_service import SUBTITLE_STATUS_LABELS from app.services.task_service import ( OUTPUT_STATUS_LABELS, - WORKFLOW_STEPS, _parse_time_to_seconds, get_default_subtitle_style, get_task, @@ -60,6 +61,81 @@ def _batch_completed_output_clip_counts(task_ids: list[str]) -> dict[str, int]: return {row["task_id"]: int(row["cnt"] or 0) for row in rows} +def _dashboard_pending_publish_counts(task_ids: list[str]) -> dict[str, int]: + """统计仍有有效切片未进入发布终态的任务和切片。""" + if not task_ids: + return {"task_count": 0, "clip_count": 0} + + task_placeholders = ",".join("?" for _ in task_ids) + terminal_statuses = sorted(TERMINAL_PUBLISH_STATUSES) + terminal_placeholders = ",".join("?" for _ in terminal_statuses) + with get_connection() as connection: + row = connection.execute( + f""" + WITH latest_publish_job AS ( + SELECT + output_clip_id, + UPPER(status) AS status, + ROW_NUMBER() OVER ( + PARTITION BY output_clip_id + ORDER BY created_at DESC, updated_at DESC, id DESC + ) AS row_number + FROM publish_jobs + ) + SELECT + COUNT(DISTINCT output_clip.task_id) AS task_count, + COUNT(*) AS clip_count + FROM output_clip + LEFT JOIN latest_publish_job + ON latest_publish_job.output_clip_id = output_clip.id + AND latest_publish_job.row_number = 1 + WHERE output_clip.task_id IN ({task_placeholders}) + AND output_clip.status = 'completed' + AND output_clip.is_active = 1 + AND ( + latest_publish_job.status IS NULL + OR latest_publish_job.status NOT IN ({terminal_placeholders}) + ) + """, + [*task_ids, *terminal_statuses], + ).fetchone() + return { + "task_count": int(row["task_count"] or 0) if row else 0, + "clip_count": int(row["clip_count"] or 0) if row else 0, + } + + +def _dashboard_weekly_summary(tasks: list[dict], *, now: datetime | None = None) -> dict: + """按应用时区生成本周新增任务总数和日期范围。""" + zone = app_zone(settings.app_timezone) + current = now or datetime.now(zone) + current = current.replace(tzinfo=zone) if current.tzinfo is None else current.astimezone(zone) + week_start = (current - timedelta(days=current.weekday())).replace( + hour=0, + minute=0, + second=0, + microsecond=0, + ) + week_end = week_start + timedelta(days=7) + weekly_total = 0 + + for task in tasks: + raw_created_at = task.get("created_at_raw") + if not raw_created_at: + continue + try: + created_at = parse_datetime(raw_created_at, settings.app_timezone).astimezone(zone) + except ValueError: + continue + if week_start <= created_at < week_end: + weekly_total += 1 + + return { + "total": weekly_total, + "range_label": f"{week_start:%m.%d} - {(week_end - timedelta(days=1)):%m.%d}", + } + + def _batch_clip_candidate_counts(task_ids: list[str]) -> dict[str, dict[str, int]]: """一次查询获得多个任务的候选片段总数和已启用数量。""" if not task_ids: @@ -106,13 +182,20 @@ def _batch_all_output_clips(task_ids: list[str]) -> dict[str, list[dict]]: clip_candidates.enabled AS clip_enabled, subtitle_jobs.id AS subtitle_job_id, subtitle_jobs.status AS subtitle_status, + subtitle_jobs.revision_id AS subtitle_revision_id, subtitle_jobs.subtitle_file_path, subtitle_jobs.output_file_path AS subtitled_output_file_path, subtitle_jobs.error_message AS subtitle_error_message, + subtitle_jobs.validation_status AS subtitle_validation_status, + subtitle_jobs.validation_json AS subtitle_validation_json, + subtitle_jobs.encoder AS subtitle_encoder, + subtitle_jobs.verified_at AS subtitle_verified_at, + subtitle_revisions.status AS subtitle_revision_status, subtitle_jobs.updated_at AS subtitle_updated_at FROM output_clip LEFT JOIN clip_candidates ON clip_candidates.id = output_clip.clip_candidate_id - LEFT JOIN subtitle_jobs ON subtitle_jobs.output_clip_id = output_clip.id + LEFT JOIN subtitle_jobs ON subtitle_jobs.output_clip_id = output_clip.id AND subtitle_jobs.is_active = 1 + LEFT JOIN subtitle_revisions ON subtitle_revisions.id = subtitle_jobs.revision_id WHERE output_clip.task_id IN ({placeholders}) AND output_clip.is_active = 1 ORDER BY CASE WHEN output_clip.output_file_name IS NULL OR output_clip.output_file_name = '' THEN 1 ELSE 0 END, @@ -148,6 +231,14 @@ def _batch_all_output_clips(task_ids: list[str]) -> dict[str, list[dict]]: "clip_end_seconds": clip_end_seconds, "subtitle_status": subtitle_status, "subtitle_status_label": SUBTITLE_STATUS_LABELS.get(subtitle_status, subtitle_status), + "subtitle_publish_ready": bool( + subtitle_status == "completed" + and output.get("subtitle_validation_status") == "verified" + and output.get("subtitle_revision_status") == "approved" + and subtitled_path + and subtitled_path.exists() + and subtitled_path.is_file() + ), "subtitle_stage": SUBTITLE_STATUS_LABELS.get(subtitle_status, subtitle_status), "subtitled_file_exists": bool(subtitled_path and subtitled_path.exists() and subtitled_path.is_file()), "subtitled_media_url": f"/media/tasks/{output['task_id']}/subtitled-clips/{output['id']}", @@ -157,53 +248,44 @@ def _batch_all_output_clips(task_ids: list[str]) -> dict[str, list[dict]]: return result -def get_dashboard_context() -> dict: +def get_dashboard_context(*, now: datetime | None = None) -> dict: """Dashboard 首页统计上下文""" tasks = list_tasks() - today = datetime.now().date() - today_count = 0 - for task in tasks: - raw_created_at = task.get("created_at_raw") - if not raw_created_at: - continue - try: - if datetime.fromisoformat(raw_created_at).astimezone().date() == today: - today_count += 1 - except ValueError: - continue - - pending_count = sum( + task_ids = [task["id"] for task in tasks] + weekly_summary = _dashboard_weekly_summary(tasks, now=now) + completed_oc_map = _batch_completed_output_clip_counts(task_ids) + completed_task_count = sum(1 for task in tasks if completed_oc_map.get(task["id"], 0) > 0) + completed_clip_count = sum(completed_oc_map.values()) + pending_publish = _dashboard_pending_publish_counts(task_ids) + failed_count = sum( 1 for task in tasks - if task["status"] in {TaskStatus.pending_video.value, TaskStatus.pending_processing.value} + if task["status"] == TaskStatus.failed.value or str(task["status"]).startswith("FAILED_") ) - review_count = sum(1 for task in tasks if task["status"] == TaskStatus.pending_review.value) - completed_count = sum( - 1 - for task in tasks - if task["status"] in {TaskStatus.completed.value, TaskStatus.completed_with_errors.value} - ) - output_clip_count = sum(int(task.get("output_clip_count") or 0) for task in tasks) - completed_oc_map = _batch_completed_output_clip_counts([task["id"] for task in tasks]) - ready_for_subtitle_count = sum(completed_oc_map.get(task["id"], 0) for task in tasks) - failed_count = sum(1 for task in tasks if task["status"] == TaskStatus.failed.value) return { "stats": [ - {"label": "今日新增任务", "value": today_count, "note": "来自 SQLite", "tone": "blue"}, - {"label": "待处理", "value": pending_count, "note": "可继续推进", "tone": "amber"}, - {"label": "待检查", "value": review_count, "note": "AI 结果可生成切片", "tone": "purple"}, - {"label": "已切片任务", "value": completed_count, "note": f"输出 {output_clip_count} 条切片", "tone": "green"}, - {"label": "待加字幕", "value": ready_for_subtitle_count, "note": "切片后工作流", "tone": "blue"}, - {"label": "待推送", "value": ready_for_subtitle_count, "note": "需字幕和发布确认", "tone": "red"}, + { + "label": "本周新增任务", + "value": weekly_summary["total"], + "note": f"{weekly_summary['range_label']} · 上海时间", + "tone": "blue", + }, + { + "label": "已切片任务", + "value": completed_task_count, + "note": f"共 {completed_clip_count} 条有效切片", + "tone": "green", + }, + { + "label": "待推送任务", + "value": pending_publish["task_count"], + "note": f"涉及 {pending_publish['clip_count']} 条待发送或复核切片", + "tone": "amber", + }, {"label": "失败任务", "value": failed_count, "note": "需排查", "tone": "red"}, ], - "focus_stats": [ - {"label": "输出切片", "value": output_clip_count, "description": "条短视频已生成记录"}, - {"label": "待加字幕", "value": ready_for_subtitle_count, "description": "条切片可进入字幕工作台"}, - {"label": "待推送", "value": ready_for_subtitle_count, "description": "条切片等待发布前确认"}, - ], - "workflow_steps": WORKFLOW_STEPS, + "weekly_summary": weekly_summary, "recent_tasks": tasks[:5], } @@ -219,19 +301,20 @@ def get_clips_overview_context() -> dict: enabled_count = counts["enabled"] review_ready = clip_count > 0 can_cut = enabled_count > 0 and task["source_exists"] - if task["status"] == TaskStatus.failed.value: + normalized_status = str(task.get("status") or "").lower() + if normalized_status == TaskStatus.failed.value: review_stage = "异常" review_tone = "red" - elif task["status"] == TaskStatus.completed.value: + elif normalized_status == TaskStatus.completed.value: review_stage = "已完成" review_tone = "green" - elif task["status"] == TaskStatus.completed_with_errors.value: + elif normalized_status == TaskStatus.completed_with_errors.value: review_stage = "部分完成" review_tone = "amber" - elif task["status"] == TaskStatus.pending_review.value or review_ready: + elif normalized_status == TaskStatus.pending_review.value or review_ready: review_stage = "待检查" review_tone = "purple" - elif task["status"] in {TaskStatus.pending_ai.value, TaskStatus.ai_analyzing.value}: + elif normalized_status in {TaskStatus.pending_ai.value, TaskStatus.ai_analyzing.value}: review_stage = "待 AI" review_tone = "blue" else: @@ -250,42 +333,39 @@ def get_clips_overview_context() -> dict: } ) + completed_statuses = { + TaskStatus.completed.value, + TaskStatus.completed_with_errors.value, + } + reviewed_task_count = sum(1 for task in enriched_tasks if task["real_clip_count"] > 0) + passed_clip_count = sum(task["enabled_clip_count"] for task in enriched_tasks) + completed_task_count = sum( + 1 + for task in tasks + if str(task.get("status") or "").lower() in completed_statuses + ) + return { "tasks": enriched_tasks, "stats": [ { - "label": "待 AI 分析", - "value": sum( - 1 - for task in tasks - if task["status"] in {TaskStatus.pending_ai.value, TaskStatus.ai_analyzing.value} - ), + "label": "累计审核任务", + "value": reviewed_task_count, + "note": "已进入候选片段审核流程", "tone": "blue", }, { - "label": "待检查", - "value": sum(1 for task in enriched_tasks if task["review_stage"] == "待检查"), - "tone": "purple", - }, - { - "label": "可生成切片", - "value": sum(1 for task in enriched_tasks if task["can_cut"]), + "label": "已通过视频", + "value": passed_clip_count, + "note": "当前启用的视频片段", "tone": "green", }, { - "label": "已完成", - "value": sum( - 1 - for task in tasks - if task["status"] in {TaskStatus.completed.value, TaskStatus.completed_with_errors.value} - ), + "label": "已完成任务", + "value": completed_task_count, + "note": "含手动与全自动完成状态", "tone": "green", }, - { - "label": "异常任务", - "value": sum(1 for task in tasks if task["status"] == TaskStatus.failed.value), - "tone": "red", - }, ], } diff --git a/app/services/task_service.py b/app/services/task_service.py index 0678e39..8805e82 100644 --- a/app/services/task_service.py +++ b/app/services/task_service.py @@ -13,6 +13,7 @@ from app.models.task import ClipCandidateBatchItem, ClipCandidateUpdate, TaskStatus from app.services.ai_analysis_workflow_service import ( AI_CLIP_MIN_RECOMMENDED_SECONDS, + AIAnalysisConflictError, _ai_model_name, _ai_provider_label, _analysis_payload_to_preview, @@ -30,12 +31,16 @@ _summarize_ai_error, _summarize_analysis_clips, _write_analysis_payload, + ensure_task_ai_analysis_artifact, get_ai_analysis_run, get_latest_ai_analysis_run, + get_task_ai_analysis_meta, + validate_ai_analysis_meta_for_cut, get_task_ai_analysis_status, get_task_ai_source_label, list_ai_analysis_runs, process_task_ai_analysis, + queue_task_ai_analysis, restore_ai_analysis_run, ) from app.services.clip_feedback_service import save_clip_feedback @@ -44,9 +49,12 @@ get_source_video_path, resolve_video_file_path, ) +from app.services.publish_domain import TERMINAL_PUBLISH_STATUSES from app.services.task_lifecycle_service import ( + TaskStatusConflictError, create_task_record, soft_delete_task, + transition_task_status, update_task_ai_preference, update_task_candidate_clip_count, update_task_selection_settings, @@ -112,6 +120,7 @@ "AI 分析", "自动选片", "原片切割", + "字幕审核", "标题文案", "发送队列", "发布任务", @@ -126,16 +135,20 @@ TaskStatus.AI_ANALYZING.value: "AI 分析中", TaskStatus.CLIP_SELECTING.value: "自动选片中", TaskStatus.VIDEO_CUTTING.value: "原片切割中", + TaskStatus.SUBTITLE_DRAFTING.value: "生成字幕草稿中", + TaskStatus.PENDING_SUBTITLE_REVIEW.value: "字幕待审核", TaskStatus.METADATA_GENERATING.value: "生成标题文案中", TaskStatus.SCHEDULE_CREATING.value: "准备发送队列中", TaskStatus.PUBLISH_JOB_CREATING.value: "创建发布任务中", TaskStatus.READY_TO_PUBLISH.value: "待人工确认发布", TaskStatus.COMPLETED.value: "全自动流程完成", + TaskStatus.CANCELLED.value: "已取消", TaskStatus.FAILED_PREPARING_SOURCE.value: "准备视频失败", TaskStatus.FAILED_TRANSCRIBING.value: "转写失败", TaskStatus.FAILED_AI_ANALYZING.value: "AI 分析失败", TaskStatus.FAILED_CLIP_SELECTING.value: "自动选片失败", TaskStatus.FAILED_VIDEO_CUTTING.value: "原片切割失败", + TaskStatus.FAILED_SUBTITLE_DRAFTING.value: "字幕草稿生成失败", TaskStatus.FAILED_METADATA_GENERATING.value: "标题文案生成失败", TaskStatus.FAILED_SCHEDULE_CREATING.value: "发送队列准备失败", TaskStatus.FAILED_PUBLISH_JOB_CREATING.value: "发布任务创建失败", @@ -159,19 +172,23 @@ TaskStatus.AI_ANALYZING.value: 45, TaskStatus.CLIP_SELECTING.value: 58, TaskStatus.VIDEO_CUTTING.value: 70, - TaskStatus.METADATA_GENERATING.value: 80, - TaskStatus.SCHEDULE_CREATING.value: 88, - TaskStatus.PUBLISH_JOB_CREATING.value: 94, + TaskStatus.SUBTITLE_DRAFTING.value: 75, + TaskStatus.PENDING_SUBTITLE_REVIEW.value: 78, + TaskStatus.METADATA_GENERATING.value: 84, + TaskStatus.SCHEDULE_CREATING.value: 90, + TaskStatus.PUBLISH_JOB_CREATING.value: 95, TaskStatus.READY_TO_PUBLISH.value: 100, TaskStatus.COMPLETED.value: 100, + TaskStatus.CANCELLED.value: 0, TaskStatus.FAILED_PREPARING_SOURCE.value: 8, TaskStatus.FAILED_TRANSCRIBING.value: 25, TaskStatus.FAILED_AI_ANALYZING.value: 45, TaskStatus.FAILED_CLIP_SELECTING.value: 58, TaskStatus.FAILED_VIDEO_CUTTING.value: 70, - TaskStatus.FAILED_METADATA_GENERATING.value: 80, - TaskStatus.FAILED_SCHEDULE_CREATING.value: 88, - TaskStatus.FAILED_PUBLISH_JOB_CREATING.value: 94, + TaskStatus.FAILED_SUBTITLE_DRAFTING.value: 75, + TaskStatus.FAILED_METADATA_GENERATING.value: 84, + TaskStatus.FAILED_SCHEDULE_CREATING.value: 90, + TaskStatus.FAILED_PUBLISH_JOB_CREATING.value: 95, TaskStatus.pending_video.value: 0, TaskStatus.pending_processing.value: 5, TaskStatus.audio_extracting.value: 20, @@ -192,6 +209,7 @@ TaskStatus.AI_ANALYZING.value, TaskStatus.CLIP_SELECTING.value, TaskStatus.VIDEO_CUTTING.value, + TaskStatus.SUBTITLE_DRAFTING.value, TaskStatus.METADATA_GENERATING.value, TaskStatus.SCHEDULE_CREATING.value, TaskStatus.PUBLISH_JOB_CREATING.value, @@ -206,12 +224,14 @@ TaskStatus.FAILED_AI_ANALYZING.value, TaskStatus.FAILED_CLIP_SELECTING.value, TaskStatus.FAILED_VIDEO_CUTTING.value, + TaskStatus.FAILED_SUBTITLE_DRAFTING.value, TaskStatus.FAILED_METADATA_GENERATING.value, TaskStatus.FAILED_SCHEDULE_CREATING.value, TaskStatus.FAILED_PUBLISH_JOB_CREATING.value, } AUTO_PIPELINE_RESUMABLE_STATUSES = { + TaskStatus.CANCELLED.value, TaskStatus.pending_review.value, TaskStatus.completed_with_errors.value, TaskStatus.failed.value, @@ -225,7 +245,6 @@ SOURCE_TYPE_LABELS = { "upload": "上传视频", - "nas": "选择 NAS / 本地文件", } OUTPUT_STATUS_LABELS = { @@ -253,22 +272,32 @@ def get_task_workflow_steps(task: dict) -> list[dict[str, str]]: TaskStatus.AI_ANALYZING.value: 4, TaskStatus.CLIP_SELECTING.value: 5, TaskStatus.VIDEO_CUTTING.value: 6, - TaskStatus.METADATA_GENERATING.value: 7, - TaskStatus.SCHEDULE_CREATING.value: 8, - TaskStatus.PUBLISH_JOB_CREATING.value: 9, - TaskStatus.READY_TO_PUBLISH.value: 10, - TaskStatus.COMPLETED.value: 10, + TaskStatus.SUBTITLE_DRAFTING.value: 7, + TaskStatus.PENDING_SUBTITLE_REVIEW.value: 7, + TaskStatus.METADATA_GENERATING.value: 8, + TaskStatus.SCHEDULE_CREATING.value: 9, + TaskStatus.PUBLISH_JOB_CREATING.value: 10, + TaskStatus.READY_TO_PUBLISH.value: 11, + TaskStatus.COMPLETED.value: 11, + TaskStatus.CANCELLED.value: 1, TaskStatus.FAILED_PREPARING_SOURCE.value: 2, TaskStatus.FAILED_TRANSCRIBING.value: 3, TaskStatus.FAILED_AI_ANALYZING.value: 4, TaskStatus.FAILED_CLIP_SELECTING.value: 5, TaskStatus.FAILED_VIDEO_CUTTING.value: 6, - TaskStatus.FAILED_METADATA_GENERATING.value: 7, - TaskStatus.FAILED_SCHEDULE_CREATING.value: 8, - TaskStatus.FAILED_PUBLISH_JOB_CREATING.value: 9, + TaskStatus.FAILED_SUBTITLE_DRAFTING.value: 7, + TaskStatus.FAILED_METADATA_GENERATING.value: 8, + TaskStatus.FAILED_SCHEDULE_CREATING.value: 9, + TaskStatus.FAILED_PUBLISH_JOB_CREATING.value: 10, TaskStatus.pending_review.value: 5, - TaskStatus.completed.value: 10, - TaskStatus.completed_with_errors.value: 10, + TaskStatus.pending_processing.value: 2, + TaskStatus.audio_extracting.value: 2, + TaskStatus.transcribing.value: 3, + TaskStatus.pending_ai.value: 4, + TaskStatus.ai_analyzing.value: 4, + TaskStatus.cutting.value: 6, + TaskStatus.completed.value: 11, + TaskStatus.completed_with_errors.value: 11, TaskStatus.failed.value: 1, } failed_statuses = { @@ -277,11 +306,13 @@ def get_task_workflow_steps(task: dict) -> list[dict[str, str]]: TaskStatus.FAILED_AI_ANALYZING.value, TaskStatus.FAILED_CLIP_SELECTING.value, TaskStatus.FAILED_VIDEO_CUTTING.value, + TaskStatus.FAILED_SUBTITLE_DRAFTING.value, TaskStatus.FAILED_METADATA_GENERATING.value, TaskStatus.FAILED_SCHEDULE_CREATING.value, TaskStatus.FAILED_PUBLISH_JOB_CREATING.value, TaskStatus.completed_with_errors.value, TaskStatus.failed.value, + TaskStatus.CANCELLED.value, } completed_statuses = { TaskStatus.READY_TO_PUBLISH.value, @@ -381,8 +412,6 @@ def get_platform_label(platform: str) -> str: def get_source_path(task: dict) -> str: - if task.get("source_type") == "nas": - return task.get("nas_file_path") or "尚未选择 NAS / 本地文件" return task.get("original_video_path") or "尚未上传视频" @@ -435,33 +464,42 @@ def _format_seconds_as_time(seconds: int) -> str: def _probe_video(path: Path | None) -> dict[str, str]: - if not path or not path.exists(): + if not path: return {"duration": "尚未读取", "video_size": "尚未读取"} + try: + file_size = path.stat().st_size + except OSError: + return {"duration": "读取失败", "video_size": "读取失败"} duration = None ffprobe = shutil.which("ffprobe") if ffprobe: - result = subprocess.run( - [ - ffprobe, - "-v", - "error", - "-show_entries", - "format=duration", - "-of", - "default=noprint_wrappers=1:nokey=1", - str(path), - ], - capture_output=True, - text=True, - encoding="utf-8", - errors="replace", - ) - if result.returncode == 0: + try: + result = subprocess.run( + [ + ffprobe, + "-v", + "error", + "-show_entries", + "format=duration", + "-of", + "default=noprint_wrappers=1:nokey=1", + str(path), + ], + capture_output=True, + text=True, + encoding="utf-8", + errors="replace", + timeout=settings.ffprobe_timeout, + check=False, + ) + except (OSError, subprocess.TimeoutExpired): + result = None + if result and result.returncode == 0: try: duration = float(result.stdout.strip()) except ValueError: duration = None - return {"duration": _format_duration(duration), "video_size": _format_file_size(path.stat().st_size)} + return {"duration": _format_duration(duration), "video_size": _format_file_size(file_size)} def _row_to_task(row: Row, include_video_probe: bool = False) -> dict: @@ -494,10 +532,14 @@ def _row_to_task(row: Row, include_video_probe: bool = False) -> dict: "progress": progress, "candidate_count": task.get("candidate_clip_count") or 0, "selection_profile": task.get("selection_profile") or "general", - "selection_profile_label": ( - "康熙笑点选片模式" if task.get("selection_profile") == "variety_comedy" else "通用模式(历史任务)" - ), + "selection_profile_label": { + "general": "通用内容价值", + "variety_comedy": "康熙笑点选片模式", + "long_live_talk": "长直播高光(语言类)", + }.get(task.get("selection_profile") or "general", "通用内容价值"), "final_clip_target": int(task.get("final_clip_target") or 5), + "highlight_density_per_hour": int(task.get("highlight_density_per_hour") or 4), + "highlight_total_limit": int(task.get("highlight_total_limit") or 30), "duration": video_meta["duration"], "video_size": video_meta["video_size"], "owner": "本地用户", @@ -536,6 +578,7 @@ def list_tasks(include_deleted: bool = False) -> list[dict]: SELECT id, task_name, task_dir_name, source_type, platform, original_video_path, nas_file_path, max_clip_duration, candidate_clip_count, selection_profile, final_clip_target, + highlight_density_per_hour, highlight_total_limit, ai_preference, ai_prompt_preset_id, auto_mode, auto_config_json, status, progress, error_message, last_error, is_deleted, deleted_at, created_at, updated_at @@ -559,30 +602,201 @@ def list_task_name_history() -> list[str]: AND TRIM(task_name) <> '' GROUP BY TRIM(task_name) ORDER BY latest_created_at DESC, task_name ASC - LIMIT 100 + LIMIT 5 """ ).fetchall() return [str(row["task_name"]) for row in rows] -def get_task(task_id: str, include_video_probe: bool = True) -> dict | None: +def get_task( + task_id: str, + include_video_probe: bool = True, + *, + include_deleted: bool = False, +) -> dict | None: + deleted_clause = "" if include_deleted else " AND COALESCE(is_deleted, 0) = 0" with get_connection() as connection: row = connection.execute( - """ + f""" SELECT id, task_name, task_dir_name, source_type, platform, original_video_path, nas_file_path, max_clip_duration, candidate_clip_count, selection_profile, final_clip_target, + highlight_density_per_hour, highlight_total_limit, ai_preference, ai_prompt_preset_id, auto_mode, auto_config_json, status, progress, error_message, last_error, is_deleted, deleted_at, created_at, updated_at FROM tasks - WHERE id = ? + WHERE id = ?{deleted_clause} """, (task_id,), ).fetchone() return _row_to_task(row, include_video_probe=include_video_probe) if row else None +def _get_active_workflow_job(task_id: str) -> dict: + from app.services import job_service + + with get_connection() as connection: + row = connection.execute( + """ + SELECT id, job_type, status, progress, message, updated_at + FROM workflow_jobs + WHERE task_id = ? AND status IN (?, ?) + ORDER BY + CASE status WHEN ? THEN 0 ELSE 1 END, + updated_at DESC, + created_at DESC, + id DESC + LIMIT 1 + """, + ( + task_id, + job_service.JOB_STATUS_RUNNING, + job_service.JOB_STATUS_QUEUED, + job_service.JOB_STATUS_RUNNING, + ), + ).fetchone() + if not row: + return {} + job = dict(row) + job["job_type_label"] = job_service.JOB_TYPE_LABELS.get(job["job_type"], job["job_type"]) + job["status_label"] = job_service.JOB_STATUS_LABELS.get(job["status"], job["status"]) + return job + + +def _get_publish_live_summary(task_id: str) -> dict: + terminal_statuses = {status.upper() for status in TERMINAL_PUBLISH_STATUSES} + with get_connection() as connection: + rows = connection.execute( + """ + WITH latest_publish_job AS ( + SELECT + publish_jobs.status, + ROW_NUMBER() OVER ( + PARTITION BY publish_jobs.output_clip_id, publish_jobs.platform + ORDER BY publish_jobs.created_at DESC, + publish_jobs.updated_at DESC, + publish_jobs.id DESC + ) AS row_number + FROM publish_jobs + INNER JOIN output_clip + ON output_clip.id = publish_jobs.output_clip_id + AND output_clip.task_id = ? + AND output_clip.status = 'completed' + AND output_clip.is_active = 1 + ) + SELECT UPPER(status) AS status, COUNT(*) AS count + FROM latest_publish_job + WHERE row_number = 1 + GROUP BY UPPER(status) + """, + (task_id,), + ).fetchall() + + statuses = {str(row["status"] or "").upper(): int(row["count"] or 0) for row in rows} + total = sum(statuses.values()) + success = sum(statuses.get(status, 0) for status in {"PUBLISHED", "EXPORTED"}) + cancelled = statuses.get("CANCELLED", 0) + resolved = sum(statuses.get(status, 0) for status in terminal_statuses) + pending = sum(statuses.get(status, 0) for status in {"DRAFT", "WAITING", "SCHEDULED"}) + publishing = statuses.get("PUBLISHING", 0) + failed = statuses.get("FAILED", 0) + need_review = statuses.get("NEED_REVIEW", 0) + attention = failed + need_review + progress = round(success / total * 100) if total else 0 + + if not total: + state = "none" + label = "尚未创建发布任务" + message = "处理完成后可同步到发送中心。" + elif attention: + state = "attention" + label = f"发布需处理 · {attention} 条" + message = f"已成功 {success}/{total} 条,另有 {attention} 条失败或需要人工复核。" + elif publishing: + state = "publishing" + label = f"正在发布 · {success}/{total}" + message = f"平台正在处理 {publishing} 条,已成功 {success}/{total} 条。" + elif pending: + state = "scheduled" + label = f"托管发布中 · {success}/{total}" + message = f"已成功 {success}/{total} 条,另有 {pending} 条正在等待排期发送。" + elif resolved == total and success == total: + state = "completed" + label = f"发布完成 · {success}/{total}" + message = f"全部 {total} 条发布任务均已完成。" + else: + state = "resolved" + label = f"发布已结束 · {success}/{total}" + message = f"已成功 {success}/{total} 条,取消 {cancelled} 条。" + + return { + "state": state, + "label": label, + "message": message, + "progress": progress, + "total": total, + "success": success, + "resolved": resolved, + "pending": pending, + "publishing": publishing, + "failed": failed, + "need_review": need_review, + "cancelled": cancelled, + "should_poll": bool(pending or publishing), + } + + +def _build_live_activity(task: dict, active_job: dict, publish: dict) -> dict: + status = task["status"] + transcript = task.get("transcript_progress") or {} + publish_task_statuses = { + TaskStatus.READY_TO_PUBLISH.value, + TaskStatus.COMPLETED.value, + TaskStatus.completed.value, + TaskStatus.completed_with_errors.value, + } + + if publish["total"] and status in publish_task_statuses: + return { + "kind": "publish", + "label": "托管发布", + "status": publish["state"], + "progress": publish["progress"], + "message": publish["message"], + "updated_at": "", + } + + if status in {TaskStatus.TRANSCRIBING.value, TaskStatus.transcribing.value} and transcript: + return { + "kind": "transcript", + "label": "转写文本", + "status": str(transcript.get("status") or "running"), + "progress": max(0, min(100, int(transcript.get("percent") or 0))), + "message": str(transcript.get("message") or "正在转写文本"), + "updated_at": str(transcript.get("updated_at") or ""), + } + + if active_job: + return { + "kind": str(active_job.get("job_type") or "workflow"), + "label": str(active_job.get("job_type_label") or "后台任务"), + "status": str(active_job.get("status") or "running"), + "progress": max(0, min(100, int(active_job.get("progress") or 0))), + "message": str(active_job.get("message") or active_job.get("status_label") or "正在处理"), + "updated_at": str(active_job.get("updated_at") or ""), + } + + return { + "kind": "task", + "label": task["status_label"], + "status": status, + "progress": max(0, min(100, int(task.get("progress") or 0))), + "message": task.get("error_message") or f"当前阶段:{task['status_label']}", + "updated_at": str(task.get("updated_at_raw") or ""), + } + + def get_task_live_status(task_id: str) -> dict: task = get_task(task_id, include_video_probe=False) if not task: @@ -590,15 +804,47 @@ def get_task_live_status(task_id: str) -> dict: status = task["status"] auto_mode = bool(task.get("auto_mode")) - is_running = auto_mode and status in AUTO_PIPELINE_RUNNING_STATUSES - should_poll = is_running or (auto_mode and status == TaskStatus.READY_TO_PUBLISH.value) + active_job = _get_active_workflow_job(task_id) + publish = _get_publish_live_summary(task_id) + activity = _build_live_activity(task, active_job, publish) + task_is_running = status in AUTO_PIPELINE_RUNNING_STATUSES + is_running = bool(task_is_running or active_job or publish["should_poll"]) + should_poll = is_running candidate_count = count_clip_candidates(task_id) output_clip_count = int(task.get("output_clip_count") or 0) + workflow_steps = get_task_workflow_steps(task) + + publish_task_statuses = { + TaskStatus.READY_TO_PUBLISH.value, + TaskStatus.COMPLETED.value, + TaskStatus.completed.value, + TaskStatus.completed_with_errors.value, + } + display_status_label = task["status_label"] + overall_progress = int(task.get("progress") or 0) + if publish["total"] and status in publish_task_statuses: + display_status_label = publish["label"] + overall_progress = 100 if publish["state"] == "completed" else min( + 99, + 90 + round(publish["progress"] * 0.09), + ) + if workflow_steps: + workflow_steps[-1]["name"] = "平台发布" + if publish["state"] == "attention": + workflow_steps[-1]["state"] = "warning" + elif publish["state"] in {"scheduled", "publishing"}: + workflow_steps[-1]["state"] = "current" + + runtime_status = "running" if should_poll else ("completed" if overall_progress >= 100 else "idle") + if task.get("error_message") or publish["state"] == "attention": + runtime_status = "failed" primary_action = "none" if auto_mode: if status in AUTO_PIPELINE_FAILED_STATUSES: primary_action = "retry" + elif status == TaskStatus.PENDING_SUBTITLE_REVIEW.value: + primary_action = "subtitle_review" elif status in { TaskStatus.READY_TO_PUBLISH.value, TaskStatus.COMPLETED.value, @@ -612,14 +858,21 @@ def get_task_live_status(task_id: str) -> dict: return { "task_id": task_id, + "snapshot_at": _now_iso(), "status": status, - "status_label": task["status_label"], - "progress": int(task.get("progress") or 0), + "status_label": display_status_label, + "task_status_label": task["status_label"], + "progress": overall_progress, + "task_progress": int(task.get("progress") or 0), "updated_at": task["updated_at"], "error_message": task.get("error_message") or "", "is_running": is_running, "should_poll": should_poll, - "workflow_steps": get_task_workflow_steps(task), + "runtime_status": runtime_status, + "runtime_status_label": display_status_label, + "workflow_steps": workflow_steps, + "active_operation": activity, + "publish": publish, "log_lines": _read_task_log_tail(task_id), "counts": { "candidates": candidate_count, @@ -920,6 +1173,13 @@ def sync_reviewed_clips_to_publish_center( task_id: str, payloads: list[ClipCandidateBatchItem], ) -> dict: + with get_connection() as connection: + task = connection.execute( + "SELECT auto_mode, status FROM tasks WHERE id = ? AND COALESCE(is_deleted, 0) = 0", + (task_id,), + ).fetchone() + if task and bool(task["auto_mode"]) and task["status"] == TaskStatus.PENDING_SUBTITLE_REVIEW.value: + raise ValueError("自动流水线正在等待字幕审核,请先完成字幕决定,再同步发送中心") save_result = update_clip_candidates_batch(task_id, payloads) needs_regeneration = bool(save_result["changed_count"]) or not _active_outputs_match_enabled_candidates(task_id) @@ -942,6 +1202,22 @@ def sync_reviewed_clips_to_publish_center( ) action_message = "已保存当前审核选择,现有切片与选择一致,无需重复生成。" + link_state = publish_sync.get("link_state") or {} + publish_sync_ok = ( + publish_sync.get("status") == "ok" + and not (publish_sync.get("errors") or []) + and link_state.get("state") == "linked" + and int(link_state.get("missing_count") or 0) == 0 + ) + current_task = get_task(task_id, include_video_probe=False) + if publish_sync_ok and current_task and current_task.get("status") == TaskStatus.pending_review.value: + transition_task_status(task_id, TaskStatus.completed) + elif not publish_sync_ok and current_task and current_task.get("status") in { + TaskStatus.completed.value, + TaskStatus.completed_with_errors.value, + }: + update_task_status(task_id, TaskStatus.pending_review) + return { "status": publish_sync.get("status") or "ok", "message": f"{action_message}{publish_sync.get('message') or '发送中心同步完成。'}", @@ -951,7 +1227,7 @@ def sync_reviewed_clips_to_publish_center( "review_save": save_result, "cut_result": cut_result, "publish_sync": publish_sync, - "link_state": publish_sync.get("link_state") or {}, + "link_state": link_state, "errors": publish_sync.get("errors") or [], "warnings": publish_sync.get("warnings") or [], } @@ -1063,13 +1339,20 @@ def list_output_clips(task_id: str) -> list[dict]: clip_candidates.enabled AS clip_enabled, subtitle_jobs.id AS subtitle_job_id, subtitle_jobs.status AS subtitle_status, + subtitle_jobs.revision_id AS subtitle_revision_id, subtitle_jobs.subtitle_file_path, subtitle_jobs.output_file_path AS subtitled_output_file_path, subtitle_jobs.error_message AS subtitle_error_message, + subtitle_jobs.validation_status AS subtitle_validation_status, + subtitle_jobs.validation_json AS subtitle_validation_json, + subtitle_jobs.encoder AS subtitle_encoder, + subtitle_jobs.verified_at AS subtitle_verified_at, + subtitle_revisions.status AS subtitle_revision_status, subtitle_jobs.updated_at AS subtitle_updated_at FROM output_clip LEFT JOIN clip_candidates ON clip_candidates.id = output_clip.clip_candidate_id LEFT JOIN subtitle_jobs ON subtitle_jobs.output_clip_id = output_clip.id AND subtitle_jobs.is_active = 1 + LEFT JOIN subtitle_revisions ON subtitle_revisions.id = subtitle_jobs.revision_id WHERE output_clip.task_id = ? AND output_clip.is_active = 1 ORDER BY CASE WHEN output_clip.output_file_name IS NULL OR output_clip.output_file_name = '' THEN 1 ELSE 0 END, @@ -1105,6 +1388,14 @@ def list_output_clips(task_id: str) -> list[dict]: "clip_end_seconds": clip_end_seconds, "subtitle_status": subtitle_status, "subtitle_status_label": SUBTITLE_STATUS_LABELS.get(subtitle_status, subtitle_status), + "subtitle_publish_ready": bool( + subtitle_status == "completed" + and output.get("subtitle_validation_status") == "verified" + and output.get("subtitle_revision_status") == "approved" + and subtitled_path + and subtitled_path.exists() + and subtitled_path.is_file() + ), "subtitle_stage": SUBTITLE_STATUS_LABELS.get(subtitle_status, subtitle_status), "subtitled_file_exists": bool(subtitled_path and subtitled_path.exists() and subtitled_path.is_file()), "subtitled_media_url": f"/media/tasks/{task_id}/subtitled-clips/{output['id']}", @@ -1131,8 +1422,9 @@ def get_output_clip(task_id: str, output_clip_id: str) -> dict | None: with get_connection() as connection: subtitle_row = connection.execute( """ - SELECT * + SELECT subtitle_jobs.*, subtitle_revisions.status AS subtitle_revision_status FROM subtitle_jobs + LEFT JOIN subtitle_revisions ON subtitle_revisions.id = subtitle_jobs.revision_id WHERE task_id = ? AND output_clip_id = ? AND is_active = 1 """, (task_id, output_clip_id), @@ -1145,6 +1437,12 @@ def get_output_clip(task_id: str, output_clip_id: str) -> dict | None: "subtitle_file_path": subtitle_row["subtitle_file_path"], "subtitled_output_file_path": subtitle_row["output_file_path"], "subtitle_error_message": subtitle_row["error_message"], + "subtitle_revision_id": subtitle_row["revision_id"], + "subtitle_revision_status": subtitle_row["subtitle_revision_status"], + "subtitle_validation_status": subtitle_row["validation_status"], + "subtitle_validation_json": subtitle_row["validation_json"], + "subtitle_encoder": subtitle_row["encoder"], + "subtitle_verified_at": subtitle_row["verified_at"], "subtitle_status_label": SUBTITLE_STATUS_LABELS.get(subtitle_row["status"], subtitle_row["status"]), } ) diff --git a/app/services/transcript_service.py b/app/services/transcript_service.py index 8974504..5743f1f 100644 --- a/app/services/transcript_service.py +++ b/app/services/transcript_service.py @@ -5,7 +5,11 @@ from pathlib import Path import re import shutil +import socket import subprocess +import queue +import threading +import time from tempfile import TemporaryDirectory from typing import Callable from urllib.error import HTTPError, URLError @@ -13,6 +17,12 @@ from uuid import uuid4 from app.core.config import settings +from app.services import job_service +from app.services.managed_process_service import popen_process_group, terminate_process_tree +from app.services.transcription_checkpoint_service import ( + RemoteTranscriptionResultUncertainError, + TranscriptionCheckpoint, +) TIME_TABLE_PATTERN = re.compile( @@ -26,11 +36,21 @@ PLACEHOLDER_TEXT_MARKERS = ("这里会保存", "后续可接入", "占位转写内容") +@dataclass(frozen=True) +class TranscriptWord: + start_ms: int + end_ms: int + text: str + confidence: float | None = None + + @dataclass(frozen=True) class TranscriptSegment: start_seconds: float end_seconds: float text: str + confidence: float | None = None + words: tuple[TranscriptWord, ...] = () @dataclass(frozen=True) @@ -63,13 +83,49 @@ def duration_seconds(self) -> float: "nvidia", ) _RESERVED_REMOTE_PROVIDERS = ("aliyun", "tencent", "xunfei") +_REMOTE_SAFE_RETRY_ATTEMPTS = 3 + +class RemoteTranscriptionError(RuntimeError): + """远程转写失败,并明确是否允许在本进程内安全重试。""" -def run_ffmpeg_audio_extract(video_path: Path, output_path: Path) -> dict[str, str]: + def __init__( + self, + message: str, + *, + category: str, + safe_to_retry: bool = False, + billing_uncertain: bool = False, + retry_after_seconds: float | None = None, + ) -> None: + suffix = ";本次是否计费不确定,未自动重试" if billing_uncertain else "" + super().__init__(f"[{category}] {message}{suffix}") + self.category = category + self.safe_to_retry = safe_to_retry + self.billing_uncertain = billing_uncertain + self.retry_after_seconds = retry_after_seconds + + +class TranscriptCancelledError(RuntimeError): + """用户取消转写;必须穿透 Provider 包装并由工作流记为 cancelled。""" + + +def run_ffmpeg_audio_extract( + video_path: Path, + output_path: Path, + *, + cancel_check: Callable[[], bool] | None = None, + progress_callback: Callable[[int], None] | None = None, +) -> dict[str, str]: output_path.parent.mkdir(parents=True, exist_ok=True) + temporary_path = output_path.with_name(f"{output_path.stem}.part{output_path.suffix}") + temporary_path.unlink(missing_ok=True) command = [ "ffmpeg", "-y", + "-nostats", + "-loglevel", + "error", "-i", str(video_path), "-vn", @@ -79,12 +135,67 @@ def run_ffmpeg_audio_extract(video_path: Path, output_path: Path) -> dict[str, s "16000", "-ac", "1", - str(output_path), + "-progress", + "pipe:1", + str(temporary_path), ] - result = subprocess.run(command, capture_output=True, text=True, encoding="utf-8", errors="replace", - timeout=settings.ffmpeg_audio_extract_timeout) - if result.returncode != 0: - raise RuntimeError(result.stderr.strip() or "FFmpeg 音频提取失败") + duration = get_audio_duration_seconds(video_path) + process = popen_process_group( + command, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + text=True, + encoding="utf-8", + errors="replace", + ) + output_lines: queue.Queue[str] = queue.Queue() + stderr_lines: list[str] = [] + + def read_stream(stream, target) -> None: + if not stream: + return + for line in iter(stream.readline, ""): + target(line.rstrip()) + + stdout_thread = threading.Thread(target=read_stream, args=(process.stdout, output_lines.put), daemon=True) + stderr_thread = threading.Thread(target=read_stream, args=(process.stderr, stderr_lines.append), daemon=True) + stdout_thread.start() + stderr_thread.start() + last_progress_at = time.monotonic() + try: + while process.poll() is None: + if cancel_check and cancel_check(): + terminate_process_tree(process) + raise RuntimeError("用户已取消音频提取") + try: + line = output_lines.get(timeout=0.5) + except queue.Empty: + line = "" + if line: + last_progress_at = time.monotonic() + if line.startswith(("out_time_ms=", "out_time_us=")): + try: + microseconds = int(line.split("=", 1)[1]) + percent = round((microseconds / 1_000_000) / max(duration, 0.001) * 100) + if progress_callback: + progress_callback(max(0, min(99, percent))) + except ValueError: + pass + if time.monotonic() - last_progress_at > settings.ffmpeg_audio_extract_timeout: + terminate_process_tree(process) + raise RuntimeError(f"FFmpeg 连续 {settings.ffmpeg_audio_extract_timeout} 秒没有进展,已终止进程树") + stdout_thread.join(timeout=2) + stderr_thread.join(timeout=2) + if process.returncode != 0: + raise RuntimeError("\n".join(stderr_lines[-20:]).strip() or "FFmpeg 音频提取失败") + temporary_path.replace(output_path) + if progress_callback: + progress_callback(100) + except Exception: + if process.poll() is None: + terminate_process_tree(process) + temporary_path.unlink(missing_ok=True) + raise return { "status": "ok", "message": "音频提取完成", @@ -99,6 +210,7 @@ def write_transcript_markdown( transcript_path: Path, progress_callback: Callable[[dict], None] | None = None, provider: str | None = None, + allow_uncertain_retry: bool = False, ) -> dict[str, str]: if not audio_path.exists(): raise RuntimeError("未找到音频文件,请先提取音频") @@ -106,6 +218,20 @@ def write_transcript_markdown( transcript_path.parent.mkdir(parents=True, exist_ok=True) progress_path = get_transcript_progress_path(transcript_path) _set_configured_transcription_runtime(provider) + checkpoint = None + task_id = str(task.get("id") or "").strip() + if task_id: + checkpoint = TranscriptionCheckpoint( + task_id=task_id, + source_path=audio_path, + provider=_ACTIVE_TRANSCRIPTION_PROVIDER, + model=_ACTIVE_TRANSCRIPTION_MODEL, + device=_ACTIVE_TRANSCRIPTION_DEVICE, + compute_type=_ACTIVE_TRANSCRIPTION_COMPUTE_TYPE, + chunk_seconds=settings.transcription_chunk_seconds, + overlap_seconds=settings.transcription_chunk_overlap_seconds, + allow_uncertain_retry=allow_uncertain_retry, + ) _emit_transcript_progress( progress_path, progress_callback, @@ -121,10 +247,13 @@ def write_transcript_markdown( progress_path, progress_callback, provider=provider, + checkpoint=checkpoint, ) + job_service.require_active_job_lease() content = build_transcript_markdown(task, audio_path, segments) temp_path = transcript_path.with_name(f"{transcript_path.name}.tmp") temp_path.write_text(content, encoding="utf-8") + job_service.require_active_job_lease() temp_path.replace(transcript_path) final_progress = read_transcript_progress(transcript_path) total_chunks = int(final_progress.get("total_chunks") or 0) @@ -153,11 +282,18 @@ def transcribe_audio_with_configured_provider( progress_callback: Callable[[dict], None] | None = None, provider: str | None = None, allow_fallback: bool = False, + checkpoint: TranscriptionCheckpoint | None = None, ) -> list[TranscriptSegment]: provider = _normalize_provider_name(provider or settings.transcription_provider) fallback_provider = _normalize_provider_name(settings.transcription_fallback_provider) try: - return transcribe_audio_with_provider(audio_path, working_dir, progress_path, provider, progress_callback) + return transcribe_audio_with_provider( + audio_path, working_dir, progress_path, provider, progress_callback, checkpoint=checkpoint + ) + except job_service.JobLeaseLostError: + raise + except TranscriptCancelledError: + raise except Exception as exc: if allow_fallback and fallback_provider and fallback_provider != provider: _emit_transcript_progress( @@ -177,6 +313,10 @@ def transcribe_audio_with_configured_provider( fallback_provider, progress_callback, ) + except job_service.JobLeaseLostError: + raise + except TranscriptCancelledError: + raise except Exception as fallback_exc: raise RuntimeError( f"{_provider_label(provider)} 转写失败:{exc};" @@ -191,6 +331,7 @@ def transcribe_audio_with_provider( progress_path: Path, provider: str, progress_callback: Callable[[dict], None] | None = None, + checkpoint: TranscriptionCheckpoint | None = None, ) -> list[TranscriptSegment]: provider = _normalize_provider_name(provider) if provider == "local": @@ -201,9 +342,13 @@ def transcribe_audio_with_provider( device=(_WHISPER_MODEL_KEY or _primary_model_key())[1], compute_type=(_WHISPER_MODEL_KEY or _primary_model_key())[2], ) - return transcribe_audio_in_chunks(audio_path, working_dir, progress_path, progress_callback) + return transcribe_audio_in_chunks( + audio_path, working_dir, progress_path, progress_callback, checkpoint=checkpoint + ) if provider == "volcengine": - return transcribe_audio_with_volcengine(audio_path, working_dir, progress_path, progress_callback) + return transcribe_audio_with_volcengine( + audio_path, working_dir, progress_path, progress_callback, checkpoint=checkpoint + ) if provider in _RESERVED_REMOTE_PROVIDERS: raise RuntimeError(f"{_provider_label(provider)} 转写接口已预留,但当前版本还没有完整接入") raise RuntimeError(f"未知转写服务商:{provider or '空'}") @@ -214,6 +359,7 @@ def transcribe_audio_in_chunks( working_dir: Path, progress_path: Path, progress_callback: Callable[[dict], None] | None = None, + checkpoint: TranscriptionCheckpoint | None = None, ) -> list[TranscriptSegment]: duration_seconds = get_audio_duration_seconds(audio_path) chunks = build_transcript_chunks( @@ -223,6 +369,8 @@ def transcribe_audio_in_chunks( ) if not chunks: raise RuntimeError("本地语音转写失败:音频时长无效,无法分段") + if checkpoint: + checkpoint.ensure_run(chunks) _emit_transcript_progress( progress_path, progress_callback, @@ -237,6 +385,17 @@ def transcribe_audio_in_chunks( with TemporaryDirectory(prefix=_TRANSCRIPT_CHUNK_DIR_PREFIX, dir=working_dir) as temp_dir: temp_dir_path = Path(temp_dir) for chunk in chunks: + completed_segments = checkpoint.load_completed(chunk.index, _segment_from_checkpoint) if checkpoint else None + if completed_segments is not None: + all_segments.extend( + _offset_chunk_segments(completed_segments, chunk, settings.transcription_chunk_overlap_seconds) + ) + _emit_transcript_progress( + progress_path, progress_callback, status="running", current_chunk=chunk.index, + total_chunks=len(chunks), percent=_chunk_percent(chunk.index, len(chunks)), + message=f"已复用第 {chunk.index}/{len(chunks)} 段 checkpoint", + ) + continue _emit_transcript_progress( progress_path, progress_callback, @@ -257,7 +416,14 @@ def transcribe_audio_in_chunks( percent=_chunk_start_percent(chunk.index, len(chunks)), message=f"正在加载模型并转写第 {chunk.index}/{len(chunks)} 段", ) - chunk_segments = transcribe_audio(chunk_path, allow_empty=True) + try: + chunk_segments = transcribe_audio(chunk_path, allow_empty=True) + if checkpoint: + checkpoint.save_completed(chunk.index, chunk_segments) + except Exception as exc: + if checkpoint: + checkpoint.save_failed(chunk.index, str(exc)) + raise all_segments.extend(_offset_chunk_segments(chunk_segments, chunk, settings.transcription_chunk_overlap_seconds)) _emit_transcript_progress( progress_path, @@ -271,6 +437,8 @@ def transcribe_audio_in_chunks( if not all_segments: raise RuntimeError("本地语音转写完成,但没有识别到可用语音内容") + if checkpoint: + checkpoint.complete() return sorted(all_segments, key=lambda segment: (segment.start_seconds, segment.end_seconds)) @@ -279,6 +447,7 @@ def transcribe_audio_with_volcengine( working_dir: Path, progress_path: Path, progress_callback: Callable[[dict], None] | None = None, + checkpoint: TranscriptionCheckpoint | None = None, ) -> list[TranscriptSegment]: _ensure_volcengine_configured() _set_active_transcription_runtime( @@ -297,6 +466,8 @@ def transcribe_audio_with_volcengine( ) if not chunks: raise RuntimeError("火山引擎远程转写失败:音频时长无效,无法分段") + if checkpoint: + checkpoint.ensure_run(chunks) _emit_transcript_progress( progress_path, progress_callback, @@ -311,6 +482,17 @@ def transcribe_audio_with_volcengine( with TemporaryDirectory(prefix=_TRANSCRIPT_CHUNK_DIR_PREFIX, dir=working_dir) as temp_dir: temp_dir_path = Path(temp_dir) for chunk in chunks: + completed_segments = checkpoint.load_completed(chunk.index, _segment_from_checkpoint) if checkpoint else None + if completed_segments is not None: + all_segments.extend( + _offset_chunk_segments(completed_segments, chunk, settings.transcription_chunk_overlap_seconds) + ) + _emit_transcript_progress( + progress_path, progress_callback, status="running", current_chunk=chunk.index, + total_chunks=len(chunks), percent=_chunk_percent(chunk.index, len(chunks)), + message=f"已复用第 {chunk.index}/{len(chunks)} 段 checkpoint", + ) + continue _emit_transcript_progress( progress_path, progress_callback, @@ -331,7 +513,24 @@ def transcribe_audio_with_volcengine( percent=_chunk_start_percent(chunk.index, len(chunks)), message=f"正在请求火山引擎转写第 {chunk.index}/{len(chunks)} 段", ) - chunk_segments = transcribe_audio_with_volcengine_flash(chunk_path, allow_empty=True) + request_id = checkpoint.prepare_remote_request(chunk.index) if checkpoint else None + try: + chunk_segments = transcribe_audio_with_volcengine_flash( + chunk_path, + allow_empty=True, + request_id=request_id, + ) + except Exception as exc: + if checkpoint: + if isinstance(exc, RemoteTranscriptionResultUncertainError) or ( + isinstance(exc, RemoteTranscriptionError) and exc.billing_uncertain + ): + checkpoint.save_uncertain(chunk.index, str(exc)) + else: + checkpoint.save_failed(chunk.index, str(exc), attempt_already_counted=True) + raise + if checkpoint: + _save_remote_checkpoint_completed(checkpoint, chunk.index, chunk_segments) all_segments.extend(_offset_chunk_segments(chunk_segments, chunk, settings.transcription_chunk_overlap_seconds)) _emit_transcript_progress( progress_path, @@ -345,13 +544,64 @@ def transcribe_audio_with_volcengine( if not all_segments: raise RuntimeError("火山引擎远程转写完成,但没有识别到可用语音内容") + if checkpoint: + checkpoint.complete() return sorted(all_segments, key=lambda segment: (segment.start_seconds, segment.end_seconds)) -def transcribe_audio_with_volcengine_flash(audio_path: Path, allow_empty: bool = False) -> list[TranscriptSegment]: +def _save_remote_checkpoint_completed( + checkpoint: TranscriptionCheckpoint, + chunk_index: int, + segments: list[TranscriptSegment], +) -> None: + """Provider 已返回后若落账失败,保留 requesting/uncertain,绝不降为可自动重试。""" + try: + checkpoint.save_completed(chunk_index, segments, attempt_already_counted=True) + except job_service.JobLeaseLostError: + # 旧 worker 不得写回;requesting 会让新 owner 停止自动重发。 + raise + except Exception as exc: + uncertain = RemoteTranscriptionResultUncertainError( + f"第 {chunk_index} 段远程转写已返回,但 checkpoint 保存失败;" + "结果与计费状态不确定,普通重试不会再次请求。" + ) + try: + checkpoint.save_uncertain(chunk_index, str(uncertain)) + except job_service.JobLeaseLostError: + raise + except Exception: + # 数据库仍不可写时保留请求前的 requesting,恢复端同样会 fail closed。 + pass + raise uncertain from exc + + +def transcribe_audio_with_volcengine_flash( + audio_path: Path, + allow_empty: bool = False, + *, + sleep_fn: Callable[[float], None] = time.sleep, + request_id: str | None = None, +) -> list[TranscriptSegment]: _ensure_volcengine_configured() - headers = _volcengine_headers() + headers = _volcengine_headers(request_id=request_id) payload = _build_volcengine_flash_payload(audio_path) + for attempt in range(1, _REMOTE_SAFE_RETRY_ATTEMPTS + 1): + try: + return _request_volcengine_transcript(payload, headers, allow_empty=allow_empty) + except RemoteTranscriptionError as exc: + if not exc.safe_to_retry or attempt >= _REMOTE_SAFE_RETRY_ATTEMPTS: + raise + delay = exc.retry_after_seconds if exc.retry_after_seconds is not None else 2 ** (attempt - 1) + sleep_fn(max(0.0, min(delay, 60.0))) + raise AssertionError("远程转写安全重试循环异常结束") + + +def _request_volcengine_transcript( + payload: dict, + headers: dict[str, str], + *, + allow_empty: bool, +) -> list[TranscriptSegment]: request = Request( settings.volcengine_asr_api_url, data=json.dumps(payload).encode("utf-8"), @@ -366,29 +616,90 @@ def transcribe_audio_with_volcengine_flash(audio_path: Path, allow_empty: bool = log_id = response.headers.get("X-Tt-Logid", "") except HTTPError as exc: error_text = exc.read().decode("utf-8", errors="replace") - raise RuntimeError(f"火山引擎接口返回 HTTP {exc.code}:{error_text[:500]}") from exc + retry_after = ( + _parse_retry_after_seconds((exc.headers or {}).get("Retry-After")) + if exc.code == 429 + else None + ) + raise RemoteTranscriptionError( + f"火山引擎接口返回 HTTP {exc.code}:{error_text[:500]}", + category="rate_limited" if exc.code == 429 else "http_error", + safe_to_retry=exc.code == 429, + billing_uncertain=exc.code >= 500 or exc.code == 408, + retry_after_seconds=retry_after, + ) from exc except URLError as exc: - raise RuntimeError(f"无法连接火山引擎转写接口:{exc}") from exc + reason = exc.reason + is_timeout = isinstance(reason, TimeoutError) + is_preconnect_failure = isinstance(reason, (ConnectionRefusedError, socket.gaierror)) + raise RemoteTranscriptionError( + "火山引擎转写接口连接超时" if is_timeout else f"无法连接火山引擎转写接口:{reason}", + category="timeout" if is_timeout else "network_error", + safe_to_retry=is_preconnect_failure, + billing_uncertain=not is_preconnect_failure, + ) from exc + except TimeoutError as exc: + raise RemoteTranscriptionError( + "火山引擎转写接口连接或读取超时", + category="timeout", + billing_uncertain=True, + ) from exc + except OSError as exc: + raise RemoteTranscriptionError( + f"火山引擎转写响应读取失败:{exc}", + category="network_error", + billing_uncertain=True, + ) from exc if status_code and status_code != "20000000": if allow_empty and status_code == "20000003": return [] - raise RuntimeError( + raise RemoteTranscriptionError( "火山引擎接口返回业务错误:" - f"code={status_code}, message={status_message or 'unknown'}, logid={log_id or 'unknown'}" + f"code={status_code}, message={status_message or 'unknown'}, logid={log_id or 'unknown'}", + category="business_error", + billing_uncertain=True, ) try: payload = json.loads(response_text) except json.JSONDecodeError as exc: - raise RuntimeError(f"火山引擎接口返回的不是 JSON:{response_text[:500]}") from exc + raise RemoteTranscriptionError( + f"火山引擎接口返回的不是 JSON:{response_text[:500]}", + category="invalid_response_json", + billing_uncertain=True, + ) from exc - segments = parse_volcengine_transcript_segments(payload) + if not isinstance(payload, dict) or not isinstance(payload.get("result"), dict): + raise RemoteTranscriptionError( + "火山引擎成功响应缺少 result 对象", + category="invalid_response_schema", + billing_uncertain=True, + ) + try: + segments = parse_volcengine_transcript_segments(payload) + except (TypeError, ValueError, KeyError, OverflowError) as exc: + raise RemoteTranscriptionError( + f"火山引擎成功响应的转写字段无效:{exc}", + category="invalid_response_schema", + billing_uncertain=True, + ) from exc if not segments and not allow_empty: - raise RuntimeError(f"火山引擎远程转写没有返回可用文本:{json.dumps(payload, ensure_ascii=False)[:500]}") + raise RemoteTranscriptionError( + f"火山引擎远程转写没有返回可用文本:{json.dumps(payload, ensure_ascii=False)[:500]}", + category="empty_model_output", + billing_uncertain=True, + ) return segments +def _parse_retry_after_seconds(value: str | None) -> float | None: + try: + return max(0.0, min(float(str(value or "").strip()), 60.0)) + except ValueError: + return None + + def parse_volcengine_transcript_segments(payload: dict) -> list[TranscriptSegment]: result = payload.get("result") if isinstance(payload, dict) else None if not isinstance(result, dict): @@ -399,7 +710,7 @@ def parse_volcengine_transcript_segments(payload: dict) -> list[TranscriptSegmen segments = [_segment_from_volcengine_utterance(item) for item in utterances if isinstance(item, dict)] return [segment for segment in segments if segment and segment.text] - text = normalize_transcript_text(str(result.get("text") or result.get("message") or "")) if isinstance(result, dict) else "" + text = normalize_transcript_text(str(result.get("text") or "")) if isinstance(result, dict) else "" if text: return [TranscriptSegment(start_seconds=0, end_seconds=1, text=text)] return [] @@ -416,8 +727,20 @@ def get_audio_duration_seconds(audio_path: Path) -> float: "default=noprint_wrappers=1:nokey=1", str(audio_path), ] - result = subprocess.run(command, capture_output=True, text=True, encoding="utf-8", errors="replace", - timeout=settings.ffprobe_timeout) + try: + result = subprocess.run( + command, + capture_output=True, + text=True, + encoding="utf-8", + errors="replace", + timeout=settings.ffprobe_timeout, + check=False, + ) + except subprocess.TimeoutExpired as exc: + raise RuntimeError(f"FFprobe 读取音频时长超过 {settings.ffprobe_timeout} 秒") from exc + except OSError as exc: + raise RuntimeError(f"FFprobe 无法读取音频时长:{exc}") from exc if result.returncode != 0: raise RuntimeError(result.stderr.strip() or "FFprobe 无法读取音频时长") try: @@ -475,9 +798,25 @@ def _extract_audio_chunk(audio_path: Path, chunk_path: Path, chunk: TranscriptCh "1", str(chunk_path), ] - result = subprocess.run(command, capture_output=True, text=True, encoding="utf-8", errors="replace", - timeout=settings.ffmpeg_chunk_timeout) + chunk_path.unlink(missing_ok=True) + try: + result = subprocess.run( + command, + capture_output=True, + text=True, + encoding="utf-8", + errors="replace", + timeout=settings.ffmpeg_chunk_timeout, + check=False, + ) + except subprocess.TimeoutExpired as exc: + chunk_path.unlink(missing_ok=True) + raise RuntimeError(f"FFmpeg 音频分段超过 {settings.ffmpeg_chunk_timeout} 秒:第 {chunk.index} 段") from exc + except OSError as exc: + chunk_path.unlink(missing_ok=True) + raise RuntimeError(f"FFmpeg 无法执行音频分段:第 {chunk.index} 段:{exc}") from exc if result.returncode != 0: + chunk_path.unlink(missing_ok=True) raise RuntimeError(result.stderr.strip() or f"FFmpeg 音频分段失败:第 {chunk.index} 段") @@ -504,9 +843,27 @@ def _extract_remote_audio_chunk(audio_path: Path, chunk_path: Path, chunk: Trans *codec_args, str(chunk_path), ] - result = subprocess.run(command, capture_output=True, text=True, encoding="utf-8", errors="replace", - timeout=settings.ffmpeg_chunk_timeout) + chunk_path.unlink(missing_ok=True) + try: + result = subprocess.run( + command, + capture_output=True, + text=True, + encoding="utf-8", + errors="replace", + timeout=settings.ffmpeg_chunk_timeout, + check=False, + ) + except subprocess.TimeoutExpired as exc: + chunk_path.unlink(missing_ok=True) + raise RuntimeError( + f"FFmpeg 远程转写音频压缩超过 {settings.ffmpeg_chunk_timeout} 秒:第 {chunk.index} 段" + ) from exc + except OSError as exc: + chunk_path.unlink(missing_ok=True) + raise RuntimeError(f"FFmpeg 无法执行远程转写音频压缩:第 {chunk.index} 段:{exc}") from exc if result.returncode != 0: + chunk_path.unlink(missing_ok=True) raise RuntimeError(result.stderr.strip() or f"FFmpeg 远程转写音频压缩失败:第 {chunk.index} 段") @@ -537,11 +894,11 @@ def _ensure_volcengine_configured() -> None: raise RuntimeError("缺少火山引擎资源 ID,请在系统状态页的“1. 音频转写”填写资源 ID") -def _volcengine_headers() -> dict[str, str]: +def _volcengine_headers(*, request_id: str | None = None) -> dict[str, str]: headers = { "Content-Type": "application/json", "X-Api-Resource-Id": settings.volcengine_asr_resource_id, - "X-Api-Request-Id": uuid4().hex, + "X-Api-Request-Id": request_id or uuid4().hex, "X-Api-Sequence": "-1", } if settings.volcengine_asr_api_key: @@ -561,7 +918,34 @@ def _segment_from_volcengine_utterance(item: dict) -> TranscriptSegment | None: end_seconds = _volcengine_utterance_time_to_seconds(item, "end_time", "end") if end_seconds <= start_seconds: end_seconds = start_seconds + 1 - return TranscriptSegment(start_seconds=start_seconds, end_seconds=end_seconds, text=text) + raw_words = item.get("words") or item.get("word_info") or [] + words: list[TranscriptWord] = [] + if isinstance(raw_words, list): + for word in raw_words: + if not isinstance(word, dict): + continue + word_text = normalize_transcript_text(str(word.get("text") or word.get("word") or "")) + if not word_text: + continue + word_start = _volcengine_utterance_time_to_seconds(word, "start_time", "start") + word_end = _volcengine_utterance_time_to_seconds(word, "end_time", "end") + confidence = word.get("confidence") or word.get("probability") + words.append( + TranscriptWord( + start_ms=round(word_start * 1000), + end_ms=round(max(word_start, word_end) * 1000), + text=word_text, + confidence=float(confidence) if confidence is not None else None, + ) + ) + segment_confidence = item.get("confidence") + return TranscriptSegment( + start_seconds=start_seconds, + end_seconds=end_seconds, + text=text, + confidence=float(segment_confidence) if segment_confidence is not None else None, + words=tuple(words), + ) def _volcengine_utterance_time_to_seconds(item: dict, millisecond_key: str, fallback_key: str) -> float: @@ -608,11 +992,41 @@ def _offset_chunk_segments( start_seconds=adjusted_start, end_seconds=adjusted_end, text=segment.text, + confidence=segment.confidence, + words=tuple( + TranscriptWord( + start_ms=word.start_ms + round(chunk.start_seconds * 1000), + end_ms=word.end_ms + round(chunk.start_seconds * 1000), + text=word.text, + confidence=word.confidence, + ) + for word in segment.words + ), ) ) return adjusted_segments +def _segment_from_checkpoint(item: dict) -> TranscriptSegment: + words = tuple( + TranscriptWord( + start_ms=int(word.get("start_ms") or 0), + end_ms=int(word.get("end_ms") or 0), + text=str(word.get("text") or ""), + confidence=float(word["confidence"]) if word.get("confidence") is not None else None, + ) + for word in (item.get("words") or []) + if isinstance(word, dict) + ) + return TranscriptSegment( + start_seconds=float(item.get("start_seconds") or 0), + end_seconds=float(item.get("end_seconds") or 0), + text=str(item.get("text") or ""), + confidence=float(item["confidence"]) if item.get("confidence") is not None else None, + words=words, + ) + + def get_transcript_progress_path(transcript_path: Path) -> Path: return transcript_path.with_name(_TRANSCRIPT_PROGRESS_FILE_NAME) @@ -743,16 +1157,35 @@ def _transcribe_audio_with_model(model, audio_path: Path) -> list[TranscriptSegm language=settings.transcription_language or None, vad_filter=True, beam_size=5, + word_timestamps=True, ) - return [ - TranscriptSegment( - start_seconds=float(segment.start), - end_seconds=float(segment.end), - text=normalize_transcript_text(segment.text), + results: list[TranscriptSegment] = [] + for segment in raw_segments: + text = normalize_transcript_text(segment.text) + if not text: + continue + words = tuple( + TranscriptWord( + start_ms=round(float(word.start) * 1000), + end_ms=round(float(word.end) * 1000), + text=normalize_transcript_text(word.word), + confidence=float(word.probability) if getattr(word, "probability", None) is not None else None, + ) + for word in (getattr(segment, "words", None) or []) + if normalize_transcript_text(getattr(word, "word", "")) ) - for segment in raw_segments - if normalize_transcript_text(segment.text) - ] + avg_logprob = getattr(segment, "avg_logprob", None) + confidence = max(0.0, min(1.0, 2.718281828 ** float(avg_logprob))) if avg_logprob is not None else None + results.append( + TranscriptSegment( + start_seconds=float(segment.start), + end_seconds=float(segment.end), + text=text, + confidence=confidence, + words=words, + ) + ) + return results def _get_whisper_model(model_key: tuple[str, str, str] | None = None): @@ -796,13 +1229,35 @@ def _get_whisper_model(model_key: tuple[str, str, str] | None = None): def _primary_model_key() -> tuple[str, str, str]: + configured_device = settings.transcription_device.strip().lower() + device = _detect_transcription_device() if configured_device == "auto" else configured_device + configured_compute = settings.transcription_compute_type.strip().lower() + compute_type = ("float16" if device == "cuda" else "int8") if configured_compute == "auto" else configured_compute return ( settings.transcription_model, - settings.transcription_device, - settings.transcription_compute_type, + device, + compute_type, ) +def _detect_transcription_device() -> str: + """仅在显式配置 TRANSCRIPTION_DEVICE=auto 时检测 CUDA;cpu 配置永不覆盖。""" + executable = shutil.which("nvidia-smi") + if not executable: + return "cpu" + try: + result = subprocess.run( + [executable, "--query-gpu=name", "--format=csv,noheader"], + capture_output=True, + text=True, + timeout=10, + check=False, + ) + except (OSError, subprocess.TimeoutExpired): + return "cpu" + return "cuda" if result.returncode == 0 and result.stdout.strip() else "cpu" + + def _cpu_fallback_model_key() -> tuple[str, str, str]: return ( settings.transcription_cpu_fallback_model, diff --git a/app/services/transcript_workflow_service.py b/app/services/transcript_workflow_service.py index 1b011dc..25358aa 100644 --- a/app/services/transcript_workflow_service.py +++ b/app/services/transcript_workflow_service.py @@ -9,9 +9,11 @@ from app.core.config import settings from app.models.task import TaskStatus +from app.services import job_service from app.services.storage_service import get_artifact_paths, get_source_video_path, validate_source_video_path from app.services.task_log_service import append_task_log from app.services.transcript_service import ( + TranscriptCancelledError, cleanup_transcript_chunk_dirs, read_transcript_preview, read_transcript_progress, @@ -26,10 +28,6 @@ _CANCEL_TRANSCRIPT_TASKS: set[str] = set() -class TranscriptCancelledError(RuntimeError): - pass - - _TRANSCRIPT_STALE_AFTER = timedelta(minutes=10) _DEFAULT_REMOTE_TRANSCRIPTION_PROVIDER = "volcengine" @@ -152,7 +150,7 @@ def get_task_transcript_status(task_id: str) -> dict: # ---------- 音频提取 ---------- -def process_task_audio(task_id: str) -> dict: +def process_task_audio(task_id: str, job_id: str | None = None) -> dict: from app.services.task_service import get_task, update_task_status # noqa: F811 task = get_task(task_id) @@ -169,7 +167,25 @@ def process_task_audio(task_id: str) -> dict: update_task_status(task_id, TaskStatus.audio_extracting) append_task_log(task_id, "开始使用 FFmpeg 提取音频") try: - result = run_ffmpeg_audio_extract(source_path, paths["audio_path"]) + cancel_check = None + audio_progress_callback = None + if job_id: + def cancel_check() -> bool: + return job_service.is_cancel_requested(job_id) + + def audio_progress_callback(percent: int) -> None: + job_service.update_job_progress( + job_id, max(1, min(20, round(percent * 0.2))), f"正在提取音频:{percent}%" + ) + job_service.heartbeat_job(job_id) + result = run_ffmpeg_audio_extract( + source_path, + paths["audio_path"], + cancel_check=cancel_check, + progress_callback=audio_progress_callback, + ) + except job_service.JobLeaseLostError: + raise except Exception as exc: error = str(exc) update_task_status(task_id, TaskStatus.failed, error) @@ -187,6 +203,8 @@ def process_task_transcript( task_id: str, background_tasks: Any | None = None, provider: str | None = None, + job_id: str | None = None, + allow_uncertain_retry: bool = False, ) -> dict: from app.services.task_service import get_task, update_task_status # noqa: F811 @@ -229,9 +247,15 @@ def process_task_transcript( _CANCEL_TRANSCRIPT_TASKS.discard(task_id) _RUNNING_TRANSCRIPT_TASKS.add(task_id) if background_tasks is not None: - background_tasks.add_task(_run_task_transcript_background, task_id, provider_name) + background_tasks.add_task( + _run_task_transcript_background, + task_id, + provider_name, + job_id, + allow_uncertain_retry, + ) else: - _run_task_transcript_background(task_id, provider_name) + _run_task_transcript_background(task_id, provider_name, job_id, allow_uncertain_retry) return { "status": "started", "message": f"已开始{provider_label}分段转写,请稍后刷新查看进度。", @@ -285,6 +309,7 @@ def process_task_transcript_workflow( background_tasks: Any | None = None, force: bool = False, provider: str | None = None, + job_id: str | None = None, ) -> dict: from app.services.task_service import get_task # noqa: F811 @@ -294,23 +319,36 @@ def process_task_transcript_workflow( paths = get_artifact_paths(task_id) if paths["transcript_path"].exists() and not force: - return { - "status": "completed", - "message": "转写 Markdown 已经生成,无需重复处理。如需重做,请点击\"重新生成转写\"。", - "task": get_task(task_id), - } + if read_transcript_preview(paths["transcript_path"], max_lines=1): + return { + "status": "completed", + "message": "转写 Markdown 已经生成,无需重复处理。如需重做,请点击\"重新生成转写\"。", + "task": get_task(task_id), + } + append_task_log(task_id, "检测到空白、截断或不可解析的转写 Markdown,本次不会误判为已完成") if not paths["audio_path"].exists(): append_task_log(task_id, "一键处理:未发现音频文件,先自动提取音频") - process_task_audio(task_id) + process_task_audio(task_id, job_id=job_id) if force: append_task_log(task_id, "用户明确要求重新生成转写 Markdown") - return process_task_transcript(task_id, background_tasks=background_tasks, provider=provider) + return process_task_transcript( + task_id, + background_tasks=background_tasks, + provider=provider, + job_id=job_id, + allow_uncertain_retry=force, + ) -def _run_task_transcript_background(task_id: str, provider: str | None = None) -> None: +def _run_task_transcript_background( + task_id: str, + provider: str | None = None, + job_id: str | None = None, + allow_uncertain_retry: bool = False, +) -> None: from app.services.task_service import get_task, update_task_status # noqa: F811 task = get_task(task_id) @@ -322,6 +360,13 @@ def _run_task_transcript_background(task_id: str, provider: str | None = None) - provider_label = _transcription_choice_label(provider_name) def progress_callback(progress: dict) -> None: + if job_id: + if job_service.is_cancel_requested(job_id): + raise TranscriptCancelledError("用户已停止当前转写任务") + percent = int(progress.get("percent") or 0) + job_percent = 20 + round(max(0, min(100, percent)) * 0.79) + job_service.update_job_progress(job_id, job_percent, str(progress.get("message") or "转写中")) + job_service.heartbeat_job(job_id) if task_id in _CANCEL_TRANSCRIPT_TASKS: raise TranscriptCancelledError("用户已停止当前转写任务") message = progress.get("message") or "转写进度已更新" @@ -342,7 +387,10 @@ def progress_callback(progress: dict) -> None: paths["transcript_path"], progress_callback=progress_callback, provider=provider_name, + allow_uncertain_retry=allow_uncertain_retry, ) + except job_service.JobLeaseLostError: + raise except TranscriptCancelledError as exc: last_progress = read_transcript_progress(paths["transcript_path"]) write_transcript_progress( diff --git a/app/services/transcription_checkpoint_service.py b/app/services/transcription_checkpoint_service.py new file mode 100644 index 0000000..98dbc93 --- /dev/null +++ b/app/services/transcription_checkpoint_service.py @@ -0,0 +1,309 @@ +"""SQLite 转写分块 checkpoint。 + +每个块成功后独立提交;进程失败或重启时只读取同一源指纹和同一运行配置下的成功块。 +""" + +from __future__ import annotations + +import hashlib +import json +from dataclasses import asdict +from datetime import datetime, timezone +from pathlib import Path +from uuid import uuid4 + +from app.db.database import get_connection +from app.services import job_service + + +def _now_iso() -> str: + return datetime.now(timezone.utc).isoformat(timespec="seconds") + + +def _assert_current_lease(connection) -> None: + active = job_service.current_job_lease() + if active is None: + return + job_id, lease_owner, lease_token = active + row = connection.execute( + """ + SELECT 1 FROM workflow_jobs + WHERE id = ? AND status = 'running' AND lease_owner = ? AND lease_token = ? + AND lease_expires_at > ? + """, + (job_id, lease_owner, lease_token, _now_iso()), + ).fetchone() + if not row: + raise job_service.JobLeaseLostError(f"Workflow Job 租约已失效:{job_id}") + + +def fingerprint_file(path_value: str | Path) -> str: + path = Path(path_value).resolve() + digest = hashlib.sha256() + size = path.stat().st_size + digest.update(str(size).encode("ascii")) + with path.open("rb") as source: + digest.update(source.read(1024 * 1024)) + if size > 1024 * 1024: + source.seek(max(0, size - 1024 * 1024)) + digest.update(source.read(1024 * 1024)) + return digest.hexdigest() + + +class TranscriptionCheckpoint: + def __init__( + self, + *, + task_id: str, + source_path: str | Path, + provider: str, + model: str, + device: str, + compute_type: str, + chunk_seconds: int, + overlap_seconds: int, + allow_uncertain_retry: bool = False, + ) -> None: + self.task_id = task_id + self.source_fingerprint = fingerprint_file(source_path) + self.provider = provider + self.model = model + self.device = device + self.compute_type = compute_type + self.chunk_seconds = chunk_seconds + self.overlap_seconds = overlap_seconds + self.allow_uncertain_retry = allow_uncertain_retry + self.run_id = "" + + def ensure_run(self, chunks) -> str: + if self.run_id: + return self.run_id + now = _now_iso() + with get_connection() as connection: + connection.execute("BEGIN IMMEDIATE") + _assert_current_lease(connection) + uncertain = connection.execute( + """ + SELECT id FROM transcription_runs + WHERE task_id = ? AND source_fingerprint = ? AND provider = ? AND model = ? + AND device = ? AND compute_type = ? AND chunk_seconds = ? AND overlap_seconds = ? + AND status = 'uncertain' + ORDER BY updated_at DESC LIMIT 1 + """, + ( + self.task_id, self.source_fingerprint, self.provider, self.model, + self.device, self.compute_type, self.chunk_seconds, self.overlap_seconds, + ), + ).fetchone() + if uncertain and not self.allow_uncertain_retry: + connection.rollback() + raise RemoteTranscriptionResultUncertainError( + "上次远程转写请求可能已计费,但结果未可靠保存;" + "普通任务重试不会再次请求。请由用户明确选择重新生成转写。" + ) + row = connection.execute( + """ + SELECT id FROM transcription_runs + WHERE task_id = ? AND source_fingerprint = ? AND provider = ? AND model = ? + AND device = ? AND compute_type = ? AND chunk_seconds = ? AND overlap_seconds = ? + AND status IN ('processing', 'failed', 'completed') + ORDER BY updated_at DESC LIMIT 1 + """, + ( + self.task_id, self.source_fingerprint, self.provider, self.model, + self.device, self.compute_type, self.chunk_seconds, self.overlap_seconds, + ), + ).fetchone() + connection.execute( + "UPDATE transcription_runs SET is_active = 0 WHERE task_id = ?", + (self.task_id,), + ) + if row: + self.run_id = row["id"] + connection.execute( + "UPDATE transcription_runs SET status = 'processing', is_active = 1, error_message = NULL, total_chunks = ?, updated_at = ? WHERE id = ?", + (len(chunks), now, self.run_id), + ) + else: + self.run_id = uuid4().hex + connection.execute( + """ + INSERT INTO transcription_runs ( + id, task_id, source_fingerprint, provider, model, device, compute_type, + chunk_seconds, overlap_seconds, status, total_chunks, completed_chunks, + is_active, error_message, created_at, updated_at, completed_at + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, 'processing', ?, 0, 1, NULL, ?, ?, NULL) + """, + ( + self.run_id, self.task_id, self.source_fingerprint, self.provider, + self.model, self.device, self.compute_type, self.chunk_seconds, + self.overlap_seconds, len(chunks), now, now, + ), + ) + for chunk in chunks: + connection.execute( + """ + INSERT OR IGNORE INTO transcription_chunks ( + id, run_id, task_id, chunk_index, start_ms, end_ms, status, + attempt_count, result_json, result_checksum, error_message, created_at, updated_at + ) VALUES (?, ?, ?, ?, ?, ?, 'queued', 0, NULL, NULL, NULL, ?, ?) + """, + ( + uuid4().hex, self.run_id, self.task_id, chunk.index, + round(chunk.start_seconds * 1000), round(chunk.end_seconds * 1000), now, now, + ), + ) + connection.commit() + return self.run_id + + def load_completed(self, chunk_index: int, segment_factory) -> list | None: + with get_connection() as connection: + _assert_current_lease(connection) + row = connection.execute( + "SELECT result_json, result_checksum FROM transcription_chunks WHERE run_id = ? AND chunk_index = ? AND status = 'completed'", + (self.run_id, chunk_index), + ).fetchone() + if not row or not row["result_json"]: + return None + raw = str(row["result_json"]) + if hashlib.sha256(raw.encode("utf-8")).hexdigest() != str(row["result_checksum"] or ""): + self._invalidate_completed_chunk(chunk_index, "checkpoint checksum 不一致,已重新排队") + return None + try: + payload = json.loads(raw) + if not isinstance(payload, list): + raise ValueError("checkpoint 顶层不是数组") + return [segment_factory(item) for item in payload if isinstance(item, dict)] + except (json.JSONDecodeError, TypeError, ValueError, KeyError) as exc: + self._invalidate_completed_chunk(chunk_index, f"checkpoint 内容损坏,已重新排队:{exc}") + return None + + def _invalidate_completed_chunk(self, chunk_index: int, error: str) -> None: + with get_connection() as connection: + connection.execute("BEGIN IMMEDIATE") + _assert_current_lease(connection) + connection.execute( + """ + UPDATE transcription_chunks + SET status = 'queued', result_json = NULL, result_checksum = NULL, + error_message = ?, updated_at = ? + WHERE run_id = ? AND chunk_index = ? AND status = 'completed' + """, + (error[:2000], _now_iso(), self.run_id, chunk_index), + ) + connection.commit() + + def prepare_remote_request(self, chunk_index: int) -> str: + """在远程副作用前落账;遗留 requesting 表示上次结果不确定,禁止自动重发。""" + now = _now_iso() + request_id = hashlib.sha256( + f"{self.run_id}:{self.source_fingerprint}:{chunk_index}".encode("utf-8") + ).hexdigest()[:32] + with get_connection() as connection: + connection.execute("BEGIN IMMEDIATE") + _assert_current_lease(connection) + row = connection.execute( + "SELECT status FROM transcription_chunks WHERE run_id = ? AND chunk_index = ?", + (self.run_id, chunk_index), + ).fetchone() + if not row: + connection.rollback() + raise RuntimeError(f"找不到远程转写分片 checkpoint:{chunk_index}") + if row["status"] in {"requesting", "uncertain"}: + connection.rollback() + raise RemoteTranscriptionResultUncertainError( + f"第 {chunk_index} 段上次远程请求已发出但没有可靠结果;" + "本次未自动重发,请由用户明确重新生成转写。" + ) + connection.execute( + """ + UPDATE transcription_chunks + SET status = 'requesting', attempt_count = attempt_count + 1, + error_message = ?, updated_at = ? + WHERE run_id = ? AND chunk_index = ? + """, + (f"request_id={request_id}", now, self.run_id, chunk_index), + ) + connection.commit() + return request_id + + def save_completed(self, chunk_index: int, segments, *, attempt_already_counted: bool = False) -> None: + now = _now_iso() + raw = json.dumps([asdict(segment) for segment in segments], ensure_ascii=False, separators=(",", ":")) + checksum = hashlib.sha256(raw.encode("utf-8")).hexdigest() + with get_connection() as connection: + connection.execute("BEGIN IMMEDIATE") + _assert_current_lease(connection) + connection.execute( + """ + UPDATE transcription_chunks + SET status = 'completed', attempt_count = attempt_count + ?, result_json = ?, + result_checksum = ?, error_message = NULL, updated_at = ? + WHERE run_id = ? AND chunk_index = ? + """, + (0 if attempt_already_counted else 1, raw, checksum, now, self.run_id, chunk_index), + ) + completed = connection.execute( + "SELECT COUNT(*) FROM transcription_chunks WHERE run_id = ? AND status = 'completed'", + (self.run_id,), + ).fetchone()[0] + connection.execute( + "UPDATE transcription_runs SET completed_chunks = ?, updated_at = ? WHERE id = ?", + (completed, now, self.run_id), + ) + connection.commit() + + def save_failed(self, chunk_index: int, error: str, *, attempt_already_counted: bool = False) -> None: + now = _now_iso() + with get_connection() as connection: + connection.execute("BEGIN IMMEDIATE") + _assert_current_lease(connection) + connection.execute( + """UPDATE transcription_chunks SET status = 'failed', attempt_count = attempt_count + ?, + error_message = ?, updated_at = ? WHERE run_id = ? AND chunk_index = ?""", + (0 if attempt_already_counted else 1, error[:2000], now, self.run_id, chunk_index), + ) + connection.execute( + "UPDATE transcription_runs SET status = 'failed', error_message = ?, updated_at = ? WHERE id = ?", + (error[:2000], now, self.run_id), + ) + connection.commit() + + def save_uncertain(self, chunk_index: int, error: str) -> None: + now = _now_iso() + with get_connection() as connection: + connection.execute("BEGIN IMMEDIATE") + _assert_current_lease(connection) + connection.execute( + """ + UPDATE transcription_chunks + SET status = 'uncertain', error_message = ?, updated_at = ? + WHERE run_id = ? AND chunk_index = ? + """, + (error[:2000], now, self.run_id, chunk_index), + ) + connection.execute( + """ + UPDATE transcription_runs + SET status = 'uncertain', error_message = ?, updated_at = ? + WHERE id = ? + """, + (error[:2000], now, self.run_id), + ) + connection.commit() + + def complete(self) -> None: + now = _now_iso() + with get_connection() as connection: + connection.execute("BEGIN IMMEDIATE") + _assert_current_lease(connection) + connection.execute( + """UPDATE transcription_runs SET status = 'completed', is_active = 1, + error_message = NULL, completed_at = ?, updated_at = ? WHERE id = ?""", + (now, now, self.run_id), + ) + connection.commit() + + +class RemoteTranscriptionResultUncertainError(RuntimeError): + """远程调用已发出但结果未可靠持久化,必须停止自动重试。""" diff --git a/app/services/video_cut_service.py b/app/services/video_cut_service.py index a41295e..e303f60 100644 --- a/app/services/video_cut_service.py +++ b/app/services/video_cut_service.py @@ -182,26 +182,87 @@ def cut_single_clip( strategy: str = "accurate", ) -> CutResult: plan.output_path.parent.mkdir(parents=True, exist_ok=True) - command = build_ffmpeg_cut_command(ffmpeg_path, source_video, plan, strategy=strategy) - result = subprocess.run(command, capture_output=True, text=True, encoding="utf-8", errors="replace", - timeout=settings.ffmpeg_cut_timeout) + plan.output_path.unlink(missing_ok=True) + temporary_path = plan.output_path.with_name(f"{plan.output_path.stem}.part{plan.output_path.suffix}") + temporary_path.unlink(missing_ok=True) + temporary_plan = CutPlan( + clip_candidate_id=plan.clip_candidate_id, + title=plan.title, + start_time=plan.start_time, + end_time=plan.end_time, + output_path=temporary_path, + duration_seconds=plan.duration_seconds, + ) + command = build_ffmpeg_cut_command(ffmpeg_path, source_video, temporary_plan, strategy=strategy) + try: + result = subprocess.run( + command, + capture_output=True, + text=True, + encoding="utf-8", + errors="replace", + timeout=settings.ffmpeg_cut_timeout, + check=False, + ) + except subprocess.TimeoutExpired: + temporary_path.unlink(missing_ok=True) + return CutResult( + clip_candidate_id=plan.clip_candidate_id, + output_file_path="", + output_file_name="", + status="failed", + error_message=f"FFmpeg 切片超过 {settings.ffmpeg_cut_timeout} 秒,已清理未完成文件", + ) + except OSError as exc: + temporary_path.unlink(missing_ok=True) + return CutResult( + clip_candidate_id=plan.clip_candidate_id, + output_file_path="", + output_file_name="", + status="failed", + error_message=f"FFmpeg 无法启动:{exc}", + ) if result.returncode != 0: + temporary_path.unlink(missing_ok=True) return CutResult( clip_candidate_id=plan.clip_candidate_id, - output_file_path=str(plan.output_path), - output_file_name=plan.output_path.name, + output_file_path="", + output_file_name="", status="failed", error_message=summarize_stderr(result.stderr), ) - if not plan.output_path.exists() or plan.output_path.stat().st_size == 0: + try: + valid_output = temporary_path.exists() and temporary_path.stat().st_size > 0 + except OSError as exc: + temporary_path.unlink(missing_ok=True) + return CutResult( + clip_candidate_id=plan.clip_candidate_id, + output_file_path="", + output_file_name="", + status="failed", + error_message=f"FFmpeg 输出文件无法校验:{exc}", + ) + if not valid_output: + temporary_path.unlink(missing_ok=True) return CutResult( clip_candidate_id=plan.clip_candidate_id, - output_file_path=str(plan.output_path), - output_file_name=plan.output_path.name, + output_file_path="", + output_file_name="", status="failed", error_message="FFmpeg 已结束,但没有生成有效的视频文件", ) + try: + temporary_path.replace(plan.output_path) + except OSError as exc: + temporary_path.unlink(missing_ok=True) + return CutResult( + clip_candidate_id=plan.clip_candidate_id, + output_file_path="", + output_file_name="", + status="failed", + error_message=f"FFmpeg 输出文件切换失败,已清理未完成文件:{exc}", + ) return CutResult( clip_candidate_id=plan.clip_candidate_id, diff --git a/app/services/video_cut_workflow_service.py b/app/services/video_cut_workflow_service.py index 881cef5..0ebf025 100644 --- a/app/services/video_cut_workflow_service.py +++ b/app/services/video_cut_workflow_service.py @@ -4,20 +4,29 @@ 支持 cut_run 版本化:每次切片创建新的 run,成功后才切换 active,失败保留旧结果。 """ +from pathlib import Path +import shutil +from sqlite3 import Connection from uuid import uuid4 from app.core.config import settings from app.models.task import TaskStatus from app.services.storage_service import get_artifact_paths, get_source_video_path, validate_source_video_path from app.services.task_log_service import append_task_log -from app.services.video_cut_service import CutResult, cut_clips +from app.services.video_cut_service import CutResult, cut_clips, parse_time_to_seconds # ---------- Cut Run 数据库操作 ---------- -def _next_cut_run_number(task_id: str) -> int: +def _next_cut_run_number(task_id: str, connection: Connection | None = None) -> int: from app.db.database import get_connection + if connection is not None: + row = connection.execute( + "SELECT COALESCE(MAX(run_number), 0) AS max_run_number FROM cut_runs WHERE task_id = ?", + (task_id,), + ).fetchone() + return int(row["max_run_number"] or 0) + 1 with get_connection() as connection: row = connection.execute( "SELECT COALESCE(MAX(run_number), 0) AS max_run_number FROM cut_runs WHERE task_id = ?", @@ -29,12 +38,31 @@ def _next_cut_run_number(task_id: str) -> int: def _create_cut_run(task_id: str) -> dict: """创建新的切割 run,状态为 processing,is_active=0""" from app.db.database import get_connection - from app.services.task_service import _now_iso + from app.services import job_service + from app.services.task_service import _now_iso, STATUS_PROGRESS - now = _now_iso() run_id = uuid4().hex[:12] - run_number = _next_cut_run_number(task_id) + active_lease = job_service.current_job_lease() with get_connection() as connection: + connection.execute("BEGIN IMMEDIATE") + now = _now_iso() + if active_lease: + workflow_job_id, lease_owner, lease_token = active_lease + valid_lease = connection.execute( + """ + SELECT 1 FROM workflow_jobs + WHERE id = ? AND task_id = ? AND status = 'running' + AND lease_owner = ? AND lease_token = ? + AND lease_expires_at > ? AND cancel_requested = 0 + """, + (workflow_job_id, task_id, lease_owner, lease_token, now), + ).fetchone() + if not valid_lease: + connection.rollback() + raise job_service.JobLeaseLostError( + f"Workflow Job 租约已失效,拒绝创建切片批次:{workflow_job_id}" + ) + run_number = _next_cut_run_number(task_id, connection) connection.execute( """ INSERT INTO cut_runs (id, task_id, run_number, status, is_active, created_at, updated_at) @@ -42,10 +70,48 @@ def _create_cut_run(task_id: str) -> dict: """, (run_id, task_id, run_number, now, now), ) + final_now = _now_iso() + lease_condition = "" + lease_params: tuple[str, ...] = () + if active_lease: + workflow_job_id, lease_owner, lease_token = active_lease + lease_condition = """ + AND EXISTS ( + SELECT 1 FROM workflow_jobs + WHERE id = ? AND task_id = ? AND status = 'running' + AND lease_owner = ? AND lease_token = ? + AND lease_expires_at > ? AND cancel_requested = 0 + ) + """ + lease_params = (workflow_job_id, task_id, lease_owner, lease_token, final_now) + cursor = connection.execute( + f""" + UPDATE tasks + SET status = ?, progress = ?, error_message = NULL, last_error = NULL, updated_at = ? + WHERE id = ? AND COALESCE(is_deleted, 0) = 0 AND status != ? + {lease_condition} + """, + ( + TaskStatus.cutting.value, + STATUS_PROGRESS.get(TaskStatus.cutting.value, 0), + final_now, + task_id, + TaskStatus.CANCELLED.value, + *lease_params, + ), + ) + if cursor.rowcount != 1: + connection.rollback() + raise RuntimeError("任务已取消、删除或不存在,不能创建新的切片批次") connection.commit() return {"id": run_id, "run_number": run_number, "task_id": task_id} +def _cut_run_output_dir(clips_dir: Path, cut_run: dict) -> Path: + """每个 cut_run 使用独立目录,重试或并发不会覆盖同名媒体。""" + return clips_dir / f"run_{int(cut_run['run_number']):04d}_{cut_run['id']}" + + def _activate_cut_run(task_id: str, run_id: str) -> None: """激活指定的 cut_run,同时将该 task 下其他 run 和旧 output_clip 标记为非活跃""" from app.db.database import get_connection @@ -79,42 +145,334 @@ def _fail_cut_run(run_id: str, error_message: str = "") -> None: now = _now_iso() with get_connection() as connection: connection.execute( - "UPDATE cut_runs SET status = 'failed', error_message = ?, updated_at = ? WHERE id = ?", + """ + UPDATE cut_runs + SET status = 'failed', error_message = ?, updated_at = ? + WHERE id = ? AND status = 'processing' + """, (error_message, now, run_id), ) connection.commit() -# ---------- Output Clip 数据库操作 ---------- - -def _insert_output_clip_record(task_id: str, cut_run_id: str, result: CutResult) -> None: +def _finalize_task_for_cut_run( + task_id: str, + cut_run_id: str, + status: TaskStatus, + error_message: str | None, + *, + connection: Connection | None = None, +) -> bool: + """仅允许最新 cut_run 在有效 Workflow Job 代际内写回 Task 主状态。""" from app.db.database import get_connection - from app.services.task_service import _now_iso + from app.services import job_service + from app.services.task_service import _now_iso, STATUS_PROGRESS + + if connection is None: + with get_connection() as owned_connection: + owned_connection.execute("BEGIN IMMEDIATE") + try: + finalized = _finalize_task_for_cut_run( + task_id, + cut_run_id, + status, + error_message, + connection=owned_connection, + ) + owned_connection.commit() + return finalized + except Exception: + owned_connection.rollback() + raise now = _now_iso() - with get_connection() as connection: - connection.execute( - """ - INSERT INTO output_clip ( - id, task_id, clip_candidate_id, output_file_path, output_file_name, - status, error_message, cut_run_id, is_active, created_at, updated_at + active_lease = job_service.current_job_lease() + lease_condition = "" + lease_params: tuple[str, ...] = () + if active_lease: + workflow_job_id, lease_owner, lease_token = active_lease + lease_condition = """ + AND EXISTS ( + SELECT 1 FROM workflow_jobs + WHERE id = ? AND task_id = ? AND status = 'running' + AND lease_owner = ? AND lease_token = ? + AND lease_expires_at > ? AND cancel_requested = 0 ) - VALUES (?, ?, ?, ?, ?, ?, ?, ?, 1, ?, ?) + """ + lease_params = (workflow_job_id, task_id, lease_owner, lease_token, now) + cursor = connection.execute( + f""" + UPDATE tasks + SET status = ?, progress = ?, error_message = ?, last_error = ?, updated_at = ? + WHERE id = ? AND COALESCE(is_deleted, 0) = 0 AND status = ? + AND EXISTS ( + SELECT 1 FROM cut_runs current_run + WHERE current_run.id = ? AND current_run.task_id = ? + AND NOT EXISTS ( + SELECT 1 FROM cut_runs newer_run + WHERE newer_run.task_id = current_run.task_id + AND newer_run.run_number > current_run.run_number + ) + ) + {lease_condition} """, ( - uuid4().hex[:12], + status.value, + STATUS_PROGRESS.get(status.value, 0), + error_message, + error_message, + now, task_id, - result.clip_candidate_id, - result.output_file_path, - result.output_file_name, - result.status, - result.error_message, + TaskStatus.cutting.value, cut_run_id, - now, - now, + task_id, + *lease_params, ), ) - connection.commit() + return cursor.rowcount == 1 + + +# ---------- Output Clip 数据库操作 ---------- + +def _insert_output_clip_record( + task_id: str, + cut_run_id: str, + result: CutResult, + *, + source_fingerprint: str = "", + connection: Connection | None = None, + is_active: bool = True, +) -> None: + from app.db.database import get_connection + from app.services.task_service import _now_iso + + now = _now_iso() + if connection is None: + with get_connection() as owned_connection: + _insert_output_clip_record_on_connection( + owned_connection, + task_id, + cut_run_id, + result, + source_fingerprint=source_fingerprint, + is_active=is_active, + now=now, + ) + owned_connection.commit() + return + _insert_output_clip_record_on_connection( + connection, + task_id, + cut_run_id, + result, + source_fingerprint=source_fingerprint, + is_active=is_active, + now=now, + ) + + +def _insert_output_clip_record_on_connection( + connection: Connection, + task_id: str, + cut_run_id: str, + result: CutResult, + *, + source_fingerprint: str, + is_active: bool, + now: str, +) -> None: + candidate = connection.execute( + "SELECT start_time, end_time FROM clip_candidates WHERE id = ? AND task_id = ?", + (result.clip_candidate_id, task_id), + ).fetchone() + source_start_ms = None + source_end_ms = None + snapshot_source = "legacy_inferred" + if candidate: + source_start_ms = round(parse_time_to_seconds(candidate["start_time"]) * 1000) + source_end_ms = round(parse_time_to_seconds(candidate["end_time"]) * 1000) + snapshot_source = "cut_commit" + connection.execute( + """ + INSERT INTO output_clip ( + id, task_id, clip_candidate_id, output_file_path, output_file_name, + status, error_message, cut_run_id, is_active, + source_start_ms, source_end_ms, source_duration_ms, + source_fingerprint, snapshot_source, created_at, updated_at + ) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + """, + ( + uuid4().hex[:12], + task_id, + result.clip_candidate_id, + result.output_file_path, + result.output_file_name, + result.status, + result.error_message, + cut_run_id, + 1 if is_active else 0, + source_start_ms, + source_end_ms, + source_end_ms - source_start_ms if source_start_ms is not None and source_end_ms is not None else None, + source_fingerprint, + snapshot_source, + now, + now, + ), + ) + + +def _commit_cut_run_results( + task_id: str, + cut_run_id: str, + results: list[CutResult], + *, + source_fingerprint: str, + error_message: str = "", +) -> dict[str, bool]: + """原子写入一个批次的所有结果,并按 run_number 决定是否激活。""" + from app.db.database import get_connection + from app.services import job_service + from app.services.task_service import _now_iso + + active_lease = job_service.current_job_lease() + try: + with get_connection() as connection: + connection.execute("BEGIN IMMEDIATE") + now = _now_iso() + if active_lease: + workflow_job_id, lease_owner, lease_token = active_lease + valid_lease = connection.execute( + """ + SELECT 1 FROM workflow_jobs + WHERE id = ? AND task_id = ? AND status = 'running' + AND lease_owner = ? AND lease_token = ? + AND lease_expires_at > ? AND cancel_requested = 0 + """, + (workflow_job_id, task_id, lease_owner, lease_token, now), + ).fetchone() + if not valid_lease: + raise job_service.JobLeaseLostError( + f"Workflow Job 租约已失效,拒绝提交切片批次:{workflow_job_id}" + ) + run = connection.execute( + "SELECT run_number, status, is_active FROM cut_runs WHERE id = ? AND task_id = ?", + (cut_run_id, task_id), + ).fetchone() + if not run: + raise ValueError("切片批次不存在") + if run["status"] in {"completed", "completed_with_errors"}: + newer = connection.execute( + "SELECT 1 FROM cut_runs WHERE task_id = ? AND run_number > ? LIMIT 1", + (task_id, int(run["run_number"])), + ).fetchone() + connection.rollback() + return { + "activated": bool(run["is_active"]), + "is_latest": newer is None, + "already_committed": True, + } + if run["status"] != "processing": + raise RuntimeError(f"切片批次状态不是 processing:{run['status']}") + + for result in results: + _insert_output_clip_record( + task_id, + cut_run_id, + result, + source_fingerprint=source_fingerprint, + connection=connection, + is_active=False, + ) + + has_success = any(result.status == "completed" for result in results) + has_failures = any(result.status != "completed" for result in results) + committed_status = ( + "completed_with_errors" + if has_success and has_failures + else "completed" if has_success else "failed" + ) + newer = connection.execute( + """ + SELECT 1 + FROM cut_runs + WHERE task_id = ? AND run_number > ? + LIMIT 1 + """, + (task_id, int(run["run_number"])), + ).fetchone() + newer_completed = connection.execute( + """ + SELECT 1 + FROM cut_runs + WHERE task_id = ? AND run_number > ? + AND status IN ('completed', 'completed_with_errors') + LIMIT 1 + """, + (task_id, int(run["run_number"])), + ).fetchone() + is_latest = newer is None + activated = has_success and newer_completed is None + + connection.execute( + """ + UPDATE cut_runs + SET status = ?, is_active = ?, error_message = ?, updated_at = ? + WHERE id = ? + """, + ( + committed_status, + 1 if activated else 0, + error_message or None, + now, + cut_run_id, + ), + ) + if activated: + connection.execute( + "UPDATE cut_runs SET is_active = 0, updated_at = ? WHERE task_id = ? AND id != ?", + (now, task_id, cut_run_id), + ) + connection.execute( + """ + UPDATE output_clip + SET is_active = 0 + WHERE task_id = ? AND (cut_run_id IS NULL OR cut_run_id != ?) + """, + (task_id, cut_run_id), + ) + connection.execute( + "UPDATE output_clip SET is_active = 1 WHERE task_id = ? AND cut_run_id = ?", + (task_id, cut_run_id), + ) + final_task_status = ( + TaskStatus.completed_with_errors + if committed_status == "completed_with_errors" + else TaskStatus.completed if committed_status == "completed" else TaskStatus.failed + ) + task_finalized = _finalize_task_for_cut_run( + task_id, + cut_run_id, + final_task_status, + error_message or None, + connection=connection, + ) + connection.commit() + return { + "activated": activated, + "is_latest": is_latest, + "already_committed": False, + "task_finalized": task_finalized, + } + except Exception as exc: + try: + _fail_cut_run(cut_run_id, str(exc)) + except Exception as fail_mark_error: + raise RuntimeError( + f"切片批次提交失败:{exc};失败状态也未能写入:{fail_mark_error}" + ) from exc + raise def _deactivate_output_clips(task_id: str, except_run_id: str | None = None) -> None: @@ -167,6 +525,43 @@ def process_task_video_cuts(task_id: str, *, sync_publish_jobs: bool = True) -> if not task: raise ValueError("任务不存在") + from app.services.ai_analysis_workflow_service import ( + get_task_ai_analysis_meta, + validate_ai_analysis_meta_for_cut, + ) + + meta = get_task_ai_analysis_meta(task_id) + profile = str(task.get("selection_profile") or "general") + if not meta: + error = "AI 分析缺少可信的完整性元数据;请重新分析或恢复可信历史,当前不会生成切片。" + append_task_log(task_id, f"视频切割已阻止:{error}") + raise ValueError(error) + meta = validate_ai_analysis_meta_for_cut(meta, profile) + coverage = float(meta.get("coverage_percent") or 0) + if profile == "long_live_talk" and ( + meta.get("analysis_incomplete") or float(meta.get("coverage_ratio") or 0) < 0.90 + ): + error = ( + f"长直播分析覆盖率仅 {coverage:.2f}%,低于 90%;" + "请先重试 AI 分析补齐缺失窗口,当前不会生成切片或同步发送中心。" + ) + append_task_log(task_id, f"视频切割已阻止:{error}") + raise ValueError(error) + if meta.get("analysis_incomplete"): + error = ( + f"{profile} AI 分析存在未完成单元,当前覆盖率 {coverage:.2f}%;" + "请先重试 AI 分析补齐失败单元,当前不会生成切片或同步发送中心。" + ) + append_task_log(task_id, f"视频切割已阻止:{error}") + raise ValueError(error) + if meta.get("quality_degraded"): + error = ( + f"{profile} AI 分析质量评审未完整通过;" + "候选可供人工检查,但当前不会生成切片或同步发送中心。" + ) + append_task_log(task_id, f"视频切割已阻止:{error}") + raise ValueError(error) + source_path = get_source_video_path(task) valid, error_message = validate_source_video_path(str(source_path) if source_path else None) if not valid: @@ -182,49 +577,61 @@ def process_task_video_cuts(task_id: str, *, sync_publish_jobs: bool = True) -> raise ValueError(error) paths = get_artifact_paths(task_id) - update_task_status(task_id, TaskStatus.cutting) - append_task_log(task_id, f"开始自动切割视频,启用片段数:{len(enabled_clips)}") - # === 版本化:创建新的 cut_run,不删除旧记录 === cut_run = _create_cut_run(task_id) cut_run_id = cut_run["id"] + append_task_log(task_id, f"开始自动切割视频,启用片段数:{len(enabled_clips)}") append_task_log(task_id, f"创建切割批次:第 {cut_run['run_number']} 次切割") try: + from app.services.transcription_checkpoint_service import fingerprint_file + + source_fingerprint = fingerprint_file(source_path) + cut_output_dir = _cut_run_output_dir(paths["clips_dir"], cut_run) results = cut_clips( source_video=source_path, clips=enabled_clips, - output_dir=paths["clips_dir"], + output_dir=cut_output_dir, strategy=settings.default_cut_strategy, ) except Exception as exc: error = str(exc) # 失败时:标记 cut_run 为 failed,保留旧的 active output_clip _fail_cut_run(cut_run_id, error) - update_task_status(task_id, TaskStatus.failed, error) + _finalize_task_for_cut_run(task_id, cut_run_id, TaskStatus.failed, error) append_task_log(task_id, f"视频切割失败:{error}") raise - # 插入新 output_clip 记录,关联到当前 cut_run + final_status, final_error = _resolve_final_cut_status(results) + try: + commit_result = _commit_cut_run_results( + task_id, + cut_run_id, + results, + source_fingerprint=source_fingerprint, + error_message=final_error or "", + ) + except Exception as exc: + error = f"切片文件已生成,但批次结果未能原子写入数据库:{exc}" + try: + if cut_output_dir.exists(): + shutil.rmtree(cut_output_dir) + except OSError as cleanup_error: + error = f"{error};未提交的批次目录清理失败:{cleanup_error}" + _finalize_task_for_cut_run(task_id, cut_run_id, TaskStatus.failed, error) + append_task_log(task_id, error) + raise RuntimeError(error) from exc + for result in results: - _insert_output_clip_record(task_id, cut_run_id, result) if result.status == "completed": append_task_log(task_id, f"切片完成:{result.output_file_name}") else: append_task_log(task_id, f"切片失败:{result.clip_candidate_id},原因:{result.error_message}") - final_status, final_error = _resolve_final_cut_status(results) - publish_sync = None if final_status == TaskStatus.failed: - # 全部失败:不激活新 run,旧 active 保持不变 - _fail_cut_run(cut_run_id, final_error or "全部切片失败") - update_task_status(task_id, final_status, final_error) append_task_log(task_id, "切割批次失败,旧切片结果保留不变") - else: - # 成功或部分成功:激活新 run,旧 output_clip 标记为非活跃 - _activate_cut_run(task_id, cut_run_id) - update_task_status(task_id, final_status, final_error) + elif commit_result["activated"] and commit_result["task_finalized"]: append_task_log(task_id, f"自动切割结束:{get_status_label(final_status.value)}") if sync_publish_jobs: try: @@ -242,12 +649,14 @@ def process_task_video_cuts(task_id: str, *, sync_publish_jobs: bool = True) -> "errors": [str(exc)], } append_task_log(task_id, f"切片完成后的发送中心同步失败:{exc}") + else: + append_task_log(task_id, "切割批次已完成,但任务已取消或已有更新批次,未覆盖当前主状态") return { "status": final_status.value, "status_label": get_status_label(final_status.value), "message": "视频切割流程已完成", - "output_dir": str(paths["clips_dir"]), + "output_dir": str(cut_output_dir), "cut_run_id": cut_run_id, "cut_run_number": cut_run["run_number"], "results": [result.__dict__ for result in results], diff --git a/app/static/css/styles.css b/app/static/css/styles.css index 7caf64f..8ccd4ef 100644 --- a/app/static/css/styles.css +++ b/app/static/css/styles.css @@ -645,6 +645,19 @@ tbody tr:last-child td { font-size: 30px; } +.live-progress-summary { + display: flex; + align-items: baseline; + justify-content: space-between; + gap: 12px; +} + +.live-progress-summary span { + color: var(--muted); + font-size: 12px; + font-weight: 700; +} + .toolbar { flex-wrap: wrap; justify-content: flex-start; @@ -765,6 +778,10 @@ dd { gap: 12px; } +#long-live-settings[hidden] { + display: none; +} + .segmented label, .toggle-line, .inline-switch { @@ -854,6 +871,51 @@ dd { line-height: 1.6; } +.segmented-control { + display: flex; + flex-wrap: wrap; + gap: 10px; + margin-bottom: 14px; +} + +.segmented-control label { + display: inline-flex; + flex-direction: row; + align-items: center; + gap: 8px; + padding: 10px 14px; + border: 1px solid var(--border); + border-radius: 999px; + background: #fff; + cursor: pointer; +} + +.file-browser-list { + display: grid; + max-height: 300px; + overflow: auto; + border: 1px solid var(--border); + border-radius: 10px; + background: #fff; +} + +.file-browser-item { + width: 100%; + padding: 11px 13px; + border: 0; + border-bottom: 1px solid var(--border); + background: transparent; + color: var(--text); + text-align: left; + cursor: pointer; +} + +.file-browser-item:hover, +.file-browser-item.is-selected { + background: rgba(37, 111, 255, 0.08); + color: var(--blue); +} + .file-upload-button { display: inline-flex; grid-auto-flow: column; @@ -1065,6 +1127,65 @@ fieldset input.visually-hidden-file { min-height: 210px; } +.live-current-operation { + display: grid; + grid-template-columns: minmax(0, 1fr) auto; + gap: 5px 14px; + align-items: center; + margin-top: 14px; + padding: 12px 14px; + border: 1px solid rgba(37, 111, 255, 0.14); + border-radius: 8px; + background: rgba(37, 111, 255, 0.06); +} + +.live-current-operation div { + display: flex; + flex-wrap: wrap; + gap: 6px 10px; + align-items: baseline; + min-width: 0; +} + +.live-current-operation div span { + color: var(--muted); + font-size: 12px; +} + +.live-current-operation > strong { + color: var(--blue); + font-size: 18px; +} + +.live-current-operation p { + grid-column: 1 / -1; + margin: 0; + color: var(--muted); + font-size: 12px; + line-height: 1.55; +} + +.live-current-operation[data-status="attention"], +.live-current-operation[data-status="failed"] { + border-color: rgba(230, 159, 0, 0.24); + background: rgba(230, 159, 0, 0.08); +} + +.live-current-operation[data-status="attention"] > strong, +.live-current-operation[data-status="failed"] > strong { + color: var(--amber); +} + +[data-task-live-status-pill][data-status="running"] { + background: var(--blue-soft); + color: var(--blue); +} + +[data-task-live-status-pill][data-status="failed"] { + background: var(--amber-soft); + color: var(--amber); +} + .metro-timeline { position: relative; display: grid; @@ -1688,6 +1809,10 @@ button:disabled { grid-template-columns: repeat(5, minmax(150px, 1fr)); } +.clip-review-stat-grid { + grid-template-columns: repeat(3, minmax(180px, 1fr)); +} + .review-board { display: grid; gap: 12px; @@ -3933,6 +4058,15 @@ body.transcript-drawer-open .main-panel { grid-template-columns: repeat(2, minmax(0, 1fr)); } + .live-current-operation { + grid-template-columns: 1fr; + } + + .live-current-operation > strong, + .live-current-operation p { + grid-column: 1; + } + .segmented, .source-grid, .upload-card, @@ -4889,6 +5023,33 @@ td a, .system-chip { display: none; } + + .review-layout { + grid-template-columns: minmax(0, 1fr); + } + + .review-layout > * { + min-width: 0; + } + + .clip-list > * { + min-width: 0; + } + + .clip-card { + grid-template-columns: minmax(0, 1fr); + } + + .clip-card > * { + min-width: 0; + max-width: 100%; + } + + .preview-panel { + position: static; + max-height: none; + overflow: visible; + } } @media (max-width: 860px) { @@ -5175,7 +5336,29 @@ td a, flex-wrap: wrap; } -.publish-content-header [data-backfill-covers][data-loading="true"] { +.publish-content-maintenance { + display: flex; + align-items: center; + justify-content: space-between; + gap: 18px; + margin-bottom: 16px; + padding: 12px 14px; + border: 1px solid rgba(126, 151, 187, 0.18); + border-radius: 14px; + background: rgba(247, 250, 255, 0.78); +} + +.publish-content-maintenance > div:first-child { + display: grid; + gap: 3px; +} + +.publish-content-maintenance small, +.publish-copy-toolbar small { + color: var(--muted); +} + +.publish-content-maintenance [data-backfill-covers][data-loading="true"] { cursor: wait; opacity: 0.72; } @@ -5280,6 +5463,23 @@ td a, flex-wrap: wrap; gap: 10px; } + +.publish-copy-toolbar { + display: flex; + align-items: center; + justify-content: space-between; + gap: 14px; + margin-bottom: 12px; + padding: 10px 12px; + border-radius: 12px; + background: rgba(238, 245, 255, 0.7); +} + +.publish-copy-toolbar > div { + display: grid; + gap: 2px; +} + .publish-schedule-badge { display: inline-flex; align-items: center; @@ -5671,6 +5871,12 @@ td a, } @media (max-width: 760px) { + .publish-content-maintenance, + .publish-copy-toolbar { + align-items: stretch; + flex-direction: column; + } + .publish-copy-toolbar .secondary-button { width: 100%; } .publish-content-header-actions { width: 100%; justify-content: stretch; @@ -5719,3 +5925,576 @@ td a, .publish-history-view-switch .secondary-button { flex: 1; } .publish-history-toolbar-actions .compact-filter { width: 100%; } } + +/* ── Subtitle editor v2: 原片主时间轴 + 切片继承 ── */ +.subtitle-editor { + margin-bottom: 24px; + overflow: hidden; +} +.subtitle-editor-heading, +.subtitle-editor-trackbar, +.subtitle-editor-toolbar, +.subtitle-waveform-toolbar, +.subtitle-editor-state { + display: flex; + align-items: center; + gap: 10px; +} +.subtitle-editor-heading { align-items: flex-start; } +.subtitle-editor-heading > div:first-child { flex: 1; } +.subtitle-editor-heading h2 { margin-bottom: 5px; } +.subtitle-editor-heading p:last-child { margin: 0; color: var(--muted); } +.subtitle-editor-state { justify-content: flex-end; flex-wrap: wrap; } +.subtitle-editor-trackbar { + justify-content: space-between; + flex-wrap: wrap; + margin: 18px 0 14px; + padding: 13px 14px; + border: 1px solid rgba(126, 151, 187, 0.2); + border-radius: 15px; + background: rgba(245, 249, 255, 0.78); +} +.subtitle-editor-trackbar > label { display: grid; gap: 5px; min-width: min(380px, 100%); } +.subtitle-editor-trackbar > label > span, +.subtitle-editor-toolbar label > span, +.subtitle-cue-time-inputs span, +.subtitle-speaker span { + color: var(--muted); + font-size: 0.72rem; + font-weight: 750; +} +.subtitle-editor-revision-meta { flex: 1; min-width: 230px; color: var(--muted); font-size: 0.82rem; } +.subtitle-import-button { cursor: pointer; } +.subtitle-editor-media-grid { + display: grid; + grid-template-columns: minmax(280px, 0.72fr) minmax(420px, 1.28fr); + gap: 14px; +} +.subtitle-video-stage, +.subtitle-waveform-card { + position: relative; + overflow: hidden; + min-width: 0; + border: 1px solid rgba(126, 151, 187, 0.22); + border-radius: 16px; + background: #101521; +} +.subtitle-video-stage { + display: grid; + place-items: center; + min-height: 248px; +} +.subtitle-video-stage video { width: 100%; max-height: 390px; background: #080b12; } +.subtitle-video-overlay { + position: absolute; + left: 6%; + right: 6%; + bottom: 6%; + min-height: 1.4em; + color: #fff; + font-size: clamp(16px, 2vw, 28px); + font-weight: 800; + line-height: 1.35; + text-align: center; + white-space: pre-line; + text-shadow: 0 2px 4px #000, 1px 0 2px #000, -1px 0 2px #000; + pointer-events: none; +} +.subtitle-waveform-card { align-self: stretch; padding: 14px; color: #d9e8ff; background: linear-gradient(145deg, #111a2a, #192842); } +.subtitle-waveform-toolbar { justify-content: space-between; margin-bottom: 13px; font-size: 0.78rem; } +.subtitle-waveform-toolbar span { color: #9bb6da; } +#subtitle-waveform { min-height: 92px; border-radius: 10px; background: rgba(255, 255, 255, 0.045); } +#subtitle-timeline { min-height: 24px; color: #b7cae6; } +.subtitle-editor-toolbar { + flex-wrap: wrap; + align-items: flex-end; + margin: 14px 0 12px; + padding: 13px; + border-radius: 15px; + background: #f4f7fb; +} +.subtitle-search-field { display: grid; flex: 1 1 170px; gap: 5px; } +.subtitle-shift-field { display: grid; flex: 0 1 125px; gap: 5px; } +.subtitle-editor-toolbar .control { min-height: 36px; } +.subtitle-quality-summary { + margin-bottom: 10px; + padding: 10px 13px; + border: 1px solid rgba(38, 118, 255, 0.18); + border-radius: 12px; + color: #355274; + background: #eef6ff; + font-size: 0.82rem; +} +.subtitle-quality-summary.has-errors { color: #a72530; border-color: rgba(199, 50, 63, 0.24); background: #fff0f1; } +.subtitle-quality-summary.has-warnings { color: #765307; border-color: rgba(213, 151, 20, 0.25); background: #fff8e5; } +.subtitle-quality-summary.is-clean { color: #146840; border-color: rgba(24, 146, 89, 0.2); background: #ecfaf3; } +.subtitle-cue-list { + position: relative; + height: min(640px, 64vh); + overflow: auto; + border: 1px solid rgba(126, 151, 187, 0.22); + border-radius: 16px; + background: #f8fafc; + contain: strict; +} +.subtitle-cue-spacer { width: 1px; opacity: 0; } +.subtitle-cue-viewport { position: absolute; inset: 0 0 auto 0; } +.subtitle-cue-row { + position: absolute; + top: 0; + left: 0; + right: 0; + display: grid; + grid-template-columns: 42px 106px 160px minmax(260px, 1fr) 126px; + grid-template-rows: 83px 31px; + gap: 7px 10px; + height: 124px; + margin: 4px 8px; + padding: 10px; + border: 1px solid rgba(126, 151, 187, 0.18); + border-radius: 13px; + background: #fff; + box-shadow: 0 5px 15px rgba(33, 62, 97, 0.04); + box-sizing: border-box; +} +.subtitle-cue-row.is-selected { border-color: rgba(38, 118, 255, 0.55); background: #f4f8ff; } +.subtitle-cue-row.is-current { box-shadow: inset 4px 0 0 var(--blue), 0 7px 20px rgba(38, 118, 255, 0.12); } +.subtitle-cue-check { display: grid; grid-template-rows: 26px auto; place-items: center; color: var(--muted); font-weight: 750; } +.subtitle-cue-time { + align-self: center; + padding: 8px; + border: 0; + border-radius: 9px; + color: var(--blue); + background: #edf5ff; + font: inherit; + font-size: 0.74rem; + font-weight: 780; + cursor: pointer; +} +.subtitle-cue-time-inputs { display: grid; grid-template-columns: 1fr; gap: 5px; } +.subtitle-cue-time-inputs label, +.subtitle-speaker { display: grid; gap: 3px; } +.subtitle-cue-time-inputs input, +.subtitle-speaker input, +.subtitle-cue-row textarea { + width: 100%; + min-width: 0; + padding: 7px 8px; + border: 1px solid rgba(126, 151, 187, 0.3); + border-radius: 8px; + color: var(--text); + background: #fff; + font: inherit; + box-sizing: border-box; +} +.subtitle-cue-row textarea { resize: none; line-height: 1.45; } +.subtitle-cue-issues { + grid-column: 2 / -1; + overflow: hidden; + color: var(--muted); + font-size: 0.74rem; + text-overflow: ellipsis; + white-space: nowrap; +} +.subtitle-cue-empty { padding: 50px 20px; color: var(--muted); text-align: center; } + +@media (max-width: 1080px) { + .subtitle-editor-media-grid { grid-template-columns: 1fr; } + .subtitle-cue-row { grid-template-columns: 38px 100px 150px minmax(220px, 1fr); } + .subtitle-speaker { display: none; } +} + +@media (max-width: 760px) { + .subtitle-editor-heading, + .subtitle-editor-trackbar, + .subtitle-editor-state { align-items: stretch; flex-direction: column; } + .subtitle-editor-trackbar .button-row { display: grid; grid-template-columns: 1fr 1fr; width: 100%; } + .subtitle-cue-row { + grid-template-columns: 34px 96px minmax(185px, 1fr); + grid-template-rows: 83px 31px; + min-width: 560px; + } + .subtitle-cue-time-inputs { display: none; } + .subtitle-cue-issues { grid-column: 2 / -1; } +} + +.subtitle-review-gate { + display: grid; + grid-template-columns: minmax(0, 1fr) auto; + gap: 24px; + align-items: center; + margin-bottom: 24px; + border-color: rgba(0, 122, 255, 0.24); + background: linear-gradient(135deg, rgba(0, 122, 255, 0.08), rgba(255, 255, 255, 0.88)); +} +.subtitle-review-gate h2, +.subtitle-ai-suggestion-panel h3 { margin: 4px 0 8px; } +.subtitle-batch-job { + grid-column: 1 / -1; + padding-top: 16px; + border-top: 1px solid var(--line); +} +.subtitle-batch-job p { margin: 8px 0; } +.subtitle-ai-suggestion-panel { + margin: 14px 0; + padding: 16px; + border: 1px solid rgba(0, 122, 255, 0.2); + border-radius: 18px; + background: rgba(245, 249, 255, 0.92); +} +.subtitle-ai-diff-list { + display: grid; + gap: 8px; + max-height: 360px; + overflow: auto; +} +.subtitle-ai-diff-row { + display: grid; + grid-template-columns: auto 110px minmax(0, 1fr) auto minmax(0, 1fr); + gap: 10px; + align-items: start; + padding: 12px; + border: 1px solid var(--line); + border-radius: 12px; + background: #fff; +} +.subtitle-ai-diff-time { + color: var(--muted); + font-variant-numeric: tabular-nums; +} +.subtitle-ai-diff-before { color: #8e4b4b; text-decoration: line-through; } +.subtitle-ai-diff-after { color: #126b42; white-space: pre-wrap; } +.subtitle-ai-diff-arrow { color: var(--muted); } + +@media (max-width: 900px) { + .subtitle-review-gate { grid-template-columns: 1fr; } + .subtitle-ai-diff-row { grid-template-columns: auto 1fr; } + .subtitle-ai-diff-time, + .subtitle-ai-diff-before, + .subtitle-ai-diff-arrow, + .subtitle-ai-diff-after { grid-column: 2; } +} + +/* Dashboard weekly overview */ +.dashboard-overview-panel { + position: relative; + margin-bottom: 18px; + overflow: hidden; + padding: 0; +} + +.dashboard-overview-panel::before { + position: absolute; + top: -120px; + right: -80px; + width: 360px; + height: 260px; + border-radius: 50%; + background: radial-gradient(circle, rgba(31, 111, 255, 0.12), rgba(31, 111, 255, 0)); + content: ""; + pointer-events: none; +} + +.dashboard-overview-header { + position: relative; + z-index: 1; + display: flex; + align-items: flex-start; + justify-content: space-between; + gap: 20px; + padding: 22px 24px 18px; +} + +.dashboard-overview-header h2 { + margin: 0; + color: #111827; +} + +.dashboard-overview-header > div > p:last-child { + margin: 7px 0 0; + color: var(--muted); + font-size: 13px; +} + +.dashboard-week-chip { + flex: 0 0 auto; + padding: 8px 12px; + border: 1px solid rgba(31, 111, 255, 0.14); + border-radius: 999px; + background: rgba(239, 246, 255, 0.86); + color: var(--blue); + font-size: 12px; + font-weight: 760; + white-space: nowrap; +} + +.dashboard-stat-strip { + position: relative; + z-index: 1; + display: grid; + grid-template-columns: repeat(4, minmax(0, 1fr)); + margin: 0 24px 24px; + overflow: hidden; + border: 1px solid rgba(31, 45, 71, 0.08); + border-radius: 14px; + background: rgba(248, 251, 255, 0.72); +} + +.dashboard-stat-item { + position: relative; + display: grid; + min-width: 0; + gap: 7px; + padding: 18px 20px; +} + +.dashboard-stat-item + .dashboard-stat-item { + border-left: 1px solid rgba(31, 45, 71, 0.08); +} + +.dashboard-stat-item::after { + position: absolute; + top: 18px; + right: 18px; + width: 8px; + height: 8px; + border-radius: 50%; + background: currentColor; + box-shadow: 0 0 0 5px color-mix(in srgb, currentColor 10%, transparent); + content: ""; +} + +.dashboard-stat-item > span { + color: var(--muted); + font-size: 12px; + font-weight: 740; +} + +.dashboard-stat-item > strong { + color: #111827; + font-size: clamp(28px, 2.1vw, 36px); + line-height: 1; +} + +.dashboard-stat-item > small { + min-height: 34px; + color: currentColor; + font-size: 12px; + font-weight: 700; + line-height: 1.45; + overflow-wrap: anywhere; +} + +@media (max-width: 1180px) { + .dashboard-stat-strip { + grid-template-columns: repeat(2, minmax(0, 1fr)); + } + + .dashboard-stat-item:nth-child(3) { + border-left: 0; + } + + .dashboard-stat-item:nth-child(n + 3) { + border-top: 1px solid rgba(31, 45, 71, 0.08); + } + +} + +@media (max-width: 640px) { + .dashboard-overview-header { + align-items: stretch; + flex-direction: column; + } + + .dashboard-overview-header { + padding: 18px 16px 14px; + } + + .dashboard-week-chip { + align-self: flex-start; + } + + .dashboard-stat-strip { + margin: 0 16px 20px; + } + + .dashboard-stat-item { + padding: 16px; + } + +} + +/* Motion system: page-specific, local-only and safe to ignore when JS is unavailable. */ +body.motion-enabled { + --motion-duration: 420ms; + --motion-fast: 260ms; + --motion-ease: cubic-bezier(0.22, 1, 0.36, 1); +} + +.motion-enabled .motion-pending { + opacity: 0; + will-change: opacity, transform; +} + +.motion-enabled .motion-pending.motion-rise { + transform: translate3d(0, 10px, 0); +} + +.motion-enabled .motion-pending.motion-soft { + transform: translate3d(0, 4px, 0); +} + +.motion-enabled .motion-pending.motion-in { + animation: motion-rise-in var(--motion-duration) var(--motion-ease) both; + animation-delay: calc(min(var(--motion-order, 0), 9) * 36ms); +} + +.motion-enabled .motion-pending.motion-soft.motion-in { + animation-name: motion-soft-in; + animation-duration: 360ms; +} + +.motion-enabled :is(.progress, .large-progress) > span, +.motion-enabled [data-task-live-progress-bar], +.motion-enabled #cut-job-progress-bar, +.motion-enabled #subtitle-batch-progress-bar { + transition: width 360ms var(--motion-ease); +} + +.motion-enabled :is(.metro-step, .status-pill, .publish-calendar-day, .publish-plan-row, .publish-content-card) { + transition: + color var(--motion-fast) ease, + background-color var(--motion-fast) ease, + border-color var(--motion-fast) ease, + box-shadow var(--motion-fast) ease; +} + +.motion-enabled :is(.primary-button, .secondary-button, .icon-button, .text-button, .link-button):not(:disabled):active { + transform: translateY(0) scale(0.985); +} + +.motion-enabled .motion-data-updated { + animation: motion-data-highlight 620ms ease-out both; +} + +.motion-enabled .motion-item-added { + animation: motion-item-added 360ms var(--motion-ease) both; + animation-delay: calc(min(var(--motion-order, 0), 7) * 28ms); +} + +.motion-enabled .motion-confirm { + animation: motion-confirm 480ms var(--motion-ease) both; +} + +.motion-enabled .motion-panel-activated, +.motion-enabled.motion-settled .publish-calendar-day-detail:not([hidden]), +.motion-enabled.motion-settled .subtitle-batch-job:not([hidden]) { + animation: motion-panel-swap 300ms var(--motion-ease) both; +} + +.motion-enabled .clip-card.is-removing { + transition: opacity 180ms ease, transform 180ms ease; + transform: scale(0.995); +} + +@media (hover: hover) { + .motion-enabled :is(.review-task-row, table tbody tr) { + transition: background-color 180ms ease, border-color 180ms ease, box-shadow 180ms ease, transform 180ms ease; + } + + .motion-enabled .review-task-row:hover { + border-color: rgba(37, 111, 255, 0.2); + box-shadow: 0 10px 28px rgba(24, 39, 75, 0.07); + transform: translateY(-1px); + } + + .motion-enabled table tbody tr:hover { + background: rgba(234, 242, 255, 0.38); + } +} + +.motion-enabled.motion-settled .modal-backdrop:not([hidden]), +.motion-enabled.motion-settled .source-monitor-backdrop:not([hidden]), +.motion-enabled.motion-settled .schedule-drawer-backdrop:not([hidden]) { + animation: motion-overlay-in 220ms ease-out both; +} + +.motion-enabled.motion-settled .modal-backdrop:not([hidden]) > .modal-panel, +.motion-enabled.motion-settled .source-monitor-backdrop:not([hidden]) > .source-monitor-panel { + animation: motion-dialog-in 300ms var(--motion-ease) both; +} + +.motion-enabled.motion-settled .transcript-drawer:not([hidden]), +.motion-enabled.motion-settled .schedule-drawer:not([hidden]) { + animation: motion-drawer-in 320ms var(--motion-ease) both; +} + +.motion-enabled.motion-settled details[open] > :not(summary) { + animation: motion-soft-in 240ms ease-out both; +} + +@keyframes motion-rise-in { + from { opacity: 0; transform: translate3d(0, 10px, 0); } + to { opacity: 1; transform: translate3d(0, 0, 0); } +} + +@keyframes motion-soft-in { + from { opacity: 0; transform: translate3d(0, 4px, 0); } + to { opacity: 1; transform: translate3d(0, 0, 0); } +} + +@keyframes motion-data-highlight { + 0% { box-shadow: 0 0 0 4px rgba(37, 111, 255, 0.12); filter: saturate(1.12); } + 100% { box-shadow: 0 0 0 0 rgba(37, 111, 255, 0); filter: saturate(1); } +} + +@keyframes motion-item-added { + from { opacity: 0; transform: translate3d(0, 7px, 0); } + to { opacity: 1; transform: translate3d(0, 0, 0); } +} + +@keyframes motion-confirm { + 0%, 100% { box-shadow: 0 0 0 0 rgba(34, 160, 107, 0); } + 45% { border-color: rgba(34, 160, 107, 0.42); box-shadow: 0 0 0 5px rgba(34, 160, 107, 0.1); } +} + +@keyframes motion-panel-swap { + from { opacity: 0; transform: translate3d(0, 5px, 0); } + to { opacity: 1; transform: translate3d(0, 0, 0); } +} + +@keyframes motion-overlay-in { + from { opacity: 0; } + to { opacity: 1; } +} + +@keyframes motion-dialog-in { + from { opacity: 0; transform: translate3d(0, 10px, 0) scale(0.985); } + to { opacity: 1; transform: translate3d(0, 0, 0) scale(1); } +} + +@keyframes motion-drawer-in { + from { opacity: 0; transform: translate3d(18px, 0, 0); } + to { opacity: 1; transform: translate3d(0, 0, 0); } +} + +@media print, (prefers-reduced-motion: reduce) { + html:focus-within { + scroll-behavior: auto; + } + + *, + *::before, + *::after { + scroll-behavior: auto !important; + animation-duration: 1ms !important; + animation-delay: 0ms !important; + animation-iteration-count: 1 !important; + transition-duration: 1ms !important; + } + + .send-publishing-spinner { + animation: none !important; + } +} diff --git a/app/static/js/app.js b/app/static/js/app.js index 1e4b73d..8c3cfa4 100644 --- a/app/static/js/app.js +++ b/app/static/js/app.js @@ -1,13 +1,9 @@ async function apiFetch(url, options = {}) { const requestOptions = { ...options }; const headers = new Headers(options.headers || {}); - const token = document.querySelector('meta[name="local-admin-token"]')?.content || ""; if (options.body && !(options.body instanceof FormData) && !headers.has("Content-Type")) { headers.set("Content-Type", "application/json"); } - if (token && !headers.has("Authorization")) { - headers.set("Authorization", `Bearer ${token}`); - } requestOptions.headers = headers; const response = await fetch(url, requestOptions); let data = {}; @@ -31,9 +27,17 @@ async function apiFetch(url, options = {}) { window.apiFetch = apiFetch; +function preferredScrollBehavior() { + return window.matchMedia("(prefers-reduced-motion: reduce)").matches ? "auto" : "smooth"; +} + +window.preferredScrollBehavior = preferredScrollBehavior; + const newTaskForm = document.querySelector("#new-task-form"); const newTaskAutoMode = newTaskForm?.querySelector("input[name='auto_mode']"); const newTaskSubmitButton = document.querySelector("#new-task-submit-button"); +const selectionProfileInput = document.querySelector("#selection-profile"); +const longLiveSettings = document.querySelector("#long-live-settings"); function updateNewTaskSubmitLabel() { if (!newTaskSubmitButton) return; @@ -45,6 +49,14 @@ if (newTaskAutoMode) { updateNewTaskSubmitLabel(); } +if (selectionProfileInput && longLiveSettings) { + const updateLongLiveSettings = () => { + longLiveSettings.hidden = selectionProfileInput.value !== "long_live_talk"; + }; + selectionProfileInput.addEventListener("change", updateLongLiveSettings); + updateLongLiveSettings(); +} + if (newTaskForm) { newTaskForm.addEventListener("submit", async (event) => { event.preventDefault(); @@ -58,29 +70,22 @@ if (newTaskForm) { result.textContent = "正在创建任务..."; try { - if (!videoFileInput.files.length) { - throw new Error("请选择要上传的视频文件"); - } + if (!payload.selection_profile) throw new Error("请选择选片模式"); + if (!videoFileInput.files.length) throw new Error("请选择要上传的视频文件"); const uploadData = new FormData(); - uploadData.append("task_name", payload.task_name || ""); - uploadData.append("platform", payload.platform || "general"); - uploadData.append("max_clip_duration", payload.max_clip_duration || "10"); - uploadData.append("candidate_clip_count", payload.candidate_clip_count || "12"); - uploadData.append("selection_profile", payload.selection_profile || "variety_comedy"); - uploadData.append("final_clip_target", payload.final_clip_target || "5"); + for (const key of [ + "task_name", "platform", "max_clip_duration", "candidate_clip_count", + "selection_profile", "final_clip_target", + ]) uploadData.append(key, payload[key] || ""); + if (payload.selection_profile === "long_live_talk") { + uploadData.append("highlight_density_per_hour", payload.highlight_density_per_hour || "4"); + uploadData.append("highlight_total_limit", payload.highlight_total_limit || "30"); + } uploadData.append("ai_preference", ""); uploadData.append("auto_mode", payload.auto_mode === "true" ? "true" : "false"); uploadData.append("auto_metadata_use_ai", "false"); uploadData.append("video_file", videoFileInput.files[0]); - const response = await fetch("/api/tasks/upload", { - method: "POST", - body: uploadData, - }); - - const data = await response.json(); - if (!response.ok) { - throw new Error(data.detail || "任务创建失败"); - } + const data = await apiFetch("/api/tasks/upload", { method: "POST", body: uploadData }); result.textContent = `${data.message}${payload.auto_mode === "true" ? " 全自动流水线已启动。" : ""} 正在进入详情页...`; window.location.href = data.detail_url; } catch (error) { @@ -120,6 +125,7 @@ async function handleProcessAction(button) { if (result) result.textContent = data.message || "处理完成,正在刷新页面..."; if (button.dataset.endpoint.includes("/process/transcript")) { startTranscriptPolling(true); + startTaskLiveStatusPolling(true); if (data.status === "completed") { window.setTimeout(() => window.location.reload(), 600); } @@ -333,9 +339,11 @@ const aiAnalysisSummary = document.querySelector("#ai-analysis-summary"); const aiCandidateCountPill = document.querySelector("#ai-candidate-count-pill"); const aiCandidateCountInput = document.querySelector("#ai-candidate-count-input"); const aiSelectionProfile = aiAnalysisForm - ? aiAnalysisForm.dataset.selectionProfile || "variety_comedy" - : "variety_comedy"; + ? aiAnalysisForm.dataset.selectionProfile || "general" + : "general"; const aiFinalClipTarget = document.querySelector("#ai-final-clip-target"); +const aiHighlightDensity = document.querySelector("#ai-highlight-density"); +const aiHighlightTotalLimit = document.querySelector("#ai-highlight-total-limit"); const showAiHistoryButton = document.querySelector("#show-ai-history-button"); const refreshAiHistoryButton = document.querySelector("#refresh-ai-history-button"); const aiAnalysisHistory = document.querySelector("#ai-analysis-history"); @@ -363,11 +371,16 @@ const taskLiveUpdatedAt = document.querySelector("[data-task-live-updated-at]"); const taskLiveCandidateCount = document.querySelector("[data-task-live-candidate-count]"); const taskLiveOutputCount = document.querySelector("[data-task-live-output-count]"); const taskLiveActions = document.querySelector("[data-live-task-actions]"); +const taskLiveOperation = document.querySelector("[data-task-live-operation]"); +const taskLiveOperationLabel = document.querySelector("[data-task-live-operation-label]"); +const taskLiveOperationProgress = document.querySelector("[data-task-live-operation-progress]"); +const taskLiveOperationMessage = document.querySelector("[data-task-live-operation-message]"); let aiStatusPollingTimer = null; let isAiAnalysisBusy = false; let aiAnalysisControlStates = new Map(); let taskLiveStatusTimer = null; let taskLiveForcedPollingUntil = 0; +let taskLiveStatusRequestInFlight = false; const TASK_LIVE_STATUS_INTERVAL_MS = 3000; function setAiAnalysisControlsDisabled(disabled) { @@ -664,10 +677,7 @@ aiProcessButtons.forEach((button) => { percent: 18, message: "正在保存 Prompt 方案并启动 AI 分析...", }); - renderRuntimeLog({ - status: "running", - log_lines: ["正在启动 AI 分析,请稍等..."], - }); + startTaskLiveStatusPolling(true); try { await saveTaskAiPromptSettings(); @@ -681,13 +691,20 @@ aiProcessButtons.forEach((button) => { if (!response.ok) { throw new Error(data.detail || "AI 分析失败"); } - if (aiProcessResult) aiProcessResult.textContent = data.message || "AI 分析完成。"; + if (!data.job_id) throw new Error("AI 分析队列没有返回 job_id"); + if (aiProcessResult) aiProcessResult.textContent = data.message || "AI 分析已加入队列。"; + const completedJob = await waitForAiAnalysisJob(data.job_id); + const result = completedJob.result_json || {}; + if (aiProcessResult) aiProcessResult.textContent = result.message || completedJob.message || "AI 分析完成。"; await pollAiAnalysisStatus(false).catch(() => {}); - if (aiCandidateCountPill && Array.isArray(data.clips)) { - aiCandidateCountPill.textContent = `${data.clips.length} 条候选`; + if (aiCandidateCountPill && Number.isFinite(Number(result.clip_count))) { + aiCandidateCountPill.textContent = `${Number(result.clip_count)} 条候选`; } - renderAiAnalysisSummary(data.analysis_run || data); - renderAiAnalysisHistory(data.runs || aiAnalysisRuns); + const historyResponse = await fetch(`/api/tasks/${taskId}/ai-analysis-runs`); + const historyData = await historyResponse.json(); + if (!historyResponse.ok) throw new Error(historyData.detail || "读取 AI 分析历史失败"); + renderAiAnalysisSummary(historyData.latest || result); + renderAiAnalysisHistory(historyData.runs || aiAnalysisRuns); } catch (error) { if (aiProcessResult) aiProcessResult.textContent = `AI 分析失败:${summarizeErrorMessage(error.message)}`; await pollAiAnalysisStatus(false).catch(() => {}); @@ -1344,7 +1361,7 @@ function ensureClipPreviewVisible() { const viewportHeight = window.innerHeight || document.documentElement.clientHeight || 0; const isVisible = rect.top >= 0 && rect.top < viewportHeight * 0.72 && rect.bottom > Math.min(120, viewportHeight); if (isVisible) return; - previewTarget.scrollIntoView({ behavior: "smooth", block: "start", inline: "nearest" }); + previewTarget.scrollIntoView({ behavior: preferredScrollBehavior(), block: "start", inline: "nearest" }); } function closeTranscriptDrawer() { @@ -1579,13 +1596,18 @@ async function saveTaskSelectionSettings() { if (!Number.isInteger(finalTarget) || finalTarget < 1 || finalTarget > 12) { throw new Error("最终启用目标必须是 1 到 12 之间的整数。"); } + const settingsPayload = { + selection_profile: aiSelectionProfile, + final_clip_target: finalTarget, + }; + if (aiSelectionProfile === "long_live_talk") { + settingsPayload.highlight_density_per_hour = Number(aiHighlightDensity?.value || 4); + settingsPayload.highlight_total_limit = Number(aiHighlightTotalLimit?.value || 30); + } const response = await fetch(`/api/tasks/${aiAnalysisForm.dataset.taskId}/selection-settings`, { method: "PATCH", headers: { "Content-Type": "application/json" }, - body: JSON.stringify({ - selection_profile: aiSelectionProfile || "variety_comedy", - final_clip_target: finalTarget, - }), + body: JSON.stringify(settingsPayload), }); const data = await response.json(); if (!response.ok) throw new Error(data.detail || "选片设置保存失败"); @@ -1626,7 +1648,11 @@ document.querySelectorAll("[data-sync-publish-task]").forEach((button) => { } const summary = document.querySelector("[data-publish-link-summary]"); if (summary) { - summary.innerHTML = `发送中心关联:${data.link_state?.label || "同步完成"}${data.message || ""}`; + const heading = document.createElement("strong"); + const detail = document.createElement("span"); + heading.textContent = `发送中心关联:${data.link_state?.label || "同步完成"}`; + detail.textContent = data.message || ""; + summary.replaceChildren(heading, detail); } showClipReviewMessage(data.message || "发送中心同步完成。", data.status === "partial" ? "error" : "success"); if (!document.querySelector("#process-result")) { @@ -1654,7 +1680,7 @@ document.querySelectorAll("[data-sync-publish-task]").forEach((button) => { document.querySelectorAll(".js-hide-task").forEach((button) => { button.addEventListener("click", async () => { const taskTitle = button.dataset.taskTitle || "这条任务"; - const confirmed = window.confirm(`确认永久删除“${taskTitle}”吗?\n\n系统会永久删除 E 盘任务目录内的原片副本、音频、转写、切片、字幕、封面和发布包,删除后无法恢复。\n\nNAS 或任务目录外的原始视频不会被删除。`); + const confirmed = window.confirm(`确认永久删除“${taskTitle}”吗?\n\n系统会永久删除 E 盘任务目录内的原片副本、音频、转写、切片、字幕、封面和发布包,删除后无法恢复。\n\n任务目录外的原始视频不会被删除。`); if (!confirmed) return; const originalText = button.textContent; @@ -1689,6 +1715,15 @@ if (subtitleStyleForm) { const formData = new FormData(subtitleStyleForm); const payload = Object.fromEntries(formData.entries()); payload.font_size = Number(payload.font_size || 42); + payload.outline_width = Number(payload.outline_width || 3); + payload.shadow_depth = Number(payload.shadow_depth || 1); + payload.safe_area_percent = Number(payload.safe_area_percent || 5); + payload.speaker_styles = { + 主播: { font_color: payload.speaker_host_color || "#ffffff" }, + 嘉宾: { font_color: payload.speaker_guest_color || "#ffd60a" }, + }; + delete payload.speaker_host_color; + delete payload.speaker_guest_color; payload.shadow_enabled = Boolean(subtitleStyleForm.elements.shadow_enabled?.checked); if (submitButton) submitButton.disabled = true; if (subtitleStyleResult) subtitleStyleResult.textContent = "正在保存字幕样式..."; @@ -1736,8 +1771,13 @@ function renderRuntimeLog(status) { completed: "已完成", failed: "失败", }; - runtimeLogState.textContent = status.status_label || labelMap[status.status] || status.task_status_label || "已刷新"; - runtimeLogState.dataset.status = status.status || "idle"; + const runtimeStatus = status.runtime_status || status.status || "idle"; + runtimeLogState.textContent = status.runtime_status_label + || labelMap[runtimeStatus] + || status.status_label + || status.task_status_label + || "已刷新"; + runtimeLogState.dataset.status = runtimeStatus; } if (!runtimeLogLines) return; const lines = Array.isArray(status.log_lines) ? status.log_lines : []; @@ -1769,13 +1809,38 @@ function renderTaskLiveActions(data) { const reviewAction = taskLiveActions.querySelector("[data-live-review-action]"); if (reviewAction) reviewAction.hidden = !actions.review; const syncAction = taskLiveActions.querySelector("[data-live-sync-action]"); - if (syncAction) syncAction.hidden = Number(data.counts?.outputs || 0) <= 0; + if (syncAction) { + syncAction.hidden = primaryAction === "subtitle_review" || Number(data.counts?.outputs || 0) <= 0; + } + const subtitleSkip = taskLiveActions.querySelector("[data-live-subtitle-skip]"); + if (subtitleSkip) subtitleSkip.hidden = primaryAction !== "subtitle_review"; +} + +async function waitForAiAnalysisJob(jobId) { + while (true) { + const response = await fetch(`/api/tasks/jobs/${jobId}`); + const job = await response.json(); + if (!response.ok) throw new Error(job.detail || "查询 AI 分析任务进度失败"); + renderAiAnalysisProgress({ + status: job.status, + percent: Number(job.progress || 0), + message: job.status === "failed" + ? (job.error_message || job.message || "AI 分析失败") + : (job.message || "AI 分析正在排队..."), + }); + if (job.status === "completed") return job; + if (job.status === "failed" || job.status === "cancelled") { + throw new Error(job.error_message || job.message || "AI 分析任务未完成"); + } + await wait(1000); + } } function renderTaskLiveStatus(data) { const progress = Math.max(0, Math.min(100, Number(data.progress || 0))); document.querySelectorAll("[data-task-live-status-label]").forEach((node) => { node.textContent = data.status_label || data.status || "状态未知"; + node.dataset.status = data.runtime_status || (data.should_poll ? "running" : "completed"); }); const headerStatus = document.querySelector("[data-task-live-header-status]"); if (headerStatus) headerStatus.textContent = data.status_label || data.status || "状态未知"; @@ -1789,6 +1854,18 @@ function renderTaskLiveStatus(data) { if (taskLiveOutputCount) taskLiveOutputCount.textContent = `${outputCount} 条`; if (aiCandidateCountPill) aiCandidateCountPill.textContent = `${candidateCount} 条候选`; + const operation = data.active_operation || {}; + const operationProgress = Math.max(0, Math.min(100, Number(operation.progress || 0))); + if (taskLiveOperation) taskLiveOperation.dataset.status = operation.status || "idle"; + if (taskLiveOperationLabel) { + taskLiveOperationLabel.textContent = operation.label || data.task_status_label || "当前任务"; + } + if (taskLiveOperationProgress) taskLiveOperationProgress.textContent = `${operationProgress}%`; + if (taskLiveOperationMessage) { + taskLiveOperationMessage.textContent = operation.message + || `当前阶段:${data.task_status_label || data.status_label || "状态未知"}`; + } + const allowedStepStates = new Set(["done", "current", "pending", "warning"]); (Array.isArray(data.workflow_steps) ? data.workflow_steps : []).forEach((step) => { const node = taskLiveOverview?.querySelector(`[data-task-live-step="${step.index}"]`); @@ -1808,6 +1885,7 @@ function renderTaskLiveStatus(data) { autoPipelineMonitor.dataset.running = data.should_poll ? "true" : "false"; } if (taskLiveNote) { + taskLiveNote.dataset.snapshotAt = data.snapshot_at || ""; if (data.error_message) { taskLiveNote.dataset.state = "error"; taskLiveNote.textContent = `流程已暂停:${summarizeErrorMessage(data.error_message)}`; @@ -1831,6 +1909,7 @@ function scheduleTaskLiveStatusPolling() { async function pollTaskLiveStatus() { if (!autoPipelineMonitor) return null; + if (taskLiveStatusRequestInFlight) return null; const taskId = autoPipelineMonitor.dataset.taskId; if (!taskId) return null; if (taskLiveStatusTimer) { @@ -1838,6 +1917,7 @@ async function pollTaskLiveStatus() { taskLiveStatusTimer = null; } + taskLiveStatusRequestInFlight = true; try { const data = await apiFetch(`/api/tasks/${encodeURIComponent(taskId)}/live-status`); renderTaskLiveStatus(data); @@ -1853,6 +1933,8 @@ async function pollTaskLiveStatus() { } scheduleTaskLiveStatusPolling(); return null; + } finally { + taskLiveStatusRequestInFlight = false; } } @@ -1883,7 +1965,6 @@ async function pollAiAnalysisStatus(keepPolling = false) { throw new Error(data.detail || "读取 AI 分析状态失败"); } renderAiAnalysisProgress(data); - renderRuntimeLog(data); if (aiStatusPollingTimer) { window.clearTimeout(aiStatusPollingTimer); aiStatusPollingTimer = null; @@ -1912,8 +1993,8 @@ document.querySelectorAll("[data-render-subtitle]").forEach((button) => { const errorNode = card.querySelector("[data-subtitle-error]"); const originalText = button.textContent; button.disabled = true; - button.textContent = "加字幕中..."; - if (statusNode) statusNode.textContent = "字幕生成中"; + button.textContent = "正在入队..."; + if (statusNode) statusNode.textContent = "字幕排队中"; if (errorNode) errorNode.textContent = ""; try { @@ -1922,9 +2003,10 @@ document.querySelectorAll("[data-render-subtitle]").forEach((button) => { if (!response.ok) { throw new Error(data.detail || "自动加字幕失败"); } - if (statusNode) statusNode.textContent = data.output_clip?.subtitle_status_label || "已加字幕"; - if (errorNode) errorNode.textContent = data.message || "自动加字幕完成。"; - window.setTimeout(() => window.location.reload(), 700); + if (errorNode) errorNode.textContent = data.message || "字幕任务已加入队列。"; + button.textContent = "后台烧录中"; + await pollSubtitleWorkflowJob(data.job_id, statusNode, errorNode); + window.location.reload(); } catch (error) { if (statusNode) statusNode.textContent = "字幕失败"; if (errorNode) errorNode.textContent = `自动加字幕失败:${error.message}`; @@ -1935,6 +2017,43 @@ document.querySelectorAll("[data-render-subtitle]").forEach((button) => { }); }); +async function pollSubtitleWorkflowJob(jobId, statusNode, messageNode) { + if (!jobId) throw new Error("后台没有返回字幕 job id"); + while (true) { + const response = await fetch(`/api/tasks/jobs/${encodeURIComponent(jobId)}`); + const job = await response.json(); + if (!response.ok) throw new Error(job.detail || "读取字幕任务状态失败"); + if (statusNode) statusNode.textContent = `${job.status_label || job.status} · ${Number(job.progress || 0)}%`; + if (messageNode) messageNode.textContent = job.message || "字幕任务处理中"; + if (job.status === "completed") return job; + if (job.status === "failed" || job.status === "cancelled") { + throw new Error(job.error_message || job.message || "字幕任务未完成"); + } + await new Promise((resolve) => window.setTimeout(resolve, 1500)); + } +} + +document.querySelectorAll("[data-live-subtitle-skip]").forEach((button) => { + button.addEventListener("click", async () => { + if (!window.confirm("确认跳过字幕并进入片段审核吗?审核保存后才会同步发送中心。")) return; + const taskId = button.dataset.taskId; + button.disabled = true; + const originalText = button.textContent; + button.textContent = "正在进入审核..."; + try { + const data = await apiFetch(`/api/subtitles/tasks/${encodeURIComponent(taskId)}/skip-to-review`, { + method: "POST", + }); + if (taskLiveNote) taskLiveNote.textContent = data.message || "已跳过字幕,正在进入片段审核"; + window.location.href = data.review_url || `/tasks/${encodeURIComponent(taskId)}/clips/review`; + } catch (error) { + window.alert(`跳过字幕失败:${summarizeErrorMessage(error.message)}`); + button.disabled = false; + button.textContent = originalText; + } + }); +}); + const cutEditModal = document.querySelector("#cut-edit-modal"); const closeCutEditButton = document.querySelector("#close-cut-edit"); const cutEditVideo = document.querySelector("#cut-edit-video"); @@ -2329,15 +2448,11 @@ if (aiConfigForm) { if (aiConfigResult) aiConfigResult.textContent = "正在保存三类 AI 接口配置..."; try { - const response = await fetch("/api/settings/ai", { + const data = await window.apiFetch("/api/settings/ai", { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify(payload), }); - const data = await response.json(); - if (!response.ok) { - throw new Error(data.detail || "保存失败"); - } if (aiConfigResult) aiConfigResult.textContent = data.message || "保存成功。"; window.setTimeout(() => { toggleAiConfigModal(false); diff --git a/app/static/js/motion.js b/app/static/js/motion.js new file mode 100644 index 0000000..961bcf8 --- /dev/null +++ b/app/static/js/motion.js @@ -0,0 +1,208 @@ +(() => { + "use strict"; + + const body = document.body; + const mainPanel = document.querySelector(".main-panel"); + if (!body || !mainPanel) return; + + const path = window.location.pathname.replace(/\/+$/, "") || "/"; + const reduceMotion = window.matchMedia("(prefers-reduced-motion: reduce)").matches; + + const profiles = [ + { + name: "dashboard", + matches: () => path === "/", + reveals: [ + [".dashboard-stat-item", "soft", 5], + ], + }, + { + name: "new-task", + matches: () => path === "/tasks/new", + reveals: [], + }, + { + name: "tasks", + matches: () => path === "/tasks", + reveals: [], + }, + { + name: "transcript", + matches: () => /^\/tasks\/[^/]+\/transcript$/.test(path), + reveals: [], + }, + { + name: "clip-review", + matches: () => /^\/tasks\/[^/]+\/clips(?:\/review)?$/.test(path), + reveals: [], + }, + { + name: "task-detail", + matches: () => /^\/tasks\/[^/]+$/.test(path), + reveals: [], + }, + { + name: "clips", + matches: () => path === "/clips", + reveals: [], + }, + { + name: "subtitle-task", + matches: () => /^\/subtitles\/[^/]+$/.test(path), + reveals: [], + }, + { + name: "subtitles", + matches: () => path === "/subtitles", + reveals: [], + }, + { + name: "publish", + matches: () => path === "/publish", + reveals: [], + }, + { + name: "system", + matches: () => path === "/system", + reveals: [], + }, + ]; + + const profile = profiles.find((item) => item.matches()) || { + name: body.dataset.page || "static", + reveals: [], + }; + + body.dataset.motionPage = profile.name; + body.dataset.motion = reduceMotion ? "reduced" : "full"; + if (reduceMotion) return; + + body.classList.add("motion-enabled"); + + const canObserveViewport = "IntersectionObserver" in window; + const revealObserver = canObserveViewport + ? new IntersectionObserver((entries, observer) => { + entries.forEach((entry) => { + if (!entry.isIntersecting) return; + const node = entry.target; + node.classList.add("motion-in"); + node.dataset.motionSeen = "true"; + observer.unobserve(node); + }); + }, { rootMargin: "0px 0px -4% 0px", threshold: 0.04 }) + : null; + + const isVisible = (node) => !node.hidden && !node.closest("[hidden]"); + const registered = new Set(); + + profile.reveals.forEach(([selector, kind, limit], groupIndex) => { + const nodes = Array.from(document.querySelectorAll(selector)) + .filter((node) => isVisible(node) && !registered.has(node)) + .slice(0, limit); + nodes.forEach((node, index) => { + registered.add(node); + node.classList.add("motion-pending", `motion-${kind}`); + node.style.setProperty("--motion-order", String(Math.min(groupIndex + index, 9))); + const finishInitialMotion = (event) => { + if (event.target !== node) return; + node.classList.remove("motion-pending", "motion-rise", "motion-soft", "motion-in"); + node.removeEventListener("animationend", finishInitialMotion); + }; + node.addEventListener("animationend", finishInitialMotion); + if (revealObserver) { + revealObserver.observe(node); + } else { + node.classList.add("motion-in"); + } + }); + }); + + const replayClass = (node, className) => { + if (!node || reduceMotion || !isVisible(node)) return; + node.classList.remove(className); + window.requestAnimationFrame(() => { + node.classList.add(className); + const finishReplay = (event) => { + if (event.target !== node) return; + node.classList.remove(className); + node.removeEventListener("animationend", finishReplay); + }; + node.addEventListener("animationend", finishReplay); + }); + }; + + const liveValueSelectors = [ + "[data-task-live-header-status]", + "[data-task-live-progress-number]", + "[data-task-live-operation-label]", + "[data-task-live-operation-progress]", + "[data-task-live-candidate-count]", + "[data-task-live-output-count]", + "[data-worker-status]", + "[data-scheduler-runtime]", + "#subtitle-save-state", + ]; + + liveValueSelectors.forEach((selector) => { + document.querySelectorAll(selector).forEach((node) => { + let lastValue = node.textContent; + const observer = new MutationObserver(() => { + const nextValue = node.textContent; + if (nextValue === lastValue) return; + lastValue = nextValue; + replayClass(node, "motion-data-updated"); + }); + observer.observe(node, { childList: true, characterData: true, subtree: true }); + }); + }); + + const dynamicSelector = [ + ".ai-analysis-result-item", + ".ai-history-item", + ".schedule-preview-list > div", + ].join(","); + + const animateAddedContent = (root) => { + if (!(root instanceof Element)) return; + const candidates = []; + if (root.matches(dynamicSelector)) candidates.push(root); + candidates.push(...root.querySelectorAll(dynamicSelector)); + candidates.slice(0, 8).forEach((node, index) => { + if (node.dataset.motionDynamicSeen === "true" || !isVisible(node)) return; + node.dataset.motionDynamicSeen = "true"; + node.style.setProperty("--motion-order", String(Math.min(index, 7))); + replayClass(node, "motion-item-added"); + }); + }; + + const dynamicObserver = new MutationObserver((records) => { + records.forEach((record) => record.addedNodes.forEach(animateAddedContent)); + }); + dynamicObserver.observe(mainPanel, { childList: true, subtree: true }); + + const selectionProfile = document.querySelector("#selection-profile"); + const longLiveSettings = document.querySelector("#long-live-settings"); + selectionProfile?.addEventListener("change", () => { + if (longLiveSettings && !longLiveSettings.hidden) replayClass(longLiveSettings, "motion-item-added"); + }); + + const videoFileInput = document.querySelector("#video-file-input"); + const uploadPanel = document.querySelector("#upload-source-panel"); + videoFileInput?.addEventListener("change", () => replayClass(uploadPanel, "motion-confirm")); + + document.querySelectorAll("[data-center-tab]").forEach((button) => { + button.addEventListener("click", () => { + const panel = document.querySelector(`[data-center-panel="${CSS.escape(button.dataset.centerTab || "")}"]`); + replayClass(panel, "motion-panel-activated"); + }); + }); + + document.querySelectorAll("[data-prompt-preset-tab] input").forEach((input) => { + input.addEventListener("change", () => { + const card = document.querySelector(`[data-prompt-preset-card][data-preset-id="${CSS.escape(input.value)}"]`); + replayClass(card, "motion-panel-activated"); + }); + }); + + window.setTimeout(() => body.classList.add("motion-settled"), 620); +})(); diff --git a/app/static/js/publish-center.js b/app/static/js/publish-center.js index d9d2e76..1c656f6 100644 --- a/app/static/js/publish-center.js +++ b/app/static/js/publish-center.js @@ -100,7 +100,7 @@ if (publishCenterRoot) { backfillCoversButton.disabled = loading || count === 0; backfillCoversButton.textContent = loading ? `正在补充${platformLabel()} ${count} 条封面…` - : `一键补充${platformLabel()}缺失封面${count ? `(${count})` : ""}`; + : `补齐${platformLabel()}缺失封面${count ? `(${count})` : ""}`; } function beijingDatetimeValue(timestamp) { @@ -270,7 +270,7 @@ if (publishCenterRoot) { `[data-publish-row][data-section="schedule"][data-job-id="${CSS.escape(jobId || "")}"]`, ); if (!row || row.hidden) return; - row.scrollIntoView({ behavior: "smooth", block: "center" }); + row.scrollIntoView({ behavior: window.preferredScrollBehavior(), block: "center" }); row.classList.add("is-calendar-focus"); window.setTimeout(() => row.classList.remove("is-calendar-focus"), 1600); } @@ -1128,6 +1128,8 @@ if (publishCenterRoot) { panel.hidden = !active; panel.classList.toggle("active", active); }); + const batchAiButton = document.querySelector("[data-batch-ai]"); + if (batchAiButton) batchAiButton.hidden = tab !== "content"; if (tab === "history") void refreshHistory({ calendar: true, records: true }); } @@ -1337,7 +1339,7 @@ if (publishCenterRoot) { const readiness = effectiveReadiness(row); if (readiness.action === "start_worker") { document.querySelector("[data-worker-help]")?.removeAttribute("hidden"); - document.querySelector("[data-scheduler-health]")?.scrollIntoView({ behavior: "smooth", block: "center" }); + document.querySelector("[data-scheduler-health]")?.scrollIntoView({ behavior: window.preferredScrollBehavior(), block: "center" }); await refreshSchedulerHealth(true); return; } @@ -1375,7 +1377,7 @@ if (publishCenterRoot) { } switchTab("content"); const editorRow = document.querySelector(`[data-publish-row][data-section="content"][data-job-id="${CSS.escape(row.dataset.jobId)}"]`); - editorRow?.scrollIntoView({ behavior: "smooth", block: "center" }); + editorRow?.scrollIntoView({ behavior: window.preferredScrollBehavior(), block: "center" }); editorRow?.querySelector("[data-account-select]")?.focus(); showMessage("请在内容准备中选择本次使用的同平台账号并保存。", "error"); return; @@ -1384,7 +1386,7 @@ if (publishCenterRoot) { switchTab("content"); const editorRow = document.querySelector(`[data-publish-row][data-section="content"][data-job-id="${CSS.escape(row.dataset.jobId)}"]`); if (editorRow && !editorRow.hidden) { - editorRow.scrollIntoView({ behavior: "smooth", block: "center" }); + editorRow.scrollIntoView({ behavior: window.preferredScrollBehavior(), block: "center" }); editorRow.querySelector("input, textarea, select")?.focus(); } showMessage(readiness.message || "请先补齐发布内容并保存。", "error"); @@ -1400,24 +1402,6 @@ if (publishCenterRoot) { } } - async function upgradePendingDouyinMetadata() { - try { - const data = await window.apiFetch("/api/publish/jobs/metadata/upgrade-pending-douyin", { method: "POST" }); - (data.jobs || []).forEach((job) => { - const row = document.querySelector( - `[data-publish-row][data-section="content"][data-job-id="${CSS.escape(job.id || "")}"]`, - ); - applyGeneratedMetadataToForm(row, job); - updateRowFromJob(job); - }); - if (Number(data.upgraded_count || 0) > 0 || Number(data.failed_count || 0) > 0) { - showMessage(data.message || "抖音旧草稿文案升级完成。", data.failed_count ? "error" : "success"); - } - } catch (error) { - showMessage(`旧草稿文案升级已中止:${error.message}`, "error"); - } - } - async function refreshAccounts() { try { const data = await window.apiFetch("/api/publish/accounts"); @@ -1496,7 +1480,7 @@ if (publishCenterRoot) { if (!cell) return; selectedCalendarDate = cell.dataset.calendarDate || ""; renderCalendar(); - calendarDayDetail?.scrollIntoView({ behavior: "smooth", block: "nearest" }); + calendarDayDetail?.scrollIntoView({ behavior: window.preferredScrollBehavior(), block: "nearest" }); }); calendarNode?.addEventListener("keydown", (event) => { const cell = event.target.closest("[data-calendar-date]"); @@ -1830,7 +1814,7 @@ if (publishCenterRoot) { ); setTaskGroupExpanded(contentRow?.closest("[data-publish-task-group]"), true); switchTab("content"); - contentRow?.scrollIntoView({ behavior: "smooth", block: "center" }); + contentRow?.scrollIntoView({ behavior: window.preferredScrollBehavior(), block: "center" }); showMessage(data.message || "已取消发送并返回内容准备。", "success"); } catch (error) { showMessage(`取消发送失败:${error.message}`, "error"); @@ -1866,8 +1850,8 @@ if (publishCenterRoot) { const data = await window.apiFetch(`/api/publish/jobs/${jobId}/metadata?use_ai=true`, { method: "POST" }); applyGeneratedMetadataToForm(row, data.job); updateRowFromJob(data.job); - showMessage("AI 文案已重写并同步到最终发送字段。", "success"); - } catch (error) { showMessage(`AI 重写失败:${error.message}`, "error"); } + showMessage("本条 AI 文案已重写并保存到最终发送字段。", "success"); + } catch (error) { showMessage(`本条 AI 重写失败:${error.message}`, "error"); } finally { metadataButton.disabled = false; } return; } @@ -2052,7 +2036,7 @@ if (publishCenterRoot) { } } showMessage( - `批量 AI 重写完成:成功 ${succeeded} 条,失败 ${failed} 条;失败项已保留原文。`, + `已选文案 AI 重写完成:成功 ${succeeded} 条,失败 ${failed} 条;失败项已保留原文。`, failed ? "error" : "success", ); }); @@ -2149,7 +2133,11 @@ if (publishCenterRoot) { latestPreviewItems.forEach((item, index) => { const row = document.querySelector(`[data-publish-row][data-job-id="${CSS.escape(item.job_id)}"]`); const line = document.createElement("div"); - line.innerHTML = `第 ${index + 1} 条:${row?.querySelector("[data-row-title]")?.textContent || item.job_id}`; + const title = document.createElement("strong"); + const scheduledAt = document.createElement("time"); + title.textContent = `第 ${index + 1} 条:${row?.querySelector("[data-row-title]")?.textContent || item.job_id}`; + scheduledAt.textContent = item.scheduled_at_local_display || ""; + line.append(title, scheduledAt); previewList.appendChild(line); }); latestPreviewSignature = previewSignature(payload); @@ -2196,8 +2184,8 @@ if (publishCenterRoot) { const button = event.currentTarget; button.disabled = true; try { const data = await window.apiFetch(`/api/publish/queue/refresh?use_ai=false&platform=${encodeURIComponent(activePlatform)}`, { method: "POST" }); - showMessage(data.message || "缺失任务已补充,请稍后查看内容准备区。", "success"); - } catch (error) { showMessage(`补充任务失败:${error.message}`, "error"); } + showMessage(data.message || "遗漏切片已同步,请在内容准备区核对。", "success"); + } catch (error) { showMessage(`同步遗漏切片失败:${error.message}`, "error"); } finally { button.disabled = false; } }); @@ -2212,14 +2200,13 @@ if (publishCenterRoot) { if (scheduleForm?.elements.start_at_local) scheduleForm.elements.start_at_local.value = beijingDatetimeValue(Date.now() + 10 * 60 * 1000); document.querySelectorAll('[data-publish-row][data-section="schedule"], [data-publish-row][data-section="history"]').forEach((row) => applyRowReadiness(row)); updateSelectionUi(); applyHistoryFilter(); refreshScheduleViews(); updateBackfillCoversButton(); - void upgradePendingDouyinMetadata(); if (focus?.dataset.taskId) { const group = document.querySelector( `[data-publish-task-group][data-task-id="${CSS.escape(focus.dataset.taskId)}"]`, ); if (group && !group.hidden) { setTaskGroupExpanded(group, true); - group.scrollIntoView({ behavior: "smooth", block: "start" }); + group.scrollIntoView({ behavior: window.preferredScrollBehavior(), block: "start" }); } else { showMessage("已定位到该处理任务,但当前没有可准备的抖音新版本内容。可返回任务页重新同步。"); } diff --git a/app/static/js/subtitle-editor.js b/app/static/js/subtitle-editor.js new file mode 100644 index 0000000..2b5db9f --- /dev/null +++ b/app/static/js/subtitle-editor.js @@ -0,0 +1,802 @@ +(() => { + "use strict"; + + const root = document.querySelector("#subtitle-editor"); + if (!root) return; + + const elements = { + track: root.querySelector("#subtitle-track-select"), + saveState: root.querySelector("#subtitle-save-state"), + revisionMeta: root.querySelector("#subtitle-revision-meta"), + approve: root.querySelector("#subtitle-approve"), + video: root.querySelector("#subtitle-editor-video"), + overlay: root.querySelector("#subtitle-video-overlay"), + waveform: root.querySelector("#subtitle-waveform"), + waveformStatus: root.querySelector("#subtitle-waveform-status"), + list: root.querySelector("#subtitle-cue-list"), + spacer: root.querySelector("#subtitle-cue-spacer"), + viewport: root.querySelector("#subtitle-cue-viewport"), + quality: root.querySelector("#subtitle-quality-summary"), + search: root.querySelector("#subtitle-search"), + replacement: root.querySelector("#subtitle-replacement"), + replaceAll: root.querySelector("#subtitle-replace-all"), + add: root.querySelector("#subtitle-add"), + split: root.querySelector("#subtitle-split"), + merge: root.querySelector("#subtitle-merge"), + remove: root.querySelector("#subtitle-delete"), + shiftMs: root.querySelector("#subtitle-shift-ms"), + shift: root.querySelector("#subtitle-shift"), + undo: root.querySelector("#subtitle-undo"), + redo: root.querySelector("#subtitle-redo"), + aiSuggest: root.querySelector("#subtitle-ai-suggest"), + save: root.querySelector("#subtitle-save-now"), + importFile: root.querySelector("#subtitle-import-file"), + exports: ["srt", "vtt", "ass"].map((format) => [ + format, + root.querySelector(`#subtitle-export-${format}`), + ]), + aiPanel: root.querySelector("#subtitle-ai-suggestion-panel"), + aiSummary: root.querySelector("#subtitle-ai-suggestion-summary"), + aiDiffList: root.querySelector("#subtitle-ai-diff-list"), + aiSelectAll: root.querySelector("#subtitle-ai-select-all"), + aiAccept: root.querySelector("#subtitle-ai-accept"), + aiClose: root.querySelector("#subtitle-ai-close"), + }; + + const ROW_HEIGHT = 132; + const PAGE_SIZE = 2000; + const state = { + taskId: root.dataset.taskId, + tracks: [], + track: null, + revision: null, + cues: [], + visibleIndices: [], + selectedIds: new Set(), + currentCueId: null, + undo: [], + redo: [], + waveSurfer: null, + regions: null, + selectedRegion: null, + saveTimer: null, + saving: false, + dirty: false, + changeVersion: 0, + requestToken: 0, + suggestion: null, + }; + + const batch = { + root: document.querySelector("#subtitle-review-gate"), + approve: document.querySelector("#subtitle-batch-approve-render"), + skip: document.querySelector("#subtitle-skip-and-resume"), + panel: document.querySelector("#subtitle-batch-job"), + bar: document.querySelector("#subtitle-batch-progress-bar"), + label: document.querySelector("#subtitle-batch-progress-label"), + message: document.querySelector("#subtitle-batch-message"), + cancel: document.querySelector("#subtitle-batch-cancel"), + retry: document.querySelector("#subtitle-batch-retry"), + jobId: "", + pollTimer: null, + }; + + function setStatus(message, tone = "blue") { + elements.saveState.textContent = message; + elements.saveState.className = `status-pill tone-${tone}`; + } + + async function api(url, options = {}) { + const response = await fetch(url, options); + let payload = {}; + try { + payload = await response.json(); + } catch (_error) { + payload = {}; + } + if (!response.ok) { + const detail = typeof payload.detail === "string" ? payload.detail : "字幕请求失败"; + const error = new Error(detail); + error.status = response.status; + throw error; + } + return payload; + } + + function cueSnapshot() { + return state.cues.map((cue) => ({ + ...cue, + start_ms: Number(cue.start_ms), + end_ms: Number(cue.end_ms), + })); + } + + function restoreSnapshot(snapshot) { + state.cues = snapshot.map((cue) => ({ ...cue })); + state.selectedIds.clear(); + state.currentCueId = null; + updateAiButton(); + markChanged(); + applySearch(); + updateCurrentCue(); + } + + function mutate(callback) { + state.undo.push(cueSnapshot()); + if (state.undo.length > 50) state.undo.shift(); + state.redo = []; + callback(); + state.cues.sort((left, right) => left.start_ms - right.start_ms || left.end_ms - right.end_ms); + markChanged(); + applySearch(); + updateUndoButtons(); + } + + function markChanged(schedule = true) { + state.dirty = true; + state.changeVersion += 1; + elements.save.disabled = false; + setStatus("有未保存修改", "amber"); + renderQuality(); + if (schedule) { + window.clearTimeout(state.saveTimer); + state.saveTimer = window.setTimeout(() => saveRevision(false), 1800); + } + } + + function updateUndoButtons() { + elements.undo.disabled = state.undo.length === 0; + elements.redo.disabled = state.redo.length === 0; + } + + function applySearch() { + const query = elements.search.value.trim().toLocaleLowerCase(); + state.visibleIndices = state.cues + .map((cue, index) => ({ cue, index })) + .filter(({ cue }) => !query || `${cue.text} ${cue.speaker || ""}`.toLocaleLowerCase().includes(query)) + .map(({ index }) => index); + elements.spacer.style.height = `${state.visibleIndices.length * ROW_HEIGHT}px`; + renderVirtualRows(); + } + + function formatMs(value) { + const total = Math.max(0, Number(value) || 0); + const hours = Math.floor(total / 3600000); + const minutes = Math.floor((total % 3600000) / 60000); + const seconds = Math.floor((total % 60000) / 1000); + const milliseconds = Math.floor(total % 1000); + return `${String(hours).padStart(2, "0")}:${String(minutes).padStart(2, "0")}:${String(seconds).padStart(2, "0")}.${String(milliseconds).padStart(3, "0")}`; + } + + function escapeHtml(value) { + return String(value ?? "") + .replaceAll("&", "&") + .replaceAll("<", "<") + .replaceAll(">", ">") + .replaceAll('"', """) + .replaceAll("'", "'"); + } + + function cueIssues(cue, previous) { + const issues = []; + const duration = cue.end_ms - cue.start_ms; + const lines = String(cue.text || "").split(/\r?\n/); + const chinese = (value) => (String(value).match(/[\u4e00-\u9fff]/g) || []).length; + if (lines.length > 2) issues.push("超过 2 行"); + if (lines.some((line) => chinese(line) > 18)) issues.push("单行超过 18 个中文字符"); + if (duration < 800) issues.push("短于 800ms"); + if (duration > 7000) issues.push("长于 7 秒"); + if (chinese(cue.text) / Math.max(0.001, duration / 1000) > 12) issues.push("阅读速度过快"); + if (previous) { + const gap = cue.start_ms - previous.end_ms; + if (gap < 0) issues.push("与上一条重叠"); + else if (gap < 80) issues.push("间隔小于 80ms"); + } + return issues; + } + + function renderVirtualRows() { + updateAiButton(); + if (!state.visibleIndices.length) { + elements.viewport.innerHTML = '
没有匹配的字幕行
'; + return; + } + const scrollTop = elements.list.scrollTop; + const first = Math.max(0, Math.floor(scrollTop / ROW_HEIGHT) - 4); + const count = Math.ceil(elements.list.clientHeight / ROW_HEIGHT) + 8; + const last = Math.min(state.visibleIndices.length, first + count); + const rows = []; + for (let virtualIndex = first; virtualIndex < last; virtualIndex += 1) { + const cueIndex = state.visibleIndices[virtualIndex]; + const cue = state.cues[cueIndex]; + const previous = cueIndex > 0 ? state.cues[cueIndex - 1] : null; + const issues = cueIssues(cue, previous); + const isSelected = state.selectedIds.has(cue.id); + const isCurrent = state.currentCueId === cue.id; + rows.push(` +
+ + +
+ + +
+ + +
${issues.length ? escapeHtml(issues.join(" · ")) : "时间与阅读速度正常"}
+
+ `); + } + elements.viewport.innerHTML = rows.join(""); + bindVisibleRows(); + } + + function bindVisibleRows() { + elements.viewport.querySelectorAll(".subtitle-cue-row").forEach((row) => { + const cue = state.cues.find((item) => item.id === row.dataset.cueId); + if (!cue) return; + row.querySelector("[data-cue-select]").addEventListener("change", (event) => { + if (event.target.checked) state.selectedIds.add(cue.id); + else state.selectedIds.delete(cue.id); + updateAiButton(); + renderVirtualRows(); + }); + row.querySelector("[data-cue-seek]").addEventListener("click", () => selectCue(cue, true)); + row.querySelectorAll("[data-cue-field]").forEach((input) => { + let editStarted = false; + let beforeEdit = null; + input.addEventListener("focus", () => { + editStarted = false; + beforeEdit = cueSnapshot(); + }); + input.addEventListener("input", () => { + if (!editStarted) { + state.undo.push(beforeEdit || cueSnapshot()); + if (state.undo.length > 50) state.undo.shift(); + state.redo = []; + editStarted = true; + updateUndoButtons(); + } + const field = input.dataset.cueField; + cue[field] = field.endsWith("_ms") ? Number(input.value) : input.value; + markChanged(); + }); + input.addEventListener("change", () => { + if (cue.end_ms <= cue.start_ms) cue.end_ms = cue.start_ms + 1; + state.cues.sort((left, right) => left.start_ms - right.start_ms || left.end_ms - right.end_ms); + applySearch(); + selectCue(cue, false); + }); + }); + }); + } + + function renderQuality() { + let errors = 0; + let warnings = 0; + state.cues.forEach((cue, index) => { + const issues = cueIssues(cue, index ? state.cues[index - 1] : null); + warnings += issues.length; + if (issues.includes("与上一条重叠")) errors += 1; + }); + elements.quality.className = `subtitle-quality-summary${errors ? " has-errors" : warnings ? " has-warnings" : " is-clean"}`; + elements.quality.textContent = errors + ? `${errors} 个重叠错误,另有 ${Math.max(0, warnings - errors)} 条质量提醒;请先处理红色问题。` + : warnings + ? `没有重叠错误,共 ${warnings} 条时长、间隔、行长或阅读速度提醒。` + : `共 ${state.cues.length} 条字幕,当前没有发现质量问题。`; + } + + function findCurrentCue(timeMs) { + let low = 0; + let high = state.cues.length - 1; + let candidate = null; + while (low <= high) { + const middle = Math.floor((low + high) / 2); + if (state.cues[middle].start_ms <= timeMs) { + candidate = state.cues[middle]; + low = middle + 1; + } else { + high = middle - 1; + } + } + return candidate && candidate.end_ms >= timeMs ? candidate : null; + } + + function updateCurrentCue() { + const cue = findCurrentCue((elements.video.currentTime || 0) * 1000); + const nextId = cue?.id || null; + if (state.currentCueId !== nextId) { + state.currentCueId = nextId; + renderVirtualRows(); + } + elements.overlay.textContent = cue ? `${cue.speaker ? `${cue.speaker}:` : ""}${cue.text}` : ""; + } + + function selectCue(cue, seek) { + state.currentCueId = cue.id; + if (seek) elements.video.currentTime = cue.start_ms / 1000; + elements.overlay.textContent = `${cue.speaker ? `${cue.speaker}:` : ""}${cue.text}`; + renderSelectedRegion(cue); + const visibleIndex = state.visibleIndices.indexOf(state.cues.indexOf(cue)); + if (visibleIndex >= 0) { + const top = visibleIndex * ROW_HEIGHT; + if (top < elements.list.scrollTop || top + ROW_HEIGHT > elements.list.scrollTop + elements.list.clientHeight) { + elements.list.scrollTop = Math.max(0, top - ROW_HEIGHT); + } + } + renderVirtualRows(); + } + + function renderSelectedRegion(cue) { + if (!state.regions) return; + state.regions.clearRegions(); + state.selectedRegion = state.regions.addRegion({ + id: `cue-${cue.id}`, + start: cue.start_ms / 1000, + end: cue.end_ms / 1000, + color: "rgba(38, 118, 255, 0.22)", + drag: true, + resize: true, + }); + } + + async function loadTracks() { + setStatus("正在生成或读取统一字幕轨…", "blue"); + try { + const payload = await api(`/api/subtitles/tasks/${encodeURIComponent(state.taskId)}/tracks`); + state.tracks = payload.tracks || []; + elements.track.innerHTML = state.tracks.map((track) => { + const label = track.track_type === "source" ? "原片主字幕" : `切片 · ${track.output_file_name || track.name}`; + const suffix = track.has_manual_edits ? "(人工版)" : track.sync_status === "pending_sync" ? "(待同步)" : ""; + return ``; + }).join(""); + elements.track.disabled = state.tracks.length === 0; + if (!state.tracks.length) throw new Error("当前任务没有可用字幕轨"); + await loadTrack(state.tracks[0].id); + } catch (error) { + setStatus(error.message, "red"); + elements.revisionMeta.textContent = "请先完成结构化转写并生成切片。"; + } + } + + async function loadTrack(trackId) { + const token = ++state.requestToken; + window.clearTimeout(state.saveTimer); + setStatus("正在载入毫秒级字幕…", "blue"); + const track = state.tracks.find((item) => item.id === trackId); + if (!track) return; + try { + const first = await api(`/api/subtitles/tracks/${encodeURIComponent(trackId)}/cues?offset=0&limit=${PAGE_SIZE}`); + const cues = [...(first.cues || [])]; + for (let offset = cues.length; offset < Number(first.total || 0); offset += PAGE_SIZE) { + const page = await api(`/api/subtitles/tracks/${encodeURIComponent(trackId)}/cues?offset=${offset}&limit=${PAGE_SIZE}`); + cues.push(...(page.cues || [])); + } + if (token !== state.requestToken) return; + state.track = first.track; + state.revision = first.revision; + state.cues = cues; + state.selectedIds.clear(); + state.suggestion = null; + state.currentCueId = null; + state.undo = []; + state.redo = []; + state.dirty = false; + state.changeVersion = 0; + elements.track.value = trackId; + elements.video.src = state.track.media_url; + elements.approve.disabled = !state.revision; + updateAiButton(); + elements.save.disabled = true; + elements.exports.forEach(([, button]) => { button.disabled = !state.revision; }); + renderRevisionMeta(); + applySearch(); + renderQuality(); + updateUndoButtons(); + setStatus(`已载入 ${state.cues.length} 条,自动保存已开启`, "green"); + loadWaveform(token); + } catch (error) { + setStatus(error.message, "red"); + } + } + + function renderRevisionMeta() { + if (!state.revision) { + elements.revisionMeta.textContent = "尚无 revision"; + return; + } + const status = state.revision.status === "approved" ? "已审核" : "草稿"; + elements.revisionMeta.textContent = `Revision ${state.revision.revision_number} · ${status} · ${state.cues.length} 条 · ${state.track.sync_status}`; + } + + async function loadWaveform(token) { + if (state.waveSurfer) { + state.waveSurfer.destroy(); + state.waveSurfer = null; + state.regions = null; + } + elements.waveform.innerHTML = ""; + document.querySelector("#subtitle-timeline").innerHTML = ""; + elements.waveformStatus.textContent = "正在读取 peaks…"; + try { + const payload = await api(`${state.track.peaks_url}?max_points=12000`); + if (token !== state.requestToken) return; + if (!window.WaveSurfer || !window.WaveSurfer.Regions || !window.WaveSurfer.Timeline) { + throw new Error("本地 wavesurfer.js 未正确载入"); + } + state.regions = window.WaveSurfer.Regions.create(); + const timeline = window.WaveSurfer.Timeline.create({ container: "#subtitle-timeline", height: 24 }); + state.waveSurfer = window.WaveSurfer.create({ + container: elements.waveform, + media: elements.video, + peaks: [Float32Array.from(payload.peaks || [])], + duration: Number(payload.duration_ms || 0) / 1000, + height: 92, + minPxPerSec: state.track.track_type === "source" ? 0.08 : 8, + normalize: true, + waveColor: "#9fbce8", + progressColor: "#2676ff", + cursorColor: "#ff9f0a", + plugins: [state.regions, timeline], + }); + state.regions.on("region-update-end", (region) => { + if (!state.currentCueId) return; + const cue = state.cues.find((item) => item.id === state.currentCueId); + if (!cue) return; + mutate(() => { + cue.start_ms = Math.max(0, Math.round(region.start * 1000)); + cue.end_ms = Math.max(cue.start_ms + 1, Math.round(region.end * 1000)); + }); + }); + elements.waveformStatus.textContent = `${payload.point_count} 个 peaks${payload.cached ? " · 已复用缓存" : " · 新生成"}`; + } catch (error) { + elements.waveformStatus.textContent = `波形暂不可用:${error.message}`; + } + } + + async function saveRevision(force) { + if (!state.dirty || state.saving || !state.track || !state.revision) return; + window.clearTimeout(state.saveTimer); + state.saving = true; + const version = state.changeVersion; + const cues = cueSnapshot(); + const baseRevisionId = state.revision.id; + setStatus("正在自动保存新 revision…", "blue"); + try { + const payload = await api(`/api/subtitles/tracks/${encodeURIComponent(state.track.id)}/revisions`, { + method: "POST", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify({ base_revision_id: baseRevisionId, cues, note: force ? "手动立即保存" : "字幕编辑器自动保存" }), + }); + state.revision = payload.revision; + if (version === state.changeVersion) { + state.cues = (payload.revision.cues || []).map((cue) => ({ ...cue })); + state.dirty = false; + state.selectedIds.clear(); + updateAiButton(); + elements.save.disabled = true; + applySearch(); + setStatus(`Revision ${state.revision.revision_number} 已保存`, "green"); + } else { + state.dirty = true; + setStatus("保存期间又有修改,正在继续保存…", "amber"); + } + renderRevisionMeta(); + } catch (error) { + setStatus(error.status === 409 ? "版本已变化,请重新选择字幕轨后再编辑" : `保存失败:${error.message}`, "red"); + } finally { + state.saving = false; + if (state.dirty && version !== state.changeVersion) { + state.saveTimer = window.setTimeout(() => saveRevision(false), 300); + } + } + } + + function selectedCues() { + return state.cues.filter((cue) => state.selectedIds.has(cue.id)); + } + + function updateAiButton() { + if (!elements.aiSuggest) return; + elements.aiSuggest.disabled = !state.revision || state.selectedIds.size === 0 || state.selectedIds.size > 500; + } + + elements.track.addEventListener("change", () => loadTrack(elements.track.value)); + elements.list.addEventListener("scroll", renderVirtualRows, { passive: true }); + elements.video.addEventListener("timeupdate", updateCurrentCue); + elements.search.addEventListener("input", applySearch); + elements.save.addEventListener("click", () => saveRevision(true)); + + elements.undo.addEventListener("click", () => { + if (!state.undo.length) return; + state.redo.push(cueSnapshot()); + restoreSnapshot(state.undo.pop()); + updateUndoButtons(); + }); + elements.redo.addEventListener("click", () => { + if (!state.redo.length) return; + state.undo.push(cueSnapshot()); + restoreSnapshot(state.redo.pop()); + updateUndoButtons(); + }); + + elements.add.addEventListener("click", () => { + const start = Math.max(0, Math.round((elements.video.currentTime || 0) * 1000)); + const cue = { id: `local-${crypto.randomUUID()}`, start_ms: start, end_ms: start + 2000, text: "新字幕", speaker: "", confidence: null, source_cue_id: null }; + mutate(() => state.cues.push(cue)); + selectCue(cue, false); + }); + + elements.remove.addEventListener("click", () => { + if (!state.selectedIds.size) return setStatus("请先勾选要删除的字幕行", "amber"); + mutate(() => { state.cues = state.cues.filter((cue) => !state.selectedIds.has(cue.id)); }); + state.selectedIds.clear(); + updateAiButton(); + }); + + elements.merge.addEventListener("click", () => { + const cues = selectedCues().sort((left, right) => left.start_ms - right.start_ms); + if (cues.length < 2) return setStatus("合并至少需要勾选两行", "amber"); + mutate(() => { + const merged = { ...cues[0], end_ms: Math.max(...cues.map((cue) => cue.end_ms)), text: cues.map((cue) => cue.text).join(" ") }; + const ids = new Set(cues.map((cue) => cue.id)); + state.cues = state.cues.filter((cue) => !ids.has(cue.id)); + state.cues.push(merged); + state.selectedIds = new Set([merged.id]); + }); + }); + + elements.split.addEventListener("click", () => { + const cue = state.cues.find((item) => item.id === state.currentCueId) || selectedCues()[0]; + if (!cue) return setStatus("请先点击一行字幕", "amber"); + let splitMs = Math.round((elements.video.currentTime || 0) * 1000); + if (splitMs <= cue.start_ms || splitMs >= cue.end_ms) splitMs = Math.round((cue.start_ms + cue.end_ms) / 2); + const middle = Math.max(1, Math.floor(cue.text.length / 2)); + const originalEndMs = cue.end_ms; + mutate(() => { + cue.end_ms = splitMs; + const second = { ...cue, id: `local-${crypto.randomUUID()}`, start_ms: splitMs, end_ms: originalEndMs, text: cue.text.slice(middle).trim() || cue.text }; + cue.text = cue.text.slice(0, middle).trim() || cue.text; + state.cues.push(second); + }); + }); + + elements.shift.addEventListener("click", () => { + const delta = Number(elements.shiftMs.value || 0); + if (!Number.isFinite(delta) || delta === 0) return setStatus("请输入非 0 的毫秒位移", "amber"); + const selected = state.selectedIds; + mutate(() => state.cues.forEach((cue) => { + if (!selected.size || selected.has(cue.id)) { + const duration = cue.end_ms - cue.start_ms; + cue.start_ms = Math.max(0, cue.start_ms + delta); + cue.end_ms = cue.start_ms + duration; + } + })); + }); + + elements.replaceAll.addEventListener("click", () => { + const search = elements.search.value; + if (!search) return setStatus("请先输入要搜索的文字", "amber"); + const replacement = elements.replacement.value; + mutate(() => state.cues.forEach((cue) => { cue.text = cue.text.split(search).join(replacement); })); + }); + + elements.approve.addEventListener("click", async () => { + if (state.dirty) await saveRevision(false); + if (state.dirty || !state.revision) return; + try { + const payload = await api(`/api/subtitles/tracks/${encodeURIComponent(state.track.id)}/approve`, { + method: "POST", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify({ revision_id: state.revision.id }), + }); + state.revision = payload.revision; + renderRevisionMeta(); + setStatus(`Revision ${state.revision.revision_number} 已审核`, "green"); + } catch (error) { + setStatus(`审核失败:${error.message}`, "red"); + } + }); + + elements.aiSuggest?.addEventListener("click", async () => { + if (state.dirty) await saveRevision(false); + const cueIds = [...state.selectedIds]; + if (state.dirty || !state.revision || !cueIds.length) return; + elements.aiSuggest.disabled = true; + setStatus(`正在分批生成 ${cueIds.length} 条 AI 建议…`, "blue"); + try { + const payload = await api(`/api/subtitles/tracks/${encodeURIComponent(state.track.id)}/ai-suggestions`, { + method: "POST", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify({ revision_id: state.revision.id, cue_ids: cueIds, instructions: "" }), + }); + state.suggestion = payload; + renderAiSuggestionDiff(); + setStatus(`AI 返回 ${payload.diff?.length || 0} 条文字差异,尚未应用`, "green"); + } catch (error) { + setStatus(`AI 建议失败:${error.message}`, "red"); + } finally { + updateAiButton(); + } + }); + + function renderAiSuggestionDiff() { + const diff = state.suggestion?.diff || []; + elements.aiPanel.hidden = false; + elements.aiSummary.textContent = diff.length + ? `共 ${diff.length} 条文字差异。建议版本不会覆盖当前人工版本,请勾选后接受。` + : "AI 没有建议修改;当前字幕保持不变。"; + elements.aiDiffList.innerHTML = diff.length + ? diff.map((item) => ` + + `).join("") + : '

没有差异可接受。

'; + elements.aiAccept.disabled = diff.length === 0; + } + + elements.aiSelectAll?.addEventListener("click", () => { + elements.aiDiffList.querySelectorAll("[data-ai-diff-cue]").forEach((input) => { input.checked = true; }); + }); + + elements.aiClose?.addEventListener("click", () => { + elements.aiPanel.hidden = true; + state.suggestion = null; + }); + + elements.aiAccept?.addEventListener("click", async () => { + if (!state.suggestion || !state.revision) return; + const cueIds = [...elements.aiDiffList.querySelectorAll("[data-ai-diff-cue]:checked")].map((input) => input.value); + if (!cueIds.length) return setStatus("请至少勾选一条 AI 文字建议", "amber"); + elements.aiAccept.disabled = true; + try { + const payload = await api( + `/api/subtitles/tracks/${encodeURIComponent(state.track.id)}/ai-suggestions/${encodeURIComponent(state.suggestion.revision.id)}/accept`, + { + method: "POST", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify({ base_revision_id: state.suggestion.base_revision_id, cue_ids: cueIds }), + }, + ); + state.suggestion = null; + elements.aiPanel.hidden = true; + await loadTrack(state.track.id); + setStatus(`已接受 ${cueIds.length} 条建议并创建人工草稿 Revision ${payload.revision.revision_number}`, "green"); + } catch (error) { + setStatus(`接受 AI 建议失败:${error.message}`, "red"); + elements.aiAccept.disabled = false; + } + }); + + elements.importFile.addEventListener("change", async () => { + const file = elements.importFile.files?.[0]; + if (!file || !state.track) return; + const form = new FormData(); + form.append("file", file); + setStatus(`正在导入 ${file.name}…`, "blue"); + try { + await api(`/api/subtitles/tracks/${encodeURIComponent(state.track.id)}/import`, { method: "POST", body: form }); + await loadTrack(state.track.id); + } catch (error) { + setStatus(`导入失败:${error.message}`, "red"); + } finally { + elements.importFile.value = ""; + } + }); + + elements.exports.forEach(([format, button]) => button.addEventListener("click", () => { + if (!state.track || !state.revision) return; + window.location.href = `/api/subtitles/tracks/${encodeURIComponent(state.track.id)}/export?format_name=${format}&revision_id=${encodeURIComponent(state.revision.id)}`; + })); + + function renderBatchJob(job) { + if (!batch.panel || !job) return; + batch.panel.hidden = false; + batch.jobId = job.id; + const progress = Math.max(0, Math.min(100, Number(job.progress || 0))); + batch.bar.style.width = `${progress}%`; + batch.label.textContent = `${job.status_label || job.status} · ${progress}%`; + batch.message.textContent = job.error_message || job.message || "字幕任务状态已更新"; + const active = job.status === "queued" || job.status === "running"; + batch.cancel.hidden = !active; + batch.retry.hidden = !(job.status === "failed" || job.status === "cancelled"); + batch.approve.disabled = active; + batch.skip.disabled = active; + } + + async function pollBatchJob(jobId) { + window.clearTimeout(batch.pollTimer); + try { + const job = await api(`/api/tasks/jobs/${encodeURIComponent(jobId)}`); + renderBatchJob(job); + if (job.status === "queued" || job.status === "running") { + batch.pollTimer = window.setTimeout(() => pollBatchJob(jobId), 1500); + } else if (job.status === "completed") { + batch.message.textContent = "字幕成片全部验证通过,自动流水线已恢复。即将返回任务详情。"; + window.setTimeout(() => { window.location.href = `/tasks/${encodeURIComponent(state.taskId)}`; }, 1600); + } + } catch (error) { + batch.message.textContent = `读取字幕 Job 失败:${error.message}`; + batch.pollTimer = window.setTimeout(() => pollBatchJob(jobId), 3000); + } + } + + batch.approve?.addEventListener("click", async () => { + if (state.dirty) await saveRevision(false); + if (state.dirty) return; + if (!window.confirm("确认审核所有切片的当前字幕版本并批量烧录吗?全部验证通过后流水线会自动继续。")) return; + batch.approve.disabled = true; + batch.skip.disabled = true; + try { + const payload = await api(`/api/subtitles/tasks/${encodeURIComponent(state.taskId)}/approve-and-render`, { + method: "POST", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify({ approve_active_revisions: true }), + }); + renderBatchJob(payload.job); + pollBatchJob(payload.job_id); + } catch (error) { + batch.message.textContent = `批量烧录未启动:${error.message}`; + batch.panel.hidden = false; + batch.approve.disabled = false; + batch.skip.disabled = false; + } + }); + + batch.skip?.addEventListener("click", async () => { + if (!window.confirm("确认跳过字幕并进入片段审核吗?审核保存后才会同步发送中心。")) return; + batch.skip.disabled = true; + try { + const payload = await api(`/api/subtitles/tasks/${encodeURIComponent(state.taskId)}/skip-to-review`, { method: "POST" }); + window.location.href = payload.review_url || `/tasks/${encodeURIComponent(state.taskId)}/clips/review`; + } catch (error) { + batch.message.textContent = `跳过字幕失败:${error.message}`; + batch.panel.hidden = false; + batch.skip.disabled = false; + } + }); + + batch.cancel?.addEventListener("click", async () => { + if (!batch.jobId || !window.confirm("确认取消当前字幕烧录吗?已完成并验证的切片会保留,可稍后重试缺失部分。")) return; + const payload = await api(`/api/tasks/jobs/${encodeURIComponent(batch.jobId)}/cancel`, { method: "POST" }); + renderBatchJob(payload.job); + pollBatchJob(batch.jobId); + }); + + batch.retry?.addEventListener("click", async () => { + if (!batch.jobId) return; + const payload = await api(`/api/tasks/jobs/${encodeURIComponent(batch.jobId)}/retry`, { method: "POST" }); + renderBatchJob(payload.job); + pollBatchJob(batch.jobId); + }); + + async function loadLatestBatchJob() { + if (!batch.root) return; + try { + const payload = await api(`/api/subtitles/tasks/${encodeURIComponent(state.taskId)}/jobs`); + const latest = (payload.jobs || [])[0]; + if (latest) { + renderBatchJob(latest); + if (latest.status === "queued" || latest.status === "running") pollBatchJob(latest.id); + } + } catch (_error) { + // 页面仍可编辑字幕;Job 状态读取失败时由用户再次点击触发明确错误。 + } + } + + window.addEventListener("beforeunload", (event) => { + if (!state.dirty) return; + event.preventDefault(); + event.returnValue = ""; + }); + + loadTracks(); + loadLatestBatchJob(); +})(); diff --git a/app/static/vendor/wavesurfer/LICENSE b/app/static/vendor/wavesurfer/LICENSE new file mode 100644 index 0000000..88998ae --- /dev/null +++ b/app/static/vendor/wavesurfer/LICENSE @@ -0,0 +1,29 @@ +BSD 3-Clause License + +Copyright (c) 2012-2023, katspaugh and contributors +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +* Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +* Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +* Neither the name of the copyright holder nor the names of its + contributors may be used to endorse or promote products derived from + this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. diff --git a/app/static/vendor/wavesurfer/regions.min.js b/app/static/vendor/wavesurfer/regions.min.js new file mode 100644 index 0000000..ff8303e --- /dev/null +++ b/app/static/vendor/wavesurfer/regions.min.js @@ -0,0 +1 @@ +!function(t,e){"object"==typeof exports&&"undefined"!=typeof module?module.exports=e():"function"==typeof define&&define.amd?define(e):((t="undefined"!=typeof globalThis?globalThis:t||self).WaveSurfer=t.WaveSurfer||{},t.WaveSurfer.Regions=e())}(this,(function(){"use strict";class t{constructor(){this.listeners={}}on(t,e,i){if(this.listeners[t]||(this.listeners[t]=new Set),null==i?void 0:i.once){const i=(...n)=>{this.un(t,i),e(...n)};return this.listeners[t].add(i),()=>this.un(t,i)}return this.listeners[t].add(e),()=>this.un(t,e)}un(t,e){var i;null===(i=this.listeners[t])||void 0===i||i.delete(e)}once(t,e){return this.on(t,e,{once:!0})}unAll(){this.listeners={}}emit(t,...e){this.listeners[t]&&this.listeners[t].forEach((t=>t(...e)))}}class e extends t{constructor(t){super(),this.subscriptions=[],this.isDestroyed=!1,this.options=t}onInit(){}_init(t){this.isDestroyed&&(this.subscriptions=[],this.isDestroyed=!1),this.wavesurfer=t,this.onInit()}destroy(){this.emit("destroy"),this.subscriptions.forEach((t=>t())),this.subscriptions=[],this.isDestroyed=!0,this.wavesurfer=void 0}}function i(t,e){const n=e.xmlns?document.createElementNS(e.xmlns,t):document.createElement(t);for(const[t,s]of Object.entries(e))if("children"===t&&s)for(const[t,e]of Object.entries(s))e instanceof Node?n.appendChild(e):"string"==typeof e?n.appendChild(document.createTextNode(e)):n.appendChild(i(t,e));else"style"===t?Object.assign(n.style,s):"textContent"===t?n.textContent=s:n.setAttribute(t,s.toString());return n}function n(t,e,n){const s=i(t,e||{});return null==n||n.appendChild(s),s}function s(t){let e=t;const i=new Set;return{get value(){return e},set(t){Object.is(e,t)||(e=t,i.forEach((t=>t(e))))},update(t){this.set(t(e))},subscribe:t=>(i.add(t),()=>i.delete(t))}}function r(t,e){let i;const n=()=>{i&&(i(),i=void 0),i=t()},s=e.map((t=>t.subscribe(n)));return n(),()=>{i&&(i(),i=void 0),s.forEach((t=>t()))}}function o(t,e){const i=s(null),n=t=>{i.set(t)};return t.addEventListener(e,n),i._cleanup=()=>{t.removeEventListener(e,n)},i}function l(t){const e=t._cleanup;"function"==typeof e&&e()}function h(t,e={}){const{threshold:i=3,mouseButton:n=0,touchDelay:r=100}=e,o=s(null),h=new Map,a=matchMedia("(pointer: coarse)").matches;let d=()=>{};const c=e=>{if(e.button!==n)return;if(h.has(e.pointerId))return;if(h.set(e.pointerId,e),h.size>1)return;const s=e.pointerId;let l=e.clientX,c=e.clientY,u=!1;const p=Date.now(),v=t.getBoundingClientRect(),{left:g,top:m}=v,f=t=>{if(t.pointerId!==s)return;if(t.defaultPrevented||h.size>1)return;if(a&&Date.now()-pi||Math.abs(v)>i)&&(t.preventDefault(),t.stopPropagation(),u||(o.set({type:"start",x:l-g,y:c-m}),u=!0),o.set({type:"move",x:e-g,y:n-m,deltaX:d,deltaY:v}),l=e,c=n)},b=t=>{if(h.delete(t.pointerId)){if(t.pointerId===s&&u){const e=t.clientX,i=t.clientY;o.set({type:"end",x:e-g,y:i-m})}0===h.size&&d()}},E=t=>{t.relatedTarget&&t.relatedTarget!==document.documentElement||b(t)},C=t=>{u&&(t.stopPropagation(),t.preventDefault())},L=t=>{t.defaultPrevented||h.size>1||u&&t.preventDefault()};document.addEventListener("pointermove",f),document.addEventListener("pointerup",b),document.addEventListener("pointerout",E),document.addEventListener("pointercancel",E),document.addEventListener("touchmove",L,{passive:!1}),document.addEventListener("click",C,{capture:!0}),d=()=>{document.removeEventListener("pointermove",f),document.removeEventListener("pointerup",b),document.removeEventListener("pointerout",E),document.removeEventListener("pointercancel",E),document.removeEventListener("touchmove",L),setTimeout((()=>{document.removeEventListener("click",C,{capture:!0})}),10)}};t.addEventListener("pointerdown",c);return{signal:o,cleanup:()=>{d(),t.removeEventListener("pointerdown",c),h.clear(),l(o)}}}class a extends t{constructor(t,e,i=0){var n,s,r,o,l,h,a,d,c,u;super(),this.totalDuration=e,this.numberOfChannels=i,this.element=null,this.minLength=0,this.maxLength=1/0,this.contentEditable=!1,this.subscriptions=[],this.updatingSide=void 0,this.isRemoved=!1,this.subscriptions=[],this.id=t.id||`region-${Math.random().toString(32).slice(2)}`,this.start=this.clampPosition(t.start),this.end=this.clampPosition(null!==(n=t.end)&&void 0!==n?n:t.start),this.drag=null===(s=t.drag)||void 0===s||s,this.resize=null===(r=t.resize)||void 0===r||r,this.resizeStart=null===(o=t.resizeStart)||void 0===o||o,this.resizeEnd=null===(l=t.resizeEnd)||void 0===l||l,this.color=null!==(h=t.color)&&void 0!==h?h:"rgba(0, 0, 0, 0.1)",this.minLength=null!==(a=t.minLength)&&void 0!==a?a:this.minLength,this.maxLength=null!==(d=t.maxLength)&&void 0!==d?d:this.maxLength,this.channelIdx=null!==(c=t.channelIdx)&&void 0!==c?c:-1,this.contentEditable=null!==(u=t.contentEditable)&&void 0!==u?u:this.contentEditable,this.element=this.initElement(),this.setContent(t.content),this.setPart(),this.renderPosition(),this.initMouseEvents()}clampPosition(t){return Math.max(0,Math.min(this.totalDuration,t))}setPart(){var t;const e=this.start===this.end;null===(t=this.element)||void 0===t||t.setAttribute("part",`${e?"marker":"region"} ${this.id}`)}addResizeHandles(t){const e={position:"absolute",zIndex:"2",width:"6px",height:"100%",top:"0",cursor:"ew-resize",wordBreak:"keep-all"},i=n("div",{part:"region-handle region-handle-left",style:Object.assign(Object.assign({},e),{left:"0",borderLeft:"2px solid rgba(0, 0, 0, 0.5)",borderRadius:"2px 0 0 2px"})},t),s=n("div",{part:"region-handle region-handle-right",style:Object.assign(Object.assign({},e),{right:"0",borderRight:"2px solid rgba(0, 0, 0, 0.5)",borderRadius:"0 2px 2px 0"})},t),o=h(i,{threshold:1}),l=h(s,{threshold:1}),a=r((()=>{const t=o.signal.value;t&&("move"===t.type&&void 0!==t.deltaX?this.onResize(t.deltaX,"start"):"end"===t.type&&this.onEndResizing("start"))}),[o.signal]),d=r((()=>{const t=l.signal.value;t&&("move"===t.type&&void 0!==t.deltaX?this.onResize(t.deltaX,"end"):"end"===t.type&&this.onEndResizing("end"))}),[l.signal]);this.subscriptions.push((()=>{a(),d(),o.cleanup(),l.cleanup()}))}removeResizeHandles(t){const e=t.querySelector('[part*="region-handle-left"]'),i=t.querySelector('[part*="region-handle-right"]');e&&t.removeChild(e),i&&t.removeChild(i)}initElement(){if(this.isRemoved)return null;const t=this.start===this.end;let e=0,i=100;this.channelIdx>=0&&this.numberOfChannels>0&&this.channelIdxt&&this.emit("click",t))),u=i.subscribe((t=>t&&this.emit("over",t))),p=n.subscribe((t=>t&&this.emit("leave",t))),v=s.subscribe((t=>t&&this.emit("dblclick",t))),g=a.subscribe((t=>t&&this.toggleCursor(!0))),m=d.subscribe((t=>t&&this.toggleCursor(!1)));this.subscriptions.push((()=>{c(),u(),p(),v(),g(),m(),l(e),l(i),l(n),l(s),l(a),l(d)}));const f=h(t),b=r((()=>{const t=f.signal.value;t&&("start"===t.type?this.toggleCursor(!0):"move"===t.type&&void 0!==t.deltaX?this.onMove(t.deltaX):"end"===t.type&&(this.toggleCursor(!1),this.drag&&this.emit("update-end")))}),[f.signal]);this.subscriptions.push((()=>{b(),f.cleanup()})),this.contentEditable&&this.content&&(this.contentClickListener=t=>this.onContentClick(t),this.contentBlurListener=()=>this.onContentBlur(),this.content.addEventListener("click",this.contentClickListener),this.content.addEventListener("blur",this.contentBlurListener))}_onUpdate(t,e,i){var n;if(!(null===(n=this.element)||void 0===n?void 0:n.parentElement))return;const{width:s}=this.element.parentElement.getBoundingClientRect(),r=t/s*this.totalDuration;let o=e&&"start"!==e?this.start:this.start+r,l=e&&"end"!==e?this.end:this.end+r;const h=void 0!==i;h&&this.updatingSide&&this.updatingSide!==e&&("start"===this.updatingSide?o=i:l=i),o=Math.max(0,o),l=Math.min(this.totalDuration,l);const a=l-o;this.updatingSide=e;const d=a>=this.minLength&&a<=this.maxLength;o<=l&&(d||h)&&(this.start=o,this.end=l,this.renderPosition(),this.emit("update",e))}onMove(t){this.drag&&this._onUpdate(t)}onResize(t,e){this.resize&&(this.resizeStart||"start"!==e)&&(this.resizeEnd||"end"!==e)&&this._onUpdate(t,e)}onEndResizing(t){this.resize&&(this.emit("update-end",t),this.updatingSide=void 0)}onContentClick(t){t.stopPropagation();t.target.focus(),this.emit("click",t)}onContentBlur(){this.emit("update-end")}_setTotalDuration(t){this.totalDuration=t,this.renderPosition()}play(t){this.emit("play",t&&this.end!==this.start?this.end:void 0)}getContent(t=!1){var e;return t?this.content||void 0:this.element instanceof HTMLElement?(null===(e=this.content)||void 0===e?void 0:e.innerHTML)||void 0:""}setContent(t){var e;if(this.element)if(this.content&&this.contentEditable&&(this.contentClickListener&&this.content.removeEventListener("click",this.contentClickListener),this.contentBlurListener&&this.content.removeEventListener("blur",this.contentBlurListener)),null===(e=this.content)||void 0===e||e.remove(),t){if("string"==typeof t){const e=this.start===this.end;this.content=n("div",{style:{padding:`0.2em ${e?.2:.4}em`,display:"inline-block"},textContent:t})}else this.content=t;this.contentEditable&&(this.content.contentEditable="true",this.contentClickListener=t=>this.onContentClick(t),this.contentBlurListener=()=>this.onContentBlur(),this.content.addEventListener("click",this.contentClickListener),this.content.addEventListener("blur",this.contentBlurListener)),this.content.setAttribute("part","region-content"),this.element.appendChild(this.content),this.emit("content-changed")}else this.content=void 0}setOptions(t){var e,i;if(this.element){if(t.color&&(this.color=t.color,this.element.style.backgroundColor=this.color),void 0!==t.drag&&(this.drag=t.drag,this.element.style.cursor=this.drag?"grab":"default"),void 0!==t.start||void 0!==t.end){const n=this.start===this.end;this.start=this.clampPosition(null!==(e=t.start)&&void 0!==e?e:this.start),this.end=this.clampPosition(null!==(i=t.end)&&void 0!==i?i:n?this.start:this.end),this.renderPosition(),this.setPart(),this.emit("render")}if(t.content&&this.setContent(t.content),t.id&&(this.id=t.id,this.setPart()),void 0!==t.resize&&t.resize!==this.resize){const e=this.start===this.end;this.resize=t.resize,this.resize&&!e?this.addResizeHandles(this.element):this.removeResizeHandles(this.element)}void 0!==t.resizeStart&&(this.resizeStart=t.resizeStart),void 0!==t.resizeEnd&&(this.resizeEnd=t.resizeEnd)}}remove(){this.isRemoved=!0,this.emit("remove"),this.subscriptions.forEach((t=>t())),this.subscriptions=[],this.content&&this.contentEditable&&(this.contentClickListener&&(this.content.removeEventListener("click",this.contentClickListener),this.contentClickListener=void 0),this.contentBlurListener&&(this.content.removeEventListener("blur",this.contentBlurListener),this.contentBlurListener=void 0)),this.element&&(this.element.remove(),this.element=null),this.unAll()}}class d extends e{constructor(t){super(t),this.regions=[],this.regionsContainer=this.initRegionsContainer()}static create(t){return new d(t)}onInit(){if(!this.wavesurfer)throw Error("WaveSurfer is not initialized");this.wavesurfer.getWrapper().appendChild(this.regionsContainer),this.subscriptions.push(this.wavesurfer.on("ready",(t=>{this.regions.forEach((e=>e._setTotalDuration(t)))})));let t=[];this.subscriptions.push(this.wavesurfer.on("timeupdate",(e=>{const i=this.regions.filter((t=>t.start<=e&&(t.end===t.start?t.start+.05:t.end)>=e));i.forEach((e=>{t.includes(e)||this.emit("region-in",e)})),t.forEach((t=>{i.includes(t)||this.emit("region-out",t)})),t=i})))}initRegionsContainer(){return n("div",{part:"regions-container",style:{position:"absolute",top:"0",left:"0",width:"100%",height:"100%",zIndex:"5",pointerEvents:"none"}})}getRegions(){return this.regions}avoidOverlapping(t){t.content&&!t.isRemoved&&setTimeout((()=>{if(!t.content)return;const e=t.content;e.style.marginTop="0";const i=e.getBoundingClientRect(),n=this.regions.indexOf(t);if(n<0)return;const s=this.regions.slice(0,n).filter((t=>!t.isRemoved)).reduce(((e,n)=>{if(n===t||!n.content)return e;const s=n.content.getBoundingClientRect();return i.leftt.top-e.top)).reduce(((t,e)=>{const n=i.top+t,s=n+i.height;return nthis.avoidOverlapping(t)))}adjustScroll(t){var e,i;if(!t.element)return;const n=null===(i=null===(e=this.wavesurfer)||void 0===e?void 0:e.getWrapper())||void 0===i?void 0:i.parentElement;if(!n)return;const{clientWidth:s,scrollWidth:r}=n;if(r<=s)return;const o=n.getBoundingClientRect(),l=t.element.getBoundingClientRect(),h=l.left-o.left,a=l.right-o.left;h<0?n.scrollLeft+=h:a>s&&(n.scrollLeft+=a-s)}virtualAppend(t,e,i){const n=()=>{if(!this.wavesurfer)return;const n=this.wavesurfer.getWidth(),s=this.wavesurfer.getScroll(),r=e.clientWidth,o=this.wavesurfer.getDuration(),l=Math.round(t.start/o*r),h=l+(Math.round((t.end-t.start)/o*r)||1)>s&&l{if(!this.wavesurfer||!t.element)return;n();const e=this.wavesurfer.on("scroll",n),i=this.wavesurfer.on("zoom",n),s=this.wavesurfer.on("resize",n),r=t.on("render",n),o=[e,i,s,r];this.subscriptions.push(...o),t.once("remove",(()=>{e(),i(),s(),r(),this.subscriptions=this.subscriptions.filter((t=>!o.includes(t)))}))}),0)}saveRegion(t){if(!t.element)return;this.virtualAppend(t,this.regionsContainer,t.element),this.avoidOverlapping(t),this.regions.push(t);const e=[t.on("update",(e=>{e||this.adjustScroll(t),this.emit("region-update",t,e)})),t.on("update-end",(e=>{this.avoidOverlappingAll(),this.emit("region-updated",t,e)})),t.on("play",(e=>{var i;null===(i=this.wavesurfer)||void 0===i||i.play(t.start,e)})),t.on("click",(e=>{this.emit("region-clicked",t,e)})),t.on("dblclick",(e=>{this.emit("region-double-clicked",t,e)})),t.on("content-changed",(()=>{this.emit("region-content-changed",t)})),t.once("remove",(()=>{e.forEach((t=>t())),this.subscriptions=this.subscriptions.filter((t=>!e.includes(t))),this.regions=this.regions.filter((e=>e!==t)),this.emit("region-removed",t)}))];this.subscriptions.push(...e),this.emit("region-created",t)}addRegion(t){var e,i;if(!this.wavesurfer)throw Error("WaveSurfer is not initialized");const n=this.wavesurfer.getDuration(),s=null===(i=null===(e=this.wavesurfer)||void 0===e?void 0:e.getDecodedData())||void 0===i?void 0:i.numberOfChannels,r=new a(t,n,s);if(this.emit("region-initialized",r),n)this.saveRegion(r);else{const t=this.wavesurfer.once("ready",(e=>{r._setTotalDuration(e),this.saveRegion(r),this.subscriptions=this.subscriptions.filter((e=>e!==t))}));this.subscriptions.push(t)}return r}enableDragSelection(t,e=3){var i;const n=null===(i=this.wavesurfer)||void 0===i?void 0:i.getWrapper();if(!(n&&n instanceof HTMLElement))return()=>{};let s=null,o=0,l=0;const d=h(n,{threshold:e}),c=r((()=>{var e,i;const n=d.signal.value;if(n)if("start"===n.type){if(o=n.x,!this.wavesurfer)return;const r=this.wavesurfer.getDuration(),h=null===(i=null===(e=this.wavesurfer)||void 0===e?void 0:e.getDecodedData())||void 0===i?void 0:i.numberOfChannels,{width:d}=this.wavesurfer.getWrapper().getBoundingClientRect();l=o/d*r;const c=n.x/d*r,u=(n.x+5)/d*r;s=new a(Object.assign(Object.assign({},t),{start:c,end:u}),r,h),this.emit("region-initialized",s),s.element&&this.regionsContainer.appendChild(s.element)}else"move"===n.type&&void 0!==n.deltaX?s&&s._onUpdate(n.deltaX,n.x>o?"end":"start",l):"end"===n.type&&s&&(this.saveRegion(s),s.updatingSide=void 0,s=null)}),[d.signal]);return()=>{c(),d.cleanup()}}clearRegions(){this.regions.slice().forEach((t=>t.remove())),this.regions=[]}destroy(){this.clearRegions(),super.destroy(),this.regionsContainer.remove()}}return d})); diff --git a/app/static/vendor/wavesurfer/timeline.min.js b/app/static/vendor/wavesurfer/timeline.min.js new file mode 100644 index 0000000..25f6c11 --- /dev/null +++ b/app/static/vendor/wavesurfer/timeline.min.js @@ -0,0 +1 @@ +!function(t,e){"object"==typeof exports&&"undefined"!=typeof module?module.exports=e():"function"==typeof define&&define.amd?define(e):((t="undefined"!=typeof globalThis?globalThis:t||self).WaveSurfer=t.WaveSurfer||{},t.WaveSurfer.Timeline=e())}(this,(function(){"use strict";class t{constructor(){this.listeners={}}on(t,e,i){if(this.listeners[t]||(this.listeners[t]=new Set),null==i?void 0:i.once){const i=(...s)=>{this.un(t,i),e(...s)};return this.listeners[t].add(i),()=>this.un(t,i)}return this.listeners[t].add(e),()=>this.un(t,e)}un(t,e){var i;null===(i=this.listeners[t])||void 0===i||i.delete(e)}once(t,e){return this.on(t,e,{once:!0})}unAll(){this.listeners={}}emit(t,...e){this.listeners[t]&&this.listeners[t].forEach((t=>t(...e)))}}class e extends t{constructor(t){super(),this.subscriptions=[],this.isDestroyed=!1,this.options=t}onInit(){}_init(t){this.isDestroyed&&(this.subscriptions=[],this.isDestroyed=!1),this.wavesurfer=t,this.onInit()}destroy(){this.emit("destroy"),this.subscriptions.forEach((t=>t())),this.subscriptions=[],this.isDestroyed=!0,this.wavesurfer=void 0}}function i(t,e){const s=e.xmlns?document.createElementNS(e.xmlns,t):document.createElement(t);for(const[t,n]of Object.entries(e))if("children"===t&&n)for(const[t,e]of Object.entries(n))e instanceof Node?s.appendChild(e):"string"==typeof e?s.appendChild(document.createTextNode(e)):s.appendChild(i(t,e));else"style"===t?Object.assign(s.style,n):"textContent"===t?s.textContent=n:s.setAttribute(t,n.toString());return s}function s(t,e,s){return i(t,e||{})}const n={height:20,timeOffset:0,formatTimeCallback:t=>{if(t/60>1){return`${Math.floor(t/60)}:${`${(t=Math.round(t%60))<10?"0":""}${t}`}`}return`${Math.round(1e3*t)/1e3}`}};class o extends e{constructor(t){super(t||{}),this.notchElements=new Map,this.currentTimeline=null,this.options=Object.assign({},n,t),this.timelineWrapper=this.initTimelineWrapper()}static create(t){return new o(t)}onInit(){var t;if(!this.wavesurfer)throw Error("WaveSurfer is not initialized");let e=this.wavesurfer.getWrapper();if((i=this.options.container)instanceof HTMLElement||"object"==typeof i&&null!==i&&i.nodeType===Node.ELEMENT_NODE&&"object"==typeof i.style)e=this.options.container;else if("string"==typeof this.options.container){const t=document.querySelector(this.options.container);if(!t)throw Error(`No Timeline container found matching ${this.options.container}`);e=t}var i;this.options.insertPosition?(e.firstElementChild||e).insertAdjacentElement(this.options.insertPosition,this.timelineWrapper):e.appendChild(this.timelineWrapper);const s=this.wavesurfer.getState();this.subscriptions.push(function(t,e){let i;const s=()=>{i&&(i(),i=void 0),i=t()},n=e.map((t=>t.subscribe(s)));return s(),()=>{i&&(i(),i=void 0),n.forEach((t=>t()))}}((()=>{(s.duration.value>0||this.options.duration)&&this.initTimeline()}),[s.duration])),this.subscriptions.push(this.wavesurfer.on("redraw",(()=>this.initTimeline()))),this.subscriptions.push(this.wavesurfer.on("scroll",((t,e,i,s)=>{this.currentTimeline&&this.updateVisibleNotches(i,s,this.currentTimeline)}))),((null===(t=this.wavesurfer)||void 0===t?void 0:t.getDuration())||this.options.duration)&&this.initTimeline()}destroy(){this.timelineWrapper.remove(),super.destroy()}initTimelineWrapper(){return s("div",{part:"timeline-wrapper",style:{pointerEvents:"none"}})}defaultTimeInterval(t){return t>=25?1:5*t>=25?5:15*t>=25?15:60*Math.ceil(.5/t)}defaultPrimaryLabelInterval(t){return t>=25?10:5*t>=25?6:4}defaultSecondaryLabelInterval(t){return t>=25?5:2}virtualAppend(t,e,i){if(this.notchElements.set(i,{start:t,width:i.clientWidth,wasVisible:!1}),!this.wavesurfer)return;const s=this.wavesurfer.getScroll(),n=s+this.wavesurfer.getWidth(),o=this.notchElements.get(i),r=t>=s&&t+o.width{const o=s.start>=t&&s.start+s.width{this.un(t,i),e(...n)};return this.listeners[t].add(i),()=>this.un(t,i)}return this.listeners[t].add(e),()=>this.un(t,e)}un(t,e){var i;null===(i=this.listeners[t])||void 0===i||i.delete(e)}once(t,e){return this.on(t,e,{once:!0})}unAll(){this.listeners={}}emit(t,...e){this.listeners[t]&&this.listeners[t].forEach((t=>t(...e)))}}const i={decode:function(e,i){return t(this,void 0,void 0,(function*(){const t=new AudioContext({sampleRate:i});try{return yield t.decodeAudioData(e)}finally{"closed"!==t.state&&(yield t.close().catch((()=>{})))}}))},createBuffer:function(t,e){if(!t||0===t.length)throw new Error("channelData must be a non-empty array");if(e<=0)throw new Error("duration must be greater than 0");if("number"==typeof t[0]&&(t=[t]),!t[0]||0===t[0].length)throw new Error("channelData must contain non-empty channel arrays");!function(t){const e=t[0];if(e.some((t=>t>1||t<-1))){const i=e.length;let n=0;for(let t=0;tn&&(n=i)}for(const e of t)for(let t=0;tt instanceof Float32Array?t:Float32Array.from(t)));return{duration:e,length:i[0].length,sampleRate:i[0].length/e,numberOfChannels:i.length,getChannelData:t=>{const e=i[t];if(!e)throw new Error(`Channel ${t} not found`);return e},copyFromChannel:AudioBuffer.prototype.copyFromChannel,copyToChannel:AudioBuffer.prototype.copyToChannel}}};function n(t,e){const i=e.xmlns?document.createElementNS(e.xmlns,t):document.createElement(t);for(const[t,s]of Object.entries(e))if("children"===t&&s)for(const[t,e]of Object.entries(s))e instanceof Node?i.appendChild(e):"string"==typeof e?i.appendChild(document.createTextNode(e)):i.appendChild(n(t,e));else"style"===t?Object.assign(i.style,s):"textContent"===t?i.textContent=s:i.setAttribute(t,s.toString());return i}function s(t,e,i){const s=n(t,e||{});return null==i||i.appendChild(s),s}function r(t){return t instanceof HTMLElement||"object"==typeof t&&null!==t&&t.nodeType===Node.ELEMENT_NODE&&"object"==typeof t.style}var o=Object.freeze({__proto__:null,createElement:s,default:s,isHTMLElement:r});const a={fetchBlob:function(e,i,n){return t(this,void 0,void 0,(function*(){var s;const r=yield fetch(e,n);if(r.status>=400)throw new Error(`Failed to fetch ${e}: ${r.status} (${r.statusText})`);return function(e,i,n){t(this,void 0,void 0,(function*(){var t;if(!e.body||!e.headers)return;const s=e.body.getReader(),r=Number(e.headers.get("Content-Length"))||0;let o=0;const a=()=>{s.cancel()};if(n){if(n.aborted)return void s.cancel();n.addEventListener("abort",a,{once:!0})}try{for(;;){const e=yield s.read();if(e.done)break;if(o+=(null===(t=e.value)||void 0===t?void 0:t.length)||0,r>0){const t=Math.round(o/r*100);i(t)}}}catch(t){if(t instanceof DOMException&&"AbortError"===t.name)return;console.warn("Progress tracking error:",t)}finally{n&&n.removeEventListener("abort",a)}}))}(r.clone(),i,null!==(s=null==n?void 0:n.signal)&&void 0!==s?s:void 0),r.blob()}))}};function l(t){let e=t;const i=new Set;return{get value(){return e},set(t){Object.is(e,t)||(e=t,i.forEach((t=>t(e))))},update(t){this.set(t(e))},subscribe:t=>(i.add(t),()=>i.delete(t))}}function h(t,e){const i=l(t());return e.forEach((e=>e.subscribe((()=>{const e=t();Object.is(i.value,e)||i.set(e)})))),{get value(){return i.value},subscribe:t=>i.subscribe(t)}}function c(t,e){let i;const n=()=>{i&&(i(),i=void 0),i=t()},s=e.map((t=>t.subscribe(n)));return n(),()=>{i&&(i(),i=void 0),s.forEach((t=>t()))}}class u extends e{get isPlayingSignal(){return this._isPlaying}get currentTimeSignal(){return this._currentTime}get durationSignal(){return this._duration}get volumeSignal(){return this._volume}get mutedSignal(){return this._muted}get playbackRateSignal(){return this._playbackRate}get seekingSignal(){return this._seeking}constructor(t){super(),this.isExternalMedia=!1,this._ownBlobUrl=null,this.reactiveMediaEventCleanups=[],t.media?(this.media=t.media,this.isExternalMedia=!0):this.media=document.createElement("audio"),this._isPlaying=l(!1),this._currentTime=l(0),this._duration=l(0),this._volume=l(this.media.volume),this._muted=l(this.media.muted),this._playbackRate=l(this.media.playbackRate||1),this._seeking=l(!1),this.setupReactiveMediaEvents(),t.mediaControls&&(this.media.controls=!0),t.autoplay&&(this.media.autoplay=!0),null!=t.playbackRate&&this.onMediaEvent("canplay",(()=>{null!=t.playbackRate&&(this.media.playbackRate=t.playbackRate)}),{once:!0})}setupReactiveMediaEvents(){this.reactiveMediaEventCleanups.push(this.onMediaEvent("play",(()=>{this._isPlaying.set(!0)}))),this.reactiveMediaEventCleanups.push(this.onMediaEvent("pause",(()=>{this._isPlaying.set(!1)}))),this.reactiveMediaEventCleanups.push(this.onMediaEvent("ended",(()=>{this._isPlaying.set(!1)}))),this.reactiveMediaEventCleanups.push(this.onMediaEvent("timeupdate",(()=>{this._currentTime.set(this.media.currentTime)}))),this.reactiveMediaEventCleanups.push(this.onMediaEvent("durationchange",(()=>{this._duration.set(this.media.duration||0)}))),this.reactiveMediaEventCleanups.push(this.onMediaEvent("loadedmetadata",(()=>{this._duration.set(this.media.duration||0)}))),this.reactiveMediaEventCleanups.push(this.onMediaEvent("seeking",(()=>{this._seeking.set(!0)}))),this.reactiveMediaEventCleanups.push(this.onMediaEvent("seeked",(()=>{this._seeking.set(!1)}))),this.reactiveMediaEventCleanups.push(this.onMediaEvent("volumechange",(()=>{this._volume.set(this.media.volume),this._muted.set(this.media.muted)}))),this.reactiveMediaEventCleanups.push(this.onMediaEvent("ratechange",(()=>{this._playbackRate.set(this.media.playbackRate)})))}onMediaEvent(t,e,i){return this.media.addEventListener(t,e,i),()=>this.media.removeEventListener(t,e,i)}getSrc(){return this.media.currentSrc||this.media.src||""}revokeSrc(){this._ownBlobUrl&&(URL.revokeObjectURL(this._ownBlobUrl),this._ownBlobUrl=null)}canPlayType(t){return""!==this.media.canPlayType(t)}setSrc(t,e){const i=this.getSrc();if(t&&i===t)return;this.revokeSrc();const n=e instanceof Blob&&(this.canPlayType(e.type)||!t)?URL.createObjectURL(e):t;if(n!==t&&(this._ownBlobUrl=n),i&&this.media.removeAttribute("src"),n||t)try{this.media.src=n}catch(e){this.media.src=t}}destroy(){this.reactiveMediaEventCleanups.forEach((t=>t())),this.reactiveMediaEventCleanups=[],this.revokeSrc(),this.unAll(),this.isExternalMedia||(this.media.pause(),this.media.removeAttribute("src"),this.media.load(),this.media.remove())}setMediaElement(t){this.reactiveMediaEventCleanups.forEach((t=>t())),this.reactiveMediaEventCleanups=[],this.media=t,this.setupReactiveMediaEvents()}play(){return t(this,void 0,void 0,(function*(){try{return yield this.media.play()}catch(t){if(t instanceof DOMException&&"AbortError"===t.name)return;throw t}}))}pause(){this.media.pause()}isPlaying(){return!this.media.paused&&!this.media.ended}setTime(t){this.media.currentTime=Math.max(0,Math.min(t,this.getDuration()))}getDuration(){return this.media.duration}getCurrentTime(){return this.media.currentTime}getVolume(){return this.media.volume}setVolume(t){this.media.volume=t}getMuted(){return this.media.muted}setMuted(t){this.media.muted=t}getPlaybackRate(){return this.media.playbackRate}isSeeking(){return this.media.seeking}setPlaybackRate(t,e){null!=e&&(this.media.preservesPitch=e),this.media.playbackRate=t}getMediaElement(){return this.media}setSinkId(t){return this.media.setSinkId(t)}}function d({maxTop:t,maxBottom:e,halfHeight:i,vScale:n,barMinHeight:s=0,barAlign:r}){let o=Math.round(t*i*n);let a=o+Math.round(e*i*n)||1;return afunction(t){const{scrollLeft:e,scrollWidth:i,clientWidth:n}=t;if(0===i)return{startX:0,endX:1};const s=e/i,r=(e+n)/i;return{startX:Math.max(0,Math.min(1,s)),endX:Math.max(0,Math.min(1,r))}}(e.value)),[e]),n=h((()=>function(t){return{left:t.scrollLeft,right:t.scrollLeft+t.clientWidth}}(e.value)),[e]),s=()=>{e.set({scrollLeft:t.scrollLeft,scrollWidth:t.scrollWidth,clientWidth:t.clientWidth})};t.addEventListener("scroll",s,{passive:!0});return{scrollData:e,percentages:i,bounds:n,cleanup:()=>{t.removeEventListener("scroll",s),b(e)}}}class C extends e{constructor(t,e){super(),this.timeouts=[],this.isScrollable=!1,this.audioData=null,this.resizeObserver=null,this.lastContainerWidth=0,this.isDragging=!1,this.subscriptions=[],this.unsubscribeOnScroll=[],this.dragStream=null,this.scrollStream=null,this.containerInlinePadding=0,this.onClickWrapper=t=>{const e=this.wrapper.getBoundingClientRect(),[i,n]=m(e,t.clientX,t.clientY);this.emit("click",i,n)},this.onDblClickWrapper=t=>{const e=this.wrapper.getBoundingClientRect(),[i,n]=m(e,t.clientX,t.clientY);this.emit("dblclick",i,n)},this.subscriptions=[],this.options=t;const i=this.parentFromOptionsContainer(t.container);this.parent=i;const[n,s]=this.initHtml();i.appendChild(n),this.container=n,this.scrollContainer=s.querySelector(".scroll"),this.wrapper=s.querySelector(".wrapper"),this.canvasWrapper=s.querySelector(".canvases"),this.progressWrapper=s.querySelector(".progress"),this.cursor=s.querySelector(".cursor"),this.calculateInlinePadding(),e&&s.appendChild(e),this.initEvents()}parentFromOptionsContainer(t){let e;if("string"==typeof t?e=document.querySelector(t):r(t)&&(e=t),!e)throw new Error("Container not found");return e}initEvents(){this.wrapper.addEventListener("click",this.onClickWrapper),this.wrapper.addEventListener("dblclick",this.onDblClickWrapper),!0!==this.options.dragToSeek&&"object"!=typeof this.options.dragToSeek||this.initDrag(),this.scrollStream=y(this.scrollContainer);const t=c((()=>{const{startX:t,endX:e}=this.scrollStream.percentages.value,{left:i,right:n}=this.scrollStream.bounds.value;this.emit("scroll",t,e,i,n)}),[this.scrollStream.percentages,this.scrollStream.bounds]);if(this.subscriptions.push(t),"function"==typeof ResizeObserver){const t=this.createDelay(100);this.resizeObserver=new ResizeObserver((()=>{t().then((()=>this.onContainerResize())).catch((()=>{}))})),this.resizeObserver.observe(this.scrollContainer)}}onContainerResize(){const t=this.parent.clientWidth;this.calculateInlinePadding(),t===this.lastContainerWidth&&"auto"!==this.options.height||(this.lastContainerWidth=t,this.reRender(),this.emit("resize"))}initDrag(){if(this.dragStream)return;this.dragStream=function(t,e={}){const{threshold:i=3,mouseButton:n=0,touchDelay:s=100}=e,r=l(null),o=new Map,a=matchMedia("(pointer: coarse)").matches;let h=()=>{};const c=e=>{if(e.button!==n)return;if(o.has(e.pointerId))return;if(o.set(e.pointerId,e),o.size>1)return;const l=e.pointerId;let c=e.clientX,u=e.clientY,d=!1;const p=Date.now(),m=t.getBoundingClientRect(),{left:f,top:g}=m,v=t=>{if(t.pointerId!==l)return;if(t.defaultPrevented||o.size>1)return;if(a&&Date.now()-pi||Math.abs(m)>i)&&(t.preventDefault(),t.stopPropagation(),d||(r.set({type:"start",x:c-f,y:u-g}),d=!0),r.set({type:"move",x:e-f,y:n-g,deltaX:h,deltaY:m}),c=e,u=n)},b=t=>{if(o.delete(t.pointerId)){if(t.pointerId===l&&d){const e=t.clientX,i=t.clientY;r.set({type:"end",x:e-f,y:i-g})}0===o.size&&h()}},y=t=>{t.relatedTarget&&t.relatedTarget!==document.documentElement||b(t)},C=t=>{d&&(t.stopPropagation(),t.preventDefault())},S=t=>{t.defaultPrevented||o.size>1||d&&t.preventDefault()};document.addEventListener("pointermove",v),document.addEventListener("pointerup",b),document.addEventListener("pointerout",y),document.addEventListener("pointercancel",y),document.addEventListener("touchmove",S,{passive:!1}),document.addEventListener("click",C,{capture:!0}),h=()=>{document.removeEventListener("pointermove",v),document.removeEventListener("pointerup",b),document.removeEventListener("pointerout",y),document.removeEventListener("pointercancel",y),document.removeEventListener("touchmove",S),setTimeout((()=>{document.removeEventListener("click",C,{capture:!0})}),10)}};return t.addEventListener("pointerdown",c),{signal:r,cleanup:()=>{h(),t.removeEventListener("pointerdown",c),o.clear(),b(r)}}}(this.wrapper);const t=c((()=>{const t=this.dragStream.signal.value;if(!t)return;const e=this.wrapper.getBoundingClientRect().width,i=(n=t.x/e)<0?0:n>1?1:n;var n;"start"===t.type?(this.isDragging=!0,this.emit("dragstart",i)):"move"===t.type?this.emit("drag",i):"end"===t.type&&(this.isDragging=!1,this.emit("dragend",i))}),[this.dragStream.signal]);this.subscriptions.push(t)}calculateInlinePadding(){const{paddingLeft:t,paddingRight:e}=getComputedStyle(this.scrollContainer),i=parseFloat(t)+parseFloat(e);this.containerInlinePadding=Number.isNaN(i)?0:i}initHtml(){const t=document.createElement("div"),e=t.attachShadow({mode:"open"}),i=this.options.cspNonce&&"string"==typeof this.options.cspNonce?this.options.cspNonce.replace(/"/g,""):"";return e.innerHTML=`\n \n :host {\n user-select: none;\n min-width: 1px;\n }\n :host audio {\n display: block;\n width: 100%;\n }\n :host .scroll {\n overflow-x: auto;\n overflow-y: hidden;\n width: 100%;\n position: relative;\n }\n :host .noScrollbar {\n scrollbar-color: transparent;\n scrollbar-width: none;\n }\n :host .noScrollbar::-webkit-scrollbar {\n display: none;\n -webkit-appearance: none;\n }\n :host .wrapper {\n position: relative;\n overflow: visible;\n z-index: 2;\n }\n :host .canvases {\n min-height: ${this.getHeight(this.options.height,this.options.splitChannels)}px;\n pointer-events: none;\n }\n :host .canvases > div {\n position: relative;\n }\n :host canvas {\n display: block;\n position: absolute;\n top: 0;\n image-rendering: pixelated;\n }\n :host .progress {\n pointer-events: none;\n position: absolute;\n z-index: 2;\n top: 0;\n left: 0;\n width: 0;\n height: 100%;\n overflow: hidden;\n }\n :host .progress > div {\n position: relative;\n }\n :host .cursor {\n pointer-events: none;\n position: absolute;\n z-index: 5;\n top: 0;\n left: 0;\n height: 100%;\n border-radius: 2px;\n }\n \n\n
\n
\n
\n
\n
\n
\n
\n `,[t,e]}setOptions(t){var e;if(this.options.container!==t.container){const e=this.parentFromOptionsContainer(t.container);e.appendChild(this.container),this.parent=e}!0===t.dragToSeek||"object"==typeof this.options.dragToSeek?this.initDrag():(null===(e=this.dragStream)||void 0===e||e.cleanup(),this.dragStream=null),this.options=t,this.reRender()}getWrapper(){return this.wrapper}getWidth(){return this.scrollContainer.clientWidth-this.containerInlinePadding}getScroll(){return this.scrollContainer.scrollLeft}setScroll(t){this.scrollContainer.scrollLeft=t}setScrollPercentage(t){const{scrollWidth:e}=this.scrollContainer,i=e*t;this.setScroll(i)}destroy(){var t;this.wrapper.removeEventListener("click",this.onClickWrapper),this.wrapper.removeEventListener("dblclick",this.onDblClickWrapper),this.timeouts.forEach((t=>t())),this.timeouts=[],this.subscriptions.forEach((t=>t())),this.container.remove(),this.resizeObserver&&(this.resizeObserver.disconnect(),this.resizeObserver=null),null===(t=this.unsubscribeOnScroll)||void 0===t||t.forEach((t=>t())),this.unsubscribeOnScroll=[],this.dragStream&&(this.dragStream.cleanup(),this.dragStream=null),this.scrollStream&&(this.scrollStream.cleanup(),this.scrollStream=null)}createDelay(t=10){let e,i;const n=()=>{e&&(clearTimeout(e),e=void 0),i&&(i(),i=void 0)};return this.timeouts.push(n),()=>new Promise(((s,r)=>{n(),i=r,e=setTimeout((()=>{e=void 0,i=void 0,s()}),t)}))}getHeight(t,e){var i;const n=(null===(i=this.audioData)||void 0===i?void 0:i.numberOfChannels)||1;return function({optionsHeight:t,optionsSplitChannels:e,parentHeight:i,numberOfChannels:n,defaultHeight:s=128}){if(null==t)return s;const r=Number(t);if(!isNaN(r))return r;if("auto"===t){const t=i||s;return(null==e?void 0:e.every((t=>!t.overlay)))?t/n:t}return s}({optionsHeight:t,optionsSplitChannels:e,parentHeight:this.parent.clientHeight,numberOfChannels:n,defaultHeight:128})}convertColorValues(t,e){return function(t,e,i){if(!Array.isArray(t))return t||"";if(0===t.length)return"#999";if(t.length<2)return t[0]||"";const n=document.createElement("canvas"),s=n.getContext("2d");if(!s)return t[0]||"";const r=i||n.height*e,o=s.createLinearGradient(0,0,0,r),a=1/(t.length-1);return t.forEach(((t,e)=>{o.addColorStop(e*a,t)})),o}(t,this.getPixelRatio(),null==e?void 0:e.canvas.height)}getPixelRatio(){return t=window.devicePixelRatio,Math.max(1,t||1);var t}renderBarWaveform(t,e,i,n){const{width:s,height:r}=i.canvas,{halfHeight:o,barWidth:a,barRadius:l,barIndexScale:h,barSpacing:c,barMinHeight:u}=function({width:t,height:e,length:i,options:n,pixelRatio:s}){const r=e/2,o=n.barWidth?n.barWidth*s:1,a=n.barGap?n.barGap*s:n.barWidth?o/2:0,l=o+a||1;return{halfHeight:r,barWidth:o,barGap:a,barRadius:n.barRadius||0,barMinHeight:n.barMinHeight?n.barMinHeight*s:0,barIndexScale:i>0?t/l/i:0,barSpacing:l}}({width:s,height:r,length:(t[0]||[]).length,options:e,pixelRatio:this.getPixelRatio()}),m=function({channelData:t,barIndexScale:e,barSpacing:i,barWidth:n,halfHeight:s,vScale:r,canvasHeight:o,barAlign:a,barMinHeight:l}){const h=t[0]||[],c=t[1]||h,u=h.length,m=[];let f=0,g=0,v=0;for(let t=0;t<=u;t++){const u=Math.round(t*e);if(u>f){const{topHeight:t,totalHeight:e}=d({maxTop:g,maxBottom:v,halfHeight:s,vScale:r,barMinHeight:l,barAlign:a}),h=p({barAlign:a,halfHeight:s,topHeight:t,totalHeight:e,canvasHeight:o});m.push({x:f*i,y:h,width:n,height:e}),f=u,g=0,v=0}const b=Math.abs(h[t]||0),y=Math.abs(c[t]||0);b>g&&(g=b),y>v&&(v=y)}return m}({channelData:t,barIndexScale:h,barSpacing:c,barWidth:a,halfHeight:o,vScale:n,canvasHeight:r,barAlign:e.barAlign,barMinHeight:u});i.beginPath();for(const t of m)l&&"roundRect"in i?i.roundRect(t.x,t.y,t.width,t.height,l):i.rect(t.x,t.y,t.width,t.height);i.fill(),i.closePath()}renderLineWaveform(t,e,i,n){const{width:s,height:r}=i.canvas,o=function({channelData:t,width:e,height:i,vScale:n}){const s=i/2,r=t[0]||[];return[r,t[1]||r].map(((t,i)=>{const r=t.length,o=r?e/r:0,a=s,l=0===i?-1:1,h=[{x:0,y:a}];let c=0,u=0;for(let e=0;e<=r;e++){const i=Math.round(e*o);if(i>c){const t=a+(Math.round(u*s*n)||1)*l;h.push({x:c,y:t}),c=i,u=0}const r=Math.abs(t[e]||0);r>u&&(u=r)}return h.push({x:c,y:a}),h}))}({channelData:t,width:s,height:r,vScale:n});i.beginPath();for(const t of o)if(t.length){i.moveTo(t[0].x,t[0].y);for(let e=1;ea&&(a=i)}return a?r/a:r}({channelData:t,barHeight:e.barHeight,normalize:e.normalize,maxPeak:e.maxPeak});f(e)?this.renderBarWaveform(t,e,i,n):this.renderLineWaveform(t,e,i,n)}renderSingleCanvas(t,e,i,n,s,r,o){const a=this.getPixelRatio(),l=document.createElement("canvas");l.width=Math.round(i*a),l.height=Math.round(n*a),l.style.width=`${i}px`,l.style.height=`${n}px`,l.style.left=`${Math.round(s)}px`,r.appendChild(l);const h=l.getContext("2d");if(e.renderFunction?(h.fillStyle=this.convertColorValues(e.waveColor,h),e.renderFunction(t,h)):this.renderWaveform(t,e,h),l.width>0&&l.height>0){const t=l.cloneNode(),i=t.getContext("2d");i.drawImage(l,0,0),i.globalCompositeOperation="source-in",i.fillStyle=this.convertColorValues(e.progressColor,i),i.fillRect(0,0,l.width,l.height),o.appendChild(t)}}renderMultiCanvas(t,e,i,n,s,r){const o=this.getPixelRatio(),{clientWidth:a}=this.scrollContainer,l=i/o,h=function({clientWidth:t,totalWidth:e,options:i}){return g(Math.min(8e3,t,e),i)}({clientWidth:a,totalWidth:l,options:e});let c={};if(0===h)return;const u=i=>{if(i<0||i>=d)return;if(c[i])return;c[i]=!0;const o=i*h;let a=Math.min(l-o,h);if(a=g(a,e),a<=0)return;const u=function({channelData:t,offset:e,clampedWidth:i,totalWidth:n}){return t.map((t=>{const s=Math.floor(e/n*t.length),r=Math.floor((e+i)/n*t.length);return t.slice(s,r)}))}({channelData:t,offset:o,clampedWidth:a,totalWidth:l});this.renderSingleCanvas(u,e,a,n,o,s,r)},d=Math.ceil(l/h);if(!this.isScrollable){for(let t=0;tu(t))),d>1){const t=this.on("scroll",(()=>{const{scrollLeft:t}=this.scrollContainer;Object.keys(c).length>10&&(s.innerHTML="",r.innerHTML="",c={}),v({scrollLeft:t,totalWidth:l,numCanvases:d}).forEach((t=>u(t)))}));this.unsubscribeOnScroll.push(t)}}renderChannel(t,e,i,n){var{overlay:s}=e,r=function(t,e){var i={};for(var n in t)Object.prototype.hasOwnProperty.call(t,n)&&e.indexOf(n)<0&&(i[n]=t[n]);if(null!=t&&"function"==typeof Object.getOwnPropertySymbols){var s=0;for(n=Object.getOwnPropertySymbols(t);s0&&(o.style.marginTop=`-${a}px`),this.canvasWrapper.style.minHeight=`${a}px`,this.canvasWrapper.appendChild(o);const l=o.cloneNode();this.progressWrapper.appendChild(l),this.renderMultiCanvas(t,r,i,a,o,l)}render(e){return t(this,void 0,void 0,(function*(){var t;this.timeouts.forEach((t=>t())),this.timeouts=[],this.unsubscribeOnScroll.forEach((t=>t())),this.unsubscribeOnScroll=[],this.canvasWrapper.innerHTML="",this.progressWrapper.innerHTML="",null!=this.options.width&&(this.scrollContainer.style.width="number"==typeof this.options.width?`${this.options.width}px`:this.options.width);const i=this.getPixelRatio(),n=this.scrollContainer.clientWidth-this.containerInlinePadding,{scrollWidth:s,isScrollable:r,useParentWidth:o,width:a}=function({duration:t,minPxPerSec:e=0,parentWidth:i,fillParent:n,pixelRatio:s}){const r=Math.ceil(t*e),o=r>i,a=Boolean(n&&!o);return{scrollWidth:r,isScrollable:o,useParentWidth:a,width:(a?i:r)*s}}({duration:e.duration,minPxPerSec:this.options.minPxPerSec||0,parentWidth:n,fillParent:this.options.fillParent,pixelRatio:i});if(this.isScrollable=r,this.wrapper.style.width=o?"100%":`${s}px`,this.scrollContainer.style.overflowX=this.isScrollable?"auto":"hidden",this.scrollContainer.classList.toggle("noScrollbar",!!this.options.hideScrollbar),this.cursor.style.backgroundColor=`${this.options.cursorColor||this.options.progressColor}`,this.cursor.style.width=`${this.options.cursorWidth}px`,this.audioData=e,this.emit("render"),this.options.splitChannels)for(let i=0;i1&&t.push(e.getChannelData(1)),this.renderChannel(t,this.options,a,0)}Promise.resolve().then((()=>this.emit("rendered")))}))}reRender(){if(this.unsubscribeOnScroll.forEach((t=>t())),this.unsubscribeOnScroll=[],!this.audioData)return;const{scrollWidth:t}=this.scrollContainer,{right:e}=this.progressWrapper.getBoundingClientRect();if(this.render(this.audioData),!this.isScrollable&&this.scrollContainer.scrollLeft)this.scrollContainer.scrollLeft=0;else if(this.isScrollable&&t!==this.scrollContainer.scrollWidth){const{right:t}=this.progressWrapper.getBoundingClientRect(),i=function(t){const e=2*t;return(e<0?Math.floor(e):Math.ceil(e))/2}(t-e);this.scrollContainer.scrollLeft+=i}}zoom(t){this.options.minPxPerSec=t,this.reRender()}scrollIntoView(t,e=!1){var i;const{scrollLeft:n,scrollWidth:s,clientWidth:r}=this.scrollContainer,o=t*s,a=n,l=n+r,h=r/2;if(this.isDragging){const t=30;o+t>l?this.scrollContainer.scrollLeft+=t:o-tl)&&(this.scrollContainer.scrollLeft=o-(this.options.autoCenter?h:0));const t=o-n-h;if(e&&this.options.autoCenter&&t>0){const e=null===(i=this.audioData)||void 0===i?void 0:i.duration;if(void 0===e||e<=0)return void(this.scrollContainer.scrollLeft+=t);const n=s/e;this.scrollContainer.scrollLeft+=n<=600?Math.min(t,10):t}}}renderProgress(t,e){if(isNaN(t))return;const i=100*t;this.canvasWrapper.style.clipPath=`polygon(${i}% 0%, 100% 0%, 100% 100%, ${i}% 100%)`,this.progressWrapper.style.width=`${i}%`,this.cursor.style.left=`${i}%`,this.cursor.style.transform=this.options.cursorWidth?`translateX(-${t*this.options.cursorWidth}px)`:"",this.isScrollable&&this.options.autoScroll&&this.audioData&&this.audioData.duration>0&&this.scrollIntoView(t,e)}exportImage(e,i,n){return t(this,void 0,void 0,(function*(){const t=this.canvasWrapper.querySelectorAll("canvas");if(!t.length)throw new Error("No waveform data");if("dataURL"===n){const n=Array.from(t).map((t=>t.toDataURL(e,i)));return Promise.resolve(n)}return Promise.all(Array.from(t).map((t=>new Promise(((n,s)=>{t.toBlob((t=>{t?n(t):s(new Error("Could not export image"))}),e,i)})))))}))}}class S extends e{constructor(){super(...arguments),this.animationFrameId=null,this.isRunning=!1}start(){if(this.isRunning)return;this.isRunning=!0;const t=()=>{this.isRunning&&(this.emit("tick"),this.animationFrameId=requestAnimationFrame(t))};t()}stop(){this.isRunning=!1,null!==this.animationFrameId&&(cancelAnimationFrame(this.animationFrameId),this.animationFrameId=null)}destroy(){this.stop(),this.unAll()}}class E extends e{constructor(t){super(),this.bufferNode=null,this.playStartTime=0,this.playbackPosition=0,this._muted=!1,this._playbackRate=1,this._duration=void 0,this.buffer=null,this.currentSrc="",this.paused=!0,this.crossOrigin=null,this.seeking=!1,this.autoplay=!1,this.addEventListener=this.on,this.removeEventListener=this.un,this._destroyed=!1,function(){const t=globalThis.navigator;if(null==t?void 0:t.audioSession)try{t.audioSession.type="playback"}catch(t){console.warn("Setting navigator.audioSession.type failed:",t)}}(),this.audioContext=t||new AudioContext,this.gainNode=this.audioContext.createGain(),this.gainNode.connect(this.audioContext.destination)}load(){return t(this,void 0,void 0,(function*(){}))}remove(){this.destroy()}destroy(){if(!this._destroyed){if(this._destroyed=!0,this.currentSrc="",this.bufferNode){this.bufferNode.onended=null;try{this.bufferNode.stop()}catch(t){}this.bufferNode.disconnect(),this.bufferNode=null}this.gainNode.disconnect(),"function"==typeof this.audioContext.close&&Promise.resolve(this.audioContext.close.call(this.audioContext)).catch((()=>{})),this.buffer=null,this.unAll()}}get src(){return this.currentSrc}set src(t){if(this.currentSrc=t,this._duration=void 0,!t)return this.buffer=null,void this.emit("emptied");fetch(t).then((e=>{if(e.status>=400)throw new Error(`Failed to fetch ${t}: ${e.status} (${e.statusText})`);return e.arrayBuffer()})).then((e=>this.currentSrc!==t?null:this.audioContext.decodeAudioData(e))).then((e=>{this.currentSrc===t&&(this.buffer=e,this.emit("loadedmetadata"),this.emit("canplay"),this.autoplay&&this.play())})).catch((t=>{console.error("WebAudioPlayer load error:",t)}))}_play(){if(!this.paused)return;this.paused=!1,this.bufferNode&&(this.bufferNode.onended=null,this.bufferNode.disconnect()),this.bufferNode=this.audioContext.createBufferSource(),this.buffer&&(this.bufferNode.buffer=this.buffer),this.bufferNode.playbackRate.value=this._playbackRate,this.bufferNode.connect(this.gainNode);let t=this.playbackPosition;(t>=this.duration||t<0)&&(t=0,this.playbackPosition=0),this.bufferNode.start(this.audioContext.currentTime,t),this.playStartTime=this.audioContext.currentTime,this.bufferNode.onended=()=>{!this.paused&&this.duration-this.currentTime<.01&&(this.pause(),this.emit("ended"))}}_pause(){if(this.paused=!0,this.bufferNode){this.bufferNode.onended=null;try{this.bufferNode.stop()}catch(t){}}this.playbackPosition+=(this.audioContext.currentTime-this.playStartTime)*this._playbackRate}play(){return t(this,void 0,void 0,(function*(){this.paused&&(this._play(),this.emit("play"))}))}pause(){this.paused||(this._pause(),this.emit("pause"))}stopAt(t){const e=(t-this.currentTime)/this._playbackRate,i=this.bufferNode;null==i||i.stop(this.audioContext.currentTime+e),null==i||i.addEventListener("ended",(()=>{i===this.bufferNode&&(this.bufferNode=null,this.pause(),this.playbackPosition=Math.min(t,this.duration),this.emit("timeupdate"))}),{once:!0})}setSinkId(e){return t(this,void 0,void 0,(function*(){return this.audioContext.setSinkId(e)}))}get playbackRate(){return this._playbackRate}set playbackRate(t){const e=!this.paused;e&&this._pause(),this._playbackRate=t,e&&this._play(),this.bufferNode&&(this.bufferNode.playbackRate.value=t)}get currentTime(){return this.paused?this.playbackPosition:this.playbackPosition+(this.audioContext.currentTime-this.playStartTime)*this._playbackRate}set currentTime(t){const e=!this.paused;e&&this._pause(),this.playbackPosition=t,e&&this._play(),this.emit("seeking"),this.emit("timeupdate")}get duration(){var t,e;return null!==(t=this._duration)&&void 0!==t?t:(null===(e=this.buffer)||void 0===e?void 0:e.duration)||0}set duration(t){this._duration=t}get volume(){return this.gainNode.gain.value}set volume(t){this.gainNode.gain.value=t,this.emit("volumechange")}get muted(){return this._muted}set muted(t){this._muted!==t&&(this._muted=t,this._muted?this.gainNode.disconnect():this.gainNode.connect(this.audioContext.destination))}canPlayType(t){return/^(audio|video)\//.test(t)}getGainNode(){return this.gainNode}getChannelData(){const t=[];if(!this.buffer)return t;const e=this.buffer.numberOfChannels;for(let i=0;i!u.value),[u]),S=h((()=>null!==f.value),[f]),E=h((()=>S.value&&c.value>0),[S,c]),P=h((()=>a.value),[a]),w=h((()=>c.value>0?a.value/c.value:0),[a,c]);return{state:{currentTime:a,duration:c,isPlaying:u,isPaused:C,isSeeking:d,volume:p,playbackRate:m,audioBuffer:f,peaks:g,url:v,zoom:b,scrollPosition:y,canPlay:S,isReady:E,progress:P,progressPercent:w},actions:{setCurrentTime:t=>{const e=Math.max(0,Math.min(c.value||1/0,t));a.set(e)},setDuration:t=>{c.set(Math.max(0,t))},setPlaying:t=>{u.set(t)},setSeeking:t=>{d.set(t)},setVolume:t=>{const e=Math.max(0,Math.min(1,t));p.set(e)},setPlaybackRate:t=>{const e=Math.max(.1,Math.min(16,t));m.set(e)},setAudioBuffer:t=>{f.set(t),t&&c.set(t.duration)},setPeaks:t=>{g.set(t)},setUrl:t=>{v.set(t)},setZoom:t=>{b.set(Math.max(0,t))},setScrollPosition:t=>{y.set(Math.max(0,t))}}}}({isPlaying:this.isPlayingSignal,currentTime:this.currentTimeSignal,duration:this.durationSignal,volume:this.volumeSignal,playbackRate:this.playbackRateSignal,isSeeking:this.seekingSignal});this.wavesurferState=i,this.wavesurferActions=n,this.timer=new S;const s=e?void 0:this.getMediaElement();this.renderer=new C(this.options,s),this.initPlayerEvents(),this.initRendererEvents(),this.initTimerEvents(),this.initReactiveState(),this.initPlugins();const r=this.options.url||this.getSrc()||"";Promise.resolve().then((()=>{this.emit("init");const{peaks:t,duration:e}=this.options;(r||t&&e)&&this.load(r,t,e).catch((()=>{}))}))}updateProgress(t=this.getCurrentTime()){return this.renderer.renderProgress(t/this.getDuration(),this.isPlaying()),t}initTimerEvents(){this.subscriptions.push(this.timer.on("tick",(()=>{if(!this.isSeeking()){const t=this.updateProgress();if(this.emit("timeupdate",t),this.emit("audioprocess",t),null!=this.stopAtPosition&&this.isPlaying()&&t>=this.stopAtPosition){const t=this.stopAtPosition;this.pause(),this.setTime(t)}}})))}initReactiveState(){this.reactiveCleanups.push(function(t,e){const i=[];i.push(c((()=>{const i=t.isPlaying.value;e.emit(i?"play":"pause")}),[t.isPlaying])),i.push(c((()=>{const i=t.currentTime.value;e.emit("timeupdate",i),t.isPlaying.value&&e.emit("audioprocess",i)}),[t.currentTime,t.isPlaying])),i.push(c((()=>{t.isSeeking.value&&e.emit("seeking",t.currentTime.value)}),[t.isSeeking,t.currentTime]));let n=!1;i.push(c((()=>{t.isReady.value&&!n&&(n=!0,e.emit("ready",t.duration.value))}),[t.isReady,t.duration])),i.push(c((()=>{null===t.audioBuffer.value&&(n=!1)}),[t.audioBuffer]));let s=!1;return i.push(c((()=>{const i=t.isPlaying.value,n=t.currentTime.value,r=t.duration.value,o=r>0&&n>=r;s&&!i&&o&&e.emit("finish"),s=i&&o}),[t.isPlaying,t.currentTime,t.duration])),i.push(c((()=>{const i=t.zoom.value;i>0&&e.emit("zoom",i)}),[t.zoom])),()=>{i.forEach((t=>t()))}}(this.wavesurferState,{emit:this.emit.bind(this)}))}initPlayerEvents(){this.isPlaying()&&(this.emit("play"),this.timer.start()),this.mediaSubscriptions.push(this.onMediaEvent("timeupdate",(()=>{const t=this.updateProgress();this.emit("timeupdate",t)})),this.onMediaEvent("play",(()=>{this.emit("play"),this.timer.start()})),this.onMediaEvent("pause",(()=>{this.emit("pause"),this.timer.stop(),this.stopAtPosition=null})),this.onMediaEvent("emptied",(()=>{this.timer.stop(),this.stopAtPosition=null})),this.onMediaEvent("ended",(()=>{this.emit("timeupdate",this.getDuration()),this.emit("finish"),this.stopAtPosition=null})),this.onMediaEvent("seeking",(()=>{this.emit("seeking",this.getCurrentTime())})),this.onMediaEvent("error",(()=>{var t;this.emit("error",null!==(t=this.getMediaElement().error)&&void 0!==t?t:new Error("Media error")),this.stopAtPosition=null})))}initRendererEvents(){this.subscriptions.push(this.renderer.on("click",((t,e)=>{this.options.interact&&(this.seekTo(t),this.emit("interaction",t*this.getDuration()),this.emit("click",t,e))})),this.renderer.on("dblclick",((t,e)=>{this.emit("dblclick",t,e)})),this.renderer.on("scroll",((t,e,i,n)=>{const s=this.getDuration();this.emit("scroll",t*s,e*s,i,n)})),this.renderer.on("render",(()=>{this.emit("redraw")})),this.renderer.on("rendered",(()=>{this.emit("redrawcomplete")})),this.renderer.on("dragstart",(t=>{this.emit("dragstart",t)})),this.renderer.on("dragend",(t=>{this.emit("dragend",t)})),this.renderer.on("resize",(()=>{this.emit("resize")})));{let t;const e=this.renderer.on("drag",(e=>{var i;if(!this.options.interact)return;this.renderer.renderProgress(e),clearTimeout(t);let n=0;const s=this.options.dragToSeek;this.isPlaying()?n=0:!0===s?n=200:s&&"object"==typeof s&&(n=null!==(i=s.debounceTime)&&void 0!==i?i:200),t=setTimeout((()=>{this.seekTo(e)}),n),this.emit("interaction",e*this.getDuration()),this.emit("drag",e)}));this.subscriptions.push((()=>{clearTimeout(t),e()}))}}initPlugins(){var t;(null===(t=this.options.plugins)||void 0===t?void 0:t.length)&&this.options.plugins.forEach((t=>{this.registerPlugin(t)}))}unsubscribePlayerEvents(){this.mediaSubscriptions.forEach((t=>t())),this.mediaSubscriptions=[]}setOptions(t){this.options=Object.assign({},this.options,t),t.duration&&!t.peaks&&(this.decodedData=i.createBuffer(this.exportPeaks(),t.duration)),t.peaks&&t.duration&&(this.decodedData=i.createBuffer(t.peaks,t.duration)),this.renderer.setOptions(this.options),t.audioRate&&this.setPlaybackRate(t.audioRate),null!=t.mediaControls&&(this.getMediaElement().controls=t.mediaControls)}registerPlugin(t){if(this.plugins.includes(t))return t;t._init(this),this.plugins.push(t);const e=t.once("destroy",(()=>{this.plugins=this.plugins.filter((e=>e!==t)),this.subscriptions=this.subscriptions.filter((t=>t!==e))}));return this.subscriptions.push(e),t}unregisterPlugin(t){this.plugins=this.plugins.filter((e=>e!==t)),t.destroy()}getWrapper(){return this.renderer.getWrapper()}getWidth(){return this.renderer.getWidth()}getScroll(){return this.renderer.getScroll()}setScroll(t){return this.renderer.setScroll(t)}setScrollTime(t){const e=t/this.getDuration();this.renderer.setScrollPercentage(e)}getActivePlugins(){return this.plugins}loadAudio(e,n,s,r){return t(this,void 0,void 0,(function*(){var t;const o=++this._loadVersion;if(this._isDestroyed=!1,this.emit("load",e),!this.options.media&&this.isPlaying()&&this.pause(),this.decodedData=null,this.stopAtPosition=null,null===(t=this.abortController)||void 0===t||t.abort(),this.abortController=null,!n&&!s){const t=this.options.fetchParams||{};window.AbortController&&!t.signal&&(this.abortController=new AbortController,t.signal=this.abortController.signal);const i=t=>this.emit("loading",t);if(n=yield a.fetchBlob(e,i,t),this._isDestroyed||o!==this._loadVersion)return;const s=this.options.blobMimeType;s&&(n=new Blob([n],{type:s}))}if(this._isDestroyed||o!==this._loadVersion)return;this.setSrc(e,n);const l=yield new Promise((t=>{const e=r||this.getDuration();e?t(e):this.mediaSubscriptions.push(this.onMediaEvent("loadedmetadata",(()=>t(this.getDuration())),{once:!0}))}));if(!this._isDestroyed&&o===this._loadVersion){if(!e&&!n){const t=this.getMediaElement();t instanceof E&&(t.duration=l)}if(s)this.decodedData=i.createBuffer(s,l||0);else if(n){const t=yield n.arrayBuffer();if(this._isDestroyed||o!==this._loadVersion)return;this.decodedData=yield i.decode(t,this.options.sampleRate)}this._isDestroyed||o!==this._loadVersion||(this.decodedData&&(this.emit("decode",this.getDuration()),this.renderer.render(this.decodedData)),this.emit("ready",this.getDuration()))}}))}load(e,i,n){return t(this,void 0,void 0,(function*(){try{return yield this.loadAudio(e,void 0,i,n)}catch(t){throw this.emit("error",t),t}}))}loadBlob(e,i,n){return t(this,void 0,void 0,(function*(){try{return yield this.loadAudio("",e,i,n)}catch(t){throw this.emit("error",t),t}}))}zoom(t){if(!this.decodedData)throw new Error("No audio loaded");this.renderer.zoom(t),this.emit("zoom",t)}getDecodedData(){return this.decodedData}exportPeaks({channels:t=2,maxLength:e=8e3,precision:i=1e4}={}){if(!this.decodedData)throw new Error("The audio has not been decoded yet");const n=Math.min(t,this.decodedData.numberOfChannels),s=[];for(let t=0;tMath.abs(s)&&(s=i)}r.push(Math.round(s*i)/i)}s.push(r)}return s}getDuration(){let t=super.getDuration()||0;return 0!==t&&t!==1/0||!this.decodedData||(t=this.decodedData.duration),t}toggleInteraction(t){this.options.interact=t}setTime(t){this.stopAtPosition=null,super.setTime(t),this.updateProgress(t),this.emit("timeupdate",t)}seekTo(t){const e=this.getDuration()*t;this.setTime(e)}play(e,i){const n=Object.create(null,{play:{get:()=>super.play}});return t(this,void 0,void 0,(function*(){null!=e&&this.setTime(e);const t=yield n.play.call(this);return null!=i&&(this.media instanceof E?this.media.stopAt(i):this.stopAtPosition=i),t}))}playPause(){return t(this,void 0,void 0,(function*(){return this.isPlaying()?this.pause():this.play()}))}stop(){this.pause(),this.setTime(0)}skip(t){this.setTime(this.getCurrentTime()+t)}empty(){this.load("",[[0]],.001)}setMediaElement(t){this.unsubscribePlayerEvents(),super.setMediaElement(t),this.initPlayerEvents()}exportImage(){return t(this,arguments,void 0,(function*(t="image/png",e=1,i="dataURL"){return this.renderer.exportImage(t,e,i)}))}destroy(){var t;this._isDestroyed=!0,this.emit("destroy"),null===(t=this.abortController)||void 0===t||t.abort(),this.plugins.forEach((t=>t.destroy())),this.subscriptions.forEach((t=>t())),this.unsubscribePlayerEvents(),this.reactiveCleanups.forEach((t=>t())),this.reactiveCleanups=[],this.timer.destroy(),this.renderer.destroy(),super.destroy()}}return w.BasePlugin=class extends e{constructor(t){super(),this.subscriptions=[],this.isDestroyed=!1,this.options=t}onInit(){}_init(t){this.isDestroyed&&(this.subscriptions=[],this.isDestroyed=!1),this.wavesurfer=t,this.onInit()}destroy(){this.emit("destroy"),this.subscriptions.forEach((t=>t())),this.subscriptions=[],this.isDestroyed=!0,this.wavesurfer=void 0}},w.dom=o,w})); diff --git a/app/templates/base.html b/app/templates/base.html index 9214ea5..ed88c20 100644 --- a/app/templates/base.html +++ b/app/templates/base.html @@ -8,11 +8,10 @@ - - + {% block extra_head %}{% endblock %} - + - + {% block extra_scripts %}{% endblock %} + diff --git a/app/templates/clips_overview.html b/app/templates/clips_overview.html index f00539e..03a4d80 100644 --- a/app/templates/clips_overview.html +++ b/app/templates/clips_overview.html @@ -6,18 +6,18 @@

Clip Review

片段审核工作台

-

按工作流查看每个任务是否已经进入 AI 分析、人工审核、切片生成或异常排查阶段。

+

集中查看累计审核任务、通过视频和已完成任务,并继续进入单任务审核。

新建任务 -
+
{% for item in stats %}
{{ item.label }} {{ item.value }} - 来自当前任务库 + {{ item.note }}
{% endfor %}
diff --git a/app/templates/index.html b/app/templates/index.html index 066725e..04e65d8 100644 --- a/app/templates/index.html +++ b/app/templates/index.html @@ -11,48 +11,25 @@

工作台

查看全部任务
-
- {% for item in stats %} -
-
- {{ item.label }} - {{ item.value }} - {{ item.note }} -
- {% endfor %} -
- -
-
-
-
-

Workflow

-

工作流程

-
-
-
- {% for step in workflow_steps %} -
- {{ loop.index }} - {{ step }} -
- {% endfor %} +
+
+
+

Weekly overview

+

本周任务概览

+

周一至周日统计,发布终态内容不会重复计入待推送。

-
+ {{ weekly_summary.range_label }} · 上海时间 + -
-
-
-

Today

-

今日处理焦点

-
-
-
    -
  • {{ stats[2].value }}条任务等待检查并生成切片
  • -
  • {{ stats[1].value }}条任务等待进入处理流程
  • -
  • {{ stats[4].value }}条任务需要排查异常
  • -
-
+
+ {% for item in stats %} +
+ {{ item.label }} + {{ item.value }} + {{ item.note }} +
+ {% endfor %} +
diff --git a/app/templates/new_task.html b/app/templates/new_task.html index 685d1f2..a3127e7 100644 --- a/app/templates/new_task.html +++ b/app/templates/new_task.html @@ -29,18 +29,17 @@

基本信息

- 视频来源 - -
+ 上传本机视频 +
- 上传本机视频 - 选择视频后,会直接写入 E 盘任务的 source 目录;上传临时文件也在 E 盘。 + 选择本机视频文件 + 默认最大 4 GB,上传后统一保存到 E 盘任务目录。
- +

尚未选择视频文件。

@@ -56,10 +55,32 @@

基本信息

+ +
-

开启后,系统按“最终启用目标”生成切片;康熙笑点选片模式只自动处理达到 A 级质量门槛的片段。

+

开启后会按所选模式执行;长直播高光的分层选片将在下一阶段接入。

@@ -123,7 +144,7 @@

智能 MVP 流程

小贴士

-

本任务固定使用康熙笑点选片模式。候选池默认最多 12 条,真正进入切片队列的 A 级片段默认不超过 5 条。

+

请选择与素材相符的模式。创建前会检查音轨、编码、可解码性和剩余空间。

diff --git a/app/templates/publish.html b/app/templates/publish.html index 9ed62b5..ba870cf 100644 --- a/app/templates/publish.html +++ b/app/templates/publish.html @@ -33,6 +33,13 @@ +
+
+ 发布文案 + AI 重写会立即生成并保存本条标题、话题和简介 +
+ +
- @@ -145,7 +151,7 @@ >

Publish Center

发送中心

先准备内容,再按北京时间排期,最后在执行记录中确认平台结果。

-
+
@@ -183,17 +189,30 @@ {% set initial_platform = "douyin" %} {% set initial_missing_cover_count = missing_cover_counts.get(initial_platform, 0) %}
-

内容准备

这里不展示历史错误;内容、平台和账号完整后再加入排期。

+

内容准备

逐条编辑文案,或勾选多条后批量重写;内容、平台和账号完整后再加入排期。

+
+
{% for group in publish_task_groups %}
@@ -318,7 +337,7 @@

全部执行记录

仅限 抖音 - +
@@ -357,4 +376,4 @@

全部执行记录

{% endblock %} -{% block extra_scripts %}{% endblock %} +{% block extra_scripts %}{% endblock %} diff --git a/app/templates/subtitle_workflow.html b/app/templates/subtitle_workflow.html index de23c58..6913846 100644 --- a/app/templates/subtitle_workflow.html +++ b/app/templates/subtitle_workflow.html @@ -16,7 +16,9 @@

{{ task.title }} · 字幕工作台

返回字幕任务列表 返回片段审核 {% if output_clips %} + {% if not (task.auto_mode and task.status == "PENDING_SUBTITLE_REVIEW") %} + {% endif %} 查看本任务发送内容 {% endif %} @@ -38,6 +40,124 @@

{{ task.title }} · 字幕工作台

{% endfor %} +{% if task.auto_mode and task.status == "PENDING_SUBTITLE_REVIEW" %} +
+
+

Review Gate

+

自动流水线已在字幕审核处暂停

+

请先检查和编辑字幕。点击“审核并批量烧录”会批准每条切片的当前版本,全部成片验证通过后才恢复发送中心流程。

+
+
+ + +
+ +
+{% endif %} + +
+
+
+

Professional Subtitle Editor

+

原片主时间轴 · 切片继承编辑器

+

先在原片修正统一字幕,再按切片边界自动继承;已经人工编辑的切片不会被覆盖。

+
+
+ 正在载入字幕轨… + +
+
+ +
+ +
尚未载入 revision
+
+ + + + +
+
+ +
+
+ +
+
+
+
+ 服务端预计算波形 + 等待载入 +
+
+
+
+
+ +
+ + + + + + + + + + + + + +
+ + + +
+ 质量检查会提示重叠、间隔、时长、行数与中文阅读速度,但不会自动改字。 +
+
+
+
+
+
+
@@ -164,6 +284,30 @@

字幕样式模板

启用阴影和描边,适合短视频平台观看 +
+ + +
+ +
+ + +

样式会保存到 SQLite,后续自动加字幕会使用这套模板。

@@ -304,4 +448,10 @@

还没有进入字幕流程的切片

{% endblock %} {% block extra_scripts %} +{% if subtitle_task_mode %} + + + + +{% endif %} {% endblock %} diff --git a/app/templates/system_status.html b/app/templates/system_status.html index 2df6c06..8cefa69 100644 --- a/app/templates/system_status.html +++ b/app/templates/system_status.html @@ -95,7 +95,7 @@

三类 AI 接口配置

- + @@ -103,7 +103,7 @@

三类 AI 接口配置

- + @@ -154,10 +154,10 @@

1. 音频转写

- +
- - + +
@@ -176,7 +176,7 @@

1. 音频转写

2. 分析文字稿,生成候选切片

- +
- +
-{% if task.auto_mode %}
全自动模式已接管这个任务。 -

系统会自动推进音频提取、转写、AI 选片和视频切割;状态概览与运行日志每 3 秒自动更新,不会刷新整张页面。只有失败或历史任务中断时才需要点击重试/继续。

+

系统会自动推进音频提取、转写、AI 选片和视频切割;切片后会生成字幕草稿并暂停。审核烧录可恢复后续流程;跳过字幕则先进入片段审核,保存确认后才同步发送中心。

-{% endif %}
@@ -65,7 +66,6 @@

基础信息

平台
{{ task.platform_label }}
-
视频来源
{{ task.source_type_label }}
视频时长
{{ task.duration }}
视频大小
{{ task.video_size }}
候选片段
{{ task.candidate_count }} 条
@@ -84,12 +84,23 @@

基础信息

Status

状态概览

- {{ task.status_label }} + {{ task.status_label }}
- {{ task.progress }}% +
+ {{ task.progress }}% + 总流程进度 +
+
+
+ 当前操作 + {{ task.status_label }} +
+ {{ task.progress }}% +

正在读取最新任务进度…

+
    {% for step in workflow_steps %}
  1. @@ -103,7 +114,7 @@

    状态概览

    当前阶段 异常 / 待处理 -

    正在连接任务进度…

    +

    正在连接任务进度…

@@ -115,6 +126,10 @@

AI 分析

{{ task.candidate_count }} 条候选 + {% if task.selection_profile == "long_live_talk" %} + + + {% endif %}
- 待刷新 + 正在连接
任务开始后,这里会自动显示最新运行日志。

diff --git a/docker-compose.yml b/docker-compose.yml index 2187fa9..ea63499 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -13,11 +13,12 @@ services: container_name: niuma-studio command: uvicorn app.main:app --host 0.0.0.0 --port 8001 ports: - - "8001:8001" + - "127.0.0.1:8001:8001" env_file: - path: .env required: false environment: + NIUMA_TRUST_DOCKER_LOOPBACK_PROXY: "true" DATA_DIR: /app/data DATABASE_PATH: /app/data/workflow.sqlite3 STORAGE_ROOT: /workspace/tasks diff --git a/docs/AI_ANALYSIS.md b/docs/AI_ANALYSIS.md index 7b1d7cf..1004fc7 100644 --- a/docs/AI_ANALYSIS.md +++ b/docs/AI_ANALYSIS.md @@ -1,5 +1,16 @@ # AI 片段分析说明 +## 2026-08-23:长直播语言高光模式 + +- 模式键:`long_live_talk`。 +- 窗口:300 秒,重叠 60 秒;单窗口最多召回 5 个高光,允许返回空数组,不强制凑数。 +- 类别:`quote_opinion`、`story_experience`、`emotional_peak`、`conflict_reversal`、`practical_knowledge`、`interactive_humor`。 +- 恢复:checkpoint 绑定转写 SHA-256、Provider、模型和窗口边界;任一键变化都会创建新的窗口集合。 +- 完整性:以成功窗口区间并集计算 `coverage_ratio`,小于 `0.90` 写入 `analysis_incomplete=true`。 +- 排序:先跨窗口去重,再按小时最多 N 条,最后跨小时轮询到总量上限;默认 4 条/小时、总计 30 条。 + +`general` 和 `variety_comedy` 继续使用原算法;本次没有迁移或重算既有分析历史。 + ## 2026-05-23:AI Prompt 方案 任务详情页现在使用“AI Prompt 方案”管理 AI 分析 Prompt: diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 15480c5..94e95ee 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -1,5 +1,66 @@ # 系统架构 +## 2026-08-24:字幕审核与交付证据链 + +```text +切片完成 +→ source/clip 字幕草稿 +→ pending_subtitle_review(自动流水线暂停) + ├─ 明确跳过 → delivery_mode=original → 恢复元数据/发送任务 + └─ 审核 revision → workflow_jobs:subtitle → 临时渲染 + → FFprobe 验证 → 原子激活 → delivery_mode=subtitled + → 恢复元数据/发送任务 +``` + +- `subtitle_auto_workflow_service.py` 负责暂停点、交付决定、批量 checkpoint、恢复流水线及父进程异常清理。 +- `job_worker.py` 仍采用单重型 Job 子进程;字幕 Job 与转写/切片共享 lease、heartbeat、取消和人工重试接口。 +- 渲染尝试顺序为可用 `h264_nvenc`、`libx264`,音频可安全复制时优先复制,否则转 AAC;最终统一验证 H.264、`yuv420p`、音轨、时长和文件大小。 +- 发布任务保存 `subtitle_delivery_mode` 及 revision/验证证据。自动字幕来源只有在审核、验证和文件三项同时成立时可进入发送中心。 +- AI 纠错独立于自动流水线:Provider 返回的内容只能映射既有 cue 的文本,不允许改变时间、说话人或未知 cue;建议 revision 必须人工接受。 + +## 2026-08-23:统一字幕架构 + +```text +转写 checkpoint / 旧 transcript.md + ↓ +source subtitle_track → immutable subtitle_revision → subtitle_cues(ms) + ↓ 按 output_clip 原片边界快照截取 +clip subtitle_track → immutable subtitle_revision → SRT/VTT/ASS/编辑器 +``` + +- `subtitle_data_service.py` 是字幕数据事实入口,负责轨、版本、cue、同步、导入导出、质量检查和服务端波形 peaks。 +- 原片轨只保存一条 active track;内容变化产生新 revision。切片轨记录来源 track/revision,未人工编辑时可同步,人工编辑后只进入 `pending_sync`。 +- revision 内容不可原地更新;审核只改变 revision 状态,渲染必须固定引用 revision id,避免编辑过程中改变已排队输出。 +- `pysubs2` 负责字幕格式与 ASS,不再手写固定分辨率 ASS。`wavesurfer.js` 仅消费服务端 peaks 与媒体流,不读取六小时完整音频到浏览器内存。 +- 单条兼容入口和批量入口都只负责创建持久化字幕 Job;HTTP 请求不再同步等待 FFmpeg。批量入口额外负责全自动流水线恢复。 + +## 2026-08-23:长直播选片层 + +`long_live_talk` 使用独立的 `long_live_talk_analyzer`: + +```text +结构化转写 +→ 300 秒窗口(60 秒重叠) +→ 每窗口独立 AI 召回与 SQLite checkpoint +→ 成功窗口时间轴并集 / 完整转写时间轴 = coverage_ratio +→ 时间重叠 + 语义相似去重 +→ 每小时密度筛选 +→ 跨小时轮询合并 +→ 总量上限 +→ analysis.json + ai_analysis_runs + clip_candidates +``` + +自动流水线在 `CLIP_SELECTING` 入口读取当前 `analysis_meta`。`analysis_incomplete=true` 或 `coverage_ratio < 0.90` 会直接中断,所以后续 `VIDEO_CUTTING`、内容准备和发送任务创建均不会执行。手动切片入口使用相同门禁。 + +## 2026-08-23:长直播基础层 + +- 新建任务必须显式选择 `general`、`variety_comedy` 或 `long_live_talk`;数据库的 `general` 默认值只用于旧数据兼容。 +- 数小时重型流程由 SQLite `workflow_jobs` 单 worker 串行领取,并为每个 Job 启动独立 Python 子进程。Job 使用 lease、heartbeat、尝试次数、取消标志和 checkpoint;Web 重启后可接管过期 lease,取消时终止子进程树。 +- 创建任务前检查视频轨、音轨、时长、编码、分辨率、帧率、首尾抽样解码与 E 盘剩余空间;超过 6 小时只提示。 +- 转写事实来源改为 `transcription_runs + transcription_chunks`。每块独立提交带校验和的结构化结果;`transcript.md` 是兼容导出。 +- 本地 faster-whisper 保存词级毫秒时间戳与置信度。仅 `TRANSCRIPTION_DEVICE=auto` 自动探测 CUDA,显式 `cpu` 不覆盖。 +- FFmpeg 音频提取写临时文件后原子替换,支持无进展超时、Job 取消和 Windows 进程树终止。 + ## 1. 当前架构概览 ### 1.1 架构形态 @@ -55,8 +116,8 @@ v2.1 的架构目标不是云端多租户,而是把一台 Windows 电脑上的 - **单体业务应用**:页面、路由、视频、AI、SQLite 和 Scheduler 保持同一 FastAPI 应用;Windows Worker 只隔离宿主 Chrome 操作。 - **SQLite 单写入者**:只有 Docker 内的 FastAPI 可以读写 `workflow.sqlite3`;Windows Worker 不导入数据库仓储、不打开 SQLite,只通过 HTTP 返回账号检查/发布结果并写独立执行日志。 -- **同步 FFmpeg**:视频处理通过 `subprocess` 同步调用,阻塞当前请求直到完成。 -- **无外部消息队列**:发布队列直接使用 SQLite 原子状态更新,无 Redis / Celery。 +- **受管 FFmpeg**:长流程由持久化 Job worker 调用;已接入的音频提取支持进度、无进展超时、取消和进程树终止。 +- **无外部消息队列**:工作流与发布队列均使用 SQLite 原子领取,无 Redis / Celery。 - **无用户体系**:单用户本地使用,通过 `LOCAL_ADMIN_TOKEN` 做简易鉴权。 - **统一定时调度**:立即发送与未来排期都先写 `SCHEDULED`,再由 `PublishScheduler` 原子领取。 - **终态原子提交**:平台结果、任务终态和事件由 FastAPI 在同一 SQLite 事务写入;任一步失败都会回滚,避免出现“平台结果已记但任务仍在发送中”。 diff --git a/docs/DATABASE_SCHEMA.md b/docs/DATABASE_SCHEMA.md index 6e93b09..cf5202f 100644 --- a/docs/DATABASE_SCHEMA.md +++ b/docs/DATABASE_SCHEMA.md @@ -1,5 +1,59 @@ # 数据库结构说明 +## 2026-08-24:字幕自动流水线字段 + +`subtitle_jobs` 在原有不可变 revision 引用上增加: + +| 字段 | 说明 | +| --- | --- | +| `workflow_job_id` | 所属持久化字幕 Job,用于取消、恢复和精确清理临时文件 | +| `validation_status` | `pending / verified`;旧记录默认为未验证,不能自动发布 | +| `validation_json` | FFprobe 的编码、像素格式、音轨、时长和音频处理证据 | +| `encoder` | 实际成功的 `h264_nvenc` 或 `libx264` | +| `verified_at` | 输出验证通过时间 | + +- 全自动字幕交付决定保存在 `tasks.auto_config_json.subtitle_delivery_mode`,值仅允许 `original / subtitled`,并记录 `subtitle_decided_at`。 +- `workflow_jobs.payload_json` 固定保存每个 output clip 与 revision 的组合;`checkpoint_json.completed` 只复用对应字幕 job 仍为 `completed + verified` 且文件存在的条目。 +- 缺少新字段的已有数据库先创建 `subtitle-auto-workflow` SQLite 在线备份,再执行幂等列/索引迁移;旧字幕文件和 job 不删除,但未验证旧记录不会获得自动发布资格。 + +## 2026-08-23:字幕 revision 数据结构 + +- `output_clip` 新增 `source_start_ms / source_end_ms / source_duration_ms / source_fingerprint / snapshot_source`。新切片使用 `cut_commit`,旧切片第一次使用时可从候选边界生成 `legacy_inferred`;之后候选修改不会漂移已保存边界。 +- `subtitle_tracks`:任务级原片轨或切片轨,记录 `source_track_id / source_revision_id / active_revision_id / sync_status / has_manual_edits`。 +- `subtitle_revisions`:不可变内容版本,记录来源 `asr / markdown / source_sync / manual / import / ai_suggestion`、父版本、状态、cue 数和 checksum。 +- `subtitle_cues`:毫秒级 `start_ms / end_ms`、文字、置信度、手工说话人和 `source_cue_id`;按 revision 和时间查询。 +- `subtitle_jobs.revision_id` 固定渲染使用的版本;样式表新增描边宽度、阴影深度、安全区百分比与说话人样式 JSON。 +- 启动迁移保持幂等;已有数据库缺少上述结构时先通过 SQLite 在线 backup 创建 `subtitle-editor-rebuild` 迁移前快照,不删除旧字幕 job 或历史切片。 + +## 2026-08-23:长直播 AI 窗口 checkpoint + +新增 `ai_analysis_windows`: + +| 字段 | 说明 | +| --- | --- | +| `task_id` | 所属任务 | +| `transcript_fingerprint` | 完整转写内容 SHA-256;内容变化后不会复用旧窗口 | +| `provider` / `model` | Provider 与模型隔离键 | +| `window_index` | 当前窗口序号 | +| `start_seconds` / `end_seconds` | 原片主时间轴范围 | +| `status` | `queued / running / completed / failed` | +| `attempt_count` | 累计真实请求次数 | +| `result_json` / `result_checksum` | 成功结果与 SHA-256 校验和 | +| `error_message` / `next_retry_at` | 最后错误与退避时间 | +| `created_at / updated_at / completed_at` | 生命周期时间 | + +唯一键由任务、转写指纹、Provider、模型、窗口序号和起止时间组成。迁移使用 `CREATE TABLE/INDEX IF NOT EXISTS`,已有数据库在变更前继续执行 SQLite 在线备份。 + +## 2026-08-23:长直播基础设施迁移 + +- `tasks` 增加 `highlight_density_per_hour INTEGER NOT NULL DEFAULT 4` 与 `highlight_total_limit INTEGER NOT NULL DEFAULT 30`;历史任务模式和值不改写。 +- `selection_profile` 合法值扩展为 `general / variety_comedy / long_live_talk`,数据库继续保留 `DEFAULT 'general'` 兼容旧数据。 +- `workflow_jobs` 增加尝试、退避时间、lease、heartbeat、取消和 checkpoint 字段。 +- 新增 `transcription_runs`,按任务、源指纹、Provider、模型、设备、计算类型和分块参数标识一次转写。 +- 新增 `transcription_chunks`,逐块保存毫秒边界、状态、尝试次数、结构化 JSON、SHA-256 校验和与错误。 +- 新索引覆盖 Job 领取、任务类型状态、活跃转写 run 与转写块状态。 +- 已存在数据库缺少新结构时,迁移前使用 SQLite Online Backup API 写入 `data/backups/workflow-before-long-live-foundation-*.sqlite3`,通过 `PRAGMA quick_check` 后才执行幂等增量迁移。 + ## 2026-08-01:康熙笑点优先 V2 兼容迁移 - `tasks` 新增 `selection_profile TEXT NOT NULL DEFAULT 'general'` 与 `final_clip_target INTEGER NOT NULL DEFAULT 5`。历史任务自动保持 `general`,不会改变原有分析行为。 @@ -65,7 +119,7 @@ - `tasks` 表新增 `task_dir_name` 字段,用来记录任务在存储盘里的实际文件夹名。 - `id` 仍是任务唯一 ID,用于数据库关联和网页地址;本地文件夹不再默认使用短 ID,而是使用 `task_dir_name`。 - 新建任务时会根据 `task_name` 生成安全的 Windows 文件夹名;重名时自动追加序号,避免覆盖旧目录。 -- `DELETE /api/tasks/{task_id}` 会永久删除系统托管的任务目录和发布包,再把 `is_deleted` 设为 `1` 并写入 `deleted_at`;NAS 或任务目录外的原片不会删除。 +- `DELETE /api/tasks/{task_id}` 会永久删除系统托管的任务目录和发布包,再把 `is_deleted` 设为 `1` 并写入 `deleted_at`;任务目录外的原片不会删除。 - 一次性迁移脚本为 `scripts/migrate_task_dirs_to_project_names.py`,默认 dry-run,带 `--apply` 才会移动文件夹并更新路径字段。 ## 2026-05-25:发布后台新增表 @@ -219,10 +273,10 @@ data/workflow.sqlite3 | `id` | TEXT | 任务唯一 ID,创建时自动生成 | | `task_name` | TEXT | 任务名称 | | `task_dir_name` | TEXT | 存储盘实际任务文件夹名;永久删除后保留原值作为隐藏历史记录,但对应目录不再存在 | -| `source_type` | TEXT | 视频来源:`upload` 或 `nas` | +| `source_type` | TEXT | 兼容字段;当前统一为 `upload`,新接口不再接受该参数 | | `platform` | TEXT | 平台类型:`douyin`、`bilibili`、`general` | -| `original_video_path` | TEXT | 本地上传视频路径,后续接真实上传后写入 | -| `nas_file_path` | TEXT | NAS / 本地已有视频路径 | +| `original_video_path` | TEXT | 当前统一使用的本机上传视频路径 | +| `nas_file_path` | TEXT | 旧版兼容字段;迁移后清空,新接口不再读写 | | `max_clip_duration` | INTEGER | 单条切片最长时长,单位:分钟;新建任务默认 10 分钟 | | `candidate_clip_count` | INTEGER | 希望 AI 输出的候选片段数量;新建任务默认 12 条 | | `ai_preference` | TEXT | AI 片段选择偏好 | @@ -365,8 +419,9 @@ data/workflow.sqlite3 早期项目骨架曾使用过 `title`、`source_path`、`max_clip_minutes`、`target_clip_count` 等草稿字段。当前初始化逻辑会自动补齐新字段,并把旧字段数据迁移到当前字段中。 为了不破坏已有本地数据库,旧字段不会被强制删除。后续代码以本文件列出的当前字段为准。 新任务默认值调整不会批量更新已有 `tasks` 记录;历史任务已经保存的最大时长和候选数量保持不变。 +账本迁移 `20260824_02_task_upload_only` 会在写入前创建 SQLite Online Backup:旧 `nas_file_path` 在 `original_video_path` 为空时归一到该字段,随后统一写为 `source_type='upload'` 并清空 `nas_file_path`。物理列继续保留,不删除或移动任何外部视频。 -任务删除采用“媒体永久删除、数据库历史隐藏保留”的方式:`DELETE /api/tasks/{task_id}` 只允许删除 `TASKS_DIR` 下与任务精确对应的目录、该任务的手动发布包和可确认归属的旧版项目 `tasks` 目录。删除成功后把 `is_deleted` 改为 `1` 并写入 `deleted_at`,候选片段、切片、字幕和发布历史仍留在 SQLite 中用于审计。NAS 或任务目录外的原片永远不参与删除;运行中的转写、切片或真实发送任务返回 409,避免后台进程重新生成文件。 +任务删除采用“媒体永久删除、数据库历史隐藏保留”的方式:`DELETE /api/tasks/{task_id}` 只允许删除 `TASKS_DIR` 下与任务精确对应的目录、该任务的手动发布包和可确认归属的旧版项目 `tasks` 目录。删除成功后把 `is_deleted` 改为 `1` 并写入 `deleted_at`,候选片段、切片、字幕和发布历史仍留在 SQLite 中用于审计。任务目录外的原片永远不参与删除;运行中的转写、切片或真实发送任务返回 409,避免后台进程重新生成文件。 `clip_candidates.reason` 是早期推荐理由字段,当前审核页优先读取 `highlight_reason`。数据库初始化时会把已有 `reason` 自动补到 `highlight_reason`。 diff --git a/docs/SUBTITLE_AND_PUBLISH_PLAN.md b/docs/SUBTITLE_AND_PUBLISH_PLAN.md index 8cbd3a2..05e9283 100644 --- a/docs/SUBTITLE_AND_PUBLISH_PLAN.md +++ b/docs/SUBTITLE_AND_PUBLISH_PLAN.md @@ -1,5 +1,19 @@ # 字幕与一键推送功能计划书 +## 2026-08-24 PR 4 实现状态 + +- 自动流水线已在字幕草稿后暂停,人工可审核并批量烧录,或明确跳过后使用原片恢复流程。 +- AI 纠错已使用建议 revision 和差异接受流程,不自动覆盖人工版本,也不允许修改时间或说话人。 +- 字幕烧录已改为持久化异步 Job,具备真实进度、取消、重试、checkpoint、临时输出、原子替换、NVENC 回退和 FFprobe 验证。 +- 发送中心已增加审核与验证证据门禁;自动任务不会把未审核或未验证的字幕成片用于发布。 +- 第一版仍不包含自动说话人分离、翻译、卡拉 OK、游戏画面多模态选片或绕过平台验证。 + +## 2026-08-23 PR 3 实现状态 + +- 已完成统一原片/切片 track、不可变 revision、毫秒 cue、切片边界快照、人工编辑保护。 +- 已完成 pysubs2 SRT/VTT/ASS、动态 ASS 样式、本地 wavesurfer 波形、虚拟列表和专业编辑操作。 +- 自动流水线审核暂停、AI 建议、持久化批量渲染、NVENC 回退和发送中心审核门禁已在 PR 4 完成。 + ## 1. 最终目标 把当前“上传视频 → 转写 → AI 分析 → 片段审核 → 自动切片”的流程继续向后延伸,形成: diff --git a/docs/TASK_FLOW.md b/docs/TASK_FLOW.md index 5f45b82..26b9af4 100644 --- a/docs/TASK_FLOW.md +++ b/docs/TASK_FLOW.md @@ -1,5 +1,43 @@ # 任务状态流转 +## 2026-08-24:自动字幕审核与恢复(PR 4) + +1. `video_cutting` 成功后进入 `subtitle_drafting`,为原片及成功切片准备字幕轨,然后写入 `pending_subtitle_review` 并结束当前流水线 Job。 +2. “审核并批量烧录”对 active revision 运行质量检查;字幕重叠属于阻断错误,其他中文质量项保持告警。审核后创建 `subtitle` 类型的持久化 Job。 +3. 字幕 Job 按固定 revision 串行处理切片,每条成功后写 checkpoint。失败或重启只重跑未验证条目;取消会终止 FFmpeg 进程树并保留旧 active 成片。 +4. 全部输出经 FFprobe 验证后创建从 `metadata_generating` 开始的恢复 Job;发送任务固定为 `subtitled` 来源。 +5. “跳过字幕”必须由用户在暂停状态明确操作;`POST /api/subtitles/tasks/{task_id}/skip-to-review` 写入 `original` 决定并进入 `pending_review`,不创建自动恢复 Job。旧 `skip-and-resume` 暂时保留为同语义兼容入口。 +6. 用户在片段审核页保存后才同步发送中心;完整关联时任务进入 `completed`,关联失败或缺失时保留 `pending_review` 并展示错误,不把整个任务误标为失败。 +7. 发送准备检查再次验证交付决定、revision 审核状态、渲染验证状态和文件存在性;任何证据缺失都不会进入实际发布。 + +新增主任务状态:`subtitle_drafting`、`pending_subtitle_review`、`failed_subtitle_drafting`。暂停状态不是失败,也不会每 3 秒继续轮询;页面提供字幕审核和显式跳过操作。 + +## 2026-08-23:字幕编辑数据流(PR 3) + +1. 完成转写后,从成功的结构化 checkpoint 创建原片主字幕 revision;旧任务没有 checkpoint 时兼容读取完整 `transcript.md`。 +2. 切片完成时固化原片起止毫秒快照;首次进入字幕工作台时按快照截取主字幕并换算为切片局部时间。 +3. 编辑、导入、拆分、合并、位移和替换均创建新 revision;自动保存使用当前 active revision 做乐观并发校验,冲突时返回 409,不覆盖他人或新版本。 +4. 未人工编辑的切片可跟随原片新 revision;人工切片只标记 `pending_sync`,用户明确强制同步前保留其 active revision。 +5. 审核将指定 revision 标记为 `approved`;单条和批量烧录都通过持久化 Job 异步执行,自动任务在审核点暂停并等待用户决定。 + +## 2026-08-23:长直播高光流程 + +1. 逐句时间戳转写按约 5 分钟、60 秒重叠生成窗口。 +2. 每个窗口分别召回金句观点、故事经历、情绪峰值、冲突反转、实用知识和互动幽默;远程失败单轮最多尝试 3 次。 +3. 成功窗口立即保存,重启或重试时直接复用;只重新请求未成功窗口。 +4. 跨窗口重复事件按时间重叠、关键时刻距离和文本语义合并。 +5. 每小时最多使用任务的 `highlight_density_per_hour`,再按跨小时轮询应用 `highlight_total_limit`。 +6. 成功窗口覆盖完整时间轴不足 90% 时,任务保留候选供查看但标记“分析不完整”;自动/手动切片和发送中心同步均被阻止。 + +## 2026-08-23:长直播基础流程 + +`明确选择模式 → 上传本机视频 → 媒体/磁盘预检 → SQLite Job 排队 → 音频提取 → 分块转写 checkpoint → transcript.md 兼容导出`。 + +- 同一源指纹和转写配置下,重试只处理缺失或失败块;成功块通过 SHA-256 校验后复用。 +- 源内容或 Provider、模型、设备、分块参数改变时创建新的 run,旧 run 保留但不复用。 +- 默认只有一个本地重型 Job;Web 重启后 worker 接管 lease 已过期的任务。 +- `long_live_talk` 本阶段只完成创建与基础设施,分层高光选片由下一阶段实现。 + ## 1. 主任务状态列表(tasks.status) | 状态码 | 中文展示 | 说明 | @@ -198,7 +236,7 @@ output_clip 生成成功 └─ opencli_publish → 显式兼容开关 → PUBLISHED / FAILED / NEED_REVIEW ``` -`platform` 只能是 `douyin` / `bilibili`;`publish_mode` 只能表示执行方式,禁止互相混用。发送中心的“补充缺失任务”只补缺,不覆盖已有任务的执行方式。 +`platform` 只能是 `douyin` / `bilibili`;`publish_mode` 只能表示执行方式,禁止互相混用。发送中心内容准备区的“同步遗漏切片”固定使用 `use_ai=false`,只补建尚未进入发送中心的草稿与默认封面,不覆盖已有任务的执行方式或文案。AI 文案仅由内容卡“AI 重写本条文案”或明确勾选后的“AI 重写已选文案”触发;页面加载、排期计划和执行记录不会调用 AI 重写。 ### 发送任务状态(publish_jobs.status) diff --git a/docs/UI_REFERENCE.md b/docs/UI_REFERENCE.md index 9758f9a..b4d4a62 100644 --- a/docs/UI_REFERENCE.md +++ b/docs/UI_REFERENCE.md @@ -1,10 +1,98 @@ # UI 参考说明 +## 2026-08-26 更新:全站克制动效系统 + +- 动效遵循“状态反馈优先、装饰最少”:使用原生 CSS/JavaScript,统一短时缓动,不引入 Motion、AutoAnimate、Animate.css、AOS、GSAP 或 View Transitions 运行时。 +- 工作台四项核心统计轻微渐显;不做数字滚动,也不让最近任务逐行延迟。 +- 新建任务只为长直播条件设置和文件选择提供确认反馈;任务详情的进度条平滑变化,状态、当前操作和数量只在值真的变化时短暂强调,运行日志保持稳定。 +- 片段审核和字幕工作台的进度条、弹窗、抽屉与保存状态使用短过渡;视频、波形、播放头、长转写文本和字幕虚拟列表保持静态,避免播放或输入时反复重播。 +- 发送中心的标签页、日历日期详情、账号/排期抽屉、排期预览和用户主动生成的结果使用一次性过渡;后台轮询重建的历史记录与 42 格日历不播放批量入场。 +- 任务列表、片段/字幕总览通过行 hover 和按钮按压提供定位反馈;完整转写与系统静态状态没有合适的入场语义,不为“每页都有动画”强行动画整页。 +- `prefers-reduced-motion: reduce` 下非必要动画、过渡、旋转与脚本平滑滚动均降为即时;无 JavaScript 时内容默认可见。 +- 1180px 以下片段审核主区固定为单列,并清除卡片的最小内容宽度,390px 页面无整页横向溢出。 + +## 2026-08-26 更新:发送中心 AI 文案操作分层 + +- 页面顶栏只保留账号管理;原“补充缺失任务”改名为“同步遗漏切片”,与“补齐抖音缺失封面”共同放入内容准备维护区。 +- 维护区使用独立浅色条明确说明:同步动作只补建遗漏草稿和默认封面,不调用 AI、不修改已有文案。 +- 每条内容卡的 AI 按钮紧贴发布文案字段,命名为“AI 重写本条文案”;勾选内容后的固定操作栏使用“AI 重写已选文案”,范围一眼可见。 +- 批量 AI 仅在内容准备标签页显示;排期计划只处理时间,执行记录只处理结果和人工复核。 +- 页面打开和刷新不再静默执行旧草稿全局 AI 升级,AI 文案必须由用户明确点击触发。后端兼容接口保留,不改变历史数据或发布状态机。 +- 窄屏下维护区与单条文案工具栏改为纵向排列,按钮使用整行宽度。 + +## 2026-08-26 更新:片段审核累计统计 + +- 片段审核工作台顶部只保留三项结果指标:“累计审核任务、已通过视频、已完成任务”,移除待 AI、待检查、可生成切片和异常任务等过程型统计。 +- “累计审核任务”表示当前任务库中已经生成候选片段并进入审核流程的未删除任务;“已通过视频”表示这些任务下当前启用且未删除的视频片段;“已完成任务”兼容手动、全自动和部分完成状态,任务行徽标也按同一大小写规范化口径显示。 +- 每张卡片显示自己的业务说明,不再统一写“来自当前任务库”,避免把自动启用片段误解为全部人工确认。 +- 桌面端三张卡片等宽排列;1180px 及以下沿用现有单列响应式规则。该布局使用片段审核专属类,不影响字幕工作台的统计卡片。 + +## 2026-08-26 更新:任务详情实时进度同步 + +- 状态概览新增“总流程进度”和“当前操作”两层信息:总进度保留全流程位置,当前操作显示转写、AI、切片或托管发布的独立百分比和最新说明。 +- 手动任务与全自动任务共用同一个 `/live-status` 快照;有活动任务或待执行发布排期时每 3 秒局部刷新,页面隐藏时暂停,回到前台立即读取最新状态。 +- 运行日志、状态标签、进度条、地铁步骤、候选数和输出数来自同一次快照;AI 区域的独立轮询不再单独覆盖右侧日志。 +- 全自动流程兼容内部的小写运行状态,转写、AI 和切片分别稳定定位在第 3、4、6 步,不会短暂退回第 1 步。 +- 已同步发送中心的任务会继续显示发布汇总:等待/排期/发送中保持蓝色当前阶段,失败或需复核显示黄色警告,全部发布成功后才显示发布完成。手动和全自动任务遵循相同口径。 +- 小屏下“当前操作”卡改为单列,百分比、说明和时间线保持可读,不引入新的前端框架。 + +## 2026-08-26 更新:工作台周统计总览 + +- 工作台顶部由分散的统计卡、工作流程和今日焦点合并为一个“本周任务概览”面板,保留清晰留白、浅色玻璃背景和蓝色主强调。 +- 核心指标精简为“本周新增任务、已切片任务、待推送任务、失败任务”;不再显示“待处理、待检查、待加字幕”。 +- 周范围按上海时区计算并显示在概览标题区;本周新增总数直接由核心指标呈现,不再增加重复的每日柱状图。 +- “待推送任务”使用有效切片的最新发布状态统计;已发布、已导出或已取消内容不会重复计入,说明文字同时给出仍待发送或复核的切片数。 +- 最近任务表保持原有结构和入口。 + +## 2026-08-25 更新:任务 20 与上传单入口 + +- 新建任务的历史名称下拉最多显示最近 5 个未删除、去重名称;仍允许手工输入新名称。 +- 新建入口只保留“上传本机视频”,不再显示视频来源单选、本地/NAS 路径浏览器或已有文件创建入口。 +- 每小时高光密度和候选总上限只在“长直播高光(语言类)”模式显示;通用与康熙模式不显示、也不提交这两个参数。 +- 已完成切片或进入发送中心的任务再次点击 AI 分析时显示明确冲突提示,原“已完成”状态和现有候选、切片、发送记录保持不变。 +- “跳过字幕”统一命名为“跳过字幕,进入片段审核”;跳过后不自动进入发送中心,必须在片段审核页保存确认后才同步。 + +## 2026-08-24 更新:字幕审核暂停与批量 Job + +- 全自动任务详情在切片后显示“字幕待审核”主操作;状态卡隐藏处理中提示,明确说明未确认前不会创建发送任务。 +- 页面同时提供“进入字幕审核”和“跳过字幕,进入片段审核”。跳过是有确认提示的显式决定,不使用默认值,也不直接续跑发送中心。 +- 字幕工作台顶部新增自动流程门禁卡:可审核所有 active revision 并批量烧录,也可明确跳过;运行时展示 Job 状态、百分比、当前条目、取消和失败重试。 +- AI 建议入口作用于勾选 cue,差异面板并排显示原文、建议和理由;用户可逐条勾选后接受,未接受内容不会写入 active revision。 +- 单条“自动加字幕”改为入队并轮询状态,不再让页面等待同步 FFmpeg;完成后刷新带字幕预览。 +- 小屏下门禁操作、Job 进度和差异行按单列排列,继续使用现有 Apple 风格卡片、蓝色强调色和原生 HTML/CSS/JavaScript。 + +## 2026-08-23 更新:专业字幕编辑器 + +- 字幕任务页顶部新增“原片主时间轴 · 切片继承编辑器”,轨下拉明确区分原片和每条切片,并显示人工版、待同步、revision 与审核状态。 +- 视频与字幕双向联动;视频上覆盖当前字幕,波形使用本地 wavesurfer Regions/Timeline,拖动蓝色区间直接修改毫秒起止。 +- 字幕列表固定高度并虚拟渲染,只创建可见行 DOM;适合数千 cue。每行提供勾选、跳转、开始/结束毫秒、文字、说话人和质量提醒。 +- 工具栏包含搜索替换、新增、拆分、合并、删除、批量位移、撤销、重做和立即保存;普通编辑 1.8 秒后自动保存为新 revision。 +- 质量卡将重叠显示为错误,时长、间隔、行长、行数和阅读速度显示为提醒;页面不会自动改字。 +- 样式侧栏增加描边宽度、阴影深度和安全区;实际 ASS 画布按视频 9:16、16:9 或 1:1 尺寸生成。 +- 小屏将视频与波形改为单列,字幕行横向保留关键编辑字段;不引入 React/Vue。 + +## 2026-08-23 更新:长直播分析完整性状态 + +- 长直播 AI 窗口失败时,任务详情的 AI 状态显示“分析不完整”和当前覆盖率,不把部分结果冒充为已完整分析。 +- 用户可再次点击原 AI 分析按钮;系统复用已确认成功窗口,只补“明确未被 Provider 接受”的安全失败。已开始但本地未确认结果的窗口会显示计费不确定,不自动再次请求。 +- 覆盖率低于 90% 时,自动流水线和“生成切片”返回中文说明,不进入发送中心;页面布局和现有通用/综艺审核卡片保持不变。 + +## 2026-08-23 更新:新建任务与长直播入口 + +1. 当前只提供上传本机视频入口,上传默认最大 4 GB;本地/NAS 已有文件与目录浏览入口已移除。 +2. 后端统一以 `original_video_path` 读取任务原片,不再接受已有文件 JSON 创建请求。 +3. “选片模式”首项是不可提交的“请选择选片模式”,没有隐藏默认值;可选通用、康熙笑点、长直播语言高光。 +4. 长直播模式显示每小时密度(1~10,默认 4)和候选总上限(1~50,默认 30);其他模式隐藏。 +5. 创建前检查媒体与空间;超过 6 小时为非阻断提示,无音轨、不可解码或空间不足时不创建任务。 + ## 2026-08-23 更新:v2.1 片段审核批量操作与后台切片进度 - 片段审核列表顶部增加粘性“全选当前列表 / 取消全选”,实时显示启用数量并支持半选状态;切换只改前端草稿,仍由“保存修改”统一写入数据库。 - “生成切片”先保存当前审核选择,再创建后台切片任务并显示排队、运行、完成或失败进度;重复点击或并发请求复用同一条进行中任务,不重复切割。 - AI 分析运行期间统一锁定分析按钮、Prompt、候选数量和相关表单控件,完成或失败后恢复原来的禁用状态,避免同一任务并发覆盖候选结果。 +- AI 分析按钮现在只负责创建或复用持久化后台 Job,页面按 Job 进度轮询;关闭或刷新页面不会取消任务,应用重启后可由新 Worker 接管过期 lease。Job 完成后页面重新读取分析摘要和历史,不依赖长 HTTP 请求保持连接。 +- 通用、综艺和长直播分析只要存在局部失败,页面仍保留成功候选供人工检查,但“生成切片”和全自动选片都会显示覆盖率/未完成提示并停止;补齐失败单元前不会进入切片或发送中心。 +- 综艺全局质量评审缺失、只覆盖部分候选或返回无效 `source_id` 时同样只允许人工检查;系统不会用默认评分冒充完整评审后自动切片。 - 页面继续沿用 Apple 风格浅色卡片、蓝色主强调和轻量玻璃效果;侧栏版本统一显示“v2.1 本地高光生产版”。 ## 2026-08-22 更新:发送中心固定抖音与文案规则 @@ -12,8 +100,8 @@ - 发送中心全局平台卡只保留抖音;`platform=bilibili` 深链参数会被忽略。B站历史记录、API 和 Publisher 继续保留在后端,但当前前台不展示 B站任务或发送入口。 - 内容卡账号状态直接复用统一发送预检:唯一正常抖音账号自动选中并显示“内容完整”,多账号显示“缺少:发布账号”,登录失效显示“账号需登录”。 - 每个原始处理任务组头增加“全选本任务 / 取消全选”复选框;只作用于本组当前可操作的抖音视频,支持部分选择状态,任务组折叠时仍可使用。 -- 标题输入限制 30 字并显示实时计数;标签提示 4~6 个、每个 2~3 字;简介限制 15~35 字并显示实时计数。单条和批量按钮统一命名为“AI 重写文案”,成功后立即刷新三个输入框。 -- 页面首次打开会异步调用幂等旧草稿升级;成功项原地更新,失败项提示保留原文。页面仍保持 Apple 风格浅色卡片、玻璃感和响应式布局,不引入前端框架。 +- 标题输入限制 30 字并显示实时计数;标签提示 4~6 个、每个 2~3 字;简介限制 15~35 字并显示实时计数。单条与批量入口分别命名为“AI 重写本条文案”和“AI 重写已选文案”,成功后立即刷新三个输入框。 +- 旧草稿升级接口继续保留后台兼容,但页面不再自动触发;AI 文案只通过单条或明确勾选后的批量入口执行。页面仍保持 Apple 风格浅色卡片、玻璃感和响应式布局,不引入前端框架。 ## v2.1 界面总览 @@ -199,7 +287,7 @@ - 系统状态页不再使用“AI 配置”弹窗,页面中间直接展示“三类 AI 接口配置”表单。 - 三块配置统一命名为 `1. 音频转写`、`2. 分析文字稿,生成候选切片`、`3. 发送中心生成发布文案`。 - 音频转写区域配置火山引擎远程转写;文字稿分析和发布文案分别配置独立 OpenAI-compatible 远程接口。 -- API Key 输入框改为普通文本框,个人本地使用时直接完整显示,不再做脱敏占位。 +- API Key 输入框为密码框且读取时始终留空;已配置时显示“已配置,留空保持”,避免密钥进入页面源码或浏览器状态。留空保存会保留现有密钥。 - 关键路径列表同步展示三类接口的地址、模型或资源 ID、Key 状态,方便一眼定位哪一类接口未配置。 ## 2026-06-04 更新:任务详情日志侧栏精简 @@ -216,7 +304,7 @@ ## 2026-08-02 更新:任务列表永久删除 - 任务列表右侧危险操作为“永久删除”,不再提供 E 盘回收站;点击后明确提示原片副本、音频、转写、切片、字幕、封面和发布包都将永久删除且无法恢复。 -- 确认框同时说明 NAS 或任务目录外的原始视频不会被删除;运行中的处理或真实发布任务由后台拒绝删除并展示具体原因。 +- 确认框说明任务目录外的原始视频不会被删除;运行中的处理或真实发布任务由后台拒绝删除并展示具体原因。 - 新建任务后的本地文件夹名与项目名保持一致;任务详情页仍展示任务目录,方便用户直接对照 E 盘项目文件夹查找完成视频。 ## 2026-05-27 更新:AI 片段完整性与检查入口 @@ -313,7 +401,7 @@ docs/design/live_streaming_slicing_workflow_ui_16x9.png ### 1. 工作台首页 -包含左侧导航、顶部搜索、任务统计卡片、v1.1 流程展示和最近任务列表。 +包含左侧导航、顶部搜索、本周任务概览和最近任务列表。概览只保留本周新增、已切片、待推送和失败四项核心指标,不再使用低样本量的每日新增任务柱状图占用首页空间。 ### 2. 任务列表 @@ -327,7 +415,7 @@ docs/design/live_streaming_slicing_workflow_ui_16x9.png 包含任务名称、视频上传、平台类型、单条切片最大时长、候选片段数量和流程说明。 -任务名称输入框提供受应用控制的历史候选:点击输入框会按创建时间从新到旧展示任务列表中的可见任务名称,最新任务排第一,已隐藏/永久删除的任务不会出现;点击候选可完整填入,仍支持手动输入任意新标题。该候选只读展示,不改变创建请求字段与提交流程。 +任务名称输入框提供受应用控制的历史候选:点击输入框会按创建时间从新到旧展示最多 5 个可见且去重的任务名称,最新任务排第一,已隐藏/永久删除的任务不会出现;点击候选可完整填入,仍支持手动输入任意新标题。 当前新建任务页只保留上传本机视频入口。上传视频会保存到 E 盘任务目录;NAS / 本地路径选择、浏览路径和打开目录入口已从页面移除。 @@ -375,7 +463,9 @@ v1.3.0 起,新建任务页可勾选“新建后自动跑完整流水线”。 包含 E 盘存储目录、FFmpeg、FFprobe、SQLite 数据库、任务数量和最近异常任务状态。 -当前系统状态页已补充推荐运行地址 `http://127.0.0.1:8001`、AI 接口状态、任务状态统计和页面内三类 AI 接口配置表单。配置保存到 `.env`,API Key 输入框完整显示当前值;本地 AI 区域保留为 Ollama 手动兜底模型选择。 +当前系统状态页已补充推荐运行地址 `http://127.0.0.1:8001`、AI 接口状态、任务状态统计和页面内三类 AI 接口配置表单。配置仍保存到 `.env`,但读取接口和页面不再返回 API Key 原值;密码框默认留空,已配置时显示“已配置,留空保持”。本地 AI 区域保留为 Ollama 手动兜底模型选择。 + +系统页面和业务页面默认只允许从本机回环地址访问;Docker 端口也只绑定 `127.0.0.1`。若未来确实需要非本机客户端访问,必须显式配置本地管理员 Token,并由客户端使用 Bearer Token。页面不注入该 Token,避免密钥进入 HTML、JavaScript 或浏览器日志。 系统状态页现在会直接展示音频转写、文字稿分析和发送中心发布文案三类接口的配置状态。如果 Key 为空或明显过短,页面会把对应接口标记为待配置或异常。远程接口默认面向 DeepSeek-compatible 配置,推荐 `https://api.deepseek.com`、`deepseek-v4-flash` 和 `chat_completions`;本地 AI 默认推荐 `qwen3:8b`,避免 `qwen3:14b` 在当前机器上长时间超时。 diff --git a/docs/agent_tasks/2026-08-23-long-live-foundation.md b/docs/agent_tasks/2026-08-23-long-live-foundation.md new file mode 100644 index 0000000..23cbcc0 --- /dev/null +++ b/docs/agent_tasks/2026-08-23-long-live-foundation.md @@ -0,0 +1,54 @@ +# PR 1 执行任务:长直播基础设施 + +## 背景 + +NiuMa Studio 现有链路能按固定窗口处理长素材,但新建任务默认写死综艺模式,已有文件入口未接入页面,转写分块结果不能恢复,长任务依赖 Web 进程内后台任务,也缺少媒体与磁盘预检。 + +## 目标 + +在 `feature/long-live-foundation` 分支完成可独立验收的长直播基础层,为后续长直播选片、字幕编辑器和字幕自动流水线提供稳定数据与任务基础。 + +## 允许修改范围 + +- `app/models/`、`app/routers/`、`app/services/`、`app/db/`、`app/templates/`、`app/static/` +- `tests/` 中与任务创建、媒体预检、Job、转写有关的测试 +- `docs/`、`DEVELOPMENT_LOG.md`、`NEXT_STEPS.md`、依赖声明 + +## 禁止修改范围 + +- 不批量修改已有任务的 `selection_profile` +- 不删除或移动用户视频、数据库、日志和配置 +- 不改变现有 AI 选片算法和字幕工作台业务逻辑 +- 不覆盖明确配置的 faster-whisper CPU/CUDA 选项 +- 不提交 `.env`、密钥、Cookie、媒体、数据库、日志、缓存或构建产物 + +## 已确定实现要求 + +1. 新建任务必须明确选择 `general`、`variety_comedy` 或 `long_live_talk`;页面、上传 API 和 JSON API 均不得静默兜底。 +2. 长直播专用参数为每小时密度 1~10(默认 4)和总上限 1~50(默认 30),仅长直播模式使用。 +3. 页面同时支持浏览器上传和受根目录保护的本地/NAS已有文件;浏览器上传仍默认限制 4 GB。 +4. 创建任务前用 FFprobe 检查音视频流、时长、编码、分辨率、帧率与可读性,并按源文件、PCM、预计输出和安全余量检查可用空间;超过 6 小时只警告。 +5. `workflow_jobs` 增加 lease、heartbeat、尝试次数、取消标志和 checkpoint;单 worker 接管过期任务。 +6. `transcription_runs`、`transcription_chunks` 持久化每个块的结果、校验与错误;源文件指纹改变时创建新 run,不复用旧块。 +7. 统一保存毫秒级段落、词时间戳和置信度;`transcript.md` 只保留兼容导出。 +8. FFmpeg/ASR 支持进度、无进展超时和进程树终止;保留显式 CPU 配置,并提供可验证的 `auto` CUDA 选择。 + +## 验收标准 + +- 页面没有隐藏选片默认值;三种模式均可创建;缺失或非法模式返回中文错误。 +- 已有文件可浏览并创建任务,且不会复制或删除外部唯一原片。 +- 无音轨、不可探测、空间不足时拒绝创建;超过 6 小时显示非阻断提示。 +- 第 N 个转写块失败后重试只处理缺失块;指纹改变后不复用旧 checkpoint。 +- queued/running Job 在进程重启后可重新接管;取消和重试接口有持久状态。 +- 历史任务行为与数据不变。 + +## 测试命令 + +```powershell +python -m pytest tests/test_selection_profile_default.py tests/test_media_storage_lifecycle.py tests/test_job_queue.py tests/test_split_services.py -q +python -m pytest tests/ -q +``` + +## 返回格式 + +主代理验收时报告:实际修改文件、迁移与兼容策略、测试命令和结果、diff/敏感信息检查、提交哈希、推送分支与 PR 链接。 diff --git a/docs/agent_tasks/2026-08-23-long-live-selection.md b/docs/agent_tasks/2026-08-23-long-live-selection.md new file mode 100644 index 0000000..754a5fc --- /dev/null +++ b/docs/agent_tasks/2026-08-23-long-live-selection.md @@ -0,0 +1,62 @@ +# PR 2 执行任务:长直播高光分层选片 + +## 背景 + +PR 1 已完成长直播任务字段、媒体预检、持久化 Job 与转写分块 checkpoint。现有通用/综艺 AI 分析仍是请求内顺序处理,不能按窗口恢复,也没有长直播的时段覆盖和总量门禁。 + +## 目标 + +1. 为 `long_live_talk` 新增独立的 5 分钟重叠窗口召回。 +2. 将每个 AI 窗口的状态、尝试次数、结果和错误持久化,成功窗口可跨重启复用。 +3. 在每小时内去重排序,再进行全局合并,默认每小时 4 条、总计最多 30 条。 +4. 支持金句观点、故事经历、情绪峰值、冲突反转、实用知识、互动幽默六类高光。 +5. 覆盖率低于 90% 时明确标记分析不完整,禁止自动切片和发送中心同步。 + +## 允许修改范围 + +- `app/db/database.py` +- `app/services/ai/` 下的长直播分析实现及必要导出 +- `app/services/ai_analysis_workflow_service.py` +- `app/services/pipeline_engine.py` +- 与分析完成门禁直接相关的视频切片工作流 +- PR 2 专项测试、数据库/架构/任务流程/UI 文档、开发日志与下一步 + +## 禁止修改范围 + +- 不改变 `general` 与 `variety_comedy` 的算法和历史结果。 +- 不实现字幕数据层、字幕编辑器或字幕渲染;这些属于 PR 3/PR 4。 +- 不读取或写入任何 `.env`、密钥、Token、Cookie。 +- 不合并 PR,不删除分支,不改写 Git 历史。 + +## 已确定实现要求 + +- 窗口默认 300 秒、重叠 60 秒。 +- 远程窗口失败最多 3 次,指数退避;本地失败也记录错误但不做隐藏兜底。 +- checkpoint 以任务、转写内容指纹、Provider、模型及窗口参数隔离。 +- 同一事件用时间重叠/接近度与文本语义相似度合并。 +- 先按小时分桶和配额,再按时间排序输出,不能让前段高分候选耗尽全局名额。 +- 覆盖率按成功窗口对原时间轴的并集计算;不足 90% 保留成功结果供查看,但设置 `analysis_incomplete=true`。 +- 自动流水线在选片前检查最新分析元数据;不完整时停止,不进入切片、内容准备或发送中心。 + +## 验收标准 + +- 六小时结构化时间轴能生成完整窗口清单,并验证覆盖率、每小时密度、总上限和时间覆盖。 +- 模拟第 N 窗口失败后,失败状态和错误可查询;再次运行只请求缺失窗口。 +- 模拟跨窗口重复事件可以合并。 +- 覆盖率不足 90% 时自动流水线明确报错且不调用切片。 +- 原有通用和综艺测试不回归。 + +## 测试命令 + +```powershell +python -m pytest tests/test_long_live_selection.py tests/test_auto_pipeline.py -q +python -m pytest tests/ -q +python -m compileall app tests +``` + +## 返回格式 + +- 修改文件与关键设计 +- 专项及全量测试结果 +- Git diff/敏感信息检查结论 +- 中文 commit、推送分支、堆叠 PR 链接 diff --git a/docs/agent_tasks/2026-08-23-subtitle-auto-workflow.md b/docs/agent_tasks/2026-08-23-subtitle-auto-workflow.md new file mode 100644 index 0000000..05ad8f0 --- /dev/null +++ b/docs/agent_tasks/2026-08-23-subtitle-auto-workflow.md @@ -0,0 +1,67 @@ +# PR4 执行任务:字幕审核、渲染与自动流水线整合 + +## 背景 + +PR3 已建立原片/切片统一字幕轨、不可变 revision、毫秒级 cue、专业编辑器和动态 ASS。当前自动流水线仍在切片后跳过字幕,同步 FFmpeg 会阻塞 HTTP,发送中心也只凭 `subtitle_jobs.status=completed` 判断带字幕成片。 + +## 目标 + +1. 自动任务在切片后生成字幕草稿并进入 `PENDING_SUBTITLE_REVIEW`,不创建发布任务。 +2. 用户可明确选择“审核并批量烧录”或“跳过字幕,使用原片继续”。 +3. 字幕烧录使用 `workflow_jobs` 持久化队列,支持真实进度、取消、失败重试、checkpoint 和重启接管。 +4. 渲染固定引用 approved revision,使用临时输出、NVENC 失败回退 libx264、FFprobe 验证和成功后原子切换;失败保留旧 active 成片。 +5. AI 纠错只创建非 active 建议 revision,只能改文字和 cue 内断行;用户查看 diff 后选择接受。 +6. 发送中心只自动使用“approved revision + verified render”的字幕成片;明确跳过字幕时允许原片继续。 + +## 允许修改范围 + +- `app/models/task.py`、`app/models/subtitle.py` +- `app/db/database.py` +- `app/services/pipeline_engine.py`、`job_service.py`、`job_worker.py` +- `app/services/subtitle_data_service.py`、`subtitle_workflow_service.py`、新增字幕 AI 服务 +- `app/services/auto_publish_service.py`、`publish_service.py`、必要的发布 readiness 门禁 +- `app/routers/tasks.py`、`app/routers/subtitles.py` +- 字幕工作台、任务详情页及其 JS/CSS +- PR4 相关测试与项目文档 + +## 禁止修改范围 + +- 不改抖音/B站真实登录、验证码、风控和 Publisher 实现。 +- 不自动发布、部署、合并 PR 或删除用户数据。 +- 不引入说话人自动分离、翻译、卡拉 OK、多模态游戏选片。 +- 不读取或写入 API Key、Token、Cookie、`.env` 内容。 + +## 已确定实现要求 + +- 自动任务暂停是正常终态,不作为失败;恢复从 `METADATA_GENERATING` 开始。 +- 批量烧录的 payload 固定保存每条 `output_clip_id/revision_id`,重试复用 checkpoint,已成功条目不重复渲染。 +- 单条和批量 HTTP 入口都只入队,不同步运行 FFmpeg。 +- 字幕成片新增验证状态;旧版 completed 字幕按未验证处理,不自动进入发送中心。 +- explicit skip 决策写入任务 `auto_config_json`,不得由“没有字幕”静默推断。 +- AI 建议不可激活、不可改变时间戳/说话人;接受建议后生成新的人工草稿 revision。 + +## 验收标准 + +- 切片后生成全部 clip track 草稿并暂停,未创建 publish job。 +- 审核批量烧录成功后自动排队恢复流水线,发布任务使用 verified subtitle。 +- 明确跳过后恢复流水线,发布任务使用 original。 +- 未审批、未验证、revision 已过期的字幕不能作为发布源。 +- Job 可查询、取消、失败重试;取消/失败不覆盖旧 active 成片,不残留 `.part`。 +- 9:16、16:9、1:1、中文字体、音频映射、NVENC 回退、FFprobe 验证均有测试。 +- AI diff 支持选择接受,建议 revision 永不自动覆盖 active revision。 + +## 测试命令 + +```powershell +.venv\Scripts\python.exe -m ruff check app tests +.venv\Scripts\python.exe -m compileall -q app tests +node --check app/static/js/app.js +node --check app/static/js/subtitle-editor.js +.venv\Scripts\python.exe -m pytest tests/test_subtitle_auto_workflow.py tests/test_subtitle_editor.py tests/test_auto_pipeline.py tests/test_job_queue.py tests/test_publish_readiness.py tests/test_publish_task_linkage.py -q +.venv\Scripts\python.exe -m pytest tests/ -q +``` + +## 返回格式 + +- Luna Operator 只返回准确命令、通过/失败数、耗时、警告和失败证据,不修改生产文件。 +- 主代理检查实际 diff、范围外修改、敏感信息、TODO/debug、临时文件和 Git 状态后再验收。 diff --git a/docs/agent_tasks/2026-08-23-subtitle-editor-rebuild.md b/docs/agent_tasks/2026-08-23-subtitle-editor-rebuild.md new file mode 100644 index 0000000..ad971f2 --- /dev/null +++ b/docs/agent_tasks/2026-08-23-subtitle-editor-rebuild.md @@ -0,0 +1,65 @@ +# PR 3 执行任务:字幕数据层与专业编辑器重构 + +## 背景 + +现有字幕工作台从 `transcript.md` 最多读取 120 行,按整秒时间生成固定 1080×1920 ASS,并在 HTTP 请求内同步烧录。长直播需要以原片毫秒时间轴为事实来源,切片字幕必须继承且不能覆盖人工编辑。 + +## 目标 + +1. 建立 `subtitle_tracks / subtitle_revisions / subtitle_cues` 统一字幕模型。 +2. 保存 output clip 的不可变原片起止毫秒快照,完成原片→切片本地时间换算。 +3. 使用 `pysubs2==1.9.0` 实现 SRT/VTT/ASS 导入导出与 ASS 序列化。 +4. 本地固定 `wavesurfer.js@7.12.11`,使用 Regions/Timeline 和服务端 peaks 构建长视频编辑器。 +5. 提供文本、毫秒时间、说话人、增删、拆分、合并、批量位移、搜索替换、撤销重做、自动保存和质量告警。 + +## 允许修改范围 + +- 字幕数据库表、output_clip 快照和 subtitle_jobs revision 引用。 +- 新字幕服务、模型、API 路由、媒体 peaks 接口。 +- 字幕工作台模板、专用 JS/CSS 和固定版本 vendor 文件。 +- requirements、第三方许可证、测试和项目文档。 + +## 禁止修改范围 + +- 不实现自动说话人分离、翻译、卡拉 OK 或 AI 自动覆盖。 +- 不复制 GPL-3.0 VideoCaptioner 源码;仅借鉴流程。 +- PR 3 不接入自动流水线审核暂停和异步批量烧录,这属于 PR 4。 +- 不删除旧 `subtitle_jobs` 或旧带字幕成片;旧同步烧录入口暂时兼容。 +- 不读取或写入 secrets,不合并 PR,不改写 Git 历史。 + +## 已确定实现要求 + +- revision 内容创建后不可原地改写;每次人工保存创建子 revision 并切换 active。 +- 渲染 Job 增加 `revision_id`,后续只能引用固定 revision。 +- source track 来自结构化转写 checkpoint;无结构化结果时完整读取 Markdown,不再截断 120 行。 +- clip track 使用 `source_start_ms/source_end_ms` 快照截取 source cues 并换算本地毫秒。 +- 未人工编辑 clip 自动跟随 source active revision;已有人工 revision 只标记 `pending_sync`。 +- 质量规则只返回 warning/error,不自动修改文字。 +- 波形 peaks 服务端缓存并限制点数,浏览器不解码完整 6 小时音频。 +- 样式序列化使用实际视频宽高,并提供安全区与说话人样式。 + +## 验收标准 + +- SRT/VTT/ASS 往返保留毫秒时间,中文不乱码。 +- 超过 120 cues 不截断;重叠、时长、间隔、行长和阅读速度规则正确。 +- 原片→切片边界截取准确;人工 clip revision 不被 source 更新覆盖。 +- 数千 cues 范围查询分页;前端实现虚拟滚动及编辑操作。 +- 六小时 peaks 输出点数受控且有缓存。 +- 旧字幕任务、通用/综艺/长直播流程不回归。 + +## 测试命令 + +```powershell +.venv\Scripts\ruff.exe check app tests +.venv\Scripts\python.exe -m compileall app tests +node --check app/static/js/subtitle-editor.js +.venv\Scripts\python.exe -m pytest tests/test_subtitle_editor.py tests/test_split_services.py tests/test_versioning_rollback.py -q +.venv\Scripts\python.exe -m pytest tests/ -q +``` + +## 返回格式 + +- 数据模型、API 和编辑器能力摘要 +- 第三方版本与许可证 +- 专项/全量测试证据 +- 中文 commit、推送分支、堆叠 PR 链接 diff --git a/docs/agent_tasks/2026-08-24-task20-workflow-ui.md b/docs/agent_tasks/2026-08-24-task20-workflow-ui.md new file mode 100644 index 0000000..72d932e --- /dev/null +++ b/docs/agent_tasks/2026-08-24-task20-workflow-ui.md @@ -0,0 +1,55 @@ +# 任务 20 状态修复与新建流程精简 + +## 背景 + +任务 `3210d91ee1fb` 已完成 AI、第二次切片和发送中心 12/12 关联,但随后一次手动 AI 重入删除仍被切片引用的候选记录,触发 SQLite 外键约束并把主状态覆盖为失败。新建任务页同时仍暴露 NAS 入口,长直播设置的 `hidden` 被 CSS 覆盖,跳过字幕会越过片段审核直接恢复自动流水线。 + +## 目标 + +- 阻止并发或已有成片任务再次启动手动 AI,且冲突不得降级成功状态。 +- 新建任务仅允许上传本机视频,任务名称历史限制为 5 条。 +- 长直播密度只在长直播模式显示和提交。 +- 跳过字幕后进入片段审核,审核同步成功后任务正确完成。 +- 在数据证据完整时仅修复任务 20 主状态,不重跑 AI、切片或发布同步。 + +## 允许修改范围 + +- `app/` 下相关模型、路由、服务、模板、JavaScript、CSS 和数据库迁移账本。 +- 对应 `tests/` 回归测试。 +- `DEVELOPMENT_LOG.md`、`NEXT_STEPS.md`、`docs/UI_REFERENCE.md`、`docs/TASK_FLOW.md`、`docs/DATABASE_SCHEMA.md`。 +- 正式 SQLite 中任务 20 的 `status/progress/error_message/last_error/updated_at`,以及一条审计日志;上传单入口迁移只归一旧来源字段。 + +## 禁止修改范围 + +- AI Provider、Codex 认证、Prompt、超时、分析算法和任务 20 的候选分析结果。 +- 任务 20 的切片文件、候选内容、发布任务、排期和字幕决定。 +- 真实平台发布、账号状态、Cookie、Token、`.env` 和外部原片。 + +## 已确定实现要求 + +- 手动 AI 冲突返回 HTTP 409;自动流水线持有有效租约的首次 AI 不受影响。 +- 候选替换在同一事务内检查 `output_clip` 引用并返回领域冲突,不泄漏 SQLite 错误。 +- JSON 已有文件创建和 `/api/files/browse` 下线;数据库旧列保留但不再提供能力。 +- `skip-to-review` 写入 `subtitle_delivery_mode=original`,不创建自动恢复 Job,返回审核地址;旧路径保留兼容。 +- 审核同步无须重切且完整关联时显式完成任务;失败时保留审核状态。 + +## 验收标准 + +- 任务 20 显示已完成、100%、无外键错误,12 个活跃文件和 12 条排期均保持原 ID。 +- 全自动运行中或已有活跃切片/发布关联的手动 AI 请求均被拒绝且数据不变。 +- 新建页无 NAS 文案、控件和请求;历史名称最多 5 条。 +- 通用和康熙模式不显示/提交长直播参数,长直播模式正常显示/提交。 +- 跳过字幕后进入审核,保存同步后进入发送中心且不触发真实发布。 + +## 测试命令 + +- `python -m pytest -q`(使用项目 `.venv`) +- `python -m compileall -q app tests` +- `node --check app/static/js/app.js` +- `node --check app/static/js/subtitle-editor.js` +- `git diff --check` +- SQLite `PRAGMA integrity_check`、`PRAGMA foreign_key_check` + +## 返回格式 + +报告修改文件、测试结果、任务 20 恢复证据、分支、提交、推送和 PR;任何失败需给出准确命令与错误,不得用模拟测试声称真实 AI 或真实发布成功。 diff --git a/docs/agent_tasks/2026-08-25-p1-3d-subtitle-atomicity.md b/docs/agent_tasks/2026-08-25-p1-3d-subtitle-atomicity.md new file mode 100644 index 0000000..df807ab --- /dev/null +++ b/docs/agent_tasks/2026-08-25-p1-3d-subtitle-atomicity.md @@ -0,0 +1,70 @@ +# P1.3d 字幕批准原子性与跨进程恢复任务 + +## 背景 + +工程审计确认字幕批量烧录仍有三个稳定性缺口:批量批准逐条提交、渲染完成与 active 结果切换分两次提交、进程被接管后旧执行仍可能落下最终文件或激活旧结果。现有实现已经使用 workflow job lease、attempt 专属 `.part.mp4` 和 checkpoint,本轮在这些机制上收口,不引入新队列或新架构。 + +## 目标 + +1. 批量批准、固定待渲染 revision、创建或复用字幕 workflow job,以及全自动任务的字幕交付模式写入形成单个数据库事务。 +2. 任一 revision 校验或 job 创建失败时,整批批准不得出现部分成功。 +3. 只有仍持有当前 workflow job lease 的执行可以把验证通过的临时文件切换为最终文件并激活 subtitle job。 +4. 新执行接管或重试时,清理同一 workflow job 遗留的未激活 processing 记录和 attempt 临时文件;不得删除已激活结果、已验证最终文件或其他 job 的临时文件。 +5. checkpoint 只接受当前已激活、已验证且文件仍存在的字幕结果,避免恢复时复用已被后续版本替代的记录。 +6. 字幕 Job 完成与后续自动流水线 Job 的创建/复用必须同事务提交,取消或 lease 失效时两者都不得落库。 + +## 允许修改范围 + +- `app/services/subtitle_data_service.py` +- `app/services/subtitle_auto_workflow_service.py` +- `app/services/subtitle_workflow_service.py` +- 为复用原子 job 创建所必需的 `app/services/job_service.py` +- 为父进程异常退出收口所必需的 `app/services/job_worker.py` +- 为剩余 Workflow lease/follow-up/发布草稿跨进程边界所必需的 `app/services/auto_publish_service.py` +- 为发布草稿已提交但 checkpoint 尚未写入恢复所必需的 `app/services/pipeline_engine.py` +- 字幕、job fencing、pipeline checkpoint 相关测试 +- `.codemap/modules.json`、`PROJECT_AUDIT.md`、`DEVELOPMENT_LOG.md`、`NEXT_STEPS.md` + +## 禁止修改范围 + +- 不改变用户可见的字幕审核流程和页面结构。 +- 不新增数据库 Schema,不迁移或写入活动 `data/workflow.sqlite3`。 +- 不执行真实 FFmpeg、AI Provider、Chrome 或平台投稿。 +- 不处理 P1.4 的通用超时/重试策略,不处理 P1.5 的管理员门禁和 XSS。 +- 不做字幕模块的大规模拆分或格式化。 + +## 已确定实现要求 + +- 所有批量输入先完成无副作用收集;正式批准必须在 `BEGIN IMMEDIATE` 中重新校验 track、revision、cue 数量与时间重叠。 +- workflow job 的“查找活动任务或插入新任务”必须复用同一事务连接,不能在批准提交后才创建。 +- 渲染成功落库必须检查 `workflow_job_id + status=running + lease_owner + lease_token + 未过期`;无 workflow job 的同步单条渲染保持兼容。 +- 文件最终切换与数据库 active 切换在同一短事务保护区完成;后续数据库失败时删除本次唯一命名的孤儿最终文件。 +- 清理函数只接受受管理字幕目录中的、名称包含精确 workflow job 标记的 `.part.mp4` 或无数据库引用的本次最终文件,单文件失败需可诊断且不能伪装成清理成功。 +- 所有 lease 校验时间必须在取得数据库写锁后计算;续跑 Job 与当前字幕 Job 终态必须原子提交。 +- 源轨生成、切片轨同步和字幕导入必须在写锁内重读 active revision,不能用事务外旧基线覆盖人工新版本。 +- ASS 文件按 revision 确定性生成并可复用,不作为 attempt 临时文件删除;只有带本次 workflow 标记的临时/孤儿视频属于中断清理范围。 + +## 验收标准 + +- 批次第 N 项校验失败,前 N-1 项仍保持原状态,且不创建 workflow job。 +- job 插入失败时没有批准或 delivery mode 的半提交。 +- stale lease 不能完成、激活或覆盖新执行;旧 active 结果保持不变。 +- 新 lease 可标记遗留 processing 记录并清理自己的 `.part.mp4`,其他 job 文件不受影响。 +- 有效 checkpoint 可恢复;未激活、revision 不匹配、文件缺失或已被替代的 checkpoint 会重做。 +- Pytest 必须明确使用唯一临时 `test_workflow.sqlite3`;活动数据库不得作为测试目标。 + +## 测试命令 + +```powershell +.venv\Scripts\python.exe -m pytest -q tests/test_subtitle_editor.py tests/test_subtitle_auto_workflow.py tests/test_job_fencing.py tests/test_job_queue.py tests/test_auto_pipeline.py tests/test_pipeline_checkpoint.py tests/test_pipeline_state_stability.py tests/test_task_state_machine.py tests/test_publish_task_linkage.py tests/test_versioning_rollback.py +.venv\Scripts\python.exe -m ruff check app tests +.venv\Scripts\python.exe -m compileall -q app +``` + +## 返回格式 + +- 修改文件与关键行为变化 +- 测试命令、通过数、失败证据 +- 活动数据库未变化证据 +- Codemap 受影响模块复评 +- commit、分支、push 与 PR 链接 diff --git a/docs/agent_tasks/2026-08-25-p1-4-provider-resilience.md b/docs/agent_tasks/2026-08-25-p1-4-provider-resilience.md new file mode 100644 index 0000000..84cf9f8 --- /dev/null +++ b/docs/agent_tasks/2026-08-25-p1-4-provider-resilience.md @@ -0,0 +1,65 @@ +# P1.4 AI / FFmpeg 失败边界与幂等治理 + +## 背景 + +P1.3d 已完成字幕批准批次、目录清理和跨进程恢复。本轮处理审计路线图中工作量最大的 P1.4:AI / ASR / FFmpeg 超时、429 / 5xx、空响应、Markdown / 错误 JSON、重试幂等与重复计费。 + +## 目标 + +- 为 AI HTTP、Codex CLI、火山引擎转写与核心 FFmpeg / FFprobe 路径提供明确的超时和可诊断错误。 +- 区分“请求明确未被执行,可安全自动重试”和“请求可能已经执行,不得自动重试”。 +- 删除会因格式校验失败而自动再次计费的模型调用,优先本地容错解析并保留人工重试入口。 +- 长直播窗口只对有明确安全证据的暂时性错误自动重试,并继续复用已完成 checkpoint。 +- FFmpeg 超时或异常时清理本次未完成文件,不把部分输出当作成功产物。 + +## 允许修改范围 + +- `app/services/ai/**` +- `app/services/ai_analysis_workflow_service.py` +- `app/services/subtitle_ai_service.py` +- `app/services/transcript_service.py` +- `app/services/task_service.py` +- `app/services/video_cut_service.py` +- `app/services/subtitle_data_service.py` +- `app/core/config.py` 与 `.env.example`(仅必要的非敏感超时 / 重试配置) +- 与本轮直接相关的测试、Codemap、审计和进度文档 + +## 禁止修改范围 + +- 不调用真实 AI、远程转写、真实 FFmpeg 生产素材或发布平台。 +- 不改活动数据库、不执行生产 migration、不删除用户素材。 +- 不引入自动 Provider fallback,不把 5xx、超时或坏 JSON 一概机械重试。 +- 不改变业务页面结构,不扩大到 P1.5 安全门禁范围。 + +## 已确定实现要求 + +1. AI Provider 错误必须带阶段、HTTP 状态、是否适合自动重试、是否存在重复计费不确定性。 +2. 429 可依据“请求被限流拒绝”进行有限退避重试;连接建立前失败可有限重试;5xx、读取超时、空响应和错误 JSON 默认视为结果不确定,不自动再次调用计费模型。 +3. 通用、综艺、字幕 AI 的 JSON 纠错不得通过第二次模型调用完成;Markdown 围栏和可局部提取 JSON 由本地解析器处理。 +4. 长直播 checkpoint 必须记录失败类别;仅安全错误进入自动重试,其余停在明确可恢复状态。 +5. 核心 ffprobe 详情查询不得无限阻塞或因文件 `stat` 异常直接 500;切片超时 / 启动失败必须清理部分文件并返回明确失败。 +6. 火山引擎 HTTP 429 / 5xx / 超时必须分类,并且 checkpoint 只在确定结果成功后标记完成。 + +## 验收标准 + +- 错误 JSON、Markdown、空响应、HTTP 429、HTTP 5xx、网络超时均有隔离单测。 +- 证明格式错误不会产生第二次 AI 调用。 +- 证明长直播只重试安全错误,并复用已完成窗口。 +- 证明 ffprobe 超时 / OSError 不阻断任务详情,FFmpeg 切片失败不遗留可误认成成功的部分文件。 +- Ruff、compileall 和 P1.4 定向回归通过;测试数据库必须与活动数据库隔离。 + +## 测试命令 + +```powershell +.\.venv\Scripts\python.exe -m ruff check app tests +.\.venv\Scripts\python.exe -m compileall -q app +.\.venv\Scripts\python.exe -m pytest -q tests/test_codex_cli_provider.py tests/test_ai_json_parsing.py tests/test_long_live_selection.py tests/test_variety_comedy_selection.py tests/test_auto_pipeline.py tests/test_pipeline_checkpoint.py tests/test_long_live_foundation.py tests/test_subtitle_auto_workflow.py tests/test_p0_security.py tests/test_versioning_rollback.py +``` + +## 返回格式 + +- 修改文件与行为变化 +- 错误分类 / 自动重试矩阵 +- 测试命令、通过数、失败数、warnings +- 是否触发真实 AI / ASR / FFmpeg / 平台调用 +- 剩余风险与下一轮 P1.5 衔接点 diff --git a/docs/agent_tasks/2026-08-25-p1-4b-ai-consistency.md b/docs/agent_tasks/2026-08-25-p1-4b-ai-consistency.md new file mode 100644 index 0000000..b75f706 --- /dev/null +++ b/docs/agent_tasks/2026-08-25-p1-4b-ai-consistency.md @@ -0,0 +1,80 @@ +# P1.4b AI 结果一致性与恢复任务 + +## 背景 + +P1.4 已统一 AI/ASR/FFmpeg 的超时、429/5xx、坏 JSON 和不确定计费边界。最终 Codemap 复评仍确认三条 HIGH:AI 候选、run 与任务终态分开提交;人工 AI 入口没有持久化 Job/lease;普通和综艺分析的局部失败会被当成完整成功。 + +## 目标 + +1. `/api/tasks/{task_id}/process/ai` 只创建或复用持久化 `ai_analysis` Workflow Job,由现有 Worker 子进程执行并支持过期 lease 接管。 +2. 候选替换、active run 切换、新 run 插入和任务 `pending_review` 终态在同一个 SQLite 事务内完成,并在提交点重新验证当前 lease。 +3. `candidate_clips.json` 降为可重建派生缓存;数据库 active run 是权威来源,缺失文件可恢复且不会再次调用 Provider。 +4. `general` / `variety_comedy` 的局部失败写入结构化 `analysis_meta`;不完整分析可供人工查看,但自动选片、切片和发布必须停止。 + +## 允许修改范围 + +- `app/services/ai_analysis_workflow_service.py` +- `app/services/ai/ai_clip_analyzer.py` +- `app/services/ai/variety_comedy_analyzer.py` +- `app/services/ai/long_live_talk_analyzer.py` +- `app/services/ai/unit_checkpoint.py` +- `app/services/job_service.py` +- `app/services/job_worker.py` +- `app/services/pipeline_engine.py` +- `app/models/task.py` +- `app/routers/tasks.py` +- `app/static/js/app.js` +- 对应隔离测试、Codemap 与项目文档 + +## 禁止修改范围 + +- 不新增数据库列或迁移活动数据库。 +- 不调用真实 AI、FFmpeg、Chrome、抖音或 B站。 +- 不重写整个 AI Service、Job 系统或前端框架。 +- 不处理 P2 God Service 拆分、代码风格清理或历史 dead code。 + +## 已确定实现要求 + +- Job 创建前在同一事务检查任务、下游产物和活动自动流水线;并发重复点击只能得到同一个 AI Job。 +- Worker 使用现有 owner/token/heartbeat;旧 lease 不能提交候选、run、任务终态或 Job 终态。 +- 最终事务必须检查 job id、task id、允许 job type、running、owner/token、lease 未过期且未取消。 +- Provider 成功但派生文件写入失败时,不得把已经提交的 AI 结果当成 Provider 失败而重新计费;后续从 active run 重建文件。 +- 旧 `candidate_clips.json` 和旧 run 继续可读,不批量重写历史数据。 +- 合法空窗口不算失败;超时、HTTP、坏 JSON、坏结构和无效条目需要记录失败。全窗口失败仍为 hard failure。 +- `analysis_incomplete=true` 时保留候选供人工复核,但自动流水线不得进入切片。 + +## Codemap 复评后的必要范围补充 + +- 普通/综艺每个 Provider 单元在调用前写入持久化 Job checkpoint;成功后写校验和。前次调用已开始但本地结果未确认时 fail closed,不自动重复计费。 +- 长直播严格要求 `moments` 数组;坏条目和计费不确定窗口必须进入不完整状态。综艺全局评审不完整时设置质量降级并锁住自动切片。 +- AI Job 取消、父 Worker 异常退出、停机释放和进程树终止失败必须让 Job/Task 进入明确且可恢复的状态。 +- 失败/取消后的显式重试必须复用原 Workflow Job 和单元账本;Provider model、endpoint、protocol 变化必须使输入指纹变化,但不得借新 Job 丢弃旧不确定证据。 +- active run/meta 损坏、Schema 或选片模式不一致、非规范覆盖率必须 fail closed;手动和自动切片复用同一验证器。 +- 安全交叉复审发现的静态媒体 CORS/API 写入 Origin 混用必须一并封口,但不扩大为新的权限系统。 + +## 验收标准 + +- 人工 AI API 立即返回 queued/running Job;页面轮询 Job,完成后刷新分析摘要与历史。 +- 同一任务并发请求只创建一个 active AI Job;活动 auto pipeline 时仍返回 409 且不创建 AI Job。 +- 候选/run/任务状态任一点失败全部回滚;旧 Worker 提交被拒。 +- active run 可重建缺失的 `candidate_clips.json`,重建不调用 Provider。 +- general/variety 部分失败具有覆盖率和失败单元元数据,自动切片被门禁;完整结果不受影响。 +- failed/cancelled AI Job 从原行重排队并保留 checkpoint;损坏或模式漂移的分析元数据无法进入任何切片入口。 +- 定向测试、全量隔离 pytest、Ruff、Compileall、JavaScript 语法和 `git diff --check` 通过,活动数据库哈希不变。 + +## 测试命令 + +```powershell +.\.venv\Scripts\python.exe -m pytest -q tests/test_split_services.py tests/test_job_queue.py tests/test_job_fencing.py tests/test_auto_pipeline.py tests/test_pipeline_checkpoint.py tests/test_provider_resilience.py tests/test_variety_comedy_selection.py +.\.venv\Scripts\python.exe -m pytest -q --disable-warnings -k "not test_real_ffmpeg_render_supports_three_aspect_ratios" +.\.venv\Scripts\ruff.exe check app tests +.\.venv\Scripts\python.exe -m compileall -q app +node --check app/static/js/app.js +git diff --check +``` + +## 返回格式 + +- 列出修改文件、原子性/lease/恢复语义、测试结果和活动 DB 哈希。 +- 明确剩余无法消除的外部 Provider 已受理但本地尚未落账窗口。 +- 不声称测试等于真实 AI 调用或真实发布验证。 diff --git a/docs/agent_tasks/2026-08-25-p1-5-local-security.md b/docs/agent_tasks/2026-08-25-p1-5-local-security.md new file mode 100644 index 0000000..a283d0d --- /dev/null +++ b/docs/agent_tasks/2026-08-25-p1-5-local-security.md @@ -0,0 +1,68 @@ +# P1.5 本地安全边界修复任务 + +## 背景 + +P1.3d 与 P1.4 已分别收紧跨进程恢复和 AI / FFmpeg 失败边界。本轮只处理本地单用户 V1 的高收益安全问题,不引入登录系统或企业级权限架构。 + +## 目标 + +1. AI 配置页面和 `/api/settings/ai` 不再把已保存的 API Key、Access Key、管理员 Token 或 `.env` 绝对路径送到浏览器。 +2. AI 配置输入拒绝 CR/LF/NUL、过长值、非法协议和非法 URL,阻断 `.env` 配置注入。 +3. 删除已确认的动态 `innerHTML` 注入点,服务端数据只通过 `textContent` 写入 DOM。 +4. 本地管理页面、API 与媒体入口默认只接受本机 Host;非本机 API 客户端必须提供 `LOCAL_ADMIN_TOKEN`。Docker 正式端口仅绑定 `127.0.0.1`。 + +## 允许修改范围 + +- `app/main.py` +- `app/models/settings.py` +- `app/services/ai_config_service.py` +- `app/templates/base.html` +- `app/templates/system_status.html` +- `app/static/js/app.js` +- `app/static/js/publish-center.js` +- `app/templates/publish.html` +- `app/services/publish_domain.py` +- `app/services/publish_scheduler.py` +- `app/services/publish_service.py` +- `docker-compose.yml` +- 安全回归测试、Codemap 与项目进度文档 + +## 禁止修改范围 + +- 不改任务、AI、字幕、切片、排期、发布的业务状态机。 +- 不读取或输出 `.env` 中的真实值。 +- 不访问真实 AI、平台、浏览器 Worker 或活动数据库。 +- 不引入多用户、OAuth 登录、反向代理或新的第三方依赖。 + +## 已确定实现要求 + +- Secret 字段对外值固定为空,另用布尔状态表示“已配置”;空输入继续保留旧 Secret。 +- Token 比较使用常量时间比较;不得把 Token 放在 HTML、JavaScript、日志或错误信息中。 +- 本机 Host 支持 `localhost`、`127.0.0.1`、`::1`;测试客户端单独兼容。跨站 Origin 的写请求拒绝。 +- `/health`、静态资源和 favicon 保持公开;其余非本机入口关闭,远程 API 仅允许 Bearer Token。 +- 所有动态 UI 文本使用 DOM 节点与 `textContent`。 +- Provider / Worker 返回的作品链接仅在 HTTP(S) 且域名与目标平台匹配时进入已发布结果或页面链接;异常链接进入人工复核。 +- `.env` 写入前由 Pydantic 校验字符串边界,不做自动“修复”。 + +## 验收标准 + +- 配置 GET、保存响应和系统状态 HTML 均不包含测试 Secret 或真实 `.env` 路径。 +- Secret 输入为密码框且空值表示保留现有配置。 +- 换行注入、`javascript:` / `file:` URL、非法响应路径返回 422,且没有写入配置文件。 +- 本机页面/API 可用;非本机 Host 无 Token 返回 403;错误 Token 返回 401;正确 Token 可访问 API;跨站写请求返回 403。 +- 已确认的两处 XSS payload 只显示为文本,不进入 `innerHTML`。 +- Docker 端口映射为 `127.0.0.1:8001:8001`。 + +## 测试命令 + +- `python -m ruff check app tests` +- `python -m compileall -q app` +- `node --check app/static/js/app.js` +- `node --check app/static/js/publish-center.js` +- P1.5 定向 pytest +- 使用 pytest 隔离数据库运行完整非真实外部调用测试集 + +## 返回格式 + +- 返回准确命令、隔离数据库路径、通过/失败数量。 +- 明确说明没有读取 Secret、没有触发真实 AI / FFmpeg / 平台调用、没有写活动数据库。 diff --git a/docs/agent_tasks/2026-08-26-clip-review-summary.md b/docs/agent_tasks/2026-08-26-clip-review-summary.md new file mode 100644 index 0000000..4fff001 --- /dev/null +++ b/docs/agent_tasks/2026-08-26-clip-review-summary.md @@ -0,0 +1,57 @@ +# 片段审核累计统计精简 + +## 背景 + +`/clips` 顶部当前显示“待 AI 分析、待检查、可生成切片、已完成、异常任务”五项阶段统计。用户希望只保留累计审核任务、已通过视频和已完成任务三个长期有用的汇总指标。 + +## 目标 + +- 将顶部统计精简为三项:累计审核任务、已通过视频、已完成任务。 +- 每项使用明确、可验证且排除软删除数据的 SQLite 口径。 +- 保持审核队列、单任务指标和操作入口不变。 + +## 允许修改范围 + +- `app/services/task_query_service.py` +- `app/templates/clips_overview.html` +- `app/static/css/styles.css` +- `tests/test_task_query_service.py` +- `tests/test_p1_1_db_performance.py`(仅相关回归需要时) +- `DEVELOPMENT_LOG.md`、`NEXT_STEPS.md`、`docs/UI_REFERENCE.md` + +## 禁止修改范围 + +- 任务、候选片段、切片和发布状态机。 +- SQLite Schema、迁移和真实数据。 +- AI Provider、Prompt、账号、Cookie、Token、`.env` 和认证配置。 +- 真实 AI、切片、字幕或平台发布流程。 + +## 已确定实现要求 + +- “累计审核任务”统计当前未删除任务中至少有一条未删除候选片段的去重任务数;当前数据库没有可靠的任务级人工审核事件,不能把 `reviewed=1` 冒充纯人工审核。 +- “已通过视频”统计上述未删除任务中 `is_deleted=0 AND enabled=1` 的候选片段总数,页面说明为“当前启用的视频片段”。 +- “已完成任务”兼容 `completed / completed_with_errors / COMPLETED`,统一大小写后统计当前未删除任务。 +- 三张卡分别提供自己的统计说明,不再统一显示“来自当前任务库”。 +- 统计网格由五列改为三列;现有响应式折叠逻辑继续保留。 + +## 验收标准 + +- `/clips` 顶部只显示“累计审核任务、已通过视频、已完成任务”。 +- 当前活动库预期显示 `20 / 163 / 20`;数值随数据变化时以同一口径实时渲染。 +- 软删除任务和软删除候选片段不计入累计审核与通过数。 +- 大写自动完成状态 `COMPLETED` 不再被漏算。 +- 审核队列结构、任务行指标、详情和进入审核按钮保持不变。 +- 桌面与 390px 窄屏无横向溢出。 + +## 测试命令 + +- `.venv\Scripts\python.exe -m pytest tests/test_task_query_service.py tests/test_p1_1_db_performance.py -q` +- `.venv\Scripts\python.exe -m pytest -q` +- `.venv\Scripts\python.exe -m compileall -q app tests` +- `.venv\Scripts\python.exe -m ruff check app/services/task_query_service.py tests/test_task_query_service.py` +- `git diff --check` +- 浏览器刷新 `http://127.0.0.1:8001/clips`,核对三项指标、审核队列、窄屏布局和页面错误。 + +## 返回格式 + +报告统计口径、修改文件、测试与浏览器证据、分支、提交、推送和 PR;不得把本地统计验证描述为真实 AI、切片或平台发布验证。 diff --git a/docs/agent_tasks/2026-08-26-dashboard-weekly-overview.md b/docs/agent_tasks/2026-08-26-dashboard-weekly-overview.md new file mode 100644 index 0000000..c1d7445 --- /dev/null +++ b/docs/agent_tasks/2026-08-26-dashboard-weekly-overview.md @@ -0,0 +1,56 @@ +# 工作台周统计与发布口径修正 + +## 背景 + +工作台首页当前把全部已完成切片同时显示为“待加字幕”和“待推送”,没有读取 `publish_jobs` 的真实状态;因此已经发布的内容仍被错误计入待推送。顶部的七张独立统计卡与“工作流程 / 今日处理焦点”也形成了分散的信息块,不符合本次要求的整体统计面板。 + +## 目标 + +- 把“今日新增任务”改为上海时区周一至周日的“本周新增任务”。 +- 移除“待处理”“待检查”“待加字幕”统计。 +- “待推送任务”只统计仍有有效切片尚未进入发布终态的任务,排除已发布、已导出和已取消内容。 +- 将顶部统计区、工作流程和今日焦点合并重构为一个整体周概览面板,并加入本周每日新增任务柱状图。 + +## 允许修改范围 + +- `app/services/task_query_service.py` +- `app/templates/index.html` +- `app/static/css/styles.css` +- Dashboard 相关测试文件。 +- `DEVELOPMENT_LOG.md`、`NEXT_STEPS.md`、`docs/UI_REFERENCE.md`。 + +## 禁止修改范围 + +- 任务、切片、字幕和发布状态机。 +- SQLite 表结构、迁移和真实数据。 +- 发布调度、账号、Cookie、Token、`.env`、AI Provider 和认证配置。 +- 真实抖音 / B站发送、部署和 PR 合并。 + +## 已确定实现要求 + +- 周边界使用 `settings.app_timezone`(默认 `Asia/Shanghai`),范围为周一 00:00 至下周一 00:00。 +- 时间解析兼容带时区 ISO、`Z` 结尾和旧的无时区记录;无时区记录按应用时区解释。 +- 待推送口径以每个有效、已完成切片的最新 `publish_jobs` 记录为准;最新状态为 `PUBLISHED`、`EXPORTED`、`CANCELLED` 时排除,没有发布记录或最新状态仍为活跃 / 异常状态时计入。 +- 首页展示待推送的去重任务数,并在说明中展示对应切片数,避免把切片数误称为任务数。 +- 图表使用服务端生成的语义化 HTML 和 CSS,不引入新的前端依赖。 +- 最近任务表保持不变。 + +## 验收标准 + +- 首页不再出现“今日新增任务”“待处理”“待检查”“待加字幕”“工作流程”“今日处理焦点”。 +- 首页出现“本周任务概览”、四项核心指标和周一至周日柱状图。 +- 已发布切片不计入待推送;最新发布状态仍为等待、排期、发送中、失败或需复核时继续计入。 +- 本周统计在周边界和 UTC / 上海时区跨日场景下正确。 +- 桌面和窄屏布局均无明显溢出,页面无控制台错误。 + +## 测试命令 + +- `.venv\Scripts\python.exe -m pytest tests/test_task_query_service.py tests/test_p1_1_db_performance.py -q` +- `.venv\Scripts\python.exe -m pytest -q` +- `.venv\Scripts\python.exe -m compileall -q app tests` +- `git diff --check` +- 浏览器刷新 `http://127.0.0.1:8001/`,核对文案、统计值、图表、窄屏布局和控制台。 + +## 返回格式 + +报告修改文件、统计口径、测试结果、浏览器验收证据、分支、提交、推送和 PR;不得把本地测试描述为真实平台发布验证。 diff --git a/docs/agent_tasks/2026-08-26-publish-ai-action-placement.md b/docs/agent_tasks/2026-08-26-publish-ai-action-placement.md new file mode 100644 index 0000000..e5a2538 --- /dev/null +++ b/docs/agent_tasks/2026-08-26-publish-ai-action-placement.md @@ -0,0 +1,64 @@ +# 发送中心 AI 文案操作分层 + +## 背景 + +发送中心顶部的“补充缺失任务”和勾选内容后的“批量 AI 重写文案”位置接近、说明不足,容易被理解成重复操作。实际前者通过 `use_ai=false` 补建遗漏的发布草稿并生成默认封面,后者会逐条调用 AI 并覆盖已选草稿的标题、话题和简介。页面加载时还会静默触发一次旧草稿全局 AI 升级,不符合 AI 必须由用户明确选择后执行的交互边界。 + +## 目标 + +- 将补建草稿与 AI 文案操作按“维护动作 / 内容动作”清晰分层。 +- AI 重写只保留在内容准备中的单条和显式勾选批量入口。 +- 排期计划和执行记录不展示 AI 重写。 +- 页面打开不再自动调用旧草稿全局 AI 升级。 + +## 允许修改范围 + +- `app/templates/publish.html` +- `app/static/css/styles.css` +- `app/static/js/publish-center.js` +- `tests/test_publish_center_cleanup.py` +- `tests/test_publish_center_browser.py`(仅相关断言需要时) +- `DEVELOPMENT_LOG.md` +- `NEXT_STEPS.md` +- `docs/UI_REFERENCE.md` +- `docs/TASK_FLOW.md` + +## 禁止修改范围 + +- 发布任务、调度器、Publisher 和账号的后端状态机。 +- AI Prompt、Provider、模型、认证或计费配置。 +- SQLite Schema、迁移和活动数据。 +- 真实 AI、封面生成、排期或平台发布操作。 + +## 已确定实现要求 + +- 页面顶栏只保留“账号管理”;“补充缺失任务”移入内容准备的维护区,改名为“同步遗漏切片”。 +- 维护区明确说明:只补建遗漏草稿和默认封面,不调用 AI、不修改已有文案。 +- 单条按钮放在发布文案字段上方,命名为“AI 重写本条文案”,明确会立即生成并保存本条标题、话题和简介。 +- 勾选后的按钮命名为“AI 重写已选文案”,仅在内容准备标签页显示;排期与执行记录不显示 AI 重写。 +- 移除页面加载时对 `/api/publish/jobs/metadata/upgrade-pending-douyin` 的自动调用;保留后端兼容接口,不删除历史能力。 +- 补建遗漏草稿继续固定使用 `use_ai=false`,不改变后端语义。 + +## 验收标准 + +- 顶部“账号管理”旁不再出现含糊的“补充缺失任务”。 +- 内容准备区能看到“同步遗漏切片”和清晰的无 AI 说明。 +- 每条内容卡只显示“AI 重写本条文案”;勾选后显示“AI 重写已选文案”。 +- 切换到排期计划或执行记录后,批量 AI 按钮隐藏。 +- 打开或刷新 `/publish` 不请求旧草稿全局 AI 升级接口。 +- 现有单条、批量、补建、封面和排期后端接口保持不变。 +- 桌面与 390px 窄屏布局可读,无横向溢出。 + +## 测试命令 + +- `.venv\Scripts\python.exe -m pytest tests/test_publish_center_cleanup.py tests/test_publish_center_browser.py tests/test_publish_copy_rules.py tests/test_publish_scheduler.py tests/test_publish_task_grouping.py tests/test_publish_api_flow.py -q` +- `.venv\Scripts\python.exe -m pytest -q` +- `.venv\Scripts\python.exe -m compileall -q app tests` +- `node --check app/static/js/publish-center.js` +- `.venv\Scripts\python.exe -m ruff check tests/test_publish_center_cleanup.py tests/test_publish_center_browser.py` +- `git diff --check` +- 浏览器验证 `/publish` 的桌面与 390px 窄屏按钮层级,并通过前端请求路径断言确认刷新时没有自动 AI 升级调用。 + +## 返回格式 + +报告两个动作的最终区别、AI 按钮放置、修改文件、测试与浏览器证据、分支、提交、推送和 PR;不得把本地页面验证描述为真实 AI 或平台发布验证。 diff --git a/docs/agent_tasks/2026-08-26-remove-weekly-task-chart.md b/docs/agent_tasks/2026-08-26-remove-weekly-task-chart.md new file mode 100644 index 0000000..29c7d86 --- /dev/null +++ b/docs/agent_tasks/2026-08-26-remove-weekly-task-chart.md @@ -0,0 +1,52 @@ +# 删除工作台每日任务柱状图 + +## 背景 + +工作台当前使用较大区域展示本周每日新增任务柱状图。现阶段任务创建频率较低,图表还会把未来日期显示为 0,并将单条任务按本周峰值拉满,无法提供可靠的趋势判断。 + +## 目标 + +- 删除每日新增任务柱状图,不增加替代装饰模块。 +- 保留本周任务概览、四项核心指标、周日期范围和最近任务列表。 +- 清理不再使用的每日统计、样式和动效代码。 + +## 允许修改范围 + +- Dashboard 查询上下文、首页模板、Dashboard 专用样式和动效配置。 +- Dashboard 相关测试。 +- `DEVELOPMENT_LOG.md`、`NEXT_STEPS.md`、`docs/UI_REFERENCE.md`。 + +## 禁止修改范围 + +- SQLite 表结构和真实数据。 +- 任务、切片、字幕和发布状态机。 +- AI Provider、账号、Cookie、Token、`.env` 和真实平台发布。 +- 历史 `docs/agent_tasks` 记录。 + +## 已确定实现要求 + +- `weekly_chart` 精简并改名为 `weekly_summary`,只返回本周总数和日期范围。 +- 保留应用时区、周一至下周一边界以及旧时间格式兼容逻辑。 +- 删除每日计数、星期标签、柱高百分比和当天高亮字段。 +- 删除柱状图 HTML、CSS、响应式规则、JavaScript reveal 和专用关键帧。 +- 删除图表后为统计条保留自然的底部间距。 + +## 验收标准 + +- 首页不再出现“本周每日新增任务”“Task trend”“本周累计”和七日柱图。 +- 首页继续显示四项核心指标、周日期范围和最近任务列表。 +- 本周总数和上海时区周边界统计保持正确。 +- 生产代码和测试中不再残留 `weekly_chart`、`weekly-bar` 或 `dashboard-chart`。 +- 桌面和窄屏布局无明显空洞、贴边或横向溢出。 + +## 测试命令 + +- `.venv\Scripts\python.exe -m pytest tests/test_task_query_service.py tests/test_p1_1_db_performance.py tests/test_ui_motion.py -q` +- `.venv\Scripts\python.exe -m pytest -q` +- `.venv\Scripts\python.exe -m compileall -q app tests` +- `node --check app/static/js/motion.js` +- `git diff --check` + +## 返回格式 + +报告修改文件、测试与浏览器验收结果、分支、提交、推送和 PR;不得把本地验收描述为真实平台发布验证。 diff --git a/docs/agent_tasks/2026-08-26-task-progress-live-sync.md b/docs/agent_tasks/2026-08-26-task-progress-live-sync.md new file mode 100644 index 0000000..66ae16c --- /dev/null +++ b/docs/agent_tasks/2026-08-26-task-progress-live-sync.md @@ -0,0 +1,60 @@ +# 任务详情实时进度与日志同步 + +## 背景 + +任务详情页虽然已有 `/api/tasks/{task_id}/live-status`,但前端只在全自动任务处于少数运行状态时持续轮询。手动转写、手动 AI、手动切片,以及主任务已完成但发布任务仍在排期或发送的场景,会很快停止刷新。运行日志还会被 AI 专用接口单独更新,导致日志和状态概览来自不同时间点、不同状态口径。 + +## 目标 + +- 状态概览与运行日志统一由同一份任务实时快照驱动。 +- 全自动和手动任务都能在当前页面持续看到最新阶段、总流程进度和当前操作进度。 +- 已进入发送中心的任务显示真实发布汇总,不再把仍有排期任务的流程笼统显示为 100% 已完成。 +- 页面无需整页刷新;切回浏览器标签页时立即补一次最新快照。 + +## 允许修改范围 + +- `app/services/task_service.py` +- `app/templates/task_detail.html` +- `app/static/js/app.js` +- `app/static/css/styles.css` +- 任务实时状态相关测试文件。 +- `DEVELOPMENT_LOG.md`、`NEXT_STEPS.md`、`docs/UI_REFERENCE.md`。 + +## 禁止修改范围 + +- SQLite 表结构、迁移和现有真实任务数据。 +- AI、转写、切片和发布状态机的写入逻辑。 +- Provider、账号、Cookie、Token、`.env` 和认证配置。 +- 真实 AI 调用、真实切片、真实抖音 / B站投稿、部署和 PR 合并。 + +## 已确定实现要求 + +- `/live-status` 一次返回主任务状态、活动 Workflow Job、转写细分进度、最新运行日志和发布任务汇总。 +- 主进度必须明确标为“总流程进度”;另行显示当前操作名称、操作百分比和最新说明,避免把步骤进度误认为总流程进度。 +- 手动任务不能再被 `auto_mode` 条件排除;有活动任务或待执行发布排期时,页面可见期间每 3 秒刷新一次,页面隐藏时暂停,回到前台立即刷新。 +- 运行日志只由 `/live-status` 快照更新,AI 专用轮询只负责 AI 区域自身进度。 +- 发布汇总以每个切片最新的 `publish_jobs` 记录为准;区分已发布/已导出、排期/等待/发送中、失败/需复核和取消。 +- 有发布任务尚未完成时,状态概览显示托管发布进度和 `已完成/总数`,最后一个时间线节点保持当前或警告状态;全部成功后才显示发布完成。 +- 所有新增接口逻辑保持只读,不修改真实任务、作业或发布记录。 + +## 验收标准 + +- 全自动和手动任务在运行或等待托管发布期间都持续发起 3 秒一次的局部状态刷新,不依赖 F5 或切换页面。 +- 状态概览与运行日志使用同一次响应中的数据,页面不存在第二条日志刷新数据源。 +- 转写或后台作业运行时显示当前操作百分比和说明;总流程百分比与当前操作百分比有清晰标签。 +- 存在 9 条已发布、3 条待排期的任务显示“9/12”托管发布进度,而不是笼统的“已完成 100%”。 +- 发布失败或需复核时使用警告语义,不冒充发布成功。 +- 既有全自动状态、失败重试和页面局部刷新测试继续通过。 + +## 测试命令 + +- `.venv\Scripts\python.exe -m pytest tests/test_auto_pipeline.py tests/test_pipeline_state_stability.py -q` +- `.venv\Scripts\python.exe -m pytest -q` +- `.venv\Scripts\python.exe -m compileall -q app tests` +- `node --check app/static/js/app.js` +- `git diff --check` +- 浏览器刷新 `http://127.0.0.1:8001/tasks/3210d91ee1fb`,核对发布汇总、轮询更新时间、日志同步、桌面与窄屏布局和控制台。 + +## 返回格式 + +报告修改文件、实时状态口径、测试结果、浏览器验收证据、分支、提交、推送和 PR;不得把只读页面验证描述为真实 AI、切片或平台发布验证。 diff --git a/docs/agent_tasks/2026-08-26-ui-motion-system.md b/docs/agent_tasks/2026-08-26-ui-motion-system.md new file mode 100644 index 0000000..1b3bbf4 --- /dev/null +++ b/docs/agent_tasks/2026-08-26-ui-motion-system.md @@ -0,0 +1,81 @@ +# 全站页面适配动效系统 + +## 背景 + +当前页面只有少量按钮 hover、进度条宽度变化和发送中的旋转图标,页面进入、标签切换、抽屉打开、列表新增和实时状态更新缺少统一反馈。用户希望参考 GitHub 上成熟的前端动效项目,为每个页面补充适配动效;没有明确交互价值的区域不强行添加。 + +调研参考: + +- Motion:优先使用浏览器原生动画能力,并尽量只动画 `opacity`、`transform` 等合成属性。 +- FormKit AutoAnimate:监听 DOM 真实变化,为新增内容提供短时反馈,不持续制造视觉噪声。 +- Animate.css:统一时长和延迟变量,并对 `prefers-reduced-motion` 提供完整降级。 +- AOS:页面滚动进入视口时只播放一次,避免往返滚动反复晃动。 + +本项目是无前端构建步骤的 Windows 本地后台,因此不引入 npm/CDN 依赖,只实现小型原生 CSS/JavaScript 动效层。 + +## 目标 + +- 为工作台、任务列表、新建任务、任务详情、完整转写、片段总览、单任务片段审核、字幕总览/工作台、发送中心和系统状态配置适合自身结构的动效。 +- 仅工作台周统计与柱图在首次进入时采用一次性渐显/增长;其余页面没有合适入场语义时保持静态,通过真实状态和交互变化提供反馈。 +- 实时进度、动态新增行、标签页、弹窗和抽屉的变化有明确反馈。 +- 系统启用“减少动态效果”时停用非必要动画和过渡。 +- 无 JavaScript、脚本失败或动画 API 不可用时,所有内容和操作仍可见、可用。 + +## 允许修改范围 + +- `app/templates/base.html` +- `app/static/css/styles.css` +- `app/static/js/motion.js`(新增) +- `app/static/js/app.js`(仅 reduced-motion 滚动降级) +- `app/static/js/publish-center.js`(仅 reduced-motion 滚动降级) +- `tests/test_ui_motion.py`(新增) +- 与本任务直接相关的现有前端测试(仅确有需要时) +- `DEVELOPMENT_LOG.md` +- `NEXT_STEPS.md` +- `docs/UI_REFERENCE.md` +- 本任务文件 + +## 禁止修改范围 + +- FastAPI 路由、业务服务、SQLite Schema、迁移和活动数据。 +- AI Prompt、Provider、认证、计费或模型配置。 +- 转写、切片、字幕烧录、排期、Scheduler、Publisher 和平台发送状态机。 +- 真实 AI、封面生成、排期、Windows Worker 发布或平台投稿。 +- React/Vue、npm 打包流程、远程 CDN 和第三方运行时依赖。 + +## 已确定实现要求 + +- 使用路由级页面配置映射不同选择器,不给所有卡片套同一种动画。 +- 初次进入只播放一次;列表最多采用短延迟错峰,不能让内容等待过久。 +- 工作台周统计和柱图采用一次性反馈;新建任务只反馈条件设置区和文件选择;任务详情只反馈真实变化的状态、进度和操作数据;片段审核、字幕工作台和发送中心只反馈进度、标签、弹层、抽屉及用户明确触发的新增内容。 +- 任务列表、完整转写、片段总览、字幕总览和系统状态没有合适的页面入场语义,保持静态布局,仅使用全局按钮、链接、行状态、焦点和展开收起过渡,不逐行制造动画。 +- 不对视频播放、字幕波形、滚动日志、输入中的文本和持续轮询本身添加循环动效。 +- 动态 DOM 动效限制在排期预览、AI 分析结果和 AI 历史等用户触发的新增项,不监听或动画轮询重建的执行记录、日历、字幕虚拟列表及整个页面的每次字符变化。 +- `prefers-reduced-motion: reduce` 下动画时长和过渡时长压缩到近零,关闭非必要旋转;脚本也应跳过初始/动态动效注册。 +- 现有 `scrollIntoView({ behavior: "smooth" })` 在 reduced-motion 下改用即时滚动,不能只关闭 CSS 动画。 +- 动画只改变呈现,不改 DOM 语义、业务请求、按钮可用性和已有选择器。 + +## 验收标准 + +- 所有正式页面都能得到路由匹配的动效配置,未知路由保持静态可用。 +- 动画脚本是本地静态资源,不访问外网,不引入新依赖。 +- 工作台周统计与柱图首次进入时短时播放一次;其他静态页面不为了凑数量添加整页入场动画。 +- 工作台柱状图、详情进度、弹窗/抽屉、发送中心标签切换和动态新增行有平滑反馈。 +- 长文本、日志、视频和字幕编辑高频区域不发生反复闪烁或位移。 +- 390px 窄屏无横向溢出;动效不改变最终布局尺寸。 +- 减少动态效果模式下内容立即可见,动画和过渡被禁用。 +- 现有全量测试通过,JavaScript 语法、Ruff、Compileall 和 `git diff --check` 通过。 + +## 测试命令 + +- `.venv\Scripts\python.exe -m pytest tests/test_ui_motion.py -q` +- `.venv\Scripts\python.exe -m pytest -q` +- `.venv\Scripts\python.exe -m compileall -q app tests` +- `node --check app/static/js/motion.js` +- `.venv\Scripts\python.exe -m ruff check tests/test_ui_motion.py` +- `git diff --check` +- 浏览器逐页检查桌面端与 390px 窄屏;检查 `data-motion-page`、初始动画、动态面板动画和 reduced-motion CSS 降级,不点击真实 AI、切片、封面、排期或发送按钮。 + +## 返回格式 + +报告 GitHub 调研取舍、各页面动效映射、明确未添加动效的区域、修改文件、测试与浏览器证据、分支、提交、推送和 PR;不得把本地前端验收描述为真实业务流程或平台发布验证。 diff --git a/prompts/default_ai_prompt_preset_001.txt b/prompts/default_ai_prompt_preset_001.txt index 923f923..fe398ca 100644 --- a/prompts/default_ai_prompt_preset_001.txt +++ b/prompts/default_ai_prompt_preset_001.txt @@ -1,223 +1,62 @@ -你是一个专业的直播切片内容分析专家,擅长从长直播转写文本中识别适合短视频传播的高价值片段。 - -你的任务是: -根据我提供的带时间戳直播转写文本,筛选出适合剪辑成短视频的候选片段,并严格按照指定 JSON 格式输出。 - -你必须遵守以下原则: - -1. 只能基于我提供的转写文本进行分析,不能虚构直播中没有出现的内容。 -2. 必须优先选择内容完整、观点明确、表达集中、适合单独传播的片段。 -3. 不要为了凑数量强行推荐片段。如果没有合适片段,可以返回空数组。 -4. 每个候选片段必须包含明确的开始时间和结束时间。 -5. 开始时间和结束时间必须尽量来自转写文本中已有的时间戳附近,不允许凭空编造。 -6. 单个片段时长不得超过用户设定的最大时长。 -7. 候选片段要适合短视频传播,而不是简单截取任意连续文本。 -8. 如果某段内容需要上下文才能理解,不适合单独传播,应降低推荐优先级。 -9. 如果某段内容存在明显口误、重复、寒暄、无信息量闲聊,应避免推荐。 -10. 输出必须是严格合法 JSON,不要输出 Markdown,不要输出解释,不要输出代码块,不要包含多余文字。 - ---- - -【用户配置】 - -单条切片最大时长: -{{MAX_CLIP_DURATION}} - -期望候选片段数量: -{{TARGET_CLIP_COUNT}} - -AI 偏好风格: +# Role +你是《康熙来了》类棚内访谈的抖音短视频选片总编。 + +你的任务不是总结节目,也不是寻找“经过复杂剪辑可能会好看”的素材,而是筛选可以从原片连续截取、基本不做二次剪辑就能直接发布的高留存片段。 + +# Core Goal +第一目标:观众看到片段原始开头后的前3秒,能立即理解现场正在发生什么,并产生继续观看的欲望。 +第二目标:片段中存在明确的笑点、尴尬、关系张力、意外回答、态度反差或评论冲动。 +第三目标:在最短必要长度内完成“钩子→发展→爆点→反应→收尾”。 + +# Hard Selection Gates +候选片段必须同时满足: + +1. start_time 对应的原始开头本身就有钩子,例如犀利问题、异常回答、明显尴尬、关系冲突、态度反差或强烈现场反应。 +2. 不依赖倒叙、标题党、封面大字、补充背景字幕或重新排列镜头才能成立。 +3. 开始后15秒内必须出现第一次有效刺激;不能先进行长时间人物介绍或普通铺垫。 +4. 全段只围绕一个核心话题或冲突,不混入第二个无关话题。 +5. 爆点后必须保留主持人追问、嘉宾解释、补刀、笑声、沉默或态度变化,形成完整反应闭环。 +6. 必须存在明确评论动机:观众会想站队、评价人物、分享类似经历、讨论关系或指出反差。 +7. 任一条件不满足就淘汰,不要因为明星知名度、八卦标签或目标数量而保留。 + +# Priority +按以下顺序选择: + +1. 关系张力:前任、婚恋、暧昧、好友互损、社交地位、明星关系,但必须有现场互动和反应。 +2. 尴尬反转:被问住、回答前后矛盾、慌张解释、主持人连续追问或补刀。 +3. 大众共鸣:恋爱、婚姻、年龄、面子、自尊、社交尴尬,但必须有具体人物和具体事件。 +4. 连续笑点:一个核心情境内出现两次以上递进笑点,并有现场反应。 +5. 意外观点:答案明显超出常规预期,能自然形成争议或评论。 + +# Duration +以60–90秒为主。 +只有情绪持续递进、没有明显空转时才保留到90–120秒。 +最高不得超过150秒。 +从第一个可独立理解的犀利问题、异常回答或明显反应开始;在第一次完整反应闭环后尽快结束,不保留多余尾巴。 + +# Reject +坚决淘汰: + +- 需要剪辑师“救活”的普通内容。 +- 原始开头平淡,只能把中段爆点倒叙到开头的内容。 +- 只有明星名字、前任、婚恋或身体话题,但没有现场反应的内容。 +- 只有一句金句、笑声或表情,没有完整互动回合的内容。 +- 铺垫超过15秒才出现有效刺激的内容。 +- 话题看起来刺激,但嘉宾只是平铺直叙回答的内容。 +- 同一故事的重复版本;只保留开头最强、反应最完整的一条。 +- 依赖观众了解整期节目才能看懂的内容。 + +# Score Calibration +评分必须严格使用0–100: + +- hook_score 只评价原始 start_time 后前3秒,不能评价标题或假设中的倒叙剪辑。 +- humor_score 评价真实笑点和反应闭环,不因话题敏感而加分。 +- interaction_reaction_score 评价追问、补刀、语塞、解释、沉默和现场反应。 +- completeness_score 评价最短必要闭环,不奖励冗长。 +- novelty_score 评价意外程度、关系张力和评论分歧。 +- title_score 只有原文中确实存在可支撑标题的信息时才能给高分。 + +普通可看内容不得高于65分;有潜力但不能直接发布为66–77分;只有无需复杂剪辑即可直接发布的强片段才能达到78分以上。宁可返回空结果,也不要虚高或凑数。 + +# User Preference {{AI_PREFERENCE}} - -可选风格说明: -- 强观点:优先选择观点鲜明、有判断、有结论的内容 -- 高情绪:优先选择情绪强、冲突强、表达有感染力的内容 -- 知识密度高:优先选择方法论、经验总结、步骤拆解、专业解释 -- 爆点型:优先选择有争议、有反差、有传播钩子的内容 -- 口播型:优先选择适合直接剪成口播短视频的完整表达 -- 通用:综合考虑内容价值、完整度和传播性 - ---- - -【直播转写文本】 - -{{TRANSCRIPT_TEXT}} - ---- - -【分析要求】 - -请从转写文本中筛选候选切片片段。 - -每个候选片段需要判断: - -1. 这个片段讲了什么? -2. 是否适合从长直播中独立剪出来? -3. 是否有明确主题? -4. 是否有开头、展开和结论? -5. 是否适合短视频平台传播? -6. 是否符合用户设置的最大切片时长? -7. 是否有足够的信息密度或情绪价值? -8. 是否需要避开无意义寒暄、重复话术、无效互动? - ---- - -【优先推荐的内容类型】 - -以下内容优先推荐: - -1. 明确观点: - - 主播表达了一个清晰判断 - - 有“为什么”“怎么做”“关键是”等逻辑 - -2. 方法总结: - - 包含步骤、技巧、经验、避坑、复盘 - - 适合观众收藏或转发 - -3. 冲突或反差: - - 包含常见误区、反常识观点、争议判断 - - 适合吸引点击 - -4. 案例拆解: - - 主播讲了具体案例 - - 有过程、有结论、有启发 - -5. 高情绪表达: - - 表达有感染力 - - 有惊讶、吐槽、强烈赞同或反对 - -6. 问答片段: - - 问题清晰 - - 回答完整 - - 不依赖太多上下文 - -7. 金句片段: - - 句子简洁有力 - - 适合做标题或字幕重点 - ---- - -【不推荐的内容类型】 - -以下内容不要推荐,除非其中包含明显高价值内容: - -1. 开场寒暄 -2. 纯打招呼 -3. 感谢关注、点赞、刷礼物 -4. 无意义闲聊 -5. 直播间技术调试 -6. 重复介绍商品但没有新的信息 -7. 只有情绪没有内容 -8. 需要大量上下文才能理解的片段 -9. 时间点不清晰的片段 -10. 明显不完整的半句话、断裂表达 - ---- - -【时间选择规则】 - -1. start_time 应该选择在该片段核心内容开始前 10-25 秒左右,尽量包含必要铺垫、问题、冲突起因或观点引入。 -2. end_time 应该选择在该片段核心内容自然结束后 10-25 秒左右,尽量包含结论、反应、追问、笑点落地或自然收尾。 -3. 不要只截取爆点、金句或结论本身;候选片段应该像一条可以直接发布的完整短视频。 -4. 如果转写文本只有分钟级时间戳,则允许使用最接近的分钟级时间。 -5. 不要把多个主题强行合并成一个长片段。 -6. 如果一个主题超过最大切片时长,应选择其中最完整、最有传播价值的一小段,并优先保留理解所需的前后文。 -7. 如果两个候选片段时间重叠,请优先保留价值更高、上下文更完整的一个。 - ---- - -【传播价值判断】 - -spread_value 只能从以下值中选择: - -- 高 -- 中 -- 低 - -判断标准: - -高: -- 主题明确 -- 信息密度高 -- 观点强或情绪强 -- 适合独立传播 -- 适合做标题吸引点击 - -中: -- 内容有一定价值 -- 可以剪,但爆点不强 -- 需要轻微剪辑包装 - -低: -- 内容普通 -- 信息量一般 -- 单独传播吸引力弱 - ---- - -【置信度规则】 - -confidence_score 使用 0 到 1 之间的小数。 - -参考标准: - -0.90 - 1.00: -非常适合切片,主题完整,传播价值强。 - -0.75 - 0.89: -适合切片,内容较完整,有一定传播价值。 - -0.60 - 0.74: -可以考虑,但需要人工判断或二次剪辑。 - -0.60 以下: -不建议默认选中。 - -selected_by_default 规则: -- confidence_score >= 0.75 且 spread_value 为“高”或“中”时,可设为 true -- 其他情况设为 false - ---- - -【输出格式】 - -你必须严格输出以下 JSON 结构: - -{ - "analysis_summary": "对整场直播内容的整体判断,说明主要内容方向、可切片价值、适合剪辑的内容类型。控制在150字以内。", - "clips": [ - { - "clip_id": "clip_001", - "title": "适合短视频使用的标题,控制在30字以内", - "start_time": "00:12:10", - "end_time": "00:14:40", - "duration_seconds": 150, - "cover_time_seconds": 15, - "summary": "概括这个片段讲了什么,控制在80字以内。", - "highlight_reason": "说明为什么这个片段值得剪,控制在100字以内。", - "spread_value": "高", - "suggested_editing": "给出剪辑建议,例如保留哪句开头、哪里加字幕、是否需要裁掉前后寒暄,控制在100字以内。", - "confidence_score": 0.92, - "selected_by_default": true - } - ] -} - ---- - -【严格输出要求】 - -1. 只输出 JSON。 -2. 不要输出 Markdown。 -3. 不要使用 ```json 代码块。 -4. 不要输出任何解释性文字。 -5. 不要包含 或任何 XML/HTML 标签。 -6. clips 数量不要超过用户设置的期望候选片段数量。 -7. 如果没有合适片段,请输出: - -{ - "analysis_summary": "本场直播内容未发现适合独立切片传播的高价值片段。", - "clips": [] -} - -现在请开始分析。 diff --git a/requirements.in b/requirements.in index c84c629..a50c48c 100644 --- a/requirements.in +++ b/requirements.in @@ -10,3 +10,4 @@ tzdata>=2026.1 playwright>=1.58,<1.63 aiofiles>=25.1,<26 faster-whisper>=1.2,<1.3 +pysubs2==1.9.0 diff --git a/requirements.txt b/requirements.txt index d44dc29..0b340de 100644 --- a/requirements.txt +++ b/requirements.txt @@ -13,3 +13,4 @@ tzdata==2026.3 playwright==1.62.0 aiofiles==25.1.0 faster-whisper==1.2.1 +pysubs2==1.9.0 diff --git a/scripts/publish_host_worker.py b/scripts/publish_host_worker.py index a3d8475..4b0d621 100644 --- a/scripts/publish_host_worker.py +++ b/scripts/publish_host_worker.py @@ -8,16 +8,20 @@ import argparse import json +import logging import os import subprocess import sys import threading +import time +import weakref from pathlib import Path from typing import Any +from uuid import uuid4 import uvicorn from fastapi import BackgroundTasks, Depends, FastAPI, Header, HTTPException -from pydantic import BaseModel, Field +from pydantic import BaseModel, Field, ValidationInfo, field_validator PROJECT_ROOT = Path(__file__).resolve().parents[1] if str(PROJECT_ROOT) not in sys.path: @@ -31,15 +35,25 @@ PublishOutcome, PublishResult, PublishValidationError, + sanitize_provider_response, ) from app.services.publishers.browser_runtime import BrowserRuntime # noqa: E402 from app.services.publishers.registry import get_platform_publisher # noqa: E402 +from app.services.publishers.worker_client import validate_worker_identifier # noqa: E402 + + +logger = logging.getLogger(__name__) class AccountRequest(BaseModel): platform: str = Field(pattern="^(douyin|bilibili)$") account_id: str = Field(min_length=1, max_length=120) + @field_validator("account_id") + @classmethod + def validate_account_id(cls, value: str) -> str: + return validate_worker_identifier(value, "account_id", max_length=120) + class PublishRequest(BaseModel): job_id: str = Field(min_length=1, max_length=160) @@ -61,18 +75,161 @@ class PublishRequest(BaseModel): bilibili_source: str = "" publisher: str = "local_browser" + @field_validator("job_id", "execution_id", "account_id") + @classmethod + def validate_identifiers(cls, value: str, info: ValidationInfo) -> str: + max_length = 120 if info.field_name == "account_id" else 160 + return validate_worker_identifier(value, str(info.field_name), max_length=max_length) + class OpenCliRunRequest(BaseModel): command: list[str] timeout: int = Field(default=600, ge=1, le=1800) +class _LockLease(str): + """兼容原 token 字符串,同时持有由操作系统管理的文件锁句柄。""" + + def __new__(cls, value: str, handle: Any): + lease = super().__new__(cls, value) + lease.handle = handle + return lease + + +def _try_create_lock_file(path: Path) -> _LockLease | None: + path.parent.mkdir(parents=True, exist_ok=True) + token = f"{os.getpid()}:{uuid4().hex}" + try: + path.touch(exist_ok=True) + handle = path.open("r+b") + except OSError as exc: + raise PublishValidationError( + f"无法打开 Worker 独占锁:{path.name}", "worker_lock_unavailable" + ) from exc + try: + if os.name == "nt": + import msvcrt + + handle.seek(0) + msvcrt.locking(handle.fileno(), msvcrt.LK_NBLCK, 1) + else: + import fcntl + + fcntl.flock(handle.fileno(), fcntl.LOCK_EX | fcntl.LOCK_NB) + except OSError: + handle.close() + return None + try: + handle.seek(0) + handle.write(token.encode("utf-8")) + handle.truncate() + handle.flush() + os.fsync(handle.fileno()) + except OSError as exc: + lease = _LockLease(token, handle) + _release_lock_file(path, lease) + raise PublishValidationError( + f"无法写入 Worker 独占锁:{path.name}", "worker_lock_unavailable" + ) from exc + return _LockLease(token, handle) + + +def _lock_file_is_active(path: Path) -> bool: + if not path.exists(): + return False + try: + lease = _try_create_lock_file(path) + if lease is None: + return True + _release_lock_file(path, lease) + return False + except PublishValidationError: + logger.exception("检查 Worker 独占锁失败:%s", path.name) + return True + + +def _release_lock_file(path: Path, token: _LockLease | str | None) -> None: + if not token: + return + if not isinstance(token, _LockLease): + logger.error("拒绝在缺少操作系统锁句柄时删除锁文件:%s", path.name) + return + handle = token.handle + try: + if os.name == "nt": + import msvcrt + + handle.seek(0) + msvcrt.locking(handle.fileno(), msvcrt.LK_UNLCK, 1) + else: + import fcntl + + fcntl.flock(handle.fileno(), fcntl.LOCK_UN) + except OSError: + logger.exception("释放 Worker 操作系统锁失败:%s", path.name) + finally: + handle.close() + + +class _ReentrantExecutionLock: + """进程内可重入 + 跨进程 O_EXCL,崩溃残留时 fail closed。""" + + def __init__(self, path: Path) -> None: + self.path = path + self._thread_lock = threading.RLock() + self._local = threading.local() + + @property + def acquired(self) -> bool: + return bool(getattr(self._local, "acquired", False)) + + def __enter__(self) -> "_ReentrantExecutionLock": + self._thread_lock.acquire() + depth = int(getattr(self._local, "depth", 0)) + if depth == 0: + try: + self._local.token = _try_create_lock_file(self.path) + self._local.acquired = bool(self._local.token) + except Exception: + self._thread_lock.release() + raise + self._local.depth = depth + 1 + return self + + def __exit__(self, *_args: object) -> None: + depth = int(getattr(self._local, "depth", 1)) - 1 + self._local.depth = depth + if depth == 0: + _release_lock_file(self.path, getattr(self._local, "token", None)) + self._local.token = None + self._local.acquired = False + self._thread_lock.release() + + +_EXECUTION_LOCKS: weakref.WeakValueDictionary[str, _ReentrantExecutionLock] = ( + weakref.WeakValueDictionary() +) +_EXECUTION_LOCKS_GUARD = threading.Lock() + + +def _execution_lock(execution_id: str) -> _ReentrantExecutionLock: + state_root = Path(settings.publish_worker_state_dir).resolve() + key = f"{state_root}:{execution_id}" + with _EXECUTION_LOCKS_GUARD: + return _EXECUTION_LOCKS.setdefault( + key, + _ReentrantExecutionLock(state_root / "locks" / "executions" / f"{execution_id}.lock"), + ) + + class ExecutionJournal: def __init__(self, execution_id: str) -> None: - self.execution_id = execution_id + self.execution_id = validate_worker_identifier( + execution_id, "execution_id", max_length=160 + ) self.root = Path(settings.publish_worker_state_dir) / "executions" - self.path = self.root / f"{execution_id}.json" - self._lock = threading.Lock() + self.path = self.root / f"{self.execution_id}.json" + self.lock = _execution_lock(self.execution_id) self.root.mkdir(parents=True, exist_ok=True) def read(self) -> dict[str, Any]: @@ -81,32 +238,148 @@ def read(self) -> dict[str, Any]: try: parsed = json.loads(self.path.read_text(encoding="utf-8")) except (OSError, json.JSONDecodeError): - return {"execution_id": self.execution_id, "phase": "unknown"} - return parsed if isinstance(parsed, dict) else {"execution_id": self.execution_id, "phase": "unknown"} - - def update(self, phase: str, details: dict[str, Any] | None = None) -> None: - with self._lock: + return {"execution_id": self.execution_id, "phase": "corrupt", "corrupt": True} + if not isinstance(parsed, dict): + return {"execution_id": self.execution_id, "phase": "corrupt", "corrupt": True} + if parsed.get("execution_id") != self.execution_id or not isinstance(parsed.get("phase"), str): + return {"execution_id": self.execution_id, "phase": "corrupt", "corrupt": True} + identity = parsed.get("identity") + if identity is not None: + expected_keys = {"job_id", "platform", "account_id"} + if ( + not isinstance(identity, dict) + or set(identity) != expected_keys + or not all(isinstance(identity.get(key), str) and identity.get(key) for key in expected_keys) + ): + return {"execution_id": self.execution_id, "phase": "corrupt", "corrupt": True} + return parsed + + def update( + self, + phase: str, + details: dict[str, Any] | None = None, + *, + identity: dict[str, str] | None = None, + ) -> None: + with self.lock: + if not self.lock.acquired: + raise PublishValidationError( + "相同 execution_id 正在另一 Worker 进程中执行", + "execution_in_progress", + ) current = self.read() + if current.get("corrupt"): + raise PublishValidationError( + "执行日志已损坏,为避免重复投稿已停止执行", + "execution_journal_corrupt", + ) current.update({ "execution_id": self.execution_id, "phase": phase, "updated_at": utc_now_iso(), }) + if phase == "upload_started" or current.get("upload_started") is True: + current["upload_started"] = True + if identity: + current["identity"] = dict(identity) if details: - current["details"] = details - temporary = self.path.with_suffix(".tmp") - temporary.write_text(json.dumps(current, ensure_ascii=False, indent=2), encoding="utf-8") - temporary.replace(self.path) + current["details"] = sanitize_provider_response(details) + temporary = self.path.with_name( + f".{self.path.name}.{os.getpid()}.{threading.get_ident()}.tmp" + ) + try: + temporary.write_text(json.dumps(current, ensure_ascii=False, indent=2), encoding="utf-8") + temporary.replace(self.path) + finally: + temporary.unlink(missing_ok=True) + + +def _prior_job_execution_requires_review( + job_id: str, + current_execution_id: str, +) -> tuple[str, str] | None: + """查找同一发布任务已进入不可安全重试阶段的旧 execution。""" + + safe_retry_phases = { + "unknown", + "received", + "browser_opening", + "browser_opened", + "rejected", + "failed", + } + root = Path(settings.publish_worker_state_dir) / "executions" + if not root.exists(): + return None + for path in root.glob("*.json"): + if path.stem == current_execution_id: + continue + try: + state = ExecutionJournal(path.stem).read() + except PublishValidationError: + continue + identity = state.get("identity") + if not isinstance(identity, dict) or identity.get("job_id") != job_id: + continue + phase = str(state.get("phase") or "unknown") + if state.get("upload_started") is True or phase not in safe_retry_phases: + return path.stem, phase + return None + + +class _AccountOperationLock: + def __init__(self, path: Path) -> None: + self.path = path + self._thread_lock = threading.Lock() + self._token: str | None = None + + def acquire(self, blocking: bool = True) -> bool: + if not self._thread_lock.acquire(blocking=blocking): + return False + try: + self._token = _try_create_lock_file(self.path) + except Exception: + self._thread_lock.release() + raise + if not self._token: + self._thread_lock.release() + return False + return True + def release(self) -> None: + _release_lock_file(self.path, self._token) + self._token = None + self._thread_lock.release() -_ACCOUNT_LOCKS: dict[str, threading.Lock] = {} + +_ACCOUNT_LOCKS: weakref.WeakValueDictionary[str, _AccountOperationLock] = ( + weakref.WeakValueDictionary() +) +_JOB_LOCKS: weakref.WeakValueDictionary[str, _AccountOperationLock] = weakref.WeakValueDictionary() _LOCKS_GUARD = threading.Lock() -def _account_lock(platform: str, account_id: str) -> threading.Lock: - key = f"{platform}:{account_id}" +def _account_lock(platform: str, account_id: str) -> _AccountOperationLock: + state_root = Path(settings.publish_worker_state_dir).resolve() + key = f"{state_root}:{platform}:{account_id}" with _LOCKS_GUARD: - return _ACCOUNT_LOCKS.setdefault(key, threading.Lock()) + return _ACCOUNT_LOCKS.setdefault( + key, + _AccountOperationLock( + state_root / "locks" / "accounts" / f"{platform}--{account_id}.lock" + ), + ) + + +def _job_lock(job_id: str) -> _AccountOperationLock: + safe_job_id = validate_worker_identifier(job_id, "job_id", max_length=160) + state_root = Path(settings.publish_worker_state_dir).resolve() + key = f"{state_root}:{safe_job_id}" + with _LOCKS_GUARD: + return _JOB_LOCKS.setdefault( + key, + _AccountOperationLock(state_root / "locks" / "jobs" / f"{safe_job_id}.lock"), + ) def _allowed_roots() -> list[Path]: @@ -228,10 +501,7 @@ def check_account(payload: AccountRequest) -> dict[str, Any]: finally: lock.release() - def login_background(payload: AccountRequest) -> None: - lock = _account_lock(payload.platform, payload.account_id) - if not lock.acquire(blocking=False): - return + def login_background(payload: AccountRequest, lock: _AccountOperationLock) -> None: try: runtime = BrowserRuntime(payload.platform, payload.account_id) publisher = get_platform_publisher( @@ -241,108 +511,298 @@ def login_background(payload: AccountRequest) -> None: except Exception: # Worker 只负责宿主浏览器操作,不直接写 Docker 挂载的 SQLite。 # 登录结果由 FastAPI 后续通过账号检测接口统一落库。 - return + logger.exception( + "打开平台登录窗口失败:platform=%s account_id=%s", + payload.platform, + payload.account_id, + ) finally: lock.release() @worker.post("/v1/accounts/login", dependencies=[Depends(require_token)], status_code=202) def login_account(payload: AccountRequest, background_tasks: BackgroundTasks) -> dict[str, Any]: - if _account_lock(payload.platform, payload.account_id).locked(): + lock = _account_lock(payload.platform, payload.account_id) + if not lock.acquire(blocking=False): raise HTTPException(status_code=409, detail="该账号已有浏览器窗口正在运行") - background_tasks.add_task(login_background, payload) + background_tasks.add_task(login_background, payload, lock) return {"status": "started", "message": "已打开独立 Chrome,请在窗口中完成平台登录"} @worker.post("/v1/accounts/open-center", dependencies=[Depends(require_token)], status_code=202) def open_center(payload: AccountRequest, background_tasks: BackgroundTasks) -> dict[str, Any]: - if _account_lock(payload.platform, payload.account_id).locked(): + lock = _account_lock(payload.platform, payload.account_id) + if not lock.acquire(blocking=False): raise HTTPException(status_code=409, detail="该账号已有浏览器窗口正在运行") - background_tasks.add_task(login_background, payload) + background_tasks.add_task(login_background, payload, lock) return {"status": "started", "message": "已打开平台创作者中心"} - @worker.post("/v1/publish", dependencies=[Depends(require_token)]) - def publish(payload: PublishRequest) -> dict[str, Any]: + def publish_with_job_lock(payload: PublishRequest) -> dict[str, Any]: journal = ExecutionJournal(payload.execution_id) - journal.update("received", {"job_id": payload.job_id, "platform": payload.platform}) - lock = _account_lock(payload.platform, payload.account_id) - if not lock.acquire(blocking=False): - result = PublishResult( - outcome=PublishOutcome.FAILED, - message="同一账号已有发布任务正在执行", - error_code="account_busy", - ) - journal.update("rejected", result.as_dict()) - return result.as_dict() - try: - values = payload.model_dump() - values["video_path"] = _resolve_media_path(values["video_path"], required=True) - values["cover_file_path"] = _resolve_media_path(values["cover_file_path"], required=False) - def update_phase(phase: str, details: dict[str, Any] | None = None) -> None: - journal.update(phase, details) + identity = { + "job_id": payload.job_id, + "platform": payload.platform, + "account_id": payload.account_id, + } + terminal_phases = {"confirmed_success", "exported", "failed", "manual_review"} + terminal_outcomes = { + "confirmed_success": PublishOutcome.PUBLISHED.value, + "exported": PublishOutcome.EXPORTED.value, + "failed": PublishOutcome.FAILED.value, + "manual_review": PublishOutcome.NEED_REVIEW.value, + } + safe_resume_phases = {"unknown", "received", "browser_opening", "browser_opened", "rejected"} - runtime = BrowserRuntime( - payload.platform, - payload.account_id, - phase_callback=update_phase, - ) - publisher = get_platform_publisher( - payload.platform, - runtime=runtime, - account_id=payload.account_id, - ) - result = publisher.publish(values) - journal.update("confirmed_success" if result.outcome == PublishOutcome.PUBLISHED else result.outcome.value.lower(), result.as_dict()) - return result.as_dict() - except PublishNeedsReview as exc: + def saved_terminal_result(phase: str, saved: Any) -> dict[str, Any] | None: + required_fields = { + "outcome", "message", "remote_video_id", "platform_url", "published_at", + "provider_response", "error_code", "needs_manual_review", + } + if ( + phase not in terminal_outcomes + or not isinstance(saved, dict) + or not required_fields.issubset(saved) + or saved.get("outcome") != terminal_outcomes[phase] + or not isinstance(saved.get("provider_response"), dict) + or not isinstance(saved.get("needs_manual_review"), bool) + or (phase == "confirmed_success" and not str(saved.get("published_at") or "")) + or (phase == "manual_review" and saved.get("needs_manual_review") is not True) + or (phase != "manual_review" and saved.get("needs_manual_review") is not False) + ): + return None + return sanitize_provider_response(saved) + + with journal.lock: current = journal.read() - diagnostics = current.get("details") if isinstance(current.get("details"), dict) else {} - result = PublishResult( - outcome=PublishOutcome.NEED_REVIEW, - message=exc.message, - error_code=exc.error_code, - needs_manual_review=True, - provider_response={"diagnostics": diagnostics}, + phase = str(current.get("phase") or "unknown") + existing_identity = current.get("identity") + if current.get("corrupt"): + return PublishResult( + outcome=PublishOutcome.NEED_REVIEW, + message="Worker 执行日志已损坏,为避免重复投稿已停止执行", + error_code="execution_journal_corrupt", + needs_manual_review=True, + ).as_dict() + if isinstance(existing_identity, dict) and existing_identity != identity: + return PublishResult( + outcome=PublishOutcome.NEED_REVIEW, + message="execution_id 已属于另一发布任务,已拒绝重复执行", + error_code="execution_identity_conflict", + needs_manual_review=True, + ).as_dict() + if journal.path.exists() and not isinstance(existing_identity, dict): + return PublishResult( + outcome=PublishOutcome.NEED_REVIEW, + message="旧版 Worker 日志缺少执行身份,为避免重复投稿已停止执行", + error_code="execution_identity_missing", + needs_manual_review=True, + ).as_dict() + if not journal.lock.acquired: + saved = saved_terminal_result(phase, current.get("details")) + if saved is not None: + return saved + return PublishResult( + outcome=PublishOutcome.NEED_REVIEW, + message="相同 execution_id 正在另一 Worker 进程执行,已拒绝重复投稿", + error_code="execution_in_progress", + needs_manual_review=True, + ).as_dict() + if phase in terminal_phases: + saved = saved_terminal_result(phase, current.get("details")) + if saved is not None: + return saved + return PublishResult( + outcome=PublishOutcome.NEED_REVIEW, + message="Worker 终态日志与结果不一致,为避免重复投稿已停止执行", + error_code="execution_terminal_result_inconsistent", + needs_manual_review=True, + ).as_dict() + prior_conflict = _prior_job_execution_requires_review( + payload.job_id, + payload.execution_id, ) - journal.update("manual_review", result.as_dict()) - return result.as_dict() - except PublishValidationError as exc: - result = PublishResult( - outcome=PublishOutcome.FAILED, - message=exc.message, - error_code=exc.error_code, - ) - journal.update("failed", result.as_dict()) - return result.as_dict() - except PublishError as exc: - phase = str(journal.read().get("phase") or "unknown") - current = journal.read() - diagnostics = current.get("details") if isinstance(current.get("details"), dict) else {} - uncertain = exc.needs_manual_review or phase in { - "upload_started", "upload_completed", "submit_clicked", "unknown" - } - result = PublishResult( - outcome=PublishOutcome.NEED_REVIEW if uncertain else PublishOutcome.FAILED, - message=exc.message, - error_code=exc.error_code, - needs_manual_review=uncertain, - provider_response={"diagnostics": diagnostics}, + if prior_conflict is not None: + prior_execution_id, prior_phase = prior_conflict + result = PublishResult( + outcome=PublishOutcome.NEED_REVIEW, + message=( + "同一发布任务已有旧 execution 进入不可安全重试阶段," + "已拒绝再次投稿并要求人工核对" + ), + error_code="job_execution_conflict", + needs_manual_review=True, + provider_response={ + "prior_execution_id": prior_execution_id, + "prior_phase": prior_phase, + }, + ) + journal.update("manual_review", result.as_dict(), identity=identity) + return result.as_dict() + if phase not in safe_resume_phases: + result = PublishResult( + outcome=PublishOutcome.NEED_REVIEW, + message=f"Worker 上次停在 {phase} 阶段,可能已经上传,已禁止自动重试", + error_code="execution_resume_unsafe", + needs_manual_review=True, + ) + journal.update("manual_review", result.as_dict(), identity=identity) + return result.as_dict() + + journal.update( + "received", + {"job_id": payload.job_id, "platform": payload.platform}, + identity=identity, ) - journal.update("manual_review" if uncertain else "failed", result.as_dict()) - return result.as_dict() - except Exception as exc: - result = PublishResult( + lock = _account_lock(payload.platform, payload.account_id) + if not lock.acquire(blocking=False): + result = PublishResult( + outcome=PublishOutcome.FAILED, + message="同一账号已有发布任务正在执行", + error_code="account_busy", + ) + journal.update("rejected", result.as_dict()) + return result.as_dict() + try: + values = payload.model_dump() + values["video_path"] = _resolve_media_path(values["video_path"], required=True) + values["cover_file_path"] = _resolve_media_path(values["cover_file_path"], required=False) + if Path(values["video_path"]).suffix.lower() not in { + ".mp4", ".mov", ".mkv", ".avi", ".flv", ".webm", ".m4v", + }: + raise PublishValidationError("Worker 视频文件类型不受支持", "unsupported_video_format") + if values["cover_file_path"] and Path(values["cover_file_path"]).suffix.lower() not in { + ".jpg", ".jpeg", ".png", ".webp", + }: + raise PublishValidationError("Worker 封面文件类型不受支持", "unsupported_cover_format") + + def update_phase(phase: str, details: dict[str, Any] | None = None) -> None: + journal.update(phase, details) + + runtime = BrowserRuntime( + payload.platform, + payload.account_id, + phase_callback=update_phase, + ) + publisher = get_platform_publisher( + payload.platform, + runtime=runtime, + account_id=payload.account_id, + ) + result = publisher.publish(values) + if ( + result.outcome == PublishOutcome.PUBLISHED + and (not result.published_at or result.needs_manual_review) + ): + result = PublishResult( + outcome=PublishOutcome.NEED_REVIEW, + message="Publisher 成功结果缺少时间证据或仍要求人工复核", + error_code="publish_result_inconsistent", + needs_manual_review=True, + provider_response={"invalid_result": result.as_dict()}, + ) + if ( + result.outcome == PublishOutcome.FAILED + and journal.read().get("upload_started") is True + ): + result = PublishResult( + outcome=PublishOutcome.NEED_REVIEW, + message=result.message or "上传开始后的失败结果需要人工确认", + error_code=result.error_code or "publish_result_uncertain", + needs_manual_review=True, + provider_response=result.provider_response, + ) + result_phase = { + PublishOutcome.PUBLISHED: "confirmed_success", + PublishOutcome.EXPORTED: "exported", + PublishOutcome.NEED_REVIEW: "manual_review", + PublishOutcome.FAILED: "failed", + }[result.outcome] + journal.update(result_phase, result.as_dict()) + return result.as_dict() + except PublishNeedsReview as exc: + current = journal.read() + diagnostics = current.get("details") if isinstance(current.get("details"), dict) else {} + result = PublishResult( + outcome=PublishOutcome.NEED_REVIEW, + message=exc.message, + error_code=exc.error_code, + needs_manual_review=True, + provider_response={"diagnostics": diagnostics}, + ) + journal.update("manual_review", result.as_dict()) + return result.as_dict() + except PublishValidationError as exc: + upload_started = journal.read().get("upload_started") is True + result = PublishResult( + outcome=( + PublishOutcome.NEED_REVIEW if upload_started else PublishOutcome.FAILED + ), + message=exc.message, + error_code=exc.error_code, + needs_manual_review=upload_started, + ) + journal.update("manual_review" if upload_started else "failed", result.as_dict()) + return result.as_dict() + except PublishError as exc: + phase = str(journal.read().get("phase") or "unknown") + current = journal.read() + diagnostics = current.get("details") if isinstance(current.get("details"), dict) else {} + uncertain = ( + exc.needs_manual_review + or current.get("upload_started") is True + or phase in { + "upload_started", + "upload_completed", + "submit_clicked", + "unknown", + } + ) + result = PublishResult( + outcome=PublishOutcome.NEED_REVIEW if uncertain else PublishOutcome.FAILED, + message=exc.message, + error_code=exc.error_code, + needs_manual_review=uncertain, + provider_response={"diagnostics": diagnostics}, + ) + journal.update("manual_review" if uncertain else "failed", result.as_dict()) + return result.as_dict() + except Exception as exc: + result = PublishResult( + outcome=PublishOutcome.NEED_REVIEW, + message=f"Worker 出现未识别异常,请人工确认平台是否已投稿:{exc}", + error_code="worker_result_uncertain", + needs_manual_review=True, + ) + journal.update("manual_review", result.as_dict()) + return result.as_dict() + finally: + lock.release() + + @worker.post("/v1/publish", dependencies=[Depends(require_token)]) + def publish(payload: PublishRequest) -> dict[str, Any]: + lock = _job_lock(payload.job_id) + if not lock.acquire(blocking=True): + return PublishResult( outcome=PublishOutcome.NEED_REVIEW, - message=f"Worker 出现未识别异常,请人工确认平台是否已投稿:{exc}", - error_code="worker_result_uncertain", + message="同一发布任务已有 execution 正在执行,已拒绝并行投稿", + error_code="job_execution_in_progress", needs_manual_review=True, - ) - journal.update("manual_review", result.as_dict()) - return result.as_dict() + ).as_dict() + try: + return publish_with_job_lock(payload) finally: lock.release() @worker.get("/v1/executions/{execution_id}", dependencies=[Depends(require_token)]) def execution(execution_id: str) -> dict[str, Any]: - return ExecutionJournal(execution_id).read() + try: + safe_execution_id = validate_worker_identifier( + execution_id, "execution_id", max_length=160 + ) + except PublishValidationError as exc: + raise HTTPException(status_code=422, detail=exc.message) from exc + journal = ExecutionJournal(safe_execution_id) + execution_state = journal.read() + execution_state["in_progress"] = _lock_file_is_active(journal.lock.path) + return execution_state return worker diff --git a/scripts/repair_foreign_key_integrity.py b/scripts/repair_foreign_key_integrity.py new file mode 100644 index 0000000..ee4dc16 --- /dev/null +++ b/scripts/repair_foreign_key_integrity.py @@ -0,0 +1,347 @@ +"""预演并修复 output_clip 缺失造成的 SQLite 外键异常。 + +默认只读。只有显式传入 ``--apply`` 和预期异常数量时才会写入数据库。 +修复策略是不删除发布/字幕历史,而是补充不可见、无媒体路径的占位 output_clip。 +""" + +from __future__ import annotations + +import argparse +import json +import sqlite3 +import sys +from dataclasses import dataclass +from datetime import datetime, timezone +from pathlib import Path + + +PROJECT_ROOT = Path(__file__).resolve().parents[1] +if str(PROJECT_ROOT) not in sys.path: + sys.path.insert(0, str(PROJECT_ROOT)) + +from app.core.config import settings # noqa: E402 +from app.services.database_backup_service import ( # noqa: E402 + create_schema_migration_backup, + sqlite_quick_check, +) + + +SUPPORTED_CHILD_TABLES = ("publish_jobs", "subtitle_jobs") +TOMBSTONE_STATUS = "integrity_repair_tombstone" +TOMBSTONE_SOURCE = "integrity_repair_tombstone" + + +class ForeignKeyRepairSafetyError(RuntimeError): + """当前异常不满足自动修复条件。""" + + +@dataclass(frozen=True) +class RepairTombstone: + output_clip_id: str + task_id: str + source_tables: tuple[str, ...] + + +@dataclass(frozen=True) +class ForeignKeyRepairPlan: + database_path: Path + violation_count: int + tombstones: tuple[RepairTombstone, ...] + unsupported_violations: tuple[str, ...] + + @property + def tombstone_count(self) -> int: + return len(self.tombstones) + + @property + def can_apply(self) -> bool: + return not self.unsupported_violations + + +def _open_readonly(database_path: Path) -> sqlite3.Connection: + resolved = database_path.resolve() + if not resolved.is_file(): + raise ForeignKeyRepairSafetyError(f"数据库不存在:{resolved}") + connection = sqlite3.connect( + f"{resolved.as_uri()}?mode=ro", + uri=True, + timeout=10, + ) + connection.row_factory = sqlite3.Row + return connection + + +def _foreign_key_target( + connection: sqlite3.Connection, + table: str, + foreign_key_id: int, +) -> tuple[str, str, str] | None: + rows = connection.execute(f'PRAGMA foreign_key_list("{table}")').fetchall() + for row in rows: + if int(row[0]) == foreign_key_id: + return str(row[2]), str(row[3]), str(row[4]) + return None + + +def _build_plan_from_connection( + connection: sqlite3.Connection, + database_path: Path, +) -> ForeignKeyRepairPlan: + violations = connection.execute("PRAGMA foreign_key_check").fetchall() + grouped: dict[str, dict[str, set[str]]] = {} + unsupported: list[str] = [] + + for violation in violations: + table = str(violation[0]) + row_id = int(violation[1]) + parent = str(violation[2]) + foreign_key_id = int(violation[3]) + target = ( + _foreign_key_target(connection, table, foreign_key_id) + if table in SUPPORTED_CHILD_TABLES + else None + ) + if ( + table not in SUPPORTED_CHILD_TABLES + or parent != "output_clip" + or target != ("output_clip", "output_clip_id", "id") + ): + unsupported.append( + f"不支持的外键异常:table={table}, parent={parent}, fkid={foreign_key_id}" + ) + continue + + row = connection.execute( + f'SELECT output_clip_id, task_id FROM "{table}" WHERE rowid = ?', + (row_id,), + ).fetchone() + if not row or not str(row["output_clip_id"] or "").strip(): + unsupported.append(f"{table} rowid={row_id} 缺少可修复的 output_clip_id") + continue + + output_clip_id = str(row["output_clip_id"]).strip() + task_id = str(row["task_id"] or "").strip() + task_exists = connection.execute( + "SELECT 1 FROM tasks WHERE id = ?", + (task_id,), + ).fetchone() + output_exists = connection.execute( + "SELECT 1 FROM output_clip WHERE id = ?", + (output_clip_id,), + ).fetchone() + if not task_id or not task_exists or output_exists: + unsupported.append( + f"{table} rowid={row_id} 的任务或 output_clip 状态不满足占位修复条件" + ) + continue + + item = grouped.setdefault(output_clip_id, {"task_ids": set(), "tables": set()}) + item["task_ids"].add(task_id) + item["tables"].add(table) + + tombstones: list[RepairTombstone] = [] + for output_clip_id, item in sorted(grouped.items()): + task_ids = item["task_ids"] + if len(task_ids) != 1: + unsupported.append( + f"同一缺失 output_clip 被多个任务引用,拒绝猜测归属:{output_clip_id}" + ) + continue + tombstones.append( + RepairTombstone( + output_clip_id=output_clip_id, + task_id=next(iter(task_ids)), + source_tables=tuple(sorted(item["tables"])), + ) + ) + + return ForeignKeyRepairPlan( + database_path=database_path.resolve(), + violation_count=len(violations), + tombstones=tuple(tombstones), + unsupported_violations=tuple(unsupported), + ) + + +def build_repair_plan(database_path: Path) -> ForeignKeyRepairPlan: + connection = _open_readonly(database_path) + try: + return _build_plan_from_connection(connection, database_path) + finally: + connection.close() + + +def _plan_signature(plan: ForeignKeyRepairPlan) -> tuple: + return ( + plan.violation_count, + tuple( + (item.output_clip_id, item.task_id, item.source_tables) + for item in plan.tombstones + ), + plan.unsupported_violations, + ) + + +def _insert_tombstone( + connection: sqlite3.Connection, + tombstone: RepairTombstone, + now: str, +) -> None: + columns = { + str(row[1]) + for row in connection.execute("PRAGMA table_info(output_clip)").fetchall() + } + required = {"id", "task_id", "status", "created_at", "updated_at", "is_active"} + missing = sorted(required - columns) + if missing: + raise ForeignKeyRepairSafetyError( + "output_clip 缺少安全占位所需字段:" + ", ".join(missing) + ) + + values: dict[str, object] = { + "id": tombstone.output_clip_id, + "task_id": tombstone.task_id, + "status": TOMBSTONE_STATUS, + "created_at": now, + "updated_at": now, + "is_active": 0, + } + optional_values = { + "clip_candidate_id": None, + "output_file_path": "", + "output_file_name": "", + "error_message": "外键完整性修复生成的不可见占位记录;原 output_clip 已缺失", + "cut_run_id": None, + "snapshot_source": TOMBSTONE_SOURCE, + } + values.update({key: value for key, value in optional_values.items() if key in columns}) + + names = tuple(values) + placeholders = ", ".join("?" for _ in names) + quoted_names = ", ".join(f'"{name}"' for name in names) + connection.execute( + f"INSERT INTO output_clip ({quoted_names}) VALUES ({placeholders})", + tuple(values[name] for name in names), + ) + + +def apply_repair_plan( + database_path: Path, + backup_dir: Path, + expected_violation_count: int, +) -> dict: + database_path = database_path.resolve() + backup_dir = backup_dir.resolve() + initial_plan = build_repair_plan(database_path) + if initial_plan.violation_count != expected_violation_count: + raise ForeignKeyRepairSafetyError( + "外键异常数量与人工确认值不一致:" + f"expected={expected_violation_count}, actual={initial_plan.violation_count}" + ) + if initial_plan.unsupported_violations: + raise ForeignKeyRepairSafetyError(";".join(initial_plan.unsupported_violations)) + + connection = sqlite3.connect(str(database_path), timeout=10) + connection.row_factory = sqlite3.Row + connection.execute("PRAGMA foreign_keys = ON") + connection.execute("PRAGMA busy_timeout = 5000") + try: + connection.execute("BEGIN IMMEDIATE") + locked_plan = _build_plan_from_connection(connection, database_path) + if _plan_signature(locked_plan) != _plan_signature(initial_plan): + raise ForeignKeyRepairSafetyError("数据库在预演与写入之间发生变化,已中止") + + backup_path = create_schema_migration_backup( + database_path, + backup_dir, + "foreign-key-repair", + ) + backup_plan = build_repair_plan(backup_path) + if _plan_signature(backup_plan) != _plan_signature(locked_plan): + raise ForeignKeyRepairSafetyError("锁内备份与待修复状态不一致,已中止") + + now = datetime.now(timezone.utc).isoformat(timespec="seconds") + for tombstone in locked_plan.tombstones: + _insert_tombstone(connection, tombstone, now) + + remaining = connection.execute("PRAGMA foreign_key_check").fetchall() + if remaining: + raise ForeignKeyRepairSafetyError( + f"事务内复查仍有 {len(remaining)} 条外键异常,已回滚" + ) + connection.commit() + except Exception: + connection.rollback() + raise + finally: + connection.close() + + if sqlite_quick_check(database_path) != "ok": + raise ForeignKeyRepairSafetyError("修复后数据库 quick_check 失败") + final_plan = build_repair_plan(database_path) + if final_plan.violation_count != 0: + raise ForeignKeyRepairSafetyError( + f"修复后仍有 {final_plan.violation_count} 条外键异常" + ) + + return { + "mode": "apply", + "database_path": str(database_path), + "backup_path": str(backup_path), + "before_violation_count": initial_plan.violation_count, + "after_violation_count": final_plan.violation_count, + "tombstone_count": initial_plan.tombstone_count, + "quick_check": "ok", + } + + +def _dry_run_report(plan: ForeignKeyRepairPlan) -> dict: + table_counts = {table: 0 for table in SUPPORTED_CHILD_TABLES} + for tombstone in plan.tombstones: + for table in tombstone.source_tables: + table_counts[table] += 1 + return { + "mode": "dry-run", + "database_path": str(plan.database_path), + "violation_count": plan.violation_count, + "tombstone_count": plan.tombstone_count, + "tombstone_source_table_counts": table_counts, + "can_apply": plan.can_apply, + "unsupported_violations": list(plan.unsupported_violations), + } + + +def main() -> int: + parser = argparse.ArgumentParser(description="安全修复 output_clip 孤儿外键") + parser.add_argument("--database", type=Path, default=settings.database_path) + parser.add_argument("--backup-dir", type=Path, default=settings.data_dir / "backups") + parser.add_argument("--apply", action="store_true", help="实际写入;默认只预演") + parser.add_argument( + "--expected-violation-count", + type=int, + help="应用时必须提供,且必须与预演数量完全一致", + ) + args = parser.parse_args() + + try: + if args.apply: + if args.expected_violation_count is None: + raise ForeignKeyRepairSafetyError( + "--apply 必须同时提供 --expected-violation-count" + ) + report = apply_repair_plan( + args.database, + args.backup_dir, + args.expected_violation_count, + ) + else: + report = _dry_run_report(build_repair_plan(args.database)) + print(json.dumps(report, ensure_ascii=False, indent=2)) + return 0 + except (sqlite3.Error, OSError, ForeignKeyRepairSafetyError) as exc: + print(f"外键修复已中止:{exc}", file=sys.stderr) + return 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/sonar-project.properties b/sonar-project.properties new file mode 100644 index 0000000..54014cf --- /dev/null +++ b/sonar-project.properties @@ -0,0 +1,14 @@ +sonar.projectKey=niuma-studio-local-audit +sonar.projectName=NiuMa Studio +sonar.projectVersion=2.1.0 +sonar.sourceEncoding=UTF-8 + +sonar.sources=app,scripts +sonar.tests=tests +sonar.test.inclusions=tests/**/*.py +sonar.exclusions=app/static/vendor/**,app/static/img/**,scripts/test_*.py,**/__pycache__/** + +sonar.python.version=3.12 +sonar.python.coverage.reportPaths=coverage.xml +sonar.python.xunit.reportPath=test-results.xml +sonar.working.directory=.scannerwork diff --git a/tests/conftest.py b/tests/conftest.py index 7fd1474..e9bf998 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -2,13 +2,42 @@ import os import sys +import tempfile from pathlib import Path +import pytest + # 让测试代码可以直接导入 app 模块 PROJECT_ROOT = Path(__file__).resolve().parents[1] sys.path.insert(0, str(PROJECT_ROOT)) -os.environ.setdefault("STORAGE_ROOT", str(PROJECT_ROOT / "data" / "test_storage")) -os.environ.setdefault("TASKS_DIR", str(PROJECT_ROOT / "data" / "test_storage")) -os.environ.setdefault("DATA_DIR", str(PROJECT_ROOT / "data")) -os.environ.setdefault("DATABASE_PATH", str(PROJECT_ROOT / "data" / "test_workflow.sqlite3")) +# 测试进程必须拥有自己的数据库和文件根目录。这里刻意使用无条件赋值, +# 防止调用 pytest 时继承 DATABASE_PATH、STORAGE_ROOT 等活动环境变量。 +PYTEST_SANDBOX_ROOT = Path(tempfile.mkdtemp(prefix="niuma-pytest-")).resolve() +PYTEST_STORAGE_ROOT = PYTEST_SANDBOX_ROOT / "storage" +PYTEST_TASKS_DIR = PYTEST_STORAGE_ROOT / "tasks" +PYTEST_UPLOAD_TEMP_DIR = PYTEST_STORAGE_ROOT / "_临时上传" +PYTEST_DATA_DIR = PYTEST_SANDBOX_ROOT / "data" +PYTEST_DATABASE_PATH = PYTEST_DATA_DIR / "test_workflow.sqlite3" +PYTEST_PUBLISH_EXPORT_DIR = PYTEST_STORAGE_ROOT / "_发布包" + +os.environ.update( + { + "STORAGE_ROOT": str(PYTEST_STORAGE_ROOT), + "TASKS_DIR": str(PYTEST_TASKS_DIR), + "UPLOAD_TEMP_DIR": str(PYTEST_UPLOAD_TEMP_DIR), + "DATA_DIR": str(PYTEST_DATA_DIR), + "DATABASE_PATH": str(PYTEST_DATABASE_PATH), + "PUBLISH_SCHEDULER_EXPORT_DIR": str(PYTEST_PUBLISH_EXPORT_DIR), + "NIUMA_PYTEST_SANDBOX_ROOT": str(PYTEST_SANDBOX_ROOT), + } +) + + +@pytest.fixture(scope="session", autouse=True) +def initialize_isolated_test_database(): + """在任何测试访问服务层之前初始化本进程专属临时数据库。""" + + from app.db.database import init_db + + init_db() diff --git a/tests/test_ai_job_consistency.py b/tests/test_ai_job_consistency.py new file mode 100644 index 0000000..2d0604a --- /dev/null +++ b/tests/test_ai_job_consistency.py @@ -0,0 +1,493 @@ +from __future__ import annotations + +import json +from pathlib import Path + +import pytest +from fastapi.testclient import TestClient + +from app.main import app +from app.models.task import AIClipAnalysisResult, AIClipItem, TaskCreate, TaskStatus +from app.services import job_service, job_worker +from app.services import ai_analysis_workflow_service as workflow +from app.services.ai import unit_checkpoint +from app.services.storage_service import get_artifact_paths +from app.services.task_lifecycle_service import create_task_record, update_task_status +from app.services.task_service import get_task, list_clip_candidates +from app.db.database import get_connection + + +PREFIX = "test-ai-consistency-" + + +@pytest.fixture(autouse=True) +def cleanup_rows(): + yield + with get_connection() as connection: + for table in ("publish_jobs", "workflow_jobs", "output_clip", "clip_candidates", "ai_analysis_runs"): + connection.execute(f"DELETE FROM {table} WHERE task_id LIKE ?", (f"{PREFIX}%",)) + connection.execute("DELETE FROM tasks WHERE id LIKE ?", (f"{PREFIX}%",)) + connection.commit() + + +def _create_task(suffix: str, *, status: TaskStatus = TaskStatus.pending_ai) -> str: + task_id = f"{PREFIX}{suffix}" + create_task_record(TaskCreate(task_name=suffix, selection_profile="general"), task_id=task_id) + update_task_status(task_id, status) + transcript_path = get_artifact_paths(task_id)["transcript_path"] + transcript_path.parent.mkdir(parents=True, exist_ok=True) + transcript_path.write_text("00:00:00 - 00:02:00 测试转写正文", encoding="utf-8") + return task_id + + +def _analysis(task_id: str, *, incomplete: bool = False) -> AIClipAnalysisResult: + return AIClipAnalysisResult( + task_id=task_id, + analysis_summary="隔离 AI 结果", + clips=[ + AIClipItem( + clip_id="clip_001", + title="隔离候选", + start_time="00:00:10", + end_time="00:01:10", + duration_seconds=60, + cover_time_seconds=30, + summary="摘要", + highlight_reason="亮点", + spread_value="高", + suggested_editing="保留上下文", + confidence_score=0.9, + ) + ], + analysis_meta={ + "schema_version": 2, + "coverage_basis": "chunk_count", + "expected_units": 2, + "completed_units": 1 if incomplete else 2, + "failed_units": 1 if incomplete else 0, + "coverage_ratio": 0.5 if incomplete else 1.0, + "coverage_percent": 50 if incomplete else 100, + "analysis_incomplete": incomplete, + "quality_degraded": False, + "failed_stages": [{"stage": "chunk", "unit_id": "chunk_002"}] if incomplete else [], + }, + ) + + +def _claim_ai_job(task_id: str, owner: str = "ai-test-owner") -> tuple[dict, str]: + job, _created = workflow.queue_task_ai_analysis(task_id, provider="remote") + claimed = job_service.claim_job(job["id"], owner) + assert claimed + return claimed, owner + + +def test_manual_api_creates_and_reuses_persistent_ai_job(monkeypatch): + task_id = _create_task("queue") + monkeypatch.setattr(workflow, "_analyze_with_provider", lambda *_args, **_kwargs: pytest.fail("API 不应直接调用 Provider")) + client = TestClient(app) + + first = client.post(f"/api/tasks/{task_id}/process/ai?provider=remote") + second = client.post(f"/api/tasks/{task_id}/process/ai?provider=remote") + + assert first.status_code == 200 + assert first.json()["status"] == job_service.JOB_STATUS_QUEUED + assert second.json()["job_id"] == first.json()["job_id"] + with get_connection() as connection: + count = connection.execute( + "SELECT COUNT(*) AS total FROM workflow_jobs WHERE task_id = ? AND job_type = ?", + (task_id, job_service.JOB_TYPE_AI_ANALYSIS), + ).fetchone()["total"] + assert count == 1 + + +def test_failed_ai_job_reuses_original_checkpoint_ledger(): + task_id = _create_task("retry-ledger") + claimed, owner = _claim_ai_job(task_id, owner="retry-owner") + checkpoint = { + "_ai_analysis_units_v1": { + "namespaces": { + "general_chunks": { + "input_fingerprint": "stable", + "units": {"chunk_001": {"status": "uncertain", "error": "计费结果不确定"}}, + } + } + } + } + job_service.update_job_checkpoint( + claimed["id"], + checkpoint, + lease_owner=owner, + lease_token=claimed["lease_token"], + ) + job_service.mark_job_failed( + claimed["id"], + "Provider 结果不确定", + lease_owner=owner, + lease_token=claimed["lease_token"], + ) + + retried, created = workflow.queue_task_ai_analysis(task_id, provider="remote") + + assert created is True + assert retried["id"] == claimed["id"] + assert retried["status"] == job_service.JOB_STATUS_QUEUED + assert retried["checkpoint_json"] == checkpoint + + +def test_failed_ai_job_cannot_switch_provider_and_drop_ledger(): + task_id = _create_task("retry-provider-change") + claimed, owner = _claim_ai_job(task_id, owner="provider-owner") + job_service.mark_job_failed( + claimed["id"], + "Provider 结果不确定", + lease_owner=owner, + lease_token=claimed["lease_token"], + ) + + with pytest.raises(workflow.AIAnalysisConflictError, match="不能在同一账本中切换"): + workflow.queue_task_ai_analysis(task_id, provider="local") + + assert job_service.get_job(claimed["id"])["status"] == job_service.JOB_STATUS_FAILED + + +def test_ai_job_cancel_and_parent_failure_settle_task_state(): + cancel_task_id = _create_task("cancel-settle") + cancel_job, cancel_owner = _claim_ai_job(cancel_task_id, owner="cancel-owner") + with job_service.job_lease_context(cancel_job["id"], cancel_owner, cancel_job["lease_token"]): + workflow._begin_ai_analysis( + cancel_task_id, + get_task(cancel_task_id, include_video_probe=False), + ) + requested = job_service.request_job_cancel(cancel_job["id"]) + assert requested["cancel_requested"] == 1 + assert get_task(cancel_task_id, include_video_probe=False)["status"] == TaskStatus.pending_ai.value + assert job_service.release_job_lease( + cancel_job["id"], + cancel_owner, + cancel_job["lease_token"], + ) + assert job_service.get_job(cancel_job["id"])["status"] == job_service.JOB_STATUS_CANCELLED + + failed_task_id = _create_task("parent-failure") + failed_job, failed_owner = _claim_ai_job(failed_task_id, owner="failed-owner") + with job_service.job_lease_context(failed_job["id"], failed_owner, failed_job["lease_token"]): + workflow._begin_ai_analysis( + failed_task_id, + get_task(failed_task_id, include_video_probe=False), + ) + job_service.mark_job_failed( + failed_job["id"], + "子进程异常退出", + lease_owner=failed_owner, + lease_token=failed_job["lease_token"], + ) + failed_task = get_task(failed_task_id, include_video_probe=False) + assert failed_task["status"] == TaskStatus.failed.value + assert failed_task["last_error"] == "子进程异常退出" + + +def test_confirmed_ai_unit_is_reused_after_lease_takeover(): + task_id = _create_task("unit-reuse") + claimed, owner = _claim_ai_job(task_id, owner="unit-owner-a") + calls = 0 + + def operation() -> dict: + nonlocal calls + calls += 1 + return {"clips": [{"clip_id": "confirmed"}]} + + with job_service.job_lease_context(claimed["id"], owner, claimed["lease_token"]): + first = unit_checkpoint.execute_checkpointed_ai_unit( + task_id=task_id, + namespace="test_units", + input_fingerprint="stable-input", + unit_id="unit-001", + operation=operation, + ) + with get_connection() as connection: + connection.execute( + "UPDATE workflow_jobs SET lease_expires_at = '2000-01-01T00:00:00+00:00' WHERE id = ?", + (claimed["id"],), + ) + connection.commit() + reclaimed = job_service.claim_job(claimed["id"], "unit-owner-b") + assert reclaimed + with job_service.job_lease_context(reclaimed["id"], "unit-owner-b", reclaimed["lease_token"]): + second = unit_checkpoint.execute_checkpointed_ai_unit( + task_id=task_id, + namespace="test_units", + input_fingerprint="stable-input", + unit_id="unit-001", + operation=operation, + ) + + assert first.status == "completed" + assert second.status == "completed" + assert second.reused is True + assert calls == 1 + + +def test_started_ai_unit_without_local_result_is_not_rebilled_after_takeover(monkeypatch): + task_id = _create_task("unit-uncertain") + claimed, owner = _claim_ai_job(task_id, owner="uncertain-owner-a") + original_finish = unit_checkpoint._finish_unit_success + monkeypatch.setattr( + unit_checkpoint, + "_finish_unit_success", + lambda **_kwargs: (_ for _ in ()).throw(job_service.JobLeaseLostError("crashed before save")), + ) + with job_service.job_lease_context(claimed["id"], owner, claimed["lease_token"]): + with pytest.raises(job_service.JobLeaseLostError): + unit_checkpoint.execute_checkpointed_ai_unit( + task_id=task_id, + namespace="test_uncertain", + input_fingerprint="stable-input", + unit_id="unit-001", + operation=lambda: {"clips": [{"clip_id": "provider-returned"}]}, + ) + monkeypatch.setattr(unit_checkpoint, "_finish_unit_success", original_finish) + with get_connection() as connection: + connection.execute( + "UPDATE workflow_jobs SET lease_expires_at = '2000-01-01T00:00:00+00:00' WHERE id = ?", + (claimed["id"],), + ) + connection.commit() + reclaimed = job_service.claim_job(claimed["id"], "uncertain-owner-b") + provider_called = False + + def forbidden_operation() -> dict: + nonlocal provider_called + provider_called = True + return {} + + with job_service.job_lease_context(reclaimed["id"], "uncertain-owner-b", reclaimed["lease_token"]): + result = unit_checkpoint.execute_checkpointed_ai_unit( + task_id=task_id, + namespace="test_uncertain", + input_fingerprint="stable-input", + unit_id="unit-001", + operation=forbidden_operation, + ) + + assert result.status == "uncertain" + assert result.reused is True + assert provider_called is False + + +def test_unknown_ai_unit_checkpoint_state_is_not_rebilled(): + task_id = _create_task("unit-corrupt") + claimed, owner = _claim_ai_job(task_id, owner="corrupt-owner") + with get_connection() as connection: + connection.execute( + """ + UPDATE workflow_jobs SET checkpoint_json = ? WHERE id = ? + """, + ( + json.dumps( + { + "_ai_analysis_units_v1": { + "namespaces": { + "test_corrupt": { + "input_fingerprint": "stable-input", + "units": {"unit-001": {"status": "unexpected"}}, + } + } + } + } + ), + claimed["id"], + ), + ) + connection.commit() + provider_called = False + + def forbidden_operation() -> dict: + nonlocal provider_called + provider_called = True + return {} + + with job_service.job_lease_context(claimed["id"], owner, claimed["lease_token"]): + result = unit_checkpoint.execute_checkpointed_ai_unit( + task_id=task_id, + namespace="test_corrupt", + input_fingerprint="stable-input", + unit_id="unit-001", + operation=forbidden_operation, + ) + + assert result.status == "uncertain" + assert result.reused is True + assert provider_called is False + + +def test_changed_ai_unit_fingerprint_preserves_old_ledger(): + task_id = _create_task("unit-fingerprint-change") + claimed, owner = _claim_ai_job(task_id, owner="fingerprint-owner") + with get_connection() as connection: + checkpoint = { + "_ai_analysis_units_v1": { + "namespaces": { + "test_changed": { + "input_fingerprint": "old-input", + "units": {"unit-001": {"status": "uncertain", "error": "旧请求不确定"}}, + } + } + } + } + connection.execute( + "UPDATE workflow_jobs SET checkpoint_json = ? WHERE id = ?", + (json.dumps(checkpoint), claimed["id"]), + ) + connection.commit() + provider_called = False + + def forbidden_operation() -> dict: + nonlocal provider_called + provider_called = True + return {} + + with job_service.job_lease_context(claimed["id"], owner, claimed["lease_token"]): + with pytest.raises(ValueError, match="旧恢复证据不会被覆盖"): + unit_checkpoint.execute_checkpointed_ai_unit( + task_id=task_id, + namespace="test_changed", + input_fingerprint="new-input", + unit_id="unit-001", + operation=forbidden_operation, + ) + + assert provider_called is False + assert job_service.get_job(claimed["id"])["checkpoint_json"] == checkpoint + + +def test_ai_worker_commits_candidates_run_task_and_job(monkeypatch): + task_id = _create_task("worker") + monkeypatch.setattr(workflow, "_analyze_with_provider", lambda *_args, **_kwargs: _analysis(task_id)) + claimed, owner = _claim_ai_job(task_id) + + completed = job_worker.execute_job( + claimed["id"], + lease_owner=owner, + lease_token=claimed["lease_token"], + already_claimed=True, + ) + + assert completed["status"] == job_service.JOB_STATUS_COMPLETED + assert completed["result_json"]["clip_count"] == 1 + assert get_task(task_id, include_video_probe=False)["status"] == TaskStatus.pending_review.value + assert [clip["title"] for clip in list_clip_candidates(task_id)] == ["隔离候选"] + latest = workflow.get_latest_ai_analysis_run(task_id) + assert latest["analysis_meta"]["workflow_job_id"] == claimed["id"] + + +def test_ai_result_transaction_rolls_back_on_run_insert_failure(monkeypatch): + task_id = _create_task("rollback", status=TaskStatus.pending_review) + workflow._replace_clip_candidates(task_id, [workflow.result_to_jsonable(_analysis(task_id))["clips"][0]]) + old_run = workflow._insert_ai_analysis_run( + task_id=task_id, + analysis_payload=workflow.result_to_jsonable(_analysis(task_id)), + provider="remote", + provider_label="远程 AI", + model="old-model", + fallback_notice="", + prompt_preset={}, + requested_clip_count=1, + ) + monkeypatch.setattr(workflow, "_analyze_with_provider", lambda *_args, **_kwargs: _analysis(task_id)) + monkeypatch.setattr( + workflow, + "_insert_ai_analysis_run_with_connection", + lambda *_args, **_kwargs: (_ for _ in ()).throw(RuntimeError("run insert failed")), + ) + claimed, owner = _claim_ai_job(task_id) + + with job_service.job_lease_context(claimed["id"], owner, claimed["lease_token"]): + with pytest.raises(ValueError, match="run insert failed"): + workflow.process_task_ai_analysis(task_id, provider="remote") + + assert [clip["title"] for clip in list_clip_candidates(task_id)] == ["隔离候选"] + runs = workflow.list_ai_analysis_runs(task_id) + assert len(runs) == 1 + assert runs[0]["id"] == old_run["id"] + + +def test_stale_ai_worker_cannot_commit_after_lease_takeover(monkeypatch): + task_id = _create_task("fencing") + claimed, owner = _claim_ai_job(task_id, owner="old-owner") + replacement: dict = {} + + def steal_lease(*_args, **_kwargs): + with get_connection() as connection: + connection.execute( + "UPDATE workflow_jobs SET lease_expires_at = '2000-01-01T00:00:00+00:00' WHERE id = ?", + (claimed["id"],), + ) + connection.commit() + replacement.update(job_service.claim_job(claimed["id"], "new-owner") or {}) + return _analysis(task_id) + + monkeypatch.setattr(workflow, "_analyze_with_provider", steal_lease) + + with job_service.job_lease_context(claimed["id"], owner, claimed["lease_token"]): + with pytest.raises(job_service.JobLeaseLostError): + workflow.process_task_ai_analysis(task_id, provider="remote") + + assert replacement["lease_owner"] == "new-owner" + assert list_clip_candidates(task_id) == [] + assert workflow.list_ai_analysis_runs(task_id) == [] + assert get_task(task_id, include_video_probe=False)["status"] == TaskStatus.ai_analyzing.value + + +def test_taken_over_job_reuses_committed_run_and_rebuilds_file_without_provider(monkeypatch): + task_id = _create_task("resume") + calls = 0 + + def analyze(*_args, **_kwargs): + nonlocal calls + calls += 1 + return _analysis(task_id) + + monkeypatch.setattr(workflow, "_analyze_with_provider", analyze) + claimed, owner = _claim_ai_job(task_id, owner="first-owner") + with job_service.job_lease_context(claimed["id"], owner, claimed["lease_token"]): + first = workflow.process_task_ai_analysis(task_id, provider="remote") + assert first["analysis_run_id"] + analysis_path: Path = get_artifact_paths(task_id)["analysis_path"] + analysis_path.unlink(missing_ok=True) + + with get_connection() as connection: + connection.execute( + "UPDATE workflow_jobs SET lease_expires_at = '2000-01-01T00:00:00+00:00' WHERE id = ?", + (claimed["id"],), + ) + connection.commit() + takeover = job_service.claim_job(claimed["id"], "second-owner") + assert takeover + with job_service.job_lease_context(takeover["id"], "second-owner", takeover["lease_token"]): + resumed = workflow.process_task_ai_analysis(task_id, provider="remote") + + assert resumed["analysis_run_id"] == first["analysis_run_id"] + assert calls == 1 + assert analysis_path.exists() + + +def test_corrupt_active_analysis_meta_fails_closed(): + task_id = _create_task("corrupt-meta", status=TaskStatus.pending_review) + with get_connection() as connection: + connection.execute( + """ + INSERT INTO ai_analysis_runs ( + id, task_id, run_number, provider, provider_label, model, + requested_clip_count, clip_count, analysis_summary, + analysis_payload_json, is_active, created_at + ) VALUES (?, ?, 1, 'remote', '远程 AI', 'test-model', 1, 1, '', ?, 1, CURRENT_TIMESTAMP) + """, + ("corrupt-meta-run", task_id, json.dumps({"clips": []})), + ) + connection.commit() + + meta = workflow.get_task_ai_analysis_meta(task_id) + + assert meta["analysis_incomplete"] is True + assert meta["quality_degraded"] is True + assert meta["integrity_error"] == "active_run_meta_missing" diff --git a/tests/test_auto_pipeline.py b/tests/test_auto_pipeline.py index 0f327c7..6a63aa8 100644 --- a/tests/test_auto_pipeline.py +++ b/tests/test_auto_pipeline.py @@ -5,6 +5,7 @@ from datetime import datetime, timezone from pathlib import Path from unittest.mock import Mock +from types import SimpleNamespace from fastapi.testclient import TestClient import pytest @@ -14,6 +15,7 @@ from app.main import app from app.models.task import TaskCreate, TaskStatus from app.services.auto_publish_service import create_auto_publish_jobs +from app.services import job_service from app.services.pipeline_engine import PipelineEngine, build_schedule_times from app.services.storage_service import get_artifact_paths from app.services.task_lifecycle_service import create_task_record, update_task_status @@ -22,11 +24,24 @@ @pytest.fixture(autouse=True) -def auto_pipeline_db_cleanup(): +def auto_pipeline_db_cleanup(monkeypatch): + monkeypatch.setattr( + "app.services.task_lifecycle_service.preflight_media", + lambda *_args, **_kwargs: SimpleNamespace(to_dict=lambda: {"warnings": []}), + ) init_db() with get_connection() as connection: + connection.execute("DELETE FROM ai_analysis_windows WHERE task_id LIKE 'test-auto-%'") connection.execute("DELETE FROM publish_jobs WHERE task_id LIKE 'test-auto-%'") connection.execute("DELETE FROM subtitle_jobs WHERE task_id LIKE 'test-auto-%'") + connection.execute("DELETE FROM workflow_jobs WHERE task_id LIKE 'test-auto-%'") + connection.execute( + "DELETE FROM subtitle_cues WHERE revision_id IN (SELECT id FROM subtitle_revisions WHERE track_id IN (SELECT id FROM subtitle_tracks WHERE task_id LIKE 'test-auto-%'))" + ) + connection.execute( + "DELETE FROM subtitle_revisions WHERE track_id IN (SELECT id FROM subtitle_tracks WHERE task_id LIKE 'test-auto-%')" + ) + connection.execute("DELETE FROM subtitle_tracks WHERE task_id LIKE 'test-auto-%'") connection.execute("DELETE FROM output_clip WHERE task_id LIKE 'test-auto-%'") connection.execute("DELETE FROM cut_runs WHERE task_id LIKE 'test-auto-%'") connection.execute("DELETE FROM clip_candidates WHERE task_id LIKE 'test-auto-%'") @@ -35,8 +50,17 @@ def auto_pipeline_db_cleanup(): connection.commit() yield with get_connection() as connection: + connection.execute("DELETE FROM ai_analysis_windows WHERE task_id LIKE 'test-auto-%'") connection.execute("DELETE FROM publish_jobs WHERE task_id LIKE 'test-auto-%'") connection.execute("DELETE FROM subtitle_jobs WHERE task_id LIKE 'test-auto-%'") + connection.execute("DELETE FROM workflow_jobs WHERE task_id LIKE 'test-auto-%'") + connection.execute( + "DELETE FROM subtitle_cues WHERE revision_id IN (SELECT id FROM subtitle_revisions WHERE track_id IN (SELECT id FROM subtitle_tracks WHERE task_id LIKE 'test-auto-%'))" + ) + connection.execute( + "DELETE FROM subtitle_revisions WHERE track_id IN (SELECT id FROM subtitle_tracks WHERE task_id LIKE 'test-auto-%')" + ) + connection.execute("DELETE FROM subtitle_tracks WHERE task_id LIKE 'test-auto-%'") connection.execute("DELETE FROM output_clip WHERE task_id LIKE 'test-auto-%'") connection.execute("DELETE FROM cut_runs WHERE task_id LIKE 'test-auto-%'") connection.execute("DELETE FROM clip_candidates WHERE task_id LIKE 'test-auto-%'") @@ -51,6 +75,60 @@ def _headers() -> dict[str, str]: return {} +def test_manual_ai_endpoint_returns_409_while_auto_pipeline_is_active(monkeypatch): + task_id = "test-auto-ai-job-guard" + create_task_record( + TaskCreate(task_name="自动流水线 AI 重入保护", selection_profile="general", auto_mode=True), + task_id=task_id, + ) + update_task_status(task_id, TaskStatus.pending_ai) + job_service.create_job(task_id, job_service.JOB_TYPE_AUTO_PIPELINE) + monkeypatch.setattr( + "app.services.ai_analysis_workflow_service._analyze_with_provider", + lambda *_args, **_kwargs: pytest.fail("冲突请求不应调用 AI Provider"), + ) + + response = TestClient(app).post(f"/api/tasks/{task_id}/process/ai", headers=_headers()) + + assert response.status_code == 409 + assert "全自动流水线正在处理" in response.json()["detail"] + assert get_task(task_id, include_video_probe=False)["status"] == TaskStatus.pending_ai.value + assert not [ + job + for job in job_service.list_jobs(task_id=task_id) + if job.get("job_type") == job_service.JOB_TYPE_AI_ANALYSIS + ] + + +def test_manual_ai_endpoint_returns_409_after_clips_are_materialized(monkeypatch): + task_id = "test-auto-ai-output-guard" + create_task_record(TaskCreate(task_name="切片后 AI 重入保护", selection_profile="general"), task_id=task_id) + update_task_status(task_id, TaskStatus.completed) + with get_connection() as connection: + connection.execute( + """ + INSERT INTO output_clip ( + id, task_id, output_file_path, output_file_name, + status, is_active, created_at, updated_at + ) VALUES (?, ?, 'clip.mp4', 'clip.mp4', 'completed', 1, 'now', 'now') + """, + (f"{task_id}-output", task_id), + ) + connection.commit() + monkeypatch.setattr( + "app.services.ai_analysis_workflow_service._analyze_with_provider", + lambda *_args, **_kwargs: pytest.fail("冲突请求不应调用 AI Provider"), + ) + + response = TestClient(app).post(f"/api/tasks/{task_id}/process/ai", headers=_headers()) + + assert response.status_code == 409 + assert "已经生成切片" in response.json()["detail"] + task = get_task(task_id, include_video_probe=False) + assert task["status"] == TaskStatus.completed.value + assert not task["error_message"] + + def _fake_video(name: str = "source.mp4") -> Path: path = settings.tasks_dir / "_test_inputs" / name path.parent.mkdir(parents=True, exist_ok=True) @@ -71,12 +149,13 @@ def _create_auto_task(task_id: str = "test-auto-task") -> dict: task_name=task_id, source_type="upload", platform="general", + selection_profile="general", original_video_path=str(video), max_clip_duration=5, candidate_clip_count=5, auto_mode=True, ) - create_task_record(payload, task_id=task_id) + create_task_record(payload, task_id=task_id, task_dir_name=task_id) return get_task(task_id, include_video_probe=False) @@ -93,16 +172,20 @@ def test_clip_candidates_schema_has_nullable_cover_time(): def test_auto_mode_false_does_not_start_pipeline(monkeypatch): starter = Mock(return_value={"status": "started"}) monkeypatch.setattr("app.routers.tasks.start_auto_pipeline", starter) - video = _fake_video("manual.mp4") + monkeypatch.setattr("app.routers.tasks.uuid4", lambda: SimpleNamespace(hex="test-auto-m1-upload")) payload = { "task_name": "test-auto-manual", - "source_type": "upload", "platform": "general", - "original_video_path": str(video), - "auto_mode": False, + "selection_profile": "general", + "auto_mode": "false", } with TestClient(app) as client: - response = client.post("/api/tasks", json=payload, headers=_headers()) + response = client.post( + "/api/tasks/upload", + data=payload, + files={"video_file": ("manual.mp4", b"fake mp4", "video/mp4")}, + headers=_headers(), + ) assert response.status_code == 200 starter.assert_not_called() @@ -110,20 +193,37 @@ def test_auto_mode_false_does_not_start_pipeline(monkeypatch): def test_auto_mode_true_starts_pipeline(monkeypatch): starter = Mock(return_value={"status": "started"}) monkeypatch.setattr("app.routers.tasks.start_auto_pipeline", starter) - video = _fake_video("auto.mp4") + monkeypatch.setattr("app.routers.tasks.uuid4", lambda: SimpleNamespace(hex="test-auto-a1-upload")) payload = { "task_name": "test-auto-start", - "source_type": "upload", "platform": "general", - "original_video_path": str(video), - "auto_mode": True, + "selection_profile": "general", + "auto_mode": "true", } with TestClient(app) as client: - response = client.post("/api/tasks", json=payload, headers=_headers()) + response = client.post( + "/api/tasks/upload", + data=payload, + files={"video_file": ("auto.mp4", b"fake mp4", "video/mp4")}, + headers=_headers(), + ) assert response.status_code == 200 starter.assert_called_once() +def test_json_task_creation_route_is_removed(): + with TestClient(app) as client: + response = client.post( + "/api/tasks", + json={"task_name": "不允许的已有文件任务", "source_type": "nas"}, + headers=_headers(), + ) + browse = client.get("/api/files/browse", headers=_headers()) + assert response.status_code == 405 + assert browse.status_code == 404 + assert "/api/files/browse" not in app.openapi()["paths"] + + def test_existing_transcript_skips_transcription(monkeypatch): task = _create_auto_task("test-auto-existing-transcript") paths = get_artifact_paths(task["id"]) @@ -245,7 +345,13 @@ def test_daily_window_schedule_supports_seven_to_midnight_without_looping(): def test_create_auto_publish_job_records_scheduled_at(): task = _create_auto_task("test-auto-publish-job") - clip_path = _fake_video("publish_clip.mp4") + paths = get_artifact_paths(task["id"]) + paths["clips_dir"].mkdir(parents=True, exist_ok=True) + paths["covers_dir"].mkdir(parents=True, exist_ok=True) + clip_path = paths["clips_dir"] / "publish_clip.mp4" + cover_path = paths["covers_dir"] / "publish_clip_cover.jpg" + clip_path.write_bytes(b"fake mp4") + cover_path.write_bytes(b"fake jpg") with get_connection() as connection: now = "2026-06-23T08:00:00+00:00" connection.execute( @@ -263,7 +369,7 @@ def test_create_auto_publish_job_records_scheduled_at(): { "output_clip": {"id": "out-1", "output_file_path": str(clip_path)}, "cover": { - "cover_file_path": str(_fake_cover("publish_clip_cover.jpg")), + "cover_file_path": str(cover_path), "cover_time_seconds": 12.5, "cover_source": "ai_frame", }, @@ -279,8 +385,45 @@ def test_create_auto_publish_job_records_scheduled_at(): "scheduled_at": "2026-06-23T08:10:00+00:00", } ] - result = create_auto_publish_jobs(task, scheduled_items) + workflow_job = job_service.create_job(task["id"], job_service.JOB_TYPE_AUTO_PIPELINE) + claimed = job_service.claim_job(workflow_job["id"], "auto-publish-test-worker") + with job_service.job_lease_context( + workflow_job["id"], + "auto-publish-test-worker", + claimed["lease_token"], + ): + result = create_auto_publish_jobs( + task, + scheduled_items, + subtitle_delivery_mode="original", + workflow_job_id=workflow_job["id"], + ) + cover_path.write_bytes(b"changed cover") + with pytest.raises(ValueError, match="证据已失效"): + create_auto_publish_jobs( + task, + scheduled_items, + subtitle_delivery_mode="original", + workflow_job_id=workflow_job["id"], + ) + cover_path.write_bytes(b"fake jpg") + changed_copy = [ + { + **scheduled_items[0], + "metadata": {**scheduled_items[0]["metadata"], "title": "新的排期标题"}, + } + ] + with pytest.raises(ValueError, match="文案证据已失效"): + create_auto_publish_jobs( + task, + changed_copy, + subtitle_delivery_mode="original", + workflow_job_id=workflow_job["id"], + ) assert result["created_count"] == 1 + assert result["created"][0]["provider_payload"]["workflow_job_id"] == workflow_job["id"] + assert result["created"][0]["provider_payload"]["video_file_fingerprint"] + assert result["created"][0]["provider_payload"]["cover_file_fingerprint"] with get_connection() as connection: row = connection.execute( "SELECT scheduled_at, status, video_source, cover_mode, cover_time_seconds, cover_file_path FROM publish_jobs WHERE task_id = ?", @@ -295,6 +438,47 @@ def test_create_auto_publish_job_records_scheduled_at(): assert row["cover_file_path"].endswith("publish_clip_cover.jpg") +def test_workflow_publish_rejects_artifacts_outside_task_directory(): + task = _create_auto_task("test-auto-publish-unmanaged") + clip_path = _fake_video("unmanaged_publish_clip.mp4") + cover_path = _fake_cover("unmanaged_publish_cover.jpg") + workflow_job = job_service.create_job(task["id"], job_service.JOB_TYPE_AUTO_PIPELINE) + claimed = job_service.claim_job(workflow_job["id"], "unmanaged-publish-worker") + + with job_service.job_lease_context( + workflow_job["id"], + "unmanaged-publish-worker", + claimed["lease_token"], + ): + with pytest.raises(ValueError, match="原片切片文件不存在"): + create_auto_publish_jobs( + task, + [ + { + "output_clip": {"id": "unmanaged-out", "output_file_path": str(clip_path)}, + "cover": {"cover_file_path": str(cover_path), "cover_time_seconds": 1}, + "metadata": { + "platform": "douyin", + "title": "越界测试", + "caption": "越界测试", + "hashtags": ["测试"], + "risk_flags": [], + }, + "scheduled_at": "", + } + ], + subtitle_delivery_mode="original", + workflow_job_id=workflow_job["id"], + ) + + with get_connection() as connection: + count = connection.execute( + "SELECT COUNT(*) FROM publish_jobs WHERE task_id = ?", + (task["id"],), + ).fetchone()[0] + assert count == 0 + + def test_create_auto_publish_job_without_schedule_waits_for_send_center(): task = _create_auto_task("test-auto-publish-waiting") clip_path = _fake_video("publish_waiting_clip.mp4") @@ -333,6 +517,7 @@ def test_create_auto_publish_job_without_schedule_waits_for_send_center(): "scheduled_at": "", } ], + subtitle_delivery_mode="original", ) assert result["created_count"] == 1 with get_connection() as connection: @@ -400,7 +585,7 @@ def test_prepare_source_uses_pathlib_and_writes_reference(): assert reference_path.exists() -def test_auto_selection_uses_candidate_count_and_task_max_duration(): +def test_auto_selection_uses_candidate_count_and_task_max_duration(monkeypatch): task_id = "test-auto-selection-rules" video = _fake_video(f"{task_id}.mp4") payload = TaskCreate( @@ -415,7 +600,7 @@ def test_auto_selection_uses_candidate_count_and_task_max_duration(): auto_min_clip_seconds=120, auto_max_clip_seconds=7200, ) - create_task_record(payload, task_id=task_id) + create_task_record(payload, task_id=task_id, task_dir_name=task_id) now = datetime.now(timezone.utc).isoformat() with get_connection() as connection: for clip_id, start_time, end_time, confidence in [ @@ -436,6 +621,23 @@ def test_auto_selection_uses_candidate_count_and_task_max_duration(): ) connection.commit() + monkeypatch.setattr( + "app.services.pipeline_engine.task_service.get_task_ai_analysis_meta", + lambda _task_id: { + "schema_version": 2, + "selection_profile": "general", + "analysis_incomplete": False, + "quality_degraded": False, + "coverage_ratio": 1.0, + "coverage_percent": 100.0, + "expected_units": 1, + "completed_units": 1, + "failed_units": 0, + "failed_stages": [], + "invalid_item_count": 0, + }, + ) + result = PipelineEngine()._select_clips(task_id, {"config": {}}) assert result["target_count"] == 2 @@ -480,7 +682,7 @@ def test_live_status_endpoint_tracks_running_auto_pipeline(): assert payload["is_running"] is True assert payload["should_poll"] is True assert payload["actions"]["primary"] == "processing" - assert len(payload["workflow_steps"]) == 10 + assert len(payload["workflow_steps"]) == 11 assert payload["workflow_steps"][3]["state"] == "current" assert any("实时状态测试日志" in line for line in payload["log_lines"]) @@ -518,6 +720,100 @@ def test_live_status_endpoint_returns_completed_actions(monkeypatch): assert all(step["state"] == "done" for step in payload["workflow_steps"]) +def test_live_status_endpoint_tracks_manual_workflow_job(): + task_id = "test-auto-manual-live-job" + create_task_record( + TaskCreate(task_name="手动任务实时进度", selection_profile="general"), + task_id=task_id, + ) + update_task_status(task_id, TaskStatus.ai_analyzing) + job = job_service.create_job(task_id, job_service.JOB_TYPE_AI_ANALYSIS) + claimed = job_service.claim_job(job["id"], "live-status-test-worker") + job_service.update_job_progress( + job["id"], + 61, + "正在分析第 14/23 段", + lease_owner="live-status-test-worker", + lease_token=claimed["lease_token"], + ) + + with TestClient(app) as client: + response = client.get(f"/api/tasks/{task_id}/live-status", headers=_headers()) + + assert response.status_code == 200 + payload = response.json() + assert payload["should_poll"] is True + assert payload["active_operation"]["kind"] == job_service.JOB_TYPE_AI_ANALYSIS + assert payload["active_operation"]["progress"] == 61 + assert payload["active_operation"]["message"] == "正在分析第 14/23 段" + assert payload["workflow_steps"][3]["state"] == "current" + + +def test_live_status_endpoint_maps_lowercase_status_inside_auto_pipeline(): + task = _create_auto_task("test-auto-live-lowercase") + update_task_status(task["id"], TaskStatus.transcribing) + + with TestClient(app) as client: + response = client.get(f"/api/tasks/{task['id']}/live-status", headers=_headers()) + + assert response.status_code == 200 + payload = response.json() + assert payload["should_poll"] is True + assert payload["workflow_steps"][2]["state"] == "current" + assert payload["workflow_steps"][0]["state"] == "done" + + +def test_live_status_endpoint_combines_publish_progress_with_task_timeline(): + task = _create_auto_task("test-auto-live-publish") + update_task_status(task["id"], TaskStatus.COMPLETED) + now = datetime.now(timezone.utc).isoformat() + statuses = ("PUBLISHED", "SCHEDULED", "NEED_REVIEW") + with get_connection() as connection: + for index, status in enumerate(statuses, start=1): + output_id = f"{task['id']}-output-{index}" + connection.execute( + """ + INSERT INTO output_clip ( + id, task_id, output_file_path, output_file_name, + status, is_active, created_at, updated_at + ) VALUES (?, ?, ?, ?, 'completed', 1, ?, ?) + """, + (output_id, task["id"], f"clip-{index}.mp4", f"clip-{index}.mp4", now, now), + ) + connection.execute( + """ + INSERT INTO publish_jobs ( + id, task_id, output_clip_id, platform, publish_mode, + status, created_at, updated_at + ) VALUES (?, ?, ?, 'douyin', 'local_browser', ?, ?, ?) + """, + (f"{task['id']}-publish-{index}", task["id"], output_id, status, now, now), + ) + connection.commit() + + with TestClient(app) as client: + response = client.get(f"/api/tasks/{task['id']}/live-status", headers=_headers()) + + assert response.status_code == 200 + payload = response.json() + assert payload["status"] == TaskStatus.COMPLETED.value + assert payload["task_progress"] == 100 + assert payload["progress"] == 93 + assert payload["status_label"] == "发布需处理 · 1 条" + assert payload["publish"]["total"] == 3 + assert payload["publish"]["success"] == 1 + assert payload["publish"]["pending"] == 1 + assert payload["publish"]["need_review"] == 1 + assert payload["active_operation"]["kind"] == "publish" + assert payload["active_operation"]["progress"] == 33 + assert payload["workflow_steps"][-1] == { + "name": "平台发布", + "index": "11", + "state": "warning", + } + assert payload["should_poll"] is True + + def test_live_status_endpoint_returns_404_for_missing_task(): with TestClient(app) as client: response = client.get("/api/tasks/test-auto-missing/live-status", headers=_headers()) @@ -535,8 +831,17 @@ def test_task_detail_live_status_frontend_uses_partial_refresh(): ] assert "data-task-live-overview" in template + assert "data-task-live-operation" in template assert "data-live-task-actions" in template assert "/live-status" in live_script assert "TASK_LIVE_STATUS_INTERVAL_MS = 3000" in script assert 'document.addEventListener("visibilitychange"' in live_script + assert "taskLiveStatusRequestInFlight" in live_script + assert "renderRuntimeLog(data);" in live_script assert "window.location.reload" not in live_script + + ai_poll_script = script[ + script.index("async function pollAiAnalysisStatus"): + script.index("function stopAiAnalysisStatusPolling") + ] + assert "renderRuntimeLog" not in ai_poll_script diff --git a/tests/test_clip_review_publish_sync.py b/tests/test_clip_review_publish_sync.py index a4cdc36..359197a 100644 --- a/tests/test_clip_review_publish_sync.py +++ b/tests/test_clip_review_publish_sync.py @@ -3,7 +3,7 @@ from unittest.mock import Mock from app.main import app -from app.models.task import ClipCandidateBatchItem +from app.models.task import ClipCandidateBatchItem, TaskStatus from app.services import task_service @@ -70,3 +70,68 @@ def test_review_sync_reuses_matching_outputs_when_review_is_unchanged(monkeypatc def test_review_sync_route_is_available() -> None: assert "/api/tasks/{task_id}/clips/sync-publish" in app.openapi()["paths"] + + +def test_unchanged_review_marks_pending_review_completed_after_full_link(monkeypatch) -> None: + monkeypatch.setattr( + task_service, + "update_clip_candidates_batch", + Mock(return_value={"changed_count": 0, "message": "已保存", "clips": [], "task": {}}), + ) + monkeypatch.setattr(task_service, "_active_outputs_match_enabled_candidates", lambda _task_id: True) + monkeypatch.setattr( + "app.services.publish_service.sync_task_publish_jobs", + Mock( + return_value={ + "status": "ok", + "message": "发送中心同步完成。", + "errors": [], + "link_state": {"state": "linked", "linked_count": 1, "missing_count": 0}, + } + ), + ) + monkeypatch.setattr( + task_service, + "get_task", + lambda *_args, **_kwargs: {"status": TaskStatus.pending_review.value}, + ) + transition = Mock() + monkeypatch.setattr(task_service, "transition_task_status", transition) + + task_service.sync_reviewed_clips_to_publish_center("task-review-sync", _payload()) + + transition.assert_called_once_with("task-review-sync", TaskStatus.completed) + + +def test_partial_publish_sync_keeps_review_state_after_recut(monkeypatch) -> None: + monkeypatch.setattr( + task_service, + "update_clip_candidates_batch", + Mock(return_value={"changed_count": 1, "message": "已保存", "clips": [], "task": {}}), + ) + monkeypatch.setattr(task_service, "_active_outputs_match_enabled_candidates", lambda _task_id: False) + monkeypatch.setattr( + task_service, + "process_task_video_cuts", + Mock( + return_value={ + "publish_sync": { + "status": "partial", + "message": "关联不完整", + "errors": ["同步失败"], + "link_state": {"state": "partial", "missing_count": 1}, + } + } + ), + ) + monkeypatch.setattr( + task_service, + "get_task", + lambda *_args, **_kwargs: {"status": TaskStatus.completed.value}, + ) + update = Mock() + monkeypatch.setattr(task_service, "update_task_status", update) + + task_service.sync_reviewed_clips_to_publish_center("task-review-sync", _payload()) + + update.assert_called_once_with("task-review-sync", TaskStatus.pending_review) diff --git a/tests/test_cut_atomicity.py b/tests/test_cut_atomicity.py new file mode 100644 index 0000000..0b38c07 --- /dev/null +++ b/tests/test_cut_atomicity.py @@ -0,0 +1,371 @@ +"""P1.3:切片批次编号、结果提交和版本激活原子性。""" + +from __future__ import annotations + +from concurrent.futures import ThreadPoolExecutor +from pathlib import Path + +import pytest + +from app.core.config import settings +from app.db.database import get_connection +from app.models.task import TaskCreate, TaskStatus +from app.services.task_lifecycle_service import create_task_record +from app.services.video_cut_service import CutResult + + +@pytest.fixture(autouse=True) +def cleanup_cut_data(): + yield + with get_connection() as connection: + connection.execute("DELETE FROM workflow_jobs WHERE task_id LIKE 'test-atomic-%'") + connection.execute("DELETE FROM output_clip WHERE task_id LIKE 'test-atomic-%'") + connection.execute("DELETE FROM cut_runs WHERE task_id LIKE 'test-atomic-%'") + connection.execute("DELETE FROM clip_candidates WHERE task_id LIKE 'test-atomic-%'") + connection.execute("DELETE FROM tasks WHERE id LIKE 'test-atomic-%'") + connection.commit() + + +def _create_task(task_id: str) -> None: + create_task_record( + TaskCreate( + task_name=task_id, + source_type="upload", + platform="general", + selection_profile="general", + ), + task_id=task_id, + task_dir_name=task_id, + ) + + +def _insert_candidate(task_id: str, clip_id: str) -> None: + from app.services.task_service import _now_iso + + now = _now_iso() + with get_connection() as connection: + connection.execute( + """ + INSERT INTO clip_candidates ( + id, task_id, clip_key, title, start_time, end_time, duration_seconds, + summary, reason, highlight_reason, spread_value, suggested_editing, + confidence_score, selected_by_default, enabled, reviewed, created_at, updated_at + ) VALUES (?, ?, ?, ?, '00:00:01', '00:00:03', 2, '', '', '', '', '', 0.8, 1, 1, 0, ?, ?) + """, + (clip_id, task_id, clip_id, clip_id, now, now), + ) + connection.commit() + + +def _result(clip_id: str, filename: str) -> CutResult: + return CutResult(clip_id, f"C:/tmp/{filename}", filename, "completed") + + +def test_concurrent_cut_run_numbers_are_unique(): + from app.services.video_cut_workflow_service import _create_cut_run + + task_id = "test-atomic-run-number" + _create_task(task_id) + + with ThreadPoolExecutor(max_workers=4) as executor: + runs = list(executor.map(lambda _index: _create_cut_run(task_id), range(8))) + + numbers = sorted(run["run_number"] for run in runs) + assert numbers == list(range(1, 9)) + + +def test_cut_result_batch_rolls_back_on_second_insert(monkeypatch): + import app.services.video_cut_workflow_service as workflow + + task_id = "test-atomic-rollback" + _create_task(task_id) + _insert_candidate(task_id, "test-atomic-clip-1") + _insert_candidate(task_id, "test-atomic-clip-2") + run = workflow._create_cut_run(task_id) + original_insert = workflow._insert_output_clip_record + calls = 0 + + def fail_second_insert(*args, **kwargs): + nonlocal calls + calls += 1 + if calls == 2: + raise RuntimeError("模拟第二条写入失败") + return original_insert(*args, **kwargs) + + monkeypatch.setattr(workflow, "_insert_output_clip_record", fail_second_insert) + + with pytest.raises(RuntimeError, match="第二条写入失败"): + workflow._commit_cut_run_results( + task_id, + run["id"], + [_result("test-atomic-clip-1", "one.mp4"), _result("test-atomic-clip-2", "two.mp4")], + source_fingerprint="fingerprint", + ) + + with get_connection() as connection: + count = connection.execute( + "SELECT COUNT(*) AS total FROM output_clip WHERE cut_run_id = ?", (run["id"],) + ).fetchone()["total"] + run_row = connection.execute("SELECT status, is_active FROM cut_runs WHERE id = ?", (run["id"],)).fetchone() + assert count == 0 + assert dict(run_row) == {"status": "failed", "is_active": 0} + + +def test_older_run_cannot_replace_newer_completed_run(): + from app.services.video_cut_workflow_service import _commit_cut_run_results, _create_cut_run + + task_id = "test-atomic-order" + _create_task(task_id) + _insert_candidate(task_id, "test-atomic-order-clip") + older = _create_cut_run(task_id) + newer = _create_cut_run(task_id) + newest_failed = _create_cut_run(task_id) + + newer_commit = _commit_cut_run_results( + task_id, + newer["id"], + [_result("test-atomic-order-clip", "newer.mp4")], + source_fingerprint="newer", + ) + _commit_cut_run_results( + task_id, + newest_failed["id"], + [ + CutResult( + "test-atomic-order-clip", + "C:/tmp/failed.mp4", + "failed.mp4", + "failed", + "模拟失败", + ) + ], + source_fingerprint="failed", + error_message="模拟失败", + ) + older_commit = _commit_cut_run_results( + task_id, + older["id"], + [_result("test-atomic-order-clip", "older.mp4")], + source_fingerprint="older", + ) + + assert newer_commit["activated"] is True + assert older_commit["activated"] is False + with get_connection() as connection: + active = connection.execute( + "SELECT id FROM cut_runs WHERE task_id = ? AND is_active = 1", (task_id,) + ).fetchall() + assert [row["id"] for row in active] == [newer["id"]] + + +def test_older_run_cannot_finalize_task_after_newer_run_starts(): + from app.services.task_service import get_task + from app.services.video_cut_workflow_service import _commit_cut_run_results, _create_cut_run + + task_id = "test-atomic-task-finalize-order" + _create_task(task_id) + _insert_candidate(task_id, "test-atomic-task-finalize-clip") + older = _create_cut_run(task_id) + newer = _create_cut_run(task_id) + + older_commit = _commit_cut_run_results( + task_id, + older["id"], + [_result("test-atomic-task-finalize-clip", "older-finished.mp4")], + source_fingerprint="older-finished", + ) + + assert older_commit["task_finalized"] is False + assert get_task(task_id, include_video_probe=False)["status"] == TaskStatus.cutting.value + + newer_commit = _commit_cut_run_results( + task_id, + newer["id"], + [_result("test-atomic-task-finalize-clip", "newer-finished.mp4")], + source_fingerprint="newer-finished", + ) + + assert newer_commit["task_finalized"] is True + assert get_task(task_id, include_video_probe=False)["status"] == TaskStatus.completed.value + + +def test_cut_commit_cannot_overwrite_task_status_changed_during_run(): + from app.services.task_lifecycle_service import update_task_status + from app.services.task_service import get_task + from app.services.video_cut_workflow_service import _commit_cut_run_results, _create_cut_run + + task_id = "test-atomic-task-status-cas" + _create_task(task_id) + _insert_candidate(task_id, "test-atomic-task-status-cas-clip") + run = _create_cut_run(task_id) + update_task_status(task_id, TaskStatus.failed, "外部流程已更新状态") + + committed = _commit_cut_run_results( + task_id, + run["id"], + [_result("test-atomic-task-status-cas-clip", "status-cas.mp4")], + source_fingerprint="status-cas", + ) + + assert committed["task_finalized"] is False + task = get_task(task_id, include_video_probe=False) + assert task["status"] == TaskStatus.failed.value + assert task["last_error"] == "外部流程已更新状态" + + +def test_expired_worker_cannot_create_cut_run_or_write_cutting_state(): + from app.services import job_service + from app.services.task_service import get_task + from app.services.video_cut_workflow_service import _create_cut_run + + task_id = "test-atomic-expired-create" + _create_task(task_id) + job, _created = job_service.create_or_get_active_job( + task_id=task_id, + job_type=job_service.JOB_TYPE_VIDEO_CUT, + ) + claimed = job_service.claim_job(job["id"], "expired-cut-owner") + assert claimed is not None + with get_connection() as connection: + connection.execute( + "UPDATE workflow_jobs SET lease_expires_at = '2000-01-01T00:00:00+00:00' WHERE id = ?", + (job["id"],), + ) + connection.commit() + + with job_service.job_lease_context(job["id"], "expired-cut-owner", claimed["lease_token"]): + with pytest.raises(job_service.JobLeaseLostError): + _create_cut_run(task_id) + + assert get_task(task_id, include_video_probe=False)["status"] == "pending_video" + + +def test_duplicate_commit_does_not_downgrade_completed_run(): + from app.services.video_cut_workflow_service import _commit_cut_run_results, _create_cut_run + + task_id = "test-atomic-idempotent" + _create_task(task_id) + _insert_candidate(task_id, "test-atomic-idempotent-clip") + run = _create_cut_run(task_id) + results = [_result("test-atomic-idempotent-clip", "once.mp4")] + + first = _commit_cut_run_results(task_id, run["id"], results, source_fingerprint="same") + second = _commit_cut_run_results(task_id, run["id"], results, source_fingerprint="same") + + assert first["already_committed"] is False + assert second["already_committed"] is True + with get_connection() as connection: + run_row = connection.execute("SELECT status FROM cut_runs WHERE id = ?", (run["id"],)).fetchone() + count = connection.execute( + "SELECT COUNT(*) AS total FROM output_clip WHERE cut_run_id = ?", (run["id"],) + ).fetchone()["total"] + assert run_row["status"] == "completed" + assert count == 1 + + +def test_partial_success_is_preserved_in_cut_run_status(): + from app.services.video_cut_workflow_service import _commit_cut_run_results, _create_cut_run + + task_id = "test-atomic-partial" + _create_task(task_id) + _insert_candidate(task_id, "test-atomic-partial-ok") + _insert_candidate(task_id, "test-atomic-partial-failed") + run = _create_cut_run(task_id) + + committed = _commit_cut_run_results( + task_id, + run["id"], + [ + _result("test-atomic-partial-ok", "ok.mp4"), + CutResult( + "test-atomic-partial-failed", + "C:/tmp/failed.mp4", + "failed.mp4", + "failed", + "模拟失败", + ), + ], + source_fingerprint="partial", + error_message="部分失败", + ) + + assert committed["activated"] is True + with get_connection() as connection: + run_row = connection.execute( + "SELECT status, error_message FROM cut_runs WHERE id = ?", (run["id"],) + ).fetchone() + assert dict(run_row) == {"status": "completed_with_errors", "error_message": "部分失败"} + + +def test_each_cut_run_uses_a_distinct_output_directory(tmp_path): + from app.services.video_cut_workflow_service import _cut_run_output_dir + + first = {"id": "abc123", "run_number": 1} + second = {"id": "def456", "run_number": 2} + + assert _cut_run_output_dir(tmp_path, first) != _cut_run_output_dir(tmp_path, second) + assert _cut_run_output_dir(tmp_path, first).parent == tmp_path + + +def test_process_cut_commits_batch_and_returns_run_directory(monkeypatch): + import app.services.task_service as task_service + import app.services.video_cut_workflow_service as workflow + from app.services import ai_analysis_workflow_service + + task_id = "test-atomic-process" + _create_task(task_id) + _insert_candidate(task_id, "test-atomic-process-clip") + source = settings.tasks_dir / "_test_inputs" / "test-atomic-process.mp4" + source.parent.mkdir(parents=True, exist_ok=True) + source.write_bytes(b"fake-video") + with get_connection() as connection: + connection.execute( + "UPDATE tasks SET original_video_path = ?, source_type = 'upload' WHERE id = ?", + (str(source), task_id), + ) + connection.commit() + + original_get_task = task_service.get_task + monkeypatch.setattr( + task_service, + "get_task", + lambda value, include_video_probe=True: original_get_task(value, include_video_probe=False), + ) + monkeypatch.setattr( + ai_analysis_workflow_service, + "get_task_ai_analysis_meta", + lambda _task_id: { + "schema_version": 2, + "selection_profile": "general", + "analysis_incomplete": False, + "quality_degraded": False, + "coverage_ratio": 1.0, + "coverage_percent": 100.0, + "expected_units": 1, + "completed_units": 1, + "failed_units": 0, + "failed_stages": [], + "invalid_item_count": 0, + }, + ) + + def fake_cut_clips(*, source_video, clips, output_dir, strategy): + del source_video, clips, strategy + output_dir.mkdir(parents=True, exist_ok=True) + output = output_dir / "clip.mp4" + output.write_bytes(b"clip") + return [CutResult("test-atomic-process-clip", str(output), output.name, "completed")] + + monkeypatch.setattr(workflow, "cut_clips", fake_cut_clips) + + result = workflow.process_task_video_cuts(task_id, sync_publish_jobs=False) + + assert Path(result["output_dir"]).name.startswith("run_0001_") + with get_connection() as connection: + rows = connection.execute( + "SELECT cut_run_id, is_active FROM output_clip WHERE task_id = ?", + (task_id,), + ).fetchall() + assert len(rows) == 1 + assert rows[0]["cut_run_id"] == result["cut_run_id"] + assert rows[0]["is_active"] == 1 diff --git a/tests/test_database_backup_service.py b/tests/test_database_backup_service.py index 80f52f6..3fa63f0 100644 --- a/tests/test_database_backup_service.py +++ b/tests/test_database_backup_service.py @@ -17,6 +17,7 @@ build_cleanup_plan, create_media_cleanup_backup, create_publish_migration_backup, + create_schema_migration_backup, sqlite_quick_check, ) @@ -32,6 +33,14 @@ def _create_database(path: Path, value: str = "ok") -> None: connection.close() +def _assert_portable_backup(path: Path) -> None: + with sqlite3.connect(path) as connection: + journal_mode = connection.execute("PRAGMA journal_mode").fetchone()[0] + assert str(journal_mode).lower() == "delete" + for suffix in ("-wal", "-shm", "-journal"): + assert not Path(f"{path}{suffix}").exists() + + def _set_local_time(path: Path, value: datetime) -> None: timestamp = value.timestamp() os.utime(path, (timestamp, timestamp)) @@ -106,6 +115,22 @@ def test_repeated_backup_within_24_hours_creates_only_one_file(tmp_path): assert second is None assert backups == [first] assert sqlite_quick_check(first) == "ok" + _assert_portable_backup(first) + + +def test_schema_migration_backup_is_portable(tmp_path): + database_path = tmp_path / "workflow.sqlite3" + backup_dir = tmp_path / "backups" + _create_database(database_path) + + backup = create_schema_migration_backup( + database_path, + backup_dir, + "schema-test", + ) + + assert sqlite_quick_check(backup) == "ok" + _assert_portable_backup(backup) def test_media_cleanup_backup_is_always_created_and_valid(tmp_path): @@ -117,6 +142,7 @@ def test_media_cleanup_backup_is_always_created_and_valid(tmp_path): assert backup.name.startswith("workflow-before-media-cleanup-") assert sqlite_quick_check(backup) == "ok" + _assert_portable_backup(backup) def test_concurrent_publish_migration_creates_one_valid_backup(monkeypatch, tmp_path): diff --git a/tests/test_job_fencing.py b/tests/test_job_fencing.py new file mode 100644 index 0000000..9c7b005 --- /dev/null +++ b/tests/test_job_fencing.py @@ -0,0 +1,476 @@ +"""Workflow Job 租约代际隔离回归测试。""" + +from __future__ import annotations + +import sqlite3 +from concurrent.futures import ThreadPoolExecutor +from datetime import datetime, timezone +from uuid import uuid4 + +import pytest + +from app.db.database import get_connection, init_db +from app.services import job_service, job_worker +from app.services.managed_process_service import ProcessTerminationError + + +def _create_task_and_job(*, max_attempts: int = 3) -> tuple[str, dict]: + init_db() + task_id = f"fencing-{uuid4().hex}" + now = datetime.now(timezone.utc).isoformat(timespec="seconds") + with get_connection() as connection: + connection.execute( + """ + INSERT INTO tasks ( + id, task_name, task_dir_name, source_type, platform, selection_profile, + status, progress, is_deleted, created_at, updated_at + ) VALUES (?, 'fencing', ?, 'upload', 'general', 'general', + 'pending_video', 0, 0, ?, ?) + """, + (task_id, task_id, now, now), + ) + connection.commit() + job = job_service.create_job(task_id, job_service.JOB_TYPE_VIDEO_CUT) + if max_attempts != 3: + with get_connection() as connection: + connection.execute( + "UPDATE workflow_jobs SET max_attempts = ? WHERE id = ?", + (max_attempts, job["id"]), + ) + connection.commit() + return task_id, job + + +def _expire(job_id: str) -> None: + with get_connection() as connection: + connection.execute( + "UPDATE workflow_jobs SET lease_expires_at = '2000-01-01T00:00:00+00:00' WHERE id = ?", + (job_id,), + ) + connection.commit() + + +def _cleanup(task_id: str) -> None: + with get_connection() as connection: + connection.execute("DELETE FROM workflow_jobs WHERE task_id = ?", (task_id,)) + connection.execute("DELETE FROM tasks WHERE id = ?", (task_id,)) + connection.commit() + + +def test_stale_worker_cannot_overwrite_new_claim() -> None: + task_id, created = _create_task_and_job() + try: + first = job_service.claim_job(created["id"], "worker-a") + assert first and first["lease_token"] + _expire(created["id"]) + second = job_service.claim_job(created["id"], "worker-b") + assert second and second["lease_token"] != first["lease_token"] + + stale_operations = ( + lambda: job_service.update_job_progress( + created["id"], 90, "stale", lease_owner="worker-a", lease_token=first["lease_token"] + ), + lambda: job_service.update_job_checkpoint( + created["id"], {"stale": True}, lease_owner="worker-a", lease_token=first["lease_token"] + ), + lambda: job_service.heartbeat_job(created["id"], "worker-a", first["lease_token"]), + lambda: job_service.mark_job_completed( + created["id"], lease_owner="worker-a", lease_token=first["lease_token"] + ), + lambda: job_service.mark_job_failed( + created["id"], "stale", lease_owner="worker-a", lease_token=first["lease_token"] + ), + lambda: job_service.mark_job_cancelled( + created["id"], lease_owner="worker-a", lease_token=first["lease_token"] + ), + ) + for operation in stale_operations: + with pytest.raises(job_service.JobLeaseLostError): + operation() + assert job_service.release_job_lease(created["id"], "worker-a", first["lease_token"]) is False + + current = job_service.get_job(created["id"]) + assert current["status"] == job_service.JOB_STATUS_RUNNING + assert current["lease_owner"] == "worker-b" + assert current["lease_token"] == second["lease_token"] + assert current["progress"] == 10 + assert current["checkpoint_json"] == {} + + job_service.update_job_progress( + created["id"], 35, "current", lease_owner="worker-b", lease_token=second["lease_token"] + ) + assert job_service.heartbeat_job(created["id"], "worker-b", second["lease_token"]) + finally: + _cleanup(task_id) + + +def test_expired_lease_cannot_heartbeat_or_start_claimed_subprocess(monkeypatch) -> None: + task_id, created = _create_task_and_job() + called = False + try: + claimed = job_service.claim_job(created["id"], "worker-a") + _expire(created["id"]) + with pytest.raises(job_service.JobLeaseLostError): + job_service.heartbeat_job(created["id"], "worker-a", claimed["lease_token"]) + + def forbidden_handler(_task_id: str): + nonlocal called + called = True + return {} + + monkeypatch.setattr(job_worker.task_service, "process_task_video_cuts", forbidden_handler) + with pytest.raises(job_service.JobLeaseLostError): + job_worker.execute_job( + created["id"], + lease_owner="worker-a", + lease_token=claimed["lease_token"], + already_claimed=True, + ) + assert called is False + finally: + _cleanup(task_id) + + +def test_worker_marks_job_failed_when_subprocess_cannot_start(monkeypatch) -> None: + task_id, created = _create_task_and_job() + runner = job_worker.WorkflowJobRunner() + try: + claimed = job_service.claim_job(created["id"], runner.owner) + assert claimed and claimed["lease_token"] + monkeypatch.setattr( + job_worker, + "popen_process_group", + lambda *_args, **_kwargs: (_ for _ in ()).throw(OSError("spawn denied")), + ) + + runner._run_job_subprocess(created["id"]) + + failed = job_service.get_job(created["id"]) + assert failed["status"] == job_service.JOB_STATUS_FAILED + assert "无法启动 Job 子进程:spawn denied" in failed["error_message"] + assert failed["lease_owner"] is None + assert failed["lease_token"] is None + finally: + _cleanup(task_id) + + +def test_parent_worker_failure_settles_auto_pipeline_task_state() -> None: + task_id, created = _create_task_and_job() + try: + with get_connection() as connection: + connection.execute( + "UPDATE workflow_jobs SET job_type = ? WHERE id = ?", + (job_service.JOB_TYPE_AUTO_PIPELINE, created["id"]), + ) + connection.execute( + "UPDATE tasks SET status = ? WHERE id = ?", + ("AI_ANALYZING", task_id), + ) + connection.commit() + claimed = job_service.claim_job(created["id"], "auto-parent-owner") + job_service.mark_job_failed( + created["id"], + "子进程异常退出", + lease_owner="auto-parent-owner", + lease_token=claimed["lease_token"], + ) + with get_connection() as connection: + status = connection.execute( + "SELECT status FROM tasks WHERE id = ?", + (task_id,), + ).fetchone()["status"] + assert status == "FAILED_AI_ANALYZING" + finally: + _cleanup(task_id) + + +def test_process_termination_error_is_contained_and_job_remains_recoverable(monkeypatch) -> None: + task_id, created = _create_task_and_job() + runner = job_worker.WorkflowJobRunner() + try: + claimed = job_service.claim_job(created["id"], runner.owner) + monkeypatch.setattr( + job_worker, + "terminate_process_tree", + lambda _process: (_ for _ in ()).throw(ProcessTerminationError("taskkill failed")), + ) + + assert runner._terminate_process(object(), created["id"], claimed["lease_token"], "取消任务") is False + current = job_service.get_job(created["id"]) + assert current["status"] == job_service.JOB_STATUS_RUNNING + assert current["lease_owner"] == runner.owner + assert "将继续重试" in current["message"] + finally: + _cleanup(task_id) + + +def test_claim_next_does_not_fail_live_max_attempt_worker() -> None: + active_task_id, active_created = _create_task_and_job(max_attempts=1) + queued_task_id, queued_created = _create_task_and_job() + try: + with get_connection() as connection: + connection.execute( + "UPDATE workflow_jobs SET created_at = '1900-01-01T00:00:00+00:00' WHERE id = ?", + (queued_created["id"],), + ) + connection.commit() + active = job_service.claim_job(active_created["id"], "worker-a") + assert active and active["attempt_count"] == 1 + + claimed_next = job_service.claim_next_job("worker-b") + assert claimed_next and claimed_next["id"] == queued_created["id"] + still_active = job_service.get_job(active_created["id"]) + assert still_active["status"] == job_service.JOB_STATUS_RUNNING + assert still_active["lease_owner"] == "worker-a" + finally: + _cleanup(active_task_id) + _cleanup(queued_task_id) + + +def test_expired_max_attempt_job_is_failed_before_next_claim() -> None: + task_id, created = _create_task_and_job(max_attempts=1) + try: + claimed = job_service.claim_job(created["id"], "worker-a") + assert claimed and claimed["attempt_count"] == 1 + _expire(created["id"]) + + assert job_service.claim_next_job("worker-b") is None + failed = job_service.get_job(created["id"]) + assert failed["status"] == job_service.JOB_STATUS_FAILED + assert failed["lease_owner"] is None + assert failed["lease_token"] is None + finally: + _cleanup(task_id) + + +def test_retry_invalidates_previous_lease_token() -> None: + task_id, created = _create_task_and_job() + try: + claimed = job_service.claim_job(created["id"], "worker-a") + job_service.mark_job_failed( + created["id"], + "first failure", + lease_owner="worker-a", + lease_token=claimed["lease_token"], + ) + retried = job_service.retry_job(created["id"]) + assert retried["status"] == job_service.JOB_STATUS_QUEUED + + with pytest.raises(job_service.JobLeaseLostError): + job_service.mark_job_completed( + created["id"], + lease_owner="worker-a", + lease_token=claimed["lease_token"], + ) + current = job_service.get_job(created["id"]) + assert current["status"] == job_service.JOB_STATUS_QUEUED + assert current["lease_token"] is None + finally: + _cleanup(task_id) + + +def test_workflow_job_schema_contains_lease_token() -> None: + init_db() + with get_connection() as connection: + columns = {row[1] for row in connection.execute("PRAGMA table_info(workflow_jobs)").fetchall()} + assert "lease_token" in columns + + +def test_unfenced_running_job_blocks_startup() -> None: + from app.db.database import _guard_unfenced_running_workflow_jobs + + task_id, created = _create_task_and_job() + try: + with get_connection() as connection: + connection.execute( + """ + UPDATE workflow_jobs + SET status = 'running', lease_owner = 'old-worker', lease_token = NULL + WHERE id = ? + """, + (created["id"],), + ) + connection.commit() + with pytest.raises(RuntimeError, match="未带 lease_token"): + _guard_unfenced_running_workflow_jobs(connection) + finally: + _cleanup(task_id) + + +def test_workflow_job_column_migration_is_concurrent_safe(tmp_path) -> None: + from app.db.database import _migrate_workflow_jobs_table + + database_path = tmp_path / "legacy.sqlite3" + with sqlite3.connect(database_path) as connection: + connection.execute("CREATE TABLE workflow_jobs (id TEXT PRIMARY KEY)") + connection.commit() + + def migrate() -> None: + with sqlite3.connect(database_path, timeout=10) as connection: + connection.row_factory = sqlite3.Row + connection.execute("PRAGMA busy_timeout = 5000") + _migrate_workflow_jobs_table(connection) + connection.commit() + + with ThreadPoolExecutor(max_workers=2) as executor: + list(executor.map(lambda _index: migrate(), range(2))) + + with sqlite3.connect(database_path) as connection: + columns = {row[1] for row in connection.execute("PRAGMA table_info(workflow_jobs)").fetchall()} + assert "lease_token" in columns + assert "checkpoint_json" in columns + + +def test_pipeline_lease_loss_does_not_write_failed_task_state(monkeypatch) -> None: + from app.models.task import TaskStatus + from app.services.pipeline_engine import PipelineEngine + from app.services import pipeline_engine + + state_updates: list[TaskStatus] = [] + logs: list[str] = [] + engine = PipelineEngine() + monkeypatch.setattr( + engine, + "_get_task", + lambda _task_id: {"status": TaskStatus.CREATED.value, "auto_mode": 1}, + ) + monkeypatch.setattr(pipeline_engine.task_service, "update_task_status", lambda _task_id, status, *_args: state_updates.append(status)) + monkeypatch.setattr(pipeline_engine, "append_task_log", lambda _task_id, message: logs.append(message)) + monkeypatch.setattr(pipeline_engine.job_service, "is_cancel_requested", lambda _job_id: False) + monkeypatch.setattr(pipeline_engine.job_service, "update_job_progress", lambda *_args, **_kwargs: None) + monkeypatch.setattr( + pipeline_engine.job_service, + "heartbeat_job", + lambda *_args, **_kwargs: (_ for _ in ()).throw(job_service.JobLeaseLostError("lost")), + ) + + with pytest.raises(job_service.JobLeaseLostError): + engine.run( + "task-fencing", + start_step=TaskStatus.PREPARING_SOURCE, + job_id="job-fencing", + ) + + assert state_updates == [] + assert not any("流水线失败" in message for message in logs) + + +def test_audio_lease_loss_does_not_mark_task_failed(monkeypatch, tmp_path) -> None: + from app.services import task_service, transcript_workflow_service + + source = tmp_path / "source.mp4" + audio = tmp_path / "audio.mp3" + source.write_bytes(b"source") + status_updates = [] + logs: list[str] = [] + monkeypatch.setattr(task_service, "get_task", lambda _task_id: {"id": "task-fencing"}) + monkeypatch.setattr(task_service, "update_task_status", lambda _task_id, status, *_args: status_updates.append(status)) + monkeypatch.setattr(transcript_workflow_service, "get_source_video_path", lambda _task: source) + monkeypatch.setattr(transcript_workflow_service, "validate_source_video_path", lambda _path: (True, "")) + monkeypatch.setattr(transcript_workflow_service, "get_artifact_paths", lambda _task_id: {"audio_path": audio}) + monkeypatch.setattr(transcript_workflow_service, "append_task_log", lambda _task_id, message: logs.append(message)) + monkeypatch.setattr( + transcript_workflow_service, + "run_ffmpeg_audio_extract", + lambda *_args, **_kwargs: (_ for _ in ()).throw(job_service.JobLeaseLostError("lost")), + ) + + with pytest.raises(job_service.JobLeaseLostError): + transcript_workflow_service.process_task_audio("task-fencing", job_id="job-fencing") + + assert status_updates[-1].value == "audio_extracting" + assert not any("音频提取失败" in message for message in logs) + + +def test_subtitle_lease_loss_does_not_try_fallback_encoder(monkeypatch, tmp_path) -> None: + from app.services import subtitle_workflow_service + + calls = 0 + + def lose_lease(*_args, **_kwargs): + nonlocal calls + calls += 1 + raise job_service.JobLeaseLostError("lost") + + monkeypatch.setattr(subtitle_workflow_service, "_ffmpeg_has_encoder", lambda _name: False) + monkeypatch.setattr(subtitle_workflow_service, "_run_ffmpeg_progress", lose_lease) + + with pytest.raises(job_service.JobLeaseLostError): + subtitle_workflow_service._render_with_fallback( + tmp_path / "input.mp4", + tmp_path / "subtitle.ass", + tmp_path / "output.part.mp4", + workflow_job_id="job-fencing", + duration_seconds=60, + has_audio=True, + source_audio_codec="aac", + progress_start=0, + progress_end=100, + ) + assert calls == 1 + + +def test_stale_transcription_checkpoint_cannot_overwrite_new_claim(tmp_path) -> None: + from app.services.transcript_service import TranscriptChunk, TranscriptSegment + from app.services.transcription_checkpoint_service import TranscriptionCheckpoint + + task_id, created = _create_task_and_job() + source = tmp_path / "source.mp4" + source.write_bytes(b"source") + checkpoint = TranscriptionCheckpoint( + task_id=task_id, + source_path=source, + provider="local", + model="medium", + device="cpu", + compute_type="int8", + chunk_seconds=120, + overlap_seconds=5, + ) + chunks = [TranscriptChunk(index=1, start_seconds=0, end_seconds=120)] + segments = [TranscriptSegment(0, 1, "current")] + try: + first = job_service.claim_job(created["id"], "worker-a") + with job_service.job_lease_context(created["id"], "worker-a", first["lease_token"]): + checkpoint.ensure_run(chunks) + _expire(created["id"]) + second = job_service.claim_job(created["id"], "worker-b") + + with job_service.job_lease_context(created["id"], "worker-a", first["lease_token"]): + with pytest.raises(job_service.JobLeaseLostError): + checkpoint.save_completed(1, [TranscriptSegment(0, 1, "stale")]) + + with job_service.job_lease_context(created["id"], "worker-b", second["lease_token"]): + checkpoint.save_completed(1, segments) + with get_connection() as connection: + row = connection.execute( + "SELECT status, result_json FROM transcription_chunks WHERE run_id = ? AND chunk_index = 1", + (checkpoint.run_id,), + ).fetchone() + assert row["status"] == "completed" + assert "current" in row["result_json"] + assert "stale" not in row["result_json"] + finally: + with get_connection() as connection: + connection.execute("DELETE FROM transcription_chunks WHERE task_id = ?", (task_id,)) + connection.execute("DELETE FROM transcription_runs WHERE task_id = ?", (task_id,)) + connection.commit() + _cleanup(task_id) + + +def test_transcription_provider_does_not_wrap_lease_loss(monkeypatch, tmp_path) -> None: + from app.services import transcript_service + + monkeypatch.setattr( + transcript_service, + "transcribe_audio_with_provider", + lambda *_args, **_kwargs: (_ for _ in ()).throw(job_service.JobLeaseLostError("lost")), + ) + with pytest.raises(job_service.JobLeaseLostError): + transcript_service.transcribe_audio_with_configured_provider( + tmp_path / "audio.mp3", + tmp_path, + tmp_path / "progress.json", + provider="local", + allow_fallback=True, + ) diff --git a/tests/test_job_queue.py b/tests/test_job_queue.py index 704fde1..f691033 100644 --- a/tests/test_job_queue.py +++ b/tests/test_job_queue.py @@ -33,6 +33,13 @@ def _create_test_task(prefix: str = "task") -> str: return task_id +def _lease_kwargs(job: dict) -> dict[str, str]: + return { + "lease_owner": str(job["lease_owner"]), + "lease_token": str(job["lease_token"]), + } + + class TestJobCreate: """创建 job 记录""" @@ -143,8 +150,8 @@ def test_create_or_get_active_job_allows_new_job_after_completion(self, tmp_path task_id = _create_test_task() first_job, _ = create_or_get_active_job(task_id, JOB_TYPE_VIDEO_CUT) - mark_job_running(first_job["id"]) - mark_job_completed(first_job["id"]) + running = mark_job_running(first_job["id"]) + mark_job_completed(first_job["id"], **_lease_kwargs(running)) second_job, second_created = create_or_get_active_job(task_id, JOB_TYPE_VIDEO_CUT) @@ -222,7 +229,7 @@ def test_normal_flow_queued_to_completed(self, tmp_path, monkeypatch): # 步骤 3:标记 completed result_data = {"output_count": 3, "output_dir": "/tmp/clips"} - job = mark_job_completed(job["id"], result_data) + job = mark_job_completed(job["id"], result_data, **_lease_kwargs(job)) assert job["status"] == JOB_STATUS_COMPLETED assert job["progress"] == 100 assert job["finished_at"] is not None @@ -241,10 +248,10 @@ def test_failed_flow_records_error(self, tmp_path, monkeypatch): task_id = _create_test_task() job = create_job(task_id=task_id, job_type=JOB_TYPE_VIDEO_CUT) - mark_job_running(job["id"]) + running = mark_job_running(job["id"]) error_text = "视频文件已损坏,无法读取" - job = mark_job_failed(job["id"], error_text) + job = mark_job_failed(job["id"], error_text, **_lease_kwargs(running)) assert job["status"] == JOB_STATUS_FAILED assert job["error_message"] == error_text @@ -264,19 +271,19 @@ def test_progress_update(self, tmp_path, monkeypatch): task_id = _create_test_task() job = create_job(task_id=task_id, job_type=JOB_TYPE_VIDEO_CUT) - mark_job_running(job["id"]) + running = mark_job_running(job["id"]) # 更新进度到 50 - job = update_job_progress(job["id"], 50, "正在处理第 2/4 个片段") + job = update_job_progress(job["id"], 50, "正在处理第 2/4 个片段", **_lease_kwargs(running)) assert job["progress"] == 50 assert "第 2/4" in job["message"] # 进度不应超过 100 - job = update_job_progress(job["id"], 150, "超出范围") + job = update_job_progress(job["id"], 150, "超出范围", **_lease_kwargs(running)) assert job["progress"] == 100 # 进度不应小于 0 - job = update_job_progress(job["id"], -10, "负数") + job = update_job_progress(job["id"], -10, "负数", **_lease_kwargs(running)) assert job["progress"] == 0 @@ -336,7 +343,7 @@ def test_result_json_serialization(self, tmp_path, monkeypatch): task_id = _create_test_task() job = create_job(task_id=task_id, job_type=JOB_TYPE_VIDEO_CUT) - mark_job_running(job["id"]) + running = mark_job_running(job["id"]) result = { "status": "completed", @@ -348,7 +355,7 @@ def test_result_json_serialization(self, tmp_path, monkeypatch): ], "errors": [], } - job = mark_job_completed(job["id"], result) + job = mark_job_completed(job["id"], result, **_lease_kwargs(running)) assert job["result_json"] == result assert job["result_json"]["output_count"] == 3 @@ -377,8 +384,8 @@ def test_empty_result_defaults_to_empty_dict(self, tmp_path, monkeypatch): task_id = _create_test_task() job = create_job(task_id=task_id, job_type=JOB_TYPE_VIDEO_CUT) - mark_job_running(job["id"]) - job = mark_job_completed(job["id"]) + running = mark_job_running(job["id"]) + job = mark_job_completed(job["id"], **_lease_kwargs(running)) assert job["result_json"] == {} @@ -449,8 +456,8 @@ def test_list_jobs_by_status(self, tmp_path, monkeypatch): j2 = create_job(task_id=task_id, job_type=JOB_TYPE_VIDEO_CUT) # 把 j2 完成 - mark_job_running(j2["id"]) - mark_job_completed(j2["id"]) + running = mark_job_running(j2["id"]) + mark_job_completed(j2["id"], **_lease_kwargs(running)) queued_jobs = list_jobs(task_id=task_id, status=JOB_STATUS_QUEUED) assert len(queued_jobs) == 1 diff --git a/tests/test_long_live_foundation.py b/tests/test_long_live_foundation.py new file mode 100644 index 0000000..682b63b --- /dev/null +++ b/tests/test_long_live_foundation.py @@ -0,0 +1,230 @@ +from __future__ import annotations + +import json +from datetime import datetime, timezone +from types import SimpleNamespace + +import pytest + +from app.db.database import get_connection, init_db +from app.services import job_service +from app.services.media_preflight_service import MAX_TESTED_DURATION_SECONDS, preflight_media, probe_media +from app.services.transcript_service import ( + TranscriptChunk, + TranscriptSegment, + TranscriptWord, + _segment_from_checkpoint, +) +from app.services.transcription_checkpoint_service import TranscriptionCheckpoint + + +def _ffprobe_payload( + *, + duration: float = 3600, + include_video: bool = True, + include_audio: bool = True, + filename: str = "", +) -> str: + streams = [] + if include_video: + streams.append( + { + "codec_type": "video", + "codec_name": "h264", + "width": 1920, + "height": 1080, + "avg_frame_rate": "30/1", + } + ) + if include_audio: + streams.append({"codec_type": "audio", "codec_name": "aac", "channels": 2, "sample_rate": "48000"}) + return json.dumps( + { + "streams": streams, + "format": {"duration": str(duration), "size": "1048576", "filename": filename}, + }, + ensure_ascii=False, + ) + + +def test_media_preflight_collects_streams_and_six_hour_warning(monkeypatch, tmp_path): + source = tmp_path / "long.mp4" + source.write_bytes(b"video") + + def fake_run(command, **_kwargs): + if "-show_streams" in command: + return SimpleNamespace(returncode=0, stdout=_ffprobe_payload(duration=MAX_TESTED_DURATION_SECONDS + 1), stderr="") + return SimpleNamespace(returncode=0, stdout="", stderr="") + + monkeypatch.setattr("app.services.media_preflight_service.shutil.which", lambda name: name) + monkeypatch.setattr("app.services.media_preflight_service.subprocess.run", fake_run) + monkeypatch.setattr( + "app.services.media_preflight_service.shutil.disk_usage", + lambda _path: SimpleNamespace(free=100 * 1024 ** 3), + ) + result = preflight_media(source, total_output_limit=30) + assert result.video_codec == "h264" + assert result.audio_codec == "aac" + assert result.width == 1920 + assert result.frame_rate == 30 + assert result.warnings and "超过当前 6 小时验收范围" in result.warnings[0] + + +def test_media_preflight_rejects_missing_audio(monkeypatch, tmp_path): + source = tmp_path / "silent.mp4" + source.write_bytes(b"video") + monkeypatch.setattr("app.services.media_preflight_service.shutil.which", lambda name: name) + monkeypatch.setattr( + "app.services.media_preflight_service.subprocess.run", + lambda *_args, **_kwargs: SimpleNamespace(returncode=0, stdout=_ffprobe_payload(include_audio=False), stderr=""), + ) + with pytest.raises(ValueError, match="没有音轨"): + probe_media(source) + + +def test_media_preflight_uses_utf8_for_chinese_filename(monkeypatch, tmp_path): + source = tmp_path / "康熙来了.mp4" + source.write_bytes(b"video") + calls = [] + + def fake_run(command, **kwargs): + calls.append((command, kwargs)) + if "-show_streams" in command: + return SimpleNamespace( + returncode=0, + stdout=_ffprobe_payload(duration=20, filename=str(source)), + stderr="", + ) + return SimpleNamespace(returncode=0, stdout="", stderr="") + + monkeypatch.setattr("app.services.media_preflight_service.shutil.which", lambda name: name) + monkeypatch.setattr("app.services.media_preflight_service.subprocess.run", fake_run) + monkeypatch.setattr( + "app.services.media_preflight_service.shutil.disk_usage", + lambda _path: SimpleNamespace(free=100 * 1024 ** 3), + ) + + result = preflight_media(source, total_output_limit=12) + + assert result.video_codec == "h264" + assert len(calls) == 3 + assert all(kwargs["encoding"] == "utf-8" for _command, kwargs in calls) + assert all(kwargs["errors"] == "replace" for _command, kwargs in calls) + + +def test_media_preflight_rejects_missing_video(monkeypatch, tmp_path): + source = tmp_path / "audio-only.mp4" + source.write_bytes(b"audio") + monkeypatch.setattr("app.services.media_preflight_service.shutil.which", lambda name: name) + monkeypatch.setattr( + "app.services.media_preflight_service.subprocess.run", + lambda *_args, **_kwargs: SimpleNamespace( + returncode=0, + stdout=_ffprobe_payload(include_video=False), + stderr="", + ), + ) + + with pytest.raises(ValueError, match="没有视频轨"): + probe_media(source) + + +def test_transcription_checkpoint_resumes_and_invalidates_on_source_change(tmp_path): + init_db() + task_id = "checkpoint-foundation-test" + source = tmp_path / "source.mp4" + source.write_bytes(b"source-v1") + now = datetime.now(timezone.utc).isoformat(timespec="seconds") + with get_connection() as connection: + connection.execute( + """INSERT OR REPLACE INTO tasks ( + id, task_name, task_dir_name, source_type, platform, selection_profile, + status, progress, is_deleted, created_at, updated_at + ) VALUES (?, 'checkpoint', ?, 'upload', 'general', 'long_live_talk', + 'pending_processing', 0, 0, ?, ?)""", + (task_id, task_id, now, now), + ) + connection.commit() + chunks = [TranscriptChunk(index=1, start_seconds=0, end_seconds=120)] + segment = TranscriptSegment( + 0.125, + 1.875, + "测试词级时间戳", + confidence=0.91, + words=(TranscriptWord(125, 600, "测试", 0.9),), + ) + try: + first = TranscriptionCheckpoint( + task_id=task_id, source_path=source, provider="local", model="medium", + device="cpu", compute_type="int8", chunk_seconds=120, overlap_seconds=5, + ) + first.ensure_run(chunks) + first.save_completed(1, [segment]) + + resumed = TranscriptionCheckpoint( + task_id=task_id, source_path=source, provider="local", model="medium", + device="cpu", compute_type="int8", chunk_seconds=120, overlap_seconds=5, + ) + resumed.ensure_run(chunks) + loaded = resumed.load_completed(1, _segment_from_checkpoint) + assert resumed.run_id == first.run_id + assert loaded and loaded[0].words[0].start_ms == 125 + assert loaded[0].confidence == pytest.approx(0.91) + + source.write_bytes(b"source-v2-changed") + changed = TranscriptionCheckpoint( + task_id=task_id, source_path=source, provider="local", model="medium", + device="cpu", compute_type="int8", chunk_seconds=120, overlap_seconds=5, + ) + changed.ensure_run(chunks) + assert changed.run_id != first.run_id + assert changed.load_completed(1, _segment_from_checkpoint) is None + finally: + with get_connection() as connection: + connection.execute("DELETE FROM transcription_chunks WHERE task_id = ?", (task_id,)) + connection.execute("DELETE FROM transcription_runs WHERE task_id = ?", (task_id,)) + connection.execute("DELETE FROM tasks WHERE id = ?", (task_id,)) + connection.commit() + + +def test_job_lease_takeover_cancel_and_retry(): + init_db() + task_id = "job-lease-foundation-test" + now = datetime.now(timezone.utc).isoformat(timespec="seconds") + with get_connection() as connection: + connection.execute( + """INSERT OR REPLACE INTO tasks ( + id, task_name, task_dir_name, source_type, platform, selection_profile, + status, progress, is_deleted, created_at, updated_at + ) VALUES (?, 'lease', ?, 'upload', 'general', 'general', 'pending_video', 0, 0, ?, ?)""", + (task_id, task_id, now, now), + ) + connection.commit() + try: + job = job_service.create_job(task_id, job_service.JOB_TYPE_TRANSCRIPT) + claimed = job_service.claim_job(job["id"], "worker-one") + assert claimed["status"] == job_service.JOB_STATUS_RUNNING + assert claimed["attempt_count"] == 1 + with get_connection() as connection: + connection.execute( + "UPDATE workflow_jobs SET lease_expires_at = '2000-01-01T00:00:00+00:00' WHERE id = ?", + (job["id"],), + ) + connection.commit() + reclaimed = job_service.claim_job(job["id"], "worker-two") + assert reclaimed["lease_owner"] == "worker-two" + assert reclaimed["lease_token"] != claimed["lease_token"] + assert reclaimed["attempt_count"] == 2 + assert job_service.request_job_cancel(job["id"])["cancel_requested"] == 1 + cancelled = job_service.mark_job_cancelled( + job["id"], + lease_owner="worker-two", + lease_token=reclaimed["lease_token"], + ) + assert cancelled["status"] == job_service.JOB_STATUS_CANCELLED + assert job_service.retry_job(job["id"])["status"] == job_service.JOB_STATUS_QUEUED + finally: + with get_connection() as connection: + connection.execute("DELETE FROM workflow_jobs WHERE task_id = ?", (task_id,)) + connection.execute("DELETE FROM tasks WHERE id = ?", (task_id,)) + connection.commit() diff --git a/tests/test_long_live_selection.py b/tests/test_long_live_selection.py new file mode 100644 index 0000000..4b688f6 --- /dev/null +++ b/tests/test_long_live_selection.py @@ -0,0 +1,394 @@ +from __future__ import annotations + +from collections import Counter +from datetime import datetime, timezone +import hashlib +import json +import re + +import pytest + +from app.db.database import get_connection, init_db +from app.services import job_service +from app.services.ai.ai_clip_analyzer import TranscriptRow +from app.services.ai.base import AIProviderError +from app.services.ai.long_live_talk_analyzer import ( + LongLiveAnalysisRequest, + LongLiveWindow, + _get_or_create_checkpoint, + _mark_checkpoint_running, + _parse_window_payload, + analyze_long_live_talk, + build_long_live_windows, + calculate_window_coverage, + deduplicate_long_live_moments, + list_long_live_window_checkpoints, + select_temporally_balanced_highlights, +) +from app.services.pipeline_engine import PipelineEngine +from app.services.storage_service import get_artifact_paths + + +PREFIX = "test-long-selection-" + + +@pytest.fixture(autouse=True) +def cleanup_long_live_selection_rows(): + init_db() + _cleanup() + yield + _cleanup() + + +def _cleanup() -> None: + with get_connection() as connection: + connection.execute("DELETE FROM workflow_jobs WHERE task_id LIKE ?", (f"{PREFIX}%",)) + connection.execute("DELETE FROM ai_analysis_windows WHERE task_id LIKE ?", (f"{PREFIX}%",)) + connection.execute("DELETE FROM clip_candidates WHERE task_id LIKE ?", (f"{PREFIX}%",)) + connection.execute("DELETE FROM ai_analysis_runs WHERE task_id LIKE ?", (f"{PREFIX}%",)) + connection.execute("DELETE FROM tasks WHERE id LIKE ?", (f"{PREFIX}%",)) + connection.commit() + + +def _time_text(seconds: int) -> str: + hours, remainder = divmod(seconds, 3600) + minutes, value = divmod(remainder, 60) + return f"{hours:02d}:{minutes:02d}:{value:02d}" + + +def _rows_for_duration(duration_seconds: int, step_seconds: int = 30) -> list[TranscriptRow]: + rows = [] + for start in range(0, duration_seconds, step_seconds): + end = min(duration_seconds, start + step_seconds) + rows.append( + TranscriptRow( + start_time=_time_text(start), + end_time=_time_text(end), + start_seconds=start, + end_seconds=end, + text=f"第 {start // step_seconds + 1} 句结构化直播转写", + ) + ) + return rows + + +def _write_transcript(task_id: str, duration_seconds: int) -> None: + path = get_artifact_paths(task_id)["transcript_path"] + path.parent.mkdir(parents=True, exist_ok=True) + lines = ["| 开始 | 结束 | 文本 |", "| --- | --- | --- |"] + lines.extend( + f"| {row.start_time} | {row.end_time} | {row.text} |" + for row in _rows_for_duration(duration_seconds) + ) + path.write_text("\n".join(lines), encoding="utf-8") + + +def _create_task(task_id: str) -> None: + now = datetime.now(timezone.utc).isoformat(timespec="seconds") + with get_connection() as connection: + connection.execute( + """ + INSERT INTO tasks ( + id, task_name, task_dir_name, source_type, platform, selection_profile, + highlight_density_per_hour, highlight_total_limit, + status, progress, is_deleted, created_at, updated_at + ) VALUES (?, ?, ?, 'upload', 'general', 'long_live_talk', 4, 30, + 'pending_processing', 0, 0, ?, ?) + """, + (task_id, task_id, task_id, now, now), + ) + connection.commit() + + +def _moment(start: int, *, score: float = 80, title: str = "观点") -> dict: + return { + "title": title, + "start_seconds": start, + "end_seconds": start + 60, + "key_seconds": start + 30, + "summary": f"{title}完整内容", + "highlight_reason": "有明确内容价值", + "suggested_editing": "保留完整表达", + "category": "quote_opinion", + "topic_key": title, + "score": score, + "source_window_indexes": [], + } + + +class FakeWindowProvider: + def __init__(self, failed_indexes: set[int] | None = None): + self.failed_indexes = failed_indexes or set() + self.calls: Counter[int] = Counter() + + def generate_json(self, prompt: str, retry_instruction: str | None = None) -> str: + del retry_instruction + match = re.search(r"窗口 (\d+)/(\d+),范围 (\d{2}:\d{2}:\d{2})-(\d{2}:\d{2}:\d{2})", prompt) + assert match + index = int(match.group(1)) + self.calls[index] += 1 + if index in self.failed_indexes: + raise RuntimeError(f"模拟窗口 {index} 网络失败") + start = sum( + value * factor + for value, factor in zip(map(int, match.group(3).split(":")), (3600, 60, 1), strict=True) + ) + end = sum( + value * factor + for value, factor in zip(map(int, match.group(4).split(":")), (3600, 60, 1), strict=True) + ) + clip_end = min(end, start + 60) + return json.dumps( + { + "moments": [ + { + "title": f"窗口 {index} 的观点", + "category": "quote_opinion", + "start_time": _time_text(start), + "end_time": _time_text(clip_end), + "key_time": _time_text((start + clip_end) // 2), + "topic_key": f"topic-{index}", + "summary": "完整观点", + "highlight_reason": "可独立传播", + "score": 80, + } + ] + }, + ensure_ascii=False, + ) + + +class SafeRetryWindowProvider(FakeWindowProvider): + def generate_json(self, prompt: str, retry_instruction: str | None = None) -> str: + match = re.search(r"窗口 (\d+)/(\d+),范围", prompt) + assert match + index = int(match.group(1)) + if index == 1 and self.calls[index] < 2: + self.calls[index] += 1 + raise AIProviderError( + "模拟 429", + category="rate_limited", + http_status=429, + safe_to_retry=True, + retry_after_seconds=0, + ) + return super().generate_json(prompt, retry_instruction) + + +def test_six_hour_structured_timeline_window_coverage_density_and_total_limit(): + rows = _rows_for_duration(6 * 3600) + windows = build_long_live_windows(rows) + coverage = calculate_window_coverage( + [(window.start_seconds, window.end_seconds) for window in windows], + 0, + 6 * 3600, + ) + assert 85 <= len(windows) <= 95 + assert coverage == pytest.approx(1.0) + + moments = [] + for hour in range(6): + for position in range(7): + moments.append(_moment(hour * 3600 + position * 420, score=100 - position, title=f"{hour}-{position}")) + selected = select_temporally_balanced_highlights(moments, density_per_hour=4, total_limit=30) + hour_counts = Counter(((item["start_seconds"] + item["end_seconds"]) // 2) // 3600 for item in selected) + assert len(selected) == 24 + assert set(hour_counts) == set(range(6)) + assert max(hour_counts.values()) == 4 + + +def test_cross_window_duplicate_uses_time_and_semantic_merge(): + first = _moment(100, score=82, title="创业失败后的转折") + first["source_window_indexes"] = [1] + second = _moment(125, score=91, title="创业失败以后如何翻身") + second["source_window_indexes"] = [2] + unique = deduplicate_long_live_moments([first, second, _moment(1000, title="独立知识点")]) + assert len(unique) == 2 + merged = unique[0] + assert merged["score"] == 91 + assert merged["start_seconds"] == 100 + assert merged["end_seconds"] == 185 + assert merged["source_window_indexes"] == [1, 2] + + +def test_long_live_window_schema_fails_closed_but_allows_explicit_empty_list(): + window = LongLiveWindow(1, 1, 0, 120, tuple(_rows_for_duration(120)), "text") + + with pytest.raises(Exception, match="缺少 moments"): + _parse_window_payload('{"clips": []}', window) + with pytest.raises(Exception, match="无效条目"): + _parse_window_payload( + json.dumps( + { + "moments": [ + { + "title": "有效观点", + "start_time": "00:00:10", + "end_time": "00:01:10", + "key_time": "00:00:30", + }, + "broken", + ] + } + ), + window, + ) + assert _parse_window_payload('{"moments": []}', window) == { + "moments": [], + "invalid_item_count": 0, + } + + +def test_ambiguous_window_failure_is_not_retried_after_restart(tmp_path): + task_id = f"{PREFIX}resume" + _create_task(task_id) + _write_transcript(task_id, 30 * 60) + request = LongLiveAnalysisRequest( + task_id=task_id, + transcript_path=get_artifact_paths(task_id)["transcript_path"], + provider_name="remote", + model_name="test-model", + density_per_hour=4, + total_limit=30, + ) + first_provider = FakeWindowProvider({3}) + first = analyze_long_live_talk(request, provider=first_provider, sleep_fn=lambda _seconds: None) + assert first_provider.calls[3] == 1 + assert first.meta["failed_window_count"] == 1 + checkpoints = list_long_live_window_checkpoints(task_id) + failed = [item for item in checkpoints if item["window_index"] == 3][0] + assert failed["status"] == "uncertain" + assert failed["attempt_count"] == 1 + assert "模拟窗口 3 网络失败" in failed["error_message"] + + second_provider = FakeWindowProvider() + second = analyze_long_live_talk(request, provider=second_provider, sleep_fn=lambda _seconds: None) + assert second_provider.calls == Counter() + assert second.meta["failed_window_count"] == 1 + assert second.meta["reused_window_count"] == second.meta["window_count"] - 1 + assert second.meta["analysis_incomplete"] is True + + +def test_corrupt_completed_window_checkpoint_is_not_rebilled(tmp_path): + task_id = f"{PREFIX}corrupt-completed" + _create_task(task_id) + _write_transcript(task_id, 10 * 60) + request = LongLiveAnalysisRequest( + task_id=task_id, + transcript_path=get_artifact_paths(task_id)["transcript_path"], + provider_name="remote", + model_name="test-model", + density_per_hour=4, + total_limit=30, + ) + first_provider = FakeWindowProvider() + analyze_long_live_talk(request, provider=first_provider, sleep_fn=lambda _seconds: None) + with get_connection() as connection: + checkpoint_id = connection.execute( + """ + SELECT id FROM ai_analysis_windows + WHERE task_id = ? AND status = 'completed' + ORDER BY window_index LIMIT 1 + """, + (task_id,), + ).fetchone()["id"] + invalid_result = json.dumps({"unexpected": []}, ensure_ascii=False, sort_keys=True) + invalid_checksum = hashlib.sha256(invalid_result.encode("utf-8")).hexdigest() + connection.execute( + "UPDATE ai_analysis_windows SET result_json = ?, result_checksum = ? WHERE id = ?", + (invalid_result, invalid_checksum, checkpoint_id), + ) + connection.commit() + + second_provider = FakeWindowProvider() + second = analyze_long_live_talk(request, provider=second_provider, sleep_fn=lambda _seconds: None) + + assert second_provider.calls == Counter() + assert second.meta["analysis_incomplete"] is True + corrupted = [ + item for item in list_long_live_window_checkpoints(task_id) + if item["id"] == checkpoint_id + ][0] + assert corrupted["status"] == "uncertain" + + +def test_safe_rate_limit_window_is_retried_within_bound(tmp_path): + task_id = f"{PREFIX}safe-retry" + _create_task(task_id) + _write_transcript(task_id, 10 * 60) + request = LongLiveAnalysisRequest( + task_id=task_id, + transcript_path=get_artifact_paths(task_id)["transcript_path"], + provider_name="remote", + model_name="test-model", + density_per_hour=4, + total_limit=30, + ) + provider = SafeRetryWindowProvider() + result = analyze_long_live_talk(request, provider=provider, sleep_fn=lambda _seconds: None) + assert provider.calls[1] == 3 + assert result.meta["failed_window_count"] == 0 + + +def test_stale_worker_cannot_update_long_live_checkpoint(): + task_id = f"{PREFIX}stale-worker" + _create_task(task_id) + request = LongLiveAnalysisRequest( + task_id=task_id, + transcript_path=get_artifact_paths(task_id)["transcript_path"], + provider_name="remote", + model_name="test-model", + ) + window = LongLiveWindow(1, 1, 0, 60, tuple(_rows_for_duration(60)), "text") + checkpoint = _get_or_create_checkpoint(request, "fingerprint", window) + job = job_service.create_job(task_id, job_service.JOB_TYPE_AUTO_PIPELINE) + claimed = job_service.claim_job(job["id"], "old-worker") + assert claimed + with get_connection() as connection: + connection.execute( + "UPDATE workflow_jobs SET lease_expires_at = ? WHERE id = ?", + ("2000-01-01T00:00:00+00:00", job["id"]), + ) + connection.commit() + with job_service.job_lease_context(job["id"], "old-worker", claimed["lease_token"]): + with pytest.raises(job_service.JobLeaseLostError): + _mark_checkpoint_running(checkpoint["id"]) + with get_connection() as connection: + status = connection.execute( + "SELECT status FROM ai_analysis_windows WHERE id = ?", + (checkpoint["id"],), + ).fetchone()["status"] + assert status == "queued" + + +def test_pipeline_blocks_incomplete_long_live_before_candidate_selection(): + task_id = f"{PREFIX}gate" + _create_task(task_id) + path = get_artifact_paths(task_id)["analysis_path"] + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text( + json.dumps( + { + "clips": [], + "analysis_meta": { + "selection_profile": "long_live_talk", + "analysis_incomplete": True, + "coverage_ratio": 0.72, + "coverage_percent": 72, + }, + } + ), + encoding="utf-8", + ) + with pytest.raises(ValueError, match="低于 90%"): + PipelineEngine()._select_clips(task_id, {"config": {}}) + + +def test_ai_analysis_windows_schema_is_idempotent(): + init_db() + init_db() + with get_connection() as connection: + columns = {row["name"] for row in connection.execute("PRAGMA table_info(ai_analysis_windows)")} + indexes = {row["name"] for row in connection.execute("PRAGMA index_list(ai_analysis_windows)")} + assert {"transcript_fingerprint", "window_index", "attempt_count", "result_checksum", "next_retry_at"} <= columns + assert "idx_ai_analysis_windows_resume" in indexes diff --git a/tests/test_managed_process_service.py b/tests/test_managed_process_service.py new file mode 100644 index 0000000..751fab6 --- /dev/null +++ b/tests/test_managed_process_service.py @@ -0,0 +1,102 @@ +from __future__ import annotations + +from types import SimpleNamespace +import subprocess +from unittest.mock import Mock + +import pytest + +from app.services import managed_process_service +from app.services.managed_process_service import ProcessTerminationError, terminate_process_tree + + +def _windows(monkeypatch) -> None: + monkeypatch.setattr(managed_process_service, "os", SimpleNamespace(name="nt")) + + +def test_terminate_returns_immediately_for_finished_process(monkeypatch): + process = Mock(pid=123) + process.poll.return_value = 0 + run = Mock() + monkeypatch.setattr(managed_process_service.subprocess, "run", run) + + terminate_process_tree(process) + + run.assert_not_called() + + +def test_windows_terminate_waits_for_confirmed_exit(monkeypatch): + _windows(monkeypatch) + process = Mock(pid=123) + process.poll.return_value = None + process.wait.return_value = 1 + monkeypatch.setattr( + managed_process_service.subprocess, + "run", + Mock(return_value=subprocess.CompletedProcess([], 0, stderr=b"")), + ) + + terminate_process_tree(process) + + process.wait.assert_called_once_with(timeout=5) + + +def test_windows_terminate_accepts_race_when_process_already_exited(monkeypatch): + _windows(monkeypatch) + process = Mock(pid=123) + process.poll.side_effect = [None, 0] + process.wait.return_value = 0 + monkeypatch.setattr( + managed_process_service.subprocess, + "run", + Mock(return_value=subprocess.CompletedProcess([], 128, stderr=b"not found")), + ) + + terminate_process_tree(process) + + process.wait.assert_called_once_with(timeout=5) + + +def test_windows_terminate_rejects_nonzero_taskkill_for_live_process(monkeypatch): + _windows(monkeypatch) + process = Mock(pid=123) + process.poll.side_effect = [None, None] + monkeypatch.setattr( + managed_process_service.subprocess, + "run", + Mock(return_value=subprocess.CompletedProcess([], 5, stderr=b"access denied")), + ) + + with pytest.raises(ProcessTerminationError, match="退出码 5"): + terminate_process_tree(process) + + process.wait.assert_not_called() + + +def test_windows_terminate_reports_taskkill_timeout(monkeypatch): + _windows(monkeypatch) + process = Mock(pid=123) + process.poll.side_effect = [None, None] + monkeypatch.setattr( + managed_process_service.subprocess, + "run", + Mock(side_effect=subprocess.TimeoutExpired("taskkill", 15)), + ) + + with pytest.raises(ProcessTerminationError, match="无法执行 taskkill"): + terminate_process_tree(process) + + +def test_windows_terminate_reports_process_that_does_not_exit(monkeypatch): + _windows(monkeypatch) + process = Mock(pid=123) + process.poll.return_value = None + process.wait.side_effect = subprocess.TimeoutExpired("process", 5) + monkeypatch.setattr( + managed_process_service.subprocess, + "run", + Mock(return_value=subprocess.CompletedProcess([], 0, stderr=b"")), + ) + + with pytest.raises(ProcessTerminationError, match="仍未退出"): + terminate_process_tree(process) diff --git a/tests/test_media_storage_lifecycle.py b/tests/test_media_storage_lifecycle.py index f96f270..34d7f68 100644 --- a/tests/test_media_storage_lifecycle.py +++ b/tests/test_media_storage_lifecycle.py @@ -1,8 +1,11 @@ from __future__ import annotations import os +import sqlite3 import sys import tempfile +from contextlib import contextmanager +from types import SimpleNamespace from pathlib import Path import pytest @@ -16,7 +19,7 @@ from app.services import job_service from app.services import task_lifecycle_service from app.services.storage_service import ( - StorageSafetyError, + build_task_media_cleanup_plan, configure_runtime_media_storage, save_uploaded_video, ) @@ -30,7 +33,7 @@ @pytest.fixture -def isolated_media_settings(tmp_path): +def isolated_media_settings(tmp_path, monkeypatch): original_temp = tempfile.tempdir original_env = {name: os.environ.get(name) for name in ("TEMP", "TMP")} @@ -45,6 +48,7 @@ def isolated_media_settings(tmp_path): "tasks_dir": storage_root, "upload_temp_dir": storage_root / "_临时上传", "publish_scheduler_export_dir": storage_root / "_发布包", + "max_upload_size_bytes": 4 * 1024 * 1024 * 1024, } settings_objects = [] seen_settings = set() @@ -68,6 +72,11 @@ def isolated_media_settings(tmp_path): for name, value in replacements.items(): object.__setattr__(candidate, name, value) init_db() + monkeypatch.setattr( + task_lifecycle_service, + "preflight_media", + lambda *_args, **_kwargs: SimpleNamespace(to_dict=lambda: {"warnings": []}), + ) try: yield replacements @@ -91,6 +100,7 @@ def _create_managed_task(task_id: str, task_dir: Path, *, auto_mode: bool = Fals TaskCreate( task_name=task_id, source_type="upload", + selection_profile="general", original_video_path=str(source_path), auto_mode=auto_mode, ), @@ -131,12 +141,12 @@ def capture_upload(task_id, filename, file_object, task_dir_name=None): with TestClient(app) as client: response = client.post( "/api/tasks/upload", - data={"task_name": "large-upload-e-drive", "platform": "general"}, + data={"task_name": "large-upload-e-drive", "platform": "general", "selection_profile": "general"}, files={"video_file": ("source.mp4", b"V" * (2 * 1024 * 1024), "video/mp4")}, headers=_headers(), ) - assert response.status_code == 200 + assert response.status_code == 200, response.text assert captured["rolled"] is True assert captured["tempdir"] == str(settings.upload_temp_dir.resolve()) assert tempfile.tempdir == previous_temp @@ -150,8 +160,10 @@ def capture_upload(task_id, filename, file_object, task_dir_name=None): def test_failed_upload_removes_partial_task_directory(monkeypatch, isolated_media_settings): - original_limit = settings.max_upload_size_bytes - object.__setattr__(settings, "max_upload_size_bytes", 1024) + from app.services import storage_service + + original_limit = storage_service.settings.max_upload_size_bytes + object.__setattr__(storage_service.settings, "max_upload_size_bytes", 1024) monkeypatch.setattr( tasks_router, "allocate_task_dir_name", @@ -160,12 +172,12 @@ def test_failed_upload_removes_partial_task_directory(monkeypatch, isolated_medi try: response = TestClient(app).post( "/api/tasks/upload", - data={"task_name": "failed-upload", "platform": "general"}, + data={"task_name": "failed-upload", "platform": "general", "selection_profile": "general"}, files={"video_file": ("source.mp4", b"V" * 2048, "video/mp4")}, headers=_headers(), ) finally: - object.__setattr__(settings, "max_upload_size_bytes", original_limit) + object.__setattr__(storage_service.settings, "max_upload_size_bytes", original_limit) assert response.status_code == 400 assert not (settings.tasks_dir / "failed-upload-directory").exists() @@ -211,8 +223,8 @@ def test_delete_preserves_external_source(isolated_media_settings): create_task_record( TaskCreate( task_name=task_id, - source_type="nas", - nas_file_path=str(external_source), + selection_profile="general", + original_video_path=str(external_source), ), task_id=task_id, task_dir_name=task_id, @@ -261,7 +273,7 @@ def test_delete_failure_keeps_task_visible(monkeypatch, isolated_media_settings) def fail_cleanup(_plan): raise RuntimeError("模拟文件被占用") - monkeypatch.setattr(task_lifecycle_service, "apply_task_media_cleanup_plan", fail_cleanup) + monkeypatch.setattr(task_lifecycle_service, "stage_task_media_cleanup_plan", fail_cleanup) with pytest.raises(RuntimeError, match="文件被占用"): delete_task_permanently(task_id) @@ -272,6 +284,8 @@ def fail_cleanup(_plan): def test_delete_rejects_path_traversal(isolated_media_settings): + from app.services import storage_service + now = "2026-08-02T00:00:00+00:00" with get_connection() as connection: connection.execute( @@ -283,7 +297,7 @@ def test_delete_rejects_path_traversal(isolated_media_settings): ) connection.commit() - with pytest.raises(StorageSafetyError, match="不安全路径"): + with pytest.raises(storage_service.StorageSafetyError, match="不安全路径"): delete_task_permanently("unsafe-task") @@ -343,3 +357,138 @@ def test_cleanup_aborts_before_deleting_overlapping_active_directory(isolated_me apply_report(report) assert active_dir.exists() + + +def _staged_cleanup_api(): + """返回 P0.3 约定的暂存 API;生产实现尚未接入时给出明确测试失败。""" + from app.services import storage_service + + names = ( + "stage_task_media_cleanup_plan", + "rollback_staged_task_media_cleanup", + "finalize_staged_task_media_cleanup", + ) + missing = [name for name in names if not hasattr(storage_service, name)] + if missing: + pytest.fail( + "P0.3 暂存删除 API 尚未实现:" + ", ".join(missing) + ) + return tuple(getattr(storage_service, name) for name in names) + + +def _task_cleanup_plan(task_id: str): + with get_connection() as connection: + task = connection.execute( + "SELECT * FROM tasks WHERE id = ?", + (task_id,), + ).fetchone() + assert task is not None + return build_task_media_cleanup_plan(dict(task), include_legacy=False) + + +def test_staged_cleanup_rolls_back_when_second_move_fails(monkeypatch, isolated_media_settings): + """第二个托管目录移动失败时,第一个目录必须恢复到原位置。""" + stage_cleanup, _rollback_cleanup, _finalize_cleanup = _staged_cleanup_api() + task_id = "staged-delete-second-move-failure" + task_dir = settings.tasks_dir / task_id + _create_managed_task(task_id, task_dir) + export_dir = settings.publish_scheduler_export_dir / task_id + export_dir.mkdir(parents=True) + (export_dir / "clip.mp4").write_bytes(b"export") + plan = _task_cleanup_plan(task_id) + + from app.services import storage_service + + original_move = storage_service.shutil.move + move_calls = 0 + + def fail_second_move(source, destination): + nonlocal move_calls + move_calls += 1 + if move_calls == 2: + raise OSError("模拟第二个托管目录移动失败") + return original_move(source, destination) + + monkeypatch.setattr(storage_service.shutil, "move", fail_second_move) + with pytest.raises((OSError, RuntimeError), match="移动失败"): + stage_cleanup(plan) + + assert task_dir.exists() + assert export_dir.exists() + assert (task_dir / "source" / "source.mp4").read_bytes() == b"managed-video" + assert (export_dir / "clip.mp4").read_bytes() == b"export" + + +def test_database_commit_failure_restores_staged_media(monkeypatch, isolated_media_settings): + """数据库提交失败时,删除流程不得留下文件已移走、任务仍可见的状态。""" + _staged_cleanup_api() + task_id = "staged-delete-db-commit-failure" + task_dir = settings.tasks_dir / task_id + _create_managed_task(task_id, task_dir) + export_dir = settings.publish_scheduler_export_dir / task_id + export_dir.mkdir(parents=True) + (export_dir / "clip.mp4").write_bytes(b"export") + + original_get_connection = task_lifecycle_service.get_connection + + class CommitFailingConnection: + def __init__(self, connection): + self._connection = connection + + def __getattr__(self, name): + return getattr(self._connection, name) + + def commit(self): + raise sqlite3.OperationalError("模拟数据库提交失败") + + @contextmanager + def failing_connection(): + with original_get_connection() as connection: + yield CommitFailingConnection(connection) + + monkeypatch.setattr(task_lifecycle_service, "get_connection", failing_connection) + with pytest.raises(sqlite3.OperationalError, match="提交失败"): + delete_task_permanently(task_id) + + assert task_dir.exists() + assert export_dir.exists() + with get_connection() as connection: + row = connection.execute( + "SELECT is_deleted FROM tasks WHERE id = ?", + (task_id,), + ).fetchone() + assert row["is_deleted"] == 0 + + +def test_final_cleanup_failure_returns_cleanup_pending_after_db_commit( + monkeypatch, + isolated_media_settings, +): + """数据库已提交后隔离区清理失败,应保留已删除状态并返回待清理。""" + _staged_cleanup_api() + task_id = "staged-delete-final-cleanup-failure" + task_dir = settings.tasks_dir / task_id + _create_managed_task(task_id, task_dir) + + from app.services import storage_service + + original_rmtree = storage_service.shutil.rmtree + rmtree_calls = 0 + + def fail_final_cleanup(path, *args, **kwargs): + nonlocal rmtree_calls + rmtree_calls += 1 + if rmtree_calls == 1: + raise OSError("模拟最终隔离区清理失败") + return original_rmtree(path, *args, **kwargs) + + monkeypatch.setattr(storage_service.shutil, "rmtree", fail_final_cleanup) + result = delete_task_permanently(task_id) + + assert result["status"] == "cleanup_pending" + with get_connection() as connection: + row = connection.execute( + "SELECT is_deleted FROM tasks WHERE id = ?", + (task_id,), + ).fetchone() + assert row["is_deleted"] == 1 diff --git a/tests/test_media_timeout_resilience.py b/tests/test_media_timeout_resilience.py new file mode 100644 index 0000000..cb00739 --- /dev/null +++ b/tests/test_media_timeout_resilience.py @@ -0,0 +1,110 @@ +from __future__ import annotations + +import subprocess +from pathlib import Path +from types import SimpleNamespace + +import pytest + +from app.services import media_preflight_service, subtitle_data_service, task_service, video_cut_service +from app.services.video_cut_service import CutPlan + + +def test_task_probe_timeout_returns_bounded_metadata(monkeypatch, tmp_path: Path): + source = tmp_path / "video.mp4" + source.write_bytes(b"video") + monkeypatch.setattr(task_service.shutil, "which", lambda _name: "ffprobe") + monkeypatch.setattr( + task_service.subprocess, + "run", + lambda *_args, **_kwargs: (_ for _ in ()).throw(subprocess.TimeoutExpired("ffprobe", 1)), + ) + result = task_service._probe_video(source) + assert result["duration"] == "尚未读取" + assert result["video_size"] != "读取失败" + + +def test_cut_timeout_cleans_partial_output(monkeypatch, tmp_path: Path): + source = tmp_path / "source.mp4" + source.write_bytes(b"source") + output = tmp_path / "clip.mp4" + plan = CutPlan("clip-1", "title", "0", "10", output, 10) + + def timeout(command, **_kwargs): + Path(command[-1]).write_bytes(b"partial") + raise subprocess.TimeoutExpired(command[0], 1) + + monkeypatch.setattr(video_cut_service.subprocess, "run", timeout) + result = video_cut_service.cut_single_clip("ffmpeg", source, plan) + assert result.status == "failed" + assert result.output_file_path == "" + assert "已清理" in str(result.error_message) + assert not output.exists() + + +def test_cut_nonzero_exit_cleans_partial_output(monkeypatch, tmp_path: Path): + source = tmp_path / "source.mp4" + source.write_bytes(b"source") + output = tmp_path / "clip.mp4" + plan = CutPlan("clip-1", "title", "0", "10", output, 10) + + def fail(command, **_kwargs): + Path(command[-1]).write_bytes(b"partial") + return SimpleNamespace(returncode=1, stderr="decode failed") + + monkeypatch.setattr(video_cut_service.subprocess, "run", fail) + result = video_cut_service.cut_single_clip("ffmpeg", source, plan) + assert result.status == "failed" + assert not output.exists() + + +def test_cut_atomic_replace_failure_cleans_partial_output(monkeypatch, tmp_path: Path): + source = tmp_path / "source.mp4" + source.write_bytes(b"source") + output = tmp_path / "clip.mp4" + plan = CutPlan("clip-1", "title", "0", "10", output, 10) + + def succeed(command, **_kwargs): + Path(command[-1]).write_bytes(b"complete") + return SimpleNamespace(returncode=0, stderr="") + + original_replace = Path.replace + + def fail_part_replace(path: Path, target): + if path.name.endswith(".part.mp4"): + raise OSError("rename denied") + return original_replace(path, target) + + monkeypatch.setattr(video_cut_service.subprocess, "run", succeed) + monkeypatch.setattr(Path, "replace", fail_part_replace) + result = video_cut_service.cut_single_clip("ffmpeg", source, plan) + assert result.status == "failed" + assert "切换失败" in str(result.error_message) + assert not output.exists() + assert not (tmp_path / "clip.part.mp4").exists() + + +def test_subtitle_dimension_probe_fails_closed_on_timeout(monkeypatch, tmp_path: Path): + media = tmp_path / "video.mp4" + media.write_bytes(b"video") + monkeypatch.setattr(subtitle_data_service.shutil, "which", lambda _name: "ffprobe") + monkeypatch.setattr( + subtitle_data_service.subprocess, + "run", + lambda *_args, **_kwargs: (_ for _ in ()).throw(subprocess.TimeoutExpired("ffprobe", 1)), + ) + with pytest.raises(RuntimeError, match="读取视频尺寸超过"): + subtitle_data_service._probe_media_dimensions(media) + + +def test_media_decode_sample_wraps_timeout(monkeypatch, tmp_path: Path): + media = tmp_path / "video.mp4" + media.write_bytes(b"video") + monkeypatch.setattr(media_preflight_service.shutil, "which", lambda _name: "ffmpeg") + monkeypatch.setattr( + media_preflight_service.subprocess, + "run", + lambda *_args, **_kwargs: (_ for _ in ()).throw(subprocess.TimeoutExpired("ffmpeg", 1)), + ) + with pytest.raises(ValueError, match="解码抽样超过"): + media_preflight_service._run_decode_sample(media, 0) diff --git a/tests/test_metadata_checkpoint_resilience.py b/tests/test_metadata_checkpoint_resilience.py new file mode 100644 index 0000000..9a6bec8 --- /dev/null +++ b/tests/test_metadata_checkpoint_resilience.py @@ -0,0 +1,102 @@ +from __future__ import annotations + +from pathlib import Path + +import pytest + +from app.services import pipeline_engine +from app.models.task import TaskStatus +from app.services.pipeline_engine import PipelineCheckpointError, PipelineEngine +from app.services.storage_service import get_artifact_paths + + +def test_partial_metadata_checkpoint_reuses_completed_platform(monkeypatch): + task_id = "test-metadata-partial-recovery" + output_clip = { + "id": "output-1", + "clip_candidate_id": "candidate-1", + "task_name": "测试任务", + "clip_title": "测试片段", + "clip_summary": "摘要", + "highlight_reason": "理由", + "spread_value": "中", + "suggested_editing": "保留完整表达", + "status": "completed", + "file_exists": True, + "cover_time_seconds": 1, + } + monkeypatch.setattr(pipeline_engine.task_service, "list_output_clips", lambda _task_id: [output_clip]) + monkeypatch.setattr(pipeline_engine, "platforms_for_task", lambda _task: ["douyin", "bilibili"]) + engine = PipelineEngine() + monkeypatch.setattr(engine, "_get_task", lambda _task_id: {"id": task_id, "platform": "general"}) + monkeypatch.setattr(engine, "_write_clip_metadata", lambda *_args, **_kwargs: None) + + cover_path = get_artifact_paths(task_id)["covers_dir"] / "cover.jpg" + + def generate_cover(*_args, **_kwargs): + cover_path.parent.mkdir(parents=True, exist_ok=True) + cover_path.write_bytes(b"cover") + return {"cover_file_path": str(cover_path), "cover_time_seconds": 1} + + monkeypatch.setattr(pipeline_engine, "generate_publish_cover_for_item", generate_cover) + first_calls: list[str] = [] + + class FirstGenerator: + def __init__(self, *, use_ai: bool): + assert use_ai is True + + def generate(self, _item: dict, platform: str) -> dict: + first_calls.append(platform) + if platform == "bilibili": + raise RuntimeError("模拟第二个平台中断") + return {"platform": platform, "title": platform, "risk_flags": []} + + monkeypatch.setattr(pipeline_engine, "MetadataGenerator", FirstGenerator) + with pytest.raises(RuntimeError, match="第二个平台中断"): + engine._generate_metadata(task_id, {"config": {"auto_metadata_use_ai": True}}) + assert first_calls == ["douyin", "bilibili"] + + second_calls: list[str] = [] + + class SecondGenerator: + def __init__(self, *, use_ai: bool): + assert use_ai is True + + def generate(self, _item: dict, platform: str) -> dict: + second_calls.append(platform) + return {"platform": platform, "title": platform, "risk_flags": []} + + monkeypatch.setattr(pipeline_engine, "MetadataGenerator", SecondGenerator) + result = engine._generate_metadata(task_id, {"config": {"auto_metadata_use_ai": True}}) + assert second_calls == ["bilibili"] + assert result["metadata_count"] == 2 + assert [item["metadata"]["platform"] for item in result["metadata_items"]] == ["douyin", "bilibili"] + + metadata_path = Path(result["metadata_path"]) + assert metadata_path.is_file() + + +def test_metadata_checkpoint_requires_every_output_platform_pair(monkeypatch): + engine = PipelineEngine() + task_id = "test-metadata-pair-coverage" + monkeypatch.setattr(engine, "_verify_file_evidence", lambda *_args, **_kwargs: None) + monkeypatch.setattr(engine, "_active_output_ids", lambda _task_id: ["output-1"]) + monkeypatch.setattr(engine, "_get_task", lambda _task_id: {"id": task_id}) + monkeypatch.setattr(pipeline_engine, "platforms_for_task", lambda _task: ["douyin", "bilibili"]) + monkeypatch.setattr( + engine, + "_read_json_list", + lambda _path: [ + { + "output_clip": {"id": "output-1"}, + "metadata": {"platform": "douyin", "title": "only one", "risk_flags": []}, + "cover": {}, + } + ], + ) + with pytest.raises(PipelineCheckpointError, match="active output 与平台"): + engine._restore_checkpoint_step( + task_id, + TaskStatus.METADATA_GENERATING, + {"metadata": {}, "output_clip_ids": ["output-1"]}, + ) diff --git a/tests/test_p0_security.py b/tests/test_p0_security.py index d5b6a47..f507fbe 100644 --- a/tests/test_p0_security.py +++ b/tests/test_p0_security.py @@ -2,12 +2,43 @@ import io import importlib +import os import subprocess from pathlib import Path from unittest.mock import Mock, patch import pytest + +@pytest.fixture(autouse=True) +def _restore_reloaded_runtime_modules(): + """这些用例会 reload 全局配置;结束后恢复 pytest 隔离环境,避免污染后续模块。""" + names = ( + "STORAGE_ROOT", + "TASKS_DIR", + "UPLOAD_TEMP_DIR", + "DATA_DIR", + "DATABASE_PATH", + "ALLOWED_MEDIA_ROOTS", + "MAX_UPLOAD_SIZE_BYTES", + ) + original = {name: os.environ.get(name) for name in names} + yield + for name, value in original.items(): + if value is None: + os.environ.pop(name, None) + else: + os.environ[name] = value + import app.core.config + import app.db.database + import app.services.publish_service + import app.services.storage_service + + importlib.reload(app.core.config) + importlib.reload(app.db.database) + importlib.reload(app.services.storage_service) + importlib.reload(app.services.publish_service) + # ── 辅助函数 ── @@ -45,13 +76,12 @@ def test_reject_double_dot_in_validate(self, tmp_path, monkeypatch): assert not valid assert "不安全" in msg or "跳转" in msg - def test_reject_double_dot_in_browse(self, tmp_path, monkeypatch): - """browse_video_directory 拒绝包含 .. 的路径""" + def test_directory_browse_capability_is_removed(self, tmp_path, monkeypatch): + """上传单入口下不再暴露目录浏览服务。""" _reload_storage_service(tmp_path, monkeypatch) - from app.services.storage_service import browse_video_directory + from app.services import storage_service - result = browse_video_directory("E:\\..\\Windows") - assert "不安全" in result.get("error", "") or not result.get("exists", True) + assert not hasattr(storage_service, "browse_video_directory") def test_reject_path_outside_roots(self, tmp_path, monkeypatch): """validate 拒绝不在允许根目录下的文件""" @@ -434,10 +464,10 @@ def test_cors_origin_whitelist(self): assert len(local_origins) > 0 def test_build_allow_origin_for_localhost(self): - """未知 localhost 端口也返回 Origin 本身""" + """未知 localhost 端口不能读取媒体内容。""" from app.main import _build_allow_origin_header result = _build_allow_origin_header("http://localhost:9999") - assert result == "http://localhost:9999" + assert result == "null" def test_build_allow_origin_for_unknown(self): """完全未知的 Origin 返回 'null'""" diff --git a/tests/test_p1_1_db_performance.py b/tests/test_p1_1_db_performance.py index 35c0b61..17e0510 100644 --- a/tests/test_p1_1_db_performance.py +++ b/tests/test_p1_1_db_performance.py @@ -319,7 +319,7 @@ def test_list_tasks_keeps_required_fields(self, tmp_path, monkeypatch): class TestDashboardContext: """get_dashboard_context 统计正确""" - def test_ready_for_subtitle_count(self, tmp_path, monkeypatch): + def test_completed_clip_task_count(self, tmp_path, monkeypatch): _setup_db(tmp_path, monkeypatch) from app.db.database import get_connection from app.services.task_service import get_dashboard_context @@ -333,11 +333,11 @@ def test_ready_for_subtitle_count(self, tmp_path, monkeypatch): conn.commit() ctx = get_dashboard_context() - # "待加字幕" = 已完成切片数 = 2 - ready_stat = next(s for s in ctx["stats"] if s["label"] == "待加字幕") - assert ready_stat["value"] == 2 + sliced_stat = next(s for s in ctx["stats"] if s["label"] == "已切片任务") + assert sliced_stat["value"] == 1 + assert "2 条" in sliced_stat["note"] - def test_output_clip_count_in_focus_stats(self, tmp_path, monkeypatch): + def test_pending_publish_count_in_dashboard_stats(self, tmp_path, monkeypatch): _setup_db(tmp_path, monkeypatch) from app.db.database import get_connection from app.services.task_service import get_dashboard_context @@ -349,8 +349,9 @@ def test_output_clip_count_in_focus_stats(self, tmp_path, monkeypatch): conn.commit() ctx = get_dashboard_context() - focus_output = next(s for s in ctx["focus_stats"] if s["label"] == "输出切片") - assert focus_output["value"] == 1 + pending_stat = next(s for s in ctx["stats"] if s["label"] == "待推送任务") + assert pending_stat["value"] == 1 + assert "1 条" in pending_stat["note"] class TestClipsOverviewContext: diff --git a/tests/test_p1_5_security.py b/tests/test_p1_5_security.py new file mode 100644 index 0000000..dd6ea49 --- /dev/null +++ b/tests/test_p1_5_security.py @@ -0,0 +1,435 @@ +import json +from pathlib import Path +import sqlite3 + +import pytest +from fastapi.testclient import TestClient +from pydantic import ValidationError + +from app.core.config import settings +from app.main import app +from app.models.settings import AIConfigUpdate +from app.models.task import PublishPlatformConfigUpdate, TaskStatus, TaskStatusUpdate +from app.services import ai_config_service, publish_service, task_query_service +from app.services.publish_domain import safe_platform_url, validate_platform_url +from app.services.publish_scheduler import queue_snapshot +from app.services.publishers.base import PublishOutcome, PublishResult + + +PROJECT_ROOT = Path(__file__).resolve().parents[1] +SECRET_SENTINEL = "p1-5-secret-sentinel-value" + + +def _fake_ai_values() -> dict[str, str]: + values = {key: "" for key in ai_config_service.SETTING_ATTRS} + values.update( + { + "AI_DEFAULT_PROVIDER": "remote", + "AI_PUBLISH_PROVIDER": "remote", + "AI_REQUEST_TIMEOUT_SECONDS": "120", + "AI_CODEX_PATH": "codex", + "AI_CODEX_HOME": "", + "AI_CODEX_MODEL": "gpt-5.6-sol", + "AI_CODEX_TIMEOUT_SECONDS": "300", + "TRANSCRIPTION_PROVIDER": "volcengine", + "TRANSCRIPTION_FALLBACK_PROVIDER": "", + "VOLCENGINE_ASR_API_URL": "https://example.invalid/asr", + "VOLCENGINE_ASR_RESOURCE_ID": "volc.bigasr.auc_turbo", + "VOLCENGINE_ASR_TIMEOUT_SECONDS": "300", + "VOLCENGINE_ASR_AUDIO_FORMAT": "mp3", + "AI_ANALYSIS_REMOTE_BASE_URL": "https://example.invalid/v1", + "AI_ANALYSIS_REMOTE_MODEL": "analysis-model", + "AI_ANALYSIS_REMOTE_PROTOCOL": "chat_completions", + "AI_ANALYSIS_REMOTE_REASONING_EFFORT": "", + "AI_ANALYSIS_REMOTE_RESPONSES_PATH": "/v1/responses", + "AI_ANALYSIS_REMOTE_DISABLE_RESPONSE_STORAGE": "true", + "AI_ANALYSIS_REQUEST_TIMEOUT_SECONDS": "120", + "AI_PUBLISH_REMOTE_BASE_URL": "https://example.invalid/v1", + "AI_PUBLISH_REMOTE_MODEL": "publish-model", + "AI_PUBLISH_REMOTE_PROTOCOL": "chat_completions", + "AI_PUBLISH_REMOTE_REASONING_EFFORT": "", + "AI_PUBLISH_REMOTE_RESPONSES_PATH": "/v1/responses", + "AI_PUBLISH_REMOTE_DISABLE_RESPONSE_STORAGE": "true", + "AI_PUBLISH_REQUEST_TIMEOUT_SECONDS": "120", + "AI_LOCAL_BASE_URL": "http://127.0.0.1:11434/v1", + "AI_LOCAL_MODEL": "qwen3:8b", + "AI_LOCAL_PROTOCOL": "chat_completions", + "AI_LOCAL_FALLBACK_PROTOCOL": "", + "AI_LOCAL_HEALTH_TIMEOUT_SECONDS": "30", + "AI_NETWORK_ACCESS": "enabled", + "AI_WINDOWS_WSL_SETUP_ACKNOWLEDGED": "true", + "AI_MODEL_CONTEXT_WINDOW": "1000000", + "AI_MODEL_AUTO_COMPACT_TOKEN_LIMIT": "900000", + } + ) + for key in ai_config_service.SECRET_SETTING_KEYS: + values[key] = f"{SECRET_SENTINEL}-{key.lower()}" + return values + + +def _safe_context(monkeypatch) -> dict: + monkeypatch.setattr(ai_config_service, "_current_config_values", _fake_ai_values) + monkeypatch.setattr(ai_config_service, "fetch_ollama_models", lambda **_kwargs: []) + monkeypatch.setattr( + ai_config_service.CodexCliProvider, + "version_status", + lambda _self: {"ok": True, "version": "test", "detail": "测试可用"}, + ) + return ai_config_service.get_ai_config_context() + + +def test_ai_config_context_never_returns_secret_or_env_path(monkeypatch) -> None: + context = _safe_context(monkeypatch) + serialized = json.dumps(context, ensure_ascii=False) + + assert SECRET_SENTINEL not in serialized + assert "env_path" not in context + for key in ai_config_service.SECRET_SETTING_KEYS: + assert context["values"][key] == "" + assert context["secret_configured"][key] is True + + +def test_system_page_does_not_render_admin_or_provider_secrets(monkeypatch) -> None: + context = _safe_context(monkeypatch) + monkeypatch.setattr(task_query_service, "get_ai_config_context", lambda: context) + previous_token = settings.local_admin_token + object.__setattr__(settings, "local_admin_token", f"{SECRET_SENTINEL}-admin") + try: + response = TestClient(app).get("/system") + finally: + object.__setattr__(settings, "local_admin_token", previous_token) + + assert response.status_code == 200 + assert SECRET_SENTINEL not in response.text + assert 'meta name="local-admin-token"' not in response.text + assert 'name="ai_analysis_remote_api_key" type="password" value=""' in response.text + assert "项目本地 .env" in response.text + + +def test_blank_secret_fields_keep_existing_values_without_returning_them(monkeypatch) -> None: + current = _fake_ai_values() + captured: dict[str, str] = {} + monkeypatch.setattr(ai_config_service, "_current_config_values", lambda: current) + monkeypatch.setattr(ai_config_service, "_write_env_values", lambda values: captured.update(values)) + monkeypatch.setattr(ai_config_service, "_apply_runtime_values", lambda _values: None) + monkeypatch.setattr( + ai_config_service, + "get_ai_config_context", + lambda: {"values": ai_config_service._public_config_values(current)}, + ) + + response = ai_config_service.save_ai_config(AIConfigUpdate()) + + for key in ai_config_service.SECRET_SETTING_KEYS: + assert captured[key] == current[key] + assert SECRET_SENTINEL not in json.dumps(response, ensure_ascii=False) + + +def test_publish_config_and_account_dtos_remove_raw_secrets() -> None: + config = publish_service._normalize_config( + { + "platform": "douyin", + "client_key": "public-client-key", + "client_secret": f"{SECRET_SENTINEL}-client", + } + ) + account = publish_service._normalize_account( + { + "platform": "douyin", + "login_status": "normal", + "authorization_status": "authorized", + "access_token": f"{SECRET_SENTINEL}-access", + "refresh_token": f"{SECRET_SENTINEL}-refresh", + } + ) + + assert "client_secret" not in config + assert "access_token" not in account + assert "refresh_token" not in account + assert config["client_secret_masked"] + assert account["access_token_masked"] + assert account["refresh_token_masked"] + assert SECRET_SENTINEL not in json.dumps({"config": config, "account": account}, ensure_ascii=False) + + +def test_publish_result_dto_redacts_nested_provider_secrets() -> None: + result = PublishResult( + outcome=PublishOutcome.PUBLISHED, + message=f"upstream Authorization: Bearer {SECRET_SENTINEL}-bearer", + provider_response={ + "access_token": f"{SECRET_SENTINEL}-access", + "Set-Cookie": f"{SECRET_SENTINEL}-cookie-header", + "provider_access_token": f"{SECRET_SENTINEL}-provider-access", + "nested": {"cookie": f"{SECRET_SENTINEL}-cookie", "safe": "visible"}, + }, + ) + + payload = result.as_dict() + + assert payload["provider_response"]["access_token"] == "[REDACTED]" + assert payload["provider_response"]["Set-Cookie"] == "[REDACTED]" + assert payload["provider_response"]["provider_access_token"] == "[REDACTED]" + assert payload["provider_response"]["nested"]["cookie"] == "[REDACTED]" + assert payload["provider_response"]["nested"]["safe"] == "visible" + assert "[REDACTED]" in payload["message"] + assert SECRET_SENTINEL not in json.dumps(payload, ensure_ascii=False) + + +def test_oauth_account_helper_never_returns_token_response(monkeypatch) -> None: + monkeypatch.setattr(publish_service, "_validate_and_consume_oauth_state", lambda *_args: True) + monkeypatch.setattr( + publish_service, + "_get_platform_config_record", + lambda _platform: {"platform": "douyin", "client_key": "client", "client_secret": "secret"}, + ) + monkeypatch.setattr( + publish_service.DouyinPublishProvider, + "exchange_code", + lambda _self, _code: { + "data": { + "nickname": "安全账号", + "access_token": f"{SECRET_SENTINEL}-access", + "refresh_token": f"{SECRET_SENTINEL}-refresh", + } + }, + ) + monkeypatch.setattr( + publish_service, + "create_account", + lambda payload: {"status": "ok", "account": {"account_name": payload.account_name}}, + ) + + result = publish_service.save_douyin_oauth_account("oauth-code", state="valid-state") + + assert "provider_response" not in result + assert SECRET_SENTINEL not in json.dumps(result, ensure_ascii=False) + + +def test_legacy_publish_job_dto_redacts_raw_provider_fields() -> None: + job = publish_service._normalize_job( + { + "id": "legacy-secret-job", + "platform": "douyin", + "status": "FAILED", + "provider_response": json.dumps({"token": SECRET_SENTINEL, "safe": "visible"}), + "publish_result": json.dumps({"provider_response": {"cookie": SECRET_SENTINEL}}), + } + ) + + assert "provider_response" not in job + assert "publish_result" not in job + assert job["provider_payload"]["token"] == "[REDACTED]" + assert job["provider_payload"]["safe"] == "visible" + assert job["publish_result_payload"]["provider_response"]["cookie"] == "[REDACTED]" + assert SECRET_SENTINEL not in json.dumps(job, ensure_ascii=False) + + malformed = publish_service._normalize_job( + { + "id": "legacy-malformed-secret-job", + "platform": "douyin", + "status": "FAILED", + "provider_response": f'{{"access_token":"{SECRET_SENTINEL}', + } + ) + assert malformed["provider_payload"] == {"invalid_payload": True} + assert SECRET_SENTINEL not in json.dumps(malformed, ensure_ascii=False) + + +def test_queue_snapshot_redacts_legacy_provider_fields(monkeypatch) -> None: + job_id = "test-p1-5-queue-secret" + now = "2026-08-25T00:00:00+00:00" + connection = sqlite3.connect(":memory:") + connection.row_factory = sqlite3.Row + connection.execute( + """ + CREATE TABLE publish_jobs ( + id TEXT, task_id TEXT, platform TEXT, status TEXT, + provider_response TEXT, publish_result TEXT, platform_url TEXT, + scheduled_at TEXT, created_at TEXT + ) + """ + ) + connection.execute( + """ + INSERT INTO publish_jobs ( + id, task_id, platform, status, provider_response, publish_result, created_at + ) VALUES (?, 'test-task', 'douyin', 'FAILED', ?, ?, ?) + """, + ( + job_id, + f'{{"access_token":"{SECRET_SENTINEL}', + json.dumps({"provider_response": {"cookie": SECRET_SENTINEL}}), + now, + ), + ) + connection.commit() + monkeypatch.setattr("app.services.publish_scheduler.get_connection", lambda: connection) + + try: + job = next(item for item in queue_snapshot()["all"] if item["id"] == job_id) + finally: + connection.close() + + assert "provider_response" not in job + assert "publish_result" not in job + assert job["provider_payload"] == {"invalid_payload": True} + assert job["publish_result_payload"]["provider_response"]["cookie"] == "[REDACTED]" + assert SECRET_SENTINEL not in json.dumps(job, ensure_ascii=False) + + +@pytest.mark.parametrize( + ("field", "value"), + [ + ("ai_codex_model", "safe-model\nINJECTED_CONFIG=1"), + ("ai_analysis_remote_base_url", "javascript:alert(1)"), + ("ai_publish_remote_base_url", "http://api.example.com/v1"), + ("ai_local_base_url", "http://169.254.169.254/latest/meta-data"), + ("volcengine_asr_api_url", "file:///etc/passwd"), + ("ai_analysis_remote_responses_path", "//evil.example/v1/responses"), + ], +) +def test_ai_config_rejects_unsafe_values(field: str, value: str) -> None: + with pytest.raises(ValidationError): + AIConfigUpdate(**{field: value}) + + +@pytest.mark.parametrize( + ("field", "value"), + [ + ("client_secret", "safe\nINJECTED=1"), + ("api_base_url", "http://api.example.com/v1"), + ("auth_url", "javascript:alert(1)"), + ("token_url", "https://user:pass@api.example.com/token"), + ("upload_url", "https://api.example.com/upload?token=unsafe"), + ], +) +def test_publish_platform_config_rejects_unsafe_values(field: str, value: str) -> None: + with pytest.raises(ValidationError): + PublishPlatformConfigUpdate(**{field: value}) + + +def test_task_status_error_message_has_bounded_length() -> None: + with pytest.raises(ValidationError): + TaskStatusUpdate(status=TaskStatus.failed, error_message="x" * 2001) + + +def test_validation_response_does_not_echo_rejected_secret(monkeypatch) -> None: + called = False + + def unexpected_save(_payload): + nonlocal called + called = True + return {} + + monkeypatch.setattr("app.routers.settings.save_ai_config", unexpected_save) + response = TestClient(app).post( + "/api/settings/ai", + json={"ai_analysis_remote_api_key": f"{SECRET_SENTINEL}\nINJECTED=1"}, + ) + + assert response.status_code == 422 + assert called is False + assert SECRET_SENTINEL not in response.text + assert '"input"' not in response.text + + +def test_local_gate_rejects_remote_host_and_accepts_bearer_token() -> None: + client = TestClient(app) + previous_token = settings.local_admin_token + try: + object.__setattr__(settings, "local_admin_token", "") + assert client.get("/api/tasks", headers={"Host": "lan.example"}).status_code == 403 + assert client.get("/system", headers={"Host": "lan.example"}).status_code == 403 + assert client.get("/media/tasks/missing/source-video", headers={"Host": "lan.example"}).status_code == 403 + + object.__setattr__(settings, "local_admin_token", "test-admin-token") + assert client.get("/api/tasks", headers={"Host": "lan.example"}).status_code == 401 + assert client.get( + "/api/tasks", + headers={"Host": "lan.example", "Authorization": "Bearer wrong-token"}, + ).status_code == 401 + authorized = client.get( + "/api/tasks", + headers={"Host": "lan.example", "Authorization": "Bearer test-admin-token"}, + ) + assert authorized.status_code == 200 + finally: + object.__setattr__(settings, "local_admin_token", previous_token) + + +def test_remote_peer_cannot_bypass_gate_with_loopback_host(monkeypatch) -> None: + monkeypatch.delenv("NIUMA_TRUST_DOCKER_LOOPBACK_PROXY", raising=False) + client = TestClient(app, client=("203.0.113.9", 50000)) + previous_token = settings.local_admin_token + try: + object.__setattr__(settings, "local_admin_token", "test-admin-token") + assert client.get("/api/tasks", headers={"Host": "127.0.0.1"}).status_code == 401 + assert client.get( + "/api/tasks", + headers={"Host": "127.0.0.1", "Authorization": "Bearer test-admin-token"}, + ).status_code == 200 + finally: + object.__setattr__(settings, "local_admin_token", previous_token) + + +def test_local_gate_keeps_health_public_and_blocks_cross_site_writes() -> None: + client = TestClient(app, client=("127.0.0.1", 49152)) + assert client.get("/health", headers={"Host": "lan.example"}).status_code == 200 + same_origin = client.post( + "/api/settings/ai", + headers={"Host": "127.0.0.1:49152", "Origin": "http://127.0.0.1:49152"}, + json={}, + ) + assert same_origin.status_code != 403 + response = client.post( + "/api/settings/ai", + headers={"Host": "127.0.0.1", "Origin": "https://evil.example"}, + json={}, + ) + assert response.status_code == 403 + + for external_origin in ("https://creator.douyin.com", "https://members.bilibili.com"): + response = client.post( + "/api/settings/ai", + headers={"Host": "127.0.0.1", "Origin": external_origin}, + json={}, + ) + assert response.status_code == 403 + + +def test_dynamic_frontend_text_is_not_written_with_inner_html() -> None: + app_script = (PROJECT_ROOT / "app/static/js/app.js").read_text(encoding="utf-8") + publish_script = (PROJECT_ROOT / "app/static/js/publish-center.js").read_text(encoding="utf-8") + + assert "summary.innerHTML" not in app_script + assert "summary.replaceChildren(heading, detail)" in app_script + assert "line.innerHTML" not in publish_script + assert "title.textContent = `第 ${index + 1} 条" in publish_script + assert "scheduledAt.textContent" in publish_script + + +@pytest.mark.parametrize( + ("platform", "url"), + [ + ("douyin", "javascript:alert(1)"), + ("douyin", "data:text/html,unsafe"), + ("douyin", "https://douyin.com.evil.example/video/1"), + ("bilibili", "https://evil.example/?next=bilibili.com"), + ], +) +def test_platform_url_rejects_dangerous_or_foreign_links(platform: str, url: str) -> None: + with pytest.raises(ValueError): + validate_platform_url(platform, url, allow_empty=False) + assert safe_platform_url(platform, url) == "" + + +def test_platform_url_accepts_expected_domains() -> None: + assert validate_platform_url("douyin", "https://www.douyin.com/video/1") + assert validate_platform_url("bilibili", "https://www.bilibili.com/video/BV1") + + +def test_docker_port_is_bound_to_loopback() -> None: + compose = (PROJECT_ROOT / "docker-compose.yml").read_text(encoding="utf-8") + assert '"127.0.0.1:8001:8001"' in compose + assert '\n - "8001:8001"' not in compose + assert 'NIUMA_TRUST_DOCKER_LOOPBACK_PROXY: "true"' in compose diff --git a/tests/test_partial_ai_analysis.py b/tests/test_partial_ai_analysis.py new file mode 100644 index 0000000..5614cc5 --- /dev/null +++ b/tests/test_partial_ai_analysis.py @@ -0,0 +1,406 @@ +from __future__ import annotations + +import json +from pathlib import Path + +import pytest + +from app.services.ai import ai_clip_analyzer, variety_comedy_analyzer +from app.services.ai.unit_checkpoint import AIUnitExecution +from app.services.ai.ai_clip_analyzer import AnalysisRequest, TranscriptChunk, TranscriptRow +from app.services.ai.variety_comedy_analyzer import ( + ComedyAnalysisRequest, + ComedyTranscriptWindow, +) +from app.services.ai_analysis_workflow_service import validate_ai_analysis_meta_for_cut +from app.services.pipeline_engine import PipelineEngine +from app.services.video_cut_workflow_service import process_task_video_cuts + + +def _valid_general_payload(task_id: str) -> str: + return json.dumps( + { + "task_id": task_id, + "analysis_summary": "ok", + "clips": [ + { + "clip_id": "clip-raw", + "title": "完整片段", + "start_time": "00:00:10", + "end_time": "00:01:10", + "duration_seconds": 60, + "cover_time_seconds": 30, + "summary": "摘要", + "highlight_reason": "亮点", + "spread_value": "高", + "suggested_editing": "保留上下文", + "confidence_score": 0.9, + } + ], + }, + ensure_ascii=False, + ) + + +def test_general_partial_chunk_failure_is_structured_and_not_retried(monkeypatch, tmp_path: Path): + task_id = "test-partial-general" + transcript = tmp_path / "transcript.md" + transcript.write_text("00:00:00 - 00:20:00 测试正文", encoding="utf-8") + + class Provider: + name = "remote" + + def __init__(self): + self.calls = 0 + + def generate_json(self, _prompt: str) -> str: + self.calls += 1 + return _valid_general_payload(task_id) if self.calls == 1 else "not-json" + + provider = Provider() + monkeypatch.setattr(ai_clip_analyzer, "build_provider", lambda *_args, **_kwargs: provider) + monkeypatch.setattr( + ai_clip_analyzer, + "_build_local_analysis_chunks", + lambda *_args, **_kwargs: [ + TranscriptChunk(1, 2, 0, 600, "00:00:00 - 00:10:00 第一段", 30), + TranscriptChunk(2, 2, 600, 1200, "00:10:00 - 00:20:00 第二段", 30), + ], + ) + + result = ai_clip_analyzer.analyze_task_transcript( + AnalysisRequest( + task_id=task_id, + transcript_path=transcript, + max_clip_duration_minutes=3, + target_clip_count=2, + ai_preference="", + provider_name="remote", + ) + ) + + assert provider.calls == 2 + assert len(result.clips) == 1 + assert result.analysis_meta["analysis_incomplete"] is True + assert result.analysis_meta["expected_units"] == 2 + assert result.analysis_meta["completed_units"] == 1 + assert result.analysis_meta["coverage_ratio"] == 0.5 + assert result.analysis_meta["failed_stages"][0]["unit_id"] == "chunk_002" + + +def test_variety_partial_recall_failure_is_structured(monkeypatch, tmp_path: Path): + rows = [TranscriptRow("00:00:00", "00:03:00", 0, 180, "综艺正文")] + windows = [ + ComedyTranscriptWindow(1, 2, 0, 90, tuple(rows), "第一窗"), + ComedyTranscriptWindow(2, 2, 90, 180, tuple(rows), "第二窗"), + ] + moment = { + "source_id": "moment-1", + "key_time": "00:01:00", + "key_seconds": 60, + "title": "笑点", + "topic_key": "综艺", + "humor_reason": "反差", + "recall_score": 90, + } + expanded = { + "source_id": "moment-1", + "title": "完整笑点", + "start_time": "00:00:10", + "end_time": "00:01:20", + "duration_seconds": 70, + "key_moment_time": "00:01:00", + "topic_key": "综艺", + "summary": "摘要", + "highlight_reason": "反差", + "arc_structure": "铺垫到反转", + "suggested_editing": "保留反应", + } + monkeypatch.setattr(variety_comedy_analyzer, "_read_transcript", lambda _path: "正文") + monkeypatch.setattr(variety_comedy_analyzer, "_extract_transcript_rows", lambda _text: rows) + monkeypatch.setattr(variety_comedy_analyzer, "build_provider", lambda *_args, **_kwargs: object()) + monkeypatch.setattr(variety_comedy_analyzer, "build_comedy_windows", lambda *_args, **_kwargs: windows) + monkeypatch.setattr( + variety_comedy_analyzer, + "_recall_moments", + lambda *_args, **_kwargs: ( + [moment], + ["召回窗口 2/2 跳过:invalid JSON"], + {"expected_units": 2, "completed_units": 1, "failed_units": 1, "empty_unit_count": 0, "invalid_item_count": 0}, + ), + ) + monkeypatch.setattr( + variety_comedy_analyzer, + "_expand_moments", + lambda *_args, **_kwargs: ( + [expanded], + [], + {"expected_units": 1, "completed_units": 1, "failed_units": 0, "empty_unit_count": 0, "invalid_item_count": 0}, + ), + ) + monkeypatch.setattr(variety_comedy_analyzer, "analyze_audio_reaction", lambda *_args, **_kwargs: {}) + monkeypatch.setattr(variety_comedy_analyzer, "list_recent_feedback_context", lambda *_args, **_kwargs: []) + monkeypatch.setattr(variety_comedy_analyzer, "_global_judge", lambda *_args, **_kwargs: ({}, "")) + monkeypatch.setattr( + variety_comedy_analyzer, + "score_comedy_candidate", + lambda candidate, _judge: { + **candidate, + "quality_tier": "A", + "quality_score": 90, + "text_quality_score": 90, + "humor_score": 90, + "completeness_score": 90, + "audio_reaction_score": 0, + "quality_evidence": {}, + "rejection_reason": "", + "selected_by_default": False, + }, + ) + + result = variety_comedy_analyzer.analyze_variety_comedy( + ComedyAnalysisRequest( + task_id="test-partial-variety", + transcript_path=tmp_path / "transcript.md", + audio_path=tmp_path / "audio.wav", + candidate_pool_limit=5, + final_clip_target=1, + ai_preference="", + provider_name="remote", + ) + ) + + assert len(result.clips) == 1 + assert result.analysis_meta["analysis_incomplete"] is True + assert result.analysis_meta["expected_units"] == 4 + assert result.analysis_meta["completed_units"] == 3 + assert result.analysis_meta["failed_units"] == 1 + assert result.analysis_meta["failed_stages"][0]["stage"] == "recall" + + +@pytest.mark.parametrize("profile", ["general", "variety_comedy"]) +def test_auto_pipeline_blocks_incomplete_analysis_for_every_profile(monkeypatch, profile: str): + engine = PipelineEngine() + monkeypatch.setattr(engine, "_get_task", lambda _task_id: {"selection_profile": profile}) + monkeypatch.setattr( + "app.services.pipeline_engine.task_service.get_task_ai_analysis_meta", + lambda _task_id: {"analysis_incomplete": True, "coverage_ratio": 0.5, "coverage_percent": 50}, + ) + + with pytest.raises(ValueError, match="不会进入自动切片"): + engine._select_clips("test-partial-gate", {"config": {}}) + + +@pytest.mark.parametrize("profile", ["general", "variety_comedy"]) +def test_manual_cut_blocks_incomplete_analysis_for_every_profile(monkeypatch, profile: str): + monkeypatch.setattr( + "app.services.task_service.get_task", + lambda *_args, **_kwargs: {"id": "test-partial-cut", "selection_profile": profile}, + ) + monkeypatch.setattr( + "app.services.ai_analysis_workflow_service.get_task_ai_analysis_meta", + lambda _task_id: {"analysis_incomplete": True, "coverage_ratio": 0.5, "coverage_percent": 50}, + ) + + with pytest.raises(ValueError, match="不会生成切片"): + process_task_video_cuts("test-partial-cut") + + +def test_quality_degraded_analysis_is_manual_review_only(monkeypatch): + engine = PipelineEngine() + monkeypatch.setattr(engine, "_get_task", lambda _task_id: {"selection_profile": "variety_comedy"}) + degraded = { + "schema_version": 2, + "selection_profile": "variety_comedy", + "analysis_incomplete": False, + "quality_degraded": True, + "coverage_ratio": 1.0, + "coverage_percent": 100, + } + monkeypatch.setattr( + "app.services.pipeline_engine.task_service.get_task_ai_analysis_meta", + lambda _task_id: degraded, + ) + with pytest.raises(ValueError, match="质量评审未完整通过"): + engine._select_clips("test-quality-gate", {"config": {}}) + + monkeypatch.setattr( + "app.services.task_service.get_task", + lambda *_args, **_kwargs: {"id": "test-quality-cut", "selection_profile": "variety_comedy"}, + ) + monkeypatch.setattr( + "app.services.ai_analysis_workflow_service.get_task_ai_analysis_meta", + lambda _task_id: degraded, + ) + with pytest.raises(ValueError, match="质量评审未完整通过"): + process_task_video_cuts("test-quality-cut") + + +def test_missing_analysis_meta_blocks_manual_and_auto_cut(monkeypatch): + engine = PipelineEngine() + monkeypatch.setattr(engine, "_get_task", lambda _task_id: {"selection_profile": "general"}) + monkeypatch.setattr( + "app.services.pipeline_engine.task_service.get_task_ai_analysis_meta", + lambda _task_id: {}, + ) + with pytest.raises(ValueError, match="缺少可信的完整性元数据"): + engine._select_clips("test-missing-meta-auto", {"config": {}}) + + monkeypatch.setattr( + "app.services.task_service.get_task", + lambda *_args, **_kwargs: {"id": "test-missing-meta-cut", "selection_profile": "general"}, + ) + monkeypatch.setattr( + "app.services.ai_analysis_workflow_service.get_task_ai_analysis_meta", + lambda _task_id: {}, + ) + with pytest.raises(ValueError, match="缺少可信的完整性元数据"): + process_task_video_cuts("test-missing-meta-cut") + + +def test_analysis_profile_mismatch_fails_closed(): + meta = { + "schema_version": 2, + "selection_profile": "general", + "analysis_incomplete": False, + "quality_degraded": False, + "coverage_ratio": 1.0, + "coverage_percent": 100.0, + } + + result = validate_ai_analysis_meta_for_cut(meta, "long_live_talk") + assert result["analysis_incomplete"] is True + assert result["quality_degraded"] is True + assert result["integrity_error"] == "selection_profile_mismatch" + + +def test_complete_analysis_requires_structured_unit_evidence(): + meta = { + "schema_version": 2, + "selection_profile": "general", + "analysis_incomplete": False, + "quality_degraded": False, + "coverage_ratio": 1.0, + "coverage_percent": 100.0, + "invalid_item_count": 0, + } + + result = validate_ai_analysis_meta_for_cut(meta, "general") + + assert result["analysis_incomplete"] is True + assert result["integrity_error"] == "analysis_unit_evidence_invalid" + + +def test_complete_analysis_coverage_must_match_unit_ledger(): + meta = { + "schema_version": 2, + "selection_profile": "general", + "analysis_incomplete": False, + "quality_degraded": False, + "coverage_ratio": 0.5, + "coverage_percent": 50.0, + "expected_units": 2, + "completed_units": 2, + "failed_units": 0, + "failed_stages": [], + "invalid_item_count": 0, + } + + result = validate_ai_analysis_meta_for_cut(meta, "general") + + assert result["analysis_incomplete"] is True + assert result["integrity_error"] == "analysis_unit_evidence_invalid" + + +def test_corrupt_replayed_variety_unit_is_not_counted_complete(monkeypatch): + window = ComedyTranscriptWindow( + 1, + 1, + 0, + 60, + (TranscriptRow("00:00:00", "00:01:00", 0, 60, "正文"),), + "正文", + ) + monkeypatch.setattr( + variety_comedy_analyzer, + "execute_checkpointed_ai_unit", + lambda **_kwargs: AIUnitExecution(status="completed", payload={}, reused=True), + ) + + moments, failures, stats = variety_comedy_analyzer._recall_moments( + object(), + [window], + "", + task_id="test-corrupt-variety-replay", + input_fingerprint="stable", + ) + + assert moments == [] + assert stats["completed_units"] == 0 + assert stats["failed_units"] == 1 + assert "moments 不是数组" in failures[0] + + +@pytest.mark.parametrize( + ("field", "value"), + [ + ("schema_version", 2.9), + ("coverage_ratio", True), + ("coverage_percent", True), + ("coverage_ratio", float("nan")), + ("coverage_percent", float("inf")), + ("coverage_percent", 99.0), + ], +) +def test_noncanonical_analysis_meta_fails_closed(field: str, value: object): + meta = { + "schema_version": 2, + "selection_profile": "general", + "analysis_incomplete": False, + "quality_degraded": False, + "coverage_ratio": 1.0, + "coverage_percent": 100.0, + } + meta[field] = value + + result = validate_ai_analysis_meta_for_cut(meta, "general") + + assert result["analysis_incomplete"] is True + assert result["quality_degraded"] is True + assert result["integrity_error"] in {"analysis_schema_invalid", "analysis_coverage_invalid"} + + +def test_variety_global_judge_requires_complete_candidate_coverage(): + candidates = [] + for source_id in ("candidate-a", "candidate-b"): + candidates.append( + { + "source_id": source_id, + "title": source_id, + "start_time": "00:00:00", + "end_time": "00:01:00", + "duration_seconds": 60, + "topic_key": "topic", + "summary": "summary", + "highlight_reason": "reason", + "arc_structure": "arc", + "audio_evidence": {}, + } + ) + + class Provider: + def generate_json(self, _prompt: str, retry_instruction: str | None = None) -> str: + del retry_instruction + return json.dumps({"ranked_clips": [{"source_id": "candidate-a"}]}) + + judged, warning = variety_comedy_analyzer._global_judge( + Provider(), + candidates, + "", + [], + task_id="test-partial-judge", + input_fingerprint="stable-input", + ) + + assert set(judged) == {"candidate-a"} + assert "缺少 1 个候选" in warning diff --git a/tests/test_path_resolution.py b/tests/test_path_resolution.py index 88086ab..fbaaed6 100644 --- a/tests/test_path_resolution.py +++ b/tests/test_path_resolution.py @@ -41,20 +41,31 @@ def test_sanitize_fallback_for_empty(self): assert len(result) > 0 assert result == "untitled" - def test_allocate_generates_unique_names(self, tmp_path, monkeypatch): - """重名时自动追加序号 — 需要模拟数据库已有第一条记录""" - monkeypatch.setenv("TASKS_DIR", str(tmp_path)) - monkeypatch.setenv("STORAGE_ROOT", str(tmp_path)) - # 第一次分配:创建目录并写入数据库 - name1 = allocate_task_dir_name("测试唯一性") - # 模拟写入数据库 — 第二次调用应该能检测到重名 - # 由于测试环境没有真实数据库,只验证两次调用格式正确 - assert name1.endswith("测试唯一性") or "测试唯一性" in name1 - # 验证函数不会崩溃 - name2 = allocate_task_dir_name("另一个项目") - assert "另一个项目" in name2 - assert isinstance(name1, str) - assert isinstance(name2, str) + def test_allocate_generates_unique_names(self, tmp_path): + """同名任务通过原子目录预占获得不同目录名。""" + from app.services import storage_service + + runtime_settings = storage_service.settings + + originals = { + "storage_root": runtime_settings.storage_root, + "tasks_dir": runtime_settings.tasks_dir, + "database_path": runtime_settings.database_path, + } + object.__setattr__(runtime_settings, "storage_root", tmp_path) + object.__setattr__(runtime_settings, "tasks_dir", tmp_path) + object.__setattr__(runtime_settings, "database_path", tmp_path / "missing.sqlite3") + try: + name1 = storage_service.allocate_task_dir_name("测试唯一性") + name2 = storage_service.allocate_task_dir_name("测试唯一性") + finally: + for name, value in originals.items(): + object.__setattr__(runtime_settings, name, value) + + assert name1 == "测试唯一性" + assert name2 == "测试唯一性 (2)" + assert (tmp_path / name1).is_dir() + assert (tmp_path / name2).is_dir() def test_storage_path_can_be_configured(self, tmp_path): """路径可以由环境变量配置,不是只写死 E 盘""" @@ -139,41 +150,39 @@ def test_all_expected_keys_present(self): class TestCreateTaskDirectory: """创建任务目录时以 task_dir_name 为准""" - def test_directory_created_with_dir_name(self, tmp_path, monkeypatch): + def test_directory_created_with_dir_name(self, tmp_path): """create_task_directory 使用 task_dir_name 而非 task_id 作为文件夹名""" - # 先设置临时数据目录和数据库路径,避免污染真实数据 - data_dir = tmp_path / "data" - data_dir.mkdir(parents=True, exist_ok=True) - monkeypatch.setenv("TASKS_DIR", str(tmp_path)) - monkeypatch.setenv("STORAGE_ROOT", str(tmp_path)) - monkeypatch.setenv("DATA_DIR", str(data_dir)) - monkeypatch.setenv("DATABASE_PATH", str(data_dir / "test_db.sqlite3")) - - # 强制重载 config 和 storage_service,让新环境变量生效 - import importlib - import app.core.config - import app.services.storage_service - importlib.reload(app.core.config) - importlib.reload(app.services.storage_service) - from app.services.storage_service import get_artifact_paths - - task_dir = create_task_directory(task_id="abc123", task_dir_name="我的项目") - - # 目录名以 task_dir_name 为准,不是 task_id - assert "我的项目" in str(task_dir), f"目录名应包含 task_dir_name,实际:{task_dir}" - assert "abc123" not in str(task_dir), "目录名不应包含 task_id" - - # 所有子目录都存在 - expected_subdirs = [ - "source", "audio", "transcripts", "analysis", - "05_clips", "06_subtitled", "07_covers", "logs", - ] - for sub in expected_subdirs: - sub_path = task_dir / sub - assert sub_path.exists(), f"缺少子目录:{sub}" - - # clips_dir 指向 05_clips - paths = get_artifact_paths(task_id="abc123", task_dir_name="我的项目") - assert "05_clips" in str(paths["clips_dir"]), ( - f"正式 clips_dir 应指向 05_clips,实际:{paths['clips_dir']}" - ) + from app.services import storage_service + + runtime_settings = storage_service.settings + + originals = { + "storage_root": runtime_settings.storage_root, + "tasks_dir": runtime_settings.tasks_dir, + } + object.__setattr__(runtime_settings, "storage_root", tmp_path) + object.__setattr__(runtime_settings, "tasks_dir", tmp_path) + try: + task_dir = storage_service.create_task_directory(task_id="abc123", task_dir_name="我的项目") + + # 目录名以 task_dir_name 为准,不是 task_id + assert "我的项目" in str(task_dir), f"目录名应包含 task_dir_name,实际:{task_dir}" + assert "abc123" not in str(task_dir), "目录名不应包含 task_id" + + # 所有子目录都存在 + expected_subdirs = [ + "source", "audio", "transcripts", "analysis", + "05_clips", "06_subtitled", "07_covers", "logs", + ] + for sub in expected_subdirs: + sub_path = task_dir / sub + assert sub_path.exists(), f"缺少子目录:{sub}" + + # clips_dir 指向 05_clips + paths = storage_service.get_artifact_paths(task_id="abc123", task_dir_name="我的项目") + assert "05_clips" in str(paths["clips_dir"]), ( + f"正式 clips_dir 应指向 05_clips,实际:{paths['clips_dir']}" + ) + finally: + for name, value in originals.items(): + object.__setattr__(runtime_settings, name, value) diff --git a/tests/test_pipeline_checkpoint.py b/tests/test_pipeline_checkpoint.py new file mode 100644 index 0000000..7f7efdf --- /dev/null +++ b/tests/test_pipeline_checkpoint.py @@ -0,0 +1,773 @@ +"""P1.3b auto-pipeline checkpoint and restart reconciliation tests.""" + +from __future__ import annotations + +import json +import shutil +from pathlib import Path +from unittest.mock import Mock + +import pytest + +from app.db.database import get_connection +from app.models.task import TaskCreate, TaskStatus +from app.services import job_service, task_service +from app.services.ai import unit_checkpoint +from app.services.pipeline_checkpoint_service import ( + AUTO_PIPELINE_CHECKPOINT_KIND, + AutoPipelineCheckpoint, + PipelineCheckpointError, +) +from app.services.pipeline_engine import PipelineEngine, STEP_STATUSES, start_auto_pipeline +from app.services.storage_service import get_artifact_paths +from app.services.task_lifecycle_service import create_task_record + + +@pytest.fixture(autouse=True) +def cleanup_pipeline_checkpoint_data(): + """Keep this module's records and artifacts from leaking into later tests.""" + + yield + pattern = "test-pipeline-checkpoint-%" + with get_connection() as connection: + task_ids = [ + str(row["id"]) + for row in connection.execute( + "SELECT id FROM tasks WHERE id LIKE ?", + (pattern,), + ).fetchall() + ] + connection.execute("DELETE FROM publish_jobs WHERE task_id LIKE ?", (pattern,)) + connection.execute("DELETE FROM subtitle_jobs WHERE task_id LIKE ?", (pattern,)) + connection.execute( + """ + DELETE FROM subtitle_cues WHERE revision_id IN ( + SELECT revision.id FROM subtitle_revisions AS revision + JOIN subtitle_tracks AS track ON track.id = revision.track_id + WHERE track.task_id LIKE ? + ) + """, + (pattern,), + ) + connection.execute( + """ + DELETE FROM subtitle_revisions WHERE track_id IN ( + SELECT id FROM subtitle_tracks WHERE task_id LIKE ? + ) + """, + (pattern,), + ) + connection.execute("DELETE FROM subtitle_tracks WHERE task_id LIKE ?", (pattern,)) + connection.execute("DELETE FROM clip_feedback WHERE task_id LIKE ?", (pattern,)) + connection.execute("DELETE FROM output_clip WHERE task_id LIKE ?", (pattern,)) + connection.execute("DELETE FROM cut_runs WHERE task_id LIKE ?", (pattern,)) + connection.execute("DELETE FROM clip_candidates WHERE task_id LIKE ?", (pattern,)) + connection.execute("DELETE FROM ai_analysis_windows WHERE task_id LIKE ?", (pattern,)) + connection.execute("DELETE FROM ai_analysis_runs WHERE task_id LIKE ?", (pattern,)) + connection.execute("DELETE FROM transcription_chunks WHERE task_id LIKE ?", (pattern,)) + connection.execute("DELETE FROM transcription_runs WHERE task_id LIKE ?", (pattern,)) + connection.execute("DELETE FROM workflow_jobs WHERE task_id LIKE ?", (pattern,)) + connection.execute("DELETE FROM tasks WHERE id LIKE ?", (pattern,)) + connection.commit() + for task_id in task_ids: + shutil.rmtree(get_artifact_paths(task_id)["task_dir"], ignore_errors=True) + + +def _create_auto_task(task_id: str, *, delivery_mode: str = "original") -> dict: + create_task_record( + TaskCreate( + task_name=task_id, + source_type="upload", + platform="general", + selection_profile="general", + auto_mode=True, + ), + task_id=task_id, + task_dir_name=task_id, + ) + with get_connection() as connection: + connection.execute( + "UPDATE tasks SET auto_config_json = ? WHERE id = ?", + (json.dumps({"subtitle_delivery_mode": delivery_mode}), task_id), + ) + connection.commit() + return task_service.get_task(task_id, include_video_probe=False) + + +def _claim_auto_job(task_id: str, start_step: TaskStatus, owner: str) -> dict: + job = job_service.create_job( + task_id=task_id, + job_type=job_service.JOB_TYPE_AUTO_PIPELINE, + payload={"retry": False, "start_step": start_step.value}, + ) + claimed = job_service.claim_job(job["id"], owner) + assert claimed is not None + return claimed + + +def _checkpoint( + engine: PipelineEngine, + task: dict, + job: dict, + start_step: TaskStatus, +) -> AutoPipelineCheckpoint: + return AutoPipelineCheckpoint.load( + job_id=job["id"], + task_id=task["id"], + start_step=start_step.value, + run_key=engine._pipeline_run_key(task, engine._load_auto_config(task), start_step), + ordered_steps=[step.value for step in STEP_STATUSES], + ) + + +def _insert_candidate(task_id: str, candidate_id: str, *, clip_key: str | None = None) -> None: + now = task_service._now_iso() + with get_connection() as connection: + connection.execute( + """ + INSERT INTO clip_candidates ( + id, task_id, clip_key, title, start_time, end_time, duration_seconds, + created_at, updated_at + ) VALUES (?, ?, ?, 'clip', '00:00:00', '00:00:30', 30, ?, ?) + """, + (candidate_id, task_id, clip_key or candidate_id, now, now), + ) + connection.commit() + + +def _insert_output_clip( + task_id: str, + output_id: str, + path: Path, + *, + cut_run_id: str | None = None, + clip_candidate_id: str | None = None, +) -> None: + now = task_service._now_iso() + with get_connection() as connection: + connection.execute( + """ + INSERT INTO output_clip ( + id, task_id, clip_candidate_id, output_file_path, output_file_name, status, + cut_run_id, is_active, created_at, updated_at + ) VALUES (?, ?, ?, ?, ?, 'completed', ?, 1, ?, ?) + """, + ( + output_id, + task_id, + clip_candidate_id, + str(path), + path.name, + cut_run_id, + now, + now, + ), + ) + connection.commit() + + +def test_auto_pipeline_checkpoint_is_versioned_and_fenced() -> None: + task = _create_auto_task("test-pipeline-checkpoint-envelope") + job = _claim_auto_job(task["id"], TaskStatus.PREPARING_SOURCE, "checkpoint-owner") + with job_service.job_lease_context(job["id"], "checkpoint-owner", job["lease_token"]): + checkpoint = _checkpoint(PipelineEngine(), task, job, TaskStatus.PREPARING_SOURCE) + checkpoint.begin_step(TaskStatus.PREPARING_SOURCE.value, baseline={"before": "empty"}) + checkpoint.complete_step( + TaskStatus.PREPARING_SOURCE.value, + outputs={"source_reference": {"sha256": "evidence"}}, + ) + + stored = job_service.get_job(job["id"])["checkpoint_json"] + assert stored["kind"] == AUTO_PIPELINE_CHECKPOINT_KIND + assert stored["completed_steps"] == [TaskStatus.PREPARING_SOURCE.value] + assert stored["steps"][TaskStatus.PREPARING_SOURCE.value]["state"] == "succeeded" + assert "config" not in stored + + with job_service.job_lease_context(job["id"], "old-owner", "old-token"): + with pytest.raises(job_service.JobLeaseLostError): + checkpoint.begin_step(TaskStatus.TRANSCRIBING.value) + + +def test_auto_pipeline_checkpoint_preserves_ai_unit_recovery_evidence() -> None: + task = _create_auto_task("test-pipeline-checkpoint-ai-units") + job = _claim_auto_job(task["id"], TaskStatus.AI_ANALYZING, "ai-unit-owner") + with job_service.job_lease_context(job["id"], "ai-unit-owner", job["lease_token"]): + checkpoint = _checkpoint(PipelineEngine(), task, job, TaskStatus.AI_ANALYZING) + checkpoint.begin_step(TaskStatus.AI_ANALYZING.value, baseline={}) + unit_checkpoint.execute_checkpointed_ai_unit( + task_id=task["id"], + namespace="general_chunks", + input_fingerprint="stable-input", + unit_id="chunk_001", + operation=lambda: {"clips": [{"clip_id": "confirmed"}]}, + ) + checkpoint.fail_step(TaskStatus.AI_ANALYZING.value, "模拟最终提交前中断") + + stored = job_service.get_job(job["id"])["checkpoint_json"] + assert stored["kind"] == AUTO_PIPELINE_CHECKPOINT_KIND + assert stored["steps"][TaskStatus.AI_ANALYZING.value]["state"] == "failed" + unit = stored["_ai_analysis_units_v1"]["namespaces"]["general_chunks"]["units"]["chunk_001"] + assert unit["status"] == "completed" + + +@pytest.mark.parametrize( + "raw_checkpoint", + ["{broken", {"kind": "future_auto_pipeline_v9"}], +) +def test_malformed_or_unknown_checkpoint_fails_closed(raw_checkpoint) -> None: + task = _create_auto_task(f"test-pipeline-checkpoint-bad-{type(raw_checkpoint).__name__}") + job = _claim_auto_job(task["id"], TaskStatus.PUBLISH_JOB_CREATING, "bad-checkpoint-owner") + with get_connection() as connection: + value = raw_checkpoint if isinstance(raw_checkpoint, str) else json.dumps(raw_checkpoint) + connection.execute( + "UPDATE workflow_jobs SET checkpoint_json = ? WHERE id = ?", + (value, job["id"]), + ) + connection.commit() + engine = PipelineEngine() + engine._create_publish_jobs = Mock(side_effect=AssertionError("未知 checkpoint 不得执行 handler")) + + with job_service.job_lease_context(job["id"], "bad-checkpoint-owner", job["lease_token"]): + with pytest.raises(PipelineCheckpointError): + engine.run( + task["id"], + start_step=TaskStatus.PUBLISH_JOB_CREATING, + job_id=job["id"], + ) + + engine._create_publish_jobs.assert_not_called() + assert task_service.get_task(task["id"], include_video_probe=False)["status"] == TaskStatus.CREATED.value + + +def test_completed_metadata_restores_context_without_regeneration(tmp_path: Path) -> None: + task = _create_auto_task("test-pipeline-checkpoint-metadata") + paths = get_artifact_paths(task["id"]) + clip_path = paths["clips_dir"] / "checkpoint-metadata.mp4" + clip_path.parent.mkdir(parents=True, exist_ok=True) + clip_path.write_bytes(b"video") + _insert_output_clip(task["id"], "checkpoint-metadata-output", clip_path) + output_clip = task_service.list_output_clips(task["id"])[0] + cover_path = paths["covers_dir"] / "checkpoint-cover.jpg" + cover_path.parent.mkdir(parents=True, exist_ok=True) + cover_path.write_bytes(b"cover") + metadata_items = [ + { + "output_clip": output_clip, + "metadata": { + "platform": "douyin", + "title": "checkpoint title", + "caption": "checkpoint caption", + "hashtags": ["checkpoint"], + "risk_flags": [], + }, + "cover": {"cover_file_path": str(cover_path), "cover_time_seconds": 1}, + } + ] + metadata_path = paths["analysis_path"].parent / "auto_publish_metadata.json" + metadata_path.parent.mkdir(parents=True, exist_ok=True) + metadata_path.write_text(json.dumps(metadata_items, ensure_ascii=False), encoding="utf-8") + + engine = PipelineEngine() + job = _claim_auto_job(task["id"], TaskStatus.METADATA_GENERATING, "metadata-owner") + captured_schedule: list[dict] = [] + with job_service.job_lease_context(job["id"], "metadata-owner", job["lease_token"]): + checkpoint = _checkpoint(engine, task, job, TaskStatus.METADATA_GENERATING) + checkpoint.begin_step(TaskStatus.METADATA_GENERATING.value) + metadata_result = { + "metadata_path": str(metadata_path), + "metadata_count": 1, + "need_review_count": 0, + "metadata_items": metadata_items, + } + checkpoint.complete_step( + TaskStatus.METADATA_GENERATING.value, + outputs=engine._checkpoint_outputs( + task["id"], TaskStatus.METADATA_GENERATING, metadata_result + ), + ) + engine._generate_metadata = Mock(side_effect=AssertionError("metadata 不应重复生成")) + original_schedule = engine._create_schedule + + def capture_schedule(task_id: str, context: dict) -> dict: + captured_schedule.extend(context[TaskStatus.METADATA_GENERATING.value]["metadata_items"]) + return original_schedule(task_id, context) + + engine._create_schedule = capture_schedule + engine._write_task_summary = Mock(return_value={"summary_path": str(tmp_path / "summary.json")}) + result = engine.run( + task["id"], + start_step=TaskStatus.METADATA_GENERATING, + job_id=job["id"], + ) + + assert result["status"] == "ready_to_publish" + engine._generate_metadata.assert_not_called() + assert captured_schedule[0]["metadata"]["title"] == "checkpoint title" + stored = job_service.get_job(job["id"])["checkpoint_json"] + metadata_outputs = stored["steps"][TaskStatus.METADATA_GENERATING.value]["outputs"] + assert "metadata_items" not in metadata_outputs + assert stored["completed_steps"] == [ + TaskStatus.METADATA_GENERATING.value, + TaskStatus.SCHEDULE_CREATING.value, + TaskStatus.PUBLISH_JOB_CREATING.value, + ] + publish_outputs = stored["steps"][TaskStatus.PUBLISH_JOB_CREATING.value]["outputs"] + with job_service.job_lease_context(job["id"], "metadata-owner", job["lease_token"]): + recovered_publish = engine._reconcile_interrupted_step( + task["id"], + TaskStatus.PUBLISH_JOB_CREATING, + {"baseline": {"schedule": publish_outputs["schedule_input"]}}, + ) + assert recovered_publish is not None + assert recovered_publish["created_count"] == 1 + assert recovered_publish["created"][0]["id"] == publish_outputs["created_ids"][0] + expected_job = publish_outputs["job_evidence"][0] + with get_connection() as connection: + connection.execute( + "UPDATE output_clip SET is_active = 0 WHERE id = ?", + (expected_job["output_clip_id"],), + ) + connection.commit() + with pytest.raises(PipelineCheckpointError, match="切片已失活"): + engine._restore_checkpoint_step( + task["id"], + TaskStatus.PUBLISH_JOB_CREATING, + publish_outputs, + ) + with get_connection() as connection: + connection.execute( + "UPDATE output_clip SET is_active = 1 WHERE id = ?", + (expected_job["output_clip_id"],), + ) + connection.commit() + with get_connection() as connection: + connection.execute( + "UPDATE publish_jobs SET platform = 'bilibili' WHERE id = ?", + (publish_outputs["created_ids"][0],), + ) + connection.commit() + with pytest.raises(PipelineCheckpointError, match="草稿字段不一致"): + engine._restore_checkpoint_step( + task["id"], + TaskStatus.PUBLISH_JOB_CREATING, + publish_outputs, + ) + with get_connection() as connection: + connection.execute( + "UPDATE publish_jobs SET platform = ?, title = 'tampered title' WHERE id = ?", + (expected_job["platform"], publish_outputs["created_ids"][0]), + ) + connection.commit() + with pytest.raises(PipelineCheckpointError, match="草稿字段不一致"): + engine._restore_checkpoint_step( + task["id"], + TaskStatus.PUBLISH_JOB_CREATING, + publish_outputs, + ) + + +def test_interrupted_ai_step_reconciles_new_analysis_run_without_provider_call() -> None: + task = _create_auto_task("test-pipeline-checkpoint-ai") + engine = PipelineEngine() + job = _claim_auto_job(task["id"], TaskStatus.AI_ANALYZING, "ai-checkpoint-owner") + paths = get_artifact_paths(task["id"]) + paths["transcript_path"].parent.mkdir(parents=True, exist_ok=True) + paths["transcript_path"].write_text( + "00:00:00 --> 00:00:30\ncheckpoint transcript", + encoding="utf-8", + ) + analysis_payload = { + "analysis_summary": "recovered", + "analysis_meta": {"provider": "test", "generated_at": task_service._now_iso()}, + "clips": [{"clip_id": "clip-1", "title": "clip"}], + } + with job_service.job_lease_context(job["id"], "ai-checkpoint-owner", job["lease_token"]): + checkpoint = _checkpoint(engine, task, job, TaskStatus.AI_ANALYZING) + checkpoint.begin_step( + TaskStatus.AI_ANALYZING.value, + baseline=engine._checkpoint_baseline(task["id"], TaskStatus.AI_ANALYZING), + ) + paths["analysis_path"].parent.mkdir(parents=True, exist_ok=True) + paths["analysis_path"].write_text(json.dumps(analysis_payload), encoding="utf-8") + now = task_service._now_iso() + _insert_candidate(task["id"], "checkpoint-ai-clip", clip_key="clip-1") + with get_connection() as connection: + connection.execute( + """ + INSERT INTO ai_analysis_runs ( + id, task_id, run_number, provider, provider_label, model, + requested_clip_count, clip_count, analysis_payload_json, + is_active, created_at + ) VALUES ('checkpoint-ai-run', ?, 1, 'test', 'test', 'test', 1, 1, ?, 1, ?) + """, + (task["id"], json.dumps(analysis_payload), now), + ) + connection.commit() + engine._run_ai_analysis = Mock(side_effect=AssertionError("AI Provider 不应重复调用")) + engine._select_clips = Mock(side_effect=RuntimeError("stop after AI recovery")) + result = engine.run( + task["id"], + start_step=TaskStatus.AI_ANALYZING, + job_id=job["id"], + ) + + assert result["failed_step"] == TaskStatus.CLIP_SELECTING.value + engine._run_ai_analysis.assert_not_called() + stored = job_service.get_job(job["id"])["checkpoint_json"] + assert stored["completed_steps"] == [TaskStatus.AI_ANALYZING.value] + assert stored["steps"][TaskStatus.AI_ANALYZING.value]["recovered"] is True + + +def test_auto_retry_reuses_failed_job_checkpoint_without_provider_call() -> None: + task = _create_auto_task("test-pipeline-checkpoint-ai-retry") + engine = PipelineEngine() + first_job = _claim_auto_job(task["id"], TaskStatus.AI_ANALYZING, "retry-old-owner") + paths = get_artifact_paths(task["id"]) + paths["transcript_path"].parent.mkdir(parents=True, exist_ok=True) + paths["transcript_path"].write_text("retry transcript", encoding="utf-8") + analysis_payload = { + "analysis_summary": "recover after failed job", + "analysis_meta": {"provider": "test", "generated_at": task_service._now_iso()}, + "clips": [{"clip_id": "retry-clip", "title": "retry"}], + } + with job_service.job_lease_context( + first_job["id"], + "retry-old-owner", + first_job["lease_token"], + ): + checkpoint = _checkpoint(engine, task, first_job, TaskStatus.AI_ANALYZING) + checkpoint.begin_step( + TaskStatus.AI_ANALYZING.value, + baseline=engine._checkpoint_baseline(task["id"], TaskStatus.AI_ANALYZING), + ) + paths["analysis_path"].write_text(json.dumps(analysis_payload), encoding="utf-8") + _insert_candidate(task["id"], "checkpoint-ai-retry-clip", clip_key="retry-clip") + now = task_service._now_iso() + with get_connection() as connection: + connection.execute( + """ + INSERT INTO ai_analysis_runs ( + id, task_id, run_number, provider, provider_label, model, + requested_clip_count, clip_count, analysis_payload_json, + is_active, created_at + ) VALUES ('checkpoint-ai-retry-run', ?, 1, 'test', 'test', 'test', 1, 1, ?, 1, ?) + """, + (task["id"], json.dumps(analysis_payload), now), + ) + connection.commit() + job_service.mark_job_failed(first_job["id"], "worker exited after provider result") + + queued = start_auto_pipeline(task["id"], background_tasks=object(), retry=True) + assert queued["job_id"] == first_job["id"] + assert queued["status"] == job_service.JOB_STATUS_QUEUED + assert job_service.get_job(first_job["id"])["checkpoint_json"]["current_step"] == ( + TaskStatus.AI_ANALYZING.value + ) + + reclaimed = job_service.claim_job(first_job["id"], "retry-new-owner") + assert reclaimed is not None + engine._run_ai_analysis = Mock(side_effect=AssertionError("AI Provider 不应重复调用")) + engine._select_clips = Mock(side_effect=RuntimeError("stop after retry recovery")) + with job_service.job_lease_context( + reclaimed["id"], + "retry-new-owner", + reclaimed["lease_token"], + ): + result = engine.run( + task["id"], + start_step=TaskStatus.AI_ANALYZING, + job_id=reclaimed["id"], + ) + + assert result["failed_step"] == TaskStatus.CLIP_SELECTING.value + engine._run_ai_analysis.assert_not_called() + assert job_service.get_job(reclaimed["id"])["checkpoint_json"]["completed_steps"] == [ + TaskStatus.AI_ANALYZING.value + ] + + +def test_job_id_without_active_lease_fails_before_handler() -> None: + task = _create_auto_task("test-pipeline-checkpoint-no-lease") + job = job_service.create_job( + task_id=task["id"], + job_type=job_service.JOB_TYPE_AUTO_PIPELINE, + payload={"start_step": TaskStatus.AI_ANALYZING.value}, + ) + engine = PipelineEngine() + engine._run_ai_analysis = Mock(side_effect=AssertionError("无租约不得执行 handler")) + + with pytest.raises(job_service.JobLeaseLostError): + engine.run( + task["id"], + start_step=TaskStatus.AI_ANALYZING, + job_id=job["id"], + ) + + engine._run_ai_analysis.assert_not_called() + assert task_service.get_task(task["id"], include_video_probe=False)["status"] == ( + TaskStatus.CREATED.value + ) + + +def test_interrupted_transcript_step_reconciles_final_markdown() -> None: + task = _create_auto_task("test-pipeline-checkpoint-transcript") + engine = PipelineEngine() + job = _claim_auto_job(task["id"], TaskStatus.TRANSCRIBING, "transcript-owner") + paths = get_artifact_paths(task["id"]) + + with job_service.job_lease_context(job["id"], "transcript-owner", job["lease_token"]): + checkpoint = _checkpoint(engine, task, job, TaskStatus.TRANSCRIBING) + checkpoint.begin_step( + TaskStatus.TRANSCRIBING.value, + baseline=engine._checkpoint_baseline(task["id"], TaskStatus.TRANSCRIBING), + ) + paths["transcript_path"].parent.mkdir(parents=True, exist_ok=True) + paths["transcript_path"].write_text( + "00:00:00 --> 00:00:30\ncompleted transcript", + encoding="utf-8", + ) + engine._transcribe_or_read_text = Mock( + side_effect=AssertionError("最终 Markdown 已存在时不应重复转写") + ) + engine._run_ai_analysis = Mock(side_effect=RuntimeError("stop after transcript recovery")) + result = engine.run( + task["id"], + start_step=TaskStatus.TRANSCRIBING, + job_id=job["id"], + ) + + assert result["failed_step"] == TaskStatus.AI_ANALYZING.value + engine._transcribe_or_read_text.assert_not_called() + stored = job_service.get_job(job["id"])["checkpoint_json"] + assert stored["completed_steps"] == [TaskStatus.TRANSCRIBING.value] + assert stored["steps"][TaskStatus.TRANSCRIBING.value]["recovered"] is True + + +def test_interrupted_ai_step_reruns_when_transcript_changed() -> None: + task = _create_auto_task("test-pipeline-checkpoint-ai-input-changed") + engine = PipelineEngine() + job = _claim_auto_job(task["id"], TaskStatus.AI_ANALYZING, "ai-input-owner") + paths = get_artifact_paths(task["id"]) + paths["transcript_path"].parent.mkdir(parents=True, exist_ok=True) + paths["transcript_path"].write_text("original transcript", encoding="utf-8") + analysis_payload = { + "analysis_summary": "stale result", + "analysis_meta": {"provider": "test", "generated_at": task_service._now_iso()}, + "clips": [{"clip_id": "clip-stale", "title": "stale"}], + } + + with job_service.job_lease_context(job["id"], "ai-input-owner", job["lease_token"]): + checkpoint = _checkpoint(engine, task, job, TaskStatus.AI_ANALYZING) + checkpoint.begin_step( + TaskStatus.AI_ANALYZING.value, + baseline=engine._checkpoint_baseline(task["id"], TaskStatus.AI_ANALYZING), + ) + paths["transcript_path"].write_text("changed transcript", encoding="utf-8") + paths["analysis_path"].write_text(json.dumps(analysis_payload), encoding="utf-8") + _insert_candidate(task["id"], "checkpoint-ai-stale-clip", clip_key="clip-stale") + now = task_service._now_iso() + with get_connection() as connection: + connection.execute( + """ + INSERT INTO ai_analysis_runs ( + id, task_id, run_number, provider, provider_label, model, + requested_clip_count, clip_count, analysis_payload_json, + is_active, created_at + ) VALUES ('checkpoint-ai-stale-run', ?, 1, 'test', 'test', 'test', 1, 1, ?, 1, ?) + """, + (task["id"], json.dumps(analysis_payload), now), + ) + connection.commit() + engine._run_ai_analysis = Mock(side_effect=RuntimeError("provider rerun required")) + result = engine.run( + task["id"], + start_step=TaskStatus.AI_ANALYZING, + job_id=job["id"], + ) + + assert result["failed_step"] == TaskStatus.AI_ANALYZING.value + assert result["last_error"] == "provider rerun required" + engine._run_ai_analysis.assert_called_once() + stored = job_service.get_job(job["id"])["checkpoint_json"] + assert stored["completed_steps"] == [] + assert stored["steps"][TaskStatus.AI_ANALYZING.value]["state"] == "failed" + + +def test_interrupted_cut_step_reconciles_active_run_without_ffmpeg(tmp_path: Path) -> None: + task = _create_auto_task("test-pipeline-checkpoint-cut") + engine = PipelineEngine() + job = _claim_auto_job(task["id"], TaskStatus.VIDEO_CUTTING, "cut-checkpoint-owner") + paths = get_artifact_paths(task["id"]) + output_path = paths["clips_dir"] / "run_0001_checkpoint-cut-run" / "clip.mp4" + output_path.parent.mkdir(parents=True, exist_ok=True) + output_path.write_bytes(b"video") + _insert_candidate(task["id"], "checkpoint-cut-candidate") + with job_service.job_lease_context(job["id"], "cut-checkpoint-owner", job["lease_token"]): + checkpoint = _checkpoint(engine, task, job, TaskStatus.VIDEO_CUTTING) + checkpoint.begin_step( + TaskStatus.VIDEO_CUTTING.value, + baseline=engine._checkpoint_baseline(task["id"], TaskStatus.VIDEO_CUTTING), + ) + now = task_service._now_iso() + with get_connection() as connection: + connection.execute( + """ + INSERT INTO cut_runs ( + id, task_id, run_number, status, is_active, created_at, updated_at + ) VALUES ('checkpoint-cut-run', ?, 1, 'completed', 1, ?, ?) + """, + (task["id"], now, now), + ) + connection.commit() + _insert_output_clip( + task["id"], + "checkpoint-cut-output", + output_path, + cut_run_id="checkpoint-cut-run", + clip_candidate_id="checkpoint-cut-candidate", + ) + engine._cut_video = Mock(side_effect=AssertionError("FFmpeg 不应重复执行")) + engine._prepare_subtitle_drafts = Mock(side_effect=RuntimeError("stop after cut recovery")) + result = engine.run( + task["id"], + start_step=TaskStatus.VIDEO_CUTTING, + job_id=job["id"], + ) + + assert result["failed_step"] == TaskStatus.SUBTITLE_DRAFTING.value + engine._cut_video.assert_not_called() + stored = job_service.get_job(job["id"])["checkpoint_json"] + assert stored["completed_steps"] == [TaskStatus.VIDEO_CUTTING.value] + assert stored["steps"][TaskStatus.VIDEO_CUTTING.value]["recovered"] is True + + +def test_cut_checkpoint_rejects_empty_file_and_wrong_run() -> None: + task = _create_auto_task("test-pipeline-checkpoint-cut-evidence") + engine = PipelineEngine() + paths = get_artifact_paths(task["id"]) + output_path = paths["clips_dir"] / "run_0001_checkpoint-cut-evidence" / "clip.mp4" + output_path.parent.mkdir(parents=True, exist_ok=True) + output_path.write_bytes(b"valid-video-evidence") + _insert_candidate(task["id"], "checkpoint-cut-evidence-candidate") + now = task_service._now_iso() + with get_connection() as connection: + connection.execute( + """ + INSERT INTO cut_runs ( + id, task_id, run_number, status, is_active, created_at, updated_at + ) VALUES ('checkpoint-cut-evidence-run', ?, 1, 'completed', 1, ?, ?) + """, + (task["id"], now, now), + ) + connection.commit() + _insert_output_clip( + task["id"], + "checkpoint-cut-evidence-output", + output_path, + cut_run_id="checkpoint-cut-evidence-run", + clip_candidate_id="checkpoint-cut-evidence-candidate", + ) + outputs = engine._checkpoint_outputs( + task["id"], + TaskStatus.VIDEO_CUTTING, + {"cut_run_id": "checkpoint-cut-evidence-run", "cut_run_number": 1}, + ) + + output_path.write_bytes(b"") + with pytest.raises(PipelineCheckpointError, match="文件为空"): + engine._restore_checkpoint_step(task["id"], TaskStatus.VIDEO_CUTTING, outputs) + + output_path.write_bytes(b"valid-video-evidence") + with get_connection() as connection: + connection.execute( + "UPDATE output_clip SET cut_run_id = 'another-run' WHERE id = ?", + ("checkpoint-cut-evidence-output",), + ) + connection.commit() + with pytest.raises(PipelineCheckpointError, match="不属于当前 cut run"): + engine._restore_checkpoint_step(task["id"], TaskStatus.VIDEO_CUTTING, outputs) + + +def test_interrupted_cut_step_reruns_when_selection_changed() -> None: + task = _create_auto_task("test-pipeline-checkpoint-cut-input-changed") + engine = PipelineEngine() + job = _claim_auto_job(task["id"], TaskStatus.VIDEO_CUTTING, "cut-input-owner") + paths = get_artifact_paths(task["id"]) + output_path = paths["clips_dir"] / "run_0001_checkpoint-cut-stale" / "clip.mp4" + output_path.parent.mkdir(parents=True, exist_ok=True) + output_path.write_bytes(b"video") + _insert_candidate(task["id"], "checkpoint-cut-old-candidate") + + with job_service.job_lease_context(job["id"], "cut-input-owner", job["lease_token"]): + checkpoint = _checkpoint(engine, task, job, TaskStatus.VIDEO_CUTTING) + checkpoint.begin_step( + TaskStatus.VIDEO_CUTTING.value, + baseline=engine._checkpoint_baseline(task["id"], TaskStatus.VIDEO_CUTTING), + ) + now = task_service._now_iso() + with get_connection() as connection: + connection.execute( + "UPDATE clip_candidates SET enabled = 0, updated_at = ? WHERE task_id = ?", + (now, task["id"]), + ) + connection.commit() + _insert_candidate(task["id"], "checkpoint-cut-new-candidate") + with get_connection() as connection: + connection.execute( + """ + INSERT INTO cut_runs ( + id, task_id, run_number, status, is_active, created_at, updated_at + ) VALUES ('checkpoint-cut-stale', ?, 1, 'completed', 1, ?, ?) + """, + (task["id"], now, now), + ) + connection.commit() + _insert_output_clip( + task["id"], + "checkpoint-cut-stale-output", + output_path, + cut_run_id="checkpoint-cut-stale", + clip_candidate_id="checkpoint-cut-new-candidate", + ) + engine._cut_video = Mock(side_effect=RuntimeError("ffmpeg rerun required")) + result = engine.run( + task["id"], + start_step=TaskStatus.VIDEO_CUTTING, + job_id=job["id"], + ) + + assert result["failed_step"] == TaskStatus.VIDEO_CUTTING.value + assert result["last_error"] == "ffmpeg rerun required" + engine._cut_video.assert_called_once() + stored = job_service.get_job(job["id"])["checkpoint_json"] + assert stored["completed_steps"] == [] + assert stored["steps"][TaskStatus.VIDEO_CUTTING.value]["state"] == "failed" + + +def test_completed_subtitle_checkpoint_keeps_manual_review_gate(tmp_path: Path) -> None: + task = _create_auto_task("test-pipeline-checkpoint-subtitle") + engine = PipelineEngine() + job = _claim_auto_job(task["id"], TaskStatus.SUBTITLE_DRAFTING, "subtitle-checkpoint-owner") + with get_connection() as connection: + connection.execute( + "UPDATE tasks SET status = ? WHERE id = ?", + (TaskStatus.PENDING_SUBTITLE_REVIEW.value, task["id"]), + ) + connection.commit() + with job_service.job_lease_context(job["id"], "subtitle-checkpoint-owner", job["lease_token"]): + checkpoint = _checkpoint(engine, task, job, TaskStatus.SUBTITLE_DRAFTING) + checkpoint.begin_step(TaskStatus.SUBTITLE_DRAFTING.value) + checkpoint.complete_step(TaskStatus.SUBTITLE_DRAFTING.value, outputs={"track_ids": ["track"]}) + engine._restore_checkpoint_step = Mock(return_value={"status": "pending_subtitle_review"}) + engine._prepare_subtitle_drafts = Mock(side_effect=AssertionError("字幕草稿不应重复生成")) + engine._generate_metadata = Mock(side_effect=AssertionError("人工审核前不得继续文案步骤")) + engine._write_task_summary = Mock(return_value={"summary_path": str(tmp_path / "summary.json")}) + result = engine.run( + task["id"], + start_step=TaskStatus.SUBTITLE_DRAFTING, + job_id=job["id"], + ) + + assert result["status"] == "pending_subtitle_review" + engine._prepare_subtitle_drafts.assert_not_called() + engine._generate_metadata.assert_not_called() diff --git a/tests/test_pipeline_state_stability.py b/tests/test_pipeline_state_stability.py new file mode 100644 index 0000000..7ba124b --- /dev/null +++ b/tests/test_pipeline_state_stability.py @@ -0,0 +1,387 @@ +"""P1.3:全自动流水线取消、终态和失败证据。""" + +from __future__ import annotations + +import json +from unittest.mock import Mock + +import pytest + +from app.db.database import get_connection, init_db +from app.models.task import TaskCreate, TaskStatus +from app.services import job_service, task_service +from app.services.pipeline_engine import PipelineEngine +from app.services.task_lifecycle_service import create_task_record +from app.services.task_service import get_task + + +@pytest.fixture(autouse=True) +def cleanup_pipeline_data(): + yield + with get_connection() as connection: + connection.execute("DELETE FROM publish_jobs WHERE task_id LIKE 'test-pipeline-state-%'") + connection.execute("DELETE FROM workflow_jobs WHERE task_id LIKE 'test-pipeline-state-%'") + connection.execute("DELETE FROM output_clip WHERE task_id LIKE 'test-pipeline-state-%'") + connection.execute("DELETE FROM tasks WHERE id LIKE 'test-pipeline-state-%'") + connection.commit() + + +def _create_auto_task(task_id: str) -> dict: + create_task_record( + TaskCreate( + task_name=task_id, + source_type="upload", + platform="general", + selection_profile="general", + auto_mode=True, + ), + task_id=task_id, + task_dir_name=task_id, + ) + return get_task(task_id, include_video_probe=False) + + +def _engine_for_last_step(monkeypatch, task_id: str) -> PipelineEngine: + engine = PipelineEngine() + monkeypatch.setattr(engine, "_create_publish_jobs", Mock(return_value={"created_count": 1})) + monkeypatch.setattr( + engine, + "_write_task_summary", + Mock(return_value={"summary_path": f"C:/tmp/{task_id}.json"}), + ) + return engine + + +def test_success_stays_ready_to_publish(monkeypatch): + task = _create_auto_task("test-pipeline-state-ready") + engine = _engine_for_last_step(monkeypatch, task["id"]) + + result = engine.run(task["id"], start_step=TaskStatus.PUBLISH_JOB_CREATING) + + assert result["status"] == "ready_to_publish" + assert get_task(task["id"], include_video_probe=False)["status"] == TaskStatus.READY_TO_PUBLISH.value + assert task_service.get_task_live_status(task["id"])["should_poll"] is False + + +def test_cancel_after_step_does_not_write_ready(monkeypatch): + task = _create_auto_task("test-pipeline-state-cancel") + job, _created = job_service.create_or_get_active_job( + task_id=task["id"], job_type=job_service.JOB_TYPE_AUTO_PIPELINE + ) + claimed = job_service.claim_job(job["id"], "cancel-after-step-owner") + assert claimed is not None + engine = _engine_for_last_step(monkeypatch, task["id"]) + cancelled = iter([False, True]) + monkeypatch.setattr(job_service, "is_cancel_requested", lambda _job_id: next(cancelled)) + monkeypatch.setattr(job_service, "update_job_progress", Mock()) + monkeypatch.setattr(job_service, "heartbeat_job", Mock()) + + with job_service.job_lease_context( + claimed["id"], + "cancel-after-step-owner", + claimed["lease_token"], + ): + result = engine.run( + task["id"], + start_step=TaskStatus.PUBLISH_JOB_CREATING, + job_id=job["id"], + ) + + assert result["status"] == "cancelled" + assert get_task(task["id"], include_video_probe=False)["status"] == TaskStatus.CANCELLED.value + + +def test_summary_write_failure_does_not_hide_business_failure(monkeypatch): + task = _create_auto_task("test-pipeline-state-summary") + engine = PipelineEngine() + monkeypatch.setattr(engine, "_create_publish_jobs", Mock(side_effect=ValueError("原始业务错误"))) + monkeypatch.setattr(engine, "_write_task_summary", Mock(side_effect=OSError("磁盘暂不可写"))) + + result = engine.run(task["id"], start_step=TaskStatus.PUBLISH_JOB_CREATING) + + assert result["status"] == "failed" + assert result["last_error"] == "原始业务错误" + assert result["summary_path"] == "" + assert get_task(task["id"], include_video_probe=False)["last_error"] == "原始业务错误" + + +def test_request_cancel_marks_auto_task_cancelled(): + task = _create_auto_task("test-pipeline-state-request-cancel") + job, _created = job_service.create_or_get_active_job( + task_id=task["id"], job_type=job_service.JOB_TYPE_AUTO_PIPELINE + ) + + cancelled = job_service.request_job_cancel(job["id"]) + + assert cancelled["status"] == job_service.JOB_STATUS_CANCELLED + assert get_task(task["id"], include_video_probe=False)["status"] == TaskStatus.CANCELLED.value + + +def test_cancelled_status_survives_database_initialization(): + task = _create_auto_task("test-pipeline-state-restart") + job, _created = job_service.create_or_get_active_job( + task_id=task["id"], job_type=job_service.JOB_TYPE_AUTO_PIPELINE + ) + job_service.request_job_cancel(job["id"]) + + init_db() + + assert get_task(task["id"], include_video_probe=False)["status"] == TaskStatus.CANCELLED.value + + +def test_cancel_requested_running_job_cannot_be_marked_completed(): + task = _create_auto_task("test-pipeline-state-terminal-race") + job, _created = job_service.create_or_get_active_job( + task_id=task["id"], job_type=job_service.JOB_TYPE_AUTO_PIPELINE + ) + claimed = job_service.claim_job(job["id"], "test-owner") + token = claimed["lease_token"] + job_service.request_job_cancel(job["id"]) + + with job_service.job_lease_context(job["id"], "test-owner", token): + with pytest.raises(job_service.JobLeaseLostError): + job_service.mark_job_completed(job["id"], {"status": "ready_to_publish"}) + + current = job_service.get_job(job["id"]) + assert current["status"] == job_service.JOB_STATUS_RUNNING + assert current["cancel_requested"] == 1 + assert get_task(task["id"], include_video_probe=False)["status"] == TaskStatus.CANCELLED.value + + +def test_cancel_between_final_check_and_ready_write_wins(monkeypatch): + task = _create_auto_task("test-pipeline-state-final-race") + job, _created = job_service.create_or_get_active_job( + task_id=task["id"], job_type=job_service.JOB_TYPE_AUTO_PIPELINE + ) + claimed = job_service.claim_job(job["id"], "final-race-owner") + assert claimed is not None + engine = _engine_for_last_step(monkeypatch, task["id"]) + monkeypatch.setattr(job_service, "update_job_progress", Mock()) + monkeypatch.setattr(job_service, "heartbeat_job", Mock()) + monkeypatch.setattr( + engine, + "_checkpoint_outputs", + Mock( + return_value={ + "created_ids": ["checkpoint-publish"], + "skipped_ids": [], + "created_count": 1, + "skipped_count": 0, + "schedule_input": {"sha256": "test"}, + } + ), + ) + checks = 0 + + def cancel_after_final_check(_job_id): + nonlocal checks + checks += 1 + if checks == 3: + job_service.request_job_cancel(job["id"]) + + monkeypatch.setattr(engine, "_raise_if_cancelled", cancel_after_final_check) + + with job_service.job_lease_context(job["id"], "final-race-owner", claimed["lease_token"]): + result = engine.run( + task["id"], + start_step=TaskStatus.PUBLISH_JOB_CREATING, + job_id=job["id"], + ) + + assert result["status"] == "cancelled" + assert get_task(task["id"], include_video_probe=False)["status"] == TaskStatus.CANCELLED.value + + +def test_cancel_pipeline_cancels_only_jobs_created_by_current_run(): + task = _create_auto_task("test-pipeline-state-publish-cleanup") + now = task_service._now_iso() + with get_connection() as connection: + connection.execute( + """ + INSERT INTO output_clip ( + id, task_id, output_file_path, output_file_name, status, + is_active, created_at, updated_at + ) VALUES ('pipeline-output', ?, '', 'clip.mp4', 'completed', 1, ?, ?) + """, + (task["id"], now, now), + ) + for job_id, platform in ( + ("pipeline-created-job", "douyin"), + ("pipeline-existing-job", "bilibili"), + ): + connection.execute( + """ + INSERT INTO publish_jobs ( + id, task_id, output_clip_id, platform, status, created_at, updated_at + ) VALUES (?, ?, 'pipeline-output', ?, 'WAITING', ?, ?) + """, + (job_id, task["id"], platform, now, now), + ) + connection.commit() + + cancelled = PipelineEngine()._cancel_unpublished_auto_jobs( + task["id"], + {TaskStatus.PUBLISH_JOB_CREATING.value: {"created": [{"id": "pipeline-created-job"}]}}, + ) + + with get_connection() as connection: + rows = { + row["id"]: row["status"] + for row in connection.execute( + "SELECT id, status FROM publish_jobs WHERE task_id = ?", (task["id"],) + ).fetchall() + } + assert cancelled == 1 + assert rows == { + "pipeline-created-job": "CANCELLED", + "pipeline-existing-job": "WAITING", + } + + +def test_old_worker_cannot_write_ready_after_lease_takeover(): + task = _create_auto_task("test-pipeline-state-ready-fence") + job, _created = job_service.create_or_get_active_job( + task_id=task["id"], job_type=job_service.JOB_TYPE_AUTO_PIPELINE + ) + old_claim = job_service.claim_job(job["id"], "old-owner") + with get_connection() as connection: + connection.execute( + "UPDATE workflow_jobs SET lease_expires_at = '2000-01-01T00:00:00+00:00' WHERE id = ?", + (job["id"],), + ) + connection.commit() + new_claim = job_service.claim_job(job["id"], "new-owner") + assert new_claim is not None + + with job_service.job_lease_context(job["id"], "old-owner", old_claim["lease_token"]): + with pytest.raises(job_service.JobLeaseLostError): + PipelineEngine()._mark_ready_to_publish(task["id"], job["id"]) + + assert get_task(task["id"], include_video_probe=False)["status"] == TaskStatus.CREATED.value + + +def test_old_worker_cannot_cancel_publish_draft_after_lease_takeover(): + task = _create_auto_task("test-pipeline-state-cancel-publish-fence") + job, _created = job_service.create_or_get_active_job( + task_id=task["id"], job_type=job_service.JOB_TYPE_AUTO_PIPELINE + ) + old_claim = job_service.claim_job(job["id"], "old-cancel-owner") + assert old_claim is not None + now = task_service._now_iso() + with get_connection() as connection: + connection.execute( + """ + INSERT INTO output_clip ( + id, task_id, output_file_path, output_file_name, status, + is_active, created_at, updated_at + ) VALUES ('pipeline-fenced-output', ?, '', 'clip.mp4', 'completed', 1, ?, ?) + """, + (task["id"], now, now), + ) + connection.execute( + """ + INSERT INTO publish_jobs ( + id, task_id, output_clip_id, platform, status, created_at, updated_at + ) VALUES ('pipeline-fenced-publish', ?, 'pipeline-fenced-output', 'douyin', 'WAITING', ?, ?) + """, + (task["id"], now, now), + ) + connection.execute( + "UPDATE workflow_jobs SET lease_expires_at = '2000-01-01T00:00:00+00:00' WHERE id = ?", + (job["id"],), + ) + connection.commit() + new_claim = job_service.claim_job(job["id"], "new-cancel-owner") + assert new_claim is not None + + with job_service.job_lease_context(job["id"], "old-cancel-owner", old_claim["lease_token"]): + with pytest.raises(job_service.JobLeaseLostError): + PipelineEngine()._cancel_unpublished_auto_jobs( + task["id"], + { + "workflow_job_id": job["id"], + TaskStatus.PUBLISH_JOB_CREATING.value: { + "created": [{"id": "pipeline-fenced-publish"}] + }, + }, + ) + + with get_connection() as connection: + status = connection.execute( + "SELECT status FROM publish_jobs WHERE id = 'pipeline-fenced-publish'" + ).fetchone()["status"] + assert status == "WAITING" + + +def test_expired_cancel_requested_job_is_recovered_as_cancelled(): + task = _create_auto_task("test-pipeline-state-cancel-recovery") + job, _created = job_service.create_or_get_active_job( + task_id=task["id"], job_type=job_service.JOB_TYPE_AUTO_PIPELINE + ) + claimed = job_service.claim_job(job["id"], "cancel-recovery-owner") + assert claimed is not None + job_service.request_job_cancel(job["id"]) + with get_connection() as connection: + connection.execute( + "UPDATE workflow_jobs SET lease_expires_at = '2000-01-01T00:00:00+00:00' WHERE id = ?", + (job["id"],), + ) + connection.commit() + + job_service.claim_next_job("recovery-worker") + + recovered = job_service.get_job(job["id"]) + assert recovered["status"] == job_service.JOB_STATUS_CANCELLED + assert recovered["lease_owner"] is None + assert recovered["lease_token"] is None + + +def test_job_cancel_cleans_only_publish_jobs_linked_to_same_pipeline(): + task = _create_auto_task("test-pipeline-state-linked-publish") + job, _created = job_service.create_or_get_active_job( + task_id=task["id"], job_type=job_service.JOB_TYPE_AUTO_PIPELINE + ) + now = task_service._now_iso() + with get_connection() as connection: + connection.execute( + """ + INSERT INTO output_clip ( + id, task_id, output_file_path, output_file_name, status, + is_active, created_at, updated_at + ) VALUES ('linked-pipeline-output', ?, '', 'clip.mp4', 'completed', 1, ?, ?) + """, + (task["id"], now, now), + ) + for publish_job_id, workflow_job_id, platform in ( + ("linked-pipeline-publish", job["id"], "douyin"), + ("other-pipeline-publish", "another-workflow-job", "bilibili"), + ): + provider_response = json.dumps( + {"source": "auto_pipeline", "workflow_job_id": workflow_job_id}, + ensure_ascii=False, + ) + connection.execute( + """ + INSERT INTO publish_jobs ( + id, task_id, output_clip_id, platform, status, + provider_response, created_at, updated_at + ) VALUES (?, ?, 'linked-pipeline-output', ?, 'SCHEDULED', ?, ?, ?) + """, + (publish_job_id, task["id"], platform, provider_response, now, now), + ) + connection.commit() + + job_service.request_job_cancel(job["id"]) + + with get_connection() as connection: + statuses = { + row["id"]: row["status"] + for row in connection.execute( + "SELECT id, status FROM publish_jobs WHERE task_id = ?", + (task["id"],), + ).fetchall() + } + assert statuses == { + "linked-pipeline-publish": "CANCELLED", + "other-pipeline-publish": "SCHEDULED", + } diff --git a/tests/test_provider_resilience.py b/tests/test_provider_resilience.py new file mode 100644 index 0000000..ccfbd10 --- /dev/null +++ b/tests/test_provider_resilience.py @@ -0,0 +1,349 @@ +from __future__ import annotations + +from io import BytesIO +import json +from pathlib import Path +from urllib.error import HTTPError, URLError + +import pytest + +from app.services.ai import ai_clip_analyzer +from app.services.ai.ai_clip_analyzer import AIAnalysisError, AnalysisRequest +from app.services.ai.base import ( + AIProviderError, + ProviderConfig, + extract_chat_completion_text, + generate_json_with_safe_retry, + post_json, +) +from app.services.ai.local_model_provider import LocalModelProvider +from app.services.ai.variety_comedy_analyzer import _generate_payload +from app.services import transcript_service + + +class _Response: + def __init__(self, body: str, headers: dict[str, str] | None = None): + self._body = body.encode("utf-8") + self.headers = headers or {} + + def __enter__(self): + return self + + def __exit__(self, *_args): + return False + + def read(self) -> bytes: + return self._body + + +def _http_error(code: int, body: str = "error", headers: dict[str, str] | None = None) -> HTTPError: + return HTTPError("https://provider.invalid", code, "error", headers or {}, BytesIO(body.encode("utf-8"))) + + +def test_ai_transport_classifies_429_as_safe_retry(monkeypatch): + monkeypatch.setattr( + "app.services.ai.base.urlopen", + lambda *_args, **_kwargs: (_ for _ in ()).throw(_http_error(429, headers={"Retry-After": "7"})), + ) + with pytest.raises(AIProviderError) as captured: + post_json("https://provider.invalid", {"x": 1}, "", 1) + assert captured.value.category == "rate_limited" + assert captured.value.safe_to_retry is True + assert captured.value.billing_uncertain is False + assert captured.value.retry_after_seconds == 7 + + +def test_ai_transport_does_not_mark_5xx_or_timeout_safe(monkeypatch): + monkeypatch.setattr( + "app.services.ai.base.urlopen", + lambda *_args, **_kwargs: (_ for _ in ()).throw(_http_error(503)), + ) + with pytest.raises(AIProviderError) as server_error: + post_json("https://provider.invalid", {}, "", 1) + assert server_error.value.safe_to_retry is False + assert server_error.value.billing_uncertain is True + + monkeypatch.setattr( + "app.services.ai.base.urlopen", + lambda *_args, **_kwargs: (_ for _ in ()).throw(URLError(TimeoutError("timed out"))), + ) + with pytest.raises(AIProviderError) as timeout_error: + post_json("https://provider.invalid", {}, "", 1) + assert timeout_error.value.category == "timeout" + assert timeout_error.value.safe_to_retry is False + assert timeout_error.value.billing_uncertain is True + + +def test_ai_transport_error_never_exposes_provider_body(monkeypatch): + secret = "provider-secret-sentinel" + monkeypatch.setattr( + "app.services.ai.base.urlopen", + lambda *_args, **_kwargs: (_ for _ in ()).throw( + _http_error(401, body=json.dumps({"authorization": secret, "prompt": secret})) + ), + ) + + with pytest.raises(AIProviderError) as captured: + post_json("https://provider.invalid", {}, "", 1) + + assert captured.value.http_status == 401 + assert secret not in str(captured.value) + + +def test_ai_transport_invalid_json_never_exposes_response_body(monkeypatch): + secret = "invalid-json-secret-sentinel" + monkeypatch.setattr("app.services.ai.base.urlopen", lambda *_args, **_kwargs: _Response(secret)) + + with pytest.raises(AIProviderError) as captured: + post_json("https://provider.invalid", {}, "", 1) + + assert captured.value.category == "invalid_response_json" + assert secret not in str(captured.value) + + +def test_shared_retry_only_repeats_explicitly_safe_failures(): + class Provider: + def __init__(self, *, safe: bool): + self.calls = 0 + self.safe = safe + + def generate_json(self, _prompt: str) -> str: + self.calls += 1 + if self.calls < 3: + raise AIProviderError("temporary", safe_to_retry=self.safe, retry_after_seconds=0) + return "{}" + + safe_provider = Provider(safe=True) + assert generate_json_with_safe_retry(safe_provider, "prompt", sleep_fn=lambda _seconds: None) == "{}" + assert safe_provider.calls == 3 + + ambiguous_provider = Provider(safe=False) + with pytest.raises(AIProviderError): + generate_json_with_safe_retry(ambiguous_provider, "prompt", sleep_fn=lambda _seconds: None) + assert ambiguous_provider.calls == 1 + + +@pytest.mark.parametrize( + "body, category", + [("", "empty_response"), ("not-json", "invalid_response_json"), ("[]", "invalid_response_schema")], +) +def test_ai_transport_rejects_empty_or_invalid_json_without_retry(monkeypatch, body, category): + monkeypatch.setattr("app.services.ai.base.urlopen", lambda *_args, **_kwargs: _Response(body)) + with pytest.raises(AIProviderError) as captured: + post_json("https://provider.invalid", {}, "", 1) + assert captured.value.category == category + assert captured.value.safe_to_retry is False + assert captured.value.billing_uncertain is True + + +def test_chat_completion_rejects_non_object_message_as_uncertain_schema_error(): + with pytest.raises(AIProviderError) as captured: + extract_chat_completion_text({"choices": [{"message": "broken"}]}) + assert captured.value.category == "invalid_response_schema" + assert captured.value.billing_uncertain is True + + +class _CountingInvalidProvider: + name = "remote" + + def __init__(self): + self.calls = 0 + + def generate_json(self, _prompt: str, retry_instruction: str | None = None) -> str: + del retry_instruction + self.calls += 1 + return "not-json" + + +def test_general_analysis_invalid_json_does_not_call_provider_twice(monkeypatch, tmp_path: Path): + transcript = tmp_path / "transcript.md" + transcript.write_text("00:00:00 - 00:01:00 测试正文", encoding="utf-8") + provider = _CountingInvalidProvider() + monkeypatch.setattr(ai_clip_analyzer, "build_provider", lambda *_args, **_kwargs: provider) + request = AnalysisRequest( + task_id="no-double-charge", + transcript_path=transcript, + max_clip_duration_minutes=3, + target_clip_count=1, + ai_preference="", + provider_name="remote", + ) + with pytest.raises(AIAnalysisError, match="没有生成可用候选"): + ai_clip_analyzer.analyze_task_transcript(request) + assert provider.calls == 1 + + +def test_variety_invalid_json_does_not_call_provider_twice(): + provider = _CountingInvalidProvider() + with pytest.raises(AIAnalysisError, match="JSON 解析失败"): + _generate_payload(provider, "prompt", expected_key="moments") + assert provider.calls == 1 + + +def test_local_protocol_fallback_only_runs_for_missing_endpoint(monkeypatch): + provider = LocalModelProvider( + ProviderConfig( + base_url="http://127.0.0.1:11434/v1", + api_key="", + model="local", + protocol="responses", + fallback_protocol="chat_completions", + timeout_seconds=1, + ) + ) + calls: list[str] = [] + + def fail_ambiguous(*_args): + calls.append("responses") + raise AIProviderError("server", http_status=503, billing_uncertain=True) + + monkeypatch.setattr(provider, "_responses", fail_ambiguous) + monkeypatch.setattr(provider, "_chat_completions", lambda *_args: calls.append("chat") or "{}") + with pytest.raises(AIProviderError): + provider.generate_json("prompt") + assert calls == ["responses"] + + calls.clear() + monkeypatch.setattr( + provider, + "_responses", + lambda *_args: (_ for _ in ()).throw(AIProviderError("missing", http_status=404)), + ) + assert provider.generate_json("prompt") == "{}" + assert calls == ["chat"] + + +def test_volcengine_retries_429_with_same_request_id(monkeypatch): + monkeypatch.setattr(transcript_service, "_ensure_volcengine_configured", lambda: None) + monkeypatch.setattr(transcript_service, "_build_volcengine_flash_payload", lambda _path: {}) + requests = [] + + def fake_urlopen(request, **_kwargs): + requests.append(request) + if len(requests) < 3: + raise _http_error(429, headers={"Retry-After": "0"}) + return _Response(json.dumps({"result": {"text": "转写成功"}})) + + monkeypatch.setattr(transcript_service, "urlopen", fake_urlopen) + segments = transcript_service.transcribe_audio_with_volcengine_flash( + Path("unused.audio"), + request_id="stable-request-id", + sleep_fn=lambda _seconds: None, + ) + assert [segment.text for segment in segments] == ["转写成功"] + assert len(requests) == 3 + assert {request.headers["X-api-request-id"] for request in requests} == {"stable-request-id"} + + +def test_volcengine_5xx_and_bad_schema_are_not_retried(monkeypatch): + monkeypatch.setattr(transcript_service, "_ensure_volcengine_configured", lambda: None) + monkeypatch.setattr(transcript_service, "_build_volcengine_flash_payload", lambda _path: {}) + calls = 0 + + def fail_5xx(*_args, **_kwargs): + nonlocal calls + calls += 1 + raise _http_error(500) + + monkeypatch.setattr(transcript_service, "urlopen", fail_5xx) + with pytest.raises(transcript_service.RemoteTranscriptionError) as server_error: + transcript_service.transcribe_audio_with_volcengine_flash(Path("unused.audio"), sleep_fn=lambda _: None) + assert server_error.value.billing_uncertain is True + assert calls == 1 + + calls = 0 + + def bad_schema(*_args, **_kwargs): + nonlocal calls + calls += 1 + return _Response("{}") + + monkeypatch.setattr(transcript_service, "urlopen", bad_schema) + with pytest.raises(transcript_service.RemoteTranscriptionError, match="缺少 result"): + transcript_service.transcribe_audio_with_volcengine_flash(Path("unused.audio"), sleep_fn=lambda _: None) + assert calls == 1 + + +def test_volcengine_invalid_success_fields_are_billing_uncertain(monkeypatch): + monkeypatch.setattr(transcript_service, "_ensure_volcengine_configured", lambda: None) + monkeypatch.setattr(transcript_service, "_build_volcengine_flash_payload", lambda _path: {}) + monkeypatch.setattr( + transcript_service, + "urlopen", + lambda *_args, **_kwargs: _Response( + json.dumps( + { + "result": { + "utterances": [ + {"text": "已返回文本", "start_time": 0, "end_time": 1000, "confidence": "bad"} + ] + } + } + ) + ), + ) + with pytest.raises(transcript_service.RemoteTranscriptionError) as captured: + transcript_service.transcribe_audio_with_volcengine_flash(Path("unused.audio"), sleep_fn=lambda _: None) + assert captured.value.category == "invalid_response_schema" + assert captured.value.billing_uncertain is True + + +def test_transcript_cancellation_is_not_wrapped_as_provider_failure(monkeypatch, tmp_path): + monkeypatch.setattr( + transcript_service, + "transcribe_audio_with_provider", + lambda *_args, **_kwargs: (_ for _ in ()).throw( + transcript_service.TranscriptCancelledError("cancelled") + ), + ) + with pytest.raises(transcript_service.TranscriptCancelledError, match="cancelled"): + transcript_service.transcribe_audio_with_configured_provider( + tmp_path / "audio.wav", + tmp_path, + tmp_path / "progress.json", + provider="local", + ) + + +def test_volcengine_response_reset_is_billing_uncertain(monkeypatch): + monkeypatch.setattr(transcript_service, "_ensure_volcengine_configured", lambda: None) + monkeypatch.setattr(transcript_service, "_build_volcengine_flash_payload", lambda _path: {}) + + class ResetResponse(_Response): + def read(self) -> bytes: + raise ConnectionResetError("reset while reading") + + monkeypatch.setattr( + transcript_service, + "urlopen", + lambda *_args, **_kwargs: ResetResponse(""), + ) + with pytest.raises(transcript_service.RemoteTranscriptionError) as captured: + transcript_service.transcribe_audio_with_volcengine_flash(Path("unused.audio"), sleep_fn=lambda _: None) + assert captured.value.category == "network_error" + assert captured.value.billing_uncertain is True + assert captured.value.safe_to_retry is False + + +def test_remote_checkpoint_save_failure_becomes_uncertain(): + class Checkpoint: + def __init__(self): + self.uncertain = "" + + def save_completed(self, *_args, **_kwargs): + raise RuntimeError("database locked") + + def save_uncertain(self, _chunk_index: int, error: str): + self.uncertain = error + + checkpoint = Checkpoint() + with pytest.raises( + transcript_service.RemoteTranscriptionResultUncertainError, + match="checkpoint 保存失败", + ): + transcript_service._save_remote_checkpoint_completed( + checkpoint, + 1, + [transcript_service.TranscriptSegment(0, 1, "ok")], + ) + assert "普通重试不会再次请求" in checkpoint.uncertain diff --git a/tests/test_publish_center_browser.py b/tests/test_publish_center_browser.py index d0f40b8..38e8b20 100644 --- a/tests/test_publish_center_browser.py +++ b/tests/test_publish_center_browser.py @@ -253,8 +253,9 @@ def fulfill_metadata(route): group_select.check() assert first_content.locator("[data-publish-select]").is_checked() assert second_content.locator("[data-publish-select]").is_checked() + assert page.locator("[data-batch-ai]").inner_text() == "AI 重写已选文案" page.locator("[data-batch-ai]").click() - page.locator("#send-center-message").filter(has_text="批量 AI 重写完成:成功 2 条").wait_for() + page.locator("#send-center-message").filter(has_text="已选文案 AI 重写完成:成功 2 条").wait_for() assert first_content.locator('[name="description"]').input_value() == "陈汉典刚说自己像潘玮柏,小S立刻给出另一答案" assert second_content.locator('[name="tags"]').input_value() == "综艺,高光,小S,反转" first_content.locator("[data-publish-select]").uncheck() @@ -272,6 +273,7 @@ def fulfill_metadata(route): group_select.uncheck() page.locator('[data-center-tab="schedule"]').click() + assert page.locator("[data-batch-ai]").is_hidden() assert page.locator('[data-schedule-calendar] .publish-calendar-day').count() == 42 assert "抖音" in page.locator("[data-calendar-title]").inner_text() assert page.locator( diff --git a/tests/test_publish_center_cleanup.py b/tests/test_publish_center_cleanup.py index c69eb24..a0c20b9 100644 --- a/tests/test_publish_center_cleanup.py +++ b/tests/test_publish_center_cleanup.py @@ -14,9 +14,33 @@ def test_selection_bar_has_no_legacy_batch_send_but_keeps_scheduling() -> None: assert "data-open-schedule-drawer" in selection_bar assert "data-apply-batch-target" in selection_bar assert "data-batch-ai" in selection_bar + assert "AI 重写已选文案" in selection_bar assert "data-publish-now" in template +def test_publish_ai_and_maintenance_actions_are_scoped_to_content_preparation() -> None: + template = (PROJECT_ROOT / "app/templates/publish.html").read_text(encoding="utf-8") + publish_script = (PROJECT_ROOT / "app/static/js/publish-center.js").read_text(encoding="utf-8") + page_heading = template.split('
', 1)[1].split( + "
", 1 + )[0] + content_panel = template.split('data-center-panel="content"', 1)[1].split( + 'data-center-panel="schedule"', 1 + )[0] + content_row_macro = template.split("{% macro content_row(job) %}", 1)[1].split( + "{% endmacro %}", 1 + )[0] + + assert "data-supplement-publish-jobs" not in page_heading + assert "data-supplement-publish-jobs" in content_panel + assert "同步遗漏切片" in content_panel + assert "不调用 AI,也不修改已有文案" in content_panel + assert "AI 重写本条文案" in content_row_macro + assert "use_ai=false" in publish_script + assert "/api/publish/jobs/metadata/upgrade-pending-douyin" not in publish_script + assert 'batchAiButton.hidden = tab !== "content"' in publish_script + + def test_legacy_publish_frontend_handlers_are_removed() -> None: global_script = (PROJECT_ROOT / "app/static/js/app.js").read_text(encoding="utf-8") publish_script = (PROJECT_ROOT / "app/static/js/publish-center.js").read_text(encoding="utf-8") diff --git a/tests/test_publish_copy_rules.py b/tests/test_publish_copy_rules.py index d47df26..39aad11 100644 --- a/tests/test_publish_copy_rules.py +++ b/tests/test_publish_copy_rules.py @@ -205,6 +205,38 @@ def test_ai_failure_does_not_overwrite_existing_copy(monkeypatch) -> None: ) +def test_metadata_regeneration_does_not_overwrite_concurrent_publish(monkeypatch) -> None: + job_id = _seed_job(title=VALID_TITLE, description=VALID_DESCRIPTION, tags=VALID_TAGS) + + def generate_after_concurrent_claim(*_args, **_kwargs): + with get_connection() as connection: + connection.execute( + """ + UPDATE publish_jobs + SET title = '并发发布采用的标题', status = 'PUBLISHING', updated_at = ? + WHERE id = ? + """, + (datetime.now(timezone.utc).isoformat(timespec="microseconds"), job_id), + ) + connection.commit() + return { + "title": "过期 AI 标题", + "description": VALID_DESCRIPTION, + "tags": VALID_TAGS, + "source": "ai:test", + "error": "", + } + + monkeypatch.setattr(publish_service, "generate_publish_metadata", generate_after_concurrent_claim) + + with pytest.raises(ValueError, match="内容或状态已经变化"): + publish_service.regenerate_send_job_metadata(job_id, use_ai=True) + + stored = _raw_job(job_id) + assert stored["status"] == "PUBLISHING" + assert stored["title"] == "并发发布采用的标题" + + def test_manual_save_binds_unique_account_and_synchronizes_alias_fields() -> None: account_id = _seed_account() job_id = _seed_job(title=VALID_TITLE, description=VALID_DESCRIPTION, tags=VALID_TAGS) diff --git a/tests/test_publish_fencing.py b/tests/test_publish_fencing.py new file mode 100644 index 0000000..dba2453 --- /dev/null +++ b/tests/test_publish_fencing.py @@ -0,0 +1,1392 @@ +from __future__ import annotations + +import json +import multiprocessing +import threading +import time +from concurrent.futures import ThreadPoolExecutor +from datetime import datetime, timedelta, timezone +from pathlib import Path +from uuid import uuid4 + +import pytest +from fastapi.testclient import TestClient + +from app.core.config import settings +from app.db.database import get_connection, init_db +from app.services import publish_scheduler as scheduler_module +from app.services.publish_repository import PublishRepository +from app.services.publish_scheduler import PublishScheduler +from app.services.publish_executor import execute_publish_job, is_publish_dispatch_active +from app.services.publishers.base import ( + PublishError, + PublishOutcome, + PublishResult, + PublishValidationError, + PublishWorkerUnavailable, + sanitize_provider_response, +) +from app.services.publishers import manual_export as manual_export_module +from app.services.publishers.manual_export import ManualExportPublisher +from app.services.publishers.worker_client import validate_worker_identifier +from scripts import publish_host_worker as worker_module + + +PREFIX = "test-publish-fence-" + + +def _hold_execution_lock_in_child( + state_dir: str, + execution_id: str, + ready, + release, +) -> None: + object.__setattr__(settings, "publish_worker_state_dir", Path(state_dir)) + journal = worker_module.ExecutionJournal(execution_id) + with journal.lock: + if not journal.lock.acquired: + return + ready.set() + release.wait(timeout=10) + + +def _contend_lock_in_child( + lock_path: str, + ready, + start, + release, + results, +) -> None: + token = None + try: + ready.set() + start.wait(timeout=10) + path = Path(lock_path) + token = worker_module._try_create_lock_file(path) + results.put({"acquired": bool(token), "error": ""}) + if token: + release.wait(timeout=10) + except Exception as exc: + results.put({"acquired": False, "error": repr(exc)}) + raise + finally: + worker_module._release_lock_file(Path(lock_path), token) + + +@pytest.fixture(autouse=True) +def clean_publish_fencing_rows(): + init_db() + _cleanup() + yield + _cleanup() + + +def _cleanup() -> None: + with get_connection() as connection: + connection.execute("DELETE FROM publish_jobs WHERE task_id LIKE ?", (f"{PREFIX}%",)) + connection.execute("DELETE FROM output_clip WHERE task_id LIKE ?", (f"{PREFIX}%",)) + connection.execute("DELETE FROM tasks WHERE id LIKE ?", (f"{PREFIX}%",)) + connection.commit() + + +def _iso(seconds: int = 0) -> str: + return (datetime.now(timezone.utc) + timedelta(seconds=seconds)).isoformat(timespec="seconds") + + +def _seed_job( + tmp_path: Path, + *, + status: str = "PUBLISHING", + execution_id: str = "execution-current", + publish_mode: str = "manual_export", +) -> str: + suffix = uuid4().hex[:10] + task_id = f"{PREFIX}{suffix}" + clip_id = f"{PREFIX}clip-{suffix}" + job_id = f"{PREFIX}job-{suffix}" + video = tmp_path / f"{suffix}.mp4" + video.write_bytes(b"fake video") + now = _iso() + with get_connection() as connection: + connection.execute( + """ + INSERT INTO tasks (id, task_name, task_dir_name, platform, status, created_at, updated_at) + VALUES (?, ?, ?, 'douyin', 'COMPLETED', ?, ?) + """, + (task_id, task_id, task_id, now, now), + ) + connection.execute( + """ + INSERT INTO output_clip ( + id, task_id, output_file_path, output_file_name, status, is_active, created_at, updated_at + ) VALUES (?, ?, ?, 'clip.mp4', 'completed', 1, ?, ?) + """, + (clip_id, task_id, str(video), now, now), + ) + connection.execute( + """ + INSERT INTO publish_jobs ( + id, task_id, output_clip_id, clip_id, platform, publish_mode, + video_source, video_file_path, video_path, title, description, caption, + tags, hashtags, risk_flags, scheduled_at, schedule_timezone, timezone, + status, worker_id, execution_id, execution_phase, provider_response, + created_at, updated_at + ) VALUES (?, ?, ?, ?, 'douyin', ?, 'original', ?, ?, + '测试标题', '测试正文', '测试正文', '测试', '测试', '[]', ?, + 'Asia/Shanghai', 'Asia/Shanghai', ?, 'worker-new', ?, 'claimed', ?, ?, ?) + """, + ( + job_id, + task_id, + clip_id, + clip_id, + publish_mode, + str(video), + str(video), + _iso(-60), + status, + execution_id, + json.dumps({"original": True}), + now, + now, + ), + ) + connection.commit() + return job_id + + +def _raw(job_id: str) -> dict: + with get_connection() as connection: + return dict(connection.execute("SELECT * FROM publish_jobs WHERE id = ?", (job_id,)).fetchone()) + + +def _identity(job_id: str) -> dict[str, str]: + job = _raw(job_id) + return { + "job_id": job_id, + "platform": str(job.get("platform") or ""), + "account_id": str(job.get("account_id") or ""), + } + + +def _event_count(job_id: str) -> int: + with get_connection() as connection: + return int( + connection.execute( + "SELECT COUNT(*) FROM publish_job_events WHERE job_id = ?", (job_id,) + ).fetchone()[0] + ) + + +def test_repository_rejects_stale_provider_result_and_phase(tmp_path): + job_id = _seed_job(tmp_path) + repository = PublishRepository() + result = PublishResult( + outcome=PublishOutcome.PUBLISHED, + message="旧执行结果", + remote_video_id="old-video", + provider_response={"token": "must-not-persist"}, + ) + + assert repository.record_provider_result( + job_id, result, expected_execution_id="execution-old" + ) is False + assert repository.update_execution_phase( + job_id, "upload_started", expected_execution_id="execution-old" + ) is False + + stored = _raw(job_id) + assert stored["execution_phase"] == "claimed" + assert json.loads(stored["provider_response"]) == {"original": True} + assert not stored["remote_video_id"] + + +def test_all_terminal_writes_reject_stale_execution_without_events(tmp_path): + job_id = _seed_job(tmp_path) + scheduler = PublishScheduler() + result = PublishResult( + outcome=PublishOutcome.PUBLISHED, + message="旧执行结果", + provider_response={"replacement": True}, + ) + before_events = _event_count(job_id) + + transitions = [ + scheduler._mark_published(job_id, result, expected_execution_id="execution-old"), + scheduler._mark_exported(job_id, result, expected_execution_id="execution-old"), + scheduler._mark_failed( + job_id, "old_failed", "旧执行失败", result, expected_execution_id="execution-old" + ), + scheduler._mark_need_review( + job_id, "old_review", "旧执行需复核", result, expected_execution_id="execution-old" + ), + ] + + assert {item["status"] for item in transitions} == {"skipped"} + stored = _raw(job_id) + assert stored["status"] == "PUBLISHING" + assert stored["execution_id"] == "execution-current" + assert json.loads(stored["provider_response"]) == {"original": True} + assert _event_count(job_id) == before_events + + +def test_recovery_cannot_apply_old_worker_success_to_new_execution(tmp_path): + job_id = _seed_job(tmp_path, execution_id="execution-old") + + class Worker: + @staticmethod + def execution(_execution_id: str) -> dict: + with get_connection() as connection: + connection.execute( + "UPDATE publish_jobs SET execution_id = 'execution-new', worker_id = 'worker-new' WHERE id = ?", + (job_id,), + ) + connection.commit() + return { + "phase": "confirmed_success", + "details": {"outcome": "PUBLISHED", "message": "旧执行声称成功"}, + "identity": _identity(job_id), + } + + recovered = PublishScheduler(worker_client=Worker()).recover_interrupted_jobs() + + assert recovered == 0 + stored = _raw(job_id) + assert stored["status"] == "PUBLISHING" + assert stored["execution_id"] == "execution-new" + assert json.loads(stored["provider_response"]) == {"original": True} + + +def test_recovery_query_failure_never_requeues_from_stale_database_phase(tmp_path): + job_id = _seed_job(tmp_path, execution_id="execution-query-failed") + with get_connection() as connection: + connection.execute( + "UPDATE publish_jobs SET execution_phase = 'received', updated_at = ? WHERE id = ?", + (_iso(-3600), job_id), + ) + connection.commit() + + class Worker: + @staticmethod + def execution(_execution_id: str) -> dict: + raise PublishWorkerUnavailable("Worker 无法查询") + + recovered = PublishScheduler(worker_client=Worker()).recover_interrupted_jobs() + + assert recovered == 1 + stored = _raw(job_id) + assert stored["status"] == "NEED_REVIEW" + assert stored["error_code"] == "interrupted_publish_uncertain" + + +def test_recovery_keeps_execution_pending_while_worker_lock_is_active(tmp_path): + job_id = _seed_job(tmp_path, execution_id="execution-still-running") + with get_connection() as connection: + connection.execute( + "UPDATE publish_jobs SET execution_phase = 'received', updated_at = ? WHERE id = ?", + (_iso(-3600), job_id), + ) + connection.commit() + + class Worker: + @staticmethod + def execution(_execution_id: str) -> dict: + return { + "phase": "browser_opened", + "details": {}, + "in_progress": True, + "identity": _identity(job_id), + } + + recovered = PublishScheduler(worker_client=Worker()).recover_interrupted_jobs() + + assert recovered == 0 + assert _raw(job_id)["status"] == "PUBLISHING" + + +def test_recovery_rejects_worker_terminal_phase_result_mismatch(tmp_path): + job_id = _seed_job(tmp_path, execution_id="execution-mismatched-terminal") + + class Worker: + @staticmethod + def execution(_execution_id: str) -> dict: + return { + "phase": "confirmed_success", + "details": {"outcome": "FAILED", "message": "矛盾终态"}, + "in_progress": False, + "identity": _identity(job_id), + } + + recovered = PublishScheduler(worker_client=Worker()).recover_interrupted_jobs() + + assert recovered == 1 + stored = _raw(job_id) + assert stored["status"] == "NEED_REVIEW" + assert stored["error_code"] == "recovery_result_uncertain" + + +def test_executor_refuses_stale_execution_before_loading_publisher(monkeypatch, tmp_path): + job_id = _seed_job(tmp_path, execution_id="execution-new") + monkeypatch.setattr( + "app.services.publish_executor.get_publisher", + lambda *_args, **_kwargs: pytest.fail("旧 execution 不得进入 Publisher"), + ) + + with pytest.raises(PublishValidationError) as caught: + execute_publish_job(job_id, expected_execution_id="execution-old") + + assert caught.value.error_code == "publish_execution_stale" + + +def test_executor_reserves_dispatch_after_publisher_resolution(monkeypatch, tmp_path): + job_id = _seed_job(tmp_path, execution_id="execution-old") + + class Publisher: + @staticmethod + def publish(_job): + pytest.fail("状态已变化后不得进入真实 Publisher") + + def change_state_before_dispatch(*_args, **_kwargs): + with get_connection() as connection: + connection.execute( + "UPDATE publish_jobs SET status = 'NEED_REVIEW', updated_at = ? WHERE id = ?", + (_iso(1), job_id), + ) + connection.commit() + return Publisher() + + monkeypatch.setattr("app.services.publish_executor.get_publisher", change_state_before_dispatch) + + with pytest.raises(PublishValidationError) as caught: + execute_publish_job(job_id, expected_execution_id="execution-old") + + assert caught.value.error_code == "publish_execution_stale" + + +def test_local_browser_reserves_dispatch_immediately_before_worker_publish(monkeypatch, tmp_path): + job_id = _seed_job( + tmp_path, + execution_id="execution-local-browser", + publish_mode="local_browser", + ) + + def publisher_factory(*_args, before_dispatch, **_kwargs): + class Publisher: + @staticmethod + def publish(_job): + with get_connection() as connection: + connection.execute( + "UPDATE publish_jobs SET status = 'NEED_REVIEW', updated_at = ? WHERE id = ?", + (_iso(1), job_id), + ) + connection.commit() + before_dispatch() + pytest.fail("dispatch CAS 失败后不得请求 Worker /publish") + + return Publisher() + + monkeypatch.setattr("app.services.publish_executor.get_publisher", publisher_factory) + + with pytest.raises(PublishValidationError) as caught: + execute_publish_job(job_id, expected_execution_id="execution-local-browser") + + assert caught.value.error_code == "publish_execution_stale" + assert not is_publish_dispatch_active("execution-local-browser") + + +def test_executor_marks_execution_active_before_dispatch_cas(monkeypatch, tmp_path): + execution_id = "execution-cas-window" + job_id = _seed_job(tmp_path, execution_id=execution_id) + repository = PublishRepository() + original = repository.begin_execution_dispatch + observed: list[bool] = [] + + def guarded_reservation(*args, **kwargs): + observed.append(is_publish_dispatch_active(execution_id)) + return original(*args, **kwargs) + + monkeypatch.setattr(repository, "begin_execution_dispatch", guarded_reservation) + + class Publisher: + @staticmethod + def publish(_job): + return PublishResult( + outcome=PublishOutcome.NEED_REVIEW, + message="仅测试 dispatch CAS 窗口", + needs_manual_review=True, + ) + + monkeypatch.setattr( + "app.services.publish_executor.get_publisher", + lambda *_args, **_kwargs: Publisher(), + ) + + result = execute_publish_job( + job_id, + expected_execution_id=execution_id, + repository=repository, + ) + + assert result["outcome"] == "NEED_REVIEW" + assert observed == [True] + assert not is_publish_dispatch_active(execution_id) + + +def test_recovery_does_not_interrupt_active_dispatch_window(monkeypatch, tmp_path): + job_id = _seed_job( + tmp_path, + execution_id="execution-active-dispatch", + publish_mode="local_browser", + ) + + def publisher_factory(*_args, before_dispatch, **_kwargs): + class Publisher: + @staticmethod + def publish(_job): + before_dispatch() + with get_connection() as connection: + connection.execute( + "UPDATE publish_jobs SET updated_at = ? WHERE id = ?", + (_iso(-3600), job_id), + ) + connection.commit() + assert PublishScheduler().recover_interrupted_jobs() == 0 + assert _raw(job_id)["status"] == "PUBLISHING" + return PublishResult( + outcome=PublishOutcome.NEED_REVIEW, + message="仅测试 dispatch 临界区", + needs_manual_review=True, + ) + + return Publisher() + + monkeypatch.setattr("app.services.publish_executor.get_publisher", publisher_factory) + + result = execute_publish_job(job_id, expected_execution_id="execution-active-dispatch") + + assert result["outcome"] == "NEED_REVIEW" + + +def test_recovery_snapshot_cannot_override_new_dispatch_reservation(tmp_path): + job_id = _seed_job(tmp_path, execution_id="execution-current") + repository = PublishRepository() + + class Worker: + @staticmethod + def execution(_execution_id: str) -> dict: + before = _raw(job_id) + assert repository.begin_execution_dispatch( + job_id, + "execution-current", + str(before["updated_at"]), + ) + return { + "phase": "received", + "details": {}, + "in_progress": False, + "identity": _identity(job_id), + } + + with get_connection() as connection: + connection.execute( + "UPDATE publish_jobs SET updated_at = ? WHERE id = ?", + (_iso(-3600), job_id), + ) + connection.commit() + + recovered = PublishScheduler(worker_client=Worker()).recover_interrupted_jobs() + + assert recovered == 0 + stored = _raw(job_id) + assert stored["status"] == "PUBLISHING" + assert stored["execution_phase"] == "dispatching" + + +def test_publish_now_does_not_overwrite_same_status_concurrent_edit(monkeypatch, tmp_path): + job_id = _seed_job(tmp_path, status="SCHEDULED", execution_id="") + scheduler = PublishScheduler() + + def readiness(*_args, **_kwargs): + with get_connection() as connection: + connection.execute( + "UPDATE publish_jobs SET title = '并发新标题', updated_at = ? WHERE id = ?", + (_iso(10), job_id), + ) + connection.commit() + return { + job_id: { + "resolved_account_id": "", + "resolved_publish_mode": "manual_export", + } + } + + monkeypatch.setattr(scheduler, "_require_ready_jobs", readiness) + + with pytest.raises(ValueError, match="状态已变化"): + scheduler.publish_now(job_id) + + stored = _raw(job_id) + assert stored["title"] == "并发新标题" + + +def test_worker_unavailable_from_stale_execution_cannot_reschedule_or_add_event(tmp_path): + job_id = _seed_job(tmp_path, execution_id="execution-old") + scheduler = PublishScheduler() + captured = _raw(job_id) + with get_connection() as connection: + connection.execute( + "UPDATE publish_jobs SET execution_id = 'execution-new', worker_id = 'worker-new' WHERE id = ?", + (job_id,), + ) + connection.commit() + before_events = _event_count(job_id) + + result = scheduler._handle_worker_unavailable( + captured, + PublishWorkerUnavailable("Worker 暂不可用", request_may_have_been_received=False), + ) + + assert result["status"] == "skipped" + stored = _raw(job_id) + assert stored["status"] == "PUBLISHING" + assert stored["execution_id"] == "execution-new" + assert _event_count(job_id) == before_events + + +def test_worker_timeout_after_request_keeps_known_execution_pending(tmp_path): + job_id = _seed_job(tmp_path, execution_id="execution-uploading") + + class Worker: + @staticmethod + def execution(_execution_id: str) -> dict: + return { + "phase": "upload_started", + "details": {"message": "上传中"}, + "identity": _identity(job_id), + } + + result = PublishScheduler(worker_client=Worker())._handle_worker_unavailable( + _raw(job_id), + PublishWorkerUnavailable("请求超时", request_may_have_been_received=True), + ) + + assert result["status"] == "pending" + assert _raw(job_id)["status"] == "PUBLISHING" + + +def test_worker_timeout_rejects_mismatched_terminal_result(tmp_path): + job_id = _seed_job(tmp_path, execution_id="execution-timeout-mismatch") + + class Worker: + @staticmethod + def execution(_execution_id: str) -> dict: + return { + "phase": "failed", + "details": {"outcome": "PUBLISHED", "message": "矛盾终态"}, + "identity": _identity(job_id), + } + + result = PublishScheduler(worker_client=Worker())._handle_worker_unavailable( + _raw(job_id), + PublishWorkerUnavailable("请求超时", request_may_have_been_received=True), + ) + + assert result["status"] == "need_review" + assert _raw(job_id)["status"] == "NEED_REVIEW" + + +def test_concurrent_safe_repair_creates_exactly_one_replacement(monkeypatch, tmp_path): + source_id = _seed_job( + tmp_path, + status="NEED_REVIEW", + execution_id="execution-source", + publish_mode="manual_export", + ) + + def readiness(*_args, **_kwargs): + return { + "repairable": True, + "dispatch_ready": True, + "requires_worker": False, + "resolved_publish_mode": "local_browser", + "resolved_account_id": "", + } + + monkeypatch.setattr(scheduler_module, "build_send_readiness", readiness) + scheduler = PublishScheduler() + monkeypatch.setattr(scheduler, "_public_job", lambda job_id: scheduler.repository.get_job(job_id)) + + with ThreadPoolExecutor(max_workers=2) as pool: + results = list(pool.map(lambda _: scheduler.repair_and_publish(source_id), range(2))) + + assert sorted(item["status"] for item in results) == ["already_created", "scheduled"] + with get_connection() as connection: + replacements = connection.execute( + "SELECT id FROM publish_jobs WHERE retry_of_job_id = ?", (source_id,) + ).fetchall() + source_events = connection.execute( + """ + SELECT COUNT(*) FROM publish_job_events + WHERE job_id = ? AND event_type = 'safe_repair_replacement_created' + """, + (source_id,), + ).fetchone()[0] + assert len(replacements) == 1 + assert source_events == 1 + + +def test_safe_repair_rechecks_source_state_inside_final_transaction(monkeypatch, tmp_path): + source_id = _seed_job( + tmp_path, + status="NEED_REVIEW", + execution_id="execution-source", + publish_mode="manual_export", + ) + + def readiness(*_args, **_kwargs): + return { + "repairable": True, + "dispatch_ready": True, + "requires_worker": True, + "resolved_publish_mode": "local_browser", + "resolved_account_id": "", + } + + def change_source_state(_client): + with get_connection() as connection: + connection.execute( + "UPDATE publish_jobs SET status = 'PUBLISHED', updated_at = ? WHERE id = ?", + (_iso(1), source_id), + ) + connection.commit() + + monkeypatch.setattr(scheduler_module, "build_send_readiness", readiness) + monkeypatch.setattr(scheduler_module, "require_worker_available", change_source_state) + + with pytest.raises(ValueError, match="源任务状态已变化"): + PublishScheduler().repair_and_publish(source_id) + + with get_connection() as connection: + count = connection.execute( + "SELECT COUNT(*) FROM publish_jobs WHERE retry_of_job_id = ?", (source_id,) + ).fetchone()[0] + assert count == 0 + + +def test_concurrent_failed_retry_creates_only_one_active_replacement(monkeypatch, tmp_path): + source_id = _seed_job(tmp_path, status="FAILED", publish_mode="manual_export") + scheduler = PublishScheduler() + monkeypatch.setattr( + scheduler, + "_require_ready_jobs", + lambda *_args, **_kwargs: { + source_id: { + "resolved_publish_mode": "manual_export", + "resolved_account_id": "", + } + }, + ) + + def retry() -> tuple[str, str]: + try: + return "ok", scheduler.retry_failed(source_id)["job_id"] + except ValueError as exc: + return "error", str(exc) + + with ThreadPoolExecutor(max_workers=2) as pool: + results = list(pool.map(lambda _: retry(), range(2))) + + assert [kind for kind, _ in results].count("ok") == 1 + assert [kind for kind, _ in results].count("error") == 1 + with get_connection() as connection: + count = connection.execute( + "SELECT COUNT(*) FROM publish_jobs WHERE retry_of_job_id = ?", (source_id,) + ).fetchone()[0] + assert count == 1 + + +def test_update_schedule_cannot_overwrite_job_claimed_after_readiness(monkeypatch, tmp_path): + job_id = _seed_job(tmp_path, status="SCHEDULED") + scheduler = PublishScheduler() + + def readiness(*_args, **_kwargs): + with get_connection() as connection: + connection.execute( + "UPDATE publish_jobs SET status = 'PUBLISHING', updated_at = ? WHERE id = ?", + (_iso(1), job_id), + ) + connection.commit() + return { + job_id: { + "resolved_account_id": "", + "resolved_publish_mode": "manual_export", + } + } + + monkeypatch.setattr(scheduler, "_require_ready_jobs", readiness) + + with pytest.raises(ValueError, match="任务状态已变化"): + scheduler.update_schedule(job_id, _iso(3600)) + + assert _raw(job_id)["status"] == "PUBLISHING" + + +def test_update_schedule_cannot_overwrite_same_status_concurrent_edit(monkeypatch, tmp_path): + job_id = _seed_job(tmp_path, status="SCHEDULED") + scheduler = PublishScheduler() + + def readiness(*_args, **_kwargs): + with get_connection() as connection: + connection.execute( + "UPDATE publish_jobs SET title = '并发修改', updated_at = ? WHERE id = ?", + (_iso(2), job_id), + ) + connection.commit() + return { + job_id: { + "resolved_account_id": "", + "resolved_publish_mode": "manual_export", + } + } + + monkeypatch.setattr(scheduler, "_require_ready_jobs", readiness) + + with pytest.raises(ValueError, match="任务状态已变化"): + scheduler.update_schedule(job_id, _iso(3600)) + + stored = _raw(job_id) + assert stored["status"] == "SCHEDULED" + assert stored["title"] == "并发修改" + + +def test_batch_schedule_cannot_overwrite_concurrently_claimed_job(monkeypatch, tmp_path): + job_id = _seed_job(tmp_path, status="SCHEDULED") + scheduler = PublishScheduler() + + def readiness(*_args, **_kwargs): + with get_connection() as connection: + connection.execute( + "UPDATE publish_jobs SET status = 'PUBLISHING', updated_at = ? WHERE id = ?", + (_iso(3), job_id), + ) + connection.commit() + return { + job_id: { + "dispatch_ready": True, + "resolved_account_id": "", + "resolved_publish_mode": "manual_export", + } + } + + monkeypatch.setattr(scheduler, "_require_ready_jobs", readiness) + + with pytest.raises(ValueError, match="当前状态不能修改排期"): + scheduler.update_batch_schedule( + [job_id], + platform="douyin", + action="apply", + confirmed_schedule=[{"job_id": job_id, "scheduled_at_utc": _iso(3600)}], + ) + + assert _raw(job_id)["status"] == "PUBLISHING" + + +def test_malformed_risk_flags_fail_closed_before_executor(tmp_path): + job_id = _seed_job(tmp_path, status="SCHEDULED") + with get_connection() as connection: + connection.execute("UPDATE publish_jobs SET risk_flags = '{broken' WHERE id = ?", (job_id,)) + connection.commit() + scheduler = PublishScheduler( + executor=lambda *_args, **_kwargs: pytest.fail("风险字段损坏时不得调用 Publisher") + ) + + result = scheduler.execute_job(job_id) + + assert result["status"] == "need_review" + assert _raw(job_id)["status"] == "NEED_REVIEW" + + +@pytest.mark.parametrize("risk_flags", ['"not-a-list"', '{"unexpected": true}', '{"risk_flags": "bad"}']) +def test_wrong_shape_risk_flags_fail_closed_before_executor(tmp_path, risk_flags): + job_id = _seed_job(tmp_path, status="SCHEDULED") + with get_connection() as connection: + connection.execute("UPDATE publish_jobs SET risk_flags = ? WHERE id = ?", (risk_flags, job_id)) + connection.commit() + scheduler = PublishScheduler( + executor=lambda *_args, **_kwargs: pytest.fail("风险字段结构错误时不得调用 Publisher") + ) + + result = scheduler.execute_job(job_id) + + assert result["status"] == "need_review" + assert _raw(job_id)["status"] == "NEED_REVIEW" + + +def test_manual_publish_url_validates_hostname_not_substring(tmp_path): + job_id = _seed_job(tmp_path, status="NEED_REVIEW") + + with pytest.raises(ValueError, match="有效的 douyin.com"): + PublishScheduler().mark_published_manually( + job_id, "https://attacker.example/?next=https://douyin.com/video/1" + ) + + assert _raw(job_id)["status"] == "NEED_REVIEW" + + +@pytest.mark.parametrize( + "value", + ["../escape", r"..\escape", r"C:\escape", "a/b", "a:b", ".", "..", "CON", "con.txt", "bad.", "bad value", "bad\nvalue"], +) +def test_worker_identifier_rejects_windows_unsafe_values(value): + with pytest.raises(PublishValidationError): + validate_worker_identifier(value, "execution_id", max_length=160) + + +@pytest.fixture +def isolated_worker_state(tmp_path): + original = settings.publish_worker_state_dir + object.__setattr__(settings, "publish_worker_state_dir", tmp_path) + try: + yield tmp_path + finally: + object.__setattr__(settings, "publish_worker_state_dir", original) + + +def _worker_payload(*, execution_id: str = "execution-idempotent", account_id: str = "account-1") -> dict: + return { + "job_id": "job-1", + "execution_id": execution_id, + "platform": "douyin", + "account_id": account_id, + "title": "测试标题", + "caption": "测试正文", + "video_path": "video.mp4", + } + + +def _worker_headers() -> dict[str, str]: + return {"Authorization": "Bearer test-token"} + + +def test_worker_same_execution_is_idempotent_under_concurrent_requests( + monkeypatch, isolated_worker_state +): + calls = 0 + calls_lock = threading.Lock() + + class Publisher: + def publish(self, _values): + nonlocal calls + with calls_lock: + calls += 1 + time.sleep(0.1) + return PublishResult( + outcome=PublishOutcome.PUBLISHED, + message="投稿成功", + remote_video_id="video-1", + published_at=_iso(), + provider_response={"token": "secret-value"}, + ) + + monkeypatch.setattr(worker_module, "get_platform_publisher", lambda *_args, **_kwargs: Publisher()) + monkeypatch.setattr( + worker_module, + "_resolve_media_path", + lambda raw_value, *, required: str(raw_value or "") if required else "", + ) + + def invoke(): + return TestClient(worker_module.create_worker_app(token="test-token")).post( + "/v1/publish", headers=_worker_headers(), json=_worker_payload() + ) + + with ThreadPoolExecutor(max_workers=2) as pool: + responses = list(pool.map(lambda _: invoke(), range(2))) + + assert [response.status_code for response in responses] == [200, 200] + assert {response.json()["outcome"] for response in responses} == {"PUBLISHED"} + assert calls == 1 + stored = worker_module.ExecutionJournal("execution-idempotent").read() + assert stored["identity"] == { + "job_id": "job-1", + "platform": "douyin", + "account_id": "account-1", + } + assert stored["details"]["provider_response"]["token"] == "[REDACTED]" + + +def test_worker_serializes_different_executions_for_same_job( + monkeypatch, isolated_worker_state +): + calls = 0 + + class Publisher: + def publish(self, _values): + nonlocal calls + calls += 1 + time.sleep(0.05) + return PublishResult( + outcome=PublishOutcome.PUBLISHED, + message="投稿成功", + published_at=_iso(), + ) + + monkeypatch.setattr(worker_module, "get_platform_publisher", lambda *_args, **_kwargs: Publisher()) + monkeypatch.setattr( + worker_module, + "_resolve_media_path", + lambda raw_value, *, required: str(raw_value or "") if required else "", + ) + + def invoke(execution_id: str): + return TestClient(worker_module.create_worker_app(token="test-token")).post( + "/v1/publish", + headers=_worker_headers(), + json=_worker_payload(execution_id=execution_id), + ).json() + + with ThreadPoolExecutor(max_workers=2) as pool: + responses = list(pool.map(invoke, ["execution-job-a", "execution-job-b"])) + + assert calls == 1 + assert {response["outcome"] for response in responses} == {"PUBLISHED", "NEED_REVIEW"} + assert "job_execution_conflict" in {response["error_code"] for response in responses} + + +def test_worker_cross_process_lock_fails_closed_without_calling_publisher( + monkeypatch, isolated_worker_state +): + journal = worker_module.ExecutionJournal("execution-cross-process") + token = worker_module._try_create_lock_file(journal.lock.path) + assert token + monkeypatch.setattr( + worker_module, + "get_platform_publisher", + lambda *_args, **_kwargs: pytest.fail("跨进程锁占用时不得调用 Publisher"), + ) + try: + response = TestClient(worker_module.create_worker_app(token="test-token")).post( + "/v1/publish", + headers=_worker_headers(), + json=_worker_payload(execution_id="execution-cross-process"), + ) + finally: + worker_module._release_lock_file(journal.lock.path, token) + + assert response.status_code == 200 + assert response.json()["outcome"] == "NEED_REVIEW" + assert response.json()["error_code"] == "execution_in_progress" + + +def test_worker_execution_lock_blocks_a_real_second_process( + monkeypatch, isolated_worker_state +): + context = multiprocessing.get_context("spawn") + ready = context.Event() + release = context.Event() + process = context.Process( + target=_hold_execution_lock_in_child, + args=(str(isolated_worker_state), "execution-real-process", ready, release), + ) + process.start() + try: + assert ready.wait(timeout=8) + monkeypatch.setattr( + worker_module, + "get_platform_publisher", + lambda *_args, **_kwargs: pytest.fail("另一进程持锁时不得调用 Publisher"), + ) + response = TestClient(worker_module.create_worker_app(token="test-token")).post( + "/v1/publish", + headers=_worker_headers(), + json=_worker_payload(execution_id="execution-real-process"), + ) + assert response.json()["outcome"] == "NEED_REVIEW" + assert response.json()["error_code"] == "execution_in_progress" + finally: + release.set() + process.join(timeout=8) + if process.is_alive(): + process.terminate() + process.join(timeout=3) + assert process.exitcode == 0 + + +def test_worker_reclaims_lock_left_by_dead_process(isolated_worker_state): + journal = worker_module.ExecutionJournal("execution-stale-lock") + journal.lock.path.parent.mkdir(parents=True, exist_ok=True) + journal.lock.path.write_text("2147483646:dead-process", encoding="utf-8") + + token = worker_module._try_create_lock_file(journal.lock.path) + try: + assert token + token.handle.seek(0) + assert token.handle.read().decode("utf-8") == token + finally: + worker_module._release_lock_file(journal.lock.path, token) + + +def test_two_processes_cannot_both_reclaim_same_inactive_lock(isolated_worker_state): + context = multiprocessing.get_context("spawn") + lock_path = Path(isolated_worker_state) / "locks" / "race.lock" + lock_path.parent.mkdir(parents=True, exist_ok=True) + lock_path.write_text("2147483646:dead-process", encoding="utf-8") + start = context.Event() + release = context.Event() + ready = [context.Event(), context.Event()] + results = context.Queue() + processes = [ + context.Process( + target=_contend_lock_in_child, + args=(str(lock_path), ready[index], start, release, results), + ) + for index in range(2) + ] + for process in processes: + process.start() + try: + assert all(event.wait(timeout=10) for event in ready) + start.set() + outcomes = [results.get(timeout=10), results.get(timeout=10)] + assert [item["error"] for item in outcomes] == ["", ""] + assert sorted(item["acquired"] for item in outcomes) == [False, True] + finally: + release.set() + for process in processes: + process.join(timeout=10) + if process.is_alive(): + process.terminate() + process.join(timeout=5) + assert [process.exitcode for process in processes] == [0, 0] + assert worker_module._lock_file_is_active(lock_path) is False + + +def test_execution_lock_releases_thread_lock_when_file_lock_creation_fails( + monkeypatch, isolated_worker_state +): + lock = worker_module._execution_lock("execution-lock-error") + original = worker_module._try_create_lock_file + monkeypatch.setattr( + worker_module, + "_try_create_lock_file", + lambda _path: (_ for _ in ()).throw(PublishValidationError("锁失败", "lock_failed")), + ) + with pytest.raises(PublishValidationError): + with lock: + pytest.fail("锁创建失败时不得进入临界区") + + monkeypatch.setattr(worker_module, "_try_create_lock_file", original) + with lock: + assert lock.acquired is True + + +def test_worker_rejects_execution_identity_conflict_without_republish( + monkeypatch, isolated_worker_state +): + calls = 0 + + class Publisher: + def publish(self, _values): + nonlocal calls + calls += 1 + return PublishResult( + outcome=PublishOutcome.PUBLISHED, + message="投稿成功", + published_at=_iso(), + ) + + monkeypatch.setattr(worker_module, "get_platform_publisher", lambda *_args, **_kwargs: Publisher()) + monkeypatch.setattr(worker_module, "_resolve_media_path", lambda value, *, required: str(value or "")) + client = TestClient(worker_module.create_worker_app(token="test-token")) + + first = client.post("/v1/publish", headers=_worker_headers(), json=_worker_payload()) + conflict = client.post( + "/v1/publish", + headers=_worker_headers(), + json=_worker_payload(account_id="account-2"), + ) + + assert first.json()["outcome"] == "PUBLISHED" + assert conflict.json()["outcome"] == "NEED_REVIEW" + assert conflict.json()["error_code"] == "execution_identity_conflict" + assert calls == 1 + + +def test_worker_terminal_replay_requires_matching_identity(monkeypatch, isolated_worker_state): + journal = worker_module.ExecutionJournal("execution-missing-identity") + journal.path.write_text( + json.dumps({ + "execution_id": "execution-missing-identity", + "phase": "confirmed_success", + "details": PublishResult( + outcome=PublishOutcome.PUBLISHED, + message="旧日志声称成功", + ).as_dict(), + }), + encoding="utf-8", + ) + monkeypatch.setattr( + worker_module, + "get_platform_publisher", + lambda *_args, **_kwargs: pytest.fail("缺少身份的终态不得重放或再次投稿"), + ) + + response = TestClient(worker_module.create_worker_app(token="test-token")).post( + "/v1/publish", + headers=_worker_headers(), + json=_worker_payload(execution_id="execution-missing-identity"), + ) + + assert response.json()["outcome"] == "NEED_REVIEW" + assert response.json()["error_code"] == "execution_identity_missing" + + +def test_worker_rejects_new_execution_after_same_job_reached_upload( + monkeypatch, isolated_worker_state +): + old = worker_module.ExecutionJournal("execution-old-upload") + old.update( + "upload_started", + {"message": "已开始上传"}, + identity={"job_id": "job-1", "platform": "douyin", "account_id": "account-1"}, + ) + monkeypatch.setattr( + worker_module, + "get_platform_publisher", + lambda *_args, **_kwargs: pytest.fail("同一 job 的新 execution 不得重复投稿"), + ) + + response = TestClient(worker_module.create_worker_app(token="test-token")).post( + "/v1/publish", + headers=_worker_headers(), + json=_worker_payload(execution_id="execution-new-upload"), + ) + + assert response.json()["outcome"] == "NEED_REVIEW" + assert response.json()["error_code"] == "job_execution_conflict" + + +def test_worker_keeps_post_upload_script_failure_in_manual_review( + monkeypatch, isolated_worker_state +): + class Publisher: + def __init__(self, runtime): + self.runtime = runtime + + def publish(self, _values): + self.runtime.phase("upload_started", {"message": "开始上传"}) + self.runtime.phase("title_filled", {"message": "上传后填写标题"}) + raise PublishError("页面脚本异常", "platform_form_changed") + + monkeypatch.setattr( + worker_module, + "get_platform_publisher", + lambda *_args, **kwargs: Publisher(kwargs["runtime"]), + ) + monkeypatch.setattr( + worker_module, + "_resolve_media_path", + lambda raw_value, *, required: str(raw_value or "") if required else "", + ) + client = TestClient(worker_module.create_worker_app(token="test-token")) + + first = client.post( + "/v1/publish", + headers=_worker_headers(), + json=_worker_payload(execution_id="execution-post-upload"), + ) + second = client.post( + "/v1/publish", + headers=_worker_headers(), + json=_worker_payload(execution_id="execution-post-upload-retry"), + ) + + assert first.json()["outcome"] == "NEED_REVIEW" + assert first.json()["needs_manual_review"] is True + assert second.json()["error_code"] == "job_execution_conflict" + + +def test_worker_does_not_resume_after_upload_started(monkeypatch, isolated_worker_state): + journal = worker_module.ExecutionJournal("execution-unsafe") + journal.update( + "upload_started", + {"message": "上传中"}, + identity={"job_id": "job-1", "platform": "douyin", "account_id": "account-1"}, + ) + monkeypatch.setattr( + worker_module, + "get_platform_publisher", + lambda *_args, **_kwargs: pytest.fail("危险阶段不得再次调用 Publisher"), + ) + response = TestClient(worker_module.create_worker_app(token="test-token")).post( + "/v1/publish", + headers=_worker_headers(), + json=_worker_payload(execution_id="execution-unsafe"), + ) + + assert response.status_code == 200 + assert response.json()["outcome"] == "NEED_REVIEW" + assert response.json()["error_code"] == "execution_resume_unsafe" + + +def test_corrupt_worker_journal_fails_closed(monkeypatch, isolated_worker_state): + journal = worker_module.ExecutionJournal("execution-corrupt") + journal.path.write_text("{broken", encoding="utf-8") + monkeypatch.setattr( + worker_module, + "get_platform_publisher", + lambda *_args, **_kwargs: pytest.fail("损坏 journal 不得调用 Publisher"), + ) + response = TestClient(worker_module.create_worker_app(token="test-token")).post( + "/v1/publish", + headers=_worker_headers(), + json=_worker_payload(execution_id="execution-corrupt"), + ) + + assert response.status_code == 200 + assert response.json()["outcome"] == "NEED_REVIEW" + assert response.json()["error_code"] == "execution_journal_corrupt" + + +def test_inconsistent_terminal_worker_journal_fails_closed(monkeypatch, isolated_worker_state): + journal = worker_module.ExecutionJournal("execution-inconsistent") + journal.update( + "confirmed_success", + {"outcome": "FAILED", "message": "不一致的旧日志"}, + identity={"job_id": "job-1", "platform": "douyin", "account_id": "account-1"}, + ) + monkeypatch.setattr( + worker_module, + "get_platform_publisher", + lambda *_args, **_kwargs: pytest.fail("不一致终态不得再次调用 Publisher"), + ) + + response = TestClient(worker_module.create_worker_app(token="test-token")).post( + "/v1/publish", + headers=_worker_headers(), + json=_worker_payload(execution_id="execution-inconsistent"), + ) + + assert response.json()["outcome"] == "NEED_REVIEW" + assert response.json()["error_code"] == "execution_terminal_result_inconsistent" + + +def test_worker_login_reserves_account_lock_before_background_task( + monkeypatch, isolated_worker_state +): + opened = threading.Event() + release = threading.Event() + + class Publisher: + @staticmethod + def open_login(_account_id): + opened.set() + assert release.wait(timeout=3) + + monkeypatch.setattr(worker_module, "get_platform_publisher", lambda *_args, **_kwargs: Publisher()) + + def first_login(): + return TestClient(worker_module.create_worker_app(token="test-token")).post( + "/v1/accounts/login", + headers=_worker_headers(), + json={"platform": "douyin", "account_id": "account-login"}, + ) + + with ThreadPoolExecutor(max_workers=1) as pool: + first_future = pool.submit(first_login) + assert opened.wait(timeout=3) + second = TestClient(worker_module.create_worker_app(token="test-token")).post( + "/v1/accounts/open-center", + headers=_worker_headers(), + json={"platform": "douyin", "account_id": "account-login"}, + ) + release.set() + first = first_future.result(timeout=3) + + assert first.status_code == 202 + assert second.status_code == 409 + + +def test_provider_response_redacts_camel_case_secret_keys(): + sanitized = sanitize_provider_response({ + "accessToken": "a", + "refresh-token": "b", + "session_token": "c", + "csrfToken": "d", + "nested": {"clientSecret": "e", "apiKey": "f", "private-key": "g"}, + }) + + assert sanitized == { + "accessToken": "[REDACTED]", + "refresh-token": "[REDACTED]", + "session_token": "[REDACTED]", + "csrfToken": "[REDACTED]", + "nested": { + "clientSecret": "[REDACTED]", + "apiKey": "[REDACTED]", + "private-key": "[REDACTED]", + }, + } + + +def test_manual_export_rejects_path_components_outside_export_root(tmp_path): + publisher = ManualExportPublisher(export_dir=tmp_path / "exports") + + with pytest.raises(PublishValidationError, match="不安全路径"): + publisher.build_package_dir({"task_id": "../outside", "clip_id": "clip-1"}) + + +@pytest.mark.parametrize("component", ["CON", "con.txt", "trailing.", "trailing ", "bad*name"]) +def test_manual_export_rejects_windows_unsafe_components(tmp_path, component): + publisher = ManualExportPublisher(export_dir=tmp_path / "exports") + + with pytest.raises(PublishValidationError, match="不安全路径"): + publisher.build_package_dir({"task_id": component, "clip_id": "clip-1"}) + + +def test_manual_export_failure_keeps_previous_complete_package(monkeypatch, tmp_path): + video = tmp_path / "video.mp4" + video.write_bytes(b"new-video") + publisher = ManualExportPublisher(export_dir=tmp_path / "exports") + job = { + "id": "job-1", + "task_id": "task-1", + "clip_id": "clip-1", + "title": "测试标题", + "caption": "测试正文", + "video_path": str(video), + } + package_dir = publisher.build_package_dir(job) + package_dir.mkdir(parents=True) + (package_dir / "sentinel.txt").write_text("old-complete", encoding="utf-8") + original_write_json = manual_export_module._write_json + calls = 0 + + def fail_second_json(path, value): + nonlocal calls + calls += 1 + if calls == 2: + raise OSError("磁盘写入失败") + return original_write_json(path, value) + + monkeypatch.setattr(manual_export_module, "_write_json", fail_second_json) + + with pytest.raises(OSError, match="磁盘写入失败"): + publisher.publish(job) + + assert (package_dir / "sentinel.txt").read_text(encoding="utf-8") == "old-complete" + assert not list(package_dir.parent.glob(".clip-1.staging-*")) + + +def test_worker_http_boundary_rejects_reserved_account_and_execution_ids(isolated_worker_state): + client = TestClient(worker_module.create_worker_app(token="test-token")) + + account = client.post( + "/v1/accounts/check", + headers=_worker_headers(), + json={"platform": "douyin", "account_id": "CON"}, + ) + execution = client.get("/v1/executions/a:b", headers=_worker_headers()) + + assert account.status_code == 422 + assert execution.status_code == 422 diff --git a/tests/test_publish_readiness.py b/tests/test_publish_readiness.py index 79ce993..293fd99 100644 --- a/tests/test_publish_readiness.py +++ b/tests/test_publish_readiness.py @@ -133,6 +133,7 @@ def publish(self, payload: dict) -> PublishResult: message="测试投稿成功", remote_video_id="remote-test-1", platform_url="https://www.douyin.com/video/test-1", + published_at=_iso(), ) diff --git a/tests/test_publish_scheduler.py b/tests/test_publish_scheduler.py index 0539b24..ce22ccd 100644 --- a/tests/test_publish_scheduler.py +++ b/tests/test_publish_scheduler.py @@ -144,6 +144,7 @@ def test_auto_pipeline_creates_only_metadata_target_platform(tmp_path): "metadata": {"platform": "bilibili", "title": "自动标题", "caption": "自动正文", "hashtags": ["自动"], "risk_flags": []}, "scheduled_at": "", }], + subtitle_delivery_mode="original", ) created = result["created"][0] assert created["platform"] == "bilibili" @@ -372,9 +373,18 @@ class FinishedWorker: def execution(_execution_id): return { "phase": "manual_review", + "identity": { + "job_id": job_id, + "platform": str(_raw(job_id).get("platform") or ""), + "account_id": str(_raw(job_id).get("account_id") or ""), + }, "details": { "outcome": "NEED_REVIEW", "message": "上传状态需要人工确认", + "remote_video_id": "", + "platform_url": "", + "published_at": "", + "provider_response": {}, "error_code": "video_upload_timeout", "needs_manual_review": True, }, @@ -405,9 +415,19 @@ def execution(cls, _execution_id): cls.calls += 1 return { "phase": "confirmed_success", + "identity": { + "job_id": job_id, + "platform": str(_raw(job_id).get("platform") or ""), + "account_id": str(_raw(job_id).get("account_id") or ""), + }, "details": { "outcome": "PUBLISHED", "message": "投稿成功", + "remote_video_id": "video-1", + "platform_url": "https://www.douyin.com/video/video-1", + "published_at": _utc(), + "provider_response": {}, + "error_code": "", "needs_manual_review": False, }, } diff --git a/tests/test_publish_scheduler_state_machine.py b/tests/test_publish_scheduler_state_machine.py index c6a49f1..22d97de 100644 --- a/tests/test_publish_scheduler_state_machine.py +++ b/tests/test_publish_scheduler_state_machine.py @@ -280,6 +280,7 @@ def test_due_job_follows_publisher_outcome(tmp_path, outcome, expected): message="mock result", remote_video_id="remote-1" if outcome == PublishOutcome.PUBLISHED else "", platform_url="https://www.douyin.com/video/1" if outcome == PublishOutcome.PUBLISHED else "", + published_at=_iso() if outcome == PublishOutcome.PUBLISHED else "", error_code="mock_error" if outcome != PublishOutcome.PUBLISHED else "", needs_manual_review=outcome == PublishOutcome.NEED_REVIEW, ) @@ -291,6 +292,26 @@ def test_due_job_follows_publisher_outcome(tmp_path, outcome, expected): assert row["finished_at"] +def test_untrusted_provider_platform_url_requires_manual_review(tmp_path): + calls: list[str] = [] + job_id = _job(tmp_path) + result = PublishResult( + outcome=PublishOutcome.PUBLISHED, + message="投稿成功但链接异常", + remote_video_id="remote-unsafe", + platform_url="javascript:alert(1)", + published_at=_iso(), + ) + + PublishScheduler(executor=_executor(result, calls)).run_once() + + row = _raw(job_id) + assert calls == [job_id] + assert row["status"] == "NEED_REVIEW" + assert row["error_code"] == "invalid_platform_url" + assert row["platform_url"] in {None, ""} + + def test_published_job_is_never_executed_again(tmp_path): calls: list[str] = [] job_id = _job(tmp_path, status="PUBLISHED") @@ -302,7 +323,7 @@ def test_published_job_is_never_executed_again(tmp_path): def test_two_schedulers_atomically_claim_only_once(tmp_path): calls: list[str] = [] job_id = _job(tmp_path) - executor = _executor(PublishResult(PublishOutcome.PUBLISHED), calls) + executor = _executor(PublishResult(PublishOutcome.PUBLISHED, published_at=_iso()), calls) with ThreadPoolExecutor(max_workers=2) as pool: results = list(pool.map(lambda _: PublishScheduler(executor=executor).execute_job(job_id), range(2))) assert calls == [job_id] @@ -340,6 +361,52 @@ def flaky_run_once(): assert scheduler_module._SCHEDULER_HEALTH["last_error_code"] == "" +def test_scheduler_background_task_is_tracked_and_awaited(monkeypatch): + async def scenario(): + scheduler = PublishScheduler() + started = asyncio.Event() + stopped = asyncio.Event() + + async def fake_run_forever(): + started.set() + await stopped.wait() + + def fake_stop(): + stopped.set() + + monkeypatch.setattr(scheduler, "run_forever", fake_run_forever) + monkeypatch.setattr(scheduler, "stop", fake_stop) + monkeypatch.setattr(scheduler_module, "PublishScheduler", lambda: scheduler) + original_enabled = scheduler_module.settings.publish_scheduler_enabled + object.__setattr__(scheduler_module.settings, "publish_scheduler_enabled", True) + try: + returned = await scheduler_module.start_scheduler_background() + await started.wait() + + assert returned is scheduler + assert scheduler._background_task is not None + assert scheduler._background_task.get_name() == "niuma-publish-scheduler" + + await scheduler.shutdown() + assert scheduler._background_task.done() + finally: + object.__setattr__( + scheduler_module.settings, "publish_scheduler_enabled", original_enabled + ) + + asyncio.run(scenario()) + + +def test_scheduler_shutdown_before_background_task_initializes_does_not_hang(): + async def scenario(): + scheduler = PublishScheduler() + scheduler._background_task = asyncio.create_task(scheduler.run_forever()) + await asyncio.wait_for(scheduler.shutdown(), timeout=3) + assert scheduler._background_task.done() + + asyncio.run(scenario()) + + def test_unexpected_job_error_does_not_block_later_due_jobs(monkeypatch): scheduler = PublishScheduler() calls: list[str] = [] @@ -357,10 +424,15 @@ def execute(job_id: str): return {"status": "skipped", "job_id": job_id} monkeypatch.setattr(scheduler, "execute_job", execute) + monkeypatch.setattr( + scheduler.repository, + "get_job", + lambda job_id: {"id": job_id, "status": "SCHEDULED"}, + ) monkeypatch.setattr( scheduler, "_mark_need_review", - lambda job_id, error_code, message: { + lambda job_id, error_code, message, **_kwargs: { "status": "need_review", "job_id": job_id, "error_code": error_code, diff --git a/tests/test_publish_task_grouping.py b/tests/test_publish_task_grouping.py index 2e2be48..5a7dc36 100644 --- a/tests/test_publish_task_grouping.py +++ b/tests/test_publish_task_grouping.py @@ -218,6 +218,8 @@ def test_dismiss_keeps_files_and_other_platform_and_blocks_recreation(tmp_path: assert refreshed["skipped_removed"] >= 1 assert not any(job.get("output_clip_id") == clip_id for job in refreshed["created"]) + cover_path = tmp_path / f"{clip_id}-cover.jpg" + cover_path.write_bytes(b"cover") auto_result = create_auto_publish_jobs( {"id": task_id, "platform": "douyin"}, [ @@ -230,9 +232,11 @@ def test_dismiss_keeps_files_and_other_platform_and_blocks_recreation(tmp_path: "hashtags": ["测试"], "risk_flags": [], }, + "cover": {"cover_file_path": str(cover_path), "cover_time_seconds": 1}, "scheduled_at": "", } ], + subtitle_delivery_mode="original", ) assert auto_result["created_count"] == 0 assert auto_result["skipped_count"] == 1 diff --git a/tests/test_publish_task_linkage.py b/tests/test_publish_task_linkage.py index 1b3e124..a6d7e8a 100644 --- a/tests/test_publish_task_linkage.py +++ b/tests/test_publish_task_linkage.py @@ -51,6 +51,15 @@ def _cleanup() -> None: ) connection.execute("DELETE FROM publish_jobs WHERE task_id LIKE ?", (f"{PREFIX}%",)) connection.execute("DELETE FROM subtitle_jobs WHERE task_id LIKE ?", (f"{PREFIX}%",)) + connection.execute( + "DELETE FROM subtitle_cues WHERE revision_id IN (SELECT id FROM subtitle_revisions WHERE track_id IN (SELECT id FROM subtitle_tracks WHERE task_id LIKE ?))", + (f"{PREFIX}%",), + ) + connection.execute( + "DELETE FROM subtitle_revisions WHERE track_id IN (SELECT id FROM subtitle_tracks WHERE task_id LIKE ?)", + (f"{PREFIX}%",), + ) + connection.execute("DELETE FROM subtitle_tracks WHERE task_id LIKE ?", (f"{PREFIX}%",)) connection.execute("DELETE FROM output_clip WHERE task_id LIKE ?", (f"{PREFIX}%",)) connection.execute("DELETE FROM clip_candidates WHERE task_id LIKE ?", (f"{PREFIX}%",)) connection.execute("DELETE FROM tasks WHERE id LIKE ?", (f"{PREFIX}%",)) @@ -356,20 +365,42 @@ def test_subtitle_sync_updates_only_unscheduled_video_sources( subtitled_path.write_bytes(b"subtitled") now = _time() with get_connection() as connection: + track_id = f"{PREFIX}track-{uuid4().hex[:8]}" + revision_id = f"{PREFIX}revision-{uuid4().hex[:8]}" + connection.execute( + """ + INSERT INTO subtitle_tracks ( + id, task_id, track_type, output_clip_id, name, active_revision_id, + sync_status, created_at, updated_at + ) VALUES (?, ?, 'clip', ?, '已审核字幕', ?, 'manual', ?, ?) + """, + (track_id, task_id, output_id, revision_id, now, now), + ) + connection.execute( + """ + INSERT INTO subtitle_revisions ( + id, track_id, revision_number, origin, status, cue_count, + checksum, created_at, approved_at + ) VALUES (?, ?, 1, 'manual', 'approved', 1, 'verified-test', ?, ?) + """, + (revision_id, track_id, now, now), + ) connection.execute( """ INSERT INTO subtitle_jobs ( - id, task_id, output_clip_id, status, output_file_path, - is_active, created_at, updated_at - ) VALUES (?, ?, ?, 'completed', ?, 1, ?, ?) + id, task_id, output_clip_id, revision_id, status, output_file_path, + validation_status, verified_at, is_active, created_at, updated_at + ) VALUES (?, ?, ?, ?, 'completed', ?, 'verified', ?, 1, ?, ?) """, ( f"{PREFIX}subtitle-{uuid4().hex[:8]}", task_id, output_id, + revision_id, str(subtitled_path), now, now, + now, ), ) connection.commit() diff --git a/tests/test_publish_worker_client.py b/tests/test_publish_worker_client.py index 3b727c7..2f5cdd0 100644 --- a/tests/test_publish_worker_client.py +++ b/tests/test_publish_worker_client.py @@ -71,7 +71,9 @@ def fake_urlopen(request, timeout): }) monkeypatch.setattr("app.services.publishers.worker_client.urlopen", fake_urlopen) - result = PublishWorkerClient("http://127.0.0.1:8765", "secret-token", 7).publish({"job_id": "job-1"}) + result = PublishWorkerClient("http://127.0.0.1:8765", "secret-token", 7).publish( + {"job_id": "job-1", "execution_id": "execution-1", "account_id": "account-1"} + ) assert result.outcome == PublishOutcome.PUBLISHED assert result.remote_video_id == "BV123" assert captured == {"authorization": "Bearer secret-token", "timeout": 7} @@ -83,7 +85,22 @@ def timeout(*_, **__): monkeypatch.setattr("app.services.publishers.worker_client.urlopen", timeout) with pytest.raises(PublishWorkerUnavailable) as caught: - PublishWorkerClient("http://127.0.0.1:8765", "token", 2).publish({"job_id": "job-1"}) + PublishWorkerClient("http://127.0.0.1:8765", "token", 2).publish( + {"job_id": "job-1", "execution_id": "execution-1", "account_id": "account-1"} + ) + assert caught.value.request_may_have_been_received is True + + +def test_worker_connection_reset_after_publish_is_marked_as_possibly_received(monkeypatch): + def reset(*_, **__): + raise ConnectionResetError("connection reset after response") + + monkeypatch.setattr("app.services.publishers.worker_client.urlopen", reset) + with pytest.raises(PublishWorkerUnavailable) as caught: + PublishWorkerClient("http://127.0.0.1:8765", "token", 2).publish( + {"job_id": "job-1", "execution_id": "execution-1", "account_id": "account-1"} + ) + assert caught.value.request_may_have_been_received is True diff --git a/tests/test_repair_foreign_key_integrity.py b/tests/test_repair_foreign_key_integrity.py new file mode 100644 index 0000000..d58522d --- /dev/null +++ b/tests/test_repair_foreign_key_integrity.py @@ -0,0 +1,263 @@ +"""外键完整性修复脚本的安全回归测试。 + +这些测试使用独立的临时 SQLite 数据库,不接触项目活动数据库。 +""" + +from __future__ import annotations + +import sqlite3 +from pathlib import Path + +import pytest + +from app.db import database as database_module +from scripts.repair_foreign_key_integrity import ( + apply_repair_plan, + build_repair_plan, +) + + +def _connect(database_path: Path) -> sqlite3.Connection: + connection = sqlite3.connect(str(database_path)) + connection.row_factory = sqlite3.Row + return connection + + +def _foreign_key_violations(database_path: Path) -> list[tuple]: + with _connect(database_path) as connection: + return connection.execute("PRAGMA foreign_key_check").fetchall() + + +def _assert_portable_backup(database_path: Path) -> None: + with _connect(database_path) as connection: + journal_mode = connection.execute("PRAGMA journal_mode").fetchone()[0] + assert str(journal_mode).lower() == "delete" + for suffix in ("-wal", "-shm", "-journal"): + assert not Path(f"{database_path}{suffix}").exists() + + +def _create_database(tmp_path: Path) -> Path: + database_path = tmp_path / "workflow.sqlite3" + runtime_settings = database_module.settings + original_values = { + "database_path": runtime_settings.database_path, + "data_dir": runtime_settings.data_dir, + "tasks_dir": runtime_settings.tasks_dir, + } + try: + object.__setattr__(runtime_settings, "database_path", database_path) + object.__setattr__(runtime_settings, "data_dir", tmp_path) + object.__setattr__(runtime_settings, "tasks_dir", tmp_path / "tasks") + database_module.init_db() + finally: + for name, value in original_values.items(): + object.__setattr__(runtime_settings, name, value) + return database_path + + +def _insert_task(connection: sqlite3.Connection, task_id: str) -> None: + now = "2026-08-24T00:00:00+00:00" + connection.execute( + """ + INSERT INTO tasks (id, task_name, task_dir_name, source_type, created_at, updated_at) + VALUES (?, ?, ?, 'upload', ?, ?) + """, + (task_id, task_id, task_id, now, now), + ) + + +def _insert_orphan_references( + database_path: Path, + *, + publish_task_id: str = "task-a", + subtitle_task_id: str | None = "task-a", + orphan_output_clip_id: str = "orphan-clip-001", +) -> None: + with _connect(database_path) as connection: + connection.execute("PRAGMA foreign_keys = OFF") + _insert_task(connection, publish_task_id) + if subtitle_task_id and subtitle_task_id != publish_task_id: + _insert_task(connection, subtitle_task_id) + now = "2026-08-24T00:00:00+00:00" + connection.execute( + """ + INSERT INTO publish_jobs ( + id, task_id, output_clip_id, platform, video_file_path, video_path, + created_at, updated_at + ) VALUES (?, ?, ?, 'douyin', ?, ?, ?, ?) + """, + ( + "publish-orphan-001", + publish_task_id, + orphan_output_clip_id, + r"D:\private\source.mp4", + r"D:\private\source.mp4", + now, + now, + ), + ) + if subtitle_task_id: + connection.execute( + """ + INSERT INTO subtitle_jobs ( + id, task_id, output_clip_id, subtitle_file_path, + output_file_path, created_at, updated_at + ) VALUES (?, ?, ?, ?, ?, ?, ?) + """, + ( + "subtitle-orphan-001", + subtitle_task_id, + orphan_output_clip_id, + r"D:\private\subtitle.srt", + r"D:\private\subtitle.mp4", + now, + now, + ), + ) + connection.commit() + + +def _insert_unexpected_task_orphan(database_path: Path) -> None: + with _connect(database_path) as connection: + connection.execute("PRAGMA foreign_keys = OFF") + now = "2026-08-24T00:00:00+00:00" + connection.execute( + """ + INSERT INTO output_clip ( + id, task_id, clip_candidate_id, output_file_path, output_file_name, + status, created_at, updated_at + ) VALUES (?, ?, NULL, ?, ?, 'completed', ?, ?) + """, + ( + "output-with-missing-task", + "task-does-not-exist", + r"D:\private\original.mp4", + "original.mp4", + now, + now, + ), + ) + connection.commit() + + +def test_build_repair_plan_is_dry_run_and_counts_supported_tombstone( + tmp_path: Path, +): + database_path = _create_database(tmp_path) + _insert_orphan_references(database_path) + before = _foreign_key_violations(database_path) + + plan = build_repair_plan(database_path) + + assert plan.violation_count == 2 + assert plan.tombstone_count == 1 + assert not plan.unsupported_violations + assert len(_foreign_key_violations(database_path)) == len(before) == 2 + with _connect(database_path) as connection: + assert connection.execute( + "SELECT COUNT(*) FROM output_clip WHERE id = ?", + ("orphan-clip-001",), + ).fetchone()[0] == 0 + + +def test_apply_creates_verified_backup_and_clears_foreign_key_violations( + tmp_path: Path, +): + database_path = _create_database(tmp_path) + _insert_orphan_references(database_path) + backup_dir = tmp_path / "backups" + + result = apply_repair_plan( + database_path, + backup_dir, + expected_violation_count=2, + ) + + backup_path = Path(result["backup_path"]) + assert backup_path.exists() + _assert_portable_backup(backup_path) + assert result["before_violation_count"] == 2 + assert result["after_violation_count"] == 0 + assert result["tombstone_count"] == 1 + assert _foreign_key_violations(database_path) == [] + with _connect(database_path) as connection: + assert connection.execute("PRAGMA quick_check").fetchone()[0] == "ok" + with _connect(backup_path) as backup_connection: + assert backup_connection.execute("PRAGMA quick_check").fetchone()[0] == "ok" + assert len(backup_connection.execute("PRAGMA foreign_key_check").fetchall()) == 2 + + +def test_tombstone_is_inactive_and_does_not_copy_media_paths(tmp_path: Path): + database_path = _create_database(tmp_path) + _insert_orphan_references(database_path) + + apply_repair_plan(database_path, tmp_path / "backups", expected_violation_count=2) + + with _connect(database_path) as connection: + row = connection.execute( + """ + SELECT task_id, status, is_active, output_file_path, output_file_name + FROM output_clip WHERE id = ? + """, + ("orphan-clip-001",), + ).fetchone() + assert row["task_id"] == "task-a" + assert row["status"] == "integrity_repair_tombstone" + assert row["is_active"] == 0 + assert row["output_file_path"] in (None, "") + assert row["output_file_name"] in (None, "") + + +def test_apply_rejects_unexpected_foreign_key_violation_without_writing( + tmp_path: Path, +): + database_path = _create_database(tmp_path) + _insert_orphan_references(database_path) + _insert_unexpected_task_orphan(database_path) + before = len(_foreign_key_violations(database_path)) + backup_dir = tmp_path / "backups" + + plan = build_repair_plan(database_path) + assert plan.unsupported_violations + with pytest.raises((RuntimeError, ValueError), match="(unsupported|不支持|异常|拒绝)"): + apply_repair_plan(database_path, backup_dir, expected_violation_count=before) + + assert len(_foreign_key_violations(database_path)) == before == 3 + with _connect(database_path) as connection: + assert connection.execute( + "SELECT COUNT(*) FROM output_clip WHERE id = ?", + ("orphan-clip-001",), + ).fetchone()[0] == 0 + assert not list(backup_dir.glob("*")) if backup_dir.exists() else True + + +def test_apply_rejects_cross_task_orphan_reference_without_writing(tmp_path: Path): + database_path = _create_database(tmp_path) + _insert_orphan_references(database_path, subtitle_task_id="task-b") + before = len(_foreign_key_violations(database_path)) + backup_dir = tmp_path / "backups" + + plan = build_repair_plan(database_path) + assert plan.unsupported_violations + with pytest.raises((RuntimeError, ValueError), match="(task|任务|冲突|拒绝)"): + apply_repair_plan(database_path, backup_dir, expected_violation_count=before) + + assert len(_foreign_key_violations(database_path)) == before == 2 + assert not list(backup_dir.glob("*")) if backup_dir.exists() else True + + +def test_apply_rejects_unexpected_violation_count_without_writing(tmp_path: Path): + database_path = _create_database(tmp_path) + _insert_orphan_references(database_path) + backup_dir = tmp_path / "backups" + + with pytest.raises((RuntimeError, ValueError), match="(count|数量|expected|预期)"): + apply_repair_plan(database_path, backup_dir, expected_violation_count=99) + + assert len(_foreign_key_violations(database_path)) == 2 + with _connect(database_path) as connection: + assert connection.execute( + "SELECT COUNT(*) FROM output_clip WHERE id = ?", + ("orphan-clip-001",), + ).fetchone()[0] == 0 + assert not list(backup_dir.glob("*")) if backup_dir.exists() else True diff --git a/tests/test_schema_migration_ledger.py b/tests/test_schema_migration_ledger.py new file mode 100644 index 0000000..29ae83a --- /dev/null +++ b/tests/test_schema_migration_ledger.py @@ -0,0 +1,361 @@ +import sqlite3 +from concurrent.futures import ThreadPoolExecutor +from types import SimpleNamespace + +import pytest + +from app.db import database as database_module + + +@pytest.fixture +def isolated_database(monkeypatch, tmp_path): + data_dir = tmp_path / "data" + database_path = data_dir / "migration-ledger.sqlite3" + tasks_dir = tmp_path / "tasks" + test_settings = SimpleNamespace( + data_dir=data_dir, + database_path=database_path, + tasks_dir=tasks_dir, + publish_default_mode="local_browser", + ) + monkeypatch.setattr(database_module, "settings", test_settings) + return database_path + + +def _connect(database_path): + connection = sqlite3.connect(database_path, timeout=10) + connection.row_factory = sqlite3.Row + connection.execute("PRAGMA busy_timeout = 10000") + return connection + + +def _index_names(connection): + return { + row["name"] + for row in connection.execute( + "SELECT name FROM sqlite_master WHERE type = 'index'" + ).fetchall() + } + + +def test_init_records_migration_once_and_switches_unique_index(isolated_database): + database_module.init_db() + database_module.init_db() + + with _connect(isolated_database) as connection: + migrations = connection.execute( + "SELECT version, name, checksum, applied_at FROM schema_migrations" + ).fetchall() + indexes = _index_names(connection) + + assert len(migrations) == 2 + migrations_by_version = {row["version"]: row for row in migrations} + publish_migration = migrations_by_version[database_module.PUBLISH_ACTIVE_INDEX_MIGRATION_VERSION] + assert publish_migration["name"] == database_module.PUBLISH_ACTIVE_INDEX_MIGRATION_NAME + assert publish_migration["checksum"] == database_module.PUBLISH_ACTIVE_INDEX_MIGRATION_CHECKSUM + assert publish_migration["applied_at"] + upload_migration = migrations_by_version[database_module.TASK_UPLOAD_ONLY_MIGRATION_VERSION] + assert upload_migration["name"] == database_module.TASK_UPLOAD_ONLY_MIGRATION_NAME + assert upload_migration["checksum"] == database_module.TASK_UPLOAD_ONLY_MIGRATION_CHECKSUM + assert upload_migration["applied_at"] + assert database_module.PUBLISH_ACTIVE_UNIQUE_INDEX_NAME in indexes + assert database_module.PUBLISH_ACTIVE_UNIQUE_INDEX_LEGACY_NAME not in indexes + + +def test_checksum_drift_refuses_startup(isolated_database): + database_module.init_db() + with _connect(isolated_database) as connection: + connection.execute( + "UPDATE schema_migrations SET checksum = 'tampered' WHERE version = ?", + (database_module.PUBLISH_ACTIVE_INDEX_MIGRATION_VERSION,), + ) + connection.commit() + + with pytest.raises(database_module.SchemaMigrationError, match="checksum"): + database_module.init_db() + + +def test_existing_database_is_backed_up_before_publish_index_migration(isolated_database): + database_module.init_db() + with _connect(isolated_database) as connection: + connection.execute( + "DELETE FROM schema_migrations WHERE version = ?", + (database_module.PUBLISH_ACTIVE_INDEX_MIGRATION_VERSION,), + ) + connection.execute( + f"DROP INDEX {database_module.PUBLISH_ACTIVE_UNIQUE_INDEX_NAME}" + ) + connection.commit() + + database_module.init_db() + + backups = list( + (isolated_database.parent / "backups").glob( + "workflow-before-publish-active-unique-index-v2-*.sqlite3" + ) + ) + assert len(backups) == 1 + + +def test_legacy_nas_task_is_backed_up_and_normalized(isolated_database): + database_module.init_db() + legacy_path = r"E:\\历史原片\\source.mp4" + with _connect(isolated_database) as connection: + connection.execute( + "DELETE FROM schema_migrations WHERE version = ?", + (database_module.TASK_UPLOAD_ONLY_MIGRATION_VERSION,), + ) + connection.execute( + """ + INSERT INTO tasks ( + id, task_name, source_type, nas_file_path, status, created_at, updated_at + ) VALUES ('legacy-nas-task', '历史 NAS 任务', 'nas', ?, 'completed', 'now', 'now') + """, + (legacy_path,), + ) + connection.commit() + + database_module.init_db() + + with _connect(isolated_database) as connection: + task = connection.execute( + "SELECT source_type, original_video_path, nas_file_path FROM tasks WHERE id = 'legacy-nas-task'" + ).fetchone() + ledger_count = connection.execute( + "SELECT COUNT(*) FROM schema_migrations WHERE version = ?", + (database_module.TASK_UPLOAD_ONLY_MIGRATION_VERSION,), + ).fetchone()[0] + backups = list( + (isolated_database.parent / "backups").glob("workflow-before-task-upload-only-*.sqlite3") + ) + assert dict(task) == { + "source_type": "upload", + "original_video_path": legacy_path, + "nas_file_path": None, + } + assert ledger_count == 1 + assert len(backups) == 1 + + +def test_applied_migration_with_drifted_index_refuses_startup(isolated_database): + database_module.init_db() + with _connect(isolated_database) as connection: + connection.execute( + f"DROP INDEX {database_module.PUBLISH_ACTIVE_UNIQUE_INDEX_NAME}" + ) + connection.execute( + f"CREATE UNIQUE INDEX {database_module.PUBLISH_ACTIVE_UNIQUE_INDEX_NAME} " + "ON publish_jobs(id)" + ) + connection.commit() + + with pytest.raises(database_module.SchemaMigrationError, match="定义发生漂移"): + database_module.init_db() + + +def test_applied_migration_with_missing_index_refuses_startup(isolated_database): + database_module.init_db() + with _connect(isolated_database) as connection: + connection.execute( + f"DROP INDEX {database_module.PUBLISH_ACTIVE_UNIQUE_INDEX_NAME}" + ) + connection.commit() + + with pytest.raises(database_module.SchemaMigrationError, match="不存在"): + database_module.init_db() + + +def test_applied_migration_with_legacy_index_refuses_startup(isolated_database): + database_module.init_db() + with _connect(isolated_database) as connection: + connection.execute( + f""" + CREATE UNIQUE INDEX {database_module.PUBLISH_ACTIVE_UNIQUE_INDEX_LEGACY_NAME} + ON publish_jobs(output_clip_id, platform, publish_mode) + WHERE status NOT IN ('PUBLISHED', 'EXPORTED', 'CANCELLED') + AND output_clip_id IS NOT NULL AND output_clip_id <> '' + """ + ) + connection.commit() + + with pytest.raises(database_module.SchemaMigrationError, match="仍存在"): + database_module.init_db() + + +def test_malformed_ledger_is_backed_up_then_rejected(isolated_database): + database_module.init_db() + with _connect(isolated_database) as connection: + connection.execute("DROP TABLE schema_migrations") + connection.execute("CREATE TABLE schema_migrations (version TEXT, name TEXT)") + connection.commit() + + with pytest.raises(database_module.SchemaMigrationError, match="账本结构不完整"): + database_module.init_db() + + backups = list( + (isolated_database.parent / "backups").glob( + "workflow-before-publish-active-unique-index-v2-*.sqlite3" + ) + ) + assert len(backups) == 1 + + +def test_unique_index_allows_failed_history_but_rejects_two_active_jobs(isolated_database): + database_module.init_db() + with _connect(isolated_database) as connection: + connection.executemany( + """ + INSERT INTO publish_jobs ( + id, task_id, output_clip_id, platform, publish_mode, + status, created_at, updated_at + ) VALUES (?, 'task-1', 'clip-1', 'douyin', 'local_browser', ?, ?, ?) + """, + [ + ("failed-1", "FAILED", "2026-08-24T00:00:00+08:00", "2026-08-24T00:00:00+08:00"), + ("failed-2", "FAILED", "2026-08-24T00:01:00+08:00", "2026-08-24T00:01:00+08:00"), + ("active-1", "WAITING", "2026-08-24T00:02:00+08:00", "2026-08-24T00:02:00+08:00"), + ], + ) + with pytest.raises(sqlite3.IntegrityError, match="UNIQUE constraint failed"): + connection.execute( + """ + INSERT INTO publish_jobs ( + id, task_id, output_clip_id, platform, publish_mode, + status, created_at, updated_at + ) VALUES ( + 'active-2', 'task-1', 'clip-1', 'douyin', 'local_browser', + 'NEED_REVIEW', '2026-08-24T00:03:00+08:00', '2026-08-24T00:03:00+08:00' + ) + """ + ) + + +def test_duplicate_active_jobs_fail_without_data_rewrite_or_ledger_entry(isolated_database): + database_module.init_db() + with _connect(isolated_database) as connection: + connection.execute( + "DELETE FROM schema_migrations WHERE version = ?", + (database_module.PUBLISH_ACTIVE_INDEX_MIGRATION_VERSION,), + ) + connection.execute( + f"DROP INDEX {database_module.PUBLISH_ACTIVE_UNIQUE_INDEX_NAME}" + ) + connection.executemany( + """ + INSERT INTO publish_jobs ( + id, task_id, output_clip_id, platform, publish_mode, + status, created_at, updated_at + ) VALUES (?, 'task-1', 'clip-1', 'douyin', 'local_browser', ?, ?, ?) + """, + [ + ("job-waiting", "WAITING", "2026-08-24T00:00:00+08:00", "2026-08-24T00:00:00+08:00"), + ("job-review", "NEED_REVIEW", "2026-08-24T00:01:00+08:00", "2026-08-24T00:01:00+08:00"), + ], + ) + connection.commit() + + with pytest.raises(database_module.SchemaMigrationError, match="重复的活动发布任务"): + database_module.init_db() + + with _connect(isolated_database) as connection: + statuses = dict( + connection.execute( + "SELECT id, status FROM publish_jobs WHERE id IN ('job-waiting', 'job-review')" + ).fetchall() + ) + ledger_count = connection.execute( + "SELECT COUNT(*) FROM schema_migrations WHERE version = ?", + (database_module.PUBLISH_ACTIVE_INDEX_MIGRATION_VERSION,), + ).fetchone()[0] + indexes = _index_names(connection) + + assert statuses == {"job-waiting": "WAITING", "job-review": "NEED_REVIEW"} + assert ledger_count == 0 + assert database_module.PUBLISH_ACTIVE_UNIQUE_INDEX_NAME not in indexes + + +def test_new_index_verification_failure_preserves_legacy_index( + isolated_database, + monkeypatch, +): + database_module.init_db() + with _connect(isolated_database) as connection: + connection.execute( + "DELETE FROM schema_migrations WHERE version = ?", + (database_module.PUBLISH_ACTIVE_INDEX_MIGRATION_VERSION,), + ) + connection.execute( + f"DROP INDEX {database_module.PUBLISH_ACTIVE_UNIQUE_INDEX_NAME}" + ) + connection.execute( + f""" + CREATE UNIQUE INDEX {database_module.PUBLISH_ACTIVE_UNIQUE_INDEX_LEGACY_NAME} + ON publish_jobs(output_clip_id, platform, publish_mode) + WHERE status NOT IN ('PUBLISHED', 'EXPORTED', 'CANCELLED') + AND output_clip_id IS NOT NULL AND output_clip_id <> '' + """ + ) + connection.commit() + + def fail_verification(_connection): + raise database_module.SchemaMigrationError("simulated index verification failure") + + monkeypatch.setattr( + database_module, + "_verify_publish_active_unique_index", + fail_verification, + ) + + with _connect(isolated_database) as connection: + with pytest.raises(database_module.SchemaMigrationError, match="simulated"): + database_module._run_schema_migrations(connection) + + with _connect(isolated_database) as connection: + indexes = _index_names(connection) + ledger_count = connection.execute( + "SELECT COUNT(*) FROM schema_migrations WHERE version = ?", + (database_module.PUBLISH_ACTIVE_INDEX_MIGRATION_VERSION,), + ).fetchone()[0] + + assert database_module.PUBLISH_ACTIVE_UNIQUE_INDEX_LEGACY_NAME in indexes + assert database_module.PUBLISH_ACTIVE_UNIQUE_INDEX_NAME not in indexes + assert ledger_count == 0 + + +def test_concurrent_migration_runners_share_one_ledger_entry(isolated_database): + database_module.init_db() + with _connect(isolated_database) as connection: + connection.execute( + "DELETE FROM schema_migrations WHERE version = ?", + (database_module.PUBLISH_ACTIVE_INDEX_MIGRATION_VERSION,), + ) + connection.execute( + f"DROP INDEX {database_module.PUBLISH_ACTIVE_UNIQUE_INDEX_NAME}" + ) + connection.commit() + + def run_migration(): + with _connect(isolated_database) as connection: + database_module._run_schema_migrations(connection) + + with ThreadPoolExecutor(max_workers=2) as executor: + list(executor.map(lambda _: run_migration(), range(2))) + + with _connect(isolated_database) as connection: + ledger_count = connection.execute( + "SELECT COUNT(*) FROM schema_migrations WHERE version = ?", + (database_module.PUBLISH_ACTIVE_INDEX_MIGRATION_VERSION,), + ).fetchone()[0] + indexes = _index_names(connection) + + assert ledger_count == 1 + assert database_module.PUBLISH_ACTIVE_UNIQUE_INDEX_NAME in indexes + + +def test_noncritical_index_errors_are_not_silenced(): + class FailingConnection: + def execute(self, _sql): + raise sqlite3.OperationalError("simulated index failure") + + with pytest.raises(sqlite3.OperationalError, match="simulated index failure"): + database_module._create_indexes(FailingConnection()) diff --git a/tests/test_selection_profile_default.py b/tests/test_selection_profile_default.py index 473caa1..34cf751 100644 --- a/tests/test_selection_profile_default.py +++ b/tests/test_selection_profile_default.py @@ -1,134 +1,91 @@ -"""选片模式固定为「康熙笑点选片模式」的默认值与兼容测试。 - -新建任务不再提供「通用内容价值」选项,默认使用 variety_comedy; -历史 general 任务保持原样,label 显示为「通用模式(历史任务)」。 -""" +"""新建任务必须显式选择选片模式,历史数据保持兼容。""" from __future__ import annotations +import pytest from fastapi.testclient import TestClient +from pydantic import ValidationError from app.core.config import settings from app.db.database import get_connection, init_db from app.main import app -from app.models.task import TaskCreate, TaskSelectionSettingsUpdate +from app.models.task import TaskCreate from app.routers import tasks as tasks_router def _headers() -> dict[str, str]: - if not settings.local_admin_token: - return {} - return {"Authorization": f"Bearer {settings.local_admin_token}"} - + return {"Authorization": f"Bearer {settings.local_admin_token}"} if settings.local_admin_token else {} -def test_task_create_defaults_to_variety_comedy(): - payload = TaskCreate(task_name="选片默认值测试") - assert payload.selection_profile == "variety_comedy" - settings_update = TaskSelectionSettingsUpdate() - assert settings_update.selection_profile == "variety_comedy" +def test_task_create_requires_selection_profile(): + with pytest.raises(ValidationError): + TaskCreate(task_name="缺少模式") + for profile in ("general", "variety_comedy", "long_live_talk"): + assert TaskCreate(task_name=profile, selection_profile=profile).selection_profile == profile -def test_new_task_page_fixed_to_kangxi_profile(): +def test_new_task_page_has_required_three_option_select(): response = TestClient(app).get("/tasks/new", headers=_headers()) - assert response.status_code == 200 - assert '' in response.text - assert '' in response.text + assert 'name="selection_profile" value="variety_comedy"' not in response.text + assert "通用内容价值" in response.text assert "康熙笑点选片模式" in response.text + assert "长直播高光(语言类)" in response.text -def test_upload_form_defaults_to_variety_comedy(monkeypatch, tmp_path): +def test_upload_form_rejects_missing_selection_profile(): + response = TestClient(app).post( + "/api/tasks/upload", + data={"task_name": "缺少模式", "platform": "general"}, + files={"video_file": ("source.mp4", b"fake-video", "video/mp4")}, + headers=_headers(), + ) + assert response.status_code == 422 + assert response.json()["detail"] == "请选择选片模式" + + +def test_json_task_creation_api_is_removed(): + response = TestClient(app).post("/api/tasks", json={"task_name": "缺少模式"}, headers=_headers()) + assert response.status_code == 405 + + +@pytest.mark.parametrize("profile", ["general", "variety_comedy", "long_live_talk"]) +def test_upload_accepts_each_explicit_profile(monkeypatch, tmp_path, profile): captured: dict[str, TaskCreate] = {} saved_video = tmp_path / "source.mp4" saved_video.write_bytes(b"fake-video") + monkeypatch.setattr(tasks_router, "allocate_task_dir_name", lambda *_args, **_kwargs: "profile-test") + monkeypatch.setattr(tasks_router, "save_uploaded_video", lambda *_args, **_kwargs: saved_video) - monkeypatch.setattr( - tasks_router, - "allocate_task_dir_name", - lambda task_name, exclude_task_id=None: "test-default-selection", - ) - monkeypatch.setattr( - tasks_router, - "save_uploaded_video", - lambda task_id, filename, source, task_dir_name: saved_video, - ) - - def fake_create_task_record( - upload_payload: TaskCreate, - task_id: str | None = None, - task_dir_name: str | None = None, - ) -> dict: - captured["payload"] = upload_payload - return { - "id": task_id, - "task_name": upload_payload.task_name, - "detail_url": f"/tasks/{task_id}", - "message": "任务已创建并写入数据库。", - } - - monkeypatch.setattr(tasks_router.task_service, "create_task_record", fake_create_task_record) + def fake_create(payload, task_id=None, task_dir_name=None): + captured["payload"] = payload + return {"id": task_id, "detail_url": f"/tasks/{task_id}", "message": "已创建"} + monkeypatch.setattr(tasks_router.task_service, "create_task_record", fake_create) response = TestClient(app).post( "/api/tasks/upload", - data={"task_name": "上传选片默认值测试", "platform": "general"}, + data={"task_name": profile, "selection_profile": profile}, files={"video_file": ("source.mp4", b"fake-video", "video/mp4")}, headers=_headers(), ) - assert response.status_code == 200 - assert captured["payload"].selection_profile == "variety_comedy" + assert captured["payload"].selection_profile == profile -def test_variety_comedy_task_label_is_kangxi(monkeypatch): +def test_historical_general_and_variety_modes_remain_unchanged(monkeypatch): from app.services.task_lifecycle_service import create_task_record from app.services.task_service import get_task init_db() - task_id = "sel-profile-label-test" - monkeypatch.setattr( - "app.services.task_lifecycle_service.create_task_directory", - lambda task_id, task_dir_name: None, - ) - - try: - create_task_record( - TaskCreate(task_name="康熙默认任务"), - task_id=task_id, - task_dir_name=task_id, - ) - task = get_task(task_id, include_video_probe=False) - assert task["selection_profile"] == "variety_comedy" - assert task["selection_profile_label"] == "康熙笑点选片模式" - finally: - with get_connection() as connection: - connection.execute("DELETE FROM tasks WHERE id = ?", (task_id,)) - connection.commit() - - -def test_legacy_general_task_label_kept(monkeypatch): - from app.services.task_lifecycle_service import create_task_record - from app.services.task_service import get_task - - init_db() - task_id = "sel-profile-general-test" - monkeypatch.setattr( - "app.services.task_lifecycle_service.create_task_directory", - lambda task_id, task_dir_name: None, - ) - + monkeypatch.setattr("app.services.task_lifecycle_service.create_task_directory", lambda *_args: None) + task_ids = ["sel-history-general", "sel-history-variety"] try: - create_task_record( - TaskCreate(task_name="历史通用任务", selection_profile="general"), - task_id=task_id, - task_dir_name=task_id, - ) - task = get_task(task_id, include_video_probe=False) - assert task["selection_profile"] == "general" - assert task["selection_profile_label"] == "通用模式(历史任务)" + create_task_record(TaskCreate(task_name="历史通用", selection_profile="general"), task_id=task_ids[0], task_dir_name=task_ids[0]) + create_task_record(TaskCreate(task_name="历史综艺", selection_profile="variety_comedy"), task_id=task_ids[1], task_dir_name=task_ids[1]) + assert get_task(task_ids[0], include_video_probe=False)["selection_profile"] == "general" + assert get_task(task_ids[1], include_video_probe=False)["selection_profile"] == "variety_comedy" finally: with get_connection() as connection: - connection.execute("DELETE FROM tasks WHERE id = ?", (task_id,)) + connection.executemany("DELETE FROM tasks WHERE id = ?", [(item,) for item in task_ids]) connection.commit() diff --git a/tests/test_split_services.py b/tests/test_split_services.py index 70aa1ce..edefe73 100644 --- a/tests/test_split_services.py +++ b/tests/test_split_services.py @@ -86,6 +86,7 @@ def test_create_task_record_minimal(self): task_name="测试任务-最小", source_type="upload", platform="general", + selection_profile="general", max_clip_duration=5, candidate_clip_count=8, ai_preference="", @@ -99,20 +100,11 @@ def test_create_task_record_minimal(self): } assert "detail_url" in result - def test_create_task_record_with_nas_source(self): - """NAS 来源任务应有 pending_video 状态(因为没有文件校验)""" - payload = TaskCreate( - task_name="NAS 测试", - source_type="nas", - platform="bilibili", - nas_file_path="/not/exist/video.mp4", - max_clip_duration=3, - candidate_clip_count=5, - ai_preference="", - ) - # NAS 路径不存在时会抛出 ValueError - with pytest.raises(ValueError, match="文件不存在|路径不存在"): - create_task_record(payload, task_id="test-nas-001") + def test_task_create_model_no_longer_exposes_nas_source(self): + """新建任务模型只接收上传后的原片路径,不再暴露 NAS 参数。""" + fields = getattr(TaskCreate, "model_fields", getattr(TaskCreate, "__fields__", {})) + assert "source_type" not in fields + assert "nas_file_path" not in fields def test_update_task_status_ok(self): """更新已存在任务的状态""" @@ -121,6 +113,7 @@ def test_update_task_status_ok(self): task_name="状态更新测试", source_type="upload", platform="general", + selection_profile="general", max_clip_duration=3, candidate_clip_count=5, ai_preference="", @@ -144,6 +137,7 @@ def test_permanent_delete_task(self): task_name="删除测试", source_type="upload", platform="general", + selection_profile="general", max_clip_duration=3, candidate_clip_count=5, ai_preference="", @@ -163,6 +157,7 @@ def test_update_candidate_clip_count_invalid(self): task_name="数量测试", source_type="upload", platform="general", + selection_profile="general", max_clip_duration=3, candidate_clip_count=5, ai_preference="", @@ -187,6 +182,9 @@ def cleanup_ai_rows(self): yield with get_connection() as connection: + connection.execute("DELETE FROM publish_jobs WHERE task_id LIKE 'test-ai-%'") + connection.execute("DELETE FROM workflow_jobs WHERE task_id LIKE 'test-ai-%'") + connection.execute("DELETE FROM output_clip WHERE task_id LIKE 'test-ai-%'") connection.execute("DELETE FROM clip_candidates WHERE task_id LIKE 'test-ai-%'") connection.execute("DELETE FROM ai_analysis_runs WHERE task_id LIKE 'test-ai-%'") connection.execute("DELETE FROM tasks WHERE id LIKE 'test-ai-%'") @@ -194,7 +192,8 @@ def cleanup_ai_rows(self): def test_missing_transcript_triggers_error(self): """没有转写文件时 AI 分析应报错""" - from app.services.ai_analysis_workflow_service import process_task_ai_analysis + from app.services import job_service + from app.services.ai_analysis_workflow_service import process_task_ai_analysis, queue_task_ai_analysis from app.services.task_lifecycle_service import create_task_record from app.models.task import TaskCreate @@ -202,14 +201,21 @@ def test_missing_transcript_triggers_error(self): task_name="AI 测试-无转写", source_type="upload", platform="general", + selection_profile="general", max_clip_duration=3, candidate_clip_count=5, ai_preference="", ) create_task_record(payload, task_id="test-ai-001") + update_task_status("test-ai-001", TaskStatus.pending_ai) - with pytest.raises(ValueError, match="转写"): - process_task_ai_analysis("test-ai-001") + job, created = queue_task_ai_analysis("test-ai-001") + assert created is True + claimed = job_service.claim_job(job["id"], "test-ai-owner") + assert claimed + with job_service.job_lease_context(job["id"], "test-ai-owner", claimed["lease_token"]): + with pytest.raises(ValueError, match="转写"): + process_task_ai_analysis("test-ai-001") def test_analysis_status_idle(self): """刚创建的任务 AI 分析状态应为 idle""" @@ -221,6 +227,7 @@ def test_analysis_status_idle(self): task_name="AI 状态测试", source_type="upload", platform="general", + selection_profile="general", max_clip_duration=3, candidate_clip_count=5, ai_preference="", @@ -239,7 +246,7 @@ def test_replace_clip_candidates_keeps_new_rows(self): task_id = "test-ai-replace" create_task_record( - TaskCreate(task_name="候选替换测试", source_type="upload", platform="general"), + TaskCreate(task_name="候选替换测试", source_type="upload", platform="general", selection_profile="general"), task_id=task_id, ) _replace_clip_candidates( @@ -275,7 +282,7 @@ def test_replace_clip_candidates_rolls_back_when_insert_fails(self): task_id = "test-ai-replace-rollback" create_task_record( - TaskCreate(task_name="候选回滚测试", source_type="upload", platform="general"), + TaskCreate(task_name="候选回滚测试", source_type="upload", platform="general", selection_profile="general"), task_id=task_id, ) now = _now_iso() @@ -328,6 +335,74 @@ def test_replace_clip_candidates_rolls_back_when_insert_fails(self): assert len(clips) == 1 assert clips[0]["title"] == "旧片段" + def test_replace_clip_candidates_rejects_existing_output_reference(self): + from app.db.database import get_connection + from app.services.ai_analysis_workflow_service import ( + AIAnalysisConflictError, + _replace_clip_candidates, + ) + from app.services.task_service import list_clip_candidates + + task_id = "test-ai-output-reference" + create_task_record(TaskCreate(task_name="已有切片", selection_profile="general"), task_id=task_id) + with get_connection() as connection: + connection.execute( + """ + INSERT INTO clip_candidates ( + id, task_id, clip_key, title, start_time, end_time, duration_seconds, + selected_by_default, enabled, reviewed, created_at, updated_at + ) VALUES ('test-ai-output-reference_clip_001', ?, 'old', '旧候选', + '00:00', '00:30', 30, 1, 1, 1, 'now', 'now') + """, + (task_id,), + ) + connection.execute( + """ + INSERT INTO output_clip ( + id, task_id, clip_candidate_id, output_file_path, output_file_name, + status, is_active, created_at, updated_at + ) VALUES ('test-ai-output-reference-out', ?, 'test-ai-output-reference_clip_001', + 'clip.mp4', 'clip.mp4', 'completed', 1, 'now', 'now') + """, + (task_id,), + ) + connection.commit() + + with pytest.raises(AIAnalysisConflictError, match="已有切片引用"): + _replace_clip_candidates(task_id, []) + + assert [clip["title"] for clip in list_clip_candidates(task_id)] == ["旧候选"] + with get_connection() as connection: + assert connection.execute("PRAGMA foreign_key_check").fetchall() == [] + + def test_manual_ai_reentry_is_blocked_before_status_change(self): + from app.db.database import get_connection + from app.services.ai_analysis_workflow_service import AIAnalysisConflictError, queue_task_ai_analysis + from app.services.task_service import get_task + + task_id = "test-ai-manual-reentry" + create_task_record(TaskCreate(task_name="AI 重入保护", selection_profile="general"), task_id=task_id) + update_task_status(task_id, TaskStatus.completed) + with get_connection() as connection: + connection.execute( + """ + INSERT INTO output_clip ( + id, task_id, output_file_path, output_file_name, status, is_active, created_at, updated_at + ) VALUES ('test-ai-manual-reentry-out', ?, 'clip.mp4', 'clip.mp4', + 'completed', 1, 'now', 'now') + """, + (task_id,), + ) + connection.commit() + before = get_task(task_id, include_video_probe=False) + + with pytest.raises(AIAnalysisConflictError, match="已经生成切片"): + queue_task_ai_analysis(task_id) + + after = get_task(task_id, include_video_probe=False) + assert after["status"] == before["status"] + assert after["error_message"] == before["error_message"] + def test_restore_ai_history_recreates_candidates(self): from app.services.ai_analysis_workflow_service import ( _insert_ai_analysis_run, @@ -338,7 +413,7 @@ def test_restore_ai_history_recreates_candidates(self): task_id = "test-ai-restore-history" create_task_record( - TaskCreate(task_name="历史恢复测试", source_type="upload", platform="general"), + TaskCreate(task_name="历史恢复测试", source_type="upload", platform="general", selection_profile="general"), task_id=task_id, ) payload = { @@ -393,6 +468,7 @@ def test_no_enabled_clips_triggers_error(self): task_name="切片测试-无片段", source_type="upload", platform="general", + selection_profile="general", max_clip_duration=3, candidate_clip_count=5, ai_preference="", @@ -429,6 +505,7 @@ def test_render_missing_output_clip(self): task_name="字幕测试", source_type="upload", platform="general", + selection_profile="general", max_clip_duration=3, candidate_clip_count=5, ai_preference="", @@ -454,6 +531,7 @@ def test_missing_audio_triggers_error(self): task_name="转写测试-无音频", source_type="upload", platform="general", + selection_profile="general", max_clip_duration=3, candidate_clip_count=5, ai_preference="", diff --git a/tests/test_storage_boundaries.py b/tests/test_storage_boundaries.py new file mode 100644 index 0000000..a1c758d --- /dev/null +++ b/tests/test_storage_boundaries.py @@ -0,0 +1,248 @@ +from __future__ import annotations + +import asyncio +from concurrent.futures import ThreadPoolExecutor +from pathlib import Path + +import pytest +from fastapi import HTTPException + +from app.routers import media as media_router +from app.services import storage_service + + +@pytest.fixture +def isolated_storage_boundaries(tmp_path): + runtime_settings = storage_service.settings + replacements = { + "project_root": tmp_path / "project", + "data_dir": tmp_path / "data", + "database_path": tmp_path / "data" / "missing.sqlite3", + "storage_root": tmp_path / "storage", + "tasks_dir": tmp_path / "storage" / "tasks", + } + originals = {name: getattr(runtime_settings, name) for name in replacements} + for name, value in replacements.items(): + object.__setattr__(runtime_settings, name, value) + runtime_settings.tasks_dir.mkdir(parents=True) + try: + yield replacements + finally: + for name, value in originals.items(): + object.__setattr__(runtime_settings, name, value) + + +@pytest.mark.parametrize("unsafe_name", ["..\\outside", "C:\\Windows", "\\\\server\\share"]) +def test_task_directory_rejects_unsafe_names(isolated_storage_boundaries, unsafe_name): + with pytest.raises(storage_service.StorageSafetyError, match="不安全路径"): + storage_service.create_task_directory("task-safe", unsafe_name) + + +def test_task_directory_allows_literal_tilde(isolated_storage_boundaries): + task_dir = storage_service.create_task_directory("task-tilde", "直播~精选") + + assert task_dir == storage_service.settings.tasks_dir / "直播~精选" + assert task_dir.is_dir() + + +def test_task_directory_rejects_symlink_escape(isolated_storage_boundaries, tmp_path): + outside = tmp_path / "outside" + outside.mkdir() + link = storage_service.settings.tasks_dir / "linked-task" + try: + link.symlink_to(outside, target_is_directory=True) + except OSError as exc: + pytest.skip(f"当前 Windows 环境不允许创建测试符号链接:{exc}") + + with pytest.raises(storage_service.StorageSafetyError, match="不安全"): + storage_service.get_task_directory("task-link", "linked-task") + + +def test_atomic_task_directory_reservation_avoids_same_name_collision(isolated_storage_boundaries): + def reserve() -> str: + return storage_service.allocate_task_dir_name("同名并发任务") + + with ThreadPoolExecutor(max_workers=2) as executor: + names = list(executor.map(lambda _index: reserve(), range(2))) + + assert len(set(names)) == 2 + assert all((storage_service.settings.tasks_dir / name).is_dir() for name in names) + + +def test_container_path_mapping_rejects_traversal(isolated_storage_boundaries): + with pytest.raises(storage_service.StorageSafetyError, match="不安全路径"): + storage_service.resolve_video_file_path("/workspace/tasks/../../outside.mp4") + + +def test_task_media_resolver_accepts_only_current_task_clip_directory(isolated_storage_boundaries, tmp_path): + task_dir = storage_service.create_task_directory("task-media", "task-media") + managed_clip = task_dir / "05_clips" / "clip.mp4" + managed_clip.write_bytes(b"clip") + outside_clip = tmp_path / "private.mp4" + outside_clip.write_bytes(b"private") + + accepted = storage_service.resolve_task_media_file_path( + str(managed_clip), + task_id="task-media", + task_dir_name="task-media", + allowed_subdirectories=("05_clips", "clips"), + ) + rejected = storage_service.resolve_task_media_file_path( + str(outside_clip), + task_id="task-media", + task_dir_name="task-media", + allowed_subdirectories=("05_clips", "clips"), + ) + + assert accepted == managed_clip + assert rejected is None + + +def test_task_media_resolver_rejects_symlinked_clip_escape(isolated_storage_boundaries, tmp_path): + task_dir = storage_service.create_task_directory("task-symlink", "task-symlink") + outside_clip = tmp_path / "outside.mp4" + outside_clip.write_bytes(b"private") + linked_clip = task_dir / "05_clips" / "linked.mp4" + try: + linked_clip.symlink_to(outside_clip) + except OSError as exc: + pytest.skip(f"当前 Windows 环境不允许创建测试符号链接:{exc}") + + resolved = storage_service.resolve_task_media_file_path( + str(linked_clip), + task_id="task-symlink", + task_dir_name="task-symlink", + allowed_subdirectories=("05_clips",), + ) + + assert resolved is None + + +def test_output_media_route_rejects_database_path_outside_task( + isolated_storage_boundaries, + tmp_path, + monkeypatch, +): + storage_service.create_task_directory("task-route", "task-route") + outside_clip = tmp_path / "sensitive.mp4" + outside_clip.write_bytes(b"private") + monkeypatch.setattr( + media_router.task_service, + "get_task", + lambda *_args, **_kwargs: {"id": "task-route", "task_dir_name": "task-route"}, + ) + monkeypatch.setattr( + media_router.task_service, + "get_output_clip", + lambda *_args, **_kwargs: {"id": "clip-1", "output_file_path": str(outside_clip)}, + ) + + with pytest.raises(HTTPException) as exc_info: + asyncio.run(media_router.get_task_output_clip("task-route", "clip-1")) + + assert exc_info.value.status_code == 404 + + +def test_output_media_route_accepts_current_task_clip(isolated_storage_boundaries, monkeypatch): + task_dir = storage_service.create_task_directory("task-route-ok", "task-route-ok") + clip_path = task_dir / "05_clips" / "clip.mp4" + clip_path.write_bytes(b"clip") + monkeypatch.setattr( + media_router.task_service, + "get_task", + lambda *_args, **_kwargs: {"id": "task-route-ok", "task_dir_name": "task-route-ok"}, + ) + monkeypatch.setattr( + media_router.task_service, + "get_output_clip", + lambda *_args, **_kwargs: {"id": "clip-1", "output_file_path": str(clip_path)}, + ) + + response = asyncio.run(media_router.get_task_output_clip("task-route-ok", "clip-1")) + + assert Path(response.path) == clip_path + + +def test_source_media_route_keeps_allowed_external_source(isolated_storage_boundaries, monkeypatch): + source_path = storage_service.settings.storage_root / "共享原片" / "source.mp4" + source_path.parent.mkdir(parents=True) + source_path.write_bytes(b"source") + monkeypatch.setattr( + media_router.task_service, + "get_task", + lambda *_args, **_kwargs: { + "id": "task-source", + "task_dir_name": "task-source", + "source_type": "upload", + "original_video_path": str(source_path), + }, + ) + + response = asyncio.run(media_router.get_task_source_video("task-source")) + + assert Path(response.path) == source_path + + +def test_subtitled_media_route_rejects_database_path_outside_task( + isolated_storage_boundaries, + tmp_path, + monkeypatch, +): + storage_service.create_task_directory("task-subtitle", "task-subtitle") + outside_clip = tmp_path / "sensitive.mp4" + outside_clip.write_bytes(b"private") + monkeypatch.setattr( + media_router.task_service, + "get_task", + lambda *_args, **_kwargs: {"id": "task-subtitle", "task_dir_name": "task-subtitle"}, + ) + monkeypatch.setattr( + media_router.task_service, + "get_output_clip", + lambda *_args, **_kwargs: { + "id": "clip-1", + "subtitled_output_file_path": str(outside_clip), + }, + ) + + with pytest.raises(HTTPException) as exc_info: + asyncio.run(media_router.get_task_subtitled_clip("task-subtitle", "clip-1")) + + assert exc_info.value.status_code == 404 + + +def test_cover_route_turns_unsafe_persisted_task_directory_into_404(monkeypatch): + monkeypatch.setattr( + media_router.task_service, + "get_task", + lambda *_args, **_kwargs: {"id": "task-cover", "task_dir_name": "..\\outside"}, + ) + + with pytest.raises(HTTPException) as exc_info: + asyncio.run(media_router.get_task_cover("task-cover", "cover.jpg")) + + assert exc_info.value.status_code == 404 + + +def test_cover_route_rejects_symlink_escape(isolated_storage_boundaries, tmp_path, monkeypatch): + task_dir = storage_service.create_task_directory("task-cover-link", "task-cover-link") + outside_cover = tmp_path / "private.jpg" + outside_cover.write_bytes(b"private") + linked_cover = task_dir / "07_covers" / "cover.jpg" + try: + linked_cover.symlink_to(outside_cover) + except OSError as exc: + pytest.skip(f"当前 Windows 环境不允许创建测试符号链接:{exc}") + monkeypatch.setattr( + media_router.task_service, + "get_task", + lambda *_args, **_kwargs: { + "id": "task-cover-link", + "task_dir_name": "task-cover-link", + }, + ) + + with pytest.raises(HTTPException) as exc_info: + asyncio.run(media_router.get_task_cover("task-cover-link", "cover.jpg")) + + assert exc_info.value.status_code == 404 diff --git a/tests/test_subtitle_auto_workflow.py b/tests/test_subtitle_auto_workflow.py new file mode 100644 index 0000000..0e1e8b8 --- /dev/null +++ b/tests/test_subtitle_auto_workflow.py @@ -0,0 +1,1127 @@ +from __future__ import annotations + +import hashlib +import json +import sqlite3 +import shutil +import subprocess +from pathlib import Path +from unittest.mock import Mock +from uuid import uuid4 + +import pytest +from fastapi.testclient import TestClient + +from app.core.config import settings +from app.db.database import get_connection, init_db +from app.main import app +from app.models.task import TaskStatus +from app.services import job_service +from app.services.auto_publish_service import create_auto_publish_jobs +from app.services.pipeline_engine import PipelineEngine +from app.services.publish_readiness import build_send_readiness +from app.services.publish_service import sync_task_publish_jobs +from app.services.subtitle_ai_service import generate_subtitle_suggestions +from app.services.subtitle_auto_workflow_service import ( + cleanup_interrupted_subtitle_job, + enqueue_task_subtitle_render, + execute_subtitle_render_job, + prepare_task_subtitle_review, + reconcile_interrupted_subtitle_job, + skip_task_subtitles_and_resume, + skip_task_subtitles_to_review, +) +from app.services.subtitle_data_service import ( + accept_suggestion_revision, + approve_revision, + create_manual_revision, + ensure_clip_track, + get_revision, + get_track, +) +from app.services.subtitle_workflow_service import ( + _create_subtitle_job, + _build_ffmpeg_render_command, + _finalize_subtitle_job, + _probe_media, + _render_with_fallback, + _validate_rendered_media, + render_subtitles_for_output_clip, +) +from app.services.storage_service import get_artifact_paths +from app.services.task_service import get_task + + +PREFIX = "test-subtitle-auto-" + + +def _headers() -> dict[str, str]: + if settings.local_admin_token: + return {"Authorization": f"Bearer {settings.local_admin_token}"} + return {} + + +@pytest.fixture(autouse=True) +def subtitle_auto_database(): + init_db() + _cleanup() + yield + _cleanup() + + +def _cleanup() -> None: + with get_connection() as connection: + publish_rows = connection.execute( + "SELECT id FROM publish_jobs WHERE task_id LIKE ?", (f"{PREFIX}%",) + ).fetchall() + if publish_rows: + placeholders = ",".join("?" for _ in publish_rows) + connection.execute( + f"DELETE FROM publish_job_events WHERE job_id IN ({placeholders})", + [row["id"] for row in publish_rows], + ) + connection.execute("DELETE FROM publish_jobs WHERE task_id LIKE ?", (f"{PREFIX}%",)) + connection.execute("DELETE FROM subtitle_jobs WHERE task_id LIKE ?", (f"{PREFIX}%",)) + connection.execute("DELETE FROM workflow_jobs WHERE task_id LIKE ?", (f"{PREFIX}%",)) + connection.execute( + "DELETE FROM subtitle_cues WHERE revision_id IN (SELECT id FROM subtitle_revisions WHERE track_id IN (SELECT id FROM subtitle_tracks WHERE task_id LIKE ?))", + (f"{PREFIX}%",), + ) + connection.execute( + "DELETE FROM subtitle_revisions WHERE track_id IN (SELECT id FROM subtitle_tracks WHERE task_id LIKE ?)", + (f"{PREFIX}%",), + ) + connection.execute("DELETE FROM subtitle_tracks WHERE task_id LIKE ?", (f"{PREFIX}%",)) + connection.execute("DELETE FROM output_clip WHERE task_id LIKE ?", (f"{PREFIX}%",)) + connection.execute("DELETE FROM transcription_chunks WHERE task_id LIKE ?", (f"{PREFIX}%",)) + connection.execute("DELETE FROM transcription_runs WHERE task_id LIKE ?", (f"{PREFIX}%",)) + connection.execute("DELETE FROM tasks WHERE id LIKE ?", (f"{PREFIX}%",)) + connection.commit() + + +def _create_task(tmp_path: Path, *, status: str = "VIDEO_CUTTING") -> tuple[str, str, Path]: + task_id = f"{PREFIX}{uuid4().hex[:8]}" + output_id = f"out-{uuid4().hex[:8]}" + source_path = tmp_path / f"{task_id}-source.mp4" + output_path = tmp_path / f"{output_id}.mp4" + source_path.write_bytes(b"source") + output_path.write_bytes(b"clip") + now = "2026-08-24T00:00:00+00:00" + segments = [ + {"start_seconds": 0.1, "end_seconds": 1.2, "text": "大家好", "confidence": 0.95}, + {"start_seconds": 1.3, "end_seconds": 2.8, "text": "这是直播高光", "confidence": 0.93}, + ] + result_json = json.dumps(segments, ensure_ascii=False) + result_checksum = hashlib.sha256(result_json.encode("utf-8")).hexdigest() + run_id = f"run-{uuid4().hex[:8]}" + with get_connection() as connection: + connection.execute( + """ + INSERT INTO tasks ( + id, task_name, source_type, platform, original_video_path, + selection_profile, auto_mode, auto_config_json, status, + created_at, updated_at + ) VALUES (?, '字幕自动流程测试', 'upload', 'general', ?, 'general', 1, '{}', ?, ?, ?) + """, + (task_id, str(source_path), status, now, now), + ) + connection.execute( + """ + INSERT INTO output_clip ( + id, task_id, output_file_path, output_file_name, status, is_active, + source_start_ms, source_end_ms, source_duration_ms, + source_fingerprint, snapshot_source, created_at, updated_at + ) VALUES (?, ?, ?, 'clip.mp4', 'completed', 1, 0, 5000, 5000, + 'source-v1', 'cut_commit', ?, ?) + """, + (output_id, task_id, str(output_path), now, now), + ) + connection.execute( + """ + INSERT INTO transcription_runs ( + id, task_id, source_fingerprint, provider, model, device, compute_type, + chunk_seconds, overlap_seconds, status, + total_chunks, completed_chunks, is_active, created_at, updated_at, completed_at + ) VALUES (?, ?, 'source-v1', 'local', 'small', 'cpu', 'int8', + 120, 5, 'completed', 1, 1, 1, ?, ?, ?) + """, + (run_id, task_id, now, now, now), + ) + connection.execute( + """ + INSERT INTO transcription_chunks ( + id, run_id, task_id, chunk_index, start_ms, end_ms, status, + result_json, result_checksum, created_at, updated_at + ) VALUES (?, ?, ?, 1, 0, 5000, 'completed', ?, ?, ?, ?) + """, + (f"chunk-{uuid4().hex[:8]}", run_id, task_id, result_json, result_checksum, now, now), + ) + connection.commit() + return task_id, output_id, output_path + + +def _add_output_clip(task_id: str, tmp_path: Path, *, file_name: str = "z-clip.mp4") -> tuple[str, Path]: + output_id = f"out-{uuid4().hex[:8]}" + output_path = tmp_path / f"{output_id}.mp4" + output_path.write_bytes(b"clip") + now = "2026-08-24T00:00:01+00:00" + with get_connection() as connection: + connection.execute( + """ + INSERT INTO output_clip ( + id, task_id, output_file_path, output_file_name, status, is_active, + source_start_ms, source_end_ms, source_duration_ms, + source_fingerprint, snapshot_source, created_at, updated_at + ) VALUES (?, ?, ?, ?, 'completed', 1, 0, 5000, 5000, + 'source-v1', 'cut_commit', ?, ?) + """, + (output_id, task_id, str(output_path), file_name, now, now), + ) + connection.commit() + return output_id, output_path + + +def test_schema_contains_async_render_validation_fields(): + with get_connection() as connection: + columns = {row[1] for row in connection.execute("PRAGMA table_info(subtitle_jobs)")} + assert { + "workflow_job_id", + "revision_id", + "validation_status", + "validation_json", + "encoder", + "verified_at", + } <= columns + + +@pytest.mark.parametrize("width,height", [(360, 640), (640, 360), (480, 480)]) +def test_real_ffmpeg_render_supports_three_aspect_ratios(tmp_path: Path, width: int, height: int): + if not shutil.which("ffmpeg") or not shutil.which("ffprobe"): + pytest.skip("本机没有 FFmpeg/FFprobe") + task_id, output_id, output_path = _create_task(tmp_path) + try: + subprocess.run( + [ + "ffmpeg", + "-y", + "-hide_banner", + "-loglevel", + "error", + "-f", + "lavfi", + "-i", + f"color=c=navy:s={width}x{height}:r=25:d=3", + "-f", + "lavfi", + "-i", + "sine=frequency=440:sample_rate=44100:duration=3", + "-shortest", + "-c:v", + "libx264", + "-pix_fmt", + "yuv420p", + "-c:a", + "aac", + str(output_path), + ], + capture_output=True, + text=True, + timeout=60, + check=True, + ) + prepare_task_subtitle_review(task_id) + track = ensure_clip_track(task_id, output_id) + approve_revision(track["id"], track["active_revision_id"]) + + result = render_subtitles_for_output_clip(task_id, output_id) + rendered_path = Path(result["job"]["output_file_path"]) + probe = _probe_media(rendered_path) + + assert result["job"]["validation_status"] == "verified" + assert result["job"]["revision_id"] == track["active_revision_id"] + assert rendered_path.is_file() + assert probe["video_codec"] == "h264" + assert probe["pixel_format"] == "yuv420p" + assert probe["has_audio"] is True + assert probe["duration"] == pytest.approx(3.0, abs=1.0) + finally: + shutil.rmtree(get_artifact_paths(task_id)["task_dir"], ignore_errors=True) + + +def test_prepare_review_creates_clip_draft_and_pauses(tmp_path: Path): + task_id, output_id, _ = _create_task(tmp_path) + result = prepare_task_subtitle_review(task_id) + task = get_task(task_id, include_video_probe=False) + track = ensure_clip_track(task_id, output_id) + revision = get_revision(track["active_revision_id"], include_cues=True) + assert result["status"] == "pending_subtitle_review" + assert task["status"] == TaskStatus.PENDING_SUBTITLE_REVIEW.value + assert revision["status"] == "draft" + assert revision["cue_count"] == 2 + + init_db() + assert get_task(task_id, include_video_probe=False)["status"] == TaskStatus.PENDING_SUBTITLE_REVIEW.value + + +def test_prepare_review_cannot_reopen_terminal_task(tmp_path: Path): + task_id, _, _ = _create_task(tmp_path, status=TaskStatus.READY_TO_PUBLISH.value) + + with pytest.raises(ValueError, match="不能进入字幕审核"): + prepare_task_subtitle_review(task_id) + + assert get_task(task_id, include_video_probe=False)["status"] == TaskStatus.READY_TO_PUBLISH.value + + +def test_review_page_and_api_enqueue_batch_job(tmp_path: Path): + task_id, _, _ = _create_task(tmp_path) + prepare_task_subtitle_review(task_id) + with TestClient(app) as client: + page = client.get(f"/subtitles/{task_id}", headers=_headers()) + task_page = client.get(f"/tasks/{task_id}", headers=_headers()) + live = client.get(f"/api/tasks/{task_id}/live-status", headers=_headers()) + queued = client.post( + f"/api/subtitles/tasks/{task_id}/approve-and-render", + json={"approve_active_revisions": True}, + headers=_headers(), + ) + jobs = client.get(f"/api/subtitles/tasks/{task_id}/jobs", headers=_headers()) + + assert page.status_code == 200 + assert 'id="subtitle-review-gate"' in page.text + assert 'id="subtitle-batch-approve-render"' in page.text + assert 'id="subtitle-skip-and-resume"' in page.text + assert 'id="subtitle-ai-suggest"' in page.text + assert 'data-sync-publish-task' not in page.text + assert task_page.status_code == 200 + assert f'href="/subtitles/{task_id}"' in task_page.text + assert live.status_code == 200 + assert live.json()["actions"]["primary"] == "subtitle_review" + assert live.json()["should_poll"] is False + assert queued.status_code == 200 + assert queued.json()["job"]["job_type"] == job_service.JOB_TYPE_SUBTITLE + assert jobs.status_code == 200 + assert jobs.json()["count"] == 1 + + +def test_skip_api_persists_original_delivery_decision(tmp_path: Path): + task_id, _, _ = _create_task(tmp_path) + prepare_task_subtitle_review(task_id) + with TestClient(app) as client: + response = client.post( + f"/api/subtitles/tasks/{task_id}/skip-to-review", + headers=_headers(), + ) + assert response.status_code == 200, response.text + assert response.json()["status"] == TaskStatus.pending_review.value + assert response.json()["review_url"] == f"/tasks/{task_id}/clips/review" + with get_connection() as connection: + raw_config = connection.execute( + "SELECT auto_config_json FROM tasks WHERE id = ?", + (task_id,), + ).fetchone()[0] + active_resume_jobs = connection.execute( + """ + SELECT COUNT(*) FROM workflow_jobs + WHERE task_id = ? AND job_type = ? AND status IN ('queued', 'running') + """, + (task_id, job_service.JOB_TYPE_AUTO_PIPELINE), + ).fetchone()[0] + assert json.loads(raw_config)["subtitle_delivery_mode"] == "original" + assert get_task(task_id, include_video_probe=False)["status"] == TaskStatus.pending_review.value + assert active_resume_jobs == 0 + + +def test_pipeline_stops_before_metadata_and_publish(tmp_path: Path, monkeypatch: pytest.MonkeyPatch): + task_id, _, _ = _create_task(tmp_path, status="CREATED") + engine = PipelineEngine() + for name in ("_prepare_source", "_transcribe_or_read_text", "_run_ai_analysis", "_select_clips", "_cut_video"): + monkeypatch.setattr(engine, name, Mock(return_value={"ok": True})) + + def pause(task_id_value: str, _context: dict) -> dict: + from app.services import task_service + + task_service.update_task_status(task_id_value, TaskStatus.PENDING_SUBTITLE_REVIEW) + return {"status": "pending_subtitle_review"} + + monkeypatch.setattr(engine, "_prepare_subtitle_drafts", pause) + metadata = Mock(return_value={}) + monkeypatch.setattr(engine, "_generate_metadata", metadata) + result = engine.run(task_id) + assert result["status"] == "pending_subtitle_review" + metadata.assert_not_called() + with get_connection() as connection: + assert connection.execute("SELECT COUNT(*) FROM publish_jobs WHERE task_id = ?", (task_id,)).fetchone()[0] == 0 + + +def test_batch_approval_pins_revisions_and_records_delivery_mode(tmp_path: Path): + task_id, output_id, _ = _create_task(tmp_path) + prepare_task_subtitle_review(task_id) + result = enqueue_task_subtitle_render( + task_id, + approve_active_revisions=True, + continue_pipeline=True, + ) + item = result["job"]["payload_json"]["items"][0] + revision = get_revision(item["revision_id"]) + with get_connection() as connection: + config = json.loads(connection.execute("SELECT auto_config_json FROM tasks WHERE id = ?", (task_id,)).fetchone()[0]) + assert item["output_clip_id"] == output_id + assert revision["status"] == "approved" + assert config["subtitle_delivery_mode"] == "subtitled" + + +def test_batch_approval_rolls_back_when_later_revision_is_invalid(tmp_path: Path): + task_id, first_output_id, _ = _create_task(tmp_path) + prepare_task_subtitle_review(task_id) + second_output_id, _ = _add_output_clip(task_id, tmp_path) + first_track = ensure_clip_track(task_id, first_output_id) + second_track = ensure_clip_track(task_id, second_output_id) + with get_connection() as connection: + second_cues = connection.execute( + "SELECT id FROM subtitle_cues WHERE revision_id = ? ORDER BY cue_index ASC", + (second_track["active_revision_id"],), + ).fetchall() + connection.execute( + "UPDATE subtitle_cues SET start_ms = 1000 WHERE id = ?", + (second_cues[1]["id"],), + ) + connection.commit() + + with pytest.raises(ValueError, match="时间重叠"): + enqueue_task_subtitle_render(task_id, approve_active_revisions=True, continue_pipeline=True) + + with get_connection() as connection: + statuses = { + row["id"]: row["status"] + for row in connection.execute( + "SELECT id, status FROM subtitle_revisions WHERE id IN (?, ?)", + (first_track["active_revision_id"], second_track["active_revision_id"]), + ).fetchall() + } + job_count = connection.execute( + "SELECT COUNT(*) FROM workflow_jobs WHERE task_id = ? AND job_type = ?", + (task_id, job_service.JOB_TYPE_SUBTITLE), + ).fetchone()[0] + assert statuses[first_track["active_revision_id"]] == "draft" + assert statuses[second_track["active_revision_id"]] == "draft" + assert job_count == 0 + + +def test_job_insert_failure_rolls_back_approval_and_delivery_mode(tmp_path: Path): + task_id, output_id, _ = _create_task(tmp_path) + prepare_task_subtitle_review(task_id) + track = ensure_clip_track(task_id, output_id) + with get_connection() as connection: + connection.execute( + """ + CREATE TRIGGER fail_subtitle_workflow_job_insert + BEFORE INSERT ON workflow_jobs + WHEN NEW.job_type = 'subtitle' + BEGIN + SELECT RAISE(ABORT, 'forced subtitle job insert failure'); + END + """ + ) + connection.commit() + try: + with pytest.raises(sqlite3.IntegrityError, match="forced subtitle job insert failure"): + enqueue_task_subtitle_render(task_id, approve_active_revisions=True, continue_pipeline=True) + finally: + with get_connection() as connection: + connection.execute("DROP TRIGGER IF EXISTS fail_subtitle_workflow_job_insert") + connection.commit() + + with get_connection() as connection: + revision_status = connection.execute( + "SELECT status FROM subtitle_revisions WHERE id = ?", + (track["active_revision_id"],), + ).fetchone()[0] + config = json.loads( + connection.execute("SELECT auto_config_json FROM tasks WHERE id = ?", (task_id,)).fetchone()[0] + ) + job_count = connection.execute( + "SELECT COUNT(*) FROM workflow_jobs WHERE task_id = ? AND job_type = ?", + (task_id, job_service.JOB_TYPE_SUBTITLE), + ).fetchone()[0] + assert revision_status == "draft" + assert "subtitle_delivery_mode" not in config + assert job_count == 0 + + +def test_corrupt_delivery_config_fails_closed_without_partial_approval(tmp_path: Path): + task_id, output_id, _ = _create_task(tmp_path) + prepare_task_subtitle_review(task_id) + track = ensure_clip_track(task_id, output_id) + with get_connection() as connection: + connection.execute("UPDATE tasks SET auto_config_json = '{broken' WHERE id = ?", (task_id,)) + connection.commit() + + with pytest.raises(ValueError, match="配置已损坏"): + enqueue_task_subtitle_render(task_id, approve_active_revisions=True, continue_pipeline=True) + + with get_connection() as connection: + assert connection.execute( + "SELECT status FROM subtitle_revisions WHERE id = ?", (track["active_revision_id"],) + ).fetchone()[0] == "draft" + assert connection.execute( + "SELECT auto_config_json FROM tasks WHERE id = ?", (task_id,) + ).fetchone()[0] == "{broken" + assert connection.execute( + "SELECT COUNT(*) FROM workflow_jobs WHERE task_id = ? AND job_type = ?", + (task_id, job_service.JOB_TYPE_SUBTITLE), + ).fetchone()[0] == 0 + + +def test_skip_is_explicit_and_enters_clip_review_without_resume_job(tmp_path: Path): + task_id, _, _ = _create_task(tmp_path) + prepare_task_subtitle_review(task_id) + result = skip_task_subtitles_to_review(task_id) + with get_connection() as connection: + config = json.loads(connection.execute("SELECT auto_config_json FROM tasks WHERE id = ?", (task_id,)).fetchone()[0]) + resume_jobs = connection.execute( + "SELECT COUNT(*) FROM workflow_jobs WHERE task_id = ? AND job_type = ?", + (task_id, job_service.JOB_TYPE_AUTO_PIPELINE), + ).fetchone()[0] + assert config["subtitle_delivery_mode"] == "original" + assert result["status"] == TaskStatus.pending_review.value + assert result["review_url"] == f"/tasks/{task_id}/clips/review" + assert resume_jobs == 0 + + +def test_legacy_skip_endpoint_uses_new_review_semantics(tmp_path: Path): + task_id, _, _ = _create_task(tmp_path) + prepare_task_subtitle_review(task_id) + result = skip_task_subtitles_and_resume(task_id) + assert result["status"] == TaskStatus.pending_review.value + assert result["review_url"].endswith("/clips/review") + + +def test_skip_with_corrupt_config_keeps_review_gate_and_original_payload(tmp_path: Path): + task_id, _, _ = _create_task(tmp_path) + prepare_task_subtitle_review(task_id) + with get_connection() as connection: + connection.execute("UPDATE tasks SET auto_config_json = '[broken' WHERE id = ?", (task_id,)) + connection.commit() + with pytest.raises(ValueError, match="配置已损坏"): + skip_task_subtitles_to_review(task_id) + with get_connection() as connection: + row = connection.execute( + "SELECT status, auto_config_json FROM tasks WHERE id = ?", (task_id,) + ).fetchone() + assert row["status"] == TaskStatus.PENDING_SUBTITLE_REVIEW.value + assert row["auto_config_json"] == "[broken" + + +def test_skip_rejects_active_subtitle_job(tmp_path: Path): + task_id, _, _ = _create_task(tmp_path) + prepare_task_subtitle_review(task_id) + enqueue_task_subtitle_render(task_id, approve_active_revisions=True, continue_pipeline=True) + with pytest.raises(ValueError, match="仍在运行"): + skip_task_subtitles_and_resume(task_id) + + +def test_skip_rejects_queued_pipeline_resume_after_subtitle_completion(tmp_path: Path): + task_id, _, _ = _create_task(tmp_path) + prepare_task_subtitle_review(task_id) + job_service.create_job( + task_id, + job_service.JOB_TYPE_AUTO_PIPELINE, + {"retry": False, "start_step": TaskStatus.METADATA_GENERATING.value}, + ) + + with pytest.raises(ValueError, match="后续自动流水线仍在运行"): + skip_task_subtitles_to_review(task_id) + + with get_connection() as connection: + row = connection.execute( + "SELECT status, auto_config_json FROM tasks WHERE id = ?", + (task_id,), + ).fetchone() + assert row["status"] == TaskStatus.PENDING_SUBTITLE_REVIEW.value + assert json.loads(row["auto_config_json"] or "{}").get("subtitle_delivery_mode") != "original" + + +def test_render_enqueue_rejects_existing_pipeline_resume(tmp_path: Path): + task_id, _, _ = _create_task(tmp_path) + prepare_task_subtitle_review(task_id) + job_service.create_job( + task_id, + job_service.JOB_TYPE_AUTO_PIPELINE, + {"retry": False, "start_step": TaskStatus.METADATA_GENERATING.value}, + ) + + with pytest.raises(ValueError, match="不能重复执行字幕烧录"): + enqueue_task_subtitle_render( + task_id, + approve_active_revisions=True, + continue_pipeline=True, + ) + + with get_connection() as connection: + subtitle_jobs = connection.execute( + "SELECT COUNT(*) FROM workflow_jobs WHERE task_id = ? AND job_type = ?", + (task_id, job_service.JOB_TYPE_SUBTITLE), + ).fetchone()[0] + revision_statuses = { + row[0] + for row in connection.execute( + """ + SELECT sr.status FROM subtitle_revisions sr + JOIN subtitle_tracks st ON st.id = sr.track_id + WHERE st.task_id = ? AND st.active_revision_id = sr.id + """, + (task_id,), + ).fetchall() + } + assert subtitle_jobs == 0 + assert revision_statuses == {"draft"} + + +def test_pending_subtitle_review_cannot_sync_publish_center(tmp_path: Path): + task_id, _, _ = _create_task(tmp_path) + prepare_task_subtitle_review(task_id) + with pytest.raises(ValueError, match="等待字幕审核"): + sync_task_publish_jobs(task_id) + + +def test_batch_execution_checkpoints_and_completes_with_resume_atomically( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +): + task_id, output_id, _ = _create_task(tmp_path) + prepare_task_subtitle_review(task_id) + queued = enqueue_task_subtitle_render(task_id, approve_active_revisions=True, continue_pipeline=True) + renderer = Mock( + return_value={ + "job": { + "id": "subtitle-render-1", + "output_file_path": str(tmp_path / "verified.mp4"), + } + } + ) + monkeypatch.setattr( + "app.services.subtitle_auto_workflow_service.render_subtitles_for_output_clip", + renderer, + ) + claimed = job_service.claim_job(queued["job_id"], "subtitle-test-worker") + with job_service.job_lease_context(queued["job_id"], "subtitle-test-worker", claimed["lease_token"]): + result = execute_subtitle_render_job(queued["job_id"], task_id, queued["job"]["payload_json"]) + completed_job, resume_job, created = job_service.mark_job_completed_with_followup( + queued["job_id"], + result, + followup_task_id=task_id, + followup_job_type=job_service.JOB_TYPE_AUTO_PIPELINE, + followup_payload={"retry": False, "start_step": TaskStatus.METADATA_GENERATING.value}, + result_followup_key="resume_job_id", + ) + checkpoint = job_service.get_job(queued["job_id"])["checkpoint_json"] + assert result["completed_count"] == 1 + assert checkpoint["completed"][output_id]["revision_id"] + assert result["resume_requested"] is True + assert result["resume_job_id"] == "" + assert created is True + assert completed_job["status"] == job_service.JOB_STATUS_COMPLETED + assert completed_job["result_json"]["resume_job_id"] == resume_job["id"] + assert resume_job["job_type"] == job_service.JOB_TYPE_AUTO_PIPELINE + assert resume_job["payload_json"]["start_step"] == TaskStatus.METADATA_GENERATING.value + + +def test_cancelled_subtitle_job_cannot_create_resume_job(tmp_path: Path): + task_id, _, _ = _create_task(tmp_path) + prepare_task_subtitle_review(task_id) + queued = enqueue_task_subtitle_render(task_id, approve_active_revisions=True, continue_pipeline=True) + claimed = job_service.claim_job(queued["job_id"], "subtitle-cancelled-worker") + + job_service.request_job_cancel(queued["job_id"]) + with job_service.job_lease_context( + queued["job_id"], + "subtitle-cancelled-worker", + claimed["lease_token"], + ): + with pytest.raises(job_service.JobLeaseLostError, match="租约已失效或已取消"): + job_service.mark_job_completed_with_followup( + queued["job_id"], + {"resume_requested": True}, + followup_task_id=task_id, + followup_job_type=job_service.JOB_TYPE_AUTO_PIPELINE, + followup_payload={"retry": False, "start_step": TaskStatus.METADATA_GENERATING.value}, + result_followup_key="resume_job_id", + ) + + with get_connection() as connection: + resume_count = connection.execute( + "SELECT COUNT(*) FROM workflow_jobs WHERE task_id = ? AND job_type = ?", + (task_id, job_service.JOB_TYPE_AUTO_PIPELINE), + ).fetchone()[0] + assert resume_count == 0 + assert job_service.get_job(queued["job_id"])["cancel_requested"] == 1 + + +def test_incompatible_existing_resume_job_rolls_back_subtitle_completion(tmp_path: Path): + task_id, _, _ = _create_task(tmp_path) + prepare_task_subtitle_review(task_id) + queued = enqueue_task_subtitle_render(task_id, approve_active_revisions=True, continue_pipeline=True) + existing_resume = job_service.create_job( + task_id, + job_service.JOB_TYPE_AUTO_PIPELINE, + {"retry": True, "start_step": TaskStatus.AI_ANALYZING.value}, + ) + claimed = job_service.claim_job(queued["job_id"], "subtitle-payload-worker") + + with job_service.job_lease_context( + queued["job_id"], + "subtitle-payload-worker", + claimed["lease_token"], + ): + with pytest.raises(ValueError, match="执行参数不同"): + job_service.mark_job_completed_with_followup( + queued["job_id"], + {"resume_requested": True}, + followup_task_id=task_id, + followup_job_type=job_service.JOB_TYPE_AUTO_PIPELINE, + followup_payload={"retry": False, "start_step": TaskStatus.METADATA_GENERATING.value}, + result_followup_key="resume_job_id", + ) + + current = job_service.get_job(queued["job_id"]) + assert current["status"] == job_service.JOB_STATUS_RUNNING + assert current["result_json"] == {} + assert job_service.get_job(existing_resume["id"])["payload_json"] == { + "retry": True, + "start_step": TaskStatus.AI_ANALYZING.value, + } + + +def test_restart_recovers_completed_active_result_before_checkpoint(tmp_path: Path, monkeypatch: pytest.MonkeyPatch): + task_id, output_id, _ = _create_task(tmp_path) + prepare_task_subtitle_review(task_id) + queued = enqueue_task_subtitle_render(task_id, approve_active_revisions=True, continue_pipeline=True) + workflow_job_id = queued["job_id"] + revision_id = queued["job"]["payload_json"]["items"][0]["revision_id"] + rendered_path = tmp_path / "already-committed.mp4" + rendered_path.write_bytes(b"verified") + child = _create_subtitle_job( + task_id, + output_id, + "completed", + output_file_path=str(rendered_path), + revision_id=revision_id, + workflow_job_id=workflow_job_id, + is_active=1, + ) + with get_connection() as connection: + connection.execute( + "UPDATE subtitle_jobs SET validation_status = 'verified', verified_at = updated_at WHERE id = ?", + (child["id"],), + ) + connection.commit() + renderer = Mock(side_effect=AssertionError("已提交结果不得重复渲染")) + monkeypatch.setattr( + "app.services.subtitle_auto_workflow_service.render_subtitles_for_output_clip", + renderer, + ) + claimed = job_service.claim_job(workflow_job_id, "subtitle-recovery-worker") + with job_service.job_lease_context(workflow_job_id, claimed["lease_owner"], claimed["lease_token"]): + result = execute_subtitle_render_job(workflow_job_id, task_id, queued["job"]["payload_json"]) + renderer.assert_not_called() + assert result["completed"][output_id]["subtitle_job_id"] == child["id"] + assert job_service.get_job(workflow_job_id)["checkpoint_json"]["completed"][output_id]["subtitle_job_id"] == child["id"] + + +def test_cancel_cleanup_is_precise_and_retry_keeps_checkpoint( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +): + task_id, output_id, _ = _create_task(tmp_path) + prepare_task_subtitle_review(task_id) + queued = enqueue_task_subtitle_render(task_id, approve_active_revisions=True, continue_pipeline=True) + workflow_job_id = queued["job_id"] + revision_id = queued["job"]["payload_json"]["items"][0]["revision_id"] + child = _create_subtitle_job( + task_id, + output_id, + "processing", + revision_id=revision_id, + workflow_job_id=workflow_job_id, + is_active=0, + ) + directory = tmp_path / "subtitled" + directory.mkdir(parents=True, exist_ok=True) + monkeypatch.setattr( + "app.services.subtitle_auto_workflow_service.get_artifact_paths", + lambda _task_id: {"subtitled_dir": directory}, + ) + owned_temp = directory / f".clip.{workflow_job_id}.part.mp4" + unrelated_temp = directory / ".clip.other-job.part.mp4" + owned_temp.write_bytes(b"partial") + unrelated_temp.write_bytes(b"keep") + + stale_claim = job_service.claim_job(workflow_job_id, "cleanup-test-worker-old") + with get_connection() as connection: + connection.execute( + "UPDATE workflow_jobs SET lease_expires_at = '2000-01-01T00:00:00+00:00' WHERE id = ?", + (workflow_job_id,), + ) + connection.commit() + claimed = job_service.claim_job(workflow_job_id, "cleanup-test-worker") + assert cleanup_interrupted_subtitle_job( + workflow_job_id, + lease_owner="cleanup-test-worker-old", + lease_token=stale_claim["lease_token"], + status="cancelled", + message="旧 Worker 不得清理", + ) is False + assert owned_temp.exists() is True + assert cleanup_interrupted_subtitle_job( + workflow_job_id, + lease_owner="cleanup-test-worker", + lease_token=claimed["lease_token"], + status="cancelled", + message="用户已取消字幕烧录", + ) is True + with get_connection() as connection: + child_status = connection.execute( + "SELECT status FROM subtitle_jobs WHERE id = ?", + (child["id"],), + ).fetchone()[0] + assert child_status == "cancelled" + assert owned_temp.exists() is False + assert unrelated_temp.exists() is True + + checkpoint = {"completed": {output_id: {"revision_id": revision_id}}} + with job_service.job_lease_context(workflow_job_id, claimed["lease_owner"], claimed["lease_token"]): + job_service.update_job_checkpoint(workflow_job_id, checkpoint) + job_service.mark_job_cancelled(workflow_job_id) + retried = job_service.retry_job(workflow_job_id) + assert retried["status"] == job_service.JOB_STATUS_QUEUED + assert retried["checkpoint_json"] == checkpoint + + +def test_takeover_rejects_stale_finalize_and_reconciles_owned_temp( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +): + task_id, output_id, _ = _create_task(tmp_path) + prepare_task_subtitle_review(task_id) + track = ensure_clip_track(task_id, output_id) + approve_revision(track["id"], track["active_revision_id"]) + old_final = tmp_path / "old-active.mp4" + old_final.write_bytes(b"old") + old_active = _create_subtitle_job( + task_id, + output_id, + "completed", + output_file_path=str(old_final), + revision_id=track["active_revision_id"], + is_active=1, + ) + queued = enqueue_task_subtitle_render(task_id, continue_pipeline=True) + workflow_job_id = queued["job_id"] + stale_claim = job_service.claim_job(workflow_job_id, "subtitle-stale-worker") + child = _create_subtitle_job( + task_id, + output_id, + "processing", + revision_id=track["active_revision_id"], + workflow_job_id=workflow_job_id, + is_active=0, + ) + managed_dir = tmp_path / "managed-subtitled" + managed_dir.mkdir() + temporary_path = managed_dir / f".new-output.{workflow_job_id}.part.mp4" + temporary_path.write_bytes(b"new") + workflow_marker = hashlib.sha256(workflow_job_id.encode("utf-8")).hexdigest()[:12] + orphan_final = managed_dir / f"clip_subtitled_{workflow_marker}_orphan.mp4" + orphan_final.write_bytes(b"orphan") + referenced_final = managed_dir / f"clip_subtitled_{workflow_marker}_referenced.mp4" + referenced_final.write_bytes(b"referenced") + with get_connection() as connection: + connection.execute( + "UPDATE subtitle_jobs SET output_file_path = ? WHERE id = ?", + (str(referenced_final), old_active["id"]), + ) + connection.commit() + final_path = managed_dir / "new-output.mp4" + subtitle_path = managed_dir / "revision.ass" + subtitle_path.write_text("subtitle", encoding="utf-8") + monkeypatch.setattr( + "app.services.subtitle_auto_workflow_service.get_artifact_paths", + lambda _task_id: {"subtitled_dir": managed_dir}, + ) + with get_connection() as connection: + connection.execute( + "UPDATE workflow_jobs SET lease_expires_at = '2000-01-01T00:00:00+00:00' WHERE id = ?", + (workflow_job_id,), + ) + connection.commit() + current_claim = job_service.claim_job(workflow_job_id, "subtitle-current-worker") + + with job_service.job_lease_context( + workflow_job_id, + stale_claim["lease_owner"], + stale_claim["lease_token"], + ): + with pytest.raises(job_service.JobLeaseLostError, match="租约已失效"): + _finalize_subtitle_job( + task_id=task_id, + output_clip_id=output_id, + revision_id=track["active_revision_id"], + subtitle_job_id=child["id"], + workflow_job_id=workflow_job_id, + subtitle_path=subtitle_path, + temporary_path=temporary_path, + output_path=final_path, + validation={"duration": 3.0}, + encoder="libx264", + audio_mode="aac", + ) + assert final_path.exists() is False + assert temporary_path.exists() is True + with get_connection() as connection: + assert connection.execute( + "SELECT is_active FROM subtitle_jobs WHERE id = ?", (old_active["id"],) + ).fetchone()[0] == 1 + assert connection.execute( + "SELECT status FROM subtitle_jobs WHERE id = ?", (child["id"],) + ).fetchone()[0] == "processing" + connection.execute( + "UPDATE subtitle_jobs SET status = 'queued' WHERE id = ?", + (child["id"],), + ) + connection.commit() + + with job_service.job_lease_context( + workflow_job_id, + current_claim["lease_owner"], + current_claim["lease_token"], + ): + recovery = reconcile_interrupted_subtitle_job(workflow_job_id) + assert recovery["interrupted_count"] == 1 + assert recovery["deleted_artifact_count"] == 2 + assert temporary_path.exists() is False + assert orphan_final.exists() is False + assert referenced_final.read_bytes() == b"referenced" + assert old_final.read_bytes() == b"old" + with get_connection() as connection: + assert connection.execute( + "SELECT status FROM subtitle_jobs WHERE id = ?", (child["id"],) + ).fetchone()[0] == "failed" + + +def test_old_revision_cannot_activate_after_editor_moves_forward(tmp_path: Path): + task_id, output_id, _ = _create_task(tmp_path) + prepare_task_subtitle_review(task_id) + track = ensure_clip_track(task_id, output_id) + approved = approve_revision(track["id"], track["active_revision_id"]) + old_final = tmp_path / "current-active.mp4" + old_final.write_bytes(b"current") + current_active = _create_subtitle_job( + task_id, + output_id, + "completed", + output_file_path=str(old_final), + revision_id=approved["id"], + is_active=1, + ) + child = _create_subtitle_job( + task_id, + output_id, + "processing", + revision_id=approved["id"], + is_active=0, + ) + newer = create_manual_revision( + track["id"], + base_revision_id=approved["id"], + cues=approved["cues"], + note="渲染期间的新编辑", + ) + temporary_path = tmp_path / ".old-revision.part.mp4" + temporary_path.write_bytes(b"old-revision-render") + final_path = tmp_path / "old-revision-final.mp4" + subtitle_path = tmp_path / "old-revision.ass" + subtitle_path.write_text("subtitle", encoding="utf-8") + + with pytest.raises(ValueError, match="revision 已变化"): + _finalize_subtitle_job( + task_id=task_id, + output_clip_id=output_id, + revision_id=approved["id"], + subtitle_job_id=child["id"], + workflow_job_id=None, + subtitle_path=subtitle_path, + temporary_path=temporary_path, + output_path=final_path, + validation={"duration": 3.0}, + encoder="libx264", + audio_mode="aac", + ) + assert final_path.exists() is False + assert get_track(track["id"])["active_revision_id"] == newer["id"] + with get_connection() as connection: + assert connection.execute( + "SELECT is_active FROM subtitle_jobs WHERE id = ?", (current_active["id"],) + ).fetchone()[0] == 1 + assert connection.execute( + "SELECT is_active FROM subtitle_jobs WHERE id = ?", (child["id"],) + ).fetchone()[0] == 0 + + +def test_ffmpeg_command_maps_optional_audio_and_forces_compatible_video(tmp_path: Path): + command = _build_ffmpeg_render_command( + tmp_path / "input.mp4", + tmp_path / "subtitle.ass", + tmp_path / "output.mp4", + encoder="libx264", + audio_mode="aac", + ) + assert command[command.index("-map") + 1] == "0:v:0" + assert "0:a:0?" in command + assert ["-c:v", "libx264"] == command[command.index("-c:v") : command.index("-c:v") + 2] + assert "yuv420p" in command + assert "-progress" in command + + +def test_nvenc_failure_falls_back_to_libx264(tmp_path: Path, monkeypatch: pytest.MonkeyPatch): + calls = [] + monkeypatch.setattr("app.services.subtitle_workflow_service._ffmpeg_has_encoder", lambda _name: True) + + def run(command: list[str], **_kwargs) -> None: + calls.append(command) + if "h264_nvenc" in command: + raise RuntimeError("NVENC unavailable") + + monkeypatch.setattr("app.services.subtitle_workflow_service._run_ffmpeg_progress", run) + encoder, audio_mode = _render_with_fallback( + tmp_path / "input.mp4", + tmp_path / "subtitle.ass", + tmp_path / "output.part.mp4", + workflow_job_id="job", + duration_seconds=3, + has_audio=True, + source_audio_codec="aac", + progress_start=5, + progress_end=95, + ) + assert encoder == "libx264" + assert audio_mode == "copy" + assert any("h264_nvenc" in command for command in calls) + assert any("libx264" in command for command in calls) + + +def test_ffprobe_validation_rejects_missing_audio_and_bad_pixel_format(tmp_path: Path, monkeypatch: pytest.MonkeyPatch): + output = tmp_path / "render.part.mp4" + output.write_bytes(b"video") + monkeypatch.setattr( + "app.services.subtitle_workflow_service._probe_media", + lambda _path: { + "duration": 3.0, + "video_codec": "h264", + "pixel_format": "yuv444p", + "has_audio": False, + "audio_codec": "", + }, + ) + with pytest.raises(RuntimeError, match="yuv420p"): + _validate_rendered_media(output, source_duration=3, source_has_audio=True) + + +class _SuggestionProvider: + name = "test-ai" + + def __init__(self, cue_ids: list[str]): + self.cue_ids = cue_ids + + def generate_json(self, _prompt: str) -> str: + return json.dumps( + { + "suggestions": [ + {"cue_id": cue_id, "suggested_text": f"建议{index + 1}", "reason": "错别字"} + for index, cue_id in enumerate(self.cue_ids) + ] + }, + ensure_ascii=False, + ) + + +def test_ai_suggestion_is_inactive_until_selected_diff_is_accepted(tmp_path: Path): + task_id, output_id, _ = _create_task(tmp_path) + prepare_task_subtitle_review(task_id) + track = ensure_clip_track(task_id, output_id) + base_revision = get_revision(track["active_revision_id"], include_cues=True) + cue_ids = [cue["id"] for cue in base_revision["cues"]] + suggestion = generate_subtitle_suggestions( + track["id"], + revision_id=base_revision["id"], + cue_ids=cue_ids, + provider=_SuggestionProvider(cue_ids), + ) + assert get_track(track["id"])["active_revision_id"] == base_revision["id"] + assert suggestion["revision"]["status"] == "suggested" + accepted = accept_suggestion_revision( + track["id"], + suggestion_revision_id=suggestion["revision"]["id"], + base_revision_id=base_revision["id"], + cue_ids=[cue_ids[0]], + ) + assert accepted["origin"] == "manual" + assert accepted["cues"][0]["text"] == "建议1" + assert accepted["cues"][1]["text"] == base_revision["cues"][1]["text"] + assert accepted["cues"][0]["start_ms"] == base_revision["cues"][0]["start_ms"] + + +def test_publish_readiness_blocks_unverified_subtitle_evidence(): + job = { + "status": "WAITING", + "platform": "douyin", + "publish_mode": "manual_export", + "video_source": "subtitled", + "video_file_path": "clip.mp4", + "video_path": "clip.mp4", + "title": "标题", + "caption": "正文", + "provider_payload": {"subtitle_delivery_mode": "subtitled"}, + } + readiness = build_send_readiness(job, accounts=[]) + assert readiness["dispatch_ready"] is False + assert any(issue["code"] == "subtitle_not_verified" for issue in readiness["issues"]) + + +def test_auto_publish_uses_verified_approved_subtitle(tmp_path: Path): + task_id, output_id, output_path = _create_task(tmp_path) + subtitled_path = tmp_path / "approved-subtitled.mp4" + cover_path = tmp_path / "cover.jpg" + subtitled_path.write_bytes(b"subtitled") + cover_path.write_bytes(b"cover") + result = create_auto_publish_jobs( + {"id": task_id, "platform": "general"}, + [ + { + "output_clip": { + "id": output_id, + "output_file_path": str(output_path), + "subtitle_status": "completed", + "subtitled_output_file_path": str(subtitled_path), + "subtitle_revision_id": "approved-revision", + "subtitle_revision_status": "approved", + "subtitle_validation_status": "verified", + "subtitle_verified_at": "2026-08-24T00:00:00+00:00", + }, + "cover": {"cover_file_path": str(cover_path), "cover_time_seconds": 1}, + "metadata": { + "platform": "douyin", + "title": "标题", + "caption": "正文", + "hashtags": ["测试"], + "risk_flags": [], + }, + "scheduled_at": "", + } + ], + subtitle_delivery_mode="subtitled", + ) + assert result["created"][0]["video_source"] == "subtitled" + assert result["created"][0]["video_file_path"] == str(subtitled_path) + issue_codes = {issue["code"] for issue in result["created"][0]["send_readiness"]["issues"]} + assert "subtitle_not_verified" not in issue_codes + assert "subtitle_review_required" not in issue_codes diff --git a/tests/test_subtitle_editor.py b/tests/test_subtitle_editor.py new file mode 100644 index 0000000..8468c9f --- /dev/null +++ b/tests/test_subtitle_editor.py @@ -0,0 +1,466 @@ +from __future__ import annotations + +from array import array +import hashlib +import json +from uuid import uuid4 + +import pysubs2 +import pytest +from fastapi.testclient import TestClient + +from app.db.database import get_connection, init_db +from app.main import app +from app.services.subtitle_data_service import ( + apply_revision_operations, + create_manual_revision, + ensure_clip_track, + ensure_source_track, + evaluate_subtitle_quality, + export_subtitle_text, + get_revision, + get_track, + get_waveform_peaks, + import_subtitle_text, + inherit_cues_for_clip, + serialize_revision_to_ass, +) +from app.services.video_cut_service import CutResult +from app.services.video_cut_workflow_service import _insert_output_clip_record +from app.services.subtitle_workflow_service import _write_ass_file + + +PREFIX = "test-subtitle-editor-" + + +@pytest.fixture(autouse=True) +def subtitle_editor_database(): + init_db() + _cleanup() + yield + _cleanup() + + +def _cleanup() -> None: + with get_connection() as connection: + connection.execute( + "DELETE FROM subtitle_cues WHERE revision_id IN (SELECT id FROM subtitle_revisions WHERE track_id IN (SELECT id FROM subtitle_tracks WHERE task_id LIKE ?))", + (f"{PREFIX}%",), + ) + connection.execute( + "DELETE FROM subtitle_revisions WHERE track_id IN (SELECT id FROM subtitle_tracks WHERE task_id LIKE ?)", + (f"{PREFIX}%",), + ) + connection.execute("DELETE FROM subtitle_tracks WHERE task_id LIKE ?", (f"{PREFIX}%",)) + connection.execute("DELETE FROM subtitle_jobs WHERE task_id LIKE ?", (f"{PREFIX}%",)) + connection.execute("DELETE FROM output_clip WHERE task_id LIKE ?", (f"{PREFIX}%",)) + connection.execute("DELETE FROM cut_runs WHERE task_id LIKE ?", (f"{PREFIX}%",)) + connection.execute("DELETE FROM clip_candidates WHERE task_id LIKE ?", (f"{PREFIX}%",)) + connection.execute("DELETE FROM transcription_chunks WHERE task_id LIKE ?", (f"{PREFIX}%",)) + connection.execute("DELETE FROM transcription_runs WHERE task_id LIKE ?", (f"{PREFIX}%",)) + connection.execute("DELETE FROM tasks WHERE id LIKE ?", (f"{PREFIX}%",)) + connection.commit() + + +def _create_task(*, segments: list[dict] | None = None, with_clip: bool = True) -> tuple[str, str | None]: + task_id = f"{PREFIX}{uuid4().hex[:10]}" + now = "2026-08-23T12:00:00+00:00" + with get_connection() as connection: + connection.execute( + "INSERT INTO tasks (id, task_name, platform, status, created_at, updated_at) VALUES (?, ?, 'general', 'completed', ?, ?)", + (task_id, "字幕编辑器测试", now, now), + ) + if segments is not None: + _insert_transcription(connection, task_id, segments, now) + output_id = None + if with_clip: + output_id = f"out-{uuid4().hex[:10]}" + connection.execute( + """ + INSERT INTO output_clip ( + id, task_id, output_file_path, output_file_name, status, is_active, + source_start_ms, source_end_ms, source_duration_ms, + source_fingerprint, snapshot_source, created_at, updated_at + ) VALUES (?, ?, ?, 'clip.mp4', 'completed', 1, 1000, 5000, 4000, + 'source-v1', 'cut_commit', ?, ?) + """, + (output_id, task_id, "C:/missing/clip.mp4", now, now), + ) + connection.commit() + return task_id, output_id + + +def _insert_transcription(connection, task_id: str, segments: list[dict], now: str) -> None: + run_id = f"run-{uuid4().hex[:10]}" + raw = json.dumps(segments, ensure_ascii=False, separators=(",", ":")) + checksum = hashlib.sha256(raw.encode("utf-8")).hexdigest() + connection.execute( + """ + INSERT INTO transcription_runs ( + id, task_id, source_fingerprint, provider, model, device, compute_type, + chunk_seconds, overlap_seconds, status, total_chunks, completed_chunks, + is_active, created_at, updated_at, completed_at + ) VALUES (?, ?, 'source-v1', 'local', 'small', 'cpu', 'int8', + 120, 5, 'completed', 1, 1, 1, ?, ?, ?) + """, + (run_id, task_id, now, now, now), + ) + connection.execute( + """ + INSERT INTO transcription_chunks ( + id, run_id, task_id, chunk_index, start_ms, end_ms, status, + attempt_count, result_json, result_checksum, created_at, updated_at + ) VALUES (?, ?, ?, 1, 0, 120000, 'completed', 1, ?, ?, ?, ?) + """, + (f"chunk-{uuid4().hex[:10]}", run_id, task_id, raw, checksum, now, now), + ) + + +def _segments(count: int = 4) -> list[dict]: + return [ + { + "start_seconds": index * 1.5 + 0.123, + "end_seconds": index * 1.5 + 1.345, + "text": f"第{index + 1}条中文字幕", + "confidence": 0.91, + "words": [ + { + "start_ms": round((index * 1.5 + 0.123) * 1000), + "end_ms": round((index * 1.5 + 0.5) * 1000), + "text": "第", + "confidence": 0.9, + } + ], + } + for index in range(count) + ] + + +def test_schema_migration_is_idempotent_and_contains_revision_tables(): + init_db() + init_db() + with get_connection() as connection: + names = { + row[0] + for row in connection.execute( + "SELECT name FROM sqlite_master WHERE type = 'table'" + ).fetchall() + } + output_columns = {row[1] for row in connection.execute("PRAGMA table_info(output_clip)")} + assert {"subtitle_tracks", "subtitle_revisions", "subtitle_cues"} <= names + assert {"source_start_ms", "source_end_ms", "source_duration_ms", "source_fingerprint"} <= output_columns + + +def test_source_track_uses_structured_checkpoint_with_millisecond_precision(): + task_id, _ = _create_task(segments=_segments(), with_clip=False) + track = ensure_source_track(task_id) + revision = get_revision(track["active_revision_id"], include_cues=True) + assert revision["cue_count"] == 4 + assert revision["cues"][0]["start_ms"] == 123 + assert revision["cues"][0]["end_ms"] == 1345 + assert revision["cues"][0]["confidence"] == pytest.approx(0.91) + + +def test_more_than_120_cues_are_not_truncated(): + task_id, _ = _create_task(segments=_segments(150), with_clip=False) + track = ensure_source_track(task_id) + revision = get_revision(track["active_revision_id"], include_cues=True) + assert revision["cue_count"] == 150 + assert len(revision["cues"]) == 150 + assert revision["cues"][-1]["text"] == "第150条中文字幕" + + +def test_source_to_clip_boundary_conversion_is_exact(): + source = [ + {"id": "a", "start_ms": 0, "end_ms": 1500, "text": "开头"}, + {"id": "b", "start_ms": 1500, "end_ms": 3000, "text": "中间"}, + {"id": "c", "start_ms": 4500, "end_ms": 6000, "text": "结尾"}, + {"id": "d", "start_ms": 7000, "end_ms": 8000, "text": "范围外"}, + ] + inherited = inherit_cues_for_clip(source, 1000, 5000) + assert [(cue["start_ms"], cue["end_ms"], cue["text"]) for cue in inherited] == [ + (0, 500, "开头"), + (500, 2000, "中间"), + (3500, 4000, "结尾"), + ] + assert [cue["source_cue_id"] for cue in inherited] == ["a", "b", "c"] + + +def test_clip_track_inherits_snapshot_and_manual_revision_is_not_overwritten(): + task_id, output_id = _create_task(segments=_segments(), with_clip=True) + clip_track = ensure_clip_track(task_id, output_id) + active = get_revision(clip_track["active_revision_id"], include_cues=True) + edited = [{**cue, "text": f"人工:{cue['text']}"} for cue in active["cues"]] + manual = create_manual_revision( + clip_track["id"], + base_revision_id=active["id"], + cues=edited, + note="人工精修", + ) + + with get_connection() as connection: + chunk = connection.execute( + "SELECT * FROM transcription_chunks WHERE task_id = ?", (task_id,) + ).fetchone() + changed = _segments() + changed[1]["text"] = "原片字幕已经变化" + raw = json.dumps(changed, ensure_ascii=False, separators=(",", ":")) + connection.execute( + "UPDATE transcription_chunks SET result_json = ?, result_checksum = ? WHERE id = ?", + (raw, hashlib.sha256(raw.encode("utf-8")).hexdigest(), chunk["id"]), + ) + connection.commit() + + ensure_source_track(task_id, force=True) + protected = get_track(clip_track["id"]) + assert protected["active_revision_id"] == manual["id"] + assert protected["sync_status"] == "pending_sync" + assert protected["has_manual_edits"] is True + + +def test_source_manual_revision_syncs_unedited_clip_but_only_flags_edited_clip(): + task_id, output_id = _create_task(segments=_segments(), with_clip=True) + source_track = ensure_source_track(task_id) + clip_track = ensure_clip_track(task_id, output_id) + source_revision = get_revision(source_track["active_revision_id"], include_cues=True) + source_cues = [{**cue, "text": f"原片修改:{cue['text']}"} for cue in source_revision["cues"]] + new_source = create_manual_revision( + source_track["id"], + base_revision_id=source_revision["id"], + cues=source_cues, + ) + followed = get_track(clip_track["id"]) + assert followed["source_revision_id"] == new_source["id"] + assert followed["sync_status"] == "up_to_date" + + clip_revision = get_revision(followed["active_revision_id"], include_cues=True) + manual_clip = create_manual_revision( + clip_track["id"], + base_revision_id=clip_revision["id"], + cues=[{**cue, "text": f"切片精修:{cue['text']}"} for cue in clip_revision["cues"]], + ) + unchanged_source = ensure_clip_track(task_id, output_id) + assert unchanged_source["active_revision_id"] == manual_clip["id"] + assert unchanged_source["sync_status"] == "manual" + source_after = get_revision(new_source["id"], include_cues=True) + create_manual_revision( + source_track["id"], + base_revision_id=new_source["id"], + cues=[{**cue, "text": f"再次修改:{cue['text']}"} for cue in source_after["cues"]], + ) + protected = get_track(clip_track["id"]) + assert protected["active_revision_id"] == manual_clip["id"] + assert protected["sync_status"] == "pending_sync" + + +def test_multiple_operations_keep_cue_ids_until_new_revision_is_committed(): + task_id, _ = _create_task(segments=_segments(), with_clip=False) + track = ensure_source_track(task_id) + revision = get_revision(track["active_revision_id"], include_cues=True) + cue_id = revision["cues"][0]["id"] + updated = apply_revision_operations( + track["id"], + base_revision_id=revision["id"], + operations=[ + {"type": "update", "cue_id": cue_id, "text": "连续操作"}, + {"type": "shift", "cue_ids": [cue_id], "delta_ms": 250}, + ], + ) + assert updated["cues"][0]["text"] == "连续操作" + assert updated["cues"][0]["start_ms"] == 373 + + +@pytest.mark.parametrize("format_name", ["srt", "vtt", "ass"]) +def test_pysubs2_round_trip_preserves_chinese_and_milliseconds(format_name: str): + task_id, _ = _create_task(segments=_segments(), with_clip=False) + track = ensure_source_track(task_id) + content, _media_type, _filename = export_subtitle_text(track["id"], format_name=format_name) + parsed = pysubs2.SSAFile.from_string(content, format_=format_name) + # ASS 规范使用厘秒;SRT/VTT 保留 1ms,ASS 最多产生 5ms 的量化误差。 + tolerance_ms = 5 if format_name == "ass" else 0 + assert abs(parsed.events[0].start - 123) <= tolerance_ms + assert abs(parsed.events[0].end - 1345) <= tolerance_ms + assert "中文字幕" in parsed.events[0].plaintext + + imported = import_subtitle_text(track["id"], content=content, format_name=format_name) + assert abs(imported["cues"][0]["start_ms"] - 123) <= tolerance_ms + assert abs(imported["cues"][0]["end_ms"] - 1345) <= tolerance_ms + + +@pytest.mark.parametrize("dimensions", [(1080, 1920), (1920, 1080), (1080, 1080)]) +def test_ass_resolution_follows_real_media_dimensions(monkeypatch, dimensions): + task_id, output_id = _create_task(segments=_segments(), with_clip=True) + track = ensure_clip_track(task_id, output_id) + monkeypatch.setattr("app.services.subtitle_data_service._probe_media_dimensions", lambda _path: dimensions) + ass = serialize_revision_to_ass(track["id"], track["active_revision_id"]) + document = pysubs2.SSAFile.from_string(ass, format_="ass") + assert int(document.info["PlayResX"]) == dimensions[0] + assert int(document.info["PlayResY"]) == dimensions[1] + assert document.styles["Default"].marginv == round(dimensions[1] * 0.05) + + +def test_ass_applies_default_host_and_guest_speaker_styles(monkeypatch): + task_id, output_id = _create_task(segments=_segments(), with_clip=True) + track = ensure_clip_track(task_id, output_id) + revision = get_revision(track["active_revision_id"], include_cues=True) + cues = [] + for index, cue in enumerate(revision["cues"]): + cues.append({**cue, "speaker": "主播" if index == 0 else "嘉宾"}) + manual = create_manual_revision( + track["id"], + base_revision_id=revision["id"], + cues=cues, + ) + monkeypatch.setattr( + "app.services.subtitle_data_service._probe_media_dimensions", + lambda _path: (1920, 1080), + ) + document = pysubs2.SSAFile.from_string( + serialize_revision_to_ass(track["id"], manual["id"]), + format_="ass", + ) + assert document.events[0].style != "Default" + assert document.events[1].style != "Default" + assert document.styles[document.events[0].style].primarycolor == pysubs2.Color(255, 255, 255) + assert document.styles[document.events[1].style].primarycolor == pysubs2.Color(255, 214, 10) + + +def test_ass_render_uses_explicit_immutable_revision_not_latest_active(monkeypatch, tmp_path): + task_id, output_id = _create_task(segments=_segments(), with_clip=True) + track = ensure_clip_track(task_id, output_id) + original = get_revision(track["active_revision_id"], include_cues=True) + create_manual_revision( + track["id"], + base_revision_id=original["id"], + cues=[{**cue, "text": "最新人工版本"} for cue in original["cues"]], + ) + monkeypatch.setattr( + "app.services.subtitle_workflow_service.get_artifact_paths", + lambda _task_id: {"subtitled_dir": tmp_path}, + ) + path = _write_ass_file( + task_id, + {"id": output_id, "output_file_name": "fixed.mp4"}, + {}, + revision_id=original["id"], + ) + document = pysubs2.load(str(path), encoding="utf-8") + assert "最新人工版本" not in document.events[0].plaintext + assert "中文字幕" in document.events[0].plaintext + + +def test_quality_rules_only_report_and_do_not_change_text(): + cues = [ + {"id": "a", "start_ms": 0, "end_ms": 500, "text": "这是一行非常非常非常非常非常长的中文字幕"}, + {"id": "b", "start_ms": 400, "end_ms": 9000, "text": "发生重叠\n第二行\n第三行"}, + ] + original = json.loads(json.dumps(cues, ensure_ascii=False)) + quality = evaluate_subtitle_quality(cues) + codes = {issue["code"] for issue in quality["issues"]} + assert {"too_short", "line_too_long", "reading_speed", "overlap", "too_long", "too_many_lines"} <= codes + assert quality["error_count"] == 1 + assert cues == original + + +def test_cut_commit_saves_immutable_source_bounds(): + task_id, _ = _create_task(segments=None, with_clip=False) + candidate_id = f"candidate-{uuid4().hex[:8]}" + run_id = f"cut-{uuid4().hex[:8]}" + now = "2026-08-23T12:00:00+00:00" + with get_connection() as connection: + connection.execute( + """ + INSERT INTO clip_candidates ( + id, task_id, title, start_time, end_time, duration_seconds, + created_at, updated_at + ) VALUES (?, ?, '候选', '00:00:01.250', '00:00:05.750', 5, ?, ?) + """, + (candidate_id, task_id, now, now), + ) + connection.execute( + "INSERT INTO cut_runs (id, task_id, run_number, status, is_active, created_at, updated_at) VALUES (?, ?, 1, 'processing', 0, ?, ?)", + (run_id, task_id, now, now), + ) + connection.commit() + _insert_output_clip_record( + task_id, + run_id, + CutResult(candidate_id, "C:/missing/output.mp4", "output.mp4", "completed"), + source_fingerprint="fingerprint-v1", + ) + with get_connection() as connection: + connection.execute( + "UPDATE clip_candidates SET start_time = '00:01:00', end_time = '00:02:00' WHERE id = ?", + (candidate_id,), + ) + row = connection.execute( + "SELECT * FROM output_clip WHERE task_id = ? AND clip_candidate_id = ?", + (task_id, candidate_id), + ).fetchone() + assert row["source_start_ms"] == 1250 + assert row["source_end_ms"] == 5750 + assert row["source_duration_ms"] == 4500 + assert row["source_fingerprint"] == "fingerprint-v1" + assert row["snapshot_source"] == "cut_commit" + + +def test_cue_api_supports_time_range_and_pagination(): + task_id, _ = _create_task(segments=_segments(20), with_clip=False) + track = ensure_source_track(task_id) + response = TestClient(app).get( + f"/api/subtitles/tracks/{track['id']}/cues", + params={"start_ms": 3000, "end_ms": 9000, "offset": 1, "limit": 2}, + ) + assert response.status_code == 200 + payload = response.json() + assert payload["total"] >= 4 + assert len(payload["cues"]) == 2 + assert all(cue["end_ms"] > 3000 and cue["start_ms"] < 9000 for cue in payload["cues"]) + + +def test_subtitle_page_loads_local_editor_and_vendored_wavesurfer(): + task_id, _ = _create_task(segments=_segments(), with_clip=True) + response = TestClient(app).get(f"/subtitles/{task_id}") + assert response.status_code == 200 + assert 'id="subtitle-editor"' in response.text + assert "vendor/wavesurfer/wavesurfer.min.js" in response.text + assert "vendor/wavesurfer/regions.min.js" in response.text + assert "js/subtitle-editor.js" in response.text + + +def test_waveform_peaks_are_precomputed_at_low_sample_rate_and_cached(monkeypatch, tmp_path): + media_path = tmp_path / "source.mp4" + media_path.write_bytes(b"fake-video-source") + task_id, _ = _create_task(segments=_segments(), with_clip=False) + with get_connection() as connection: + connection.execute( + "UPDATE tasks SET original_video_path = ? WHERE id = ?", + (str(media_path), task_id), + ) + connection.commit() + track = ensure_source_track(task_id) + transcript_path = tmp_path / "artifacts" / "transcript.md" + calls = [] + + class Result: + returncode = 0 + stderr = b"" + stdout = array("h", [0, 1000, -2000, 32000, -12000] * 400).tobytes() + + def fake_run(command, **_kwargs): + calls.append(command) + return Result() + + monkeypatch.setattr("app.services.subtitle_data_service.shutil.which", lambda _name: "ffmpeg") + monkeypatch.setattr("app.services.subtitle_data_service.subprocess.run", fake_run) + monkeypatch.setattr( + "app.services.subtitle_data_service.get_artifact_paths", + lambda _task_id: {"transcript_path": transcript_path}, + ) + first = get_waveform_peaks(track["id"], max_points=1000) + second = get_waveform_peaks(track["id"], max_points=1000) + assert calls and calls[0][calls[0].index("-ar") + 1] == "100" + assert first["point_count"] <= 1000 + assert first["cached"] is False + assert second["cached"] is True + assert len(calls) == 1 diff --git a/tests/test_task_defaults.py b/tests/test_task_defaults.py index 96d7e02..2a0fbc8 100644 --- a/tests/test_task_defaults.py +++ b/tests/test_task_defaults.py @@ -22,7 +22,7 @@ def _headers() -> dict[str, str]: def test_task_create_and_upload_api_use_ten_minutes_and_twelve_candidates(monkeypatch, tmp_path): - payload = TaskCreate(task_name="默认值模型测试") + payload = TaskCreate(task_name="默认值模型测试", selection_profile="general") assert payload.max_clip_duration == 10 assert payload.candidate_clip_count == 12 @@ -58,7 +58,7 @@ def fake_create_task_record( response = TestClient(app).post( "/api/tasks/upload", - data={"task_name": "上传默认值测试", "platform": "general"}, + data={"task_name": "上传默认值测试", "platform": "general", "selection_profile": "general"}, files={"video_file": ("source.mp4", b"fake-video", "video/mp4")}, headers=_headers(), ) @@ -87,13 +87,14 @@ def test_new_defaults_persist_without_rewriting_explicit_historical_values(monke try: create_task_record( - TaskCreate(task_name="新默认值"), + TaskCreate(task_name="新默认值", selection_profile="general"), task_id=default_task_id, task_dir_name=default_task_id, ) create_task_record( TaskCreate( task_name="历史显式值", + selection_profile="general", max_clip_duration=5, candidate_clip_count=5, ), diff --git a/tests/test_task_name_history.py b/tests/test_task_name_history.py index 17b710c..6128110 100644 --- a/tests/test_task_name_history.py +++ b/tests/test_task_name_history.py @@ -139,13 +139,13 @@ def test_list_task_name_history_deduplicates_non_empty_names_and_limits_results( history = task_service.list_task_name_history() - assert len(history) == 100 + assert len(history) == 5 assert history[0] == "重复任务" assert history.count("重复任务") == 1 assert "" not in history assert " " not in history - assert history[-1] == "任务 006" - assert "任务 005" not in history + assert history[-1] == "任务 101" + assert "任务 100" not in history def test_new_task_page_renders_name_history_candidates(tmp_path, monkeypatch): @@ -170,6 +170,10 @@ def test_new_task_page_renders_name_history_candidates(tmp_path, monkeypatch): assert 'name="task_name"' in response.text assert 'list="task-name-history"' in response.text assert 'autocomplete="off"' in response.text + assert "NAS" not in response.text + assert 'name="source_type"' not in response.text + assert 'name="video_file"' in response.text + assert 'id="long-live-settings" hidden' in response.text assert "九月的第二次综艺访谈精华回顾合集" in response.text assert "八月的第一次直播全程高光切片" in response.text newest_position = response.text.index("九月的第二次综艺访谈精华回顾合集") diff --git a/tests/test_task_query_service.py b/tests/test_task_query_service.py index ebdcaec..4da08f9 100644 --- a/tests/test_task_query_service.py +++ b/tests/test_task_query_service.py @@ -10,11 +10,15 @@ """ from datetime import datetime, timezone +import os +from pathlib import Path from uuid import uuid4 +from zoneinfo import ZoneInfo import pytest from app.db.database import get_connection, init_db +from app.core.config import settings from app.services.task_query_service import ( get_clips_overview_context, get_dashboard_context, @@ -152,6 +156,27 @@ def _insert_test_output_clip( connection.commit() +def _insert_test_publish_job( + job_id: str, + task_id: str, + output_clip_id: str, + status: str, + *, + created_at: str, +) -> None: + with get_connection() as connection: + connection.execute( + """ + INSERT INTO publish_jobs ( + id, task_id, output_clip_id, platform, publish_mode, + status, created_at, updated_at + ) VALUES (?, ?, ?, 'douyin', 'local_browser', ?, ?, ?) + """, + (job_id, task_id, output_clip_id, status, created_at, created_at), + ) + connection.commit() + + def _insert_test_subtitle_job( job_id: str, task_id: str, @@ -173,6 +198,21 @@ def _insert_test_subtitle_job( def _clean_test_data() -> None: + sandbox_value = os.environ.get("NIUMA_PYTEST_SANDBOX_ROOT", "").strip() + if not sandbox_value: + raise RuntimeError("拒绝清理测试数据:缺少 NIUMA_PYTEST_SANDBOX_ROOT") + sandbox_root = Path(sandbox_value).resolve() + database_path = Path(settings.database_path).resolve() + try: + is_in_sandbox = database_path.is_relative_to(sandbox_root) + except AttributeError: # pragma: no cover - Python 3.8 兼容 + is_in_sandbox = str(database_path).lower().startswith(str(sandbox_root).lower() + os.sep) + if not is_in_sandbox or database_path.name != "test_workflow.sqlite3": + raise RuntimeError( + "拒绝清理测试数据:数据库不在 pytest sandbox 内或文件名异常;" + f"database={database_path}, sandbox={sandbox_root}" + ) + with get_connection() as connection: connection.execute("DELETE FROM publish_jobs") connection.execute("DELETE FROM subtitle_jobs") @@ -185,6 +225,21 @@ def _clean_test_data() -> None: connection.commit() +def test_clean_test_data_refuses_database_outside_pytest_sandbox(): + """危险整表清理在路径异常时必须先中止,不能连接活动库。""" + original_database_path = settings.database_path + try: + object.__setattr__( + settings, + "database_path", + Path(__file__).resolve().parents[1] / "data" / "workflow.sqlite3", + ) + with pytest.raises(RuntimeError, match="拒绝清理测试数据"): + _clean_test_data() + finally: + object.__setattr__(settings, "database_path", original_database_path) + + @pytest.fixture(autouse=True) def setup_database(): """每个测试前初始化数据库并清理旧数据""" @@ -206,15 +261,13 @@ def test_empty_dashboard_fields_complete(self): # 顶层字段 assert "stats" in context - assert "focus_stats" in context - assert "workflow_steps" in context + assert "weekly_summary" in context assert "recent_tasks" in context # stats 列表结构 stat_labels = {s["label"] for s in context["stats"]} expected_labels = { - "今日新增任务", "待处理", "待检查", "已切片任务", - "待加字幕", "待推送", "失败任务", + "本周新增任务", "已切片任务", "待推送任务", "失败任务", } assert stat_labels == expected_labels @@ -223,21 +276,12 @@ def test_empty_dashboard_fields_complete(self): assert "value" in stat assert "tone" in stat - # focus_stats 列表结构 - focus_labels = {s["label"] for s in context["focus_stats"]} - assert focus_labels == {"输出切片", "待加字幕", "待推送"} - - for stat in context["focus_stats"]: - assert "label" in stat - assert "value" in stat - assert "description" in stat + assert context["weekly_summary"]["total"] == 0 + assert "days" not in context["weekly_summary"] # recent_tasks 为空列表 assert context["recent_tasks"] == [] - # workflow_steps 不为空 - assert len(context["workflow_steps"]) > 0 - def test_dashboard_counts_with_tasks(self): """有任务时统计数值正确""" task_id_1 = uuid4().hex[:12] @@ -248,14 +292,15 @@ def test_dashboard_counts_with_tasks(self): _insert_test_task(task_id_1, "任务1", status="pending_video", created_at=today) _insert_test_task(task_id_2, "任务2", status="pending_review", created_at=today) _insert_test_task(task_id_3, "任务3", status="completed", created_at="2020-01-01T00:00:00") + _insert_test_clip_candidate("clip-dashboard-completed", task_id_3) + _insert_test_output_clip("output-dashboard-completed", task_id_3, "clip-dashboard-completed") context = get_dashboard_context() stat_map = {s["label"]: s["value"] for s in context["stats"]} - assert stat_map["今日新增任务"] == 2 # task_1 + task_2 (today) - assert stat_map["待处理"] == 1 # task_1 - assert stat_map["待检查"] == 1 # task_2 + assert stat_map["本周新增任务"] == 2 assert stat_map["已切片任务"] == 1 # task_3 + assert stat_map["待推送任务"] == 1 # 已切片但还没有发布记录 assert stat_map["失败任务"] == 0 # recent_tasks 最多 5 条 @@ -275,6 +320,68 @@ def test_dashboard_counts_with_failed_tasks(self): stat_map = {s["label"]: s["value"] for s in context["stats"]} assert stat_map["失败任务"] == 2 + def test_dashboard_week_uses_shanghai_monday_boundary(self): + now = datetime(2026, 8, 26, 12, 0, tzinfo=ZoneInfo("Asia/Shanghai")) + _insert_test_task( + "week-before", "上周日任务", created_at="2026-08-23T15:59:59+00:00" + ) + _insert_test_task( + "week-monday", "本周一任务", created_at="2026-08-23T16:00:00+00:00" + ) + _insert_test_task( + "week-after", "下周一任务", created_at="2026-08-30T16:00:00+00:00" + ) + + context = get_dashboard_context(now=now) + + assert context["weekly_summary"]["range_label"] == "08.24 - 08.30" + assert context["weekly_summary"]["total"] == 1 + assert "days" not in context["weekly_summary"] + + def test_dashboard_pending_publish_uses_latest_job_state(self): + task_id = "dashboard-publish-state" + _insert_test_task(task_id, "发布状态任务", status="completed") + for index in range(1, 4): + candidate_id = f"dashboard-publish-candidate-{index}" + output_id = f"dashboard-publish-output-{index}" + _insert_test_clip_candidate(candidate_id, task_id) + _insert_test_output_clip(output_id, task_id, candidate_id) + + _insert_test_publish_job( + "dashboard-job-waiting-old", + task_id, + "dashboard-publish-output-1", + "WAITING", + created_at="2026-08-26T01:00:00+00:00", + ) + _insert_test_publish_job( + "dashboard-job-published-new", + task_id, + "dashboard-publish-output-1", + "PUBLISHED", + created_at="2026-08-26T02:00:00+00:00", + ) + _insert_test_publish_job( + "dashboard-job-published-old", + task_id, + "dashboard-publish-output-2", + "PUBLISHED", + created_at="2026-08-26T01:00:00+00:00", + ) + _insert_test_publish_job( + "dashboard-job-scheduled-new", + task_id, + "dashboard-publish-output-2", + "SCHEDULED", + created_at="2026-08-26T02:00:00+00:00", + ) + + context = get_dashboard_context() + stat_map = {s["label"]: s for s in context["stats"]} + + assert stat_map["待推送任务"]["value"] == 1 + assert "2 条" in stat_map["待推送任务"]["note"] + # ── Clips Overview Context ───────────────────────────────────────── @@ -291,8 +398,9 @@ def test_empty_clips_overview_fields_complete(self): assert context["tasks"] == [] stat_labels = {s["label"] for s in context["stats"]} - expected_labels = {"待 AI 分析", "待检查", "可生成切片", "已完成", "异常任务"} + expected_labels = {"累计审核任务", "已通过视频", "已完成任务"} assert stat_labels == expected_labels + assert all(stat["value"] == 0 for stat in context["stats"]) def test_clips_overview_with_tasks_and_clips(self): """有任务和候选片段时统计字段正确""" @@ -325,6 +433,12 @@ def test_clips_overview_with_tasks_and_clips(self): assert task["review_ready"] is True assert task["can_cut"] is False # source_exists 为 False + stat_map = {stat["label"]: stat for stat in context["stats"]} + assert stat_map["累计审核任务"]["value"] == 1 + assert stat_map["已通过视频"]["value"] == 2 + assert stat_map["已通过视频"]["note"] == "当前启用的视频片段" + assert stat_map["已完成任务"]["value"] == 0 + def test_clips_overview_with_deleted_clips(self): """已删除的候选片段不计入统计""" task_id = uuid4().hex[:12] @@ -341,20 +455,50 @@ def test_clips_overview_with_deleted_clips(self): assert task["enabled_clip_count"] == 1 def test_clips_overview_stats_correct(self): - """统计卡片数值正确""" + """累计审核、启用片段和完成任务统计正确,且排除已删除数据""" task_id_1 = uuid4().hex[:12] task_id_2 = uuid4().hex[:12] + task_id_3 = uuid4().hex[:12] + task_id_4 = uuid4().hex[:12] today = datetime.now(timezone.utc).isoformat(timespec="seconds") - _insert_test_task(task_id_1, "待AI", status="pending_ai", created_at=today) - _insert_test_task(task_id_2, "失败", status="failed", created_at=today) + _insert_test_task(task_id_1, "待检查", status="pending_review", created_at=today) + _insert_test_task(task_id_2, "大写完成", status="COMPLETED", created_at=today) + _insert_test_task( + task_id_3, + "部分完成", + status="completed_with_errors", + created_at=today, + ) + _insert_test_task( + task_id_4, + "已删除任务", + status="completed", + is_deleted=1, + created_at=today, + ) + _insert_test_clip_candidate("clip_stats_1", task_id_1, "启用片段", enabled=1) + _insert_test_clip_candidate("clip_stats_2", task_id_1, "未启用片段", enabled=0) + _insert_test_clip_candidate("clip_stats_3", task_id_2, "完成片段", enabled=1) + _insert_test_clip_candidate("clip_stats_4", task_id_4, "已删除任务片段", enabled=1) + _insert_test_clip_candidate( + "clip_stats_5", + task_id_1, + "已删除片段", + enabled=1, + is_deleted=1, + ) context = get_clips_overview_context() stat_map = {s["label"]: s["value"] for s in context["stats"]} + task_map = {task["id"]: task for task in context["tasks"]} - assert stat_map["待 AI 分析"] == 1 - assert stat_map["待检查"] == 0 - assert stat_map["异常任务"] == 1 + assert stat_map == { + "累计审核任务": 2, + "已通过视频": 2, + "已完成任务": 2, + } + assert task_map[task_id_2]["review_stage"] == "已完成" # ── Subtitle Workflow Context ────────────────────────────────────── diff --git a/tests/test_task_state_machine.py b/tests/test_task_state_machine.py new file mode 100644 index 0000000..3e3ad30 --- /dev/null +++ b/tests/test_task_state_machine.py @@ -0,0 +1,149 @@ +"""P1.3:任务状态机与删除任务写保护。""" + +from __future__ import annotations + +from fastapi.testclient import TestClient +import pytest + +from app.core.config import settings +from app.db.database import get_connection +from app.main import app +from app.models.task import TaskCreate, TaskStatus +from app.services import job_service +from app.services.task_lifecycle_service import ( + TaskStatusConflictError, + create_task_record, + update_task_status, +) + + +def _headers() -> dict[str, str]: + if settings.local_admin_token: + return {"Authorization": f"Bearer {settings.local_admin_token}"} + return {} + + +@pytest.fixture(autouse=True) +def cleanup_tasks(): + yield + with get_connection() as connection: + connection.execute("DELETE FROM publish_jobs WHERE task_id LIKE 'test-state-%'") + connection.execute("DELETE FROM workflow_jobs WHERE task_id LIKE 'test-state-%'") + connection.execute("DELETE FROM tasks WHERE id LIKE 'test-state-%'") + connection.commit() + + +def _create_task(task_id: str, *, auto_mode: bool = False) -> dict: + return create_task_record( + TaskCreate( + task_name=task_id, + source_type="upload", + platform="general", + selection_profile="general", + auto_mode=auto_mode, + ), + task_id=task_id, + task_dir_name=task_id, + ) + + +def test_status_api_rejects_jump_from_empty_task_to_completed(): + task = _create_task("test-state-no-jump") + + with TestClient(app) as client: + response = client.patch( + f"/api/tasks/{task['id']}/status", + json={"status": TaskStatus.completed.value}, + headers=_headers(), + ) + + assert response.status_code == 409 + with get_connection() as connection: + row = connection.execute("SELECT status FROM tasks WHERE id = ?", (task["id"],)).fetchone() + assert row["status"] == TaskStatus.pending_video.value + + +def test_status_api_allows_declared_adjacent_transition(): + task = _create_task("test-state-adjacent") + with get_connection() as connection: + connection.execute( + "UPDATE tasks SET original_video_path = 'C:/managed/source.mp4' WHERE id = ?", + (task["id"],), + ) + connection.commit() + + with TestClient(app) as client: + response = client.patch( + f"/api/tasks/{task['id']}/status", + json={"status": TaskStatus.pending_processing.value}, + headers=_headers(), + ) + + assert response.status_code == 200 + assert response.json()["status"] == TaskStatus.pending_processing.value + + +def test_status_api_requires_source_before_pending_processing(): + task = _create_task("test-state-source-required") + + with TestClient(app) as client: + response = client.patch( + f"/api/tasks/{task['id']}/status", + json={"status": TaskStatus.pending_processing.value}, + headers=_headers(), + ) + + assert response.status_code == 409 + assert "尚未绑定源视频" in response.json()["detail"] + + +def test_internal_status_write_rejects_deleted_task(): + task = _create_task("test-state-deleted") + with get_connection() as connection: + connection.execute("UPDATE tasks SET is_deleted = 1 WHERE id = ?", (task["id"],)) + connection.commit() + + with pytest.raises(TaskStatusConflictError, match="已永久删除"): + update_task_status(task["id"], TaskStatus.completed) + + with get_connection() as connection: + row = connection.execute("SELECT status FROM tasks WHERE id = ?", (task["id"],)).fetchone() + assert row["status"] == TaskStatus.pending_video.value + + +def test_deleted_task_is_hidden_from_detail_and_media_apis(): + task = _create_task("test-state-hidden") + with get_connection() as connection: + connection.execute("UPDATE tasks SET is_deleted = 1 WHERE id = ?", (task["id"],)) + connection.commit() + + with TestClient(app) as client: + detail = client.get(f"/api/tasks/{task['id']}", headers=_headers()) + media = client.get(f"/media/tasks/{task['id']}/source-video", headers=_headers()) + + assert detail.status_code == 404 + assert media.status_code == 404 + + +def test_public_cancel_also_requests_active_auto_pipeline_stop(): + task = _create_task("test-state-public-cancel", auto_mode=True) + update_task_status(task["id"], TaskStatus.PREPARING_SOURCE) + job, _created = job_service.create_or_get_active_job( + task_id=task["id"], + job_type=job_service.JOB_TYPE_AUTO_PIPELINE, + ) + claimed = job_service.claim_job(job["id"], "public-cancel-owner") + assert claimed is not None + + with TestClient(app) as client: + response = client.patch( + f"/api/tasks/{task['id']}/status", + json={"status": TaskStatus.CANCELLED.value}, + headers=_headers(), + ) + + assert response.status_code == 200 + assert response.json()["status"] == TaskStatus.CANCELLED.value + current_job = job_service.get_job(job["id"]) + assert current_job["status"] == job_service.JOB_STATUS_RUNNING + assert current_job["cancel_requested"] == 1 diff --git a/tests/test_test_environment_isolation.py b/tests/test_test_environment_isolation.py new file mode 100644 index 0000000..5ef7e40 --- /dev/null +++ b/tests/test_test_environment_isolation.py @@ -0,0 +1,19 @@ +"""pytest 测试环境必须与活动数据库和媒体目录隔离。""" + +import os +from pathlib import Path + + +def test_pytest_paths_are_under_process_sandbox(): + sandbox = Path(os.environ["NIUMA_PYTEST_SANDBOX_ROOT"]).resolve() + assert sandbox.exists() + assert Path(os.environ["DATABASE_PATH"]).resolve().is_relative_to(sandbox) + assert Path(os.environ["DATABASE_PATH"]).name == "test_workflow.sqlite3" + for name in ( + "STORAGE_ROOT", + "TASKS_DIR", + "UPLOAD_TEMP_DIR", + "DATA_DIR", + "PUBLISH_SCHEDULER_EXPORT_DIR", + ): + assert Path(os.environ[name]).resolve().is_relative_to(sandbox) diff --git a/tests/test_transcription_checkpoint_resilience.py b/tests/test_transcription_checkpoint_resilience.py new file mode 100644 index 0000000..69ee31e --- /dev/null +++ b/tests/test_transcription_checkpoint_resilience.py @@ -0,0 +1,125 @@ +from __future__ import annotations + +from datetime import datetime, timezone +import hashlib + +import pytest + +from app.db.database import get_connection, init_db +from app.services.transcript_service import TranscriptChunk, TranscriptSegment, _segment_from_checkpoint +from app.services.transcription_checkpoint_service import ( + RemoteTranscriptionResultUncertainError, + TranscriptionCheckpoint, +) + + +def _create_task(task_id: str) -> None: + now = datetime.now(timezone.utc).isoformat(timespec="seconds") + with get_connection() as connection: + connection.execute( + """ + INSERT INTO tasks ( + id, task_name, task_dir_name, source_type, platform, selection_profile, + status, progress, is_deleted, created_at, updated_at + ) VALUES (?, 'checkpoint resilience', ?, 'upload', 'general', 'general', + 'pending_processing', 0, 0, ?, ?) + """, + (task_id, task_id, now, now), + ) + connection.commit() + + +def _cleanup(task_id: str) -> None: + with get_connection() as connection: + connection.execute("DELETE FROM transcription_chunks WHERE task_id = ?", (task_id,)) + connection.execute("DELETE FROM transcription_runs WHERE task_id = ?", (task_id,)) + connection.execute("DELETE FROM tasks WHERE id = ?", (task_id,)) + connection.commit() + + +def _checkpoint(task_id: str, source, provider: str = "local") -> TranscriptionCheckpoint: + return TranscriptionCheckpoint( + task_id=task_id, + source_path=source, + provider=provider, + model="test-model", + device="remote" if provider == "volcengine" else "cpu", + compute_type="mp3" if provider == "volcengine" else "int8", + chunk_seconds=120, + overlap_seconds=5, + ) + + +def test_corrupted_completed_chunk_is_invalidated_and_recomputed(tmp_path): + init_db() + task_id = "test-checkpoint-corruption" + source = tmp_path / "audio.wav" + source.write_bytes(b"audio") + _create_task(task_id) + try: + checkpoint = _checkpoint(task_id, source) + checkpoint.ensure_run([TranscriptChunk(1, 0, 120)]) + checkpoint.save_completed(1, [TranscriptSegment(0, 1, "ok")]) + corrupt = "{not-json" + checksum = hashlib.sha256(corrupt.encode("utf-8")).hexdigest() + with get_connection() as connection: + connection.execute( + """ + UPDATE transcription_chunks + SET result_json = ?, result_checksum = ? + WHERE run_id = ? AND chunk_index = 1 + """, + (corrupt, checksum, checkpoint.run_id), + ) + connection.commit() + assert checkpoint.load_completed(1, _segment_from_checkpoint) is None + with get_connection() as connection: + row = connection.execute( + "SELECT status, result_json FROM transcription_chunks WHERE run_id = ? AND chunk_index = 1", + (checkpoint.run_id,), + ).fetchone() + assert row["status"] == "queued" + assert row["result_json"] is None + finally: + _cleanup(task_id) + + +def test_unfinished_remote_request_requires_explicit_new_run(tmp_path): + init_db() + task_id = "test-checkpoint-remote-uncertain" + source = tmp_path / "audio.mp3" + source.write_bytes(b"remote-audio") + chunks = [TranscriptChunk(1, 0, 120)] + _create_task(task_id) + try: + first = _checkpoint(task_id, source, provider="volcengine") + first.ensure_run(chunks) + request_id = first.prepare_remote_request(1) + assert request_id + + resumed = _checkpoint(task_id, source, provider="volcengine") + resumed.ensure_run(chunks) + assert resumed.run_id == first.run_id + with pytest.raises(RemoteTranscriptionResultUncertainError, match="没有可靠结果"): + resumed.prepare_remote_request(1) + + resumed.save_uncertain(1, "结果不确定") + blocked_retry = _checkpoint(task_id, source, provider="volcengine") + with pytest.raises(RemoteTranscriptionResultUncertainError, match="普通任务重试不会再次请求"): + blocked_retry.ensure_run(chunks) + + explicit_retry = TranscriptionCheckpoint( + task_id=task_id, + source_path=source, + provider="volcengine", + model="test-model", + device="remote", + compute_type="mp3", + chunk_seconds=120, + overlap_seconds=5, + allow_uncertain_retry=True, + ) + explicit_retry.ensure_run(chunks) + assert explicit_retry.run_id != first.run_id + finally: + _cleanup(task_id) diff --git a/tests/test_ui_motion.py b/tests/test_ui_motion.py new file mode 100644 index 0000000..70d249b --- /dev/null +++ b/tests/test_ui_motion.py @@ -0,0 +1,78 @@ +from pathlib import Path + + +PROJECT_ROOT = Path(__file__).resolve().parents[1] +BASE_TEMPLATE = PROJECT_ROOT / "app" / "templates" / "base.html" +MOTION_SCRIPT = PROJECT_ROOT / "app" / "static" / "js" / "motion.js" +APP_SCRIPT = PROJECT_ROOT / "app" / "static" / "js" / "app.js" +PUBLISH_SCRIPT = PROJECT_ROOT / "app" / "static" / "js" / "publish-center.js" +STYLESHEET = PROJECT_ROOT / "app" / "static" / "css" / "styles.css" + + +def test_base_loads_local_motion_layer_after_page_scripts() -> None: + template = BASE_TEMPLATE.read_text(encoding="utf-8") + + assert 'data-page="{{ active_page or \'\' }}"' in template + assert "js/motion.js" in template + assert template.index("{% block extra_scripts %}") < template.index("js/motion.js") + motion_script_line = next(line for line in template.splitlines() if "js/motion.js" in line) + assert "http://" not in motion_script_line + assert "https://" not in motion_script_line + + +def test_motion_profiles_cover_every_rendered_page_family() -> None: + script = MOTION_SCRIPT.read_text(encoding="utf-8") + + expected_profiles = { + 'name: "dashboard"', + 'name: "new-task"', + 'name: "tasks"', + 'name: "task-detail"', + 'name: "transcript"', + 'name: "clips"', + 'name: "clip-review"', + 'name: "subtitles"', + 'name: "subtitle-task"', + 'name: "publish"', + 'name: "system"', + } + + for profile in expected_profiles: + assert profile in script + + +def test_motion_layer_is_progressive_and_presentation_only() -> None: + script = MOTION_SCRIPT.read_text(encoding="utf-8") + + assert 'matchMedia("(prefers-reduced-motion: reduce)")' in script + assert "IntersectionObserver" in script + assert "MutationObserver" in script + assert "body.dataset.motionPage" in script + assert "if (nextValue === lastValue) return" in script + assert "#runtime-log-lines" not in script + assert "fetch(" not in script + assert "apiFetch" not in script + assert "innerHTML" not in script + assert "localStorage" not in script + + +def test_styles_disable_nonessential_motion_when_user_requests_it() -> None: + styles = STYLESHEET.read_text(encoding="utf-8") + + assert "@media print, (prefers-reduced-motion: reduce)" in styles + assert "animation-duration: 1ms !important" in styles + assert "transition-duration: 1ms !important" in styles + assert ".send-publishing-spinner" in styles + assert "@keyframes motion-drawer-in" in styles + + +def test_reduced_motion_also_disables_scripted_smooth_scrolling() -> None: + app_script = APP_SCRIPT.read_text(encoding="utf-8") + publish_script = PUBLISH_SCRIPT.read_text(encoding="utf-8") + + assert "function preferredScrollBehavior()" in app_script + assert 'matchMedia("(prefers-reduced-motion: reduce)")' in app_script + assert "window.preferredScrollBehavior = preferredScrollBehavior" in app_script + assert 'behavior: "smooth"' not in app_script + assert 'behavior: "smooth"' not in publish_script + assert publish_script.count("window.preferredScrollBehavior()") == 7 diff --git a/tests/test_variety_comedy_selection.py b/tests/test_variety_comedy_selection.py index a38dca5..268d73c 100644 --- a/tests/test_variety_comedy_selection.py +++ b/tests/test_variety_comedy_selection.py @@ -388,7 +388,7 @@ def test_three_stage_flow_allows_weak_episode_to_select_less_than_target(monkeyp assert 60 <= result.clips[0].duration_seconds <= 150 -def test_auto_pipeline_only_enables_selected_a_grade_clips(): +def test_auto_pipeline_only_enables_selected_a_grade_clips(monkeypatch): task_id = f"{PREFIX}auto" _insert_task(task_id, final_target=5) get_artifact_paths(task_id)["analysis_path"].parent.mkdir(parents=True, exist_ok=True) @@ -398,6 +398,22 @@ def test_auto_pipeline_only_enables_selected_a_grade_clips(): } _insert_candidate(task_id, 5, tier="A", quality=99, selected_by_default=False) _insert_candidate(task_id, 6, tier="B", quality=76, selected_by_default=True) + monkeypatch.setattr( + "app.services.pipeline_engine.task_service.get_task_ai_analysis_meta", + lambda _task_id: { + "schema_version": 2, + "selection_profile": "variety_comedy", + "analysis_incomplete": False, + "quality_degraded": False, + "coverage_ratio": 1.0, + "coverage_percent": 100.0, + "expected_units": 1, + "completed_units": 1, + "failed_units": 0, + "failed_stages": [], + "invalid_item_count": 0, + }, + ) result = PipelineEngine()._select_clips(task_id, {"config": {}}) diff --git a/tests/test_versioning_rollback.py b/tests/test_versioning_rollback.py index 3f93bbf..f14d617 100644 --- a/tests/test_versioning_rollback.py +++ b/tests/test_versioning_rollback.py @@ -45,6 +45,7 @@ def _create_task(self, task_id: str) -> dict: task_name=f"版本化测试-{task_id}", source_type="upload", platform="general", + selection_profile="general", max_clip_duration=5, candidate_clip_count=5, ai_preference="", diff --git a/third_party_licenses/pysubs2-LICENSE.txt b/third_party_licenses/pysubs2-LICENSE.txt new file mode 100644 index 0000000..dc62e89 --- /dev/null +++ b/third_party_licenses/pysubs2-LICENSE.txt @@ -0,0 +1,19 @@ +Copyright (c) 2014-2026 Tomas Karabela + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE.