diff --git a/.codemap/codemap.html b/.codemap/codemap.html new file mode 100644 index 0000000..cec06c8 --- /dev/null +++ b/.codemap/codemap.html @@ -0,0 +1,634 @@ + + + + + + +Functional Architecture Map + + + +
+
+
+
+

Functional Architecture Map

+
+
+
+
+ + +
+ + + + + + + +
+
+
+ + + + + + +
+
click a module · scroll to pan
+
+ +
+ + + + + diff --git a/.codemap/codemap.md b/.codemap/codemap.md new file mode 100644 index 0000000..5989372 --- /dev/null +++ b/.codemap/codemap.md @@ -0,0 +1,225 @@ + + +# GroupBrief V1 — Functional Module Quality Audit + +> **Interactive view:** [`.codemap/codemap.html`](codemap.html) — per-module scores, findings, LoC, and the dependency graph. This file is the written report. + +**Generated:** 2026-08-24 · **Modules:** 12 · **Size:** 28149 tracked LoC across 116 files + +## Health by layer + +| Layer | Modules | Avg score | +|---|--:|--:| +| 前端 · 管理界面 | 2 | 67 | +| 后端 · 入口与 API | 1 | 58 | +| 编排 · 调度与状态机 | 1 | 54 | +| 领域 · 数据到内容 | 6 | 60 | +| 持久化 · SQLite 与工件 | 1 | 62 | +| 运维 · 脚本 | 1 | 58 | + +## Per-module lines of code & score + +_LoC is the representative file/folder per module; folder-level modules overlap and are not additive._ + +### 前端 · 管理界面 + +| Module | LoC | Score | Tags | +|---|--:|:--|:--| +| 前端界面 | 10,683 | 62 C | god-component, silent-except, fallback, duplication, dual-format, legacy, bloat, glue, stub, placeholder | +| 前端 API 客户端 | 427 | 72 C | dual-format, duplication, glue, any-escape | + +### 后端 · 入口与 API + +| Module | LoC | Score | Tags | +|---|--:|:--|:--| +| 运行时与 HTTP API | 2,419 | 58 D | god-component, bloat, silent-except, fallback, legacy, dual-format, duplication, fake-output, glue | + +### 编排 · 调度与状态机 + +| Module | LoC | Score | Tags | +|---|--:|:--|:--| +| Pipeline 与调度 | 2,160 | 54 D | god-component, bloat, glue, fallback, silent-except, legacy, dual-format, duplication, placeholder | + +### 领域 · 数据到内容 + +| Module | LoC | Score | Tags | +|---|--:|:--|:--| +| AI 摘要与生图提示词 | 3,601 | 58 D | fallback, silent-except, legacy, dual-format, duplication, bloat, god-component, glue, any-escape, over-fit | +| 微信数据接入 | 2,334 | 64 C | god-component, bloat, fallback, fake-output, duplication, dual-format, glue, silent-except, legacy | +| 图片生成与恢复 | 1,951 | 61 C | god-component, bloat, fallback, silent-except, legacy, dual-format, duplication, any-escape, glue | +| 微信与邮件交付 | 1,725 | 61 C | fallback, silent-except, legacy, dual-format, duplication, bloat, god-component, glue, over-fit | +| V1 报告兼容链 | 696 | 52 D | fake-output, silent-except, legacy, dual-format, duplication, glue, bloat | +| 排行榜与统计 | 376 | 64 C | legacy, dual-format, duplication, bloat, glue, over-fit | + +### 持久化 · SQLite 与工件 + +| Module | LoC | Score | Tags | +|---|--:|:--|:--| +| 数据库与运行状态 | 1,125 | 62 C | silent-except, fallback, legacy, dual-format, duplication, bloat, god-component, glue, any-escape | + +### 运维 · 脚本 + +| Module | LoC | Score | Tags | +|---|--:|:--|:--| +| 运维与计划任务脚本 | 652 | 58 D | legacy, dual-format, fake-output, glue, over-fit, bloat | + +## Worst offenders + +- **V1 报告兼容链 (52/D)** — app/services/report_service.py:247: Prompt 失败时仍继续保存 Report 并调用 HandoffService；handoff_service.py:82-90 无条件写入 status=prompt_ready、poster_file=null，可能把空 Prompt 暴露为可交接产物。另在 268-271 行捕获文件输出异常只写 error_message，145-147 行仅检查 ranking_status/prompt_status，文件缺失仍可能被判定为成功。 +- **Pipeline 与调度 (54/D)** — app/pipeline/daily_pipeline.py:72: DailyPipeline 文件共约 1358 行，单类同时编排配置、群名同步、数据源、排行榜、Prompt、图片生成、RunStore、发送、恢复与人工确认；构造函数在 85-95 行直接连接多个外部子系统，职责和 blast radius 过大。 +- **运行时与 HTTP API (58/D)** — app/api/files.py:25: 路径安全检查使用字符串 startswith；report_date、group_dir、filename 均未规范化。可用 sibling-prefix 绕过目录边界，例如 output/2026-08-1/../2026-08-10/ranking.txt 解析后仍满足 startswith，存在跨日期目录读取风险。 +- **AI 摘要与生图提示词 (58/D)** — app/ai/prompt_builder.py:76-80,558-673; app/providers/ai/codex.py:131-152,159-221; app/providers/ai/deepseek.py:227-259: 存在多层重试与回退叠加：Builder 的事件、候选主题、版式和最终 Prompt 各自重试，Codex/DeepSeek Provider 内部再次重试，Codex 任意异常还会切换 DeepSeek。一次超时、空响应或已被服务端接受但客户端断连的请求可能产生多次模型调用、延迟放大和重复计费。 +- **运维与计划任务脚本 (58/D)** — scripts/test_wechat_data.py:31: 脚本声明只读取微信数据，但每个命令在 138 行都会先调用 _load_settings；该函数执行 repo.init_db 和 repo.apply_db_settings，初始化过程包含建表、默认值写入和迁移，健康检查/列群等只读操作可能修改生产数据库。 +- **图片生成与恢复 (61/C)** — app/image/codex_generator.py:143: CodexImageGenerator 所在文件约 892 行，单类同时负责 CLI 健康检查、跨进程互斥、子进程树终止、超时恢复、attempt manifest、候选扫描、哈希去重、图片验证、原子提升和 smoke 状态，属于高 blast-radius God Component。 +- **微信与邮件交付 (61/C)** — app/services/email_service.py:200-243; scripts/send_daily_email.py:203-243: SMTP 发送失败后直接重试同一 EmailMessage；如果 SMTP 服务端已经接收邮件但客户端在响应阶段断开，第二次尝试可能造成重复邮件。代码没有幂等键、Message-ID 去重或提交状态确认。 +- **前端界面 (62/C)** — frontend/src/pages/v2/AIImages.tsx:130: 单一 AIImages 组件从 130 行延伸至文件末尾约 696 行，132-177 行集中维护约 40 个 state/ref，同时承载运行列表、群级与运行级 Prompt、主题解析、图片再生、恢复、发送确认和轮询，是核心流程 God Component。 +- **数据库与运行状态 (62/C)** — app/db/models.py:10-79: 持久化模型缺少关系约束：wechat_group_id 仅普通索引，GroupRun.run_id/group_id 与 Report.group_run_id 都是无 ForeignKey 的裸整数，也没有复合唯一约束；重复群组、重复运行和孤儿记录只能依赖业务层避免。 +- **微信数据接入 (64/C)** — app/providers/history/wechat_data_analysis.py:92: WeChatDataAnalysisProvider 文件约 858 行，单个 Provider 同时负责 MCP 配置与健康检查、JSON 导出、群发现/解析、范围分页、旧锚点分页、响应解析、联系人映射、发送人冲突修复和消息转换，属于历史接入 God Component，修改任一上游协议都可能影响整条取数链。 + +## All findings + +### HIGH (29) + +- **前端界面** · `frontend/src/pages/v2/AIImages.tsx:130` — 单一 AIImages 组件从 130 行延伸至文件末尾约 696 行，132-177 行集中维护约 40 个 state/ref，同时承载运行列表、群级与运行级 Prompt、主题解析、图片再生、恢复、发送确认和轮询，是核心流程 God Component。 +- **运行时与 HTTP API** · `app/api/files.py:25` — 路径安全检查使用字符串 startswith；report_date、group_dir、filename 均未规范化。可用 sibling-prefix 绕过目录边界，例如 output/2026-08-1/../2026-08-10/ranking.txt 解析后仍满足 startswith，存在跨日期目录读取风险。 +- **运行时与 HTTP API** · `app/api/v2_ui.py:84` — 单文件 833 行同时承担 Dashboard、归档聚合、Prompt 编辑、图片重生成、恢复、健康检查、Pipeline 控制和文件读取等多个边界，包含约 20 个路由与大量内嵌业务逻辑，属于高耦合 God Component。 +- **Pipeline 与调度** · `app/pipeline/daily_pipeline.py:72` — DailyPipeline 文件共约 1358 行，单类同时编排配置、群名同步、数据源、排行榜、Prompt、图片生成、RunStore、发送、恢复与人工确认；构造函数在 85-95 行直接连接多个外部子系统，职责和 blast radius 过大。 +- **Pipeline 与调度** · `app/pipeline/daily_pipeline.py:124` — PeriodResolver 支持 schedule_rule（app/scheduler/period.py:35-47），但 generate_all、force_generate、rebuild_prompt_from_snapshot 均未传入 Group.schedule_rule（本行及 1045、1115 行），群级周期配置会被静默按 weekday_default 执行。 +- **Pipeline 与调度** · `app/pipeline/daily_pipeline.py:1304` — _save_json 在 1304-1306 行直接 write_text；messages/ranking 等工件与 run.json 分步写入。加载损坏快照时 1310-1313 行直接失败且明确不回源，进程中断可能留下不可恢复的半成品状态。 +- **Pipeline 与调度** · `app/scheduler/daily_v2_job.py:40` — DailyScheduleState.load 在 42-48 行将 OSError/JSONDecodeError 直接吞掉并返回仅含 run_date 的新状态；若已有状态文件损坏，_run_locked 可能把已完成的邮件阶段当成未开始，失去去重依据并触发重复外部发送。 +- **微信数据接入** · `app/providers/history/wechat_data_analysis.py:92` — WeChatDataAnalysisProvider 文件约 858 行，单个 Provider 同时负责 MCP 配置与健康检查、JSON 导出、群发现/解析、范围分页、旧锚点分页、响应解析、联系人映射、发送人冲突修复和消息转换，属于历史接入 God Component，修改任一上游协议都可能影响整条取数链。 +- **微信数据接入** · `app/data_sources/wechat_data_analysis.py:41` — V2 WeChatDataAnalysisSource 在 46-54 行只构造 WeChatDataAnalysisProvider，没有接入 app.providers.history.registry 的 wechat-cli/Mock fallback；文档虽然称 MCP/导出为回退路线，但 V2 真实数据源不可用时不会自动切换到 CLI 或 Mock，和 V1 provider registry 的契约不一致。 +- **微信数据接入** · `app/providers/history/registry.py:33` — build_providers 在 33-34 行会在 history_provider_mock_enabled 开启时自动追加 MockProvider；HistoryService.fetch 在 99-116 行把第一个 OK/EMPTY_RESULT 当作有效结果返回。真实 Provider 失败时，fixtures 可能被当成真实日报输入，形成 fake-output 风险。 +- **排行榜与统计** · `app/ranking/engine.py:18-64; app/services/ranking_service.py:17-76` — 项目同时维护两套 RankingEngine/RankingResult：V2 处理 V2Message 并输出结构化 JSON，V1 service 处理 NormalizedMessage 并内置 Top10 文本渲染。两套实现都做发言统计和确定性排序，但过滤入口、字段协议、上限和输出格式不同，规则修改容易发生语义漂移。 +- **AI 摘要与生图提示词** · `app/ai/prompt_builder.py:76-80,558-673; app/providers/ai/codex.py:131-152,159-221; app/providers/ai/deepseek.py:227-259` — 存在多层重试与回退叠加：Builder 的事件、候选主题、版式和最终 Prompt 各自重试，Codex/DeepSeek Provider 内部再次重试，Codex 任意异常还会切换 DeepSeek。一次超时、空响应或已被服务端接受但客户端断连的请求可能产生多次模型调用、延迟放大和重复计费。 +- **AI 摘要与生图提示词** · `app/ai/prompt_builder.py:225-460,512-673` — DeepSeekImagePromptBuilder 是明显 god-component：同一类同时负责模板读取、主题解析、消息分块、事件提取、候选选题、证据回查、版式导演、最终 Prompt 拼接、敏感字段检查、元数据和 Provider 调用，约 677 行且跨越多个业务阶段，修改一个阶段容易影响整条链路。 +- **图片生成与恢复** · `app/image/codex_generator.py:143` — CodexImageGenerator 所在文件约 892 行，单类同时负责 CLI 健康检查、跨进程互斥、子进程树终止、超时恢复、attempt manifest、候选扫描、哈希去重、图片验证、原子提升和 smoke 状态，属于高 blast-radius God Component。 +- **图片生成与恢复** · `scripts/codex_image_automation.py:449` — 人工 adopt 流程在 449-458 行按共享 generated_images 目录的快照增量选择唯一 PNG，但 begin/adopt 没有跨进程互斥；两个并行人工任务各自产生单个候选时，候选可能被错误群任务认领并写入错误 run.json。 +- **图片生成与恢复** · `scripts/codex_image_automation.py:336` — _sync_scheduler_result 在 336-379 行对 output/.scheduler/.json 做无锁 read-modify-write。它与 DailyScheduleState 的进程内锁不共享，可能覆盖同时写入的 generation/email 字段，造成调度状态丢失或重复阶段判断。 +- **微信与邮件交付** · `app/services/email_service.py:200-243; scripts/send_daily_email.py:203-243` — SMTP 发送失败后直接重试同一 EmailMessage；如果 SMTP 服务端已经接收邮件但客户端在响应阶段断开，第二次尝试可能造成重复邮件。代码没有幂等键、Message-ID 去重或提交状态确认。 +- **微信与邮件交付** · `app/services/email_service.py:111-154` — EmailService.send 只返回汇总结果，不写入每群 email_status、发送批次或幂等记录；同一 run 被 scheduler、手动脚本或恢复流程再次调用时，已成功群仍可能重新发送。 +- **V1 报告兼容链** · `app/services/report_service.py:247` — Prompt 失败时仍继续保存 Report 并调用 HandoffService；handoff_service.py:82-90 无条件写入 status=prompt_ready、poster_file=null，可能把空 Prompt 暴露为可交接产物。另在 268-271 行捕获文件输出异常只写 error_message，145-147 行仅检查 ranking_status/prompt_status，文件缺失仍可能被判定为成功。 +- **V1 报告兼容链** · `app/services/report_service.py:317` — _find_success_group_run 只查询 GroupRun.ranking_status == success，没有要求 prompt_status == success。此前排行榜成功但 Prompt 失败的记录会在 175-177 行被当作已完成，非 force 重试无法修复，只会再次返回失败。 +- **V1 报告兼容链** · `app/services/report_service.py:148` — GroupRun 初始 running/pending 状态在 179-187 行先提交；后续异常在 148-159 行通过新增一条 failed GroupRun 处理，原记录会遗留为 running/pending。父 Run 只有在 118-126 行全部 worker 返回后才收口，进程或 worker 异常可留下孤儿状态。 +- **V1 报告兼容链** · `app/services/handoff_service.py:34` — safe_dir_name 仅替换字符并截断 display_name，不加入稳定 group_id，也不拒绝 '.'、'..' 或 Windows 保留名；save_outputs 在 52-61 行直接使用该目录并覆盖写文件。不同群名可能碰撞覆盖，'.'/'..' 还可把写入指向日期目录或 output 根目录。 +- **V1 报告兼容链** · `app/services/handoff_service.py:105` — list_group_outputs 将未经格式校验的 report_date 直接拼入 output_dir 路径；文件接口把用户路径参数直接传入，绝对路径或 '..' 可突破预期日期目录并列出其他目录，同时返回完整本地 filesystem path。 +- **V1 报告兼容链** · `scripts/run_daily_pipeline.py:35` — _print_results 只打印每群状态；generate、send、force-generate、rebuild-prompt、force-send 分支分别在 99、105、114、119、124 行无条件 return 0。即使结果为 failed/partial，调度器仍会收到成功退出码。 +- **数据库与运行状态** · `app/db/models.py:10-79` — 持久化模型缺少关系约束：wechat_group_id 仅普通索引，GroupRun.run_id/group_id 与 Report.group_run_id 都是无 ForeignKey 的裸整数，也没有复合唯一约束；重复群组、重复运行和孤儿记录只能依赖业务层避免。 +- **数据库与运行状态** · `app/db/models.py:38-79; app/v2/run_store.py:1-7` — 存在双持久化格式：V1 将 Run/GroupRun/Report 写入 SQLite，V2 将同一类运行状态和输出元数据写入每群每日 run.json；当前模块没有统一同步或一致性边界，状态可能分叉。 +- **数据库与运行状态** · `app/v2/run_store.py:153-162` — load_run 对 JSONDecodeError/OSError 静默 pass，并回退为全新的 PENDING 状态；损坏或部分写入的状态会被当成未执行任务，可能触发重复生成或重复后续操作。 +- **运维与计划任务脚本** · `scripts/test_wechat_data.py:31` — 脚本声明只读取微信数据，但每个命令在 138 行都会先调用 _load_settings；该函数执行 repo.init_db 和 repo.apply_db_settings，初始化过程包含建表、默认值写入和迁移，健康检查/列群等只读操作可能修改生产数据库。 +- **运维与计划任务脚本** · `scripts/test_wechat_data.py:177` — main 无论 health、list-groups、resolve 或 fetch 返回什么状态都固定 return 0；数据源不可用、读取失败或空结果仍会被自动化环境视为成功，脚本没有把 Provider 状态映射为退出码。 + +### MED (65) + +- **前端界面** · `frontend/src/pages/v2/AIImages.tsx:376` — 图片再生状态通过每 2 秒或 5 秒 setInterval 轮询；请求失败在 388 行以 catch(() => undefined) 静默丢弃，既无用户错误状态也无退避，网络/API 异常时会持续轮询并隐藏失败。 +- **前端界面** · `frontend/src/pages/v2/Tasks.tsx:113` — 任务页先读取全部 runs，再对每个 run 调用 getRunDetail，形成无分页/批量接口保护的线性 N+1 请求扇出；历史记录增多时请求数和页面等待时间同步增长。 +- **前端界面** · `frontend/src/pages/v2/AIImages.tsx:47` — STATUS_LABELS、runKey、statusTone 等展示与身份逻辑在 AIImages 47/74/98、Archive 46/98/112、ChatRecords 26/48/52、Ranking 25/60/119、Tasks 29/47/51 多处重复，状态协议和标签容易出现页面间漂移。 +- **前端界面** · `frontend/src/pages/v2/Settings.tsx:28` — 前端手工维护 SENSITIVE_KEYS、BOOLEAN_KEYS、NUMBER_KEYS、LABELS、SETTING_GROUPS 多套设置契约；61-72 与 110-115 行还混有 V1 兼容邮件字段和 Provider fallback 字段，后端设置变更可能造成控件、类型或序列化不一致。 +- **前端界面** · `frontend/src/styles.css:1197` — 全局样式存在明确的‘旧页面兼容与窄屏布局’区域，3803 行开始仍有 archive-legacy 样式；5939 行单文件同时容纳新旧页面级联规则，增加跨页面回归和未使用 CSS 累积风险。 +- **前端 API 客户端** · `frontend/src/api.ts:3` — request 仅调用 fetch，没有 AbortSignal 超时、取消、重试或网络错误分类；AI 生图、刷新、发送等长任务请求可能长期悬挂，页面无法主动结束旧请求或恢复瞬时网络失败。 +- **前端 API 客户端** · `frontend/src/api.ts:8` — HTTP 错误只读取原始 response.text() 并构造普通 Error（8-11 行），没有统一解析后端结构化错误字段、error_type、状态码或可恢复性；调用页面只能按字符串处理失败。 +- **前端 API 客户端** · `frontend/src/api.ts:22` — 文件同时保留 V1 的 Group/Run/LatestReport 类型和接口，以及 65 行之后的 GroupV2/V2Run/Archive 与 V2 pipeline 接口；同一前端客户端维护两套后端协议，形成 dual-format/duplication。 +- **运行时与 HTTP API** · `app/main.py:29` — FastAPI lifespan 在 yield 前同步执行 WeChat、Codex、模板和本地环境检查；外部依赖不可用时可能拖慢服务启动，且 app/main.py:34-35 捕获 Exception 后直接置空 startup_checks，没有日志或失败原因。 +- **运行时与 HTTP API** · `app/main.py:48` — 所有设置、删除、生成、发送和日志路由均直接挂载，没有认证/授权依赖。默认 host 是本机回环，但一旦通过 APP_HOST 或容器暴露到网络，管理和发送接口即无身份边界。 +- **运行时与 HTTP API** · `app/api/settings.py:99` — 设置 API 接收任意字符串并先持久化；app/config/settings.py:163-166 对类型转换异常静默跳过，接口仍返回 ok=true，导致数据库值、运行时值和 UI 成功提示可能不一致。 +- **运行时与 HTTP API** · `app/config/settings.py:177` — 布尔配置遇到非标准字符串时执行 return bool(text)，例如 'falsee' 会被静默转换为 True；错误配置不会被拒绝，可能改变发送、邮件或 Mock Provider 行为。 +- **运行时与 HTTP API** · `app/api/system.py:92` — status 接口直接对可由设置 API 写入的 schedule_generate_time 执行 split/int，未捕获格式错误；非法配置会让状态接口返回 500，而调度器自身另有回退规则，形成配置行为分裂。 +- **运行时与 HTTP API** · `app/api/v2_ui.py:98` — Dashboard 用 display_name 作为 RunStore 的运行目录键；同一文件后续 archive_groups 使用稳定 group_id/wechat_group_id 匹配，而 retry_failed 在 app/api/v2_ui.py:724-735 又按 display_name 查询，改名或重名时可能显示 Pending、找不到任务或恢复错误群。 +- **运行时与 HTTP API** · `app/core/logging.py:68` — setup_logging 在 root.handlers 已存在时立即 return，导致 uvicorn/宿主已预配置 root handler 时不会执行 app/provider/ai/scheduler/email 文件 handler 配置，日志可能只进宿主输出而不进入声明的分类日志。 +- **运行时与 HTTP API** · `app/config/settings.py:30` — V1/V2 和主备 Provider 配置同时存在：history_provider_primary/fallback/mock_enabled、summary_provider_primary/fallback 与旧 ai_provider/ai_model/ai_api_key 并列；app/api/settings.py:24-75 又手工复制一套可编辑键，配置来源和行为边界容易漂移。 +- **Pipeline 与调度** · `app/scheduler/send_job.py:12` — run_send_due_job 在 14-19 行捕获所有 Exception，只记录日志且返回 None；APScheduler 调度层无法得到失败状态，分钟级发送异常可能表现为任务成功但无人感知。 +- **Pipeline 与调度** · `app/scheduler/daily_v2_job.py:232` — 邮件阶段只用 proc.returncode 判定 email_status=sent/failed（232-245 行），不解析子进程输出中的逐群结果或工件状态，存在子任务部分失败却被记录为 sent 的协议缺口。 +- **Pipeline 与调度** · `app/scheduler/generate_job.py:1` — generate_job.py 与 email_job.py 仍保留 V1 ReportService/EmailService 任务（generate_job.py:15-24、email_job.py:14-22），而 manager.py:44-65 只注册 V2 任务；两套调度/状态格式仍可被外部调用，形成 legacy/dual-format 维护面。 +- **Pipeline 与调度** · `app/scheduler/calendar_rules.py:11` — V1 仍定义 ReportWindow/get_report_window（11-35 行），V2 又定义独立 PeriodWindow/PeriodResolver（app/scheduler/period.py:14-57）；两套日期窗口模型实现相同的前一自然日逻辑，增加跨版本语义漂移和 duplication 风险。 +- **Pipeline 与调度** · `app/services/group_name_sync.py:81` — 健康检查、群列表读取异常或空结果在 81-94 行统一转为 unavailable，157-165 行只标记 skipped 并保留本地旧名称；DailyPipeline 在 126、682 行继续生成/发送，数据源不可用时可能继续使用过期自动发送目标。 +- **Pipeline 与调度** · `app/pipeline/daily_pipeline.py:205` — _record_group_failure 仅尽力写 FAILED；220-230 行再次捕获状态落盘异常并继续，导致返回结果显示 failed 但 run.json 可能没有失败状态，恢复与人工排查依赖日志而非持久化状态。 +- **微信数据接入** · `app/data_sources/base.py:20` — V2 定义 V2Message、FetchResult、DataSourceStatus、WeChatDataSource（20-115 行），V1 又定义 RawMessage、FetchResult、ProviderStatus、ChatHistoryProvider（app/providers/history/base.py:15-85）；V2 wrapper 在 app/data_sources/wechat_data_analysis.py:100-134 反复做两套模型转换，存在 dual-format/duplication 和额外 glue 层。 +- **微信数据接入** · `app/data_sources/wechat_data_analysis.py:82` — _group_exists 在 84-90 行捕获所有异常并直接返回 False；当上游 list_groups 临时失败且消息结果为空时，fetch_messages 在 114-123 行会把暂时不可用误判为 GROUP_NOT_FOUND，而不是数据源异常。 +- **微信数据接入** · `app/services/history_service.py:130` — discover_groups 在 133-143 行对 Provider 健康检查或 list_groups 的所有异常直接 continue 且不记录日志；所有 Provider 同时异常时调用方只得到空群列表，缺少可诊断的失败原因。 +- **微信数据接入** · `app/providers/history/contact_resolver.py:37` — find_contact_db 在 37-43 行按目录排序后返回第一个账号的 contact.db；WeChatDataAnalysisProvider 虽保存 wechat_mcp_account（app/providers/history/wechat_data_analysis.py:110），构造 ContactResolver 时并未按账号选择数据库（117-118 行），多账号环境可能把联系人显示名映射到错误账号。 +- **微信数据接入** · `app/providers/history/wechat_data_analysis.py:267` — JSON 导出路径在 284-287 行直接读取 item['timestamp']，并由 _to_raw 在 693 行直接读取 item['group_id']；单条导出记录缺字段或格式损坏会使整个群取数异常，没有逐条隔离或明确 INVALID_RESULT 状态。 +- **排行榜与统计** · `app/ranking/renderer.py:22-70; app/services/ranking_service.py:26-46` — 排行榜渲染存在两条路径：V2 使用模板变量渲染，V1 RankingResult.render 直接拼接固定文本。相同统计数据可能产生不同的标题、字段和格式，邮件/报告与 V2 ranking.txt 的展示契约不统一。 +- **排行榜与统计** · `app/ranking/template_service.py:17-20,64-75,105-110` — 默认模板同时以内嵌 DEFAULT_RANKING_TEMPLATE 和磁盘 default.txt 形式存在；_ensure_default 与 reset 都从代码常量写回文件。模板文件可被编辑而代码常量不会同步，恢复默认或重新初始化可能覆盖文件侧修改。 +- **排行榜与统计** · `app/ranking/engine_types.py:31-33; app/ranking/renderer.py:35-38,66-70` — V2 数据结构和渲染器保留多处旧协议兼容：top_limit 被追加到字段末尾以维持位置参数，top10_lines 作为旧变量别名，render_simple 继续保留无模板调用点。兼容层已进入核心类型/渲染路径，增加长期 dual-format/legacy 维护成本。 +- **排行榜与统计** · `app/ranking/engine.py:21-27,40-54; app/services/ranking_service.py:49-67` — 两套引擎的可计数判定来源不同：V2 自己检查 message_type 与 SYSTEM_KEYWORDS，V1 依赖 NormalizedMessage.countable；同一原始数据经过不同标准化链路时，消息数、发言人数和 Top 排名可能不一致。 +- **AI 摘要与生图提示词** · `app/providers/ai/deepseek.py:227-259` — _chat 将 retryable 初始设为 True，并在 except Exception 中保留可重试状态；JSON 解析异常、字段缺失、空内容、非瞬态网络/协议错误都会进入重试路径，未区分瞬态错误和确定性输入/响应错误。 +- **AI 摘要与生图提示词** · `app/providers/ai/codex.py:42-65,123-152` — CodexGPTProvider 继承 DeepSeekV4FlashProvider 却绕过父类 __init__，复用父类内部编排并自行覆盖关键方法；这是对实现细节的 over-fit 耦合，父类初始化或分块协议变化时容易产生隐蔽回归。 +- **AI 摘要与生图提示词** · `app/ai/prompt_templates.py:16-96,124-165; templates/image_prompt/default.md:1-98` — 默认 Prompt 模板存在两份来源：Python 内嵌 DEFAULT_IMAGE_PROMPT_TEMPLATE 与可编辑 default.md。_ensure_default 只在缺失时写入，reset 又把内嵌版本写回，两个文件可独立漂移，模板修改和恢复行为不一致。 +- **AI 摘要与生图提示词** · `app/ai/prompt_builder.py:97-124,393-447; app/ai/prompt_templates.py:17-96; app/providers/ai/deepseek.py:37-67; app/ai/layouts.py:222-233` — 事实约束、输出结构、主题/版式规则和禁止词分散在多个 Python 常量及 Markdown 模板中，存在重复 Prompt 契约；规则变更需要同步多处，容易出现主模型、备用模型、模板和最终检查器不一致。 +- **AI 摘要与生图提示词** · `app/providers/ai/base.py:19-20; app/providers/ai/deepseek.py:84-91; app/ai/prompt_builder.py:225-226,480-491,676-677` — 同时维护新 message_items 与旧 messages_text 两套输入格式，并通过 legacy-* 合成消息 ID；Builder 还保留旧类名 DeepSeekImagePromptBuilder 并导出 GroupSummaryImagePromptBuilder 别名，说明 V1/V2 兼容层仍渗透核心路径，存在 dual-format/legacy 维护成本。 +- **AI 摘要与生图提示词** · `app/ai/prompt_builder.py:453-455; app/ai/topic_selection.py:177-184` — 元数据和候选字段保留旧字段（deepseek_ms、interestingness_score）以兼容历史读取；代码虽能运行，但新旧协议长期并存，字段语义和版本边界不清晰。 +- **AI 摘要与生图提示词** · `app/ai/prompt_builder.py:415-447` — 最终 Prompt 约束主要依赖 forbidden term 的字符串包含检查和若干必需块的字符串包含检查，无法验证语义上的事实一致性、每段文字是否真正只出现一次或模型是否改写了证据；校验容易出现误报和漏报。 +- **图片生成与恢复** · `app/image/image_task.py:67` — verify_image 在 67-77 行只检查文件存在、大小和魔数签名，没有真正解码图片；app/image/regeneration.py:106-115 直接用该结果决定替换正式图片，损坏但带合法 PNG/JPEG 头的文件可能被接受。函数文档所称的可解析校验与实现不一致。 +- **图片生成与恢复** · `app/image/codex_generator.py:1` — 主链路使用 Codex CLI 自有 attempt manifest（文件头及 236-430 行），同时 scripts/codex_image_automation.py:1-18 保留 Desktop begin/adopt/verify marker 工作流；两套图片认领、候选发现和 run 状态同步协议并存，属于 legacy/dual-format duplication。 +- **图片生成与恢复** · `app/image/regeneration.py:46` — enqueue_regeneration 在 46-51 行先把任务 key 加入全局 _ACTIVE，随后 53-68 行才执行状态写入、生成器构造和线程提交；任一更新或 submit 异常都没有回滚 _ACTIVE，后续同一群/日期会永久被判定为正在队列中，直到进程重启。 +- **图片生成与恢复** · `app/image/image_task.py:169` — SerialImageQueue 在 169-173 行吞掉 run_hook 的全部异常并继续返回图片结果；如果 hook 的 run.json 更新失败，调用方仍可能看到成功图片但持久化状态未进入 IMAGE_READY/READY_TO_SEND。 +- **微信与邮件交付** · `app/services/email_service.py:156-243; scripts/send_daily_email.py:82-243` — 生产 EmailService 与手动脚本重复实现邮件构造、图片附件校验、SMTP 连接和重试逻辑；两套实现的降级语义不同（服务侧无效图片降级为纯排行榜，脚本侧直接跳过/失败），修复容易出现行为漂移。 +- **微信与邮件交付** · `app/sender/wechat_automation.py:131-143` — legacy CLI 的 JSON 解析失败时回退为 proc.returncode == 0；CLI 可能退出码为 0 但没有可验证成功响应，此处仍返回 success，存在把未确认发送报告为成功的风险。 +- **微信与邮件交付** · `app/sender/wechat_automation.py:104-117` — 日志将完整命令拼接输出，命令包含 --to 目标和 --content 消息正文；真实发送时群名、接收目标及部分聊天内容可能进入日志，属于敏感数据泄漏风险。 +- **微信与邮件交付** · `app/sender/wechat_native.py:37-745` — WindowsWechatDriver 集中实现 OCR、窗口激活、坐标点击、剪贴板、图片转换、提交后截图差异验证和发送互斥，文件 868 行、约 55 个定义，属于高复杂度 god-component；桌面版本、DPI 或 OCR 变化时 Blast Radius 很大。 +- **微信与邮件交付** · `app/sender/wechat_automation.py:7-13,31-56; app/sender/wechat_native.py:862-868; scripts/test_wechat_send.py:23-25,68-76` — 旧 wechat-automation 路径文档已明确当前微信版本 UIA 不兼容并会返回 WECHAT_WINDOW_NOT_FOUND，但配置为 legacy_cli 时仍会选择该 Provider，手动真实发送测试脚本也固定使用它；存在 legacy/stub 路径继续被误用的风险。 +- **V1 报告兼容链** · `app/services/handoff_service.py:116` — handoff.json 通过 json.loads 无保护读取；save_outputs 在 57-92 行顺序写入多个文件且非原子，进程中断或部分文件损坏时，归档列表接口会直接抛异常而不是返回明确的不完整状态。 +- **V1 报告兼容链** · `scripts/run_daily_pipeline.py:28` — _pipeline 每次都执行 repo.init_db 和 repo.apply_db_settings；status 分支在 83-89 行也经过该初始化路径，因此看似只读的状态查询可能创建或修改数据库设置。 +- **V1 报告兼容链** · `app/services/report_service.py:1` — ReportService 明确是 V1 生成链路，而 scripts/run_daily_pipeline.py:1 使用独立 V2 DailyPipeline；handoff_service.py:11-12 又同时维护 V1/V2 状态语义。两套生成、状态和输出协议并存，后续修复容易发生 legacy/dual-format 漂移。 +- **V1 报告兼容链** · `scripts/run_daily_pipeline.py:72` — send 命令提供 --dry-run，但 force-send 子命令没有该选项；91 行通过 getattr 缺省为 False，因此 force-send 无法在 CLI 层进行 dry-run，误调用会直接进入真实发送路径。 +- **数据库与运行状态** · `app/db/repository.py:34-184` — 迁移由多个手写 _migrate_* 函数组成，重复使用 marker 查询、更新、插入和 commit；没有版本表、统一迁移注册或全局迁移事务，属于明显 legacy/bloat/duplication。 +- **数据库与运行状态** · `app/db/repository.py:187-206` — init_db 每次启动按固定顺序执行 create_all、补列、种默认值和多轮数据迁移；多个进程并发启动或中途异常时可能出现部分迁移，且旧 schema 默认值与模型默认值容易漂移。 +- **数据库与运行状态** · `app/db/repository.py:331-425` — Repository 的 save_group、delete_group、restore_group、create_run、save_report、set_setting_value 等操作都立即 commit，缺少跨实体事务边界；核心流程的多阶段写入无法由该层保证原子性。 +- **数据库与运行状态** · `app/v2/run_store.py:164-181` — 普通 save_run/update 只使用进程内 RLock，临时文件名固定为 run.json.tmp；跨进程写同一 run 时仍可能互相覆盖临时文件或丢失更新，跨进程互斥只在发送 claim 路径启用。 +- **数据库与运行状态** · `app/v2/recovery.py:85-105` — verify_output 只检查文件存在且 size>0，不校验 JSON schema、内容可解析性、图片实际解码或状态字段一致性；损坏文件可能被判定为完整。 +- **数据库与运行状态** · `app/v2/recovery.py:113-153` — recover_incomplete 通过 display_name 查询 group_id，而不是使用持久化稳定 ID；重名群组时可能恢复到错误群组，且动态导入 DailyPipeline/DB，形成较强 glue 耦合。 +- **数据库与运行状态** · `app/v2/run_store.py:119-121` — 运行目录以 safe_dir_name(group_name)/run_date 定位，没有稳定 group_id；名称清洗、特殊字符归一化或截断会造成不同群组目录碰撞并覆盖状态/输出。 +- **数据库与运行状态** · `app/v2/constants.py:19-32` — STATUS_FLOW 只声明状态顺序，is_terminal 只做终态集合判断，没有统一的合法状态转移校验；状态推进规则仍分散在上层 pipeline，状态机约定容易被绕过。 +- **数据库与运行状态** · `app/v2/run_store.py:113-365` — RunStore 同时承担路径生成、JSON 状态读写、历史布局读取、跨进程发送锁、发送租约和运行枚举，职责集中度偏高，已接近 persistence god-component；后续修改发送状态时容易影响文件状态逻辑。 +- **运维与计划任务脚本** · `scripts/daily_auto.py:69` — already_running 被纳入成功退出集合。任务实际没有执行任何生成或发送时仍返回 0，计划任务监控无法区分‘已完成’和‘被并发锁跳过’。 +- **运维与计划任务脚本** · `scripts/install_daily_task.py:114` — 安装、卸载和状态命令的 main 固定 return 0；_install 即使 schtasks 返回失败也只打印字符串，调用方无法通过退出码发现任务安装失败。install_autostart.py:64-74 存在同样问题。 +- **运维与计划任务脚本** · `scripts/install_daily_task.py:44` — 两阶段安装先创建生成任务，再创建发送任务；发送任务创建失败时 55-71 行只返回警告，不回滚已创建的生成任务，系统会留下半安装状态。 +- **运维与计划任务脚本** · `scripts/test_wechat_data.py:45` — fetch 结果会把完整 messages、消息详情和 Provider 错误写入 output/test-data，固定文件名会覆盖旧证据，且没有脱敏、保留期限或显式清理边界；真实聊天验证可能留下敏感数据。 +- **运维与计划任务脚本** · `scripts/install_daily_task.py:35` — 计划任务的生成阶段调用 daily_auto.py，发送阶段调用另一套 run_daily_pipeline.py；两个入口拥有不同的状态文件、失败语义和退出码，调度链路存在 dual-format/glue 漂移风险。 + +### LOW (31) + +- **前端界面** · `frontend/src/components/ui.tsx:39` — useFetch 接收 caller 自行提供的 unknown[] 依赖，内部却省略 loader 依赖并在 67 行关闭 exhaustive-deps 检查；未来传入捕获筛选条件的 inline loader 时可能保留旧闭包，且失去静态提示。 +- **前端界面** · `frontend/src/pages/v2/History.tsx:3` — History 仅包装 Archive，System.tsx:3 仅包装 Settings；navigation.ts:61-66 已将 history/system 归一到 archive/settings，App.tsx 也直接渲染 Archive/Settings，两个 wrapper 当前未被引用，属于 legacy/stub 死代码候选。 +- **前端界面** · `frontend/src/components/layout/AppShell.tsx:108` — 通知按钮只有图标和 aria-label，没有 onClick、状态或目标路由，当前是可点击外观但无行为的 placeholder UI。 +- **前端 API 客户端** · `frontend/src/api.ts:367` — readV2TextFile 直接重新实现 fetch、状态码检查和错误文本处理，绕过 3-16 行的 request 通用封装；它与其他请求的 headers、错误格式和未来超时策略容易发生漂移。 +- **前端 API 客户端** · `frontend/src/api.ts:146` — V2Run 使用 [key: string]: unknown 放开任意字段，削弱 run 状态、发送字段和恢复字段的静态类型约束；调用方访问动态字段只能运行时判断。 +- **前端 API 客户端** · `frontend/src/api.ts:364` — getV2File 只对 group 做 encodeURIComponent，date/file 直接拼接 URL（364-365 行）；当前日期和白名单文件名通常安全，但通用客户端边界没有统一的路径参数编码/校验策略。 +- **运行时与 HTTP API** · `app/api/system.py:58` — 旧 API stats 只接受 Run.status 为 success/partial；V2 运行状态使用独立的大写状态协议并由 /api/v2/dashboard 读取，V1/V2 双格式并存，旧统计接口可能长期返回空或过期数据。 +- **运行时与 HTTP API** · `app/api/v2_ui.py:759` — pipeline/generate、pipeline/send-due、pipeline/send 在 HTTP 请求内同步构造 DailyPipeline 并执行长耗时取数、AI、生图或桌面发送，API 层承担任务编排和外部副作用，易阻塞请求线程并扩大重复点击的影响面。 +- **Pipeline 与调度** · `app/scheduler/manager.py:23` — 无效 schedule_generate_time 在 23-33 行静默回退到 00:15；虽然有 warning，但配置错误不会阻止启动，运行顺序依赖日志才能发现实际调度时间。 +- **Pipeline 与调度** · `app/scheduler/period.py:45` — PeriodResolver 对除 weekday_default 外的规则直接抛出 NotImplementedError（45-47 行），而 Group.schedule_rule 已暴露为可配置字段，属于已接入界面但未完成执行语义的 placeholder。 +- **微信数据接入** · `app/providers/history/wechat_cli.py:65` — wechat-cli 的 list-groups 在 65-80 行对非零退出、JSON 解析失败和任意异常均返回空列表；作为 fallback 时会把命令故障表现为无群，且没有保留错误上下文。 +- **微信数据接入** · `app/providers/history/base.py:68` — ChatHistoryProvider 的接口方法在 73-85 行仅抛出 NotImplementedError，属于未使用 ABC 的 placeholder 接口；目前不影响已接入实现，但错误实例化基类时才会暴露，类型边界约束较弱。 +- **排行榜与统计** · `app/ranking/engine.py:29-38; app/ranking/engine_types.py:23-43` — V2 支持可配置 top_limit，但只校验大于 0；没有上限范围或非整数输入边界，结果结构也不校验 rank/count 非负。当前调用方通常传固定值，属于低影响输入约束缺口。 +- **排行榜与统计** · `app/ranking/renderer.py:41-45` — 模板替换对无法匹配正则的占位符保持原文，对未知/畸形占位符没有渲染后残留检查；虽然 validate_template 能拦截常规未知变量，但最终输出仍可能含未替换模板标记。 +- **排行榜与统计** · `app/ranking/renderer.py:66-70` — render_simple 通过函数内动态导入 DEFAULT_RANKING_TEMPLATE 保留旧调用点，属于 glue 兼容实现；未直接造成错误，但继续扩大了 V1/V2 渲染边界。 +- **排行榜与统计** · `app/ranking/engine.py:1-7,46-64` — 排行榜核心本身是确定性、无 LLM、按消息数和稳定名称排序的纯计算路径，职责清晰且复杂度低；主要问题集中在 V1/V2 双实现与输出协议分裂，而非算法本身。 +- **AI 摘要与生图提示词** · `app/ai/prompt_builder.py:456-461` — build 使用宽泛 except Exception 统一记录并返回失败结果；异常不会静默丢失，但错误类型边界被压平，调用方只能依据截断后的字符串判断 Provider、模板还是验证失败。 +- **AI 摘要与生图提示词** · `app/ai/image_themes.py:351-368; app/ai/prompt_builder_types.py:9,21,30; app/ai/layouts.py:13` — 多个核心数据入口使用 Any（主题 rng、消息列表、持久化元数据、布局定义），降低了 Prompt 协议的静态约束，属于低影响 any-escape。 +- **AI 摘要与生图提示词** · `app/ai/layouts.py:478-547; app/ai/prompt_builder.py:612-622` — 版式导演失败两次后使用确定性 fallback_layout_plan；该回退不编造聊天事实且有稳定 seed，属于有界安全兼容，但会掩盖模型版式能力或协议问题并降低输出丰富度。 +- **图片生成与恢复** · `app/image/regeneration.py:38` — 重新生图入口的 generator 参数使用 Any，app/image/image_task.py:94 也以 Any 接收生成器，没有统一协议或结构化能力检查；生成器异常只能在运行时被捕获，属于 glue/any-escape。 +- **图片生成与恢复** · `app/image/codex_generator.py:865` — _load_last_smoke 在 865-872 行只要 smoke JSON 能解析就返回 ok=True，不验证记录中的 source/output 文件仍存在；健康页可能继续展示过期的最近实测成功状态。 +- **微信与邮件交付** · `scripts/send_daily_email.py:132-140,246-278` — 手动邮件脚本把未验证的 run_date 直接拼接到 output/{group}/{run_date} 路径；run_date 没有 YYYY-MM-DD 校验，虽然通常由本机操作者输入，仍缺少与 V2 RunStore 一致的路径边界。 +- **微信与邮件交付** · `app/sender/base.py:41-46; app/sender/wechat_native.py:823-857` — 文字和图片按顺序提交，不具备原子 bundle；文字成功后图片可能失败或结果未知，调用方必须正确处理部分成功。SendResult 已保留 submitted、verification_level 和 outcome_unknown 字段，这是当前较好的 fail-closed 兼容设计。 +- **微信与邮件交付** · `scripts/test_wechat_send.py:60-66; app/sender/wechat_automation.py:96-102` — dry-run 测试把 ranking_demo_group_2026-08-17.txt 作为图片路径，而 legacy send_image 只检查文件存在、不验证图片格式；测试可能显示‘图片发送’成功但实际输入不是图片。 +- **V1 报告兼容链** · `app/services/prompt_service.py:89` — _build_context_text 明确标注为兼容旧调用，ranking 参数未使用，且仓库内没有实际调用；它与 _build_message_items/新 provider message_items 路径重复构造完整上下文，属于 legacy 兼容代码候选。 +- **数据库与运行状态** · `app/v2/run_store.py:339-364` — list_runs 对损坏 run.json 直接静默跳过，调用方无法区分“没有运行”和“状态文件损坏”，导致恢复/审计列表不完整。 +- **数据库与运行状态** · `app/v2/recovery.py:38-52` — 旧 run 缺少 image_enabled 或值未知时固定回退为 True；这是兼容性 fallback，但会把旧格式或异常数据强制解释为需要图片。 +- **数据库与运行状态** · `app/db/repository.py:13,428-431` — 全局 engine 使用 Any，_now 返回 Any，削弱了 persistence 边界的静态类型约束，属于低影响 any-escape。 +- **运维与计划任务脚本** · `scripts/generate_fixtures.py:170` — 生成器按当前 date.today() 滚动生成最近 8 天 fixtures，并在 171-183 行直接覆盖 groups.json 和消息文件；没有 --output、日期或确认参数，容易覆盖手工 fixture，且输出结果随执行日期变化。 +- **运维与计划任务脚本** · `scripts/generate_fixtures.py:24` — groups.json 声明 member_count 为 48/112，但实际生成的 MEMBERS_A/MEMBERS_B 只有 15/8 人；这是 mock_fixture 的不一致数据契约，可能让 UI 或测试展示错误人数。 +- **运维与计划任务脚本** · `scripts/install_autostart.py:24` — 开机自启注释仍称启动 V1 脚本，_START_CMD 通过 start_windows.bat 启动旧入口；该兼容包装与当前 V2 调度入口并存，版本边界和实际启动流程不清晰。 + +## Cross-cutting themes + +- **V1/V2 双协议是最广泛的系统性债务.** 运行状态、持久化、排行榜、Prompt、文件 API、调度与前端契约都保留两套可触达路径；它解释了项目为何兼容旧数据，也放大了状态分叉和维护成本。 +- **数据约束债务已经产生真实后果.** SQLite 关系主要靠应用代码维持，当前只读检查发现 224 条 group_runs 中 192 条已失去群组关联；缺少 FK/唯一约束不再只是理论风险。 +- **复杂度集中在少数高 Blast Radius 模块.** DailyPipeline、V2 API、PromptBuilder、CodexImageGenerator、WDA Provider、WechatNativeSender 与 AIImages 页面同时承担多个阶段，任何局部变化都需要跨链路回归。 +- **失败安全已有正确骨架，但状态损坏仍会被误解释.** V2 的发送 claim、result_unknown、手工 hold 和图片 staging 值得保留；相反，损坏 run.json/scheduler JSON 的静默重置可能把未知结果当成未开始。 +- **配置存在可保存但未进入真实执行链的漂移.** 群级 schedule_rule、summary_model、prompt_model、provider_preference 已进入模型与 UI，但主 Pipeline 没有完整消费；设置 API 的宽松转换又可能造成数据库、运行时和 UI 三方不一致。 +- **外部副作用的幂等边界不一致.** 微信 V2 发送偏向 fail-closed，但 SMTP 重试、Legacy CLI 退出码、人工图片 adopt 与部分运维脚本仍可能重复执行、误报成功或覆盖状态。 diff --git a/.codemap/config.json b/.codemap/config.json new file mode 100644 index 0000000..2960606 --- /dev/null +++ b/.codemap/config.json @@ -0,0 +1,8 @@ +{ + "lang": "zh", + "project": "GroupBrief V1", + "subtitle": "本地微信群报生成与安全发送系统工程体检", + "outputDir": ".codemap", + "htmlFile": "codemap.html", + "mdFile": "codemap.md" +} diff --git a/.codemap/modules.json b/.codemap/modules.json new file mode 100644 index 0000000..a011444 --- /dev/null +++ b/.codemap/modules.json @@ -0,0 +1,1225 @@ +{ + "meta": { + "project": "GroupBrief V1", + "lang": "zh", + "subtitle": "本地微信群报生成与安全发送系统工程体检", + "generatedAt": "2026-08-24", + "htmlPath": ".codemap/codemap.html", + "mdPath": ".codemap/codemap.md", + "spineDesc": "管理界面发起或调度器触发任务，经 API/Pipeline 读取群配置和微信数据，生成排行榜、提示词与图片，以 run.json/SQLite 持久化状态，最后在安全认领后发送。", + "tracked_loc": 28149, + "tracked_files": 116, + "rev": "cf29e2decf5ae74e5a7037531edb0218dc4bdcea" + }, + "excludes": [ + "__pycache__", + "/node_modules/", + "/frontend/dist/", + "/frontend/.vite/", + "/.git/", + "/.venv/", + "/data/", + "/output/", + "/logs/", + "/fixtures/", + "/tests/", + "/.codemap/" + ], + "bands": [ + { + "id": "frontend", + "tier": "fe", + "t": "前端 · 管理界面", + "d": "页面、交互状态与 API 客户端" + }, + { + "id": "wire", + "wire": true, + "t": "HTTP / JSON" + }, + { + "id": "entry", + "tier": "be", + "t": "后端 · 入口与 API", + "d": "FastAPI 生命周期、配置、路由" + }, + { + "id": "orchestration", + "tier": "be", + "t": "编排 · 调度与状态机", + "d": "每日生成、恢复、发送窗口" + }, + { + "id": "domain", + "tier": "be", + "t": "领域 · 数据到内容", + "d": "微信数据、排行、AI、图片、交付" + }, + { + "id": "storage", + "tier": "db", + "t": "持久化 · SQLite 与工件", + "d": "关系数据、run.json、恢复状态" + }, + { + "id": "ops", + "tier": "ops", + "t": "运维 · 脚本", + "d": "Windows 计划任务、手工检查与恢复入口" + } + ], + "spine": [ + "frontend_ui", + "frontend_api", + "runtime_api", + "pipeline_scheduler", + "history_ingestion", + "ranking", + "ai_prompting", + "image_generation", + "persistence_state", + "delivery" + ], + "reportThemes": [ + [ + "V1/V2 双协议是最广泛的系统性债务", + "运行状态、持久化、排行榜、Prompt、文件 API、调度与前端契约都保留两套可触达路径；它解释了项目为何兼容旧数据，也放大了状态分叉和维护成本。" + ], + [ + "数据约束债务已经产生真实后果", + "SQLite 关系主要靠应用代码维持，当前只读检查发现 224 条 group_runs 中 192 条已失去群组关联；缺少 FK/唯一约束不再只是理论风险。" + ], + [ + "复杂度集中在少数高 Blast Radius 模块", + "DailyPipeline、V2 API、PromptBuilder、CodexImageGenerator、WDA Provider、WechatNativeSender 与 AIImages 页面同时承担多个阶段，任何局部变化都需要跨链路回归。" + ], + [ + "失败安全已有正确骨架，但状态损坏仍会被误解释", + "V2 的发送 claim、result_unknown、手工 hold 和图片 staging 值得保留；相反，损坏 run.json/scheduler JSON 的静默重置可能把未知结果当成未开始。" + ], + [ + "配置存在可保存但未进入真实执行链的漂移", + "群级 schedule_rule、summary_model、prompt_model、provider_preference 已进入模型与 UI，但主 Pipeline 没有完整消费；设置 API 的宽松转换又可能造成数据库、运行时和 UI 三方不一致。" + ], + [ + "外部副作用的幂等边界不一致", + "微信 V2 发送偏向 fail-closed，但 SMTP 重试、Legacy CLI 退出码、人工图片 adopt 与部分运维脚本仍可能重复执行、误报成功或覆盖状态。" + ] + ], + "modules": [ + { + "id": "frontend_ui", + "label": "前端界面", + "band": "frontend", + "path": "frontend/src/**/*.tsx + styles.css", + "paths": [ + "frontend/src/**/*.tsx", + "frontend/src/styles.css", + "frontend/src/navigation.ts" + ], + "coupling": "high", + "deps": [ + "frontend_api" + ], + "desc": "React 管理界面、页面状态、轮询、图片查看与操作入口。", + "tests": [], + "loc": 10683, + "contentHash": "d9c0dc23e2932bdececb0f7a3aad82bf53f2d7c32f6cffb9a47f6ed2590c7572", + "score": 62, + "grade": "C", + "tags": [ + "god-component", + "silent-except", + "fallback", + "duplication", + "dual-format", + "legacy", + "bloat", + "glue", + "stub", + "placeholder" + ], + "findings": [ + { + "sev": "HIGH", + "loc": "frontend/src/pages/v2/AIImages.tsx:130", + "text": "单一 AIImages 组件从 130 行延伸至文件末尾约 696 行，132-177 行集中维护约 40 个 state/ref，同时承载运行列表、群级与运行级 Prompt、主题解析、图片再生、恢复、发送确认和轮询，是核心流程 God Component。" + }, + { + "sev": "MED", + "loc": "frontend/src/pages/v2/AIImages.tsx:376", + "text": "图片再生状态通过每 2 秒或 5 秒 setInterval 轮询；请求失败在 388 行以 catch(() => undefined) 静默丢弃，既无用户错误状态也无退避，网络/API 异常时会持续轮询并隐藏失败。" + }, + { + "sev": "MED", + "loc": "frontend/src/pages/v2/Tasks.tsx:113", + "text": "任务页先读取全部 runs，再对每个 run 调用 getRunDetail，形成无分页/批量接口保护的线性 N+1 请求扇出；历史记录增多时请求数和页面等待时间同步增长。" + }, + { + "sev": "MED", + "loc": "frontend/src/pages/v2/AIImages.tsx:47", + "text": "STATUS_LABELS、runKey、statusTone 等展示与身份逻辑在 AIImages 47/74/98、Archive 46/98/112、ChatRecords 26/48/52、Ranking 25/60/119、Tasks 29/47/51 多处重复，状态协议和标签容易出现页面间漂移。" + }, + { + "sev": "MED", + "loc": "frontend/src/pages/v2/Settings.tsx:28", + "text": "前端手工维护 SENSITIVE_KEYS、BOOLEAN_KEYS、NUMBER_KEYS、LABELS、SETTING_GROUPS 多套设置契约；61-72 与 110-115 行还混有 V1 兼容邮件字段和 Provider fallback 字段，后端设置变更可能造成控件、类型或序列化不一致。" + }, + { + "sev": "MED", + "loc": "frontend/src/styles.css:1197", + "text": "全局样式存在明确的‘旧页面兼容与窄屏布局’区域，3803 行开始仍有 archive-legacy 样式；5939 行单文件同时容纳新旧页面级联规则，增加跨页面回归和未使用 CSS 累积风险。" + }, + { + "sev": "LOW", + "loc": "frontend/src/components/ui.tsx:39", + "text": "useFetch 接收 caller 自行提供的 unknown[] 依赖，内部却省略 loader 依赖并在 67 行关闭 exhaustive-deps 检查；未来传入捕获筛选条件的 inline loader 时可能保留旧闭包，且失去静态提示。" + }, + { + "sev": "LOW", + "loc": "frontend/src/pages/v2/History.tsx:3", + "text": "History 仅包装 Archive，System.tsx:3 仅包装 Settings；navigation.ts:61-66 已将 history/system 归一到 archive/settings，App.tsx 也直接渲染 Archive/Settings，两个 wrapper 当前未被引用，属于 legacy/stub 死代码候选。" + }, + { + "sev": "LOW", + "loc": "frontend/src/components/layout/AppShell.tsx:108", + "text": "通知按钮只有图标和 aria-label，没有 onClick、状态或目标路由，当前是可点击外观但无行为的 placeholder UI。" + } + ], + "auditedHash": "d9c0dc23e2932bdececb0f7a3aad82bf53f2d7c32f6cffb9a47f6ed2590c7572", + "auditedAt": "2026-08-24", + "auditedRev": "1d326e2" + }, + { + "id": "frontend_api", + "label": "前端 API 客户端", + "band": "frontend", + "path": "frontend/src/api.ts", + "paths": [ + "frontend/src/api.ts" + ], + "coupling": "high", + "deps": [ + "runtime_api" + ], + "desc": "前端全部 HTTP 类型、请求封装与后端契约集中点。", + "tests": [], + "loc": 427, + "contentHash": "338a3903e576b08ebcd2283087254f9637bc75c09aab03c6871370edaa07832c", + "score": 72, + "grade": "C", + "tags": [ + "dual-format", + "duplication", + "glue", + "any-escape" + ], + "findings": [ + { + "sev": "MED", + "loc": "frontend/src/api.ts:3", + "text": "request 仅调用 fetch，没有 AbortSignal 超时、取消、重试或网络错误分类；AI 生图、刷新、发送等长任务请求可能长期悬挂，页面无法主动结束旧请求或恢复瞬时网络失败。" + }, + { + "sev": "MED", + "loc": "frontend/src/api.ts:8", + "text": "HTTP 错误只读取原始 response.text() 并构造普通 Error（8-11 行），没有统一解析后端结构化错误字段、error_type、状态码或可恢复性；调用页面只能按字符串处理失败。" + }, + { + "sev": "MED", + "loc": "frontend/src/api.ts:22", + "text": "文件同时保留 V1 的 Group/Run/LatestReport 类型和接口，以及 65 行之后的 GroupV2/V2Run/Archive 与 V2 pipeline 接口；同一前端客户端维护两套后端协议，形成 dual-format/duplication。" + }, + { + "sev": "LOW", + "loc": "frontend/src/api.ts:367", + "text": "readV2TextFile 直接重新实现 fetch、状态码检查和错误文本处理，绕过 3-16 行的 request 通用封装；它与其他请求的 headers、错误格式和未来超时策略容易发生漂移。" + }, + { + "sev": "LOW", + "loc": "frontend/src/api.ts:146", + "text": "V2Run 使用 [key: string]: unknown 放开任意字段，削弱 run 状态、发送字段和恢复字段的静态类型约束；调用方访问动态字段只能运行时判断。" + }, + { + "sev": "LOW", + "loc": "frontend/src/api.ts:364", + "text": "getV2File 只对 group 做 encodeURIComponent，date/file 直接拼接 URL（364-365 行）；当前日期和白名单文件名通常安全，但通用客户端边界没有统一的路径参数编码/校验策略。" + } + ], + "auditedHash": "338a3903e576b08ebcd2283087254f9637bc75c09aab03c6871370edaa07832c", + "auditedAt": "2026-08-24", + "auditedRev": "1d326e2" + }, + { + "id": "runtime_api", + "label": "运行时与 HTTP API", + "band": "entry", + "path": "app/main.py, app/config, app/core, app/api", + "paths": [ + "app/main.py", + "app/config/**/*.py", + "app/core/**/*.py", + "app/api/**/*.py" + ], + "coupling": "core", + "deps": [ + "pipeline_scheduler", + "persistence_state", + "v1_reporting", + "history_ingestion", + "ai_prompting", + "delivery" + ], + "desc": "FastAPI 生命周期、设置加载、健康检查、V1/V2 路由与文件接口。", + "tests": [ + "tests/test_ui_api.py", + "tests/test_v2_archive_api.py", + "tests/test_v2_group_prompt_api.py" + ], + "loc": 2419, + "contentHash": "855c2a0150a138f15c34bfe3e6c4b76562f02873cf3810546b7f7ed632c8a5d0", + "score": 58, + "grade": "D", + "tags": [ + "god-component", + "bloat", + "silent-except", + "fallback", + "legacy", + "dual-format", + "duplication", + "fake-output", + "glue" + ], + "findings": [ + { + "sev": "HIGH", + "loc": "app/api/files.py:25", + "text": "路径安全检查使用字符串 startswith；report_date、group_dir、filename 均未规范化。可用 sibling-prefix 绕过目录边界，例如 output/2026-08-1/../2026-08-10/ranking.txt 解析后仍满足 startswith，存在跨日期目录读取风险。" + }, + { + "sev": "HIGH", + "loc": "app/api/v2_ui.py:84", + "text": "单文件 833 行同时承担 Dashboard、归档聚合、Prompt 编辑、图片重生成、恢复、健康检查、Pipeline 控制和文件读取等多个边界，包含约 20 个路由与大量内嵌业务逻辑，属于高耦合 God Component。" + }, + { + "sev": "MED", + "loc": "app/main.py:29", + "text": "FastAPI lifespan 在 yield 前同步执行 WeChat、Codex、模板和本地环境检查；外部依赖不可用时可能拖慢服务启动，且 app/main.py:34-35 捕获 Exception 后直接置空 startup_checks，没有日志或失败原因。" + }, + { + "sev": "MED", + "loc": "app/main.py:48", + "text": "所有设置、删除、生成、发送和日志路由均直接挂载，没有认证/授权依赖。默认 host 是本机回环，但一旦通过 APP_HOST 或容器暴露到网络，管理和发送接口即无身份边界。" + }, + { + "sev": "MED", + "loc": "app/api/settings.py:99", + "text": "设置 API 接收任意字符串并先持久化；app/config/settings.py:163-166 对类型转换异常静默跳过，接口仍返回 ok=true，导致数据库值、运行时值和 UI 成功提示可能不一致。" + }, + { + "sev": "MED", + "loc": "app/config/settings.py:177", + "text": "布尔配置遇到非标准字符串时执行 return bool(text)，例如 'falsee' 会被静默转换为 True；错误配置不会被拒绝，可能改变发送、邮件或 Mock Provider 行为。" + }, + { + "sev": "MED", + "loc": "app/api/system.py:92", + "text": "status 接口直接对可由设置 API 写入的 schedule_generate_time 执行 split/int，未捕获格式错误；非法配置会让状态接口返回 500，而调度器自身另有回退规则，形成配置行为分裂。" + }, + { + "sev": "MED", + "loc": "app/api/v2_ui.py:98", + "text": "Dashboard 用 display_name 作为 RunStore 的运行目录键；同一文件后续 archive_groups 使用稳定 group_id/wechat_group_id 匹配，而 retry_failed 在 app/api/v2_ui.py:724-735 又按 display_name 查询，改名或重名时可能显示 Pending、找不到任务或恢复错误群。" + }, + { + "sev": "MED", + "loc": "app/core/logging.py:68", + "text": "setup_logging 在 root.handlers 已存在时立即 return，导致 uvicorn/宿主已预配置 root handler 时不会执行 app/provider/ai/scheduler/email 文件 handler 配置，日志可能只进宿主输出而不进入声明的分类日志。" + }, + { + "sev": "MED", + "loc": "app/config/settings.py:30", + "text": "V1/V2 和主备 Provider 配置同时存在：history_provider_primary/fallback/mock_enabled、summary_provider_primary/fallback 与旧 ai_provider/ai_model/ai_api_key 并列；app/api/settings.py:24-75 又手工复制一套可编辑键，配置来源和行为边界容易漂移。" + }, + { + "sev": "LOW", + "loc": "app/api/system.py:58", + "text": "旧 API stats 只接受 Run.status 为 success/partial；V2 运行状态使用独立的大写状态协议并由 /api/v2/dashboard 读取，V1/V2 双格式并存，旧统计接口可能长期返回空或过期数据。" + }, + { + "sev": "LOW", + "loc": "app/api/v2_ui.py:759", + "text": "pipeline/generate、pipeline/send-due、pipeline/send 在 HTTP 请求内同步构造 DailyPipeline 并执行长耗时取数、AI、生图或桌面发送，API 层承担任务编排和外部副作用，易阻塞请求线程并扩大重复点击的影响面。" + } + ], + "auditedHash": "855c2a0150a138f15c34bfe3e6c4b76562f02873cf3810546b7f7ed632c8a5d0", + "auditedAt": "2026-08-24", + "auditedRev": "1d326e2" + }, + { + "id": "pipeline_scheduler", + "label": "Pipeline 与调度", + "band": "orchestration", + "path": "app/pipeline, app/scheduler, generation runtime", + "paths": [ + "app/pipeline/**/*.py", + "app/scheduler/**/*.py", + "app/services/generation_runtime.py", + "app/services/group_name_sync.py" + ], + "coupling": "core", + "deps": [ + "persistence_state", + "history_ingestion", + "ranking", + "ai_prompting", + "image_generation", + "delivery" + ], + "desc": "每日任务编排、并发隔离、阶段状态、调度恢复与发送窗口。", + "tests": [ + "tests/test_v2_pipeline.py", + "tests/test_scheduler.py", + "tests/test_daily_auto.py", + "tests/test_generation_concurrency.py" + ], + "loc": 2160, + "contentHash": "27298f50129acd93b946ff31d176f3912e39faea0f52e686daf041051ff5439d", + "score": 54, + "grade": "D", + "tags": [ + "god-component", + "bloat", + "glue", + "fallback", + "silent-except", + "legacy", + "dual-format", + "duplication", + "placeholder" + ], + "findings": [ + { + "sev": "HIGH", + "loc": "app/pipeline/daily_pipeline.py:72", + "text": "DailyPipeline 文件共约 1358 行，单类同时编排配置、群名同步、数据源、排行榜、Prompt、图片生成、RunStore、发送、恢复与人工确认；构造函数在 85-95 行直接连接多个外部子系统，职责和 blast radius 过大。" + }, + { + "sev": "HIGH", + "loc": "app/pipeline/daily_pipeline.py:124", + "text": "PeriodResolver 支持 schedule_rule（app/scheduler/period.py:35-47），但 generate_all、force_generate、rebuild_prompt_from_snapshot 均未传入 Group.schedule_rule（本行及 1045、1115 行），群级周期配置会被静默按 weekday_default 执行。" + }, + { + "sev": "HIGH", + "loc": "app/pipeline/daily_pipeline.py:1304", + "text": "_save_json 在 1304-1306 行直接 write_text；messages/ranking 等工件与 run.json 分步写入。加载损坏快照时 1310-1313 行直接失败且明确不回源，进程中断可能留下不可恢复的半成品状态。" + }, + { + "sev": "HIGH", + "loc": "app/scheduler/daily_v2_job.py:40", + "text": "DailyScheduleState.load 在 42-48 行将 OSError/JSONDecodeError 直接吞掉并返回仅含 run_date 的新状态；若已有状态文件损坏，_run_locked 可能把已完成的邮件阶段当成未开始，失去去重依据并触发重复外部发送。" + }, + { + "sev": "MED", + "loc": "app/scheduler/send_job.py:12", + "text": "run_send_due_job 在 14-19 行捕获所有 Exception，只记录日志且返回 None；APScheduler 调度层无法得到失败状态，分钟级发送异常可能表现为任务成功但无人感知。" + }, + { + "sev": "MED", + "loc": "app/scheduler/daily_v2_job.py:232", + "text": "邮件阶段只用 proc.returncode 判定 email_status=sent/failed（232-245 行），不解析子进程输出中的逐群结果或工件状态，存在子任务部分失败却被记录为 sent 的协议缺口。" + }, + { + "sev": "MED", + "loc": "app/scheduler/generate_job.py:1", + "text": "generate_job.py 与 email_job.py 仍保留 V1 ReportService/EmailService 任务（generate_job.py:15-24、email_job.py:14-22），而 manager.py:44-65 只注册 V2 任务；两套调度/状态格式仍可被外部调用，形成 legacy/dual-format 维护面。" + }, + { + "sev": "MED", + "loc": "app/scheduler/calendar_rules.py:11", + "text": "V1 仍定义 ReportWindow/get_report_window（11-35 行），V2 又定义独立 PeriodWindow/PeriodResolver（app/scheduler/period.py:14-57）；两套日期窗口模型实现相同的前一自然日逻辑，增加跨版本语义漂移和 duplication 风险。" + }, + { + "sev": "MED", + "loc": "app/services/group_name_sync.py:81", + "text": "健康检查、群列表读取异常或空结果在 81-94 行统一转为 unavailable，157-165 行只标记 skipped 并保留本地旧名称；DailyPipeline 在 126、682 行继续生成/发送，数据源不可用时可能继续使用过期自动发送目标。" + }, + { + "sev": "MED", + "loc": "app/pipeline/daily_pipeline.py:205", + "text": "_record_group_failure 仅尽力写 FAILED；220-230 行再次捕获状态落盘异常并继续，导致返回结果显示 failed 但 run.json 可能没有失败状态，恢复与人工排查依赖日志而非持久化状态。" + }, + { + "sev": "LOW", + "loc": "app/scheduler/manager.py:23", + "text": "无效 schedule_generate_time 在 23-33 行静默回退到 00:15；虽然有 warning，但配置错误不会阻止启动，运行顺序依赖日志才能发现实际调度时间。" + }, + { + "sev": "LOW", + "loc": "app/scheduler/period.py:45", + "text": "PeriodResolver 对除 weekday_default 外的规则直接抛出 NotImplementedError（45-47 行），而 Group.schedule_rule 已暴露为可配置字段，属于已接入界面但未完成执行语义的 placeholder。" + } + ], + "auditedHash": "27298f50129acd93b946ff31d176f3912e39faea0f52e686daf041051ff5439d", + "auditedAt": "2026-08-24", + "auditedRev": "1d326e2" + }, + { + "id": "history_ingestion", + "label": "微信数据接入", + "band": "domain", + "path": "app/data_sources, app/providers/history, history services", + "paths": [ + "app/data_sources/**/*.py", + "app/providers/history/**/*.py", + "app/services/history_service.py", + "app/services/message_normalizer.py", + "app/services/speaker_identity.py" + ], + "coupling": "core", + "deps": [ + "persistence_state" + ], + "desc": "WDA/MCP/CLI/Mock 数据源、联系人解析、消息归一化与身份聚合。", + "tests": [ + "tests/test_wechat_mcp.py", + "tests/test_v2_data_source.py", + "tests/test_contact_resolver.py" + ], + "loc": 2334, + "contentHash": "2d7391184024b7dec08adc33cb71ad3fd0715ecab177058cfe4ee364738eb8b0", + "score": 64, + "grade": "C", + "tags": [ + "god-component", + "bloat", + "fallback", + "fake-output", + "duplication", + "dual-format", + "glue", + "silent-except", + "legacy" + ], + "findings": [ + { + "sev": "HIGH", + "loc": "app/providers/history/wechat_data_analysis.py:92", + "text": "WeChatDataAnalysisProvider 文件约 858 行，单个 Provider 同时负责 MCP 配置与健康检查、JSON 导出、群发现/解析、范围分页、旧锚点分页、响应解析、联系人映射、发送人冲突修复和消息转换，属于历史接入 God Component，修改任一上游协议都可能影响整条取数链。" + }, + { + "sev": "HIGH", + "loc": "app/data_sources/wechat_data_analysis.py:41", + "text": "V2 WeChatDataAnalysisSource 在 46-54 行只构造 WeChatDataAnalysisProvider，没有接入 app.providers.history.registry 的 wechat-cli/Mock fallback；文档虽然称 MCP/导出为回退路线，但 V2 真实数据源不可用时不会自动切换到 CLI 或 Mock，和 V1 provider registry 的契约不一致。" + }, + { + "sev": "HIGH", + "loc": "app/providers/history/registry.py:33", + "text": "build_providers 在 33-34 行会在 history_provider_mock_enabled 开启时自动追加 MockProvider；HistoryService.fetch 在 99-116 行把第一个 OK/EMPTY_RESULT 当作有效结果返回。真实 Provider 失败时，fixtures 可能被当成真实日报输入，形成 fake-output 风险。" + }, + { + "sev": "MED", + "loc": "app/data_sources/base.py:20", + "text": "V2 定义 V2Message、FetchResult、DataSourceStatus、WeChatDataSource（20-115 行），V1 又定义 RawMessage、FetchResult、ProviderStatus、ChatHistoryProvider（app/providers/history/base.py:15-85）；V2 wrapper 在 app/data_sources/wechat_data_analysis.py:100-134 反复做两套模型转换，存在 dual-format/duplication 和额外 glue 层。" + }, + { + "sev": "MED", + "loc": "app/data_sources/wechat_data_analysis.py:82", + "text": "_group_exists 在 84-90 行捕获所有异常并直接返回 False；当上游 list_groups 临时失败且消息结果为空时，fetch_messages 在 114-123 行会把暂时不可用误判为 GROUP_NOT_FOUND，而不是数据源异常。" + }, + { + "sev": "MED", + "loc": "app/services/history_service.py:130", + "text": "discover_groups 在 133-143 行对 Provider 健康检查或 list_groups 的所有异常直接 continue 且不记录日志；所有 Provider 同时异常时调用方只得到空群列表，缺少可诊断的失败原因。" + }, + { + "sev": "MED", + "loc": "app/providers/history/contact_resolver.py:37", + "text": "find_contact_db 在 37-43 行按目录排序后返回第一个账号的 contact.db；WeChatDataAnalysisProvider 虽保存 wechat_mcp_account（app/providers/history/wechat_data_analysis.py:110），构造 ContactResolver 时并未按账号选择数据库（117-118 行），多账号环境可能把联系人显示名映射到错误账号。" + }, + { + "sev": "MED", + "loc": "app/providers/history/wechat_data_analysis.py:267", + "text": "JSON 导出路径在 284-287 行直接读取 item['timestamp']，并由 _to_raw 在 693 行直接读取 item['group_id']；单条导出记录缺字段或格式损坏会使整个群取数异常，没有逐条隔离或明确 INVALID_RESULT 状态。" + }, + { + "sev": "LOW", + "loc": "app/providers/history/wechat_cli.py:65", + "text": "wechat-cli 的 list-groups 在 65-80 行对非零退出、JSON 解析失败和任意异常均返回空列表；作为 fallback 时会把命令故障表现为无群，且没有保留错误上下文。" + }, + { + "sev": "LOW", + "loc": "app/providers/history/base.py:68", + "text": "ChatHistoryProvider 的接口方法在 73-85 行仅抛出 NotImplementedError，属于未使用 ABC 的 placeholder 接口；目前不影响已接入实现，但错误实例化基类时才会暴露，类型边界约束较弱。" + } + ], + "auditedHash": "2d7391184024b7dec08adc33cb71ad3fd0715ecab177058cfe4ee364738eb8b0", + "auditedAt": "2026-08-24", + "auditedRev": "1d326e2" + }, + { + "id": "ranking", + "label": "排行榜与统计", + "band": "domain", + "path": "app/ranking + ranking_service", + "paths": [ + "app/ranking/**/*.py", + "app/services/ranking_service.py" + ], + "coupling": "med", + "deps": [], + "desc": "消息统计、人物榜单、模板渲染与 V1 兼容排行。", + "tests": [ + "tests/test_ranking.py", + "tests/test_v2_ranking.py", + "tests/test_v2_ranking_template.py" + ], + "loc": 376, + "contentHash": "ba5462ebaf26b376b09b83cf3b77fc096bee391083fa0dabe9c3a7c2ffaafac8", + "score": 64, + "grade": "C", + "tags": [ + "legacy", + "dual-format", + "duplication", + "bloat", + "glue", + "over-fit" + ], + "findings": [ + { + "sev": "HIGH", + "loc": "app/ranking/engine.py:18-64; app/services/ranking_service.py:17-76", + "text": "项目同时维护两套 RankingEngine/RankingResult：V2 处理 V2Message 并输出结构化 JSON，V1 service 处理 NormalizedMessage 并内置 Top10 文本渲染。两套实现都做发言统计和确定性排序，但过滤入口、字段协议、上限和输出格式不同，规则修改容易发生语义漂移。" + }, + { + "sev": "MED", + "loc": "app/ranking/renderer.py:22-70; app/services/ranking_service.py:26-46", + "text": "排行榜渲染存在两条路径：V2 使用模板变量渲染，V1 RankingResult.render 直接拼接固定文本。相同统计数据可能产生不同的标题、字段和格式，邮件/报告与 V2 ranking.txt 的展示契约不统一。" + }, + { + "sev": "MED", + "loc": "app/ranking/template_service.py:17-20,64-75,105-110", + "text": "默认模板同时以内嵌 DEFAULT_RANKING_TEMPLATE 和磁盘 default.txt 形式存在；_ensure_default 与 reset 都从代码常量写回文件。模板文件可被编辑而代码常量不会同步，恢复默认或重新初始化可能覆盖文件侧修改。" + }, + { + "sev": "MED", + "loc": "app/ranking/engine_types.py:31-33; app/ranking/renderer.py:35-38,66-70", + "text": "V2 数据结构和渲染器保留多处旧协议兼容：top_limit 被追加到字段末尾以维持位置参数，top10_lines 作为旧变量别名，render_simple 继续保留无模板调用点。兼容层已进入核心类型/渲染路径，增加长期 dual-format/legacy 维护成本。" + }, + { + "sev": "MED", + "loc": "app/ranking/engine.py:21-27,40-54; app/services/ranking_service.py:49-67", + "text": "两套引擎的可计数判定来源不同：V2 自己检查 message_type 与 SYSTEM_KEYWORDS，V1 依赖 NormalizedMessage.countable；同一原始数据经过不同标准化链路时，消息数、发言人数和 Top 排名可能不一致。" + }, + { + "sev": "LOW", + "loc": "app/ranking/engine.py:29-38; app/ranking/engine_types.py:23-43", + "text": "V2 支持可配置 top_limit，但只校验大于 0；没有上限范围或非整数输入边界，结果结构也不校验 rank/count 非负。当前调用方通常传固定值，属于低影响输入约束缺口。" + }, + { + "sev": "LOW", + "loc": "app/ranking/renderer.py:41-45", + "text": "模板替换对无法匹配正则的占位符保持原文，对未知/畸形占位符没有渲染后残留检查；虽然 validate_template 能拦截常规未知变量，但最终输出仍可能含未替换模板标记。" + }, + { + "sev": "LOW", + "loc": "app/ranking/renderer.py:66-70", + "text": "render_simple 通过函数内动态导入 DEFAULT_RANKING_TEMPLATE 保留旧调用点，属于 glue 兼容实现；未直接造成错误，但继续扩大了 V1/V2 渲染边界。" + }, + { + "sev": "LOW", + "loc": "app/ranking/engine.py:1-7,46-64", + "text": "排行榜核心本身是确定性、无 LLM、按消息数和稳定名称排序的纯计算路径，职责清晰且复杂度低；主要问题集中在 V1/V2 双实现与输出协议分裂，而非算法本身。" + } + ], + "auditedHash": "ba5462ebaf26b376b09b83cf3b77fc096bee391083fa0dabe9c3a7c2ffaafac8", + "auditedAt": "2026-08-24", + "auditedRev": "1d326e2" + }, + { + "id": "ai_prompting", + "label": "AI 摘要与生图提示词", + "band": "domain", + "path": "app/ai, app/providers/ai, templates/image_prompt", + "paths": [ + "app/ai/**/*.py", + "app/providers/ai/**/*.py", + "templates/image_prompt/**/*.md" + ], + "coupling": "high", + "deps": [], + "desc": "Codex/DeepSeek 调用、事件提取、分块、主题与布局选择、提示词生成。", + "tests": [ + "tests/test_v2_prompt_builder.py", + "tests/test_codex_summary_provider.py", + "tests/test_topic_selection.py" + ], + "loc": 3601, + "contentHash": "5f58c145a3bbe60705954bbefd164c3412bede327740708bd7f70826499c1b07", + "score": 58, + "grade": "D", + "tags": [ + "fallback", + "silent-except", + "legacy", + "dual-format", + "duplication", + "bloat", + "god-component", + "glue", + "any-escape", + "over-fit" + ], + "findings": [ + { + "sev": "HIGH", + "loc": "app/ai/prompt_builder.py:76-80,558-673; app/providers/ai/codex.py:131-152,159-221; app/providers/ai/deepseek.py:227-259", + "text": "存在多层重试与回退叠加：Builder 的事件、候选主题、版式和最终 Prompt 各自重试，Codex/DeepSeek Provider 内部再次重试，Codex 任意异常还会切换 DeepSeek。一次超时、空响应或已被服务端接受但客户端断连的请求可能产生多次模型调用、延迟放大和重复计费。" + }, + { + "sev": "HIGH", + "loc": "app/ai/prompt_builder.py:225-460,512-673", + "text": "DeepSeekImagePromptBuilder 是明显 god-component：同一类同时负责模板读取、主题解析、消息分块、事件提取、候选选题、证据回查、版式导演、最终 Prompt 拼接、敏感字段检查、元数据和 Provider 调用，约 677 行且跨越多个业务阶段，修改一个阶段容易影响整条链路。" + }, + { + "sev": "MED", + "loc": "app/providers/ai/deepseek.py:227-259", + "text": "_chat 将 retryable 初始设为 True，并在 except Exception 中保留可重试状态；JSON 解析异常、字段缺失、空内容、非瞬态网络/协议错误都会进入重试路径，未区分瞬态错误和确定性输入/响应错误。" + }, + { + "sev": "MED", + "loc": "app/providers/ai/codex.py:42-65,123-152", + "text": "CodexGPTProvider 继承 DeepSeekV4FlashProvider 却绕过父类 __init__，复用父类内部编排并自行覆盖关键方法；这是对实现细节的 over-fit 耦合，父类初始化或分块协议变化时容易产生隐蔽回归。" + }, + { + "sev": "MED", + "loc": "app/ai/prompt_templates.py:16-96,124-165; templates/image_prompt/default.md:1-98", + "text": "默认 Prompt 模板存在两份来源：Python 内嵌 DEFAULT_IMAGE_PROMPT_TEMPLATE 与可编辑 default.md。_ensure_default 只在缺失时写入，reset 又把内嵌版本写回，两个文件可独立漂移，模板修改和恢复行为不一致。" + }, + { + "sev": "MED", + "loc": "app/ai/prompt_builder.py:97-124,393-447; app/ai/prompt_templates.py:17-96; app/providers/ai/deepseek.py:37-67; app/ai/layouts.py:222-233", + "text": "事实约束、输出结构、主题/版式规则和禁止词分散在多个 Python 常量及 Markdown 模板中，存在重复 Prompt 契约；规则变更需要同步多处，容易出现主模型、备用模型、模板和最终检查器不一致。" + }, + { + "sev": "MED", + "loc": "app/providers/ai/base.py:19-20; app/providers/ai/deepseek.py:84-91; app/ai/prompt_builder.py:225-226,480-491,676-677", + "text": "同时维护新 message_items 与旧 messages_text 两套输入格式，并通过 legacy-* 合成消息 ID；Builder 还保留旧类名 DeepSeekImagePromptBuilder 并导出 GroupSummaryImagePromptBuilder 别名，说明 V1/V2 兼容层仍渗透核心路径，存在 dual-format/legacy 维护成本。" + }, + { + "sev": "MED", + "loc": "app/ai/prompt_builder.py:453-455; app/ai/topic_selection.py:177-184", + "text": "元数据和候选字段保留旧字段（deepseek_ms、interestingness_score）以兼容历史读取；代码虽能运行，但新旧协议长期并存，字段语义和版本边界不清晰。" + }, + { + "sev": "MED", + "loc": "app/ai/prompt_builder.py:415-447", + "text": "最终 Prompt 约束主要依赖 forbidden term 的字符串包含检查和若干必需块的字符串包含检查，无法验证语义上的事实一致性、每段文字是否真正只出现一次或模型是否改写了证据；校验容易出现误报和漏报。" + }, + { + "sev": "LOW", + "loc": "app/ai/prompt_builder.py:456-461", + "text": "build 使用宽泛 except Exception 统一记录并返回失败结果；异常不会静默丢失，但错误类型边界被压平，调用方只能依据截断后的字符串判断 Provider、模板还是验证失败。" + }, + { + "sev": "LOW", + "loc": "app/ai/image_themes.py:351-368; app/ai/prompt_builder_types.py:9,21,30; app/ai/layouts.py:13", + "text": "多个核心数据入口使用 Any（主题 rng、消息列表、持久化元数据、布局定义），降低了 Prompt 协议的静态约束，属于低影响 any-escape。" + }, + { + "sev": "LOW", + "loc": "app/ai/layouts.py:478-547; app/ai/prompt_builder.py:612-622", + "text": "版式导演失败两次后使用确定性 fallback_layout_plan；该回退不编造聊天事实且有稳定 seed，属于有界安全兼容，但会掩盖模型版式能力或协议问题并降低输出丰富度。" + } + ], + "auditedHash": "5f58c145a3bbe60705954bbefd164c3412bede327740708bd7f70826499c1b07", + "auditedAt": "2026-08-24", + "auditedRev": "1d326e2" + }, + { + "id": "image_generation", + "label": "图片生成与恢复", + "band": "domain", + "path": "app/image + Codex image automation", + "paths": [ + "app/image/**/*.py", + "scripts/codex_image_automation.py", + "scripts/test_image_generation.py" + ], + "coupling": "high", + "deps": [ + "ai_prompting", + "persistence_state" + ], + "desc": "Codex ImageGen 子进程、串行队列、图片校验、暂存、重生成与人工恢复。", + "tests": [ + "tests/test_codex_image_automation.py", + "tests/test_v2_image_task.py", + "tests/test_v2_image_regeneration.py" + ], + "loc": 1951, + "contentHash": "2d9181c0f9b013bf8ec7ff7a3878e4ee34733093a0444ffe6d960dc3cca5799d", + "score": 61, + "grade": "C", + "tags": [ + "god-component", + "bloat", + "fallback", + "silent-except", + "legacy", + "dual-format", + "duplication", + "any-escape", + "glue" + ], + "findings": [ + { + "sev": "HIGH", + "loc": "app/image/codex_generator.py:143", + "text": "CodexImageGenerator 所在文件约 892 行，单类同时负责 CLI 健康检查、跨进程互斥、子进程树终止、超时恢复、attempt manifest、候选扫描、哈希去重、图片验证、原子提升和 smoke 状态，属于高 blast-radius God Component。" + }, + { + "sev": "MED", + "loc": "app/image/image_task.py:67", + "text": "verify_image 在 67-77 行只检查文件存在、大小和魔数签名，没有真正解码图片；app/image/regeneration.py:106-115 直接用该结果决定替换正式图片，损坏但带合法 PNG/JPEG 头的文件可能被接受。函数文档所称的可解析校验与实现不一致。" + }, + { + "sev": "HIGH", + "loc": "scripts/codex_image_automation.py:449", + "text": "人工 adopt 流程在 449-458 行按共享 generated_images 目录的快照增量选择唯一 PNG，但 begin/adopt 没有跨进程互斥；两个并行人工任务各自产生单个候选时，候选可能被错误群任务认领并写入错误 run.json。" + }, + { + "sev": "HIGH", + "loc": "scripts/codex_image_automation.py:336", + "text": "_sync_scheduler_result 在 336-379 行对 output/.scheduler/.json 做无锁 read-modify-write。它与 DailyScheduleState 的进程内锁不共享，可能覆盖同时写入的 generation/email 字段，造成调度状态丢失或重复阶段判断。" + }, + { + "sev": "MED", + "loc": "app/image/codex_generator.py:1", + "text": "主链路使用 Codex CLI 自有 attempt manifest（文件头及 236-430 行），同时 scripts/codex_image_automation.py:1-18 保留 Desktop begin/adopt/verify marker 工作流；两套图片认领、候选发现和 run 状态同步协议并存，属于 legacy/dual-format duplication。" + }, + { + "sev": "MED", + "loc": "app/image/regeneration.py:46", + "text": "enqueue_regeneration 在 46-51 行先把任务 key 加入全局 _ACTIVE，随后 53-68 行才执行状态写入、生成器构造和线程提交；任一更新或 submit 异常都没有回滚 _ACTIVE，后续同一群/日期会永久被判定为正在队列中，直到进程重启。" + }, + { + "sev": "MED", + "loc": "app/image/image_task.py:169", + "text": "SerialImageQueue 在 169-173 行吞掉 run_hook 的全部异常并继续返回图片结果；如果 hook 的 run.json 更新失败，调用方仍可能看到成功图片但持久化状态未进入 IMAGE_READY/READY_TO_SEND。" + }, + { + "sev": "LOW", + "loc": "app/image/regeneration.py:38", + "text": "重新生图入口的 generator 参数使用 Any，app/image/image_task.py:94 也以 Any 接收生成器，没有统一协议或结构化能力检查；生成器异常只能在运行时被捕获，属于 glue/any-escape。" + }, + { + "sev": "LOW", + "loc": "app/image/codex_generator.py:865", + "text": "_load_last_smoke 在 865-872 行只要 smoke JSON 能解析就返回 ok=True，不验证记录中的 source/output 文件仍存在；健康页可能继续展示过期的最近实测成功状态。" + } + ], + "auditedHash": "2d9181c0f9b013bf8ec7ff7a3878e4ee34733093a0444ffe6d960dc3cca5799d", + "auditedAt": "2026-08-24", + "auditedRev": "1d326e2" + }, + { + "id": "delivery", + "label": "微信与邮件交付", + "band": "domain", + "path": "app/sender, email service, send scripts", + "paths": [ + "app/sender/**/*.py", + "app/services/email_service.py", + "scripts/send_daily_email.py", + "scripts/test_wechat_send.py" + ], + "coupling": "core", + "deps": [ + "persistence_state" + ], + "desc": "微信桌面/OCR 目标验证、安全发送认领、邮件交付与结果未知保护。", + "tests": [ + "tests/test_v2_wechat_sender.py", + "tests/test_v2_wechat_native.py", + "tests/test_email.py" + ], + "loc": 1725, + "contentHash": "56d8a864de2706eaebbe8194ebfb4baa2cb0acfcce915fcc60f9dae136a6196f", + "score": 61, + "grade": "C", + "tags": [ + "fallback", + "silent-except", + "legacy", + "dual-format", + "duplication", + "bloat", + "god-component", + "glue", + "over-fit" + ], + "findings": [ + { + "sev": "HIGH", + "loc": "app/services/email_service.py:200-243; scripts/send_daily_email.py:203-243", + "text": "SMTP 发送失败后直接重试同一 EmailMessage；如果 SMTP 服务端已经接收邮件但客户端在响应阶段断开，第二次尝试可能造成重复邮件。代码没有幂等键、Message-ID 去重或提交状态确认。" + }, + { + "sev": "HIGH", + "loc": "app/services/email_service.py:111-154", + "text": "EmailService.send 只返回汇总结果，不写入每群 email_status、发送批次或幂等记录；同一 run 被 scheduler、手动脚本或恢复流程再次调用时，已成功群仍可能重新发送。" + }, + { + "sev": "MED", + "loc": "app/services/email_service.py:156-243; scripts/send_daily_email.py:82-243", + "text": "生产 EmailService 与手动脚本重复实现邮件构造、图片附件校验、SMTP 连接和重试逻辑；两套实现的降级语义不同（服务侧无效图片降级为纯排行榜，脚本侧直接跳过/失败），修复容易出现行为漂移。" + }, + { + "sev": "MED", + "loc": "app/sender/wechat_automation.py:131-143", + "text": "legacy CLI 的 JSON 解析失败时回退为 proc.returncode == 0；CLI 可能退出码为 0 但没有可验证成功响应，此处仍返回 success，存在把未确认发送报告为成功的风险。" + }, + { + "sev": "MED", + "loc": "app/sender/wechat_automation.py:104-117", + "text": "日志将完整命令拼接输出，命令包含 --to 目标和 --content 消息正文；真实发送时群名、接收目标及部分聊天内容可能进入日志，属于敏感数据泄漏风险。" + }, + { + "sev": "MED", + "loc": "app/sender/wechat_native.py:37-745", + "text": "WindowsWechatDriver 集中实现 OCR、窗口激活、坐标点击、剪贴板、图片转换、提交后截图差异验证和发送互斥，文件 868 行、约 55 个定义，属于高复杂度 god-component；桌面版本、DPI 或 OCR 变化时 Blast Radius 很大。" + }, + { + "sev": "MED", + "loc": "app/sender/wechat_automation.py:7-13,31-56; app/sender/wechat_native.py:862-868; scripts/test_wechat_send.py:23-25,68-76", + "text": "旧 wechat-automation 路径文档已明确当前微信版本 UIA 不兼容并会返回 WECHAT_WINDOW_NOT_FOUND，但配置为 legacy_cli 时仍会选择该 Provider，手动真实发送测试脚本也固定使用它；存在 legacy/stub 路径继续被误用的风险。" + }, + { + "sev": "LOW", + "loc": "scripts/send_daily_email.py:132-140,246-278", + "text": "手动邮件脚本把未验证的 run_date 直接拼接到 output/{group}/{run_date} 路径；run_date 没有 YYYY-MM-DD 校验，虽然通常由本机操作者输入，仍缺少与 V2 RunStore 一致的路径边界。" + }, + { + "sev": "LOW", + "loc": "app/sender/base.py:41-46; app/sender/wechat_native.py:823-857", + "text": "文字和图片按顺序提交，不具备原子 bundle；文字成功后图片可能失败或结果未知，调用方必须正确处理部分成功。SendResult 已保留 submitted、verification_level 和 outcome_unknown 字段，这是当前较好的 fail-closed 兼容设计。" + }, + { + "sev": "LOW", + "loc": "scripts/test_wechat_send.py:60-66; app/sender/wechat_automation.py:96-102", + "text": "dry-run 测试把 ranking_demo_group_2026-08-17.txt 作为图片路径，而 legacy send_image 只检查文件存在、不验证图片格式；测试可能显示‘图片发送’成功但实际输入不是图片。" + } + ], + "auditedHash": "56d8a864de2706eaebbe8194ebfb4baa2cb0acfcce915fcc60f9dae136a6196f", + "auditedAt": "2026-08-24", + "auditedRev": "1d326e2" + }, + { + "id": "v1_reporting", + "label": "V1 报告兼容链", + "band": "domain", + "path": "legacy report/prompt/handoff services", + "paths": [ + "app/services/report_service.py", + "app/services/prompt_service.py", + "app/services/handoff_service.py", + "scripts/run_daily_pipeline.py" + ], + "coupling": "high", + "deps": [ + "persistence_state", + "history_ingestion", + "ranking", + "ai_prompting", + "delivery" + ], + "desc": "仍可触达的 V1 数据库报告、Prompt fallback、工件交接与 CLI 流程。", + "tests": [ + "tests/test_ui_api.py", + "tests/test_handoff.py", + "tests/test_prompt.py" + ], + "loc": 696, + "contentHash": "af44a66794ab4fa97b307e1452bdb5ab97c4855397a690df9c6754f4a27b9e4d", + "score": 52, + "grade": "D", + "tags": [ + "fake-output", + "silent-except", + "legacy", + "dual-format", + "duplication", + "glue", + "bloat" + ], + "findings": [ + { + "sev": "HIGH", + "loc": "app/services/report_service.py:247", + "text": "Prompt 失败时仍继续保存 Report 并调用 HandoffService；handoff_service.py:82-90 无条件写入 status=prompt_ready、poster_file=null，可能把空 Prompt 暴露为可交接产物。另在 268-271 行捕获文件输出异常只写 error_message，145-147 行仅检查 ranking_status/prompt_status，文件缺失仍可能被判定为成功。" + }, + { + "sev": "HIGH", + "loc": "app/services/report_service.py:317", + "text": "_find_success_group_run 只查询 GroupRun.ranking_status == success，没有要求 prompt_status == success。此前排行榜成功但 Prompt 失败的记录会在 175-177 行被当作已完成，非 force 重试无法修复，只会再次返回失败。" + }, + { + "sev": "HIGH", + "loc": "app/services/report_service.py:148", + "text": "GroupRun 初始 running/pending 状态在 179-187 行先提交；后续异常在 148-159 行通过新增一条 failed GroupRun 处理，原记录会遗留为 running/pending。父 Run 只有在 118-126 行全部 worker 返回后才收口，进程或 worker 异常可留下孤儿状态。" + }, + { + "sev": "HIGH", + "loc": "app/services/handoff_service.py:34", + "text": "safe_dir_name 仅替换字符并截断 display_name，不加入稳定 group_id，也不拒绝 '.'、'..' 或 Windows 保留名；save_outputs 在 52-61 行直接使用该目录并覆盖写文件。不同群名可能碰撞覆盖，'.'/'..' 还可把写入指向日期目录或 output 根目录。" + }, + { + "sev": "HIGH", + "loc": "app/services/handoff_service.py:105", + "text": "list_group_outputs 将未经格式校验的 report_date 直接拼入 output_dir 路径；文件接口把用户路径参数直接传入，绝对路径或 '..' 可突破预期日期目录并列出其他目录，同时返回完整本地 filesystem path。" + }, + { + "sev": "HIGH", + "loc": "scripts/run_daily_pipeline.py:35", + "text": "_print_results 只打印每群状态；generate、send、force-generate、rebuild-prompt、force-send 分支分别在 99、105、114、119、124 行无条件 return 0。即使结果为 failed/partial，调度器仍会收到成功退出码。" + }, + { + "sev": "MED", + "loc": "app/services/handoff_service.py:116", + "text": "handoff.json 通过 json.loads 无保护读取；save_outputs 在 57-92 行顺序写入多个文件且非原子，进程中断或部分文件损坏时，归档列表接口会直接抛异常而不是返回明确的不完整状态。" + }, + { + "sev": "MED", + "loc": "scripts/run_daily_pipeline.py:28", + "text": "_pipeline 每次都执行 repo.init_db 和 repo.apply_db_settings；status 分支在 83-89 行也经过该初始化路径，因此看似只读的状态查询可能创建或修改数据库设置。" + }, + { + "sev": "MED", + "loc": "app/services/report_service.py:1", + "text": "ReportService 明确是 V1 生成链路，而 scripts/run_daily_pipeline.py:1 使用独立 V2 DailyPipeline；handoff_service.py:11-12 又同时维护 V1/V2 状态语义。两套生成、状态和输出协议并存，后续修复容易发生 legacy/dual-format 漂移。" + }, + { + "sev": "MED", + "loc": "scripts/run_daily_pipeline.py:72", + "text": "send 命令提供 --dry-run，但 force-send 子命令没有该选项；91 行通过 getattr 缺省为 False，因此 force-send 无法在 CLI 层进行 dry-run，误调用会直接进入真实发送路径。" + }, + { + "sev": "LOW", + "loc": "app/services/prompt_service.py:89", + "text": "_build_context_text 明确标注为兼容旧调用，ranking 参数未使用，且仓库内没有实际调用；它与 _build_message_items/新 provider message_items 路径重复构造完整上下文，属于 legacy 兼容代码候选。" + } + ], + "auditedHash": "af44a66794ab4fa97b307e1452bdb5ab97c4855397a690df9c6754f4a27b9e4d", + "auditedAt": "2026-08-24", + "auditedRev": "1d326e2" + }, + { + "id": "persistence_state", + "label": "数据库与运行状态", + "band": "storage", + "path": "app/db + app/v2", + "paths": [ + "app/db/**/*.py", + "app/v2/**/*.py" + ], + "coupling": "core", + "deps": [ + "v1_reporting" + ], + "desc": "SQLModel/SQLite、启动迁移、V2 run.json 状态、发送 lease 与恢复。", + "tests": [ + "tests/test_v2_recovery.py", + "tests/test_v2_group_migration.py", + "tests/test_v2_period.py" + ], + "loc": 1125, + "contentHash": "f6293bb537b00804adedf91e75ec40c2f5c913bb5688887009da44baddb2f533", + "score": 62, + "grade": "C", + "tags": [ + "silent-except", + "fallback", + "legacy", + "dual-format", + "duplication", + "bloat", + "god-component", + "glue", + "any-escape" + ], + "findings": [ + { + "sev": "HIGH", + "loc": "app/db/models.py:10-79", + "text": "持久化模型缺少关系约束：wechat_group_id 仅普通索引，GroupRun.run_id/group_id 与 Report.group_run_id 都是无 ForeignKey 的裸整数，也没有复合唯一约束；重复群组、重复运行和孤儿记录只能依赖业务层避免。" + }, + { + "sev": "HIGH", + "loc": "app/db/models.py:38-79; app/v2/run_store.py:1-7", + "text": "存在双持久化格式：V1 将 Run/GroupRun/Report 写入 SQLite，V2 将同一类运行状态和输出元数据写入每群每日 run.json；当前模块没有统一同步或一致性边界，状态可能分叉。" + }, + { + "sev": "HIGH", + "loc": "app/v2/run_store.py:153-162", + "text": "load_run 对 JSONDecodeError/OSError 静默 pass，并回退为全新的 PENDING 状态；损坏或部分写入的状态会被当成未执行任务，可能触发重复生成或重复后续操作。" + }, + { + "sev": "MED", + "loc": "app/db/repository.py:34-184", + "text": "迁移由多个手写 _migrate_* 函数组成，重复使用 marker 查询、更新、插入和 commit；没有版本表、统一迁移注册或全局迁移事务，属于明显 legacy/bloat/duplication。" + }, + { + "sev": "MED", + "loc": "app/db/repository.py:187-206", + "text": "init_db 每次启动按固定顺序执行 create_all、补列、种默认值和多轮数据迁移；多个进程并发启动或中途异常时可能出现部分迁移，且旧 schema 默认值与模型默认值容易漂移。" + }, + { + "sev": "MED", + "loc": "app/db/repository.py:331-425", + "text": "Repository 的 save_group、delete_group、restore_group、create_run、save_report、set_setting_value 等操作都立即 commit，缺少跨实体事务边界；核心流程的多阶段写入无法由该层保证原子性。" + }, + { + "sev": "MED", + "loc": "app/v2/run_store.py:164-181", + "text": "普通 save_run/update 只使用进程内 RLock，临时文件名固定为 run.json.tmp；跨进程写同一 run 时仍可能互相覆盖临时文件或丢失更新，跨进程互斥只在发送 claim 路径启用。" + }, + { + "sev": "MED", + "loc": "app/v2/recovery.py:85-105", + "text": "verify_output 只检查文件存在且 size>0，不校验 JSON schema、内容可解析性、图片实际解码或状态字段一致性；损坏文件可能被判定为完整。" + }, + { + "sev": "MED", + "loc": "app/v2/recovery.py:113-153", + "text": "recover_incomplete 通过 display_name 查询 group_id，而不是使用持久化稳定 ID；重名群组时可能恢复到错误群组，且动态导入 DailyPipeline/DB，形成较强 glue 耦合。" + }, + { + "sev": "MED", + "loc": "app/v2/run_store.py:119-121", + "text": "运行目录以 safe_dir_name(group_name)/run_date 定位，没有稳定 group_id；名称清洗、特殊字符归一化或截断会造成不同群组目录碰撞并覆盖状态/输出。" + }, + { + "sev": "MED", + "loc": "app/v2/constants.py:19-32", + "text": "STATUS_FLOW 只声明状态顺序，is_terminal 只做终态集合判断，没有统一的合法状态转移校验；状态推进规则仍分散在上层 pipeline，状态机约定容易被绕过。" + }, + { + "sev": "LOW", + "loc": "app/v2/run_store.py:339-364", + "text": "list_runs 对损坏 run.json 直接静默跳过，调用方无法区分“没有运行”和“状态文件损坏”，导致恢复/审计列表不完整。" + }, + { + "sev": "LOW", + "loc": "app/v2/recovery.py:38-52", + "text": "旧 run 缺少 image_enabled 或值未知时固定回退为 True；这是兼容性 fallback，但会把旧格式或异常数据强制解释为需要图片。" + }, + { + "sev": "LOW", + "loc": "app/db/repository.py:13,428-431", + "text": "全局 engine 使用 Any，_now 返回 Any，削弱了 persistence 边界的静态类型约束，属于低影响 any-escape。" + }, + { + "sev": "MED", + "loc": "app/v2/run_store.py:113-365", + "text": "RunStore 同时承担路径生成、JSON 状态读写、历史布局读取、跨进程发送锁、发送租约和运行枚举，职责集中度偏高，已接近 persistence god-component；后续修改发送状态时容易影响文件状态逻辑。" + } + ], + "auditedHash": "f6293bb537b00804adedf91e75ec40c2f5c913bb5688887009da44baddb2f533", + "auditedAt": "2026-08-24", + "auditedRev": "1d326e2" + }, + { + "id": "operations_scripts", + "label": "运维与计划任务脚本", + "band": "ops", + "path": "scripts/daily_auto.py 等", + "paths": [ + "scripts/daily_auto.py", + "scripts/generate_fixtures.py", + "scripts/install_autostart.py", + "scripts/install_daily_task.py", + "scripts/test_wechat_data.py" + ], + "coupling": "med", + "deps": [ + "pipeline_scheduler", + "history_ingestion" + ], + "desc": "Windows 自动启动/计划任务、每日监控入口、fixture 与数据源手工检查。", + "tests": [ + "tests/test_daily_auto.py", + "tests/test_install_daily_task.py" + ], + "loc": 652, + "contentHash": "ec805b9b5f228de0522b1755d42b5bfa244b32cf5c2202ca4a900b428c3737ee", + "score": 58, + "grade": "D", + "tags": [ + "legacy", + "dual-format", + "fake-output", + "glue", + "over-fit", + "bloat" + ], + "findings": [ + { + "sev": "HIGH", + "loc": "scripts/test_wechat_data.py:31", + "text": "脚本声明只读取微信数据，但每个命令在 138 行都会先调用 _load_settings；该函数执行 repo.init_db 和 repo.apply_db_settings，初始化过程包含建表、默认值写入和迁移，健康检查/列群等只读操作可能修改生产数据库。" + }, + { + "sev": "HIGH", + "loc": "scripts/test_wechat_data.py:177", + "text": "main 无论 health、list-groups、resolve 或 fetch 返回什么状态都固定 return 0；数据源不可用、读取失败或空结果仍会被自动化环境视为成功，脚本没有把 Provider 状态映射为退出码。" + }, + { + "sev": "MED", + "loc": "scripts/daily_auto.py:69", + "text": "already_running 被纳入成功退出集合。任务实际没有执行任何生成或发送时仍返回 0，计划任务监控无法区分‘已完成’和‘被并发锁跳过’。" + }, + { + "sev": "MED", + "loc": "scripts/install_daily_task.py:114", + "text": "安装、卸载和状态命令的 main 固定 return 0；_install 即使 schtasks 返回失败也只打印字符串，调用方无法通过退出码发现任务安装失败。install_autostart.py:64-74 存在同样问题。" + }, + { + "sev": "MED", + "loc": "scripts/install_daily_task.py:44", + "text": "两阶段安装先创建生成任务，再创建发送任务；发送任务创建失败时 55-71 行只返回警告，不回滚已创建的生成任务，系统会留下半安装状态。" + }, + { + "sev": "MED", + "loc": "scripts/test_wechat_data.py:45", + "text": "fetch 结果会把完整 messages、消息详情和 Provider 错误写入 output/test-data，固定文件名会覆盖旧证据，且没有脱敏、保留期限或显式清理边界；真实聊天验证可能留下敏感数据。" + }, + { + "sev": "MED", + "loc": "scripts/install_daily_task.py:35", + "text": "计划任务的生成阶段调用 daily_auto.py，发送阶段调用另一套 run_daily_pipeline.py；两个入口拥有不同的状态文件、失败语义和退出码，调度链路存在 dual-format/glue 漂移风险。" + }, + { + "sev": "LOW", + "loc": "scripts/generate_fixtures.py:170", + "text": "生成器按当前 date.today() 滚动生成最近 8 天 fixtures，并在 171-183 行直接覆盖 groups.json 和消息文件；没有 --output、日期或确认参数，容易覆盖手工 fixture，且输出结果随执行日期变化。" + }, + { + "sev": "LOW", + "loc": "scripts/generate_fixtures.py:24", + "text": "groups.json 声明 member_count 为 48/112，但实际生成的 MEMBERS_A/MEMBERS_B 只有 15/8 人；这是 mock_fixture 的不一致数据契约，可能让 UI 或测试展示错误人数。" + }, + { + "sev": "LOW", + "loc": "scripts/install_autostart.py:24", + "text": "开机自启注释仍称启动 V1 脚本，_START_CMD 通过 start_windows.bat 启动旧入口；该兼容包装与当前 V2 调度入口并存，版本边界和实际启动流程不清晰。" + } + ], + "auditedHash": "ec805b9b5f228de0522b1755d42b5bfa244b32cf5c2202ca4a900b428c3737ee", + "auditedAt": "2026-08-24", + "auditedRev": "1d326e2" + } + ] +} \ No newline at end of file diff --git a/.codemap/sonar-summary.json b/.codemap/sonar-summary.json new file mode 100644 index 0000000..1bc3c74 --- /dev/null +++ b/.codemap/sonar-summary.json @@ -0,0 +1,48 @@ +{ + "server": "SonarQube Community Build 26.8.0.126808", + "projectKey": "groupbrief-v1-local-audit", + "analysisId": "1a07a159-5e1b-41b1-a74b-f68ee6dfe58a", + "ceTaskId": "02a52972-25c5-4dcc-8493-1bf7f0d9c2dd", + "status": "SUCCESS", + "warnings": 0, + "metrics": { + "ncloc": 23570, + "files": 120, + "bugs": 5, + "vulnerabilities": 12, + "securityHotspots": 0, + "codeSmells": 295, + "duplicatedBlocks": 5, + "duplicatedLines": 68, + "duplicatedLinesDensity": 0.2, + "coverage": 63.9, + "lineCoverage": 63.9, + "linesToCover": 10145, + "uncoveredLines": 3664, + "cognitiveComplexity": 3703, + "cyclomaticComplexity": 4368, + "maintainabilityRating": "A", + "reliabilityRating": "D", + "securityRating": "D", + "technicalDebtMinutes": 2141, + "technicalDebtRatioPercent": 0.3 + }, + "humanReview": { + "confirmedSonarBugs": 0, + "sonarBugCandidatesNeedingLowPriorityReview": 5, + "sonarVulnerabilitiesClassifiedLowValue": 12, + "reason": "Vulnerability findings are non-cryptographic random selection and deterministic fixture hashes; source review found more material authentication and path-boundary risks not detected by SonarQube." + }, + "validation": { + "pythonTestsPassed": 413, + "pythonTestsFailed": 1, + "pythonWarnings": 1, + "pythonStatementCoveragePercent": 74.7, + "frontendAutomatedTests": 0, + "frontendBuild": "PASS", + "frontendMainJsBytes": 451300, + "frontendMainJsGzipBytes": 131500, + "npmAudit": {"moderate": 1, "high": 1, "critical": 0}, + "pipAuditKnownVulnerabilities": 0 + } +} diff --git a/.env.example b/.env.example index f0e6df8..b0c54dd 100644 --- a/.env.example +++ b/.env.example @@ -6,16 +6,21 @@ APP_HOST=127.0.0.1 APP_PORT=8766 APP_TIMEZONE=Asia/Shanghai +# 仅自动化测试或显式本地假数据演练可开启；真实运行必须保持 false。 +ALLOW_TEST_PROVIDERS=false +# 旧 V1 数据库流水线默认只读；仅短期回滚/兼容窗口可临时改为 maintenance。 +# 该开关不会从数据库或设置 UI 生效。 +LEGACY_V1_WRITE_MODE=read_only # --- 数据库 --- DATABASE_URL=sqlite:///data/groupbrief.db -# --- 微信历史读取 Provider --- -# 主 Provider：wechat_data_analysis（WeChatDataAnalysis） -# 备用 Provider：wechat_cli（wechat-cli） +# --- V1 兼容历史读取 Provider（正式 V2 固定使用 WeChatDataAnalysis MCP/导出）--- +# 不通过生产设置 UI 修改；V1 主/备仍保留一版兼容。 HISTORY_PROVIDER_PRIMARY=wechat_data_analysis HISTORY_PROVIDER_FALLBACK=wechat_cli -HISTORY_PROVIDER_MOCK_ENABLED=true +# Mock 必须同时满足 ALLOW_TEST_PROVIDERS=true 才会注册。 +HISTORY_PROVIDER_MOCK_ENABLED=false # 微信数据目录（WeChatDataAnalysis 原始数据探测/状态提示使用，不用于群名解析） WECHAT_DATA_DIR= # 微信导出数据目录（WeChatDataAnalysis 结构化 JSON 导出根目录，群名解析与消息读取使用）。 @@ -52,8 +57,7 @@ CODEX_SUMMARY_TIMEOUT_SECONDS=240 CODEX_SUMMARY_MAX_RETRIES=2 CODEX_SUMMARY_REQUEST_CONCURRENCY=2 -# DeepSeek 备用（只有 Codex 主调用失败时才使用） -AI_PROVIDER=deepseek +# DeepSeek 备用（只有 SUMMARY_PROVIDER_FALLBACK=deepseek 时才使用） AI_BASE_URL=https://api.deepseek.com AI_API_KEY= AI_MODEL=deepseek-v4-flash @@ -109,6 +113,17 @@ EMAIL_SEND_PARTIAL_REPORT=true # --- 自动任务 --- # 业务调度统一由后台 APScheduler 管理：每日 00:15 生成前一日群报，08:30 起按群发送。 +# 正式环境只能有一个 owner。默认 fastapi；切为 external 前必须先停止内部 scheduler。 +SCHEDULER_OWNER=fastapi SCHEDULE_GENERATE_TIME=00:15 SCHEDULE_EMAIL_TIME=after_generate SCHEDULE_STARTUP_CATCHUP_ENABLED=true +# 自动恢复仅扫描今天与前一天；代码还会把旧值强制收敛到 48 小时。 +RELIABILITY_WATCHDOG_ENABLED=true +RELIABILITY_LOOKBACK_DAYS=2 +RELIABILITY_WATCHDOG_INTERVAL_MINUTES=10 +# 周报能力先部署后灰度。14 天验收完成前保持关闭；周报发送另有独立闸门。 +WEEKLY_INSIGHTS_ENABLED=false +WEEKLY_SEND_ENABLED=false +WEEKLY_GENERATE_TIME=07:45 +WEEKLY_SEND_TIME=08:30 diff --git a/.github/dependabot.yml b/.github/dependabot.yml new file mode 100644 index 0000000..741b3d0 --- /dev/null +++ b/.github/dependabot.yml @@ -0,0 +1,19 @@ +version: 2 +updates: + - package-ecosystem: pip + directory: "/" + schedule: + interval: weekly + open-pull-requests-limit: 5 + + - package-ecosystem: npm + directory: "/frontend" + schedule: + interval: weekly + open-pull-requests-limit: 5 + + - package-ecosystem: github-actions + directory: "/" + schedule: + interval: weekly + open-pull-requests-limit: 3 diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 21fdc48..95d58b4 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -19,10 +19,16 @@ jobs: with: python-version: "3.12" cache: pip + cache-dependency-path: | + requirements.txt + requirements-dev.txt + requirements.lock - name: Install dependencies - run: python -m pip install -r requirements.txt -r requirements-dev.txt - - name: Run tests - run: python -m pytest tests -q + run: python -m pip install -r requirements.txt -r requirements-dev.txt -c requirements.lock + - name: Run tests and generate Sonar coverage + run: python -m pytest tests -q --cov=app --cov=scripts --cov-report=xml:coverage.xml + - name: Run tests in deterministic random order + run: python -m pytest tests -q --random-order-seed=20260825 - name: Compile Python sources run: python -m compileall -q app scripts tests @@ -36,10 +42,14 @@ jobs: - uses: actions/checkout@v7 - uses: actions/setup-node@v7 with: - node-version: "20" + node-version: "22.22.2" cache: npm cache-dependency-path: frontend/package-lock.json - name: Install dependencies run: npm ci - name: Build frontend run: npm run build + - name: Run frontend unit tests + run: npm test + - name: Run browser E2E with Fake API + run: npm run test:e2e diff --git a/.gitignore b/.gitignore index 385b33d..a5a6078 100644 --- a/.gitignore +++ b/.gitignore @@ -7,6 +7,7 @@ data/ output/ logs/ +runtime/ *.db *.sqlite *.sqlite3 @@ -27,6 +28,11 @@ build/ node_modules/ frontend/dist/ frontend/.vite/ +frontend/playwright-report/ +frontend/test-results/ +.coverage +coverage.xml +htmlcov/ *.tsbuildinfo # --- 系统文件 --- diff --git a/GROUPBRIEF_DAILY_PREVIOUS_DAY_RECOVERY_TASK.md b/GROUPBRIEF_DAILY_PREVIOUS_DAY_RECOVERY_TASK.md new file mode 100644 index 0000000..9d8c410 --- /dev/null +++ b/GROUPBRIEF_DAILY_PREVIOUS_DAY_RECOVERY_TASK.md @@ -0,0 +1,62 @@ +# GroupBrief 每日前一日规则修复与 2026-08-29 补跑任务 + +## 背景 + +2026-08-29 00:15 调度器已触发，但六个启用群均因 `weekday_default` 周末规则被过滤， +当天清单为空并以 `not_run` 结束。这与“每天生成前一天群报”的产品约定不一致。 + +## 目标 + +1. 将应用、新建群、空值回退和示例配置的默认统计规则统一为 `daily_previous_day`。 +2. 保留 `weekday_default` 作为用户显式选择的兼容规则。 +3. 在生产 SQLite 在线备份后，用单事务把群 23–28 切换为 `daily_previous_day`。 +4. 通过带 `state_version` CAS 的恢复入口修复 2026-08-29 空清单并补跑六群生成。 +5. 补跑不发送邮件、不手工调用微信发送；微信仍由既有 08:30 `send_due` 流程处理。 + +## 允许修改范围 + +- 群配置默认值、统计周期默认值、任务清单空值回退。 +- Dashboard/群详情的默认规则展示和示例配置。 +- 调度状态 CAS 更新与空清单显式恢复 API。 +- 与上述行为直接相关的后端、前端测试。 + +## 禁止修改范围 + +- 不改变显式 `weekday_default` 的工作日语义。 +- 不修改群名称、发送目标、发送时间、发送开关、Prompt、图片主题或排行榜规则。 +- 不删除或覆盖历史 `run.json`、发送凭证、图片、Prompt 或聊天数据。 +- 不手工提前发送、不重发历史内容、不访问或写入任何密钥。 + +## 已确定实现要求 + +- 默认值统一为 `daily_previous_day`；显式工作日规则仍可选。 +- 空清单恢复只允许在旧状态为 `expected_groups=[]`、`generation_status=not_run`、 + 调用方提供的 `state_version` 匹配且当天不存在群级运行记录时执行。 +- 恢复前按当前数据库重新计算预期群，并校验群 ID 集合完全匹配。 +- 状态更新必须在生成锁与状态文件锁下原子完成，记录恢复审计字段。 +- 生产数据库使用 SQLite online backup、`BEGIN IMMEDIATE`、完整性和外键检查。 + +## 验收标准 + +- 周六/周日默认规则也运行，且只覆盖前一自然日。 +- 显式 `weekday_default` 周末仍跳过，周一仍覆盖周五至周日。 +- 新建群、模型默认、旧库补列、空值回退与前端新建表单均为 `daily_previous_day`。 +- 生产群 23–28 的规则均为 `daily_previous_day`，其他配置不变。 +- 2026-08-29 清单包含且仅包含群 23–28，统计窗口为 2026-08-28 全天。 +- 六群生成达到可信终态；未手工调用邮件或微信发送。 +- 后端测试、前端测试、前端构建、Python 编译和 `git diff --check` 通过。 + +## 测试命令 + +- `.\.venv\Scripts\python.exe -m pytest tests -q` +- `npm test -- --run` +- `npm run build` +- `.\.venv\Scripts\python.exe -m compileall app scripts` +- `git diff --check` + +## 返回格式 + +- 根因与修复范围。 +- 生产备份路径、数据库更新前后值和完整性检查。 +- 2026-08-29 六群生成及 08:30 待发送状态。 +- 测试、提交哈希、分支和推送结果。 diff --git a/GROUPBRIEF_DASHBOARD_RUNTIME_WINDOWS_TASK.md b/GROUPBRIEF_DASHBOARD_RUNTIME_WINDOWS_TASK.md new file mode 100644 index 0000000..17186bd --- /dev/null +++ b/GROUPBRIEF_DASHBOARD_RUNTIME_WINDOWS_TASK.md @@ -0,0 +1,70 @@ +# GroupBrief 总览实时任务节点与日志窗口任务 + +## 背景 + +当前“运行总览”能展示每天各群的结果卡片，但 00:15 后台任务运行期间缺少实时阶段和日志视图。现有权威状态已经保存在 `output/.scheduler/.json` 与各群 `run.json`，日志已经按固定分类写入 `logs/`。 + +## 目标 + +- 在总览统计卡片下方新增“任务节点”和“运行日志”两个只读窗口。 +- 任务节点使用最新 scheduler/run 状态，展示全局阶段和每群当前阶段。 +- 日志通过固定白名单、日期过滤、条数限制和脱敏后返回结构化数据。 +- 今日运行中自动刷新，历史日期保持手动刷新。 + +## 允许修改范围 + +- `app/scheduler/runtime_status.py` +- `app/api/v2_ui_read.py` +- 新增只读日志解析服务及对应后端测试 +- `frontend/src/api.ts` +- `frontend/src/pages/v2/Dashboard.tsx` +- Dashboard 相关样式、Vitest 与 Playwright 测试 +- V2 路由契约测试 +- 本任务文件 + +## 禁止修改范围 + +- 不修改 00:15/08:30 调度时间或 APScheduler 注册逻辑。 +- 不修改取数、排行、Prompt、生图、发送算法及状态推进逻辑。 +- 不修改数据库 Schema、生产数据、发送锁、未知结果保护或外部 Provider 配置。 +- 不触发真实取数、AI 调用、图片生成、微信发送、服务重启或部署。 +- 不读取或写入 `.env`、凭据、Cookie、浏览器数据。 + +## 已确定实现要求 + +1. 抽取每日状态的纯读取构建函数；Dashboard 直接基于最新 scheduler 和启用群 run 快照生成 `runtime`，现有状态文件写入合同保持兼容。 +2. 节点固定为：调度启动、读取群消息、生成排行榜、摘要与提示词、生成图片、等待发送/发送完成。 +3. 节点状态固定为：`pending`、`running`、`success`、`retry_pending`、`held`、`failed`。 +4. 图片 `image_job.status=queued` 只能显示排队/等待，不得冒充正在生成。 +5. 损坏状态 fail-closed 为需要关注，不写回、不猜测成功。 +6. 新增 `GET /api/v2/runtime/logs`；`tail` 默认 100、范围 1..200；来源只允许 `scheduler/app/provider/ai`。 +7. 日志记录包含时间、级别、来源、消息；按运行日期过滤，跨来源按时间排序，消息最长 500 字符并脱敏。 +8. 桌面端任务节点/日志按约 60%/40% 并排，820px 以下上下排列。 +9. 今日 `not_started` 每 30 秒刷新，`running/retry_pending` 每 3 秒刷新，终态停止；页面后台暂停，恢复可见立即刷新；历史日期仅手动刷新。 +10. 日志支持来源/级别筛选、暂停/继续自动滚动和手动刷新。 + +## 验收标准 + +- Dashboard 响应新增 `runtime`，旧字段与原 URL 保持兼容。 +- 六群可展开查看当前节点；空状态、运行中、重试、暂停、失败、完成均有准确文案。 +- 日志接口拒绝非法来源/级别/日期/条数，不能读取任意路径，不能泄露敏感字段。 +- 现有 Dashboard 生成、发送与人工核对操作仍只命中原接口。 +- 窄屏无横向溢出；轮询会按状态和页面可见性启停。 +- 所有定向后端测试、前端测试、构建和 Dashboard E2E 通过。 + +## 测试命令 + +```powershell +.\.venv\Scripts\python.exe -m pytest tests/test_runtime_status.py tests/test_runtime_logs.py tests/test_v2_ui_router_contract.py tests/test_p24_observability.py -q +npm --prefix frontend test -- --run src/api.test.ts src/pages/v2/dashboardRuntime.test.ts +npm --prefix frontend run build +npm --prefix frontend run test:e2e -- --grep Dashboard +git diff --check +``` + +## 返回格式 + +- 汇报后端状态/日志接口、前端两个窗口、自动刷新与安全边界。 +- 提供测试命令和准确通过/失败数量。 +- 汇报 Git 状态、提交哈希、分支、仓库地址与推送结果。 +- 明确生产服务未重启，真实外部任务未触发。 diff --git a/GROUPBRIEF_EASON_RANKING_IMAGE_FIX_TASK.md b/GROUPBRIEF_EASON_RANKING_IMAGE_FIX_TASK.md new file mode 100644 index 0000000..4dc9166 --- /dev/null +++ b/GROUPBRIEF_EASON_RANKING_IMAGE_FIX_TASK.md @@ -0,0 +1,67 @@ +# GroupBrief Eason 排行榜与图片事实修复任务 + +## 背景 + +2026-08-28 的 `Eason张UED-4群🤘`（数据库 `group_id=28`）排行榜使用“所有非系统消息气泡”计数，群友期望改为文字主榜并单列互动数；展示名必须以 WeChatDataAnalysis 返回的 `senderDisplayName` 为准。当前再生图片还包含聊天证据中不存在的 BMI、体脂率和天气数字。 + +## 目标 + +1. 为单群提供可配置的排行榜计数与发言人名称策略。 +2. 仅为 Eason 群启用“文字主榜＋互动数”和 WeChatDataAnalysis 名称权威策略。 +3. 扩展排行榜 JSON/文本并保持旧群、旧模板和下游读取兼容。 +4. 为严格模式图片增加基于消息证据和 OCR 的事实校验，事实不明时失败并保持发送锁。 +5. 重新读取 Eason 的 2026-08-28 数据、重建榜单和 Prompt、再生合格图片，绝不触发微信发送。 + +## 允许修改范围 + +- `app/db`、`app/api/groups.py`、排行榜与模板模块。 +- WeChatDataAnalysis Provider/V2 数据源及消息快照兼容字段。 +- 每日生成阶段、图片验证和图片再生流程。 +- Eason 群配置页面、Dashboard 排行预览及相应类型。 +- 与上述行为直接相关的测试、排行榜模板和本任务文件。 +- 验证通过后更新本地 Eason 群配置与 2026-08-28 产物。 + +## 禁止修改范围 + +- 不改变其他五群的持久化策略或历史产物。 +- 不修改 OpenAI/Codex 登录、认证或 Provider 配置。 +- 不调用微信发送接口，不清除 `USER_REQUEST_NO_SEND_2026_08_28` 发送锁。 +- 不提交 `.env`、日志、缓存、运行时数据库备份、生成图片或其他敏感/大型产物。 +- 不 reset、stash、清理、强推或重写 Git 历史。 + +## 已确定实现要求 + +- 新增 `ranking_count_policy`，默认 `all_messages`；Eason 使用 `text_primary_with_interactions`。 +- 新增 `sender_name_policy`，默认 `resolved`；Eason 使用 `wechat_data_analysis`。 +- 文字主榜只按 `message_type=text` 排名，互动为其余可计数非系统类型；同文字数按规范化名称和身份键排序。 +- `message_count` 保持所有可计数消息总数；新增文字数、互动数和文字发言人数。`TopSpeaker.count` 在文字主榜中等于文字数，并新增文字/互动/名称来源字段。 +- WeChatDataAnalysis 原始展示名必须跨 MCP 与 JSON export 保存；Eason 不得被联系人备注覆盖。身份仍以 `sender_id` 聚合；空名匿名化；同名不同 ID 保持可区分。 +- Eason Prompt 追加严格可见文案合同；图片事实校验使用同目录 `messages.json`、Prompt 与 OCR。无证据数字/长文本、OCR 不可用或识别失败不得晋级为可复核图片。 +- 严格模式生图事实校验失败时最多生成两次；最终失败保持旧图和发送锁。 +- 图片尺寸只校验可解码和正尺寸，不要求固定 1024×1536。 + +## 验收标准 + +- 当前 Eason 快照基准：383 条文字、226 条互动、609 条可计数消息、42 名活跃发言人；“深圳-UI-白白”为文字 55、互动 34。 +- Eason 的名称来源标记为 `wechat_data_analysis`，不使用联系人备注覆盖。 +- OCR/事实测试拒绝 `12%`、`90÷1.72²=30.4`、`120天`、`少油少盐`，允许消息证据中的 `78.8`、`61`、`66`。 +- 其他群默认排行榜 JSON/文本和名称解析行为保持兼容。 +- Eason 新产物完成后 `send_hold=true` 且 `send_hold_reason=USER_REQUEST_NO_SEND_2026_08_28`，没有新增发送证据。 +- SQLite `integrity_check=ok`、外键检查无错误。 + +## 测试命令 + +```powershell +.\.venv\Scripts\python.exe -m pytest tests\test_v2_ranking.py tests\test_v2_ranking_template.py -q +.\.venv\Scripts\python.exe -m pytest tests\test_wechat_mcp.py tests\test_v2_data_source.py -q +.\.venv\Scripts\python.exe -m pytest tests\test_v2_image_task.py tests\test_v2_image_regeneration.py -q +.\.venv\Scripts\python.exe -m pytest tests -q +.\.venv\Scripts\python.exe -m compileall -q app tests +Set-Location frontend; npm test; npm run build +``` + +## 返回格式 + +- 精确命令、退出码、测试数量和失败证据。 +- 修改文件范围、关键行为、Eason 实际产物统计与图片事实复核结果。 +- 最终发送锁、Git 提交、分支、远端和推送结果。 diff --git a/GROUPBRIEF_FIXED_DAILY_SCHEDULER_TASK.md b/GROUPBRIEF_FIXED_DAILY_SCHEDULER_TASK.md new file mode 100644 index 0000000..3f04531 --- /dev/null +++ b/GROUPBRIEF_FIXED_DAILY_SCHEDULER_TASK.md @@ -0,0 +1,62 @@ +# GroupBrief 定点调度重构任务 + +## 背景 + +当前生产调度器每分钟第 15 秒执行一次 `send_due`，并每 10 分钟执行可靠性 Watchdog。大量无工作扫描和 `not_run` 日志容易让 Dashboard 看起来持续运行。产品已确认改为定点调度：每天 00:15 生成、每天 08:30 启动六群串行发送。 + +## 目标 + +- 只保留 00:15 生成和 08:30 发送两个固定日常 Cron。 +- 取消分钟级发送扫描和 10 分钟 Watchdog 永久轮询。 +- 六个启用群从 08:30 起按稳定群 ID 顺序串行发送。 +- 用启动检查和去重的一次性 `DateTrigger` 承担安全补偿，不牺牲发送 claim、未知结果锁和 30 分钟迟到窗口。 +- Dashboard 展示下次生成时间和下次发送批次时间，空闲时不持续输出 `not_run`。 + +## 允许修改范围 + +- `app/scheduler/`、`app/pipeline/` 中的调度与一次性补偿编排。 +- 全局设置、群聊 API 兼容校验、系统状态与 Dashboard 只读状态投影。 +- 群聊编辑页面和 Dashboard 的发送时间展示。 +- 与本次改造直接相关的后端、前端及 E2E 测试。 +- `.gitignore` 中仅与现有测试/运行产物有关的窄范围忽略项。 + +## 禁止修改范围 + +- 不改变微信目标识别、OCR、发送实现或外部提交协议。 +- 不解除 `SEND_RESULT_UNKNOWN`、`SENT`、人工暂停、状态损坏的 fail-closed 保护。 +- 不触发真实微信发送、真实 AI、真实生图或生产数据库写入。 +- 不重启 8766，不部署、不合并 PR，不修改 Provider、登录或认证配置。 +- 不删除、清理、暂存或覆盖现有未跟踪运行产物和用户文件。 + +## 已确定实现要求 + +- `schedule_generate_time` 默认 00:15；新增全局 `schedule_send_time`，默认 08:30。 +- 08:30 是整批启动时间，不是六群并发时间；发送顺序沿用稳定群 ID。 +- 明确提交前失败继续后续群；可能已提交但结果未知时停止整批。 +- 启动时最多检查当前日和前一日生成欠账；历史任务只补生成，禁止自动发送。 +- 今日发送仅允许在 08:30 至 09:00 自动补偿；超过窗口进入 `MISSED_SEND_WINDOW`。 +- 只有显式可重试且持久化了 `next_retry_at` 的任务才能注册一次性重试；任务必须去重，重启后可从持久化状态恢复。 +- 群级 `send_time` 数据库列保留兼容；API 缺省时使用 08:30，非 08:30 输入必须明确拒绝；前端不再提供群级编辑。 +- 周报关闭时不得参与日报发送任务或产生每分钟空闲日志。 + +## 验收标准 + +- 调度器固定日常任务只有 00:15 生成和 08:30 发送，不存在分钟 Cron 或 Interval Watchdog。 +- 启动补偿、迟到窗口、READY 后一次性补偿、重启恢复和任务去重均有自动化测试。 +- 六群串行顺序、提交前失败继续、结果未知停止、SENT 不重发均有回归覆盖。 +- Dashboard/API 能同时报告 `next_generate_at` 与 `next_send_at`。 +- 正式 diff 不包含运行产物、敏感数据、调试代码或范围外依赖变化。 + +## 测试命令 + +- `.venv\Scripts\python.exe -m pytest tests -q` +- `npm test`（工作目录 `frontend`） +- `npm run build`（工作目录 `frontend`） +- `git diff --check` + +## 返回格式 + +- 汇报根因与最终调度行为。 +- 列出测试命令、通过数量或准确失败证据。 +- 报告 Git 初始状态、默认分支、任务分支、提交与远端 SHA、Push/PR/CI 状态。 +- 明确说明未触发真实微信发送、未重启生产 8766、未自动合并 PR。 diff --git a/GROUPBRIEF_FLEXIBLE_IMAGE_DIMENSIONS_TASK.md b/GROUPBRIEF_FLEXIBLE_IMAGE_DIMENSIONS_TASK.md new file mode 100644 index 0000000..210c621 --- /dev/null +++ b/GROUPBRIEF_FLEXIBLE_IMAGE_DIMENSIONS_TASK.md @@ -0,0 +1,59 @@ +# 群报图片尺寸规则放宽任务 + +## 背景 + +2026-08-28 的茶馆 V3.0 与 Grok App 交流群已生成可完整解码的 PNG 图片，但由于画布为 864×1821、不是原先固定的 1024×1536，系统将其保留为候选图并进入人工确认。用户已确认尺寸不影响群报使用，要求修改规则并纳入群报。 + +## 目标 + +- 将 1024×1536 从强制尺寸改为优先尺寸。 +- 接受其他能够完整解码、尺寸为正数且格式与扩展名一致的图片。 +- 使用现有候选认领流程，把茶馆 V3.0 与 Grok App 交流群的现有候选图纳入当天群报。 +- 认领后继续保持人工发送锁，不触发任何微信发送。 + +## 允许修改范围 + +- 图片合同校验与 Codex 生图结果落盘逻辑。 +- 群报图片 Prompt 中的画布尺寸说明。 +- 与上述规则直接相关的单元测试。 +- 本任务说明文件。 +- 当天茶馆 V3.0 与 Grok App 交流群的本地运行状态和正式图片产物。 + +## 禁止修改范围 + +- 候选图片的 job_id、来源、路径边界、SHA-256 与唯一候选校验。 +- 微信发送逻辑、发送锁、群目标身份校验与发送记录。 +- 调度时间、六群顺序、聊天事实、文案与视觉内容。 +- 为匹配尺寸而裁切、拉伸或重新编码候选原图。 +- 生产数据库迁移、远端部署和任何敏感配置。 + +## 已确定实现要求 + +1. Prompt 仍优先请求 1024×1536 的竖版 2:3 画布，但明确允许其他完整可读的竖版尺寸。 +2. 图片合同只校验文件存在、大小合理、可完整解码、尺寸为正数、实际格式与扩展名一致。 +3. 结果落盘不得再因宽高不是 1024×1536 而失败。 +4. 候选认领仍必须精确匹配当前运行的 job_id 与 candidate_id，并复核文件哈希和允许根目录。 +5. 认领完成后必须保持 `send_hold=true`、`needs_manual_send=true`，且不得调用发送接口。 + +## 验收标准 + +- 864×1821 的有效 PNG 能通过生图结果落盘和候选认领测试。 +- 损坏图片、路径越界、job_id 或哈希不匹配仍被拒绝。 +- 茶馆 V3.0 与 Grok App 交流群的 `daily_image.png` 与各自已确认候选图哈希一致。 +- 两个运行进入可人工审核/待发送状态，并保持发送锁；当天实际发送数不增加。 +- 相关测试、完整测试和 Python 编译检查通过。 + +## 测试命令 + +```powershell +.\.venv\Scripts\python.exe -m pytest tests/test_v2_image_task.py tests/test_v2_image_regeneration.py tests/test_v2_prompt_builder.py -q +.\.venv\Scripts\python.exe -m pytest tests -q +.\.venv\Scripts\python.exe -m compileall -q app tests +``` + +## 返回格式 + +- 修改规则摘要。 +- 茶馆 V3.0 与 Grok App 交流群的认领结果、尺寸、哈希和发送锁状态。 +- 测试结果、Git 提交与推送结果。 +- 尚未解决或明确未执行的事项。 diff --git a/GROUPBRIEF_GROUP27_28_IMAGE_RECOVERY_TASK.md b/GROUPBRIEF_GROUP27_28_IMAGE_RECOVERY_TASK.md new file mode 100644 index 0000000..694b696 --- /dev/null +++ b/GROUPBRIEF_GROUP27_28_IMAGE_RECOVERY_TASK.md @@ -0,0 +1,78 @@ +# 米游3.2 与 Eason 群报生图恢复任务 + +## 背景 + +2026-08-28： + +- 米游涩泛二次元同好摸鱼群3.2 的 ImageGen 调用因连接失败没有生成候选图。旧结构化回执只能填写 `image_path`，错误说明被写进该字段，系统只能按“结果未知”失败关闭。 +- Eason张UED-4群🤘 的漫画编辑稿引用了一条本身以省略号结尾的完整真实原话，被通用“悬空省略号”规则误判；三次编辑重试后进入本地简化信息图兜底，并已于 08:30 发送。 +- Prompt 失败降级时只保存了最小兜底元数据，丢失已计算的话题选择与分镜信息，后续不能使用常规安全重建路径。 + +用户要求查明并修复错误，为两个群重新生图，但今天不得再次发送。 + +## 目标 + +- 区分 ImageGen 的明确失败与真正的结果未知，避免错误说明伪装成图片路径。 +- 允许逐字引用本身以省略号结尾的完整原消息，但继续拒绝人为截断出来的悬空省略号。 +- Prompt 失败时保留已经形成的非敏感话题与布局元数据，便于之后安全重建。 +- 为缺少历史话题快照的旧兜底运行提供显式授权的“仅从已保存消息重新选题并重建 Prompt”恢复入口。 +- 按顺序为米游3.2、Eason 重新生图，保持两群今日发送锁，不调用任何发送接口。 + +## 允许修改范围 + +- `app/ai/poster_copy.py` +- `app/ai/prompt_builder.py` +- `app/pipeline/generation_stages.py` +- `app/pipeline/daily_pipeline.py` +- `app/v2/run_store.py` +- `app/image/codex_generator.py` +- `app/image/regeneration.py` +- `app/image/codex_image_result.schema.json` +- 与上述逻辑直接相关的测试。 +- 两群 2026-08-28 的本地 Prompt、生图任务状态和正式图片产物。 + +## 禁止修改范围 + +- 微信发送器、群目标身份、发送记录和已发送历史。 +- 当天 messages.json、聊天事实和候选图片归属/哈希/路径边界规则。 +- 调度时间、群配置、数据库结构、Secrets 和远端部署。 +- 自动重试结果未知的旧任务、猜测候选图或并行启动两个 ImageGen。 +- 解除 `USER_REQUEST_NO_SEND_2026_08_28` 发送锁。 + +## 已确定实现要求 + +1. ImageGen 结构化回执必须包含 `status`、`image_path`、`error`；成功只接受绝对图片路径，失败只接受空路径和非空错误。 +2. 明确失败返回 `outcome_unknown=false`，不得被登记为候选或自动猜图；同一任务仍只调用一次 ImageGen。 +3. 完整原消息逐字以省略号结尾时可作为真实气泡；任何仅截取原消息一部分并以省略号结尾的气泡仍拒绝。 +4. PromptBuilder 的已形成元数据随普通失败结果返回；本地兜底元数据在其基础上追加，不覆盖话题选择。 +5. 新 Prompt 成功落盘时必须清除旧的本地图片兜底标记；缺少旧话题快照时，只有显式 `allow_topic_reselection=true` 才能从保存的 messages.json 重新选题，不得重新读取微信。 +6. Eason 重建 Prompt 后再生图；米游3.2直接使用已验证的现有 Prompt 生图；两项严格串行。 +7. 生图完成后两群保持 `send_hold=true`、`needs_manual_send=true`，且本次不新增 `sent_at`、`text_sent_at` 或 `image_sent_at`。 +8. 重生图成功后必须清除旧的本地兜底标记，并记录本次尝试数量，避免面板把真实 ImageGen 新图继续显示成 Pillow 兜底图。 + +## 验收标准 + +- 新失败回执测试证明网络/工具明确失败不会再成为“结果未知”，旧版成功回执仍可兼容读取。 +- 省略号测试同时覆盖“完整原消息允许”和“截断片段拒绝”。 +- Prompt 失败兜底测试证明 `topic_selection` 等元数据仍保留。 +- 显式话题重选测试证明只复用保存消息、不取微信、不生图，并保留 SENT 历史和发送锁。 +- 重生图成功测试证明旧 `fallback_level`、`fallback_reason`、`image_variant` 和强制本地兜底开关均被清除。 +- 两张新图片可完整解码，正式文件与任务回执的 job_id、Prompt 哈希和 SHA-256 一致。 +- Eason 原 08:30 发送历史不改变；米游3.2仍无发送历史；两个群本次均无发送调用。 + +## 测试命令 + +```powershell +.\.venv\Scripts\python.exe -m pytest tests/test_v2_prompt_builder.py tests/test_v2_pipeline.py tests/test_v2_image_task.py -q +.\.venv\Scripts\python.exe -m pytest tests/test_v2_image_regeneration.py -q +.\.venv\Scripts\python.exe -m pytest tests -q +.\.venv\Scripts\python.exe -m compileall -q app tests +``` + +## 返回格式 + +- 两群已证实根因。 +- 代码修复与运行恢复结果。 +- 两张图片路径、尺寸、哈希、任务回执和发送锁证据。 +- 测试、Git 提交、分支与推送结果。 +- 已发送事实和未执行事项。 diff --git a/GROUPBRIEF_IMAGE_AUTOMATION_RECOVERY_TASK.md b/GROUPBRIEF_IMAGE_AUTOMATION_RECOVERY_TASK.md new file mode 100644 index 0000000..a8adbc7 --- /dev/null +++ b/GROUPBRIEF_IMAGE_AUTOMATION_RECOVERY_TASK.md @@ -0,0 +1,65 @@ +# GroupBrief 图片自动化恢复任务 + +## 背景 + +2026-08-31 的六群日报中，3 群图片成功，Grok 因 Codex 返回无效 JSON 被错误归入结果未知，米游涩泛 1.1 与 Eason 的 AI 图片被事实校验拦截；Eason 的第二次图片调用又遇到明确网络失败。现有流程没有在这些“结果已知但不可用”的终态统一落到本地安全信息图，导致整批生成和后续发送被阻断。 + +## 目标 + +- 已收到但格式无效的 Codex JSON 视为已知无效响应，不切换收费 Provider、不自动发起第二次外部调用，并允许 Pipeline 使用本地信息图兜底。 +- 严格图片事实校验连续失败，或质量重试明确失败时，生成并校验本地信息图。 +- 保留真正结果未知时的人工暂停，禁止自动重复外部调用。 +- 消除已确认的 OCR 数字误判：结构化头部数字、`块/元`货币同义和纯零符号误识别。 +- 恢复 2026-08-31 未完成群，图片成功后再进入发送；不重生成已成功图片。 + +## 允许修改范围 + +- `app/providers/ai/base.py` +- `app/providers/ai/codex.py` +- `app/image/fact_verification.py` +- `app/image/image_task.py` +- `app/pipeline/generation_stages.py` +- 对应的 `tests/` 测试文件 +- 本任务文件 + +## 禁止修改范围 + +- 不更换 Codex 模型、登录方式、认证或 Provider 配置。 +- 不读取或提交 `.env`、Token、Cookie、浏览器数据。 +- 不放宽未知图片结果、未知 Prompt 结果和未知发送结果的 fail-closed 规则。 +- 不改排行榜、摘要内容、群配置、发送目标和已成功图片。 +- 不自动合并 PR，不强推，不清理用户已有改动。 + +## 已确定实现要求 + +1. 新增“外部调用已返回但响应不可解析”的明确异常类型；它既不是未提交，也不是结果未知。 +2. Codex JSON 无效或不是对象时抛出该异常，禁止调用备用 Provider；PromptBuilder 把它收敛为普通失败，由现有本地信息图路径处理。 +3. 严格图片校验的第二次结果只有在结果已知时才允许本地兜底；结果未知继续失败关闭。 +4. 数字事实校验只补充确定性结构化头部证据，并做窄范围规范化，不把整个 AI Prompt 当作数字事实来源。 +5. 运行恢复前备份当天权威 `run.json`/调度状态；发送前逐群校验微信目标和图片文件哈希。 +6. 人工显式 `force_generate` 可以越过自动重试等待与已耗尽预算，但仍必须阻止所有人工核对状态；恢复时保留历史重试账本。 + +## 验收标准 + +- 无效 Codex JSON 不进入 `PROMPT_RESULT_UNKNOWN`，不调用 DeepSeek 备用，最终能走本地图片兜底。 +- 两次严格事实校验失败或第二次明确网络失败时，任务得到可解码、事实校验通过的本地 PNG。 +- 第二次图片结果未知时不生成替代图片、不再次调用外部图片 Provider。 +- Eason 群名 `4.1`、`38块/38元`以及 OCR 纯零误识别均有回归测试。 +- 显式强制恢复可从 `WAIT_RETRY`/`FAILED_FINAL` 继续执行，但不能越过 `HOLD_MANUAL`。 +- 聚焦测试与完整后端测试通过；`git diff --check` 通过。 +- 今日六群最终均有有效图片；发送只在目标匹配后执行，并以 `run.json` 与 UI 发送证据为准。 + +## 测试命令 + +```powershell +.\.venv\Scripts\python.exe -m pytest tests\test_codex_summary_provider.py tests\test_image_fact_verification.py tests\test_v2_image_task.py -q +.\.venv\Scripts\python.exe -m pytest tests -q +git diff --check +``` + +## 返回格式 + +- 根因与修复点 +- 测试命令、通过数和失败证据 +- 今日逐群图片/发送终态 +- Git 分支、提交、远端 SHA、PR 和 CI 状态 diff --git a/GROUPBRIEF_IMAGE_QUEUE_RECOVERY_TASK.md b/GROUPBRIEF_IMAGE_QUEUE_RECOVERY_TASK.md new file mode 100644 index 0000000..1ae67f3 --- /dev/null +++ b/GROUPBRIEF_IMAGE_QUEUE_RECOVERY_TASK.md @@ -0,0 +1,76 @@ +# GroupBrief 六群顺序生图恢复任务 + +## 背景 + +2026-08-28 的每日批次在 00:23 启动补偿后继续运行。用户要求 6 个启用群全部继续生图，并要求图片任务顺序执行。运行时已将 `image_generation_concurrency` 调整为 `1`，且调整发生在首个生图进程启动前。 + +群 24、27 在 Prompt 已成功保存后进入失败状态，错误为： + +`Can't emit change event for attribute 'Group.image_enabled' - parent object of type has been garbage collected.` + +六群随后均因同一错误进入等待重试，但失败状态的普通补写会不断重算 `next_retry_at`，导致到期时间持续后移，自动恢复无法真正开始。 + +## 目标 + +1. 修复群级运行覆盖配置复制后携带失效 SQLAlchemy 状态的问题。 +2. 修复相同失败快照补写时不断推迟既有重试时间的问题。 +3. 保持当日 Prompt、排行和消息快照可复用，不重复调用已经成功的阶段。 +4. 让 6 个群按 Prompt 就绪顺序进入单并发生图队列。 +5. 不触发微信发送。 + +## 允许修改范围 + +- `app/pipeline/daily_pipeline.py` +- `app/v2/reliability.py` +- 与该缺陷直接相关的自动化测试文件 +- 本任务说明文件 + +## 禁止修改范围 + +- 微信发送、发送状态和发送目标逻辑 +- Prompt 模板、选题、排行算法和图片内容规则 +- 历史运行目录中的已有消息、排行、Prompt 和图片产物 +- API Key、Token、密码、Cookie、`.env` 或浏览器数据 + +## 已确定实现要求 + +- 不再对从 SQLModel Session 加载的 `Group` 表对象直接使用会复制 `_sa_instance_state` 的 `model_copy()`。 +- 运行覆盖值必须应用到具有独立 SQLAlchemy 状态的新 `Group` 实例。 +- 保留覆盖字段白名单及结果顺序。 +- 添加可复现“数据库对象被回收后写属性”的回归测试。 +- 相同失败指纹的状态补写必须保留第一次计算出的 `next_retry_at`，不得无限推迟重试。 +- 恢复时只使用现有幂等/重试机制，不重启第二批任务。 + +## 验收标准 + +- 回归测试在修复前可复现 `ObjectDereferencedError`，修复后通过。 +- 相关并发与 V2 流水线测试通过。 +- 今日失败群可从 `PROMPT_SAVED` 检查点继续，最终 6 个 `daily_image.png` 均有效落盘。 +- 实际生图进程最大并发为 1。 +- `sent=0`，无微信发送副作用。 +- 数据库 `integrity_check=ok` 且 `foreign_key_check` 为空。 + +## 测试命令 + +```powershell +.\.venv\Scripts\python.exe -m pytest tests\test_generation_concurrency.py -q +.\.venv\Scripts\python.exe -m pytest tests\test_reliability_state.py -q +.\.venv\Scripts\python.exe -m pytest tests\test_v2_pipeline.py -q +.\.venv\Scripts\python.exe -m pytest tests -q +``` + +## 返回格式 + +- 根因与修复摘要 +- 6 个群的最终图片路径、校验结果和实际生图顺序 +- 测试结果 +- Git 提交、分支和推送结果 +- 明确说明未触发发送 + +## 2026-08-28 本轮执行结果 + +- 六个启用群均已进入单并发生图队列，实测同时存活的生图 Codex 进程最大为 1。 +- 合格落盘：米游涩泛二次元同好摸鱼群1.1、米游涩泛二次元同好摸鱼群2.3、Eason张UED-4群🤘（本地 Pillow 兜底）。 +- 人工保持：茶馆V3.0（三周年纪念）🐮🐴、Grok App 交流群的唯一候选均为有效 `864×1821` PNG，不符合固定 `1024×1536` 合同；米游涩泛二次元同好摸鱼群3.2 因网络断连且没有可信候选而结果未知。 +- 三个失败项均未自动重试，也未裁切、拉伸或认领不合格候选；需用户针对本次失败明确授权后再决定重新生图或使用本地兜底。 +- 微信发送计数为 0。 diff --git a/GROUPBRIEF_NEXT_STAGE_TASK.md b/GROUPBRIEF_NEXT_STAGE_TASK.md new file mode 100644 index 0000000..44d5acd --- /dev/null +++ b/GROUPBRIEF_NEXT_STAGE_TASK.md @@ -0,0 +1,84 @@ +# GroupBrief 自用可靠闭环与每周洞察实施任务 + +## 背景 + +GroupBrief 已具备六群日报生成、图片、归档和当天自动发送能力。2026-08-27 的可靠性整改把 watchdog 默认扩大到最近 30 天，真实服务启动后开始自动补历史任务并触发生图；同时完整测试与生产实例共用 `Local\\GroupBrief.Generation`，导致生产运行时测试无法执行。本任务按已确认方案将恢复窗口收敛为 48 小时、补齐状态真相和测试隔离，让群级配置真正生效，并在 14 天真实稳定性门槛后提供独立每周洞察。 + +## 目标 + +1. 最近 48 小时可自动恢复；更早欠账只读预览，确认后只恢复生成，绝不自动历史补发。 +2. 每日状态必须以稳定群 ID 的预期任务清单为准，不得因“已发现的群都成功”误报完成。 +3. 生产服务运行时，普通测试与仿真使用独立锁和临时运行根目录。 +4. 群级统计规则、历史数据源偏好、摘要 Provider/模型和 Prompt Provider/模型均进入实际执行链并写入审计状态。 +5. 旧版孤儿记录只读归档；不猜测关联、不删除、不参与新统计。 +6. 每周洞察聚合既有日报工件，AI 失败仍有确定性文本和本地卡片；周一独立排队发送。 + +## 允许修改范围 + +- `app/`：恢复规划、调度、状态投影、群配置、Provider 路由、周报聚合/归档/调度/API。 +- `frontend/`：恢复清单、群配置白名单、旧版历史归档、每周洞察只读页面及测试。 +- `tests/`、`scripts/simulate_reliability.py`：隔离、故障注入、仿真和周报测试。 +- 必要的兼容迁移、文档、`.env.example`、依赖约束和本任务文件。 + +## 禁止修改范围 + +- 不删除或批量改写 `data/`、`output/`、`runtime/` 中的真实历史和工件。 +- 不自动解除 `SEND_RESULT_UNKNOWN`、`PROMPT_RESULT_UNKNOWN` 或其他结果未知状态。 +- 不执行真实微信补发、邮件补发、真实历史重生成或付费 Provider 验收。 +- 不引入云端多用户、Docker 原生微信发送、全面 Pipeline/CSS 重写或 React/Vite/TypeScript 大版本升级。 +- 不读取、输出或提交 API Key、Token、密码、Cookie、`.env` 内容和完整聊天/Prompt。 + +## 已确定实现要求 + +### 恢复与状态 + +- 自动窗口为当前日期及前 1 天；更早任务只进入 backlog 预览。 +- `GET /api/v2/recovery/backlog` 只读；`POST /api/v2/recovery/confirm` 使用版本时间/CAS，只允许选中任务恢复生成，不发送。 +- 每日 manifest 保存稳定群 ID、显示名快照、规则、发送时间、目标身份和期望终态；旧状态惰性兼容。 +- readiness 只描述服务/依赖；daily outcome 使用 `running/complete/partial/blocked/needs_attention`，完整性判断必须比较预期任务清单。 + +### 测试隔离 + +- 生产生成锁名保持 `Local\\GroupBrief.Generation`。 +- 普通测试和仿真自动使用进程唯一的命名空间；专门的互斥测试可显式使用共享测试锁。 +- 数据库、`output`、`runtime`、scheduler 状态和外部 Provider 默认隔离。 + +### 群级配置 + +- `schedule_rule` 仅支持 `weekday_default` 与 `daily_previous_day`。 +- `provider_preference` 保留兼容但明确为历史数据源偏好；新 API/前端使用清晰名称。 +- 摘要与 Prompt 分别保存 Provider/模型；空值继承全局。只允许健康检查/配置注册表返回的白名单组合。 +- 请求值、实际 Provider/模型、fallback 原因写入 `run.json`。 + +### 每周洞察 + +- 周期为上一自然周周一至周日，基于非重叠日报周期和稳定发言人身份聚合。 +- 每群最多一次 AI 周度叙述调用；失败时生成确定性统计文本和 Pillow 本地卡片。 +- 周报拥有独立状态、claim、工件和归档；周一 08:30 进入现有串行发送边界，日报与周报互不冒充成功。 + +## 验收标准 + +1. GroupBrief 停止时与运行时，完整后端测试均通过；运行时测试不竞争生产锁。 +2. 30 天 × 6 群仿真：预期任务零永久丢失、零重复生图、零重复发送、无无限重试；结果未知永久 hold。 +3. 48 小时边界、过期确认、状态半写、图片落盘后崩溃、发送前/后失败和 Provider fallback 有故障注入证据。 +4. 前端单测、生产构建、恢复/群配置/每周洞察 E2E 通过，390px 宽度可操作。 +5. 旧数据库和旧 `run.json` 可读；迁移后 SQLite `integrity_check` 与 `foreign_key_check` 通过。 +6. 实际上线只恢复当天自动任务，不执行历史发送；灰度群必须在执行时再次由用户确认。 + +## 测试命令 + +```powershell +.\.venv\Scripts\python.exe -m compileall -q app scripts tests +.\.venv\Scripts\python.exe -m pytest tests -q +.\.venv\Scripts\python.exe scripts\simulate_reliability.py --days 30 --seed 20260827 --groups 6 +npm test --prefix frontend +npm run build --prefix frontend +npm run test:e2e --prefix frontend +git diff --check +``` + +## 返回格式 + +- 按恢复/状态、测试隔离、群级配置、历史归档、每周洞察报告行为变化和测试证据。 +- 明确区分 Mock/仿真、真实本地运行和真实外部发送；不得把测试通过描述为真实送达。 +- 报告 Alter 服务状态、Git 分支、提交哈希、远端地址和推送结果，以及尚未完成的 14 天观察项。 diff --git a/GROUPBRIEF_PR1_CI_MERGE_TASK.md b/GROUPBRIEF_PR1_CI_MERGE_TASK.md new file mode 100644 index 0000000..d145fb2 --- /dev/null +++ b/GROUPBRIEF_PR1_CI_MERGE_TASK.md @@ -0,0 +1,48 @@ +# GroupBrief PR #1 CI 合并任务 + +## 背景 + +PR #1 可合并且没有未解决评论，但 Windows CI 的前端单测环境和后端跨进程状态测试失败，禁止在检查未通过时直接合并。 + +## 目标 + +- 让前端 CI Node 版本满足当前锁定的 `jsdom` / `undici` 引擎要求。 +- 修复 Windows 下运行状态文件原子替换偶发访问拒绝与损坏隔离问题。 +- 在全部必需 CI 通过后使用 Squash 合并 PR #1。 + +## 允许修改范围 + +- `.github/workflows/ci.yml` +- `app/v2/run_store.py` +- `app/scheduler/daily_v2_job.py` +- `tests/test_cross_process_state.py` +- `app/core/path_security.py` +- `tests/test_path_security.py` + +## 禁止修改范围 + +- 不跳过、删除或弱化 CI 检查。 +- 不改变调度时间、生成、发送、恢复或 Provider 行为。 +- 不启动生产服务，不生成图片，不发送微信或邮件。 +- 不改写 Git 历史，不强推，不触碰原工作区已有审计改动。 + +## 已确定实现要求 + +- CI 使用精确 Node `22.22.2`。 +- Windows named mutex 明确声明 ctypes 参数和返回类型，并显式处理等待失败。 +- 状态文件写入使用 PID 与 UUID 组成的唯一临时文件；成功或异常后均清理残留。 +- 原子替换仅对仍存在源文件的瞬时 `PermissionError` 做有限短退避重试；永久失败必须继续抛出。 +- Windows 路径边界比较先规范化等价的 extended-length 前缀，不放宽导航、盘符、UNC 或符号链接边界。 +- named mutex 使用规范化后的真实路径摘要，避免同一路径因 `\\?\` 表示差异生成两把锁。 +- `DailyScheduleState` 与 `RunStore` 共用同一原子文本写入实现。 + +## 验收标准与测试 + +- `tests/test_cross_process_state.py` 连续多轮通过，且不残留临时文件。 +- 完整后端 CI 等价测试通过。 +- `npm ci`、`npm run build`、`npm test`、Playwright Fake API E2E 通过。 +- 远端 PR #1 的必需检查全部成功，随后才执行 Squash 合并。 + +## 返回格式 + +报告修改文件、测试命令与结果、提交 SHA、远端 SHA、PR/CI/合并状态，以及原工作区是否保持不变。 diff --git a/GROUPBRIEF_RANKING_MEMBER_NAME_STABILITY_TASK.md b/GROUPBRIEF_RANKING_MEMBER_NAME_STABILITY_TASK.md new file mode 100644 index 0000000..47b9926 --- /dev/null +++ b/GROUPBRIEF_RANKING_MEMBER_NAME_STABILITY_TASK.md @@ -0,0 +1,61 @@ +# 排行榜成员名称稳定性修复任务 + +## 背景 + +- 茶馆两个不同成员被旧 WDA 错误解析为同一个“鲁布斯”，GroupBrief 随后生成了虚假的同名编号。 +- Grok 的群主/邀请人字段 `c2341298` 被旧 WDA 当成大量成员的群名片。 +- GroupBrief Provider 已有联系人回退，但群级名称策略会再次写回错误的上游名称。 +- Eason 的名称数据完整，前端单行省略样式导致长群名片看起来不完整。 + +## 目标 + +- 最终名称采用“有效群名片原样优先，异常或缺失时回退联系人，最后稳定匿名名”。 +- 统计身份始终以 `sender_id` 为准，只对最终仍真实同名的成员添加稳定编号。 +- 排行榜相关页面完整换行显示成员名称。 +- 同步修复 WDA 源码的 `chat_room.ext_buffer` 字段解析，但不替换正式安装。 + +## 允许修改范围 + +- GroupBrief 的联系人/群名片解析、WDA Provider 名称解析、群级名称策略及相关测试。 +- 排行榜页、Dashboard 排行预览及对应样式。 +- 隔离 WDA 工作树中的群名片解析函数和相关测试。 + +## 禁止修改范围 + +- 不修改排行榜模板、排名口径、数据库结构或 API/产物字段结构。 +- 不刷新或改写 2026-08-28 及历史 `output` 产物。 +- 不触发摘要、生图、微信发送、历史补跑或生产数据库迁移。 +- 不替换或重启正式 WDA 2.2.1，不重启 GroupBrief 8766。 +- 不修改 WDA 原工作区已有未提交内容，不重写 Git 历史或强制推送。 + +## 已确定实现要求 + +- `contact.db.chat_room.ext_buffer` 按字段 1=成员 ID、字段 2=群名片、字段 4=邀请人/群主解析。 +- 一旦看到字段 1 的明确成员记录，不允许字段 4 兼容猜测覆盖该成员。 +- GroupBrief 应直接读取群名片映射，使正式 WDA 尚未更新时后续任务也能正确解析。 +- 联系人库明确返回的名称即使大小写与成员 ID 相同，也视为可信联系人名称。 +- 群级策略保留 Provider 已解析的名称和来源，不再用 `upstream_sender_name` 覆盖。 +- 长名称完整换行，并提供原始完整名称的 `title`。 + +## 验收标准 + +- 茶馆 `jiangzhema123` 显示“鲁布斯”，`to1900` 回退显示“罗斯”。 +- Grok 不再将数十个不同成员统一显示为 `c2341298`。 +- Eason 长群名片在排行榜详情与 Dashboard 预览中完整显示；数据库原值“广州”保持不变。 +- 真同名仍稳定编号，消息数、发言人数和排名算法不变。 +- 正式历史产物和运行服务零改动。 + +## 测试命令 + +- GroupBrief：`python -m pytest tests/test_contact_resolver.py tests/test_wechat_mcp.py tests/test_sender_name_policy.py -q` +- GroupBrief：`python -m pytest tests -q`（限定正式测试目录，排除 `data/audit-snapshot-*` 历史源码快照） +- 前端：`npm --prefix frontend test`、`npm --prefix frontend run build` +- Python 编译：`python -m compileall -q app tests` +- WDA 隔离工作树：`python -m pytest tests/test_group_nickname_ext_buffer_parsing.py -q` +- 两仓库：`git diff --check` + +## 返回格式 + +- 报告根因、实际修改、针对性/全量测试结果、只读真实数据核验结果。 +- 报告两个仓库的分支、提交哈希、远端地址和普通推送结果。 +- 明确说明未刷新历史产物、未替换 WDA、未重启服务、未触发任何发送或生图。 diff --git a/GROUPBRIEF_RELIABILITY_AUDIT.md b/GROUPBRIEF_RELIABILITY_AUDIT.md new file mode 100644 index 0000000..6c2a7fb --- /dev/null +++ b/GROUPBRIEF_RELIABILITY_AUDIT.md @@ -0,0 +1,769 @@ +# GroupBrief 无人值守稳定性审计报告 + +> 审计日期：2026-08-27（Asia/Shanghai） +> 代码基线：`f7f391824c03d8be45b20558cee6e5d74a2ce20c` +> 审计目标：判断当前系统在无人查看、无人重试、无人临时消歧的条件下，连续运行 30 天时能否稳定完成全部群的读取、统计、整理、Prompt、生图、保存和微信发送。 +> 本轮边界：只审计并新增本报告；未修改生产代码、测试、配置、数据库、Windows 任务或运行状态；未调用收费 AI、未生图、未发送微信或邮件。 + +## 结论摘要 + +当前 GroupBrief 已经具备一批重要的安全保护：逐群 `run.json`、阶段状态、原子替换、损坏状态 fail-closed、Prompt/发送 claim、未知发送结果人工锁、群级生成隔离、图片候选归属和哈希校验、当天启动补偿等。这些措施使系统比“单个 `run_daily_task()` 黑盒”可靠很多，也显著降低了重复付费生图和重复发送的风险。 + +但当前版本**尚未达到 30 天无人值守标准**。决定性问题不是单次正常路径，而是失败后的自动收敛： + +1. 任意正常返回的失败或部分失败批次都会写入 `generation_completed_at`；普通 `FAILED` 群又被启动恢复扫描排除，临时故障可能因此成为永久欠账。 +2. 启动补偿和发送扫描都只看当天，程序跨日停机后不会自动补历史任务；当前机器也没有可证明有效的 Windows 开机启动链。 +3. AI 生图只有原 Prompt 路径，没有安全化 Prompt 和本地确定性信息图降级；内容拒绝或模型故障仍会导致当天无图。 +4. 外部发送与本地状态提交之间仍有未检查的 claim 更新窗口；系统虽然会对未知结果 fail-closed，但无法无人值守地完成消歧。 +5. 批次状态与六个逐群状态缺少统一可信终态。审计时真实状态是“六群均 `SENT`，批次仍 `blocked`”。 + +隔离仿真使用固定种子 `20260827`，运行 30 天 × 6 群共 180 个任务单元，保留真实 `DailyPipeline`、`RunStore`、`DailyScheduleState` 和发送阶段，Mock 外部取数/AI/生图/微信。最终 130 个到达 `SENT`，21 个 `FAILED`，17 个图片就绪但从未发送，12 个因离线日根本未建档；没有发现无限 retry 或重复发送。**50/180 个任务未完成，说明当前实现偏向“安全停住”，还没有“自动恢复并最终完成”的闭环。** + +--- + +## 1. 当前真实架构 + +### 1.1 运行形态与真实基线 + +审计在 2026-08-27 10:20（北京时间）记录到： + +- `127.0.0.1:8766` 的真实监听链为：`python.exe -m uvicorn app.main:app`（PID 211548）← `.venv\Scripts\python.exe -m uvicorn`（PID 69232）← `alter.exe --internal-daemon`（PID 48056）。 +- Alter 中 `GroupBrief-Backend` 为 running、enabled、autorestart，当前 restart count 为 2；这能覆盖 Alter 仍存活时的子进程崩溃。 +- `/api/system/health` 返回 200/`ok`；`/api/system/ready` 返回 ready；`/api/system/status` 显示 scheduler owner=`fastapi`、scheduler active=`true`、6 个启用群、下一次生成时间为次日 00:15。 +- WeChatDataAnalysis 的本地健康端点 `127.0.0.1:10392/api/health` 当时为 healthy。 +- Windows 任务 `GroupBriefDaily` 与 `GroupBriefDailySend` 均为 Disabled、Interactive only、`StartWhenAvailable=false`、无失败重启策略；未发现 GroupBrief/Alter 对应 Windows Service、Run 启动项或 Startup 快捷方式。 +- 因此，**已证实的是 Alter 对已启动子进程的守护，不是 Windows 重启后的自动拉起**。 + +实时证据只表示审计时刻的状态，不等价于未来 30 天持续健康，也不证明外部 AI 或微信发送成功。 + +### 1.2 主要组件 + +| 组件 | 责任 | 主要代码位置 | 持久化 | +|---|---|---|---| +| FastAPI 生命周期 | 初始化配置、目录、数据库、启动检查和内置 Scheduler | `app/main.py:50-89` | 数据库、日志 | +| APScheduler owner | 00:15 生成、每分钟扫描发送、当天启动补偿 | `app/scheduler/manager.py:54-149` | `.scheduler/.json` | +| 每日任务包装器 | 生成锁、批次状态、结果汇总、邮件决策 | `app/scheduler/daily_v2_job.py:163-330` | `.scheduler/.json` | +| DailyPipeline | 多群并发、逐群阶段编排、图片任务、发送扫描 | `app/pipeline/daily_pipeline.py:68-453` | 逐群输出目录 | +| 微信数据源 | MCP/导出读取、分页、去重、总时限 | `app/data_sources/wechat_data_analysis.py:41-138`、`app/providers/history/wechat_data_analysis.py:257-579` | `messages.json` | +| 统计与 Prompt | 排行、候选话题、结构化 Prompt、事实校验 | `app/pipeline/generation_stages.py:229-716`、`app/ai/prompt_builder.py:207-587` | ranking、prompt、run.json | +| 生图执行器 | Codex 进程、job/claim、候选归属、验证、原子提升 | `app/image/codex_generator.py:272-1455`、`app/image/image_task.py:69-266` | 图片、attempt manifest、run.json | +| 微信发送阶段 | 目标预检、文字/图片 claim、未知结果锁 | `app/pipeline/delivery_stages.py:38-483` | run.json | +| RunStore | 状态机、原子 JSON 写入、Prompt/发送 claim | `app/v2/run_store.py:117-812` | `output///run.json` | +| 恢复服务 | 启动扫描未完成任务、未知发送 fail-closed | `app/v2/recovery.py:49-199` | 读取/推进 run.json | + +### 1.3 当前状态权威来源 + +- 逐群事实权威：`output/<群>//run.json`。 +- 批次/邮件事实：`output/.scheduler/.json`。 +- 当前缺少一个把二者一致性校验后汇总的“每日可信终态”。 +- 2026-08-27 的真实样本中，批次文件为 `generation_status=blocked`、`email_status=skipped_generation_not_successful`，但当前六个群的 `run.json` 都是 `SENT`。这证明两层状态可能长期分叉。 + +--- + +## 2. 每日任务完整调用链 + +### 2.1 当前生产 DAG + +```text +FastAPI lifespan + └─ start_scheduler() + ├─ 00:15 run_daily_v2_job(date) + │ ├─ generation_mutex (thread + Windows named mutex) + │ ├─ DailyScheduleState.load(date) + │ ├─ 当 generation_completed_at 不存在时： + │ │ └─ DailyPipeline.generate_all(date) + │ │ ├─ PeriodResolver -> [前一自然日 00:00, 当日 00:00) + │ │ ├─ GroupNameSyncService + │ │ ├─ repo.list_groups(only_enabled=True) + │ │ └─ 每群 worker（群间隔离） + │ │ ├─ 载入/初始化 run.json + │ │ ├─ WeChatDataAnalysisSource.fetch_messages() + │ │ ├─ 保存 messages.json + │ │ ├─ RankingEngine + RankingRenderer + │ │ ├─ 保存 ranking.json / ranking.txt + │ │ ├─ GroupSummaryImagePromptBuilder.build() + │ │ ├─ 保存 image_prompt.txt + prompt_meta + │ │ └─ ImageStages + │ │ └─ CodexImageGenerator.generate() + │ │ ├─ 进程/线程 job claim + │ │ ├─ imagegen API/CLI + │ │ ├─ 候选归属、路径、哈希、PIL/尺寸检查 + │ │ └─ os.replace -> daily_image.png + │ ├─ 写 generation_completed_at（目前成功/部分/失败都写） + │ └─ 成功时进入邮件；失败/blocked/partial 可跳过邮件 + └─ 每分钟第 15 秒 DailyPipeline.send_due(now) + ├─ 只读取 now.date() 的 run.json + ├─ 到点且不超过 late window + └─ DeliveryStages + ├─ verify target / image preflight + ├─ text claim -> send_text -> finish claim + ├─ image claim -> send_image -> finish claim + └─ SENT，或 SEND_RESULT_UNKNOWN/manual hold +``` + +### 2.2 节点级失败点地图 + +缩写：`有/无/部分`；“恢复”指进程崩溃后无需人工即可继续；“永久丢失”指在当前自动调度策略下可能永远不再推进。 + +| 关键步骤 | 输入 | 输出 | 外部依赖 | 异常捕获 | timeout | retry | fallback | 状态持久化 | 崩溃后恢复 | 重复执行副作用 | 当天永久丢失 | +|---|---|---|---|---|---|---|---|---|---|---|---| +| Scheduler 触发 | 当前时间、配置 | 每日 job | APScheduler、常驻进程 | 有，顶层记 failed | misfire 30 分钟 | 无队列 | 仅当天启动补偿 | 批次 JSON | 部分 | 锁冲突会直接放弃 | **高**，跨日不补 | +| 日期范围 | run_date、时区 | 前一自然日窗口 | zoneinfo | 有效日期校验 | 不适用 | 无 | 无 | 写入 run.json | 有 | 幂等 | 低；但业务若需非自然日会错数 | +| 群识别/群名同步 | DB 群、WDA 群列表 | 绑定与发送目标 | SQLite、WDA | 批次前置异常未隔离 | WDA 子调用有 | 无 | 手工 target | run.json 审计字段/DB | 部分 | 全局失败可阻断所有群 | **高** | +| 聊天读取 | group id、时间窗 | V2Message[] | WDA MCP/导出 | 有，单群 FAILED | 单次 + 总时限 | 无请求级 retry | 仅配置选择；MCP 运行时不切 CLI | `messages.json`、run.json | 快照已保存则可复用 | 快照避免重抓 | **高**，FAILED 自动排除 | +| 内容过滤/快照解析 | 消息列表 | 合法消息快照 | 本地文件 | 有，损坏 fail-closed | 不适用 | 无 | 不隐式重抓 | 原子 messages.json | 有条件 | 重复复用无副作用 | 中；损坏需人工 | +| 排行榜统计 | messages | ranking result/text | 本地 CPU | 有，单群 FAILED | 无 | 无 | 无 | ranking.json/txt、run.json | 文件已落盘可复用 | 基本幂等 | 高，FAILED 被封存 | +| AI 内容整理/选题 | messages、排行 | topic selection | Codex/DeepSeek | 解析错误有重试；调用异常部分不在重试块 | 有 | 部分 | Codex/DeepSeek 配置 fallback，未知结果 fail-closed | operation claim、prompt_meta | 部分；未知结果人工 | 盲重试可能重复付费，因此被锁 | **高** | +| Prompt 渲染 | 结构化选题、昵称、引用 | image_prompt.txt | 本地 + 上游 AI | Schema/事实检查有 | 上游有 | 格式重试部分有 | 无安全化 Prompt | 原子 prompt 文件、hash、run.json | 已保存可续 | 输入 hash 降低重复 | 高，超长/敏感内容无降级 | +| 生图调用 | prompt file | 候选图片/回执 | Codex imagegen | 分类捕获 | 默认 1200 秒 | 仅确认 start_failed 最多 2 次 | 无 L2/L3 | job/attempt manifest、claim | 部分；未知回执 hold | 避免重复付费 | **高** | +| 图片返回/归属 | job/thread candidate | 可信候选 | Codex 线程输出 | 有 | 属于总调用时限 | 无独立下载 retry | 同线程候选恢复 | manifest/hash | 部分 | 强归属避免串群 | 高，失败即无图 | +| 图片有效性检查 | 图片文件 | valid/invalid | Pillow | 新图强、复用/发送前弱 | 不适用 | 无 | 无 | 校验结果部分记录 | 部分 | 截断 PNG 可能被复用 | 中高 | +| 图片保存/归档 | 临时图片 | daily_image.png | NTFS、磁盘 | 原子替换前后捕获不完整 | 不适用 | 无 | 保留部分候选 | 文件 + run.json | 存在“文件成功/状态失败”窗口 | 可触发重复生图或 hold | 高 | +| 微信文字发送 | 排行文本、target | SendResult | Windows 微信 UI | sender 内有；claim 更新结果有未检查路径 | 阶段/提交/互斥有 | 明确未提交可重试；未知不重试 | 无 | claim、sent/unknown 字段 | 未知提交需人工 | fail-closed 降低重复 | 中；会人工 hold | +| 微信图片发送 | daily_image.png、target | SendResult | Windows 微信 UI | 同上 | 有 | 明确失败可续图片且不重发文字 | 无 | claim、SENT/unknown | 部分 | 状态提交失败可能重复 | 中高 | +| 当天完成记录 | 群 run + 批次结果 | completed/SENT | 文件系统 | 有 | 不适用 | 无 | 无一致性汇总 | 两套 JSON | **不可靠** | 分叉导致错误判定 | **高** | + +### 2.3 目标 DAG（最小增量，不换框架） + +```text +Watchdog 扫描最近 30 天（旧 -> 新） + └─ 为每个 date × enabled_group 建立/核对 task identity + ├─ 读取 last_successful_checkpoint + input_hash + ├─ 若 execution_state=WAIT_RETRY 且 now>=next_retry_at：续跑当前阶段 + ├─ 若 HOLD_MANUAL / SEND_RESULT_UNKNOWN：保持 fail-closed 并进日报 + └─ 按 checkpoint 推进 + DATA -> RANKING -> SUMMARY -> PROMPT + -> IMAGE L1 + ├─ 可明确分类的安全拒绝：IMAGE L2 safe prompt + └─ L2 仍失败：IMAGE L3 local infographic + -> IMAGE_SAVED -> SEND_TEXT -> SEND_IMAGE -> SENT + 每一步：原子业务产物 + attempt ledger + checkpoint + +Daily reconciler + └─ 汇总逐群 run.json，校验批次状态，原子写 runtime/YYYY-MM-DD/status.json +``` + +--- + +## 3. 当前已有的稳定性措施 + +以下能力已经存在，后续整改不应重复造轮子： + +1. **逐群状态机和输出隔离**：`PENDING → DATA_READY → RANKING_READY → PROMPT_READY → IMAGE_READY → READY_TO_SEND → SENT/FAILED`，每群每天独立目录。 +2. **原子状态写入**：`RunStore`、`DailyScheduleState` 和图片提升使用临时文件 + `os.replace`；JSON 损坏时返回 `CORRUPT` 并阻止自动覆盖。 +3. **消息快照复用**：`messages.json` 存在时默认不重新读取微信，避免重启导致数据漂移；快照损坏不会静默回源。 +4. **多群生成隔离**：群 worker 和图片 worker 捕获单群异常，单个群的多数阶段故障不会直接终止其他群。 +5. **Prompt 幂等保护**：记录 operation id/input hash；模型调用结果未知时暂停，不盲目重复计费调用。 +6. **图片任务归属保护**：job id、thread id、prompt hash、候选路径约束、SHA、PIL 解码和目标尺寸验证；进程超时会尝试终止进程树并核对可信候选。 +7. **生图有限重试**：仅在确认进程没有成功启动的 `start_failed` 场景最多尝试 2 次；已启动但结果未知不会自动重试。这是正确的成本/幂等边界，不应改成无脑 retry。 +8. **发送 claim 与未知结果锁**：文字和图片分阶段记录；文字已确认而图片失败时可只续图片；`SEND_RESULT_UNKNOWN` 永久 fail-closed，不自动重发。 +9. **目标与图片预检**：发送前核对群目标、候选归属、文件存在、hash/尺寸等部分信息，降低串群和错图风险。 +10. **当天启动补偿**：FastAPI Scheduler 启动时会检查当天批次，在未写完成标记时补跑生成。 +11. **并发约束**：生成有线程锁和 Windows 命名互斥；发送桌面操作也有互斥，减少多进程同时控制微信。 +12. **测试覆盖基础较好**：本次完整后端测试 597 项通过，包含未知结果、claim、恢复、并发和发送幂等测试；前端 21 项通过。 + +这些保护解释了为什么仿真没有出现重复发送和无限重试；但它们也让临时失败更容易进入永久停止，因此必须补上“有预算的自动恢复”和跨日 watchdog。 + +--- + +## 4. P0 风险 + +### P0-1：失败/部分失败批次被永久标记为“生成已完成” + +- **问题**：只要 `DailyPipeline.generate_all()` 正常返回，即使结果是 `partial`、`failed` 或 `blocked`，调度层仍写 `generation_completed_at`。启动补偿看到该字段后不再生成；恢复扫描又明确排除普通 `FAILED`。 +- **为什么发生**：批次“本次调用已结束”和“所有群已完成”共用了同一个 completion marker；阶段状态和执行重试状态没有正交拆分。 +- **代码位置**：`app/scheduler/daily_v2_job.py:231-293`、`app/scheduler/manager.py:115-148`、`app/v2/recovery.py:70-100`。 +- **如何复现**：让任意群第一次取数 timeout 或 Prompt 返回错误；每日 job 正常返回 partial；检查 `.scheduler/.json` 已有 `generation_completed_at`，该群 `run.json=FAILED`；重启服务，自动补偿不会再次处理。 +- **修改方案**：保留“invocation completed”，新增 `batch_terminal_at` 或按逐群状态计算真正完成；`FAILED_RETRYABLE` 必须保留 retry budget、`next_retry_at` 和 last checkpoint；watchdog 根据逐群事实而不是批次 marker 补跑。 +- **修改风险**：错误分类会造成重复付费调用。Prompt/生图“结果未知”必须继续 hold，只对明确未提交或确定失败自动重试。 +- **验证成功**：注入一次性 timeout 后，第一次批次为 partial，重启在 `next_retry_at` 后只重跑失败群，成功群不重读、不重算、不重生图，最终统一状态为完成。 + +### P0-2：只补当天任务，跨日停机后任务永久丢失 + +- **问题**：启动恢复只检查当天；发送扫描也只构造 `now.date()`。错过 00:15 或发送时间并跨日后，历史欠账不再进入自动路径。 +- **为什么发生**：Scheduler 设计假设进程一直在线；APScheduler misfire 只保留 30 分钟，发送 late window 也只有 30 分钟。 +- **代码位置**：`app/scheduler/manager.py:54-149`、`app/pipeline/daily_pipeline.py:404-453`、`app/pipeline/daily_pipeline.py:941-1020`。 +- **运行证据**：两项 Windows 任务均 Disabled、Interactive only、无 `StartWhenAvailable`/`RestartOnFailure`；未发现可证明的 Windows 开机拉起机制。Alter 只能证明当前守护关系。 +- **如何复现**：在某日 00:15 前停止服务，次日再启动；观察只检查次日 run_date，昨日没有 `.scheduler`/`run.json`。或图片已准备后跨过发送日，后续 `send_due` 永远不扫描该日期。 +- **修改方案**：启动和周期 watchdog 扫最近 30 天，旧到新创建/核对逐群任务；生成和“明确尚未提交”的发送可补跑；未知提交继续人工消歧。Windows 启动机制需单独建立并可验证。 +- **修改风险**：历史群配置或群名可能已变化；任务 identity 应使用稳定 group id + report date，发送目标使用当天已持久化快照并重新做 action-time preflight。 +- **验证成功**：模拟关机 48 小时后启动，昨日及前日缺失生成按旧到新补齐；已经 `SENT` 不重发；`SEND_RESULT_UNKNOWN` 不自动重发；日报明确列出人工 hold。 + +### P0-3：AI 生图没有 Level 2/Level 3 降级 + +- **问题**：原 Prompt 被内容审核拒绝、昵称/引用敏感、模型不可用或候选无效时，系统只记录失败；没有安全化 Prompt，也没有本地简化信息图。 +- **为什么发生**：`CodexImageGenerator` 始终发送同一 Prompt，`ImageStages` 只有成功/失败状态；现有 regeneration 是人工审核流程，不是自动降级。 +- **代码位置**：`app/image/codex_generator.py:898-909`、`app/image/image_task.py:169-200`、`app/pipeline/image_stages.py:63-137`、`app/image/regeneration.py:59-401`。 +- **如何复现**：在话题引用中放入会触发审核的原话，或让 imagegen 返回审核拒绝；检查 run 进入 `IMAGE_GENERATION_FAILED`，无第二 Prompt、无本地图。 +- **修改方案**：Level 1 保持现有 Prompt；仅对可明确分类的安全/内容拒绝进入 Level 2，从同一结构化 topic selection 生成安全化 Prompt，泛化昵称、原话和风险表达但保留 topic id、事实与排行；仍失败则 Level 3 用 Pillow 按结构化数据确定性渲染信息图，不再调用外部模型。 +- **修改风险**：过度清洗会改变当天主题；Level 2 必须保存原/安全 Prompt、变更原因和结构化事实 hash，验证核心 topic id 集合不变。 +- **验证成功**：构造审核拒绝，L1 失败后 L2 自动成功；再让 L2 失败，L3 仍产出 1024×1536 可发送图片；三层 attempt ledger 完整且主题/排行一致。 + +### P0-4：V2 微信数据源运行时故障可让所有群同日失败，且无自动后备 + +- **问题**：V2 固定构造 `WeChatDataAnalysisSource`。MCP 已配置但运行时 timeout/服务崩溃时不会切换到另一个读取后端，也没有请求级 retry；共用依赖故障可同时击穿全部群。 +- **为什么发生**：当前 fallback 主要在“配置时选择”和 MCP 不支持范围方法时的同后端分页兼容，不是运行时 provider fallback。 +- **代码位置**：`app/pipeline/daily_pipeline.py:80-88`、`app/data_sources/wechat_data_analysis.py:41-138`、`app/providers/history/wechat_data_analysis.py:257-369`。 +- **如何复现**：保持 MCP client 配置有效但让 10392 在 fetch 时断开；六群分别得到读取失败，批次随后仍可能写 completed marker。 +- **修改方案**：先加入同 provider 的有限指数退避；只有能证明两个后端返回相同语义/日期边界且配置允许时，才切换备用读取器。每次 fallback 记录 provider、错误分类、数据窗口和去重指标。 +- **修改风险**：不同后端消息 ID、时区或分页语义不一致会造成重复/漏数；fallback 不能静默拼接两份不兼容快照。 +- **验证成功**：主读取器前两次 timeout 后第三次成功；或受控切换备用后，消息窗口、去重数、群 identity 均通过合同测试，其他群不受影响，批次不被提前封存。 + +--- + +## 5. P1 风险 + +### P1-1：图片发送 claim 的关键更新结果未检查 + +- **问题**：文字发送开始会检查 `update_send_claim` 返回值，但部分完成更新、图片开始和图片完成路径忽略布尔结果，仍继续外部发送或返回 sent。 +- **为什么发生**：状态写入被当作附属记录，而不是外部动作的事务前置/后置条件。 +- **代码位置**：`app/pipeline/delivery_stages.py:215-329`、`app/pipeline/delivery_stages.py:381-442`、`app/v2/run_store.py:619-654`。 +- **如何复现**：注入 stale claim 或让 `update_send_claim` 返回 false；观察图片仍可能发送，或外部成功但本地未形成可靠 SENT。 +- **修改方案**：所有 pre-submit claim 更新失败必须禁止发送；post-submit 更新失败必须进入 `SEND_RESULT_UNKNOWN`，保存不可变 attempt id，不得返回普通 sent。 +- **修改风险**：错误地把“状态写失败但外部未提交”归为 unknown 会增加人工 hold，但比重复发送安全。 +- **验证成功**：对每个 claim 更新点注入 false/OSError；提交前零外部调用，提交后失败一律 unknown；重启不重复发送。 + +### P1-2：批次状态与逐群状态没有一致性收敛 + +- **问题**：真实环境出现六群全 `SENT`、批次仍 `blocked`；健康接口和邮件决策无法给出唯一可信答案。 +- **为什么发生**：批次文件只记录某次生成调用，后续人工/恢复/发送不会统一重算批次终态。 +- **代码位置**：`app/scheduler/daily_v2_job.py:215-330`、`app/scheduler/daily_v2_job.py:521-617`、`app/api/system.py:22-98`。 +- **如何复现**：先产生 partial/blocked，再通过后续恢复或手工路径使群任务全部 SENT；批次状态保持旧值。 +- **修改方案**：新增只读 reconciler，以逐群 run 为事实源计算 `complete/partial/held/missing`，原子写日报；不要反向覆盖逐群历史。 +- **修改风险**：群配置当天变化会改变期望集合，需保存 daily task manifest。 +- **验证成功**：任一路径推进后，reconciler 给出一致终态；篡改/缺失一个 run 时日报明确 degraded，不伪报成功。 + +### P1-3：发送扫描缺少最外层逐群异常隔离 + +- **问题**：`send_due` 串行循环没有包住整个单群 load/路径/持久化/DeliveryStages 调用；sender 自身异常虽被处理，但更外层异常可阻断后续群。 +- **为什么发生**：生成路径有 worker 级隔离，发送路径没有对称设计。 +- **代码位置**：`app/pipeline/daily_pipeline.py:404-453`、`app/pipeline/delivery_stages.py:56-483`。 +- **如何复现**：让第一个 due 群的 run.json 读取或状态更新抛异常；验证后续群本轮没有发送。 +- **修改方案**：每群完整 send transaction 外包 `try/except`，记录该群 retryable failure 后继续；同时保留桌面互斥避免并发控制微信。 +- **修改风险**：异常后微信 UI 可能处于未知界面；仅当 sender 明确未提交且可恢复 UI 时继续下一个群，否则整个桌面发送器应短暂熔断并留下 backlog。 +- **验证成功**：第一群 pre-submit 失败不影响第二至六群；post-submit unknown 时不重复且日报准确。 + +### P1-4：多群隔离在全局前置步骤不完整 + +- **问题**：`_sync_group_names()` 与 `_load_groups()` 位于群 worker 之前；前置异常会让所有群都没有逐群结果。 +- **为什么发生**：批次级准备与逐群处理边界没有明确区分可降级/不可降级错误。 +- **代码位置**：`app/pipeline/daily_pipeline.py:115-124`、`app/pipeline/daily_pipeline.py:184-219`。 +- **如何复现**：让群名同步 list_groups 抛 timeout，或 SQLite 在 list_groups 时 locked；观察全部 worker 未启动。 +- **修改方案**：群清单使用上一次成功快照；群名同步失败时标记 stale 并继续已绑定群，只有 identity 不可信的群单独 hold。 +- **修改风险**：使用陈旧群名可能发错群；必须在发送前重新核对 target，且不可信时 fail-closed。 +- **验证成功**：同步失败时已稳定绑定的群仍生成；未知/冲突群单独 hold；不出现串群发送。 + +### P1-5：Prompt 总长度、特殊字符和 Provider 重试边界不完整 + +- **问题**：输入分段约限制 50,000 字符，但最终图片 Prompt 没有总长上限；群名、昵称、引用可原样进入固定标题合同。解析重试覆盖 JSON/Schema 错误，但部分 `_analysis_chat()` 调用位于重试 `try` 外，网络/timeout 不进入格式重试。 +- **为什么发生**：只限制了上游对话段和单字段，没有对最终产物做字节/token、控制字符和内容风险预算。 +- **代码位置**：`app/ai/conversation_segments.py:46-52,108-140`、`app/ai/prompt_builder.py:337-388,448-583`、`app/ai/poster_copy.py:76-84,419-636`。 +- **如何复现**：使用超长群名/昵称、含换行的昵称、成百条长引用；让 Provider 第一次 read timeout 或返回 Markdown fence/错误 JSON。 +- **修改方案**：最终 Prompt 加 token/字符硬上限和确定性裁剪顺序；显示文本转义控制字符；结构化数据与渲染文本分离；Provider 错误按“明确未提交/429/5xx/timeout unknown/Schema”分类。 +- **修改风险**：裁剪可能丢核心话题；优先保留 topic id、排名、事实摘要，引用和装饰性指令最后裁剪。 +- **验证成功**：属性测试覆盖中文、emoji、换行、引号和最大长度；输出始终满足合同，核心 topic 不变，无无限重试。 + +### P1-6：图片复用/发送前校验弱于新图落盘校验 + +- **问题**：新图最终路径会用 Pillow 解码并检查 1024×1536；但已有图片跳过、部分候选复用和发送前主要只检查存在、非空和文件头。带合法 PNG 头的截断文件可能通过基础检查。 +- **为什么发生**：`verify_image()` 与 `verify_image_contract()` 两套合同应用范围不同。 +- **代码位置**：`app/image/image_task.py:69-102,133-200`、`app/image/codex_generator.py:438-449,1382-1420`、`app/pipeline/delivery_stages.py:143-210`。 +- **如何复现**：把 daily_image.png 替换为仅含合法 PNG signature 的截断文件，保持非零大小；走复用或发送 preflight。 +- **修改方案**：所有进入 `IMAGE_READY`、复用和发送前统一执行 Pillow `verify/load`、尺寸、格式和可选 SHA/mtime 合同；失败退回 image checkpoint，不发送。 +- **修改风险**：Pillow 解码增加少量耗时；可缓存 hash 对应的验证结果。 +- **验证成功**：截断、错尺寸、扩展名伪装、零字节全部被拒绝；合法图只验证一次并可安全复用。 + +### P1-7：图片原子提升后仍可能形成“文件成功、状态失败” + +- **问题**：`os.replace` 已把图片放到目标路径后，smoke metadata、清理、回执删除或 hook 状态更新仍可能抛错/被吞掉，最终 run 标记 FAILED 但文件实际存在。 +- **为什么发生**：文件 commit 与状态 commit 不是一个可恢复的 checkpoint；部分 hook 使用 `except Exception: pass`。 +- **代码位置**：`app/image/codex_generator.py:1247-1295,1382-1420`、`app/image/image_task.py:232-266`、`app/pipeline/image_stages.py:63-144`。 +- **如何复现**：在 `os.replace` 后、`_save_last_smoke` 或 run hook 中注入 OSError；重启后比较文件与 run 状态。 +- **修改方案**：把“目标文件已通过强校验”的事实作为可重建 checkpoint；启动 reconciler 若 hash/尺寸/operation id 匹配则修复为 IMAGE_READY，不重新生图。 +- **修改风险**：错误认领旧图会串日/串群；必须同时校验 group task id、run date、prompt hash、job id 和 SHA。 +- **验证成功**：七个落盘窗口逐点 kill；重启均认领同一可信图片，imagegen 调用次数仍为 1。 + +### P1-8：锁冲突和普通状态更新缺少跨进程完整协调 + +- **问题**：生成锁 2 秒未取得就返回 `already_running`，没有持久队列；普通 `DailyScheduleState.update` 只有进程内锁，`RunStore` 的部分普通读改写也可能在多进程竞争时丢更新，命名互斥主要覆盖特定 claim 路径。 +- **为什么发生**：系统同时支持 FastAPI owner、脚本/Windows task 和人工 API，多 owner 风险没有用唯一调度租约统一。 +- **代码位置**：`app/services/generation_runtime.py:22-48`、`app/scheduler/daily_v2_job.py:163-183`、`app/scheduler/daily_v2_job.py:143-156`、`app/v2/run_store.py:225-253,561-654`。 +- **如何复现**：两个进程同时启动相同日期 job，并在状态 read 与 replace 之间注入延迟；观察一个直接放弃或字段被覆盖。 +- **修改方案**：确立单 Scheduler owner；busy 写入 backlog/`next_retry_at`；所有逐任务读改写使用同一跨进程 mutex 或带 version 的 compare-and-swap。 +- **修改风险**：锁顺序错误会死锁；固定 batch→group→send claim 顺序并限制等待时间。 +- **验证成功**：多进程压力测试中无 lost update、无死锁；busy 任务最终自动领取且只执行一次外部阶段。 + +### P1-9：缺少无人值守所需的结构化诊断和每日汇总 + +- **问题**：日志多为普通文本，缺少稳定 `run_id/group_task_id`、stage duration、attempt、模型、HTTP code、错误分类；`/health` 恒定 OK，`/ready` 只检查本地基础项；没有 `runtime/YYYY-MM-DD/status.json`。 +- **为什么发生**：可观测性按“终端看报错”建设，而非按 30 天任务账本建设。 +- **代码位置**：`app/core/logging.py`、`app/api/system.py:22-98`、`app/scheduler/daily_v2_job.py:163-330`。 +- **如何复现**：查询某群某日第二次生图 timeout 的响应码、退避和耗时；当前无法从单一结构化来源回答。 +- **修改方案**：结构化 JSON 日志 + 每日 reconciler；健康分为 process health、scheduler liveness、dependency readiness、daily completion，避免 200/OK 误导。 +- **修改风险**：日志可能泄露聊天和 token；只记录 hash、长度、错误摘要，统一脱敏，不写原始 Prompt/聊天/API key。 +- **验证成功**：给定 date+group_task_id 可还原所有 attempts；每日状态能回答每群卡在哪一步、下一次重试时间和是否需人工。 + +--- + +## 6. AI 生图专项风险 + +### 6.1 Prompt 风险清单 + +| 检查项 | 当前结论 | 风险 | +|---|---|---| +| Prompt 超长 | 对话分段有限制，最终 Prompt 无总上限 | Provider 拒绝、截断或超时 | +| Markdown/JSON 错误 | 有 fence/JSON/Schema/事实校验及有限格式重试 | 调用异常不一定进入重试；最终失败无降级 | +| 固定自然语言格式 | 渲染合同依赖固定标题/字段 | 昵称换行、控制字符、模型偏离可破坏结构 | +| 特殊字符/中文昵称 | 中文路径基本受 Path/UTF-8 支持；显示字段未统一转义 | 控制字符和引用边界污染 Prompt | +| 聊天安全限制 | 原话、昵称、敏感表达可能进入图片 Prompt | 单一话题可能使整图审核失败 | +| 单话题异常隔离 | 无按 topic 的安全风险隔离/替换 | 一个 topic 可拖垮整张图 | +| 事实保持 | 已有 topic selection、message id、quote 校验 | Level 2 设计必须复用这些事实锚点 | + +### 6.2 当前生图错误分类与重试 + +| 场景 | 当前分类 | 自动 retry | backoff | 当前结果 | +|---|---|---:|---:|---| +| 可执行文件不存在/确认未启动 | `start_failed` | 最多 2 次 | 无 | 重试后失败 | +| 已启动后非零退出 | `result_unknown` | 否 | 无 | hold，防重复付费 | +| 进程 timeout | 尝试杀进程树和候选恢复；否则 unknown | 否 | 无 | 可恢复可信候选，否则 hold | +| 无可信回执/无效候选 | `result_unknown` | 否 | 无 | hold | +| 内容审核拒绝 | 未形成专门稳定分类 | 否 | 无 | 通常归失败/unknown | +| 429 | Codex CLI 路径未暴露稳定 HTTP 分类 | 否 | 无 | 无专项策略 | +| 5xx/网络 | 同上 | 否 | 无 | 无专项策略 | +| 图片下载失败 | 无独立 downloader retry 接口 | 否 | 无 | 生成失败 | + +“所有异常一律 retry”不适合此链路：只要请求可能已经被 Provider 接收，就不能盲目再次付费生成。正确方向是扩展可证明的错误分类和 attempt ledger，而不是放宽 unknown 锁。 + +### 6.3 三级降级设计 + +**Level 1：原 Prompt** + +- 完整保留现有 Prompt 合同、topic selection 和图片质量要求。 +- 只对明确 `start_failed`、明确 429/5xx 且 Provider 证明未接受任务的场景做有上限的指数退避。 + +**Level 2：安全化 Prompt** + +- 触发条件仅限明确内容安全/审核拒绝或本地安全预检命中。 +- 输入必须是同一份已持久化结构化 topic selection、排行和事实 hash，不重新让 AI 自由选题。 +- 只允许：昵称泛化为“群友 A/B”、原话改为事实性转述、敏感细节抽象、删除装饰性风险词。 +- 禁止：删除主要 topic id、改变排行、编造事实、把失败主题替换成另一天内容。 +- 保存 `prompt_level=2`、原/安全 Prompt hash、变更字段和触发错误分类。 + +**Level 3：本地简化信息图** + +- 使用 Pillow 等本地确定性渲染器，读取 ranking.json、topic selection、日期和安全化摘要。 +- 固定 1024×1536 布局，包含日期、活跃榜、主要话题、数据量和“简化版”标识。 +- 不调用外部 AI，不受模型审核或配额影响。 +- 使用同一图片合同和原子写入；`image_source=local_fallback` 明确记录，不伪装成 AI 漫画。 + +### 6.4 重试预算建议 + +| 错误类型 | 建议 | +|---|---| +| DNS/connect timeout、明确未提交 | 3 次；5s、20s、60s + jitter | +| 429 | 尊重 `Retry-After`；总预算不超过当日截止时间 | +| 5xx | 2-3 次指数退避；熔断共享 Provider，避免六群同时冲击 | +| read timeout/无回执/已启动未知 | 不自动重试同一外部操作；尝试按 job/thread 回收结果，否则 hold | +| 内容审核明确拒绝 | 不重复 L1；进入一次 L2，再进入 L3 | +| 本地磁盘/图片合同失败 | 修复前短重试写入；已有可信候选不重新调用模型 | +| 配额耗尽/余额不足 | 标记 provider unavailable，直接 L3 或延后，不让六群重复撞 API | + +--- + +## 7. Scheduler 风险 + +### 7.1 当前调度能力 + +- 生成：每日 00:15，`misfire_grace_time=1800`、`coalesce=true`、`max_instances=1`。 +- 发送：每分钟第 15 秒扫描，`misfire_grace_time=45`、`coalesce=true`、`max_instances=1`。 +- 启动：只补启动当天且未写 `generation_completed_at` 的生成；`skip_email=true`。 +- 进程：FastAPI owner 当前 active；外部 Windows 任务被禁用。 + +### 7.2 Watchdog 最小设计 + +1. 每次服务启动后和每 10 分钟运行一次只读扫描。 +2. 扫描 `[today-29, today]`，按日期从旧到新；期望任务集合由“当日群 manifest”决定，没有 manifest 时用当前启用群并标记推断。 +3. 对每个任务按 execution state 分类： + - `COMPLETE/SENT`：不动作。 + - `WAIT_RETRY` 且到期：从 last checkpoint 续跑。 + - `MISSING`：建立任务并开始 DATA。 + - `HOLD_MANUAL`：只进状态报告。 + - `SEND_RESULT_UNKNOWN`：永久 fail-closed，绝不自动重发。 +4. 发送仅在证据明确为“尚未提交”时自动补；跨日发送前必须重新验证群目标和图片合同。 +5. 使用有租期的 owner lease，busy 任务写 `next_retry_at`，不能只返回 `already_running` 后消失。 +6. Windows 启动机制需满足：非交互登录也可启动、`StartWhenAvailable`、失败重启、明确工作目录、日志目录、与 FastAPI owner 不重复。 + +### 7.3 上一任务未完成与第二天关系 + +- 生成/统计可以按旧到新补，避免后一天先完成导致统计和配额竞争失序。 +- 微信发送不能简单把所有历史欠账一起重发；应逐条核对 target、提交证据和 send claim。 +- 前一天 `HOLD_MANUAL` 不应阻塞第二天生成，但必须持续出现在每日/总览报告，直到人工消歧。 +- Provider 全局故障应熔断并让各群进入共享 `next_retry_at`，避免六群独立重试形成风暴。 + +--- + +## 8. 幂等性问题 + +### 8.1 当前幂等边界 + +| 阶段 | 当前重复执行行为 | 结论 | +|---|---|---| +| 已有 messages.json | 默认复用，不重抓 | 良好 | +| 已到 IMAGE_READY/READY_TO_SEND/SENT | 非 force 跳过生成 | 良好 | +| Prompt 结果未知 | operation claim 阻止盲重试 | 安全但需人工 | +| 生图已启动结果未知 | attempt manifest/hold | 安全但需人工 | +| 文字成功、图片失败 | 可只续图片 | 良好 | +| SENT | send_due 跳过 | 良好 | +| 普通 FAILED | 自动恢复扫描排除 | **不满足无人值守** | +| 批次 partial/failed | 仍可能写 generation_completed_at | **不满足无人值守** | +| 外部成功、本地 finish claim 失败 | 可能 unknown 或状态分叉 | 需强化 | + +### 8.2 建议状态模型 + +不删除现有阶段状态，新增正交字段： + +```json +{ + "stage": "IMAGE", + "stage_status": "PROMPT_READY", + "execution_state": "WAIT_RETRY", + "last_successful_checkpoint": "PROMPT_SAVED", + "input_hash": "sha256:...", + "attempt_count": 2, + "retry_budget": 3, + "next_retry_at": "2026-08-27T01:20:00+08:00", + "last_error_type": "API_TIMEOUT_PRE_SUBMIT", + "manual_hold": false, + "version": 17 +} +``` + +建议执行状态： + +- `ACTIVE` +- `WAIT_RETRY` +- `HOLD_MANUAL` +- `COMPLETE` +- `FAILED_FINAL` + +建议 checkpoint： + +- `TASK_CREATED` +- `MESSAGES_SAVED` +- `RANKING_SAVED` +- `SUMMARY_SAVED` +- `PROMPT_SAVED` +- `IMAGE_CLAIMED` +- `IMAGE_SAVED` +- `TEXT_SEND_CLAIMED` +- `TEXT_SENT_CONFIRMED` +- `IMAGE_SEND_CLAIMED` +- `SENT_CONFIRMED` + +每次 attempt 使用 append-only ledger，包含 operation id、开始/结束、输入 hash、外部提交证据、错误分类和产物 hash。状态文件只保存当前快照，ledger 提供审计证据。 + +--- + +## 9. 状态恢复问题 + +### 9.1 七个崩溃窗口 Resume 矩阵 + +| 崩溃位置 | 当前重启判断 | 当前风险 | 目标恢复行为 | +|---|---|---|---| +| 1. 聊天记录读取后 | 若 messages.json 已原子保存则复用；保存前会重读 | 读取完成但未保存无 checkpoint；普通 FAILED 后不自动补 | 以 `MESSAGES_SAVED` 为边界；保存前可安全重读，保存后只续排行 | +| 2. AI 总结完成后 | 结果已持久化可复用；无可信结果进入 unknown | Provider 已返回但本地未保存时可能人工 hold | operation receipt + SUMMARY_SAVED；未知不盲重试 | +| 3. Prompt 完成后 | prompt 文件/run 状态一致时可进入生图 | 文件与状态提交窗口可能分叉 | 用 prompt hash reconciler 修复 `PROMPT_SAVED` | +| 4. 生图过程中 | 进程树/attempt manifest；已启动未知不重试 | 可能永久 hold | 先按 job/thread 回收；无可信结果保持人工 hold，允许 L3 但不得重复 L1 付费 | +| 5. 图片生成完成但未保存 | 可尝试同线程候选恢复 | 没有候选或归属证据则失败 | 候选携带 job/group/date/prompt hash；可信则原子提升，不可信转 L3/hold | +| 6. 图片保存后未发送 | run 为 IMAGE_READY/READY 可当天续发 | 跨日不扫描；文件/状态可能分叉 | watchdog 识别 IMAGE_SAVED，重新 preflight 后只续发送 | +| 7. 微信已发送但 DB/run 未更新 | unresolved claim 转 unknown 的部分保护 | 无法无人值守判定，盲重发会重复 | 永久 `SEND_RESULT_UNKNOWN`；人工或可验证外部 receipt 消歧，绝不以超时自动重发 | + +### 9.2 恢复原则 + +1. 本地产物恢复依据必须同时包含稳定 task id、日期、群 id、输入 hash 和产物 hash。 +2. 外部动作必须区分：未提交、已确认、结果未知。只有“明确未提交”可自动 retry。 +3. `SEND_RESULT_UNKNOWN` 的人工 hold 是正确安全边界，不能为了无人值守成功率而删除。 +4. `FAILED` 不能继续作为所有失败的单一终态；至少拆为 retryable、manual hold、final。 +5. 恢复执行必须针对单群单阶段，不允许默认 force 整日重跑。 + +--- + +## 10. 日志与监控问题 + +### 10.1 建议结构化日志 + +每条关键事件至少包含： + +```json +{ + "timestamp": "2026-08-27T01:03:12.431+08:00", + "level": "WARNING", + "run_id": "20260827-auto-01", + "group_task_id": "group-42:2026-08-27", + "group_id": 42, + "report_date": "2026-08-27", + "stage": "IMAGE_GENERATION", + "status": "WAIT_RETRY", + "duration_ms": 60012, + "attempt": 2, + "max_attempts": 3, + "model": "configured-image-model", + "provider": "codex-imagegen", + "api_response_code": 503, + "error_type": "API_5XX", + "error_summary": "provider temporarily unavailable", + "next_retry_at": "2026-08-27T01:04:12+08:00" +} +``` + +禁止记录 API key、Cookie、完整聊天、完整 Prompt、完整模型响应。群名可在本地状态报告展示，集中日志优先使用稳定 group id；错误摘要限长并脱敏。 + +### 10.2 `runtime/YYYY-MM-DD/status.json` 建议 + +```json +{ + "schema_version": 1, + "report_date": "2026-08-27", + "generated_at": "2026-08-27T09:00:00+08:00", + "run_id": "20260827-auto-01", + "expected_groups": 6, + "complete_groups": 5, + "held_groups": 1, + "overall_status": "attention_required", + "groups": [ + { + "group_task_id": "group-42:2026-08-27", + "group_name": "群 A", + "statistics": "success", + "summary": "success", + "prompt": "success", + "image": "success_after_retry", + "image_attempts": 2, + "send": "sent", + "last_checkpoint": "SENT_CONFIRMED", + "next_retry_at": null, + "manual_hold": false, + "errors": [ + {"stage": "IMAGE_GENERATION", "attempt": 1, "error_type": "API_TIMEOUT_PRE_SUBMIT"} + ] + } + ] +} +``` + +写入策略：从逐群 run/ledger 只读重建，先写临时文件再原子替换；如果源状态损坏或期望群集合不确定，`overall_status` 必须 degraded/attention_required，不能默认 success。 + +### 10.3 需要告警的最小条件 + +- 当日 00:45 仍无 batch/task manifest。 +- 任一任务超过 `next_retry_at + grace` 未推进。 +- Provider 全局熔断、配额耗尽、磁盘写失败、SQLite locked 超预算。 +- 到发送时间后仍未 `SENT`。 +- `SEND_RESULT_UNKNOWN`、run/scheduler corruption、批次与逐群状态不一致。 +- Scheduler heartbeat 超过 5 分钟、Windows 启动后服务未恢复。 + +--- + +## 11. 建议修改架构 + +### 11.1 最小增量原则 + +- 保留 FastAPI、APScheduler、DailyPipeline、RunStore 和现有状态名称。 +- 不重写框架，不引入分布式队列作为 P0 前置条件。 +- 先补正确的任务账本、重试分类、checkpoint 和 watchdog，再考虑 UI/抽象。 +- 所有外部动作继续 claim-first、unknown fail-closed。 + +### 11.2 建议新增的最小模块 + +| 模块 | 责任 | 是否修改公开 API | +|---|---|---| +| `TaskReconciler` | 从最近 30 天逐群状态计算缺失/重试/hold/complete | 否，可先内部使用 | +| `AttemptLedger` | append-only 记录外部调用和 checkpoint | 否 | +| `RetryPolicy` | 按错误类型给出预算、backoff、next_retry_at | 否 | +| `ImageFallbackService` | L2 安全 Prompt、L3 本地信息图 | 可复用现有生成入口 | +| `DailyStatusWriter` | 原子生成 runtime 状态报告 | 否 | +| `SchedulerLease` | 明确唯一 owner、busy backlog | 否 | + +### 11.3 影响—投入矩阵 + +| 项目 | 影响 | 预计投入 | 优先级 | +|---|---|---:|---| +| 修正 completed marker + retryable state | 极高 | 中 | P0-1 | +| 30 天 watchdog + Windows 启动验证 | 极高 | 中 | P0-2 | +| L2/L3 生图降级 | 极高 | 中 | P0-3 | +| 数据源 retry/受控 fallback | 高 | 中 | P0-4 | +| 发送 claim 严格检查 | 高 | 小 | P1-1 | +| 批次/逐群 reconciler + status.json | 高 | 小-中 | P1-2/P1-9 | +| 发送逐群隔离 | 中高 | 小 | P1-3 | +| Prompt 长度/转义/分类 | 中高 | 中 | P1-5 | +| 图片统一强校验与落盘恢复 | 中高 | 小-中 | P1-6/P1-7 | +| 多进程状态 version/CAS | 中高 | 中 | P1-8 | + +### 11.4 不在本轮范围 + +- 不更换 Web 框架、Scheduler 或数据库。 +- 不重写整个 Pipeline，不做纯代码风格重构。 +- 不改变公开 API/Schema/状态类型；本报告字段仅是后续建议。 +- 不自动解除 `SEND_RESULT_UNKNOWN`。 +- 不实际安装 Windows 服务/任务，不部署、不发微信/邮件。 +- 不创建技术债工单、不生成 Canvas、不修改测试。 + +--- + +## 12. 建议开发顺序 + +每项都应独立提交、独立故障注入验证；未通过前不并行扩大生产改动。 + +### P0-1：修正批次完成语义和失败分类 + +- 引入 invocation completed 与 task terminal 分离。 +- 将普通失败分类为 `FAILED_RETRYABLE / HOLD_MANUAL / FAILED_FINAL`（可用正交字段实现，不必替换现有 status）。 +- 验收：一次性取数/Prompt timeout 重启后只续失败群。 + +### P0-2：实现 30 天 TaskReconciler/Watchdog + +- 旧到新扫描缺失任务、retry due 和状态分叉。 +- 先只补生成；发送补跑必须遵守提交证据。 +- 验收：停机 48 小时、错过计划、重复启动、上一日未完成均收敛。 + +### P0-3：实现生图 Level 2/Level 3 + +- 先完成确定性 L3，确保外部模型全不可用仍有图。 +- 再增加事实保持的 L2 安全化 Prompt。 +- 验收：审核拒绝、配额耗尽、连续 5xx 均至少得到 L3 图。 + +### P0-4：数据源重试、熔断和受控 fallback + +- 先做同 provider 明确未提交错误的指数退避。 +- 证明备用后端数据合同一致后再启用 fallback。 +- 验收：10392 短暂不可用不再让六群永久失败。 + +### P1-1：严格检查所有发送 claim 更新 + +- pre-submit 失败禁止外部调用；post-submit 失败进入 unknown。 +- 验收：逐更新点 fault injection，零重复发送。 + +### P1-2：统一图片强校验和落盘 reconciler + +- 复用、发送前均执行统一合同；识别可信“文件成功/状态失败”。 +- 验收：截断图片拒绝，可信已落盘图不重复生成。 + +### P1-3：补齐逐群发送/全局前置隔离 + +- 发送单群异常不阻断其他群；群名同步使用可信缓存降级。 +- 验收：首群失败，后五群仍按安全规则推进。 + +### P1-4：Prompt 长度、安全预检和错误分类 + +- 硬上限、控制字符转义、事实优先裁剪、429/5xx/timeout 分类。 +- 验收：属性测试和超长中文输入不破 Schema。 + +### P1-5：结构化日志与每日状态报告 + +- 引入 run_id/group_task_id/attempt/duration/response code；原子日报。 +- 验收：无需查看终端即可定位任一群任一阶段。 + +### P1-6：唯一 Scheduler owner 与跨进程状态协调 + +- owner lease、busy backlog、version/CAS、固定锁顺序。 +- 验收：双进程压力测试无丢任务/死锁/重复外部动作。 + +### P2-1：稳定并发时序测试 + +- **问题**：审计早期一次并发时序测试出现 `maximum == 1` 而期望 2，随后完整测试和单文件测试均通过，属于未稳定复现的时序波动。 +- **原因/位置**：`tests/test_generation_concurrency.py` 依赖短事件窗口和线程调度。 +- **复现**：Windows 负载下循环运行该文件并记录 seed/调度时刻。 +- **方案**：改用显式 barrier/可控 clock，保留并发语义断言。 +- **风险**：只改测试同步，不能放宽生产并发约束。 +- **验证**：循环 100 次零偶发失败，并仍能捕获串行退化。 + +### P2-2：清理测试/构建警告 + +- **问题**：后端有 Starlette/httpx 弃用警告；前端 SSR 测试有 `useLayoutEffect does nothing on the server`。 +- **原因/位置**：`.venv/Lib/site-packages/fastapi/testclient.py:1`、`frontend/src/components/common/index.tsx:69-128` 及 motion SSR。 +- **复现**：运行完整 pytest 与 `npm test`。 +- **方案**：在依赖升级窗口处理 TestClient 迁移；前端测试使用合适的客户端环境或避免 SSR 路径触发 layout effect。 +- **风险**：依赖升级可能扩大范围，排在稳定性 P0/P1 之后。 +- **验证**：测试通过且无对应 warning。 + +### P2-3：使健康接口表达真实运行能力 + +- **问题**：`/health` 恒为 OK，`/ready` 不检查 Scheduler heartbeat、WDA、微信或当日任务完成度。 +- **原因/位置**：`app/api/system.py:22-98`。 +- **复现**：停止 WDA 或让调度长期无 heartbeat，health 仍可 200。 +- **方案**：保留 liveness 200，新增/扩展 readiness 和 daily status，明确各检查项，不把外部故障等同进程死亡。 +- **风险**：若部署平台把 readiness 失败自动重启，需避免因外部依赖抖动造成重启风暴。 +- **验证**：进程活着但依赖失败时 liveness 正常、readiness degraded、日报可诊断。 + +--- + +## 附录 A：30 天无人值守仿真 + +### A.1 方法 + +- 固定随机种子：`20260827`。 +- 时间范围：2026-07-29 至 2026-08-27，共 30 天。 +- 群数：当前启用群数量 6；仿真使用匿名稳定 group id，不读取真实聊天内容。 +- 真实逻辑：`DailyPipeline.generate_all/send_due`、`RunStore`、`DailyScheduleState`、排行、文件落盘、阶段状态、发送 claim。 +- Mock 边界：微信取数、AI Prompt、生图、微信 sender；没有任何真实外部动作。 +- 注入：5% 网络 timeout、5% AI Schema 错误、5% 生图失败、3% 图片下载边界失败、3% 微信发送失败、5% 离线生成日、10% 离线发送日、20% 重复启动候选。 +- 仿真输出位于 `TemporaryDirectory`，进程退出时删除，未进入仓库。 + +### A.2 结果 + +| 指标 | 结果 | +|---|---:| +| 任务单元 | 180 | +| SENT | 130 | +| FAILED | 21 | +| READY_TO_SEND 但以后未扫描 | 17 | +| PENDING/根本未建任务 | 12 | +| 未完成合计 | **50** | +| 重复生图 | 0 | +| 重复文字发送 | 0 | +| 重复图片发送 | 0 | +| 无限 retry | 0 | +| 缺失 scheduler 日期 | 2 | +| batch success / partial | 12 / 16 | + +实际命中的外部边界故障：网络 timeout 10 次、AI 错误格式 6 次、生图失败 4 次、图片下载边界失败 1 次、微信发送失败 5 次。离线生成日为 2026-08-20、2026-08-24；离线发送日为 2026-07-30、2026-08-10、2026-08-24、2026-08-25；5 个日期触发了重复启动检查。 + +解释： + +- 无重复发送/生图说明当前终态跳过和 claim 保护有效。 +- 无无限 retry 不是“恢复成功”，而是很多失败在第一次正常返回后被 batch completion marker 封存。 +- 17 个 READY_TO_SEND 在离线发送日后永远留在历史目录，直接验证了 current-date-only 扫描缺口。 +- 12 个 PENDING 来自两天完全离线且无跨日建档/补跑。 +- 随机 5% 进程中断条件在该固定种子下没有实际命中，因此“真实 OS kill 后的全链路恢复”本轮未得到统计样本；上文七窗口矩阵来自真实代码与现有测试证据，仍应在整改测试中强制逐点 kill。 +- 当前 pipeline 没有独立图片 downloader 接口，因此 3% 下载失败在 image generator 边界注入；这也是可测试性缺口。 +- 仿真没有把“外部已提交但结果未知”作为随机故障；现有测试覆盖 unknown hold，本报告不把它误报为无人值守可恢复。 + +### A.3 判定 + +以“30 天 × 全群零永久丢失、SENT 后零重复发送、unknown 永不盲重发”为标准： + +- 零重复发送：通过。 +- unknown fail-closed：静态代码/现有测试通过，仿真未随机命中。 +- 零永久丢失：失败，50 个任务未完成。 +- 中断自动恢复：证据不足，需整改后强制 kill 测试。 +- 单群失败隔离：生成 worker 大部分场景通过；全局前置和发送外层仍有缺口。 +- 第二天继续：当天任务可继续，但上一天欠账不会被处理，失败。 + +--- + +## 附录 B：验证证据 + +所有验证均未调用真实 AI、生图、微信或邮件。 + +| 命令 | 结果 | +|---|---| +| `.\.venv\Scripts\python.exe -m compileall -q app scripts tests` | exit 0 | +| `.\.venv\Scripts\python.exe -m pytest tests -q` | **597 passed**, 1 warning, 54.95s | +| `.\.venv\Scripts\python.exe -m pytest tests/test_generation_concurrency.py -q` | **5 passed**, 3.12s；本次未复现早期时序失败 | +| `.\.venv\Scripts\python.exe -m pytest tests -q --ignore=tests/test_generation_concurrency.py` | **592 passed**, 1 warning, 42.16s | +| `npm test`（`frontend`） | **5 files / 21 tests passed**, 11.33s；有 React SSR warning | +| `npm run build`（`frontend`） | exit 0，5004 modules，9.68s | + +后端 warning 为 Starlette/httpx TestClient 弃用提示；前端 warning 为 SSR 环境中的 `useLayoutEffect`。二者未阻止本次构建，但应按 P2 处理。 + +测试通过只证明已编码的合同，不证明真实 Provider、真实微信 UI、Windows 重启或 30 天调度可靠；本报告的 P0 结论来自代码、隔离仿真和实时运行状态三类证据的交叉验证。 diff --git a/GROUPBRIEF_RELIABILITY_REMEDIATION_TASK.md b/GROUPBRIEF_RELIABILITY_REMEDIATION_TASK.md new file mode 100644 index 0000000..cedcefd --- /dev/null +++ b/GROUPBRIEF_RELIABILITY_REMEDIATION_TASK.md @@ -0,0 +1,67 @@ +# GroupBrief 无人值守稳定性整改任务 + +## 背景 + +依据 `GROUPBRIEF_RELIABILITY_AUDIT.md`，完成 P0-1 至 P2-3 的全部整改。目标是在外部 AI、图片和微信均可被 Mock 的情况下，证明 30 天 × 全部启用群的任务不会永久丢失、不会无限重试、不会重复生图或重复发送，并可从持久化 checkpoint 恢复。 + +## 允许修改范围 + +- `app/`：V2 调度、状态、Pipeline、Provider、图片、发送、日志和系统状态。 +- `scripts/`：隔离仿真和 Windows 调度安装参数。 +- `tests/`：单元、集成、并发、故障注入和 30 天仿真。 +- `frontend/`：仅 P2-2 警告修复及相应测试。 +- `requirements*.txt`：仅在已有依赖无法完成目标时修改；优先复用 Pillow、APScheduler、SQLModel。 +- 本任务文档与审计报告中的整改状态说明。 + +## 禁止修改范围 + +- 不读取、写入或提交 `.env`、Token、API Key、密码、Cookie、浏览器数据。 +- 不在验证中调用收费 AI、生图、真实微信或邮件。 +- 不自动解除 `SEND_RESULT_UNKNOWN`，不把结果未知改成自动重发。 +- 不改写历史 Git，不强推，不部署，不迁移生产数据库，不修改真实 Windows 任务或实时 `output/`。 +- 不更换 FastAPI、APScheduler、SQLite/SQLModel，不做与稳定性无关的大规模重构。 + +## 已确定实现要求 + +| 编号 | 实现要求 | 权威验收证据 | +|---|---|---| +| P0-1 | invocation 完成与逐群终态分离；retryable/manual/final 正交执行状态；checkpoint、retry budget、`next_retry_at`、输入/产物 hash、attempt 记录 | 一次性故障后只续失败阶段；成功阶段调用计数不增加 | +| P0-2 | 最近 30 天 watchdog，旧到新补任务；明确未提交的历史发送可续；unknown 永久 hold；唯一 owner/busy backlog | 48 小时离线、跨日、重复启动测试零任务丢失/零重复发送 | +| P0-3 | 生图 L1 原 Prompt、L2 事实保持安全化 Prompt、L3 Pillow 本地信息图 | 审核拒绝与 Provider 不可用仍得到合同图片；topic/ranking 不变 | +| P0-4 | 微信数据读取按错误分类有限 retry、指数退避、熔断；不静默混合不兼容数据源 | timeout/5xx 可恢复，永久故障不形成重试风暴，快照来源可审计 | +| P1-1 | 所有发送 claim 更新必须检查；pre-submit 失败零外部调用，post-submit 失败进入 unknown | 每个更新点故障注入，发送调用和最终状态符合合同 | +| P1-2 | 复用、READY 和发送前统一强图片合同；可信已落盘图片可对账恢复 | 截断/错尺寸拒绝；落盘后崩溃不重复生图 | +| P1-3 | 发送逐群异常隔离；全局群同步失败只 hold 不可信群；桌面状态未知时熔断 | 首群明确 pre-submit 失败不影响后续，未知时不继续冒险 | +| P1-4 | 最终 Prompt 长度硬限、控制字符清理、事实优先裁剪、Provider 错误分类 | 中文/emoji/换行/超长属性测试稳定且不改变 topic id | +| P1-5 | JSON 结构日志；`runtime/YYYY-MM-DD/status.json`；run/group/stage/attempt/duration/model/code/error | 给定 date/task id 能恢复完整诊断；日报由逐群状态原子重建 | +| P1-6 | Scheduler owner lease、跨进程状态互斥/version、busy 持久补偿 | 双进程压力测试无 lost update、死锁、重复外部动作 | +| P2-1 | 并发测试改用 barrier/可控同步 | Windows 循环 100 次无偶发失败且仍能捕获串行退化 | +| P2-2 | 清理 Starlette/httpx 和前端 SSR warning，不扩大依赖风险 | 完整后端和前端测试无对应 warning | +| P2-3 | liveness/readiness/scheduler heartbeat/dependency/daily completion 分层 | 进程活着但依赖失败时 liveness=ok、readiness=degraded，原因明确 | + +## 兼容性与安全不变量 + +1. 现有阶段状态和值保持可读，旧 `run.json` 可惰性补齐新字段。 +2. `SENT` 永不被自动生成/发送路径降级或重发。 +3. `CORRUPT`、`PROMPT_RESULT_UNKNOWN`、`SEND_RESULT_UNKNOWN` 永久 fail-closed。 +4. 已启动且结果未知的付费调用不自动重试；L3 本地渲染不等同重复付费调用。 +5. 所有状态写入原子化；同一 date × group 的读改写具备跨进程协调或版本检查。 +6. 历史发送前重新验证当前目标和图片合同；不能证明“尚未提交”时不发送。 +7. 日志和日报不得保存凭据或完整聊天/Prompt。 + +## 测试命令 + +```powershell +.\.venv\Scripts\python.exe -m compileall -q app scripts tests +.\.venv\Scripts\python.exe -m pytest tests -q +.\.venv\Scripts\python.exe scripts\simulate_reliability.py --days 30 --seed 20260827 --groups 6 +npm test --prefix frontend +npm run build --prefix frontend +git diff --check +``` + +## 返回格式 + +- 每个 P0/P1/P2 项：修改文件、行为变化、故障注入证据、剩余风险。 +- 明确区分 Mock 验证与真实外部动作；不得用测试通过声称真实发送或真实 Provider 可用。 +- 最终报告 Git 分支、提交、远端与推送结果。 diff --git a/GROUPBRIEF_SIX_GROUP_RANKING_SYNC_TASK.md b/GROUPBRIEF_SIX_GROUP_RANKING_SYNC_TASK.md new file mode 100644 index 0000000..8e51ee9 --- /dev/null +++ b/GROUPBRIEF_SIX_GROUP_RANKING_SYNC_TASK.md @@ -0,0 +1,66 @@ +# GroupBrief 六群排行榜与生图规则统一任务 + +## 背景 + +当前仅 Eason 群使用“文字主榜＋互动数”、WeChatDataAnalysis 名称和严格图片事实校验，其他五个活动群仍使用所有非系统消息参与排名及联系人解析名称。用户已确认六群统一，并要求从下一次任务起生效。 + +## 目标 + +1. 六个活动群统一采用文字消息决定名次、互动数仅展示的排行榜口径。 +2. 排行榜名单下方固定显示：`说明：互动指图片、表情、引用等非文字消息，仅展示活跃度，不影响排名。` +3. 六群统一采用 WeChatDataAnalysis 的 `senderDisplayName`，并启用现有严格 Prompt 与 OCR 图片事实校验。 +4. Dashboard、独立排行榜页面和发送用 `ranking.txt` 保持一致。 + +## 允许修改范围 + +- `templates/ranking/text_interactions.txt` +- Dashboard 与独立排行榜页面及其样式、测试 +- 排行榜模板相关后端测试 +- 本任务文件 +- 生产 SQLite 中活动群 ID 23–27 的三个排行榜/名称配置字段 + +## 禁止修改范围 + +- 不重建或改写 2026-08-28 及更早的消息、排行榜、Prompt、图片和 `run.json`。 +- 不调用生图、Prompt 重建、微信发送或补发接口。 +- 不修改群发送时间、发送目标、图片主题、模型或其他群配置。 +- 不修改已归档群或未来新建群的默认配置。 + +## 已确定实现要求 + +- `text_interactions` 模板在 `{{top_lines}}` 后只追加一次固定说明。 +- 新口径页面显示 `文字 X｜互动 Y`；旧口径和历史 JSON 继续显示 `X 条`。 +- Dashboard 与独立排行榜页面仅在 `count_policy=text_primary_with_interactions` 时显示说明。 +- 生产数据库变更前创建备份；更新前重新核对 ID 23–27 的旧值，随后在一个事务内设置： + - `ranking_count_policy=text_primary_with_interactions` + - `ranking_template=text_interactions` + - `sender_name_policy=wechat_data_analysis` +- ID 28 只核验，不重写；完成后六个活动群的三个字段必须一致。 +- 现有 `uses_strict_image_fact_contract` 继续按排行榜口径启用，因此六群下一次生图统一执行严格事实校验。 + +## 验收标准 + +- 固定说明位于 Top 名单最后且只出现一次。 +- 文字数决定名次；互动不参与排序；系统消息不计数。 +- 新旧 `ranking.json` 均可被前端解析并正确展示。 +- SQLite `integrity_check=ok`，`foreign_key_check` 无结果，六群配置一致。 +- 2026-08-28 输出目录和发送状态未被改写；执行过程中没有发送或生图调用。 +- 后端测试、编译检查、前端测试、生产构建和 `git diff --check` 全部通过。 + +## 测试命令 + +```powershell +.\.venv\Scripts\python.exe -m pytest tests -q +.\.venv\Scripts\python.exe -m compileall -q app tests +Set-Location frontend +npm test -- --run +npm run build +Set-Location .. +git diff --check +``` + +## 返回格式 + +- 报告固定说明、六群最终配置、数据库校验、测试与构建结果。 +- 报告今天/历史产物未重建且未发送。 +- 报告 Git 提交哈希、分支、远端地址和普通推送结果。 diff --git a/GROUPBRIEF_SPEAKER_ATTRIBUTION_CONTRACT_TASK.md b/GROUPBRIEF_SPEAKER_ATTRIBUTION_CONTRACT_TASK.md new file mode 100644 index 0000000..80083d2 --- /dev/null +++ b/GROUPBRIEF_SPEAKER_ATTRIBUTION_CONTRACT_TASK.md @@ -0,0 +1,57 @@ +# 群报说话人归属契约修复任务 + +## 背景 + +- 2026-08-28 刷新消息和排行榜后，旧选题证据与旧 Prompt 未同步失效，导致同一 `message_id` 在 `messages.json` 和 `prompt_meta` 中出现不同说话人。 +- 现有 Poster 编辑模型按姓名选择原话；多人发送相同文本时，仅靠姓名和文本无法证明具体消息归属。 +- 图片署名必须使用消息记录当时的有效显示名；无效、系统事件文本或跨身份冲突时才回退已解析名称。统计身份仍以 `sender_id` 为准。 + +## 目标 + +- 建立 `message_id + sender_id + 当时显示名 + 原文` 的不可拆分证据绑定。 +- 为消息快照和说话人归属生成稳定指纹，并阻止过期选题、Prompt、图片继续进入生图或发送。 +- Poster 编辑模型只引用 `message_id`，姓名由程序从当前证据派生。 +- 保持公开 HTTP API 和数据库结构不变。 + +## 允许修改范围 + +- `app/ai/` 中的消息归属、选题、Poster 编辑和 Prompt 构建。 +- `app/pipeline/`、`app/v2/` 中的快照状态、重建和发送闸门。 +- 数据源适配器的身份溯源字段保留。 +- 对应 Python 测试和本任务文件。 + +## 禁止修改范围 + +- 不修改真实 `output/` 历史产物、生产数据库或微信联系人数据库。 +- 不触发生图、微信发送、邮件发送、服务重启或历史补跑。 +- 不增加图片事后审核或人工审核机制。 +- 不修改 OpenAI Provider、认证方式、API Key 或 `.env`。 + +## 已确定实现要求 + +- 有效且跨 `sender_id` 唯一的 `upstream_sender_name` 作为该条消息的图片显示名；其余情况回退 `sender_name`，必要时稳定同名编号。 +- `evidence_dialogue` 保留 `message_id`、`sender_id`、`speaker`、受控片段和完整原文；完整原文仅供程序校验和内部 provenance 落盘，不提交给编辑模型。 +- Poster 编辑 JSON 的人物项使用 `message_id`、`action`、`quote`；程序按消息 ID 派生姓名并校验 quote 只能来自同一条消息。 +- selection、`prompt_meta` 和 `run.json` 均写入 `message_snapshot_sha256`、`speaker_fingerprint`；Prompt 元数据和 run 另写 `speaker_bindings`。 +- 旧元数据缺少指纹、指纹不匹配、重复消息 ID或证据字段漂移时失败关闭；显式重新选题后才可重建。 +- 刷新消息后保留旧文件用于追溯，但标记 `prompt_stale/image_stale`，保持发送锁。 + +## 验收标准 + +- 相同原话由不同成员发送时，Poster 必须通过 `message_id` 精确绑定对应身份。 +- 同一消息 ID 的姓名、sender_id 或原文改变后，旧选题不能复用。 +- 2026-08-28 类型的快照漂移会被阻断；匹配的新快照可正常构建 Prompt。 +- 刷新消息不调用总结模型、生图或发送；显式重新选题才允许产生新 Prompt。 +- 现有说话人、Prompt、Pipeline、恢复、生图和发送回归测试通过。 + +## 测试命令 + +- `.\.venv\Scripts\python.exe -m pytest -q tests/test_contact_resolver.py tests/test_wechat_mcp.py tests/test_sender_name_policy.py tests/test_topic_selection.py tests/test_v2_prompt_builder.py tests/test_v2_pipeline.py` +- `.\.venv\Scripts\python.exe -m pytest -q tests/test_generation_concurrency.py tests/test_reliability_state.py tests/test_v2_image_task.py tests/test_v2_image_regeneration.py tests/test_period_rules.py tests/test_recovery_planner.py tests/test_v2_wechat_native.py tests/test_image_fact_verification.py` +- `.\.venv\Scripts\python.exe -m pytest -q tests` +- `.\.venv\Scripts\python.exe -m compileall -q app tests` +- `git diff --check` + +## 返回格式 + +- 报告根因、字段契约、实际修改、测试结果、未执行的生产动作、提交哈希、分支和推送结果。 diff --git a/GROUPBRIEF_WECHAT_DAILY_SEND_HARDENING_TASK.md b/GROUPBRIEF_WECHAT_DAILY_SEND_HARDENING_TASK.md new file mode 100644 index 0000000..d269101 --- /dev/null +++ b/GROUPBRIEF_WECHAT_DAILY_SEND_HARDENING_TASK.md @@ -0,0 +1,66 @@ +# GroupBrief 每日微信发送稳定性修复任务 + +## 背景 + +2026-08-29 日报六群在 08:30 自动发送时先因微信主窗口不可见而失败；窗口恢复后， +四群发送成功，Grok 与 Eason 在目标搜索阶段以“匹配数 0”安全停止。两群均未提交文字或图片， +随后因三次明确失败耗尽重试预算进入 `failed_final`。 + +## 目标 + +1. 在不启动新微信实例、不绕过登录的前提下，恢复唯一、已存在的隐藏微信主窗口。 +2. 稳定识别微信 4.1 搜索浮层中的“群聊/最常使用”目标，继续拒绝聊天记录和网络搜索结果。 +3. 保持人工 `send_target` 高于自动群名同步，避免已确认的发送目标被旧数据覆盖。 +4. 为“明确未提交且重试耗尽”的发送失败增加 CAS 保护的无发送恢复入口。 +5. 保留 `SEND_RESULT_UNKNOWN`、歧义目标、错误群名和已发送任务的禁止重试边界。 + +## 允许修改范围 + +- `app/sender/wechat_native.py` 的窗口准备和目标搜索纯函数。 +- `app/v2/run_store.py` 的发送失败状态恢复 CAS。 +- V2 API 中仅重置明确未提交失败的恢复入口。 +- 与上述行为直接相关的后端和前端 API 类型/测试。 + +## 禁止修改范围 + +- 不自动启动新的微信实例，不处理登录、二维码或权限提示。 +- 不放宽到网络搜索结果、聊天记录文本或模糊的多目标匹配。 +- 不清除或重试 `SEND_RESULT_UNKNOWN`，不覆盖 `SENT`、`text_sent_at`、`image_sent_at`。 +- 本任务验证期间不发送任何真实微信文字或图片。 +- 不重启微信、GroupBrief 后端或 Alter 服务。 + +## 已确定实现要求 + +- 只允许恢复标题精确为“微信/WeChat”、进程为 `Weixin.exe/WeChat.exe`、 + 属于当前 Windows 会话且窗口类受信任的唯一隐藏顶层主窗口。 +- 多个可见窗口、多个隐藏候选、锁屏、登录窗口或无法确认进程身份时继续失败关闭。 +- 搜索优先限制在“群聊/最常使用”标题与下一分区边界之间；最终仍需聊天标题复核。 +- 网络结果之后、聊天记录之后的同名文字永不成为群聊候选。 +- 失败恢复要求 `READY_TO_SEND/IMAGE_READY + failed_final + SEND_RETRY_EXHAUSTED`， + 且没有任何已提交、已验证或已发送证据；调用方必须同时提供 + `expected_updated_at` 与 `expected_state_version`。 +- 恢复操作只重置重试状态并保留失败账本，不调用微信发送器。 + +## 验收标准 + +- 唯一隐藏微信主窗口可被恢复，多个或身份不明候选仍被拒绝。 +- Grok 的真实搜索布局能唯一选中“最常使用”结果；网络和聊天记录同名项仍被拒绝。 +- Eason 人工发送目标不被自动同步覆盖。 +- 明确未提交的 `failed_final` 可经 CAS 恢复为 `ready`；过期版本、未知结果、已提交或已发送状态均拒绝。 +- 完整后端测试、前端测试、前端构建、Python 编译和 `git diff --check` 通过。 +- 无真实微信发送调用。 + +## 测试命令 + +- `.\.venv\Scripts\python.exe -m pytest tests -q` +- `npm test -- --run` +- `npm run build` +- `.\.venv\Scripts\python.exe -m compileall app scripts` +- `git diff --check` + +## 返回格式 + +- 根因与修复边界。 +- 改动文件及防错条件。 +- 无发送实机目标校验结果。 +- 测试、提交哈希、分支和推送结果。 diff --git a/GROUPBRIEF_WECHAT_UIA_SEARCH_FIX_TASK.md b/GROUPBRIEF_WECHAT_UIA_SEARCH_FIX_TASK.md new file mode 100644 index 0000000..01585a0 --- /dev/null +++ b/GROUPBRIEF_WECHAT_UIA_SEARCH_FIX_TASK.md @@ -0,0 +1,56 @@ +# GroupBrief 微信群聊搜索验证修复任务 + +## 背景 + +2026-08-30 08:30 六个群的文字与图片均已准备完成，但微信搜索浮层中的小字号“最常使用”标题未被 Windows OCR 识别。现有发送器因“可信分区 0”按设计停止，六群都在提交前失败，没有产生重复发送。 + +## 目标 + +- 保留现有 OCR 分区选择与点击后标题复核。 +- 当分区标题 OCR 缺失时，仅允许唯一且完整匹配 `search_item_<群名>` 的 Windows UI Automation 群聊搜索项作为兜底。 +- 修复后先无副作用验证六个目标，再按用户本次明确授权补发 2026-08-30 六群文字与图片。 + +## 允许修改范围 + +- `app/sender/wechat_native.py` +- `tests/test_v2_wechat_native.py` +- `requirements.txt` +- 本任务文件 + +## 禁止修改范围 + +- AI Provider、认证、`.env` 和模型配置。 +- 群名、群 ID、排行榜文案、Prompt 和已生成图片。 +- 与本故障无关的 CodeMap、前端和调度器逻辑。 +- 任何 `SEND_RESULT_UNKNOWN` 或已提交任务的自动解锁/重发。 + +## 已确定实现要求 + +- UIA 候选必须与微信主窗口属于同一进程。 +- Automation ID 必须严格等于 `search_item_<完整目标群名>`，文本标题也必须匹配，且候选中心必须位于搜索浮层范围内。 +- 匹配数不是 1 时必须失败关闭；禁止选择聊天记录或搜索网络结果。 +- UIA 点击后仍必须通过当前聊天标题 OCR 二次校验。 +- 依赖不可用或 UIA 歧义时不得降级为猜测点击。 + +## 验收标准 + +- 新增 UIA 唯一匹配、歧义、越界和 OCR 失败兜底测试通过。 +- `tests/test_v2_wechat_native.py` 及相关后端测试通过。 +- 实际 `POST /api/groups/{id}/verify-send-target` 对六群全部返回唯一目标。 +- 补发前备份当天六份 `run.json` 和数据库，并用 CAS 解除明确未提交的 `failed_final` 锁。 +- 补发后六群均需要 `SENT`、`submitted=true`、`ui_observed`、`outcome_unknown=false`，且图片 SHA-256 与回执一致。 + +## 测试命令 + +```powershell +.\.venv\Scripts\python.exe -m pytest tests\test_v2_wechat_native.py -q +.\.venv\Scripts\python.exe -m pytest tests -q +git diff --check +``` + +## 返回格式 + +- 根因与修复边界。 +- 代码与依赖变更。 +- 测试、线上目标验证、逐群发送证据。 +- Git 分支、提交、Push、PR 与 CI 状态。 diff --git a/PROJECT_AUDIT.md b/PROJECT_AUDIT.md new file mode 100644 index 0000000..b3a3dbc --- /dev/null +++ b/PROJECT_AUDIT.md @@ -0,0 +1,724 @@ +# GroupBrief V1 全面工程体检 + +> 审计日期：2026-08-24（Asia/Shanghai） +> +> 审计基线：`cf29e2decf5ae74e5a7037531edb0218dc4bdcea` 与该时点工作树 +> +> 审计方式：Codemap + Code Overhaul Full Audit + SonarQube Community 26.8 +> +> 原则：只审计，不修改生产代码，不触发真实微信、邮件或收费模型调用 + +## 0. 审计边界与结论可信度 + +本报告覆盖 `app/`、`scripts/`、`frontend/src/`、`templates/`、测试、当前 SQLite Schema/聚合数据、当前本地进程与计划任务。Codemap 跟踪 116 个生产文件、28,149 行；SonarQube 分析 120 个文件、23,570 NCLOC。 + +审计开始时已有 AI 图片主题、Prompt、前端页面和测试改动；这些改动在审计期间由外部流程提交为 `cf29e2d`。本轮没有改动任何生产业务文件。SonarQube 与测试使用审计时点的隔离快照。 + +以下结论需要明确边界： + +- `/api/system/health` 当前返回 `ok`，只证明服务进程可响应，不证明 WDA、Codex、DeepSeek、SMTP、微信桌面链路都健康。 +- `output/.scheduler/2026-08-24.json` 记录 `generation_status=success`、6 个群均 `ready_to_send`、`email_status=sent`；这说明本地状态机收口，但不是外部邮箱或微信送达的不可变证明。 +- 未读取 `.env` 值或任何密钥。Git 历史扫描覆盖 40 个提交，没有发现常见 Secret 模式或历史跟踪 `.env` 的证据，但这不是专业 Secret Scanner 的绝对保证。 +- 本报告将 Sonar 规则命中与人工源码判断分开；不会把规则分数机械等同于真实风险。 + +--- + +## 1. Executive Summary + +### 1.1 一句话结论 + +**当前 GroupBrief 属于“可用 V1”，健康度 54 / 100。** + +它已经不是 Demo：真实数据接入、群级并发、串行生图、状态持久化、安全发送认领、失败隔离、调度恢复和管理界面都已经形成完整链路；当前后端绑定 `127.0.0.1:8766`、健康接口正常，当日 6 群生成状态已收口。 + +它还不是“稳定 V1”：数据库已有 192 条失去群组关联的 `group_runs`，6 个历史 `runs` 永久停在 `running`；V1/V2 两套生成、持久化、排行、Prompt、API 和调度语义仍同时可触达；损坏的 `run.json` 或 scheduler 状态会被静默解释为“未开始”；Docker 支持路径会把无认证管理/发送 API 绑定到 `0.0.0.0`；完整测试不是全绿且存在顺序依赖。 + +### 1.2 为什么现在能够运行 + +1. `app/main.py:22-45` 在启动时创建目录、初始化 SQLite、执行手写迁移并启动调度器，历史数据库通常能被就地升级。 +2. `app/pipeline/daily_pipeline.py:99-199` 对群级工作并发执行并隔离单群异常，不要求一个群失败就终止整批。 +3. `app/v2/run_store.py:164-181,248-302` 对单个 `run.json` 使用临时文件替换，并为发送提供 claim、lease、`result_unknown` 与人工 hold。 +4. `app/image/codex_generator.py:236-430,705-852` 已有任务级 staging、结构化结果、进程树回收、候选校验、SHA256 与原子提升。 +5. `app/providers/history/wechat_mcp.py:56-145` 对 MCP 有 loopback/允许主机、超时、响应大小和 JSON 边界。 +6. `app/sender/wechat_native.py` 与 Pipeline 的发送状态偏向 fail-closed：无法确认时进入 unknown/hold，而不是直接报告成功。 +7. Python 测试已有 400+ 场景；本轮隔离执行为 413 通过、1 失败，Python statement coverage 约 74.7%。 +8. 当前 V2 默认关闭群级微信自动发送，依赖不完整时通常不会在启动阶段直接造成对外误发。 + +### 1.3 哪些部分只是“目前没出问题” + +- 数据关系靠应用代码约定而非 FK/Unique；真实数据库已经出现逻辑孤儿。 +- SQLite 写入、`run.json`、scheduler JSON、图片/文本工件不是同一个事务；进程中断可能留下跨存储半成功。 +- `schedule_rule`、`summary_model`、`prompt_model`、`provider_preference` 能保存和展示，但 V2 主流程没有完整消费。 +- FastAPI 内 APScheduler 与 Windows Task Scheduler 同时存在，当前主要依赖互斥锁和发送 claim 避免重复，而不是单一调度所有权。 +- AI/SMTP 的“请求已提交但响应丢失”没有全链路幂等键，重试可能重复调用或重复邮件。 +- 前端无自动化测试；真实微信、Codex 生图、SMTP、OCR 发送仍依赖人工实机验收。 + +--- + +## 2. 项目架构图 + +### 2.1 主架构 + +```text +┌────────────────────────── Frontend ──────────────────────────┐ +│ React/Vite Pages ── frontend/src/api.ts ── HTTP/JSON │ +└──────────────────────────────┬───────────────────────────────┘ + ▼ +┌──────────────────── FastAPI / Runtime API ───────────────────┐ +│ app/main.py + app/api/* + app/config/* + app/core/* │ +└──────────────┬───────────────────────────┬────────────────────┘ + │ V2 │ V1 兼容链 + ▼ ▼ +┌──────────────────────────┐ ┌───────────────────────────────┐ +│ DailyPipeline/Scheduler │ │ ReportService/Prompt/Handoff │ +│ generate / recover/send │ │ SQLite Run/GroupRun/Report │ +└──────────────┬───────────┘ └──────────────┬────────────────┘ + │ │ + ┌───────┼────────┬─────────┬───────────┼─────────┐ + ▼ ▼ ▼ ▼ ▼ ▼ + WDA/MCP Ranking AI Prompt ImageGen Email WeChat + Provider Engine Codex/DS Codex CLI SMTP OCR/UI + │ │ │ │ │ │ + └───────┴────────┴─────────┴───────────┴─────────┘ + │ + ▼ + SQLite + output/{group}/{date}/run.json + 工件 +``` + +### 2.2 核心 V2 数据流 + +```text +群配置/稳定 wechat_group_id + → WDA/MCP 取消息 + → messages.json + → 消息归一化/身份聚合 + → RankingEngine → ranking.json / ranking.txt + → 事件提取/选题/版式/Prompt → image_prompt.txt + → Codex ImageGen → daily_image.png + → RunStore 更新 READY_TO_SEND + → 到点 claim 发送权 + → 精确目标验证 + → 文本 + 图片 + → SENT 或 result_unknown/manual hold +``` + +### 2.3 状态流 + +```text +PENDING + └→ DATA_READY + └→ RANKING_READY + └→ PROMPT_READY + └→ IMAGE_READY + └→ READY_TO_SEND + ├→ SENT + ├→ FAILED + └→ result_unknown / manual hold + +任意生成阶段可进入 FAILED；但损坏 run.json 当前可能被重置为新的 PENDING。 +``` + +### 2.4 Codemap 模块健康 + +Codemap 平均分约 **60.5 / 100（C）**。没有 A/B 模块；5 个 D、7 个 C。 + +| 模块 | 行数 | 分数 | 主要原因 | +|---|---:|---:|---| +| V1 报告兼容链 | 696 | 52 / D | 假成功、路径边界、双状态 | +| Pipeline 与调度 | 2,160 | 54 / D | God Service、静默回退、双调度 | +| 运行时与 HTTP API | 2,419 | 58 / D | 无认证边界、833 行 V2 API、配置漂移 | +| AI 摘要与提示词 | 3,601 | 58 / D | 多层重试、Prompt 契约散落、God Builder | +| 运维脚本 | 652 | 58 / D | 失败仍退出 0、半安装、只读命令写 DB | +| 图片生成与恢复 | 1,951 | 61 / C | 高复杂度、双认领协议、无锁状态同步 | +| 微信与邮件交付 | 1,725 | 61 / C | SMTP 幂等缺口、Legacy CLI、God Driver | +| 前端界面 | 10,683 | 62 / C | 5,939 行 CSS、God 页面、轮询/N+1 | +| 数据库与运行状态 | 1,125 | 62 / C | 无 FK、双持久化、损坏状态重置 | +| 微信数据接入 | 2,334 | 64 / C | 858 行 Provider、V1/V2 双模型、Mock fallback | +| 排行榜与统计 | 376 | 64 / C | V1/V2 双引擎/双渲染 | +| 前端 API 客户端 | 427 | 72 / C | 双协议、无超时/取消、弱错误类型 | + +修改影响最大的路径是： + +```text +DailyPipeline + ├→ RunStore / SQLite + ├→ WDA/MCP + ├→ Ranking + ├→ Prompt/Codex/DeepSeek + ├→ ImageGen + ├→ GroupNameSync + └→ WeChat/Email +``` + +因此 `daily_pipeline.py` 的任何“简单修改”都需要至少覆盖生成、恢复、并发、状态持久化、发送 claim 和外部结果未知分支。 + +--- + +## 3. 项目健康度 + +### 3.1 总分：54 / 100 + +| 维度 | 分数 | 依据 | +|---|---:|---| +| 架构合理性 | 5 / 10 | 模块职责大体可识别，但 V1/V2 并存、存储层反向依赖旧服务、核心 God Service 明显 | +| 业务逻辑 | 6 / 10 | 主链可运行并隔离单群失败；群级配置未完全生效、状态与工件可分叉 | +| 代码质量 | 5 / 10 | Codemap 60.5/C；Sonar 295 smells；多个 600–1,300 行核心文件 | +| 数据设计 | 3 / 10 | 192/224 `group_runs` 孤儿、无 FK/复合唯一、6 个永久 running、手写迁移漂移 | +| 稳定性 | 6 / 10 | 有 mutex、claim、hold、staging 和原子替换；损坏状态静默重置、双调度与退出码仍弱 | +| 测试 | 6 / 10 | Python 413 通过、覆盖较广；1 失败且单测顺序依赖，前端/E2E/真实外部边界缺失 | +| 安全性 | 5 / 10 | 当前 loopback、Secret 忽略、MCP allowlist 较好；Docker 无认证暴露、旧文件路径检查和敏感日志有风险 | +| 性能与资源 | 6 / 10 | 当前规模可承受；存在前后端 N+1、轮询、串行图片长尾、多层 AI 重试和 SQLite 写竞争 | +| 可观测性 | 5 / 10 | 有分类日志、run.json、scheduler 状态；部分异常被吞、健康 GET 有副作用、脚本退出码误导 | +| 文档与可维护性 | 7 / 10 | README、架构、CI、恢复文档较多；V1/V2 文档/代码边界仍漂移，缺版本化迁移与决策记录 | + +### 3.2 已经可靠的部分 + +- V2 发送 claim/lease/unknown hold。 +- Codex 图片 staging、验证、哈希、进程树回收。 +- WDA/MCP 允许主机、超时、响应大小与 JSON 校验。 +- 群级失败隔离和全局图片串行队列。 +- 当前 SQLite `PRAGMA integrity_check=ok`；没有发现重复非空 `wechat_group_id`、重复 `(run_id, group_id)` 或孤儿 Report。 +- 当前服务实际监听 `127.0.0.1:8766`，不是 `0.0.0.0`。 +- 前端严格 TypeScript，当前 `npm run build` 通过。 + +### 3.3 逻辑完整性现状 + +| 项目 | 结果 | +|---|---:| +| SQLite 物理完整性 | `ok` | +| `group_runs` 总数 | 224 | +| 找不到 `groups.id` 的 `group_runs` | 192 | +| `runs.status=running` 且未结束 | 6 | +| 孤儿 Report | 0 | +| 重复非空 WeChat Group ID | 0 | +| 重复 `(run_id, group_id)` | 0 | +| SQLite `foreign_keys` | 0 | +| SQLite journal mode | `delete` | + +物理完整性正常不等于业务一致性正常；当前主要问题正是“数据库文件没坏，但关系语义已丢失”。 + +--- + +## 4. SonarQube 客观指标 + +### 4.1 扫描信息 + +- SonarQube：Community Build `26.8.0.126808` +- Project Key：`groupbrief-v1-local-audit` +- Analysis ID：`1a07a159-5e1b-41b1-a74b-f68ee6dfe58a` +- CE Task：`02a52972-25c5-4dcc-8493-1bf7f0d9c2dd` +- 状态：`SUCCESS`，0 warning +- 本地 Dashboard： + +### 4.2 指标 + +| 指标 | SonarQube 结果 | 人工解释 | +|---|---:|---| +| Bugs | 5 | 0 个已确认高风险；3 个正则分组、1 个变量重赋值、1 个字符串默认排序均偏低价值 | +| Vulnerabilities | 12 | 全部为非安全用途随机数或 fixture MD5，人工判定低价值命中 | +| Security Hotspots | 0 | 不代表安全；真正的 Docker/认证/路径风险由源码审查发现 | +| Code Smells | 295 | 其中复杂度/大模块值得修；大量弃用图标、API 文档、readonly props 属 P3 | +| Duplication | 0.2% | 5 blocks / 68 lines；全局不严重，局部前端最高 9.7% | +| Coverage | 63.9% | Python 约 74.7%，前端 0%，因此总体下降 | +| Cognitive Complexity | 3,703 | 与 Codemap God Service 结论高度一致 | +| Cyclomatic Complexity | 4,368 | 热点集中在 Pipeline、WDA、Codex、Native Sender 和页面 | +| Maintainability Rating | A | 技术债比率仅 0.3%；不应解读为架构健康 | +| Reliability Rating | D | 由规则级 Bug 候选拉低；需人工确认，当前 5 个均未证明真实故障 | +| Security Rating | D | 由 12 个随机数/MD5 命中拉低；与真实安全边界不一致 | +| Technical Debt | 2,141 分钟 | 约 35 小时 41 分；仅是规则修复估算，不包含迁移/数据治理 | +| Technical Debt Ratio | 0.3% | 估算开发成本基数很大，使比例看起来漂亮 | + +Sonar Quality Gate 显示 `OK`，但当前门只检查“新问题为 0”，并以同日先前扫描为比较基线。它不能证明项目达到了可发布质量门槛。 + +### 4.3 最复杂文件 + +| 文件 | Cognitive | Cyclomatic | Sonar Coverage | +|---|---:|---:|---:| +| `app/pipeline/daily_pipeline.py` | 265 | 212 | 86.8% | +| `app/providers/history/wechat_data_analysis.py` | 222 | 193 | 79.8% | +| `app/image/codex_generator.py` | 213 | 160 | 71.9% | +| `frontend/src/pages/v2/AIImages.tsx` | 162 | 250 | 0% | +| `app/sender/wechat_native.py` | 161 | 178 | 56.0% | +| `scripts/codex_image_automation.py` | 154 | 145 | 68.5% | +| `app/api/v2_ui.py` | 130 | 145 | 51.7% | +| `frontend/src/pages/v2/Archive.tsx` | 109 | 236 | 0% | + +### 4.4 最复杂函数/组件 + +| 位置 | 函数/组件 | Cognitive | +|---|---|---:| +| `app/pipeline/daily_pipeline.py:726` | `_send_one` | 59 | +| `app/pipeline/daily_pipeline.py:295` | `_generate_one` | 45 | +| `app/ai/conversation_segments.py:108` | `segment_messages` | 43 | +| `app/image/codex_generator.py:246` | `_generate_locked` | 42 | +| `frontend/src/pages/v2/AIImages.tsx:130` | `AIImages` | 38 | +| `app/ai/prompt_builder.py:243` | `build` | 37 | +| `app/services/group_name_sync.py:65` | `sync` | 34 | +| `app/services/email_service.py:52` | `build_email` | 33 | +| `app/ai/topic_selection.py:314` | `score_and_select_topics` | 31 | +| `app/providers/history/wechat_data_analysis.py:294` | `_fetch_messages_mcp` | 31 | + +### 4.5 重复最严重位置 + +| 文件 | Duplication | +|---|---:| +| `frontend/src/pages/v2/ChatRecords.tsx` | 9.7% | +| `frontend/src/pages/v2/Tasks.tsx` | 5.4% | +| `frontend/src/pages/v2/Ranking.tsx` | 4.9% | +| `frontend/src/pages/v2/AIImages.tsx` | 2.0% | + +这与源码审查发现的 `STATUS_LABELS`、`runKey`、`statusTone`、加载/错误状态重复相符，属于值得处理的局部重复；全局 0.2% 不值得为了评分做大规模抽象。 + +### 4.6 值得修与低价值规则 + +**值得修：** + +- `python:S3776` / `typescript:S3776`：47 个复杂函数候选，与 Codemap 的 God Component 结论交叉确认。 +- CSS contrast 19 处：真实可访问性问题，应在 UI 稳定后按页面修。 +- Vite/npm audit：1 high + 1 moderate，属于开发服务器路径遍历/UNC/读取风险，应规划工具链升级。 +- 前端 0% coverage、`v2_ui.py` 51.7%、`wechat_native.py` 56%、`wechat_cli.py` 37%：比追求总百分比更有价值。 + +**低价值或需人工确认：** + +- 12 个 Sonar Vulnerability：`random.Random` 用于每日视觉风格与 fixture，MD5 用作假数据 ID，不承担密码学安全。 +- 3 个 `_FENCE_RE` 正则分组 Bug：当前表达式意图就是“开头 fenced 或结尾 fenced”，未发现真实错误。 +- `messageTypes.sort()`：集合内容是字符串，默认字典序符合当前展示意图。 +- 203 个 Phosphor 图标弃用命中：主要是库 API 演进，不能为清零而一次性替换所有图标。 +- 53 个 FastAPI `responses` 文档命中、28 个 readonly props、重复主题分类文字：属于 P3 工程卫生。 + +--- + +## 5. 验证与测试体系 + +| 验证 | 结果 | 解释 | +|---|---|---| +| Python compileall | PASS | `app scripts tests` 编译通过 | +| Python 全量测试 + Coverage | 413 passed, 1 failed, 1 warning | 唯一失败是并发耗时 `<0.45s`，实际约 0.488s | +| 失败测试独立运行 | 3/3 失败 | 独立运行变为 `UnboundExecutionError`，证明还存在全局 DB engine/测试顺序依赖 | +| Python statement coverage | 74.7% | 6,481 / 8,671；没有 branch coverage | +| Sonar overall coverage | 63.9% | 包含 0% 前端 | +| Frontend build/typecheck | PASS | `tsc -b && vite build`；4,596 modules | +| Frontend bundle | 451.30 kB JS / 131.50 kB gzip | 单主 chunk；当前可接受但缺 route split | +| Frontend unit/component/E2E | N/A | 未配置 Vitest/Jest/Playwright/Cypress | +| Lint/format | N/A | Python 无 Ruff/Mypy；前端无 ESLint/Prettier | +| `pip check` | PASS | 无破损依赖 | +| `pip-audit` | PASS | 当前解析版本未发现已知 Python 漏洞 | +| `npm audit` | FAIL | 1 high（Vite）+ 1 moderate（esbuild） | +| SonarQube | PASS | 二次扫描成功，Coverage 已实际导入 | + +低覆盖重点： + +- `app/providers/history/wechat_cli.py` 37% +- `app/api/v2_templates.py` 40% +- `app/api/logs.py` 42% +- `app/api/v2_ui.py` 52% +- `app/sender/wechat_native.py` 56% +- `app/providers/ai/deepseek.py` 62% + +完全依赖人工/实机验证的核心流程： + +- 真实 WDA/MCP 数据读取与多账号联系人映射。 +- Codex CLI 登录环境、长时生图、进程树异常与真实产物归属。 +- 微信窗口、DPI、OCR、剪贴板、目标精确匹配、文本/图片实际提交。 +- SMTP 服务端“已接收但客户端断连”的结果未知场景。 +- Windows 重启、锁屏、休眠、计划任务与 FastAPI 内 scheduler 同时存在时的行为。 + +--- + +## 6. 问题优先级与详细问题 + +### P0-1 Docker 模式无认证暴露管理/发送能力，且旧文件路径边界不足 + +- **位置：** `docker-compose.yml:15-27`、`Dockerfile:42-43`、`app/main.py:48-66`、`app/api/files.py:19-34`、`app/services/handoff_service.py:34,105` +- **模块：** 运行时/API、V1 报告兼容链 +- **来源：** Codemap + Code Overhaul；Sonar 未发现 +- **原因：** Docker 使用 `8766:8766` + `APP_HOST=0.0.0.0`；所有设置、生成、发送、邮件、删除/恢复接口无认证。旧文件接口用字符串 `startswith` 做 containment，V1 日期/目录参数校验不足。 +- **实际影响：** 若宿主防火墙/路由允许访问，局域网用户可修改路径/Provider、触发生图/发送/邮件或读取越界文件。 +- **发生概率：** 当前 Windows 进程绑定 loopback 时低；启用仓库提供的 Docker 路径时中到高。 +- **收益 / 成本 / 风险：** 收益极高；成本中；若直接加复杂账号体系风险高，若先限定 loopback + 管理令牌则风险低。 +- **Blast Radius：** 全部数据、外部发送、配置、宿主文件边界。 +- **推荐：** 先将 Docker 端口显式绑定 `127.0.0.1:8766:8766`，用 `Path.is_relative_to`/等价方式统一 containment；若需要 LAN，增加最小管理令牌和路由权限分级。不要引入企业级 IAM。 + +### P0-2 历史数据关系已丢失，缺少 FK/Unique 使问题可继续累积 + +- **位置：** `app/db/models.py:10-79`、`app/db/repository.py:331-425`、`app/api/runs.py:29-37` +- **模块：** 数据库与运行状态 +- **来源：** Codemap + Code Overhaul + 真实 SQLite 聚合 +- **原因：** `GroupRun.run_id/group_id`、`Report.group_run_id` 无 FK；`wechat_group_id`、`(run_id,group_id)`、`reports.group_run_id` 无业务唯一约束；删除/历史迁移靠应用层约定。 +- **实际影响：** 224 条 `group_runs` 中 192 条失去群组关系；历史页面只能回退为 `群 {id}`，统计/恢复可能关联错误。 +- **发生概率：** 已发生。 +- **收益 / 成本 / 风险：** 收益极高；成本高；直接加约束会因现有脏数据失败，必须先备份、分类和回填。 +- **Blast Radius：** 历史报告、统计、恢复、删除策略、未来迁移。 +- **推荐：** 单独执行“只读分类 → WAL 感知备份 → 映射/归档孤儿 → 临时表迁移 → FK/Unique → foreign_key_check → 回滚演练”。不得在普通启动中顺手修。 + +### P1-1 损坏状态被静默重置，可能把“结果未知”解释为“未开始” + +- **位置：** `app/v2/run_store.py:153-162,339-364`、`app/scheduler/daily_v2_job.py:40-61` +- **模块：** 持久化、调度 +- **来源：** Codemap + Code Overhaul +- **原因：** JSONDecodeError/OSError 后返回新 PENDING/空日状态，而不是 quarantined/corrupt。 +- **实际影响：** 已生成、已发邮件或已提交微信但未写完成标记时，重启后可能重新生成或重复外部副作用。 +- **发生概率：** 低到中；断电、磁盘错误、跨进程覆盖时升高。 +- **收益 / 成本 / 风险：** 收益高；成本中；修复需兼容旧状态文件。 +- **Blast Radius：** 单群单日；scheduler 文件损坏时整批。 +- **推荐：** 引入 `CORRUPT/RESULT_UNKNOWN` 隔离状态、保留原文件、manifest-last 校验和人工恢复入口；任何外部提交阶段都不得自动回退 PENDING。 + +### P1-2 V1 状态存在假成功、永久 running 和失败退出码 0 + +- **位置：** `app/services/report_service.py:130-187,247-273,317`、`app/services/handoff_service.py:52-95`、`scripts/run_daily_pipeline.py:35,99-124` +- **模块：** V1 报告兼容链、运维脚本 +- **来源：** Codemap + Code Overhaul + 真实 SQLite +- **原因：** 多阶段 commit；worker 异常新建失败记录而非收口旧记录；成功只看 ranking/prompt 状态，不看工件；CLI 分支无条件返回 0。 +- **实际影响：** 当前有 6 个 `runs=running`；文件缺失/部分失败仍可能被监控视为成功。 +- **发生概率：** 已发生。 +- **收益 / 成本 / 风险：** 收益高；成本中；旧调用方可能依赖当前退出码，需版本化改变。 +- **Blast Radius：** V1 API、Windows 发送任务、历史统计。 +- **推荐：** 先定义可机读 terminal contract；CLI 对 failed/partial/blocked/already_running 分别返回非零或独立码；为旧 running 提供只读审计和显式归档，不自动猜测成功。 + +### P1-3 V1/V2 两套仍可触达的业务系统造成状态分叉 + +- **位置：** `app/services/report_service.py`、`app/pipeline/daily_pipeline.py`、`app/db/models.py:38-79`、`app/v2/run_store.py`、`app/ranking/*`、`app/services/ranking_service.py`、`frontend/src/api.ts:22-365` +- **模块：** 全链路 +- **来源：** 多方共同发现 +- **原因：** 迭代时保留旧 API/服务/数据库/输出/脚本，同时新 V2 使用独立 run.json 与状态协议。 +- **实际影响：** 两个页面/脚本可以对“最新状态”给出不同答案；修一个排行/Prompt/发送问题需要检查两套实现。 +- **发生概率：** 高。 +- **收益 / 成本 / 风险：** 收益高；成本高；一次性删除风险极高。 +- **Blast Radius：** API、前端、数据库、调度、邮件、历史。 +- **推荐：** 冻结 V1 新功能，先记录真实调用者与数据保留要求；用适配层只读 V1 历史，新生成只走 V2；最后分轮退役写路径。 + +### P1-4 群级配置能保存但未完整进入执行路径 + +- **位置：** `app/db/models.py:18,24-27`、`app/scheduler/period.py:33-47`、`app/pipeline/daily_pipeline.py:124,1045,1115` +- **模块：** Pipeline、设置/API +- **来源：** Codemap + Code Overhaul +- **原因：** UI/DB 先扩展字段，Pipeline 仍使用全局默认或固定 Provider。 +- **实际影响：** 用户以为已配置 `schedule_rule/summary_model/prompt_model/provider_preference`，实际运行仍可能走默认值。 +- **发生概率：** 高，只要使用这些字段。 +- **收益 / 成本 / 风险：** 收益高；成本中；真正启用模型路由可能改变费用与结果，必须显式验收。 +- **Blast Radius：** 单群日期窗口、Provider、模型调用。 +- **推荐：** 为每个字段建立“保存 → API 回读 → Pipeline 消费 → run.json 审计”的契约测试；未实现字段应在 UI 标为不可用，而不是静默保存。 + +### P1-5 外部调用重试缺少一致幂等边界 + +- **位置：** `app/providers/ai/codex.py:123-221`、`app/providers/ai/deepseek.py:143-259`、`app/services/email_service.py:111-243`、`scripts/send_daily_email.py:203-243` +- **模块：** AI、交付 +- **来源：** Codemap + Code Overhaul +- **原因：** Builder/Provider 多层重试与 fallback 叠加；SMTP 断线后重发同一消息；没有 request/message idempotency key。 +- **实际影响：** Provider 已接收但响应丢失时可能重复计费；SMTP 已接收时可能重复邮件。 +- **发生概率：** 中；日志已有连接中断类历史证据。 +- **收益 / 成本 / 风险：** 收益高；成本中到高；错误实现幂等可能阻止合法重试。 +- **Blast Radius：** 单个 Prompt chunk、单群邮件或整批。 +- **推荐：** 合并重试所有权；只对 429/明确 5xx/连接前失败重试；为邮件生成稳定 Message-ID 和本地发送 ledger；结果未知时进入 hold。 + +### P1-6 真实数据失败可在 V1 静默落入 Mock + +- **位置：** `app/providers/history/registry.py:21-47`、`app/services/history_service.py:76-128` +- **模块：** 微信数据接入 +- **来源：** Codemap + Code Overhaul +- **原因：** `history_provider_mock_enabled` 时 Mock 自动追加，首个 OK/EMPTY_RESULT 即返回。 +- **实际影响：** 真实 WDA/MCP 不可用时，可能生成结构正常但事实不真实的报告。 +- **发生概率：** 中；当前数据库设置显示 Mock 可启用，真实 Provider 故障历史存在。 +- **收益 / 成本 / 风险：** 收益高；成本低；会减少“可演示性”但提升真实性。 +- **Blast Radius：** 对应日期的全部 V1 报告。 +- **推荐：** 生产模式 fail closed；Mock 只能由显式开发变量 + fixture 标识启用，所有产物必须携带 `source=mock` 且禁止发送。 + +### P1-7 双调度入口与误导性退出码削弱无人值守可信度 + +- **位置：** `app/scheduler/manager.py:36-82`、`scripts/install_daily_task.py:35-71,114`、`scripts/daily_auto.py:69` +- **模块：** 调度、运维脚本 +- **来源：** Codemap + Code Overhaul + 当前 Windows 状态 +- **原因：** FastAPI 内 APScheduler 和 Windows `GroupBriefDaily/GroupBriefDailySend` 同时存在；`already_running` 和安装失败可返回 0。 +- **实际影响：** 任务可能只是被另一实例抢锁，却被监控视为完成；半安装计划任务无法由退出码发现。 +- **发生概率：** 中；当前两类调度都存在。 +- **收益 / 成本 / 风险：** 收益高；成本低到中；切换调度所有权时需避免漏跑。 +- **Blast Radius：** 每日整批生成与发送。 +- **推荐：** 每个部署只保留一个 scheduler owner；另一个只做 watchdog；为 `success/partial/blocked/already_running/not_run` 定义稳定退出码与监控事件。 + +### P1-8 Schema 迁移无版本链且默认值已漂移 + +- **位置：** `app/db/repository.py:34-207` +- **模块：** 数据库与运行状态 +- **来源：** Codemap + Code Overhaul + Schema 对比 +- **原因：** `create_all + PRAGMA + ALTER + settings marker` 在启动时执行，多个迁移各自 commit。 +- **实际影响：** 当前 SQLite 列默认仍保留旧 DeepSeek/blue_white，而运行数据已使用 Codex/random_preset；多进程或中断可产生部分迁移。 +- **发生概率：** 中。 +- **收益 / 成本 / 风险：** 收益高；成本中高；贸然引入 Alembic 也可能过度设计。 +- **Blast Radius：** 启动、恢复、所有新建群。 +- **推荐：** 不必立刻上复杂框架；先建立单一 `schema_version`、顺序迁移注册、每版事务/备份/校验和回滚脚本，再评估 Alembic。 + +### P2-1 复杂度集中在少数 God Module + +- **位置：** `daily_pipeline.py`、`codex_generator.py`、`wechat_data_analysis.py`、`wechat_native.py`、`v2_ui.py`、`AIImages.tsx`、`styles.css` +- **模块：** 多模块 +- **来源：** Codemap + Code Overhaul + Sonar +- **原因：** 快速 V1 迭代把协议、状态、I/O、恢复和 UI 交互集中在同一个类/文件。 +- **实际影响：** 修改成本和回归范围持续增大；Sonar top function complexity 达 59。 +- **发生概率：** 每次维护都发生。 +- **收益 / 成本 / 风险：** 收益中高；成本高；无 characterization tests 时拆分风险高。 +- **Blast Radius：** 核心生成/发送与管理页面。 +- **推荐：** 先补行为网，再按阶段提取纯函数/边界对象；不引入微服务、CQRS、事件溯源或大量模式。 + +### P2-2 测试存在顺序依赖、时间阈值脆弱和前端/E2E 空白 + +- **位置：** `tests/conftest.py:13-23`、`tests/test_generation_concurrency.py:147`、`frontend/package.json:6-9` +- **模块：** 测试体系 +- **来源：** Code Overhaul + 实测 + Sonar Coverage +- **原因：** 全局 repository engine 由其他测试隐式初始化；并发测试用硬墙钟 `<0.45s`；前端没有测试框架。 +- **实际影响：** 全量 413/1 失败，单独运行同一测试 3/3 因 unbound engine 失败；CI 结果可能随顺序/负载变化。 +- **发生概率：** 已发生。 +- **收益 / 成本 / 风险：** 收益高；成本中；不应通过放宽所有断言掩盖真实并发回归。 +- **Blast Radius：** CI 可信度、并发/状态修改安全网。 +- **推荐：** 每测试 fixture 显式 init/dispose DB；用事件/barrier 验证并发而非极窄墙钟；先为 AIImages、Archive、Settings 和发送确认补行为测试，再选少量 Playwright 实机前流程。 + +### P2-3 前后端 N+1、轮询与单主 Bundle + +- **位置：** `app/api/runs.py:25-31`、`app/api/reports.py:59-63`、`frontend/src/pages/v2/Tasks.tsx:113`、`AIImages.tsx:376-388` +- **模块：** API、前端 +- **来源：** Code Overhaul + Codemap +- **原因：** 每行/每 run 再请求明细；固定 2/5 秒轮询失败后静默继续；所有页面打入单主 JS chunk。 +- **实际影响：** 当前小数据影响有限，历史增长后请求数和页面等待线性增长；网络失败会持续轮询。 +- **发生概率：** 中。 +- **收益 / 成本 / 风险：** 收益中；成本低到中；过早引入复杂缓存风险大。 +- **Blast Radius：** Tasks/AIImages/历史列表。 +- **推荐：** 先批量查询/分页、AbortController、退避和可见错误；页面级动态 import 可后置。 + +### P2-4 可观测性存在“看似成功”的空洞 + +- **位置：** `app/main.py:29-35`、`app/scheduler/send_job.py:12-19`、`app/core/logging.py:68`、`app/api/system.py:23-48` +- **模块：** 运行时、调度 +- **来源：** Codemap + Code Overhaul +- **原因：** 启动检查异常被置空；send job 仅日志不返回失败；已有 root handler 时分类日志不初始化；Provider GET 会外呼并写 DB。 +- **实际影响：** 服务“活着”但依赖不可用；scheduler 认为任务完成；健康刷新本身消耗资源并扩张 `provider_health`。 +- **发生概率：** 中。 +- **收益 / 成本 / 风险：** 收益中高；成本中；强制启动门禁可能降低本地可用性。 +- **Blast Radius：** 运维判断、监控、Provider 表。 +- **推荐：** 区分 liveness/readiness/deep diagnostics；调度器记录终态与退出码；深健康检查显式触发并限频，ProviderHealth 设置 retention。 + +### P2-5 依赖可复现性和 Vite 开发服务器漏洞 + +- **位置：** `requirements.txt`、`requirements-dev.txt`、`frontend/package.json`、`.github/workflows/ci.yml` +- **模块：** 工具链 +- **来源：** Code Overhaul + `pip-audit` + `npm audit/outdated` +- **原因：** Python 全部使用 `>=` 且无 lock；前端锁定 Vite 5.4.21，审计发现 1 high + 1 moderate；多个前端直接依赖已落后一个或更多 major。 +- **实际影响：** Python 重建可能解析到不同组合；Vite dev server 在 Windows/路径场景存在风险。 +- **发生概率：** 中。 +- **收益 / 成本 / 风险：** 收益中；成本中；直接跳 React 19/Vite 8 有较大兼容风险。 +- **Blast Radius：** 本地开发、CI、Docker 构建。 +- **推荐：** 先引入 Python constraints/lock 和 Renovate/Dependabot；Vite 单独升级并回归 build/dev；不要把 React major 升级和核心稳定性整改绑在同一轮。 + +### P2-6 工件写入与目录身份仍可能碰撞 + +- **位置：** `app/v2/run_store.py:119-121`、`app/services/handoff_service.py:34-92`、`app/pipeline/daily_pipeline.py:433-671` +- **模块：** 持久化、V1 报告、Pipeline +- **来源：** Codemap + Code Overhaul +- **原因：** 目录使用清洗/截断后的群名；多工件顺序写入；V1 handoff 非原子。 +- **实际影响：** 同名/清洗后相同群可能覆盖；状态和工件不一致。 +- **发生概率：** 当前未发现活动群碰撞，未来中低。 +- **收益 / 成本 / 风险：** 收益中；成本中高；改目录会影响历史 URL/归档。 +- **Blast Radius：** 单群全部历史工件。 +- **推荐：** 新目录加入稳定 group_id，保留旧目录只读映射；用临时执行目录 + manifest-last 原子发布；不得直接批量搬迁现有 output。 + +### P3 低 ROI 工程卫生 + +- Phosphor 图标弃用、readonly props、FastAPI `responses` 文档、主题标签常量、少量 wrapper 页面、通知 placeholder、目录命名美化。 +- 这些问题可以在相关页面/模块被修改时顺手处理，不应占用 P0/P1 稳定性预算。 + +--- + +## 7. 技术债 Top 10 + +按“风险 × 影响 × 未来维护成本 × 修改收益”排序： + +| 排名 | 技术债 | 优先级 | 为什么排在这里 | +|---:|---|---|---| +| 1 | Docker 无认证暴露 + 旧路径边界 | P0 | 可直接扩大为配置、文件和外部发送控制权 | +| 2 | 192 条逻辑孤儿 + 无 FK/Unique | P0 | 已发生的数据关系损失，会继续积累 | +| 3 | 损坏状态静默回退 PENDING | P1 | 可能把结果未知变成重复执行 | +| 4 | V1 假成功/永久 running/退出码 0 | P1 | 已有 6 条卡死记录，破坏监控可信度 | +| 5 | V1/V2 双生成与双状态体系 | P1 | 所有未来功能都要承担双倍维护与状态分叉 | +| 6 | 群级配置未真正路由 | P1 | 用户配置与实际行为不一致，涉及日期和模型费用 | +| 7 | AI/SMTP 重试无统一幂等 | P1 | 可能重复扣费或重复邮件 | +| 8 | 双 scheduler owner + 误导退出码 | P1 | 无人值守可能“没跑却显示成功” | +| 9 | 核心 God Module 与复杂函数 | P2 | 每次修复 Blast Radius 大，长期维护成本复利增长 | +| 10 | 测试顺序依赖 + 前端/E2E 空白 | P2 | 阻碍安全拆分与真实失败场景验证 | + +--- + +## 8. 删除候选清单 + +本轮只列出，不删除。即使标为“可以安全删除”，也应在独立清理轮次执行 `rg → build/test → diff`。 + +### 8.1 可以安全删除候选 + +- `frontend/src/pages/v2/History.tsx`：仅包装 `Archive`，导航已归一到 archive，未发现引用。 +- `frontend/src/pages/v2/System.tsx`：仅包装 `Settings`，导航已归一到 settings，未发现引用。 +- `app/providers/v2/base.py`：自称预留接口，全仓未发现实际引用。 + +### 8.2 需要确认后删除 + +- `app/scheduler/generate_job.py`、`app/scheduler/email_job.py`：当前 manager 不注册，但外部脚本可能仍调用。 +- `app/services/prompt_service.py:_build_context_text`：旧调用兼容候选。 +- `app/providers/history/wechat_cli.py`：V1 registry/配置仍可能路由。 +- `scripts/codex_image_automation.py`、`docs/CODEX_IMAGE_AUTOMATION_PROMPT.md`：旧桌面流程，仍可能用于人工恢复。 +- `scripts/test_wechat_send.py`：固定 legacy sender，作为测试价值低，但可能是现场手工工具。 +- V1 `ReportService/RankingService/HandoffService` 与旧文件 API：必须先证明没有真实调用者并迁移历史读取。 +- `archive-legacy` CSS 与其它旧页面样式：必须先做实际 DOM/CSS coverage，不能凭名称删除。 +- 无引用的 notification 按钮：删除会改变 UI，不属于纯 dead code。 + +### 8.3 暂时不要删除 + +- Mock Provider 与 `fixtures/`：Python 测试和本地开发仍使用；应限制生产路由，不是直接删除。 +- 所有 migration marker/旧列兼容：尚未证明所有数据库都完成升级。 +- V1 SQLite 表与历史记录：即使停止写入，也需保留只读历史迁移期。 +- `top10_lines`、`deepseek_ms`、旧模型字段等兼容数据：先建立版本边界和读取统计。 +- 发送 claim/lease/hold、Codex staging/recovery、WDA allowlist：这些是当前可靠性保护。 + +--- + +## 9. 暂时不要动的地方 + +以下代码可能不漂亮，但运行价值高、修改收益低于风险： + +1. `app/v2/run_store.py:248-302` 的发送 claim、lease、`result_unknown` 和人工 hold。 +2. `app/pipeline/daily_pipeline.py:726-1010` 的发送前认领、提交后未知结果和 fail-closed 分支。 +3. `app/image/codex_generator.py:42-140,236-430,705-852` 的 Windows 进程树终止、跨进程互斥、staging、候选归属和原子提升。 +4. `app/providers/history/wechat_mcp.py:56-145` 的 loopback/allowed-host、超时、响应上限和 JSON 校验。 +5. `app/pipeline/daily_pipeline.py:368-388` 对消息快照无效时拒绝静默回源。 +6. `app/services/group_name_sync.py` 基于稳定 WeChat ID 同步群名并保留人工 `send_target` 的逻辑。 +7. Docker 单 worker 与全局 generation mutex。它们不是最终并发方案，但当前保护 SQLite 和外部副作用。 +8. 数据库设置覆盖 `.env` 的既有行为。来源层次需要清晰化，但不能直接反转，否则会改变生产配置。 +9. 图片全局串行队列。它会拉长总耗时，但符合当前 Codex/桌面资源约束；在有性能证据前不要盲目并行。 + +--- + +## 10. Code Overhaul 影响/成本矩阵 + +| | 低成本 | 高成本 | +|---|---|---| +| **高影响** | Docker loopback、Path containment、非法设置拒绝、脚本退出码、Mock 生产禁用 | 孤儿数据治理/FK、V1/V2 收敛、稳定 group_id 目录、幂等 ledger | +| **低影响** | CSS contrast、客户端超时/取消、错误类型、少量 dead wrapper | 全量图标替换、为清零重复率抽象、全面 CSS 重写、框架 major 迁移 | + +### 当前已有、应复用的能力 + +- `RunStore` 原子 JSON 替换和发送 lease。 +- `generation_mutex` / 单 worker。 +- `verify_image`、Codex staging/manifest/hash；应补强而不是重写。 +- MCP allowed-host/timeout/size guard。 +- `GroupNameSyncService` 稳定身份语义。 +- `tests/conftest.py` 已有外部调用隔离意图；需改为每测试独立 DB 生命周期。 +- GitHub Actions 已有 Windows Python test + frontend build 基线。 + +### 本轮不在范围 + +- 不重构、不修 Sonar、不加 Schema、不删代码、不升级依赖。 +- 不创建 Beads；用户要求报告完成后停止。 +- 不执行真实微信、邮件、WDA、Codex、DeepSeek 验收。 +- 不改变部署、计划任务或当前运行进程。 + +--- + +## 11. 整改路线图 + +每轮都必须小范围、可独立测试、可独立提交、可回滚；禁止“全面重构”。 + +### P0.1 网络与文件安全边界 + +- **范围：** Docker loopback、最小管理令牌（如需要 LAN）、统一 `Path` containment、V1 日期/文件 allowlist。 +- **测试：** API auth/unauth、路径 traversal 参数化测试、Windows 路径变体、Docker 端口检查。 +- **回滚：** 单独配置/路由提交；保留 loopback 本地开发模式。 + +### P0.2 数据一致性治理 + +- **范围：** 只读分类 192 条孤儿、备份/恢复演练、映射策略、FK/Unique 迁移设计。 +- **测试：** `integrity_check`、`foreign_key_check`、重复约束、旧库升级、回滚恢复。 +- **回滚：** 原库只读保留；迁移在副本验证后替换。 + +### P1.1 损坏状态与结果未知恢复 + +- **范围：** run/scheduler JSON schema、CORRUPT 隔离、manifest-last、人工恢复。 +- **测试：** 截断 JSON、空文件、半写、重启、邮件/微信已提交但无完成标记。 +- **回滚：** 新读取器兼容旧格式；不批量改历史文件。 + +### P1.2 单一调度所有权与退出码 + +- **范围：** FastAPI scheduler 与 Windows task 角色、稳定终态/退出码、安装事务。 +- **测试：** 双实例抢锁、already_running、partial、blocked、半安装、系统重启。 +- **回滚：** 保留另一入口为禁用的 watchdog，不同时启用。 + +### P1.3 配置契约与 Provider 真实性 + +- **范围：** `schedule_rule/model/provider_preference` 真正消费或 UI 禁用；生产 Mock fail closed。 +- **测试：** 保存/回读/执行/run.json 审计；真实失败不得落 Mock。 +- **回滚：** 每字段 feature gate；默认保持现行为直到测试通过。 + +### P1.4 AI 与邮件幂等 + +- **范围：** 单层 retry policy、错误分类、请求/邮件稳定 ID、发送 ledger、unknown hold。 +- **测试：** 429、500、连接前断、提交后断、重复进程、SMTP 已接收无响应。 +- **回滚：** 幂等表/文件独立；关闭新策略可回到现有 fail behavior。 + +### P1.5 V1 冻结与退役计划 + +- **范围：** 调用者清单、只读历史适配、新写入只走 V2、逐路由退役。 +- **测试：** 历史页面、导出、V1/V2 同日对照、旧脚本使用统计。 +- **回滚：** 每条路由独立 feature flag；不删除历史表/工件。 + +### P2.1 测试隔离与关键行为网 + +- **范围：** 每测试临时 DB/engine、并发 barrier、前端核心行为、少量 E2E。 +- **测试：** 单文件、随机顺序、重复运行、coverage + 非 coverage 对照。 +- **回滚：** 测试提交独立于生产重构。 + +### P2.2 Pipeline 阶段拆分 + +- **范围：** 先提取纯状态转移/结果分类，再分离生成、图片、发送协调；保留现有 Facade。 +- **测试：** 以 P2.1 characterization 为门；原 API 与 run.json 不变。 +- **回滚：** Facade 可切回旧内部实现。 + +### P2.3 API/前端热点拆分 + +- **范围：** `v2_ui.py` 按领域拆 router；`AIImages` 提取数据 hook/命令 hook；共享状态展示 helper；CSS 按页面渐进拆。 +- **测试：** API contract、组件行为、截图/可访问性。 +- **回滚：** 路由路径和页面对外契约不变。 + +### P2.4 依赖、可观测性与性能小步改进 + +- **范围：** Python lock/constraints、Vite 安全升级、liveness/readiness、Provider retention、批量查询/分页/退避。 +- **测试：** clean install、Docker build、bundle、API 查询数、健康检查无副作用。 +- **回滚：** 每一项独立提交，不与 React major 或核心重构捆绑。 + +--- + +## 12. 最终判断 + +这个项目现在能跑，不是纯靠运气。它已经有一套相当有价值的“外部副作用安全骨架”：发送认领、结果未知 hold、Codex staging、图片归属验证、MCP 边界、群级失败隔离。这些部分是 V1 最可靠、最不应该为了工程美学重写的资产。 + +项目目前最大的风险也不是“代码不够高级”，而是三个现实问题： + +1. **数据关系和状态真相不唯一。** SQLite 与 run.json、V1 与 V2、工件与状态之间仍可能分叉。 +2. **部署安全边界依赖环境约定。** Windows loopback 安全，但仓库提供的 Docker 路径会扩大暴露面。 +3. **测试和监控还不能完全证明无人值守。** 一个完整测试失败、单独运行又暴露顺序依赖；前端和真实外部边界没有自动化网。 + +最安全的提升顺序不是拆大文件，而是： + +```text +先封网络/文件边界 +→ 再保护和修复数据真相 +→ 再让损坏/未知状态 fail closed +→ 再统一调度、退出码、配置与幂等 +→ 最后用测试网逐步收敛 V1/V2 和拆分 God Module +``` + +在完成 P0/P1 之前，不建议进行全面架构重写、React major 升级、CSS 全量重构或为了 Sonar A 评级批量改规则问题。 + +## 13. 尚待用户决策 + +1. 正式部署是否永远只允许本机访问，还是未来需要 LAN/远程访问？这决定认证边界。 +2. V1 API/数据库历史是否仍有实际用户或外部脚本依赖？这决定退役速度。 +3. 192 条孤儿 `group_runs` 是保留审计、映射回群、匿名归档还是删除候选？必须由业务决定。 +4. 群级模型/Provider 字段是计划真实启用，还是应从 V1 UI 暂时隐藏？ +5. FastAPI APScheduler 与 Windows Task Scheduler 哪一个是唯一生产 owner？ +6. 外部邮件/微信需要怎样的不可变送达证据：Message-ID、日志、截图还是人工确认？ + +审计到此停止。等待确认整改路线图后，再进入任何代码或数据修改。 diff --git a/README.md b/README.md index fb4cb3b..ee0f21c 100644 --- a/README.md +++ b/README.md @@ -111,14 +111,14 @@ start_windows.bat 5. 在“记录与归档”中确认消息与运行文件已经保存。 6. 只有完成当前微信版本和桌面环境的实机测试后，才开启该群的微信发送。 -### Docker（可选） +### Docker（仅开发与只读） ```powershell Copy-Item .env.example .env docker compose up -d --build ``` -Docker 只运行 GroupBrief 服务本体。微信桌面客户端、WeChatDataAnalysis、Codex CLI 和微信 UI 自动化仍在 Windows 宿主机上。完整说明见 [`docs/DOCKER.md`](docs/DOCKER.md)。 +正式支持环境为 Windows 10/11 + Alter + 本机微信，且 FastAPI Scheduler 是唯一调度 owner。Docker 仅保留开发、读取与界面验证能力，不承诺原生微信发送；微信桌面客户端、WeChatDataAnalysis 和 Codex CLI 仍在 Windows 宿主机上。完整说明见 [`docs/DOCKER.md`](docs/DOCKER.md)。 ## 配置 diff --git a/app/ai/conversation_segments.py b/app/ai/conversation_segments.py index 4710288..ade8725 100644 --- a/app/ai/conversation_segments.py +++ b/app/ai/conversation_segments.py @@ -21,6 +21,9 @@ class PromptMessage: sender_name: str text: str sender_id: str = "" + resolved_sender_name: str = "" + upstream_sender_name: str = "" + attribution_name_source: str = "" @dataclass(frozen=True) diff --git a/app/ai/image_themes.py b/app/ai/image_themes.py index 200f7f7..a5038c6 100644 --- a/app/ai/image_themes.py +++ b/app/ai/image_themes.py @@ -4,21 +4,48 @@ import hashlib import random +import re from dataclasses import dataclass from datetime import date -from typing import Any +from typing import Any, Literal class ImageThemeError(ValueError): """生图主题配置不合法。""" +ThemeKind = Literal["mode", "preset"] + + @dataclass(frozen=True) class ImageThemeDefinition: key: str label: str description: str prompt: str + kind: ThemeKind = "preset" + category: str = "" + swatches: tuple[str, ...] = () + variation_count: int = 1 + + +@dataclass(frozen=True) +class StyleFamilyDefinition: + """一个可公开选择、每天在家族内细微变化的美术风格。""" + + key: str + label: str + category: str + description: str + swatches: tuple[str, str, str] + media: tuple[str, str] + palette: tuple[str, str] + texture: tuple[str, str] + light: tuple[str, str] + + @property + def variation_count(self) -> int: + return len(self.media) * len(self.palette) * len(self.texture) * len(self.light) @dataclass(frozen=True) @@ -34,13 +61,25 @@ class ResolvedImageTheme: style_seed: str = "" catalog_version: str = "" + @property + def has_explicit_style(self) -> bool: + """只有手动预设、每日随机或自定义主题才注入具体风格约束。""" + return self.requested_key != AI_FREE_THEME + + @property + def visible_text(self) -> str: + """写入 Prompt 的风格文本；AI 自由发挥只保留一条中性说明。""" + if not self.has_explicit_style: + return self.prompt + return f"{self.display_name}：{self.prompt}" + def to_meta(self) -> dict[str, str]: return { "requested_theme": self.requested_key, "resolved_theme": self.actual_key, "theme_display_name": self.display_name, "theme_prompt": self.prompt, - "theme_text": f"{self.display_name}：{self.prompt}", + "theme_text": self.visible_text, "theme_custom": self.custom_text, "style_signature": self.style_signature, "style_seed": self.style_seed, @@ -48,11 +87,13 @@ def to_meta(self) -> dict[str, str]: } -DEFAULT_IMAGE_THEME = "random_preset" +DEFAULT_IMAGE_THEME = "ai_free" RANDOM_PRESET_THEME = "random_preset" AI_FREE_THEME = "ai_free" CUSTOM_THEME = "custom" -STYLE_CATALOG_VERSION = "daily-style-v2" +STYLE_CATALOG_VERSION = "daily-style-v3" +STYLE_VARIATIONS_PER_FAMILY = 16 +STYLE_SAFETY_SUFFIX = "只控制美术语言和视觉质感，不得新增、删除或改写聊天事实、人物、数字和指定文字。" # 旧主题继续接受和解析，供历史配置与已保存 Prompt 使用；主界面不再展示。 CONCRETE_THEME_KEYS: tuple[str, ...] = ( @@ -78,25 +119,166 @@ def to_meta(self) -> dict[str, str]: } IMAGE_THEME_MODE_DEFINITIONS: dict[str, ImageThemeDefinition] = { - RANDOM_PRESET_THEME: ImageThemeDefinition(RANDOM_PRESET_THEME, "每日随机", "每群每天独立生成一套可复现风格", "按日期与群聊稳定生成兼容的画材、配色、纹理和光影组合；页面几何由漫画分镜单独决定。"), - AI_FREE_THEME: ImageThemeDefinition(AI_FREE_THEME, "AI 自由发挥（兼容）", "历史配置兼容模式", "根据当天真实聊天选择一个统一视觉主题；不得新增或改变聊天事实。"), - CUSTOM_THEME: ImageThemeDefinition(CUSTOM_THEME, "指定风格", "输入 1～80 字的群专属风格词", "严格使用用户指定的视觉主题，完全替代每日随机；只能影响视觉表现。"), + RANDOM_PRESET_THEME: ImageThemeDefinition( + RANDOM_PRESET_THEME, "每日随机", "每群每天从完整风格库生成一套可复现风格", + "按日期与群聊稳定生成兼容的画材、配色、纹理和光影组合；页面几何由漫画分镜单独决定。", + kind="mode", category="模式", variation_count=352, + ), + AI_FREE_THEME: ImageThemeDefinition( + AI_FREE_THEME, "AI 自由发挥", "默认不指定画材、配色、纹理或光影", + "根据当天真实聊天内容自由选择统一视觉风格。", + kind="mode", category="模式", + ), + CUSTOM_THEME: ImageThemeDefinition( + CUSTOM_THEME, "指定风格", "输入 1～80 字的群专属风格词", + "严格使用用户指定的视觉主题，完全替代每日随机；只能影响视觉表现。", + kind="mode", category="模式", + ), } -IMAGE_THEME_KEYS = frozenset((*CONCRETE_THEME_KEYS, *IMAGE_THEME_MODE_DEFINITIONS)) - -# 每个家族内部的选择经过人工配伍，不会把所有词库做无约束笛卡尔积。 -_STYLE_FAMILIES: tuple[dict[str, Any], ...] = ( - {"label": "丝网印刷社论漫画", "media": ("颗粒丝网印刷", "双色孔版印刷"), "palette": ("群青、奶油白与珊瑚红", "墨绿、米白与亮橙"), "texture": ("纸张颗粒和套色轻微错位", "半调网点与撕纸边缘"), "light": ("平面高对比光影", "清晰块面明暗")}, - {"label": "纸艺立体插画", "media": ("分层剪纸", "立体纸雕"), "palette": ("天蓝、暖黄与番茄红", "鼠尾草绿、杏色与靛蓝"), "texture": ("纤维纸纹与圆润切边", "折纸阴影与手工拼贴边缘"), "light": ("柔和棚拍侧光", "浅景深纸艺投影")}, - {"label": "水彩旅行手账", "media": ("透明水彩线稿", "水彩与彩铅混合"), "palette": ("湖蓝、浅赭与豆沙红", "薄荷绿、柠檬黄与灰紫"), "texture": ("水渍晕染和纸胶带", "彩铅笔触与手写箭头"), "light": ("通透自然光", "柔和晨光氛围")}, - {"label": "复古未来主义", "media": ("复古科幻杂志插画", "几何矢量科幻海报"), "palette": ("深靛蓝、铜橙与象牙白", "紫黑、青绿与暖金"), "texture": ("老印刷网点与金属刻度", "扫描线和细密星尘"), "light": ("边缘霓虹与局部辉光", "戏剧化逆光")}, - {"label": "黏土定格剧场", "media": ("手工黏土定格动画", "软陶微缩场景"), "palette": ("奶油黄、天空蓝与草莓红", "陶土橙、青绿与乳白"), "texture": ("可见手捏纹理与圆角道具", "软陶颗粒和纸板布景"), "light": ("温暖摄影棚柔光", "微缩场景侧逆光")}, - {"label": "木刻新闻画", "media": ("现代木刻版画", "粗线条凸版印刷"), "palette": ("黑、米白与朱红", "藏青、牛皮纸色与橘黄"), "texture": ("刀刻线纹和粗粝纸面", "油墨不均与印章点缀"), "light": ("高反差明暗", "版画式硬边阴影")}, - {"label": "玻璃拟态科技刊", "media": ("半透明玻璃拟态", "清洁三维插画"), "palette": ("冰蓝、白色与荧光青", "深灰蓝、薰衣草紫与亮粉"), "texture": ("磨砂玻璃和细线纹理", "透明折射与柔和渐变"), "light": ("冷色体积光", "柔亮边缘光")}, - {"label": "儿童科普绘本", "media": ("蜡笔与彩铅绘本", "不透明水粉童书插画"), "palette": ("明黄、草绿与天空蓝", "珊瑚红、奶油白与海军蓝"), "texture": ("蜡笔颗粒和手绘图标", "水粉笔触与圆润贴纸"), "light": ("明快均匀光线", "温暖午后光")}, - {"label": "建筑蓝图漫画", "media": ("工程蓝图线稿", "等距轴测技术插画"), "palette": ("蓝底白线与安全橙", "石墨灰、亮蓝与荧光黄"), "texture": ("网格纸、尺寸线和编号章", "铅笔辅助线与半透明线稿"), "light": ("理性均匀照明", "细微高光强化结构")}, - {"label": "织物刺绣拼布", "media": ("刺绣与布艺拼贴", "羊毛毡立体插画"), "palette": ("靛蓝、米白与砖红", "橄榄绿、芥末黄与莓果紫"), "texture": ("织物纹理、针脚和毛毡边缘", "绒线轮廓与布贴层次"), "light": ("柔和室内散射光", "温暖侧光突出纤维")}, + +STYLE_FAMILIES: tuple[StyleFamilyDefinition, ...] = ( + StyleFamilyDefinition( + "silkscreen_editorial", "丝网印刷社论漫画", "印刷与编辑", "粗线、套色与纸张颗粒形成醒目的社论漫画感。", + ("#21409A", "#F6E8C9", "#F25F5C"), + ("颗粒丝网印刷", "双色孔版印刷"), ("群青、奶油白与珊瑚红", "墨绿、米白与亮橙"), + ("纸张颗粒和轻微套色错位", "半调网点与撕纸边缘"), ("平面高对比光影", "清晰块面明暗"), + ), + StyleFamilyDefinition( + "paper_cut_layered", "分层纸艺插画", "立体与手作", "纤维纸、手工切边与柔和投影带来轻巧层次。", + ("#63B3ED", "#F6C453", "#E34D3B"), + ("分层剪纸", "立体纸雕"), ("天蓝、暖黄与番茄红", "鼠尾草绿、杏色与靛蓝"), + ("纤维纸纹与圆润切边", "折纸阴影与手工拼贴边缘"), ("柔和棚拍侧光", "浅景深纸艺投影"), + ), + StyleFamilyDefinition( + "watercolor_journal", "水彩旅行手账", "绘画与纸本", "透明水彩、彩铅和自然纸纹营造轻松手账感。", + ("#4FA3B7", "#B8D8BA", "#C97B84"), + ("透明水彩细墨线", "水彩与彩铅混合"), ("湖蓝、浅赭与豆沙红", "薄荷绿、柠檬黄与灰紫"), + ("水渍晕染与纸胶带质感", "彩铅笔触与棉纸纹理"), ("通透自然光", "柔和晨光氛围"), + ), + StyleFamilyDefinition( + "retro_futurism", "复古未来主义", "科技与结构", "旧科幻印刷语言与克制辉光结合的未来想象。", + ("#1E2A5E", "#C56E33", "#F2E9D8"), + ("复古科幻杂志插画", "几何矢量科幻海报"), ("深靛蓝、铜橙与象牙白", "紫黑、青绿与暖金"), + ("老印刷网点与金属刻度", "扫描线与细密星尘"), ("边缘霓虹与局部辉光", "戏剧化逆光"), + ), + StyleFamilyDefinition( + "clay_stopmotion", "黏土定格剧场", "立体与手作", "手捏质感与微缩摄影让群聊像一场定格短片。", + ("#F2C94C", "#5DADE2", "#E96B6B"), + ("手工黏土定格动画", "软陶微缩场景"), ("奶油黄、天空蓝与草莓红", "陶土橙、青绿与乳白"), + ("可见手捏纹理与圆角道具", "软陶颗粒与纸板布景"), ("温暖摄影棚柔光", "微缩场景侧逆光"), + ), + StyleFamilyDefinition( + "woodcut_editorial", "木刻新闻画", "印刷与编辑", "粗线刀刻、套印墨色与强反差形成有力叙事。", + ("#171717", "#F3E6C8", "#B52A2A"), + ("现代木刻版画", "粗线凸版印刷"), ("黑、米白与朱红", "藏青、牛皮纸色与橘黄"), + ("刀刻线纹与粗粝纸面", "油墨不均与印章色点"), ("高反差明暗", "版画式硬边阴影"), + ), + StyleFamilyDefinition( + "glassmorphism_tech", "玻璃拟态科技刊", "科技与结构", "透明折射、磨砂表面与冷色辉光呈现清洁科技感。", + ("#25304A", "#67E8F9", "#A78BFA"), + ("半透明玻璃拟态插画", "清洁三维科技插画"), ("冰蓝、白色与荧光青", "深灰蓝、薰衣草紫与亮粉"), + ("磨砂玻璃与细线纹理", "透明折射与柔和渐变"), ("冷色体积光", "柔亮边缘光"), + ), + StyleFamilyDefinition( + "children_science_picturebook", "儿童科普绘本", "绘画与纸本", "蜡笔、水粉和圆润造型带来明快、亲切的科普绘本感。", + ("#F5C542", "#67B76F", "#5AA7E8"), + ("蜡笔与彩铅绘本", "不透明水粉童书插画"), ("明黄、草绿与天空蓝", "珊瑚红、奶油白与海军蓝"), + ("蜡笔颗粒与手绘符号", "水粉笔触与圆润纸贴质感"), ("明快均匀光线", "温暖午后光"), + ), + StyleFamilyDefinition( + "architectural_blueprint", "建筑蓝图漫画", "科技与结构", "工程线稿、轴测绘制与克制高光形成理性视觉语言。", + ("#165DFF", "#F8FAFC", "#FF8A34"), + ("工程蓝图线稿", "等距轴测技术插画"), ("蓝底白线与安全橙", "石墨灰、亮蓝与荧光黄"), + ("工程网格与细线刻痕", "铅笔辅助线与半透明线稿"), ("理性均匀照明", "细微高光强化形体"), + ), + StyleFamilyDefinition( + "textile_embroidery", "织物刺绣拼布", "立体与手作", "针脚、毛毡和布贴层次形成温暖的手作触感。", + ("#344E7A", "#F4ECD8", "#A64B3C"), + ("刺绣与布艺拼贴", "羊毛毡立体插画"), ("靛蓝、米白与砖红", "橄榄绿、芥末黄与莓果紫"), + ("织物纹理、针脚与毛毡边缘", "绒线轮廓与布贴层次"), ("柔和室内散射光", "温暖侧光突出纤维"), + ), + StyleFamilyDefinition( + "ink_wash_editorial", "水墨留白漫画", "传统与复古", "干湿笔、宣纸留白与克制设色呈现代水墨叙事。", + ("#1B1D1F", "#264653", "#C43D2F"), + ("现代水墨细线漫画", "水墨设色干湿笔"), ("墨黑、黛青、朱砂与宣纸白", "松烟黑、石青、赭石与暖白"), + ("宣纸纤维、飞白与墨晕", "枯笔皴擦、水痕与印章色点"), ("雾化散射与局部硬墨", "柔亮留白与深墨对比"), + ), + StyleFamilyDefinition( + "art_deco_night", "装饰艺术夜刊", "传统与复古", "对称几何、金属质感与深色调形成精致夜刊气氛。", + ("#0D3B2E", "#D4AF37", "#F5E6C8"), + ("几何装饰插画", "金属箔复古印刷"), ("深祖母绿、黑金与奶油色", "午夜蓝、酒红与香槟金"), + ("压纹纸与细金线", "天鹅绒颗粒与金箔"), ("琥珀聚光", "高对比金属边缘光"), + ), + StyleFamilyDefinition( + "isometric_miniature", "等距微缩模型", "立体与手作", "俯视微缩物件与短投影构成清爽的立体小世界。", + ("#8EC5FC", "#F9C74F", "#90BE6D"), + ("等距微缩模型", "低多边形立体插画"), ("天空蓝、暖黄与草绿", "鼠尾草绿、陶土橙与乳白"), + ("磨砂模型树脂", "微缩木材与颗粒地形"), ("柔和俯视棚拍", "清晰日光与短投影"), + ), + StyleFamilyDefinition( + "pixel_arcade", "像素街机小志", "动漫与数字", "清晰像素簇、街机色彩与屏幕辉光形成活跃数字质感。", + ("#2B174A", "#00D4FF", "#FF4D8D"), + ("16-bit 像素画", "32-bit 像素插画"), ("紫黑、荧光青与亮粉", "深蓝、青柠绿与亮橙"), + ("抖动网点与像素簇", "扫描线与精细精灵边缘"), ("屏幕霓虹辉光", "街机式高对比光"), + ), + StyleFamilyDefinition( + "cel_animation", "赛璐璐动画", "动漫与数字", "平涂色块、手绘墨线和戏剧轮廓光带来动画张力。", + ("#243B6B", "#F2C14E", "#E85D75"), + ("手绘赛璐璐动画", "复古电视动画"), ("深蓝、明黄与珊瑚红", "青绿、奶油白与绯红"), + ("平涂色块、墨线与胶片尘点", "赛璐璐颜料边缘与轻微套色"), ("戏剧性轮廓光", "暖色夕照补光"), + ), + StyleFamilyDefinition( + "chibi_sticker", "Q版贴纸剧场", "动漫与数字", "圆润人物、白边贴纸和糖果色让群聊轻松可爱。", + ("#F8BBD0", "#B39DDB", "#81D4FA"), + ("光泽 Q 版贴纸", "圆润萌系插画"), ("粉红、薰衣草紫与天空蓝", "柠檬黄、薄荷绿与珊瑚红"), + ("白色贴纸边与高光", "柔软塑料颗粒与纸标签质感"), ("柔和正面光", "糖果色环境光"), + ), + StyleFamilyDefinition( + "pencil_storyboard", "铅笔分镜手稿", "绘画与纸本", "石墨线、纸纹和速写痕迹保留鲜活的创作现场感。", + ("#4A4A4A", "#D9CBB6", "#B76E79"), + ("石墨铅笔手稿", "彩铅速写"), ("石墨灰、米纸色与红铅色", "炭黑、象牙白与靛蓝"), + ("纸纹、橡皮痕与辅助线", "交叉排线与纸胶带质感"), ("桌面散射光", "窗边柔和侧光"), + ), + StyleFamilyDefinition( + "natural_history_engraving", "复古博物图鉴", "传统与复古", "精密排线、旧纸斑点与博物馆光线呈现古典图鉴气质。", + ("#5B4636", "#C9B27C", "#6B7D4E"), + ("铜版蚀刻插画", "钢笔线描图鉴"), ("棕褐、橄榄绿与砖红", "墨黑、旧纸色与深蓝"), + ("铜版排线与旧纸斑点", "细密交叉线与纤维纸"), ("博物馆柔光", "轻微暗角侧光"), + ), + StyleFamilyDefinition( + "minimal_vector", "极简几何插画", "印刷与编辑", "大色块、清晰轮廓与少量错位投影保持利落现代。", + ("#111827", "#F9FAFB", "#FF6B35"), + ("扁平矢量插画", "几何拼贴插画"), ("黑、白与亮橙", "深蓝、奶油白与草绿"), + ("纯色色块与清晰轮廓", "切割几何表面"), ("均匀平面光", "轻微错位投影"), + ), + StyleFamilyDefinition( + "gouache_editorial", "不透明水粉社论", "绘画与纸本", "厚实哑光笔触与纸张颗粒形成温暖的社论插画感。", + ("#D95D39", "#E9C46A", "#2A9D8F"), + ("不透明水粉插画", "水粉与彩铅混合"), ("赭黄、珊瑚红与青绿", "灰紫、芥末黄与海军蓝"), + ("厚实笔触与纸张颗粒", "叠色水粉与干刷边缘"), ("柔和哑光", "温暖散射光"), + ), + StyleFamilyDefinition( + "stained_glass", "彩色玻璃拼画", "立体与手作", "铅条接缝、宝石色玻璃与透射光形成璀璨手作质感。", + ("#2E1A47", "#1F7A8C", "#C99700"), + ("铅条彩色玻璃", "手工玻璃马赛克"), ("紫晶、青绿与金黄", "宝石红、钴蓝与琥珀色"), + ("玻璃气泡与铅条接缝", "不规则马赛克边缘"), ("透射辉光", "高对比彩色折射光"), + ), + StyleFamilyDefinition( + "mineral_pigment", "矿物岩彩插画", "传统与复古", "矿物颗粒、石面底色与细碎金箔呈现沉稳华丽的岩彩效果。", + ("#B33A3A", "#235789", "#C6A15B"), + ("矿物岩彩插画", "石色设色绘画"), ("朱砂、石青、石绿与金色", "赭石、靛蓝、玉绿与象牙白"), + ("矿物颗粒与灰泥底", "石面纹理与细碎金箔"), ("掠射侧光", "柔和宝石反射光"), + ), +) + +STYLE_FAMILY_BY_KEY = {family.key: family for family in STYLE_FAMILIES} +STYLE_FAMILY_KEYS = tuple(family.key for family in STYLE_FAMILIES) +IMAGE_THEME_KEYS = frozenset((*CONCRETE_THEME_KEYS, *IMAGE_THEME_MODE_DEFINITIONS, *STYLE_FAMILY_KEYS)) + +_HEX_COLOR_RE = re.compile(r"^#[0-9A-F]{6}$") +_PERSISTED_STYLE_FORBIDDEN = ( + "版式使用", "卡片", "数据面板", "分栏", "跨格", "路线式阅读", "信息节点", "中心主视觉", "REFERENCE_0", ) @@ -119,7 +301,7 @@ def validate_image_theme_config(theme: Any, custom: Any = "") -> tuple[str, str] custom = "" if not isinstance(custom, str): raise ImageThemeError("自定义主题必须是文本") - custom_text = normalize_custom_theme(custom) if custom.strip() or key == CUSTOM_THEME else "" + custom_text = normalize_custom_theme(custom) if key == CUSTOM_THEME else "" return key, custom_text @@ -130,32 +312,48 @@ def _signature(prompt: str) -> str: return hashlib.sha256(prompt.encode("utf-8")).hexdigest()[:16] -def _daily_style(seed_text: str, previous_signature: str = "") -> ResolvedImageTheme: +def _style_prompt(family: StyleFamilyDefinition, picker: random.Random) -> str: + media = picker.choice(family.media) + palette = picker.choice(family.palette) + texture = picker.choice(family.texture) + light = picker.choice(family.light) + return f"统一采用{media}；配色为{palette}；质感为{texture}；光影为{light}。{STYLE_SAFETY_SUFFIX}" + + +def _daily_style(requested_key: str, group_key: str, run_date: str, previous_signature: str = "") -> ResolvedImageTheme: for attempt in range(20): - material = f"{STYLE_CATALOG_VERSION}|{seed_text}|{attempt}" + material = f"{STYLE_CATALOG_VERSION}|{requested_key}|{group_key}|{run_date}|{attempt}" seed = int(hashlib.sha256(material.encode("utf-8")).hexdigest()[:16], 16) picker = random.Random(seed) - family = picker.choice(_STYLE_FAMILIES) - media = picker.choice(family["media"]) - palette = picker.choice(family["palette"]) - texture = picker.choice(family["texture"]) - light = picker.choice(family["light"]) - prompt = ( - f"统一采用{media}形式；配色为{palette}；加入{texture}；光影为{light}。" - "保持中文清晰；所有风格元素只作为视觉表现，不得新增或改变聊天事实。" - ) + family = picker.choice(STYLE_FAMILIES) if requested_key == RANDOM_PRESET_THEME else STYLE_FAMILY_BY_KEY[requested_key] + prompt = _style_prompt(family, picker) signature = _signature(prompt) if signature != previous_signature or attempt == 19: return ResolvedImageTheme( - RANDOM_PRESET_THEME, - "daily_random", - family["label"], - prompt, - style_signature=signature, - style_seed=f"{seed:016x}", - catalog_version=STYLE_CATALOG_VERSION, + requested_key, family.key, family.label, prompt, + style_signature=signature, style_seed=f"{seed:016x}", catalog_version=STYLE_CATALOG_VERSION, ) - raise AssertionError("每日随机风格解析失败") + raise AssertionError("每日风格解析失败") + + +def _restored_style(key: str, persisted_meta: dict[str, Any] | None) -> ResolvedImageTheme | None: + if not persisted_meta or persisted_meta.get("requested_theme") != key: + return None + version = str(persisted_meta.get("style_catalog_version") or "") + prompt = str(persisted_meta.get("theme_prompt") or "").strip() + if version not in {"daily-style-v2", STYLE_CATALOG_VERSION} or not prompt: + return None + if any(term in prompt for term in _PERSISTED_STYLE_FORBIDDEN): + return None + return ResolvedImageTheme( + key, + str(persisted_meta.get("resolved_theme") or ("daily_random" if key == RANDOM_PRESET_THEME else key)), + str(persisted_meta.get("theme_display_name") or "每日随机"), + prompt, + style_signature=str(persisted_meta.get("style_signature") or _signature(prompt)), + style_seed=str(persisted_meta.get("style_seed") or ""), + catalog_version=version, + ) def resolve_image_theme( @@ -169,29 +367,17 @@ def resolve_image_theme( persisted_meta: dict[str, Any] | None = None, ) -> ResolvedImageTheme: key, custom_text = validate_image_theme_config(theme, custom) - if key == RANDOM_PRESET_THEME: - if ( - persisted_meta - and persisted_meta.get("requested_theme") == RANDOM_PRESET_THEME - and persisted_meta.get("theme_prompt") - and persisted_meta.get("style_catalog_version") == STYLE_CATALOG_VERSION - ): - return ResolvedImageTheme( - RANDOM_PRESET_THEME, - str(persisted_meta.get("resolved_theme") or "daily_random"), - str(persisted_meta.get("theme_display_name") or "每日随机"), - str(persisted_meta["theme_prompt"]), - style_signature=str(persisted_meta.get("style_signature") or ""), - style_seed=str(persisted_meta.get("style_seed") or ""), - catalog_version=str(persisted_meta.get("style_catalog_version") or STYLE_CATALOG_VERSION), - ) + if key == RANDOM_PRESET_THEME or key in STYLE_FAMILY_BY_KEY: + restored = _restored_style(key, persisted_meta) + if restored is not None: + return restored # 兼容旧测试/调用方显式注入的 choice 选择器。 - if rng is not None: + if key == RANDOM_PRESET_THEME and rng is not None: actual_key = rng.choice(CONCRETE_THEME_KEYS) definition = IMAGE_THEME_DEFINITIONS[actual_key] return ResolvedImageTheme(key, actual_key, definition.label, definition.prompt, style_signature=_signature(definition.prompt)) effective_date = run_date or date.today().isoformat() - return _daily_style(f"{group_key or 'preview'}|{effective_date}", previous_signature) + return _daily_style(key, group_key or "preview", effective_date, previous_signature) if key == CUSTOM_THEME: prompt = ( f"自定义大主题「{custom_text}」：严格使用该指定风格并完全替代随机风格；" @@ -208,12 +394,46 @@ def resolve_image_theme( resolve_theme = resolve_image_theme -def public_image_theme_options() -> list[dict[str, str]]: - """主界面只展示每日随机与指定风格两种模式。""" - return [ - {"key": definition.key, "label": definition.label, "description": definition.description} - for definition in ( - IMAGE_THEME_MODE_DEFINITIONS[RANDOM_PRESET_THEME], - IMAGE_THEME_MODE_DEFINITIONS[CUSTOM_THEME], +def _public_option(definition: ImageThemeDefinition) -> dict[str, object]: + return { + "key": definition.key, + "label": definition.label, + "description": definition.description, + "kind": definition.kind, + "category": definition.category, + "swatches": list(definition.swatches), + "variation_count": definition.variation_count, + "preview_url": ( + f"/assets/image-theme-previews/{definition.key}.webp" + if definition.kind == "preset" + else "" + ), + } + + +def public_image_theme_options() -> list[dict[str, object]]: + """返回三种选择模式和稳定排序的 22 个公开风格家族。""" + modes = ( + IMAGE_THEME_MODE_DEFINITIONS[AI_FREE_THEME], + IMAGE_THEME_MODE_DEFINITIONS[RANDOM_PRESET_THEME], + IMAGE_THEME_MODE_DEFINITIONS[CUSTOM_THEME], + ) + presets = ( + ImageThemeDefinition( + family.key, family.label, family.description, "", kind="preset", category=family.category, + swatches=family.swatches, variation_count=family.variation_count, ) - ] + for family in STYLE_FAMILIES + ) + return [_public_option(definition) for definition in (*modes, *presets)] + + +def validate_style_catalog() -> None: + """启动测试可调用的目录自检；生产解析不在请求热路径重复运行。""" + if len(STYLE_FAMILIES) != 22 or len(STYLE_FAMILY_BY_KEY) != len(STYLE_FAMILIES): + raise ImageThemeError("公开风格家族必须恰好为 22 个且键唯一") + for family in STYLE_FAMILIES: + if family.variation_count != STYLE_VARIATIONS_PER_FAMILY: + raise ImageThemeError(f"风格 {family.key} 的变化数量不是 16") + if len(family.swatches) != 3 or any(not _HEX_COLOR_RE.fullmatch(color) for color in family.swatches): + raise ImageThemeError(f"风格 {family.key} 的色板不合法") diff --git a/app/ai/poster_copy.py b/app/ai/poster_copy.py new file mode 100644 index 0000000..4a4a385 --- /dev/null +++ b/app/ai/poster_copy.py @@ -0,0 +1,715 @@ +"""把证据化选题编辑为固定结构的微信群日报漫画 Prompt。""" + +from __future__ import annotations + +from dataclasses import dataclass +import json +import re +from typing import Any, Iterable + +from app.ai.layouts import LayoutPlan, SHOT_LABELS + + +POSTER_COPY_VERSION = "fixed-chat-comic-v2" +MAX_VISIBLE_PARTICIPANTS = 4 +MAX_VISIBLE_QUOTE_CHARS = 48 +_GENERIC_COPY = ("信息量拉满", "一天顶一周", "比过山车还刺激") +_FORBIDDEN_RENDERED_TERMS = ( + "topic-", + "message_id", + "selected_topics", + "storyboard_plan", + "participant_options", + "evidence_dialogue", + "speaker_bindings", +) +_ELLIPSIS_END_RE = re.compile(r"(?:\.{3}|…)$") +_SENTENCE_SEGMENT_RE = re.compile(r"[^。！？!?]+[。！？!?]?") +_CLAUSE_SEGMENT_RE = re.compile(r"[^，,；;：:]+[，,；;：:]?") +_NUMBER_RE = re.compile(r"\d+(?:\.\d+)?") +_GROUNDING_STOPWORDS = { + "真实", "群友", "话题", "当天", "讨论", "聊天", "漫画", "事件", "画面", + "继续", "开始", "一起", "现场", "今天", "日报", "群聊", +} +_CONFLICTING_RULES = ( + "不得绘制群名称", + "不显示群名称", + "不设置副标题", + "不显示副标题", + "不设置底部总结", + "不显示底部总结", + "每题只允许一个姓名", + "每个话题只允许一个姓名", + "每格只允许一个姓名", +) + + +class PosterCopyError(ValueError): + """最终编辑稿不符合真实证据或固定结构。""" + + +@dataclass(frozen=True) +class ParticipantCopy: + message_id: str + sender_id: str + name: str + action: str + quote: str = "" + + +@dataclass(frozen=True) +class PanelCopy: + topic_id: str + title: str + event_summary: str + composition: str + participants: tuple[ParticipantCopy, ...] + visual_gag: str + fact_line: str + + +@dataclass(frozen=True) +class DailyPosterCopy: + title: str + subtitle: str + panels: tuple[PanelCopy, ...] + footer_summary: str + + +def _clean_text( + value: object, + *, + maximum: int, + label: str, + allow_terminal_ellipsis: bool = False, +) -> str: + text = re.sub(r"\s+", " ", str(value or "")).strip() + if not text: + raise PosterCopyError(f"{label}不能为空") + if len(text) > maximum: + raise PosterCopyError(f"{label}超过 {maximum} 个字符") + if _ELLIPSIS_END_RE.search(text) and not allow_terminal_ellipsis: + raise PosterCopyError(f"{label}不能以悬空省略号结尾") + return text + + +def _selected_topics(selection: dict, topic_order: Iterable[str]) -> list[dict[str, Any]]: + selected = [ + item + for item in selection.get("candidates", []) + if isinstance(item, dict) and item.get("selected") + ] + by_id = {str(item.get("topic_id") or ""): item for item in selected} + ordered = [by_id[topic_id] for topic_id in topic_order if topic_id in by_id] + if len(ordered) != len(selected): + raise PosterCopyError("漫画阅读顺序没有覆盖全部入选主题") + return ordered + + +def build_poster_editor_source( + selection: dict, + layout: LayoutPlan, +) -> dict[str, Any]: + """生成仅供总结模型使用的证据编辑包；不会直接进入最终 Prompt。""" + ordered = _selected_topics(selection, layout.topic_order) + beats = {beat.topic_id: beat for beat in layout.panel_beats} + topics: list[dict[str, Any]] = [] + for item in ordered: + topic_id = str(item.get("topic_id") or "") + dialogue = item.get("evidence_dialogue") + if not isinstance(dialogue, list): + dialogue = [] + evidence_dialogue = [ + { + "message_id": str(entry.get("message_id") or ""), + "sender_id": str(entry.get("sender_id") or ""), + "speaker": str(entry.get("speaker") or "").strip(), + "text": str(entry.get("text") or "").strip(), + "original_text": str( + entry.get("original_text") or entry.get("text") or "" + ).strip(), + } + for entry in dialogue + if isinstance(entry, dict) + and str(entry.get("message_id") or "").strip() + and str(entry.get("speaker") or "").strip() + and str(entry.get("text") or "").strip() + ] + # 署名是 message-scoped：同一 sender_id 当天改名时，两条绑定都保留， + # Poster 模型选择哪条 message_id，程序就使用那条消息当时的显示名。 + participants: list[str] = [] + for entry in evidence_dialogue: + name = entry["speaker"] + if name not in participants: + participants.append(name) + beat = beats.get(topic_id) + topics.append( + { + "topic_id": topic_id, + "source_title": str(item.get("title") or "").strip(), + "source_summary": str(item.get("summary") or "").strip(), + "source_visual_gag": str(item.get("visual_gag") or "").strip(), + "participant_options": participants, + "evidence_dialogue": evidence_dialogue, + "speaker_bindings": evidence_dialogue, + "shot_hints": [ + SHOT_LABELS.get(shot, shot) + for shot in (beat.shots if beat is not None else ()) + ], + } + ) + return { + "copy_version": POSTER_COPY_VERSION, + "layout_hint": { + "name": layout.layout_name, + "structure_mode": layout.structure_mode, + "comedy_device": layout.comedy_device, + "reason": layout.layout_reason, + }, + "topics": topics, + } + + +POSTER_EDITOR_SYSTEM = """你是「群报 GroupBrief」的漫画日报内容编辑。 +你只能根据给定的 evidence package 写一个 JSON 对象，不得输出 Markdown 或解释。 +所有事实、姓名和对白必须来自对应 topic；不得改变金额、时间、地点和人物关系。 +每个 panel 必须对应一个 topic_id，顺序不得改变。优先使用 2～4 位 participant_options； +每位人物都要从 speaker_bindings 选择一个真实 message_id，并写可绘制的动作、站位或反应。 +不得输出或改写人物姓名；程序会从 message_id 绑定中填入姓名。quote 只能逐字复制同一 +message_id 的完整原消息或其中连续、语义完整的片段，不得跨消息拼接，不得改写， +不得自行添加悬空省略号截断。 +如果完整原消息本身以省略号结尾，只能逐字引用整条原消息，不得截取其中一部分。 +有两名以上真实发言人时，优先为至少两人各选一条能形成接话关系的真实气泡。 +event_summary 与 fact_line 必须是完整句，只能概括 source_summary 和 evidence_dialogue。 +视觉笑点只能字面化或放大已有内容，不得伪装成群里真实发生的新事件。 +主标题、副标题和底部总结必须回收入选主题，不使用通用套话。 +主标题不超过 24 个字符，副标题不超过 36 个字符，底部总结不超过 42 个字符； +每个版面标题不超过 18 个字符，事实说明不超过 72 个字符。写数字前必须确认该数字 +原样存在于对应 topic 的 source_summary 或 evidence_dialogue 中。每个 quote 最多 48 个字符； +原消息更长时，只能选择其中一个连续、语义完整且不带悬空省略号的短句，不得复制整段长消息。 + +严格 JSON 结构： +{ + "title": "当天主标题", + "subtitle": "当天副标题", + "panels": [ + { + "topic_id": "原 topic_id", + "title": "自然话题标题", + "event_summary": "一至两句完整背景", + "composition": "格子大小、景别、人物相对站位", + "participants": [ + {"message_id": "speaker_bindings 中的真实消息ID", "action": "动作、站位或反应", "quote": "该消息的可选真实原话"} + ], + "visual_gag": "不改变事实的视觉笑点", + "fact_line": "完整简短事实说明" + } + ], + "footer_summary": "当天底部总结" +}""" + + +def build_poster_editor_prompt(source: dict[str, Any]) -> str: + # 完整原文只用于程序侧 message_id 校验与 provenance 落盘。编辑模型只需 + # 看到受控短片段，避免超长消息扩大 Prompt 或诱导它重写无关内容。 + model_source = dict(source) + model_topics: list[dict[str, Any]] = [] + for topic in source.get("topics") or []: + if not isinstance(topic, dict): + continue + model_topic = dict(topic) + for field in ("evidence_dialogue", "speaker_bindings"): + model_topic[field] = [ + {key: value for key, value in entry.items() if key != "original_text"} + for entry in topic.get(field) or [] + if isinstance(entry, dict) + ] + model_topics.append(model_topic) + model_source["topics"] = model_topics + return ( + "请把以下已证据校验的群聊主题内化成固定群聊漫画编辑稿。" + "只返回 JSON；内部字段稍后由程序移除，不会交给图片模型。\n\n" + + json.dumps(model_source, ensure_ascii=False, separators=(",", ":")) + ) + + +def _strip_json_fence(raw: str) -> str: + text = (raw or "").strip() + if text.startswith("```"): + text = re.sub(r"^```(?:json)?\s*", "", text, flags=re.IGNORECASE) + text = re.sub(r"\s*```$", "", text) + return text.strip() + + +def _bindings_by_message_id(topic: dict[str, Any]) -> dict[str, dict[str, str]]: + result: dict[str, dict[str, str]] = {} + for entry in topic.get("speaker_bindings") or []: + if not isinstance(entry, dict): + continue + message_id = str(entry.get("message_id") or "").strip() + sender_id = str(entry.get("sender_id") or "").strip() + speaker = str(entry.get("speaker") or "").strip() + text = re.sub( + r"\s+", + " ", + str(entry.get("original_text") or entry.get("text") or ""), + ).strip() + if not message_id or not speaker or not text or message_id in result: + continue + result[message_id] = { + "message_id": message_id, + "sender_id": sender_id, + "speaker": speaker, + "text": text, + } + return result + + +def _quote_is_contiguous(quote: str, messages: Iterable[str]) -> bool: + candidate = re.sub(r"\s+", " ", quote).strip().strip("“”\"") + if not candidate: + return False + return any(candidate in message for message in messages) + + +def _quote_is_full_message(quote: str, messages: Iterable[str]) -> bool: + candidate = re.sub(r"\s+", " ", quote).strip().strip("“”\"") + if not candidate: + return False + return any( + candidate == re.sub(r"\s+", " ", str(message)).strip().strip("“”\"") + for message in messages + ) + + +def _shorten_grounded_quote(quote: str, messages: Iterable[str]) -> str: + """把已证实来自原消息的长气泡缩成一个真实、连续、完整的短句。""" + normalized_messages = [re.sub(r"\s+", " ", str(message)).strip() for message in messages] + if not _quote_is_contiguous(quote, normalized_messages): + return "" + + def candidates(pattern: re.Pattern[str], text: str) -> list[str]: + result: list[str] = [] + for match in pattern.finditer(text): + candidate = match.group(0).strip().strip("“”\"") + candidate = candidate.rstrip("，,；;：:").strip() + if 2 <= len(candidate) <= MAX_VISIBLE_QUOTE_CHARS and not _ELLIPSIS_END_RE.search(candidate): + if _quote_is_contiguous(candidate, normalized_messages): + result.append(candidate) + return result + + sentence_candidates = candidates(_SENTENCE_SEGMENT_RE, quote) + if sentence_candidates: + return max(sentence_candidates, key=len) + clause_candidates = candidates(_CLAUSE_SEGMENT_RE, quote) + if clause_candidates: + return max(clause_candidates, key=len) + return "" + + +def _assert_numbers_grounded(text: str, topic: dict[str, Any], label: str) -> None: + evidence = " ".join( + ( + str(topic.get("source_summary") or ""), + *(str(item.get("text") or "") for item in topic.get("evidence_dialogue") or [] if isinstance(item, dict)), + ) + ) + allowed = set(_NUMBER_RE.findall(evidence)) + invented = [number for number in _NUMBER_RE.findall(text) if number not in allowed] + if invented: + raise PosterCopyError(f"{label}包含证据中不存在的数字：{invented[0]}") + + +def _grounding_tokens(text: str) -> set[str]: + tokens: set[str] = set() + for run in re.findall(r"[\u3400-\u9fff]{2,}", text): + tokens.update(run[index : index + 2] for index in range(len(run) - 1)) + tokens.update(re.findall(r"[A-Za-z][A-Za-z0-9!~._-]+|\d+(?:\.\d+)?", text)) + return {token for token in tokens if token not in _GROUNDING_STOPWORDS} + + +def _assert_text_grounded(text: str, evidence: str, label: str) -> None: + tokens = _grounding_tokens(text) + if not tokens or not any(token in evidence for token in tokens): + raise PosterCopyError(f"{label}没有回收到对应真实话题") + + +def parse_poster_copy(raw: str, source: dict[str, Any]) -> DailyPosterCopy: + try: + payload = json.loads(_strip_json_fence(raw)) + except json.JSONDecodeError as exc: + raise PosterCopyError(f"漫画编辑稿不是有效 JSON：{exc.msg}") from exc + if not isinstance(payload, dict): + raise PosterCopyError("漫画编辑稿必须是 JSON 对象") + + title = _clean_text(payload.get("title"), maximum=24, label="主标题") + subtitle = _clean_text(payload.get("subtitle"), maximum=36, label="副标题") + footer = _clean_text(payload.get("footer_summary"), maximum=42, label="底部总结") + if any(phrase in footer for phrase in _GENERIC_COPY): + raise PosterCopyError("底部总结使用了禁用的通用套话") + + raw_panels = payload.get("panels") + topics = source.get("topics") + if not isinstance(raw_panels, list) or not isinstance(topics, list): + raise PosterCopyError("漫画编辑稿缺少 panels") + if len(raw_panels) != len(topics): + raise PosterCopyError("版面数量与入选话题数量不一致") + + panels: list[PanelCopy] = [] + for index, (raw_panel, topic) in enumerate(zip(raw_panels, topics), start=1): + if not isinstance(raw_panel, dict) or not isinstance(topic, dict): + raise PosterCopyError(f"版面{index}格式无效") + topic_id = str(raw_panel.get("topic_id") or "").strip() + expected_topic_id = str(topic.get("topic_id") or "").strip() + if topic_id != expected_topic_id: + raise PosterCopyError(f"版面{index}未按既定话题顺序输出") + + panel_title = _clean_text(raw_panel.get("title"), maximum=18, label=f"版面{index}标题") + event_summary = _clean_text(raw_panel.get("event_summary"), maximum=120, label=f"版面{index}背景") + composition = _clean_text(raw_panel.get("composition"), maximum=160, label=f"版面{index}构图") + visual_gag = _clean_text(raw_panel.get("visual_gag"), maximum=180, label=f"版面{index}视觉笑点") + fact_line = _clean_text(raw_panel.get("fact_line"), maximum=72, label=f"版面{index}事实说明") + _assert_numbers_grounded(event_summary, topic, f"版面{index}背景") + _assert_numbers_grounded(fact_line, topic, f"版面{index}事实说明") + topic_evidence = " ".join( + ( + str(topic.get("source_title") or ""), + str(topic.get("source_summary") or ""), + " ".join(str(value) for value in topic.get("participant_options") or []), + " ".join( + str(entry.get("text") or "") + for entry in topic.get("evidence_dialogue") or [] + if isinstance(entry, dict) + ), + ) + ) + _assert_text_grounded(panel_title, topic_evidence, f"版面{index}标题") + _assert_text_grounded(event_summary, topic_evidence, f"版面{index}背景") + _assert_text_grounded(fact_line, topic_evidence, f"版面{index}事实说明") + + participant_options = [str(value).strip() for value in topic.get("participant_options") or [] if str(value).strip()] + bindings = _bindings_by_message_id(topic) + available_identities = { + ("id", binding["sender_id"].casefold()) + if binding["sender_id"] + else ("name", binding["speaker"]) + for binding in bindings.values() + } + raw_participants = raw_panel.get("participants") + if not isinstance(raw_participants, list): + raise PosterCopyError(f"版面{index}缺少 participants") + minimum = 2 if len(available_identities) >= 2 else 1 + maximum = min(MAX_VISIBLE_PARTICIPANTS, len(available_identities)) + if not bindings or not (minimum <= len(raw_participants) <= maximum): + raise PosterCopyError(f"版面{index}应使用 {minimum}～{maximum} 位真实参与者") + + participants: list[ParticipantCopy] = [] + identities: set[tuple[str, str]] = set() + quoted_speakers: set[tuple[str, str]] = set() + for person_index, raw_person in enumerate(raw_participants, start=1): + if not isinstance(raw_person, dict): + raise PosterCopyError(f"版面{index}第{person_index}位人物格式无效") + if str(raw_person.get("name") or "").strip(): + raise PosterCopyError(f"版面{index}第{person_index}位人物不得由模型提供姓名") + message_id = str(raw_person.get("message_id") or "").strip() + binding = bindings.get(message_id) + if binding is None: + raise PosterCopyError(f"版面{index}包含未授权的消息ID：{message_id or '空'}") + name = binding["speaker"] + sender_id = binding["sender_id"] + identity = ("id", sender_id.casefold()) if sender_id else ("name", name) + if name not in participant_options or identity in identities: + raise PosterCopyError(f"版面{index}包含未授权或重复的群友身份：{name}") + identities.add(identity) + action = _clean_text(raw_person.get("action"), maximum=120, label=f"版面{index}人物动作") + quote = re.sub(r"\s+", " ", str(raw_person.get("quote") or "")).strip().strip("“”\"") + if quote: + source_messages = [binding["text"]] + if not _quote_is_contiguous(quote, source_messages): + raise PosterCopyError(f"版面{index}中“{name}”的气泡无法回查绑定消息") + if len(quote) > MAX_VISIBLE_QUOTE_CHARS: + quote = _shorten_grounded_quote(quote, source_messages) + if not quote: + raise PosterCopyError( + f"版面{index}真实气泡超过 {MAX_VISIBLE_QUOTE_CHARS} 个字符且无法安全缩短" + ) + quote = _clean_text( + quote, + maximum=MAX_VISIBLE_QUOTE_CHARS, + label=f"版面{index}真实气泡", + allow_terminal_ellipsis=_quote_is_full_message( + quote, + source_messages, + ), + ) + quoted_speakers.add(identity) + participants.append( + ParticipantCopy( + message_id=message_id, + sender_id=sender_id, + name=name, + action=action, + quote=quote, + ) + ) + + required_quotes = 2 if len(available_identities) >= 2 and len(participants) >= 2 else 1 + if len(quoted_speakers) < required_quotes: + raise PosterCopyError(f"版面{index}缺少足够的真实多人对白") + + panels.append( + PanelCopy( + topic_id=topic_id, + title=panel_title, + event_summary=event_summary, + composition=composition, + participants=tuple(participants), + visual_gag=visual_gag, + fact_line=fact_line, + ) + ) + all_evidence = " ".join( + " ".join( + ( + str(topic.get("source_title") or ""), + str(topic.get("source_summary") or ""), + " ".join(str(value) for value in topic.get("participant_options") or []), + " ".join( + str(entry.get("text") or "") + for entry in topic.get("evidence_dialogue") or [] + if isinstance(entry, dict) + ), + ) + ) + for topic in topics + if isinstance(topic, dict) + ) + _assert_text_grounded(title, all_evidence, "主标题") + _assert_text_grounded(subtitle, all_evidence, "副标题") + _assert_text_grounded(footer, all_evidence, "底部总结") + return DailyPosterCopy(title=title, subtitle=subtitle, panels=tuple(panels), footer_summary=footer) + + +def _overall_visual(style_text: str, *, explicit_style: bool) -> str: + style_line = ( + f"本次手动视觉风格：{style_text}" + if explicit_style + else "根据当天真实聊天内容自由选择统一视觉风格。" + ) + return "\n\n".join( + ( + "生成一张适合微信手机端阅读的竖版漫画群报，优先采用 1024×1536、2:3 画布；其他完整可读的竖版尺寸也可以采用，不要为了匹配尺寸裁切或拉伸。", + style_line, + "整张图像一页热闹的群聊漫画：顶部是群名称、完整统计时间、主标题和副标题，中间由多个大小错落的话题漫画格组成，底部展示当天总结和统计数据。", + "每个话题都要画成一个真实的“群友讨论现场”，而不是单人物插画。每个话题优先选择 2～4 位真正参与该段聊天的群友出镜，人物旁边直接标注对应的真实群昵称。", + "不同群友使用不同动作、表情和站位，以真实聊天气泡、人物反应、道具、动作线和视觉笑点表现讨论过程。所有剧情、人物关系和聊天内容均来自当天真实群聊，不额外编造新的聊天事实。", + ) + ) + + +def _render_panel(index: int, panel: PanelCopy) -> str: + paragraphs = [f"【版面{index}】", panel.title, panel.event_summary, panel.composition] + for participant in panel.participants: + if participant.quote: + paragraphs.append( + f"{participant.name}{participant.action}。人物旁清晰标注“{participant.name}”；" + f"与此人物不可拆分的真实聊天气泡写“{participant.quote}”。" + "姓名牌、人物和该气泡必须处于同一局部区域，不得与其他人物交换。" + ) + else: + paragraphs.append( + f"{participant.name}{participant.action}。人物旁清晰标注“{participant.name}”。" + ) + paragraphs.extend( + ( + panel.visual_gag, + "版面中自然加入一句简短事实说明：\n\n" + panel.fact_line, + ) + ) + return "\n\n".join(paragraphs) + + +TEXT_RULES = """图片中允许出现： + +群名称、完整统计时间、主标题、副标题、统计数据、每个话题标题、真实群友昵称、精选真实聊天气泡、简短事实说明和底部总结。 + +群友昵称应贴近对应人物显示，让读者能够直接识别每个人是谁。同一个话题可以显示多位真实群友姓名；优先展示真正参与该段聊天的人，不为了凑人数添加无关群友。 + +海报顶部的群名称、完整统计时间、主标题和副标题，以及底部总结、消息数和发言人数均不可省略、改写或缩小至不可读。 + +不要把“任务”“版面”“画面”“人物”“参与者”“事实说明”“文字规则”“主气泡”“代表人物”“可用文字”等 Prompt 结构性字段画进图片。 + +不要额外生成程序字段、序号、英文装饰词、Logo、网址、无关品牌或与聊天无关的文字。真实聊天中必要的产品名可作为普通文字保留，但不绘制品牌 Logo。 + +空间不足时先减少装饰、道具和次要反应，不删除顶部、底部、真实姓名和主要对白。""" + + +def render_poster_prompt( + copy: DailyPosterCopy, + *, + group_name: str, + period_line: str, + message_line: str, + speaker_line: str, + style_text: str, + explicit_style: bool, + template_text: str = "", +) -> str: + panels = "\n\n".join( + _render_panel(index, panel) for index, panel in enumerate(copy.panels, start=1) + ) + overall_visual = _overall_visual(style_text, explicit_style=explicit_style) + if template_text: + from app.ai.prompt_templates import render_image_prompt_template + + period_start, separator, period_end = period_line.partition(" ~ ") + if not separator: + raise PosterCopyError("完整统计时间格式无效") + message_count = message_line.removesuffix(" 条消息") + speaker_count = speaker_line.removesuffix(" 人发言") + prompt = render_image_prompt_template( + template_text, + { + "group_name": group_name, + "period_start": period_start, + "period_end": period_end, + "message_count": message_count, + "speaker_count": speaker_count, + "main_title": copy.title, + "subtitle": copy.subtitle, + "overall_visual": overall_visual, + "panels": panels, + "text_rules": TEXT_RULES, + "footer_summary": copy.footer_summary, + }, + ).strip() + else: + parts = [ + "【任务】\n\n生成一张竖版微信群日报漫画信息图。", + f"【群名称】\n\n{group_name}", + f"【统计时间】\n\n{period_line}", + f"【数据】\n\n{message_line}\n{speaker_line}", + f"【主标题】\n\n{copy.title}", + f"【副标题】\n\n{copy.subtitle}", + "【整体视觉】\n\n" + overall_visual, + panels, + "【文字规则】\n\n" + TEXT_RULES, + f"【底部总结】\n\n{copy.footer_summary}", + ] + prompt = "\n\n".join(parts).strip() + validate_rendered_poster_prompt( + prompt, + copy=copy, + group_name=group_name, + period_line=period_line, + message_line=message_line, + speaker_line=speaker_line, + ) + return prompt + + +def _section_headings(prompt: str) -> list[str]: + return re.findall(r"(?m)^【([^\n】]+)】\s*$", prompt) + + +def validate_fixed_prompt_contract( + prompt: str, + *, + expected_panel_count: int | None = None, +) -> int: + """校验人工编辑后的固定合同；不依赖内部编辑 JSON。""" + headings = _section_headings(prompt) + panel_headings = [heading for heading in headings if re.fullmatch(r"版面\d+", heading)] + panel_count = len(panel_headings) + if not 2 <= panel_count <= 7: + raise PosterCopyError("最终 Prompt 必须包含连续的 2～7 个版面") + if expected_panel_count is not None and panel_count != expected_panel_count: + raise PosterCopyError("版面数量与本次已校验入选话题数量不一致") + expected = [ + "任务", + "群名称", + "统计时间", + "数据", + "主标题", + "副标题", + "整体视觉", + "漫画分镜", + *(f"版面{index}" for index in range(1, panel_count + 1)), + "文字规则", + "底部总结", + ] + if headings != expected: + raise PosterCopyError("最终 Prompt 区块名称、顺序或版面编号不符合固定合同") + + sections = re.split(r"(?m)^【[^\n】]+】\s*$", prompt)[1:] + if len(sections) != len(headings): + raise PosterCopyError("最终 Prompt 区块解析失败") + for heading, body in zip(headings, sections): + if not body.strip(): + raise PosterCopyError(f"【{heading}】缺少真实内容") + + if not re.search( + r"\d{4}-\d{2}-\d{2}\s+\d{2}:\d{2}:\d{2}\s*~\s*" + r"\d{4}-\d{2}-\d{2}\s+\d{2}:\d{2}:\d{2}", + prompt, + ): + raise PosterCopyError("最终 Prompt 缺少完整统计开始和结束时间") + if not re.search(r"\d+\s*条消息", prompt) or not re.search(r"\d+\s*人发言", prompt): + raise PosterCopyError("最终 Prompt 缺少消息数或发言人数") + for phrase in _CONFLICTING_RULES: + if phrase in prompt: + raise PosterCopyError(f"最终 Prompt 包含冲突规则：{phrase}") + for forbidden in _FORBIDDEN_RENDERED_TERMS: + if forbidden in prompt: + raise PosterCopyError(f"最终 Prompt 暴露内部字段：{forbidden}") + return panel_count + + +def validate_rendered_poster_prompt( + prompt: str, + *, + copy: DailyPosterCopy, + group_name: str, + period_line: str, + message_line: str, + speaker_line: str, +) -> None: + validate_fixed_prompt_contract(prompt, expected_panel_count=len(copy.panels)) + expected = [ + "任务", + "群名称", + "统计时间", + "数据", + "主标题", + "副标题", + "整体视觉", + "漫画分镜", + *(f"版面{index}" for index in range(1, len(copy.panels) + 1)), + "文字规则", + "底部总结", + ] + headings = _section_headings(prompt) + if headings != expected: + raise PosterCopyError("最终 Prompt 区块名称、顺序或版面编号不符合固定合同") + for literal, label in ( + (group_name, "群名称"), + (period_line, "完整统计时间"), + (message_line, "消息数"), + (speaker_line, "发言人数"), + (copy.title, "主标题"), + (copy.subtitle, "副标题"), + (copy.footer_summary, "底部总结"), + ): + if literal not in prompt: + raise PosterCopyError(f"最终 Prompt 缺少{label}") + for forbidden in _FORBIDDEN_RENDERED_TERMS: + if forbidden in prompt: + raise PosterCopyError(f"最终 Prompt 暴露内部字段：{forbidden}") + for panel in copy.panels: + for participant in panel.participants: + if participant.name not in prompt: + raise PosterCopyError(f"最终 Prompt 缺少群友姓名：{participant.name}") + if participant.quote and participant.quote not in prompt: + raise PosterCopyError(f"最终 Prompt 缺少真实气泡：{participant.quote}") + if panel.fact_line not in prompt: + raise PosterCopyError(f"最终 Prompt 缺少事实说明：{panel.fact_line}") diff --git a/app/ai/prompt_builder.py b/app/ai/prompt_builder.py index 6a8f655..af55bca 100644 --- a/app/ai/prompt_builder.py +++ b/app/ai/prompt_builder.py @@ -16,16 +16,30 @@ import logging import re +from copy import deepcopy from time import perf_counter from datetime import datetime from app.ai.prompt_templates import ( ImagePromptTemplateError, ImagePromptTemplateService, - render_image_prompt_template, validate_image_prompt_template, ) +from app.ai.poster_copy import ( + POSTER_COPY_VERSION, + POSTER_EDITOR_SYSTEM, + PosterCopyError, + build_poster_editor_prompt, + build_poster_editor_source, + parse_poster_copy, + render_poster_prompt, +) from app.ai.prompt_builder_types import PromptInput, PromptOutput +from app.ai.prompt_safety import enforce_prompt_budget, sanitize_prompt_text +from app.ai.speaker_attribution import ( + AttributionName, + build_attribution_contract, +) from app.ai.image_themes import ImageThemeError, resolve_image_theme from app.ai.layouts import ( IMAGE_LAYOUT_DEFINITIONS, @@ -36,10 +50,8 @@ detect_explicit_style_layout, fallback_layout_plan, fixed_layout_plan, - layout_plan_json, parse_layout_plan, preferred_layout_from_style, - resolved_layout_instruction, restored_layout_plan, selected_topic_ids, ) @@ -69,6 +81,7 @@ selected_topics_json, ) from app.config.settings import Settings, get_settings +from app.providers.ai.base import ExternalCallResultUnknownError from app.providers.ai.codex import build_summary_provider logger = logging.getLogger("groupbrief.ai") @@ -77,7 +90,7 @@ TOPIC_CANDIDATE_MAX_TOKENS = 8_000 EVENT_CARD_MAX_TOKENS = 6_000 LAYOUT_DIRECTOR_MAX_TOKENS = 1_800 -FINAL_PROMPT_MAX_ATTEMPTS = 2 +FINAL_PROMPT_MAX_ATTEMPTS = 3 _STRUCTURED_RETRY_INSTRUCTION = """\ 上一次响应不完整、被截断或不符合约定 JSON。请重新阅读原材料，并重新输出一个完整、紧凑、可解析的 JSON 对象。 @@ -88,51 +101,7 @@ 必须使用合法 layout_id 和 structure_mode；featured_topic_ids 数量必须匹配结构模式， topic_order 与 panel_beats 必须恰好覆盖全部入选主题；至少一个话题使用两个镜头，镜头总数必须合法。""" -_FINAL_PROMPT_RETRY_INSTRUCTION = """\ - -上一次最终 Prompt 暴露了内部字段名、主题 ID，或退化成等大模块列表。请完整重写： -只用自然短标题、事实旁白、人物姓名和逐字气泡；保留全部已选话题、指定画风、统计日期与漫画分镜； -不要输出任何数据字段式栏目名，不要输出 topic ID，不要把一个话题机械装进一个等大的矩形区域。""" - -SYSTEM_BASE = """你是「群报 GroupBrief」的漫画日报海报 Prompt 设计师。 -你的唯一任务：根据给定的微信群聊内容，生成一份可以直接复制给 GPT 图片生成能力的完整中文 Prompt， -用于绘制「竖版微信群日报漫画信息图」。 - -硬性要求（必须严格遵守）： -1. 只能使用聊天内容中真实存在的事件、人物、对话，禁止编造任何聊天中不存在的事件。 -2. 不得凭空补充金额、时间、地点、身份关系。 -3. 原话引用必须来自真实聊天，可适当缩写，但不能改写事实。 -4. 事实真实性是准入门槛；通过真实性校验后，好玩程度、群内识别度和视觉笑点是第一优化目标。 - 可以使用字面化、反差、回环、误会与反转、一本正经地荒诞，但不能改变事实。 -5. 海报人物只能采用程序从对应消息回查得到的真实姓名，不得只画匿名人物或自由生成人名。 -6. 数据（消息数、发言人数）必须使用给定数字，禁止自行计算。 -7. 必须严格按给定的【输出结构】组织最终 Prompt；给定的漫画分镜骨架控制整张图的大小格与阅读节奏。 -8. 候选主题已经过证据校验和程序评分；最终只能使用给定的 2～7 个入选主题，并且每个恰好使用一次。 -9. 【大主题】是全图最高视觉约束，控制配色、画材、服装、造型、装饰、纹理、光影和画风； - 漫画分镜只控制格子几何、阅读路径和镜头节拍，不得替换或削弱【大主题】。 -10. 一个话题不等于一个矩形模块；5～7 个话题可以展开为 7～12 个镜头，至少一个话题使用连续镜头。 -11. 每段内容用自然短标题、一句事实旁白、真实人物姓名和至少一句逐字气泡呈现； - 禁止输出内部字段名、topic ID、表格栏目或说明性标签。 -12. 格子必须有明显的大、中、小三级尺寸差，并按计划使用嵌套特写、连续动作或跨格主体； - 禁止整齐两列等高矩形和“每个话题一块”的列表式构图。 -13. 必须把给定的“统计日期：YYYY-MM-DD”作为清晰可见的画面文字，放在海报顶部或底部，不得省略或改写。""" - -CHUNK_ANALYZE_SYSTEM = """你是群聊事件分析助手。只提取聊天中真实存在的事件/人物/原话， -输出严格 JSON（不输出其他内容），没有事件就返回空数组。""" - -CHUNK_ANALYZE_PROMPT = """以下是微信群聊记录片段（{label}）。 - -请分析并输出 JSON： -{{ - "events": [ - {{"title": "事件短标题", "people": ["提到的人名"], "content": "事件描述（真实基于聊天）", "quotes": ["1-3条逐字真实原话"]}} - ] -}} - -要求：只提取真实存在的内容；没有事件就返回空数组；每个片段最多提取 10 个事件，最终候选最多 10 个。""" - _HTML_COMMENT_RE = re.compile(r"", re.DOTALL) -_FORBIDDEN_FINAL_PROMPT_TERMS = ("参与群友", "事实信息", "真实原话", "信息卡", "topic-") def _strip_html_comments(text: str) -> str: @@ -140,6 +109,86 @@ def _strip_html_comments(text: str) -> str: return _HTML_COMMENT_RE.sub("", text).strip() +def _validated_persisted_selection( + selection: object, + messages: list[PromptMessage], + persisted_meta: object, + *, + message_snapshot_sha256: str, + speaker_fingerprint: str, +) -> dict: + """验证已落盘选题仍完整且能回查快照；不重新调用模型选题。""" + if not isinstance(selection, dict) or not isinstance(selection.get("candidates"), list): + raise ValueError("已保存的选题总结缺少 candidates，已停止重建") + result = deepcopy(selection) + selected = [item for item in result["candidates"] if isinstance(item, dict) and item.get("selected")] + selected_ids = [str(item.get("topic_id") or "").strip() for item in selected] + stored_ids = result.get("selected_topic_ids") + if not isinstance(stored_ids, list) or selected_ids != [str(value) for value in stored_ids]: + raise ValueError("已保存的选题 ID 与入选标记不一致，已停止重建") + if not selected_ids or len(selected_ids) != len(set(selected_ids)): + raise ValueError("已保存的选题 ID 为空或重复，已停止重建") + if not isinstance(persisted_meta, dict): + raise ValueError("已保存的 Prompt 元数据缺少消息快照指纹，已停止重建") + stored_snapshot = str(persisted_meta.get("message_snapshot_sha256") or "") + stored_speakers = str(persisted_meta.get("speaker_fingerprint") or "") + if not stored_snapshot or not stored_speakers: + raise ValueError("已保存的 Prompt 元数据缺少消息快照指纹，已停止重建") + if stored_snapshot != message_snapshot_sha256 or stored_speakers != speaker_fingerprint: + raise ValueError("messages.json 已变化，旧选题证据已过期，必须重新选题") + if ( + str(result.get("message_snapshot_sha256") or "") != message_snapshot_sha256 + or str(result.get("speaker_fingerprint") or "") != speaker_fingerprint + ): + raise ValueError("已保存的选题缺少匹配的消息快照指纹，必须重新选题") + + evidence_by_id: dict[str, PromptMessage] = {} + for message in messages: + if not message.message_id: + continue + if message.message_id in evidence_by_id: + raise ValueError("messages.json 包含重复 message_id，已停止重建") + evidence_by_id[message.message_id] = message + allowed_message_ids = set(evidence_by_id) + for item in selected: + if not str(item.get("title") or "").strip() or not str(item.get("summary") or "").strip(): + raise ValueError("已保存的入选主题缺少标题或总结，已停止重建") + evidence_ids = item.get("message_ids") if isinstance(item.get("message_ids"), list) else [] + if not evidence_ids or any(str(message_id) not in allowed_message_ids for message_id in evidence_ids): + raise ValueError("已保存的入选主题无法从 messages.json 回查，已停止重建") + quotes = item.get("quotes") if isinstance(item.get("quotes"), list) else [] + if not any(str(value).strip() for value in quotes): + raise ValueError("已保存的入选主题缺少真实原话，已停止重建") + visible_people = item.get("visible_participants") if isinstance(item.get("visible_participants"), list) else [] + if not any(str(value).strip() for value in visible_people) and not str(item.get("participant_label") or "").strip(): + raise ValueError("已保存的入选主题缺少可见人物，已停止重建") + dialogue = item.get("evidence_dialogue") + if not isinstance(dialogue, list) or not dialogue: + raise ValueError("已保存的入选主题缺少结构化说话人证据，已停止重建") + for entry in dialogue: + if not isinstance(entry, dict): + raise ValueError("已保存的说话人证据格式无效，已停止重建") + message_id = str(entry.get("message_id") or "") + current = evidence_by_id.get(message_id) + if current is None: + raise ValueError("已保存的说话人证据无法从 messages.json 回查，已停止重建") + if str(entry.get("sender_id") or "") != current.sender_id: + raise ValueError("已保存的说话人身份与 messages.json 不一致，必须重新选题") + if str(entry.get("speaker") or "").strip() != current.sender_name: + raise ValueError("已保存的说话人姓名与 messages.json 不一致，必须重新选题") + stored_text = re.sub(r"\s+", " ", str(entry.get("text") or "")).strip() + current_text = re.sub(r"\s+", " ", current.text or "").strip() + if not stored_text or stored_text not in current_text: + raise ValueError("已保存的说话人原文与 messages.json 不一致，必须重新选题") + stored_original = re.sub( + r"\s+", " ", str(entry.get("original_text") or "") + ).strip() + if stored_original and stored_original != current_text: + raise ValueError("已保存的说话人完整原文与 messages.json 不一致，必须重新选题") + selected_topics_json(result) + return result + + _MEDIA_PREFIX = { "image": "[图片]", "emoji": "[表情]", @@ -164,52 +213,6 @@ def _to_ai_text(m) -> str: return content -def _compact_visible_text(value: object, maximum: int) -> str: - """把事实或原话压成单行；只截显示文本，不改变证据与完整姓名。""" - text = re.sub(r"\s+", " ", str(value or "")).strip() - if len(text) <= maximum: - return text - for marker in ("。", "！", "？", "；"): - position = text.rfind(marker, 0, maximum + 1) - if position >= max(12, maximum // 2): - return text[: position + 1] - return text[: maximum - 1].rstrip() + "…" - - -def build_grounded_story_material(selection: dict, topic_order: tuple[str, ...]) -> str: - """把证据整理成自然剧情句；不把内部数据字段交给图片模型绘制。""" - selected = [ - item - for item in selection.get("candidates", []) - if isinstance(item, dict) and item.get("selected") - ] - if not selected: - raise ValueError("没有可生成漫画剧情的入选主题") - by_id = {str(item.get("topic_id") or ""): item for item in selected} - ordered = [by_id[topic_id] for topic_id in topic_order if topic_id in by_id] - if len(ordered) != len(selected): - raise ValueError("漫画阅读顺序没有覆盖全部入选主题") - - lines = [ - f"整页按阅读顺序讲清以下 {len(ordered)} 段真实群聊剧情。序号仅表示阅读次序，不得画进图片：" - ] - for index, item in enumerate(ordered, start=1): - title = _compact_visible_text(item.get("title"), 24) or "群聊话题" - participant_label = str(item.get("participant_label") or "群友（昵称未识别）").strip() - fact = _compact_visible_text(item.get("summary"), 72) or "（仅按该话题的真实消息证据绘制）" - quotes = item.get("quotes") if isinstance(item.get("quotes"), list) else [] - quote = next((_compact_visible_text(value, 48) for value in quotes if str(value or "").strip()), "") - lines.append( - f"{index}. 小标题写《{title}》。画面讲清“{fact}”。让 {participant_label} 出现在对应场景附近，" - f"其中一个气泡逐字写“{quote or fact}”。" - ) - lines.append( - "这些剧情句只用于指导绘画；画面只显示小标题、短旁白、人物姓名和气泡正文，" - "不要显示序号、说明文字、字段名称或程序标识。" - ) - return "\n".join(lines) - - class DeepSeekImagePromptBuilder: """保留历史类名以兼容现有注入点；默认实现已切换为 Codex GPT。""" @@ -220,17 +223,42 @@ def __init__( settings: Settings | None = None, templates: ImagePromptTemplateService | None = None, provider=None, + *, + summary_settings: Settings | None = None, + summary_provider=None, + prompt_provider=None, ): self.settings = settings or get_settings() + self.summary_settings = summary_settings or self.settings self.templates = templates or ImagePromptTemplateService() - # V1/V2 使用同一主备 Provider，不重复实现模型调用。 - self._provider = provider or build_summary_provider(self.settings) + # 兼容旧注入点：显式 provider 仍同时承担分析与海报 Prompt。 + # 生产路径允许两项能力分别选择白名单 Provider/模型。 + if provider is not None: + self._summary_provider = provider + self._prompt_provider = provider + else: + self._summary_provider = summary_provider or build_summary_provider( + self.summary_settings + ) + self._prompt_provider = prompt_provider or build_summary_provider( + self.settings + ) + self._provider = self._summary_provider # ---------- 对外 ---------- def build(self, data: PromptInput) -> PromptOutput: started_at = perf_counter() - api_model = self._provider.model + api_model = self._prompt_provider.model + seen_providers: set[int] = set() + for provider_instance in (self._summary_provider, self._prompt_provider): + if id(provider_instance) in seen_providers: + continue + seen_providers.add(id(provider_instance)) + reset_usage = getattr(provider_instance, "reset_usage", None) + if callable(reset_usage): + reset_usage() + meta: dict | None = None try: theme = resolve_image_theme( data.image_theme, @@ -246,37 +274,81 @@ def build(self, data: PromptInput) -> PromptOutput: report_date = (data.report_date or data.period_end[:10]).strip() if not re.fullmatch(r"\d{4}-\d{2}-\d{2}", report_date): raise ValueError("report_date 必须来自统计周期并使用 YYYY-MM-DD") - theme_text = f"{theme.display_name}：{theme.prompt}" - date_line = f"统计日期：{report_date}" - - messages = [self._to_prompt_message(message, index) for index, message in enumerate(data.messages, start=1)] + theme_text = theme.visible_text + explicit_theme_text = theme_text if theme.has_explicit_style else "" + visible_group_name, _ = sanitize_prompt_text( + data.visible_group_name or data.group_name, + allow_newlines=False, + ) + if not visible_group_name: + raise ValueError("群聊显示名不能为空") + period_line = f"{data.period_start} ~ {data.period_end}" + message_line = f"{data.message_count} 条消息" + speaker_line = f"{data.speaker_count} 人发言" + + attribution = build_attribution_contract(data.messages) + message_snapshot_sha256 = ( + str(data.message_snapshot_sha256 or "").strip() + or attribution.message_snapshot_sha256 + ) + speaker_fingerprint = ( + str(data.speaker_fingerprint or "").strip() + or attribution.speaker_fingerprint + ) + messages = [ + self._to_prompt_message(message, index, attribution.names[index - 1]) + for index, message in enumerate(data.messages, start=1) + ] messages = [message for message in messages if message.text] direct_chars = max(1_000, int(self.settings.max_context_chars or 50_000)) - chunks = segment_messages( - messages, - direct_chars=direct_chars, - target_chars=min(TARGET_CHUNK_CHARS, direct_chars), - hard_chars=max(HARD_CHUNK_CHARS, direct_chars), - session_gap_minutes=SESSION_GAP_MINUTES, - overlap_messages=OVERLAP_MESSAGES, - ) - if not chunks: - raise ValueError("没有可提交给总结模型的聊天文本") - meta: dict = { + chunks: list[ConversationChunk] = [] + if data.persisted_topic_selection is None: + chunks = segment_messages( + messages, + direct_chars=direct_chars, + target_chars=min(TARGET_CHUNK_CHARS, direct_chars), + hard_chars=max(HARD_CHUNK_CHARS, direct_chars), + session_gap_minutes=SESSION_GAP_MINUTES, + overlap_messages=OVERLAP_MESSAGES, + ) + if not chunks: + raise ValueError("没有可提交给总结模型的聊天文本") + elif not messages: + raise ValueError("messages.json 为空，无法回查已保存选题") + meta = { "template": data.template, "template_source": "group_override" if data.template_override else "global", "api_model": api_model, "primary_provider": self.settings.summary_provider_primary, "fallback_provider": self.settings.summary_provider_fallback, + "summary_primary_provider": self.summary_settings.summary_provider_primary, + "summary_fallback_provider": self.summary_settings.summary_provider_fallback, + "prompt_primary_provider": self.settings.summary_provider_primary, + "prompt_fallback_provider": self.settings.summary_provider_fallback, "message_lines": len(messages), "context_chars": sum(len(message.text) for message in messages), "chunk_count": len(chunks), "generated_at": datetime.now().isoformat(), "report_date": report_date, + "message_snapshot_sha256": message_snapshot_sha256, + "speaker_fingerprint": speaker_fingerprint, } meta.update(theme.to_meta()) + meta["style_intervention"] = theme.has_explicit_style - if len(chunks) <= 1: + if data.persisted_topic_selection is not None: + selection = _validated_persisted_selection( + data.persisted_topic_selection, + messages, + data.persisted_theme_meta, + message_snapshot_sha256=message_snapshot_sha256, + speaker_fingerprint=speaker_fingerprint, + ) + meta["mode"] = "persisted_topic_selection" + meta["topic_selection_reused"] = True + meta["reuse_source"] = "run.prompt_meta" + analysis_calls = 0 + elif len(chunks) <= 1: meta["mode"] = "direct" candidates, candidate_calls = self._topic_candidates_with_retry( TOPIC_CANDIDATE_SYSTEM, @@ -290,8 +362,11 @@ def build(self, data: PromptInput) -> PromptOutput: def analyze(item: tuple[int, ConversationChunk]) -> tuple[list[dict], int]: idx, chunk = item + event_system = EVENT_ANALYZE_SYSTEM + if explicit_theme_text: + event_system += "\n\n" + self._theme_constraint(explicit_theme_text) return self._event_cards_with_retry( - EVENT_ANALYZE_SYSTEM + "\n\n" + self._theme_constraint(f"{theme.display_name}：{theme.prompt}"), + event_system, build_event_prompt(chunk, f"第 {idx}/{len(chunks)} 块"), chunk, ) @@ -314,7 +389,11 @@ def analyze(item: tuple[int, ConversationChunk]) -> tuple[list[dict], int]: ) analysis_calls = event_calls + candidate_calls - selection = score_and_select_topics(candidates, messages) + if data.persisted_topic_selection is None: + selection = score_and_select_topics(candidates, messages) + meta["topic_selection_reused"] = False + selection["message_snapshot_sha256"] = message_snapshot_sha256 + selection["speaker_fingerprint"] = speaker_fingerprint meta["topic_selection_version"] = selection["topic_selection_version"] meta["topic_selection"] = selection selected_payload = selected_topics_json(selection) @@ -324,7 +403,16 @@ def analyze(item: tuple[int, ConversationChunk]) -> tuple[list[dict], int]: style_layout_locked = detect_explicit_style_layout(custom_style_text) preferred_layout = preferred_layout_from_style(custom_style_text) - if preferred_layout: + if data.persisted_topic_selection is not None: + layout = restored_layout_plan( + data.persisted_theme_meta, + topic_ids, + style_layout_locked=style_layout_locked, + ) + if layout is None: + raise ValueError("已保存的漫画分镜无法覆盖全部入选主题，已停止重建") + layout_calls = 0 + elif preferred_layout: layout = fixed_layout_plan( preferred_layout, topic_ids, @@ -343,14 +431,12 @@ def analyze(item: tuple[int, ConversationChunk]) -> tuple[list[dict], int]: layout, layout_calls = self._layout_plan_with_retry( selected_payload, topic_ids, - theme_text=theme_text, + theme_text=explicit_theme_text, recent_history=recent_history, style_layout_locked=style_layout_locked, seed_text=f"{data.group_id or data.group_name}|{data.run_date}", ) - layout_instruction = resolved_layout_instruction(layout, custom_style_text) - story_material = build_grounded_story_material(selection, layout.topic_order) meta.update(layout.to_meta()) meta["recent_layout_ids"] = [ str(item.get("layout_id") or "") @@ -358,111 +444,171 @@ def analyze(item: tuple[int, ConversationChunk]) -> tuple[list[dict], int]: if isinstance(item, dict) and item.get("layout_id") ] - structure = render_image_prompt_template( - template_text, - { - "group_name": data.group_name, - "period_start": data.period_start, - "period_end": data.period_end, - "report_date": report_date, - "message_count": str(data.message_count), - "speaker_count": str(data.speaker_count), - "image_theme": theme_text, - "layout_name": layout.layout_name, - "layout_instruction": layout_instruction, - }, - ) - # 群级模板覆盖也必须服从真实剧情和漫画分镜契约。 - structure = f"{structure}\n\n{story_material}" - if date_line not in structure: - # 兼容没有新增占位符的旧/群级模板，同时保证每个最终 Prompt 都收到日期区块。 - structure = f"【固定画面日期】\n{date_line}\n\n{structure}" - - final_user_prompt = ( - "以下主题已经过原消息证据回查和喜剧优先固定评分。" - "最终 Prompt 只能使用 selected_topics，并严格服从 storyboard_plan 的阅读顺序和逐话题镜头；" - "不得加入未入选候选、临时改选、遗漏或重复主题；JSON 字段名和 topic ID 只供内部对应，" - "绝对不要出现在最终 Prompt 或画面文字中：\n\n" - + selected_payload - + "\n\n【已校验漫画分镜方案】\n" - + layout_plan_json(layout) - + "\n\n" - + story_material - ) + editor_source = build_poster_editor_source(selection, layout) + meta["speaker_bindings"] = [ + {"topic_id": str(topic.get("topic_id") or ""), **dict(binding)} + for topic in editor_source.get("topics", []) + if isinstance(topic, dict) + for binding in topic.get("speaker_bindings", []) + if isinstance(binding, dict) + ] + final_user_prompt = build_poster_editor_prompt(editor_source) text = "" final_calls = 0 + last_violations: list[str] = [] for attempt in range(FINAL_PROMPT_MAX_ATTEMPTS): - prompt = final_user_prompt if attempt == 0 else final_user_prompt + _FINAL_PROMPT_RETRY_INSTRUCTION - candidate_text = self._chat( - structure, + prompt = final_user_prompt + if attempt: + prompt += ( + "\n\n上一次漫画编辑 JSON 未通过证据或结构校验。" + "请重新输出完整 JSON，不要解释，不要复用错误内容。" + ) + if last_violations: + prompt += "\n上次具体违反：" + "；".join(last_violations[:8]) + raw_copy = self._prompt_chat( + POSTER_EDITOR_SYSTEM, prompt, - theme_text, - layout_instruction, + response_format="json_object", + temperature=0.35, + max_tokens=6000, ) final_calls += 1 - forbidden = [term for term in _FORBIDDEN_FINAL_PROMPT_TERMS if term in candidate_text] - if forbidden: - logger.warning("最终 Prompt 暴露内部字段或主题 ID（第 %s/%s 次）：%s", attempt + 1, FINAL_PROMPT_MAX_ATTEMPTS, forbidden) + try: + copy = parse_poster_copy(raw_copy, editor_source) + candidate_text = render_poster_prompt( + copy, + group_name=visible_group_name, + period_line=period_line, + message_line=message_line, + speaker_line=speaker_line, + style_text=theme_text, + explicit_style=theme.has_explicit_style, + template_text=template_text, + ) + except PosterCopyError as exc: + last_violations = [str(exc)] + logger.warning( + "漫画编辑稿校验失败（第 %s/%s 次）：%s", + attempt + 1, + FINAL_PROMPT_MAX_ATTEMPTS, + last_violations, + ) continue text = candidate_text + meta["poster_copy_version"] = POSTER_COPY_VERSION + meta["poster_topic_count"] = len(copy.panels) + meta["poster_visible_participant_count"] = sum( + len(panel.participants) for panel in copy.panels + ) break if not text: - raise ValueError("最终生图 Prompt 连续暴露内部字段或主题 ID") - - mandatory_blocks: list[str] = [] - if story_material not in text: - mandatory_blocks.append(story_material) - if theme_text not in text: - mandatory_blocks.append( - "【大主题】\n" - + theme_text - + "\n漫画分镜不得替换或削弱该指定风格。" - ) - if layout.layout_name not in text: - mandatory_blocks.append("【漫画分镜｜整张图只使用一种骨架】\n" + layout_instruction) - if date_line not in text: - mandatory_blocks.append( - "【必须在画面中清晰绘制的固定文字】\n" - + date_line - + "\n该日期标识必须位于海报顶部或底部，不得省略或改写。" - ) - if mandatory_blocks: - text = "\n\n".join((*mandatory_blocks, text)) - if date_line not in text: - raise ValueError("最终生图 Prompt 缺少准确统计日期") - forbidden = [term for term in _FORBIDDEN_FINAL_PROMPT_TERMS if term in text] - if forbidden: - raise ValueError("最终生图 Prompt 仍含内部字段或主题 ID：" + "、".join(forbidden)) + raise ValueError("最终生图 Prompt 未通过固定漫画合同：" + "；".join(last_violations[:8])) meta["api_call_count"] = analysis_calls + layout_calls + final_calls + summary_actual = ( + self._provider_actual( + self._summary_provider, + self.summary_settings, + ) + if analysis_calls + else { + "provider": "", + "model": "", + "providers_used": [], + "fallback_reason": "", + } + ) + prompt_actual = self._provider_actual( + self._prompt_provider, + self.settings, + ) + meta["summary_provider_actual"] = summary_actual["provider"] + meta["summary_model_actual"] = summary_actual["model"] + meta["summary_fallback_reason"] = summary_actual["fallback_reason"] + meta["summary_providers_used"] = summary_actual["providers_used"] + meta["summary_api_call_count"] = analysis_calls + meta["prompt_provider_actual"] = prompt_actual["provider"] + meta["prompt_model_actual"] = prompt_actual["model"] + meta["prompt_fallback_reason"] = prompt_actual["fallback_reason"] + meta["prompt_providers_used"] = prompt_actual["providers_used"] + meta["prompt_api_call_count"] = layout_calls + final_calls + # 旧字段继续表示最终海报 Prompt 能力，供旧 run.json 读取器兼容。 + meta["actual_provider"] = prompt_actual["provider"] + meta["actual_model"] = prompt_actual["model"] + meta["providers_used"] = sorted( + set(summary_actual["providers_used"] + prompt_actual["providers_used"]) + ) + meta["fallback_reason"] = prompt_actual["fallback_reason"] summary_ms = round((perf_counter() - started_at) * 1000) meta["summary_ms"] = summary_ms # 保留旧字段一版，避免历史运行分析与外部读取立即失效。 meta["deepseek_ms"] = summary_ms - return PromptOutput(success=True, prompt=text.strip(), model=api_model, meta=meta) + text, prompt_budget_meta = enforce_prompt_budget( + text, + max_chars=self.settings.image_prompt_max_chars, + max_bytes=self.settings.image_prompt_max_bytes, + ) + meta.update(prompt_budget_meta) + return PromptOutput(success=True, prompt=text, model=api_model, meta=meta) except (ImagePromptTemplateError, ImageThemeError, LayoutPlanError, ValueError) as e: logger.warning("Prompt 模板错误：%s", e) - return PromptOutput(success=False, error=str(e)[:300], model=api_model) + return PromptOutput( + success=False, + error=str(e)[:300], + model=api_model, + meta=meta, + ) + except ExternalCallResultUnknownError: + # Pipeline 需要把提交后断线/超时持久化为 result_unknown，不能降格成普通失败。 + raise except Exception as e: # 主备模型调用失败等 logger.exception("ImagePromptBuilder 生成失败") - return PromptOutput(success=False, error=str(e)[:300], model=api_model) + return PromptOutput( + success=False, + error=str(e)[:300], + model=api_model, + meta=meta, + ) # ---------- 内部 ---------- - def _to_prompt_message(self, message, index: int) -> PromptMessage: + def _to_prompt_message( + self, + message, + index: int, + attribution: AttributionName | None = None, + ) -> PromptMessage: timestamp = message.timestamp if hasattr(message.timestamp, "strftime") else None message_id = ( getattr(message, "message_id", "") or getattr(message, "content_hash", "") or f"v2-{index}" ) + if attribution is None: + attribution = build_attribution_contract([message]).names[0] + safe_sender, _ = sanitize_prompt_text( + attribution.display_name or "(未知)", + allow_newlines=False, + ) + resolved_sender, _ = sanitize_prompt_text( + getattr(message, "sender_name", "") or "", + allow_newlines=False, + ) + upstream_sender, _ = sanitize_prompt_text( + getattr(message, "upstream_sender_name", "") or "", + allow_newlines=False, + ) + safe_text, _ = sanitize_prompt_text(_to_ai_text(message)) return PromptMessage( message_id=str(message_id), timestamp=timestamp, - sender_name=message.sender_name or "(未知)", - text=_to_ai_text(message).strip(), + sender_name=safe_sender or "(未知)", + text=safe_text, sender_id=str(getattr(message, "sender_id", "") or ""), + resolved_sender_name=resolved_sender, + upstream_sender_name=upstream_sender, + attribution_name_source=attribution.source, ) def _to_line(self, message) -> str: @@ -486,43 +632,30 @@ def _theme_constraint(theme_prompt: str) -> str: + theme_prompt + "\n大主题控制全图配色、画材、服装、造型、装饰、纹理、光影和画风；" "不得创造、补充或改写聊天事实，也不得被整体版式替换或削弱。" + "重新生图只允许改变所选美术家族及当天已解析的视觉细节，日期、数字、人物、逐字气泡、话题覆盖和既定分镜都是不变量。" ) - @staticmethod - def _layout_constraint(layout_prompt: str) -> str: - return ( - "【漫画分镜约束｜整张图只使用一种骨架】\n" - + layout_prompt - + "\n漫画分镜只控制格子几何、阅读路径和镜头节拍；必须服从大主题。" - ) - - def _chat( + def _analysis_chat( self, - structure: str, + system: str, user_prompt: str, - theme_prompt: str = "", - layout_prompt: str = "", *, - response_format: str = "text", - temperature: float = 0.7, - max_tokens: int = 3000, + response_format: str = "json_object", + temperature: float = 0.1, + max_tokens: int = 4000, ) -> str: - """调用主备总结模型。system 含固定约束 + 模板输出结构。""" - theme_block = "\n\n" + self._theme_constraint(theme_prompt) if theme_prompt else "" - layout_block = "\n\n" + self._layout_constraint(layout_prompt) if layout_prompt else "" - system = SYSTEM_BASE + theme_block + layout_block + "\n\n【输出结构】\n" + structure - messages = [ - {"role": "system", "content": system}, - {"role": "user", "content": user_prompt}, - ] - return self._provider._chat( - messages, + """结构化分析调用不混入最终海报格式约束，避免候选阶段角色冲突。""" + return self._summary_provider._chat( + [ + {"role": "system", "content": system}, + {"role": "user", "content": user_prompt}, + ], response_format=response_format, temperature=temperature, max_tokens=max_tokens, ) - def _analysis_chat( + def _prompt_chat( self, system: str, user_prompt: str, @@ -531,8 +664,8 @@ def _analysis_chat( temperature: float = 0.1, max_tokens: int = 4000, ) -> str: - """结构化分析调用不混入最终海报格式约束，避免候选阶段角色冲突。""" - return self._provider._chat( + """最终版式与海报编辑调用使用群级 Prompt Provider。""" + return self._prompt_provider._chat( [ {"role": "system", "content": system}, {"role": "user", "content": user_prompt}, @@ -542,6 +675,27 @@ def _analysis_chat( max_tokens=max_tokens, ) + @staticmethod + def _provider_actual(provider, settings: Settings) -> dict: + actual_provider = str( + getattr(provider, "last_provider_used", "") + or getattr(provider, "name", "unknown") + ) + actual_model = str( + settings.ai_model + if actual_provider == "deepseek" + else getattr(provider, "model", settings.codex_summary_model) + ) + providers_used = list(getattr(provider, "providers_used", []) or []) + return { + "provider": actual_provider, + "model": actual_model, + "providers_used": providers_used or [actual_provider], + "fallback_reason": str( + getattr(provider, "last_fallback_reason", "") or "" + ), + } + def _layout_plan_with_retry( self, selected_topics_payload: str, @@ -571,7 +725,7 @@ def _layout_plan_with_retry( last_error: LayoutPlanError | None = None for attempt in range(STRUCTURED_ANALYSIS_MAX_ATTEMPTS): prompt = user_prompt if attempt == 0 else user_prompt + _LAYOUT_RETRY_INSTRUCTION - raw = self._analysis_chat( + raw = self._prompt_chat( LAYOUT_DIRECTOR_SYSTEM, prompt, response_format="json_object", diff --git a/app/ai/prompt_builder_types.py b/app/ai/prompt_builder_types.py index 1b5dfde..bb189d1 100644 --- a/app/ai/prompt_builder_types.py +++ b/app/ai/prompt_builder_types.py @@ -23,11 +23,15 @@ class PromptInput: template: str = "default" group_id: str = "" run_date: str = "" - image_theme: str = "random_preset" + message_snapshot_sha256: str = "" + speaker_fingerprint: str = "" + visible_group_name: str = "" + image_theme: str = "ai_free" image_theme_custom: str = "" template_override: str = "" previous_theme_signature: str = "" persisted_theme_meta: dict[str, Any] | None = None + persisted_topic_selection: dict[str, Any] | None = None recent_layout_history: tuple[dict[str, Any], ...] = () diff --git a/app/ai/prompt_editing.py b/app/ai/prompt_editing.py index d8a90ce..a8fb05c 100644 --- a/app/ai/prompt_editing.py +++ b/app/ai/prompt_editing.py @@ -9,11 +9,21 @@ MAX_PROMPT_CHARS = 50_000 _THEME_SECTION_RE = re.compile( - r"(?ms)^【大主题】\s*\n.*?(?=^【[^\n】]+】\s*$|\Z)" + r"(?ms)^【(?:大主题|视觉风格)】\s*\n.*?(?=^【[^\n】]+】\s*$|\Z)" +) +_OVERALL_VISUAL_SECTION_RE = re.compile( + r"(?ms)^【整体视觉】\s*\n(?P.*?)(?=^【[^\n】]+】\s*$|\Z)" +) +_STYLE_LINE_RE = re.compile( + r"(?m)^\s*(?:根据当天真实聊天内容自由选择统一视觉风格。|本次手动视觉风格：[^\n]*)\s*$" ) -def validate_prompt_text(value: object) -> str: +def validate_prompt_text( + value: object, + *, + expected_panel_count: int | None = None, +) -> str: if not isinstance(value, str): raise ValueError("Prompt 必须是文本") text = value.strip() @@ -23,6 +33,9 @@ def validate_prompt_text(value: object) -> str: raise ValueError(f"Prompt 不能超过 {MAX_PROMPT_CHARS} 字") if "\x00" in text: raise ValueError("Prompt 不得包含空字符") + from app.ai.poster_copy import validate_fixed_prompt_contract + + validate_fixed_prompt_contract(text, expected_panel_count=expected_panel_count) return text + "\n" @@ -31,12 +44,28 @@ def prompt_revision(text: str) -> str: def resolved_theme_text(theme: ResolvedImageTheme) -> str: - return f"{theme.display_name}：{theme.prompt}" + return theme.visible_text def replace_theme_section(prompt: str, theme: ResolvedImageTheme) -> str: - """只替换规范主题段；不存在时在开头插入，不改写其他内容。""" - block = f"【大主题】\n{resolved_theme_text(theme)}\n\n" + """新 Prompt 只替换【整体视觉】中的风格句；兼容历史独立主题段。""" + overall = _OVERALL_VISUAL_SECTION_RE.search(prompt) + if overall: + body = _STYLE_LINE_RE.sub("", overall.group("body")).strip() + style_line = ( + f"本次手动视觉风格：{resolved_theme_text(theme)}" + if theme.has_explicit_style + else "根据当天真实聊天内容自由选择统一视觉风格。" + ) + replacement = "【整体视觉】\n" + body + "\n\n" + style_line + "\n\n" + return ( + prompt[: overall.start()] + + replacement + + prompt[overall.end() :].lstrip() + ).strip() + "\n" + + heading = "大主题" if theme.has_explicit_style else "视觉风格" + block = f"【{heading}】\n{resolved_theme_text(theme)}\n\n" if _THEME_SECTION_RE.search(prompt): return _THEME_SECTION_RE.sub(block.rstrip(), prompt, count=1).strip() + "\n" return block + prompt.lstrip() diff --git a/app/ai/prompt_safety.py b/app/ai/prompt_safety.py new file mode 100644 index 0000000..a7c848d --- /dev/null +++ b/app/ai/prompt_safety.py @@ -0,0 +1,96 @@ +"""最终生图 Prompt 的确定性清洗与长度预算。""" + +from __future__ import annotations + +import re +import unicodedata + +_SECTION_RE = re.compile(r"(?m)^(【[^\n】]{1,80}】[^\n]*)$") + + +def sanitize_prompt_text(value: object, *, allow_newlines: bool = True) -> tuple[str, int]: + """移除可能破坏文件/模型边界的控制字符，并统一换行。""" + text = unicodedata.normalize("NFC", str(value or "")) + text = text.replace("\r\n", "\n").replace("\r", "\n") + removed = 0 + output: list[str] = [] + for char in text: + if char == "\n" and allow_newlines: + output.append(char) + continue + if char == "\t": + output.append(" ") + continue + if unicodedata.category(char) in {"Cc", "Cf", "Cs"}: + removed += 1 + continue + output.append(char) + cleaned = "".join(output) + if allow_newlines: + cleaned = re.sub(r"\n{4,}", "\n\n\n", cleaned) + else: + cleaned = re.sub(r"\s+", " ", cleaned) + return cleaned.strip(), removed + + +def _truncate_utf8(text: str, max_bytes: int) -> str: + if len(text.encode("utf-8")) <= max_bytes: + return text + return text.encode("utf-8")[: max(max_bytes, 0)].decode("utf-8", errors="ignore") + + +def _clip_middle(text: str, char_budget: int) -> str: + if len(text) <= char_budget: + return text + marker = "\n…（超长内容已确定性压缩）…\n" + if char_budget <= len(marker) + 20: + return text[:char_budget] + remaining = char_budget - len(marker) + head = max(1, remaining * 2 // 3) + return text[:head] + marker + text[-(remaining - head):] + + +def enforce_prompt_budget( + value: object, + *, + max_chars: int, + max_bytes: int, +) -> tuple[str, dict[str, int | bool]]: + """保留每个 ``【章节】`` 标题及章节首尾，把最终 Prompt 限定在硬预算内。""" + cleaned, removed = sanitize_prompt_text(value) + char_limit = max(1_000, int(max_chars)) + byte_limit = max(4_000, int(max_bytes)) + original_chars = len(cleaned) + original_bytes = len(cleaned.encode("utf-8")) + compacted = original_chars > char_limit or original_bytes > byte_limit + result = cleaned + + if compacted: + matches = list(_SECTION_RE.finditer(cleaned)) + if matches: + preamble = cleaned[: matches[0].start()].strip() + sections: list[tuple[str, str]] = [] + for index, match in enumerate(matches): + end = matches[index + 1].start() if index + 1 < len(matches) else len(cleaned) + sections.append((match.group(1).strip(), cleaned[match.end():end].strip())) + fixed = sum(len(title) + 2 for title, _ in sections) + len(preamble) + content_budget = max(char_limit - fixed - 64, len(sections) * 24) + per_section = max(24, content_budget // max(len(sections), 1)) + parts = [preamble] if preamble else [] + for title, body in sections: + parts.append(title) + if body: + parts.append(_clip_middle(body, per_section)) + result = "\n".join(parts) + result = _clip_middle(result, char_limit) + result = _truncate_utf8(result, byte_limit) + result = result.rstrip() + + return result, { + "prompt_controls_removed": removed, + "prompt_compacted": compacted, + "prompt_original_chars": original_chars, + "prompt_original_bytes": original_bytes, + "prompt_final_chars": len(result), + "prompt_final_bytes": len(result.encode("utf-8")), + } diff --git a/app/ai/prompt_templates.py b/app/ai/prompt_templates.py index b8e7cb1..37788bb 100644 --- a/app/ai/prompt_templates.py +++ b/app/ai/prompt_templates.py @@ -1,9 +1,9 @@ """生图 Prompt 模板服务。 模板以 UTF-8 文本文件存储在 templates/image_prompt/ 下（默认 default.md）。 -与排行榜模板（app/ranking/template_service.py）结构一致，变量不同： -group_name / report_date / period_start / period_end / message_count / speaker_count / -image_theme / layout_name / layout_instruction。 +与排行榜模板（app/ranking/template_service.py）结构一致。最终固定区块由 +main_title / subtitle / overall_visual / panels / text_rules / footer_summary 等 +结构化变量填入；历史变量仅保留解析兼容，不进入默认模板。 """ from __future__ import annotations @@ -13,20 +13,14 @@ from app.config.settings import PROJECT_ROOT -# 默认生图 Prompt 模板（与 templates/image_prompt/default.md 同步；恢复默认时写回此内容） +# 默认生图 Prompt 模板（与 templates/image_prompt/default.md 同步）。动态内容先经 +# 结构化证据校验，再由代码填入，模型不能自行增删区块。 DEFAULT_IMAGE_PROMPT_TEMPLATE = """【任务】 生成一张竖版微信群日报漫画信息图。 -【创作优先级】 -事实真实性是准入门槛；通过真实性校验后，好玩程度、群内识别度和视觉笑点是第一优化目标。 -正常保留 5～7 个独立话题的密度，用漫画镜头表现“谁做了什么、别人怎样接话”，不要画成栏目列表。 - 【群名称】 {{group_name}} -【固定画面日期】 -统计日期：{{report_date}} - 【统计时间】 {{period_start}} ~ {{period_end}} @@ -34,49 +28,31 @@ {{message_count}} 条消息 {{speaker_count}} 人发言 -【大主题】 -{{image_theme}} - -【漫画分镜】 -{{layout_name}} -{{layout_instruction}} - 【主标题】 -（优先使用群聊原句、群内梗、反差或回环；建议不超过 18 个汉字） +{{main_title}} 【副标题】 -（一句话概括当天讨论；建议不超过 26 个汉字） +{{subtitle}} + +【整体视觉】 +{{overall_visual}} -【剧情与镜头分配】 -按给定阅读顺序使用全部入选话题，不得遗漏、重复、增删或改选。 -一个话题不等于一个矩形模块；同一话题可以用连续的环境、动作、对白、反应或特写镜头展开。 -正常 5～7 个话题应形成 7～12 个视觉格，至少一个话题使用两个以上连续镜头。 +【漫画分镜】 +每个话题先作为一个独立漫画框，框内再根据聊天内容自由拆成1～3个连续分镜，表现发言、动作、群友回应和表情反应。 +可自由使用：头条大格＋反应小格、宽格＋小格、纵向长镜头＋侧挂格、大场景＋表情特写、上下连续镜头、两个不等大场景、对角线动作跨格、自由切角拼贴。 +简单话题可以只画一个完整场景，不强制分镜。 +同一话题内可以出现多名真实群友，姓名贴近对应人物，真实聊天气泡放在说话人物附近。内部小镜头不要重复话题标题和姓名。 +分镜只在当前话题框内展开，不跨到其他话题；根据真实聊天强化误会、吐槽、动作和群友反应，不额外编造剧情。 +允许人物动作、道具、速度线或视觉效果在同一个话题框内部轻微跨越小分镜边界，以增加漫画连续感。 +漫画分镜负责强化真实聊天中的动作、误会、吐槽、反差和群友反应 -【画面文字白名单】 -只清晰绘制：主标题、统计日期、给定数据、自然的话题短标题、短事实旁白、真实姓名和精选群聊气泡。 -不得绘制程序字段、主题编号、说明性栏目名或 JSON；空间不足时先减少装饰和副标题，保留事实与气泡。 +{{panels}} -【分镜表现】 -整页至少有大、中、小三级格子尺寸差；使用嵌套反应小格、连续动作、局部特写或一次跨格主体建立节奏。 -气泡尾巴、人物视线和动作线共同引导从上到下、从左到右阅读；禁止整齐两列等高矩形和重复模板块。 +【文字规则】 +{{text_rules}} 【底部总结】 -可用一句短文案回收当天讨论；不使用“信息量拉满”“一天顶一周”“比过山车还刺激”等通用套话。 - -【硬性要求】 -1. 只使用聊天内容中真实存在的事件、人物、对话，禁止编造。 -2. 不得凭空补充金额、时间、地点、身份关系。 -3. 气泡文字必须来自程序给定的真实聊天，可缩短长度，但不能改写事实。 -4. 可以使用字面化、反差、回环、误会与反转、一本正经地荒诞，但不能改变事实。 -5. 海报人物依据聊天事件中的真实人员，而不是发言排行榜 Top10；姓名只能使用程序回查得到的人员。 -6. 数据（消息数、发言人数）必须使用给定数字，禁止自行计算。 -7. 【大主题】是全图最高视觉约束，控制配色、画材、造型、装饰、纹理、光影和画风；【漫画分镜】不得替换或削弱它。 -8. 【漫画分镜】只控制格子几何、阅读路径和镜头节拍；每张图只能使用给定的一种骨架。 -9. 不得把法庭、菜单、地图、新闻台等无关主题包装强加给真实聊天。 -10. 最终 Prompt 必须严格包含给定的 2～7 个入选主题且各使用一次；证据不足时由上游减少数量，不得编造。 -11. 每个入选话题至少显示一个真实姓名、一句事实短句和一句给定气泡，不得用泛化头像替代人物。 -12. 漫画主体与对话必须和对应聊天事实直接相关，视觉比喻只能放大已有笑点，不能另写故事。 -13. 必须把“统计日期：{{report_date}}”逐字作为清晰可见的画面文字，放在海报顶部或底部。 +{{footer_summary}} """ # 生图 Prompt 模板支持的变量 @@ -91,6 +67,12 @@ "image_theme", "layout_name", "layout_instruction", + "main_title", + "subtitle", + "overall_visual", + "panels", + "text_rules", + "footer_summary", } ) @@ -98,7 +80,40 @@ _RENDER_DEFAULTS = { "layout_name": "（生成时自动选择漫画分镜骨架）", "layout_instruction": "（生成时根据入选主题和最近分镜历史写入大小格与镜头节拍）", + "main_title": "（生成时填入当天真实主标题）", + "subtitle": "（生成时填入当天真实副标题）", + "overall_visual": "（生成时填入固定整体视觉与当前风格）", + "panels": "【版面1】\n（生成时按真实入选话题填入）", + "text_rules": "（生成时填入固定文字规则）", + "footer_summary": "（生成时填入当天真实底部总结）", } +_FIXED_TEMPLATE_HEADINGS = ( + "任务", + "群名称", + "统计时间", + "数据", + "主标题", + "副标题", + "整体视觉", + "漫画分镜", + "文字规则", + "底部总结", +) +_FIXED_TEMPLATE_VARS = frozenset( + { + "group_name", + "period_start", + "period_end", + "message_count", + "speaker_count", + "main_title", + "subtitle", + "overall_visual", + "panels", + "text_rules", + "footer_summary", + } +) class ImagePromptTemplateError(ValueError): @@ -150,14 +165,26 @@ def reset(self, name: str = "default") -> str: def validate_image_prompt_template(text: str) -> None: - """校验模板：所有 {{var}} 占位符必须属于受支持变量。""" + """校验模板变量以及固定群聊漫画区块合同。""" + used_vars: set[str] = set() for m in re.finditer(r"\{\{\s*(\w+)\s*\}\}", text): var = m.group(1) + used_vars.add(var) if var not in IMAGE_PROMPT_VARS: raise ImagePromptTemplateError( f"模板包含不支持的变量：{{{{{var}}}}}。" f"支持的变量：{sorted(IMAGE_PROMPT_VARS)}" ) + headings = tuple(re.findall(r"(?m)^【([^\n】]+)】\s*$", text)) + if headings != _FIXED_TEMPLATE_HEADINGS: + raise ImagePromptTemplateError( + "模板区块必须严格为：" + " → ".join(_FIXED_TEMPLATE_HEADINGS) + ) + missing = sorted(_FIXED_TEMPLATE_VARS - used_vars) + if missing: + raise ImagePromptTemplateError( + "固定漫画模板缺少变量：" + ", ".join(f"{{{{{name}}}}}" for name in missing) + ) def render_image_prompt_template(text: str, values: dict[str, str]) -> str: diff --git a/app/ai/speaker_attribution.py b/app/ai/speaker_attribution.py new file mode 100644 index 0000000..5239a46 --- /dev/null +++ b/app/ai/speaker_attribution.py @@ -0,0 +1,184 @@ +"""Build deterministic, message-scoped speaker attribution for image prompts.""" + +from __future__ import annotations + +from dataclasses import dataclass +from datetime import datetime +import hashlib +import json +from typing import Any, Iterable + +from app.providers.history.wechat_data_analysis import ( + _sanitize_sender_name, + _usable_sender_name, +) +from app.services.speaker_identity import speaker_identity_key + + +@dataclass(frozen=True) +class AttributionName: + """The display name to use for one specific chat message.""" + + display_name: str + source: str + + +@dataclass(frozen=True) +class AttributionContract: + """Snapshot hashes and per-message names used by the summary pipeline.""" + + names: tuple[AttributionName, ...] + message_snapshot_sha256: str + speaker_fingerprint: str + + +def _field(message: object, name: str) -> Any: + if isinstance(message, dict): + return message.get(name) + return getattr(message, name, None) + + +def _timestamp_text(value: object) -> str: + if isinstance(value, datetime): + return value.isoformat() + return str(value or "") + + +def _snapshot_record(message: object) -> dict[str, str]: + return { + "message_id": str(_field(message, "message_id") or ""), + "group_id": str(_field(message, "group_id") or ""), + "group_name": str(_field(message, "group_name") or ""), + "sender_id": str(_field(message, "sender_id") or ""), + "sender_name": str(_field(message, "sender_name") or ""), + "timestamp": _timestamp_text(_field(message, "timestamp")), + "message_type": str(_field(message, "message_type") or "text"), + "content": str(_field(message, "content") or ""), + "upstream_sender_name": str( + _field(message, "upstream_sender_name") or "" + ), + "sender_name_source": str(_field(message, "sender_name_source") or ""), + } + + +def _sha256(payload: object) -> str: + canonical = json.dumps( + payload, + ensure_ascii=False, + sort_keys=True, + separators=(",", ":"), + ) + return hashlib.sha256(canonical.encode("utf-8")).hexdigest() + + +def _resolved_fallback(message: object, sender_id: str) -> str: + resolved = _sanitize_sender_name(_field(message, "sender_name")) + if resolved and resolved.casefold() not in {"none", "null", "(未知)", "未知"}: + return resolved + identity = speaker_identity_key(sender_id, resolved) + if identity is None: + return "(未知)" + digest = hashlib.sha256(f"{identity[0]}:{identity[1]}".encode("utf-8")).hexdigest()[:8] + return f"未命名成员-{digest}" + + +def resolve_attribution_names(messages: Iterable[object]) -> tuple[AttributionName, ...]: + """Prefer the valid chat-time name; fall back only when it is unsafe. + + The choice is message-scoped. A member who changed their group card during + the day keeps the name that accompanied each individual message. + """ + + rows = list(messages) + sender_ids = { + str(_field(message, "sender_id") or "").strip() + for message in rows + if str(_field(message, "sender_id") or "").strip() + } + sender_ids_casefold = {sender_id.casefold() for sender_id in sender_ids} + upstream_identities: dict[str, set[str]] = {} + normalized_upstream: list[str] = [] + for message in rows: + sender_id = str(_field(message, "sender_id") or "").strip() + upstream = _sanitize_sender_name(_field(message, "upstream_sender_name")) + normalized_upstream.append(upstream) + if sender_id and _usable_sender_name(upstream, sender_id): + upstream_identities.setdefault(upstream.casefold(), set()).add( + sender_id.casefold() + ) + + base: list[AttributionName] = [] + identities: list[tuple[str, str] | None] = [] + for message, upstream in zip(rows, normalized_upstream): + sender_id = str(_field(message, "sender_id") or "").strip() + usable_upstream = bool( + sender_id + and _usable_sender_name(upstream, sender_id) + and upstream.casefold() + not in (sender_ids_casefold - {sender_id.casefold()}) + and len(upstream_identities.get(upstream.casefold(), set())) == 1 + ) + if usable_upstream: + name = upstream + source = "upstream_sender_name" + else: + name = _resolved_fallback(message, sender_id) + source = str(_field(message, "sender_name_source") or "resolved") + base.append(AttributionName(name, source)) + identities.append(speaker_identity_key(sender_id.casefold(), name)) + + # If the fallback still leaves two identities with the same label, keep the + # existing stable suffix convention without changing names that only vary + # over time for the same sender_id. + duplicate_identities: dict[str, set[tuple[str, str]]] = {} + for attribution, identity in zip(base, identities): + if identity is not None: + duplicate_identities.setdefault( + attribution.display_name.casefold(), set() + ).add(identity) + suffixes: dict[tuple[str, tuple[str, str]], int] = {} + for normalized_name, identity_set in duplicate_identities.items(): + if len(identity_set) <= 1: + continue + for number, identity in enumerate(sorted(identity_set), start=1): + suffixes[(normalized_name, identity)] = number + + result: list[AttributionName] = [] + for attribution, identity in zip(base, identities): + number = ( + suffixes.get((attribution.display_name.casefold(), identity)) + if identity is not None + else None + ) + if number is None: + result.append(attribution) + else: + result.append( + AttributionName( + f"{attribution.display_name}（同名 {number}）", + attribution.source, + ) + ) + return tuple(result) + + +def build_attribution_contract(messages: Iterable[object]) -> AttributionContract: + rows = list(messages) + names = resolve_attribution_names(rows) + snapshot_records = [_snapshot_record(message) for message in rows] + speaker_records = [ + { + "message_id": record["message_id"], + "sender_id": record["sender_id"], + "display_name": attribution.display_name, + "display_name_source": attribution.source, + "upstream_sender_name": record["upstream_sender_name"], + "sender_name_source": record["sender_name_source"], + } + for record, attribution in zip(snapshot_records, names) + ] + return AttributionContract( + names=names, + message_snapshot_sha256=_sha256(snapshot_records), + speaker_fingerprint=_sha256(speaker_records), + ) diff --git a/app/ai/strict_prompt_contract.py b/app/ai/strict_prompt_contract.py new file mode 100644 index 0000000..ff7951f --- /dev/null +++ b/app/ai/strict_prompt_contract.py @@ -0,0 +1,47 @@ +"""严格群报图片的可见文案与事实边界。""" + +from __future__ import annotations + +import re + + +STRICT_IMAGE_FACT_MARKER = "【严格事实边界】" + +STRICT_IMAGE_FACT_CONTRACT = f""" + +{STRICT_IMAGE_FACT_MARKER} +这是一项硬性验收合同，优先级高于任何视觉丰富化建议： +1. 图片中所有可见文字只能来自本 Prompt 已明确列出的群名、标题、话题标题、人物昵称、真实聊天气泡、事实说明、统计数据和底部总结。 +2. 禁止补充任何未逐字列出的数字、单位、百分比、金额、体重、身高、健康计算公式、温度、天气持续天数、日期、时间、饮食建议或健康结论。 +3. 禁止把“体脂率低”“健康指标很标准”“雨天包年”等口语扩写成具体百分比、计算公式、温度或天数。 +4. 禁止自行改写计划或建议；例如原文没有“少油少盐”时不得生成该文字。 +5. 视觉笑点只能通过人物动作、表情、道具形状和无文字装饰表达，不得添加新的事实性标签、App 数值、仪表盘或数据卡片。 +6. 无法确认的文字宁可不画；不要用占位符、乱码、伪汉字或装饰性英文填充。 +""".strip("\n") + + +def sanitize_strict_image_prompt(prompt: str) -> str: + """移除 Eason 合同明确禁用的 BMI 展示，同时保留有证据的猜体重话题。""" + safe_lines: list[str] = [] + for raw_line in str(prompt or "").splitlines(): + line = raw_line + line = re.sub(r"手指一路猜到\s*BMI", "从手指猜体重", line, flags=re.IGNORECASE) + line = re.sub(r"身高[、,，]\s*BMI\s*(?:和|与|及)", "身高和", line, flags=re.IGNORECASE) + line = re.sub( + r"(?:旁边的?)?计算器[^。\n]{0,16}弹出\s*BMI\s*窗口", + "旁边连续弹出“猜体重”的聊天气泡", + line, + flags=re.IGNORECASE, + ) + # 用户要求图片不出现 BMI；无法安全改写的整行直接删除。 + if re.search(r"BMI", line, flags=re.IGNORECASE): + continue + safe_lines.append(line) + return "\n".join(safe_lines) + + +def append_strict_image_fact_contract(prompt: str) -> str: + text = sanitize_strict_image_prompt(prompt).rstrip() + if STRICT_IMAGE_FACT_MARKER in text: + return text + "\n" + return f"{text}\n\n{STRICT_IMAGE_FACT_CONTRACT}\n" diff --git a/app/ai/topic_selection.py b/app/ai/topic_selection.py index e7e4f5b..28cb0f4 100644 --- a/app/ai/topic_selection.py +++ b/app/ai/topic_selection.py @@ -14,7 +14,7 @@ from app.ai.conversation_segments import ConversationChunk, PromptMessage -TOPIC_SELECTION_VERSION = "4.0" +TOPIC_SELECTION_VERSION = "6.0" MAX_CANDIDATES = 10 MIN_SELECTED = 2 TARGET_SELECTED = 5 @@ -22,8 +22,8 @@ HIGH_VOLUME_MESSAGE_THRESHOLD = 200 SELECTION_MIN_SCORE = 60.0 SELECTION_MAX_GAP = 15.0 -VISIBLE_PARTICIPANT_LIMIT = 3 -VISIBLE_PARTICIPANT_CHAR_BUDGET = 24 +VISIBLE_PARTICIPANT_LIMIT = 4 +VISIBLE_PARTICIPANT_CHAR_BUDGET = 48 UNRESOLVED_PARTICIPANT_LABEL = "群友（昵称未识别）" SCORE_WEIGHTS = { @@ -37,13 +37,15 @@ TOPIC_CANDIDATE_SYSTEM = """你是群聊日报选题编辑。只能基于给定消息或事件卡整理候选主题。 必须返回一个 JSON 对象，不得输出 Markdown 或解释。候选主题必须引用真实 message_ids，禁止虚构。 +你只负责选择 message_ids 和概括事件，不得输出 people、quotes 或任何人物姓名； +人物身份、显示名和逐字原话全部由程序按 message_id 从当前快照回填。 事实真实性是准入门槛；通过真实性校验后，好玩程度是第一排序目标。 内容充足时输出 10 个候选，证据不足时允许少于 10 个；共享同一核心事实、里程碑或结论的相似话题必须合并， 不得为了凑数把一个事件拆成“发起/回应”或重复角度。 comedy_score 为 0～40，group_recognition_score 为 0～20，visual_score 为 0～20。 comedy_angle 说明真实笑点，visual_gag 说明不改变事实的视觉笑点，并给出简短 score_reason。 -为避免响应截断，标题、人物和评分理由必须简洁；summary 必须是一句可直接放进图片的信息，建议不超过 48 个汉字； -quotes 只能逐字摘录原消息，禁止改写或把摘要伪装成原话； +为避免响应截断，标题和评分理由必须简洁；summary 必须是不含人物姓名的一句完整事件概括，建议不超过 60 个汉字， +不得以省略号、半句话或残缺表情代码结尾； 每个候选的 message_ids 最多保留 100 条有效证据。""" _CANDIDATE_SCHEMA = """返回结构： @@ -52,9 +54,7 @@ { "topic_id": "topic-01", "title": "主题短标题", - "summary": "基于证据的一句事实过程或结论", - "people": ["真实参与者"], - "quotes": ["1-3 条逐字摘录的真实原话"], + "summary": "不含人物姓名、基于证据的一句事实过程或结论", "start_time": "YYYY-MM-DD HH:MM", "end_time": "YYYY-MM-DD HH:MM", "message_ids": ["真实消息ID"], @@ -83,6 +83,7 @@ def __init__(self, code: str, detail: str): @dataclass(frozen=True) class TopicEvidence: message_id: str + sender_id: str sender_key: str sender_name: str text: str @@ -191,9 +192,15 @@ def _evidence(messages: Iterable[PromptMessage]) -> dict[str, TopicEvidence]: for source_index, item in enumerate(messages): if not item.message_id: continue + if item.message_id in result: + raise TopicSelectionError( + "TOPIC_EVIDENCE_DUPLICATE_ID", + f"messages.json 包含重复 message_id：{item.message_id}", + ) result[item.message_id] = TopicEvidence( message_id=item.message_id, - sender_key=item.sender_id or item.sender_name or "(未知)", + sender_id=item.sender_id, + sender_key=(item.sender_id or "").casefold() or item.sender_name or "(未知)", sender_name=item.sender_name or "(未知)", text=item.text or "", timestamp=item.timestamp, @@ -209,6 +216,26 @@ def _normalized_quote(value: str) -> str: return _QUOTE_PUNCTUATION_RE.sub("", unicodedata.normalize("NFKC", value or "")).casefold() +def _complete_excerpt(value: str, maximum: int) -> str: + """保留完整可读句或连续短句，不制造悬空省略号。""" + text = re.sub(r"\s+", " ", value or "").strip() + if len(text) <= maximum: + return text + sentences = [part.strip() for part in re.split(r"(?<=[。！？!?；;])", text) if part.strip()] + selected = "" + for sentence in sentences: + if len(sentence) > maximum: + continue + combined = selected + sentence + if len(combined) > maximum: + break + selected = combined + if selected: + return selected + clauses = [part.strip() for part in re.split(r"[，,、：:]", text) if part.strip()] + return next((part for part in clauses if len(part) <= maximum), "") + + def _verified_quotes(candidate_quotes: Iterable[str], items: Iterable[TopicEvidence]) -> list[str]: """只保留可从所引消息回查的原话；无有效候选时使用真实原文短句。""" evidence_items = list(items) @@ -220,14 +247,15 @@ def _verified_quotes(candidate_quotes: Iterable[str], items: Iterable[TopicEvide continue if any(normalized in message for message, _ in normalized_messages) and quote not in verified: verified.append(quote.strip()) - if len(verified) >= 2: + if len(verified) >= 3: break if verified: return verified for _, original in normalized_messages: - if original: - return [original[:80]] + excerpt = _complete_excerpt(original, 80) + if excerpt: + return [excerpt] return [] @@ -238,11 +266,16 @@ def _evidence_dialogue(items: Iterable[TopicEvidence]) -> list[dict[str, str]]: text = item.text.strip() if not text: continue + excerpt = _complete_excerpt(text, 240) + if not excerpt: + continue dialogue.append( { "message_id": item.message_id, + "sender_id": item.sender_id, "speaker": item.sender_name, - "text": text[:240], + "text": excerpt, + "original_text": text, } ) return dialogue @@ -336,19 +369,29 @@ def score_and_select_topics( if len(timestamps) >= 2: duration = max(0.0, (max(timestamps) - min(timestamps)).total_seconds() / 60.0) attribution = _participant_attribution(items) - verified_quotes = _verified_quotes(candidate.get("quotes", []), items) + # 姓名和原话不能采用模型自由输出；统一按 message_id 从当前快照回填。 + verified_quotes = _verified_quotes((), items) if not verified_quotes: raise TopicSelectionError( "TOPIC_CANDIDATES_INVALID", f"候选主题“{candidate['title']}”没有可回查的原话证据", ) + evidence_dialogue = _evidence_dialogue(items) + dialogue_speaker_count = len( + { + str(entry.get("speaker") or "").strip() + for entry in evidence_dialogue + if _resolved_participant_name(str(entry.get("speaker") or "")) + } + ) metrics.append( { **candidate, "message_ids": ids, "people": attribution["participants"], "quotes": verified_quotes, - "evidence_dialogue": _evidence_dialogue(items), + "evidence_dialogue": evidence_dialogue, + "dialogue_speaker_count": dialogue_speaker_count, "evidence_message_count": len(ids), **attribution, "duration_minutes": round(duration, 1), @@ -384,7 +427,12 @@ def score_and_select_topics( if guaranteed_selected < rank <= MAX_SELECTED: total = item["scores"]["total"] gap = (previous_total - total) if previous_total is not None else 0.0 - selected = total >= SELECTION_MIN_SCORE and gap < SELECTION_MAX_GAP + selected = ( + total >= SELECTION_MIN_SCORE + and gap < SELECTION_MAX_GAP + and item["participant_count"] >= 2 + and item["dialogue_speaker_count"] >= 2 + ) if not selected: # 分数降序；第一个不满足后，后续候选也不再入选。 for tail in scored[rank - 1 :]: diff --git a/app/api/email.py b/app/api/email.py index eb97eec..a0890c6 100644 --- a/app/api/email.py +++ b/app/api/email.py @@ -2,18 +2,26 @@ from __future__ import annotations -from fastapi import APIRouter, Depends +from fastapi import APIRouter, Depends, HTTPException from sqlmodel import Session +from app.config.settings import Settings, get_settings from app.db import repository as repo from app.services.email_service import EmailService +from app.services.legacy_v1_policy import ( + LegacyV1WriteBlockedError, + require_legacy_v1_write, +) -router = APIRouter(prefix="/api/email", tags=["email"]) +router = APIRouter(prefix="/api/email", tags=["email"], deprecated=True) @router.get("/preview") -def preview(session: Session = Depends(repo.get_session)): - service = EmailService() +def preview( + session: Session = Depends(repo.get_session), + settings: Settings = Depends(get_settings), +): + service = EmailService(settings) result = service.build_email(session) return { "subject": result.subject, @@ -24,7 +32,18 @@ def preview(session: Session = Depends(repo.get_session)): @router.post("/send") -def send(session: Session = Depends(repo.get_session)): - service = EmailService() +def send( + session: Session = Depends(repo.get_session), + settings: Settings = Depends(get_settings), +): + try: + require_legacy_v1_write( + settings, + operation="email.send", + replacement="V2 每日任务或 scripts/send_daily_email.py", + ) + except LegacyV1WriteBlockedError as exc: + raise HTTPException(status_code=410, detail=exc.as_detail()) from exc + service = EmailService(settings) ok, detail = service.send(session) return {"ok": ok, "detail": detail} diff --git a/app/api/files.py b/app/api/files.py index a9d71a0..97d1b4b 100644 --- a/app/api/files.py +++ b/app/api/files.py @@ -2,32 +2,63 @@ from __future__ import annotations -from fastapi import APIRouter, Depends +from fastapi import APIRouter, Depends, HTTPException from fastapi.responses import FileResponse, PlainTextResponse +from app.config.settings import Settings, get_settings +from app.core.path_security import resolve_within from app.services.handoff_service import HandoffService -router = APIRouter(prefix="/api/files", tags=["files"]) +router = APIRouter(prefix="/api/files", tags=["files"], deprecated=True) + +ALLOWED_RAW_FILES = frozenset( + { + "ranking.txt", + "image_prompt.txt", + "meta.json", + "normalized_messages.json", + "handoff.json", + "messages.json", + "ranking.json", + "run.json", + "image_prompt.original.txt", + "daily_image.png", + "daily_image.previous.png", + } +) @router.get("/dates") -def dates(): - service = HandoffService() +def dates(settings: Settings = Depends(get_settings)): + service = HandoffService(settings) return service.list_output_dates() @router.get("/{report_date}") -def list_day(report_date: str): - service = HandoffService() - return service.list_group_outputs(report_date) +def list_day(report_date: str, settings: Settings = Depends(get_settings)): + service = HandoffService(settings) + try: + return service.list_group_outputs(report_date) + except ValueError as exc: + raise HTTPException(status_code=400, detail=str(exc)) from exc @router.get("/{report_date}/{group_dir}/raw/{filename}") -def raw_file(report_date: str, group_dir: str, filename: str): - service = HandoffService() - day_dir = service.settings.output_dir / report_date - file = (day_dir / group_dir / filename).resolve() - if not file.is_file() or not str(file).startswith(str(day_dir.resolve())): +def raw_file( + report_date: str, + group_dir: str, + filename: str, + settings: Settings = Depends(get_settings), +): + if filename not in ALLOWED_RAW_FILES: + raise HTTPException(status_code=400, detail=f"不允许访问的文件：{filename}") + service = HandoffService(settings) + try: + day_dir = service.output_day_dir(report_date) + file = resolve_within(day_dir, group_dir, filename) + except ValueError as exc: + raise HTTPException(status_code=400, detail=str(exc)) from exc + if not file.is_file(): return PlainTextResponse("not found", status_code=404) if filename.endswith(".json"): return PlainTextResponse(file.read_text(encoding="utf-8"), media_type="application/json") diff --git a/app/api/groups.py b/app/api/groups.py index 8301133..2cf61a9 100644 --- a/app/api/groups.py +++ b/app/api/groups.py @@ -6,8 +6,12 @@ from pydantic import BaseModel from sqlmodel import Session, select -from app.config.settings import Settings, get_settings -from app.ai.image_themes import DEFAULT_IMAGE_THEME, ImageThemeError, resolve_image_theme, validate_image_theme_config +from app.ai.image_themes import ( + DEFAULT_IMAGE_THEME, + ImageThemeError, + resolve_image_theme, + validate_image_theme_config, +) from app.ai.prompt_builder import _strip_html_comments from app.ai.prompt_editing import prompt_revision, resolved_theme_text from app.ai.prompt_templates import ( @@ -16,6 +20,8 @@ render_image_prompt_template, validate_image_prompt_template, ) +from app.config.settings import Settings, get_settings +from app.core.path_security import PathBoundaryError, validate_path_label from app.db import repository as repo from app.db.models import Group from app.data_sources.wechat_data_analysis import WeChatDataAnalysisSource @@ -24,6 +30,7 @@ effective_send_target, send_target_mode, ) +from app.services.group_provider_config import validate_group_provider_values router = APIRouter(prefix="/api/groups", tags=["groups"]) @@ -35,15 +42,19 @@ class GroupCreate(BaseModel): enabled: bool = True provider_preference: str = "" # V2 扩展 - schedule_rule: str = "weekday_default" + schedule_rule: str = "daily_previous_day" send_time: str = "08:30" - summary_model: str = "gpt-5.6-sol" - prompt_model: str = "gpt-5.6-sol" + summary_provider: str = "" + prompt_provider: str = "" + summary_model: str = "" + prompt_model: str = "" image_enabled: bool = True send_target: str = "" ranking_template: str = "default" + ranking_count_policy: str = "all_messages" + sender_name_policy: str = "resolved" image_prompt_template: str = "default" - image_theme: str = "random_preset" + image_theme: str = DEFAULT_IMAGE_THEME image_theme_custom: str = "" image_prompt_override: str = "" wechat_send_enabled: bool = False @@ -58,11 +69,15 @@ class GroupUpdate(BaseModel): # V2 扩展 schedule_rule: str | None = None send_time: str | None = None + summary_provider: str | None = None + prompt_provider: str | None = None summary_model: str | None = None prompt_model: str | None = None image_enabled: bool | None = None send_target: str | None = None ranking_template: str | None = None + ranking_count_policy: str | None = None + sender_name_policy: str | None = None image_prompt_template: str | None = None image_theme: str | None = None image_theme_custom: str | None = None @@ -91,6 +106,15 @@ def _validate_group_theme(theme: object, custom: object = "") -> tuple[str, str] raise HTTPException(status_code=422, detail=str(exc)) from exc +def _validate_output_group_name(value: object, *, field_name: str) -> str: + """拒绝会被当作文件路径的群名称，同时保留普通显示名标点。""" + text = str(value or "") + try: + return validate_path_label(text, field_name=field_name) + except PathBoundaryError as exc: + raise HTTPException(status_code=422, detail=str(exc)) from exc + + def _validate_prompt_override(content: object) -> str: if content is None: return "" @@ -119,7 +143,7 @@ def _group_prompt_payload(group: Group) -> dict: preview = render_image_prompt_template( _strip_html_comments(content), { - "group_name": group.display_name or group.wechat_group_name, + "group_name": group.wechat_group_name or group.display_name, "period_start": "（生成时写入统计开始时间）", "period_end": "（生成时写入统计结束时间）", "message_count": "（生成时写入消息数）", @@ -147,8 +171,28 @@ def _require_active_group(session: Session, group_id: int) -> Group: return group +def _apply_global_send_time( + values: dict, + settings: Settings, + *, + explicitly_provided: bool, +) -> dict: + configured = str(getattr(settings, "schedule_send_time", "08:30") or "08:30") + provided = str(values.get("send_time") or "") + if explicitly_provided and provided != configured: + raise HTTPException( + status_code=422, + detail=f"群聊不能单独设置发送时间；日报发送批次固定为 {configured}", + ) + values["send_time"] = configured + return values + + @router.get("") -def list_groups(session: Session = Depends(repo.get_session)): +def list_groups( + session: Session = Depends(repo.get_session), + settings: Settings = Depends(get_settings), +): groups = repo.list_groups(session) return [ { @@ -158,8 +202,13 @@ def list_groups(session: Session = Depends(repo.get_session)): "wechat_group_name": g.wechat_group_name, "enabled": g.enabled, "provider_preference": g.provider_preference, + "history_provider_preference": g.provider_preference, "schedule_rule": g.schedule_rule, - "send_time": g.send_time, + "send_time": str( + getattr(settings, "schedule_send_time", "08:30") or "08:30" + ), + "summary_provider": g.summary_provider, + "prompt_provider": g.prompt_provider, "summary_model": g.summary_model, "prompt_model": g.prompt_model, "image_enabled": g.image_enabled, @@ -167,6 +216,8 @@ def list_groups(session: Session = Depends(repo.get_session)): "effective_send_target": effective_send_target(g), "send_target_mode": send_target_mode(g), "ranking_template": g.ranking_template, + "ranking_count_policy": g.ranking_count_policy, + "sender_name_policy": g.sender_name_policy, "image_prompt_template": g.image_prompt_template, "image_theme": g.image_theme, "image_theme_custom": g.image_theme_custom, @@ -180,11 +231,31 @@ def list_groups(session: Session = Depends(repo.get_session)): @router.post("") -def create_group(payload: GroupCreate, session: Session = Depends(repo.get_session)): - values = payload.model_dump() +def create_group( + payload: GroupCreate, + session: Session = Depends(repo.get_session), + settings: Settings = Depends(get_settings), +): + raw_values = payload.model_dump() + values = _apply_global_send_time( + raw_values, + settings, + explicitly_provided="send_time" in payload.model_fields_set, + ) + try: + values = validate_group_provider_values(values, settings) + except ValueError as exc: + raise HTTPException(status_code=422, detail=str(exc)) from exc + values["display_name"] = _validate_output_group_name( + values.get("display_name"), field_name="display_name" + ) + if values.get("wechat_group_name"): + values["wechat_group_name"] = _validate_output_group_name( + values["wechat_group_name"], field_name="wechat_group_name" + ) values["send_target"] = str(values.get("send_target") or "").strip() values["image_theme"], values["image_theme_custom"] = _validate_group_theme( - values.get("image_theme", "random_preset"), values.get("image_theme_custom", "") + values.get("image_theme", DEFAULT_IMAGE_THEME), values.get("image_theme_custom", "") ) values["image_prompt_override"] = _validate_prompt_override(values.get("image_prompt_override", "")) wechat_group_id = str(values.get("wechat_group_id") or "").strip() @@ -205,10 +276,31 @@ def create_group(payload: GroupCreate, session: Session = Depends(repo.get_sessi @router.put("/{group_id}") def update_group( - group_id: int, payload: GroupUpdate, session: Session = Depends(repo.get_session) + group_id: int, + payload: GroupUpdate, + session: Session = Depends(repo.get_session), + settings: Settings = Depends(get_settings), ): group = _require_active_group(session, group_id) - updates = payload.model_dump(exclude_unset=True) + raw_updates = payload.model_dump(exclude_unset=True) + updates = _apply_global_send_time( + raw_updates, + settings, + explicitly_provided="send_time" in payload.model_fields_set, + ) + try: + updates = validate_group_provider_values( + updates, + settings, + base=group, + ) + except ValueError as exc: + raise HTTPException(status_code=422, detail=str(exc)) from exc + for field_name in ("display_name", "wechat_group_name"): + if updates.get(field_name): + updates[field_name] = _validate_output_group_name( + updates[field_name], field_name=field_name + ) if "image_theme" in updates or "image_theme_custom" in updates: theme, custom = _validate_group_theme( updates.get("image_theme", group.image_theme), @@ -330,14 +422,17 @@ def bind_group_from_name( }, ) + selected_name = selected.group_name or name + _validate_output_group_name(selected_name, field_name="group_name") + existing = session.exec( select(Group).where(Group.wechat_group_id == selected.group_id) ).first() if existing: if existing.deleted_at is not None: - existing.wechat_group_name = selected.group_name or existing.wechat_group_name + existing.wechat_group_name = selected_name or existing.wechat_group_name if not existing.display_name: - existing.display_name = selected.group_name or name + existing.display_name = selected_name restored = repo.restore_group(session, existing.id) return { "id": restored.id, @@ -349,9 +444,9 @@ def bind_group_from_name( return {"id": existing.id, "bound": True, "already_existed": True} group = Group( - display_name=selected.group_name or name, + display_name=selected_name, wechat_group_id=selected.group_id, - wechat_group_name=selected.group_name, + wechat_group_name=selected_name, ) group = repo.save_group(session, group) return {"id": group.id, "bound": True, "already_existed": False} diff --git a/app/api/reports.py b/app/api/reports.py index 8d17665..b09577d 100644 --- a/app/api/reports.py +++ b/app/api/reports.py @@ -8,12 +8,17 @@ from pydantic import BaseModel from sqlmodel import Session +from app.config.settings import Settings, get_settings from app.db import repository as repo from app.db.models import GroupRun, Report from app.services.report_service import ReportService from app.services.generation_runtime import GenerationBusyError +from app.services.legacy_v1_policy import ( + LegacyV1WriteBlockedError, + require_legacy_v1_write, +) -router = APIRouter(prefix="/api/reports", tags=["reports"]) +router = APIRouter(prefix="/api/reports", tags=["reports"], deprecated=True) class GenerateRequest(BaseModel): @@ -26,9 +31,29 @@ class PromptUpdate(BaseModel): text: str +def _require_write(settings: Settings, *, operation: str, replacement: str) -> None: + try: + require_legacy_v1_write( + settings, + operation=operation, + replacement=replacement, + ) + except LegacyV1WriteBlockedError as exc: + raise HTTPException(status_code=410, detail=exc.as_detail()) from exc + + @router.post("/generate") -def generate(payload: GenerateRequest, session: Session = Depends(repo.get_session)): - service = ReportService() +def generate( + payload: GenerateRequest, + session: Session = Depends(repo.get_session), + settings: Settings = Depends(get_settings), +): + _require_write( + settings, + operation="report.generate", + replacement="POST /api/v2/pipeline/generate", + ) + service = ReportService(settings=settings) group = None if payload.group_id is not None: group = repo.get_active_group(session, payload.group_id) @@ -65,6 +90,9 @@ def latest(session: Session = Depends(repo.get_session)): "id": r.id, "group_run_id": r.group_run_id, "group_id": group_run.group_id if group_run else None, + "legacy_group_id": group_run.legacy_group_id if group_run else None, + "identity_state": group_run.identity_state if group_run else "unresolved", + "orphan_reason": group_run.orphan_reason if group_run else "group_run_missing", "ranking_text": r.ranking_text, "prompt_text": r.prompt_text, "ranking_file": r.ranking_file, @@ -79,7 +107,17 @@ def latest(session: Session = Depends(repo.get_session)): @router.put("/{report_id}/prompt") -def update_prompt(report_id: int, payload: PromptUpdate, session: Session = Depends(repo.get_session)): +def update_prompt( + report_id: int, + payload: PromptUpdate, + session: Session = Depends(repo.get_session), + settings: Settings = Depends(get_settings), +): + _require_write( + settings, + operation="report.prompt.update", + replacement="PUT /api/v2/runs/{group}/{run_date}/prompt", + ) report = session.get(Report, report_id) if not report: raise HTTPException(404, "报告不存在") diff --git a/app/api/runs.py b/app/api/runs.py index ca71931..576c22f 100644 --- a/app/api/runs.py +++ b/app/api/runs.py @@ -7,7 +7,7 @@ from app.db import repository as repo -router = APIRouter(prefix="/api/runs", tags=["runs"]) +router = APIRouter(prefix="/api/runs", tags=["runs"], deprecated=True) @router.get("/{run_id}") @@ -27,14 +27,21 @@ def run_detail(run_id: int, session: Session = Depends(repo.get_session)): ).all() details = [] for gr in group_runs: - group = session.get(Group, gr.group_id) + group = session.get(Group, gr.group_id) if gr.group_id is not None else None + if group: + group_name = group.display_name or group.wechat_group_name + elif gr.identity_state == "orphaned": + group_name = f"历史群（旧 ID {gr.legacy_group_id}）" + else: + group_name = "未知群" details.append( { "id": gr.id, "group_id": gr.group_id, - "group_name": (group.display_name or group.wechat_group_name) - if group - else f"群 {gr.group_id}", + "legacy_group_id": gr.legacy_group_id, + "identity_state": gr.identity_state, + "orphan_reason": gr.orphan_reason, + "group_name": group_name, "provider_used": gr.provider_used, "message_count": gr.message_count, "speaker_count": gr.speaker_count, diff --git a/app/api/settings.py b/app/api/settings.py index 9d71fa8..1e7dc83 100644 --- a/app/api/settings.py +++ b/app/api/settings.py @@ -5,11 +5,15 @@ from __future__ import annotations -from fastapi import APIRouter, Depends +from fastapi import APIRouter, Depends, HTTPException from pydantic import BaseModel from sqlmodel import Session from app.db import repository as repo +from app.config.settings import get_settings as get_runtime_settings +from app.providers.ai.codex import validate_summary_provider_config +from app.sender.wechat_native import validate_wechat_sender_mode +from app.services.email_service import email_delivery_config_error router = APIRouter(prefix="/api/settings", tags=["settings"]) @@ -22,9 +26,6 @@ } EDITABLE_KEYS = { - "history_provider_primary", - "history_provider_fallback", - "history_provider_mock_enabled", "wechat_data_dir", "wechat_export_dir", "wechat_cli_path", @@ -41,7 +42,6 @@ "codex_summary_timeout_seconds", "codex_summary_max_retries", "codex_summary_request_concurrency", - "ai_provider", "ai_base_url", "ai_api_key", "ai_model", @@ -55,8 +55,13 @@ "codex_home", "codex_timeout_seconds", "codex_generated_images_dir", + "image_generation_concurrency", + "image_fallback_font_path", "wechat_sender_mode", "wechat_native_action_delay_seconds", + "wechat_native_stage_timeout_seconds", + "wechat_native_submit_timeout_seconds", + "wechat_native_poll_interval_seconds", "wechat_native_mutex_timeout_seconds", "wechat_send_claim_seconds", "wechat_late_send_window_minutes", @@ -76,6 +81,18 @@ ALL_KEYS = EDITABLE_KEYS | SENSITIVE_KEYS +_SUMMARY_CONFIG_KEYS = {"summary_provider_primary", "summary_provider_fallback"} +_EMAIL_CONFIG_KEYS = { + "email_enabled", + "email_recipient", + "email_from", + "email_smtp_host", + "email_smtp_port", + "email_smtp_user", + "email_smtp_password", + "email_use_ssl", +} + class SettingsPayload(BaseModel): values: dict[str, str] @@ -95,17 +112,35 @@ def get_settings(session: Session = Depends(repo.get_session)): @router.put("") def update_settings(payload: SettingsPayload, session: Session = Depends(repo.get_session)): - applied: dict[str, str] = {} + requested: dict[str, str] = {} for key, value in payload.values.items(): if key not in EDITABLE_KEYS: continue if key in SENSITIVE_KEYS and value in ("", "******"): continue # 不覆盖已有密钥 - repo.set_setting_value(session, key, value) - applied[key] = value - if applied: - # 让本次修改立即在运行中的 Settings 实例生效（类型安全、忽略掩码值）。 - from app.config.settings import get_settings + requested[key] = value + if requested: + runtime_settings = get_runtime_settings() + candidate = runtime_settings.model_copy(deep=True) + converted = set(candidate.apply_runtime_values(requested)) + rejected = sorted(set(requested) - converted) + if rejected: + raise HTTPException(status_code=422, detail=f"设置值类型无效：{', '.join(rejected)}") + changed = set(requested) + try: + if changed & _SUMMARY_CONFIG_KEYS: + validate_summary_provider_config(candidate) + if "wechat_sender_mode" in changed: + validate_wechat_sender_mode(candidate) + if candidate.email_enabled and changed & _EMAIL_CONFIG_KEYS: + email_error = email_delivery_config_error(candidate) + if email_error: + raise ValueError(email_error) + except ValueError as exc: + raise HTTPException(status_code=422, detail=str(exc)) from exc - get_settings().apply_runtime_values(applied) + # 先完整校验，再写入持久化设置，避免部分无效配置已经入库。 + for key, value in requested.items(): + repo.set_setting_value(session, key, value) + runtime_settings.apply_runtime_values(requested) return {"ok": True} diff --git a/app/api/system.py b/app/api/system.py index fdffefb..449016f 100644 --- a/app/api/system.py +++ b/app/api/system.py @@ -2,14 +2,19 @@ from __future__ import annotations -from datetime import datetime +from datetime import datetime, timedelta +import json +import os from zoneinfo import ZoneInfo -from fastapi import APIRouter, Depends -from sqlmodel import Session +from fastapi import APIRouter, Depends, Request +from fastapi.responses import JSONResponse +from sqlalchemy import text +from sqlmodel import Session, select from app.config.settings import Settings, get_settings from app.db import repository as repo +from app.db.models import ProviderHealth as StoredProviderHealth from app.scheduler.calendar_rules import get_report_window router = APIRouter(prefix="/api/system", tags=["system"]) @@ -17,7 +22,250 @@ @router.get("/health") def health(): - return {"status": "ok"} + return { + "status": "ok", + "service": "groupbrief", + "timestamp": datetime.now().astimezone().isoformat(), + } + + +@router.get("/ready") +def readiness( + request: Request, + session: Session = Depends(repo.get_session), + settings: Settings = Depends(get_settings), +): + """被动检查真实运行能力；不主动调用外部 Provider，也不写任何状态。""" + checks: dict[str, dict] = {} + try: + session.exec(text("SELECT 1")).first() + checks["database"] = { + "ok": True, + "status": "OK", + "detail": "数据库连接可用", + } + except Exception as exc: + checks["database"] = { + "ok": False, + "status": "UNAVAILABLE", + "detail": str(exc)[:200], + } + + output_ok = settings.output_dir.is_dir() and os.access( + settings.output_dir, + os.W_OK, + ) + checks["output"] = { + "ok": output_ok, + "status": "OK" if output_ok else "UNAVAILABLE", + "detail": "output 目录存在且可写" if output_ok else "output 目录不存在或不可写", + } + + try: + from app.ai.prompt_templates import ImagePromptTemplateService + from app.ranking.template_service import RankingTemplateService + + templates_ok = ( + "default" in RankingTemplateService().list_templates() + and "default" in ImagePromptTemplateService().list_templates() + ) + template_detail = "默认排行和生图模板可读" if templates_ok else "缺少默认模板" + except Exception as exc: + templates_ok = False + template_detail = str(exc)[:200] + checks["templates"] = { + "ok": templates_ok, + "status": "OK" if templates_ok else "UNAVAILABLE", + "detail": template_detail, + } + + startup_error = str( + getattr(request.app.state, "startup_check_error", "") or "" + ) + checks["startup_capture"] = { + "ok": not startup_error, + "status": "OK" if not startup_error else "ERROR", + "detail": startup_error or "启动检查结果已保留", + } + + startup_checks = list(getattr(request.app.state, "startup_checks", []) or []) + + def startup_dependency(name_fragment: str, label: str, *, required: bool = True) -> dict: + row = next( + ( + item + for item in startup_checks + if name_fragment in str(item.get("name") or "") + ), + None, + ) + if not required: + return {"ok": True, "status": "NOT_REQUIRED", "detail": f"{label} 当前无启用任务"} + if row is None: + return {"ok": False, "status": "UNKNOWN", "detail": f"缺少 {label} 启动检查快照"} + return { + "ok": bool(row.get("ok")), + "status": str(row.get("status") or "UNKNOWN"), + "detail": str(row.get("detail") or "")[:200], + "checked_at": str(getattr(request.app.state, "startup_checks_at", "") or ""), + } + + try: + enabled_groups = repo.list_groups(session, only_enabled=True) + except Exception: + enabled_groups = [] + has_groups = bool(enabled_groups) + has_send_groups = any(bool(getattr(group, "wechat_send_enabled", False)) for group in enabled_groups) + checks["wechat_data"] = startup_dependency( + "WeChatDataAnalysis", + "WeChatDataAnalysis", + required=has_groups, + ) + checks["wechat_client"] = startup_dependency( + "微信客户端", + "微信客户端", + required=has_send_groups, + ) + codex = startup_dependency("Codex GPT", "Codex GPT", required=has_groups) + deepseek = startup_dependency("DeepSeek", "DeepSeek 备用", required=has_groups) + if not has_groups: + checks["summary_provider"] = codex + else: + checks["summary_provider"] = { + "ok": bool(codex["ok"] or deepseek["ok"]), + "status": "OK" if codex["ok"] or deepseek["ok"] else "UNAVAILABLE", + "detail": f"Codex={codex['status']}；DeepSeek={deepseek['status']}", + } + + owner = str(getattr(request.app.state, "scheduler_owner", settings.scheduler_owner)) + scheduler_active = bool(getattr(request.app.state, "scheduler_active", False)) + if owner != "fastapi": + checks["scheduler_heartbeat"] = { + "ok": True, + "status": "EXTERNAL_OWNER", + "detail": f"Scheduler owner={owner}，本进程不负责心跳", + } + else: + from app.scheduler.heartbeat import load_scheduler_heartbeat + from app.scheduler.manager import get_scheduler + + heartbeat = load_scheduler_heartbeat(settings) + try: + beat_at = datetime.fromisoformat(str(heartbeat.get("last_beat_at") or "")) + now = datetime.now(beat_at.tzinfo) if beat_at.tzinfo else datetime.now() + age_seconds = max(0, int((now - beat_at).total_seconds())) + except ValueError: + age_seconds = -1 + active_scheduler = get_scheduler() + scheduler_running = bool( + active_scheduler is not None + and getattr(active_scheduler, "running", False) + ) + next_run_at = "" + if scheduler_running: + scheduled_times = [ + job.next_run_time + for job in active_scheduler.get_jobs() + if getattr(job, "next_run_time", None) is not None + ] + if scheduled_times: + next_run_at = min(scheduled_times).isoformat() + heartbeat_ok = scheduler_active and scheduler_running + checks["scheduler_heartbeat"] = { + "ok": heartbeat_ok, + "status": "OK" if heartbeat_ok else "STOPPED", + "detail": ( + f"scheduler=running next_run_at={next_run_at or 'pending'} " + f"last_event={heartbeat.get('last_job', '')} age={age_seconds}s" + if heartbeat_ok + else "APScheduler 未运行" + ), + "next_run_at": next_run_at, + "last_beat_at": str(heartbeat.get("last_beat_at") or ""), + } + + try: + tz = ZoneInfo(settings.app_timezone) + now = datetime.now(tz) + generate_at = datetime.combine( + now.date(), + datetime.strptime(str(settings.schedule_generate_time), "%H:%M").time(), + tzinfo=tz, + ) + grace = timedelta(minutes=30) + state_path = settings.output_dir / ".scheduler" / f"{now.date().isoformat()}.json" + try: + daily_state = json.loads(state_path.read_text(encoding="utf-8")) + except (OSError, UnicodeError, json.JSONDecodeError): + daily_state = {} + if now <= generate_at + grace: + daily_ok = True + daily_status = "NOT_DUE" if now < generate_at else "GRACE_PERIOD" + daily_detail = f"检查宽限截止 {generate_at + grace}" + else: + daily_ok = bool(daily_state.get("generation_completed_at")) + daily_status = "OK" if daily_ok else "OVERDUE" + daily_detail = ( + f"generation_status={daily_state.get('generation_status', 'missing')}" + ) + except Exception as exc: + daily_ok = False + daily_status = "ERROR" + daily_detail = str(exc)[:200] + daily_result = { + "ok": daily_ok, + "status": daily_status, + "detail": daily_detail, + } + # readiness 仅表示进程和依赖可用性；今日任务结果单独投影，不再拖低 ready。 + ready = all(item["ok"] for item in checks.values()) + payload = { + "ready": ready, + "status": "ready" if ready else "degraded", + "checks": checks, + "daily_result": daily_result, + "scheduler_owner": getattr( + request.app.state, + "scheduler_owner", + settings.scheduler_owner, + ), + "scheduler_active": bool( + getattr(request.app.state, "scheduler_active", False) + ), + } + return JSONResponse(payload, status_code=200 if ready else 503) + + +def _provider_health_payload(row: StoredProviderHealth) -> dict: + return { + "status": row.status, + "detail": row.detail, + "ok": row.status == "OK", + "checked_at": row.checked_at.isoformat() if row.checked_at else "", + } + + +def _prune_provider_health( + session: Session, + *, + max_per_provider: int = 100, +) -> int: + rows = session.exec( + select(StoredProviderHealth).order_by( + StoredProviderHealth.provider, + StoredProviderHealth.checked_at.desc(), + StoredProviderHealth.id.desc(), + ) + ).all() + counts: dict[str, int] = {} + removed = 0 + for row in rows: + counts[row.provider] = counts.get(row.provider, 0) + 1 + if counts[row.provider] <= max_per_provider: + continue + session.delete(row) + removed += 1 + return removed @router.get("/providers") @@ -25,30 +273,58 @@ def providers( session: Session = Depends(repo.get_session), settings: Settings = Depends(get_settings), ): - from datetime import datetime + """读取最近一次已保存的 Provider 健康结果，不执行外部检查。""" + rows = session.exec( + select(StoredProviderHealth).order_by( + StoredProviderHealth.checked_at.desc(), + StoredProviderHealth.id.desc(), + ) + ).all() + latest: dict[str, StoredProviderHealth] = {} + for row in rows: + latest.setdefault(row.provider, row) + from app.services.group_provider_config import provider_catalog + + return { + **{name: _provider_health_payload(row) for name, row in latest.items()}, + "catalog": provider_catalog(settings), + } + - from app.db.models import ProviderHealth +@router.post("/providers/refresh") +def refresh_providers( + session: Session = Depends(repo.get_session), + settings: Settings = Depends(get_settings), +): + """显式执行 Provider 深度检查，保存结果并限制历史记录数量。""" from app.providers.history.registry import check_all_health health = check_all_health(settings) now = datetime.now() for name, h in health.items(): session.add( - ProviderHealth( + StoredProviderHealth( provider=name, status=h.status.value, detail=h.detail[:500], checked_at=now, ) ) + session.flush() + _prune_provider_health(session) session.commit() return { - name: {"status": h.status.value, "detail": h.detail, "ok": h.ok} + name: { + "status": h.status.value, + "detail": h.detail, + "ok": h.ok, + "checked_at": now.isoformat(), + } for name, h in health.items() } -@router.get("/stats") +@router.get("/stats", deprecated=True) def stats(session: Session = Depends(repo.get_session)): """仪表盘统计卡数据：最近一次成功 run 的消息总数 / 发言人数。""" from sqlmodel import select @@ -65,7 +341,10 @@ def stats(session: Session = Depends(repo.get_session)): "run_id": None, } rows = session.exec( - select(GroupRun).where(GroupRun.run_id == latest.id) + select(GroupRun).where( + GroupRun.run_id == latest.id, + GroupRun.identity_state == "linked", + ) ).all() return { "total_messages": sum(r.message_count for r in rows), @@ -77,6 +356,8 @@ def stats(session: Session = Depends(repo.get_session)): @router.get("/status") def status(session: Session = Depends(repo.get_session), settings: Settings = Depends(get_settings)): + from app.scheduler.manager import get_scheduler + try: tz = ZoneInfo(settings.app_timezone) now = datetime.now(tz) @@ -85,21 +366,27 @@ def status(session: Session = Depends(repo.get_session), settings: Settings = De tz = None window = get_report_window(now.date(), settings.app_timezone) - next_run = "" - if window.should_run: - from datetime import time - - gen_time = settings.schedule_generate_time # HH:MM - hour, minute = (int(x) for x in gen_time.split(":")) + def next_daily_at(value: str, fallback: str) -> str: + try: + hour, minute = (int(x) for x in str(value).split(":")) + except (TypeError, ValueError): + hour, minute = (int(x) for x in fallback.split(":")) next_dt = now.replace(hour=hour, minute=minute, second=0, microsecond=0) if next_dt <= now: - next_dt = next_dt.replace(day=next_dt.day + 1) - next_run = next_dt.isoformat() + next_dt += timedelta(days=1) + return next_dt.isoformat() + + next_generate_at = next_daily_at(settings.schedule_generate_time, "00:15") + next_send_at = next_daily_at(settings.schedule_send_time, "08:30") groups = repo.list_groups(session, only_enabled=True) return { "version": "1.0.0", "status": "running", + "scheduler_owner": settings.scheduler_owner, + "scheduler_active": get_scheduler() is not None, + "legacy_v1_write_mode": settings.legacy_v1_write_mode, + "legacy_v1_writes_active": settings.legacy_v1_write_mode == "maintenance", "now": now.isoformat() if tz else None, "timezone": settings.app_timezone, "report_date": window.report_date.isoformat(), @@ -107,7 +394,8 @@ def status(session: Session = Depends(repo.get_session), settings: Settings = De "range_end": window.range_end.isoformat() if window.should_run else "", "should_run_today": window.should_run, "is_weekend_summary": window.is_weekend_summary, - "next_generate_at": next_run, + "next_generate_at": next_generate_at, + "next_send_at": next_send_at, "enabled_groups": len(groups), "total_groups": len(repo.list_groups(session)), } diff --git a/app/api/v2_recovery.py b/app/api/v2_recovery.py new file mode 100644 index 0000000..9cbb12f --- /dev/null +++ b/app/api/v2_recovery.py @@ -0,0 +1,77 @@ +"""V2 历史恢复预览与显式确认 API。""" + +from __future__ import annotations + +from fastapi import APIRouter, Depends, HTTPException, Query +from pydantic import BaseModel, Field + +from app.config.settings import Settings, get_settings +from app.scheduler.recovery_planner import ( + RecoveryPlanChangedError, + RecoveryPlanner, + RecoverySelectionError, +) +from app.services.generation_runtime import GenerationBusyError + +router = APIRouter(prefix="/recovery", tags=["v2-recovery"]) + + +class RecoverySelection(BaseModel): + run_date: str + group_id: int = Field(gt=0) + + +class RecoveryConfirmBody(BaseModel): + expected_version: str = Field(min_length=64, max_length=64) + tasks: list[RecoverySelection] = Field(min_length=1, max_length=180) + + +class EmptyManifestRepairBody(BaseModel): + run_date: str + expected_state_version: int = Field(ge=1) + expected_group_ids: list[int] = Field(min_length=1, max_length=100) + + +@router.get("/backlog") +def recovery_backlog( + lookback_days: int = Query(default=30, ge=3, le=30), + settings: Settings = Depends(get_settings), +): + return RecoveryPlanner(settings).preview(lookback_days=lookback_days) + + +@router.post("/confirm") +def confirm_recovery( + body: RecoveryConfirmBody, + settings: Settings = Depends(get_settings), +): + try: + return RecoveryPlanner(settings).confirm_generation( + [item.model_dump() for item in body.tasks], + expected_version=body.expected_version, + ) + except RecoveryPlanChangedError as exc: + raise HTTPException(status_code=409, detail=str(exc)) from exc + except RecoverySelectionError as exc: + raise HTTPException(status_code=400, detail=str(exc)) from exc + except GenerationBusyError as exc: + raise HTTPException(status_code=409, detail=str(exc)) from exc + + +@router.post("/repair-empty-manifest") +def repair_empty_manifest( + body: EmptyManifestRepairBody, + settings: Settings = Depends(get_settings), +): + try: + return RecoveryPlanner(settings).repair_empty_manifest_and_generate( + body.run_date, + expected_state_version=body.expected_state_version, + expected_group_ids=body.expected_group_ids, + ) + except RecoveryPlanChangedError as exc: + raise HTTPException(status_code=409, detail=str(exc)) from exc + except RecoverySelectionError as exc: + raise HTTPException(status_code=400, detail=str(exc)) from exc + except GenerationBusyError as exc: + raise HTTPException(status_code=409, detail=str(exc)) from exc diff --git a/app/api/v2_ui.py b/app/api/v2_ui.py index b3818ba..f4064de 100644 --- a/app/api/v2_ui.py +++ b/app/api/v2_ui.py @@ -1,590 +1,55 @@ -"""V2 管理 API（P8 前端依赖）。 - -Dashboard / 历史日报 / 系统健康 / Pipeline 手动操作 / 输出文件读取。 -""" +"""V2 管理 API 聚合入口（P8 前端依赖）。""" from __future__ import annotations -from datetime import datetime, time -from pathlib import Path -import shutil -from zoneinfo import ZoneInfo - -from fastapi import APIRouter, Depends, HTTPException -from fastapi.responses import FileResponse -from pydantic import BaseModel -from sqlmodel import Session - -from app.config.settings import Settings, get_settings -from app.db import repository as repo -from app.scheduler.period import PeriodResolver -from app.v2.constants import FILE_IMAGE -from app.v2.run_store import RunStore, validate_run_date - -router = APIRouter(prefix="/api/v2", tags=["v2-ui"]) - -# 允许前端直接读取的输出文件 -ALLOWED_FILES = frozenset( - { - "messages.json", - "ranking.json", - "ranking.txt", - "image_prompt.txt", - "image_prompt.original.txt", - "daily_image.png", - "daily_image.previous.png", - "run.json", - } +from fastapi import APIRouter, Depends, HTTPException, Request + +from app.api.v2_ui_common import ( + ALLOWED_FILES, + ImageThemeResolveBody, + PipelineGenerateBody, + PipelineSendBody, + ResetSendFailureBody, + ResolveManualSendBody, + ResolvePromptUnknownBody, + ResolveSendUnknownBody, + RetryBody, + RunPromptUpdateBody, + make_store as _store, + safe_group_dir as _safe_group_dir, + timezone_for as _tz, + validate_api_run_date as _validate_run_date, ) - - -class PipelineGenerateBody(BaseModel): - group_id: int | None = None - run_date: str | None = None - force: bool = False - refresh_messages: bool = False - - -class PipelineSendBody(BaseModel): - group_id: int - run_date: str | None = None - confirm_regenerated: bool = False - confirm_late_send: bool = False - - -class ImageThemeResolveBody(BaseModel): - image_theme: str - image_theme_custom: str = "" - prompt: str = "" - group_id: int | str | None = None - run_date: str = "" - - -class RunPromptUpdateBody(BaseModel): - content: str - expected_revision: str - image_theme: str - image_theme_custom: str = "" - - -def _tz(settings: Settings) -> ZoneInfo: - try: - return ZoneInfo(settings.app_timezone) - except Exception: - return ZoneInfo("Asia/Shanghai") - - -def _store(settings: Settings) -> RunStore: - return RunStore(settings.output_dir) - - -# ---------- Dashboard ---------- - - -@router.get("/dashboard") -def dashboard(session: Session = Depends(repo.get_session), settings: Settings = Depends(get_settings)): - tz = _tz(settings) - now = datetime.now(tz) - today = now.date().isoformat() - - resolver = PeriodResolver() - window = resolver.resolve(run_date=now.date(), timezone=settings.app_timezone) - store = _store(settings) - groups = repo.list_groups(session, only_enabled=True) - - cards: list[dict] = [] - counts = {"pending": 0, "generated": 0, "sent": 0, "failed": 0} - for g in groups: - name = g.display_name or g.wechat_group_name - run = store.load_run(name, today) - status = run.get("status", "PENDING") - image_path = store.image_path(name, today) - image_url = "" - if image_path.exists() and Path(image_path).stat().st_size > 0: - from urllib.parse import quote - - image_url = f"/api/v2/files/{quote(name)}/{today}/{FILE_IMAGE}" - cards.append( - { - "group_id": g.id, - "group_name": name, - "send_time": g.send_time, - "schedule_rule": g.schedule_rule, - "image_enabled": bool(g.image_enabled), - "wechat_send_enabled": bool(getattr(g, "wechat_send_enabled", False)), - "ranking_template": g.ranking_template, - "image_prompt_template": g.image_prompt_template, - "status": status, - "period_start": run.get("period_start", ""), - "period_end": run.get("period_end", ""), - "message_count": run.get("message_count", 0), - "speaker_count": run.get("speaker_count", 0), - "image_url": image_url, - "error": run.get("error") or run.get("image_error") or run.get("error_type") or "", - "sent_at": run.get("sent_at", ""), - "send_hold": bool(run.get("send_hold")), - "updated_at": run.get("updated_at", ""), - } - ) - if status in ("SENT",): - counts["sent"] += 1 - elif status in ("IMAGE_READY", "READY_TO_SEND"): - counts["generated"] += 1 - elif status == "FAILED": - counts["failed"] += 1 - else: - counts["pending"] += 1 - - # 下一次发送时间（最早到点且未发送的群） - next_send = "" - upcoming = [] - for card in cards: - if card["status"] not in ("IMAGE_READY", "READY_TO_SEND"): - continue - if card["sent_at"]: - continue - if not card["wechat_send_enabled"] or card["send_hold"]: - continue - try: - h, m = card["send_time"].split(":") - send_at = now.replace(hour=int(h), minute=int(m), second=0, microsecond=0) - except Exception: - send_at = now.replace(hour=8, minute=30, second=0, microsecond=0) - upcoming.append((send_at, card["group_name"])) - if upcoming: - earliest, name = min(upcoming, key=lambda x: x[0]) - next_send = f"{earliest.strftime('%H:%M')}（{name}）" - - return { - "today": today, - "should_run": window.should_run, - "period_start": window.period_start_str(), - "period_end": window.period_end_str(), - "enabled_groups": len(cards), - "counts": counts, - "next_send": next_send, - "cards": cards, - } - - -# ---------- 历史日报 ---------- - - -@router.get("/runs") -def list_runs(settings: Settings = Depends(get_settings), run_date: str | None = None): - if run_date is not None: - _validate_run_date(run_date) - runs = _store(settings).list_runs(run_date) - return {"runs": runs, "total": len(runs)} - - -_ARCHIVE_RUN_FIELDS = ( - "group_id", - "wechat_group_id", - "group_name", - "run_date", - "report_date", - "status", - "period_start", - "period_end", - "message_count", - "speaker_count", - "sent_at", - "error", - "error_type", - "image_error", - "updated_at", +from app.api.v2_ui_images import ( + get_run_prompt, + image_themes, + rebuild_run_prompt, + refresh_run_messages, + regenerate_run_image, + resolve_theme_preview, + restore_run_prompt, + router as image_router, + update_run_prompt, ) +from app.api.v2_ui_read import ( + archive_groups, + dashboard, + list_runs, + read_output_file, + router as read_router, + run_detail, +) +from app.config.settings import Settings, get_settings +from app.v2.constants import RUN_STATE_CORRUPT +from app.api.v2_recovery import router as recovery_router +from app.api.v2_weekly import router as weekly_router -def _archive_text(value: object) -> str: - return str(value).strip() if value is not None else "" - - -def _archive_local_id(value: object) -> int | None: - text = _archive_text(value) - return int(text) if text.isdigit() else None - - -def _archive_run_summary(run: dict) -> dict | None: - if not isinstance(run, dict): - return None - run_date = _archive_text(run.get("run_date")) - try: - validate_run_date(run_date) - except ValueError: - return None - return {field: run.get(field, "") for field in _ARCHIVE_RUN_FIELDS} - - -def _archive_group_entry(group) -> dict: - deleted_at = getattr(group, "deleted_at", None) - return { - "archive_key": f"group:{group.id}", - "group_id": group.id, - "wechat_group_id": group.wechat_group_id or "", - "display_name": group.display_name or group.wechat_group_name or f"群 {group.id}", - "state": "deleted" if deleted_at is not None else "active", - "enabled": bool(group.enabled) if deleted_at is None else False, - "deleted_at": deleted_at.isoformat() if deleted_at is not None else None, - "created_at": group.created_at.isoformat() if group.created_at else "", - "runs": [], - } - - -def _match_archive_group(run: dict, groups_by_id: dict, groups_by_wechat: dict): - raw_group_id = _archive_text(run.get("group_id")) - local_id = _archive_local_id(raw_group_id) - wechat_id = _archive_text(run.get("wechat_group_id")) - if not wechat_id and raw_group_id and local_id is None: - wechat_id = raw_group_id - - if local_id is not None and local_id in groups_by_id: - candidate = groups_by_id[local_id] - candidate_wechat = _archive_text(candidate.wechat_group_id) - if wechat_id and candidate_wechat and wechat_id != candidate_wechat: - return None - return candidate - - if wechat_id: - candidates = groups_by_wechat.get(wechat_id, []) - if len(candidates) == 1: - return candidates[0] - if len(candidates) > 1: - return None - - return None - - -def _orphan_archive_key(run: dict) -> str: - raw_group_id = _archive_text(run.get("group_id")) - wechat_id = _archive_text(run.get("wechat_group_id")) - if not wechat_id and raw_group_id and _archive_local_id(raw_group_id) is None: - wechat_id = raw_group_id - if wechat_id: - return f"orphan:wechat:{wechat_id}" - if raw_group_id: - return f"orphan:local:{raw_group_id}" - return f"orphan:name:{_archive_text(run.get('group_name')) or 'unknown'}" - - -@router.get("/archive/groups") -def archive_groups( - session: Session = Depends(repo.get_session), - settings: Settings = Depends(get_settings), -): - """按稳定群身份聚合 V2 归档；不返回任何本机绝对路径。""" - db_groups = repo.list_groups(session, include_deleted=True) - entries = {group.id: _archive_group_entry(group) for group in db_groups if group.id is not None} - groups_by_id = {group.id: group for group in db_groups if group.id is not None} - groups_by_wechat: dict[str, list] = {} - - for group in db_groups: - wechat_id = _archive_text(group.wechat_group_id) - if wechat_id: - groups_by_wechat.setdefault(wechat_id, []).append(group) - - orphan_entries: dict[str, dict] = {} - for raw_run in _store(settings).list_runs(): - run = _archive_run_summary(raw_run) - if run is None: - continue - group = _match_archive_group(raw_run, groups_by_id, groups_by_wechat) - if group is not None and group.id in entries: - entries[group.id]["runs"].append(run) - continue - - key = _orphan_archive_key(raw_run) - orphan = orphan_entries.setdefault( - key, - { - "archive_key": key, - "group_id": None, - "wechat_group_id": _archive_text(raw_run.get("wechat_group_id")), - "display_name": _archive_text(raw_run.get("group_name")) or "历史遗留群", - "state": "orphaned", - "enabled": False, - "deleted_at": None, - "created_at": "", - "runs": [], - }, - ) - orphan["runs"].append(run) - - for entry in [*entries.values(), *orphan_entries.values()]: - entry["runs"].sort( - key=lambda item: (_archive_text(item.get("run_date")), _archive_text(item.get("updated_at"))), - reverse=True, - ) - entry["run_count"] = len(entry["runs"]) - entry["run_dates"] = sorted( - {_archive_text(item.get("run_date")) for item in entry["runs"] if item.get("run_date")}, - reverse=True, - ) - - active = sorted( - (entry for entry in entries.values() if entry["state"] == "active"), - key=lambda entry: entry["group_id"], - ) - deleted = sorted( - (entry for entry in entries.values() if entry["state"] == "deleted"), - key=lambda entry: entry.get("deleted_at") or "", - reverse=True, - ) - orphaned = sorted( - orphan_entries.values(), - key=lambda entry: entry["runs"][0].get("run_date", "") if entry["runs"] else "", - reverse=True, - ) - return { - "groups": [*active, *deleted, *orphaned], - "active_count": len(active), - "trash_count": len(deleted) + len(orphaned), - } - - -@router.get("/runs/{group}/{run_date}") -def run_detail(group: str, run_date: str, settings: Settings = Depends(get_settings)): - _validate_run_date(run_date) - store = _store(settings) - run = store.load_run(group, run_date) - group_dir = store.group_dir(group, run_date) - files = ( - sorted(p.name for p in group_dir.glob("*") if p.is_file() and p.name in ALLOWED_FILES) - if group_dir.exists() - else [] - ) - return {"run": run, "files": files} - - -# ---------- 生图主题与 Prompt 编辑 ---------- - - -@router.get("/image-themes") -def image_themes(): - from app.ai.image_themes import public_image_theme_options - - return {"themes": public_image_theme_options()} - - -@router.post("/image-themes/resolve") -def resolve_theme_preview( - body: ImageThemeResolveBody, - session: Session = Depends(repo.get_session), - settings: Settings = Depends(get_settings), -): - from app.ai.image_themes import ImageThemeError, resolve_image_theme - from app.ai.prompt_editing import replace_theme_section, resolved_theme_text - - try: - if body.run_date: - _validate_run_date(body.run_date) - group_key = str(body.group_id or "preview") - group_name = group_key - if isinstance(body.group_id, int) or (isinstance(body.group_id, str) and body.group_id.isdigit()): - group = repo.get_active_group(session, int(body.group_id)) - if group is not None: - group_name = group.display_name or group.wechat_group_name or group_key - previous_signature = ( - _store(settings).previous_theme_signature(group_name, body.run_date) - if body.run_date and body.group_id is not None - else "" - ) - theme = resolve_image_theme( - body.image_theme, - body.image_theme_custom, - group_key=group_key, - run_date=body.run_date, - previous_signature=previous_signature, - ) - except ImageThemeError as exc: - raise HTTPException(status_code=422, detail=str(exc)) from exc - return { - "requested_key": theme.requested_key, - "actual_key": theme.actual_key, - "display_name": theme.display_name, - "theme_text": resolved_theme_text(theme), - "style_signature": theme.style_signature, - "style_seed": theme.style_seed, - "prompt": replace_theme_section(body.prompt, theme) if body.prompt else "", - } - - -def _read_run_prompt(store: RunStore, group: str, run_date: str) -> tuple[str, dict]: - if not store.run_path(group, run_date).exists(): - raise HTTPException(status_code=404, detail="运行记录不存在") - path = store.prompt_path(group, run_date) - if not path.is_file(): - raise HTTPException(status_code=404, detail="image_prompt.txt 不存在") - try: - content = path.read_text(encoding="utf-8") - except (OSError, UnicodeError) as exc: - raise HTTPException(status_code=500, detail=f"Prompt 读取失败：{exc}") from exc - return content, store.load_run(group, run_date) - - -@router.get("/runs/{group}/{run_date}/prompt") -def get_run_prompt(group: str, run_date: str, settings: Settings = Depends(get_settings)): - from app.ai.prompt_editing import prompt_revision - - _validate_run_date(run_date) - store = _store(settings) - content, run = _read_run_prompt(store, group, run_date) - prompt_meta = run.get("prompt_meta") if isinstance(run.get("prompt_meta"), dict) else {} - return { - "group_name": group, - "run_date": run_date, - "content": content, - "revision": prompt_revision(content), - "has_original": store.original_prompt_path(group, run_date).is_file(), - # random_preset 在 Prompt 构建时已经解析；运行级编辑必须继续使用当次固定主题。 - "image_theme": prompt_meta.get("requested_theme") or run.get("image_theme") or "random_preset", - "image_theme_custom": run.get("image_theme_custom") or "", - "prompt_edited_at": run.get("prompt_edited_at") or "", - "topic_selection": prompt_meta.get("topic_selection"), - } - - -@router.put("/runs/{group}/{run_date}/prompt") -def update_run_prompt( - group: str, - run_date: str, - body: RunPromptUpdateBody, - settings: Settings = Depends(get_settings), -): - from app.ai.image_themes import ImageThemeError, resolve_image_theme - from app.ai.prompt_editing import prompt_revision, replace_theme_section, validate_prompt_text - - _validate_run_date(run_date) - store = _store(settings) - current, current_run = _read_run_prompt(store, group, run_date) - if body.expected_revision != prompt_revision(current): - raise HTTPException(status_code=409, detail="Prompt 已被其他页面修改，请刷新后重试") - try: - current_meta = current_run.get("prompt_meta") if isinstance(current_run.get("prompt_meta"), dict) else None - theme = resolve_image_theme( - body.image_theme, - body.image_theme_custom, - group_key=str(current_run.get("group_id") or group), - run_date=run_date, - persisted_meta=current_meta, - ) - content = validate_prompt_text(replace_theme_section(body.content, theme)) - except (ImageThemeError, ValueError) as exc: - raise HTTPException(status_code=422, detail=str(exc)) from exc - - prompt_path = store.prompt_path(group, run_date) - original_path = store.original_prompt_path(group, run_date) - if not original_path.exists(): - shutil.copy2(prompt_path, original_path) - temp = prompt_path.with_suffix(".txt.tmp") - temp.write_text(content, encoding="utf-8") - temp.replace(prompt_path) - meta = theme.to_meta() - store.update( - group, - run_date, - image_theme=body.image_theme, - image_theme_custom=body.image_theme_custom.strip(), - prompt_meta={**(store.load_run(group, run_date).get("prompt_meta") or {}), **meta, "source": "manual"}, - prompt_edited_at=datetime.now().strftime("%Y-%m-%d %H:%M:%S"), - image_regen_status="prompt_saved", - image_regen_error="", - send_hold=True, - needs_manual_send=True, - ) - return get_run_prompt(group, run_date, settings) - - -@router.post("/runs/{group}/{run_date}/prompt/restore") -def restore_run_prompt(group: str, run_date: str, settings: Settings = Depends(get_settings)): - _validate_run_date(run_date) - store = _store(settings) - if not store.run_path(group, run_date).exists(): - raise HTTPException(status_code=404, detail="运行记录不存在") - original = store.original_prompt_path(group, run_date) - if not original.is_file(): - raise HTTPException(status_code=404, detail="没有可恢复的原始 Prompt") - shutil.copy2(original, store.prompt_path(group, run_date)) - store.update( - group, - run_date, - prompt_edited_at=datetime.now().strftime("%Y-%m-%d %H:%M:%S"), - image_regen_status="prompt_restored", - send_hold=True, - needs_manual_send=True, - ) - return get_run_prompt(group, run_date, settings) - - -@router.post("/runs/{group}/{run_date}/regenerate-image", status_code=202) -def regenerate_run_image(group: str, run_date: str, settings: Settings = Depends(get_settings)): - from app.image.regeneration import enqueue_regeneration - - _validate_run_date(run_date) - try: - run = enqueue_regeneration(settings, group, run_date) - except FileNotFoundError as exc: - raise HTTPException(status_code=404, detail=str(exc)) from exc - except RuntimeError as exc: - raise HTTPException(status_code=409, detail=str(exc)) from exc - return {"accepted": True, "run": run} - - -def _run_group_id(store: RunStore, group: str, run_date: str) -> int: - run = store.load_run(group, run_date) - try: - return int(run.get("group_id")) - except (TypeError, ValueError) as exc: - raise HTTPException(status_code=409, detail="运行记录缺少可用群 ID") from exc - - -@router.post("/runs/{group}/{run_date}/refresh-messages") -def refresh_run_messages(group: str, run_date: str, settings: Settings = Depends(get_settings)): - """更新当天消息和确定性排行榜；不会重建 Prompt、生图或发送。""" - from app.pipeline.daily_pipeline import DailyPipeline - from app.services.generation_runtime import GenerationBusyError - - _validate_run_date(run_date) - store = _store(settings) - if not store.run_path(group, run_date).exists(): - raise HTTPException(status_code=404, detail="运行记录不存在") - try: - result = DailyPipeline(settings=settings, dry_run=False).force_generate( - _run_group_id(store, group, run_date), - run_date, - refresh_messages=True, - ) - except GenerationBusyError as exc: - raise HTTPException(status_code=409, detail=str(exc)) from exc - if result.get("status") == "failed": - raise HTTPException(status_code=409, detail=result.get("detail") or result.get("error") or "消息刷新失败") - return {"result": result, "run": store.load_run(group, run_date)} - - -@router.post("/runs/{group}/{run_date}/rebuild-prompt") -def rebuild_run_prompt(group: str, run_date: str, settings: Settings = Depends(get_settings)): - """只从已保存的 messages.json 重建排行榜和 Prompt；不会取数或生图。""" - from app.pipeline.daily_pipeline import DailyPipeline - from app.services.generation_runtime import GenerationBusyError - - _validate_run_date(run_date) - store = _store(settings) - if not store.run_path(group, run_date).exists(): - raise HTTPException(status_code=404, detail="运行记录不存在") - try: - result = DailyPipeline(settings=settings, dry_run=False).rebuild_prompt_from_snapshot( - _run_group_id(store, group, run_date), - run_date, - ) - except GenerationBusyError as exc: - raise HTTPException(status_code=409, detail=str(exc)) from exc - if result.get("status") == "failed": - raise HTTPException(status_code=409, detail=result.get("detail") or result.get("error") or "Prompt 重建失败") - return {"result": result, "run": store.load_run(group, run_date)} - - -# ---------- 系统健康 ---------- +router = APIRouter(prefix="/api/v2", tags=["v2-ui"]) +router.include_router(read_router) +router.include_router(image_router) +router.include_router(recovery_router) +router.include_router(weekly_router) @router.get("/system/health") @@ -594,14 +59,20 @@ def system_health(settings: Settings = Depends(get_settings)): from app.data_sources.wechat_data_analysis import WeChatDataAnalysisSource source = WeChatDataAnalysisSource(settings=settings) - h = source.health_check() - checks["wechat_data_analysis"] = {"ok": h.ok, "status": h.status.value, "detail": h.detail} + health = source.health_check() + checks["wechat_data_analysis"] = { + "ok": health.ok, + "status": health.status.value, + "detail": health.detail, + } from app.providers.ai.codex import CodexGPTProvider codex_summary = CodexGPTProvider(settings) codex_summary_report = codex_summary.health_report() - codex_summary_ok, codex_summary_detail = codex_summary.health_check() + codex_summary_ok, codex_summary_detail = codex_summary.health_check( + codex_summary_report + ) checks["codex_summary"] = { "ok": codex_summary_ok, "status": "OK" if codex_summary_ok else "UNAVAILABLE", @@ -614,14 +85,17 @@ def system_health(settings: Settings = Depends(get_settings)): checks["deepseek_fallback"] = { "ok": bool(settings.ai_api_key), "status": "OK" if settings.ai_api_key else "UNAVAILABLE", - "detail": f"备用模型 {settings.ai_model}，API Key {'已配置' if settings.ai_api_key else '未配置'}", + "detail": ( + f"备用模型 {settings.ai_model}，" + f"API Key {'已配置' if settings.ai_api_key else '未配置'}" + ), } from app.image.codex_generator import CodexImageGenerator codex = CodexImageGenerator(settings=settings) codex_report = codex.health_report() - codex_ok, codex_detail = codex.health_check() + codex_ok, codex_detail = codex.health_check(codex_report) checks["codex_imagegen"] = { "ok": codex_ok, "status": "OK" if codex_ok else "UNAVAILABLE", @@ -631,51 +105,86 @@ def system_health(settings: Settings = Depends(get_settings)): "last_image_smoke": codex_report["last_image_smoke"], } - from app.sender.wechat_native import create_wechat_sender + from app.sender.wechat_native import WechatNativeSender, create_wechat_sender sender = create_wechat_sender(settings=settings) - send_ok, send_detail = sender.health_check() - sender_report = getattr(sender, "health_report", lambda: {"ok": send_ok})() + if isinstance(sender, WechatNativeSender): + sender_report = sender.health_report() + send_ok, send_detail = sender.health_check(sender_report) + else: + send_ok, send_detail = sender.health_check() + sender_report = {"ok": send_ok} checks["wechat_sender"] = { "ok": send_ok, "status": "OK" if send_ok else "UNAVAILABLE", "detail": send_detail, - "dependencies": sender_report.get("dependencies", {"ok": send_ok, "detail": send_detail}), - "desktop": sender_report.get("desktop", {"ok": send_ok, "detail": send_detail}), - "ocr": sender_report.get("ocr", {"ok": send_ok, "detail": send_detail}), - "clipboard": sender_report.get("clipboard", {"ok": send_ok, "detail": send_detail}), - "window": sender_report.get("window", {"ok": send_ok, "detail": send_detail}), + "dependencies": sender_report.get( + "dependencies", + {"ok": send_ok, "detail": send_detail}, + ), + "desktop": sender_report.get( + "desktop", + {"ok": send_ok, "detail": send_detail}, + ), + "ocr": sender_report.get( + "ocr", + {"ok": send_ok, "detail": send_detail}, + ), + "clipboard": sender_report.get( + "clipboard", + {"ok": send_ok, "detail": send_detail}, + ), + "window": sender_report.get( + "window", + {"ok": send_ok, "detail": send_detail}, + ), } - # output 可写 try: settings.ensure_dirs() - test = settings.output_dir / ".write_test" - test.write_text("ok", encoding="utf-8") - test.unlink() - checks["output"] = {"ok": True, "status": "OK", "detail": "output 目录可写"} - except Exception as e: - checks["output"] = {"ok": False, "status": "UNAVAILABLE", "detail": str(e)[:200]} - - # 模板完整性 + test_path = settings.output_dir / ".write_test" + test_path.write_text("ok", encoding="utf-8") + test_path.unlink() + checks["output"] = { + "ok": True, + "status": "OK", + "detail": "output 目录可写", + } + except Exception as exc: + checks["output"] = { + "ok": False, + "status": "UNAVAILABLE", + "detail": str(exc)[:200], + } + from app.ranking.template_service import RankingTemplateService try: ranking_names = RankingTemplateService().list_templates() - checks["templates"] = {"ok": True, "status": "OK", "detail": f"排行榜模板 {ranking_names}"} - except Exception as e: - checks["templates"] = {"ok": False, "status": "UNAVAILABLE", "detail": str(e)[:200]} + checks["templates"] = { + "ok": True, + "status": "OK", + "detail": f"排行榜模板 {ranking_names}", + } + except Exception as exc: + checks["templates"] = { + "ok": False, + "status": "UNAVAILABLE", + "detail": str(exc)[:200], + } - # 最近一次完整任务 + 未完成任务 store = _store(settings) runs = store.list_runs() last_run = runs[0] if runs else {} - incomplete = sum(1 for r in runs if r.get("status") not in ("SENT", "FAILED")) + incomplete = sum( + 1 for run in runs if run.get("status") not in ("SENT", "FAILED") + ) checks["recent_task"] = { "ok": True, "status": "OK", "detail": ( - f"最近：{last_run.get('group_name', '—')} {last_run.get('run_date', '—')} " + f"最近：{last_run.get('group_name', '—')} " + f"{last_run.get('run_date', '—')} " f"{last_run.get('status', '—')}；未完成任务 {incomplete} 个" ), } @@ -683,14 +192,13 @@ def system_health(settings: Settings = Depends(get_settings)): return {"checks": checks, "warnings": _environment_warnings()} -# ---------- 启动检查 / 恢复（P9） ---------- - - @router.get("/system/startup") -def startup_checks(settings: Settings = Depends(get_settings)): - from app.core.startup_check import run_startup_checks - - return {"checks": run_startup_checks(settings)} +def startup_checks(request: Request): + """返回服务启动时保存的检查快照，不在 GET 请求中重新探测外部依赖。""" + return { + "checks": list(getattr(request.app.state, "startup_checks", [])), + "error": str(getattr(request.app.state, "startup_check_error", "") or ""), + } @router.get("/system/recovery") @@ -698,62 +206,99 @@ def recovery_info(settings: Settings = Depends(get_settings)): from app.v2.recovery import scan_incomplete, verify_output store = _store(settings) + runs = store.list_runs() return { - "incomplete": scan_incomplete(store), - "integrity": verify_output(store), + "incomplete": scan_incomplete(store, runs=runs), + "integrity": verify_output(store, runs=runs), } -class RetryBody(BaseModel): - group_id: int | None = None - run_date: str | None = None - - @router.post("/pipeline/retry-failed") -def retry_failed(body: RetryBody | None = None, settings: Settings = Depends(get_settings)): +def retry_failed( + body: RetryBody | None = None, + settings: Settings = Depends(get_settings), +): from app.pipeline.daily_pipeline import DailyPipeline from app.v2.recovery import scan_incomplete body = body or RetryBody() if body.run_date is not None: _validate_run_date(body.run_date) - pipeline = DailyPipeline() if body.group_id: - r = pipeline.force_generate(body.group_id, body.run_date) - return {"results": [{"group_id": body.group_id, "status": r.get("status"), "detail": r.get("error") or ""}]} + pipeline = DailyPipeline() + result = pipeline.force_generate(body.group_id, body.run_date) + return { + "results": [ + { + "group_id": body.group_id, + "status": result.get("status"), + "detail": result.get("error") or "", + } + ] + } + incomplete = scan_incomplete(_store(settings), body.run_date) results: list[dict] = [] + pipeline = None for run in incomplete: group_name = run["group_name"] + if run.get("recovery_type") == "manual_review": + error_type = run.get("error_type") or RUN_STATE_CORRUPT + results.append( + { + "group_name": group_name, + "status": "blocked", + "error_type": error_type, + "detail": ( + "AI 调用结果未知，需人工复核" + if error_type == "PROMPT_RESULT_UNKNOWN" + else "微信发送结果未知，需人工核对后消歧" + if error_type == "SEND_RESULT_UNKNOWN" + else "运行状态文件损坏，需人工复核" + ), + } + ) + continue + if pipeline is None: + pipeline = DailyPipeline() + from sqlmodel import Session, select from app.db import repository as repo from app.db.models import Group with Session(repo.engine) as session: - group = session.exec(select(Group).where(Group.display_name == group_name)).first() + group = session.exec( + select(Group).where(Group.display_name == group_name) + ).first() if group is None: - results.append({"group_name": group_name, "status": "skipped", "detail": "群不存在/已停用"}) + results.append( + { + "group_name": group_name, + "status": "skipped", + "detail": "群不存在/已停用", + } + ) continue if run.get("recovery_type") == "send": - r = pipeline.force_send(group.id, run["run_date"]) + result = pipeline.force_send(group.id, run["run_date"]) else: - r = pipeline.force_generate(group.id, run["run_date"]) - results.append({"group_name": group_name, "status": r.get("status"), "detail": r.get("error") or ""}) + result = pipeline.force_generate(group.id, run["run_date"]) + results.append( + { + "group_name": group_name, + "status": result.get("status"), + "detail": result.get("error") or "", + } + ) return {"results": results} def _environment_warnings() -> list[str]: - """P9：休眠/锁屏等无人值守风险提示。""" - warnings: list[str] = [] - warnings.append( + return [ "休眠/锁屏风险：GroupBrief 自动发送依赖桌面会话。请保持电脑开机、不休眠、不锁屏；" "Windows 电源设置请关闭自动休眠与自动锁屏。" - ) - return warnings - - -# ---------- Pipeline 手动操作 ---------- + ] @router.post("/pipeline/generate") @@ -807,27 +352,75 @@ def pipeline_send(body: PipelineSendBody): return {"result": result} -# ---------- 输出文件读取 ---------- +@router.post("/pipeline/resolve-send-unknown") +def pipeline_resolve_send_unknown(body: ResolveSendUnknownBody): + from app.pipeline.daily_pipeline import DailyPipeline + _validate_run_date(body.run_date) + result = DailyPipeline(dry_run=False).resolve_send_unknown( + body.group_id, + body.run_date, + resolution=body.resolution, + expected_send_unknown_at=body.expected_send_unknown_at, + ) + if result.get("status") == "conflict": + raise HTTPException(status_code=409, detail=result) + if result.get("status") == "failed": + raise HTTPException(status_code=400, detail=result) + return {"result": result} -@router.get("/files/{group}/{run_date}/{file_name}") -def read_output_file( - group: str, - run_date: str, - file_name: str, - settings: Settings = Depends(get_settings), -): - _validate_run_date(run_date) - if file_name not in ALLOWED_FILES: - raise HTTPException(400, f"不允许访问的文件：{file_name}") - path = _store(settings).group_dir(group, run_date) / file_name - if not path.exists(): - raise HTTPException(404, "文件不存在") - return FileResponse(path, filename=file_name) +@router.post("/pipeline/reset-send-failure") +def pipeline_reset_send_failure(body: ResetSendFailureBody): + """解除明确未提交的最终失败；该接口不会调用微信 Sender。""" + from app.pipeline.daily_pipeline import DailyPipeline -def _validate_run_date(run_date: str) -> str: - try: - return validate_run_date(run_date) - except ValueError as exc: - raise HTTPException(status_code=400, detail=str(exc)) from exc + _validate_run_date(body.run_date) + result = DailyPipeline(dry_run=False).reset_explicit_send_failure( + body.group_id, + body.run_date, + expected_updated_at=body.expected_updated_at, + expected_state_version=body.expected_state_version, + ) + if result.get("status") == "conflict": + raise HTTPException(status_code=409, detail=result) + if result.get("status") == "failed": + raise HTTPException(status_code=400, detail=result) + return {"result": result} + + +@router.post("/pipeline/resolve-prompt-unknown") +def pipeline_resolve_prompt_unknown(body: ResolvePromptUnknownBody): + """解除 Prompt 结果未知暂停；该接口本身不会调用外部模型。""" + from app.pipeline.daily_pipeline import DailyPipeline + + _validate_run_date(body.run_date) + result = DailyPipeline(dry_run=False).resolve_prompt_unknown( + body.group_id, + body.run_date, + expected_operation_id=body.expected_operation_id, + ) + if result.get("status") == "conflict": + raise HTTPException(status_code=409, detail=result) + if result.get("status") == "failed": + raise HTTPException(status_code=400, detail=result) + return {"result": result} + + +@router.post("/pipeline/resolve-manual-send") +def pipeline_resolve_manual_send(body: ResolveManualSendBody): + """记录人工核对结论；该接口不会调用微信 Sender。""" + from app.pipeline.daily_pipeline import DailyPipeline + + _validate_run_date(body.run_date) + result = DailyPipeline(dry_run=False).resolve_manual_send( + body.group_id, + body.run_date, + resolution=body.resolution, + expected_updated_at=body.expected_updated_at, + ) + if result.get("status") == "conflict": + raise HTTPException(status_code=409, detail=result) + if result.get("status") == "failed": + raise HTTPException(status_code=400, detail=result) + return {"result": result} diff --git a/app/api/v2_ui_common.py b/app/api/v2_ui_common.py new file mode 100644 index 0000000..fff01a6 --- /dev/null +++ b/app/api/v2_ui_common.py @@ -0,0 +1,129 @@ +"""V2 管理 API 的共享请求模型与路径边界。""" + +from __future__ import annotations + +from pathlib import Path +from typing import Literal +from zoneinfo import ZoneInfo + +from fastapi import HTTPException +from pydantic import BaseModel, Field + +from app.config.settings import Settings +from app.v2.run_store import RunStore, validate_run_date + + +ALLOWED_FILES = frozenset( + { + "messages.json", + "ranking.json", + "ranking.txt", + "image_prompt.txt", + "image_prompt.original.txt", + "daily_image.png", + "daily_image.previous.png", + "run.json", + } +) + + +class PipelineGenerateBody(BaseModel): + group_id: int | None = None + run_date: str | None = None + force: bool = False + refresh_messages: bool = False + + +class PipelineSendBody(BaseModel): + group_id: int + run_date: str | None = None + confirm_regenerated: bool = False + confirm_late_send: bool = False + + +class ResolveSendUnknownBody(BaseModel): + group_id: int + run_date: str + resolution: Literal["text_sent", "not_sent"] + expected_send_unknown_at: str = Field(min_length=10, max_length=80) + + +class ResolvePromptUnknownBody(BaseModel): + group_id: int + run_date: str + expected_operation_id: str = Field(min_length=8, max_length=80) + + +class ResetSendFailureBody(BaseModel): + group_id: int + run_date: str + expected_updated_at: str = Field(min_length=10, max_length=80) + expected_state_version: int = Field(ge=1) + + +class ResolveManualSendBody(BaseModel): + group_id: int + run_date: str + resolution: Literal["all_sent", "text_sent", "not_sent"] + expected_updated_at: str = Field(min_length=10, max_length=80) + + +class ImageThemeResolveBody(BaseModel): + image_theme: str + image_theme_custom: str = "" + prompt: str = "" + group_id: int | str | None = None + run_date: str = "" + + +class RunPromptUpdateBody(BaseModel): + content: str + expected_revision: str + image_theme: str + image_theme_custom: str = "" + + +class StableRunTarget(BaseModel): + group_id: int + wechat_group_id: str + run_date: str + + +class BatchRunTargetsBody(BaseModel): + targets: list[StableRunTarget] = Field(min_length=1, max_length=50) + + +class CandidateClaimBody(BaseModel): + job_id: str = Field(min_length=8, max_length=64) + candidate_id: str = Field(min_length=64, max_length=64) + + +class RetryBody(BaseModel): + group_id: int | None = None + run_date: str | None = None + + +def timezone_for(settings: Settings) -> ZoneInfo: + try: + return ZoneInfo(settings.app_timezone) + except Exception: + return ZoneInfo("Asia/Shanghai") + + +def make_store(settings: Settings) -> RunStore: + return RunStore(settings.output_dir) + + +def safe_group_dir(store: RunStore, group: str, run_date: str) -> Path: + """把 RunStore 的路径拒绝统一转换成明确的客户端错误。""" + try: + return store.group_dir(group, run_date) + except ValueError as exc: + raise HTTPException(status_code=400, detail=str(exc)) from exc + + +def validate_api_run_date(run_date: str) -> str: + try: + return validate_run_date(run_date) + except ValueError as exc: + raise HTTPException(status_code=400, detail=str(exc)) from exc diff --git a/app/api/v2_ui_images.py b/app/api/v2_ui_images.py new file mode 100644 index 0000000..870a5dc --- /dev/null +++ b/app/api/v2_ui_images.py @@ -0,0 +1,496 @@ +"""V2 生图主题、Prompt 编辑与运行级图片命令。""" + +from __future__ import annotations + +from datetime import datetime +import shutil +from urllib.parse import quote + +from fastapi import APIRouter, Depends, HTTPException +from fastapi.responses import FileResponse +from sqlmodel import Session + +from app.api.v2_ui_common import ( + BatchRunTargetsBody, + CandidateClaimBody, + ImageThemeResolveBody, + RunPromptUpdateBody, + make_store as _store, + safe_group_dir as _safe_group_dir, + validate_api_run_date as _validate_run_date, +) +from app.config.settings import Settings, get_settings +from app.db import repository as repo +from app.v2.run_store import RunStore + + +router = APIRouter() + + +@router.get("/image-themes") +def image_themes(): + from app.ai.image_themes import public_image_theme_options + + return {"themes": public_image_theme_options()} + + +@router.post("/image-themes/resolve") +def resolve_theme_preview( + body: ImageThemeResolveBody, + session: Session = Depends(repo.get_session), + settings: Settings = Depends(get_settings), +): + from app.ai.image_themes import ImageThemeError, resolve_image_theme + from app.ai.prompt_editing import replace_theme_section, resolved_theme_text + + try: + if body.run_date: + _validate_run_date(body.run_date) + group_key = str(body.group_id or "preview") + group_name = group_key + if isinstance(body.group_id, int) or ( + isinstance(body.group_id, str) and body.group_id.isdigit() + ): + group = repo.get_active_group(session, int(body.group_id)) + if group is not None: + group_name = ( + group.display_name or group.wechat_group_name or group_key + ) + previous_signature = ( + _store(settings).previous_theme_signature(group_name, body.run_date) + if body.run_date and body.group_id is not None + else "" + ) + theme = resolve_image_theme( + body.image_theme, + body.image_theme_custom, + group_key=group_key, + run_date=body.run_date, + previous_signature=previous_signature, + ) + except ImageThemeError as exc: + raise HTTPException(status_code=422, detail=str(exc)) from exc + except ValueError as exc: + raise HTTPException(status_code=400, detail=str(exc)) from exc + return { + "requested_key": theme.requested_key, + "actual_key": theme.actual_key, + "display_name": theme.display_name, + "theme_text": resolved_theme_text(theme), + "style_signature": theme.style_signature, + "style_seed": theme.style_seed, + "prompt": replace_theme_section(body.prompt, theme) if body.prompt else "", + } + + +def _read_run_prompt( + store: RunStore, + group: str, + run_date: str, +) -> tuple[str, dict]: + _safe_group_dir(store, group, run_date) + if not store.run_path(group, run_date).exists(): + raise HTTPException(status_code=404, detail="运行记录不存在") + path = store.prompt_path(group, run_date) + if not path.is_file(): + raise HTTPException(status_code=404, detail="image_prompt.txt 不存在") + try: + content = path.read_text(encoding="utf-8") + except (OSError, UnicodeError) as exc: + raise HTTPException(status_code=500, detail=f"Prompt 读取失败：{exc}") from exc + return content, store.load_run(group, run_date) + + +@router.get("/runs/{group}/{run_date}/prompt") +def get_run_prompt( + group: str, + run_date: str, + settings: Settings = Depends(get_settings), +): + from app.ai.prompt_editing import prompt_revision + + _validate_run_date(run_date) + store = _store(settings) + content, run = _read_run_prompt(store, group, run_date) + prompt_meta = ( + run.get("prompt_meta") if isinstance(run.get("prompt_meta"), dict) else {} + ) + return { + "group_name": group, + "run_date": run_date, + "content": content, + "revision": prompt_revision(content), + "has_original": store.original_prompt_path(group, run_date).is_file(), + "image_theme": ( + prompt_meta.get("requested_theme") + or run.get("image_theme") + or "ai_free" + ), + "image_theme_custom": run.get("image_theme_custom") or "", + "prompt_edited_at": run.get("prompt_edited_at") or "", + "topic_selection": prompt_meta.get("topic_selection"), + } + + +@router.put("/runs/{group}/{run_date}/prompt") +def update_run_prompt( + group: str, + run_date: str, + body: RunPromptUpdateBody, + settings: Settings = Depends(get_settings), +): + from app.ai.image_themes import ImageThemeError, resolve_image_theme + from app.ai.prompt_editing import ( + prompt_revision, + replace_theme_section, + validate_prompt_text, + ) + + _validate_run_date(run_date) + store = _store(settings) + current, current_run = _read_run_prompt(store, group, run_date) + if body.expected_revision != prompt_revision(current): + raise HTTPException( + status_code=409, + detail="Prompt 已被其他页面修改，请刷新后重试", + ) + try: + current_meta = ( + current_run.get("prompt_meta") + if isinstance(current_run.get("prompt_meta"), dict) + else None + ) + theme = resolve_image_theme( + body.image_theme, + body.image_theme_custom, + group_key=str(current_run.get("group_id") or group), + run_date=run_date, + persisted_meta=current_meta, + ) + topic_selection = ( + current_meta.get("topic_selection") + if isinstance(current_meta, dict) + else None + ) + expected_panel_count = ( + int(topic_selection.get("selected_count") or 0) + if isinstance(topic_selection, dict) + else None + ) + content = validate_prompt_text( + replace_theme_section(body.content, theme), + expected_panel_count=expected_panel_count or None, + ) + except (ImageThemeError, ValueError) as exc: + raise HTTPException(status_code=422, detail=str(exc)) from exc + + prompt_path = store.prompt_path(group, run_date) + original_path = store.original_prompt_path(group, run_date) + if not original_path.exists(): + shutil.copy2(prompt_path, original_path) + temp = prompt_path.with_suffix(".txt.tmp") + temp.write_text(content, encoding="utf-8") + temp.replace(prompt_path) + meta = theme.to_meta() + store.update( + group, + run_date, + image_theme=body.image_theme, + image_theme_custom=body.image_theme_custom.strip(), + prompt_meta={ + **(store.load_run(group, run_date).get("prompt_meta") or {}), + **meta, + "source": "manual", + }, + prompt_edited_at=datetime.now().strftime("%Y-%m-%d %H:%M:%S"), + image_regen_status="prompt_saved", + image_regen_error="", + send_hold=True, + needs_manual_send=True, + ) + return get_run_prompt(group, run_date, settings) + + +@router.post("/runs/{group}/{run_date}/prompt/restore") +def restore_run_prompt( + group: str, + run_date: str, + settings: Settings = Depends(get_settings), +): + _validate_run_date(run_date) + store = _store(settings) + _safe_group_dir(store, group, run_date) + if not store.run_path(group, run_date).exists(): + raise HTTPException(status_code=404, detail="运行记录不存在") + original = store.original_prompt_path(group, run_date) + if not original.is_file(): + raise HTTPException(status_code=404, detail="没有可恢复的原始 Prompt") + shutil.copy2(original, store.prompt_path(group, run_date)) + store.update( + group, + run_date, + prompt_edited_at=datetime.now().strftime("%Y-%m-%d %H:%M:%S"), + image_regen_status="prompt_restored", + send_hold=True, + needs_manual_send=True, + ) + return get_run_prompt(group, run_date, settings) + + +@router.post("/runs/{group}/{run_date}/regenerate-image", status_code=202) +def regenerate_run_image( + group: str, + run_date: str, + settings: Settings = Depends(get_settings), +): + from app.image.regeneration import enqueue_regeneration + + _validate_run_date(run_date) + _safe_group_dir(_store(settings), group, run_date) + try: + run = enqueue_regeneration(settings, group, run_date) + except FileNotFoundError as exc: + raise HTTPException(status_code=404, detail=str(exc)) from exc + except RuntimeError as exc: + raise HTTPException(status_code=409, detail=str(exc)) from exc + return {"accepted": True, "run": run} + + +@router.get("/runs/{group}/{run_date}/image-candidates") +def get_run_image_candidates( + group: str, + run_date: str, + settings: Settings = Depends(get_settings), +): + from app.image.regeneration import list_regeneration_candidates + + _validate_run_date(run_date) + _safe_group_dir(_store(settings), group, run_date) + candidates = list_regeneration_candidates(settings, group, run_date) + return { + "candidates": [ + { + **candidate, + "preview_url": ( + f"/api/v2/runs/{quote(group, safe='')}/{run_date}/image-candidates/" + f"{candidate['candidate_id']}" + ), + } + for candidate in candidates + ] + } + + +@router.get("/runs/{group}/{run_date}/image-candidates/{candidate_id}") +def preview_run_image_candidate( + group: str, + run_date: str, + candidate_id: str, + settings: Settings = Depends(get_settings), +): + from app.image.regeneration import candidate_preview_path + + _validate_run_date(run_date) + _safe_group_dir(_store(settings), group, run_date) + try: + path = candidate_preview_path(settings, group, run_date, candidate_id) + except FileNotFoundError as exc: + raise HTTPException(status_code=404, detail=str(exc)) from exc + except ValueError as exc: + raise HTTPException(status_code=400, detail=str(exc)) from exc + return FileResponse(path) + + +@router.post("/runs/{group}/{run_date}/image-candidates/claim") +def claim_run_image_candidate( + group: str, + run_date: str, + body: CandidateClaimBody, + settings: Settings = Depends(get_settings), +): + from app.image.regeneration import claim_regeneration_candidate + + _validate_run_date(run_date) + _safe_group_dir(_store(settings), group, run_date) + try: + run = claim_regeneration_candidate( + settings, + group, + run_date, + job_id=body.job_id, + candidate_id=body.candidate_id, + ) + except FileNotFoundError as exc: + raise HTTPException(status_code=404, detail=str(exc)) from exc + except ValueError as exc: + raise HTTPException(status_code=409, detail=str(exc)) from exc + return {"claimed": True, "run": run} + + +def _run_group_id(store: RunStore, group: str, run_date: str) -> int: + run = store.load_run(group, run_date) + try: + return int(run.get("group_id")) + except (TypeError, ValueError) as exc: + raise HTTPException(status_code=409, detail="运行记录缺少可用群 ID") from exc + + +@router.post("/runs/{group}/{run_date}/refresh-messages") +def refresh_run_messages( + group: str, + run_date: str, + settings: Settings = Depends(get_settings), +): + """更新当天消息和确定性排行榜；不会重建 Prompt、生图或发送。""" + from app.pipeline.daily_pipeline import DailyPipeline + from app.services.generation_runtime import GenerationBusyError + + _validate_run_date(run_date) + store = _store(settings) + _safe_group_dir(store, group, run_date) + if not store.run_path(group, run_date).exists(): + raise HTTPException(status_code=404, detail="运行记录不存在") + try: + result = DailyPipeline(settings=settings, dry_run=False).force_generate( + _run_group_id(store, group, run_date), + run_date, + refresh_messages=True, + ) + except GenerationBusyError as exc: + raise HTTPException(status_code=409, detail=str(exc)) from exc + if result.get("status") == "failed": + raise HTTPException( + status_code=409, + detail=( + result.get("detail") + or result.get("error") + or "消息刷新失败" + ), + ) + return {"result": result, "run": store.load_run(group, run_date)} + + +@router.post("/runs/{group}/{run_date}/rebuild-prompt") +def rebuild_run_prompt( + group: str, + run_date: str, + settings: Settings = Depends(get_settings), +): + """只从已保存的 messages.json 重建排行榜和 Prompt；不会取数或生图。""" + from app.pipeline.daily_pipeline import DailyPipeline + from app.services.generation_runtime import GenerationBusyError + + _validate_run_date(run_date) + store = _store(settings) + _safe_group_dir(store, group, run_date) + if not store.run_path(group, run_date).exists(): + raise HTTPException(status_code=404, detail="运行记录不存在") + try: + result = DailyPipeline( + settings=settings, + dry_run=False, + ).rebuild_prompt_from_snapshot( + _run_group_id(store, group, run_date), + run_date, + ) + except GenerationBusyError as exc: + raise HTTPException(status_code=409, detail=str(exc)) from exc + if result.get("status") == "failed": + raise HTTPException( + status_code=409, + detail=( + result.get("detail") + or result.get("error") + or "Prompt 重建失败" + ), + ) + return {"result": result, "run": store.load_run(group, run_date)} + + +@router.post("/runs/batch/rebuild-prompts") +def rebuild_run_prompts_batch( + body: BatchRunTargetsBody, + settings: Settings = Depends(get_settings), +): + """六群等批量场景：只复用快照和已校验选题，受控并行重建 Prompt。""" + from app.pipeline.daily_pipeline import DailyPipeline + from app.services.generation_runtime import GenerationBusyError + + targets: list[tuple[int, str, str]] = [] + for target in body.targets: + targets.append( + ( + target.group_id, + target.wechat_group_id.strip(), + _validate_run_date(target.run_date), + ) + ) + try: + results = DailyPipeline(settings=settings, dry_run=False).rebuild_prompts_from_snapshots( + targets + ) + except GenerationBusyError as exc: + raise HTTPException(status_code=409, detail=str(exc)) from exc + return {"results": results} + + +@router.post("/runs/batch/regenerate-images", status_code=202) +def regenerate_run_images_batch( + body: BatchRunTargetsBody, + settings: Settings = Depends(get_settings), +): + """按稳定身份提交多张重画；执行器和 Codex 侧共同限制为最多 2 路。""" + from app.image.regeneration import enqueue_regeneration + + store = _store(settings) + accepted: list[dict] = [] + for target in body.targets: + run_date = _validate_run_date(target.run_date) + matching = [ + run + for run in store.list_runs(run_date) + if str(run.get("group_id") or "") == str(target.group_id) + and str(run.get("wechat_group_id") or "") == target.wechat_group_id.strip() + ] + if len(matching) != 1: + accepted.append( + { + "group_id": target.group_id, + "wechat_group_id": target.wechat_group_id, + "run_date": run_date, + "accepted": False, + "detail": "无法用 group_id 与 wechat_group_id 唯一锁定运行", + } + ) + continue + group_name = str(matching[0].get("group_name") or "") + try: + run = enqueue_regeneration( + settings, + group_name, + run_date, + expected_group_id=target.group_id, + expected_wechat_group_id=target.wechat_group_id, + ) + accepted.append( + { + "group_id": target.group_id, + "wechat_group_id": target.wechat_group_id, + "group_name": group_name, + "run_date": run_date, + "accepted": True, + "job": run.get("image_regen_job") or {}, + } + ) + except (FileNotFoundError, RuntimeError, ValueError) as exc: + accepted.append( + { + "group_id": target.group_id, + "wechat_group_id": target.wechat_group_id, + "group_name": group_name, + "run_date": run_date, + "accepted": False, + "detail": str(exc), + } + ) + return {"results": accepted} diff --git a/app/api/v2_ui_read.py b/app/api/v2_ui_read.py new file mode 100644 index 0000000..074037c --- /dev/null +++ b/app/api/v2_ui_read.py @@ -0,0 +1,467 @@ +"""V2 Dashboard、归档、运行详情与输出文件查询。""" + +from __future__ import annotations + +import json +from datetime import date, datetime +from pathlib import Path +from urllib.parse import quote + +from fastapi import APIRouter, Depends, HTTPException, Query +from fastapi.responses import FileResponse +from sqlmodel import Session + +from app.api.v2_ui_common import ( + ALLOWED_FILES, + make_store as _store, + safe_group_dir as _safe_group_dir, + timezone_for as _tz, + validate_api_run_date as _validate_run_date, +) +from app.config.settings import Settings, get_settings +from app.db import repository as repo +from app.scheduler.period import PeriodResolver +from app.scheduler.runtime_status import build_daily_status +from app.services.runtime_logs import read_runtime_logs +from app.v2.constants import FILE_IMAGE +from app.v2.run_store import validate_run_date + + +router = APIRouter() + + +@router.get("/dashboard") +def dashboard( + session: Session = Depends(repo.get_session), + settings: Settings = Depends(get_settings), + run_date: str | None = None, +): + tz = _tz(settings) + now = datetime.now(tz) + selected_date = now.date() + if run_date is not None: + selected_date = date.fromisoformat(_validate_run_date(run_date)) + selected_run_date = selected_date.isoformat() + window = PeriodResolver().resolve( + run_date=selected_date, + timezone=settings.app_timezone, + ) + store = _store(settings) + groups = repo.list_groups(session, only_enabled=True) + + cards: list[dict] = [] + runtime_runs: list[dict] = [] + counts = {"pending": 0, "generated": 0, "sent": 0, "failed": 0, "held": 0} + for group in groups: + name = group.display_name or group.wechat_group_name + run = store.load_run(name, selected_run_date) + runtime_run = dict(run) + runtime_run.setdefault("group_id", str(group.id or "")) + runtime_run.setdefault("group_name", name) + runtime_runs.append(runtime_run) + status = run.get("status", "PENDING") + image_path = store.image_path(name, selected_run_date) + image_url = "" + if image_path.exists() and Path(image_path).stat().st_size > 0: + image_url = f"/api/v2/files/{quote(name)}/{selected_run_date}/{FILE_IMAGE}" + ranking_preview: list[dict[str, object]] = [] + ranking_error = "" + ranking_path = store.ranking_json_path(name, selected_run_date) + if ranking_path.exists() and ranking_path.stat().st_size > 0: + try: + ranking = json.loads(ranking_path.read_text(encoding="utf-8")) + speakers = ranking.get("top_speakers", []) if isinstance(ranking, dict) else [] + if not isinstance(speakers, list): + raise ValueError("top_speakers 不是数组") + for index, item in enumerate(speakers[:5], start=1): + if not isinstance(item, dict): + continue + speaker_name = str(item.get("name") or "").strip() + count = item.get("count") + if not speaker_name or not isinstance(count, (int, float)): + continue + ranking_preview.append( + { + "rank": int(item.get("rank") or index), + "name": speaker_name, + "count": int(count), + "text_count": int(item.get("text_count") or 0), + "interaction_count": int( + item.get("interaction_count") or 0 + ), + "name_source": str(item.get("name_source") or "resolved"), + } + ) + except (OSError, UnicodeError, json.JSONDecodeError, ValueError, TypeError) as exc: + ranking_error = f"排行榜预览不可用：{exc}" + cards.append( + { + "group_id": group.id, + "group_name": name, + "send_time": settings.schedule_send_time, + "schedule_rule": group.schedule_rule, + "image_enabled": bool(group.image_enabled), + "wechat_send_enabled": bool( + getattr(group, "wechat_send_enabled", False) + ), + "ranking_template": group.ranking_template, + "ranking_count_policy": getattr( + group, "ranking_count_policy", "all_messages" + ), + "image_prompt_template": group.image_prompt_template, + "status": status, + "period_start": run.get("period_start", ""), + "period_end": run.get("period_end", ""), + "message_count": run.get("message_count", 0), + "speaker_count": run.get("speaker_count", 0), + "image_url": image_url, + "ranking_preview": ranking_preview, + "ranking_error": ranking_error, + "error": ( + run.get("error") + or run.get("image_error") + or run.get("send_error") + or run.get("error_type") + or "" + ), + "sent_at": run.get("sent_at", ""), + "prompt_hold": bool(run.get("prompt_hold")), + "prompt_hold_reason": str(run.get("prompt_hold_reason") or ""), + "prompt_operation_id": str(run.get("prompt_operation_id") or ""), + "prompt_operation_status": str(run.get("prompt_operation_status") or ""), + "send_hold": bool(run.get("send_hold")), + "send_state": str(run.get("send_state") or ""), + "send_hold_reason": str(run.get("send_hold_reason") or ""), + "send_error": str(run.get("send_error") or ""), + "send_error_type": str(run.get("send_error_type") or ""), + "send_unknown_at": str(run.get("send_unknown_at") or ""), + "updated_at": run.get("updated_at", ""), + } + ) + if status == "SENT": + counts["sent"] += 1 + elif run.get("prompt_hold") or run.get("send_hold"): + counts["held"] += 1 + elif status in ("IMAGE_READY", "READY_TO_SEND"): + counts["generated"] += 1 + elif status == "FAILED": + counts["failed"] += 1 + else: + counts["pending"] += 1 + + next_send = "" + if selected_date == now.date() and any( + card["status"] in ("IMAGE_READY", "READY_TO_SEND") + and not card["sent_at"] + and card["wechat_send_enabled"] + and not card["send_hold"] + for card in cards + ): + next_send = f"{settings.schedule_send_time}（按群 ID 串行批次）" + + runtime_status = build_daily_status( + store, + selected_run_date, + runs=runtime_runs, + output_root=settings.output_dir, + schedule_generate_time=settings.schedule_generate_time, + schedule_send_time=settings.schedule_send_time, + app_timezone=settings.app_timezone, + ) + daily_status = { + "overall_status": runtime_status["overall_status"], + "summary": runtime_status["summary"], + "updated_at": runtime_status["updated_at"], + } + + return { + "today": selected_run_date, + "run_date": selected_run_date, + "should_run": window.should_run, + "period_start": window.period_start_str(), + "period_end": window.period_end_str(), + "enabled_groups": len(cards), + "counts": counts, + "next_send": next_send, + "daily_status": daily_status, + "runtime": runtime_status, + "cards": cards, + } + + +@router.get("/runtime/logs") +def runtime_logs( + run_date: str, + tail: int = Query(default=100, ge=1, le=200), + sources: str | None = None, + levels: str | None = None, + settings: Settings = Depends(get_settings), +): + selected_run_date = _validate_run_date(run_date) + try: + return read_runtime_logs( + settings.logs_dir, + selected_run_date, + tail=tail, + sources=sources, + levels=levels, + app_timezone=settings.app_timezone, + ) + except ValueError as exc: + raise HTTPException(status_code=422, detail=str(exc)) from exc + + +@router.get("/runs") +def list_runs( + settings: Settings = Depends(get_settings), + run_date: str | None = None, + include_files: bool = False, +): + if run_date is not None: + _validate_run_date(run_date) + store = _store(settings) + runs = store.list_runs(run_date) + if include_files: + for run in runs: + run["files"] = _run_files( + store, + str(run.get("group_name") or ""), + str(run.get("run_date") or ""), + ) + return {"runs": runs, "total": len(runs)} + + +def _run_files(store, group: str, run_date: str) -> list[str]: + try: + _validate_run_date(run_date) + group_dir = _safe_group_dir(store, group, run_date) + except HTTPException: + return [] + if not group_dir.exists(): + return [] + return sorted( + path.name + for path in group_dir.glob("*") + if path.is_file() and path.name in ALLOWED_FILES + ) + + +_ARCHIVE_RUN_FIELDS = ( + "group_id", + "wechat_group_id", + "group_name", + "run_date", + "report_date", + "status", + "period_start", + "period_end", + "message_count", + "speaker_count", + "sent_at", + "error", + "error_type", + "image_error", + "send_state", + "send_hold", + "send_hold_reason", + "send_error", + "send_error_type", + "send_unknown_at", + "send_unknown_stage", + "text_sent_at", + "image_sent_at", + "text_submitted_at", + "image_submitted_at", + "verification_level", + "updated_at", +) + + +def _archive_text(value: object) -> str: + return str(value).strip() if value is not None else "" + + +def _archive_local_id(value: object) -> int | None: + text = _archive_text(value) + return int(text) if text.isdigit() else None + + +def _archive_run_summary(run: dict) -> dict | None: + if not isinstance(run, dict): + return None + run_date = _archive_text(run.get("run_date")) + try: + validate_run_date(run_date) + except ValueError: + return None + return {field: run.get(field, "") for field in _ARCHIVE_RUN_FIELDS} + + +def _archive_group_entry(group) -> dict: + deleted_at = getattr(group, "deleted_at", None) + return { + "archive_key": f"group:{group.id}", + "group_id": group.id, + "wechat_group_id": group.wechat_group_id or "", + "display_name": ( + group.display_name or group.wechat_group_name or f"群 {group.id}" + ), + "state": "deleted" if deleted_at is not None else "active", + "enabled": bool(group.enabled) if deleted_at is None else False, + "deleted_at": deleted_at.isoformat() if deleted_at is not None else None, + "created_at": group.created_at.isoformat() if group.created_at else "", + "runs": [], + } + + +def _match_archive_group(run: dict, groups_by_id: dict, groups_by_wechat: dict): + raw_group_id = _archive_text(run.get("group_id")) + local_id = _archive_local_id(raw_group_id) + wechat_id = _archive_text(run.get("wechat_group_id")) + if not wechat_id and raw_group_id and local_id is None: + wechat_id = raw_group_id + + if local_id is not None and local_id in groups_by_id: + candidate = groups_by_id[local_id] + candidate_wechat = _archive_text(candidate.wechat_group_id) + if wechat_id and candidate_wechat and wechat_id != candidate_wechat: + return None + return candidate + if wechat_id: + candidates = groups_by_wechat.get(wechat_id, []) + if len(candidates) == 1: + return candidates[0] + if len(candidates) > 1: + return None + return None + + +def _orphan_archive_key(run: dict) -> str: + raw_group_id = _archive_text(run.get("group_id")) + wechat_id = _archive_text(run.get("wechat_group_id")) + if not wechat_id and raw_group_id and _archive_local_id(raw_group_id) is None: + wechat_id = raw_group_id + if wechat_id: + return f"orphan:wechat:{wechat_id}" + if raw_group_id: + return f"orphan:local:{raw_group_id}" + return f"orphan:name:{_archive_text(run.get('group_name')) or 'unknown'}" + + +@router.get("/archive/groups") +def archive_groups( + session: Session = Depends(repo.get_session), + settings: Settings = Depends(get_settings), +): + """按稳定群身份聚合 V2 归档；不返回任何本机绝对路径。""" + db_groups = repo.list_groups(session, include_deleted=True) + entries = { + group.id: _archive_group_entry(group) + for group in db_groups + if group.id is not None + } + groups_by_id = { + group.id: group for group in db_groups if group.id is not None + } + groups_by_wechat: dict[str, list] = {} + for group in db_groups: + wechat_id = _archive_text(group.wechat_group_id) + if wechat_id: + groups_by_wechat.setdefault(wechat_id, []).append(group) + + orphan_entries: dict[str, dict] = {} + for raw_run in _store(settings).list_runs(): + run = _archive_run_summary(raw_run) + if run is None: + continue + group = _match_archive_group(raw_run, groups_by_id, groups_by_wechat) + if group is not None and group.id in entries: + entries[group.id]["runs"].append(run) + continue + key = _orphan_archive_key(raw_run) + orphan = orphan_entries.setdefault( + key, + { + "archive_key": key, + "group_id": None, + "wechat_group_id": _archive_text(raw_run.get("wechat_group_id")), + "display_name": ( + _archive_text(raw_run.get("group_name")) or "历史遗留群" + ), + "state": "orphaned", + "enabled": False, + "deleted_at": None, + "created_at": "", + "runs": [], + }, + ) + orphan["runs"].append(run) + + for entry in [*entries.values(), *orphan_entries.values()]: + entry["runs"].sort( + key=lambda item: ( + _archive_text(item.get("run_date")), + _archive_text(item.get("updated_at")), + ), + reverse=True, + ) + entry["run_count"] = len(entry["runs"]) + entry["run_dates"] = sorted( + { + _archive_text(item.get("run_date")) + for item in entry["runs"] + if item.get("run_date") + }, + reverse=True, + ) + + active = sorted( + (entry for entry in entries.values() if entry["state"] == "active"), + key=lambda entry: entry["group_id"], + ) + deleted = sorted( + (entry for entry in entries.values() if entry["state"] == "deleted"), + key=lambda entry: entry.get("deleted_at") or "", + reverse=True, + ) + orphaned = sorted( + orphan_entries.values(), + key=lambda entry: ( + entry["runs"][0].get("run_date", "") if entry["runs"] else "" + ), + reverse=True, + ) + return { + "groups": [*active, *deleted, *orphaned], + "active_count": len(active), + "trash_count": len(deleted) + len(orphaned), + } + + +@router.get("/runs/{group}/{run_date}") +def run_detail( + group: str, + run_date: str, + settings: Settings = Depends(get_settings), +): + _validate_run_date(run_date) + store = _store(settings) + _safe_group_dir(store, group, run_date) + run = store.load_run(group, run_date) + return {"run": run, "files": _run_files(store, group, run_date)} + + +@router.get("/files/{group}/{run_date}/{file_name}") +def read_output_file( + group: str, + run_date: str, + file_name: str, + settings: Settings = Depends(get_settings), +): + _validate_run_date(run_date) + if file_name not in ALLOWED_FILES: + raise HTTPException(400, f"不允许访问的文件：{file_name}") + path = _safe_group_dir(_store(settings), group, run_date) / file_name + if not path.exists(): + raise HTTPException(404, "文件不存在") + return FileResponse(path, filename=file_name) diff --git a/app/api/v2_weekly.py b/app/api/v2_weekly.py new file mode 100644 index 0000000..0c993a5 --- /dev/null +++ b/app/api/v2_weekly.py @@ -0,0 +1,68 @@ +"""只读周报归档 API。""" + +from __future__ import annotations + +from fastapi import APIRouter, Depends, HTTPException +from fastapi.responses import FileResponse + +from app.config.settings import Settings, get_settings +from app.weekly.store import WeeklyStore + +router = APIRouter(prefix="/weekly", tags=["v2-weekly"]) + + +def _store(settings: Settings) -> WeeklyStore: + return WeeklyStore(settings.output_dir) + + +def _public_state(state: dict) -> dict: + return { + key: value + for key, value in state.items() + if key not in {"send_claim_id"} + } + + +@router.get("") +def list_weekly_insights(settings: Settings = Depends(get_settings)): + states = [_public_state(item) for item in _store(settings).list_states()] + return {"schema_version": 1, "items": states} + + +@router.get("/{week_start}/{group_id}") +def weekly_insight_detail( + week_start: str, + group_id: int, + settings: Settings = Depends(get_settings), +): + store = _store(settings) + matches = [ + item + for item in store.list_states() + if item.get("week_start") == week_start and int(item.get("group_id") or 0) == group_id + ] + if not matches: + raise HTTPException(status_code=404, detail="周报不存在") + state = _public_state(matches[0]) + state["card_url"] = f"/api/v2/weekly/{week_start}/{group_id}/card" + return state + + +@router.get("/{week_start}/{group_id}/card") +def weekly_insight_card( + week_start: str, + group_id: int, + settings: Settings = Depends(get_settings), +): + store = _store(settings) + matches = [ + item + for item in store.list_states() + if item.get("week_start") == week_start and int(item.get("group_id") or 0) == group_id + ] + if not matches: + raise HTTPException(status_code=404, detail="周报不存在") + path = store.card_path(week_start, str(matches[0].get("week_end") or ""), group_id) + if not path.is_file(): + raise HTTPException(status_code=404, detail="周报卡片不存在") + return FileResponse(path, media_type="image/png", filename="weekly_card.png") diff --git a/app/config/settings.py b/app/config/settings.py index 7f5d030..3eb072f 100644 --- a/app/config/settings.py +++ b/app/config/settings.py @@ -4,11 +4,37 @@ from functools import lru_cache from pathlib import Path -from typing import Any +from typing import Any, Literal from pydantic_settings import BaseSettings, SettingsConfigDict PROJECT_ROOT = Path(__file__).resolve().parents[2] +_ENVIRONMENT_ONLY_FIELDS = frozenset( + { + "allow_test_providers", + "legacy_v1_write_mode", + "scheduler_owner", + "schedule_send_time", + "reliability_watchdog_enabled", + "reliability_lookback_days", + "reliability_watchdog_interval_minutes", + "wechat_fetch_max_attempts", + "wechat_fetch_retry_backoff_seconds", + "wechat_fetch_circuit_failure_threshold", + "wechat_fetch_circuit_cooldown_seconds", + "wechat_runtime_export_fallback_enabled", + "image_prompt_max_chars", + "image_prompt_max_bytes", + "sqlite_busy_timeout_seconds", + "sqlite_retry_max_attempts", + "scheduler_heartbeat_stale_seconds", + "weekly_insights_enabled", + "weekly_send_enabled", + "weekly_generate_time", + "weekly_send_time", + "output_root_override", + } +) class Settings(BaseSettings): @@ -22,14 +48,21 @@ class Settings(BaseSettings): app_host: str = "127.0.0.1" app_port: int = 8766 app_timezone: str = "Asia/Shanghai" + # 测试 Provider 安全闸门：真实运行默认关闭，且不通过设置 API/数据库修改。 + allow_test_providers: bool = False + # 旧 V1 数据库流水线默认只读；短期兼容只能通过环境显式进入 maintenance。 + # 该字段不得加入设置 API/数据库，避免运行时误开双写。 + legacy_v1_write_mode: Literal["read_only", "maintenance"] = "read_only" # 数据库 database_url: str = "sqlite:///data/groupbrief.db" + sqlite_busy_timeout_seconds: int = 15 + sqlite_retry_max_attempts: int = 3 - # 微信历史读取 + # V1 兼容历史读取；正式 V2 使用下方 WeChatDataAnalysis MCP/导出配置。 history_provider_primary: str = "wechat_data_analysis" history_provider_fallback: str = "wechat_cli" - history_provider_mock_enabled: bool = True + history_provider_mock_enabled: bool = False wechat_data_dir: str = "" wechat_export_dir: str = "" wechat_cli_path: str = "" @@ -44,6 +77,12 @@ class Settings(BaseSettings): # 单页范围读取允许更长超时；整组读取总时限独立控制，兼容旧分页路径。 wechat_mcp_range_timeout_seconds: int = 60 wechat_fetch_total_timeout_seconds: int = 600 + wechat_fetch_max_attempts: int = 3 + wechat_fetch_retry_backoff_seconds: float = 1.0 + wechat_fetch_circuit_failure_threshold: int = 3 + wechat_fetch_circuit_cooldown_seconds: int = 300 + # 仅当 MCP 明确读取失败且完整 JSON 导出能独立返回同一时间窗时切换；不拼接两源。 + wechat_runtime_export_fallback_enabled: bool = True # 额外允许的 MCP 主机（逗号分隔，仅 Docker 容器访问宿主机场景使用， # 如 host.docker.internal）。默认空：仍只允许本机回环地址。 wechat_mcp_allowed_hosts: str = "" @@ -55,11 +94,15 @@ class Settings(BaseSettings): summary_provider_primary: str = "codex" summary_provider_fallback: str = "deepseek" codex_summary_model: str = "gpt-5.6-sol" - codex_summary_timeout_seconds: int = 240 + # 结构化群聊整理在高峰期可能超过 4 分钟；600 秒仍有明确上限， + # 同时避免把正常的长响应误判成不可自动恢复的结果未知。 + codex_summary_timeout_seconds: int = 600 codex_summary_max_retries: int = 2 codex_summary_request_concurrency: int = 2 # DeepSeek 备用 + # 旧设置兼容字段；真实路由只使用 summary_provider_primary/fallback。 + # 不再通过设置 API 暴露，保留一版以兼容旧 .env/数据库。 ai_provider: str = "deepseek" ai_base_url: str = "https://api.deepseek.com" ai_api_key: str = "" @@ -82,11 +125,21 @@ class Settings(BaseSettings): codex_home: str = "" codex_timeout_seconds: int = 1200 codex_generated_images_dir: str = "" # 留空时默认 ~/.codex/generated_images + # Codex/ImageGen 任务在可靠结构化回执下允许受控并发；默认两路。 + image_generation_concurrency: int = 2 + # 本地 Level 3 信息图字体；留空时按 Windows 常见中文字体顺序探测。 + image_fallback_font_path: str = "" + # 最终 image_prompt.txt 的硬边界；仅部署环境可调，避免运行时误设为无限。 + image_prompt_max_chars: int = 24000 + image_prompt_max_bytes: int = 65536 # 微信发送（V2）。默认适配微信 4.1.x 的 Windows 键盘/剪贴板/OCR 驱动； # legacy_cli 保留旧 wechat-automation-api 兼容入口。 wechat_sender_mode: str = "native" wechat_native_action_delay_seconds: float = 0.6 + wechat_native_stage_timeout_seconds: float = 5.0 + wechat_native_submit_timeout_seconds: float = 8.0 + wechat_native_poll_interval_seconds: float = 0.2 wechat_native_mutex_timeout_seconds: float = 20.0 wechat_send_claim_seconds: int = 180 wechat_late_send_window_minutes: int = 30 @@ -110,9 +163,30 @@ class Settings(BaseSettings): email_send_partial_report: bool = True # 自动任务 + # fastapi：8766 内 APScheduler 是唯一 owner；external：仅允许外部调度； + # disabled：不注册任何自动任务。该字段只由环境配置，不通过设置 API 修改。 + scheduler_owner: Literal["fastapi", "external", "disabled"] = "fastapi" schedule_generate_time: str = "00:15" + # 日报微信发送采用唯一全局批次时间；群级 send_time 仅保留数据库兼容。 + schedule_send_time: str = "08:30" schedule_email_time: str = "after_generate" schedule_startup_catchup_enabled: bool = True + # 无人值守恢复只在进程启动时检查一次，不再注册固定频率 Watchdog。 + # 字段名保留一版以兼容既有部署环境。 + reliability_watchdog_enabled: bool = True + # 自动恢复严格限制为当前日和前一日（48 小时产品边界）。更早任务只预览。 + reliability_lookback_days: int = 2 + reliability_watchdog_interval_minutes: int = 10 # 已弃用，仅兼容旧配置 + # 周报能力先部署、后灰度：14 天可靠性验收完成前保持关闭。 + weekly_insights_enabled: bool = False + weekly_send_enabled: bool = False + weekly_generate_time: str = "07:45" + weekly_send_time: str = "08:30" + scheduler_heartbeat_stale_seconds: int = 300 + + # 只供测试/离线执行通过环境变量隔离 output 与相邻 runtime； + # 生产默认留空，路径合同保持不变，且设置 API 无权修改。 + output_root_override: str = "" # 路径 @property @@ -121,6 +195,8 @@ def data_dir(self) -> Path: @property def output_dir(self) -> Path: + if self.output_root_override: + return Path(self.output_root_override).expanduser().resolve() return PROJECT_ROOT / "output" @property @@ -155,7 +231,7 @@ def apply_runtime_values(self, values: dict[str, Any]) -> list[str]: applied: list[str] = [] field_map = Settings.model_fields for key, raw in values.items(): - if key not in field_map: + if key not in field_map or key in _ENVIRONMENT_ONLY_FIELDS: continue if isinstance(raw, str) and raw.strip() == "******": continue @@ -182,7 +258,7 @@ def _coerce_setting_value(key: str, raw: Any, annotation: Any) -> Any: return True if text in _BOOL_FALSE: return False - return bool(text) + raise ValueError(f"{key} 必须是 true/false") if annotation is int or annotation == int: if isinstance(raw, bool): return int(raw) diff --git a/app/core/logging.py b/app/core/logging.py index b464ee0..f41956d 100644 --- a/app/core/logging.py +++ b/app/core/logging.py @@ -19,8 +19,6 @@ _FORMAT = "%(asctime)s [%(levelname)s] %(name)s: %(message)s" -_configured: set[str] = set() - # P9：日志保留天数（超出即删除） LOG_RETENTION_DAYS = 30 @@ -58,21 +56,19 @@ def clean_old_logs(logs_dir: Path, max_days: int = LOG_RETENTION_DAYS) -> int: def setup_logging(logs_dir: Path, level: int = logging.INFO) -> None: logs_dir.mkdir(parents=True, exist_ok=True) # P9：启动时清理过期日志 + removed = 0 + cleanup_error: Exception | None = None try: removed = clean_old_logs(logs_dir) - if removed: - logging.getLogger("app").info("清理过期日志文件 %d 个（保留 %d 天）", removed, LOG_RETENTION_DAYS) - except Exception: - pass + except Exception as exc: + cleanup_error = exc root = logging.getLogger() - if root.handlers: - return root.setLevel(level) - - console = logging.StreamHandler() - console.setFormatter(logging.Formatter(_FORMAT)) - root.addHandler(console) + if not root.handlers: + console = logging.StreamHandler() + console.setFormatter(logging.Formatter(_FORMAT)) + root.addHandler(console) for name, filename in ( ("app", "app.log"), @@ -83,20 +79,31 @@ def setup_logging(logs_dir: Path, level: int = logging.INFO) -> None: ): _configure_logger(name, logs_dir / filename, level) + app_logger = logging.getLogger("app") + if removed: + app_logger.info( + "清理过期日志文件 %d 个（保留 %d 天）", + removed, + LOG_RETENTION_DAYS, + ) + if cleanup_error is not None: + app_logger.warning("清理过期日志失败：%s", cleanup_error, exc_info=True) + def _configure_logger(name: str, file_path: Path, level: int) -> None: - if name in _configured: - return _ensure_file(file_path) logger = logging.getLogger(name) logger.setLevel(level) + target = str(file_path.resolve()) + for existing in logger.handlers: + if isinstance(existing, RotatingFileHandler) and existing.baseFilename == target: + return handler = RotatingFileHandler( file_path, maxBytes=5 * 1024 * 1024, backupCount=5, encoding="utf-8" ) handler.setFormatter(logging.Formatter(_FORMAT)) logger.addHandler(handler) logger.propagate = False - _configured.add(name) def get_logger(name: str) -> logging.Logger: diff --git a/app/core/observability.py b/app/core/observability.py new file mode 100644 index 0000000..698f0f0 --- /dev/null +++ b/app/core/observability.py @@ -0,0 +1,23 @@ +"""不含业务正文和凭据的结构化运行事件。""" + +from __future__ import annotations + +import json +from datetime import datetime +from typing import Any + + +def log_event(logger, event: str, **fields: Any) -> None: + payload = { + "event": str(event), + "timestamp": datetime.now().astimezone().isoformat(), + } + for key, value in fields.items(): + if value is None or value == "": + continue + if key in {"prompt", "messages", "api_key", "token", "password", "target"}: + continue + if key in {"error", "error_summary", "detail"}: + value = str(value)[:300] + payload[key] = value + logger.info("GB_EVENT %s", json.dumps(payload, ensure_ascii=False, default=str)) diff --git a/app/core/path_security.py b/app/core/path_security.py new file mode 100644 index 0000000..7e2a529 --- /dev/null +++ b/app/core/path_security.py @@ -0,0 +1,75 @@ +"""本地文件路径安全边界。 + +所有来自 API、数据库显示名或历史文件的路径片段，在进入 output 目录前 +都必须经过这里的导航检查和真实路径 containment 校验。 +""" + +from __future__ import annotations + +import os +import re +from datetime import datetime +from pathlib import Path, PurePosixPath, PureWindowsPath + + +_ISO_DATE = re.compile(r"^\d{4}-\d{2}-\d{2}$") +_PATH_SPLIT = re.compile(r"[\\/]+") + + +class PathBoundaryError(ValueError): + """用户输入可能逃出预期文件根目录。""" + + +def _strip_windows_extended_prefix(value: str) -> str: + """把 Windows 同一路径的 extended-length 表示还原为普通 drive/UNC 表示。""" + + if value.startswith("\\\\?\\UNC\\"): + return "\\\\" + value[8:] + if value.startswith("\\\\?\\") and len(value) >= 7 and value[5] == ":": + return value[4:] + return value + + +def _resolved_path(value: Path | str) -> Path: + resolved = Path(value).resolve() + if os.name == "nt": + return Path(_strip_windows_extended_prefix(str(resolved))) + return resolved + + +def validate_iso_date(value: str, *, field_name: str = "date") -> str: + """只接受真实存在的 YYYY-MM-DD 日期。""" + if not isinstance(value, str) or not _ISO_DATE.fullmatch(value): + raise ValueError(f"{field_name} 必须是有效的 YYYY-MM-DD 日期") + try: + datetime.strptime(value, "%Y-%m-%d") + except ValueError as exc: + raise ValueError(f"{field_name} 必须是有效的 YYYY-MM-DD 日期") from exc + return value + + +def validate_path_label(value: str, *, field_name: str = "name") -> str: + """允许普通显示名中的标点，但拒绝路径导航、盘符和绝对路径。""" + if not isinstance(value, str) or "\x00" in value: + raise PathBoundaryError(f"{field_name} 包含不安全的路径内容") + if PurePosixPath(value).is_absolute(): + raise PathBoundaryError(f"{field_name} 不能是绝对路径") + windows_path = PureWindowsPath(value) + if windows_path.is_absolute() or windows_path.drive: + raise PathBoundaryError(f"{field_name} 不能包含盘符或 UNC 路径") + if any(part in {".", ".."} for part in _PATH_SPLIT.split(value)): + raise PathBoundaryError(f"{field_name} 不能包含路径导航段") + return value + + +def resolve_within(root: Path | str, *parts: Path | str, allow_root: bool = False) -> Path: + """解析路径并证明它位于 root 内；同时阻断 symlink 和 sibling-prefix 绕过。""" + try: + resolved_root = _resolved_path(root) + candidate = _resolved_path(resolved_root.joinpath(*(Path(part) for part in parts))) + candidate.relative_to(resolved_root) + except (OSError, RuntimeError, ValueError) as exc: + raise PathBoundaryError("路径超出允许的文件目录") from exc + if not allow_root and candidate == resolved_root: + raise PathBoundaryError("路径不能指向文件根目录") + return candidate diff --git a/app/data_sources/base.py b/app/data_sources/base.py index 55646dc..b6c2313 100644 --- a/app/data_sources/base.py +++ b/app/data_sources/base.py @@ -65,6 +65,8 @@ class V2Message: timestamp: datetime message_type: str = "text" content: str = "" + upstream_sender_name: str = "" + sender_name_source: str = "" raw: dict[str, Any] = field(default_factory=dict) def to_dict(self) -> dict: @@ -77,6 +79,8 @@ def to_dict(self) -> dict: "timestamp": self.timestamp.isoformat(), "message_type": self.message_type, "content": self.content, + "upstream_sender_name": self.upstream_sender_name, + "sender_name_source": self.sender_name_source, } diff --git a/app/data_sources/history_provider.py b/app/data_sources/history_provider.py new file mode 100644 index 0000000..68b65a8 --- /dev/null +++ b/app/data_sources/history_provider.py @@ -0,0 +1,92 @@ +"""把 V1 历史 Provider 安全适配到 V2 只读数据源契约。""" + +from __future__ import annotations + +import hashlib + +from app.data_sources.base import ( + DataSourceHealth, + DataSourceStatus, + FetchResult, + ResolvedGroup, + V2Message, + WeChatDataSource, +) +from app.providers.history.base import ChatHistoryProvider, ProviderStatus, RawMessage +from app.v2.constants import GROUP_NOT_FOUND, MESSAGE_FETCH_FAILED, WECHAT_DATA_UNAVAILABLE + + +class HistoryProviderDataSource(WeChatDataSource): + def __init__(self, provider: ChatHistoryProvider): + self.provider = provider + self.name = provider.name + + def health_check(self) -> DataSourceHealth: + health = self.provider.health_check() + status = DataSourceStatus.OK if health.ok else DataSourceStatus.UNAVAILABLE + return DataSourceHealth(status, health.detail) + + def list_groups(self) -> list[ResolvedGroup]: + return [ + ResolvedGroup(item.group_id, item.group_name, item.member_count) + for item in self.provider.list_groups() + ] + + def resolve_group(self, group_name: str) -> list[ResolvedGroup]: + query = group_name.strip().casefold() + return [ + item + for item in self.list_groups() + if query in item.group_name.casefold() + ] + + def fetch_messages(self, group_id, start_time, end_time) -> FetchResult: + result = self.provider.fetch_messages(group_id, start_time, end_time) + if result.status == ProviderStatus.OK and result.messages: + return FetchResult( + [_message(item) for item in result.messages], + DataSourceStatus.OK, + result.detail, + meta={ + **(result.meta if isinstance(result.meta, dict) else {}), + "provider_chain": [self.name], + "fallback_used": False, + }, + ) + status = ( + DataSourceStatus.GROUP_NOT_FOUND + if result.status == ProviderStatus.GROUP_NOT_FOUND + else DataSourceStatus.EMPTY_RESULT + if result.status == ProviderStatus.EMPTY_RESULT + else DataSourceStatus.UNAVAILABLE + if result.status == ProviderStatus.UNAVAILABLE + else DataSourceStatus.READ_FAILED + ) + error_type = ( + GROUP_NOT_FOUND + if status == DataSourceStatus.GROUP_NOT_FOUND + else WECHAT_DATA_UNAVAILABLE + if status == DataSourceStatus.UNAVAILABLE + else MESSAGE_FETCH_FAILED + ) + return FetchResult([], status, result.detail, error_type, {"provider_chain": [self.name]}) + + +def _message(item: RawMessage) -> V2Message: + message_id = item.source_message_id or item.content_hash + if not message_id: + payload = f"{item.sender_id}|{item.timestamp.isoformat()}|{item.content}" + message_id = hashlib.sha256(payload.encode("utf-8")).hexdigest()[:24] + return V2Message( + message_id=message_id, + group_id=item.group_id, + group_name=item.group_name, + sender_id=item.sender_id, + sender_name=item.sender_name or "(未知)", + timestamp=item.timestamp, + message_type=item.message_type, + content=item.content, + upstream_sender_name=item.upstream_sender_name or item.sender_name or "", + sender_name_source=item.sender_name_source or "history_provider", + raw={"source": item.source or "history_provider"}, + ) diff --git a/app/data_sources/resilient.py b/app/data_sources/resilient.py new file mode 100644 index 0000000..30d4233 --- /dev/null +++ b/app/data_sources/resilient.py @@ -0,0 +1,154 @@ +"""V2 数据源有限重试与进程内熔断，不拼接多个 Provider 的消息。""" + +from __future__ import annotations + +import random +import threading +import time +from datetime import datetime +from typing import Callable + +from app.config.settings import Settings +from app.data_sources.base import ( + DataSourceHealth, + DataSourceStatus, + FetchResult, + ResolvedGroup, + WeChatDataSource, +) +from app.v2.constants import MESSAGE_FETCH_FAILED, WECHAT_DATA_UNAVAILABLE + + +class ProviderCircuitBreaker: + def __init__(self, *, threshold: int, cooldown_seconds: float) -> None: + self.threshold = max(int(threshold), 1) + self.cooldown_seconds = max(float(cooldown_seconds), 1.0) + self._failures = 0 + self._opened_at: float | None = None + self._half_open_probe = False + self._lock = threading.Lock() + + def allow(self, now: float) -> bool: + with self._lock: + if self._opened_at is None: + return True + if now - self._opened_at >= self.cooldown_seconds: + # 半开只放行一个探测，避免多个群在冷却结束时同时冲击 Provider。 + if self._half_open_probe: + return False + self._half_open_probe = True + return True + return False + + def success(self) -> None: + with self._lock: + self._failures = 0 + self._opened_at = None + self._half_open_probe = False + + def failure(self, now: float) -> None: + with self._lock: + if self._half_open_probe: + self._failures = self.threshold + else: + self._failures += 1 + if self._failures >= self.threshold: + self._opened_at = now + self._half_open_probe = False + + +class ResilientWeChatDataSource(WeChatDataSource): + """只包装 fetch_messages；群解析仍委托原数据源并保留原语义。""" + + def __init__( + self, + source: WeChatDataSource, + settings: Settings, + *, + sleep: Callable[[float], None] = time.sleep, + clock: Callable[[], float] = time.monotonic, + jitter: Callable[[], float] = random.random, + ) -> None: + self.source = source + self.settings = settings + self.name = source.name + self._sleep = sleep + self._clock = clock + self._jitter = jitter + self._circuit = ProviderCircuitBreaker( + threshold=settings.wechat_fetch_circuit_failure_threshold, + cooldown_seconds=settings.wechat_fetch_circuit_cooldown_seconds, + ) + + def health_check(self) -> DataSourceHealth: + return self.source.health_check() + + def list_groups(self) -> list[ResolvedGroup]: + return self.source.list_groups() + + def resolve_group(self, group_name: str) -> list[ResolvedGroup]: + return self.source.resolve_group(group_name) + + @staticmethod + def _retryable(result: FetchResult) -> bool: + return bool( + result.status in {DataSourceStatus.UNAVAILABLE, DataSourceStatus.READ_FAILED} + and result.error_type + in {"", MESSAGE_FETCH_FAILED, WECHAT_DATA_UNAVAILABLE, "API_5XX", "API_TIMEOUT_PRE_SUBMIT"} + ) + + def fetch_messages(self, group_id, start_time, end_time) -> FetchResult: + now = self._clock() + if not self._circuit.allow(now): + return FetchResult( + [], + DataSourceStatus.UNAVAILABLE, + "微信数据源熔断中，等待冷却后自动探测", + WECHAT_DATA_UNAVAILABLE, + {"attempts": 0, "circuit_open": True, "provider_chain": [self.name]}, + ) + + max_attempts = min(max(int(self.settings.wechat_fetch_max_attempts), 1), 5) + base = max(float(self.settings.wechat_fetch_retry_backoff_seconds), 0.0) + attempts: list[dict] = [] + last_result: FetchResult | None = None + for attempt in range(1, max_attempts + 1): + started = self._clock() + try: + result = self.source.fetch_messages(group_id, start_time, end_time) + except (TimeoutError, ConnectionError, OSError) as exc: + result = FetchResult( + [], + DataSourceStatus.READ_FAILED, + str(exc)[:300], + MESSAGE_FETCH_FAILED, + ) + elapsed_ms = round((self._clock() - started) * 1000) + attempts.append( + { + "attempt": attempt, + "status": result.status.value, + "error_type": result.error_type, + "elapsed_ms": elapsed_ms, + } + ) + result.meta = { + **(result.meta if isinstance(result.meta, dict) else {}), + "attempt_count": attempt, + "attempts": attempts, + "provider_chain": [self.name], + } + last_result = result + if result.status == DataSourceStatus.OK: + self._circuit.success() + return result + if not self._retryable(result) or attempt >= max_attempts: + break + delay = base * (2 ** (attempt - 1)) + delay += delay * 0.2 * self._jitter() + if delay > 0: + self._sleep(delay) + + self._circuit.failure(self._clock()) + assert last_result is not None + return last_result diff --git a/app/data_sources/wechat_data_analysis.py b/app/data_sources/wechat_data_analysis.py index 3751b45..05408d7 100644 --- a/app/data_sources/wechat_data_analysis.py +++ b/app/data_sources/wechat_data_analysis.py @@ -18,6 +18,7 @@ from datetime import datetime from app.config.settings import Settings, get_settings +from app.core.logging import get_logger from app.v2.constants import ( GROUP_NOT_FOUND, MESSAGE_FETCH_FAILED, @@ -36,6 +37,7 @@ # 默认本机地址（与 .env 一致；覆盖时仅允许本机回环，见 V1 wechat_mcp.py） DEFAULT_MCP_URL = "http://127.0.0.1:10392/mcp" +logger = get_logger("groupbrief.data_source") class WeChatDataAnalysisSource(WeChatDataSource): @@ -86,7 +88,7 @@ def _group_exists(self, group_id: str) -> bool: if g.group_id == group_id: return True except Exception: - pass + logger.warning("群存在性校验失败：group_id=%s", group_id, exc_info=True) return False # ---------- 消息读取 ---------- @@ -98,6 +100,37 @@ def fetch_messages( end_time: datetime, ) -> FetchResult: result = self._provider.fetch_messages(group_id, start_time, end_time) + primary_result = result + fallback_used = False + if ( + self.settings.wechat_runtime_export_fallback_enabled + and getattr(self._provider, "_mcp_client", None) is not None + and result.status in {ProviderStatus.READ_FAILED, ProviderStatus.UNAVAILABLE} + and getattr(self._provider, "export_dir", None) is not None + and self._provider.export_dir.is_dir() + ): + # 完整切换到同一 WeChatDataAnalysis 的 JSON 导出结果；绝不拼接 MCP 消息。 + fallback = self._provider._fetch_messages_export( + group_id, + start_time, + end_time, + ) + if fallback.status in {ProviderStatus.OK, ProviderStatus.EMPTY_RESULT}: + result = fallback + fallback_used = True + + source_meta = { + **(result.meta if isinstance(result.meta, dict) else {}), + "provider_chain": ( + ["wechat_data_analysis_mcp", "wechat_data_analysis_export"] + if fallback_used + else ["wechat_data_analysis_mcp"] + if getattr(self._provider, "_mcp_client", None) is not None + else ["wechat_data_analysis_export"] + ), + "fallback_used": fallback_used, + "primary_error": primary_result.detail[:200] if fallback_used else "", + } if result.status == ProviderStatus.OK and result.messages: messages = [_to_v2_message(m) for m in result.messages] @@ -105,7 +138,7 @@ def fetch_messages( messages=messages, status=DataSourceStatus.OK, detail=f"{self.name}：{len(messages)} 条消息", - meta=result.meta, + meta=source_meta, ) if result.status == ProviderStatus.GROUP_NOT_FOUND: return FetchResult( @@ -122,7 +155,7 @@ def fetch_messages( result.meta, ) return FetchResult( - [], DataSourceStatus.EMPTY_RESULT, result.detail, "", result.meta + [], DataSourceStatus.EMPTY_RESULT, result.detail, "", source_meta ) # READ_FAILED / UNAVAILABLE 等一律归为取数失败 error_type = WECHAT_DATA_UNAVAILABLE if result.status in ( @@ -162,6 +195,15 @@ def _to_v2_message(m: RawMessage) -> V2Message: timestamp=m.timestamp, message_type=m.message_type, content=m.content, + upstream_sender_name=( + m.upstream_sender_name + or ( + m.sender_name + if m.sender_name_source in {"", "wechat_data_analysis"} + else "" + ) + ), + sender_name_source=m.sender_name_source or "wechat_data_analysis", raw={ "source": m.source, "source_message_id": m.source_message_id, diff --git a/app/db/models.py b/app/db/models.py index 4533983..764d0b0 100644 --- a/app/db/models.py +++ b/app/db/models.py @@ -4,11 +4,20 @@ from datetime import datetime +from sqlalchemy import CheckConstraint, Index, text from sqlmodel import Field, SQLModel class Group(SQLModel, table=True): __tablename__ = "groups" + __table_args__ = ( + Index( + "uq_groups_wechat_group_id_active", + "wechat_group_id", + unique=True, + sqlite_where=text("TRIM(wechat_group_id) <> '' AND deleted_at IS NULL"), + ), + ) id: int | None = Field(default=None, primary_key=True) display_name: str = Field(default="", max_length=128) @@ -21,15 +30,19 @@ class Group(SQLModel, table=True): deleted_at: datetime | None = Field(default=None, index=True) # ---------- V2 扩展字段（P3 起使用，P7 pipeline 落地） ---------- - schedule_rule: str = "weekday_default" # 统计周期规则 - send_time: str = "08:30" # 本群独立发送时间 HH:MM + schedule_rule: str = "daily_previous_day" # 默认每天统计前一自然日 + send_time: str = "08:30" # 兼容旧数据；实际发送读取全局 schedule_send_time + summary_provider: str = "" # 周报摘要 AI；空值继承全局 + prompt_provider: str = "" # 日报 Prompt AI；空值继承全局 summary_model: str = "gpt-5.6-sol" # 总结主模型 prompt_model: str = "gpt-5.6-sol" # Prompt 主模型 image_enabled: bool = True # 是否生图 send_target: str = "" # 可选人工发送目标；为空时自动跟随 wechat_group_name ranking_template: str = "default" # 排行榜模板名 + ranking_count_policy: str = "all_messages" # all_messages / text_primary_with_interactions + sender_name_policy: str = "resolved" # resolved / wechat_data_analysis image_prompt_template: str = "default" # 生图 Prompt 模板名 - image_theme: str = "random_preset" # 生图大主题键（默认每日随机） + image_theme: str = "ai_free" # 生图主题键（默认由 AI 按聊天内容自由发挥） image_theme_custom: str = "" # 自定义生图大主题（image_theme=custom 时使用） image_prompt_override: str = "" # 本群专属 Prompt 模板；为空时继承全局模板 wechat_send_enabled: bool = False # 独立于生成开关，默认禁止自动对外发送 @@ -37,6 +50,9 @@ class Group(SQLModel, table=True): class Run(SQLModel, table=True): __tablename__ = "runs" + __table_args__ = ( + Index("ix_runs_report_date_status", "report_date", "status"), + ) id: int | None = Field(default=None, primary_key=True) report_date: str = "" # YYYY-MM-DD（报告归属日） @@ -51,10 +67,38 @@ class Run(SQLModel, table=True): class GroupRun(SQLModel, table=True): __tablename__ = "group_runs" + __table_args__ = ( + CheckConstraint( + """ + ( + identity_state = 'linked' + AND group_id IS NOT NULL + AND legacy_group_id IS NULL + AND orphan_reason = '' + ) + OR + ( + identity_state = 'orphaned' + AND group_id IS NULL + AND legacy_group_id IS NOT NULL + AND orphan_reason = 'historical_group_missing' + ) + """, + name="ck_group_runs_identity", + ), + ) id: int | None = Field(default=None, primary_key=True) - run_id: int = Field(index=True) - group_id: int = Field(index=True) + run_id: int = Field(foreign_key="runs.id", ondelete="RESTRICT", index=True) + group_id: int | None = Field( + default=None, + foreign_key="groups.id", + ondelete="RESTRICT", + index=True, + ) + legacy_group_id: int | None = None + identity_state: str = "linked" # linked / orphaned + orphan_reason: str = "" provider_used: str = "" message_count: int = 0 speaker_count: int = 0 @@ -67,7 +111,12 @@ class Report(SQLModel, table=True): __tablename__ = "reports" id: int | None = Field(default=None, primary_key=True) - group_run_id: int = Field(index=True) + group_run_id: int = Field( + foreign_key="group_runs.id", + ondelete="RESTRICT", + index=True, + unique=True, + ) ranking_text: str = "" prompt_text: str = "" ranking_file: str = "" @@ -101,7 +150,12 @@ class ExecutionLog(SQLModel, table=True): __tablename__ = "execution_logs" id: int | None = Field(default=None, primary_key=True) - run_id: int | None = None + run_id: int | None = Field( + default=None, + foreign_key="runs.id", + ondelete="RESTRICT", + index=True, + ) level: str = "info" # info / warning / error message: str = "" created_at: datetime = Field(default_factory=datetime.now) diff --git a/app/db/offline_migrations.py b/app/db/offline_migrations.py new file mode 100644 index 0000000..f36e3fb --- /dev/null +++ b/app/db/offline_migrations.py @@ -0,0 +1,696 @@ +"""GroupBrief 的显式离线数据库迁移。 + +该模块故意不接入应用启动流程。迁移只读取明确指定的源数据库，并把结果 +写入一个全新的目标文件；源数据库不会被原地修改或覆盖。 +""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import os +import sqlite3 +import sys +import uuid +from dataclasses import asdict, dataclass +from datetime import datetime, timezone +from pathlib import Path +from typing import Any, Sequence + + +MIGRATION_ID = "p0_2b_group_run_identity_v1" +TARGET_USER_VERSION = 1 +_MIGRATION_SIGNATURE = """group_runs:nullable-group,legacy-id,identity-state,restrict-fks +reports:unique-group-run,restrict-fk +execution_logs:nullable-run,restrict-fk +indexes:runs-date-status,execution-run,active-wechat-id +""" +MIGRATION_CHECKSUM = hashlib.sha256(_MIGRATION_SIGNATURE.encode("utf-8")).hexdigest() + +_CORE_TABLES = ("groups", "runs", "group_runs", "reports", "execution_logs") +_REQUIRED_COLUMNS: dict[str, set[str]] = { + "groups": {"id", "wechat_group_id", "deleted_at"}, + "runs": {"id", "report_date", "status"}, + "group_runs": { + "id", + "run_id", + "group_id", + "provider_used", + "message_count", + "speaker_count", + "ranking_status", + "prompt_status", + "error_message", + }, + "reports": { + "id", + "group_run_id", + "ranking_text", + "prompt_text", + "ranking_file", + "prompt_file", + "poster_file", + "poster_status", + "email_status", + "created_at", + "updated_at", + }, + "execution_logs": {"id", "run_id", "level", "message", "created_at"}, +} +_REBUILT_TABLES = ("group_runs", "reports", "execution_logs") +_KNOWN_REBUILT_INDEXES = { + "ix_group_runs_group_id", + "ix_group_runs_run_id", + "ix_reports_group_run_id", +} + + +class MigrationError(RuntimeError): + """迁移前置条件、执行或验证失败。""" + + +@dataclass(frozen=True) +class DatabaseSnapshot: + integrity_check: str + user_version: int + table_counts: dict[str, int] + orphan_group_runs: int + orphan_reports: int + group_runs_missing_run: int + execution_logs_missing_run: int + duplicate_report_relations: int + duplicate_active_wechat_ids: int + + +def _sha256(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as handle: + for chunk in iter(lambda: handle.read(1024 * 1024), b""): + digest.update(chunk) + return digest.hexdigest() + + +def _connect_read_only(path: Path) -> sqlite3.Connection: + connection = sqlite3.connect(f"{path.resolve().as_uri()}?mode=ro", uri=True) + connection.row_factory = sqlite3.Row + connection.execute("PRAGMA query_only = ON") + return connection + + +def _table_exists(connection: sqlite3.Connection, table: str) -> bool: + row = connection.execute( + "SELECT 1 FROM sqlite_master WHERE type='table' AND name=?", + (table,), + ).fetchone() + return row is not None + + +def _table_columns(connection: sqlite3.Connection, table: str) -> set[str]: + return {str(row[1]) for row in connection.execute(f'PRAGMA table_info("{table}")')} + + +def _scalar(connection: sqlite3.Connection, sql: str) -> int: + return int(connection.execute(sql).fetchone()[0]) + + +def _validate_source_schema(connection: sqlite3.Connection) -> None: + missing_tables = [table for table in _CORE_TABLES if not _table_exists(connection, table)] + if missing_tables: + raise MigrationError(f"数据库缺少必要表：{', '.join(missing_tables)}") + + if _table_exists(connection, "schema_migrations"): + migration_columns = _table_columns(connection, "schema_migrations") + expected = {"migration_id", "applied_at", "checksum"} + if not expected.issubset(migration_columns): + raise MigrationError("schema_migrations 表结构不兼容，拒绝继续") + applied = connection.execute( + "SELECT checksum FROM schema_migrations WHERE migration_id=?", + (MIGRATION_ID,), + ).fetchone() + if applied is not None: + raise MigrationError(f"迁移 {MIGRATION_ID} 已经执行，拒绝重复迁移") + + for table, required in _REQUIRED_COLUMNS.items(): + actual = _table_columns(connection, table) + missing = sorted(required - actual) + if missing: + raise MigrationError(f"表 {table} 缺少必要列：{', '.join(missing)}") + if table in _REBUILT_TABLES: + unexpected = sorted(actual - required) + if unexpected: + raise MigrationError( + f"待重建表 {table} 包含未知列，拒绝静默丢弃：{', '.join(unexpected)}" + ) + + unknown_indexes = [ + str(row[0]) + for row in connection.execute( + """ + SELECT name + FROM sqlite_master + WHERE type='index' + AND tbl_name IN ('group_runs', 'reports', 'execution_logs') + AND sql IS NOT NULL + ORDER BY name + """ + ) + if str(row[0]) not in _KNOWN_REBUILT_INDEXES + ] + if unknown_indexes: + raise MigrationError( + "待重建表包含未知显式索引，拒绝静默丢弃：" + "，".join(unknown_indexes) + ) + + dependent_objects = [ + f"{row[0]}:{row[1]}" + for row in connection.execute( + """ + SELECT type, name + FROM sqlite_master + WHERE + (type='trigger' AND tbl_name IN ('group_runs', 'reports', 'execution_logs')) + OR + ( + type='view' + AND ( + LOWER(sql) LIKE '%group_runs%' + OR LOWER(sql) LIKE '%reports%' + OR LOWER(sql) LIKE '%execution_logs%' + ) + ) + ORDER BY type, name + """ + ) + ] + if dependent_objects: + raise MigrationError( + "待重建表存在未知触发器或依赖视图，拒绝静默破坏：" + "，".join(dependent_objects) + ) + +def _validate_no_sqlite_sidecars(source: Path) -> None: + sidecars = [ + Path(f"{source}-wal"), + Path(f"{source}-shm"), + Path(f"{source}-journal"), + ] + present = [path.name for path in sidecars if path.exists()] + if present: + raise MigrationError( + "源数据库存在 SQLite 写入/日志侧文件，不能视为离线源：" + "，".join(present) + ) + + +def _snapshot(connection: sqlite3.Connection) -> DatabaseSnapshot: + integrity_rows = [str(row[0]) for row in connection.execute("PRAGMA integrity_check")] + integrity = "ok" if integrity_rows == ["ok"] else "; ".join(integrity_rows) + counts = {table: _scalar(connection, f'SELECT COUNT(*) FROM "{table}"') for table in _CORE_TABLES} + return DatabaseSnapshot( + integrity_check=integrity, + user_version=int(connection.execute("PRAGMA user_version").fetchone()[0]), + table_counts=counts, + orphan_group_runs=_scalar( + connection, + """ + SELECT COUNT(*) + FROM group_runs gr + LEFT JOIN groups g ON g.id = gr.group_id + WHERE g.id IS NULL + """, + ), + orphan_reports=_scalar( + connection, + """ + SELECT COUNT(*) + FROM reports rep + LEFT JOIN group_runs gr ON gr.id = rep.group_run_id + WHERE gr.id IS NULL + """, + ), + group_runs_missing_run=_scalar( + connection, + """ + SELECT COUNT(*) + FROM group_runs gr + LEFT JOIN runs r ON r.id = gr.run_id + WHERE r.id IS NULL + """, + ), + execution_logs_missing_run=_scalar( + connection, + """ + SELECT COUNT(*) + FROM execution_logs log + LEFT JOIN runs r ON r.id = log.run_id + WHERE log.run_id IS NOT NULL AND r.id IS NULL + """, + ), + duplicate_report_relations=_scalar( + connection, + """ + SELECT COUNT(*) + FROM ( + SELECT group_run_id + FROM reports + GROUP BY group_run_id + HAVING COUNT(*) > 1 + ) + """, + ), + duplicate_active_wechat_ids=_scalar( + connection, + """ + SELECT COUNT(*) + FROM ( + SELECT wechat_group_id + FROM groups + WHERE TRIM(wechat_group_id) <> '' AND deleted_at IS NULL + GROUP BY wechat_group_id + HAVING COUNT(*) > 1 + ) + """, + ), + ) + + +def _validate_preflight(snapshot: DatabaseSnapshot) -> None: + if snapshot.integrity_check != "ok": + raise MigrationError(f"源数据库完整性检查失败：{snapshot.integrity_check}") + if snapshot.user_version != 0: + raise MigrationError( + f"源数据库 user_version={snapshot.user_version}，本迁移只接受旧版 user_version=0" + ) + failures = { + "缺失父 Run 的 GroupRun": snapshot.group_runs_missing_run, + "孤儿 Report": snapshot.orphan_reports, + "同一 GroupRun 的重复 Report": snapshot.duplicate_report_relations, + "重复的活动微信群 ID": snapshot.duplicate_active_wechat_ids, + "缺失父 Run 的执行日志": snapshot.execution_logs_missing_run, + } + present = [f"{name}={count}" for name, count in failures.items() if count] + if present: + raise MigrationError("迁移前置检查失败：" + "，".join(present)) + + +def preflight_database(source: str | Path) -> dict[str, Any]: + """只读检查一个候选源数据库，不创建任何输出文件。""" + source_path = Path(source).expanduser().resolve() + if not source_path.is_file(): + raise MigrationError(f"源数据库不存在或不是文件：{source_path}") + _validate_no_sqlite_sidecars(source_path) + with _connect_read_only(source_path) as connection: + _validate_source_schema(connection) + snapshot = _snapshot(connection) + _validate_preflight(snapshot) + return { + "migration_id": MIGRATION_ID, + "source": str(source_path), + "source_sha256": _sha256(source_path), + "snapshot": asdict(snapshot), + "ready": True, + } + + +def _backup_database(source: Path, destination: Path) -> None: + source_connection = _connect_read_only(source) + destination_connection = sqlite3.connect(destination) + try: + source_connection.backup(destination_connection) + destination_connection.commit() + finally: + destination_connection.close() + source_connection.close() + + +def _apply_relationship_migration(database: Path) -> None: + connection = sqlite3.connect(database) + try: + connection.execute("PRAGMA foreign_keys = OFF") + connection.execute("BEGIN IMMEDIATE") + + connection.execute("ALTER TABLE reports RENAME TO reports_p0_2b_legacy") + connection.execute("ALTER TABLE execution_logs RENAME TO execution_logs_p0_2b_legacy") + connection.execute("ALTER TABLE group_runs RENAME TO group_runs_p0_2b_legacy") + + connection.execute( + """ + CREATE TABLE group_runs ( + id INTEGER NOT NULL PRIMARY KEY, + run_id INTEGER NOT NULL REFERENCES runs(id) ON DELETE RESTRICT, + group_id INTEGER REFERENCES groups(id) ON DELETE RESTRICT, + legacy_group_id INTEGER, + identity_state TEXT NOT NULL DEFAULT 'linked', + orphan_reason TEXT NOT NULL DEFAULT '', + provider_used VARCHAR NOT NULL, + message_count INTEGER NOT NULL, + speaker_count INTEGER NOT NULL, + ranking_status VARCHAR NOT NULL, + prompt_status VARCHAR NOT NULL, + error_message VARCHAR NOT NULL, + CHECK ( + ( + identity_state = 'linked' + AND group_id IS NOT NULL + AND legacy_group_id IS NULL + AND orphan_reason = '' + ) + OR + ( + identity_state = 'orphaned' + AND group_id IS NULL + AND legacy_group_id IS NOT NULL + AND orphan_reason = 'historical_group_missing' + ) + ) + ) + """ + ) + connection.execute( + """ + INSERT INTO group_runs ( + id, run_id, group_id, legacy_group_id, identity_state, + orphan_reason, provider_used, message_count, speaker_count, + ranking_status, prompt_status, error_message + ) + SELECT + gr.id, + gr.run_id, + CASE WHEN g.id IS NULL THEN NULL ELSE gr.group_id END, + CASE WHEN g.id IS NULL THEN gr.group_id ELSE NULL END, + CASE WHEN g.id IS NULL THEN 'orphaned' ELSE 'linked' END, + CASE WHEN g.id IS NULL THEN 'historical_group_missing' ELSE '' END, + gr.provider_used, + gr.message_count, + gr.speaker_count, + gr.ranking_status, + gr.prompt_status, + gr.error_message + FROM group_runs_p0_2b_legacy gr + LEFT JOIN groups g ON g.id = gr.group_id + """ + ) + + connection.execute( + """ + CREATE TABLE reports ( + id INTEGER NOT NULL PRIMARY KEY, + group_run_id INTEGER NOT NULL UNIQUE + REFERENCES group_runs(id) ON DELETE RESTRICT, + ranking_text VARCHAR NOT NULL, + prompt_text VARCHAR NOT NULL, + ranking_file VARCHAR NOT NULL, + prompt_file VARCHAR NOT NULL, + poster_file VARCHAR NOT NULL, + poster_status VARCHAR NOT NULL, + email_status VARCHAR NOT NULL, + created_at DATETIME NOT NULL, + updated_at DATETIME NOT NULL + ) + """ + ) + connection.execute( + """ + INSERT INTO reports ( + id, group_run_id, ranking_text, prompt_text, ranking_file, + prompt_file, poster_file, poster_status, email_status, + created_at, updated_at + ) + SELECT + id, group_run_id, ranking_text, prompt_text, ranking_file, + prompt_file, poster_file, poster_status, email_status, + created_at, updated_at + FROM reports_p0_2b_legacy + """ + ) + + connection.execute( + """ + CREATE TABLE execution_logs ( + id INTEGER NOT NULL PRIMARY KEY, + run_id INTEGER REFERENCES runs(id) ON DELETE RESTRICT, + level VARCHAR NOT NULL, + message VARCHAR NOT NULL, + created_at DATETIME NOT NULL + ) + """ + ) + connection.execute( + """ + INSERT INTO execution_logs (id, run_id, level, message, created_at) + SELECT id, run_id, level, message, created_at + FROM execution_logs_p0_2b_legacy + """ + ) + + connection.execute("DROP TABLE reports_p0_2b_legacy") + connection.execute("DROP TABLE execution_logs_p0_2b_legacy") + connection.execute("DROP TABLE group_runs_p0_2b_legacy") + + connection.execute("CREATE INDEX ix_group_runs_run_id ON group_runs(run_id)") + connection.execute("CREATE INDEX ix_group_runs_group_id ON group_runs(group_id)") + connection.execute("CREATE INDEX ix_runs_report_date_status ON runs(report_date, status)") + connection.execute("CREATE INDEX ix_execution_logs_run_id ON execution_logs(run_id)") + connection.execute( + """ + CREATE UNIQUE INDEX uq_groups_wechat_group_id_active + ON groups(wechat_group_id) + WHERE TRIM(wechat_group_id) <> '' AND deleted_at IS NULL + """ + ) + + connection.execute( + """ + CREATE TABLE IF NOT EXISTS schema_migrations ( + migration_id TEXT NOT NULL PRIMARY KEY, + applied_at TEXT NOT NULL, + checksum TEXT NOT NULL + ) + """ + ) + connection.execute( + "INSERT INTO schema_migrations(migration_id, applied_at, checksum) VALUES (?, ?, ?)", + (MIGRATION_ID, datetime.now(timezone.utc).isoformat(), MIGRATION_CHECKSUM), + ) + connection.execute(f"PRAGMA user_version = {TARGET_USER_VERSION}") + connection.commit() + except Exception: + connection.rollback() + raise + finally: + connection.close() + + +def _validate_migrated_database(database: Path, before: DatabaseSnapshot) -> dict[str, Any]: + connection = sqlite3.connect(database) + connection.row_factory = sqlite3.Row + try: + connection.execute("PRAGMA foreign_keys = ON") + integrity_rows = [str(row[0]) for row in connection.execute("PRAGMA integrity_check")] + if integrity_rows != ["ok"]: + raise MigrationError("迁移后完整性检查失败：" + "; ".join(integrity_rows)) + + foreign_key_rows = [tuple(row) for row in connection.execute("PRAGMA foreign_key_check")] + if foreign_key_rows: + raise MigrationError(f"迁移后外键检查失败，共 {len(foreign_key_rows)} 行") + + counts = {table: _scalar(connection, f'SELECT COUNT(*) FROM "{table}"') for table in _CORE_TABLES} + if counts != before.table_counts: + raise MigrationError(f"迁移前后核心表行数不一致：before={before.table_counts}, after={counts}") + + orphaned = _scalar(connection, "SELECT COUNT(*) FROM group_runs WHERE identity_state='orphaned'") + linked = _scalar(connection, "SELECT COUNT(*) FROM group_runs WHERE identity_state='linked'") + invalid_links = _scalar( + connection, + """ + SELECT COUNT(*) + FROM group_runs gr + LEFT JOIN groups g ON g.id = gr.group_id + WHERE gr.group_id IS NOT NULL AND g.id IS NULL + """, + ) + preserved_legacy_ids = _scalar( + connection, + """ + SELECT COUNT(*) + FROM group_runs + WHERE identity_state='orphaned' + AND group_id IS NULL + AND legacy_group_id IS NOT NULL + AND orphan_reason='historical_group_missing' + """, + ) + if orphaned != before.orphan_group_runs or preserved_legacy_ids != before.orphan_group_runs: + raise MigrationError("历史孤儿数量或旧 ID 保留数量不一致") + if linked != before.table_counts["group_runs"] - before.orphan_group_runs: + raise MigrationError("可关联 GroupRun 数量不一致") + if invalid_links: + raise MigrationError(f"迁移后仍存在 {invalid_links} 条无效活动 group_id") + + migration_row = connection.execute( + "SELECT checksum FROM schema_migrations WHERE migration_id=?", + (MIGRATION_ID,), + ).fetchone() + if migration_row is None or str(migration_row[0]) != MIGRATION_CHECKSUM: + raise MigrationError("schema_migrations 记录缺失或 checksum 不一致") + user_version = int(connection.execute("PRAGMA user_version").fetchone()[0]) + if user_version != TARGET_USER_VERSION: + raise MigrationError(f"user_version 不正确：{user_version}") + + foreign_keys = { + table: [dict(row) for row in connection.execute(f'PRAGMA foreign_key_list("{table}")')] + for table in ("group_runs", "reports", "execution_logs") + } + actual_foreign_keys = { + (table, str(row["from"]), str(row["table"]), str(row["to"]), str(row["on_delete"])) + for table, rows in foreign_keys.items() + for row in rows + } + expected_foreign_keys = { + ("group_runs", "run_id", "runs", "id", "RESTRICT"), + ("group_runs", "group_id", "groups", "id", "RESTRICT"), + ("reports", "group_run_id", "group_runs", "id", "RESTRICT"), + ("execution_logs", "run_id", "runs", "id", "RESTRICT"), + } + if actual_foreign_keys != expected_foreign_keys: + raise MigrationError( + f"迁移后外键结构不符合预期：actual={sorted(actual_foreign_keys)}" + ) + return { + "integrity_check": "ok", + "foreign_key_check_rows": 0, + "table_counts": counts, + "linked_group_runs": linked, + "orphaned_group_runs": orphaned, + "preserved_legacy_group_ids": preserved_legacy_ids, + "invalid_linked_group_ids": invalid_links, + "user_version": user_version, + "foreign_keys": foreign_keys, + } + finally: + connection.close() + + +def _write_json(path: Path, payload: dict[str, Any]) -> None: + path.write_text(json.dumps(payload, ensure_ascii=False, indent=2) + "\n", encoding="utf-8") + + +def _resolve_and_validate_targets( + source: str | Path, + output: str | Path, + manifest: str | Path | None, +) -> tuple[Path, Path, Path]: + source_path = Path(source).expanduser().resolve() + output_path = Path(output).expanduser().resolve() + manifest_path = ( + Path(manifest).expanduser().resolve() + if manifest is not None + else output_path.with_suffix(output_path.suffix + ".manifest.json") + ) + if source_path == output_path: + raise MigrationError("源数据库和输出数据库不能是同一个文件") + if manifest_path in {source_path, output_path}: + raise MigrationError("Manifest 路径不能与源数据库或输出数据库相同") + if output_path.exists(): + raise MigrationError(f"输出文件已存在，拒绝覆盖：{output_path}") + if manifest_path.exists(): + raise MigrationError(f"Manifest 已存在，拒绝覆盖：{manifest_path}") + return source_path, output_path, manifest_path + + +def migrate_database( + source: str | Path, + output: str | Path, + *, + manifest: str | Path | None = None, +) -> dict[str, Any]: + """将旧数据库迁移到一个全新的输出文件，并返回验证 Manifest。""" + source_path, output_path, manifest_path = _resolve_and_validate_targets( + source, + output, + manifest, + ) + + preflight = preflight_database(source_path) + before = DatabaseSnapshot(**preflight["snapshot"]) + source_hash_before = str(preflight["source_sha256"]) + + output_path.parent.mkdir(parents=True, exist_ok=True) + manifest_path.parent.mkdir(parents=True, exist_ok=True) + token = uuid.uuid4().hex + temporary_database = output_path.with_name(f".{output_path.name}.{token}.tmp") + temporary_manifest = manifest_path.with_name(f".{manifest_path.name}.{token}.tmp") + output_promoted = False + manifest_promoted = False + try: + _backup_database(source_path, temporary_database) + _apply_relationship_migration(temporary_database) + after = _validate_migrated_database(temporary_database, before) + + source_hash_after = _sha256(source_path) + if source_hash_after != source_hash_before: + raise MigrationError("迁移期间源数据库文件发生变化，拒绝产出结果") + + result: dict[str, Any] = { + "migration_id": MIGRATION_ID, + "migration_checksum": MIGRATION_CHECKSUM, + "created_at": datetime.now(timezone.utc).isoformat(), + "source": str(source_path), + "output": str(output_path), + "manifest": str(manifest_path), + "source_sha256_before": source_hash_before, + "source_sha256_after": source_hash_after, + "source_unchanged": True, + "output_sha256": _sha256(temporary_database), + "before": asdict(before), + "after": after, + } + _write_json(temporary_manifest, result) + os.replace(temporary_database, output_path) + output_promoted = True + os.replace(temporary_manifest, manifest_path) + manifest_promoted = True + return result + except Exception: + for temporary_path in (temporary_database, temporary_manifest): + if temporary_path.exists(): + temporary_path.unlink() + if output_promoted and not manifest_promoted and output_path.exists(): + output_path.unlink() + raise + + +def _build_parser() -> argparse.ArgumentParser: + parser = argparse.ArgumentParser( + description="把 GroupBrief 旧数据库迁移到一个全新的、经过校验的输出副本。", + ) + parser.add_argument("--source", required=True, type=Path, help="只读源数据库路径") + parser.add_argument("--output", required=True, type=Path, help="必须不存在的输出数据库路径") + parser.add_argument("--manifest", type=Path, help="可选 Manifest 路径") + actions = parser.add_mutually_exclusive_group(required=True) + actions.add_argument("--dry-run", action="store_true", help="仅执行只读前置检查") + actions.add_argument("--apply", action="store_true", help="创建并迁移新的输出副本") + return parser + + +def main(argv: Sequence[str] | None = None) -> int: + args = _build_parser().parse_args(argv) + try: + if args.dry_run: + source_path, output_path, _ = _resolve_and_validate_targets( + args.source, + args.output, + args.manifest, + ) + result = preflight_database(source_path) + result["planned_output"] = str(output_path) + else: + result = migrate_database(args.source, args.output, manifest=args.manifest) + except (MigrationError, OSError, sqlite3.Error) as exc: + print(json.dumps({"ok": False, "error": str(exc)}, ensure_ascii=False), file=sys.stderr) + return 2 + print(json.dumps({"ok": True, **result}, ensure_ascii=False, indent=2)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/app/db/repository.py b/app/db/repository.py index 1a18f93..bebc2e0 100644 --- a/app/db/repository.py +++ b/app/db/repository.py @@ -2,28 +2,191 @@ from __future__ import annotations +from datetime import datetime, timezone +from functools import partial from pathlib import Path from typing import Any +from sqlalchemy import event from sqlmodel import Session, SQLModel, create_engine, select from app.config.settings import Settings +from app.db.offline_migrations import MIGRATION_CHECKSUM, MIGRATION_ID, TARGET_USER_VERSION from app.db.models import Group, Report, Run, Setting engine: Any = None +_RELATIONSHIP_COLUMNS = {"legacy_group_id", "identity_state", "orphan_reason"} +_EXPECTED_FOREIGN_KEYS = { + ("group_runs", "run_id", "runs", "id", "RESTRICT"), + ("group_runs", "group_id", "groups", "id", "RESTRICT"), + ("reports", "group_run_id", "group_runs", "id", "RESTRICT"), + ("execution_logs", "run_id", "runs", "id", "RESTRICT"), +} + + +class DatabaseSchemaError(RuntimeError): + """正式数据库尚未迁移或关系 Schema 与当前代码不一致。""" + + +def _enable_sqlite_connection_pragmas( + dbapi_connection, + _connection_record, + *, + busy_timeout_ms: int = 15_000, +) -> None: + """每个 SQLite 连接启用外键，并给短暂锁竞争留出有限等待。""" + cursor = dbapi_connection.cursor() + try: + cursor.execute("PRAGMA foreign_keys = ON") + cursor.execute(f"PRAGMA busy_timeout = {max(int(busy_timeout_ms), 1000)}") + finally: + cursor.close() + + +def _schema_error(detail: str) -> DatabaseSchemaError: + return DatabaseSchemaError( + f"数据库 Schema 与当前 GroupBrief 不兼容：{detail}。" + "请先停止所有写入者，再使用 scripts/migrate_db.py 执行 P0.2B 离线迁移。" + ) + + +def _ensure_relationship_schema_current() -> None: + """接受 Fresh/已迁移数据库，拒绝旧非空或伪迁移数据库。""" + with engine.begin() as connection: + foreign_keys_enabled = int(connection.exec_driver_sql("PRAGMA foreign_keys").scalar_one()) + if foreign_keys_enabled != 1: + raise _schema_error("当前 SQLite 连接没有启用 foreign_keys") + + columns = { + str(row[1]) + for row in connection.exec_driver_sql("PRAGMA table_info(group_runs)") + } + missing_columns = sorted(_RELATIONSHIP_COLUMNS - columns) + if missing_columns: + raise _schema_error("group_runs 缺少列 " + ", ".join(missing_columns)) + + tables = { + str(row[0]) + for row in connection.exec_driver_sql( + "SELECT name FROM sqlite_master WHERE type='table'" + ) + } + has_migration_table = "schema_migrations" in tables + user_version = int(connection.exec_driver_sql("PRAGMA user_version").scalar_one()) + + if not has_migration_table: + core_rows = sum( + int(connection.exec_driver_sql(f'SELECT COUNT(*) FROM "{table}"').scalar_one()) + for table in ("groups", "runs", "group_runs", "reports", "execution_logs") + ) + if core_rows: + raise _schema_error("非空数据库缺少 schema_migrations 记录") + connection.exec_driver_sql( + """ + CREATE TABLE schema_migrations ( + migration_id TEXT NOT NULL PRIMARY KEY, + applied_at TEXT NOT NULL, + checksum TEXT NOT NULL + ) + """ + ) + connection.exec_driver_sql( + "INSERT INTO schema_migrations(migration_id, applied_at, checksum) VALUES (?, ?, ?)", + (MIGRATION_ID, datetime.now(timezone.utc).isoformat(), MIGRATION_CHECKSUM), + ) + connection.exec_driver_sql(f"PRAGMA user_version = {TARGET_USER_VERSION}") + user_version = TARGET_USER_VERSION + + migration = connection.exec_driver_sql( + "SELECT checksum FROM schema_migrations WHERE migration_id=?", + (MIGRATION_ID,), + ).first() + if migration is None: + raise _schema_error(f"缺少迁移记录 {MIGRATION_ID}") + if str(migration[0]) != MIGRATION_CHECKSUM: + raise _schema_error(f"迁移 {MIGRATION_ID} checksum 不一致") + if user_version != TARGET_USER_VERSION: + raise _schema_error( + f"user_version={user_version}，预期 {TARGET_USER_VERSION}" + ) + + actual_foreign_keys = { + (table, str(row[3]), str(row[2]), str(row[4]), str(row[6])) + for table in ("group_runs", "reports", "execution_logs") + for row in connection.exec_driver_sql(f'PRAGMA foreign_key_list("{table}")') + } + if actual_foreign_keys != _EXPECTED_FOREIGN_KEYS: + raise _schema_error("外键结构或删除策略不符合 P0.2B 目标") + + report_unique = False + for index_row in connection.exec_driver_sql("PRAGMA index_list(reports)"): + if not bool(index_row[2]): + continue + index_name = str(index_row[1]).replace('"', '""') + index_columns = [ + str(row[2]) + for row in connection.exec_driver_sql(f'PRAGMA index_info("{index_name}")') + ] + if index_columns == ["group_run_id"]: + report_unique = True + break + if not report_unique: + raise _schema_error("reports.group_run_id 缺少唯一约束") + + required_indexes = { + ("groups", "uq_groups_wechat_group_id_active"): (True, ["wechat_group_id"]), + ("runs", "ix_runs_report_date_status"): (False, ["report_date", "status"]), + ("group_runs", "ix_group_runs_run_id"): (False, ["run_id"]), + ("group_runs", "ix_group_runs_group_id"): (False, ["group_id"]), + ("execution_logs", "ix_execution_logs_run_id"): (False, ["run_id"]), + } + for (table, index_name), (expected_unique, expected_columns) in required_indexes.items(): + index_row = next( + ( + row + for row in connection.exec_driver_sql(f'PRAGMA index_list("{table}")') + if str(row[1]) == index_name + ), + None, + ) + if index_row is None or bool(index_row[2]) != expected_unique: + raise _schema_error(f"缺少索引 {index_name} 或唯一性不一致") + escaped_name = index_name.replace('"', '""') + actual_columns = [ + str(row[2]) + for row in connection.exec_driver_sql( + f'PRAGMA index_info("{escaped_name}")' + ) + ] + if actual_columns != expected_columns: + raise _schema_error(f"索引 {index_name} 列定义不一致") + + active_group_index_sql = connection.exec_driver_sql( + "SELECT sql FROM sqlite_master WHERE type='index' AND name=?", + ("uq_groups_wechat_group_id_active",), + ).scalar_one_or_none() + normalized_index_sql = " ".join(str(active_group_index_sql or "").lower().split()) + expected_predicate = "where trim(wechat_group_id) <> '' and deleted_at is null" + if expected_predicate not in normalized_index_sql: + raise _schema_error("uq_groups_wechat_group_id_active 条件定义不一致") + # V2 群配置扩展列（幂等迁移：仅在列不存在时 ALTER TABLE ADD COLUMN） _V2_GROUP_COLUMNS: dict[str, str] = { - "schedule_rule": "VARCHAR(64) NOT NULL DEFAULT 'weekday_default'", + "schedule_rule": "VARCHAR(64) NOT NULL DEFAULT 'daily_previous_day'", "send_time": "VARCHAR(8) NOT NULL DEFAULT '08:30'", + "summary_provider": "VARCHAR(32) NOT NULL DEFAULT ''", + "prompt_provider": "VARCHAR(32) NOT NULL DEFAULT ''", "summary_model": "VARCHAR(64) NOT NULL DEFAULT 'gpt-5.6-sol'", "prompt_model": "VARCHAR(64) NOT NULL DEFAULT 'gpt-5.6-sol'", "image_enabled": "BOOLEAN NOT NULL DEFAULT 1", "send_target": "VARCHAR(256) NOT NULL DEFAULT ''", "ranking_template": "VARCHAR(64) NOT NULL DEFAULT 'default'", + "ranking_count_policy": "VARCHAR(64) NOT NULL DEFAULT 'all_messages'", + "sender_name_policy": "VARCHAR(64) NOT NULL DEFAULT 'resolved'", "image_prompt_template": "VARCHAR(64) NOT NULL DEFAULT 'default'", - "image_theme": "VARCHAR(64) NOT NULL DEFAULT 'random_preset'", + "image_theme": "VARCHAR(64) NOT NULL DEFAULT 'ai_free'", "image_theme_custom": "VARCHAR(80) NOT NULL DEFAULT ''", "image_prompt_override": "TEXT NOT NULL DEFAULT ''", "wechat_send_enabled": "BOOLEAN NOT NULL DEFAULT 0", @@ -191,9 +354,28 @@ def init_db(settings: Settings) -> Any: db_path.parent.mkdir(parents=True, exist_ok=True) engine = create_engine( f"sqlite:///{db_path}", - connect_args={"check_same_thread": False}, + connect_args={ + "check_same_thread": False, + "timeout": max(int(settings.sqlite_busy_timeout_seconds), 1), + }, ) + busy_timeout_ms = max(int(settings.sqlite_busy_timeout_seconds), 1) * 1000 + event.listen( + engine, + "connect", + partial( + _enable_sqlite_connection_pragmas, + busy_timeout_ms=busy_timeout_ms, + ), + ) + with engine.begin() as connection: + connection.exec_driver_sql( + f"PRAGMA busy_timeout = {busy_timeout_ms}" + ) + connection.exec_driver_sql("PRAGMA journal_mode = WAL") + connection.exec_driver_sql("PRAGMA synchronous = NORMAL") SQLModel.metadata.create_all(engine) + _ensure_relationship_schema_current() _migrate_group_v2_columns() _seed_defaults(settings) # 先种默认值再迁移，确保旧 .env 中的 deepseek-chat / 12000 也会升级。 @@ -231,8 +413,13 @@ def _seed_defaults(settings: Settings) -> None: "codex_home": settings.codex_home, "codex_timeout_seconds": str(settings.codex_timeout_seconds), "codex_generated_images_dir": settings.codex_generated_images_dir, + "image_generation_concurrency": str(settings.image_generation_concurrency), + "image_fallback_font_path": settings.image_fallback_font_path, "wechat_sender_mode": settings.wechat_sender_mode, "wechat_native_action_delay_seconds": str(settings.wechat_native_action_delay_seconds), + "wechat_native_stage_timeout_seconds": str(settings.wechat_native_stage_timeout_seconds), + "wechat_native_submit_timeout_seconds": str(settings.wechat_native_submit_timeout_seconds), + "wechat_native_poll_interval_seconds": str(settings.wechat_native_poll_interval_seconds), "wechat_native_mutex_timeout_seconds": str(settings.wechat_native_mutex_timeout_seconds), "wechat_send_claim_seconds": str(settings.wechat_send_claim_seconds), "wechat_late_send_window_minutes": str(settings.wechat_late_send_window_minutes), diff --git a/app/db/resilience.py b/app/db/resilience.py new file mode 100644 index 0000000..94082b5 --- /dev/null +++ b/app/db/resilience.py @@ -0,0 +1,33 @@ +"""SQLite 短暂 busy/locked 的有限退避。""" + +from __future__ import annotations + +import time +from collections.abc import Callable +from typing import TypeVar + +from sqlalchemy.exc import OperationalError + +T = TypeVar("T") + + +def is_sqlite_busy(exc: BaseException) -> bool: + text = str(exc).lower() + return "database is locked" in text or "database is busy" in text or "sqlite_busy" in text + + +def run_with_sqlite_retry( + operation: Callable[[], T], + *, + max_attempts: int = 3, + base_delay_seconds: float = 0.1, +) -> T: + attempts = min(max(int(max_attempts), 1), 5) + for attempt in range(1, attempts + 1): + try: + return operation() + except OperationalError as exc: + if not is_sqlite_busy(exc) or attempt >= attempts: + raise + time.sleep(max(float(base_delay_seconds), 0.01) * (2 ** (attempt - 1))) + raise AssertionError("unreachable") diff --git a/app/image/codex_generator.py b/app/image/codex_generator.py index 3e68e4e..78de7a6 100644 --- a/app/image/codex_generator.py +++ b/app/image/codex_generator.py @@ -1,8 +1,8 @@ r"""Codex CLI ``$imagegen`` 图片生成器。 主链路使用官方 ``codex exec --json``，认证目录与图片目录统一由 -``CODEX_HOME`` 派生。生成请求通过 Windows 命名互斥锁跨进程串行；只接收 -本次执行后出现且能唯一归属的图片，验证后再原子替换正式文件。 +``CODEX_HOME`` 派生。生成请求通过进程级与 Windows 命名信号量受控并发； +自动流程只接收带匹配 job_id 的结构化回执，其他候选仅供人工恢复。 """ from __future__ import annotations @@ -11,6 +11,7 @@ import hashlib import json import os +import re import signal import shutil import subprocess @@ -18,10 +19,12 @@ import time import uuid from contextlib import contextmanager +from dataclasses import dataclass, field from pathlib import Path from typing import Callable, Iterator from app.config.settings import Settings, get_settings +from app.ai.concurrency import bounded_slot, normalized_limit from app.core.logging import get_logger from app.image.image_task import ImageTaskResult, detect_image_format, verify_image @@ -31,12 +34,40 @@ _POLL_INTERVAL = 0.5 _RECOVERY_POLL_ROUNDS = 30 _MAX_ATTEMPTS = 2 -_ATTEMPT_MANIFEST = ".codex-image-attempt.json" +_ATTEMPT_MANIFEST = "attempt.json" +_JOB_DIR = ".imagegen-jobs" _RESULT_SCHEMA = Path(__file__).with_name("codex_image_result.schema.json") -_PROCESS_IMAGE_LOCK = threading.Lock() -_MUTEX_NAME = "Local\\GroupBrief.CodexImagegen" +_PROCESS_IMAGE_LOCK = threading.Lock() # 兼容旧测试/扩展，不再作为全局单槽锁。 +_MUTEX_NAME = "Local\\GroupBrief.CodexImagegen.v2" _WAIT_OBJECT_0 = 0 _WAIT_ABANDONED = 0x80 +_JOB_ID_RE = re.compile(r"^[A-Za-z0-9_-]{8,80}$") +_THREAD_ID_RE = re.compile(r"^[A-Za-z0-9-]{8,80}$") +_PROCESS_CAPTURE_LIMIT = 256 * 1024 +_DIAGNOSTIC_TAIL_LIMIT = 2000 + + +@dataclass +class _TextTail: + """线程安全的有界文本尾部，避免 Codex 长输出无限占用内存。""" + + limit: int = _PROCESS_CAPTURE_LIMIT + _value: str = "" + _lock: threading.Lock = field(default_factory=threading.Lock) + + def append(self, value: str) -> None: + if not value: + return + with self._lock: + self._value = (self._value + value)[-self.limit :] + + def get(self) -> str: + with self._lock: + return self._value + + +def re_fullmatch_job_id(value: str) -> bool: + return bool(_JOB_ID_RE.fullmatch(value or "")) def _terminate_process_tree(process: subprocess.Popen) -> None: @@ -52,17 +83,17 @@ def _terminate_process_tree(process: subprocess.Popen) -> None: creationflags=getattr(subprocess, "CREATE_NO_WINDOW", 0), ) except (OSError, subprocess.SubprocessError): - pass + logger.warning("Windows Codex 进程树终止失败", exc_info=True) else: try: os.killpg(process.pid, signal.SIGKILL) except OSError: - pass + logger.warning("Codex 进程组终止失败", exc_info=True) if process.poll() is None: try: process.kill() except OSError: - pass + logger.warning("Codex 子进程强制终止失败", exc_info=True) def _run_codex_process( @@ -73,8 +104,9 @@ def _run_codex_process( input: str, env: dict[str, str], on_start: Callable[[int], None] | None = None, + on_event: Callable[[dict], None] | None = None, ) -> subprocess.CompletedProcess: - """运行 Codex；超时时先杀进程树，再回收管道并抛出 TimeoutExpired。""" + """流式运行 Codex；持续解析 JSONL，超时后终止进程树并排空管道。""" popen_kwargs: dict = { "stdin": subprocess.PIPE, "stdout": subprocess.PIPE, @@ -98,46 +130,141 @@ def _run_codex_process( try: process.communicate(timeout=5) except (subprocess.TimeoutExpired, OSError): - pass + logger.warning("on_start 失败后的 Codex 管道回收未完成", exc_info=True) raise + + stdout_tail = _TextTail() + stderr_tail = _TextTail() + + def drain(stream, target: _TextTail, *, parse_jsonl: bool) -> None: + if stream is None: + return + try: + for line in iter(stream.readline, ""): + target.append(line) + if not parse_jsonl or on_event is None: + continue + try: + event = json.loads(line) + except json.JSONDecodeError: + continue + if isinstance(event, dict): + try: + on_event(event) + except Exception: + logger.warning("Codex JSONL 事件记录失败，继续等待进程", exc_info=True) + except (OSError, ValueError): + logger.warning("Codex 输出管道读取异常", exc_info=True) + finally: + try: + stream.close() + except (OSError, ValueError): + logger.debug("Codex 输出管道关闭异常", exc_info=True) + + readers = [ + threading.Thread( + target=drain, + args=(process.stdout, stdout_tail), + kwargs={"parse_jsonl": True}, + name="groupbrief-codex-stdout", + daemon=True, + ), + threading.Thread( + target=drain, + args=(process.stderr, stderr_tail), + kwargs={"parse_jsonl": False}, + name="groupbrief-codex-stderr", + daemon=True, + ), + ] + for reader in readers: + reader.start() + try: - stdout, stderr = process.communicate(input=input, timeout=timeout) - except subprocess.TimeoutExpired: + if process.stdin is not None: + try: + process.stdin.write(input) + process.stdin.flush() + except (BrokenPipeError, OSError): + logger.warning("Codex stdin 写入失败", exc_info=True) + finally: + try: + process.stdin.close() + except (OSError, ValueError): + logger.debug("Codex stdin 关闭异常", exc_info=True) + process.wait(timeout=timeout) + except subprocess.TimeoutExpired as exc: _terminate_process_tree(process) try: - process.communicate(timeout=5) + process.wait(timeout=5) except (subprocess.TimeoutExpired, OSError): try: process.kill() except OSError: - pass - raise - return subprocess.CompletedProcess(command, process.returncode, stdout, stderr) + logger.warning("Codex 超时后的强制终止失败", exc_info=True) + for reader in readers: + reader.join(timeout=5) + raise subprocess.TimeoutExpired( + exc.cmd, + exc.timeout, + output=stdout_tail.get(), + stderr=stderr_tail.get(), + ) from None + finally: + for reader in readers: + reader.join(timeout=5) + return subprocess.CompletedProcess( + command, + process.returncode, + stdout_tail.get(), + stderr_tail.get(), + ) @contextmanager -def _imagegen_mutex(timeout_seconds: float) -> Iterator[None]: - """防止定时、手动和重生成任务跨线程/跨进程同时认领图片。""" - if not _PROCESS_IMAGE_LOCK.acquire(timeout=max(timeout_seconds, 0.1)): - raise TimeoutError("另一个 Codex 生图任务正在运行") +def _imagegen_mutex(timeout_seconds: float, limit: int = 1) -> Iterator[None]: + """兼容旧函数名的跨进程受控并发槽。""" + limit = normalized_limit(limit, 1, maximum=6) handle = None owns_handle = False - try: - if os.name == "nt": - handle = ctypes.windll.kernel32.CreateMutexW(None, False, _MUTEX_NAME) - if not handle: - raise OSError("无法创建 Codex 生图互斥锁") - wait_code = ctypes.windll.kernel32.WaitForSingleObject(handle, int(timeout_seconds * 1000)) - if wait_code not in (_WAIT_OBJECT_0, _WAIT_ABANDONED): - raise TimeoutError("等待 Codex 生图互斥锁超时") - owns_handle = True - yield - finally: - if handle: - if owns_handle: - ctypes.windll.kernel32.ReleaseMutex(handle) - ctypes.windll.kernel32.CloseHandle(handle) - _PROCESS_IMAGE_LOCK.release() + with bounded_slot("codex_image_request", limit): + try: + if os.name == "nt": + kernel32 = ctypes.WinDLL("kernel32", use_last_error=True) + kernel32.CreateSemaphoreW.argtypes = [ + ctypes.c_void_p, + ctypes.c_long, + ctypes.c_long, + ctypes.c_wchar_p, + ] + kernel32.CreateSemaphoreW.restype = ctypes.c_void_p + kernel32.WaitForSingleObject.argtypes = [ctypes.c_void_p, ctypes.c_uint] + kernel32.WaitForSingleObject.restype = ctypes.c_uint + kernel32.ReleaseSemaphore.argtypes = [ + ctypes.c_void_p, + ctypes.c_long, + ctypes.c_void_p, + ] + kernel32.ReleaseSemaphore.restype = ctypes.c_int + kernel32.CloseHandle.argtypes = [ctypes.c_void_p] + kernel32.CloseHandle.restype = ctypes.c_int + handle = kernel32.CreateSemaphoreW( + None, limit, limit, _MUTEX_NAME + ) + if not handle: + raise OSError("无法创建 Codex 生图并发信号量") + wait_code = kernel32.WaitForSingleObject( + handle, int(timeout_seconds * 1000) + ) + if wait_code not in (_WAIT_OBJECT_0, _WAIT_ABANDONED): + raise TimeoutError("等待 Codex 生图并发槽超时") + owns_handle = True + yield + finally: + if handle: + if owns_handle: + kernel32.ReleaseSemaphore(handle, 1, None) + kernel32.CloseHandle(handle) class CodexImageGenerator: @@ -172,6 +299,7 @@ def __init__( self.generated_images_dir = self.codex_home / "generated_images" self._resolved_binary = "" + self._attempt_manifest_lock = threading.RLock() # ---------- 健康检查 ---------- @@ -214,8 +342,8 @@ def health_report(self) -> dict: "generated_images_dir": str(self.generated_images_dir), } - def health_check(self) -> tuple[bool, str]: - report = self.health_report() + def health_check(self, report: dict | None = None) -> tuple[bool, str]: + report = report or self.health_report() if not report["binary"]["ok"]: return False, ( f"codex CLI 不可用（未找到命令：{self.codex_path}）。" @@ -233,31 +361,115 @@ def health_check(self) -> tuple[bool, str]: # ---------- 生成 ---------- - def generate(self, prompt_file: Path, output_path: Path) -> ImageTaskResult: + def generate( + self, + prompt_file: Path, + output_path: Path, + *, + force: bool = False, + job_id: str = "", + revision: int = 1, + prompt_sha256: str = "", + ) -> ImageTaskResult: + # 配置/可执行性错误不应排队等待全局生图锁；进入锁后仍会再次检查， + # 以覆盖等待期间 CLI 状态发生变化的情况。 + ok, detail = self.health_check() + if not ok: + return ImageTaskResult(False, error=detail, detail={"stage": "health"}) try: - with _imagegen_mutex((self.timeout * _MAX_ATTEMPTS) + 60): - return self._generate_locked(Path(prompt_file), Path(output_path)) + with _imagegen_mutex( + (self.timeout * _MAX_ATTEMPTS) + 60, + normalized_limit(self.settings.image_generation_concurrency, 2, maximum=6), + ): + return self._generate_locked( + Path(prompt_file), + Path(output_path), + force=force, + job_id=job_id, + revision=revision, + prompt_sha256=prompt_sha256, + ) except TimeoutError as exc: return ImageTaskResult(False, error=str(exc), detail={"stage": "mutex"}) except Exception as exc: logger.exception("Codex 生图互斥阶段异常") return ImageTaskResult(False, error=str(exc)[:300], detail={"stage": "mutex"}) - def _generate_locked(self, prompt_file: Path, output_path: Path) -> ImageTaskResult: + def can_reconcile_without_generation(self, prompt_file: Path, job_id: str) -> bool: + """仅在已有 attempt 能按 thread_id 找到可信图片时允许调度收口。""" + if not re_fullmatch_job_id(job_id): + return False + task_dir = prompt_file.parent.resolve() + manifest = task_dir / _JOB_DIR / job_id / _ATTEMPT_MANIFEST + attempt = self._load_attempt_manifest(manifest) + if not attempt or attempt.get("state") not in {"running", "result_unknown"}: + return False + thread_id = str(attempt.get("codex_thread_id") or "") + if not _THREAD_ID_RE.fullmatch(thread_id) or attempt.get("codex_thread_conflict"): + return False + try: + pid = int(attempt.get("pid") or 0) + except (TypeError, ValueError): + return False + if self._pid_is_running(pid): + return False + records = self._candidate_records(attempt, task_dir) + selected, _ = self._select_candidate(records) + return selected is not None + + def _generate_locked( + self, + prompt_file: Path, + output_path: Path, + *, + force: bool = False, + job_id: str = "", + revision: int = 1, + prompt_sha256: str = "", + ) -> ImageTaskResult: ok, detail = self.health_check() if not ok: return ImageTaskResult(False, error=detail, detail={"stage": "health"}) if not prompt_file.exists(): return ImageTaskResult(False, error=f"image_prompt.txt 不存在：{prompt_file}", detail={"stage": "input"}) + # ImageJob 的锁外检查只能优化单进程路径；跨进程排队后必须在同一 + # 生图 mutex 内重新确认，避免两个执行者依次重复生成同一张图。 + if not force: + existing_ok, _ = verify_image(output_path) + if existing_ok: + return ImageTaskResult( + True, + image_path=output_path, + detail={ + "stage": "existing", + "recovery_status": "existing_output_reused", + "attempt_count": 0, + }, + ) + try: prompt_text = prompt_file.read_text(encoding="utf-8") except (OSError, UnicodeError) as exc: return ImageTaskResult(False, error=f"无法读取 image_prompt.txt：{exc}", detail={"stage": "input"}) + # 任务哈希按磁盘原始字节创建；Windows 的 read_text 会把 CRLF 规范化 + # 为 LF，不能再对解码后的文本计算哈希，否则内容未变也会被误判。 + actual_prompt_sha256 = hashlib.sha256(prompt_file.read_bytes()).hexdigest() + if prompt_sha256 and prompt_sha256.lower() != actual_prompt_sha256: + return ImageTaskResult( + False, + error="Prompt 内容已变化，拒绝使用旧生图任务", + detail={"stage": "input", "prompt_sha256": actual_prompt_sha256}, + ) + job_id = (job_id or uuid.uuid4().hex).strip() + if not re_fullmatch_job_id(job_id): + return ImageTaskResult(False, error="生图 job_id 格式无效", detail={"stage": "input"}) task_dir = prompt_file.parent.resolve() output_path = output_path.resolve() - manifest_path = task_dir / _ATTEMPT_MANIFEST + job_dir = task_dir / _JOB_DIR / job_id + job_dir.mkdir(parents=True, exist_ok=True) + manifest_path = job_dir / _ATTEMPT_MANIFEST history: list[dict] = [] next_attempt = 1 @@ -265,6 +477,19 @@ def _generate_locked(self, prompt_file: Path, output_path: Path) -> ImageTaskRes if previous: history = list(previous.get("attempt_history") or []) previous_number = self._safe_attempt_number(previous.get("attempt_number")) + if previous.get("state") == "completed" and not force: + return ImageTaskResult( + False, + error="上次生图已确认完成但正式输出缺失，禁止自动重复生成", + detail={ + "stage": "resume", + "outcome_unknown": False, + "attempt_count": previous_number, + "recovery_status": "completed_output_missing", + "candidate_diagnostics": previous.get("candidate_diagnostics") or [], + "attempts": history, + }, + ) if previous.get("state") == "exhausted": return self._attempts_exhausted_result(previous_number, history, previous) if previous.get("state") == "blocked_process": @@ -280,18 +505,30 @@ def _generate_locked(self, prompt_file: Path, output_path: Path) -> ImageTaskRes "attempts": history, }, ) - if previous.get("state") == "retrying": - next_attempt = previous_number + 1 - if previous.get("state") == "running": + if previous.get("state") == "result_unknown": + if not self._ensure_recorded_process_stopped(previous): + return ImageTaskResult( + False, + error="上次 Codex 生图进程仍未确认结束，禁止恢复或重试", + detail={ + "stage": "resume", + "outcome_unknown": True, + "attempt_count": previous_number, + "recovery_status": "result_unknown_process_still_running", + "codex_thread_id": str(previous.get("codex_thread_id") or ""), + "candidate_diagnostics": previous.get("candidate_diagnostics") or [], + "attempts": history, + }, + ) source, diagnostics, reason = self._reconcile_attempt(previous, task_dir) recovered_history = [ *history, - self._attempt_audit(previous, "interrupted", reason, diagnostics), + self._attempt_audit(previous, "result_unknown_reconcile", reason, diagnostics), ] recovered = self._promote_recovered_candidate( source, output_path, - recovery_status="recovered_after_interruption", + recovery_status="recovered_from_result_unknown", attempt_number=previous_number, diagnostics=diagnostics, attempts=recovered_history, @@ -300,21 +537,114 @@ def _generate_locked(self, prompt_file: Path, output_path: Path) -> ImageTaskRes ) if recovered is not None: return recovered + explicit_failure = self._structured_failure_detail( + Path(str(previous.get("result_path") or "")), + str(previous.get("job_id") or ""), + ) + if explicit_failure: + recovered_history.append( + self._attempt_audit( + previous, + "explicit_failure", + explicit_failure, + diagnostics, + ) + ) + return self._explicit_failure_result( + previous, + manifest_path, + explicit_failure, + attempt_number=previous_number, + diagnostics=diagnostics, + attempts=recovered_history, + ) + return ImageTaskResult( + False, + error="上次 Codex 生图已启动但结果未知，禁止自动重复生成", + detail={ + "stage": "resume", + "outcome_unknown": True, + "attempt_count": previous_number, + "recovery_status": "result_unknown_hold", + "codex_thread_id": str(previous.get("codex_thread_id") or ""), + "candidate_diagnostics": diagnostics, + "attempts": recovered_history, + }, + ) + if previous.get("state") == "retrying": + if previous.get("outcome") == "start_failed": + next_attempt = previous_number + 1 + else: + previous.update( + state="result_unknown", + recovery_reason="旧重试记录无法证明外部调用未发生，已转人工复核", + ) + self._write_attempt_manifest(manifest_path, previous) + return ImageTaskResult( + False, + error="旧 Codex 生图重试记录结果未知，禁止自动重复生成", + detail={ + "stage": "resume", + "outcome_unknown": True, + "attempt_count": previous_number, + "recovery_status": "legacy_retry_result_unknown", + "candidate_diagnostics": previous.get("candidate_diagnostics") or [], + "attempts": history, + }, + ) + if previous.get("state") == "running": if not self._ensure_recorded_process_stopped(previous): return ImageTaskResult( False, - error="上次 Codex 生图进程仍未确认结束，已停止自动重试", + error="上次 Codex 生图进程仍未确认结束，已停止自动恢复", detail={ "stage": "resume", "outcome_unknown": True, "attempt_count": previous_number, "recovery_status": "interrupted_process_still_running", - "candidate_diagnostics": diagnostics, + "codex_thread_id": str(previous.get("codex_thread_id") or ""), + "candidate_diagnostics": previous.get("candidate_diagnostics") or [], "attempts": history, }, ) - history = recovered_history - next_attempt = previous_number + 1 + source, diagnostics, reason = self._reconcile_attempt(previous, task_dir) + recovered_history = [ + *history, + self._attempt_audit(previous, "interrupted", reason, diagnostics), + ] + recovered = self._promote_recovered_candidate( + source, + output_path, + recovery_status="recovered_after_interruption", + attempt_number=previous_number, + diagnostics=diagnostics, + attempts=recovered_history, + manifest_path=manifest_path, + attempt=previous, + ) + if recovered is not None: + return recovered + previous.update( + state="result_unknown", + finished_at=datetime_now_iso(), + outcome="interrupted", + recovery_reason="进程已停止但没有可信候选，外部调用结果未知", + attempt_history=recovered_history, + candidate_diagnostics=diagnostics, + ) + self._write_attempt_manifest(manifest_path, previous) + return ImageTaskResult( + False, + error="上次 Codex 生图进程已停止但结果未知，禁止自动重试", + detail={ + "stage": "resume", + "outcome_unknown": True, + "attempt_count": previous_number, + "recovery_status": "interrupted_result_unknown", + "candidate_diagnostics": diagnostics, + "attempts": recovered_history, + }, + ) environment = os.environ.copy() environment["CODEX_HOME"] = str(self.codex_home) @@ -322,7 +652,15 @@ def _generate_locked(self, prompt_file: Path, output_path: Path) -> ImageTaskRes last_diagnostics: list[dict] = [] for attempt_number in range(next_attempt, _MAX_ATTEMPTS + 1): - attempt = self._new_attempt(task_dir, attempt_number, history) + attempt = self._new_attempt( + task_dir, + attempt_number, + history, + job_id=job_id, + revision=revision, + prompt_sha256=actual_prompt_sha256, + job_dir=job_dir, + ) self._write_attempt_manifest(manifest_path, attempt) command = self._build_command(Path(attempt["result_path"])) logger.info( @@ -338,19 +676,32 @@ def _generate_locked(self, prompt_file: Path, output_path: Path) -> ImageTaskRes command, timeout=self.timeout, cwd=str(task_dir), - input=self._attempt_prompt(prompt_text, Path(attempt["staging_path"])), + input=self._attempt_prompt(prompt_text, job_id), env=environment, on_start=lambda pid, record=attempt: self._record_attempt_pid( manifest_path, record, pid ), + on_event=lambda event, record=attempt: self._record_codex_event( + manifest_path, record, event + ), ) exit_code = int(proc.returncode) + self._record_process_diagnostics( + manifest_path, + attempt, + stderr=proc.stderr or "", + ) if proc.returncode != 0: outcome = "nonzero_exit" except FileNotFoundError: outcome = "start_failed" - except subprocess.TimeoutExpired: + except subprocess.TimeoutExpired as exc: outcome = "timeout" + self._record_process_diagnostics( + manifest_path, + attempt, + stderr=str(exc.stderr or ""), + ) except Exception: logger.exception("调用 codex 异常") outcome = "exec_error" @@ -373,6 +724,31 @@ def _generate_locked(self, prompt_file: Path, output_path: Path) -> ImageTaskRes manifest_path=manifest_path, attempt=attempt, ) + explicit_failure = ( + self._structured_failure_detail( + Path(str(attempt.get("result_path") or "")), + str(attempt.get("job_id") or ""), + ) + if source is None + else "" + ) + if explicit_failure: + audit = self._attempt_audit( + attempt, + "explicit_failure", + explicit_failure, + diagnostics, + exit_code, + ) + history.append(audit) + return self._explicit_failure_result( + attempt, + manifest_path, + explicit_failure, + attempt_number=attempt_number, + diagnostics=diagnostics, + attempts=history, + ) audit = self._attempt_audit(attempt, outcome, reason, diagnostics, exit_code) history.append(audit) if recovered is not None: @@ -382,6 +758,48 @@ def _generate_locked(self, prompt_file: Path, output_path: Path) -> ImageTaskRes reason = "唯一候选未通过完整图片验证" last_reason = reason audit["recovery_reason"] = reason + if outcome != "start_failed": + attempt.update( + state="result_unknown", + finished_at=datetime_now_iso(), + outcome="invalid_output", + exit_code=exit_code, + recovery_reason=reason, + attempt_history=history, + candidate_diagnostics=diagnostics, + ) + self._write_attempt_manifest(manifest_path, attempt) + return ImageTaskResult( + False, + error="Codex 已产生候选但验证未通过，结果未知且禁止自动重试", + detail={ + "stage": "verify", + "outcome_unknown": True, + "attempt_count": attempt_number, + "recovery_status": "invalid_candidate_hold", + "codex_thread_id": str(attempt.get("codex_thread_id") or ""), + "codex_event_summary": list( + attempt.get("codex_event_summary") or [] + ), + "codex_stderr_tail": str( + attempt.get("codex_stderr_tail") or "" + ), + "candidate_diagnostics": diagnostics, + "attempts": history, + }, + ) + attempt.update( + state="retrying" if attempt_number < _MAX_ATTEMPTS else "exhausted", + finished_at=datetime_now_iso(), + outcome="invalid_output", + exit_code=exit_code, + recovery_reason=reason, + attempt_history=history, + candidate_diagnostics=diagnostics, + ) + self._write_attempt_manifest(manifest_path, attempt) + self._cleanup_attempt_files(attempt) + continue if outcome == "timeout" and not self._ensure_recorded_process_stopped(attempt): reason = "超时进程树未确认结束，禁止启动重试" @@ -403,6 +821,38 @@ def _generate_locked(self, prompt_file: Path, output_path: Path) -> ImageTaskRes "outcome_unknown": True, "attempt_count": attempt_number, "recovery_status": "timeout_process_still_running", + "codex_thread_id": str(attempt.get("codex_thread_id") or ""), + "codex_event_summary": list(attempt.get("codex_event_summary") or []), + "codex_stderr_tail": str(attempt.get("codex_stderr_tail") or ""), + "candidate_diagnostics": diagnostics, + "attempts": history, + }, + ) + + if outcome != "start_failed": + reason = reason or "外部生图已启动但没有可信结果" + audit["recovery_reason"] = reason + attempt.update( + state="result_unknown", + finished_at=datetime_now_iso(), + outcome=outcome, + exit_code=exit_code, + recovery_reason=reason, + attempt_history=history, + candidate_diagnostics=diagnostics, + ) + self._write_attempt_manifest(manifest_path, attempt) + return ImageTaskResult( + False, + error="Codex 生图已启动但结果未知，已禁止自动重试", + detail={ + "stage": "ambiguous" if len(diagnostics) > 1 else "exec", + "outcome_unknown": True, + "attempt_count": attempt_number, + "recovery_status": "result_unknown_hold", + "codex_thread_id": str(attempt.get("codex_thread_id") or ""), + "codex_event_summary": list(attempt.get("codex_event_summary") or []), + "codex_stderr_tail": str(attempt.get("codex_stderr_tail") or ""), "candidate_diagnostics": diagnostics, "attempts": history, }, @@ -483,25 +933,50 @@ def _safe_attempt_number(value: object) -> int: return 1 @staticmethod - def _attempt_prompt(prompt_text: str, staging_path: Path) -> str: + def _attempt_prompt(prompt_text: str, job_id: str) -> str: return ( "$imagegen " + prompt_text + "\n\n" - "只为本次任务生成一张最终图片。不要读取、引用或复用任何已有图片。" - f"完成后把最终图片复制到这个精确路径：{staging_path.resolve()}。" - "最终回复必须严格符合 output schema，并在 image_path 中返回该最终图片路径。" + f"本次生图任务 ID 是 {job_id}。" + "只为本次任务调用一次 ImageGen，并只生成、选择一张最终图片。" + "优先使用 1024×1536 像素的竖版 2:3 画布；" + "只要图片完整可读，其他竖版尺寸也可以直接采用，不要为了匹配尺寸裁切或拉伸。" + "不要读取、引用或复用任何已有图片，也不要复制或另存生成结果。" + "最终回复必须严格符合 output schema，并在 job_id 中逐字返回本次任务 ID。" + "生成成功时 status=success、image_path 返回 ImageGen 产生图片当前存在的绝对路径、error 为空；" + "生成失败时 status=failed、image_path 为空、error 返回简短失败原因。" + "禁止把错误说明伪装成 image_path。" ) - def _new_attempt(self, task_dir: Path, attempt_number: int, history: list[dict]) -> dict: + def _new_attempt( + self, + task_dir: Path, + attempt_number: int, + history: list[dict], + *, + job_id: str = "", + revision: int = 1, + prompt_sha256: str = "", + job_dir: Path | None = None, + ) -> dict: attempt_id = uuid.uuid4().hex + job_id = job_id or uuid.uuid4().hex + job_dir = job_dir or (task_dir / _JOB_DIR / job_id) + job_dir.mkdir(parents=True, exist_ok=True) return { - "version": 1, + "version": 3, "state": "running", + "job_id": job_id, + "revision": max(1, int(revision)), + "prompt_sha256": prompt_sha256, "attempt_id": attempt_id, "attempt_number": attempt_number, "started_at": datetime_now_iso(), "pid": None, - "staging_path": str(task_dir / f".codex-image-{attempt_id}.png"), - "result_path": str(task_dir / f".codex-image-{attempt_id}.result.json"), + "codex_thread_id": "", + "codex_event_summary": [], + "codex_stderr_tail": "", + "staging_path": str(job_dir / f"candidate-{attempt_id}.png"), + "result_path": str(job_dir / f"receipt-{attempt_id}.json"), "before": self._snapshot(task_dir), "attempt_history": list(history), } @@ -527,8 +1002,70 @@ def _load_attempt_manifest(path: Path) -> dict | None: return None def _record_attempt_pid(self, manifest_path: Path, attempt: dict, pid: int) -> None: - attempt["pid"] = int(pid) - self._write_attempt_manifest(manifest_path, attempt) + with self._attempt_manifest_lock: + attempt["pid"] = int(pid) + self._write_attempt_manifest(manifest_path, attempt) + + def _record_codex_event( + self, + manifest_path: Path, + attempt: dict, + event: dict, + ) -> None: + event_type = str(event.get("type") or "") + if event_type not in {"thread.started", "turn.completed", "turn.failed", "error"}: + return + summary: dict[str, str] = { + "type": event_type, + "observed_at": datetime_now_iso(), + } + if event_type == "thread.started": + thread_id = str(event.get("thread_id") or "").strip() + if not _THREAD_ID_RE.fullmatch(thread_id): + return + summary["thread_id"] = thread_id + elif event_type in {"turn.failed", "error"}: + raw_error = event.get("error") + if isinstance(raw_error, dict): + raw_error = raw_error.get("message") + summary["message"] = self._sanitize_diagnostic(str(raw_error or ""), 300) + + with self._attempt_manifest_lock: + if event_type == "thread.started": + existing = str(attempt.get("codex_thread_id") or "") + thread_id = summary["thread_id"] + if existing and existing != thread_id: + attempt["codex_thread_conflict"] = True + return + attempt["codex_thread_id"] = thread_id + events = list(attempt.get("codex_event_summary") or []) + events.append(summary) + attempt["codex_event_summary"] = events[-20:] + self._write_attempt_manifest(manifest_path, attempt) + + def _record_process_diagnostics( + self, + manifest_path: Path, + attempt: dict, + *, + stderr: str, + ) -> None: + with self._attempt_manifest_lock: + attempt["codex_stderr_tail"] = self._sanitize_diagnostic( + stderr, + _DIAGNOSTIC_TAIL_LIMIT, + ) + self._write_attempt_manifest(manifest_path, attempt) + + @staticmethod + def _sanitize_diagnostic(value: str, limit: int) -> str: + text = str(value or "")[-max(int(limit), 1) :] + text = re.sub( + r"(?i)(api[_ -]?key|authorization|bearer|token|password|cookie)(\s*[:=]\s*)\S+", + r"\1\2[REDACTED]", + text, + ) + return text def _snapshot(self, task_dir: Path) -> dict[str, dict[str, int]]: result: dict[str, dict[str, int]] = {} @@ -579,21 +1116,65 @@ def _is_new_candidate(self, path: Path, task_dir: Path, before: dict) -> bool: or int(old.get("size") or -1) != stat.st_size ) - def _structured_result_path(self, result_path: Path, task_dir: Path) -> Path | None: + @staticmethod + def _structured_result(result_path: Path, expected_job_id: str) -> dict | None: try: parsed = json.loads(result_path.read_text(encoding="utf-8")) except (OSError, json.JSONDecodeError): return None - if not isinstance(parsed, dict) or set(parsed) != {"image_path"}: + if not isinstance(parsed, dict): + return None + if str(parsed.get("job_id") or "") != expected_job_id: + return None + + # 兼容升级前已经落盘的成功回执；新回执必须显式区分成功与失败， + # 避免把“connection failed”之类的错误文字误当成图片路径。 + if set(parsed) == {"job_id", "image_path"}: + raw = parsed.get("image_path") + if not isinstance(raw, str): + return None + return { + "job_id": expected_job_id, + "status": "success", + "image_path": raw, + "error": "", + } + if set(parsed) != {"job_id", "status", "image_path", "error"}: + return None + if any( + not isinstance(parsed.get(key), str) + for key in ("job_id", "status", "image_path", "error") + ): + return None + if parsed["status"] not in {"success", "failed"}: + return None + if parsed["status"] == "success": + if not parsed["image_path"].strip() or parsed["error"].strip(): + return None + elif parsed["image_path"].strip() or not parsed["error"].strip(): + return None + return parsed + + @classmethod + def _structured_result_path(cls, result_path: Path, expected_job_id: str) -> Path | None: + parsed = cls._structured_result(result_path, expected_job_id) + if parsed is None or parsed.get("status") != "success": return None raw = parsed.get("image_path") if not isinstance(raw, str) or not raw.strip(): return None candidate = Path(raw.strip()).expanduser() if not candidate.is_absolute(): - candidate = task_dir / candidate + return None return candidate + @classmethod + def _structured_failure_detail(cls, result_path: Path, expected_job_id: str) -> str: + parsed = cls._structured_result(result_path, expected_job_id) + if parsed is None or parsed.get("status") != "failed": + return "" + return cls._sanitize_diagnostic(str(parsed.get("error") or ""), 500).strip() + @staticmethod def _sha256(path: Path) -> str: digest = hashlib.sha256() @@ -604,12 +1185,17 @@ def _sha256(path: Path) -> str: def _candidate_records(self, attempt: dict, task_dir: Path) -> list[tuple[Path, dict]]: before = attempt.get("before") if isinstance(attempt.get("before"), dict) else {} + codex_thread_id = str(attempt.get("codex_thread_id") or "").strip() + if not _THREAD_ID_RE.fullmatch(codex_thread_id): + codex_thread_id = "" staging_path = Path(str(attempt.get("staging_path") or "")) result_path = Path(str(attempt.get("result_path") or "")) paths: list[tuple[Path, str]] = [] if self._is_new_candidate(staging_path, task_dir, before): paths.append((staging_path, "staging")) - structured = self._structured_result_path(result_path, task_dir) + structured = self._structured_result_path( + result_path, str(attempt.get("job_id") or "") + ) if structured is not None and self._is_new_candidate(structured, task_dir, before): paths.append((structured, "structured")) current = self._snapshot(task_dir) @@ -619,10 +1205,20 @@ def _candidate_records(self, attempt: dict, task_dir: Path) -> list[tuple[Path, continue label, relative = key.split(":", 1) root = task_dir if label == "task" else self.generated_images_dir - paths.append((root / Path(relative), "scan")) + relative_parts = Path(relative).parts + source = ( + "thread" + if label == "generated_images" + and old is None + and codex_thread_id + and relative_parts + and relative_parts[0] == codex_thread_id + else "scan" + ) + paths.append((root / Path(relative), source)) - by_hash: dict[tuple[int, str], tuple[Path, dict]] = {} - priority = {"staging": 0, "structured": 1, "scan": 2} + by_path: dict[Path, tuple[Path, dict]] = {} + priority = {"staging": 0, "structured": 1, "thread": 2, "scan": 3} for path, source in paths: label = self._path_label(path, task_dir) if label is None: @@ -632,7 +1228,7 @@ def _candidate_records(self, attempt: dict, task_dir: Path) -> list[tuple[Path, digest = self._sha256(path) except OSError: continue - key = (stat.st_size, digest) + resolved_path = path.resolve() record = { "source": source, "sources": [source], @@ -642,9 +1238,9 @@ def _candidate_records(self, attempt: dict, task_dir: Path) -> list[tuple[Path, "size_bytes": int(stat.st_size), "sha256": digest, } - existing = by_hash.get(key) + existing = by_path.get(resolved_path) if existing is None: - by_hash[key] = (path.resolve(), record) + by_path[resolved_path] = (resolved_path, record) continue existing_path, existing_record = existing sources = set(existing_record.get("sources") or []) @@ -652,24 +1248,29 @@ def _candidate_records(self, attempt: dict, task_dir: Path) -> list[tuple[Path, existing_record["sources"] = sorted(sources, key=lambda item: priority[item]) if priority[source] < priority[str(existing_record.get("source") or "scan")]: record["sources"] = existing_record["sources"] - by_hash[key] = (path.resolve(), record) + by_path[resolved_path] = (resolved_path, record) else: - by_hash[key] = (existing_path, existing_record) - return sorted(by_hash.values(), key=lambda item: (priority[item[1]["source"]], item[1]["relative_path"])) + by_path[resolved_path] = (existing_path, existing_record) + return sorted( + by_path.values(), + key=lambda item: (priority[item[1]["source"]], item[1]["relative_path"]), + ) @staticmethod def _select_candidate(records: list[tuple[Path, dict]]) -> tuple[Path | None, str]: if not records: return None, "未发现本次尝试新增或修改的有效候选" - staged = [item for item in records if "staging" in item[1].get("sources", [])] - if len(staged) == 1: - return staged[0][0], "staging 路径唯一" structured = [item for item in records if "structured" in item[1].get("sources", [])] if len(structured) == 1: - return structured[0][0], "结构化最终路径唯一" - if len(records) == 1: - return records[0][0], "本次增量候选唯一" - return None, f"发现 {len(records)} 个不同内容候选，无法唯一归属" + return structured[0][0], "job_id 匹配的结构化最终路径唯一" + thread_attributed = [ + item for item in records if "thread" in item[1].get("sources", []) + ] + if len(thread_attributed) == 1: + return thread_attributed[0][0], "Codex thread_id 目录内的新增图片唯一" + if len(thread_attributed) > 1: + return None, f"同一 Codex thread_id 下发现 {len(thread_attributed)} 个候选，禁止自动猜图" + return None, f"发现 {len(records)} 个候选但缺少匹配 job_id 的可信回执，禁止自动猜图" def _reconcile_attempt(self, attempt: dict, task_dir: Path) -> tuple[Path | None, list[dict], str]: last_records: list[tuple[Path, dict]] = [] @@ -692,16 +1293,73 @@ def _attempt_audit( exit_code: int | None = None, ) -> dict: return { + "job_id": str(attempt.get("job_id") or ""), "attempt_id": str(attempt.get("attempt_id") or ""), "attempt_number": int(attempt.get("attempt_number") or 1), "started_at": str(attempt.get("started_at") or ""), "finished_at": datetime_now_iso(), "outcome": outcome, "exit_code": exit_code, + "codex_thread_id": str(attempt.get("codex_thread_id") or ""), "recovery_reason": reason, "candidate_count": len(diagnostics), } + def _explicit_failure_result( + self, + attempt: dict, + manifest_path: Path, + failure_detail: str, + *, + attempt_number: int, + diagnostics: list[dict], + attempts: list[dict], + ) -> ImageTaskResult: + reason = f"Codex ImageGen 明确失败：{failure_detail}" + attempt.update( + state="exhausted", + finished_at=datetime_now_iso(), + outcome="explicit_failure", + recovery_reason=reason, + attempt_history=list(attempts), + candidate_diagnostics=diagnostics, + ) + self._write_attempt_manifest(manifest_path, attempt) + return ImageTaskResult( + False, + error=reason, + detail={ + "stage": "exec", + "outcome_unknown": False, + "attempt_count": attempt_number, + "recovery_status": "explicit_generation_failure", + "codex_thread_id": str(attempt.get("codex_thread_id") or ""), + "codex_event_summary": list(attempt.get("codex_event_summary") or []), + "codex_stderr_tail": str(attempt.get("codex_stderr_tail") or ""), + "candidate_diagnostics": diagnostics, + "attempts": list(attempts), + }, + ) + + def _receipt_source( + self, + source: Path, + diagnostics: list[dict], + ) -> str: + try: + digest = self._sha256(source) + except OSError: + return "" + for record in diagnostics: + if str(record.get("sha256") or "") != digest: + continue + sources = set(record.get("sources") or []) + if "structured" in sources: + return "structured_receipt" + if "thread" in sources: + return "codex_thread_scan" + return "" + def _promote_recovered_candidate( self, source: Path | None, @@ -721,18 +1379,44 @@ def _promote_recovered_candidate( except Exception: logger.warning("候选图片验证失败，将按有限重试策略继续", exc_info=True) return None + post_promote_warnings: list[str] = [] if self._is_project_output(output_path): - self._save_last_smoke(source, output_path, image_detail) + try: + self._save_last_smoke(source, output_path, image_detail) + except Exception as exc: + post_promote_warnings.append(f"smoke_state:{type(exc).__name__}") + logger.warning( + "正式图片已原子落盘，但健康快照写入失败；保留图片成功结果", + exc_info=True, + ) + receipt_source = self._receipt_source(source, diagnostics) + recovered_at = datetime_now_iso() self._cleanup_attempt_files(attempt) - manifest_path.unlink(missing_ok=True) + try: + manifest_path.unlink(missing_ok=True) + except OSError as exc: + post_promote_warnings.append(f"manifest_cleanup:{type(exc).__name__}") + logger.warning( + "正式图片已原子落盘，但尝试清单清理失败；保留图片成功结果", + exc_info=True, + ) return ImageTaskResult( True, image_path=output_path, detail={ + "job_id": str(attempt.get("job_id") or ""), + "revision": int(attempt.get("revision") or 1), + "prompt_sha256": str(attempt.get("prompt_sha256") or ""), "attempt_count": attempt_number, "recovery_status": recovery_status, + "recovered_at": recovered_at, + "receipt_source": receipt_source, + "codex_thread_id": str(attempt.get("codex_thread_id") or ""), + "codex_event_summary": list(attempt.get("codex_event_summary") or []), + "codex_stderr_tail": str(attempt.get("codex_stderr_tail") or ""), "candidate_diagnostics": diagnostics, "attempts": list(attempts), + "post_promote_warnings": post_promote_warnings, **image_detail, }, ) @@ -745,7 +1429,7 @@ def _cleanup_attempt_files(attempt: dict) -> None: try: Path(raw).unlink(missing_ok=True) except OSError: - pass + logger.warning("Codex 尝试临时文件清理失败：%s", raw, exc_info=True) @staticmethod def _pid_is_running(pid: int) -> bool: @@ -797,7 +1481,7 @@ def _ensure_recorded_process_stopped(self, attempt: dict) -> bool: try: os.killpg(pid, signal.SIGKILL) except OSError: - pass + logger.warning("记录中的 Codex 进程组终止失败：pid=%s", pid, exc_info=True) for _ in range(10): if not self._pid_is_running(pid): return True @@ -823,7 +1507,10 @@ def _attempts_exhausted_result(attempt_number: int, history: list[dict], attempt ) @staticmethod - def _promote_valid_image(source: Path, output_path: Path) -> dict: + def _promote_valid_image( + source: Path, + output_path: Path, + ) -> dict: from PIL import Image output_path.parent.mkdir(parents=True, exist_ok=True) @@ -847,6 +1534,7 @@ def _promote_valid_image(source: Path, output_path: Path) -> dict: return { "format": detect_image_format(output_path), "size_bytes": output_path.stat().st_size, + "sha256": CodexImageGenerator._sha256(output_path), "width": width, "height": height, } @@ -868,7 +1556,7 @@ def _load_last_smoke(self) -> dict: if isinstance(parsed, dict): return {"ok": True, **parsed} except (OSError, json.JSONDecodeError): - pass + logger.debug("Codex 生图健康快照不可读", exc_info=True) return {"ok": False, "status": "NOT_RUN", "detail": "尚未完成真实图片生成"} def _save_last_smoke(self, source: Path, output_path: Path, detail: dict) -> None: diff --git a/app/image/codex_image_result.schema.json b/app/image/codex_image_result.schema.json index a31e21c..61c436b 100644 --- a/app/image/codex_image_result.schema.json +++ b/app/image/codex_image_result.schema.json @@ -2,11 +2,21 @@ "$schema": "https://json-schema.org/draft/2020-12/schema", "type": "object", "properties": { - "image_path": { + "job_id": { + "type": "string", + "minLength": 8 + }, + "status": { "type": "string", - "minLength": 1 + "enum": ["success", "failed"] + }, + "image_path": { + "type": "string" + }, + "error": { + "type": "string" } }, - "required": ["image_path"], + "required": ["job_id", "status", "image_path", "error"], "additionalProperties": false } diff --git a/app/image/fact_verification.py b/app/image/fact_verification.py new file mode 100644 index 0000000..3156792 --- /dev/null +++ b/app/image/fact_verification.py @@ -0,0 +1,280 @@ +"""严格群报图片的 OCR 与聊天证据一致性校验。""" + +from __future__ import annotations + +import asyncio +from dataclasses import asdict, dataclass +from datetime import datetime +from difflib import SequenceMatcher +import hashlib +import json +from pathlib import Path +import re +import unicodedata + +from app.ranking.policies import RANKING_POLICY_TEXT_PRIMARY +from app.ai.strict_prompt_contract import STRICT_IMAGE_FACT_MARKER + + +FACT_REVIEW_FILE = "image_fact_review.json" +_NUMERIC_FACT_RE = re.compile( + r"(? bool: + run_path = prompt_file.with_name("run.json") + try: + run = json.loads(run_path.read_text(encoding="utf-8")) + except (OSError, UnicodeError, json.JSONDecodeError): + return False + if not isinstance(run, dict): + return False + return bool( + run.get("image_fact_contract") == "strict_evidence_v1" + or run.get("ranking_count_policy") == RANKING_POLICY_TEXT_PRIMARY + ) + + +def extract_image_text(image_path: Path) -> str: + """复用 Windows 微信发送器的 WinRT OCR，只读取本地图片。""" + from app.sender.wechat_native import WindowsWechatDriver + + lines = asyncio.run(WindowsWechatDriver._ocr_png(image_path.read_bytes())) + return "\n".join(str(line.text or "").strip() for line in lines if line.text) + + +def _compact_text(value: object) -> str: + normalized = unicodedata.normalize("NFKC", str(value or "")).casefold() + return "".join( + char + for char in normalized + if not unicodedata.category(char).startswith(("P", "S", "Z", "C")) + ) + + +def _canonical_number(value: str) -> str: + normalized = re.sub( + r"\s+", + "", + value.replace("，", ".").replace(",", "."), + ).casefold() + normalized = normalized.replace("万元", "w").replace("万", "w") + normalized = normalized.replace("块", "元") + if re.fullmatch(r"0+", normalized): + return "0" + return normalized + + +def _numeric_facts(value: str) -> set[str]: + return {_canonical_number(match.group(0)) for match in _NUMERIC_FACT_RE.finditer(value)} + + +def _numeric_fact_is_allowed(candidate: str, allowed: set[str]) -> bool: + if candidate in allowed: + return True + # 漫画 OCR 常把已知小数或三位数拆成 18/78/8万等片段。 + compact = re.sub(r"[.,，。\s]", "", candidate) + if len(compact) >= 2: + for expected in allowed: + expected_compact = re.sub(r"[.,，。\s]", "", expected) + if compact in expected_compact: + return True + return False + + +def _load_evidence(prompt_file: Path) -> tuple[list[str], str, int]: + prompt = prompt_file.read_text(encoding="utf-8") + visible_prompt = prompt.split(STRICT_IMAGE_FACT_MARKER, 1)[0] + messages_path = prompt_file.with_name("messages.json") + payload = json.loads(messages_path.read_text(encoding="utf-8")) + if not isinstance(payload, list) or not payload: + raise ValueError("messages.json 必须是非空数组") + evidence_lines: list[str] = [ + line.strip() for line in visible_prompt.splitlines() if line.strip() + ] + numeric_evidence_lines: list[str] = [] + for heading in ("群名称", "统计时间", "数据"): + match = re.search( + rf"【{heading}】\s*(.*?)(?=\n【|\Z)", + visible_prompt, + flags=re.DOTALL, + ) + if match: + numeric_evidence_lines.extend( + line.strip() for line in match.group(1).splitlines() if line.strip() + ) + for item in payload: + if not isinstance(item, dict): + continue + for field in ("group_name", "sender_name", "content"): + text = str(item.get(field) or "").strip() + if text: + evidence_lines.append(text) + numeric_evidence_lines.append(text) + ranking_path = prompt_file.with_name("ranking.json") + try: + ranking = json.loads(ranking_path.read_text(encoding="utf-8")) + except (OSError, UnicodeError, json.JSONDecodeError): + ranking = {} + if isinstance(ranking, dict): + ranking_text = json.dumps(ranking, ensure_ascii=False) + evidence_lines.append(ranking_text) + numeric_evidence_lines.append(ranking_text) + return evidence_lines, "\n".join(numeric_evidence_lines), len(payload) + + +def _forbidden_facts(ocr_text: str, message_evidence: str) -> tuple[str, ...]: + compact = unicodedata.normalize("NFKC", ocr_text) + findings: list[str] = [] + checks = ( + ("BMI", re.compile(r"bmi", re.IGNORECASE)), + ("体脂率百分比", re.compile(r"体脂率[^\n]{0,16}(?:\d|%|％)")), + ("体重身高计算式", re.compile(r"\d+(?:[.,]\d+)?\s*÷\s*\d+(?:[.,]\d+)?")), + ("天气天数", re.compile(r"(?:雨|天气)[^\n]{0,16}\d+\s*天|\d+\s*天[^\n]{0,16}(?:雨|天气)")), + ("温度", re.compile(r"\d+(?:[.,]\d+)?\s*(?:℃|°\s*[cC])")), + ) + for label, pattern in checks: + if pattern.search(compact): + findings.append(label) + if "少油少盐" in compact and "少油少盐" not in message_evidence: + findings.append("无聊天依据的饮食改写") + return tuple(findings) + + +def _matches_allowed_text(candidate: str, allowed: list[str], corpus: str) -> bool: + compact = _compact_text(candidate) + if len(compact) < 4: + return True + if compact in corpus: + return True + for expected in allowed: + expected_compact = _compact_text(expected) + if len(expected_compact) < 4: + continue + if compact in expected_compact or expected_compact in compact: + return True + length_ratio = min(len(compact), len(expected_compact)) / max( + len(compact), len(expected_compact) + ) + matcher = SequenceMatcher(None, compact, expected_compact) + if length_ratio >= 0.45 and matcher.ratio() >= 0.6: + return True + longest = matcher.find_longest_match() + if longest.size >= 4 and longest.size / len(compact) >= 0.6: + return True + return False + + +def _looks_like_factual_text(value: str) -> bool: + compact = _compact_text(value) + return any(marker in compact for marker in _FACTUAL_TEXT_MARKERS) + + +def review_image_facts( + prompt_file: Path, + image_path: Path, + *, + ocr_text: str | None = None, +) -> ImageFactReview: + """把 OCR 文字与 Prompt/消息证据对照；严格模式下任何不明项均失败。""" + try: + evidence_lines, numeric_evidence, message_count = _load_evidence(prompt_file) + except (OSError, UnicodeError, json.JSONDecodeError, ValueError) as exc: + return ImageFactReview(False, f"图片事实校验缺少有效证据：{exc}") + + try: + text = extract_image_text(image_path) if ocr_text is None else str(ocr_text) + except Exception as exc: + return ImageFactReview(False, f"图片事实 OCR 不可用：{type(exc).__name__}: {exc}") + if not text.strip(): + return ImageFactReview(False, "图片事实 OCR 未识别到任何文字") + + allowed_blob = "\n".join(evidence_lines) + allowed_numbers = _numeric_facts(numeric_evidence) + # 分镜序号属于版式，不是聊天事实。 + allowed_numbers.update(str(number) for number in range(0, 11)) + unknown_numeric = tuple( + sorted( + ( + item + for item in _numeric_facts(text) + if not _numeric_fact_is_allowed(item, allowed_numbers) + ), + key=lambda item: (len(item), item), + ) + ) + + normalized_lines = [line.strip() for line in text.splitlines() if line.strip()] + allowed_compact = _compact_text(allowed_blob) + unknown_text = tuple( + line[:120] + for line in normalized_lines + if _looks_like_factual_text(line) + and not _matches_allowed_text(line, evidence_lines, allowed_compact) + ) + forbidden_facts = _forbidden_facts(text, numeric_evidence) + digest = hashlib.sha256(image_path.read_bytes()).hexdigest() + if unknown_numeric or unknown_text or forbidden_facts: + problems: list[str] = [] + if forbidden_facts: + problems.append(f"禁止事实：{', '.join(forbidden_facts)}") + if unknown_numeric: + problems.append(f"无证据数字：{', '.join(unknown_numeric[:12])}") + if unknown_text: + problems.append(f"无证据文字：{'；'.join(unknown_text[:8])}") + return ImageFactReview( + False, + "图片事实校验失败：" + "；".join(problems), + text, + unknown_numeric, + unknown_text, + forbidden_facts, + message_count, + digest, + ) + return ImageFactReview( + True, + "图片 OCR 文案均可在 Prompt 或消息证据中找到", + text, + evidence_message_count=message_count, + image_sha256=digest, + ) + + +def write_fact_review(prompt_file: Path, review: ImageFactReview) -> None: + payload = { + **asdict(review), + "checked_at": datetime.now().astimezone().isoformat(), + "ocr_text": review.ocr_text[:8000], + } + prompt_file.with_name(FACT_REVIEW_FILE).write_text( + json.dumps(payload, ensure_ascii=False, indent=2), + encoding="utf-8", + ) diff --git a/app/image/fallback.py b/app/image/fallback.py new file mode 100644 index 0000000..85a9d91 --- /dev/null +++ b/app/image/fallback.py @@ -0,0 +1,244 @@ +"""图片生成的确定性安全化 Prompt 与 Pillow Level 3 信息图。""" + +from __future__ import annotations + +import hashlib +import json +import os +import re +import unicodedata +import uuid +from pathlib import Path +from typing import Any, Mapping + +from PIL import Image, ImageDraw, ImageFont + + +POLICY_ERROR_CODES = frozenset( + {"POLICY_REJECTED", "CONTENT_FILTER", "SAFETY_FILTER", "PROMPT_BLOCKED"} +) + +_SENSITIVE_TERMS = ( + "自杀", + "血腥", + "裸露", + "色情", + "毒品", + "武器制作", + "仇恨言论", +) + + +def _clean_text(value: object) -> str: + text = unicodedata.normalize("NFKC", str(value or "")) + cleaned: list[str] = [] + for char in text: + if char in "\r\n\t": + cleaned.append(" ") + continue + category = unicodedata.category(char) + if category in {"Cc", "Cf"}: + continue + cleaned.append(char) + return re.sub(r"[ ]{2,}", " ", "".join(cleaned)).strip() + + +def image_failure_code(detail: Mapping[str, Any] | None) -> str: + detail = detail if isinstance(detail, Mapping) else {} + for key in ("error_type", "error_code", "code", "status"): + value = str(detail.get(key) or "").strip().upper() + if value in POLICY_ERROR_CODES: + return value + return "" + + +def image_result_is_unknown(detail: Mapping[str, Any] | None) -> bool: + detail = detail if isinstance(detail, Mapping) else {} + return bool( + detail.get("outcome_unknown") + or str(detail.get("recovery_status") or "").lower() + in {"result_unknown", "timeout_process_still_running"} + or str(detail.get("stage") or "").lower() == "ambiguous" + ) + + +def sanitize_image_prompt( + prompt: str, + *, + group_name: str, + ranking: Mapping[str, Any] | None = None, +) -> tuple[str, list[dict[str, str]]]: + """只泛化身份和风险表达，不重新选题，也不改变数字事实。""" + safe = _clean_text(prompt) + redactions: list[dict[str, str]] = [] + clean_group = _clean_text(group_name) + if clean_group and clean_group in safe: + safe = safe.replace(clean_group, "今日群聊") + redactions.append({"type": "group_name", "replacement": "今日群聊"}) + + speakers = [] + if isinstance(ranking, Mapping): + rows = ranking.get("top_speakers") + if isinstance(rows, list): + speakers = [ + _clean_text(row.get("name")) + for row in rows + if isinstance(row, Mapping) and _clean_text(row.get("name")) + ] + for index, name in enumerate(dict.fromkeys(speakers)): + alias = f"群友{chr(ord('A') + min(index, 25))}" + if name in safe: + safe = safe.replace(name, alias) + redactions.append({"type": "nickname", "replacement": alias}) + + for term in _SENSITIVE_TERMS: + if term in safe: + safe = safe.replace(term, "相关风险表达") + redactions.append({"type": "sensitive_expression", "replacement": "相关风险表达"}) + + header = ( + "【安全化版本】仅泛化昵称和可能触发审核的表达；必须保留原日期、" + "数字事实、话题顺序、主要话题与排行榜，不新增人物或事实。\n" + ) + return header + safe, redactions + + +def _font_candidates(explicit: str = "") -> list[Path]: + paths: list[Path] = [] + if explicit: + paths.append(Path(explicit)) + windows = Path(os.environ.get("WINDIR", "C:/Windows")) / "Fonts" + paths.extend( + [ + windows / "msyhbd.ttc", + windows / "msyh.ttc", + windows / "simhei.ttf", + windows / "simsun.ttc", + Path("/usr/share/fonts/truetype/dejavu/DejaVuSans.ttf"), + ] + ) + return paths + + +def _load_font(size: int, explicit: str = "") -> tuple[ImageFont.ImageFont, str]: + for path in _font_candidates(explicit): + if not path.is_file(): + continue + try: + return ImageFont.truetype(str(path), size=size), str(path) + except OSError: + continue + return ImageFont.load_default(), "Pillow/default" + + +def _fit_lines(draw: ImageDraw.ImageDraw, text: str, font, width: int, limit: int) -> list[str]: + text = _clean_text(text) + if not text: + return [] + lines: list[str] = [] + current = "" + for char in text: + candidate = current + char + if current and draw.textlength(candidate, font=font) > width: + lines.append(current) + current = char + if len(lines) >= limit: + break + else: + current = candidate + if current and len(lines) < limit: + lines.append(current) + if len(lines) == limit and sum(len(line) for line in lines) < len(text): + lines[-1] = lines[-1][:-1] + "…" + return lines + + +def _load_json(path: Path) -> dict: + try: + value = json.loads(path.read_text(encoding="utf-8")) + except (OSError, UnicodeError, json.JSONDecodeError): + return {} + return value if isinstance(value, dict) else {} + + +def render_local_infographic( + *, + group_name: str, + run_date: str, + ranking_path: Path, + run_path: Path, + output_path: Path, + font_path: str = "", + failure_class: str = "IMAGE_GENERATION_FAILED", +) -> dict[str, Any]: + """从当天结构化产物渲染固定 1024×1536 PNG，并原子提升。""" + ranking = _load_json(ranking_path) + run = _load_json(run_path) + canvas = Image.new("RGB", (1024, 1536), "#F4F7FB") + draw = ImageDraw.Draw(canvas) + title_font, used_font = _load_font(52, font_path) + section_font, _ = _load_font(34, font_path) + body_font, _ = _load_font(28, font_path) + small_font, _ = _load_font(22, font_path) + + draw.rounded_rectangle((56, 48, 968, 238), radius=28, fill="#173B57") + title = _clean_text(group_name) or "今日群聊" + draw.text((92, 82), title[:22], font=title_font, fill="white") + draw.text((92, 160), f"{run_date} · 简化版数据卡片", font=body_font, fill="#D7EAF7") + + message_count = int(ranking.get("message_count") or run.get("message_count") or 0) + speaker_count = int(ranking.get("speaker_count") or run.get("speaker_count") or 0) + draw.rounded_rectangle((56, 270, 968, 410), radius=24, fill="white") + draw.text((92, 294), f"消息 {message_count}", font=section_font, fill="#173B57") + draw.text((520, 294), f"参与 {speaker_count}", font=section_font, fill="#173B57") + draw.text((92, 355), "外部生图不可用，已自动生成本地信息图", font=small_font, fill="#63778A") + + draw.text((72, 458), "活跃排行", font=section_font, fill="#173B57") + speakers = ranking.get("top_speakers") if isinstance(ranking.get("top_speakers"), list) else [] + max_count = max([int(row.get("count") or 0) for row in speakers if isinstance(row, Mapping)] or [1]) + y = 520 + for index, row in enumerate(speakers[:8]): + if not isinstance(row, Mapping): + continue + name = _clean_text(row.get("name")) or f"群友{index + 1}" + count = int(row.get("count") or 0) + draw.text((84, y), f"{index + 1}. {name[:12]}", font=body_font, fill="#263746") + bar_width = max(12, int(360 * count / max_count)) + draw.rounded_rectangle((480, y + 5, 480 + bar_width, y + 35), radius=14, fill="#55A7D9") + draw.text((864, y), str(count), font=body_font, fill="#263746") + y += 68 + + prompt_meta = run.get("prompt_meta") if isinstance(run.get("prompt_meta"), Mapping) else {} + selection = prompt_meta.get("topic_selection") if isinstance(prompt_meta.get("topic_selection"), Mapping) else {} + candidates = selection.get("candidates") if isinstance(selection.get("candidates"), list) else [] + draw.text((72, 1088), "主要话题", font=section_font, fill="#173B57") + y = 1146 + for row in [item for item in candidates if isinstance(item, Mapping) and item.get("selected")][:3]: + title = _clean_text(row.get("title")) or "当天主要话题" + summary = _clean_text(row.get("summary")) + for line in _fit_lines(draw, f"• {title}：{summary}", body_font, 840, 2): + draw.text((88, y), line, font=body_font, fill="#263746") + y += 40 + y += 16 + + draw.text( + (72, 1464), + f"fallback=L3 · reason={_clean_text(failure_class)[:40]}", + font=small_font, + fill="#7B8792", + ) + output_path.parent.mkdir(parents=True, exist_ok=True) + temp_path = output_path.with_name(f".{output_path.name}.{uuid.uuid4().hex}.tmp.png") + canvas.save(temp_path, format="PNG", optimize=True) + with Image.open(temp_path) as check: + check.load() + if check.size != (1024, 1536): + raise ValueError(f"本地信息图尺寸异常：{check.size}") + os.replace(temp_path, output_path) + return { + "fallback_level": 3, + "image_variant": "pillow", + "fallback_reason": failure_class, + "fallback_font": used_font, + "sha256": hashlib.sha256(output_path.read_bytes()).hexdigest(), + } diff --git a/app/image/image_task.py b/app/image/image_task.py index 9c0e9c0..fffd9f5 100644 --- a/app/image/image_task.py +++ b/app/image/image_task.py @@ -1,22 +1,34 @@ -"""V2 图片生成任务：验证工具 + 串行调度器。 +"""V2 图片生成任务：验证工具 + 受控并发调度器。 - verify_image：文件存在 / 大小 > 0 / 可被识别为常见图片格式（零依赖签名校验）； -- SerialImageQueue：多群图片严格串行——当前群生成成功并确认文件存在， - 才允许开始下一个群；单群失败不阻塞其他群（结果标记失败，继续下一群）； +- SerialImageQueue：兼容旧类名，默认串行并支持显式并发上限； + 单群失败不阻塞其他群（结果标记失败，继续下一群）； - 每个群每天最多 1 张。 """ from __future__ import annotations +import hashlib +import inspect +import json import shutil +from concurrent.futures import ThreadPoolExecutor, as_completed from dataclasses import dataclass from pathlib import Path from time import perf_counter from typing import Any, Callable from app.v2.constants import ( + IMAGE_CONTENT_VERIFICATION_FAILED, IMAGE_GENERATION_FAILED, IMAGE_FILE_MISSING, + PROMPT_FAILED, +) +from app.image.fallback import ( + image_failure_code, + image_result_is_unknown, + render_local_infographic, + sanitize_image_prompt, ) @@ -65,16 +77,53 @@ def detect_image_format(path: Path) -> str | None: def verify_image(path: Path) -> tuple[bool, str]: - """验证生成图片：存在 / 大小>0 / 可解析为图片。""" + """统一强校验：完整解码、格式与扩展名一致、尺寸非零。""" if not path.exists(): return False, f"图片文件不存在：{path}" size = path.stat().st_size if size <= 0: return False, f"图片文件为空（{size} 字节）：{path}" - fmt = detect_image_format(path) - if fmt is None: - return False, f"文件不是可识别的图片格式：{path}" - return True, f"OK：{fmt} 图片，{size} 字节" + if size > 50 * 1024 * 1024: + return False, f"图片文件超过 50MiB：{path}" + try: + from PIL import Image + + with Image.open(path) as image: + detected = str(image.format or "").lower() + image.verify() + with Image.open(path) as image: + image.load() + width, height = image.size + except Exception as exc: + return False, f"图片无法完整解码：{exc}" + if width <= 0 or height <= 0: + return False, f"图片尺寸无效：{width}×{height}" + suffix = path.suffix.lower() + expected_formats = {".png": "png", ".jpg": "jpeg", ".jpeg": "jpeg", ".gif": "gif", ".webp": "webp", ".tif": "tiff", ".tiff": "tiff", ".bmp": "bmp"} + expected = expected_formats.get(suffix) + if expected and detected != expected: + return False, f"图片格式与扩展名不一致：{detected}/{suffix}" + return True, f"OK：{detected} 图片，{size} 字节，尺寸 {width}×{height}" + + +def verify_image_contract(prompt_file: Path, image_path: Path) -> tuple[bool, str]: + """校验群报图片完整性；Prompt 中的画布尺寸仅作为生成偏好。""" + ok, detail = verify_image(image_path) + if not ok: + return ok, detail + from app.image.fact_verification import ( + review_image_facts, + strict_fact_verification_enabled, + write_fact_review, + ) + + if not strict_fact_verification_enabled(prompt_file): + return True, detail + review = review_image_facts(prompt_file, image_path) + write_fact_review(prompt_file, review) + if not review.ok: + return False, review.detail + return True, f"{detail}；{review.detail}" def copy_generated_image(src: Path, dst: Path) -> None: @@ -93,18 +142,84 @@ def __init__( output_path: Path, generator: Any, force: bool = False, + job_id: str = "", + revision: int = 1, + prompt_sha256: str = "", ): self.group_name = group_name self.prompt_file = prompt_file self.output_path = output_path self.generator = generator self.force = force + self.job_id = job_id + self.revision = max(1, int(revision)) + self.prompt_sha256 = prompt_sha256 + + def _call_generator( + self, + prompt_file: Path, + *, + safe_variant: bool = False, + quality_retry: bool = False, + ): + generate_parameters = inspect.signature(self.generator.generate).parameters + prompt_sha256 = hashlib.sha256(prompt_file.read_bytes()).hexdigest() + if safe_variant and self.job_id: + job_id = f"{self.job_id}-safe" + elif quality_retry and self.job_id: + job_id = f"{self.job_id}-quality-2" + else: + job_id = self.job_id + if "job_id" in generate_parameters: + return self.generator.generate( + prompt_file, + self.output_path, + force=self.force, + job_id=job_id, + revision=self.revision + (1 if safe_variant or quality_retry else 0), + prompt_sha256=prompt_sha256, + ) + if "force" in generate_parameters: + return self.generator.generate(prompt_file, self.output_path, force=self.force) + return self.generator.generate(prompt_file, self.output_path) + + def _local_fallback(self, failure_class: str) -> dict: + settings = getattr(self.generator, "settings", None) + detail = render_local_infographic( + group_name=self.group_name, + run_date=self.output_path.parent.name, + ranking_path=self.output_path.parent / "ranking.json", + run_path=self.output_path.parent / "run.json", + output_path=self.output_path, + font_path=str(getattr(settings, "image_fallback_font_path", "") or ""), + failure_class=failure_class, + ) + from app.image.fact_verification import strict_fact_verification_enabled + + if strict_fact_verification_enabled(self.prompt_file): + ok, verification_detail = verify_image_contract( + self.prompt_file, + self.output_path, + ) + if not ok: + if self.output_path.exists(): + self.output_path.unlink() + raise ValueError(verification_detail) + detail["fact_verification"] = verification_detail + return { + "group_name": self.group_name, + "status": "success", + "success": True, + "detail": f"外部生图失败，已生成本地简化信息图：{self.output_path}", + "error_type": "", + "generator_detail": detail, + } def run(self) -> dict: """执行生图并验证落盘。返回结构化结果。""" # 已存在有效图片且非 force：跳过，不重复生成 if not self.force: - ok, _ = verify_image(self.output_path) + ok, _ = verify_image_contract(self.prompt_file, self.output_path) if ok: return { "group_name": self.group_name, @@ -114,33 +229,191 @@ def run(self) -> dict: "error_type": "", } try: - result = self.generator.generate(self.prompt_file, self.output_path) + run_state = json.loads( + (self.output_path.parent / "run.json").read_text(encoding="utf-8") + ) + except (OSError, UnicodeError, json.JSONDecodeError): + run_state = {} + if isinstance(run_state, dict) and run_state.get("image_force_local_fallback"): + try: + return self._local_fallback( + str(run_state.get("prompt_fallback_reason") or PROMPT_FAILED) + ) + except Exception as fallback_exc: + return { + "group_name": self.group_name, + "status": "failed", + "success": False, + "detail": f"本地信息图生成失败：{str(fallback_exc)[:240]}", + "error_type": IMAGE_GENERATION_FAILED, + } + try: + result = self._call_generator(self.prompt_file) except Exception as e: # 生成器内部异常 - return { - "group_name": self.group_name, - "status": "failed", - "success": False, - "detail": str(e)[:300], - "error_type": IMAGE_GENERATION_FAILED, - } + try: + return self._local_fallback(type(e).__name__) + except Exception as fallback_exc: + return { + "group_name": self.group_name, + "status": "failed", + "success": False, + "detail": f"{str(e)[:180]}；本地兜底失败：{str(fallback_exc)[:100]}", + "error_type": IMAGE_GENERATION_FAILED, + } if not result.success: - return { - "group_name": self.group_name, - "status": "failed", - "success": False, - "detail": result.error, - "error_type": IMAGE_GENERATION_FAILED, - "generator_detail": result.detail or {}, - } - ok, detail = verify_image(self.output_path) + generator_detail = result.detail or {} + if image_result_is_unknown(generator_detail): + return { + "group_name": self.group_name, + "status": "failed", + "success": False, + "detail": result.error, + "error_type": IMAGE_GENERATION_FAILED, + "generator_detail": generator_detail, + } + policy_code = image_failure_code(generator_detail) + safe_failure_detail = "" + if policy_code: + try: + ranking_path = self.output_path.parent / "ranking.json" + try: + ranking = json.loads(ranking_path.read_text(encoding="utf-8")) + except (OSError, UnicodeError, json.JSONDecodeError): + ranking = {} + safe_prompt, redactions = sanitize_image_prompt( + self.prompt_file.read_text(encoding="utf-8"), + group_name=self.group_name, + ranking=ranking, + ) + safe_path = self.prompt_file.with_name("image_prompt.safe.txt") + safe_path.write_text(safe_prompt, encoding="utf-8") + safe_result = self._call_generator(safe_path, safe_variant=True) + if safe_result.success: + ok, detail = verify_image_contract(safe_path, self.output_path) + if ok: + safe_detail = dict(safe_result.detail or {}) + safe_detail.update( + fallback_level=2, + image_variant="safe", + safety_error_code=policy_code, + safety_redactions=redactions, + ) + return { + "group_name": self.group_name, + "status": "success", + "success": True, + "detail": f"安全化 Prompt 生图成功：{detail}", + "error_type": "", + "generator_detail": safe_detail, + } + if image_result_is_unknown(safe_result.detail or {}): + return { + "group_name": self.group_name, + "status": "failed", + "success": False, + "detail": safe_result.error, + "error_type": IMAGE_GENERATION_FAILED, + "generator_detail": safe_result.detail or {}, + } + safe_failure_detail = str(safe_result.error or "安全化 Prompt 生图失败")[:160] + except Exception as safe_exc: + safe_failure_detail = f"安全化 Prompt 阶段异常：{str(safe_exc)[:140]}" + try: + return self._local_fallback(policy_code or IMAGE_GENERATION_FAILED) + except Exception as fallback_exc: + return { + "group_name": self.group_name, + "status": "failed", + "success": False, + "detail": ( + f"{result.error}；{safe_failure_detail}；" + f"本地兜底失败：{str(fallback_exc)[:120]}" + ), + "error_type": IMAGE_GENERATION_FAILED, + "generator_detail": generator_detail, + } + ok, detail = verify_image_contract(self.prompt_file, self.output_path) if not ok: - return { - "group_name": self.group_name, - "status": "failed", - "success": False, - "detail": f"生成后验证失败：{detail}", - "error_type": IMAGE_FILE_MISSING, - } + from app.image.fact_verification import strict_fact_verification_enabled + + if strict_fact_verification_enabled(self.prompt_file): + try: + if self.output_path.exists(): + self.output_path.unlink() + retry_result = self._call_generator( + self.prompt_file, + quality_retry=True, + ) + except Exception as exc: + try: + return self._local_fallback(IMAGE_CONTENT_VERIFICATION_FAILED) + except Exception as fallback_exc: + return { + "group_name": self.group_name, + "status": "failed", + "success": False, + "detail": ( + f"第一次{detail}；第二次生图异常：{str(exc)[:140]}；" + f"本地兜底失败：{str(fallback_exc)[:100]}" + ), + "error_type": IMAGE_CONTENT_VERIFICATION_FAILED, + } + if retry_result.success: + retry_ok, retry_detail = verify_image_contract( + self.prompt_file, + self.output_path, + ) + if retry_ok: + retry_meta = dict(retry_result.detail or {}) + retry_meta["fact_verification_retry"] = 2 + return { + "group_name": self.group_name, + "status": "success", + "success": True, + "detail": f"第二次生图通过事实校验：{retry_detail}", + "error_type": "", + "generator_detail": retry_meta, + } + retry_error = retry_detail + else: + retry_error = retry_result.error or "第二次生图失败" + if image_result_is_unknown(retry_result.detail or {}): + if self.output_path.exists(): + self.output_path.unlink() + return { + "group_name": self.group_name, + "status": "failed", + "success": False, + "detail": f"第一次{detail}；第二次{retry_error}", + "error_type": IMAGE_GENERATION_FAILED, + "generator_detail": retry_result.detail or {}, + } + if self.output_path.exists(): + self.output_path.unlink() + try: + return self._local_fallback(IMAGE_CONTENT_VERIFICATION_FAILED) + except Exception as fallback_exc: + return { + "group_name": self.group_name, + "status": "failed", + "success": False, + "detail": ( + f"第一次{detail}；第二次{retry_error}；" + f"本地兜底失败：{str(fallback_exc)[:100]}" + ), + "error_type": IMAGE_CONTENT_VERIFICATION_FAILED, + "generator_detail": retry_result.detail or {}, + } + try: + return self._local_fallback(IMAGE_FILE_MISSING) + except Exception as fallback_exc: + return { + "group_name": self.group_name, + "status": "failed", + "success": False, + "detail": f"生成后验证失败：{detail}；本地兜底失败：{str(fallback_exc)[:100]}", + "error_type": IMAGE_FILE_MISSING, + } return { "group_name": self.group_name, "status": "success", @@ -152,23 +425,72 @@ def run(self) -> dict: class SerialImageQueue: - """严格串行生图队列（多群共用单队列）。""" + """兼容旧类名的受控图片队列；默认串行，可显式设置并发上限。""" - def __init__(self, run_hook: Callable[[ImageJob, dict], None] | None = None): + def __init__( + self, + run_hook: Callable[[ImageJob, dict], None] | None = None, + *, + max_workers: int = 1, + ): # run_hook：每群完成后回调（用于写 run.json / 日志） self.run_hook = run_hook + self.max_workers = max(1, int(max_workers)) + + @staticmethod + def _run_timed(job: ImageJob) -> dict: + started_at = perf_counter() + result = job.run() + result["imagegen_ms"] = round((perf_counter() - started_at) * 1000) + return result + + def _apply_hook(self, job: ImageJob, result: dict) -> dict: + if not self.run_hook: + return result + try: + self.run_hook(job, result) + except Exception as exc: + result.update( + status="failed", + success=False, + error_type="IMAGE_STATE_PERSIST_FAILED", + detail=f"图片结果状态持久化失败：{type(exc).__name__}: {str(exc)[:220]}", + hook_error=True, + ) + return result def run_all(self, jobs: list[ImageJob]) -> list[dict]: - """按顺序逐群执行；上一群完成（成功或失败）后才执行下一群。""" - results: list[dict] = [] - for job in jobs: - started_at = perf_counter() - result = job.run() - result["imagegen_ms"] = round((perf_counter() - started_at) * 1000) - results.append(result) - if self.run_hook: + """受控并发执行，结果顺序始终与输入任务一致。""" + if not jobs: + return [] + if self.max_workers == 1 or len(jobs) == 1: + results: list[dict] = [] + for job in jobs: + result = self._run_timed(job) + results.append(self._apply_hook(job, result)) + return results + + results_by_index: dict[int, dict] = {} + with ThreadPoolExecutor( + max_workers=min(self.max_workers, len(jobs)), + thread_name_prefix="groupbrief-image", + ) as executor: + futures = { + executor.submit(self._run_timed, job): (index, job) + for index, job in enumerate(jobs) + } + for future in as_completed(futures): + index, job = futures[future] try: - self.run_hook(job, result) - except Exception: - pass - return results + result = future.result() + except Exception as exc: + result = { + "group_name": job.group_name, + "status": "failed", + "success": False, + "detail": str(exc)[:300], + "error_type": IMAGE_GENERATION_FAILED, + "imagegen_ms": 0, + } + results_by_index[index] = self._apply_hook(job, result) + return [results_by_index[index] for index in range(len(jobs))] diff --git a/app/image/regeneration.py b/app/image/regeneration.py index 385acfc..f2f307b 100644 --- a/app/image/regeneration.py +++ b/app/image/regeneration.py @@ -1,25 +1,37 @@ -"""运行级图片重新生成：持久化状态、全局串行、失败保留旧图。""" +"""运行级图片重画：稳定任务身份、受控并发、失败关闭与人工候选认领。""" from __future__ import annotations +import hashlib +import inspect +import os import re import shutil import threading +import uuid from concurrent.futures import ThreadPoolExecutor from datetime import datetime from pathlib import Path from typing import Any +from app.ai.concurrency import bounded_slot, normalized_limit from app.config.settings import Settings +from app.core.logging import get_logger from app.image.codex_generator import CodexImageGenerator -from app.image.image_task import verify_image -from app.v2.constants import READY_TO_SEND, SENT +from app.image.fact_verification import strict_fact_verification_enabled +from app.image.image_task import ImageTaskResult, verify_image_contract +from app.v2.constants import ( + IMAGE_FILE_MISSING, + IMAGE_GENERATION_FAILED, + READY_TO_SEND, + SENT, +) from app.v2.run_store import RunStore -_EXECUTOR = ThreadPoolExecutor(max_workers=1, thread_name_prefix="groupbrief-image-regen") +_EXECUTOR = ThreadPoolExecutor(max_workers=6, thread_name_prefix="groupbrief-image-regen") _ACTIVE_LOCK = threading.Lock() _ACTIVE: set[str] = set() -_POLICY_RE = re.compile(r"policy|safety|moderation|违规|安全策略|不允许", re.IGNORECASE) +logger = get_logger("groupbrief.image.regeneration") def _job_key(group_name: str, run_date: str) -> str: @@ -30,12 +42,253 @@ def _now() -> str: return datetime.now().strftime("%Y-%m-%d %H:%M:%S") +def _prompt_sha256(path: Path) -> str: + return hashlib.sha256(path.read_bytes()).hexdigest() + + +def _sha256(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as stream: + for chunk in iter(lambda: stream.read(1024 * 1024), b""): + digest.update(chunk) + return digest.hexdigest().upper() + + +def _job_revision(run: dict[str, Any]) -> int: + previous = run.get("image_regen_job") + if not isinstance(previous, dict): + return 1 + try: + return max(1, int(previous.get("revision") or 0) + 1) + except (TypeError, ValueError): + return 1 + + +def _prepare_job( + store: RunStore, + group_name: str, + run_date: str, + *, + expected_group_id: int | None = None, + expected_wechat_group_id: str = "", +) -> dict[str, Any]: + run = store.load_run(group_name, run_date) + if expected_group_id is not None: + try: + actual_group_id = int(run.get("group_id")) + except (TypeError, ValueError) as exc: + raise ValueError("run.json 缺少稳定 group_id") from exc + if actual_group_id != int(expected_group_id): + raise ValueError("目标 group_id 与 run.json 不匹配") + if expected_wechat_group_id: + actual_wechat_group_id = str(run.get("wechat_group_id") or "").strip() + if actual_wechat_group_id != expected_wechat_group_id.strip(): + raise ValueError("目标 wechat_group_id 与 run.json 不匹配") + + prompt_path = store.prompt_path(group_name, run_date) + job_id = uuid.uuid4().hex + revision = _job_revision(run) + job = { + "job_id": job_id, + "revision": revision, + "prompt_sha256": _prompt_sha256(prompt_path), + "group_id": run.get("group_id"), + "wechat_group_id": run.get("wechat_group_id") or "", + "group_name": group_name, + "run_date": run_date, + "status": "queued", + "requested_at": _now(), + "attempt_dir": f".imagegen-jobs/{job_id}", + "receipt": {}, + "candidates": [], + } + store.update( + group_name, + run_date, + image_regen_job=job, + image_regen_status="queued", + image_regen_error="", + image_regen_requested_at=job["requested_at"], + desktop_regen_requested=False, + send_hold=True, + needs_manual_send=True, + ) + return job + + +def _merge_job( + store: RunStore, + group_name: str, + run_date: str, + job_id: str, + **fields: Any, +) -> dict[str, Any]: + run = store.load_run(group_name, run_date) + current = run.get("image_regen_job") + if not isinstance(current, dict) or str(current.get("job_id") or "") != job_id: + raise RuntimeError("生图任务身份已变化，拒绝写入旧任务结果") + merged = {**current, **fields} + store.update(group_name, run_date, image_regen_job=merged) + return merged + + +def _invoke_generator( + generator: Any, + prompt_path: Path, + output_path: Path, + job: dict[str, Any], +) -> ImageTaskResult: + parameters = inspect.signature(generator.generate).parameters + if "job_id" in parameters: + return generator.generate( + prompt_path, + output_path, + force=True, + job_id=str(job["job_id"]), + revision=int(job["revision"]), + prompt_sha256=str(job["prompt_sha256"]), + ) + if "force" in parameters: + return generator.generate(prompt_path, output_path, force=True) + return generator.generate(prompt_path, output_path) + + +def _enforce_job_identity( + generator: Any, + result: ImageTaskResult, + job: dict[str, Any], +) -> ImageTaskResult: + """支持稳定任务回执的生成器必须返回与本次尝试一致的身份。""" + detail = result.detail if isinstance(result.detail, dict) else {} + supports_identity = "job_id" in inspect.signature(generator.generate).parameters + if not result.success or not supports_identity: + return result + if ( + str(detail.get("job_id") or "") == str(job["job_id"]) + and str(detail.get("prompt_sha256") or "").lower() + == str(job["prompt_sha256"]).lower() + ): + return result + return ImageTaskResult( + False, + error="生图回执的 job_id 或 Prompt 哈希不匹配，拒绝自动认领", + detail={ + **detail, + "stage": "ambiguous", + "outcome_unknown": True, + "candidate_diagnostics": detail.get("candidate_diagnostics") or [], + }, + ) + + +def normalize_candidate_diagnostics(detail: dict[str, Any]) -> list[dict[str, Any]]: + raw = detail.get("candidate_diagnostics") + if not isinstance(raw, list): + return [] + candidates: list[dict[str, Any]] = [] + for item in raw: + if not isinstance(item, dict): + continue + sha = str(item.get("sha256") or "").upper() + root = str(item.get("root") or "") + relative = str(item.get("relative_path") or "") + if not re.fullmatch(r"[A-F0-9]{64}", sha) or root not in {"task", "generated_images"} or not relative: + continue + candidates.append( + { + "candidate_id": sha.lower(), + "sha256": sha, + "root": root, + "relative_path": relative, + "size_bytes": int(item.get("size_bytes") or 0), + "sources": list(item.get("sources") or []), + } + ) + return candidates + + +def _failure_status(detail: dict[str, Any], candidates: list[dict[str, Any]]) -> str: + if candidates or str(detail.get("stage") or "") == "ambiguous": + return "ambiguous_result" + if bool(detail.get("outcome_unknown")): + return "result_unknown" + return "failed" + + +def _promote_image( + store: RunStore, + group_name: str, + run_date: str, + source: Path, + *, + recovery_status: str, + generator_detail: dict[str, Any] | None = None, +) -> dict[str, Any]: + generator_meta = generator_detail if isinstance(generator_detail, dict) else {} + prompt_path = store.prompt_path(group_name, run_date) + ok, contract_detail = verify_image_contract(prompt_path, source) + if not ok: + raise ValueError(contract_detail) + target = store.image_path(group_name, run_date) + previous = store.previous_image_path(group_name, run_date) + staging = store.regenerating_image_path(group_name, run_date) + source = source.resolve() + staging = staging.resolve() + if source != staging: + if staging.exists(): + staging.unlink() + shutil.copy2(source, staging) + if target.is_file() and target.stat().st_size > 0: + shutil.copy2(target, previous) + os.replace(staging, target) + current = store.load_run(group_name, run_date) + next_status = SENT if current.get("status") == SENT else READY_TO_SEND + success_fields: dict[str, Any] = {} + if str(current.get("error_type") or "") in {IMAGE_GENERATION_FAILED, IMAGE_FILE_MISSING}: + success_fields.update(error=None, error_type=None, failed_stage=None) + store.update( + group_name, + run_date, + status=next_status, + image_regen_status="ready_for_review", + image_regen_error="", + image_regen_finished_at=_now(), + image_regenerated_at=_now(), + desktop_regen_requested=False, + send_hold=True, + needs_manual_send=True, + image_status="regenerated", + image_error=None, + image_recovery_status=recovery_status, + image_recovered_at=str(generator_meta.get("recovered_at") or ""), + image_receipt_source=str(generator_meta.get("receipt_source") or ""), + image_attempt_count=int(generator_meta.get("attempt_count") or 0), + image_attempts=generator_meta.get("attempts") or [], + image_fallback_level=int(generator_meta.get("fallback_level") or 0), + image_variant=str(generator_meta.get("image_variant") or "normal"), + image_fallback_reason=str(generator_meta.get("fallback_reason") or ""), + image_fallback_font=str(generator_meta.get("fallback_font") or ""), + image_safety_redactions=generator_meta.get("safety_redactions") or [], + image_force_local_fallback=False, + codex_thread_id=str(generator_meta.get("codex_thread_id") or ""), + codex_event_summary=generator_meta.get("codex_event_summary") or [], + codex_stderr_tail=str(generator_meta.get("codex_stderr_tail") or ""), + image_candidate_diagnostics=generator_meta.get("candidate_diagnostics") or [], + image_size_bytes=target.stat().st_size, + image_sha256=_sha256(target), + **success_fields, + ) + return store.load_run(group_name, run_date) + + def enqueue_regeneration( settings: Settings, group_name: str, run_date: str, *, generator: Any | None = None, + expected_group_id: int | None = None, + expected_wechat_group_id: str = "", ) -> dict[str, Any]: store = RunStore(settings.output_dir) if not store.run_path(group_name, run_date).exists(): @@ -49,96 +302,199 @@ def enqueue_regeneration( if key in _ACTIVE or run.get("image_regen_status") in {"queued", "running"}: raise RuntimeError("该运行已在重新生图队列中") _ACTIVE.add(key) - - store.update( - group_name, - run_date, - image_regen_status="queued", - image_regen_error="", - image_regen_requested_at=_now(), - desktop_regen_requested=False, - send_hold=True, - ) - _EXECUTOR.submit( - _run_regeneration, - settings, - group_name, - run_date, - generator or CodexImageGenerator(settings=settings), - ) + try: + job = _prepare_job( + store, + group_name, + run_date, + expected_group_id=expected_group_id, + expected_wechat_group_id=expected_wechat_group_id, + ) + _EXECUTOR.submit( + _run_regeneration, + settings, + group_name, + run_date, + generator or CodexImageGenerator(settings=settings), + job, + ) + except Exception: + with _ACTIVE_LOCK: + _ACTIVE.discard(key) + raise return store.load_run(group_name, run_date) -def _fallback_allowed(error: str, detail: dict[str, Any] | None) -> bool: - if _POLICY_RE.search(error or ""): - return False - stage = str((detail or {}).get("stage") or "") - return stage in {"health", "exec", "save", "copy"} - - def _run_regeneration( settings: Settings, group_name: str, run_date: str, generator: Any, + job: dict[str, Any], ) -> None: store = RunStore(settings.output_dir) key = _job_key(group_name, run_date) temp_path = store.regenerating_image_path(group_name, run_date) + job_id = str(job["job_id"]) try: if temp_path.exists(): temp_path.unlink() - store.update(group_name, run_date, image_regen_status="running", image_regen_started_at=_now()) - result = generator.generate(store.prompt_path(group_name, run_date), temp_path) + started_at = _now() + _merge_job(store, group_name, run_date, job_id, status="running", started_at=started_at) + store.update( + group_name, + run_date, + image_regen_status="running", + image_regen_started_at=started_at, + send_hold=True, + needs_manual_send=True, + ) + limit = normalized_limit(getattr(settings, "image_generation_concurrency", 2), 2, maximum=6) + prompt_path = store.prompt_path(group_name, run_date) + with bounded_slot("image_regeneration", limit): + result = _enforce_job_identity( + generator, + _invoke_generator(generator, prompt_path, temp_path, job), + job, + ) + if result.success and strict_fact_verification_enabled(prompt_path): + first_ok, first_detail = verify_image_contract(prompt_path, temp_path) + if not first_ok: + if temp_path.exists(): + temp_path.unlink() + retry_job = { + **job, + "job_id": f"{job_id}-quality-2", + "revision": int(job["revision"]) + 1, + } + retry_result = _enforce_job_identity( + generator, + _invoke_generator(generator, prompt_path, temp_path, retry_job), + retry_job, + ) + if retry_result.success: + retry_ok, retry_detail = verify_image_contract( + prompt_path, + temp_path, + ) + if retry_ok: + retry_meta = dict(retry_result.detail or {}) + retry_meta.update( + generation_attempt_job_id=str( + retry_meta.get("job_id") or retry_job["job_id"] + ), + job_id=job_id, + revision=int(job["revision"]), + prompt_sha256=str(job["prompt_sha256"]), + fact_verification_retry=2, + ) + result = ImageTaskResult( + True, + image_path=temp_path, + detail=retry_meta, + ) + else: + result = ImageTaskResult( + False, + error=f"第一次{first_detail}；第二次{retry_detail}", + detail={ + **dict(retry_result.detail or {}), + "stage": "fact_verify", + "fact_verification_retry": 2, + }, + ) + else: + result = ImageTaskResult( + False, + error=( + f"第一次{first_detail}；第二次" + f"{retry_result.error or '生图失败'}" + ), + detail={ + **dict(retry_result.detail or {}), + "fact_verification_retry": 2, + }, + ) + detail = result.detail if isinstance(result.detail, dict) else {} + candidates = normalize_candidate_diagnostics(detail) + receipt = { + "success": bool(result.success), + "job_id": str(detail.get("job_id") or job_id), + "revision": int(detail.get("revision") or job["revision"]), + "prompt_sha256": str(detail.get("prompt_sha256") or job["prompt_sha256"]), + "recovery_status": str(detail.get("recovery_status") or ""), + "width": detail.get("width"), + "height": detail.get("height"), + "sha256": detail.get("sha256"), + "attempt_count": int(detail.get("attempt_count") or 0), + "receipt_source": str(detail.get("receipt_source") or ""), + "codex_thread_id": str(detail.get("codex_thread_id") or ""), + } if not result.success: - fallback = _fallback_allowed(result.error, result.detail) - store.update( + status = _failure_status(detail, candidates) + _merge_job( + store, group_name, run_date, - image_regen_status="fallback_queued" if fallback else "failed", - image_regen_error=result.error[:500], - image_regen_finished_at=_now(), - desktop_regen_requested=fallback, - send_hold=True, + job_id, + status=status, + finished_at=_now(), + error=result.error[:500], + receipt=receipt, + candidates=candidates, ) - return - - ok, detail = verify_image(temp_path) - if not ok: store.update( group_name, run_date, - image_regen_status="fallback_queued", - image_regen_error=detail[:500], + image_regen_status=status, + image_regen_error=result.error[:500], image_regen_finished_at=_now(), - desktop_regen_requested=True, + desktop_regen_requested=False, send_hold=True, + needs_manual_send=True, ) return - target = store.image_path(group_name, run_date) - previous = store.previous_image_path(group_name, run_date) - if target.is_file() and target.stat().st_size > 0: - shutil.copy2(target, previous) - temp_path.replace(target) - current = store.load_run(group_name, run_date) - next_status = SENT if current.get("status") == SENT else READY_TO_SEND - store.update( + run = _promote_image( + store, group_name, run_date, - status=next_status, - image_regen_status="ready_for_review", - image_regen_error="", - image_regen_finished_at=_now(), - image_regenerated_at=_now(), - desktop_regen_requested=False, - send_hold=True, - needs_manual_send=True, - text_sent_at="", - image_status="regenerated", - image_error=None, + temp_path, + recovery_status="job_receipt_matched", + generator_detail=detail, + ) + _merge_job( + store, + group_name, + run_date, + job_id, + status="ready_for_review", + finished_at=_now(), + receipt={ + **receipt, + "success": True, + "sha256": run.get("image_sha256"), + }, + candidates=[], ) except Exception as exc: + try: + _merge_job( + store, + group_name, + run_date, + job_id, + status="failed", + finished_at=_now(), + error=str(exc)[:500], + ) + except Exception: + logger.exception( + "重生图失败状态无法写入 job ledger：group=%s date=%s job=%s", + group_name, + run_date, + job_id, + ) store.update( group_name, run_date, @@ -146,17 +502,212 @@ def _run_regeneration( image_regen_error=str(exc)[:500], image_regen_finished_at=_now(), send_hold=True, + needs_manual_send=True, ) finally: try: if temp_path.exists(): temp_path.unlink() except OSError: - pass + logger.warning("重生图临时文件清理失败：%s", temp_path, exc_info=True) with _ACTIVE_LOCK: _ACTIVE.discard(key) +def _candidate_roots(settings: Settings, store: RunStore, group_name: str, run_date: str) -> dict[str, Path]: + generator = CodexImageGenerator(settings=settings) + return { + "task": store.group_dir(group_name, run_date).resolve(), + "generated_images": generator.generated_images_dir.resolve(), + } + + +def _resolve_candidate( + settings: Settings, + group_name: str, + run_date: str, + candidate_id: str, +) -> tuple[Path, dict[str, Any], dict[str, Any], str]: + store = RunStore(settings.output_dir) + run = store.load_run(group_name, run_date) + job_key = "image_regen_job" + job = run.get(job_key) + if not isinstance(job, dict) or not isinstance(job.get("candidates"), list) or not job.get("candidates"): + job_key = "image_job" + job = run.get(job_key) + if not isinstance(job, dict): + raise FileNotFoundError("该运行没有可恢复的生图任务") + candidates = job.get("candidates") + if not isinstance(candidates, list): + raise FileNotFoundError("该运行没有候选图片") + record = next( + ( + item + for item in candidates + if isinstance(item, dict) + and str(item.get("candidate_id") or "") == candidate_id.lower() + ), + None, + ) + if record is None: + raise FileNotFoundError("候选图片不存在") + roots = _candidate_roots(settings, store, group_name, run_date) + root = roots.get(str(record.get("root") or "")) + if root is None: + raise ValueError("候选图片根目录无效") + path = (root / str(record.get("relative_path") or "")).resolve() + try: + path.relative_to(root) + except ValueError as exc: + raise ValueError("候选图片路径越界") from exc + if not path.is_file() or _sha256(path) != str(record.get("sha256") or "").upper(): + raise FileNotFoundError("候选图片已缺失或内容发生变化") + return path, record, job, job_key + + +def list_regeneration_candidates( + settings: Settings, + group_name: str, + run_date: str, +) -> list[dict[str, Any]]: + store = RunStore(settings.output_dir) + run = store.load_run(group_name, run_date) + job = run.get("image_regen_job") + if not isinstance(job, dict) or not job.get("candidates"): + job = run.get("image_job") + if not isinstance(job, dict): + return [] + result: list[dict[str, Any]] = [] + for item in job.get("candidates") or []: + if not isinstance(item, dict): + continue + candidate_id = str(item.get("candidate_id") or "") + try: + path, record, _, _ = _resolve_candidate( + settings, + group_name, + run_date, + candidate_id, + ) + except (FileNotFoundError, ValueError): + continue + result.append( + { + **record, + "job_id": str(job.get("job_id") or ""), + "group_id": run.get("group_id"), + "wechat_group_id": run.get("wechat_group_id") or "", + "group_name": group_name, + "run_date": run_date, + "size_bytes": path.stat().st_size, + } + ) + return result + + +def candidate_preview_path( + settings: Settings, + group_name: str, + run_date: str, + candidate_id: str, +) -> Path: + path, _, _, _ = _resolve_candidate(settings, group_name, run_date, candidate_id) + return path + + +def claim_regeneration_candidate( + settings: Settings, + group_name: str, + run_date: str, + *, + job_id: str, + candidate_id: str, +) -> dict[str, Any]: + store = RunStore(settings.output_dir) + source, record, job, job_key = _resolve_candidate( + settings, + group_name, + run_date, + candidate_id, + ) + if str(job.get("job_id") or "") != job_id: + raise ValueError("job_id 与当前运行不匹配") + run = _promote_image( + store, + group_name, + run_date, + source, + recovery_status="manually_claimed_candidate", + ) + fields = { + **job, + "status": "ready_for_review", + "finished_at": _now(), + "receipt": { + "success": True, + "job_id": job_id, + "revision": job.get("revision"), + "prompt_sha256": job.get("prompt_sha256"), + "sha256": run.get("image_sha256"), + "recovery_status": "manually_claimed_candidate", + }, + "claimed_candidate": record, + "candidates": [], + } + if job_key == "image_regen_job": + _merge_job( + store, + group_name, + run_date, + job_id, + status="ready_for_review", + finished_at=fields["finished_at"], + receipt=fields["receipt"], + claimed_candidate=record, + candidates=[], + ) + else: + current = store.load_run(group_name, run_date) + current_job = current.get("image_job") + if not isinstance(current_job, dict) or str(current_job.get("job_id") or "") != job_id: + raise ValueError("job_id 与当前运行不匹配") + store.update(group_name, run_date, image_job=fields) + return store.load_run(group_name, run_date) + + +def recover_pending_regenerations(settings: Settings) -> int: + """服务重启后复用同一 job_id；结果未知由生成器转人工，不新建付费任务。""" + store = RunStore(settings.output_dir) + recovered = 0 + for run in store.list_runs(): + if run.get("image_regen_status") not in {"queued", "running"}: + continue + job = run.get("image_regen_job") + group_name = str(run.get("group_name") or "") + run_date = str(run.get("run_date") or "") + if not isinstance(job, dict) or not group_name or not run_date: + continue + key = _job_key(group_name, run_date) + with _ACTIVE_LOCK: + if key in _ACTIVE: + continue + _ACTIVE.add(key) + try: + _EXECUTOR.submit( + _run_regeneration, + settings, + group_name, + run_date, + CodexImageGenerator(settings=settings), + job, + ) + recovered += 1 + except Exception: + with _ACTIVE_LOCK: + _ACTIVE.discard(key) + return recovered + + def run_regeneration_now( settings: Settings, group_name: str, @@ -164,8 +715,17 @@ def run_regeneration_now( generator: Any, ) -> dict[str, Any]: """同步测试入口，不进入线程池。""" + store = RunStore(settings.output_dir) key = _job_key(group_name, run_date) with _ACTIVE_LOCK: + if key in _ACTIVE: + raise RuntimeError("该运行已在重新生图队列中") _ACTIVE.add(key) - _run_regeneration(settings, group_name, run_date, generator) - return RunStore(settings.output_dir).load_run(group_name, run_date) + try: + job = _prepare_job(store, group_name, run_date) + except Exception: + with _ACTIVE_LOCK: + _ACTIVE.discard(key) + raise + _run_regeneration(settings, group_name, run_date, generator, job) + return store.load_run(group_name, run_date) diff --git a/app/main.py b/app/main.py index 049ae7c..d7c8153 100644 --- a/app/main.py +++ b/app/main.py @@ -3,7 +3,9 @@ from __future__ import annotations from contextlib import asynccontextmanager +import logging import os +from datetime import datetime from pathlib import Path from fastapi import FastAPI @@ -19,27 +21,71 @@ APP_VERSION = "1.0.0" +def _should_start_scheduler(settings) -> bool: + return ( + settings.scheduler_owner == "fastapi" + and os.environ.get("GROUPBRIEF_NO_SCHEDULER", "") != "1" + ) + + +def _capture_startup_checks(settings, runner=None) -> tuple[list[dict], str]: + if runner is None: + from app.core.startup_check import run_startup_checks + + runner = run_startup_checks + try: + return runner(settings), "" + except Exception as exc: + logging.getLogger("app").exception("启动检查执行失败") + detail = str(exc)[:200] + return [ + { + "name": "启动检查执行", + "ok": False, + "status": "ERROR", + "detail": detail, + } + ], detail + + @asynccontextmanager async def lifespan(app: FastAPI): settings = get_settings() settings.ensure_dirs() setup_logging(settings.logs_dir) + if settings.legacy_v1_write_mode == "maintenance": + logging.getLogger("groupbrief.legacy_v1").warning( + "旧 V1 写入 maintenance 模式已启用；正式生成与发送仍应使用 V2" + ) + else: + logging.getLogger("groupbrief.legacy_v1").info( + "旧 V1 写入已冻结为只读模式" + ) repository.init_db(settings) app.state.settings = settings # P9：启动检查（记录日志，不阻止启动） - try: - from app.core.startup_check import run_startup_checks - - app.state.startup_checks = run_startup_checks(settings) - except Exception: - app.state.startup_checks = [] + app.state.startup_checks, app.state.startup_check_error = ( + _capture_startup_checks(settings) + ) + app.state.startup_checks_at = datetime.now().astimezone().isoformat() + from app.image.regeneration import recover_pending_regenerations + + recovered_image_jobs = recover_pending_regenerations(settings) + if recovered_image_jobs: + logging.getLogger("groupbrief.image").warning( + "已按原 job_id 恢复 %d 个中断的生图任务；不会新建结果未知任务", + recovered_image_jobs, + ) # P9：日志轮转清理已在 setup_logging 中执行 - if os.environ.get("GROUPBRIEF_NO_SCHEDULER", "") != "1": + scheduler_started = _should_start_scheduler(settings) + app.state.scheduler_owner = settings.scheduler_owner + app.state.scheduler_active = scheduler_started + if scheduler_started: from app.scheduler.manager import start_scheduler start_scheduler(settings) yield - if os.environ.get("GROUPBRIEF_NO_SCHEDULER", "") != "1": + if scheduler_started: from app.scheduler.manager import stop_scheduler stop_scheduler() diff --git a/app/pipeline/daily_pipeline.py b/app/pipeline/daily_pipeline.py index 4309c77..437a29b 100644 --- a/app/pipeline/daily_pipeline.py +++ b/app/pipeline/daily_pipeline.py @@ -3,13 +3,13 @@ 生成阶段（默认 00:15，run_date 决定周期）： PENDING → 首次取数/复用当天快照(messages.json) → DATA_READY → 排行(ranking.json/txt) → RANKING_READY → Codex GPT / DeepSeek 备用(image_prompt.txt) → PROMPT_READY - → Codex 串行生图(daily_image.png) → IMAGE_READY → READY_TO_SEND + → Codex 受控并发生图(daily_image.png) → IMAGE_READY → READY_TO_SEND 发送阶段（每群 send_time）： READY_TO_SEND/IMAGE_READY → 发排行榜文字 → 发图片 → SENT 约束： - 每个群独立状态；某群失败不阻塞其他群； -- 生图阶段使用全局单队列严格串行； +- 生图阶段默认最多 2 路并发，每个结果按独立 job_id 归属； - 同一群同一统计周期已到终态则跳过（force 可重跑）； - 同一日报日期的消息默认只读取一次；显式 refresh_messages 只覆盖当天快照，不连带重建 Prompt 或生图； - SENT 绝不重复发送（force_send 允许重发内容）。 @@ -17,52 +17,58 @@ from __future__ import annotations +import hashlib import json from concurrent.futures import ThreadPoolExecutor, as_completed from datetime import date, datetime, time, timedelta from pathlib import Path -from time import perf_counter from zoneinfo import ZoneInfo -from app.ai.concurrency import bounded_slot, normalized_limit +from app.ai.concurrency import normalized_limit from app.ai.prompt_builder import GroupSummaryImagePromptBuilder from app.ai.prompt_builder_types import PromptInput +from app.ai.speaker_attribution import build_attribution_contract from app.config.settings import Settings, get_settings from app.core.logging import get_logger from app.data_sources.base import V2Message, WeChatDataSource from app.data_sources.wechat_data_analysis import WeChatDataAnalysisSource +from app.data_sources.history_provider import HistoryProviderDataSource +from app.data_sources.resilient import ResilientWeChatDataSource from app.db import repository as repo from app.db.models import Group +from app.db.resilience import run_with_sqlite_retry from app.image.codex_generator import CodexImageGenerator -from app.image.image_task import ImageJob, SerialImageQueue, verify_image +from app.image.image_task import ImageJob +from app.pipeline.delivery_stages import DeliveryStages +from app.pipeline.generation_stages import GenerationStages +from app.pipeline.image_stages import ImageStages from app.ranking.engine import RankingEngine from app.ranking.renderer import RankingRenderer -from app.scheduler.period import PeriodResolver +from app.scheduler.period import PeriodResolver, PeriodWindow +from app.scheduler.runtime_status import write_daily_status from app.sender.base import WechatSender from app.sender.wechat_native import create_wechat_sender from app.services.generation_runtime import generation_mutex +from app.services.group_provider_config import ( + normalize_history_provider, + resolve_group_ai_settings, +) from app.services.group_name_sync import ( GroupNameSyncReport, GroupNameSyncService, - effective_send_target, send_target_mode, ) +from app.providers.history.wechat_cli import WechatCliProvider from app.v2.constants import ( - DATA_READY, + CORRUPT, FAILED, IMAGE_GENERATION_FAILED, IMAGE_READY, - IMAGE_FILE_MISSING, - MESSAGE_FETCH_FAILED, - MESSAGE_SNAPSHOT_INVALID, - PENDING, PROMPT_FAILED, PROMPT_READY, - RANKING_FAILED, - RANKING_READY, READY_TO_SEND, + RUN_STATE_CORRUPT, SENT, - WECHAT_DATA_UNAVAILABLE, ) from app.v2.run_store import RunStore, validate_run_date @@ -83,11 +89,21 @@ def __init__( dry_run: bool = False, ): self.settings = settings or get_settings() - self.data_source = data_source or WeChatDataAnalysisSource(self.settings) + self._data_source_injected = data_source is not None + self.data_source = data_source or ResilientWeChatDataSource( + WeChatDataAnalysisSource(self.settings), + self.settings, + ) self.period_resolver = PeriodResolver() self.ranking_engine = ranking_engine or RankingEngine() self.renderer = renderer or RankingRenderer() - self.prompt_builder = prompt_builder or GroupSummaryImagePromptBuilder(self.settings) + self._prompt_builder_injected = prompt_builder is not None + self.prompt_builder = prompt_builder or GroupSummaryImagePromptBuilder( + self.settings, + summary_settings=self.settings, + ) + self._data_source_cache: dict[str, WeChatDataSource] = {} + self._prompt_builder_cache: dict[tuple[str, ...], GroupSummaryImagePromptBuilder] = {} self.image_generator = image_generator or CodexImageGenerator(self.settings) self.sender = sender or create_wechat_sender(settings=self.settings, dry_run=dry_run) self.store = store or RunStore(self.settings.output_dir) @@ -102,6 +118,7 @@ def generate_all( group_ids: list[int] | None = None, force: bool = False, refresh_messages: bool = False, + group_overrides: dict[int, dict] | None = None, *, acquire_lock: bool = True, ) -> list[dict]: @@ -112,6 +129,7 @@ def generate_all( group_ids=group_ids, force=force, refresh_messages=refresh_messages, + group_overrides=group_overrides, acquire_lock=False, ) requested_date = parse_date(run_date) @@ -121,23 +139,67 @@ def generate_all( "error_type": "INVALID_RUN_DATE", "detail": "run_date 必须是有效的 YYYY-MM-DD 日期", }] - window = self.period_resolver.resolve(run_date=requested_date, timezone=self.settings.app_timezone) - run_date_str = window.run_date.isoformat() - self._last_name_sync_report = self._sync_group_names(group_ids) + base_window = self.period_resolver.resolve( + run_date=requested_date, + timezone=self.settings.app_timezone, + schedule_rule="daily_previous_day", + ) + run_date_str = base_window.run_date.isoformat() + self._last_name_sync_report = self._sync_group_names_safe(group_ids) groups = self._load_groups(group_ids) + if group_overrides: + allowed_override_fields = { + "wechat_group_id", "wechat_group_name", "provider_preference", + "schedule_rule", "summary_provider", "summary_model", + "prompt_provider", "prompt_model", "image_enabled", "ranking_template", + "ranking_count_policy", "sender_name_policy", + "image_prompt_template", "image_theme", "image_theme_custom", + "image_prompt_override", "send_target", + } + groups = [ + Group.model_validate( + { + **group.model_dump(), + **{ + key: value + for key, value in group_overrides.get( + int(group.id or 0), {} + ).items() + if key in allowed_override_fields + }, + } + ) + for group in groups + ] if not groups: return [{"status": "no_groups", "reason": "无启用群"}] + scheduled: list[tuple[Group, PeriodWindow]] = [] + for group in groups: + window = self.period_resolver.resolve( + run_date=base_window.run_date, + timezone=self.settings.app_timezone, + schedule_rule=str(group.schedule_rule or "daily_previous_day"), + ) + if window.should_run: + scheduled.append((group, window)) + if not scheduled: + return [{"status": "no_groups", "reason": "当日没有符合群级统计规则的任务"}] + groups = [item[0] for item in scheduled] + windows = [item[1] for item in scheduled] group_limit = normalized_limit(self.settings.generation_group_concurrency, 5) + image_limit = normalized_limit(self.settings.image_generation_concurrency, 2) logger.info( - "开始并行生成：groups=%d group_limit=%d fetch_limit=%d ai_limit=%d", + "开始并行生成：groups=%d group_limit=%d fetch_limit=%d ai_limit=%d image_limit=%d", len(groups), group_limit, normalized_limit(self.settings.wechat_fetch_concurrency, 1), normalized_limit(self.settings.ai_request_concurrency, 6), + image_limit, ) if len(groups) == 1: group = groups[0] + window = windows[0] try: if refresh_messages: result = self._generate_one_safe( @@ -153,14 +215,22 @@ def generate_all( except Exception as exc: logger.exception("群 %s worker 未捕获异常，已隔离", self._group_name(group)) result = self._record_group_failure(group, run_date_str, exc, "unexpected") - return [self._run_image_when_ready_safe(group, result, run_date_str, force)] + final_results = [ + self._run_image_when_ready_safe(group, result, run_date_str, force) + ] + self._schedule_ready_send_once_safe(run_date_str) + return final_results else: with ThreadPoolExecutor( max_workers=min(group_limit, len(groups)), thread_name_prefix="groupbrief-v2-group", - ) as executor: + ) as executor, ThreadPoolExecutor( + max_workers=min(image_limit, len(groups)), + thread_name_prefix="groupbrief-v2-image", + ) as image_executor: future_indexes = {} for index, group in enumerate(groups): + window = windows[index] if refresh_messages: future = executor.submit( self._generate_one_safe, @@ -180,7 +250,7 @@ def generate_all( force, ) future_indexes[future] = index - results_by_index: dict[int, dict] = {} + image_future_indexes = {} for future in as_completed(future_indexes): result_index = future_indexes[future] group = groups[result_index] @@ -191,17 +261,95 @@ def generate_all( result = self._record_group_failure( group, run_date_str, exc, "unexpected" ) - results_by_index[result_index] = self._run_image_when_ready_safe( - group, result, run_date_str, force + image_future = image_executor.submit( + self._run_image_when_ready_safe, + group, + result, + run_date_str, + force, ) + image_future_indexes[image_future] = result_index + + results_by_index: dict[int, dict] = {} + for image_future in as_completed(image_future_indexes): + result_index = image_future_indexes[image_future] + group = groups[result_index] + try: + results_by_index[result_index] = image_future.result() + except Exception as exc: + logger.exception( + "群 %s 图片 worker 未捕获异常，已隔离", + self._group_name(group), + ) + results_by_index[result_index] = self._record_group_failure( + group, run_date_str, exc, "image" + ) # 生图按 Prompt 完成顺序启动，API 结果仍按群配置顺序返回。 - return [results_by_index[index] for index in range(len(groups))] + final_results = [results_by_index[index] for index in range(len(groups))] + self._schedule_ready_send_once_safe(run_date_str) + return final_results + + def _schedule_ready_send_once_safe(self, run_date: str) -> None: + """生成结束后若已进入发送窗口，去重注册一次性发送任务。""" + try: + from app.scheduler.manager import ( + _schedule_on_demand_jobs, + get_scheduler, + ) + + scheduler = get_scheduler() + if scheduler is not None: + _schedule_on_demand_jobs( + scheduler, + self.settings, + run_dates=[run_date], + ) + except Exception: + # 调度投影失败不能改变已完成的生成结果；启动恢复会从 run.json 重建。 + logger.exception("READY_TO_SEND 一次性任务注册失败：run_date=%s", run_date) @staticmethod def _group_name(group: Group) -> str: return group.display_name or group.wechat_group_name + def _prompt_operation_hash( + self, + data: PromptInput, + prompt_settings: Settings | None = None, + summary_settings: Settings | None = None, + ) -> str: + """生成稳定输入指纹；不把 API Key 或原始输入写入运行状态。""" + payload = dict(vars(data)) + payload["messages"] = [ + message.to_dict() if hasattr(message, "to_dict") else str(message) + for message in data.messages + ] + selected_prompt = prompt_settings or self.settings + selected_summary = summary_settings or self.settings + payload["provider_config"] = { + "summary": { + "primary": selected_summary.summary_provider_primary, + "fallback": selected_summary.summary_provider_fallback, + "codex_model": selected_summary.codex_summary_model, + "deepseek_model": selected_summary.ai_model, + }, + "prompt": { + "primary": selected_prompt.summary_provider_primary, + "fallback": selected_prompt.summary_provider_fallback, + "codex_model": selected_prompt.codex_summary_model, + "deepseek_model": selected_prompt.ai_model, + }, + } + canonical = json.dumps( + payload, + ensure_ascii=False, + sort_keys=True, + separators=(",", ":"), + default=str, + ) + return hashlib.sha256(canonical.encode("utf-8")).hexdigest() + def _record_group_failure( self, group: Group, @@ -255,11 +403,11 @@ def _run_image_when_ready( run_date: str, force: bool, ) -> dict: - """单群 Prompt 就绪后立即生图；调用方保证图片任务严格串行。""" + """单群 Prompt 就绪后立即提交受控并发生图。""" if not result.get("need_image"): return result group_name = group.display_name or group.wechat_group_name - logger.info("群 %s Prompt 已就绪，立即进入串行生图", group_name) + logger.info("群 %s Prompt 已就绪，立即进入受控并发生图", group_name) job = self._make_image_job(group, run_date, force) return self._run_image_jobs([job], run_date)[0] @@ -284,6 +432,7 @@ def _generate_one_safe( status=FAILED, failed_stage="unexpected", error=str(exc)[:300], + error_type="UNEXPECTED_GENERATION_ERROR", ) return { "group_name": group_name, @@ -300,428 +449,365 @@ def _generate_one( force: bool, *, refresh_messages: bool = False, + reuse_persisted_topic_selection: bool = False, ) -> dict: - group_name = group.display_name or group.wechat_group_name - store = self.store - started_at = perf_counter() - timings: dict[str, int] = {} - - def finish(result: dict) -> dict: - timings["total_ms"] = round((perf_counter() - started_at) * 1000) - current = store.load_run(group_name, run_date) - meta = current.get("prompt_meta") if isinstance(current.get("prompt_meta"), dict) else {} - timings["summary_calls"] = int(meta.get("api_call_count") or 0) - timings["deepseek_calls"] = timings["summary_calls"] # 旧运行分析字段兼容 - timings["chunk_count"] = int(meta.get("chunk_count") or 0) - store.update(group_name, run_date, stage_timings=timings) - logger.info( - "群生成耗时 group=%s fetch_ms=%d ranking_ms=%d summary_ms=%d " - "summary_calls=%d chunks=%d total_ms=%d status=%s", - group_name, - timings.get("fetch_ms", 0), - timings.get("ranking_ms", 0), - timings.get("summary_ms", timings.get("deepseek_ms", 0)), - timings["summary_calls"], - timings["chunk_count"], - timings["total_ms"], - result.get("status", ""), - ) - return result - - # 防重复：同一群同一周期已到终态 - run = store.load_run(group_name, run_date) - if not force and not refresh_messages and run.get("status") in (IMAGE_READY, READY_TO_SEND, SENT): - logger.info("群 %s %s 已到 %s，跳过生成", group_name, run_date, run.get("status")) - return finish({"group_name": group_name, "status": "skipped", "detail": f"已{run.get('status')}"}) - - period_start = window.period_start_str() - period_end = window.period_end_str() - base = { - "group_id": str(group.id), - "wechat_group_id": group.wechat_group_id, - "wechat_group_name": group.wechat_group_name, - "effective_send_target": effective_send_target(group), - "send_target_mode": send_target_mode(group), - **self._name_sync_audit(group), - "period_start": period_start, - "period_end": period_end, - "send_time": group.send_time, - "image_enabled": bool(group.image_enabled), - "ranking_template": group.ranking_template, - "image_prompt_template": group.image_prompt_template, - "image_theme": group.image_theme, - "image_theme_custom": group.image_theme_custom, - "wechat_send_enabled": bool(getattr(group, "wechat_send_enabled", False)), - "provider": self.data_source.name, - "failed_stage": None, - "error": None, - } - store.update(group_name, run_date, status=PENDING, **base) - - # ---- 1) 数据快照（同一日报日期默认只读取一次）---- - if not group.wechat_group_id: - store.update(group_name, run_date, status=FAILED, failed_stage="data", error="群未绑定微信群 ID") - return finish({"group_name": group_name, "status": "failed", "error_type": WECHAT_DATA_UNAVAILABLE}) - fetch_started = perf_counter() - snapshot_path = store.messages_path(group_name, run_date) - messages: list[V2Message] - if snapshot_path.is_file() and not refresh_messages: - try: - messages = self._load_message_snapshot(snapshot_path) - except (OSError, UnicodeError, ValueError) as exc: - timings["fetch_ms"] = round((perf_counter() - fetch_started) * 1000) - detail = f"当天消息快照无法读取，已停止且不会隐式重抓：{str(exc)[:220]}" - store.update( - group_name, - run_date, - status=FAILED, - failed_stage="data", - error=detail, - error_type=MESSAGE_SNAPSHOT_INVALID, - message_snapshot_reused=False, - ) - return finish({ - "group_name": group_name, - "status": "failed", - "error_type": MESSAGE_SNAPSHOT_INVALID, - "detail": detail, - }) - timings["fetch_ms"] = round((perf_counter() - fetch_started) * 1000) - store.update( - group_name, - run_date, - status=DATA_READY, - message_count=len(messages), - message_snapshot_reused=True, - message_snapshot_refreshed=False, - message_snapshot_path=snapshot_path.name, - ) - logger.info("群 %s 复用当天消息快照：%s（%d 条）", group_name, snapshot_path, len(messages)) - else: - try: - with bounded_slot("wechat_fetch", self.settings.wechat_fetch_concurrency): - fetch = self.data_source.fetch_messages( - group.wechat_group_id, window.period_start, window.period_end - ) - except Exception as exc: - timings["fetch_ms"] = round((perf_counter() - fetch_started) * 1000) - store.update( - group_name, - run_date, - status=FAILED, - failed_stage="data", - error=str(exc)[:300], - error_type=MESSAGE_FETCH_FAILED, - ) - return finish({ - "group_name": group_name, - "status": "failed", - "error_type": MESSAGE_FETCH_FAILED, - "detail": str(exc)[:300], - }) - timings["fetch_ms"] = round((perf_counter() - fetch_started) * 1000) - fetch_metrics = fetch.meta if isinstance(getattr(fetch, "meta", None), dict) else {} - if fetch_metrics: - store.update(group_name, run_date, fetch_metrics=fetch_metrics) - if fetch.status.value != "OK" or not fetch.messages: - error_type = fetch.error_type or MESSAGE_FETCH_FAILED - store.update(group_name, run_date, status=FAILED, failed_stage="data", - error=fetch.detail or fetch.status.value, error_type=error_type) - return finish({"group_name": group_name, "status": "failed", "error_type": error_type, - "detail": fetch.detail}) - messages = list(fetch.messages) - if not refresh_messages: - self._save_json(snapshot_path, [m.to_dict() for m in messages]) - store.update( - group_name, - run_date, - status=DATA_READY, - message_count=len(messages), - message_snapshot_reused=False, - message_snapshot_refreshed=False, - message_snapshot_saved_at=datetime.now().astimezone().isoformat(), - message_snapshot_path=snapshot_path.name, - message_snapshot_period_start=period_start, - message_snapshot_period_end=period_end, - ) + """保留原注入点；单群生成由显式阶段执行器负责。""" + data_source = self._data_source_for_group(group) + prompt_settings, prompt_config = resolve_group_ai_settings( + self.settings, + group, + capability="prompt", + ) + summary_settings, summary_config = resolve_group_ai_settings( + self.settings, + group, + capability="summary", + ) + self.store.update( + self._group_name(group), + run_date, + config_version=1, + history_provider_requested=( + normalize_history_provider(group.provider_preference) or self.data_source.name + ), + prompt_provider_requested=prompt_config["provider"], + prompt_model_requested=prompt_config["model"], + prompt_config_inherited=prompt_config["inherited"], + summary_provider_requested=summary_config["provider"], + summary_model_requested=summary_config["model"], + summary_config_inherited=summary_config["inherited"], + ) + prompt_builder = self._prompt_builder_for_group( + summary_settings, + prompt_settings, + ) + return GenerationStages( + settings=prompt_settings, + data_source=data_source, + ranking_engine=self.ranking_engine, + renderer=self.renderer, + prompt_builder=prompt_builder, + store=self.store, + group_name=self._group_name, + name_sync_audit=self._name_sync_audit, + get_group=self._get_group, + prompt_operation_hash=lambda data: self._prompt_operation_hash( + data, + prompt_settings, + summary_settings, + ), + save_json=self._save_json, + load_message_snapshot=self._load_message_snapshot, + logger=logger, + ).run( + group, + window, + run_date, + force, + refresh_messages=refresh_messages, + reuse_persisted_topic_selection=reuse_persisted_topic_selection, + ) - if refresh_messages: - # 手动重取消息只更新消息与确定性排行榜。先在内存中完成计算和渲染， - # 成功后再落盘，避免留下“新消息 + 旧排行榜”的明显不一致状态。 - ranking_started = perf_counter() - try: - ranking = self.ranking_engine.compute( - messages, - group_name, - period_start, - period_end, - top_limit=10, - ) - ranking_txt = self.renderer.render( - ranking, template_name=group.ranking_template - ) - except Exception as exc: - timings["ranking_ms"] = round((perf_counter() - ranking_started) * 1000) - store.update( - group_name, - run_date, - status=run.get("status") or PENDING, - failed_stage=run.get("failed_stage"), - error=run.get("error"), - message_refresh_status="failed", - message_refresh_error=str(exc)[:300], - ) - return finish({ - "group_name": group_name, - "status": "failed", - "error_type": RANKING_FAILED, - "detail": "新消息已读取，但排行榜计算失败，旧快照未被替换", - }) - timings["ranking_ms"] = round((perf_counter() - ranking_started) * 1000) - - self._save_json(snapshot_path, [m.to_dict() for m in messages]) - self._save_json(store.ranking_json_path(group_name, run_date), ranking.to_dict()) - store.ranking_txt_path(group_name, run_date).write_text(ranking_txt, encoding="utf-8") - next_status = SENT if run.get("status") == SENT else RANKING_READY - store.update( - group_name, - run_date, - status=next_status, - failed_stage=None, - error=None, - speaker_count=ranking.speaker_count, - message_count=ranking.message_count, - message_snapshot_reused=False, - message_snapshot_refreshed=True, - message_snapshot_saved_at=datetime.now().astimezone().isoformat(), - message_snapshot_path=snapshot_path.name, - message_snapshot_period_start=period_start, - message_snapshot_period_end=period_end, - message_refresh_status="completed", - message_refresh_error="", - prompt_rebuild_status="required", - prompt_rebuild_error="", - send_hold=True, - send_hold_reason="MESSAGE_SNAPSHOT_REFRESHED", - needs_manual_send=True, + def _data_source_for_group(self, group: Group) -> WeChatDataSource: + if self._data_source_injected: + return self.data_source + selected = normalize_history_provider(group.provider_preference) + if selected in {"", "wechat_data_analysis"}: + return self.data_source + if selected not in self._data_source_cache: + source = HistoryProviderDataSource(WechatCliProvider(settings=self.settings)) + self._data_source_cache[selected] = ResilientWeChatDataSource( + source, + self.settings, ) - return finish({ - "group_name": group_name, - "status": "data_ready", - "detail": "当天消息快照和排行榜已更新；未重建 Prompt，未生图", - }) + return self._data_source_cache[selected] - # ---- 2) 排行榜 ---- - ranking_started = perf_counter() - try: - ranking = self.ranking_engine.compute( - messages, - group_name, - period_start, - period_end, - top_limit=10, - ) - except Exception as e: - store.update(group_name, run_date, status=FAILED, failed_stage="ranking", error=str(e)[:300]) - timings["ranking_ms"] = round((perf_counter() - ranking_started) * 1000) - return finish({"group_name": group_name, "status": "failed", "error_type": RANKING_FAILED}) - timings["ranking_ms"] = round((perf_counter() - ranking_started) * 1000) - self._save_json(store.ranking_json_path(group_name, run_date), ranking.to_dict()) - ranking_txt = self.renderer.render(ranking, template_name=group.ranking_template) - store.ranking_txt_path(group_name, run_date).write_text(ranking_txt, encoding="utf-8") - store.update(group_name, run_date, status=RANKING_READY, - speaker_count=ranking.speaker_count, message_count=ranking.message_count) - - # ---- 3) 生图 Prompt（Codex GPT 主用，DeepSeek 备用）---- - prompt_msgs = [m for m in messages if RankingEngine._countable(m)] - prompt_input = PromptInput( - group_name=group_name, - group_id=str(group.id or group.wechat_group_id or group_name), - run_date=run_date, - period_start=period_start, - period_end=period_end, - report_date=window.period_end.date().isoformat(), - message_count=ranking.message_count, - speaker_count=ranking.speaker_count, - messages=prompt_msgs, - template=group.image_prompt_template, - image_theme=group.image_theme, - image_theme_custom=group.image_theme_custom, - template_override=getattr(group, "image_prompt_override", "") or "", - previous_theme_signature=store.previous_theme_signature(group_name, run_date), - persisted_theme_meta=run.get("prompt_meta") if isinstance(run.get("prompt_meta"), dict) else None, - recent_layout_history=store.recent_layout_history(group_name, run_date, limit=3), + def _prompt_builder_for_group( + self, + summary_settings: Settings, + prompt_settings: Settings, + ) -> GroupSummaryImagePromptBuilder: + if self._prompt_builder_injected: + return self.prompt_builder + key = ( + summary_settings.summary_provider_primary, + summary_settings.summary_provider_fallback, + summary_settings.codex_summary_model, + summary_settings.ai_model, + prompt_settings.summary_provider_primary, + prompt_settings.summary_provider_fallback, + prompt_settings.codex_summary_model, + prompt_settings.ai_model, ) - prompt_started = perf_counter() - prompt_out = self.prompt_builder.build(prompt_input) - timings["summary_ms"] = round((perf_counter() - prompt_started) * 1000) - timings["deepseek_ms"] = timings["summary_ms"] # 旧运行分析字段兼容 - if not prompt_out.success: - store.update(group_name, run_date, status=FAILED, failed_stage="prompt", - error=prompt_out.error, error_type=PROMPT_FAILED) - return finish({"group_name": group_name, "status": "failed", "error_type": PROMPT_FAILED, - "detail": prompt_out.error}) - store.prompt_path(group_name, run_date).write_text(prompt_out.prompt, encoding="utf-8") - store.update(group_name, run_date, status=PROMPT_READY, prompt_meta=prompt_out.meta) - - # Prompt 落盘后重新读取群配置，避免流水线开始后用户打开/关闭生图 - # 时仍使用旧的 group 对象和 image_enabled 快照。 - current_group = group - if group.id is not None: - try: - refreshed_group = self._get_group(group.id) - except Exception as exc: - logger.warning("群 %s 生图开关刷新失败，沿用本次读取的配置：%s", group_name, exc) - else: - if refreshed_group is not None: - current_group = refreshed_group - - current_image_enabled = bool(current_group.image_enabled) - group.image_enabled = current_image_enabled - store.update(group_name, run_date, image_enabled=current_image_enabled) - - # ---- 4) 生图判断 ---- - if not current_image_enabled: - store.update(group_name, run_date, status=READY_TO_SEND) - return finish({"group_name": group_name, "status": "ready_to_send", "detail": "未启用生图"}) - return finish({"group_name": group_name, "status": "prompt_ready", "need_image": True}) + if key not in self._prompt_builder_cache: + self._prompt_builder_cache[key] = GroupSummaryImagePromptBuilder( + prompt_settings, + summary_settings=summary_settings, + ) + return self._prompt_builder_cache[key] def _make_image_job(self, group: Group, run_date: str, force: bool) -> ImageJob: - group_name = group.display_name or group.wechat_group_name - return ImageJob( - group_name=group_name, - prompt_file=self.store.prompt_path(group_name, run_date), - output_path=self.store.image_path(group_name, run_date), - generator=self.image_generator, - force=force, - ) + """保留原注入点；图片任务构造由图片阶段负责。""" + return ImageStages( + store=self.store, + image_generator=self.image_generator, + ).make_job(self._group_name(group), run_date, force) def _image_hook(self, job: ImageJob, result: dict) -> None: - # 每群生图完成后更新 run.json（不在此处判断 need_image） - status = IMAGE_READY if result["success"] else FAILED - error_type = result.get("error_type") or IMAGE_GENERATION_FAILED - current = self.store.load_run(job.group_name, job.output_path.parent.name) - stage_timings = dict(current.get("stage_timings") or {}) - imagegen_ms = int(result.get("imagegen_ms") or 0) - stage_timings["imagegen_ms"] = imagegen_ms - image_size_bytes = job.output_path.stat().st_size if result["success"] and job.output_path.is_file() else 0 - generator_detail = result.get("generator_detail") - if not isinstance(generator_detail, dict): - generator_detail = {} - self.store.update( - job.group_name, job.output_path.parent.name, - status=status, - image_error=result.get("detail") if not result["success"] else None, - image_status=result["status"], - error_type=error_type if not result["success"] else None, - stage_timings=stage_timings, - imagegen_ms=imagegen_ms, - image_generated_at=( - datetime.now().astimezone().isoformat() - if result["success"] - else current.get("image_generated_at") - ), - image_size_bytes=image_size_bytes, - image_attempt_count=int(generator_detail.get("attempt_count") or 0), - image_recovery_status=str(generator_detail.get("recovery_status") or ""), - image_candidate_diagnostics=generator_detail.get("candidate_diagnostics") or [], - image_attempts=generator_detail.get("attempts") or [], - ) + """保留原 hook；图片结果字段和状态推进保持不变。""" + ImageStages( + store=self.store, + image_generator=self.image_generator, + ).record_result(job, result) def _after_image(self, job: ImageJob, run_date: str) -> None: - run = self.store.load_run(job.group_name, run_date) - if run.get("status") == IMAGE_READY: - self.store.update(job.group_name, run_date, status=READY_TO_SEND) + ImageStages( + store=self.store, + image_generator=self.image_generator, + ).advance_ready(job, run_date) def _run_image_jobs(self, image_jobs: list[ImageJob], run_date: str) -> list[dict]: """串行执行图片任务，并以每个 run.json 的最终状态返回结果。""" - queue = SerialImageQueue(run_hook=self._image_hook) - queue_results = queue.run_all(image_jobs) - final_results: list[dict] = [] - for job, queue_result in zip(image_jobs, queue_results): - self._after_image(job, run_date) - run = self.store.load_run(job.group_name, run_date) - final_status = run.get("status") - if final_status == READY_TO_SEND: - final_results.append( - { - "group_name": job.group_name, - "status": "ready_to_send", - "detail": "图片已准备，可以发送", - } - ) - continue - if final_status == FAILED: - final_results.append( - { - "group_name": job.group_name, - "status": "failed", - "error_type": run.get("error_type") or queue_result.get("error_type") or IMAGE_GENERATION_FAILED, - "detail": run.get("image_error") or run.get("error") or queue_result.get("detail") or "生图失败", - } - ) - continue - final_results.append( - { - "group_name": job.group_name, - "status": str(final_status or queue_result.get("status") or "failed").lower(), - "detail": queue_result.get("detail") or "图片任务未进入终态", - } - ) - return final_results + return ImageStages( + store=self.store, + image_generator=self.image_generator, + ).run_jobs( + image_jobs, + run_date, + run_hook=self._image_hook, + after_hook=self._after_image, + ) # ================= 发送阶段 ================= def send_due(self, now: datetime | None = None) -> list[dict]: now = now or datetime.now(ZoneInfo(self.settings.app_timezone)) - run_date = now.date().isoformat() + return self.send_due_for_dates([now.date().isoformat()], now=now, recovery=False) + + def send_due_for_dates( + self, + run_dates: list[str], + *, + now: datetime | None = None, + recovery: bool = False, + ) -> list[dict]: + """扫描指定日报日期。 + + 默认 ``send_due`` 只处理当天。恢复入口可以显式传入历史日期；历史任务 + 仍必须通过现有 claim、未知结果锁、目标预检和图片预检。 + """ + now = now or datetime.now(ZoneInfo(self.settings.app_timezone)) + normalized_dates = sorted({validate_run_date(value) for value in run_dates}) results: list[dict] = [] groups = self._load_groups() - due_group_ids = self._due_sync_group_ids(groups, run_date, now) + due_group_ids: list[int] = [] + for run_date in normalized_dates: + due_group_ids.extend( + self._due_sync_group_ids( + groups, + run_date, + now, + recovery=recovery, + ) + ) + due_group_ids = sorted(set(due_group_ids)) if due_group_ids: - self._last_name_sync_report = self._sync_group_names(due_group_ids) + self._last_name_sync_report = self._sync_group_names_safe(due_group_ids) groups = self._load_groups() - for group in groups: - if not bool(getattr(group, "wechat_send_enabled", False)): - continue - group_name = group.display_name or group.wechat_group_name + for run_date in normalized_dates: + report_date = date.fromisoformat(run_date) + for group in groups: + if not bool(getattr(group, "wechat_send_enabled", False)): + continue + group_name = group.display_name or group.wechat_group_name + run = self.store.load_run(group_name, run_date) + status = run.get("status") + if status not in (IMAGE_READY, READY_TO_SEND): + continue + if run.get("sent_at"): + continue # 已发送，绝不重复 + if run.get("send_hold"): + continue # unknown / 手工审核必须保持 fail-closed + send_time = parse_send_time(self.settings.schedule_send_time) + due_at = datetime.combine(report_date, send_time, tzinfo=now.tzinfo) + if now < due_at: + continue + late_window = timedelta( + minutes=max(int(self.settings.wechat_late_send_window_minutes), 0) + ) + if not recovery and now > due_at + late_window: + self.store.update( + group_name, + run_date, + send_state="held", + send_hold=True, + send_hold_reason="MISSED_SEND_WINDOW", + needs_manual_send=True, + send_error="已超过到点后 30 分钟自动补发窗口，需人工确认", + send_error_type="MISSED_SEND_WINDOW", + missed_send_window_at=now.isoformat(), + ) + results.append( + { + "group_name": group_name, + "status": "held", + "error_type": "MISSED_SEND_WINDOW", + "detail": "已超过自动补发窗口，需人工确认", + } + ) + continue + if recovery and report_date < now.date(): + self.store.update( + group_name, + run_date, + send_state="held", + send_hold=True, + send_hold_reason="HISTORICAL_SEND_REQUIRES_CONFIRMATION", + needs_manual_send=True, + send_error="历史任务禁止自动发送，需重新核对目标并人工确认", + send_error_type="HISTORICAL_SEND_REQUIRES_CONFIRMATION", + ) + results.append( + { + "group_name": group_name, + "status": "held", + "error_type": "HISTORICAL_SEND_REQUIRES_CONFIRMATION", + "detail": "历史任务禁止自动发送", + } + ) + continue + if recovery: + self.store.update( + group_name, + run_date, + send_recovery=True, + send_recovery_checked_at=now.isoformat(), + ) + run = self.store.load_run(group_name, run_date) + try: + result = self._send_one(group, group_name, run, run_date, now) + except Exception as exc: + logger.exception( + "群 %s 发送异常，检查是否已进入外部提交窗口", + group_name, + ) + result, abort_batch = self._handle_send_exception( + group_name, + run_date, + now, + exc, + ) + results.append(result) + if abort_batch: + logger.error( + "微信桌面状态可能未知，停止本批次后续群发送 date=%s", + run_date, + ) + self._write_runtime_status_safe(normalized_dates) + return results + continue + results.append(result) + if ( + result.get("error_type") == "SEND_RESULT_UNKNOWN" + or result.get("status") == "unknown" + ): + logger.error( + "微信发送结果未知，停止本批次后续群发送 date=%s group=%s", + run_date, + group_name, + ) + self._write_runtime_status_safe(normalized_dates) + return results + self._write_runtime_status_safe(normalized_dates) + return results + + def _write_runtime_status_safe(self, run_dates: list[str]) -> None: + for run_date in run_dates: + try: + write_daily_status(self.store, run_date) + except Exception: + logger.exception("每日运行报告写入失败：run_date=%s", run_date) + + def _handle_send_exception( + self, + group_name: str, + run_date: str, + now: datetime, + exc: Exception, + ) -> tuple[dict, bool]: + """隔离明确的提交前异常；未决外部提交则锁单并中止本批次。""" + detail = f"发送阶段异常：{type(exc).__name__}: {str(exc)[:220]}" + try: run = self.store.load_run(group_name, run_date) - status = run.get("status") - if status not in (IMAGE_READY, READY_TO_SEND): - continue - if run.get("sent_at"): - continue # 已发送，绝不重复 - if run.get("send_hold"): - continue # 手工 Prompt/重生图必须先审核 - send_time = parse_send_time(group.send_time or run.get("send_time", "08:30")) - if now.time() < send_time: - continue # 未到发送时间 - due_at = datetime.combine(now.date(), send_time, tzinfo=now.tzinfo) - late_window = timedelta(minutes=max(int(self.settings.wechat_late_send_window_minutes), 0)) - if now > due_at + late_window: + except Exception as state_exc: + return ( + { + "group_name": group_name, + "status": "held", + "error_type": "SEND_STATE_UNREADABLE", + "detail": f"{detail}；且无法读取发送状态：{type(state_exc).__name__}", + }, + True, + ) + + unresolved_stage = "" + for stage in ("image", "text"): + if ( + run.get(f"{stage}_attempt_started_at") + and not run.get(f"{stage}_attempt_finished_at") + and not (run.get(f"{stage}_verified_at") or run.get(f"{stage}_sent_at")) + ): + unresolved_stage = stage + break + if not unresolved_stage: + try: self.store.update( group_name, run_date, - send_state="held", + send_state="failed_final", send_hold=True, - send_hold_reason="MISSED_SEND_WINDOW", + send_hold_reason="SEND_PRE_SUBMIT_EXCEPTION", needs_manual_send=True, - send_error="已超过到点后 30 分钟自动补发窗口，需人工确认", - send_error_type="MISSED_SEND_WINDOW", - missed_send_window_at=now.isoformat(), + send_error=detail, + send_error_type="SEND_PRE_SUBMIT_FAILED", + send_next_retry_at="", ) - results.append( - { - "group_name": group_name, - "status": "held", - "error_type": "MISSED_SEND_WINDOW", - "detail": "已超过自动补发窗口，需人工确认", - } + except Exception: + logger.exception( + "发送提交前异常状态写入失败：group=%s date=%s", + group_name, + run_date, ) - continue - result = self._send_one(group, group_name, run, run_date, now) - results.append(result) - return results + return ( + { + "group_name": group_name, + "status": "failed", + "error_type": "SEND_PRE_SUBMIT_FAILED", + "detail": detail, + }, + False, + ) + + claim_id = str(run.get("send_claim_id") or "") + marked, _, reason = self.store.mark_send_result_unknown( + group_name, + run_date, + claim_id, + stage=unresolved_stage, + detail=detail, + now=now, + ) + if not marked: + detail = f"{detail}；unknown 状态持久化失败（{reason}）" + return ( + { + "group_name": group_name, + "status": "held", + "error_type": "SEND_RESULT_UNKNOWN", + "detail": detail, + }, + True, + ) def _send_one( self, @@ -734,282 +820,280 @@ def _send_one( allow_hold: bool = False, allow_sent: bool = False, ) -> dict: - target = effective_send_target(group) - self.store.update( + """保留原注入点；发送 claim 和部分成功由发送阶段负责。""" + return DeliveryStages( + settings=self.settings, + sender=self.sender, + store=self.store, + name_sync_audit=self._name_sync_audit, + logger=logger, + ).run( + group, group_name, + run, run_date, - wechat_group_name=str(group.wechat_group_name or "").strip(), - effective_send_target=target, - send_target_mode=send_target_mode(group), - **self._name_sync_audit(group), + now, + allow_hold=allow_hold, + allow_sent=allow_sent, ) - claim_id, run, claim_reason = self.store.claim_send( + + def _finish_unknown_send( + self, + group_name: str, + run_date: str, + claim_id: str, + stage: str, + detail: str, + *, + submitted_at: str = "", + ) -> dict: + """保留原注入点；未知结果继续 fail-closed。""" + return DeliveryStages( + settings=self.settings, + sender=self.sender, + store=self.store, + name_sync_audit=self._name_sync_audit, + logger=logger, + ).finish_unknown( group_name, run_date, - now=now, - lease_seconds=self.settings.wechat_send_claim_seconds, - allow_hold=allow_hold, - allow_sent=allow_sent, + claim_id, + stage, + detail, + submitted_at=submitted_at, ) - if not claim_id: - if claim_reason == "result_unknown": - return { - "group_name": group_name, - "status": "held", - "error_type": "SEND_RESULT_UNKNOWN", - "detail": "上次发送结果未知，已暂停自动重试", - } + + # ================= 手动操作 ================= + + def resolve_prompt_unknown( + self, + group_id: int, + run_date: str, + *, + expected_operation_id: str, + ) -> dict: + """人工确认丢弃未知 Prompt 结果；本方法本身不调用任何外部模型。""" + parsed_run_date = parse_date(run_date) + if parsed_run_date is None: + return { + "status": "failed", + "error_type": "INVALID_RUN_DATE", + "detail": "run_date 必须是有效的 YYYY-MM-DD 日期", + } + group = self._get_group(group_id) + if group is None: + return { + "status": "failed", + "error_type": "GROUP_NOT_FOUND", + "detail": f"群不存在 {group_id}", + } + run_date = parsed_run_date.isoformat() + group_name = self._group_name(group) + resolved, run, reason = self.store.resolve_prompt_result_unknown( + group_name, + run_date, + expected_operation_id=expected_operation_id, + now=datetime.now(ZoneInfo(self.settings.app_timezone)), + ) + if not resolved: + messages = { + "state_corrupt": "运行状态损坏，禁止人工覆盖", + "not_unknown": "当前任务已不是 Prompt 结果未知状态", + "stale": "Prompt 未知状态已变化，请刷新后重新核对", + "result_available": "已有可恢复 Prompt 结果，禁止丢弃", + } return { "group_name": group_name, - "status": "skipped", - "detail": f"发送任务未领取：{claim_reason}", + "status": "conflict", + "error_type": "PROMPT_RESOLUTION_CONFLICT", + "detail": messages.get(reason, "Prompt 未知状态无法消歧"), + "reason": reason, } + return { + "group_name": group_name, + "status": "resolved", + "resolution": "discard_and_retry", + "next_stage": "prompt", + "updated_at": run.get("updated_at"), + "detail": "已解除 Prompt 未知暂停；本次确认没有调用外部模型", + } - ranking_txt = self.store.ranking_txt_path(group_name, run_date) - image = self.store.image_path(group_name, run_date) - - try: - ranking_text = ranking_txt.read_text(encoding="utf-8") - except (OSError, UnicodeError): - self.store.finish_send_claim( - group_name, run_date, claim_id, send_state="failed", - status=FAILED, failed_stage="send", error="ranking.txt 缺失或无法读取", - error_type="SEND_TEXT_FAILED", - ) + def resolve_send_unknown( + self, + group_id: int, + run_date: str, + *, + resolution: str, + expected_send_unknown_at: str, + ) -> dict: + """人工消歧文字提交检查点；不会调用微信 Sender。""" + parsed_run_date = parse_date(run_date) + if parsed_run_date is None: return { - "group_name": group_name, "status": "failed", - "error_type": "SEND_TEXT_FAILED", - "detail": "ranking.txt 缺失或无法读取", + "error_type": "INVALID_RUN_DATE", + "detail": "run_date 必须是有效的 YYYY-MM-DD 日期", } - if not ranking_text.strip(): - self.store.finish_send_claim( - group_name, run_date, claim_id, send_state="failed", - status=FAILED, failed_stage="send", error="ranking.txt 为空", - error_type="SEND_TEXT_FAILED", - ) + group = self._get_group(group_id) + if group is None: return { - "group_name": group_name, "status": "failed", - "error_type": "SEND_TEXT_FAILED", - "detail": "ranking.txt 为空", + "error_type": "GROUP_NOT_FOUND", + "detail": f"群不存在 {group_id}", } - - image_enabled = bool(group.image_enabled) - if image_enabled: - image_ok, image_detail = verify_image(image) - if not image_ok: - self.store.finish_send_claim( - group_name, run_date, claim_id, send_state="failed", - status=FAILED, failed_stage="send", error=image_detail, - error_type=IMAGE_FILE_MISSING, - ) - return { - "group_name": group_name, - "status": "failed", - "error_type": IMAGE_FILE_MISSING, - "detail": image_detail, - } - - # 图片曾失败而文字已经确认时，只补发图片，避免分钟级重试重复发文字。 - text_already_sent = bool(run.get("text_sent_at")) - image_path = str(image.resolve()) if image_enabled else None - text_sent_at = str(run.get("text_sent_at") or "") - verification_levels: list[str] = [] - - if not text_already_sent: - started_at = datetime.now(now.tzinfo).isoformat() - updated, _ = self.store.update_send_claim( - group_name, - run_date, - claim_id, - send_state="sending_text", - text_attempt_started_at=started_at, - text_attempt_finished_at="", - text_submitted_at="", - text_verified_at="", - ) - if not updated: - return {"group_name": group_name, "status": "skipped", "detail": "发送 claim 已失效"} - try: - text_result = self.sender.send_text(target, ranking_text) - except Exception as exc: - return self._finish_unknown_send( - group_name, run_date, claim_id, "text", f"文字发送异常：{exc}" - ) - finished_at = datetime.now(now.tzinfo).isoformat() - if text_result.outcome_unknown: - return self._finish_unknown_send( - group_name, run_date, claim_id, "text", text_result.detail, - submitted_at=finished_at if text_result.submitted else "", - ) - if not text_result.success: - self.store.finish_send_claim( - group_name, - run_date, - claim_id, - send_state="ready", - status=run.get("status", READY_TO_SEND), - text_attempt_finished_at=finished_at, - text_submitted_at=finished_at if text_result.submitted else "", - send_error=text_result.detail, - send_error_type="SEND_TEXT_FAILED", - ) - return { - "group_name": group_name, - "status": "failed", - "error_type": "SEND_TEXT_FAILED", - "detail": text_result.detail, - } - text_sent_at = text_result.sent_at or finished_at - text_level = text_result.verification_level or "provider_reported" - verification_levels.append(text_level) - self.store.update_send_claim( - group_name, - run_date, - claim_id, - send_state="text_verified", - text_attempt_finished_at=finished_at, - text_submitted_at=finished_at if text_result.submitted or text_result.success else "", - text_verified_at=finished_at, - text_sent_at=text_sent_at, - text_verification_level=text_level, - send_error="", - send_error_type="", - ) - elif run.get("text_verification_level"): - verification_levels.append(str(run["text_verification_level"])) - - image_sent_at = str(run.get("image_sent_at") or "") - if image_enabled: - started_at = datetime.now(now.tzinfo).isoformat() - self.store.update_send_claim( - group_name, - run_date, - claim_id, - send_state="sending_image", - image_attempt_started_at=started_at, - image_attempt_finished_at="", - image_submitted_at="", - image_verified_at="", - ) - try: - image_result = self.sender.send_image(target, image_path) - except Exception as exc: - return self._finish_unknown_send( - group_name, run_date, claim_id, "image", f"图片发送异常：{exc}" - ) - finished_at = datetime.now(now.tzinfo).isoformat() - if image_result.outcome_unknown: - return self._finish_unknown_send( - group_name, run_date, claim_id, "image", image_result.detail, - submitted_at=finished_at if image_result.submitted else "", - ) - if not image_result.success: - self.store.finish_send_claim( - group_name, - run_date, - claim_id, - send_state="ready", - status=run.get("status", READY_TO_SEND), - text_sent_at=text_sent_at, - image_attempt_finished_at=finished_at, - image_submitted_at=finished_at if image_result.submitted else "", - send_error=image_result.detail, - send_error_type="SEND_IMAGE_FAILED", - ) - return { - "group_name": group_name, - "status": "failed", - "error_type": "SEND_IMAGE_FAILED", - "detail": image_result.detail, - } - image_sent_at = image_result.sent_at or finished_at - image_level = image_result.verification_level or "provider_reported" - verification_levels.append(image_level) - self.store.update_send_claim( - group_name, - run_date, - claim_id, - send_state="image_verified", - image_attempt_finished_at=finished_at, - image_submitted_at=finished_at if image_result.submitted or image_result.success else "", - image_verified_at=finished_at, - image_sent_at=image_sent_at, - image_verification_level=image_level, - send_error="", - send_error_type="", - ) - - if verification_levels and all(level == "ui_observed" for level in verification_levels): - verification_level = "ui_observed" - elif verification_levels and all(level == "dry_run" for level in verification_levels): - verification_level = "dry_run" - else: - verification_level = "provider_reported" - self.store.finish_send_claim( + run_date = parsed_run_date.isoformat() + group_name = self._group_name(group) + resolved, run, reason = self.store.resolve_text_send_unknown( group_name, run_date, - claim_id, - send_state="sent", - status=SENT, - sent_at=now.isoformat(), - sent_target=target, - text_sent_at=text_sent_at, - image_sent_at=image_sent_at, - send_error="", - send_error_type="", - verification_level=verification_level, - send_hold=False, - send_hold_reason="", - needs_manual_send=False, - image_regen_status="sent" if run.get("image_regen_status") == "ready_for_review" else run.get("image_regen_status"), + resolution=resolution, + expected_send_unknown_at=expected_send_unknown_at, + now=datetime.now(ZoneInfo(self.settings.app_timezone)), ) - if image_enabled: - logger.info("群 %s 已发送（文字+图片）→ SENT", group_name) - detail = "文字和图片已发送" - else: - logger.info("群 %s 已发送（仅文字，未启用图片）→ SENT", group_name) - detail = "文字已发送（未启用图片）" - return {"group_name": group_name, "status": "sent", "detail": detail, "sent_at": now.isoformat()} + if not resolved: + messages = { + "state_corrupt": "运行状态损坏，禁止人工覆盖", + "not_unknown": "当前任务已不是发送结果未知状态", + "stale": "发送未知状态已变化，请刷新后重新核对", + "unsupported_stage": "当前未知发生在图片阶段，此接口只处理文字提交", + "invalid_resolution": "人工核对结论无效", + "text_not_submitted": "没有文字提交动作记录，不能确认文字已发送", + } + return { + "group_name": group_name, + "status": "conflict", + "error_type": "SEND_RESOLUTION_CONFLICT", + "detail": messages.get(reason, "发送未知状态无法消歧"), + "reason": reason, + } + return { + "group_name": group_name, + "status": "resolved", + "resolution": resolution, + "next_stage": "image" if resolution == "text_sent" and bool(group.image_enabled) else "text" if resolution == "not_sent" else "complete", + "send_state": run.get("send_state"), + "detail": "已记录人工核对结论；本次操作没有发送任何微信内容", + } - def _finish_unknown_send( + def reset_explicit_send_failure( self, - group_name: str, + group_id: int, run_date: str, - claim_id: str, - stage: str, - detail: str, *, - submitted_at: str = "", + expected_updated_at: str, + expected_state_version: int, ) -> dict: - finished_at = datetime.now(ZoneInfo(self.settings.app_timezone)).isoformat() - fields = { - f"{stage}_attempt_finished_at": "", - f"{stage}_submitted_at": submitted_at, - } - self.store.finish_send_claim( + """解除明确未提交的重试耗尽状态；不会调用微信 Sender。""" + parsed_run_date = parse_date(run_date) + if parsed_run_date is None: + return { + "status": "failed", + "error_type": "INVALID_RUN_DATE", + "detail": "run_date 必须是有效的 YYYY-MM-DD 日期", + } + group = self._get_group(group_id) + if group is None: + return { + "status": "failed", + "error_type": "GROUP_NOT_FOUND", + "detail": f"群不存在 {group_id}", + } + run_date = parsed_run_date.isoformat() + group_name = self._group_name(group) + reset, run, reason = self.store.reset_explicit_send_failure( group_name, run_date, - claim_id, - send_state="unknown", - send_hold=True, - send_hold_reason="SEND_RESULT_UNKNOWN", - needs_manual_send=True, - send_error=detail, - send_error_type="SEND_RESULT_UNKNOWN", - verification_level="unknown", - send_unknown_at=finished_at, - **fields, + expected_updated_at=expected_updated_at, + expected_state_version=expected_state_version, + now=datetime.now(ZoneInfo(self.settings.app_timezone)), ) + if not reset: + messages = { + "state_corrupt": "运行状态损坏，禁止恢复发送", + "stale": "任务状态已变化，请刷新后重新核对", + "not_resolvable": "当前任务状态不能恢复发送", + "not_explicit_failure": "当前任务不是明确未提交的重试耗尽状态", + "active_claim": "当前任务仍有发送 claim，禁止并发恢复", + "unresolved_attempt": "存在未决发送动作，必须人工核对结果", + "submission_evidence": "已有提交、验证或发送证据，禁止自动重试", + } + return { + "group_name": group_name, + "status": "conflict", + "error_type": "SEND_FAILURE_RESET_CONFLICT", + "detail": messages.get(reason, "当前发送失败不能安全恢复"), + "reason": reason, + } return { "group_name": group_name, - "status": "held", - "error_type": "SEND_RESULT_UNKNOWN", - "detail": detail, + "status": "prepared", + "send_state": run.get("send_state"), + "run_status": run.get("status"), + "updated_at": run.get("updated_at"), + "state_version": run.get("state_version"), + "detail": "已解除明确未提交的失败暂停；本次操作没有调用微信发送器", } - # ================= 手动操作 ================= + def resolve_manual_send( + self, + group_id: int, + run_date: str, + *, + resolution: str, + expected_updated_at: str, + ) -> dict: + """人工核对整单发送状态；只更新 run.json，不调用微信 Sender。""" + parsed_run_date = parse_date(run_date) + if parsed_run_date is None: + return { + "status": "failed", + "error_type": "INVALID_RUN_DATE", + "detail": "run_date 必须是有效的 YYYY-MM-DD 日期", + } + group = self._get_group(group_id) + if group is None: + return { + "status": "failed", + "error_type": "GROUP_NOT_FOUND", + "detail": f"群不存在 {group_id}", + } + run_date = parsed_run_date.isoformat() + group_name = self._group_name(group) + resolved, run, reason = self.store.resolve_manual_send( + group_name, + run_date, + resolution=resolution, + expected_updated_at=expected_updated_at, + image_required=bool(group.image_enabled), + now=datetime.now(ZoneInfo(self.settings.app_timezone)), + ) + if not resolved: + messages = { + "state_corrupt": "运行状态损坏，禁止人工覆盖", + "stale": "任务状态已变化，请刷新后重新核对", + "not_resolvable": "当前任务状态不能进行人工发送核对", + "not_held": "当前任务并未暂停待核对，请刷新后确认", + "invalid_resolution": "人工核对结论无效", + "ranking_missing": "排行榜文案不存在或为空，不能确认已发送", + "image_missing": "日报图片不存在或为空，不能确认文字和图片均已发送", + } + return { + "group_name": group_name, + "status": "conflict", + "error_type": "MANUAL_SEND_RESOLUTION_CONFLICT", + "detail": messages.get(reason, "当前发送状态无法人工处理"), + "reason": reason, + } + return { + "group_name": group_name, + "status": "resolved", + "resolution": resolution, + "next_stage": "complete" if resolution == "all_sent" else "image" if resolution == "text_sent" and bool(group.image_enabled) else "text", + "send_state": run.get("send_state"), + "run_status": run.get("status"), + "updated_at": run.get("updated_at"), + "detail": "已写入人工核对结论；本次操作没有调用微信发送器", + } def force_generate( self, @@ -1038,7 +1122,19 @@ def force_generate( "error": "run_date 必须是有效的 YYYY-MM-DD 日期", } run_date = parsed_run_date.isoformat() - self._last_name_sync_report = self._sync_group_names([group_id]) + group = self._get_group(group_id) + if not group: + return {"status": "failed", "error": f"群不存在 {group_id}"} + group_name = self._group_name(group) + current = self.store.load_run(group_name, run_date) + if current.get("status") == CORRUPT: + return { + "group_name": group_name, + "status": "blocked", + "error_type": RUN_STATE_CORRUPT, + "detail": "运行状态文件损坏，需人工复核", + } + self._last_name_sync_report = self._sync_group_names_safe([group_id]) group = self._get_group(group_id) if not group: return {"status": "failed", "error": f"群不存在 {group_id}"} @@ -1060,6 +1156,7 @@ def rebuild_prompt_from_snapshot( group_id: int, run_date: str, *, + allow_topic_reselection: bool = False, acquire_lock: bool = True, ) -> dict: """只从当天 messages.json 重建排行榜和 Prompt，不取数、不生图。""" @@ -1068,6 +1165,7 @@ def rebuild_prompt_from_snapshot( return self.rebuild_prompt_from_snapshot( group_id, run_date, + allow_topic_reselection=allow_topic_reselection, acquire_lock=False, ) @@ -1101,15 +1199,86 @@ def rebuild_prompt_from_snapshot( "error_type": "IMAGE_REGEN_BUSY", "detail": "该运行正在生图，请完成后再重建 Prompt", } + current_prompt_meta = ( + current.get("prompt_meta") + if isinstance(current.get("prompt_meta"), dict) + else {} + ) + has_topic_selection = isinstance(current_prompt_meta.get("topic_selection"), dict) + try: + snapshot_messages = self._load_message_snapshot(snapshot_path) + attribution = build_attribution_contract(snapshot_messages) + except (OSError, UnicodeError, json.JSONDecodeError, ValueError) as exc: + return { + "group_name": group_name, + "status": "failed", + "error_type": "MESSAGE_SNAPSHOT_INVALID", + "detail": f"messages.json 无法建立说话人归属契约：{exc}", + } + stored_snapshot = str( + current_prompt_meta.get("message_snapshot_sha256") or "" + ) + stored_speakers = str(current_prompt_meta.get("speaker_fingerprint") or "") + stored_selection = ( + current_prompt_meta.get("topic_selection") + if isinstance(current_prompt_meta.get("topic_selection"), dict) + else {} + ) + attribution_matches = bool( + stored_snapshot + and stored_speakers + and stored_snapshot == attribution.message_snapshot_sha256 + and stored_speakers == attribution.speaker_fingerprint + and str(stored_selection.get("message_snapshot_sha256") or "") + == attribution.message_snapshot_sha256 + and str(stored_selection.get("speaker_fingerprint") or "") + == attribution.speaker_fingerprint + ) + persisted_selection_valid = has_topic_selection and attribution_matches + if not persisted_selection_valid and not allow_topic_reselection: + current_hold_reason = str(current.get("send_hold_reason") or "") + preserved_user_hold_reason = ( + current_hold_reason + if current_hold_reason.startswith("USER_REQUEST_NO_SEND_") + else "" + ) + self.store.update( + group_name, + run_date, + prompt_rebuild_status="required", + prompt_rebuild_error="消息快照或说话人指纹与旧选题不一致", + prompt_stale=True, + image_stale=True, + artifact_stale_reason="TOPIC_SELECTION_SNAPSHOT_INVALID", + send_hold=True, + send_hold_reason=( + preserved_user_hold_reason + or "TOPIC_SELECTION_SNAPSHOT_INVALID" + ), + needs_manual_send=True, + ) + return { + "group_name": group_name, + "status": "failed", + "error_type": "TOPIC_SELECTION_SNAPSHOT_INVALID", + "detail": "旧选题缺少匹配的消息快照/说话人指纹；已停止且不会隐式重新选题", + } + reselect_topics = not persisted_selection_valid keep_sent = current.get("status") == SENT + current_hold_reason = str(current.get("send_hold_reason") or "") + preserved_user_hold_reason = ( + current_hold_reason + if current_hold_reason.startswith("USER_REQUEST_NO_SEND_") + else "" + ) self.store.update( group_name, run_date, prompt_rebuild_status="running", prompt_rebuild_error="", send_hold=True, - send_hold_reason="PROMPT_REBUILDING", + send_hold_reason=preserved_user_hold_reason or "PROMPT_REBUILDING", needs_manual_send=True, ) window = self.period_resolver.resolve( @@ -1122,17 +1291,32 @@ def rebuild_prompt_from_snapshot( run_date, force=True, refresh_messages=False, + reuse_persisted_topic_selection=not reselect_topics, ) - if result.get("status") == "failed": + rebuilt = self.store.load_run(group_name, run_date) + if result.get("status") == "failed" or bool(rebuilt.get("image_force_local_fallback")): + detail = str( + result.get("detail") + or result.get("error") + or rebuilt.get("prompt_original_error") + or "重建失败" + )[:500] self.store.update( group_name, run_date, + status=SENT if keep_sent else FAILED, prompt_rebuild_status="failed", - prompt_rebuild_error=str(result.get("detail") or result.get("error") or "重建失败")[:500], + prompt_rebuild_error=detail, + prompt_topic_reselected=reselect_topics, send_hold=True, needs_manual_send=True, ) - return result + return { + "group_name": group_name, + "status": "failed", + "error_type": str(result.get("error_type") or PROMPT_FAILED), + "detail": detail, + } self.store.update( group_name, @@ -1142,16 +1326,119 @@ def rebuild_prompt_from_snapshot( prompt_rebuild_error="", image_regen_status="prompt_rebuilt", image_regen_error="", + prompt_topic_reselected=reselect_topics, send_hold=True, - send_hold_reason="PROMPT_REBUILT_REVIEW_REQUIRED", + send_hold_reason=( + preserved_user_hold_reason or "PROMPT_REBUILT_REVIEW_REQUIRED" + ), needs_manual_send=True, ) return { "group_name": group_name, "status": "prompt_ready", - "detail": "已从当天 messages.json 重建排行榜和 Prompt；未取数，未生图", + "detail": ( + "已从保存的 messages.json 显式重新选题并重建 Prompt；未取数，未生图" + if reselect_topics + else "已复用 run.json 中已校验选题和既定分镜重建 Prompt；未取数，未生图" + ), } + def rebuild_prompts_from_snapshots( + self, + targets: list[tuple[int, str, str]], + *, + acquire_lock: bool = True, + ) -> list[dict]: + """按稳定群 ID 并行重建已有快照；整个批次不访问微信或生图。""" + if acquire_lock: + with generation_mutex(): + return self.rebuild_prompts_from_snapshots( + targets, + acquire_lock=False, + ) + if not targets: + return [] + + seen: set[tuple[int, str]] = set() + normalized: list[tuple[int, str, str]] = [] + for group_id, wechat_group_id, run_date in targets: + key = (int(group_id), str(run_date)) + if key in seen: + continue + seen.add(key) + normalized.append((int(group_id), str(wechat_group_id), str(run_date))) + + def rebuild_one(target: tuple[int, str, str]) -> dict: + group_id, expected_wechat_group_id, run_date = target + group = self._get_group(group_id) + if group is None: + return { + "group_id": group_id, + "status": "failed", + "error_type": "GROUP_NOT_FOUND", + "detail": f"群不存在 {group_id}", + } + actual_wechat_group_id = str(group.wechat_group_id or "").strip() + if not expected_wechat_group_id or actual_wechat_group_id != expected_wechat_group_id.strip(): + return { + "group_id": group_id, + "status": "failed", + "error_type": "GROUP_IDENTITY_MISMATCH", + "detail": "group_id 与 wechat_group_id 不匹配，已停止重建", + } + group_name = self._group_name(group) + current = self.store.load_run(group_name, run_date) + try: + run_group_id = int(current.get("group_id")) + except (TypeError, ValueError): + run_group_id = -1 + run_wechat_group_id = str(current.get("wechat_group_id") or "").strip() + if run_group_id != group_id or ( + run_wechat_group_id and run_wechat_group_id != actual_wechat_group_id + ): + return { + "group_id": group_id, + "group_name": group_name, + "status": "failed", + "error_type": "RUN_IDENTITY_MISMATCH", + "detail": "run.json 群身份与目标不一致，已停止重建", + } + result = self.rebuild_prompt_from_snapshot( + group_id, + run_date, + acquire_lock=False, + ) + result.setdefault("group_id", group_id) + result.setdefault("wechat_group_id", actual_wechat_group_id) + return result + + # 快照重建本身只读本地文件，允许六个群同时进入工作池；真正的 + # Codex 文本调用仍由 provider 侧的 2 路信号量限流。 + limit = min(len(normalized), 6) + if limit <= 1: + return [rebuild_one(target) for target in normalized] + results: dict[int, dict] = {} + with ThreadPoolExecutor( + max_workers=limit, + thread_name_prefix="groupbrief-prompt-rebuild", + ) as executor: + futures = { + executor.submit(rebuild_one, target): index + for index, target in enumerate(normalized) + } + for future in as_completed(futures): + index = futures[future] + try: + results[index] = future.result() + except Exception as exc: + results[index] = { + "group_id": normalized[index][0], + "status": "failed", + "error_type": "PROMPT_REBUILD_FAILED", + "detail": str(exc)[:500], + } + return [results[index] for index in range(len(normalized))] + def force_send( self, group_id: int, @@ -1172,7 +1459,7 @@ def force_send( "error": "run_date 必须是有效的 YYYY-MM-DD 日期", } run_date = parsed_run_date.isoformat() - self._last_name_sync_report = self._sync_group_names([group_id]) + self._last_name_sync_report = self._sync_group_names_safe([group_id]) group = self._get_group(group_id) if not group: return {"status": "failed", "error": f"群不存在 {group_id}"} @@ -1186,7 +1473,7 @@ def force_send( if run.get("status") not in (IMAGE_READY, READY_TO_SEND) and not can_resend_review: return {"status": "failed", "error": f"状态 {run.get('status')} 不可发送"} scheduled_date = parse_date(run_date) - send_clock = parse_send_time(group.send_time or run.get("send_time", "08:30")) + send_clock = parse_send_time(self.settings.schedule_send_time) scheduled_at = datetime.combine(scheduled_date, send_clock, tzinfo=now.tzinfo) late_cutoff = scheduled_at + timedelta( minutes=max(int(self.settings.wechat_late_send_window_minutes), 0) @@ -1238,8 +1525,14 @@ def force_send( def _sync_group_names(self, group_ids: list[int] | None = None) -> GroupNameSyncReport: from sqlmodel import Session - with Session(repo.engine) as session: - report = GroupNameSyncService(self.data_source).sync(session, group_ids=group_ids) + def operation() -> GroupNameSyncReport: + with Session(repo.engine) as session: + return GroupNameSyncService(self.data_source).sync(session, group_ids=group_ids) + + report = run_with_sqlite_retry( + operation, + max_attempts=self.settings.sqlite_retry_max_attempts, + ) logger.info( "流水线群名同步：status=%s checked=%d updated=%d skipped=%d", report.status, @@ -1249,6 +1542,22 @@ def _sync_group_names(self, group_ids: list[int] | None = None) -> GroupNameSync ) return report + def _sync_group_names_safe( + self, + group_ids: list[int] | None = None, + ) -> GroupNameSyncReport: + """实时群名同步失败时保留缓存名称，不让全批生成/发送消失。""" + try: + return self._sync_group_names(group_ids) + except Exception as exc: + logger.exception("流水线群名同步异常，降级使用数据库缓存名称") + return GroupNameSyncReport( + status="unavailable", + source=str(getattr(self.data_source, "name", "unknown") or "unknown"), + checked=len(group_ids or []), + detail=f"群名同步异常：{type(exc).__name__}", + ) + def _name_sync_audit(self, group: Group) -> dict: mode = send_target_mode(group) report = self._last_name_sync_report @@ -1268,6 +1577,8 @@ def _due_sync_group_ids( groups: list[Group], run_date: str, now: datetime, + *, + recovery: bool = False, ) -> list[int]: late_window = timedelta(minutes=max(int(self.settings.wechat_late_send_window_minutes), 0)) group_ids: list[int] = [] @@ -1280,17 +1591,23 @@ def _due_sync_group_ids( continue if run.get("sent_at") or run.get("send_hold"): continue - send_time = parse_send_time(group.send_time or run.get("send_time", "08:30")) - due_at = datetime.combine(now.date(), send_time, tzinfo=now.tzinfo) - if due_at <= now <= due_at + late_window: + send_time = parse_send_time(self.settings.schedule_send_time) + due_at = datetime.combine(date.fromisoformat(run_date), send_time, tzinfo=now.tzinfo) + if due_at <= now and (recovery or now <= due_at + late_window): group_ids.append(int(group.id)) return group_ids def _load_groups(self, group_ids: list[int] | None = None) -> list[Group]: from sqlmodel import Session - with Session(repo.engine) as session: - groups = repo.list_groups(session, only_enabled=True) + def operation() -> list[Group]: + with Session(repo.engine) as session: + return repo.list_groups(session, only_enabled=True) + + groups = run_with_sqlite_retry( + operation, + max_attempts=self.settings.sqlite_retry_max_attempts, + ) if group_ids: groups = [g for g in groups if g.id in group_ids] return groups @@ -1298,8 +1615,14 @@ def _load_groups(self, group_ids: list[int] | None = None) -> list[Group]: def _get_group(self, group_id: int) -> Group | None: from sqlmodel import Session - with Session(repo.engine) as session: - return repo.get_active_group(session, group_id) + def operation() -> Group | None: + with Session(repo.engine) as session: + return repo.get_active_group(session, group_id) + + return run_with_sqlite_retry( + operation, + max_attempts=self.settings.sqlite_retry_max_attempts, + ) def _save_json(self, path: Path, data) -> None: path.parent.mkdir(parents=True, exist_ok=True) @@ -1313,6 +1636,7 @@ def _load_message_snapshot(path: Path) -> list[V2Message]: raise ValueError("messages.json 必须是非空数组") messages: list[V2Message] = [] + seen_message_ids: set[str] = set() for index, item in enumerate(payload, start=1): if not isinstance(item, dict): raise ValueError(f"第 {index} 条消息不是对象") @@ -1325,6 +1649,9 @@ def _load_message_snapshot(path: Path) -> list[V2Message]: group_id = str(item.get("group_id") or "").strip() if not message_id or not group_id: raise ValueError(f"第 {index} 条消息缺少 message_id/group_id") + if message_id in seen_message_ids: + raise ValueError(f"第 {index} 条消息的 message_id 重复：{message_id}") + seen_message_ids.add(message_id) messages.append( V2Message( message_id=message_id, @@ -1335,6 +1662,12 @@ def _load_message_snapshot(path: Path) -> list[V2Message]: timestamp=timestamp, message_type=str(item.get("message_type") or "text"), content=str(item.get("content") or ""), + upstream_sender_name=str(item.get("upstream_sender_name") or ""), + sender_name_source=str( + item.get("sender_name_source") + if "sender_name_source" in item + else "snapshot" + ), ) ) return messages diff --git a/app/pipeline/delivery_stages.py b/app/pipeline/delivery_stages.py new file mode 100644 index 0000000..b7334e2 --- /dev/null +++ b/app/pipeline/delivery_stages.py @@ -0,0 +1,746 @@ +"""DailyPipeline 的微信发送阶段实现。""" + +from __future__ import annotations + +import hashlib +from dataclasses import dataclass, field +from datetime import datetime +from zoneinfo import ZoneInfo + +from app.config.settings import Settings +from app.core.observability import log_event +from app.db.models import Group +from app.image.image_task import verify_image +from app.pipeline.stage_result import StageResult +from app.sender.base import WechatSender +from app.services.group_name_sync import effective_send_target, send_target_mode +from app.v2.constants import FAILED, IMAGE_FILE_MISSING, READY_TO_SEND, SENT +from app.v2.run_store import RunStore + + +@dataclass +class DeliveryContext: + group: Group + group_name: str + run: dict + run_date: str + now: datetime + allow_hold: bool + allow_sent: bool + target: str + claim_id: str = "" + ranking_text: str = "" + image_enabled: bool = False + image_path: str | None = None + text_sha256: str = "" + image_sha256: str = "" + text_sent_at: str = "" + image_sent_at: str = "" + verification_levels: list[str] = field(default_factory=list) + + +class DeliveryStages: + """按 claim、预检、文字、图片、终态顺序执行微信发送。""" + + def __init__( + self, + *, + settings: Settings, + sender: WechatSender, + store: RunStore, + name_sync_audit, + logger, + ) -> None: + self.settings = settings + self.sender = sender + self.store = store + self._name_sync_audit = name_sync_audit + self.logger = logger + + def run( + self, + group: Group, + group_name: str, + run: dict, + run_date: str, + now: datetime, + *, + allow_hold: bool = False, + allow_sent: bool = False, + ) -> dict: + context = DeliveryContext( + group=group, + group_name=group_name, + run=run, + run_date=run_date, + now=now, + allow_hold=allow_hold, + allow_sent=allow_sent, + target=effective_send_target(group), + ) + + claimed = self._claim(context) + if claimed.is_terminal: + return claimed.terminal_response() + context = claimed.next_value() + + prepared = self._prepare_payload(context) + if prepared.is_terminal: + return prepared.terminal_response() + context = prepared.next_value() + + text_stage = self._send_text(context) + if text_stage.is_terminal: + return text_stage.terminal_response() + context = text_stage.next_value() + + image_stage = self._send_image(context) + if image_stage.is_terminal: + return image_stage.terminal_response() + return self._complete(image_stage.next_value()) + + def _claim(self, context: DeliveryContext) -> StageResult[DeliveryContext]: + group = context.group + self.store.update( + context.group_name, + context.run_date, + wechat_group_name=str(group.wechat_group_name or "").strip(), + effective_send_target=context.target, + send_target_mode=send_target_mode(group), + **self._name_sync_audit(group), + ) + claim_id, run, claim_reason = self.store.claim_send( + context.group_name, + context.run_date, + now=context.now, + lease_seconds=self.settings.wechat_send_claim_seconds, + allow_hold=context.allow_hold, + allow_sent=context.allow_sent, + ) + if not claim_id: + if claim_reason == "result_unknown": + return StageResult.stop( + { + "group_name": context.group_name, + "status": "held", + "error_type": "SEND_RESULT_UNKNOWN", + "detail": "上次发送结果未知，已暂停自动重试", + } + ) + if claim_reason == "failed_final": + return StageResult.stop( + { + "group_name": context.group_name, + "status": "failed_final", + "error_type": "SEND_RETRY_EXHAUSTED", + "detail": "发送重试预算已耗尽，已暂停自动重试", + } + ) + if claim_reason == "retry_not_due": + return StageResult.stop( + { + "group_name": context.group_name, + "status": "retry_scheduled", + "detail": "发送重试尚未到期", + "next_retry_at": run.get("send_next_retry_at"), + } + ) + return StageResult.stop( + { + "group_name": context.group_name, + "status": "skipped", + "detail": f"发送任务未领取：{claim_reason}", + } + ) + context.claim_id = claim_id + context.run = run + return StageResult.proceed(context) + + def _prepare_payload( + self, + context: DeliveryContext, + ) -> StageResult[DeliveryContext]: + ranking_path = self.store.ranking_txt_path( + context.group_name, + context.run_date, + ) + image_path = self.store.image_path(context.group_name, context.run_date) + try: + ranking_text = ranking_path.read_text(encoding="utf-8") + except (OSError, UnicodeError): + self.store.finish_send_claim( + context.group_name, + context.run_date, + context.claim_id, + send_state="failed", + status=FAILED, + failed_stage="send", + error="ranking.txt 缺失或无法读取", + error_type="SEND_TEXT_FAILED", + ) + return StageResult.stop( + { + "group_name": context.group_name, + "status": "failed", + "error_type": "SEND_TEXT_FAILED", + "detail": "ranking.txt 缺失或无法读取", + } + ) + if not ranking_text.strip(): + self.store.finish_send_claim( + context.group_name, + context.run_date, + context.claim_id, + send_state="failed", + status=FAILED, + failed_stage="send", + error="ranking.txt 为空", + error_type="SEND_TEXT_FAILED", + ) + return StageResult.stop( + { + "group_name": context.group_name, + "status": "failed", + "error_type": "SEND_TEXT_FAILED", + "detail": "ranking.txt 为空", + } + ) + + context.ranking_text = ranking_text + context.text_sha256 = hashlib.sha256(ranking_text.encode("utf-8")).hexdigest() + context.image_enabled = bool(context.group.image_enabled) + if context.image_enabled: + image_ok, image_detail = verify_image(image_path) + if not image_ok: + self.store.finish_send_claim( + context.group_name, + context.run_date, + context.claim_id, + send_state="failed", + status=FAILED, + failed_stage="send", + error=image_detail, + error_type=IMAGE_FILE_MISSING, + ) + return StageResult.stop( + { + "group_name": context.group_name, + "status": "failed", + "error_type": IMAGE_FILE_MISSING, + "detail": image_detail, + } + ) + context.image_path = str(image_path.resolve()) + try: + context.image_sha256 = hashlib.sha256(image_path.read_bytes()).hexdigest() + except OSError: + self.store.finish_send_claim( + context.group_name, + context.run_date, + context.claim_id, + send_state="failed", + status=FAILED, + failed_stage="send", + error="图片无法读取以生成送达证据", + error_type=IMAGE_FILE_MISSING, + ) + return StageResult.stop( + { + "group_name": context.group_name, + "status": "failed", + "error_type": IMAGE_FILE_MISSING, + "detail": "图片无法读取以生成送达证据", + } + ) + evidence = { + "target": context.target, + "prepared_at": datetime.now(context.now.tzinfo).isoformat(), + "text_sha256": context.text_sha256, + "image_sha256": context.image_sha256, + "image_enabled": context.image_enabled, + "result": "pending", + } + persisted, latest = self.store.update_send_claim( + context.group_name, + context.run_date, + context.claim_id, + delivery_evidence=evidence, + ) + if not persisted: + return StageResult.stop( + { + "group_name": context.group_name, + "status": "skipped", + "error_type": "SEND_CLAIM_LOST", + "detail": "送达证据落盘前发送 claim 已失效，未调用发送器", + } + ) + context.run = latest + context.text_sent_at = str(context.run.get("text_sent_at") or "") + context.image_sent_at = str(context.run.get("image_sent_at") or "") + return StageResult.proceed(context) + + def _send_text( + self, + context: DeliveryContext, + ) -> StageResult[DeliveryContext]: + if context.text_sent_at: + level = context.run.get("text_verification_level") + if level: + context.verification_levels.append(str(level)) + return StageResult.proceed(context) + + started_at = datetime.now(context.now.tzinfo).isoformat() + updated, _ = self.store.update_send_claim( + context.group_name, + context.run_date, + context.claim_id, + send_state="sending_text", + text_attempt_started_at=started_at, + text_attempt_finished_at="", + text_submitted_at="", + text_verified_at="", + ) + if not updated: + return StageResult.stop( + { + "group_name": context.group_name, + "status": "skipped", + "detail": "发送 claim 已失效", + } + ) + try: + result = self.sender.send_text(context.target, context.ranking_text) + except Exception as exc: + return StageResult.stop( + self.finish_unknown( + context.group_name, + context.run_date, + context.claim_id, + "text", + f"文字发送异常：{exc}", + ) + ) + finished_at = datetime.now(context.now.tzinfo).isoformat() + if result.outcome_unknown: + return StageResult.stop( + self.finish_unknown( + context.group_name, + context.run_date, + context.claim_id, + "text", + result.detail, + submitted_at=finished_at if result.submitted else "", + diagnostics=result.diagnostics, + ) + ) + if not result.success: + if result.submitted: + return StageResult.stop( + self.finish_unknown( + context.group_name, + context.run_date, + context.claim_id, + "text", + result.detail or "文字已提交，但发送结果未确认", + submitted_at=finished_at, + diagnostics=result.diagnostics, + ) + ) + persisted, failed_run, final = self.store.finish_send_failure( + context.group_name, + context.run_date, + context.claim_id, + stage="text", + error_type="SEND_TEXT_FAILED", + detail=result.detail, + now=context.now, + diagnostics=result.diagnostics, + status=context.run.get("status", READY_TO_SEND), + text_attempt_finished_at=finished_at, + text_submitted_at="", + text_verification_diagnostics=result.diagnostics, + delivery_evidence=self._delivery_evidence( + context, + result="failed", + completed_at=finished_at, + detail=result.detail, + ), + ) + if not persisted: + return StageResult.stop( + self.finish_unknown( + context.group_name, + context.run_date, + context.claim_id, + "text", + f"文字发送失败状态无法持久化：{result.detail}", + diagnostics=result.diagnostics, + ) + ) + return StageResult.stop( + { + "group_name": context.group_name, + "status": "failed_final" if final else "retry_scheduled", + "error_type": "SEND_TEXT_FAILED", + "detail": result.detail, + "next_retry_at": failed_run.get("send_next_retry_at"), + } + ) + + context.text_sent_at = result.sent_at or finished_at + level = result.verification_level or "provider_reported" + context.verification_levels.append(level) + persisted, _ = self.store.update_send_claim( + context.group_name, + context.run_date, + context.claim_id, + send_state="text_verified", + text_attempt_finished_at=finished_at, + text_submitted_at=( + finished_at if result.submitted or result.success else "" + ), + text_verified_at=finished_at, + text_sent_at=context.text_sent_at, + text_verification_level=level, + text_verification_diagnostics=result.diagnostics, + send_error="", + send_error_type="", + ) + if not persisted: + return StageResult.stop( + self.finish_unknown( + context.group_name, + context.run_date, + context.claim_id, + "text", + "文字发送已成功，但成功状态无法持久化", + submitted_at=finished_at, + diagnostics=result.diagnostics, + ) + ) + return StageResult.proceed(context) + + def _send_image( + self, + context: DeliveryContext, + ) -> StageResult[DeliveryContext]: + if not context.image_enabled: + return StageResult.proceed(context) + + started_at = datetime.now(context.now.tzinfo).isoformat() + updated, _ = self.store.update_send_claim( + context.group_name, + context.run_date, + context.claim_id, + send_state="sending_image", + image_attempt_started_at=started_at, + image_attempt_finished_at="", + image_submitted_at="", + image_verified_at="", + ) + if not updated: + return StageResult.stop( + { + "group_name": context.group_name, + "status": "skipped", + "error_type": "SEND_CLAIM_LOST", + "detail": "图片提交前发送 claim 已失效，未调用发送器", + } + ) + try: + result = self.sender.send_image(context.target, context.image_path) + except Exception as exc: + return StageResult.stop( + self.finish_unknown( + context.group_name, + context.run_date, + context.claim_id, + "image", + f"图片发送异常：{exc}", + ) + ) + finished_at = datetime.now(context.now.tzinfo).isoformat() + if result.outcome_unknown: + return StageResult.stop( + self.finish_unknown( + context.group_name, + context.run_date, + context.claim_id, + "image", + result.detail, + submitted_at=finished_at if result.submitted else "", + diagnostics=result.diagnostics, + ) + ) + if not result.success: + if result.submitted: + return StageResult.stop( + self.finish_unknown( + context.group_name, + context.run_date, + context.claim_id, + "image", + result.detail or "图片已提交，但发送结果未确认", + submitted_at=finished_at, + diagnostics=result.diagnostics, + ) + ) + persisted, failed_run, final = self.store.finish_send_failure( + context.group_name, + context.run_date, + context.claim_id, + stage="image", + error_type="SEND_IMAGE_FAILED", + detail=result.detail, + now=context.now, + diagnostics=result.diagnostics, + status=context.run.get("status", READY_TO_SEND), + text_sent_at=context.text_sent_at, + image_attempt_finished_at=finished_at, + image_submitted_at="", + image_verification_diagnostics=result.diagnostics, + delivery_evidence=self._delivery_evidence( + context, + result="failed", + completed_at=finished_at, + detail=result.detail, + ), + ) + if not persisted: + return StageResult.stop( + self.finish_unknown( + context.group_name, + context.run_date, + context.claim_id, + "image", + f"图片发送失败状态无法持久化：{result.detail}", + diagnostics=result.diagnostics, + ) + ) + return StageResult.stop( + { + "group_name": context.group_name, + "status": "failed_final" if final else "retry_scheduled", + "error_type": "SEND_IMAGE_FAILED", + "detail": result.detail, + "next_retry_at": failed_run.get("send_next_retry_at"), + } + ) + + context.image_sent_at = result.sent_at or finished_at + level = result.verification_level or "provider_reported" + context.verification_levels.append(level) + persisted, _ = self.store.update_send_claim( + context.group_name, + context.run_date, + context.claim_id, + send_state="image_verified", + image_attempt_finished_at=finished_at, + image_submitted_at=( + finished_at if result.submitted or result.success else "" + ), + image_verified_at=finished_at, + image_sent_at=context.image_sent_at, + image_verification_level=level, + image_verification_diagnostics=result.diagnostics, + send_error="", + send_error_type="", + send_next_retry_at="", + ) + if not persisted: + return StageResult.stop( + self.finish_unknown( + context.group_name, + context.run_date, + context.claim_id, + "image", + "图片发送已成功，但成功状态无法持久化", + submitted_at=finished_at, + diagnostics=result.diagnostics, + ) + ) + return StageResult.proceed(context) + + def _complete(self, context: DeliveryContext) -> dict: + levels = context.verification_levels + if levels and all(level == "ui_observed" for level in levels): + verification_level = "ui_observed" + elif levels and all(level == "dry_run" for level in levels): + verification_level = "dry_run" + elif "manual_ui_observed" in levels: + verification_level = "manual_ui_observed" + else: + verification_level = "provider_reported" + persisted, run = self.store.finish_send_claim( + context.group_name, + context.run_date, + context.claim_id, + send_state="sent", + status=SENT, + sent_at=context.now.isoformat(), + sent_target=context.target, + text_sent_at=context.text_sent_at, + image_sent_at=context.image_sent_at, + send_error="", + send_error_type="", + verification_level=verification_level, + delivery_evidence=self._delivery_evidence( + context, + result="sent", + completed_at=context.now.isoformat(), + verification_level=verification_level, + ), + send_hold=False, + send_hold_reason="", + needs_manual_send=False, + image_regen_status=( + "sent" + if context.run.get("image_regen_status") == "ready_for_review" + else context.run.get("image_regen_status") + ), + ) + if not persisted: + if run.get("status") == SENT or run.get("sent_at"): + persisted = True + else: + return self.finish_unknown( + context.group_name, + context.run_date, + context.claim_id, + "finalize", + "微信发送已完成，但 SENT 终态无法持久化", + ) + if context.image_enabled: + self.logger.info("群 %s 已发送（文字+图片）→ SENT", context.group_name) + detail = "文字和图片已发送" + else: + self.logger.info("群 %s 已发送（仅文字，未启用图片）→ SENT", context.group_name) + detail = "文字已发送（未启用图片）" + latest = self.store.load_run(context.group_name, context.run_date) + log_event( + self.logger, + "WECHAT_SEND_FINISHED", + group_task_id=latest.get("group_task_id"), + group_name=context.group_name, + run_date=context.run_date, + stage="SEND", + status="sent", + attempt=latest.get("retry_attempt_count", 0), + ) + return { + "group_name": context.group_name, + "status": "sent", + "detail": detail, + "sent_at": context.now.isoformat(), + } + + @staticmethod + def _delivery_evidence( + context: DeliveryContext, + *, + result: str, + completed_at: str, + detail: str = "", + verification_level: str = "", + ) -> dict: + evidence = dict( + context.run.get("delivery_evidence") + if isinstance(context.run.get("delivery_evidence"), dict) + else {} + ) + evidence.update( + target=context.target, + completed_at=completed_at, + text_sha256=context.text_sha256, + image_sha256=context.image_sha256, + image_enabled=context.image_enabled, + result=result, + detail=str(detail or "")[:300], + verification_level=verification_level, + ) + return evidence + + def finish_unknown( + self, + group_name: str, + run_date: str, + claim_id: str, + stage: str, + detail: str, + *, + submitted_at: str = "", + diagnostics: dict[str, object] | None = None, + ) -> dict: + finished_at = datetime.now( + ZoneInfo(self.settings.app_timezone) + ).isoformat() + current = self.store.load_run(group_name, run_date) + evidence = dict( + current.get("delivery_evidence") + if isinstance(current.get("delivery_evidence"), dict) + else {} + ) + evidence.update( + completed_at=finished_at, + result="unknown", + detail=str(detail or "")[:300], + verification_level="unknown", + ) + fields = { + f"{stage}_attempt_finished_at": "", + f"{stage}_submitted_at": submitted_at, + f"{stage}_verification_diagnostics": diagnostics or {}, + "delivery_evidence": evidence, + } + persisted, run = self.store.finish_send_claim( + group_name, + run_date, + claim_id, + send_state="unknown", + send_hold=True, + send_hold_reason="SEND_RESULT_UNKNOWN", + needs_manual_send=True, + send_error=detail, + send_error_type="SEND_RESULT_UNKNOWN", + verification_level="unknown", + send_unknown_at=finished_at, + send_unknown_stage=stage, + **fields, + ) + if not persisted: + persisted, run, reason = self.store.mark_send_result_unknown( + group_name, + run_date, + claim_id, + stage=stage, + detail=detail, + submitted_at=submitted_at, + diagnostics=diagnostics, + now=datetime.now(ZoneInfo(self.settings.app_timezone)), + ) + if not persisted: + self.logger.error( + "发送结果未知且状态无法持久化 group=%s date=%s stage=%s reason=%s", + group_name, + run_date, + stage, + reason, + ) + detail = f"{detail}；且 unknown 状态持久化失败（{reason}）" + log_event( + self.logger, + "WECHAT_SEND_UNKNOWN", + group_name=group_name, + run_date=run_date, + stage=stage, + status="held", + error_type="SEND_RESULT_UNKNOWN", + error_summary=detail, + ) + return { + "group_name": group_name, + "status": "held", + "error_type": "SEND_RESULT_UNKNOWN", + "detail": detail, + } diff --git a/app/pipeline/generation_stages.py b/app/pipeline/generation_stages.py new file mode 100644 index 0000000..5242855 --- /dev/null +++ b/app/pipeline/generation_stages.py @@ -0,0 +1,984 @@ +"""DailyPipeline 的单群生成阶段实现。""" + +from __future__ import annotations + +from dataclasses import dataclass, field +from datetime import datetime +from pathlib import Path +from time import perf_counter +from typing import Any, Callable + +from app.ai.concurrency import bounded_slot +from app.ai.prompt_builder import GroupSummaryImagePromptBuilder +from app.ai.prompt_builder_types import PromptInput +from app.ai.speaker_attribution import build_attribution_contract +from app.ai.strict_prompt_contract import append_strict_image_fact_contract +from app.config.settings import Settings +from app.core.observability import log_event +from app.data_sources.base import V2Message, WeChatDataSource +from app.db.models import Group +from app.pipeline.stage_result import StageResult +from app.providers.ai.base import ExternalCallResultUnknownError +from app.ranking.engine import RankingEngine +from app.ranking.engine_types import RankingResult +from app.ranking.renderer import RankingRenderer +from app.ranking.policies import uses_strict_image_fact_contract +from app.services.sender_name_policy import apply_sender_name_policy +from app.scheduler.period import PeriodWindow +from app.services.group_name_sync import effective_send_target, send_target_mode +from app.v2.constants import ( + DATA_READY, + FAILED, + IMAGE_READY, + MESSAGE_FETCH_FAILED, + MESSAGE_SNAPSHOT_INVALID, + PENDING, + PROMPT_FAILED, + PROMPT_READY, + RANKING_FAILED, + RANKING_READY, + READY_TO_SEND, + RUN_STATE_CORRUPT, + SENT, + WECHAT_DATA_UNAVAILABLE, + EXECUTION_FAILED_FINAL, + EXECUTION_HOLD_MANUAL, + EXECUTION_WAIT_RETRY, +) +from app.v2.run_store import RunStore +from app.v2.reliability import retry_is_due + + +@dataclass +class GenerationContext: + group: Group + window: PeriodWindow + run_date: str + force: bool + refresh_messages: bool + reuse_persisted_topic_selection: bool + group_name: str + run: dict + persisted_prompt_meta: dict + period_start: str + period_end: str + started_at: float = field(default_factory=perf_counter) + timings: dict[str, int] = field(default_factory=dict) + + +@dataclass(frozen=True) +class PromptStageOutput: + prompt_meta: dict | None + + +class GenerationStages: + """按快照、排行、Prompt、图片决策顺序执行单群生成。""" + + def __init__( + self, + *, + settings: Settings, + data_source: WeChatDataSource, + ranking_engine: RankingEngine, + renderer: RankingRenderer, + prompt_builder: GroupSummaryImagePromptBuilder, + store: RunStore, + group_name: Callable[[Group], str], + name_sync_audit: Callable[[Group], dict], + get_group: Callable[[int], Group | None], + prompt_operation_hash: Callable[[PromptInput], str], + save_json: Callable[[Path, Any], None], + load_message_snapshot: Callable[[Path], list[V2Message]], + logger, + ) -> None: + self.settings = settings + self.data_source = data_source + self.ranking_engine = ranking_engine + self.renderer = renderer + self.prompt_builder = prompt_builder + self.store = store + self._group_name = group_name + self._name_sync_audit = name_sync_audit + self._get_group = get_group + self._prompt_operation_hash = prompt_operation_hash + self._save_json = save_json + self._load_message_snapshot = load_message_snapshot + self.logger = logger + + def run( + self, + group: Group, + window: PeriodWindow, + run_date: str, + force: bool, + *, + refresh_messages: bool = False, + reuse_persisted_topic_selection: bool = False, + ) -> dict: + context = self._context( + group, + window, + run_date, + force, + refresh_messages, + reuse_persisted_topic_selection, + ) + + prepared = self._prepare_run(context) + if prepared.is_terminal: + return self._finish(context, prepared.terminal_response()) + + message_stage = self._load_or_fetch_messages(context) + if message_stage.is_terminal: + return self._finish(context, message_stage.terminal_response()) + messages = message_stage.next_value() + + if context.refresh_messages: + refreshed = self._refresh_snapshot_and_ranking(context, messages) + return self._finish(context, refreshed.terminal_response()) + + ranking_stage = self._build_ranking(context, messages) + if ranking_stage.is_terminal: + return self._finish(context, ranking_stage.terminal_response()) + + prompt_stage = self._build_prompt( + context, + messages, + ranking_stage.next_value(), + ) + if prompt_stage.is_terminal: + return self._finish(context, prompt_stage.terminal_response()) + prompt_meta = prompt_stage.next_value().prompt_meta + if isinstance(prompt_meta, dict): + prompt_meta = dict(prompt_meta) + attribution = build_attribution_contract(messages) + prompt_meta.setdefault( + "message_snapshot_sha256", + attribution.message_snapshot_sha256, + ) + prompt_meta.setdefault( + "speaker_fingerprint", + attribution.speaker_fingerprint, + ) + prompt_meta.setdefault("speaker_bindings", []) + self.store.update( + context.group_name, + context.run_date, + prompt_meta=prompt_meta, + summary_provider_actual=str( + prompt_meta.get("summary_provider_actual") or "" + ), + summary_model_actual=str(prompt_meta.get("summary_model_actual") or ""), + summary_fallback_reason=str( + prompt_meta.get("summary_fallback_reason") or "" + ), + prompt_provider_actual=str( + prompt_meta.get("prompt_provider_actual") + or prompt_meta.get("actual_provider") + or "" + ), + prompt_model_actual=str( + prompt_meta.get("prompt_model_actual") + or prompt_meta.get("actual_model") + or "" + ), + prompt_fallback_reason=str( + prompt_meta.get("prompt_fallback_reason") + or prompt_meta.get("fallback_reason") + or "" + ), + external_call_count=int(prompt_meta.get("api_call_count") or 0), + message_snapshot_sha256=str( + prompt_meta.get("message_snapshot_sha256") or "" + ), + speaker_fingerprint=str( + prompt_meta.get("speaker_fingerprint") or "" + ), + speaker_bindings=prompt_meta.get("speaker_bindings") or [], + prompt_stale=False, + image_stale=bool(context.group.image_enabled), + artifact_stale_reason=( + "PROMPT_CHANGED_IMAGE_REBUILD_REQUIRED" + if bool(context.group.image_enabled) + else "" + ), + ) + image_stage = self._decide_image(context) + return self._finish(context, image_stage.terminal_response()) + + def _context( + self, + group: Group, + window: PeriodWindow, + run_date: str, + force: bool, + refresh_messages: bool, + reuse_persisted_topic_selection: bool, + ) -> GenerationContext: + started_at = perf_counter() + group_name = self._group_name(group) + run = self.store.load_run(group_name, run_date) + prompt_meta = run.get("prompt_meta") + persisted_prompt_meta = prompt_meta if isinstance(prompt_meta, dict) else {} + return GenerationContext( + group=group, + window=window, + run_date=run_date, + force=force, + refresh_messages=refresh_messages, + reuse_persisted_topic_selection=reuse_persisted_topic_selection, + group_name=group_name, + run=run, + persisted_prompt_meta=persisted_prompt_meta, + period_start=window.period_start_str(), + period_end=window.period_end_str(), + started_at=started_at, + ) + + def _prepare_run( + self, + context: GenerationContext, + ) -> StageResult[GenerationContext]: + run = context.run + if ( + not context.force + and not context.refresh_messages + and run.get("status") in (IMAGE_READY, READY_TO_SEND, SENT) + ): + self.logger.info( + "群 %s %s 已到 %s，跳过生成", + context.group_name, + context.run_date, + run.get("status"), + ) + return StageResult.stop( + { + "group_name": context.group_name, + "status": "skipped", + "detail": f"已{run.get('status')}", + } + ) + + execution_state = str(run.get("execution_state") or "") + if run.get("status") == FAILED and execution_state == EXECUTION_HOLD_MANUAL: + return StageResult.stop( + { + "group_name": context.group_name, + "status": "held", + "error_type": str(run.get("error_type") or "HOLD_MANUAL"), + "detail": str(run.get("error") or "任务需人工核对")[:300], + } + ) + if ( + not context.force + and run.get("status") == FAILED + and execution_state == EXECUTION_FAILED_FINAL + ): + return StageResult.stop( + { + "group_name": context.group_name, + "status": "failed_final", + "error_type": str(run.get("error_type") or "FAILED_FINAL"), + "detail": str(run.get("error") or "重试预算已耗尽")[:300], + } + ) + if ( + not context.force + and run.get("status") == FAILED + and execution_state == EXECUTION_WAIT_RETRY + and not retry_is_due(run) + ): + return StageResult.stop( + { + "group_name": context.group_name, + "status": "retry_scheduled", + "error_type": str(run.get("error_type") or "RETRY_SCHEDULED"), + "detail": f"等待自动重试：{run.get('next_retry_at') or '未指定时间'}", + } + ) + retrying = bool( + run.get("status") == FAILED + and ( + ( + execution_state == EXECUTION_WAIT_RETRY + and (context.force or retry_is_due(run)) + ) + or (context.force and execution_state == EXECUTION_FAILED_FINAL) + ) + ) + group = context.group + base = { + "group_id": str(group.id), + "wechat_group_id": group.wechat_group_id, + "wechat_group_name": group.wechat_group_name, + "effective_send_target": effective_send_target(group), + "send_target_mode": send_target_mode(group), + **self._name_sync_audit(group), + "period_start": context.period_start, + "period_end": context.period_end, + "send_time": self.settings.schedule_send_time, + "image_enabled": bool(group.image_enabled), + "ranking_template": group.ranking_template, + "ranking_count_policy": getattr( + group, "ranking_count_policy", "all_messages" + ), + "sender_name_policy": getattr(group, "sender_name_policy", "resolved"), + "image_prompt_template": group.image_prompt_template, + "image_theme": group.image_theme, + "image_theme_custom": group.image_theme_custom, + "wechat_send_enabled": bool(getattr(group, "wechat_send_enabled", False)), + "provider": self.data_source.name, + "failed_stage": None, + "error": None, + "error_type": None, + } + if retrying: + # 清除“上一轮失败快照”的去重指纹。这样同一错误在同一次落盘中 + # 不会重复计数，但真正重新执行后再次失败会消耗下一次预算。 + base.update( + failure_fingerprint="", + retry_started_at=datetime.now().astimezone().isoformat(), + retry_stage=str(run.get("failed_stage") or "unknown"), + ) + self.store.update(context.group_name, context.run_date, status=PENDING, **base) + return StageResult.proceed(context) + + def _load_or_fetch_messages( + self, + context: GenerationContext, + ) -> StageResult[list[V2Message]]: + group = context.group + if not group.wechat_group_id: + self.store.update( + context.group_name, + context.run_date, + status=FAILED, + failed_stage="data", + error="群未绑定微信群 ID", + ) + return StageResult.stop( + { + "group_name": context.group_name, + "status": "failed", + "error_type": WECHAT_DATA_UNAVAILABLE, + } + ) + + started_at = perf_counter() + snapshot_path = self.store.messages_path( + context.group_name, + context.run_date, + ) + if snapshot_path.is_file() and not context.refresh_messages: + return self._reuse_snapshot(context, snapshot_path, started_at) + return self._fetch_messages(context, snapshot_path, started_at) + + def _reuse_snapshot( + self, + context: GenerationContext, + snapshot_path: Path, + started_at: float, + ) -> StageResult[list[V2Message]]: + try: + messages = self._load_message_snapshot(snapshot_path) + except (OSError, UnicodeError, ValueError) as exc: + context.timings["fetch_ms"] = round((perf_counter() - started_at) * 1000) + detail = f"当天消息快照无法读取，已停止且不会隐式重抓：{str(exc)[:220]}" + self.store.update( + context.group_name, + context.run_date, + status=FAILED, + failed_stage="data", + error=detail, + error_type=MESSAGE_SNAPSHOT_INVALID, + message_snapshot_reused=False, + ) + return StageResult.stop( + { + "group_name": context.group_name, + "status": "failed", + "error_type": MESSAGE_SNAPSHOT_INVALID, + "detail": detail, + } + ) + + apply_sender_name_policy( + messages, + getattr(context.group, "sender_name_policy", "resolved"), + ) + context.timings["fetch_ms"] = round((perf_counter() - started_at) * 1000) + self.store.update( + context.group_name, + context.run_date, + status=DATA_READY, + message_count=len(messages), + message_snapshot_reused=True, + message_snapshot_refreshed=False, + message_snapshot_path=snapshot_path.name, + ) + self.logger.info( + "群 %s 复用当天消息快照：%s（%d 条）", + context.group_name, + snapshot_path, + len(messages), + ) + return StageResult.proceed(messages) + + def _fetch_messages( + self, + context: GenerationContext, + snapshot_path: Path, + started_at: float, + ) -> StageResult[list[V2Message]]: + group = context.group + try: + with bounded_slot("wechat_fetch", self.settings.wechat_fetch_concurrency): + fetch = self.data_source.fetch_messages( + group.wechat_group_id, + context.window.period_start, + context.window.period_end, + ) + except Exception as exc: + context.timings["fetch_ms"] = round((perf_counter() - started_at) * 1000) + self.store.update( + context.group_name, + context.run_date, + status=FAILED, + failed_stage="data", + error=str(exc)[:300], + error_type=MESSAGE_FETCH_FAILED, + ) + return StageResult.stop( + { + "group_name": context.group_name, + "status": "failed", + "error_type": MESSAGE_FETCH_FAILED, + "detail": str(exc)[:300], + } + ) + + context.timings["fetch_ms"] = round((perf_counter() - started_at) * 1000) + fetch_metrics = fetch.meta if isinstance(getattr(fetch, "meta", None), dict) else {} + if fetch_metrics: + self.store.update( + context.group_name, + context.run_date, + fetch_metrics=fetch_metrics, + ) + if fetch.status.value != "OK" or not fetch.messages: + error_type = fetch.error_type or MESSAGE_FETCH_FAILED + self.store.update( + context.group_name, + context.run_date, + status=FAILED, + failed_stage="data", + error=fetch.detail or fetch.status.value, + error_type=error_type, + ) + return StageResult.stop( + { + "group_name": context.group_name, + "status": "failed", + "error_type": error_type, + "detail": fetch.detail, + } + ) + + messages = list(fetch.messages) + apply_sender_name_policy( + messages, + getattr(context.group, "sender_name_policy", "resolved"), + ) + if not context.refresh_messages: + attribution = build_attribution_contract(messages) + self._save_json( + snapshot_path, + [message.to_dict() for message in messages], + ) + self.store.update( + context.group_name, + context.run_date, + status=DATA_READY, + message_count=len(messages), + message_snapshot_reused=False, + message_snapshot_refreshed=False, + message_snapshot_saved_at=datetime.now().astimezone().isoformat(), + message_snapshot_path=snapshot_path.name, + message_snapshot_period_start=context.period_start, + message_snapshot_period_end=context.period_end, + message_snapshot_sha256=attribution.message_snapshot_sha256, + speaker_fingerprint=attribution.speaker_fingerprint, + ) + return StageResult.proceed(messages) + + def _refresh_snapshot_and_ranking( + self, + context: GenerationContext, + messages: list[V2Message], + ) -> StageResult[dict]: + started_at = perf_counter() + try: + ranking = self.ranking_engine.compute( + messages, + context.group_name, + context.period_start, + context.period_end, + top_limit=10, + count_policy=getattr( + context.group, "ranking_count_policy", "all_messages" + ), + name_source=getattr(context.group, "sender_name_policy", "resolved"), + ) + ranking_txt = self.renderer.render( + ranking, + template_name=context.group.ranking_template, + ) + except Exception as exc: + context.timings["ranking_ms"] = round((perf_counter() - started_at) * 1000) + self.store.update( + context.group_name, + context.run_date, + status=context.run.get("status") or PENDING, + failed_stage=context.run.get("failed_stage"), + error=context.run.get("error"), + message_refresh_status="failed", + message_refresh_error=str(exc)[:300], + ) + return StageResult.stop( + { + "group_name": context.group_name, + "status": "failed", + "error_type": RANKING_FAILED, + "detail": "新消息已读取，但排行榜计算失败，旧快照未被替换", + } + ) + + context.timings["ranking_ms"] = round((perf_counter() - started_at) * 1000) + attribution = build_attribution_contract(messages) + snapshot_path = self.store.messages_path(context.group_name, context.run_date) + self._save_json(snapshot_path, [message.to_dict() for message in messages]) + self._save_json( + self.store.ranking_json_path(context.group_name, context.run_date), + ranking.to_dict(), + ) + self.store.ranking_txt_path(context.group_name, context.run_date).write_text( + ranking_txt, + encoding="utf-8", + ) + next_status = SENT if context.run.get("status") == SENT else RANKING_READY + prior_hold_reason = str(context.run.get("send_hold_reason") or "") + hold_reason = ( + prior_hold_reason + if prior_hold_reason.startswith("USER_REQUEST_") + else "MESSAGE_SNAPSHOT_REFRESHED" + ) + self.store.update( + context.group_name, + context.run_date, + status=next_status, + failed_stage=None, + error=None, + speaker_count=ranking.speaker_count, + message_count=ranking.message_count, + text_message_count=ranking.text_message_count, + interaction_message_count=ranking.interaction_message_count, + text_speaker_count=ranking.text_speaker_count, + message_snapshot_reused=False, + message_snapshot_refreshed=True, + message_snapshot_saved_at=datetime.now().astimezone().isoformat(), + message_snapshot_path=snapshot_path.name, + message_snapshot_period_start=context.period_start, + message_snapshot_period_end=context.period_end, + message_refresh_status="completed", + message_refresh_error="", + prompt_rebuild_status="required", + prompt_rebuild_error="", + message_snapshot_sha256=attribution.message_snapshot_sha256, + speaker_fingerprint=attribution.speaker_fingerprint, + prompt_stale=True, + image_stale=True, + artifact_stale_reason="MESSAGE_SNAPSHOT_REFRESHED", + send_hold=True, + send_hold_reason=hold_reason, + needs_manual_send=True, + ) + return StageResult.stop( + { + "group_name": context.group_name, + "status": "data_ready", + "detail": "当天消息快照和排行榜已更新；未重建 Prompt，未生图", + } + ) + + def _build_ranking( + self, + context: GenerationContext, + messages: list[V2Message], + ) -> StageResult[RankingResult]: + started_at = perf_counter() + try: + ranking = self.ranking_engine.compute( + messages, + context.group_name, + context.period_start, + context.period_end, + top_limit=10, + count_policy=getattr( + context.group, "ranking_count_policy", "all_messages" + ), + name_source=getattr(context.group, "sender_name_policy", "resolved"), + ) + except Exception as exc: + self.store.update( + context.group_name, + context.run_date, + status=FAILED, + failed_stage="ranking", + error=str(exc)[:300], + ) + context.timings["ranking_ms"] = round((perf_counter() - started_at) * 1000) + return StageResult.stop( + { + "group_name": context.group_name, + "status": "failed", + "error_type": RANKING_FAILED, + } + ) + + context.timings["ranking_ms"] = round((perf_counter() - started_at) * 1000) + self._save_json( + self.store.ranking_json_path(context.group_name, context.run_date), + ranking.to_dict(), + ) + ranking_txt = self.renderer.render( + ranking, + template_name=context.group.ranking_template, + ) + self.store.ranking_txt_path(context.group_name, context.run_date).write_text( + ranking_txt, + encoding="utf-8", + ) + self.store.update( + context.group_name, + context.run_date, + status=RANKING_READY, + speaker_count=ranking.speaker_count, + message_count=ranking.message_count, + text_message_count=ranking.text_message_count, + interaction_message_count=ranking.interaction_message_count, + text_speaker_count=ranking.text_speaker_count, + ) + return StageResult.proceed(ranking) + + def _build_prompt( + self, + context: GenerationContext, + messages: list[V2Message], + ranking: RankingResult, + ) -> StageResult[PromptStageOutput]: + group = context.group + attribution = build_attribution_contract(messages) + prompt_messages = [ + message for message in messages if RankingEngine._countable(message) + ] + prompt_input = PromptInput( + group_name=context.group_name, + visible_group_name=str( + context.run.get("wechat_group_name") + or group.wechat_group_name + or context.group_name + ).strip(), + group_id=str(group.id or group.wechat_group_id or context.group_name), + run_date=context.run_date, + message_snapshot_sha256=attribution.message_snapshot_sha256, + speaker_fingerprint=attribution.speaker_fingerprint, + period_start=context.period_start, + period_end=context.period_end, + report_date=context.window.period_end.date().isoformat(), + message_count=ranking.message_count, + speaker_count=ranking.speaker_count, + messages=prompt_messages, + template=group.image_prompt_template, + image_theme=group.image_theme, + image_theme_custom=group.image_theme_custom, + template_override=getattr(group, "image_prompt_override", "") or "", + previous_theme_signature=self.store.previous_theme_signature( + context.group_name, + context.run_date, + ), + persisted_theme_meta=context.persisted_prompt_meta or None, + persisted_topic_selection=( + context.persisted_prompt_meta.get("topic_selection") + if context.reuse_persisted_topic_selection + and isinstance(context.persisted_prompt_meta.get("topic_selection"), dict) + else None + ), + recent_layout_history=self.store.recent_layout_history( + context.group_name, + context.run_date, + limit=3, + ), + ) + return self._execute_prompt_operation(context, prompt_input) + + def _execute_prompt_operation( + self, + context: GenerationContext, + prompt_input: PromptInput, + ) -> StageResult[PromptStageOutput]: + input_hash = self._prompt_operation_hash(prompt_input) + operation_id, operation_state, claim_reason = self.store.claim_prompt_operation( + context.group_name, + context.run_date, + input_hash=input_hash, + force=context.force, + ) + if claim_reason == "state_corrupt": + return StageResult.stop( + { + "group_name": context.group_name, + "status": "blocked", + "error_type": RUN_STATE_CORRUPT, + "detail": "运行状态文件损坏，已阻止 AI 调用", + } + ) + if claim_reason == "result_unknown": + self.store.update( + context.group_name, + context.run_date, + status=FAILED, + failed_stage="prompt", + error="上次 AI 调用结果未知，需人工核对后才能再次生成", + error_type="PROMPT_RESULT_UNKNOWN", + ) + return StageResult.stop( + { + "group_name": context.group_name, + "status": "held", + "error_type": "PROMPT_RESULT_UNKNOWN", + "detail": "上次 AI 调用结果未知，需人工核对后才能再次生成", + } + ) + + started_at = perf_counter() + if claim_reason == "result_recorded": + operation_id = str(operation_state.get("prompt_operation_id") or "") + committed = self.store.commit_recorded_prompt( + context.group_name, + context.run_date, + operation_id, + ) + prompt_meta = committed.get("prompt_meta") + elif claim_reason == "already_completed": + prompt_meta = operation_state.get("prompt_meta") + self.store.update(context.group_name, context.run_date, status=PROMPT_READY) + else: + if not operation_id: + raise RuntimeError(f"无法领取 Prompt 操作：{claim_reason}") + try: + prompt_out = self.prompt_builder.build(prompt_input) + except ExternalCallResultUnknownError as exc: + self.store.mark_prompt_result_unknown( + context.group_name, + context.run_date, + operation_id, + error=str(exc), + ) + self._record_prompt_timing(context, started_at) + self.store.update( + context.group_name, + context.run_date, + status=FAILED, + failed_stage="prompt", + error=str(exc)[:300], + error_type="PROMPT_RESULT_UNKNOWN", + ) + return StageResult.stop( + { + "group_name": context.group_name, + "status": "held", + "error_type": "PROMPT_RESULT_UNKNOWN", + "detail": str(exc)[:300], + } + ) + if not prompt_out.success: + self.store.fail_prompt_operation( + context.group_name, + context.run_date, + operation_id, + error=prompt_out.error, + ) + self._record_prompt_timing(context, started_at) + if bool(context.group.image_enabled) and not context.reuse_persisted_topic_selection: + fallback_meta = dict(prompt_out.meta or {}) + fallback_meta.update( + mode="local_infographic", + fallback_level=3, + fallback_reason=PROMPT_FAILED, + ) + fallback_prompt = ( + "【任务】\n" + "外部内容整理失败，仅使用当天 ranking.json 生成本地简化信息图。\n" + "【画布】\n优先 1024×1536；其他完整可读的竖版尺寸也可接受\n" + ) + self.store.prompt_path(context.group_name, context.run_date).write_text( + fallback_prompt, + encoding="utf-8", + ) + self.store.update( + context.group_name, + context.run_date, + status=PROMPT_READY, + failed_stage=None, + error=None, + error_type=None, + prompt_fallback_level=3, + image_force_local_fallback=True, + prompt_fallback_reason=PROMPT_FAILED, + prompt_original_error=str(prompt_out.error)[:300], + ) + return StageResult.proceed( + PromptStageOutput( + prompt_meta=fallback_meta, + ) + ) + self.store.update( + context.group_name, + context.run_date, + status=FAILED, + failed_stage="prompt", + error=prompt_out.error, + error_type=PROMPT_FAILED, + ) + return StageResult.stop( + { + "group_name": context.group_name, + "status": "failed", + "error_type": PROMPT_FAILED, + "detail": prompt_out.error, + } + ) + self.store.record_prompt_result( + context.group_name, + context.run_date, + operation_id, + prompt=prompt_out.prompt, + meta=prompt_out.meta, + ) + committed = self.store.commit_recorded_prompt( + context.group_name, + context.run_date, + operation_id, + ) + prompt_meta = committed.get("prompt_meta") + + self._record_prompt_timing(context, started_at) + if uses_strict_image_fact_contract( + getattr(context.group, "ranking_count_policy", "all_messages") + ): + prompt_path = self.store.prompt_path(context.group_name, context.run_date) + strict_prompt = append_strict_image_fact_contract( + prompt_path.read_text(encoding="utf-8") + ) + prompt_path.write_text(strict_prompt, encoding="utf-8") + self.store.update( + context.group_name, + context.run_date, + image_fact_contract="strict_evidence_v1", + ) + return StageResult.proceed( + PromptStageOutput( + prompt_meta=prompt_meta if isinstance(prompt_meta, dict) else None + ) + ) + + def _decide_image( + self, + context: GenerationContext, + ) -> StageResult[dict]: + group = context.group + current_group = group + if group.id is not None: + try: + refreshed_group = self._get_group(group.id) + except Exception as exc: + self.logger.warning( + "群 %s 生图开关刷新失败，沿用本次读取的配置：%s", + context.group_name, + exc, + ) + else: + if refreshed_group is not None: + current_group = refreshed_group + + current_image_enabled = bool(current_group.image_enabled) + group.image_enabled = current_image_enabled + self.store.update( + context.group_name, + context.run_date, + image_enabled=current_image_enabled, + ) + if not current_image_enabled: + self.store.update(context.group_name, context.run_date, status=READY_TO_SEND) + return StageResult.stop( + { + "group_name": context.group_name, + "status": "ready_to_send", + "detail": "未启用生图", + } + ) + return StageResult.stop( + { + "group_name": context.group_name, + "status": "prompt_ready", + "need_image": True, + } + ) + + @staticmethod + def _record_prompt_timing(context: GenerationContext, started_at: float) -> None: + context.timings["summary_ms"] = round((perf_counter() - started_at) * 1000) + context.timings["deepseek_ms"] = context.timings["summary_ms"] + + def _finish(self, context: GenerationContext, result: dict) -> dict: + timings = context.timings + timings["total_ms"] = round((perf_counter() - context.started_at) * 1000) + current = self.store.load_run(context.group_name, context.run_date) + prompt_meta = current.get("prompt_meta") + meta = prompt_meta if isinstance(prompt_meta, dict) else {} + timings["summary_calls"] = int(meta.get("api_call_count") or 0) + timings["deepseek_calls"] = timings["summary_calls"] + timings["chunk_count"] = int(meta.get("chunk_count") or 0) + self.store.update( + context.group_name, + context.run_date, + stage_timings=timings, + ) + self.logger.info( + "群生成耗时 group=%s fetch_ms=%d ranking_ms=%d summary_ms=%d " + "summary_calls=%d chunks=%d total_ms=%d status=%s", + context.group_name, + timings.get("fetch_ms", 0), + timings.get("ranking_ms", 0), + timings.get("summary_ms", timings.get("deepseek_ms", 0)), + timings["summary_calls"], + timings["chunk_count"], + timings["total_ms"], + result.get("status", ""), + ) + current = self.store.load_run(context.group_name, context.run_date) + log_event( + self.logger, + "GROUP_GENERATION_FINISHED", + group_task_id=current.get("group_task_id"), + group_name=context.group_name, + run_date=context.run_date, + stage=current.get("stage"), + status=result.get("status"), + duration_ms=timings["total_ms"], + attempt=current.get("retry_attempt_count", 0), + model=meta.get("api_model", ""), + error_type=result.get("error_type", ""), + error_summary=result.get("detail", ""), + ) + return result diff --git a/app/pipeline/image_stages.py b/app/pipeline/image_stages.py new file mode 100644 index 0000000..b9f7856 --- /dev/null +++ b/app/pipeline/image_stages.py @@ -0,0 +1,263 @@ +"""DailyPipeline 的受控并发图片任务阶段实现。""" + +from __future__ import annotations + +from datetime import datetime +import hashlib +from typing import Callable +import uuid + +from app.image.image_task import ImageJob, SerialImageQueue +from app.image.regeneration import normalize_candidate_diagnostics +from app.core.logging import get_logger +from app.core.observability import log_event +from app.v2.constants import ( + FAILED, + IMAGE_GENERATION_FAILED, + IMAGE_READY, + READY_TO_SEND, +) +from app.v2.run_store import RunStore + +logger = get_logger("groupbrief.pipeline") + + +class ImageStages: + """构造、记录并收口受控并发的图片任务。""" + + def __init__(self, *, store: RunStore, image_generator) -> None: + self.store = store + self.image_generator = image_generator + + def make_job(self, group_name: str, run_date: str, force: bool) -> ImageJob: + prompt_path = self.store.prompt_path(group_name, run_date) + current = self.store.load_run(group_name, run_date) + prompt_meta = ( + current.get("prompt_meta") + if isinstance(current.get("prompt_meta"), dict) + else {} + ) + snapshot_hash = str(current.get("message_snapshot_sha256") or "") + speaker_fingerprint = str(current.get("speaker_fingerprint") or "") + if ( + current.get("prompt_stale") is not False + or not snapshot_hash + or not speaker_fingerprint + or str(prompt_meta.get("message_snapshot_sha256") or "") != snapshot_hash + or str(prompt_meta.get("speaker_fingerprint") or "") + != speaker_fingerprint + ): + raise ValueError("Prompt 与消息快照归属契约不一致，已停止生图") + prompt_sha256 = hashlib.sha256(prompt_path.read_bytes()).hexdigest() + previous = current.get("image_job") if isinstance(current.get("image_job"), dict) else {} + if ( + not force + and previous.get("job_id") + and previous.get("prompt_sha256") == prompt_sha256 + ): + job_id = str(previous["job_id"]) + revision = max(1, int(previous.get("revision") or 1)) + else: + job_id = uuid.uuid4().hex + revision = max(1, int(previous.get("revision") or 0) + 1) + image_job = { + "job_id": job_id, + "revision": revision, + "prompt_sha256": prompt_sha256, + "status": "queued", + "queued_at": datetime.now().astimezone().isoformat(), + "candidates": [], + } + self.store.update(group_name, run_date, image_job=image_job) + return ImageJob( + group_name=group_name, + prompt_file=prompt_path, + output_path=self.store.image_path(group_name, run_date), + generator=self.image_generator, + force=force, + job_id=job_id, + revision=revision, + prompt_sha256=prompt_sha256, + ) + + def record_result(self, job: ImageJob, result: dict) -> None: + status = IMAGE_READY if result["success"] else FAILED + error_type = result.get("error_type") or IMAGE_GENERATION_FAILED + error_detail = ( + str(result.get("detail") or "图片生成失败")[:300] + if not result["success"] + else None + ) + run_date = job.output_path.parent.name + current = self.store.load_run(job.group_name, run_date) + stage_timings = dict(current.get("stage_timings") or {}) + imagegen_ms = int(result.get("imagegen_ms") or 0) + stage_timings["imagegen_ms"] = imagegen_ms + image_size_bytes = ( + job.output_path.stat().st_size + if result["success"] and job.output_path.is_file() + else 0 + ) + generator_detail = result.get("generator_detail") + if not isinstance(generator_detail, dict): + generator_detail = {} + image_job = current.get("image_job") if isinstance(current.get("image_job"), dict) else {} + candidates = normalize_candidate_diagnostics(generator_detail) + next_job = { + **image_job, + "status": "completed" if result["success"] else ( + "result_unknown" + if generator_detail.get("outcome_unknown") + else "ambiguous_result" + if generator_detail.get("stage") == "ambiguous" or candidates + else "failed" + ), + "finished_at": datetime.now().astimezone().isoformat(), + "receipt": { + "job_id": job.job_id, + "revision": job.revision, + "prompt_sha256": job.prompt_sha256, + "image_path": str(job.output_path.resolve()) if result["success"] else "", + "sha256": str(generator_detail.get("sha256") or ""), + "source": str(generator_detail.get("receipt_source") or ""), + }, + "candidates": candidates, + "codex_thread_id": str(generator_detail.get("codex_thread_id") or ""), + } + self.store.update( + job.group_name, + run_date, + status=status, + failed_stage="image" if not result["success"] else None, + error=error_detail, + image_error=error_detail, + image_status=result["status"], + error_type=error_type if not result["success"] else None, + stage_timings=stage_timings, + imagegen_ms=imagegen_ms, + image_generated_at=( + datetime.now().astimezone().isoformat() + if result["success"] + else current.get("image_generated_at") + ), + image_size_bytes=image_size_bytes, + image_attempt_count=int(generator_detail.get("attempt_count") or 0), + image_recovery_status=str(generator_detail.get("recovery_status") or ""), + image_recovered_at=str(generator_detail.get("recovered_at") or ""), + image_receipt_source=str(generator_detail.get("receipt_source") or ""), + recovery_status=str(generator_detail.get("recovery_status") or ""), + recovered_at=str(generator_detail.get("recovered_at") or ""), + receipt_source=str(generator_detail.get("receipt_source") or ""), + codex_thread_id=str(generator_detail.get("codex_thread_id") or ""), + codex_event_summary=generator_detail.get("codex_event_summary") or [], + codex_stderr_tail=str(generator_detail.get("codex_stderr_tail") or ""), + image_candidate_diagnostics=generator_detail.get("candidate_diagnostics") or [], + image_attempts=generator_detail.get("attempts") or [], + image_fallback_level=int(generator_detail.get("fallback_level") or 0), + image_variant=str(generator_detail.get("image_variant") or "normal"), + image_fallback_reason=str(generator_detail.get("fallback_reason") or ""), + image_fallback_font=str(generator_detail.get("fallback_font") or ""), + image_safety_redactions=generator_detail.get("safety_redactions") or [], + image_force_local_fallback=( + False if result["success"] else current.get("image_force_local_fallback", False) + ), + image_stale=False if result["success"] else current.get("image_stale", True), + artifact_stale_reason=( + "" if result["success"] else current.get("artifact_stale_reason", "") + ), + image_job=next_job, + ) + latest = self.store.load_run(job.group_name, run_date) + log_event( + logger, + "IMAGE_GENERATION_FINISHED", + group_task_id=latest.get("group_task_id"), + group_name=job.group_name, + run_date=run_date, + stage="IMAGE", + status="success" if result["success"] else "failed", + duration_ms=imagegen_ms, + attempt=latest.get("image_attempt_count", 0), + error_type=error_type if not result["success"] else "", + error_summary=error_detail or "", + ) + + def advance_ready(self, job: ImageJob, run_date: str) -> None: + run = self.store.load_run(job.group_name, run_date) + if run.get("status") == IMAGE_READY: + self.store.update(job.group_name, run_date, status=READY_TO_SEND) + + def run_jobs( + self, + image_jobs: list[ImageJob], + run_date: str, + *, + run_hook: Callable[[ImageJob, dict], None], + after_hook: Callable[[ImageJob, str], None], + ) -> list[dict]: + settings = getattr(self.image_generator, "settings", None) + max_workers = int(getattr(settings, "image_generation_concurrency", 1) or 1) + queue = SerialImageQueue(run_hook=run_hook, max_workers=max_workers) + queue_results = queue.run_all(image_jobs) + final_results: list[dict] = [] + for job, queue_result in zip(image_jobs, queue_results): + if queue_result.get("hook_error"): + final_results.append( + { + "group_name": job.group_name, + "status": "failed", + "error_type": "IMAGE_STATE_PERSIST_FAILED", + "detail": queue_result.get("detail") + or "图片结果状态持久化失败", + "failed_stage": "image", + } + ) + continue + after_hook(job, run_date) + run = self.store.load_run(job.group_name, run_date) + final_status = run.get("status") + if final_status == READY_TO_SEND: + final_results.append( + { + "group_name": job.group_name, + "status": "ready_to_send", + "detail": "图片已准备，可以发送", + "receipt_source": str(run.get("image_receipt_source") or ""), + "recovery_status": str(run.get("image_recovery_status") or ""), + "recovered_at": str(run.get("image_recovered_at") or ""), + "codex_thread_id": str(run.get("codex_thread_id") or ""), + } + ) + continue + if final_status == FAILED: + final_results.append( + { + "group_name": job.group_name, + "status": "failed", + "error_type": ( + run.get("error_type") + or queue_result.get("error_type") + or IMAGE_GENERATION_FAILED + ), + "detail": ( + run.get("image_error") + or run.get("error") + or queue_result.get("detail") + or "生图失败" + ), + "failed_stage": str(run.get("failed_stage") or "image"), + "recovery_status": str(run.get("image_recovery_status") or ""), + "codex_thread_id": str(run.get("codex_thread_id") or ""), + } + ) + continue + final_results.append( + { + "group_name": job.group_name, + "status": str( + final_status or queue_result.get("status") or "failed" + ).lower(), + "detail": queue_result.get("detail") or "图片任务未进入终态", + } + ) + return final_results diff --git a/app/pipeline/stage_result.py b/app/pipeline/stage_result.py new file mode 100644 index 0000000..fc4e029 --- /dev/null +++ b/app/pipeline/stage_result.py @@ -0,0 +1,46 @@ +"""Pipeline 阶段间的显式继续/终止协议。""" + +from __future__ import annotations + +from dataclasses import dataclass +from enum import StrEnum +from typing import Generic, TypeVar + + +PayloadT = TypeVar("PayloadT") + + +class StageDisposition(StrEnum): + CONTINUE = "continue" + TERMINAL = "terminal" + + +@dataclass(frozen=True) +class StageResult(Generic[PayloadT]): + """阶段要么携带下一阶段输入，要么携带对外终止结果。""" + + disposition: StageDisposition + value: PayloadT | None = None + response: dict | None = None + + @classmethod + def proceed(cls, value: PayloadT) -> "StageResult[PayloadT]": + return cls(StageDisposition.CONTINUE, value=value) + + @classmethod + def stop(cls, response: dict) -> "StageResult[PayloadT]": + return cls(StageDisposition.TERMINAL, response=response) + + @property + def is_terminal(self) -> bool: + return self.disposition is StageDisposition.TERMINAL + + def terminal_response(self) -> dict: + if not self.is_terminal or self.response is None: + raise RuntimeError("继续阶段没有终止响应") + return self.response + + def next_value(self) -> PayloadT: + if self.is_terminal or self.value is None: + raise RuntimeError("终止阶段没有下一阶段输入") + return self.value diff --git a/app/providers/ai/base.py b/app/providers/ai/base.py index 14d0666..62836f4 100644 --- a/app/providers/ai/base.py +++ b/app/providers/ai/base.py @@ -7,6 +7,22 @@ from app.ai.conversation_segments import PromptMessage +class ExternalCallError(RuntimeError): + """外部 AI 调用失败的统一基类。""" + + +class ExternalCallNotSubmittedError(ExternalCallError): + """可以确认请求尚未提交给 Provider。""" + + +class ExternalCallResultUnknownError(ExternalCallError): + """请求可能已经被 Provider 接收，禁止自动重试或切换备用。""" + + +class ExternalCallInvalidResponseError(ExternalCallError): + """Provider 已明确返回，但响应不可解析；禁止切备用，可安全本地降级。""" + + @dataclass class PromptContext: group_id: str diff --git a/app/providers/ai/codex.py b/app/providers/ai/codex.py index c1c0526..775eed4 100644 --- a/app/providers/ai/codex.py +++ b/app/providers/ai/codex.py @@ -8,22 +8,40 @@ from __future__ import annotations import json +import hashlib import os import shutil import subprocess import tempfile -import time from pathlib import Path from app.ai.concurrency import bounded_slot, normalized_limit from app.config.settings import Settings from app.core.logging import get_logger -from app.providers.ai.base import PromptGeneratorProvider +from app.providers.ai.base import ( + ExternalCallInvalidResponseError, + ExternalCallNotSubmittedError, + ExternalCallResultUnknownError, + PromptGeneratorProvider, +) from app.providers.ai.deepseek import DeepSeekV4FlashProvider logger = get_logger("groupbrief.ai") _CODEX_PROVIDER_NAMES = frozenset({"codex", "codex_gpt", "gpt"}) +_DISABLED_FALLBACK_NAMES = frozenset({"", "none", "disabled"}) +_SUMMARY_FALLBACK_NAMES = _DISABLED_FALLBACK_NAMES | {"deepseek"} + + +def validate_summary_provider_config(settings: Settings) -> tuple[str, str]: + """校验 V1/V2 共用的总结 Provider 配置，未知值禁止静默回退。""" + primary = str(settings.summary_provider_primary or "").strip().lower() + fallback = str(settings.summary_provider_fallback or "").strip().lower() + if primary not in (_CODEX_PROVIDER_NAMES | {"deepseek"}): + raise ValueError(f"不支持的群聊总结主 Provider：{settings.summary_provider_primary}") + if fallback not in _SUMMARY_FALLBACK_NAMES: + raise ValueError(f"不支持的群聊总结备用 Provider：{settings.summary_provider_fallback}") + return primary, fallback def _strip_json_fence(text: str) -> str: @@ -49,6 +67,7 @@ def __init__(self, settings: Settings): # 不调用父类初始化：父类会把 ai_model（DeepSeek 备用模型）写入 # self.model。这里的主模型必须与备用模型配置完全分离。 self.settings = settings + _, fallback = validate_summary_provider_config(settings) self.model = (settings.codex_summary_model or self.model).strip() or self.model self.codex_path = settings.codex_path or "codex" configured_home = ( @@ -60,9 +79,17 @@ def __init__(self, settings: Settings): self._resolved_binary = "" self._fallback = ( DeepSeekV4FlashProvider(settings) - if (settings.summary_provider_fallback or "").strip().lower() == "deepseek" + if fallback == "deepseek" else None ) + self.last_provider_used = "" + self.last_fallback_reason = "" + self.providers_used: list[str] = [] + + def reset_usage(self) -> None: + self.last_provider_used = "" + self.last_fallback_reason = "" + self.providers_used = [] def _resolve_binary(self) -> str: if self._resolved_binary: @@ -112,8 +139,8 @@ def health_report(self) -> dict: }, } - def health_check(self) -> tuple[bool, str]: - report = self.health_report() + def health_check(self, report: dict | None = None) -> tuple[bool, str]: + report = report or self.health_report() fallback = report["fallback"] fallback_text = "已配置" if fallback["configured"] else "未配置" if not report["ok"]: @@ -129,12 +156,19 @@ def _chat( max_tokens: int = 3000, ) -> str: try: - return self._codex_chat(messages, response_format=response_format) - except Exception as primary_exc: - logger.warning("Codex GPT 主调用失败，将检查 DeepSeek 备用：%s", str(primary_exc)[:200]) + result = self._codex_chat(messages, response_format=response_format) + self.last_provider_used = self.name + if self.name not in self.providers_used: + self.providers_used.append(self.name) + return result + except ExternalCallResultUnknownError: + # 主请求可能已经产生费用；此时切备用会形成第二次收费调用。 + raise + except ExternalCallNotSubmittedError as primary_exc: + logger.warning("Codex GPT 确认未提交，将检查 DeepSeek 备用：%s", str(primary_exc)[:200]) if self._fallback is None or not self._fallback.health_check()[0]: raise RuntimeError( - f"Codex GPT 主模型失败，DeepSeek 备用未配置：{str(primary_exc)[:180]}" + f"Codex GPT 未提交，DeepSeek 备用未配置：{str(primary_exc)[:180]}" ) from primary_exc try: result = self._fallback._chat( @@ -143,82 +177,106 @@ def _chat( temperature=temperature, max_tokens=max_tokens, ) - logger.info("Codex GPT 失败后已由 DeepSeek 备用完成本次调用") + fallback_name = str(getattr(self._fallback, "name", "deepseek")) + self.last_provider_used = fallback_name + self.last_fallback_reason = str(primary_exc)[:200] + if fallback_name not in self.providers_used: + self.providers_used.append(fallback_name) + logger.info("Codex GPT 未提交，已由 DeepSeek 备用完成本次调用") return result + except ExternalCallResultUnknownError: + raise except Exception as fallback_exc: raise RuntimeError( - "Codex GPT 主模型与 DeepSeek 备用均失败：" + "Codex GPT 未提交且 DeepSeek 备用失败：" f"Codex={str(primary_exc)[:120]}；DeepSeek={str(fallback_exc)[:120]}" ) from fallback_exc def _codex_chat(self, messages: list[dict], *, response_format: str = "text") -> str: resolved = self._resolve_binary() if not resolved: - raise RuntimeError(f"未找到 Codex CLI：{self.codex_path}") + raise ExternalCallNotSubmittedError(f"未找到 Codex CLI：{self.codex_path}") prompt = self._build_codex_prompt(messages, response_format=response_format) - attempts = max(1, int(self.settings.codex_summary_max_retries)) - last_error = "" - for attempt in range(1, attempts + 1): + request_id = hashlib.sha256(prompt.encode("utf-8")).hexdigest()[:24] + with tempfile.TemporaryDirectory(prefix="groupbrief-codex-summary-") as temp_dir: + output_path = Path(temp_dir) / "final.txt" + command = [ + resolved, + "exec", + "-C", + temp_dir, + "--ignore-user-config", + "--ignore-rules", + "--config", + 'model_reasoning_effort="medium"', + "--sandbox", + "read-only", + "--skip-git-repo-check", + "--ephemeral", + "--model", + self.model, + "--output-last-message", + str(output_path), + "-", + ] + environment = os.environ.copy() + environment["CODEX_HOME"] = str(self.codex_home) + timeout_seconds = max(1, int(self.settings.codex_summary_timeout_seconds)) try: - with tempfile.TemporaryDirectory(prefix="groupbrief-codex-summary-") as temp_dir: - output_path = Path(temp_dir) / "final.txt" - command = [ - resolved, - "exec", - "-C", - temp_dir, - "--sandbox", - "read-only", - "--skip-git-repo-check", - "--ephemeral", - "--model", - self.model, - "--output-last-message", - str(output_path), - "-", - ] - environment = os.environ.copy() - environment["CODEX_HOME"] = str(self.codex_home) - with bounded_slot( - "codex_summary_request", - normalized_limit(self.settings.codex_summary_request_concurrency, 2), - ): - proc = subprocess.run( - command, - capture_output=True, - text=True, - encoding="utf-8", - errors="replace", - timeout=max(1, int(self.settings.codex_summary_timeout_seconds)), - cwd=temp_dir, - input=prompt, - env=environment, + with bounded_slot( + "codex_summary_request", + normalized_limit(self.settings.codex_summary_request_concurrency, 2), + ): + proc = subprocess.run( + command, + capture_output=True, + text=True, + encoding="utf-8", + errors="replace", + timeout=timeout_seconds, + cwd=temp_dir, + input=prompt, + env=environment, ) - if proc.returncode != 0: - # stderr/stdout 由外部 CLI 产生，不能假定其中绝不回显输入； - # 错误只保留退出码，避免聊天正文进入日志或 API 响应。 - raise RuntimeError(f"Codex CLI 退出码 {proc.returncode}") - if not output_path.is_file(): - raise RuntimeError("Codex CLI 未生成最终文本") - text = output_path.read_text(encoding="utf-8").strip() - if not text: - raise RuntimeError("Codex GPT 返回空内容") - if response_format == "json_object": - text = _strip_json_fence(text) - parsed = json.loads(text) - if not isinstance(parsed, dict): - raise ValueError("Codex GPT JSON 输出不是对象") - logger.info("Codex GPT 调用成功（model=%s attempt=%d）", self.model, attempt) - return text - except subprocess.TimeoutExpired: - last_error = f"Codex GPT 超时（>{self.settings.codex_summary_timeout_seconds}s）" - except Exception as exc: - last_error = str(exc)[:240] - logger.warning("Codex GPT attempt %d 失败：%s", attempt, last_error) - if attempt < attempts: - time.sleep(min(4.0, float(2 ** (attempt - 1)))) - raise RuntimeError(last_error or "Codex GPT 调用失败") + except subprocess.TimeoutExpired as exc: + raise ExternalCallResultUnknownError( + "Codex GPT 超时且结果未知" + f"（timeout={timeout_seconds}s request_id={request_id}）" + ) from exc + except OSError as exc: + raise ExternalCallNotSubmittedError( + f"Codex CLI 未能启动（request_id={request_id}）：{str(exc)[:160]}" + ) from exc + + if proc.returncode != 0: + # CLI 已经启动，不能证明 Provider 没有接收请求；禁止内部重试和备用调用。 + raise ExternalCallResultUnknownError( + f"Codex CLI 退出码 {proc.returncode}，结果未知（request_id={request_id}）" + ) + if not output_path.is_file(): + raise ExternalCallResultUnknownError( + f"Codex CLI 未生成最终文本，结果未知（request_id={request_id}）" + ) + text = output_path.read_text(encoding="utf-8").strip() + if not text: + raise ExternalCallResultUnknownError( + f"Codex GPT 返回空内容（request_id={request_id}）" + ) + if response_format == "json_object": + try: + text = _strip_json_fence(text) + parsed = json.loads(text) + except (ValueError, json.JSONDecodeError) as exc: + raise ExternalCallInvalidResponseError( + f"Codex GPT JSON 无效（request_id={request_id}）" + ) from exc + if not isinstance(parsed, dict): + raise ExternalCallInvalidResponseError( + f"Codex GPT JSON 输出不是对象（request_id={request_id}）" + ) + logger.info("Codex GPT 调用成功（model=%s request_id=%s）", self.model, request_id) + return text @staticmethod def _build_codex_prompt(messages: list[dict], *, response_format: str) -> str: @@ -239,9 +297,9 @@ def _build_codex_prompt(messages: list[dict], *, response_format: str) -> str: def build_summary_provider(settings: Settings) -> PromptGeneratorProvider: """按配置构造 V1/V2 共用的群聊总结 Provider。""" - primary = (settings.summary_provider_primary or "codex").strip().lower() + primary, _ = validate_summary_provider_config(settings) if primary in _CODEX_PROVIDER_NAMES: return CodexGPTProvider(settings) if primary == "deepseek": return DeepSeekV4FlashProvider(settings) - raise ValueError(f"不支持的群聊总结主 Provider：{settings.summary_provider_primary}") + raise AssertionError("总结 Provider 配置校验未覆盖已知类型") diff --git a/app/providers/ai/deepseek.py b/app/providers/ai/deepseek.py index 5c60203..0d3c96c 100644 --- a/app/providers/ai/deepseek.py +++ b/app/providers/ai/deepseek.py @@ -6,6 +6,8 @@ from __future__ import annotations +import hashlib +import json import random import time @@ -30,7 +32,13 @@ ) from app.config.settings import Settings from app.core.logging import get_logger -from app.providers.ai.base import ImagePromptResult, PromptContext, PromptGeneratorProvider +from app.providers.ai.base import ( + ExternalCallNotSubmittedError, + ExternalCallResultUnknownError, + ImagePromptResult, + PromptContext, + PromptGeneratorProvider, +) logger = get_logger("groupbrief.ai") @@ -224,10 +232,15 @@ def _chat( if response_format == "json_object": payload["response_format"] = {"type": "json_object"} + request_id = hashlib.sha256( + json.dumps(payload, ensure_ascii=False, sort_keys=True, separators=(",", ":")).encode( + "utf-8" + ) + ).hexdigest()[:24] attempts = max(1, int(self.settings.ai_max_retries)) last_error = "" + last_failure_was_pre_submit = False for attempt in range(1, attempts + 1): - retryable = True try: with bounded_slot( "deepseek_request", @@ -239,21 +252,67 @@ def _chat( json=payload, timeout=self.settings.ai_timeout_seconds, ) - if response.status_code == 200: + except (httpx.ConnectTimeout, httpx.PoolTimeout, httpx.ConnectError) as exc: + # 连接尚未建立或尚未取得连接池槽位，可以确认未提交。 + last_error = str(exc)[:160] or exc.__class__.__name__ + last_failure_was_pre_submit = True + logger.warning( + "DeepSeek 连接前失败（attempt=%d request_id=%s）：%s", + attempt, + request_id, + last_error, + ) + if attempt < attempts: + delay = min(8.0, 2 ** (attempt - 1)) + random.uniform(0.0, 0.35) + time.sleep(delay) + continue + break + except httpx.InvalidURL as exc: + raise ExternalCallNotSubmittedError( + f"DeepSeek URL 无效（request_id={request_id}）：{str(exc)[:160]}" + ) from exc + except httpx.RequestError as exc: + # 写入超时、读取超时、远端中断都无法证明请求未到达 Provider。 + raise ExternalCallResultUnknownError( + f"DeepSeek 请求结果未知（request_id={request_id}）：{exc.__class__.__name__}" + ) from exc + + if response.status_code == 200: + try: data = response.json() content = data["choices"][0]["message"]["content"] if not isinstance(content, str) or not content.strip(): raise ValueError("DeepSeek 返回空内容") - logger.info("DeepSeek 调用成功（attempt %d）", attempt) - return content - last_error = f"HTTP {response.status_code}: {response.text[:200]}" - retryable = response.status_code in {429, 503} or response.status_code >= 500 - logger.warning("DeepSeek attempt %d 失败：%s", attempt, last_error) - except Exception as exc: - last_error = str(exc)[:200] - logger.warning("DeepSeek attempt %d 异常：%s", attempt, last_error) + except (KeyError, IndexError, TypeError, ValueError, json.JSONDecodeError) as exc: + raise ExternalCallResultUnknownError( + f"DeepSeek 成功响应无法解析（request_id={request_id}）" + ) from exc + logger.info( + "DeepSeek 调用成功（attempt=%d request_id=%s）", attempt, request_id + ) + return content + + # 429/503 明确表示限流或暂不可用，可以受控重试；其他 5xx 不能 + # 证明 Provider 没有在内部完成请求，按结果未知处理。 + last_failure_was_pre_submit = False + last_error = f"HTTP {response.status_code}" + if response.status_code >= 500 and response.status_code != 503: + raise ExternalCallResultUnknownError( + f"DeepSeek 服务端错误且结果未知（request_id={request_id}）：{last_error}" + ) + retryable = response.status_code in {429, 503} + logger.warning( + "DeepSeek 明确拒绝（attempt=%d request_id=%s）：%s", + attempt, + request_id, + last_error, + ) if attempt >= attempts or not retryable: break delay = min(8.0, 2 ** (attempt - 1)) + random.uniform(0.0, 0.35) time.sleep(delay) + if last_failure_was_pre_submit: + raise ExternalCallNotSubmittedError( + f"DeepSeek 未提交：{last_error}（request_id={request_id}）" + ) raise RuntimeError(f"DeepSeek 调用失败：{last_error}") diff --git a/app/providers/history/base.py b/app/providers/history/base.py index ac09a06..60656e4 100644 --- a/app/providers/history/base.py +++ b/app/providers/history/base.py @@ -53,6 +53,8 @@ class RawMessage: source: str = "" source_message_id: str = "" content_hash: str = "" + upstream_sender_name: str = "" + sender_name_source: str = "" @dataclass diff --git a/app/providers/history/contact_resolver.py b/app/providers/history/contact_resolver.py index a10fc7c..d8ec860 100644 --- a/app/providers/history/contact_resolver.py +++ b/app/providers/history/contact_resolver.py @@ -16,13 +16,163 @@ from __future__ import annotations import os +import re import sqlite3 from pathlib import Path +from typing import Iterator from app.core.logging import get_logger logger = get_logger("groupbrief.contact") +_SYSTEM_DISPLAY_NAME_MARKERS = ( + "加入群聊", + "进入群聊", + "退出群聊", + "移出群聊", + "邀请了", + "修改群名", + "修改了群名", + "红包待领取", + "撤回了一条消息", + "拍了拍", +) + + +def is_plausible_group_card(value: object) -> bool: + """排除 ext_buffer 中混入的系统事件文本和结构性脏值。""" + text = str(value or "").strip() + return bool( + text + and len(text) <= 64 + and "\n" not in text + and "\r" not in text + and not any(marker in text for marker in _SYSTEM_DISPLAY_NAME_MARKERS) + ) + + +def _decode_varint(raw: bytes, offset: int) -> tuple[int | None, int]: + value = 0 + shift = 0 + position = int(offset) + while position < len(raw): + byte = raw[position] + position += 1 + value |= (byte & 0x7F) << shift + if (byte & 0x80) == 0: + return value, position + shift += 7 + if shift > 63: + break + return None, len(raw) + + +def _iter_protobuf_fields(raw: bytes) -> Iterator[tuple[int, int, bytes]]: + """遍历群成员 ext_buffer 中的 length-delimited protobuf 字段。""" + position = 0 + while position < len(raw): + tag, next_position = _decode_varint(raw, position) + if tag is None or next_position <= position: + break + position = next_position + field_number = int(tag) >> 3 + wire_type = int(tag) & 0x07 + if wire_type == 0: + _, next_position = _decode_varint(raw, position) + if next_position <= position: + break + position = next_position + continue + if wire_type == 1: + position += 8 + continue + if wire_type == 5: + position += 4 + continue + if wire_type != 2: + break + size, next_position = _decode_varint(raw, position) + if size is None or next_position <= position: + break + position = next_position + end = position + int(size) + if end > len(raw): + break + yield field_number, wire_type, raw[position:end] + position = end + + +def _looks_like_username(value: str) -> bool: + text = str(value or "").strip() + if not text: + return False + if text.startswith(("wxid_", "gh_")) or text.endswith("@chatroom") or "@" in text: + return True + return bool( + 6 <= len(text) <= 32 + and not re.search(r"\s", text) + and re.fullmatch(r"[A-Za-z][A-Za-z0-9_-]+", text) + ) + + +def _pick_legacy_group_card(fields: list[tuple[int, str]], username: str) -> str: + """兼容旧布局：field 4 是成员 ID、field 1 是显示名。""" + candidates: list[tuple[int, int, str]] = [] + for index, (field_number, value) in enumerate(fields): + text = str(value or "").strip() + if not text or text == username or len(text) > 64 or "\n" in text or "\r" in text: + continue + if text.startswith(("wxid_", "gh_")) or text.endswith("@chatroom") or "@" in text: + continue + score = (100 if field_number == 2 else 0) + (20 if not _looks_like_username(text) else 0) + score += max(0, 32 - len(text)) + candidates.append((score, -index, text)) + return max(candidates, default=(-1, 0, ""))[2] + + +def _parse_group_nicknames(ext_buffer: bytes, usernames: set[str]) -> dict[str, str]: + """按当前字段语义解析群名片，并保留受限的旧布局兼容。""" + result: dict[str, str] = {} + primary_seen: set[str] = set() + for _, wire_type, chunk in _iter_protobuf_fields(ext_buffer): + if wire_type != 2 or not chunk: + continue + text_fields: list[tuple[int, str]] = [] + for field_number, nested_wire_type, value in _iter_protobuf_fields(chunk): + if nested_wire_type != 2 or not value or len(value) > 256: + continue + try: + text = bytes(value).decode("utf-8", errors="strict").strip() + except UnicodeDecodeError: + continue + if text: + text_fields.append((field_number, text)) + if not text_fields: + continue + + field1 = [value for field_number, value in text_fields if field_number == 1] + field2 = [value for field_number, value in text_fields if field_number == 2] + primary_members = [value for value in field1 if value in usernames] + if primary_members: + for username in primary_members: + primary_seen.add(username) + if field2 and is_plausible_group_card(field2[0]): + result[username] = field2[0] + else: + result.pop(username, None) + continue + + legacy_members = [ + value + for field_number, value in text_fields + if field_number == 4 and value in usernames and value not in primary_seen + ] + for username in legacy_members: + display = _pick_legacy_group_card(text_fields, username) + if display: + result[username] = display + return result + def find_contact_db() -> Path | None: """自动探测 WeChatDataAnalysis 解密的联系人数据库。""" @@ -104,3 +254,30 @@ def resolve_name(self, username: str, fallback: str = "") -> str: """解析微信号对应的显示名，找不到时回退到 fallback。""" name = self.display_name(username) return name if name else fallback + + def group_nicknames(self, chatroom_id: str, usernames: list[str]) -> dict[str, str]: + """只读解析指定群的成员群名片；任何读取或格式异常均安全回退为空。""" + chatroom = str(chatroom_id or "").strip() + targets = { + str(username or "").strip() + for username in usernames + if str(username or "").strip() + } + if not chatroom.endswith("@chatroom") or not targets or not self.available: + return {} + try: + con = sqlite3.connect(f"file:{self._db_path}?mode=ro", uri=True) + try: + row = con.execute( + "SELECT ext_buffer FROM chat_room WHERE username = ? LIMIT 1", + (chatroom,), + ).fetchone() + finally: + con.close() + if row is None or row[0] is None: + return {} + raw = row[0].tobytes() if isinstance(row[0], memoryview) else bytes(row[0]) + return _parse_group_nicknames(raw, targets) if raw else {} + except Exception as exc: + logger.warning("读取群名片映射失败：%s", str(exc)[:200]) + return {} diff --git a/app/providers/history/mock.py b/app/providers/history/mock.py index e0fca20..6f8037f 100644 --- a/app/providers/history/mock.py +++ b/app/providers/history/mock.py @@ -37,8 +37,12 @@ def _normalize_ts(raw: str) -> datetime: class MockProvider(ChatHistoryProvider): name = "mock" - def __init__(self, fixtures_dir: Path | None = None): - settings: Settings = get_settings() + def __init__( + self, + fixtures_dir: Path | None = None, + settings: Settings | None = None, + ): + settings = settings or get_settings() self.fixtures_dir = fixtures_dir or settings.fixtures_dir def health_check(self) -> ProviderHealth: diff --git a/app/providers/history/registry.py b/app/providers/history/registry.py index e211979..e2f0864 100644 --- a/app/providers/history/registry.py +++ b/app/providers/history/registry.py @@ -1,6 +1,6 @@ -"""Provider 注册与自动降级。 +"""Provider 注册与显式降级。 -优先级：主 Provider → 备用 Provider → Mock（仅开发模式启用）。 +优先级：主 Provider → 备用 Provider → Mock（仅显式测试模式启用）。 """ from __future__ import annotations @@ -11,6 +11,10 @@ from app.providers.history.wechat_cli import WechatCliProvider from app.providers.history.wechat_data_analysis import WeChatDataAnalysisProvider + +class ProviderConfigurationError(ValueError): + """历史 Provider 配置无效，禁止静默改用其他实现。""" + PROVIDER_CLASSES = { "wechat_data_analysis": WeChatDataAnalysisProvider, "wechat_cli": WechatCliProvider, @@ -18,20 +22,37 @@ } -def build_providers(settings: Settings | None = None) -> list[ChatHistoryProvider]: - settings = settings or get_settings() - providers: list[ChatHistoryProvider] = [] +def validate_history_provider_config(settings: Settings) -> list[str]: + """只校验并返回 Provider 顺序，不实例化或探测外部依赖。""" order = [settings.history_provider_primary, settings.history_provider_fallback] + names: list[str] = [] seen: set[str] = set() - for name in order: + for raw_name in order: + name = str(raw_name or "").strip().lower() if not name or name in seen: continue seen.add(name) cls = PROVIDER_CLASSES.get(name) - if cls: - providers.append(cls()) - if settings.history_provider_mock_enabled and "mock" not in seen: - providers.append(MockProvider()) + if cls is None: + raise ProviderConfigurationError(f"不支持的历史 Provider：{raw_name}") + if name == "mock" and not settings.allow_test_providers: + raise ProviderConfigurationError("真实运行禁止使用 mock 历史 Provider") + names.append(name) + if not names: + raise ProviderConfigurationError("至少需要配置一个历史 Provider") + return names + + +def build_providers(settings: Settings | None = None) -> list[ChatHistoryProvider]: + settings = settings or get_settings() + names = validate_history_provider_config(settings) + providers = [PROVIDER_CLASSES[name](settings=settings) for name in names] + if ( + settings.history_provider_mock_enabled + and settings.allow_test_providers + and "mock" not in names + ): + providers.append(MockProvider(settings=settings)) return providers diff --git a/app/providers/history/wechat_cli.py b/app/providers/history/wechat_cli.py index 9105120..45547cd 100644 --- a/app/providers/history/wechat_cli.py +++ b/app/providers/history/wechat_cli.py @@ -27,8 +27,12 @@ class WechatCliProvider(ChatHistoryProvider): name = "wechat_cli" - def __init__(self, cli_path: str | None = None): - settings: Settings = get_settings() + def __init__( + self, + cli_path: str | None = None, + settings: Settings | None = None, + ): + settings = settings or get_settings() self.cli_path = cli_path or settings.wechat_cli_path or "wechat-cli" self.export_dir = settings.data_dir / "wechat_cli_export" diff --git a/app/providers/history/wechat_data_analysis.py b/app/providers/history/wechat_data_analysis.py index eab5838..207b026 100644 --- a/app/providers/history/wechat_data_analysis.py +++ b/app/providers/history/wechat_data_analysis.py @@ -26,7 +26,7 @@ from app.config.settings import Settings, get_settings from app.core.logging import get_logger -from app.providers.history.contact_resolver import ContactResolver +from app.providers.history.contact_resolver import ContactResolver, is_plausible_group_card from app.providers.history.base import ( ChatHistoryProvider, FetchResult, @@ -193,7 +193,7 @@ def _mcp_health(self) -> ProviderHealth: ) def _find_wechat_dir(self) -> Path | None: - settings = get_settings() + settings = self._settings if settings.wechat_data_dir: p = Path(settings.wechat_data_dir) if p.exists(): @@ -289,7 +289,11 @@ def _fetch_messages_export( if not messages: return FetchResult(self.name, group_id, [], ProviderStatus.EMPTY_RESULT, "该时间段无消息") - return FetchResult(self.name, group_id, messages, ProviderStatus.OK) + messages.sort(key=lambda item: (item.timestamp, item.source_message_id)) + stats: dict = {"read_strategy": "export"} + _resolve_sender_names(messages, self._contacts, group_id, stats) + stats["message_count"] = len(messages) + return FetchResult(self.name, group_id, messages, ProviderStatus.OK, meta=stats) def _fetch_messages_mcp( self, @@ -359,7 +363,7 @@ def collect(items: list) -> None: ) messages.sort(key=lambda item: (item.timestamp, item.source_message_id)) - _resolve_sender_names(messages, self._contacts, stats) + _resolve_sender_names(messages, self._contacts, group_id, stats) stats["fetch_elapsed_ms"] = round((perf_counter() - started_at) * 1000) stats["message_count"] = len(messages) if not messages: @@ -591,14 +595,21 @@ def _sanitize_sender_name(value: object) -> str: for char in str(value): if char in _INVISIBLE_NAME_CHARS: continue - if unicodedata.category(char) in {"Cf", "Cc"}: + category = unicodedata.category(char) + if category == "Cf": + continue + if category == "Cc" or char.isspace(): + visible.append(" ") continue visible.append(char) - return "".join(visible).strip() + return " ".join("".join(visible).split()) def _usable_sender_name(name: str, sender_id: str) -> bool: - if not name or name.lower() in {"none", "null"}: + if ( + not is_plausible_group_card(name) + or name.lower() in {"none", "null", "(未知)", "未知"} + ): return False return not sender_id or name.casefold() != sender_id.strip().casefold() @@ -609,13 +620,25 @@ def _anonymous_sender_name(sender_id: str) -> str: def _resolve_sender_names( - messages: list[RawMessage], contacts: ContactResolver, stats: dict | None = None + messages: list[RawMessage], + contacts: ContactResolver, + group_id: str, + stats: dict | None = None, ) -> None: - """拆分上游错误共享昵称，并为所有身份生成唯一、稳定的展示名。""" + """群名片优先，异常上游名回退联系人，并稳定拆分真实同名。""" upstream_ids: dict[str, set[str]] = {} + sender_ids = { + (message.sender_id or "").strip() + for message in messages + if (message.sender_id or "").strip() + } + sender_ids_casefold = {sender_id.casefold() for sender_id in sender_ids} for message in messages: sender_id = (message.sender_id or "").strip() upstream_name = _sanitize_sender_name(message.sender_name) + message.upstream_sender_name = _sanitize_sender_name( + message.upstream_sender_name or upstream_name + ) message.sender_name = upstream_name if sender_id and _usable_sender_name(upstream_name, sender_id): upstream_ids.setdefault(upstream_name.casefold(), set()).add(sender_id) @@ -623,32 +646,54 @@ def _resolve_sender_names( collision_names = { normalized_name for normalized_name, sender_ids in upstream_ids.items() if len(sender_ids) > 1 } + group_nicknames = contacts.group_nicknames(group_id, sorted(sender_ids)) contact_identities: set[str] = set() anonymous_identities: set[str] = set() for message in messages: sender_id = (message.sender_id or "").strip() upstream_name = _sanitize_sender_name(message.sender_name) - upstream_usable = _usable_sender_name(upstream_name, sender_id) + group_nickname = _sanitize_sender_name(group_nicknames.get(sender_id, "")) + group_nickname_usable = bool( + _usable_sender_name(group_nickname, sender_id) + and group_nickname.casefold() not in (sender_ids_casefold - {sender_id.casefold()}) + ) + upstream_usable = bool( + _usable_sender_name(upstream_name, sender_id) + and upstream_name.casefold() not in (sender_ids_casefold - {sender_id.casefold()}) + ) upstream_conflicted = bool( sender_id and upstream_usable and upstream_name.casefold() in collision_names ) - resolved_name = "" - if sender_id and (upstream_conflicted or not upstream_usable): - resolved_name = _sanitize_sender_name(contacts.resolve_name(sender_id)) + contact_name = "" + if sender_id and not group_nickname_usable and (upstream_conflicted or not upstream_usable): + contact_name = _sanitize_sender_name(contacts.resolve_name(sender_id)) - if _usable_sender_name(resolved_name, sender_id): - message.sender_name = resolved_name - contact_identities.add(sender_id) + if group_nickname_usable: + message.sender_name = group_nickname + message.sender_name_source = "wechat_data_analysis" elif upstream_usable and not upstream_conflicted: message.sender_name = upstream_name + message.sender_name_source = "wechat_data_analysis" + elif contact_name and contact_name.casefold() not in {"none", "null", "(未知)", "未知"}: + message.sender_name = contact_name + message.sender_name_source = "contact" + contact_identities.add(sender_id) else: message.sender_name = _anonymous_sender_name(sender_id or upstream_name) + message.sender_name_source = "anonymous" anonymous_identities.add(sender_id or upstream_name or message.sender_name) # 即便 contact.db 中存在真实同名，最终展示也必须保持一身份一名称。 labels = { item.key: item.name - for item in build_speaker_stats((message.sender_id, message.sender_name) for message in messages) + for item in build_speaker_stats( + ( + message.sender_id, + message.sender_name, + message.sender_name_source == "contact", + ) + for message in messages + ) } for message in messages: key = speaker_identity_key(message.sender_id, message.sender_name) @@ -693,17 +738,27 @@ def _normalize_ts(raw: str) -> datetime: def _to_raw(item: dict, ts: datetime) -> RawMessage: + upstream_name = _first_visible_str( + item, + "senderDisplayName", + "sender_display_name", + "sender_name", + "senderName", + "fromNickName", + ) return RawMessage( group_id=item["group_id"], group_name=item.get("group_name", ""), sender_id=item.get("sender_id", ""), - sender_name=item.get("sender_name", ""), + sender_name=upstream_name, timestamp=ts, message_type=item.get("message_type", "text"), content=item.get("content", ""), source=item.get("source", "wechat_data_analysis"), source_message_id=item.get("source_message_id", ""), content_hash=item.get("content_hash", ""), + upstream_sender_name=upstream_name, + sender_name_source="wechat_data_analysis", ) @@ -842,17 +897,20 @@ def _mcp_message_type(render_type) -> str: def _mcp_to_raw(item: dict, group_id: str, ts: datetime) -> RawMessage: + upstream_name = _first_visible_str( + item, "senderDisplayName", "sender_display_name", "senderName", "fromNickName" + ) return RawMessage( group_id=group_id, group_name="", sender_id=_first_str(item, "senderUsername", "sender_username", "sender", "fromUser"), - sender_name=_first_visible_str( - item, "senderDisplayName", "sender_display_name", "senderName", "fromNickName" - ), + sender_name=upstream_name, timestamp=ts, message_type=_mcp_message_type(item.get("renderType", item.get("render_type"))), content=str(item.get("content") or ""), source="wechat_data_analysis", source_message_id=str(item.get("id") or item.get("messageId") or ""), content_hash="", + upstream_sender_name=upstream_name, + sender_name_source="wechat_data_analysis", ) diff --git a/app/providers/history/wechat_mcp.py b/app/providers/history/wechat_mcp.py index bd0913f..0e1e1a3 100644 --- a/app/providers/history/wechat_mcp.py +++ b/app/providers/history/wechat_mcp.py @@ -17,6 +17,8 @@ from time import monotonic from urllib.parse import urlsplit +from app.core.logging import get_logger + LOOPBACK_HOSTS = frozenset({"127.0.0.1", "localhost", "::1"}) # 默认回环主机集合（可传入额外允许主机，如 Docker 的 host.docker.internal） @@ -26,6 +28,7 @@ _PROXY_FREE_OPENER = urllib.request.build_opener(urllib.request.ProxyHandler({})) _READ_CHUNK_SIZE = 64 * 1024 _MAX_RESPONSE_BYTES = 128 * 1024 * 1024 +logger = get_logger("groupbrief.wechat_mcp") class MCPError(Exception): @@ -185,16 +188,16 @@ def _abort() -> None: try: response_socket.shutdown(socket.SHUT_RDWR) except OSError: - pass + logger.debug("MCP 截止计时器关闭响应 socket 失败", exc_info=True) try: response_socket.close() except OSError: - pass + logger.debug("MCP 截止计时器释放响应 socket 失败", exc_info=True) return try: response.close() except (AttributeError, OSError): - pass + logger.debug("MCP 截止计时器关闭响应失败", exc_info=True) timer = Timer(remaining, _abort) timer.daemon = True @@ -217,7 +220,7 @@ def _set_response_socket_timeout(response, remaining: float) -> None: try: response_socket.settimeout(max(remaining, 0.001)) except OSError: - pass + logger.debug("MCP 响应 socket 超时设置失败", exc_info=True) def _result_error_message(result: dict) -> str: diff --git a/app/ranking/engine.py b/app/ranking/engine.py index 39cdb03..814e4d5 100644 --- a/app/ranking/engine.py +++ b/app/ranking/engine.py @@ -9,9 +9,21 @@ from __future__ import annotations +from collections import Counter +import hashlib + from app.data_sources.base import V2Message +from app.ranking.policies import ( + RANKING_POLICY_TEXT_PRIMARY, + normalize_ranking_policy, + normalize_sender_name_policy, +) from app.services.message_normalizer import COUNTABLE_TYPES, SYSTEM_KEYWORDS -from app.services.speaker_identity import build_speaker_stats, speaker_name_sort_key +from app.services.speaker_identity import ( + build_speaker_stats, + speaker_identity_key, + speaker_name_sort_key, +) from app.ranking.engine_types import RankingResult, TopSpeaker @@ -33,23 +45,71 @@ def compute( period_start: str, period_end: str, top_limit: int = 10, + count_policy: str = "all_messages", + name_source: str = "resolved", ) -> RankingResult: if top_limit <= 0: raise ValueError("排行榜上限必须大于 0") + policy = normalize_ranking_policy(count_policy) + normalized_name_source = normalize_sender_name_policy(name_source) + countable_messages = [message for message in messages if self._countable(message)] speakers = build_speaker_stats( - (m.sender_id, m.sender_name) for m in messages if self._countable(m) + ( + message.sender_id, + message.sender_name, + message.sender_name_source == "contact", + ) + for message in countable_messages ) - message_count = sum(item.count for item in speakers) + text_counts: Counter[tuple[str, str]] = Counter() + interaction_counts: Counter[tuple[str, str]] = Counter() + for message in countable_messages: + key = speaker_identity_key(message.sender_id, message.sender_name) + if key is None: + continue + if message.message_type == "text": + text_counts[key] += 1 + else: + interaction_counts[key] += 1 + + message_count = len(countable_messages) speaker_count = len(speakers) + text_message_count = sum(text_counts.values()) + interaction_message_count = sum(interaction_counts.values()) + text_speaker_count = len(text_counts) - # 确定性排序：消息数降序，同数量按名称稳定升序 - ordered = sorted( - speakers, - key=lambda item: (-item.count, speaker_name_sort_key(item.name), item.key), - ) + if policy == RANKING_POLICY_TEXT_PRIMARY: + # 互动数只展示，不参与名次或同分排序。 + ordered = sorted( + (item for item in speakers if text_counts[item.key] > 0), + key=lambda item: ( + -text_counts[item.key], + speaker_name_sort_key(item.name), + item.key, + ), + ) + else: + ordered = sorted( + speakers, + key=lambda item: (-item.count, speaker_name_sort_key(item.name), item.key), + ) top_speakers = [ - TopSpeaker(rank=i + 1, name=item.name, count=item.count) + TopSpeaker( + rank=i + 1, + name=item.name, + count=( + text_counts[item.key] + if policy == RANKING_POLICY_TEXT_PRIMARY + else item.count + ), + identity_key=hashlib.sha256( + f"{item.key[0]}:{item.key[1]}".encode("utf-8") + ).hexdigest()[:16], + text_count=text_counts[item.key], + interaction_count=interaction_counts[item.key], + name_source=normalized_name_source, + ) for i, item in enumerate(ordered[:top_limit]) ] @@ -61,4 +121,8 @@ def compute( message_count=message_count, top_limit=top_limit, top_speakers=top_speakers, + count_policy=policy, + text_message_count=text_message_count, + interaction_message_count=interaction_message_count, + text_speaker_count=text_speaker_count, ) diff --git a/app/ranking/engine_types.py b/app/ranking/engine_types.py index 4f00b20..c87cf72 100644 --- a/app/ranking/engine_types.py +++ b/app/ranking/engine_types.py @@ -14,9 +14,23 @@ class TopSpeaker: rank: int name: str count: int + identity_key: str = "" + text_count: int = 0 + interaction_count: int = 0 + name_source: str = "resolved" def to_dict(self) -> dict: - return {"rank": self.rank, "name": self.name, "count": self.count} + payload = { + "rank": self.rank, + "name": self.name, + "count": self.count, + "text_count": self.text_count, + "interaction_count": self.interaction_count, + "name_source": self.name_source, + } + if self.identity_key: + payload["identity_key"] = self.identity_key + return payload @dataclass @@ -31,6 +45,10 @@ class RankingResult: top_speakers: list[TopSpeaker] = field(default_factory=list) # 放在原有字段之后，保留旧代码按位置传入 top_speakers 的兼容性。 top_limit: int = 10 + count_policy: str = "all_messages" + text_message_count: int = 0 + interaction_message_count: int = 0 + text_speaker_count: int = 0 def to_dict(self) -> dict: return { @@ -39,6 +57,10 @@ def to_dict(self) -> dict: "period_end": self.period_end, "speaker_count": self.speaker_count, "message_count": self.message_count, + "count_policy": self.count_policy, + "text_message_count": self.text_message_count, + "interaction_message_count": self.interaction_message_count, + "text_speaker_count": self.text_speaker_count, "top_limit": self.top_limit, "top_speakers": [s.to_dict() for s in self.top_speakers], } diff --git a/app/ranking/policies.py b/app/ranking/policies.py new file mode 100644 index 0000000..a8abde9 --- /dev/null +++ b/app/ranking/policies.py @@ -0,0 +1,34 @@ +"""群级排行榜与发言人名称策略。""" + +from __future__ import annotations + + +RANKING_POLICY_ALL_MESSAGES = "all_messages" +RANKING_POLICY_TEXT_PRIMARY = "text_primary_with_interactions" +RANKING_POLICIES = frozenset( + {RANKING_POLICY_ALL_MESSAGES, RANKING_POLICY_TEXT_PRIMARY} +) + +SENDER_NAME_POLICY_RESOLVED = "resolved" +SENDER_NAME_POLICY_WECHAT_DATA_ANALYSIS = "wechat_data_analysis" +SENDER_NAME_POLICIES = frozenset( + {SENDER_NAME_POLICY_RESOLVED, SENDER_NAME_POLICY_WECHAT_DATA_ANALYSIS} +) + + +def normalize_ranking_policy(value: object) -> str: + policy = str(value or RANKING_POLICY_ALL_MESSAGES).strip().lower() + if policy not in RANKING_POLICIES: + raise ValueError(f"不支持的排行榜统计口径：{value}") + return policy + + +def normalize_sender_name_policy(value: object) -> str: + policy = str(value or SENDER_NAME_POLICY_RESOLVED).strip().lower() + if policy not in SENDER_NAME_POLICIES: + raise ValueError(f"不支持的发言人名称策略：{value}") + return policy + + +def uses_strict_image_fact_contract(ranking_policy: object) -> bool: + return normalize_ranking_policy(ranking_policy) == RANKING_POLICY_TEXT_PRIMARY diff --git a/app/ranking/renderer.py b/app/ranking/renderer.py index b6a8cf6..ef81361 100644 --- a/app/ranking/renderer.py +++ b/app/ranking/renderer.py @@ -9,6 +9,7 @@ import re from app.ranking.engine_types import RankingResult +from app.ranking.policies import RANKING_POLICY_TEXT_PRIMARY from app.ranking.template_service import ( SUPPORTED_VARS, TemplateError, @@ -23,15 +24,25 @@ def render_ranking(result: RankingResult, template_text: str) -> str: """把模板与统计结果渲染成最终排行榜文本。""" validate_template(template_text) - top_lines = "\n".join( - f"{s.rank}.{s.name}【{s.count}】" for s in result.top_speakers - ) + if result.count_policy == RANKING_POLICY_TEXT_PRIMARY: + top_lines = "\n".join( + f"{s.rank}.{s.name}【文字 {s.text_count}｜互动 {s.interaction_count}】" + for s in result.top_speakers + ) + else: + top_lines = "\n".join( + f"{s.rank}.{s.name}【{s.count}】" for s in result.top_speakers + ) values = { "group_name": result.group_name, "period_start": result.period_start, "period_end": result.period_end, "speaker_count": str(result.speaker_count), "message_count": str(result.message_count), + "count_policy": result.count_policy, + "text_message_count": str(result.text_message_count), + "interaction_message_count": str(result.interaction_message_count), + "text_speaker_count": str(result.text_speaker_count), "top_limit": str(result.top_limit), "top_lines": top_lines, # 兼容已有自定义模板；变量名虽为 top10，内容仍以本次实际上限为准。 diff --git a/app/ranking/template_service.py b/app/ranking/template_service.py index 9916f4a..2506cf4 100644 --- a/app/ranking/template_service.py +++ b/app/ranking/template_service.py @@ -48,6 +48,10 @@ "period_end", "speaker_count", "message_count", + "count_policy", + "text_message_count", + "interaction_message_count", + "text_speaker_count", "top_limit", "top_lines", "top10_lines", diff --git a/app/scheduler/daily_v2_job.py b/app/scheduler/daily_v2_job.py index 81194f7..2f252cc 100644 --- a/app/scheduler/daily_v2_job.py +++ b/app/scheduler/daily_v2_job.py @@ -8,26 +8,42 @@ from __future__ import annotations import json -import os import subprocess import sys -import threading -from datetime import date, datetime +import uuid +from datetime import date, datetime, timedelta from pathlib import Path from zoneinfo import ZoneInfo from app.config.settings import PROJECT_ROOT, Settings, get_settings from app.core.logging import get_logger +from app.core.observability import log_event from app.db import repository as repo from app.pipeline.daily_pipeline import DailyPipeline, parse_date from app.services.generation_runtime import GenerationBusyError, generation_mutex +from app.services.email_service import email_delivery_config_error +from app.v2.constants import IMAGE_GENERATION_FAILED, SCHEDULER_STATE_CORRUPT +from app.v2.run_store import _atomic_write_text, _run_mutex +from app.scheduler.outcome import ProcessExitCode, attach_outcome, summarize_results +from app.scheduler.task_manifest import ( + build_expected_groups, + expected_group_ids, + manifest_fields, +) logger = get_logger("groupbrief.scheduler") -_STATE_LOCK = threading.RLock() # 兼容旧测试/调用名，底层已改为 V1/V2 共用锁。 _daily_mutex = generation_mutex +class ScheduleStateCorruptionError(RuntimeError): + """已有 scheduler 状态损坏；禁止用新任务状态覆盖。""" + + +class ScheduleStateVersionConflictError(RuntimeError): + """scheduler 状态版本与调用方确认的版本不一致。""" + + class DailyScheduleState: def __init__(self, output_root: Path | str): self.root = Path(output_root) / ".scheduler" @@ -39,25 +55,164 @@ def path(self, run_date: str) -> Path: def load(self, run_date: str) -> dict: path = self.path(run_date) + if not path.exists(): + return {"run_date": run_date} try: parsed = json.loads(path.read_text(encoding="utf-8")) - if isinstance(parsed, dict): - return parsed - except (OSError, json.JSONDecodeError): - pass - return {"run_date": run_date} + except (OSError, UnicodeError): + return self._corrupt_state(run_date, path, "read_failed") + except json.JSONDecodeError: + return self._corrupt_state(run_date, path, "json_invalid") + schema_error = self._schema_error(parsed, run_date) + if schema_error: + return self._corrupt_state(run_date, path, schema_error) + return parsed + + def _corrupt_state(self, run_date: str, path: Path, reason: str) -> dict: + return { + "run_date": run_date, + "state_status": "corrupt", + "error_type": SCHEDULER_STATE_CORRUPT, + "state_error_reason": reason, + "state_file": path.name, + "generation_hold": True, + "email_hold": True, + "needs_manual_review": True, + "detail": "调度状态文件损坏，已阻止自动补偿、生成和邮件发送", + } + + @staticmethod + def _schema_error(data: object, run_date: str) -> str | None: + if not isinstance(data, dict): + return "root_not_object" + if data.get("run_date") != run_date: + return "run_date_invalid" + timestamp_fields = ( + "generation_started_at", + "generation_completed_at", + "generation_invocation_completed_at", + "generation_resumed_at", + "generation_recovered_at", + "email_started_at", + "email_completed_at", + "last_invocation_completed_at", + "updated_at", + "owner_busy_at", + "next_retry_at", + "manifest_created_at", + ) + for field in timestamp_fields: + value = data.get(field) + if value is None: + continue + if not isinstance(value, str) or not value.strip(): + return f"{field}_invalid" + try: + datetime.fromisoformat(value.strip()) + except ValueError: + return f"{field}_invalid" + for field in ("generation_status", "email_status"): + value = data.get(field) + if value is not None and (not isinstance(value, str) or not value.strip()): + return f"{field}_invalid" + for field in ("generation_hold", "email_hold"): + value = data.get(field) + if value is not None and not isinstance(value, bool): + return f"{field}_invalid" + exit_code = data.get("last_invocation_exit_code") + if exit_code is not None and (isinstance(exit_code, bool) or not isinstance(exit_code, int)): + return "last_invocation_exit_code_invalid" + invocation_status = data.get("last_invocation_status") + if invocation_status is not None and ( + not isinstance(invocation_status, str) or not invocation_status.strip() + ): + return "last_invocation_status_invalid" + generation_results = data.get("generation_results") + if generation_results is not None and ( + not isinstance(generation_results, list) + or any(not isinstance(item, dict) for item in generation_results) + ): + return "generation_results_invalid" + expected_groups = data.get("expected_groups") + if expected_groups is not None: + if not isinstance(expected_groups, list) or any( + not isinstance(item, dict) + or not isinstance(item.get("group_id"), int) + or item.get("group_id", 0) <= 0 + for item in expected_groups + ): + return "expected_groups_invalid" + if data.get("manifest_version") != 1: + return "manifest_version_invalid" + if data.get("generation_started_at") and not data.get("generation_status"): + return "generation_status_missing" + if data.get("generation_completed_at") and not data.get("generation_status"): + return "generation_status_missing" + if data.get("email_started_at") and not data.get("email_status"): + return "email_status_missing" + if data.get("email_completed_at") and not data.get("email_status"): + return "email_status_missing" + if not any( + data.get(field) + for field in ( + "generation_started_at", + "generation_completed_at", + "email_started_at", + "email_completed_at", + "owner_busy_at", + "manifest_created_at", + ) + ): + return "lifecycle_marker_missing" + return None def update(self, run_date: str, **fields) -> dict: - with _STATE_LOCK: + path = self.path(run_date) + with _run_mutex(path): data = self.load(run_date) + if data.get("state_status") == "corrupt": + raise ScheduleStateCorruptionError("调度状态文件损坏，禁止自动覆盖") data.update(fields) data["run_date"] = run_date + data.setdefault("run_id", f"groupbrief:{run_date}:{uuid.uuid4().hex[:12]}") + data["state_version"] = int(data.get("state_version") or 0) + 1 data["updated_at"] = _now_iso() - path = self.path(run_date) path.parent.mkdir(parents=True, exist_ok=True) - temp = path.with_suffix(".json.tmp") - temp.write_text(json.dumps(data, ensure_ascii=False, indent=2), encoding="utf-8") - os.replace(temp, path) + _atomic_write_text( + path, + json.dumps(data, ensure_ascii=False, indent=2), + ) + return data + + def compare_and_update( + self, + run_date: str, + *, + expected_state_version: int, + **fields, + ) -> dict: + """在同一个文件锁内校验版本并原子更新，供显式恢复操作使用。""" + + path = self.path(run_date) + with _run_mutex(path): + data = self.load(run_date) + if data.get("state_status") == "corrupt": + raise ScheduleStateCorruptionError("调度状态文件损坏，禁止自动覆盖") + current_version = int(data.get("state_version") or 0) + if current_version != expected_state_version: + raise ScheduleStateVersionConflictError( + f"调度状态已变化：expected={expected_state_version} actual={current_version}" + ) + data.update(fields) + data["run_date"] = run_date + data.setdefault("run_id", f"groupbrief:{run_date}:{uuid.uuid4().hex[:12]}") + data["state_version"] = current_version + 1 + data["updated_at"] = _now_iso() + path.parent.mkdir(parents=True, exist_ok=True) + _atomic_write_text( + path, + json.dumps(data, ensure_ascii=False, indent=2), + ) return data @@ -72,25 +227,141 @@ def run_daily_v2_job( run_date = run_date or datetime.now(tz).date().isoformat() parsed_date = parse_date(run_date) if parsed_date is None: - return {"status": "failed", "error_type": "INVALID_RUN_DATE", "detail": "run_date 格式无效"} + return attach_outcome( + {"status": "failed", "error_type": "INVALID_RUN_DATE", "detail": "run_date 格式无效"} + ) try: with _daily_mutex(): - return _run_locked(settings, parsed_date, skip_email=skip_email) + result = _run_locked(settings, parsed_date, skip_email=skip_email) except GenerationBusyError as exc: logger.info("V2 每日任务未领取：%s", exc) - return {"status": "already_running", "detail": str(exc)} + result = { + "status": "already_running", + "error_type": "GENERATION_OWNER_BUSY", + "retryable": True, + "detail": str(exc), + } except Exception as exc: logger.exception("V2 每日任务异常") - return {"status": "failed", "detail": str(exc)[:300]} + result = {"status": "failed", "detail": str(exc)[:300]} + return _finalize_invocation(settings, parsed_date.isoformat(), result) + + +def _finalize_invocation(settings: Settings, run_date: str, result: dict) -> dict: + finalized = attach_outcome(result) + logger.info( + "V2 每日任务终态：run_date=%s source_status=%s outcome=%s exit_code=%d", + run_date, + finalized.get("status"), + finalized["outcome_status"], + finalized["exit_code"], + ) + state_snapshot = DailyScheduleState(settings.output_dir).load(run_date) + log_event( + logger, + "DAILY_INVOCATION_FINISHED", + run_id=state_snapshot.get("run_id"), + run_date=run_date, + stage="DAILY", + status=finalized.get("outcome_status"), + response_code=finalized.get("exit_code"), + error_type=finalized.get("error_type", ""), + error_summary=finalized.get("detail", ""), + ) + state_store = DailyScheduleState(settings.output_dir) + path = state_store.path(run_date) + if not path.is_file() and finalized["outcome_status"] != "already_running": + return finalized + state = state_store.load(run_date) + if state.get("state_status") == "corrupt": + return finalized + fields = { + "last_invocation_source_status": str(finalized.get("status") or ""), + "last_invocation_status": finalized["outcome_status"], + "last_invocation_exit_code": finalized["exit_code"], + "last_invocation_completed_at": _now_iso(), + "next_retry_at": None, + } + if finalized["outcome_status"] == "already_running": + busy_count = int(state.get("owner_busy_count") or 0) + 1 + now = datetime.now().astimezone() + fields.update( + owner_busy_at=now.isoformat(), + owner_busy_count=busy_count, + next_retry_at=(now.replace(microsecond=0) + timedelta(minutes=5)).isoformat(), + ) + state_store.update(run_date, **fields) + return finalized + + +def ensure_daily_manifest( + settings: Settings, + run_date: str, + *, + pipeline: DailyPipeline | None = None, + state_store: DailyScheduleState | None = None, + state: dict | None = None, +) -> dict: + """为 48 小时活动窗口惰性补齐任务清单，不执行生成或发送。""" + parsed = parse_date(run_date) + if parsed is None: + raise ValueError("run_date 必须是有效的 YYYY-MM-DD 日期") + state_store = state_store or DailyScheduleState(settings.output_dir) + state = state or state_store.load(run_date) + if state.get("state_status") == "corrupt": + return state + if isinstance(state.get("expected_groups"), list): + return state + if pipeline is None: + repo.init_db(settings) + repo.apply_db_settings(settings) + pipeline = DailyPipeline(settings=settings) + loader = getattr(pipeline, "_load_groups", None) + resolver = getattr(pipeline, "period_resolver", None) + if not callable(loader) or resolver is None: + return state + expected = build_expected_groups( + loader(), + parsed, + timezone=settings.app_timezone, + schedule_send_time=settings.schedule_send_time, + resolver=resolver, + ) + manifest = manifest_fields(expected) + if state.get("generation_completed_at"): + # 部署新清单合同前已经完成的活动窗口只补状态投影。 + manifest["manifest_source"] = "legacy_current_config_compat" + return state_store.update(run_date, **manifest) def _run_locked(settings: Settings, run_date: date, *, skip_email: bool) -> dict: run_date_text = run_date.isoformat() state_store = DailyScheduleState(settings.output_dir) state = state_store.load(run_date_text) + if state.get("state_status") == "corrupt": + logger.error("V2 每日任务已阻断：run_date=%s scheduler state corrupt", run_date_text) + return { + "status": "blocked", + "run_date": run_date_text, + "error_type": SCHEDULER_STATE_CORRUPT, + "detail": "调度状态文件损坏，需人工复核", + } repo.init_db(settings) repo.apply_db_settings(settings) + pipeline = DailyPipeline(settings=settings) + state = ensure_daily_manifest( + settings, + run_date_text, + pipeline=pipeline, + state_store=state_store, + state=state, + ) + manifest_ids = ( + expected_group_ids(state) + if isinstance(state.get("expected_groups"), list) + else None + ) generation_results = state.get("generation_results") or [] if not state.get("generation_completed_at"): @@ -121,9 +392,31 @@ def _run_locked(settings: Settings, run_date: date, *, skip_email: bool) -> dict generation_error="", ) try: - generation_results = DailyPipeline(settings=settings).generate_all( - run_date=run_date_text, acquire_lock=False - ) + if manifest_ids is None: + # 兼容显式测试替身与上一版注入点；生产 DailyPipeline 必有任务清单。 + generation_results = pipeline.generate_all( + run_date=run_date_text, + acquire_lock=False, + ) + elif manifest_ids: + generation_results = pipeline.generate_all( + run_date=run_date_text, + group_ids=manifest_ids, + group_overrides={ + int(row["group_id"]): row + for row in state.get("expected_groups", []) + if isinstance(row, dict) + and isinstance(row.get("group_id"), int) + }, + acquire_lock=False, + ) + else: + generation_results = [ + { + "status": "no_groups", + "reason": "当日没有符合群级统计规则的任务", + } + ] except Exception as exc: state_store.update( run_date_text, @@ -131,18 +424,39 @@ def _run_locked(settings: Settings, run_date: date, *, skip_email: bool) -> dict generation_hold=True, generation_error=str(exc)[:300], ) + writer = getattr(pipeline, "_write_runtime_status_safe", None) + if callable(writer): + writer([run_date_text]) raise generation_status = _generation_status(generation_results) - state = state_store.update( - run_date_text, - generation_completed_at=_now_iso(), - generation_status=generation_status, - generation_results=_compact_results(generation_results), - generation_hold=False, - generation_error="", - ) + completion_fields = { + "generation_invocation_completed_at": _now_iso(), + "generation_status": generation_status, + "generation_results": _compact_results(generation_results), + "generation_hold": generation_status in {"blocked", "failed", "partial"}, + "generation_error": "", + } + if _generation_results_terminal(generation_results): + completion_fields["generation_completed_at"] = _now_iso() + state = state_store.update(run_date_text, **completion_fields) + writer = getattr(pipeline, "_write_runtime_status_safe", None) + if callable(writer): + writer([run_date_text]) else: state = state_store.load(run_date_text) + try: + state = _reconcile_completed_generation( + settings, + run_date_text, + state_store, + state, + ) + except Exception: + logger.exception( + "已完成生成批次的可信图片对账失败，保留原终态：run_date=%s", + run_date_text, + ) + generation_results = state.get("generation_results") or generation_results if skip_email: return { @@ -152,14 +466,83 @@ def _run_locked(settings: Settings, run_date: date, *, skip_email: bool) -> dict "email_status": "skipped_by_request", } + generation_status = str(state.get("generation_status") or "failed") + if generation_status in {"failed", "blocked", "not_run"}: + state_store.update( + run_date_text, + email_status="skipped_generation_not_successful", + email_completed_at=_now_iso(), + email_detail=f"生成终态为 {generation_status}，未调用邮件", + ) + return { + "status": generation_status, + "run_date": run_date_text, + "generation_status": generation_status, + "email_status": "skipped_generation_not_successful", + } + if generation_status == "partial" and not settings.email_send_partial_report: + state_store.update( + run_date_text, + email_status="skipped_partial_disabled", + email_completed_at=_now_iso(), + email_detail="生成部分成功且未启用部分报告邮件", + ) + return { + "status": "partial", + "run_date": run_date_text, + "generation_status": "partial", + "email_status": "skipped_partial_disabled", + } + + if ( + state.get("email_recovery_required") + and state.get("email_completed_at") + and state.get("email_status") != "unknown" + ): + email_history = list(state.get("email_history") or []) + email_history.append( + { + "status": str(state.get("email_status") or ""), + "completed_at": str(state.get("email_completed_at") or ""), + "detail": str(state.get("email_detail") or "")[-800:], + } + ) + state = state_store.update( + run_date_text, + email_history=email_history[-10:], + email_status="recovery_pending", + email_started_at=None, + email_completed_at=None, + email_hold=False, + email_error="", + email_detail="可信生图恢复完成；逐群邮件账本将只提交尚未确认发送的群", + ) + if state.get("email_completed_at"): + if state.get("email_status") == "unknown": + return { + "status": "blocked", + "run_date": run_date_text, + "generation_status": generation_status, + "email_status": "unknown", + "error_type": "EMAIL_RESULT_UNKNOWN", + "detail": "邮件发送结果未知，需人工检查", + } + if state.get("email_status") in {"partial", "failed", "failed_before_submit"}: + return { + "status": "partial", + "run_date": run_date_text, + "generation_status": generation_status, + "email_status": state.get("email_status"), + } + completed_status = generation_status if generation_status != "success" else "already_completed" return { - "status": "already_completed", + "status": completed_status, "run_date": run_date_text, - "generation_status": state.get("generation_status"), + "generation_status": generation_status, "email_status": state.get("email_status"), } - if not settings.email_enabled or not settings.email_smtp_host: + if not settings.email_enabled: state_store.update( run_date_text, email_status="skipped_disabled", @@ -167,10 +550,27 @@ def _run_locked(settings: Settings, run_date: date, *, skip_email: bool) -> dict email_detail="邮件未启用或 SMTP 未配置", ) return { - "status": state.get("generation_status", "completed"), + "status": generation_status, "run_date": run_date_text, "email_status": "skipped_disabled", } + email_config_error = email_delivery_config_error(settings) + if email_config_error: + state_store.update( + run_date_text, + email_status="failed_config", + email_completed_at=_now_iso(), + email_error=email_config_error, + email_detail="邮件配置无效，未启动发送子进程", + ) + return { + "status": "partial", + "run_date": run_date_text, + "generation_status": generation_status, + "email_status": "failed_config", + "error_type": "EMAIL_PROVIDER_CONFIG_INVALID", + "detail": email_config_error, + } if state.get("email_started_at"): state_store.update( run_date_text, @@ -197,6 +597,10 @@ def _run_locked(settings: Settings, run_date: date, *, skip_email: bool) -> dict "--run-date", run_date_text, ] + if state.get("email_recovery_required"): + for group_name in state.get("generation_recovery_groups") or []: + if isinstance(group_name, str) and group_name.strip(): + command.extend(["--group", group_name.strip()]) try: proc = subprocess.run( command, @@ -229,31 +633,171 @@ def _run_locked(settings: Settings, run_date: date, *, skip_email: bool) -> dict return {"status": "failed", "detail": str(exc)[:300]} output_tail = ((proc.stdout or "") + "\n" + (proc.stderr or ""))[-800:] - email_status = "sent" if proc.returncode == 0 else "failed" + if proc.returncode == int(ProcessExitCode.SUCCESS): + email_status = "sent" + result_status = generation_status + email_hold = False + elif proc.returncode == int(ProcessExitCode.PARTIAL): + email_status = "partial" + result_status = "partial" + email_hold = False + elif proc.returncode == int(ProcessExitCode.BLOCKED): + email_status = "unknown" + result_status = "blocked" + email_hold = True + else: + email_status = "failed_before_submit" + result_status = "partial" + email_hold = False state_store.update( run_date_text, email_status=email_status, email_completed_at=_now_iso(), email_exit_code=proc.returncode, email_detail=output_tail, + email_hold=email_hold, + email_error=( + "逐群邮件账本存在结果未知项，禁止自动重复发送" + if email_status == "unknown" + else "" + ), + email_recovery_required=False, + email_recovered_at=( + _now_iso() if state.get("email_recovery_required") else state.get("email_recovered_at") + ), ) - return { - "status": "success" if proc.returncode == 0 else "partial", + result = { + "status": result_status, "run_date": run_date_text, - "generation_status": state.get("generation_status"), + "generation_status": generation_status, "email_status": email_status, } + if email_status == "unknown": + result.update( + error_type="EMAIL_RESULT_UNKNOWN", + detail="逐群邮件账本存在结果未知项，需人工核对", + ) + return result def _generation_status(results: list[dict]) -> str: - statuses = {str(item.get("status") or "") for item in results} - if statuses and statuses <= {"ready_to_send", "skipped", "no_groups"}: - return "success" - if "failed" in statuses and len(statuses) == 1: - return "failed" - if "failed" in statuses: - return "partial" - return "success" + return str(summarize_results(results)["outcome_status"]) + + +def _generation_results_terminal(results: list[dict]) -> bool: + """批次内所有群都已成功或进入明确人工/最终终态时才封存批次。""" + if not results: + return False + terminal_statuses = { + "success", + "ready_to_send", + "already_completed", + "skipped", + "no_groups", + "held", + "blocked", + "failed_final", + } + return all(str(item.get("status") or "").lower() in terminal_statuses for item in results) + + +def _reconcile_completed_generation( + settings: Settings, + run_date: str, + state_store: DailyScheduleState, + state: dict, +) -> dict: + """只对带可信 Codex thread_id 候选的失败群做无新调用收口。""" + if state.get("generation_status") != "partial": + return state + original_results = state.get("generation_results") + if not isinstance(original_results, list): + return state + + pipeline = DailyPipeline(settings=settings) + generator = pipeline.image_generator + can_reconcile = getattr(generator, "can_reconcile_without_generation", None) + if not callable(can_reconcile): + return state + + groups = pipeline._load_groups() + groups_by_name = { + (group.display_name or group.wechat_group_name): group for group in groups + } + recovery_ids: list[int] = [] + for result in original_results: + if not isinstance(result, dict): + continue + if result.get("error_type") != IMAGE_GENERATION_FAILED: + continue + group_name = str(result.get("group_name") or "") + group = groups_by_name.get(group_name) + if group is None or group.id is None: + continue + run = pipeline.store.load_run(group_name, run_date) + image_job = run.get("image_job") if isinstance(run.get("image_job"), dict) else {} + job_id = str(image_job.get("job_id") or "") + prompt_path = pipeline.store.prompt_path(group_name, run_date) + if prompt_path.is_file() and can_reconcile( + prompt_path, + job_id, + ): + recovery_ids.append(int(group.id)) + + if not recovery_ids: + return state + + recovery_results = pipeline.generate_all( + run_date=run_date, + group_ids=recovery_ids, + force=False, + acquire_lock=False, + ) + replacements = { + str(item.get("group_name") or ""): item + for item in recovery_results + if isinstance(item, dict) and item.get("group_name") + } + merged_results = [ + replacements.get(str(item.get("group_name") or ""), item) + if isinstance(item, dict) + else item + for item in original_results + ] + recovered_groups = sorted( + name + for name, item in replacements.items() + if str(item.get("status") or "") in {"ready_to_send", "success", "skipped"} + ) + history = list(state.get("generation_history") or []) + history.append( + { + "status": str(state.get("generation_status") or ""), + "completed_at": str(state.get("generation_completed_at") or ""), + "results": original_results, + } + ) + next_status = _generation_status(merged_results) + logger.info( + "V2 已完成批次可信图片对账：run_date=%s groups=%s status=%s results=%s", + run_date, + recovered_groups, + next_status, + _compact_results(recovery_results), + ) + return state_store.update( + run_date, + generation_original_status=( + state.get("generation_original_status") or state.get("generation_status") + ), + generation_history=history[-10:], + generation_status=next_status, + generation_results=_compact_results(merged_results), + generation_recovered_at=_now_iso() if recovered_groups else state.get("generation_recovered_at"), + generation_recovery_groups=recovered_groups, + generation_recovery_results=_compact_results(recovery_results), + email_recovery_required=bool(recovered_groups), + ) def _compact_results(results: list[dict]) -> list[dict]: @@ -262,7 +806,18 @@ def _compact_results(results: list[dict]) -> list[dict]: compact.append( { key: item.get(key) - for key in ("group_name", "status", "error_type", "detail", "reason") + for key in ( + "group_name", + "status", + "error_type", + "detail", + "reason", + "failed_stage", + "receipt_source", + "recovery_status", + "recovered_at", + "codex_thread_id", + ) if item.get(key) not in (None, "") } ) diff --git a/app/scheduler/email_job.py b/app/scheduler/email_job.py index e131774..6449f9e 100644 --- a/app/scheduler/email_job.py +++ b/app/scheduler/email_job.py @@ -4,17 +4,36 @@ from sqlmodel import Session +from app.config.settings import Settings, get_settings from app.core.logging import get_logger from app.db import repository as repo +from app.scheduler.outcome import attach_outcome from app.services.email_service import EmailService +from app.services.legacy_v1_policy import ( + LEGACY_V1_WRITE_BLOCKED, + LegacyV1WriteBlockedError, + require_legacy_v1_write, +) logger = get_logger("groupbrief.scheduler") -def run_email_job() -> dict: - from app.config.settings import get_settings - - settings = get_settings() +def run_email_job(settings: Settings | None = None) -> dict: + settings = settings or get_settings() + try: + require_legacy_v1_write( + settings, + operation="scheduler.email.send", + replacement="daily_v2_generate_email", + ) + except LegacyV1WriteBlockedError as exc: + return attach_outcome( + { + "status": "blocked", + "error_type": LEGACY_V1_WRITE_BLOCKED, + "detail": str(exc), + } + ) with Session(repo.engine) as session: service = EmailService(settings) ok, detail = service.send(session) diff --git a/app/scheduler/generate_job.py b/app/scheduler/generate_job.py index 8cdf33f..77fb51a 100644 --- a/app/scheduler/generate_job.py +++ b/app/scheduler/generate_job.py @@ -4,17 +4,39 @@ from sqlmodel import Session +from app.config.settings import Settings, get_settings from app.core.logging import get_logger from app.db import repository as repo +from app.scheduler.outcome import attach_outcome from app.services.report_service import ReportService from app.services.generation_runtime import GenerationBusyError +from app.services.legacy_v1_policy import ( + LEGACY_V1_WRITE_BLOCKED, + LegacyV1WriteBlockedError, + require_legacy_v1_write, +) logger = get_logger("groupbrief.scheduler") -def run_generate_job() -> dict: +def run_generate_job(settings: Settings | None = None) -> dict: + settings = settings or get_settings() + try: + require_legacy_v1_write( + settings, + operation="scheduler.report.generate", + replacement="daily_v2_generate_email", + ) + except LegacyV1WriteBlockedError as exc: + return attach_outcome( + { + "status": "blocked", + "error_type": LEGACY_V1_WRITE_BLOCKED, + "detail": str(exc), + } + ) with Session(repo.engine) as session: - service = ReportService() + service = ReportService(settings=settings) try: run = service.generate(session, trigger_type="auto") except GenerationBusyError as exc: diff --git a/app/scheduler/heartbeat.py b/app/scheduler/heartbeat.py new file mode 100644 index 0000000..9229286 --- /dev/null +++ b/app/scheduler/heartbeat.py @@ -0,0 +1,49 @@ +"""APScheduler owner 的持久化心跳。""" + +from __future__ import annotations + +import json +import os +from datetime import datetime +from pathlib import Path + +from app.config.settings import Settings + + +def heartbeat_path(settings: Settings) -> Path: + return settings.output_dir.parent / "runtime" / "scheduler-heartbeat.json" + + +def record_scheduler_heartbeat( + settings: Settings, + *, + job: str, + status: str, + detail: str = "", + now: datetime | None = None, +) -> dict: + now = now or datetime.now().astimezone() + payload = { + "schema_version": 1, + "owner": settings.scheduler_owner, + "pid": os.getpid(), + "last_beat_at": now.isoformat(), + "last_job": str(job), + "last_status": str(status), + "detail": str(detail)[:300], + } + path = heartbeat_path(settings) + path.parent.mkdir(parents=True, exist_ok=True) + temp = path.with_name(f".{path.name}.{os.getpid()}.tmp") + temp.write_text(json.dumps(payload, ensure_ascii=False, indent=2), encoding="utf-8") + os.replace(temp, path) + return payload + + +def load_scheduler_heartbeat(settings: Settings) -> dict: + path = heartbeat_path(settings) + try: + parsed = json.loads(path.read_text(encoding="utf-8")) + except (OSError, UnicodeError, json.JSONDecodeError): + return {} + return parsed if isinstance(parsed, dict) else {} diff --git a/app/scheduler/manager.py b/app/scheduler/manager.py index ecbcb22..ffeab55 100644 --- a/app/scheduler/manager.py +++ b/app/scheduler/manager.py @@ -1,4 +1,4 @@ -"""APScheduler 调度管理：每日唯一 V2 任务 + 分钟级微信发送。""" +"""APScheduler 调度管理：每日定点生成、定点串行发送与按需一次性恢复。""" from __future__ import annotations @@ -9,19 +9,33 @@ from apscheduler.triggers.date import DateTrigger from zoneinfo import ZoneInfo -from app.config.settings import Settings +from app.config.settings import Settings, get_settings from app.core.logging import get_logger from app.scheduler.daily_v2_job import DailyScheduleState, run_daily_v2_job +from app.scheduler.heartbeat import record_scheduler_heartbeat +from app.scheduler.outcome import require_scheduler_success, summarize_results +from app.scheduler.reliability_watchdog import recovery_dates, run_reliability_watchdog from app.scheduler.send_job import run_send_due_job +from app.v2.constants import EXECUTION_WAIT_RETRY, IMAGE_READY, READY_TO_SEND +from app.v2.run_store import RunStore +from app.weekly.service import WeeklyInsightsService logger = get_logger("groupbrief.scheduler") _scheduler: BackgroundScheduler | None = None _DEFAULT_GENERATE_TIME = time(0, 15) +_DEFAULT_SEND_TIME = time(8, 30) -def _parse_generate_time(value: str) -> time: - """解析每日生成时间；无效配置安全回退到 00:15。""" +def _normalize_now(settings: Settings, now: datetime | None = None) -> datetime: + tz = ZoneInfo(settings.app_timezone) + value = now or datetime.now(tz) + if value.tzinfo is None: + return value.replace(tzinfo=tz) + return value.astimezone(tz) + + +def _parse_clock(value: str, *, fallback: time, field_name: str) -> time: text = str(value or "").strip() try: parsed = datetime.strptime(text, "%H:%M").time() @@ -29,8 +43,303 @@ def _parse_generate_time(value: str) -> time: raise ValueError("时间必须使用 HH:MM 格式") return parsed except (TypeError, ValueError): - logger.warning("无效的 schedule_generate_time=%r，已回退到 00:15", value) - return _DEFAULT_GENERATE_TIME + logger.warning( + "无效的 %s=%r，已回退到 %s", + field_name, + value, + fallback.strftime("%H:%M"), + ) + return fallback + + +def _parse_generate_time(value: str) -> time: + return _parse_clock( + value, + fallback=_DEFAULT_GENERATE_TIME, + field_name="schedule_generate_time", + ) + + +def _parse_send_time(value: str) -> time: + return _parse_clock( + value, + fallback=_DEFAULT_SEND_TIME, + field_name="schedule_send_time", + ) + + +def _parse_weekly_time(value: str) -> time: + return _parse_clock( + value, + fallback=time(7, 45), + field_name="weekly_generate_time", + ) + + +def _timestamp(value: object, *, now: datetime) -> datetime | None: + if not isinstance(value, str) or not value.strip(): + return None + try: + parsed = datetime.fromisoformat(value.strip()) + except ValueError: + return None + if parsed.tzinfo is None: + parsed = parsed.replace(tzinfo=now.tzinfo) + return parsed.astimezone(now.tzinfo) + + +def _add_one_shot( + scheduler: BackgroundScheduler, + *, + job_id: str, + name: str, + func, + run_at: datetime, + args: list | None = None, + kwargs: dict | None = None, +) -> None: + scheduler.add_job( + func, + trigger=DateTrigger(run_date=run_at, timezone=run_at.tzinfo), + args=args or [], + kwargs=kwargs or {}, + id=job_id, + name=name, + replace_existing=True, + misfire_grace_time=300, + max_instances=1, + ) + + +def _schedule_on_demand_jobs( + scheduler: BackgroundScheduler | None, + settings: Settings, + *, + now: datetime | None = None, + run_dates: list[str] | None = None, + include_newly_ready_send: bool = True, +) -> list[str]: + """从持久化状态重建有限的一次性任务；不进行任何外部调用。""" + if scheduler is None: + return [] + now = _normalize_now(settings, now) + today = now.date().isoformat() + allowed_dates = recovery_dates(now, settings.reliability_lookback_days) + selected_dates = sorted(set(run_dates or allowed_dates) & set(allowed_dates)) + state_store = DailyScheduleState(settings.output_dir) + run_store = RunStore(settings.output_dir) + scheduled: list[str] = [] + + for run_date in selected_dates: + state = state_store.load(run_date) + if state.get("state_status") == "corrupt" or state.get("generation_completed_at"): + continue + retry_times: list[datetime] = [] + state_retry = _timestamp(state.get("next_retry_at"), now=now) + if state_retry is not None: + retry_times.append(state_retry) + for run in run_store.list_runs(run_date): + if str(run.get("execution_state") or "") != EXECUTION_WAIT_RETRY: + continue + retry_at = _timestamp(run.get("next_retry_at"), now=now) + if retry_at is not None: + retry_times.append(retry_at) + if not retry_times: + continue + run_at = max(min(retry_times), now + timedelta(seconds=1)) + job_id = f"daily_v2_retry_{run_date.replace('-', '')}" + _add_one_shot( + scheduler, + job_id=job_id, + name=f"DailyV2Retry:{run_date}", + func=run_scheduled_daily_v2_job, + run_at=run_at, + args=[run_date], + kwargs={"skip_email": True}, + ) + scheduled.append(job_id) + + if today not in selected_dates: + return scheduled + + send_clock = _parse_send_time(settings.schedule_send_time) + due_at = datetime.combine(now.date(), send_clock, tzinfo=now.tzinfo) + if now < due_at: + return scheduled + cutoff = due_at + timedelta( + minutes=max(int(settings.wechat_late_send_window_minutes), 0) + ) + ready_runs = [ + run + for run in run_store.list_runs(today) + if run.get("status") in {IMAGE_READY, READY_TO_SEND} + and bool(run.get("wechat_send_enabled")) + and not run.get("sent_at") + and not run.get("send_hold") + ] + if not ready_runs: + return scheduled + + retry_times: list[datetime] = [] + has_immediate = False + for run in ready_runs: + retry_at = _timestamp(run.get("send_next_retry_at"), now=now) + if retry_at is None: + has_immediate = has_immediate or include_newly_ready_send + elif retry_at <= now: + has_immediate = True + else: + retry_times.append(retry_at) + if not has_immediate and not retry_times: + return scheduled + if now > cutoff or has_immediate: + run_at = now + timedelta(seconds=1) + else: + run_at = min(retry_times) + if run_at > cutoff: + run_at = cutoff + timedelta(seconds=1) + + job_id = f"daily_send_once_{today.replace('-', '')}" + _add_one_shot( + scheduler, + job_id=job_id, + name=f"DailySendOnce:{today}", + func=run_scheduled_send_batch, + run_at=run_at, + args=[today], + ) + scheduled.append(job_id) + return scheduled + + +def run_scheduled_daily_v2_job( + run_date: str | None = None, + *, + skip_email: bool = False, +) -> dict: + """APScheduler 生成包装器：完成后仅按持久化状态安排一次性后续任务。""" + settings = get_settings() + record_scheduler_heartbeat(settings, job="daily_v2", status="started") + try: + result = run_daily_v2_job(run_date, settings=settings, skip_email=skip_email) + except Exception as exc: + record_scheduler_heartbeat( + settings, + job="daily_v2", + status="error", + detail=f"{type(exc).__name__}: {exc}", + ) + raise + record_scheduler_heartbeat( + settings, + job="daily_v2", + status=str(result.get("outcome_status") or result.get("status") or "unknown"), + ) + logger.info( + "APScheduler 每日任务结果：status=%s outcome=%s exit_code=%s", + result.get("status"), + result.get("outcome_status"), + result.get("exit_code"), + ) + target_date = run_date or _normalize_now(settings).date().isoformat() + _schedule_on_demand_jobs( + _scheduler, + settings, + run_dates=[target_date], + ) + require_scheduler_success(result) + return result + + +def run_scheduled_send_batch(run_date: str | None = None) -> dict: + """08:30 核心批次和按需补偿共用的串行发送入口。""" + settings = get_settings() + now = _normalize_now(settings) + target_date = run_date or now.date().isoformat() + try: + return run_send_due_job( + settings=settings, + now=now, + run_date=target_date, + ) + finally: + _schedule_on_demand_jobs( + _scheduler, + settings, + now=now, + run_dates=[target_date], + include_newly_ready_send=False, + ) + + +def run_scheduled_startup_recovery() -> dict: + """进程启动后只执行一次恢复检查，并重建尚未执行的一次性任务。""" + settings = get_settings() + now = _normalize_now(settings) + record_scheduler_heartbeat(settings, job="startup_recovery", status="started") + try: + result = run_reliability_watchdog(settings=settings, now=now) + except Exception as exc: + record_scheduler_heartbeat( + settings, + job="startup_recovery", + status="error", + detail=f"{type(exc).__name__}: {exc}", + ) + raise + _schedule_on_demand_jobs( + _scheduler, + settings, + now=now, + run_dates=recovery_dates(now, settings.reliability_lookback_days), + include_newly_ready_send=False, + ) + record_scheduler_heartbeat( + settings, + job="startup_recovery", + status=str(result.get("status") or "unknown"), + ) + return result + + +def run_scheduled_weekly_insights() -> dict: + """周一独立生成上一自然周归档；不读取原始聊天、不发送。""" + settings = get_settings() + record_scheduler_heartbeat(settings, job="weekly_insights", status="started") + result = WeeklyInsightsService(settings).generate_previous_week() + record_scheduler_heartbeat( + settings, + job="weekly_insights", + status=str(result.get("status") or "unknown"), + ) + return result + + +def run_scheduled_weekly_send() -> dict: + """可选周报发送独立运行，不再参与日报批次或空闲日志。""" + settings = get_settings() + results = WeeklyInsightsService(settings).send_due() + outcome = summarize_results(results) + require_scheduler_success(outcome, allow_not_run=True) + return outcome + + +def _schedule_startup_recovery( + scheduler: BackgroundScheduler, + settings: Settings, + now: datetime | None = None, +) -> bool: + if not settings.reliability_watchdog_enabled: + return False + now = _normalize_now(settings, now) + _add_one_shot( + scheduler, + job_id="startup_recovery", + name="StartupRecovery", + func=run_scheduled_startup_recovery, + run_at=now + timedelta(seconds=5), + ) + return True def start_scheduler(settings: Settings) -> BackgroundScheduler: @@ -40,12 +349,14 @@ def start_scheduler(settings: Settings) -> BackgroundScheduler: tz = ZoneInfo(settings.app_timezone) generate_time = _parse_generate_time(settings.schedule_generate_time) + send_time = _parse_send_time(settings.schedule_send_time) scheduler = BackgroundScheduler(timezone=tz) scheduler.add_job( - run_daily_v2_job, + run_scheduled_daily_v2_job, trigger=CronTrigger( hour=generate_time.hour, minute=generate_time.minute, + second=0, timezone=tz, ), id="daily_v2_generate_email", @@ -55,20 +366,65 @@ def start_scheduler(settings: Settings) -> BackgroundScheduler: max_instances=1, ) scheduler.add_job( - run_send_due_job, - trigger=CronTrigger(minute="*", second=15, timezone=tz), - id="send_wechat_due", - name="SendWechatDue", - misfire_grace_time=45, + run_scheduled_send_batch, + trigger=CronTrigger( + hour=send_time.hour, + minute=send_time.minute, + second=0, + timezone=tz, + ), + id="daily_wechat_send_batch", + name="DailyWechatSendBatch", + misfire_grace_time=max(int(settings.wechat_late_send_window_minutes) * 60, 60), coalesce=True, max_instances=1, ) + if settings.weekly_insights_enabled: + weekly_time = _parse_weekly_time(settings.weekly_generate_time) + scheduler.add_job( + run_scheduled_weekly_insights, + trigger=CronTrigger( + day_of_week="mon", + hour=weekly_time.hour, + minute=weekly_time.minute, + second=0, + timezone=tz, + ), + id="weekly_insights_generate", + name="WeeklyInsightsGenerate", + misfire_grace_time=1800, + coalesce=True, + max_instances=1, + ) + if settings.weekly_send_enabled: + weekly_send_time = _parse_clock( + settings.weekly_send_time, + fallback=_DEFAULT_SEND_TIME, + field_name="weekly_send_time", + ) + scheduler.add_job( + run_scheduled_weekly_send, + trigger=CronTrigger( + day_of_week="mon", + hour=weekly_send_time.hour, + minute=weekly_send_time.minute, + second=0, + timezone=tz, + ), + id="weekly_insights_send", + name="WeeklyInsightsSend", + misfire_grace_time=1800, + coalesce=True, + max_instances=1, + ) scheduler.start() - _schedule_startup_catchup(scheduler, settings) _scheduler = scheduler + record_scheduler_heartbeat(settings, job="scheduler", status="started") + _schedule_startup_recovery(scheduler, settings) logger.info( - "调度已启动：%s V2 生成+邮件，微信每分钟第 15 秒 send_due（时区 %s）", + "调度已启动：每日 %s 生成，%s 微信串行发送批次（时区 %s）", generate_time.strftime("%H:%M"), + send_time.strftime("%H:%M"), settings.app_timezone, ) logger.info( @@ -92,37 +448,3 @@ def stop_scheduler() -> None: def get_scheduler() -> BackgroundScheduler | None: return _scheduler - - -def _schedule_startup_catchup( - scheduler: BackgroundScheduler, - settings: Settings, - now: datetime | None = None, -) -> bool: - """每日生成时间后若当天没有完成标记，安排一次启动补偿。""" - if not settings.schedule_startup_catchup_enabled: - return False - tz = ZoneInfo(settings.app_timezone) - now = now or datetime.now(tz) - if now.tzinfo is None: - now = now.replace(tzinfo=tz) - generate_time = _parse_generate_time(settings.schedule_generate_time) - if now.time() < generate_time: - return False - run_date = now.date().isoformat() - state = DailyScheduleState(settings.output_dir).load(run_date) - if state.get("generation_completed_at"): - return False - scheduler.add_job( - run_daily_v2_job, - trigger=DateTrigger(run_date=now + timedelta(seconds=3), timezone=tz), - args=[run_date], - kwargs={"skip_email": True}, - id="daily_v2_startup_catchup", - name="DailyV2StartupCatchup", - replace_existing=True, - misfire_grace_time=300, - max_instances=1, - ) - logger.info("已安排 V2 启动补偿：run_date=%s", run_date) - return True diff --git a/app/scheduler/outcome.py b/app/scheduler/outcome.py new file mode 100644 index 0000000..7f9d22b --- /dev/null +++ b/app/scheduler/outcome.py @@ -0,0 +1,142 @@ +"""调度与命令行共用的业务终态和退出码契约。""" + +from __future__ import annotations + +from enum import IntEnum +from typing import Iterable, Mapping + + +class ProcessExitCode(IntEnum): + SUCCESS = 0 + FAILED = 1 + PARTIAL = 2 + BLOCKED = 3 + ALREADY_RUNNING = 4 + NOT_RUN = 5 + + +_SUCCESS = frozenset( + { + "success", + "sent", + "ready_to_send", + "already_completed", + "skipped", + "skipped_disabled", + "skipped_by_request", + } +) +_FAILED = frozenset({"failed", "error"}) +_PARTIAL = frozenset({"partial"}) +_BLOCKED = frozenset( + { + "blocked", + "held", + "unknown", + "result_unknown", + "retry_scheduled", + "failed_final", + } +) +_ALREADY_RUNNING = frozenset({"already_running"}) +_NOT_RUN = frozenset({"not_run", "no_groups", "no_work", "not_due", "empty"}) + +_EXIT_CODES = { + "success": ProcessExitCode.SUCCESS, + "failed": ProcessExitCode.FAILED, + "partial": ProcessExitCode.PARTIAL, + "blocked": ProcessExitCode.BLOCKED, + "already_running": ProcessExitCode.ALREADY_RUNNING, + "not_run": ProcessExitCode.NOT_RUN, +} + + +class SchedulerOutcomeError(RuntimeError): + """业务终态不是可信成功，必须让 APScheduler 记录本次执行失败。""" + + def __init__(self, outcome: Mapping[str, object]): + self.outcome = dict(outcome) + super().__init__( + "scheduler outcome=" + f"{self.outcome.get('outcome_status', 'failed')} " + f"exit_code={self.outcome.get('exit_code', int(ProcessExitCode.FAILED))}" + ) + + +def normalize_status(value: object) -> str: + """把业务状态收敛成六种稳定终态；未知值一律 fail closed。""" + status = str(value or "").strip().lower() + if status in _SUCCESS: + return "success" + if status in _FAILED: + return "failed" + if status in _PARTIAL: + return "partial" + if status in _BLOCKED: + return "blocked" + if status in _ALREADY_RUNNING: + return "already_running" + if status in _NOT_RUN: + return "not_run" + return "failed" + + +def outcome_for_status(value: object) -> dict: + outcome_status = normalize_status(value) + return { + "outcome_status": outcome_status, + "exit_code": int(_EXIT_CODES[outcome_status]), + } + + +def attach_outcome(payload: Mapping[str, object]) -> dict: + result = dict(payload) + result.update(outcome_for_status(result.get("status"))) + return result + + +def summarize_results(results: Iterable[Mapping[str, object]]) -> dict: + rows = [dict(item) for item in results] + if not rows: + summary = outcome_for_status("not_run") + summary.update({"result_count": 0, "source_statuses": []}) + return summary + + normalized = [normalize_status(item.get("status")) for item in rows] + kinds = set(normalized) + if "blocked" in kinds: + outcome_status = "blocked" + elif "already_running" in kinds: + outcome_status = "already_running" if kinds <= {"already_running", "not_run"} else "partial" + elif "partial" in kinds: + outcome_status = "partial" + elif "failed" in kinds: + outcome_status = "failed" if kinds == {"failed"} else "partial" + elif kinds == {"not_run"}: + outcome_status = "not_run" + elif "not_run" in kinds: + outcome_status = "partial" + else: + outcome_status = "success" + + summary = outcome_for_status(outcome_status) + summary.update( + { + "result_count": len(rows), + "source_statuses": sorted({str(item.get("status") or "") for item in rows}), + } + ) + return summary + + +def require_scheduler_success( + outcome: Mapping[str, object], + *, + allow_not_run: bool = False, +) -> None: + exit_code = int(outcome.get("exit_code", ProcessExitCode.FAILED)) + if exit_code == ProcessExitCode.SUCCESS: + return + if allow_not_run and exit_code == ProcessExitCode.NOT_RUN: + return + raise SchedulerOutcomeError(outcome) diff --git a/app/scheduler/period.py b/app/scheduler/period.py index 48dda47..5e5133a 100644 --- a/app/scheduler/period.py +++ b/app/scheduler/period.py @@ -1,4 +1,4 @@ -"""V2 每日前一自然日统计周期解析器。""" +"""V2 群级统计周期解析器。""" from __future__ import annotations @@ -19,8 +19,8 @@ class PeriodWindow: period_end: datetime # 统计终点（含） should_run: bool # 今天是否生成 weekday: int # 0=周一 ... 6=周日 - rule: str = "weekday_default" - covered_dates: list[date] | None = None # 覆盖的自然日（每日固定一天） + rule: str = "daily_previous_day" + covered_dates: list[date] | None = None def period_start_str(self) -> str: return self.period_start.strftime("%Y-%m-%d %H:%M:%S") @@ -36,24 +36,35 @@ def resolve( self, run_date: date | None = None, timezone: str = "Asia/Shanghai", - schedule_rule: str = "weekday_default", + schedule_rule: str = "daily_previous_day", ) -> PeriodWindow: tz = ZoneInfo(timezone) today = run_date or datetime.now(tz).date() weekday = today.weekday() - if schedule_rule != "weekday_default": - # 预留扩展：其他周期规则在此注册 + if schedule_rule == "daily_previous_day": + targets = [today - timedelta(days=1)] + should_run = True + elif schedule_rule == "weekday_default": + if weekday >= 5: + targets = [today - timedelta(days=1)] + should_run = False + elif weekday == 0: + targets = [today - timedelta(days=offset) for offset in (3, 2, 1)] + else: + targets = [today - timedelta(days=1)] + should_run = weekday < 5 + else: raise NotImplementedError(f"暂不支持的统计周期规则：{schedule_rule}") - target = today - timedelta(days=1) return PeriodWindow( run_date=today, - period_start=datetime.combine(target, time.min), - period_end=datetime.combine(target, _END_OF_DAY), - should_run=True, + period_start=datetime.combine(min(targets), time.min), + period_end=datetime.combine(max(targets), _END_OF_DAY), + should_run=should_run, weekday=weekday, - covered_dates=[target], + rule=schedule_rule, + covered_dates=targets, ) def format_dt(self, dt: datetime) -> str: diff --git a/app/scheduler/recovery_planner.py b/app/scheduler/recovery_planner.py new file mode 100644 index 0000000..02b3e37 --- /dev/null +++ b/app/scheduler/recovery_planner.py @@ -0,0 +1,395 @@ +"""历史恢复预览与显式生成确认。""" + +from __future__ import annotations + +import hashlib +import json +from datetime import datetime, timedelta +from zoneinfo import ZoneInfo + +from app.config.settings import Settings +from app.db import repository as repo +from app.pipeline.daily_pipeline import DailyPipeline +from app.scheduler.daily_v2_job import ( + DailyScheduleState, + ScheduleStateVersionConflictError, + _compact_results, + _generation_results_terminal, + _generation_status, +) +from app.scheduler.reliability_watchdog import recovery_dates +from app.scheduler.runtime_status import write_daily_status +from app.scheduler.task_manifest import build_expected_groups, manifest_fields +from app.services.generation_runtime import generation_mutex +from app.v2.constants import IMAGE_READY, READY_TO_SEND, SENT +from app.v2.run_store import RunStore, validate_run_date + +_TERMINAL_GENERATION_STATUSES = {IMAGE_READY, READY_TO_SEND, SENT} + + +class RecoveryPlanChangedError(RuntimeError): + """确认前 backlog 已改变。""" + + +class RecoverySelectionError(ValueError): + """选择的任务不允许自动恢复。""" + + +def _is_fail_closed(run: dict) -> bool: + values = ( + run.get("execution_state"), + run.get("error_type"), + run.get("last_error_type"), + run.get("send_hold_reason"), + run.get("prompt_hold_reason"), + ) + text = "|".join(str(value or "").upper() for value in values) + return "UNKNOWN" in text or "CORRUPT" in text + + +def _version_for(items: list[dict]) -> str: + payload = json.dumps(items, ensure_ascii=False, sort_keys=True, separators=(",", ":")) + return hashlib.sha256(payload.encode("utf-8")).hexdigest() + + +class RecoveryPlanner: + def __init__( + self, + settings: Settings, + *, + store: RunStore | None = None, + state_store: DailyScheduleState | None = None, + ): + self.settings = settings + self.store = store or RunStore(settings.output_dir) + self.state_store = state_store or DailyScheduleState(self.store.root) + + def preview( + self, + *, + now: datetime | None = None, + lookback_days: int = 30, + ) -> dict: + tz = ZoneInfo(self.settings.app_timezone) + now = now or datetime.now(tz) + if now.tzinfo is None: + now = now.replace(tzinfo=tz) + days = min(max(int(lookback_days), 3), 30) + start = now.date() - timedelta(days=days - 1) + automatic_dates = set( + recovery_dates(now, self.settings.reliability_lookback_days) + ) + groups = self._load_enabled_groups() + current_by_id = {int(group.id): group for group in groups if group.id is not None} + items: list[dict] = [] + for offset in range(days): + run_date = (start + timedelta(days=offset)).isoformat() + if run_date in automatic_dates: + continue + state = self.state_store.load(run_date) + if state.get("state_status") == "corrupt": + items.append( + { + "run_date": run_date, + "group_id": None, + "group_name": "调度状态", + "status": "CORRUPT", + "execution_state": "HOLD_MANUAL", + "reason": "SCHEDULER_STATE_CORRUPT", + "safe_stage": "manual_review_only", + "recoverable": False, + "manifest_source": "corrupt", + "updated_at": "", + } + ) + continue + manifest = state.get("expected_groups") + if isinstance(manifest, list): + expected = [row for row in manifest if isinstance(row, dict)] + source = "recorded_manifest" + else: + expected = build_expected_groups( + groups, + datetime.fromisoformat(run_date).date(), + timezone=self.settings.app_timezone, + schedule_send_time=self.settings.schedule_send_time, + ) + source = "current_config_preview" + runs = { + str(run.get("group_id") or ""): run + for run in self.store.list_runs(run_date) + if isinstance(run, dict) + } + for snapshot in expected: + group_id = snapshot.get("group_id") + if not isinstance(group_id, int) or group_id <= 0: + continue + run = runs.get(str(group_id), {}) + status = str(run.get("status") or "MISSING") + if status in _TERMINAL_GENERATION_STATUSES: + continue + fail_closed = _is_fail_closed(run) + current_group = current_by_id.get(group_id) + recoverable = current_group is not None and not fail_closed + reason = ( + "RESULT_UNKNOWN_OR_CORRUPT" + if fail_closed + else "GROUP_NOT_ACTIVE" + if current_group is None + else str(run.get("error_type") or "MISSING_GENERATION") + ) + items.append( + { + "run_date": run_date, + "group_id": group_id, + "group_name": str(snapshot.get("group_name") or ""), + "status": status, + "execution_state": str(run.get("execution_state") or ""), + "reason": reason, + "safe_stage": "generation_only" if recoverable else "manual_review_only", + "recoverable": recoverable, + "manifest_source": source, + "estimated_summary_calls": 1 if recoverable else 0, + "estimated_image_calls": ( + 1 if recoverable and bool(snapshot.get("image_enabled", True)) else 0 + ), + "updated_at": str(run.get("updated_at") or state.get("updated_at") or ""), + } + ) + items.sort(key=lambda item: (item["run_date"], item.get("group_id") or 0)) + return { + "generated_at": now.isoformat(), + "automatic_recovery_dates": sorted(automatic_dates), + "lookback_days": days, + "version": _version_for(items), + "items": items, + } + + def confirm_generation( + self, + selections: list[dict], + *, + expected_version: str, + now: datetime | None = None, + ) -> dict: + preview = self.preview(now=now) + if not expected_version or expected_version != preview["version"]: + raise RecoveryPlanChangedError("恢复清单已变化，请刷新后重新确认") + allowed = { + (item["run_date"], item["group_id"]): item + for item in preview["items"] + if item.get("recoverable") and item.get("group_id") is not None + } + normalized: list[tuple[str, int]] = [] + for selection in selections: + run_date = validate_run_date(str(selection.get("run_date") or "")) + group_id = selection.get("group_id") + if not isinstance(group_id, int) or (run_date, group_id) not in allowed: + raise RecoverySelectionError("所选任务已不可恢复或需要人工复核") + normalized.append((run_date, group_id)) + normalized = sorted(set(normalized)) + if not normalized: + raise RecoverySelectionError("至少选择一个可恢复任务") + + results: list[dict] = [] + with generation_mutex(): + repo.init_db(self.settings) + repo.apply_db_settings(self.settings) + pipeline = DailyPipeline(settings=self.settings) + groups = self._load_enabled_groups() + by_id = {int(group.id): group for group in groups if group.id is not None} + for run_date in sorted({item[0] for item in normalized}): + ids = [group_id for date_value, group_id in normalized if date_value == run_date] + selected_groups = [by_id[group_id] for group_id in ids if group_id in by_id] + expected = build_expected_groups( + selected_groups, + datetime.fromisoformat(run_date).date(), + timezone=self.settings.app_timezone, + schedule_send_time=self.settings.schedule_send_time, + resolver=pipeline.period_resolver, + ) + state = self.state_store.load(run_date) + fields = { + "manual_recovery_confirmed_at": datetime.now().astimezone().isoformat(), + "manual_recovery_generation_only": True, + } + if not isinstance(state.get("expected_groups"), list): + # 这份清单只为人工确认的历史生成而创建；接口不承担历史发送， + # 因此不能把新建清单的期望终态写成 SENT。 + for row in expected: + row["expected_terminal"] = "READY_TO_SEND" + row["wechat_send_enabled"] = False + fields.update(manifest_fields(expected)) + fields["manifest_source"] = "manual_selection" + state = self.state_store.update(run_date, **fields) + day_results = pipeline.generate_all( + run_date=run_date, + group_ids=ids, + group_overrides={ + int(row["group_id"]): row + for row in state.get("expected_groups", []) + if isinstance(row, dict) and row.get("group_id") in ids + }, + acquire_lock=False, + ) + self.state_store.update( + run_date, + manual_recovery_results=day_results, + manual_recovery_status=_generation_status(day_results), + ) + write_daily_status(self.store, run_date) + results.extend(day_results) + return { + "status": _generation_status(results), + "generation_only": True, + "send_invoked": False, + "results": results, + } + + def repair_empty_manifest_and_generate( + self, + run_date: str, + *, + expected_state_version: int, + expected_group_ids: list[int], + ) -> dict: + """显式修复因配置回归形成的空清单，并只执行当天生成阶段。""" + + run_date = validate_run_date(run_date) + requested_ids = sorted(set(expected_group_ids)) + if not requested_ids or any(group_id <= 0 for group_id in requested_ids): + raise RecoverySelectionError("必须提供有效的预期群 ID") + + with generation_mutex(): + repo.init_db(self.settings) + repo.apply_db_settings(self.settings) + pipeline = DailyPipeline(settings=self.settings) + groups = self._load_enabled_groups() + expected = build_expected_groups( + groups, + datetime.fromisoformat(run_date).date(), + timezone=self.settings.app_timezone, + schedule_send_time=self.settings.schedule_send_time, + resolver=pipeline.period_resolver, + ) + actual_ids = sorted( + int(row["group_id"]) + for row in expected + if isinstance(row.get("group_id"), int) + ) + if actual_ids != requested_ids: + raise RecoveryPlanChangedError( + f"当前预期群已变化：expected={requested_ids} actual={actual_ids}" + ) + + state = self.state_store.load(run_date) + if state.get("state_status") == "corrupt": + raise RecoverySelectionError("调度状态损坏，只能人工复核") + if int(state.get("state_version") or 0) != expected_state_version: + raise RecoveryPlanChangedError("调度状态版本已变化，请重新确认") + if state.get("expected_groups") != [] or state.get("generation_status") != "not_run": + raise RecoverySelectionError("仅允许修复已完成的 no_groups 空清单") + if not state.get("generation_completed_at"): + raise RecoverySelectionError("原调度任务尚未形成可确认的完成终态") + if self.store.list_runs(run_date): + raise RecoverySelectionError("当天已存在群级运行记录，禁止重建空清单") + + now = datetime.now().astimezone().isoformat() + history = list(state.get("empty_manifest_repair_history") or []) + history.append( + { + "repaired_at": now, + "previous_state_version": expected_state_version, + "previous_manifest_created_at": str(state.get("manifest_created_at") or ""), + "previous_generation_completed_at": str( + state.get("generation_completed_at") or "" + ), + "previous_generation_status": str(state.get("generation_status") or ""), + } + ) + fields = manifest_fields(expected) + fields.update( + manifest_source="manual_empty_manifest_repair_current_config", + empty_manifest_repair_history=history[-10:], + generation_started_at=now, + generation_completed_at=None, + generation_invocation_completed_at=None, + generation_resumed_at=None, + generation_recovered_at=None, + generation_status="running", + generation_results=[], + generation_hold=False, + generation_error="", + email_started_at=None, + email_completed_at=now, + email_status="skipped_by_recovery_request", + email_hold=False, + email_error="", + email_detail="空清单显式恢复仅补生成，未调用邮件", + manual_recovery_confirmed_at=now, + manual_recovery_generation_only=True, + manual_recovery_scheduled_send_allowed=True, + ) + try: + repaired_state = self.state_store.compare_and_update( + run_date, + expected_state_version=expected_state_version, + **fields, + ) + except ScheduleStateVersionConflictError as exc: + raise RecoveryPlanChangedError("调度状态版本已变化，请重新确认") from exc + + try: + results = pipeline.generate_all( + run_date=run_date, + group_ids=actual_ids, + group_overrides={ + int(row["group_id"]): row + for row in repaired_state.get("expected_groups", []) + if isinstance(row, dict) and row.get("group_id") in actual_ids + }, + acquire_lock=False, + ) + except Exception as exc: + self.state_store.update( + run_date, + generation_status="interrupted", + generation_hold=True, + generation_error=str(exc)[:300], + generation_invocation_completed_at=datetime.now().astimezone().isoformat(), + ) + write_daily_status(self.store, run_date) + raise + + generation_status = _generation_status(results) + completion_fields = { + "generation_status": generation_status, + "generation_results": _compact_results(results), + "generation_invocation_completed_at": datetime.now().astimezone().isoformat(), + "generation_hold": generation_status in {"blocked", "failed", "partial"}, + "generation_error": "", + "manual_recovery_results": _compact_results(results), + "manual_recovery_status": generation_status, + } + if _generation_results_terminal(results): + completion_fields["generation_completed_at"] = ( + datetime.now().astimezone().isoformat() + ) + self.state_store.update(run_date, **completion_fields) + write_daily_status(self.store, run_date) + return { + "status": generation_status, + "run_date": run_date, + "generation_only": True, + "send_invoked": False, + "scheduled_send_allowed": True, + "expected_group_ids": actual_ids, + "results": results, + } + + def _load_enabled_groups(self): + from sqlmodel import Session + + repo.init_db(self.settings) + with Session(repo.engine) as session: + return repo.list_groups(session, only_enabled=True) diff --git a/app/scheduler/reliability_watchdog.py b/app/scheduler/reliability_watchdog.py new file mode 100644 index 0000000..bd5b964 --- /dev/null +++ b/app/scheduler/reliability_watchdog.py @@ -0,0 +1,141 @@ +"""进程启动时执行一次的最近 48 小时欠账恢复。 + +生成按日期从旧到新补跑。发送只处理现有 run.json 明确处于 READY 且未被 +hold 的任务，并继续复用 DeliveryStages 的 claim/unknown/目标预检合同。 +""" + +from __future__ import annotations + +from datetime import date, datetime, time, timedelta +from zoneinfo import ZoneInfo + +from app.config.settings import Settings, get_settings +from app.core.logging import get_logger +from app.pipeline.daily_pipeline import DailyPipeline +from app.scheduler.daily_v2_job import ( + DailyScheduleState, + ensure_daily_manifest, + run_daily_v2_job, +) +from app.scheduler.runtime_status import write_daily_status +from app.v2.run_store import RunStore + +logger = get_logger("groupbrief.scheduler") + + +def _generate_time(value: str) -> time: + try: + return datetime.strptime(str(value or ""), "%H:%M").time() + except (TypeError, ValueError): + return time(0, 15) + + +def recovery_dates(now: datetime, lookback_days: int) -> list[str]: + # 产品安全边界：部署配置即使遗留 30，也不得重新触发整月 AI/生图。 + days = min(max(int(lookback_days), 1), 2) + start = now.date() - timedelta(days=days - 1) + return [(start + timedelta(days=index)).isoformat() for index in range(days)] + + +def run_reliability_watchdog( + *, + settings: Settings | None = None, + now: datetime | None = None, +) -> dict: + settings = settings or get_settings() + tz = ZoneInfo(settings.app_timezone) + now = now or datetime.now(tz) + if now.tzinfo is None: + now = now.replace(tzinfo=tz) + if not settings.reliability_watchdog_enabled: + return {"status": "disabled", "generation": [], "send": []} + + dates = recovery_dates(now, settings.reliability_lookback_days) + state_store = DailyScheduleState(settings.output_dir) + generation_results: list[dict] = [] + generate_time = _generate_time(settings.schedule_generate_time) + for run_date in dates: + if run_date == now.date().isoformat() and now.time() < generate_time: + continue + state = state_store.load(run_date) + if state.get("state_status") == "corrupt": + generation_results.append( + { + "run_date": run_date, + "status": "held", + "error_type": "SCHEDULER_STATE_CORRUPT", + } + ) + continue + try: + state = ensure_daily_manifest( + settings, + run_date, + state_store=state_store, + state=state, + ) + write_daily_status(RunStore(settings.output_dir), run_date) + except Exception as exc: + logger.exception("启动恢复任务清单投影失败：run_date=%s", run_date) + generation_results.append( + { + "run_date": run_date, + "status": "failed", + "error_type": type(exc).__name__, + "detail": str(exc)[:300], + } + ) + continue + if state.get("generation_completed_at"): + continue + try: + result = run_daily_v2_job( + run_date, + settings=settings, + skip_email=True, + ) + except Exception as exc: + logger.exception("启动恢复生成补偿异常：run_date=%s", run_date) + result = { + "run_date": run_date, + "status": "failed", + "error_type": type(exc).__name__, + "detail": str(exc)[:300], + } + generation_results.append(result) + + try: + # 历史任务只能生成恢复；微信自动发送仍只扫描当天。 + send_results = DailyPipeline(settings=settings).send_due_for_dates( + [now.date().isoformat()], now=now, recovery=False + ) + except Exception as exc: + logger.exception("启动恢复发送检查异常") + send_results = [ + { + "status": "failed", + "error_type": type(exc).__name__, + "detail": str(exc)[:300], + } + ] + + status = "success" + if any(item.get("status") in {"failed", "partial"} for item in generation_results + send_results): + status = "partial" + if any(item.get("status") in {"held", "blocked"} for item in generation_results + send_results): + status = "attention_required" if status == "success" else status + result = { + "status": status, + "checked_at": now.isoformat(), + "dates": dates, + "generation": generation_results, + "send": send_results, + } + logger.info( + "启动恢复检查完成：status=%s dates=%d generation=%d send=%d", + status, + len(dates), + len(generation_results), + len(send_results), + ) + return result diff --git a/app/scheduler/runtime_status.py b/app/scheduler/runtime_status.py new file mode 100644 index 0000000..b7f87e1 --- /dev/null +++ b/app/scheduler/runtime_status.py @@ -0,0 +1,566 @@ +"""生成 runtime/YYYY-MM-DD/status.json 的脱敏每日运行报告。""" + +from __future__ import annotations + +import json +import os +from datetime import datetime, timedelta +from pathlib import Path +from typing import Iterable +from zoneinfo import ZoneInfo, ZoneInfoNotFoundError + +from app.v2.constants import ( + CORRUPT, + EXECUTION_ACTIVE, + EXECUTION_COMPLETE, + EXECUTION_FAILED_FINAL, + EXECUTION_HOLD_MANUAL, + EXECUTION_WAIT_RETRY, + IMAGE_READY, + READY_TO_SEND, + SENT, +) +from app.v2.run_store import RunStore, validate_run_date + +_CHECKPOINT_ORDER = { + "TASK_CREATED": 0, + "MESSAGES_SAVED": 1, + "RANKING_SAVED": 2, + "PROMPT_SAVED": 3, + "IMAGE_SAVED": 4, + "SENT_CONFIRMED": 5, +} + +_NODE_DEFINITIONS = ( + ("scheduler", "调度启动", 0, ""), + ("data", "读取群消息", 1, "fetch"), + ("ranking", "生成排行榜", 2, "ranking"), + ("prompt", "摘要与提示词", 3, "prompt"), + ("image", "生成图片", 4, "image"), + ("send", "等待发送 / 发送完成", 5, "send"), +) + +_NODE_LABELS = {node_id: label for node_id, label, _, _ in _NODE_DEFINITIONS} +_STAGE_NODE = { + "DATA": "data", + "RANKING": "ranking", + "PROMPT": "prompt", + "IMAGE": "image", + "SEND": "send", + "COMPLETE": "send", +} +_ACTIVE_SCHEDULER_STATUSES = {"running", "resuming"} + + +def _scheduler_snapshot(output_root: Path, run_date: str) -> dict: + path = output_root / ".scheduler" / f"{run_date}.json" + try: + parsed = json.loads(path.read_text(encoding="utf-8")) + except (OSError, UnicodeError, json.JSONDecodeError): + if path.exists(): + return { + "state_status": "corrupt", + "error_type": "SCHEDULER_STATE_CORRUPT", + "state_error_reason": "read_or_json_invalid", + "generation_hold": True, + } + return {} + if not isinstance(parsed, dict) or parsed.get("run_date") != run_date: + return { + "state_status": "corrupt", + "error_type": "SCHEDULER_STATE_CORRUPT", + "state_error_reason": "run_date_or_root_invalid", + "generation_hold": True, + } + return { + key: parsed.get(key) + for key in ( + "run_id", + "generation_status", + "generation_started_at", + "generation_completed_at", + "generation_invocation_completed_at", + "last_invocation_status", + "last_invocation_exit_code", + "manifest_version", + "manifest_created_at", + "expected_group_count", + "expected_groups", + "state_version", + "state_status", + "error_type", + "state_error_reason", + "generation_hold", + "generation_error", + "email_started_at", + "email_completed_at", + "email_status", + ) + if parsed.get(key) not in (None, "") + } + + +def _step_status(run: dict, required_checkpoint: int, stage: str) -> str: + checkpoint = _CHECKPOINT_ORDER.get(str(run.get("last_successful_checkpoint") or ""), 0) + if checkpoint >= required_checkpoint: + return "success" + failed_stage = str(run.get("failed_stage") or "").lower() + if failed_stage == stage or (stage == "fetch" and failed_stage == "data"): + return _failure_status(run) + return "pending" + + +def _failure_status(run: dict) -> str: + execution = str(run.get("execution_state") or "") + send_state = str(run.get("send_state") or "") + if ( + execution == EXECUTION_HOLD_MANUAL + or run.get("prompt_hold") + or run.get("send_hold") + or send_state in {"unknown", "failed_final"} + or str(run.get("status") or "") == CORRUPT + ): + return "held" + if execution == EXECUTION_WAIT_RETRY or run.get("next_retry_at") or run.get("send_next_retry_at"): + return "retry_pending" + return "failed" + + +def _run_has_started(run: dict) -> bool: + return bool( + run.get("updated_at") + or run.get("group_task_id") + or run.get("period_start") + or run.get("prompt_operation_started_at") + or run.get("sent_at") + ) + + +def _current_node_id(run: dict) -> str: + failed_stage = str(run.get("failed_stage") or "").lower() + if failed_stage in {"data", "fetch"}: + return "data" + if failed_stage in {"ranking", "prompt", "image", "send"}: + return failed_stage + stage = str(run.get("stage") or "").upper() + if stage in _STAGE_NODE: + return _STAGE_NODE[stage] + status = str(run.get("status") or "PENDING") + return { + "PENDING": "data", + "DATA_READY": "ranking", + "RANKING_READY": "prompt", + "PROMPT_READY": "image", + "IMAGE_READY": "send", + "READY_TO_SEND": "send", + "SENT": "send", + "FAILED": failed_stage if failed_stage in _NODE_LABELS else "data", + "CORRUPT": "data", + }.get(status, "data") + + +def _group_node_status( + run: dict, + node_id: str, + required_checkpoint: int, + stage: str, + *, + scheduler_active: bool, + scheduler_started: bool, +) -> str: + if node_id == "scheduler": + return "success" if scheduler_started else "pending" + + status = _step_status(run, required_checkpoint, stage) + if status != "pending": + return status + + if node_id == "send": + send_state = str(run.get("send_state") or "") + if send_state in {"claimed", "sending_text", "sending_image"}: + return "running" + if send_state in {"unknown", "failed_final"} or run.get("send_hold"): + return "held" + if run.get("send_next_retry_at"): + return "retry_pending" + + if _current_node_id(run) != node_id or not _run_has_started(run): + return "pending" + if node_id == "image": + image_job = run.get("image_job") + image_job_status = ( + str(image_job.get("status") or "") if isinstance(image_job, dict) else "" + ) + if image_job_status == "queued": + return "pending" + if image_job_status in {"running", "started"}: + return "running" + if node_id == "prompt" and str(run.get("prompt_operation_status") or "") == "started": + return "running" + if scheduler_active and str(run.get("execution_state") or EXECUTION_ACTIVE) == EXECUTION_ACTIVE: + return "running" + return "pending" + + +def _aggregate_node_status(statuses: list[str]) -> str: + if not statuses: + return "pending" + for candidate in ("held", "failed", "retry_pending", "running"): + if candidate in statuses: + return candidate + if all(status == "success" for status in statuses): + return "success" + return "pending" + + +def _scheduled_at(run_date: str, clock_time: str, timezone: str) -> str: + try: + hour, minute = (int(part) for part in clock_time.split(":", 1)) + value = datetime.fromisoformat(run_date).replace( + hour=hour, + minute=minute, + second=0, + microsecond=0, + tzinfo=ZoneInfo(timezone), + ) + except (TypeError, ValueError, ZoneInfoNotFoundError): + return "" + return value.isoformat() + + +def _next_scheduled_at(clock_time: str, timezone: str) -> str: + try: + tz = ZoneInfo(timezone) + now = datetime.now(tz) + hour, minute = (int(part) for part in clock_time.split(":", 1)) + value = now.replace(hour=hour, minute=minute, second=0, microsecond=0) + if value <= now: + value += timedelta(days=1) + except (TypeError, ValueError, ZoneInfoNotFoundError): + return "" + return value.isoformat() + + +def _group_snapshot( + run: dict, + *, + scheduler_active: bool = False, + scheduler_started: bool = False, +) -> dict: + status = str(run.get("status") or "PENDING") + send_state = str(run.get("send_state") or "") + if status == SENT or run.get("sent_at"): + send_status = "success" + elif send_state in {"unknown", "failed_final"} or run.get("send_hold"): + send_status = "held" + elif run.get("send_next_retry_at"): + send_status = "retry_pending" + elif send_state in {"claimed", "sending_text", "sending_image"}: + send_status = "running" + else: + send_status = _step_status(run, 5, "send") + + node_items = [ + { + "id": node_id, + "label": label, + "status": _group_node_status( + run, + node_id, + required_checkpoint, + stage, + scheduler_active=scheduler_active, + scheduler_started=scheduler_started, + ), + } + for node_id, label, required_checkpoint, stage in _NODE_DEFINITIONS + ] + current_node = _current_node_id(run) + current_status = next( + (item["status"] for item in node_items if item["id"] == current_node), + "pending", + ) + execution = str(run.get("execution_state") or "") + if status == SENT or execution == EXECUTION_COMPLETE: + current_status = "success" + elif status == CORRUPT: + current_status = "held" + elif execution == EXECUTION_FAILED_FINAL: + current_status = "failed" + elif execution == EXECUTION_HOLD_MANUAL: + current_status = "held" + elif execution == EXECUTION_WAIT_RETRY: + current_status = "retry_pending" + if current_status in {"held", "failed", "retry_pending"}: + for item in node_items: + if item["id"] == current_node: + item["status"] = current_status + break + + return { + "group_task_id": str(run.get("group_task_id") or ""), + "group_id": str(run.get("group_id") or ""), + "group_name": str(run.get("group_name") or ""), + "run_status": status, + "execution_state": execution, + "has_started": _run_has_started(run), + "current_node": current_node, + "current_node_label": _NODE_LABELS.get(current_node, "读取群消息"), + "node_status": current_status, + "nodes": node_items, + "last_successful_checkpoint": str(run.get("last_successful_checkpoint") or ""), + "next_retry_at": str(run.get("next_retry_at") or ""), + "retry_attempt_count": int(run.get("retry_attempt_count") or 0), + "retry_budget": int(run.get("retry_budget") or 0), + "data": { + "status": next(item["status"] for item in node_items if item["id"] == "data") + }, + "ranking": { + "status": next(item["status"] for item in node_items if item["id"] == "ranking") + }, + "summary": { + "status": next(item["status"] for item in node_items if item["id"] == "prompt"), + "model": str((run.get("prompt_meta") or {}).get("api_model") or "") + if isinstance(run.get("prompt_meta"), dict) + else "", + }, + "prompt": { + "status": next(item["status"] for item in node_items if item["id"] == "prompt") + }, + "image": { + "status": next(item["status"] for item in node_items if item["id"] == "image"), + "job_status": str((run.get("image_job") or {}).get("status") or "") + if isinstance(run.get("image_job"), dict) + else "", + "attempts": int(run.get("image_attempt_count") or 0), + "fallback_level": int(run.get("image_fallback_level") or 0), + }, + "send": { + "status": send_status, + "state": send_state, + "hold_reason": str(run.get("send_hold_reason") or ""), + "next_retry_at": str(run.get("send_next_retry_at") or ""), + "attempts": int(run.get("send_retry_attempt_count") or 0), + "retry_budget": int(run.get("send_retry_budget") or 0), + }, + "last_error_type": str( + run.get("last_error_type") + or run.get("error_type") + or run.get("send_error_type") + or run.get("prompt_hold_reason") + or run.get("send_hold_reason") + or "" + ), + "last_error_summary": str( + run.get("last_error_summary") + or run.get("error") + or run.get("send_error") + or run.get("prompt_operation_error") + or run.get("prompt_hold_reason") + or run.get("send_hold_reason") + or "" + )[:300], + "updated_at": str(run.get("updated_at") or ""), + } + + +def build_daily_status( + store: RunStore, + run_date: str, + *, + runs: Iterable[dict] | None = None, + output_root: Path | None = None, + schedule_generate_time: str = "00:15", + schedule_send_time: str = "08:30", + app_timezone: str = "Asia/Shanghai", +) -> dict: + """只读构建每日运行投影；不会写回 scheduler 或 run.json。""" + + run_date = validate_run_date(run_date) + resolved_output_root = Path(output_root) if output_root is not None else store.root + scheduler = _scheduler_snapshot(resolved_output_root, run_date) + scheduler_status = str(scheduler.get("generation_status") or "").lower() + scheduler_active = scheduler_status in _ACTIVE_SCHEDULER_STATUSES + scheduler_started = bool( + scheduler.get("generation_started_at") + or scheduler.get("generation_completed_at") + or scheduler_active + ) + selected_runs = runs if runs is not None else store.list_runs(run_date) + raw_runs = [dict(run) for run in selected_runs if isinstance(run, dict)] + groups = [ + _group_snapshot( + run, + scheduler_active=scheduler_active, + scheduler_started=scheduler_started, + ) + for run in raw_runs + ] + groups.sort(key=lambda item: (item["group_id"], item["group_name"])) + states = {item["execution_state"] for item in groups if item["has_started"]} + expected_rows = scheduler.get("expected_groups") + has_manifest = isinstance(expected_rows, list) + expected_rows = expected_rows if has_manifest else [] + expected_by_id = { + str(item.get("group_id")): item + for item in expected_rows + if isinstance(item, dict) and item.get("group_id") is not None + } + actual_by_id = { + item["group_id"]: item + for item in groups + if item["group_id"] and item["has_started"] + } + missing_expected_ids = sorted(set(expected_by_id) - set(actual_by_id)) + + def reached_expected(group_id: str, item: dict) -> bool: + expected_terminal = str( + (expected_by_id.get(group_id) or {}).get("expected_terminal") or "" + ) + status = item["run_status"] + if expected_terminal == SENT: + return status == SENT + return status in {READY_TO_SEND, IMAGE_READY, SENT} + + completed_count = sum( + 1 for group_id, item in actual_by_id.items() if reached_expected(group_id, item) + ) + retry_count = sum(1 for item in groups if item["execution_state"] == "WAIT_RETRY") + manual_count = sum( + 1 + for item in groups + if item["execution_state"] == "HOLD_MANUAL" or item["send"]["status"] == "held" + ) + external_call_count = sum( + int(item.get("external_call_count") or 0) + for item in raw_runs + ) + actual_providers = sorted( + { + str(item.get(field) or "").strip() + for item in raw_runs + for field in ("summary_provider_actual", "prompt_provider_actual") + if str(item.get(field) or "").strip() + } + ) + started_count = sum(1 for item in groups if item["has_started"]) + failed_count = sum( + 1 + for item in groups + if item["execution_state"] == EXECUTION_FAILED_FINAL or item["node_status"] == "failed" + ) + corrupt = scheduler.get("state_status") == "corrupt" or any( + item["run_status"] == CORRUPT for item in groups + ) + any_group_running = any(item["node_status"] == "running" for item in groups) + + if corrupt: + overall = "needs_attention" + elif scheduler_active or any_group_running: + overall = "running" + elif manual_count: + overall = "blocked" + elif retry_count or EXECUTION_WAIT_RETRY in states: + overall = "retry_pending" + elif failed_count: + overall = "failed" + elif not has_manifest and (started_count or scheduler_started): + overall = "needs_attention" + elif missing_expected_ids: + overall = "needs_attention" + elif expected_by_id and completed_count == len(expected_by_id): + overall = "complete" + elif completed_count or (started_count and scheduler.get("generation_completed_at")): + overall = "partial" + elif started_count or scheduler_started: + overall = "running" + else: + overall = "not_started" + + configured_group_count = max(len(groups), len(expected_by_id)) + scheduler_node_status = "pending" + if scheduler.get("state_status") == "corrupt" or scheduler.get("generation_hold"): + scheduler_node_status = "held" + elif scheduler_started: + scheduler_node_status = "running" if scheduler_active and not started_count else "success" + + nodes = [ + { + "id": "scheduler", + "label": _NODE_LABELS["scheduler"], + "status": scheduler_node_status, + "completed_groups": configured_group_count if scheduler_started else 0, + "total_groups": configured_group_count, + } + ] + for node_id, label, _, _ in _NODE_DEFINITIONS[1:]: + statuses = [ + next(node["status"] for node in group["nodes"] if node["id"] == node_id) + for group in groups + ] + nodes.append( + { + "id": node_id, + "label": label, + "status": _aggregate_node_status(statuses), + "completed_groups": sum(1 for status in statuses if status == "success"), + "total_groups": configured_group_count, + } + ) + + scheduler = { + **scheduler, + "scheduled_at": _scheduled_at( + run_date, + schedule_generate_time, + app_timezone, + ), + "send_scheduled_at": _scheduled_at( + run_date, + schedule_send_time, + app_timezone, + ), + "next_generate_at": _next_scheduled_at( + schedule_generate_time, + app_timezone, + ), + "next_send_at": _next_scheduled_at( + schedule_send_time, + app_timezone, + ), + } + payload = { + "schema_version": 2, + "run_date": run_date, + "run_id": str(scheduler.get("run_id") or f"groupbrief:{run_date}"), + "updated_at": datetime.now().astimezone().isoformat(), + "overall_status": overall, + "scheduler": scheduler, + "summary": { + "expected_group_count": len(expected_by_id), + "discovered_group_count": len(actual_by_id), + "configured_group_count": configured_group_count, + "completed_group_count": completed_count, + "retry_group_count": retry_count, + "manual_group_count": manual_count, + "missing_expected_group_ids": missing_expected_ids, + "manifest_complete": has_manifest and not missing_expected_ids, + "external_call_count": external_call_count, + "actual_providers": actual_providers, + }, + "nodes": nodes, + "groups": groups, + } + return payload + + +def write_daily_status(store: RunStore, run_date: str) -> Path: + run_date = validate_run_date(run_date) + payload = build_daily_status(store, run_date) + runtime_root = store.root.parent / "runtime" + path = runtime_root / run_date / "status.json" + path.parent.mkdir(parents=True, exist_ok=True) + temp = path.with_name(f".{path.name}.{os.getpid()}.tmp") + temp.write_text(json.dumps(payload, ensure_ascii=False, indent=2), encoding="utf-8") + os.replace(temp, path) + return path diff --git a/app/scheduler/send_job.py b/app/scheduler/send_job.py index 706198f..87a9ad3 100644 --- a/app/scheduler/send_job.py +++ b/app/scheduler/send_job.py @@ -1,19 +1,67 @@ -"""分钟级微信待发送扫描任务。""" +"""每日微信串行发送批次。""" from __future__ import annotations -from app.config.settings import get_settings +from datetime import datetime + +from app.config.settings import Settings, get_settings from app.core.logging import get_logger from app.pipeline.daily_pipeline import DailyPipeline +from app.scheduler.outcome import require_scheduler_success, summarize_results +from app.scheduler.heartbeat import record_scheduler_heartbeat logger = get_logger("groupbrief.scheduler") -def run_send_due_job() -> None: - """只处理已显式启用、到时、未发送且未被人工审核拦截的运行。""" +def run_send_due_job( + *, + settings: Settings | None = None, + now: datetime | None = None, + run_date: str | None = None, +) -> dict: + """按稳定群 ID 串行处理一天的日报发送，不再夹带周报扫描。""" + settings = settings or get_settings() + record_scheduler_heartbeat(settings, job="send_batch", status="started") try: - results = DailyPipeline(settings=get_settings()).send_due() - if results: - logger.info("微信 send_due 扫描结果：%s", results) - except Exception: - logger.exception("微信 send_due 调度异常") + pipeline = DailyPipeline(settings=settings) + if run_date is None: + results = pipeline.send_due() if now is None else pipeline.send_due(now=now) + else: + results = pipeline.send_due_for_dates( + [run_date], + now=now, + recovery=False, + ) + except Exception as exc: + logger.exception("微信发送批次调度异常") + record_scheduler_heartbeat( + settings, + job="send_batch", + status="error", + detail=f"{type(exc).__name__}: {exc}", + ) + raise + outcome = summarize_results(results) + if results: + logger.info( + "微信发送批次终态：outcome=%s exit_code=%d result_count=%d source_statuses=%s groups=%s", + outcome["outcome_status"], + outcome["exit_code"], + outcome["result_count"], + outcome["source_statuses"], + [ + { + key: item.get(key) + for key in ("group_name", "status", "error_type", "detail") + if item.get(key) not in (None, "") + } + for item in results + ], + ) + record_scheduler_heartbeat( + settings, + job="send_batch", + status=str(outcome.get("outcome_status") or "unknown"), + ) + require_scheduler_success(outcome, allow_not_run=True) + return outcome diff --git a/app/scheduler/task_manifest.py b/app/scheduler/task_manifest.py new file mode 100644 index 0000000..e2a8c1f --- /dev/null +++ b/app/scheduler/task_manifest.py @@ -0,0 +1,96 @@ +"""每日预期任务清单。 + +清单只保存调度与审计所需的脱敏配置快照，不保存聊天、Prompt 或凭据。 +""" + +from __future__ import annotations + +from datetime import date, datetime +from typing import Iterable + +from app.db.models import Group +from app.scheduler.period import PeriodResolver + +MANIFEST_VERSION = 1 +SUPPORTED_SCHEDULE_RULES = frozenset({"weekday_default", "daily_previous_day"}) + + +def build_expected_groups( + groups: Iterable[Group], + run_date: date, + *, + timezone: str, + schedule_send_time: str = "08:30", + resolver: PeriodResolver | None = None, +) -> list[dict]: + resolver = resolver or PeriodResolver() + expected: list[dict] = [] + for group in groups: + if group.id is None: + continue + rule = str(group.schedule_rule or "daily_previous_day") + if rule not in SUPPORTED_SCHEDULE_RULES: + raise ValueError(f"不支持的群级统计周期规则：{rule}") + window = resolver.resolve(run_date, timezone, schedule_rule=rule) + if not window.should_run: + continue + expected.append( + { + "group_id": int(group.id), + "group_name": str(group.display_name or group.wechat_group_name), + "wechat_group_id": str(group.wechat_group_id or ""), + "wechat_group_name": str(group.wechat_group_name or ""), + "schedule_rule": rule, + "history_provider_preference": str(group.provider_preference or ""), + "summary_provider": str(getattr(group, "summary_provider", "") or ""), + "summary_model": str(group.summary_model or ""), + "prompt_provider": str(getattr(group, "prompt_provider", "") or ""), + "prompt_model": str(group.prompt_model or ""), + "send_time": str(schedule_send_time or "08:30"), + "image_enabled": bool(group.image_enabled), + "ranking_template": str(group.ranking_template or "default"), + "ranking_count_policy": str( + getattr(group, "ranking_count_policy", "all_messages") + or "all_messages" + ), + "sender_name_policy": str( + getattr(group, "sender_name_policy", "resolved") or "resolved" + ), + "image_prompt_template": str(group.image_prompt_template or "default"), + "image_theme": str(group.image_theme or "ai_free"), + "image_theme_custom": str(group.image_theme_custom or ""), + "image_prompt_override": str(group.image_prompt_override or ""), + "send_target": str(group.send_target or ""), + "wechat_send_enabled": bool(group.wechat_send_enabled), + "expected_terminal": ( + "SENT" if group.wechat_send_enabled else "READY_TO_SEND" + ), + "period_start": window.period_start.isoformat(), + "period_end": window.period_end.isoformat(), + } + ) + expected.sort(key=lambda item: item["group_id"]) + return expected + + +def manifest_fields(expected_groups: list[dict]) -> dict: + return { + "manifest_version": MANIFEST_VERSION, + "manifest_created_at": datetime.now().astimezone().isoformat(), + "expected_groups": expected_groups, + "expected_group_count": len(expected_groups), + } + + +def expected_group_ids(state: dict) -> list[int]: + rows = state.get("expected_groups") + if not isinstance(rows, list): + return [] + result: list[int] = [] + for row in rows: + if not isinstance(row, dict): + continue + value = row.get("group_id") + if isinstance(value, int) and value > 0: + result.append(value) + return result diff --git a/app/sender/base.py b/app/sender/base.py index 3504e9e..620975f 100644 --- a/app/sender/base.py +++ b/app/sender/base.py @@ -7,7 +7,7 @@ from __future__ import annotations -from dataclasses import dataclass +from dataclasses import dataclass, field @dataclass @@ -20,6 +20,7 @@ class SendResult: submitted: bool = False verification_level: str = "" outcome_unknown: bool = False + diagnostics: dict[str, object] = field(default_factory=dict) class WechatSender: diff --git a/app/sender/wechat_native.py b/app/sender/wechat_native.py index ff078b3..f6235dc 100644 --- a/app/sender/wechat_native.py +++ b/app/sender/wechat_native.py @@ -1,7 +1,8 @@ """微信 4.1.x Windows 原生发送器。 -微信 4.1 使用自绘界面，标准 UI Automation 无法可靠定位控件。本模块改用： -键盘搜索 + Windows OCR 精确校验 + 剪贴板粘贴，并在任何歧义或校验失败时停止。 +微信 4.1 使用自绘界面，标准 UI Automation 只暴露部分控件。本模块组合： +键盘搜索 + Windows OCR 精确校验 + 唯一 UIA 搜索项兜底 + 剪贴板粘贴， +并在任何歧义或校验失败时停止。 生产入口默认使用 :class:`WindowsWechatDriver`；测试可注入 fake driver，绝不操作桌面。 """ @@ -16,8 +17,9 @@ import threading import time import unicodedata +import uuid from contextlib import contextmanager -from dataclasses import dataclass +from dataclasses import dataclass, field from datetime import datetime from pathlib import Path from typing import Iterator, Protocol @@ -50,6 +52,16 @@ class OcrLine: height: float +@dataclass(frozen=True) +class UiaSearchItem: + text: str + automation_id: str + left: float + top: float + right: float + bottom: float + + @dataclass(frozen=True) class NativeActionResult: success: bool @@ -57,6 +69,7 @@ class NativeActionResult: submitted: bool = False verification_level: str = "" outcome_unknown: bool = False + diagnostics: dict[str, object] = field(default_factory=dict) def _now() -> str: @@ -99,6 +112,8 @@ def _selected_header_matches(value: str, target: str) -> bool: return False if value_n[:3].casefold() != target_n[:3].casefold(): return False + if not _ascii_token_identity_matches(value, target): + return False suffix_pattern = r"\d+(?:[._-]\d+)+$" target_suffix = re.search(suffix_pattern, target_n) @@ -140,12 +155,36 @@ def _has_stable_ascii_anchor(value: str, target: str) -> bool: return any(len(re.sub(r"[^a-z0-9]", "", anchor)) >= 3 and anchor in value_n for anchor in anchors) +def _ascii_identity_tokens(value: str) -> list[str]: + normalized = unicodedata.normalize("NFKC", value or "").casefold() + return [token for token in re.findall(r"[a-z0-9]+", normalized) if len(token) >= 3] + + +def _ascii_token_identity_matches(value: str, target: str) -> bool: + """英文群名允许受限 OCR 错字，但每个稳定 token 都必须有对应项。""" + + target_tokens = _ascii_identity_tokens(target) + if not target_tokens: + return True + value_tokens = _ascii_identity_tokens(value) + return all( + any( + _edit_distance(target_token, value_token) + <= 2 + for value_token in value_tokens + ) + for target_token in target_tokens + ) + + def _search_title_score(value: str, target: str, *, max_distance: int | None = None) -> int | None: """返回受限 OCR 距离；版本号不同的相似群名永不匹配。""" value_n = _normalized_title(value) target_n = _normalized_title(target) if _title_matches(value_n, target_n): return 0 + if not _ascii_token_identity_matches(value, target): + return None anchored = _has_stable_ascii_anchor(value_n, target_n) tolerance = max_distance if max_distance is not None else (4 if anchored else 3) if len(target_n) < 8 or abs(len(value_n) - len(target_n)) > tolerance: @@ -179,6 +218,18 @@ def _section_label(value: str) -> str: return _normalized_title(value).strip("[]【】()（）<>《》") +def _trusted_group_section(value: str) -> bool: + """微信搜索浮层中允许作为群聊候选上边界的分区。""" + + label = _section_label(value) + if label == "群聊": + return True + # Windows OCR 在绿色标题上偶尔会把“用”识别成形近字（例如“岸”）。 + # 仅对较长且固定的“最常使用”标题容忍一个替换；两字“群聊”仍须精确， + # 避免把普通聊天文本误当成可信分区。 + return len(label) == len("最常使用") and _edit_distance(label, "最常使用") <= 1 + + def _select_group_search_match(lines: list[OcrLine], target: str) -> tuple[OcrLine | None, str]: """只从搜索浮层的“群聊”分区选择目标。 @@ -189,29 +240,62 @@ def _select_group_search_match(lines: list[OcrLine], target: str) -> tuple[OcrLi """ ordered = sorted(lines, key=lambda line: (line.top, line.left)) chat_sections = [line for line in ordered if _section_label(line.text) == "聊天记录"] + trusted_sections = [line for line in ordered if _trusted_group_section(line.text)] candidates: list[OcrLine] = [] if len(chat_sections) == 1: chat_top = chat_sections[0].top - candidates = [ - line - for line in ordered - if line.top + line.height <= chat_top - and 0 <= chat_top - (line.top + line.height) <= max(line.height * 6.0, 120.0) + section_before_chat = [ + line for line in trusted_sections if line.top + line.height <= chat_top ] + if section_before_chat: + section_bottom = max(line.top + line.height for line in section_before_chat) + candidates = [ + line + for line in ordered + if line.top + line.height / 2 >= section_bottom + and line.top + line.height <= chat_top + and not _trusted_group_section(line.text) + ] - scored = [(score, line) for line in candidates if (score := _search_title_score(line.text, target)) is not None] + scored = [ + (score, line) + for line in candidates + if (score := _search_title_score(line.text, target, max_distance=5)) is not None + ] + if not scored and _ascii_identity_tokens(target): + # 英文群名在绿色搜索项中可能同时出现多个 OCR 错字。只有处在可信 + # 群聊分区、稳定英文 token 仍逐项对应、且最终聊天标题还会再次复核 + # 时，才额外扩大搜索项的距离；中文群名不使用此兜底。 + scored = [ + (score, line) + for line in candidates + if (score := _search_title_score(line.text, target, max_distance=8)) is not None + ] if not scored and not chat_sections: # 新改名或很少在聊天中提及的群，搜索结果可能只有 # “最常使用”中的群聊项，没有“聊天记录”分区。此时只允许 # 选择“搜索网络结果”之前、且明显低于顶部输入框的唯一匹配。 network_sections = [line for line in ordered if "搜索网络结果" in _section_label(line.text)] - if network_sections: + if network_sections and trusted_sections: network_top = min(line.top for line in network_sections) before_network = [ line for line in ordered if line.top >= 55.0 and line.top + line.height <= network_top ] + section_before_network = [ + line for line in trusted_sections if line.top + line.height <= network_top + ] + if section_before_network: + section_bottom = max(line.top + line.height for line in section_before_network) + before_network = [ + line + for line in before_network + if line.top + line.height / 2 >= section_bottom + and not _trusted_group_section(line.text) + ] + else: + before_network = [] scored = [ (score, line) for line in before_network @@ -220,8 +304,17 @@ def _select_group_search_match(lines: list[OcrLine], target: str) -> tuple[OcrLi # “V4.0”被识别成“V4℃”时产生的一个 OCR 距离。 if (score := _search_title_score(line.text, target, max_distance=5)) is not None ] + if not scored and _ascii_identity_tokens(target): + scored = [ + (score, line) + for line in before_network + if (score := _search_title_score(line.text, target, max_distance=8)) is not None + ] if not scored: - return None, "群聊分区未得到可验证的目标匹配（匹配数 0）" + return None, ( + "群聊分区未得到可验证的目标匹配" + f"（匹配数 0；可信分区 {len(trusted_sections)}；聊天记录分区 {len(chat_sections)}）" + ) best_score = min(score for score, _ in scored) best = [line for score, line in scored if score == best_score] if len(best) != 1: @@ -229,6 +322,49 @@ def _select_group_search_match(lines: list[OcrLine], target: str) -> tuple[OcrLi return best[0], "" +def _select_uia_group_search_match( + items: list[UiaSearchItem], + target: str, + search_box: tuple[int, int, int, int], +) -> tuple[OcrLine | None, str]: + """Select one exact WeChat group result exposed by UI Automation. + + WeChat 4.1 gives only the real group-search item an automation id in the + form ``search_item_``. Chat-history rows and network results do + not carry that id. Keep the final header OCR check in ``open_and_verify`` + as a second, independent target verification layer. + """ + + expected_id = f"search_item_{target}" + box_left, box_top, box_right, box_bottom = search_box + matches: list[UiaSearchItem] = [] + for item in items: + center_x = (item.left + item.right) / 2 + center_y = (item.top + item.bottom) / 2 + if ( + item.automation_id == expected_id + and _title_matches(item.text, target) + and item.right > item.left + and item.bottom > item.top + and box_left <= center_x <= box_right + and box_top <= center_y <= box_bottom + ): + matches.append(item) + if len(matches) != 1: + return None, f"UIA 精确群聊项数量不是 1（当前 {len(matches)}）" + item = matches[0] + return ( + OcrLine( + item.text, + item.left - box_left, + item.top - box_top, + item.right - item.left, + item.bottom - item.top, + ), + "", + ) + + def _coerce_action_result(value: tuple[bool, str] | NativeActionResult) -> NativeActionResult: if isinstance(value, NativeActionResult): return value @@ -281,6 +417,18 @@ class WindowsWechatDriver: def __init__(self, settings: Settings | None = None): self.settings = settings or get_settings() self.delay = max(float(self.settings.wechat_native_action_delay_seconds), 0.2) + self.stage_timeout = max( + float(self.settings.wechat_native_stage_timeout_seconds), + self.delay, + ) + self.submit_timeout = max( + float(self.settings.wechat_native_submit_timeout_seconds), + self.delay, + ) + self.poll_interval = min( + max(float(self.settings.wechat_native_poll_interval_seconds), 0.05), + 1.0, + ) self._window: int | None = None self._enable_dpi_awareness() @@ -346,6 +494,17 @@ def open_and_verify(self, target: str) -> tuple[bool, str]: target = (target or "").strip() if not target: return False, "发送目标为空" + if os.name != "nt": + return False, "Windows 原生微信发送器仅支持 Windows" + try: + self._imports() + except Exception as exc: + return False, f"Windows 微信发送依赖不可用：{exc}" + if not self._desktop_unlocked(): + return False, "Windows 桌面已锁定，未尝试恢复或操作微信窗口" + prepared, prepare_detail = self._prepare_wechat_window() + if not prepared: + return False, prepare_detail ok, detail = self.health_check() if not ok: return False, detail @@ -400,6 +559,10 @@ def open_and_verify(self, target: str) -> tuple[bool, str]: if matched is not None: break time.sleep(self.delay) + if matched is None: + matched, uia_error = self._find_uia_group_search_match(target, search_box) + if matched is None and uia_error: + match_error = f"{match_error}；{uia_error}" if matched is None: return False, f"{match_error}，已停止发送" self._click(search_box[0] + matched.left + matched.width / 2, search_box[1] + matched.top + matched.height / 2) @@ -418,22 +581,69 @@ def paste_text(self, text: str) -> NativeActionResult: return NativeActionResult(False, "发送文字为空") submitted = False try: - before_composer, before_chat = self._capture_send_regions() self._focus_composer() + composer_empty, empty_detail = self._composer_is_empty() + if not composer_empty: + return NativeActionResult( + False, + empty_detail, + diagnostics={"phase": "composer_preflight", "composer_empty": False}, + ) + self._focus_composer() + stable, before_composer, before_chat, baseline_attempts = self._capture_stable_baseline() + if not stable: + return NativeActionResult( + False, + "文字发送前输入区无法达到稳定状态，已停止且未修改可能存在的草稿", + diagnostics={ + "phase": "composer_preflight", + "baseline_stable": False, + "baseline_attempts": baseline_attempts, + }, + ) self._set_clipboard_text(text) self._hotkey("ctrl", "v") - time.sleep(self.delay) - staged_composer, _ = self._capture_send_regions() + staged_composer, staged_change, stage_attempts = self._wait_for_staged_change( + before_composer + ) + diagnostics = { + "phase": "composer_staged", + "baseline_stable": True, + "baseline_attempts": baseline_attempts, + "stage_attempts": stage_attempts, + "staged_change": round(staged_change, 6), + } + if staged_composer is None: + return NativeActionResult( + False, + "文字粘贴后未观察到输入区暂存，已停止且未按 Enter", + diagnostics=diagnostics, + ) self._key("enter") submitted = True - time.sleep(self.delay * 1.5) - after_composer, after_chat = self._capture_send_regions() - ok, detail = self._verify_submission( - before_composer, staged_composer, after_composer, before_chat, after_chat + ok, detail, submit_diagnostics = self._wait_for_submission( + before_composer, + staged_composer, + before_chat, ) + diagnostics.update(submit_diagnostics) if not ok: - return NativeActionResult(False, f"文字已按 Enter，但 {detail}", True, "unknown", True) - return NativeActionResult(True, "文字已提交，且输入区清空和聊天区域变化已由 UI 观察", True, "ui_observed") + return NativeActionResult( + False, + f"文字已按 Enter，但 {detail}", + True, + "unknown", + True, + diagnostics, + ) + return NativeActionResult( + True, + "文字已提交，且输入区清空和聊天区域变化已由 UI 观察", + True, + "ui_observed", + False, + diagnostics, + ) except Exception as exc: return NativeActionResult( False, @@ -441,31 +651,77 @@ def paste_text(self, text: str) -> NativeActionResult: submitted, "unknown" if submitted else "", submitted, + {"phase": "exception_after_submit" if submitted else "exception_before_submit"}, ) def paste_image(self, image_path: Path) -> NativeActionResult: submitted = False try: - before_composer, before_chat = self._capture_send_regions() self._focus_composer() + composer_empty, empty_detail = self._composer_is_empty() + if not composer_empty: + return NativeActionResult( + False, + empty_detail, + diagnostics={"phase": "composer_preflight", "composer_empty": False}, + ) + self._focus_composer() + stable, before_composer, before_chat, baseline_attempts = self._capture_stable_baseline() + if not stable: + return NativeActionResult( + False, + "图片发送前输入区无法达到稳定状态，已停止且未修改可能存在的草稿", + diagnostics={ + "phase": "composer_preflight", + "baseline_stable": False, + "baseline_attempts": baseline_attempts, + }, + ) self._set_clipboard_image(image_path) self._hotkey("ctrl", "v") - time.sleep(self.delay * 1.5) - staged_composer, _ = self._capture_send_regions() - if self._difference_ratio(before_composer, staged_composer) < 0.0005: - return NativeActionResult(False, "图片粘贴后未观察到预览，已停止发送") + staged_composer, staged_change, stage_attempts = self._wait_for_staged_change( + before_composer + ) + diagnostics = { + "phase": "composer_staged", + "baseline_stable": True, + "baseline_attempts": baseline_attempts, + "stage_attempts": stage_attempts, + "staged_change": round(staged_change, 6), + } + if staged_composer is None: + return NativeActionResult( + False, + "图片粘贴后未观察到预览，已停止且未按 Enter", + diagnostics=diagnostics, + ) if not self._window: return NativeActionResult(False, "微信窗口尚未验证") self._key("enter") submitted = True - time.sleep(self.delay * 2.5) - after_composer, after_chat = self._capture_send_regions() - ok, detail = self._verify_submission( - before_composer, staged_composer, after_composer, before_chat, after_chat + ok, detail, submit_diagnostics = self._wait_for_submission( + before_composer, + staged_composer, + before_chat, ) + diagnostics.update(submit_diagnostics) if not ok: - return NativeActionResult(False, f"图片已按 Enter，但 {detail}", True, "unknown", True) - return NativeActionResult(True, "图片已提交，且预览清空和聊天区域变化已由 UI 观察", True, "ui_observed") + return NativeActionResult( + False, + f"图片已按 Enter，但 {detail}", + True, + "unknown", + True, + diagnostics, + ) + return NativeActionResult( + True, + "图片已提交，且预览清空和聊天区域变化已由 UI 观察", + True, + "ui_observed", + False, + diagnostics, + ) except Exception as exc: return NativeActionResult( False, @@ -473,6 +729,7 @@ def paste_image(self, image_path: Path) -> NativeActionResult: submitted, "unknown" if submitted else "", submitted, + {"phase": "exception_after_submit" if submitted else "exception_before_submit"}, ) @staticmethod @@ -528,21 +785,132 @@ def _check_clipboard() -> None: win32clipboard.CloseClipboard() @staticmethod - def _wechat_windows() -> list[int]: + def _window_process_name(hwnd: int) -> str: + """只读取窗口所属进程名；无法确认身份时返回空字符串。""" + + if os.name != "nt": + return "" + kernel32 = ctypes.windll.kernel32 + user32 = ctypes.windll.user32 + open_process = kernel32.OpenProcess + open_process.argtypes = [ctypes.c_ulong, ctypes.c_int, ctypes.c_ulong] + open_process.restype = ctypes.c_void_p + query_process_image = kernel32.QueryFullProcessImageNameW + query_process_image.argtypes = [ + ctypes.c_void_p, + ctypes.c_ulong, + ctypes.c_wchar_p, + ctypes.POINTER(ctypes.c_ulong), + ] + query_process_image.restype = ctypes.c_int + close_handle = kernel32.CloseHandle + close_handle.argtypes = [ctypes.c_void_p] + close_handle.restype = ctypes.c_int + process_id = ctypes.c_ulong(0) + user32.GetWindowThreadProcessId(hwnd, ctypes.byref(process_id)) + if not process_id.value: + return "" + handle = open_process(0x1000, False, process_id.value) + if not handle: + return "" + try: + size = ctypes.c_ulong(32768) + buffer = ctypes.create_unicode_buffer(size.value) + if not query_process_image(handle, 0, buffer, ctypes.byref(size)): + return "" + return Path(buffer.value).name.casefold() + finally: + close_handle(handle) + + @staticmethod + def _window_is_current_session(hwnd: int) -> bool: + """窗口进程必须与发送器处在同一 Windows 会话。""" + + if os.name != "nt": + return False + kernel32 = ctypes.windll.kernel32 + user32 = ctypes.windll.user32 + process_id = ctypes.c_ulong(0) + user32.GetWindowThreadProcessId(hwnd, ctypes.byref(process_id)) + if not process_id.value: + return False + window_session = ctypes.c_ulong(0) + current_session = ctypes.c_ulong(0) + if not kernel32.ProcessIdToSessionId(process_id.value, ctypes.byref(window_session)): + return False + current_pid = kernel32.GetCurrentProcessId() + if not kernel32.ProcessIdToSessionId(current_pid, ctypes.byref(current_session)): + return False + return window_session.value == current_session.value + + @staticmethod + def _wechat_main_window_class(value: str) -> bool: + normalized = (value or "").strip() + return normalized == "WeChatMainWndForPC" or bool( + re.fullmatch(r"Qt\d+QWindowIcon", normalized) + ) + + @classmethod + def _enumerate_wechat_windows(cls, *, visible: bool) -> list[int]: + import win32con import win32gui matches: list[int] = [] def visit(hwnd, _): - if not win32gui.IsWindowVisible(hwnd): + if bool(win32gui.IsWindowVisible(hwnd)) is not visible: return title = (win32gui.GetWindowText(hwnd) or "").strip() - if title in {"微信", "WeChat"}: - matches.append(hwnd) + if title not in {"微信", "WeChat"}: + return + if win32gui.GetParent(hwnd) or win32gui.GetWindow(hwnd, win32con.GW_OWNER): + return + if not cls._wechat_main_window_class(win32gui.GetClassName(hwnd)): + return + if cls._window_process_name(hwnd) not in {"weixin.exe", "wechat.exe"}: + return + if not cls._window_is_current_session(hwnd): + return + matches.append(hwnd) win32gui.EnumWindows(visit, None) return matches + @classmethod + def _wechat_windows(cls) -> list[int]: + return cls._enumerate_wechat_windows(visible=True) + + @classmethod + def _hidden_wechat_windows(cls) -> list[int]: + return cls._enumerate_wechat_windows(visible=False) + + def _prepare_wechat_window(self) -> tuple[bool, str]: + """恢复唯一、已存在的隐藏主窗口；绝不启动微信或处理登录。""" + + visible = self._wechat_windows() + if len(visible) == 1: + return True, "找到唯一可见微信主窗口" + if len(visible) > 1: + return False, f"必须存在唯一可见微信主窗口，当前找到 {len(visible)} 个" + + hidden = self._hidden_wechat_windows() + if len(hidden) != 1: + return False, ( + "没有唯一可恢复的微信主窗口" + f"（可见 {len(visible)} 个，隐藏候选 {len(hidden)} 个）" + ) + if not self._activate(hidden[0]): + return False, "唯一隐藏微信主窗口无法安全恢复" + deadline = time.monotonic() + max(self.stage_timeout, self.delay * 3) + while time.monotonic() < deadline: + visible = self._wechat_windows() + if len(visible) == 1 and visible[0] == hidden[0]: + return True, "已恢复唯一隐藏微信主窗口" + if len(visible) > 1: + return False, f"恢复后出现多个可见微信主窗口（{len(visible)} 个）" + time.sleep(self.poll_interval) + return False, "隐藏微信主窗口恢复后仍不可见" + @staticmethod def _activate(hwnd: int) -> bool: import win32con @@ -590,6 +958,45 @@ def _window_rect(hwnd: int) -> tuple[int, int, int, int]: return win32gui.GetWindowRect(hwnd) + def _find_uia_group_search_match( + self, + target: str, + search_box: tuple[int, int, int, int], + ) -> tuple[OcrLine | None, str]: + """Read the uniquely identified group result from WeChat's UIA tree.""" + + if not self._window: + return None, "微信窗口尚未验证" + try: + import win32process + from pywinauto import Desktop + + _, process_id = win32process.GetWindowThreadProcessId(self._window) + if not process_id: + return None, "无法确认微信窗口进程" + items: list[UiaSearchItem] = [] + for window in Desktop(backend="uia").windows(process=process_id, visible_only=True): + for control in window.descendants(control_type="ListItem"): + info = control.element_info + automation_id = str(getattr(info, "automation_id", "") or "") + if not automation_id.startswith("search_item_"): + continue + rectangle = info.rectangle + items.append( + UiaSearchItem( + text=str(control.window_text() or ""), + automation_id=automation_id, + left=float(rectangle.left), + top=float(rectangle.top), + right=float(rectangle.right), + bottom=float(rectangle.bottom), + ) + ) + return _select_uia_group_search_match(items, target, search_box) + except Exception as exc: + logger.warning("WeChat UIA search fallback unavailable: %s", exc) + return None, f"UIA 搜索兜底不可用：{type(exc).__name__}" + @staticmethod def _click(x: float, y: float) -> None: import win32api @@ -634,13 +1041,109 @@ def _difference_ratio(first, second) -> float: means = ImageStat.Stat(difference).mean return sum(means) / (255.0 * max(len(means), 1)) + def _poll_rounds(self, timeout_seconds: float) -> int: + return max(1, int(timeout_seconds / self.poll_interval) + 1) + + def _capture_stable_baseline(self): + previous_composer, previous_chat = self._capture_send_regions() + attempts = self._poll_rounds(self.stage_timeout) + for attempt in range(1, attempts + 1): + time.sleep(self.poll_interval) + composer, chat = self._capture_send_regions() + if self._difference_ratio(previous_composer, composer) <= 0.0003: + return True, composer, chat, attempt + previous_composer, previous_chat = composer, chat + return False, previous_composer, previous_chat, attempts + + def _wait_for_staged_change(self, before_composer): + attempts = self._poll_rounds(self.stage_timeout) + max_change = 0.0 + for attempt in range(1, attempts + 1): + composer, _ = self._capture_send_regions() + change = self._difference_ratio(before_composer, composer) + max_change = max(max_change, change) + if change >= 0.0005: + return composer, change, attempt + if attempt < attempts: + time.sleep(self.poll_interval) + return None, max_change, attempts + + def _wait_for_submission( + self, + before_composer, + staged_composer, + before_chat, + ) -> tuple[bool, str, dict[str, object]]: + attempts = self._poll_rounds(self.submit_timeout) + last_detail = "尚未观察到提交变化" + last_metrics: dict[str, float] = {} + for attempt in range(1, attempts + 1): + after_composer, after_chat = self._capture_send_regions() + ok, detail = self._verify_submission( + before_composer, + staged_composer, + after_composer, + before_chat, + after_chat, + ) + last_detail = detail + last_metrics = self._submission_metrics( + before_composer, + staged_composer, + after_composer, + before_chat, + after_chat, + ) + if ok: + return True, detail, { + "phase": "submit_verified", + "submit_attempts": attempt, + **{key: round(value, 6) for key, value in last_metrics.items()}, + } + if attempt < attempts: + time.sleep(self.poll_interval) + return False, last_detail, { + "phase": "submit_unknown", + "submit_attempts": attempts, + **{key: round(value, 6) for key, value in last_metrics.items()}, + } + @classmethod - def _verify_submission(cls, before_composer, staged_composer, after_composer, before_chat, after_chat) -> tuple[bool, str]: + def _submission_metrics( + cls, + before_composer, + staged_composer, + after_composer, + before_chat, + after_chat, + ) -> dict[str, float]: staged_change = cls._difference_ratio(before_composer, staged_composer) cleared_change = cls._difference_ratio(staged_composer, after_composer) returned_toward_empty = cls._difference_ratio(before_composer, after_composer) chat_change = cls._difference_ratio(before_chat, after_chat) empty_delta_limit = min(max(staged_change * 0.25, 0.0003), 0.0015) + return { + "staged_change": staged_change, + "cleared_change": cleared_change, + "returned_toward_empty": returned_toward_empty, + "empty_delta_limit": empty_delta_limit, + "chat_change": chat_change, + } + + @classmethod + def _verify_submission(cls, before_composer, staged_composer, after_composer, before_chat, after_chat) -> tuple[bool, str]: + metrics = cls._submission_metrics( + before_composer, + staged_composer, + after_composer, + before_chat, + after_chat, + ) + staged_change = metrics["staged_change"] + cleared_change = metrics["cleared_change"] + returned_toward_empty = metrics["returned_toward_empty"] + chat_change = metrics["chat_change"] + empty_delta_limit = metrics["empty_delta_limit"] composer_ok = ( staged_change >= 0.0005 and cleared_change >= 0.0005 @@ -663,6 +1166,7 @@ def _key(name: str, key_up: bool = False) -> None: codes = { "ctrl": win32con.VK_CONTROL, "a": ord("A"), + "c": ord("C"), "f": ord("F"), "v": ord("V"), "enter": win32con.VK_RETURN, @@ -675,6 +1179,46 @@ def _hotkey(self, modifier: str, key: str) -> None: self._key(key, key_up=True) self._key(modifier, key_up=True) + def _composer_is_empty(self) -> tuple[bool, str]: + """通过无破坏复制探测输入区；无法证明为空时保守停止。""" + import win32clipboard + import win32con + + sentinel = f"GroupBrief-empty-check-{uuid.uuid4().hex}" + self._set_clipboard_text(sentinel) + self._hotkey("ctrl", "a") + self._hotkey("ctrl", "c") + time.sleep(self.poll_interval) + try: + win32clipboard.OpenClipboard() + formats: list[int] = [] + value = "" + try: + current = 0 + while True: + current = win32clipboard.EnumClipboardFormats(current) + if current == 0: + break + formats.append(int(current)) + if win32clipboard.IsClipboardFormatAvailable(win32con.CF_UNICODETEXT): + value = str(win32clipboard.GetClipboardData(win32con.CF_UNICODETEXT) or "") + finally: + win32clipboard.CloseClipboard() + except Exception as exc: + return False, f"无法确认微信输入区为空，已停止发送：{str(exc)[:120]}" + finally: + # 重新点击输入区只取消选择，不删除或覆盖任何草稿。 + self._focus_composer() + draft_media_formats = { + win32con.CF_BITMAP, + win32con.CF_DIB, + getattr(win32con, "CF_DIBV5", 17), + win32con.CF_HDROP, + } + if value == sentinel and not (set(formats) & draft_media_formats): + return True, "输入区为空" + return False, "微信输入区可能存在文字或图片草稿，已停止且未清除草稿" + @staticmethod def _set_clipboard_text(text: str) -> None: import win32clipboard @@ -753,10 +1297,17 @@ def __init__(self, settings: Settings | None = None, driver: NativeWechatDriver self.driver = driver or WindowsWechatDriver(self.settings) self.dry_run = dry_run - def health_check(self) -> tuple[bool, str]: + def health_check(self, report: dict | None = None) -> tuple[bool, str]: if self.dry_run: return True, "Windows 微信原生发送器 dry-run" - return self.driver.health_check() + report = report or self.health_report() + if report["ok"]: + return True, "微信原生发送器可用（依赖、桌面、中文 OCR、剪贴板、唯一窗口均通过）" + for stage in ("dependencies", "desktop", "ocr", "clipboard", "window"): + item = report[stage] + if not item["ok"]: + return False, item["detail"] + return False, "微信原生发送器健康检查失败" def health_report(self) -> dict: if self.dry_run: @@ -815,6 +1366,7 @@ def send_image(self, target: str, image_path) -> SendResult: action.submitted, action.verification_level, action.outcome_unknown, + action.diagnostics, ) except Exception as exc: image_result = SendResult(False, str(exc), _now()) @@ -843,6 +1395,7 @@ def send_bundle(self, target: str, text: str, image_path: str | Path | None) -> text_action.submitted, text_action.verification_level, text_action.outcome_unknown, + text_action.diagnostics, ) if not text_action.success or path is None: return text_result, None @@ -854,15 +1407,27 @@ def send_bundle(self, target: str, text: str, image_path: str | Path | None) -> image_action.submitted, image_action.verification_level, image_action.outcome_unknown, + image_action.diagnostics, ) except Exception as exc: return SendResult(False, str(exc), _now()), None +def validate_wechat_sender_mode(settings: Settings) -> str: + """返回规范化 sender mode；未知值禁止默认落到 native。""" + mode = str(settings.wechat_sender_mode or "").strip().lower() + if mode not in {"native", "legacy_cli"}: + raise ValueError(f"不支持的微信发送 Provider：{settings.wechat_sender_mode}") + return mode + + def create_wechat_sender(settings: Settings | None = None, dry_run: bool = False) -> WechatSender: settings = settings or get_settings() - if settings.wechat_sender_mode.strip().lower() == "legacy_cli": + mode = validate_wechat_sender_mode(settings) + if mode == "legacy_cli": from app.sender.wechat_automation import WechatAutomationSender return WechatAutomationSender(settings=settings, dry_run=dry_run) - return WechatNativeSender(settings=settings, dry_run=dry_run) + if mode == "native": + return WechatNativeSender(settings=settings, dry_run=dry_run) + raise AssertionError("微信 sender 配置校验未覆盖已知类型") diff --git a/app/services/email_delivery.py b/app/services/email_delivery.py new file mode 100644 index 0000000..a1fcc6b --- /dev/null +++ b/app/services/email_delivery.py @@ -0,0 +1,342 @@ +"""SMTP 单封邮件幂等交付。 + +本模块不假设 SMTP 服务端支持 Idempotency-Key。稳定 Message-ID 只用于审计； +真正的本地保护来自逐封 delivery ledger 和提交前状态落盘。 +""" + +from __future__ import annotations + +import ctypes +import hashlib +import json +import os +import smtplib +import threading +import time +import uuid +from contextlib import contextmanager +from dataclasses import dataclass +from datetime import datetime +from email.message import EmailMessage +from pathlib import Path +from typing import Iterator + +from app.config.settings import Settings + + +_DELIVERY_LOCK = threading.RLock() +_WAIT_OBJECT_0 = 0 +_WAIT_ABANDONED = 0x80 + + +@dataclass(frozen=True) +class EmailDeliveryIdentity: + key: str + message_id: str + fingerprint: str + + +@dataclass(frozen=True) +class EmailDeliveryResult: + status: str + detail: str = "" + message_id: str = "" + + @property + def success(self) -> bool: + return self.status in {"sent", "already_sent"} + + @property + def outcome_unknown(self) -> bool: + return self.status == "unknown" + + +def _message_fingerprint(message: EmailMessage) -> str: + """按稳定语义字段计算指纹，避免 MIME 随机 boundary 影响结果。""" + parts: list[dict[str, str]] = [] + for part in message.walk(): + if part.is_multipart(): + continue + payload = part.get_payload(decode=True) + if payload is None: + payload = str(part.get_payload() or "").encode("utf-8", errors="replace") + parts.append( + { + "content_type": part.get_content_type(), + "filename": str(part.get_filename() or ""), + "content_disposition": str(part.get_content_disposition() or ""), + "sha256": hashlib.sha256(payload).hexdigest(), + } + ) + canonical = json.dumps( + { + "subject": str(message.get("Subject") or ""), + "from": str(message.get("From") or ""), + "to": str(message.get("To") or ""), + "cc": str(message.get("Cc") or ""), + "parts": parts, + }, + ensure_ascii=False, + sort_keys=True, + separators=(",", ":"), + ) + return hashlib.sha256(canonical.encode("utf-8")).hexdigest() + + +def ensure_email_identity(message: EmailMessage) -> EmailDeliveryIdentity: + fingerprint = _message_fingerprint(message) + message_id = f"" + if "Message-ID" in message: + del message["Message-ID"] + message["Message-ID"] = message_id + return EmailDeliveryIdentity( + key=fingerprint, + message_id=message_id, + fingerprint=fingerprint, + ) + + +@contextmanager +def _delivery_mutex(path: Path, timeout_seconds: float = 30.0) -> Iterator[None]: + acquired = _DELIVERY_LOCK.acquire(timeout=max(timeout_seconds, 0.1)) + if not acquired: + raise TimeoutError("等待邮件交付锁超时") + handle = None + owns_handle = False + try: + if os.name == "nt": + digest = hashlib.sha256(str(path.resolve()).lower().encode("utf-8")).hexdigest()[:32] + handle = ctypes.windll.kernel32.CreateMutexW( + None, False, f"Local\\GroupBrief.Email.{digest}" + ) + if not handle: + raise OSError("无法创建邮件交付互斥锁") + wait_code = ctypes.windll.kernel32.WaitForSingleObject( + handle, int(timeout_seconds * 1000) + ) + if wait_code not in (_WAIT_OBJECT_0, _WAIT_ABANDONED): + raise TimeoutError("等待跨进程邮件交付锁超时") + owns_handle = True + yield + finally: + if handle: + if owns_handle: + ctypes.windll.kernel32.ReleaseMutex(handle) + ctypes.windll.kernel32.CloseHandle(handle) + _DELIVERY_LOCK.release() + + +class EmailDeliveryLedger: + def __init__(self, root: Path | str): + self.root = Path(root) + + def path_for(self, identity: EmailDeliveryIdentity) -> Path: + return self.root / identity.key[:2] / f"{identity.key}.json" + + @staticmethod + def _read(path: Path) -> tuple[dict, bool]: + if not path.is_file(): + return {}, False + try: + data = json.loads(path.read_text(encoding="utf-8")) + except (OSError, UnicodeError, json.JSONDecodeError): + return {}, True + return (data, False) if isinstance(data, dict) else ({}, True) + + @staticmethod + def _write(path: Path, data: dict) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + temp = path.with_name(f".{path.name}.{uuid.uuid4().hex}.tmp") + temp.write_text(json.dumps(data, ensure_ascii=False, indent=2), encoding="utf-8") + os.replace(temp, path) + + def transaction(self, identity: EmailDeliveryIdentity) -> "EmailDeliveryTransaction": + return EmailDeliveryTransaction(self, identity) + + +class EmailDeliveryTransaction: + def __init__(self, ledger: EmailDeliveryLedger, identity: EmailDeliveryIdentity): + self.ledger = ledger + self.identity = identity + self.path = ledger.path_for(identity) + self.attempt_id = uuid.uuid4().hex + self.disposition = "prepared" + self.record: dict = {} + self._mutex = _delivery_mutex(self.path) + self._entered = False + + def __enter__(self) -> "EmailDeliveryTransaction": + self._mutex.__enter__() + self._entered = True + try: + record, corrupt = self.ledger._read(self.path) + if corrupt: + self.disposition = "unknown" + self.record = { + "state": "unknown", + "error": "邮件交付账本损坏，禁止自动重发", + } + return self + state = str(record.get("state") or "") + if state == "sent": + self.disposition = "already_sent" + self.record = record + return self + if state in {"submitting", "unknown"}: + self.disposition = "unknown" + self.record = record + return self + + self.record = { + "version": 1, + "state": "prepared", + "attempt_id": self.attempt_id, + "message_id": self.identity.message_id, + "fingerprint": self.identity.fingerprint, + "prepared_at": datetime.now().astimezone().isoformat(), + "pid": os.getpid(), + } + self.ledger._write(self.path, self.record) + return self + except Exception: + self._mutex.__exit__(None, None, None) + self._entered = False + raise + + def _update(self, state: str, **fields) -> None: + self.record.update(state=state, **fields) + self.ledger._write(self.path, self.record) + self.disposition = state + + def mark_prepared(self) -> None: + self._update("prepared", error="") + + def mark_submitting(self) -> None: + self._update( + "submitting", + submitting_at=datetime.now().astimezone().isoformat(), + error="", + ) + + def mark_sent(self) -> None: + self._update( + "sent", + sent_at=datetime.now().astimezone().isoformat(), + error="", + ) + + def mark_failed_before_submit(self, error: str) -> None: + self._update( + "failed_before_submit", + failed_at=datetime.now().astimezone().isoformat(), + error=str(error)[:300], + ) + + def mark_unknown(self, error: str) -> None: + self._update( + "unknown", + unknown_at=datetime.now().astimezone().isoformat(), + error=str(error)[:300], + ) + + def __exit__(self, exc_type, exc, traceback) -> bool: + try: + if exc is not None and self.disposition == "submitting": + self.mark_unknown(str(exc)) + elif exc is not None and self.disposition == "prepared": + self.mark_failed_before_submit(str(exc)) + finally: + if self._entered: + self._mutex.__exit__(exc_type, exc, traceback) + self._entered = False + return False + + +def _close_server(server) -> None: + if server is None: + return + try: + quit_method = getattr(server, "quit", None) + if quit_method is not None: + quit_method() + except Exception: + # sent 已在 quit 前持久化；关闭连接失败不能触发重复发送。 + pass + + +def deliver_email( + message: EmailMessage, + settings: Settings, + *, + ledger: EmailDeliveryLedger | None = None, + max_attempts: int = 2, +) -> EmailDeliveryResult: + """发送一封邮件；只重试 send_message 之前的连接/认证失败。""" + identity = ensure_email_identity(message) + ledger_root = ( + getattr(settings, "output_dir", None) + or getattr(settings, "data_dir", None) + or (Path.cwd() / "output") + ) + ledger = ledger or EmailDeliveryLedger(Path(ledger_root) / ".email-delivery") + + with ledger.transaction(identity) as transaction: + if transaction.disposition == "already_sent": + return EmailDeliveryResult("already_sent", "相同邮件已确认发送，已跳过", identity.message_id) + if transaction.disposition == "unknown": + return EmailDeliveryResult( + "unknown", + str(transaction.record.get("error") or "上次邮件提交结果未知，禁止自动重发"), + identity.message_id, + ) + + last_error = "" + for attempt in range(1, max(1, int(max_attempts)) + 1): + server = None + try: + if settings.email_use_ssl: + server = smtplib.SMTP_SSL( + settings.email_smtp_host, + settings.email_smtp_port, + timeout=30, + ) + else: + server = smtplib.SMTP( + settings.email_smtp_host, + settings.email_smtp_port, + timeout=30, + ) + server.starttls() + if settings.email_smtp_user: + server.login(settings.email_smtp_user, settings.email_smtp_password) + except Exception as exc: + last_error = str(exc)[:300] + transaction.mark_failed_before_submit(last_error) + _close_server(server) + if attempt < max_attempts: + transaction.mark_prepared() + time.sleep(3) + continue + return EmailDeliveryResult( + "failed_before_submit", last_error, identity.message_id + ) + + try: + transaction.mark_submitting() + refused = server.send_message(message) + if refused: + transaction.mark_unknown("SMTP 返回部分或全部收件人拒绝，结果需人工核对") + return EmailDeliveryResult( + "unknown", + "SMTP 返回部分或全部收件人拒绝，结果需人工核对", + identity.message_id, + ) + transaction.mark_sent() + return EmailDeliveryResult("sent", "", identity.message_id) + except Exception as exc: + transaction.mark_unknown(str(exc)) + return EmailDeliveryResult("unknown", str(exc)[:300], identity.message_id) + finally: + _close_server(server) + + return EmailDeliveryResult("failed_before_submit", last_error, identity.message_id) diff --git a/app/services/email_service.py b/app/services/email_service.py index 0064635..82cf39d 100644 --- a/app/services/email_service.py +++ b/app/services/email_service.py @@ -8,8 +8,8 @@ from __future__ import annotations -import smtplib -import time +import smtplib # 兼容旧测试注入；真实发送实现在 email_delivery +import time # 兼容旧测试注入；真实退避实现在 email_delivery from dataclasses import dataclass, field from email.message import EmailMessage from pathlib import Path @@ -22,11 +22,34 @@ from app.db.models import GroupRun, Report, Run from app.image.image_task import detect_image_format, verify_image from app.scheduler.calendar_rules import email_subject, get_report_window +from app.services.email_delivery import EmailDeliveryLedger, deliver_email from app.services.handoff_service import safe_dir_name +from app.services.legacy_v1_policy import require_legacy_v1_write logger = get_logger("groupbrief.email") +def email_delivery_config_error(settings: Settings) -> str: + """返回真实邮件发送前的配置错误；空字符串表示配置完整。""" + if not settings.email_enabled: + return "邮件未启用" + if not str(settings.email_smtp_host or "").strip(): + return "邮件 SMTP 主机未配置" + try: + port = int(settings.email_smtp_port) + except (TypeError, ValueError): + return "邮件 SMTP 端口无效" + if not 1 <= port <= 65535: + return "邮件 SMTP 端口无效" + if not str(settings.email_recipient or "").strip(): + return "邮件收件人未配置" + if not str(settings.email_from or settings.email_smtp_user or "").strip(): + return "邮件发件人未配置" + if settings.email_smtp_user and not settings.email_smtp_password: + return "邮件 SMTP 用户已配置但密码缺失" + return "" + + @dataclass class GroupMailBlock: group_name: str @@ -46,8 +69,13 @@ class EmailBuildResult: class EmailService: - def __init__(self, settings: Settings | None = None): + def __init__( + self, + settings: Settings | None = None, + delivery_ledger: EmailDeliveryLedger | None = None, + ): self.settings = settings or get_settings() + self.delivery_ledger = delivery_ledger def build_email(self, session: Session, run: Run | None = None) -> EmailBuildResult: """读取各群邮件数据；body 只拼接排行榜原文，不再额外包装。""" @@ -68,6 +96,11 @@ def build_email(self, session: Session, run: Run | None = None) -> EmailBuildRes group_runs = session.exec(select(GroupRun).where(GroupRun.run_id == run.id)).all() for gr in group_runs: + if gr.identity_state != "linked" or gr.group_id is None: + missing.append( + f"历史群（旧 ID {gr.legacy_group_id}）：关联已归档，不发送" + ) + continue if gr.ranking_status != "success": missing.append(f"群 {gr.group_id}：排行榜未生成（{gr.ranking_status}）") continue @@ -109,8 +142,14 @@ def build_email(self, session: Session, run: Run | None = None) -> EmailBuildRes ) def send(self, session: Session, run: Run | None = None) -> tuple[bool, str]: - if not self.settings.email_enabled or not self.settings.email_smtp_host: - return False, "邮件未启用或未配置 SMTP" + require_legacy_v1_write( + self.settings, + operation="email.send", + replacement="V2 每日任务或 scripts/send_daily_email.py", + ) + config_error = email_delivery_config_error(self.settings) + if config_error: + return False, config_error result = self.build_email(session, run) if not result.blocks: @@ -119,7 +158,8 @@ def send(self, session: Session, run: Run | None = None) -> tuple[bool, str]: return False, f"存在失败群且 SEND_PARTIAL_REPORT=false：{result.missing[0]}" sent_count = 0 - failed_count = 0 + # 允许发送部分报告不等于可以把缺失群伪装成全量成功。 + failed_count = len(result.missing) details: list[str] = [] for block in result.blocks: try: @@ -198,46 +238,15 @@ def _build_group_message(self, block: GroupMailBlock) -> EmailMessage: return message def _send_group_message(self, message: EmailMessage, max_attempts: int = 2) -> tuple[bool, str]: - """单群 SMTP 失败最多重试两次，最终结果交给 send() 汇总。""" - last_error = "" - for attempt in range(1, max_attempts + 1): - server = None - attempt_error: Exception | None = None - try: - if self.settings.email_use_ssl: - server = smtplib.SMTP_SSL( - self.settings.email_smtp_host, - self.settings.email_smtp_port, - timeout=30, - ) - else: - server = smtplib.SMTP( - self.settings.email_smtp_host, - self.settings.email_smtp_port, - timeout=30, - ) - server.starttls() - if self.settings.email_smtp_user: - server.login( - self.settings.email_smtp_user, - self.settings.email_smtp_password, - ) - server.send_message(message) - except Exception as exc: - attempt_error = exc - last_error = str(exc) - finally: - if server is not None: - try: - quit_method = getattr(server, "quit", None) - if quit_method is not None: - quit_method() - except Exception as exc: - # send_message 已成功时，QUIT 异常不能触发重复发送。 - logger.warning("SMTP 连接关闭失败：%s", str(exc)[:200]) - if attempt_error is None: - return True, "" - logger.warning("邮件发送 attempt %d 失败：%s", attempt, last_error[:200]) - if attempt < max_attempts: - time.sleep(3) - return False, last_error + """兼容 V1 调用形态；实际幂等和重试策略由统一交付模块负责。""" + result = deliver_email( + message, + self.settings, + ledger=self.delivery_ledger, + max_attempts=max_attempts, + ) + if result.success: + return True, result.detail + if result.outcome_unknown: + return False, f"结果未知，已禁止自动重发：{result.detail}" + return False, result.detail diff --git a/app/services/generation_runtime.py b/app/services/generation_runtime.py index 7174485..dc798c6 100644 --- a/app/services/generation_runtime.py +++ b/app/services/generation_runtime.py @@ -4,6 +4,7 @@ import ctypes import os +import re import threading from contextlib import contextmanager from typing import Iterator @@ -19,6 +20,21 @@ class GenerationBusyError(TimeoutError): _WAIT_ABANDONED = 0x80 +def generation_mutex_name() -> str: + """返回当前进程使用的生成锁名。 + + 生产环境不设置命名空间，继续使用历史锁名。测试和离线仿真必须设置 + ``GROUPBRIEF_GENERATION_MUTEX_NAMESPACE``，从而不会与正在运行的 8766 + 实例争抢同一个 Windows 内核对象。 + """ + + raw = os.environ.get("GROUPBRIEF_GENERATION_MUTEX_NAMESPACE", "").strip() + if not raw: + return _MUTEX_NAME + namespace = re.sub(r"[^A-Za-z0-9_.-]+", "-", raw).strip("-.")[:80] + return f"{_MUTEX_NAME}.{namespace}" if namespace else _MUTEX_NAME + + @contextmanager def generation_mutex(timeout_seconds: float = 2.0) -> Iterator[None]: """同时覆盖当前进程线程与 Windows 跨进程实例。""" @@ -30,7 +46,9 @@ def generation_mutex(timeout_seconds: float = 2.0) -> Iterator[None]: owns_handle = False try: if os.name == "nt": - handle = ctypes.windll.kernel32.CreateMutexW(None, False, _MUTEX_NAME) + handle = ctypes.windll.kernel32.CreateMutexW( + None, False, generation_mutex_name() + ) if not handle: raise OSError("无法创建群报生成互斥锁") wait_code = ctypes.windll.kernel32.WaitForSingleObject( diff --git a/app/services/group_provider_config.py b/app/services/group_provider_config.py new file mode 100644 index 0000000..33ced73 --- /dev/null +++ b/app/services/group_provider_config.py @@ -0,0 +1,169 @@ +"""群级历史数据源与 AI Provider 白名单。""" + +from __future__ import annotations + +import shutil + +from app.config.settings import Settings +from app.db.models import Group +from app.ranking.policies import ( + normalize_ranking_policy, + normalize_sender_name_policy, +) + +HISTORY_PROVIDERS = frozenset({"", "wechat_data_analysis", "wechat_cli"}) +AI_PROVIDERS = frozenset({"codex", "deepseek"}) + + +def normalize_history_provider(value: object) -> str: + name = str(value or "").strip().lower().replace("-", "_") + if name not in HISTORY_PROVIDERS: + raise ValueError(f"不支持的历史数据源：{value}") + return name + + +def provider_catalog(settings: Settings) -> dict: + codex_model = str(settings.codex_summary_model or "gpt-5.6-sol").strip() + deepseek_model = str(settings.ai_model or "deepseek-v4-flash").strip() + codex_binary = shutil.which(settings.codex_path or "codex") + return { + "history": [ + { + "provider": "wechat_data_analysis", + "label": "WeChatDataAnalysis", + "available": True, + "capabilities": ["history"], + }, + { + "provider": "wechat_cli", + "label": "wechat-cli", + "available": bool(shutil.which(settings.wechat_cli_path or "wechat-cli")), + "capabilities": ["history"], + }, + ], + "ai": [ + { + "provider": "codex", + "label": "Codex GPT", + "available": bool(codex_binary), + "models": [codex_model], + "capabilities": ["summary", "prompt"], + }, + { + "provider": "deepseek", + "label": "DeepSeek", + "available": bool(settings.ai_api_key), + "models": [deepseek_model], + "capabilities": ["summary", "prompt"], + }, + ], + } + + +def resolve_group_ai_settings( + settings: Settings, + group: Group, + *, + capability: str, +) -> tuple[Settings, dict]: + if capability not in {"summary", "prompt"}: + raise ValueError(f"未知 AI 能力：{capability}") + provider_field = f"{capability}_provider" + model_field = f"{capability}_model" + configured_provider = str(getattr(group, provider_field, "") or "").strip().lower() + configured_model = str(getattr(group, model_field, "") or "").strip() + inherited = not configured_provider and not configured_model + provider = configured_provider + if not provider and configured_model: + # 旧数据库只有 model 字段：按当前受控白名单推导 Provider，不混用未知值。 + if configured_model == str(settings.codex_summary_model or "gpt-5.6-sol").strip(): + provider = "codex" + elif configured_model == str(settings.ai_model or "deepseek-v4-flash").strip(): + provider = "deepseek" + provider = provider or str(settings.summary_provider_primary or "").strip().lower() + if provider in {"codex_gpt", "gpt"}: + provider = "codex" + if provider not in AI_PROVIDERS: + raise ValueError(f"不支持的 {capability} Provider：{configured_provider or provider}") + + default_model = ( + str(settings.codex_summary_model or "gpt-5.6-sol").strip() + if provider == "codex" + else str(settings.ai_model or "deepseek-v4-flash").strip() + ) + model = configured_model or default_model + allowed = { + str(row_model) + for row in provider_catalog(settings)["ai"] + if row["provider"] == provider + for row_model in row["models"] + } + if model not in allowed: + raise ValueError(f"模型 {model} 不在 {provider} 已验证白名单中") + + updates = { + "summary_provider_primary": provider, + "summary_provider_fallback": ( + str(settings.summary_provider_fallback or "").strip().lower() + if provider == "codex" + else "" + ), + } + if provider == "codex": + updates["codex_summary_model"] = model + else: + updates["ai_model"] = model + resolved = settings.model_copy(update=updates) + return resolved, { + "provider": provider, + "model": model, + "inherited": inherited and not configured_model, + "capability": capability, + } + + +def validate_group_provider_values( + values: dict, + settings: Settings, + *, + base: Group | None = None, +) -> dict: + normalized = dict(values) + if "provider_preference" in normalized: + normalized["provider_preference"] = normalize_history_provider( + normalized.get("provider_preference") + ) + schedule_rule = normalized.get("schedule_rule") + if schedule_rule is not None and schedule_rule not in { + "weekday_default", + "daily_previous_day", + }: + raise ValueError(f"不支持的统计周期规则：{schedule_rule}") + if "ranking_count_policy" in normalized: + normalized["ranking_count_policy"] = normalize_ranking_policy( + normalized.get("ranking_count_policy") + ) + if "sender_name_policy" in normalized: + normalized["sender_name_policy"] = normalize_sender_name_policy( + normalized.get("sender_name_policy") + ) + candidate_values = ( + {key: getattr(base, key) for key in Group.model_fields if hasattr(base, key)} + if base is not None + else {} + ) + candidate_values.update( + { + key: value + for key, value in normalized.items() + if key in Group.model_fields + } + ) + candidate = Group(**candidate_values) + for capability in ("summary", "prompt"): + if any( + field in normalized + for field in (f"{capability}_provider", f"{capability}_model") + ): + resolve_group_ai_settings(settings, candidate, capability=capability) + return normalized diff --git a/app/services/handoff_service.py b/app/services/handoff_service.py index 43d8c38..d37868d 100644 --- a/app/services/handoff_service.py +++ b/app/services/handoff_service.py @@ -21,6 +21,7 @@ from app.config.settings import Settings, get_settings from app.core.logging import get_logger +from app.core.path_security import resolve_within, validate_iso_date, validate_path_label from app.db.models import Group from app.scheduler.calendar_rules import ReportWindow from app.services.message_normalizer import NormalizedMessage @@ -33,13 +34,19 @@ def safe_dir_name(name: str, fallback: str = "group") -> str: cleaned = _INVALID_CHARS.sub("-", name).strip("-") - return cleaned[:60] or fallback + cleaned = cleaned[:60] + return fallback if cleaned in {"", ".", ".."} else cleaned class HandoffService: def __init__(self, settings: Settings | None = None): self.settings = settings or get_settings() + def output_day_dir(self, report_date: str) -> Path: + """返回 output 内的合法日期目录，不接受任意路径片段。""" + valid_date = validate_iso_date(report_date, field_name="report_date") + return resolve_within(self.settings.output_dir, valid_date) + def save_outputs( self, group: Group, @@ -49,9 +56,10 @@ def save_outputs( normalized: list[NormalizedMessage], provider: str, ) -> Path: - base_dir = self.settings.output_dir / window.report_date.isoformat() + base_dir = self.output_day_dir(window.report_date.isoformat()) base = group.display_name or group.wechat_group_name or f"group-{group.id}" - group_dir = base_dir / safe_dir_name(base) + validate_path_label(base, field_name="group_name") + group_dir = resolve_within(base_dir, safe_dir_name(base)) group_dir.mkdir(parents=True, exist_ok=True) ranking_file = group_dir / "ranking.txt" @@ -97,28 +105,41 @@ def save_outputs( def list_output_dates(self) -> list[str]: if not self.settings.output_dir.exists(): return [] - return sorted( - (d.name for d in self.settings.output_dir.iterdir() if d.is_dir()), - reverse=True, - ) + dates: list[str] = [] + for directory in self.settings.output_dir.iterdir(): + if not directory.is_dir(): + continue + try: + valid_date = validate_iso_date(directory.name, field_name="report_date") + resolve_within(self.settings.output_dir, directory) + except ValueError: + continue + dates.append(valid_date) + return sorted(dates, reverse=True) def list_group_outputs(self, report_date: str) -> list[dict]: - day_dir = self.settings.output_dir / report_date + day_dir = self.output_day_dir(report_date) if not day_dir.exists(): return [] result = [] - for group_dir in sorted(day_dir.iterdir()): - if not group_dir.is_dir(): + output_root = self.settings.output_dir.resolve() + for directory in sorted(day_dir.iterdir()): + if not directory.is_dir(): + continue + try: + group_dir = resolve_within(day_dir, directory) + except ValueError: continue handoff_file = group_dir / "handoff.json" handoff = {} if handoff_file.exists(): - handoff = json.loads(handoff_file.read_text(encoding="utf-8")) + safe_handoff = resolve_within(group_dir, handoff_file) + handoff = json.loads(safe_handoff.read_text(encoding="utf-8")) result.append( { "date": report_date, "directory": group_dir.name, - "path": str(group_dir), + "path": group_dir.relative_to(output_root).as_posix(), "handoff": handoff, "files": sorted(p.name for p in group_dir.iterdir()), } diff --git a/app/services/history_service.py b/app/services/history_service.py index bc8eea9..e823942 100644 --- a/app/services/history_service.py +++ b/app/services/history_service.py @@ -1,7 +1,7 @@ """历史聊天业务服务。 业务层只依赖 ChatHistoryProvider 接口，不直接依赖任何开源项目内部实现。 -自动降级：主 Provider 失败 → 备用 Provider → Mock。 +自动降级：主 Provider 失败 → 备用 Provider；Mock 仅限显式测试安全闸门。 群名解析（resolve）路径只搜索真实的非 Mock Provider，绝不回落到 fixtures。 """ diff --git a/app/services/legacy_v1_policy.py b/app/services/legacy_v1_policy.py new file mode 100644 index 0000000..71b62e2 --- /dev/null +++ b/app/services/legacy_v1_policy.py @@ -0,0 +1,52 @@ +"""旧 V1 数据库流水线的集中冻结策略。""" + +from __future__ import annotations + +from app.config.settings import Settings +from app.core.logging import get_logger + +logger = get_logger("groupbrief.legacy_v1") + +LEGACY_V1_WRITE_BLOCKED = "LEGACY_V1_WRITE_BLOCKED" + + +class LegacyV1WriteBlockedError(RuntimeError): + def __init__(self, operation: str, replacement: str): + self.operation = operation + self.replacement = replacement + super().__init__( + f"旧 V1 写入已冻结：{operation}；请使用 {replacement}" + ) + + def __str__(self) -> str: + return str(self.args[0]) + + def as_detail(self) -> dict[str, str]: + return { + "code": LEGACY_V1_WRITE_BLOCKED, + "message": str(self), + "operation": self.operation, + "replacement": self.replacement, + } + + +def require_legacy_v1_write( + settings: Settings, + *, + operation: str, + replacement: str, +) -> None: + """只允许环境级 maintenance 模式执行旧写入。""" + if settings.legacy_v1_write_mode == "maintenance": + logger.warning( + "旧 V1 maintenance 写入已启用：operation=%s replacement=%s", + operation, + replacement, + ) + return + logger.warning( + "旧 V1 写入已阻断：operation=%s replacement=%s", + operation, + replacement, + ) + raise LegacyV1WriteBlockedError(operation, replacement) diff --git a/app/services/prompt_service.py b/app/services/prompt_service.py index f08efea..0d8aea0 100644 --- a/app/services/prompt_service.py +++ b/app/services/prompt_service.py @@ -3,7 +3,7 @@ Codex GPT 主负责：群聊内容 → 理解事件 → 整理话题 → 生成 GPT 生图 Prompt； 主调用失败时使用 DeepSeek 备用。 不负责排行榜计算 / 微信读取 / 邮件 / 调度。 -主备都不可用时优雅降级到本地模板，不阻塞其余流程。 +主备都不可用时真实运行返回失败；本地模板只允许显式测试安全闸门。 """ from __future__ import annotations @@ -40,8 +40,12 @@ def __init__(self, settings: Settings | None = None): def _get_provider(self) -> PromptGeneratorProvider: if self._provider is not None: return self._provider - primary = (self.settings.summary_provider_primary or "codex").strip().lower() - if primary == "deepseek" and not self.settings.ai_api_key: + primary = str(self.settings.summary_provider_primary or "").strip().lower() + if ( + primary == "deepseek" + and not self.settings.ai_api_key + and self.settings.allow_test_providers + ): self._provider = TemplatePromptProvider() else: self._provider = build_summary_provider(self.settings) @@ -54,7 +58,16 @@ def generate( ranking: RankingResult, normalized: list[NormalizedMessage], ) -> PromptOutcome: - provider = self._get_provider() + try: + provider = self._get_provider() + except ValueError as exc: + logger.error("Prompt Provider 配置无效：%s", str(exc)[:200]) + return PromptOutcome( + False, + "", + str(exc), + {"error_type": "SUMMARY_PROVIDER_CONFIG_INVALID"}, + ) message_items = self._build_message_items(normalized) context_text = "\n".join( @@ -76,7 +89,7 @@ def generate( result: ImagePromptResult = provider.generate_image_prompt(context) if result.success: return PromptOutcome(True, result.prompt, meta=result.meta) - if provider.name != "template": + if provider.name != "template" and self.settings.allow_test_providers: template_result = TemplatePromptProvider().generate_image_prompt(context) if template_result.success: logger.warning("主备模型均未完成 Prompt，V1 已降级到本地模板") diff --git a/app/services/report_service.py b/app/services/report_service.py index 93930ed..9395dbc 100644 --- a/app/services/report_service.py +++ b/app/services/report_service.py @@ -13,7 +13,7 @@ from sqlmodel import Session from app.ai.concurrency import bounded_slot, normalized_limit -from app.config.settings import get_settings +from app.config.settings import Settings, get_settings from app.core.logging import get_logger from app.db import repository as repo from app.db.models import Group, GroupRun, Report, Run @@ -24,14 +24,22 @@ from app.services.ranking_service import RankingEngine from app.services.prompt_service import PromptService from app.services.generation_runtime import generation_mutex +from app.services.legacy_v1_policy import require_legacy_v1_write logger = get_logger("app") class ReportService: - def __init__(self, history: HistoryService | None = None, prompt: PromptService | None = None): + def __init__( + self, + history: HistoryService | None = None, + prompt: PromptService | None = None, + *, + settings: Settings | None = None, + ): self.history = history or HistoryService() self.prompt = prompt or PromptService() + self.settings = settings or get_settings() self.ranking = RankingEngine() self.handoff = HandoffService() @@ -46,6 +54,11 @@ def generate( acquire_lock: bool = True, ) -> Run: """生成群报。group=None 表示全部启用群。""" + require_legacy_v1_write( + self.settings, + operation="report.generate", + replacement="POST /api/v2/pipeline/generate", + ) if acquire_lock: with generation_mutex(): return self.generate( @@ -85,7 +98,7 @@ def generate( ) group_specs = [(int(g.id), g.display_name or g.wechat_group_name) for g in groups if g.id is not None] - settings = get_settings() + settings = self.settings group_limit = normalized_limit(settings.generation_group_concurrency, 5) logger.info( "V1 开始并行生成：groups=%d group_limit=%d fetch_limit=%d ai_limit=%d", @@ -192,7 +205,7 @@ def _generate_one( # 1. 读取聊天 wechat_id = group.wechat_group_id or group.wechat_group_name or group.display_name fetch_started = perf_counter() - with bounded_slot("wechat_fetch", get_settings().wechat_fetch_concurrency): + with bounded_slot("wechat_fetch", self.settings.wechat_fetch_concurrency): outcome = self.history.fetch( group.wechat_group_id or wechat_id, group.wechat_group_name or group.display_name, diff --git a/app/services/runtime_logs.py b/app/services/runtime_logs.py new file mode 100644 index 0000000..9af018e --- /dev/null +++ b/app/services/runtime_logs.py @@ -0,0 +1,185 @@ +"""Dashboard 使用的只读、限量、脱敏运行日志投影。""" + +from __future__ import annotations + +from collections import deque +from datetime import datetime +from pathlib import Path +import re +from zoneinfo import ZoneInfo, ZoneInfoNotFoundError + + +LOG_SOURCE_FILES = { + "scheduler": "scheduler.log", + "app": "app.log", + "provider": "provider.log", + "ai": "ai.log", +} +LOG_LEVELS = frozenset({"DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL"}) + +_LOG_LINE = re.compile( + r"^(?P\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}(?:,\d{3,6})?) " + r"\[(?P[A-Z]+)] (?P[^:]+): (?P.*)$" +) +_SENSITIVE_ASSIGNMENT = re.compile( + r"(?i)(?P[\"']?(?:api[_-]?key|access[_-]?token|refresh[_-]?token|token|password|passwd|cookie|authorization|prompt)[\"']?\s*[:=]\s*)" + r"(?P\"[^\"]*\"|'[^']*'|[^\s,;]+)" +) +_BEARER = re.compile(r"(?i)\bBearer\s+[^\s,;]+") +_URL_CREDENTIALS = re.compile(r"(?i)(://)[^/\s:@]+:[^@\s/]+@") +_MAX_MESSAGE_LENGTH = 500 +_CURRENT_DATE_READ_LINES = 5000 + + +def _selection(value: str | None, allowed: set[str] | frozenset[str], *, upper: bool) -> list[str]: + if value is None or not value.strip(): + return sorted(allowed) + selected = [] + for raw in value.split(","): + item = raw.strip() + item = item.upper() if upper else item.lower() + if not item: + continue + if item not in allowed: + raise ValueError(f"不支持的日志筛选值：{item}") + if item not in selected: + selected.append(item) + if not selected: + raise ValueError("日志筛选值不能为空") + return selected + + +def _redact(message: str) -> tuple[str, bool]: + value = _BEARER.sub("Bearer [REDACTED]", message) + value = _URL_CREDENTIALS.sub(r"\1[REDACTED]@", value) + value = _SENSITIVE_ASSIGNMENT.sub( + lambda match: f"{match.group('prefix')}[REDACTED]", + value, + ) + truncated = len(value) > _MAX_MESSAGE_LENGTH + if truncated: + value = f"{value[:_MAX_MESSAGE_LENGTH - 1]}…" + return value, truncated or value != message + + +def _parse_timestamp(value: str, timezone: ZoneInfo) -> datetime | None: + normalized = value.replace(",", ".") + try: + parsed = datetime.fromisoformat(normalized) + except ValueError: + return None + return parsed.replace(tzinfo=timezone) + + +def _read_lines(path: Path, *, current_date: bool) -> list[str]: + try: + with path.open("r", encoding="utf-8", errors="replace") as handle: + if current_date: + return list(deque(handle, maxlen=_CURRENT_DATE_READ_LINES)) + return list(handle) + except OSError: + return [] + + +def _parse_entries( + lines: list[str], + *, + source: str, + run_date: str, + levels: set[str], + timezone: ZoneInfo, +) -> list[dict]: + entries: list[dict] = [] + current: dict | None = None + + def flush() -> None: + nonlocal current + if current is None: + return + if current["run_date"] == run_date and current["level"] in levels: + message, redacted_or_truncated = _redact(current["message"]) + entries.append( + { + "timestamp": current["timestamp"], + "level": current["level"], + "source": source, + "message": message, + "redacted_or_truncated": redacted_or_truncated, + "_sort_key": current["sort_key"], + } + ) + current = None + + for raw_line in lines: + line = raw_line.rstrip("\r\n") + match = _LOG_LINE.match(line) + if match: + flush() + parsed = _parse_timestamp(match.group("timestamp"), timezone) + if parsed is None: + continue + level = match.group("level").upper() + if level == "WARN": + level = "WARNING" + current = { + "timestamp": parsed.isoformat(), + "run_date": parsed.date().isoformat(), + "level": level, + "message": match.group("message"), + "sort_key": parsed, + } + elif current is not None and line: + current["message"] = f"{current['message']}\n{line}" + flush() + return entries + + +def read_runtime_logs( + logs_dir: Path, + run_date: str, + *, + tail: int = 100, + sources: str | None = None, + levels: str | None = None, + app_timezone: str = "Asia/Shanghai", +) -> dict: + """读取固定分类日志；不接受调用方提供的文件名或路径。""" + + if tail < 1 or tail > 200: + raise ValueError("tail 必须在 1 到 200 之间") + try: + timezone = ZoneInfo(app_timezone) + except ZoneInfoNotFoundError as exc: + raise ValueError("日志时区配置无效") from exc + + selected_sources = _selection(sources, set(LOG_SOURCE_FILES), upper=False) + selected_levels = set(_selection(levels, LOG_LEVELS, upper=True)) + today = datetime.now(timezone).date().isoformat() + entries: list[dict] = [] + for source in selected_sources: + path = Path(logs_dir) / LOG_SOURCE_FILES[source] + entries.extend( + _parse_entries( + _read_lines(path, current_date=run_date == today), + source=source, + run_date=run_date, + levels=selected_levels, + timezone=timezone, + ) + ) + + entries.sort(key=lambda item: (item["_sort_key"], item["source"])) + result_truncated = len(entries) > tail + selected = entries[-tail:] + sanitized = [] + for item in selected: + row = dict(item) + row.pop("_sort_key", None) + sanitized.append(row) + return { + "run_date": run_date, + "updated_at": datetime.now(timezone).isoformat(), + "items": sanitized, + "truncated": result_truncated + or any(item["redacted_or_truncated"] for item in sanitized), + } diff --git a/app/services/sender_name_policy.py b/app/services/sender_name_policy.py new file mode 100644 index 0000000..dbb7e0d --- /dev/null +++ b/app/services/sender_name_policy.py @@ -0,0 +1,60 @@ +"""将群级发言人名称策略应用到已读取的消息。""" + +from __future__ import annotations + +from app.data_sources.base import V2Message +from app.providers.history.wechat_data_analysis import ( + _sanitize_sender_name, + _usable_sender_name, +) +from app.ranking.policies import ( + SENDER_NAME_POLICY_WECHAT_DATA_ANALYSIS, + normalize_sender_name_policy, +) +from app.services.speaker_identity import build_speaker_stats, speaker_identity_key + + +def apply_sender_name_policy( + messages: list[V2Message], + policy: object, +) -> list[V2Message]: + """就地应用名称策略并返回消息；默认策略保持 Provider 结果不变。""" + normalized = normalize_sender_name_policy(policy) + if normalized != SENDER_NAME_POLICY_WECHAT_DATA_ANALYSIS: + return messages + + stats = build_speaker_stats( + ( + message.sender_id, + _sanitize_sender_name(message.sender_name), + message.sender_name_source == "contact", + ) + for message in messages + ) + display_names = {item.key: item.name for item in stats} + seen_keys: set[tuple[str, str]] = set() + usable_keys: set[tuple[str, str]] = set() + for message in messages: + key = speaker_identity_key(message.sender_id, message.sender_name) + if key is None: + continue + seen_keys.add(key) + name = _sanitize_sender_name(message.sender_name) + is_contact = message.sender_name_source == "contact" + contact_usable = bool( + is_contact + and name + and name.casefold() not in {"none", "null", "(未知)", "未知"} + ) + if contact_usable or _usable_sender_name(name, str(message.sender_id or "")): + usable_keys.add(key) + anonymous_keys = seen_keys - usable_keys + + for message in messages: + key = speaker_identity_key(message.sender_id, message.sender_name) + if key is None: + continue + message.sender_name = display_names[key] + if key in anonymous_keys: + message.sender_name_source = "anonymous" + return messages diff --git a/app/services/speaker_identity.py b/app/services/speaker_identity.py index b6b5072..a8021f2 100644 --- a/app/services/speaker_identity.py +++ b/app/services/speaker_identity.py @@ -52,17 +52,25 @@ def speaker_name_sort_key(name: str) -> tuple[str, str]: return ((visible_core or str(name or "")).casefold(), str(name or "")) -def _usable_name(name: str, key: IdentityKey) -> bool: +def _usable_name(name: str, key: IdentityKey, *, trusted: bool = False) -> bool: normalized = name.strip() if not normalized or normalized.casefold() in {"none", "null", "(未知)", "未知"}: return False - return key[0] != "id" or normalized.casefold() != key[1].casefold() + return trusted or key[0] != "id" or normalized.casefold() != key[1].casefold() -def build_speaker_stats(records: Iterable[tuple[object, object]]) -> list[SpeakerStat]: - """按身份聚合记录，并为重名身份生成不重复的确定性展示名。""" +def build_speaker_stats( + records: Iterable[tuple[object, object] | tuple[object, object, object]], +) -> list[SpeakerStat]: + """按身份聚合记录，并为重名身份生成不重复的确定性展示名。 + + 第三个可选值表示名称来自可信解析源。联系人名称即使仅与微信 ID + 大小写不同也应保留；未提供该标志的旧调用继续执行原有匿名保护。 + """ aggregated: dict[IdentityKey, dict[str, object]] = {} - for index, (sender_id, sender_name) in enumerate(records): + for index, record in enumerate(records): + sender_id, sender_name = record[0], record[1] + trusted = bool(record[2]) if len(record) > 2 else False key = speaker_identity_key(sender_id, sender_name) if key is None: continue @@ -77,7 +85,7 @@ def build_speaker_stats(records: Iterable[tuple[object, object]]) -> list[Speake aggregated[key] = current current["count"] = int(current["count"]) + 1 name = str(sender_name or "").strip() - if _usable_name(name, key): + if _usable_name(name, key, trusted=trusted): names = current["names"] name_first_index = current["name_first_index"] assert isinstance(names, Counter) diff --git a/app/v2/constants.py b/app/v2/constants.py index 76d09f7..6040b8a 100644 --- a/app/v2/constants.py +++ b/app/v2/constants.py @@ -15,6 +15,7 @@ READY_TO_SEND = "READY_TO_SEND" # 内容齐备，等待发送 SENT = "SENT" # 已发送完成 FAILED = "FAILED" # 失败（failed_stage 记录失败阶段） +CORRUPT = "CORRUPT" # 状态文件存在但损坏；只读隔离，禁止自动推进 STATUS_FLOW = ( PENDING, @@ -25,11 +26,31 @@ READY_TO_SEND, SENT, FAILED, + CORRUPT, ) # 是否允许跳过生图直接进入发送就绪（image_enabled=false 时 PROMPT_READY → READY_TO_SEND） # 已发送状态不可再发送 -TERMINAL_STATUSES = frozenset({SENT, FAILED}) +TERMINAL_STATUSES = frozenset({SENT, FAILED, CORRUPT}) + +# ---------- 正交执行状态（不替换上面的阶段状态） ---------- +EXECUTION_ACTIVE = "ACTIVE" +EXECUTION_WAIT_RETRY = "WAIT_RETRY" +EXECUTION_HOLD_MANUAL = "HOLD_MANUAL" +EXECUTION_COMPLETE = "COMPLETE" +EXECUTION_FAILED_FINAL = "FAILED_FINAL" + +EXECUTION_STATES = frozenset( + { + EXECUTION_ACTIVE, + EXECUTION_WAIT_RETRY, + EXECUTION_HOLD_MANUAL, + EXECUTION_COMPLETE, + EXECUTION_FAILED_FINAL, + } +) + +DEFAULT_RETRY_BUDGET = 3 # ---------- V2 错误类型 ---------- @@ -42,9 +63,12 @@ PROMPT_FAILED = "PROMPT_FAILED" IMAGE_GENERATION_FAILED = "IMAGE_GENERATION_FAILED" IMAGE_FILE_MISSING = "IMAGE_FILE_MISSING" +IMAGE_CONTENT_VERIFICATION_FAILED = "IMAGE_CONTENT_VERIFICATION_FAILED" WECHAT_OFFLINE = "WECHAT_OFFLINE" SEND_TEXT_FAILED = "SEND_TEXT_FAILED" SEND_IMAGE_FAILED = "SEND_IMAGE_FAILED" +RUN_STATE_CORRUPT = "RUN_STATE_CORRUPT" +SCHEDULER_STATE_CORRUPT = "SCHEDULER_STATE_CORRUPT" # ---------- V2 输出文件命名（output/{群}/{日期}/） ---------- FILE_MESSAGES = "messages.json" @@ -52,6 +76,7 @@ FILE_RANKING_TXT = "ranking.txt" FILE_PROMPT = "image_prompt.txt" FILE_PROMPT_ORIGINAL = "image_prompt.original.txt" +FILE_PROMPT_SAFE = "image_prompt.safe.txt" FILE_IMAGE = "daily_image.png" FILE_IMAGE_PREVIOUS = "daily_image.previous.png" FILE_IMAGE_REGENERATING = "daily_image.regenerating.png" diff --git a/app/v2/recovery.py b/app/v2/recovery.py index 68c72ac..2c4d3c2 100644 --- a/app/v2/recovery.py +++ b/app/v2/recovery.py @@ -12,12 +12,14 @@ from app.core.logging import get_logger from app.v2.constants import ( + CORRUPT, FILE_IMAGE, FILE_PROMPT, FILE_RANKING_TXT, FAILED, IMAGE_READY, READY_TO_SEND, + RUN_STATE_CORRUPT, SENT, ) from app.v2.run_store import RunStore @@ -59,7 +61,12 @@ def _required_files(status: str, run: dict) -> list[str]: return required -def scan_incomplete(store: RunStore, run_date: str | None = None) -> list[dict]: +def scan_incomplete( + store: RunStore, + run_date: str | None = None, + *, + runs: list[dict] | None = None, +) -> list[dict]: """找出未到终态的 run（需要恢复/重跑的）。 排除 SENT（绝不重发）与 FAILED（保留错误，手动重跑）。 @@ -69,8 +76,26 @@ def scan_incomplete(store: RunStore, run_date: str | None = None) -> list[dict]: 返回按更新时间排序的列表。 """ incomplete: list[dict] = [] - for run in store.list_runs(run_date): + source_runs = store.list_runs(run_date) if runs is None else runs + for run in source_runs: status = run.get("status", "") + if status == CORRUPT: + item = dict(run) + item["recovery_type"] = "manual_review" + incomplete.append(item) + continue + if run.get("prompt_hold") or run.get("prompt_operation_status") == "unknown": + item = dict(run) + item["recovery_type"] = "manual_review" + item["error_type"] = "PROMPT_RESULT_UNKNOWN" + incomplete.append(item) + continue + if run.get("send_hold_reason") == "SEND_RESULT_UNKNOWN" or run.get("send_state") == "unknown": + item = dict(run) + item["recovery_type"] = "manual_review" + item["error_type"] = "SEND_RESULT_UNKNOWN" + incomplete.append(item) + continue if status in (SENT, FAILED): continue item = dict(run) @@ -82,11 +107,30 @@ def scan_incomplete(store: RunStore, run_date: str | None = None) -> list[dict]: return incomplete -def verify_output(store: RunStore, run_date: str | None = None) -> list[dict]: +def verify_output( + store: RunStore, + run_date: str | None = None, + *, + runs: list[dict] | None = None, +) -> list[dict]: """检查所有 run 的输出文件完整性，返回 [{group_name, run_date, status, missing, ok}]。""" results: list[dict] = [] - for run in store.list_runs(run_date): + source_runs = store.list_runs(run_date) if runs is None else runs + for run in source_runs: status = run.get("status", "") + if status == CORRUPT: + results.append( + { + "group_name": run.get("group_name", "未知群"), + "run_date": run.get("run_date", ""), + "status": CORRUPT, + "missing": [], + "ok": False, + "error_type": RUN_STATE_CORRUPT, + "detail": "运行状态文件损坏，需人工复核", + } + ) + continue required = _required_files(status, run) missing = [] for f in required: @@ -122,13 +166,32 @@ def recover_incomplete( """ from app.pipeline.daily_pipeline import DailyPipeline - pipeline = DailyPipeline() incomplete = scan_incomplete(store, run_date) if not incomplete: return [{"status": "ok", "detail": "无未完成任务"}] results: list[dict] = [] + pipeline = None for run in incomplete: group_name = run["group_name"] + if run.get("recovery_type") == "manual_review": + error_type = str(run.get("error_type") or RUN_STATE_CORRUPT) + results.append( + { + "group_name": group_name, + "status": "blocked", + "error_type": error_type, + "detail": ( + "AI 调用结果未知，需人工复核" + if error_type == "PROMPT_RESULT_UNKNOWN" + else "微信发送结果未知，需人工核对后消歧" + if error_type == "SEND_RESULT_UNKNOWN" + else "运行状态文件损坏，需人工复核" + ), + } + ) + continue + if pipeline is None: + pipeline = DailyPipeline() # 找到群 id（按显示名） gid = _find_group_id_by_name(group_name) if gid is None: diff --git a/app/v2/reliability.py b/app/v2/reliability.py new file mode 100644 index 0000000..95ac608 --- /dev/null +++ b/app/v2/reliability.py @@ -0,0 +1,265 @@ +"""V2 逐群任务的可靠性元数据与自动重试判定。 + +阶段状态继续使用 ``PENDING -> ... -> SENT/FAILED``；本模块只增加正交的 +execution_state、checkpoint 和 retry 元数据，使旧 run.json 可以惰性升级， +同时不会把结果未知的外部调用误判为可自动重试。 +""" + +from __future__ import annotations + +from datetime import datetime, timedelta +from typing import Mapping + +from app.v2.constants import ( + CORRUPT, + DATA_READY, + DEFAULT_RETRY_BUDGET, + EXECUTION_ACTIVE, + EXECUTION_COMPLETE, + EXECUTION_FAILED_FINAL, + EXECUTION_HOLD_MANUAL, + EXECUTION_WAIT_RETRY, + FAILED, + IMAGE_FILE_MISSING, + IMAGE_GENERATION_FAILED, + IMAGE_READY, + MESSAGE_FETCH_FAILED, + PENDING, + PROMPT_FAILED, + PROMPT_READY, + RANKING_FAILED, + RANKING_READY, + READY_TO_SEND, + SENT, + WECHAT_DATA_UNAVAILABLE, +) + +CHECKPOINT_BY_STATUS = { + PENDING: "TASK_CREATED", + DATA_READY: "MESSAGES_SAVED", + RANKING_READY: "RANKING_SAVED", + PROMPT_READY: "PROMPT_SAVED", + IMAGE_READY: "IMAGE_SAVED", + READY_TO_SEND: "IMAGE_SAVED", + SENT: "SENT_CONFIRMED", +} + +STAGE_BY_STATUS = { + PENDING: "DATA", + DATA_READY: "RANKING", + RANKING_READY: "PROMPT", + PROMPT_READY: "IMAGE", + IMAGE_READY: "SEND", + READY_TO_SEND: "SEND", + SENT: "COMPLETE", + FAILED: "FAILED", + CORRUPT: "STATE", +} + +_RETRYABLE_ERROR_TYPES = frozenset( + { + WECHAT_DATA_UNAVAILABLE, + MESSAGE_FETCH_FAILED, + RANKING_FAILED, + PROMPT_FAILED, + IMAGE_GENERATION_FAILED, + IMAGE_FILE_MISSING, + "DEEPSEEK_FAILED", + "UNEXPECTED_GENERATION_ERROR", + "SEND_TEXT_FAILED", + "SEND_IMAGE_FAILED", + "API_TIMEOUT_PRE_SUBMIT", + "API_429", + "API_5XX", + } +) + +_MANUAL_ERROR_TYPES = frozenset( + { + "PROMPT_RESULT_UNKNOWN", + "SEND_RESULT_UNKNOWN", + "RUN_STATE_CORRUPT", + "SCHEDULER_STATE_CORRUPT", + "MESSAGE_SNAPSHOT_INVALID", + "MISSED_SEND_WINDOW", + "GROUP_TARGET_MISMATCH", + } +) + + +def _positive_int(value: object, default: int) -> int: + try: + parsed = int(value) + except (TypeError, ValueError): + return default + return parsed if parsed > 0 else default + + +def _is_unknown_external_result(data: Mapping[str, object]) -> bool: + image_job = data.get("image_job") + return bool( + data.get("prompt_hold") + or data.get("send_hold_reason") == "SEND_RESULT_UNKNOWN" + or data.get("send_state") == "unknown" + or ( + isinstance(image_job, Mapping) + and image_job.get("status") == "result_unknown" + ) + ) + + +def failure_is_manual(data: Mapping[str, object]) -> bool: + error_type = str(data.get("error_type") or data.get("send_error_type") or "") + error = str(data.get("error") or data.get("send_error") or "") + return bool( + error_type in _MANUAL_ERROR_TYPES + or _is_unknown_external_result(data) + or "未绑定微信群" in error + ) + + +def failure_is_retryable(data: Mapping[str, object]) -> bool: + if failure_is_manual(data): + return False + error_type = str(data.get("error_type") or data.get("send_error_type") or "") + return error_type in _RETRYABLE_ERROR_TYPES + + +def retry_delay_seconds(attempt_count: int) -> int: + """有限指数退避：1m、5m、30m，之后保持 30m。""" + schedule = (60, 300, 1800) + index = min(max(int(attempt_count), 1), len(schedule)) - 1 + return schedule[index] + + +def enrich_run_state( + data: Mapping[str, object], + previous: Mapping[str, object] | None = None, + *, + now: datetime | None = None, +) -> dict: + """为即将保存的 run 快照补齐可靠性字段。 + + 同一 failure_fingerprint 的后续状态补写不会重复消耗 retry budget。 + """ + result = dict(data) + previous = dict(previous or {}) + now = now or datetime.now().astimezone() + status = str(result.get("status") or PENDING) + result.setdefault("reliability_schema_version", 1) + result.setdefault("retry_budget", _positive_int(previous.get("retry_budget"), DEFAULT_RETRY_BUDGET)) + result.setdefault("retry_attempt_count", _positive_int(previous.get("retry_attempt_count"), 0) if previous.get("retry_attempt_count") else 0) + result["stage"] = STAGE_BY_STATUS.get(status, str(result.get("stage") or "UNKNOWN")) + + checkpoint = CHECKPOINT_BY_STATUS.get(status) + if checkpoint: + result["last_successful_checkpoint"] = checkpoint + + if status == SENT: + result.update( + execution_state=EXECUTION_COMPLETE, + retryable=False, + next_retry_at="", + ) + return result + if status == CORRUPT: + result.update( + execution_state=EXECUTION_HOLD_MANUAL, + retryable=False, + manual_hold=True, + next_retry_at="", + ) + return result + if status != FAILED: + result.update( + execution_state=EXECUTION_ACTIVE, + retryable=False, + manual_hold=False, + next_retry_at="", + ) + return result + + result.setdefault( + "last_successful_checkpoint", + str(previous.get("last_successful_checkpoint") or "TASK_CREATED"), + ) + error_type = str(result.get("error_type") or result.get("send_error_type") or "") + failed_stage = str(result.get("failed_stage") or "unknown") + error_summary = str(result.get("error") or result.get("send_error") or "")[:300] + fingerprint = f"{failed_stage}|{error_type}|{error_summary}" + previous_fingerprint = str(previous.get("failure_fingerprint") or "") + attempts = int(previous.get("retry_attempt_count") or 0) + if fingerprint != previous_fingerprint: + attempts += 1 + budget = _positive_int(result.get("retry_budget"), DEFAULT_RETRY_BUDGET) + result.update( + failure_fingerprint=fingerprint, + last_error_type=error_type, + last_error_summary=error_summary, + last_failure_at=( + str(previous.get("last_failure_at") or "") + if fingerprint == previous_fingerprint + else now.isoformat() + ), + retry_attempt_count=attempts, + retry_budget=budget, + ) + ledger = list(previous.get("attempt_ledger") or []) + if fingerprint != previous_fingerprint: + ledger.append( + { + "attempt": attempts, + "stage": failed_stage, + "error_type": error_type, + "error_summary": error_summary, + "failed_at": now.isoformat(), + } + ) + result["attempt_ledger"] = ledger[-20:] + + if failure_is_manual(result): + result.update( + execution_state=EXECUTION_HOLD_MANUAL, + retryable=False, + manual_hold=True, + next_retry_at="", + ) + elif failure_is_retryable(result) and attempts < budget: + previous_next_retry_at = str(previous.get("next_retry_at") or "") + next_retry_at = ( + previous_next_retry_at + if fingerprint == previous_fingerprint and previous_next_retry_at + else (now + timedelta(seconds=retry_delay_seconds(attempts))).isoformat() + ) + result.update( + execution_state=EXECUTION_WAIT_RETRY, + retryable=True, + manual_hold=False, + next_retry_at=next_retry_at, + ) + else: + result.update( + execution_state=EXECUTION_FAILED_FINAL, + retryable=False, + manual_hold=False, + next_retry_at="", + ) + return result + + +def retry_is_due(run: Mapping[str, object], now: datetime | None = None) -> bool: + if run.get("execution_state") != EXECUTION_WAIT_RETRY: + return False + now = now or datetime.now().astimezone() + raw = str(run.get("next_retry_at") or "") + if not raw: + return True + try: + retry_at = datetime.fromisoformat(raw) + except ValueError: + return False + if retry_at.tzinfo is None and now.tzinfo is not None: + retry_at = retry_at.replace(tzinfo=now.tzinfo) + if retry_at.tzinfo is not None and now.tzinfo is None: + now = now.replace(tzinfo=retry_at.tzinfo) + return now >= retry_at diff --git a/app/v2/run_store.py b/app/v2/run_store.py index f4321db..dd14bbf 100644 --- a/app/v2/run_store.py +++ b/app/v2/run_store.py @@ -3,6 +3,7 @@ run.json 是每个群每次运行的唯一状态文件（路线文档 §十）。 状态机：PENDING → DATA_READY → RANKING_READY → PROMPT_READY → IMAGE_READY → READY_TO_SEND → SENT / FAILED。 +已有状态文件无法可信解析时进入合成的 CORRUPT 只读隔离态，不参与自动推进。 同时统一管理该群该日期的输出文件命名与目录。 """ @@ -12,16 +13,23 @@ import json import hashlib import os -import re import threading +import time import uuid from contextlib import contextmanager from datetime import datetime, timedelta from pathlib import Path from typing import Iterator +from app.core.path_security import ( + _resolved_path, + resolve_within, + validate_iso_date, + validate_path_label, +) from app.services.handoff_service import safe_dir_name from app.v2.constants import ( + CORRUPT, FILE_IMAGE, FILE_IMAGE_PREVIOUS, FILE_IMAGE_REGENERATING, @@ -31,25 +39,70 @@ FILE_RANKING_JSON, FILE_RANKING_TXT, FILE_RUN, + IMAGE_READY, PENDING, + PROMPT_READY, + READY_TO_SEND, + RUN_STATE_CORRUPT, + SENT, + STATUS_FLOW, ) +from app.v2.reliability import enrich_run_state, retry_delay_seconds -_ISO_DATE = re.compile(r"^\d{4}-\d{2}-\d{2}$") _RUN_WRITE_LOCK = threading.RLock() _WAIT_OBJECT_0 = 0 _WAIT_ABANDONED = 0x80 +_WAIT_FAILED = 0xFFFFFFFF +_PERSISTED_STATUSES = frozenset(STATUS_FLOW) - {CORRUPT} + + +class RunStateCorruptionError(RuntimeError): + """已有 run.json 损坏；任何自动写入都必须 fail closed。""" def validate_run_date(value: str) -> str: """校验 V2 运行目录日期，拒绝路径段和不存在的日历日期。""" - if not isinstance(value, str) or not _ISO_DATE.fullmatch(value): - raise ValueError("run_date 必须是有效的 YYYY-MM-DD 日期") + return validate_iso_date(value, field_name="run_date") + + +def _atomic_write_text(path: Path, content: str, *, encoding: str = "utf-8") -> None: + """使用唯一 staging 文件原子替换目标，且不遗留失败临时文件。""" + + temp = path.with_name(f".{path.name}.{os.getpid()}.{uuid.uuid4().hex}.tmp") try: - datetime.strptime(value, "%Y-%m-%d") - except ValueError as exc: - raise ValueError("run_date 必须是有效的 YYYY-MM-DD 日期") from exc - return value + temp.write_text(content, encoding=encoding) + for attempt in range(8): + try: + os.replace(temp, path) + break + except PermissionError: + if attempt == 7 or not temp.exists(): + raise + time.sleep(min(0.01 * (2**attempt), 0.5)) + finally: + try: + temp.unlink(missing_ok=True) + except OSError: + pass + + +def _windows_kernel32(): + """返回声明了 64 位 HANDLE 签名的 Windows 同步 API。""" + + import ctypes + from ctypes import wintypes + + kernel32 = ctypes.WinDLL("kernel32", use_last_error=True) + kernel32.CreateMutexW.argtypes = [wintypes.LPVOID, wintypes.BOOL, wintypes.LPCWSTR] + kernel32.CreateMutexW.restype = wintypes.HANDLE + kernel32.WaitForSingleObject.argtypes = [wintypes.HANDLE, wintypes.DWORD] + kernel32.WaitForSingleObject.restype = wintypes.DWORD + kernel32.ReleaseMutex.argtypes = [wintypes.HANDLE] + kernel32.ReleaseMutex.restype = wintypes.BOOL + kernel32.CloseHandle.argtypes = [wintypes.HANDLE] + kernel32.CloseHandle.restype = wintypes.BOOL + return kernel32 @contextmanager @@ -63,26 +116,29 @@ def _run_mutex(path: Path, timeout_seconds: float = 10.0) -> Iterator[None]: raise TimeoutError(f"等待运行状态锁超时：{path}") handle = None owns_handle = False + kernel32 = None + digest = "" try: if os.name == "nt": import ctypes - digest = hashlib.sha256(str(path.resolve()).lower().encode("utf-8")).hexdigest()[:32] - handle = ctypes.windll.kernel32.CreateMutexW(None, False, f"Local\\GroupBrief.Run.{digest}") + digest = hashlib.sha256(str(_resolved_path(path)).lower().encode("utf-8")).hexdigest()[:32] + kernel32 = _windows_kernel32() + handle = kernel32.CreateMutexW(None, False, f"Local\\GroupBrief.Run.{digest}") if not handle: - raise OSError(f"无法创建运行状态互斥锁：{path}") - wait_code = ctypes.windll.kernel32.WaitForSingleObject(handle, int(timeout_seconds * 1000)) + raise ctypes.WinError(ctypes.get_last_error()) + wait_code = kernel32.WaitForSingleObject(handle, int(timeout_seconds * 1000)) + if wait_code == _WAIT_FAILED: + raise ctypes.WinError(ctypes.get_last_error()) if wait_code not in (_WAIT_OBJECT_0, _WAIT_ABANDONED): raise TimeoutError(f"等待运行状态互斥锁超时：{path}") owns_handle = True yield finally: - if handle: - import ctypes - + if handle and kernel32 is not None: if owns_handle: - ctypes.windll.kernel32.ReleaseMutex(handle) - ctypes.windll.kernel32.CloseHandle(handle) + kernel32.ReleaseMutex(handle) + kernel32.CloseHandle(handle) _RUN_WRITE_LOCK.release() @@ -116,9 +172,13 @@ def __init__(self, output_root: Path | str): # ---------- 路径 ---------- + def _group_root(self, group_name: str) -> Path: + validate_path_label(group_name, field_name="group_name") + return resolve_within(self.root, safe_dir_name(group_name)) + def group_dir(self, group_name: str, run_date: str) -> Path: - validate_run_date(run_date) - return self.root / safe_dir_name(group_name) / run_date + valid_date = validate_run_date(run_date) + return resolve_within(self._group_root(group_name), valid_date) def run_path(self, group_name: str, run_date: str) -> Path: return self.group_dir(group_name, run_date) / FILE_RUN @@ -150,40 +210,358 @@ def regenerating_image_path(self, group_name: str, run_date: str) -> Path: # ---------- run.json ---------- + def _corrupt_run(self, group_name: str, run_date: str, path: Path, reason: str) -> dict: + try: + state_file = str(path.relative_to(self.root)) + except ValueError: + state_file = path.name + try: + updated_at = datetime.fromtimestamp(path.stat().st_mtime).astimezone().isoformat() + except OSError: + updated_at = "" + return { + "group_name": group_name, + "run_date": run_date, + "status": CORRUPT, + "state_status": "corrupt", + "error_type": RUN_STATE_CORRUPT, + "state_error_reason": reason, + "state_file": state_file, + "updated_at": updated_at, + "send_hold": True, + "needs_manual_review": True, + "detail": "运行状态文件损坏，已阻止自动覆盖、生成和发送", + } + + @staticmethod + def _run_schema_error(data: object, run_date: str) -> str | None: + if not isinstance(data, dict): + return "root_not_object" + group_name = data.get("group_name") + if not isinstance(group_name, str) or not group_name.strip(): + return "group_name_invalid" + stored_date = data.get("run_date") + if not isinstance(stored_date, str) or stored_date != run_date: + return "run_date_invalid" + status = data.get("status") + if not isinstance(status, str) or status not in _PERSISTED_STATUSES: + return "status_invalid" + return None + + def _read_run_file(self, path: Path, group_name: str, run_date: str) -> dict: + try: + raw = path.read_text(encoding="utf-8") + except (OSError, UnicodeError): + return self._corrupt_run(group_name, run_date, path, "read_failed") + try: + data = json.loads(raw) + except json.JSONDecodeError: + return self._corrupt_run(group_name, run_date, path, "json_invalid") + schema_error = self._run_schema_error(data, run_date) + if schema_error: + return self._corrupt_run(group_name, run_date, path, schema_error) + return data + + @staticmethod + def _is_corrupt(data: dict) -> bool: + return data.get("status") == CORRUPT and data.get("error_type") == RUN_STATE_CORRUPT + def load_run(self, group_name: str, run_date: str) -> dict: path = self.run_path(group_name, run_date) if path.exists(): - try: - data = json.loads(path.read_text(encoding="utf-8")) - if isinstance(data, dict): - return data - except (json.JSONDecodeError, OSError): - pass + return self._read_run_file(path, group_name, run_date) return {"group_name": group_name, "run_date": run_date, "status": PENDING} def save_run(self, group_name: str, run_date: str, data: dict) -> dict: - with _RUN_WRITE_LOCK: - path = self.run_path(group_name, run_date) + path = self.run_path(group_name, run_date) + with _run_mutex(path): + existing: dict = {} + if path.exists(): + existing = self._read_run_file(path, group_name, run_date) + if self._is_corrupt(existing): + raise RunStateCorruptionError("运行状态文件损坏，禁止自动覆盖") path.parent.mkdir(parents=True, exist_ok=True) + data = dict(data) data.setdefault("group_name", group_name) + data.setdefault("status", PENDING) data["run_date"] = run_date + task_identity = str(data.get("group_id") or "").strip() + if task_identity: + data["group_task_id"] = f"groupbrief:{run_date}:group-{task_identity}" + else: + name_digest = hashlib.sha256(group_name.encode("utf-8")).hexdigest()[:12] + data.setdefault("group_task_id", f"groupbrief:{run_date}:name-{name_digest}") + data["state_version"] = int(existing.get("state_version") or 0) + 1 + data = enrich_run_state(data, existing) + schema_error = self._run_schema_error(data, run_date) + if schema_error: + raise ValueError(f"run.json 写入数据不符合 Schema：{schema_error}") data["updated_at"] = datetime.now().strftime("%Y-%m-%d %H:%M:%S") - temp = path.with_suffix(".json.tmp") - temp.write_text(json.dumps(data, ensure_ascii=False, indent=2), encoding="utf-8") - temp.replace(path) + _atomic_write_text( + path, + json.dumps(data, ensure_ascii=False, indent=2), + ) return data def update(self, group_name: str, run_date: str, **fields) -> dict: """加载 → 合并字段 → 保存，返回最新 run。""" - with _RUN_WRITE_LOCK: + path = self.run_path(group_name, run_date) + with _run_mutex(path): data = self.load_run(group_name, run_date) + if self._is_corrupt(data): + raise RunStateCorruptionError("运行状态文件损坏，禁止自动覆盖") data.update(fields) return self.save_run(group_name, run_date, data) + # ---------- Prompt 外部调用 claim / result ---------- + + def claim_prompt_operation( + self, + group_name: str, + run_date: str, + *, + input_hash: str, + force: bool = False, + ) -> tuple[str | None, dict, str]: + """原子领取一次 Prompt 生成操作。 + + 已开始但没有完成/结果记录的调用一律转为 unknown；普通恢复不得再次 + 调用外部模型。结果已先写入 run.json 时可由调用方无费用地继续提交文件。 + """ + path = self.run_path(group_name, run_date) + with _run_mutex(path): + data = self.load_run(group_name, run_date) + if self._is_corrupt(data): + return None, data, "state_corrupt" + + operation_status = str(data.get("prompt_operation_status") or "") + same_input = data.get("prompt_operation_input_hash") == input_hash + if operation_status == "result_recorded" and same_input: + result = data.get("prompt_operation_result") + if isinstance(result, dict) and isinstance(result.get("prompt"), str): + return None, data, "result_recorded" + if operation_status == "result_recorded" and not same_input: + data.update( + prompt_operation_status="unknown", + prompt_hold=True, + prompt_hold_reason="PROMPT_INPUT_CHANGED_AFTER_RESULT", + prompt_operation_error="AI 结果已记录但恢复输入发生变化，需人工复核", + needs_manual_review=True, + ) + self.save_run(group_name, run_date, data) + return None, data, "result_unknown" + if ( + operation_status == "succeeded" + and same_input + and self.prompt_path(group_name, run_date).is_file() + and not force + ): + return None, data, "already_completed" + if operation_status == "unknown": + return None, data, "result_unknown" + if ( + operation_status == "started" + and not data.get("prompt_operation_finished_at") + ): + data.update( + prompt_operation_status="unknown", + prompt_hold=True, + prompt_hold_reason="PROMPT_RESULT_UNKNOWN", + prompt_operation_error="上次 AI 调用已开始但没有可信结果，禁止自动重复调用", + needs_manual_review=True, + ) + self.save_run(group_name, run_date, data) + return None, data, "result_unknown" + + operation_id = uuid.uuid4().hex + data.update( + prompt_operation_id=operation_id, + prompt_operation_input_hash=input_hash, + prompt_operation_status="started", + prompt_operation_started_at=datetime.now().astimezone().isoformat(), + prompt_operation_finished_at="", + prompt_operation_error="", + prompt_operation_result=None, + prompt_hold=False, + prompt_hold_reason="", + ) + self.save_run(group_name, run_date, data) + return operation_id, data, "claimed" + + def record_prompt_result( + self, + group_name: str, + run_date: str, + operation_id: str, + *, + prompt: str, + meta: dict | None, + ) -> dict: + """在写最终 Prompt 文件前，先持久化已付费调用的结果。""" + path = self.run_path(group_name, run_date) + with _run_mutex(path): + data = self.load_run(group_name, run_date) + if self._is_corrupt(data): + raise RunStateCorruptionError("运行状态文件损坏，禁止记录 Prompt 结果") + if data.get("prompt_operation_id") != operation_id: + raise RuntimeError("Prompt 操作 claim 已失效") + data.update( + prompt_operation_status="result_recorded", + prompt_operation_result={ + "prompt": prompt, + "meta": dict(meta or {}), + "sha256": hashlib.sha256(prompt.encode("utf-8")).hexdigest(), + }, + ) + return self.save_run(group_name, run_date, data) + + def commit_recorded_prompt( + self, + group_name: str, + run_date: str, + operation_id: str, + ) -> dict: + """把已记录结果原子提升为 image_prompt.txt，并完成操作状态。""" + run_path = self.run_path(group_name, run_date) + with _run_mutex(run_path): + data = self.load_run(group_name, run_date) + if self._is_corrupt(data): + raise RunStateCorruptionError("运行状态文件损坏，禁止提交 Prompt 结果") + if data.get("prompt_operation_id") != operation_id: + raise RuntimeError("Prompt 操作 claim 已失效") + result = data.get("prompt_operation_result") + if not isinstance(result, dict) or not isinstance(result.get("prompt"), str): + raise RuntimeError("Prompt 操作没有可提交的已记录结果") + + prompt = result["prompt"] + expected_hash = str(result.get("sha256") or "") + if hashlib.sha256(prompt.encode("utf-8")).hexdigest() != expected_hash: + raise RunStateCorruptionError("已记录 Prompt 结果哈希不一致") + prompt_path = self.prompt_path(group_name, run_date) + prompt_path.parent.mkdir(parents=True, exist_ok=True) + temp_path = prompt_path.with_name(f".{prompt_path.name}.{operation_id}.tmp") + temp_path.write_text(prompt, encoding="utf-8") + os.replace(temp_path, prompt_path) + + data.update( + status=PROMPT_READY, + prompt_meta=dict(result.get("meta") or {}), + prompt_fallback_level=0, + prompt_fallback_reason="", + prompt_original_error="", + image_force_local_fallback=False, + prompt_operation_status="succeeded", + prompt_operation_finished_at=datetime.now().astimezone().isoformat(), + prompt_operation_result=None, + prompt_operation_error="", + prompt_hold=False, + prompt_hold_reason="", + ) + return self.save_run(group_name, run_date, data) + + def fail_prompt_operation( + self, + group_name: str, + run_date: str, + operation_id: str, + *, + error: str, + ) -> dict: + """记录可确认的失败；该状态可以由显式重试重新领取。""" + path = self.run_path(group_name, run_date) + with _run_mutex(path): + data = self.load_run(group_name, run_date) + if data.get("prompt_operation_id") != operation_id: + raise RuntimeError("Prompt 操作 claim 已失效") + data.update( + prompt_operation_status="failed", + prompt_operation_finished_at=datetime.now().astimezone().isoformat(), + prompt_operation_error=str(error)[:300], + prompt_operation_result=None, + prompt_hold=False, + prompt_hold_reason="", + ) + return self.save_run(group_name, run_date, data) + + def mark_prompt_result_unknown( + self, + group_name: str, + run_date: str, + operation_id: str, + *, + error: str, + ) -> dict: + """提交后结果不明时进入人工 hold，finished_at 故意保持为空。""" + path = self.run_path(group_name, run_date) + with _run_mutex(path): + data = self.load_run(group_name, run_date) + if data.get("prompt_operation_id") != operation_id: + raise RuntimeError("Prompt 操作 claim 已失效") + data.update( + prompt_operation_status="unknown", + prompt_operation_error=str(error)[:300], + prompt_hold=True, + prompt_hold_reason="PROMPT_RESULT_UNKNOWN", + needs_manual_review=True, + ) + return self.save_run(group_name, run_date, data) + + def resolve_prompt_result_unknown( + self, + group_name: str, + run_date: str, + *, + expected_operation_id: str, + now: datetime, + ) -> tuple[bool, dict, str]: + """CAS 确认丢弃一次未知 Prompt 结果；本方法不调用外部模型。""" + path = self.run_path(group_name, run_date) + with _run_mutex(path): + data = self.load_run(group_name, run_date) + if self._is_corrupt(data): + return False, data, "state_corrupt" + if ( + data.get("prompt_operation_status") != "unknown" + or not data.get("prompt_hold") + or data.get("prompt_hold_reason") != "PROMPT_RESULT_UNKNOWN" + ): + return False, data, "not_unknown" + operation_id = str(data.get("prompt_operation_id") or "") + if not expected_operation_id or operation_id != expected_operation_id: + return False, data, "stale" + if isinstance(data.get("prompt_operation_result"), dict): + return False, data, "result_available" + + resolved_at = now.isoformat() + history = list(data.get("prompt_resolution_history") or []) + history.append( + { + "operation_id": operation_id, + "resolution": "discard_and_retry", + "resolved_at": resolved_at, + "previous_error": str(data.get("prompt_operation_error") or data.get("error") or ""), + } + ) + data.update( + prompt_operation_status="failed", + prompt_operation_finished_at=resolved_at, + prompt_operation_error="已人工确认丢弃上次未知结果，可显式重新生成", + prompt_operation_result=None, + prompt_hold=False, + prompt_hold_reason="", + prompt_resolution_history=history[-20:], + prompt_last_resolution="discard_and_retry", + prompt_last_resolved_at=resolved_at, + needs_manual_review=False, + ) + if str(data.get("error_type") or "") == "PROMPT_RESULT_UNKNOWN": + data.update(error="", error_type="", failed_stage="") + return True, self.save_run(group_name, run_date, data), "resolved" + def previous_theme_signature(self, group_name: str, run_date: str) -> str: """读取当前日期之前最近一次运行的实际风格签名。""" validate_run_date(run_date) - group_dir = self.root / safe_dir_name(group_name) + group_dir = self._group_root(group_name) if not group_dir.is_dir(): return "" candidates = sorted( @@ -210,7 +588,7 @@ def recent_layout_history( if limit <= 0: return () limit = min(int(limit), 12) - group_dir = self.root / safe_dir_name(group_name) + group_dir = self._group_root(group_name) if not group_dir.is_dir(): return () candidates = sorted( @@ -263,12 +641,38 @@ def claim_send( path = self.run_path(group_name, run_date) with _run_mutex(path): data = self.load_run(group_name, run_date) + if self._is_corrupt(data): + return None, data, "state_corrupt" if data.get("sent_at") and not allow_sent: return None, data, "already_sent" if data.get("send_state") == "unknown": return None, data, "result_unknown" + if data.get("send_state") == "failed_final": + return None, data, "failed_final" if data.get("send_hold") and not allow_hold: return None, data, "send_hold" + prompt_meta = ( + data.get("prompt_meta") + if isinstance(data.get("prompt_meta"), dict) + else {} + ) + snapshot_hash = str(data.get("message_snapshot_sha256") or "") + speaker_fingerprint = str(data.get("speaker_fingerprint") or "") + image_contract_required = bool(data.get("image_enabled", True)) + if ( + data.get("prompt_stale") is not False + or (image_contract_required and data.get("image_stale") is not False) + or not snapshot_hash + or not speaker_fingerprint + or str(prompt_meta.get("message_snapshot_sha256") or "") + != snapshot_hash + or str(prompt_meta.get("speaker_fingerprint") or "") + != speaker_fingerprint + ): + return None, data, "artifact_stale" + retry_at = _parse_timestamp(data.get("send_next_retry_at"), now) + if retry_at and retry_at > now: + return None, data, "retry_not_due" existing_claim = str(data.get("send_claim_id") or "") expires_at = _parse_timestamp(data.get("send_claim_expires_at"), now) @@ -312,6 +716,8 @@ def update_send_claim( path = self.run_path(group_name, run_date) with _run_mutex(path): data = self.load_run(group_name, run_date) + if self._is_corrupt(data): + return False, data if data.get("send_claim_id") != claim_id: return False, data data.update(fields) @@ -336,6 +742,423 @@ def finish_send_claim( ) return self.update_send_claim(group_name, run_date, claim_id, **fields) + def finish_send_failure( + self, + group_name: str, + run_date: str, + claim_id: str, + *, + stage: str, + error_type: str, + detail: str, + now: datetime, + diagnostics: dict | None = None, + **fields, + ) -> tuple[bool, dict, bool]: + """记录一次明确未提交的发送失败，并执行独立的有限重试预算。 + + 只有 Provider 明确返回 ``submitted=False`` 时才允许进入这里。达到预算 + 后进入 ``failed_final`` 人工暂停；未知提交结果仍走 SEND_RESULT_UNKNOWN。 + """ + path = self.run_path(group_name, run_date) + with _run_mutex(path): + data = self.load_run(group_name, run_date) + if self._is_corrupt(data) or data.get("send_claim_id") != claim_id: + return False, data, False + + try: + attempts = max(int(data.get("send_retry_attempt_count") or 0), 0) + 1 + except (TypeError, ValueError): + attempts = 1 + try: + budget = max(int(data.get("send_retry_budget") or 3), 1) + except (TypeError, ValueError): + budget = 3 + final = attempts >= budget + failed_at = now.isoformat() + ledger = list(data.get("send_failure_ledger") or []) + ledger.append( + { + "attempt": attempts, + "stage": stage, + "error_type": error_type, + "error_summary": str(detail)[:300], + "failed_at": failed_at, + } + ) + fields.setdefault(f"{stage}_verification_diagnostics", diagnostics or {}) + fields.update( + send_state="failed_final" if final else "ready", + send_claim_id="", + send_claimed_at="", + send_claim_expires_at="", + send_retry_attempt_count=attempts, + send_retry_budget=budget, + send_failure_ledger=ledger[-20:], + send_last_failure_at=failed_at, + send_next_retry_at=( + "" + if final + else (now + timedelta(seconds=retry_delay_seconds(attempts))).isoformat() + ), + send_hold=final, + send_hold_reason="SEND_RETRY_EXHAUSTED" if final else "", + needs_manual_send=final, + send_error=str(detail)[:500], + send_error_type=error_type, + ) + data.update(fields) + return True, self.save_run(group_name, run_date, data), final + + def reset_explicit_send_failure( + self, + group_name: str, + run_date: str, + *, + expected_updated_at: str, + expected_state_version: int, + now: datetime, + ) -> tuple[bool, dict, str]: + """解除明确未提交的最终发送失败；本方法绝不执行外部发送。 + + ``updated_at`` 与 ``state_version`` 组成 CAS。只接受重试预算耗尽、 + 且文字和图片均没有提交/验证/发送证据的任务；未知结果和已发送状态 + 必须继续人工核对，不能通过本入口重试。 + """ + path = self.run_path(group_name, run_date) + with _run_mutex(path): + data = self.load_run(group_name, run_date) + if self._is_corrupt(data): + return False, data, "state_corrupt" + try: + state_version = int(data.get("state_version") or 0) + attempts = max(int(data.get("send_retry_attempt_count") or 0), 0) + budget = max(int(data.get("send_retry_budget") or 3), 1) + except (TypeError, ValueError): + return False, data, "state_corrupt" + if ( + str(data.get("updated_at") or "") != expected_updated_at + or state_version != expected_state_version + ): + return False, data, "stale" + if data.get("status") not in {IMAGE_READY, READY_TO_SEND}: + return False, data, "not_resolvable" + if ( + data.get("send_state") != "failed_final" + or data.get("send_hold") is not True + or data.get("send_hold_reason") != "SEND_RETRY_EXHAUSTED" + or attempts < budget + ): + return False, data, "not_explicit_failure" + if data.get("send_claim_id"): + return False, data, "active_claim" + if _has_unresolved_send_attempt(data): + return False, data, "unresolved_attempt" + + evidence_fields = ( + "sent_at", + "send_unknown_at", + "text_submitted_at", + "text_verified_at", + "text_sent_at", + "image_submitted_at", + "image_verified_at", + "image_sent_at", + ) + if any(str(data.get(field) or "").strip() for field in evidence_fields): + return False, data, "submission_evidence" + + reset_at = now.isoformat() + history = list(data.get("send_retry_reset_history") or []) + history.append( + { + "reset_at": reset_at, + "expected_updated_at": expected_updated_at, + "expected_state_version": expected_state_version, + "previous_attempt_count": attempts, + "previous_error_type": str(data.get("send_error_type") or ""), + "previous_error": str(data.get("send_error") or "")[:300], + "previous_last_failure_at": str(data.get("send_last_failure_at") or ""), + } + ) + data.update( + send_state="ready", + send_hold=False, + send_hold_reason="", + needs_manual_send=False, + send_retry_attempt_count=0, + send_next_retry_at="", + send_last_failure_at="", + send_error="", + send_error_type="", + send_claim_id="", + send_claimed_at="", + send_claim_expires_at="", + send_retry_reset_history=history[-20:], + send_last_retry_reset_at=reset_at, + ) + return True, self.save_run(group_name, run_date, data), "reset" + + def mark_send_result_unknown( + self, + group_name: str, + run_date: str, + claim_id: str, + *, + stage: str, + detail: str, + submitted_at: str = "", + diagnostics: dict | None = None, + now: datetime, + ) -> tuple[bool, dict, str]: + """在常规 claim CAS 失败后，把已经触发的外部发送 fail-closed。 + + 只有当前 claim、磁盘上仍有未决发送尝试，或最终 SENT 提交失败时才 + 允许进入 unknown。若另一个写入者已经可靠地提交 SENT，则保留 SENT。 + """ + path = self.run_path(group_name, run_date) + with _run_mutex(path): + data = self.load_run(group_name, run_date) + if self._is_corrupt(data): + return False, data, "state_corrupt" + if data.get("status") == SENT or data.get("sent_at"): + return True, data, "already_sent" + if data.get("send_state") == "unknown": + return True, data, "already_unknown" + + owns_claim = data.get("send_claim_id") == claim_id + unresolved = _has_unresolved_send_attempt(data) + if not owns_claim and not unresolved and stage != "finalize": + return False, data, "claim_lost" + + finished_at = now.isoformat() + fields = { + "send_state": "unknown", + "send_hold": True, + "send_hold_reason": "SEND_RESULT_UNKNOWN", + "needs_manual_send": True, + "send_error": str(detail)[:500], + "send_error_type": "SEND_RESULT_UNKNOWN", + "verification_level": "unknown", + "send_unknown_at": finished_at, + "send_unknown_stage": stage, + "send_claim_id": "", + "send_claimed_at": "", + "send_claim_expires_at": "", + f"{stage}_submitted_at": submitted_at, + f"{stage}_verification_diagnostics": diagnostics or {}, + } + if stage in {"text", "image"}: + fields[f"{stage}_attempt_finished_at"] = "" + data.update(fields) + return True, self.save_run(group_name, run_date, data), "marked_unknown" + + def resolve_text_send_unknown( + self, + group_name: str, + run_date: str, + *, + resolution: str, + expected_send_unknown_at: str, + now: datetime, + ) -> tuple[bool, dict, str]: + """用时间戳 CAS 人工消歧文字提交；本方法本身不执行任何发送。""" + path = self.run_path(group_name, run_date) + with _run_mutex(path): + data = self.load_run(group_name, run_date) + if self._is_corrupt(data): + return False, data, "state_corrupt" + if data.get("send_state") != "unknown" or data.get("send_hold_reason") != "SEND_RESULT_UNKNOWN": + return False, data, "not_unknown" + if str(data.get("send_unknown_at") or "") != expected_send_unknown_at: + return False, data, "stale" + unknown_stage = str(data.get("send_unknown_stage") or "") + if not unknown_stage: + unknown_stage = "image" if data.get("image_submitted_at") or data.get("text_sent_at") else "text" + if unknown_stage != "text": + return False, data, "unsupported_stage" + if resolution not in {"text_sent", "not_sent"}: + return False, data, "invalid_resolution" + + resolved_at = now.isoformat() + history = list(data.get("send_resolution_history") or []) + history.append( + { + "stage": "text", + "resolution": resolution, + "unknown_at": expected_send_unknown_at, + "resolved_at": resolved_at, + } + ) + common = { + "send_state": "ready", + "send_hold": False, + "send_hold_reason": "", + "send_error": "", + "send_error_type": "", + "send_unknown_at": "", + "send_unknown_stage": "", + "send_claim_id": "", + "send_claimed_at": "", + "send_claim_expires_at": "", + "needs_manual_send": True, + "send_resolution_history": history[-20:], + "send_last_resolution": resolution, + "send_last_resolved_at": resolved_at, + } + if resolution == "text_sent": + submitted_at = str(data.get("text_submitted_at") or "") + if not submitted_at: + return False, data, "text_not_submitted" + data.update( + **common, + text_attempt_finished_at=resolved_at, + text_verified_at=resolved_at, + text_sent_at=submitted_at, + text_verification_level="manual_ui_observed", + verification_level="manual_ui_observed", + ) + else: + data.update( + **common, + text_attempt_started_at="", + text_attempt_finished_at="", + text_submitted_at="", + text_verified_at="", + text_sent_at="", + text_verification_level="", + verification_level="", + ) + return True, self.save_run(group_name, run_date, data), "resolved" + + def resolve_manual_send( + self, + group_name: str, + run_date: str, + *, + resolution: str, + expected_updated_at: str, + image_required: bool, + now: datetime, + ) -> tuple[bool, dict, str]: + """用 run.updated_at 做 CAS 写入人工发送结论；绝不执行外部发送。""" + path = self.run_path(group_name, run_date) + with _run_mutex(path): + data = self.load_run(group_name, run_date) + if self._is_corrupt(data): + return False, data, "state_corrupt" + if str(data.get("updated_at") or "") != expected_updated_at: + return False, data, "stale" + if data.get("status") not in {IMAGE_READY, READY_TO_SEND}: + return False, data, "not_resolvable" + if not data.get("send_hold"): + return False, data, "not_held" + if resolution not in {"all_sent", "text_sent", "not_sent"}: + return False, data, "invalid_resolution" + + ranking_path = self.ranking_txt_path(group_name, run_date) + if resolution in {"all_sent", "text_sent"} and ( + not ranking_path.exists() or ranking_path.stat().st_size <= 0 + ): + return False, data, "ranking_missing" + image_path = self.image_path(group_name, run_date) + if resolution == "all_sent" and image_required and ( + not image_path.exists() or image_path.stat().st_size <= 0 + ): + return False, data, "image_missing" + + resolved_at = now.isoformat() + history = list(data.get("send_resolution_history") or []) + history.append( + { + "stage": "all" if resolution == "all_sent" else "text" if resolution == "text_sent" else "none", + "resolution": resolution, + "expected_updated_at": expected_updated_at, + "resolved_at": resolved_at, + "previous_send_state": str(data.get("send_state") or ""), + "previous_send_hold_reason": str(data.get("send_hold_reason") or ""), + "previous_send_error": str(data.get("send_error") or data.get("error") or ""), + } + ) + common = { + "send_state": "sent" if resolution == "all_sent" else "ready", + "send_hold": False, + "send_hold_reason": "", + "send_error": "", + "send_error_type": "", + "send_unknown_at": "", + "send_unknown_stage": "", + "send_claim_id": "", + "send_claimed_at": "", + "send_claim_expires_at": "", + "send_resolution_history": history[-20:], + "send_last_resolution": resolution, + "send_last_resolved_at": resolved_at, + "manual_send_resolution": resolution, + "manual_send_resolved_at": resolved_at, + } + if str(data.get("error_type") or "") == "SEND_RESULT_UNKNOWN": + common.update(error="", error_type="", failed_stage="") + + if resolution == "all_sent": + data.update( + **common, + status=SENT, + sent_at=resolved_at, + text_attempt_finished_at=resolved_at, + text_submitted_at=str(data.get("text_submitted_at") or resolved_at), + text_verified_at=resolved_at, + text_sent_at=str(data.get("text_sent_at") or resolved_at), + text_verification_level="manual_user_confirmed", + image_attempt_finished_at=resolved_at if image_required else str(data.get("image_attempt_finished_at") or ""), + image_submitted_at=str(data.get("image_submitted_at") or resolved_at) if image_required else "", + image_verified_at=resolved_at if image_required else "", + image_sent_at=str(data.get("image_sent_at") or resolved_at) if image_required else "", + image_verification_level="manual_user_confirmed" if image_required else "", + verification_level="manual_user_confirmed", + needs_manual_send=False, + ) + elif resolution == "text_sent": + data.update( + **common, + status=READY_TO_SEND, + sent_at="", + text_attempt_finished_at=resolved_at, + text_submitted_at=str(data.get("text_submitted_at") or resolved_at), + text_verified_at=resolved_at, + text_sent_at=str(data.get("text_sent_at") or resolved_at), + text_verification_level="manual_user_confirmed", + image_attempt_started_at="", + image_attempt_finished_at="", + image_submitted_at="", + image_verified_at="", + image_sent_at="", + image_verification_level="", + verification_level="manual_user_confirmed", + needs_manual_send=True, + ) + else: + data.update( + **common, + status=READY_TO_SEND, + sent_at="", + text_attempt_started_at="", + text_attempt_finished_at="", + text_submitted_at="", + text_verified_at="", + text_sent_at="", + text_verification_level="", + image_attempt_started_at="", + image_attempt_finished_at="", + image_submitted_at="", + image_verified_at="", + image_sent_at="", + image_verification_level="", + verification_level="", + needs_manual_send=True, + ) + return True, self.save_run(group_name, run_date, data), "resolved" + def list_runs(self, run_date: str | None = None) -> list[dict]: """列出全部已存在的 run（可按日期过滤）。 @@ -358,8 +1181,17 @@ def list_runs(self, run_date: str | None = None) -> list[dict]: run_path = d / FILE_RUN if run_path.exists(): try: - runs.append(json.loads(run_path.read_text(encoding="utf-8"))) - except (json.JSONDecodeError, OSError): + valid_date = validate_run_date(d.name) + except ValueError: + runs.append( + self._corrupt_run( + group_dir.name, + d.name, + run_path, + "directory_date_invalid", + ) + ) continue + runs.append(self._read_run_file(run_path, group_dir.name, valid_date)) runs.sort(key=lambda r: r.get("updated_at", ""), reverse=True) return runs diff --git a/app/weekly/__init__.py b/app/weekly/__init__.py new file mode 100644 index 0000000..4566dc1 --- /dev/null +++ b/app/weekly/__init__.py @@ -0,0 +1 @@ +"""独立周报聚合、归档与发送。""" diff --git a/app/weekly/service.py b/app/weekly/service.py new file mode 100644 index 0000000..3dab09e --- /dev/null +++ b/app/weekly/service.py @@ -0,0 +1,475 @@ +"""上一自然周洞察：确定性聚合、一次 AI 叙述、本地卡片和独立发送。""" + +from __future__ import annotations + +import hashlib +import json +from collections import Counter +from datetime import date, datetime, time, timedelta +from pathlib import Path +from typing import Callable +from zoneinfo import ZoneInfo + +from PIL import Image, ImageDraw +from sqlmodel import Session + +from app.config.settings import Settings, get_settings +from app.db import repository as repo +from app.db.models import Group +from app.image.fallback import _fit_lines, _load_font +from app.providers.ai.base import ExternalCallResultUnknownError +from app.providers.ai.codex import build_summary_provider +from app.sender.base import WechatSender +from app.sender.wechat_native import create_wechat_sender +from app.services.generation_runtime import generation_mutex +from app.services.group_name_sync import effective_send_target +from app.services.group_provider_config import resolve_group_ai_settings +from app.v2.run_store import RunStore +from app.weekly.store import WeeklyStore + + +def previous_natural_week(reference: date) -> tuple[date, date]: + current_monday = reference - timedelta(days=reference.weekday()) + end = current_monday - timedelta(days=1) + return end - timedelta(days=6), end + + +def _safe_json(path: Path) -> dict: + try: + value = json.loads(path.read_text(encoding="utf-8")) + except (OSError, UnicodeError, json.JSONDecodeError): + return {} + return value if isinstance(value, dict) else {} + + +def _sha256_bytes(value: bytes) -> str: + return hashlib.sha256(value).hexdigest() + + +class WeeklyInsightsService: + def __init__( + self, + settings: Settings | None = None, + *, + daily_store: RunStore | None = None, + weekly_store: WeeklyStore | None = None, + provider_factory: Callable[[Settings], object] | None = None, + sender: WechatSender | None = None, + ) -> None: + self.settings = settings or get_settings() + self.daily_store = daily_store or RunStore(self.settings.output_dir) + self.store = weekly_store or WeeklyStore(self.daily_store.root) + self.provider_factory = provider_factory or build_summary_provider + self.sender = sender or create_wechat_sender(settings=self.settings) + + def generate_previous_week( + self, + *, + now: datetime | None = None, + group_ids: list[int] | None = None, + acquire_lock: bool = True, + ) -> dict: + tz = ZoneInfo(self.settings.app_timezone) + now = now or datetime.now(tz) + if now.tzinfo is None: + now = now.replace(tzinfo=tz) + if acquire_lock: + with generation_mutex(): + return self.generate_previous_week( + now=now, + group_ids=group_ids, + acquire_lock=False, + ) + start, end = previous_natural_week(now.date()) + groups = self._groups(group_ids) + results = [self._generate_group(group, start, end, now) for group in groups] + return { + "status": "complete" if all(item["status"] in {"ready_to_send", "skipped"} for item in results) else "partial", + "week_start": start.isoformat(), + "week_end": end.isoformat(), + "results": results, + } + + def _groups(self, group_ids: list[int] | None = None) -> list[Group]: + repo.init_db(self.settings) + with Session(repo.engine) as session: + groups = repo.list_groups(session, only_enabled=True) + if group_ids is not None: + wanted = {int(value) for value in group_ids} + groups = [group for group in groups if group.id in wanted] + return groups + + def _generate_group( + self, + group: Group, + week_start: date, + week_end: date, + now: datetime, + ) -> dict: + assert group.id is not None + start_text, end_text = week_start.isoformat(), week_end.isoformat() + existing = self.store.load(start_text, end_text, group.id) + if existing.get("status") in {"ready_to_send", "sent", "needs_attention"}: + return { + "group_id": group.id, + "group_name": group.display_name, + "status": "skipped", + "detail": f"周报已处于 {existing.get('status')}", + } + if existing.get("status") in {"building", "sending"}: + # 外部调用或发送中断无法证明是否已提交；重新执行可能产生重复调用。 + error_type = ( + "WEEKLY_AI_RESULT_UNKNOWN" + if existing.get("status") == "building" + else "WEEKLY_SEND_RESULT_UNKNOWN" + ) + self.store.update( + start_text, + end_text, + group.id, + status="needs_attention", + error_type=error_type, + ) + return { + "group_id": group.id, + "group_name": group.display_name, + "status": "skipped", + "detail": "上次周报外部操作中断，已转人工复核", + } + + aggregate = self._aggregate(group, week_start, week_end) + deterministic = self._deterministic_narrative(group, aggregate, start_text, end_text) + ai_status = "not_attempted" + ai_error = "" + narrative = deterministic + actual_provider = "local_deterministic" + actual_model = "none" + ai_call_count = 0 + requested_provider = "" + requested_model = "" + try: + provider_settings, provider_meta = resolve_group_ai_settings( + self.settings, + group, + capability="summary", + ) + # 周报每群最多一次外部调用；不在同一周报里追加 Provider fallback 调用。 + provider_settings = provider_settings.model_copy(update={"summary_provider_fallback": ""}) + requested_provider = provider_meta["provider"] + requested_model = provider_meta["model"] + provider = self.provider_factory(provider_settings) + ai_call_count = 1 + self.store.save( + start_text, + end_text, + group.id, + { + "status": "building", + "group_name": group.display_name, + "ai_attempt_started_at": now.isoformat(), + "ai_call_count": 1, + "aggregation": aggregate, + }, + ) + narrative = str( + provider._chat( + [ + { + "role": "system", + "content": "你只根据聚合统计写一段简洁中文周度洞察，不补充原始聊天或不存在的事实。", + }, + { + "role": "user", + "content": json.dumps(aggregate, ensure_ascii=False, separators=(",", ":")), + }, + ], + response_format="text", + temperature=0.2, + max_tokens=1200, + ) + ).strip() + if not narrative: + raise ValueError("周报 AI 返回空文本") + ai_status = "completed" + actual_provider = str( + getattr(provider, "last_provider_used", "") + or getattr(provider, "name", requested_provider) + ) + actual_model = str(getattr(provider, "model", requested_model)) + except ExternalCallResultUnknownError as exc: + ai_status = "result_unknown" + ai_error = str(exc)[:300] + narrative = deterministic + except Exception as exc: + ai_status = "failed" + ai_error = str(exc)[:300] + narrative = deterministic + + text_path = self.store.text_path(start_text, end_text, group.id) + text_path.parent.mkdir(parents=True, exist_ok=True) + text_path.write_text(narrative, encoding="utf-8") + card_path = self.store.card_path(start_text, end_text, group.id) + self._render_card(group, aggregate, start_text, end_text, card_path) + text_bytes = narrative.encode("utf-8") + card_bytes = card_path.read_bytes() + payload = self.store.save( + start_text, + end_text, + group.id, + { + "status": "ready_to_send", + "group_name": group.display_name, + "wechat_group_id": group.wechat_group_id, + "send_target_snapshot": effective_send_target(group), + "aggregation": aggregate, + "narrative": narrative, + "narrative_source": "ai" if ai_status == "completed" else "local_deterministic", + "ai_status": ai_status, + "ai_error": ai_error, + "ai_call_count": ai_call_count, + "requested_provider": requested_provider, + "requested_model": requested_model, + "actual_provider": actual_provider, + "actual_model": actual_model, + "text_sha256": _sha256_bytes(text_bytes), + "card_sha256": _sha256_bytes(card_bytes), + "generated_at": now.isoformat(), + }, + ) + return { + "group_id": group.id, + "group_name": group.display_name, + "status": payload["status"], + "ai_status": ai_status, + "missing_days": aggregate["missing_days"], + } + + def _aggregate(self, group: Group, week_start: date, week_end: date) -> dict: + contributors: dict[str, dict] = {} + topics: Counter[str] = Counter() + daily: list[dict] = [] + missing_days: list[str] = [] + current = week_start + while current <= week_end: + run_date = current.isoformat() + run = self.daily_store.load_run(group.display_name, run_date) + ranking = _safe_json(self.daily_store.ranking_json_path(group.display_name, run_date)) + if not ranking: + missing_days.append(run_date) + message_count = int(ranking.get("message_count") or run.get("message_count") or 0) + speaker_count = int(ranking.get("speaker_count") or run.get("speaker_count") or 0) + daily.append( + { + "date": run_date, + "message_count": message_count, + "speaker_count": speaker_count, + "status": str(run.get("status") or "MISSING"), + } + ) + rows = ranking.get("top_speakers") if isinstance(ranking.get("top_speakers"), list) else [] + for row in rows: + if not isinstance(row, dict): + continue + name = str(row.get("name") or "(未知)") + identity = str(row.get("identity_key") or f"name:{name.casefold()}") + item = contributors.setdefault(identity, {"identity_key": identity, "name": name, "count": 0}) + item["name"] = name + item["count"] += int(row.get("count") or 0) + prompt_meta = run.get("prompt_meta") if isinstance(run.get("prompt_meta"), dict) else {} + selection = prompt_meta.get("topic_selection") if isinstance(prompt_meta.get("topic_selection"), dict) else {} + candidates = selection.get("candidates") if isinstance(selection.get("candidates"), list) else [] + for candidate in candidates: + if isinstance(candidate, dict) and candidate.get("selected"): + title = str(candidate.get("title") or "").strip() + if title: + topics[title] += 1 + current += timedelta(days=1) + ordered_contributors = sorted( + contributors.values(), + key=lambda item: (-int(item["count"]), str(item["name"]).casefold(), str(item["identity_key"])), + ) + return { + "message_count": sum(item["message_count"] for item in daily), + "daily": daily, + "contributors": ordered_contributors[:10], + "topics": [ + {"title": title, "days": count} + for title, count in sorted(topics.items(), key=lambda item: (-item[1], item[0]))[:8] + ], + "missing_days": missing_days, + "source": "saved_daily_rankings_and_summaries", + "raw_messages_uploaded": False, + } + + @staticmethod + def _deterministic_narrative(group: Group, aggregate: dict, start: str, end: str) -> str: + daily = aggregate["daily"] + peak = max(daily, key=lambda item: (item["message_count"], item["date"])) + contributors = "、".join( + f"{item['name']}（{item['count']}）" for item in aggregate["contributors"][:3] + ) or "暂无可用排行" + topics = "、".join(item["title"] for item in aggregate["topics"][:3]) or "暂无已保存话题摘要" + missing = f"；缺少 {len(aggregate['missing_days'])} 天日报" if aggregate["missing_days"] else "" + return ( + f"{group.display_name}｜{start} 至 {end} 周度洞察\n" + f"本周共 {aggregate['message_count']} 条可统计消息，活跃峰值为 {peak['date']}（{peak['message_count']} 条）{missing}。\n" + f"主要贡献：{contributors}。\n主要话题：{topics}。" + ) + + def _render_card(self, group: Group, aggregate: dict, start: str, end: str, path: Path) -> None: + canvas = Image.new("RGB", (1024, 1280), "#F4F7FB") + draw = ImageDraw.Draw(canvas) + title_font, _ = _load_font(48, self.settings.image_fallback_font_path) + section_font, _ = _load_font(32, self.settings.image_fallback_font_path) + body_font, _ = _load_font(26, self.settings.image_fallback_font_path) + small_font, _ = _load_font(21, self.settings.image_fallback_font_path) + draw.rounded_rectangle((52, 44, 972, 220), radius=28, fill="#173B57") + draw.text((84, 74), str(group.display_name)[:22], font=title_font, fill="white") + draw.text((84, 152), f"{start} — {end} · 每周洞察", font=body_font, fill="#D7EAF7") + draw.rounded_rectangle((52, 252, 972, 390), radius=22, fill="white") + draw.text((84, 280), f"消息 {aggregate['message_count']}", font=section_font, fill="#173B57") + draw.text((520, 280), f"覆盖 {7 - len(aggregate['missing_days'])}/7 天", font=section_font, fill="#173B57") + daily = aggregate["daily"] + max_count = max([int(item["message_count"]) for item in daily] or [1]) or 1 + y = 450 + draw.text((64, y), "活跃趋势", font=section_font, fill="#173B57") + y += 60 + for item in daily: + draw.text((76, y), item["date"][5:], font=small_font, fill="#536879") + width = max(8, int(620 * int(item["message_count"]) / max_count)) + draw.rounded_rectangle((210, y + 2, 210 + width, y + 28), radius=12, fill="#55A7D9") + draw.text((870, y), str(item["message_count"]), font=small_font, fill="#263746") + y += 52 + draw.text((64, 900), "主要贡献", font=section_font, fill="#173B57") + y = 956 + for index, item in enumerate(aggregate["contributors"][:4], start=1): + draw.text((84, y), f"{index}. {item['name']} {item['count']}", font=body_font, fill="#263746") + y += 46 + draw.text((550, 900), "话题变化", font=section_font, fill="#173B57") + y = 956 + for item in aggregate["topics"][:4]: + for line in _fit_lines(draw, f"• {item['title']}", body_font, 390, 1): + draw.text((566, y), line, font=body_font, fill="#263746") + y += 46 + draw.text((64, 1234), "仅聚合已保存日报，不上传整周原始聊天", font=small_font, fill="#758493") + path.parent.mkdir(parents=True, exist_ok=True) + temp = path.with_suffix(".png.tmp") + canvas.save(temp, format="PNG") + temp.replace(path) + + def send_due(self, *, now: datetime | None = None) -> list[dict]: + if not self.settings.weekly_send_enabled: + return [{"status": "not_run", "detail": "周报发送灰度闸门未开启"}] + tz = ZoneInfo(self.settings.app_timezone) + now = now or datetime.now(tz) + if now.tzinfo is None: + now = now.replace(tzinfo=tz) + if now.weekday() != 0: + return [{"status": "not_run", "detail": "今天不是周一"}] + due = time.fromisoformat(self.settings.weekly_send_time) + due_at = datetime.combine(now.date(), due, tzinfo=tz) + if now < due_at: + return [{"status": "not_run", "detail": "尚未到周报发送时间"}] + week_start, week_end = previous_natural_week(now.date()) + all_states = [ + item + for item in self.store.list_states() + if item.get("week_start") == week_start.isoformat() + and item.get("week_end") == week_end.isoformat() + ] + results: list[dict] = [] + for state in all_states: + if state.get("status") != "sending": + continue + expires_text = str(state.get("send_claim_expires_at") or "") + try: + expired = datetime.fromisoformat(expires_text) <= now + except (TypeError, ValueError): + expired = True + if expired: + group_id = int(state.get("group_id") or 0) + self.store.update( + week_start.isoformat(), + week_end.isoformat(), + group_id, + status="needs_attention", + error_type="WEEKLY_SEND_RESULT_UNKNOWN", + send_claim_id="", + send_claim_expires_at="", + ) + results.append( + { + "group_name": str(state.get("group_name") or group_id), + "status": "held", + "error_type": "WEEKLY_SEND_RESULT_UNKNOWN", + } + ) + states = [item for item in all_states if item.get("status") == "ready_to_send"] + groups = {int(group.id): group for group in self._groups() if group.id is not None} + for state in states: + group_id = int(state.get("group_id") or 0) + group = groups.get(group_id) + if group is None or not bool(group.wechat_send_enabled): + continue + target = effective_send_target(group) + if target != str(state.get("send_target_snapshot") or ""): + self.store.update( + week_start.isoformat(), week_end.isoformat(), group_id, + status="needs_attention", error_type="WEEKLY_SEND_TARGET_CHANGED", + ) + results.append({"group_name": group.display_name, "status": "held", "error_type": "WEEKLY_SEND_TARGET_CHANGED"}) + continue + claim_id, state = self.store.claim_send( + week_start.isoformat(), + week_end.isoformat(), + group_id, + now=now, + ) + if not claim_id: + continue + text_path = self.store.text_path(week_start.isoformat(), week_end.isoformat(), group_id) + card_path = self.store.card_path(week_start.isoformat(), week_end.isoformat(), group_id) + try: + text_result, image_result = self.sender.send_bundle( + target, + text_path.read_text(encoding="utf-8"), + card_path, + ) + except Exception as exc: + self.store.update( + week_start.isoformat(), week_end.isoformat(), group_id, + status="needs_attention", error_type="WEEKLY_SEND_RESULT_UNKNOWN", + send_error=str(exc)[:300], send_claim_id="", send_claim_expires_at="", + ) + results.append({"group_name": group.display_name, "status": "held", "error_type": "WEEKLY_SEND_RESULT_UNKNOWN"}) + break + image_ok = image_result is not None and image_result.success + if text_result.success and image_ok: + self.store.update( + week_start.isoformat(), week_end.isoformat(), group_id, + status="sent", sent_at=now.isoformat(), send_target=target, + send_claim_id="", send_claim_expires_at="", + send_result={ + "text": text_result.detail, + "image": image_result.detail, + "verification_level": image_result.verification_level or text_result.verification_level, + }, + ) + results.append({"group_name": group.display_name, "status": "sent"}) + continue + unknown = bool( + text_result.outcome_unknown + or text_result.submitted + or (image_result and (image_result.outcome_unknown or image_result.submitted)) + ) + error_type = "WEEKLY_SEND_RESULT_UNKNOWN" if unknown else "WEEKLY_SEND_FAILED" + self.store.update( + week_start.isoformat(), week_end.isoformat(), group_id, + status="needs_attention", error_type=error_type, + send_claim_id="", send_claim_expires_at="", + send_error=f"text={text_result.detail}; image={getattr(image_result, 'detail', '')}"[:300], + ) + results.append({"group_name": group.display_name, "status": "held", "error_type": error_type}) + if unknown: + break + return results or [{"status": "not_run", "detail": "没有待发送周报"}] diff --git a/app/weekly/store.py b/app/weekly/store.py new file mode 100644 index 0000000..e91a89d --- /dev/null +++ b/app/weekly/store.py @@ -0,0 +1,124 @@ +"""周报独立状态与工件存储。""" + +from __future__ import annotations + +import json +import threading +import uuid +from datetime import datetime, timedelta +from pathlib import Path + +from app.core.path_security import resolve_within, validate_iso_date + +_LOCK = threading.RLock() + + +class WeeklyStore: + def __init__(self, output_root: Path | str): + self.root = Path(output_root) / ".weekly" + + def period_dir(self, week_start: str, week_end: str) -> Path: + start = validate_iso_date(week_start, field_name="week_start") + end = validate_iso_date(week_end, field_name="week_end") + return resolve_within(self.root, f"{start}_{end}") + + def group_dir(self, week_start: str, week_end: str, group_id: int) -> Path: + if int(group_id) <= 0: + raise ValueError("group_id 必须为正整数") + return resolve_within(self.period_dir(week_start, week_end), f"group-{int(group_id)}") + + def state_path(self, week_start: str, week_end: str, group_id: int) -> Path: + return self.group_dir(week_start, week_end, group_id) / "weekly.json" + + def text_path(self, week_start: str, week_end: str, group_id: int) -> Path: + return self.group_dir(week_start, week_end, group_id) / "weekly.txt" + + def card_path(self, week_start: str, week_end: str, group_id: int) -> Path: + return self.group_dir(week_start, week_end, group_id) / "weekly_card.png" + + def load(self, week_start: str, week_end: str, group_id: int) -> dict: + path = self.state_path(week_start, week_end, group_id) + if not path.is_file(): + return {} + try: + value = json.loads(path.read_text(encoding="utf-8")) + except (OSError, UnicodeError, json.JSONDecodeError): + return { + "schema_version": 1, + "week_start": week_start, + "week_end": week_end, + "group_id": group_id, + "status": "needs_attention", + "error_type": "WEEKLY_STATE_CORRUPT", + } + return value if isinstance(value, dict) else {} + + def save(self, week_start: str, week_end: str, group_id: int, value: dict) -> dict: + with _LOCK: + path = self.state_path(week_start, week_end, group_id) + path.parent.mkdir(parents=True, exist_ok=True) + payload = dict(value) + payload.update( + schema_version=1, + week_start=week_start, + week_end=week_end, + group_id=int(group_id), + ) + temp = path.with_suffix(".json.tmp") + temp.write_text( + json.dumps(payload, ensure_ascii=False, indent=2), + encoding="utf-8", + ) + temp.replace(path) + return payload + + def update(self, week_start: str, week_end: str, group_id: int, **fields) -> dict: + with _LOCK: + current = self.load(week_start, week_end, group_id) + current.update(fields) + return self.save(week_start, week_end, group_id, current) + + def claim_send( + self, + week_start: str, + week_end: str, + group_id: int, + *, + now: datetime, + lease_seconds: int = 600, + ) -> tuple[str | None, dict]: + """只允许 READY 周报原子进入 sending；崩溃后绝不自动重提。""" + with _LOCK: + current = self.load(week_start, week_end, group_id) + if current.get("status") != "ready_to_send": + return None, current + claim_id = uuid.uuid4().hex + current.update( + status="sending", + send_claim_id=claim_id, + send_attempt_started_at=now.isoformat(), + send_claim_expires_at=( + now + timedelta(seconds=max(int(lease_seconds), 60)) + ).isoformat(), + ) + return claim_id, self.save(week_start, week_end, group_id, current) + + def list_states(self) -> list[dict]: + if not self.root.is_dir(): + return [] + states: list[dict] = [] + for path in self.root.glob("*_*/group-*/weekly.json"): + try: + value = json.loads(path.read_text(encoding="utf-8")) + except (OSError, UnicodeError, json.JSONDecodeError): + continue + if isinstance(value, dict): + states.append(value) + states.sort( + key=lambda item: ( + str(item.get("week_start") or ""), + int(item.get("group_id") or 0), + ), + reverse=True, + ) + return states diff --git a/config/app.example.yaml b/config/app.example.yaml index af6a6eb..b7bfe28 100644 --- a/config/app.example.yaml +++ b/config/app.example.yaml @@ -11,7 +11,7 @@ app: timezone: "Asia/Shanghai" # 默认统计周期规则 - default_schedule_rule: "weekday_default" + default_schedule_rule: "daily_previous_day" data_source: # WeChatDataAnalysis 本地 MCP 服务（本机回环地址） diff --git a/config/groups.example.yaml b/config/groups.example.yaml index 580c9db..e640c9e 100644 --- a/config/groups.example.yaml +++ b/config/groups.example.yaml @@ -9,8 +9,8 @@ groups: enabled: true - # 统计周期规则：weekday_default（默认：周一=周五~周日，周六日不生成） - schedule_rule: "weekday_default" + # 统计周期规则：daily_previous_day（默认：每天生成前一自然日群报） + schedule_rule: "daily_previous_day" # 本群发送时间（每群独立） send_time: "08:30" diff --git a/docker-compose.yml b/docker-compose.yml index b782c54..baf2880 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -13,7 +13,8 @@ services: image: groupbrief-v2:latest container_name: groupbrief-v2 ports: - - "8766:8766" + # 管理与发送 API 没有远程认证；默认只发布到宿主机回环地址。 + - "127.0.0.1:8766:8766" volumes: # 数据持久化（SQLite / 日报输出 / 日志）挂载到宿主机当前目录 - ./data:/app/data diff --git a/docs/DOCKER.md b/docs/DOCKER.md index a9b101a..19a3a6c 100644 --- a/docs/DOCKER.md +++ b/docs/DOCKER.md @@ -1,6 +1,6 @@ -# 使用 Docker 运行 GroupBrief +# 使用 Docker 开发或只读运行 GroupBrief -Docker 方式适合希望让 GroupBrief 后端、Web UI 和调度器持续运行的用户。微信桌面客户端、WeChatDataAnalysis 和微信 UI 自动化仍运行在 Windows 宿主机上。 +Docker 只作为开发、读取和界面验证环境。正式支持环境是 Windows + Alter + 本机微信，由 FastAPI Scheduler 独占调度；Docker 不承诺原生微信发送。微信桌面客户端、WeChatDataAnalysis 和微信 UI 自动化仍运行在 Windows 宿主机上。 ## 运行边界 diff --git a/docs/audits/P0_2A_DATA_INTEGRITY.md b/docs/audits/P0_2A_DATA_INTEGRITY.md new file mode 100644 index 0000000..03c5f95 --- /dev/null +++ b/docs/audits/P0_2A_DATA_INTEGRITY.md @@ -0,0 +1,256 @@ +# GroupBrief P0.2A 数据一致性取证 + +> 取证时间：2026-08-24 19:22（Asia/Shanghai） +> +> 数据基线：`data/groupbrief.db` 的 SQLite Online Backup 一致性快照 +> +> 原则：只备份、只读聚合、只设计迁移；未修改正式数据库、Schema 或业务数据 + +## 1. 结论 + +正式数据库的物理结构当前完整，`PRAGMA integrity_check=ok`；问题属于**逻辑关系损失**，不是数据库文件损坏。 + +224 条 `group_runs` 中有 192 条（85.7%）引用已不存在的群 ID `1-22`。当前群 ID 只有 `23-29`。Git 历史证实旧版本曾对群执行物理删除，直到 2026-08-22 才改为软删除。因此，最可信解释是：旧群被物理删除并重新创建后，历史 `group_runs`/`reports` 被保留，但数据库没有外键阻止关系断裂。 + +目前不能安全自动修复旧群关系：`group_runs` 只保存本地整数 `group_id`，没有历史 `wechat_group_id` 快照；不能假设旧 ID `1` 对应新 ID `23`，也不能按顺序映射。 + +本轮决定： + +- 保留全部 192 条历史记录。 +- 不删除、不伪造群、不自动重关联。 +- P0.2B 先建立“历史孤儿”正式表达，再加外键。 +- 正式迁移前必须停止所有数据库写入者，并只在备份副本完成演练。 + +## 2. 一致性备份 + +使用 `sqlite3.Connection.backup()` 从只读源连接创建一致性快照；没有复制、移动或替换活动数据库。 + +备份文件： + +```text +data/backups/groupbrief-p0-2a-consistent-20260824-192257.db +``` + +校验 Manifest： + +```text +data/backups/groupbrief-p0-2a-consistent-20260824-192257.manifest.json +``` + +备份证据： + +```text +字节数 1,052,672 +SHA256 a17fca934b40d4c076e605df899c53a97082f415ea9a7e3699f725cba10fd6a4 +integrity ok +groups 7 +runs 68 +group_runs 224 +reports 214 +settings 55 +provider 48 +execution_log 0 +``` + +源库与备份的四个核心表行数完全一致。备份目标的 `schema_version` cookie 从源库的 `25` 变为 `1`，这是 SQLite 内部 schema cookie，不是业务迁移版本，不能用于迁移判断。真正可用于版本判断的 `PRAGMA user_version` 在源库和备份中都为 `0`。 + +`PRAGMA foreign_key_check` 返回 0 行，但当前表根本没有声明外键，因此这不表示逻辑关系健康。 + +## 3. 已证实的数据问题 + +### 3.1 192 条孤儿 GroupRun + +当前活动/软删除群 ID： + +```text +23, 24, 25, 26, 27, 28, 29 +``` + +孤儿记录引用的旧 ID 及数量： + +```text +1:19 2:18 3:15 4:14 5:13 6:11 +7:10 8:10 9:8 10:8 11:8 12:6 +13:7 14:6 15:6 16:6 17:5 18:5 +19:4 20:5 21:3 22:5 +``` + +按报告日期： + +```text +2026-08-13 166 +2026-08-17 24 +2026-08-18 2 +``` + +最早日期为 2026-08-13，最晚日期为 2026-08-18。2026-08-13 的 166 条 `group_runs` 全部已经失去群关系，说明这是一批历史关系断裂，不是偶发单行错误。 + +### 3.2 孤儿 GroupRun 的内容完整程度 + +父 Run 状态： + +```text +success 181 +running 10 +failed 1 +``` + +阶段状态： + +```text +ranking=success, prompt=success 164 +ranking=success, prompt=skipped 22 +ranking=success, prompt=pending 5 +ranking=failed, prompt=skipped 1 +``` + +报告关系： + +```text +有且仅有 1 条 Report 186 +没有 Report 6 +多条 Report 0 +``` + +这意味着绝大多数孤儿仍包含可用历史结果。直接删除会丢失 186 条已有报告的归属上下文，不可接受。 + +### 3.3 五条“空成功”Run + +以下父 Run 状态为 `success`，但没有任何 `GroupRun`： + +```text +Run 32 success auto 2026-08-17 +Run 37 success auto 2026-08-17 +Run 60 success manual 2026-08-18 +Run 61 success manual 2026-08-18 +Run 65 success auto 2026-08-18 +``` + +它们不能证明核心业务成功，属于状态语义问题。当前没有足够日志确认是空群执行、旧流程写入还是中途清理；P0.2B 不应猜测性改成失败，先标记为待分类。 + +### 3.4 当前没有重复关系 + +以下聚合均为 0： + +- 重复 `(run_id, group_id)` 关系 +- 同一 `group_run_id` 对应多条 Report +- 非空 `wechat_group_id` 重复 +- `group_runs` 缺失父 Run +- Report 缺失父 GroupRun + +没有重复数据是好消息，但它完全依赖应用代码；数据库没有唯一约束阻止未来重复。 + +## 4. 根因置信度 + +### 已证实 + +- 初始版本到提交 `05b7da0` 之前，`repository.delete_group()` 使用 `session.delete(group)` 物理删除。 +- 2026-08-22 才切换为当前软删除实现。 +- 数据库没有 `group_runs.group_id → groups.id` 外键。 +- 当前群 ID 从 23 开始，而全部孤儿只引用 1-22。 +- 孤儿数据集中在旧群重新创建前后的 2026-08-13 至 2026-08-18。 + +### 高概率推断 + +旧群 ID `1-22` 曾存在，后来通过旧版本物理删除；群重新绑定/创建后获得 ID `23-29`。由于没有外键和稳定身份快照，旧 `group_runs`/`reports` 被保留成逻辑孤儿。 + +### 尚未证实 + +- 每个旧群 ID 对应哪个新群 ID。 +- 五条空成功 Run 的具体形成动作。 +- 10 条父 Run 为 `running` 的孤儿是否真的中断，还是旧流程没有收口。 +- 是否能从历史 output 工件为每条孤儿恢复稳定微信群 ID；本轮没有读取业务内容。 + +## 5. Schema 与迁移问题 + +当前状态： + +```text +PRAGMA foreign_keys = 0 +PRAGMA user_version = 0 +正式 migration version table = 不存在 +``` + +缺少的关系保护： + +- `group_runs.run_id → runs.id` +- `group_runs.group_id → groups.id` +- `reports.group_run_id → group_runs.id` +- `execution_logs.run_id → runs.id` + +缺少的唯一/查询保护： + +- 每个 GroupRun 最多一条 Report +- 非空 `wechat_group_id` 的唯一约束 +- `runs(report_date, status)` 查询索引 +- `execution_logs(run_id)` 查询索引 + +`UNIQUE(run_id, group_id)` 暂时不能直接添加。当前失败路径可能在同一个 Run/Group 上追加另一条失败记录；需要先统一 attempt/revision 语义，或者引入 `attempt_no`。 + +数据库列默认值也已经漂移： + +```text +groups.summary_model 数据库默认 deepseek-v4-flash / 代码默认 gpt-5.6-sol +groups.image_theme 数据库默认 blue_white / 代码默认 random_preset +``` + +现有 `ALTER TABLE ADD COLUMN` 与 settings marker 不能表达列重建、约束升级和回滚，因此 P0.2B 需要正式 Schema 版本机制。 + +## 6. P0.2B 推荐设计 + +### 6.1 先建立历史身份表达 + +不要把外键直接套到当前非空 `group_id`。建议先将历史身份拆开： + +```text +group_id INTEGER NULL，引用当前 groups.id +legacy_group_id INTEGER NULL，保存旧本地 ID +identity_state TEXT NOT NULL，例如 active / legacy_orphan / unresolved +orphan_reason TEXT NOT NULL，例如 legacy_group_hard_deleted +``` + +迁移规则： + +- 当前能关联到 `groups` 的 32 条 GroupRun：保留 `group_id`。 +- 192 条孤儿：将旧整数移入 `legacy_group_id`，`group_id=NULL`。 +- 不创建假群，不做顺序映射。 +- Reports 继续关联原 GroupRun，保留 186 条历史报告。 + +### 6.2 外键删除策略 + +建议历史数据优先： + +- Group → GroupRun 使用 `ON DELETE RESTRICT`，禁止再次物理删除仍有历史的群。 +- Run → GroupRun 使用 `ON DELETE RESTRICT`。 +- GroupRun → Report 使用 `ON DELETE RESTRICT`。 +- 业务删除继续使用现有 `deleted_at` 软删除。 + +不要使用级联删除历史报告。 + +### 6.3 正式迁移顺序 + +1. 停止 FastAPI scheduler、Windows 计划任务及所有数据库写入者。 +2. 再创建一份停机前备份并验证 hash/integrity。 +3. 在独立副本创建 Schema V1 迁移表和新表。 +4. 按确定规则复制数据，不原地批量 UPDATE。 +5. 建立外键、唯一约束和索引。 +6. 开启 `PRAGMA foreign_keys=ON`，执行 `foreign_key_check` 和 `integrity_check`。 +7. 用归档 API、历史 Report 数、Run 状态和完整测试验收。 +8. 通过后才原子替换正式数据库。 + +### 6.4 回滚 + +- 保留原库和迁移后库，不在原库执行 down migration。 +- 失败时停止所有写入者，用已验证备份原子恢复。 +- 恢复迁移前代码提交。 +- 再次核对完整性、核心表行数和 8766 健康。 + +## 7. 本轮明确未做 + +- 未修改正式数据库、Schema、索引或 PRAGMA。 +- 未删除或重关联任何孤儿记录。 +- 未读取聊天正文、Prompt、群名、发送目标或 Secret。 +- 未停止/重启 8766 服务、调度器或计划任务。 +- 未运行真实 AI、微信或邮件动作。 + +P0.2A 至此完成。下一轮只有在确认“历史孤儿建模方案”后，才进入 P0.2B 迁移实现与副本演练。 diff --git a/docs/audits/P0_2B_MIGRATION_REHEARSAL.md b/docs/audits/P0_2B_MIGRATION_REHEARSAL.md new file mode 100644 index 0000000..c07805c --- /dev/null +++ b/docs/audits/P0_2B_MIGRATION_REHEARSAL.md @@ -0,0 +1,211 @@ +# GroupBrief P0.2B-1 离线迁移与副本演练 + +> 执行时间：2026-08-24（Asia/Shanghai） +> +> 范围：离线迁移工具、自动化测试、P0.2A 备份副本演练 +> +> 未执行：正式数据库替换、ORM/API 切换、服务重启、真实外部调用 + +## 1. 结论 + +P0.2B-1 已达到“可以进入切换设计”的状态，但尚未部署到正式数据库。 + +显式离线迁移工具已经在 P0.2A 一致性备份上成功完成一次真实演练：192 条历史孤儿 `group_runs` 全部保留旧 ID，并从错误的活动关联转换为明确的 `orphaned` 状态；32 条仍能关联当前群的记录保持 `linked`；224 条 GroupRun、214 条 Report、68 条 Run 和 7 条 Group 总数均未变化。 + +迁移后副本的物理完整性、关系完整性和关键语义检查全部通过： + +```text +integrity_check ok +foreign_key_check_rows 0 +linked_group_runs 32 +orphaned_group_runs 192 +preserved_legacy_group_ids 192 +invalid_linked_group_ids 0 +user_version 1 +``` + +正式数据库 `data/groupbrief.db` 没有被迁移或替换，当前应用仍运行旧 Schema。 + +## 2. 交付内容 + +```text +app/db/offline_migrations.py 离线迁移、前置检查、事务重建和结果验证 +scripts/migrate_db.py 薄 CLI 入口 +tests/test_db_migration.py 迁移成功与失败边界测试 +``` + +CLI 必须显式选择动作： + +```powershell +.\.venv\Scripts\python.exe scripts\migrate_db.py ` + --source <只读源数据库> ` + --output <必须不存在的新数据库> ` + --dry-run + +.\.venv\Scripts\python.exe scripts\migrate_db.py ` + --source <只读源数据库> ` + --output <必须不存在的新数据库> ` + --apply +``` + +该模块没有接入 `app.main` 或 `repository.init_db()`，应用启动不会自动迁移。 + +## 3. 真实演练证据 + +演练源： + +```text +data/backups/groupbrief-p0-2a-consistent-20260824-192257.db +SHA256 a17fca934b40d4c076e605df899c53a97082f415ea9a7e3699f725cba10fd6a4 +``` + +最终演练输出： + +```text +data/backups/groupbrief-p0-2b-rehearsal-final-20260824.db +SHA256 5990c894be7e05a073fb9b69447df41bb71b9d0be36fb31e3f97ae2e1ffa3e37 +``` + +Manifest： + +```text +data/backups/groupbrief-p0-2b-rehearsal-final-20260824.db.manifest.json +``` + +源文件迁移前后 SHA256 完全相同。演练输出和 Manifest 位于已被 Git 忽略的 `data/backups/`，不会进入仓库。 + +第一次演练产生的以下两个文件已被最终演练取代： + +```text +data/backups/groupbrief-p0-2b-rehearsal-20260824.db +data/backups/groupbrief-p0-2b-rehearsal-20260824.db.manifest.json +``` + +本机安全策略阻止了自动删除，因此它们仍被保留为可恢复的旧演练产物；正式备份未受影响。 + +## 4. 迁移后的关系设计 + +### GroupRun 身份 + +仍可关联当前群： + +```text +group_id 当前 groups.id +legacy_group_id NULL +identity_state linked +orphan_reason 空字符串 +``` + +历史孤儿： + +```text +group_id NULL +legacy_group_id 原旧本地 ID +identity_state orphaned +orphan_reason historical_group_missing +``` + +数据库 `CHECK` 约束禁止出现“显示 linked 但没有当前群”或“显示 orphaned 但丢失旧 ID”的矛盾状态。 + +### 外键与约束 + +- `group_runs.run_id → runs.id ON DELETE RESTRICT` +- `group_runs.group_id → groups.id ON DELETE RESTRICT` +- `reports.group_run_id → group_runs.id ON DELETE RESTRICT` +- `execution_logs.run_id → runs.id ON DELETE RESTRICT` +- 同一 GroupRun 最多一个 Report +- 非空、未删除群的 `wechat_group_id` 唯一 + +没有添加 `UNIQUE(run_id, group_id)`：当前强制重试/重新生成语义尚未正式建模，贸然添加可能阻断合法重试。 + +## 5. 安全边界和失败模式 + +- **输入输出相同：** 迁移拒绝，不打开写连接。 +- **输出或 Manifest 已存在：** 拒绝覆盖，原文件保持不变。 +- **WAL/SHM/Journal 侧文件存在：** 认为源库可能仍有写入者，迁移拒绝。 +- **未知 `user_version`：** 拒绝覆盖其他迁移体系。 +- **未知列、显式索引、触发器或依赖视图：** 拒绝静默丢失未来 Schema。 +- **缺失父 Run、孤儿 Report、重复 Report、重复活动微信群 ID：** 前置检查失败，不创建输出。 +- **表重建中断：** 事务回滚并删除本轮精确命名的临时文件。 +- **迁移已执行：** 依据 `schema_migrations` 明确拒绝重复应用。 +- **迁移后计数、外键或语义不一致：** 不提升临时数据库为正式输出。 + +迁移工具只使用 Python 标准库 `sqlite3`，没有新增第三方依赖。 + +## 6. Impact / Effort + +```text + LOW EFFORT HIGH EFFORT + ┌────────────────────────┬──────────────────────────┐ +HIGH │ 已完成：离线迁移工具 │ 下一轮：ORM/API 正式切换 │ +IMPACT │ 已完成：副本演练和约束验证 │ 下一轮：停机迁移与恢复演练 │ + ├────────────────────────┼──────────────────────────┤ +LOW │ 可后续：CLI 文案美化 │ 暂缓：引入完整 Alembic 体系 │ +IMPACT │ │ 暂缓：自动映射旧群到新群 │ + └────────────────────────┴──────────────────────────┘ +``` + +## 7. 为什么本轮不修改 ORM/API + +当前旧正式库没有 `legacy_group_id`、`identity_state` 和 `orphan_reason`。`SQLModel.metadata.create_all()` 不会给既有表增加这些列。 + +如果先更新 `models.py` 并重启应用，ORM 查询会直接报 `no such column`。因此必须把下一轮设计成一个有明确停机窗口的原子切换： + +```text +停止所有写入者 + → 最终备份 + → 离线迁移新库 + → 更新 ORM/API 与连接级 foreign_keys=ON + → 替换数据库 + → 启动并验证 + → 失败则停止服务并恢复原库/原代码 +``` + +## 8. 下一轮 P0.2B-2 的切换条件 + +1. 明确 FastAPI scheduler、Windows Task 和其他写入者全部停止。 +2. 迁移前确认没有 `-wal`、`-shm`、`-journal`。 +3. 创建新的停机备份并记录 SHA256。 +4. 同一个发布轮次更新 ORM、Run API、Dashboard 和邮件孤儿过滤。 +5. SQLAlchemy 每个 SQLite 连接显式开启 `PRAGMA foreign_keys=ON`。 +6. 定向测试、全量测试、旧库迁移测试和回滚测试全部通过。 +7. 启动后只验证本地读取和健康状态，不触发 AI、微信或邮件。 + +在这些条件满足前，不应把演练数据库替换成正式数据库。 + +## 9. 本轮不在范围内 + +- 不修复 5 条“success 但无 GroupRun”的历史 Run。 +- 不修复历史 `running` 状态。 +- 不自动猜测旧群与当前群的对应关系。 +- 不改 V1/V2 业务流程。 +- 不引入 Alembic、微服务、事件总线或其他高复杂度方案。 + +## 10. 最终验证 + +当前最终工作树的独立验证结果： + +```text +迁移定向测试 17 passed +完整 Python 测试 455 passed, 1 warning +Python 编译检查 passed +git diff --check passed +8766 根页面 HTTP 200 +``` + +唯一 warning 是现有 Starlette `TestClient` 对 `httpx` 调用方式的弃用提示，与本次迁移逻辑无关；应在依赖维护轮次处理，不值得阻塞本轮。 + +```text +╔══════════════════════════════════════════════╗ +║ P0.2B-1 CODE OVERHAUL SUMMARY ║ +╠══════════════════════════════════════════════╣ +║ Mode: SURGICAL ║ +║ Stack: Python / SQLite ║ +║ Production DB: unchanged ║ +║ Orphans preserved: 192 / 192 ║ +║ Reports preserved: 214 / 214 ║ +║ Test failures: 0 ║ +║ Critical gaps: runtime cutover pending ║ +║ Beads filed: 0 ║ +╚══════════════════════════════════════════════╝ +``` diff --git a/docs/audits/P0_2B_PRODUCTION_CUTOVER.md b/docs/audits/P0_2B_PRODUCTION_CUTOVER.md new file mode 100644 index 0000000..373426c --- /dev/null +++ b/docs/audits/P0_2B_PRODUCTION_CUTOVER.md @@ -0,0 +1,158 @@ +# GroupBrief P0.2B-2 正式数据库关系切换 + +> 执行时间：2026-08-24（Asia/Shanghai） +> +> 模式：SURGICAL +> +> 范围：ORM/API、连接级外键、测试隔离、正式 SQLite 离线迁移与切换 + +## 1. 结论 + +P0.2B-2 已完成正式切换。正式服务当前使用 P0.2B Schema，8766 已由原 Alter 登记进程恢复运行，Windows 计划任务已恢复原启用状态。 + +迁移没有猜测历史群映射，也没有丢弃历史记录： + +```text +groups 7 → 7 +runs 68 → 68 +group_runs 224 → 224 +reports 214 → 214 +linked - → 32 +orphaned 192 → 192 +``` + +192 条已经失去现存群关系的历史 GroupRun 现在使用 `identity_state=orphaned`、`group_id=NULL` 和 `legacy_group_id=<旧 ID>` 明确表达。32 条仍能关联当前群的记录保持 `linked`。 + +## 2. 正式切换证据 + +旧正式库在所有写入者停止后保持稳定： + +```text +旧库路径 data/groupbrief.db +旧库 SHA256 a677eaa7e2654a1aee07f6a5e8712a1e73c228b3db0d1746ffd887fb3324bbd4 +integrity ok +user_version 0 +``` + +正式迁移输出： + +```text +迁移 Manifest data/backups/groupbrief-p0-2b2-candidate-20260824-224331.json +新库 SHA256 6de4c332cf51b4313aeb52c5cdd4dd624bdeda656e78455133860445bef35f6a +integrity ok +FK violations 0 +user_version 1 +migration_id p0_2b_group_run_identity_v1 +checksum c14ecdc63408fa30cbfe02803098ddf9ffb2ce25fa3440d758f5e3a60accbb24 +``` + +旧库已移动到精确回滚路径： + +```text +data/backups/groupbrief-p0-2b2-original-20260824-224331.db +SHA256 a677eaa7e2654a1aee07f6a5e8712a1e73c228b3db0d1746ffd887fb3324bbd4 +``` + +上述数据库与 Manifest 位于被 Git 忽略的 `data/backups/`，不会进入仓库。 + +## 3. 运行时改动 + +- ORM 增加 GroupRun 身份状态、历史群 ID、外键和一致性 `CHECK`。 +- Report 与 GroupRun 建立 `RESTRICT` 外键及一对一唯一约束。 +- ExecutionLog 与 Run 建立 `RESTRICT` 外键。 +- 活动群 `wechat_group_id` 使用条件唯一索引；Run 增加日期/状态复合索引。 +- SQLite 每个 SQLAlchemy 连接都执行 `PRAGMA foreign_keys=ON`。 +- 启动时校验迁移记录、checksum、`user_version`、外键和关键索引；旧非空 Schema 会 fail closed，不会自动原地迁移。 +- Run/Report API 显式返回历史孤儿身份；统计和邮件只使用 `linked` 记录。 + +## 4. 停写与原子替换 + +切换窗口内执行了以下保护： + +1. 确认原 Alter 进程已停止、8766/8767 无监听。 +2. 禁用 `GroupBriefDaily` 与 `GroupBriefDailySend`。 +3. 确认没有 Uvicorn、日报生成或发送 Python 写入进程。 +4. 确认没有 `groupbrief.db-wal`、`-shm`、`-journal`。 +5. 对正式源库执行 dry-run 和 apply，迁移前后源 SHA256 相同。 +6. 候选库先在 8767 启动并验证六个只读接口。 +7. 同盘移动旧库到回滚路径，再移动候选库到正式路径；第二步失败时会立即移回旧库。 +8. 启动 8766、验证数据库与接口后，恢复两个计划任务。 + +## 5. 启动后验收 + +正式库： + +```text +integrity_check ok +foreign_key_check rows 0 +connection foreign_keys 1 +user_version 1 +groups/runs/gr/reports 7 / 68 / 224 / 214 +linked/orphaned 32 / 192 +``` + +只读 HTTP 验收： + +```text +/ 200 +/api/system/health 200 +/api/system/status 200 +/api/system/stats 200 +/api/runs 200 +/api/reports/latest 200 +``` + +运行状态： + +```text +Alter process ID 94507bc8-4b0e-4f37-89a2-1bffedb15fd3 +Alter status running +8766 listener python/uvicorn child of the registered Alter process +GroupBriefDaily enabled, Ready, next 2026-08-25 00:15 +GroupBriefDailySend enabled, Ready, next 2026-08-25 08:30 +``` + +没有执行真实 AI、微信或邮件发送。 + +## 6. 测试证据 + +主控与独立 Operator 最终结果一致： + +```text +完整测试 458 passed, 1 warning +定向测试 80 passed, 1 warning +git diff check passed +8767 shadow six read-only endpoints HTTP 200 +8766 formal six read-only endpoints HTTP 200 +``` + +唯一 warning 是现有 Starlette `TestClient` 对 `httpx` 调用方式的弃用提示，与数据库切换无关。 + +测试数据库改为每个 pytest 进程独占的临时 SQLite 文件，避免并行测试、失败重跑和固定测试库残留互相污染。旧测试中依赖“无外键时可直接删父记录”的清场逻辑也已按依赖顺序修正。 + +## 7. 运行异常与处置 + +首次恢复服务时误用了 `alter start GroupBrief-Backend`。当前 Alter 版本把该参数解释为新脚本，因此短暂创建了一个同名空壳登记项，但它没有监听 8766，也没有访问正式数据库。 + +该空壳已按精确 ID 删除，并确认不存在。随后使用原登记 ID 执行 `alter restart`，原进程成功恢复；其他 Alter 项目没有被修改。 + +## 8. 回滚边界 + +P0.2B-2 的新代码会主动拒绝旧 Schema，因此回滚必须成对执行，不能只换回数据库： + +1. 停止原 Alter 进程并禁用两个 GroupBrief 计划任务。 +2. 确认无 8766/8767 监听、无 Python 写入者、无 SQLite sidecar。 +3. 将代码恢复到本轮父提交 `df8966f58305a54b641e3301f648fdf445eba9eb`。 +4. 将当前正式库移到新的故障留存路径。 +5. 将 `groupbrief-p0-2b2-original-20260824-224331.db` 恢复为 `data/groupbrief.db`。 +6. 启动并验证旧版本，再恢复计划任务。 + +不要在新代码下直接恢复旧库；Schema guard 会按设计拒绝启动。 + +## 9. 本轮未处理 + +- 不猜测 192 条历史孤儿对应哪个现存群。 +- 不重写历史 Run 的 success/running 状态。 +- 不添加 `UNIQUE(run_id, group_id)`，避免误伤现有强制重跑语义。 +- 不重构 V1/V2 Pipeline、Provider 或调度架构。 +- 不触发真实生成、发送或第三方服务。 diff --git a/docs/audits/P1_1_CORRUPT_STATE_RECOVERY.md b/docs/audits/P1_1_CORRUPT_STATE_RECOVERY.md new file mode 100644 index 0000000..6c9d74a --- /dev/null +++ b/docs/audits/P1_1_CORRUPT_STATE_RECOVERY.md @@ -0,0 +1,85 @@ +# P1.1 损坏状态与“结果未知”恢复 + +日期：2026-08-25 + +## 结论 + +P1.1 将“文件不存在”和“文件存在但不可信”拆成了两个完全不同的状态： + +```text +文件不存在 ───────────────────→ PENDING / 新任务 +文件合法且未完成 ─────────────→ 沿用现有安全续跑 +外部调用已开始但结果无法确认 ─→ RESULT_UNKNOWN + hold +文件存在但损坏或结构不可信 ───→ CORRUPT + manual_review + │ + └─ 禁止覆盖、生成、邮件和微信发送 +``` + +此前，非法 JSON、空文件、截断文件和部分结构错误会被宽松回退为 PENDING 或空调度状态。这可能把已经发生过外部副作用的任务误判为“从未开始”，造成重复生成、重复扣费或重复发送。本轮采取 fail-closed：不猜测、不自动重建、不覆盖原证据，由人工根据日志、产物和外部平台结果决定后续处理。 + +## 修改范围 + +- `app/v2/run_store.py`：统一解析和结构校验，合成不含业务正文的 CORRUPT 摘要；所有写入口拒绝覆盖损坏文件；列表不再静默漏掉损坏运行。 +- `app/v2/recovery.py`：扫描和完整性检查显式返回损坏项；自动恢复在创建 Pipeline 前阻断。 +- `app/scheduler/daily_v2_job.py`：调度状态损坏时返回 blocked/manual_review，不初始化数据库、不生成、不调用邮件。 +- `app/scheduler/manager.py`：启动补偿跳过损坏调度状态。 +- `app/api/v2_ui.py`：手动 retry 在查库或创建 Pipeline 前阻断损坏运行。 +- `app/pipeline/daily_pipeline.py`：直接指定群的强制生成在群名同步和生成前检查损坏状态，并返回结构化 blocked。 +- `app/v2/constants.py`：增加只读隔离状态和统一错误类型。 +- `tests/test_v2_recovery.py`、`tests/test_scheduler.py`、`tests/test_v2_pipeline.py`：覆盖损坏文件、结构错误、证据保留、直接群重试及结果未知状态。 + +## 风险与收益 + +| 项目 | 评估 | +| --- | --- | +| 实际影响 | 避免损坏状态触发重复外部调用，是高影响防线 | +| 发生概率 | 当前磁盘未发现损坏，但异常退出、磁盘故障或人工编辑都可能触发 | +| 修改收益 | 高；把不可证明安全的恢复从自动执行改成人工确认 | +| 修改成本 | 低至中；集中在状态边界和恢复入口 | +| 修改风险 | 低；正常 PENDING、合法中断续跑和结果未知语义保持不变 | +| Blast Radius | V2 运行状态、每日调度、恢复 API 和启动补偿 | + +## 失败模式与处理 + +| 失败模式 | 旧行为 | 新行为 | +| --- | --- | --- | +| `run.json` 不存在 | PENDING | PENDING，不变 | +| `run.json` 非法、空、截断、非对象 | PENDING 或被列表忽略 | CORRUPT，显式可见，禁止写入和恢复 | +| `run.json` 缺身份或状态字段 | 可能继续使用 | CORRUPT，人工复核 | +| scheduler 状态非法或结构错误 | 当作从未执行，可能重跑 | blocked，禁止生成和邮件 | +| 邮件 started 但无 completed | 结果未知保护 | EMAIL_RESULT_UNKNOWN + hold，不变 | +| 合法生成中断 | 从断点续跑 | 从断点续跑，不变 | + +## 数据检查 + +本轮只读扫描了当前正式状态文件，没有读取或输出业务正文，也没有重写任何状态： + +- V2 `run.json`：45 个；结构损坏 0 个。 +- 状态分布：SENT 19、PROMPT_READY 1、READY_TO_SEND 20、FAILED 5。 +- scheduler 状态：4 个；结构损坏 0 个。 + +因此本次改动是补齐故障边界，不是对当前数据进行修复。 + +## 验证 + +- 定向恢复/调度/Pipeline 测试：最终 76 项通过。 +- 将仅 P1.1 暂存差异应用到基于 `05b1f64` 的隔离 worktree 后，完整 Python 测试 478 项通过，1 个上游弃用 warning。 +- 独立验证首次完整测试出现一次并发 timing 抖动：阈值 `<0.45s`，实测 `0.466s`；未修改阈值，完整重跑通过。该证据留到 P2.1 处理测试稳定性。 +- Python 编译检查：通过。 +- 前端生产构建：通过（4,596 个模块）。 +- `git diff --check`：通过，仅有工作区 LF/CRLF 提示。 +- 8766：核心状态隔离改动写入后，由原 Alter 项目 `94507bc8-4b0e-4f37-89a2-1bffedb15fd3` 重启；监听进程仍由该项目的进程树持有，`/api/version` 返回 `1.0.0`，恢复接口可读且报告 CORRUPT 数量为 0。当天调度状态已完成，Windows 调度任务均为 Ready；重启日志无 catch-up 和 ERROR。随后补充的“直接指定群重试”保护已在隔离完整测试中验证；为避免把同期另一任务的未提交源码一起载入 8766，没有再次重启。未触发真实 AI、邮件或微信发送。 + +## 回滚 + +代码可通过单独回滚本次提交恢复。P1.1 没有数据库迁移，也没有修改现有 `run.json` 或 scheduler 文件。若未来出现 CORRUPT，必须先备份原文件，再由人工依据日志、磁盘产物和外部发送结果重建可信状态；不能直接删除文件让系统回到 PENDING。 + +## 并发工作区说明 + +验收期间工作区出现了另一任务正在进行的 AI 图片主题、Prompt、群 API、数据库默认值和前端修改。它们没有纳入 P1.1 暂存或提交；直接在混合工作区运行全量测试时有 7 项相关断言失败。为避免污染结论，本轮以“HEAD + 仅 P1.1 暂存差异”的隔离 worktree 完整测试作为提交验收依据。 + +## 非目标 + +- 不自动猜测或修复损坏状态。 +- 不改变调度进程所有权、退出码和假成功语义；这些属于 P1.2。 +- 不统一 Provider、Mock、AI/邮件幂等或 V1 双轨；这些按后续轮次处理。 diff --git a/docs/audits/P1_2_SCHEDULER_OWNERSHIP.md b/docs/audits/P1_2_SCHEDULER_OWNERSHIP.md new file mode 100644 index 0000000..b9dd176 --- /dev/null +++ b/docs/audits/P1_2_SCHEDULER_OWNERSHIP.md @@ -0,0 +1,101 @@ +# P1.2 单一调度所有权、退出码与假成功 + +日期：2026-08-25 + +## 结论 + +正式唯一调度 owner 统一为 8766 FastAPI 进程内的 APScheduler。Windows `GroupBriefDaily` 与 `GroupBriefDailySend` 不删除，只禁用保留为可回滚入口；配置和安装脚本会阻止它们在 `fastapi` owner 下重新启用。 + +```text +整改前 +FastAPI APScheduler ─┬─ 00:15 生成+邮件 + └─ 每分钟微信 send_due +Windows Task ───────┬─ 00:15 daily_auto.py + └─ 08:30~09:00 每分钟 send CLI + ↓ + 依赖 mutex/claim 碰撞避重 + +整改后 +Alter 管理的 8766 + └─ FastAPI APScheduler（唯一 owner） + ├─ 00:15 生成+邮件 + ├─ 启动补偿 + └─ 每分钟微信 send_due + +Windows 两任务：Disabled（只作回滚，不同时启用） +手动 CLI：保留，输出稳定 outcome 与退出码 +``` + +## 关键问题 + +- `daily_auto.py` 把 `already_running` 返回为 0，Windows 会误记成成功。 +- `run_daily_pipeline.py` 除参数缺失外所有业务分支固定返回 0；failed、held、结果未知都能假成功。 +- `run_send_due_job()` 捕获异常后正常返回，APScheduler 会把异常扫描记为正常执行。 +- `_generation_status()` 把未识别状态和 `no_groups` 落入 success。 +- 生成 partial 后，只要邮件子进程返回 0，整批状态会被覆盖成 success。 +- Windows 两阶段安装的第二步失败时，第一步残留，且安装命令仍返回 0。 + +## 终态与退出码 + +| outcome | exit code | 运维语义 | +| --- | ---: | --- | +| success | 0 | 本次完整成功或已有可信完成态 | +| failed | 1 | 本次失败 | +| partial | 2 | 部分成功，仍有失败项 | +| blocked | 3 | 结果未知或人工复核，禁止自动推进 | +| already_running | 4 | 本次没有取得执行所有权 | +| not_run | 5 | 没有可执行对象，本次没有实际工作 | + +未知业务状态不再默认 success，而是 fail closed 为 failed。APScheduler 没有进程退出码，因此使用同一 outcome；除分钟级正常空扫描 `not_run` 外，非 success 会抛出 `SchedulerOutcomeError`，让调度器记录失败。 + +## 修改范围 + +- `app/scheduler/outcome.py`：唯一业务终态、聚合和退出码契约。 +- `app/config/settings.py`、`app/main.py`：环境级 `scheduler_owner=fastapi|external|disabled`；只在 fastapi owner 下注册内部调度。 +- `app/api/system.py`：只读返回配置 owner 与实际 scheduler active 状态。 +- `app/scheduler/daily_v2_job.py`：no_groups/未知状态 fail closed；partial 不再被邮件成功覆盖；记录最近调用 outcome/exit code。 +- `app/scheduler/manager.py`、`app/scheduler/send_job.py`：APScheduler 对业务失败不再假成功或吞异常。 +- `scripts/daily_auto.py`、`scripts/run_daily_pipeline.py`：打印可机读 `OUTCOME` 并返回稳定退出码。 +- `scripts/install_daily_task.py`：owner 冲突时拒绝安装/启用，半安装自动回滚，状态冲突返回 blocked。 + +## 失败模式 + +| 场景 | 新行为 | +| --- | --- | +| 两实例争抢生成锁 | 未取得锁的一方 `already_running / exit 4` | +| 全部群生成失败 | `failed / exit 1` | +| 部分群失败、邮件成功 | 整批仍为 `partial / exit 2` | +| scheduler/run 状态损坏或发送结果未知 | `blocked / exit 3` | +| 无启用群 | `not_run / exit 5`，不调用邮件 | +| send_due 没有到点对象 | `not_run`，作为正常分钟扫描，不向 APScheduler 抛错 | +| send_due 返回 failed/held | 调度异常，不能记为成功 | +| Windows 第二个任务安装失败 | 删除刚创建的第一个任务并返回非零 | +| fastapi owner 下安装/启用 Windows 任务 | 操作被阻断 | + +## 验证 + +- P1.2 定向测试：55 项通过（2.42 秒）。 +- 隔离工作区完整测试：在新增最后 1 项 API 断言前为 518 项通过、1 条既有 warning；新增断言已包含在上述 55 项定向测试中。 +- Python `compileall`：通过。 +- 前端生产构建：通过（4596 modules transformed，约 5.65 秒）。 +- `git diff --check`：通过。 +- 独立完整测试随后两次被全局生成 mutex 阻断；进程取证确认锁持有者是另一个正在执行 23–28 号群组 Prompt 重建并调用摘要模型的真实任务，不是 P1.2 回归，也未强制终止。待该写入任务自然结束后补做一次无并发完整复验。 +- 2026-08-25 10:10 正式 owner 切换完成：`GroupBriefDaily` 与 `GroupBriefDailySend` 均为 `Disabled`，导出的任务 XML `Enabled=false`。 +- 切换后 `scripts/install_daily_task.py status` 返回 `scheduler_owner=fastapi`、两任务 disabled、`outcome=success / exit 0`。 +- 切换后 Alter 管理的 8766 仍正常运行，`/api/system/status` 返回 HTTP 200，下一生成时间为 2026-08-26 00:15。 +- 由于同一工作区另有未提交的 Prompt 生产代码正在执行，本轮没有冒险重启 8766；当前内置 APScheduler 原本已在运行，单一 owner 切换立即生效，新退出码/状态接口将在安全重启后加载。 +- 未触发真实 AI、邮件或微信发送。 + +## 回滚 + +回滚必须按顺序进行，不能先启用 Windows 任务： + +1. 将 8766 的 `SCHEDULER_OWNER` 切为 `external` 并重启，确认 API 返回 `scheduler_active=false`。 +2. 再启用 `GroupBriefDaily` 和 `GroupBriefDailySend`。 +3. 如需回滚代码，单独 revert 本次提交；没有数据库 Schema 变更。 + +## 非目标 + +- 不在本轮重写 V1 运行状态；V1 写路径在 P1.5 冻结和退役。 +- 不修改 Provider、Mock、AI/邮件幂等策略。 +- 不新增队列、分布式锁或外部调度平台。 diff --git a/docs/audits/P1_3_PROVIDER_CONFIG.md b/docs/audits/P1_3_PROVIDER_CONFIG.md new file mode 100644 index 0000000..8f37e4e --- /dev/null +++ b/docs/audits/P1_3_PROVIDER_CONFIG.md @@ -0,0 +1,98 @@ +# P1.3 配置与 Provider fail-closed + +日期：2026-08-25 + +## 结论 + +真实运行现在默认拒绝测试 Provider 和未知 Provider 配置。Mock/本地模板不再因为真实依赖缺失而把任务伪装成成功；设置 API 会在任何数据库写入前完成类型、Provider 名称和邮件配置校验。 + +```text +真实运行（默认） + allow_test_providers=false + ├─ History Mock：不注册 + ├─ V1 Template：不降级为成功 + ├─ 未知 summary/sender：配置失败 + └─ 邮件配置不完整：SMTP 前失败 + +显式自动化测试 + allow_test_providers=true + └─ 可按测试设置启用 Mock/Template，不触发外部依赖 +``` + +## 整改前问题 + +- `history_provider_mock_enabled` 默认 true，所有真实历史 Provider 不可用时会自动读 fixtures。 +- history registry 接收了 Settings，却用 `cls()` 重新读取全局缓存；API 刚保存的路径可能不生效。 +- 未知 history Provider 名称被静默忽略。 +- V1 DeepSeek 缺 Key、或 AI 主备都失败时，本地模板会返回 `success=True`。 +- summary 备用 Provider 未知值被当作“没有备用”，没有配置错误。 +- `wechat_sender_mode` 只判断 `legacy_cli`，其他任意拼写都会默认选择 native。 +- `ai_provider` 可在 UI/API 编辑，但真实路由只读取 `summary_provider_primary/fallback`。 +- V1 history provider 选择可在 UI/API 编辑，但正式 V2 始终使用 WeChatDataAnalysis MCP/导出。 +- 邮件只检查 enabled/host；缺 recipient/from、端口错误或账号缺密码时仍可能进入构造/SMTP。 + +## 修改 + +### 测试 Provider 安全闸门 + +- 新增环境级 `allow_test_providers=false`，不通过数据库或设置 API 开启。 +- `history_provider_mock_enabled` 默认 false；即使旧数据库仍存 true，只要安全闸门关闭，registry 也不会追加 Mock。 +- 显式把 `mock` 配成主/备 Provider 且安全闸门关闭时，直接抛配置错误。 +- V1 Template 只在安全闸门开启时用于测试；真实运行 AI 失败会保持失败。 + +### 配置真正传递 + +- history registry 把同一个 Settings 实例传给 WeChatDataAnalysis、wechat-cli 和 Mock。 +- WeChatDataAnalysis 的微信目录探测改用实例 Settings，不再回到全局缓存。 +- history/summary/sender 未知名称均 fail-closed。 +- 图片 Provider 的健康预检移到全局生图锁之前；CLI 配置错误不会因另一个真实生图任务持锁而等待最长一小时，进入锁后仍会二次检查。 + +### 设置 API 与 UI + +- API 先复制运行时 Settings、完成类型和业务校验，再写数据库并应用运行时值。 +- 非法布尔值不再按“任意非空字符串=true”处理。 +- 生产设置入口移除 Mock 开关、V1-only history provider 选择和无实际路由作用的 `ai_provider`。 +- summary、sender 和邮件仍是正式可编辑配置，错误值返回 HTTP 422，且不会产生部分设置写入。 + +### 邮件预检 + +真实发送前统一检查： + +- email enabled +- SMTP host +- 1–65535 端口 +- recipient +- from 或 SMTP user +- 配置 SMTP user 时必须有 password + +调度发现“邮件已启用但配置不完整”时写入 `email_status=failed_config`、`EMAIL_PROVIDER_CONFIG_INVALID`，整批返回 partial，并且不启动邮件子进程。 + +## 配置边界 + +- V2 历史读取唯一正式入口仍是 WeChatDataAnalysis MCP/JSON 导出；没有为追求统一而引入新的数据源抽象。 +- V1 history provider 字段仅保留环境/旧数据库兼容，为 P1.5 冻结退役做准备。 +- Codex → DeepSeek 是两个真实 Provider 之间的显式 fallback，继续保留。 +- V2 Prompt 的确定性版式 fallback 不是外部 Provider Mock，不在本轮删除。 +- 邮件整体关闭仍是合法配置；只有“已启用但配置残缺”才是失败。 + +## 验证 + +- 主工作区定向测试：87 项通过（2.85 秒）。 +- 图片 Provider/锁顺序补充定向测试：52 项通过（2.32 秒）；图片任务测试文件全量 30 项通过（1.88 秒）。 +- 最新隔离快照其余测试 510 项通过（26.04 秒），与图片任务 30 项合计覆盖当前 540 项测试。 +- 隔离快照 Python compileall 通过；最后两项图片改动另行通过 `py_compile`。 +- 隔离前端 production build 通过：TypeScript project build 和 Vite build 成功，4596 个模块完成转换。 +- `git diff --cached --check` 通过。 +- 全局生成 mutex 在验证前已确认可用；此前占锁的 23–28 号群 Prompt 重建任务已自然结束，未强制终止。 +- 首次全量验证又发现 8766 正在真实生图时持有图片 mutex；测试进程已停止且未终止真实生图，并据此修复了 Provider 健康预检与锁的先后顺序。 +- 两次 Operator 全量测试均因与真实生图 mutex 竞争而超出既有时长基线；按执行边界停止重试后，主控通过独立 mutex 的图片测试和排除图片文件的隔离全量测试完成验收。 +- 未调用真实 AI、SMTP、微信或其他外部 Provider。 + +## 部署说明 + +本轮代码提交后需要在工作区无未验证并行生产改动时安全重启 8766，才能让常驻进程加载新安全边界。当前另一个 Prompt/UI 任务仍有未提交文件，因此本轮不把它们一起加载进服务。 + +## 回滚 + +- 代码可整体 revert 本次提交；没有 Schema 变更，也没有修改真实数据库设置值。 +- 回滚后 Mock/Template 的旧 fail-open 行为会恢复，因此只应用于紧急代码回退，不应作为长期配置方案。 diff --git a/docs/audits/P1_4_EXTERNAL_CALL_IDEMPOTENCY.md b/docs/audits/P1_4_EXTERNAL_CALL_IDEMPOTENCY.md new file mode 100644 index 0000000..83ca954 --- /dev/null +++ b/docs/audits/P1_4_EXTERNAL_CALL_IDEMPOTENCY.md @@ -0,0 +1,87 @@ +# P1.4 AI 与邮件外部调用幂等 + +日期：2026-08-25 + +## 结论 + +AI Prompt、生图和 SMTP 发送现在都遵循同一条安全边界：只有能够确认“请求尚未提交”的失败才允许自动重试；一旦请求可能已经到达外部服务但结果无法确认，立即进入 `result_unknown`/人工 hold，禁止自动切换 Provider、重复生成或重复发送。 + +```text +准备/领取操作 + ├─ 明确未提交失败 → 可受控重试 + ├─ 外部调用成功 → 先记录结果/发送凭据，再推进业务状态 + └─ 已提交但结果未知 → hold，必须人工核对 +``` + +## 整改前风险 + +- Codex CLI 会在超时、非零退出或无输出后自动重试，并可能切换 DeepSeek；第一次调用实际成功但本地没拿到结果时会重复扣费。 +- DeepSeek 把读取超时、响应解析错误等“是否已提交未知”的异常当成普通重试条件。 +- Prompt 没有外部操作 claim，也没有“先记录付费结果、再写最终文件”的恢复点；进程在两者之间退出会再次调用模型。 +- 图片生成虽有 manifest，但“进程结束且没有候选文件”会继续下一次外部调用。 +- 邮件按群循环发送，没有逐封稳定身份和交付账本；中途失败后重跑会重复发送前面已经成功的群。 +- 邮件子进程只返回粗粒度成功/失败，调度器无法区分“未提交失败”和“提交结果未知”。 + +## 修改 + +### AI Provider + +- 新增 `ExternalCallNotSubmittedError` 与 `ExternalCallResultUnknownError`，把外部调用失败按是否可能已提交分类。 +- Codex CLI 只在二进制不存在或进程无法启动时视为未提交；超时、非零退出、缺输出、空输出、非法 JSON 都视为结果未知，不再内部重试或切 DeepSeek。 +- DeepSeek 仅对连接前失败、明确 HTTP 429/503 做受控重试；其他 5xx、读取/写入超时、连接中断、HTTP 200 响应不可解析均进入结果未知。 +- 日志只记录稳定 `request_id`、模型和错误类型，不记录 Provider 原始响应或聊天正文。 + +### Prompt 操作恢复 + +- `run.json` 新增逐次 Prompt 操作的 operation id、输入指纹、started/result_recorded/succeeded/unknown 状态和人工 hold。 +- 外部调用前原子 claim；相同运行发现 started 且无可信结束时直接转 unknown，`force` 也不能绕过。 +- 模型返回后先把 Prompt、元数据和 SHA-256 记录到 `run.json`，再原子替换最终 Prompt 文件。 +- 若在结果记录后、最终文件提交前退出，恢复流程直接提交已记录结果，不再调用 AI。 +- Provider 明确返回结果未知时，API、恢复扫描和调度均显示 `PROMPT_RESULT_UNKNOWN`，不降格成可自动重试的普通失败。 + +### 图片生成 + +- 在全局图片锁内部再次检查有效输出，防止两个进程先后等待锁后各调用一次 Provider。 +- 相同输出在非 force 模式下直接复用。 +- manifest 已是 `result_unknown`，或外部进程已经启动但没有可信候选文件时，禁止第二次外部调用并保留人工复核状态。 +- manifest 已确认完成但正式图片后来缺失时，普通恢复也不会重新付费生成；只有显式 force 才能启动新调用。 +- 兼容现有两参数图片生成器；只有支持 `force` 的生成器才传递该参数。 + +### 邮件发送 + +- 新增稳定语义指纹和稳定 `Message-ID`；MIME 随机 boundary 不影响同一封业务邮件的身份。 +- 在 `output/.email-delivery/` 建立逐封 JSON 账本，状态为 prepared/submitting/sent/failed_before_submit/unknown。 +- 逐封发送全程持有进程内锁和 Windows 命名互斥锁；同一邮件已 sent 时直接跳过。 +- 只有连接、TLS 或认证阶段失败可以重试；进入 `send_message` 后的异常、部分收件人拒绝或中断统一为 unknown，后续运行禁止自动重发。 +- SMTP `quit` 在 sent 持久化之后执行；关闭连接失败不会触发重复发送。 +- 日报脚本按 sent/already_sent/failed_before_submit/unknown 汇总，使用稳定退出码：0 成功、2 部分、3 结果未知、1 失败。 +- 有群因报告或附件缺失而跳过时，日报脚本和 V1 邮件服务不再把其余群发送成功误报成全量成功。 +- V2 调度把退出码 3 映射为 `EMAIL_RESULT_UNKNOWN`、`email_hold=true`，不能再把它记成成功。 + +## 保留边界 + +- 微信发送已有 claim/lease/unknown 账本，本轮不重复改写。 +- SMTP 服务端没有通用 Idempotency-Key；稳定 Message-ID 便于对账，但真正防重依赖本地账本和 fail-closed 状态。 +- V1 数据库级生成唯一约束留到 P1.5 双轨冻结时处理，避免在双轨仍活动时引入新的状态竞争。 +- unknown 不自动猜测成功或失败；人工核对后的解除流程应作为独立、可审计操作实现。 + +## 验证 + +- P1.4 相关回归套件：165 项通过（12.72 秒）。 +- 图片专项：30 项通过（1.94 秒）。 +- 邮件专项：43 项通过（2.44 秒）。 +- 项目 `tests/` 最终全量：559 项通过、1 项失败、1 条弃用 warning（25.59 秒）。 +- 唯一失败为 `test_five_groups_overlap_with_limits_order_and_failure_isolation` 的机器计时阈值：业务结果和并发上限断言均通过，最终实测 0.470 秒，高于固定 0.45 秒；归入 P2.1 的随机顺序/时序测试稳定性整改，不在本轮放宽断言。 +- Python compileall、前端 production build 和 `git diff --check` 通过。 +- 测试全部使用 Fake Provider/Fake SMTP；未调用真实 AI、SMTP、微信或其他外部服务。 +- 测试产生的临时邮件账本已核对并删除，真实 `output` 未留下测试交付记录。 + +## 部署说明 + +8766 常驻进程需要在工作区达到下一个安全检查点后由 Alter 管理器安全重启，才能加载本轮代码。本轮不直接触发真实 AI、邮件或微信来做验收。 + +## 回滚 + +- 没有数据库 Schema 变更，可整体 revert 本次提交。 +- 已产生的生产邮件账本属于防重审计记录，代码回滚时也不应删除。 +- 回滚会恢复自动重试/切备用和逐群重复邮件风险，只适合作为紧急代码回退。 diff --git a/docs/audits/P1_5_V1_FREEZE.md b/docs/audits/P1_5_V1_FREEZE.md new file mode 100644 index 0000000..44ced5f --- /dev/null +++ b/docs/audits/P1_5_V1_FREEZE.md @@ -0,0 +1,85 @@ +# P1.5 V1 双轨冻结与逐步退役 + +日期：2026-08-25 + +## 结论 + +旧 V1 数据库流水线现在默认处于 `read_only`：历史数据库、历史文件和 GET 查询继续保留，但旧生成、旧 Prompt 写入、旧邮件发送和旧 scheduler job 均不能再与正式 V2 Pipeline 形成双写。短期回滚兼容只能通过环境级 `LEGACY_V1_WRITE_MODE=maintenance` 显式开启，设置 UI 和数据库值不能开启它。 + +```text +正式路径 + 8766 APScheduler → V2 DailyPipeline → V2 run.json/output → V2 邮件/微信 + +旧 V1 兼容链 + GET 历史/文件 → 保留只读 + POST/PUT 生成/编辑/邮件 → HTTP 410 + 旧 generate/email job → blocked / exit 3 + ReportService.generate / EmailService.send → 中央策略再次阻断 +``` + +## 侦察证据 + +- 当前前端只使用 `/api/v2/*`，没有发现 `/api/reports/*` 或 `/api/email/*` 写调用。 +- 当前 APScheduler 只注册 `daily_v2_generate_email` 与 `send_wechat_due`，没有注册旧 generate/email job。 +- Windows 正式任务也使用 `daily_auto.py` 和 `run_daily_pipeline.py`，均属于 V2 链路。 +- V1 与 V2 都会调用 AI/SMTP，但分别写数据库状态和 `run.json` 状态；共用 mutex/邮件指纹只能减小并发，不能让两套状态成为同一个真相。 +- `legacy_cli` 是 V2 可选微信 Sender，不是独立 V1 Pipeline，本轮不误删。 + +## 默认冻结的写入口 + +| 入口 | 新行为 | 替代入口 | +| --- | --- | --- | +| `POST /api/reports/generate` | HTTP 410 / `LEGACY_V1_WRITE_BLOCKED` | `POST /api/v2/pipeline/generate` | +| `PUT /api/reports/{id}/prompt` | HTTP 410 / `LEGACY_V1_WRITE_BLOCKED` | V2 run Prompt API | +| `POST /api/email/send` | HTTP 410 / `LEGACY_V1_WRITE_BLOCKED` | V2 每日任务或 V2 邮件脚本 | +| `run_generate_job()` | blocked / exit 3 | `daily_v2_generate_email` | +| `run_email_job()` | blocked / exit 3 | `daily_v2_generate_email` | +| `ReportService.generate()` | 外部调用和数据库写入前抛冻结异常 | `DailyPipeline` | +| `EmailService.send()` | 构建邮件和 SMTP 前抛冻结异常 | V2 per-group 邮件交付 | + +API 层与 Service 层同时检查：API 给旧客户端稳定的 410 契约，Service guard 防止人工 import 或遗留脚本绕过 API。 + +## 保留的只读兼容面 + +- `GET /api/reports/latest` +- `GET /api/runs`、`GET /api/runs/{id}` +- `GET /api/files/*` +- `GET /api/email/preview` +- `GET /api/system/stats` +- V1 SQLite 表、历史 Report/Run/GroupRun 与旧输出目录 + +这些 router/operation 已在 OpenAPI 标记 deprecated，但不会在本轮删除。共享的 `/api/groups/*`、`/api/settings/*` 仍是 V2 正式配置入口，不属于冻结对象。 + +## 配置与防绕过 + +- 默认：`LEGACY_V1_WRITE_MODE=read_only`。 +- 临时兼容：`LEGACY_V1_WRITE_MODE=maintenance`，每次旧写入记录 warning。 +- `/api/system/status` 返回当前模式和 `legacy_v1_writes_active`。 +- `legacy_v1_write_mode` 不在设置 API 可编辑键中。 +- `Settings.apply_runtime_values()` 明确拒绝从数据库应用 `legacy_v1_write_mode`；同时统一保护既有的 `allow_test_providers` 和 `scheduler_owner` 环境级边界。 +- 阻断日志只记录操作名和替代入口，不记录 Prompt、邮件正文、Token 或消息内容。 + +## 验证 + +- P1.5 专项与受影响 V1/V2 回归：133 项通过（17.67 秒）。 +- 项目 `tests/` 全量：567 项通过、1 项失败、1 条弃用 warning（26.78 秒）。 +- 唯一失败仍是 `test_five_groups_overlap_with_limits_order_and_failure_isolation` 的固定 0.45 秒机器计时阈值；业务结果和并发上限断言均通过，本次耗时 0.474 秒。该问题按计划归入 P2.1，不在冻结轮次放宽阈值。 +- Python compileall 通过。 +- 前端 production build 通过：TypeScript 与 Vite 成功，4596 个模块完成转换，Vite 约 5.13 秒。 +- `git diff --check` 通过，仅有工作区既有 LF/CRLF 提示。 +- 全部测试使用本地 Fake/临时数据库；未调用真实 AI、SMTP 或微信。 + +## 部署与观察期 + +1. 提交后由 Alter 安全重启 8766，确认 `/api/system/status` 显示 `read_only`。 +2. 至少观察 7 个完整调度日的 `groupbrief.legacy_v1` 阻断日志，识别是否仍有合法旧客户端。 +3. V1 历史读取、SQLite 表和旧输出至少保留 30 天；没有调用证据后再独立退役旧写服务。 +4. 最终删除路由、表或依赖必须另开轮次，不与本冻结提交混在一起。 + +本轮不会通过真实邮件/AI/微信做验收，也不会删除或改写历史数据。 + +## 回滚 + +- 紧急兼容时设置 `LEGACY_V1_WRITE_MODE=maintenance` 并安全重启 8766；不要重新启用旧 Windows 调度任务。 +- 没有 Schema 或生产数据变更，代码可整体 revert。 +- maintenance 只用于短期回滚窗口；它会恢复 V1/V2 双状态和重复外部调用风险。 diff --git a/docs/audits/P2_1_TEST_SYSTEM.md b/docs/audits/P2_1_TEST_SYSTEM.md new file mode 100644 index 0000000..70abbb9 --- /dev/null +++ b/docs/audits/P2_1_TEST_SYSTEM.md @@ -0,0 +1,39 @@ +# P2.1 前端、E2E 与随机顺序测试验收 + +## 结论 + +P2.1 已补齐可自动执行的前端单元测试、浏览器 E2E 和后端随机顺序测试。此次只修改测试、测试配置、CI 与开发依赖，没有修改生产业务逻辑，也没有连接真实 AI、邮件或微信发送端。 + +## 新增保护 + +- 前端单元测试：覆盖 Hash 路由、V2 生成/发送请求体、结构化 API 错误与损坏归档 JSON。 +- 浏览器 E2E：覆盖 Dashboard 加载、立即生成、发送前确认、确认后发送。`/api/**` 全部由 Playwright 拦截；出现未声明 API 会直接让用例失败。 +- 后端随机顺序：提供 `--random-order-seed=<整数>`，CI 固定使用 `20260825`，便于失败复现。 +- CI：保留正常顺序全量测试，并新增固定随机顺序、前端单测和 Fake API E2E。 + +## 本轮暴露并修复的测试问题 + +1. 并发测试用墙钟时间小于 0.45 秒判断并发。该断言受 Windows 调度、SQLite 和机器负载影响，属于假失败；现在保留并发峰值、结果顺序、失败隔离等行为断言。 +2. 并发测试单独运行时意外依赖其他测试先绑定全局数据库。现在该测试显式隔离不属于其目标的群名同步。 +3. 微信 MCP 的范围读取能力缓存会跨用例泄漏，使正常顺序通过、随机顺序失败。现在每个用例保存、清空并恢复缓存。 + +## 验证结果 + +| 检查 | 结果 | +| --- | --- | +| 后端正常顺序 | `568 passed` | +| 后端随机顺序 seed `20260825` | `568 passed` | +| 后端随机顺序 seed `8675309` | `568 passed` | +| 前端单元测试 | `2 files / 7 tests passed` | +| Dashboard Fake API E2E | `1 passed` | +| 前端生产构建 | 通过，Vite `6.4.3` | +| `npm audit` | `0 vulnerabilities` | +| `npm audit --omit=dev` | `0 vulnerabilities` | + +后端测试仍有一条 Starlette/httpx 兼容性弃用警告，不影响本轮通过；归入 P2.4 依赖治理。 + +## 边界与后续 + +- 当前 E2E 只覆盖最高风险的 Dashboard 生成/发送确认主路径，尚未覆盖 AIImages、Settings、Tasks 重试等完整页面矩阵。 +- E2E 使用 Fake API，只证明前端交互和请求契约，不证明真实 Provider、邮件或微信可用。 +- 本轮不为覆盖率数字添加无业务价值用例。后续新增或拆分核心流程时，应沿 API 契约和失败路径逐步扩充。 diff --git a/docs/audits/P2_2_PIPELINE_STAGES.md b/docs/audits/P2_2_PIPELINE_STAGES.md new file mode 100644 index 0000000..46b7bed --- /dev/null +++ b/docs/audits/P2_2_PIPELINE_STAGES.md @@ -0,0 +1,50 @@ +# P2.2 Pipeline 分阶段拆分验收 + +## 结论 + +`DailyPipeline` 继续作为原 API、CLI、调度器和恢复流程的兼容 Facade；内部单群生成、图片任务和微信发送已拆成独立阶段模块。此次是行为保持型重构，没有修改数据库、`run.json` 契约、状态值、Provider、Prompt、图片生成器或 Sender。 + +## 拆分结果 + +| 项目 | 拆分前 | 拆分后 | +| --- | ---: | ---: | +| `daily_pipeline.py` | 1,498 行 | 852 行 | +| `_generate_one` | 378 行 | 33 行 Facade | +| `_send_one` | 249 行 | 27 行 Facade | + +新增模块： + +- `stage_result.py`：显式区分“继续下一阶段”和“携带业务结果终止”。 +- `generation_stages.py`：运行初始化/防重、消息快照、显式刷新、排行榜、Prompt claim/commit/unknown、图片决策、耗时收口。 +- `image_stages.py`：图片任务构造、串行队列、结果字段落盘、`IMAGE_READY → READY_TO_SEND` 收口。 +- `delivery_stages.py`：发送 claim、文件预检、文字、图片、部分成功恢复、unknown hold、`SENT` 收口。 + +## 保留的兼容边界 + +- `DailyPipeline` 构造参数及 `generate_all`、`send_due`、`force_generate`、`force_send`、`rebuild_prompt_from_snapshot` 未改变。 +- `_generate_one_safe`、`_generate_one`、`_run_image_when_ready`、`_make_image_job`、`_image_hook`、`_run_image_jobs`、`_send_one`、`_finish_unknown_send`、`_sync_group_names`、`_load_groups` 仍存在。 +- 图片仍通过 `SerialImageQueue` 严格串行，现有 hook 仍可注入。 +- Prompt 的 `claim → record → commit`、`result_recorded` 恢复和 `result_unknown` hold 未改变。 +- 微信发送仍保留 claim/lease、文字成功后图片失败只补图片、提交后结果未知禁止重复发送、人工审核和逾期确认。 +- 群级失败隔离、并发上限、结果稳定排序和 V1 冻结未改变。 + +## 验证结果 + +| 检查 | 结果 | +| --- | --- | +| 阶段/Pipeline/恢复/调度/幂等定向回归 | `122 passed` | +| 后端正常顺序全量 | `570 passed` | +| 随机顺序 seed `20260825` | `570 passed` | +| 随机顺序 seed `8675309` | `570 passed` | +| Python compileall | 通过 | +| 前端单元测试 | `7 passed` | +| 前端正式构建 | 通过 | + +唯一 warning 仍是已知 Starlette/httpx 弃用提示，归入 P2.4 依赖治理。 + +## 明确未做 + +- 没有把 V1 `ReportService` 合并进 V2。 +- 没有重写状态机、引入事件总线或新框架。 +- 没有改变调度所有权、API 同步执行方式或数据库事务模型。 +- 没有启动/停止 8766，也没有调用真实 AI、邮件或微信。 diff --git a/docs/audits/P2_3_API_FRONTEND_MODULES.md b/docs/audits/P2_3_API_FRONTEND_MODULES.md new file mode 100644 index 0000000..34cc88a --- /dev/null +++ b/docs/audits/P2_3_API_FRONTEND_MODULES.md @@ -0,0 +1,49 @@ +# P2.3 API 与前端 God Module 拆分 + +日期：2026-08-25 + +## 目标与边界 + +本轮只按已有职责拆分两个高耦合模块，不改变 Pipeline、RunStore、数据库、Provider、真实生图或发送流程。 + +- 保留全部 `/api/v2` 路径、HTTP 方法、状态码、响应结构和默认 OpenAPI operation ID。 +- 保留 `app.api.v2_ui` 的 `router`、`RunPromptUpdateBody`、`RetryBody`、`update_run_prompt`、`restore_run_prompt`、`retry_failed` 与 `_store` 兼容入口。 +- 保留 `AIImages` 默认导出、`/#/images` 路由、原有 `.ai-images-*` CSS 类、可见文案、aria 标签和 API 封装。 +- 验证只使用隔离测试或前端 Fake API，不连接真实 8766，不调用真实 AI、生图、邮件或微信。 + +## 拆分结果 + +### 后端 API + +| 模块 | 职责 | 行数 | +| --- | --- | ---: | +| `app/api/v2_ui.py` | 聚合入口、系统诊断、Pipeline 命令、兼容导出 | 333 | +| `app/api/v2_ui_common.py` | 请求模型、RunStore/时区/路径校验公共边界 | 86 | +| `app/api/v2_ui_read.py` | Dashboard、运行列表、归档、输出文件读取 | 356 | +| `app/api/v2_ui_images.py` | 图片主题、运行 Prompt、重建与重新生图命令 | 321 | + +原 `v2_ui.py` 为 866 行且同时承担查询、归档、图片编辑、系统探测和 Pipeline 命令；拆分后最大文件为 356 行，聚合入口降为 333 行。总行数增加来自明确的模块接口、独立 imports 和说明，不是新增业务分支。 + +### 前端 AI 图片工作台 + +| 模块 | 职责 | 行数 | +| --- | --- | ---: | +| `AIImages.tsx` | 页面组合与顶层加载/失败边界 | 29 | +| `ai-images/useAIImageCatalogs.ts` | 群、主题、默认 Prompt 一次性并行加载 | 71 | +| `ai-images/ImageStylePanel.tsx` | 群级默认生图风格状态、预览与保存 | 242 | +| `ai-images/useAIImageRuns.ts` | 运行选择、详情、Prompt 命令、轮询和发送确认 | 332 | +| `ai-images/AIImageRunWorkspace.tsx` | 运行筛选、详情、图片和操作区视图 | 144 | +| `ai-images/model.tsx` | 状态标签、错误说明、稳定 key 与 Prompt 预览 | 98 | + +原 `AIImages.tsx` 为 696 行，约 40 个状态与目录加载、轮询、Prompt 编辑、重画、发送和 JSX 混在同一组件。拆分后页面入口为 29 行，最大内部模块为 332 行；共享目录仍只加载一次，群默认风格与当天运行的状态边界明确分离。 + +## 契约保护 + +- `tests/test_v2_ui_router_contract.py` 固定 20 个 V2 UI 路由的路径、方法和 operation ID，并检查旧模块兼容导出。 +- `frontend/src/pages/v2/ai-images/model.test.ts` 覆盖运行 key、404 可操作错误、Prompt 预览变量和时间展示。 +- `frontend/e2e/ai-images.spec.ts` 通过全量拦截 `/api/**` 的 Fake API 验证目录、运行列表、详情与筛选；任何漏拦截请求都会直接失败。 +- 既有归档、Prompt 编辑、损坏状态恢复测试继续覆盖移动后的后端实现。 + +## 当前结论 + +P2.3 完成的是低风险职责拆分，不是业务重写。两个 God Module 的修改半径已经缩小：以后修改归档查询不会同时碰 Prompt 编辑，修改群默认风格也不会同时进入运行级发送视图。`useAIImageRuns.ts` 仍集中管理同一运行工作区的状态机，暂不继续强拆，避免把相互依赖的状态分散成跨 Hook 隐式耦合。 diff --git a/docs/audits/P2_4_DEPENDENCIES_OBSERVABILITY_PERFORMANCE.md b/docs/audits/P2_4_DEPENDENCIES_OBSERVABILITY_PERFORMANCE.md new file mode 100644 index 0000000..fa6a32c --- /dev/null +++ b/docs/audits/P2_4_DEPENDENCIES_OBSERVABILITY_PERFORMANCE.md @@ -0,0 +1,80 @@ +# P2.4 依赖、可观测性与证据化性能优化 + +日期：2026-08-25 + +## 结论 + +本轮没有盲目升级 React、提高生图并发或压缩 Prompt，而是只处理审计中已有直接证据的问题：Python 环境不可复现、分类日志可能未初始化、启动检查异常被吞、健康检查读写边界混乱、重复深度探测、两处前端 N+1、恢复扫描重复、固定轮询吞错和单主 Bundle。 + +## 依赖治理 + +- 新增 `requirements.lock`，记录当前 Windows / Python 3.12 已通过完整测试的直接与传递依赖版本；`requirements.txt` 和 `requirements-dev.txt` 继续表达兼容下限，安装时用 `-c requirements.lock` 收敛到验证组合。 +- CI 改为使用 lock constraints，并通过 `pytest-cov` 生成 Sonar 已配置读取的 `coverage.xml`。 +- `.gitignore` 明确排除 `.coverage`、`coverage.xml` 和 `htmlcov/`，不提交临时覆盖率产物。 +- 新增 Dependabot 周更检查 pip、npm 与 GitHub Actions，只创建可审阅 PR，不自动做大版本升级。 +- 当前 `pip check` 无破损依赖；`npm audit` 为 0 vulnerability。旧审计中的 Vite 5 漏洞结论已过期，当前为 Vite 6.4.3，本轮不再无依据升级。 +- Starlette/FastAPI 仍发出 TestClient/httpx 迁移 warning；当前测试功能正常，等待上游支持路径明确后单独升级，不用忽略规则掩盖。 + +## 可观测性 + +### 日志与启动检查 + +- `setup_logging()` 不再因为 Uvicorn/宿主已安装 root handler 就提前退出；分类文件 handler 始终按目标路径幂等配置。 +- 日志清理异常不再静默吞掉，会进入 `app.log`。 +- 启动检查发生未捕获异常时，保存明确的 `ERROR` 检查项和错误摘要，并记录 exception；不再用空数组伪装成“没有问题”。 + +### 健康检查分层 + +- `GET /api/system/health`：纯 liveness，无外部调用、无写入。 +- `GET /api/system/ready`：只读检查数据库、输出目录、默认模板和启动检查捕获状态；不调用 Provider、不创建探测文件。关键本地依赖失败时返回 HTTP 503。 +- `GET /api/system/providers`：只读返回最近一次持久化结果，不再因 GET 执行外部检查或写库。 +- `POST /api/system/providers/refresh`：显式执行 Provider 检查并持久化，每个 Provider 最多保留最近 100 条记录。 +- `GET /api/v2/system/startup`：读取启动时保存的快照，不在浏览器刷新时再次执行启动探测。 +- V2 深度健康响应结构保持不变，但 Codex summary、Codex image 和原生微信 sender 在同一请求内只生成一次 health report。 + +任务中心已改用轻量 readiness；设置中心仍保留用户明确触发的深度诊断，Dashboard 为发送按钮安全门禁仍读取真实 sender 健康状态。 + +## 有证据的性能优化 + +### 批量运行文件摘要 + +`GET /api/v2/runs` 新增可选 `include_files=true`。默认响应不变；显式启用时，每个 run 附带 allowlist 内的文件名。 + +- Tasks 和 ChatRecords 使用批量字段，不再对 N 条 run 再发 N 个详情请求。 +- 如果连接到尚未支持新字段的旧后端，前端仍回退到原逐条详情请求，不牺牲兼容性。 +- Fake API E2E 明确断言任务中心没有发出 `/api/v2/runs/{group}/{date}` 请求，也没有误用深度 health。 + +### 恢复扫描 + +`/api/v2/system/recovery` 现在只读取一次 run 快照，并把同一快照交给未完成扫描和完整性验证；此前会完整遍历并读取两次文件系统。 + +### 重新生图轮询 + +- 用完成后再排下一次的 `setTimeout` 替代固定 `setInterval`，避免慢请求重叠。 +- 正常状态为 2 秒，Desktop fallback 为 5 秒；连续失败指数退避，最大 30 秒。 +- 失败不再静默，页面会显示错误和下一次重试时间；成功后清除错误，状态机与人工发送门禁不变。 + +### 前端拆包 + +所有页面改为 React lazy route chunks。生产构建实测： + +| 指标 | 修改前 | 修改后 | 变化 | +| --- | ---: | ---: | ---: | +| 主 JS | 469.02 kB | 192.99 kB | -58.9% | +| 主 JS gzip | 132.78 kB | 60.00 kB | -54.8% | + +AIImages、Settings、Archive 等页面成为独立 chunk，hash 路由和页面行为不变。 + +## 测试与安全边界 + +- 新增 logging 已有 root handler、启动异常保存、readiness 200/503、Provider retention、深度报告复用、恢复单次扫描、批量 files、轮询退避和 Tasks Fake API E2E 测试。 +- 覆盖率 XML 已成功生成：Python `app + scripts` 行覆盖率 78.88%（8,203 / 10,399）。覆盖率用于识别缺口，不设为了数字而写测试的门槛。 +- 最新回归：后端全量 582/582；两个随机种子各 582/582；前端单测 13/13；Fake API E2E 4/4；compileall、带 `requirements.lock` 的 pip dry-run、`pip check`、`npm audit` 和正式 build 均通过。 +- 全部浏览器测试拦截 `/api/**`；本轮未访问真实 8766，未调用真实 AI、生图、邮件或微信。 + +## 明确保留的非目标 + +- 图片单线程队列与 Codex 全局互斥是当前桌面资源保护，不提高并发。 +- AI 有界重试和结果未知保护不动。 +- RunStore 全量扫描未来仍需分页；当前规模下先消除重复扫描和 HTTP N+1，不引入缓存失效复杂度。 +- 通用 request/job correlation ID 可在独立可观测性轮次增加；本轮先保证错误不会丢失、健康边界不会假成功。 diff --git a/docs/tasks/2026-08-24-image-style-library-prompt-hardening.md b/docs/tasks/2026-08-24-image-style-library-prompt-hardening.md new file mode 100644 index 0000000..b83a44f --- /dev/null +++ b/docs/tasks/2026-08-24-image-style-library-prompt-hardening.md @@ -0,0 +1,72 @@ +# 生图风格库与 Prompt 补强执行任务 + +## 背景 + +GroupBrief 已支持每日可复现随机风格、自定义风格、动态漫画分镜与运行级 Prompt 编辑，但随机风格家族尚未作为可搜索的正式目录开放，群级配置也只能在每日随机和自定义文本之间切换。现有 Prompt 对真实聊天事实已有约束，仍需进一步明确手机端画布、逐话题文字配额、可绘制镜头、空间不足时的降级顺序及重新生图不变量。 + +本任务只抽象借鉴外部风格分类与 OpenAI 官方图像提示词指南，不复制外部仓库的 Prompt、图片、品牌、角色、艺术家姓名或参考图依赖。 + +## 目标 + +- 把现有 10 个随机风格家族整理为带稳定键、分类、说明和色板的正式目录，并新增 12 个家族。 +- 每个命名预设固定家族，但在家族内提供 16 种可复现的每日微变化；每日随机覆盖 22 个家族的 352 种组合。 +- 群级默认风格和运行级风格共用紧凑、可搜索、可筛选且支持键盘操作的选择器。 +- 强化微信手机端竖版画布、逐字文字、动作镜头、空间降级和重新生图不变量，不降低正常 5～7 个真实话题的信息密度。 +- 修复 AI 图片页选题评分字段与后端不一致导致的潜在运行时错误。 + +## 允许修改范围 + +- `app/ai/image_themes.py` 风格目录、解析、兼容和公开元数据。 +- V2 风格目录与解析 API 的响应元数据，不改变现有请求体。 +- `templates/image_prompt/default.md`、`app/ai/prompt_templates.py` 与 `app/ai/prompt_builder.py` 的图片 Prompt 约束。 +- `frontend/src/pages/v2/AIImages.tsx`、必要的前端类型和 `frontend/src/styles.css`。 +- 与风格、Prompt、群 API、运行级编辑和 UI API 直接相关的测试。 +- 本任务说明文件。 + +## 禁止修改范围 + +- 不新增或迁移数据库列，不改变调度、归档、微信发送、邮件发送或生图调用链路。 +- 不读取真实聊天，不修改 `data/groupbrief.db`，不写入 `output/`。 +- 不调用真实 ImageGen，不发送微信或邮件，不修改真实群配置。 +- 不引入外部图片、预览图、品牌、角色 IP、艺术家姓名、`REFERENCE_0` 或参考图依赖。 +- 不读取、输出或提交 `.env`、API Key、Token、Cookie、浏览器数据或其他 secrets。 +- 不修改 Codex 模型提供商、登录方式或认证配置。 + +## 已确定实现要求 + +1. 公开目录固定包含 `random_preset`、`custom` 两个模式和 22 个命名预设；每项提供稳定键、名称、说明、类型、分类、三色色板和变化数量。 +2. 每个家族提供两组画材、配色、纹理、光影候选，共 16 种组合；统一安全尾句只控制美术语言和视觉质感，不得删改事实、人物、数字或指定文字。 +3. 目录版本升级为 `daily-style-v3`。种子包含版本、主题键、群键和运行日期；同群同日一致，跨日尽量避免与上一签名完全相同。 +4. 安全的 `daily-style-v2` 已保存 `theme_prompt` 原样复用；旧版中夹带版式结构的 Prompt 拒绝复用。历史具体主题键继续解析但不在公开目录展示。 +5. 群级保存命名预设时写入预设键并清空自定义文本；未保存草稿跨目标群切换保留。运行级切换只替换 Prompt 的【大主题】段。 +6. Prompt 明确微信手机端 `1024×1536` 竖版、安全边距、逐话题短标题/真实姓名/事实句/主气泡、可绘制镜头、逐字文字恰好一次、空间降级顺序和重新生图不变量。 +7. 前端评分字段使用后端真实的 `comedy`、`group_recognition`、`visual`、`discussion`、`participation`、`continuity`。 + +## 验收标准 + +- 后端目录测试覆盖 22 个公开家族、每个 16 种变化、唯一键、合法色板及禁用词/IP/参考图依赖。 +- 确定性和兼容测试覆盖同日一致、跨日微变化、上一签名排除、v2 复用、旧版污染拒绝与历史主题键解析。 +- API 测试覆盖稳定顺序、新字段、群配置预设保存及运行级只替换主题段。 +- Prompt 测试确认文件模板与内置模板一致，新画布、文字配额、镜头动作、降级顺序和不变量存在，2～7 个真实话题校验不回退。 +- 前端构建、后端定向测试、完整测试、`git diff --check` 通过。 +- 隔离数据库浏览器检查覆盖搜索、分类、预设/自定义/随机、保存重载、运行级替换、评分卡、Esc 和 1280×720 横向布局。 +- 最终检查实际 diff、依赖、硬编码、TODO/debug、临时文件和范围外修改。 + +## 测试命令 + +```powershell +$env:DATABASE_URL='sqlite:///data/style-library-smoke.db' +.\.venv\Scripts\python.exe -m pytest -q tests/test_daily_random_theme.py tests/test_v2_group_prompt_api.py tests/test_v2_prompt_builder.py tests/test_image_layouts.py tests/test_v2_prompt_editing.py tests/test_ui_api.py +.\.venv\Scripts\python.exe -m pytest -q +npm --prefix frontend run build +git diff --check +``` + +浏览器检查使用隔离数据库和本地测试服务；不得连接真实生产数据库或触发任何外部发送/生图动作。 + +## 返回格式 + +- 风格目录、选择器、Prompt 补强和兼容性说明。 +- 修改文件、定向/全量测试、前端构建和隔离 UI 检查结果。 +- 明确声明未执行真实生图、未读取真实聊天、未发送微信或邮件、未修改生产数据库。 +- Git 分支、提交哈希、远端仓库地址和推送结果。 diff --git a/docs/tasks/2026-08-26-automation-hardening.md b/docs/tasks/2026-08-26-automation-hardening.md new file mode 100644 index 0000000..f8353f3 --- /dev/null +++ b/docs/tasks/2026-08-26-automation-hardening.md @@ -0,0 +1,58 @@ +# 2026-08-26 自动化恢复链路加固 + +## 背景 + +2026-08-26 的每日任务暴露出两类独立问题：Codex 生图已经落盘但进程超时且最终结构化回执缺失，任务无法安全认领图片；微信文字实际提交后，固定延迟截图没有观察到输入区暂存状态，发送结果被保守标记为未知并暂停图片阶段。 + +## 目标 + +- 使用同一 Codex exec 线程标识安全恢复超时后已经落盘的唯一图片，不猜图、不重复付费生成。 +- 将微信提交验证改为有界轮询，并在按 Enter 前强制证明输入区已经暂存内容。 +- 为文字发送未知增加带并发保护的人工消歧，只更新阶段检查点，不在消歧接口内发送。 +- 让 Dashboard、任务中心和调度日志准确展示暂停任务和恢复来源。 +- 保持逐群邮件账本幂等；仅对本次新恢复的群重新进入邮件收口。 + +## 允许修改范围 + +- `app/image`、`app/pipeline`、`app/scheduler`、`app/sender`、`app/v2`、`app/api`、`app/config`、`app/db` 中与本任务直接相关的代码。 +- `frontend/src` 中 Dashboard、任务中心、设置与 API 类型。 +- 对应后端与前端测试、本文档。 + +## 禁止修改范围 + +- 不修改 2026-08-26 的 `output`、调度账本、邮件账本或发送时间戳。 +- 不触发真实 Codex 生图、SMTP 邮件、微信消息、部署或数据库迁移。 +- 不读取或写入 API Key、Token、Cookie、密码、`.env` 内容和浏览器数据。 +- 不改变 Codex Provider、登录方式或认证配置。 + +## 已确定实现要求 + +- `codex exec --json` 的 `thread.started.thread_id` 必须流式、原子写入 attempt manifest；结构化回执优先，线程目录候选仅在新增、唯一、路径归属、尺寸和哈希全部通过时可信。 +- 没有可信证据时保持 `result_unknown`，不得启动下一次外部生成。 +- 微信暂存轮询默认 5 秒、提交验证默认 8 秒、轮询间隔默认 0.2 秒；暂存未观察到时不得按 Enter。 +- 人工消歧请求必须携带 `expected_send_unknown_at`，只支持文字未知；`text_sent` 续跑图片阶段，`not_sent` 重置文字阶段，两者均不直接发送。 +- 新字段向后兼容；Dashboard 增加 `held` 计数并展示 `send_error`。 +- 2026-08-26 不补发任何缺失内容。 + +## 验收标准 + +- 超时进程创建的同线程唯一有效图片能被认领；目录不匹配、旧文件、多候选和无效尺寸均保持未知。 +- 暂存延迟可以成功等待；暂存未出现绝不提交；提交后歧义仍 fail closed。 +- 人工消歧具备时间戳 CAS、幂等和阶段续跑测试。 +- Dashboard、任务中心、API 和调度日志测试通过。 +- 所有测试使用临时目录和假发送器，不产生真实外部动作。 +- 最终检查工作区 diff、依赖、调试残留和范围外修改。 + +## 测试命令 + +- `.venv\\Scripts\\python.exe -m pytest tests/test_v2_image_task.py tests/test_v2_wechat_native.py tests/test_v2_pipeline.py tests/test_v2_ui_router_contract.py tests/test_scheduler.py tests/test_send_daily_email.py -q` +- `.venv\\Scripts\\python.exe -m pytest tests -q` +- `npm test -- --run` +- `npm run lint` +- `npm run build` + +前端命令从 `frontend` 目录运行；最终具体脚本以 `package.json` 为准。 + +## 返回格式 + +最终报告应包含：两类根因、实际修改、测试命令与结果、运行时健康验证、未执行的真实外部动作、Git 提交哈希、分支、远端地址和推送结果。 diff --git a/docs/tasks/2026-08-26-status-style-motion.md b/docs/tasks/2026-08-26-status-style-motion.md new file mode 100644 index 0000000..a69aa4e --- /dev/null +++ b/docs/tasks/2026-08-26-status-style-motion.md @@ -0,0 +1,73 @@ +# GroupBrief 状态闭环、风格中心与全站克制动效改造 + +## 背景 + +当前仪表盘会把发送结果无法自动确认的任务持续显示为“暂停待核对”，刷新不会把用户在微信中手工完成的发送写回 `run.json`；排行榜与图片分散展示，聊天记录与排行榜也没有默认筛选当天。AI 图片风格选择器使用大型下拉层，缺少可核对的示例图。全站只有零散 CSS 过渡，没有统一的减弱动态效果支持。 + +## 目标 + +- 让 Dashboard 支持指定运行日期、排行榜预览和每群完整任务卡。 +- 增加不触发微信发送的人工状态处理闭环，并保留冲突保护与审计历史。 +- 调整主导航顺序，聊天记录和排行榜默认使用上海本地当天。 +- 将图片风格选择器改为有草稿/确认语义的风格中心，并提供 22 张本地 WebP 示例图。 +- 接入 `motion@^13.1.1`，只在页面、弹窗和内容切换等有业务意义的位置使用克制动效。 +- 对减少动态效果、键盘焦点、Escape、遮罩关闭、`aria-live` 和窄屏布局提供完整降级。 + +## 允许修改范围 + +- `app/api/`、`app/pipeline/`、`app/v2/` 中与本次 API、状态和主题目录直接相关的文件。 +- `frontend/` 的源码、测试、依赖清单、锁文件与本次新增静态资源。 +- `tests/` 中本次功能的后端测试。 +- 本任务文档。 + +## 禁止修改范围 + +- 主题解析规则、Prompt 事实约束、图片生成模型和真实微信发送实现。 +- 现有历史 Prompt、历史图片和非目标群任务状态。 +- 数据库结构、生产数据库和远程服务器。 +- 任何 secrets、`.env`、Cookie、浏览器数据或用户未授权的外部发送。 + +## 已确定实现要求 + +- `run.json` 继续作为任务状态权威来源；人工处理使用 `expected_updated_at` 做 CAS。 +- 新接口只写状态和审计历史，绝不调用发送器；支持 `all_sent`、`text_sent`、`not_sent`。 +- `MotionConfig reducedMotion="user"` 与 `LazyMotion + domAnimation` 统一接入；动画以 160–240ms 的 opacity/transform 为主。 +- 长列表不做逐项 stagger，不添加持续闪烁、弹跳、数字滚动、虚构进度或装饰性循环。 +- 风格修改先保存在模态草稿中，只有“使用这个风格”才提交；关闭和取消不保存。 +- 示例图不包含真实群数据、文字、品牌、版权角色或水印；压缩为 WebP 后单张目标不超过 300KB，总体不超过 8MB。 + +## 验收标准 + +- Dashboard 指定日期能正确返回任务统计、Top 5 排行和图片；人工确认后统计即时更新。 +- 人工确认写入 `send_resolution_history`；过期 `expected_updated_at` 返回冲突；重复确认不触发发送。 +- 主导航顺序正确；聊天记录和排行榜默认显示上海本地当天，且可以清空查看全部历史。 +- 风格中心四页签、搜索/分类、22 个风格、示例/色板、草稿确认和取消语义可用。 +- 普通动画和 reduced-motion 下页面最终 DOM、Toast、弹窗、内容切换与焦点行为一致。 +- Motion gzip 增量目标不超过约 30KB；窄屏、长消息列表和图片页无明显布局抖动或点击阻塞。 +- Eason 当天 `run.json` 先备份，再以 `all_sent` 核对为已发送；不调用微信发送，Dashboard 为已发送 6、暂停待核对 0。 + +## 测试命令 + +```powershell +.\.venv\Scripts\python.exe -m pytest tests\test_v2_ui_router_contract.py tests\test_v2_pipeline.py tests\test_daily_random_theme.py -q +Set-Location frontend +npm test +npm run build +npm run test:e2e +``` + +## 返回格式 + +- 列出核心功能、测试/构建/E2E 结果、资源体积与 Motion gzip 增量。 +- 报告服务重启后 8766 的监听归属和真实页面核验结果。 +- 报告 Eason 备份路径、状态变更与“未触发微信发送”的证据。 +- 报告 Git 分支、提交哈希、远端地址和普通推送结果。 + +## 完成记录 + +- 后端完整测试：592 passed；前端单元测试：20 passed；E2E：10 passed。 +- 前端构建通过，主 JS gzip 90.06KB、CSS gzip 18.14KB；相对改造前主 JS 约增加 30.07KB。 +- 22 张 WebP 示例图均为 1024×1536，合计 4,855,496 bytes，最大 300,318 bytes。 +- Alter 登记 `GroupBrief-Backend` 已按精确 ID 重启；8766 监听进程祖先为 Alter daemon。 +- Eason 当天状态备份：`output/Eason张UED-4群🤘/2026-08-26/run.json.before-manual-all-sent-20260826-163028.bak`，SHA-256 `9A262868AD653D8C4BE3EC4CD99997AB64675AB00D160524F32FA0A169BC6091`。 +- `all_sent` 人工核对完成：`SENT`、`manual_user_confirmed`，Dashboard 为已发送 6、暂停待核对 0；接口未调用微信发送器。 diff --git a/docs/tasks/2026-08-27-prompt-generation-reliability.md b/docs/tasks/2026-08-27-prompt-generation-reliability.md new file mode 100644 index 0000000..8cf7a9b --- /dev/null +++ b/docs/tasks/2026-08-27-prompt-generation-reliability.md @@ -0,0 +1,70 @@ +# 2026-08-27 Prompt 生成可靠性修复 + +## 背景 + +2026-08-27 自动生成中出现两类 Prompt 阶段故障： + +1. `Grok App 交流群` 的 Codex GPT 调用超过 240 秒，进入 `PROMPT_RESULT_UNKNOWN`；现有 Dashboard 仍显示“立即生成”，但后台会永久拒绝再次领取，缺少人工消歧闭环。 +2. `米游涩泛二次元同好摸鱼群2.3` 的漫画编辑 JSON 连续三次返回超过 48 字的真实气泡；系统提示没有明确气泡上限，也没有对可安全缩短的真实连续片段做确定性归一化。 + +## 目标 + +- 降低 Codex 文本整理调用因继承用户级推理配置和过短超时导致的超时概率。 +- 保留“结果未知禁止自动重试”的安全语义，并提供显式、CAS 保护的人工确认入口。 +- 让超过 48 字但可从真实原话中提取完整连续短句的气泡稳定通过；不可安全缩短时仍拒绝。 +- Dashboard 对 Prompt 结果未知显示准确操作，不再提供注定失败的普通“立即生成”。 + +## 允许修改范围 + +- `app/providers/ai/codex.py` +- `app/config/settings.py` +- `app/db/repository.py` +- `app/api/settings.py` +- `app/ai/poster_copy.py` +- `app/v2/run_store.py` +- `app/pipeline/daily_pipeline.py` +- `app/api/v2_ui.py` +- `frontend/src/api.ts` +- `frontend/src/pages/v2/Dashboard.tsx` +- 与上述行为直接相关的测试、文档和前端样式（仅在确有需要时） + +## 禁止修改范围 + +- 不修改 Codex 登录方式、认证文件、API Key、Token、Cookie 或 `.env` 内容。 +- 不自动重跑当天真实 Codex/GPT/ImageGen 生成，不发送微信，不发送邮件。 +- 不改变发送结果未知的 fail-closed 语义。 +- 不修改历史 `run.json`、图片、排行榜或聊天快照。 +- 不重置、清理或覆盖用户已有 Git 工作。 + +## 已确定实现要求 + +- Codex 文本调用继续固定使用已配置的 OpenAI 模型，显式隔离用户级非必要运行配置，并使用适合结构化文本整理的受控推理强度。 +- 默认文本调用超时从 240 秒提高到 600 秒；运行中持久化设置通过正式设置 API 单独更新，不直接改 SQLite。 +- 超时、非零退出、缺少最终文本仍视为结果未知，禁止自动切备用或自动重试。 +- Prompt 未知消歧必须校验当前 operation id，且消歧本身不调用外部模型；只有用户随后确认生成时才新建调用。 +- 真实气泡自动缩短只能取自同一 speaker 的原消息连续片段，不得改写、拼接或添加省略号。 +- 无法找到不超过 48 字的完整连续短句时继续返回合同错误。 + +## 验收标准 + +- Codex 命令包含显式隔离/推理参数，模型、认证目录和只读 sandbox 保持不变。 +- Prompt 未知状态无法被普通 force 绕过；错误 operation id 的人工消歧失败；正确确认后可重新领取，但确认动作本身零外部调用。 +- Dashboard 能识别 `prompt_hold`，显示清晰风险说明并要求确认。 +- 超长多句真实原话可确定性选出不超过 48 字的连续完整短句；无安全边界的超长原话仍失败。 +- 针对性后端测试、`pytest tests -q`、前端测试和生产构建通过。 + +## 测试命令 + +```powershell +.venv\Scripts\python.exe -m pytest tests\test_codex_summary_provider.py tests\test_p14_generation_idempotency.py tests\test_v2_prompt_builder.py tests\test_v2_ui_router_contract.py -q +.venv\Scripts\python.exe -m pytest tests -q +npm --prefix frontend test -- --run +npm --prefix frontend run build +``` + +## 返回格式 + +- 根因与修复摘要 +- 测试/运行时证据 +- 未执行的真实外部动作 +- Git 提交、分支、远端与推送结果 diff --git a/frontend/e2e/ai-images.spec.ts b/frontend/e2e/ai-images.spec.ts new file mode 100644 index 0000000..c8e1e4e --- /dev/null +++ b/frontend/e2e/ai-images.spec.ts @@ -0,0 +1,201 @@ +import { expect, test, type Page, type Route } from "@playwright/test"; + +const runDate = "2026-08-25"; +const secondaryGroup = "第二测试群"; + +function topicSelection(prefix: string) { + const candidates = Array.from({ length: 4 }, (_, index) => { + const rank = index + 1; + return { + topic_id: `${prefix}-topic-${rank}`, + rank, + title: `${prefix}候选 ${rank}`, + summary: `${prefix}候选摘要 ${rank}`, + evidence_message_count: 8 - index, + participant_count: 4, + duration_minutes: 12, + score_reason: `${prefix}评分原因 ${rank}`, + selected: rank <= 2, + scores: { + discussion: 8, + participation: 7, + comedy: 9, + group_recognition: 8, + visual: 7, + continuity: 6, + total: 92 - index, + }, + }; + }); + return { + topic_selection_version: "5.0", + candidate_count: candidates.length, + selected_count: 2, + selected_topic_ids: candidates.slice(0, 2).map((candidate) => candidate.topic_id), + candidates, + }; +} + +async function json(route: Route, body: unknown) { + await route.fulfill({ + status: 200, + contentType: "application/json", + body: JSON.stringify(body), + }); +} + +async function installFakeApi(page: Page) { + const calls: string[] = []; + await page.route("**/api/**", async (route) => { + const request = route.request(); + const url = new URL(request.url()); + const path = url.pathname; + calls.push(`${request.method()} ${path}`); + + if (path === "/api/groups") { + return json(route, [{ + id: 7, + display_name: "测试群", + wechat_group_id: "wx-test", + wechat_group_name: "测试群", + enabled: true, + provider_preference: "default", + created_at: "", + updated_at: "", + schedule_rule: "daily", + send_time: "08:30", + summary_model: "default", + prompt_model: "default", + image_enabled: true, + send_target: "测试群", + effective_send_target: "测试群", + send_target_mode: "manual", + ranking_template: "default", + image_prompt_template: "default", + image_theme: "ai_free", + image_theme_custom: "", + has_image_prompt_override: false, + wechat_send_enabled: false, + }]); + } + if (path === "/api/v2/image-themes") { + return json(route, { themes: [ + { key: "ai_free", label: "AI 自由发挥", description: "不注入预设风格", kind: "mode", category: "模式", swatches: [], variation_count: 1, preview_url: "" }, + { key: "random_preset", label: "每日随机", description: "每天稳定随机", kind: "mode", category: "模式", swatches: [], variation_count: 352, preview_url: "" }, + { key: "custom", label: "指定风格", description: "自定义描述", kind: "mode", category: "模式", swatches: [], variation_count: 1, preview_url: "" }, + { key: "paper_cut_layered", label: "分层纸艺插画", description: "纤维纸与柔和投影", kind: "preset", category: "立体与手作", swatches: ["#63B3ED", "#F6C453", "#E34D3B"], variation_count: 16, preview_url: "/assets/image-theme-previews/paper_cut_layered.webp" }, + ] }); + } + if (path === "/api/v2/image-themes/resolve") { + const body = request.postDataJSON() as { image_theme?: string; image_theme_custom?: string }; + return json(route, { requested_key: body.image_theme, display_name: body.image_theme === "custom" ? body.image_theme_custom : "分层纸艺插画", theme_text: "测试风格约束", prompt: "测试 Prompt", style_signature: "test", style_seed: "test" }); + } + if (path === "/api/v2/templates/image_prompt/default") { + return json(route, { name: "default", content: "{{group_name}} {{image_theme}}" }); + } + if (path === "/api/v2/runs") { + return json(route, { runs: [ + { group_name: "测试群", group_id: 7, run_date: runDate, status: "PROMPT_READY", updated_at: `${runDate}T08:00:00`, image_regen_status: "idle" }, + { group_name: secondaryGroup, group_id: 8, run_date: runDate, status: "PROMPT_READY", updated_at: `${runDate}T07:30:00`, image_regen_status: "idle" }, + ], total: 2 }); + } + const runPath = path.match(/^\/api\/v2\/runs\/([^/]+)\/(\d{4}-\d{2}-\d{2})(\/prompt)?$/); + if (runPath) { + const groupName = decodeURIComponent(runPath[1]); + const groupId = groupName === secondaryGroup ? 8 : 7; + if (runPath[3]) { + return json(route, { group_name: groupName, run_date: runDate, content: groupName === "测试群" ? "测试 Prompt" : "第二测试 Prompt", revision: "r1", has_original: true, image_theme: "ai_free", image_theme_custom: "", prompt_edited_at: "", topic_selection: topicSelection(groupName) }); + } + return json(route, { run: { group_name: groupName, group_id: groupId, run_date: runDate, status: "PROMPT_READY", updated_at: `${runDate}T08:00:00`, image_regen_status: "idle" }, files: [] }); + } + throw new Error(`E2E 出现未拦截 API：${request.method()} ${path}`); + }); + return calls; +} + +test("AI 图片工作台通过 Fake API 加载目录、运行与详情", async ({ page }) => { + const calls = await installFakeApi(page); + await page.goto("/#/images"); + + await expect(page.getByRole("heading", { name: "设置群聊生图风格" })).toBeVisible(); + await expect(page.getByRole("heading", { name: "运行记录" })).toBeVisible(); + await expect(page.getByRole("heading", { name: `测试群 · ${runDate}` })).toBeVisible(); + await expect(page.getByText("测试 Prompt", { exact: true })).toBeVisible(); + await expect(page.getByText("显示 2 / 2 条")).toBeVisible(); + + await page.getByPlaceholder("搜索群名").fill("不存在"); + await expect(page.getByText("显示 0 / 2 条")).toBeVisible(); + expect(calls).toEqual(expect.arrayContaining([ + "GET /api/groups", + "GET /api/v2/image-themes", + "GET /api/v2/templates/image_prompt/default", + "GET /api/v2/runs", + ])); +}); + +test("选题评分默认显示前两项，可展开收起并在切换运行时重置", async ({ page }) => { + await installFakeApi(page); + await page.goto("/#/images"); + + const scoreCard = page.getByRole("region", { name: "选题评分" }); + await expect(scoreCard.locator(".ai-images-topic-score-item")).toHaveCount(2); + await expect(scoreCard.getByText("测试群候选 1", { exact: true })).toBeVisible(); + await expect(scoreCard.getByText("测试群候选 3", { exact: true })).toHaveCount(0); + + const expandButton = scoreCard.getByRole("button", { name: "展开其余 2 个" }); + await expect(expandButton).toHaveAttribute("aria-expanded", "false"); + await expect(expandButton).toHaveAttribute("aria-controls", /.+/); + await expandButton.click(); + await expect(scoreCard.locator(".ai-images-topic-score-item")).toHaveCount(4); + await expect(scoreCard.getByRole("button", { name: "收起至 2 个" })).toHaveAttribute("aria-expanded", "true"); + + await scoreCard.getByRole("button", { name: "收起至 2 个" }).click(); + await expect(scoreCard.locator(".ai-images-topic-score-item")).toHaveCount(2); + + await scoreCard.getByRole("button", { name: "展开其余 2 个" }).click(); + await page.locator(".ai-images-run-item").filter({ hasText: secondaryGroup }).click(); + await expect(page.getByRole("heading", { name: `${secondaryGroup} · ${runDate}` })).toBeVisible(); + await expect(scoreCard.locator(".ai-images-topic-score-item")).toHaveCount(2); + await expect(scoreCard.getByText(`${secondaryGroup}候选 1`, { exact: true })).toBeVisible(); + await expect(scoreCard.getByRole("button", { name: "展开其余 2 个" })).toHaveAttribute("aria-expanded", "false"); +}); + +test("风格中心保留草稿，取消不应用，确认后一次提交", async ({ page }) => { + const calls = await installFakeApi(page); + await page.goto("/#/images"); + + await page.getByRole("button", { name: /AI 自由发挥/ }).first().click(); + const dialog = page.getByRole("dialog", { name: "风格中心" }); + await expect(dialog.getByRole("tab")).toHaveCount(4); + await dialog.getByRole("tab", { name: "预设风格" }).click(); + await dialog.getByRole("button", { name: /分层纸艺插画/ }).click(); + await expect(dialog.getByRole("img", { name: /分层纸艺插画/ })).toBeVisible(); + await dialog.getByRole("button", { name: "取消" }).click(); + await expect(page.getByRole("button", { name: /AI 自由发挥/ }).first()).toBeVisible(); + + await page.getByRole("button", { name: /AI 自由发挥/ }).first().click(); + await dialog.getByRole("tab", { name: "自定义描述" }).click(); + await dialog.getByPlaceholder(/低饱和黏土摄影/).fill("低饱和黏土摄影"); + await dialog.getByRole("button", { name: "使用这个风格" }).click(); + await expect(page.getByRole("button", { name: /指定风格/ }).first()).toBeVisible(); + expect(calls.filter((call) => call === "POST /api/v2/image-themes/resolve")).toHaveLength(1); +}); + +test("风格中心在窄屏保持可滚动且操作按钮可达", async ({ page }) => { + await page.setViewportSize({ width: 390, height: 844 }); + await installFakeApi(page); + await page.goto("/#/images"); + + await page.getByRole("button", { name: /AI 自由发挥/ }).first().click(); + const dialog = page.getByRole("dialog", { name: "风格中心" }); + await dialog.getByRole("tab", { name: "预设风格" }).click(); + await expect(dialog.getByRole("img", { name: /分层纸艺插画/ })).toBeVisible(); + await expect(dialog.getByRole("button", { name: "取消" })).toBeVisible(); + await expect(dialog.getByRole("button", { name: "使用这个风格" })).toBeVisible(); + + const box = await dialog.boundingBox(); + expect(box).not.toBeNull(); + expect(box!.x).toBeGreaterThanOrEqual(0); + expect(box!.width).toBeLessThanOrEqual(390); + expect(box!.height).toBeLessThanOrEqual(844); +}); diff --git a/frontend/e2e/dashboard.spec.ts b/frontend/e2e/dashboard.spec.ts new file mode 100644 index 0000000..733528d --- /dev/null +++ b/frontend/e2e/dashboard.spec.ts @@ -0,0 +1,223 @@ +import { expect, test, type Page, type Route } from "@playwright/test"; + +const runDate = "2026-08-25"; + +const dashboard = { + today: runDate, + should_run: true, + period_start: "2026-08-24 00:00:00", + period_end: "2026-08-24 23:59:59", + enabled_groups: 1, + counts: { pending: 0, generated: 1, sent: 0, failed: 0, held: 0 }, + next_send: "08:30（测试群）", + daily_status: { overall_status: "partial", summary: {}, updated_at: "2026-08-25T08:00:00+08:00" }, + runtime: { + schema_version: 2, + run_date: runDate, + run_id: "groupbrief:2026-08-25:test", + updated_at: "2026-08-25T08:00:00+08:00", + overall_status: "partial", + scheduler: { + scheduled_at: "2026-08-25T00:15:00+08:00", + generation_started_at: "2026-08-25T00:15:01+08:00", + generation_completed_at: "2026-08-25T00:20:00+08:00", + generation_status: "success", + }, + summary: { configured_group_count: 1 }, + nodes: [ + { id: "scheduler", label: "调度启动", status: "success", completed_groups: 1, total_groups: 1 }, + { id: "data", label: "读取群消息", status: "success", completed_groups: 1, total_groups: 1 }, + { id: "ranking", label: "生成排行榜", status: "success", completed_groups: 1, total_groups: 1 }, + { id: "prompt", label: "摘要与提示词", status: "success", completed_groups: 1, total_groups: 1 }, + { id: "image", label: "生成图片", status: "success", completed_groups: 1, total_groups: 1 }, + { id: "send", label: "等待发送 / 发送完成", status: "pending", completed_groups: 0, total_groups: 1 }, + ], + groups: [ + { + group_id: "7", + group_name: "测试群", + run_status: "READY_TO_SEND", + current_node: "send", + current_node_label: "等待发送 / 发送完成", + node_status: "pending", + nodes: [ + { id: "scheduler", label: "调度启动", status: "success" }, + { id: "data", label: "读取群消息", status: "success" }, + { id: "ranking", label: "生成排行榜", status: "success" }, + { id: "prompt", label: "摘要与提示词", status: "success" }, + { id: "image", label: "生成图片", status: "success" }, + { id: "send", label: "等待发送 / 发送完成", status: "pending" }, + ], + last_error_type: "", + last_error_summary: "", + updated_at: "2026-08-25T08:00:00+08:00", + }, + ], + }, + cards: [ + { + group_id: 7, + group_name: "测试群", + send_time: "08:30", + schedule_rule: "daily", + image_enabled: false, + ranking_template: "default", + ranking_count_policy: "all_messages", + image_prompt_template: "default", + status: "READY_TO_SEND", + period_start: "2026-08-24 00:00:00", + period_end: "2026-08-24 23:59:59", + message_count: 12, + speaker_count: 3, + image_url: "", + ranking_preview: [{ rank: 1, name: "成员甲", count: 8 }], + ranking_error: "", + error: "", + sent_at: "", + wechat_send_enabled: true, + send_hold: false, + send_state: "ready", + send_hold_reason: "", + send_error: "", + send_error_type: "", + send_unknown_at: "", + updated_at: "2026-08-25 08:00:00", + }, + ], +}; + +async function json(route: Route, body: unknown) { + await route.fulfill({ + status: 200, + contentType: "application/json", + body: JSON.stringify(body), + }); +} + +async function installFakeApi(page: Page, held = false) { + const calls: { path: string; search: string; body: unknown }[] = []; + await page.route("**/api/**", async (route) => { + const request = route.request(); + const url = new URL(request.url()); + const path = url.pathname; + const body = request.postDataJSON?.() ?? null; + calls.push({ path, search: url.search, body }); + + if (path === "/api/v2/dashboard") return json(route, held ? { + ...dashboard, + counts: { ...dashboard.counts, generated: 0, held: 1 }, + cards: dashboard.cards.map((card) => ({ ...card, send_hold: true, send_state: "unknown", send_hold_reason: "SEND_RESULT_UNKNOWN", error: "发送结果需要人工核对" })), + } : dashboard); + if (path === "/api/v2/runtime/logs") { + return json(route, { + run_date: runDate, + updated_at: "2026-08-25T08:00:03+08:00", + truncated: false, + items: [ + { timestamp: "2026-08-25T00:15:01+08:00", level: "INFO", source: "scheduler", message: "00:15 每日任务已启动" }, + { timestamp: "2026-08-25T00:15:03+08:00", level: "WARNING", source: "provider", message: "测试群数据读取稍慢" }, + ], + }); + } + if (path === "/api/v2/system/health") { + return json(route, { + checks: { + wechat_sender: { ok: true, status: "OK", detail: "Fake sender，仅用于 E2E" }, + }, + }); + } + if (path === "/api/v2/pipeline/generate") { + return json(route, { + results: [{ status: "ready_to_send", group_name: "测试群", detail: "Fake 生成完成" }], + }); + } + if (path === "/api/v2/pipeline/send") { + return json(route, { result: { status: "sent", group_name: "测试群" } }); + } + if (path === "/api/v2/pipeline/resolve-manual-send") { + return json(route, { result: { status: "resolved", group_name: "测试群", detail: "已写入人工核对结论；本次操作没有调用微信发送器" } }); + } + throw new Error(`E2E 出现未拦截 API：${request.method()} ${path}`); + }); + return calls; +} + +test("Dashboard 生成与发送确认只命中 Fake API", async ({ page }) => { + const calls = await installFakeApi(page); + await page.goto("/#/dashboard"); + + await expect(page.getByRole("heading", { name: "运行总览" })).toBeVisible(); + await expect(page.getByRole("heading", { name: "任务节点" })).toBeVisible(); + await expect(page.getByRole("heading", { name: "运行日志" })).toBeVisible(); + await expect(page.getByText("00:15 每日任务已启动")).toBeVisible(); + await expect(page.getByText("测试群", { exact: true }).first()).toBeVisible(); + await page.getByLabel("运行日期").fill(runDate); + + await page.getByRole("button", { name: "立即生成" }).click(); + await expect(page.getByText("生成完成：Fake 生成完成")).toBeVisible(); + const generate = calls.find((call) => call.path === "/api/v2/pipeline/generate"); + expect(generate?.body).toEqual({ group_id: 7, force: true, run_date: runDate }); + + await page.getByRole("button", { name: "立即发送" }).click(); + const dialog = page.getByRole("dialog", { name: "确认立即发送" }); + await expect(dialog).toContainText("测试群"); + expect(calls.some((call) => call.path === "/api/v2/pipeline/send")).toBe(false); + + await dialog.getByRole("button", { name: "确认发送" }).click(); + await expect(page.getByText("「测试群」已发送")).toBeVisible(); + const send = calls.find((call) => call.path === "/api/v2/pipeline/send"); + expect(send?.body).toEqual({ group_id: 7, run_date: runDate }); +}); + +test("Dashboard 日志筛选和窄屏布局只使用只读 Fake API", async ({ page }) => { + const calls = await installFakeApi(page); + await page.setViewportSize({ width: 390, height: 844 }); + await page.goto("/#/dashboard"); + + await expect(page.getByRole("heading", { name: "任务节点" })).toBeVisible(); + await expect(page.getByRole("heading", { name: "运行日志" })).toBeVisible(); + await page.getByLabel("日志来源").selectOption("provider"); + await page.getByLabel("日志级别").selectOption("WARNING"); + await page.getByRole("button", { name: "暂停滚动" }).click(); + await expect(page.getByRole("button", { name: "继续滚动" })).toBeVisible(); + await expect.poll(() => calls.filter((call) => call.path === "/api/v2/runtime/logs").length).toBeGreaterThanOrEqual(3); + expect(calls.some((call) => call.search.includes("sources=provider") && call.search.includes("levels=WARNING"))).toBe(true); + expect(await page.evaluate(() => document.documentElement.scrollWidth <= window.innerWidth)).toBe(true); + expect(calls.some((call) => call.path.startsWith("/api/v2/pipeline/"))).toBe(false); +}); + +test("Dashboard 人工核对只写状态，不调用发送接口", async ({ page }) => { + const calls = await installFakeApi(page, true); + await page.goto("/#/dashboard"); + await page.getByLabel("运行日期").fill(runDate); + + const resolveButton = page.getByRole("button", { name: "人工核对" }); + await resolveButton.click(); + const dialog = page.getByRole("dialog", { name: /核对.*测试群.*发送结果/ }); + await expect(dialog).toContainText("不会再次发送任何内容"); + await page.keyboard.press("Escape"); + await expect(dialog).toBeHidden(); + await expect(resolveButton).toBeFocused(); + await resolveButton.click(); + await dialog.getByRole("button", { name: "确认核对结果" }).click(); + await expect(page.getByText("已写入人工核对结论；本次操作没有调用微信发送器")).toBeVisible(); + + const resolved = calls.find((call) => call.path === "/api/v2/pipeline/resolve-manual-send"); + expect(resolved?.body).toEqual({ + group_id: 7, + run_date: runDate, + resolution: "all_sent", + expected_updated_at: "2026-08-25 08:00:00", + }); + expect(calls.some((call) => call.path === "/api/v2/pipeline/send")).toBe(false); +}); + +test.describe("减少动态效果", () => { + test.use({ reducedMotion: "reduce" }); + test("页面转场降级后仍到达最终可访问状态", async ({ page }) => { + await installFakeApi(page); + await page.goto("/#/dashboard"); + await expect(page.getByRole("heading", { name: "运行总览" })).toBeVisible(); + await expect(page.getByText("成员甲")).toBeVisible(); + }); +}); diff --git a/frontend/e2e/date-filters.spec.ts b/frontend/e2e/date-filters.spec.ts new file mode 100644 index 0000000..e388656 --- /dev/null +++ b/frontend/e2e/date-filters.spec.ts @@ -0,0 +1,44 @@ +import { expect, test, type Page, type Route } from "@playwright/test"; + +function shanghaiToday(): string { + return new Intl.DateTimeFormat("en-CA", { + timeZone: "Asia/Shanghai", + year: "numeric", + month: "2-digit", + day: "2-digit", + }).format(new Date()); +} + +async function installRunsApi(page: Page) { + const urls: string[] = []; + await page.route("**/api/v2/runs**", async (route: Route) => { + urls.push(route.request().url()); + await route.fulfill({ + status: 200, + contentType: "application/json", + body: JSON.stringify({ runs: [] }), + }); + }); + return urls; +} + +for (const [name, path] of [ + ["聊天记录", "/#/messages"], + ["排行榜", "/#/ranking"], +] as const) { + test(`${name}默认筛选上海当天，并可清空查看历史`, async ({ page }) => { + const requests = await installRunsApi(page); + await page.goto(path); + + const dateInput = page.getByLabel("运行日期"); + const expected = shanghaiToday(); + await expect(dateInput).toHaveValue(expected); + await expect.poll(() => requests.some((url) => new URL(url).searchParams.get("run_date") === expected)).toBe(true); + + await dateInput.fill(""); + await expect.poll(() => { + const latest = requests.at(-1); + return latest ? new URL(latest).searchParams.has("run_date") : true; + }).toBe(false); + }); +} diff --git a/frontend/e2e/recovery-config.spec.ts b/frontend/e2e/recovery-config.spec.ts new file mode 100644 index 0000000..2752a12 --- /dev/null +++ b/frontend/e2e/recovery-config.spec.ts @@ -0,0 +1,129 @@ +import { expect, test, type Route } from "@playwright/test"; + +async function json(route: Route, body: unknown) { + await route.fulfill({ + status: 200, + contentType: "application/json", + body: JSON.stringify(body), + }); +} + +test("390px 窄屏可核对 48 小时外恢复清单且确认接口不包含发送参数", async ({ page }) => { + await page.setViewportSize({ width: 390, height: 844 }); + let confirmBody: Record | null = null; + await page.route("**/api/**", async (route) => { + const request = route.request(); + const url = new URL(request.url()); + if (url.pathname === "/api/system/ready") { + return json(route, { ready: true, checks: { database: { ok: true } } }); + } + if (url.pathname === "/api/v2/runs") { + return json(route, { runs: [], total: 0 }); + } + if (url.pathname === "/api/v2/system/recovery") { + return json(route, { incomplete: [], integrity: [] }); + } + if (url.pathname === "/api/v2/recovery/backlog" && request.method() === "GET") { + return json(route, { + generated_at: "2026-08-27T09:00:00+08:00", + automatic_recovery_dates: ["2026-08-26", "2026-08-27"], + lookback_days: 30, + version: "b".repeat(64), + items: [{ + run_date: "2026-08-20", + group_id: 7, + group_name: "低风险测试群", + status: "FAILED", + execution_state: "WAIT_RETRY", + reason: "IMAGE_GENERATION_FAILED", + safe_stage: "generation_only", + recoverable: true, + manifest_source: "recorded", + estimated_summary_calls: 1, + estimated_image_calls: 1, + updated_at: "2026-08-20T09:00:00+08:00", + }], + }); + } + if (url.pathname === "/api/v2/recovery/confirm" && request.method() === "POST") { + confirmBody = request.postDataJSON(); + return json(route, { + status: "success", + generation_only: true, + send_invoked: false, + results: [{ group_name: "低风险测试群", status: "ready_to_send" }], + }); + } + throw new Error(`E2E 出现未拦截 API：${request.method()} ${url.pathname}`); + }); + + await page.goto("/#/tasks"); + await expect(page.getByRole("heading", { name: "48 小时外恢复待核对" })).toBeVisible(); + await page.getByRole("checkbox").check(); + await page.getByRole("button", { name: "确认恢复生成（1）" }).click(); + await expect(page.getByText("绝不会发送历史微信", { exact: false })).toBeVisible(); + await page.getByRole("button", { name: "确认，仅恢复生成" }).click(); + await expect.poll(() => confirmBody).not.toBeNull(); + expect(confirmBody).toEqual({ + expected_version: "b".repeat(64), + tasks: [{ run_date: "2026-08-20", group_id: 7 }], + }); + expect(JSON.stringify(confirmBody)).not.toContain("send"); +}); + +test("群配置只展示后端白名单并支持两种统计规则", async ({ page }) => { + await page.setViewportSize({ width: 390, height: 844 }); + await page.route("**/api/**", async (route) => { + const url = new URL(route.request().url()); + if (url.pathname === "/api/groups") { + return json(route, [{ + id: 7, + display_name: "低风险测试群", + wechat_group_id: "test@chatroom", + wechat_group_name: "低风险测试群", + enabled: true, + provider_preference: "wechat_data_analysis", + schedule_rule: "daily_previous_day", + send_time: "08:30", + summary_provider: "codex", + summary_model: "gpt-5.6-sol", + prompt_provider: "codex", + prompt_model: "gpt-5.6-sol", + image_enabled: true, + send_target: "", + ranking_template: "default", + image_prompt_template: "default", + wechat_send_enabled: false, + }]); + } + if (url.pathname === "/api/v2/templates/ranking") { + return json(route, { templates: ["default"], previews: {} }); + } + if (url.pathname === "/api/v2/templates/image_prompt") { + return json(route, { templates: ["default"] }); + } + if (url.pathname === "/api/system/providers") { + return json(route, { + catalog: { + history: [ + { provider: "wechat_data_analysis", label: "WeChatDataAnalysis", available: true, capabilities: ["history"] }, + { provider: "wechat_cli", label: "wechat-cli", available: false, capabilities: ["history"] }, + ], + ai: [ + { provider: "codex", label: "Codex GPT", available: true, models: ["gpt-5.6-sol"], capabilities: ["summary", "prompt"] }, + { provider: "deepseek", label: "DeepSeek", available: false, models: ["deepseek-v4-flash"], capabilities: ["summary", "prompt"] }, + ], + }, + }); + } + throw new Error(`E2E 出现未拦截 API：${route.request().method()} ${url.pathname}`); + }); + + await page.goto("/#/groups/7"); + await expect(page.getByRole("heading", { name: "群配置详情" })).toBeVisible(); + await expect(page.getByLabel("统计周期规则")).toHaveValue("daily_previous_day"); + await page.getByLabel("统计周期规则").selectOption("weekday_default"); + await expect(page.getByLabel("统计周期规则")).toHaveValue("weekday_default"); + await expect(page.getByLabel("摘要 Provider").locator("option[value=deepseek]")).toBeDisabled(); + await expect(page.getByLabel("日报 Prompt Provider").locator("option[value=deepseek]")).toBeDisabled(); +}); diff --git a/frontend/e2e/tasks.spec.ts b/frontend/e2e/tasks.spec.ts new file mode 100644 index 0000000..7753211 --- /dev/null +++ b/frontend/e2e/tasks.spec.ts @@ -0,0 +1,116 @@ +import { expect, test, type Page, type Route } from "@playwright/test"; + +const runDate = "2026-08-25"; + +async function json(route: Route, body: unknown) { + await route.fulfill({ + status: 200, + contentType: "application/json", + body: JSON.stringify(body), + }); +} + +test("任务中心使用批量 files 且不再逐条请求运行详情", async ({ page }) => { + const calls: string[] = []; + await page.route("**/api/**", async (route) => { + const request = route.request(); + const url = new URL(request.url()); + calls.push(`${request.method()} ${url.pathname}${url.search}`); + + if (url.pathname === "/api/system/ready") { + return json(route, { + ready: true, + scheduler_owner: "windows", + scheduler_active: false, + checks: { database: { ok: true, status: "OK", detail: "Fake DB" } }, + }); + } + if (url.pathname === "/api/v2/runs") { + expect(url.searchParams.get("include_files")).toBe("true"); + return json(route, { + runs: [{ + group_name: "测试群", + group_id: 7, + run_date: runDate, + status: "PROMPT_READY", + updated_at: `${runDate}T08:00:00`, + files: ["messages.json", "run.json"], + }], + total: 1, + }); + } + if (url.pathname === "/api/v2/system/recovery") { + return json(route, { + incomplete: [], + integrity: [{ group_name: "测试群", run_date: runDate, status: "PROMPT_READY", missing: [], ok: true }], + }); + } + if (url.pathname === "/api/v2/recovery/backlog") { + return json(route, { + generated_at: "2026-08-27T09:00:00+08:00", + automatic_recovery_dates: ["2026-08-26", "2026-08-27"], + lookback_days: 30, + version: "a".repeat(64), + items: [], + }); + } + throw new Error(`E2E 出现未拦截 API：${request.method()} ${url.pathname}`); + }); + + await page.goto("/#/tasks"); + + await expect(page.getByRole("heading", { name: "任务中心" })).toBeVisible(); + const activeTab = page.getByRole("tab", { name: "任务中心" }); + await expect(activeTab.locator(".workspace-tab-label")).toHaveText("任务中心"); + await expect(activeTab.locator(".workspace-tab-label")).toHaveCSS("z-index", "1"); + await expect(activeTab.locator(".workspace-tab-indicator")).toHaveCSS("z-index", "0"); + await expect(page.getByText("messages.json", { exact: true })).toBeVisible(); + expect(calls.filter((call) => call.startsWith("GET /api/v2/runs/"))).toEqual([]); + expect(calls.some((call) => call.startsWith("GET /api/v2/system/health"))).toBe(false); +}); + +test("聊天记录使用批量 files 且只读取选中的消息文件", async ({ page }) => { + const calls: string[] = []; + await page.route("**/api/**", async (route) => { + const request = route.request(); + const url = new URL(request.url()); + calls.push(`${request.method()} ${url.pathname}${url.search}`); + + if (url.pathname === "/api/v2/runs") { + expect(url.searchParams.get("include_files")).toBe("true"); + return json(route, { + runs: [{ + group_name: "测试群", + run_date: runDate, + status: "SENT", + files: ["messages.json", "run.json"], + }], + total: 1, + }); + } + if (url.pathname.endsWith(`/2026-08-25/messages.json`)) { + return json(route, [{ + message_id: "m1", + group_id: "wx-test", + group_name: "测试群", + sender_id: "u1", + sender_name: "小明", + timestamp: `${runDate}T08:00:00`, + message_type: "text", + content: "Fake 归档消息", + }]); + } + throw new Error(`E2E 出现未拦截 API：${request.method()} ${url.pathname}`); + }); + + await page.goto("/#/messages"); + + await expect(page.getByRole("heading", { name: "聊天记录" })).toBeVisible(); + const activeTab = page.getByRole("tab", { name: "聊天记录" }); + await expect(activeTab.locator(".workspace-tab-label")).toHaveText("聊天记录"); + await expect(activeTab.locator(".workspace-tab-label")).toHaveCSS("z-index", "1"); + await expect(activeTab.locator(".workspace-tab-indicator")).toHaveCSS("z-index", "0"); + await expect(page.getByText("Fake 归档消息", { exact: true })).toBeVisible(); + expect(calls.filter((call) => call.startsWith("GET /api/v2/runs/"))).toEqual([]); + expect(calls.filter((call) => call.endsWith("/messages.json"))).toHaveLength(1); +}); diff --git a/frontend/package-lock.json b/frontend/package-lock.json index 35f835b..7eeb3b7 100644 --- a/frontend/package-lock.json +++ b/frontend/package-lock.json @@ -10,16 +10,73 @@ "dependencies": { "@fontsource-variable/inter": "^5.3.0", "@phosphor-icons/react": "^2.1.10", + "motion": "^13.1.1", "react": "^18.3.1", "react-dom": "^18.3.1", "yet-another-react-lightbox": "3.32.2" }, "devDependencies": { + "@playwright/test": "^1.62.1", "@types/react": "^18.3.3", "@types/react-dom": "^18.3.0", "@vitejs/plugin-react": "^4.3.1", + "jsdom": "^30.0.1", "typescript": "^5.5.3", - "vite": "^5.4.0" + "vite": "^6.4.3", + "vitest": "^3.2.6" + } + }, + "node_modules/@asamuzakjp/css-color": { + "version": "6.0.7", + "resolved": "https://registry.npmjs.org/@asamuzakjp/css-color/-/css-color-6.0.7.tgz", + "integrity": "sha512-vC/bk1Lz7Tn/EfU9/apOTBk80/8dyGyWMowPoV1tJ52muDGsDqt2HPT2klrFUiY60MQmQv9q8yIht15JnBgDGw==", + "dev": true, + "license": "MIT", + "dependencies": { + "@csstools/css-calc": "^3.3.0", + "@csstools/css-color-parser": "^4.1.10", + "@csstools/css-parser-algorithms": "^4.0.0", + "@csstools/css-tokenizer": "^4.0.0", + "lru-cache": "^11.5.2" + }, + "engines": { + "node": "^22.13.0 || >=24.0.0" + } + }, + "node_modules/@asamuzakjp/css-color/node_modules/lru-cache": { + "version": "11.5.2", + "resolved": "https://registry.npmjs.org/lru-cache/-/lru-cache-11.5.2.tgz", + "integrity": "sha512-4pfM1Ff0x50o0tQwb5ucw/RzNyD0/YJME6IVcStalZuMWxdt3sR3huStTtxz4PUmvZfRguvDejasvQ2kifR11g==", + "dev": true, + "license": "BlueOak-1.0.0", + "engines": { + "node": "20 || >=22" + } + }, + "node_modules/@asamuzakjp/dom-selector": { + "version": "8.3.2", + "resolved": "https://registry.npmjs.org/@asamuzakjp/dom-selector/-/dom-selector-8.3.2.tgz", + "integrity": "sha512-93Z1N+BQNXysodoicpOIyNh2drHfz/CTf9nnT0FEx72GJcIiwgydD7tGAr78j41LsYn3hlRn+LdGPuBLn1Bl8Q==", + "dev": true, + "license": "MIT", + "dependencies": { + "bidi-js": "^1.0.3", + "css-tree": "^3.2.1", + "is-potential-custom-element-name": "^1.0.1", + "lru-cache": "^11.5.2" + }, + "engines": { + "node": "^22.13.0 || >=24.0.0" + } + }, + "node_modules/@asamuzakjp/dom-selector/node_modules/lru-cache": { + "version": "11.5.2", + "resolved": "https://registry.npmjs.org/lru-cache/-/lru-cache-11.5.2.tgz", + "integrity": "sha512-4pfM1Ff0x50o0tQwb5ucw/RzNyD0/YJME6IVcStalZuMWxdt3sR3huStTtxz4PUmvZfRguvDejasvQ2kifR11g==", + "dev": true, + "license": "BlueOak-1.0.0", + "engines": { + "node": "20 || >=22" } }, "node_modules/@babel/code-frame": { @@ -304,10 +361,163 @@ "node": ">=6.9.0" } }, + "node_modules/@bramus/specificity": { + "version": "2.4.2", + "resolved": "https://registry.npmjs.org/@bramus/specificity/-/specificity-2.4.2.tgz", + "integrity": "sha512-ctxtJ/eA+t+6q2++vj5j7FYX3nRu311q1wfYH3xjlLOsczhlhxAg2FWNUXhpGvAw3BWo1xBcvOV6/YLc2r5FJw==", + "dev": true, + "license": "MIT", + "dependencies": { + "css-tree": "^3.0.0" + }, + "bin": { + "specificity": "bin/cli.js" + } + }, + "node_modules/@csstools/color-helpers": { + "version": "6.1.1", + "resolved": "https://registry.npmjs.org/@csstools/color-helpers/-/color-helpers-6.1.1.tgz", + "integrity": "sha512-gLNsunvwf3mCi5u5o46/Z/JcJMnhbHSaZ69rkgPzNM3J4s8hWwpPUQB6/tt0EDFyCiWzxANlx+2LJwpYj4zS1w==", + "dev": true, + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/csstools" + }, + { + "type": "opencollective", + "url": "https://opencollective.com/csstools" + } + ], + "license": "MIT-0", + "engines": { + "node": ">=20.19.0" + } + }, + "node_modules/@csstools/css-calc": { + "version": "3.3.0", + "resolved": "https://registry.npmjs.org/@csstools/css-calc/-/css-calc-3.3.0.tgz", + "integrity": "sha512-c5ihYsPkdG6JCkU2zTMm4+k6r7RXuGxtWYhu5DHMIiF1FHzrfmHL5so11AoFpUv/tu61xfcmT4AmKoFfMPoqdQ==", + "dev": true, + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/csstools" + }, + { + "type": "opencollective", + "url": "https://opencollective.com/csstools" + } + ], + "license": "MIT", + "engines": { + "node": ">=20.19.0" + }, + "peerDependencies": { + "@csstools/css-parser-algorithms": "^4.0.0", + "@csstools/css-tokenizer": "^4.0.0" + } + }, + "node_modules/@csstools/css-color-parser": { + "version": "4.2.0", + "resolved": "https://registry.npmjs.org/@csstools/css-color-parser/-/css-color-parser-4.2.0.tgz", + "integrity": "sha512-5+5LEmFuY1AjXdYhmgjTJogtQnP1evJ1zrBZGUNZ0thkpwnnmKxcHdAMn/OtFjAb25zA+jKDVYVRl+5G7rjv1A==", + "dev": true, + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/csstools" + }, + { + "type": "opencollective", + "url": "https://opencollective.com/csstools" + } + ], + "license": "MIT", + "dependencies": { + "@csstools/color-helpers": "^6.1.1", + "@csstools/css-calc": "^3.3.0" + }, + "engines": { + "node": ">=20.19.0" + }, + "peerDependencies": { + "@csstools/css-parser-algorithms": "^4.0.0", + "@csstools/css-tokenizer": "^4.0.0" + } + }, + "node_modules/@csstools/css-parser-algorithms": { + "version": "4.0.0", + "resolved": "https://registry.npmjs.org/@csstools/css-parser-algorithms/-/css-parser-algorithms-4.0.0.tgz", + "integrity": "sha512-+B87qS7fIG3L5h3qwJ/IFbjoVoOe/bpOdh9hAjXbvx0o8ImEmUsGXN0inFOnk2ChCFgqkkGFQ+TpM5rbhkKe4w==", + "dev": true, + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/csstools" + }, + { + "type": "opencollective", + "url": "https://opencollective.com/csstools" + } + ], + "license": "MIT", + "engines": { + "node": ">=20.19.0" + }, + "peerDependencies": { + "@csstools/css-tokenizer": "^4.0.0" + } + }, + "node_modules/@csstools/css-syntax-patches-for-csstree": { + "version": "1.1.8", + "resolved": "https://registry.npmjs.org/@csstools/css-syntax-patches-for-csstree/-/css-syntax-patches-for-csstree-1.1.8.tgz", + "integrity": "sha512-CpMLjAvwQg3BL5S0IeqsZNMH7EQrEWi0kLKOC13ZBF0ZwERiLWlibNPJr8G1kdU3Ms/r2KiNrF81pUh2HwAHdg==", + "dev": true, + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/csstools" + }, + { + "type": "opencollective", + "url": "https://opencollective.com/csstools" + } + ], + "license": "MIT-0", + "peerDependencies": { + "css-tree": "^3.2.1" + }, + "peerDependenciesMeta": { + "css-tree": { + "optional": true + } + } + }, + "node_modules/@csstools/css-tokenizer": { + "version": "4.0.0", + "resolved": "https://registry.npmjs.org/@csstools/css-tokenizer/-/css-tokenizer-4.0.0.tgz", + "integrity": "sha512-QxULHAm7cNu72w97JUNCBFODFaXpbDg+dP8b/oWFAZ2MTRppA3U00Y2L1HqaS4J6yBqxwa/Y3nMBaxVKbB/NsA==", + "dev": true, + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/csstools" + }, + { + "type": "opencollective", + "url": "https://opencollective.com/csstools" + } + ], + "license": "MIT", + "engines": { + "node": ">=20.19.0" + } + }, "node_modules/@esbuild/aix-ppc64": { - "version": "0.21.5", - "resolved": "https://registry.npmjs.org/@esbuild/aix-ppc64/-/aix-ppc64-0.21.5.tgz", - "integrity": "sha512-1SDgH6ZSPTlggy1yI6+Dbkiz8xzpHJEVAlF/AM1tHPLsf5STom9rwtjE4hKAF20FfXXNTFqEYXyJNWh1GiZedQ==", + "version": "0.25.12", + "resolved": "https://registry.npmjs.org/@esbuild/aix-ppc64/-/aix-ppc64-0.25.12.tgz", + "integrity": "sha512-Hhmwd6CInZ3dwpuGTF8fJG6yoWmsToE+vYgD4nytZVxcu1ulHpUQRAB1UJ8+N1Am3Mz4+xOByoQoSZf4D+CpkA==", "cpu": [ "ppc64" ], @@ -318,13 +528,13 @@ "aix" ], "engines": { - "node": ">=12" + "node": ">=18" } }, "node_modules/@esbuild/android-arm": { - "version": "0.21.5", - "resolved": "https://registry.npmjs.org/@esbuild/android-arm/-/android-arm-0.21.5.tgz", - "integrity": "sha512-vCPvzSjpPHEi1siZdlvAlsPxXl7WbOVUBBAowWug4rJHb68Ox8KualB+1ocNvT5fjv6wpkX6o/iEpbDrf68zcg==", + "version": "0.25.12", + "resolved": "https://registry.npmjs.org/@esbuild/android-arm/-/android-arm-0.25.12.tgz", + "integrity": "sha512-VJ+sKvNA/GE7Ccacc9Cha7bpS8nyzVv0jdVgwNDaR4gDMC/2TTRc33Ip8qrNYUcpkOHUT5OZ0bUcNNVZQ9RLlg==", "cpu": [ "arm" ], @@ -335,13 +545,13 @@ "android" ], "engines": { - "node": ">=12" + "node": ">=18" } }, "node_modules/@esbuild/android-arm64": { - "version": "0.21.5", - "resolved": "https://registry.npmjs.org/@esbuild/android-arm64/-/android-arm64-0.21.5.tgz", - "integrity": "sha512-c0uX9VAUBQ7dTDCjq+wdyGLowMdtR/GoC2U5IYk/7D1H1JYC0qseD7+11iMP2mRLN9RcCMRcjC4YMclCzGwS/A==", + "version": "0.25.12", + "resolved": "https://registry.npmjs.org/@esbuild/android-arm64/-/android-arm64-0.25.12.tgz", + "integrity": "sha512-6AAmLG7zwD1Z159jCKPvAxZd4y/VTO0VkprYy+3N2FtJ8+BQWFXU+OxARIwA46c5tdD9SsKGZ/1ocqBS/gAKHg==", "cpu": [ "arm64" ], @@ -352,13 +562,13 @@ "android" ], "engines": { - "node": ">=12" + "node": ">=18" } }, "node_modules/@esbuild/android-x64": { - "version": "0.21.5", - "resolved": "https://registry.npmjs.org/@esbuild/android-x64/-/android-x64-0.21.5.tgz", - "integrity": "sha512-D7aPRUUNHRBwHxzxRvp856rjUHRFW1SdQATKXH2hqA0kAZb1hKmi02OpYRacl0TxIGz/ZmXWlbZgjwWYaCakTA==", + "version": "0.25.12", + "resolved": "https://registry.npmjs.org/@esbuild/android-x64/-/android-x64-0.25.12.tgz", + "integrity": "sha512-5jbb+2hhDHx5phYR2By8GTWEzn6I9UqR11Kwf22iKbNpYrsmRB18aX/9ivc5cabcUiAT/wM+YIZ6SG9QO6a8kg==", "cpu": [ "x64" ], @@ -369,13 +579,13 @@ "android" ], "engines": { - "node": ">=12" + "node": ">=18" } }, "node_modules/@esbuild/darwin-arm64": { - "version": "0.21.5", - "resolved": "https://registry.npmjs.org/@esbuild/darwin-arm64/-/darwin-arm64-0.21.5.tgz", - "integrity": "sha512-DwqXqZyuk5AiWWf3UfLiRDJ5EDd49zg6O9wclZ7kUMv2WRFr4HKjXp/5t8JZ11QbQfUS6/cRCKGwYhtNAY88kQ==", + "version": "0.25.12", + "resolved": "https://registry.npmjs.org/@esbuild/darwin-arm64/-/darwin-arm64-0.25.12.tgz", + "integrity": "sha512-N3zl+lxHCifgIlcMUP5016ESkeQjLj/959RxxNYIthIg+CQHInujFuXeWbWMgnTo4cp5XVHqFPmpyu9J65C1Yg==", "cpu": [ "arm64" ], @@ -386,13 +596,13 @@ "darwin" ], "engines": { - "node": ">=12" + "node": ">=18" } }, "node_modules/@esbuild/darwin-x64": { - "version": "0.21.5", - "resolved": "https://registry.npmjs.org/@esbuild/darwin-x64/-/darwin-x64-0.21.5.tgz", - "integrity": "sha512-se/JjF8NlmKVG4kNIuyWMV/22ZaerB+qaSi5MdrXtd6R08kvs2qCN4C09miupktDitvh8jRFflwGFBQcxZRjbw==", + "version": "0.25.12", + "resolved": "https://registry.npmjs.org/@esbuild/darwin-x64/-/darwin-x64-0.25.12.tgz", + "integrity": "sha512-HQ9ka4Kx21qHXwtlTUVbKJOAnmG1ipXhdWTmNXiPzPfWKpXqASVcWdnf2bnL73wgjNrFXAa3yYvBSd9pzfEIpA==", "cpu": [ "x64" ], @@ -403,13 +613,13 @@ "darwin" ], "engines": { - "node": ">=12" + "node": ">=18" } }, "node_modules/@esbuild/freebsd-arm64": { - "version": "0.21.5", - "resolved": "https://registry.npmjs.org/@esbuild/freebsd-arm64/-/freebsd-arm64-0.21.5.tgz", - "integrity": "sha512-5JcRxxRDUJLX8JXp/wcBCy3pENnCgBR9bN6JsY4OmhfUtIHe3ZW0mawA7+RDAcMLrMIZaf03NlQiX9DGyB8h4g==", + "version": "0.25.12", + "resolved": "https://registry.npmjs.org/@esbuild/freebsd-arm64/-/freebsd-arm64-0.25.12.tgz", + "integrity": "sha512-gA0Bx759+7Jve03K1S0vkOu5Lg/85dou3EseOGUes8flVOGxbhDDh/iZaoek11Y8mtyKPGF3vP8XhnkDEAmzeg==", "cpu": [ "arm64" ], @@ -420,13 +630,13 @@ "freebsd" ], "engines": { - "node": ">=12" + "node": ">=18" } }, "node_modules/@esbuild/freebsd-x64": { - "version": "0.21.5", - "resolved": "https://registry.npmjs.org/@esbuild/freebsd-x64/-/freebsd-x64-0.21.5.tgz", - "integrity": "sha512-J95kNBj1zkbMXtHVH29bBriQygMXqoVQOQYA+ISs0/2l3T9/kj42ow2mpqerRBxDJnmkUDCaQT/dfNXWX/ZZCQ==", + "version": "0.25.12", + "resolved": "https://registry.npmjs.org/@esbuild/freebsd-x64/-/freebsd-x64-0.25.12.tgz", + "integrity": "sha512-TGbO26Yw2xsHzxtbVFGEXBFH0FRAP7gtcPE7P5yP7wGy7cXK2oO7RyOhL5NLiqTlBh47XhmIUXuGciXEqYFfBQ==", "cpu": [ "x64" ], @@ -437,13 +647,13 @@ "freebsd" ], "engines": { - "node": ">=12" + "node": ">=18" } }, "node_modules/@esbuild/linux-arm": { - "version": "0.21.5", - "resolved": "https://registry.npmjs.org/@esbuild/linux-arm/-/linux-arm-0.21.5.tgz", - "integrity": "sha512-bPb5AHZtbeNGjCKVZ9UGqGwo8EUu4cLq68E95A53KlxAPRmUyYv2D6F0uUI65XisGOL1hBP5mTronbgo+0bFcA==", + "version": "0.25.12", + "resolved": "https://registry.npmjs.org/@esbuild/linux-arm/-/linux-arm-0.25.12.tgz", + "integrity": "sha512-lPDGyC1JPDou8kGcywY0YILzWlhhnRjdof3UlcoqYmS9El818LLfJJc3PXXgZHrHCAKs/Z2SeZtDJr5MrkxtOw==", "cpu": [ "arm" ], @@ -454,13 +664,13 @@ "linux" ], "engines": { - "node": ">=12" + "node": ">=18" } }, "node_modules/@esbuild/linux-arm64": { - "version": "0.21.5", - "resolved": "https://registry.npmjs.org/@esbuild/linux-arm64/-/linux-arm64-0.21.5.tgz", - "integrity": "sha512-ibKvmyYzKsBeX8d8I7MH/TMfWDXBF3db4qM6sy+7re0YXya+K1cem3on9XgdT2EQGMu4hQyZhan7TeQ8XkGp4Q==", + "version": "0.25.12", + "resolved": "https://registry.npmjs.org/@esbuild/linux-arm64/-/linux-arm64-0.25.12.tgz", + "integrity": "sha512-8bwX7a8FghIgrupcxb4aUmYDLp8pX06rGh5HqDT7bB+8Rdells6mHvrFHHW2JAOPZUbnjUpKTLg6ECyzvas2AQ==", "cpu": [ "arm64" ], @@ -471,13 +681,13 @@ "linux" ], "engines": { - "node": ">=12" + "node": ">=18" } }, "node_modules/@esbuild/linux-ia32": { - "version": "0.21.5", - "resolved": "https://registry.npmjs.org/@esbuild/linux-ia32/-/linux-ia32-0.21.5.tgz", - "integrity": "sha512-YvjXDqLRqPDl2dvRODYmmhz4rPeVKYvppfGYKSNGdyZkA01046pLWyRKKI3ax8fbJoK5QbxblURkwK/MWY18Tg==", + "version": "0.25.12", + "resolved": "https://registry.npmjs.org/@esbuild/linux-ia32/-/linux-ia32-0.25.12.tgz", + "integrity": "sha512-0y9KrdVnbMM2/vG8KfU0byhUN+EFCny9+8g202gYqSSVMonbsCfLjUO+rCci7pM0WBEtz+oK/PIwHkzxkyharA==", "cpu": [ "ia32" ], @@ -488,13 +698,13 @@ "linux" ], "engines": { - "node": ">=12" + "node": ">=18" } }, "node_modules/@esbuild/linux-loong64": { - "version": "0.21.5", - "resolved": "https://registry.npmjs.org/@esbuild/linux-loong64/-/linux-loong64-0.21.5.tgz", - "integrity": "sha512-uHf1BmMG8qEvzdrzAqg2SIG/02+4/DHB6a9Kbya0XDvwDEKCoC8ZRWI5JJvNdUjtciBGFQ5PuBlpEOXQj+JQSg==", + "version": "0.25.12", + "resolved": "https://registry.npmjs.org/@esbuild/linux-loong64/-/linux-loong64-0.25.12.tgz", + "integrity": "sha512-h///Lr5a9rib/v1GGqXVGzjL4TMvVTv+s1DPoxQdz7l/AYv6LDSxdIwzxkrPW438oUXiDtwM10o9PmwS/6Z0Ng==", "cpu": [ "loong64" ], @@ -505,13 +715,13 @@ "linux" ], "engines": { - "node": ">=12" + "node": ">=18" } }, "node_modules/@esbuild/linux-mips64el": { - "version": "0.21.5", - "resolved": "https://registry.npmjs.org/@esbuild/linux-mips64el/-/linux-mips64el-0.21.5.tgz", - "integrity": "sha512-IajOmO+KJK23bj52dFSNCMsz1QP1DqM6cwLUv3W1QwyxkyIWecfafnI555fvSGqEKwjMXVLokcV5ygHW5b3Jbg==", + "version": "0.25.12", + "resolved": "https://registry.npmjs.org/@esbuild/linux-mips64el/-/linux-mips64el-0.25.12.tgz", + "integrity": "sha512-iyRrM1Pzy9GFMDLsXn1iHUm18nhKnNMWscjmp4+hpafcZjrr2WbT//d20xaGljXDBYHqRcl8HnxbX6uaA/eGVw==", "cpu": [ "mips64el" ], @@ -522,13 +732,13 @@ "linux" ], "engines": { - "node": ">=12" + "node": ">=18" } }, "node_modules/@esbuild/linux-ppc64": { - "version": "0.21.5", - "resolved": "https://registry.npmjs.org/@esbuild/linux-ppc64/-/linux-ppc64-0.21.5.tgz", - "integrity": "sha512-1hHV/Z4OEfMwpLO8rp7CvlhBDnjsC3CttJXIhBi+5Aj5r+MBvy4egg7wCbe//hSsT+RvDAG7s81tAvpL2XAE4w==", + "version": "0.25.12", + "resolved": "https://registry.npmjs.org/@esbuild/linux-ppc64/-/linux-ppc64-0.25.12.tgz", + "integrity": "sha512-9meM/lRXxMi5PSUqEXRCtVjEZBGwB7P/D4yT8UG/mwIdze2aV4Vo6U5gD3+RsoHXKkHCfSxZKzmDssVlRj1QQA==", "cpu": [ "ppc64" ], @@ -539,13 +749,13 @@ "linux" ], "engines": { - "node": ">=12" + "node": ">=18" } }, "node_modules/@esbuild/linux-riscv64": { - "version": "0.21.5", - "resolved": "https://registry.npmjs.org/@esbuild/linux-riscv64/-/linux-riscv64-0.21.5.tgz", - "integrity": "sha512-2HdXDMd9GMgTGrPWnJzP2ALSokE/0O5HhTUvWIbD3YdjME8JwvSCnNGBnTThKGEB91OZhzrJ4qIIxk/SBmyDDA==", + "version": "0.25.12", + "resolved": "https://registry.npmjs.org/@esbuild/linux-riscv64/-/linux-riscv64-0.25.12.tgz", + "integrity": "sha512-Zr7KR4hgKUpWAwb1f3o5ygT04MzqVrGEGXGLnj15YQDJErYu/BGg+wmFlIDOdJp0PmB0lLvxFIOXZgFRrdjR0w==", "cpu": [ "riscv64" ], @@ -556,13 +766,13 @@ "linux" ], "engines": { - "node": ">=12" + "node": ">=18" } }, "node_modules/@esbuild/linux-s390x": { - "version": "0.21.5", - "resolved": "https://registry.npmjs.org/@esbuild/linux-s390x/-/linux-s390x-0.21.5.tgz", - "integrity": "sha512-zus5sxzqBJD3eXxwvjN1yQkRepANgxE9lgOW2qLnmr8ikMTphkjgXu1HR01K4FJg8h1kEEDAqDcZQtbrRnB41A==", + "version": "0.25.12", + "resolved": "https://registry.npmjs.org/@esbuild/linux-s390x/-/linux-s390x-0.25.12.tgz", + "integrity": "sha512-MsKncOcgTNvdtiISc/jZs/Zf8d0cl/t3gYWX8J9ubBnVOwlk65UIEEvgBORTiljloIWnBzLs4qhzPkJcitIzIg==", "cpu": [ "s390x" ], @@ -573,13 +783,13 @@ "linux" ], "engines": { - "node": ">=12" + "node": ">=18" } }, "node_modules/@esbuild/linux-x64": { - "version": "0.21.5", - "resolved": "https://registry.npmjs.org/@esbuild/linux-x64/-/linux-x64-0.21.5.tgz", - "integrity": "sha512-1rYdTpyv03iycF1+BhzrzQJCdOuAOtaqHTWJZCWvijKD2N5Xu0TtVC8/+1faWqcP9iBCWOmjmhoH94dH82BxPQ==", + "version": "0.25.12", + "resolved": "https://registry.npmjs.org/@esbuild/linux-x64/-/linux-x64-0.25.12.tgz", + "integrity": "sha512-uqZMTLr/zR/ed4jIGnwSLkaHmPjOjJvnm6TVVitAa08SLS9Z0VM8wIRx7gWbJB5/J54YuIMInDquWyYvQLZkgw==", "cpu": [ "x64" ], @@ -590,13 +800,30 @@ "linux" ], "engines": { - "node": ">=12" + "node": ">=18" + } + }, + "node_modules/@esbuild/netbsd-arm64": { + "version": "0.25.12", + "resolved": "https://registry.npmjs.org/@esbuild/netbsd-arm64/-/netbsd-arm64-0.25.12.tgz", + "integrity": "sha512-xXwcTq4GhRM7J9A8Gv5boanHhRa/Q9KLVmcyXHCTaM4wKfIpWkdXiMog/KsnxzJ0A1+nD+zoecuzqPmCRyBGjg==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "netbsd" + ], + "engines": { + "node": ">=18" } }, "node_modules/@esbuild/netbsd-x64": { - "version": "0.21.5", - "resolved": "https://registry.npmjs.org/@esbuild/netbsd-x64/-/netbsd-x64-0.21.5.tgz", - "integrity": "sha512-Woi2MXzXjMULccIwMnLciyZH4nCIMpWQAs049KEeMvOcNADVxo0UBIQPfSmxB3CWKedngg7sWZdLvLczpe0tLg==", + "version": "0.25.12", + "resolved": "https://registry.npmjs.org/@esbuild/netbsd-x64/-/netbsd-x64-0.25.12.tgz", + "integrity": "sha512-Ld5pTlzPy3YwGec4OuHh1aCVCRvOXdH8DgRjfDy/oumVovmuSzWfnSJg+VtakB9Cm0gxNO9BzWkj6mtO1FMXkQ==", "cpu": [ "x64" ], @@ -607,13 +834,30 @@ "netbsd" ], "engines": { - "node": ">=12" + "node": ">=18" + } + }, + "node_modules/@esbuild/openbsd-arm64": { + "version": "0.25.12", + "resolved": "https://registry.npmjs.org/@esbuild/openbsd-arm64/-/openbsd-arm64-0.25.12.tgz", + "integrity": "sha512-fF96T6KsBo/pkQI950FARU9apGNTSlZGsv1jZBAlcLL1MLjLNIWPBkj5NlSz8aAzYKg+eNqknrUJ24QBybeR5A==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "openbsd" + ], + "engines": { + "node": ">=18" } }, "node_modules/@esbuild/openbsd-x64": { - "version": "0.21.5", - "resolved": "https://registry.npmjs.org/@esbuild/openbsd-x64/-/openbsd-x64-0.21.5.tgz", - "integrity": "sha512-HLNNw99xsvx12lFBUwoT8EVCsSvRNDVxNpjZ7bPn947b8gJPzeHWyNVhFsaerc0n3TsbOINvRP2byTZ5LKezow==", + "version": "0.25.12", + "resolved": "https://registry.npmjs.org/@esbuild/openbsd-x64/-/openbsd-x64-0.25.12.tgz", + "integrity": "sha512-MZyXUkZHjQxUvzK7rN8DJ3SRmrVrke8ZyRusHlP+kuwqTcfWLyqMOE3sScPPyeIXN/mDJIfGXvcMqCgYKekoQw==", "cpu": [ "x64" ], @@ -624,13 +868,30 @@ "openbsd" ], "engines": { - "node": ">=12" + "node": ">=18" + } + }, + "node_modules/@esbuild/openharmony-arm64": { + "version": "0.25.12", + "resolved": "https://registry.npmjs.org/@esbuild/openharmony-arm64/-/openharmony-arm64-0.25.12.tgz", + "integrity": "sha512-rm0YWsqUSRrjncSXGA7Zv78Nbnw4XL6/dzr20cyrQf7ZmRcsovpcRBdhD43Nuk3y7XIoW2OxMVvwuRvk9XdASg==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "openharmony" + ], + "engines": { + "node": ">=18" } }, "node_modules/@esbuild/sunos-x64": { - "version": "0.21.5", - "resolved": "https://registry.npmjs.org/@esbuild/sunos-x64/-/sunos-x64-0.21.5.tgz", - "integrity": "sha512-6+gjmFpfy0BHU5Tpptkuh8+uw3mnrvgs+dSPQXQOv3ekbordwnzTVEb4qnIvQcYXq6gzkyTnoZ9dZG+D4garKg==", + "version": "0.25.12", + "resolved": "https://registry.npmjs.org/@esbuild/sunos-x64/-/sunos-x64-0.25.12.tgz", + "integrity": "sha512-3wGSCDyuTHQUzt0nV7bocDy72r2lI33QL3gkDNGkod22EsYl04sMf0qLb8luNKTOmgF/eDEDP5BFNwoBKH441w==", "cpu": [ "x64" ], @@ -641,13 +902,13 @@ "sunos" ], "engines": { - "node": ">=12" + "node": ">=18" } }, "node_modules/@esbuild/win32-arm64": { - "version": "0.21.5", - "resolved": "https://registry.npmjs.org/@esbuild/win32-arm64/-/win32-arm64-0.21.5.tgz", - "integrity": "sha512-Z0gOTd75VvXqyq7nsl93zwahcTROgqvuAcYDUr+vOv8uHhNSKROyU961kgtCD1e95IqPKSQKH7tBTslnS3tA8A==", + "version": "0.25.12", + "resolved": "https://registry.npmjs.org/@esbuild/win32-arm64/-/win32-arm64-0.25.12.tgz", + "integrity": "sha512-rMmLrur64A7+DKlnSuwqUdRKyd3UE7oPJZmnljqEptesKM8wx9J8gx5u0+9Pq0fQQW8vqeKebwNXdfOyP+8Bsg==", "cpu": [ "arm64" ], @@ -658,13 +919,13 @@ "win32" ], "engines": { - "node": ">=12" + "node": ">=18" } }, "node_modules/@esbuild/win32-ia32": { - "version": "0.21.5", - "resolved": "https://registry.npmjs.org/@esbuild/win32-ia32/-/win32-ia32-0.21.5.tgz", - "integrity": "sha512-SWXFF1CL2RVNMaVs+BBClwtfZSvDgtL//G/smwAc5oVK/UPu2Gu9tIaRgFmYFFKrmg3SyAjSrElf0TiJ1v8fYA==", + "version": "0.25.12", + "resolved": "https://registry.npmjs.org/@esbuild/win32-ia32/-/win32-ia32-0.25.12.tgz", + "integrity": "sha512-HkqnmmBoCbCwxUKKNPBixiWDGCpQGVsrQfJoVGYLPT41XWF8lHuE5N6WhVia2n4o5QK5M4tYr21827fNhi4byQ==", "cpu": [ "ia32" ], @@ -675,13 +936,13 @@ "win32" ], "engines": { - "node": ">=12" + "node": ">=18" } }, "node_modules/@esbuild/win32-x64": { - "version": "0.21.5", - "resolved": "https://registry.npmjs.org/@esbuild/win32-x64/-/win32-x64-0.21.5.tgz", - "integrity": "sha512-tQd/1efJuzPC6rCFwEvLtci/xNFcTZknmXs98FYDfGE4wP9ClFV98nyKrzJKVPMhdDnjzLhdUyMX4PsQAPjwIw==", + "version": "0.25.12", + "resolved": "https://registry.npmjs.org/@esbuild/win32-x64/-/win32-x64-0.25.12.tgz", + "integrity": "sha512-alJC0uCZpTFrSL0CCDjcgleBXPnCrEAhTBILpeAp7M/OFgoqtAetfBzX0xM00MUsVVPpVjlPuMbREqnZCXaTnA==", "cpu": [ "x64" ], @@ -692,7 +953,25 @@ "win32" ], "engines": { - "node": ">=12" + "node": ">=18" + } + }, + "node_modules/@exodus/bytes": { + "version": "1.15.1", + "resolved": "https://registry.npmjs.org/@exodus/bytes/-/bytes-1.15.1.tgz", + "integrity": "sha512-S6mL0yNB/Abt9Ei4tq8gDhcczc4S3+vQ4ra7vxnAf+YHC02srtqxKKZghx2Dq6p0e66THKwR6r8N6P95wEty7Q==", + "dev": true, + "license": "MIT", + "engines": { + "node": "^20.19.0 || ^22.12.0 || >=24.0.0" + }, + "peerDependencies": { + "@noble/hashes": "^1.8.0 || ^2.0.0" + }, + "peerDependenciesMeta": { + "@noble/hashes": { + "optional": true + } } }, "node_modules/@fontsource-variable/inter": { @@ -787,6 +1066,22 @@ "react-dom": ">= 16.8" } }, + "node_modules/@playwright/test": { + "version": "1.62.1", + "resolved": "https://registry.npmjs.org/@playwright/test/-/test-1.62.1.tgz", + "integrity": "sha512-DTcUc8qii+cpHvtOwggMtBRMjKZHXYWdw8syRYu2vtzuq4Wxphqq4NfCs5Zt44L6mA8rfDfj+PHnxFc/FeK6mQ==", + "dev": true, + "license": "Apache-2.0", + "dependencies": { + "playwright": "1.62.1" + }, + "bin": { + "playwright": "cli.js" + }, + "engines": { + "node": ">=20" + } + }, "node_modules/@rolldown/pluginutils": { "version": "1.0.0-beta.27", "resolved": "https://registry.npmjs.org/@rolldown/pluginutils/-/pluginutils-1.0.0-beta.27.tgz", @@ -1228,6 +1523,24 @@ "@babel/types": "^7.28.2" } }, + "node_modules/@types/chai": { + "version": "5.2.3", + "resolved": "https://registry.npmjs.org/@types/chai/-/chai-5.2.3.tgz", + "integrity": "sha512-Mw558oeA9fFbv65/y4mHtXDs9bPnFMZAL/jxdPFUpOHHIXX91mcgEHbS5Lahr+pwZFR8A7GQleRWeI6cGFC2UA==", + "dev": true, + "license": "MIT", + "dependencies": { + "@types/deep-eql": "*", + "assertion-error": "^2.0.1" + } + }, + "node_modules/@types/deep-eql": { + "version": "4.0.2", + "resolved": "https://registry.npmjs.org/@types/deep-eql/-/deep-eql-4.0.2.tgz", + "integrity": "sha512-c9h9dVVMigMPc4bwTvC5dxqtqJZwQPePsWjPlpSOnojbor6pGqdk541lfA7AqFQr5pB1BRdq0juY9db81BwyFw==", + "dev": true, + "license": "MIT" + }, "node_modules/@types/estree": { "version": "1.0.9", "resolved": "https://registry.npmjs.org/@types/estree/-/estree-1.0.9.tgz", @@ -1284,32 +1597,193 @@ "vite": "^4.2.0 || ^5.0.0 || ^6.0.0 || ^7.0.0" } }, - "node_modules/baseline-browser-mapping": { - "version": "2.11.14", - "resolved": "https://registry.npmjs.org/baseline-browser-mapping/-/baseline-browser-mapping-2.11.14.tgz", - "integrity": "sha512-JyJ954WzuIR8/FFzX0o5krdSTrBAkcCSRfWSleRsIHSWV+cZe2FI1PKggVkFke1hBldRs+LRxUczzE9iPmgZww==", + "node_modules/@vitest/expect": { + "version": "3.2.6", + "resolved": "https://registry.npmjs.org/@vitest/expect/-/expect-3.2.6.tgz", + "integrity": "sha512-1+7q9BtaKzEmO+fmNT3kYvoNn5Y71XWAx2Q5HRim4tTVRQVRv4uJFAQ5FbK0OPUeNP/WmVCpxYxoJdvuHVjzBQ==", "dev": true, - "license": "Apache-2.0", - "bin": { - "baseline-browser-mapping": "dist/cli.cjs" + "license": "MIT", + "dependencies": { + "@types/chai": "^5.2.2", + "@vitest/spy": "3.2.6", + "@vitest/utils": "3.2.6", + "chai": "^5.2.0", + "tinyrainbow": "^2.0.0" }, - "engines": { - "node": ">=6.0.0" + "funding": { + "url": "https://opencollective.com/vitest" } }, - "node_modules/browserslist": { - "version": "4.28.8", - "resolved": "https://registry.npmjs.org/browserslist/-/browserslist-4.28.8.tgz", - "integrity": "sha512-V2NpofLblG64mfOtSgDhOJESZEGogzDMBv/q+W6oc4LXWP/q75eOXoOaaOu1EOadB9U4Bwx/e0yzbvwKH8zalA==", + "node_modules/@vitest/mocker": { + "version": "3.2.6", + "resolved": "https://registry.npmjs.org/@vitest/mocker/-/mocker-3.2.6.tgz", + "integrity": "sha512-EZOrpDbkKotFAP7wPAQV1UIyoGOk4oX7ynWhBhLB7v+meMHbQhU16oPpIYGTTe4oFlhpryGpgpcZP/sin3hYuw==", "dev": true, - "funding": [ - { - "type": "opencollective", - "url": "https://opencollective.com/browserslist" - }, - { - "type": "tidelift", - "url": "https://tidelift.com/funding/github/npm/browserslist" + "license": "MIT", + "dependencies": { + "@vitest/spy": "3.2.6", + "estree-walker": "^3.0.3", + "magic-string": "^0.30.17" + }, + "funding": { + "url": "https://opencollective.com/vitest" + }, + "peerDependencies": { + "msw": "^2.4.9", + "vite": "^5.0.0 || ^6.0.0 || ^7.0.0-0" + }, + "peerDependenciesMeta": { + "msw": { + "optional": true + }, + "vite": { + "optional": true + } + } + }, + "node_modules/@vitest/pretty-format": { + "version": "3.2.7", + "resolved": "https://registry.npmjs.org/@vitest/pretty-format/-/pretty-format-3.2.7.tgz", + "integrity": "sha512-KUHlwqVu0sRlhCdyPdQ/wBoTfRahjUky1MubOmYw9fWfIZy1gNoHpuaaQBPAaMaVYdQYHJLurzj8ECCj5OwTqA==", + "dev": true, + "license": "MIT", + "dependencies": { + "tinyrainbow": "^2.0.0" + }, + "funding": { + "url": "https://opencollective.com/vitest" + } + }, + "node_modules/@vitest/runner": { + "version": "3.2.6", + "resolved": "https://registry.npmjs.org/@vitest/runner/-/runner-3.2.6.tgz", + "integrity": "sha512-HYcoSj1w5tcgUnzoF0HcyaAQjpA1gj9ftUJ7iSJSuipc02jW9gKkigwZbjFldAfYHA1fa8UZVRftdMY5msWM9Q==", + "dev": true, + "license": "MIT", + "dependencies": { + "@vitest/utils": "3.2.6", + "pathe": "^2.0.3", + "strip-literal": "^3.0.0" + }, + "funding": { + "url": "https://opencollective.com/vitest" + } + }, + "node_modules/@vitest/snapshot": { + "version": "3.2.6", + "resolved": "https://registry.npmjs.org/@vitest/snapshot/-/snapshot-3.2.6.tgz", + "integrity": "sha512-H+ZjNTWGpObenh0YnlBctAPnJSI20P81PL8BPzWpx54YXLLTm8hEsWawtcYLMrwvpK48hGxLLbCS+1KRXhsKhw==", + "dev": true, + "license": "MIT", + "dependencies": { + "@vitest/pretty-format": "3.2.6", + "magic-string": "^0.30.17", + "pathe": "^2.0.3" + }, + "funding": { + "url": "https://opencollective.com/vitest" + } + }, + "node_modules/@vitest/snapshot/node_modules/@vitest/pretty-format": { + "version": "3.2.6", + "resolved": "https://registry.npmjs.org/@vitest/pretty-format/-/pretty-format-3.2.6.tgz", + "integrity": "sha512-lb7XXXzmm2h2ASzFnRvQpDo6onT1NmMJA3tkGTWiBFtRJ9lxGY3d3mm/Apt36gej2bkkOVLL/yTOtufDaFa/jA==", + "dev": true, + "license": "MIT", + "dependencies": { + "tinyrainbow": "^2.0.0" + }, + "funding": { + "url": "https://opencollective.com/vitest" + } + }, + "node_modules/@vitest/spy": { + "version": "3.2.6", + "resolved": "https://registry.npmjs.org/@vitest/spy/-/spy-3.2.6.tgz", + "integrity": "sha512-oq6BbH68WzcWmwtBrU9nqLeaXTR4XwJF7FSLkKEZo4i6eoXcrxjcwSuTvWBIRUTC6VC72nXYunzqgZA+IKdtxg==", + "dev": true, + "license": "MIT", + "dependencies": { + "tinyspy": "^4.0.3" + }, + "funding": { + "url": "https://opencollective.com/vitest" + } + }, + "node_modules/@vitest/utils": { + "version": "3.2.6", + "resolved": "https://registry.npmjs.org/@vitest/utils/-/utils-3.2.6.tgz", + "integrity": "sha512-lI23nIs4bnT3T8NIoh+vFaz5s2/DdP0Jgt2jxwgWljvwn82cLJtyi/If+fjFyoLMGIOz0U/fKvWE0d4jsNQEfg==", + "dev": true, + "license": "MIT", + "dependencies": { + "@vitest/pretty-format": "3.2.6", + "loupe": "^3.1.4", + "tinyrainbow": "^2.0.0" + }, + "funding": { + "url": "https://opencollective.com/vitest" + } + }, + "node_modules/@vitest/utils/node_modules/@vitest/pretty-format": { + "version": "3.2.6", + "resolved": "https://registry.npmjs.org/@vitest/pretty-format/-/pretty-format-3.2.6.tgz", + "integrity": "sha512-lb7XXXzmm2h2ASzFnRvQpDo6onT1NmMJA3tkGTWiBFtRJ9lxGY3d3mm/Apt36gej2bkkOVLL/yTOtufDaFa/jA==", + "dev": true, + "license": "MIT", + "dependencies": { + "tinyrainbow": "^2.0.0" + }, + "funding": { + "url": "https://opencollective.com/vitest" + } + }, + "node_modules/assertion-error": { + "version": "2.0.1", + "resolved": "https://registry.npmjs.org/assertion-error/-/assertion-error-2.0.1.tgz", + "integrity": "sha512-Izi8RQcffqCeNVgFigKli1ssklIbpHnCYc6AknXGYoB6grJqyeby7jv12JUQgmTAnIDnbck1uxksT4dzN3PWBA==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=12" + } + }, + "node_modules/baseline-browser-mapping": { + "version": "2.11.14", + "resolved": "https://registry.npmjs.org/baseline-browser-mapping/-/baseline-browser-mapping-2.11.14.tgz", + "integrity": "sha512-JyJ954WzuIR8/FFzX0o5krdSTrBAkcCSRfWSleRsIHSWV+cZe2FI1PKggVkFke1hBldRs+LRxUczzE9iPmgZww==", + "dev": true, + "license": "Apache-2.0", + "bin": { + "baseline-browser-mapping": "dist/cli.cjs" + }, + "engines": { + "node": ">=6.0.0" + } + }, + "node_modules/bidi-js": { + "version": "1.0.3", + "resolved": "https://registry.npmjs.org/bidi-js/-/bidi-js-1.0.3.tgz", + "integrity": "sha512-RKshQI1R3YQ+n9YJz2QQ147P66ELpa1FQEg20Dk8oW9t2KgLbpDLLp9aGZ7y8WHSshDknG0bknqGw5/tyCs5tw==", + "dev": true, + "license": "MIT", + "dependencies": { + "require-from-string": "^2.0.2" + } + }, + "node_modules/browserslist": { + "version": "4.28.8", + "resolved": "https://registry.npmjs.org/browserslist/-/browserslist-4.28.8.tgz", + "integrity": "sha512-V2NpofLblG64mfOtSgDhOJESZEGogzDMBv/q+W6oc4LXWP/q75eOXoOaaOu1EOadB9U4Bwx/e0yzbvwKH8zalA==", + "dev": true, + "funding": [ + { + "type": "opencollective", + "url": "https://opencollective.com/browserslist" + }, + { + "type": "tidelift", + "url": "https://tidelift.com/funding/github/npm/browserslist" }, { "type": "github", @@ -1331,6 +1805,16 @@ "node": "^6 || ^7 || ^8 || ^9 || ^10 || ^11 || ^12 || >=13.7" } }, + "node_modules/cac": { + "version": "6.7.14", + "resolved": "https://registry.npmjs.org/cac/-/cac-6.7.14.tgz", + "integrity": "sha512-b6Ilus+c3RrdDk+JhLKUAQfzzgLEPy6wcXqS7f/xe1EETvsDP6GORG7SFuOs6cID5YkqchW/LXZbX5bc8j7ZcQ==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=8" + } + }, "node_modules/caniuse-lite": { "version": "1.0.30001809", "resolved": "https://registry.npmjs.org/caniuse-lite/-/caniuse-lite-1.0.30001809.tgz", @@ -1352,6 +1836,33 @@ ], "license": "CC-BY-4.0" }, + "node_modules/chai": { + "version": "5.3.3", + "resolved": "https://registry.npmjs.org/chai/-/chai-5.3.3.tgz", + "integrity": "sha512-4zNhdJD/iOjSH0A05ea+Ke6MU5mmpQcbQsSOkgdaUMJ9zTlDTD/GYlwohmIE2u0gaxHYiVHEn1Fw9mZ/ktJWgw==", + "dev": true, + "license": "MIT", + "dependencies": { + "assertion-error": "^2.0.1", + "check-error": "^2.1.1", + "deep-eql": "^5.0.1", + "loupe": "^3.1.0", + "pathval": "^2.0.0" + }, + "engines": { + "node": ">=18" + } + }, + "node_modules/check-error": { + "version": "2.1.3", + "resolved": "https://registry.npmjs.org/check-error/-/check-error-2.1.3.tgz", + "integrity": "sha512-PAJdDJusoxnwm1VwW07VWwUN1sl7smmC3OKggvndJFadxxDRyFJBX/ggnu/KE4kQAB7a3Dp8f/YXC1FlUprWmA==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">= 16" + } + }, "node_modules/convert-source-map": { "version": "2.0.0", "resolved": "https://registry.npmjs.org/convert-source-map/-/convert-source-map-2.0.0.tgz", @@ -1359,6 +1870,20 @@ "dev": true, "license": "MIT" }, + "node_modules/css-tree": { + "version": "3.2.1", + "resolved": "https://registry.npmjs.org/css-tree/-/css-tree-3.2.1.tgz", + "integrity": "sha512-X7sjQzceUhu1u7Y/ylrRZFU2FS6LRiFVp6rKLPg23y3x3c3DOKAwuXGDp+PAGjh6CSnCjYeAul8pcT8bAl+lSA==", + "dev": true, + "license": "MIT", + "dependencies": { + "mdn-data": "2.27.1", + "source-map-js": "^1.2.1" + }, + "engines": { + "node": "^10 || ^12.20.0 || ^14.13.0 || >=15.0.0" + } + }, "node_modules/csstype": { "version": "3.2.3", "resolved": "https://registry.npmjs.org/csstype/-/csstype-3.2.3.tgz", @@ -1366,6 +1891,35 @@ "devOptional": true, "license": "MIT" }, + "node_modules/data-urls": { + "version": "7.0.0", + "resolved": "https://registry.npmjs.org/data-urls/-/data-urls-7.0.0.tgz", + "integrity": "sha512-23XHcCF+coGYevirZceTVD7NdJOqVn+49IHyxgszm+JIiHLoB2TkmPtsYkNWT1pvRSGkc35L6NHs0yHkN2SumA==", + "dev": true, + "license": "MIT", + "dependencies": { + "whatwg-mimetype": "^5.0.0", + "whatwg-url": "^16.0.0" + }, + "engines": { + "node": "^20.19.0 || ^22.12.0 || >=24.0.0" + } + }, + "node_modules/data-urls/node_modules/whatwg-url": { + "version": "16.0.1", + "resolved": "https://registry.npmjs.org/whatwg-url/-/whatwg-url-16.0.1.tgz", + "integrity": "sha512-1to4zXBxmXHV3IiSSEInrreIlu02vUOvrhxJJH5vcxYTBDAx51cqZiKdyTxlecdKNSjj8EcxGBxNf6Vg+945gw==", + "dev": true, + "license": "MIT", + "dependencies": { + "@exodus/bytes": "^1.11.0", + "tr46": "^6.0.0", + "webidl-conversions": "^8.0.1" + }, + "engines": { + "node": "^20.19.0 || ^22.12.0 || >=24.0.0" + } + }, "node_modules/debug": { "version": "4.4.3", "resolved": "https://registry.npmjs.org/debug/-/debug-4.4.3.tgz", @@ -1384,6 +1938,23 @@ } } }, + "node_modules/decimal.js": { + "version": "10.6.0", + "resolved": "https://registry.npmjs.org/decimal.js/-/decimal.js-10.6.0.tgz", + "integrity": "sha512-YpgQiITW3JXGntzdUmyUR1V812Hn8T1YVXhCu+wO3OpS4eU9l4YdD3qjyiKdV6mvV29zapkMeD390UVEf2lkUg==", + "dev": true, + "license": "MIT" + }, + "node_modules/deep-eql": { + "version": "5.0.2", + "resolved": "https://registry.npmjs.org/deep-eql/-/deep-eql-5.0.2.tgz", + "integrity": "sha512-h5k/5U50IJJFpzfL6nO9jaaumfjO/f2NjK/oYB2Djzm4p9L+3T9qWpZqZ2hAbLPuuYq9wrU08WQyBTL5GbPk5Q==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=6" + } + }, "node_modules/electron-to-chromium": { "version": "1.5.408", "resolved": "https://registry.npmjs.org/electron-to-chromium/-/electron-to-chromium-1.5.408.tgz", @@ -1391,10 +1962,30 @@ "dev": true, "license": "ISC" }, + "node_modules/entities": { + "version": "8.0.0", + "resolved": "https://registry.npmjs.org/entities/-/entities-8.0.0.tgz", + "integrity": "sha512-zwfzJecQ/Uej6tusMqwAqU/6KL2XaB2VZ2Jg54Je6ahNBGNH6Ek6g3jjNCF0fG9EWQKGZNddNjU5F1ZQn/sBnA==", + "dev": true, + "license": "BSD-2-Clause", + "engines": { + "node": ">=20.19.0" + }, + "funding": { + "url": "https://github.com/fb55/entities?sponsor=1" + } + }, + "node_modules/es-module-lexer": { + "version": "1.7.0", + "resolved": "https://registry.npmjs.org/es-module-lexer/-/es-module-lexer-1.7.0.tgz", + "integrity": "sha512-jEQoCwk8hyb2AZziIOLhDqpm5+2ww5uIE6lkO/6jcOCusfk6LhMHpXXfBLXTZ7Ydyt0j4VoUQv6uGNYbdW+kBA==", + "dev": true, + "license": "MIT" + }, "node_modules/esbuild": { - "version": "0.21.5", - "resolved": "https://registry.npmjs.org/esbuild/-/esbuild-0.21.5.tgz", - "integrity": "sha512-mg3OPMV4hXywwpoDxu3Qda5xCKQi+vCTZq8S9J/EpkhB2HzKXq4SNFZE3+NK93JYxc8VMSep+lOUSC/RVKaBqw==", + "version": "0.25.12", + "resolved": "https://registry.npmjs.org/esbuild/-/esbuild-0.25.12.tgz", + "integrity": "sha512-bbPBYYrtZbkt6Os6FiTLCTFxvq4tt3JKall1vRwshA3fdVztsLAatFaZobhkBC8/BrPetoa0oksYoKXoG4ryJg==", "dev": true, "hasInstallScript": true, "license": "MIT", @@ -1402,32 +1993,35 @@ "esbuild": "bin/esbuild" }, "engines": { - "node": ">=12" + "node": ">=18" }, "optionalDependencies": { - "@esbuild/aix-ppc64": "0.21.5", - "@esbuild/android-arm": "0.21.5", - "@esbuild/android-arm64": "0.21.5", - "@esbuild/android-x64": "0.21.5", - "@esbuild/darwin-arm64": "0.21.5", - "@esbuild/darwin-x64": "0.21.5", - "@esbuild/freebsd-arm64": "0.21.5", - "@esbuild/freebsd-x64": "0.21.5", - "@esbuild/linux-arm": "0.21.5", - "@esbuild/linux-arm64": "0.21.5", - "@esbuild/linux-ia32": "0.21.5", - "@esbuild/linux-loong64": "0.21.5", - "@esbuild/linux-mips64el": "0.21.5", - "@esbuild/linux-ppc64": "0.21.5", - "@esbuild/linux-riscv64": "0.21.5", - "@esbuild/linux-s390x": "0.21.5", - "@esbuild/linux-x64": "0.21.5", - "@esbuild/netbsd-x64": "0.21.5", - "@esbuild/openbsd-x64": "0.21.5", - "@esbuild/sunos-x64": "0.21.5", - "@esbuild/win32-arm64": "0.21.5", - "@esbuild/win32-ia32": "0.21.5", - "@esbuild/win32-x64": "0.21.5" + "@esbuild/aix-ppc64": "0.25.12", + "@esbuild/android-arm": "0.25.12", + "@esbuild/android-arm64": "0.25.12", + "@esbuild/android-x64": "0.25.12", + "@esbuild/darwin-arm64": "0.25.12", + "@esbuild/darwin-x64": "0.25.12", + "@esbuild/freebsd-arm64": "0.25.12", + "@esbuild/freebsd-x64": "0.25.12", + "@esbuild/linux-arm": "0.25.12", + "@esbuild/linux-arm64": "0.25.12", + "@esbuild/linux-ia32": "0.25.12", + "@esbuild/linux-loong64": "0.25.12", + "@esbuild/linux-mips64el": "0.25.12", + "@esbuild/linux-ppc64": "0.25.12", + "@esbuild/linux-riscv64": "0.25.12", + "@esbuild/linux-s390x": "0.25.12", + "@esbuild/linux-x64": "0.25.12", + "@esbuild/netbsd-arm64": "0.25.12", + "@esbuild/netbsd-x64": "0.25.12", + "@esbuild/openbsd-arm64": "0.25.12", + "@esbuild/openbsd-x64": "0.25.12", + "@esbuild/openharmony-arm64": "0.25.12", + "@esbuild/sunos-x64": "0.25.12", + "@esbuild/win32-arm64": "0.25.12", + "@esbuild/win32-ia32": "0.25.12", + "@esbuild/win32-x64": "0.25.12" } }, "node_modules/escalade": { @@ -1440,6 +2034,67 @@ "node": ">=6" } }, + "node_modules/estree-walker": { + "version": "3.0.3", + "resolved": "https://registry.npmjs.org/estree-walker/-/estree-walker-3.0.3.tgz", + "integrity": "sha512-7RUKfXgSMMkzt6ZuXmqapOurLGPPfgj6l9uRZ7lRGolvk0y2yocc35LdcxKC5PQZdn2DMqioAQ2NoWcrTKmm6g==", + "dev": true, + "license": "MIT", + "dependencies": { + "@types/estree": "^1.0.0" + } + }, + "node_modules/expect-type": { + "version": "1.4.0", + "resolved": "https://registry.npmjs.org/expect-type/-/expect-type-1.4.0.tgz", + "integrity": "sha512-KfYbmpRm0VbLjEvVa9yGwCi9GI34xvi7A/HXYWQO65CSD2u3MczUJSuwXKFIxlGsgBQizV9q5J9NHj4VG0n+pA==", + "dev": true, + "license": "Apache-2.0", + "engines": { + "node": ">=12.0.0" + } + }, + "node_modules/fdir": { + "version": "6.5.0", + "resolved": "https://registry.npmjs.org/fdir/-/fdir-6.5.0.tgz", + "integrity": "sha512-tIbYtZbucOs0BRGqPJkshJUYdL+SDH7dVM8gjy+ERp3WAUjLEFJE+02kanyHtwjWOnwrKYBiwAmM0p4kLJAnXg==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=12.0.0" + }, + "peerDependencies": { + "picomatch": "^3 || ^4" + }, + "peerDependenciesMeta": { + "picomatch": { + "optional": true + } + } + }, + "node_modules/framer-motion": { + "version": "13.1.1", + "resolved": "https://registry.npmjs.org/framer-motion/-/framer-motion-13.1.1.tgz", + "integrity": "sha512-B/xn2TPS4f61cEBLFjiYlQFnBZUW1YVj/LM+C+N4OP8Rs95VLEI2ot/RlfBg111la/EiyECFaJJi/A3FWA8MUA==", + "license": "MIT", + "dependencies": { + "motion-dom": "^13.1.1", + "motion-utils": "^13.0.0", + "tslib": "^2.4.0" + }, + "peerDependencies": { + "react": "^18.0.0 || ^19.0.0", + "react-dom": "^18.0.0 || ^19.0.0" + }, + "peerDependenciesMeta": { + "react": { + "optional": true + }, + "react-dom": { + "optional": true + } + } + }, "node_modules/fsevents": { "version": "2.3.3", "resolved": "https://registry.npmjs.org/fsevents/-/fsevents-2.3.3.tgz", @@ -1465,12 +2120,83 @@ "node": ">=6.9.0" } }, + "node_modules/html-encoding-sniffer": { + "version": "6.0.0", + "resolved": "https://registry.npmjs.org/html-encoding-sniffer/-/html-encoding-sniffer-6.0.0.tgz", + "integrity": "sha512-CV9TW3Y3f8/wT0BRFc1/KAVQ3TUHiXmaAb6VW9vtiMFf7SLoMd1PdAc4W3KFOFETBJUb90KatHqlsZMWV+R9Gg==", + "dev": true, + "license": "MIT", + "dependencies": { + "@exodus/bytes": "^1.6.0" + }, + "engines": { + "node": "^20.19.0 || ^22.12.0 || >=24.0.0" + } + }, + "node_modules/is-potential-custom-element-name": { + "version": "1.0.1", + "resolved": "https://registry.npmjs.org/is-potential-custom-element-name/-/is-potential-custom-element-name-1.0.1.tgz", + "integrity": "sha512-bCYeRA2rVibKZd+s2625gGnGF/t7DSqDs4dP7CrLA1m7jKWz6pps0LpYLJN8Q64HtmPKJ1hrN3nzPNKFEKOUiQ==", + "dev": true, + "license": "MIT" + }, "node_modules/js-tokens": { "version": "4.0.0", "resolved": "https://registry.npmjs.org/js-tokens/-/js-tokens-4.0.0.tgz", "integrity": "sha512-RdJUflcE3cUzKiMqQgsCu06FPu9UdIJO0beYbPhHN4k6apgJtifcoCtT9bcxOpYBtpD2kCM6Sbzg4CausW/PKQ==", "license": "MIT" }, + "node_modules/jsdom": { + "version": "30.0.1", + "resolved": "https://registry.npmjs.org/jsdom/-/jsdom-30.0.1.tgz", + "integrity": "sha512-52v7mUVUfNQVYYqE1lcdaymWL0njO7lTLUog6ZvW2U5KsbiLk/GnZlVJ+qx0xfNJZ6Gn+KSpPNE52vurbxZwrA==", + "dev": true, + "license": "MIT", + "dependencies": { + "@asamuzakjp/css-color": "^6.0.5", + "@asamuzakjp/dom-selector": "^8.3.0", + "@bramus/specificity": "^2.4.2", + "@csstools/css-syntax-patches-for-csstree": "^1.1.7", + "@exodus/bytes": "^1.15.1", + "css-tree": "^3.2.1", + "data-urls": "^7.0.0", + "decimal.js": "^10.6.0", + "html-encoding-sniffer": "^6.0.0", + "is-potential-custom-element-name": "^1.0.1", + "lru-cache": "^11.5.2", + "parse5": "^8.0.1", + "saxes": "^6.0.0", + "symbol-tree": "^3.2.4", + "tough-cookie": "^6.0.2", + "undici": "^8.9.0", + "w3c-xmlserializer": "^5.0.0", + "webidl-conversions": "^8.0.1", + "whatwg-mimetype": "^5.0.0", + "whatwg-url": "^17.1.0", + "xml-name-validator": "^5.0.0" + }, + "engines": { + "node": "^22.22.2 || ^24.15.0 || >=26.0.0" + }, + "peerDependencies": { + "canvas": "^3.2.3" + }, + "peerDependenciesMeta": { + "canvas": { + "optional": true + } + } + }, + "node_modules/jsdom/node_modules/lru-cache": { + "version": "11.5.2", + "resolved": "https://registry.npmjs.org/lru-cache/-/lru-cache-11.5.2.tgz", + "integrity": "sha512-4pfM1Ff0x50o0tQwb5ucw/RzNyD0/YJME6IVcStalZuMWxdt3sR3huStTtxz4PUmvZfRguvDejasvQ2kifR11g==", + "dev": true, + "license": "BlueOak-1.0.0", + "engines": { + "node": "20 || >=22" + } + }, "node_modules/jsesc": { "version": "3.1.0", "resolved": "https://registry.npmjs.org/jsesc/-/jsesc-3.1.0.tgz", @@ -1509,6 +2235,13 @@ "loose-envify": "cli.js" } }, + "node_modules/loupe": { + "version": "3.2.1", + "resolved": "https://registry.npmjs.org/loupe/-/loupe-3.2.1.tgz", + "integrity": "sha512-CdzqowRJCeLU72bHvWqwRBBlLcMEtIvGrlvef74kMnV2AolS9Y8xUv1I0U/MNAWMhBlKIoyuEgoJ0t/bbwHbLQ==", + "dev": true, + "license": "MIT" + }, "node_modules/lru-cache": { "version": "5.1.1", "resolved": "https://registry.npmjs.org/lru-cache/-/lru-cache-5.1.1.tgz", @@ -1519,6 +2252,60 @@ "yallist": "^3.0.2" } }, + "node_modules/magic-string": { + "version": "0.30.21", + "resolved": "https://registry.npmjs.org/magic-string/-/magic-string-0.30.21.tgz", + "integrity": "sha512-vd2F4YUyEXKGcLHoq+TEyCjxueSeHnFxyyjNp80yg0XV4vUhnDer/lvvlqM/arB5bXQN5K2/3oinyCRyx8T2CQ==", + "dev": true, + "license": "MIT", + "dependencies": { + "@jridgewell/sourcemap-codec": "^1.5.5" + } + }, + "node_modules/mdn-data": { + "version": "2.27.1", + "resolved": "https://registry.npmjs.org/mdn-data/-/mdn-data-2.27.1.tgz", + "integrity": "sha512-9Yubnt3e8A0OKwxYSXyhLymGW4sCufcLG6VdiDdUGVkPhpqLxlvP5vl1983gQjJl3tqbrM731mjaZaP68AgosQ==", + "dev": true, + "license": "CC0-1.0" + }, + "node_modules/motion": { + "version": "13.1.1", + "resolved": "https://registry.npmjs.org/motion/-/motion-13.1.1.tgz", + "integrity": "sha512-WNZoK6xiF+kkTqkZ5K7FDDh6A8BG4i5Hc7KXtW8gtTxkpJFds+hIOrDaQGKjQj/AE/i4hJqAaUHEqp/Qo02y6Q==", + "license": "MIT", + "dependencies": { + "framer-motion": "^13.1.1", + "tslib": "^2.4.0" + }, + "peerDependencies": { + "react": "^18.0.0 || ^19.0.0", + "react-dom": "^18.0.0 || ^19.0.0" + }, + "peerDependenciesMeta": { + "react": { + "optional": true + }, + "react-dom": { + "optional": true + } + } + }, + "node_modules/motion-dom": { + "version": "13.1.1", + "resolved": "https://registry.npmjs.org/motion-dom/-/motion-dom-13.1.1.tgz", + "integrity": "sha512-XSf8VYWSB6G/0IY3rWVbyLcxWXtAVHkN1PQE2agTaCv3u8RGvbwu56TyyR/MNzBqqNavEBTZzErcxI1TxBrjcA==", + "license": "MIT", + "dependencies": { + "motion-utils": "^13.0.0" + } + }, + "node_modules/motion-utils": { + "version": "13.0.0", + "resolved": "https://registry.npmjs.org/motion-utils/-/motion-utils-13.0.0.tgz", + "integrity": "sha512-7DnN7TmbLcYXcG4RVadXIihWlyuM9afoUww8Y5Agg431kGKiuL2/OMyP4mJ5wLz+pvN3t5ySClLOaVXJ+wekRQ==", + "license": "MIT" + }, "node_modules/ms": { "version": "2.1.3", "resolved": "https://registry.npmjs.org/ms/-/ms-2.1.3.tgz", @@ -1555,6 +2342,36 @@ "node": ">=18" } }, + "node_modules/parse5": { + "version": "8.0.1", + "resolved": "https://registry.npmjs.org/parse5/-/parse5-8.0.1.tgz", + "integrity": "sha512-z1e/HMG90obSGeidlli3hj7cbocou0/wa5HacvI3ASx34PecNjNQeaHNo5WIZpWofN9kgkqV1q5YvXe3F0FoPw==", + "dev": true, + "license": "MIT", + "dependencies": { + "entities": "^8.0.0" + }, + "funding": { + "url": "https://github.com/inikulin/parse5?sponsor=1" + } + }, + "node_modules/pathe": { + "version": "2.0.3", + "resolved": "https://registry.npmjs.org/pathe/-/pathe-2.0.3.tgz", + "integrity": "sha512-WUjGcAqP1gQacoQe+OBJsFA7Ld4DyXuUIjZ5cc75cLHvJ7dtNsTugphxIADwspS+AraAUePCKrSVtPLFj/F88w==", + "dev": true, + "license": "MIT" + }, + "node_modules/pathval": { + "version": "2.0.1", + "resolved": "https://registry.npmjs.org/pathval/-/pathval-2.0.1.tgz", + "integrity": "sha512-//nshmD55c46FuFw26xV/xFAaB5HF9Xdap7HJBBnrKdAd6/GxDBaNA1870O79+9ueg61cZLSVc+OaFlfmObYVQ==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">= 14.16" + } + }, "node_modules/picocolors": { "version": "1.1.1", "resolved": "https://registry.npmjs.org/picocolors/-/picocolors-1.1.1.tgz", @@ -1562,6 +2379,66 @@ "dev": true, "license": "ISC" }, + "node_modules/picomatch": { + "version": "4.0.7", + "resolved": "https://registry.npmjs.org/picomatch/-/picomatch-4.0.7.tgz", + "integrity": "sha512-qcJu88Q2IWqJsDD529JKMdwGm/dvInW4HvQnRwiH9JtihJvzGOscDtHE3x1pBKeUOTysQ8kVmLnJ2kJu7yhcGA==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=12" + }, + "funding": { + "url": "https://github.com/sponsors/jonschlinkert" + } + }, + "node_modules/playwright": { + "version": "1.62.1", + "resolved": "https://registry.npmjs.org/playwright/-/playwright-1.62.1.tgz", + "integrity": "sha512-0M+L3LAD8/nm554LOla9Ayx0j0tmFZ0FBcoQ7F1VuVHpM/XpiC8RcDzBQB8W5+hA8L22THxELzeF+2WcUzvcLg==", + "dev": true, + "license": "Apache-2.0", + "dependencies": { + "playwright-core": "1.62.1" + }, + "bin": { + "playwright": "cli.js" + }, + "engines": { + "node": ">=20" + }, + "optionalDependencies": { + "fsevents": "2.3.2" + } + }, + "node_modules/playwright-core": { + "version": "1.62.1", + "resolved": "https://registry.npmjs.org/playwright-core/-/playwright-core-1.62.1.tgz", + "integrity": "sha512-wPYSwEBJY9GHraISXqyqtx0na0LpO3XEX7jNDhntbex7tzUS7kLnZsOlFruFJB4Hi/rhDMjXGqHewDZ68nYZVw==", + "dev": true, + "license": "Apache-2.0", + "bin": { + "playwright-core": "cli.js" + }, + "engines": { + "node": ">=20" + } + }, + "node_modules/playwright/node_modules/fsevents": { + "version": "2.3.2", + "resolved": "https://registry.npmjs.org/fsevents/-/fsevents-2.3.2.tgz", + "integrity": "sha512-xiqMQR4xAeHTuB9uWm+fFRcIOgKBMiOBP+eXiyT7jsgVCq1bkVygt00oASowB7EdtpOHaaPgKt812P9ab+DDKA==", + "dev": true, + "hasInstallScript": true, + "license": "MIT", + "optional": true, + "os": [ + "darwin" + ], + "engines": { + "node": "^8.16.0 || ^10.6.0 || >=11.0.0" + } + }, "node_modules/postcss": { "version": "8.5.26", "resolved": "https://registry.npmjs.org/postcss/-/postcss-8.5.26.tgz", @@ -1591,6 +2468,16 @@ "node": "^10 || ^12 || >=14" } }, + "node_modules/punycode": { + "version": "2.3.1", + "resolved": "https://registry.npmjs.org/punycode/-/punycode-2.3.1.tgz", + "integrity": "sha512-vYt7UD1U9Wg6138shLtLOvdAu+8DsC/ilFtEVHcH+wydcSpNE20AfSOduf6MkRFahL5FY7X1oU7nKVZFtfq8Fg==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=6" + } + }, "node_modules/react": { "version": "18.3.1", "resolved": "https://registry.npmjs.org/react/-/react-18.3.1.tgz", @@ -1626,6 +2513,16 @@ "node": ">=0.10.0" } }, + "node_modules/require-from-string": { + "version": "2.0.2", + "resolved": "https://registry.npmjs.org/require-from-string/-/require-from-string-2.0.2.tgz", + "integrity": "sha512-Xf0nWe6RseziFMu+Ap9biiUbmplq6S9/p+7w7YXP/JBHhrUDDUhwa+vANyubuqfZWTveU//DYVGsDG7RKL/vEw==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=0.10.0" + } + }, "node_modules/rollup": { "version": "4.62.4", "resolved": "https://registry.npmjs.org/rollup/-/rollup-4.62.4.tgz", @@ -1672,6 +2569,19 @@ "fsevents": "~2.3.2" } }, + "node_modules/saxes": { + "version": "6.0.0", + "resolved": "https://registry.npmjs.org/saxes/-/saxes-6.0.0.tgz", + "integrity": "sha512-xAg7SOnEhrm5zI3puOOKyy1OMcMlIJZYNJY7xLBwSze0UjhPLnWfj2GF2EpT0jmzaJKIWKHLsaSSajf35bcYnA==", + "dev": true, + "license": "ISC", + "dependencies": { + "xmlchars": "^2.2.0" + }, + "engines": { + "node": ">=v12.22.7" + } + }, "node_modules/scheduler": { "version": "0.23.2", "resolved": "https://registry.npmjs.org/scheduler/-/scheduler-0.23.2.tgz", @@ -1691,6 +2601,13 @@ "semver": "bin/semver.js" } }, + "node_modules/siginfo": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/siginfo/-/siginfo-2.0.0.tgz", + "integrity": "sha512-ybx0WO1/8bSBLEWXZvEd7gMW3Sn3JFlW3TvX1nREbDLRNQNaeNN8WK0meBwPdAaOI7TtRRRJn/Es1zhrrCHu7g==", + "dev": true, + "license": "ISC" + }, "node_modules/source-map-js": { "version": "1.2.1", "resolved": "https://registry.npmjs.org/source-map-js/-/source-map-js-1.2.1.tgz", @@ -1701,6 +2618,160 @@ "node": ">=0.10.0" } }, + "node_modules/stackback": { + "version": "0.0.2", + "resolved": "https://registry.npmjs.org/stackback/-/stackback-0.0.2.tgz", + "integrity": "sha512-1XMJE5fQo1jGH6Y/7ebnwPOBEkIEnT4QF32d5R1+VXdXveM0IBMJt8zfaxX1P3QhVwrYe+576+jkANtSS2mBbw==", + "dev": true, + "license": "MIT" + }, + "node_modules/std-env": { + "version": "3.10.0", + "resolved": "https://registry.npmjs.org/std-env/-/std-env-3.10.0.tgz", + "integrity": "sha512-5GS12FdOZNliM5mAOxFRg7Ir0pWz8MdpYm6AY6VPkGpbA7ZzmbzNcBJQ0GPvvyWgcY7QAhCgf9Uy89I03faLkg==", + "dev": true, + "license": "MIT" + }, + "node_modules/strip-literal": { + "version": "3.1.0", + "resolved": "https://registry.npmjs.org/strip-literal/-/strip-literal-3.1.0.tgz", + "integrity": "sha512-8r3mkIM/2+PpjHoOtiAW8Rg3jJLHaV7xPwG+YRGrv6FP0wwk/toTpATxWYOW0BKdWwl82VT2tFYi5DlROa0Mxg==", + "dev": true, + "license": "MIT", + "dependencies": { + "js-tokens": "^9.0.1" + }, + "funding": { + "url": "https://github.com/sponsors/antfu" + } + }, + "node_modules/strip-literal/node_modules/js-tokens": { + "version": "9.0.1", + "resolved": "https://registry.npmjs.org/js-tokens/-/js-tokens-9.0.1.tgz", + "integrity": "sha512-mxa9E9ITFOt0ban3j6L5MpjwegGz6lBQmM1IJkWeBZGcMxto50+eWdjC/52xDbS2vy0k7vIMK0Fe2wfL9OQSpQ==", + "dev": true, + "license": "MIT" + }, + "node_modules/symbol-tree": { + "version": "3.2.4", + "resolved": "https://registry.npmjs.org/symbol-tree/-/symbol-tree-3.2.4.tgz", + "integrity": "sha512-9QNk5KwDF+Bvz+PyObkmSYjI5ksVUYtjW7AU22r2NKcfLJcXp96hkDWU3+XndOsUb+AQ9QhfzfCT2O+CNWT5Tw==", + "dev": true, + "license": "MIT" + }, + "node_modules/tinybench": { + "version": "2.9.0", + "resolved": "https://registry.npmjs.org/tinybench/-/tinybench-2.9.0.tgz", + "integrity": "sha512-0+DUvqWMValLmha6lr4kD8iAMK1HzV0/aKnCtWb9v9641TnP/MFb7Pc2bxoxQjTXAErryXVgUOfv2YqNllqGeg==", + "dev": true, + "license": "MIT" + }, + "node_modules/tinyexec": { + "version": "0.3.2", + "resolved": "https://registry.npmjs.org/tinyexec/-/tinyexec-0.3.2.tgz", + "integrity": "sha512-KQQR9yN7R5+OSwaK0XQoj22pwHoTlgYqmUscPYoknOoWCWfj/5/ABTMRi69FrKU5ffPVh5QcFikpWJI/P1ocHA==", + "dev": true, + "license": "MIT" + }, + "node_modules/tinyglobby": { + "version": "0.2.17", + "resolved": "https://registry.npmjs.org/tinyglobby/-/tinyglobby-0.2.17.tgz", + "integrity": "sha512-wXR/dYpcqKmfWpEdZjiKJOwCNFndD0DMnrW/cYjVGttEkBfVgcLFHoNrlj47mjOVic9yyNu65alsgF4NQyTa2g==", + "dev": true, + "license": "MIT", + "dependencies": { + "fdir": "^6.5.0", + "picomatch": "^4.0.4" + }, + "engines": { + "node": ">=12.0.0" + }, + "funding": { + "url": "https://github.com/sponsors/SuperchupuDev" + } + }, + "node_modules/tinypool": { + "version": "1.1.1", + "resolved": "https://registry.npmjs.org/tinypool/-/tinypool-1.1.1.tgz", + "integrity": "sha512-Zba82s87IFq9A9XmjiX5uZA/ARWDrB03OHlq+Vw1fSdt0I+4/Kutwy8BP4Y/y/aORMo61FQ0vIb5j44vSo5Pkg==", + "dev": true, + "license": "MIT", + "engines": { + "node": "^18.0.0 || >=20.0.0" + } + }, + "node_modules/tinyrainbow": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/tinyrainbow/-/tinyrainbow-2.0.0.tgz", + "integrity": "sha512-op4nsTR47R6p0vMUUoYl/a+ljLFVtlfaXkLQmqfLR1qHma1h/ysYk4hEXZ880bf2CYgTskvTa/e196Vd5dDQXw==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=14.0.0" + } + }, + "node_modules/tinyspy": { + "version": "4.0.4", + "resolved": "https://registry.npmjs.org/tinyspy/-/tinyspy-4.0.4.tgz", + "integrity": "sha512-azl+t0z7pw/z958Gy9svOTuzqIk6xq+NSheJzn5MMWtWTFywIacg2wUlzKFGtt3cthx0r2SxMK0yzJOR0IES7Q==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=14.0.0" + } + }, + "node_modules/tldts": { + "version": "7.4.11", + "resolved": "https://registry.npmjs.org/tldts/-/tldts-7.4.11.tgz", + "integrity": "sha512-aBiNayCfTQxuIJBm06M+xR14cYaYlDlSXZbgsnKzKNxDKUVq7KFwTjwBSsb7m9Y5xO8WfPnBc63WaYFMTGlvqw==", + "dev": true, + "license": "MIT", + "dependencies": { + "tldts-core": "^7.4.11" + }, + "bin": { + "tldts": "bin/cli.js" + } + }, + "node_modules/tldts-core": { + "version": "7.4.11", + "resolved": "https://registry.npmjs.org/tldts-core/-/tldts-core-7.4.11.tgz", + "integrity": "sha512-CW3WN2rIIE/Of21mulhgnGOwoDyEFNygyIBOONSdyAuSATgMMUCpLeUlB+E8sAwA5xRV9hYPl+kyZ9citHCaKg==", + "dev": true, + "license": "MIT" + }, + "node_modules/tough-cookie": { + "version": "6.0.2", + "resolved": "https://registry.npmjs.org/tough-cookie/-/tough-cookie-6.0.2.tgz", + "integrity": "sha512-exgYmnmL/sJpR3upZfXG5PoatXQii55xAiXGXzY+sROLZ/Y+SLcp9PgJNI9Vz37HpQ74WvDcLT8eqm+kV3FzrA==", + "dev": true, + "license": "BSD-3-Clause", + "dependencies": { + "tldts": "^7.0.5" + }, + "engines": { + "node": ">=16" + } + }, + "node_modules/tr46": { + "version": "6.0.0", + "resolved": "https://registry.npmjs.org/tr46/-/tr46-6.0.0.tgz", + "integrity": "sha512-bLVMLPtstlZ4iMQHpFHTR7GAGj2jxi8Dg0s2h2MafAE4uSWF98FC/3MomU51iQAMf8/qDUbKWf5GxuvvVcXEhw==", + "dev": true, + "license": "MIT", + "dependencies": { + "punycode": "^2.3.1" + }, + "engines": { + "node": ">=20" + } + }, + "node_modules/tslib": { + "version": "2.8.1", + "resolved": "https://registry.npmjs.org/tslib/-/tslib-2.8.1.tgz", + "integrity": "sha512-oJFu94HQb+KVduSUQL7wnpmqnfmLsOA/nAh6b6EH0wCEoK0/mPeXU6c3wKDV83MkOuHPRHtSXKKU99IBazS/2w==", + "license": "0BSD" + }, "node_modules/typescript": { "version": "5.9.3", "resolved": "https://registry.npmjs.org/typescript/-/typescript-5.9.3.tgz", @@ -1715,6 +2786,16 @@ "node": ">=14.17" } }, + "node_modules/undici": { + "version": "8.10.0", + "resolved": "https://registry.npmjs.org/undici/-/undici-8.10.0.tgz", + "integrity": "sha512-HvltHd7avK13QIw/oLe4qoOLyoVSoafqJ2jYOrtMRBkbYT31eiBQ8O0ehRKZiEZCMEyLFQNIADpgCWC5fALvYQ==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=22.19.0" + } + }, "node_modules/update-browserslist-db": { "version": "1.3.1", "resolved": "https://registry.npmjs.org/update-browserslist-db/-/update-browserslist-db-1.3.1.tgz", @@ -1747,21 +2828,24 @@ } }, "node_modules/vite": { - "version": "5.4.21", - "resolved": "https://registry.npmjs.org/vite/-/vite-5.4.21.tgz", - "integrity": "sha512-o5a9xKjbtuhY6Bi5S3+HvbRERmouabWbyUcpXXUA1u+GNUKoROi9byOJ8M0nHbHYHkYICiMlqxkg1KkYmm25Sw==", + "version": "6.4.3", + "resolved": "https://registry.npmjs.org/vite/-/vite-6.4.3.tgz", + "integrity": "sha512-NTKlcQjlAK7MlQoyb6LgaqHc8sso/pVyUJYWMws3jg21uTJw/LddqIFPcPqP6PzpgbIcZyKI85sFE4HBrQDA8A==", "dev": true, "license": "MIT", "dependencies": { - "esbuild": "^0.21.3", - "postcss": "^8.4.43", - "rollup": "^4.20.0" + "esbuild": "^0.25.0", + "fdir": "^6.4.4", + "picomatch": "^4.0.2", + "postcss": "^8.5.3", + "rollup": "^4.34.9", + "tinyglobby": "^0.2.13" }, "bin": { "vite": "bin/vite.js" }, "engines": { - "node": "^18.0.0 || >=20.0.0" + "node": "^18.0.0 || ^20.0.0 || >=22.0.0" }, "funding": { "url": "https://github.com/vitejs/vite?sponsor=1" @@ -1770,19 +2854,25 @@ "fsevents": "~2.3.3" }, "peerDependencies": { - "@types/node": "^18.0.0 || >=20.0.0", + "@types/node": "^18.0.0 || ^20.0.0 || >=22.0.0", + "jiti": ">=1.21.0", "less": "*", "lightningcss": "^1.21.0", "sass": "*", "sass-embedded": "*", "stylus": "*", "sugarss": "*", - "terser": "^5.4.0" + "terser": "^5.16.0", + "tsx": "^4.8.1", + "yaml": "^2.4.2" }, "peerDependenciesMeta": { "@types/node": { "optional": true }, + "jiti": { + "optional": true + }, "less": { "optional": true }, @@ -1803,9 +2893,193 @@ }, "terser": { "optional": true + }, + "tsx": { + "optional": true + }, + "yaml": { + "optional": true + } + } + }, + "node_modules/vite-node": { + "version": "3.2.4", + "resolved": "https://registry.npmjs.org/vite-node/-/vite-node-3.2.4.tgz", + "integrity": "sha512-EbKSKh+bh1E1IFxeO0pg1n4dvoOTt0UDiXMd/qn++r98+jPO1xtJilvXldeuQ8giIB5IkpjCgMleHMNEsGH6pg==", + "dev": true, + "license": "MIT", + "dependencies": { + "cac": "^6.7.14", + "debug": "^4.4.1", + "es-module-lexer": "^1.7.0", + "pathe": "^2.0.3", + "vite": "^5.0.0 || ^6.0.0 || ^7.0.0-0" + }, + "bin": { + "vite-node": "vite-node.mjs" + }, + "engines": { + "node": "^18.0.0 || ^20.0.0 || >=22.0.0" + }, + "funding": { + "url": "https://opencollective.com/vitest" + } + }, + "node_modules/vitest": { + "version": "3.2.6", + "resolved": "https://registry.npmjs.org/vitest/-/vitest-3.2.6.tgz", + "integrity": "sha512-xejya+bT/j/+R/AGa1XOfRxLmNUlLtlwjRsFUILF+xHfzElmGcmFydy2gqqIrd62ptIEfwVMofd19uNWD9L7Nw==", + "dev": true, + "license": "MIT", + "dependencies": { + "@types/chai": "^5.2.2", + "@vitest/expect": "3.2.6", + "@vitest/mocker": "3.2.6", + "@vitest/pretty-format": "^3.2.6", + "@vitest/runner": "3.2.6", + "@vitest/snapshot": "3.2.6", + "@vitest/spy": "3.2.6", + "@vitest/utils": "3.2.6", + "chai": "^5.2.0", + "debug": "^4.4.1", + "expect-type": "^1.2.1", + "magic-string": "^0.30.17", + "pathe": "^2.0.3", + "picomatch": "^4.0.2", + "std-env": "^3.9.0", + "tinybench": "^2.9.0", + "tinyexec": "^0.3.2", + "tinyglobby": "^0.2.14", + "tinypool": "^1.1.1", + "tinyrainbow": "^2.0.0", + "vite": "^5.0.0 || ^6.0.0 || ^7.0.0-0", + "vite-node": "3.2.4", + "why-is-node-running": "^2.3.0" + }, + "bin": { + "vitest": "vitest.mjs" + }, + "engines": { + "node": "^18.0.0 || ^20.0.0 || >=22.0.0" + }, + "funding": { + "url": "https://opencollective.com/vitest" + }, + "peerDependencies": { + "@edge-runtime/vm": "*", + "@types/debug": "^4.1.12", + "@types/node": "^18.0.0 || ^20.0.0 || >=22.0.0", + "@vitest/browser": "3.2.6", + "@vitest/ui": "3.2.6", + "happy-dom": "*", + "jsdom": "*" + }, + "peerDependenciesMeta": { + "@edge-runtime/vm": { + "optional": true + }, + "@types/debug": { + "optional": true + }, + "@types/node": { + "optional": true + }, + "@vitest/browser": { + "optional": true + }, + "@vitest/ui": { + "optional": true + }, + "happy-dom": { + "optional": true + }, + "jsdom": { + "optional": true } } }, + "node_modules/w3c-xmlserializer": { + "version": "5.0.0", + "resolved": "https://registry.npmjs.org/w3c-xmlserializer/-/w3c-xmlserializer-5.0.0.tgz", + "integrity": "sha512-o8qghlI8NZHU1lLPrpi2+Uq7abh4GGPpYANlalzWxyWteJOCsr/P+oPBA49TOLu5FTZO4d3F9MnWJfiMo4BkmA==", + "dev": true, + "license": "MIT", + "dependencies": { + "xml-name-validator": "^5.0.0" + }, + "engines": { + "node": ">=18" + } + }, + "node_modules/webidl-conversions": { + "version": "8.0.1", + "resolved": "https://registry.npmjs.org/webidl-conversions/-/webidl-conversions-8.0.1.tgz", + "integrity": "sha512-BMhLD/Sw+GbJC21C/UgyaZX41nPt8bUTg+jWyDeg7e7YN4xOM05YPSIXceACnXVtqyEw/LMClUQMtMZ+PGGpqQ==", + "dev": true, + "license": "BSD-2-Clause", + "engines": { + "node": ">=20" + } + }, + "node_modules/whatwg-mimetype": { + "version": "5.0.0", + "resolved": "https://registry.npmjs.org/whatwg-mimetype/-/whatwg-mimetype-5.0.0.tgz", + "integrity": "sha512-sXcNcHOC51uPGF0P/D4NVtrkjSU2fNsm9iog4ZvZJsL3rjoDAzXZhkm2MWt1y+PUdggKAYVoMAIYcs78wJ51Cw==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=20" + } + }, + "node_modules/whatwg-url": { + "version": "17.1.0", + "resolved": "https://registry.npmjs.org/whatwg-url/-/whatwg-url-17.1.0.tgz", + "integrity": "sha512-3GeworPmc2ZfEEHP7lEbUfBX/L75wdEsi0rLNhXcXxnoN5jyq0SL5gCy06SGW2cyTIZdTvWIDQNQoza++vKeaw==", + "dev": true, + "license": "MIT", + "dependencies": { + "@exodus/bytes": "^1.15.1", + "tr46": "^6.0.0", + "webidl-conversions": "^8.0.1" + }, + "engines": { + "node": "^22.14.0 || >=24.0.0" + } + }, + "node_modules/why-is-node-running": { + "version": "2.3.0", + "resolved": "https://registry.npmjs.org/why-is-node-running/-/why-is-node-running-2.3.0.tgz", + "integrity": "sha512-hUrmaWBdVDcxvYqnyh09zunKzROWjbZTiNy8dBEjkS7ehEDQibXJ7XvlmtbwuTclUiIyN+CyXQD4Vmko8fNm8w==", + "dev": true, + "license": "MIT", + "dependencies": { + "siginfo": "^2.0.0", + "stackback": "0.0.2" + }, + "bin": { + "why-is-node-running": "cli.js" + }, + "engines": { + "node": ">=8" + } + }, + "node_modules/xml-name-validator": { + "version": "5.0.0", + "resolved": "https://registry.npmjs.org/xml-name-validator/-/xml-name-validator-5.0.0.tgz", + "integrity": "sha512-EvGK8EJ3DhaHfbRlETOWAS5pO9MZITeauHKJyb8wyajUfQUenkIg2MvLDTZ4T/TgIcm3HU0TFBgWWboAZ30UHg==", + "dev": true, + "license": "Apache-2.0", + "engines": { + "node": ">=18" + } + }, + "node_modules/xmlchars": { + "version": "2.2.0", + "resolved": "https://registry.npmjs.org/xmlchars/-/xmlchars-2.2.0.tgz", + "integrity": "sha512-JZnDKK8B0RCDw84FNdDAIpZK+JuJw+s7Lz8nksI7SIuU3UXJJslUthsi+uWBUYOwPFwW7W7PRLRfUKpxjtjFCw==", + "dev": true, + "license": "MIT" + }, "node_modules/yallist": { "version": "3.1.1", "resolved": "https://registry.npmjs.org/yallist/-/yallist-3.1.1.tgz", diff --git a/frontend/package.json b/frontend/package.json index b36c657..de3c833 100644 --- a/frontend/package.json +++ b/frontend/package.json @@ -6,20 +6,27 @@ "scripts": { "dev": "vite", "build": "tsc -b && vite build", - "preview": "vite preview" + "preview": "vite preview", + "test": "vitest run", + "test:watch": "vitest", + "test:e2e": "npm run build && playwright test" }, "dependencies": { "@fontsource-variable/inter": "^5.3.0", "@phosphor-icons/react": "^2.1.10", + "motion": "^13.1.1", "react": "^18.3.1", "react-dom": "^18.3.1", "yet-another-react-lightbox": "3.32.2" }, "devDependencies": { + "@playwright/test": "^1.62.1", "@types/react": "^18.3.3", "@types/react-dom": "^18.3.0", "@vitejs/plugin-react": "^4.3.1", + "jsdom": "^30.0.1", "typescript": "^5.5.3", - "vite": "^5.4.0" + "vite": "^6.4.3", + "vitest": "^3.2.6" } } diff --git a/frontend/playwright.config.ts b/frontend/playwright.config.ts new file mode 100644 index 0000000..9e2c0c9 --- /dev/null +++ b/frontend/playwright.config.ts @@ -0,0 +1,21 @@ +import { defineConfig } from "@playwright/test"; + +export default defineConfig({ + testDir: "./e2e", + fullyParallel: false, + workers: 1, + retries: 0, + reporter: "line", + use: { + baseURL: "http://127.0.0.1:4173", + channel: "chrome", + headless: true, + trace: "retain-on-failure", + }, + webServer: { + command: "npm run preview -- --host 127.0.0.1 --port 4173", + url: "http://127.0.0.1:4173", + reuseExistingServer: false, + timeout: 30_000, + }, +}); diff --git a/frontend/public/assets/image-theme-previews/architectural_blueprint.webp b/frontend/public/assets/image-theme-previews/architectural_blueprint.webp new file mode 100644 index 0000000..fc308f7 Binary files /dev/null and b/frontend/public/assets/image-theme-previews/architectural_blueprint.webp differ diff --git a/frontend/public/assets/image-theme-previews/art_deco_night.webp b/frontend/public/assets/image-theme-previews/art_deco_night.webp new file mode 100644 index 0000000..640875f Binary files /dev/null and b/frontend/public/assets/image-theme-previews/art_deco_night.webp differ diff --git a/frontend/public/assets/image-theme-previews/cel_animation.webp b/frontend/public/assets/image-theme-previews/cel_animation.webp new file mode 100644 index 0000000..5345d95 Binary files /dev/null and b/frontend/public/assets/image-theme-previews/cel_animation.webp differ diff --git a/frontend/public/assets/image-theme-previews/chibi_sticker.webp b/frontend/public/assets/image-theme-previews/chibi_sticker.webp new file mode 100644 index 0000000..1845bd9 Binary files /dev/null and b/frontend/public/assets/image-theme-previews/chibi_sticker.webp differ diff --git a/frontend/public/assets/image-theme-previews/children_science_picturebook.webp b/frontend/public/assets/image-theme-previews/children_science_picturebook.webp new file mode 100644 index 0000000..a79694e Binary files /dev/null and b/frontend/public/assets/image-theme-previews/children_science_picturebook.webp differ diff --git a/frontend/public/assets/image-theme-previews/clay_stopmotion.webp b/frontend/public/assets/image-theme-previews/clay_stopmotion.webp new file mode 100644 index 0000000..3c8ccd3 Binary files /dev/null and b/frontend/public/assets/image-theme-previews/clay_stopmotion.webp differ diff --git a/frontend/public/assets/image-theme-previews/glassmorphism_tech.webp b/frontend/public/assets/image-theme-previews/glassmorphism_tech.webp new file mode 100644 index 0000000..0b79b69 Binary files /dev/null and b/frontend/public/assets/image-theme-previews/glassmorphism_tech.webp differ diff --git a/frontend/public/assets/image-theme-previews/gouache_editorial.webp b/frontend/public/assets/image-theme-previews/gouache_editorial.webp new file mode 100644 index 0000000..a40e10b Binary files /dev/null and b/frontend/public/assets/image-theme-previews/gouache_editorial.webp differ diff --git a/frontend/public/assets/image-theme-previews/ink_wash_editorial.webp b/frontend/public/assets/image-theme-previews/ink_wash_editorial.webp new file mode 100644 index 0000000..8e28474 Binary files /dev/null and b/frontend/public/assets/image-theme-previews/ink_wash_editorial.webp differ diff --git a/frontend/public/assets/image-theme-previews/isometric_miniature.webp b/frontend/public/assets/image-theme-previews/isometric_miniature.webp new file mode 100644 index 0000000..fecab7c Binary files /dev/null and b/frontend/public/assets/image-theme-previews/isometric_miniature.webp differ diff --git a/frontend/public/assets/image-theme-previews/mineral_pigment.webp b/frontend/public/assets/image-theme-previews/mineral_pigment.webp new file mode 100644 index 0000000..c83ac07 Binary files /dev/null and b/frontend/public/assets/image-theme-previews/mineral_pigment.webp differ diff --git a/frontend/public/assets/image-theme-previews/minimal_vector.webp b/frontend/public/assets/image-theme-previews/minimal_vector.webp new file mode 100644 index 0000000..1504125 Binary files /dev/null and b/frontend/public/assets/image-theme-previews/minimal_vector.webp differ diff --git a/frontend/public/assets/image-theme-previews/natural_history_engraving.webp b/frontend/public/assets/image-theme-previews/natural_history_engraving.webp new file mode 100644 index 0000000..de52679 Binary files /dev/null and b/frontend/public/assets/image-theme-previews/natural_history_engraving.webp differ diff --git a/frontend/public/assets/image-theme-previews/paper_cut_layered.webp b/frontend/public/assets/image-theme-previews/paper_cut_layered.webp new file mode 100644 index 0000000..b50e5a3 Binary files /dev/null and b/frontend/public/assets/image-theme-previews/paper_cut_layered.webp differ diff --git a/frontend/public/assets/image-theme-previews/pencil_storyboard.webp b/frontend/public/assets/image-theme-previews/pencil_storyboard.webp new file mode 100644 index 0000000..5ab6796 Binary files /dev/null and b/frontend/public/assets/image-theme-previews/pencil_storyboard.webp differ diff --git a/frontend/public/assets/image-theme-previews/pixel_arcade.webp b/frontend/public/assets/image-theme-previews/pixel_arcade.webp new file mode 100644 index 0000000..e1e3a9d Binary files /dev/null and b/frontend/public/assets/image-theme-previews/pixel_arcade.webp differ diff --git a/frontend/public/assets/image-theme-previews/retro_futurism.webp b/frontend/public/assets/image-theme-previews/retro_futurism.webp new file mode 100644 index 0000000..dc914ce Binary files /dev/null and b/frontend/public/assets/image-theme-previews/retro_futurism.webp differ diff --git a/frontend/public/assets/image-theme-previews/silkscreen_editorial.webp b/frontend/public/assets/image-theme-previews/silkscreen_editorial.webp new file mode 100644 index 0000000..71780e4 Binary files /dev/null and b/frontend/public/assets/image-theme-previews/silkscreen_editorial.webp differ diff --git a/frontend/public/assets/image-theme-previews/stained_glass.webp b/frontend/public/assets/image-theme-previews/stained_glass.webp new file mode 100644 index 0000000..94af782 Binary files /dev/null and b/frontend/public/assets/image-theme-previews/stained_glass.webp differ diff --git a/frontend/public/assets/image-theme-previews/textile_embroidery.webp b/frontend/public/assets/image-theme-previews/textile_embroidery.webp new file mode 100644 index 0000000..f79ccfe Binary files /dev/null and b/frontend/public/assets/image-theme-previews/textile_embroidery.webp differ diff --git a/frontend/public/assets/image-theme-previews/watercolor_journal.webp b/frontend/public/assets/image-theme-previews/watercolor_journal.webp new file mode 100644 index 0000000..4b1a03f Binary files /dev/null and b/frontend/public/assets/image-theme-previews/watercolor_journal.webp differ diff --git a/frontend/public/assets/image-theme-previews/woodcut_editorial.webp b/frontend/public/assets/image-theme-previews/woodcut_editorial.webp new file mode 100644 index 0000000..805102a Binary files /dev/null and b/frontend/public/assets/image-theme-previews/woodcut_editorial.webp differ diff --git a/frontend/src/App.tsx b/frontend/src/App.tsx index e5e989e..d64370e 100644 --- a/frontend/src/App.tsx +++ b/frontend/src/App.tsx @@ -1,14 +1,19 @@ -import Dashboard from "./pages/v2/Dashboard"; -import Groups from "./pages/v2/Groups"; -import GroupDetail from "./pages/v2/GroupDetail"; -import Ranking from "./pages/v2/Ranking"; -import AIImages from "./pages/v2/AIImages"; -import ChatRecords from "./pages/v2/ChatRecords"; -import Tasks from "./pages/v2/Tasks"; -import Archive from "./pages/v2/Archive"; -import Settings from "./pages/v2/Settings"; +import { lazy, Suspense } from "react"; + import AppShell from "./components/layout/AppShell"; +import { LoadingState } from "./components/common"; import { usePageNavigation } from "./navigation"; +import { m, MotionProvider, PageTransition } from "./components/motion"; + +const Dashboard = lazy(() => import("./pages/v2/Dashboard")); +const Groups = lazy(() => import("./pages/v2/Groups")); +const GroupDetail = lazy(() => import("./pages/v2/GroupDetail")); +const Ranking = lazy(() => import("./pages/v2/Ranking")); +const AIImages = lazy(() => import("./pages/v2/AIImages")); +const ChatRecords = lazy(() => import("./pages/v2/ChatRecords")); +const Tasks = lazy(() => import("./pages/v2/Tasks")); +const Archive = lazy(() => import("./pages/v2/Archive")); +const Settings = lazy(() => import("./pages/v2/Settings")); function WorkspaceTabs({ tabs, @@ -30,7 +35,8 @@ function WorkspaceTabs({ className={active === tab.key ? "is-active" : ""} onClick={() => onNavigate(tab.key)} > - {tab.label} + {active === tab.key &&