Skip to content

feat(sentiment): harden news layer — market-flow filter, evidence floor, event-override off by default#20

Open
Zeeechenn wants to merge 1 commit into
mainfrom
feat/news-sentiment-pack
Open

feat(sentiment): harden news layer — market-flow filter, evidence floor, event-override off by default#20
Zeeechenn wants to merge 1 commit into
mainfrom
feat/news-sentiment-pack

Conversation

@Zeeechenn

@Zeeechenn Zeeechenn commented Jun 15, 2026

Copy link
Copy Markdown
Owner

背景

2026-06-15 M27 情感 IC 诊断的落地。event_taxonomy0.65×事件极性 覆盖被量出对 IC 净负(event delta 三次测量 −0.0027 / −0.0146 / −0.0280),且非个股新闻驱动个股甩尾(兆易 603986:2 条板块 IPO 新闻、0 条公司新闻 → 情感 −71.3)。

改动(阶段0「便宜消融」,在现有标题管线上)

  • 行情/资金流过滤is_market_flow / is_company_specific / company_specific_titles;市场/资金流/榜单标题在分类前剔除,避免「市场下滑」里的「下滑」误判成 earnings_warning 硬事件。
  • event_override 默认关settings.sentiment_event_override_enabled(默认 False)。事件类型仍上报(可观测),但不再混入分数。可为干净单 provider OOS 复测重新打开。
  • 个股证据地板analyze_news 中,窗口无个股特定标题(仅噪声/跨公司,按 company_aliases 判)时返回中性 0 并跳过 LLM 调用(兼省成本)。
  • mixed-noise 收窄:窗口里只要存在个股标题,就只把个股特定标题送入 LLM prompt/cache/event score,不再把跨公司/资金流标题一并送给 LLM。
  • 历史回填覆盖:生产 postmarket 和 backtest news-cache 路径都会在可用时传入股票名+代码别名,避免过滤只在生产盘后路径生效。
  • event-ab harness 强制 override ON:让 m27 --event-ab 始终测量 override 的反事实 IC 效应,独立于生产默认值。

验证

  • 本地相关测试:46 passed / 1 skipped
  • 本地门禁:ruff clean、mypy clean、backend pytest 1224 passed / 6 skipped
  • 本地 frontend step:该 linked worktree 没有 frontend/node_modules,因此未在本地完成;GitHub Actions 的 frontend test/build 已通过。
  • GitHub Actions:push 与 pull_request 两个 CI run 均通过 backend lint/typecheck、backend tests、frontend test/build、security/dependency audit。

性质与采用门槛

这是防御性改动:移除一个已测量的 IC 拖累 + 增加鲁棒性(灭兆易式甩尾),不制造新情感 alpha(情感原始 IC ~0.02 的天花板是输入数据决定的,需阶段1 完整 iFinD pack 才可能验证能否抬升)。

⚠️ 采用边界:改动了镜像生产的情感行为 → test2 epoch 重置。合并/用于 live test2 信号前,应先做一次干净单 provider OOS 复测确认(与本会话纪律一致)。本 PR 不直接合并 main、不部署。

🤖 Generated with Claude Code

@Zeeechenn

Copy link
Copy Markdown
Owner Author

干净单 provider OOS 复测确认(2026-06-15)

把 test2 全池 387 个情感窗口用 Sonnet 4.6 单一 provider 重打(DB 副本,生产库未动,372/387 覆盖),消除之前 Codex+4.6 混合 confound:

样本 pure IC ICIR 正率 +event IC delta
满覆盖混合(Codex+4.6) 0.020 0.073 51.9% 0.006 −0.0146
干净单 provider 4.6 0.0735 0.282 60% 0.0449 −0.0286
  • 本 PR 的 override-off 验收通过:干净数据 delta 仍 −0.0286(override 把 pure 0.0735 砍到 0.0449),四次测量全负。
  • 📌 附带发现(超出本 PR 范围):混合样本的 pure IC 0.020 是 Codex 打分污染的伪低值;干净 4.6 打分 pure IC = 0.0735(过 IC 门、正率 60%),即情感打分模型质量是最大杠杆(local_cli/Codex→0.02 vs 4.6→0.0735)。

边界:同一时间窗、25 IC 天、ICIR 仍<0.4、exploratory。合并/部署仍需你点头。

@Zeeechenn Zeeechenn force-pushed the feat/news-sentiment-pack branch from df122c0 to 477ad33 Compare June 15, 2026 15:36
…oor, event-override off by default

Follow-up to the 2026-06-15 M27 sentiment IC diagnosis. The event_taxonomy
0.65 override was measured net-negative for IC across every variant tested
(event delta -0.0027 / -0.0146 / -0.0280), and noisy non-company headlines
drove individual-stock whipsaws (兆易 603986: 2 sector-IPO headlines, 0 company
news -> -71.3 sentiment).

Changes:
- event_taxonomy: market/fund-flow/list headlines are filtered before
  classification so they cannot misfire as hard corporate events (for example,
  "下滑" inside "市场下滑"). Adds is_market_flow / is_company_specific /
  company_specific_titles helpers.
- event_score: the 0.65 override blend is now gated by
  settings.sentiment_event_override_enabled (default OFF). Matched event types
  are still reported for observability; raw sentiment is kept. Re-enableable for
  a clean single-provider OOS re-test.
- sentiment.analyze_news: evidence floor returns neutral 0 and skips the LLM
  when a window has no company-specific headline. Mixed windows now send only
  company-specific titles into the LLM prompt/cache/event scoring.
- postmarket and backtest news-cache paths pass stock name+symbol aliases when
  available, so cross-company contamination filtering is not production-only.
- m27_alpha_diagnostic --event-ab forces the override ON so it keeps measuring
  the override's counterfactual IC effect independent of the production default.

Verification:
- PYTHONPATH=. pytest -q tests/test_news_sentiment_pack.py tests/test_m27_alpha_event_universe.py tests/test_m27_m28_integration.py tests/test_tavily_news.py tests/test_sentiment_cache.py tests/test_llm_runtime_provider.py
  -> 46 passed / 1 skipped
- make verify PYTHON=/Users/zeeechenn/mingcang/.venv/bin/python
  -> ruff clean, mypy clean, backend pytest 1224 passed / 6 skipped; local
     frontend step not runnable in this linked worktree because frontend
     node_modules is absent
- GitHub Actions on PR #20 after force-push -> backend lint/typecheck, backend
  tests, frontend test/build, and security/dependency audit all passed for both
  push and pull_request runs

This is a defensive change (removes a measured drag + adds robustness); it does
not manufacture new sentiment alpha. Production adoption for live test2 signals
should wait for a clean single-provider OOS confirmation (epoch reset).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@Zeeechenn Zeeechenn force-pushed the feat/news-sentiment-pack branch from 477ad33 to 97bbbbd Compare June 15, 2026 15:43
@Zeeechenn

Copy link
Copy Markdown
Owner Author

#2 换打分模型已加入本 PR(commit 4255c1b

依据干净 OOS(情感 IC 0.0735@sonnet-4.6 vs ~0.02@fast/Codex),把情感打分从硬编码 fast 改为可配 sentiment_model_tier(默认 capableclaude-sonnet-4-6,每个 provider 都映射到它)。同时证据地板升级为完整过滤(只有个股特定标题进 LLM/缓存键),backtest news_cache 也接了 alias。

⚠️ 生产启用需配 env(.env,未提交)

  • AI_PROVIDER=anthropic → 立即生效,云 API 付费、稳定,适合 88 股日批;或
  • AI_PROVIDER=local_cli + LOCAL_CLI_PREFER_CODEX=false → 走 claude -p --model claude-sonnet-4-6(CC 订阅,无 API 费),但批量有日配额截断风险(test2 25 股可接受,88 股 postmarket 慎用)。

验证:全量 1225 passed / 6 skipped,ruff + mypy clean。

@Zeeechenn

Copy link
Copy Markdown
Owner Author

更新(拆分):(sonnet-4.6 模型升级)已从本 PR 移出,单独到 #21(stacked on this PR)。本 PR 现在是纯新闻层加固(override-off + 个股相关性过滤 + 证据地板),1 个 commit 97bbbbd,零 model_tier、零 M51。上面提到 model_tier 的旧评论以 #21 为准。

@Zeeechenn

Copy link
Copy Markdown
Owner Author

更正(上一条评论吞了字):已拆分 —— sentiment_model_tier(sonnet-4.6 模型升级)从本 PR 移出,单独到 #21(stacked on this PR)。本 PR 现在是纯新闻层加固(event-override 默认关 + 个股相关性过滤 + 证据地板),1 个 commit 97bbbbd零 model_tier、零 M51。早先提到 model_tier 的评论改以 #21 为准。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant