Skip to content

fix: respect save path changes and localize X media - #6

Open
aiirux-opc wants to merge 1 commit into
izscc:mainfrom
aiirux-opc:aiirux/x2md-fix
Open

aiirux-opc wants to merge 1 commit into
izscc:mainfrom
aiirux-opc:aiirux/x2md-fix

Conversation

@aiirux-opc

Copy link
Copy Markdown

中文说明

变更摘要

本 PR 修复两个彼此独立、但都会表现为“设置已保存却没有生效”的问题:

  1. 更换主要保存目录后,已经保存过的内容仍被旧目录中的去重记录命中,导致新目录没有生成 Markdown。
  2. 在使用代理 Fake-IP DNS 的网络环境中,X 图片域名会解析到 198.18.0.0/15,图片下载在发出 HTTP 请求前就被 SSRF 防护拒绝;同时补齐了 Bun 1.3.13 对 DNS lookup all: true 回调形式的兼容。

问题根因

保存目录切换

duplicate_policy=skip 原先只检查最新修订记录中的文件是否仍然存在,没有检查这些文件是否位于当前配置的 save_paths。因此,即使用户已经切换主要保存目录,只要旧目录文件还在,保存流程就会直接返回 skipped。

X 图片本地化

代理软件可将 pbs.twimg.com、video.twimg.com 等域名解析到基准测试网段 198.18.0.0/15。该网段默认被视为非公网地址,原有安全下载逻辑会在连接前拒绝它。

通过应用内置 Bun 1.3.13 做真实网络验证时还发现,HTTPS 客户端可能以 all: true 调用自定义 lookup。原实现始终通过单地址回调返回结果,导致 Bun 报出 results.sort is not a function。

实现方案

保存去重

  • 仅把“文件仍存在,并且父目录与当前某个 save_path 完全匹配”的索引文件视为有效目标。
  • 只有当前所有保存目录都已被有效文件覆盖时,skip 策略才返回 skipped。
  • 当前目录列表中尚未覆盖的目标会进入写入事务。
  • 旧目录中的文件保留,不迁移、不删除,也不再作为当前保存结果返回。
  • update 和 always_new 策略保持原有语义。

安全下载与 Fake-IP

  • Fake-IP 例外严格限定为 pbs.twimg.com 和 video.twimg.com。
  • 例外严格限定为 IPv4 198.18.0.0/15。
  • 任意其他域名解析到 Fake-IP 时仍然拒绝。
  • 即使是受信任媒体域名,解析到 loopback、私网或其他保留地址时仍然拒绝。
  • 每次重定向仍会重新执行协议、端口、DNS 和地址校验。
  • 固定 DNS lookup 同时支持单地址回调和 all: true 数组回调,兼容 Node.js 与应用内置 Bun。

行为变化

场景 修复前 修复后
保存过内容后切换主要目录 返回旧目录文件,新目录没有文件 在新目录补存一份,旧文件保留
多个当前目标中有一个文件缺失 恢复缺失目标 行为保持不变,只恢复缺失目标
X 图片域名解析为受信任 Fake-IP 下载前被拒绝,Markdown 回退远程链接 下载附件并写入本地相对引用
任意域名解析为 Fake-IP 拒绝 继续拒绝
受信任域名解析为回环或私网地址 拒绝 继续拒绝
Bun 使用 lookup(..., { all: true }) 回调结构不兼容 返回固定地址数组

文件变更

  • app/core/save.ts
    • 将去重判断限定在当前保存目录范围内。
    • 只为当前尚未覆盖的目录创建写入事务。
  • app/core/safe-download.ts
    • 增加受信任 X 媒体域名的 Fake-IP 边界判断。
    • 增加兼容 Node/Bun 两种回调契约的 pinned lookup。
  • app/tests/save-index.test.ts
    • 覆盖切换保存目录后补存同一内容的回归场景。
  • app/tests/safe-download.test.ts
    • 覆盖受信任 Fake-IP、任意域名 Fake-IP、回环地址和两种 lookup 回调契约。
  • app/tests/api.test.ts
    • 将 X 图片本地化契约更新为本地附件引用。

验证结果

  • npm run check
    • TypeScript 类型检查通过。
    • JavaScript/TypeScript:403 项测试全部通过。
    • Python:11 项测试全部通过。
  • npm run smoke:mac
    • 打包应用启动、/ping、/save、/status、/log 和 /open 冒烟测试通过。
  • 使用 /Applications/X2MD.app 内置 Bun 1.3.13 进行真实下载:
    • pbs.twimg.com 在测试网络中解析为 198.18.0.52。
    • 成功下载 124,267 字节 JPEG,尺寸为 1672x941。
  • 使用真实 /config -> /save 链路验证:
    • HTTP 状态 200。
    • outcome=saved。
    • media.completed=1、media.failed=0。
    • 附件实际存在,Markdown 使用本地相对路径。
  • 本地构建验证:
    • Mac ZIP 结构与版本检查通过。
    • DMG 校验和及挂载检查通过。
    • Chrome 扩展 ZIP 完整性、Manifest V3 和版本检查通过。
    • 构建产物未包含在本 PR 中。

兼容性与风险

  • 没有配置格式或状态数据迁移。
  • 没有新增或升级依赖。
  • 没有放宽通用 SSRF 规则。
  • Fake-IP 兼容范围保持在明确的平台媒体域名和固定网段内。
  • 旧目录文件不会被自动移动或删除。

English Description

Summary

This PR fixes two independent issues that both made saved settings appear ineffective:

  1. After changing the primary save directory, an existing deduplication record could still point to a file in the previous directory, preventing Markdown from being written to the newly configured destination.
  2. In proxy environments that use Fake-IP DNS, X media hosts can resolve into 198.18.0.0/15. The download was rejected by SSRF protection before any HTTP request was made. This PR also fixes the DNS lookup callback contract used by Bun 1.3.13 when all: true is requested.

Root Causes

Save path changes

For duplicate_policy=skip, the save pipeline previously checked only whether every file in the latest indexed revision still existed. It did not verify that those files belonged to the currently configured save_paths. A file left in the previous directory therefore caused an immediate skipped result after the destination changed.

X image localization

Some proxy configurations resolve pbs.twimg.com and video.twimg.com to the benchmarking range 198.18.0.0/15. The existing downloader correctly treated that range as non-public, but had no narrowly scoped way to distinguish a trusted platform media host using proxy Fake-IP DNS from an arbitrary untrusted host.

Real-network testing with the Bun 1.3.13 bundled in the macOS app also showed that the HTTPS client may invoke a custom DNS lookup with all: true. The previous callback always returned one address instead of an address array, producing results.sort is not a function.

Implementation

Save deduplication

  • Indexed files count as intact targets only when they still exist and their parent directory exactly matches a currently configured save path.
  • The skip policy returns skipped only when every current save path is already covered.
  • The write transaction receives only current destinations that are not yet covered.
  • Files in previous directories are preserved, but are not returned as current save results.
  • The existing update and always_new semantics remain unchanged.

Safe download and Fake-IP support

  • The Fake-IP exception is restricted to pbs.twimg.com and video.twimg.com.
  • The exception is restricted to IPv4 198.18.0.0/15.
  • Arbitrary hosts resolving to Fake-IP addresses remain blocked.
  • Trusted media hosts resolving to loopback, private, or other reserved addresses remain blocked.
  • Every redirect hop continues to revalidate the protocol, port, DNS result, and resolved address.
  • The pinned DNS lookup now supports both single-address callbacks and all: true array callbacks for Node.js and the bundled Bun runtime.

Behavior Changes

Scenario Before After
Change the primary directory after saving content Returns the old file and writes nothing to the new directory Writes a copy to the new directory and preserves the old file
One current target is missing Restores only the missing target Unchanged
Trusted X media host resolves to proxy Fake-IP Rejected before download; Markdown keeps the remote URL Downloads the attachment and renders a local relative reference
Arbitrary host resolves to proxy Fake-IP Rejected Still rejected
Trusted host resolves to loopback/private address Rejected Still rejected
Bun requests lookup(..., { all: true }) Callback shape is incompatible Returns an array containing the pinned address

Files Changed

  • app/core/save.ts
    • Scopes deduplication to the currently configured save directories.
    • Writes only to current destinations that are not already covered.
  • app/core/safe-download.ts
    • Adds narrowly scoped Fake-IP handling for trusted X media hosts.
    • Adds a pinned lookup compatible with Node.js and Bun callback contracts.
  • app/tests/save-index.test.ts
    • Adds regression coverage for saving the same capture after changing directories.
  • app/tests/safe-download.test.ts
    • Covers trusted Fake-IP, arbitrary-host Fake-IP, loopback rejection, and both lookup callback shapes.
  • app/tests/api.test.ts
    • Updates the X image localization contract to expect a local attachment reference.

Validation

  • npm run check
    • TypeScript type checking passed.
    • 403 JavaScript/TypeScript tests passed.
    • 11 Python tests passed.
  • npm run smoke:mac
    • Packaged app startup, /ping, /save, /status, /log, and /open passed.
  • Real download using Bun 1.3.13 bundled with /Applications/X2MD.app:
    • pbs.twimg.com resolved to 198.18.0.52 in the test environment.
    • Downloaded a 124,267-byte JPEG with dimensions 1672x941.
  • Real /config -> /save flow:
    • HTTP 200.
    • outcome=saved.
    • media.completed=1, media.failed=0.
    • The attachment existed on disk and Markdown contained the local relative reference.
  • Local packaging checks:
    • Mac ZIP structure and version validation passed.
    • DMG checksum and mounted bundle validation passed.
    • Chrome extension ZIP integrity, Manifest V3, and version validation passed.
    • Build artifacts are not included in this PR.

Compatibility and Risk

  • No configuration or state-schema migration.
  • No dependency additions or upgrades.
  • No relaxation of the general SSRF policy.
  • Fake-IP compatibility remains limited to explicit platform media hosts and one fixed network range.
  • Existing files in previous save directories are never moved or deleted.

Scope duplicate detection to files in the currently configured save directories so captures are written after the primary destination changes.

Allow trusted X media hosts to use proxy Fake-IP DNS results while preserving private-network and redirect validation, and support Bun's all-address lookup callback contract.

Add regression coverage for save-path changes, Fake-IP boundaries, pinned DNS callbacks, and localized X image references.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant