Skip to content

feat(transport): add DoH connection self-healing (transparent retry + pool rebuild) - #42

Merged
olicesx merged 2 commits into
mainfrom
fix/doh-connection-self-healing
Aug 12, 2026
Merged

feat(transport): add DoH connection self-healing (transparent retry + pool rebuild)#42
olicesx merged 2 commits into
mainfrom
fix/doh-connection-self-healing

Conversation

@olicesx

@olicesx olicesx commented Aug 12, 2026

Copy link
Copy Markdown
Owner

背景

Closes #41

DohClient 此前是四个传输协议中唯一没有连接自愈的。代理节点抖动导致 DoH 连接变成半开后,reqwest 会无限复用死连接(POST 非幂等,hyper 不自动重试),每次请求 3s 超时,只能重启恢复。

方案

补齐与 TcpMuxClient/DotMuxClient/DoqMuxClient 对等的双保险,用 reqwest 的原语表达:

机制 mux client 做法 DoH 适配
透明重试 is_reused 判断失败 → 新连接重试,≥1.5s 预算 reqwest 无 reuse 信号 → 无条件重试一次(DNS 查询语义幂等)
错误计数重建 连续错误 ≥ 阈值 → reset() 连接 reqwest 无法 reset 单连接 → 重建整个 Client(drop = 清池)
  • ArcSwap<DohHttpClient> 热替换连接池(仓库已有依赖)
  • DashMap<Arc<str>, usize> per-upstream 错误计数
  • 新增配置 doh_health_check_error_threshold(默认 3,对齐 tcp 命名)
  • send() 签名和所有调用方不变

不照搬 max_age/idle_timeout:前者 reqwest 不暴露连接创建时间无法实现,后者由 pool_idle_timeout(90s) 覆盖。

验证

  • cargo build 编译通过
  • cargo test 全量 151 测试通过(含 9 个 DoH 集成测试),无回归
  • ✅ 新增单元测试覆盖:阈值重建、计数清零、per-upstream 隔离、错误分类
  • ✅ 实证测试:reqwest::Client::drop 立即关闭连接池的 keep-alive 连接(重建治本的前提,0.01s 内观察到连接关闭,pool_idle_timeout=60s 排除 idle 回收干扰)
  • ✅ 独立审查 6 个正确性盲点(并发/重入、ArcSwap 用法、错误分类、重试预算、资源生命周期、维度对比)均无缺陷

待确认(诚实声明)

代码逻辑层面已验证,但 issue 描述的真实代理抖动场景(dae HK 组)我无法复现,端到端效果待实测。建议在原环境下验证:节点抖动恢复后是否不再持续刷屏、是否无需重启即自愈。

已知权衡

阈值默认 3,节点恢复后前 2 次查询仍可能超时失败(透明重试时未达阈值、仍用同一死连接),第 3 次连续错误才整体重建。这与 TcpMuxClient(同阈值)行为一致。若需消除头部失败,可后续补 max_age 式主动轮换。

… pool rebuild)

DohClient previously had no self-healing, unlike TcpMuxClient/DotMuxClient/
DoqMuxClient. When a proxy node flapped and DoH connections turned half-open,
reqwest reused the dead connections indefinitely (POST is non-idempotent so
hyper won't auto-retry), causing persistent 3s timeouts until restart (#41).

This adds the two safeguards mirroring the mux clients, adapted to reqwest's
constraints (no per-connection reset, no reuse signal):

- Transparent retry: on transport error (timeout/connect/IO), retry once with a
  fresh-connection budget (>=1.5s); DNS queries are semantically idempotent.
- Pool rebuild: per-upstream consecutive error counter; at threshold (default 3,
  configurable via doh_health_check_error_threshold), rebuild the reqwest::Client
  — drop evicts the dead connection pool.

Uses ArcSwap<DohHttpClient> for a hot-swappable pool (already a repo dependency)
and DashMap for per-upstream error counts. The send() signature and all callers
are unchanged.

Tests cover the healing contract (threshold rebuild, counter reset, per-upstream
isolation, error classification) plus an empirical test proving that dropping a
reqwest::Client closes its pooled keep-alive connections (the rebuild premise).

Closes #41
@olicesx
olicesx merged commit 7da59f5 into main Aug 12, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant