feat(transport): add DoH connection self-healing (transparent retry + pool rebuild) - #42
Merged
Merged
Conversation
… pool rebuild) DohClient previously had no self-healing, unlike TcpMuxClient/DotMuxClient/ DoqMuxClient. When a proxy node flapped and DoH connections turned half-open, reqwest reused the dead connections indefinitely (POST is non-idempotent so hyper won't auto-retry), causing persistent 3s timeouts until restart (#41). This adds the two safeguards mirroring the mux clients, adapted to reqwest's constraints (no per-connection reset, no reuse signal): - Transparent retry: on transport error (timeout/connect/IO), retry once with a fresh-connection budget (>=1.5s); DNS queries are semantically idempotent. - Pool rebuild: per-upstream consecutive error counter; at threshold (default 3, configurable via doh_health_check_error_threshold), rebuild the reqwest::Client — drop evicts the dead connection pool. Uses ArcSwap<DohHttpClient> for a hot-swappable pool (already a repo dependency) and DashMap for per-upstream error counts. The send() signature and all callers are unchanged. Tests cover the healing contract (threshold rebuild, counter reset, per-upstream isolation, error classification) plus an empirical test proving that dropping a reqwest::Client closes its pooled keep-alive connections (the rebuild premise). Closes #41
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
背景
Closes #41
DohClient此前是四个传输协议中唯一没有连接自愈的。代理节点抖动导致 DoH 连接变成半开后,reqwest 会无限复用死连接(POST 非幂等,hyper 不自动重试),每次请求 3s 超时,只能重启恢复。方案
补齐与
TcpMuxClient/DotMuxClient/DoqMuxClient对等的双保险,用 reqwest 的原语表达:is_reused判断失败 → 新连接重试,≥1.5s 预算reset()连接Client(drop = 清池)ArcSwap<DohHttpClient>热替换连接池(仓库已有依赖)DashMap<Arc<str>, usize>per-upstream 错误计数doh_health_check_error_threshold(默认 3,对齐 tcp 命名)send()签名和所有调用方不变不照搬
max_age/idle_timeout:前者 reqwest 不暴露连接创建时间无法实现,后者由pool_idle_timeout(90s)覆盖。验证
cargo build编译通过cargo test全量 151 测试通过(含 9 个 DoH 集成测试),无回归reqwest::Client::drop立即关闭连接池的 keep-alive 连接(重建治本的前提,0.01s 内观察到连接关闭,pool_idle_timeout=60s 排除 idle 回收干扰)待确认(诚实声明)
代码逻辑层面已验证,但 issue 描述的真实代理抖动场景(dae HK 组)我无法复现,端到端效果待实测。建议在原环境下验证:节点抖动恢复后是否不再持续刷屏、是否无需重启即自愈。
已知权衡
阈值默认 3,节点恢复后前 2 次查询仍可能超时失败(透明重试时未达阈值、仍用同一死连接),第 3 次连续错误才整体重建。这与
TcpMuxClient(同阈值)行为一致。若需消除头部失败,可后续补 max_age 式主动轮换。