Repository navigation
Conversation
|
Important Review skippedToo many files! This PR contains 270 files, which is 170 over the limit of 100. To get a review, reduce the PR to 100 files or fewer by splitting it into smaller PRs or changing its base branch. Upgrade to a paid plan to raise the limit. This review couldn't start because sufficient usage credits or metered capacity aren't available. Add credits or update usage-based reviews in the billing tab, then retry. ⚙️ Run configuration
📒 Files selected for processing (270)
You can disable this status message by setting the
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
Author
|
Closing: this branch is based on the internal pingkai-master lineage. The PR will be opened against the internal repository instead. |
The change previously proposed in this branch is being submitted to our internal repository instead. This branch intentionally contains no code.
Meowooh
force-pushed
the
fix/cold-range-metadata-lock-inversion
branch
from
October 8, 2026 15:27
237b9cd to
22ecc07
Compare
liangjchen
added a commit
that referenced
this pull request
Oct 19, 2026
Follow EloqKV #575 and use the RustFS binary already installed in the shared ubuntu-dev image. Remove the MinIO source build and cache, check for RustFS before compiling, and use its readiness endpoint. Preserve failure status and logs while stopping the server and removing its private temporary data on exit. Verified the published amd64/arm64 image manifests and the image-pinned amd64 RustFS checksum. The exact workflow cloud test passed locally: 1 case and 11 assertions. Missing-binary and injected-exit-7 cleanup checks passed, as did actionlint, YAML, shell syntax, and whitespace.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
结论
冷加载 range slices 时持有 fast-reader 再去申请
meta_data_mux_，与 fast writer（先拿meta_data_mux_写锁、再等 per-shard fast reader 归零）构成闭环。修复方式是：在调用方已经持有的 metadata guard 内取 catalog/schema 快照，冷加载路径不再重新申请该锁。故障现场
2026-10-08 08:28–08:45（UTC+8），三节点只读取证（
/proc/<pid>/mem的O_RDONLY+pread，未 attach、未暂停进程；Build IDlibtxservice.so=0f3cb26c…，libstorage_eloq_core.so=853d8fd6…）。.13/ PID 18246：meta_data_mux_.__cur_writer183240x80000000/0x80000001meta_data_mux_ + 8（0x55dcc973c30）range_slices_ == nullptr，owner 属于本 NG）闭环两半：
meta_data_mux_写锁 →FastMetaDataMutex::lock在0x4b8500–0x4b8507循环等待低 31 位清零，即等待 shard 1 的 reader 退出。PinRangeSlice已持有 shard 1 fast-reader → 进入FetchRangeSlices→GetSharedTableSchema→ 申请meta_data_mux_读锁。它拿不到，也就不会释放 fast-reader。栈上的返回地址与部署指令逐条交叉验证过调用链（
libtxservice.so+0x4bdfa0→GetSharedTableSchema，+0x396c2c→FetchRangeSlices，libstorage_eloq_core.so+0x170243→PinRangeSlice<MongoKey>）；同一现场脚本重复采样输出verdict: DEADLOCK_CONFIRMED。入口节点首次阻塞在 02:36:39，与 02:36:36 的 range 2926 在线 split 时间线完全对应。修复
meta_data_mux_。LocalCcShards新增私有GetRangeCatalogSnapshotInternal：校验调用方真实持有的 fast/ordinary guard，在 guard 下读取table_catalogs_，并在CatalogEntry::s_mux_下复制 current/dirty schema。FetchRangeSlices/FetchTableRanges改为接收这个 owning 快照并设为私有（…WithCatalogGuard）；公开 wrapper 为 replay、Analyze、DataSync 等原本没有 guard 的调用方保留普通加锁路径。guard 覆盖注册与存储投递，投递后不再访问请求对象（inlineSetFinish会销毁它）。FetchRangeSlicesReq::SetFinish不再查 catalog。 key cache 的 advisory 估算改用随请求携带的 admission 时刻 current schema——identity 选择本就基于它；回调里回查等于把同一个重入放回 inline 完成路径。KickoutRangeSlices()，它会再取meta_data_mux_。现在只投递一个无指针、可合并的通知，由独立 eviction pthread 在自己的队列锁之外消费。该线程在容器、bucket、metrics、ranges heap 就绪之后启动，startup 期间的通知保持 pending，shutdown 在 shard 拆解前 join。它执行的就是普通KickoutRangeSlices逻辑；ranges-heap owner 保持 idle，因为它的 mimalloc 后备 TLD 与 scanner 分配共享，不能并发做 scratch 分配。影响范围
PinRangeSlice/PinRangeSlices的冷加载分支、FetchTableRanges的冷加载分支、冷加载完成回调、以及内存打满时的淘汰触发路径。测试
测试先空着。 本 PR 只包含修复与文档，回归测试由另一个 agent 验证后单独提交到本分支。
需要覆盖的交错：reader 已持有 fast guard → writer 已取
meta_data_mux_写锁并等待 reader 排空 → reader 再解析 schema/identity，要求两侧都有界完成；以及 inline 完成（成功 / ranges 预算打满通知）这条路径。仅仅照抄锁算法的 fixture 不算证据。验证情况（截至提交时）
-Wall -Wextra -Werror下可编译（按测试夹具的编译边界抽取生产源码构建）；同一夹具在修复前源码上复现了死锁（RUNTIME_TIMEOUT）。PinRangeSlices、inline 完成、预算打满通知）；多节点真实集群上 range 2701 的锁环消失；淘汰线程的启停时序。.11输出NOT_OBSERVED，但那只是单点诊断探针，不等于集群健康，也不能作为验收依据。诊断报告与取证细节按需补充。