feat: distribute bazel JKS truststore via squid-ca-cert (Vault) - #19
Open
ccijunk wants to merge 25 commits into
Open
feat: distribute bazel JKS truststore via squid-ca-cert (Vault)#19ccijunk wants to merge 25 commits into
ccijunk wants to merge 25 commits into
Conversation
只依赖已有数据源,不引入新组件: - 各 Squid 的 access.log(命中/字节/客户端/bump-splice) - HAProxy stats CSV :8404(后端分发与健康) - cgroup cpu.stat(CPU 秒 → 每 GB CPU 成本) 按 begin/end 框定区间输出四层指标: Squid 缓存效果 / SSL Bump CPU 成本 / HAProxy 分发健康 / 客户端归因。 采集上有三个坑,都在脚本里处理了: 1. 必须框定区间。累计值把预热、健康检查、历史负载混在一起, 算出来的命中率和 CPU/GB 没有意义。 2. 健康检查要剔除。每 3 秒 × 3 后端 × 2 节点,日志里是 NONE_NONE/400、 方法为 -,不滤掉会把请求数和命中率彻底冲淡。 3. CONNECT 隧道不是对象请求。实测 bump 的 CONNECT 记录是 NONE_NONE/200 bytes=0,只是建隧道,真正负载会作为解密后的 GET https://... 再记一行 —— 混进去会凭空翻倍请求数并稀释命中率。 splice 则相反,是 TCP_TUNNEL 带真实字节且不可能命中。 最初的实现把这两者搞混了,4 次下载被记成 8 个请求。 实测(4 个从未取过的包,分别跑纯 MISS 与纯 HIT 两个窗口): 纯 MISS: 命中率 0% , 44.42 CPU秒/GB 纯 HIT : 命中率 76.4%, 37.13 CPU秒/GB 注意这个小样本(4 请求、低并发)不足以复现 k8s 压测里 「纯 HIT 比 MISS 更吃 CPU」的结论 —— 那是 N=120 高并发下 TLS 批量加密 打满 CPU 的效应,低并发时开销主要在握手和证书生成。 脚本的作用是提供采集手段,系数仍需在目标并发下实测。 归因层依赖 PROXY protocol(见 PR #9),未生效时脚本会给出提示。 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
原先只报一个「每 GB CPU」,这个数不可信,有两个问题:
1. CPU 不与字节成正比。它由「每请求固定成本」(TLS 握手、证书生成、缓存查找)
和「每字节边际成本」(加密、I/O)两项构成。只报每 GB 会把固定成本摊进字节,
于是同一套环境下这个数随对象大小漂移 —— 实测 14MB 对象约 28 秒/GB,
58KB-6MB 混合约 37-44 秒/GB,相差 1.6 倍。当成常数用会算错容量。
2. 窗口里的空闲 CPU 被算进了请求成本。Squid 零流量时也在烧 CPU:
健康检查(每3秒×3后端×2节点)、日志写入、缓存索引维护。
实测空闲基线约 0.004 CPU秒/秒(三副本合计)。窗口越稀疏(比如里面有大量
docker run 启动等待时间),这部分占比越高,实测能占到测量值的三分之一。
改法:
- 新增 metrics.sh baseline [秒]: 零负载时测一次空闲 CPU 速率,存 .metrics-baseline
- end 时按窗口时长扣除空闲基线,输出「原始 − 基线 = 归因」
- 拆成两项输出: 每请求 CPU / 每 GB CPU, 并同时给出平均对象大小
- 明确标注每 GB CPU 不是常数, 并给出容量公式:
核数 ≈ 请求速率 × 每请求CPU + 吞吐(GB/s) × 每GB CPU
拟合这两项需要在不同对象大小/并发下各跑一个窗口
顺带纠正之前基于这个指标得出的结论: 早先用「纯 MISS 44.42 秒/GB vs
纯 HIT 37.13 秒/GB」比较两者 CPU 成本是无效的 —— 两个窗口对象集不同,
差值又落在空闲基线污染的范围内,不足以支撑任何 HIT/MISS 结论。
低并发本来也测不出 k8s 压测里 N=120 时「HIT 比 MISS 更吃 CPU」的效应,
脚本里已加注,避免再用小样本去推翻高并发结论。
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
两层都从已经在采集的 access.log 字段算出来,不需要新数据源。 延迟层(字段 $2 耗时ms),按 HIT/MISS 分开出 p50/p95/p99: - 均值会被长尾骗过去。实测同一个 14MB 对象的 TCP_HIT 耗时分布为 133/246/251/256/273/5106 ms —— 同样是命中,最慢的比最快的高 38 倍。 这种尾巴用两次 wall-clock 采样(用例 04 现在的做法)根本看不见。 - 按命中结果分开是必要的: MISS 延迟基本由源站带宽支配,HIT 延迟才是 缓存真正买到的东西,混在一起两者互相污染。 - 对在线业务而言 p99 才是用户实际感受,这是从「省了多少带宽」转向 「用户等多久」时最先需要的指标。 错误层,关键是把代理故障与源站故障分开: - 判据是层级字段: Squid 自己生成的错误是 HIER_NONE(压根没连上源站), 真正到达源站再返回的错误带 HIER_DIRECT 等。 - 为什么重要: 源站 404(包不存在)是正常现象,不该告警;而 DNS 失败、 连接失败、ACL 拒绝才是要告警的。k8s 压测里 120 并发中 31-37 个「失败」 其实全是源站 404,不是 Squid 故障(见 reports/stress-benchmark-20260811.md)。 - 另外单列客户端中断(*_ABORTED)与隧道失败(CONNECT 阶段即失败)。 验证: 故意混入一个不存在的包,输出正确归类为「源站4xx 1 / 代理故障 0」。 实现上把窗口日志先落到临时文件再多趟分析,避免为每个指标重复 docker exec。 注意分位数的口径可以搬到生产,代码不能: Prometheus 要用预先分桶的 histogram,无法从计数器抓取值反算真实 p99。已在脚本头部注明。 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
分层性能指标采集脚本:命中 / 延迟分位 / 错误分类 / 两项 CPU 成本
依据 PR #12(scripts/metrics.sh)的建议改造用例 04。旧版只打印首次(MISS)/ 二次(HIT)两个孤立 wall-clock,看不见长尾、不分类错误、不测 CPU 成本。 新版用 metrics.sh begin/end 框定 MISS 与 HIT 两个窗口: - 固定客户端 IP(balance source → 同一台 Squid)+ cache-busting URL, 保证窗口1是真 MISS、窗口2是真 HIT(各 Squid 缓存独立、无 peer) - 先测空闲 CPU 基线并在归因时扣除 - 保留 HTTP 200 真断言,分层指标作为信息输出 实测效果(本地 Compose): - MISS p50=3638ms / 命中率0% / 每请求CPU 0.247s - HIT p50=130 p95=148 p99=149ms / 命中率100% / 每请求CPU 0.085s - 缓存价值 = 延迟 28× 提速; CONNECT 隧道正确去重(未重复计数) - 归因层正确检测到 Compose 侧无 PROXY protocol(#9)并提示 README 同步更新测试点表与预期输出。 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
把 tests/04 的分层指标口径搬到 K8s(与 Compose 版 metrics.sh 完全同口径:
健康检查过滤 / CONNECT 去重 / HIER 错误分类 / HIT-MISS 延迟分位 / CPU 两项成本),
只把数据源从 docker+HAProxy 换成 kubectl exec(access.log、cgroup v1 cpuacct.usage)
+ kubectl get endpoints。
- k8s/metrics-k8s.sh: K8s 版采集(baseline/begin/end/now)
- k8s/latency-test.sh: 一键 driver(起 client pod → baseline → MISS/HIT 两窗口 → 清理),
等价 Compose 的 tests/04
- .gitignore: 忽略 .metrics-k8s-{state,baseline}
- README: 新增「K8s 上的同款采集」小节
测试集群(test-husheng, squid StatefulSet 2/3 Ready)实测:
- MISS p50≈1000ms / 命中率0% ; HIT n=30 p50≈167 p95≈176 p99≈177ms / 命中率100%
- CONNECT 隧道正确去重(bump 1 / bump 30)
- HIT 每请求 CPU(0.177)略高于 MISS(0.140),方向与「HIT 重新加密更吃 CPU」一致
- 归因层直接显示真实客户端 pod 段 IP —— K8s Service 保留源 IP,无需 PROXY protocol
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- 从 gmarmstrong/buildkit@proxy-chaining 编译镜像, 正对照/缓存/负对照三场景实测通过 - 关键结论: 无 cert 配置 → CA 经 SSL_CERT_FILE 走系统信任库, 实测成立; PULL+RUN 共用一套 env 是 BuildKit 既有标准行为, 与 deploy 的 splice+registry-proxy 互补 - 新增 buildkitd-6996.toml / docker-compose.6996.yml, 与 fork 版并列 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
docs(buildkit): 上游 PR #6996(env 变量式上游代理)端到端验证报告与部署资产
test: 分层指标取代两次采样(Compose 用例04 + K8s 同款采集,两套实跑)
…sults - 3 new refresh_patterns: .crate / .zip / .pth|pt|safetensors - helm upgrade REVISION 14 (squid-rpardini-0.1.4), ArgoCD ownership + kubelet volume-sync gotchas - retest: cargo 11% -> ~95% hit, wget 94MB .pth TCP_HIT zero-origin feat(chart): 0.1.4 — force-cache rules for cargo/model-weight traffic - refresh_pattern for .crate (cargo), .zip, .pth/.pt/.safetensors (model weights) - measured: cargo hit 11% -> ~95%, wget 94MB .pth TCP_HIT zero-origin (gy-006) - verified via helm upgrade REVISION 14
JKS truststore (Bazel/JDK) fetched from Vault (secrets/data/ascend/ci) and synced into squid-ca-cert SecretDefinition for CI namespaces (buildkitd, ascend-gha-runners, nv-action, squid). Align caBundleKey/ caPublicKey to v3 vault properties.
CLA Signature Passccijunk, thanks for your pull request. All authors of the commits have signed the CLA. 👍 |
Linking Issue Notice@ccijunk , the pull request must be linked to at least one issue. |
…alid options - immutable rules only for content-addressed/A-class artifacts (.whl/.crate/.deb) - .zip/.tar.gz/.pth demoted to LM-aging (0/20%/4320): ref-addressed & overwritable - codeload refs/tags + github releases/download get immutable; refs/heads short TTL - remove dead .docker.io rule, unescape domains, max 525960->525600 (no cropped warning) - add CACHE-STRATEGY.md (full per-rule audit + CI scenario analysis)
- wait_case: wait on Volcano Job phase (pods scheduled in batches polluted subsequent case windows); fix missing -n $NS - analyze: awk printf %d truncated >2GB byte counters to -2^32 (%.0f) - 01-pip: add set -o pipefail so pip failures are not masked by tail
CLA Signature Passccijunk, thanks for your pull request. All authors of the commits have signed the CLA. 👍 |
- analyze: window ends DONE+180s; Volcano marks Job Completed while late-batch pods still stream (case 01 window held only background api.github CONNECTs → false 0% HIT) - run-tool-traffic.sh: wait_pods_done helper (pods may outlive vj phase) - CACHE-STRATEGY.md §10.1: document Volcano batching, background TLS noise (A000412 = bad_certificate alert from unknown 10.0.0.209/225), epoch access.log timestamps - verified case 01 (pip) = 99.9% HIT (7.5GB out, 0.1% origin)
CLA Signature Passccijunk, thanks for your pull request. All authors of the commits have signed the CLA. 👍 |
…data fallback Document first-match-wins ordering: extension immutable rules (whl/crate/deb) intercept artifacts before domain rules, so .pypi.org/.golang.org/.debian.org/ .ubuntu.com actually only fallback to index metadata + unclassified content. Ordering-sensitive: moving domain rules above extension rules drops .deb/.whl from immutable to 0 20% 4320.
CLA Signature Passccijunk, thanks for your pull request. All authors of the commits have signed the CLA. 👍 |
…& gitcode 0% - CACHE-STRATEGY.md §10.2: squid 7.6 stable-build swapout race triggered by SWAPFAIL under cold-cache concurrency; SFS shared volume 92% full as amplifier; fixes: free registry-cache / collapsed_forwarding / aufs-disks; 7.6 is a stable tag (SQUID_7_6), not dev — no upstream fix yet - TOOL-RESULTS.md: gitcode clone 0% (POST pack, structural); non-git paths WAF-blocked (418/403/verification page); single-case cold apt = 65.4% HIT - run-tool-traffic.sh: wait_pods_done must 'seen' a pod before trusting zero running pods (vj may be Completed before any pod is scheduled)
CLA Signature Passccijunk, thanks for your pull request. All authors of the commits have signed the CLA. 👍 |
- run-tool-traffic.sh wait_pods_done: 'seen' must trigger on ANY pod
(incl. Succeeded); previous version stuck 30min when pods finished
before the first poll
- analyze-tool-traffic.py:
- cluster clock offset (+128s observed) — timeline is local, access.log
is cluster; without it every window is misaligned
- drop DONE+180 tail (DONE is now precise; back-to-back cases made
+180 swallow the next case's traffic) — clamp e to next SUBMIT
- read access.log from PREVIOUS container stdout (current access.log
is recreated on restart, ~2min only); pod-IP filtering removed
(Volcano rebuilds pods, IPs no longer match)
- tool/03-github.yaml: curl raw.githubusercontent.com --max-time 120
(was hanging forever without timeout)
CLA Signature Passccijunk, thanks for your pull request. All authors of the commits have signed the CLA. 👍 |
…probe - 01-pip.yaml: replace jsdelivr CDN with gh-proxy for requirements.txt fetch - 03-github.yaml: remove github.com reachability probe (meaningless per request) - cachedemo.yaml: add hf-mirror.com to NO_PROXY - run-tool-traffic.sh: make LOG_DIR configurable via env var - run-per-case-wlcb.sh: new wrapper for sequential per-case monitoring on wlcb-001
CLA Signature Passccijunk, thanks for your pull request. All authors of the commits have signed the CLA. 👍 |
16-tool sequential run on wlcb-001 with per-case Prometheus monitoring. Reports client/origin throughput, hitrate, and pattern classification (steady, burst, spike, decay) for each tool.
…s, huaweicloud mirror - statefulset.yaml: add nodeAffinity and tolerations support (template) - values.yaml: add nodeAffinity default (empty), switch alpine mirror to huaweicloud - values-wlcb-001.yaml: new — wlcb-001 cluster values (amd64, node 192.168.1.49, cache toleration) - values-gy-001.yaml: new — gy-001 cluster values (amd64, node 192.168.1.191, cache toleration)
CLA Signature Passccijunk, thanks for your pull request. All authors of the commits have signed the CLA. 👍 |
ccijunk
force-pushed
the
feat/jks-truststore
branch
from
August 26, 2026 01:52
cde9564 to
df58737
Compare
CLA Signature Passccijunk, thanks for your pull request. All authors of the commits have signed the CLA. 👍 |
1 similar comment
CLA Signature Passccijunk, thanks for your pull request. All authors of the commits have signed the CLA. 👍 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Distribution of the Bazel/JVM truststore (
squid-bazel-trust.jks) from Vault via thesquid-ca-certSecretDefinition, plus the 0.1.6 cache strategy (per-addressing refresh_pattern rules) and traffic-test tooling fixes.Changes
squid-ca-certkeysMap 增加squid-bazel-trust.jks（Vault propertysquid_bazel_trust_v3_jks），随 CA 一起分发到 buildkitd / ascend-gha-runners / nv-action / squidcaBundleKey/caPublicKey对齐线上 v3 名（squid_ca_bundle_v3_pem/squid_ca_v3_pem），新增caTruststoreKey.whl/.crate/.deb：10080 100% 525600 ignore-reload override-expire ignore-no-store）.pth/.pt/.safetensors、codeloadrefs/heads、索引页）降级0 20% 4320（尊重 LM 老化）refs/tags+ githubreleases/download→ immutableignore-no-cache/override-vary（ERROR/Unknown option）、删除失效.docker.io规则、域名转义\\.pypi\\.org等、max=525600（消除 cropped 告警）wait_case改为等待 Volcano Job phase（vj 分批调度导致窗口污染），修复缺失-n $NS；新增wait_pods_done（pods 可能比 vj phase 晚结束）；LOG_DIR改为可配置环境变量printf %d将 >2GB 字节计数截断为负值 →%.0f；窗口终点DONE+180s捕获分批调度尾部流量；HIT% 与 Prometheusclient_out/origin_in交叉验证set -o pipefail（避免 tail 掩盖失败）、PIP_CERT 去重；jsdelivr CDN 替换为 gh-proxy 获取 requirements.txthf-mirror.com加入 NO_PROXY/etc/squid-bazel-trust/squid-bazel-trust.jks再写.bazelrc关键坑
JAVA_TOOL_OPTIONS，trustStore 参数只能走.bazelrcstartupignore-no-cache；override-vary是未知选项（ERROR）；override-expire/ignore-reload/ignore-no-store为 legacy WARNING 但有效subPath挂载不热更新，配置变更须rollout restart（kubelet 不会重载 subPath 文件）10.0.0.209/10.0.0.225，不在任何 pod/Service 列表）每 1-5s 对api.github.com:443发 CONNECT + TLS 失败（cache.logA000412= SSL alertbad certificate），每分钟 ~10-20 条 24/7 持续，与 case 流量无关Test (gy-006, 0.1.6 复测)
DONE+180s修复后实测 99.9% HITgh-proxy.test.osinfra.cn（03-github.yaml 用insteadOf把 github.com 重写到 gh-proxy 镜像），client_out/origin_in交叉验证全程走代理wlcb-001 测试 (warm cache, 16 工具)
traffic-test/WLCB-001-TRAFFIC-REPORT.md— 逐 case 流量模式分析。关键发现: