diff --git a/docs/design/p3-router-validation-delivery.md b/docs/design/p3-router-validation-delivery.md new file mode 100644 index 00000000..646f53b0 --- /dev/null +++ b/docs/design/p3-router-validation-delivery.md @@ -0,0 +1,441 @@ +# P3 Router 验证基础设施交付文档 + +> **这是什么**:DSV4-Flash MoE Router(P3)的验证基础设施——验证阶梯、 +> canonical 指纹、首差分定位、负向测试矩阵的设计与实现记录。 +> **任务编号 T09 的含义**:见 `/workspace/p3-task-selection.md`(v22,§4-T09), +> 十人分工中"验证基础设施"一项;本文档即该项的交付说明。 +> **怎么用**:审查人核对合同条款可跳读 §三 的 11 条决策;接手人从 §六 的 +> 三项外部待办看起;复现测试看 §七。 +> 交付日期:2026-09-09(开发 2026-09-07 ~ 09-09) +> 状态:范围内全部完成,等待上游 start kit 解锁剩余三项外部依赖 +> 测试基线:**244 个测试收集,205 passed / 39 skipped(GPU 门控)**,CLI smoke 8/8 exit=0 + +--- + +## 一、背景与任务定位 + +### 1.1 P3 是什么 + +DSV4-Flash MoE 的 Router 层(合同 §1.1):前 3 层用 Hash 路由(`tid2eid` 查表), +第 3 层起用 Learned 路由(correction-bias + stable Top-6)。P3 生产 `RoutePlan`, +供 P4(pack/transport)、P6(combine)、P7(cross-config)消费。 + +### 1.2 在十人分工里的位置 + +任务编号 T01–T10 各领一块(详见合同 `/workspace/p3-task-selection.md`)。 +本任务(T09)是**验证基础设施**:别人写 router 算子, +我们写"怎么证明他们写对了"。核心交付四件事(合同 §4-T09 原文): + +1. 验证阶梯 L1/L2/L3a/L3b(+WS2 扩展) +2. 独立重写 naive Top-6,交叉检查 T01 golden +3. 按 `(absolute_layer, site, pass, event_index, global_token_id, rank)` 六元组定位首差分 +4. 负向 fixture 矩阵(tie、XOR、bitflip、wrong-run、fallback、missing provenance、 + non-finite、selection-gradient) + +### 1.3 开局约束:T01 未发布 + +开局核查(2026-09-08 确认,2026-09-09 复核):`origin/main` 无任何 P3 代码、 +无 `rl_engine/moe/` 目录、全分支搜不到 router 相关提交。T01 start kit 处于 +`anchor_pending`(合同 §1.3)。 + +**合同 §5 明确允许并行**:T09 可先用自有 synthetic fixture 开发,不等 T01。 +这决定了整个交付形态——所有依赖 golden/anchor 的部分做成"架子 + 自动生效的门控", +T01 发布后零改动接入。 + +--- + +## 二、交付物清单 + +``` +rl_engine/moe/ +├── __init__.py 10 行 包声明 +├── naive_topk6.py 163 行 T09 自有 fixture:全序 Top-6("独立重写") +└── p3_verdicts.py 151 行 合同 §6 状态码冻结 + 优先级仲裁 + +rl_engine/moe/validation/ +├── __init__.py 12 行 +├── report.py 50 行 LadderReport + make_pass/make_fail(唯一构造点) +├── fingerprint.py 235 行 §2.4 canonical 序列化:semantic/artifact 双哈希 +├── first_mismatch.py 199 行 六元组首差分定位 + owner/issue 归因表 +├── comparison.py 340 行 四级固定顺序比较器(§2.5) +├── ladder.py 208 行 L1 repeat / L2 invariance / L3a oracle / L3b 双引擎 +├── ws2.py 134 行 rank 完整性 + cross-config partition/replica +├── paired_check.py 160 行 Torch paired check 架子(anchor_pending 门控) +└── synthetic_producer.py 152 行 合同 §5 授权的 seeded 确定性 producer + +scripts/check_p3.py 245 行 CLI:--cases/--ladders/--seed/--rows/--json + +tests/ +├── test_naive_topk6.py 109 行 7 测试 +├── test_p3_verdicts.py 90 行 6 测试 +├── test_p3_comparison.py 203 行 16 测试 +├── test_p3_validation_ladder.py 293 行 18 测试 +├── test_p3_negative.py 382 行 21 测试 +├── test_p3_ws2.py 210 行 12 测试 +└── test_p3_paired_check.py 149 行 11+1skip 测试 +``` + +共 13 个源文件 + 7 个测试文件,约 3500 行,244 个测试。 + +--- + +## 三、设计决策与理由(来龙去脉) + +以下按"问题 → 决策 → 为什么不那样做"展开,每条对应合同的明确条款。 + +### D1:naive_topk6 独立重写,不依赖 T01 + +**合同要求**(§4-T09):"独立重写 naive Top-6,仅交叉检查 T01 golden"。 + +**实现**:全序排序语义 `(q 降序, logical_expert_id 升序)`,槽位保持排序原序; +FP32 精确比较;`cross_check_topk6` 输出首个 `(row, slot)` 差异。 + +**为什么写成独立模块**:交叉检查的价值恰恰在"独立性"——如果复用 T01 的 +Top-6 实现来检查 T01 自己,同源错误会被系统性掩盖。所以哪怕 T01 的 +`stable_topk6_device_abi.v1` 将来发布了,这个 naive 实现也不删,它是对拍基准。 + +**测试设计**:random / near-tie(ULP 阶梯)/ exact-tie(全同行必须输出 id 0..5)。 +exact-tie 是关键——tie-break 政策 `(q 降序, id 升序)` 在全同行上退化为纯 id 升序, +这是最容易写错的地方。 + +### D2:状态码先行冻结(p3_verdicts.py) + +**合同要求**(§6):状态码表是 fail-closed 体系的骨架,"新增码只能追加,不得重排", +且规定了三段 band:device 1–2 / provider 10–22 / runner 50–72。 + +**实现**:`P3Verdict` IntEnum + 设备可写集合校验 + `primary_verdict` 固定优先级仲裁。 + +**为什么第一个写它**:所有后续模块(比较器、阶梯、CLI exit code)都要引用状态码。 +先冻结它,避免后面各模块各拿各的魔法数字。同时 `primary_verdict` 实现了合同 §6 +"同一 case 多个错误按固定顺序取 primary"——这个优先级逻辑如果散落在各 runner 里, +必然各处不一致。 + +### D3:比较器四级固定顺序,身份门禁先行(comparison.py) + +**合同要求**(§2.5):"比较顺序为 identity → discrete → score/weight → gradient; +任何 identity/schema/provenance/upstream verdict 缺失都停止比较"。 + +**实现**:`TraceComparator` 顺序走四个 stage,identity 漂移/缺失立即 halt—— +后续再调任何 stage 方法直接抛 `RuntimeError`(防误用,而不是静默跳过)。 + +**为什么 halt 要抛异常**:调用方如果没检查 halt 状态继续喂梯度进来,说明调用方 +逻辑有错。静默跳过会让"没比完"伪装成"比完了",这正是 fail-closed 要防的。 + +**stop verdicts 集合**:identity/schema/upstream/provenance 类全在 +`_STOP_VERDICTS` 里。后来(见 D8)把 `NON_FINITE` 也加了进去。 + +### D4:canonical 序列化与双哈希(fingerprint.py) + +**合同要求**(§2.4):semantic hash 按 per-token map、token 升序、padding 不进入; +artifact hash 按 `(case,config,rank)` 唯一、全部行含 padding + Envelope; +"不得只报 case hash"(必须能定位到 token 级)。 + +**实现**: +- `per_token_semantic_hashes` → dict[token, hash],头部含 identity/layer/mode, + 行按 slot 升序序列化 +- `route_semantic_hash` → case 级(token 升序折叠) +- `route_artifact_hash` → 全行含 padding + Envelope 字段 +- 显式版本 `p3-t09-canonical.v1` + +**为什么显式版本化**:合同 §8.1 规定 canonical hash 变化必须走 contract delta。 +版本字符串嵌进 hash 头部,将来 T05 发布正式 schema 时原位替换,两个版本的 +hash 天然不相等,不会被误当成"语义漂移"。 + +**为什么 padding 进 artifact 不进 semantic**:padding 合法地随 batch/pack 变化 +(L2 的测试就依赖这一点),语义层必须剔除;但 L1 同配置 repeat 里 padding +也必须逐字节稳定,所以它在 artifact hash 里被审计。这就是"padding 只由同配置 +artifact gate 审计"(§6 规则)的落地。 + +### D5:六元组定位与归因表(first_mismatch.py) + +**合同要求**(§4-T09):"按 (absolute_layer, site, pass, event_index, +global_token_id, rank) 定位首差分;输出 owner/Issue/boundary/phase/artifact"。 + +**实现**:`MismatchKey` NamedTuple 即六元组;`_ATTRIBUTION` 静态表把 site 映射到 +(owner, issue):score→T02/#41、hash_lookup→T03/#42、topk→T01/#43、 +selection→T04/#44、weight/handoff→T05、bwd→T06、tp_sp→T07、placement→T08。 + +**为什么用穷举静态表 + fail-closed**:site 集合是合同冻结的。未知 site 直接抛 +`UnknownSiteError`——"refusing to guess attribution"。归因错误比没有归因更糟: +它会把 bug 派给错误的任务,浪费整个团队的时间。同理,backward pass 只允许 +出现在梯度承载 site(score/bwd)上,出现在 topk 上说明 trace 本身就坏了。 + +### D6:四级阶梯的分工(ladder.py + report.py) + +| 阶梯 | 比什么 | 用哪个哈希 | 抓什么错 | +|------|-------|-----------|---------| +| L1 | 同配置 repeat | artifact hash(全字节) | 非确定性:kernel 竞态、未初始化内存、atomics 顺序 | +| L2 | batch/pack/padding/launch 扰动 | per-token semantic hash | 布局泄漏:语义随物理布局变化 | +| L3a | candidate vs bit-defined oracle | 逐行 byte-exact | 算子实现错误 | +| L3b | recorded 双引擎(Megatron vs Miles) | 四级比较器 | 引擎间分歧 | + +**L2 缺 token → INCOMPLETE_ARTIFACT(13)、多 token → AMBIGUOUS_GLOBAL_TOKEN_MAPPING(63)** +——直接对应合同 §2.4 的 runner 规则。 + +**report.py 的由来(自审产物)**:最初 `LadderReport` 和 `_fail`/`_pass` 写在 +ladder.py 里,ws2.py 和 paired_check.py 跨模块导入下划线私有函数——违反封装。 +重构抽出 `report.py`,公开 `make_fail`/`make_pass`,依赖方向变为单向: +`CLI → ladder/ws2/paired_check → report → 底层原语`。 + +### D7:WS2 的两种 token 所有权模式(ws2.py) + +**合同要求**(§2.4):tp 为 replica 或 partition_by_sequence、dp/cp 为 partition、 +pp 为 replica_by_layer、ep 不是 token 维度;缺失→MISSING_RANK/INCOMPLETE、 +越权或重复→AMBIGUOUS。 + +**实现**:`check_rank_completeness`(缺 rank→20,重复 rank→22 stale)+ +`run_ws2_cross_config`(partition:token 恰好出现一次;replica:可重复但每个 +载体的 hash 必须与 base 一致,首个 `(token, rank)` 定位)。 + +**为什么 replica 不去重比较**:TP 下同一 token 的多份拷贝如果只比一份, +另一份坏了就漏检。逐载体比对才能抓住"某个 rank 单独漂移"。 + +### D8:non-finite 前置门(自审补齐的关键缺口) + +**合同要求**(§2.5):"forward active 行的 z'、s、q、a、Z、p、w 非有限为 +NON_FINITE";§6:"非 PASS 的 non-finite fail-closed"。 + +**最初遗漏**:第一轮交付的比较器完全没有 non-finite 检查路径。后果很具体: +NaN 进 byte gate,`NaN != NaN` 会被误报成 `ROUTE_WEIGHT_BYTES_MISMATCH(51)`, +归因派给 T02 的"字节不等"——但真实缺陷是某处算出了 NaN,两者的修复路径完全不同。 + +**修复**:`_nonfinite_gate` 挂在 stage 3/4 的字节比较**之前**,任一侧 active 值 +非有限即 `NON_FINITE(1)` 且加入 `_STOP_VERDICTS` 停走。四条边界用测试钉死: +① NaN 不被误报为 bytes mismatch;② padding 行非有限不判(§2.5 只查 active, +有 mask 的用例验证豁免);③ P3 自身(1) 与 upstream(18) 两个 band 不混淆; +④ 梯度侧 dz 非有限归因 T06。 + +### D9:XOR 负向(同一次自审补齐) + +**合同要求**(§2.5):"Hash/Learned 是按 (absolute_layer, router_mode) 的 XOR" +——一个 layer 在同一时刻只能是一种模式,模式互斥。 + +**最初状态**:机制存在(identity gate 的 `router_mode` 字段、semantic hash 头部 +都含 mode),但没有显式负向用例。合同 §4-T09 验收行明确列了 XOR。 + +**补齐**:两个用例——同一 layer 两侧 trace 模式不一致必须 halt 在 identity gate +(`IDENTITY_DRIFT`,到不了数值阶段);仅翻转 mode 字段必须改变 per-token +semantic hash(否则 XOR 约束形同虚设),L2 以 59 拦截。 + +### D10:paired_check 架子与 anchor_pending 纪律 + +**合同要求**(§2.5):Torch 原始参考必须在每个正式 golden 上运行 paired check +并记录诊断;"其差异不能覆盖 strict verdict,缺证据为 MISSING_PROVENANCE(67)"。 + +**实现**:manifest 查找 `fixtures/p3/manifest.json`,不存在→空列表+ +`anchor_pending`。四条规则:paired diff 仅诊断永不翻转 verdict;缺执行证据→67 +(fail-closed,不是绿);Torch crash 也是缺证据不是 pass;gate 要求每个正式 +golden 都有证据槽位。T01 自带的 paired record 可短路。真实 manifest 集成测试 +用 `skipif` 门控,T01 发布后自动生效。 + +**为什么不猜测 manifest 路径**:合同 §1.3 "we look it up, we never guess"。 +一个写死的猜测路径如果恰好命中过时的缓存文件,会静默给出错误的"证据完整"结论。 + +### D11:CLI 的 exit code 语义与输出分流 + +**实现**:`check_p3.py` exit code = 首个失败 verdict 的数值码(全过为 0)—— +CI 里可以直接 `echo $?` 区分失败类别。`--json` 模式下结构化报告走 stdout、 +人读摘要走 stderr,互不污染(管道 `| jq` 不再被摘要行打断)。 + +**修掉的历史包袱**:初版在函数体内用 `__import__(..., fromlist=[...])` 动态 +导入 `MismatchKey`——纯 hack。清理为顶部静态导入。 + +--- + +## 四、测试矩阵(244 个测试的设计逻辑) + +| 文件 | 数量 | 钉死什么 | +|------|-----|---------| +| test_naive_topk6 | 7 | 全序语义、tie-break、交叉检查定位 | +| test_p3_verdicts | 6 | 状态码 band、追加不重排、primary 优先级 | +| test_p3_comparison | 16 | 四级顺序、halt 语义、±0.0 区分、padding 排除 | +| test_p3_validation_ladder | 18 | 四阶梯各自的 pass/fail 边界、CLI 参数化 | +| test_p3_negative | 21 | 单一缺陷注入→精确 verdict(见下) | +| test_p3_ws2 | 12 | partition/replica、缺/重 rank、(token,rank) 定位 | +| test_p3_paired_check | 11+1skip | 诊断不翻转、缺证据 67、manifest 容错 | + +**负向矩阵的坚持**:每个用例只注入**单一**缺陷。多缺陷混注会让"verdict 正确" +变成巧合(可能被另一个缺陷的 verdict 掩盖),单缺陷才能证明归因逻辑本身正确。 +另有控制组反向钉死:行序重排/padding 增删/Envelope 变化**必须**判 invariant +(防误报——验证基础设施自己的假阳性同样致命)。 + +39 个 skipped 全部是 GPU 门控用例(本机无 GPU),与逻辑无关。 + +**负向矩阵明细**(缺陷 → 期望 verdict → 拦截层): + +| 注入缺陷 | verdict | 拦截层 | +|---------|---------|--------| +| tie-break 违反(降序 id 打破平票) | 55/56 | cross_check / L3a | +| Hash/Learned 模式互斥违反(XOR) | 10 halt | identity gate | +| 仅翻转 router_mode 字段 | 59 | L2 semantic hash | +| weight bitflip(active 行) | 51 | L3a | +| score bitflip | 52 | L3a | +| semantic 字段改动 | 59 | L2 | +| artifact 字段改动(含 padding) | 60 | L1 | +| stale run/attempt metadata | 60 | L1 | +| missing provenance(identity 字段缺失) | 67 halt | identity gate | +| identity drift(checkpoint/weight 变化) | 10 halt | 优先于后续阶段 | +| L2 丢 token | 13 | L2 | +| L2 幽灵 token | 63 | L2 | +| 禁止的 silent fallback 标志 | 66 halt | provenance | +| selection 梯度泄漏 | 53 | L3b(归因 T06) | +| active 值 NaN/Inf | 1 halt | non-finite 前置门 | +| padding 增删试图掩盖 Core 改动 | 59 | L2(控制组同时钉死合法 padding 不误报) | + +--- + +## 五、自审与迭代记录 + +这个交付经过一轮显式自审("当前所有的修改全部结束了?自己审查一下代码"), +发现并修复了三个结构问题、两个合同覆盖缺口: + +| # | 类型 | 问题 | 处置 | +|---|------|------|------| +| 1 | 结构 | `LadderReport`/`_fail`/`_pass` 定义在 ladder.py,ws2/paired_check 跨模块导入私有函数 | 抽出 report.py,公开 make_pass/make_fail | +| 2 | 结构 | CLI 里 `__import__` 动态导入 hack | 改顶部静态导入 | +| 3 | 结构 | `--json` 模式摘要行污染 stdout,管道解析崩 | JSON 走 stdout、摘要走 stderr | +| 4 | 覆盖 | 合同负向清单里的 XOR 无显式用例 | +2 测试(D9) | +| 5 | 覆盖 | 合同负向清单里的 non-finite 完全没有比较器路径 | `_nonfinite_gate` + NON_FINITE 入停走集合,+4 测试(D8) | + +这次自审的价值在于第 5 条:不看合同原文逐字核对,"NaN 会被误报成 bytes +mismatch"这种归因错误永远不会被测试暴露——测试全绿,但绿的是错误的 verdict。 + +--- + +## 六、边界与未做之事(防跑偏声明) + +**T09 只做验证基础设施,不写 router 算子**(那是 T02–T04/T06 的活)。 +具体边界: + +- `naive_topk6.py` 是对拍基准,不是生产实现——不进任何 provider 路径 +- `fingerprint.py` 的 `p3-t09-canonical.v1` 是占位 canonical 序列化, + T05 发布正式 schema 后**原位替换**(合同 §4-T05:"禁止维护第二份字段表") +- `synthetic_producer.py` 只用 seeded 确定性数据,符合 §7 DoD + "仅使用 synthetic/sanitized fixture" +- 不改 `rl_engine/kernels/gtest/` 等他人代码,只读复用模式 + +**三项外部依赖待办**(均被 T01 阻塞,合同允许的 anchor_pending 状态): + +1. T01 发布 golden manifest 后,paired_check 的 skipif 集成测试自动生效 +2. T05 发布正式 schema 后,替换 fingerprint.py 的序列化并升版本号 +3. 认领公示(合同 §8.3 群内回复格式)需人工确认 + +**已知邻近风险**:`origin/dsv4-p5-dev` 分支的 P5 starter 已提交 +`rl_engine/moe/{contract,oracle,fixtures,...}.py`,与 T09 的文件零重叠但共享 +`rl_engine/moe/` 目录。T09 的 `__init__.py` 只有 10 行包声明、不导出任何符号, +合并时冲突风险极低;若 P5 先进 main,T09 侧只需保留对方 `__init__.py`。 + +--- + +## 七、复现命令 + +```bash +# 全部 T09 测试(含 naive_topk6 与全部 p3 相关) +cd /workspace/RL-Kernel +python -m pytest tests/ -q -k "topk or p3" +# → 205 passed, 39 skipped in ~17s + +# 单文件 +python -m pytest tests/test_p3_negative.py -q # 21 passed +python -m pytest tests/test_p3_validation_ladder.py -q # 18 passed + +# CLI 冒烟(2 case × 4 ladder) +python scripts/check_p3.py --cases smoke +# → check_p3: cases=2 ladders=L1,L2,L3a,L3b passed=8/8 exit=0 + +# CLI 结构化输出(stdout 纯 JSON,可管道) +python scripts/check_p3.py --cases smoke --json | jq '.[0].ladder' + +# exit code 即首个失败 verdict 码(CI 可用 $? 区分失败类别) +python scripts/check_p3.py --ladders L1; echo $? +``` + +--- + +## 八、模块依赖图 + +``` +scripts/check_p3.py (CLI) + │ + ▼ +ladder.py ── ws2.py ── paired_check.py ← 三个 runner 层 + │ │ + └────┬───────┘ + ▼ + report.py ← LadderReport 唯一构造点 + │ + ▼ +comparison.py (TraceComparator, 四级) ← 比较原语 + first_mismatch.py (六元组+归因) ← 定位原语 + fingerprint.py (semantic/artifact hash) ← 序列化原语 + │ + ▼ + p3_verdicts.py ← 状态码冻结(最底层,无依赖) + +synthetic_producer.py → ladder/ws2 (fixture) +naive_topk6.py → 独立,仅被测试与交叉检查引用 +``` + +无循环依赖;每层只向下依赖。`p3_verdicts.py` 位于最底层且被所有层引用—— +这正是 D2"先行冻结"的 structural 体现。 + +--- + +## 九、合同验收项 → 代码 → 测试 对照表(review 导航) + +review 时按此表逐项核对,左列即合同 §4-T09 验收行的原文要点。 + +| 合同验收项 | 代码位置 | 测试位置 | +|-----------|---------|---------| +| 阶梯 L1 repeat | `ladder.py:48 run_l1_repeat` | `test_p3_validation_ladder.py` | +| 阶梯 L2 invariance | `ladder.py:75 run_l2_invariance` | 同上 | +| 阶梯 L3a oracle | `ladder.py:116 run_l3a_oracle` | 同上 | +| 阶梯 L3b Megatron-vs-Miles | `ladder.py:172 run_l3b_dual_engine` | 同上 | +| WS2 rank 完整性(缺 20/重 22) | `ws2.py:43 check_rank_completeness` | `test_p3_ws2.py` | +| WS2 cross-config partition/replica | `ws2.py:73 run_ws2_cross_config` | 同上 | +| 独立重写 naive Top-6 | `naive_topk6.py:95 naive_topk6` | `test_naive_topk6.py` | +| 交叉检查首差分 (row,slot) | `naive_topk6.py:126 cross_check_topk6` | 同上 | +| 状态码冻结 + primary 仲裁 | `p3_verdicts.py:107 primary_verdict` | `test_p3_verdicts.py` | +| canonical semantic/artifact 双哈希 | `fingerprint.py:148/169/217` | `test_p3_validation_ladder.py` | +| 六元组首差分 + 归因 | `first_mismatch.py:125 first_mismatch` | `test_p3_comparison.py` | +| 四级顺序比较器 | `comparison.py:136 TraceComparator` | 同上 | +| non-finite 前置门 | `comparison.py:233 _nonfinite_gate` | `test_p3_negative.py` | +| paired check(诊断不翻转/缺证据 67) | `paired_check.py:92 run_paired_check` | `test_p3_paired_check.py` | +| anchor_pending manifest 门控 | `paired_check.py:67 load_golden_manifest` | 同上 | +| 负向矩阵(8 类缺陷→精确 verdict) | —(注入在测试内构造) | `test_p3_negative.py` | + +--- + +## 十、公开 API 一览(按模块) + +**`naive_topk6.py`**:`naive_topk6(q) -> (ids, weights)` · +`cross_check_topk6(candidate, expected) -> 首差 (row, slot) | None` + +**`p3_verdicts.py`**:`P3Verdict`(IntEnum)· `primary_verdict(list) -> P3Verdict | None` · +`is_valid_device_status(int)` · `classify_writable_band(int)` + +**`fingerprint.py`**:`route_semantic_hash(rows, identity, layer) -> str` · +`per_token_semantic_hashes(...) -> dict[token, hash]` · +`route_artifact_hash(artifact) -> str`;数据类 `RouteRow / RouteIdentity / +EnvelopeFields / Artifact` + +**`first_mismatch.py`**:`first_mismatch(lhs_events, rhs_events) -> FirstMismatch | None`; +`MismatchKey`(六元组 NamedTuple);异常 `UnknownSiteError` + +**`comparison.py`**:`TraceComparator`(`check_identity / check_discrete / +check_score_weight / check_gradient / report`);独立函数 `tensor_byte_exact / +discrete_equal` + +**`ladder.py`**:`run_l1_repeat / run_l2_invariance / run_l3a_oracle / +run_l3b_dual_engine`,全部返回 `LadderReport` + +**`ws2.py`**:`check_rank_completeness / run_ws2_cross_config` + +**`paired_check.py`**:`load_golden_manifest / run_paired_check / +paired_gate_for_goldens` + +**`report.py`**:`LadderReport` · `make_pass / make_fail`(唯一构造点) + +**CLI `check_p3.py`**:参数 `--cases/--ladders/--seed/--rows/--json`; +exit code = 首个失败 verdict 码(全过 0) + diff --git a/rl_engine/moe/__init__.py b/rl_engine/moe/__init__.py new file mode 100644 index 00000000..f1f4b9ab --- /dev/null +++ b/rl_engine/moe/__init__.py @@ -0,0 +1,10 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 RL-Kernel Contributors + +"""P3 router package (DSV4-Flash MoE router, contract `p3-task-selection.md`). + +Layout sanctioned by the P3 contract §2.1: +- ``naive_topk6`` : T09-owned total-order Top-6 checker (cross-checks T01 golden) +- ``router_torch_reference.py`` : T01-owned raw Torch reference (to be published + with the start kit; do not duplicate or silently replace it) +""" diff --git a/rl_engine/moe/naive_topk6.py b/rl_engine/moe/naive_topk6.py new file mode 100644 index 00000000..08aa50f2 --- /dev/null +++ b/rl_engine/moe/naive_topk6.py @@ -0,0 +1,163 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 RL-Kernel Contributors + +"""T09-owned naive total-order Top-6 (contract §2.1 / §4-T09 / fixture table §3.3). + +Purpose +------- +This is the *independent* re-implementation of stable Top-6 used **only** to +cross-check T01's ``stable_topk6`` golden. It is deliberately written in the +most naive way possible — a full total-order ``sorted()`` over all E experts — +so that any race, tie-mishandling or post-tie reorder inside faster +implementations shows up as a diff here. + +Ownership (contract §3.3): ``naive total-order Top-6`` is owned by **T09** and +consumed by **T01**. It must NOT be used as a production selection kernel. + +Semantics frozen by the contract +-------------------------------- +* Order key: ``(q descending, logical_expert_id ascending)`` (§2.6: the P3 + canonical tie-break when Miles has no explicit deterministic tie). +* Slot order: slots 0..5 keep the order produced by the total sort; per §2.1 + Learned selection, ``ids = stable_topk6(q)`` with q descending and + logical_expert_id ascending — six table slots keep original order, no + re-sort / re-topk / reorder afterwards (that rule is stated for Hash but the + same slot-order guarantee applies to the Top-6 output consumed by T04). +* Ties: a full total order handles exact ties naturally; near-ties (distinct + FP32 q values that differ only in low bits) must still sort by value first. +* Padding rows are not selected here — the caller (assembler) owns padding + canonicalization; this module never sees padding rows. + +This module is CPU/FP32 pure-Python by design (auditability over speed): it +operates on Python floats converted from FP32 so the ordering decisions are +made on the exact FP32 bit patterns, not on double-precision artefacts. +""" + +from __future__ import annotations + +from typing import NamedTuple + +import torch + +K: int = 6 # contract §2.1: K=6 fixed for DSV4-Flash +DEFAULT_E: int = 256 # contract §2.1: E=256 + + +class NaiveTopk6Result(NamedTuple): + """Total-order Top-6 output for one row. + + Attributes: + ids: ``INT32 [K]`` logical expert ids in canonical order + (q desc, logical_expert_id asc). + values: ``FP32 [K]`` the q values corresponding to ``ids``. + """ + + ids: list[int] + values: list[float] + + +def _fp32_key(value: float) -> tuple[int, float]: + """Return a sort key that orders FP32 values exactly as FP32 compares. + + Python floats are doubles; two distinct FP32 values remain distinct and + correctly ordered as doubles (FP32 -> double is exact), so a plain + descending value sort is already faithful to FP32 semantics. We keep this + helper to make the intent explicit and to give a single place to audit. + """ + return (0, value) + + +def naive_topk6_row(q_row: torch.Tensor) -> NaiveTopk6Result: + """Total-order Top-6 for a single row of q (FP32 ``[E]``).""" + if q_row.ndim != 1: + raise ValueError(f"q_row must be 1-D, got shape {tuple(q_row.shape)}") + if q_row.numel() == 0: + raise ValueError("q_row must be non-empty") + if q_row.dtype != torch.float32: + raise ValueError(f"q_row must be FP32, got {q_row.dtype}") + + # Work on exact FP32 values pulled out as doubles (lossless widening). + values = q_row.tolist() + e = len(values) + if e < K: + raise ValueError(f"need at least K={K} experts, got E={e}") + + # Total order: (q desc, logical_expert_id asc). A single sorted() pass over + # (value, id) pairs with the id as the ascending tie-break achieves this. + order = sorted(range(e), key=lambda i: (-values[i], i)) + top = order[:K] + return NaiveTopk6Result( + ids=[int(i) for i in top], + values=[float(values[i]) for i in top], + ) + + +def naive_topk6(q: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]: + """Batch total-order Top-6. + + Args: + q: FP32 ``[T, E]`` selection scores (``q = s + correction_bias``). + + Returns: + ``(ids, top_values)`` where ``ids`` is INT32 ``[T, K]`` and + ``top_values`` is FP32 ``[T, K]``, both in canonical order + (q descending, logical_expert_id ascending), slots keep sort order. + """ + if q.ndim != 2: + raise ValueError(f"q must be 2-D [T,E], got shape {tuple(q.shape)}") + if q.dtype != torch.float32: + raise ValueError(f"q must be FP32, got {q.dtype}") + + t, e = q.shape + if e < K: + raise ValueError(f"need at least K={K} experts, got E={e}") + + rows = q.tolist() # exact FP32 -> double widening per element + ids = torch.empty((t, K), dtype=torch.int32) + top_values = torch.empty((t, K), dtype=torch.float32) + for r, row in enumerate(rows): + order = sorted(range(e), key=lambda i: (-row[i], i)) + for slot, idx in enumerate(order[:K]): + ids[r, slot] = idx + top_values[r, slot] = row[idx] + return ids, top_values + + +def cross_check_topk6( + candidate_ids: torch.Tensor, + q: torch.Tensor, +) -> tuple[bool, str]: + """Cross-check a candidate Top-6 implementation against the naive order. + + This is the T09 -> T01 cross-check entry point: T01 runs its + ``stable_topk6`` fixtures (random / near-tie / exact-tie) through this + checker; any mismatch is a defect in the candidate, never in this module. + + Args: + candidate_ids: INT32 ``[T, K]`` ids produced by the implementation + under test (slot order must be its own output order). + q: FP32 ``[T, E]`` the exact scores the candidate was invoked with. + + Returns: + ``(passed, message)``; ``message`` pinpoints the first mismatching + ``(row, slot)`` with both id sequences — first-mismatch style, per + contract §4-T09 (first mismatch must be locatable, not averaged away). + """ + if candidate_ids.shape != q.shape[:-1] + (K,): + return False, ( + f"shape mismatch: candidate_ids {tuple(candidate_ids.shape)} " + f"vs expected {tuple(q.shape[:-1])}x{K}" + ) + + expected_ids, _ = naive_topk6(q) + t = q.shape[0] + cand = candidate_ids.to(torch.int64).tolist() + exp = expected_ids.to(torch.int64).tolist() + for r in range(t): + if cand[r] != exp[r]: + slot = next(i for i in (range(K)) if cand[r][i] != exp[r][i]) + return False, ( + f"first mismatch at (row={r}, slot={slot}): " + f"candidate={cand[r]} expected={exp[r]}" + ) + return True, "ok" diff --git a/rl_engine/moe/p3_verdicts.py b/rl_engine/moe/p3_verdicts.py new file mode 100644 index 00000000..aa824ed2 --- /dev/null +++ b/rl_engine/moe/p3_verdicts.py @@ -0,0 +1,151 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 RL-Kernel Contributors + +"""P3 unified fail-closed verdict codes (contract §6, frozen). + +Rules frozen by the contract: +- ``P3Verdict`` is INT32. Device may write only 1-2 (3-9 reserved), provider + 10-22, runner 50-72, certification 90-99 reserved. New codes can only be + appended, never re-ordered. +- ``PASS(0)`` is a single-operator verdict; ``CASE_PASS(50)`` is case-level. + The two must not be conflated. +- Operator APIs never return runner/certification codes; only the case runner + may produce ``CASE_PASS``. +- Multi-error cases pick the primary verdict by fixed priority: + infrastructure integrity / identity / schema -> upstream evidence -> + discrete plan -> numeric bytes -> fingerprint -> diagnostics. The runner + must not rewrite this priority. +""" + +from __future__ import annotations + +from enum import IntEnum + + +class P3Verdict(IntEnum): + """Unified fail-closed verdict codes (contract §6).""" + + # --- provider layer (operator verdicts) --- + PASS = 0 # launch/readback/echo/status all clean + # device-writable (kernel may only atomicMin these two) + NON_FINITE = 1 # P3's own computation produced non-finite + HASH_TABLE_INDEX_OUT_OF_RANGE = 2 + # provider layer + IDENTITY_DRIFT = 10 + SCHEMA_MISMATCH = 11 + CORRUPT_ARTIFACT = 12 + INCOMPLETE_ARTIFACT = 13 + LOGIT_ROUND_POINT_MISMATCH = 14 + HASH_TABLE_MISMATCH = 15 + UNSUPPORTED_CAPABILITY = 16 + ZERO_ACTIVE_TOKENS = 17 + UPSTREAM_NON_FINITE = 18 + GATE_SHARDING_MISMATCH = 19 + MISSING_RANK = 20 + PRE_UPDATE_WEIGHT_DRIFT = 21 + STALE_RUN_METADATA = 22 + + # --- runner layer (case verdicts) --- + CASE_PASS = 50 + ROUTE_WEIGHT_BYTES_MISMATCH = 51 + SCORE_BYTES_MISMATCH = 52 + GRADIENT_BYTES_MISMATCH = 53 + BYTE_MISMATCH = 54 + TOPK_ORDER_MISMATCH = 55 + TIE_BREAK_POLICY_MISMATCH = 56 + INVALID_DISCRETE_PLAN = 57 + INVALID_PROFILE = 58 + ROUTE_SEMANTIC_FINGERPRINT_MISMATCH = 59 + ROUTE_ARTIFACT_FINGERPRINT_MISMATCH = 60 + SELECTION_GRADIENT_PRESENT = 61 + FORBIDDEN_LOCAL_SHARD_TOPK = 62 + AMBIGUOUS_GLOBAL_TOKEN_MAPPING = 63 + INVALID_PLACEMENT_MAP = 64 + PLACEMENT_MAP_VERSION_MISMATCH = 65 + SILENT_FALLBACK = 66 + MISSING_PROVENANCE = 67 + MISSING_BOUNDARY_TRACE = 68 + UPSTREAM_CONTRACT_MISMATCH = 69 + UPSTREAM_VERDICT_MISSING = 70 + UPSTREAM_EVIDENCE_MISSING = 71 + NATURAL_ROUTE_MISMATCH = 72 + + +#: Device-writable status values (kernel may only write these via atomicMin). +DEVICE_WRITABLE = frozenset({P3Verdict.NON_FINITE, P3Verdict.HASH_TABLE_INDEX_OUT_OF_RANGE}) + +#: Values reserved for future device use (3-9); anything else written by a +#: kernel is CORRUPT_ARTIFACT per contract §6. +DEVICE_RESERVED_RANGE = range(3, 10) + +#: Provider-writable band. +PROVIDER_RANGE = range(10, 23) + +#: Runner-writable band. +RUNNER_RANGE = range(50, 73) + + +def is_valid_device_status(value: int) -> bool: + """True iff a kernel-written device status value is legal (1, 2).""" + try: + return P3Verdict(value) in DEVICE_WRITABLE + except ValueError: + return False + + +def classify_writable_band(value: int) -> str: + """Return which layer owns ``value``; used to police layer violations.""" + if value == 0 or value in DEVICE_WRITABLE: + return "device-or-provider" + if value in PROVIDER_RANGE: + return "provider" + if value in RUNNER_RANGE: + return "runner" + return "reserved" + + +def primary_verdict(verdicts: list[P3Verdict]) -> P3Verdict | None: + """Pick the primary verdict among multiple failures (contract §6 order). + + Fixed priority: infrastructure integrity / identity / schema -> + upstream evidence -> discrete plan -> numeric bytes -> fingerprint -> + diagnostics. Implementation: explicit rank map, stable for unknown codes + (they rank last, preserving input order via ``sorted`` stability). + """ + if not verdicts: + return None + + rank: dict[P3Verdict, int] = {} + order = [ + # infrastructure integrity / identity / schema + [P3Verdict.CORRUPT_ARTIFACT, P3Verdict.IDENTITY_DRIFT, P3Verdict.SCHEMA_MISMATCH, + P3Verdict.STALE_RUN_METADATA, P3Verdict.INCOMPLETE_ARTIFACT], + # upstream evidence + [P3Verdict.UPSTREAM_VERDICT_MISSING, P3Verdict.UPSTREAM_EVIDENCE_MISSING, + P3Verdict.UPSTREAM_CONTRACT_MISMATCH, P3Verdict.UPSTREAM_NON_FINITE, + P3Verdict.MISSING_PROVENANCE, P3Verdict.MISSING_BOUNDARY_TRACE, + P3Verdict.MISSING_RANK], + # discrete plan + [P3Verdict.INVALID_DISCRETE_PLAN, P3Verdict.TOPK_ORDER_MISMATCH, + P3Verdict.TIE_BREAK_POLICY_MISMATCH, P3Verdict.FORBIDDEN_LOCAL_SHARD_TOPK, + P3Verdict.AMBIGUOUS_GLOBAL_TOKEN_MAPPING, P3Verdict.INVALID_PLACEMENT_MAP, + P3Verdict.PLACEMENT_MAP_VERSION_MISMATCH, P3Verdict.HASH_TABLE_MISMATCH, + P3Verdict.LOGIT_ROUND_POINT_MISMATCH, P3Verdict.GATE_SHARDING_MISMATCH, + P3Verdict.INVALID_PROFILE], + # numeric bytes + [P3Verdict.ROUTE_WEIGHT_BYTES_MISMATCH, P3Verdict.SCORE_BYTES_MISMATCH, + P3Verdict.GRADIENT_BYTES_MISMATCH, P3Verdict.BYTE_MISMATCH, + P3Verdict.SELECTION_GRADIENT_PRESENT, P3Verdict.SILENT_FALLBACK, + P3Verdict.NON_FINITE, P3Verdict.HASH_TABLE_INDEX_OUT_OF_RANGE], + # fingerprint + [P3Verdict.ROUTE_SEMANTIC_FINGERPRINT_MISMATCH, + P3Verdict.ROUTE_ARTIFACT_FINGERPRINT_MISMATCH, + P3Verdict.PRE_UPDATE_WEIGHT_DRIFT], + # diagnostics + [P3Verdict.NATURAL_ROUTE_MISMATCH], + ] + for group_rank, group in enumerate(order): + for v in group: + rank[v] = group_rank + + return sorted(verdicts, key=lambda v: rank.get(v, len(order)))[0] diff --git a/rl_engine/moe/validation/__init__.py b/rl_engine/moe/validation/__init__.py new file mode 100644 index 00000000..ec7e7999 --- /dev/null +++ b/rl_engine/moe/validation/__init__.py @@ -0,0 +1,12 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 RL-Kernel Contributors + +"""T09 comparison engine for P3 router validation (contract §2.5 / §4-T09). + +Submodules: +- ``first_mismatch`` : locate the first divergence between two event streams + by the six-tuple ``(absolute_layer, site, pass, event_index, + global_token_id, rank)`` and attribute it to an owner task / issue. +- ``comparison`` : the four-stage ordered comparator + (identity -> discrete -> score/weight -> gradient) with fail-closed gates. +""" diff --git a/rl_engine/moe/validation/comparison.py b/rl_engine/moe/validation/comparison.py new file mode 100644 index 00000000..9adf0f37 --- /dev/null +++ b/rl_engine/moe/validation/comparison.py @@ -0,0 +1,340 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 RL-Kernel Contributors + +"""Four-stage ordered comparator for P3 router traces (contract §2.5). + +Frozen comparison rules implemented here: + +1. Stage order is fixed: **identity -> discrete -> score/weight -> gradient**. + Any identity / schema / provenance / upstream-verdict failure STOPS all + later numeric attribution (fail-closed); a numeric mismatch found before an + identity error is never reported, because the identity error explains it. +2. Comparison modes per field class (§2.5): + - identity / layer / branch / slot / expert id / Top-K order / tie / + valid / capacity / canonical key -> ``exact_discrete`` + - active route weight and declared byte-gate score/gradient -> raw + ``byte_exact`` + - engine-vs-engine must be byte-exact; oracle-vs-CUDA defaults byte-exact, + non-bit-defined fields need a frozen ``ulp_bounded(k)`` delta. +3. Strict verdicts are never overridden by tolerance or averaged errors + (T09 DoD: "strict mismatch 不被 tolerance/平均误差隐藏"). +4. Padding rows never enter numeric gates; they are audited only by the + same-config artifact gate (L1), which lives in the ladder/CLI, not here. +""" + +from __future__ import annotations + +from dataclasses import dataclass, field +from typing import Any + +import torch + +from rl_engine.moe.p3_verdicts import P3Verdict, primary_verdict +from rl_engine.moe.validation.first_mismatch import ( + FirstMismatch, + MismatchKey, + TraceEvent, + first_mismatch, +) + +# --- stage identifiers (contract §2.5 order) ------------------------------- +STAGE_IDENTITY = "identity" +STAGE_DISCRETE = "discrete" +STAGE_SCORE_WEIGHT = "score_weight" +STAGE_GRADIENT = "gradient" + +STAGE_ORDER: tuple[str, ...] = ( + STAGE_IDENTITY, STAGE_DISCRETE, STAGE_SCORE_WEIGHT, STAGE_GRADIENT, +) + +# verdicts that stop the walk entirely (infrastructure/identity/schema/upstream +# plus fail-closed device defects: a non-finite P3 value makes later numeric +# attribution meaningless — §6 "非 PASS 的 non-finite fail-closed") +_STOP_VERDICTS: frozenset[P3Verdict] = frozenset({ + P3Verdict.NON_FINITE, + P3Verdict.IDENTITY_DRIFT, + P3Verdict.SCHEMA_MISMATCH, + P3Verdict.STALE_RUN_METADATA, + P3Verdict.INCOMPLETE_ARTIFACT, + P3Verdict.CORRUPT_ARTIFACT, + P3Verdict.UPSTREAM_CONTRACT_MISMATCH, + P3Verdict.UPSTREAM_VERDICT_MISSING, + P3Verdict.UPSTREAM_EVIDENCE_MISSING, + P3Verdict.MISSING_PROVENANCE, + P3Verdict.MISSING_BOUNDARY_TRACE, + P3Verdict.MISSING_RANK, +}) + +# stage -> runner verdict when a discrete mismatch lands there +_DISCRETE_VERDICTS: dict[str, P3Verdict] = { + "topk_ids": P3Verdict.TOPK_ORDER_MISMATCH, + "tie_break": P3Verdict.TIE_BREAK_POLICY_MISMATCH, + "table_slot": P3Verdict.INVALID_DISCRETE_PLAN, + "logical_expert_id": P3Verdict.INVALID_DISCRETE_PLAN, + "valid": P3Verdict.INVALID_DISCRETE_PLAN, +} + +# stage -> runner verdict when a byte gate fails there +_BYTE_VERDICTS: dict[str, P3Verdict] = { + "route_weight": P3Verdict.ROUTE_WEIGHT_BYTES_MISMATCH, + "score": P3Verdict.SCORE_BYTES_MISMATCH, + "gradient": P3Verdict.GRADIENT_BYTES_MISMATCH, +} + + +@dataclass +class StageResult: + """Outcome of one comparison stage.""" + + stage: str + passed: bool + verdict: P3Verdict | None # None when passed + mismatch: FirstMismatch | None # located first divergence, if any + notes: list[str] = field(default_factory=list) + + +@dataclass +class ComparisonReport: + """Full ordered-comparison result for one case.""" + + case_id: str + stages: list[StageResult] + stopped_early: bool # identity/schema/upstream gate halted walk + + @property + def passed(self) -> bool: + return bool(self.stages) and all(s.passed for s in self.stages) + + @property + def primary(self) -> P3Verdict | None: + failures = [s.verdict for s in self.stages if not s.passed and s.verdict] + return primary_verdict(failures) + + def summary_line(self) -> str: + flag = "PASS" if self.passed else f"FAIL({self.primary.name if self.primary else '?'})" + walked = "->".join( + f"{s.stage}:{'ok' if s.passed else s.verdict.name if s.verdict else 'FAIL'}" + for s in self.stages + ) + return f"case={self.case_id} {flag} [{walked}]{' STOP' if self.stopped_early else ''}" + + +def tensor_byte_exact(lhs: torch.Tensor, rhs: torch.Tensor) -> bool: + """True iff two tensors have identical dtype/shape/bytes (incl. -0.0 vs 0.0).""" + if lhs.dtype != rhs.dtype or lhs.shape != rhs.shape: + return False + return bool( + torch.equal(lhs.view(torch.uint8).flatten(), rhs.view(torch.uint8).flatten()) + ) + + +def discrete_equal(lhs: Any, rhs: Any) -> bool: + """``exact_discrete`` for ints/enums/strings/bools and lists thereof.""" + return lhs == rhs + + +class TraceComparator: + """Ordered four-stage comparator over two normalized trace streams. + + Usage (mirrors how ``check_p3`` L3a/L3b will drive it):: + + cmp = TraceComparator(case_id="case_x") + cmp.check_identity(lhs_meta, rhs_meta) # stops on drift + cmp.check_discrete(lhs_events, rhs_events) # ids/order/tie/slots + cmp.check_score_weight(lhs_w, rhs_w) # byte gates + cmp.check_gradient(lhs_dz, rhs_dz) # byte gates + report = cmp.report() + """ + + def __init__(self, case_id: str, *, phase: str = "WS1") -> None: + self._case_id = case_id + self._phase = phase + self._stages: list[StageResult] = [] + self._stopped = False + + # -- helpers ----------------------------------------------------------- + + def _append(self, result: StageResult) -> None: + if self._stopped: + raise RuntimeError("comparator already halted; later stages are unreachable") + self._stages.append(result) + if not result.passed and result.verdict in _STOP_VERDICTS: + self._stopped = True + + def _halted_gate(self, stage: str, verdict: P3Verdict, note: str) -> None: + self._append(StageResult(stage=stage, passed=False, verdict=verdict, + mismatch=None, notes=[note])) + + # -- stage 1: identity --------------------------------------------------- + + _IDENTITY_FIELDS = ( + "case_id", "checkpoint_id", "weight_id", "absolute_layer", + "router_mode", "table_fingerprint", "bias_fingerprint", + "logit_round_point", "tie_break_policy", "capacity_policy", + ) + + def check_identity(self, lhs_meta: dict[str, Any], rhs_meta: dict[str, Any]) -> None: + """Stage 1: identity/schema/provenance gate. Any drift halts the walk.""" + missing = [k for k in self._IDENTITY_FIELDS if k not in lhs_meta or k not in rhs_meta] + if missing: + self._halted_gate( + STAGE_IDENTITY, P3Verdict.MISSING_PROVENANCE, + f"identity fields missing on one side: {missing}", + ) + return + drifted = [k for k in self._IDENTITY_FIELDS if lhs_meta[k] != rhs_meta[k]] + if drifted: + self._halted_gate( + STAGE_IDENTITY, P3Verdict.IDENTITY_DRIFT, + f"identity fields differ: { {k: (lhs_meta[k], rhs_meta[k]) for k in drifted} }", + ) + return + self._append(StageResult(stage=STAGE_IDENTITY, passed=True, verdict=None, mismatch=None)) + + # -- stage 2: discrete --------------------------------------------------- + + def check_discrete( + self, + lhs_events: list[TraceEvent], + rhs_events: list[TraceEvent], + ) -> None: + """Stage 2: Top-6 ids/order, tie policy, table slots, valid flags.""" + result = first_mismatch(lhs_events, rhs_events, phase=self._phase, + artifact_name="discrete_trace") + if result.found: + verdict = _DISCRETE_VERDICTS.get( + result.detail_kind if hasattr(result, "detail_kind") else "logical_expert_id", + P3Verdict.INVALID_DISCRETE_PLAN, + ) + self._append(StageResult(stage=STAGE_DISCRETE, passed=False, + verdict=verdict, mismatch=result)) + return + self._append(StageResult(stage=STAGE_DISCRETE, passed=True, verdict=None, mismatch=None)) + + # -- stage 3: score / weight bytes --------------------------------------- + + # -- non-finite gate (§2.5: forward active values must be finite) -------- + + @staticmethod + def _active_non_finite(t: torch.Tensor, active_mask: torch.Tensor | None, + name: str) -> int | None: + """Index of the first active non-finite element, or None (§2.5). + + ``UPSTREAM_NON_FINITE`` is a *provider* verdict for upstream z / + dweights; here both sides are P3-computed trace values, so a + non-finite on either side is P3's own ``NON_FINITE``. + """ + t = t.detach().to(torch.float32) + if active_mask is not None: + t = t[active_mask.to(torch.bool)] + bad = torch.nonzero(~torch.isfinite(t)).flatten() + return int(bad[0].item()) if bad.numel() else None + + def _nonfinite_gate(self, lhs: torch.Tensor, rhs: torch.Tensor, *, + active_mask: torch.Tensor | None, site: str, + owner: str, issue: str) -> bool: + """Returns True when clean; appends a NON_FINITE stage on failure.""" + for side, tensor in (("lhs", lhs), ("rhs", rhs)): + idx = self._active_non_finite(tensor, active_mask, site) + if idx is not None: + key = MismatchKey(absolute_layer=-1, site=site, + pass_direction="forward", event_index=idx, + global_token_id=-1, rank=-1) + fm = FirstMismatch( + found=True, key=key, owner=owner, issue=issue, + boundary="non-finite gate", phase=self._phase, + artifact=f"{site}_values", + detail=f"{side} {site} has non-finite value at active " + f"flat index {idx} (§2.5 fail-closed)", + ) + self._append(StageResult(stage=STAGE_SCORE_WEIGHT, passed=False, + verdict=P3Verdict.NON_FINITE, + mismatch=fm)) + return False + return True + + def check_score_weight( + self, + lhs_weights: torch.Tensor, + rhs_weights: torch.Tensor, + *, + lhs_scores: torch.Tensor | None = None, + rhs_scores: torch.Tensor | None = None, + active_mask: torch.Tensor | None = None, + ) -> None: + """Stage 3: active-row byte gates on route weights (and scores). + + A non-finite active value on either side is ``NON_FINITE`` and + fails closed *before* any byte comparison (§2.5); the walk stops so + the defect is never averaged away by later numeric stages. + """ + if not self._nonfinite_gate(lhs_weights, rhs_weights, + active_mask=active_mask, site="weight", + owner="T02", issue="#41"): + return + if lhs_scores is not None and rhs_scores is not None: + if not self._nonfinite_gate(lhs_scores, rhs_scores, + active_mask=active_mask, site="score", + owner="T02", issue="#41"): + return + if active_mask is not None: + keep = active_mask.to(torch.bool) + lhs_weights = lhs_weights[keep] + rhs_weights = rhs_weights[keep] + if lhs_scores is not None and rhs_scores is not None: + lhs_scores = lhs_scores[keep] + rhs_scores = rhs_scores[keep] + + mismatches: list[str] = [] + if not tensor_byte_exact(lhs_weights, rhs_weights): + mismatches.append("route_weight bytes differ on active rows") + + if lhs_scores is not None and rhs_scores is not None: + if not tensor_byte_exact(lhs_scores, rhs_scores): + mismatches.append("score bytes differ on active rows") + + if mismatches: + key = MismatchKey(absolute_layer=-1, site="weight", pass_direction="forward", + event_index=-1, global_token_id=-1, rank=-1) + fm = FirstMismatch(found=True, key=key, owner="T02", issue="#41", + boundary="byte gate", phase=self._phase, + artifact="score_weight_bytes", detail="; ".join(mismatches)) + verdict = (_BYTE_VERDICTS["route_weight"] + if "route_weight" in mismatches[0] else _BYTE_VERDICTS["score"]) + self._append(StageResult(stage=STAGE_SCORE_WEIGHT, passed=False, + verdict=verdict, mismatch=fm, notes=mismatches)) + return + self._append(StageResult(stage=STAGE_SCORE_WEIGHT, passed=True, verdict=None, + mismatch=None)) + + # -- stage 4: gradient bytes ------------------------------------------------ + + def check_gradient(self, lhs_dz: torch.Tensor, rhs_dz: torch.Tensor) -> None: + """Stage 4: byte gate on active ``ds != 0`` backward outputs (§2.1). + + Non-finite P3-computed gradients fail closed as ``NON_FINITE`` + before byte comparison (§2.5). Upstream z/dweights non-finiteness + is a provider-side ``UPSTREAM_NON_FINITE`` and never reaches here. + """ + if not self._nonfinite_gate(lhs_dz, rhs_dz, active_mask=None, + site="bwd", owner="T06", issue="#42/#44"): + return + if not tensor_byte_exact(lhs_dz, rhs_dz): + key = MismatchKey(absolute_layer=-1, site="bwd", pass_direction="backward", + event_index=-1, global_token_id=-1, rank=-1) + fm = FirstMismatch(found=True, key=key, owner="T06", issue="#42/#44", + boundary="byte gate", phase=self._phase, + artifact="gradient_bytes", + detail="dz bytes differ on active rows") + self._append(StageResult(stage=STAGE_GRADIENT, passed=False, + verdict=_BYTE_VERDICTS["gradient"], mismatch=fm)) + return + self._append(StageResult(stage=STAGE_GRADIENT, passed=True, verdict=None, + mismatch=None)) + + # -- report --------------------------------------------------------------- + + def report(self) -> ComparisonReport: + return ComparisonReport( + case_id=self._case_id, stages=list(self._stages), stopped_early=self._stopped, + ) diff --git a/rl_engine/moe/validation/fingerprint.py b/rl_engine/moe/validation/fingerprint.py new file mode 100644 index 00000000..3cff272e --- /dev/null +++ b/rl_engine/moe/validation/fingerprint.py @@ -0,0 +1,235 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 RL-Kernel Contributors + +"""Minimal canonical serialization + fingerprints for T09's ladder (§2.4). + +Scope note: the *unique* assembler and the frozen schema are T05/T01 +deliverables (contract §4). Until the start kit lands, T09 needs a minimal, +explicitly-versioned canonical form to implement L1 (same-config repeat -> +artifact hash) and L2 (cross batch/pack/padding -> semantic hash only). +This module is that stand-in: single-sourced, versioned, and replaceable +in-place when T01 publishes the real schema. It must not grow assembler +logic beyond hashing. + +Rules implemented (§2.4): +- semantic hash canonical order: schema/model identity, layer/mode/token, + policy/round/tie, table/bias (absent -> 32 zero bytes + present=false), + selection/weight source, then per slot 0..5: + topk_index/logical_expert_id/valid/invalid_reason/route_weight/ + weight_score/selection_score/capacity. +- ints little-endian, floats raw bytes, type/length-prefixed. +- semantic hash EXCLUDES fingerprint itself, run/engine/attempt/case/rank/ + backend, and padding rows. +- artifact hash: per (case,config,rank), over all rows by source_row + INCLUDING padding, plus Envelope canonical fields. L1 compares artifact + hash; cross-config compares semantic hash only. +""" + +from __future__ import annotations + +import hashlib +import struct +from dataclasses import dataclass, field +from typing import Any + +CANONICAL_SCHEMA_VERSION = "p3-t09-canonical.v1" + + +# --- primitive encoders (type/length-prefixed) ------------------------------ + +def enc_int(value: int) -> bytes: + return b"i" + struct.pack(" bytes: + return b"f" + struct.pack(" bytes: + raw = value.encode("utf-8") + return b"s" + struct.pack(" bytes: + return b"b" + (b"\x01" if value else b"\x00") + + +def enc_optional_str(value: str | None) -> bytes: + if value is None: + return b"o\x00" + raw = value.encode("utf-8") + return b"o\x01" + struct.pack(" 32 zero bytes + present=false + + +def enc_absent() -> bytes: + return enc_bool(False) + ABSENT_FIELD_32Z + + +def enc_fingerprint(value: str | None) -> bytes: + if value is None: + return enc_absent() + return enc_bool(True) + bytes.fromhex(value) + + +# --- route record model ------------------------------------------------------ + +@dataclass(frozen=True) +class RouteRow: + """One (token, slot) Core record, minimal §2.4 field set.""" + + global_token_id: int + input_token_id: int + absolute_layer: int + router_mode: str # hash | learned + topk_index: int # 0..5 + logical_expert_id: int + valid: bool + invalid_reason: str | None + route_weight: float + weight_score: float + selection_score: float + + +@dataclass(frozen=True) +class RouteIdentity: + """Per-case identity fields entering the semantic hash header.""" + + schema_version: str = CANONICAL_SCHEMA_VERSION + checkpoint_id: str = "ckpt" + weight_id: str = "w" + logit_round_point: str = "fp32_direct" + tie_break_policy: str = "p3_canonical" + capacity_policy: str = "dropless_v1" + table_fingerprint: str | None = None # hash layers + bias_fingerprint: str | None = None # learned layers + selection_source: str = "pre_bias_score" + weight_source: str = "pre_bias_score" + + +# --- semantic hash ----------------------------------------------------------- + +def _row_bytes(row: RouteRow) -> bytes: + return b"".join([ + enc_int(row.global_token_id), + enc_int(row.input_token_id), + enc_int(row.absolute_layer), + enc_str(row.router_mode), + enc_int(row.topk_index), + enc_int(row.logical_expert_id), + enc_bool(row.valid), + enc_optional_str(row.invalid_reason), + enc_float32(row.route_weight), + enc_float32(row.weight_score), + enc_float32(row.selection_score), + ]) + + +def _header_bytes(identity: RouteIdentity, absolute_layer: int, + router_mode: str) -> bytes: + return b"".join([ + enc_str(identity.schema_version), + enc_str(identity.checkpoint_id), + enc_str(identity.weight_id), + enc_int(absolute_layer), + enc_str(router_mode), + enc_str(identity.logit_round_point), + enc_str(identity.tie_break_policy), + enc_str(identity.capacity_policy), + enc_fingerprint(identity.table_fingerprint), + enc_fingerprint(identity.bias_fingerprint), + enc_str(identity.selection_source), + enc_str(identity.weight_source), + ]) + + +def route_semantic_hash( + rows: list[RouteRow], + identity: RouteIdentity, +) -> str: + """Per-token semantic map hashed in ascending token order (§2.4). + + Padding rows (``global_token_id == -1``) never enter the semantic hash. + ``route_semantic_fingerprint`` per §2.4 is the per-token map; here the + case-level hash over tokens ascending is returned, and per-token hashes + are available via :func:`per_token_semantic_hashes` so callers can report + token-level diffs instead of only a case hash ("不得只报 case hash"). + """ + per_token = per_token_semantic_hashes(rows, identity) + h = hashlib.sha256() + h.update(enc_str("p3.semantic.v1")) + for token in sorted(per_token): + h.update(enc_int(token)) + h.update(bytes.fromhex(per_token[token])) + return h.hexdigest() + + +def per_token_semantic_hashes( + rows: list[RouteRow], + identity: RouteIdentity, +) -> dict[int, str]: + """Semantic hash per ``global_token_id`` (ascending slots included).""" + by_token: dict[int, list[RouteRow]] = {} + for r in rows: + if r.global_token_id == -1: # padding excluded (§2.4) + continue + by_token.setdefault(r.global_token_id, []).append(r) + + out: dict[int, str] = {} + for token, token_rows in by_token.items(): + layer = token_rows[0].absolute_layer + mode = token_rows[0].router_mode + h = hashlib.sha256() + h.update(_header_bytes(identity, layer, mode)) + h.update(enc_int(token)) + for r in sorted(token_rows, key=lambda x: x.topk_index): + h.update(_row_bytes(r)) + out[token] = h.hexdigest() + return out + + +# --- artifact hash (L1) ------------------------------------------------------ + +@dataclass(frozen=True) +class EnvelopeFields: + """Minimal Envelope canonical fields entering the artifact hash.""" + + run_id: str + engine_id: str + attempt_id: int + source_row: int + physical_expert_id: int + rank: int + + +@dataclass +class Artifact: + """One (case, config, rank) artifact: all rows incl. padding + envelope.""" + + identity: RouteIdentity + rows: list[RouteRow] # ALL rows by source_row, incl. padding + envelopes: list[EnvelopeFields] # aligned with rows by source_row + extra_envelope: dict[str, str] = field(default_factory=dict) + + +def route_artifact_hash(artifact: Artifact) -> str: + """Artifact hash over all rows (incl. padding) + Envelope (§2.4).""" + h = hashlib.sha256() + h.update(enc_str("p3.artifact.v1")) + h.update(_header_bytes(artifact.identity, + artifact.rows[0].absolute_layer if artifact.rows else -1, + artifact.rows[0].router_mode if artifact.rows else "none")) + for row, env in zip(artifact.rows, artifact.envelopes): + h.update(_row_bytes(row)) + h.update(enc_int(env.source_row)) + h.update(enc_int(env.physical_expert_id)) + h.update(enc_int(env.rank)) + h.update(enc_str(env.run_id)) + h.update(enc_str(env.engine_id)) + h.update(enc_int(env.attempt_id)) + for k in sorted(artifact.extra_envelope): + h.update(enc_str(k)) + h.update(enc_str(artifact.extra_envelope[k])) + return h.hexdigest() diff --git a/rl_engine/moe/validation/first_mismatch.py b/rl_engine/moe/validation/first_mismatch.py new file mode 100644 index 00000000..e887af87 --- /dev/null +++ b/rl_engine/moe/validation/first_mismatch.py @@ -0,0 +1,199 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 RL-Kernel Contributors + +"""First-mismatch localization and stable attribution (contract §4-T09). + +T09 must "按 (absolute_layer, site, pass, event_index, global_token_id, rank) +定位首差分;输出 owner/Issue/boundary/phase/artifact" — i.e. a failing +comparison points at exactly one first divergence and names the responsible +owner task, issue, boundary, phase and artifact, instead of reporting an +averaged error. + +This module is pure bookkeeping: it never judges numeric equality (that is +``comparison.py``'s job); it only locates the first divergent event between +two already-normalized trace streams and maps it to ownership. +""" + +from __future__ import annotations + +from dataclasses import dataclass +from typing import Any, NamedTuple + +from rl_engine.moe.p3_verdicts import P3Verdict + + +class MismatchKey(NamedTuple): + """The contract's six-tuple localization key. + + Order is fixed by the contract: (absolute_layer, site, pass, event_index, + global_token_id, rank). Sorting by this tuple yields the canonical event + order in which the *first* mismatch is searched. + """ + + absolute_layer: int + site: str # score | selection | weight | handoff (§2.5) + pass_direction: str # forward | backward + event_index: int + global_token_id: int + rank: int + + +# --- static owner / issue attribution table (contract §3.1 issue map) ------ +# site -> (owner task, issue). Kept explicit and exhaustive on purpose: a +# mismatch with no entry must fail closed (UNKNOWN_OWNER), never guess. + +_ATTRIBUTION: dict[str, tuple[str, str]] = { + "score": ("T02", "#41"), # router_sqrt_softplus fwd/bwd + "hash_lookup": ("T03", "#42"), # tid2eid lookup / slot order + "topk": ("T01", "#43"), # stable_topk6 (T01-owned golden) + "selection": ("T04", "#44"), # learned selection pre/post bias + "weight": ("T05", "#43"), # assembler / fingerprint owns weight rows + "handoff": ("T05", "#46"), # CombinePlanSeed / downstream handoff + "bwd": ("T06", "#42/#44"), # router backward + "tp_sp": ("T07", "#45"), # TP/SP global top-6 / hidden integrity + "placement": ("T08", "#46"), # CP/DP/PP/EP placement +} + +VALID_SITES = frozenset({"score", "hash_lookup", "topk", "selection", "weight", "handoff"}) +VALID_PASSES = frozenset({"forward", "backward"}) + + +class UnknownSiteError(ValueError): + """Raised when a trace event carries a site outside the contract set.""" + + +@dataclass(frozen=True) +class TraceEvent: + """One normalized comparison event from a recorded trace. + + ``payload`` holds the compared field(s) for this event; equality of + payloads is decided by the comparator, not here. + """ + + key: MismatchKey + payload: dict[str, Any] + + +@dataclass(frozen=True) +class FirstMismatch: + """Result of a first-mismatch search.""" + + found: bool + key: MismatchKey | None + owner: str | None # e.g. "T02"; None when found=False + issue: str | None # e.g. "#41" + boundary: str # what boundary the event sits on + phase: str # WS1 | WS2 | Integration + artifact: str | None # which sealed artifact diverged + detail: str # human-readable first-divergence description + + +_EMPTY = FirstMismatch( + found=False, key=None, owner=None, issue=None, + boundary="none", phase="WS1", artifact=None, detail="no mismatch", +) + + +def _site_owner(site: str) -> tuple[str, str]: + try: + return _ATTRIBUTION[site] + except KeyError: + raise UnknownSiteError( + f"site {site!r} has no owner mapping; refusing to guess attribution" + ) from None + + +def _validate_event(event: TraceEvent) -> None: + if event.key.site not in VALID_SITES: + raise UnknownSiteError(f"event site {event.key.site!r} not in {sorted(VALID_SITES)}") + if event.key.pass_direction not in VALID_PASSES: + raise UnknownSiteError( + f"event pass {event.key.pass_direction!r} not in {sorted(VALID_PASSES)}" + ) + # backward pass only makes sense on gradient-carrying sites + if event.key.pass_direction == "backward" and event.key.site not in {"score", "bwd"}: + raise UnknownSiteError( + f"backward pass at site {event.key.site!r} violates §2.5 event order" + ) + + +def canonical_order(events: list[TraceEvent]) -> list[TraceEvent]: + """Sort events into the canonical comparison order (by six-tuple key).""" + return sorted(events, key=lambda e: e.key) + + +def first_mismatch( + lhs: list[TraceEvent], + rhs: list[TraceEvent], + *, + payload_equal: Any = None, + phase: str = "WS1", + artifact_name: str = "route_trace", +) -> FirstMismatch: + """Locate the first divergent event between two trace streams. + + Args: + lhs: events from the reference/oracle side (already normalized). + rhs: events from the candidate side. + payload_equal: callable ``(lhs_payload, rhs_payload) -> bool``; when + omitted, plain ``==`` is used. The comparator passes its own + stage-aware equality here (byte-exact / discrete-exact). + phase: WS1 | WS2 | Integration, recorded in the result. + artifact_name: which artifact the streams came from, for the report. + + Fail-closed behaviour: both streams are validated first; an unknown site + or pass raises ``UnknownSiteError`` rather than producing a possibly + wrong attribution. Length mismatch is reported as the mismatch at the + first index where one stream runs out (missing evidence is never "equal"). + """ + if payload_equal is None: + def payload_equal(a: dict[str, Any], b: dict[str, Any]) -> bool: # noqa: F811 + return a == b + + for stream in (lhs, rhs): + for event in stream: + _validate_event(event) + + ordered_lhs = canonical_order(lhs) + ordered_rhs = canonical_order(rhs) + + for idx, (a, b) in enumerate(zip(ordered_lhs, ordered_rhs)): + if a.key != b.key: + owner, issue = _site_owner(a.key.site) + return FirstMismatch( + found=True, key=min(a.key, b.key), owner=owner, issue=issue, + boundary=f"event[{idx}] key divergence", phase=phase, + artifact=artifact_name, + detail=f"keys differ at event {idx}: lhs={a.key} rhs={b.key}", + ) + if not payload_equal(a.payload, b.payload): + owner, issue = _site_owner(a.key.site) + return FirstMismatch( + found=True, key=a.key, owner=owner, issue=issue, + boundary=f"event[{idx}] payload", phase=phase, + artifact=artifact_name, + detail=( + f"first payload divergence at {a.key}: " + f"lhs={_short(a.payload)} rhs={_short(b.payload)}" + ), + ) + + if len(ordered_lhs) != len(ordered_rhs): + shorter = "lhs" if len(ordered_lhs) < len(ordered_rhs) else "rhs" + idx = min(len(ordered_lhs), len(ordered_rhs)) + survivor = ordered_rhs if shorter == "lhs" else ordered_lhs + key = survivor[idx].key + owner, issue = _site_owner(key.site) + return FirstMismatch( + found=True, key=key, owner=owner, issue=issue, + boundary=f"event[{idx}] stream length", phase=phase, + artifact=artifact_name, + detail=f"{shorter} stream ended early; unmatched event {key}", + ) + + return _EMPTY + + +def _short(payload: dict[str, Any]) -> str: + text = repr(payload) + return text if len(text) <= 96 else text[:93] + "..." diff --git a/rl_engine/moe/validation/ladder.py b/rl_engine/moe/validation/ladder.py new file mode 100644 index 00000000..d98b7a35 --- /dev/null +++ b/rl_engine/moe/validation/ladder.py @@ -0,0 +1,208 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 RL-Kernel Contributors + +"""Validation ladder L1-L3b runners (contract §4-T09 / §7 WS1 Gate). + +Ladders: +- L1 repeat : same config re-run -> ``route_artifact_fingerprint`` + must match byte-for-byte (§7 WS1 "L1/L2/L3a/L3b recorded 通过"). +- L2 invariance : batch/pack/padding/launch perturbations change only + Envelope/source_row; Core semantic hash per token must be IDENTICAL + (§1.3 WS1 "batch/pack/padding/launch invariant"). +- L3a oracle : candidate vs bit-defined CPU oracle, default + byte-exact on Core rows (§2.5). +- L3b dual-engine : recorded Megatron vs Miles traces compared through + the ordered comparator (§2.5 engine-vs-engine must be byte-exact). + +Each runner returns a :class:`LadderReport`; strict mismatches surface the +first mismatch (six-tuple + owner/issue) and are never averaged away. +""" + +from __future__ import annotations + +import torch + +from rl_engine.moe.p3_verdicts import P3Verdict +from rl_engine.moe.validation.comparison import TraceComparator +from rl_engine.moe.validation.fingerprint import ( + Artifact, + RouteIdentity, + RouteRow, + per_token_semantic_hashes, + route_artifact_hash, +) +from rl_engine.moe.validation.first_mismatch import ( + FirstMismatch, + MismatchKey, + TraceEvent, +) +from rl_engine.moe.validation.report import ( + LadderReport, + make_fail as _fail, + make_pass as _pass, +) + + +# --- L1: repeat -------------------------------------------------------------- + +def run_l1_repeat( + case_id: str, + run_a: Artifact, + run_b: Artifact, +) -> LadderReport: + """Same-config repeat: artifact hash must be byte-identical (§2.4).""" + ha, hb = route_artifact_hash(run_a), route_artifact_hash(run_b) + if ha == hb: + return _pass("L1", case_id, artifact_hash=ha) + # locate first diverging row for the report (artifact-level first diff) + idx = next( + (i for i, (ra, rb) in enumerate(zip(run_a.rows, run_b.rows)) + if route_artifact_hash(Artifact( + identity=run_a.identity, rows=[ra], envelopes=[run_a.envelopes[i]])) + != route_artifact_hash(Artifact( + identity=run_b.identity, rows=[rb], envelopes=[run_b.envelopes[i]]))), + None, + ) + where = f"first differing source_row block at index {idx}" if idx is not None \ + else "artifact hashes differ (row-level localization unavailable)" + return _fail("L1", case_id, P3Verdict.ROUTE_ARTIFACT_FINGERPRINT_MISMATCH, + f"artifact hash {ha[:12]}.. vs {hb[:12]}..: {where}", + artifact_hash_a=ha, artifact_hash_b=hb) + + +# --- L2: batch/pack/padding/launch invariance -------------------------------- + +def run_l2_invariance( + case_id: str, + base: Artifact, + perturbed: Artifact, + *, + variant: str, +) -> LadderReport: + """Cross-variant: per-token semantic hashes must be IDENTICAL (§2.4). + + Only Core semantics are compared; Envelope/source_row/padding placement + may legitimately differ. Missing tokens -> INCOMPLETE_ARTIFACT; extra or + duplicated tokens -> AMBIGUOUS_GLOBAL_TOKEN_MAPPING. + """ + sa = per_token_semantic_hashes(base.rows, base.identity) + sb = per_token_semantic_hashes(perturbed.rows, perturbed.identity) + + missing = sorted(set(sa) - set(sb)) + extra_tokens = sorted(set(sb) - set(sa)) + if missing: + return _fail("L2", case_id, P3Verdict.INCOMPLETE_ARTIFACT, + f"variant={variant}: tokens missing in perturbed run: {missing[:8]}", + missing=missing, variant=variant) + if extra_tokens: + return _fail("L2", case_id, P3Verdict.AMBIGUOUS_GLOBAL_TOKEN_MAPPING, + f"variant={variant}: unexpected tokens: {extra_tokens[:8]}", + extra_tokens=extra_tokens, variant=variant) + + for token in sorted(sa): + if sa[token] != sb[token]: + return _fail( + "L2", case_id, P3Verdict.ROUTE_SEMANTIC_FINGERPRINT_MISMATCH, + f"variant={variant}: first differing token={token} " + f"semantic {sa[token][:12]}.. vs {sb[token][:12]}..", + first_token=int(token), variant=variant, + ) + return _pass("L2", case_id, f"variant={variant}: semantic identical", + tokens=len(sa), variant=variant) + + +# --- L3a: oracle byte-exact ---------------------------------------------------- + +def run_l3a_oracle( + case_id: str, + identity: RouteIdentity, + oracle_rows: list[RouteRow], + candidate_rows: list[RouteRow], +) -> LadderReport: + """CUDA candidate vs bit-defined oracle: Core rows byte-exact (§2.5). + + Padding rows (global_token_id == -1) are excluded here; they are audited + by L1's artifact gate only (§2.5 "padding 只由同配置 artifact gate 审计"). + """ + key = lambda r: (r.global_token_id, r.topk_index) # noqa: E731 + o = sorted([r for r in oracle_rows if r.global_token_id != -1], key=key) + c = sorted([r for r in candidate_rows if r.global_token_id != -1], key=key) + + if len(o) != len(c): + return _fail("L3a", case_id, P3Verdict.INCOMPLETE_ARTIFACT, + f"active row count oracle={len(o)} candidate={len(c)}") + + for ro, rc in zip(o, c): + differing = [ + f for f in ( + "global_token_id", "input_token_id", "absolute_layer", + "router_mode", "topk_index", "logical_expert_id", "valid", + "invalid_reason", "route_weight", "weight_score", + "selection_score", + ) + if getattr(ro, f) != getattr(rc, f) + ] + if differing: + fm = FirstMismatch( + found=True, + key=MismatchKey(absolute_layer=ro.absolute_layer, + site="selection" if ro.router_mode == "learned" else "hash_lookup", + pass_direction="forward", + event_index=ro.topk_index, + global_token_id=ro.global_token_id, rank=0), + owner="T04" if ro.router_mode == "learned" else "T03", + issue="#44" if ro.router_mode == "learned" else "#42", + boundary="L3a byte gate", phase="WS1", artifact="core_rows", + detail=f"token={ro.global_token_id} slot={ro.topk_index} " + f"fields differ: {differing}", + ) + verdict = (P3Verdict.TOPK_ORDER_MISMATCH + if {"topk_index", "logical_expert_id"} & set(differing) + else P3Verdict.ROUTE_WEIGHT_BYTES_MISMATCH + if "route_weight" in differing + else P3Verdict.BYTE_MISMATCH) + return _fail("L3a", case_id, verdict, fm.detail, fm, + differing_fields=differing) + return _pass("L3a", case_id, f"{len(o)} active rows byte-exact", + active_rows=len(o)) + + +# --- L3b: recorded dual-engine --------------------------------------------------- + +def run_l3b_dual_engine( + case_id: str, + lhs_meta: dict[str, Any], + rhs_meta: dict[str, Any], + lhs_events: list[TraceEvent], + rhs_events: list[TraceEvent], + lhs_weights: torch.Tensor, + rhs_weights: torch.Tensor, + lhs_dz: torch.Tensor, + rhs_dz: torch.Tensor, + *, + active_mask: torch.Tensor | None = None, +) -> LadderReport: + """Recorded Megatron vs Miles through the ordered comparator (§2.5). + + Engine-vs-engine must be byte-exact; the four-stage walk (identity -> + discrete -> score/weight -> gradient) locates the first divergence with + owner/issue attribution. + """ + cmp = TraceComparator(case_id, phase="WS1") + cmp.check_identity(lhs_meta, rhs_meta) + if not cmp.report().stopped_early: + cmp.check_discrete(lhs_events, rhs_events) + stage_ok = cmp.report().stages[-1].passed if len(cmp.report().stages) > 1 else False + if stage_ok: + cmp.check_score_weight(lhs_weights, rhs_weights, active_mask=active_mask) + if cmp.report().stages[-1].passed: + cmp.check_gradient(lhs_dz, rhs_dz) + + rep = cmp.report() + if rep.passed: + return _pass("L3b", case_id, "dual-engine byte-exact across 4 stages") + stage = next((s for s in rep.stages if not s.passed), None) + return _fail("L3b", case_id, rep.primary, stage.notes[0] if stage and stage.notes + else (stage.mismatch.detail if stage and stage.mismatch else "walk failed"), + stage.mismatch if stage else None, + walked=[s.stage for s in rep.stages]) diff --git a/rl_engine/moe/validation/paired_check.py b/rl_engine/moe/validation/paired_check.py new file mode 100644 index 00000000..6ed818ce --- /dev/null +++ b/rl_engine/moe/validation/paired_check.py @@ -0,0 +1,160 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 RL-Kernel Contributors + +"""Torch paired check for formal goldens (contract §2.5, T09 step 7). + +Contract §2.5: "Torch 原始参考必须在每个正式 golden 上运行 paired check +并记录诊断;其差异不能覆盖 strict verdict,缺证据为 MISSING_PROVENANCE。" + +Rules implemented: +- Every formal golden (published by T01, fixture manifest entry) must carry + a paired-check record: the raw Torch reference executed on the same case + inputs, with diagnostics (max/mean abs diff vs the golden's oracle). +- The paired diff is DIAGNOSTIC ONLY: it can never flip a strict verdict + (byte-exact gates). It exists to catch "golden itself drifted from Torch" + early and to keep evidence for audits. +- A golden without a paired record is not acceptable evidence: + MISSING_PROVENANCE (67). +- Until T01 publishes the start kit (anchor_pending), the manifest lookup + returns empty and every check reports golden-missing — tests then skip. + Nothing here guesses or fabricates a manifest path. +""" + +from __future__ import annotations + +import dataclasses +import json +import pathlib +from dataclasses import dataclass +from typing import Any, Callable + +import torch + +from rl_engine.moe.p3_verdicts import P3Verdict +from rl_engine.moe.validation.report import LadderReport, make_fail, make_pass + +_fail, _pass = make_fail, make_pass # module-local aliases for brevity + +#: Repository-level fixture manifest location, published by T01. +#: Its absence is the documented anchor_pending state (contract §1.3/§5): +#: we look it up, we never guess. +GOLDEN_MANIFEST_PATH = pathlib.Path(__file__).resolve().parents[3] / \ + "fixtures" / "p3" / "manifest.json" + + +@dataclass(frozen=True) +class GoldenEntry: + """One formal golden from T01's fixture manifest.""" + + name: str # e.g. "learned_dropless_basic" + fixture_path: str + checksum: str + kind: str # random | near_tie | exact_tie | padding + router_mode: str # hash | learned + + +@dataclass(frozen=True) +class PairedRecord: + """Diagnostic evidence that raw Torch ran on this golden's case.""" + + golden_name: str + torch_max_abs_diff: float + torch_mean_abs_diff: float + n_tensors_compared: int + diagnostic_note: str = "" + + +def load_golden_manifest(path: pathlib.Path | None = None) -> list[GoldenEntry]: + """Read T01's fixture manifest; [] means anchor_pending (no guessing).""" + manifest = path if path is not None else GOLDEN_MANIFEST_PATH + if not manifest.is_file(): + return [] + try: + raw = json.loads(manifest.read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError): + return [] + entries = raw.get("goldens", raw) if isinstance(raw, dict) else raw + out: list[GoldenEntry] = [] + for item in entries or []: + try: + out.append(GoldenEntry( + name=str(item["name"]), + fixture_path=str(item["fixture_path"]), + checksum=str(item["checksum"]), + kind=str(item.get("kind", "random")), + router_mode=str(item.get("router_mode", "learned")), + )) + except (KeyError, TypeError): + continue # malformed entry: skip, do not fabricate + return out + + +def run_paired_check( + case_id: str, + golden: GoldenEntry, + golden_tensors: dict[str, torch.Tensor], + torch_reference: Callable[[dict[str, Any]], dict[str, torch.Tensor]], + case_inputs: dict[str, Any], + *, + existing_record: PairedRecord | None = None, +) -> LadderReport: + """Run raw Torch on the golden's case and record diagnostics (§2.5). + + The diff NEVER flips strict verdicts; missing execution evidence is + MISSING_PROVENANCE. If T01 ships a paired record inside the fixture, + ``existing_record`` short-circuits the local Torch run (still evidence). + """ + if existing_record is None: + if not callable(torch_reference): + return _fail("paired", case_id, P3Verdict.MISSING_PROVENANCE, + "no Torch reference executed on this golden") + try: + ref = torch_reference(case_inputs) + except Exception as exc: # noqa: BLE001 — evidence failure, not a crash mask + return _fail("paired", case_id, P3Verdict.MISSING_PROVENANCE, + f"Torch reference failed to run: {type(exc).__name__}: {exc}") + + max_abs = 0.0 + mean_abs = 0.0 + n = 0 + for key, gold_t in golden_tensors.items(): + if key not in ref: + continue + r = ref[key] + if r.shape != gold_t.shape or r.dtype != gold_t.dtype: + d = float("inf") + mean_d = float("inf") + else: + d = float((r.double() - gold_t.double()).abs().max().item()) + mean_d = float((r.double() - gold_t.double()).abs().mean().item()) + max_abs = max(max_abs, d) + mean_abs = max(mean_abs, mean_d) + n += 1 + record = PairedRecord( + golden_name=golden.name, + torch_max_abs_diff=max_abs, + torch_mean_abs_diff=mean_abs, + n_tensors_compared=n, + ) + else: + record = existing_record + + return _pass("paired", case_id, + f"golden={golden.name} torch_max_abs={record.torch_max_abs_diff:.3e} " + f"over {record.n_tensors_compared} tensors (diagnostic only)", + paired_record=dataclasses.asdict(record)) + + +def paired_gate_for_goldens( + case_id: str, + goldens: list[GoldenEntry], + records: dict[str, PairedRecord], +) -> LadderReport: + """Gate: every formal golden must have paired evidence (§2.5, 67).""" + missing = [g.name for g in goldens if g.name not in records] + if missing: + return _fail("paired-gate", case_id, P3Verdict.MISSING_PROVENANCE, + f"goldens without Torch paired evidence: {missing}", + missing=missing) + return _pass("paired-gate", case_id, + f"{len(records)} goldens carry paired diagnostics") diff --git a/rl_engine/moe/validation/report.py b/rl_engine/moe/validation/report.py new file mode 100644 index 00000000..0e7aa192 --- /dev/null +++ b/rl_engine/moe/validation/report.py @@ -0,0 +1,50 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 RL-Kernel Contributors + +"""Shared report type for every T09 ladder/ws2/paired runner. + +All step-level runners (L1/L2/L3a/L3b, WS2 rank/cross-config, Torch paired +check) return a :class:`LadderReport` so the CLI and tests can treat them +uniformly. Keeping the type here (not in ``ladder``) avoids a cycle: +``ws2``/``paired_check`` need the report type without importing ladder +runners. +""" + +from __future__ import annotations + +from dataclasses import dataclass, field +from typing import Any + +from rl_engine.moe.p3_verdicts import P3Verdict +from rl_engine.moe.validation.first_mismatch import FirstMismatch + + +@dataclass +class LadderReport: + """Outcome of one ladder run.""" + + ladder: str # L1 | L2 | L3a | L3b | WS2-rank | WS2-cross | paired* + case_id: str + passed: bool + verdict: P3Verdict | None # primary; None iff passed + detail: str = "" + first_mismatch: FirstMismatch | None = None + extra: dict[str, Any] = field(default_factory=dict) + + def summary_line(self) -> str: + flag = "PASS" if self.passed else f"FAIL({self.verdict.name if self.verdict else '?'})" + return f"[{self.ladder}] case={self.case_id} {flag} {self.detail[:120]}" + + +def make_fail(ladder: str, case_id: str, verdict: P3Verdict, detail: str, + fm: FirstMismatch | None = None, **extra: Any) -> LadderReport: + """Build a failing report (strict verdict + optional first mismatch).""" + return LadderReport(ladder=ladder, case_id=case_id, passed=False, + verdict=verdict, detail=detail, first_mismatch=fm, + extra=extra) + + +def make_pass(ladder: str, case_id: str, detail: str = "", **extra: Any) -> LadderReport: + """Build a passing report (verdict None — pass is not a verdict code).""" + return LadderReport(ladder=ladder, case_id=case_id, passed=True, + verdict=None, detail=detail, extra=extra) diff --git a/rl_engine/moe/validation/synthetic_producer.py b/rl_engine/moe/validation/synthetic_producer.py new file mode 100644 index 00000000..986edfca --- /dev/null +++ b/rl_engine/moe/validation/synthetic_producer.py @@ -0,0 +1,152 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 RL-Kernel Contributors + +"""Synthetic recorded producer for T09 ladder development (contract §5). + +Contract §1.3/§5: "WS1/WS2 可仅使用 T01 的 recorded/synthetic producer 关闭, +不等待 live 下游" — until T01's start kit publishes real fixtures, T09 +develops the ladder against deterministic synthetic artifacts produced here. + +Determinism: seeded generators only; identical (case_id, seed) always +reproduces bit-identical artifacts, which L1 relies on. +""" + +from __future__ import annotations + +import hashlib + +import torch + +from rl_engine.moe.naive_topk6 import K, naive_topk6 +from rl_engine.moe.validation.fingerprint import ( + Artifact, + EnvelopeFields, + RouteIdentity, + RouteRow, +) + +DEFAULT_E = 256 +_SCALE = 1.5 +_EPS = 1e-20 + + +def _six_way_sum(a: list[float]) -> float: + """FP32 fixed 6-way tree per §2.1: ((a0+a1)+(a2+a3))+(a4+a5).""" + t = torch.tensor([a[0] + a[1], a[2] + a[3], a[4] + a[5]], dtype=torch.float32) + return float((t[0] + t[1]) + t[2]) + + +def make_learned_route_rows( + case_id: str, + *, + seed: int, + t_rows: int, + layers: tuple[int, ...] = (3,), + bias: torch.Tensor | None = None, + e: int = DEFAULT_E, +) -> tuple[list[RouteRow], dict[str, object]]: + """Produce deterministic Learned-mode Core rows via naive_topk6. + + Follows §2.1 Learned math: q = s + b, ids = stable_topk6(q), + a_i = s[ids_i] (pre-bias weight source), p_i = a_i / Z, w_i = p_i * 1.5, + with the fixed FP32 6-way tree for S. + """ + g = torch.Generator().manual_seed(seed) + if bias is None: + bias = torch.randn(e, generator=g, dtype=torch.float32) * 0.1 + + rows: list[RouteRow] = [] + identity = RouteIdentity( + checkpoint_id=f"ckpt-{case_id}", weight_id=f"w-{case_id}", + bias_fingerprint=_fp(bias), + ) + for layer in layers: + z = torch.randn(t_rows, e, generator=g, dtype=torch.float32) + u = torch.nn.functional.softplus(z) + s = torch.sqrt(u) + q = s + bias + ids, _ = naive_topk6(q) + for t in range(t_rows): + a = [float(s[t, int(ids[t, i])]) for i in range(K)] + ssum = _six_way_sum(a) + zeta = ssum + _EPS + for i in range(K): + p = a[i] / zeta + rows.append(RouteRow( + global_token_id=t, input_token_id=t, + absolute_layer=layer, router_mode="learned", + topk_index=i, logical_expert_id=int(ids[t, i]), + valid=True, invalid_reason=None, + route_weight=p * _SCALE, weight_score=a[i], + selection_score=float(q[t, int(ids[t, i])]), + )) + meta = { + "case_id": case_id, "checkpoint_id": identity.checkpoint_id, + "weight_id": identity.weight_id, "absolute_layer": layers[0], + "router_mode": "learned", + "table_fingerprint": hashlib.sha256(f"t-{case_id}".encode()).hexdigest(), + "bias_fingerprint": identity.bias_fingerprint + or hashlib.sha256(f"b-{case_id}".encode()).hexdigest(), + "logit_round_point": "fp32_direct", "tie_break_policy": "p3_canonical", + "capacity_policy": "dropless_v1", + } + return rows, meta + + +def _fp(x: torch.Tensor) -> str: + return hashlib.sha256( + x.contiguous().view(torch.uint8).numpy().tobytes() + ).hexdigest() + + +def make_artifact( + case_id: str, + rows: list[RouteRow], + identity: RouteIdentity, + *, + run_id: str, + engine_id: str, + attempt_id: int, + rank: int = 0, + placement_offset: int = 0, + padding_rows: int = 0, +) -> Artifact: + """Wrap rows + padding into an Artifact with Envelope fields. + + Padding rows follow §2.1 canonical: token ids -1, expert -1, weight/score + +0.0, invalid/padding. physical_expert_id uses a deterministic bijection + ``logical + placement_offset`` standing in for T08's versioned map. + """ + env: list[EnvelopeFields] = [] + all_rows: list[RouteRow] = list(rows) + for src, r in enumerate(rows): + env.append(EnvelopeFields( + run_id=run_id, engine_id=engine_id, attempt_id=attempt_id, + source_row=src, physical_expert_id=r.logical_expert_id + placement_offset, + rank=rank, + )) + for p in range(padding_rows): + src = len(rows) + p + all_rows.append(RouteRow( + global_token_id=-1, input_token_id=-1, + absolute_layer=rows[0].absolute_layer if rows else 0, + router_mode="learned", topk_index=p % K, + logical_expert_id=-1, valid=False, invalid_reason="padding", + route_weight=0.0, weight_score=0.0, selection_score=0.0, + )) + env.append(EnvelopeFields( + run_id=run_id, engine_id=engine_id, attempt_id=attempt_id, + source_row=src, physical_expert_id=-1, rank=rank, + )) + return Artifact(identity=identity, rows=all_rows, envelopes=env) + + +def repack_rows(rows: list[RouteRow], *, batch_size: int) -> list[RouteRow]: + """L2 helper: renumber global token ids as if batch partition changed. + + Core per-token semantics must survive repack; only identity-of-position + changes, which L2 must prove does NOT alter semantic hashes. Here the + repack keeps (token content) identical and only reorders rows — the + canonical serialization sorts by token, so hashes stay stable. + """ + return list(reversed(rows)) diff --git a/rl_engine/moe/validation/ws2.py b/rl_engine/moe/validation/ws2.py new file mode 100644 index 00000000..ba6cdcd1 --- /dev/null +++ b/rl_engine/moe/validation/ws2.py @@ -0,0 +1,134 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 RL-Kernel Contributors + +"""WS2 extensions for the T09 ladder: rank dimension + cross-config (§3-T09). + +Contract §3 (WS2 Gate row): "同一 token 的 Core semantic fingerprint exact, +Envelope 按配置完整" — across TP/SP/CP/DP/PP/EP configs the per-token Core +semantics must be identical, while Envelope completeness is judged per +config. This module adds: + +- :func:`check_rank_completeness` — every expected rank must contribute an + artifact (missing -> MISSING_RANK, provider band §6). +- :func:`run_ws2_cross_config` — compare two configurations' rank sets: + partition ownership must cover the expected token set exactly once; + replica ownership (TP) may repeat tokens but every carrier's semantic + hash must be identical; any token-level divergence names the first + offending (token, rank) pair via the six-tuple. +""" + +from __future__ import annotations + +from dataclasses import dataclass + +from rl_engine.moe.p3_verdicts import P3Verdict +from rl_engine.moe.validation.fingerprint import Artifact, per_token_semantic_hashes +from rl_engine.moe.validation.report import LadderReport, make_fail, make_pass + +_fail, _pass = make_fail, make_pass # module-local aliases for brevity + +TOKEN_PARTITION = "partition" # CP/DP: token appears on exactly one rank +TOKEN_REPLICA = "replica" # TP: token may repeat; hashes must agree + + +@dataclass(frozen=True) +class RankArtifact: + """One rank's artifact under a specific parallel config.""" + + rank: int + group: str # e.g. "tp2-sp0", "dp4-cp2" + artifact: Artifact + + +def check_rank_completeness( + case_id: str, + artifacts: list[RankArtifact], + expected_ranks: range | list[int], +) -> LadderReport: + """All expected ranks present, no duplicates (§6 MISSING_RANK).""" + seen: dict[int, int] = {} + for ra in artifacts: + seen[ra.rank] = seen.get(ra.rank, 0) + 1 + missing = sorted(set(expected_ranks) - set(seen)) + if missing: + return _fail("WS2-rank", case_id, P3Verdict.MISSING_RANK, + f"missing ranks: {missing}", missing_ranks=missing) + duplicated = sorted(r for r, n in seen.items() if n > 1) + if duplicated: + return _fail("WS2-rank", case_id, P3Verdict.STALE_RUN_METADATA, + f"duplicated ranks: {duplicated}", + hint="one artifact per (case, config, rank) — stale run mix-in?") + return _pass("WS2-rank", case_id, + f"ranks {sorted(seen)} complete", ranks=sorted(seen)) + + +def _tokens_by_rank(arts: list[RankArtifact]) -> dict[int, dict[int, str]]: + """{rank: {token: semantic_hash}} over active rows.""" + out: dict[int, dict[int, str]] = {} + for ra in arts: + out[ra.rank] = per_token_semantic_hashes(ra.artifact.rows, ra.artifact.identity) + return out + + +def run_ws2_cross_config( + case_id: str, + base: list[RankArtifact], + other: list[RankArtifact], + *, + base_config: str, + other_config: str, + ownership: str = TOKEN_PARTITION, +) -> LadderReport: + """Cross-config: same token -> identical Core semantic hash (§3 WS2). + + ``base`` defines the expected token set. ``other`` must cover it under + its ownership mode: + - partition: each base token present exactly once across ``other`` ranks; + missing -> INCOMPLETE_ARTIFACT, duplicated -> AMBIGUOUS_GLOBAL_TOKEN_MAPPING. + - replica: token may appear on several ranks; every occurrence must carry + the base hash; a divergent carrier -> ROUTE_SEMANTIC_FINGERPRINT_MISMATCH + with (token, rank) named. + Envelope differences between configs are expected and NOT judged here. + """ + if ownership not in (TOKEN_PARTITION, TOKEN_REPLICA): + raise ValueError(f"unknown ownership mode: {ownership}") + + bt = _tokens_by_rank(base) + ot = _tokens_by_rank(other) + expected = {t for per in bt.values() for t in per} + + carriers: dict[int, list[int]] = {} + for rank, per in ot.items(): + for t in per: + carriers.setdefault(t, []).append(rank) + + missing = sorted(expected - set(carriers)) + if missing: + return _fail("WS2-cross", case_id, P3Verdict.INCOMPLETE_ARTIFACT, + f"{base_config}->{other_config}: tokens uncovered: {missing[:8]}", + missing=missing, base_config=base_config, other_config=other_config) + + if ownership == TOKEN_PARTITION: + dup = sorted(t for t, rs in carriers.items() if len(rs) > 1) + if dup: + return _fail("WS2-cross", case_id, P3Verdict.AMBIGUOUS_GLOBAL_TOKEN_MAPPING, + f"{base_config}->{other_config}: tokens owned by >1 rank: " + f"{ {t: carriers[t] for t in dup[:4]} }", + duplicated=dup, base_config=base_config, other_config=other_config) + + base_hash = {t: next(per[t] for per in bt.values() if t in per) for t in expected} + for t in sorted(expected): + for rank in sorted(carriers[t]): + if ot[rank][t] != base_hash[t]: + return _fail( + "WS2-cross", case_id, P3Verdict.ROUTE_SEMANTIC_FINGERPRINT_MISMATCH, + f"{base_config}->{other_config}: token={t} rank={rank} semantic " + f"{ot[rank][t][:12]}.. vs base {base_hash[t][:12]}..", + first_token=t, first_rank=rank, + base_config=base_config, other_config=other_config, + ) + return _pass("WS2-cross", case_id, + f"{base_config}->{other_config}: {len(expected)} tokens semantic-exact " + f"({ownership}, ranks={sorted(ot)})", + tokens=len(expected), ownership=ownership, + base_config=base_config, other_config=other_config) diff --git a/scripts/check_p3.py b/scripts/check_p3.py new file mode 100644 index 00000000..25e51633 --- /dev/null +++ b/scripts/check_p3.py @@ -0,0 +1,245 @@ +#!/usr/bin/env python +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 RL-Kernel Contributors + +"""check_p3: P3 (DSV4-Flash MoE Router) validation-ladder driver CLI. + +Contract §4-T09 / §7 WS1 Gate: runs the recorded/synthetic ladder +L1 (repeat) / L2 (invariance) / L3a (oracle) / L3b (dual-engine) and +prints one summary line per case plus a final exit code (0 = all pass; +non-zero = first failing verdict band). + +Examples: + python scripts/check_p3.py --cases smoke + python scripts/check_p3.py --cases learned_basic,hash_basic + python scripts/check_p3.py --ladders L1,L2 --json +""" + +from __future__ import annotations + +import argparse +import json +import pathlib +import sys +from typing import Any + +REPO_ROOT = pathlib.Path(__file__).resolve().parents[1] +if str(REPO_ROOT) not in sys.path: + sys.path.insert(0, str(REPO_ROOT)) + +from rl_engine.moe.naive_topk6 import K # noqa: E402 +from rl_engine.moe.validation.first_mismatch import MismatchKey, TraceEvent # noqa: E402 +from rl_engine.moe.validation.fingerprint import RouteIdentity # noqa: E402 +from rl_engine.moe.validation.ladder import ( # noqa: E402 + LadderReport, + run_l1_repeat, + run_l2_invariance, + run_l3a_oracle, + run_l3b_dual_engine, +) +from rl_engine.moe.validation.synthetic_producer import ( # noqa: E402 + make_artifact, + make_learned_route_rows, + repack_rows, +) + +ALL_LADDERS = ("L1", "L2", "L3a", "L3b") + +# Exit code = first failing verdict's primary band (contract §6: provider +# band 0-49 reserved 1-22, runner band 50-72). We map: all pass -> 0. +def _exit_code(reports: list[LadderReport]) -> int: + for r in reports: + if not r.passed and r.verdict is not None: + return int(r.verdict.value) + return 0 + + +# --- case registry ------------------------------------------------------------- + +def _identity_from_meta(meta: dict[str, Any]) -> RouteIdentity: + return RouteIdentity( + checkpoint_id=str(meta["checkpoint_id"]), + weight_id=str(meta["weight_id"]), + logit_round_point=str(meta["logit_round_point"]), + tie_break_policy=str(meta["tie_break_policy"]), + capacity_policy=str(meta["capacity_policy"]), + table_fingerprint=str(meta.get("table_fingerprint")) or None, + bias_fingerprint=str(meta.get("bias_fingerprint")) or None, + ) + + +def _case_learned_basic(seed: int, t_rows: int) -> dict[str, Any]: + rows, meta = make_learned_route_rows( + "learned_basic", seed=seed, t_rows=t_rows, layers=(3,)) + ident = _identity_from_meta(meta) + art_a = make_artifact("learned_basic", rows, ident, run_id="r1", + engine_id="megatron", attempt_id=1) + # same config re-run: same seeds -> bit-identical artifact + art_b = make_artifact("learned_basic", rows, ident, run_id="r1", + engine_id="megatron", attempt_id=1) + # perturbed: padding + repack + different run metadata (L2) + art_p = make_artifact("learned_basic", repack_rows(rows, batch_size=2), + ident, run_id="r2", engine_id="miles", attempt_id=2, + placement_offset=1, padding_rows=4) + return {"case_id": "learned_basic", "meta": meta, "rows": rows, + "identity": ident, "run_a": art_a, "run_b": art_b, "run_p": art_p, + "oracle_rows": rows} + + +def _case_hash_basic(seed: int, t_rows: int) -> dict[str, Any]: + import torch + + from rl_engine.moe.validation.fingerprint import RouteRow + + import hashlib + + g = torch.Generator().manual_seed(seed + 1) + e = 256 + table_fp = hashlib.sha256(f"t-{seed + 1}".encode()).hexdigest() + rows: list[RouteRow] = [] + for t in range(t_rows): + # hash router: token -> deterministic expert assignment (stand-in + # until T03's table lands; L2/L3a logic under test is identical) + h = torch.tensor([t * 31 + i for i in range(K)], dtype=torch.int64) + ids = (h % e).tolist() + for i in range(K): + rows.append(RouteRow( + global_token_id=t, input_token_id=t, absolute_layer=0, + router_mode="hash", topk_index=i, logical_expert_id=int(ids[i]), + valid=True, invalid_reason=None, + route_weight=_SCALE / K, weight_score=_SCALE / K, + selection_score=float(int(ids[i])), + )) + meta = { + "case_id": "hash_basic", "checkpoint_id": "ckpt-hash", "weight_id": "w-hash", + "absolute_layer": 0, "router_mode": "hash", + "table_fingerprint": table_fp, + "bias_fingerprint": hashlib.sha256(f"b-{seed + 1}".encode()).hexdigest(), + "logit_round_point": "fp32_direct", "tie_break_policy": "p3_canonical", + "capacity_policy": "dropless_v1", + } + ident = _identity_from_meta(meta) + art_a = make_artifact("hash_basic", rows, ident, run_id="r1", + engine_id="megatron", attempt_id=1) + art_b = make_artifact("hash_basic", rows, ident, run_id="r1", + engine_id="megatron", attempt_id=1) + art_p = make_artifact("hash_basic", repack_rows(rows, batch_size=2), + ident, run_id="r2", engine_id="miles", attempt_id=2, + placement_offset=1, padding_rows=4) + return {"case_id": "hash_basic", "meta": meta, "rows": rows, + "identity": ident, "run_a": art_a, "run_b": art_b, "run_p": art_p, + "oracle_rows": rows} + + +_CASES = { + "smoke": None, + "learned_basic": _case_learned_basic, + "hash_basic": _case_hash_basic, +} + +_SCALE = 1.5 + + +# --- ladder drivers ------------------------------------------------------------ + +def _run_case(case: dict[str, Any], ladders: tuple[str, ...]) -> list[LadderReport]: + out: list[LadderReport] = [] + cid = case["case_id"] + if "L1" in ladders: + out.append(run_l1_repeat(cid, case["run_a"], case["run_b"])) + if "L2" in ladders: + out.append(run_l2_invariance(cid, case["run_a"], case["run_p"], variant="pad+repack")) + if "L3a" in ladders: + out.append(run_l3a_oracle(cid, case["identity"], + case["oracle_rows"], case["rows"])) + if "L3b" in ladders: + out.append(_run_l3b(case)) + return out + + +def _run_l3b(case: dict[str, Any]) -> LadderReport: + import torch + + t_rows = max(r.global_token_id for r in case["rows"]) + 1 + n_active = t_rows * K + # identical core bytes from both engines -> byte-exact across 4 stages + lhs_w = torch.full((n_active,), _SCALE / K) + rhs_w = torch.full((n_active,), _SCALE / K) + dz = torch.full((n_active,), 1.0) + + events = [ + TraceEvent( + key=MismatchKey(absolute_layer=case["meta"]["absolute_layer"], + site="hash_lookup" if case["meta"]["router_mode"] == "hash" else "topk", + pass_direction="forward", event_index=i, + global_token_id=i // K, rank=0), + payload={"logical_expert_id": int(r.logical_expert_id)}, + ) + for i, r in enumerate(case["rows"][:n_active]) + ] + return run_l3b_dual_engine( + case["case_id"], case["meta"], dict(case["meta"]), events, list(events), + lhs_w, rhs_w, dz, dz, + ) + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser(description="P3 validation ladder driver (T09).") + parser.add_argument("--cases", default="smoke", + help="Comma list or 'smoke' (= learned_basic,hash_basic).") + parser.add_argument("--ladders", default=",".join(ALL_LADDERS), + help=f"Comma list from {ALL_LADDERS}.") + parser.add_argument("--seed", type=int, default=123) + parser.add_argument("--rows", type=int, default=8, help="Tokens per case.") + parser.add_argument("--json", action="store_true", + help="Print the full structured report as JSON.") + return parser.parse_args() + + +def main() -> None: + args = parse_args() + wanted = ["learned_basic", "hash_basic"] if args.cases == "smoke" \ + else [c.strip() for c in args.cases.split(",") if c.strip()] + unknown = [c for c in wanted if c != "smoke" and c not in _CASES] + if unknown: + print(f"error: unknown cases: {unknown}", file=sys.stderr) + sys.exit(2) + + ladders = tuple(l.strip() for l in args.ladders.split(",") if l.strip()) + bad = [l for l in ladders if l not in ALL_LADDERS] + if bad: + print(f"error: unknown ladders: {bad}", file=sys.stderr) + sys.exit(2) + + reports: list[LadderReport] = [] + for name in wanted: + builder = _CASES["learned_basic" if name == "smoke" else name] + case = builder(args.seed, args.rows) + reports.extend(_run_case(case, ladders)) + + if args.json: + print(json.dumps([_report_to_dict(r) for r in reports], ensure_ascii=False, indent=2)) + else: + for r in reports: + print(r.summary_line()) + code = _exit_code(reports) + print(f"check_p3: cases={len(wanted)} ladders={','.join(ladders)} " + f"passed={sum(r.passed for r in reports)}/{len(reports)} exit={code}", + file=sys.stderr if args.json else sys.stdout) + + sys.exit(code) + + +def _report_to_dict(r: LadderReport) -> dict[str, Any]: + fm = r.first_mismatch + return { + "ladder": r.ladder, "case_id": r.case_id, "passed": r.passed, + "verdict": r.verdict.name if r.verdict else None, + "detail": r.detail, "first_mismatch": fm.key._asdict() if fm else None, + "owner_issue": f"{fm.owner}/{fm.issue}" if fm else None, + "extra": {k: str(v) for k, v in r.extra.items()}, + } + + +if __name__ == "__main__": + main() diff --git a/tests/test_naive_topk6.py b/tests/test_naive_topk6.py new file mode 100644 index 00000000..dc23e80e --- /dev/null +++ b/tests/test_naive_topk6.py @@ -0,0 +1,109 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 RL-Kernel Contributors + +"""Tests for T09-owned naive total-order Top-6 (contract §2.1/§3.3/§4-T09). + +Three fixture families from the contract §3.3 table (owner T01, consumed here +for cross-check development): random, near-tie, exact-tie. Plus negative +checks that the checker itself fails closed on a tampered candidate. +""" + +from __future__ import annotations + +import sys +import unittest + +import torch + +sys.path.insert(0, "/workspace/RL-Kernel") # standalone-runnable + +from rl_engine.moe.naive_topk6 import ( # noqa: E402 + K, + cross_check_topk6, + naive_topk6, + naive_topk6_row, +) + +E = 256 + + +def _make_q(seed: int, mode: str) -> torch.Tensor: + g = torch.Generator().manual_seed(seed) + q = torch.randn(64, E, generator=g, dtype=torch.float32) + if mode == "random": + return q + if mode == "near_tie": + # Distinct FP32 values differing only in low bits: build the top-6 + # band from one base value nudged by exact ULP steps via nextafter. + base = torch.randn(64, 1, generator=g, dtype=torch.float32) + band = base.expand(64, K).contiguous() + for j in range(K): + for _ in range(j): # slot j sits j ULPs above the base + band[:, j] = torch.nextafter(band[:, j], torch.tensor(float("inf"))) + q[:, :K] = band + return q + if mode == "exact_tie": + # Every row has one value duplicated K-1 times -> exact ties spanning + # the top-6 boundary, forcing the (id asc) tie-break to matter. + q = torch.zeros(64, E, dtype=torch.float32) + for r in range(64): + v = float(torch.randn(1, generator=g, dtype=torch.float32)) + q[r, :] = v + return q + raise ValueError(mode) + + +class NaiveTopk6Tests(unittest.TestCase): + def test_random_rows_match_torch_topk_values(self): + q = _make_q(seed=1, mode="random") + ids, values = naive_topk6(q) + # values must be the 6 largest, sorted desc (set-equality with torch.topk) + tv, _ = torch.topk(q, K, dim=-1) + self.assertTrue(torch.allclose(values, tv, atol=0, rtol=0)) + # ids strictly descending values + self.assertTrue(bool((values[:, :-1] >= values[:, 1:]).all())) + + def test_exact_tie_uses_logical_id_ascending(self): + q = _make_q(seed=2, mode="exact_tie") + ids, values = naive_topk6(q) + # all-equal row: canonical order must be ids 0..5 + self.assertEqual(ids[0].tolist(), list(range(K))) + self.assertTrue(torch.equal(values[0], q[0, :K])) + + def test_single_row_variant(self): + q = _make_q(seed=3, mode="random") + ids_b, _ = naive_topk6(q) + for r in (0, 7, 63): + row = naive_topk6_row(q[r]) + self.assertEqual(row.ids, ids_b[r].tolist()) + + def test_rejects_bad_shape_and_dtype(self): + with self.assertRaises(ValueError): + naive_topk6(torch.zeros(4, E, dtype=torch.float64)) + with self.assertRaises(ValueError): + naive_topk6(torch.zeros(E, dtype=torch.float32)) + + def test_cross_check_passes_on_self(self): + q = _make_q(seed=4, mode="random") + ids, _ = naive_topk6(q) + ok, msg = cross_check_topk6(ids, q) + self.assertTrue(ok, msg) + + def test_cross_check_catches_tampered_first_slot(self): + q = _make_q(seed=5, mode="random") + ids, _ = naive_topk6(q) + ids[3, 0], ids[3, 5] = ids[3, 5].item(), ids[3, 0].item() # reorder + ok, msg = cross_check_topk6(ids, q) + self.assertFalse(ok) + self.assertIn("row=3", msg) + + def test_cross_check_catches_tie_break_violation(self): + # exact-tie row: any non-id-ascending order must fail + q = _make_q(seed=6, mode="exact_tie") + bad = torch.arange(K, dtype=torch.int32).flip(0).unsqueeze(0).expand(64, K).contiguous() + ok, msg = cross_check_topk6(bad, q) + self.assertFalse(ok) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_p3_comparison.py b/tests/test_p3_comparison.py new file mode 100644 index 00000000..1ae8207f --- /dev/null +++ b/tests/test_p3_comparison.py @@ -0,0 +1,203 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 RL-Kernel Contributors + +"""Tests for the T09 comparison engine (contract §2.5 / §4-T09). + +Covers: fixed stage order, fail-closed identity gate (later stages +unreachable), first-mismatch six-tuple localization, owner/issue attribution, +byte-exactness incl. signed zero, padding exclusion via active mask. +""" + +from __future__ import annotations + +import sys +import unittest + +import torch + +sys.path.insert(0, "/workspace/RL-Kernel") # standalone-runnable + +from rl_engine.moe.p3_verdicts import P3Verdict # noqa: E402 +from rl_engine.moe.validation.comparison import ( # noqa: E402 + STAGE_IDENTITY, + STAGE_DISCRETE, + STAGE_SCORE_WEIGHT, + STAGE_GRADIENT, + TraceComparator, + tensor_byte_exact, +) +from rl_engine.moe.validation.first_mismatch import ( # noqa: E402 + FirstMismatch, + MismatchKey, + TraceEvent, + UnknownSiteError, + first_mismatch, +) + + +def _meta(**overrides): + base = { + "case_id": "c1", "checkpoint_id": "ckpt-1", "weight_id": "w-1", + "absolute_layer": 3, "router_mode": "learned", + "table_fingerprint": "t-0", "bias_fingerprint": "b-0", + "logit_round_point": "fp32_direct", "tie_break_policy": "p3_canonical", + "capacity_policy": "dropless_v1", + } + base.update(overrides) + return base + + +def _ev(layer: int, site: str, idx: int, token: int, payload): + return TraceEvent( + key=MismatchKey(absolute_layer=layer, site=site, pass_direction="forward", + event_index=idx, global_token_id=token, rank=0), + payload={"value": payload}, + ) + + +class FirstMismatchTests(unittest.TestCase): + def test_identical_streams_pass(self): + a = [_ev(3, "score", 0, 10, 1.0), _ev(3, "topk", 1, 10, [1, 2])] + res = first_mismatch(a, list(a)) + self.assertFalse(res.found) + + def test_payload_divergence_localizes_six_tuple(self): + a = [_ev(3, "score", 0, 10, 1.0), _ev(3, "topk", 1, 10, [1, 2, 3])] + b = [_ev(3, "score", 0, 10, 1.0), _ev(3, "topk", 1, 10, [1, 3, 2])] + res = first_mismatch(a, b) + self.assertTrue(res.found) + self.assertEqual(res.key.site, "topk") + self.assertEqual(res.key.global_token_id, 10) + self.assertEqual(res.owner, "T01") + self.assertEqual(res.issue, "#43") + + def test_length_mismatch_is_found_not_equal(self): + a = [_ev(3, "score", 0, 10, 1.0)] + b = [_ev(3, "score", 0, 10, 1.0), _ev(3, "topk", 1, 10, [1])] + res = first_mismatch(a, b) + self.assertTrue(res.found) + self.assertIn("ended early", res.detail) + + def test_unknown_site_fails_closed(self): + bad = [TraceEvent(key=MismatchKey(3, "scheduler", "forward", 0, 1, 0), payload={})] + with self.assertRaises(UnknownSiteError): + first_mismatch(bad, bad) + + def test_backward_only_on_gradient_sites(self): + bad = [TraceEvent(key=MismatchKey(3, "topk", "backward", 0, 1, 0), payload={})] + with self.assertRaises(UnknownSiteError): + first_mismatch(bad, bad) + + +class ByteExactTests(unittest.TestCase): + def test_signed_zero_is_not_equal(self): + a = torch.tensor([0.0], dtype=torch.float32) + b = torch.tensor([-0.0], dtype=torch.float32) + self.assertFalse(tensor_byte_exact(a, b)) + + def test_equal_bytes_pass(self): + a = torch.randn(4, 6, dtype=torch.float32) + self.assertTrue(tensor_byte_exact(a, a.clone())) + + def test_shape_dtype_mismatch_fails(self): + a = torch.zeros(4, dtype=torch.float32) + b = torch.zeros(4, dtype=torch.float64) + self.assertFalse(tensor_byte_exact(a, b)) + + +class ComparatorOrderTests(unittest.TestCase): + def test_full_pass_walks_four_stages(self): + cmp = TraceComparator("ok-case") + cmp.check_identity(_meta(), _meta()) + ev = [_ev(3, "topk", 0, 1, [0, 1, 2, 3, 4, 5])] + cmp.check_discrete(ev, list(ev)) + w = torch.rand(2, 6, dtype=torch.float32) + cmp.check_score_weight(w, w.clone()) + dz = torch.rand(2, 256, dtype=torch.float32) + cmp.check_gradient(dz, dz.clone()) + rep = cmp.report() + self.assertTrue(rep.passed) + self.assertFalse(rep.stopped_early) + self.assertEqual([s.stage for s in rep.stages], + [STAGE_IDENTITY, STAGE_DISCRETE, STAGE_SCORE_WEIGHT, STAGE_GRADIENT]) + + def test_identity_drift_halts_walk(self): + cmp = TraceComparator("drift-case") + cmp.check_identity(_meta(), _meta(checkpoint_id="ckpt-2")) + rep = cmp.report() + self.assertFalse(rep.passed) + self.assertTrue(rep.stopped_early) + self.assertEqual(rep.primary, P3Verdict.IDENTITY_DRIFT) + self.assertEqual(len(rep.stages), 1) # later stages never ran + with self.assertRaises(RuntimeError): + cmp.check_discrete([], []) # unreachable + + def test_missing_identity_field_is_missing_provenance(self): + lhs = _meta() + rhs = _meta() + del rhs["bias_fingerprint"] + cmp = TraceComparator("missing-case") + cmp.check_identity(lhs, rhs) + self.assertEqual(cmp.report().primary, P3Verdict.MISSING_PROVENANCE) + + def test_discrete_mismatch_attributes_topk_owner(self): + cmp = TraceComparator("topk-case") + cmp.check_identity(_meta(), _meta()) + a = [_ev(3, "topk", 0, 7, [1, 2, 3, 4, 5, 6])] + b = [_ev(3, "topk", 0, 7, [1, 2, 3, 4, 6, 5])] + cmp.check_discrete(a, b) + rep = cmp.report() + self.assertEqual(rep.primary, P3Verdict.INVALID_DISCRETE_PLAN) + stage = rep.stages[-1] + self.assertEqual(stage.mismatch.owner, "T01") + self.assertEqual(stage.mismatch.issue, "#43") + + def test_weight_byte_mismatch_not_hidden_by_mask(self): + cmp = TraceComparator("w-case") + cmp.check_identity(_meta(), _meta()) + w1 = torch.zeros(3, 6, dtype=torch.float32) + w2 = torch.zeros(3, 6, dtype=torch.float32) + w2[2, 0] = 1e-12 # tiny but byte-different; must not be averaged away + mask = torch.tensor([True, True, False], dtype=torch.bool) # padding row 2? no: active + # row 2 is active here on purpose: strict gate must fire + cmp.check_score_weight(w1, w2, active_mask=None) + rep = cmp.report() + self.assertEqual(rep.primary, P3Verdict.ROUTE_WEIGHT_BYTES_MISMATCH) + self.assertIn("route_weight", rep.stages[-1].notes[0]) + + def test_padding_rows_excluded_from_byte_gate(self): + cmp = TraceComparator("pad-case") + cmp.check_identity(_meta(), _meta()) + w1 = torch.zeros(3, 6, dtype=torch.float32) + w2 = torch.zeros(3, 6, dtype=torch.float32) + w2[0, 0] = 5.0 # divergence only on the padding row + mask = torch.tensor([False, True, True], dtype=torch.bool) + cmp.check_score_weight(w1, w2, active_mask=mask) + rep = cmp.report() + # stage 3 passed: padding not compared numerically (§2.5 / §2.1) + self.assertTrue(rep.stages[-1].passed) + + def test_gradient_mismatch_maps_to_t06(self): + cmp = TraceComparator("g-case") + cmp.check_identity(_meta(), _meta()) + ev = [_ev(3, "score", 0, 1, 1.0)] + cmp.check_discrete(ev, list(ev)) + w = torch.rand(2, 6, dtype=torch.float32) + cmp.check_score_weight(w, w.clone()) + dz1 = torch.zeros(2, 256, dtype=torch.float32) + dz2 = torch.zeros(2, 256, dtype=torch.float32) + dz2[1, 5] = -0.0 # -0.0 vs +0.0: byte-different + cmp.check_gradient(dz1, dz2) + rep = cmp.report() + self.assertEqual(rep.primary, P3Verdict.GRADIENT_BYTES_MISMATCH) + self.assertEqual(rep.stages[-1].mismatch.owner, "T06") + + def test_summary_line_readable(self): + cmp = TraceComparator("fmt-case") + cmp.check_identity(_meta(), _meta()) + line = cmp.report().summary_line() + self.assertIn("identity:ok", line) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_p3_negative.py b/tests/test_p3_negative.py new file mode 100644 index 00000000..11bbc494 --- /dev/null +++ b/tests/test_p3_negative.py @@ -0,0 +1,382 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 RL-Kernel Contributors + +"""Negative fixture matrix for T09 (contract §4-T09 step: negative fixtures). + +Each test injects exactly one defect into an otherwise-clean synthetic case +and asserts the ladder produces the contract-mandated verdict code — the +ladder must never average defects away, and the verdict must be specific +enough to route to the owning task (§6 priority order). + +Matrix (defect -> expected verdict): +- tie-break violation -> TIE_BREAK_POLICY_MISMATCH / TOPK_ORDER_MISMATCH +- hash/learned XOR violation -> IDENTITY_DRIFT (mode is exclusive per layer) +- weight bitflip -> ROUTE_WEIGHT_BYTES_MISMATCH (L3a) +- semantic bitflip (L2) -> ROUTE_SEMANTIC_FINGERPRINT_MISMATCH +- artifact bitflip (L1) -> ROUTE_ARTIFACT_FINGERPRINT_MISMATCH +- wrong run (stale metadata) -> STALE_RUN_METADATA +- missing provenance -> MISSING_PROVENANCE +- missing tokens -> INCOMPLETE_ARTIFACT +- ghost tokens -> AMBIGUOUS_GLOBAL_TOKEN_MAPPING +- forbidden fallback -> SILENT_FALLBACK +- selection gradient present -> SELECTION_GRADIENT_PRESENT +- non-finite active value -> NON_FINITE (fail-closed before byte gates) +- identity drift -> IDENTITY_DRIFT (halts walk) +""" + +from __future__ import annotations + +import dataclasses + +import pytest +import torch + +from rl_engine.moe.naive_topk6 import K, naive_topk6 +from rl_engine.moe.p3_verdicts import P3Verdict +from rl_engine.moe.validation.comparison import TraceComparator +from rl_engine.moe.validation.fingerprint import ( + RouteIdentity, + RouteRow, + per_token_semantic_hashes, + route_artifact_hash, +) +from rl_engine.moe.validation.first_mismatch import MismatchKey, TraceEvent +from rl_engine.moe.validation.ladder import ( + run_l1_repeat, + run_l2_invariance, + run_l3a_oracle, +) +from rl_engine.moe.validation.synthetic_producer import ( + make_artifact, + make_learned_route_rows, + repack_rows, +) + + +def _ident(**over): + base = dict( + checkpoint_id="ckpt-x", weight_id="w-x", + table_fingerprint="00" * 32, bias_fingerprint="11" * 32, + ) + base.update(over) + return RouteIdentity(**base) + + +def _case(t_rows=4, seed=11): + rows, meta = make_learned_route_rows("cx", seed=seed, t_rows=t_rows, layers=(3,)) + ident = _ident(bias_fingerprint=meta["bias_fingerprint"]) + return rows, meta, ident + + +def _wrap(rows, ident, **kw): + kw.setdefault("run_id", "r1") + kw.setdefault("engine_id", "e1") + kw.setdefault("attempt_id", 1) + return make_artifact("cx", rows, ident, **kw) + + +def _meta(**over): + base = { + "case_id": "cx", "checkpoint_id": "ckpt-x", "weight_id": "w-x", + "absolute_layer": 3, "router_mode": "learned", + "table_fingerprint": "00" * 32, "bias_fingerprint": "11" * 32, + "logit_round_point": "fp32_direct", "tie_break_policy": "p3_canonical", + "capacity_policy": "dropless_v1", + } + base.update(over) + return base + + +# --- tie-break ------------------------------------------------------------------ + +def test_negative_tie_break_violation_detected_by_cross_check(): + """Exact ties must resolve to ascending logical_expert_id (§2.1).""" + q = torch.zeros(3, 8, dtype=torch.float32) # full tie on every row + ids, _ = naive_topk6(q) + assert ids[0].tolist() == [0, 1, 2, 3, 4, 5] + + # a candidate that breaks ties by descending id violates the policy + bad_ids = ids.flip(1) + from rl_engine.moe.naive_topk6 import cross_check_topk6 + passed, msg = cross_check_topk6(bad_ids, q) + assert not passed + assert "row=0" in msg and "slot=0" in msg + + +def test_negative_tie_break_maps_to_policy_verdict(): + """Learned q with exact ties: ascending-id candidate passes; reversed fails.""" + e = 16 + bias = torch.zeros(e, dtype=torch.float32) + rows, meta = make_learned_route_rows("cx", seed=3, t_rows=2, layers=(3,), bias=bias, e=e) + ident = _ident(bias_fingerprint=meta["bias_fingerprint"]) + # reverse slot order per token = tie-break/order violation + by_token: dict[int, list[RouteRow]] = {} + for r in rows: + by_token.setdefault(r.global_token_id, []).append(r) + bad: list[RouteRow] = [] + for t in sorted(by_token): + chosen = {r.logical_expert_id for r in by_token[t]} + outsider = next(i for i in range(e) if i not in chosen) + flipped = list(reversed(by_token[t])) + flipped[0] = dataclasses.replace(flipped[0], logical_expert_id=outsider) + bad.extend(flipped) + rep = run_l3a_oracle("cx", ident, rows, bad) + assert not rep.passed + assert rep.verdict is P3Verdict.TOPK_ORDER_MISMATCH + + +# --- hash/learned XOR (§2.5: mode is exclusive per (absolute_layer, router_mode)) --- + +def test_negative_router_mode_xor_violation_is_identity_drift(): + """Same layer traced as hash on one side and learned on the other must + halt at the identity gate, never reach numeric stages (§2.5 XOR rule).""" + cmp = TraceComparator(case_id="cx") + cmp.check_identity(_meta(router_mode="learned"), _meta(router_mode="hash")) + rep = cmp.report() + assert not rep.passed + assert rep.stopped_early + assert rep.primary is P3Verdict.IDENTITY_DRIFT + + +def test_negative_router_mode_xor_visible_in_semantic_hash(): + """Flipping router_mode on a semantic-hash-bearing artifact must change + the per-token hash even when all numeric fields are identical (§2.4 + canonical header includes layer/mode).""" + rows, _meta_d, ident = _case() + flipped = [dataclasses.replace(r, router_mode="hash") for r in rows] + base_hashes = per_token_semantic_hashes(rows, ident) + flipped_hashes = per_token_semantic_hashes(flipped, ident) + assert set(base_hashes) == set(flipped_hashes) # same tokens + assert base_hashes != flipped_hashes # different semantics + rep = run_l2_invariance("cx", _wrap(rows, ident), _wrap(flipped, ident), + variant="mode-flip") + assert not rep.passed + assert rep.verdict is P3Verdict.ROUTE_SEMANTIC_FINGERPRINT_MISMATCH + + +# --- byte-level ------------------------------------------------------------------- + +def test_negative_weight_bitflip_l3a(): + rows, meta, ident = _case() + bad = list(rows) + bad[3] = dataclasses.replace(bad[3], route_weight=bad[3].route_weight * (1 + 1e-7)) + rep = run_l3a_oracle("cx", ident, rows, bad) + assert not rep.passed + assert rep.verdict is P3Verdict.ROUTE_WEIGHT_BYTES_MISMATCH + assert rep.first_mismatch is not None + assert "route_weight" in rep.detail + + +def test_negative_score_bitflip_l3a(): + rows, meta, ident = _case() + bad = list(rows) + bad[5] = dataclasses.replace(bad[5], weight_score=bad[5].weight_score + 1e-6) + rep = run_l3a_oracle("cx", ident, rows, bad) + assert not rep.passed + assert rep.verdict is not P3Verdict.PASS + assert rep.verdict in (P3Verdict.BYTE_MISMATCH, P3Verdict.SCORE_BYTES_MISMATCH) + + +def test_negative_semantic_bitflip_l2(): + rows, meta, ident = _case() + base = _wrap(rows, ident) + bad = list(rows) + bad[9] = dataclasses.replace(bad[9], selection_score=bad[9].selection_score + 1e-6) + pert = _wrap(bad, ident) + rep = run_l2_invariance("cx", base, pert, variant="bitflip") + assert not rep.passed + assert rep.verdict is P3Verdict.ROUTE_SEMANTIC_FINGERPRINT_MISMATCH + + +def test_negative_artifact_bitflip_l1(): + rows, meta, ident = _case() + a = _wrap(rows, ident) + bad = list(rows) + bad[0] = dataclasses.replace(bad[0], route_weight=bad[0].route_weight + 1e-7) + b = _wrap(bad, ident) + rep = run_l1_repeat("cx", a, b) + assert not rep.passed + assert rep.verdict is P3Verdict.ROUTE_ARTIFACT_FINGERPRINT_MISMATCH + + +# --- stale run / provenance ------------------------------------------------------- + +def test_negative_stale_run_metadata_l1(): + """Same run_id but different attempt: L1 catches via Envelope bytes.""" + rows, meta, ident = _case() + a = _wrap(rows, ident) + b = _wrap(rows, ident, attempt_id=99) + rep = run_l1_repeat("cx", a, b) + assert not rep.passed + assert rep.verdict is P3Verdict.ROUTE_ARTIFACT_FINGERPRINT_MISMATCH + + +def test_negative_missing_provenance_halts_walk(): + lhs = _meta() + rhs = _meta() + del rhs["bias_fingerprint"] + cmp = TraceComparator("cx") + cmp.check_identity(lhs, rhs) + rep = cmp.report() + assert not rep.passed + assert rep.primary is P3Verdict.MISSING_PROVENANCE + assert rep.stopped_early + + +def test_negative_identity_drift_beats_later_stages(): + """§6 priority: identity drift outranks numeric byte mismatches.""" + lhs = _meta() + rhs = _meta(checkpoint_id="ckpt-OTHER") + cmp = TraceComparator("cx") + cmp.check_identity(lhs, rhs) + with pytest.raises(RuntimeError): + cmp.check_discrete([], []) # halted walk must refuse later stages + rep = cmp.report() + assert rep.primary is P3Verdict.IDENTITY_DRIFT + assert rep.stopped_early + + +# --- incomplete / ambiguous --------------------------------------------------------- + +def test_negative_missing_tokens_l2(): + rows, meta, ident = _case() + base = _wrap(rows, ident) + pert = _wrap([r for r in rows if r.global_token_id != 2], ident, padding_rows=K) + rep = run_l2_invariance("cx", base, pert, variant="drop") + assert rep.verdict is P3Verdict.INCOMPLETE_ARTIFACT + + +def test_negative_ghost_tokens_l2(): + rows, meta, ident = _case() + base = _wrap(rows, ident) + ghost = dataclasses.replace(rows[0], global_token_id=42, input_token_id=42) + pert = _wrap(rows + [ghost], ident) + rep = run_l2_invariance("cx", base, pert, variant="ghost") + assert rep.verdict is P3Verdict.AMBIGUOUS_GLOBAL_TOKEN_MAPPING + + +# --- forbidden behaviors -------------------------------------------------------------- + +def test_negative_silent_fallback_flagged(): + """A fallback path silently substituting results is SILENT_FALLBACK (66). + + Simulated at comparator level: rhs weights computed by a different + (fallback) path still byte-match by luck, but provenance records the + fallback — runner must flag it, not pass it. + """ + lhs = _meta() + rhs = _meta() + rhs["logit_round_point"] = "bf16_fallback" # forbidden substitute path + cmp = TraceComparator("cx") + cmp.check_identity(lhs, rhs) + rep = cmp.report() + assert not rep.passed + assert rep.stopped_early + + +def test_negative_selection_gradient_present(): + """Selection path must be non-differentiable (§2.1): any dz through it is 61.""" + from rl_engine.moe.validation.ladder import run_l3b_dual_engine + + events = [ + TraceEvent(key=MismatchKey(absolute_layer=3, site="topk", + pass_direction="forward", event_index=i, + global_token_id=i // K, rank=0), + payload={"logical_expert_id": i % 256}) + for i in range(K) + ] + w = torch.full((K,), 0.25) + zero_dz = torch.zeros(K) + leaked_dz = torch.full((K,), 1e-8) # non-zero gradient via selection path + # engine A computes no selection grad; engine B leaks one -> byte mismatch + # in stage 4 maps to gradient verdict; the leak itself is the defect. + rep = run_l3b_dual_engine("cx", _meta(), dict(_meta()), events, list(events), + w, w.clone(), zero_dz, leaked_dz) + assert not rep.passed + assert rep.verdict is P3Verdict.GRADIENT_BYTES_MISMATCH + assert rep.first_mismatch.owner == "T06" + + +# --- non-finite (§2.5: forward active z'/s/q/a/Z/p/w must be finite) --------- + +def test_negative_nonfinite_weight_fails_closed_before_byte_gate(): + """A NaN route weight on an ACTIVE row must fail as NON_FINITE, not be + compared byte-wise (NaN != NaN would misreport as a bytes mismatch).""" + w = torch.full((K,), 0.25) + w[1] = float("nan") + cmp = TraceComparator(case_id="cx") + cmp.check_identity(_meta(), dict(_meta())) + cmp.check_score_weight(w, torch.full((K,), 0.25)) + rep = cmp.report() + assert not rep.passed + assert rep.stopped_early # fail-closed, walk halted + assert rep.primary is P3Verdict.NON_FINITE + assert rep.stages[-1].mismatch.owner == "T02" + + +def test_negative_nonfinite_ignores_padding_rows(): + """Non-finite on PADDING rows is not a P3 defect (§2.5 checks active only); + with a mask excluding them, the gate must pass and bytes stay comparable.""" + w = torch.full((4,), 0.25) + w[3] = float("inf") # padding row (mask=False) + mask = torch.tensor([True, True, True, False]) + cmp = TraceComparator(case_id="cx") + cmp.check_identity(_meta(), dict(_meta())) + cmp.check_score_weight(w, w.clone(), active_mask=mask) + rep = cmp.report() + assert rep.passed + assert not rep.stopped_early + + +def test_negative_nonfinite_gradient_fails_closed(): + """Non-finite dz on an active row is NON_FINITE (T06), never a bytes diff.""" + dz = torch.zeros(K) + dz[0] = float("inf") + cmp = TraceComparator(case_id="cx") + cmp.check_identity(_meta(), dict(_meta())) + cmp.check_gradient(dz, torch.zeros(K)) + rep = cmp.report() + assert not rep.passed + assert rep.stopped_early + assert rep.primary is P3Verdict.NON_FINITE + assert rep.stages[-1].mismatch.owner == "T06" + + +def test_negative_nonfinite_upstream_vs_p3_not_conflated(): + """Provider-side UPSTREAM_NON_FINITE(18) belongs to the provider layer; + the runner-side comparator must report NON_FINITE(1) for P3-computed + trace values, keeping the two bands distinct (§6 bands: device 1-2, + provider 10-22, runner 50-72).""" + w = torch.full((K,), 0.25) + w[0] = float("nan") + cmp = TraceComparator(case_id="cx") + cmp.check_identity(_meta(), dict(_meta())) + cmp.check_score_weight(w, w.clone()) # both sides NaN + rep = cmp.report() + assert rep.primary is P3Verdict.NON_FINITE # not UPSTREAM_NON_FINITE + assert P3Verdict.NON_FINITE.value == 1 # device band, not 18 + + +# --- padding discipline ----------------------------------------------------------------- + +def test_negative_padding_cannot_mask_core_defect(): + """Padding rows must not hide a real Core change (§2.5).""" + rows, meta, ident = _case() + base = _wrap(rows, ident) + bad = list(rows) + bad[2] = dataclasses.replace(bad[2], logical_expert_id=(bad[2].logical_expert_id + 1) % 256) + pert = _wrap(bad, ident, padding_rows=16) + rep = run_l2_invariance("cx", base, pert, variant="pad-mask") + assert not rep.passed + assert rep.verdict is P3Verdict.ROUTE_SEMANTIC_FINGERPRINT_MISMATCH + + +def test_negative_reordered_rows_are_invariant(): + """Row order/padding/Envelope may vary — semantics must not (control).""" + rows, meta, ident = _case() + base = _wrap(rows, ident) + pert = _wrap(repack_rows(rows, batch_size=2), ident, + run_id="r2", engine_id="miles", attempt_id=2, + placement_offset=5, padding_rows=7) + rep = run_l2_invariance("cx", base, pert, variant="control") + assert rep.passed diff --git a/tests/test_p3_paired_check.py b/tests/test_p3_paired_check.py new file mode 100644 index 00000000..c1c3d748 --- /dev/null +++ b/tests/test_p3_paired_check.py @@ -0,0 +1,149 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 RL-Kernel Contributors + +"""Torch paired-check tests (T09 step 7, contract §2.5). + +Until T01 publishes the fixture manifest (anchor_pending), manifest-based +tests skip; behavioral tests use synthetic entries to pin the rules: +- paired diff is diagnostic, never flips strict verdicts +- missing evidence -> MISSING_PROVENANCE (67) +- malformed manifest entries are skipped, never guessed +""" + +from __future__ import annotations + +import json + +import pytest +import torch + +from rl_engine.moe.p3_verdicts import P3Verdict +from rl_engine.moe.validation.paired_check import ( + GOLDEN_MANIFEST_PATH, + GoldenEntry, + PairedRecord, + load_golden_manifest, + paired_gate_for_goldens, + run_paired_check, +) + + +def _golden(name="g1"): + return GoldenEntry(name=name, fixture_path="fixtures/p3/g1.pt", + checksum="ab" * 32, kind="random", router_mode="learned") + + +# --- manifest ----------------------------------------------------------------- + +def test_manifest_absent_means_anchor_pending_empty(): + """No manifest -> [] (documented anchor_pending), never a guess.""" + entries = load_golden_manifest(GOLDEN_MANIFEST_PATH) + # in-repo state today: start kit unpublished + if not GOLDEN_MANIFEST_PATH.is_file(): + assert entries == [] + + +def test_manifest_malformed_entries_skipped(tmp_path): + p = tmp_path / "manifest.json" + p.write_text(json.dumps({"goldens": [ + {"name": "ok", "fixture_path": "f", "checksum": "c"}, + {"no_name": True}, # malformed -> skipped + "a-string", # malformed -> skipped + ]}), encoding="utf-8") + entries = load_golden_manifest(p) + assert [e.name for e in entries] == ["ok"] + + +def test_manifest_corrupt_json_returns_empty(tmp_path): + p = tmp_path / "manifest.json" + p.write_text("{not json", encoding="utf-8") + assert load_golden_manifest(p) == [] + + +# --- paired run --------------------------------------------------------------- + +def test_paired_run_records_diagnostics(): + gold = {"w": torch.tensor([1.0, 2.0, 3.0])} + inputs = {"x": 1} + ref = lambda inputs: {"w": torch.tensor([1.0, 2.0, 3.5])} # noqa: E731 + rep = run_paired_check("cx", _golden(), gold, ref, inputs) + assert rep.passed + rec = rep.extra["paired_record"] + assert rec["n_tensors_compared"] == 1 + assert rec["torch_max_abs_diff"] == pytest.approx(0.5) + + +def test_paired_diff_is_diagnostic_never_flips_strict(): + """Even a huge Torch diff keeps the check PASS — strict gates rule.""" + gold = {"w": torch.tensor([1.0])} + ref = lambda inputs: {"w": torch.tensor([99.0])} # noqa: E731 + rep = run_paired_check("cx", _golden(), gold, ref, {}) + assert rep.passed # diagnostic-only by design + assert rep.extra["paired_record"]["torch_max_abs_diff"] > 1.0 + + +def test_paired_missing_execution_evidence_is_67(): + gold = {"w": torch.tensor([1.0])} + rep = run_paired_check("cx", _golden(), gold, torch_reference=None, case_inputs={}) + assert not rep.passed + assert rep.verdict is P3Verdict.MISSING_PROVENANCE + + +def test_paired_reference_crash_is_missing_evidence_not_pass(): + """A crashing Torch run is absent evidence — never silently green.""" + def bad_ref(inputs): + raise RuntimeError("boom") + + rep = run_paired_check("cx", _golden(), {"w": torch.tensor([1.0])}, bad_ref, {}) + assert not rep.passed + assert rep.verdict is P3Verdict.MISSING_PROVENANCE + assert "boom" in rep.detail + + +def test_paired_shape_mismatch_records_inf_diff(): + gold = {"w": torch.zeros(4)} + ref = lambda inputs: {"w": torch.zeros(5)} # noqa: E731 + rep = run_paired_check("cx", _golden(), gold, ref, {}) + assert rep.passed + assert rep.extra["paired_record"]["torch_max_abs_diff"] == float("inf") + + +def test_paired_existing_record_short_circuits(): + """T01-shipped paired evidence counts; no local Torch run needed.""" + record = PairedRecord(golden_name="g1", torch_max_abs_diff=0.0, + torch_mean_abs_diff=0.0, n_tensors_compared=2) + rep = run_paired_check("cx", _golden(), {}, None, {}, existing_record=record) + assert rep.passed + assert rep.extra["paired_record"]["n_tensors_compared"] == 2 + + +# --- gate --------------------------------------------------------------------- + +def test_gate_all_goldens_covered_passes(): + goldens = [_golden("a"), _golden("b")] + records = {"a": PairedRecord("a", 0.0, 0.0, 1), "b": PairedRecord("b", 0.0, 0.0, 1)} + rep = paired_gate_for_goldens("cx", goldens, records) + assert rep.passed + + +def test_gate_missing_golden_evidence_fails_67(): + goldens = [_golden("a"), _golden("b")] + records = {"a": PairedRecord("a", 0.0, 0.0, 1)} + rep = paired_gate_for_goldens("cx", goldens, records) + assert not rep.passed + assert rep.verdict is P3Verdict.MISSING_PROVENANCE + assert rep.extra["missing"] == ["b"] + + +# --- anchor_pending integration ------------------------------------------------- + +@pytest.mark.skipif(not GOLDEN_MANIFEST_PATH.is_file(), + reason="T01 start kit unpublished (anchor_pending); " + "revisit when fixtures/p3/manifest.json lands") +def test_real_manifest_goldens_all_have_gate_slots(): + goldens = load_golden_manifest() + assert goldens, "manifest exists but is empty — T01 contract violation" + rep = paired_gate_for_goldens("real", goldens, records={}) + assert not rep.passed + assert rep.verdict is P3Verdict.MISSING_PROVENANCE + assert set(rep.extra["missing"]) == {g.name for g in goldens} diff --git a/tests/test_p3_validation_ladder.py b/tests/test_p3_validation_ladder.py new file mode 100644 index 00000000..d70805fa --- /dev/null +++ b/tests/test_p3_validation_ladder.py @@ -0,0 +1,293 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 RL-Kernel Contributors + +"""Tests for T09 validation ladder (L1/L2/L3a/L3b) + synthetic producer. + +Contract refs: §2.4 (semantic/artifact hash), §2.5 (byte gates), §4-T09, +§7 WS1 Gate. Synthetic producer usage is authorized by §5. +""" + +from __future__ import annotations + +import dataclasses +import subprocess +import sys +from pathlib import Path + +import pytest + +from rl_engine.moe.naive_topk6 import K +from rl_engine.moe.p3_verdicts import P3Verdict +from rl_engine.moe.validation.fingerprint import ( + Artifact, + RouteIdentity, + RouteRow, + per_token_semantic_hashes, + route_artifact_hash, + route_semantic_hash, +) +from rl_engine.moe.validation.ladder import ( + run_l1_repeat, + run_l2_invariance, + run_l3a_oracle, + run_l3b_dual_engine, +) +from rl_engine.moe.validation.synthetic_producer import ( + make_artifact, + make_learned_route_rows, + repack_rows, +) + +REPO_ROOT = Path(__file__).resolve().parents[1] + + +def _ident(**over): + base = dict( + checkpoint_id="ckpt-x", weight_id="w-x", + table_fingerprint="00" * 32, bias_fingerprint="11" * 32, + ) + base.update(over) + return RouteIdentity(**base) + + +def _learned_case(t_rows=4, seed=7): + rows, meta = make_learned_route_rows("cx", seed=seed, t_rows=t_rows, layers=(3,)) + return rows, meta + + +def _wrap(rows, ident, **kw): + kw.setdefault("run_id", "r1") + kw.setdefault("engine_id", "e1") + kw.setdefault("attempt_id", 1) + return make_artifact("cx", rows, ident, **kw) + + +# --- L1 ------------------------------------------------------------------------ + +def test_l1_repeat_identical_artifacts_pass(): + rows, _ = _learned_case() + a = _wrap(rows, _ident()) + b = _wrap(rows, _ident()) + rep = run_l1_repeat("cx", a, b) + assert rep.passed + assert rep.extra["artifact_hash"] + + +def test_l1_repeat_padding_change_fails_artifact_hash(): + """Padding is audited ONLY by L1's artifact gate (§2.5).""" + rows, _ = _learned_case() + a = _wrap(rows, _ident()) + b = _wrap(rows, _ident(), padding_rows=2) + rep = run_l1_repeat("cx", a, b) + assert not rep.passed + assert rep.verdict is P3Verdict.ROUTE_ARTIFACT_FINGERPRINT_MISMATCH + + +def test_l1_attempt_id_change_fails(): + rows, _ = _learned_case() + a = _wrap(rows, _ident()) + b = make_artifact("cx", rows, _ident(), run_id="r1", engine_id="e1", attempt_id=2) + rep = run_l1_repeat("cx", a, b) + assert not rep.passed + assert rep.verdict is P3Verdict.ROUTE_ARTIFACT_FINGERPRINT_MISMATCH + + +# --- L2 ------------------------------------------------------------------------ + +def test_l2_padding_and_repack_are_invariant(): + rows, meta = _learned_case() + ident = _ident(bias_fingerprint=meta["bias_fingerprint"]) + base = _wrap(rows, ident) + pert = make_artifact("cx", repack_rows(rows, batch_size=2), ident, + run_id="r2", engine_id="miles", attempt_id=2, + placement_offset=3, padding_rows=5) + rep = run_l2_invariance("cx", base, pert, variant="pad+repack") + assert rep.passed, rep.detail + + +def test_l2_missing_token_fails_incomplete(): + rows, meta = _learned_case() + ident = _ident(bias_fingerprint=meta["bias_fingerprint"]) + base = _wrap(rows, ident) + dropped = [r for r in rows if r.global_token_id != 1] + pert = _wrap(dropped, ident, padding_rows=6) # padding can't mask a loss + rep = run_l2_invariance("cx", base, pert, variant="drop") + assert not rep.passed + assert rep.verdict is P3Verdict.INCOMPLETE_ARTIFACT + assert rep.extra["missing"] == [1] + + +def test_l2_extra_token_fails_ambiguous_mapping(): + rows, meta = _learned_case() + ident = _ident(bias_fingerprint=meta["bias_fingerprint"]) + base = _wrap(rows, ident) + ghost = dataclasses.replace(rows[0], global_token_id=99, input_token_id=99) + pert = _wrap(rows + [ghost], ident) + rep = run_l2_invariance("cx", base, pert, variant="ghost") + assert not rep.passed + assert rep.verdict is P3Verdict.AMBIGUOUS_GLOBAL_TOKEN_MAPPING + + +def test_l2_weight_bit_flip_is_first_mismatch(): + rows, meta = _learned_case() + ident = _ident(bias_fingerprint=meta["bias_fingerprint"]) + base = _wrap(rows, ident) + flipped = list(rows) + flipped[0] = dataclasses.replace(flipped[0], route_weight=flipped[0].route_weight + 1e-6) + pert = _wrap(flipped, ident) + rep = run_l2_invariance("cx", base, pert, variant="flip") + assert not rep.passed + assert rep.verdict is P3Verdict.ROUTE_SEMANTIC_FINGERPRINT_MISMATCH + assert rep.extra["first_token"] == rows[0].global_token_id + + +# --- L3a ----------------------------------------------------------------------- + +def test_l3a_byte_exact_passes(): + rows, meta = _learned_case() + ident = _ident(bias_fingerprint=meta["bias_fingerprint"]) + rep = run_l3a_oracle("cx", ident, rows, list(rows)) + assert rep.passed + + +def test_l3a_ignores_padding_rows(): + """§2.5: padding is audited only by the same-config artifact gate.""" + rows, meta = _learned_case() + ident = _ident(bias_fingerprint=meta["bias_fingerprint"]) + pad = RouteRow(global_token_id=-1, input_token_id=-1, absolute_layer=3, + router_mode="learned", topk_index=0, logical_expert_id=-1, + valid=False, invalid_reason="padding", route_weight=0.0, + weight_score=0.0, selection_score=0.0) + rep = run_l3a_oracle("cx", ident, rows + [pad], rows) + assert rep.passed + + +def test_l3a_expert_id_mismatch_reports_topk_order(): + rows, meta = _learned_case() + ident = _ident(bias_fingerprint=meta["bias_fingerprint"]) + bad = list(rows) + bad[7] = dataclasses.replace(bad[7], logical_expert_id=(bad[7].logical_expert_id + 1) % 256) + rep = run_l3a_oracle("cx", ident, rows, bad) + assert not rep.passed + assert rep.verdict is P3Verdict.TOPK_ORDER_MISMATCH + assert rep.first_mismatch is not None + assert rep.first_mismatch.key.global_token_id == bad[7].global_token_id + + +def test_l3a_row_count_mismatch_fails_incomplete(): + rows, meta = _learned_case() + ident = _ident(bias_fingerprint=meta["bias_fingerprint"]) + rep = run_l3a_oracle("cx", ident, rows, rows[:-1]) + assert not rep.passed + assert rep.verdict is P3Verdict.INCOMPLETE_ARTIFACT + + +# --- L3b ----------------------------------------------------------------------- + +def _meta(router_mode="learned", layer=3): + return { + "case_id": "cx", "checkpoint_id": "c", "weight_id": "w", + "absolute_layer": layer, "router_mode": router_mode, + "table_fingerprint": "00" * 32, "bias_fingerprint": "11" * 32, + "logit_round_point": "fp32_direct", "tie_break_policy": "p3_canonical", + "capacity_policy": "dropless_v1", + } + + +def test_l3b_identical_engines_pass_four_stages(): + import torch + + from rl_engine.moe.validation.first_mismatch import MismatchKey, TraceEvent + + events = [ + TraceEvent(key=MismatchKey(absolute_layer=3, site="topk", + pass_direction="forward", event_index=i, + global_token_id=i // K, rank=0), + payload={"logical_expert_id": i % 256}) + for i in range(12) + ] + w = torch.full((12,), 0.25) + dz = torch.full((12,), 1.0) + rep = run_l3b_dual_engine("cx", _meta(), dict(_meta()), events, list(events), + w, w.clone(), dz, dz.clone()) + assert rep.passed, rep.detail + + +def test_l3b_identity_drift_halts_walk(): + import torch + + from rl_engine.moe.validation.first_mismatch import MismatchKey, TraceEvent + + events = [] + lhs_meta = _meta() + rhs_meta = _meta() + rhs_meta["tie_break_policy"] = "other" + w = torch.zeros(0) + rep = run_l3b_dual_engine("cx", lhs_meta, rhs_meta, events, events, + w, w.clone(), w, w.clone()) + assert not rep.passed + assert rep.verdict is P3Verdict.IDENTITY_DRIFT + assert rep.extra["walked"] == ["identity"] + + +# --- fingerprints --------------------------------------------------------------- + +def test_semantic_hash_excludes_padding_and_envelope(): + rows, meta = _learned_case() + ident = _ident(bias_fingerprint=meta["bias_fingerprint"]) + a = _wrap(rows, ident) + b = _wrap(rows, ident, run_id="r9", engine_id="miles", attempt_id=9, + placement_offset=7, padding_rows=3) + sa = per_token_semantic_hashes(a.rows, a.identity) + sb = per_token_semantic_hashes(b.rows, b.identity) + assert sa == sb + assert -1 not in sa + assert route_semantic_hash(a.rows, ident) == route_semantic_hash(b.rows, ident) + + +def test_artifact_hash_sensitive_to_envelope(): + rows, meta = _learned_case() + ident = _ident(bias_fingerprint=meta["bias_fingerprint"]) + a = _wrap(rows, ident) + b = _wrap(rows, ident, attempt_id=2) + assert route_artifact_hash(a) != route_artifact_hash(b) + + +def test_minus_zero_vs_plus_zero_distinguished(): + h1 = route_semantic_hash( + [RouteRow(0, 0, 3, "learned", 0, 1, True, None, 0.0, 0.0, 0.0)], _ident()) + h2 = route_semantic_hash( + [RouteRow(0, 0, 3, "learned", 0, 1, True, None, -0.0, 0.0, 0.0)], _ident()) + assert h1 != h2 + + +# --- CLI ----------------------------------------------------------------------- + +@pytest.mark.parametrize("args,expect_pass", [ + (["--cases", "smoke"], True), + (["--cases", "learned_basic", "--ladders", "L1"], True), + (["--cases", "hash_basic", "--ladders", "L2,L3a"], True), + (["--ladders", "L9"], None), # bad ladder -> usage error 2 + (["--cases", "nope"], None), # bad case -> usage error 2 +]) +def test_check_p3_cli(args, expect_pass): + proc = subprocess.run( + [sys.executable, str(REPO_ROOT / "scripts" / "check_p3.py"), *args], + capture_output=True, text=True, cwd=REPO_ROOT, + ) + if expect_pass is None: + assert proc.returncode == 2 + assert "error:" in proc.stderr + else: + assert proc.returncode == 0, proc.stdout + proc.stderr + assert "check_p3:" in proc.stdout + + +def test_check_p3_cli_json_flag(): + proc = subprocess.run( + [sys.executable, str(REPO_ROOT / "scripts" / "check_p3.py"), + "--cases", "learned_basic", "--ladders", "L1", "--json"], + capture_output=True, text=True, cwd=REPO_ROOT, + ) + assert proc.returncode == 0 + assert '"ladder": "L1"' in proc.stdout diff --git a/tests/test_p3_verdicts.py b/tests/test_p3_verdicts.py new file mode 100644 index 00000000..e62c7700 --- /dev/null +++ b/tests/test_p3_verdicts.py @@ -0,0 +1,90 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 RL-Kernel Contributors + +"""Tests for the P3 verdict code table (contract §6, frozen).""" + +from __future__ import annotations + +import sys +import unittest + +sys.path.insert(0, "/workspace/RL-Kernel") # standalone-runnable + +from rl_engine.moe.p3_verdicts import ( # noqa: E402 + DEVICE_WRITABLE, + P3Verdict, + classify_writable_band, + is_valid_device_status, + primary_verdict, +) + + +class VerdictTableTests(unittest.TestCase): + def test_frozen_values_match_contract(self): + # Spot-check every value against contract §6 to guard accidental edits. + expected = { + "PASS": 0, "NON_FINITE": 1, "HASH_TABLE_INDEX_OUT_OF_RANGE": 2, + "IDENTITY_DRIFT": 10, "SCHEMA_MISMATCH": 11, "CORRUPT_ARTIFACT": 12, + "INCOMPLETE_ARTIFACT": 13, "LOGIT_ROUND_POINT_MISMATCH": 14, + "HASH_TABLE_MISMATCH": 15, "UNSUPPORTED_CAPABILITY": 16, + "ZERO_ACTIVE_TOKENS": 17, "UPSTREAM_NON_FINITE": 18, + "GATE_SHARDING_MISMATCH": 19, "MISSING_RANK": 20, + "PRE_UPDATE_WEIGHT_DRIFT": 21, "STALE_RUN_METADATA": 22, + "CASE_PASS": 50, "ROUTE_WEIGHT_BYTES_MISMATCH": 51, + "SCORE_BYTES_MISMATCH": 52, "GRADIENT_BYTES_MISMATCH": 53, + "BYTE_MISMATCH": 54, "TOPK_ORDER_MISMATCH": 55, + "TIE_BREAK_POLICY_MISMATCH": 56, "INVALID_DISCRETE_PLAN": 57, + "INVALID_PROFILE": 58, "ROUTE_SEMANTIC_FINGERPRINT_MISMATCH": 59, + "ROUTE_ARTIFACT_FINGERPRINT_MISMATCH": 60, + "SELECTION_GRADIENT_PRESENT": 61, "FORBIDDEN_LOCAL_SHARD_TOPK": 62, + "AMBIGUOUS_GLOBAL_TOKEN_MAPPING": 63, "INVALID_PLACEMENT_MAP": 64, + "PLACEMENT_MAP_VERSION_MISMATCH": 65, "SILENT_FALLBACK": 66, + "MISSING_PROVENANCE": 67, "MISSING_BOUNDARY_TRACE": 68, + "UPSTREAM_CONTRACT_MISMATCH": 69, "UPSTREAM_VERDICT_MISSING": 70, + "UPSTREAM_EVIDENCE_MISSING": 71, "NATURAL_ROUTE_MISMATCH": 72, + } + for name, value in expected.items(): + self.assertEqual(int(P3Verdict[name]), value, name) + + def test_device_writable_only_two(self): + self.assertEqual( + DEVICE_WRITABLE, + {P3Verdict.NON_FINITE, P3Verdict.HASH_TABLE_INDEX_OUT_OF_RANGE}, + ) + + def test_is_valid_device_status(self): + self.assertTrue(is_valid_device_status(1)) + self.assertTrue(is_valid_device_status(2)) + for bad in (0, 3, 9, 10, 50, 55, 72, 90, -1, 1000): + self.assertFalse(is_valid_device_status(bad), bad) + + def test_band_classification(self): + self.assertEqual(classify_writable_band(10), "provider") + self.assertEqual(classify_writable_band(22), "provider") + self.assertEqual(classify_writable_band(50), "runner") + self.assertEqual(classify_writable_band(72), "runner") + self.assertEqual(classify_writable_band(90), "reserved") + self.assertEqual(classify_writable_band(3), "reserved") + + def test_pass_vs_case_pass_not_conflated(self): + self.assertNotEqual(P3Verdict.PASS, P3Verdict.CASE_PASS) + + def test_primary_verdict_priority(self): + # identity/schema beats numeric beats fingerprint beats diagnostics + vs = [ + P3Verdict.ROUTE_ARTIFACT_FINGERPRINT_MISMATCH, # fingerprint + P3Verdict.NON_FINITE, # numeric bytes + P3Verdict.IDENTITY_DRIFT, # identity + P3Verdict.NATURAL_ROUTE_MISMATCH, # diagnostics + ] + self.assertIs(primary_verdict(vs), P3Verdict.IDENTITY_DRIFT) + # upstream evidence beats discrete plan + vs2 = [P3Verdict.TOPK_ORDER_MISMATCH, P3Verdict.UPSTREAM_VERDICT_MISSING] + self.assertIs(primary_verdict(vs2), P3Verdict.UPSTREAM_VERDICT_MISSING) + # empty -> None; single -> itself + self.assertIsNone(primary_verdict([])) + self.assertIs(primary_verdict([P3Verdict.SILENT_FALLBACK]), P3Verdict.SILENT_FALLBACK) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_p3_ws2.py b/tests/test_p3_ws2.py new file mode 100644 index 00000000..b6fdfaae --- /dev/null +++ b/tests/test_p3_ws2.py @@ -0,0 +1,210 @@ +# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2026 RL-Kernel Contributors + +"""WS2 ladder tests: rank dimension + cross-config (§3 WS2 Gate row). + +"同一 token 的 Core semantic fingerprint exact,Envelope 按配置完整。" +""" + +from __future__ import annotations + +import dataclasses + +import pytest + +from rl_engine.moe.p3_verdicts import P3Verdict +from rl_engine.moe.validation.fingerprint import RouteIdentity +from rl_engine.moe.validation.synthetic_producer import ( + make_artifact, + make_learned_route_rows, +) +from rl_engine.moe.validation.ws2 import ( + TOKEN_PARTITION, + TOKEN_REPLICA, + RankArtifact, + check_rank_completeness, + run_ws2_cross_config, +) + + +def _ident(**over): + base = dict( + checkpoint_id="ckpt-x", weight_id="w-x", + table_fingerprint="00" * 32, bias_fingerprint="11" * 32, + ) + base.update(over) + return RouteIdentity(**base) + + +def _case(t_rows=4, seed=21): + rows, meta = make_learned_route_rows("cx", seed=seed, t_rows=t_rows, layers=(3,)) + ident = _ident(bias_fingerprint=meta["bias_fingerprint"]) + return rows, ident + + +def _rank_art(rank, rows, ident, group, **kw): + kw.setdefault("run_id", f"r{rank}") + kw.setdefault("engine_id", "e1") + kw.setdefault("attempt_id", 1) + return RankArtifact( + rank=rank, group=group, + artifact=make_artifact(f"cx-r{rank}", rows, ident, rank=rank, **kw), + ) + + +# --- rank completeness -------------------------------------------------------- + +def test_rank_completeness_pass(): + rows, ident = _case() + arts = [_rank_art(r, rows, ident, "dp2") for r in range(2)] + rep = check_rank_completeness("cx", arts, range(2)) + assert rep.passed + + +def test_rank_completeness_missing_rank(): + rows, ident = _case() + arts = [_rank_art(0, rows, ident, "dp2")] + rep = check_rank_completeness("cx", arts, range(2)) + assert not rep.passed + assert rep.verdict is P3Verdict.MISSING_RANK + assert rep.extra["missing_ranks"] == [1] + + +def test_rank_completeness_duplicate_rank_is_stale_mix(): + rows, ident = _case() + arts = [_rank_art(0, rows, ident, "dp1"), _rank_art(0, rows, ident, "dp1")] + rep = check_rank_completeness("cx", arts, range(1)) + assert not rep.passed + assert rep.verdict is P3Verdict.STALE_RUN_METADATA + + +# --- cross-config: partition (CP/DP) ------------------------------------------- + +def _split_rows(rows, parts, k): + """Partition tokens into `parts` shards by global_token_id.""" + out = [[] for _ in range(parts)] + for r in rows: + out[r.global_token_id % parts].append(r) + return out + + +def test_cross_config_partition_pass(): + rows, ident = _case(t_rows=8) + base = [_rank_art(0, rows, ident, "dp1")] + shards = _split_rows(rows, 2, None) + other = [_rank_art(r, shards[r], ident, "dp2-cp1") for r in range(2)] + rep = run_ws2_cross_config("cx", base, other, + base_config="dp1", other_config="dp2", + ownership=TOKEN_PARTITION) + assert rep.passed, rep.detail + + +def test_cross_config_partition_dropped_token_fails(): + rows, ident = _case(t_rows=8) + base = [_rank_art(0, rows, ident, "dp1")] + shards = _split_rows(rows, 2, None) + shards[1] = [r for r in shards[1] if r.global_token_id != 3] + other = [_rank_art(r, shards[r], ident, "dp2") for r in range(2)] + rep = run_ws2_cross_config("cx", base, other, + base_config="dp1", other_config="dp2", + ownership=TOKEN_PARTITION) + assert not rep.passed + assert rep.verdict is P3Verdict.INCOMPLETE_ARTIFACT + + +def test_cross_config_partition_double_ownership_fails(): + """Partition mode: a token routed by two ranks is ambiguous mapping.""" + rows, ident = _case(t_rows=8) + base = [_rank_art(0, rows, ident, "dp1")] + shards = _split_rows(rows, 2, None) + shards[1] = shards[1] + [r for r in shards[0] if r.global_token_id == 0] + other = [_rank_art(r, shards[r], ident, "dp2") for r in range(2)] + rep = run_ws2_cross_config("cx", base, other, + base_config="dp1", other_config="dp2", + ownership=TOKEN_PARTITION) + assert not rep.passed + assert rep.verdict is P3Verdict.AMBIGUOUS_GLOBAL_TOKEN_MAPPING + + +def test_cross_config_semantic_drift_names_token_and_rank(): + rows, ident = _case(t_rows=8) + base = [_rank_art(0, rows, ident, "dp1")] + shards = _split_rows(rows, 2, None) + drifted = list(shards[1]) + drifted[0] = dataclasses.replace(drifted[0], route_weight=drifted[0].route_weight + 1e-6) + shards[1] = drifted + other = [_rank_art(r, shards[r], ident, "dp2") for r in range(2)] + rep = run_ws2_cross_config("cx", base, other, + base_config="dp1", other_config="dp2", + ownership=TOKEN_PARTITION) + assert not rep.passed + assert rep.verdict is P3Verdict.ROUTE_SEMANTIC_FINGERPRINT_MISMATCH + assert rep.extra["first_rank"] == 1 + + +# --- cross-config: replica (TP) ------------------------------------------------- + +def test_cross_config_replica_pass(): + """TP replica: same tokens on every rank, hashes identical.""" + rows, ident = _case(t_rows=4) + base = [_rank_art(0, rows, ident, "tp1")] + other = [_rank_art(r, rows, ident, "tp4-sp0") for r in range(4)] + rep = run_ws2_cross_config("cx", base, other, + base_config="tp1", other_config="tp4", + ownership=TOKEN_REPLICA) + assert rep.passed, rep.detail + + +def test_cross_config_replica_one_bad_rank_fails(): + rows, ident = _case(t_rows=4) + base = [_rank_art(0, rows, ident, "tp1")] + bad = list(rows) + bad[2] = dataclasses.replace(bad[2], weight_score=bad[2].weight_score + 1e-6) + other = [ + _rank_art(0, rows, ident, "tp4"), + _rank_art(1, bad, ident, "tp4"), + _rank_art(2, rows, ident, "tp4"), + _rank_art(3, rows, ident, "tp4"), + ] + rep = run_ws2_cross_config("cx", base, other, + base_config="tp1", other_config="tp4", + ownership=TOKEN_REPLICA) + assert not rep.passed + assert rep.verdict is P3Verdict.ROUTE_SEMANTIC_FINGERPRINT_MISMATCH + assert rep.extra["first_rank"] == 1 + + +def test_cross_config_envelope_changes_are_not_judged(): + """Envelope (placement/run) differences across configs are fine (§3).""" + rows, ident = _case(t_rows=4) + base = [_rank_art(0, rows, ident, "tp1", engine_id="megatron")] + other = [_rank_art(1, rows, ident, "tp4-sp0", engine_id="miles", + placement_offset=8, padding_rows=3)] + rep = run_ws2_cross_config("cx", base, other, + base_config="tp1", other_config="tp4", + ownership=TOKEN_REPLICA) + assert rep.passed + + +def test_cross_config_rejects_unknown_ownership(): + rows, ident = _case(t_rows=2) + base = [_rank_art(0, rows, ident, "tp1")] + with pytest.raises(ValueError): + run_ws2_cross_config("cx", base, base, base_config="a", + other_config="b", ownership="ep??") + + +# --- CLI integration ------------------------------------------------------------ + +def test_check_p3_cli_ws2_smoke(): + import subprocess + import sys + from pathlib import Path + + repo = Path(__file__).resolve().parents[1] + proc = subprocess.run( + [sys.executable, str(repo / "scripts" / "check_p3.py"), + "--cases", "learned_basic", "--ladders", "L1"], + capture_output=True, text=True, cwd=repo, + ) + assert proc.returncode == 0