diff --git a/assets/completion-matrix-7way.en.png b/assets/completion-matrix-7way.en.png new file mode 100644 index 0000000..77aa571 Binary files /dev/null and b/assets/completion-matrix-7way.en.png differ diff --git a/assets/completion-matrix-7way.png b/assets/completion-matrix-7way.png new file mode 100644 index 0000000..e4e49ed Binary files /dev/null and b/assets/completion-matrix-7way.png differ diff --git a/assets/effort-curves-20260911.en.png b/assets/effort-curves-20260911.en.png new file mode 100644 index 0000000..ed26347 Binary files /dev/null and b/assets/effort-curves-20260911.en.png differ diff --git a/assets/effort-curves-20260911.png b/assets/effort-curves-20260911.png new file mode 100644 index 0000000..60ff9dd Binary files /dev/null and b/assets/effort-curves-20260911.png differ diff --git a/assets/score-vs-tokens-2026-w37.en.png b/assets/score-vs-tokens-2026-w37.en.png new file mode 100644 index 0000000..5b911e4 Binary files /dev/null and b/assets/score-vs-tokens-2026-w37.en.png differ diff --git a/assets/score-vs-tokens-2026-w37.png b/assets/score-vs-tokens-2026-w37.png new file mode 100644 index 0000000..670de6b Binary files /dev/null and b/assets/score-vs-tokens-2026-w37.png differ diff --git a/assets/top5-2026-w38.en.png b/assets/top5-2026-w38.en.png new file mode 100644 index 0000000..848a984 Binary files /dev/null and b/assets/top5-2026-w38.en.png differ diff --git a/assets/top5-2026-w38.png b/assets/top5-2026-w38.png new file mode 100644 index 0000000..2736ab2 Binary files /dev/null and b/assets/top5-2026-w38.png differ diff --git a/assets/trust-chain.en.png b/assets/trust-chain.en.png new file mode 100644 index 0000000..e9f2e42 Binary files /dev/null and b/assets/trust-chain.en.png differ diff --git a/assets/trust-chain.png b/assets/trust-chain.png new file mode 100644 index 0000000..68f266e Binary files /dev/null and b/assets/trust-chain.png differ diff --git a/assets/wallclock-strip-2026-w37.en.png b/assets/wallclock-strip-2026-w37.en.png new file mode 100644 index 0000000..56af5e5 Binary files /dev/null and b/assets/wallclock-strip-2026-w37.en.png differ diff --git a/assets/wallclock-strip-2026-w37.png b/assets/wallclock-strip-2026-w37.png new file mode 100644 index 0000000..b2340c8 Binary files /dev/null and b/assets/wallclock-strip-2026-w37.png differ diff --git a/assets/what-is-amber.en.png b/assets/what-is-amber.en.png new file mode 100644 index 0000000..67eb3a1 Binary files /dev/null and b/assets/what-is-amber.en.png differ diff --git a/assets/what-is-amber.png b/assets/what-is-amber.png new file mode 100644 index 0000000..76cc3cb Binary files /dev/null and b/assets/what-is-amber.png differ diff --git a/en.html b/en.html new file mode 100644 index 0000000..ba24092 --- /dev/null +++ b/en.html @@ -0,0 +1,150 @@ + + + + + +AskClaw — AMBER: sealed-in-amber replay evaluation + + + + +
+
+ + +
+
+ +
+ +
+

AMBER — real history,
sealed in amber, replayed

+

We take real, auditable incidents and freeze them at the moment before the answer was revealed. Models must redo the work with only what was knowable then, against a rubric frozen before any output existed. The cases are never published. The scores always are.

+
+
23 cases / 26 papersprivate library, hash-sealed
+
11 result reposone per vendor × endpoint
+
Updated weeklysame library, same band, public hash index
+
+
+ +
+

The board right now

+

Same band (high), same 23 cases, same public hash index. Five rank tiers; ties are labeled as ties.

+
+ AMBER board: five rank tiers (2026-W38) +
2026-W38 · leader swe-2-max @ Devin at 18/23; a five-way tie at 17/23.
+
+
+ +
+

Same score, different shape

+

The nine-axis completion matrix: each cell is a face's pin-level completion. Tied totals hide completely different profiles — the matrix shows what the sum can't.

+
+ Nine-axis completion matrix +
Five lanes tied at 17/23, plus doubao 16/23 and gp27b 14/23.
+
+
+ +
+

Why the scores can be trusted

+

Every weekly issue runs like an exam: cases sealed in advance, a clean-room sitting, per-paper wire audits, anonymized publication, and a public index anyone can re-check.

+
+ How a result is produced +
+
+ +
+

Three counterintuitive findings

+

Each comes with named boundaries and counterexamples in the original writeups.

+
+ Effort curves +
"Think longer ≠ score better": most families backfire at top bands; swe-2 is monotone to the top.
+
+
+
+ Score vs thinking budget +
Same band, same library: output-token bills span 17× while scores differ by one case.
+
+
+ Per-case wall clock +
A vendor's high TPS only holds on easy problems; hard ones slow the token stream down.
+
+
+
+ +
+

Result repos

+

One repo per lane (model × endpoint × band), one matrix per week, wire audits and hash cross-checks included.

+
+ amberspec

Method spec + public hash index + cross-repo weekly board

 + amber-devin18/23

swe-2-max @ Devin — current leader

 + amber-kimi17/23

k3 @ Kimi official coding endpoint + the K2.8 duel

 + amber-ollama17/23

glm-5.3-flash / d41f @ Ollama Cloud

 + amber-commandcode17/23

d41f @ CommandCode — five-band ladder

 + amber-workbuddy17/23

hy4-preview-f @ WorkBuddy ACP

 + amber-gpt16/23

gpt-6-astra / gpt-5.6 family @ OpenAI Codex

 + amber-crof16/23

qwen3.8-27b @ CrofAI — the $1.15 value king

 + amber-deepseek16/23

deepseek-flash @ DeepSeek official

 + amber-opencode16/23

deepseek-flash @ OpenCode Go

 + amber-doubao16/23

doubao-seed-evolving @ Volcengine Ark Agent Plan

 + amber-goldenpotato14/23

Qwen3.8-27B @ community self-hosted (3× V100)

 +
+
+ +
+

The rules

+
Private side, never public: the case library, oracles, transcripts, internal case IDs and variant names.
+
Public side, always checkable: scores, aggregates, verdicts, the hash index — anyone can verify the library didn't change and the scores add up.
+
Historical outcomes are evidence, not the only correct answer; runtime sealing ≠ clean training data. Both warnings are printed on every issue.
+

Read the method: AMBER Core Specification · public hash index · roadmap

+
+ +
+ + + + + diff --git a/index.html b/index.html new file mode 100644 index 0000000..57e053e --- /dev/null +++ b/index.html @@ -0,0 +1,150 @@ + + + + + +AskClaw — AMBER 琥珀式封存回放评测 + + + + +
+
+ + +
+
+ +
+ +
+

AMBER — 琥珀式封存的
历史回放评测

+

把真实、可审计的历史事件封进琥珀，精确还原到「答案揭晓之前」，让 AI 只用当时可得的信息重做一遍。评分标准在看到任何输出之前就已冻结。题目永不公开，分数永远可查。

+
+
23 案 / 26 卷私有题库，哈希封存
+
11 个成绩仓按厂商 × 端点分仓
+
每周更新同库、同档、公开哈希索引
+
+
+ +
+

当前总榜

+

同档（high）、同 23 案、同公开哈希索引。五个名次档，并列如实标注。

+
+ AMBER 当前总榜：五个名次档（2026-W38，全库 23 案） +
2026-W38 · 榜首 swe-2-max @ Devin 18/23；#2 并列五席 17/23。
+
+
+ +
+

同分 ≠ 同款

+

九轴完成度矩阵：每格 = 该面全部案子的钉级完成度。同分的几家，形状完全不同——总分看不见的东西，矩阵看得见。

+
+ 九轴完成度矩阵 +
五家 17/23 + doubao 16/23 + gp27b 14/23，九轴同场。
+
+
+ +
+

分数凭什么可信

+

每期成绩像一场考试：出卷封存、净室应考、逐卷验脑、脱敏发布、人人可核对。

+
+ 一期成绩怎么炼成 +
+
+ +
+

三个反直觉发现

+

都有边界，都只代表已测家族——原文里每条都标着反例。

+
+ effort 曲线 +
「想更久 ≠ 考更好」：多数家族高档反噬，swe-2 单调到底。
+
+
+
+ 分数 × 思考预算 +
同档同库，输出 token 账单跨 17 倍，分差不过一案。
+
+
+ 逐题耗时 +
厂商的高 TPS 只在简单题上成立；难题想得多，吐字当场变慢。
+
+
+
+ +
+

成绩仓

+

每条车道（模型 × 端点 × 档位）一个仓，每期一篇矩阵，验脑记录与哈希对照齐全。

+
+ amber规范

方法规范 + 公开哈希索引 + 跨仓周榜

 + amber-devin18/23

swe-2-max @ Devin — 当前榜首

 + amber-kimi17/23

k3 @ Kimi 官方 coding 端点 + K2.8 对决

 + amber-ollama17/23

glm-5.3-flash / d41f @ Ollama Cloud

 + amber-commandcode17/23

d41f @ CommandCode — 五档天梯

 + amber-workbuddy17/23

hy4-preview-f @ WorkBuddy ACP

 + amber-gpt16/23

gpt-6-astra / gpt-5.6 系 @ OpenAI Codex

 + amber-crof16/23

qwen3.8-27b @ CrofAI — $1.15 性价比之王

 + amber-deepseek16/23

deepseek-flash @ DeepSeek 官方

 + amber-opencode16/23

deepseek-flash @ OpenCode Go

 + amber-doubao16/23

doubao-seed-evolving @ 火山方舟 Agent Plan

 + amber-goldenpotato14/23

Qwen3.8-27B @ 社区自部署（3×V100）

 +
+
+ +
+

规则

+
私有面永不公开：题库、oracle、transcript、内部案号与变体名。
+
公开面永远可查：分数、聚合、裁决、哈希索引——任何人都能核对题集没换、分数可对。
+
历史结果是证据，不是唯一正解；运行时密封 ≠ 训练数据纯净——这两条警告印在每期成绩上。
+

方法全文：AMBER Core Specification · 公开哈希索引 · 路线图

+
+ +
+ + + + +