Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,9 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); ver

### Added

- Intern-Decision seven-suite collection with raw HTTP evidence and H800 results
for the 0.8B, 2B and 4B checkpoints.

- Intern-Decision input verification and offline replay with the official evaluator
and explicit coverage for incomplete runs.

Expand Down
37 changes: 37 additions & 0 deletions benchmarks/INTERN_DECISION.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,40 @@ parse and refusal failures remain failed attempts. Scored ledger columns are ign
by hash. Retain the original run because raw files are referenced, not copied.
Recorded configuration is provenance, not proof of the loaded weights.

## Collect seven suites

Use an already running authorized endpoint and an environment JSON containing
`checkpoint.repo`/`revision`, `backend.name`/`version`, positive
`sampling.temperature`, and `hardware`. Include actual dtype, libraries, context
limits and JobID when applicable. The original file is saved unchanged and hashed.
It must not contain collector-owned `endpoint`, `path`, `model_configured`,
`env_sha256`, `dataset`, `plan`, `stop_reason`, `attempted` or `collected` fields.

```bash
python -m benchmarks.intern_decision collect \
--intern-root /path/to/Intern-Decision --scorer-root /path/to/jevbench \
--endpoint http://127.0.0.1:8000 --path /v1/decisions \
--run-meta /path/to/actual-environment.json --out /path/to/fresh-seven-suite-run
python -m benchmarks.intern_decision replay \
--intern-root /path/to/Intern-Decision --scorer-root /path/to/jevbench \
--round /path/to/fresh-seven-suite-run/round-1 --out /path/to/fresh-seven-suite-replay
```

The shared client sends five separate warmups, then one serial accuracy round.
All fields of a Typed Decisions row stay in one POST. Only state and questions
(`type`, `instructions`, `criteria`) are model evidence; targets and IDs are omitted.
Use `--model` only if the endpoint requires it; `--timeout` defaults to 120 seconds.
`--path /v1/systemone` is also supported. No API key, automatic retry or resume is
provided. This is separate from the public231 three-round latency matrix.

Every attempt saves original raw evidence before its ledger record. HTTP
401/403/429, three consecutive infrastructure failures, or an interrupt stop
collection; 422 remains failed and resets the consecutive-failure counter.
`run.meta.json` retains `stop_reason` and `attempted`. `collected=true` means every
request was attempted, including failed answers; it does not mean scoring passed.
A stop during warmup leaves all measured requests unattempted. Controlled early
stops exit 2; unexpected exceptions exit nonzero; full collection exits 0.

## Scoring and completeness

The unmodified upstream `src.eval.jev.evaluate` and table renderer perform scoring.
Expand All @@ -79,3 +113,6 @@ different input. This entry calls official `evaluate` directly because upstream'
main/merge CLI requires those 10 references even for the bundled input. The table's
Brier/ECE columns refer to JevBench hard. A complete replay establishes scoring
coverage, not reproduction of published model quality or inference latency.

[H800 results](results/intern-accuracy-v1-h800.md) report one complete run for the
0.8B, 2B and 4B HF checkpoints.
130 changes: 128 additions & 2 deletions benchmarks/intern_decision.py
Original file line number Diff line number Diff line change
@@ -1,8 +1,9 @@
"""Verify and replay pinned Intern-Decision suites with the official evaluator."""
"""Verify, collect and replay pinned Intern-Decision suites with the official evaluator."""

from __future__ import annotations

import argparse
from contextlib import closing
import hashlib
import importlib
import importlib.util
Expand Down Expand Up @@ -227,6 +228,117 @@ def load_round(intern_root, scorer_root, round_dir):
return responses, provenance, config


def collect(args, *, transport=None):
"""Collect one complete accuracy-v1 round, with five separate warmup requests."""
from benchmarks.client import DecisionClient, decode
from benchmarks.evidence import dump, outside_repo, write_json
from benchmarks.legs.leg1_public231 import validate_environment

verified = verify_sources(args.intern_root, args.scorer_root)
environment = args.run_meta.read_bytes()
meta = decode(environment)
validate_environment(meta)
controlled = {
"endpoint",
"path",
"model_configured",
"env_sha256",
"dataset",
"plan",
"stop_reason",
"attempted",
"collected",
}
if controlled.intersection(meta):
raise ValueError(f"run-meta contains collector-controlled fields: {sorted(controlled.intersection(meta))}")
bundle = Path(verified["intern_root"]) / "benchmarks/accuracy-v1"
items = [
(suite, line, row) for suite in SUITES for line, row in _rows(bundle / verified["datasets"][suite]["path"])
]
output = outside_repo(args.out)
meta.update(
endpoint=args.endpoint,
path=args.path,
model_configured=args.model,
env_sha256=hashlib.sha256(environment).hexdigest(),
dataset=dict(
intern_revision=INTERN_REVISION,
jevbench_pin=SCORER_REVISION,
hashes=verified["files"],
dataset_hash=verified["manifest_sha256"],
),
plan=dict(datasets=verified["datasets"], warmup=5, rounds=1),
stop_reason=None,
)
with closing(
DecisionClient(args.endpoint, path=args.path, model=args.model, timeout_s=args.timeout, transport=transport)
) as client:
output.mkdir(parents=True, exist_ok=False)
(output / "environment.json").write_bytes(environment)
write_json(output / "run.meta.json", meta)
stop, errors, attempted = None, 0, 0
try:
for number, selected in ((0, items[:5]), (1, items)):
directory = output / ("warmup" if number == 0 else "round-1")
directory.mkdir()
with (directory / ("records.jsonl" if number == 0 else "results.jsonl")).open(
"x", encoding="utf-8"
) as stream:
for suite, line, row in selected:
body = _body(row)
raw = client.request(body)
path = _raw_path(directory, (suite, line, row["id"]), True)
path.parent.mkdir(parents=True, exist_ok=True)
write_json(path, raw)
code, response = raw["http_status"], raw["response"]
answers = response.get("answers") if isinstance(response, dict) else None
parsed = isinstance(answers, dict) and all(
isinstance(answers.get(field), dict) and answers[field].get("type") == question["type"]
for field, question in body["questions"].items()
)
status = "ok" if parsed else "parse_error"
if raw["client_error"] is not None:
status = "client_error"
elif code is None or not 200 <= code < 300:
status = "refused" if code == 422 else "http_error"
record = dict(
suite=suite,
source_line=line,
task_id=row["id"],
source_file=verified["datasets"][suite]["path"],
round=number,
kind="warmup" if number == 0 else "measured",
attempted=True,
ok=status == "ok",
status=status,
status_code=code,
error=raw["client_error"] or (status if status != "ok" else None),
ts=raw["ts"],
latency_s=raw["latency_s"],
raw_sha256=_sha256(path),
)
stream.write(dump(record) + "\n")
stream.flush()
os.fsync(stream.fileno())
attempted += number
errors = errors + 1 if status != "ok" and code != 422 else 0
if raw["client_error"] == "KeyboardInterrupt" or code in (401, 403, 429) or errors >= 3:
stop = raw["client_error"] or "access/rate limit or three consecutive infrastructure errors"
break
if stop:
break
except KeyboardInterrupt:
stop = "interrupted"
except Exception as exc:
stop = f"{type(exc).__name__}: {exc}"
raise
finally:
meta.update(stop_reason=stop, attempted=attempted, collected=attempted == len(items) and stop is None)
write_json(output / "run.meta.final.json", meta)
(output / "run.meta.final.json").replace(output / "run.meta.json")
return output


class SavedResponseEngine:
"""Sequential, unsharded evaluate() adapter; never normalizes saved answers."""

Expand Down Expand Up @@ -374,22 +486,36 @@ def replay_bundle(intern_root, scorer_root, responses, *, checkpoint, backend, t
def main():
parser = argparse.ArgumentParser(description=__doc__)
commands = parser.add_subparsers(dest="command", required=True)
for command in ("verify", "replay"):
for command in ("verify", "replay", "collect"):
child = commands.add_parser(command)
child.add_argument("--intern-root", type=Path, required=True)
child.add_argument("--scorer-root", type=Path, required=True)
if command == "replay":
child.add_argument("--round", type=Path, required=True, help="A's saved round-N directory")
child.add_argument("--out", type=Path, required=True, help="New replay output directory")
if command == "collect":
child.add_argument("--endpoint", required=True)
child.add_argument("--path", choices=("/v1/decisions", "/v1/systemone"), required=True)
child.add_argument("--model")
child.add_argument("--timeout", type=float, default=120)
child.add_argument("--run-meta", type=Path, required=True)
child.add_argument("--out", type=Path, required=True)
args = parser.parse_args()
if args.command == "verify":
report = verify_sources(args.intern_root, args.scorer_root)
elif args.command == "collect":
report = {"run": str(collect(args))}
else:
responses, provenance, config = load_round(args.intern_root, args.scorer_root, args.round)
report = replay_bundle(
args.intern_root, args.scorer_root, responses, **config, output=args.out, provenance=provenance
)
print(json.dumps(report, indent=2))
if (
args.command == "collect"
and not json.loads((Path(report["run"]) / "run.meta.json").read_text(encoding="utf-8"))["collected"]
):
raise SystemExit(2)


if __name__ == "__main__":
Expand Down
50 changes: 50 additions & 0 deletions benchmarks/results/intern-accuracy-v1-h800.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
# Intern-Decision accuracy-v1 on one H800

Three pinned Intern checkpoints used the official HF service, BF16, SDPA and
temperature 1. Each received five warmups, then one serial pass over all seven
suites: 10,751 requests and 12,351 decisions. All responses were valid.

## Results

| Suite | Requests / decisions | Intern 0.8B | Intern 2B | Intern 4B |
|---|---:|---:|---:|---:|
| JevBench easy | 48 / 48 | 47/48 | 48/48 | 48/48 |
| JevBench original | 72 / 72 | 57/72 | 61/72 | 71/72 |
| JevBench hard | 111 / 111 | 60/111 | 72/111 | 81/111 |
| AG News | 7,600 / 7,600 | 6,735/7,600 | 6,833/7,600 | 6,903/7,600 |
| ToolACE | 310 / 310 | 293/310 | 299/310 | 299/310 |
| Typed Decisions | 400 / 2,000 | 1,552/2,000 | 1,582/2,000 | 1,613/2,000 |
| WildJailBreak | 2,210 / 2,210 | 1,414/2,210 | 1,731/2,210 | 1,982/2,210 |
| **Mean of seven accuracies** | Each suite has equal weight | **79.4077%** | **84.7675%** | **89.8854%** |

The mean uses each suite's unrounded accuracy, not the pooled fraction over
12,351 decisions. Typed Decisions keeps all five fields in one request.
These are HF T=1 results; no XTuner calibration or published-score equivalence
is claimed. The pinned Hard input has no `gold_probs`, so TVD is unmeasured.

## Reproduce

Use the [verify, collect and replay commands](../INTERN_DECISION.md). Inputs and
the official evaluator are pinned to Intern-Decision
`3572c8a68b5df5dafe02d0e093989ba8ec0183bc`; JevBench is
`7ce310c7262ed49cc85853339a8a42459298e3f3`.

| Checkpoint | Revision |
|---|---|
| Intern-Decision-0.8B | `85a0cc5a99d67ea8d56dfe98115689212867171d` |
| Intern-Decision-2B | `8797836c65fc91a2435b1fb6850b5f0aabd75cc3` |
| Intern-Decision-4B | `0e5e6aa7d6d750e2b1504ba11a8136cb58aeb3cd` |

The service used PyTorch 2.9.1+cu128, Transformers 5.14.1, an 8192-token limit,
thinking off and no KV cache. Optional FLA/causal-conv1d paths were unavailable.
This run was collected on 2026-10-07 UTC, Slurm job 415106. It is a separate
allocation from the [five-model public231 comparison](public231-h800.md);
latencies from those two runs should not be combined.

A non-author checked every saved request and response against the fixed inputs,
then independently re-scored them and called the unmodified official `evaluate`
on CPU. Counts and metrics agreed. Source, environment and response hashes are
retained with the run. The data bundle is not vendored; its manifest leaves
AG News licensing unresolved. The public attachment retains predictions, scores
and timing, omitting input states and returned legends. Full raw evidence stays
with the original run; the public copy is not a complete replay bundle.
Loading