Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions docs/community-benchmarks.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ tensorfold serve example-org/public-checkpoint --name local-model
tensorfold benchmark example-org/public-checkpoint --server http://127.0.0.1:8080 --model-id local-model --backend mlx
```

Attached-server results do not claim the client's hardware is the server's hardware. The client reads `GET /v1/benchmark/manifest` from the server it attached to. A manifest that passes the receipt checks fills the version, hardware, checkpoint hashes, revision and rank count (1 or 2). Without that document, or if any field fails those checks, the receipt keeps unknown hardware and an unknown revision and remains unranked. An attached run records the backend, model and context you declare, unchecked against the server. The runner does not clear or restart an attached server's caches.
Attached-server results do not claim the client's hardware is the server's hardware. The client reads `GET /v1/benchmark/manifest` from the server it attached to. A manifest that passes the receipt checks fills the version, hardware (GPU cores included), checkpoint hashes, revision and rank count (1 or 2). The manifest names the backend as the receipt does (`rocm` for the HIP engine, `mlx` for Metal) and probes the hardware as a managed run of that backend would. The server attests the revision a managed run records: the name of the Hugging Face cache snapshot it loaded. `TF_CHECKPOINT_REVISION` stands in only when the server's checkpoint directory is not a snapshot; when it names another commit than the loaded snapshot, the manifest attests no revision. Without that document, or if any field fails those checks, the receipt keeps unknown hardware and an unknown revision and remains unranked. An attached run records the backend, model and context you declare, unchecked against the server. The runner does not clear or restart an attached server's caches.

The runner records failures, incomplete streams, missing usage, early EOS and cancellations. Exit code 0 means every measured attempt completed its requested output with a delivery interval; 1 means an error, and with `--publish` the receipt is still saved when the run finished; 2 means incomplete attempts remain in the saved receipt; 130 means the run was cancelled with Ctrl-C: an attempt in flight is kept in the receipt, and a cancel during server startup saves nothing. SIGTERM and SIGHUP exit 143 and 129 without a receipt. Errors never silently disappear from an average.

Expand All @@ -59,7 +59,7 @@ Open [the benchmark board](https://tensorfold.dev/benchmarks), choose **Upload r

That browser profile keeps later submissions together, with up to 20 uploads per day. Clearing browser data or using another browser creates a separate profile. Display names are self-chosen and do not verify an identity; matching names do not merge profiles. New-profile limits apply to reduce spam.

Complete standard community-v1 submissions publish automatically after quota checks, strict receipt validation, public-checkpoint lookup and server-side statistics/completeness checks. A complete standard run includes both workloads at the suite temperatures and seeds, five successful repetitions per cell, 256 output tokens, a native managed run or a validated attached manifest, an attested rank count of one or two, a checkpoint revision/configuration digest and reported GPU hardware. Attached runs without accepted manifest facts remain unranked. A published submission returns its public result URL.
Complete standard community-v1 submissions publish automatically after quota checks, strict receipt validation, public-checkpoint lookup and server-side statistics/completeness checks. A complete standard run includes both workloads at the suite temperatures and seeds, five successful repetitions per cell, 256 output tokens, a native managed run or a validated attached manifest, an attested rank count of one or two, a checkpoint revision/configuration digest and reported GPU hardware. Attached runs without accepted manifest facts remain unranked. A published submission returns its public result URL. `run`, `inspect` and `publish` say when a public managed receipt names no GPU, checkpoint revision or configuration digest, before anything is uploaded.

If the website's model-checkpoint service is temporarily unavailable, a valid receipt is saved privately and its public-checkpoint check is retried automatically. A complete standard run becomes public only after that check passes. You do not need to rerun the benchmark or upload the same receipt again. Private or inaccessible checkpoints are never published automatically. Valid incomplete or nonstandard runs remain pending review after public-checkpoint verification; invalid receipts are rejected.

Expand Down Expand Up @@ -106,4 +106,4 @@ The parameter-versus-speed plot uses the selected parameter basis on a logarithm

## AMD (ROCm)

A ROCm build runs the same suite and saves its receipt locally, with backend `rocm` on platform `linux`. `--backend auto` picks `rocm` on that build; an attached run names it with `--backend rocm`. A managed ROCm run records the CPU and system memory, while GPU names and the driver version stay unknown. `inspect` shows a ROCm receipt as usual. The board accepts mlx and cuda receipts for now, so `--publish` with ROCm is refused before the run starts and `benchmark publish` refuses a ROCm receipt before any upload; publishing waits for the board to accept rocm.
A ROCm build runs the same suite and saves its receipt locally, with backend `rocm` on platform `linux`. `--backend auto` picks `rocm` on that build; an attached run names it with `--backend rocm`. A managed ROCm run records the CPU (an AMD name's `w/` is written `with`, so `AMD RYZEN AI MAX+ 395 w/ Radeon 8060S` fits the board's name pattern) and system memory, and each GPU agent `rocminfo` lists: its vendor and marketing name (for example `AMD Radeon 8060S Graphics`), its first global memory pool and its compute units. `rocminfo` already lists only the GPUs `ROCR_VISIBLE_DEVICES` leaves visible, renumbered in that variable's order, so the probe does not apply it again. It keeps the agents HIP keeps, by `HIP_VISIBLE_DEVICES` or, when that is unset or only a pair of quotes, `CUDA_VISIBLE_DEVICES` (HIP on AMD reads both), with HIP's rules: an entry is an index written plainly (`1`; not `01`, `+1` or with a space) or `GPU-` text found in an agent's `Uuid` from `rocminfo`; the first entry that is invalid or past the last GPU ends the list; a repeated entry is skipped; one pair of double quotes around the value is dropped; the variable in use, set to nothing, hides every GPU. The `Uuid` is only compared, never recorded. An index counts every GPU agent, so an agent whose name fails the allowlist keeps its place: selecting it records no GPU, never the next card. When a GPU HIP keeps is not named, or a `GPU-` entry cannot be matched because `rocminfo` prints no `Uuid`, the hardware source is `partial`. An AMD APU, whose GPU is named "... Graphics", is recorded as `unified` memory; discrete cards as `dedicated`. The in-kernel amdgpu driver has no version, so the driver stays unknown. Without `rocminfo` the GPUs stay unknown, as before. `inspect` shows a ROCm receipt as usual. The board accepts mlx and cuda receipts for now, so `--publish` with ROCm is refused before the run starts and `benchmark publish` refuses a ROCm receipt before any upload; publishing waits for the board to accept rocm.
Loading