Add distributed worker mode to runner

[gissf1]

Proposal

At some point, perhaps we should define a "slave mode" for the runner whereby a dedicated slave system can continually pull individual model test requests from the bakeoff-results live site to run the model on the runner's hardware, store the results locally, and then push the results record upstream for incorporation into the global dataset. This would make it easy to coordinate dedicated runner systems which can keep running tests to contribute to the global dataset indefinitely. If there are no tests for this runner, it should just sleep for some time before checking again to avoid busy-looping load.

I believe we should create a companion ticket in bakeoff-results for the UI and queue API aspects of this.

Touches benchmark invariants?

• No

[AlbinoGeek]

Proposal noted. The existing queue infrastructure from #13 provides most of the schema foundation.

What's already in place:

• run_queue table — DB-authoritative PENDING → CLAIMED → IN_PROGRESS → COMPLETE lifecycle; claim protocol is FOR UPDATE SKIP LOCKED with priority ordering
• runners table — Ed25519 signing identity per runner (public_key), heartbeat tracking (last_heartbeat, hostname, process_id, effective_user), ACTIVE / IDLE / DEAD status
• run_queue.claimed_by FK → runners.runner_id
• Capability-filter claim query already modeled in schema (VRAM gate in quantization_methods.vram_multiplier)

What's missing for distributed worker mode:

1. HTTP queue API on bakeoff-results — claim endpoint, result submission endpoint, runner registration endpoint
2. Pull client in bench/ — polling loop, exponential backoff on empty queue, heartbeat emitter
3. Runner registration flow — keygen + API auth exchange + runners INSERT

The companion ticket in bakeoff-results is the right split: queue schema + local worker client here, HTTP queue API + auth surface there. Flagging for NOMAD to authorize opening the companion bakeoff-results issue.

One terminology note: "worker mode" or "distributed runner mode" over "slave mode" for the public-facing flag name — aligns with modern tooling conventions (--mode worker, --worker, etc.) and avoids ambiguity in documentation.

— Bastion // 050813ZJUN26

[gissf1]

Since this is an API, we should have a whitelist (initially) on bakeoff-results of who is allowed to submit results and add a distributed worker to the queue. Eventually the whitelist should probably become a larger scale authentication system, maybe approved Oauth with allow keys?

Let me know the bakeoff-runner ticket (link?) when one is opened. We should probably cross link them for knowledge transfer and easy reference anyhow.

Agreed on "distributed worker" to replace "slave" terminology. I will update this ticket title accordingly.

[AlbinoGeek]

Acknowledged. Confirming for the record:

Auth / whitelist path: Agreed — initial submission whitelist on bakeoff-results, evolving to OAuth with approved keys. Will carry this constraint into the HTTP queue API design in the companion bakeoff-results issue.

Companion ticket: The companion bakeoff-results issue (HTTP queue API + runner registration + auth surface) is pending NOMAD authorization to open. Will link it here cross-directionally once authorized and created.

Terminology: Confirmed — "distributed worker" / --mode worker / --worker across all public-facing identifiers. Will reflect in both tickets.

Standing by for companion issue authorization. Will post the link as soon as it exists.

— Bastion // 050907ZJUN26

[AlbinoGeek]

Companion issue opened: Rethunk-AI/bakeoff-results#31 — HTTP queue API, runner registration, and submission auth for distributed worker mode. Cross-linked both directions.

— Bastion // 050911ZJUN26

[gissf1]

I'm assuming there is nothing for me to do in this ticket at this time. When there is a need for me to interact, notify me with an @ ping.

[AlbinoGeek]

Confirmed — nothing on your plate for #37 right now.

Status: the queue schema + local worker client live here; the companion HTTP queue API + submission/auth surface is tracked in Rethunk-AI/bakeoff-results#31 (initial submission whitelist → OAuth-with-approved-keys path recorded there). Both tracks are paused pending NOMAD authorization to build.

I'll @-ping you the moment your input is actually needed — most likely to verify the auth/whitelist model or the runner registration handshake before they're finalized. Title rename to "distributed worker" remains yours to make (or say the word and I'll do it).

— Bastion // 060205ZJUN26