feat: let Reef own inference and training coordination

[BobbyZhouZijian]

Motivation

Slime's bootstrap and bridge owned inference lifecycle, weight publication and recovery. This coupled future training and inference backends to Slime's rollout management. Reef now coordinates independent backend implementations, including when a trained version may begin serving.

Implements RFC #425 in this draft PR. The RFC remains proposed; maintainer acceptance and GPU validation remain open. This does not implement the separate vLLM/MLX work in #381/#325.

Changes

• Define independent TrainingRuntime / InferenceRuntime scheduling interfaces and native TrainingBackend / InferenceBackend ABCs in reef/runtime/interfaces.py. Concrete implementations inherit them; there is no aggregate ModelRuntime.
• Restrict reef/runtime/ to five modules and executor/: interfaces.py for contracts/values, scheduler.py for scenario scheduling and serialized training/resource handoffs, deployment.py for plans/resources/factory discovery, publication.py for transfer/version/commit coordination and LoRA residency, and recovery.py for durable state and reconstruction. Directory, dependency and installed-wheel checks enforce the layout. Runtime is a PEP 420 namespace package without a package facade.
• Keep RuntimeScheduler and worker-side TrainingCoordinator as distinct objects in scheduler.py. Actual weight transfer ordering, target allocation, receiver verification and adapter replay live in publication.BackendWeightPublisher; startup restoration lives in recovery.TrainingRecovery. Shared store ABCs avoid circular imports while preserving disk-first job transitions.
• Move HTTP/generic inference runtimes to reef/inference/, generic training connections to reef/train/runtime.py, and coordinator RPC transports to runtime/executor/connection.py. Generic pair assembly lives in reef/service/runtime.py. Builtin factories resolve lazily without package-import registration side effects.
• Put concrete SGLang engine/router launch, request capture, plugin, configuration, weight/LoRA receipt and memory operations in reef/inference/sglang/. SGLangInferenceRuntime owns its default request handler. SlimeTrainingRuntime connects training and uses the independently selected inference factory. Runtime imports neither integration; training and inference integrations do not import each other, including from local functions.
• Let Reef assemble separate unallocated training/inference components, validate native transport compatibility, reserve resources, fence inference before colocated training initialization, attach a fresh weight-transfer session, and start the generic coordinator. Cleanup, current-worker monitoring and bounded reconstruction follow component ownership.
• Preserve full-weight/LoRA publication and colocated resource handoffs. Reef persists each transfer target before sending, verifies the sender and every receiver, and keeps generation fenced until durable artifact commit acknowledgement. Sender preparation/export and trainer release precede receiver memory restoration. Native senders cannot independently pause, resume or restart inference.
• Name recipe update contracts CandidateBackend / RuntimeCandidateBackend and request execution InferenceHandler. Replace remaining first-party Protocol contracts with explicit inheritance and add CI enforcement against Protocol, runtime_checkable and TYPE_CHECKING, including alias forms. Protocol violations cannot be baselined; existing Callable debt does not grow. Repository-local uv caches are excluded as third-party dependencies, with regression coverage that still rejects authored Protocol definitions.
• Keep managed inference resources under inference.*, update maintained YAML/request-handler examples, backend-extension documentation, plugin registration and wheel checks. Fix base-wheel test collection when optional Ray is absent.

Tinker on this branch

• Tinker's inference side moved to reef/inference/tinker.py (SDK sampler, TinkerInferenceRuntime, chat handler). The trainer in reef/train/tinker_backend/ keeps its incumbent on disk; the two share only each artifact's tinker-checkpoint.json manifest. TrainingRuntime.commit_candidate (no-op default) lets the scheduler tell a trainer with immutable incumbents which candidate Reef committed.
• Local SGLang inference with Tinker training: training.backend: tinker plus inference.backend: sglang runs Reef's model driver with a GPU-less trainer (ModelGpuLayout accepts zero training GPUs), SGLang engines in LoRA serving mode, and TinkerTrainingBackend behind Reef's coordinator. Adapter delivery is a Reef-owned transfer (reef-adapter-files-v1): TrainingBackend.adapter_files returns the PEFT directory, InferenceBackend.load_adapter_files loads it at the receiver, the publisher drives both; inference services list the transfers they accept. The HTTP service connects through the generic coordinator_training runtime kind.
• tinker-cookbook==0.5.7 is pinned in the tinker extra for the checkpoint-to-PEFT conversion.
• recipes/tttd/examples/tttd/serve-tinker.yaml runs the Erdős example on Tinker; the harness reads its stack file from TTTD_STACK. Running it exposed two judge bugs (no cvxpy in the judge image although the instruction offers it; numpy results rejected by the runner's json.dumps), both fixed.

Compatibility and operational impact

• Extensions import runtime/native contracts from reef.runtime.interfaces and factory APIs from reef.runtime.deployment. The previous runtime package facade and base, backends, adapters, control, weights, training_job module paths are removed. ModelRuntime, ExecutorModelRuntime and aggregate RayRuntime aliases are removed. Model processes must run the same revision.
• Runtime factories and connection helpers return (training_runtime, inference_runtime). Generic executor_training / ray_training kinds remain. Automatic Slime assembly selects slime_training with an independent inference_runtime selector; SGLang is the currently supported native receiver. Coordinator transports use CoordinatorClient and Executor/Ray variants. connect_ray_runtime and connect_executor_runtimes live in reef.service.runtime.
• Native components, optional capabilities and factories require explicit ABC inheritance. Recipe extensions use CandidateBackend and Trainer.candidate_backend; request adapters use InferenceHandler, inference_handler, and its from_config classmethod. Evaluation plugin factories implement .build(...); header providers implement .headers(...). Existing native harness entrypoints retain boundary adapters.
• Request adapter configuration uses inference.handler-factory / inference.handler-config; former request backend-factory / backend-config keys are rejected. Native inference.backend / training.backend selection is separate. Complete dotted CLI flags override YAML; no public services section is added.
• inference.num-gpus, inference.tensor-parallel-size and inference.options are authoritative for managed inference; equivalent training-native rollout resource flags are rejected. training.colocate (default false) decides whether training and inference share the same GPUs; Reef makes the reservation and passes the colocation flags to the Slime driver. Model drivers return create_training_plan plus an independently selected inference definition. Direct start_bridge / TrainBridgeActorImpl construction is replaced by Reef assembly; legacy argument-file/healthcheck inputs remain supported.
• The existing slime-sglang-control-v2 native attachment and direct worker-to-worker weight transport remain. Runtime IDs retain <incarnation>:<sequence>; pending markers may contain target_runtime_load_id, and older markers remain readable. Artifact formats, provider bodies, scenario isolation and receipt linkage retain their contracts.
• Managed recovery retains HTTP and external Ray, allowing three component replacements per five minutes with 1/2/4-second delays. In-flight mutation/inference calls are not automatically replayed. Unconfirmed cleanup or ambiguous RUNNING state stops recovery; Slime rejected candidates require restoring committed trainer state before cold reconstruction.
• Borrowed engines, PRM and user-simulation services retain external ownership. Reef reserves the model GPUs itself (reef.runtime.executor.placement: one placement group sized from inference.num-gpus, the trainer's GPU count and training.colocate, with bundles sorted by node and device) and hands the training and inference portions to the backends; Slime no longer creates its own placement groups. training.colocate is the only colocation switch: training.options.colocate / offload-rollout are rejected, and the driver receives the matching native flags from Reef. This adds no cluster autoscaling, dynamic parallelism resizing or arbitrary backend transport interoperability. Initial colocated training releases all inference memory; keeping the LoRA base resident applies only to later steps.

Verification

Validation on macOS ARM64 without CUDA for the merged source in 8607194d, plus the subsequent CI cache-scan correction:

• Full source suite: python -m pytest tests -n 4 --dist loadfile --cov=reef --cov-report=term -q --tb=short --durations=15 — 3,800 passed, 60 skipped; 88.50% coverage. Repository environment, Node.js 22/uv on PATH, proxy bypass and isolated Git configuration.
• Installed base wheel, outside the repository with no Ray/Torch/Slime: package-CI service suite — 3,043 passed, 62 skipped. Used CI's training-dependent test exclusions.
• Real Ray CPU workers: REEF_TEST_RAY=1 over test_inference_control.py, test_training_restart.py and test_model_supervision_ray.py — 12 passed. Covers moved coordinator/control paths, named discovery, publication, commit barriers, reconstruction and cleanup across processes. Run on 56bfbf87; the merge and cache-scan fix leave these runtime/inference modules unchanged.
• pre-commit run --all-files passed. python -m mypy passed for 275 source files. The design baseline retains 25 existing Callable entries; only moved-file paths changed. Directory/import boundary suite: 35 passed. Protocol policy suite: 45 passed, including reproduction of CI uv-cache false positives while still rejecting Protocol in owned source.
• Node.js 22: npm ci, npm run check:docs, npm run lint, npm run build passed.
• Built sdist/wheel, twine check and uv lock --check passed. An installed-wheel smoke check blocked Ray, SGLang, Slime, Torch, Megatron and Transformers imports while validating exact namespace contents, independent contracts, lazy native runtime factory discovery, generic connections, HTTP/model override construction and the SGLang plugin entry point.

Remote CI passed for cb51707a: lint and the test/package matrices on Python 3.10, 3.11 and 3.12 are all green. CI run.

Actual GPU checkpoint loading, LoRA IPC/NCCL transfer, colocated memory capacity and performance remain unverified. CPU contracts and process tests do not satisfy those acceptance gates.

Real Tinker runs on this branch (Qwen3-8B, CPU host, both in-process topology):

• tutorials/tinker smoke: four sampled rollouts, one remote optimizer step, one committed release.
• recipes/tttd/examples/tttd/serve-tinker.yaml on the Erdős task through harbor, thinking on, 26,000-token budget, 2 x 2 grid: two of four programs valid (C₅ ≈ 0.4995 / 0.5000), one training step committed, incumbent advanced. The 2 x 2 grid degenerates TTTD's leave-one-out advantages (±1e12), so that step checks the mechanism, not the method; a meaningful step needs more rollouts per group.
• Restart and rollback on real Tinker (in-process topology): the restarted stack rebound the serving version to its new incarnation and trained step 2 from the committed adapter (best reward 2.612, C₅ ≈ 0.383); a rollback to the step-1 release minted a new served version and moved the trainer's incumbent back to that checkpoint.
• A non-degenerate step, 2 groups x 8 rollouts with the paper's settings: advantages in [-0.90, 3.46], adaptive betas ≈ 1.2–1.6, loss:sum ≈ -5.5e4, four valid programs, best C₅ ≈ 0.386, one committed step.
• The local-engine topology end to end on CPU (tests/reef_service/test_tinker_local_engine_ray.py): a Ray cluster with pretend GPUs, the real model driver, coordinator and HTTP-side connection, a stub engine accepting the adapter-file transfer in place of SGLang. Passes with the fake SDK in CI and, run locally with REEF_TEST_TINKER_REAL=1, with the real SDK: real training step, archive download and cookbook conversion (110 s). The converted adapter has 506 tensors with Hugging Face parameter names and shapes matching Qwen3-8B at rank 32.
• Not run on GPU: an actual SGLang engine loading the adapter and serving it. Open risk found by the adapter check: Tinker also adapts lm_head, which SGLang's LoRA loader may not accept.

AI assistance

OpenAI Codex assisted with implementation, review, tests, documentation and this description at the contributor's direction. The human contributor remains responsible for reviewing and understanding the submitted changes and validation claims.

Checklist

• The change is focused and contains no unrelated cleanup.
• Tests cover behavior changes, or this pull request does not change behavior.
• Public interface changes include contract tests, or no public interface changes are present.
• Affected user and developer documentation is updated, or no documentation update is required.
• Root README changes update both languages and README.i18n.yaml, or do not affect README content.
• An accepted RFC is linked, or this change does not require an RFC.
• Relevant pre-commit and test checks pass.
• Non-trivial AI assistance is disclosed above, or none was used.

Review and merge

RFC #425 acceptance and GPU validation remain open. Follow the repository's maintenance model for review and merge requirements.

[github-actions]

Merge Oncall: @BobbyZhouZijian

The merge oncall will route reviews and keep this pull request moving toward a clear decision.

[hanfeiyu]

Overall this looks great. Thanks for taking on this substantial refactor!

approved