fix(metaserver): remove owner TTL from lifecycle sweep - #2
Closed
GentleCold wants to merge 16 commits into
Closed
GentleCold wants to merge 16 commits into
GentleCold wants to merge 16 commits into
Conversation
## Summary - classify `FullAttentionSpec` subclasses as full-attention cache groups - allow specialized full-attention specs used by the latest Kimi K3 vLLM release to initialize with aligned Mamba groups - cover subclass and MLA inheritance behavior in cache-group layout tests - bump the workspace and Python package version to `0.23.10` for release ## Testing - `cd python && uv run --extra test pytest` (256 passed, 13 deselected) - `cd python && uv run --isolated --no-project --with pytest --with numpy --with 'requests>=2.26.0' pytest` (256 passed, 13 deselected) - pre-commit hooks, including `cargo test --release`, Ruff, and Commitizen checks - `cargo metadata --locked --no-deps --format-version 1` - Cargo workspace, Python package, and Commitizen versions verified at `0.23.10` ## Release After merge, create and push tag `v0.23.10` to trigger the release workflow. ## Performance Work Not a performance change; no benchmark was required.
## Summary - keep polling an in-flight prefetch with its original query identity when local prefix hits change - let TP shard validation check the stale result against the query that created it - release stale per-shard leases before issuing the updated query ## Root cause The scheduler already snapshots `computed_blocks` and query hashes when `query_prefetch` returns Loading. After TP shard support was added, each shard validated a later Ready result against the current, shorter query before the scheduler could compare it with the saved snapshot. A valid stale result therefore raised `hits > queried blocks` instead of being discarded. ## Tests - `cd python && uv run --extra test pytest -q` (257 passed, 13 deselected) - `cd python && uv run --extra test pytest -q tests/test_tp_shards.py tests/test_combine_hashes.py` (66 passed) - commit hooks, including `cargo test --release`, ruff, format, typos, and Commitizen
…ntion and recurrent layers (novitalabs#433) ## What this changes End-to-end PegaFlow support for vLLM v1 hybrid KV cache models (full attention layers mixed with Mamba/KDA/GDN recurrent layers), landed on top of the group-aware plumbing introduced in recent connector work and proved out on GB300 (SM100) with Kimi-K3. Before this, every engine-side keying decision assumed a single cache group: group 0's content hashes addressed all segments, so recurrent groups were never saved, and a warm restore for a hybrid model could only answer attention hash queries. This patch teaches the whole stack about cache groups: ### Engine (`feat(core)`) - `pegaflow-common::block::group_hash` encodes a block content hash with its vLLM cache-group id. Group 0 stays byte-identical to the raw hash, so all existing single-group deployments produce exactly the same block keys as before. - `RegisterContextRequest` now carries `layer_group_ids`; the instance registry seals a per-group layer set and slot topology and refuses to serve instances whose workers disagree about group layout. - Read cache, write path, layup and offload all carry explicit group ids; slot derivation dedupes per group instead of per layer. - `QueryRequest.group_id` selects semantics: group 0 keeps the existing prefix query, other groups answer sparse-checkpoint membership queries. Membership results do not feed remote RDMA fetch in this release (see Limitations). - gRPC service routes save fan-out and per-(slot, hash) reports so the connector can place recurrent checkpoints where its membership leases require. ### Connector (`feat(connector)`) - Every group's block hashes are derived from the same prefix hash chain (group id + checkpoint-offset/private-salt emptiness mixed in) so a restore is only counted as a hit when every group's data can come back together. Group 0 provides prompt-layout blocks; recurrent groups provide sparse checkpoints at per-model-config positions, including the final HMA blocks (`fix(connector): save final HMA cache blocks`). - `_reconcile_hybrid` combines per-group answers into the longest fully restorable prefix; a partial reconcile is dropped entirely rather than restored with stale recurrent context. - Save/load intents keep per-group slot positions and write recurrent groups under their own hash namespaces inside the same request flow. - TP shard topology now accepts an explicit `pegaflow.tp_shard_endpoints` list so rank-to-server mapping no longer depends on every server being reachable at the same host:port pattern; without it the previous deterministic default layout is preserved. ## Verification - New coverage: `pegaflow-core/tests/hybrid_groups.rs`, instance-slot sealing tests for multi-group topologies, and `python/tests/test_hybrid_reconcile.py` for the reconcile/drop/final- block contracts. All existing engine, connector and fault-tolerance gates pass unmodified (`prek run` green: fmt, clippy, release-profile tests, ruff, typos). - Cluster validation on GB300 (SM100, aarch64): Kimi-K3 (MLA + KDA), 6 `xingming-k3-debug` LWS groups of TP8x2 with per-tray `pegaflow-server` shards (`550gb` hugepage pools each). Under Kimi production prefill traffic, port 8001 shows 24/24-25/25 layout block hits after a cold pass (TTFT 4.9s -> 1.7s for a ~39K-token prompt), and pooled saves/loads run at multi-TB/hour overnight with `pegaflow_cache_block_hits`-driven throughput scaling as expected. ## Limitations - The vLLM connector's reconcile keeps using the local membership form: its sparse queries stay resident-cache-only, so hybrid models still recompute after local eviction of recurrent checkpoints. Engine-side the limitation is closed: `wait_for_full_prefix` on a group > 0 query selects an all-or-nothing set fetch that routes misses through the same SSD prefetch and MetaServer + RDMA machinery as prefix queries (the want-set either completes or reports short, which callers treat as a miss) — the form a prefill/decode handoff consumer needs. Wiring the connector's warm-restore path onto it is follow-up work. ## Test plan - `cargo test --workspace --no-default-features --features cuda-13,rdma` - `cd python && uv run --extra test pytest` - `python/tests/test_vllm_e2e_correctness.py -m e2e` (hybrid model, GB300 cluster): cold -> warm verify loop with `hit_blocks` matching the full layout block count. --------- Co-authored-by: xiaguan <rsh56dwgsw@privaterelay.appleid.com>
## Summary - reclaim only cache entries exclusively owned by the resident cache - keep leased, queried, loading, and weakly referenced blocks indexed during memory pressure - preserve reclaimable-before-retained ordering with a bounded LRU scan - strengthen the GPU eviction integration test to prove an unleased batch is reclaimed while a leased batch remains queryable and loadable ## Testing - `cargo test --no-default-features --features cuda-13,rdma -p pegaflow-core storage::read_cache::tests -- --nocapture` - `cargo test --no-default-features --features cuda-13,rdma -p pegaflow-core --test eviction -- --nocapture` - `cargo clippy --workspace --all-targets --no-default-features --features cuda-13,rdma -- -D warnings` - `prek run` (includes release-mode Rust test suite) Signed-off-by: xiaguan <751080330@qq.com>
## Summary - bump the Rust workspace packages to `0.23.12` - bump the Python package and Commitizen version to `0.23.12` - refresh the workspace package versions in `Cargo.lock` ## Validation - `cargo metadata --locked --no-deps --format-version 1` - `prek run` (includes Clippy and release-mode Rust tests) Signed-off-by: xiaguan <751080330@qq.com>
## Summary - Treat only additional strong `Arc<SealedBlock>` owners as pressure-eviction pins. - Allow SSD write-queue `Weak<SealedBlock>` references to yield to memory pressure, preserving the existing fire-and-forget SSD contract. - Add regression coverage for weak-only eviction and strong-reference retention. ## Validation Ran on RTX 4090 / Linux with CUDA 13 and RDMA enabled: - `cargo fmt --all -- --check` - `cargo test --no-default-features --features cuda-13,rdma -p pegaflow-core storage::read_cache::tests -- --nocapture` (22 passed) - `cargo clippy --workspace --all-targets --no-default-features --features cuda-13,rdma -- -D warnings` - `cargo test --no-default-features --features cuda-13,rdma -p pegaflow-core --test eviction -- --nocapture --test-threads=1` (3 passed) The existing `ssd_cache` integration suite was also attempted on the 4090 host, but its fixed 32 KiB test pool is split across the host's NUMA-local pools and exhausts before the first save; all 11 failures are `pinned pool exhausted`, before SSD assertions run.
## Summary - Increase the default local HLL size from `bucket_bits=14` (16,384 registers) to `16` (65,536 registers), reducing the theoretical standard error from about 0.8% to about 0.4%. - Record only the final miss suffix in HLL cardinality while keeping all queried blocks in `pegaflow_hll_total_requests`. - Add `pegaflow_hll_estimated_hit_rate`, calculated and clamped in the tracker, while retaining the existing cardinality and total metrics and labels. - Use stable `(namespace, block hash)` identities so local measurements do not collide across namespaces. - Update the metrics documentation and default windows to `15m,1h,1d`. ## Compatibility This is not an `/metrics` protocol breaking change: the HTTP endpoint, existing metric names, types, and labels remain available. The new gauge is additive. The existing PromQL remains valid, but the HLL cardinality semantics are now explicitly miss-only and the default precision is higher. MetaServer HLL aggregation, heartbeat, protobuf, and MetaServer metrics are intentionally excluded and will be handled by a follow-up PR. ## Validation - `cargo fmt --all -- --check` - `cargo test -p pegaflow-common` (34 passed) - `git diff --check` Server compilation could not complete on the macOS development host because the workspace's Linux-only `io-uring`/CUDA dependencies require Linux headers/toolchains; Linux CI remains the authoritative server build check.
## Summary - Fix `ConnectorContext.virtual_block_size` to match vLLM's scheduler block-hash granularity. - Keep DCP in the virtual block size, but do not multiply by PCP. - Update unit coverage for PCP-only and DCP+PCP configurations. Fixes novitalabs#428. ## Why vLLM scheduler block hashes use `block_size * dcp`; prefill context parallelism changes rank ownership, not token granularity. Pegaflow currently multiplies by `pcp_world_size` as well, making the expected block size 8x too large with PCP8 and causing the first prefix-cache hit to fail. PCP deployments currently work around this with a startup `sed` patch; this makes the behavior native to Pegaflow. ## Duplicate-work check - Checked issue novitalabs#428 and open PRs for `428`, `PCP`, `context parallel`, and `virtual_block_size`. - No open PR covering this fix was found. ## Tests - `UV_CACHE_DIR=/tmp/pegaflow-uv-cache uv run --isolated --no-project python -c '...'` (syntax compilation passed) - `git diff --check` (passed) - Full pytest could not run because the environment cannot resolve `pypi.org` to install test dependencies. AI assistance was used to prepare this change; human review is required before merge.
## Summary - replace the single-owner prefix result with an ordered `FetchSegment` plan that covers the longest contiguous remote prefix - exclude the requester from owner candidates and greedily choose the owner that reaches the farthest from each offset, with a stable node-name tie break - validate and fetch segments serially in core; stop immediately on the first short read or RDMA failure, keep the already fetched contiguous prefix, and do not retry another owner or query MetaServer again - preserve the single-owner fast path as one segment and add per-segment block counts to the fetch summary log - reserve the old protobuf response field so mixed-version deployments degrade to a cache miss instead of misinterpreting the payload ## Why In multi-turn workloads, an older prefix segment may survive only on one remote P node while a newer segment is stored on another. The previous API selected only one owner, so the requester could fetch the first segment but had to recompute the rest even though the full prefix existed across nodes. ## Validation Executed on `192.168.172.117` with CUDA 13 and RDMA: - `cargo fmt --all -- --check` - `cargo test -p pegaflow-metaserver`: 38 passed - `cargo test -p pegaflow-core rdma_fetch --no-default-features --features cuda-13,rdma`: 8 passed - `cargo test -p pegaflow-core metaserver_client --no-default-features --features cuda-13,rdma`: 9 passed - `cargo test -p pegaflow-proto`: passed - `cargo clippy -p pegaflow-core -p pegaflow-metaserver --all-targets --no-default-features --features cuda-13,rdma -- -D warnings` - `cargo build --release --no-default-features --features cuda-13,rdma` The full core run passed 157/157 non-GPU unit tests and the integration suites reached `ssd_cache`; those 11 tests failed during harness setup with `pinned pool exhausted`. The same single test fails identically in the baseline workspace on this host, so it is an environment condition rather than a feature regression. ## 6P2D E2E evidence Real fragmented multi-turn A/B, Qwen3-4B, 6P2D, 128-token blocks, 8 active conversations, no HTTP retry: | Metric | Baseline | Feature | | --- | ---: | ---: | | Target RDMA blocks | 20/42 | 42/42 (20+22) | | Target hit ratio | 47.6% | 100.0% | | Mean target TTFT | 2776.2 ms | 2123.6 ms | | Whole-workload hit ratio | 48.6% | 53.6% | All four paired target requests returned HTTP 200 with identical prompt hashes. The feature recovered 88 additional RDMA blocks, avoided recomputing 88 blocks, and reduced mean target TTFT by 652.6 ms (23.5%). A separate fixed-transcript 96 conversations x 12 turns run also used identical prompts and exercised many multi-segment plans. Its aggregate hit ratio was effectively flat (74.9645% vs 74.9586%) because target-path requests were diluted by 1152 total requests; median TTFT improved from 5019.7 ms to 4713.1 ms, while tail latency remained noisy. The fragmented paired A/B above isolates the mechanism without synthetic cache operations or retries. Private benchmark harness and artifacts remain under `.local/` and are intentionally not part of this PR.
## Summary - Replace one whole-batch SSD prefetch slab per NUMA node with bounded 256 MiB staging chunks. - Preserve per-slot NUMA placement, slot ordering, oversized-slot handling, and the existing all-or-nothing batch allocation contract. - Add a representative multi-group regression shape that verifies the allocation bound. ## Production evidence before this change Observed on one GB300 4-GPU node running a hybrid multi-group workload: | Metric | Observed value | | --- | ---: | | SSD-prefetched data | 83 blocks / 8.52 GB | | Largest contiguous staging request | 3.5 GiB per NUMA node | | Cache blocks evicted | 4,608 | | Logical cache bytes evicted | 415.6 GiB | | Allocator bytes reclaimed | 462.8 GB | | Pinned-pool usage | 484.5 GB -> 30.2 GB | The allocator reclaims until its largest free extent can satisfy the request. A multi-GiB whole-batch slab therefore turns fragmentation into a large eviction burst. This is the SSD equivalent of the RDMA staging issue addressed by novitalabs#387. ## Allocation-shape A/B The deterministic multi-group regression uses 56 blocks with eight 16 MiB slots split evenly across two NUMA nodes: | Allocation shape | Baseline | This PR | | --- | ---: | ---: | | Bytes staged per NUMA | 3.5 GiB | 3.5 GiB | | Maximum contiguous allocation | 3.5 GiB | 256 MiB | | Allocations per NUMA | 1 | 14 | | Slots preserved | 224 | 224 | This establishes the allocation bound. Runtime eviction and hit-ratio impact still need to be re-measured after deployment. ## Validation - `cargo fmt --all -- --check` - `cargo clippy -p pegaflow-core --all-targets --no-default-features --features cuda-13,rdma -- -D warnings` - `cargo test -p pegaflow-core --no-default-features --features cuda-13,rdma -- --test-threads=1` (154 passed, 1 existing GPU-only test ignored) - `prek run` --------- Signed-off-by: xiaguan <751080330@qq.com>
…ovitalabs#441) ## Summary - Hybrid (HMA) requests now skip mid-flight saves and emit a single recurrent checkpoint from `request_finished`. - That checkpoint is resolved through vLLM's committed prefix cache instead of the live align-mode block table, which otherwise stores a mid-block running state or a speculative draft block under the wrong hash. - Regression tests build the recurrent table with vLLM's align-mode formulas and encode provenance in block ids so a wrong save is visible. This is not duplicating an existing PR: the 5090 exploration session never landed a commit or PR. ## Test plan - [x] `cd python && uv run --extra test pytest` — 278 passed, 13 deselected - [x] `tests/test_hma_checkpoint_index.py` — 9 cases (mid-block / aligned / speculative, plus skip-when-stored and new-boundary-only) - [x] Local Qwen3.5-4B GPU invariant script (`logs/k3-hma/verify_hma_invariants.py`): vbs=528, mid-block keys last committed boundary, short skip, warm hit no re-save with matching greedy text, prefix-extend files only the new boundary, independent miss, two-block checkpoint at 1056 - [x] `pytest -m e2e tests/test_vllm_e2e_correctness.py --model /data/models/Qwen3.5-4B --max-model-len 4096` — 5 passed (9m30s); same-process HMA load used PegaFlow; no data-path RPC / KV load failures --------- Co-authored-by: Cursor <cursoragent@cursor.com>
## Summary - bump the Rust workspace packages to `0.24.0` - bump the Python package and Commitizen version to `0.24.0` - refresh the workspace package versions in `Cargo.lock` ## Validation - `cargo metadata --locked --no-deps --format-version 1` - `prek run` (includes Clippy and release-mode Rust tests) Signed-off-by: xiaguan <751080330@qq.com>
novitalabs#448) ## Problem On hybrid models (Kimi-K3 / Kimi-Linear: KDA recurrent state + MLA attention) the PegaFlow connector's hit rate on agent workloads is far below the Mooncake store connector's, and it is not a capacity issue. The connector stored **one** recurrent checkpoint per request, at `request_finished`, taken from whatever boundary block vLLM still had cached. Agent traffic shares a system prompt across sessions and branches mid-history, so the shared prefix almost never ends at that single checkpoint. `reconcile_hybrid_hit` then finds the attention pages but no usable recurrent state and logs ``` [PegaKVConnector] req=... HMA attention prefix of N blocks has no common recurrent checkpoint; recomputing instead ``` i.e. a 0-block hit on exactly the traffic Mooncake serves from the second session on. (novitalabs#444 does not address this; it only restores local hits.) ## Fix Store recurrent states the way vLLM offers them (same hand-off the Mooncake connector consumes since vllm-project/vllm#51358): - **Per-boundary saves.** Consume `SchedulerOutput.kv_connector_block_state.boundary_state_offloads` every step. Each `(group, block, boundary_tokens)` entry becomes a boundary save job filed under the request hash ending at that boundary: chunk ends, internal prefill checkpoints, every block crossed while decoding. The block is pinned in the GPU block pool (`pool.touch`) until every worker reports the job via the new `PegaWorkerMetadata` (`KVConnectorWorkerMetadata.aggregate` across TP), so the save is decoupled from the request's block lifetime; `has_pending_push_work` keeps the engine stepping while jobs are in flight. HMA now requires a vLLM with this hand-off and fails loudly without it. - **Attention pages save mid-flight** at the normal per-block cadence with recurrent rows nulled; recurrent blocks are never saved positionally from the connector's table mirror. Request saves run asynchronously again (the synchronous D2H in `wait_for_save` is gone). - **Junction hint.** When the external attention prefix runs past the last usable checkpoint, set `Request.shared_prefix_boundary` so vLLM stops the prefill chunk there and commits (and hands off) the state at the end of the shared prefix. The first sharer recomputes once; every later sharer resumes. The first commit is a prerequisite for running on current vLLM main at all: vLLM now allocates one KV buffer and hands each layer a strided `[B, H, N, C]` view at a nonzero storage offset, which registration rejected (zero-offset assertion; MLA split ratio read from the head axis). ## Verification Kimi-Linear-48B-A3B-Instruct, single GB300, vLLM main (2026-09-01 nightly, 1920-token blocks, mamba `align`), `pegaflow-server` 40 GiB pool, greedy decoding. Prompts share a ~10.5k-token system prefix `S` (5 full blocks); E/F share only the first ~4.3k tokens of `S` (2 full blocks) and then diverge for ~3k unique tokens, so their common prefix ends where no earlier checkpoint exists. | request | master | this PR | |---|---|---| | A cold | 0 / 5 blocks | 0 / 5 | | A' exact repeat | 5 | 5 | | B, C sharers of full `S` | 5 | 5 | | D next turn of A | 5 | 5 | | E first sharer of the 2-block prefix | 0 (`no common recurrent checkpoint`) | 0 (`no common recurrent checkpoint`, junction committed) | | **F second sharer of the 2-block prefix** | **0** (`no common recurrent checkpoint`) | **2 / 2 blocks** | Greedy output of A' equals A in both runs; no load/save failures; `pegaflow_cache_block_hits_total` matches the per-request hit counts. Unit gate: `uv run --isolated --no-project --with pytest --with numpy --with 'requests>=2.26.0' pytest` → 288 passed. New `tests/test_hma_boundary_offloads.py` covers pinned saves and release across workers, worker-meta aggregation, unusable/duplicate hand-offs, attention-only mid-request saves, and the junction hint. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_016pSeF3SWkaeomAW4GQ7M6z --------- Signed-off-by: JinYan Su <751080330@qq.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
… table (novitalabs#449) ## Summary Production K3 (TP8, MTP, PegaFlow master with novitalabs#448) hit this on every attention layer once KV pressure started preempting requests: ``` RuntimeError: save block/hash count mismatch for language_model.model.layers.3.self_attn: blocks=1 hashes=10 ``` - **Root cause** (`scheduler.py::_consume_full_block_saves`): the save window was bounded by `base_block_idx + len(table)`, but the mirrored table already contains the externally loaded prefix, so the external offset was counted twice. It stayed masked while the scheduled-token accumulator only covered the current life. After a preemption vLLM resets `num_computed_tokens` and rebuilds the table from the resume chunk, while the accumulator and the first life's offset stay: the window runs past the rebuilt table, Python slicing silently shortens the block IDs, and the prompt hashes (known up front) are not shortened. In save-only mode the same stale watermark could save a partially recomputed block under a valid hash. - **Fix**: bound the window by the table itself; on resume restart the per-request bookkeeping from the refreshed external hit (`_rebase_resumed_request`), keeping already-stored progress. - **Blast radius fix** (`worker.py`): the raise killed the save thread, so nothing drained the queue afterwards, every later request stayed held for a save that never completed, and the engine leaked KV cache until restart (2 running / 18 waiting / 2 tok/s in the incident). The save worker now survives a failing batch, a malformed intent drops only its own request's save, and completion is always reported so blocks get released. - **Second bug, found while reproducing under preemption on Kimi-Linear** (`scheduler.py::_load_block_ids_by_group`): the load's destination vector was sized by the *usable* hit, but the engine walks it in lock step with the query lease. When the hybrid reconcile fell back to an earlier checkpoint (an exact-repeat prompt whose final token must be recomputed), the load carried one target fewer than the lease and the engine rejected it with `query lease block count 5 does not match destination block count 4`, which the worker escalated to "Service unavailable" and the engine died. The probe now keeps the leased block count and the load pads unwanted leased blocks with `None` targets (already skipped by the engine). - Bumps the version to 0.24.2. ## Test plan - [x] `tests/test_save_intent_preempt_resume.py`: the production shape (19-block prompt, 9 loaded, preempted after a sub-block chunk, resumed) plus save-only partial-block and rebase cases. All 5 fail on master, pass here. - [x] `tests/test_connector_save_lifecycle.py`: malformed intent is skipped and the request still completes; save worker survives a failing batch. - [x] `tests/test_hma_boundary_offloads.py::test_load_targets_cover_every_leased_block_when_the_hit_shrinks`: fails on master, passes here. - [x] `cd python && uv run --isolated --no-project --with pytest --with numpy --with 'requests>=2.26.0' pytest` on the connector suites (109 passed), ruff check/format clean. - [x] Single-GPU Kimi-Linear e2e (tray08, 512 GiB pool, 400-block GPU KV budget to force preemption, 2048-token chunks): master died on the lease/destination mismatch under a warm-prefix + 96 long-decode fillers + long-tail scenario; this branch completes 3 rounds (294 requests, 3908 preemptions, 0 save mismatches, 0 load failures). Multi-turn vllm-bench load (128 conv, 3 turns, 2000+ preemptions) is clean on both. The exact save-mismatch sequence from production was not reproduced e2e on this model (its resume hits cover everything stored, so the rebuilt table is never shorter than the hashes); the unit test pins that shape. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01SXoijrttVSfhrQF6EAq5CC --------- Signed-off-by: JinYan Su <751080330@qq.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
POST /admin/cleanup-expired-blocksfor explicit one-hour owner cleanup.Validation
192.168.172.117):cargo test -p pegaflow-metaserver(42 passed).cargo clippy -p pegaflow-metaserver --all-targets -- -D warningspassed./healthreturnedok,POST /admin/cleanup-expired-blocksreturned valid JSON, and metrics were exposed.A complete 2P2D GPU workload was not started because the host already had a long-running vLLM workload occupying the GPUs; no existing process was stopped.