fix(metaserver): preserve active blocks and skip healthy sweep scans

[GentleCold]

Problem

The 600-second MetaServer sweep walked every block and rebuilt redundancy state even when all nodes were healthy. During a real vLLM run this can hold block shard locks long enough for query-prefetch retries to grow.

Change

• A healthy sweep checks node liveness only. A block walk runs after an actual node removal or pending session takeover.
• Redundancy counters are maintained incrementally under existing shard guards; metrics read the counters without scanning block shards.
• Registration age is removed from background sweep decisions.
• Add POST /admin/cleanup-expired-blocks to the existing --http-addr listener (default 0.0.0.0:9092). It removes owners registered strictly more than one hour before the call, preserves nodes and physical KV data, and returns removed_owners/removed_keys.

RTX 4090 vLLM A/B

Master 0c49ca3 and feature 6365f1a ran sequentially on the same host with two TP2 DeepSeek-V2-Lite-Chat instances, block size 16, two PegaFlow servers, 400 GB pools, 1M+ retained logical keys, 660 seconds of steady traffic, and an added identical 20-request/s real short-cold stream. The measurement proxy adds one identical local gRPC hop to both variants.

Metric	Master	Feature
Keys after seed drain	1,009,664	1,001,472
Steady QueryPrefixBlocks mean / P99 / max	0.751 / 1.750 / 382.747 ms	0.570 / 1.432 / 24.074 ms
Steady InsertBlockHashes mean / P99 / max	0.554 / 1.574 / 66.977 ms	0.519 / 1.522 / 33.749 ms
Steady HeartbeatNode mean / P99 / max	0.609 / 1.474 / 4.857 ms	0.543 / 1.183 / 1.930 ms
Query max near the 600-second sweep (3-second window)	382.747 ms	1.140 ms
Insert max in the same window	1.568 ms	7.448 ms
query-prefetch rate in sweep bin / nearby median	147.5 / 51.7 s⁻¹	49.2 / 49.5 s⁻¹
Steady MetaServer RPC errors	0	0

The master query stall and 2.85x query-prefetch increase occur in the same sweep bin. Feature has no corresponding query stall or rate spike, but its isolated Insert maximum is higher in that window; causality is unresolved. Each steady run includes approximately 25k Inserts and 88 Heartbeats, limiting heartbeat tail conclusions. All steady RPCs and inference requests succeeded; one feature pressure request returned 500 during teardown after finished.

The proxy captured QueryPrefixBlocks, InsertBlockHashes and HeartbeatNode. No RemoveBlockHashes or UnregisterNode calls were recorded, so their latency is unmeasured. HTTP cleanup measures an internal MetaServer scan; it does not cover the server-to-MetaServer RemoveBlockHashes RPC. RPC timing starts at proxy forwarding and excludes time queued in the PegaFlow server.

Cleanup endpoint measurement

All feature requests were made while the real vLLM load continued.

Call	HTTP	Removed owners / keys	Duration	Overlapping Query / Insert max
Fresh scan 1/2/3	200	0 / 0 each	291.1 / 161.2 / 200.9 ms	≤1.148 / ≤0.947 ms
Aged (>1h)	200	182,272 / 182,272	394.4 ms	1.087 / 0.825 ms
Aged repeat (+30s)	200	77,952 / 77,952	265.8 ms	5.476 / 1.881 ms

The repeat call overlapped only 4 Queries / 9 Inserts. Query mean before/during/after was 0.588/4.709/0.600 ms; Insert was 0.572/1.084/0.507 ms (30-second before/after windows). This shows a transient increase, with too few samples for stable tail estimates. The extra 77,952 deletions match seed registrations crossing the one-hour cutoff during the 30-second gap.

The cleanup POST is served on the existing HTTP listener and remains POST-only (GET returns 405). Two nodes remained active throughout; final retained keys were 1,066,748. The aged calls delete metadata owners only, so existing physical KV allocations are not reclaimed by this endpoint.

This is one sequential A/B on a single host using real vLLM-generated metadata registrations; the vLLM correctness gate was not run. Full raw artifacts and analysis are in .local/reports/pr451-vllm/; the remote run directories are under /root/kexi/vllm_meta_ab/.

[Copilot]

🟡 Changes recommended

There is a confirmed redundancy-counter correctness bug during heartbeats (causing metric drift), and the new unauthenticated destructive admin endpoint is exposed on the default 0.0.0.0 HTTP listener.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR updates the MetaServer’s lifecycle management to stop expiring block owners based on registration age during the periodic sweep, and instead provides an explicit operator-triggered cleanup endpoint while maintaining redundancy metrics without full scans.

Changes:

• Reworked sweep_expired to remove owners via a node→block reverse index and maintain redundancy counters incrementally.
• Added POST /admin/cleanup-expired-blocks to explicitly remove owners older than 1 hour.
• Deprecated ttl_minutes behavior in the MetaServer CLI and logs, keeping the flag for compatibility.

File summaries

File	Description
pegaflow-metaserver/src/store.rs	Removes owner-age TTL from periodic sweep, adds reverse index + incremental redundancy counters, and adds manual cleanup API support logic.
pegaflow-metaserver/src/lib.rs	Deprecates ttl_minutes semantics and wires the store into the HTTP server startup.
pegaflow-metaserver/src/http_server.rs	Adds the admin cleanup endpoint and a small route-level test.
pegaflow-metaserver/Cargo.toml	Adds deps needed for JSON response serialization and HTTP route testing.

Review details

• Files reviewed: 4/4 changed files
• Comments generated: 3
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[GentleCold]

Addressed the review feedback in ab4215e: heartbeat redundancy accounting uses one definition, admin cleanup is loopback-only on a separate listener, and synchronous store work is batched/offloaded to the blocking pool. Added regression coverage and updated the PR description with RTX 4090 validation data. Please re-review.

[xiaguan]

Since these services are internal by default, could we reuse the existing HTTP listener for cleanup and drop the separate admin listener and configuration?

[GentleCold]

Implemented in commit 90859d2.

The manual cleanup route now reuses the existing --http-addr listener. Removed the separate --admin-http-addr CLI option, second listener/task, loopback validation, and the corresponding README configuration. The route remains POST-only at /admin/cleanup-expired-blocks; the HTTP listener is intended for the internal MetaServer network.

Updated the route tests and verified git diff --check. This environment does not have Cargo/rustfmt installed, so a local Rust build/test could not be run.

[GentleCold]

Follow-up: the PR description now reflects the shared listener as well. The implementation is pushed as 90859d2; PR head is updated and the review request is addressed.

[xiaguan]

LGTM on 2e139c1.

Reviewed lifecycle cleanup, session replacement and unregister races, incremental owner accounting, and the shared HTTP listener. Local validation: 45 metaserver tests, strict Clippy, formatting, and a real HTTP smoke test passed; all 12 CI checks are green.

Non-blocking cleanup: remove the obsolete README statement that the public listener returns 404 for the cleanup route, and consider dropping the redundant route-availability test already covered by the preceding cleanup test.

The known lost-removal metadata risk remains a follow-up. The dedicated vLLM correctness E2E gate was not run; the maintainer has elected to proceed with this limitation disclosed. No independent performance claim is made here.