Skip to content

fix(prefill): isolate progress HTTP event loop - #193

Merged
FluffyAIcode merged 1 commit into
mainfrom
AgentMemory/prefill-progress-http-thread-0717
Jul 17, 2026
Merged

fix(prefill): isolate progress HTTP event loop#193
FluffyAIcode merged 1 commit into
mainfrom
AgentMemory/prefill-progress-http-thread-0717

Conversation

@FluffyAIcode

Copy link
Copy Markdown
Owner

Summary

  • run the inference-network Uvicorn server in its own thread and event loop
  • keep gRPC model operations on their existing MLX-affine execution path
  • allow heartbeat polling to read segment token progress while AppendTokens blocks
  • shut down and join the HTTP thread during graceful runtime termination

Root cause

The worker correctly reported segmented tokens_computed, but Primary's synchronous AppendTokens path occupied the same asyncio loop that hosted Uvicorn. Dashboard requests timed out during Prefill, so Terminal heartbeats displayed stale Generator progress instead of live Critic segments.

Tests

  • HTTP/gRPC event-loop isolation source contract
  • targeted bridge/network suite — 20 passed
  • full platform-neutral suite — 874 passed, 1 skipped
  • Python compile and diff checks pass

Made with Cursor

Run dashboard telemetry on a dedicated thread so blocking model RPCs cannot hide segmented Prefill progress or stale the Terminal ETA.

Co-authored-by: Cursor <cursoragent@cursor.com>
@cursor

cursor Bot commented Jul 17, 2026

Copy link
Copy Markdown

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.

@FluffyAIcode
FluffyAIcode merged commit 884c784 into main Jul 17, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant