Skip to content

Make Eval SDK sheet operation waits websocket-first - #353

Open
hasaan21 wants to merge 2 commits into
masterfrom
hasaan/fix-eval-sdk-progress
Open

Make Eval SDK sheet operation waits websocket-first#353
hasaan21 wants to merge 2 commits into
masterfrom
hasaan/fix-eval-sdk-progress

Conversation

@hasaan21

@hasaan21 hasaan21 commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

Summary

Make Eval SDK sheet operation waits websocket-first so CLI cell_progress tracks the dashboard 1:1 via SMART_TABLE_EXECUTION_STATUS_UPDATE, with REST polling as fallback.

Changes

  • Add evaluations/live_progress.py Centrifugo listener for sheet execution-status updates (filter by operation execution_ids)
  • Wire wait_for_sheet_operations / await_for_sheet_operations to WS-first progress; sync path runs async under the hood
  • Keep light REST safety poll; fall back to REST if WS token/connect/subscribe fails
  • Support optional statuses on wait helpers (None = STALE-only API default; [] = force all cells)
  • Add SDK tests for matching IDs, ignored foreign IDs, and REST fallback

Test Plan

  • Tested locally
  • Added/updated tests
  • Verified in staging
  • poetry run pytest tests/test_eval_polling.py
  • Manual: sample-promptlayer-python/recalc_traces_download.py with statuses=[] → live cells N/M aligned with UI

Checklist

  • Code follows project conventions
  • No any types in TypeScript (frontend)
  • Tests pass locally
  • Documentation updated if needed
  • Update promptlayer-helm if you add a new envvar (env/ is tied to Cloud Run only and will be removed soon)

@hasaan21
hasaan21 force-pushed the hasaan/fix-eval-sdk-progress branch from 0750943 to 4947478 Compare August 7, 2026 20:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant