Skip to content

Fix flaky Simulator stream E2E that blocked the 0.57.0 release - #508

Merged
sambitcreate merged 1 commit into
mainfrom
claude/project-thread-ur6whr
Oct 16, 2026
Merged

sambitcreate merged 1 commit into
mainfrom
claude/project-thread-ur6whr

Conversation

@sambitcreate

Copy link
Copy Markdown
Owner

Requested by sbt · project thread

Before: the 0.57.0 Release desktop run (run 38853835788) failed twice without publishing. Attempt 1 passed every test, then lost network during Apple notarization. Attempt 2 failed the deterministic Electron E2E gate because --fail-on-flaky-tests caught one flaky spec: tests/e2e/environment-devices-stream.spec.ts:107 › Simulator stream › paints MJPEG through the proxy, sends touches, reconnects input, and attaches a screenshot. The failure was expect.poll(... boundingBox().x).toBe(floated.x), which got 285 when it expected 285.15625. It passed on its retry.

After: that assertion compares whole pixels, the same way the drag assertion a few lines earlier in this test already does. The behavior it checks still holds: the floating player comes back where it was left after docking and floating again.

How: one line in the spec. --fail-on-flaky-tests, timeouts and retries are unchanged. Merging this to main re-triggers Release desktop, and because v0.57.0 does not exist yet, that run publishes 0.57.0.

Local checks: eslint on the spec and root tsc --noEmit pass. The spec itself needs macOS (Simulator), so the CI and release runs are its acceptance record.

🤖 Generated with Claude Code

https://claude.ai/code/session_01SJgWwVbexyFmdCVEcQPw7v


Generated by Claude Code

The Simulator stream E2E failed the 0.57.0 release gate as flaky: the
re-floated player's x was 285 while the dragged frame was 285.15625, so
the exact float comparison never settled. Compare whole pixels, matching
the drag assertion earlier in the same test.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SJgWwVbexyFmdCVEcQPw7v
@sambitcreate sambitcreate self-assigned this Oct 16, 2026
@very-hermes-bot

very-hermes-bot commented Oct 16, 2026 •

Copy link
Copy Markdown
Collaborator

Hermes review failed

Hermes could not produce a code-review verdict for this head SHA.

Engine: agy/gemini-3.8-flash-high
Review mode: full
Head: c6dc0535fb7ab06db05d73f33222bb2dd7c71a22
Generated: 2026-10-16T10:19:54+00:00
Failure phase: engine

Safe summary:

agy failed (3)
STDOUT:
{"event":"init","conversation_id":"c1d9d190-ec48-4aa1-9090-8eed2e41ee5f","init":{"model":"gemini-3.8-flash-high","cwd":"/tmp/hermes-review-bot/aiden-agent-pr-508-c6dc0535fb7a","tools":["ask_custom_permission","ask_permission","ask_question","browser_click_element","browser_drag_pixel_to_pixel","browser_get_dom","browser_get_network_request","browser_input","browser_list_network_requests","browser_mouse_down","browser_mouse_up","browser_move_mouse","browser_press_key","browser_refresh_page","browser_resize_window","browser_scroll","browser_scroll_dom","browser_select_option","browser_subagent","call_mcp_tool","capture_browser_console_logs","capture_browser_screenshot","click_browser_pixel","command_status","define_subagent","delete_knowledge","execute_browser_javascript","find_by_name","finish","generate_image","grep_search","invoke_subagent","list_browser_pages","list_dir","list_permissions","list_plugin_accounts","list_resources","manage_inbox","manage_subagents","manage_task","multi_replace_file_content","notebook_edit","notebook_execution","open_browser_url","read_browser_page","read_resource","read_url_content","replace_file_content","run_command","run_workflow","schedule","search_marketplace","search_web","sed_file","send_command_input","send_message","view_file","wait","wait_5_seconds","write_to_file"],"permission_mode":"always-proceed"}}
{"event":"

Local logs: /home/sambit1/.hermes/logs/review-bot/aiden-agent.log

Copy link
Copy Markdown
Owner Author

The Hermes Review Bot status failed inside its own engine (agy failed (3), failure phase engine) before producing any verdict, the same failure it hit on #507. It is not caused by this one-line test change, there is no fix to port from this PR, and I can't re-run that external status from here. #507 merged with this status red, so it does not block merging.


Generated by Claude Code

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes in the Simulator stream E2E, including the surrounding drag/dock/re-float flow and production geometry persistence.

  • Whole-pixel comparison: The restored player’s x-coordinate is compared at whole-pixel precision, preserving the position-restoration check without requiring identical fractional layout coordinates. Retries, timeouts, and --fail-on-flaky-tests are unchanged.
  • Validation: E2E type-checking, ESLint on the changed spec, and all seven floating-player geometry tests passed. The macOS-only Electron spec could not run on this Linux runner; its head-commit CI results remain the acceptance check.

Pullfrog  | View workflow run | Using gpt-6.1-sol | 𝕏

@sambitcreate
sambitcreate merged commit fa72a89 into main Oct 16, 2026
25 of 26 checks passed
@sambitcreate
sambitcreate deleted the claude/project-thread-ur6whr branch October 16, 2026 15:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants