refactor(actions): extract _ref_result_coordinates for ref-lookup coordinate resolution

[dhruvbatra]

The structural issue

actions.py has exactly two places that turn a Navigator element ref into concrete viewport coordinates by running the SDK's GET_ELEMENT_BY_REF_SCRIPT, and each hand-rolled the identical decode sequence over that script's result payload — check success, pull coordinates, validate the shape with _is_coordinate_pair, then round(float(...)) both components:

• ActionExecutor._resolve_coordinates — the click / hover / mouse_down / mouse_up / scroll targeting path:

  if result.get("success"):
      resolved_coordinates = result.get("coordinates")
      if _is_coordinate_pair(resolved_coordinates):
          return round(float(resolved_coordinates[0])), round(float(resolved_coordinates[1]))

• ActionExecutor._execute_expanded_tool, set_element_value paste-preview branch:

  coords = ref_info.get("coordinates") if ref_info.get("success") else None
  if _is_coordinate_pair(coords):
      await self._best_effort_overlay_preview_action(
          action_type="set_element_value",
          x=round(float(coords[0])),
          y=round(float(coords[1])),
      )

Same script, same payload contract, same three-step decode, expressed two different ways ~70 lines apart.

Why the refactor is an improvement

Extracts a module-level _ref_result_coordinates(ref_result) -> tuple[int, int] | None next to the existing _is_coordinate_pair helper it builds on. Both call sites now just ask for the coordinates and check for None, then fall through to their own (genuinely different) fallback: raw coordinates / a BrowserActionError for the targeting path, silently skipping the overlay preview for the paste path.

That puts the "what a successful ref lookup looks like, and how its floats become viewport ints" rule in one place. Today the two spellings agree only by coincidence — the success gate, the pair validation, and the rounding are each an independent chance to drift (e.g. one site gaining a truncate-vs-round or a missing-success tolerance the other doesn't). It also makes the payload-shaped decode independently readable and named, rather than inlined inside a 60-line dispatch branch.

Why it is safe

• Equivalent by construction. The helper returns None in exactly the two cases the inline code fell through on — success falsy, or coordinates failing _is_coordinate_pair — and applies the same round(float(...)) conversion on the happy path. In _resolve_coordinates the "success but malformed coordinates" case still falls through to the ref-resolution-failed / coordinate-fallback branch, as before. Neither call site's surrounding try/except scope changed, so an evaluate failure or a malformed payload is still caught where it was.
• Test suite unchanged before/after: uv run pytest tests/ → 322 passed / 21 failed, byte-identical to the pre-change baseline on this branch point (the 21 are the repo's pre-existing environment-only Chromium-sandbox failures in test_browser.py / test_live_runner.py). tests/test_actions.py alone: 40/40.
• uv run ruff check and ruff format --check clean on the touched file; python -m py_compile clean.
• Note on type checking: this repo has no mypy/pyright configuration and no type-check CI job, so there was no configured type checker to run. The one new annotation (dict[str, Any] -> tuple[int, int] | None) matches what both call sites already do with the value.
• 1 file touched, +27/−8, no public API or behavior change.

---

Generated by Claude Code

---

Note

Low Risk
Internal refactor of coordinate decoding with no intended behavior or API change.

Overview
Deduplicates how GET_ELEMENT_BY_REF_SCRIPT results become viewport ints.

Adds _ref_result_coordinates, which checks success, validates the pair with _is_coordinate_pair, and rounds to ints (or returns None). _resolve_coordinates and the set_element_value overlay preview now share that helper instead of inlining the same decode. Behavior and fallbacks are unchanged.

^{Reviewed by Cursor Bugbot for commit 25bd779. Bugbot is set up for automated code reviews on this repo. Configure here.}

Summary by CodeRabbit

• Bug Fixes
  • Improved coordinate handling for visual interactions.
  • Standardized validation and rounding of successful reference results.
  • Improved consistency of overlay previews and element coordinate resolution.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: f841740c-00a4-4f66-a508-1406172ed29f

📥 Commits

Reviewing files that changed from the base of the PR and between 2a36b75 and 25bd779.

📒 Files selected for processing (1)

• src/frontend_visualqa/actions.py

Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review.

---

📝 Walkthrough

Walkthrough

The change adds a shared helper for validating and rounding coordinates from successful reference lookups. The overlay preview and ref-based action resolution now use this helper.

Changes

Coordinate validation

Layer / File(s)	Summary
Shared coordinate helper and action integration src/frontend_visualqa/actions.py	The new private helper returns rounded coordinates for valid finite reference results. The overlay preview and ref-based action resolution use the helper. Coordinate fallback and action errors remain unchanged.

Estimated code review effort: 2 (Simple) | ~5 minutes

Merge Risk: ⚪ Minimal · up to 25bd7

This change centralizes existing coordinate decoding without an indicated behavior or API change. No actionable merge-blocking risk remains beyond normal checks and review.

Suggested reviewers: juanpin

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes extracting the shared helper for reference-lookup coordinate resolution.
Docstring Coverage	✅ Passed	Docstring check was indeterminate for this PR — some files could not be analyzed in time. Not blocking.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch claude/admiring-hawking-ojmafy

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}