Skip to content

Add ref-first visual/text find fallback for stale browser targets #117

Description

@coldmans

Problem

Claude in Chrome handled the Musinsa sort dropdown by using a find/read-page style primitive: locate visible text, click the resolved target, then verify by screenshot/URL. GAIA currently prefers OpenClaw role-tree refs, which is good for benchmark evidence, but custom dropdowns can visually update while refs/role snapshots are stale.

Goal

Add a ref-first visual/text find fallback that keeps benchmark evidence intact while improving recovery for visible-but-missing targets.

Desired behavior

  • Keep ref_id based actions as the primary path.
  • When a WAIT/action failure indicates the desired visible text/option exists but no fresh ref is available, run a find-style resolver.
  • Prefer resolving visible text back into an OpenClaw ref from a forced snapshot.
  • Only if no ref can be promoted, allow a constrained coordinate fallback for read-only, non-destructive targets.
  • Re-verify with URL/UI/snapshot evidence after fallback.

Safety constraints

  • Do not use coordinate fallback for login, purchase, cart, submit, delete, write, account, reservation, or CAPTCHA/security gates.
  • Keep reason codes and benchmark artifacts explicit so failures remain explainable.

Reference case

Musinsa search sort dropdown: selecting 낮은 가격순 from https://www.musinsa.com/search/goods?keyword=%EA%B0%80%EB%B0%A9

Acceptance

  • Unit tests cover ref promotion, unsafe fallback blocking, and coordinate fallback gating.
  • Musinsa strict probe remains successful.
  • Existing goal-driven and OpenClaw unit suites pass.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions