Problem
Claude in Chrome handled the Musinsa sort dropdown by using a find/read-page style primitive: locate visible text, click the resolved target, then verify by screenshot/URL. GAIA currently prefers OpenClaw role-tree refs, which is good for benchmark evidence, but custom dropdowns can visually update while refs/role snapshots are stale.
Goal
Add a ref-first visual/text find fallback that keeps benchmark evidence intact while improving recovery for visible-but-missing targets.
Desired behavior
- Keep ref_id based actions as the primary path.
- When a WAIT/action failure indicates the desired visible text/option exists but no fresh ref is available, run a find-style resolver.
- Prefer resolving visible text back into an OpenClaw ref from a forced snapshot.
- Only if no ref can be promoted, allow a constrained coordinate fallback for read-only, non-destructive targets.
- Re-verify with URL/UI/snapshot evidence after fallback.
Safety constraints
- Do not use coordinate fallback for login, purchase, cart, submit, delete, write, account, reservation, or CAPTCHA/security gates.
- Keep reason codes and benchmark artifacts explicit so failures remain explainable.
Reference case
Musinsa search sort dropdown: selecting 낮은 가격순 from https://www.musinsa.com/search/goods?keyword=%EA%B0%80%EB%B0%A9
Acceptance
- Unit tests cover ref promotion, unsafe fallback blocking, and coordinate fallback gating.
- Musinsa strict probe remains successful.
- Existing goal-driven and OpenClaw unit suites pass.
Problem
Claude in Chrome handled the Musinsa sort dropdown by using a find/read-page style primitive: locate visible text, click the resolved target, then verify by screenshot/URL. GAIA currently prefers OpenClaw role-tree refs, which is good for benchmark evidence, but custom dropdowns can visually update while refs/role snapshots are stale.
Goal
Add a ref-first visual/text find fallback that keeps benchmark evidence intact while improving recovery for visible-but-missing targets.
Desired behavior
Safety constraints
Reference case
Musinsa search sort dropdown: selecting 낮은 가격순 from https://www.musinsa.com/search/goods?keyword=%EA%B0%80%EB%B0%A9
Acceptance