Skip to content

Real task completion: auto-fill fields instead of only narrating them #9

Description

@NITISH-R-G

Objective

For steps where the user has already stated the value in their spoken utterance (e.g. "pay 2400 rupees" while on the amount step), fill the field automatically via the accessibility action API instead of only pointing at it and asking the user to type it themselves.

Rationale

The repo audit's central product criticism: today the app narrates, it does not automate. The user still types the amount and the account number themselves after being pointed at the field - the accessibility permission's risk (reading the full foreground screen) is not currently repaid by a proportional benefit. Auto-fill, where the value is unambiguous, is the single change most likely to make the core value proposition defensible against "this is TalkBack with extra steps."

Acceptance criteria

  • When the planner or a deterministic parser can extract a clear value from the transcript for a step marked expects_value: true, the value is set on the target EditText via AccessibilityNodeInfo.performAction(ACTION_SET_TEXT, ...) and the user is told what was filled in (not just what to do)
  • If no value can be confidently extracted, current behavior (point + ask the user to type) is unchanged - this is additive, not a replacement
  • Never auto-submits a form - filling is assistive, submission is still a deliberate user tap, to avoid an accidental real-world transaction (this matters most on book_taxi, which touches real ride apps)
  • Works for the demo pay_bill task; explicitly out of scope for third-party apps until # is resolved, since a wrong auto-fill in a real ride app is a worse failure mode than a wrong auto-fill in a demo screen

Implementation steps

  1. Extend the planner's tool schema with an optional extracted_value field (additive - the frozen contract permits new optional fields, never removals)
  2. Add a deterministic fallback extractor (regex over digits/currency words) for the keyless path, since auto-fill should work without a Sarvam key too, consistent with every other feature in this app
  3. Implement the accessibility ACTION_SET_TEXT call in ScreenReaderService or a new narrowly-scoped helper - do not expand ScreenReaderService's read-only contract without discussing it first, since "no gestures" is a stated design principle
  4. Update the spoken instruction to reflect what was actually done ("I've entered 2400 rupees" vs "Enter the amount here")
  5. Add unit tests for the value extractor and manual device verification that the real field updates

Estimated effort

High (this is a genuine scope expansion of what the app does, not a bug fix - budget a week including the discussion in step 3 about whether ScreenReaderService's "no gestures" principle needs to become "read-only except for explicit, user-narrated value entry")

Risks

  • This is the one change in this list that most looks like "adding a sixth capability" to a frozen five-piece architecture. It should go through the discussion CONTRIBUTING.md asks for before code is written, not be merged as a fait accompli
  • Auto-filling the wrong value into a real financial form (even the demo one) is a much worse failure than a wrong highlight - this needs a higher confidence bar than the planner's existing 0.5 floor
  • Third-party apps (Uber/Ola/Rapido) should very deliberately NOT be in scope for v1 of this - see the separate RideApps matching issue

Testing checklist

  • Unit tests: value extraction from "pay 2400 rupees", "the amount is two thousand four hundred", ambiguous phrasing that should NOT auto-fill
  • Manual device test: spoken amount auto-fills the demo screen's field correctly
  • Manual device test: ambiguous/no value falls back to point-and-ask exactly as today
  • Confirm no auto-submit occurs under any code path

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    audit-followupFlagged by the repo audit; too large/risky for a small PRenhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions