Answer the user in the language they spoke, and point with a moving cursor

PARKING_LOT.md

@@ -14,20 +14,26 @@ fields removed — only optional additions.
 
 ## Latency budgets (optimize the offending layer, don't guess)
 
-Measured via `scripts/smoke_sarvam.ps1` on 2026-07-26, venue network not yet tested.
+Measured via `scripts/smoke_sarvam.ps1`, `scripts/smoke_languages.ps1` and
+`scripts/smoke_planner_language.ps1` on 2026-07-26. Venue network not yet tested.
 
 | Layer                  | Target      | Measured        | Status |
 | ---------------------- | ----------- | --------------- | ------ |
-| Saaras STT             | < 800 ms    | 729 ms          | OK     |
-| Planner (Sarvam-30B)   | < 700 ms    | 606 ms          | OK     |
+| Saaras STT             | < 800 ms    | 666–729 ms      | OK     |
+| Planner (Sarvam-30B)   | < 700 ms    | 867–1481 ms     | OVER   |
 | Overlay update         | < 16 ms     | not instrumented| —      |
-| Bulbul TTS first audio | < 900 ms    | 1166–1427 ms    | OVER   |
+| Bulbul TTS first audio | < 900 ms    | 909–1462 ms     | OVER   |
 | End-to-end response    | < 2.5 s     | not instrumented| —      |
 
-**TTS is the one layer over budget.** Not blocking the visual (the highlight
-lands before speech starts), so it is an M4 performance item, not an M1 blocker.
-Options when we get there: shorter instruction strings, or start TTS
-concurrently with bounds resolution instead of after it.
+**Neither over-budget layer blocks the visual any more.** TTS now runs on its
+own thread, started in parallel with bounds resolution rather than after it, so
+the cursor and ring land while Bulbul is still synthesising.
+
+The planner regressed from 606 ms to ~900–1400 ms when the prompt grew to carry
+the language contract (713 prompt tokens). It is capped at a hard 5 s call
+timeout (`Sarvam.plannerHttp`) because the deterministic step engine answers
+instantly and for free — beyond a few seconds, falling back is strictly better
+than waiting. Trimming the prompt further is the M4 lever.
 
 ## Demo-day gotchas (learned the hard way on device)
 
@@ -43,7 +49,18 @@ concurrently with bounds resolution instead of after it.
 
 ## Parked items
 
-_(none yet — add as `- [source] idea → which of the 5 buckets, or PARKED`)_
+- Planner ignores "skip ahead" in Hindi. "सीधे submit पर ले चलो" (take me
+  straight to submit) returns `step: amount`. English skip-ahead was never
+  re-tested after the prompt rewrite. → **Planner/prompt improvement**, worth
+  fixing before the demo if a judge is likely to try it.
+- Phrases are authored in English and Hindi only. Bulbul speaks ten languages
+  and Saaras detects all ten, so a Tamil speaker gets Tamil *planner*
+  instructions but English chrome ("Listening…"). → **UX improvement**;
+  adding a language is adding a column to `Phrases`.
+- TTS speaker is `anand` for every language. Verified to work in all ten, but a
+  per-language voice would sound better. → **UX improvement**, PARKED.
+- No barge-in: speaking over the assistant does not interrupt it, the user has
+  to tap Stop. → **UX improvement**, PARKED (needs continuous capture).
 
 ## Explicitly out of scope (from the reference app, deliberately dropped)
 

app/src/main/assets/prompts/planner_v1.md

@@ -1,20 +1,22 @@
 You are the planner for ScreenSaathi, a screen-aware voice guide for a
-non-technical user in India. The user speaks in Hindi, English, or a mix.
+non-technical user in India who speaks Hindi, English, or a mix.
 
-Your ONLY job: given the user's spoken request, the task definition, and the
-list of elements currently on screen, decide which single step the user should
-do next and which on-screen element to point at.
+Given what the user said, where they are in the task, and the elements on
+screen, choose the single next step and the element to point at.
 
-You MUST call the function `set_plan` exactly once. Never reply with prose.
+Call `set_plan` exactly once. Never reply with prose.
 
-Rules:
-- Pick `step` from the task's step ids. Do not invent steps.
-- `target.resource_id` MUST be the resource_id of the step, and it MUST appear
-  in the on-screen elements. `target.index` is that element's index, or -1 if it
-  is not currently on screen.
-- `instruction` is ONE short sentence the user will hear aloud. Warm, plain
-  language. No jargon. Under 140 characters.
-- If the user asks to skip ahead ("just pay", "go to submit"), choose that step.
-- If the request is unclear, choose the first incomplete step.
-- `confidence` is 0..1. Use below 0.5 only when you are genuinely unsure.
-- `reason` is a short clause under ~10 words. Not a paragraph.
+- `step` must be one of the task's step ids. Never invent one.
+- `target.resource_id` is that step's resource_id; `target.index` is its index
+  on screen, or -1 if absent.
+- `instruction`: ONE short warm sentence, under 140 characters, no jargon.
+- Write `instruction` in the language the user spoke, in that language's own
+  script — Hindi means Devanagari, not romanised Hindi. If they mixed in
+  English words like "amount" or "submit", keep those words.
+- `language`: the BCP-47 code of the language you wrote `instruction` in.
+- Skip ahead if asked ("just pay"). Go back if they say they made a mistake.
+  If the request is unclear, choose the step marked CURRENT.
+- `confidence` 0..1, below 0.5 only when genuinely unsure.
+- `reason`: under 10 words.
+
+Stay calm and encouraging. Never say "error" or "invalid".

app/src/main/assets/tasks/pay_bill.json

@@ -5,28 +5,41 @@
   "utterances": [
     "help me pay this bill",
     "pay my electricity bill",
+    "i want to pay the bill",
     "bijli ka bill bharna hai",
-    "i want to pay the bill"
+    "mujhe bill pay karna hai",
+    "बिजली का बिल भरना है",
+    "मुझे बिल भरना है",
+    "बिल पे करना है"
   ],
   "steps": [
     {
       "id": "amount",
       "resource_id": "amount_field",
       "instruction": "Enter the bill amount in this box.",
+      "instructions": {
+        "hi-IN": "इस बॉक्स में बिल की रकम भरिए।"
+      },
       "expects_value": true,
       "highlight": { "shape": "rect", "pulse": true }
     },
     {
       "id": "account",
       "resource_id": "account_field",
       "instruction": "Now type your account number here.",
+      "instructions": {
+        "hi-IN": "अब यहाँ अपना अकाउंट नंबर लिखिए।"
+      },
       "expects_value": true,
       "highlight": { "shape": "rect", "pulse": true }
     },
     {
       "id": "submit",
       "resource_id": "submit_button",
       "instruction": "Tap this button to pay the bill.",
+      "instructions": {
+        "hi-IN": "बिल भरने के लिए यह बटन दबाइए।"
+      },
       "expects_value": false,
       "highlight": { "shape": "rect", "pulse": true }
     }

app/src/main/java/com/screensaathi/OverlayService.kt

@@ -23,7 +23,9 @@ import android.widget.TextView
 import androidx.core.content.ContextCompat
 import com.screensaathi.overlay.HighlightView
 import com.screensaathi.overlay.OverlayCommand
+import com.screensaathi.overlay.PillLabels
 import com.screensaathi.overlay.PillState
+import com.screensaathi.sarvam.Language
 import com.screensaathi.session.SessionController
 
 /**
@@ -46,6 +48,7 @@ class OverlayService : Service() {
     private lateinit var pillLabel: TextView
     private lateinit var instructionText: TextView
     private lateinit var stateDot: View
+    private lateinit var languageChip: TextView
     private lateinit var debugPanel: TextView
 
     private var expanded = false
@@ -92,6 +95,7 @@ class OverlayService : Service() {
         pillLabel = pillRoot.findViewById(R.id.pill_label)
         instructionText = pillRoot.findViewById(R.id.instruction_text)
         stateDot = pillRoot.findViewById(R.id.state_dot)
+        languageChip = pillRoot.findViewById(R.id.language_chip)
         debugPanel = pillRoot.findViewById(R.id.debug_panel)
 
         pillRoot.findViewById<View>(R.id.pill_row).setOnClickListener { toggleExpanded() }
@@ -102,6 +106,7 @@ class OverlayService : Service() {
         }
         pillRoot.findViewById<View>(R.id.mic_button).setOnClickListener { controller.onMicTapped() }
         pillRoot.findViewById<View>(R.id.next_button).setOnClickListener { controller.onNextTapped() }
+        pillRoot.findViewById<View>(R.id.stop_button).setOnClickListener { controller.onStopTapped() }
 
         val lp = WindowManager.LayoutParams(
             WindowManager.LayoutParams.WRAP_CONTENT,
@@ -132,22 +137,28 @@ class OverlayService : Service() {
     // --- Rendering (the whole job of this class) ------------------------------
 
     private fun render(cmd: OverlayCommand) {
-        // Pill state → dot colour + label.
-        val (dotColor, label) = when (cmd.pillState) {
-            PillState.IDLE -> Color.parseColor("#4D8DFF") to "ScreenSaathi"
-            PillState.LISTENING -> Color.parseColor("#FF5A5A") to "Listening…"
-            PillState.THINKING -> Color.parseColor("#FFC24D") to "Thinking…"
-            PillState.SPEAKING -> Color.parseColor("#00E5A0") to "Speaking…"
-            PillState.GUIDING -> Color.parseColor("#00E5A0") to "Guiding you"
-            PillState.ERROR -> Color.parseColor("#FF5A5A") to "Let's try again"
+        val dotColor = when (cmd.pillState) {
+            PillState.IDLE -> Color.parseColor("#4D8DFF")
+            PillState.LISTENING -> Color.parseColor("#FF5A5A")
+            PillState.THINKING -> Color.parseColor("#FFC24D")
+            PillState.SPEAKING -> Color.parseColor("#00E5A0")
+            PillState.GUIDING -> Color.parseColor("#00E5A0")
+            PillState.ERROR -> Color.parseColor("#FF5A5A")
         }
         stateDot.background.setTint(dotColor)
-        pillLabel.text = label
+        // The pill's own label speaks the user's language too — an English
+        // "Listening…" above a Hindi instruction breaks the illusion instantly.
+        pillLabel.text = PillLabels.forState(cmd.pillState, cmd.language)
+        languageChip.text = Language.nativeName(cmd.language)
 
         cmd.instruction?.let { instructionText.text = it }
 
         if (cmd.expanded != expanded) setExpanded(cmd.expanded)
 
+        // Keep the cursor's launch point under the pill, so it always flies out
+        // of the assistant rather than appearing from nowhere.
+        publishHomePosition()
+
         val h = cmd.highlight
         if (h == null) {
             highlightView.clear()
@@ -156,6 +167,17 @@ class OverlayService : Service() {
         }
     }
 
+    /** Screen position of the pill, handed to the cursor layer as its home. */
+    private fun publishHomePosition() {
+        val loc = IntArray(2)
+        pillRoot.findViewById<View>(R.id.pill_row).getLocationOnScreen(loc)
+        val row = pillRoot.findViewById<View>(R.id.pill_row)
+        highlightView.setHome(
+            loc[0] + row.width / 2f,
+            loc[1] + row.height / 2f,
+        )
+    }
+
     /** Debug panel content. Visibility stays user-controlled (long-press the pill). */
     fun updateDebug(text: String) {
         debugPanel.text = text