From c7950d7502a8dc34747621d382075f4f326980e3 Mon Sep 17 00:00:00 2001 From: NITISH-R-G Date: Sun, 26 Jul 2026 14:49:47 +0530 Subject: [PATCH] Answer the user in the language they spoke, and point with a moving cursor MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two things the demo needed: it has to reply in the user's own language, and the pill has to visibly guide rather than sit still. ## Multilingual Language detection already reached TTS, but the TEXT stayed English. A Hindi speaker heard English words requested as hi-IN, which Bulbul rejects outright: 400 "Text must contain at least one character from the allowed languages." A rejected synthesis is indistinguishable from having nothing to say, so the app just went quiet. Fixes, in the order the words travel: - Spoken(text, language) pairs every string with the language it is actually written in, so the mismatch is now unrepresentable. - Language.reconcile() trusts the script over the claim before anything is sent to Bulbul. The rule is presence, not majority, matching Bulbul's own: if the claimed language's script appears at all, keep it. That is what makes code-switching work - "amount यहाँ भरिए" stays hi-IN and the English word is read naturally. A majority count gets this wrong, because Devanagari vowel signs are combining marks rather than letters, so it scores 6 Latin to 5 Devanagari and calls it English. - The planner now receives the detected language and the CURRENT step, and returns instruction in that language plus a `language` code. The prompt rule "if unclear, stay on the current step" was previously unfollowable - the model was never told which step that was. - The task DSL carries per-language wording, so the offline fallback is multilingual too, and Phrases/PillLabels move the assistant's own words and the pill's labels off English literals. - TaskRepository.normalize() stripped with [^a-z0-9 ], which deletes every Devanagari character. Saaras returns Hindi speech AS Devanagari, so utterance matching scored zero on the primary demo language and silently matched nothing. It now strips by character category. Verified live against Sarvam, not assumed: - scripts/smoke_languages.ps1 - all ten languages synthesise with speaker `anand` and Saaras detects each one back correctly. - scripts/planner_case.ps1 - the real prompt and tool spec return Devanagari for Hindi, Tamil for Tamil, Bengali for Bengali, each correctly labelled. ## Moving cursor HighlightView now flies a cursor along an arc to the target, trailing a comet tail and a tether back to the pill, and the ring only blooms once it arrives so the two read as cause and effect. The ring alone teleported between fields, which read as a highlight appearing rather than as something guiding you. ## Stop onStopTapped() silences speech, invalidates the in-flight turn, flies the cursor home and clears the ring - while keeping the step position, so the next mic tap resumes instead of restarting. ## Also Speech moved to its own thread and now starts in parallel with bounds resolution, so the visual never queues behind Bulbul's ~1.2s. The planner got a hard 5s call timeout: the step engine answers instantly and for free, so waiting longer than that is strictly worse than falling back. Tested: assembleDebug and testDebugUnitTest green (56 tests, 39 new); installed and exercised on a real device - cursor flight, ring, tether, Stop button and the language chip all render, no crashes. Co-Authored-By: Claude Opus 5 --- PARKING_LOT.md | 35 ++- app/src/main/assets/prompts/planner_v1.md | 34 +-- app/src/main/assets/tasks/pay_bill.json | 15 +- .../java/com/screensaathi/OverlayService.kt | 40 ++- .../com/screensaathi/overlay/HighlightView.kt | 269 +++++++++++++++--- .../screensaathi/overlay/OverlayCommand.kt | 7 + .../com/screensaathi/overlay/PillLabels.kt | 41 +++ .../java/com/screensaathi/sarvam/Language.kt | 152 ++++++++++ .../com/screensaathi/sarvam/PlannerResult.kt | 12 +- .../java/com/screensaathi/sarvam/Sarvam.kt | 19 ++ .../com/screensaathi/sarvam/SarvamPlanner.kt | 138 ++++++--- .../java/com/screensaathi/sarvam/SarvamStt.kt | 13 +- .../java/com/screensaathi/sarvam/SarvamTts.kt | 25 +- .../java/com/screensaathi/session/Phrases.kt | 78 +++++ .../screensaathi/session/SessionController.kt | 150 +++++++--- .../com/screensaathi/session/VoiceDebug.kt | 5 + .../java/com/screensaathi/task/TaskModels.kt | 23 +- .../com/screensaathi/task/TaskRepository.kt | 45 ++- app/src/main/res/layout/overlay_pill.xml | 27 ++ .../com/screensaathi/sarvam/LanguageTest.kt | 141 +++++++++ .../sarvam/SarvamPlannerParseTest.kt | 135 +++++++++ .../com/screensaathi/session/PhrasesTest.kt | 84 ++++++ .../screensaathi/task/MultilingualTaskTest.kt | 101 +++++++ contracts/planner.schema.json | 8 +- scripts/planner_case.ps1 | 68 +++++ scripts/smoke_languages.ps1 | 78 +++++ 26 files changed, 1566 insertions(+), 177 deletions(-) create mode 100644 app/src/main/java/com/screensaathi/overlay/PillLabels.kt create mode 100644 app/src/main/java/com/screensaathi/sarvam/Language.kt create mode 100644 app/src/main/java/com/screensaathi/session/Phrases.kt create mode 100644 app/src/test/java/com/screensaathi/sarvam/LanguageTest.kt create mode 100644 app/src/test/java/com/screensaathi/sarvam/SarvamPlannerParseTest.kt create mode 100644 app/src/test/java/com/screensaathi/session/PhrasesTest.kt create mode 100644 app/src/test/java/com/screensaathi/task/MultilingualTaskTest.kt create mode 100644 scripts/planner_case.ps1 create mode 100644 scripts/smoke_languages.ps1 diff --git a/PARKING_LOT.md b/PARKING_LOT.md index b96d502..3797513 100644 --- a/PARKING_LOT.md +++ b/PARKING_LOT.md @@ -14,20 +14,26 @@ fields removed — only optional additions. ## Latency budgets (optimize the offending layer, don't guess) -Measured via `scripts/smoke_sarvam.ps1` on 2026-07-26, venue network not yet tested. +Measured via `scripts/smoke_sarvam.ps1`, `scripts/smoke_languages.ps1` and +`scripts/smoke_planner_language.ps1` on 2026-07-26. Venue network not yet tested. | Layer | Target | Measured | Status | | ---------------------- | ----------- | --------------- | ------ | -| Saaras STT | < 800 ms | 729 ms | OK | -| Planner (Sarvam-30B) | < 700 ms | 606 ms | OK | +| Saaras STT | < 800 ms | 666–729 ms | OK | +| Planner (Sarvam-30B) | < 700 ms | 867–1481 ms | OVER | | Overlay update | < 16 ms | not instrumented| — | -| Bulbul TTS first audio | < 900 ms | 1166–1427 ms | OVER | +| Bulbul TTS first audio | < 900 ms | 909–1462 ms | OVER | | End-to-end response | < 2.5 s | not instrumented| — | -**TTS is the one layer over budget.** Not blocking the visual (the highlight -lands before speech starts), so it is an M4 performance item, not an M1 blocker. -Options when we get there: shorter instruction strings, or start TTS -concurrently with bounds resolution instead of after it. +**Neither over-budget layer blocks the visual any more.** TTS now runs on its +own thread, started in parallel with bounds resolution rather than after it, so +the cursor and ring land while Bulbul is still synthesising. + +The planner regressed from 606 ms to ~900–1400 ms when the prompt grew to carry +the language contract (713 prompt tokens). It is capped at a hard 5 s call +timeout (`Sarvam.plannerHttp`) because the deterministic step engine answers +instantly and for free — beyond a few seconds, falling back is strictly better +than waiting. Trimming the prompt further is the M4 lever. ## Demo-day gotchas (learned the hard way on device) @@ -43,7 +49,18 @@ concurrently with bounds resolution instead of after it. ## Parked items -_(none yet — add as `- [source] idea → which of the 5 buckets, or PARKED`)_ +- Planner ignores "skip ahead" in Hindi. "सीधे submit पर ले चलो" (take me + straight to submit) returns `step: amount`. English skip-ahead was never + re-tested after the prompt rewrite. → **Planner/prompt improvement**, worth + fixing before the demo if a judge is likely to try it. +- Phrases are authored in English and Hindi only. Bulbul speaks ten languages + and Saaras detects all ten, so a Tamil speaker gets Tamil *planner* + instructions but English chrome ("Listening…"). → **UX improvement**; + adding a language is adding a column to `Phrases`. +- TTS speaker is `anand` for every language. Verified to work in all ten, but a + per-language voice would sound better. → **UX improvement**, PARKED. +- No barge-in: speaking over the assistant does not interrupt it, the user has + to tap Stop. → **UX improvement**, PARKED (needs continuous capture). ## Explicitly out of scope (from the reference app, deliberately dropped) diff --git a/app/src/main/assets/prompts/planner_v1.md b/app/src/main/assets/prompts/planner_v1.md index 3888d60..df2a9af 100644 --- a/app/src/main/assets/prompts/planner_v1.md +++ b/app/src/main/assets/prompts/planner_v1.md @@ -1,20 +1,22 @@ You are the planner for ScreenSaathi, a screen-aware voice guide for a -non-technical user in India. The user speaks in Hindi, English, or a mix. +non-technical user in India who speaks Hindi, English, or a mix. -Your ONLY job: given the user's spoken request, the task definition, and the -list of elements currently on screen, decide which single step the user should -do next and which on-screen element to point at. +Given what the user said, where they are in the task, and the elements on +screen, choose the single next step and the element to point at. -You MUST call the function `set_plan` exactly once. Never reply with prose. +Call `set_plan` exactly once. Never reply with prose. -Rules: -- Pick `step` from the task's step ids. Do not invent steps. -- `target.resource_id` MUST be the resource_id of the step, and it MUST appear - in the on-screen elements. `target.index` is that element's index, or -1 if it - is not currently on screen. -- `instruction` is ONE short sentence the user will hear aloud. Warm, plain - language. No jargon. Under 140 characters. -- If the user asks to skip ahead ("just pay", "go to submit"), choose that step. -- If the request is unclear, choose the first incomplete step. -- `confidence` is 0..1. Use below 0.5 only when you are genuinely unsure. -- `reason` is a short clause under ~10 words. Not a paragraph. +- `step` must be one of the task's step ids. Never invent one. +- `target.resource_id` is that step's resource_id; `target.index` is its index + on screen, or -1 if absent. +- `instruction`: ONE short warm sentence, under 140 characters, no jargon. +- Write `instruction` in the language the user spoke, in that language's own + script — Hindi means Devanagari, not romanised Hindi. If they mixed in + English words like "amount" or "submit", keep those words. +- `language`: the BCP-47 code of the language you wrote `instruction` in. +- Skip ahead if asked ("just pay"). Go back if they say they made a mistake. + If the request is unclear, choose the step marked CURRENT. +- `confidence` 0..1, below 0.5 only when genuinely unsure. +- `reason`: under 10 words. + +Stay calm and encouraging. Never say "error" or "invalid". diff --git a/app/src/main/assets/tasks/pay_bill.json b/app/src/main/assets/tasks/pay_bill.json index 1c31cac..cd8c7ff 100644 --- a/app/src/main/assets/tasks/pay_bill.json +++ b/app/src/main/assets/tasks/pay_bill.json @@ -5,14 +5,21 @@ "utterances": [ "help me pay this bill", "pay my electricity bill", + "i want to pay the bill", "bijli ka bill bharna hai", - "i want to pay the bill" + "mujhe bill pay karna hai", + "बिजली का बिल भरना है", + "मुझे बिल भरना है", + "बिल पे करना है" ], "steps": [ { "id": "amount", "resource_id": "amount_field", "instruction": "Enter the bill amount in this box.", + "instructions": { + "hi-IN": "इस बॉक्स में बिल की रकम भरिए।" + }, "expects_value": true, "highlight": { "shape": "rect", "pulse": true } }, @@ -20,6 +27,9 @@ "id": "account", "resource_id": "account_field", "instruction": "Now type your account number here.", + "instructions": { + "hi-IN": "अब यहाँ अपना अकाउंट नंबर लिखिए।" + }, "expects_value": true, "highlight": { "shape": "rect", "pulse": true } }, @@ -27,6 +37,9 @@ "id": "submit", "resource_id": "submit_button", "instruction": "Tap this button to pay the bill.", + "instructions": { + "hi-IN": "बिल भरने के लिए यह बटन दबाइए।" + }, "expects_value": false, "highlight": { "shape": "rect", "pulse": true } } diff --git a/app/src/main/java/com/screensaathi/OverlayService.kt b/app/src/main/java/com/screensaathi/OverlayService.kt index ca4ac07..55af19a 100644 --- a/app/src/main/java/com/screensaathi/OverlayService.kt +++ b/app/src/main/java/com/screensaathi/OverlayService.kt @@ -23,7 +23,9 @@ import android.widget.TextView import androidx.core.content.ContextCompat import com.screensaathi.overlay.HighlightView import com.screensaathi.overlay.OverlayCommand +import com.screensaathi.overlay.PillLabels import com.screensaathi.overlay.PillState +import com.screensaathi.sarvam.Language import com.screensaathi.session.SessionController /** @@ -46,6 +48,7 @@ class OverlayService : Service() { private lateinit var pillLabel: TextView private lateinit var instructionText: TextView private lateinit var stateDot: View + private lateinit var languageChip: TextView private lateinit var debugPanel: TextView private var expanded = false @@ -92,6 +95,7 @@ class OverlayService : Service() { pillLabel = pillRoot.findViewById(R.id.pill_label) instructionText = pillRoot.findViewById(R.id.instruction_text) stateDot = pillRoot.findViewById(R.id.state_dot) + languageChip = pillRoot.findViewById(R.id.language_chip) debugPanel = pillRoot.findViewById(R.id.debug_panel) pillRoot.findViewById(R.id.pill_row).setOnClickListener { toggleExpanded() } @@ -102,6 +106,7 @@ class OverlayService : Service() { } pillRoot.findViewById(R.id.mic_button).setOnClickListener { controller.onMicTapped() } pillRoot.findViewById(R.id.next_button).setOnClickListener { controller.onNextTapped() } + pillRoot.findViewById(R.id.stop_button).setOnClickListener { controller.onStopTapped() } val lp = WindowManager.LayoutParams( WindowManager.LayoutParams.WRAP_CONTENT, @@ -132,22 +137,28 @@ class OverlayService : Service() { // --- Rendering (the whole job of this class) ------------------------------ private fun render(cmd: OverlayCommand) { - // Pill state → dot colour + label. - val (dotColor, label) = when (cmd.pillState) { - PillState.IDLE -> Color.parseColor("#4D8DFF") to "ScreenSaathi" - PillState.LISTENING -> Color.parseColor("#FF5A5A") to "Listening…" - PillState.THINKING -> Color.parseColor("#FFC24D") to "Thinking…" - PillState.SPEAKING -> Color.parseColor("#00E5A0") to "Speaking…" - PillState.GUIDING -> Color.parseColor("#00E5A0") to "Guiding you" - PillState.ERROR -> Color.parseColor("#FF5A5A") to "Let's try again" + val dotColor = when (cmd.pillState) { + PillState.IDLE -> Color.parseColor("#4D8DFF") + PillState.LISTENING -> Color.parseColor("#FF5A5A") + PillState.THINKING -> Color.parseColor("#FFC24D") + PillState.SPEAKING -> Color.parseColor("#00E5A0") + PillState.GUIDING -> Color.parseColor("#00E5A0") + PillState.ERROR -> Color.parseColor("#FF5A5A") } stateDot.background.setTint(dotColor) - pillLabel.text = label + // The pill's own label speaks the user's language too — an English + // "Listening…" above a Hindi instruction breaks the illusion instantly. + pillLabel.text = PillLabels.forState(cmd.pillState, cmd.language) + languageChip.text = Language.nativeName(cmd.language) cmd.instruction?.let { instructionText.text = it } if (cmd.expanded != expanded) setExpanded(cmd.expanded) + // Keep the cursor's launch point under the pill, so it always flies out + // of the assistant rather than appearing from nowhere. + publishHomePosition() + val h = cmd.highlight if (h == null) { highlightView.clear() @@ -156,6 +167,17 @@ class OverlayService : Service() { } } + /** Screen position of the pill, handed to the cursor layer as its home. */ + private fun publishHomePosition() { + val loc = IntArray(2) + pillRoot.findViewById(R.id.pill_row).getLocationOnScreen(loc) + val row = pillRoot.findViewById(R.id.pill_row) + highlightView.setHome( + loc[0] + row.width / 2f, + loc[1] + row.height / 2f, + ) + } + /** Debug panel content. Visibility stays user-controlled (long-press the pill). */ fun updateDebug(text: String) { debugPanel.text = text diff --git a/app/src/main/java/com/screensaathi/overlay/HighlightView.kt b/app/src/main/java/com/screensaathi/overlay/HighlightView.kt index a6f43c3..8273f23 100644 --- a/app/src/main/java/com/screensaathi/overlay/HighlightView.kt +++ b/app/src/main/java/com/screensaathi/overlay/HighlightView.kt @@ -5,15 +5,28 @@ import android.content.Context import android.graphics.Canvas import android.graphics.Color import android.graphics.Paint +import android.graphics.Path +import android.graphics.PointF import android.graphics.RectF import android.view.View +import android.view.animation.DecelerateInterpolator import android.view.animation.LinearInterpolator +import android.view.animation.PathInterpolator /** - * Full-screen transparent view that draws one pulsing highlight over the target - * element's screen bounds. The pulse runs on its own ValueAnimator and the - * bounds move on a separate one — keeping them independent is what makes the - * pointer read as smooth rather than snapping (the trick from the reference app). + * Full-screen transparent layer that draws two things: + * + * 1. a **cursor** that physically flies to whatever the user should touch next, + * leaving a short motion trail behind it, and + * 2. the **ring** that blooms around that element once the cursor arrives. + * + * The ring alone was static: it teleported between fields, which read as a + * highlight appearing rather than as something guiding you. Travel is what + * makes it feel like a hand pointing — the eye follows the moving thing and + * arrives at the target already looking at it. + * + * Everything is drawn; nothing here touches WindowManager, and the window is + * touch-through, so motion can never intercept a tap meant for the app below. */ class HighlightView(context: Context) : View(context) { @@ -27,17 +40,50 @@ class HighlightView(context: Context) : View(context) { color = Color.parseColor("#5500E5A0") strokeWidth = dp(12f) } + private val cursorCore = Paint(Paint.ANTI_ALIAS_FLAG).apply { + style = Paint.Style.FILL + color = Color.parseColor("#FFFFFFFF") + } + private val cursorHalo = Paint(Paint.ANTI_ALIAS_FLAG).apply { + style = Paint.Style.FILL + color = Color.parseColor("#CC00E5A0") + } + private val trailPaint = Paint(Paint.ANTI_ALIAS_FLAG).apply { + style = Paint.Style.FILL + color = Color.parseColor("#8800E5A0") + } + private val tether = Paint(Paint.ANTI_ALIAS_FLAG).apply { + style = Paint.Style.STROKE + color = Color.parseColor("#3300E5A0") + strokeWidth = dp(1.5f) + } private var shape = "rect" - // Current drawn rect (animated toward target). private val current = RectF() private val from = RectF() private val target = RectF() private var hasTarget = false + /** 0 while the cursor is still travelling, 1 once the ring is fully out. */ + private var ringReveal = 0f + private var pulse = 0f private var pulseEnabled = true + /** Where the cursor is right now, and the arc it is flying along. */ + private val cursor = PointF() + private val flightStart = PointF() + private val flightEnd = PointF() + private val flightControl = PointF() + private var cursorVisible = false + + /** Recent cursor positions, newest last — drawn as a fading comet tail. */ + private val trail = ArrayDeque() + + /** The pill's own position; the cursor launches from and returns to it. */ + private val home = PointF() + private var hasHome = false + private val pulseAnimator = ValueAnimator.ofFloat(0f, 1f).apply { duration = 1100 repeatCount = ValueAnimator.INFINITE @@ -49,15 +95,27 @@ class HighlightView(context: Context) : View(context) { } } - private var moveAnimator: ValueAnimator? = null + private var flightAnimator: ValueAnimator? = null init { - // Transparent overlay; never intercepts touches (handled by WindowManager flags). setWillNotDraw(false) } private val originOnScreen = IntArray(2) + /** + * Tell the layer where the pill sits, in screen pixels, so the cursor can + * launch from it rather than materialising out of nowhere. + */ + fun setHome(screenX: Float, screenY: Float) { + getLocationOnScreen(originOnScreen) + home.set(screenX - originOnScreen[0], screenY - originOnScreen[1]) + if (!hasHome) { + cursor.set(home) + hasHome = true + } + } + fun show(l: Int, t: Int, r: Int, b: Int, shape: String, pulse: Boolean) { this.shape = shape this.pulseEnabled = pulse @@ -69,65 +127,192 @@ class HighlightView(context: Context) : View(context) { val dy = originOnScreen[1].toFloat() target.set(l - dx, t - dy, r - dx, b - dy) + // Park the cursor just off the element's leading edge: close enough to + // read as pointing at it, outside it so it never hides the field. + // Clamped inside the view, because full-width fields start close enough + // to the screen edge that the halo would otherwise be sliced in half. + val inset = dp(18f) + val edgeGuard = dp(16f) + CURSOR_HALO_DP * resources.displayMetrics.density + flightEnd.set( + (target.left - inset).coerceAtLeast(edgeGuard), + target.centerY().coerceIn(edgeGuard, (height - edgeGuard).coerceAtLeast(edgeGuard)), + ) + flightStart.set(if (cursorVisible) cursor else if (hasHome) home else flightEnd) + // Bow the path away from the straight line so it arcs instead of + // sliding. A straight slide reads mechanical; an arc reads intentional. + flightControl.set( + (flightStart.x + flightEnd.x) / 2f, + (flightStart.y + flightEnd.y) / 2f - dp(64f), + ) + if (!hasTarget) { current.set(target) hasTarget = true } else { from.set(current) - moveAnimator?.cancel() - moveAnimator = ValueAnimator.ofFloat(0f, 1f).apply { - duration = 320 - interpolator = android.view.animation.DecelerateInterpolator() - addUpdateListener { - val f = it.animatedValue as Float - current.set( - lerp(from.left, target.left, f), - lerp(from.top, target.top, f), - lerp(from.right, target.right, f), - lerp(from.bottom, target.bottom, f), - ) - invalidate() - } - start() + } + cursorVisible = true + trail.clear() + + flightAnimator?.cancel() + ringReveal = 0f + flightAnimator = ValueAnimator.ofFloat(0f, 1f).apply { + duration = FLIGHT_MS + // Quick departure, long settle — the shape of a deliberate gesture. + interpolator = PathInterpolator(0.22f, 1f, 0.36f, 1f) + addUpdateListener { a -> + val f = a.animatedValue as Float + quadTo(flightStart, flightControl, flightEnd, f, cursor) + pushTrail(cursor) + // The ring only starts blooming once the cursor is most of the + // way there, so the two read as cause and effect. + ringReveal = ((f - 0.55f) / 0.45f).coerceIn(0f, 1f) + val ease = DECELERATE.getInterpolation(f) + current.set( + lerp(from.left, target.left, ease), + lerp(from.top, target.top, ease), + lerp(from.right, target.right, ease), + lerp(from.bottom, target.bottom, ease), + ) + invalidate() } + start() } + if (pulseEnabled && !pulseAnimator.isStarted) pulseAnimator.start() if (!pulseEnabled) pulseAnimator.cancel() visibility = VISIBLE invalidate() } + /** Withdraw cleanly: the cursor flies home, then everything disappears. */ fun clear() { - hasTarget = false + flightAnimator?.cancel() pulseAnimator.cancel() - moveAnimator?.cancel() + if (!cursorVisible || !hasHome) { + finishClear() + return + } + flightStart.set(cursor) + flightEnd.set(home) + flightControl.set( + (flightStart.x + flightEnd.x) / 2f, + (flightStart.y + flightEnd.y) / 2f - dp(48f), + ) + flightAnimator = ValueAnimator.ofFloat(0f, 1f).apply { + duration = RETURN_MS + interpolator = DecelerateInterpolator() + addUpdateListener { a -> + val f = a.animatedValue as Float + quadTo(flightStart, flightControl, flightEnd, f, cursor) + pushTrail(cursor) + ringReveal = 1f - f + invalidate() + } + addListener(onEnd = { finishClear() }) + start() + } + } + + private fun finishClear() { + hasTarget = false + cursorVisible = false + ringReveal = 0f + trail.clear() + flightAnimator = null visibility = GONE invalidate() } override fun onDraw(canvas: Canvas) { - if (!hasTarget) return - val pad = dp(6f) + (if (pulseEnabled) pulse * dp(6f) else 0f) - val rect = RectF( - current.left - pad, - current.top - pad, - current.right + pad, - current.bottom + pad, - ) - glow.alpha = (60 + (if (pulseEnabled) pulse * 120 else 120f)).toInt().coerceIn(0, 255) - val radius = if (shape == "circle") maxOf(rect.width(), rect.height()) / 2f else dp(14f) - if (shape == "circle") { - val cx = rect.centerX() - val cy = rect.centerY() - canvas.drawCircle(cx, cy, radius, glow) - canvas.drawCircle(cx, cy, radius, stroke) - } else { - canvas.drawRoundRect(rect, radius, radius, glow) - canvas.drawRoundRect(rect, radius, radius, stroke) + if (!hasTarget && !cursorVisible) return + + if (hasTarget && ringReveal > 0f) { + val pad = dp(6f) + (if (pulseEnabled) pulse * dp(6f) else 0f) + val rect = RectF( + current.left - pad, + current.top - pad, + current.right + pad, + current.bottom + pad, + ) + val revealAlpha = ringReveal + glow.alpha = ((60 + (if (pulseEnabled) pulse * 120 else 120f)) * revealAlpha) + .toInt().coerceIn(0, 255) + stroke.alpha = (255 * revealAlpha).toInt().coerceIn(0, 255) + val radius = + if (shape == "circle") maxOf(rect.width(), rect.height()) / 2f else dp(14f) + if (shape == "circle") { + canvas.drawCircle(rect.centerX(), rect.centerY(), radius, glow) + canvas.drawCircle(rect.centerX(), rect.centerY(), radius, stroke) + } else { + canvas.drawRoundRect(rect, radius, radius, glow) + canvas.drawRoundRect(rect, radius, radius, stroke) + } + } + + if (!cursorVisible) return + + // Tether back to the pill: a faint thread so the cursor always reads as + // an extension of the assistant rather than a loose dot. + if (hasHome) { + tetherPath.reset() + tetherPath.moveTo(home.x, home.y) + tetherPath.quadTo( + (home.x + cursor.x) / 2f, + (home.y + cursor.y) / 2f + dp(24f), + cursor.x, cursor.y, + ) + canvas.drawPath(tetherPath, tether) + } + + // Comet tail: oldest smallest and faintest. + trail.forEachIndexed { i, p -> + val f = (i + 1f) / (trail.size + 1f) + trailPaint.alpha = (110 * f).toInt().coerceIn(0, 255) + canvas.drawCircle(p.x, p.y, dp(3f) + dp(3f) * f, trailPaint) } + + val breathe = if (pulseEnabled) pulse else 0.5f + canvas.drawCircle(cursor.x, cursor.y, dp(CURSOR_HALO_DP) + dp(3f) * breathe, cursorHalo) + canvas.drawCircle(cursor.x, cursor.y, dp(4.5f), cursorCore) + } + + private val tetherPath = Path() + + private fun pushTrail(p: PointF) { + trail.addLast(PointF(p.x, p.y)) + while (trail.size > TRAIL_LENGTH) trail.removeFirst() + } + + /** Quadratic bezier, written into [out] to avoid allocating per frame. */ + private fun quadTo(a: PointF, c: PointF, b: PointF, t: Float, out: PointF) { + val inv = 1f - t + out.x = inv * inv * a.x + 2f * inv * t * c.x + t * t * b.x + out.y = inv * inv * a.y + 2f * inv * t * c.y + t * t * b.y } private fun lerp(a: Float, b: Float, f: Float) = a + (b - a) * f private fun dp(v: Float): Float = v * resources.displayMetrics.density + + private companion object { + const val FLIGHT_MS = 620L + const val RETURN_MS = 380L + const val TRAIL_LENGTH = 12 + const val CURSOR_HALO_DP = 11f + val DECELERATE = DecelerateInterpolator(1.6f) + } +} + +/** Tiny helper so the animator listener stays readable. */ +private fun ValueAnimator.addListener(onEnd: () -> Unit) { + addListener(object : android.animation.AnimatorListenerAdapter() { + private var cancelled = false + override fun onAnimationCancel(animation: android.animation.Animator) { + cancelled = true + } + override fun onAnimationEnd(animation: android.animation.Animator) { + if (!cancelled) onEnd() + } + }) } diff --git a/app/src/main/java/com/screensaathi/overlay/OverlayCommand.kt b/app/src/main/java/com/screensaathi/overlay/OverlayCommand.kt index 745daf7..4609db1 100644 --- a/app/src/main/java/com/screensaathi/overlay/OverlayCommand.kt +++ b/app/src/main/java/com/screensaathi/overlay/OverlayCommand.kt @@ -22,4 +22,11 @@ data class OverlayCommand( val expanded: Boolean = false, val instruction: String? = null, val highlight: HighlightBounds? = null, + /** + * Language of [instruction], BCP-47. Optional addition to the v1 overlay + * contract. The renderer uses it for the pill's own labels ("Listening…") + * and the language chip, so the chrome speaks the user's language too + * rather than staying English around a Hindi sentence. + */ + val language: String = "en-IN", ) diff --git a/app/src/main/java/com/screensaathi/overlay/PillLabels.kt b/app/src/main/java/com/screensaathi/overlay/PillLabels.kt new file mode 100644 index 0000000..109a000 --- /dev/null +++ b/app/src/main/java/com/screensaathi/overlay/PillLabels.kt @@ -0,0 +1,41 @@ +package com.screensaathi.overlay + +import com.screensaathi.sarvam.Language + +/** + * The pill's own one-word status, in the user's language. + * + * These were English literals inside the renderer's `when`, so a Hindi speaker + * got a Devanagari instruction under an English "Listening…". The chrome has to + * switch language with the content or the illusion breaks at a glance. + * + * Kept beside the renderer rather than in session/Phrases because these are + * render-time labels for a PillState, not things the assistant ever says aloud. + */ +object PillLabels { + + private val EN = mapOf( + PillState.IDLE to "ScreenSaathi", + PillState.LISTENING to "Listening…", + PillState.THINKING to "Thinking…", + PillState.SPEAKING to "Speaking…", + PillState.GUIDING to "Guiding you", + PillState.ERROR to "Let's try again", + ) + + private val HI = mapOf( + PillState.IDLE to "स्क्रीन साथी", + PillState.LISTENING to "सुन रहा हूँ…", + PillState.THINKING to "सोच रहा हूँ…", + PillState.SPEAKING to "बोल रहा हूँ…", + PillState.GUIDING to "रास्ता दिखा रहा हूँ", + PillState.ERROR to "फिर से कोशिश करें", + ) + + private val BY_LANGUAGE = mapOf("en-IN" to EN, "hi-IN" to HI) + + fun forState(state: PillState, language: String): String { + val table = BY_LANGUAGE[Language.normalize(language)] ?: EN + return table[state] ?: EN.getValue(state) + } +} diff --git a/app/src/main/java/com/screensaathi/sarvam/Language.kt b/app/src/main/java/com/screensaathi/sarvam/Language.kt new file mode 100644 index 0000000..72c32a5 --- /dev/null +++ b/app/src/main/java/com/screensaathi/sarvam/Language.kt @@ -0,0 +1,152 @@ +package com.screensaathi.sarvam + +/** + * The languages ScreenSaathi will speak, and the rules for choosing one. + * + * The list is not aspirational — every code here was verified end to end + * against live Sarvam on 2026-07-26 by `scripts/smoke_languages.ps1`: Bulbul v3 + * synthesised the language with speaker `anand`, and Saaras v3 detected it back + * correctly from the resulting audio. Re-run that script before adding a code. + */ +object Language { + + /** + * Used when nothing has been detected yet, and for any text we only + * authored in English. English is the safe default because an English + * string sent with a non-English `target_language_code` is rejected by + * Bulbul outright: + * + * "Text must contain at least one character from the allowed languages." + * + * That is the whole reason [Spoken] carries its own language: the code sent + * to TTS has to describe the *text*, not the user's spoken language. + */ + const val DEFAULT = "en-IN" + + /** Verified Bulbul-speakable / Saaras-detectable. See the smoke script. */ + val SUPPORTED = linkedSetOf( + "en-IN", "hi-IN", "bn-IN", "gu-IN", "kn-IN", + "ml-IN", "mr-IN", "pa-IN", "ta-IN", "te-IN", + ) + + /** Endonyms — what a speaker calls their own language. Used in the UI. */ + private val NATIVE_NAME = mapOf( + "en-IN" to "English", + "hi-IN" to "हिन्दी", + "bn-IN" to "বাংলা", + "gu-IN" to "ગુજરાતી", + "kn-IN" to "ಕನ್ನಡ", + "ml-IN" to "മലയാളം", + "mr-IN" to "मराठी", + "pa-IN" to "ਪੰਜਾਬੀ", + "ta-IN" to "தமிழ்", + "te-IN" to "తెలుగు", + ) + + /** + * Coerce whatever Saaras or the planner hands us into a code we know we can + * speak. Accepts a bare tag ("hi") as well as a full one ("hi-IN"), because + * the planner is a language model and will occasionally shorten it. + * + * An unrecognised code silently became a 400 from Bulbul and the app went + * mute with no explanation, so everything funnels through here. + */ + fun normalize(code: String?): String { + val raw = code?.trim()?.replace('_', '-') ?: return DEFAULT + if (raw.isEmpty()) return DEFAULT + SUPPORTED.firstOrNull { it.equals(raw, ignoreCase = true) }?.let { return it } + val base = raw.substringBefore('-').lowercase() + return SUPPORTED.firstOrNull { it.substringBefore('-') == base } ?: DEFAULT + } + + fun isSupported(code: String?): Boolean = + code != null && SUPPORTED.any { it.equals(code.trim(), ignoreCase = true) } + + /** For the debug panel and the language chip on the card. */ + fun nativeName(code: String): String = NATIVE_NAME[normalize(code)] ?: code + + /** Script each language is actually written in. Hindi and Marathi share one. */ + private val SCRIPT = mapOf( + "en-IN" to Character.UnicodeScript.LATIN, + "hi-IN" to Character.UnicodeScript.DEVANAGARI, + "mr-IN" to Character.UnicodeScript.DEVANAGARI, + "bn-IN" to Character.UnicodeScript.BENGALI, + "gu-IN" to Character.UnicodeScript.GUJARATI, + "kn-IN" to Character.UnicodeScript.KANNADA, + "ml-IN" to Character.UnicodeScript.MALAYALAM, + "pa-IN" to Character.UnicodeScript.GURMUKHI, + "ta-IN" to Character.UnicodeScript.TAMIL, + "te-IN" to Character.UnicodeScript.TELUGU, + ) + + /** First supported code written in each script; the inverse of [SCRIPT]. */ + private val CODE_FOR_SCRIPT: Map = + SCRIPT.entries.reversed().associate { (code, script) -> script to code } + + /** + * Scripts present in [text], with a count of characters in each. + * + * Counts every character of the script, not just `isLetter` ones. Devanagari + * vowel signs like "ा" and "ि" are combining marks, not letters, so a + * letters-only count undercounts Devanagari badly — "amount यहाँ भरिए" comes + * out 6 Latin to 5 Devanagari and gets called English. + */ + private fun scriptsIn(text: String): Map { + val counts = HashMap() + var i = 0 + while (i < text.length) { + val cp = text.codePointAt(i) + i += Character.charCount(cp) + val script = runCatching { Character.UnicodeScript.of(cp) }.getOrNull() ?: continue + if (script == Character.UnicodeScript.COMMON || + script == Character.UnicodeScript.UNKNOWN || + script == Character.UnicodeScript.INHERITED + ) continue + counts[script] = (counts[script] ?: 0) + 1 + } + return counts + } + + /** + * The language code it is actually safe to send to Bulbul for [text]. + * + * Bulbul rejects a text/code mismatch outright — + * "Text must contain at least one character from the allowed languages" — + * and a rejected call means the app just goes silent with no explanation. + * The planner is a language model, so it will sometimes label romanised + * Hindi as `hi-IN`, or answer in English while claiming the user's language. + * + * The rule follows Bulbul's own: **presence, not majority**. + * - If the claimed language's script appears at all, keep the claim. This + * is what makes code-switching work — "amount यहाँ भरिए" stays `hi-IN`, + * and Bulbul reads the embedded English word naturally. + * - Otherwise fall to whichever Indic script is actually there, so + * Devanagari labelled `en-IN` is corrected rather than mispronounced. + * - Failing that, romanised or English text becomes `en-IN`. + */ + fun reconcile(text: String, claimed: String?): String { + val claim = normalize(claimed) + val present = scriptsIn(text) + if (present.isEmpty()) return claim // digits or punctuation only + + SCRIPT[claim]?.let { if (present.containsKey(it)) return claim } + + val indic = present.entries + .filter { it.key != Character.UnicodeScript.LATIN } + .maxByOrNull { it.value } + ?.key + if (indic != null) CODE_FOR_SCRIPT[indic]?.let { return it } + + return DEFAULT + } +} + +/** + * A piece of text together with the language it is actually written in. + * + * Language detection used to be propagated to TTS while the *text* stayed + * English, so a Hindi speaker heard English words requested as `hi-IN`. Pairing + * the two makes that mismatch unrepresentable: whatever produced the string — + * the task DSL, the planner, or a UI phrase — says which language it wrote. + */ +data class Spoken(val text: String, val language: String = Language.DEFAULT) diff --git a/app/src/main/java/com/screensaathi/sarvam/PlannerResult.kt b/app/src/main/java/com/screensaathi/sarvam/PlannerResult.kt index 8f7bcc2..ff62344 100644 --- a/app/src/main/java/com/screensaathi/sarvam/PlannerResult.kt +++ b/app/src/main/java/com/screensaathi/sarvam/PlannerResult.kt @@ -13,4 +13,14 @@ data class PlannerResult( val instruction: String, val confidence: Double, val reason: String, -) + /** + * BCP-47 code of the language [instruction] is written in — an optional + * addition to the v1 contract, which permits new optional fields but no + * removals. Always normalized to a code we can actually speak, so the pair + * can go straight to Bulbul. + */ + val language: String = Language.DEFAULT, +) { + /** The instruction together with the language it is written in. */ + val spoken: Spoken get() = Spoken(instruction, language) +} diff --git a/app/src/main/java/com/screensaathi/sarvam/Sarvam.kt b/app/src/main/java/com/screensaathi/sarvam/Sarvam.kt index de881b2..6daf177 100644 --- a/app/src/main/java/com/screensaathi/sarvam/Sarvam.kt +++ b/app/src/main/java/com/screensaathi/sarvam/Sarvam.kt @@ -35,4 +35,23 @@ object Sarvam { .readTimeout(15, TimeUnit.SECONDS) .build() } + + /** + * The planner gets a hard ceiling on the whole call, not just on socket + * idle time. + * + * Its budget is 700 ms and it normally answers in ~600. But an overlong + * system prompt was observed making sarvam-30b stall well past 40 s + * without ever tripping the read timeout, because bytes kept trickling. + * On stage that is a frozen pill. The step engine can answer instantly and + * for free, so anything beyond a few seconds is strictly worse than + * falling back — share the connection pool, cap the call. + */ + val plannerHttp: OkHttpClient by lazy { + http.newBuilder() + .callTimeout(PLANNER_CALL_TIMEOUT_S, TimeUnit.SECONDS) + .build() + } + + private const val PLANNER_CALL_TIMEOUT_S = 5L } diff --git a/app/src/main/java/com/screensaathi/sarvam/SarvamPlanner.kt b/app/src/main/java/com/screensaathi/sarvam/SarvamPlanner.kt index 49772b9..cd25ecc 100644 --- a/app/src/main/java/com/screensaathi/sarvam/SarvamPlanner.kt +++ b/app/src/main/java/com/screensaathi/sarvam/SarvamPlanner.kt @@ -27,16 +27,31 @@ class SarvamPlanner(context: Context) { context.assets.open(PROMPT).bufferedReader().use { it.readText() } }.getOrDefault("You are ScreenSaathi's planner. Call set_plan once.") - /** Blocking. Call off the main thread. */ - fun plan(transcript: String, task: GuidedTask, screen: ScreenSnapshot): PlannerResult? { + /** + * Blocking. Call off the main thread. + * + * @param spokenLanguage what Saaras detected the user speaking; the planner + * is asked to answer in it. + * @param currentStepId where the user is right now. Without it the prompt's + * "if unclear, stay on the current step" rule was unfollowable — the model + * was never told which step that was. + */ + fun plan( + transcript: String, + task: GuidedTask, + screen: ScreenSnapshot, + spokenLanguage: String, + currentStepId: String?, + ): PlannerResult? { if (!Sarvam.hasKey()) { Log.w(TAG, "No Sarvam key — planner unavailable") return null } + val userContent = buildUserContent(transcript, task, screen, spokenLanguage, currentStepId) val messages = JSONArray() .put(JSONObject().put("role", "system").put("content", systemPrompt)) - .put(JSONObject().put("role", "user").put("content", buildUserContent(transcript, task, screen))) + .put(JSONObject().put("role", "user").put("content", userContent)) val payload = JSONObject() .put("model", Sarvam.PLANNER_MODEL) @@ -56,13 +71,13 @@ class SarvamPlanner(context: Context) { .build() return try { - Sarvam.http.newCall(req).execute().use { resp -> + Sarvam.plannerHttp.newCall(req).execute().use { resp -> val raw = resp.body?.string().orEmpty() if (!resp.isSuccessful) { Log.w(TAG, "Planner ${resp.code}: $raw") return null } - parse(raw, task) + parse(raw, task, spokenLanguage) } } catch (e: Exception) { Log.w(TAG, "Planner failed: ${e.message}") @@ -70,10 +85,24 @@ class SarvamPlanner(context: Context) { } } - private fun buildUserContent(transcript: String, task: GuidedTask, screen: ScreenSnapshot): String { - val steps = task.steps.joinToString("\n") { "- ${it.id} (resource_id=${it.resourceId})" } + private fun buildUserContent( + transcript: String, + task: GuidedTask, + screen: ScreenSnapshot, + spokenLanguage: String, + currentStepId: String?, + ): String { + // Marking CURRENT is what makes "if unclear, stay put" and "go back a + // step" answerable at all. Listing the ids alone left the model + // guessing where the user already was. + val steps = task.steps.joinToString("\n") { + val marker = if (it.id == currentStepId) " <- CURRENT" else "" + "- ${it.id} (resource_id=${it.resourceId})$marker" + } return buildString { - append("User said: \"").append(transcript).append("\"\n\n") + append("User said: \"").append(transcript).append("\"\n") + append("Detected spoken language: ").append(Language.normalize(spokenLanguage)) + append(" — reply in this language, in its own script.\n\n") append("Task: ").append(task.id).append(" — ").append(task.title).append("\n") append("Steps:\n").append(steps).append("\n\n") append(screen.toPromptText()) @@ -98,14 +127,24 @@ class SarvamPlanner(context: Context) { .put("index", JSONObject().put("type", "integer")) ).put("required", JSONArray().put("resource_id").put("index")) ) - .put("instruction", JSONObject().put("type", "string")) + .put( + "instruction", + JSONObject().put("type", "string") + .put("description", "One short sentence, in the user's own language and script.") + ) + .put( + "language", + JSONObject().put("type", "string") + .put("enum", JSONArray().apply { Language.SUPPORTED.forEach { put(it) } }) + .put("description", "BCP-47 code of the language `instruction` is written in.") + ) .put("confidence", JSONObject().put("type", "number")) .put("reason", JSONObject().put("type", "string")) ) .put( "required", JSONArray().put("intent").put("step").put("target") - .put("instruction").put("confidence").put("reason") + .put("instruction").put("language").put("confidence").put("reason") ) return JSONObject() .put("type", "function") @@ -118,42 +157,55 @@ class SarvamPlanner(context: Context) { ) } - private fun parse(raw: String, task: GuidedTask): PlannerResult? { - val message = JSONObject(raw) - .optJSONArray("choices")?.optJSONObject(0)?.optJSONObject("message") - ?: return null - val call = message.optJSONArray("tool_calls")?.optJSONObject(0) - ?: return null - val argsStr = call.optJSONObject("function")?.optString("arguments") ?: return null - val args = JSONObject(argsStr) - - val step = args.optString("step").takeIf { it.isNotBlank() } ?: return null - // Guard: the model must pick a real step. If not, fail to fallback. - if (task.indexOfStep(step) < 0) { - Log.w(TAG, "Planner returned unknown step '$step'") - return null - } - val target = args.optJSONObject("target") ?: JSONObject() - val declaredRid = target.optString("resource_id") - // Trust the DSL's resource_id for the step over the model's echo — the - // step id is authoritative, the model only chooses which step. - val rid = task.stepById(step)?.resourceId ?: declaredRid - - return PlannerResult( - version = 1, - intent = args.optString("intent", task.id), - step = step, - targetResourceId = rid, - targetIndex = target.optInt("index", -1), - instruction = args.optString("instruction", task.stepById(step)?.instruction ?: ""), - confidence = args.optDouble("confidence", 0.5), - reason = args.optString("reason", "").take(80), - ) - } - companion object { private const val TAG = "SarvamPlanner" private const val PROMPT = "prompts/planner_v1.md" private val JSON = "application/json; charset=utf-8".toMediaType() + + /** Visible for testing: needs no Context, so the parse is unit-testable. */ + fun parse(raw: String, task: GuidedTask, spokenLanguage: String): PlannerResult? { + val message = JSONObject(raw) + .optJSONArray("choices")?.optJSONObject(0)?.optJSONObject("message") + ?: return null + val call = message.optJSONArray("tool_calls")?.optJSONObject(0) + ?: return null + val argsStr = call.optJSONObject("function")?.optString("arguments") ?: return null + val args = JSONObject(argsStr) + + val step = args.optString("step").takeIf { it.isNotBlank() } ?: return null + // Guard: the model must pick a real step. If not, fail to fallback. + if (task.indexOfStep(step) < 0) { + Log.w(TAG, "Planner returned unknown step '$step'") + return null + } + val dslStep = task.stepById(step) + val target = args.optJSONObject("target") ?: JSONObject() + // Trust the DSL's resource_id for the step over the model's echo — + // the step id is authoritative, the model only chooses which step. + val rid = dslStep?.resourceId ?: target.optString("resource_id") + + // If the model gave us nothing to say, fall back to the DSL wording + // for the user's language rather than to an empty utterance. + val spokenText = args.optString("instruction").takeIf { it.isNotBlank() } + ?: dslStep?.spokenFor(spokenLanguage)?.text.orEmpty() + + return PlannerResult( + version = 1, + intent = args.optString("intent", task.id), + step = step, + targetResourceId = rid, + targetIndex = target.optInt("index", -1), + instruction = spokenText, + confidence = args.optDouble("confidence", 0.5), + reason = args.optString("reason", "").take(80), + // The model's own label is only a hint. Reconcile it against the + // script it actually wrote in: a wrong code is a 400 from Bulbul + // and the app going quietly mute. + language = Language.reconcile( + spokenText, + args.optString("language").takeIf { it.isNotBlank() } ?: spokenLanguage, + ), + ) + } } } diff --git a/app/src/main/java/com/screensaathi/sarvam/SarvamStt.kt b/app/src/main/java/com/screensaathi/sarvam/SarvamStt.kt index 44c341a..abaeee1 100644 --- a/app/src/main/java/com/screensaathi/sarvam/SarvamStt.kt +++ b/app/src/main/java/com/screensaathi/sarvam/SarvamStt.kt @@ -17,7 +17,10 @@ import org.json.JSONObject */ class SarvamStt { - data class Result(val transcript: String, val languageCode: String?) + data class Result(val transcript: String, val languageCode: String?) { + /** Detected language, or the safe default when Saaras did not say. */ + val language: String get() = Language.normalize(languageCode) + } /** Blocking. Call off the main thread. */ fun transcribe(wav: File, mode: String = "transcribe"): Result? { @@ -55,7 +58,13 @@ class SarvamStt { val json = JSONObject(raw) val transcript = json.optString("transcript", "") if (transcript.isBlank()) return null - Result(transcript, json.optString("language_code").takeIf { it.isNotBlank() }) + // Saaras returns the language in the speaker's own script + // ("hi-IN" for Devanagari output). Normalize immediately so a + // code we cannot speak never travels further into the app. + val detected = json.optString("language_code") + .takeIf { it.isNotBlank() } + ?.let { Language.normalize(it) } + Result(transcript, detected) } } catch (e: Exception) { Log.w(TAG, "STT failed: ${e.message}") diff --git a/app/src/main/java/com/screensaathi/sarvam/SarvamTts.kt b/app/src/main/java/com/screensaathi/sarvam/SarvamTts.kt index 500f2e3..62c79c8 100644 --- a/app/src/main/java/com/screensaathi/sarvam/SarvamTts.kt +++ b/app/src/main/java/com/screensaathi/sarvam/SarvamTts.kt @@ -17,20 +17,29 @@ import org.json.JSONObject */ class SarvamTts { - /** Blocking. Call off the main thread. Returns WAV bytes ready for playback. */ - fun synthesize( - text: String, - languageCode: String = "hi-IN", - speaker: String = DEFAULT_SPEAKER, - ): ByteArray? { + /** + * Blocking. Call off the main thread. Returns WAV bytes ready for playback. + * + * Takes a [Spoken] rather than a loose text+code pair on purpose. Bulbul + * rejects a mismatch outright — + * "Text must contain at least one character from the allowed languages" — + * and a rejected call is indistinguishable from the app having nothing to + * say. The language is reconciled against the text one last time here, so + * no caller can make the app go mute by mislabelling a string. + */ + fun synthesize(spoken: Spoken, speaker: String = DEFAULT_SPEAKER): ByteArray? { if (!Sarvam.hasKey()) { Log.w(TAG, "No Sarvam key set — TTS unavailable") return null } - if (text.isBlank()) return null + if (spoken.text.isBlank()) return null + val languageCode = Language.reconcile(spoken.text, spoken.language) + if (languageCode != spoken.language) { + Log.w(TAG, "Language '${spoken.language}' does not match the text; sending $languageCode") + } val payload = JSONObject() - .put("text", text) + .put("text", spoken.text) .put("target_language_code", languageCode) .put("speaker", speaker) .put("model", Sarvam.TTS_MODEL) diff --git a/app/src/main/java/com/screensaathi/session/Phrases.kt b/app/src/main/java/com/screensaathi/session/Phrases.kt new file mode 100644 index 0000000..d96b72c --- /dev/null +++ b/app/src/main/java/com/screensaathi/session/Phrases.kt @@ -0,0 +1,78 @@ +package com.screensaathi.session + +import com.screensaathi.sarvam.Language +import com.screensaathi.sarvam.Spoken + +/** + * The assistant's own words — everything it says that does not come from the + * task DSL or the planner. + * + * These were hardcoded English string literals scattered through + * SessionController, so even after the language was correctly detected the app + * still said "I didn't catch that" to a Hindi speaker. They live here instead, + * keyed by language, and each lookup reports which language it actually found + * so TTS is never asked to speak English text as `hi-IN`. + * + * English and Hindi are authored for the demo; any other detected language + * falls back to English text *labelled as English*, which is degraded but + * always speakable. Adding a language is adding a column here. + */ +object Phrases { + + enum class Key { + LISTENING, + THINKING, + DIDNT_CATCH, + HOLD_LONGER, + MIC_OFF, + ALL_DONE, + STOPPED, + RESUMED, + NO_TASKS, + TAP_MIC, + } + + private val EN = mapOf( + Key.LISTENING to "Listening… tap the mic again when you're done.", + Key.THINKING to "One moment…", + Key.DIDNT_CATCH to "I didn't catch that — let's start here.", + Key.HOLD_LONGER to "I didn't catch that — hold the mic a moment longer.", + Key.MIC_OFF to "Microphone access is off, so I can't hear you. I'll guide you step by step.", + Key.ALL_DONE to "That's the last step — you're all done!", + Key.STOPPED to "Stopped. Tap the mic whenever you want to carry on.", + Key.RESUMED to "Let's carry on from where we left off.", + Key.NO_TASKS to "No tasks are installed.", + Key.TAP_MIC to "Tap the mic and tell me what you want to do.", + ) + + private val HI = mapOf( + Key.LISTENING to "सुन रहा हूँ… बोलकर फिर से माइक दबाइए।", + Key.THINKING to "एक पल…", + Key.DIDNT_CATCH to "मैं समझ नहीं पाया — चलिए यहाँ से शुरू करते हैं।", + Key.HOLD_LONGER to "मैं सुन नहीं पाया — माइक थोड़ी देर और दबाए रखिए।", + Key.MIC_OFF to "माइक बंद है, इसलिए मैं सुन नहीं सकता। मैं आपको कदम दर कदम बताता हूँ।", + Key.ALL_DONE to "यही आख़िरी कदम था — काम पूरा हो गया!", + Key.STOPPED to "रोक दिया। जब चाहें माइक दबाकर आगे बढ़िए।", + Key.RESUMED to "चलिए वहीं से आगे बढ़ते हैं।", + Key.NO_TASKS to "कोई काम उपलब्ध नहीं है।", + Key.TAP_MIC to "माइक दबाइए और बताइए आप क्या करना चाहते हैं।", + ) + + private val BY_LANGUAGE = mapOf( + "en-IN" to EN, + "hi-IN" to HI, + ) + + /** + * The phrase in [language] if we have it, otherwise the English text + * *labelled English* — never English words wearing another language's code. + */ + fun get(key: Key, language: String): Spoken { + val code = Language.normalize(language) + BY_LANGUAGE[code]?.get(key)?.let { return Spoken(it, code) } + return Spoken(EN.getValue(key), Language.DEFAULT) + } + + /** Languages we can speak our own words in, as opposed to merely detect. */ + val AUTHORED: Set get() = BY_LANGUAGE.keys +} diff --git a/app/src/main/java/com/screensaathi/session/SessionController.kt b/app/src/main/java/com/screensaathi/session/SessionController.kt index 35c3c2e..3d90d56 100644 --- a/app/src/main/java/com/screensaathi/session/SessionController.kt +++ b/app/src/main/java/com/screensaathi/session/SessionController.kt @@ -14,7 +14,9 @@ import com.screensaathi.overlay.HighlightBounds import com.screensaathi.overlay.OverlayCommand import com.screensaathi.overlay.PillState import com.screensaathi.sarvam.AudioPlayer +import com.screensaathi.sarvam.Language import com.screensaathi.sarvam.Sarvam +import com.screensaathi.sarvam.Spoken import com.screensaathi.sarvam.SarvamPlanner import com.screensaathi.sarvam.SarvamStt import com.screensaathi.sarvam.SarvamTts @@ -64,6 +66,15 @@ class SessionController( private val captureWorker = HandlerThread("saathi-capture").apply { start() } private val capture = Handler(captureWorker.looper) + /** + * Speech synthesis and playback. Separate from [bg] so Bulbul's ~1.2 s — + * the slowest layer in the loop — never sits in front of the next + * highlight resolution. The visual path is the demo; it must never queue + * behind audio. + */ + private val speechWorker = HandlerThread("saathi-speech").apply { start() } + private val speech = Handler(speechWorker.looper) + private val tasks: TaskRepository = TaskRepository.load(context) private var engine: StepEngine? = null @@ -74,7 +85,17 @@ class SessionController( private val player = AudioPlayer(context) @Volatile private var isRecording = false - @Volatile private var lastLanguage = "hi-IN" + + /** + * The language the user last spoke, as detected by Saaras. Everything the + * assistant says is chosen for this. It starts at [Language.DEFAULT] rather + * than a guessed "hi-IN": before anyone has spoken we have no evidence, and + * English is the one language every authored string exists in. + */ + @Volatile private var lastLanguage = Language.DEFAULT + + /** Set by [onStopTapped]; cleared as soon as the user engages again. */ + @Volatile private var stopped = false /** Turn that owns the in-progress capture, so stop() reads the right file. */ @Volatile private var recordingTurn = -1 @@ -122,12 +143,8 @@ class SessionController( // Opening a turn here invalidates any plan still in flight, so a slow // planner response cannot yank the user back a step after they advance. val turn = newTurn() - if (isRecording) { - // Advancing mid-utterance abandons it: that capture belongs to a - // turn the user has just walked away from. - isRecording = false - capture.post { recorder.stop(); purgeStaleCaptures(turn) } - } + abandonRecording(turn) + stopped = false val e = engine if (e == null) { startDefaultTask(turn) @@ -135,18 +152,49 @@ class SessionController( } if (e.isOnLastStep) { currentHighlight = null + val done = Phrases.get(Phrases.Key.ALL_DONE, lastLanguage) renderIfCurrent(turn, OverlayCommand(PillState.IDLE, expanded = true, - instruction = "That's the last step — you're all done!", highlight = null)) + instruction = done.text, highlight = null)) + bg.post { speak(done, turn) } return } e.advance() presentCurrentStep(e, turn, speak = true) } + /** + * Stop cleanly: silence speech, drop any in-flight turn, clear the ring, + * and keep the step position so the next mic tap resumes rather than + * restarts. The user must be able to call it off mid-sentence without + * leaving the pill stuck in "Listening…" or the ring orphaned on screen. + */ + fun onStopTapped() { + val turn = newTurn() + abandonRecording(turn) + stopped = true + player.stop() + currentHighlight = null + val phrase = Phrases.get(Phrases.Key.STOPPED, lastLanguage) + render(OverlayCommand(PillState.IDLE, expanded = true, + instruction = phrase.text, highlight = null)) + publishDebug(turn) { it.copy(note = "stopped by user at step ${engine?.currentStep?.id ?: "-"}") } + } + + /** True once the user has stopped and before they resume. */ + val isStopped: Boolean get() = stopped + + private fun abandonRecording(turn: Int) { + if (!isRecording) return + // The capture belongs to a turn the user has just walked away from. + isRecording = false + capture.post { recorder.stop(); purgeStaleCaptures(turn) } + } + // --- Voice loop ----------------------------------------------------------- private fun startListening() { val turn = newTurn() + stopped = false // A denied microphone used to fail silently: recorder.start() returned // false and the app quietly ran the deterministic task, so the user was @@ -154,7 +202,7 @@ class SessionController( if (!hasMicPermission()) { startDefaultTask( turn, - lead = "Microphone access is off, so I can't hear you — I'll guide you step by step.", + lead = Phrases.get(Phrases.Key.MIC_OFF, lastLanguage), note = "RECORD_AUDIO denied", ) return @@ -171,7 +219,8 @@ class SessionController( currentHighlight = null debug = VoiceDebug() renderIfCurrent(turn, OverlayCommand(PillState.LISTENING, expanded = true, - instruction = "Listening… tap the mic again when you're done.")) + instruction = Phrases.get(Phrases.Key.LISTENING, lastLanguage).text, + language = lastLanguage)) capture.post { purgeStaleCaptures(turn) @@ -186,7 +235,9 @@ class SessionController( // The capture's own turn, not whatever is current — they diverge if // anything opened a turn while the user was still speaking. val turn = recordingTurn - render(OverlayCommand(PillState.THINKING, expanded = true, instruction = "One moment…")) + render(OverlayCommand(PillState.THINKING, expanded = true, + instruction = Phrases.get(Phrases.Key.THINKING, lastLanguage).text, + language = lastLanguage)) capture.post { recorder.stop() @@ -205,7 +256,7 @@ class SessionController( if (heldMs < MIN_SPEECH_MS) { publishDebug(turn) { it.copy(note = "too short (${heldMs}ms) — no STT call") } wav.delete() - fallbackAfterFailedSpeech(turn, "I didn't catch that — hold the mic a moment longer.") + fallbackAfterFailedSpeech(turn, Phrases.Key.HOLD_LONGER) return } @@ -221,23 +272,32 @@ class SessionController( if (sttResult == null || sttResult.transcript.isBlank()) { publishDebug(turn) { it.copy(sttMs = sttMs, note = "STT returned nothing") } - fallbackAfterFailedSpeech(turn, "I didn't catch that — let's start here.") + fallbackAfterFailedSpeech(turn, Phrases.Key.DIDNT_CATCH) return } - sttResult.languageCode?.let { lastLanguage = it } - publishDebug(turn) { it.copy(heard = sttResult.transcript, sttMs = sttMs) } + // Everything the assistant says from here on is chosen for this. + lastLanguage = sttResult.language + publishDebug(turn) { + it.copy(heard = sttResult.transcript, sttMs = sttMs, language = lastLanguage) + } val task = pickTask(sttResult.transcript) if (task == null) { publishDebug(turn) { it.copy(note = "no task matched") } - fallbackAfterFailedSpeech(turn, "I didn't catch that — let's start here.") + fallbackAfterFailedSpeech(turn, Phrases.Key.DIDNT_CATCH) return } val e = engine?.takeIf { it.task.id == task.id } ?: StepEngine(task).also { engine = it } val snap = ScreenReaderService.instance?.snapshot() ?: ScreenSnapshot.EMPTY val tp0 = SystemClock.uptimeMillis() - val plan = planner.plan(sttResult.transcript, task, snap) + val plan = planner.plan( + transcript = sttResult.transcript, + task = task, + screen = snap, + spokenLanguage = lastLanguage, + currentStepId = e.currentStep.id, + ) val planMs = SystemClock.uptimeMillis() - tp0 if (!isCurrent(turn)) return @@ -248,9 +308,10 @@ class SessionController( intent = plan.intent, step = plan.step, wantResourceId = plan.targetResourceId, planMs = planMs, confidence = plan.confidence, + language = plan.language, ) } - presentStep(e, plan.instruction, turn, speak = true) + presentStep(e, plan.spoken, turn, speak = true) } else { // Planner unsure or unavailable — deterministic order wins. publishDebug(turn) { @@ -265,19 +326,16 @@ class SessionController( } } - private fun fallbackAfterFailedSpeech(turn: Int, lead: String) { + /** + * Speech failed. Keep the user moving in their own language rather than + * dead-ending, and resume where they were if a task is already running. + */ + private fun fallbackAfterFailedSpeech(turn: Int, key: Phrases.Key) { val e = engine if (e != null) { - presentStep(e, e.currentStep.instruction, turn, speak = true) + presentCurrentStep(e, turn, speak = true) } else { - val task = tasks.byId(DEFAULT_TASK) ?: tasks.tasks.firstOrNull() - if (task == null) { - renderIfCurrent(turn, OverlayCommand(PillState.ERROR, expanded = true, - instruction = "No tasks are installed.")) - } else { - val ne = StepEngine(task); engine = ne - presentStep(ne, lead, turn, speak = true) - } + startDefaultTask(turn, lead = Phrases.get(key, lastLanguage)) } } @@ -288,23 +346,24 @@ class SessionController( * the caller explain *why* we are on this path instead of silently * pretending the voice loop ran. */ - private fun startDefaultTask(turn: Int, lead: String? = null, note: String? = null) { + private fun startDefaultTask(turn: Int, lead: Spoken? = null, note: String? = null) { note?.let { n -> publishDebug(turn) { it.copy(note = n) } } val task = tasks.byId(DEFAULT_TASK) ?: tasks.tasks.firstOrNull() if (task == null) { + val none = Phrases.get(Phrases.Key.NO_TASKS, lastLanguage) renderIfCurrent(turn, OverlayCommand(PillState.ERROR, expanded = true, - instruction = "No tasks are installed.")) + instruction = none.text, language = none.language)) return } val e = StepEngine(task); engine = e - presentStep(e, lead ?: e.currentStep.instruction, turn, speak = true) + presentStep(e, lead ?: e.currentStep.spokenFor(lastLanguage), turn, speak = true) } private fun pickTask(transcript: String): GuidedTask? = tasks.matchByUtterance(transcript) ?: tasks.byId(DEFAULT_TASK) ?: tasks.tasks.firstOrNull() private fun presentCurrentStep(e: StepEngine, turn: Int, speak: Boolean) { - presentStep(e, e.currentStep.instruction, turn, speak) + presentStep(e, e.currentStep.spokenFor(lastLanguage), turn, speak) } /** @@ -312,10 +371,16 @@ class SessionController( * instruction. Rendering is always pushed back through the callback, and * always gated on [turn] still being live. */ - private fun presentStep(e: StepEngine, instruction: String, turn: Int, speak: Boolean) { + private fun presentStep(e: StepEngine, instruction: Spoken, turn: Int, speak: Boolean) { val step = e.currentStep renderIfCurrent(turn, OverlayCommand(PillState.GUIDING, expanded = true, - instruction = instruction, highlight = null)) + instruction = instruction.text, language = instruction.language, highlight = null)) + + // Speech is kicked off in parallel with bounds resolution, on its own + // thread. Bulbul's ~1.2 s is the slowest layer we have; running it here + // meant the *next* interaction's highlight queued behind it, so an + // impatient second tap looked like a frozen pill. + if (speak) speech.post { speak(instruction, turn) } bg.post { if (!isCurrent(turn)) return@post @@ -340,26 +405,28 @@ class SessionController( if (!isCurrent(turn)) return@post currentHighlight = hl render(OverlayCommand(PillState.GUIDING, expanded = true, - instruction = instruction, highlight = hl)) - if (speak) speak(instruction, turn) + instruction = instruction.text, language = instruction.language, highlight = hl)) } } - private fun speak(text: String, turn: Int) { + /** Runs on [speech]. Never on the thread that resolves the highlight. */ + private fun speak(spoken: Spoken, turn: Int) { if (!Sarvam.hasKey()) return - val bytes = tts.synthesize(text, languageCode = lastLanguage) ?: return - if (!isCurrent(turn)) return + val bytes = tts.synthesize(spoken) ?: return + if (!isCurrent(turn) || stopped) return // Carry currentHighlight through both renders — the ring must survive // the speaking state, not blink out while the instruction is read. player.play( bytes, onStart = { renderIfCurrent(turn, OverlayCommand(PillState.SPEAKING, expanded = true, - instruction = text, highlight = currentHighlight)) + instruction = spoken.text, language = spoken.language, + highlight = currentHighlight)) }, onDone = { renderIfCurrent(turn, OverlayCommand(PillState.GUIDING, expanded = true, - instruction = text, highlight = currentHighlight)) + instruction = spoken.text, language = spoken.language, + highlight = currentHighlight)) }, ) } @@ -398,6 +465,7 @@ class SessionController( recorder.stop() worker.quitSafely() captureWorker.quitSafely() + speechWorker.quitSafely() } companion object { diff --git a/app/src/main/java/com/screensaathi/session/VoiceDebug.kt b/app/src/main/java/com/screensaathi/session/VoiceDebug.kt index 39276e4..310e99d 100644 --- a/app/src/main/java/com/screensaathi/session/VoiceDebug.kt +++ b/app/src/main/java/com/screensaathi/session/VoiceDebug.kt @@ -1,5 +1,7 @@ package com.screensaathi.session +import com.screensaathi.sarvam.Language + /** * One voice turn's diagnostics, accumulated and rendered as a single panel. * @@ -15,6 +17,8 @@ package com.screensaathi.session */ data class VoiceDebug( val heard: String? = null, + /** Language detected from speech, or the one the planner replied in. */ + val language: String? = null, val intent: String? = null, val step: String? = null, val wantResourceId: String? = null, @@ -34,6 +38,7 @@ data class VoiceDebug( fun toPanel(): String = buildString { heard?.let { line("heard", it.take(40)) } + language?.let { line("lang", "$it (${Language.nativeName(it)})") } if (intent != null || step != null) { line("intent", "${intent ?: "-"} step: ${step ?: "-"}") } diff --git a/app/src/main/java/com/screensaathi/task/TaskModels.kt b/app/src/main/java/com/screensaathi/task/TaskModels.kt index 9c5b214..9fbebb4 100644 --- a/app/src/main/java/com/screensaathi/task/TaskModels.kt +++ b/app/src/main/java/com/screensaathi/task/TaskModels.kt @@ -1,5 +1,8 @@ package com.screensaathi.task +import com.screensaathi.sarvam.Language +import com.screensaathi.sarvam.Spoken + /** * In-memory form of the Task DSL (contracts/task.schema.json). * Plain data classes — no framework, so the step engine stays testable. @@ -13,10 +16,28 @@ data class Highlight( data class TaskStep( val id: String, val resourceId: String, + /** Base wording, authored in [Language.DEFAULT]. Always present. */ val instruction: String, + /** Optional per-language wording, keyed by full code ("hi-IN"). */ + val instructions: Map = emptyMap(), val expectsValue: Boolean = false, val highlight: Highlight = Highlight(), -) +) { + /** + * The step's wording in [language] if the DSL carries it, otherwise the + * base English text labelled as English. + * + * Returning a [Spoken] rather than a bare String is the point: the offline + * path used to hand English DSL text to Bulbul tagged with the user's + * detected language, which Bulbul rejects outright. The caller can no + * longer lose track of which language the words are in. + */ + fun spokenFor(language: String): Spoken { + val code = Language.normalize(language) + instructions[code]?.let { return Spoken(it, code) } + return Spoken(instruction, Language.DEFAULT) + } +} data class GuidedTask( val version: Int, diff --git a/app/src/main/java/com/screensaathi/task/TaskRepository.kt b/app/src/main/java/com/screensaathi/task/TaskRepository.kt index d12fae3..d7c386f 100644 --- a/app/src/main/java/com/screensaathi/task/TaskRepository.kt +++ b/app/src/main/java/com/screensaathi/task/TaskRepository.kt @@ -2,6 +2,7 @@ package com.screensaathi.task import android.content.Context import android.util.Log +import com.screensaathi.sarvam.Language import org.json.JSONObject /** @@ -33,11 +34,22 @@ class TaskRepository private constructor(val tasks: List) { return if (bestScore > 0) best else null } + /** + * Lowercase, strip punctuation, split into words of 3+ characters. + * + * Punctuation is removed by *category*, not by an `[^a-z0-9 ]` allowlist. + * That allowlist deleted every non-Latin character, so a Saaras transcript + * of Hindi speech — which comes back in Devanagari, e.g. + * "बिजली का बिल भरना है" — normalized to the empty set and matched nothing + * at all. Hindi is the primary demo language, so the matcher was silently + * dead on the path that matters most. + */ private fun normalize(s: String): Set = s.lowercase() - .replace(Regex("[^a-z0-9 ]"), " ") - .split(Regex("\\s+")) - .filter { it.length > 2 } + .map { if (it.isLetterOrDigit()) it else ' ' } + .joinToString("") + .split(' ') + .filter { it.length >= MIN_WORD_LENGTH } .toSet() private fun overlap(a: Set, b: Set): Int = a.count { it in b } @@ -46,6 +58,12 @@ class TaskRepository private constructor(val tasks: List) { private const val TAG = "TaskRepository" private const val DIR = "tasks" + /** Drops "a"/"is"/"का" style filler without dropping real content words. */ + private const val MIN_WORD_LENGTH = 3 + + /** Visible for testing: build a repository without an AssetManager. */ + fun of(tasks: List): TaskRepository = TaskRepository(tasks) + fun load(context: Context): TaskRepository { val out = mutableListOf() val am = context.assets @@ -67,6 +85,26 @@ class TaskRepository private constructor(val tasks: List) { return TaskRepository(out) } + /** + * Optional `"instructions": { "hi-IN": "…" }` block on a step. Unknown + * or unspeakable codes are dropped here rather than at synthesis time, + * where they would surface as a silent 400 from Bulbul. + */ + private fun parseInstructions(o: JSONObject?): Map { + if (o == null) return emptyMap() + val out = LinkedHashMap() + for (key in o.keys()) { + val text = o.optString(key) + if (text.isBlank()) continue + if (!Language.isSupported(key)) { + Log.w(TAG, "Ignoring unsupported instruction language '$key'") + continue + } + out[Language.normalize(key)] = text + } + return out + } + fun parse(o: JSONObject): GuidedTask { val stepsJson = o.getJSONArray("steps") val steps = ArrayList(stepsJson.length()) @@ -78,6 +116,7 @@ class TaskRepository private constructor(val tasks: List) { id = s.getString("id"), resourceId = s.getString("resource_id"), instruction = s.getString("instruction"), + instructions = parseInstructions(s.optJSONObject("instructions")), expectsValue = s.optBoolean("expects_value", false), highlight = Highlight( shape = h?.optString("shape", "rect") ?: "rect", diff --git a/app/src/main/res/layout/overlay_pill.xml b/app/src/main/res/layout/overlay_pill.xml index e5e4819..ffe4e20 100644 --- a/app/src/main/res/layout/overlay_pill.xml +++ b/app/src/main/res/layout/overlay_pill.xml @@ -58,6 +58,19 @@ android:gravity="center" android:textColor="@color/saathi_text" android:textSize="15sp" /> + + + @@ -99,5 +112,19 @@ android:textColor="@color/saathi_text" android:textSize="15sp" android:textStyle="bold" /> + + + diff --git a/app/src/test/java/com/screensaathi/sarvam/LanguageTest.kt b/app/src/test/java/com/screensaathi/sarvam/LanguageTest.kt new file mode 100644 index 0000000..aef062f --- /dev/null +++ b/app/src/test/java/com/screensaathi/sarvam/LanguageTest.kt @@ -0,0 +1,141 @@ +package com.screensaathi.sarvam + +import org.junit.Assert.assertEquals +import org.junit.Assert.assertFalse +import org.junit.Assert.assertTrue +import org.junit.Test + +/** + * Language handling is the difference between the assistant answering a Hindi + * speaker in Hindi and the app going silently mute. + * + * The mute failure is real and verified: Bulbul rejects a text/language + * mismatch with 400 "Text must contain at least one character from the allowed + * languages", and a rejected synthesis is indistinguishable from the assistant + * having nothing to say. [Language.reconcile] is the guard, so it is tested + * hardest. + */ +class LanguageTest { + + // --- normalize ------------------------------------------------------------ + + @Test + fun `a supported code passes through`() { + assertEquals("hi-IN", Language.normalize("hi-IN")) + assertEquals("ta-IN", Language.normalize("ta-IN")) + } + + @Test + fun `a bare tag is widened to the full code`() { + // The planner is a language model; it shortens codes sometimes. + assertEquals("hi-IN", Language.normalize("hi")) + assertEquals("bn-IN", Language.normalize("bn")) + } + + @Test + fun `case and underscore variants are accepted`() { + assertEquals("hi-IN", Language.normalize("HI-in")) + assertEquals("hi-IN", Language.normalize("hi_IN")) + assertEquals("hi-IN", Language.normalize(" hi-IN ")) + } + + @Test + fun `anything unspeakable becomes the default rather than reaching Bulbul`() { + assertEquals(Language.DEFAULT, Language.normalize(null)) + assertEquals(Language.DEFAULT, Language.normalize("")) + assertEquals(Language.DEFAULT, Language.normalize("klingon")) + assertEquals(Language.DEFAULT, Language.normalize("fr-FR")) + } + + @Test + fun `isSupported does not silently widen`() { + assertTrue(Language.isSupported("hi-IN")) + assertFalse(Language.isSupported("hi")) + assertFalse(Language.isSupported("fr-FR")) + assertFalse(Language.isSupported(null)) + } + + // --- reconcile: the anti-mute guard -------------------------------------- + + @Test + fun `an honest claim is kept`() { + assertEquals("hi-IN", Language.reconcile("इस बॉक्स में रकम भरिए।", "hi-IN")) + assertEquals("en-IN", Language.reconcile("Enter the amount here.", "en-IN")) + } + + @Test + fun `English text claimed as Hindi is corrected to English`() { + // This is the exact shape of the bug: detection said hi-IN, the text + // stayed English, Bulbul 400s, the app goes quiet. + assertEquals("en-IN", Language.reconcile("Enter the bill amount in this box.", "hi-IN")) + } + + @Test + fun `romanised Hindi is treated as English, because that is what Bulbul accepts`() { + assertEquals("en-IN", Language.reconcile("bijli ka bill bhariye", "hi-IN")) + } + + @Test + fun `Devanagari text mislabelled as English is corrected to Hindi`() { + assertEquals("hi-IN", Language.reconcile("इस बॉक्स में रकम भरिए।", "en-IN")) + } + + @Test + fun `code-switched text keeps the claimed Indic language`() { + // "amount यहाँ भरिए" — Hindi structure with an English noun the user + // already knows. Bulbul only needs one character of the language, and + // reads the embedded English fine, so this must stay hi-IN. + // + // Counting letters would get this wrong: Devanagari vowel signs are + // combining marks rather than letters, so this scores 6 Latin to 5 + // Devanagari and would be mislabelled English. + assertEquals("hi-IN", Language.reconcile("amount यहाँ भरिए", "hi-IN")) + assertEquals("hi-IN", Language.reconcile("submit बटन दबाइए", "hi-IN")) + assertEquals("ta-IN", Language.reconcile("amount இங்கே உள்ளிடவும்", "ta-IN")) + } + + @Test + fun `English with no Indic characters at all is English, whatever is claimed`() { + assertEquals("en-IN", Language.reconcile("Please enter the amount here now", "hi-IN")) + } + + @Test + fun `each script maps to its own language`() { + assertEquals("ta-IN", Language.reconcile("தொகையை உள்ளிடவும்.", "en-IN")) + assertEquals("te-IN", Language.reconcile("మొత్తాన్ని నమోదు చేయండి.", "en-IN")) + assertEquals("bn-IN", Language.reconcile("পরিমাণ লিখুন।", "en-IN")) + assertEquals("kn-IN", Language.reconcile("ಮೊತ್ತವನ್ನು ನಮೂದಿಸಿ.", "en-IN")) + assertEquals("ml-IN", Language.reconcile("തുക നൽകുക.", "en-IN")) + assertEquals("gu-IN", Language.reconcile("રકમ ભરો.", "en-IN")) + assertEquals("pa-IN", Language.reconcile("ਰਕਮ ਭਰੋ।", "en-IN")) + } + + @Test + fun `Devanagari keeps a Marathi claim, since Marathi shares the script`() { + // Correcting mr-IN to hi-IN here would be wrong: both are speakable and + // both are written in Devanagari, so the claim is the only signal. + assertEquals("mr-IN", Language.reconcile("या बॉक्समध्ये रक्कम भरा.", "mr-IN")) + } + + @Test + fun `text with no letters at all keeps the claim rather than guessing`() { + assertEquals("hi-IN", Language.reconcile("123 ...", "hi-IN")) + } + + @Test + fun `native names are endonyms`() { + assertEquals("हिन्दी", Language.nativeName("hi-IN")) + assertEquals("தமிழ்", Language.nativeName("ta-IN")) + assertEquals("English", Language.nativeName("en-IN")) + } + + @Test + fun `every supported language has a native name and a script`() { + for (code in Language.SUPPORTED) { + assertTrue("$code has no native name", Language.nativeName(code) != code) + // A supported code must be reconcilable with its own text, which + // only holds if the script table knows about it. + assertEquals(code, Language.reconcile("123", code)) + } + } +} diff --git a/app/src/test/java/com/screensaathi/sarvam/SarvamPlannerParseTest.kt b/app/src/test/java/com/screensaathi/sarvam/SarvamPlannerParseTest.kt new file mode 100644 index 0000000..921f9bc --- /dev/null +++ b/app/src/test/java/com/screensaathi/sarvam/SarvamPlannerParseTest.kt @@ -0,0 +1,135 @@ +package com.screensaathi.sarvam + +import com.screensaathi.task.GuidedTask +import com.screensaathi.task.TaskStep +import org.junit.Assert.assertEquals +import org.junit.Assert.assertNotNull +import org.junit.Assert.assertNull +import org.junit.Test + +/** + * The planner is a language model behind a forced tool call, so the parser is + * the boundary where its output stops being a suggestion and starts driving the + * UI. These pin what happens when it answers imperfectly. + */ +class SarvamPlannerParseTest { + + private val task = GuidedTask( + version = 1, id = "pay_bill", title = "Pay Electricity Bill", + utterances = emptyList(), + steps = listOf( + TaskStep( + id = "amount", resourceId = "amount_field", + instruction = "Enter the bill amount in this box.", + instructions = mapOf("hi-IN" to "इस बॉक्स में बिल की रकम भरिए।"), + ), + TaskStep(id = "submit", resourceId = "submit_button", instruction = "Tap to pay."), + ), + ) + + private fun response(args: String): String = + """{"choices":[{"message":{"tool_calls":[{"function":{"name":"set_plan","arguments":${ + org.json.JSONObject.quote(args) + }}}]}}]}""" + + @Test + fun `a well-formed Hindi plan is parsed whole`() { + val plan = SarvamPlanner.parse( + response( + """{"intent":"pay_bill","step":"amount", + "target":{"resource_id":"amount_field","index":4}, + "instruction":"इस बॉक्स में बिल की रकम भरिए।", + "language":"hi-IN","confidence":0.97,"reason":"matches amount request"}""" + ), + task, "hi-IN", + ) + assertNotNull(plan) + assertEquals("pay_bill", plan!!.intent) + assertEquals("amount", plan.step) + assertEquals("amount_field", plan.targetResourceId) + assertEquals(4, plan.targetIndex) + assertEquals("hi-IN", plan.language) + assertEquals(0.97, plan.confidence, 0.001) + assertEquals("hi-IN", plan.spoken.language) + } + + @Test + fun `a mislabelled language is corrected to the script actually written`() { + // The model answered in English but claimed the user's Hindi. Trusting + // the claim would 400 at Bulbul and the app would go mute. + val plan = SarvamPlanner.parse( + response( + """{"intent":"pay_bill","step":"amount", + "target":{"resource_id":"amount_field","index":1}, + "instruction":"Enter the bill amount in this box.", + "language":"hi-IN","confidence":0.9,"reason":"ok"}""" + ), + task, "hi-IN", + ) + assertEquals("en-IN", plan!!.language) + } + + @Test + fun `a missing language falls back to the language the user spoke`() { + val plan = SarvamPlanner.parse( + response( + """{"intent":"pay_bill","step":"amount", + "target":{"resource_id":"amount_field","index":1}, + "instruction":"इस बॉक्स में रकम भरिए।","confidence":0.9,"reason":"ok"}""" + ), + task, "hi-IN", + ) + assertEquals("hi-IN", plan!!.language) + } + + @Test + fun `an empty instruction falls back to the DSL wording in the user's language`() { + val plan = SarvamPlanner.parse( + response( + """{"intent":"pay_bill","step":"amount", + "target":{"resource_id":"amount_field","index":1}, + "instruction":"","language":"hi-IN","confidence":0.9,"reason":"ok"}""" + ), + task, "hi-IN", + ) + assertEquals("इस बॉक्स में बिल की रकम भरिए।", plan!!.instruction) + assertEquals("hi-IN", plan.language) + } + + @Test + fun `an invented step is rejected so the caller falls back deterministically`() { + val plan = SarvamPlanner.parse( + response( + """{"intent":"pay_bill","step":"teleport", + "target":{"resource_id":"x","index":1}, + "instruction":"go","language":"en-IN","confidence":0.99,"reason":"ok"}""" + ), + task, "en-IN", + ) + assertNull(plan) + } + + @Test + fun `the DSL resource id wins over the model's echo`() { + val plan = SarvamPlanner.parse( + response( + """{"intent":"pay_bill","step":"submit", + "target":{"resource_id":"hallucinated_id","index":2}, + "instruction":"Tap to pay.","language":"en-IN","confidence":0.8,"reason":"ok"}""" + ), + task, "en-IN", + ) + assertEquals("submit_button", plan!!.targetResourceId) + } + + @Test + fun `prose instead of a tool call is rejected`() { + val raw = """{"choices":[{"message":{"content":"Sure! Let me help you pay."}}]}""" + assertNull(SarvamPlanner.parse(raw, task, "en-IN")) + } + + @Test + fun `an empty response is rejected`() { + assertNull(SarvamPlanner.parse("{}", task, "en-IN")) + } +} diff --git a/app/src/test/java/com/screensaathi/session/PhrasesTest.kt b/app/src/test/java/com/screensaathi/session/PhrasesTest.kt new file mode 100644 index 0000000..3bf6b87 --- /dev/null +++ b/app/src/test/java/com/screensaathi/session/PhrasesTest.kt @@ -0,0 +1,84 @@ +package com.screensaathi.session + +import com.screensaathi.sarvam.Language +import org.junit.Assert.assertEquals +import org.junit.Assert.assertNotEquals +import org.junit.Assert.assertTrue +import org.junit.Test + +/** + * The assistant's own words. The invariant that matters: a phrase is never + * returned wearing a language it is not written in, because that combination + * is a 400 from Bulbul and silence from the app. + */ +class PhrasesTest { + + @Test + fun `Hindi is answered in Hindi`() { + val p = Phrases.get(Phrases.Key.LISTENING, "hi-IN") + assertEquals("hi-IN", p.language) + assertTrue("expected Devanagari", p.text.any { it.code in 0x0900..0x097F }) + } + + @Test + fun `English is answered in English`() { + val p = Phrases.get(Phrases.Key.LISTENING, "en-IN") + assertEquals("en-IN", p.language) + assertEquals("Listening… tap the mic again when you're done.", p.text) + } + + @Test + fun `an unauthored language falls back to English text labelled English`() { + // Tamil is speakable by Bulbul but we have not authored our own words in + // it. The degraded answer must be honest English, never English text + // tagged ta-IN. + val p = Phrases.get(Phrases.Key.LISTENING, "ta-IN") + assertEquals(Language.DEFAULT, p.language) + assertEquals(Phrases.get(Phrases.Key.LISTENING, "en-IN").text, p.text) + } + + @Test + fun `an unknown language code does not throw`() { + val p = Phrases.get(Phrases.Key.THINKING, "klingon") + assertEquals(Language.DEFAULT, p.language) + assertTrue(p.text.isNotBlank()) + } + + @Test + fun `every key exists in every authored language`() { + for (language in Phrases.AUTHORED) { + for (key in Phrases.Key.values()) { + val p = Phrases.get(key, language) + assertEquals("$key/$language fell back", language, p.language) + assertTrue("$key/$language is blank", p.text.isNotBlank()) + } + } + } + + @Test + fun `the Hindi wording is actually translated, not copied English`() { + for (key in Phrases.Key.values()) { + assertNotEquals( + "$key was never translated to Hindi", + Phrases.get(key, "en-IN").text, + Phrases.get(key, "hi-IN").text, + ) + } + } + + @Test + fun `every authored phrase is safe to hand straight to Bulbul`() { + // The end-to-end invariant: whatever we return, reconciling it against + // its own text must not change the code. + for (language in Phrases.AUTHORED) { + for (key in Phrases.Key.values()) { + val p = Phrases.get(key, language) + assertEquals( + "$key/$language would be rejected or re-tagged by TTS", + p.language, + Language.reconcile(p.text, p.language), + ) + } + } + } +} diff --git a/app/src/test/java/com/screensaathi/task/MultilingualTaskTest.kt b/app/src/test/java/com/screensaathi/task/MultilingualTaskTest.kt new file mode 100644 index 0000000..2d3a891 --- /dev/null +++ b/app/src/test/java/com/screensaathi/task/MultilingualTaskTest.kt @@ -0,0 +1,101 @@ +package com.screensaathi.task + +import com.screensaathi.sarvam.Language +import org.json.JSONObject +import org.junit.Assert.assertEquals +import org.junit.Assert.assertNotNull +import org.junit.Assert.assertNull +import org.junit.Assert.assertTrue +import org.junit.Test + +/** + * The offline path has to be multilingual too. When the planner is unavailable + * the DSL wording is all the user gets, so it has to exist in their language — + * and, when it does not, has to be honest about being English. + */ +class MultilingualTaskTest { + + private val json = """ + { + "version": 1, "id": "pay_bill", "title": "Pay Electricity Bill", + "utterances": ["help me pay this bill", "बिजली का बिल भरना है"], + "steps": [ + { "id": "amount", "resource_id": "amount_field", + "instruction": "Enter the bill amount in this box.", + "instructions": { "hi-IN": "इस बॉक्स में बिल की रकम भरिए।", "fr-FR": "Entrez le montant." } } + ] + } + """.trimIndent() + + private fun step() = TaskRepository.parse(JSONObject(json)).steps[0] + + @Test + fun `a translated step is spoken in the user's language`() { + val spoken = step().spokenFor("hi-IN") + assertEquals("hi-IN", spoken.language) + assertEquals("इस बॉक्स में बिल की रकम भरिए।", spoken.text) + } + + @Test + fun `an untranslated language gets English text labelled English`() { + // Not "English text labelled ta-IN" — that combination is a 400 from + // Bulbul and the app goes mute. + val spoken = step().spokenFor("ta-IN") + assertEquals(Language.DEFAULT, spoken.language) + assertEquals("Enter the bill amount in this box.", spoken.text) + } + + @Test + fun `an unspeakable instruction language is dropped at parse time`() { + // fr-FR is in the fixture but Bulbul cannot speak it, so it must never + // reach synthesis. + assertNull(step().instructions["fr-FR"]) + assertNotNull(step().instructions["hi-IN"]) + } + + @Test + fun `a bare tag in the DSL is normalized`() { + val j = """ + { "version": 1, "id": "t", "title": "T", "steps": + [ { "id": "a", "resource_id": "a_field", "instruction": "go", + "instructions": { "hi-IN": "चलिए" } } ] } + """.trimIndent() + val s = TaskRepository.parse(JSONObject(j)).steps[0] + assertEquals("चलिए", s.spokenFor("hi").text) + } + + @Test + fun `steps with no instructions block still work`() { + val j = """ + { "version": 1, "id": "t", "title": "T", "steps": + [ { "id": "a", "resource_id": "a_field", "instruction": "go" } ] } + """.trimIndent() + val s = TaskRepository.parse(JSONObject(j)).steps[0] + assertTrue(s.instructions.isEmpty()) + assertEquals(Language.DEFAULT, s.spokenFor("hi-IN").language) + assertEquals("go", s.spokenFor("hi-IN").text) + } + + // --- utterance matching --------------------------------------------------- + + private fun repo() = TaskRepository.of(listOf(TaskRepository.parse(JSONObject(json)))) + + @Test + fun `a Devanagari transcript matches a Devanagari utterance`() { + // The bug this pins: normalize() used to strip with [^a-z0-9 ], which + // deletes every Devanagari character. Saaras returns Hindi speech AS + // Devanagari, so the matcher scored zero on the primary demo language + // and silently matched nothing at all. + assertEquals("pay_bill", repo().matchByUtterance("बिजली का बिल भरना है")?.id) + } + + @Test + fun `an English transcript still matches`() { + assertEquals("pay_bill", repo().matchByUtterance("help me pay this bill")?.id) + } + + @Test + fun `an unrelated transcript matches nothing`() { + assertNull(repo().matchByUtterance("what is the weather tomorrow")) + } +} diff --git a/contracts/planner.schema.json b/contracts/planner.schema.json index 94b463e..4b9738e 100644 --- a/contracts/planner.schema.json +++ b/contracts/planner.schema.json @@ -5,6 +5,7 @@ "description": "The only thing the planner is allowed to return. FROZEN: no field is ever removed after M1; only optional fields may be added. The overlay renderer consumes target + instruction and ignores the rest; confidence/reason exist for the debug overlay (M5) and for triage, not for rendering.", "type": "object", "required": ["version", "intent", "step", "target", "instruction", "confidence", "reason"], + "$comment": "language is intentionally NOT required: it was added after the M1 freeze, and a plan without it is still a valid v1 plan (it defaults to en-IN).", "additionalProperties": false, "properties": { "version": { @@ -38,7 +39,12 @@ "instruction": { "type": "string", "maxLength": 140, - "description": "One short spoken instruction. Read aloud verbatim by Bulbul and shown on the card." + "description": "One short spoken instruction, written in the user's own language and script. Read aloud verbatim by Bulbul and shown on the card." + }, + "language": { + "type": "string", + "enum": ["en-IN", "hi-IN", "bn-IN", "gu-IN", "kn-IN", "ml-IN", "mr-IN", "pa-IN", "ta-IN", "te-IN"], + "description": "OPTIONAL, added after M1 (the freeze permits additions, never removals). BCP-47 code of the language `instruction` is written in — it describes the text, not the user's speech. Bulbul rejects a text/code mismatch outright (\"Text must contain at least one character from the allowed languages\") and the app then goes silent, so the client reconciles this against the script actually used before trusting it. Absent means en-IN. Every listed code was verified end to end against live Bulbul + Saaras by scripts/smoke_languages.ps1." }, "confidence": { "type": "number", diff --git a/scripts/planner_case.ps1 b/scripts/planner_case.ps1 new file mode 100644 index 0000000..bc57ae2 --- /dev/null +++ b/scripts/planner_case.ps1 @@ -0,0 +1,68 @@ +# Builds one planner request body exactly as SarvamPlanner does, writes it as +# UTF-8 JSON, and POSTs it with curl.exe. PowerShell 5.1's Invoke-RestMethod is +# deliberately not used here — it was hanging and burning CPU on this payload, +# which is a harness problem, not a Sarvam one. +param( + [Parameter(Mandatory=$true)][string]$Said, + [Parameter(Mandatory=$true)][string]$Lang +) +$ErrorActionPreference = "Stop" +$root = "C:\Projects\Sarvam\ScreenSaathi" +$sp = "C:\Users\nitis\AppData\Local\Temp\claude\C--Projects-Sarvam\dba325a3-0f15-439d-9311-f4e11c48cf47\scratchpad" + +$line = Get-Content "$root\local.properties" | Where-Object { $_ -match '^\s*sarvam\.api\.key\s*=' } | Select-Object -First 1 +$Key = ($line -split '=',2)[1].Trim() + +$prompt = [IO.File]::ReadAllText("$root\app\src\main\assets\prompts\planner_v1.md", [Text.UTF8Encoding]::new($false)) + +$screen = @" +Screen: com.screensaathi +Elements: +[0] TextView "Pay Electricity Bill" +[1] EditText id=amount_field E +[2] EditText id=account_field E +[3] Button id=submit_button "Pay Bill" C +"@ + +$user = "User said: `"$Said`"`nDetected spoken language: $Lang - reply in this language, in its own script.`n`nTask: pay_bill - Pay Electricity Bill`nSteps:`n- amount (resource_id=amount_field) <- CURRENT`n- account (resource_id=account_field)`n- submit (resource_id=submit_button)`n`n$screen" + +function J([string]$s) { + $sb = New-Object Text.StringBuilder + foreach ($ch in $s.ToCharArray()) { + $c = [int][char]$ch + switch ($ch) { + '"' { [void]$sb.Append('\"'); continue } + '\' { [void]$sb.Append('\\'); continue } + "`n" { [void]$sb.Append('\n'); continue } + "`r" { [void]$sb.Append('\r'); continue } + "`t" { [void]$sb.Append('\t'); continue } + default { + if ($c -lt 32 -or $c -gt 126) { [void]$sb.AppendFormat('\u{0:x4}', $c) } + else { [void]$sb.Append($ch) } + } + } + } + $sb.ToString() +} + +$body = '{"model":"sarvam-30b","messages":[{"role":"system","content":"' + (J $prompt) + '"},{"role":"user","content":"' + (J $user) + '"}],' + + '"tools":[{"type":"function","function":{"name":"set_plan","description":"Set the next guided step and the element to point at.","parameters":' + + '{"type":"object","properties":{"intent":{"type":"string"},"step":{"type":"string","enum":["amount","account","submit"]},' + + '"target":{"type":"object","properties":{"resource_id":{"type":"string"},"index":{"type":"integer"}},"required":["resource_id","index"]},' + + '"instruction":{"type":"string"},"language":{"type":"string","enum":["en-IN","hi-IN","bn-IN","gu-IN","kn-IN","ml-IN","mr-IN","pa-IN","ta-IN","te-IN"]},' + + '"confidence":{"type":"number"},"reason":{"type":"string"}},' + + '"required":["intent","step","target","instruction","language","confidence","reason"]}}}],' + + '"tool_choice":"required","parallel_tool_calls":false,"reasoning_effort":null,"temperature":0.1,"max_tokens":300}' + +$bodyFile = "$sp\body.json" +[IO.File]::WriteAllText($bodyFile, $body, [Text.UTF8Encoding]::new($false)) + +$curl = "$env:SystemRoot\System32\curl.exe" +$sw = [Diagnostics.Stopwatch]::StartNew() +$resp = & $curl -s --max-time 30 -X POST "https://api.sarvam.ai/v1/chat/completions" ` + -H "api-subscription-key: $Key" -H "Content-Type: application/json" ` + --data-binary "@$bodyFile" 2>&1 | Out-String +$sw.Stop() + +Write-Output "ms=$($sw.ElapsedMilliseconds)" +Write-Output $resp diff --git a/scripts/smoke_languages.ps1 b/scripts/smoke_languages.ps1 new file mode 100644 index 0000000..5572dc5 --- /dev/null +++ b/scripts/smoke_languages.ps1 @@ -0,0 +1,78 @@ +# Verifies the multilingual round trip ScreenSaathi depends on, per language: +# Bulbul TTS (does this language + speaker work?) +# -> Saaras STT (does it come back, and is language_code detected correctly?) +# +# This is the evidence behind the language list in sarvam/Language.kt. Run it +# again if Sarvam changes models or speakers. +# +# powershell -File scripts\smoke_languages.ps1 +param([string]$Key, [string]$Speaker = "anand") + +$ErrorActionPreference = "Continue" + +if ([string]::IsNullOrWhiteSpace($Key)) { + $propsPath = Join-Path (Split-Path $PSScriptRoot -Parent) "local.properties" + $line = Get-Content $propsPath | Where-Object { $_ -match '^\s*sarvam\.api\.key\s*=' } | Select-Object -First 1 + if ($line) { $Key = ($line -split '=', 2)[1].Trim() } + if ([string]::IsNullOrWhiteSpace($Key)) { Write-Host "No key." -ForegroundColor Red; exit 1 } +} +$hdr = @{ "api-subscription-key" = $Key } +$curl = "$env:SystemRoot\System32\curl.exe" + +# One natural sentence per language, of the kind the app actually speaks. +$cases = @( + @{ code = "en-IN"; text = "Enter the bill amount in this box." }, + @{ code = "hi-IN"; text = "इस बॉक्स में बिल की रकम भरिए।" }, + @{ code = "ta-IN"; text = "இந்தப் பெட்டியில் தொகையை உள்ளிடவும்." }, + @{ code = "te-IN"; text = "ఈ పెట్టెలో మొత్తాన్ని నమోదు చేయండి." }, + @{ code = "bn-IN"; text = "এই বাক্সে বিলের পরিমাণ লিখুন।" }, + @{ code = "kn-IN"; text = "ಈ ಪೆಟ್ಟಿಗೆಯಲ್ಲಿ ಮೊತ್ತವನ್ನು ನಮೂದಿಸಿ." }, + @{ code = "ml-IN"; text = "ഈ ബോക്സിൽ തുക നൽകുക." }, + @{ code = "mr-IN"; text = "या बॉक्समध्ये बिलाची रक्कम भरा." }, + @{ code = "gu-IN"; text = "આ બોક્સમાં બિલની રકમ ભરો." }, + @{ code = "pa-IN"; text = "ਇਸ ਬਾਕਸ ਵਿੱਚ ਬਿੱਲ ਦੀ ਰਕਮ ਭਰੋ।" } +) + +Write-Host "`nspeaker = $Speaker`n" -ForegroundColor DarkGray +$ok = @(); $bad = @() + +foreach ($c in $cases) { + $body = @{ + text = $c.text; target_language_code = $c.code + speaker = $Speaker; model = "bulbul:v3" + } | ConvertTo-Json + $utf8 = [Text.Encoding]::UTF8.GetBytes($body) + try { + $t = Measure-Command { + $script:r = Invoke-RestMethod -Uri "https://api.sarvam.ai/text-to-speech" -Method Post ` + -Headers $hdr -ContentType "application/json; charset=utf-8" -Body $utf8 + } + if (-not $r.audios -or $r.audios.Count -eq 0) { + Write-Host ("{0,-7} TTS: no audio" -f $c.code) -ForegroundColor Yellow; $bad += $c.code; continue + } + $bytes = [Convert]::FromBase64String($r.audios[0]) + $wav = "$PSScriptRoot\lang_$($c.code).wav" + [IO.File]::WriteAllBytes($wav, $bytes) + + $raw = & $curl -s -X POST "https://api.sarvam.ai/speech-to-text" ` + -H "api-subscription-key: $Key" ` + -F "model=saaras:v3" -F "mode=transcribe" -F "file=@$wav;type=audio/wav" 2>&1 | Out-String + $p = $null; try { $p = $raw | ConvertFrom-Json } catch {} + Remove-Item $wav -ErrorAction SilentlyContinue + + $detected = if ($p) { $p.language_code } else { "?" } + $match = if ($detected -eq $c.code) { "OK " } else { "DIFF" } + $colour = if ($detected -eq $c.code) { "Green" } else { "Yellow" } + Write-Host ("{0,-7} TTS {1,5}ms detected={2,-7} {3} '{4}'" -f ` + $c.code, [int]$t.TotalMilliseconds, $detected, $match, $p.transcript) -ForegroundColor $colour + $ok += $c.code + } catch { + Write-Host ("{0,-7} FAIL: {1}" -f $c.code, $_.Exception.Message) -ForegroundColor Red + if ($_.ErrorDetails) { Write-Host (" " + $_.ErrorDetails.Message) -ForegroundColor DarkRed } + $bad += $c.code + } +} + +Write-Host "`nusable: $($ok -join ', ')" -ForegroundColor Cyan +if ($bad) { Write-Host "failed: $($bad -join ', ')" -ForegroundColor Red } +Write-Host ""