diff --git a/.github/ISSUE_TEMPLATE/feature_request.md b/.github/ISSUE_TEMPLATE/feature_request.md index c0223ff..8bb2757 100644 --- a/.github/ISSUE_TEMPLATE/feature_request.md +++ b/.github/ISSUE_TEMPLATE/feature_request.md @@ -18,7 +18,7 @@ What's missing or wrong today, concretely — not "it would be nice if." **Does this touch a frozen contract or one of the five core pieces?** -(overlay pill / ScreenReaderService / Saaras STT / Bulbul TTS / Sarvam-30B +(overlay pill / ScreenReaderService / Saaras STT / Bulbul TTS / Sarvam-105B planner / any file in `contracts/`) If yes, explain why the current design can't absorb this change without breaking it. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 6549d79..4ab2653 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -7,7 +7,7 @@ round-trip. ## Before you write code The five core pieces (overlay pill, `ScreenReaderService`, Saaras STT, -Bulbul TTS, Sarvam-30B planner) and the contracts in `contracts/*.schema.json` +Bulbul TTS, Sarvam-105B planner) and the contracts in `contracts/*.schema.json` are **frozen**. That means: - No sixth subsystem. No new top-level component. diff --git a/README.md b/README.md index 594f7e0..f7528ae 100644 --- a/README.md +++ b/README.md @@ -34,7 +34,7 @@ on-device screenshots, not mockups. ```mermaid flowchart TD U([User speaks]) --> STT[Saaras STT
transcript + language] - STT --> P[Sarvam-30B planner
forced tool call] + STT --> P[Sarvam-105B planner
forced tool call] R[ScreenReaderService
read-only accessibility snapshot] --> P P -->|step + target + instruction
+ language, all in the user's script| SC[SessionController] SC --> OV[OverlayService] @@ -51,7 +51,7 @@ Five pieces, and only five — this is a deliberate freeze, not an oversight: | Overlay pill | `OverlayService.kt`, `overlay/HighlightView.kt` | Renders an `OverlayCommand`. Never reasons. | | Screen reader | `ScreenReaderService.kt`, `screen/ScreenSnapshot.kt` | Read-only accessibility snapshot. No gestures. | | Saaras STT | `sarvam/SarvamStt.kt`, `sarvam/WavRecorder.kt` | 16 kHz mono WAV → transcript + language. | -| Sarvam-30B planner | `sarvam/SarvamPlanner.kt` | Forced tool call → the frozen planner contract. | +| Sarvam-105B planner | `sarvam/SarvamPlanner.kt` | Forced tool call → the frozen planner contract. | | Bulbul TTS | `sarvam/SarvamTts.kt`, `sarvam/AudioPlayer.kt` | Instruction → warm spoken audio (`anand`). | `session/SessionController.kt` orchestrates them. `session/StepEngine.kt` is the diff --git a/SECURITY.md b/SECURITY.md index 6fd6559..afc030f 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -25,7 +25,7 @@ buried: and skips its own overlay's views. It does **not** take screenshots and does **not** read arbitrary background apps — only the foreground window, only while the service is enabled. -- That snapshot is sent, as text, to the Sarvam-30B planner API +- That snapshot is sent, as text, to the Sarvam-105B planner API (`sarvam/SarvamPlanner.kt`) to decide which step and element to point at. Recorded audio is sent to Saaras STT; the spoken instruction is sent to Bulbul TTS. All three are outbound calls to `api.sarvam.ai` over HTTPS. diff --git a/app/src/main/assets/prompts/planner_v1.md b/app/src/main/assets/prompts/planner_v1.md index df2a9af..fa6f3c2 100644 --- a/app/src/main/assets/prompts/planner_v1.md +++ b/app/src/main/assets/prompts/planner_v1.md @@ -14,8 +14,11 @@ Call `set_plan` exactly once. Never reply with prose. script — Hindi means Devanagari, not romanised Hindi. If they mixed in English words like "amount" or "submit", keep those words. - `language`: the BCP-47 code of the language you wrote `instruction` in. -- Skip ahead if asked ("just pay"). Go back if they say they made a mistake. - If the request is unclear, choose the step marked CURRENT. +- Skip ahead if asked, in ANY language: "just pay", "go to submit", + "सीधे submit पर ले चलो", "submit पर ले चलो", "बस पे कर दो", "आखिरी step", + "நேரடியா submit பண்ணு" all mean the same thing — jump straight to the last + step, do not restate an earlier one first. Go back if they say they made a + mistake. If the request is unclear, choose the step marked CURRENT. - `confidence` 0..1, below 0.5 only when genuinely unsure. - `reason`: under 10 words. diff --git a/app/src/main/java/com/screensaathi/sarvam/Sarvam.kt b/app/src/main/java/com/screensaathi/sarvam/Sarvam.kt index 6daf177..605286c 100644 --- a/app/src/main/java/com/screensaathi/sarvam/Sarvam.kt +++ b/app/src/main/java/com/screensaathi/sarvam/Sarvam.kt @@ -18,7 +18,15 @@ object Sarvam { const val STT_MODEL = "saaras:v3" const val TTS_MODEL = "bulbul:v3" - const val PLANNER_MODEL = "sarvam-30b" + // sarvam-30b (the model this app was originally built and verified + // against) was retired by Sarvam sometime between 2026-07-26 and + // 2026-07-31 - a live call now returns a hard 400 ("Model 'sarvam-30b' has + // been deprecated... use sarvam-105b"), which the planner correctly turns + // into a null and falls back to the deterministic StepEngine, but that + // means the planner was silently dead for every real user in the + // meantime. Re-verified live against sarvam-105b before switching this + // constant - see scripts/planner_case.ps1. + const val PLANNER_MODEL = "sarvam-105b" const val AUTH_HEADER = "api-subscription-key" diff --git a/app/src/main/java/com/screensaathi/sarvam/SarvamPlanner.kt b/app/src/main/java/com/screensaathi/sarvam/SarvamPlanner.kt index 93fa672..bfb4345 100644 --- a/app/src/main/java/com/screensaathi/sarvam/SarvamPlanner.kt +++ b/app/src/main/java/com/screensaathi/sarvam/SarvamPlanner.kt @@ -11,7 +11,7 @@ import org.json.JSONArray import org.json.JSONObject /** - * Sarvam-30B planner. Forces a single structured tool call (tool_choice + * Sarvam-105B planner. Forces a single structured tool call (tool_choice * required) so the model always returns the FROZEN planner schema, never prose. * reasoning_effort is disabled — this is a fast routing decision, not a * chain-of-thought task, and reasoning tokens would blow the latency budget. diff --git a/app/src/main/java/com/screensaathi/session/SessionController.kt b/app/src/main/java/com/screensaathi/session/SessionController.kt index 05d3ef0..fe5da04 100644 --- a/app/src/main/java/com/screensaathi/session/SessionController.kt +++ b/app/src/main/java/com/screensaathi/session/SessionController.kt @@ -34,7 +34,7 @@ import java.util.concurrent.atomic.AtomicInteger /** * The orchestration layer. Turns UI taps into OverlayCommands, driving the * deterministic StepEngine plus (when a key is present and calls succeed) the - * Sarvam voice loop: Saaras STT -> Sarvam-30B planner -> highlight -> Bulbul TTS. + * Sarvam voice loop: Saaras STT -> Sarvam-105B planner -> highlight -> Bulbul TTS. * * Every network step has a deterministic fallback. If the key is missing or any * call fails, mic-tap starts the task and Next advances in order — the overlay diff --git a/docs/DATA_HANDLING.md b/docs/DATA_HANDLING.md index 0b819a5..e10edf0 100644 --- a/docs/DATA_HANDLING.md +++ b/docs/DATA_HANDLING.md @@ -34,7 +34,7 @@ Three outbound calls, all HTTPS to `api.sarvam.ai`, all defined in | Call | What is sent | What comes back | | --- | --- | --- | | Saaras STT | The recorded `.wav` clip | Transcript + detected language | -| Sarvam-30B planner | The transcript, the current task's step ids, and the current screen snapshot as plain text | A step id, a target resource id, a spoken instruction, and a language code | +| Sarvam-105B planner | The transcript, the current task's step ids, and the current screen snapshot as plain text | A step id, a target resource id, a spoken instruction, and a language code | | Bulbul TTS | The instruction text and its language code | Synthesized audio (WAV) | No third party other than Sarvam ever receives data from this app. There is diff --git a/docs/PARKING_LOT.md b/docs/PARKING_LOT.md index 3797513..aae34a3 100644 --- a/docs/PARKING_LOT.md +++ b/docs/PARKING_LOT.md @@ -14,26 +14,49 @@ fields removed — only optional additions. ## Latency budgets (optimize the offending layer, don't guess) -Measured via `scripts/smoke_sarvam.ps1`, `scripts/smoke_languages.ps1` and -`scripts/smoke_planner_language.ps1` on 2026-07-26. Venue network not yet tested. +Measured via `scripts/smoke_sarvam.ps1`, `scripts/smoke_languages.ps1`, and +`scripts/planner_case.ps1` on 2026-07-26 and re-measured 2026-07-31 after the +forced model swap below. Venue network not yet tested. -| Layer | Target | Measured | Status | -| ---------------------- | ----------- | --------------- | ------ | -| Saaras STT | < 800 ms | 666–729 ms | OK | -| Planner (Sarvam-30B) | < 700 ms | 867–1481 ms | OVER | -| Overlay update | < 16 ms | not instrumented| — | -| Bulbul TTS first audio | < 900 ms | 909–1462 ms | OVER | -| End-to-end response | < 2.5 s | not instrumented| — | +| Layer | Target | Measured (07-26) | Measured (07-31) | Status | +| ---------------------- | ----------- | ----------------- | ----------------- | ------ | +| Saaras STT | < 800 ms | 666–729 ms | not re-measured | OK | +| Planner | < 700 ms | 867–1481 ms | **1261–3300 ms** | OVER | +| Overlay update | < 16 ms | not instrumented | not instrumented | — | +| Bulbul TTS first audio | < 900 ms | 909–1462 ms | not re-measured | OVER | +| End-to-end response | < 2.5 s | not instrumented | not instrumented | — | **Neither over-budget layer blocks the visual any more.** TTS now runs on its own thread, started in parallel with bounds resolution rather than after it, so the cursor and ring land while Bulbul is still synthesising. -The planner regressed from 606 ms to ~900–1400 ms when the prompt grew to carry -the language contract (713 prompt tokens). It is capped at a hard 5 s call -timeout (`Sarvam.plannerHttp`) because the deterministic step engine answers -instantly and for free — beyond a few seconds, falling back is strictly better -than waiting. Trimming the prompt further is the M4 lever. +**2026-07-31: forced model swap, `sarvam-30b` → `sarvam-105b`.** Sarvam +deprecated `sarvam-30b` sometime between 07-26 and 07-31 — a live call now +returns a hard 400 (`"Model 'sarvam-30b' has been deprecated... use +sarvam-105b"`). This is not optional or scheduled work: the planner was +silently dead (falling back to the deterministic engine on every real call) +until this was caught and fixed. `Sarvam.PLANNER_MODEL` now points at +`sarvam-105b`, re-verified live via `scripts/planner_case.ps1` against the +same six-case regression set used on 07-26. + +The swap is a net negative on latency: 105B is a bigger, slower model, and +measured planner latency roughly doubled (1261–3300 ms vs 867–1481 ms). It is +still capped at the same hard 5 s call timeout (`Sarvam.plannerHttp`) — beyond +that, falling back to the free, instant `StepEngine` is still strictly better +than waiting. Reducing planner latency is now a harder problem than trimming +the prompt (M4 item, unchanged) — it may require asking Sarvam for a smaller +model tier once one exists, or moving skip-ahead/correction handling into +deterministic keyword matching so the LLM is only consulted when genuinely +ambiguous. + +One behavioral difference worth watching, not yet resolved: on `sarvam-105b`, +"अरे नहीं, गलती हो गई" (correction, said on the first step) now returns +`step: account` instead of the previous `step: amount` — moving the user +forward on a "we go back" utterance where there is no earlier step to return +to. Not confirmed as objectively wrong (there's no clean target for "go back" +from step one), but it's a change in behavior from the same prompt against a +different model, worth a second look before relying on the correction path +in a demo. ## Demo-day gotchas (learned the hard way on device) @@ -49,18 +72,27 @@ than waiting. Trimming the prompt further is the M4 lever. ## Parked items -- Planner ignores "skip ahead" in Hindi. "सीधे submit पर ले चलो" (take me - straight to submit) returns `step: amount`. English skip-ahead was never - re-tested after the prompt rewrite. → **Planner/prompt improvement**, worth - fixing before the demo if a judge is likely to try it. -- Phrases are authored in English and Hindi only. Bulbul speaks ten languages - and Saaras detects all ten, so a Tamil speaker gets Tamil *planner* - instructions but English chrome ("Listening…"). → **UX improvement**; - adding a language is adding a column to `Phrases`. +- ~~Planner ignores "skip ahead" in Hindi.~~ **Fixed 2026-07-31**: the prompt + now enumerates skip-ahead phrasing across languages explicitly + (`planner_v1.md`); re-verified live, "सीधे submit पर ले चलो" now correctly + returns `step: submit`. See the latency section above for what else changed + in the same pass (the forced `sarvam-105b` swap). +- Phrases are authored in English, Hindi, **and Tamil** (`session/Phrases.kt`, + `overlay/PillLabels.kt`) as of the taxi-flow work. Bulbul speaks ten + languages and Saaras detects all ten; the remaining seven (Bengali, + Gujarati, Kannada, Malayalam, Marathi, Punjabi, Telugu) still get planner + instructions in the right language but English chrome ("Listening…"). + → **UX improvement**; adding a language is adding a column to `Phrases` and + `PillLabels`. - TTS speaker is `anand` for every language. Verified to work in all ten, but a per-language voice would sound better. → **UX improvement**, PARKED. - No barge-in: speaking over the assistant does not interrupt it, the user has to tap Stop. → **UX improvement**, PARKED (needs continuous capture). +- `book_taxi.json`'s `pick_ride`/`destination` steps match Uber/Ola/Rapido by + hardcoded English button text (`text_any`), with no fallback if any of the + three apps change their copy. → tracked as a GitHub issue, not fixed here — + it needs a real design decision (semantic matching? OCR? per-app adapters?), + not a quick patch. See issues labeled `audit-followup`. ## Explicitly out of scope (from the reference app, deliberately dropped) diff --git a/scripts/planner_case.ps1 b/scripts/planner_case.ps1 index bc57ae2..aad51cd 100644 --- a/scripts/planner_case.ps1 +++ b/scripts/planner_case.ps1 @@ -45,7 +45,7 @@ function J([string]$s) { $sb.ToString() } -$body = '{"model":"sarvam-30b","messages":[{"role":"system","content":"' + (J $prompt) + '"},{"role":"user","content":"' + (J $user) + '"}],' + +$body = '{"model":"sarvam-105b","messages":[{"role":"system","content":"' + (J $prompt) + '"},{"role":"user","content":"' + (J $user) + '"}],' + '"tools":[{"type":"function","function":{"name":"set_plan","description":"Set the next guided step and the element to point at.","parameters":' + '{"type":"object","properties":{"intent":{"type":"string"},"step":{"type":"string","enum":["amount","account","submit"]},' + '"target":{"type":"object","properties":{"resource_id":{"type":"string"},"index":{"type":"integer"}},"required":["resource_id","index"]},' +