diff --git a/README.md b/README.md index ffab7e40..fc456b1f 100644 --- a/README.md +++ b/README.md @@ -11,11 +11,11 @@ Mural is a native iPhone and Android app for learning through conversation. Speak to a warm, animated orb, follow the meaning when you need it, and practise words again in later conversations. Mural adjusts the challenge from the evidence in your replies. -Built with SwiftUI and Liquid Glass on iPhone, and Jetpack Compose on Android. Learning records stay on your device. This version connects directly to OpenAI using your own API key. It needs an internet connection, but no Mural account or running Mac. +Built with SwiftUI and Liquid Glass on iPhone, and Jetpack Compose on Android. Learning records stay on your device. This version connects directly to OpenAI or Google AI Studio using your own API key. It needs an internet connection, but no Mural account or running Mac. ## Android -A native Android client is available in [`apps/android/`](apps/android/README.md), with voice and written conversation, the same eight language modules, local learning records and iPhone-compatible JSON backups. Its interface is English, and Spanish on a phone set to Spanish. It runs on Android 8.0 or later and uses your own OpenAI API key stored with Android Keystore. The iPhone client remains available below. +A native Android client is available in [`apps/android/`](apps/android/README.md), with voice and written conversation, the same nine language modules, local learning records and iPhone-compatible JSON backups. Its interface is English, and Spanish on a phone set to Spanish. It runs on Android 8.0 or later and supports OpenAI or Google AI Studio keys stored with Android Keystore. The iPhone client remains available below. See the [Android installation/build guide](docs/run-on-android.md) and [Android verification record](verification/android-validation.md). Build a personal-install APK with Java 17 and Android SDK 36: @@ -26,7 +26,7 @@ cd apps/android ## Get started -You need a Mac with Xcode 26 or later, an iPhone running iOS 26.1 or later, an Apple Account, and an OpenAI API project with billing and access to GPT-Live-1 and GPT-5.6 Luna. A ChatGPT subscription does not provide API credit. +You need a Mac with Xcode 26 or later, an iPhone running iOS 26.1 or later, an Apple Account, and either an OpenAI API project with billing or a Google AI Studio API key. A ChatGPT subscription does not provide API credit. ### Install with a local AI agent @@ -49,8 +49,8 @@ Detect my connected iPhone, build with the configured signing team, install Mural and launch it. Tell me when I need to unlock the phone, trust this Mac or the developer profile, enable Developer Mode, or approve a system prompt. -I will choose my learning and subtitle languages, then enter my own OpenAI -API key in Settings > Advanced > Use your own API key. Do not ask me to paste +I will choose my learning and subtitle languages, then choose OpenAI or Google AI Studio +and enter its API key in Settings > Advanced > Use your own API key. Do not ask me to paste the key into chat, read it from Keychain, or put it in source files or logs. Leave managed accounts, hosted trials and purchases disabled. @@ -67,7 +67,7 @@ Updating an earlier checkout? The iPhone project now lives in `apps/ios/`. Befor 3. Select the **Mural** target, open **Signing & Capabilities**, enable automatic signing, and choose your team. For your own fork, replace the bundle identifier with a unique value such as `com.yourname.mural`. Keep that value stable for later updates. 4. Connect and unlock your iPhone. Trust the Mac if prompted. Turn on **Settings → Privacy & Security → Developer Mode** on the phone, restart, and confirm the setting. 5. Select **Mural** as the scheme and your iPhone as the destination, then click **Run**. If iOS asks you to trust the developer, do so in **Settings → General → VPN & Device Management**. -6. Choose your learning and subtitle languages in the welcome screens. In **Settings → Advanced → Use your own API key**, save your own OpenAI project key. Start a conversation and allow microphone access. +6. Choose your learning and subtitle languages in the welcome screens. In **Settings → Advanced → Use your own API key**, choose OpenAI or Google AI Studio and save that provider’s key. Start a conversation and allow microphone access. You should hear Mural greet you in your chosen language. You can now disconnect your phone from the Mac and use Wi-Fi or cellular. @@ -83,17 +83,17 @@ A free Personal Team can run the app on your own phone; TestFlight and App Store - **A fresh start:** the Talk screen returns to its greeting 15 seconds after a conversation ends. Tap **New conversation** to reset immediately. Your saved conversations and learning remain. - **Local records:** export or import a JSON learning backup, delete a conversation, or delete all learning data from Settings. -The modules teach Norwegian Bokmål with an Eastern Norwegian voice target, Spanish from Spain, international English, French from France, German from Germany, Italian from Italy, Brazilian Portuguese and Standard Mandarin with Simplified Chinese. Each language has its own conversation themes, teaching guidance and progress. Valid regional alternatives are accepted. +The modules teach Norwegian Bokmål with an Eastern Norwegian voice target, Spanish from Spain, international English, French from France, German from Germany, Italian, Brazilian Portuguese, Standard Mandarin with Simplified Chinese, and Thai. Each language has its own conversation themes, teaching guidance and progress. Valid regional alternatives are accepted. On iPhone, Mandarin includes optional pinyin in Talk, transcripts and word details. Chinese word lookup uses word boundaries, and the original characters remain available for copying from transcripts. Pinyin uses system dictionary readings; names, ambiguous words and tone changes in connected speech still need listening checks. Voice accent and teaching guidance are model instructions, and fluent-speaker review is still needed before making pronunciation or learning-effectiveness claims. ## Privacy and API costs -Mural stores conversations, vocabulary and preferences on your device. There is no Mural cloud sync, analytics SDK or advertising. The optional iPhone account feature stores signup data on the account service; conversations and vocabulary stay local. Your API key is stored in the device’s Keychain, excluded from learning exports, and sent only to OpenAI. +Mural stores conversations, vocabulary and preferences on your device. There is no Mural cloud sync, analytics SDK or advertising. The optional iPhone account feature stores signup data on the account service; conversations and vocabulary stay local. The selected provider’s API key is stored in protected platform credential storage (Apple Keychain on iPhone and Android Keystore/private encrypted preferences on Android), excluded from learning exports, and sent only to that provider. -During practice, audio, selected conversation text, learning context and requested searches go to OpenAI. Mural does not save raw audio. API requests set `store: false` where supported, but that does not disable all provider retention; OpenAI’s abuse-monitoring rules and your project’s settings still apply. [OpenAI data controls](https://developers.openai.com/api/docs/guides/your-data) +During practice, audio, selected conversation text, learning context and requested searches go to the selected provider. Mural does not save raw audio. OpenAI requests set `store: false`; Google requests use Google AI Studio’s `generateContent` and Live API directly. Provider retention, abuse-monitoring and project settings still apply. See [OpenAI data controls](https://developers.openai.com/api/docs/guides/your-data) and [Google’s Gemini API privacy guidance](https://ai.google.dev/gemini-api/docs/usage-policies). -OpenAI bills your project for voice, text and search. The app’s usage display is an estimate, and its conversation time limit is not a billing cap. Check your OpenAI project’s usage and spending settings. +The provider bills your project for voice, text and search. The app’s usage display is an estimate, and its conversation time limit is not a billing cap. Check the selected provider’s usage and spending settings. ## Planned public service diff --git a/apps/android/README.md b/apps/android/README.md index 4626c2af..13eec3df 100644 --- a/apps/android/README.md +++ b/apps/android/README.md @@ -1,8 +1,8 @@ # Mural for Android -Native Kotlin and Jetpack Compose client for Android 8.0 or later. It offers voice and written conversation, eight learning languages, 24 themes, meanings, vocabulary and local history, using the owner's OpenAI API key. Mandarin captions link each word and show optional pinyin on Android 10 or later; Android 8 and 9 keep word links without pinyin. +Native Kotlin and Jetpack Compose client for Android 8.0 or later. It offers voice and written conversation, nine learning languages, 24 themes, meanings, vocabulary and local history. OpenAI remains the default, and Google AI Studio can be selected in Settings. Mandarin captions link each word and show optional pinyin on Android 10 or later; Android 8 and 9 keep word links without pinyin. -Word taps in all eight languages open a contextual meaning sheet, matching the iOS flow. Mandarin uses a bundled offline phrase dictionary with an ICU fallback; unresolved common ambiguous readings retain their source characters. +Word taps in all nine languages open a contextual meaning sheet, matching the iOS flow. Mandarin uses a bundled offline phrase dictionary with an ICU fallback; unresolved common ambiguous readings retain their source characters. Interface copy lives in `res/values` (English) and `res/values-es` (Spanish); Android picks the translation from the phone's language. diff --git a/apps/android/app/src/androidTest/java/chat/mural/CaptionParityTest.kt b/apps/android/app/src/androidTest/java/chat/mural/CaptionParityTest.kt index 9b69f1a9..724e27d0 100644 --- a/apps/android/app/src/androidTest/java/chat/mural/CaptionParityTest.kt +++ b/apps/android/app/src/androidTest/java/chat/mural/CaptionParityTest.kt @@ -71,7 +71,7 @@ class CaptionParityTest { field.set(vm, APIClient("fixture-only", client, "https://offline.invalid/v1/".toHttpUrl())) state("hasKey", true) vm.updatePreferences(vm.archive.preferences.copy(learningLanguageID = "zh", meaningLanguage = "English", - meaningVisible = true, hasOnboarded = true, aiConsentVersion = 1)) + meaningVisible = true, hasOnboarded = true, aiConsentVersion = AI_CONSENT_VERSION)) } } @Suppress("UNCHECKED_CAST") diff --git a/apps/android/app/src/androidTest/java/chat/mural/ConversationPolicyTest.kt b/apps/android/app/src/androidTest/java/chat/mural/ConversationPolicyTest.kt index 1824f31b..30578847 100644 --- a/apps/android/app/src/androidTest/java/chat/mural/ConversationPolicyTest.kt +++ b/apps/android/app/src/androidTest/java/chat/mural/ConversationPolicyTest.kt @@ -23,7 +23,7 @@ class ConversationPolicyTest { vm = compose.awaitHistoryLoaded() compose.runOnIdle { original = ArchiveCodec.encode(vm.archive) - vm.updatePreferences(vm.archive.preferences.copy(hasOnboarded = true, aiConsentVersion = 1)) + vm.updatePreferences(vm.archive.preferences.copy(hasOnboarded = true, aiConsentVersion = AI_CONSENT_VERSION)) } } @Suppress("UNCHECKED_CAST") private fun state(name: String, value: Any?) { diff --git a/apps/android/app/src/androidTest/java/chat/mural/StartupErrorTest.kt b/apps/android/app/src/androidTest/java/chat/mural/StartupErrorTest.kt index 9241aef3..89e3b4c4 100644 --- a/apps/android/app/src/androidTest/java/chat/mural/StartupErrorTest.kt +++ b/apps/android/app/src/androidTest/java/chat/mural/StartupErrorTest.kt @@ -29,7 +29,7 @@ class StartupErrorTest { AccountViewModel::class.java.getDeclaredField("configuration").apply { isAccessible = true } .set(account, requireNotNull(ManagedAccountConfiguration.parse("https://offline.invalid", "123-offline.apps.googleusercontent.com"))) preferences = vm.archive.preferences - vm.updatePreferences(preferences.copy(hasOnboarded = true, aiConsentVersion = 1)) + vm.updatePreferences(preferences.copy(hasOnboarded = true, aiConsentVersion = chat.mural.core.AI_CONSENT_VERSION)) } } @After fun restore() { diff --git a/apps/android/app/src/androidTest/java/chat/mural/network/HanReadingsDeviceTest.kt b/apps/android/app/src/androidTest/java/chat/mural/network/HanReadingsDeviceTest.kt index 06d522da..1a35ae7e 100644 --- a/apps/android/app/src/androidTest/java/chat/mural/network/HanReadingsDeviceTest.kt +++ b/apps/android/app/src/androidTest/java/chat/mural/network/HanReadingsDeviceTest.kt @@ -57,4 +57,11 @@ class HanReadingsDeviceTest { for (word in listOf("行", "重", "还", "教")) assertNull(word, reader.reading(word)) assertEquals(listOf("教", "我", "中文"), CaptionWords.segments("教我中文", "zh", reader).mapNotNull { it.lookup }) } + + @Test fun thaiCaptionLinksUseIcuWordBoundaries() { + val text = "ฉันอยากไปตลาด" + val words = reader.languageWords(text, "th") ?: error("Thai ICU segmentation unavailable") + assertEquals(text, words.joinToString("")) + assertEquals(listOf("ฉัน", "อยาก", "ไป", "ตลาด"), CaptionWords.segments(text, "th", reader).mapNotNull { it.lookup }) + } } diff --git a/apps/android/app/src/androidTest/java/chat/mural/network/NativeCompatibilityTest.kt b/apps/android/app/src/androidTest/java/chat/mural/network/NativeCompatibilityTest.kt index 3a4a5b7d..e6599461 100644 --- a/apps/android/app/src/androidTest/java/chat/mural/network/NativeCompatibilityTest.kt +++ b/apps/android/app/src/androidTest/java/chat/mural/network/NativeCompatibilityTest.kt @@ -18,14 +18,21 @@ class NativeCompatibilityTest { val namespace = UUID.randomUUID().toString() val store = CredentialStore(context, "mural_test_$namespace", "chat.mural.test.$namespace") val fake = "sk-offline-test-credential-never-sent" + val google = "AIza-offline-test-credential-never-sent" try { store.save(fake) + store.save(google, AIProvider.GOOGLE_AI_STUDIO) val raw = context.getSharedPreferences("mural_test_$namespace", Context.MODE_PRIVATE).all.values.joinToString() assertFalse(raw.contains(fake)) + assertFalse(raw.contains(google)) assertTrue(store.hasKey) assertEquals(fake, CredentialStore(context, "mural_test_$namespace", "chat.mural.test.$namespace").read()) + assertEquals(google, store.read(AIProvider.GOOGLE_AI_STUDIO)) assertThrows(CredentialStore.CredentialException.Invalid::class.java) { store.save("bad") } assertEquals(fake, store.read()) + store.delete(AIProvider.GOOGLE_AI_STUDIO) + assertFalse(store.hasKey(AIProvider.GOOGLE_AI_STUDIO)) + assertEquals(fake, store.read()) store.delete(); assertFalse(store.hasKey) } finally { store.delete() } } diff --git a/apps/android/app/src/main/java/chat/mural/MuralViewModel.kt b/apps/android/app/src/main/java/chat/mural/MuralViewModel.kt index b9ce107e..311375ee 100644 --- a/apps/android/app/src/main/java/chat/mural/MuralViewModel.kt +++ b/apps/android/app/src/main/java/chat/mural/MuralViewModel.kt @@ -126,6 +126,7 @@ class MuralViewModel(application: Application) : AndroidViewModel(application) { private var lookupJob: Job? = null var topicResult by mutableStateOf(null); private set var hasKey by mutableStateOf(false); private set + var aiProvider by mutableStateOf(AIProviderSelection.read(application)); private set val language get() = LanguageRegistry.get(archive.preferences.learningLanguageID)!! val learner get() = LearningEngine.project(archive.sessions, language.id, archive.preferences.hiddenWords) val isRunning get() = state in listOf("connecting", "active", "closing") @@ -134,7 +135,7 @@ class MuralViewModel(application: Application) : AndroidViewModel(application) { private val json = Json { ignoreUnknownKeys = true; encodeDefaults = true } private val repository = LearningRepository(application) private val credentials = CredentialStore(application) - private val api = APIClient(credentials) + private val api = APIClient(credentials).also { it.provider = aiProvider } private val transport = LiveTransport(application, viewModelScope) private val providerStore = ConversationProviderStore(application) private val hostedConfiguration = HostedConfiguration.parse(BuildConfig.MANAGED_API_ORIGIN) @@ -201,7 +202,7 @@ class MuralViewModel(application: Application) : AndroidViewModel(application) { private var actionJob: Job? = null private val meanings = MeaningController(viewModelScope, canRetryFailure = HostedHelperRetry::canRetryAtBoundary, retryDelay = HostedHelperRetry::automaticDelay) { request -> - if (archive.preferences.aiConsentVersion != 1) throw IllegalStateException("AI processing consent is required.") + if (archive.preferences.aiConsentVersion != AI_CONSENT_VERSION) throw IllegalStateException("AI processing consent is required.") val module = LanguageRegistry.get(request.learningLanguageID) ?: throw IllegalStateException("Unsupported language.") if (request.sessionID in hostedSessionIDs && request.translationInput.toByteArray(Charsets.UTF_8).size > 24_576) throw MeaningInputLimitException() val result = teaching(request.sessionID, HelperPurpose.MEANING, request.cacheKey, @@ -225,7 +226,7 @@ class MuralViewModel(application: Application) : AndroidViewModel(application) { guests?.let { controller -> viewModelScope.launch { controller.state.collect { guestState = it } } } viewModelScope.launch { try { - val loaded = withContext(Dispatchers.IO) { repository.load() to credentials.hasKey } + val loaded = withContext(Dispatchers.IO) { repository.load() to credentials.hasKey(aiProvider) } archive = loaded.first.archive val providers = providerStore.read(if (loaded.second) ConversationProvider.PERSONAL_KEY else ConversationProvider.HOSTED_MINUTES) hostedSessionIDs = providers.hostedIDs @@ -291,7 +292,7 @@ class MuralViewModel(application: Application) : AndroidViewModel(application) { val current = archive.sessions.firstOrNull { it.id == snapshot.id } val ticket = finalAssessmentTickets.firstOrNull { it.matches(snapshot, passage) } if (snapshot.id in hostedSessionIDs) false - else if (!storageReady || !hasKey || archive.preferences.aiConsentVersion != 1 || current == null || + else if (!storageReady || !hasKey || archive.preferences.aiConsentVersion != AI_CONSENT_VERSION || current == null || ticket == null || !FinalAssessmentRecovery.canAttempt(ticket, current, nowSeconds())) false else { finalAssessmentTickets = finalAssessmentTickets.map { @@ -311,7 +312,7 @@ class MuralViewModel(application: Application) : AndroidViewModel(application) { } private fun recoverFinalAssessments() { - if (!storageReady || !hasKey || archive.preferences.aiConsentVersion != 1) return + if (!storageReady || !hasKey || archive.preferences.aiConsentVersion != AI_CONSENT_VERSION) return val remaining = FinalAssessmentRecovery.MAX_RECOVERED_PER_LAUNCH - recoveredAssessmentIDs.size if (remaining <= 0) return FinalAssessmentRecovery.recover(archive.sessions, finalAssessmentTickets, nowSeconds()) @@ -369,7 +370,7 @@ class MuralViewModel(application: Application) : AndroidViewModel(application) { } private fun cloudReady(): Boolean { if (!storageReady) { presentError(getApplication().getString(R.string.error_resolve_local_history_first)); return false } - if (archive.preferences.aiConsentVersion != 1) { + if (archive.preferences.aiConsentVersion != AI_CONSENT_VERSION) { presentError(getApplication().getString(R.string.error_accept_ai_consent)); return false } val currentHosted = session?.id in hostedSessionIDs @@ -412,7 +413,7 @@ class MuralViewModel(application: Application) : AndroidViewModel(application) { readiness.selectAccount(member.accountID, false) readiness.refresh() } else { - if (archive.preferences.aiConsentVersion != 1 || conversationProvider != ConversationProvider.HOSTED_MINUTES) { + if (archive.preferences.aiConsentVersion != AI_CONSENT_VERSION || conversationProvider != ConversationProvider.HOSTED_MINUTES) { readiness.selectAccount(null, false); return@launch } guests?.acquire() @@ -495,7 +496,7 @@ class MuralViewModel(application: Application) : AndroidViewModel(application) { /** The creating session, rather than the currently selected settings option, chooses every helper. */ private suspend fun teaching(localID: String?, purpose: HelperPurpose, logicalID: String, instructions: String, input: String, schema: JsonObject? = null, search: Boolean = false): APIResult { - if (archive.preferences.aiConsentVersion != 1) throw HostedFailure.Unavailable + if (archive.preferences.aiConsentVersion != AI_CONSENT_VERSION) throw HostedFailure.Unavailable if (localID != null && localID in hostedSessionIDs) { return hostedBindings.respond(localID, purpose, logicalID, instructions, input, schema, search) } @@ -605,21 +606,30 @@ class MuralViewModel(application: Application) : AndroidViewModel(application) { } fun saveKey(key: String) { if (isRunning) return - try { credentials.save(key); hasKey = credentials.hasKey; selectConversationProvider(ConversationProvider.PERSONAL_KEY); recoverFinalAssessments(); notice = getApplication().getString(R.string.notice_key_saved) } + try { credentials.save(key, aiProvider); hasKey = credentials.hasKey(aiProvider); selectConversationProvider(ConversationProvider.PERSONAL_KEY); recoverFinalAssessments(); notice = getApplication().getString(R.string.notice_key_saved) } catch (e: Exception) { presentError(e, R.string.error_key_save_failed) } } fun deleteKey() { if (isRunning) return - try { credentials.delete(); hasKey = false } + try { credentials.delete(aiProvider); hasKey = false } catch (e: Exception) { presentError(e, R.string.error_key_delete_failed) } - finally { hasKey = credentials.hasKey } + finally { hasKey = credentials.hasKey(aiProvider) } + } + fun selectAIProvider(provider: AIProvider) { + if (isRunning || aiProvider == provider) return + aiProvider = provider + api.provider = provider + AIProviderSelection.save(getApplication(), provider) + hasKey = credentials.hasKey(provider) + if (storageReady) updatePreferences(archive.preferences.copy(aiConsentVersion = null)) + dismissError() } fun updatePreferences(preferences: Preferences) { if (isRunning || !storageReady) return if (LanguageRegistry.get(preferences.learningLanguageID) == null || preferences.meaningLanguage !in MeaningLanguages.all || preferences.sessionMinutes !in 1..60) return val languageChanged = preferences.learningLanguageID != language.id actionJob?.cancel(); clearLookup(); working = false - if (preferences.aiConsentVersion != 1) { + if (preferences.aiConsentVersion != AI_CONSENT_VERSION) { finalAssessments.cancelAll(); hostedBindings.disableHelpers() hostedFinalAssessmentJobs.values.toList().forEach { it.cancel() }; hostedFinalAssessmentJobs.clear() } @@ -754,6 +764,7 @@ class MuralViewModel(application: Application) : AndroidViewModel(application) { connectionJob?.cancel(); durationJob?.cancel(); closeJob?.cancel(); assessmentJob?.cancel() actionJob?.cancel(); clearLookup(); languageCheckJob?.cancel() delegations.values.toList().forEach { it.cancel() }; delegations.clear() + if (session?.providerID == "gemini-live") updateSession { it.voiceSeconds = (nowSeconds() - it.startedAt).coerceAtLeast(0.0) } transport.disconnect(); inputLevel = 0.0; outputLevel = 0.0; working = false; isMuted = false updateSession { it.endedAt = nowSeconds(); it.usageFinal = final } state = "ended" @@ -791,7 +802,10 @@ class MuralViewModel(application: Application) : AndroidViewModel(application) { private fun handle(event: JsonObject) { if (session == null || !isRunning) return when (event["type"]?.jsonPrimitive?.content) { - "mural.session.created" -> updateSession { it.providerID = (event["session"] as? JsonObject)?.get("id")?.jsonPrimitive?.content; it.voiceSeconds = 15.0 } + "mural.session.created" -> updateSession { + it.providerID = (event["session"] as? JsonObject)?.get("id")?.jsonPrimitive?.content + if (it.providerID != "gemini-live") it.voiceSeconds = 15.0 + } "session.started" -> if (state == "connecting") { state = "active"; activity = ConversationActivity(activityNow()); conversationPace = ConversationPace(); inactivitySeconds = null updateSession { it.providerID = (event["session"] as? JsonObject)?.get("id")?.jsonPrimitive?.content ?: it.providerID } @@ -851,7 +865,7 @@ class MuralViewModel(application: Application) : AndroidViewModel(application) { s.searchCalls = (s.searchCalls.toLong() + usage.searches).coerceAtMost(1_000_000_000).toInt() } private fun scheduleTranslation(utteranceComplete: Boolean = state == "ended") { - if (!archive.preferences.meaningVisible || archive.preferences.aiConsentVersion != 1) return + if (!archive.preferences.meaningVisible || archive.preferences.aiConsentVersion != AI_CONSENT_VERSION) return val current = session ?: return val passage = current.passages.lastOrNull { it.speaker == Speaker.assistant } ?: return val request = MeaningRequest(current.id, passage, current.languageID, archive.preferences.meaningLanguage) @@ -894,7 +908,7 @@ class MuralViewModel(application: Application) : AndroidViewModel(application) { )) } private suspend fun requestAssessment(snapshot: SessionRecord, passage: Passage): FinalAssessmentResult { - if (archive.preferences.aiConsentVersion != 1) throw IllegalStateException("AI processing consent is required.") + if (archive.preferences.aiConsentVersion != AI_CONSENT_VERSION) throw IllegalStateException("AI processing consent is required.") val module = LanguageRegistry.get(snapshot.languageID) ?: throw IllegalStateException("Unsupported language.") val result = teaching(snapshot.id, HelperPurpose.ASSESSMENT, passage.revisionKey, TeachingPolicy.assessment(module), helperContext(snapshot, passage), assessmentSchema(module.id)) @@ -910,7 +924,7 @@ class MuralViewModel(application: Application) : AndroidViewModel(application) { try { withTimeout(HostedConversationBindings.POST_END_MILLIS) { val result = requestAssessment(snapshot, passage) - if (archive.preferences.aiConsentVersion != 1) return@withTimeout + if (archive.preferences.aiConsentVersion != AI_CONSENT_VERSION) return@withTimeout result.applying(archive.sessions.firstOrNull { it.id == result.sessionID })?.let { updated -> save(updated) if (session?.id == updated.id) session = clone(updated) @@ -962,7 +976,7 @@ class MuralViewModel(application: Application) : AndroidViewModel(application) { val result = teaching(snapshot.id, HelperPurpose.DELEGATION, id, TeachingPolicy.delegation(language), helperContext(snapshot), search = snapshot.searchCalls < 3) if (state != "active" || session?.id != sessionID) return@launch - updateSession { addUsage(it, result.usage); if (result.sources.isNotEmpty()) it.topics += TopicBrief(languageID = it.languageID, query = getApplication().getString(R.string.topics_from_conversation), text = result.text, sources = result.sources) } + updateSession { addUsage(it, result.usage); if (result.sources.isNotEmpty()) it.topics += TopicBrief(languageID = it.languageID, query = getApplication().getString(R.string.topics_from_conversation), text = result.text, sources = result.sources, searchEntryPointHTML = result.searchEntryPointHTML) } command("commentary", result.text, id) } catch (_: CancellationException) { } catch (_: Exception) { if (session?.id == sessionID) command("commentary", language.lookupUnavailableReply, id) } @@ -1042,7 +1056,7 @@ class MuralViewModel(application: Application) : AndroidViewModel(application) { val result = teaching(session?.id, HelperPurpose.TOPIC, UUID.randomUUID().toString(), TeachingPolicy.currentTopic(module), query.trim().take(500), search = true) if (token != generation) return@launch if (result.sources.isEmpty()) { presentError(getApplication().getString(R.string.error_topic_unsourced)); return@launch } - val brief = TopicBrief(languageID = module.id, query = query.trim().take(500), text = result.text, sources = result.sources) + val brief = TopicBrief(languageID = module.id, query = query.trim().take(500), text = result.text, sources = result.sources, searchEntryPointHTML = result.searchEntryPointHTML) topicResult = brief if (isRunning) updateSession { it.topics += brief; addUsage(it, result.usage) } else { val record = SessionRecord(languageID = module.id, title = brief.query, endedAt = nowSeconds()); record.topics += brief; addUsage(record, result.usage); save(record) } diff --git a/apps/android/app/src/main/java/chat/mural/core/Languages.kt b/apps/android/app/src/main/java/chat/mural/core/Languages.kt index 8a7292af..9ad638a7 100644 --- a/apps/android/app/src/main/java/chat/mural/core/Languages.kt +++ b/apps/android/app/src/main/java/chat/mural/core/Languages.kt @@ -31,6 +31,7 @@ object Themes { val shared = listOf( data class LanguageModule( val id: String, val name: String, val nativeName: String, val variety: String, val locale: String, + val usesPlatformWordSegmentation: Boolean = false, val greeting: String, val greetingWord: String, val speechGuidance: String, val writingGuidance: String, val lemmaGuidance: String, val teachingFocus: List, val topicPlaceholder: String, val lookupUnavailableReply: String, val themeOverrides: Map = emptyMap() @@ -49,6 +50,7 @@ object LanguageRegistry { nativeName = "Norsk", variety = "Bokmål", locale = "nb-NO", + usesPlatformWordSegmentation = false, greeting = "Hei!", greetingWord = "hei", speechGuidance = "Use natural Eastern Norwegian pronunciation. Accept other Norwegian dialects without treating dialect differences as errors.", @@ -69,6 +71,7 @@ object LanguageRegistry { nativeName = "Español", variety = "Spain", locale = "es-ES", + usesPlatformWordSegmentation = false, greeting = "¡Hola!", greetingWord = "hola", speechGuidance = "Use clear Spanish from Spain, with a natural distinction between s and z/soft c, tú for friendly singular address and vosotros for informal plural address. Accept seseo, ustedes, voseo and other valid regional forms without marking them wrong. Do not imitate a regional caricature.", @@ -89,6 +92,7 @@ object LanguageRegistry { nativeName = "English", variety = "International", locale = "en", + usesPlatformWordSegmentation = false, greeting = "Hi!", greetingWord = "hi", speechGuidance = "Use clear, broadly intelligible English with a consistent, natural pronunciation. Accept valid regional accents, vocabulary and grammar, including British and American forms. Do not treat an accent difference as an error or require imitation of a native accent. Correct pronunciation only when meaning is unclear and the audio supports the correction.", @@ -107,6 +111,7 @@ object LanguageRegistry { nativeName = "Français", variety = "France", locale = "fr-FR", + usesPlatformWordSegmentation = false, greeting = "Salut !", greetingWord = "salut", speechGuidance = "Use clear, natural metropolitan French pronunciation. Use tu in a friendly conversation and vous when the situation calls for formality or plural address. Accept valid regional accents, vocabulary and grammar from across the French-speaking world. Do not treat regional variation, informal omission of ne or a non-native accent alone as an error. Do not imitate a regional caricature.", @@ -127,6 +132,7 @@ object LanguageRegistry { nativeName = "Deutsch", variety = "Germany", locale = "de-DE", + usesPlatformWordSegmentation = false, greeting = "Hallo!", greetingWord = "hallo", speechGuidance = "Use clear, natural Standard German as spoken in Germany. Use du for friendly conversation and Sie when the situation calls for formality. Accept valid Austrian, Swiss and other regional pronunciation, vocabulary and grammar. Do not treat a regional difference or a non-native accent alone as an error. Correct pronunciation only when supported by the audio, not a transcript alone.", @@ -147,6 +153,7 @@ object LanguageRegistry { nativeName = "Italiano", variety = "Italy", locale = "it-IT", + usesPlatformWordSegmentation = false, greeting = "Ciao!", greetingWord = "ciao", speechGuidance = "Use clear, natural Standard Italian pronunciation. Use tu for friendly conversation and Lei when the situation calls for formality. Model vowel sounds, word stress and consonant length naturally. Accept valid regional accents and vocabulary without treating regional variation or a non-native accent alone as an error. Do not infer a pronunciation error from spelling alone.", @@ -167,6 +174,7 @@ object LanguageRegistry { nativeName = "Português", variety = "Brazil", locale = "pt-BR", + usesPlatformWordSegmentation = false, greeting = "Olá!", greetingWord = "olá", speechGuidance = "Use clear, natural Brazilian Portuguese with broadly intelligible pronunciation and consistent Brazilian vocabulary. Use você in friendly conversation and formal address when appropriate. Accept valid uses of tu, regional Brazilian accents and grammar, and European, African and other Portuguese varieties without marking them wrong. Do not imitate a regional caricature or infer pronunciation errors from a transcript alone.", @@ -187,6 +195,7 @@ object LanguageRegistry { nativeName = "普通话", variety = "Mainland China", locale = "zh-CN", + usesPlatformWordSegmentation = false, greeting = "你好!", greetingWord = "你好", speechGuidance = "Use clear, natural Standard Mandarin pronunciation. Treat tones, tone changes, retroflex and non-retroflex sounds, and distinctions between initials and finals as meaningful when they affect understanding. Accept valid regional accents and vocabulary without treating a regional difference or a non-native accent alone as an error. Do not imitate a regional caricature.", @@ -201,11 +210,32 @@ object LanguageRegistry { "cabin" to ConversationTheme("cabin", "周末出游", "A change of scene", "mountain.2", "Local life", "一起设想一个周末旅行,选择城市、海边或乡村,讨论实际安排和喜欢做的事情。", 2), "traditions" to ConversationTheme("traditions", "日常习俗", "Small customs, big stories", "flag", "Local life", "用普通话聊日常习俗和节日。比较学习者熟悉的地方,避免把任何一种习惯说成所有人的共同体验。", 2)) ) - val all = listOf(norwegian, spanish, english, french, german, italian, portuguese, mandarin) + private val thai = LanguageModule( + id = "th", + name = "Thai", + nativeName = "ภาษาไทย", + variety = "Thailand", + locale = "th-TH", + usesPlatformWordSegmentation = true, + greeting = "สวัสดี!", + greetingWord = "สวัสดี", + speechGuidance = "Use clear, natural Central Thai pronunciation. Treat tones, vowel length, final consonants and polite particles as meaningful when they affect understanding. Accept valid regional accents and vocabulary without treating a regional difference or a non-native accent alone as an error. Do not imitate a regional caricature.", + writingGuidance = "Use natural modern Thai with Thai script and appropriate spacing between phrases. Keep polite particles and pronouns natural to the learner’s context. Do not append transliteration or translations to ordinary spoken replies; explain pronunciation briefly only when asked.", + lemmaGuidance = "Give vocabulary in natural Thai script and preserve the exact observed form and quote. Prefer useful dictionary forms or everyday chunks, and include polite particles only when they are part of the meaningful expression. Do not infer tone or pronunciation accuracy from typed text alone.", + topicPlaceholder = "อาหาร การเดินทาง เพลง ชีวิตประจำวัน…", + lookupUnavailableReply = "ตอนนี้ฉันยังตรวจสอบเรื่องนี้ไม่ได้ ถ้าคุณต้องการ เราคุยเกี่ยวกับภาพรวมของหัวข้อนี้ก่อนได้นะ", + teachingFocus = listOf("Greetings, introductions, names, polite particles and useful everyday chunks such as ผมชื่อ and ฉันอยากได้.", "Everyday questions, classifiers, numbers, food and short present-time exchanges.", "Connected stories, completed actions, time expressions and familiar situations.", "Reasons and opinions, comparisons, requests and natural linking phrases.", "Nuance, conditionals, register, particles and regional variation.", "Flexible advanced discussion with precise, natural Thai and context-appropriate politeness."), + themeOverrides = mapOf("coffee" to ConversationTheme("coffee", "ดื่มกาแฟกันไหม", "Something warm, please", "cup.and.saucer", "Everyday", "Meet in a neighbourhood café in Thailand. Order a drink and chat in Thai, following the learner’s interests.", 0), + "groceries" to ConversationTheme("groceries", "ไปตลาดกัน", "Find something good", "basket", "Everyday", "Shop for everyday ingredients at a Thai market or supermarket. Practise quantities, prices and polite questions while respecting regional food vocabulary.", 2), + "travel" to ConversationTheme("travel", "สถานีต่อไป", "A ticket to somewhere", "tram", "Everyday", "Plan an imagined trip in Thailand. Discuss transport, directions and tickets without inventing current schedules.", 1), + "cabin" to ConversationTheme("cabin", "เที่ยวสุดสัปดาห์", "A change of scene", "mountain.2", "Local life", "Imagine a weekend away in a city, by the sea or in the countryside. Discuss practical plans and things the learner enjoys.", 2), + "traditions" to ConversationTheme("traditions", "เรื่องเล็ก ๆ ในชีวิตประจำวัน", "Small customs, big stories", "flag", "Local life", "Talk in Thai about everyday customs and festivals. Compare experiences without treating any habit as universal.", 2)) + ) + val all = listOf(norwegian, spanish, english, french, german, italian, portuguese, mandarin, thai) fun get(id: String) = all.firstOrNull { it.id == id } } object MeaningLanguages { - val all = listOf("English", "French", "German", "Spanish", "Norwegian", "Portuguese", "Italian", "Chinese (Simplified)", "Polish", "Arabic", "Ukrainian") - fun greeting(language: String) = mapOf("English" to "Hi!", "French" to "Salut !", "German" to "Hallo!", "Spanish" to "¡Hola!", "Norwegian" to "Hei!", "Portuguese" to "Olá!", "Italian" to "Ciao!", "Chinese (Simplified)" to "你好!", "Chinese" to "你好!", "Polish" to "Cześć!", "Arabic" to "مرحبًا!", "Ukrainian" to "Привіт!")[language] ?: "Hi!" + val all = listOf("English", "French", "German", "Spanish", "Norwegian", "Portuguese", "Italian", "Chinese (Simplified)", "Thai", "Polish", "Arabic", "Ukrainian") + fun greeting(language: String) = mapOf("English" to "Hi!", "French" to "Salut !", "German" to "Hallo!", "Spanish" to "¡Hola!", "Norwegian" to "Hei!", "Portuguese" to "Olá!", "Italian" to "Ciao!", "Chinese (Simplified)" to "你好!", "Chinese" to "你好!", "Thai" to "สวัสดี!", "Polish" to "Cześć!", "Arabic" to "مرحبًا!", "Ukrainian" to "Привіт!")[language] ?: "Hi!" } diff --git a/apps/android/app/src/main/java/chat/mural/core/MandarinPinyin.kt b/apps/android/app/src/main/java/chat/mural/core/MandarinPinyin.kt index 6d162643..c374b8c5 100644 --- a/apps/android/app/src/main/java/chat/mural/core/MandarinPinyin.kt +++ b/apps/android/app/src/main/java/chat/mural/core/MandarinPinyin.kt @@ -9,6 +9,9 @@ interface HanReader { /** Splits text into consecutive pieces whose concatenation is the original text. */ fun words(text: String): List + /** Platform word boundaries for languages whose writing system normally omits spaces. */ + fun languageWords(text: String, languageID: String): List? = null + /** A Latin transcription of one word, or null when the platform has none. */ fun reading(word: String): String? @@ -86,6 +89,12 @@ object CaptionWords { val words = MandarinPinyin.words(text, reader) ?: return listOf(CaptionSegment(text, null)) return words.map { CaptionSegment(it, it.takeIf(::containsLetter)) } } + if (LanguageRegistry.get(languageID)?.usesPlatformWordSegmentation == true) { + val words = reader?.languageWords(languageID = languageID, text = text) + ?.takeIf { it.joinToString("") == text } + ?: return listOf(CaptionSegment(text, null)) + return words.map(::segment) + } val result = mutableListOf() val run = StringBuilder() for (character in text) { diff --git a/apps/android/app/src/main/java/chat/mural/core/Models.kt b/apps/android/app/src/main/java/chat/mural/core/Models.kt index 663690fc..3f5082c6 100644 --- a/apps/android/app/src/main/java/chat/mural/core/Models.kt +++ b/apps/android/app/src/main/java/chat/mural/core/Models.kt @@ -10,6 +10,7 @@ import kotlinx.serialization.json.* @Serializable enum class Outcome { success, partial, breakdown, uncertain } const val APPLE_EPOCH_UNIX_SECONDS = 978307200.0 +const val AI_CONSENT_VERSION = 2 fun nowSeconds(): Double = System.currentTimeMillis() / 1000.0 - APPLE_EPOCH_UNIX_SECONDS @Serializable @@ -104,7 +105,7 @@ data class SourceLink(val title: String, val url: String) { @Serializable data class TopicBrief( val id: String = UUID.randomUUID().toString(), val languageID: String, var query: String, - var text: String, var sources: List, val retrievedAt: Double = nowSeconds() + var text: String, var sources: List, var searchEntryPointHTML: String? = null, val retrievedAt: Double = nowSeconds() ) { val isFresh get() = nowSeconds() - retrievedAt < 6 * 3600 } @Serializable diff --git a/apps/android/app/src/main/java/chat/mural/network/AIProvider.kt b/apps/android/app/src/main/java/chat/mural/network/AIProvider.kt new file mode 100644 index 00000000..cd0379df --- /dev/null +++ b/apps/android/app/src/main/java/chat/mural/network/AIProvider.kt @@ -0,0 +1,31 @@ +package chat.mural.network + +import android.content.Context + +enum class AIProvider( + val displayName: String, + val helperModel: String, + val liveModel: String, + val keyUrl: String, + val usageUrl: String, + val dataControlsUrl: String, +) { + OPENAI("OpenAI", "gpt-5.6-luna", "gpt-live-1", "https://platform.openai.com/api-keys", "https://platform.openai.com/usage", "https://developers.openai.com/api/docs/guides/your-data"), + GOOGLE_AI_STUDIO("Google AI Studio", "gemma-4-31b-it", "gemini-3.8-live", "https://aistudio.google.com/app/apikey", "https://aistudio.google.com/app/usage", "https://ai.google.dev/gemini-api/docs/usage-policies"), +} + +object AIProviderSelection { + private const val PREFERENCES = "mural_ai_provider" + private const val PROVIDER = "provider" + + fun read(context: Context): AIProvider { + val raw = context.applicationContext.getSharedPreferences(PREFERENCES, Context.MODE_PRIVATE) + .getString(PROVIDER, null) + return AIProvider.entries.firstOrNull { it.name == raw } ?: AIProvider.OPENAI + } + + fun save(context: Context, provider: AIProvider) { + context.applicationContext.getSharedPreferences(PREFERENCES, Context.MODE_PRIVATE) + .edit().putString(PROVIDER, provider.name).apply() + } +} diff --git a/apps/android/app/src/main/java/chat/mural/network/APIClient.kt b/apps/android/app/src/main/java/chat/mural/network/APIClient.kt index a5686f90..bddff43e 100644 --- a/apps/android/app/src/main/java/chat/mural/network/APIClient.kt +++ b/apps/android/app/src/main/java/chat/mural/network/APIClient.kt @@ -10,9 +10,13 @@ import kotlinx.coroutines.suspendCancellableCoroutine import kotlinx.serialization.json.Json import kotlinx.serialization.json.JsonObject import kotlinx.serialization.json.JsonPrimitive +import kotlinx.serialization.json.JsonArray +import kotlinx.serialization.json.JsonElement import kotlinx.serialization.json.buildJsonArray import kotlinx.serialization.json.buildJsonObject import kotlinx.serialization.json.contentOrNull +import kotlinx.serialization.json.intOrNull +import kotlinx.serialization.json.jsonArray import kotlinx.serialization.json.jsonObject import kotlinx.serialization.json.jsonPrimitive import kotlinx.serialization.json.put @@ -28,18 +32,29 @@ import okhttp3.Response import okio.Buffer data class APIUsage(val input: Int = 0, val output: Int = 0, val searches: Int = 0) -data class APIResult(val text: String, val sources: List, val usage: APIUsage) +data class APIResult( + val text: String, + val sources: List, + val usage: APIUsage, + val searchEntryPointHTML: String? = null, +) class APIClient private constructor( - private val readCredential: () -> String?, + private val readCredential: (AIProvider) -> String?, private val client: OkHttpClient = defaultClient(), private val baseUrl: HttpUrl = API_BASE_URL, + private val googleBaseUrl: HttpUrl = GOOGLE_API_BASE_URL, ) : TeachingClient, LiveSessionProvider { - constructor(credentials: CredentialStore) : this(credentials::read) + constructor(credentials: CredentialStore) : this({ provider -> credentials.read(provider) }) + + var provider: AIProvider = AIProvider.OPENAI internal constructor(key: String?, client: OkHttpClient, baseUrl: HttpUrl) : this({ key }, client, baseUrl) + internal constructor(key: String?, client: OkHttpClient, baseUrl: HttpUrl, googleBaseUrl: HttpUrl) : + this({ key }, client, baseUrl, googleBaseUrl) + override suspend fun createLiveSession(request: LiveSessionRequest): LiveSessionConnection { val result = post("live/sessions", buildJsonObject { put("session", buildJsonObject { @@ -61,46 +76,8 @@ class APIClient private constructor( if (!VALID_PATH.matches(path) || path.contains("..") || path.startsWith('/')) { throw APIException.InvalidResponse } - val key = readCredential() ?: throw APIException.MissingKey - val request = Request.Builder() - .url(baseUrl.newBuilder().addPathSegments(path).build()) - .header("Authorization", "Bearer $key") - .header("Content-Type", JSON_MEDIA_TYPE.toString()) - .post(body.toString().toRequestBody(JSON_MEDIA_TYPE)) - .build() - - // Parse on OkHttp's worker while the continuation remains cancellable. - // Cancellation closes a response even if the peer stalls halfway through its body. - return suspendCancellableCoroutine { continuation -> - val call = client.newCall(request) - continuation.invokeOnCancellation { call.cancel() } - call.enqueue(object : Callback { - override fun onFailure(call: Call, error: IOException) { - if (continuation.isActive) continuation.resumeWithException(error) - } - override fun onResponse(call: Call, response: Response) { - try { - val value = response.use { - if (it.code !in 200..299) { - val errorCode = runCatching { - val payload = it.peekBody(16_385).string() - if (payload.toByteArray(Charsets.UTF_8).size > 16_384) null else - (JSON.parseToJsonElement(payload).jsonObject["error"] as? JsonObject) - ?.get("code")?.jsonPrimitive?.contentOrNull - }.getOrNull() - throw APIException.Http(it.code, errorCode, it.header("x-request-id")) - } - val payload = it.readBoundedBody() - try { JSON.parseToJsonElement(payload).jsonObject } - catch (_: Exception) { throw APIException.InvalidResponse } - } - if (continuation.isActive) continuation.resume(value) - } catch (error: Exception) { - if (continuation.isActive) continuation.resumeWithException(error) - } - } - }) - } + val key = readCredential(AIProvider.OPENAI) ?: throw APIException.MissingKey + return directPost(baseUrl.newBuilder().addPathSegments(path).build(), key, body, "OpenAI") } override suspend fun respond( @@ -110,6 +87,9 @@ class APIClient private constructor( search: Boolean, purpose: HelperPurpose?, ): APIResult { + if (provider == AIProvider.GOOGLE_AI_STUDIO) { + return geminiRespond(instructions, input, schema, search) + } val body = buildJsonObject { put("model", "gpt-5.6-luna") put("store", false) @@ -143,6 +123,108 @@ class APIClient private constructor( return decodeTeachingResponse(response) } + private suspend fun geminiRespond(instructions: String, input: String, schema: JsonObject?, search: Boolean): APIResult { + val key = readCredential(AIProvider.GOOGLE_AI_STUDIO) ?: throw APIException.MissingKey + val body = buildJsonObject { + put("systemInstruction", buildJsonObject { put("parts", buildJsonArray { add(buildJsonObject { put("text", instructions) }) }) }) + put("contents", buildJsonArray { add(buildJsonObject { + put("role", "user") + put("parts", buildJsonArray { add(buildJsonObject { put("text", input) }) }) + }) }) + put("generationConfig", buildJsonObject { + put("maxOutputTokens", if (schema == null) 1_400 else 2_200) + if (schema != null) { + put("responseMimeType", "application/json") + put("responseJsonSchema", geminiCompatibleSchema(schema)) + } + }) + if (search) put("tools", buildJsonArray { add(buildJsonObject { put("googleSearch", buildJsonObject { }) }) }) + } + val path = "models/${AIProvider.GOOGLE_AI_STUDIO.helperModel}:generateContent" + val response = directPost(googleBaseUrl.newBuilder().addPathSegments(path).build(), key, body, "Google AI Studio", google = true) + val candidate = response["candidates"]?.jsonArray?.firstOrNull()?.jsonObject ?: throw APIException.InvalidResponse + if (candidate["finishReason"]?.jsonPrimitive?.contentOrNull != "STOP") throw APIException.Incomplete + val text = buildString { + val parts = candidate["content"]?.jsonObject?.get("parts")?.jsonArray ?: JsonArray(emptyList()) + for (part in parts) append(part.jsonObject["text"]?.jsonPrimitive?.contentOrNull.orEmpty()) + } + if (text.isEmpty()) throw APIException.Incomplete + val grounding = candidate["groundingMetadata"]?.jsonObject + val sources = linkedMapOf() + val chunks = grounding?.get("groundingChunks")?.jsonArray ?: JsonArray(emptyList()) + for (chunk in chunks) { + val web = chunk.jsonObject["web"]?.jsonObject ?: continue + val url = web["uri"]?.jsonPrimitive?.contentOrNull ?: continue + if (SourceLink("Source", url).safeUrl() != null) sources.putIfAbsent(url, SourceLink(web["title"]?.jsonPrimitive?.contentOrNull ?: "Source", url)) + } + val searchQueries = grounding?.get("webSearchQueries")?.jsonArray + ?.mapNotNull { it.jsonPrimitive.contentOrNull?.trim()?.takeIf(String::isNotEmpty) } + ?.distinct() + ?.size ?: 0 + val searchEntryPointHtml = grounding?.get("searchEntryPoint")?.jsonObject + ?.get("renderedContent")?.jsonPrimitive?.contentOrNull?.take(32_000) + val usage = response["usageMetadata"]?.jsonObject + return APIResult( + text = text, + sources = sources.values.toList(), + searchEntryPointHTML = searchEntryPointHtml, + usage = APIUsage( + input = usage?.get("promptTokenCount")?.jsonPrimitive?.intOrNull ?: 0, + output = usage?.get("candidatesTokenCount")?.jsonPrimitive?.intOrNull ?: 0, + searches = if (search) searchQueries.takeIf { it > 0 } ?: if (sources.isNotEmpty()) 1 else 0 else 0, + ), + ) + } + + private fun geminiCompatibleSchema(value: JsonElement): JsonElement = when (value) { + is JsonObject -> JsonObject(value + .filterKeys { it != "additionalProperties" } + .mapValues { (_, nested) -> geminiCompatibleSchema(nested) }) + is JsonArray -> JsonArray(value.map(::geminiCompatibleSchema)) + else -> value + } + + private suspend fun directPost(url: HttpUrl, key: String, body: JsonObject, providerName: String, google: Boolean = false): JsonObject { + val builder = Request.Builder() + .url(url) + .header("Content-Type", JSON_MEDIA_TYPE.toString()) + .post(body.toString().toRequestBody(JSON_MEDIA_TYPE)) + if (google) builder.header("x-goog-api-key", key) else builder.header("Authorization", "Bearer $key") + val request = builder.build() + // Parse on OkHttp's worker while the continuation remains cancellable. + // Cancellation closes a response even if the peer stalls halfway through its body. + return suspendCancellableCoroutine { continuation -> + val call = client.newCall(request) + continuation.invokeOnCancellation { call.cancel() } + call.enqueue(object : Callback { + override fun onFailure(call: Call, error: IOException) { + if (continuation.isActive) continuation.resumeWithException(error) + } + override fun onResponse(call: Call, response: Response) { + try { + val value = response.use { + if (it.code !in 200..299) { + val errorCode = runCatching { + val payload = it.peekBody(16_385).string() + if (payload.toByteArray(Charsets.UTF_8).size > 16_384) null else + (JSON.parseToJsonElement(payload).jsonObject["error"] as? JsonObject) + ?.get("code")?.jsonPrimitive?.contentOrNull + }.getOrNull() + throw APIException.Http(it.code, errorCode, it.header("x-request-id"), providerName) + } + val payload = it.readBoundedBody() + try { JSON.parseToJsonElement(payload).jsonObject } + catch (_: Exception) { throw APIException.InvalidResponse } + } + if (continuation.isActive) continuation.resume(value) + } catch (error: Exception) { + if (continuation.isActive) continuation.resumeWithException(error) + } + } + }) + } + } + private fun Response.readBoundedBody(): String { val responseBody = body ?: throw APIException.InvalidResponse if (responseBody.contentLength() > MAX_RESPONSE_BYTES) throw APIException.InvalidResponse @@ -159,22 +241,22 @@ class APIClient private constructor( } sealed class APIException(message: String, cause: Throwable? = null) : IOException(message, cause) { - data object MissingKey : APIException("Add your OpenAI key in Settings to begin.") - data object InvalidResponse : APIException("OpenAI returned an incomplete response. Please try again.") - data object Incomplete : APIException("OpenAI returned an incomplete response. Please try again.") + data object MissingKey : APIException("Add your selected AI provider key in Settings to begin.") + data object InvalidResponse : APIException("The AI provider returned an incomplete response. Please try again.") + data object Incomplete : APIException("The AI provider returned an incomplete response. Please try again.") data object Refused : APIException("Mural couldn't complete that request. Try a different topic.") - class Http(val status: Int, code: String? = null, reference: String? = null) : APIException(messageFor(status)) { + class Http(val status: Int, code: String? = null, reference: String? = null, providerName: String = "OpenAI") : APIException(messageFor(status, providerName)) { val code = ProviderFailureKind.safeCode(code) val reference = ProviderFailureKind.safeReference(reference) val kind get() = ProviderFailureKind.classify(status, code) } companion object { - private fun messageFor(status: Int): String = when (status) { - 401 -> "Your OpenAI key wasn't accepted. Check it in Settings." - 403, 404 -> "This API key may not have access to the requested model. Check your OpenAI project." - 429 -> "OpenAI's usage or rate limit was reached. Check your project billing and limits." - else -> "OpenAI couldn't complete the request (HTTP $status). Please try again." + private fun messageFor(status: Int, providerName: String): String = when (status) { + 401 -> "Your $providerName key wasn't accepted. Check it in Settings." + 403, 404 -> "This API key may not have access to the requested model. Check your $providerName project." + 429 -> "$providerName's usage or rate limit was reached. Check your project billing and limits." + else -> "$providerName couldn't complete the request (HTTP $status). Please try again." } } } @@ -186,6 +268,12 @@ class APIClient private constructor( .addPathSegment("v1") .addPathSegment("") .build() + private val GOOGLE_API_BASE_URL = HttpUrl.Builder() + .scheme("https") + .host("generativelanguage.googleapis.com") + .addPathSegment("v1beta") + .addPathSegment("") + .build() private val JSON_MEDIA_TYPE = "application/json; charset=utf-8".toMediaType() private val VALID_PATH = Regex("[a-z0-9][a-z0-9_/-]*") private const val MAX_RESPONSE_BYTES = 1_048_576L diff --git a/apps/android/app/src/main/java/chat/mural/network/CredentialStore.kt b/apps/android/app/src/main/java/chat/mural/network/CredentialStore.kt index 5cb66761..07bd876b 100644 --- a/apps/android/app/src/main/java/chat/mural/network/CredentialStore.kt +++ b/apps/android/app/src/main/java/chat/mural/network/CredentialStore.kt @@ -10,7 +10,7 @@ import javax.crypto.KeyGenerator import javax.crypto.SecretKey import javax.crypto.spec.GCMParameterSpec -/** Stores the OpenAI API key encrypted by a non-exportable Android Keystore key. */ +/** Stores provider API keys encrypted by non-exportable Android Keystore keys. */ class CredentialStore internal constructor( context: Context, preferencesName: String, @@ -23,20 +23,31 @@ class CredentialStore internal constructor( val hasKey: Boolean get() = read() != null + fun hasKey(provider: AIProvider): Boolean = read(provider) != null + @Synchronized fun save(key: String) { + save(key, AIProvider.OPENAI) + } + + @Synchronized + fun save(key: String, provider: AIProvider) { val value = key.trim() - if (!value.startsWith("sk-") || value.length < 20 || value.any(Char::isWhitespace)) { + val valid = when (provider) { + AIProvider.OPENAI -> value.startsWith("sk-") && value.length >= 20 + AIProvider.GOOGLE_AI_STUDIO -> value.length >= 20 + } + if (!valid || value.any(Char::isWhitespace)) { throw CredentialException.Invalid } try { val cipher = Cipher.getInstance(TRANSFORMATION) - cipher.init(Cipher.ENCRYPT_MODE, encryptionKey()) + cipher.init(Cipher.ENCRYPT_MODE, encryptionKey(provider)) val ciphertext = cipher.doFinal(value.toByteArray(Charsets.UTF_8)) val saved = preferences.edit() - .putString(CIPHERTEXT, Base64.encodeToString(ciphertext, Base64.NO_WRAP)) - .putString(IV, Base64.encodeToString(cipher.iv, Base64.NO_WRAP)) + .putString(ciphertextKey(provider), Base64.encodeToString(ciphertext, Base64.NO_WRAP)) + .putString(ivKey(provider), Base64.encodeToString(cipher.iv, Base64.NO_WRAP)) .commit() if (!saved) throw CredentialException.Save } catch (error: CredentialException) { @@ -48,10 +59,15 @@ class CredentialStore internal constructor( @Synchronized fun read(): String? { - val encodedCiphertext = preferences.getString(CIPHERTEXT, null) ?: return null - val encodedIv = preferences.getString(IV, null) ?: return clearUnreadableCredential() + return read(AIProvider.OPENAI) + } + + @Synchronized + fun read(provider: AIProvider): String? { + val encodedCiphertext = preferences.getString(ciphertextKey(provider), null) ?: return null + val encodedIv = preferences.getString(ivKey(provider), null) ?: return clearUnreadableCredential(provider) return try { - val key = keyStore().getKey(keyAlias, null) as? SecretKey ?: return clearUnreadableCredential() + val key = keyStore().getKey(keyAlias(provider), null) as? SecretKey ?: return clearUnreadableCredential(provider) val cipher = Cipher.getInstance(TRANSFORMATION) cipher.init( Cipher.DECRYPT_MODE, @@ -59,29 +75,34 @@ class CredentialStore internal constructor( GCMParameterSpec(GCM_TAG_BITS, Base64.decode(encodedIv, Base64.NO_WRAP)), ) cipher.doFinal(Base64.decode(encodedCiphertext, Base64.NO_WRAP)).toString(Charsets.UTF_8) - .takeIf { it.startsWith("sk-") && it.length >= 20 && it.none(Char::isWhitespace) } - ?: clearUnreadableCredential() + .takeIf { isValidStoredKey(it, provider) } + ?: clearUnreadableCredential(provider) } catch (_: Exception) { - clearUnreadableCredential() + clearUnreadableCredential(provider) } } @Synchronized fun delete() { - if (!preferences.edit().clear().commit()) throw CredentialException.Remove + delete(AIProvider.OPENAI) + } + + @Synchronized + fun delete(provider: AIProvider) { + if (!preferences.edit().remove(ciphertextKey(provider)).remove(ivKey(provider)).commit()) throw CredentialException.Remove try { val store = keyStore() - if (store.containsAlias(keyAlias)) store.deleteEntry(keyAlias) + if (store.containsAlias(keyAlias(provider))) store.deleteEntry(keyAlias(provider)) } catch (_: Exception) { } } - private fun encryptionKey(): SecretKey { + private fun encryptionKey(provider: AIProvider): SecretKey { val store = keyStore() - (store.getKey(keyAlias, null) as? SecretKey)?.let { return it } + (store.getKey(keyAlias(provider), null) as? SecretKey)?.let { return it } return KeyGenerator.getInstance(KeyProperties.KEY_ALGORITHM_AES, ANDROID_KEY_STORE).run { init( KeyGenParameterSpec.Builder( - keyAlias, + keyAlias(provider), KeyProperties.PURPOSE_ENCRYPT or KeyProperties.PURPOSE_DECRYPT, ) .setBlockModes(KeyProperties.BLOCK_MODE_GCM) @@ -96,19 +117,33 @@ class CredentialStore internal constructor( private fun keyStore(): KeyStore = KeyStore.getInstance(ANDROID_KEY_STORE).apply { load(null) } - private fun clearUnreadableCredential(): Nothing? { - preferences.edit().clear().commit() + private fun clearUnreadableCredential(provider: AIProvider): Nothing? { + preferences.edit().remove(ciphertextKey(provider)).remove(ivKey(provider)).commit() try { val store = keyStore() - if (store.containsAlias(keyAlias)) store.deleteEntry(keyAlias) + if (store.containsAlias(keyAlias(provider))) store.deleteEntry(keyAlias(provider)) } catch (_: Exception) { // A stale ciphertext is already gone; a later save can retry key replacement. } return null } + private fun isValidStoredKey(value: String, provider: AIProvider): Boolean = when (provider) { + AIProvider.OPENAI -> value.startsWith("sk-") && value.length >= 20 + AIProvider.GOOGLE_AI_STUDIO -> value.length >= 20 + } && value.none(Char::isWhitespace) + + private fun suffix(provider: AIProvider): String = when (provider) { + AIProvider.OPENAI -> "" + AIProvider.GOOGLE_AI_STUDIO -> "_google" + } + + private fun ciphertextKey(provider: AIProvider) = CIPHERTEXT + suffix(provider) + private fun ivKey(provider: AIProvider) = IV + suffix(provider) + private fun keyAlias(provider: AIProvider) = keyAlias + suffix(provider).replace('_', '.') + sealed class CredentialException(message: String) : IllegalStateException(message) { - data object Invalid : CredentialException("Enter a valid OpenAI API key.") + data object Invalid : CredentialException("Enter a valid API key for the selected provider.") data object Save : CredentialException("The key couldn't be saved securely on this device.") data object Remove : CredentialException("The key couldn't be removed. Unlock this device and try again.") } diff --git a/apps/android/app/src/main/java/chat/mural/network/GeminiLiveTransport.kt b/apps/android/app/src/main/java/chat/mural/network/GeminiLiveTransport.kt new file mode 100644 index 00000000..44c28883 --- /dev/null +++ b/apps/android/app/src/main/java/chat/mural/network/GeminiLiveTransport.kt @@ -0,0 +1,473 @@ +package chat.mural.network + +import android.Manifest +import android.content.Context +import android.content.pm.PackageManager +import android.media.AudioAttributes +import android.media.AudioFormat +import android.media.AudioRecord +import android.media.AudioTrack +import android.media.MediaRecorder +import android.os.Process +import androidx.core.content.ContextCompat +import java.nio.ByteBuffer +import java.nio.ByteOrder +import java.util.Base64 +import java.util.concurrent.TimeUnit +import kotlinx.coroutines.CoroutineScope +import kotlinx.coroutines.Dispatchers +import kotlinx.coroutines.Job +import kotlinx.coroutines.delay +import kotlinx.coroutines.isActive +import kotlinx.coroutines.launch +import kotlinx.serialization.json.Json +import kotlinx.serialization.json.JsonArray +import kotlinx.serialization.json.JsonObject +import kotlinx.serialization.json.buildJsonArray +import kotlinx.serialization.json.buildJsonObject +import kotlinx.serialization.json.contentOrNull +import kotlinx.serialization.json.jsonArray +import kotlinx.serialization.json.jsonObject +import kotlinx.serialization.json.jsonPrimitive +import kotlinx.serialization.json.put +import okhttp3.HttpUrl +import okhttp3.OkHttpClient +import okhttp3.Request +import okhttp3.Response +import okhttp3.WebSocket +import okhttp3.WebSocketListener +import java.util.ArrayDeque +import java.util.concurrent.atomic.AtomicLong + +/** Direct Google AI Studio Live API transport using 16 kHz input and 24 kHz output PCM. */ +class GeminiLiveTransport( + context: Context, + private val scope: CoroutineScope, +) { + var onEvent: ((JsonObject) -> Unit)? = null + var onFailure: ((String) -> Unit)? = null + var onLevels: ((Double, Double) -> Unit)? = null + + private val applicationContext = context.applicationContext + private val client = OkHttpClient.Builder().readTimeout(0, TimeUnit.MILLISECONDS).build() + @Volatile private var socket: WebSocket? = null + @Volatile private var active = false + @Volatile private var muted = false + @Volatile private var closing = false + @Volatile private var socketReady = false + private val connectionGeneration = AtomicLong(0) + private val controlLock = Any() + private val pendingControlMessages = ArrayDeque() + private var flushingControlMessages = false + private var record: AudioRecord? = null + private var track: AudioTrack? = null + private var recordJob: Job? = null + private var connectedAt = 0L + private var sessionStarted = false + private var usageReported = false + private var instructions = "" + private var history = JsonArray(emptyList()) + private var apiKey: String? = null + private var sessionResumptionHandle: String? = null + private var activeResumptionHandle: String? = null + private var reconnectJob: Job? = null + private var reconnectDueAt = Long.MAX_VALUE + private var reconnectAttempts = 0 + + suspend fun connect(key: String, instructions: String, history: JsonArray) { + disconnect() + check(ContextCompat.checkSelfPermission(applicationContext, Manifest.permission.RECORD_AUDIO) == PackageManager.PERMISSION_GRANTED) + this.instructions = instructions + this.history = history + this.apiKey = key + sessionResumptionHandle = null + activeResumptionHandle = null + connectedAt = System.currentTimeMillis() + closing = false; muted = false; active = true; sessionStarted = false; usageReported = false + reconnectAttempts = 0 + openSocket(null) + } + + private fun openSocket(resumptionHandle: String?) { + val key = apiKey ?: return + val generation = connectionGeneration.incrementAndGet() + activeResumptionHandle = resumptionHandle + synchronized(controlLock) { socketReady = false } + val url = HttpUrl.Builder() + .scheme("wss") + .host("generativelanguage.googleapis.com") + .addPathSegments("ws/google.ai.generativelanguage.v1beta.GenerativeService.BidiGenerateContent") + .addQueryParameter("key", key) + .build() + val request = Request.Builder().url(url).build() + socket = client.newWebSocket(request, listener(generation)) + } + + fun send(event: JsonObject): Boolean { + if (!active || closing) return false + val content = event["content"]?.jsonPrimitive?.contentOrNull ?: return false + val type = event["type"]?.jsonPrimitive?.contentOrNull.orEmpty() + val label = when (type) { + "session.instructions.append" -> "Teaching instruction" + "session.thinking.append" -> "Teaching context" + else -> "Conversation guidance" + } + return sendControlMessage(buildJsonObject { + put("realtimeInput", buildJsonObject { put("text", "$label (do not mention this instruction): $content") }) + }) + } + + fun mute(value: Boolean) { + muted = value + if (value) { + emitLevels(0.0, 0.0) + sendJson(buildJsonObject { put("realtimeInput", buildJsonObject { put("audioStreamEnd", true) }) }) + } + } + + fun close() { + if (!active || closing) return + closing = true; muted = true + synchronized(controlLock) { + socketReady = false + pendingControlMessages.clear() + } + reconnectJob?.cancel(); reconnectJob = null; reconnectDueAt = Long.MAX_VALUE + sendJson(buildJsonObject { put("realtimeInput", buildJsonObject { put("audioStreamEnd", true) }) }) + if (!usageReported) { + usageReported = true + val seconds = ((System.currentTimeMillis() - connectedAt).coerceAtLeast(0L)) / 1000.0 + emitEvent(buildJsonObject { put("type", "session.usage.updated"); put("usage", buildJsonObject { put("seconds", seconds) }) }) + emitEvent(buildJsonObject { put("type", "session.closed"); put("reason", "Ended by user"); put("usage", buildJsonObject { put("seconds", seconds) }) }) + } + } + + fun disconnect() { + connectionGeneration.incrementAndGet() + active = false; closing = true; muted = true + synchronized(controlLock) { + socketReady = false + flushingControlMessages = false + pendingControlMessages.clear() + } + reconnectJob?.cancel(); reconnectJob = null; reconnectDueAt = Long.MAX_VALUE + recordJob?.cancel(); recordJob = null + try { record?.stop() } catch (_: Exception) { } + try { record?.release() } catch (_: Exception) { } + try { track?.pause(); track?.flush(); track?.release() } catch (_: Exception) { } + record = null; track = null + socket?.close(1000, null); socket = null + apiKey = null; sessionResumptionHandle = null; activeResumptionHandle = null + emitLevels(0.0, 0.0) + } + + private fun listener(expectedGeneration: Long) = object : WebSocketListener() { + override fun onOpen(webSocket: WebSocket, response: Response) { + if (!isCurrent(webSocket, expectedGeneration)) return + sendJson(setupMessage(), webSocket, expectedGeneration) + } + + override fun onMessage(webSocket: WebSocket, text: String) { + parse(text, webSocket, expectedGeneration) + } + + override fun onMessage(webSocket: WebSocket, bytes: okio.ByteString) { + parse(bytes.utf8(), webSocket, expectedGeneration) + } + + override fun onFailure(webSocket: WebSocket, t: Throwable, response: Response?) { + handleSocketEnded(webSocket, expectedGeneration, "The Google AI Studio voice connection ended. Check your key and connection, then try again.") + } + + override fun onClosing(webSocket: WebSocket, code: Int, reason: String) { + if (isCurrent(webSocket, expectedGeneration)) { + handleSocketEnded(webSocket, expectedGeneration, "The Google AI Studio voice connection ended. Check your key and connection, then try again.") + webSocket.close(code, reason) + } + } + + override fun onClosed(webSocket: WebSocket, code: Int, reason: String) { + handleSocketEnded(webSocket, expectedGeneration, "The Google AI Studio voice connection ended. Check your key and connection, then try again.") + } + } + + private fun handleSocketEnded(webSocket: WebSocket, expectedGeneration: Long, message: String) { + if (!isCurrent(webSocket, expectedGeneration) || closing) return + synchronized(controlLock) { socketReady = false } + if (sessionResumptionHandle.isNullOrBlank()) fail(message) else scheduleReconnect(0L) + } + + private fun setupMessage(): JsonObject = buildJsonObject { + put("setup", buildJsonObject { + put("model", "models/${AIProvider.GOOGLE_AI_STUDIO.liveModel}") + put("generationConfig", buildJsonObject { + put("responseModalities", buildJsonArray { add("AUDIO") }) + put("speechConfig", buildJsonObject { put("voiceConfig", buildJsonObject { + put("prebuiltVoiceConfig", buildJsonObject { put("voiceName", "Kore") }) + }) }) + }) + put("systemInstruction", buildJsonObject { put("parts", buildJsonArray { add(buildJsonObject { put("text", instructions) }) }) }) + put("inputAudioTranscription", buildJsonObject { }) + put("outputAudioTranscription", buildJsonObject { }) + put("contextWindowCompression", buildJsonObject { put("slidingWindow", buildJsonObject { }) }) + put("sessionResumption", buildJsonObject { + activeResumptionHandle?.takeIf { it.isNotBlank() }?.let { put("handle", it) } + }) + if (activeResumptionHandle == null && historyTurns().isNotEmpty()) { + put("historyConfig", buildJsonObject { put("initialHistoryInClientContent", true) }) + } + }) + } + + private fun scheduleReconnect(delayMillis: Long) { + if (!active || closing || sessionResumptionHandle.isNullOrBlank()) return + val dueAt = System.currentTimeMillis() + delayMillis.coerceAtLeast(0L) + if (reconnectJob?.isActive == true && dueAt >= reconnectDueAt) return + reconnectJob?.cancel() + reconnectDueAt = dueAt + reconnectJob = scope.launch { + delay(delayMillis.coerceAtLeast(0L)) + reconnectDueAt = Long.MAX_VALUE + reconnectJob = null + reconnect() + } + } + + private fun reconnect() { + if (!active || closing) return + val handle = sessionResumptionHandle?.takeIf { it.isNotBlank() } ?: run { + fail("The Google AI Studio voice connection ended. Check your key and connection, then try again.") + return + } + if (++reconnectAttempts > MAX_RECONNECT_ATTEMPTS) { + fail("The Google AI Studio voice connection could not be resumed. Check your connection, then try again.") + return + } + recordJob?.cancel(); recordJob = null + try { record?.stop() } catch (_: Exception) { } + try { record?.release() } catch (_: Exception) { } + record = null + val oldSocket = socket + synchronized(controlLock) { + socketReady = false + socket = null + } + oldSocket?.close(1000, "Resuming session") + openSocket(handle) + } + + private fun timeLeftMillis(value: String?): Long { + val text = value?.trim()?.lowercase() ?: return 0L + return when { + text.endsWith("ms") -> text.removeSuffix("ms").toDoubleOrNull()?.toLong() ?: 0L + text.endsWith("s") -> ((text.removeSuffix("s").toDoubleOrNull() ?: 0.0) * 1_000.0).toLong() + else -> text.toDoubleOrNull()?.toLong() ?: 0L + } + } + + private fun historyTurns(): JsonArray = buildJsonArray { + for (item in history) { + val objectValue = item as? JsonObject ?: continue + val role = objectValue["role"]?.jsonPrimitive?.contentOrNull ?: continue + val content = objectValue["content"]?.jsonArray ?: continue + val text = content.mapNotNull { (it as? JsonObject)?.get("text")?.jsonPrimitive?.contentOrNull }.joinToString(" ") + if (text.isNotBlank()) add(buildJsonObject { + put("role", if (role == "assistant") "model" else "user") + put("parts", buildJsonArray { add(buildJsonObject { put("text", text) }) }) + }) + } + } + + private fun sendHistory(webSocket: WebSocket, generation: Long) { + val turns = historyTurns() + if (turns.isEmpty()) return + sendJson(buildJsonObject { put("clientContent", buildJsonObject { put("turns", turns); put("turnComplete", true) }) }, webSocket, generation) + } + + private fun startAudio(expectedGeneration: Long, expectedSocket: WebSocket) { + if (!isCurrent(expectedSocket, expectedGeneration)) return + val minBuffer = AudioRecord.getMinBufferSize(SAMPLE_RATE_IN, AudioFormat.CHANNEL_IN_MONO, AudioFormat.ENCODING_PCM_16BIT) + .coerceAtLeast(SAMPLES_PER_PACKET * 2) + val recorder = record ?: AudioRecord( + MediaRecorder.AudioSource.VOICE_COMMUNICATION, + SAMPLE_RATE_IN, + AudioFormat.CHANNEL_IN_MONO, + AudioFormat.ENCODING_PCM_16BIT, + minBuffer * 2, + ).also { record = it } + val player = track ?: AudioTrack.Builder() + .setAudioAttributes(AudioAttributes.Builder().setUsage(AudioAttributes.USAGE_VOICE_COMMUNICATION).setContentType(AudioAttributes.CONTENT_TYPE_SPEECH).build()) + .setAudioFormat(AudioFormat.Builder().setSampleRate(SAMPLE_RATE_OUT).setEncoding(AudioFormat.ENCODING_PCM_16BIT).setChannelMask(AudioFormat.CHANNEL_OUT_MONO).build()) + .setTransferMode(AudioTrack.MODE_STREAM) + .setBufferSizeInBytes(SAMPLE_RATE_OUT / 2) + .build() + .also { track = it } + if (recorder.recordingState != AudioRecord.RECORDSTATE_RECORDING) recorder.startRecording() + if (player.playState != AudioTrack.PLAYSTATE_PLAYING) player.play() + recordJob?.cancel() + recordJob = scope.launch(Dispatchers.IO) { + Process.setThreadPriority(Process.THREAD_PRIORITY_AUDIO) + val packet = ByteArray(SAMPLES_PER_PACKET * 2) + while (isActive && isCurrent(expectedSocket, expectedGeneration)) { + val count = recorder.read(packet, 0, packet.size) + if (count <= 0 || muted) continue + val data = packet.copyOf(count) + emitLevels(minOf(1.0, rms(data) * 7.0), 0.0, expectedSocket, expectedGeneration) + sendJson(buildJsonObject { + put("realtimeInput", buildJsonObject { + put("audio", buildJsonObject { + put("mimeType", "audio/pcm;rate=16000") + put("data", Base64.getEncoder().encodeToString(data)) + }) + }) + }, expectedSocket, expectedGeneration) + } + } + } + + private fun parse(text: String, expectedSocket: WebSocket, expectedGeneration: Long) { + if (!isCurrent(expectedSocket, expectedGeneration)) return + val json = runCatching { JSON.parseToJsonElement(text).jsonObject }.getOrNull() ?: return + json["sessionResumptionUpdate"]?.jsonObject?.get("newHandle")?.jsonPrimitive?.contentOrNull + ?.takeIf { it.isNotBlank() } + ?.let { sessionResumptionHandle = it } + json["goAway"]?.jsonObject?.let { goAway -> + scheduleReconnect(timeLeftMillis(goAway["timeLeft"]?.jsonPrimitive?.contentOrNull)) + return + } + if (json["setupComplete"] != null) { + try { startAudio(expectedGeneration, expectedSocket) } catch (_: Exception) { + fail("The Google AI Studio microphone couldn’t start.") + return + } + if (!isCurrent(expectedSocket, expectedGeneration)) return + val firstSession = !sessionStarted + if (firstSession && activeResumptionHandle == null) sendHistory(expectedSocket, expectedGeneration) + synchronized(controlLock) { + if (!isCurrent(expectedSocket, expectedGeneration)) return + socketReady = true + flushPendingControlMessages(expectedSocket, expectedGeneration) + } + reconnectAttempts = 0 + if (firstSession) { + sessionStarted = true + val session = buildJsonObject { put("id", "gemini-live"); put("model", AIProvider.GOOGLE_AI_STUDIO.liveModel) } + emitEvent(buildJsonObject { put("type", "mural.session.created"); put("session", session) }, expectedGeneration) + emitEvent(buildJsonObject { put("type", "session.started"); put("session", session) }, expectedGeneration) + } + return + } + val error = json["error"]?.jsonObject + if (error != null) { + fail(error["message"]?.jsonPrimitive?.contentOrNull?.take(240) ?: "The Google AI Studio voice request failed.") + return + } + val content = json["serverContent"]?.jsonObject ?: return + if (content["interrupted"]?.jsonPrimitive?.contentOrNull == "true") { + try { track?.pause(); track?.flush(); track?.play() } catch (_: Exception) { } + emitLevels(0.0, 0.0, expectedSocket, expectedGeneration) + } + val now = (System.currentTimeMillis() - connectedAt).toInt().coerceAtLeast(0) + content["inputTranscription"]?.jsonObject?.get("text")?.jsonPrimitive?.contentOrNull?.takeIf { it.isNotBlank() }?.let { + emitTranscript("session.input_transcript.delta", it, now, expectedGeneration) + } + content["outputTranscription"]?.jsonObject?.get("text")?.jsonPrimitive?.contentOrNull?.takeIf { it.isNotBlank() }?.let { + emitTranscript("session.output_transcript.delta", it, now, expectedGeneration) + } + val parts = content["modelTurn"]?.jsonObject?.get("parts")?.jsonArray ?: JsonArray(emptyList()) + for (part in parts) { + val inline = (part as? JsonObject)?.get("inlineData")?.jsonObject ?: continue + val encoded = inline["data"]?.jsonPrimitive?.contentOrNull ?: continue + val bytes = runCatching { Base64.getDecoder().decode(encoded) }.getOrNull() ?: continue + if (!isCurrent(expectedSocket, expectedGeneration)) return + track?.write(bytes, 0, bytes.size, AudioTrack.WRITE_BLOCKING) + emitLevels(0.0, 0.35, expectedSocket, expectedGeneration) + } + } + + private fun emitTranscript(type: String, text: String, now: Int, generation: Long) { + emitEvent(buildJsonObject { + put("type", type); put("event_id", java.util.UUID.randomUUID().toString()); put("delta", text) + put("start_ms", (now - 1).coerceAtLeast(0)); put("end_ms", now) + }, generation) + } + + private fun isCurrent(expectedSocket: WebSocket, expectedGeneration: Long): Boolean = + active && connectionGeneration.get() == expectedGeneration && socket === expectedSocket + + private fun sendControlMessage(value: JsonObject): Boolean { + synchronized(controlLock) { + if (!active || closing) return false + val currentSocket = socket + if (!socketReady || flushingControlMessages || currentSocket == null) { + pendingControlMessages.addLast(value) + return true + } + if (sendJson(value, currentSocket, connectionGeneration.get())) return true + socketReady = false + pendingControlMessages.addFirst(value) + return true + } + } + + private fun flushPendingControlMessages(expectedSocket: WebSocket, expectedGeneration: Long) { + synchronized(controlLock) { + if (!socketReady || !isCurrent(expectedSocket, expectedGeneration)) return + flushingControlMessages = true + try { + while (pendingControlMessages.isNotEmpty()) { + val message = pendingControlMessages.first() + if (!sendJson(message, expectedSocket, expectedGeneration)) { + socketReady = false + return + } + pendingControlMessages.removeFirst() + } + } finally { + flushingControlMessages = false + } + } + } + + private fun sendJson(value: JsonObject, expectedSocket: WebSocket? = null, expectedGeneration: Long? = null): Boolean { + val current = socket ?: return false + if (expectedSocket != null && current !== expectedSocket) return false + if (expectedGeneration != null && connectionGeneration.get() != expectedGeneration) return false + return current.send(value.toString()) + } + + private fun emitEvent(event: JsonObject, expectedGeneration: Long = connectionGeneration.get()) { + scope.launch { if (active && connectionGeneration.get() == expectedGeneration) onEvent?.invoke(event) } + } + private fun emitLevels(input: Double, output: Double, expectedSocket: WebSocket? = null, expectedGeneration: Long? = null) { + scope.launch { + if (active && (expectedSocket == null || socket === expectedSocket) && (expectedGeneration == null || connectionGeneration.get() == expectedGeneration)) { + onLevels?.invoke(input, output) + } + } + } + private fun fail(message: String) { + reconnectJob?.cancel(); reconnectJob = null; reconnectDueAt = Long.MAX_VALUE + val expectedGeneration = connectionGeneration.get() + scope.launch { if (active && connectionGeneration.get() == expectedGeneration && !closing) onFailure?.invoke(message) } + } + + private fun rms(data: ByteArray): Double { + if (data.size < 2) return 0.0 + val samples = ByteBuffer.wrap(data).order(ByteOrder.LITTLE_ENDIAN).asShortBuffer() + var total = 0.0 + while (samples.hasRemaining()) { + val value = samples.get().toDouble() / Short.MAX_VALUE + total += value * value + } + return kotlin.math.sqrt(total / (data.size / 2.0)) + } + + companion object { + private const val SAMPLE_RATE_IN = 16_000 + private const val SAMPLE_RATE_OUT = 24_000 + private const val SAMPLES_PER_PACKET = 1_600 + private const val MAX_RECONNECT_ATTEMPTS = 5 + private val JSON = Json { ignoreUnknownKeys = true } + } +} diff --git a/apps/android/app/src/main/java/chat/mural/network/HanReadings.kt b/apps/android/app/src/main/java/chat/mural/network/HanReadings.kt index 7c53fafb..f4f1c8ec 100644 --- a/apps/android/app/src/main/java/chat/mural/network/HanReadings.kt +++ b/apps/android/app/src/main/java/chat/mural/network/HanReadings.kt @@ -26,7 +26,16 @@ class IcuHanReader : HanReader { } override fun words(text: String): List { - val iterator = BreakIterator.getWordInstance(ULocale.CHINA) + return HanWords.merge(localeWords(text, ULocale.CHINA), CAPTION_WORDS) + } + + override fun languageWords(text: String, languageID: String): List? = when (languageID) { + "th" -> localeWords(text, ULocale("th_TH")) + else -> null + } + + private fun localeWords(text: String, locale: ULocale): List { + val iterator = BreakIterator.getWordInstance(locale) iterator.setText(text) val pieces = mutableListOf() var start = iterator.first() @@ -35,7 +44,7 @@ class IcuHanReader : HanReader { pieces += text.substring(start, end) start = end; end = iterator.next() } - return HanWords.merge(pieces, CAPTION_WORDS) + return pieces } /** Prefer a word reading; use ICU only when a known ambiguous character does not need context. */ diff --git a/apps/android/app/src/main/java/chat/mural/network/LiveTransport.kt b/apps/android/app/src/main/java/chat/mural/network/LiveTransport.kt index 6851f44b..d1179125 100644 --- a/apps/android/app/src/main/java/chat/mural/network/LiveTransport.kt +++ b/apps/android/app/src/main/java/chat/mural/network/LiveTransport.kt @@ -67,6 +67,7 @@ class LiveTransport( private val applicationContext = context.applicationContext private val audioManager = applicationContext.getSystemService(Context.AUDIO_SERVICE) as AudioManager + private val gemini = GeminiLiveTransport(applicationContext, scope) private val generation = AtomicLong(0) private val lock = Any() private val retiredAttempts = ArrayDeque() @@ -75,16 +76,60 @@ class LiveTransport( @Volatile private var activeAttempt: Attempt? = null @Volatile private var startedState = false @Volatile private var mutedState = false + @Volatile private var geminiActive = false + @Volatile private var geminiAudioAttempt: Attempt? = null val started: Boolean get() = startedState val isMuted: Boolean get() = mutedState + init { + gemini.onEvent = { event -> + if (event["type"]?.jsonPrimitive?.contentOrNull == "session.started") startedState = true + scope.launch { if (geminiActive) onEvent?.invoke(event) } + } + gemini.onFailure = { message -> scope.launch { if (geminiActive) onFailure?.invoke(message) } } + gemini.onLevels = { input, output -> scope.launch { if (geminiActive) onLevels?.invoke(input, output) } } + } + suspend fun connect( api: LiveSessionProvider, instructions: String, history: JsonArray = JsonArray(emptyList()), language: String? = null, ) = withContext(AUDIO_DISPATCHER) { + if (api is APIClient && api.provider == AIProvider.GOOGLE_AI_STUDIO) { + disconnect() + drainRetiredAttempts() + if (applicationContext.checkSelfPermission(Manifest.permission.RECORD_AUDIO) != PackageManager.PERMISSION_GRANTED) { + throw microphoneException() + } + val key = CredentialStore(applicationContext).read(AIProvider.GOOGLE_AI_STUDIO) ?: throw APIClient.APIException.MissingKey + val attemptGeneration = generation.get() + val audioAttempt = Attempt( + id = attemptGeneration, + ownership = LiveSessionOwnership(audioScope), + previousAudioMode = audioManager.mode, + previousSpeakerphone = if (Build.VERSION.SDK_INT < 31) legacySpeakerphoneState() else false, + ) + synchronized(lock) { + if (generation.get() != attemptGeneration) throw CancellationException("Voice connection superseded") + geminiAudioAttempt = audioAttempt + geminiActive = true + startedState = false; mutedState = false + } + try { + configureAudio(audioAttempt) + gemini.connect(key, instructions, history) + return@withContext + } catch (error: Throwable) { + geminiActive = false; gemini.disconnect(); cleanupIfCurrent(audioAttempt) + throw error + } + } + if (geminiActive || geminiAudioAttempt != null) { + gemini.disconnect(); geminiActive = false + synchronized(lock) { geminiAudioAttempt?.let { retiredAttempts.addLast(it); geminiAudioAttempt = null } } + } val attemptGeneration = detachAttempt() drainRetiredAttempts() emitZeroLevels(attemptGeneration) @@ -153,6 +198,7 @@ class LiveTransport( /** True means accepted for delivery; every native operation runs on the audio worker. */ fun send(event: JsonObject): Boolean { + if (geminiActive) return gemini.send(event) val attempt = activeAttempt ?: return false if (!isCurrent(attempt) || !attempt.channelOpen.get()) return false audioScope.launch { @@ -173,6 +219,7 @@ class LiveTransport( } fun mute(muted: Boolean) { + if (geminiActive) { mutedState = muted; gemini.mute(muted); return } val attempt = activeAttempt ?: return mutedState = muted audioScope.launch { @@ -186,6 +233,7 @@ class LiveTransport( } fun close() { + if (geminiActive) { mutedState = true; gemini.close(); return } val attempt = activeAttempt ?: return attempt.closing.set(true) attempt.ownership.close() @@ -201,6 +249,8 @@ class LiveTransport( } fun disconnect() { + if (geminiActive) { gemini.disconnect(); geminiActive = false } + synchronized(lock) { geminiAudioAttempt?.let { retiredAttempts.addLast(it); geminiAudioAttempt = null } } val detached = detachAttempt() // This scope outlives the ViewModel so clearing the screen cannot cancel native cleanup. audioScope.launch { drainRetiredAttempts() } @@ -532,11 +582,12 @@ class LiveTransport( private fun cleanupIfCurrent(attempt: Attempt): Long? { val cleanupGeneration = synchronized(lock) { - if (activeAttempt !== attempt) null + if (activeAttempt !== attempt && geminiAudioAttempt !== attempt) null else { val nextGeneration = generation.incrementAndGet() retiredAttempts.addLast(attempt) - activeAttempt = null + if (activeAttempt === attempt) activeAttempt = null + if (geminiAudioAttempt === attempt) geminiAudioAttempt = null startedState = false mutedState = false nextGeneration @@ -606,7 +657,7 @@ class LiveTransport( } private fun isCurrent(attempt: Attempt): Boolean = - activeAttempt === attempt && generation.get() == attempt.id && !attempt.cleaned.get() + (activeAttempt === attempt || geminiAudioAttempt === attempt) && generation.get() == attempt.id && !attempt.cleaned.get() private fun requireCurrent(attempt: Attempt) { if (!isCurrent(attempt)) throw CancellationException("Voice connection superseded") diff --git a/apps/android/app/src/main/java/chat/mural/ui/LibraryScreens.kt b/apps/android/app/src/main/java/chat/mural/ui/LibraryScreens.kt index 365a8b99..2b188e06 100644 --- a/apps/android/app/src/main/java/chat/mural/ui/LibraryScreens.kt +++ b/apps/android/app/src/main/java/chat/mural/ui/LibraryScreens.kt @@ -1,5 +1,10 @@ package chat.mural.ui +import android.graphics.Color +import android.net.Uri +import android.webkit.WebResourceRequest +import android.webkit.WebView +import android.webkit.WebViewClient import androidx.annotation.StringRes import androidx.compose.foundation.background import androidx.compose.foundation.clickable @@ -54,6 +59,7 @@ import androidx.compose.ui.semantics.heading import androidx.compose.ui.semantics.semantics import androidx.compose.ui.text.font.FontWeight import androidx.compose.ui.unit.dp +import androidx.compose.ui.viewinterop.AndroidView import androidx.compose.ui.window.Dialog import chat.mural.MuralViewModel import chat.mural.R @@ -170,6 +176,9 @@ private fun CurrentTopicDialog( } vm.topicResult?.let { brief -> item { Text(inlineMarkdown(brief.text), style = MaterialTheme.typography.bodyLarge) } + brief.searchEntryPointHTML?.takeIf { it.isNotBlank() }?.let { html -> + item { GoogleSearchSuggestions(html) } + } if (brief.sources.isNotEmpty()) { item { Text(stringResource(R.string.topics_sources_heading), fontWeight = FontWeight.SemiBold, modifier = Modifier.semantics { heading() }) } items(brief.sources.filter { it.safeUrl() != null }) { source -> @@ -195,6 +204,38 @@ private fun CurrentTopicDialog( } } +@Composable +internal fun GoogleSearchSuggestions(html: String) { + val uriHandler = LocalUriHandler.current + Column(verticalArrangement = Arrangement.spacedBy(8.dp)) { + Text(stringResource(R.string.topics_google_search_suggestions), style = MaterialTheme.typography.labelMedium, color = MuralColors.Secondary) + AndroidView( + factory = { context -> + WebView(context).apply { + setBackgroundColor(Color.TRANSPARENT) + settings.javaScriptEnabled = false + settings.domStorageEnabled = false + webViewClient = object : WebViewClient() { + override fun shouldOverrideUrlLoading(view: WebView, request: WebResourceRequest): Boolean { + val uri: Uri = request.url + if (uri.scheme == "https" && !uri.host.isNullOrBlank()) uriHandler.openUri(uri.toString()) + return true + } + } + } + }, + update = { view -> + if (view.tag != html) { + view.tag = html + val policy = "" + view.loadDataWithBaseURL("https://www.google.com/", policy + html, "text/html", "UTF-8", null) + } + }, + modifier = Modifier.fillMaxWidth().height(104.dp), + ) + } +} + @Composable fun WordsScreen(vm: MuralViewModel) { var search by rememberSaveable { mutableStateOf("") } diff --git a/apps/android/app/src/main/java/chat/mural/ui/MuralApp.kt b/apps/android/app/src/main/java/chat/mural/ui/MuralApp.kt index e65ba319..75f56cec 100644 --- a/apps/android/app/src/main/java/chat/mural/ui/MuralApp.kt +++ b/apps/android/app/src/main/java/chat/mural/ui/MuralApp.kt @@ -48,6 +48,7 @@ import androidx.compose.ui.unit.dp import chat.mural.MuralViewModel import chat.mural.R import chat.mural.core.CloudAction +import chat.mural.core.AI_CONSENT_VERSION import chat.mural.AccountViewModel import chat.mural.MinutePurchaseViewModel import androidx.lifecycle.compose.collectAsStateWithLifecycle diff --git a/apps/android/app/src/main/java/chat/mural/ui/Onboarding.kt b/apps/android/app/src/main/java/chat/mural/ui/Onboarding.kt index 721a26e0..a3efc0bd 100644 --- a/apps/android/app/src/main/java/chat/mural/ui/Onboarding.kt +++ b/apps/android/app/src/main/java/chat/mural/ui/Onboarding.kt @@ -62,8 +62,7 @@ import chat.mural.R import chat.mural.core.LanguageModule import chat.mural.core.LanguageRegistry import chat.mural.core.MeaningLanguages - -const val AI_CONSENT_VERSION = 1 +import chat.mural.core.AI_CONSENT_VERSION @Composable fun OnboardingScreen( diff --git a/apps/android/app/src/main/java/chat/mural/ui/SettingsScreen.kt b/apps/android/app/src/main/java/chat/mural/ui/SettingsScreen.kt index 89cae4fb..7b82c206 100644 --- a/apps/android/app/src/main/java/chat/mural/ui/SettingsScreen.kt +++ b/apps/android/app/src/main/java/chat/mural/ui/SettingsScreen.kt @@ -65,6 +65,8 @@ import chat.mural.core.Passage import chat.mural.core.SessionRecord import chat.mural.core.Speaker import chat.mural.core.UsageSummary +import chat.mural.core.AI_CONSENT_VERSION +import chat.mural.network.AIProvider @Composable fun SettingsScreen(vm: MuralViewModel, onExport: () -> Unit, onImport: () -> Unit, onReviewConsent: () -> Unit, @@ -123,7 +125,12 @@ fun SettingsScreen(vm: MuralViewModel, onExport: () -> Unit, onImport: () -> Uni } item { SettingsGroup(stringResource(R.string.settings_advanced), - if (!vm.hasKey) stringResource(R.string.settings_byok_version_footer) else null) { + if (!vm.hasKey) stringResource(if (vm.aiProvider == AIProvider.OPENAI) R.string.settings_byok_version_footer else R.string.settings_google_byok_version_footer) else null) { + SettingsChoiceRow(stringResource(R.string.settings_ai_provider), vm.aiProvider.displayName, vm.aiProvider.name, + AIProvider.entries.map { it.name to it.displayName }, "settings-ai-provider", !vm.isRunning) { selected -> + vm.selectAIProvider(AIProvider.entries.first { it.name == selected }) + } + SettingsDivider() SettingsRow(stringResource(R.string.settings_use_own_key), symbol = SettingsSymbol.KEY, chevron = !advanced, modifier = Modifier.testTag("advanced-api-key"), onClick = { advanced = !advanced }) AnimatedVisibility(advanced, enter = expandVertically() + fadeIn(), exit = shrinkVertically() + fadeOut()) { @@ -134,14 +141,14 @@ fun SettingsScreen(vm: MuralViewModel, onExport: () -> Unit, onImport: () -> Uni SettingsRow(stringResource(if (vm.hasKey) R.string.settings_replace_key else R.string.settings_save_key), enabled = !vm.isRunning, tint = MuralColors.Secondary, chevron = true, onClick = { keyDialog = true }) SettingsDivider() - SettingsRow(stringResource(R.string.settings_open_api_keys), tint = MuralColors.Secondary, - onClick = { open("https://platform.openai.com/api-keys") }) + SettingsRow(stringResource(if (vm.aiProvider == AIProvider.OPENAI) R.string.settings_open_api_keys else R.string.settings_open_google_api_keys), tint = MuralColors.Secondary, + onClick = { open(vm.aiProvider.keyUrl) }) if (vm.hasKey) { SettingsDivider() SettingsRow(stringResource(R.string.settings_remove_key), enabled = !vm.isRunning, tint = MuralColors.Red, onClick = { deleteKey = true }) } - Text(stringResource(R.string.settings_key_owner_footer), style = MaterialTheme.typography.bodySmall, + Text(stringResource(if (vm.aiProvider == AIProvider.OPENAI) R.string.settings_key_owner_footer else R.string.settings_google_key_owner_footer), style = MaterialTheme.typography.bodySmall, color = MuralColors.Secondary, modifier = Modifier.padding(start = 16.dp, end = 16.dp, bottom = 16.dp)) } } @@ -149,7 +156,7 @@ fun SettingsScreen(vm: MuralViewModel, onExport: () -> Unit, onImport: () -> Uni } item { val usage = UsageSummary.of(vm.archive.sessions) - SettingsGroup(stringResource(R.string.settings_keep_comfortable), stringResource(R.string.settings_usage_footer)) { + SettingsGroup(stringResource(R.string.settings_keep_comfortable), stringResource(if (vm.aiProvider == AIProvider.OPENAI) R.string.settings_usage_footer else R.string.settings_google_usage_footer)) { val limits = (listOf(5, 10, 15, 20, 30, 60) + prefs.sessionMinutes).distinct().sorted() SettingsChoiceRow(stringResource(R.string.settings_conversation_limit), stringResource(R.string.settings_limit_minutes, prefs.sessionMinutes), prefs.sessionMinutes.toString(), limits.map { it.toString() to stringResource(R.string.settings_limit_minutes, it) }, @@ -157,12 +164,10 @@ fun SettingsScreen(vm: MuralViewModel, onExport: () -> Unit, onImport: () -> Uni SettingsDivider() SettingsRow(stringResource(R.string.settings_voice_time_label), usage.voiceTime) SettingsDivider() - SettingsRow(stringResource(R.string.settings_voice_estimate_label), usage.voiceEstimate) - SettingsDivider() SettingsRow(stringResource(R.string.settings_search_calls_label), usage.searchCalls.toString()) SettingsDivider() - SettingsRow(stringResource(R.string.settings_usage_billing_link), tint = MuralColors.Secondary, - onClick = { open("https://platform.openai.com/usage") }) + SettingsRow(stringResource(if (vm.aiProvider == AIProvider.OPENAI) R.string.settings_usage_billing_link else R.string.settings_google_usage_billing_link), tint = MuralColors.Secondary, + onClick = { open(vm.aiProvider.usageUrl) }) } } item { @@ -199,12 +204,12 @@ fun SettingsScreen(vm: MuralViewModel, onExport: () -> Unit, onImport: () -> Uni SettingsGroup { Column(Modifier.padding(16.dp), verticalArrangement = Arrangement.spacedBy(8.dp)) { Text(stringResource(R.string.settings_app_version_footer, version), style = MaterialTheme.typography.bodySmall, color = MuralColors.Secondary) - Text(stringResource(R.string.settings_models_footer), style = MaterialTheme.typography.bodySmall, color = MuralColors.Secondary) + Text(stringResource(if (vm.aiProvider == AIProvider.OPENAI) R.string.settings_models_footer else R.string.settings_google_models_footer), style = MaterialTheme.typography.bodySmall, color = MuralColors.Secondary) } SettingsDivider() - SettingsRow(stringResource(R.string.settings_openai_data_controls), tint = MuralColors.Secondary, - onClick = { open("https://developers.openai.com/api/docs/guides/your-data") }) - Text(stringResource(R.string.settings_data_use_footer), style = MaterialTheme.typography.bodySmall, + SettingsRow(stringResource(if (vm.aiProvider == AIProvider.OPENAI) R.string.settings_openai_data_controls else R.string.settings_google_data_controls), tint = MuralColors.Secondary, + onClick = { open(vm.aiProvider.dataControlsUrl) }) + Text(stringResource(if (vm.aiProvider == AIProvider.OPENAI) R.string.settings_data_use_footer else R.string.settings_google_data_use_footer), style = MaterialTheme.typography.bodySmall, color = MuralColors.Secondary, modifier = Modifier.padding(horizontal = 16.dp, vertical = 8.dp)) SettingsDivider() SettingsRow(stringResource(R.string.settings_open_source_notices), chevron = true, onClick = { notices = true }) @@ -289,7 +294,7 @@ private fun KeyDialog(vm: MuralViewModel, onDismiss: () -> Unit) { } Surface(shape = RoundedCornerShape(28.dp), color = MuralColors.Surface) { Column(Modifier.padding(22.dp), verticalArrangement = Arrangement.spacedBy(15.dp)) { - Text(stringResource(R.string.settings_key_dialog_title), style = MaterialTheme.typography.headlineMedium) + Text(stringResource(if (vm.aiProvider == AIProvider.OPENAI) R.string.settings_key_dialog_title else R.string.settings_google_key_dialog_title), style = MaterialTheme.typography.headlineMedium) Text(stringResource(R.string.settings_key_dialog_note), color = MuralColors.Secondary) MuralTextField( key, @@ -356,6 +361,11 @@ fun TranscriptDialog(vm: MuralViewModel, session: SessionRecord, onDismiss: () - if (passage.speaker == Speaker.user && !vm.isRunning) MuralTextButton(onClick = { correcting = passage }) { Text(stringResource(R.string.history_edit_passage_button)) } } } + liveSession.topics.forEach { topic -> + topic.searchEntryPointHTML?.takeIf { html -> html.isNotBlank() }?.let { html -> + item(key = "topic-search-${topic.id}") { GoogleSearchSuggestions(html) } + } + } liveSession.topics.flatMap { it.sources }.filter { it.safeUrl() != null }.takeIf { it.isNotEmpty() }?.let { sources -> item { Text(stringResource(R.string.history_saved_sources), fontWeight = FontWeight.SemiBold) } items(sources) { source -> diff --git a/apps/android/app/src/main/res/values-es/conversation_errors.xml b/apps/android/app/src/main/res/values-es/conversation_errors.xml index 7e254966..01166fa8 100644 --- a/apps/android/app/src/main/res/values-es/conversation_errors.xml +++ b/apps/android/app/src/main/res/values-es/conversation_errors.xml @@ -1,10 +1,10 @@ - Tu proyecto de OpenAI no tiene crédito de API disponible. Revisa la facturación y el límite de uso antes de volver a intentarlo. + Tu proyecto del proveedor de IA no tiene crédito de API disponible. Revisa la facturación y el límite de uso antes de volver a intentarlo. El servicio de voz o de aprendizaje no está disponible temporalmente. Inténtalo de nuevo en unos momentos. El servicio no pudo aceptar esta solicitud. Si el problema continúa, contacta con soporte. La solicitud tardó demasiado. Revisa tu conexión e inténtalo de nuevo. Mural no pudo conectarse. Revisa tu conexión a internet e inténtalo de nuevo. - Referencia de OpenAI: %1$s + Referencia del proveedor: %1$s Todavía se está preparando otra explicación. Espera un momento e inténtalo de nuevo. No hay suficiente saldo disponible para esta explicación. Revisa tu saldo en Cuenta. Mural todavía está comprobando tu saldo tras una solicitud anterior. Inténtalo de nuevo en unos momentos. diff --git a/apps/android/app/src/main/res/values-es/settings_parity.xml b/apps/android/app/src/main/res/values-es/settings_parity.xml index bc1897d6..380042ff 100644 --- a/apps/android/app/src/main/res/values-es/settings_parity.xml +++ b/apps/android/app/src/main/res/values-es/settings_parity.xml @@ -6,9 +6,13 @@ Cada idioma conserva sus propias palabras y progreso. Mural encuentra tu ritmo a través de la conversación. Termina esta conversación para cambiar de idioma. Cada idioma conserva sus propias palabras y progreso. Avanzado + Proveedor de IA Usar tu propia clave de API Una forma opcional de usar Mural. OpenAI te cobra directamente el uso de tu propia clave. + Una forma opcional de usar Mural. Google AI Studio te cobra directamente el uso de tu propia clave. El uso se factura a tu cuenta de OpenAI. La clave se guarda cifrada en este teléfono y solo se envía a OpenAI. + Abrir claves de Google AI Studio + El uso se factura a tu cuenta de Google AI Studio. La clave se guarda cifrada en este teléfono y solo se envía a Google. Con calma %1$d minutos diff --git a/apps/android/app/src/main/res/values-es/strings.xml b/apps/android/app/src/main/res/values-es/strings.xml index c8c1a33c..956b1e99 100644 --- a/apps/android/app/src/main/res/values-es/strings.xml +++ b/apps/android/app/src/main/res/values-es/strings.xml @@ -17,9 +17,9 @@ No se pudo abrir el historial local. No se sobrescribirá. Conserva los datos de la aplicación y revisa el archivo antes de continuar. No se pudo guardar el progreso. Exporta un respaldo antes de cerrar la aplicación. Primero resuelve el error del historial local. - Acepta el uso de OpenAI en Ajustes antes de empezar. - Añade tu clave de OpenAI en Ajustes para empezar. - Introduce una clave de la API de OpenAI válida. + Acepta el uso del proveedor de IA en Ajustes antes de empezar. + Añade la clave del proveedor de IA seleccionado en Ajustes para empezar. + Introduce una clave válida del proveedor de IA seleccionado. No se pudo guardar la clave de forma segura en este dispositivo. No se pudo guardar la clave. No se pudo eliminar la clave. Desbloquea el dispositivo e inténtalo de nuevo. @@ -31,12 +31,12 @@ No se pudo buscar el tema. La búsqueda no devolvió fuentes verificables. Prueba un tema más específico. No se pudo importar: verifica el formato de Mural, los idiomas y el límite de 30 MB. - OpenAI devolvió una respuesta incompleta. Inténtalo de nuevo. + El proveedor de IA devolvió una respuesta incompleta. Inténtalo de nuevo. Mural no pudo completar esa solicitud. Prueba con otro tema. - Tu clave de OpenAI no fue aceptada. Revísala en Ajustes. - Esta clave de API puede no tener acceso al modelo solicitado. Revisa tu proyecto de OpenAI. - OpenAI está limitando las solicitudes. Espera un momento e inténtalo de nuevo. Si el problema continúa, revisa la facturación y los límites de tu proyecto. - OpenAI no pudo completar la solicitud (HTTP %1$d). Inténtalo de nuevo. + La clave del proveedor de IA no fue aceptada. Revísala en Ajustes. + Esta clave de API puede no tener acceso al modelo solicitado. Revisa tu proyecto del proveedor. + El proveedor de IA está limitando las solicitudes. Espera un momento e inténtalo de nuevo. Si el problema continúa, revisa la facturación y los límites de tu proyecto. + El proveedor de IA no pudo completar la solicitud (HTTP %1$d). Inténtalo de nuevo. La copia supera el límite de 30 MB. No se pudo abrir el archivo elegido. Permite el acceso al micrófono en los ajustes de Android para iniciar una conversación. @@ -54,7 +54,7 @@ Clave guardada en este dispositivo. Vamos a hacerlo más sencillo. No se pudo enviar una actualización. Puedes seguir hablando. - OpenAI rechazó una actualización de voz. Si deja de responder, reinicia la conversación. + El proveedor de IA rechazó una actualización de voz. Si deja de responder, reinicia la conversación. Se alcanzó el límite de tiempo. Terminamos la sesión por inactividad. Tema listo. Pulsa Hablar o escribe para comenzar. @@ -118,6 +118,7 @@ Buscar un tema Buscando… Fuentes + Sugerencias de búsqueda de Google Hablar de esto Las fuentes quedan guardadas junto al tema. Actualidad @@ -177,11 +178,13 @@ Estimación de voz Búsquedas registradas Uso y facturación de OpenAI - La estimación de voz usa 0,05 USD/min a fecha de 11 de septiembre de 2026. La traducción, la enseñanza y las búsquedas se cobran aparte. Una solicitud interrumpida puede cobrarse sin quedar registrada aquí. Tu panel de OpenAI es la referencia. La duración orientativa es local, no un tope de gasto. + El tiempo de voz registrado es local. La traducción, la enseñanza y las búsquedas pueden cobrarse aparte. Una solicitud interrumpida puede cobrarse sin quedar registrada aquí. Tu panel de OpenAI es la referencia. La duración orientativa es local, no un tope de gasto. + Uso y facturación de Google AI Studio + Los precios y el uso de Google AI Studio varían según el modelo y el proyecto. Consulta tu panel de Google AI Studio; la duración orientativa es local, no un tope de gasto. Permiso para IA Aceptado para esta versión No aceptado - Con permiso, Mural envía audio y el texto elegido a OpenAI. Puedes usar el historial y las palabras sin concederlo. + Con permiso, Mural envía audio, el texto elegido y el texto de las búsquedas de temas al proveedor de IA seleccionado. Puedes usar el historial y las palabras sin concederlo. Retirar permiso Revisar y aceptar Tus datos @@ -191,13 +194,16 @@ Eliminar conversaciones y aprendizaje Ayuda y privacidad Controles de datos de OpenAI + Políticas de uso de Google AI Studio Avisos de código abierto - El audio y el texto elegido se envían a OpenAI durante la práctica. Mural no guarda audio sin procesar. + El audio y el texto elegido se envían al proveedor seleccionado durante la práctica. Mural no guarda audio sin procesar. + El audio y el texto elegido se envían a Google AI Studio durante la práctica. Pueden aplicarse las políticas de retención y supervisión contra abusos del proveedor. Mural no guarda audio sin procesar. Mural %1$s · Android Correcciones Con suavidad, mientras hablamos Abrir las claves de API de OpenAI Voz: GPT-Live-1 · Profesor: GPT-5.6 Luna + Voz: Gemini 3.8 Live · Profesor: Gemma 4 31B IT ¿Eliminar la clave guardada? Necesitarás otra clave para volver a usar las funciones de IA. ¿Eliminar todo el aprendizaje? @@ -206,6 +212,7 @@ ¿Retirar el permiso para IA? Las conversaciones, búsquedas y significados en línea quedarán desactivados. Tu historial local seguirá disponible. Clave de OpenAI + Clave de API de Google AI Studio Se guarda cifrada por Android y no forma parte de las copias. Nueva clave Copia exportada @@ -237,7 +244,7 @@ Antes de conversar. - Con tu permiso, Mural envía audio y el texto que elijas a OpenAI para ofrecer conversaciones, explicaciones y temas actuales. + Con tu permiso, Mural envía audio, el texto que elijas y el texto de las búsquedas de temas al proveedor de IA seleccionado para ofrecer conversaciones y significados. Se aplican las reglas de conservación del proveedor. Tu historial de aprendizaje se guarda en este dispositivo. Mural no guarda audio sin procesar. Puedes consultar tus palabras y conversaciones sin aceptar. Ver política de privacidad Aceptar y continuar diff --git a/apps/android/app/src/main/res/values/conversation_errors.xml b/apps/android/app/src/main/res/values/conversation_errors.xml index 92dca75c..8d6877ed 100644 --- a/apps/android/app/src/main/res/values/conversation_errors.xml +++ b/apps/android/app/src/main/res/values/conversation_errors.xml @@ -1,10 +1,10 @@ - Your OpenAI project has no available API credit. Check its billing and usage limit before trying again. + Your AI provider project has no available API credit. Check its billing and usage limit before trying again. The voice or teaching service is temporarily unavailable. Please try again shortly. The service could not accept this request. If this continues, contact support. The request took too long. Check your connection and try again. Mural could not connect. Check your internet connection and try again. - OpenAI reference: %1$s + Provider reference: %1$s Another explanation is still being prepared. Wait a moment, then try again. There is not enough available balance for this explanation. Check your balance in Account. Mural is still checking your balance after an earlier request. Please try again shortly. diff --git a/apps/android/app/src/main/res/values/settings_parity.xml b/apps/android/app/src/main/res/values/settings_parity.xml index 0ab66289..1d43c4a5 100644 --- a/apps/android/app/src/main/res/values/settings_parity.xml +++ b/apps/android/app/src/main/res/values/settings_parity.xml @@ -6,9 +6,13 @@ Each language keeps its own words and progress. Mural finds your pace through conversation. End this conversation to switch languages. Each language keeps its own words and progress. Advanced + AI provider Use your own API key An optional way to use Mural. OpenAI bills usage on your own key directly to you. + An optional way to use Mural. Google AI Studio bills usage on your own key directly to you. Your OpenAI account pays for usage. The key stays encrypted on this phone and is sent only to OpenAI. + Open Google AI Studio keys + Your Google AI Studio account pays for usage. The key stays encrypted on this phone and is sent only to Google. Keep it comfortable %1$d minutes diff --git a/apps/android/app/src/main/res/values/strings.xml b/apps/android/app/src/main/res/values/strings.xml index 41035788..a249b2e2 100644 --- a/apps/android/app/src/main/res/values/strings.xml +++ b/apps/android/app/src/main/res/values/strings.xml @@ -17,9 +17,9 @@ The local history couldn\'t be opened. It won\'t be overwritten. Keep the app\'s data and check the file before continuing. Progress couldn\'t be saved. Export a backup before closing the app. Resolve the local history error first. - Accept OpenAI use in Settings before you begin. - Add your OpenAI key in Settings to begin. - Enter a valid OpenAI API key. + Accept AI provider use in Settings before you begin. + Add your selected AI provider key in Settings to begin. + Enter a valid key for the selected AI provider. The key couldn\'t be saved securely on this device. The key couldn\'t be saved. The key couldn\'t be removed. Unlock this device and try again. @@ -31,12 +31,12 @@ The topic couldn\'t be found. The search didn\'t return verifiable sources. Try a more specific topic. Import failed. Check the file is a Mural backup, matches supported languages, and is under 30 MB. - OpenAI returned an incomplete response. Please try again. + The AI provider returned an incomplete response. Please try again. Mural couldn\'t complete that request. Try a different topic. - Your OpenAI key wasn\'t accepted. Check it in Settings. - This API key may not have access to the requested model. Check your OpenAI project. - OpenAI is limiting requests. Wait briefly and try again. If this continues, check your project’s billing and limits. - OpenAI couldn\'t complete the request (HTTP %1$d). Please try again. + Your AI provider key wasn\'t accepted. Check it in Settings. + This API key may not have access to the requested model. Check your provider project. + The AI provider is limiting requests. Wait briefly and try again. If this continues, check your project’s billing and limits. + The AI provider couldn\'t complete the request (HTTP %1$d). Please try again. The backup exceeds the 30 MB limit. The selected file couldn\'t be opened. Allow microphone access in Android Settings to start a conversation. @@ -118,6 +118,7 @@ Find a topic Finding something interesting… Sources + Google Search suggestions Talk about this Sources stay attached to the topic. From the world today @@ -177,11 +178,13 @@ Voice estimate Search calls recorded OpenAI usage and billing - Voice estimate uses $0.05/min as of 11 September 2026. Translation, teaching and search cost extra. Interrupted requests can be billed without a usage record here. Your OpenAI dashboard is authoritative. The time limit is local, not a billing cap. + Recorded voice time is local. Translation, teaching and search usage may be billed separately. Interrupted requests can be billed without a usage record here. Your OpenAI dashboard is authoritative. The time limit is local, not a billing cap. + Google AI Studio usage and billing + Google AI Studio pricing and usage vary by model and project. Check your Google AI Studio dashboard; the time limit is local, not a billing cap. AI permission Accepted for this version Not accepted - With permission, Mural sends audio and the text you choose to OpenAI. You can use your history and words without granting it. + With permission, Mural sends audio, the text you choose, and topic search text to the selected AI provider. You can use your history and words without granting it. Withdraw permission Review and accept Your words belong to you @@ -191,13 +194,16 @@ Delete all conversations and learning Help and privacy OpenAI data controls + Google AI Studio usage policies Open-source notices Audio and selected text go to OpenAI while you practise. Requests disable provider storage where supported; abuse-monitoring retention may still apply. Raw audio is not saved by Mural. + Audio and selected text go to Google AI Studio while you practise. Provider retention and abuse-monitoring policies may apply. Raw audio is not saved by Mural. Mural %1$s · Android Corrections Gently, as we talk Open OpenAI API keys Voice: GPT-Live-1 · Teacher: GPT-5.6 Luna + Voice: Gemini 3.8 Live · Teacher: Gemma 4 31B IT Delete the saved key? You\'ll need another key to use AI features again. Delete all learning data on this device? @@ -206,6 +212,7 @@ Withdraw AI permission? Conversations, searches, and online meanings will be turned off. Your local history stays available. OpenAI API key + Google AI Studio API key Stored encrypted by Android and never included in backups. New key Backup exported @@ -237,7 +244,7 @@ Before we talk. - With your permission, Mural sends audio and selected text to OpenAI to provide conversations and meanings. Provider retention rules apply. + With your permission, Mural sends audio, selected text, and topic search text to the selected AI provider to provide conversations and meanings. Provider retention rules apply. Your learning record is stored on this device. Mural does not save raw audio. You can keep browsing your saved words and conversations without agreeing. Privacy policy Agree and continue diff --git a/apps/android/app/src/test/java/chat/mural/core/CoreTest.kt b/apps/android/app/src/test/java/chat/mural/core/CoreTest.kt index 6f2187b7..d019089f 100644 --- a/apps/android/app/src/test/java/chat/mural/core/CoreTest.kt +++ b/apps/android/app/src/test/java/chat/mural/core/CoreTest.kt @@ -65,14 +65,16 @@ class CoreTest { assertEquals(2,merged.sessions.size) } @Test fun languageRegistryAndThemesStayStable() { - assertEquals(listOf("nb","es","en","fr","de","it","pt","zh"),LanguageRegistry.all.map { it.id }) + assertEquals(listOf("nb","es","en","fr","de","it","pt","zh","th"),LanguageRegistry.all.map { it.id }) assertEquals(24,Themes.shared.map { it.id }.toSet().size) assertEquals("Salut !",LanguageRegistry.get("fr")!!.greeting) - for ((id, locale, greeting) in listOf(Triple("de","de-DE","Hallo!"),Triple("it","it-IT","Ciao!"),Triple("pt","pt-BR","Olá!"),Triple("zh","zh-CN","你好!"))) { + for ((id, locale, greeting) in listOf(Triple("de","de-DE","Hallo!"),Triple("it","it-IT","Ciao!"),Triple("pt","pt-BR","Olá!"),Triple("zh","zh-CN","你好!"),Triple("th","th-TH","สวัสดี!"))) { assertEquals(locale,LanguageRegistry.get(id)!!.locale); assertEquals(greeting,LanguageRegistry.get(id)!!.greeting) } assertTrue(MeaningLanguages.all.contains("Chinese (Simplified)")) assertEquals("你好!",MeaningLanguages.greeting("Chinese (Simplified)")) + assertTrue(MeaningLanguages.all.contains("Thai")) + assertEquals("สวัสดี!",MeaningLanguages.greeting("Thai")) assertEquals("Norwegian",LanguageRegistry.get("nb")!!.name) assertEquals("Hei!",MeaningLanguages.greeting("Norwegian")) } diff --git a/apps/android/app/src/test/java/chat/mural/core/MandarinPinyinTest.kt b/apps/android/app/src/test/java/chat/mural/core/MandarinPinyinTest.kt index 6c02f07a..e46f686a 100644 --- a/apps/android/app/src/test/java/chat/mural/core/MandarinPinyinTest.kt +++ b/apps/android/app/src/test/java/chat/mural/core/MandarinPinyinTest.kt @@ -112,4 +112,17 @@ class MandarinPinyinTest { } assertEquals(listOf(), CaptionWords.segments("2026 — ☕️", "en", reader).mapNotNull { it.lookup }) } + + @Test fun ThaiCaptionsUsePlatformWordBoundariesWhenTheLanguageRequestsThem() { + val thaiReader = object : HanReader { + override fun words(text: String) = listOf(text) + override fun languageWords(text: String, languageID: String) = + if (languageID == "th") listOf("ฉัน", "อยาก", "ไป", "ตลาด") else null + override fun reading(word: String) = null + } + val text = "ฉันอยากไปตลาด" + val segments = CaptionWords.segments(text, "th", thaiReader) + assertEquals(text, segments.joinToString("") { it.text }) + assertEquals(listOf("ฉัน", "อยาก", "ไป", "ตลาด"), segments.mapNotNull { it.lookup }) + } } diff --git a/apps/android/app/src/test/java/chat/mural/network/APIClientTest.kt b/apps/android/app/src/test/java/chat/mural/network/APIClientTest.kt index 541220c2..b8d89b58 100644 --- a/apps/android/app/src/test/java/chat/mural/network/APIClientTest.kt +++ b/apps/android/app/src/test/java/chat/mural/network/APIClientTest.kt @@ -46,6 +46,20 @@ class APIClientTest { assertFalse(body["store"]!!.jsonPrimitive.boolean) assertEquals("Hola",result.text); assertEquals(APIUsage(12,7,0),result.usage) } + @Test fun googleProviderUsesGemmaEndpointGoogleAuthGroundingSourcesAndSearchSuggestions() = runBlocking { + val google = APIClient("AIza-fake-test-only-credential", OkHttpClient.Builder().followRedirects(false).build(), server.url("/v1/"), server.url("/v1/")) + google.provider = AIProvider.GOOGLE_AI_STUDIO + server.enqueue(MockResponse().setBody("""{"candidates":[{"finishReason":"STOP","content":{"parts":[{"text":"สวัสดี"}]},"groundingMetadata":{"webSearchQueries":["thai news","thai culture"],"searchEntryPoint":{"renderedContent":"Search suggestions"},"groundingChunks":[{"web":{"uri":"https://example.com","title":"Example"}}]}}],"usageMetadata":{"promptTokenCount":3,"candidatesTokenCount":4}}""")) + val result = google.respond("policy", "hello", search = true) + val request = server.takeRequest(2, TimeUnit.SECONDS)!! + assertEquals("/v1/models/gemma-4-31b-it:generateContent", request.path) + assertEquals("AIza-fake-test-only-credential", request.getHeader("x-goog-api-key")) + assertNull(request.getHeader("Authorization")) + assertEquals("สวัสดี", result.text) + assertEquals(APIUsage(3, 4, 2), result.usage) + assertEquals("https://example.com", result.sources.single().url) + assertTrue(result.searchEntryPointHTML.orEmpty().contains("Search suggestions")) + } @Test fun rejectsRedirectWithoutFollowingOrLeakingCredentials() = runBlocking { server.enqueue(MockResponse().setResponseCode(302).setHeader("Location",server.url("/other"))) try { api.post("responses",buildJsonObject{}); fail("accepted redirect") } catch (e: APIClient.APIException.Http) { assertEquals(302,e.status) } diff --git a/apps/ios/App/AIProvider.swift b/apps/ios/App/AIProvider.swift new file mode 100644 index 00000000..b335343c --- /dev/null +++ b/apps/ios/App/AIProvider.swift @@ -0,0 +1,78 @@ +import Foundation + +enum AIProvider: String, CaseIterable, Identifiable { + case openAI + case googleAIStudio + + var id: String { rawValue } + + var title: String { + switch self { + case .openAI: "OpenAI" + case .googleAIStudio: "Google AI Studio" + } + } + + var helperModel: String { + switch self { + case .openAI: "gpt-5.6-luna" + case .googleAIStudio: "gemma-4-31b-it" + } + } + + var liveModel: String { + switch self { + case .openAI: "gpt-live-1" + case .googleAIStudio: "gemini-3.8-live" + } + } + + var keyURL: URL { + switch self { + case .openAI: URL(string: "https://platform.openai.com/api-keys")! + case .googleAIStudio: URL(string: "https://aistudio.google.com/app/apikey")! + } + } + + var usageURL: URL { + switch self { + case .openAI: URL(string: "https://platform.openai.com/usage")! + case .googleAIStudio: URL(string: "https://aistudio.google.com/app/usage")! + } + } + + var dataControlsURL: URL { + switch self { + case .openAI: URL(string: "https://developers.openai.com/api/docs/guides/your-data")! + case .googleAIStudio: URL(string: "https://ai.google.dev/gemini-api/docs/usage-policies")! + } + } + + var keyPlaceholder: String { + switch self { + case .openAI: "OpenAI API key" + case .googleAIStudio: "Google AI Studio API key" + } + } + + var keyHelp: String { + switch self { + case .openAI: "Stored only on this iPhone. Mural sends it directly to OpenAI." + case .googleAIStudio: "Stored only on this iPhone. Mural sends it directly to Google AI Studio." + } + } +} + +enum AIProviderSelection { + private static let defaultsKey = "mural.ai-provider" + + static var current: AIProvider { + guard let rawValue = UserDefaults.standard.string(forKey: defaultsKey), + let provider = AIProvider(rawValue: rawValue) else { return .openAI } + return provider + } + + static func save(_ provider: AIProvider) { + UserDefaults.standard.set(provider.rawValue, forKey: defaultsKey) + } +} diff --git a/apps/ios/App/APIClient.swift b/apps/ios/App/APIClient.swift index 0c0ef512..bfb406a3 100644 --- a/apps/ios/App/APIClient.swift +++ b/apps/ios/App/APIClient.swift @@ -6,18 +6,29 @@ final class NoRedirect: NSObject, URLSessionTaskDelegate { } struct APIUsage { var input = 0; var output = 0; var searches = 0 } -struct APIResult { var text: String; var sources: [SourceLink]; var usage: APIUsage } +struct APIResult { + var text: String + var sources: [SourceLink] + var usage: APIUsage + var searchEntryPointHTML: String? = nil +} @MainActor final class APIClient { private let session: URLSession - init() { + private(set) var provider: AIProvider + init(provider: AIProvider = AIProviderSelection.current) { + self.provider = provider let config = URLSessionConfiguration.ephemeral config.timeoutIntervalForRequest = 45; config.timeoutIntervalForResource = 60 config.httpCookieStorage = nil; config.urlCache = nil session = URLSession(configuration: config, delegate: NoRedirect(), delegateQueue: nil) } + func setProvider(_ provider: AIProvider) { + guard self.provider != provider else { return } + self.provider = provider + } func post(_ path: String, body: [String: Any]) async throws -> [String: Any] { - guard let key = CredentialStore.read() else { throw APIError.missingKey } + guard let key = CredentialStore.read(.openAI) else { throw APIError.missingKey } var request = URLRequest(url: URL(string: "https://api.openai.com/v1/" + path)!) request.httpMethod = "POST"; request.setValue("Bearer " + key, forHTTPHeaderField: "Authorization") request.setValue("application/json", forHTTPHeaderField: "Content-Type") @@ -30,6 +41,9 @@ struct APIResult { var text: String; var sources: [SourceLink]; var usage: APIUs return json } func respond(instructions: String, input: String, schema: [String: Any]? = nil, search: Bool = false) async throws -> APIResult { + if provider == .googleAIStudio { + return try await geminiRespond(instructions: instructions, input: input, schema: schema, search: search) + } var body: [String: Any] = ["model": "gpt-5.6-luna", "store": false, "instructions": instructions, "input": [["role": "user", "content": input]], "max_output_tokens": schema == nil ? 1400 : 2200, "reasoning": ["effort": "low"]] @@ -54,6 +68,69 @@ struct APIResult { var text: String; var sources: [SourceLink]; var usage: APIUs guard !text.isEmpty else { throw APIError.incomplete } return APIResult(text: text, sources: sources, usage: usage) } + private func geminiRespond(instructions: String, input: String, schema: [String: Any]?, search: Bool) async throws -> APIResult { + guard let key = CredentialStore.read(.googleAIStudio) else { throw APIError.missingKey } + let model = AIProvider.googleAIStudio.helperModel + var generationConfig: [String: Any] = ["maxOutputTokens": schema == nil ? 1400 : 2200] + if schema != nil { + generationConfig["responseMimeType"] = "application/json" + generationConfig["responseJsonSchema"] = Self.geminiCompatibleSchema(schema!) + } + var body: [String: Any] = [ + "systemInstruction": ["parts": [["text": instructions]]], + "contents": [["role": "user", "parts": [["text": input]]]], + "generationConfig": generationConfig + ] + if search { body["tools"] = [["googleSearch": [:]]] } + let url = URL(string: "https://generativelanguage.googleapis.com/v1beta/models/\(model):generateContent")! + var request = URLRequest(url: url) + request.httpMethod = "POST" + request.setValue(key, forHTTPHeaderField: "x-goog-api-key") + request.setValue("application/json", forHTTPHeaderField: "Content-Type") + request.httpBody = try JSONSerialization.data(withJSONObject: body) + let (data, response) = try await session.data(for: request) + try Task.checkCancellation() + guard let http = response as? HTTPURLResponse else { throw APIError.invalidResponse } + guard (200..<300).contains(http.statusCode) else { + throw ProviderFailure(status: http.statusCode, body: data, reference: http.value(forHTTPHeaderField: "x-request-id"), providerName: "Google AI Studio") + } + guard let json = try JSONSerialization.jsonObject(with: data) as? [String: Any], + let candidate = (json["candidates"] as? [[String: Any]])?.first else { throw APIError.invalidResponse } + guard candidate["finishReason"] as? String == "STOP" else { throw APIError.incomplete } + var text = "" + for part in (candidate["content"] as? [String: Any])?["parts"] as? [[String: Any]] ?? [] { + text += part["text"] as? String ?? "" + } + let grounding = candidate["groundingMetadata"] as? [String: Any] + var sources: [SourceLink] = [] + for chunk in (grounding?["groundingChunks"] as? [[String: Any]]) ?? [] { + guard let web = chunk["web"] as? [String: Any], let url = web["uri"] as? String else { continue } + let source = SourceLink(title: web["title"] as? String ?? "Source", url: url) + if source.safeURL != nil && !sources.contains(where: { $0.url == url }) { sources.append(source) } + } + var usage = APIUsage() + if let metadata = json["usageMetadata"] as? [String: Any] { + usage.input = metadata["promptTokenCount"] as? Int ?? 0 + usage.output = metadata["candidatesTokenCount"] as? Int ?? 0 + } + let searchQueries = Set((grounding?["webSearchQueries"] as? [String] ?? []) + .map { $0.trimmingCharacters(in: .whitespacesAndNewlines) } + .filter { !$0.isEmpty }) + usage.searches = search ? (searchQueries.isEmpty && !sources.isEmpty ? 1 : searchQueries.count) : 0 + let searchEntryPointHTML = ((grounding?["searchEntryPoint"] as? [String: Any])?["renderedContent"] as? String).map { String($0.prefix(32_000)) } + guard !text.isEmpty else { throw APIError.incomplete } + return APIResult(text: text, sources: sources, usage: usage, searchEntryPointHTML: searchEntryPointHTML) + } + private static func geminiCompatibleSchema(_ value: Any) -> Any { + if let dictionary = value as? [String: Any] { + return dictionary.reduce(into: [String: Any]()) { result, entry in + guard entry.key != "additionalProperties" else { return } + result[entry.key] = geminiCompatibleSchema(entry.value) + } + } + if let array = value as? [Any] { return array.map { geminiCompatibleSchema($0) } } + return value + } static func object(_ fields: [String: Any]) -> [String: Any] { ["type": "object", "properties": fields, "required": fields.keys.sorted(), "additionalProperties": false] } static let string: [String: Any] = ["type": "string"] static func assessmentSchema(language: LanguageModule) -> [String: Any] { object([ @@ -69,13 +146,13 @@ struct APIResult { var text: String; var sources: [SourceLink]; var usage: APIUs case missingKey, invalidResponse, incomplete, refused, http(Int) var errorDescription: String? { switch self { - case .missingKey: "Add your OpenAI key in Settings to begin." - case .invalidResponse, .incomplete: "OpenAI returned an incomplete response. Please try again." + case .missingKey: "Add your selected AI provider key in Settings to begin." + case .invalidResponse, .incomplete: "The AI provider returned an incomplete response. Please try again." case .refused: "Mural couldn’t complete that request. Try a different topic." - case .http(401): "Your OpenAI key wasn’t accepted. Check it in Settings." - case .http(403), .http(404): "This API key may not have access to the requested model. Check your OpenAI project." - case .http(429): "OpenAI’s usage or rate limit was reached. Check your project’s billing and limits." - case .http(let status): "OpenAI couldn’t complete the request (HTTP \(status)). Please try again." + case .http(401): "Your AI provider key wasn’t accepted. Check it in Settings." + case .http(403), .http(404): "This API key may not have access to the requested model. Check your provider project." + case .http(429): "The AI provider’s usage or rate limit was reached. Check your project’s billing and limits." + case .http(let status): "The AI provider couldn’t complete the request (HTTP \(status)). Please try again." } } } diff --git a/apps/ios/App/ConversationCoordinator.swift b/apps/ios/App/ConversationCoordinator.swift index bf2cf9e4..fbb1b6e5 100644 --- a/apps/ios/App/ConversationCoordinator.swift +++ b/apps/ios/App/ConversationCoordinator.swift @@ -89,6 +89,7 @@ import MuralCore }) } var isRunning: Bool { state == .active || state == .connecting || state == .closing } + var aiProvider: AIProvider { api.provider } var language: LanguageModule { store.language } var assistantPassage: Passage? { session?.passages.last(where: { $0.speaker == .assistant }) } var userPassage: Passage? { session?.passages.last(where: { $0.speaker == .user }) } @@ -117,7 +118,7 @@ import MuralCore #if DEBUG if ProcessInfo.processInfo.arguments.contains("--preview") { showSettings = true; return } #endif - guard CredentialStore.hasKey else { showSettings = true; return } + guard CredentialStore.hasKey(for: api.provider) else { showSettings = true; return } cancelReset(); meanings.reset() error = nil; notice = nil; lastAssessmentKey = "" lastLanguageCheck = ""; pendingCommands = [:] @@ -165,6 +166,14 @@ import MuralCore inputLevel = 0; outputLevel = 0; state = .idle; isMuted = false store.selectLanguage(id) } + func selectAIProvider(_ provider: AIProvider) { + guard !isRunning else { return } + let providerChanged = api.provider != provider + api.setProvider(provider) + AIProviderSelection.save(provider) + if providerChanged { store.updatePreferences { $0.aiConsentVersion = nil } } + error = nil + } func selectMeaningLanguage(_ value: String) { guard MeaningLanguages.all.contains(value) else { return } meanings.reset() @@ -232,6 +241,9 @@ import MuralCore closeTask?.cancel(); durationTask?.cancel(); connectionTask?.cancel() assessmentTask?.cancel(); saveTask?.cancel(); saveTask = nil delegationTasks.values.forEach { $0.cancel() }; delegationTasks.removeAll() + if session?.providerID == "gemini-live", let startedAt = session?.startedAt { + session?.voiceSeconds = max(0, Date().timeIntervalSince(startedAt)) + } transport.disconnect(); pendingCommands = [:]; working = false session?.endedAt = .now; session?.usageFinal = final save(); state = .ended @@ -272,8 +284,10 @@ import MuralCore guard let type = event["type"] as? String, session != nil else { return } switch type { case "mural.session.created": - session?.providerID = (event["session"] as? [String: Any])?["id"] as? String - session?.voiceSeconds = 15; save() + let providerID = (event["session"] as? [String: Any])?["id"] as? String + session?.providerID = providerID + if providerID != "gemini-live" { session?.voiceSeconds = 15 } + save() case "session.started": guard state == .connecting else { return } state = .active; activity = ConversationActivity(now: activityNow); conversationPace = ConversationPace(); inactivitySeconds = nil @@ -479,7 +493,7 @@ import MuralCore guard self.session?.id == snapshot.id, self.state == .active else { return } self.addUsage(result.usage) if !result.sources.isEmpty { - self.session?.topics.append(TopicBrief(languageID: targetLanguage.id, query: "From our conversation", text: result.text, sources: result.sources)) + self.session?.topics.append(TopicBrief(languageID: targetLanguage.id, query: "From our conversation", text: result.text, sources: result.sources, searchEntryPointHTML: result.searchEntryPointHTML)) } self.append("commentary", result.text, delegationID: id); self.save() } catch is CancellationError { } @@ -557,7 +571,7 @@ import MuralCore let result = try await api.respond(instructions: TeachingPolicy.currentTopic(language: targetLanguage), input: String(query.prefix(500)), search: true) guard generation == languageGeneration else { throw CancellationError() } guard !result.sources.isEmpty else { throw TopicError.unsourced } - let brief = TopicBrief(languageID: targetLanguage.id, query: query, text: result.text, sources: result.sources) + let brief = TopicBrief(languageID: targetLanguage.id, query: query, text: result.text, sources: result.sources, searchEntryPointHTML: result.searchEntryPointHTML) if session == nil || !isRunning { var saved = SessionRecord(languageID: targetLanguage.id, title: query); saved.endedAt = .now; saved.topics = [brief] saved.inputTokens = result.usage.input; saved.outputTokens = result.usage.output; saved.searchCalls = result.usage.searches; store.save(saved) diff --git a/apps/ios/App/GeminiLiveTransport.swift b/apps/ios/App/GeminiLiveTransport.swift new file mode 100644 index 00000000..e7756da9 --- /dev/null +++ b/apps/ios/App/GeminiLiveTransport.swift @@ -0,0 +1,461 @@ +import Foundation +import AVFoundation + +/// Direct Google AI Studio Live API transport. The Live API uses Gemini's native +/// audio model, while Gemma remains the text teacher. +@MainActor final class GeminiLiveTransport: NSObject { + private struct ControlMessage { + let id: UUID + let payload: [String: Any] + } + + private struct InFlightControlMessage { + let message: ControlMessage + let socket: URLSessionWebSocketTask + let generation: Int + } + + var onEvent: (([String: Any]) -> Void)? + var onLevels: ((Double, Double) -> Void)? + var onFailure: ((String) -> Void)? + + private var socket: URLSessionWebSocketTask? + private var session: URLSession? + private var receiveTask: Task? + private var audioEngine: AVAudioEngine? + private var player: AVAudioPlayerNode? + private var ownsAudioActivation = false + private var muted = false + private var closing = false + private var connectedAt = Date() + private var inputLevel = 0.0 + private var pendingHistory: [[String: Any]] = [] + private var pendingControlMessages: [ControlMessage] = [] + private var inFlightControlMessage: InFlightControlMessage? + private var connectionGeneration = 0 + private var sessionStarted = false + private var usageReported = false + private var apiKey = "" + private var instructions = "" + private var socketReady = false + private var sessionResumptionHandle: String? + private var activeResumptionHandle: String? + private var reconnectTask: Task? + private var reconnectDeadline: Date? + private var reconnectAttempts = 0 + + func connect(key: String, instructions: String, history: [[String: Any]]) async throws { + disconnect() + closing = false; muted = false; connectedAt = .now; pendingHistory = history; sessionStarted = false; usageReported = false + apiKey = key; self.instructions = instructions + sessionResumptionHandle = nil; activeResumptionHandle = nil; reconnectAttempts = 0 + let granted = await AVAudioApplication.requestRecordPermission() + guard granted else { throw LiveTransport.TransportError.microphone } + try Task.checkCancellation() + + let audio = AVAudioSession.sharedInstance() + try audio.setCategory(.playAndRecord, mode: .voiceChat, options: [.defaultToSpeaker, .allowBluetoothHFP]) + try audio.setActive(true) + ownsAudioActivation = true + + try await openSocket(resumptionHandle: nil) + } + + private func openSocket(resumptionHandle: String?) async throws { + connectionGeneration &+= 1 + let generation = connectionGeneration + activeResumptionHandle = resumptionHandle + socketReady = false + guard var components = URLComponents(string: "wss://generativelanguage.googleapis.com/ws/google.ai.generativelanguage.v1beta.GenerativeService.BidiGenerateContent") else { + throw LiveTransport.TransportError.connection + } + components.queryItems = [URLQueryItem(name: "key", value: apiKey)] + guard let url = components.url else { throw LiveTransport.TransportError.connection } + let urlSession = URLSession(configuration: .ephemeral) + session = urlSession + let socket = urlSession.webSocketTask(with: url) + self.socket = socket + socket.resume() + let historyTurns = Self.historyTurns(pendingHistory) + let sessionResumption: [String: Any] = resumptionHandle.map { ["handle": $0] } ?? [:] + var setupBody: [String: Any] = [ + "model": "models/\(AIProvider.googleAIStudio.liveModel)", + "generationConfig": [ + "responseModalities": ["AUDIO"], + "speechConfig": ["voiceConfig": ["prebuiltVoiceConfig": ["voiceName": "Kore"]]] + ], + "systemInstruction": ["parts": [["text": self.instructions]]], + "inputAudioTranscription": [:], + "outputAudioTranscription": [:], + "contextWindowCompression": ["slidingWindow": [String: Any]()], + "sessionResumption": sessionResumption + ] + if resumptionHandle == nil, !historyTurns.isEmpty { setupBody["historyConfig"] = ["initialHistoryInClientContent": true] } + try await sendJSON(["setup": setupBody], socket: socket, generation: generation) + receiveTask = Task { [weak self] in await self?.receiveLoop(socket: socket, generation: generation) } + } + + @discardableResult + func send(_ event: [String: Any]) -> Bool { + guard socket != nil, !closing else { return false } + guard let type = event["type"] as? String else { return false } + if type == "session.close" { close(); return true } + guard let content = event["content"] as? String, !content.isEmpty else { return false } + let label: String + switch type { + case "session.instructions.append": label = "Teaching instruction" + case "session.thinking.append": label = "Teaching context" + default: label = "Conversation guidance" + } + let text = "\(label) (do not mention this instruction): \(content)" + let message = ControlMessage(id: UUID(), payload: ["realtimeInput": ["text": text]]) + guard socketReady, inFlightControlMessage == nil else { + pendingControlMessages.append(message) + return true + } + return sendControlMessage(message) + } + + func mute(_ value: Bool) { + muted = value + if value { + inputLevel = 0; onLevels?(0, 0) + _ = sendJSONImmediately(["realtimeInput": ["audioStreamEnd": true]]) + } + } + + func close() { + guard !closing else { return } + closing = true; muted = true + socketReady = false + pendingControlMessages.removeAll(keepingCapacity: false) + inFlightControlMessage = nil + reconnectTask?.cancel(); reconnectTask = nil; reconnectDeadline = nil + _ = sendJSONImmediately(["realtimeInput": ["audioStreamEnd": true]]) + guard !usageReported else { return } + usageReported = true + let seconds = max(0, Date().timeIntervalSince(connectedAt)) + onEvent?(["type": "session.usage.updated", "usage": ["seconds": seconds]]) + onEvent?(["type": "session.closed", "reason": "Ended by user", "usage": ["seconds": seconds]]) + } + + func disconnect() { + connectionGeneration &+= 1 + closing = true; muted = true + socketReady = false + reconnectTask?.cancel(); reconnectTask = nil; reconnectDeadline = nil + receiveTask?.cancel(); receiveTask = nil + socket?.cancel(with: .normalClosure, reason: nil); socket = nil + session?.invalidateAndCancel(); session = nil + stopAudio() + apiKey = ""; instructions = ""; pendingHistory = []; pendingControlMessages = []; inFlightControlMessage = nil; sessionResumptionHandle = nil; activeResumptionHandle = nil + onLevels?(0, 0) + } + + private func startAudio() throws { + let engine = AVAudioEngine() + let input = engine.inputNode + let sourceFormat = input.inputFormat(forBus: 0) + guard let targetFormat = AVAudioFormat(commonFormat: .pcmFormatInt16, sampleRate: 16_000, channels: 1, interleaved: true), + let converter = AVAudioConverter(from: sourceFormat, to: targetFormat) else { throw LiveTransport.TransportError.connection } + input.installTap(onBus: 0, bufferSize: 1024, format: sourceFormat) { [weak self] buffer, _ in + guard let self, !self.muted else { return } + let ratio = targetFormat.sampleRate / max(sourceFormat.sampleRate, 1) + let capacity = AVAudioFrameCount(Double(buffer.frameLength) * ratio + 2) + guard let converted = AVAudioPCMBuffer(pcmFormat: targetFormat, frameCapacity: capacity) else { return } + var supplied = false + var conversionError: NSError? + converter.convert(to: converted, error: &conversionError) { _, status in + if supplied { status.pointee = .noDataNow; return nil } + supplied = true; status.pointee = .haveData; return buffer + } + guard conversionError == nil, converted.frameLength > 0, + let raw = converted.audioBufferList.pointee.mBuffers.mData else { return } + let byteCount = Int(converted.frameLength) * Int(targetFormat.streamDescription.pointee.mBytesPerFrame) + let data = Data(bytes: raw, count: byteCount) + Task { @MainActor [weak self] in self?.sendAudio(data) } + } + let player = AVAudioPlayerNode() + let outputFormat = AVAudioFormat(commonFormat: .pcmFormatInt16, sampleRate: 24_000, channels: 1, interleaved: true)! + engine.attach(player) + engine.connect(player, to: engine.mainMixerNode, format: outputFormat) + try engine.start() + self.audioEngine = engine; self.player = player + } + + private func stopAudio() { + audioEngine?.inputNode.removeTap(onBus: 0) + audioEngine?.stop(); audioEngine = nil + player?.stop(); player = nil + if ownsAudioActivation { + let audio = AVAudioSession.sharedInstance() + try? audio.setActive(false) + ownsAudioActivation = false + } + } + + private func sendAudio(_ data: Data) { + guard !muted, !closing, socketReady, socket != nil else { return } + inputLevel = min(1, Self.rms(data) * 7) + onLevels?(inputLevel, player?.isPlaying == true ? 0.35 : 0) + _ = sendJSONImmediately(["realtimeInput": ["audio": [ + "mimeType": "audio/pcm;rate=16000", "data": data.base64EncodedString() + ]]]) + } + + private func sendHistory(_ history: [[String: Any]], socket: URLSessionWebSocketTask, generation: Int) { + let turns = Self.historyTurns(history) + guard !turns.isEmpty else { return } + _ = sendJSONImmediately(["clientContent": ["turns": turns, "turnComplete": true]], socket: socket, generation: generation) + } + + private static func historyTurns(_ history: [[String: Any]]) -> [[String: Any]] { + history.compactMap { item -> [String: Any]? in + guard let role = item["role"] as? String, let text = item["text"] as? String else { return nil } + return ["role": role == "assistant" ? "model" : "user", "parts": [["text": text]]] + } + } + + private func sendJSONImmediately(_ object: [String: Any], socket expectedSocket: URLSessionWebSocketTask? = nil, generation expectedGeneration: Int? = nil) -> Bool { + guard let socket, (expectedSocket == nil || socket === expectedSocket), + (expectedGeneration == nil || connectionGeneration == expectedGeneration), + let data = try? JSONSerialization.data(withJSONObject: object), + let text = String(data: data, encoding: .utf8) else { return false } + let sendGeneration = connectionGeneration + socket.send(.string(text)) { [weak self] error in + guard let self, error != nil else { return } + Task { @MainActor in + if !self.closing, self.socket === socket, self.connectionGeneration == sendGeneration { + self.connectionEnded(generation: sendGeneration, socket: socket) + } + } + } + return true + } + + private func sendControlMessage(_ message: ControlMessage) -> Bool { + guard let socket, + socketReady, + inFlightControlMessage == nil, + let data = try? JSONSerialization.data(withJSONObject: message.payload), + let text = String(data: data, encoding: .utf8) else { return false } + let generation = connectionGeneration + inFlightControlMessage = InFlightControlMessage(message: message, socket: socket, generation: generation) + socket.send(.string(text)) { [weak self] error in + let succeeded = error == nil + Task { @MainActor [weak self] in + self?.completeControlMessage(id: message.id, socket: socket, generation: generation, succeeded: succeeded) + } + } + return true + } + + private func completeControlMessage(id: UUID, socket: URLSessionWebSocketTask, generation: Int, succeeded: Bool) { + guard let inFlight = inFlightControlMessage, + inFlight.message.id == id, + inFlight.socket === socket, + inFlight.generation == generation else { return } + if !succeeded { + connectionEnded(generation: generation, socket: socket) + return + } + inFlightControlMessage = nil + guard !closing, self.socket === socket, connectionGeneration == generation else { return } + flushPendingControlMessages() + } + + private func requeueInFlightControlMessage(socket: URLSessionWebSocketTask, generation: Int) { + guard let inFlight = inFlightControlMessage, + inFlight.socket === socket, + inFlight.generation == generation else { return } + inFlightControlMessage = nil + pendingControlMessages.insert(inFlight.message, at: 0) + } + + private func sendJSON(_ object: [String: Any], socket expectedSocket: URLSessionWebSocketTask? = nil, generation expectedGeneration: Int? = nil) async throws { + guard let socket, (expectedSocket == nil || socket === expectedSocket), + (expectedGeneration == nil || connectionGeneration == expectedGeneration), + let data = try? JSONSerialization.data(withJSONObject: object), + let text = String(data: data, encoding: .utf8) else { throw LiveTransport.TransportError.connection } + try await socket.send(.string(text)) + } + + private func receiveLoop(socket: URLSessionWebSocketTask, generation: Int) async { + while !Task.isCancelled { + guard self.socket === socket, self.connectionGeneration == generation, !closing else { return } + do { + let message = try await socket.receive() + guard self.socket === socket, self.connectionGeneration == generation else { return } + switch message { + case .string(let text): + if let data = text.data(using: .utf8), let json = try? JSONSerialization.jsonObject(with: data) as? [String: Any] { handle(json, generation: generation) } + case .data(let data): + if let json = try? JSONSerialization.jsonObject(with: data) as? [String: Any] { handle(json, generation: generation) } + @unknown default: break + } + } catch { + if self.socket === socket, self.connectionGeneration == generation, !closing { + connectionEnded(generation: generation, socket: socket) + } + return + } + } + } + + private func connectionEnded(generation: Int, socket expectedSocket: URLSessionWebSocketTask? = nil) { + guard self.connectionGeneration == generation, !closing else { return } + if let expectedSocket { + requeueInFlightControlMessage(socket: expectedSocket, generation: generation) + } + socketReady = false + if sessionResumptionHandle?.isEmpty == false { + scheduleReconnect(after: 0) + } else { + fail("The Google AI Studio voice connection ended. Check your key and connection, then try again.") + } + } + + private func scheduleReconnect(after delay: TimeInterval) { + guard !closing else { return } + guard sessionResumptionHandle?.isEmpty == false else { + fail("The Google AI Studio voice connection ended. Check your key and connection, then try again.") + return + } + let deadline = Date().addingTimeInterval(max(0, delay)) + if let reconnectDeadline, reconnectTask != nil, deadline >= reconnectDeadline { return } + reconnectTask?.cancel() + reconnectDeadline = deadline + let nanoseconds = UInt64(max(0, deadline.timeIntervalSinceNow) * 1_000_000_000) + reconnectTask = Task { [weak self] in + do { try await Task.sleep(nanoseconds: nanoseconds) } catch { return } + guard !Task.isCancelled else { return } + await self?.reconnect() + } + } + + private func reconnect() async { + reconnectTask = nil; reconnectDeadline = nil + guard !closing, let handle = sessionResumptionHandle?.trimmingCharacters(in: .whitespacesAndNewlines), !handle.isEmpty else { return } + if reconnectAttempts >= 5 { + fail("The Google AI Studio voice connection could not be resumed. Check your connection, then try again.") + return + } + reconnectAttempts += 1 + if let socket { + requeueInFlightControlMessage(socket: socket, generation: connectionGeneration) + } + receiveTask?.cancel(); receiveTask = nil + socket?.cancel(with: .normalClosure, reason: nil); socket = nil + session?.invalidateAndCancel(); session = nil + do { + try await openSocket(resumptionHandle: handle) + } catch { + if !closing { scheduleReconnect(after: 1) } + } + } + + private func fail(_ message: String) { + reconnectTask?.cancel(); reconnectTask = nil; reconnectDeadline = nil + guard !closing else { return } + onFailure?(message) + } + + private func handle(_ json: [String: Any], generation: Int) { + guard self.connectionGeneration == generation, !closing else { return } + if let update = json["sessionResumptionUpdate"] as? [String: Any], + let handle = update["newHandle"] as? String, + !handle.trimmingCharacters(in: .whitespacesAndNewlines).isEmpty { + sessionResumptionHandle = handle + } + if let goAway = json["goAway"] as? [String: Any] { + scheduleReconnect(after: Self.timeInterval(from: goAway["timeLeft"])) + return + } + if json["setupComplete"] != nil { + do { + if audioEngine == nil { try startAudio() } + } catch { + socketReady = false + onFailure?("The Google AI Studio microphone couldn’t start.") + return + } + let firstSession = !sessionStarted + if firstSession, activeResumptionHandle == nil, !pendingHistory.isEmpty, let socket { + sendHistory(pendingHistory, socket: socket, generation: generation) + } + socketReady = true + flushPendingControlMessages() + reconnectAttempts = 0 + if firstSession { + sessionStarted = true + let session: [String: Any] = ["id": "gemini-live", "model": AIProvider.googleAIStudio.liveModel] + onEvent?(["type": "mural.session.created", "session": session]) + onEvent?(["type": "session.started", "session": session]) + } + return + } + if let error = json["error"] as? [String: Any] { + let message = error["message"] as? String ?? "The Google AI Studio voice request failed." + onFailure?(String(message.prefix(240))) + return + } + guard let content = json["serverContent"] as? [String: Any] else { return } + if content["interrupted"] as? Bool == true { + player?.stop() + onLevels?(muted ? 0 : inputLevel, 0) + } + let now = max(0, Int(Date().timeIntervalSince(connectedAt) * 1000)) + if let input = content["inputTranscription"] as? [String: Any], let text = input["text"] as? String, !text.isEmpty { + onEvent?(["type": "session.input_transcript.delta", "event_id": UUID().uuidString, "delta": text, "start_ms": max(0, now - 1), "end_ms": now]) + } + if let output = content["outputTranscription"] as? [String: Any], let text = output["text"] as? String, !text.isEmpty { + onEvent?(["type": "session.output_transcript.delta", "event_id": UUID().uuidString, "delta": text, "start_ms": max(0, now - 1), "end_ms": now]) + } + for part in (content["modelTurn"] as? [String: Any])?["parts"] as? [[String: Any]] ?? [] { + guard let inline = part["inlineData"] as? [String: Any], let encoded = inline["data"] as? String, + let data = Data(base64Encoded: encoded) else { continue } + playAudio(data) + } + } + + private func flushPendingControlMessages() { + guard socketReady, inFlightControlMessage == nil, let message = pendingControlMessages.first else { return } + pendingControlMessages.removeFirst() + if !sendControlMessage(message) { + pendingControlMessages.insert(message, at: 0) + socketReady = false + } + } + + private func playAudio(_ data: Data) { + guard let player else { return } + let format = AVAudioFormat(commonFormat: .pcmFormatInt16, sampleRate: 24_000, channels: 1, interleaved: true)! + let frames = AVAudioFrameCount(data.count / Int(format.streamDescription.pointee.mBytesPerFrame)) + guard let buffer = AVAudioPCMBuffer(pcmFormat: format, frameCapacity: frames), frames > 0, + let destination = buffer.audioBufferList.pointee.mBuffers.mData else { return } + data.copyBytes(to: destination.assumingMemoryBound(to: UInt8.self), count: data.count) + buffer.frameLength = frames + player.scheduleBuffer(buffer) + if !player.isPlaying { player.play() } + onLevels?(muted ? 0 : inputLevel, 0.35) + } + + private static func timeInterval(from value: Any?) -> TimeInterval { + if let number = value as? NSNumber { return max(0, number.doubleValue) } + guard let raw = value as? String else { return 0 } + let text = raw.trimmingCharacters(in: .whitespacesAndNewlines).lowercased() + if text.hasSuffix("ms") { return max(0, (Double(String(text.dropLast(2))) ?? 0) / 1_000) } + if text.hasSuffix("s") { return max(0, Double(String(text.dropLast())) ?? 0) } + return max(0, Double(text) ?? 0) + } + + private static func rms(_ data: Data) -> Double { + guard data.count >= 2 else { return 0 } + var total = 0.0 + data.withUnsafeBytes { raw in + let samples = raw.bindMemory(to: Int16.self) + for sample in samples { let value = Double(sample) / Double(Int16.max); total += value * value } + } + return sqrt(total / Double(max(1, data.count / 2))) + } +} diff --git a/apps/ios/App/Info.plist b/apps/ios/App/Info.plist index 20dfc2ca..7e1a5908 100644 --- a/apps/ios/App/Info.plist +++ b/apps/ios/App/Info.plist @@ -9,7 +9,7 @@ CFBundleShortVersionString$(MARKETING_VERSION) CFBundleVersion$(CURRENT_PROJECT_VERSION) LSRequiresIPhoneOS -NSMicrophoneUsageDescriptionMural listens while you practise speaking. Your voice is sent to OpenAI during a conversation. +NSMicrophoneUsageDescriptionMural listens while you practise speaking. Your voice is sent to the AI provider you select during a conversation. UIUserInterfaceStyleLight UILaunchScreen UIApplicationSceneManifestUIApplicationSupportsMultipleScenes diff --git a/apps/ios/App/LibraryViews.swift b/apps/ios/App/LibraryViews.swift index c6dddb92..84a9df75 100644 --- a/apps/ios/App/LibraryViews.swift +++ b/apps/ios/App/LibraryViews.swift @@ -1,5 +1,7 @@ import SwiftUI import UniformTypeIdentifiers +import UIKit +import WebKit import MuralCore struct ThemesView: View { @@ -72,11 +74,11 @@ struct CurrentTopicView: View { if let error { Text(error).font(.footnote).foregroundStyle(MuralColor.secondary) } if let brief { Text(.init(brief.text)).font(.body).textSelection(.enabled) - SourcesView(sources: brief.sources, date: brief.retrievedAt) + SourcesView(sources: brief.sources, date: brief.retrievedAt, searchEntryPointHTML: brief.searchEntryPointHTML) Button("Talk about this", systemImage: "waveform") { coordinator.discuss(brief); selected(); dismiss() } .font(.headline).padding(18).frame(maxWidth: .infinity).background(MuralColor.orange, in: Capsule()) } - Text("Search uses your OpenAI API account. Sources stay attached to the topic.").font(.footnote).foregroundStyle(MuralColor.secondary) + Text("Search uses your \(coordinator.aiProvider.title) account. Sources stay attached to the topic.").font(.footnote).foregroundStyle(MuralColor.secondary) }.padding(26) }.background(MuralColor.cream).foregroundStyle(MuralColor.ink) .toolbar { ToolbarItem(placement: .cancellationAction) { Button("Close") { dismiss() } } } @@ -164,14 +166,59 @@ struct WordDetailView: View { struct SourcesView: View { var sources: [SourceLink] var date: Date + var searchEntryPointHTML: String? = nil var body: some View { VStack(alignment: .leading, spacing: 12) { + if let searchEntryPointHTML, !searchEntryPointHTML.isEmpty { + Text("Google Search suggestions").font(.caption).foregroundStyle(MuralColor.secondary) + GoogleSearchSuggestionsView(html: searchEntryPointHTML).frame(height: 100).clipShape(RoundedRectangle(cornerRadius: 14)) + } Text("Sources · \(date.formatted(date: .abbreviated, time: .omitted))").font(.caption).foregroundStyle(MuralColor.secondary) ForEach(sources) { source in if let url = source.safeURL { Link(destination: url) { Label(source.title, systemImage: "arrow.up.right").font(.subheadline) } } } } } } +struct GoogleSearchSuggestionsView: UIViewRepresentable { + let html: String + + func makeCoordinator() -> Coordinator { Coordinator() } + + func makeUIView(context: Context) -> WKWebView { + let configuration = WKWebViewConfiguration() + configuration.preferences.javaScriptEnabled = false + let view = WKWebView(frame: .zero, configuration: configuration) + view.isOpaque = false + view.backgroundColor = .clear + view.scrollView.isScrollEnabled = false + view.navigationDelegate = context.coordinator + return view + } + + func updateUIView(_ view: WKWebView, context: Context) { + guard context.coordinator.loadedHTML != html else { return } + context.coordinator.loadedHTML = html + let policy = "" + view.loadHTMLString(policy + html, baseURL: URL(string: "https://www.google.com")) + } + + final class Coordinator: NSObject, WKNavigationDelegate { + var loadedHTML: String? + + func webView(_ webView: WKWebView, decidePolicyFor action: WKNavigationAction, decisionHandler: @escaping (WKNavigationActionPolicy) -> Void) { + guard action.navigationType == .linkActivated, + let url = action.request.url, + url.scheme == "https", + url.host != nil else { + decisionHandler(.cancel) + return + } + UIApplication.shared.open(url) + decisionHandler(.cancel) + } + } +} + struct TranscriptView: View { let session: SessionRecord? var meaningLanguage = "English" @@ -192,7 +239,7 @@ struct TranscriptView: View { } }.frame(maxWidth: .infinity, alignment: .leading) } - ForEach(session.topics) { topic in Text(.init(topic.text)); SourcesView(sources: topic.sources, date: topic.retrievedAt) } + ForEach(session.topics) { topic in Text(.init(topic.text)); SourcesView(sources: topic.sources, date: topic.retrievedAt, searchEntryPointHTML: topic.searchEntryPointHTML) } if session.fragments.isEmpty && session.topics.isEmpty { Text("Your conversation will appear here.").foregroundStyle(MuralColor.secondary) } } else { Text("Start a conversation and your words will appear here.") } }.padding(26) @@ -263,7 +310,7 @@ struct EditableTranscriptView: View { if session?.languageID == "zh" { PinyinHelp(text: passage.text) } } } - ForEach(session?.topics ?? []) { topic in Text(.init(topic.text)); SourcesView(sources: topic.sources, date: topic.retrievedAt) } + ForEach(session?.topics ?? []) { topic in Text(.init(topic.text)); SourcesView(sources: topic.sources, date: topic.retrievedAt, searchEntryPointHTML: topic.searchEntryPointHTML) } }.padding(26) }.background(MuralColor.cream).foregroundStyle(MuralColor.ink) .navigationTitle("Our conversation").navigationBarTitleDisplayMode(.inline) @@ -320,38 +367,48 @@ struct SettingsView: View { } } Section { + Picker("AI provider", selection: Binding(get: { coordinator.aiProvider }, set: { provider in + coordinator.selectAIProvider(provider) + key = "" + hasKey = CredentialStore.hasKey(for: provider) + })) { + ForEach(AIProvider.allCases) { provider in Text(provider.title).tag(provider) } + }.disabled(coordinator.isRunning) DisclosureGroup(isExpanded: $showingAPIKey) { if hasKey { Label("Your key is saved on this iPhone", systemImage: "checkmark.shield") } - SecureField(hasKey ? "Replace OpenAI key" : "OpenAI API key", text: $key) + SecureField(hasKey ? "Replace \(coordinator.aiProvider.title) key" : coordinator.aiProvider.keyPlaceholder, text: $key) .textInputAutocapitalization(.never).autocorrectionDisabled().privacySensitive().accessibilityIdentifier("api-key") Button(hasKey ? "Save replacement key" : "Save key") { - do { try CredentialStore.save(key); key = ""; hasKey = true; message = "Saved securely. Start a conversation to connect." } + do { try CredentialStore.save(key, for: coordinator.aiProvider); key = ""; hasKey = true; message = "Saved securely. Start a conversation to connect." } catch { message = error.localizedDescription } }.disabled(key.isEmpty || coordinator.isRunning) - Link("Open OpenAI API keys", destination: URL(string: "https://platform.openai.com/api-keys")!) + Link("Open \(coordinator.aiProvider.title) keys", destination: coordinator.aiProvider.keyURL) if hasKey { Button("Remove key", role: .destructive) { - do { try CredentialStore.delete(); hasKey = false; message = "Your key has been removed." } + do { try CredentialStore.delete(for: coordinator.aiProvider); hasKey = false; message = "Your key has been removed." } catch { message = error.localizedDescription } }.disabled(coordinator.isRunning) } - Text("Your OpenAI account pays for usage. The key stays in this iPhone’s Keychain and is sent only to OpenAI.") + Text(coordinator.aiProvider.keyHelp) .font(.footnote).foregroundStyle(MuralColor.secondary) } label: { Label("Use your own API key", systemImage: "key").accessibilityIdentifier("advanced-api-key") } if let message { Text(message).font(.footnote).foregroundStyle(MuralColor.secondary) } } header: { Text("Advanced") } footer: { - if !hasKey { Text("This version uses your OpenAI API key to start a conversation.") } - } + if !hasKey { Text("Add a \(coordinator.aiProvider.title) API key to start a conversation.") } + }.onAppear { hasKey = CredentialStore.hasKey(for: coordinator.aiProvider) } Section { Picker("Conversation limit", selection: Binding(get: { store.preferences.sessionMinutes }, set: { value in store.updatePreferences { $0.sessionMinutes = value } })) { ForEach([5, 10, 15, 20, 30, 60], id: \.self) { Text("\($0) minutes").tag($0) } } LabeledContent("Recorded voice time", value: "\(Int(totalVoiceSeconds / 60)) min \(Int(totalVoiceSeconds) % 60) sec") - LabeledContent("Voice estimate", value: String(format: "$%.2f USD", totalVoiceSeconds / 60 * 0.05)) LabeledContent("Search calls recorded", value: "\(store.sessions.reduce(0) { $0 + $1.searchCalls })") - Link("OpenAI usage and billing", destination: URL(string: "https://platform.openai.com/usage")!) + Link("\(coordinator.aiProvider.title) usage and billing", destination: coordinator.aiProvider.usageURL) } header: { Text("Keep it comfortable") } footer: { - Text("Voice estimate uses $0.05/min as of 11 September 2026. Translation, teaching and search cost extra. Interrupted requests can be billed without a usage record here. Your OpenAI dashboard is authoritative. The time limit is local, not a billing cap.") + if coordinator.aiProvider == .openAI { + Text("Recorded voice time is local. Translation, teaching and search usage may be billed separately. Interrupted requests can be billed without a usage record here. Your OpenAI dashboard is authoritative. The time limit is local, not a billing cap.") + } else { + Text("Google AI Studio pricing and usage vary by model and project. Check your Google AI Studio dashboard; the time limit is local, not a billing cap.") + } } Section { Button("Export learning backup", systemImage: "square.and.arrow.up") { @@ -372,9 +429,9 @@ struct SettingsView: View { } header: { Text("Help and privacy") } Section { Text("Mural 0.1 · Personal build").font(.footnote) - Text("Voice: GPT-Live-1 · Teacher: GPT-5.6 Luna").font(.footnote) - Link("OpenAI data controls", destination: URL(string: "https://developers.openai.com/api/docs/guides/your-data")!) - Text("Audio and selected text go to OpenAI while you practise. Requests disable provider storage where supported; abuse-monitoring retention may still apply. Raw audio is not saved by Mural.").font(.footnote) + Text("Voice: \(coordinator.aiProvider.liveModel) · Teacher: \(coordinator.aiProvider.helperModel)").font(.footnote) + Link("\(coordinator.aiProvider.title) data controls", destination: coordinator.aiProvider.dataControlsURL) + Text("Audio and selected text go to \(coordinator.aiProvider.title) while you practise. Provider retention and abuse-monitoring policies may apply. Raw audio is not saved by Mural.").font(.footnote) Button("Open-source notices") { notices = true } } }.scrollContentBackground(.hidden).background(MuralColor.cream).tint(MuralColor.secondary) diff --git a/apps/ios/App/LiveTransport.swift b/apps/ios/App/LiveTransport.swift index 0d876892..39373fa0 100644 --- a/apps/ios/App/LiveTransport.swift +++ b/apps/ios/App/LiveTransport.swift @@ -20,6 +20,21 @@ enum ConnectionState: Equatable { case idle, connecting, active, closing, ended, private var closing = false private var ownsAudioActivation = false private var lastInput = 0.0, lastOutput = 0.0 + private var usingGemini = false + private lazy var gemini: GeminiLiveTransport = { + let transport = GeminiLiveTransport() + transport.onEvent = { [weak self] event in + guard let self else { return } + if event["type"] as? String == "session.started" { self.started = true } + self.onEvent?(event) + } + transport.onLevels = { [weak self] input, output in self?.onLevels?(input, output) } + transport.onFailure = { [weak self] message in + guard let self, self.usingGemini else { return } + self.onFailure?(message) + } + return transport + }() private lazy var networkRecovery = makeNetworkRecovery() private func makeNetworkRecovery(timeout: Duration = .seconds(8)) -> VoiceConnectionRecovery { VoiceConnectionRecovery(timeout: timeout) { [weak self] in @@ -32,6 +47,18 @@ enum ConnectionState: Equatable { case idle, connecting, active, closing, ended, disconnect() closing = false let token = UUID(); attempt = token + if api.provider == .googleAIStudio { + guard let key = CredentialStore.read(.googleAIStudio) else { throw APIClient.APIError.missingKey } + usingGemini = true; started = false + do { + try await gemini.connect(key: key, instructions: instructions, history: history) + guard attempt == token else { gemini.disconnect(); usingGemini = false; throw CancellationError() } + return + } catch { + gemini.disconnect(); usingGemini = false + throw error + } + } let granted = await AVAudioApplication.requestRecordPermission() guard granted else { throw TransportError.microphone } try Task.checkCancellation() @@ -104,19 +131,24 @@ enum ConnectionState: Equatable { case idle, connecting, active, closing, ended, } @discardableResult func send(_ event: [String: Any]) -> Bool { + if usingGemini { return gemini.send(event) } guard let channel, channel.readyState == .open, let data = try? JSONSerialization.data(withJSONObject: event) else { return false } return channel.sendData(RTCDataBuffer(data: data, isBinary: false)) } func mute(_ muted: Bool) { + if usingGemini { gemini.mute(muted); isMuted = muted; return } isMuted = muted; localTrack?.isEnabled = !muted _ = send(["type": muted ? "session.input_audio.mute" : "session.input_audio.unmute", "event_id": UUID().uuidString]) } func close() { + if usingGemini { closing = true; isMuted = true; gemini.close(); return } networkRecovery.connected() closing = true; localTrack?.isEnabled = false; isMuted = true _ = send(["type": "session.close", "event_id": UUID().uuidString]) } func disconnect() { + gemini.disconnect() + usingGemini = false networkRecovery.connected() attempt = UUID(); meterTask?.cancel(); meterTask = nil started = false; closing = true diff --git a/apps/ios/App/ManagedAccountView.swift b/apps/ios/App/ManagedAccountView.swift index 86d735f4..cd02ce0e 100644 --- a/apps/ios/App/ManagedAccountView.swift +++ b/apps/ios/App/ManagedAccountView.swift @@ -17,7 +17,7 @@ struct ManagedAccountView: View { .font(.subheadline).foregroundStyle(MuralColor.secondary).multilineTextAlignment(.center) } if store.configuration == nil { - Text("Accounts aren’t available in this build yet. You can keep practising with your own OpenAI API key in Settings.") + Text("Accounts aren’t available in this build yet. You can keep practising with your own AI provider key in Settings.") .font(.body).multilineTextAlignment(.center) .padding(24).background(.white.opacity(0.7), in: RoundedRectangle(cornerRadius: 24)) .accessibilityIdentifier("managedAccountUnavailable") diff --git a/apps/ios/App/OnboardingView.swift b/apps/ios/App/OnboardingView.swift index 7dc3970f..4492fe8e 100644 --- a/apps/ios/App/OnboardingView.swift +++ b/apps/ios/App/OnboardingView.swift @@ -184,11 +184,11 @@ struct OnboardingView: View { } enum AIProcessingConsent { - static let version = 1 - static let summary = "With your permission, Mural sends audio and selected text to OpenAI to provide conversations and meanings. Provider retention rules apply." + static let version = 2 + static let summary = "With your permission, Mural sends audio, selected text, and topic search text to the AI provider you select to provide conversations and meanings. Provider retention rules apply." enum ConsentError: LocalizedError { case required - var errorDescription: String? { "Before using AI features, open Talk and tap the microphone to review how OpenAI processes your audio and text." } + var errorDescription: String? { "Before using AI features, open Talk and tap the microphone to review how your selected AI provider processes your audio and text." } } } diff --git a/apps/ios/App/Storage.swift b/apps/ios/App/Storage.swift index 1b56e76c..ba6f01ab 100644 --- a/apps/ios/App/Storage.swift +++ b/apps/ios/App/Storage.swift @@ -93,35 +93,48 @@ import MuralCore } enum CredentialStore { - private static let service = "no.william.mural.openai" - private static var query: [String: Any] { [kSecClass as String: kSecClassGenericPassword, kSecAttrService as String: service, kSecAttrAccount as String: "owner", kSecAttrSynchronizable as String: false] } - static func read() -> String? { - var q = query; q[kSecReturnData as String] = true; q[kSecMatchLimit as String] = kSecMatchLimitOne + private static func service(for provider: AIProvider) -> String { + switch provider { + case .openAI: "no.william.mural.openai" + case .googleAIStudio: "no.william.mural.google-ai-studio" + } + } + private static func query(for provider: AIProvider) -> [String: Any] { + [kSecClass as String: kSecClassGenericPassword, kSecAttrService as String: service(for: provider), kSecAttrAccount as String: "owner", kSecAttrSynchronizable as String: false] + } + static func read(_ provider: AIProvider = .openAI) -> String? { + var q = query(for: provider); q[kSecReturnData as String] = true; q[kSecMatchLimit as String] = kSecMatchLimitOne var item: CFTypeRef? guard SecItemCopyMatching(q as CFDictionary, &item) == errSecSuccess, let data = item as? Data else { return nil } return String(data: data, encoding: .utf8) } - static var hasKey: Bool { read() != nil } - static func save(_ key: String) throws { + static var hasKey: Bool { hasKey(for: .openAI) } + static func hasKey(for provider: AIProvider) -> Bool { read(provider) != nil } + static func save(_ key: String, for provider: AIProvider = .openAI) throws { let value = key.trimmingCharacters(in: .whitespacesAndNewlines) - guard value.hasPrefix("sk-"), value.count >= 20, !value.contains(where: \.isWhitespace) else { throw KeyError.invalid } + let valid = switch provider { + case .openAI: value.hasPrefix("sk-") && value.count >= 20 + case .googleAIStudio: value.count >= 20 + } + guard valid, !value.contains(where: \.isWhitespace) else { throw KeyError.invalid } let data = Data(value.utf8) - let status = SecItemUpdate(query as CFDictionary, [kSecValueData as String: data] as CFDictionary) + let itemQuery = query(for: provider) + let status = SecItemUpdate(itemQuery as CFDictionary, [kSecValueData as String: data] as CFDictionary) if status == errSecItemNotFound { - var q = query; q[kSecValueData as String] = data + var q = itemQuery; q[kSecValueData as String] = data q[kSecAttrAccessible as String] = kSecAttrAccessibleWhenUnlockedThisDeviceOnly guard SecItemAdd(q as CFDictionary, nil) == errSecSuccess else { throw KeyError.save } } else if status != errSecSuccess { throw KeyError.save } } - static func delete() throws { - let status = SecItemDelete(query as CFDictionary) + static func delete(for provider: AIProvider = .openAI) throws { + let status = SecItemDelete(query(for: provider) as CFDictionary) guard status == errSecSuccess || status == errSecItemNotFound else { throw KeyError.remove } } enum KeyError: LocalizedError { case invalid, save, remove var errorDescription: String? { switch self { - case .invalid: "Enter a valid OpenAI API key." + case .invalid: "Enter a valid API key for the selected provider." case .save: "The key couldn’t be saved to this device’s Keychain." case .remove: "The key couldn’t be removed. Unlock this iPhone and try again." } diff --git a/apps/ios/Core/Languages/English.swift b/apps/ios/Core/Languages/English.swift index e550978b..6d5d4056 100644 --- a/apps/ios/Core/Languages/English.swift +++ b/apps/ios/Core/Languages/English.swift @@ -2,7 +2,7 @@ import Foundation extension LanguageModule { public static let english = LanguageModule( - id: "en", name: "English", nativeName: "English", variety: "International", locale: "en", + id: "en", name: "English", nativeName: "English", variety: "International", locale: "en", usesPlatformWordSegmentation: false, greeting: "Hi!", greetingWord: "hi", speechGuidance: "Use clear, broadly intelligible English with a consistent, natural pronunciation. Accept valid regional accents, vocabulary and grammar, including British and American forms. Do not treat an accent difference as an error or require imitation of a native accent. Correct pronunciation only when meaning is unclear and the audio supports the correction.", writingGuidance: "Use standard English spelling and punctuation. Keep one spelling convention within your own reply, but accept valid regional spelling and usage from the learner.", diff --git a/apps/ios/Core/Languages/French.swift b/apps/ios/Core/Languages/French.swift index 9a663821..85e7423a 100644 --- a/apps/ios/Core/Languages/French.swift +++ b/apps/ios/Core/Languages/French.swift @@ -2,7 +2,7 @@ import Foundation extension LanguageModule { public static let french = LanguageModule( - id: "fr", name: "French", nativeName: "Français", variety: "France", locale: "fr-FR", + id: "fr", name: "French", nativeName: "Français", variety: "France", locale: "fr-FR", usesPlatformWordSegmentation: false, greeting: "Salut !", greetingWord: "salut", speechGuidance: "Use clear, natural metropolitan French pronunciation. Use tu in a friendly conversation and vous when the situation calls for formality or plural address. Accept valid regional accents, vocabulary and grammar from across the French-speaking world. Do not treat regional variation, informal omission of ne or a non-native accent alone as an error. Do not imitate a regional caricature.", writingGuidance: "Use standard French spelling, accents, apostrophes and punctuation. Preserve accents on capital letters. Match the register to the situation and accept valid regional usage from the learner.", diff --git a/apps/ios/Core/Languages/German.swift b/apps/ios/Core/Languages/German.swift index 29063b92..3eb469c9 100644 --- a/apps/ios/Core/Languages/German.swift +++ b/apps/ios/Core/Languages/German.swift @@ -2,7 +2,7 @@ import Foundation extension LanguageModule { public static let german = LanguageModule( - id: "de", name: "German", nativeName: "Deutsch", variety: "Germany", locale: "de-DE", + id: "de", name: "German", nativeName: "Deutsch", variety: "Germany", locale: "de-DE", usesPlatformWordSegmentation: false, greeting: "Hallo!", greetingWord: "hallo", speechGuidance: "Use clear, natural Standard German as spoken in Germany. Use du for friendly conversation and Sie when the situation calls for formality. Accept valid Austrian, Swiss and other regional pronunciation, vocabulary and grammar. Do not treat a regional difference or a non-native accent alone as an error. Correct pronunciation only when supported by the audio, not a transcript alone.", writingGuidance: "Use standard German spelling, noun capitalization, umlauts and ß. Accept Swiss ss spellings and valid regional wording. Match the register to the situation.", diff --git a/apps/ios/Core/Languages/Italian.swift b/apps/ios/Core/Languages/Italian.swift index d389866f..0d55ed6b 100644 --- a/apps/ios/Core/Languages/Italian.swift +++ b/apps/ios/Core/Languages/Italian.swift @@ -2,7 +2,7 @@ import Foundation extension LanguageModule { public static let italian = LanguageModule( - id: "it", name: "Italian", nativeName: "Italiano", variety: "Italy", locale: "it-IT", + id: "it", name: "Italian", nativeName: "Italiano", variety: "Italy", locale: "it-IT", usesPlatformWordSegmentation: false, greeting: "Ciao!", greetingWord: "ciao", speechGuidance: "Use clear, natural Standard Italian pronunciation. Use tu for friendly conversation and Lei when the situation calls for formality. Model vowel sounds, word stress and consonant length naturally. Accept valid regional accents and vocabulary without treating regional variation or a non-native accent alone as an error. Do not infer a pronunciation error from spelling alone.", writingGuidance: "Use standard Italian spelling, accents, apostrophes and punctuation. Preserve meaningful contrasts such as e and è. Match the register to the situation and accept valid regional usage.", diff --git a/apps/ios/Core/Languages/LanguageModule.swift b/apps/ios/Core/Languages/LanguageModule.swift index b3914a4f..409e8049 100644 --- a/apps/ios/Core/Languages/LanguageModule.swift +++ b/apps/ios/Core/Languages/LanguageModule.swift @@ -7,6 +7,7 @@ public struct LanguageModule: Identifiable, Sendable { public let nativeName: String public let variety: String public let locale: String + public let usesPlatformWordSegmentation: Bool public let greeting: String public let greetingWord: String public let speechGuidance: String @@ -27,13 +28,13 @@ public struct LanguageModule: Identifiable, Sendable { public enum LanguageRegistry { public static let defaultID = "nb" - public static let all: [LanguageModule] = [.norwegian, .spanish, .english, .french, .german, .italian, .portuguese, .mandarin] + public static let all: [LanguageModule] = [.norwegian, .spanish, .english, .french, .german, .italian, .portuguese, .mandarin, .thai] public static func module(for id: String) -> LanguageModule? { all.first { $0.id == id } } } public enum MeaningLanguages { - public static let all = ["English", "French", "German", "Spanish", "Norwegian", "Portuguese", "Italian", "Chinese (Simplified)", "Polish", "Arabic", "Ukrainian"] + public static let all = ["English", "French", "German", "Spanish", "Norwegian", "Portuguese", "Italian", "Chinese (Simplified)", "Thai", "Polish", "Arabic", "Ukrainian"] public static func greeting(in language: String) -> String { - ["English": "Hi!", "French": "Salut !", "German": "Hallo!", "Spanish": "¡Hola!", "Norwegian": "Hei!", "Portuguese": "Olá!", "Italian": "Ciao!", "Chinese (Simplified)": "你好!", "Chinese": "你好!", "Polish": "Cześć!", "Arabic": "مرحبًا!", "Ukrainian": "Привіт!"][language] ?? "Hi!" + ["English": "Hi!", "French": "Salut !", "German": "Hallo!", "Spanish": "¡Hola!", "Norwegian": "Hei!", "Portuguese": "Olá!", "Italian": "Ciao!", "Chinese (Simplified)": "你好!", "Chinese": "你好!", "Thai": "สวัสดี!", "Polish": "Cześć!", "Arabic": "مرحبًا!", "Ukrainian": "Привіт!"][language] ?? "Hi!" } } diff --git a/apps/ios/Core/Languages/Mandarin.swift b/apps/ios/Core/Languages/Mandarin.swift index 661bf3c7..6a61940f 100644 --- a/apps/ios/Core/Languages/Mandarin.swift +++ b/apps/ios/Core/Languages/Mandarin.swift @@ -2,7 +2,7 @@ import Foundation extension LanguageModule { public static let mandarin = LanguageModule( - id: "zh", name: "Mandarin Chinese", nativeName: "普通话", variety: "Mainland China", locale: "zh-CN", + id: "zh", name: "Mandarin Chinese", nativeName: "普通话", variety: "Mainland China", locale: "zh-CN", usesPlatformWordSegmentation: false, greeting: "你好!", greetingWord: "你好", speechGuidance: "Use clear, natural Standard Mandarin pronunciation. Treat tones, tone changes, retroflex and non-retroflex sounds, and distinctions between initials and finals as meaningful when they affect understanding. Accept valid regional accents and vocabulary without treating a regional difference or a non-native accent alone as an error. Do not imitate a regional caricature.", writingGuidance: "Use natural Simplified Chinese and standard modern punctuation. Prefer everyday Mainland usage while accepting valid regional wording and Traditional Chinese input. Keep Chinese text free of unnecessary spaces. The app displays pinyin separately; do not append pinyin or translations to ordinary spoken replies. Explain characters and tones briefly in Mandarin when asked.", diff --git a/apps/ios/Core/Languages/Norwegian.swift b/apps/ios/Core/Languages/Norwegian.swift index 4e9e0777..10b1cf22 100644 --- a/apps/ios/Core/Languages/Norwegian.swift +++ b/apps/ios/Core/Languages/Norwegian.swift @@ -2,7 +2,7 @@ import Foundation extension LanguageModule { public static let norwegian = LanguageModule( - id: "nb", name: "Norwegian", nativeName: "Norsk", variety: "Bokmål", locale: "nb-NO", + id: "nb", name: "Norwegian", nativeName: "Norsk", variety: "Bokmål", locale: "nb-NO", usesPlatformWordSegmentation: false, greeting: "Hei!", greetingWord: "hei", speechGuidance: "Use natural Eastern Norwegian pronunciation. Accept other Norwegian dialects without treating dialect differences as errors.", writingGuidance: "Use Norwegian Bokmål spelling and wording.", diff --git a/apps/ios/Core/Languages/Portuguese.swift b/apps/ios/Core/Languages/Portuguese.swift index 6a13a783..80de0378 100644 --- a/apps/ios/Core/Languages/Portuguese.swift +++ b/apps/ios/Core/Languages/Portuguese.swift @@ -2,7 +2,7 @@ import Foundation extension LanguageModule { public static let portuguese = LanguageModule( - id: "pt", name: "Portuguese", nativeName: "Português", variety: "Brazil", locale: "pt-BR", + id: "pt", name: "Portuguese", nativeName: "Português", variety: "Brazil", locale: "pt-BR", usesPlatformWordSegmentation: false, greeting: "Olá!", greetingWord: "olá", speechGuidance: "Use clear, natural Brazilian Portuguese with broadly intelligible pronunciation and consistent Brazilian vocabulary. Use você in friendly conversation and formal address when appropriate. Accept valid uses of tu, regional Brazilian accents and grammar, and European, African and other Portuguese varieties without marking them wrong. Do not imitate a regional caricature or infer pronunciation errors from a transcript alone.", writingGuidance: "Use standard contemporary Brazilian Portuguese spelling, accents, ã, õ and ç. Prefer everyday Brazilian wording, including a gente and conversational pronoun placement when natural. Accept valid regional and European Portuguese usage from the learner.", diff --git a/apps/ios/Core/Languages/Spanish.swift b/apps/ios/Core/Languages/Spanish.swift index 0409af53..8efb3c74 100644 --- a/apps/ios/Core/Languages/Spanish.swift +++ b/apps/ios/Core/Languages/Spanish.swift @@ -2,7 +2,7 @@ import Foundation extension LanguageModule { public static let spanish = LanguageModule( - id: "es", name: "Spanish", nativeName: "Español", variety: "Spain", locale: "es-ES", + id: "es", name: "Spanish", nativeName: "Español", variety: "Spain", locale: "es-ES", usesPlatformWordSegmentation: false, greeting: "¡Hola!", greetingWord: "hola", speechGuidance: "Use clear Spanish from Spain, with a natural distinction between s and z/soft c, tú for friendly singular address and vosotros for informal plural address. Accept seseo, ustedes, voseo and other valid regional forms without marking them wrong. Do not imitate a regional caricature.", writingGuidance: "Use standard Spanish spelling, accents and opening question and exclamation marks.", diff --git a/apps/ios/Core/Languages/Thai.swift b/apps/ios/Core/Languages/Thai.swift new file mode 100644 index 00000000..2f9c2fbd --- /dev/null +++ b/apps/ios/Core/Languages/Thai.swift @@ -0,0 +1,28 @@ +import Foundation + +extension LanguageModule { + public static let thai = LanguageModule( + id: "th", name: "Thai", nativeName: "ภาษาไทย", variety: "Thailand", locale: "th-TH", usesPlatformWordSegmentation: true, + greeting: "สวัสดี!", greetingWord: "สวัสดี", + speechGuidance: "Use clear, natural Central Thai pronunciation. Treat tones, vowel length, final consonants and polite particles as meaningful when they affect understanding. Accept valid regional accents and vocabulary without treating a regional difference or a non-native accent alone as an error. Do not imitate a regional caricature.", + writingGuidance: "Use natural modern Thai with Thai script and appropriate spacing between phrases. Keep polite particles and pronouns natural to the learner’s context. Do not append transliteration or translations to ordinary spoken replies; explain pronunciation briefly only when asked.", + lemmaGuidance: "Give vocabulary in natural Thai script and preserve the exact observed form and quote. Prefer useful dictionary forms or everyday chunks, and include polite particles only when they are part of the meaningful expression. Do not infer tone or pronunciation accuracy from typed text alone.", + teachingFocus: [ + "Greetings, introductions, names, polite particles and useful everyday chunks such as ผมชื่อ and ฉันอยากได้.", + "Everyday questions, classifiers, numbers, food and short present-time exchanges.", + "Connected stories, completed actions, time expressions and familiar situations.", + "Reasons and opinions, comparisons, requests and natural linking phrases.", + "Nuance, conditionals, register, particles and regional variation.", + "Flexible advanced discussion with precise, natural Thai and context-appropriate politeness." + ], + topicPlaceholder: "อาหาร การเดินทาง เพลง ชีวิตประจำวัน…", + lookupUnavailableReply: "ตอนนี้ฉันยังตรวจสอบเรื่องนี้ไม่ได้ ถ้าคุณต้องการ เราคุยเกี่ยวกับภาพรวมของหัวข้อนี้ก่อนได้นะ", + themeOverrides: [ + "coffee": .init("coffee", "ดื่มกาแฟกันไหม", "Something warm, please", "cup.and.saucer", "Everyday", "Meet in a neighbourhood café in Thailand. Order a drink and chat in Thai, following the learner’s interests.", 0), + "groceries": .init("groceries", "ไปตลาดกัน", "Find something good", "basket", "Everyday", "Shop for everyday ingredients at a Thai market or supermarket. Practise quantities, prices and polite questions while respecting regional food vocabulary.", 2), + "travel": .init("travel", "สถานีต่อไป", "A ticket to somewhere", "tram", "Everyday", "Plan an imagined trip in Thailand. Discuss transport, directions and tickets without inventing current schedules.", 1), + "cabin": .init("cabin", "เที่ยวสุดสัปดาห์", "A change of scene", "mountain.2", "Local life", "Imagine a weekend away in a city, by the sea or in the countryside. Discuss practical plans and things the learner enjoys.", 2), + "traditions": .init("traditions", "เรื่องเล็ก ๆ ในชีวิตประจำวัน", "Small customs, big stories", "flag", "Local life", "Talk in Thai about everyday customs and festivals. Compare experiences without treating any habit as universal.", 2) + ] + ) +} diff --git a/apps/ios/Core/MandarinPinyin.swift b/apps/ios/Core/MandarinPinyin.swift index 1e6b30ca..f228995e 100644 --- a/apps/ios/Core/MandarinPinyin.swift +++ b/apps/ios/Core/MandarinPinyin.swift @@ -84,6 +84,9 @@ public enum CaptionWords { CaptionSegment(text: $0.text, lookup: $0.text.contains(where: \.isLetter) ? $0.text : nil) } } + if LanguageRegistry.module(for: languageID)?.usesPlatformWordSegmentation == true { + return platformWordSegments(text) + } var result: [CaptionSegment] = [] var run = "" for character in text { @@ -100,4 +103,18 @@ public enum CaptionWords { let word = text.trimmingCharacters(in: .punctuationCharacters.union(.whitespacesAndNewlines)) return CaptionSegment(text: text, lookup: word.contains(where: \.isLetter) ? word : nil) } + + private static func platformWordSegments(_ text: String) -> [CaptionSegment] { + guard !text.isEmpty else { return [] } + var result: [CaptionSegment] = [] + var cursor = text.startIndex + text.enumerateSubstrings(in: text.startIndex..= cursor else { return } + if cursor < range.lowerBound { result.append(CaptionSegment(text: String(text[cursor.. SessionRecord { @@ -24,13 +25,15 @@ final class AdditionalLanguageTests: XCTestCase { } func testRegistrationPreservesOldIDsAndSetsRequestedVarieties() { - XCTAssertEqual(LanguageRegistry.all.map(\.id), ["nb", "es", "en", "fr", "de", "it", "pt", "zh"]) - for (id, locale, greeting) in [("de", "de-DE", "Hallo!"), ("it", "it-IT", "Ciao!"), ("pt", "pt-BR", "Olá!"), ("zh", "zh-CN", "你好!")] { + XCTAssertEqual(LanguageRegistry.all.map(\.id), ["nb", "es", "en", "fr", "de", "it", "pt", "zh", "th"]) + for (id, locale, greeting) in [("de", "de-DE", "Hallo!"), ("it", "it-IT", "Ciao!"), ("pt", "pt-BR", "Olá!"), ("zh", "zh-CN", "你好!"), ("th", "th-TH", "สวัสดี!")] { XCTAssertEqual(LanguageRegistry.module(for: id)?.locale, locale) XCTAssertEqual(LanguageRegistry.module(for: id)?.greeting, greeting) } XCTAssertTrue(MeaningLanguages.all.contains("Chinese (Simplified)")) XCTAssertEqual(MeaningLanguages.greeting(in: "Chinese (Simplified)"), "你好!") + XCTAssertTrue(MeaningLanguages.all.contains("Thai")) + XCTAssertEqual(MeaningLanguages.greeting(in: "Thai"), "สวัสดี!") } func testAllPromptPathsUseEachNewTargetAndItsRegionalGuidance() throws { @@ -57,7 +60,7 @@ final class AdditionalLanguageTests: XCTestCase { } } - func testAllEightLanguagesRoundTripWithIsolatedProgressAndHiddenWords() throws { + func testAllRegisteredLanguagesRoundTripWithIsolatedProgressAndHiddenWords() throws { var archive = Archive() archive.sessions = LanguageRegistry.all.flatMap { [session($0.id), session($0.id, day: 2)] } archive.preferences.meaningLanguage = "Chinese (Simplified)" @@ -79,7 +82,7 @@ final class AdditionalLanguageTests: XCTestCase { let hidden = LearningEngine.project(restored.sessions, languageID: language.id, hiddenWords: restored.preferences.hiddenWords) XCTAssertEqual(hidden.words.count, language.id == "pt" ? 0 : 1) } - XCTAssertEqual(keys.count, 8) + XCTAssertEqual(keys.count, 9) } } @@ -157,4 +160,11 @@ final class AdditionalLanguageTests: XCTestCase { XCTAssertEqual(segments.compactMap(\.lookup), expected) } } + + func testThaiWordLinksUsePlatformWordBoundaries() { + let text = "ฉันอยากไปตลาด" + let segments = CaptionWords.segments(text, languageID: "th") + XCTAssertEqual(segments.map(\.text).joined(), text) + XCTAssertEqual(segments.compactMap(\.lookup), ["ฉัน", "อยาก", "ไป", "ตลาด"]) + } } diff --git a/apps/ios/Tests/ProviderFailureTests.swift b/apps/ios/Tests/ProviderFailureTests.swift index cab9f530..2c598e3c 100644 --- a/apps/ios/Tests/ProviderFailureTests.swift +++ b/apps/ios/Tests/ProviderFailureTests.swift @@ -22,4 +22,9 @@ final class ProviderFailureTests: XCTestCase { XCTAssertEqual(ProviderFailure(status: 401, body: Data(#"{"error":{"code":"insufficient_quota"}}"#.utf8)).kind, .authentication) XCTAssertEqual(ProviderFailure(status: 403).kind, .modelAccess) } + func testProviderNameIsRenderedInSafeRecoveryAdvice() { + let error = ProviderFailure(status: 401, providerName: "Google AI Studio") + XCTAssertTrue(error.errorDescription!.contains("Google AI Studio")) + XCTAssertFalse(error.errorDescription!.contains("(providerName)")) + } } diff --git a/docs/android/design.md b/docs/android/design.md index 518fb880..7b9bc2a5 100644 --- a/docs/android/design.md +++ b/docs/android/design.md @@ -4,7 +4,7 @@ Mural keeps two native clients: SwiftUI and SwiftData in `apps/ios/`, and Kotlin with Jetpack Compose in `apps/android/`. The API lives in `services/api/`. Shared fixtures and contracts keep learning data compatible while each app retains its platform audio, accessibility and animation tools. -The Android client keeps the eight language modules, the 24 themes and their cultural variants, WebRTC voice, written replies, meanings, word lookup, current topics with sources, history, corrections, vocabulary and learning projection. It adds written conversation without the microphone permission. Android account and minute-purchase integration is in progress. The public hosted conversation service remains disabled; no cloud learning sync is planned. +The Android client keeps the nine language modules, the 24 themes and their cultural variants, OpenAI WebRTC or Google AI Studio Live voice, written replies, meanings, word lookup, current topics with sources, history, corrections, vocabulary and learning projection. It adds written conversation without the microphone permission. Android account and minute-purchase integration is in progress. The public hosted conversation service remains disabled; no cloud learning sync is planned. ## Visual system @@ -16,7 +16,7 @@ The orb has a smooth twelve-point outline, warm blended color, a feathered shado Both onboarding steps use dropdown menus over a softly moving warm background. Continue stays at the bottom; content can scroll at larger text sizes. Reply entry opens in a rounded sheet with keyboard padding and a visible send action. Search, settings and other input fields share the same rounded surfaces and warm focus color. -Caption taps open a cream bottom sheet with the selected word, its original sentence and an automatic contextual explanation, following the iOS lookup flow. All eight languages use this interaction. The sentence stays fixed while the conversation advances; closing the sheet cancels its request. Lookup has its own loading state so another helper cannot leave the sheet empty or have its work cancelled by dismissal. +Caption taps open a cream bottom sheet with the selected word, its original sentence and an automatic contextual explanation, following the iOS lookup flow. All nine languages use this interaction. The sentence stays fixed while the conversation advances; closing the sheet cancels its request. Lookup has its own loading state so another helper cannot leave the sheet empty or have its work cancelled by dismissal. Mandarin keeps the inline Show/Hide pinyin control between the Chinese caption and the English meaning. The Chinese passage and its reading scroll together; the meaning retains its own visible area. Small screens and large system text use a scrolling page. Source characters and caption tap boundaries remain independent of pronunciation phrases. @@ -25,7 +25,7 @@ Pinyin uses the MIT-licensed [phrase-pinyin-data](https://github.com/mozillazg/p ## Components - `core/`: serializable models, the v1/v2 archive, language modules, evidence rules, teaching policy, `MeaningController`, `FinalAssessmentQueue`, session limits and usage summary. No Android dependencies, so it runs in JVM tests. -- `network/`: direct HTTPS client for OpenAI, AES-GCM credentials protected by Android Keystore, native WebRTC transport and the platform language detector. +- `network/`: direct HTTPS clients for OpenAI and Google AI Studio, AES-GCM credentials protected by Android Keystore, native WebRTC/Live transports and the platform language detector. - `LearningRepository`: private `learning.json` written with `AtomicFile`. It never contains the API key. - `MuralViewModel`: conversation state, duration and inactivity limits, cancellation, assessments, meanings and persistence. Isolated copies prevent a late response from changing another conversation. - `ui/` and `MainActivity`: Compose screens, explicit consent, system permissions, export and import through the Android document picker, and accessibility. Copy lives in `res/values` (English) and `res/values-es` (Spanish). @@ -51,7 +51,7 @@ The iPhone JSON archive (`schemaVersion` 2) and the v1 Norwegian migration are p The microphone works only in the foreground. Ending the conversation, losing audio focus, leaving the app or cancelling the connection releases audio and WebRTC. Only the permissions Mural needs are requested, and no recordings are stored. System backups and device transfers exclude the archive and credentials; learners move their learning through explicit export. -The repository protocol is unchanged: OpenAI `POST /v1/live/sessions` with `gpt-live-1`, voice `marin` and the `oai-events` channel, and helper operations through `POST /v1/responses` with `gpt-5.6-luna`. The key is entered on the device, never in code. There is no shared key, no required Mural server and no paid call in automated tests. +The OpenAI protocol remains available: `POST /v1/live/sessions` with `gpt-live-1`, voice `marin` and the `oai-events` channel, and helper operations through `POST /v1/responses` with `gpt-5.6-luna`. Google AI Studio uses `gemini-3.8-live` over its Live WebSocket and `gemma-4-31b-it:generateContent` for helper operations. The key is entered on the device, never in code. There is no shared key, no required Mural server and no paid call in automated tests. ## Keeping both clients in sync diff --git a/docs/google-ai-studio.md b/docs/google-ai-studio.md new file mode 100644 index 00000000..910f54d6 --- /dev/null +++ b/docs/google-ai-studio.md @@ -0,0 +1,14 @@ +# Google AI Studio provider + +Mural can use a personal Google AI Studio API key directly from the iPhone or Android app. Select **Google AI Studio** in **Settings → Advanced → Use your own API key**, then save a key created at [Google AI Studio](https://aistudio.google.com/app/apikey). The key is stored in the platform’s protected credential store and is never included in learning backups or source files. + +The provider uses two Gemini models because Google exposes text generation and low-latency voice through different model contracts: + +- `gemma-4-31b-it` handles translations, teaching, assessments, lookups and optional grounded search. +- `gemini-3.8-live` handles the Live API voice session over a direct WebSocket, with 16 kHz PCM input, 24 kHz PCM output and input/output transcription. + +The requested “Flash live” combination is represented by Gemma for the text teacher and Gemini Live for the voice transport; `gemma-4-31b-it` itself is not a Live API model. Google documents Gemma’s hosted Gemini API support, including grounded search, in its [Gemma API guide](https://ai.google.dev/gemma/docs/core/gemma_on_gemini_api); see the [Live API WebSocket guide](https://ai.google.dev/gemini-api/docs/live-api/get-started-websocket) for voice. + +OpenAI remains the default and its WebRTC/server-session path is unchanged. Google AI Studio is a direct client path, so no Mural API, database migration, or production deployment is required for this feature; Live voice still requires the platform microphone permission. Hosted-account conversations continue to use the existing hosted provider path. + +Google AI Studio keys are personal billing credentials. Use a restricted key where possible, keep it out of logs and screenshots, and rotate it in AI Studio if the device is lost. Google recommends ephemeral tokens for production client applications; this repository’s personal-build flow follows the requested API-key model instead. diff --git a/docs/managed-accounts.md b/docs/managed-accounts.md index 2e51efa8..fe40f237 100644 --- a/docs/managed-accounts.md +++ b/docs/managed-accounts.md @@ -1,6 +1,6 @@ # How to enable optional sign-in on iPhone -Configure Google or Apple sign-in without enabling hosted conversations, trial minutes or purchases. Users can still practise with their own OpenAI API key without an account. The source defaults to accounts disabled; the current local Mural build enables Google only. Mural’s Google project is in production with verified, published consent branding as of September 12, 2026. +Configure Google or Apple sign-in without enabling hosted conversations, trial minutes or purchases. Users can still practise with their own OpenAI or Google AI Studio API key without an account. The source defaults to accounts disabled; the current local Mural build enables Google only. Mural’s Google project is in production with verified, published consent branding as of September 12, 2026. On 12 September 2026, the owner confirmed that real Google sign-in worked on the installed iPhone build. Session persistence, sign-out, expiry and account deletion still need their separate device checks below. Apple remains disabled pending enrollment and revocation setup. diff --git a/docs/run-on-android.md b/docs/run-on-android.md index 14b6494e..c3b7864a 100644 --- a/docs/run-on-android.md +++ b/docs/run-on-android.md @@ -2,7 +2,7 @@ ## Personal installation -You need Android 8.0 or later and an OpenAI project key with access to the models Mural uses. A ChatGPT subscription does not include API credit. +You need Android 8.0 or later and either an OpenAI project key or a Google AI Studio API key. A ChatGPT subscription does not include API credit. Install the generated APK (`apps/android/app/build/outputs/apk/debug/app-debug.apk`) by opening it on the phone and allowing the installation, or from a computer: @@ -24,13 +24,13 @@ If `apps/android/.signing/debug.keystore` exists (it is ignored by Git), the deb Mural speaks the interface language of your phone: Spanish on a Spanish phone, English otherwise. That is separate from the language you practise and the language you read meanings in, which you choose next. -Choose the language you practise and the language for meanings. When you practise Mandarin, captions link each word and show pinyin underneath on Android 10 or later; **Hide pinyin** keeps the characters only. Read the consent to send audio and text to OpenAI. You can decline and still browse your local data. +Choose the language you practise and the language for meanings. When you practise Mandarin, captions link each word and show pinyin underneath on Android 10 or later; **Hide pinyin** keeps the characters only. Read the consent to send audio and text to the selected AI provider. You can decline and still browse your local data. -In **Settings**, save your own OpenAI key. Do not send it through chat or put it in repository files. It is encrypted with an Android Keystore key and is never included in learning backups. +In **Settings**, choose OpenAI or Google AI Studio and save that provider’s key. Do not send it through chat or put it in repository files. It is encrypted with an Android Keystore key and is never included in learning backups. On **Talk**, start a conversation and allow the microphone. You should hear a greeting in the language you chose. **Type instead** lets you practise without the microphone. You can mute, ask for a little help, show meanings, tap a word to look it up, and end the conversation. Any conversation ends when the app moves to the background. A voice conversation also ends when audio is interrupted, when it reaches the chosen duration, or after 30 seconds of quiet, with a gentle check-in and a five-second countdown. Speaking or typing gives you time to continue; waiting for an answer also receives a bounded grace period; a written conversation stays open while you compose a reply. -Mural needs the internet to talk, translate and search. History and vocabulary are available offline. Requests are billed to your OpenAI project; **Settings** shows recorded voice time and a voice cost estimate. The app's time limit is not a billing cap. +Mural needs the internet to talk, translate and search. History and vocabulary are available offline. Requests are billed to the selected provider’s project; **Settings** shows recorded voice time and provider-specific billing guidance. The app's time limit is not a billing cap. ## Move data from iPhone @@ -56,7 +56,7 @@ cd apps/android ./gradlew :app:connectedUiTestAndroidTest ``` -The interface tests install as a separate app, `chat.mural.android.uitest`, so they never read or change the data of your Mural installation. They do not need an API key and do not call OpenAI. +The interface tests install as a separate app, `chat.mural.android.uitest`, so they never read or change the data of your Mural installation. They do not need an API key and do not call an AI provider. ## Troubleshooting @@ -64,7 +64,7 @@ The interface tests install as a separate app, `chat.mural.android.uitest`, so t - **No device is listed:** check the cable, File transfer mode and USB debugging. - **Microphone blocked:** Android Settings → Apps → Mural → Permissions → Microphone. You can also type. - **Sound plays on the speaker with a Bluetooth headset:** connect the headset before starting the conversation. Mural chooses a connected headset on Android 12 or later; on Android 8–11 it uses the speaker. -- **Key rejected or usage limit reached:** check the key, model access and project limits in your OpenAI account. Mural never needs you to share your key with anyone. +- **Key rejected or usage limit reached:** check the selected provider’s key, model access and project limits. Mural never needs you to share your key with anyone. - **Switching to another app:** Mural ends a voice conversation to release the microphone. Come back and start another; history is kept. - **Backup error:** make sure it is a complete Mural file under 30 MB. The importer rejects a file before merging invalid records. - **Update rejected because of the signature:** use the same signing key as the previous installation. Export a backup before any uninstall. diff --git a/docs/run-on-iphone.md b/docs/run-on-iphone.md index 81c0467b..56d58d5a 100644 --- a/docs/run-on-iphone.md +++ b/docs/run-on-iphone.md @@ -1,15 +1,15 @@ # How to run Mural on your iPhone -Use this guide to install a personal build from source. After installation, the phone connects directly to OpenAI and works away from your Mac. +Use this guide to install a personal build from source. After installation, the phone connects directly to the provider selected in Settings and works away from your Mac. ## Before you start - A Mac with Xcode 26 or later, downloaded from Apple. - An iPhone running iOS 26.1 or later and a USB cable for initial setup. - An Apple Account added to **Xcode → Settings → Accounts**. -- An OpenAI project with API billing enabled and access to the models configured in `apps/ios/App/APIClient.swift` and `apps/ios/App/LiveTransport.swift`. +- Either an OpenAI project with API billing enabled, or a Google AI Studio API key. Mural uses `gpt-5.6-luna`/`gpt-live-1` for OpenAI and `gemma-4-31b-it`/`gemini-3.8-live` for Google AI Studio. -OpenAI API usage is paid separately from ChatGPT. Use your own key for this personal build. Never add a key to source code, an Xcode build setting, a screenshot or a GitHub issue. +API usage is paid separately from ChatGPT. Use your own key for this personal build. Never add a key to source code, an Xcode build setting, a screenshot or a GitHub issue. ## Update an earlier checkout @@ -44,7 +44,7 @@ Both paths remain ignored by Git. If both files exist, compare them privately an 5. On the iPhone, enable **Settings → Privacy & Security → Developer Mode**. Restart and confirm **Turn On** when prompted. 6. In Xcode’s toolbar, select the **Mural** scheme and your iPhone as the destination. Click **Run** or press **⌘R**. If macOS requests access to the signing key, allow Xcode to use it. 7. If iOS blocks the first launch because the developer is untrusted, open **Settings → General → VPN & Device Management**, select your developer profile, and tap **Trust**. Return to Xcode and run again. -8. Choose your learning and subtitle languages in the welcome screens. In **Settings → Advanced → Use your own API key**, save your OpenAI project key. Tap the main conversation button and allow microphone access. +8. Choose your learning and subtitle languages in the welcome screens. In **Settings → Advanced → Use your own API key**, choose OpenAI or Google AI Studio and save that provider’s key. Tap the main conversation button and allow microphone access. Mural should greet you aloud. Disconnect the cable and confirm a short conversation over Wi-Fi or cellular. @@ -62,7 +62,7 @@ Use **Settings → Export learning backup** before changing your bundle identifi | The phone does not appear as a destination | Unlock it, check the cable, and open Devices and Simulators to finish pairing. | | Developer Mode is missing | Pair with Xcode first, then check Privacy & Security again. | | The app will not launch after a week | Refresh the build through Xcode without uninstalling. | -| Key rejected or model unavailable | Check the key’s project, API billing, permissions and model access in OpenAI. | +| Key rejected or model unavailable | Check the selected provider’s project, billing, permissions and model access. | | No microphone input | Enable Mural under iPhone Settings → Privacy & Security → Microphone. | | Sound uses the wrong output | Check iOS’s current audio destination; disconnect an unwanted Bluetooth route. | | A connection fails on cellular | Confirm Mural is allowed to use cellular data and retry with a stable connection. | diff --git a/scripts/export_android_content.py b/scripts/export_android_content.py index 53cd9363..21fd73b6 100644 --- a/scripts/export_android_content.py +++ b/scripts/export_android_content.py @@ -15,6 +15,7 @@ # Fields whose value is a nested structure (array/dict), extracted separately from the # simple quoted-string fields. STRUCTURED_FIELDS = ('teachingFocus', 'themeOverrides') +DEFAULTED_FIELDS = {'usesPlatformWordSegmentation'} def quoted(text): @@ -94,6 +95,7 @@ def generate(core): 'object Themes { val shared = listOf(', ',\n'.join(' ' + x for x in shared_themes), ') }', '', '''data class LanguageModule( val id: String, val name: String, val nativeName: String, val variety: String, val locale: String, + val usesPlatformWordSegmentation: Boolean = false, val greeting: String, val greetingWord: String, val speechGuidance: String, val writingGuidance: String, val lemmaGuidance: String, val teachingFocus: List, val topicPlaceholder: String, val lookupUnavailableReply: String, val themeOverrides: Map = emptyMap() @@ -115,8 +117,14 @@ def generate(core): module_names.append(module_name) args = [] for key in simple_fields: + boolean_match = re.search(r'\b' + key + r':\s*(true|false)', text) + if boolean_match: + args.append(f' {key} = {boolean_match.group(1)}') + continue match = re.search(r'\b' + key + r':\s*("(?:[^"\\]|\\.)*")', text) if not match: + if key in DEFAULTED_FIELDS: + continue raise SystemExit(f'LanguageModule field {key} not found in {path}. Add it or update {KOTLIN_DEST}.') args.append(f' {key} = {quoted(json.loads(match.group(1)))}') focus_match = re.search(r'teachingFocus:\s*\[(.*?)\]', text, re.S)