Repository navigation
iOS keyboard: stale microphone errors clear, Hex key audio calls leave the main thread, counts read 1, 2, 3 - #2203
Merged
Merged
Conversation
…the Hex key retries it X testers on build 56 and earlier: "The keyboard keeps either bringing me back to the app or telling me another app is using the microphone" (@itsjling) and "The another app is using audio message seems inaccurate cause it's showing for me all the time although it's working fine" (@RedaSjakur). Cause, reproduced on build 56 (2ef08aa) in an iOS 27.2 simulator on the Mac Studio (no microphone input): Hex's dictation failed ("Hex couldn't start the microphone. Try again."), Hex was then closed, and a minute later the Hex keyboard's strip in Settings' search field still said so. Hex writes its snapshot only while it runs, so a failed snapshot (heartbeat 0, no time) stayed in the App Group, and KeyboardViewModel read any .failed snapshot as failed whatever its age. The code path is unchanged on origin/dev. With a warm session up it was worse: Hex stayed in .failed, consumeKeyboardCommand ignored every start unless the phase was .ready, and the full keyboard's Hex key (isWarm = Full Access && warm available) kept sending starts to it, so nothing recorded until Hex was opened. The keyboard's own failure snapshot also reads as available for its 6 s, so a Hex key tap right after an in-keyboard failure went to a warm session that doesn't exist. Now: - KeyboardSnapshot.messageAt: when Hex wrote the message (DictationController stamps a failure when its phase becomes failed, and keyboardMessage when set; the keyboard's own failure stamps its snapshot). The keyboard drops a message, and reads a failure as offline, 8 s after it (messageShownFor), and at once for a snapshot with no time (an earlier build's). Whether a warm session is up and any result stay as they were. - KeyboardViewModel.dictatesThroughHex (Full Access, warm available, not failed), used by the full keyboard's Hex key and its touch-down capture: after a failure both keyboards dictate in the keyboard, as the Dictation bar already did. - Hex takes a keyboard start while failed when the failure was the microphone for a keyboard dictation with the model loaded (captureFailed): the next Hex key is its Try again. Model failures still wait for Hex to get ready again. Not changed: the 32 MB memory threshold and keyboardNeedsHex, the two ways the Hex key opens Hex. In the simulator the handoffs logged "speech model missing here" after the keyboard's own asset check failed ("anomaly.hex.mobile.keyboard is not subscribed to assistant.en_US") on a sim whose Hex had never installed a model, which is the intended path; a device's reason needs its probe log. Verified: new KeyboardBridgeTests.testAFailureHexWroteAWhileAgoIsOver and DictationControllerTests.testAWarmSessionsMicrophoneFailureIsRetriedFromTheKeyboard; KeyboardBridgeTests and DictationControllerTests pass (79 tests, 0 failures, iPhone 17 Pro iOS 27.2 simulator). In the simulator with this build, the same failure no longer shows in the keyboard a minute later.
X tester @leeladhar (iPhone 16 Pro Max, build 56 or earlier): "the keyboard freezes while using apps sometimes ... the only way to close the keyboard is to close the app completely". Not reproduced in the iOS 27.2 simulator on build 56 (2ef08aa) or origin/dev: 24 host visits (Messages, Contacts, Settings search) with typing, the Hex key and globe switches kept one keyboard process, no crash or hang report, footprint 29-40 MB with no growth. The device cause isn't confirmed. The likeliest one left on origin/dev is the last synchronous audio server round trips on the keyboard's main thread: every Hex key touch-down set the session's category, read and preferred its inputs (KeyboardLocalDictation.microphone(for:)), asked isOtherAudioPlaying, and read the Speech Recognition status for the probe log, all before the page could start. These wait on the audio server (and TCC) with no timeout; a build 56 crash report from TestFlight shows Hex's own main thread parked in such a call (setPreferredSampleRate waiting on SetBatchProperties). #2185 moved the one on the launch path for the same reason. While blocked, no key, Done or globe responds, and the keyboard can't dismiss. Now begin() runs those on the keyboard's audio session queue (after any earlier capture's release there) and starts the page when they return, unless the job was dropped, stopped or replaced meanwhile. Done before the page was asked to start ends quietly at once instead of leaving the page to start a capture nobody wants. Verified: builds clean; in an iPhone 17 Pro iOS 27.2 simulator (WebKit's mock audio device) the Hex key's touch-down capture still starts and is kept: "first audio +124 ms after touch-down (... getUserMedia +83 ...)", 13.5 s recorded as 643,072 frames at 48 kHz (no gap), then transcription as before.
… mid-dictation TestFlight tester on X (iPhone 13 Pro Max, iOS 27, build 56 or earlier): "in progress recording dropped when I took a screenshot". What a screenshot does to an in-keyboard dictation, in an iPhone 17 Pro iOS 27.2 simulator on the Mac Studio (WebKit's mock audio device records there): - The host only going inactive doesn't stop it: with Control Center pulled down for 4 s mid-dictation the page kept sending audio (700,416 frames at 48 kHz over a 14.8 s recording, no gap). - A screenshot taken with Cmd-Shift-3 hid the keyboard and the probe log said "keyboard hidden while recording: finishing with what was said" at the shutter. The simulator can only take one through its hardware keyboard, which also turns on keyboard minimization, so it doesn't show which of the two hid it. On an iPhone, iOS 26 and 27 open the screenshot's full-screen preview over the app, and a keyboard that hides is torn down and its WebKit processes suspended, so a dictation can't keep recording through that. - On build 56 a keyboard hidden mid-recording lost the dictation: viewWillDisappear only dropped a touch-down capture, and the recording went with the view controller (reproduced in #2163). Since #2163 on origin/dev it stops, keeps what was said, and the text goes into the same field when the keyboard comes back within 30 s, else is offered as Paste dictation. The one way left for a capture to end early with the keyboard still up is the page's audio context: iOS may suspend or interrupt it for other audio and the page never resumed it, so no more audio came and the 6 s watchdog ended the dictation. The page now resumes a context that isn't running every 250 ms while it records, and stops trying when the capture stops. Not changed: a keyboard hidden by a screenshot's preview still ends the recording (with its text kept, as above). Verified: builds clean; in the simulator the touch-down capture still starts and records through Control Center with no gap (numbers above).
TestFlight (@kevin_miller, Nov 20, build 56 = 2ef08aa): "Test 1, 2, 3" came out "Test three." / "Test 3" from the keyboard's in-keyboard dictation and "test 123" in the app's Dictate box. What each speech model writes (macOS 27.2.1 on the Mac Studio, `say` clips, 4 voices): Apple's SpeechTranscriber (in-keyboard dictation) writes "Test one, 2, 3.", "Test, one, two, three.", "Testing 123.", "Counting one, two, three, four."; DictationTranscriber writes "Test 123", "Counting 1234"; Parakeet v2 through FluidAudio 0.15.5 (the app) writes "Test 1, 2, 3", "Testing 123" (no pauses), "1, 2, 3, testing". Responsible steps: - "Test three." / "Test 3": at build 56 Clean up what I say deleted every word over P(DELETE) 0.98 with no guard. The real Core ML model gives the count words but the last 0.97-0.999 ("Test one two three." -> "Test three.", "Test 1, 2, 3." -> "Test 1, 3."): it reads a count as a self-correction. CleanupGuard (#2132, after build 56) already keeps every count on origin/dev; this commit adds tests that pin it. - "test 123": Parakeet and DictationTranscriber write "123" themselves; no Hex step merges digits. Apple Intelligence's cleanup (same prompt, macOS 27.2 model) kept "Test 1, 2, 3." in 3 of 3 runs per input. - The paths still disagreed on origin/dev: the keyboard inserted "Test one, 2, 3." where the app inserted "Test 1, 2, 3.". Change: SpokenCleanupRules.spokenCounts, run by CleanupTagger.cleanIfEnabled right after spoken symbols (every style but Verbatim, English, Clean up what I say on or off, before the tagger, in the app and in-keyboard dictation), writes a count in digits: three numbers or more counting up from one, in words or digits, with only a comma or a space between them ("one, 2, 3", "one two three" -> "1, 2, 3"), and "123"... right after "test"/"testing" when it ends its clause ("Test 123" -> "Test 1, 2, 3"; "test 123 passed", "room 123", "Counting 1234" stay). Decision: the "123" split is limited to test/testing because a bare "123" is usually a real number. Format lists already never makes a list of a count (each item would be empty); tests prove it. README gains the Spoken counts paragraph. Verified on the iOS 27.2 simulator (Mac Studio), real Core ML tagger: TranscriptCleanupGuardTests 9, CleanupGuardTests 6, SpokenCleanupRulesTests 19, HinglishRomanizerTests 7, SpokenListsTests 10, InformalContractionTests 4, FrenchSpacingTests 3, CleanupTaggerTests 15, DictationControllerTests 50: 123 tests, 0 failures. New: testSpokenCountsBecomeDigits, testOtherNumbersStayAsSaid, testCountsAreNoList, testCountsKeepEveryNumberFromEverySpeechModel (also shows the guardless tagger's "Test three."), testACountReadsTheSameFromTheAppAndTheKeyboard (app with Parakeet's text vs keyboard with both Apple modules' text, every style, Lowercase).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Four fixes from X and TestFlight reports, one commit each: a failure from Hex shows for 8 s instead of forever and the Hex key retries it (false 'another app is using the microphone'); the Hex key's audio-session calls run on the audio queue, not the keyboard's main thread (likely freeze cause, seen in a build 56 crash report); the capture page resumes an audio context iOS suspended mid-dictation; a spoken count reads 'Test 1, 2, 3.' on every speech model. Verified: xcodebuild test 606 passed, 0 failures (1 pre-existing skip); generic iOS build succeeds on dev 976267e; simulator repros before/after for the error and the sample showing setCategory off the main thread.