Skip to content

iOS keyboard: stale microphone errors clear, Hex key audio calls leave the main thread, counts read 1, 2, 3 - #2203

Merged
R44VC0RP merged 4 commits into
devfrom
ios-keyboard-reliability
Nov 21, 2026
Merged

R44VC0RP merged 4 commits into
devfrom
ios-keyboard-reliability

Conversation

@R44VC0RP

Copy link
Copy Markdown
Collaborator

Four fixes from X and TestFlight reports, one commit each: a failure from Hex shows for 8 s instead of forever and the Hex key retries it (false 'another app is using the microphone'); the Hex key's audio-session calls run on the audio queue, not the keyboard's main thread (likely freeze cause, seen in a build 56 crash report); the capture page resumes an audio context iOS suspended mid-dictation; a spoken count reads 'Test 1, 2, 3.' on every speech model. Verified: xcodebuild test 606 passed, 0 failures (1 pre-existing skip); generic iOS build succeeds on dev 976267e; simulator repros before/after for the error and the sample showing setCategory off the main thread.

…the Hex key retries it

X testers on build 56 and earlier: "The keyboard keeps either bringing me back to
the app or telling me another app is using the microphone" (@itsjling) and "The
another app is using audio message seems inaccurate cause it's showing for me all
the time although it's working fine" (@RedaSjakur).

Cause, reproduced on build 56 (2ef08aa) in an iOS 27.2 simulator on the Mac Studio
(no microphone input): Hex's dictation failed ("Hex couldn't start the microphone.
Try again."), Hex was then closed, and a minute later the Hex keyboard's strip in
Settings' search field still said so. Hex writes its snapshot only while it runs,
so a failed snapshot (heartbeat 0, no time) stayed in the App Group, and
KeyboardViewModel read any .failed snapshot as failed whatever its age. The code
path is unchanged on origin/dev. With a warm session up it was worse: Hex stayed
in .failed, consumeKeyboardCommand ignored every start unless the phase was .ready,
and the full keyboard's Hex key (isWarm = Full Access && warm available) kept
sending starts to it, so nothing recorded until Hex was opened. The keyboard's own
failure snapshot also reads as available for its 6 s, so a Hex key tap right after
an in-keyboard failure went to a warm session that doesn't exist.

Now:
- KeyboardSnapshot.messageAt: when Hex wrote the message (DictationController
  stamps a failure when its phase becomes failed, and keyboardMessage when set; the
  keyboard's own failure stamps its snapshot). The keyboard drops a message, and
  reads a failure as offline, 8 s after it (messageShownFor), and at once for a
  snapshot with no time (an earlier build's). Whether a warm session is up and any
  result stay as they were.
- KeyboardViewModel.dictatesThroughHex (Full Access, warm available, not failed),
  used by the full keyboard's Hex key and its touch-down capture: after a failure
  both keyboards dictate in the keyboard, as the Dictation bar already did.
- Hex takes a keyboard start while failed when the failure was the microphone for
  a keyboard dictation with the model loaded (captureFailed): the next Hex key is
  its Try again. Model failures still wait for Hex to get ready again.

Not changed: the 32 MB memory threshold and keyboardNeedsHex, the two ways the Hex
key opens Hex. In the simulator the handoffs logged "speech model missing here"
after the keyboard's own asset check failed ("anomaly.hex.mobile.keyboard is not
subscribed to assistant.en_US") on a sim whose Hex had never installed a model,
which is the intended path; a device's reason needs its probe log.

Verified: new KeyboardBridgeTests.testAFailureHexWroteAWhileAgoIsOver and
DictationControllerTests.testAWarmSessionsMicrophoneFailureIsRetriedFromTheKeyboard;
KeyboardBridgeTests and DictationControllerTests pass (79 tests, 0 failures, iPhone
17 Pro iOS 27.2 simulator). In the simulator with this build, the same failure no
longer shows in the keyboard a minute later.
X tester @leeladhar (iPhone 16 Pro Max, build 56 or earlier): "the keyboard freezes
while using apps sometimes ... the only way to close the keyboard is to close the
app completely".

Not reproduced in the iOS 27.2 simulator on build 56 (2ef08aa) or origin/dev: 24
host visits (Messages, Contacts, Settings search) with typing, the Hex key and globe
switches kept one keyboard process, no crash or hang report, footprint 29-40 MB with
no growth. The device cause isn't confirmed. The likeliest one left on origin/dev is
the last synchronous audio server round trips on the keyboard's main thread: every
Hex key touch-down set the session's category, read and preferred its inputs
(KeyboardLocalDictation.microphone(for:)), asked isOtherAudioPlaying, and read the
Speech Recognition status for the probe log, all before the page could start. These
wait on the audio server (and TCC) with no timeout; a build 56 crash report from
TestFlight shows Hex's own main thread parked in such a call (setPreferredSampleRate
waiting on SetBatchProperties). #2185 moved the one on the launch path for the same
reason. While blocked, no key, Done or globe responds, and the keyboard can't
dismiss.

Now begin() runs those on the keyboard's audio session queue (after any earlier
capture's release there) and starts the page when they return, unless the job was
dropped, stopped or replaced meanwhile. Done before the page was asked to start
ends quietly at once instead of leaving the page to start a capture nobody wants.

Verified: builds clean; in an iPhone 17 Pro iOS 27.2 simulator (WebKit's mock audio
device) the Hex key's touch-down capture still starts and is kept: "first audio
+124 ms after touch-down (... getUserMedia +83 ...)", 13.5 s recorded as 643,072
frames at 48 kHz (no gap), then transcription as before.
… mid-dictation

TestFlight tester on X (iPhone 13 Pro Max, iOS 27, build 56 or earlier): "in progress
recording dropped when I took a screenshot".

What a screenshot does to an in-keyboard dictation, in an iPhone 17 Pro iOS 27.2
simulator on the Mac Studio (WebKit's mock audio device records there):
- The host only going inactive doesn't stop it: with Control Center pulled down for
  4 s mid-dictation the page kept sending audio (700,416 frames at 48 kHz over a
  14.8 s recording, no gap).
- A screenshot taken with Cmd-Shift-3 hid the keyboard and the probe log said
  "keyboard hidden while recording: finishing with what was said" at the shutter.
  The simulator can only take one through its hardware keyboard, which also turns
  on keyboard minimization, so it doesn't show which of the two hid it. On an
  iPhone, iOS 26 and 27 open the screenshot's full-screen preview over the app, and
  a keyboard that hides is torn down and its WebKit processes suspended, so a
  dictation can't keep recording through that.
- On build 56 a keyboard hidden mid-recording lost the dictation:
  viewWillDisappear only dropped a touch-down capture, and the recording went with
  the view controller (reproduced in #2163). Since #2163 on origin/dev it stops,
  keeps what was said, and the text goes into the same field when the keyboard
  comes back within 30 s, else is offered as Paste dictation.

The one way left for a capture to end early with the keyboard still up is the
page's audio context: iOS may suspend or interrupt it for other audio and the page
never resumed it, so no more audio came and the 6 s watchdog ended the dictation.
The page now resumes a context that isn't running every 250 ms while it records,
and stops trying when the capture stops.

Not changed: a keyboard hidden by a screenshot's preview still ends the recording
(with its text kept, as above).

Verified: builds clean; in the simulator the touch-down capture still starts and
records through Control Center with no gap (numbers above).
TestFlight (@kevin_miller, Nov 20, build 56 = 2ef08aa): "Test 1, 2, 3"
came out "Test three." / "Test 3" from the keyboard's in-keyboard dictation
and "test 123" in the app's Dictate box.

What each speech model writes (macOS 27.2.1 on the Mac Studio, `say`
clips, 4 voices): Apple's SpeechTranscriber (in-keyboard dictation) writes
"Test one, 2, 3.", "Test, one, two, three.", "Testing 123.", "Counting one,
two, three, four."; DictationTranscriber writes "Test 123", "Counting 1234";
Parakeet v2 through FluidAudio 0.15.5 (the app) writes "Test 1, 2, 3",
"Testing 123" (no pauses), "1, 2, 3, testing".

Responsible steps:
- "Test three." / "Test 3": at build 56 Clean up what I say deleted every
  word over P(DELETE) 0.98 with no guard. The real Core ML model gives the
  count words but the last 0.97-0.999 ("Test one two three." -> "Test
  three.", "Test 1, 2, 3." -> "Test 1, 3."): it reads a count as a
  self-correction. CleanupGuard (#2132, after build 56) already keeps every
  count on origin/dev; this commit adds tests that pin it.
- "test 123": Parakeet and DictationTranscriber write "123" themselves; no
  Hex step merges digits. Apple Intelligence's cleanup (same prompt, macOS
  27.2 model) kept "Test 1, 2, 3." in 3 of 3 runs per input.
- The paths still disagreed on origin/dev: the keyboard inserted "Test one,
  2, 3." where the app inserted "Test 1, 2, 3.".

Change: SpokenCleanupRules.spokenCounts, run by CleanupTagger.cleanIfEnabled
right after spoken symbols (every style but Verbatim, English, Clean up what
I say on or off, before the tagger, in the app and in-keyboard dictation),
writes a count in digits: three numbers or more counting up from one, in
words or digits, with only a comma or a space between them ("one, 2, 3",
"one two three" -> "1, 2, 3"), and "123"... right after "test"/"testing"
when it ends its clause ("Test 123" -> "Test 1, 2, 3"; "test 123 passed",
"room 123", "Counting 1234" stay). Decision: the "123" split is limited to
test/testing because a bare "123" is usually a real number. Format lists
already never makes a list of a count (each item would be empty); tests
prove it. README gains the Spoken counts paragraph.

Verified on the iOS 27.2 simulator (Mac Studio), real Core ML tagger:
TranscriptCleanupGuardTests 9, CleanupGuardTests 6,
SpokenCleanupRulesTests 19, HinglishRomanizerTests 7, SpokenListsTests 10,
InformalContractionTests 4, FrenchSpacingTests 3, CleanupTaggerTests 15,
DictationControllerTests 50: 123 tests, 0 failures. New:
testSpokenCountsBecomeDigits, testOtherNumbersStayAsSaid,
testCountsAreNoList, testCountsKeepEveryNumberFromEverySpeechModel (also
shows the guardless tagger's "Test three."),
testACountReadsTheSameFromTheAppAndTheKeyboard (app with Parakeet's text
vs keyboard with both Apple modules' text, every style, Lowercase).
@R44VC0RP
R44VC0RP merged commit 56a93e8 into dev Nov 21, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant