First off, thanks for Dictate Extended. It has become a fixed part of how I write on mobile and my most used app besides my regular (offline) keyboard app.
Why
Right now a rewording is a single non-streamed POST /v1/chat/completions, so the keyboard shows "Rewording…" until the entire answer is finished. Streaming would turn that wait into visible progress:
- The first words appear after roughly a second instead of after the full generation, so you can start reading along and decide whether the result is going in the right direction while the rest is still arriving.
- A longer rewording stops feeling like the app has frozen.
- If a backend stalls, it becomes obvious within seconds instead of after a long timeout, because a stream that should be producing tokens produces none.
- Possibility to.interrupt if model goes the wrong way (e.g. inserting multiple options etc.), especially for local models
The effect is biggest with slower backends (self-hosted servers, larger models, machines that need to wake up first), where a rewording can take tens of seconds. Local models benefit as well: on older phones an on-device rewording runs long enough that seeing early output makes a noticeable difference.
What it would take
Send "stream": true on the chat request and read the text/event-stream response: each data: line is a JSON chunk, data: [DONE] ends it. Text arrives as choices[0].delta.content instead of choices[0].message.content.
Three details worth knowing before anyone starts:
Reasoning models use a separate field. Some models return their thinking in delta.reasoning (and message.reasoning in the non-streamed shape), right next to delta.content. Only content belongs in the result; reasoning has to be ignored rather than concatenated, otherwise the model's thinking lands in the user's text field. On one rewording I measured 16 characters of content against 953 characters of reasoning.
Usage arrives late or not at all. Many servers only send a usage object in the final chunk, and some need stream_options: {"include_usage": true} for it, so token accounting should tolerate its absence instead of treating it as an error.
Not every endpoint streams cleanly. Buffering proxies, gateways that strip SSE, providers that ignore the flag. A per-account switch, or a fallback to the non-streamed path on the first malformed chunk, would keep this from becoming a regression for setups that work fine today.
What it is not
This is not about typing the answer into the text field character by character. The result can still be committed in one piece once the stream ends. The value is in the transport: knowing that the server started answering, knowing that it is still answering, and being able to show that (and being able to react faster if something failed silently e.g.).
First off, thanks for Dictate Extended. It has become a fixed part of how I write on mobile and my most used app besides my regular (offline) keyboard app.
Why
Right now a rewording is a single non-streamed
POST /v1/chat/completions, so the keyboard shows "Rewording…" until the entire answer is finished. Streaming would turn that wait into visible progress:The effect is biggest with slower backends (self-hosted servers, larger models, machines that need to wake up first), where a rewording can take tens of seconds. Local models benefit as well: on older phones an on-device rewording runs long enough that seeing early output makes a noticeable difference.
What it would take
Send
"stream": trueon the chat request and read thetext/event-streamresponse: eachdata:line is a JSON chunk,data: [DONE]ends it. Text arrives aschoices[0].delta.contentinstead ofchoices[0].message.content.Three details worth knowing before anyone starts:
Reasoning models use a separate field. Some models return their thinking in
delta.reasoning(andmessage.reasoningin the non-streamed shape), right next todelta.content. Onlycontentbelongs in the result;reasoninghas to be ignored rather than concatenated, otherwise the model's thinking lands in the user's text field. On one rewording I measured 16 characters of content against 953 characters of reasoning.Usage arrives late or not at all. Many servers only send a
usageobject in the final chunk, and some needstream_options: {"include_usage": true}for it, so token accounting should tolerate its absence instead of treating it as an error.Not every endpoint streams cleanly. Buffering proxies, gateways that strip SSE, providers that ignore the flag. A per-account switch, or a fallback to the non-streamed path on the first malformed chunk, would keep this from becoming a regression for setups that work fine today.
What it is not
This is not about typing the answer into the text field character by character. The result can still be committed in one piece once the stream ends. The value is in the transport: knowing that the server started answering, knowing that it is still answering, and being able to show that (and being able to react faster if something failed silently e.g.).