Repository navigation
Conversation
…I to the Web Speech API
…I to the Web Speech API
|
@padenot - I just realized this PR was never merged. Can you please take a look when you get a chance? |
…I to the Web Speech API
- Fix formatting for `true` code span - Fix typo: 'an per-API' -> 'a per-API' - Remove redundant Alternate Options subsection
hoch
left a comment
hoch
left a comment
There was a problem hiding this comment.
Mostly nits, but I do have some questions.
Review comments addressed:
- install(): add the transient user activation requirement to the
algorithm so it matches the privacy section's claim. The check is
keyed off the (masked) availability status so that whether install()
requires activation cannot itself leak the true download status.
Document the NotAllowedError rejection and link the privacy bullet to
the new step.
- Download Masking: drop the vestigial "API" component from the masking
state tuple.
- Installation-time friction: name the "on-device-speech-recognition"
policy-controlled feature directly instead of "per-API".
- Disk Space / Runtime Shared Resources: "set to `true`" and
"a SpeechRecognitionErrorEvent".
- OS-Provided Models: reword to match the Writing Assistance source text
("still need to ensure ... even if the user agent has less control"),
making clear this restates requirements defined elsewhere rather than
adding an unmarked normative one.
Editorial cleanups of the same classes found while reviewing:
- Remove remaining vestigial multi-API wording ("these APIs") and the
"UA" abbreviation; spell out "user agent".
- Mark the user-agent transparency "should" in the cloud section per
this section's <strong> convention; de-2119 developer guidance.
- Nest Download Masking under Language Pack Availability so that
masking/friction/cancelation/eviction are siblings, as in the source.
- Title-case the "Installation-Time Friction" heading; fix a subject
shift in the unmasking sentence; strip trailing whitespace.
Also merges upstream/main (needed because WebAudio#200 restructured install()).
padenot
left a comment
padenot
left a comment
There was a problem hiding this comment.
Do we know why we can't have this be normal normative requirements, and we're putting them in their own sections? It makes it harder to read no?
|
@padenot Done, I moved all the normative requirements into the API Description section (under |
padenot
left a comment
padenot
left a comment
There was a problem hiding this comment.
I like this, thanks a lot for doing it, and revising it to inline the behaviour inside the normative text. Some comments inline, but generally in favour.
|
|
||
| The Web Speech API can support both server-based (cloud) and client-based/embedded (on-device) recognition and synthesis. The {{SpeechRecognition/processLocally}} attribute allows developers to indicate a requirement for on-device processing. | ||
|
|
||
| When {{SpeechRecognition/processLocally}} is `false` (the default), user speech data might be sent to a remote server for processing. Web developers need to be aware of this possibility and the associated privacy implications if they do not explicitly request local processing, and <a href="#req-cloud-transparency">user agents are encouraged to be transparent</a> with users about where speech processing occurs. |
There was a problem hiding this comment.
This reminds me we might need investigate investigate having processLocally to be true initially, and/or disallowing changing it.
In Firefox, it's always ever local, and will never be remote, and it seems to me that that the advances in technology will make this possible more and more, with high quality.
|
|
||
| As specified in <a href="#req-disk-space">the storage pressure rules</a> for {{SpeechRecognition/install()}}, in the event of storage pressure, user agents are expected to balance the utility of on-device speech recognition with the disk space its language packs take up—possibly by having {{SpeechRecognition/install()}} resolve to `false` for new downloads or by freeing up disk space in other ways, while remaining mindful of the privacy impacts discussed in <a href="#on-device-model-privacy-availability-eviction">Download Eviction</a> and potentially involving the user via download-time prompts or a browser UI for managing downloaded language packs. | ||
|
|
||
| If a previously installed language pack is evicted (e.g., by the user or due to extreme storage pressure) while it might be in use or expected to be available, subsequent attempts to use it (e.g., via {{SpeechRecognition/start()}} with {{SpeechRecognition/lang}} set to that language and {{SpeechRecognition/processLocally}} set to `true`) are expected to <a href="#req-eviction-graceful-failure">fail gracefully</a>. This might involve {{SpeechRecognition/available()}} returning {{"downloadable"}} (or {{"unavailable"}} if no longer supported), and {{SpeechRecognition/start()}} firing a {{SpeechRecognitionErrorEvent}} with an appropriate error code like {{SpeechRecognitionErrorCode/service-not-allowed}} or {{SpeechRecognitionErrorCode/language-not-supported}}. |
There was a problem hiding this comment.
It doesn't have to fail, but it can. In Firefox, you can manage the installed models in couple ways, and delete them when you want, but if a session is running, it will continue to work.
We could imagine making subsequent sessions still work since the model is still in memory. Generally after this it won't work though, e.g. after stop().
|
|
||
| On-device speech recognition can consume significant runtime resources like CPU, memory, and potentially specialized hardware accelerators. | ||
|
|
||
| As described in <a href="#req-runtime-resources">the runtime resource requirements</a> of the [=start session algorithm=], user agents are expected to ensure that one web page's use of on-device speech recognition does not overly interfere with another web page's use of the API, another web page's general operation, or overall system stability. For example, a background tab ought not to be able to monopolize speech processing resources and prevent a foreground tab from using them. |
There was a problem hiding this comment.
This usually doesn't apply for audio-related things. There's is very little notion of "background" for audio, since we can hear things not in foreground. It's also quite hard to know if the text is displayed, e.g. a subtitles, in which case we could conceivably discard them / stop processing the audio.
But realistically, if I'm dictating a document, transcribing a meeting, or talking with an AI agent (that will likely answer in audio), I really want this to work, regardless of background/foreground.
|
|
||
| As described in <a href="#req-runtime-resources">the runtime resource requirements</a> of the [=start session algorithm=], user agents are expected to ensure that one web page's use of on-device speech recognition does not overly interfere with another web page's use of the API, another web page's general operation, or overall system stability. For example, a background tab ought not to be able to monopolize speech processing resources and prevent a foreground tab from using them. | ||
|
|
||
| This specification does not mandate any particular mitigation strategy, but possible approaches include queuing requests, rate limiting, prioritizing foreground tabs, or detecting abusive behavior, and failing speech recognition operations (e.g., by firing a {{SpeechRecognitionErrorEvent}} with {{SpeechRecognitionErrorCode/service-not-allowed}}) when necessary to prevent resource exhaustion or instability. |
There was a problem hiding this comment.
Our current approach (until we generalize hardware inference) is to limit to a single concurrent session. This isn't amazing for fingerprinting, but exhausting the machine's resources isn't amazing at all.
This PR adds the privacy and security mitigations used by the Writing Assistance APIs to the on-device speech recognition part of the Web Speech API.
Closes #158
Preview | Diff