True realtime buffer writes/re-writes

[cbenz]

Problem

When transcription_backend: realtime-ws is used with websocket_provider: elevenlabs and websocket_model: scribe_v2_realtime, hyprwhspr waits for the user to press "stop" before transcribing and injecting text via ydotool. It should inject each committed segment as it arrives from the server, providing real-time dictation.

Root Cause

Two issues in the code:

1. ElevenLabsRealtimeClient never forwards transcripts in real-time

In lib/src/elevenlabs_realtime_client.py, the on_committed_transcript and on_partial_transcript event handlers only store text in internal buffers:

def on_committed_transcript(data):
    text = data.get('text', '')
    with self.lock:
        self._current_transcript = text
        self._committed_segments.append(text.strip())  # stored, never pushed
    self._transcript_event.set()  # only consumed by commit_and_get_text()

There is no set_partial_transcript_callback() implementation, so the partial preview shown in the OSD never receives data for ElevenLabs.

2. Partial preview callback hard-gated to OpenAI only

In lib/src/backends/realtime_ws_backend.py, _is_realtime_whisper_preview_enabled():

def _is_realtime_whisper_preview_enabled(self, provider_id, model_id, realtime_mode):
    return (
        self.config.get_setting('mic_osd_enabled', True)
        and provider_id == 'openai'              # ← only OpenAI
        and model_id == 'gpt-realtime-whisper'
        and realtime_mode == 'transcribe'
        and self._realtime_partial_callback is not None
    )

This means apply_partial_callback() always calls set_partial_transcript_callback(None) for ElevenLabs, clearing any preview.

Expected Behavior

• Each COMMITTED_TRANSCRIPT from ElevenLabs should be immediately injected via ydotool (real-time dictation)
• PARTIAL_TRANSCRIPT updates should be forwarded to the OSD for live preview (when mic_osd_enabled)

Environment

• hyprwhspr 1.37.0-1 (Arch Linux)
• ElevenLabs API (Scribe v2 Realtime — scribe_v2_realtime)
• Config: transcription_backend: realtime-ws, websocket_provider: elevenlabs, websocket_model: scribe_v2_realtime

[goodroot]

Hey there - try 1.39.0, this backend was mostly re-written to add some new OSD behaviour.

Edit:

Actually, this won't address exactly what you're after.

I will look at incremental pasting soon!

[goodroot]

Alright, okay. I initially misunderstood. But you are correct, and there's a good reason for this.

When you see behavior within a real time transcription on macOS or a phone or something like that, there's a very flexible and consistent keyboard API that allows programs to backspace and rewrite changes.

Pasting is just one aspect. You also need to be able to safely and consistently revise what has already been pasted.

This is actually extremely difficult to do across Linux ecosystems broadly because of how many edges there are across every desktop environment and protocol.

The compromise as of now is that you can see what you're saying when the partial previews are enabled.

It's been a while since I last looked at this, but I will look again. This may be a very significant feature addition.

[cbenz]

Yes, I am aware of the implications, I thought about that too, since I posted the issue.

I started a POC to do the dictation real-time in a layer-shell window in which I have much more control than with any window.
When validating, the paste would be done.

Actually my goal would be to be able to edit text manually (select/delete/insert/replace) during the dictation. For example, start dictating, select a misspelled group of words, dictate to replace them, move again the cursor at the end by pressing "end" on the keyboard, or anywhere in the text, etc.

The first step is to be able to inject the text as we speak.

[goodroot]

Ahh ok I am very keen to hear how your experiments go.

---

Not the same thing, but there is this:

https://github.com/goodroot/hyprwhspr/blob/main/docs/CONFIGURATION.md#continuous-mode

To that point, I suppose too there is a philosophical angle. 🤔

1. There's real time streaming, which would paste and re-shape text as it forms. It would be great to have this. If this was a web or Mac app, done. A browser/OS could do that no problem.

2. Again, not the same. But there is quicker, automatic pasting on silence detection so that the moment a thought is over, it's pasted. I don't use this mode personally because I'm not at all eloquent. Depending on how you speak, and how "quickly" VAD detects silence (configurable), one might mirror "almost real time". But this seems perilous and probably not satisfying.

---

If we rule out the second option, which we probably should, then what we'd need to identify window managers that are supportive of real-timeliness.

Which window manager you're using? I see that you're on Arch.

I use Arch/Niri.

https://github.com/niri-wm/niri/blob/7f26c3ee804fb6ed458ef7fb0e3c794f14e0b3bc/src/handlers/mod.rs#L281

This is something similar to macOS (input-method-v2 which is experimental src), and it's the primitive that we would need to apply. Though this is where we get into complicated matrix style support. As in, this is Wayland, this is Niri, or hyprland; it could "work*".

A second caveat is that the actual method itself still needs to be supported by a client. Many of them would, like GTK probably. But some terminals, electron apps, and whatever else, it might not work; it would be random and brittle.

There's also an exclusivity problem: the protocol spec is explicit that only one input method can be bound per seat at a time (documented). So this would directly conflict with anyone already running ibus/fcitx for another language/keyboard. So we could not gracefully layer alongside it, we'd have to detect and refuse to grab it. Basically any non-latin user would have that conflict, and there are other potentially conflicting cases.

So immediately there are very real challenges. 🫠

---

If I gaze into a crystal ball, which I don't have, I imagine that input method maturity will improve now that most of us have moved over to Wayland. And these sorts of input related challenges, especially with LLMs and agentic control planes, are generally top of mind, I think, for some developers.

So, maybe if we reflect on this in a few months, it might be trivial to use a new version of input-method to get real-time for free, with only a few edges.

But always encourage experimentation.

Sorry for the TED talk. 😀