Repository navigation
Windows: Maki voice agent (ElevenLabs + Hermes), songs, apps, reminders, credits gauge - #578
Open
MithunSidhaarth wants to merge 32 commits into
Open
MithunSidhaarth wants to merge 32 commits into
MithunSidhaarth wants to merge 32 commits into
Conversation
…edia controls Reads every player through GlobalSystemMediaTransportControls (event-driven, no timers) and feeds the existing music card and pill. Apple Music and Spotify get their own pills; Now Playing follows any other app (browser tabs, VLC). The card's volume slider drives the system volume through Core Audio. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A low-level hook takes the volume keys from Windows' popup (only while the setting is on; injected keys pass through) and the island shows the level. Brightness, AC/battery and battery level come from power-setting notifications; waking from sleep now triggers the Monday recap check. The clipboard listener keeps recent texts in memory only, skipping anything password managers mark private. All of it is event-driven, nothing polls. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
"Claude (your plan)" in the chat's model picker answers through the Claude Code CLI logged in with a Pro or Max account: no tools, no user settings (so Coucou's hooks don't see it), no MCP, no saved session. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…aptop volume keys - The island re-asserts topmost whenever another app comes forward (event-driven). - Volume keys injected by laptop hotkey services are taken too. - Music card: blurred album-art backdrop, bigger cover, artist in full. - Auto-close offers 5 s; long pill names no longer run under the mini Mochi. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Calendar: Google, Outlook or iCloud through the calendar's private iCal link, kept in the credential store; https/webcal only; next events on the card. Polled every 5 minutes only while the pill is on. - Island extras → Always show the island (on by default): the compact island no longer hides after a minute. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The chat passed only a PDF's name, so the CLI answered that no file matched. It now runs in the inbox with the Read tool allowed for that folder only (--allowedTools "Read(./**)", dontAsk) and is pointed at ./<file>. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- iPhone → laptop: a Share Sheet Shortcut saves into iCloudDrive/Coucou/ to-laptop; a ReadDirectoryChangesW watch (no polling) copies it to the clipboard and shows it in the island; a link opens on click. - Laptop → iPhone (opt-in): copies go to iCloudDrive/Coucou/to-iphone.txt for a "Get from laptop" Shortcut. - ntfy: finished / failed / needs-your-answer alerts to the phone when the PC has been idle for a minute; topic kept in the credential store. - Settings → iPhone with the Shortcut steps and a test button. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…Link notify.rs reads toasts with UserNotificationListener once Coucou has a package identity (identity/: sparse AppxManifest + make-identity.ps1, the exe manifest names it). Phone Link calls show as "X is calling", messages and other apps as a card; nothing is stored. Off by default. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ng and a sound for messages - The caller is the toast line that isn't Phone Link's own wording. - A call's card stays until Phone Link removes its toast (answered, declined, missed), at most 90 s; it rings meanwhile. A click brings Phone Link's call window forward; messages never cover a ringing call. - Notifications get Mochi's pop. - No package identity: Windows 11 lets Coucou ask for notification access without one, and the identity kept the island's web view from starting. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… on two pages - Focus: 25/50 min countdown in the compact island; notifications held (calls still ring); a sound and a phone ping at the end. - To-do: add, tick, delete; kept in the island's storage. - Inbox: calls and messages from Phone Link; back at the keyboard after two minutes away, the island sums up what came. - Claude plan at 80 % / 95 % of the 5-hour window: a card and a phone alert. - `coucou --notify "Build done"` sends to the phone from any script. - Claude finishing always reaches the phone. - Up to 8 declared pills, four per page on a sliding track (wheel, swipe, dots). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…compact island - Top bar: Focus (orange while running), To-do and Inbox (dot when unread) buttons; a click declares the pill if needed and opens its card. The three also show in Settings → Integrations. - Compact island: while music plays, the cover, title · artist, play/pause and next; a live activity shows over it and it comes back after. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…i-player buttons - Focus, To-do and Inbox are views of the island, tabs next to Overview and Ask, no longer pills (removed from the catalog and Settings). - Focus takes any length typed in (1–600 min) besides 25 and 50. - The focus countdown shows only in the compact island, so it never sits over the top bar's tabs. - The mini player's buttons keep their mousedown: a click pauses or skips instead of opening the island. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… go through - Call card: ✓ answers and ✕ declines/ends by pressing Phone Link's own buttons through UI Automation (callctl.rs); with nothing to press it brings Phone Link forward. After answering: "On call · name" with ✕. - ntfy: the title is RFC 2047-encoded when not ASCII (an accent or emoji made the request fail silently), "ntfy.sh/topic" pasted works, and every alert logs whether it was sent and why not. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… resets Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… top middle Always show the island is now off by default, and the invisible strip that brings the island back is 360×8 instead of 240×6, easier to find with the cursor at the top middle of the screen. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… window changes - The compact island's mini Mochis (an agent at work, e.g. the orange Claude Desktop one) are no longer replaced by the mini player: the island widens and the player stops short of them, clear of play/pause and next. - Mode changes and wake-strip collapses go to the log, to diagnose an island that won't wake. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The ↻ runs `claude -p /usage` (a slash command: no model call, no tokens), reads the session and week percentages and reset times out of its text (parseUsageText, tested) and applies them as if the status line had sent them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- voice.rs: the wake word on the device (a list grammar on Windows' speech engine), then one dictation turn for the command, words streamed to the island; the wake recogniser lets go of the mic first. Retries until Windows' online speech recognition is on. Nothing recorded, transcripts never logged. - core/intents.ts (tested): music, volume, pills, focus, timers and reminders, to-dos, open apps/sites, Claude plan usage, agents, time, decline a permission (approving by voice: never). - Anything else: Claude (the user's plan via Claude Code, Haiku) with Coucou's context; the answer opens in the island and is spoken with a Windows voice. - Settings → Voice: wake word (Maki), language, voice, speak answers; a mic in the top bar listens without the wake word. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Windows' online dictation ends every turn empty for a desktop app without a package identity (tried in every English variant), so the command is now recognised offline like the wake word: the page hands voice.rs every phrase it understands (voicePhrases, tested: each one parses to a command), and "Maki <command>" works in one breath. Nothing leaves the PC. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A one-shot RecognizeAsync ended at once on this PC ("unknown"), so every
command after a bare "Maki" came back as not understood. The command turn now
runs the same continuous session as the wake listener, for up to 7 s, and
accepts low confidence (only listed commands can match). "Maki <command>" in
one breath also counts at low confidence; "Maki" alone still has to be sure.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The wake word stays on the device; once heard, the listener steps aside and the island opens a live conversation with the user's ElevenLabs agent (signed URL fetched in Rust with the key from the credential store; the page never sees it). Mic streamed as 16 kHz PCM, the agent's voice played back, both sides captioned in the island; a click ends it, the agent hangs up on "bye" or after 20 s of silence. - Client tools: ask_hermes runs `hermes -z` on this PC (115 s cap); coucou does music, volume, focus, timers, to-dos, plan usage, agents, open. - WebView2's microphone request is granted for the island's own page only (a notch window can't show the pop-up). CSP allows ElevenLabs' socket. - Settings → Voice: ElevenLabs agent switch and API key. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…usic while listening - open_app finds Start menu apps (Store ones too) instead of a web search - lock_screen - voice timers become to-do reminders: kept across restarts, shown with their time, ring in the notch and on the phone - music pauses while Maki listens (the speakers drowned the mic) and resumes after Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…app list Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… toolset (~3x faster) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… by voice, 5 min Hermes budget Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s out of the log Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts: # windows/src-tauri/src/settings.rs # windows/src/i18n/extra.json
Author
|
Merged |
…he talks and wears her moods - Focus: a progress ring with the time inside, 25/50 chips or any length, Start/Pause/Stop, the end time - Inbox: avatars with a call/message badge, who and what on one line, a count, a calm empty state - During a Maki call Mochi listens (cyan), thinks (purple), works (blue), bobs to her voice and widens his eyes at yours; her [happy]/[surprised]/[sad]… tags become his face and leave the caption - The voice card shows bars on the voice instead of a mic Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… sound-alikes, fairer confidence - Windows ends a continuous session after silence or a mic hiccup: it's started again (new recogniser if needed) - With the ElevenLabs agent on, only the wake phrases are in the grammar (not 400+ command phrases) - "Mackie", "Macky"… count as Maki; a word boundary keeps "makiko" out - A bare wake counts at High/Medium or a raw score of 0.5+; too-unsure ones are logged (score only) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Builds on #432 (and #418). The first commits are theirs; this PR adds the last 7, all under
windows/.Maki, a voice for Mochi on Windows
Say "Maki": the wake word stays on the device (WinRT offline grammar, from #432). Then, opt-in in Settings → Voice:
Get-StartApps, never a web search), websites, web search, a note to the phone through ntfy, lock the screen.hermes -z, lean toolset). It runs in the background and Maki calls back with the answer, so no call minutes are spent waiting; Hermes' steps (the commands it runs) stream into the island while it works, read from its session store.Notes
eleven.rshasDEFAULT_AGENTpointing at my agent as a placeholder; anyone else sets their own agent id (plus.elevenAgent). Happy to turn that into a field in Settings, or drop the ElevenLabs part entirely if you'd rather keep this upstream lean. Hermes is only used when installed.i18n/extra.json).Tests
npm test: 368 pass (new: Hermes callback line).cargo test: new tests for app-name matching and YouTube result parsing.🤖 Generated with Claude Code