clicky-chess

A personal chess coach that lives in your Mac's menu bar. It watches your chess.com games, listens to your questions via push-to-talk, and gives you one clear piece of advice at a time — spoken aloud so you never have to look away from the board.

This is a fork of Clicky by @FarzaTV. Clicky is an open-source AI companion that can see your screen, talk to you, and point at things. clicky-chess takes that foundation and reshapes it into a chess-only coach that understands board positions, remembers your games, and learns your patterns over time.

What it does

• Press Control + Option during a chess.com game to talk to your coach
• It reads the board position using local OCR (Apple Vision) on the move list — no guessing at pieces from pixels
• Gives you one actionable piece of advice per interaction — what to do and why, in plain English
• Speaks its response aloud via text-to-speech so you can keep your eyes on the board
• A blue rook follows your cursor and can point at specific squares when relevant
• Remembers every move of your current game, even in long 60+ move endgames
• Learns across games — tracks what openings you've studied, mistakes you repeat, and patterns in your play

How it remembers

clicky-chess has two layers of memory:

Per-game memory: The full move history and board position (FEN) live in the system prompt, which is sent fresh with every request. This never gets truncated. Even if you're on move 50, the coach knows exactly what happened on move 3.

Cross-game memory: A persistent coaching log (stored as JSON on your Mac) tracks concepts the coach has taught you, mistakes you've made, and patterns it's noticed. When you start a new game, the coach already knows your tendencies. It won't re-explain the Sicilian Defense if it taught you that last week. It will watch for that knight fork you keep missing.

The coaching log lives at ~/Library/Application Support/clicky-chess/coaching-log.json.

What changed from Clicky

This fork makes targeted changes to turn a general-purpose screen companion into a chess-specific coach. The core infrastructure (voice pipeline, screenshot capture, TTS, overlay animations, multi-monitor support) is untouched.

New files:

• ChessGameState.swift — Game state model: FEN, move history, player color, key moments, FEN validation
• ChessMoveListOCR.swift — Local OCR using Apple's Vision framework to extract chess notation from screenshots as plain text, so the AI doesn't have to read moves from a JPEG
• MasterCoachingLog.swift — Persistent coaching log that tracks learning across games

Modified files:

• CompanionManager.swift — Chess system prompt, game state tracking, FEN/move tag parsing, OCR pipeline integration, 300ms screenshot delay for animation settling
• CompanionPanelView.swift — "clicky-chess" branding, "New Game" button, model picker with Auto option
• OverlayWindow.swift — Blue rook cursor shape instead of triangle
• MenuBarPanelManager.swift — Rook icon in the menu bar
• CompanionScreenCaptureUtility.swift — Higher resolution (1568px) and quality (92%) screenshots for better board reading
• ClaudeAPI.swift — Dual backend support for Anthropic and OpenRouter APIs
• worker/src/index.ts — New /chat-openrouter route that converts Anthropic message format to OpenAI format

Model support:

• Anthropic (Claude Sonnet, Claude Opus) — direct via Anthropic API
• OpenRouter (any model) — select "Auto" in the model picker to let OpenRouter choose the best model, or configure any specific model. The Cloudflare Worker handles format conversion between Anthropic and OpenAI message formats automatically.

Setup

Prerequisites

• macOS 14.2+
• Xcode 15+
• Node.js 18+
• A Cloudflare account (free tier works)
• API keys for: Anthropic, AssemblyAI, ElevenLabs
• (Optional) OpenRouter API key if you want to use non-Anthropic models

1. Set up the Cloudflare Worker

The Worker is a proxy that holds your API keys so they never ship in the app.

cd worker
npm install

For local development, create a worker/.dev.vars file with your keys:

ANTHROPIC_API_KEY=sk-ant-...
ASSEMBLYAI_API_KEY=...
ELEVENLABS_API_KEY=...
OPENROUTER_API_KEY=...        # optional, only if using OpenRouter models

Run it locally:

npx wrangler dev

This starts a local server at http://localhost:8787.

To deploy for real usage:

npx wrangler secret put ANTHROPIC_API_KEY
npx wrangler secret put ASSEMBLYAI_API_KEY
npx wrangler secret put ELEVENLABS_API_KEY
npx wrangler secret put OPENROUTER_API_KEY    # optional
npx wrangler deploy

If you deploy, update the worker URL in CompanionManager.swift (search for workerBaseURL).

2. Build and run

open leanring-buddy.xcodeproj

In Xcode:

1. Select the leanring-buddy scheme (the typo is inherited from Clicky — it stays)
2. Set your signing team under Signing & Capabilities
3. Cmd + R to build and run

The app appears in your menu bar (not the dock). Click the rook icon to open the panel.

3. Grant permissions

• Microphone — for push-to-talk voice capture
• Accessibility — for the global keyboard shortcut
• Screen Recording — for capturing your chess board

4. Play chess

1. Click "New Game" in the clicky-chess panel
2. Start a game on chess.com
3. Press Control + Option to talk to your coach
4. Listen, make your move, repeat

Architecture

For the full technical breakdown, see CLAUDE.md. The short version:

Menu bar app with two transparent overlay windows — one for the control panel, one for the blue rook cursor. Push-to-talk streams audio to AssemblyAI, captures a screenshot, runs local OCR to extract the move list, then sends everything to the AI. The response streams back via SSE and plays through ElevenLabs TTS. Game state lives in the system prompt so it's never truncated. All API calls go through a Cloudflare Worker proxy.

Credits

This project is a fork of Clicky by Farza. Clicky is the open-source AI companion that made this possible — the voice pipeline, screenshot system, TTS integration, cursor overlay, and Cloudflare Worker proxy are all Farza's work. clicky-chess just stands on those shoulders and points them at a chess board.