clawbunny · clawbunny · Jul 13, 2026
diff --git a/DISCOVERY.md b/DISCOVERY.md
@@ -0,0 +1,375 @@
+# GrokAuto Discovery Notes
+
+**Host:** `dsd.890.be`  
+**User:** `grok`  
+**Project path:** `/home/grok/grokauto`  
+**Local mirror:** `/Users/martin/dsd_890_be/grokauto`
+
+GrokAuto is a headless-browser automation tool that drives `grok.com` through the Chrome DevTools Protocol (CDP). It reads tasks from a JSON config, sends each prompt to Grok, waits for the streaming response to finish, and writes the results to timestamped Markdown files.
+
+---
+
+## 1. Architecture
+
+```
+main.py
+├── modules/deps_checker.py      System dependency checks (Chrome, Python, pip, Xvfb, cron)
+├── modules/config_validator.py  JSON config parsing and validation
+├── modules/browser_ctl.py       Launch Chromium and talk CDP over WebSocket
+├── modules/login_validator.py   Heuristic grok.com login detection
+├── modules/task_executor.py     Send prompts, wait for streaming, capture response text
+└── modules/response_writer.py   Format and persist Markdown output files
+```
+
+### 1.1 `main.py`
+
+CLI orchestrator. Sub-commands:
+
+| Flag | Purpose |
+|------|---------|
+| `--diagnose` | Full dependency + configuration diagnostic |
+| `--check-deps` | Dependency report only |
+| `--validate-config` | Config report only |
+| `--run` | Execute tasks from config (default action) |
+| `--prompt "..."` | Run a single inline prompt |
+| `--task-id ID` | Run only one task from config |
+| `--setup-cron "SCHEDULE"` | Install a crontab entry |
+| `--remove-cron` | Remove the crontab entry |
+| `--list-outputs` | List saved `.md` outputs |
+
+The default action is `--run`. Execution flow:
+
+1. Validate dependencies.
+2. Validate and load config.
+3. Launch Chrome with `--remote-debugging-port=9222`.
+4. Check grok.com login state; abort if not authenticated.
+5. Run selected tasks.
+6. Save responses to `output_dir` with mode `0o600`.
+7. Print summary and close browser.
+
+### 1.2 `modules/browser_ctl.py`
+
+- Launches Chromium with `--remote-debugging-port`, `--user-data-dir`, `--headless=new`, and anti-automation flags.
+- Connects to CDP via `websocket-client`.
+- Exposes `navigate_to`, `execute_js`, `wait_for_element`, `get_page_source`, `close_browser`.
+- Pre-flight port check: refuses to launch if `127.0.0.1:debug_port` is already open.
+- Registers an `atexit` cleanup handler.
+
+### 1.3 `modules/task_executor.py`
+
+- Navigates to `https://grok.com`, waits 4s for React hydration.
+- Finds the chat input using a fallback selector chain (`contenteditable`, `textarea`, etc.).
+- Handles both ProseMirror/tiptap `contenteditable` inputs and plain `<textarea>` inputs.
+- Submits by clicking a dynamically-appearing send button, or falls back to `Enter` key events.
+- Detects completion by:
+  1. Disappearance of stop/streaming indicators.
+  2. Response text remaining stable for 3 seconds.
+  3. Hard task timeout.
+- Captures response text from markdown containers, falling back to `main`/body paragraphs.
+- Pauses 2 seconds between successful tasks.
+
+### 1.4 `modules/login_validator.py`
+
+Heuristic login check on `grok.com`:
+
+1. If a "Sign in" button/link is visible → not logged in.
+2. If account/sign-out/avatar UI is present → logged in.
+3. If auth-like cookies exist → logged in.
+4. Otherwise ambiguous.
+
+### 1.5 `modules/response_writer.py`
+
+- `write_response` → `grok_response_<task_id>_<timestamp>.md`.
+- `write_batch_results` → `grok_batch_<timestamp>.md` plus individual files.
+- All files are created with `0o600` permissions.
+
+---
+
+## 2. Configuration
+
+Live config: `/home/grok/.config/grokauto/config.json`
+
+```json
+{
+  "chrome_profile_path": "/home/grok/.config/chromium/Default",
+  "chrome_binary": "/usr/bin/chromium",
+  "debug_port": 9222,
+  "headless": true,
+  "xvfb_display": ":99",
+  "output_dir": "/home/grok/grokauto_outputs",
+  "tasks": [
+    { "id": "gmail_check", "prompt": "...", "wait_for_complete": true, "timeout_seconds": 360 },
+    { "id": "voltrader",   "prompt": "...", "wait_for_complete": true, "timeout_seconds": 360 },
+    { "id": "invest",      "prompt": "...", "wait_for_complete": true, "timeout_seconds": 360 }
+  ]
+}
+```
+
+Validation rules discovered and encoded in tests:
+
+- Required top-level fields: `chrome_profile_path`, `chrome_binary`, `debug_port`, `headless`, `output_dir`, `tasks`.
+- `debug_port` must be an integer.
+- `headless` should be boolean (warned otherwise).
+- `tasks` must be a list.
+- Each task must have a non-empty `id` and `prompt`.
+- Task `timeout_seconds` must be at least 5.
+- Empty task list produces a warning, not a failure.
+
+---
+
+## 3. Tasks
+
+| ID | Purpose | Schedule |
+|----|---------|----------|
+| `gmail_check` | Search unread Gmail, draft/send replies, remove UNREAD label | 09:00 CT daily |
+| `invest` | Screen stocks, compare to Tesla, email analysis to martin@pivetta.be | 07:00 CT daily |
+| `voltrader` | Short-term momentum trading via connected tools (Robinhood, Google Docs) | **Not currently scheduled** |
+
+The `voltrader` task is still defined in config but has been removed from cron. The most recent `voltrader` outputs are from 2026-07-10.
+
+---
+
+## 4. Cron Scheduling
+
+Current crontab for `grok@grok`:
+
+```cron
+CRON_TZ=America/Chicago
+
+# GrokAuto scheduled task
+0 9 * * * cd /home/grok/grokauto && /usr/bin/python3 /home/grok/grokauto/main.py --run --config /home/grok/.config/grokauto/config.json --task-id gmail_check >> /home/grok/.config/grokauto/cron.log 2>&1
+0 7 * * * cd /home/grok/grokauto && /usr/bin/python3 /home/grok/grokauto/main.py --run --config /home/grok/.config/grokauto/config.json --task-id invest >> /home/grok/.config/grokauto/cron.log 2>&1
+```
+
+Cron log: `/home/grok/.config/grokauto/cron.log`  
+Application log: `/home/grok/.config/grokauto/grokauto.log`
+
+Recent cron runs succeeded:
+
+- `gmail_check` on 2026-07-12 09:00 → 63.4s, SUCCESS.
+- `invest` on 2026-07-12 07:02 → SUCCESS.
+
+---
+
+## 5. Live State (as of inspection)
+
+- **Chrome for grokauto:** not running persistently; GrokAuto launches a fresh Chromium for each run and closes it afterwards. No listener on TCP `9222` between runs.
+- **Chrome for gvgrok:** a separate Chromium process is running under the `gvgrok` user with `--remote-debugging-pipe` and `--user-data-dir=/home/gvgrok/.config/chromium`. This belongs to the Google Voice → Grok gateway, not GrokAuto.
+- **Outputs:** `/home/grok/grokauto_outputs` contains timestamped Markdown files, mode `0o600`.
+- **Config dir permissions:** `drwx------` (`0o700`).
+- **Authentication:** grok.com session is valid; login validation reports "Authenticated session detected on grok.com".
+
+---
+
+## 6. Local Test Suite
+
+A `pytest` suite was added under `tests/` to document the discovered behaviour without needing a browser, network, or the remote server.
+
+Run:
+
+```bash
+cd /Users/martin/dsd_890_be/grokauto
+source .venv/bin/activate
+pytest -v
+```
+
+### Coverage
+
+| File | What is tested |
+|------|----------------|
+| `tests/test_config_validator.py` | JSON validation, required fields, type checks, profile health, port availability, task rules, `load_config` |
+| `tests/test_deps_checker.py` | Chrome detection, Python version, pip packages, Xvfb, cron status, profile directory |
+| `tests/test_browser_ctl.py` | CDP dataclasses, `_fetch_json`, `_wait_for_cdp`, `_get_ws_url`, `launch_browser` command shape, cleanup |
+| `tests/test_login_validator.py` | Login heuristics, warning text, browser launch hint |
+| `tests/test_task_executor.py` | Selector builder, `send_prompt` paths, streaming completion heuristics, response capture, task execution |
+| `tests/test_response_writer.py` | Markdown formatting, file permissions, batch summaries |
+
+All 80 tests pass.
+
+---
+
+## 7. Observations and Notes
+
+- The project is intentionally simple: no async framework, no ORM, no external database; just CDP + files.
+- `websocket-client` is imported lazily inside `_send_cdp_cmd` and `launch_browser`, so missing it surfaces as a runtime error rather than an import-time error.
+- `headless=True` combined with `xvfb_display=:99` means the tool relies on Xvfb even in headless mode.
+- Task prompts are long and include tool-calling instructions (`gmail_search`, `gmail_send_message`, `gmail_batch_modify_labels`, `search_connected_tools`, etc.). Grok must have those tools connected in the saved Chrome profile for the prompts to work.
+- The `voltrader` task references a Google Doc named "VolTrader - Current State" and Robinhood schema via `search_connected_tools`. Its removal from cron may be intentional (e.g., market-hours only, manual trigger, or deprecation).
+
+---
+
+## 8. Useful Remote Commands
+
+```bash
+# Diagnose
+ssh -i ~/.ssh/martin_dsd_890_be grok@dsd.890.be 'cd /home/grok/grokauto && python3 main.py --diagnose'
+
+# Validate config
+ssh -i ~/.ssh/martin_dsd_890_be grok@dsd.890.be 'cd /home/grok/grokauto && python3 main.py --validate-config'
+
+# Tail cron log
+ssh -i ~/.ssh/martin_dsd_890_be grok@dsd.890.be 'tail -f ~/.config/grokauto/cron.log'
+
+# List recent outputs
+ssh -i ~/.ssh/martin_dsd_890_be grok@dsd.890.be 'ls -lt ~/grokauto_outputs | head'
+```
+
+---
+
+## 9. Multi-Agent Support (grok.com + kimi.com)
+
+GrokAuto now supports pluggable chat-site agents. The default agent remains
+`grok.com` for full backward compatibility.
+
+### Agent architecture
+
+```
+modules/
+  agents/
+    base.py    # Agent Protocol + shared helpers
+    grok.py    # Grok.com implementation
+    kimi.py    # Kimi.com implementation
+    __init__.py # Registry + factory
+```
+
+Each agent encapsulates:
+
+- `base_url` — the site to navigate to.
+- `input_selectors`, `submit_selectors`, `response_container_selectors`, `streaming_indicator_selectors` — DOM selector chains.
+- `check_login(session)` — heuristic login detection.
+- `warn_not_logged_in(status)` — site-specific login instructions.
+
+New agents can be added by creating a class that satisfies the `Agent` Protocol
+and registering it in `modules/agents/__init__.py`.
+
+### Supported agents
+
+| Name | URL | Notes |
+|------|-----|-------|
+| `grok.com` | https://grok.com | Fully implemented and tested |
+| `kimi.com` | https://kimi.com | Best-effort selectors; verify against live site |
+
+Set the environment variable `KIMI_BASE_URL` to override the Kimi base URL.
+
+### Configuring agents
+
+Global default:
+
+```json
+{
+  "agent": "kimi.com",
+  ...
+}
+```
+
+Per-task override:
+
+```json
+{
+  "id": "grok_summary",
+  "agent": "grok.com",
+  "prompt": "Summarize this..."
+}
+```
+
+If a task omits `agent`, it inherits the global default.
+
+---
+
+## 10. Rich Multi-Line Prompts
+
+Prompts can be supplied in three ways:
+
+### A. Single JSON string (existing behaviour)
+
+```json
+"prompt": "Check email and..."
+```
+
+### B. JSON array of strings
+
+Each array element is trimmed and joined with `\n`:
+
+```json
+"prompt": [
+  "You are a helpful assistant.",
+  "",
+  "Check the user's Gmail for unread messages.",
+  "For each message, draft a polite reply."
+]
+```
+
+### C. External file
+
+The file path is resolved relative to the config file directory (`~` is expanded):
+
+```json
+"prompt_file": "prompts/gmail_check.md"
+```
+
+Rules:
+- A task must provide `prompt` or `prompt_file`.
+- If both are present, `prompt_file` wins.
+- Empty resolved prompts are rejected by the validator.
+
+Example config combining agents and prompt files:
+
+```json
+{
+  "agent": "grok.com",
+  "chrome_profile_path": "/home/grok/.config/chromium/Default",
+  "chrome_binary": "/usr/bin/chromium",
+  "debug_port": 9222,
+  "headless": true,
+  "xvfb_display": ":99",
+  "output_dir": "/home/grok/grokauto_outputs",
+  "tasks": [
+    {
+      "id": "gmail_check",
+      "prompt_file": "prompts/gmail_check.md",
+      "wait_for_complete": true,
+      "timeout_seconds": 360
+    },
+    {
+      "id": "kimi_summary",
+      "agent": "kimi.com",
+      "prompt": [
+        "Read the attached article.",
+        "Produce a 3-bullet summary in English."
+      ],
+      "timeout_seconds": 180
+    }
+  ]
+}
+```
+
+---
+
+## 11. Test Suite After Refactor
+
+The test suite grew from 80 to 115 tests and now covers:
+
+- `tests/test_agents.py` — Agent registry, Grok agent, Kimi agent, Protocol fakes.
+- `tests/test_prompt_resolver.py` — String, list, and file prompt resolution.
+- Updated `tests/test_config_validator.py` — Agent validation and prompt forms.
+- Updated `tests/test_task_executor.py` — Agent-injected execution.
+- Updated `tests/test_login_validator.py` — Agent-delegated login checks.
+
+Run:
+
+```bash
+cd /Users/martin/dsd_890_be/grokauto
+source .venv/bin/activate
+pytest -v
+```
+
+All 115 tests pass.
+
+---
+
+## 12. Backward Compatibility
+
+- Config files without `agent` default to `grok.com`.
+- Existing single-string `prompt` fields work unchanged.
+- Existing `grok.com` selectors and login behaviour were moved, not modified.
+- No changes to CLI arguments, output file naming, or cron format.