openclaw · steipete · Aug 3, 2026 · Aug 3, 2026
diff --git a/README.md b/README.md
@@ -1,105 +1,82 @@
-# CrawlBar
+# CrawlBar 🕷️ — One menu bar for your local crawlers
 
-![CrawlBar banner](docs/assets/readme-banner.jpg)
-
-CrawlBar is a macOS menu bar control plane for local-first `*crawl` apps.
+[![CI](https://img.shields.io/github/actions/workflow/status/openclaw/crawlbar/ci.yml?branch=main&style=flat-square&label=ci)](https://github.com/openclaw/crawlbar/actions/workflows/ci.yml)
+[![GitHub release](https://img.shields.io/github/v/release/openclaw/crawlbar?style=flat-square)](https://github.com/openclaw/crawlbar/releases/latest)
+[![macOS 14+](https://img.shields.io/badge/macOS-14%2B-000000?style=flat-square&logo=apple&logoColor=white)](https://www.apple.com/macos/)
+[![License](https://img.shields.io/github/license/openclaw/crawlbar?style=flat-square)](LICENSE)
+[![Homebrew](https://img.shields.io/badge/Homebrew-openclaw%2Ftap-FBB040?style=flat-square&logo=homebrew&logoColor=black)](https://github.com/openclaw/homebrew-tap)
 
-It discovers crawler CLIs, reads metadata manifests, shows status/freshness/counts, runs refresh and doctor actions, writes redacted job logs, and keeps per-app config in `~/.crawlbar/config.json`.
+![CrawlBar banner](docs/assets/readme-banner.jpg)
 
-For `gitcrawl` and `discrawl`, CrawlBar also understands Cloudflare-backed remote archives: `remote status`, `remote archives`, and compressed SQLite `cloud publish` actions appear in the app when the installed CLI exposes them.
+CrawlBar is a native macOS menu bar app and CLI for operating local source crawlers. It discovers crawler manifests, shows status and freshness, runs supported actions, and keeps local configuration and redacted action logs in one place.
 
-## Apps
+![CrawlBar settings showing discovered local crawlers](docs/assets/crawlbar-settings.png)
 
-Built-in manifests ship for:
+## Install
 
-- `gitcrawl`
-- `slacrawl`
-- `discrawl`
-- `telecrawl`
-- `imsgcrawl`
-- `photoscrawl` (coming soon)
-- `weicrawl`
-- `notcrawl`
-- `gogcli` through the `gog` executable
-- `wacli`
-- `birdclaw`
-- `graincrawl`
+Install the notarized app and its `crawlbar` CLI with Homebrew:
 
-`gogcli`, `wacli`, and `birdclaw` can run either locally or over SSH from
-their CrawlBar settings. This covers setups where the real archive or account
-tooling lives on a server.
+```sh
+brew install openclaw/tap/crawlbar
+```
 
-Future crawler repos can drop a manifest JSON file into `~/.crawlbar/apps/*.json` to appear without a CrawlBar code change.
+CrawlBar requires macOS 14 or newer. You can also download the notarized universal build from [GitHub Releases](https://github.com/openclaw/crawlbar/releases/latest).
 
-## Build
+## Quick start
 
 ```sh
-swift build
-swift run crawlbar-selftest
-swift run crawlbarctl apps --json
+open "$(brew --prefix crawlbar)/CrawlBar.app"
+crawlbar apps
+crawlbar config validate
 ```
 
-## CLI
+The app appears in the menu bar. Open its menu to inspect crawler state, refresh all sources, or open Settings for per-crawler actions and configuration.
 
-```sh
-crawlbar apps [--json]
-crawlbar metadata [--app gitcrawl] [--json]
-crawlbar status [--app all] [--json]
-crawlbar query --app all -- manifest
-crawlbar query --app slacrawl -- 'select count(*) from messages;'
-crawlbar doctor --app discrawl [--json]
-crawlbar refresh --app slacrawl [--json]
-crawlbar action desktop-cache-import --app discrawl [--json]
-crawlbar action cloud-publish --app discrawl [--json]
-crawlbar action remote-status --app gitcrawl [--json]
-crawlbar logs [--json]
-crawlbar config path|validate|init
-crawlbar dev register --app gitcrawl --binary /path/to/gitcrawl [--json]
-crawlbar dev unregister --app gitcrawl [--json]
-crawlbar dev list [--json]
-```
+## Crawler discovery
 
-During development the SwiftPM product is `crawlbarctl` to avoid colliding with
-the `CrawlBar` app binary on case-insensitive macOS filesystems. The install
-script places it on PATH as `crawlbar`.
+CrawlBar ships manifests for GitHub, Slack, Discord, Telegram, iMessage, Apple Photos, WeChat, Notion, Google, WhatsApp, X, and Granola crawler tools. Each manifest describes the executable, supported actions, configuration fields, paths, and privacy metadata; the UI only exposes capabilities that the manifest provides.
 
-## Install
+Additional crawlers can install a manifest at `~/.crawlbar/apps/*.json` without changing CrawlBar. The [control protocol](docs/control-protocol.md) documents the manifest schema, status normalization, command contract, remote execution, and privacy rules.
 
-```sh
-brew install openclaw/tap/crawlbar
-```
+## Actions and remote crawlers
 
-## Config
+Common crawler capabilities include status, refresh, doctor, search, and local archive actions. GitHub and Discord crawlers can also expose Cloudflare remote-archive status and compressed SQLite publish actions when their installed CLIs advertise those capabilities.
 
-Main config lives at:
+Google, WhatsApp, and X crawler commands can run locally or over SSH. Remote mode resolves the crawler binary on the configured host while keeping the same status and action model in CrawlBar.
 
-```text
-~/.crawlbar/config.json
-```
+## CLI and automation
 
-Action logs live at:
+The installed app includes a `crawlbar` CLI. SwiftPM names the development executable `crawlbarctl` so it does not collide with the `CrawlBar` app binary on case-insensitive filesystems.
 
-```text
-~/.crawlbar/logs
-```
+| Task | Command |
+|---|---|
+| List discovered crawlers | `crawlbar apps [--json]` |
+| Read normalized status | `crawlbar status --app <id\|all> [--json]` |
+| Run a crawler operation | `crawlbar doctor\|refresh --app <id> [--json]` |
+| Inspect or validate config | `crawlbar config path\|validate` |
+| Show all commands | `crawlbar --help` |
+
+See the [CLI reference](docs/cli.md) for queries, actions, backups, configuration, development binary overrides, and JSON output.
 
-Both are written with private file permissions. Command output is redacted before it is returned to the UI or persisted as an action log.
+## Configuration and privacy
 
-## Package The App
+CrawlBar stores its main configuration at `~/.crawlbar/config.json`, external manifests under `~/.crawlbar/apps`, and action logs under `~/.crawlbar/logs`. Configuration and logs use private file permissions.
+
+Crawler command output is redacted before it reaches the UI, CLI response, or action log. Source crawlers continue to own their archives, authentication, parsing, search, and source-specific privacy policy.
+
+## Development
+
+The package requires Swift 6.1 or newer.
 
 ```sh
+swift build
+swift run crawlbar-selftest
 Scripts/package_app.sh
+codesign --verify --deep --strict --verbose=2 dist/CrawlBar.app
 ```
 
-The packaged `.app` is written to `dist/CrawlBar.app`.
-
-Local and CI packages use ad-hoc signing and do not need release credentials.
-Official release packages fail closed unless they use the OpenClaw Foundation
-Developer ID identity. Runtime keychain and notarization profile locators belong
-in the ignored `.mac-release.local.env`, never in committed configuration.
+The packaged app is written to `dist/CrawlBar.app`. See the [development and packaging guide](docs/development.md) for signing and release-build details. Architecture and UI conventions live in the [quality rubric](docs/quality-rubric.md) and [UI rules](docs/ui-rules.md).
 
-## Releases
+## License
 
-Release notes are kept in [CHANGELOG.md](CHANGELOG.md). The packaged app bundle
-version is read from `version.env`. Official artifacts are universal, hardened,
-notarized, stapled, and verified by `Scripts/package_release.sh`.
+MIT. See [LICENSE](LICENSE).
diff --git a/docs/assets/crawlbar-settings.png b/docs/assets/crawlbar-settings.png
diff --git a/docs/cli.md b/docs/cli.md
@@ -0,0 +1,87 @@
+# CrawlBar CLI reference
+
+The packaged app installs its helper as `crawlbar`. In a source checkout, use `swift run crawlbarctl` in place of `crawlbar`; the different SwiftPM name avoids a collision with the `CrawlBar` app binary on case-insensitive macOS filesystems.
+
+Run `crawlbar --help` for the authoritative command list provided by your installed version.
+
+## Discovery and status
+
+```text
+crawlbar apps [--json]
+crawlbar metadata [--app <id>] [--json] [--diagnostics]
+crawlbar status [--app <id|all>] [--json]
+crawlbar logs [--json]
+```
+
+`apps` reports every discovered manifest and whether its executable is available. `metadata` prints the normalized manifest contract. `status` runs the read-only status command exposed by one crawler or all crawlers. `logs` lists recent redacted action-log paths.
+
+Built-in manifests use these IDs and executable names:
+
+| Source | CrawlBar ID | Executable |
+|---|---|---|
+| GitHub | `gitcrawl` | `gitcrawl` |
+| Slack | `slacrawl` | `slacrawl` |
+| Discord | `discrawl` | `discrawl` |
+| Telegram | `telecrawl` | `telecrawl` |
+| iMessage | `imsgcrawl` | `imsgcrawl` |
+| Apple Photos | `photoscrawl` | `photoscrawl` |
+| WeChat | `weicrawl` | `weicrawl` |
+| Notion | `notcrawl` | `notcrawl` |
+| Google | `gogcli` | `gog` |
+| WhatsApp | `wacli` | `wacli` |
+| X | `birdclaw` | `bird` |
+| Granola | `graincrawl` | `graincrawl` |
+
+## Crawler operations
+
+```text
+crawlbar doctor --app <id> [--json]
+crawlbar refresh --app <id> [--json]
+crawlbar query --app <id|all> [--json] -- <text-or-sql>
+crawlbar action <action-id> --app <id> [--json]
+crawlbar install --app <id> [--json]
+crawlbar backup --app <id> [--json]
+crawlbar folder --app <id> [--json]
+```
+
+Available actions come from each crawler manifest. `query` passes every argument after `--` to the crawler without shell expansion. `backup` copies the crawler's reported primary database into CrawlBar's backup location, and `folder` opens the reported data folder.
+
+Examples:
+
+```sh
+crawlbar status --app all --json
+crawlbar query --app slacrawl -- 'select count(*) from messages;'
+crawlbar action cloud-publish --app discrawl --json
+```
+
+Commands that contact a source, mutate an archive, publish data, or use SSH require the corresponding crawler's configuration and credentials.
+
+## Configuration
+
+```text
+crawlbar config path
+crawlbar config validate
+crawlbar config init
+crawlbar config get --app <id> [--key <id>] [--json] [--reveal]
+crawlbar config set --app <id> --key <id> --value <value> [--json]
+```
+
+The main configuration file is `~/.crawlbar/config.json`. External manifests live under `~/.crawlbar/apps`, and redacted action logs live under `~/.crawlbar/logs`.
+
+Secret configuration values are hidden by default. Use `--reveal` only in a private terminal when the raw value is explicitly required.
+
+## Development binary overrides
+
+Register a locally built crawler without replacing a Homebrew or system binary:
+
+```text
+crawlbar dev register --app <id> --binary <path> [--json]
+crawlbar dev unregister --app <id> [--json]
+crawlbar dev list [--json]
+```
+
+Registration enables the crawler and records the explicit binary path in CrawlBar's configuration. Unregistering removes that override but leaves the crawler's other settings intact.
+
+## JSON output
+
+Commands with `--json` emit machine-readable output and keep diagnostics on standard error. CrawlBar redacts known secret values from crawler command output before returning or persisting it.
diff --git a/docs/development.md b/docs/development.md
@@ -0,0 +1,33 @@
+# Development and packaging
+
+`Package.swift` requires Swift tools 6.1 and targets macOS 14 or newer.
+
+## Build and test
+
+```sh
+swift build
+swift run crawlbar-selftest
+swift run crawlbarctl apps --json
+swift run crawlbarctl metadata --json
+swift run crawlbarctl config validate
+```
+
+SwiftPM names the development CLI `crawlbarctl` to avoid colliding with the `CrawlBar` app binary on case-insensitive macOS filesystems. Packaged and Homebrew installations expose the helper as `crawlbar`.
+
+## Package the app
+
+```sh
+Scripts/package_app.sh
+codesign --verify --deep --strict --verbose=2 dist/CrawlBar.app
+dist/CrawlBar.app/Contents/Helpers/crawlbar config validate
+```
+
+The packaging script writes `dist/CrawlBar.app`. Local and CI packages use ad-hoc signing and do not need release credentials.
+
+## Official artifacts
+
+The packaged app bundle version comes from `version.env`. Release notes live in [CHANGELOG.md](../CHANGELOG.md).
+
+`Scripts/package_release.sh` builds the official universal, hardened app, then notarizes, staples, and verifies it. Official packaging fails closed unless it uses the OpenClaw Foundation Developer ID identity. Runtime keychain and notarization-profile locators belong in the ignored `.mac-release.local.env`, never in committed configuration.
+
+`Scripts/verify_release.sh` checks the completed release artifact. Publishing tags or release artifacts is a separate maintainer action and is not part of local packaging.