From d7b5a428329aff3d7af3af77f942b3957922c33a Mon Sep 17 00:00:00 2001 From: Rick Date: Fri, 12 Jun 2026 17:37:52 -0500 Subject: [PATCH] docs: clarify token analytics install workflow --- .../hermes-token-analytics-install-runbook.md | 59 +++++++++++++++++++ docs/hermes-token-analytics-plugin.md | 55 +++++++++++++++-- plugins/hermes-token-analytics/README.md | 21 +++++++ 3 files changed, 130 insertions(+), 5 deletions(-) diff --git a/docs/hermes-token-analytics-install-runbook.md b/docs/hermes-token-analytics-install-runbook.md index 25348d1..8745e79 100644 --- a/docs/hermes-token-analytics-install-runbook.md +++ b/docs/hermes-token-analytics-install-runbook.md @@ -43,6 +43,25 @@ Before you start, confirm: ## Step 1. Install the plugin into the Hermes checkout +> [!WARNING] +> This repository is a monorepo. The Hermes plugin is **not** at the repo root. +> Do **not** install the repo root directly with `hermes plugins install `. +> The actual plugin source lives at `plugins/hermes-token-analytics/`. + +### Install target options + +There are two valid install targets. They are not the same thing. + +1. **Bundled checkout plugin path**, for development inside a Hermes repo checkout + - target root: `~/.hermes/hermes-agent` + - resulting plugin path: `~/.hermes/hermes-agent/plugins/hermes-token-analytics/` +2. **User plugin path**, preferred for a normal operator install + - resulting plugin path: `~/.hermes/plugins/hermes-token-analytics/` + +Preferred default: use the **user plugin path** unless you are intentionally working inside a bundled Hermes checkout. + +### Bundled checkout install + From this repo root, copy the plugin into the target Hermes checkout: ```bash @@ -67,6 +86,22 @@ What the helper does: - writes a compatibility shim at `plugins/observability/token_analytics/` - lets older `plugins.enabled` entries keep working during migration +### User plugin install + +If you are doing a normal user-level install instead of editing a Hermes checkout in place, copy this repo subdirectory into: + +```text +~/.hermes/plugins/hermes-token-analytics +``` + +Use the contents of: + +```text +plugins/hermes-token-analytics/ +``` + +The main operational point is simple: copy the plugin directory, not the monorepo root. + ## Step 2. Enable the plugin in Hermes Enable the plugin by its path-derived key: @@ -75,6 +110,12 @@ Enable the plugin by its path-derived key: hermes plugins enable hermes-token-analytics ``` +Then restart the gateway so Hermes reloads the plugin command surface: + +```bash +hermes gateway restart +``` + Then verify Hermes sees it: ```bash @@ -84,6 +125,7 @@ hermes plugins list What you want to see: - `hermes-token-analytics` listed as enabled +- the plugin commands available after the restart If the target install still references the old compatibility path, the shim also supports the legacy key: @@ -156,6 +198,10 @@ What success looks like: - `show-config` shows the resolved endpoint, workspace fields, and redacted shared secret - `sync` posts successfully to `/api/ingest/hermes-usage` +Important: a successful `hermes token-analytics sync` only proves the plugin works manually. +It does **not** mean continuous reporting is active. +Continuous reporting requires a cron job. + If `sync` fails, stop here and fix config before adding cron. ## Step 5. Create the scheduled job @@ -214,6 +260,19 @@ Confirm: - the schedule is correct - the cron-triggered run behaves the same as the manual `sync` +## Production readiness checklist + +- [ ] plugin copied from `plugins/hermes-token-analytics/` into the intended install target +- [ ] plugin enabled +- [ ] gateway restarted +- [ ] `hermes token-analytics doctor` passes +- [ ] `hermes token-analytics show-config` is correct +- [ ] `hermes token-analytics sync` succeeds +- [ ] `hermes token-analytics install-cron-wrapper` run +- [ ] cron job created +- [ ] `hermes cron list --all` shows `token-analytics-sync` +- [ ] one cron-triggered run verified + ## Normal operating procedure Use this sequence for changes or repairs: diff --git a/docs/hermes-token-analytics-plugin.md b/docs/hermes-token-analytics-plugin.md index 989223e..550eb64 100644 --- a/docs/hermes-token-analytics-plugin.md +++ b/docs/hermes-token-analytics-plugin.md @@ -4,20 +4,27 @@ This integration is now a **Hermes-native Python plugin**, not a Node or bash sidecar. +> [!WARNING] +> This repository is a monorepo. The Hermes plugin is **not** at the repo root. +> Do **not** install the repo root directly with `hermes plugins install `. +> The actual plugin source lives at `plugins/hermes-token-analytics/`. + Operator setup is: -1. Set the Cloudflare Worker secret `HERMES_TOKEN_ANALYTICS_SHARED_SECRET`. -2. Export the plugin env vars shown below. -3. Validate the install with: +1. Copy `plugins/hermes-token-analytics/` into the correct install target. +2. Enable the plugin and restart the Hermes gateway. +3. Set the Cloudflare Worker secret `HERMES_TOKEN_ANALYTICS_SHARED_SECRET`. +4. Export the plugin env vars shown below. +5. Validate the install with: ```bash hermes token-analytics doctor hermes token-analytics show-config ``` -4. Push one manual sync: +6. Push one manual sync: ```bash hermes token-analytics sync ``` -5. Let **Hermes cron own the schedule** by creating a single cron job that runs the sync command on a cadence such as every 15 minutes. That one job covers both rollups and heartbeat freshness. +7. Let **Hermes cron own the schedule** by creating a single cron job that runs the sync command on a cadence such as every 15 minutes. That one job covers both rollups and heartbeat freshness. If you just need the exact install sequence, use: @@ -46,6 +53,18 @@ This repo is the dashboard and ingest target. The operator-facing runtime is the For local Hermes checkout installs, the helper now copies the plugin into `plugins/hermes-token-analytics/` and also writes a legacy compatibility shim at `plugins/observability/token_analytics/` so existing enabled-plugin configs do not break mid-upgrade. +## Common operational failure modes + +When operators report that token analytics is "not working," the failure is often in the workflow, not the plugin logic. + +The main causes are: + +- using the monorepo root instead of `plugins/hermes-token-analytics/` +- enabling the plugin without restarting the Hermes gateway +- proving only that a manual sync works, without creating a cron job for continuous reporting + +Manual sync working is useful, but it is not the same thing as production readiness. + ## Architecture and ownership ### Hermes-native plugin, not a sidecar @@ -72,6 +91,28 @@ UI note: the dashboard may refer to these imported workspaces as **Agents** in u If a sync is running at the wrong time, fix Hermes cron. If a sync is reading the wrong DB or posting to the wrong workspace, fix plugin config. +### Install target choices + +There are two supported install targets: + +1. **Bundled checkout path** + - use this when you are editing a Hermes repo checkout directly + - plugin lives under `~/.hermes/hermes-agent/plugins/hermes-token-analytics/` +2. **User plugin path**, preferred for normal operator installs + - plugin lives under `~/.hermes/plugins/hermes-token-analytics/` + +If both are technically available, prefer the **user plugin path** unless you specifically need the plugin to live inside a Hermes checkout for development. + +### Gateway reload requirement + +After `hermes plugins enable hermes-token-analytics`, run: + +```bash +hermes gateway restart +``` + +Without the restart, Hermes may still be running without the newly enabled plugin command surface. + ## Required Worker-side configuration The Worker route rejects unauthenticated syncs. @@ -206,6 +247,10 @@ A successful run should: 4. receive a successful response from the ingest route 5. make the imported workspace visible in the dashboard +A successful manual `hermes token-analytics sync` only proves that the plugin works in the foreground. +It does **not** prove that continuous reporting is active. +Continuous reporting requires a Hermes cron job. + ### `hermes token-analytics install-cron-wrapper` Install a thin shell wrapper for Hermes cron `no_agent` jobs. diff --git a/plugins/hermes-token-analytics/README.md b/plugins/hermes-token-analytics/README.md index d2b3b3b..e3962e6 100644 --- a/plugins/hermes-token-analytics/README.md +++ b/plugins/hermes-token-analytics/README.md @@ -2,6 +2,11 @@ This directory is the source-of-truth copy of the Hermes token analytics plugin inside the monorepo. +> [!WARNING] +> This repository is a monorepo. The Hermes plugin is **not** at the repo root. +> Do **not** install the repo root directly with `hermes plugins install `. +> The actual plugin source lives at `plugins/hermes-token-analytics/`. + ## What lives here - `plugins/hermes-token-analytics/` — the plugin files as they should exist inside a Hermes checkout @@ -30,6 +35,13 @@ python3 -m pytest plugins/hermes-token-analytics/tests -q ## Local install helper +There are two install patterns: + +1. **Bundled checkout install**, useful when you are modifying a Hermes repo checkout directly +2. **User plugin install**, preferred for normal operator usage under `~/.hermes/plugins/hermes-token-analytics` + +Bundled checkout helper: + ```bash plugins/hermes-token-analytics/scripts/install-local-plugin.sh /path/to/hermes-agent ``` @@ -39,3 +51,12 @@ Default target if no argument is passed: - `~/.hermes/hermes-agent` The install helper also writes a small compatibility shim at `plugins/observability/token_analytics/` inside the target Hermes checkout so older `plugins.enabled` entries keep working during migration. + +After enabling the plugin, restart the gateway: + +```bash +hermes plugins enable hermes-token-analytics +hermes gateway restart +``` + +A successful manual `hermes token-analytics sync` proves the plugin works manually. It does **not** prove continuous reporting is active. For that, create the cron job described in `docs/hermes-token-analytics-install-runbook.md`.