Skip to content

Add Hermes model usage to the Agents panel - #13505

Closed
sccashma-hermes wants to merge 2 commits into
omacom:quattrofrom
sccashma-hermes:fix/hermes-model-usage
Closed

sccashma-hermes wants to merge 2 commits into
omacom:quattrofrom
sccashma-hermes:fix/hermes-model-usage

Conversation

@sccashma-hermes

Copy link
Copy Markdown

Summary

Hermes Agent sessions currently do not appear in Omarchy's Agents usage panel, so Hermes users only see usage from other collectors such as Codex. This adds a Hermes collector that exposes the existing local session totals and preserves each model's token usage, including models such as gpt-6-sol.

Changes

  • Add bin/omarchy-agent-usage-hermes, using read-only access to ~/.hermes/state.db.
  • Emit the standard Agents panel record with daily, recent, all-time, and per-model totals.
  • Support atomic --write output for manual or timer-driven refreshes.
  • Add a regression test covering multiple Hermes models and the updater's record shape.

Validation

  • python3 -m py_compile bin/omarchy-agent-usage-hermes
  • ./test/shell.d/agent-usage-hermes-test.sh
  • ./test/shell.d/agent-usage-update-test.sh
  • ./test/shell.d/agent-usage-fireworks-scanner-test.sh
  • ./test/shell

The focused tests pass. The full shell suite reached a pre-existing passwordless-grant-lifecycle-test.sh failure (set OMARCHY_PKGS_PATH to the companion package checkout) and then timed out during later tests.

Related: #9871

@llstrk

llstrk commented Sep 29, 2026

Copy link
Copy Markdown

Automated AI review

Community review: Independent automated community review, unaffiliated with the Omarchy team, intended to help prepare PRs for their review.

Outcome: The Hermes daily token chart shows prompt counts.

Daily token series holds prompt counts

recentDays[].messageCount gets prompt counts, but Panel.qml draws it as TOKENS BY DAY, and the other collectors put tokens there. Impact: a day with one prompt and 104 tokens shows 1.

Suggested change: Sum the four token columns per date(s.started_at, 'unixepoch', 'localtime'), as todayTokensByModel does (long chats still count on their start day; state.db has no per-day tokens).

Reproducer (fails at 5689583)

Save as test/shell.d/repro-recentdays-test.sh in a checkout of the PR head and run bash test/shell.d/repro-recentdays-test.sh.

#!/bin/bash

# Reproducer for PR #13505: the Agents panel draws recentDays[].messageCount
# as "TOKENS BY DAY" (Panel.qml dayTooltip/DayRow; the claude, codex and
# fireworks collectors all put token totals there). With one session today
# that used 104 tokens, today's bar should read 104, not the prompt count.

source "$(dirname "$0")/base-test.sh"

require_command jq
require_command python3

TEST_HOME=$(mktemp -d)
trap 'rm -rf "$TEST_HOME"' EXIT

mkdir -p "$TEST_HOME/.hermes"
HOME="$TEST_HOME" python3 - <<'PY'
import os
import sqlite3
import time
from pathlib import Path

path = Path(os.environ["HOME"]) / ".hermes" / "state.db"
conn = sqlite3.connect(path)
conn.executescript(
    """
    CREATE TABLE sessions (id TEXT PRIMARY KEY, started_at REAL NOT NULL);
    CREATE TABLE messages (id INTEGER PRIMARY KEY, session_id TEXT NOT NULL, role TEXT NOT NULL, timestamp REAL NOT NULL);
    CREATE TABLE session_model_usage (
      session_id TEXT NOT NULL, model TEXT NOT NULL,
      input_tokens INTEGER NOT NULL DEFAULT 0, output_tokens INTEGER NOT NULL DEFAULT 0,
      cache_read_tokens INTEGER NOT NULL DEFAULT 0, cache_write_tokens INTEGER NOT NULL DEFAULT 0,
      PRIMARY KEY (session_id, model)
    );
    """
)
now = time.time()
conn.execute("INSERT INTO sessions VALUES (?, ?)", ("session-1", now))
conn.executemany(
    "INSERT INTO messages VALUES (?, ?, ?, ?)",
    [(1, "session-1", "user", now), (2, "session-1", "assistant", now)],
)
conn.execute("INSERT INTO session_model_usage VALUES (?, ?, ?, ?, ?, ?)", ("session-1", "gpt-6-sol", 11, 21, 31, 41))
conn.commit()
conn.close()
PY

record=$(HOME="$TEST_HOME" "$ROOT/bin/omarchy-agent-usage-hermes") ||
  fail "Hermes collector exits successfully"
pass "Hermes collector exits successfully"

printf '%s\n' "$record" | jq -e '.recentDays[-1].messageCount == .todayTotalTokens' >/dev/null ||
  fail "Hermes collector puts today's tokens in the tokens-by-day series" \
    "$(printf '%s\n' "$record" | jq -c '{today: .recentDays[-1], todayTotalTokens, todayPrompts}')"
pass "Hermes collector puts today's tokens in the tokens-by-day series"

Actual output at the PR head (5689583), stdout then stderr:

ok - Hermes collector exits successfully
{"today":{"date":"2026-09-29","messageCount":1},"todayTotalTokens":104,"todayPrompts":1}
not ok - Hermes collector puts today's tokens in the tokens-by-day series

HERMES_HOME and profiles are ignored

Only ~/.hermes/state.db is read; Hermes also uses $HERMES_HOME and ~/.hermes/profiles/<name>/, as omarchy-theme-set-hermes handles. Impact: their usage is missing: no tab, or only what ~/.hermes/state.db holds.

Suggested change: Read $HERMES_HOME/state.db, falling back to ~/.hermes, and decide how profiles count.

Reproducer (fails at 5689583)

Save as test/shell.d/repro-hermes-home-test.sh in a checkout of the PR head and run bash test/shell.d/repro-hermes-home-test.sh.

#!/bin/bash

# Reproducer for PR #13505: Hermes keeps its state in $HERMES_HOME/state.db
# (hermes_constants.get_hermes_home: HERMES_HOME, then ~/.hermes), and
# omarchy-theme-set-hermes / omarchy-install-hermes-cli already honor
# HERMES_HOME. The collector only ever opens ~/.hermes/state.db.

source "$(dirname "$0")/base-test.sh"

require_command jq
require_command python3

TEST_HOME=$(mktemp -d)
trap 'rm -rf "$TEST_HOME"' EXIT

HERMES_HOME="$TEST_HOME/hermes-data"
mkdir -p "$HERMES_HOME"
HERMES_HOME="$HERMES_HOME" python3 - <<'PY'
import os
import sqlite3
import time
from pathlib import Path

path = Path(os.environ["HERMES_HOME"]) / "state.db"
conn = sqlite3.connect(path)
conn.executescript(
    """
    CREATE TABLE sessions (id TEXT PRIMARY KEY, started_at REAL NOT NULL);
    CREATE TABLE messages (id INTEGER PRIMARY KEY, session_id TEXT NOT NULL, role TEXT NOT NULL, timestamp REAL NOT NULL);
    CREATE TABLE session_model_usage (
      session_id TEXT NOT NULL, model TEXT NOT NULL,
      input_tokens INTEGER NOT NULL DEFAULT 0, output_tokens INTEGER NOT NULL DEFAULT 0,
      cache_read_tokens INTEGER NOT NULL DEFAULT 0, cache_write_tokens INTEGER NOT NULL DEFAULT 0,
      PRIMARY KEY (session_id, model)
    );
    """
)
now = time.time()
conn.execute("INSERT INTO sessions VALUES (?, ?)", ("session-1", now))
conn.execute("INSERT INTO messages VALUES (?, ?, ?, ?)", (1, "session-1", "user", now))
conn.execute("INSERT INTO session_model_usage VALUES (?, ?, ?, ?, ?, ?)", ("session-1", "gpt-6-sol", 11, 21, 31, 41))
conn.commit()
conn.close()
PY

record=$(HOME="$TEST_HOME" HERMES_HOME="$HERMES_HOME" "$ROOT/bin/omarchy-agent-usage-hermes") ||
  fail "Hermes collector exits successfully"
pass "Hermes collector exits successfully"

printf '%s\n' "$record" | jq -e '.totalSessions == 1 and .usageStatusText == ""' >/dev/null ||
  fail "Hermes collector reads the state database under HERMES_HOME" \
    "$(printf '%s\n' "$record" | jq -c '{totalSessions, usageStatusText}')"
pass "Hermes collector reads the state database under HERMES_HOME"

Actual output at the PR head (5689583), stdout then stderr:

ok - Hermes collector exits successfully
{"totalSessions":0,"usageStatusText":"Hermes state not found"}
not ok - Hermes collector reads the state database under HERMES_HOME
More findings

A failed read replaces the last good record

On a sqlite3.Error the collector prints a record marked "Hermes state unreadable" (zeroed, or partial if earlier queries succeeded) and exits 0, and omarchy-agent-usage-update keeps the previous file only on a non-zero exit or invalid output. Impact: a corrupt or unreadable file, or (in rollback-journal mode) a lock held past the 5 s busy timeout, overwrites hermes.json with a zero record, and the panel drops the Hermes tab until the next good run. A pre-v20 schema without session_model_usage gets a partial record: the tab shows "Hermes state unreadable" with an empty status card and no chart.

Suggested change: On a read error (not on a missing database), exit non-zero without printing, so the updater keeps the last good record; a persistent failure then shows stale numbers, so a status text for that case may still be wanted.

Save as test/shell.d/repro-failed-read-overwrites-test.sh in a checkout of the PR head and run bash test/shell.d/repro-failed-read-overwrites-test.sh.

#!/bin/bash

# Reproducer for PR #13505: when the Hermes collector cannot read state.db it
# still exits 0 with a zeroed record, so omarchy-agent-usage-update replaces
# the last good hermes.json (the updater only keeps the old file when a
# collector exits non-zero or prints invalid JSON).

source "$(dirname "$0")/base-test.sh"

require_command jq
require_command python3

TEST_HOME=$(mktemp -d)
trap 'rm -rf "$TEST_HOME"' EXIT

mkdir -p "$TEST_HOME/.hermes"
HOME="$TEST_HOME" python3 - <<'PY'
import os
import sqlite3
import time
from pathlib import Path

path = Path(os.environ["HOME"]) / ".hermes" / "state.db"
conn = sqlite3.connect(path)
conn.executescript(
    """
    CREATE TABLE sessions (id TEXT PRIMARY KEY, started_at REAL NOT NULL);
    CREATE TABLE messages (id INTEGER PRIMARY KEY, session_id TEXT NOT NULL, role TEXT NOT NULL, timestamp REAL NOT NULL);
    CREATE TABLE session_model_usage (
      session_id TEXT NOT NULL, model TEXT NOT NULL,
      input_tokens INTEGER NOT NULL DEFAULT 0, output_tokens INTEGER NOT NULL DEFAULT 0,
      cache_read_tokens INTEGER NOT NULL DEFAULT 0, cache_write_tokens INTEGER NOT NULL DEFAULT 0,
      PRIMARY KEY (session_id, model)
    );
    """
)
now = time.time()
conn.execute("INSERT INTO sessions VALUES (?, ?)", ("session-1", now))
conn.execute("INSERT INTO messages VALUES (?, ?, ?, ?)", (1, "session-1", "user", now))
conn.execute("INSERT INTO session_model_usage VALUES (?, ?, ?, ?, ?, ?)", ("session-1", "gpt-6-sol", 11, 21, 31, 41))
conn.commit()
conn.close()
PY

usage_dir="$TEST_HOME/.local/state/omarchy/agents/usage"

HOME="$TEST_HOME" OMARCHY_PATH="$ROOT" XDG_STATE_HOME="" "$ROOT/bin/omarchy-agent-usage-update" hermes ||
  fail "update writes the Hermes record"
[[ $(jq -r '.totalSessions' "$usage_dir/hermes.json") == "1" ]] ||
  fail "update writes the Hermes record"
pass "update writes the Hermes record"

# The database becomes unreadable; a corrupt file stands in for any sqlite3.Error.
printf 'not an sqlite database\n' >"$TEST_HOME/.hermes/state.db"

HOME="$TEST_HOME" OMARCHY_PATH="$ROOT" XDG_STATE_HOME="" "$ROOT/bin/omarchy-agent-usage-update" hermes 2>/dev/null
[[ $(jq -r '.totalSessions' "$usage_dir/hermes.json") == "1" ]] ||
  fail "a failed Hermes read keeps the last good record" \
    "$(jq -c '{totalSessions, totalPrompts, todayTotalTokens, hasLocalStats, usageStatusText, authHelpText}' "$usage_dir/hermes.json")"
pass "a failed Hermes read keeps the last good record"

Actual output at the PR head (5689583), stdout then stderr:

ok - update writes the Hermes record
{"totalSessions":0,"totalPrompts":0,"todayTotalTokens":0,"hasLocalStats":true,"usageStatusText":"Hermes state unreadable","authHelpText":""}
not ok - a failed Hermes read keeps the last good record

The --write rationale does not apply

The docstring says --write exists because the updater "only discovers packaged collectors", but this collector is in bin/, and omarchy-agent-usage-update globs $OMARCHY_PATH/bin/omarchy-agent-usage-* and already writes hermes.json (a PR-style test of that passes at 5689583). The PR description's "the updater's record shape" is also not exercised: the test never runs the updater.

Suggested change: Drop --write and that sentence (or keep the flag without the rationale), and declare [--force] [--limits-only] like the other collectors.

Details

Verified:

  • Table and column names, epoch-second timestamps and the four token buckets match Hermes's current state.db schema; the token sum equals Hermes's own total_tokens.
  • Day bounds, today counts and activeDates are correct in six time zones, including a 23-hour DST day and messages 0.5 s either side of local midnight.
  • A missing database gives "Hermes state not found" with exit 0; a WAL database is read while a writer holds a transaction.
  • --write is atomic and leaves no temporary files; omarchy commands --check, the bin-style test and agent-usage-update-test.sh pass with the new files.

Optional notes:

  • todaySessions and today's tokens count sessions started today, while todayPrompts uses message time, so a gateway chat started yesterday with prompts today shows prompts but 0 sessions and 0 tokens today. Counting sessions with a message today (for example DISTINCT session_id over today's assistant messages) would be closer to the other collectors.
  • The SQLite URI is built by string interpolation, so a # in HOME makes a valid database read as "unreadable"; DB_PATH.as_uri() + "?mode=ro" avoids it.
  • The new test never checks recentDays, so the first finding passes it; adding the reproducer's assertion would catch it.

Related PRs: #9871's bin/omarchy-agent-usage-hermes is the same blob, and this PR's first commit is that PR's author's commit on a newer base. #6647 and #10400 each add a different bin/omarchy-agent-usage-hermes, so whichever lands second has an add/add conflict.


Review information

Test scope: Source and sandbox tests with synthetic Hermes databases at 5689583d; no real Hermes or running panel.

AI process: Opus 5.5 Medium coordination and synthesis, Opus 5.5 Xhigh technical review and final fact check, GPT 6 Sol Xhigh search for related issues, Opus 5.5 Medium editorial check.

Opt out: To stop receiving these reviews, reply to this comment saying so.

@dhh

dhh commented Oct 2, 2026

Copy link
Copy Markdown
Contributor

Thank you! Brought in with your authorship in #14039, adapted to read HERMES_HOME and every named profile, with tokens per day in recentDays and the --write mode dropped.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants