Skip to content

Add a Local tab for self-hosted model usage - #12034

Open
Ronin11 wants to merge 2 commits into
omacom:quattrofrom
Ronin11:local-agent-tab
Open

Ronin11 wants to merge 2 commits into
omacom:quattrofrom
Ronin11:local-agent-tab

Conversation

@Ronin11

@Ronin11 Ronin11 commented Sep 16, 2026

Copy link
Copy Markdown
Contributor

Implements #12033.

Stacked on #12032. This branch contains that fix plus one new commit; GitHub will show both until it merges. The only new work here is 792b50f, and the diff worth reviewing is dab5dad..local-agent-tab.

The subscription collectors each answer "what did this cost me", so every turn served by something else is dropped on the floor. #12032 makes the Codex collector drop them correctly; this gives them somewhere to go.

A local collector counts exactly what the subscription collectors skip, across the same three sources they already read: native Codex rollouts, pi/omp sessions, and the opencode database.

What counts as local

The endpoint, not a hardcoded runtime list. A URL qualifies when it is loopback, private, link-local, or a bare/LAN hostname — so a model served by another machine on your network counts the same as one on this one:

provider endpoint Local Codex
homelab http://server.home:11434/v1 yes ·
gpu http://10.0.0.50:8000/v1 yes ·
lanbox http://gpu-box:8080 yes ·
ollama no URL; well-known runtime yes ·
openrouter https://openrouter.ai/api/v1 · ·
openai the subscription · yes

Ids with no URL to resolve fall back to the well-known runtimes (Ollama, LM Studio, llama.cpp, vLLM, LocalAI, …), and a custom gateway can be named in ~/.config/omarchy/agents/local.json.

The property that makes the pair safe

With the fix in place the two collectors partition the sessions directory — no turn lost between them, none counted twice. On real data:

codex today (before #12032) : 52,617
codex today (after)         : 31,580   gpt-6-astra
local today                 : 21,037   qwen3-coder:30b
                              ------
                              52,617   ✓

That is asserted in the test, not merely observed once.

Shape

  • No limits, no balance — nothing meters self-hosted work, so the tab draws token history alone.
  • Cached behind the same versioned envelope the other collectors use, with the same reuse windows, the same lock, and the same rules: corrupt / wrong-date / future-dated / unwritable degrade to a direct scan, and an interrupted scan is never cached.
  • The opencode provider filter is pushed into SQL rather than run over every historical message in Python, since the candidate id set is known before the query runs.
  • Cross-device aggregation needed no changes: aggregateSnapshots() builds provider accumulators on demand, so a homelab's numbers travel like any other agent's. Untested, as I have only one machine.
  • The tab self-hides like every other agent, so this costs nothing for people who never run a local model.

Tests

test/shell.d/agent-usage-local-scanner-test.sh, 11 assertions: LAN and loopback endpoints, configured ids, the no-limits shape, the partition property across both collectors, opencode filtering (paid providers, user messages, reasoning tokens, malformed rows), and the full cache lifecycle.

Whole agent-usage suite is 77 assertions across 6 files, green. Full test/shell is 236/242 files, with the same 6 environmental failures that reproduce on a clean quattro.

Deliberate omissions

  • A third-party API that is neither the subscription nor your own hardware stays uncounted. A fourth bucket is arguable; I did not want to invent one unasked.
  • No assets/local.svg — the bar glyph stands in. Happy to add a mark if you want one, but it seemed wrong to invent branding in a drive-by PR.

Entirely understand if the panel is not meant to grow this way — filed #12033 first so the idea could be rejected before the code mattered.

Ronin11 and others added 2 commits September 15, 2026 16:49
Codex CLI fronts any OpenAI-compatible backend: `--oss`, or a custom
model_provider in config.toml aimed at Ollama, LM Studio, or a gateway.
Those rollouts land in the same sessions directory as real ones, and the
native scan counted them all, so a local model showed up in the agents
panel as Codex subscription usage. On one machine `qwen3-coder:30b` was
40% of the tokens on that tab.

The provider is recorded on session_meta, which the scan skipped past;
turn_context carries the model but no provider at all. Read it there and
drop the rollout when it names anything but the built-in openai backend,
matching what the pi and opencode scans already do with foreign
providers. Rollouts written before Codex recorded the field carry no
provider, so the check only fires when one is present and differs -
older history keeps counting.

Limits are unaffected; they come from the app-server RPC, not from
session files.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The subscription collectors each answer "what did this cost me", and every
turn served by something else is dropped on the floor. Run Codex CLI
against Ollama and the work simply vanishes from the panel - which is the
right answer for the Codex tab and the wrong answer overall.

Add a collector for the other side of that line. It counts exactly what
the subscription collectors skip: Codex, pi, and opencode sessions served
by an endpoint you host, whether that is this machine or a box on the
network. An endpoint qualifies when its URL is loopback, private,
link-local, or a bare/LAN hostname, so `server.home` counts the same as
`localhost`; ids with no URL to resolve fall back to the well-known
runtimes, and a custom gateway can be named in
~/.config/omarchy/agents/local.json.

The two collectors partition the sessions directory, so no turn is lost
between them and none is counted twice - asserted in the test. A
third-party API that is neither the subscription nor your own hardware
stays uncounted on purpose.

No limits and no balance: nothing meters self-hosted work, so the panel
draws the token history alone. Cross-device aggregation needs no changes;
snapshots merge by provider id, so a homelab's numbers travel like any
other agent's.

Scans are cached behind the same versioned envelope the other collectors
use, and the opencode provider filter is pushed into SQL rather than run
over every historical message in Python.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@llstrk

llstrk commented Sep 27, 2026

Copy link
Copy Markdown

Automated AI review

Community review: Independent automated community review, unaffiliated with the Omarchy team, intended to help prepare PRs for their review.

Outcome: four behaviours do not match the description: turns still in opencode's WAL, pi on llama.cpp, a partition gap with the Codex collector, and one opencode row with an unexpected shape. Each has a small fix. A related open PR, #12803, also affects the #12032 change. Details follow, checked at 792b50f0.

Verified: the Local collector is wired in correctly.

  • omarchy-agent-usage-update picks the new collector up by glob. In the panel, the tab stays hidden with no data, the limits section and its separator are hidden when limits is empty, and the hero shows "Self-hosted". The panel and sync aggregation were checked by reading the code (Main.qml, Panel.qml), not by rendering.
  • The six agent-usage test files pass at this head with 77 assertions, as stated.
  • Count only subscription-backed sessions as Codex usage #12032's new Codex test fails against the base Codex collector ("Codex collector counts only subscription-backed native sessions") and passes with the change.
  • The head merges cleanly with the current quattro tip.

opencode rows still in the WAL are not counted

bin/omarchy-agent-usage-local:315 opens the database with ?mode=ro&immutable=1. opencode runs its database in WAL mode (opencode v1.18.32, packages/core/src/database/database.ts:27). With immutable=1, SQLite skips locking and change detection and does not read the WAL, so rows committed since the last checkpoint are invisible. The Codex and Claude collectors open the same file with db.resolve().as_uri() + "?mode=ro" and timeout=2.

Synthetic WAL-mode database, writer connection still open, rows committed but not checkpointed:

                           Local collector   Codex collector
ollama row (100 tokens)          0                 -
openai row (50 tokens)           -                 50
after wal_checkpoint             100               -

Impact: while opencode is running, the Local tab misses its newest turns. The test builds its database with the default rollback journal, so it cannot catch this. The size of the lag on a live opencode was not measured.

Suggested change: sqlite3.connect(db.resolve().as_uri() + "?mode=ro", uri=True, timeout=2), as the sibling collectors do. With only that line changed, the same fixture reports 100.

pi's llama.cpp provider is not recognised

KNOWN_LOCAL_PROVIDERS has llama-cpp and llamacpp, but pi's built-in llama.cpp provider records the id llama.cpp (pi v0.87.1, packages/coding-agent/src/extensions/llama/provider.ts:19). opencode's provider docs use the same id in their llama-server example. pi and opencode sessions record only the provider id, so this list, local.json and any same-named Codex model_providers entry are the only ways their turns get classified.

Impact: "pi on llama.cpp", named in the docstring and README as an example, is not counted. A synthetic pi session with a llama.cpp assistant message (111 tokens) and an ollama one (22) reports 22.

Suggested change: add llama.cpp to the list. Longer term, pi's models.json (providers.<id>.baseUrl, for custom providers) and opencode's config (provider.<id>.options.baseURL) do record endpoints, so reading them would let most pi and opencode providers be classified by URL, as the README describes. pi's built-in llama.cpp provider keeps its URL elsewhere, so its id still needs to be listed.

Rollouts without a readable session_meta are counted by both tabs

With a session_meta, the two collectors decide in opposite directions; without one, both count the rollout:

rollout first line                Codex (#12032)    Local
session_meta, provider openai     counts            skips
session_meta, local provider      skips             counts
no session_meta / unreadable      counts            counts   <- both

#12032's own "legacy" fixture has this shape. Recreated, Codex counts gpt-legacy 40 and Local also counts gpt-legacy 40. The partition test compares only model names across rollouts that all have a session_meta, and does not compare token sums, so it passes.

Impact: limited in practice. Current Codex writes session_meta first when it creates a rollout, so this takes a missing or truncated first line. It is still the case the description says is asserted.

Suggested change: in _scan_codex_file, count token_count lines only after a local session_meta has been seen. That variant passes the PR's Local test (11/11) and removes the overlap in the same probe.

One opencode row with an unexpected shape stops the rest of the opencode scan

In scan_opencode, the per-row try covers only json.loads. The .get calls on tokens, cache and time (:346-348) are outside it, so a row that is valid JSON with an unexpected shape (for example "cache": 7) raises into the outer handler, which marks the scan incomplete and returns. With rows of 10 tokens, that shape, and 1000 tokens, Local reports 10 and writes no cache. The Codex collector keeps these shape checks inside its per-row try.

Impact: the Local tab undercounts every opencode row after the first one with that shape. Because an incomplete scan is not cached, every run stops at the same row.

Suggested change: move the token and time extraction inside the per-row try.

Interaction with open #12803

#12803 adds a prefilter to the Codex collector that skips any line without "token_count" or "turn_context" before JSON parsing. The collector file merges with this PR's Codex change without a conflict (the Codex scanner test file conflicts, since both PRs append to it). A session_meta line contains neither string, so #12032's provider check is never reached:

for raw in handle:
  if '"token_count"' not in raw and '"turn_context"' not in raw:
    continue                      # from #12803: session_meta lines stop here
  ...
  if entry.get("type") == "session_meta":
    ...                           # from #12032: provider check, now unreachable

Impact: if both land, custom-provider rollouts count as Codex usage again, and the local ones are also counted by Local. With both test additions kept, #12032's provider test fails against the merged collector, so keeping both tests would catch it. Suggested change: whichever lands second should let session_meta lines through the prefilter.

Separately, open #10330 (pi) and #12635 (opencode) add collectors that count every provider, so they would also count the turns this tab counts. Whether overlapping tabs are intended is a maintainer decision.

Optional test improvement: three changes to the Local collector still pass all 11 assertions of the new test: removing the scan.scan_pi() call (there is no pi fixture), removing the scanDate check (the "another day" step adds no row between the cached and the fresh scan, so a hit and a miss both print 200), and removing the localhost, IP or bare-hostname rules (the only self-hosted URL fixture is server.home). A pi fixture including llama.cpp, a row inserted before the wrong-date check, a small is_self_hosted URL table and a no-session_meta rollout with a token-sum check in the partition test would cover these.

Documentation:

  • README: "Every collector counts only the turns its own subscription paid for" does not hold for the Claude collector, whose ~/.claude/projects scan has no provider filter. "Nothing a coding agent does goes uncounted" conflicts with the later note that third-party APIs are counted by no collector.
  • README: ids "with no URL to resolve (…) fall back to" the runtime list, but the code checks the list first, so a Codex vllm provider configured with a public HTTPS base_url counts as Local.
  • manual/17-ai.md:39 still lists only Claude Code, Codex and Fireworks.

Review information

Test scope: Source review of this head, the PR base and the current quattro tip, plus Codex CLI rust-v0.157.1, opencode v1.18.32 and pi v0.87.1 sources and the SQLite URI documentation. The PR's agent-usage tests, mutation variants and probes ran in an isolated sandbox with synthetic Codex rollouts, pi sessions and opencode databases. No live opencode, Codex or pi process, no real usage data and no panel rendering were involved. Cross-device sync was checked by reading the code only. The #12803 interaction comes from a merge of both heads, run in the same sandbox, and source reading.

AI process: Opus 5.5 Medium coordination and synthesis, Opus 5.5 Xhigh technical review and final fact check, GPT 6 Sol Xhigh search for related issues, Opus 5.5 Medium editorial check.

Opt out: To stop receiving these reviews, reply to this comment saying so.

@jandrusk

Copy link
Copy Markdown

This is stacked on #12032, which is now closed unmerged, so the branch can't land as-is. Please restack the Local-tab commit onto current quattro (drop or re-home the Codex subscription-only commit) and update the PR description.

dhh pushed a commit that referenced this pull request Oct 15, 2026
The collector and test parts of #12034; its Codex provider fix is already
upstream.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants