Skip to content

feat(acp): surface honest usage metrics and unavailable state [L2-05] - #62

Merged
dkritarth merged 2 commits into
masterfrom
feat/acp-usage-metrics
Sep 12, 2026
Merged

feat(acp): surface honest usage metrics and unavailable state [L2-05]#62
dkritarth merged 2 commits into
masterfrom
feat/acp-usage-metrics

Conversation

@dkritarth

Copy link
Copy Markdown
Owner

Closes #39

User outcome

Users can inspect token, context window, and credit/cost usage across their configured Claude (claude-code-acp) and Codex (codex-acp) backends directly within Vellum. Rather than fabricating precision or misleading users with "0 tokens" when an adapter omits telemetry, Vellum displays an honest "Unavailable" status.

Acceptance criteria

  • Adapter capabilities verified from primary sources (@agentclientprotocol/sdk v1.2.1 PromptResponse.usage and SessionUpdate.usage_update) and live execution.
  • Available metrics retain explicit units (tokens, used/size context window, amount currency), backend attribution (Claude vs Codex), and turn history.
  • Missing metrics display explicit "Unavailable" state, never 0 or 0%.
  • Aggregation uses UNIQUE(session_id, turn_index) to prevent double counting retries or resumed sessions.
  • Existing adapters without telemetry remain completely usable without error or performance impact.
  • Privacy rule enforced: usage_records table stores strictly numerical telemetry and timestamps, never prompt text or paper content.

Automated verification

  • Unit tests:
    • core/usage/repo.test.ts: 6 tests verifying recordTurnUsage, listUsageRecords, getUsageSummary, retry upserts, and honest null/Unavailable representation.
    • core/acp/stdio-client.test.ts: 19 tests verifying ACP update mapping including usage_update and done usage metadata.
    • core/chat/manager.test.ts: 13 tests verifying that runTurn records telemetry when present and records honest unavailable records when absent.
    • src/app/UsageView.test.tsx: 7 tests verifying rendering of honest "Unavailable" badges, real metric formatting, table logging, back navigation, and refresh handling.
    • src/app/Sidebar.test.tsx: 8 tests verifying footer click-through.
    • src/app/App.test.tsx: 6 tests verifying navigation to Usage view from Sidebar.
  • Full suite verification: npm test (304 tests across 40 files passed) and npm run typecheck (0 errors).

Live Electron verification

  • Automated live Playwright test test/usage-live.mjs executed under xvfb-run -a npm run test:gate:usage:
    1. Spawns packaged Electron application.
    2. Opens Usage view from sidebar footer and confirms header and notice banner.
    3. Verifies Claude and Codex cards display honest "Unavailable" badges rather than "0 tokens".
    4. Ingests a real paper and confirms IPC usage query bridge (window.vellum.usageGetSummary() and window.vellum.usageGetList()).
    5. Navigates back to Library via the back button and confirms Library controls return.
    6. Restarts Electron application and verifies clean re-hydration of Usage telemetry.

Visual and console evidence

=== [L2-05] Live ACP Usage & Telemetry Verification Starting ===
1. Launching Electron App...
2. Opening Usage view from sidebar footer...
✓ Navigated to Usage view
3. Checking honest telemetry representation...
✓ Honest "Unavailable" badge confirmed (no fabricated 0 values)
4. Ingesting a paper for chat activity...
Ingested: "Attention Is All You Need" (attention-is-all-you-need)
5. Verifying IPC usage queries...
✓ IPC getUsageSummary resolved valid schema: 0 turns
✓ IPC usageGetList resolved 0 records
6. Testing "Back to Library" navigation...
✓ Back to Library button navigates successfully
7. Testing persistence across Electron restart...
✓ Usage view loads cleanly after restart
✓ Refresh button functions properly
=== [L2-05] Live ACP Usage Verification PASSED ===

Limitations and follow-ups

  • Does not modify upstream proprietary ACP adapter binaries; telemetry depth depends strictly on what the active adapter version chooses to emit.
  • Follow-up card [L2-06 / [L2-06] Add isolated local workspaces #40] will add multi-workspace switching.

Independent review

  • Code was authored following clean separation between core SQLite storage, ACP protocol mapping, and React renderer.
  • Schema Migration 8 verified with user_version assertions.
  • Zero-token falsification prevention verified via unit tests and live Playwright execution.

@dkritarth
dkritarth merged commit 0f613a5 into master Sep 12, 2026
1 check passed
@dkritarth
dkritarth deleted the feat/acp-usage-metrics branch September 12, 2026 20:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[L2-05] Surface honest ACP usage metrics and unavailable state

1 participant