Fix live capture reliability, observability, and dashboard metrics

docs/COMPETITIVE_GAPS.md

@@ -5,8 +5,8 @@ from its README and documentation site rather than from memory.
 
 ## A correction I owe first
 
-On issue #201 I wrote that they "audit waste in your *setup*" while we "change
-what the *operations* cost", and that those compose. **That was wrong.**
+On issue #201 I wrote that they "audit waste in your _setup_" while we "change
+what the _operations_ cost", and that those compose. **That was wrong.**
 
 They ship active compression that changes operation cost directly:
 
@@ -24,14 +24,14 @@ model choosing to cooperate. Ours requires the model to reissue the call against
 
 ## Where we are genuinely ahead
 
-| | Us | Them |
-|---|---|---|
-| **License** | MIT | PolyForm **Noncommercial** — unusable at work |
-| Clients | 15 | 6 |
-| Savings measurement | **withheld control arm** | counterfactual baseline + metered receipts, no control |
-| Operational tooling | ~100 MCP tools (git, build, db, API, search) | none — it is not an MCP server |
-| Cross-session knowledge | anchored graph with computed staleness | decision extraction, capped at 10/session |
-| Enforcement | refusal with the replacement named | nudges and transparent rewriting |
+|                         | Us                                           | Them                                                   |
+| ----------------------- | -------------------------------------------- | ------------------------------------------------------ |
+| **License**             | MIT                                          | PolyForm **Noncommercial** — unusable at work          |
+| Clients                 | 15                                           | 6                                                      |
+| Savings measurement     | **withheld control arm**                     | counterfactual baseline + metered receipts, no control |
+| Operational tooling     | ~100 MCP tools (git, build, db, API, search) | none — it is not an MCP server                         |
+| Cross-session knowledge | anchored graph with computed staleness       | decision extraction, capped at 10/session              |
+| Enforcement             | refusal with the replacement named           | nudges and transparent rewriting                       |
 
 The licence is still the single most decisive line for anyone using this at
 work, and the control-arm measurement is a claim they cannot make. The MCP tool
@@ -61,7 +61,7 @@ decision density, agent efficiency), letter grades S–F, a status-line integrat
 with colour transitions, and degradation tracking framed against MRCR falling
 93% → 76% as context grows.
 
-We have nothing here. It is also the most *visible* feature they have — it is on
+We have nothing here. It is also the most _visible_ feature they have — it is on
 screen constantly, which is worth more than its accuracy.
 
 ### 3. Progressive disclosure of tool results
@@ -108,10 +108,14 @@ per-message cost breakdown.
 We have `get_optimization_report`. The gap is less about capability than about
 there being an obvious thing to run that tells you what to do next.
 
-### 8. Dollars, not tokens
+### 8. Billing evidence, not a universal token price
 
-They price against four tiers and report `$/month` with receipts. We report
-tokens. Same data, and the unit people actually feel.
+Provider/model tables alone cannot reconstruct a CLI user's bill because cache
+reads and writes, routing, long-context tiers, subscriptions, included credits,
+and enterprise agreements change marginal cost. Token Optimizer now reports
+verified transport tokens and fails closed on dollars unless the user supplies
+an effective blended rate. Provider-native billing exports remain the source of
+truth for actual spend.
 
 ### 9. Skills and configuration health
 

docs/HOOKS-INSTALLATION.md

@@ -32,11 +32,13 @@ The global hooks system provides **7-phase token optimization** that runs automa
 ## Prerequisites
 
 ### All Platforms
+
 - **Claude Code** CLI installed globally
 - **Node.js** 18+ and npm
 - **token-optimizer-mcp** package (will be installed by installer)
 
 ### Platform-Specific
+
 - **Windows**: PowerShell 5.1+
 - **macOS**: Bash 4.0+, Homebrew recommended
 - **Linux**: Bash 4.0+
@@ -434,18 +436,21 @@ chmod +x ~/.claude-global/hooks/helpers/*.sh
 ### 1. Check Hooks Are Firing
 
 **Windows:**
+
 ```powershell
 # View dispatcher log
 Get-Content "$env:USERPROFILE\.claude-global\hooks\logs\dispatcher.log" -Tail 20
 ```
 
 **macOS / Linux:**
+
 ```bash
 # View dispatcher log
 tail -20 ~/.claude-global/hooks/logs/dispatcher.log
 ```
 
 You should see entries like:
+
 ```
 [2025-10-26 10:00:00] [PreToolUse] DISPATCHER INVOKED
 [2025-10-26 10:00:01] [PreToolUse] Tool: Read
@@ -455,18 +460,21 @@ You should see entries like:
 ### 2. Check MCP Tools Are Being Called
 
 **Windows:**
+
 ```powershell
 # View MCP invocation log
 Get-Content "$env:USERPROFILE\.claude-global\hooks\logs\mcp-invocation.log" -Tail 20
 ```
 
 **macOS / Linux:**
+
 ```bash
 # View MCP invocation log
 tail -20 ~/.claude-global/hooks/logs/mcp-invocation.log
 ```
 
 You should see:
+
 ```
 [2025-10-26 10:00:00] [DEBUG] Invoking MCP: token-optimizer -> smart_read
 [2025-10-26 10:00:01] [DEBUG] Request: {"jsonrpc":"2.0",...}
@@ -475,25 +483,29 @@ You should see:
 ### 3. Check Token Optimization
 
 **Windows:**
+
 ```powershell
 # View token optimizer log
 Get-Content "$env:USERPROFILE\.claude-global\hooks\logs\token-optimizer.log" -Tail 10
 ```
 
 **macOS / Linux:**
+
 ```bash
 # View token optimizer log
 tail -10 ~/.claude-global/hooks/logs/token-optimizer.log
 ```
 
 You should see savings like:
+
 ```
 [2025-10-26 10:00:00] [SUGGEST] Read File (~5000 tokens) → Token Optimizer (~1850 tokens) | Savings: 3150 tokens (63%)
 ```
 
 ### 4. Check Operations Tracking
 
 **Windows:**
+
 ```powershell
 # View most recent operations CSV
 Get-ChildItem "$env:USERPROFILE\.claude-global\hooks\data\operations-*.csv" |
@@ -503,6 +515,7 @@ Get-ChildItem "$env:USERPROFILE\.claude-global\hooks\data\operations-*.csv" |
 ```
 
 **macOS / Linux:**
+
 ```bash
 # View most recent operations CSV
 ls -t ~/.claude-global/hooks/data/operations-*.csv | head -1 | xargs tail -10
@@ -515,6 +528,7 @@ ls -t ~/.claude-global/hooks/data/operations-*.csv | head -1 | xargs tail -10
 **Symptom**: No log entries in dispatcher.log
 
 **Solutions**:
+
 1. Check workspace trust is accepted (see Manual Installation Step 5 for your platform)
 2. Verify settings.json hook commands have correct paths
 3. Restart Claude Code
@@ -527,14 +541,17 @@ ls -t ~/.claude-global/hooks/data/operations-*.csv | head -1 | xargs tail -10
 **Symptom**: dispatcher.log shows hooks firing but no MCP calls
 
 **Solutions**:
+
 1. Verify token-optimizer MCP server is running:
 
    **Windows:**
+
    ```powershell
    Get-Process | Where-Object { $_.Name -like "*node*" -and $_.CommandLine -like "*token-optimizer*" }
    ```
 
    **macOS / Linux:**
+
    ```bash
    ps aux | grep token-optimizer | grep -v grep
    ```
@@ -552,6 +569,7 @@ ls -t ~/.claude-global/hooks/data/operations-*.csv | head -1 | xargs tail -10
 **Symptom**: "File cannot be loaded because running scripts is disabled"
 
 **Solution**:
+
 ```powershell
 Set-ExecutionPolicy RemoteSigned -Scope CurrentUser
 ```
@@ -561,6 +579,7 @@ Set-ExecutionPolicy RemoteSigned -Scope CurrentUser
 **Symptom**: "Permission denied" errors when running hooks
 
 **Solution**:
+
 ```bash
 # Make all hook scripts executable
 chmod +x ~/.claude-global/hooks/dispatcher.sh
@@ -576,6 +595,7 @@ ls -la ~/.claude-global/hooks/dispatcher.sh
 **Symptom**: "Bad substitution" or syntax errors
 
 **Solution**: macOS ships with Bash 3.x, but hooks require Bash 4+
+
 ```bash
 # Install Bash 4+ via Homebrew
 brew install bash
@@ -731,23 +751,17 @@ C:\Users\YOUR_USERNAME\
         └── token-optimizer-mcp         # MCP server
 ```
 
-## Token Reduction Metrics
-
-Based on production usage across 38,000+ operations:
-
-| Tool | Average Tokens Before | Average Tokens After | Reduction % |
-|------|----------------------|---------------------|-------------|
-| Read | 5,000 | 1,850 | 63% |
-| Grep | 2,000 | 740 | 63% |
-| Glob | 1,500 | 555 | 63% |
-| Edit | 3,500 | 1,295 | 63% |
-| **Overall** | **3,000** | **1,110** | **63%** |
+## Token reduction measurement
 
-### Session-Level Impact
+Static reduction percentages and repository scan sizes are not production
+savings evidence. Current releases require a materialized MCP payload before
+progressive disclosure and the actual returned payload, then debit every linked
+`expand` response. Run `get_optimization_report` for the verified net or
+`npm run dashboard:audit-savings` to inspect both qualifying and excluded rows.
 
-- **Operations per session**: ~200-500
-- **Token savings per session**: 300,000-700,000 tokens
-- **Cost savings (at $3/M tokens)**: $0.90-$2.10 per session
+Dollar values are **Not priced** unless the user configures an effective blended
+input-token rate that reflects their provider, model, cache read/write mix,
+route, tier, plan, and credits. See [Token accounting](./TOKEN_ACCOUNTING.md).
 
 ## Advanced Configuration
 

docs/TOKEN_ACCOUNTING.md

@@ -0,0 +1,101 @@
+# Token accounting contract
+
+Token Optimizer reports four different quantities because combining them would
+turn assumptions into evidence:
+
+1. **Observed returned context** is the local tokenizer count of text that
+   crossed the MCP response boundary. It is not a provider invoice.
+2. **Gross verified transport reduction** is the difference between two
+   materialized payloads: the tool result before progressive disclosure and
+   the smaller payload actually returned.
+3. **Expansion debit** is every later `expand` response linked to that preview.
+4. **Net verified MCP transport avoided** is gross reduction minus expansion
+   debits. It may be negative.
+
+Graph substitutions, causal graph-reuse estimates, repository bytes scanned,
+tool-reported compression fields, and legacy rows without provenance never
+enter the verified MCP headline.
+
+## Qualification rules
+
+A reduction qualifies only when its durable row contains all of the following:
+
+- measurement schema version 2;
+- a unique MCP request identity;
+- exact UTF-8 byte counts for both materialized payloads and a consistent byte
+  delta;
+- SHA-256 fingerprints for both payloads without storing their contents in the
+  analytics row;
+- locally counted tokens with `saved = before - returned`;
+- the content-addressed expansion reference printed in the preview.
+
+Request retries with the same process/session/RPC identity are stored once.
+Historical rows remain available through `npm run dashboard:audit-savings` but
+cannot be promoted by a boolean migration or a similarly named tool field.
+
+## What a token means here
+
+The MCP boundary does not expose the provider's final billing tokenizer. The
+ledger therefore uses a GPT-4-compatible local tokenizer estimate so the same
+transport bytes can be compared consistently across Codex, Claude Code,
+Gemini, Copilot, and other clients. The dashboard says this explicitly.
+
+Provider-native study receipts keep uncached input, cache creation, cache read,
+and output fields separate when a client supplies them. Those receipt fields
+are not substituted into MCP transport measurements because they answer a
+different question.
+
+## Provider usage and cost
+
+The dashboard reads native local usage receipts from supported CLI transcripts
+and keeps their billable dimensions separate: uncached input, cached reads,
+5-minute/1-hour/generic cache writes, and output (including thinking tokens when
+the provider bills them as output).
+
+Adapters preserve each provider's semantics. Codex/OpenAI and Gemini prompt
+totals include cached input, while Anthropic `input_tokens` excludes cache reads
+and cache creation. Claude response fragments are deduplicated by request id,
+and repeated Codex usage snapshots are deduplicated by cumulative state.
+
+A versioned price catalog selects an exact provider, model, route, request
+timestamp, and long-context tier. Every priced result carries an official source
+link and a catalog verification date. An ambiguous id such as `claude-sonnet`,
+an unknown route, or a captured dimension without a published rate remains
+**Not priced**; the nearest model is never guessed.
+
+The dashboard distinguishes two money quantities:
+
+1. **API/list-price equivalent** applies published rates to native observed
+   usage. A subscription, enterprise agreement, included credits, processing
+   tier, or cloud marketplace may make the user's invoice different.
+2. **Provider-reported actual charge** appears only when the native receipt
+   contains a charge. Token Optimizer does not reconstruct an invoice from list
+   prices.
+
+For a verified MCP payload reduction, the conservative direct saving is one
+immediate uncached-input equivalent for that operation's exact model. The ledger
+does not multiply that saving by hypothetical future cache reads or writes.
+Paired experiments can separately measure downstream effects from native usage.
+
+Current provider references are the authoritative source behind catalog rows:
+
+- [OpenAI model pricing and cached-input rates](https://developers.openai.com/api/docs/models/compare)
+- [Anthropic API pricing](https://platform.claude.com/docs/en/about-claude/pricing)
+- [Anthropic prompt-cache usage and read/write multipliers](https://platform.claude.com/docs/en/build-with-claude/prompt-caching)
+- [Gemini API model pricing](https://ai.google.dev/gemini-api/docs/pricing)
+- [GitHub Copilot models and token pricing](https://docs.github.com/en/copilot/reference/copilot-billing/models-and-pricing)
+
+The separate `analyze_project_tokens` report still accepts an explicit
+`costPerMillionTokens` because repository analysis has no model request receipt.
+That caller-supplied value remains labeled as a configured equivalent.
+
+## Auditing an existing ledger
+
+```bash
+npm run dashboard:audit-savings
+npm run dashboard:audit-savings -- /absolute/path/to/analytics.db
+```
+
+The audit is read-only. It reports verified net savings, gross reductions,
+expansion debits, observed returned context, excluded historical/tool claims,
+per-tool concentration, and the largest excluded rows.