Skip to content

Usage-divergence flag: independent-count baseline (deferred from #48) #50

Description

@c-1k

The usage-divergence flag was removed from PR #48 because its baseline was wrong: it compared provider-reported cost against the pre-call estimate, which prices output at the full max_tokens cap — so it flagged flagged: true on essentially every normal short completion (e.g. 1000in/300out under an 8192 cap → ratio ~0.06 < 1/4 → flagged) and spammed the audit chain with usage_divergence events.

The right design (do this here)

The flag's purpose is to catch a server under-reporting usage. Compare provider-reported usage against usertrust's OWN independent token count of the ACTUAL content:

  • independentCost = estimateCost(model, estimateInputTokens(actualInputMessages), countTokens(actualOutputText))
  • flag when actualCost / independentCost < 1/factor (reported materially below what we independently counted) — optionally also the over direction.
  • Acceptance: a normal 1000in/300out call → reported ≈ independent count → ratio ≈ 1 → NOT flagged; a server claiming ~50 tokens for ~2000 tokens of content we counted → ratio ≈ 0.03 → flagged.

Why it's non-trivial (the blast radius that made it too large for #48)

No output TEXT is accumulated anywhere today — every stream path accumulates provider-reported usage NUMBERS only. This needs:

  1. Per-provider output-text accumulation across THREE paths: generic createGovernedStream, the settleViaMessageStream streamEvent tap, and non-stream response bodies (OpenAI choices[].message.content, Anthropic content[].text, Responses output_text/output).
  2. A text→token counter (reuse/extend estimateInputTokens' tokenizer).
  3. Thread the independent count into the divergence computation at both settle sites.
  4. Re-introduce receipt.divergence + config.divergence + the usage_divergence audit event with the corrected semantics.

This closes the 'blind trust in provider-reported usage' limitation properly (it is intentionally left open in LIMITATIONS.md until this lands).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions