Estimate cost from tokens and rank report findings by waste

CHANGELOG.md

@@ -6,6 +6,10 @@ All notable changes to AgentProf will be documented here.
 
 ### Added
 
+- Token-based cost estimation for spans that arrive without provider cost fields, with a configurable `pricing` model rate table and a bundled default table.
+- `agentprof pricing list` to show the effective model rates and where each came from.
+- Estimated-versus-provider cost reporting in `agentprof normalize`, `agentprof cost ledger`, and report ledger tables.
+- `Top Findings` section in Markdown and HTML reports, plus `top_findings` and `summary.top_issue_kind` in the JSON payload.
 - One-command `agentprof demo` workflow with bundled sample traces.
 - Static HTML report generation alongside Markdown and JSON reports.
 - Multi-agent waste SVG visual artifacts.
@@ -14,5 +18,7 @@ All notable changes to AgentProf will be documented here.
 
 ### Changed
 
+- Report issues are ranked by attributed waste, then severity and confidence, instead of by issue ID.
+- Retry-loop severity now escalates with repeated wasted attempts, and multi-agent severity tracks the cost multiple over baseline, instead of being fixed.
 - README now leads with concrete trace-waste findings and the demo workflow.
 - CI now tests Python 3.11 and 3.12, runs demo smoke checks, builds the package, and validates wheel installation.

README.md

@@ -118,12 +118,14 @@ AgentProf complements tracing tools by profiling exported traces locally and tur
 - Store raw and normalized trace data in a local DuckDB database.
 - Normalize Langfuse observations into canonical span and trace tables.
 - Show data-quality coverage for parent links, status, costs, tokens, models, and I/O hashes.
+- Estimate span cost from token counts and model names when an export carries no provider cost fields.
 - Build an idempotent cost ledger from normalized span costs.
 - Print a status-based cost waterfall for successful, failed, and unknown span costs.
 - Detect retry loops where the same failing call repeats with the same input fingerprint and error signature.
 - Detect configured tool/spec contract violations from normalized redacted previews and error messages.
 - Estimate multi-agent orchestration overhead against configured or observed single-agent baselines.
 - Generate local Markdown, JSON, and static HTML reports from persisted issues, evidence, costs, and optional visuals.
+- Rank report issues by attributed waste and lead with a top-findings section.
 - List and show generated reports from the local store.
 
 ## Current Status
@@ -162,7 +164,7 @@ uv run agentprof report list
 uv run agentprof store stats
 ```
 
-The default fixture focuses on retry/spec behavior and does not include cost fields, so `agentprof cost ledger` will produce zero ledger entries for that sample. Use `tests/fixtures/langfuse_multi_agent_observations.json` for a costed multi-agent waste demo.
+The default fixture focuses on retry/spec behavior and carries neither cost fields nor token counts, so `agentprof cost ledger` will produce zero ledger entries for that sample. Use `tests/fixtures/langfuse_multi_agent_observations.json` for a costed multi-agent waste demo, or `tests/fixtures/langfuse_token_only_observations.json` for an export that is priced from token counts alone.
 
 The workflow creates local files only. After `agentprof report generate`, the default report artifacts are written under `.agentprof/reports/` and can be inspected with `agentprof report show`.
 
@@ -330,9 +332,51 @@ Observed mode matches costed successful single-agent traces by normalized root t
 | `agentprof report generate` | Generate Markdown, JSON, and HTML reports from persisted analysis results. |
 | `agentprof report list` | List generated reports recorded in the local store. |
 | `agentprof report show REPORT_ID` | Print a generated report's Markdown, JSON, or HTML artifact. |
+| `agentprof pricing list` | Show the model rates used to estimate cost for uncosted spans. |
 | `agentprof store stats` | Show row counts for all store tables. |
 | `agentprof store reset --yes` | Delete and recreate the local DuckDB store. |
 
+## Cost Estimation
+
+Many trace exports carry token counts and model names but no provider cost fields. Without a price, every analyzer reports $0.000000000, because waste attribution is built from span cost.
+
+AgentProf fills that gap during `agentprof normalize`: a span that has a model name and token counts but no provider cost is priced from a model rate table and stored with `cost_confidence = estimated`. Spans that already carry a provider cost are never overwritten, so estimation can only add attribution where there was none.
+
+Inspect the effective rates:
+
+```bash
+uv run agentprof pricing list
+```
+
+Override any model, or add one AgentProf does not know, in `agentprof.yml`:
+
+```yaml
+pricing:
+  enabled: true
+  use_default_table: true
+  models:
+    - model: house-fine-tune-v3
+      input_per_1m_usd: '0.90'
+      output_per_1m_usd: '2.70'
+```
+
+- Rates are USD per 1,000,000 tokens. Quote them as strings to avoid float rounding.
+- `model` matches the normalized model name exactly, or as a prefix followed by a version or date suffix, so `gpt-4o-mini` prices `gpt-4o-mini-2024-07-18` and `claude-opus-4-1` prices `claude-opus-4-1-20250805`. Longer matches win, and configured entries beat bundled defaults.
+- Only a date or build stamp counts as the same model. A separately priced release does not inherit a base rate: `gpt-4` does not price `gpt-4.1`, `gpt-4o` does not price `gpt-4o-mini`, and `claude-sonnet-4` does not price `claude-sonnet-4-5`. Those models are reported as unpriced instead, so a missing rate shows up as a prompt to configure one rather than as a wrong number.
+- Spans with a negative token count are left unpriced, since a negative component would subtract from the other and can drive a cost below zero.
+- A provider prefix such as `anthropic/claude-sonnet-4-5` is matched with and without the prefix.
+- Set `use_default_table: false` to price only from your own entries, or `enabled: false` to turn estimation off entirely.
+
+Estimated spend is always labelled rather than blended into provider-reported cost:
+
+- `agentprof normalize` reports estimated span coverage and names any model it had tokens for but no price.
+- `agentprof cost ledger` splits provider-reported cost from estimated cost.
+- Reports carry a `confidence` column per ledger row and a `span_costs_by_confidence_usd` summary field. That field covers the normalized span ledger only, because analyzer attributions re-point at spend the span ledger already measured and carry their own notion of confidence.
+
+Provider-reported spend is never displaced. A span is left alone both when it already carries a cost and when it shares a trace path with a span that does, because cost attribution counts only leaf costs: estimating a token-bearing child underneath a provider-costed parent would drop the parent's authoritative amount in favour of the estimate.
+
+The bundled rates are public list prices recorded on a fixed date, and provider pricing changes. Treat estimates as a starting point and override the models that matter to you before acting on a total.
+
 ## Spec Contracts
 
 Spec-violation analysis is opt-in. Add contracts under `analyzers.spec_violations.contracts`:
@@ -422,6 +466,13 @@ trace/span spend. The summary also includes gross and overlapping attribution
 fields so consumers can inspect how much per-issue waste was removed from the
 recoverable total.
 
+Issues are ranked by attributed waste, so the most expensive finding leads the
+report instead of whichever issue ID happened to sort first. Severity and
+confidence break ties, which is what orders findings when a trace carries no cost
+data at all. Every report opens with a `Top Findings` section covering the three
+leading issues with their recommendation and one evidence line, and the JSON
+payload exposes the same list as `top_findings` alongside `summary.top_issue_kind`.
+
 When persisted `multi_agent_waste` issues exist, `agentprof report generate` also writes `<report-id>-multi-agent-waste.svg` next to the Markdown/JSON/HTML files, embeds or links it from the Markdown and HTML reports, and records the artifact filename in `summary.artifacts.multi_agent_waste_svg`.
 
 List and inspect generated reports:

agentprof/analyze/multi_agent_waste.py

@@ -29,6 +29,8 @@
 AGENT_SPAN_TYPES = {"root", "agent"}
 BASELINE_MODES = {"estimated", "observed"}
 FAILURE_STATUSES = {"error", "timeout", "cancelled"}
+HIGH_SEVERITY_COST_MULTIPLE = Decimal("3")
+MEDIUM_SEVERITY_COST_MULTIPLE = Decimal("1.5")
 
 
 @dataclass(frozen=True)
@@ -299,6 +301,16 @@ def _normalized_name(value: str | None) -> str:
     return " ".join((value or "").replace("_", " ").replace("-", " ").lower().split())
 
 
+def _severity_from_finding(finding: MultiAgentWasteFinding) -> str:
+    """Severity tracks how far the orchestration ran over its baseline."""
+
+    if finding.cost_multiple >= HIGH_SEVERITY_COST_MULTIPLE:
+        return "high"
+    if finding.cost_multiple >= MEDIUM_SEVERITY_COST_MULTIPLE:
+        return "medium"
+    return "low"
+
+
 def _issue_from_finding(
     finding: MultiAgentWasteFinding, *, affected_spans: int
 ) -> IssueRecord:
@@ -307,7 +319,7 @@ def _issue_from_finding(
         issue_id=finding.issue_id,
         kind=ISSUE_KIND,
         title=f"Estimated orchestration overhead in {root_label}",
-        severity="medium",
+        severity=_severity_from_finding(finding),
         confidence=finding.confidence,
         first_seen=finding.first_seen,
         last_seen=finding.last_seen,

agentprof/analyze/retry_loop.py

@@ -20,6 +20,8 @@
 ATTRIBUTION_METHOD = "retry_loop"
 WASTED_COST_TYPE = "wasted_retry_cost"
 FAILURE_STATUSES = {"error", "timeout", "cancelled"}
+HIGH_SEVERITY_WASTED_ATTEMPTS = 4
+MEDIUM_SEVERITY_WASTED_ATTEMPTS = 2
 
 
 def analyze_retry_loops(
@@ -144,12 +146,22 @@ def _finding_from_group(
     )
 
 
+def _severity_from_finding(finding: RetryLoopFinding) -> str:
+    """Severity tracks how many times the agent repeated a known-failing call."""
+
+    if finding.wasted_attempts >= HIGH_SEVERITY_WASTED_ATTEMPTS:
+        return "high"
+    if finding.wasted_attempts >= MEDIUM_SEVERITY_WASTED_ATTEMPTS:
+        return "medium"
+    return "medium" if finding.wasted_cost_usd > 0 else "low"
+
+
 def _issue_from_finding(finding: RetryLoopFinding) -> IssueRecord:
     return IssueRecord(
         issue_id=finding.issue_id,
         kind=ISSUE_KIND,
         title=f"Repeated failing call to {finding.name}",
-        severity="medium" if finding.wasted_cost_usd > 0 else "low",
+        severity=_severity_from_finding(finding),
         confidence="high",
         first_seen=finding.first_seen,
         last_seen=finding.last_seen,

agentprof/cli.py

@@ -28,6 +28,7 @@
     write_workspace_gitignore,
     write_default_config,
 )
+from agentprof.cost.pricing import DEFAULT_PRICE_TABLE_AS_OF, build_pricing_table
 from agentprof.cost.runner import build_cost_ledger
 from agentprof.ingest.langfuse_export import (
     LangfuseExportFormat,
@@ -69,11 +70,13 @@ class MultiAgentBaselineMode(StrEnum):
 cost_app = typer.Typer(help="Analyze normalized trace costs.")
 analyze_app = typer.Typer(help="Run deterministic analyzers over normalized traces.")
 report_app = typer.Typer(help="Generate local AgentProf reports.")
+pricing_app = typer.Typer(help="Inspect model pricing used for cost estimation.")
 app.add_typer(store_app, name="store")
 app.add_typer(import_app, name="import")
 app.add_typer(cost_app, name="cost")
 app.add_typer(analyze_app, name="analyze")
 app.add_typer(report_app, name="report")
+app.add_typer(pricing_app, name="pricing")
 
 
 def _version_callback(value: bool) -> None:
@@ -229,7 +232,7 @@ def _run_demo_pipeline(
         )
     console.print(f"  imported {import_result.observations_imported} sample observations")
 
-    normalize_store(store)
+    normalize_store(store, pricing=config.pricing)
     console.print("  normalized spans and traces")
 
     analyze_retry_loops(store)
@@ -332,25 +335,44 @@ def normalize(
 
     config = _load_config_or_exit()
     store = DuckDBStore(config.store.path)
-    result = normalize_store(store, source=source)
+    result = normalize_store(store, source=source, pricing=config.pricing)
     quality = result.data_quality
 
     console.print("[green]Normalized imported spans.[/green]")
     console.print(f"  raw spans seen: {result.raw_spans_seen}")
     console.print(f"  normalized spans: {result.normalized_spans}")
     console.print(f"  normalized traces: {result.normalized_traces}")
+    if result.estimated_cost_spans:
+        console.print(
+            f"  estimated cost for {result.estimated_cost_spans} span(s) "
+            f"without provider cost: {_format_usd(result.estimated_cost_usd)}"
+        )
 
     table = Table(title="Data quality")
     table.add_column("Metric")
     table.add_column("Value", justify="right")
     table.add_row("Parent coverage", f"{quality.parent_coverage_pct:.1f}%")
     table.add_row("Status coverage", f"{quality.status_coverage_pct:.1f}%")
     table.add_row("Cost coverage", f"{quality.cost_coverage_pct:.1f}%")
+    table.add_row("Cost coverage (source)", f"{quality.source_cost_coverage_pct:.1f}%")
+    table.add_row(
+        "Cost coverage (estimated)", f"{quality.estimated_cost_coverage_pct:.1f}%"
+    )
     table.add_row("Token coverage", f"{quality.token_coverage_pct:.1f}%")
     table.add_row("Model coverage", f"{quality.model_coverage_pct:.1f}%")
     table.add_row("I/O hash coverage", f"{quality.io_hash_coverage_pct:.1f}%")
     console.print(table)
 
+    if result.unpriced_models:
+        console.print(
+            "[yellow]No price configured for "
+            f"{', '.join(result.unpriced_models)}.[/yellow]"
+        )
+        console.print(
+            "  Add rates under `pricing.models` in agentprof.yml to attribute "
+            "their cost."
+        )
+
 
 @cost_app.command("ledger")
 def cost_ledger() -> None:
@@ -365,6 +387,11 @@ def cost_ledger() -> None:
     console.print(f"  ledger entries: {result.ledger_entries}")
     console.print(f"  traces with cost: {result.traces_with_cost}")
     console.print(f"  total cost: {_format_usd(result.total_cost_usd)}")
+    console.print(f"  provider-reported cost: {_format_usd(result.source_cost_usd)}")
+    console.print(
+        f"  estimated cost: {_format_usd(result.estimated_cost_usd)} "
+        f"across {result.estimated_entries} entry(s)"
+    )
 
     table = Table(title="Cost waterfall")
     table.add_column("Cost type")
@@ -584,6 +611,8 @@ def report_generate(
     console.print("[green]Generated AgentProf report.[/green]")
     console.print(f"  report id: {result.report_id}")
     console.print(f"  issues: {result.issues}")
+    if result.top_finding_title:
+        console.print(f"  top finding: {result.top_finding_title}")
     console.print(f"  evidence items: {result.evidence_items}")
     console.print(f"  cost entries: {result.cost_entries}")
     console.print(f"  total wasted cost: {_format_usd(result.total_wasted_cost_usd)}")
@@ -658,6 +687,46 @@ def report_show(
         raise typer.Exit(code=2) from exc
 
 
+@pricing_app.command("list")
+def pricing_list() -> None:
+    """Show the model prices used to estimate cost for uncosted spans."""
+
+    config = _load_config_or_exit()
+    if not config.pricing.enabled:
+        console.print(
+            "[yellow]Cost estimation is disabled (`pricing.enabled: false`).[/yellow]"
+        )
+        console.print("  Spans without provider cost fields stay unattributed.")
+        return
+
+    table_prices = build_pricing_table(config.pricing).effective_prices()
+    if not table_prices:
+        console.print("No model prices are configured.")
+        console.print(
+            "  Add rates under `pricing.models` in agentprof.yml, or set "
+            "`pricing.use_default_table: true`."
+        )
+        return
+
+    table = Table(title="Model prices (USD per 1M tokens)")
+    table.add_column("Model")
+    table.add_column("Input", justify="right")
+    table.add_column("Output", justify="right")
+    table.add_column("Source")
+    for price in sorted(table_prices, key=lambda item: (item.source, item.model)):
+        table.add_row(
+            price.model,
+            f"${price.input_per_1m_usd}",
+            f"${price.output_per_1m_usd}",
+            price.source,
+        )
+    console.print(table)
+    console.print(
+        f"  Bundled default rates were recorded on {DEFAULT_PRICE_TABLE_AS_OF}. "
+        "Verify them against your provider's current pricing."
+    )
+
+
 @store_app.command("stats")
 def store_stats() -> None:
     """Show local store row counts."""

agentprof/config.py

@@ -1,10 +1,11 @@
 from __future__ import annotations
 
+from decimal import Decimal, InvalidOperation
 from pathlib import Path
 from typing import Any
 
 import yaml
-from pydantic import BaseModel, Field
+from pydantic import BaseModel, Field, field_validator
 
 
 APP_DIR = Path(".agentprof")
@@ -43,6 +44,14 @@
   spec_violations:
     contracts: []
 
+pricing:
+  # Estimated costs are only used for spans that arrive without provider cost
+  # fields. Verify rates against your provider's current pricing before you
+  # rely on estimated totals.
+  enabled: true
+  use_default_table: true
+  models: []
+
 store:
   path: .agentprof/data/agentprof.duckdb
 """
@@ -104,6 +113,30 @@ class AnalyzersConfig(BaseModel):
     )
 
 
+class ModelPriceConfig(BaseModel):
+    """Per-million-token rates used to estimate span cost for one model."""
+
+    model: str = Field(min_length=1)
+    input_per_1m_usd: Decimal = Field(ge=0)
+    output_per_1m_usd: Decimal = Field(ge=0)
+
+    @field_validator("input_per_1m_usd", "output_per_1m_usd", mode="before")
+    @classmethod
+    def _exact_decimal(cls, value: Any) -> Any:
+        if isinstance(value, Decimal) or value is None:
+            return value
+        try:
+            return Decimal(str(value))
+        except InvalidOperation as exc:
+            raise ValueError(f"{value!r} is not a valid price") from exc
+
+
+class PricingConfig(BaseModel):
+    enabled: bool = True
+    use_default_table: bool = True
+    models: list[ModelPriceConfig] = Field(default_factory=list)
+
+
 class StoreConfig(BaseModel):
     path: Path = DEFAULT_STORE_PATH
 
@@ -113,6 +146,7 @@ class AgentProfConfig(BaseModel):
     privacy: PrivacyConfig = Field(default_factory=PrivacyConfig)
     sources: SourcesConfig = Field(default_factory=SourcesConfig)
     analyzers: AnalyzersConfig = Field(default_factory=AnalyzersConfig)
+    pricing: PricingConfig = Field(default_factory=PricingConfig)
     store: StoreConfig = Field(default_factory=StoreConfig)