The bundled Killer Demo is deterministic and uses declared action costs. It explains the mechanism; it is not provider telemetry or a universal savings claim.
-
Public evaluation requires matched model, prompt, tools, limits and verifier, then reports tokens per verified successful task, quality delta and uncertainty.
-
Read the benchmark protocol →
+
+
Next milestone
Codex Reference Integration
The core is being prepared now; the adapter and measured Codex runs remain the next implementation milestone.
+
+
v0.2Learning Loop Foundationuniversal protocol · ledger · privacy · replay
+
HardeningNet-value evidence layergovernance tax · false stops · diminishing returns
+
v0.3Codex integrationone-command target · telemetry · canary · public matched benchmark
+
-
Build economically disciplined agents
-
Observe first. Measure honestly. Enforce what the evidence supports.
+
Build the evidence first
+
Observe. Measure. Let intervention earn enforcement.
diff --git a/site/styles.css b/site/styles.css
index 692786a..833a460 100644
--- a/site/styles.css
+++ b/site/styles.css
@@ -1,19 +1,144 @@
-:root{--bg:#080b10;--surface:#10151d;--surface2:#151c26;--text:#f4f7fb;--muted:#aab5c4;--line:#273140;--accent:#91ff63;--max:1180px}
-*{box-sizing:border-box}html{scroll-behavior:smooth}body{margin:0;background:var(--bg);color:var(--text);font-family:Inter,ui-sans-serif,system-ui,-apple-system,BlinkMacSystemFont,"Segoe UI",sans-serif;line-height:1.65}a{color:inherit}
-.skip-link{position:absolute;left:-9999px}.skip-link:focus{left:1rem;top:1rem;z-index:100;background:var(--text);color:var(--bg);padding:.75rem 1rem}
-.shell{width:min(calc(100% - 2rem),var(--max));margin-inline:auto}.narrow{max-width:790px}.center{text-align:center;justify-content:center}
-.site-header{position:sticky;top:0;z-index:20;background:rgba(8,11,16,.9);border-bottom:1px solid var(--line);backdrop-filter:blur(16px)}
-.nav{min-height:72px;display:flex;align-items:center;justify-content:space-between;gap:2rem}.brand{display:inline-flex;align-items:center;gap:.7rem;text-decoration:none;font-weight:800;letter-spacing:.08em}.brand-mark{display:grid;place-items:center;width:34px;height:34px;color:var(--bg);background:var(--accent);border-radius:8px}
-.nav-links{display:flex;align-items:center;gap:1.35rem}.nav-links a{text-decoration:none;color:var(--muted);font-size:.94rem}.nav-links a:hover,.nav-links a:focus{color:var(--text)}.nav-toggle{display:none}
-.button{display:inline-flex;align-items:center;justify-content:center;min-height:48px;padding:.7rem 1.15rem;border-radius:10px;background:var(--accent);color:#071006;text-decoration:none;font-weight:750;border:1px solid var(--accent)}.button:hover,.button:focus{filter:brightness(1.08);transform:translateY(-1px)}.button-secondary{background:transparent;color:var(--text);border-color:var(--line)}.button-small{min-height:38px;padding:.45rem .8rem;color:#071006!important}
-.hero{min-height:720px;display:grid;grid-template-columns:1.25fr .75fr;align-items:center;gap:4rem;padding-block:7rem 5rem}.eyebrow{color:var(--accent);text-transform:uppercase;letter-spacing:.14em;font-size:.78rem;font-weight:800}
-h1{font-size:clamp(3rem,7vw,6.4rem);line-height:.96;letter-spacing:-.055em;margin:.5rem 0 1.4rem;max-width:900px}h2{font-size:clamp(2rem,4.2vw,4rem);line-height:1.05;letter-spacing:-.04em;margin:.45rem 0 1.2rem}h3{line-height:1.2}.hero-lead{color:var(--muted);font-size:clamp(1.1rem,2vw,1.35rem);max-width:720px}.hero-actions{display:flex;gap:.8rem;margin-top:2rem;flex-wrap:wrap}.trust-row{display:flex;flex-wrap:wrap;gap:.7rem 1.25rem;list-style:none;padding:0;margin:2.2rem 0 0;color:var(--muted);font-size:.9rem}.trust-row li:before{content:"✓";color:var(--accent);margin-right:.4rem}
-.hero-panel{border:1px solid var(--line);background:linear-gradient(160deg,var(--surface2),var(--surface));padding:2rem;border-radius:20px;box-shadow:0 30px 80px rgba(0,0,0,.35)}.flow-node{padding:1rem;border:1px solid var(--line);border-radius:10px;text-align:center;background:rgba(255,255,255,.02)}.flow-node.accent{border-color:var(--accent)}.flow-arrow{text-align:center;color:var(--accent);padding:.4rem}.decision-grid{display:grid;grid-template-columns:repeat(3,1fr);gap:.5rem}.decision-grid span{border:1px solid var(--line);border-radius:8px;padding:.7rem .3rem;text-align:center;font-size:.75rem;font-weight:800}
-.section{padding-block:6.5rem;border-top:1px solid var(--line)}.problem,.feature-section,.privacy-section,.evidence-section{background:var(--surface)}.problem{text-align:center}.problem p:last-child{color:var(--muted)}.section-heading{max-width:760px;margin-bottom:3rem}
-.steps{display:grid;grid-template-columns:repeat(5,1fr);gap:1rem}.steps article,.feature-grid article,.roadmap-grid article{border:1px solid var(--line);border-radius:14px;padding:1.35rem;background:var(--surface)}.steps span,.roadmap-grid span{color:var(--accent);font-size:.8rem;font-weight:800;text-transform:uppercase;letter-spacing:.1em}.steps p,.feature-grid p,.roadmap-grid p{color:var(--muted);font-size:.94rem}
-.feature-grid{display:grid;grid-template-columns:repeat(3,1fr);gap:1rem}.split{display:grid;grid-template-columns:1fr 1fr;gap:5rem;align-items:start}.loop-list{list-style:none;padding:0;margin:0;counter-reset:loop}.loop-list li{counter-increment:loop;padding:1rem 0;border-bottom:1px solid var(--line);font-weight:700}.loop-list li:before{content:"0" counter(loop);color:var(--accent);margin-right:1rem;font-size:.8rem}.fine-print{color:var(--muted);font-size:.9rem}
-.privacy-cards{display:grid;gap:.8rem}.privacy-cards article{display:grid;gap:.2rem;padding:1.2rem;border:1px solid var(--line);border-radius:12px}.privacy-cards strong{color:var(--accent)}.privacy-cards span{color:var(--muted)}.roadmap-grid{display:grid;grid-template-columns:repeat(5,1fr);gap:1rem;margin-bottom:2.5rem}.roadmap-grid .complete{border-color:var(--accent)}.text-link{color:var(--accent);font-weight:750;text-decoration:none}.cta{background:radial-gradient(circle at 50% 20%,rgba(145,255,99,.13),transparent 45%)}
-footer{padding-block:3rem;border-top:1px solid var(--line);color:var(--muted)}.footer-grid{display:grid;grid-template-columns:2fr 1fr 1fr;gap:2rem}.footer-grid div{display:flex;flex-direction:column;gap:.4rem;align-items:flex-start}.footer-grid p{margin:0}.footer-grid a{text-decoration:none}
-@media(max-width:900px){.hero,.split{grid-template-columns:1fr}.hero{min-height:auto;gap:2.5rem;padding-top:5rem}.steps{grid-template-columns:repeat(2,1fr)}.feature-grid{grid-template-columns:repeat(2,1fr)}.roadmap-grid{grid-template-columns:repeat(2,1fr)}.nav-toggle{display:inline-flex;background:transparent;border:1px solid var(--line);color:var(--text);padding:.55rem .75rem;border-radius:8px}.nav-links{display:none;position:absolute;left:1rem;right:1rem;top:72px;background:var(--surface);border:1px solid var(--line);border-radius:12px;padding:1rem;flex-direction:column;align-items:stretch}.nav-links.open{display:flex}}
-@media(max-width:600px){h1{font-size:3.2rem}.section{padding-block:4.5rem}.steps,.feature-grid,.roadmap-grid,.footer-grid{grid-template-columns:1fr}.decision-grid{grid-template-columns:1fr}}
-@media(prefers-reduced-motion:reduce){html{scroll-behavior:auto}*,*:before,*:after{transition:none!important;animation:none!important}}
+:root {
+ --bg: #080b10;
+ --surface: #10151d;
+ --surface-2: #151c26;
+ --surface-3: #0c1118;
+ --text: #f4f7fb;
+ --muted: #aab5c4;
+ --line: #273140;
+ --accent: #91ff63;
+ --accent-soft: rgba(145, 255, 99, 0.09);
+ --warning: #ffd166;
+ --danger: #ff7b72;
+ --max: 1180px;
+}
+
+* { box-sizing: border-box; }
+html { scroll-behavior: smooth; }
+body {
+ margin: 0;
+ background: var(--bg);
+ color: var(--text);
+ font-family: Inter, ui-sans-serif, system-ui, -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif;
+ line-height: 1.65;
+}
+a { color: inherit; }
+code { font-family: ui-monospace, SFMono-Regular, Menlo, Consolas, monospace; color: var(--accent); }
+.skip-link { position: absolute; left: -9999px; }
+.skip-link:focus { left: 1rem; top: 1rem; z-index: 100; background: var(--text); color: var(--bg); padding: .75rem 1rem; }
+.shell { width: min(calc(100% - 2rem), var(--max)); margin-inline: auto; }
+.narrow { max-width: 820px; }
+.center { text-align: center; justify-content: center; }
+
+.site-header { position: sticky; top: 0; z-index: 20; background: rgba(8,11,16,.9); border-bottom: 1px solid var(--line); backdrop-filter: blur(16px); }
+.nav { min-height: 72px; display: flex; align-items: center; justify-content: space-between; gap: 2rem; }
+.brand { display: inline-flex; align-items: center; gap: .7rem; text-decoration: none; font-weight: 800; letter-spacing: .08em; }
+.brand-mark { display: grid; place-items: center; width: 34px; height: 34px; color: var(--bg); background: var(--accent); border-radius: 8px; }
+.nav-links { display: flex; align-items: center; gap: 1.25rem; }
+.nav-links a { text-decoration: none; color: var(--muted); font-size: .92rem; }
+.nav-links a:hover, .nav-links a:focus { color: var(--text); }
+.nav-toggle { display: none; }
+
+.button { display: inline-flex; align-items: center; justify-content: center; min-height: 48px; padding: .7rem 1.15rem; border-radius: 10px; background: var(--accent); color: #071006; text-decoration: none; font-weight: 760; border: 1px solid var(--accent); }
+.button:hover, .button:focus { filter: brightness(1.08); transform: translateY(-1px); }
+.button-secondary { background: transparent; color: var(--text); border-color: var(--line); }
+.button-small { min-height: 38px; padding: .45rem .8rem; color: #071006 !important; }
+
+.hero { min-height: 760px; display: grid; grid-template-columns: 1.08fr .92fr; align-items: center; gap: 4.5rem; padding-block: 7rem 5rem; }
+.eyebrow { color: var(--accent); text-transform: uppercase; letter-spacing: .14em; font-size: .77rem; font-weight: 820; }
+h1 { font-size: clamp(3rem, 6.7vw, 6rem); line-height: .98; letter-spacing: -.055em; margin: .5rem 0 1.4rem; max-width: 900px; }
+h2 { font-size: clamp(2rem, 4.2vw, 4rem); line-height: 1.05; letter-spacing: -.04em; margin: .45rem 0 1.2rem; }
+h3 { line-height: 1.2; }
+.hero-lead { color: var(--muted); font-size: clamp(1.08rem, 2vw, 1.3rem); max-width: 720px; }
+.hero-actions { display: flex; gap: .8rem; margin-top: 2rem; flex-wrap: wrap; }
+.trust-row { display: flex; flex-wrap: wrap; gap: .7rem 1.25rem; list-style: none; padding: 0; margin: 2.2rem 0 0; color: var(--muted); font-size: .9rem; }
+.trust-row li::before { content: "✓"; color: var(--accent); margin-right: .4rem; }
+
+.trace-card { border: 1px solid var(--line); background: linear-gradient(160deg, var(--surface-2), var(--surface)); padding: 1.25rem; border-radius: 20px; box-shadow: 0 30px 80px rgba(0,0,0,.36); }
+.trace-label { display: flex; justify-content: space-between; gap: 1rem; color: var(--muted); font-size: .78rem; text-transform: uppercase; letter-spacing: .08em; margin-bottom: .9rem; }
+.trace-label strong { color: var(--warning); }
+.trace-row { display: grid; grid-template-columns: 34px 1fr auto; gap: .8rem; align-items: center; padding: 1rem .65rem; border-top: 1px solid var(--line); }
+.trace-num { color: var(--muted); font-family: ui-monospace, monospace; font-size: .78rem; }
+.trace-row div { display: grid; }
+.trace-row small { color: var(--muted); }
+.status { font-size: .68rem; font-weight: 850; letter-spacing: .08em; border-radius: 999px; padding: .28rem .5rem; border: 1px solid var(--line); }
+.status.ok { color: var(--accent); }
+.status.warn { color: var(--warning); }
+.status.stop { color: var(--danger); }
+.trace-note { color: var(--muted); font-size: .83rem; margin: .85rem .65rem .2rem; }
+
+.section { padding-block: 6.5rem; border-top: 1px solid var(--line); }
+.thesis, .community-section, .roadmap-section { background: var(--surface); }
+.dark-band { background: #06080c; }
+.section-heading { max-width: 790px; margin-bottom: 3rem; }
+.section-heading > p:last-child, .body-copy { color: var(--muted); }
+.body-copy strong { color: var(--text); }
+.split { display: grid; grid-template-columns: 1fr 1fr; gap: 5rem; align-items: start; }
+.compact-gap { gap: 3rem; }
+
+.metric-grid { display: grid; grid-template-columns: repeat(3, 1fr); gap: 1rem; }
+.metric-grid article, .decision-cards article { border: 1px solid var(--line); border-radius: 14px; padding: 1.35rem; background: var(--surface); }
+.metric-grid article > span:first-child { color: var(--accent); font-size: .75rem; font-weight: 850; letter-spacing: .1em; }
+.metric-grid p, .decision-cards p { color: var(--muted); font-size: .94rem; }
+.metric-grid .accent-card { border-color: var(--accent); background: linear-gradient(160deg, var(--accent-soft), var(--surface)); }
+.evidence-banner { margin-top: 1rem; border: 1px solid var(--line); border-radius: 14px; padding: 1.15rem 1.35rem; display: grid; grid-template-columns: auto 1fr; gap: 1.2rem; align-items: center; }
+.evidence-key { color: var(--accent); font-size: .75rem; font-weight: 850; text-transform: uppercase; letter-spacing: .1em; }
+
+.terminal { display: grid; gap: .4rem; margin: 1.5rem 0; border: 1px solid var(--line); border-radius: 12px; padding: 1.2rem; background: #05080c; font-family: ui-monospace, SFMono-Regular, Menlo, Consolas, monospace; }
+.terminal span, .terminal small { color: var(--muted); }
+.terminal b { color: var(--accent); }
+.fine-print { color: var(--muted); font-size: .88rem; }
+
+.logic-list { border-top: 1px solid var(--line); }
+.logic-list div { display: grid; grid-template-columns: 180px 1fr; gap: 1rem; border-bottom: 1px solid var(--line); padding: 1rem 0; }
+.logic-list span { color: var(--muted); }
+.text-link { color: var(--accent); font-weight: 760; text-decoration: none; }
+
+.decision-cards { display: grid; grid-template-columns: repeat(2, 1fr); gap: 1rem; }
+.decision { display: inline-flex; border-radius: 999px; padding: .25rem .55rem; font-size: .69rem; font-weight: 850; letter-spacing: .08em; text-transform: uppercase; border: 1px solid var(--line); }
+.decision.accept { color: var(--accent); }
+.decision.partial { color: var(--warning); }
+.decision.reject { color: var(--danger); }
+
+.proof-list { display: grid; grid-template-columns: repeat(2, 1fr); gap: .7rem; }
+.proof-list span { border: 1px solid var(--line); border-radius: 9px; padding: .8rem; color: var(--muted); }
+.proof-list span::before { content: "✓"; color: var(--accent); margin-right: .5rem; }
+
+.roadmap-line { display: grid; grid-template-columns: repeat(3, 1fr); gap: 1rem; margin-bottom: 2.5rem; }
+.roadmap-line div { min-height: 170px; display: flex; flex-direction: column; gap: .45rem; border: 1px solid var(--line); border-radius: 14px; padding: 1.35rem; }
+.roadmap-line span { color: var(--muted); font-size: .74rem; font-weight: 850; text-transform: uppercase; letter-spacing: .1em; }
+.roadmap-line small { color: var(--muted); }
+.roadmap-line .done { border-color: rgba(145,255,99,.4); }
+.roadmap-line .current { border-color: var(--accent); background: var(--accent-soft); }
+
+.cta { background: radial-gradient(circle at 50% 20%, rgba(145,255,99,.13), transparent 45%); }
+footer { padding-block: 3rem; border-top: 1px solid var(--line); color: var(--muted); }
+.footer-grid { display: grid; grid-template-columns: 2fr 1fr 1fr; gap: 2rem; }
+.footer-grid div { display: flex; flex-direction: column; gap: .4rem; align-items: flex-start; }
+.footer-grid p { margin: 0; }
+.footer-grid a { text-decoration: none; }
+
+@media (max-width: 900px) {
+ .hero, .split { grid-template-columns: 1fr; }
+ .hero { min-height: auto; gap: 2.5rem; padding-top: 5rem; }
+ .metric-grid { grid-template-columns: repeat(2, 1fr); }
+ .roadmap-line { grid-template-columns: 1fr; }
+ .nav-toggle { display: inline-flex; background: transparent; border: 1px solid var(--line); color: var(--text); padding: .55rem .75rem; border-radius: 8px; }
+ .nav-links { display: none; position: absolute; left: 1rem; right: 1rem; top: 72px; background: var(--surface); border: 1px solid var(--line); border-radius: 12px; padding: 1rem; flex-direction: column; align-items: stretch; }
+ .nav-links.open { display: flex; }
+}
+
+@media (max-width: 620px) {
+ h1 { font-size: 3.1rem; }
+ .section { padding-block: 4.5rem; }
+ .metric-grid, .decision-cards, .footer-grid, .proof-list { grid-template-columns: 1fr; }
+ .trace-row { grid-template-columns: 28px 1fr; }
+ .trace-row .status { grid-column: 2; justify-self: start; }
+ .logic-list div { grid-template-columns: 1fr; gap: .25rem; }
+ .evidence-banner { grid-template-columns: 1fr; }
+}
+
+@media (prefers-reduced-motion: reduce) {
+ html { scroll-behavior: auto; }
+ *, *::before, *::after { transition: none !important; animation: none !important; }
+}
diff --git a/src/marginal/__init__.py b/src/marginal/__init__.py
index e393076..ef38b5c 100644
--- a/src/marginal/__init__.py
+++ b/src/marginal/__init__.py
@@ -12,6 +12,12 @@
funded_call,
)
from .budget import BudgetExceeded, BudgetLedger, BudgetLimits, BudgetOverrun, BudgetUsage
+from .controls import (
+ DiminishingReturnConfig,
+ DiminishingReturnDetector,
+ DiminishingReturnSignal,
+ GovernanceTracker,
+)
from .estimator import EstimatorIdentity, ValueEstimate, ValueEstimator
from .killer_demo import run_killer_demo
from .ledger import (
@@ -82,10 +88,14 @@
"Decision",
"DecisionLedgerContext",
"DeduplicationScope",
+ "DiminishingReturnConfig",
+ "DiminishingReturnDetector",
+ "DiminishingReturnSignal",
"EstimatorIdentity",
"EstimatorRegistry",
"ExecutionMode",
"FailureUsageExtractor",
+ "GovernanceTracker",
"JsonlDecisionLedger",
"JsonlTraceSink",
"LocalPseudonymizer",
diff --git a/src/marginal/cli.py b/src/marginal/cli.py
index 97ad644..456ca81 100644
--- a/src/marginal/cli.py
+++ b/src/marginal/cli.py
@@ -128,6 +128,18 @@ def _build_parser() -> argparse.ArgumentParser:
public_eval.add_argument("--bootstrap-samples", type=int, default=2_000)
public_eval.add_argument("--confidence-level", type=float, default=0.95)
public_eval.add_argument("--quality-margin-pp", type=float, default=1.0)
+ public_eval.add_argument(
+ "--minimum-net-token-savings-percent",
+ type=float,
+ default=0.0,
+ help="minimum net token saving required before intervention is classified supported",
+ )
+ public_eval.add_argument(
+ "--max-false-stop-rate",
+ type=float,
+ default=0.0,
+ help="maximum reviewed false-stop rate allowed for a supported intervention",
+ )
public_eval.add_argument("--seed", type=int, default=42)
return parser
@@ -209,6 +221,8 @@ def main(argv: Sequence[str] | None = None) -> int:
bootstrap_samples=args.bootstrap_samples,
confidence_level=args.confidence_level,
quality_margin_pp=args.quality_margin_pp,
+ minimum_net_token_savings_percent=(args.minimum_net_token_savings_percent),
+ max_false_stop_rate=args.max_false_stop_rate,
seed=args.seed,
)
except (OSError, ValueError) as exc:
diff --git a/src/marginal/controls/__init__.py b/src/marginal/controls/__init__.py
new file mode 100644
index 0000000..b341b57
--- /dev/null
+++ b/src/marginal/controls/__init__.py
@@ -0,0 +1,15 @@
+"""Optional controls that harden MARGINAL without coupling the core to one engine."""
+
+from .diminishing import (
+ DiminishingReturnConfig,
+ DiminishingReturnDetector,
+ DiminishingReturnSignal,
+)
+from .governance import GovernanceTracker
+
+__all__ = [
+ "DiminishingReturnConfig",
+ "DiminishingReturnDetector",
+ "DiminishingReturnSignal",
+ "GovernanceTracker",
+]
diff --git a/src/marginal/controls/diminishing.py b/src/marginal/controls/diminishing.py
new file mode 100644
index 0000000..5346c26
--- /dev/null
+++ b/src/marginal/controls/diminishing.py
@@ -0,0 +1,171 @@
+"""State-aware diminishing-return detection for repeated agent actions.
+
+The detector is intentionally provider neutral. It does not special-case a model, tool,
+or file type. A repeat only becomes less valuable when the same semantic action is proposed
+against the same observable state without new evidence.
+"""
+
+from __future__ import annotations
+
+import math
+from dataclasses import dataclass
+
+from ..models import Action
+
+
+@dataclass(frozen=True, slots=True)
+class DiminishingReturnConfig:
+ """Conservative controls for state-aware repetition."""
+
+ gain_decay: float = 0.5
+ max_same_state_repeats: int = 2
+
+ def __post_init__(self) -> None:
+ if isinstance(self.gain_decay, bool) or not isinstance(self.gain_decay, (int, float)):
+ raise TypeError("gain_decay must be a number")
+ decay = float(self.gain_decay)
+ if not math.isfinite(decay) or not 0.0 < decay <= 1.0:
+ raise ValueError("gain_decay must be finite and in (0, 1]")
+ if isinstance(self.max_same_state_repeats, bool) or not isinstance(
+ self.max_same_state_repeats, int
+ ):
+ raise TypeError("max_same_state_repeats must be an integer")
+ if self.max_same_state_repeats < 1:
+ raise ValueError("max_same_state_repeats must be at least 1")
+ object.__setattr__(self, "gain_decay", decay)
+
+
+@dataclass(frozen=True, slots=True)
+class DiminishingReturnSignal:
+ """Explain how prior same-state executions affect the next action."""
+
+ semantic_key: str
+ same_state_repeats: int
+ gain_multiplier: float
+ should_stop: bool
+ reason_code: str
+ reason: str
+
+
+@dataclass(slots=True)
+class _Observation:
+ state_hash: str
+ evidence_hash: str
+ executions_in_state: int
+
+
+class DiminishingReturnDetector:
+ """Track repeated semantic work without confusing proposal with execution.
+
+ ``evaluate`` is pure: it never advances history. Call ``observe`` only after an action
+ actually executes successfully. Missing state information fails open because MARGINAL
+ should not invent certainty it cannot observe.
+ """
+
+ def __init__(self, config: DiminishingReturnConfig | None = None) -> None:
+ self.config = config or DiminishingReturnConfig()
+ self._observations: dict[str, _Observation] = {}
+
+ def evaluate(self, action: Action) -> DiminishingReturnSignal:
+ semantic_key = self._semantic_key(action)
+ state_hash = self._metadata_text(action, "state_hash")
+ evidence_hash = self._metadata_text(action, "evidence_hash")
+
+ if not state_hash:
+ return DiminishingReturnSignal(
+ semantic_key=semantic_key,
+ same_state_repeats=0,
+ gain_multiplier=1.0,
+ should_stop=False,
+ reason_code="DIMINISHING_RETURN_UNOBSERVABLE",
+ reason="state is not observable; repetition control fails open",
+ )
+
+ previous = self._observations.get(semantic_key)
+ if (
+ previous is None
+ or previous.state_hash != state_hash
+ or (
+ evidence_hash
+ and (not previous.evidence_hash or previous.evidence_hash != evidence_hash)
+ )
+ ):
+ repeats = 0
+ else:
+ repeats = previous.executions_in_state
+
+ multiplier = self.config.gain_decay**repeats
+ should_stop = repeats >= self.config.max_same_state_repeats
+ if should_stop:
+ return DiminishingReturnSignal(
+ semantic_key=semantic_key,
+ same_state_repeats=repeats,
+ gain_multiplier=multiplier,
+ should_stop=True,
+ reason_code="DIMINISHING_RETURN_REJECTED",
+ reason=(
+ "same semantic action has already executed "
+ f"{repeats} time(s) against unchanged state without new evidence"
+ ),
+ )
+ if repeats:
+ return DiminishingReturnSignal(
+ semantic_key=semantic_key,
+ same_state_repeats=repeats,
+ gain_multiplier=multiplier,
+ should_stop=False,
+ reason_code="DIMINISHING_RETURN_DISCOUNTED",
+ reason=(
+ f"same-state repetition detected; expected gain discounted by {multiplier:.3f}"
+ ),
+ )
+ return DiminishingReturnSignal(
+ semantic_key=semantic_key,
+ same_state_repeats=0,
+ gain_multiplier=1.0,
+ should_stop=False,
+ reason_code="DIMINISHING_RETURN_CLEAR",
+ reason="new state or new evidence; no repetition penalty",
+ )
+
+ def observe(self, action: Action) -> None:
+ """Record one action only after the caller confirms it executed."""
+
+ semantic_key = self._semantic_key(action)
+ state_hash = self._metadata_text(action, "state_hash")
+ evidence_hash = self._metadata_text(action, "evidence_hash")
+ if not state_hash:
+ return
+
+ previous = self._observations.get(semantic_key)
+ same_state = previous is not None and previous.state_hash == state_hash
+ same_evidence = previous is not None and (
+ (not evidence_hash and not previous.evidence_hash)
+ or evidence_hash == previous.evidence_hash
+ )
+ executions = (
+ previous.executions_in_state + 1
+ if previous is not None and same_state and same_evidence
+ else 1
+ )
+ self._observations[semantic_key] = _Observation(
+ state_hash=state_hash,
+ evidence_hash=evidence_hash,
+ executions_in_state=executions,
+ )
+
+ def reset(self) -> None:
+ self._observations.clear()
+
+ @staticmethod
+ def _metadata_text(action: Action, key: str) -> str:
+ value = action.metadata.get(key, "")
+ return value.strip() if isinstance(value, str) else str(value).strip() if value else ""
+
+ @classmethod
+ def _semantic_key(cls, action: Action) -> str:
+ explicit = cls._metadata_text(action, "marginal_semantic_key")
+ if explicit:
+ return explicit
+ phase = cls._metadata_text(action, "phase")
+ return "|".join((action.kind.strip().lower(), action.name.strip().lower(), phase.lower()))
diff --git a/src/marginal/controls/governance.py b/src/marginal/controls/governance.py
new file mode 100644
index 0000000..ce5296f
--- /dev/null
+++ b/src/marginal/controls/governance.py
@@ -0,0 +1,77 @@
+"""First-class accounting for the cost and mistakes of MARGINAL itself."""
+
+from __future__ import annotations
+
+import math
+from dataclasses import dataclass
+from typing import Any
+
+
+@dataclass(slots=True)
+class GovernanceTracker:
+ """Account for governance overhead and explicitly reviewed false stops.
+
+ False stops are never inferred from task success. They are counted only when an external
+ reviewer or counterfactual process explicitly labels a denied recommendation as an action
+ that would have helped.
+ """
+
+ decisions: int = 0
+ decision_latency_ms: float = 0.0
+ external_tokens: int = 0
+ external_usd: float = 0.0
+ external_latency_ms: float = 0.0
+ reviewed_stops: int = 0
+ false_stops: int = 0
+
+ def record_decision(self, *, latency_ms: float = 0.0) -> None:
+ self._validate_non_negative_number("latency_ms", latency_ms)
+ self.decisions += 1
+ self.decision_latency_ms += float(latency_ms)
+
+ def record_external_overhead(
+ self,
+ *,
+ tokens: int = 0,
+ usd: float = 0.0,
+ latency_ms: float = 0.0,
+ ) -> None:
+ if isinstance(tokens, bool) or not isinstance(tokens, int):
+ raise TypeError("tokens must be an integer")
+ if tokens < 0:
+ raise ValueError("tokens must be non-negative")
+ self._validate_non_negative_number("usd", usd)
+ self._validate_non_negative_number("latency_ms", latency_ms)
+ self.external_tokens += tokens
+ self.external_usd += float(usd)
+ self.external_latency_ms += float(latency_ms)
+
+ def record_stop_review(self, *, would_have_helped: bool) -> None:
+ if not isinstance(would_have_helped, bool):
+ raise TypeError("would_have_helped must be a boolean")
+ self.reviewed_stops += 1
+ if would_have_helped:
+ self.false_stops += 1
+
+ def summary(self) -> dict[str, Any]:
+ false_stop_rate = self.false_stops / self.reviewed_stops if self.reviewed_stops else None
+ total_latency = self.decision_latency_ms + self.external_latency_ms
+ return {
+ "scope": "treasury_tree",
+ "decisions": self.decisions,
+ "decision_latency_ms": round(self.decision_latency_ms, 6),
+ "external_tokens": self.external_tokens,
+ "external_usd": round(self.external_usd, 9),
+ "external_latency_ms": round(self.external_latency_ms, 6),
+ "total_latency_ms": round(total_latency, 6),
+ "reviewed_stops": self.reviewed_stops,
+ "false_stops": self.false_stops,
+ "false_stop_rate": round(false_stop_rate, 6) if false_stop_rate is not None else None,
+ }
+
+ @staticmethod
+ def _validate_non_negative_number(name: str, value: float) -> None:
+ if isinstance(value, bool) or not isinstance(value, (int, float)):
+ raise TypeError(f"{name} must be a number")
+ if not math.isfinite(float(value)) or value < 0:
+ raise ValueError(f"{name} must be finite and non-negative")
diff --git a/src/marginal/killer_demo.py b/src/marginal/killer_demo.py
index 2eab027..d67fd1b 100644
--- a/src/marginal/killer_demo.py
+++ b/src/marginal/killer_demo.py
@@ -340,6 +340,7 @@ def run_killer_demo(output_dir: str | Path | None = None) -> dict[str, Any]:
policy=policy,
trace_sink=trace,
name="killer-demo",
+ clock=lambda: 0,
)
stage_results: list[dict[str, Any]] = []
@@ -1959,7 +1960,7 @@ def render_killer_demo_html(result: dict[str, Any]) -> str:
Results
Trace
Benchmark
diff --git a/src/marginal/policy.py b/src/marginal/policy.py
index 009dbf3..a628e15 100644
--- a/src/marginal/policy.py
+++ b/src/marginal/policy.py
@@ -8,6 +8,7 @@
from dataclasses import asdict, dataclass
from .budget import BudgetLedger
+from .controls import DiminishingReturnDetector, DiminishingReturnSignal
from .estimator import EstimatorIdentity, ValueEstimate, ValueEstimator
from .models import Action, Decision
@@ -65,7 +66,12 @@ def __post_init__(self) -> None:
class MarginalPolicy:
- """Authorize actions only when expected marginal value justifies total cost."""
+ """Authorize actions only when expected marginal value justifies total cost.
+
+ State-aware diminishing-return control is opt-in. This preserves v0.2 behavior while
+ allowing engine adapters to enable repetition control first in Shadow/Recommend mode and
+ promote it to enforcement only after measured validation.
+ """
def __init__(
self,
@@ -74,9 +80,11 @@ def __init__(
*,
name: str = "marginal-reference",
version: str = "2.0.0",
+ diminishing_detector: DiminishingReturnDetector | None = None,
) -> None:
self.config = config or PolicyConfig()
self.estimator = estimator or ValueEstimator()
+ self.diminishing_detector = diminishing_detector
if not isinstance(name, str):
raise TypeError("name must be a string")
if not name.strip():
@@ -96,9 +104,24 @@ def __init__(
self._executed_fingerprints: set[str] = set()
def mark_executed(self, fingerprint: str) -> None:
+ """Backward-compatible exact duplicate accounting."""
+
if fingerprint:
self._executed_fingerprints.add(fingerprint)
+ def observe_execution(self, action: Action) -> None:
+ """Record one successfully executed action for exact and semantic repetition control."""
+
+ if action.fingerprint:
+ self.mark_executed(action.fingerprint)
+ if self.diminishing_detector is not None:
+ self.diminishing_detector.observe(action)
+
+ def diminishing_signal(self, action: Action) -> DiminishingReturnSignal | None:
+ if self.diminishing_detector is None:
+ return None
+ return self.diminishing_detector.evaluate(action)
+
def evaluate(self, action: Action, ledger: BudgetLedger) -> Decision:
if action.current_success_probability >= self.config.target_success_probability:
return self._decision(
@@ -117,12 +140,21 @@ def evaluate(self, action: Action, ledger: BudgetLedger) -> Decision:
"BUDGET_REJECTED",
)
+ diminishing = self.diminishing_signal(action)
+ if diminishing is not None and diminishing.should_stop:
+ return self._decision(
+ False,
+ f"rejected: {diminishing.reason}",
+ diminishing.reason_code,
+ )
+
estimate = self._estimate(action)
+ gain_multiplier = diminishing.gain_multiplier if diminishing is not None else 1.0
remaining_probability = max(
0.0,
self.config.target_success_probability - action.current_success_probability,
)
- expected_gain = min(estimate.expected_gain, remaining_probability)
+ expected_gain = min(estimate.expected_gain * gain_multiplier, remaining_probability)
if expected_gain < self.config.minimum_expected_gain:
return self._decision(
False,
diff --git a/src/marginal/public_eval.py b/src/marginal/public_eval.py
index db75eb0..2596b12 100644
--- a/src/marginal/public_eval.py
+++ b/src/marginal/public_eval.py
@@ -7,12 +7,17 @@
import random
from dataclasses import dataclass
from pathlib import Path
-from typing import Any
+from typing import Any, cast
@dataclass(frozen=True, slots=True)
class RunRecord:
- """Measured result for one benchmark instance."""
+ """Measured result for one benchmark instance.
+
+ ``tokens/usd/latency_ms`` describe the agent workload. Governance overhead is stored
+ separately so reports can show both gross savings and net savings after MARGINAL's own
+ cost. Existing v0.2 JSONL rows remain valid because all new fields default to zero.
+ """
instance_id: str
resolved: bool
@@ -20,25 +25,54 @@ class RunRecord:
usd: float = 0.0
latency_ms: int = 0
tool_calls: int = 0
+ repeated_calls: int = 0
+ governance_tokens: int = 0
+ governance_usd: float = 0.0
+ governance_latency_ms: int = 0
+ reviewed_stops: int = 0
+ false_stops: int = 0
def __post_init__(self) -> None:
if not isinstance(self.instance_id, str) or not self.instance_id:
raise ValueError("instance_id must not be empty")
if not isinstance(self.resolved, bool):
raise TypeError("resolved must be a boolean")
- for name in ("tokens", "latency_ms", "tool_calls"):
+ for name in (
+ "tokens",
+ "latency_ms",
+ "tool_calls",
+ "repeated_calls",
+ "governance_tokens",
+ "governance_latency_ms",
+ "reviewed_stops",
+ "false_stops",
+ ):
value = getattr(self, name)
if isinstance(value, bool) or not isinstance(value, int):
raise TypeError(f"{name} must be an integer")
if value < 0:
raise ValueError("metrics must be non-negative")
- if isinstance(self.usd, bool) or not isinstance(self.usd, (int, float)):
- raise TypeError("usd must be a number")
- if not math.isfinite(float(self.usd)):
- raise ValueError("usd must be finite")
- if self.usd < 0:
- raise ValueError("metrics must be non-negative")
- object.__setattr__(self, "usd", float(self.usd))
+ for name in ("usd", "governance_usd"):
+ value = getattr(self, name)
+ if isinstance(value, bool) or not isinstance(value, (int, float)):
+ raise TypeError(f"{name} must be a number")
+ if not math.isfinite(float(value)) or value < 0:
+ raise ValueError("metrics must be finite and non-negative")
+ object.__setattr__(self, name, float(value))
+ if self.false_stops > self.reviewed_stops:
+ raise ValueError("false_stops cannot exceed reviewed_stops")
+
+ @property
+ def effective_tokens(self) -> int:
+ return self.tokens + self.governance_tokens
+
+ @property
+ def effective_usd(self) -> float:
+ return self.usd + self.governance_usd
+
+ @property
+ def effective_latency_ms(self) -> int:
+ return self.latency_ms + self.governance_latency_ms
def load_runs(path: Path) -> dict[str, RunRecord]:
@@ -60,6 +94,12 @@ def load_runs(path: Path) -> dict[str, RunRecord]:
usd=item.get("usd", 0.0),
latency_ms=item.get("latency_ms", 0),
tool_calls=item.get("tool_calls", 0),
+ repeated_calls=item.get("repeated_calls", 0),
+ governance_tokens=item.get("governance_tokens", 0),
+ governance_usd=item.get("governance_usd", 0.0),
+ governance_latency_ms=item.get("governance_latency_ms", 0),
+ reviewed_stops=item.get("reviewed_stops", 0),
+ false_stops=item.get("false_stops", 0),
)
except (KeyError, TypeError, ValueError, json.JSONDecodeError) as exc:
raise ValueError(f"invalid benchmark row on line {line_number}") from exc
@@ -74,14 +114,32 @@ def load_runs(path: Path) -> dict[str, RunRecord]:
def _aggregate(records: list[RunRecord]) -> dict[str, Any]:
tasks = len(records)
resolved = sum(record.resolved for record in records)
+ governance_tokens = sum(record.governance_tokens for record in records)
+ governance_usd = sum(record.governance_usd for record in records)
+ governance_latency_ms = sum(record.governance_latency_ms for record in records)
+ tokens = sum(record.tokens for record in records)
+ usd = sum(record.usd for record in records)
+ latency_ms = sum(record.latency_ms for record in records)
+ reviewed_stops = sum(record.reviewed_stops for record in records)
+ false_stops = sum(record.false_stops for record in records)
return {
"tasks": tasks,
"resolved": resolved,
"resolve_rate": resolved / tasks,
- "tokens": sum(record.tokens for record in records),
- "usd": round(sum(record.usd for record in records), 6),
- "latency_ms": sum(record.latency_ms for record in records),
+ "tokens": tokens,
+ "usd": round(usd, 6),
+ "latency_ms": latency_ms,
"tool_calls": sum(record.tool_calls for record in records),
+ "repeated_calls": sum(record.repeated_calls for record in records),
+ "governance_tokens": governance_tokens,
+ "governance_usd": round(governance_usd, 6),
+ "governance_latency_ms": governance_latency_ms,
+ "effective_tokens": tokens + governance_tokens,
+ "effective_usd": round(usd + governance_usd, 6),
+ "effective_latency_ms": latency_ms + governance_latency_ms,
+ "reviewed_stops": reviewed_stops,
+ "false_stops": false_stops,
+ "false_stop_rate": false_stops / reviewed_stops if reviewed_stops else None,
}
@@ -94,6 +152,8 @@ def _bootstrap_token_savings(
samples: int,
seed: int,
confidence_level: float,
+ *,
+ include_governance: bool,
) -> tuple[float, float]:
if samples <= 0:
raise ValueError("bootstrap_samples must be positive")
@@ -103,8 +163,12 @@ def _bootstrap_token_savings(
estimates: list[float] = []
for _ in range(samples):
draw = [pairs[rng.randrange(len(pairs))] for _ in pairs]
- baseline = sum(left.tokens for left, _ in draw)
- marginal = sum(right.tokens for _, right in draw)
+ if include_governance:
+ baseline = sum(left.effective_tokens for left, _ in draw)
+ marginal = sum(right.effective_tokens for _, right in draw)
+ else:
+ baseline = sum(left.tokens for left, _ in draw)
+ marginal = sum(right.tokens for _, right in draw)
estimates.append(_saving(float(baseline), float(marginal)))
estimates.sort()
tail = (1.0 - confidence_level) / 2.0
@@ -121,16 +185,37 @@ def compare_runs(
seed: int = 42,
confidence_level: float = 0.95,
quality_margin_pp: float = 1.0,
+ minimum_net_token_savings_percent: float = 0.0,
+ max_false_stop_rate: float = 0.0,
) -> dict[str, Any]:
- """Compare matched executions without imputing missing tasks."""
+ """Compare matched executions without imputing missing tasks.
+
+ The intervention earns a ``supported`` status only when quality is preserved, reviewed
+ false stops stay within the configured threshold, and net token savings after governance
+ overhead exceed the configured minimum. Otherwise MARGINAL should be treated as unsafe or
+ pass through rather than manufacturing a savings claim.
+ """
- if isinstance(quality_margin_pp, bool) or not isinstance(quality_margin_pp, (int, float)):
- raise TypeError("quality_margin_pp must be a number")
- if not math.isfinite(float(quality_margin_pp)) or quality_margin_pp < 0:
- raise ValueError("quality_margin_pp must be finite and non-negative")
+ for name, value in (
+ ("quality_margin_pp", quality_margin_pp),
+ ("minimum_net_token_savings_percent", minimum_net_token_savings_percent),
+ ("max_false_stop_rate", max_false_stop_rate),
+ ):
+ if isinstance(value, bool) or not isinstance(value, (int, float)):
+ raise TypeError(f"{name} must be a number")
+ if not math.isfinite(float(value)):
+ raise ValueError(f"{name} must be finite")
+ if quality_margin_pp < 0:
+ raise ValueError("quality_margin_pp must be non-negative")
+ if minimum_net_token_savings_percent < 0:
+ raise ValueError("minimum_net_token_savings_percent must be non-negative")
+ if not 0.0 <= max_false_stop_rate <= 1.0:
+ raise ValueError("max_false_stop_rate must be between 0 and 1")
if not 0.0 < confidence_level < 1.0:
raise ValueError("confidence_level must be between 0 and 1")
quality_margin_pp = float(quality_margin_pp)
+ minimum_net_token_savings_percent = float(minimum_net_token_savings_percent)
+ max_false_stop_rate = float(max_false_stop_rate)
if set(baseline) != set(marginal):
missing = sorted(set(baseline) ^ set(marginal))
@@ -144,46 +229,115 @@ def compare_runs(
(marginal_total["resolve_rate"] - baseline_total["resolve_rate"]) * 100.0,
2,
)
- token_ci = _bootstrap_token_savings(
- list(zip(baseline_rows, marginal_rows, strict=True)),
+ pairs = list(zip(baseline_rows, marginal_rows, strict=True))
+ net_token_ci = _bootstrap_token_savings(
+ pairs,
bootstrap_samples,
seed,
confidence_level,
+ include_governance=True,
+ )
+ gross_token_ci = _bootstrap_token_savings(
+ pairs,
+ bootstrap_samples,
+ seed,
+ confidence_level,
+ include_governance=False,
)
- def efficiency(total: dict[str, Any]) -> dict[str, float | None]:
+ def efficiency(total: dict[str, Any], *, include_governance: bool) -> dict[str, float | None]:
resolved = int(total["resolved"])
if resolved == 0:
return {"tokens_per_resolved": None, "usd_per_resolved": None}
+ token_key = "effective_tokens" if include_governance else "tokens"
+ usd_key = "effective_usd" if include_governance else "usd"
return {
- "tokens_per_resolved": round(float(total["tokens"]) / resolved, 6),
- "usd_per_resolved": round(float(total["usd"]) / resolved, 6),
+ "tokens_per_resolved": round(float(total[token_key]) / resolved, 6),
+ "usd_per_resolved": round(float(total[usd_key]) / resolved, 6),
}
+ gross_savings = {
+ "tokens_percent": _saving(float(baseline_total["tokens"]), float(marginal_total["tokens"])),
+ "usd_percent": _saving(float(baseline_total["usd"]), float(marginal_total["usd"])),
+ "latency_percent": _saving(
+ float(baseline_total["latency_ms"]),
+ float(marginal_total["latency_ms"]),
+ ),
+ "tool_calls_percent": _saving(
+ float(baseline_total["tool_calls"]),
+ float(marginal_total["tool_calls"]),
+ ),
+ "repeated_calls_percent": _saving(
+ float(baseline_total["repeated_calls"]),
+ float(marginal_total["repeated_calls"]),
+ ),
+ "confidence_level": confidence_level,
+ "tokens_confidence_interval": list(gross_token_ci),
+ }
+ net_savings = {
+ "tokens_percent": _saving(
+ float(baseline_total["effective_tokens"]),
+ float(marginal_total["effective_tokens"]),
+ ),
+ "usd_percent": _saving(
+ float(baseline_total["effective_usd"]),
+ float(marginal_total["effective_usd"]),
+ ),
+ "latency_percent": _saving(
+ float(baseline_total["effective_latency_ms"]),
+ float(marginal_total["effective_latency_ms"]),
+ ),
+ "tool_calls_percent": gross_savings["tool_calls_percent"],
+ "repeated_calls_percent": gross_savings["repeated_calls_percent"],
+ "confidence_level": confidence_level,
+ "tokens_confidence_interval": list(net_token_ci),
+ "tokens_95pct_ci": list(net_token_ci),
+ }
+
+ quality_margin_value = float(quality_margin_pp)
+ quality_preserved = delta_pp >= -quality_margin_value
+ false_stop_rate = marginal_total["false_stop_rate"]
+ false_stops_acceptable = false_stop_rate is None or float(false_stop_rate) <= float(
+ max_false_stop_rate
+ )
+ if not quality_preserved:
+ intervention_status = "quality_regression"
+ elif not false_stops_acceptable:
+ intervention_status = "false_stop_risk"
+ elif float(cast(float, net_savings["tokens_percent"])) <= float(
+ minimum_net_token_savings_percent
+ ):
+ intervention_status = "pass_through"
+ else:
+ intervention_status = "supported"
+
return {
- "benchmark": "public-agent-benchmark-comparison-v2",
+ "benchmark": "public-agent-benchmark-comparison-v3",
"tasks": len(ids),
"baseline": baseline_total,
"marginal": marginal_total,
"efficiency": {
- "baseline": efficiency(baseline_total),
- "marginal": efficiency(marginal_total),
+ "baseline": efficiency(baseline_total, include_governance=True),
+ "marginal": efficiency(marginal_total, include_governance=True),
},
- "savings": {
- "tokens_percent": _saving(baseline_total["tokens"], marginal_total["tokens"]),
- "usd_percent": _saving(baseline_total["usd"], marginal_total["usd"]),
- "latency_percent": _saving(baseline_total["latency_ms"], marginal_total["latency_ms"]),
- "tool_calls_percent": _saving(
- baseline_total["tool_calls"], marginal_total["tool_calls"]
- ),
- "confidence_level": confidence_level,
- "tokens_confidence_interval": list(token_ci),
- "tokens_95pct_ci": list(token_ci),
+ "agent_only_efficiency": {
+ "baseline": efficiency(baseline_total, include_governance=False),
+ "marginal": efficiency(marginal_total, include_governance=False),
+ },
+ "gross_savings": gross_savings,
+ "net_savings": net_savings,
+ # Backward-compatible key. For rows without governance overhead it is numerically
+ # identical to v0.2. For new evidence it deliberately points to the net result.
+ "savings": net_savings,
+ "governance": {
+ "tokens": marginal_total["governance_tokens"],
+ "usd": marginal_total["governance_usd"],
+ "latency_ms": marginal_total["governance_latency_ms"],
},
"quality": {
"resolved_delta_pp": delta_pp,
"non_inferiority_margin_pp": quality_margin_pp,
- "preserved_within_margin": delta_pp >= -quality_margin_pp,
+ "preserved_within_margin": quality_preserved,
"preserved_within_one_pp": delta_pp >= -1.0,
"regressions": sum(
baseline[item].resolved and not marginal[item].resolved for item in ids
@@ -191,6 +345,17 @@ def efficiency(total: dict[str, Any]) -> dict[str, float | None]:
"recoveries": sum(
not baseline[item].resolved and marginal[item].resolved for item in ids
),
+ "reviewed_stops": marginal_total["reviewed_stops"],
+ "false_stops": marginal_total["false_stops"],
+ "false_stop_rate": false_stop_rate,
+ "max_false_stop_rate": max_false_stop_rate,
+ "false_stops_acceptable": false_stops_acceptable,
+ },
+ "intervention": {
+ "status": intervention_status,
+ "minimum_net_token_savings_percent": minimum_net_token_savings_percent,
+ "net_positive": float(cast(float, net_savings["tokens_percent"])) > 0.0,
+ "graceful_irrelevance": intervention_status == "pass_through",
},
}
@@ -203,10 +368,15 @@ def _format_optional_usd(value: float | None) -> str:
return "n/a" if value is None else f"${value:.6f}"
+def _format_optional_rate(value: float | None) -> str:
+ return "n/a" if value is None else f"{value * 100.0:.2f}%"
+
+
def render_public_report(result: dict[str, Any]) -> str:
baseline = result["baseline"]
marginal = result["marginal"]
savings = result["savings"]
+ gross = result.get("gross_savings", savings)
quality = result["quality"]
efficiency = result["efficiency"]
baseline_efficiency = efficiency["baseline"]
@@ -214,11 +384,14 @@ def render_public_report(result: dict[str, Any]) -> str:
ci = savings["tokens_confidence_interval"]
confidence_percent = float(savings["confidence_level"]) * 100.0
margin = float(quality["non_inferiority_margin_pp"])
+ intervention = result.get("intervention", {"status": "unclassified"})
+ governance = result.get("governance", {"tokens": 0, "usd": 0.0, "latency_ms": 0})
return "\n".join(
[
"# Measured public benchmark comparison",
"",
"This report compares matched executions. It does not estimate or impute missing runs.",
+ "MARGINAL overhead is counted in net efficiency and net savings.",
"",
"| Metric | Baseline | MARGINAL | Change |",
"|---|---:|---:|---:|",
@@ -228,22 +401,30 @@ def render_public_report(result: dict[str, Any]) -> str:
f"{quality['resolved_delta_pp']:+.2f} pp |"
),
(
- f"| Tokens | {baseline['tokens']:,} | "
- f"{marginal['tokens']:,} | {savings['tokens_percent']:.2f}% fewer |"
+ f"| Agent tokens | {baseline['tokens']:,} | "
+ f"{marginal['tokens']:,} | {gross['tokens_percent']:.2f}% fewer |"
),
(
- f"| USD | ${baseline['usd']:.4f} | "
- f"${marginal['usd']:.4f} | {savings['usd_percent']:.2f}% lower |"
+ f"| Effective tokens (incl. governance) | {baseline['effective_tokens']:,} | "
+ f"{marginal['effective_tokens']:,} | {savings['tokens_percent']:.2f}% fewer |"
),
(
- f"| Latency | {baseline['latency_ms']:,} ms | "
- f"{marginal['latency_ms']:,} ms | "
+ f"| Effective USD | ${baseline['effective_usd']:.4f} | "
+ f"${marginal['effective_usd']:.4f} | {savings['usd_percent']:.2f}% lower |"
+ ),
+ (
+ f"| Effective latency | {baseline['effective_latency_ms']:,} ms | "
+ f"{marginal['effective_latency_ms']:,} ms | "
f"{savings['latency_percent']:.2f}% lower |"
),
(
f"| Tool calls | {baseline['tool_calls']} | "
- f"{marginal['tool_calls']} | "
- f"{savings['tool_calls_percent']:.2f}% fewer |"
+ f"{marginal['tool_calls']} | {savings['tool_calls_percent']:.2f}% fewer |"
+ ),
+ (
+ f"| Repeated calls | {baseline['repeated_calls']} | "
+ f"{marginal['repeated_calls']} | "
+ f"{savings['repeated_calls_percent']:.2f}% fewer |"
),
(
"| Tokens per resolved task | "
@@ -256,8 +437,21 @@ def render_public_report(result: dict[str, Any]) -> str:
f"{_format_optional_usd(marginal_efficiency['usd_per_resolved'])} | — |"
),
"",
+ "## Governance tax",
+ "",
+ (
+ f"MARGINAL overhead: **{governance['tokens']:,} tokens**, "
+ f"**${governance['usd']:.6f}**, **{governance['latency_ms']:,} ms**."
+ ),
+ (
+ f"Gross agent-token savings: **{gross['tokens_percent']:.2f}%**. "
+ f"Net token savings after governance: **{savings['tokens_percent']:.2f}%**."
+ ),
+ "",
+ "## Quality and intervention decision",
+ "",
(
- f"Token savings {confidence_percent:.1f}% bootstrap interval: "
+ f"Net token savings {confidence_percent:.1f}% bootstrap interval: "
f"**{ci[0]:.2f}% to {ci[1]:.2f}%**."
),
(
@@ -265,6 +459,17 @@ def render_public_report(result: dict[str, Any]) -> str:
f"**{quality['preserved_within_margin']}**."
),
f"Regressions: **{quality['regressions']}**. Recoveries: **{quality['recoveries']}**.",
+ (
+ f"Reviewed deny recommendations: **{quality.get('reviewed_stops', 0)}**. "
+ f"False stops: **{quality.get('false_stops', 0)}** "
+ f"({_format_optional_rate(quality.get('false_stop_rate'))})."
+ ),
+ f"Intervention status: **{intervention['status']}**.",
+ "",
+ (
+ "`pass_through` is a valid result: it means MARGINAL did not demonstrate "
+ "enough net value to justify intervention under the preregistered threshold."
+ ),
"",
]
)
diff --git a/src/marginal/treasury.py b/src/marginal/treasury.py
index f120c30..bf948a0 100644
--- a/src/marginal/treasury.py
+++ b/src/marginal/treasury.py
@@ -3,11 +3,13 @@
from __future__ import annotations
import threading
-from collections.abc import Iterable
+import time
+from collections.abc import Callable, Iterable
from dataclasses import asdict, replace
from typing import Any
from .budget import BudgetLedger, BudgetLimits, BudgetOverrun, BudgetUsage
+from .controls import GovernanceTracker
from .fingerprint import fingerprint_action
from .models import Action, Allocation, Cost, Decision
from .modes import ExecutionMode
@@ -21,7 +23,11 @@ class AuthorizationRequired(RuntimeError):
class Treasury:
- """Allocate and account for agent compute as a scarce resource."""
+ """Allocate and account for agent compute as a scarce resource.
+
+ Governance overhead is measured separately from agent workload cost. Parent/child
+ treasuries share one tracker so the reported governance tax covers the full treasury tree.
+ """
def __init__(
self,
@@ -32,6 +38,8 @@ def __init__(
name: str = "root",
parent: Treasury | None = None,
mode: ExecutionMode | str = ExecutionMode.ENFORCE,
+ governance_tracker: GovernanceTracker | None = None,
+ clock: Callable[[], int] | None = None,
) -> None:
self.name = name
self.ledger = BudgetLedger(limits)
@@ -45,6 +53,10 @@ def __init__(
self._pending_semantics: dict[str, list[str]] = (
parent._pending_semantics if parent is not None else {}
)
+ self._clock: Callable[[], int] = clock or time.perf_counter_ns
+ self.governance: GovernanceTracker = (
+ parent.governance if parent is not None else governance_tracker or GovernanceTracker()
+ )
if parent is None:
self._reservation_counter = 0
self._approved_count = 0
@@ -55,6 +67,8 @@ def __init__(
self._failed_settled_count = 0
self._outcome_count = 0
self._observation_count = 0
+ self._recommended_denials: set[str] = set()
+ self._reviewed_denials: set[str] = set()
@property
def usage(self) -> BudgetUsage:
@@ -68,12 +82,12 @@ def propose(self, action: Action) -> Decision:
return self.authorize(action)
def evaluate(self, action: Action) -> Decision:
- """Evaluate an action without reserving resources or mutating counters."""
+ """Evaluate an action without reserving resources or mutating decision counters."""
prepared = self._prepare(action)
assert prepared.fingerprint is not None
with self._lock:
- return self._recommended_decision(prepared)
+ return self._timed_recommendation(prepared)
def is_authorized(self, action: Action) -> bool:
prepared = self._prepare(action)
@@ -95,7 +109,7 @@ def fund_best(self, actions: Iterable[Action]) -> Allocation | None:
for action in actions:
prepared = self._prepare(action)
assert prepared.fingerprint is not None
- decision = self._recommended_decision(prepared)
+ decision = self._timed_recommendation(prepared)
evaluated.append(
{
"action": action_payload(prepared),
@@ -134,7 +148,9 @@ def authorize(self, action: Action, *, apply_mode: bool = True) -> Decision:
prepared = self._prepare(action)
assert prepared.fingerprint is not None
with self._lock:
- recommended = self._recommended_decision(prepared)
+ recommended = self._timed_recommendation(prepared)
+ if not recommended.allowed:
+ self._recommended_denials.add(prepared.fingerprint)
decision = self._apply_mode(recommended) if apply_mode else recommended
reservation_action: Action | None = None
@@ -160,6 +176,7 @@ def authorize(self, action: Action, *, apply_mode: bool = True) -> Decision:
"decision": decision_payload(decision),
"usage": usage_payload(self.usage),
"reserved": usage_payload(self.ledger.reserved_usage),
+ "governance": self.governance.summary(),
}
)
except Exception:
@@ -222,7 +239,11 @@ def _settle(
violations.append(decision.reason)
if not failed:
- self.policy.mark_executed(prepared.fingerprint)
+ observe_execution = getattr(self.policy, "observe_execution", None)
+ if callable(observe_execution):
+ observe_execution(prepared)
+ else:
+ self.policy.mark_executed(prepared.fingerprint)
self._unregister_pending(prepared.fingerprint, reservation_fingerprint)
self._committed_count += 1
if failed:
@@ -238,6 +259,7 @@ def _settle(
"usage": usage_payload(self.usage),
"budget_overrun": bool(violations),
"violations": sorted(set(violations)),
+ "governance": self.governance.summary(),
}
if failed:
event["reason"] = failure_reason
@@ -271,6 +293,51 @@ def abort(self, action: Action, *, reason: str = "execution aborted") -> None:
}
)
+ def record_governance_overhead(
+ self,
+ *,
+ tokens: int = 0,
+ usd: float = 0.0,
+ latency_ms: float = 0.0,
+ ) -> None:
+ """Record overhead produced outside the local policy, such as an adapter-side model call."""
+
+ with self._lock:
+ self.governance.record_external_overhead(
+ tokens=tokens,
+ usd=usd,
+ latency_ms=latency_ms,
+ )
+
+ def record_stop_review(self, action: Action, *, would_have_helped: bool) -> None:
+ """Attach an explicit counterfactual label to a prior deny recommendation.
+
+ This method intentionally refuses to infer false stops from task success and rejects
+ duplicate labels for the same semantic action fingerprint.
+ """
+
+ prepared = self._prepare(action)
+ assert prepared.fingerprint is not None
+ with self._lock:
+ fingerprint = prepared.fingerprint
+ if fingerprint not in self._recommended_denials:
+ raise ValueError("action was not previously recommended for denial")
+ if fingerprint in self._reviewed_denials:
+ raise ValueError("deny recommendation has already been reviewed")
+ self.governance.record_stop_review(would_have_helped=would_have_helped)
+ self._reviewed_denials.add(fingerprint)
+ self.trace_sink.emit(
+ {
+ **self._identity_payload(),
+ "event": "counterfactual_review",
+ "treasury": self.name,
+ "action": action_payload(prepared),
+ "would_have_helped": would_have_helped,
+ "false_stop": would_have_helped,
+ "governance": self.governance.summary(),
+ }
+ )
+
def observe_value(self, action: Action, realized_gain: float) -> None:
"""Record explicit action-level realized gain for the configured estimator."""
@@ -337,8 +404,17 @@ def summary(self) -> dict[str, Any]:
"limits": asdict(self.limits),
"policy": self.policy.identity.to_dict(),
"estimator": self.policy.estimator_identity.to_dict(),
+ "governance": self.governance.summary(),
}
+ def _timed_recommendation(self, prepared: Action) -> Decision:
+ started = self._clock()
+ try:
+ return self._recommended_decision(prepared)
+ finally:
+ latency_ms = (self._clock() - started) / 1_000_000
+ self.governance.record_decision(latency_ms=latency_ms)
+
def _recommended_decision(self, prepared: Action) -> Decision:
assert prepared.fingerprint is not None
if self._pending_semantics.get(prepared.fingerprint):
diff --git a/tests/controls/test_diminishing.py b/tests/controls/test_diminishing.py
new file mode 100644
index 0000000..e6ff0d4
--- /dev/null
+++ b/tests/controls/test_diminishing.py
@@ -0,0 +1,73 @@
+from __future__ import annotations
+
+from marginal import Action, Cost, DiminishingReturnConfig, DiminishingReturnDetector
+
+
+def _action(*, state: str, evidence: str = "", fingerprint: str = "a") -> Action:
+ return Action(
+ name="verify the same file",
+ kind="verification",
+ cost=Cost(tokens=100),
+ expected_gain=0.4,
+ fingerprint=fingerprint,
+ metadata={
+ "phase": "verify",
+ "state_hash": state,
+ "evidence_hash": evidence,
+ "marginal_semantic_key": "verify:file:README.md",
+ },
+ )
+
+
+def test_diminishing_returns_discount_same_state_then_stop() -> None:
+ detector = DiminishingReturnDetector(
+ DiminishingReturnConfig(gain_decay=0.5, max_same_state_repeats=2)
+ )
+
+ first = detector.evaluate(_action(state="s1"))
+ assert first.gain_multiplier == 1.0
+ assert first.should_stop is False
+ detector.observe(_action(state="s1"))
+
+ second = detector.evaluate(_action(state="s1", fingerprint="b"))
+ assert second.same_state_repeats == 1
+ assert second.gain_multiplier == 0.5
+ assert second.should_stop is False
+ detector.observe(_action(state="s1", fingerprint="b"))
+
+ third = detector.evaluate(_action(state="s1", fingerprint="c"))
+ assert third.same_state_repeats == 2
+ assert third.gain_multiplier == 0.25
+ assert third.should_stop is True
+ assert third.reason_code == "DIMINISHING_RETURN_REJECTED"
+
+
+def test_new_state_resets_repetition_pressure() -> None:
+ detector = DiminishingReturnDetector()
+ detector.observe(_action(state="s1"))
+
+ signal = detector.evaluate(_action(state="s2", fingerprint="b"))
+
+ assert signal.same_state_repeats == 0
+ assert signal.gain_multiplier == 1.0
+ assert signal.should_stop is False
+
+
+def test_new_evidence_resets_repetition_pressure() -> None:
+ detector = DiminishingReturnDetector()
+ detector.observe(_action(state="s1", evidence="e1"))
+
+ signal = detector.evaluate(_action(state="s1", evidence="e2", fingerprint="b"))
+
+ assert signal.same_state_repeats == 0
+ assert signal.gain_multiplier == 1.0
+
+
+def test_missing_state_fails_open() -> None:
+ detector = DiminishingReturnDetector()
+
+ signal = detector.evaluate(_action(state=""))
+
+ assert signal.should_stop is False
+ assert signal.gain_multiplier == 1.0
+ assert signal.reason_code == "DIMINISHING_RETURN_UNOBSERVABLE"
diff --git a/tests/controls/test_governance.py b/tests/controls/test_governance.py
new file mode 100644
index 0000000..da6ad09
--- /dev/null
+++ b/tests/controls/test_governance.py
@@ -0,0 +1,36 @@
+from __future__ import annotations
+
+import pytest
+
+from marginal import GovernanceTracker
+
+
+def test_governance_tracker_separates_self_cost_from_agent_cost() -> None:
+ tracker = GovernanceTracker()
+ tracker.record_decision(latency_ms=1.25)
+ tracker.record_external_overhead(tokens=120, usd=0.002, latency_ms=50)
+
+ summary = tracker.summary()
+
+ assert summary["decisions"] == 1
+ assert summary["external_tokens"] == 120
+ assert summary["external_usd"] == 0.002
+ assert summary["total_latency_ms"] == 51.25
+
+
+def test_false_stop_rate_requires_explicit_reviews() -> None:
+ tracker = GovernanceTracker()
+ assert tracker.summary()["false_stop_rate"] is None
+
+ tracker.record_stop_review(would_have_helped=False)
+ tracker.record_stop_review(would_have_helped=True)
+
+ assert tracker.summary()["reviewed_stops"] == 2
+ assert tracker.summary()["false_stops"] == 1
+ assert tracker.summary()["false_stop_rate"] == 0.5
+
+
+def test_governance_tracker_rejects_invalid_overhead() -> None:
+ tracker = GovernanceTracker()
+ with pytest.raises(ValueError):
+ tracker.record_external_overhead(tokens=-1)
diff --git a/tests/controls/test_policy_diminishing.py b/tests/controls/test_policy_diminishing.py
new file mode 100644
index 0000000..4d24a5c
--- /dev/null
+++ b/tests/controls/test_policy_diminishing.py
@@ -0,0 +1,48 @@
+from __future__ import annotations
+
+from marginal import (
+ Action,
+ BudgetLedger,
+ BudgetLimits,
+ Cost,
+ DiminishingReturnConfig,
+ DiminishingReturnDetector,
+ MarginalPolicy,
+)
+
+
+def _retry(number: int) -> Action:
+ return Action(
+ name="verify README",
+ kind="verification",
+ cost=Cost(tokens=100),
+ expected_gain=0.4,
+ fingerprint=f"retry-{number}",
+ metadata={
+ "phase": "verify",
+ "state_hash": "workspace-unchanged",
+ "marginal_semantic_key": "verify:file:README.md",
+ },
+ )
+
+
+def test_policy_can_discount_and_reject_repeated_same_state_work() -> None:
+ policy = MarginalPolicy(
+ diminishing_detector=DiminishingReturnDetector(
+ DiminishingReturnConfig(gain_decay=0.5, max_same_state_repeats=2)
+ )
+ )
+ ledger = BudgetLedger(BudgetLimits(max_tokens=10_000))
+
+ first = policy.evaluate(_retry(1), ledger)
+ assert first.allowed is True
+ policy.observe_execution(_retry(1))
+
+ second = policy.evaluate(_retry(2), ledger)
+ assert second.allowed is True
+ assert second.expected_gain == 0.2
+ policy.observe_execution(_retry(2))
+
+ third = policy.evaluate(_retry(3), ledger)
+ assert third.allowed is False
+ assert third.reason_code == "DIMINISHING_RETURN_REJECTED"
diff --git a/tests/controls/test_treasury_governance.py b/tests/controls/test_treasury_governance.py
new file mode 100644
index 0000000..eb79082
--- /dev/null
+++ b/tests/controls/test_treasury_governance.py
@@ -0,0 +1,44 @@
+from __future__ import annotations
+
+import pytest
+
+from marginal import Action, BudgetLimits, Cost, Treasury
+
+
+def test_shadow_mode_can_review_a_false_stop_without_inferring_it() -> None:
+ treasury = Treasury(
+ BudgetLimits(max_tokens=10_000, max_usd=10.0),
+ mode="shadow",
+ )
+ action = Action(
+ name="expensive review",
+ kind="review",
+ cost=Cost(tokens=100, usd=1.0),
+ expected_gain=0.0,
+ fingerprint="review-1",
+ )
+
+ decision = treasury.authorize(action)
+ assert decision.allowed is True
+ assert decision.recommended is False
+ treasury.commit(action)
+ treasury.record_stop_review(action, would_have_helped=True)
+
+ governance = treasury.summary()["governance"]
+ assert governance["reviewed_stops"] == 1
+ assert governance["false_stops"] == 1
+ assert governance["false_stop_rate"] == 1.0
+
+
+def test_stop_review_rejects_actions_that_were_not_denied() -> None:
+ treasury = Treasury(BudgetLimits(max_tokens=10_000), mode="shadow")
+ action = Action(
+ name="free useful action",
+ kind="verification",
+ expected_gain=0.5,
+ fingerprint="useful-1",
+ )
+ treasury.authorize(action)
+
+ with pytest.raises(ValueError, match="not previously recommended"):
+ treasury.record_stop_review(action, would_have_helped=False)
diff --git a/tests/evaluation/test_cli_public_eval_governance.py b/tests/evaluation/test_cli_public_eval_governance.py
new file mode 100644
index 0000000..6fe79c4
--- /dev/null
+++ b/tests/evaluation/test_cli_public_eval_governance.py
@@ -0,0 +1,43 @@
+from __future__ import annotations
+
+import json
+from pathlib import Path
+
+from marginal.cli import main
+
+
+def _write(path: Path, row: dict[str, object]) -> None:
+ path.write_text(json.dumps(row) + "\n", encoding="utf-8")
+
+
+def test_public_eval_cli_exposes_net_value_gates(tmp_path: Path, capsys) -> None:
+ baseline = tmp_path / "baseline.jsonl"
+ marginal = tmp_path / "marginal.jsonl"
+ _write(baseline, {"instance_id": "task", "resolved": True, "tokens": 1000})
+ _write(
+ marginal,
+ {
+ "instance_id": "task",
+ "resolved": True,
+ "tokens": 850,
+ "governance_tokens": 100,
+ },
+ )
+
+ exit_code = main(
+ [
+ "public-eval",
+ str(baseline),
+ str(marginal),
+ "--bootstrap-samples",
+ "20",
+ "--minimum-net-token-savings-percent",
+ "10",
+ "--json",
+ ]
+ )
+
+ assert exit_code == 0
+ report = json.loads(capsys.readouterr().out)
+ assert report["net_savings"]["tokens_percent"] == 5.0
+ assert report["intervention"]["status"] == "pass_through"
diff --git a/tests/evaluation/test_public_eval_governance.py b/tests/evaluation/test_public_eval_governance.py
new file mode 100644
index 0000000..5e6775b
--- /dev/null
+++ b/tests/evaluation/test_public_eval_governance.py
@@ -0,0 +1,63 @@
+from __future__ import annotations
+
+from marginal.public_eval import RunRecord, compare_runs, render_public_report
+
+
+def test_public_eval_reports_gross_and_net_savings() -> None:
+ baseline = {"task": RunRecord(instance_id="task", resolved=True, tokens=1000, usd=1.0)}
+ marginal = {
+ "task": RunRecord(
+ instance_id="task",
+ resolved=True,
+ tokens=700,
+ usd=0.7,
+ governance_tokens=200,
+ governance_usd=0.2,
+ repeated_calls=1,
+ )
+ }
+
+ result = compare_runs(baseline, marginal, bootstrap_samples=20)
+
+ assert result["gross_savings"]["tokens_percent"] == 30.0
+ assert result["net_savings"]["tokens_percent"] == 10.0
+ assert result["savings"]["tokens_percent"] == 10.0
+ assert result["intervention"]["status"] == "supported"
+ assert "Governance tax" in render_public_report(result)
+
+
+def test_governance_tax_can_make_pass_through_the_correct_result() -> None:
+ baseline = {"task": RunRecord(instance_id="task", resolved=True, tokens=1000)}
+ marginal = {
+ "task": RunRecord(
+ instance_id="task",
+ resolved=True,
+ tokens=850,
+ governance_tokens=200,
+ )
+ }
+
+ result = compare_runs(baseline, marginal, bootstrap_samples=20)
+
+ assert result["gross_savings"]["tokens_percent"] == 15.0
+ assert result["net_savings"]["tokens_percent"] == -5.0
+ assert result["intervention"]["status"] == "pass_through"
+ assert result["intervention"]["graceful_irrelevance"] is True
+
+
+def test_reviewed_false_stop_can_fail_the_intervention_gate() -> None:
+ baseline = {"task": RunRecord(instance_id="task", resolved=True, tokens=1000)}
+ marginal = {
+ "task": RunRecord(
+ instance_id="task",
+ resolved=True,
+ tokens=500,
+ reviewed_stops=1,
+ false_stops=1,
+ )
+ }
+
+ result = compare_runs(baseline, marginal, bootstrap_samples=20)
+
+ assert result["quality"]["false_stop_rate"] == 1.0
+ assert result["intervention"]["status"] == "false_stop_risk"