Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,23 @@
All notable changes to Mendmark are documented here. The project follows
Semantic Versioning for its Python and JSON contracts.

## 0.7.1 - 2026-08-12

### Added

- Five generalized real-world outcome scenarios covering customer refunds,
employee offboarding, vendor bank-detail changes, production incident
remediation, and shipment exceptions.
- Explicit read-versus-side-effect tool metadata and pinned business safeguards
for money movement, access revocation, separation of duties, emergency
changes, and duplicate fulfillment.

### Changed

- The Enterprise Outcome Golden Set now contains eight workflows, 16 system
boundaries, 16 invariants, and 64 mutations; state-only assurance detects
32/64 while complete outcome assurance detects 64/64.

## 0.7.0 - 2026-08-12

### Added
Expand Down
16 changes: 9 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,8 +45,8 @@ mendmark demo
```

The command compares a conventional state-only evaluator with complete outcome
assurance across CRM + ticketing, ERP + payments, and HRIS + identity workflows.
It writes a reviewable JSON suite and both privacy-safe reports to
assurance across eight customer, finance, identity, operations, and fulfillment
workflows. It writes a reviewable JSON suite and both privacy-safe reports to
`mendmark-enterprise-demo/`.

Audit a reviewed outcome suite directly—without an evaluator framework or
Expand Down Expand Up @@ -96,7 +96,7 @@ Mendmark has dependency-light adapters for LangChain/LangGraph, CrewAI, and the
OpenAI Agents SDK. In an existing agent repository:

```bash
python -m pip install 'mendmark-evals==0.7.0'
python -m pip install 'mendmark-evals==0.7.1'
mendmark equip --framework auto --agent auto
```

Expand Down Expand Up @@ -150,9 +150,11 @@ contract model and decision rule.
## Agent Eval Golden Set

The [Enterprise Outcome Golden Set](golden/outcome-v1/) targets the business
surface directly: three common workflows, six system boundaries, six reviewed
invariants, and 24 high-importance mutations. Its state-only profile detects
12/24; the complete outcome-contract profile detects 24/24. Run it instantly
surface directly: eight common workflows, 16 system boundaries, 16 reviewed
invariants, and 64 high-importance mutations. It covers support escalation,
invoice approval, onboarding, refunds, offboarding, vendor bank changes,
incident remediation, and shipment exceptions. Its state-only profile detects
32/64; the complete outcome-contract profile detects 64/64. Run it instantly
with `mendmark demo`.

The [Mendmark Agent Eval Golden Set](golden/agent-eval-v1/) is the canonical,
Expand Down Expand Up @@ -427,7 +429,7 @@ and the [ML evaluation card](https://github.com/danielgaskins/mendmark/blob/main

## Current boundary

Version 0.6 is a local, open-source engine. It does not yet provide a hosted
Version 0.7 is a local, open-source engine. It does not yet provide a hosted
dashboard, team accounts, remote trace ingestion, or a secrets service. The
planned control plane is described in [the product design](https://github.com/danielgaskins/mendmark/blob/main/docs/product.md).

Expand Down
40 changes: 39 additions & 1 deletion benchmarks/benchmark_outcome_golden_set.py
Original file line number Diff line number Diff line change
Expand Up @@ -27,8 +27,33 @@ def main() -> int:
if _digest(root / "suite.json") != manifest["suite_sha256"]:
print("outcome golden set: suite digest mismatch", file=sys.stderr)
return 1
asset_paths = {
"README.md": root / "README.md",
"results.json": root / "results.json",
"suite-v1.schema.json": PROJECT_ROOT
/ "src"
/ "mendmark"
/ "schemas"
/ "suite-v1.schema.json",
"report-v1.schema.json": PROJECT_ROOT
/ "src"
/ "mendmark"
/ "schemas"
/ "report-v1.schema.json",
}
for name, expected_digest in manifest["assets"].items():
if _digest(asset_paths[name]) != expected_digest:
print(f"outcome golden set: {name} digest mismatch", file=sys.stderr)
return 1
with tempfile.TemporaryDirectory() as directory:
result = run_enterprise_demo(Path(directory))
generated_suite_digest = _digest(Path(directory) / "suite.json")
if generated_suite_digest != manifest["suite_sha256"]:
print(
"outcome golden set: generated suite differs from pinned corpus",
file=sys.stderr,
)
return 1
observed = {}
for source, target in (("state_only", "state-only"), ("protected", "outcome-contract")):
report = result[source]
Expand All @@ -37,11 +62,24 @@ def main() -> int:
**{key: report["summary"][key] for key in ("cases", "mutants", "killed", "survived", "kill_rate")},
"affected_workflows": report["business_assurance"]["affected_workflows"],
"estimated_exposure_usd": report["business_assurance"]["estimated_exposure_usd"],
"critical_survivors": report["summary"]["critical_survivors"],
}
if observed != expected:
print(f"outcome golden set: expected {expected!r}, got {observed!r}", file=sys.stderr)
return 1
print("outcome golden set: PASS (state-only 12/24; outcome-contract 24/24)")
protected = result["protected"]
operator_counts = {
name: coverage["mutants"]
for name, coverage in protected["coverage"]["by_operator"].items()
}
if operator_counts != manifest["operator_counts"]:
print(
f"outcome golden set: expected operator counts "
f"{manifest['operator_counts']!r}, got {operator_counts!r}",
file=sys.stderr,
)
return 1
print("outcome golden set: PASS (state-only 32/64; outcome-contract 64/64)")
return 0


Expand Down
9 changes: 5 additions & 4 deletions docs/benchmark.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,11 @@

## Enterprise Outcome Golden Set

The [Enterprise Outcome Golden Set](../golden/outcome-v1/) pins three common
business workflows spanning CRM/ticketing, ERP/payments, and HRIS/identity. It
contains six reviewed invariants and 24 outcome-first mutations. A state-only
profile kills 12/24; the complete outcome-contract evaluator kills 24/24.
The [Enterprise Outcome Golden Set](../golden/outcome-v1/) pins eight common
business workflows spanning CRM/ticketing, ERP/payments, HRIS/identity,
vendor-master controls, incident operations, and fulfillment. It contains 16
reviewed invariants and 64 outcome-first mutations. A state-only profile kills
32/64; the complete outcome-contract evaluator kills 64/64.

```bash
mendmark demo --output-dir outcome-review
Expand Down
2 changes: 1 addition & 1 deletion docs/harness-integrations.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ guide](https://openai.github.io/openai-agents-python/tracing/).
From the agent application repository:

```bash
python -m pip install 'mendmark-evals==0.7.0'
python -m pip install 'mendmark-evals==0.7.1'
mendmark equip --framework auto --agent auto
```

Expand Down
11 changes: 6 additions & 5 deletions docs/outcome-assurance.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,11 +53,12 @@ mendmark demo invoice-approval --output-dir demo-review
```

The scenarios use vendor-neutral shapes common to CRM/ticketing,
ERP/accounts-payable, and HRIS/identity systems. They are deterministic local
snapshots—not live connectors—so demos require no accounts, credentials, model
calls, or customer data. Replace the snapshot fields and tool names with exports
from Salesforce or Dynamics/HubSpot, ServiceNow/Jira/Zendesk, SAP/Oracle/NetSuite,
Workday, and Okta/Entra-style systems during a pilot.
ERP/accounts-payable, HRIS/identity, order/payment, vendor-master, monitoring,
and fulfillment systems. They are deterministic local snapshots—not live
connectors—so demos require no accounts, credentials, model calls, or customer
data. Replace the snapshot fields and tool names with exports from Salesforce or
Dynamics/HubSpot, ServiceNow/Jira/Zendesk, SAP/Oracle/NetSuite, Workday,
Okta/Entra, and comparable operational systems during a pilot.

The command writes `suite.json`, `state-only-report.json`, and
`outcome-assurance-report.json`. The contrast demonstrates precisely which
Expand Down
8 changes: 6 additions & 2 deletions docs/product.md
Original file line number Diff line number Diff line change
Expand Up @@ -97,8 +97,9 @@ do not require raw trace ingestion.
- Native single-agent traces and multi-agent causal event graphs.
- DeepEval suite adapter.
- Framework-neutral JSON suite and local batch evaluator protocol.
- Thirty-one built-in tool, response, delegation, authorization, shared-state,
aggregation, causality, and termination mutation operators.
- Thirty-six built-in outcome, invariant, efficiency, tool, response,
delegation, authorization, shared-state, aggregation, causality, and
termination mutation operators.
- Validated domain-specific mutation plugins.
- Per-tool coverage and tool schema tracking.
- Baseline regression gates.
Expand All @@ -112,6 +113,9 @@ do not require raw trace ingestion.
- Immutable Multi-Agent Golden Set v1 plus v2 with six topologies, 17 agent
declarations, 41 events, 294 mutations, weak/strong/permuted profiles, and
fully pinned graph-and-outcome behavior.
- Enterprise Outcome Golden Set with eight workflows, 16 system boundaries,
16 reviewed invariants, 64 high-importance mutations, and pinned state-only
versus complete outcome-assurance profiles.
- Enterprise assurance at 1,000 single-agent and 250 multi-agent cases with
enforced time, memory, report-size, JUnit, SARIF, and incremental-audit checks.
- A machine-validated, privacy-safe design-partner evidence rubric and utility
Expand Down
28 changes: 22 additions & 6 deletions golden/outcome-v1/README.md
Original file line number Diff line number Diff line change
@@ -1,22 +1,38 @@
# Mendmark Enterprise Outcome Golden Set

This golden dataset targets durable business results and high-consequence
safeguards rather than requiring one exact execution path. Its three reviewable
safeguards rather than requiring one exact execution path. Its eight reviewable
cases represent common enterprise workflow shapes:

- customer escalation across CRM and ticketing;
- invoice approval across ERP and payment scheduling; and
- employee onboarding across HRIS and identity provisioning.
- employee onboarding across HRIS and identity provisioning;
- approved customer refunds across order and payment systems;
- employee offboarding across HRIS and identity revocation;
- vendor bank-detail changes with payment holds and dual control;
- production incident remediation with authorized rollback; and
- shipment exceptions across carrier and customer-notification systems.

| Scenario | Required end state | High-importance safeguards | Represented consequence |
| --- | --- | --- | --- |
| Support escalation | Ticket resolved for an enterprise customer | Exactly one remedy; SLA protected | Incorrectly closed escalation |
| Invoice approval | Invoice approved and payment scheduled | No duplicate payment; authorized approval | Incorrect payment |
| Employee onboarding | Employment active and account provisioned | Least privilege; manager approval | Unauthorized access |
| Customer refund | Refund issued and request closed | Exactly one refund; amount within approval | Duplicate or excessive refund |
| Employee offboarding | Employment terminated and access revoked | No privileged sessions; legal hold preserved | Retained access or lost records |
| Vendor bank change | Change pending review and payments held | Dual control; requester cannot self-approve | Misdirected payment |
| Production incident | Incident contained and service restored | Verified rollback; authorized change | Extended outage or unsafe change |
| Shipment exception | Shipment rerouted and customer notified | At most one replacement; validated address | Duplicate fulfillment or misdelivery |

Each case declares expected state, two business invariants, a cost ceiling, a
latency ceiling, and report-safe consequence metadata. The five outcome-first
operators generate 24 pinned mutations: 12 missing or corrupt state changes,
six invariant violations, three cost overruns, and three latency overruns.
operators generate 64 pinned mutations: 32 missing or corrupt state changes,
16 invariant violations, eight cost overruns, and eight latency overruns.

| Evaluator | Killed | Survived | Result |
| --- | ---: | ---: | --- |
| State only | 12 | 12 | At risk |
| Outcome + invariants + budgets | 24 | 0 | Protected |
| State only | 32 | 32 | At risk |
| Outcome + invariants + budgets | 64 | 0 | Protected |

The corpus is deterministic, offline, vendor-neutral, and contains no customer
data. Review [suite.json](suite.json), the pinned [manifest](manifest.json), and
Expand Down
40 changes: 34 additions & 6 deletions golden/outcome-v1/manifest.json
Original file line number Diff line number Diff line change
Expand Up @@ -4,17 +4,45 @@
"version": "outcome-v1",
"license": "MIT",
"suite": "suite.json",
"suite_sha256": "ec3150e5fbc4639ff09454ce68a58e0999cde52eb04b73a216d48f6a3594a2de",
"suite_sha256": "2f146b2a73d34c2015bbcbcb4b039f94ef0f828632abc85995895b5d2c424fd6",
"assets": {
"README.md": "44ba11b6e924da5e52d2429a348b7d0b09e9ca4ff7db689bcaee0e2715ed02db",
"results.json": "0924bc0926b02d89dbf9326e66bdec52b4443e466dce4e2d4d98ce7dd9fc0b7d",
"suite-v1.schema.json": "ef18320bce140a7e05b5d40b245be9248c17ee0784ec66325651355234563735",
"report-v1.schema.json": "6bff193ec67b38d7970fdeff3ce931bd828fa737775b98ea40c02493d6607874"
},
"contents": {
"cases": 3,
"system_boundaries": 6,
"invariants": 6,
"mutations": 24,
"domains": ["customer-support", "finance", "identity"]
"cases": 8,
"system_boundaries": 16,
"invariants": 16,
"mutations": 64,
"domains": [
"customer-support",
"accounts-payable",
"identity-onboarding",
"refunds",
"identity-offboarding",
"vendor-risk",
"incident-operations",
"fulfillment"
],
"scenarios": [
"support-escalation",
"invoice-approval",
"employee-access",
"customer-refund",
"employee-offboarding",
"vendor-bank-change",
"production-incident",
"shipment-exception"
]
},
"operator_counts": {
"outcome.required_state_missing": 16,
"outcome.state_corrupted": 16,
"outcome.invariant_violated": 16,
"outcome.cost_budget_exceeded": 8,
"outcome.latency_budget_exceeded": 8
},
"methodology": {
"unit": "A passing workflow with reviewed outcome state, safeguards, and operating limits.",
Expand Down
22 changes: 12 additions & 10 deletions golden/outcome-v1/results.json
Original file line number Diff line number Diff line change
Expand Up @@ -3,23 +3,25 @@
"profiles": {
"state-only": {
"status": "at-risk",
"cases": 3,
"mutants": 24,
"killed": 12,
"survived": 12,
"cases": 8,
"mutants": 64,
"killed": 32,
"survived": 32,
"kill_rate": 0.5,
"affected_workflows": 3,
"estimated_exposure_usd": 85000
"affected_workflows": 8,
"estimated_exposure_usd": 517000,
"critical_survivors": 13
},
"outcome-contract": {
"status": "protected",
"cases": 3,
"mutants": 24,
"killed": 24,
"cases": 8,
"mutants": 64,
"killed": 64,
"survived": 0,
"kill_rate": 1.0,
"affected_workflows": 0,
"estimated_exposure_usd": 0
"estimated_exposure_usd": 0,
"critical_survivors": 0
}
}
}
Loading