-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathmendmark.html
More file actions
313 lines (301 loc) · 15 KB
/
Copy pathmendmark.html
File metadata and controls
313 lines (301 loc) · 15 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
<!doctype html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<meta
name="description"
content="Mendmark mutation-tests agent eval suites by planting controlled tool-use failures and checking whether the team's existing evaluators catch them."
/>
<meta name="theme-color" content="#0b0d0c" />
<meta property="og:title" content="Mendmark | Mutation Testing for Agent Evals" />
<meta
property="og:description"
content="An open-source Python package that plants controlled trace faults and shows which agent failures an eval suite misses."
/>
<meta property="og:type" content="article" />
<meta property="og:url" content="https://danielgaskins.com/mendmark.html" />
<meta property="og:site_name" content="Daniel Gaskins" />
<meta property="article:author" content="Daniel Gaskins" />
<meta property="article:published_time" content="2026-07-23" />
<meta property="article:modified_time" content="2026-08-04" />
<meta property="og:image" content="https://danielgaskins.com/assets/img/mendmark-og-v1.jpg" />
<meta property="og:image:secure_url" content="https://danielgaskins.com/assets/img/mendmark-og-v1.jpg" />
<meta property="og:image:width" content="1200" />
<meta property="og:image:height" content="630" />
<meta property="og:image:alt" content="A controlled fault being exposed inside an AI agent tool-call chain" />
<meta name="twitter:card" content="summary_large_image" />
<meta name="twitter:title" content="Mendmark | Mutation Testing for Agent Evals" />
<meta
name="twitter:description"
content="An open-source Python package that plants controlled trace faults and shows which agent failures an eval suite misses."
/>
<meta name="twitter:image" content="https://danielgaskins.com/assets/img/mendmark-og-v1.jpg" />
<meta name="twitter:image:alt" content="A controlled fault being exposed inside an AI agent tool-call chain" />
<link rel="canonical" href="https://danielgaskins.com/mendmark.html" />
<title>Mendmark | Mutation Testing for Agent Evals | Daniel Gaskins</title>
<link rel="icon" href="./favicon.svg" type="image/svg+xml" />
<link rel="preconnect" href="https://fonts.googleapis.com" />
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin />
<link
href="https://fonts.googleapis.com/css2?family=DM+Mono:wght@300;400;500&family=Manrope:wght@400;500;600;700&display=swap"
rel="stylesheet"
/>
<link rel="stylesheet" href="./styles.css" />
<script src="./script.js" defer></script>
<script type="application/ld+json">
{
"@context": "https://schema.org",
"@type": "SoftwareSourceCode",
"@id": "https://danielgaskins.com/mendmark.html#software",
"name": "Mendmark",
"description": "Open-source mutation testing for AI agent evaluation suites.",
"url": "https://danielgaskins.com/mendmark.html",
"codeRepository": "https://github.com/danielgaskins/mendmark",
"downloadUrl": "https://pypi.org/project/mendmark-evals/",
"programmingLanguage": "Python",
"runtimePlatform": "Python 3",
"license": "https://opensource.org/license/mit",
"version": "0.4.2",
"author": {"@id": "https://danielgaskins.com/#daniel-gaskins", "@type": "Person", "name": "Daniel Gaskins"}
}
</script>
</head>
<body>
<a class="skip-link" href="#main">Skip to content</a>
<header class="site-header" data-header>
<a class="wordmark" href="./" aria-label="Daniel Gaskins, home">
<span>DG</span>
<span class="wordmark__name">Daniel Gaskins</span>
</a>
<button class="nav-toggle" type="button" aria-expanded="false" aria-controls="site-nav">
<span></span><span></span>
<span class="sr-only">Toggle navigation</span>
</button>
<nav id="site-nav" class="site-nav" aria-label="Primary navigation">
<a href="./#work">Work</a>
<a href="./#capabilities">How I work</a>
<a href="./blog.html">Blog</a>
<a href="./#about">About</a>
<a href="./resume.html">Résumé</a>
<a class="nav-cta" href="mailto:hello@danielgaskins.com">Talk with me</a>
</nav>
</header>
<main id="main" class="case-study">
<section class="case-hero section-shell">
<p class="eyebrow"><span class="status-dot"></span> Open source / PyPI 0.4.2 / 2026</p>
<h1>Your agent passed the test. Would the test notice if it broke?</h1>
<p class="case-hero__lede">
Mendmark plants controlled faults in passing agent traces, then runs the same evals
again. A missed tool call, wrong argument, repeated side effect, or hidden error becomes
a concrete test gap instead of a surprise in production.
</p>
<div class="case-hero__actions">
<a class="button button--primary" href="https://pypi.org/project/mendmark-evals/" target="_blank" rel="noreferrer">Install from PyPI</a>
<a class="button button--quiet" href="https://github.com/danielgaskins/mendmark" target="_blank" rel="noreferrer">View source on GitHub</a>
</div>
</section>
<section class="case-signal" aria-label="Current project scope">
<div class="section-shell case-signal__grid">
<div><strong>24</strong><span>reviewable golden cases</span></div>
<div><strong>263</strong><span>pinned mutations</span></div>
<div><strong>10</strong><span>operating domains</span></div>
<div><strong>100%</strong><span>complete-profile kill rate</span></div>
</div>
</section>
<section id="method" class="case-section section-shell">
<div class="case-section__label">
<p class="section-index">01 / THE TEST OF THE TEST</p>
</div>
<div class="case-section__body">
<h2>Plant a fault. See if the eval catches it.</h2>
<p class="case-section__lead">
A passing case is the starting point. Mendmark changes one part of its output or tool
trace, then reruns the team’s existing evaluators. The mutation is killed when an eval
changes from pass to fail. If every eval still passes, Mendmark names the blind spot.
</p>
<div class="case-flow" aria-label="Mendmark task lifecycle">
<div><span>01</span><strong>Load</strong><small>passing cases + tool contracts</small></div>
<i>→</i>
<div><span>02</span><strong>Mutate</strong><small>one controlled fault at a time</small></div>
<i>→</i>
<div><span>03</span><strong>Rerun</strong><small>the same evaluator suite</small></div>
<i>→</i>
<div><span>04</span><strong>Gate</strong><small>block survivors and regressions</small></div>
</div>
<p>
The resulting kill rate measures the eval suite, not the agent. A surviving critical
fault is a specific evaluator problem the team can reproduce and fix.
</p>
</div>
</section>
<section class="case-section case-section--paper-dark">
<div class="section-shell case-section__inner">
<div class="case-section__label">
<p class="section-index">02 / PUBLIC GOLDEN SET</p>
</div>
<div class="case-section__body">
<h2>Three evaluators. The same 263 broken runs.</h2>
<p class="case-section__lead">
The versioned golden set contains 24 passing cases, 13 tool contracts, and 39
tool calls across search, commerce, payments, CRM, communications, databases,
scheduling, weather, support, and multi-step workflows. Every case and expected
trace is public.
</p>
<div class="case-task-list">
<article>
<span>RESPONSE ONLY</span>
<h3>87 caught. 176 missed.</h3>
<p>The final answer exposed some failures, but 162 critical tool-behavior faults still passed.</p>
</article>
<article>
<span>TRACE ONLY</span>
<h3>215 caught. 48 missed.</h3>
<p>The ordered calls were correct, but removed or replaced final responses escaped.</p>
</article>
<article>
<span>TRACE + OUTCOME</span>
<h3>263 caught. 0 missed.</h3>
<p>Exact checks of the ordered tool trace and final outcome detected every planted fault in the set.</p>
</article>
</div>
<div class="case-note">
<span>SCOPE</span>
<p>
This result measures evaluator sensitivity to these controlled faults. It does
not establish overall agent accuracy or safety. The benchmark is deterministic,
runs offline without model calls, and is pinned so an unreviewed change to its
cases, mutations, or expected results fails CI.
</p>
</div>
</div>
</div>
</section>
<section class="case-section case-section--dark">
<div class="section-shell case-section__inner">
<div class="case-section__label">
<p class="section-index">03 / CONTROLLED FAILURES</p>
</div>
<div class="case-section__body">
<h2>What Mendmark breaks on purpose.</h2>
<div class="case-task-list">
<article>
<span>TOOL SELECTION</span>
<h3>A required call disappears</h3>
<p>Mendmark removes each tool call and adds an undeclared tool to see whether the trace still passes.</p>
</article>
<article>
<span>ARGUMENTS</span>
<h3>The right tool gets the wrong input</h3>
<p>One argument changes at a time, exposing evals that check the tool name but ignore its payload.</p>
</article>
<article>
<span>SIDE EFFECTS</span>
<h3>The payment happens twice</h3>
<p>Calls marked as side effecting are duplicated to test whether the suite notices unsafe retries.</p>
</article>
<article>
<span>RECOVERY</span>
<h3>The tool fails. The agent says it worked.</h3>
<p>A successful result becomes an error while the final response still claims completion.</p>
</article>
<article>
<span>FINAL RESPONSE</span>
<h3>The answer vanishes or changes</h3>
<p>The response is removed or replaced to confirm that outcome checks are doing real work.</p>
</article>
</div>
</div>
</div>
</section>
<section class="case-section section-shell">
<div class="case-section__label">
<p class="section-index">04 / TOOL ROLLOUTS</p>
</div>
<div class="case-section__body">
<h2>A new tool should change the release conversation.</h2>
<p class="case-section__lead">
Mendmark hashes every declared tool contract. When a tool appears or its schema
changes, the report shows whether any case exercises it and whether those evals catch
faults in its calls. Basic schema checks also find missing required arguments and
wrong value types without recording the values themselves.
</p>
<div class="case-comparison">
<div>
<span>BEFORE</span>
<p>A pull request adds a refund tool. The aggregate agent score stays green.</p>
</div>
<div>
<span>AFTER</span>
<p>The report marks the new side effect, shows its mutation coverage, and blocks any critical survivor.</p>
</div>
</div>
<p>
A committed baseline remembers which faults the suite caught before. If a changed
metric lets one of those faults pass, CI reports the exact regression.
</p>
</div>
</section>
<section class="case-section case-section--paper-dark">
<div class="section-shell case-section__inner">
<div class="case-section__label">
<p class="section-index">05 / WHAT SHIPS TODAY</p>
</div>
<div class="case-section__body">
<h2>A released local engine with a small data footprint.</h2>
<div class="case-columns">
<div>
<h3>Open-source engine</h3>
<ul>
<li>DeepEval suite adapter</li>
<li>Framework-neutral JSON adapter</li>
<li>Nine built-in mutation operators</li>
<li>Per-tool mutation coverage</li>
<li>Tool contract and schema tracking</li>
<li>Regression-aware CI gates</li>
</ul>
</div>
<div>
<h3>Report boundary</h3>
<ul>
<li>JSON, JUnit, and SARIF output</li>
<li>Custom mutation plugins</li>
<li>Source, policy, and CI provenance</li>
<li>Signed report verification with Cosign</li>
<li>PyPI Trusted Publishing provenance</li>
</ul>
</div>
</div>
<div class="case-note">
<span>LIMIT</span>
<p>
A passing mutation audit does not prove an agent is safe. It shows that the
configured eval suite caught the faults Mendmark tested. Teams still need
product-specific cases and mutations for their highest-risk behavior.
</p>
</div>
</div>
</div>
</section>
<section class="case-next section-shell">
<p class="section-index">06 / TRY IT</p>
<h2>Run a complete audit without an API key.</h2>
<p>
The repository includes both a small refund-agent demonstration and the public golden
set with 263 pinned mutations. Clone the project and reproduce the results without an
API key. Teams with an existing suite can use the DeepEval adapter or connect any local
evaluator through the versioned JSON protocol.
</p>
<div class="case-next__actions">
<a class="button button--primary" href="https://github.com/danielgaskins/mendmark#quick-start" target="_blank" rel="noreferrer">Run the quick start</a>
<a class="button button--quiet" href="https://github.com/danielgaskins/mendmark/issues/1" target="_blank" rel="noreferrer">Request a design-partner pilot</a>
</div>
</section>
</main>
<footer class="site-footer section-shell">
<p>© <span data-year></span> Daniel Gaskins</p>
<p>Physics · Models · Products</p>
<a href="#main">Back to top ↑</a>
</footer>
</body>
</html>