MGBench

MGBench is a benchmark for memory governance in agentic long-term memory systems.

It does not measure whether an agent can solve a GitHub issue. It measures whether a memory system can provide useful long-term context while preventing stale, invalidated, cross-scope, contradictory, or failed historical memory from becoming agent-usable context.

What MGBench Measures

• Continuity: recover current useful memory across sessions.
• Negative-transfer blocking: avoid reusing wrong historical workflow memory.
• Context efficiency: provide shorter agent context without losing current evidence.
• Governance ownership: distinguish runtime-internal governance from host-supplied filtering.
• Self-learning loop: promote repeated useful memory and suppress it when later invalidated.
• Execution-state continuity: preserve current execution state while isolating failed branches.

Track Model

MGBench is organized around memory-governance tracks:

• Admission/Lifecycle tracks measure static memory admission: given a memory set and a query, which memories may enter agent-usable context.
• Interference Robustness track measures dynamic disturbance recovery: after interruption, tool failure, context pollution, memory conflict, or environment drift, can a memory system recover the active trajectory and keep corrupted history out of direct-use context?

The Interference Robustness track is currently included as auxiliary evidence, not as part of the public 0.1 leaderboard. It has a seed fixture for adapter development and a first 40-scenario real-source GitHub PR batch with Aionis and Mem0 arms. It remains outside the aggregate leaderboard until a larger frozen batch from long-running execution traces is available.

Public 0.1 Release

The public 0.1.1 release indexes 608 frozen deterministic scenarios and is archived on Zenodo as DOI 10.5281/zenodo.20793097:

suite	scenarios	source
credibility governance	200	real GitHub PR-derived workflow pairs
ordinary-memory governance	40	active/stale/archive/scope memory tasks
controlled forgetting	24	Aionis, Graphiti, and Tencent Agent Memory reports
high-trust conflict governance	24	high-trust conflicting prior memory
scope isolation	40	real GitHub issue-derived scope fixtures
lifecycle inference	40	unlabelled lifecycle inference fixtures
execution-tree effect	40	execution-state branch isolation
execution-tree stress	200	execution-state continuity under stress

The source scenarios and reports are frozen. Scoring is deterministic and does not use an LLM judge.

Run

npm install
npm run verify
npm run score
npm run interference:verify
npm run interference:score

Generated reports are written to reports/.

Current public report:

• Markdown
• JSON

Aionis v0.3 evidence bundle:

• Aionis v0.3 Context Reliability Evidence Bundle
• JSON

This bundle summarizes the current Aionis v0.3 product-path evidence around state-preserving, auditable execution context under interference. It is a bounded memory-governance report, not a GitHub issue-solving benchmark.

Current Reference Results

The current 0.1 manifest includes reference arms for Aionis, raw memory, no memory, Mem0, Supermemory, Graphiti, and Tencent Agent Memory where reports are available.

The candidate Interference Robustness track has its seed manifest at benchmarks/mgbench-interference-v0.1.json and a real-source GitHub PR derived manifest at benchmarks/mgbench-interference-real-v0.1.json. Its built-in sanity_governed_reference arm is an oracle-like scorer check and must not be interpreted as a real system result.

Leaderboard Snapshot

rank	arm	governance owner	completed	wrong reused	product positive	avg context chars	MGBench score
1	Aionis	runtime_internal	368	0/368	368/368	1,224	100.00
2	Aionis execution tree	runtime_internal	240	0/240	240/240	3,193	93.83
3	Supermemory filtered	external_host	280	0/280	279/280	1,484	92.93
4	Mem0 filtered	external_host	280	0/280	280/280	1,815	91.74
5	Mem0 normal	none	280	274/280	3/280	4,671	30.56
6	Raw memory	none	200	200/200	0/200	1,792	30.00
7	Supermemory normal	none	280	280/280	0/280	3,170	30.00
8	No memory	none	200	0/200	0/200	0	0.00

Read the leaderboard with these boundaries:

• Raw per-suite metrics are the primary evidence.
• The aggregate MGBench score is a compact ranking for the current manifest.
• Arms do not all cover the same number of suites; coverage is reported explicitly.
• Filtered Mem0 and Supermemory modes are valid arms, but are labeled external_host because the caller supplies lifecycle/filter knowledge.
• runtime_internal means the memory system itself decides what is safe to expose to the agent.

Auxiliary Interference Evidence

The first real-source Interference Robustness run uses the 40-scenario GitHub PR-derived batch and compares Aionis observe->guide against Mem0 Platform V3 modes. These results are indexed as auxiliary evidence and are not included in the aggregate MGBench 0.1 score.

Report: Interference real-source competitor run

arm	governance owner	completed	product positive	unsafe direct-use	rehydrate recall	trace present	avg context chars
Aionis observe->guide	runtime_internal	40/40	40/40	0/40	100%	100%	1,320
Mem0 raw	none	39/40	0/40	39/40	75%	0%	1,276
Mem0 host-filtered	external_host	40/40	30/40	0/40	75%	0%	526

Interpretation: Mem0 raw retrieves active history but leaks stale, failed, and inspect-only memory into direct-use context. Mem0 host-filtered is safe only when the caller supplies an external active-state metadata filter; it still lacks rehydrate hooks and decision trace. Aionis is the only completed arm in this run that recovers active state, blocks corrupted history, preserves rehydrate hooks, and emits traceable memory decisions without fixture lifecycle labels.

Strict ID-neutral product-contract run: Interference strict clean-DB run

The strict holdout keeps the same 40 real-source scenarios but rewrites fixture memory ids to neutral m001/m002/m003 values and removes role-bearing memory ids from observed Aionis text. The Aionis adapter emits product-contract execution surfaces (current_state, failed_branch, workflow_candidate, raw_trace_pointer, lifecycle state, and warm rehydrate pointers) without passing fixture gold labels.

arm	fixture	product positive	active recovery	unsafe direct-use	rehydrate recall	trace present	avg context chars
Aionis observe->guide	mgbench-interference-real-strict-v0.1	40/40	100%	0/40	100%	100%	1,304

This clean-DB run used fresh Lite Runtime SQLite stores and a fresh scope prefix. It is the current headline strict result for the Aionis product-contract adapter.

Zvec ANN recall backend reruns:

arm	fixture	product positive	active recovery	unsafe direct-use	rehydrate recall	trace present	avg context chars
Aionis observe->guide + Zvec ANN	mgbench-interference-real-strict-v0.1	40/40	100%	0/40	100%	100%	1,306
Aionis observe->guide + Zvec ANN rerun	mgbench-interference-real-strict-v0.1	40/40	100%	0/40	100%	100%	1,305

Reports: Zvec ANN run, Zvec ANN rerun.

Interpretation: these reruns keep Aionis' governance boundary unchanged while using the optional Zvec ANN backend for stronger candidate generation. The strict ID-neutral fixture still preserves active state, blocks unsafe direct-use, keeps rehydrate hooks, and emits traceable memory decisions.

Adapter Contract

Any memory system can be evaluated by producing a deterministic report with the fields in docs/adapter-contract.md. An adapter may call a local runtime or a hosted API, but provider failures must be reported as blocked cases, not silently dropped.

Benchmark Contract

The scoring and interpretation contract is documented in docs/benchmark-contract.md.

Core principles:

1. Frozen fixtures are not edited after generation.
2. Scoring is deterministic.
3. Filtered modes must disclose whether governance is runtime-internal or host-managed.
4. Benchmark code stays outside the memory runtime under test.
5. Patch success is not a MGBench metric.

Non-Goals

• Not an Aionis-only benchmark.
• Not a patch-success benchmark.
• Not an agent reasoning benchmark.
• Not a replacement for downstream end-to-end task benchmarks.
• Not a benchmark where filtered competitors are hidden; external-host filtering is reported explicitly.