Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,24 @@
All notable changes to Mendmark are documented here. The project follows
Semantic Versioning for its Python and JSON contracts.

## 0.6.1 - 2026-08-10

### Added

- Repo-scoped Mendmark skills for Codex and Claude Code, with explicit
`$mendmark` and `/mendmark` invocation paths and non-destructive auto/all
targeting through `mendmark equip --agent`.
- An always-generated, agent-neutral `SELF-EQUIP.md` protocol for unsupported
coding agents, covering harness discovery, JSON fallback, causal trace
fidelity, reviewed golden behavior, privacy, side effects, baselines, CI,
and final evidence reporting.

### Changed

- Self-equip metadata now records coding-agent targets, and distribution
assurance verifies both native skills without modifying existing
`AGENTS.md`, `CLAUDE.md`, or repository policy.

## 0.6.0 - 2026-08-10

### Added
Expand Down
16 changes: 12 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,19 +70,27 @@ Mendmark has dependency-light adapters for LangChain/LangGraph, CrewAI, and the
OpenAI Agents SDK. In an existing agent repository:

```bash
python -m pip install 'mendmark-evals==0.6.0'
mendmark equip --framework auto
python -m pip install 'mendmark-evals==0.6.1'
mendmark equip --framework auto --agent auto
```

The command detects bounded dependency files and creates a reviewed capture
guide, offline evaluator, and inactive CI template under `.mendmark/`. It does
not edit application code, upload a trace, overwrite existing work, enable CI,
or accept a baseline.

Want the repository's coding agent to perform the integration?
Want Codex or Claude Code to perform the integration? Install its native,
repo-scoped skill (use `all` to install both):

```bash
mendmark equip --print-agent-prompt
mendmark equip --framework auto --agent codex # invoke with $mendmark
mendmark equip --framework auto --agent claude-code # invoke with /mendmark
```

For any other repository-capable agent, print a portable self-equip prompt:

```bash
mendmark equip --agent generic --print-agent-prompt
```

See the [agent harness integration guide](docs/harness-integrations.md) for the
Expand Down
4 changes: 4 additions & 0 deletions docs/assurance.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,10 @@ The automated assurance suite checks that:
is tested for idempotency, bounded detection, conflict refusal, symlink
containment, explicit expectation approval, and a complete generated
evaluator journey.
- Codex and Claude Code receive repo-scoped skills in their documented native
locations, while an always-generated portable protocol covers unrecognized
coding agents. Tests verify auto-detection, explicit/all targeting, native
invocation names, existing-instruction preservation, and safety boundaries.
- Reports, console output, JUnit, and SARIF do not expose canary values placed in
prompts, answers, metadata, tags, tool arguments, tool outputs, or descriptions.
- Repeated audits preserve mutation IDs, ordering, decisions, JUnit, SARIF, and
Expand Down
57 changes: 47 additions & 10 deletions docs/harness-integrations.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,14 +36,15 @@ guide](https://openai.github.io/openai-agents-python/tracing/).
From the agent application repository:

```bash
python -m pip install 'mendmark-evals==0.6.0'
mendmark equip --framework auto
python -m pip install 'mendmark-evals==0.6.1'
mendmark equip --framework auto --agent auto
```

Detection reads only bounded dependency files; it does not import or execute the
application. The command creates five reviewable files under `.mendmark/`:
application. The command creates six reviewable files under `.mendmark/`:

- `agent-setup.md`: harness-specific capture code and acceptance criteria.
- `SELF-EQUIP.md`: portable instructions for any repository-capable agent.
- `evaluator.py`: a deterministic offline evaluator for reviewed snapshots.
- `mendmark-ci.yml`: an inactive, pinned CI template.
- `config.json`: detected integration metadata.
Expand All @@ -55,24 +56,60 @@ preview its targets.

## Let a coding agent self-equip the repository

### Codex

```bash
mendmark equip --framework auto --agent codex
```

This installs a repo-scoped skill at `.agents/skills/mendmark/SKILL.md`, the
location Codex discovers from the repository root. Ask Codex to equip or audit
agent evaluations naturally, or invoke the skill explicitly with `$mendmark`.
UI metadata supplies a concise skill title and default prompt. Mendmark does
not create or modify the repository's `AGENTS.md`.
See OpenAI's official [Codex skills documentation](https://learn.chatgpt.com/docs/build-skills)
and [AGENTS.md discovery rules](https://learn.chatgpt.com/docs/agent-configuration/agents-md).

### Claude Code

```bash
mendmark equip --framework auto --agent claude-code
```

This installs `.claude/skills/mendmark/SKILL.md`. Claude Code can select it when
the request matches its description, or the user can invoke `/mendmark`.
Mendmark does not create or modify `CLAUDE.md`, `CLAUDE.local.md`, or existing
Claude rules. Use `--agent all` to install both native skills.
See Anthropic's official [Claude Code skills documentation](https://docs.claude.com/en/docs/claude-code/skills)
and [project-memory documentation](https://docs.claude.com/en/docs/claude-code/memory).

### Any other coding agent

Print a prompt that works with repository-capable coding agents:

```bash
mendmark equip --print-agent-prompt
mendmark equip --agent generic --print-agent-prompt
```

The prompt instructs the agent to run detection, read the generated setup file,
capture a real tool-using case, pass the audit, and meet every review criterion.
It explicitly forbids uploading trace content or silently treating observed
production behavior as correct.

The short prompt can also be copied directly:
Every setup includes `.mendmark/SELF-EQUIP.md`, a framework-neutral procedure
covering capability discovery, JSON 2.0 fallback, stable call correlation,
explicit multi-agent causality, golden-behavior review, privacy, side-effect
isolation, audit execution, baseline review, CI activation, and final reporting.
An unrepresented agent needs no Mendmark-specific plugin: give it the printed
prompt and repository access.

The short generic prompt can also be copied directly:

> Run `mendmark equip --framework auto`, read `.mendmark/agent-setup.md`
> completely, integrate the detected harness, capture at least one reviewed
> tool-using case, run the local audit, and satisfy every acceptance criterion
> before enabling CI. Do not upload trace content or approve observed behavior
> without human review.
> Run `mendmark equip --framework auto --agent generic`, read
> `.mendmark/SELF-EQUIP.md` and `.mendmark/agent-setup.md` completely, integrate
> the detected harness, capture at least one reviewed tool-using case, run the
> local audit, and satisfy every acceptance criterion before enabling CI. Do
> not upload trace content or approve observed behavior without human review.

## Direct Python API

Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"

[project]
name = "mendmark-evals"
version = "0.6.0"
version = "0.6.1"
description = "Mutation testing for agent evaluation suites"
readme = "README.md"
requires-python = ">=3.10"
Expand Down
17 changes: 16 additions & 1 deletion scripts/assure_distribution.py
Original file line number Diff line number Diff line change
Expand Up @@ -125,7 +125,7 @@ def main() -> int:
cwd=workspace,
env=clean_env,
)
if "agent-setup.md" not in prompt or "human review" not in prompt:
if "SELF-EQUIP.md" not in prompt or "human review" not in prompt:
raise RuntimeError("installed wheel did not expose safe agent self-equip guidance")
integration_api = run(
[
Expand All @@ -150,6 +150,8 @@ def main() -> int:
"equip",
"--framework",
"langgraph",
"--agent",
"all",
"--project-root",
str(workspace),
],
Expand All @@ -161,12 +163,25 @@ def main() -> int:
for generated in (
"evaluator.py",
"agent-setup.md",
"SELF-EQUIP.md",
"mendmark-ci.yml",
"config.json",
".gitignore",
):
if not (workspace / ".mendmark" / generated).is_file():
raise RuntimeError(f"installed wheel omitted equip asset: {generated}")
for skill in (
workspace / ".agents" / "skills" / "mendmark" / "SKILL.md",
workspace / ".claude" / "skills" / "mendmark" / "SKILL.md",
workspace
/ ".agents"
/ "skills"
/ "mendmark"
/ "agents"
/ "openai.yaml",
):
if not skill.is_file():
raise RuntimeError(f"installed wheel omitted agent skill: {skill}")
tasks = run([str(mendmark), "tasks"], cwd=workspace, env=clean_env)
if len([line for line in tasks.splitlines() if line.strip()]) != 5:
raise RuntimeError("installed wheel did not expose all five ML integrity tasks")
Expand Down
2 changes: 1 addition & 1 deletion src/mendmark/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

from .agent_cases import AgentCase, AgentEvent, AgentSpec, ToolCallRecord, ToolSpec

__version__ = "0.6.0"
__version__ = "0.6.1"

__all__ = [
"AgentCase",
Expand Down
14 changes: 12 additions & 2 deletions src/mendmark/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,12 @@ def build_parser() -> argparse.ArgumentParser:
default="auto",
help="detect from dependency files or select an integration (default: auto)",
)
equip.add_argument(
"--agent",
choices=("auto", "codex", "claude-code", "generic", "all"),
default="auto",
help="detect or select repository coding-agent instructions (default: auto)",
)
equip.add_argument("--project-root", default=".", help="project to equip")
equip.add_argument(
"--dry-run", action="store_true", help="show files without writing them"
Expand Down Expand Up @@ -446,18 +452,22 @@ def main(argv: list[str] | None = None) -> int:
return 0

if args.command == "equip":
from .equip import agent_prompt, equip_project
from .equip import agent_prompt, equip_project, resolve_agent_hosts

if args.print_agent_prompt:
print(agent_prompt(args.project_root))
print(agent_prompt(args.project_root, agent=args.agent))
return 0
root = Path(args.project_root).expanduser().resolve()
agent_hosts = resolve_agent_hosts(root, args.agent)
frameworks, created, unchanged = equip_project(
args.project_root,
framework=args.framework,
agent=args.agent,
dry_run=args.dry_run,
)
action = "Would create" if args.dry_run else "Created"
print("Detected: " + ", ".join(frameworks))
print("Agent host: " + ", ".join(agent_hosts))
for path in created:
print(f"{action}: {path}")
for path in unchanged:
Expand Down
Loading