Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 26 additions & 1 deletion .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -44,8 +44,33 @@ jobs:
'import site; print(site.getsitepackages()[0])')
python -m pip_audit --path "$product_site"

harness-assurance:
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
include:
- profile: langgraph
package: "langchain-core>=1,<2"
- profile: crewai
package: "crewai>=1,<2"
- profile: openai-agents
package: "openai-agents>=0.19,<1"
steps:
- uses: actions/checkout@fbc6f3992d24b796d5a048ff273f7fcc4a7b6c09 # v5
- uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6
with:
python-version: "3.13"
cache: pip
- run: python -m pip install --upgrade pip
- run: python -m pip install -e '.[dev]' '${{ matrix.package }}'
- name: Exercise current public harness objects
env:
MENDMARK_HARNESS: ${{ matrix.profile }}
run: python -m pytest tests/test_harness_live_compatibility.py

build:
needs: assurance
needs: [assurance, harness-assurance]
runs-on: ubuntu-latest
outputs:
version: ${{ steps.package-version.outputs.version }}
Expand Down
26 changes: 26 additions & 0 deletions .github/workflows/tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ name: tests

on:
push:
branches: [main]
pull_request:
workflow_dispatch:

Expand Down Expand Up @@ -92,6 +93,31 @@ jobs:
run: python -m pip install -e '.[deepeval,rubric,dev]'
- run: python -m pytest tests/test_deepeval.py tests/test_rubric_example.py

harness-compatibility:
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
include:
- profile: langgraph
package: "langchain-core>=1,<2"
- profile: crewai
package: "crewai>=1,<2"
- profile: openai-agents
package: "openai-agents>=0.19,<1"
steps:
- uses: actions/checkout@fbc6f3992d24b796d5a048ff273f7fcc4a7b6c09 # v5
- uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6
with:
python-version: "3.13"
cache: pip
- run: python -m pip install --upgrade pip
- run: python -m pip install -e '.[dev]' '${{ matrix.package }}'
- name: Exercise current public harness objects
env:
MENDMARK_HARNESS: ${{ matrix.profile }}
run: python -m pytest tests/test_harness_live_compatibility.py

distribution-assurance:
runs-on: ubuntu-latest
steps:
Expand Down
19 changes: 18 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,24 @@
All notable changes to Mendmark are documented here. The project follows
Semantic Versioning for its Python and JSON contracts.

## Unreleased
## 0.6.0 - 2026-08-10

### Added

- Dependency-light, public-object adapters for LangChain/LangGraph messages,
CrewAI events, and OpenAI Agents SDK run items, including tool-schema and
side-effect metadata conversion.
- `mendmark equip` for bounded harness detection, non-destructive local
scaffolding, an offline evaluator, an inactive pinned CI template, and a
copyable coding-agent self-equip prompt.
- Live compatibility assurance against current releases of all three harness
paths, plus explicit human approval before observed traces can become
expected behavior.
- A fluent causal-case builder for reviewed multi-agent delegation, parallel
dependencies, tool authority, state changes, results, and aggregation without
hand-authoring schema 2.0 JSON.

## 0.5.0 - 2026-08-06

### Added

Expand Down
28 changes: 27 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@
</p>
<p>
<a href="#quick-start">Quick start</a> ·
<a href="#equip-an-agent-harness">Harnesses</a> ·
<a href="#agent-eval-golden-set">Golden set</a> ·
<a href="docs/multi-agent.md">Multi-agent</a> ·
<a href="docs/agent-mutation-audits.md">How it works</a> ·
Expand Down Expand Up @@ -63,6 +64,31 @@ mendmark audit-json examples/multi_agent_suite.json \
--evaluator-command "python3 examples/multi_agent_evaluator.py"
```

## Equip an agent harness

Mendmark has dependency-light adapters for LangChain/LangGraph, CrewAI, and the
OpenAI Agents SDK. In an existing agent repository:

```bash
python -m pip install 'mendmark-evals==0.6.0'
mendmark equip --framework auto
```

The command detects bounded dependency files and creates a reviewed capture
guide, offline evaluator, and inactive CI template under `.mendmark/`. It does
not edit application code, upload a trace, overwrite existing work, enable CI,
or accept a baseline.

Want the repository's coding agent to perform the integration?

```bash
mendmark equip --print-agent-prompt
```

See the [agent harness integration guide](docs/harness-integrations.md) for the
direct Python APIs, explicit trace-approval boundary, framework compatibility,
and multi-agent guidance.

## See the blind spot in two minutes

**[▶ Watch the narrated weak-eval demonstration (original v1 fault inventory)](docs/assets/mendmark-weak-eval-demo.mp4)**
Expand Down Expand Up @@ -346,7 +372,7 @@ and the [ML evaluation card](https://github.com/danielgaskins/mendmark/blob/main

## Current boundary

Version 0.5 is a local, open-source engine. It does not yet provide a hosted
Version 0.6 is a local, open-source engine. It does not yet provide a hosted
dashboard, team accounts, remote trace ingestion, or a secrets service. The
planned control plane is described in [the product design](https://github.com/danielgaskins/mendmark/blob/main/docs/product.md).

Expand Down
5 changes: 5 additions & 0 deletions docs/assurance.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,11 @@ The automated assurance suite checks that:
- `mendmark --version`, `--help`, packaged tasks, packaged schemas, and the
complete single- and multi-agent JSON audit journeys work from that clean
installation.
- Harness converters are tested with privacy-safe facsimiles and current public
objects from LangChain Core, CrewAI, and OpenAI Agents SDK. `mendmark equip`
is tested for idempotency, bounded detection, conflict refusal, symlink
containment, explicit expectation approval, and a complete generated
evaluator journey.
- Reports, console output, JUnit, and SARIF do not expose canary values placed in
prompts, answers, metadata, tags, tool arguments, tool outputs, or descriptions.
- Repeated audits preserve mutation IDs, ordering, decisions, JUnit, SARIF, and
Expand Down
7 changes: 6 additions & 1 deletion docs/compatibility.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,9 +18,14 @@ Mendmark 0.x is evolving, but automation still needs predictable contracts.
- Historical benchmarks select immutable `agent-eval-v1` or `multi-agent-v1`
mutation profiles. Ordinary audits use `current`; adding a new operator never
rewrites an older golden-set contract.
- Harness adapters are dependency-light and use documented public object
fields. Hosted CI exercises LangChain Core 1.5.3, CrewAI 1.15.14, and OpenAI
Agents SDK 0.19.4 as the 2026-08-10 compatibility snapshot. Newer supported
releases are tested through the latest-version harness matrix; Mendmark does
not install or pin a harness in customer environments.
- A custom operator name is globally unique within an audit and is part of the
customer's accepted baseline contract.
- `audit`, `audit-json`, `prepare`, `grade`, and `show` exit with 0 for success,
- `audit`, `audit-json`, `equip`, `prepare`, `grade`, and `show` exit with 0 for success,
1 for a failed product gate, and 2 for invalid input or infrastructure failure.

The JSON Schemas under `mendmark/schemas` are the machine-readable contract.
Expand Down
Loading