|
| 1 | +# CLAIMS.md — What MoA Actually Proves |
| 2 | + |
| 3 | +This document separates proven claims from aspirational ones. |
| 4 | +"Proven" means there is executable code or a test you can run right now. |
| 5 | + |
| 6 | +--- |
| 7 | + |
| 8 | +## ✅ Proven — runnable code, zero deps |
| 9 | + |
| 10 | +### P1: FORBIDDEN action types are rejected by code, not prompts |
| 11 | +**Evidence:** `tests/test_safety.py::test_gate_rejects_deceive` |
| 12 | +`SafetyGate.evaluate()` checks `HardConstraints.FORBIDDEN_ACTIONS` (a Python frozenset) before |
| 13 | +executing any action. The LLM never sees a FORBIDDEN action because the check runs outside the LLM context. |
| 14 | + |
| 15 | +```python |
| 16 | +assert ActionType.DECEIVE_USER in HardConstraints.FORBIDDEN_ACTIONS |
| 17 | +gate = SafetyGate() |
| 18 | +d = gate.evaluate(Action(type=ActionType.DECEIVE_USER)) |
| 19 | +assert not d.approved # always False, regardless of LLM output |
| 20 | +``` |
| 21 | + |
| 22 | +### P2: SIMULATION and FICTION claims cannot become FACT |
| 23 | +**Evidence:** `tests/test_safety.py::test_verifier_rejects_simulation` and `test_verifier_rejects_fiction` |
| 24 | +`FormalVerifier.verify()` checks `claim.modality` before any other check. No amount of evidence |
| 25 | +or confidence can override this: |
| 26 | + |
| 27 | +```python |
| 28 | +claim = create_claim("event", "would_cause", "effect", Modality.SIMULATION, confidence=0.99) |
| 29 | +promoted, _ = FormalVerifier().promote_to_fact(claim, [evidence]) |
| 30 | +assert promoted == False # always |
| 31 | +``` |
| 32 | + |
| 33 | +### P3: EventLog is append-only — nothing is deleted |
| 34 | +**Evidence:** `tests/test_ir.py`, `verify_moa.py::check_memory` |
| 35 | +`EventLog` has no `.delete()`, `.pop()`, or `.clear()` method. The only public mutation method is |
| 36 | +`.write()`. This is enforced by the class definition, not by runtime checks. |
| 37 | + |
| 38 | +### P4: Actions above threshold are rejected regardless of LLM output |
| 39 | +**Evidence:** `tests/test_safety.py::test_gate_rejects_high_irreversibility` |
| 40 | +`HardConstraints.MAX_IRREVERSIBILITY = 0.8`. Any action with `ActionMeta.irreversibility_score > 0.8` |
| 41 | +is rejected by `SafetyGate.evaluate()`. Same for `blast_radius > 0.7` and `deception_risk > 0.1`. |
| 42 | + |
| 43 | +### P5: Framework works with any LLM implementing BaseLLMBackend |
| 44 | +**Evidence:** `verify_moa.py`, `tests/test_agent.py` |
| 45 | +All 48 tests run with `MockBackend` — zero external dependencies. The same code path runs with |
| 46 | +OpenAI, Anthropic, Ollama, HuggingFace, and LayerCake backends. |
| 47 | + |
| 48 | +### P6: Formal verification requires evidence + confidence ≥ 0.8 |
| 49 | +**Evidence:** `tests/test_safety.py::test_verifier_rejects_low_confidence` |
| 50 | +Claims with `confidence < 0.8` cannot be promoted to FACT even with supporting evidence. |
| 51 | +This prevents confident-sounding hallucinations from entering the FACT graph. |
| 52 | + |
| 53 | +--- |
| 54 | + |
| 55 | +## ⬜ Aspirational — require production deployment to fully evaluate |
| 56 | + |
| 57 | +### A1: Self-evolving domain modules |
| 58 | +**Status:** Architecture exists (`LayerCakeBackend.paste_domain()`). Bit-exact paste proven in |
| 59 | +the companion [LayerCake repo](https://github.com/Yoder23/layercake). "Self-evolution" at scale |
| 60 | +(agent improves itself over time) requires training infrastructure and long evaluation runs. |
| 61 | + |
| 62 | +### A2: Production-scale multi-agent coordination |
| 63 | +**Status:** The OODA loop runs. Multi-agent orchestration (multiple `MoAAgent` instances sharing |
| 64 | +memory) is not yet implemented. The memory layer is designed for it (shared `AgentMemory` is |
| 65 | +threadsafe for reads), but it has not been stress-tested. |
| 66 | + |
| 67 | +### A3: Semantic memory at scale |
| 68 | +**Status:** `EpisodicBuffer.retrieve_semantic()` works with any `embed()` function. Quality of |
| 69 | +retrieval depends entirely on the embedding model. `MockBackend` uses MD5-based embeddings that |
| 70 | +test the pipeline but not semantic accuracy. |
| 71 | + |
| 72 | +--- |
| 73 | + |
| 74 | +## ❌ Not claimed |
| 75 | + |
| 76 | +- **AGI** — This is an agent framework with safety constraints. It is not general intelligence. |
| 77 | +- **Superintelligence** — No component here exhibits superhuman capability on any benchmark. |
| 78 | +- **Perfect safety** — Hard constraints eliminate specific forbidden action types. They do not |
| 79 | + prevent all possible misuse. Security depends on the threat model. |
| 80 | +- **SOTA text generation** — MoA delegates generation to whatever LLM you provide. Generation |
| 81 | + quality equals the quality of the backend model. |
| 82 | + |
| 83 | +--- |
| 84 | + |
| 85 | +## How to challenge these claims |
| 86 | + |
| 87 | +Every ✅ claim has a test you can run: |
| 88 | + |
| 89 | +```bash |
| 90 | +python verify_moa.py # 10 checks, 0 deps, < 1 second |
| 91 | +pytest tests/ -v # 48 tests, no API key |
| 92 | +``` |
| 93 | + |
| 94 | +If a test fails, open an issue. Claims without passing tests get downgraded. |
0 commit comments