Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -44,5 +44,8 @@ jobs:
- name: Build release
run: cargo build --workspace --release

- name: Run benchmark smoke
run: cargo bench -p manas-benches -- --quick

- name: Smoke test CLI binary
run: ./target/release/manas --help
52 changes: 48 additions & 4 deletions ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,8 @@
14. [The Importance Scoring System](#14-the-importance-scoring-system)
15. [The Freshness System](#15-the-freshness-system)
16. [Error Handling Strategy](#16-error-handling-strategy)
17. [What Manas Is Not](#17-what-manas-is-not)
17. [Benchmarks and Integration Gates](#17-benchmarks-and-integration-gates)
18. [What Manas Is Not](#18-what-manas-is-not)

---

Expand Down Expand Up @@ -450,13 +451,19 @@ manas/
│ ├── toml.rs
│ └── csv.rs
── manas-cli/ ← USER INTERFACE
── manas-cli/ ← USER INTERFACE
├── Cargo.toml ← deps: all crates above
└── src/
└── main.rs ← std-only arg parsing, command routing, formatting
└── manas-benches/ ← TOOLING ONLY
└── benches/
└── bench.rs ← B1-B8 benchmark harness, BENCHMARKS.md generator
```

**Total crates: 5** (v1 had 9 — simpler is better)
**Runtime crates: 5** (v1 had 9 — simpler is better)

**Tooling crates: 1** (`manas-benches`, not part of the runtime path)

**No `manas-language` crate** — the transformer path from v1 is removed.
Language generation is a future milestone, not the foundation.
Expand Down Expand Up @@ -1178,7 +1185,44 @@ impl std::error::Error for ManasError { ... }

---

## 17. What Manas Is Not
## 17. Benchmarks and Integration Gates

Stage 16 keeps performance and regression claims measurable.

`manas-benches` is a dedicated non-runtime crate with a custom no-dependency
benchmark harness:

| ID | What it measures |
|---|---|
| B1 | Single `teach` call |
| B2 | Single `ask` call |
| B3 | `.manas` save |
| B4 | `.manas` load |
| B5 | Tokenizer throughput on 1000 words |
| B6 | Full anti-forgetting proof |
| B7 | Estimated 1000-neuron memory footprint |
| B8 | Brain file growth per new fact |

The committed benchmark report is generated with:

```bash
cargo bench -p manas-benches -- --write-markdown BENCHMARKS.md
```

CI runs the quick benchmark smoke:

```bash
cargo bench -p manas-benches -- --quick
```

The Stage 16 integration gate lives in `manas-cli/tests/stage16_integration.rs`
because the workspace root is virtual and `cargo test --workspace` only runs
tests attached to workspace packages. It covers the real demo, anti-forgetting,
persistence, growth, protection, compression, freshness, and ingestion.

---

## 18. What Manas Is Not

Manas v2 is honest about its scope:

Expand Down
21 changes: 21 additions & 0 deletions BENCHMARKS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
# Benchmarks

Generated by `cargo bench -p manas-benches -- --write-markdown BENCHMARKS.md`.

Mode: `full`

| ID | Benchmark | Result | Unit | Detail |
|---|---|---:|---|---|
| B1 | single teach | 0.0384 | ms/op | 25 iterations |
| B2 | single ask | 0.0190 | ms/op | 200 iterations |
| B3 | .manas save | 0.4677 | ms/op | 25 iterations |
| B4 | .manas load | 0.1514 | ms/op | 50 iterations |
| B5 | tokenizer 1000 words | 0.3946 | ms/op | 100 iterations |
| B6 | anti-forgetting proof | 0.3630 | s | single fixed seed |
| B7 | 1000-neuron footprint | 419.6875 | KiB | estimated heap footprint, total=1000 |
| B8 | brain growth per fact | 768.8750 | bytes/fact | n=32, min=646, max=4534 |

B7 reports an internal heap-footprint estimate for network-owned buffers and neuron storage, not process RSS.

Run full benchmarks with `cargo bench -p manas-benches`.
Run CI smoke benchmarks with `cargo bench -p manas-benches -- --quick`.
11 changes: 10 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ Manas uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
- Stage 13 — The real demo.
- Stage 14 — Inspect, neurons, and debug commands.
- Stage 15 — Compression and forget command.
- Stage 16 — Benchmarks and test suite.

### Added

Expand Down Expand Up @@ -105,10 +106,18 @@ Manas uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
- Added compression tests proving neuron count reduction, frozen-neuron safety,
high-importance survival, anti-forgetting after compression, dry-run behavior,
and brain file-size shrinkage.
- Added the non-runtime `manas-benches` workspace crate with B1-B8 benchmark
coverage for teach, ask, save, load, tokenization, anti-forgetting, memory
footprint, and brain file growth.
- Added `BENCHMARKS.md` with generated full-run benchmark results.
- Added Stage 16 integration coverage for the 22-fact demo, anti-forgetting,
persistence, growth, protection, compression, freshness, and ingestion.
- Added CI benchmark smoke coverage through
`cargo bench -p manas-benches -- --quick`.

### Next

- Stage 16Benchmarks and test suite.
- Stage 17Layer growth.

---

Expand Down
9 changes: 9 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ members = [
"manas-learn",
"manas-ingest",
"manas-cli",
"manas-benches",
]

[workspace.package]
Expand Down
26 changes: 24 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -133,7 +133,8 @@ Manas v2 is in active development. The roadmap follows a strict rule:
| Stage 13 | The real demo | Complete |
| Stage 14 | Inspect, neurons, and debug commands | Complete |
| Stage 15 | Compression and forget command | Complete |
| Stage 16+ | Benchmarks, layer growth, future agents | Planned |
| Stage 16 | Benchmarks and test suite | Complete |
| Stage 17+ | Layer growth, future agents | Planned |

Stages 1 and 2 are preserved as a standalone proof in
`manas-core/src/experiment.rs`. Stage 3 promotes the proven engine into
Expand Down Expand Up @@ -163,6 +164,10 @@ inspectable with rich `inspect`, `neurons`, and `trace` commands for debugging
learned weights, metadata, and query flow. Stage 15 adds conservative
compression through `manas forget`, merging only stale low-importance `Open`
hidden neurons that have a highly similar retained neighbor.
Stage 16 adds a dedicated benchmark crate, committed benchmark report, CI
benchmark smoke test, and an eight-test integration gate covering the demo,
anti-forgetting, persistence, growth, protection, compression, freshness, and
ingestion.

Run the proof:

Expand Down Expand Up @@ -236,6 +241,19 @@ cargo test -p manas-core compression
cargo test -p manas-learn compression
```

Run the Stage 16 integration proof:

```bash
cargo test -p manas-cli stage16
```

Run the benchmarks:

```bash
cargo bench -p manas-benches -- --quick
cargo bench -p manas-benches -- --write-markdown BENCHMARKS.md
```

Run the CLI proof:

```bash
Expand All @@ -249,6 +267,7 @@ Run the ingestion proof:
cargo test -p manas-ingest
```

See [BENCHMARKS.md](./BENCHMARKS.md) for the latest committed benchmark run.
See [ROADMAP.md](./ROADMAP.md) for the full plan with tests.
See [ARCHITECTURE.md](./ARCHITECTURE.md) for the full design.

Expand Down Expand Up @@ -281,7 +300,7 @@ cargo build --workspace --release

## Architecture

Manas v2 is built from 5 Rust crates, each with a single responsibility:
Manas v2 is built from 5 runtime Rust crates plus one benchmark tooling crate:

```
┌──────────────────────────────────────────┐
Expand Down Expand Up @@ -325,6 +344,9 @@ Manas v2 is built from 5 Rust crates, each with a single responsibility:
The transformer path from v1 is removed. The text sidecar from v1 is removed.
The answering system from v1 is replaced with direct neural weight retrieval.

`manas-benches` is a non-runtime workspace crate used for Stage 16 benchmark
measurement and CI smoke coverage.

---

## The `.manas` File Format
Expand Down
28 changes: 21 additions & 7 deletions ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,7 @@ from Stage 2 onward.**
| Stage 13 | The real demo | Complete |
| Stage 14 | Inspect, neurons, and debug commands | Complete |
| Stage 15 | Compression and forget command | Complete |
| Stage 16 | Benchmarks and test suite | Planned |
| Stage 16 | Benchmarks and test suite | Complete |
| Stage 17 | Layer growth | Planned |
| Stage 18 | Internet agent (future) | Planned |
| Stage 19 | Language generation (future) | Planned |
Expand Down Expand Up @@ -1949,7 +1949,7 @@ fn anti_forgetting_test_still_passes_after_compression() {
### Integration Tests

```rust
// tests/integration/
// manas-cli/tests/stage16_integration.rs

// IT-1: 22-fact demo test (Stage 13) runs in CI
// IT-2: Anti-forgetting: 5 facts survive 50 new facts
Expand All @@ -1973,15 +1973,29 @@ jobs:
- cargo test --workspace
- cargo clippy --workspace -- -D warnings
- cargo fmt --check
- cargo bench -p manas-benches -- --quick
```

### Done When

- [ ] All 8 benchmarks produce stable numbers
- [ ] All 8 integration tests pass in CI
- [ ] `cargo test --workspace` passes with zero failures
- [ ] `cargo clippy` passes with zero warnings
- [ ] Benchmark results documented in `BENCHMARKS.md`
- [x] All 8 benchmarks produce stable numbers
- [x] All 8 integration tests pass in CI
- [x] `cargo test --workspace` passes with zero failures
- [x] `cargo clippy` passes with zero warnings
- [x] Benchmark results documented in `BENCHMARKS.md`

### Stage 16 Implementation Notes

- Added the non-runtime `manas-benches` workspace crate with a deterministic
custom benchmark harness covering B1 through B8.
- Added `BENCHMARKS.md`, generated by
`cargo bench -p manas-benches -- --write-markdown BENCHMARKS.md`.
- Added the `manas-cli/tests/stage16_integration.rs` integration gate covering
IT-1 through IT-8.
- Added CI benchmark smoke coverage with
`cargo bench -p manas-benches -- --quick`.
- Kept Stage 16 coverage inside the existing crates so the runtime architecture
remains unchanged.

---

Expand Down
14 changes: 14 additions & 0 deletions manas-benches/Cargo.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
[package]
name = "manas-benches"
version.workspace = true
edition.workspace = true
publish = false

[[bench]]
name = "bench"
harness = false

[dependencies]
manas-core = { workspace = true }
manas-store = { workspace = true }
manas-learn = { workspace = true }
Loading
Loading