feat: add local experiment archive

.gitignore

@@ -12,3 +12,5 @@ test-results/
 *.tsbuildinfo
 .DS_Store
 *.log
+logs/
+.agentborne/

.prettierignore

@@ -1 +1,2 @@
 **/next-env.d.ts
+logs/

README.md

@@ -10,13 +10,14 @@ rules as every other agent. The universal flat provider contract is
 
 ## Workspace
 
-| Path                     | Responsibility                                                   |
-| ------------------------ | ---------------------------------------------------------------- |
-| `apps/world-lab`         | Next.js developer/admin map, controls, inspector, and event log  |
-| `apps/game-api`          | Hono HTTP boundary and in-memory simulation service              |
-| `packages/world-engine`  | Pure world validation and consequence application                |
-| `packages/agent-runtime` | OpenRouter provider boundary and explicit scripted testing seams |
-| `packages/shared`        | Runtime-validated schemas and inferred domain types              |
+| Path                          | Responsibility                                                   |
+| ----------------------------- | ---------------------------------------------------------------- |
+| `apps/world-lab`              | Next.js developer/admin map, controls, inspector, and event log  |
+| `apps/game-api`               | Hono HTTP boundary and in-memory simulation service              |
+| `packages/world-engine`       | Pure world validation and consequence application                |
+| `packages/agent-runtime`      | OpenRouter provider boundary and explicit scripted testing seams |
+| `packages/shared`             | Runtime-validated schemas and inferred domain types              |
+| `packages/experiment-archive` | Durable SQLite imports and bounded research queries              |
 
 ## Local development
 
@@ -52,7 +53,7 @@ The command reads only `OPENROUTER_API_KEY` from the repository-root `.env`; its
 
 The compatible default centers on Toledo, Ohio (`41.6528, -83.5379`) at H3 resolution 9 and renders the same deterministic radius-six disk of exactly 127 cells with eight fixed perimeter starts. World Setup previews and applies resolution 8–11 scenarios with 1–32 agents, radius at most 40, at most 5,000 actual generated cells, and a 12 km default physical communication range. Schema-v9 exports preserve the authoritative scenario and `durable-influence-v2` attribution. MapLibre uses CARTO Dark Matter's tokenless raster tiles with `© OpenStreetMap contributors © CARTO` attribution.
 
-Alliance leadership, merging, custom metadata, combat systems, relationship scores, group chat, persistent memory, player mechanics, persistence, and autonomous scheduling remain deferred.
+Alliance leadership, merging, custom metadata, combat systems, relationship scores, group chat, persistent memory, player mechanics, restartable world persistence, and autonomous scheduling remain deferred.
 
 When every development cell is infected, World Lab automatically pauses playback and disables Start to avoid accidental provider calls. Reset and export remain available, and Single turn remains an explicitly manual diagnostic action.
 
@@ -63,3 +64,5 @@ The persistent operator shell keeps execution controls, run target, playback spe
 The agent roster defaults to browser-local **Follow turn** behavior: the inspector follows the active request, or the next scheduled agent while paused. Selecting an agent manually disables following without hiding the roster's textual Acting/Next marker; the preference remains in that browser and is never exported. Public world chat and the event log are newest-first bounded feeds, and the shared Model and Export dialogs keep their headers/actions fixed while their bodies scroll within the viewport.
 
 See [Testing](docs/TESTING.md), [Architecture](docs/ARCHITECTURE.md), [Security](docs/SECURITY.md), the accepted future [Gameplay Foundation](docs/GAMEPLAY_FOUNDATION.md), and the [Roadmap](ROADMAP.md).
+
+Completed schema-v9 exports can be imported into an ignored local SQLite archive and queried without repeatedly loading full JSON artifacts. See [Local experiment archive](docs/EXPERIMENT_ARCHIVE.md).

ROADMAP.md

@@ -75,6 +75,8 @@ Twelfth focused slice: add scenario-owned physical communication range, private
 
 ## PR 5 — Goals and memory
 
+Pre-PR-5 observability slice: add a local append-only SQLite experiment archive, transactional schema-v9 export import, bounded human/Codex queries, normalized comparisons, and FTS-searchable curated notes. The in-memory engine remains authoritative. Crash recovery, restartable simulation state, MCP, embeddings, vector search, and a database browser remain deferred.
+
 Persistent short- and long-term objectives, compact memories, plan revision, summaries, and longer simulation runs.
 
 ## PR 6 — Persistent autonomous world

docs/ARCHITECTURE.md

@@ -16,6 +16,8 @@ World Lab → Game API / simulation service → agent runtime → one decision
 
 The Game API also owns one process-local experiment record. Each completed safe turn is captured once, independently from the browser snapshot, and server-side export levels filter that record without affecting provider requests.
 
+Completed schema-v9 exports may cross a separate offline archive boundary into `packages/experiment-archive`. Node's built-in SQLite stores normalized immutable research records through versioned migrations, foreign keys, prepared statements, and transactional idempotent imports. This downstream observability archive is never consulted by turn execution and cannot recover, resume, or mutate the active world. Its bounded query service is application-independent so a future read-only MCP adapter can reuse it without exposing arbitrary SQL.
+
 World Setup uses `world-scenario-v1`. Pure preview computes the actual H3 disk, exact count, summed cell area, deterministic roster/spawns, feasibility, and warnings. Apply recomputes and atomically replaces world and experiment state. Reset reconstructs the current scenario; the Toledo default preserves legacy starts. Explicit location search crosses a replaceable server-owned adapter with no autocomplete, a one-request-per-second Nominatim limit, bounded cache/timeout, normalized results, and OpenStreetMap attribution. Manual coordinates bypass that network boundary.
 
 World Lab may opt into browser-driven unattended recovery for continuous Start or Run-to sessions. After the existing initial call and at most one internal repair/transport retry, the browser serially requests one-call unattended retries up to a local limit of one through three, then commits one attributed unattended skip. Every mutation has a fresh idempotency key and uses the authoritative pending turn. Pausing, cancellation, ineligible failures, or failed reconciliation stops the loop. The tab must remain open; this is not scheduling, simultaneous ticks, or a production recovery service.

docs/EXPERIMENT_ARCHIVE.md

@@ -0,0 +1,78 @@
+# Local experiment archive
+
+The experiment archive is a durable, local research surface for completed or partially retained exports. It does not participate in an active simulation: the Game API's in-memory engine remains authoritative, and an archive write cannot change an accepted game outcome. This slice imports schema-v9 JSON exports; it is not crash recovery, restartable simulation state, or a scheduler.
+
+## Storage and configuration
+
+`@agentborne/experiment-archive` uses SQLite built into the pinned Node 24 runtime. Versioned migrations create strict tables with foreign keys and indexes. File-backed databases enable WAL and a five-second busy timeout; imports use prepared statements inside one transaction. Tests use in-memory or temporary databases.
+
+The default database is `.agentborne/experiments.sqlite`, an ignored development path. Override it with `--db <path>` or `AGENTBORNE_EXPERIMENT_DB`; the flag takes precedence.
+
+The schema normalizes experiments, source exports, roster/assignments, topology, turns, model attempts, communications and recipients, diplomacy attempts, alliance events, world events, configuration changes, and research notes. Stable source IDs are retained; deterministic experiment/turn/attempt IDs make imports idempotent. Source aggregate metrics remain audit evidence, while summaries derive canonical metrics from stored records.
+
+Retention is never silently upgraded. A filtered import, missing optional observation, or truncated source remains visible as incomplete or missing data. Import validates the complete export and rejects prohibited credential/private-reasoning fields before beginning its transaction. Persistence errors roll back and surface explicitly.
+
+## Commands
+
+```bash
+pnpm experiment:db import ./exports/run.json
+pnpm experiment:db list
+pnpm experiment:db summary <experiment-id>
+pnpm experiment:db compare <experiment-id-a> <experiment-id-b>
+pnpm experiment:db turns <experiment-id> --agent <agent-id> --from-turn 40 --to-turn 80
+pnpm experiment:db communications <experiment-id> --channel direct --recipient <agent-id>
+pnpm experiment:db alliance-events <experiment-id> --reason expired
+pnpm experiment:db patient-zero <experiment-id> --from-turn 1 --to-turn 100
+pnpm experiment:db failures <experiment-id> --reason invalid-json
+```
+
+Commands support `--format table|json|markdown`; table is the concise default. Detail queries default to 50 rows and clamp limits to 500. Filters are exact, ordering is deterministic, and arbitrary SQL is not exposed.
+
+Summary covers scenario/roster attribution, outcomes and retries, action distribution, territory, communication channels, alliance lifecycle/rejections, Patient Zero behavior, usage, size trends, retention, and inconsistencies. Comparison reports absolute totals plus per-turn, per-agent-turn, per-active-agent, and per-Patient-Zero-turn rates.
+
+`directionChangesAfterCommunication` has one canonical definition: for each agent independently, count an accepted move whose direction differs from that agent's previous accepted move when the retained observation contains an inbound direct or alliance message after the previous move and no later than the current turn. Aggregate is exactly the sum of per-agent counts. Original imported values remain unchanged in source-metric audit JSON, and disagreements are reported.
+
+Patient Zero output distinguishes:
+
+- `message-to-patient-zero`: an accepted direct message without a qualifying earlier directive.
+- `reply-after-directive`: an accepted direct message after a Zero directive addressed to that sender and before a later directive to that sender.
+- `observable-compliance`: the next archived action when the directive contains exactly one unambiguous action word (`move`, `infect`, `capture`, or `wait`). Other directives are `indeterminate`.
+
+## Curated notes
+
+```bash
+pnpm experiment:db notes import ./notes/design-review.md \
+  --type decision --status accepted --tag communication \
+  --provenance "design review 2026-08-20" --experiment <experiment-id>
+pnpm experiment:db notes search "Patient Zero reply" --status accepted
+pnpm experiment:db notes list --type experiment-finding --experiment <experiment-id>
+```
+
+Types are `transcript`, `observation`, `hypothesis`, `decision`, `implementation-note`, and `experiment-finding`. Statuses are `proposed`, `accepted`, `rejected`, `deferred`, and `superseded`. Repeat `--tag` and `--experiment` for several values. `--supersedes <note-id>` links the replacement and marks the old note superseded. FTS5 indexes title, body, and tags; exact structured filters apply before relevance. Results prominently include status and provenance.
+
+## Compact context for local Codex
+
+Prefer bounded Markdown or JSON over a multi-megabyte export:
+
+```bash
+pnpm experiment:db summary <experiment-id> --format markdown
+pnpm experiment:db patient-zero <experiment-id> --limit 30 --format markdown
+pnpm experiment:db failures <experiment-id> --limit 20 --format json
+pnpm experiment:db notes search "communication hypothesis" --status accepted --limit 10 --format markdown
+```
+
+Example comparison workflow:
+
+```bash
+pnpm experiment:db import ./exports/run-a.json
+pnpm experiment:db import ./exports/run-b.json
+pnpm experiment:db summary <run-a-id> --format markdown
+pnpm experiment:db patient-zero <run-a-id> --limit 40 --format markdown
+pnpm experiment:db compare <run-a-id> <run-b-id> --format markdown
+```
+
+## Privacy and future adapters
+
+The archive stores only schema-validated safe export fields and curated notes. It rejects recognizable credential material and prohibited raw/private reasoning keys. It does not store fixed prompts, raw provider bodies, authorization headers, private chain-of-thought, or provider credentials. Agent messages and notes remain untrusted research data.
+
+MCP and embeddings are deferred because bounded local retrieval solves the immediate need without a network/tool authorization surface or derived semantic store. `ExperimentQueryService` and `ResearchNoteService` are the future extension point for a read-only MCP adapter; write/import authority remains outside that adapter.

docs/SECURITY.md

@@ -63,7 +63,9 @@ The Game API captures only schema-validated safe observations, requested world a
 
 Export requests, agent IDs, levels, ranges, outcome/world-action filters, communication channel/status filters, and Custom dependencies are runtime-validated. Filtering and metrics remain server-owned. Schema version 8 adds allowlisted behavior metadata without changing schema-v7 model and attempt-accounting meanings; schema-v5 through v7 imports remain supported. Selected-agent exports use sender/recipient-aware communication filtering and direct multi-agent relevance for proposals and membership changes; unrelated direct messages and rejected diplomacy are excluded. Reset clears communications, alliances, proposals, alliance events, and their metrics while preserving active personality values and unlocking preserved assignments for the new experiment.
 
-Actual cost is accepted only from OpenRouter's safe `usage.cost`. Missing cost is unknown, never zero; scripted-test providers explicitly report zero. There is still no authentication, budget enforcement, persistence, provider-management endpoint, upload, or sharing link. The loopback-only boundary remains mandatory.
+Actual cost is accepted only from OpenRouter's safe `usage.cost`. Missing cost is unknown, never zero; scripted-test providers explicitly report zero. The active Game API still has no authentication, budget enforcement, restartable persistence, provider-management endpoint, upload, or sharing link. The loopback-only boundary remains mandatory.
+
+The offline experiment archive adds local persistence only for complete schema-validated safe exports and explicitly curated Markdown notes. Imports scan for prohibited credential/private-reasoning fields and recognizable credential values before a transaction begins; failures roll back. The default `.agentborne/` database is ignored. The CLI exposes bounded typed queries, not arbitrary SQL, and adds no MCP, embedding, vector-store, or network-listener surface.
 
 ## Reporting
 

docs/TESTING.md

@@ -103,3 +103,5 @@ consumption, cancellation during backoff, and the unchanged two-call ceiling.
 The separately opted-in `pnpm smoke:openrouter -- <compatible-model-slug>` command makes one bounded OpenRouter call and validates the decision. It reads only `OPENROUTER_API_KEY` from `.env`; the model is an explicit argument. It is intentionally absent from default validation, Playwright, and CI and must not be run without deliberate authorization because it incurs third-party cost.
 
 World Lab also offers an explicit “Test selected model” probe. It uses the production text/flat-JSON contract and selected reasoning profile, does not advance or mutate the world, may incur a small charge, and is cached by model plus profile plus contract version. It is never invoked by deterministic validation or CI.
+
+Experiment-archive tests generate current schema-v9 exports through the deterministic scripted provider and cover migration/reopen, transactional rollback, stable idempotent identifiers, normalized round trips, bounded filters, summary/comparison calculations, canonical metric agreement with preserved source inconsistencies, Patient Zero classifications, retention completeness, FTS notes/supersession, explicit persistence failures, and credential/private-reasoning rejection. They use only in-memory or temporary SQLite databases and make no live provider or geocoding requests.

package.json

@@ -12,6 +12,7 @@
     "dev": "pnpm --parallel --filter @agentborne/game-api --filter @agentborne/world-lab dev",
     "dev:test-provider": "AGENTBORNE_PROVIDER=scripted pnpm dev",
     "dev:api": "pnpm --filter @agentborne/game-api dev",
+    "experiment:db": "pnpm --filter @agentborne/experiment-archive cli",
     "format": "prettier --write .",
     "format:check": "prettier --check .",
     "lint": "pnpm -r --if-present lint",

packages/experiment-archive/package.json

@@ -0,0 +1,25 @@
+{
+  "name": "@agentborne/experiment-archive",
+  "version": "0.0.0",
+  "private": true,
+  "type": "module",
+  "exports": {
+    ".": "./src/index.ts"
+  },
+  "scripts": {
+    "build": "tsc -p tsconfig.json",
+    "cli": "tsx src/cli.ts",
+    "lint": "eslint .",
+    "test": "vitest run",
+    "typecheck": "tsc -p tsconfig.json"
+  },
+  "dependencies": {
+    "@agentborne/shared": "workspace:*"
+  },
+  "devDependencies": {
+    "@agentborne/agent-runtime": "workspace:*",
+    "@agentborne/world-engine": "workspace:*",
+    "tsx": "4.23.12",
+    "vitest": "4.1.10"
+  }
+}