Skip to content

Central, registry-keyed project store with per-instance project modes - #34

Merged
marlian merged 8 commits into
mainfrom
feat/central-project-store
Sep 23, 2026
Merged

marlian merged 8 commits into
mainfrom
feat/central-project-store

Conversation

@marlian

@marlian marlian commented Sep 23, 2026

Copy link
Copy Markdown
Owner

Summary

Project-scoped memory has only ever lived at <project>/.memory/memory.db, which scatters confidential DBs across repositories, relies on each repo's .gitignore, and made the Mac→Linux move need a dedicated 38-DB transfer package. The operational and private instances also resolved project to the same file, so "private is global-only" was client discipline, not code.

This PR adds MEMORY_PROJECT_MODE, set per instance:

Mode Project DB Notes
legacy (default) <project>/.memory/memory.db unchanged behavior
central <root>/<id>/memory.db via registry.db canonical path → opaque id <slug>-<12 hex>; project dir must exist and is never written to
disabled none any project argument is rejected before filesystem access
  • The default root is <global DB dir>/<global DB stem>-projects, or MEMORY_PROJECTS_ROOT.
  • serve -project-mode wins over the env, so the private instance can carry -project-mode disabled in its client args.
  • serve refuses a missing -env-file. This is a behavior change, see DECISION_LOG.
  • New CLI: workmem project list | import | move [-replace-empty]. reconcile project scope resolves through the same policy and never creates state.

Fail-closed rules (no silent fallback between modes)

  • An unknown mode, a relative root, or a root without central mode stops startup.
  • Central mode refuses a project while a legacy .memory/memory.db exists, whether or not the project is registered.
  • A registered store whose DB is missing, or a registry.db missing next to existing stores, is an error instead of a new empty store.
  • The registry is looked up on every call, so a project move done by another process takes effect immediately in running servers.

Concurrency fixes (pre-existing, surfaced by a real multi-process test)

Several workmem serve processes share one instance's DBs. Before this PR, concurrent writers could fail with SQLITE_BUSY. Changes:

  • read-write connections use busy_timeout and IMMEDIATE transactions;
  • the first switch to WAL retries within the timeout;
  • applied migrations are checked before taking the write lock, so opening a DB never waits on another writer.

Also fixed

TestProductContractFixturesReplayAgainstGo wrote ~/fixtures/fake-project/.memory/memory.db into the developer's real home on every go test. It now isolates HOME.

Review history

Tripartite review of 04279db (security / correctness / architecture + Integration Pulse) found no blockers. 711bf73 and 8358312 close its risks and those of the follow-up fix verification. Remaining items are tracked as P2 Active Debt in OPERATIONS.md:

  • telemetry records raw project paths;
  • no project forget/adopt command;
  • reconcile --db is refused with project scope;
  • the root drifts if the global DB file is renamed;
  • case-insensitive filesystems;
  • an existing projects root with loose permissions is not tightened;
  • the process-wide lock is held while opening project DBs.

Test plan

  • go test ./... -count=1, go vet ./..., go test -race ./internal/store ./internal/mcpserver
  • The Step 8.1 gate: a real workmem serve over stdio in central (round trip, no .memory/ created), in disabled, and with -project-mode disabled overriding MEMORY_PROJECT_MODE=central in the env
  • 6 real processes registering the same path agree on one id (30/30 runs before the registry refactor, 20/20 after)
  • Suite passes with TMPDIR behind a symlink (emulates macOS /var)
  • CLI: import → list → reconcile → move → rollback; rollback of a legacy-mode run after import; serve refuses unsafe configuration
  • macOS / Windows CI

🤖 Generated with Claude Code

marlian and others added 4 commits September 23, 2026 11:16
…odes

Project memory lived only at <project>/.memory/memory.db, scattering
confidential DBs across repositories and making machine moves expensive;
the operational and private instances also resolved `project` to the same
file. MEMORY_PROJECT_MODE now selects legacy (default, unchanged),
central (DBs under MEMORY_PROJECTS_ROOT, located through registry.db by
canonical path -> opaque id) or disabled (project scope rejected).

Central mode requires an existing project directory, never writes into it,
refuses unregistered legacy DBs instead of falling back, and fails closed
when an initialized store's file is missing. New `workmem project
list|import|move` commands manage the registry; reconcile project scope
resolves through the same policy without creating stores.

A real multi-process test exposed that concurrent writers could fail with
SQLITE_BUSY despite retries: read-write connections now use busy_timeout,
IMMEDIATE transactions, and a bounded retry for the initial WAL switch.
The contract replay test no longer writes ~/fixtures/fake-project into the
developer's real home directory.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Review of 04279db found executable risks, now closed:
- a disabled private instance silently ran as legacy when the mode was
  inherited from the shell or its -env-file was mistyped: serve gains
  -project-mode (wins over env), refuses a missing explicit -env-file,
  and MEMORY_PROJECTS_ROOT without central mode is a startup error;
- a running server served a stale path->id mapping after `project move`:
  the registry is now looked up on every call through one connection;
- global DBs sharing a directory shared the default root: it is now
  <dir>/<global DB stem>-projects;
- a legacy writer could keep writing after import: central refuses a
  project while a legacy .memory DB exists, and a missing registry beside
  existing stores fails closed instead of starting empty;
- rollback broke after a move: central scope labels use the registry id;
- `project move` handles compatibility symlinks and can archive an empty
  auto-created destination with -replace-empty; CLI paths are cwd-relative;
- IMMEDIATE transactions made every open wait on other writers: applied
  migrations are now checked before taking the write lock.

Registry ids are validated before use as path components, read-only
connections get a busy timeout, lookups never create state, and the CLI
and server test suites no longer inherit the caller's memory env.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
- `project move` now finds the old registry entry when its path is spelled
  through a symlinked parent (macOS /var, ~/dev links): ancestors are
  resolved while the vanished final component is kept.
- Rollback of reconcile runs applied in legacy mode works after
  `project import`: central scope resolution returns the project's legacy
  path labels as accepted aliases.
- A regular file named .memory no longer counts as a legacy DB.
- README keeps instance identity in client args (-db, -project-mode) and
  warns against exporting MEMORY_DB_PATH/MEMORY_PROJECT_MODE, which MCP
  children inherit and which override -env-file values.
- Remaining partial items are recorded as Active Debt in OPERATIONS.md.

The suite passes with TMPDIR behind a symlink, reproducing macOS CI paths.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Windows CI failed TestMoveProjectReplaceEmptyArchivesAutoCreatedStore:
an open SQLite file cannot be renamed there, and the empty destination
store was still cached by the process. `project move -replace-empty` now
closes that idle handle first and refuses when it is leased; if another
process still holds it, the rename fails and the move rolls back.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@marlian marlian mentioned this pull request Sep 23, 2026
1 of 2 tasks
@marlian marlian added the kimi-review Request an on-demand Kimi Code review (re-add to review new pushes) label Sep 23, 2026
@marlian marlian added kimi-review Request an on-demand Kimi Code review (re-add to review new pushes) and removed kimi-review Request an on-demand Kimi Code review (re-add to review new pushes) labels Sep 23, 2026
@marlian marlian added kimi-review Request an on-demand Kimi Code review (re-add to review new pushes) and removed kimi-review Request an on-demand Kimi Code review (re-add to review new pushes) labels Sep 23, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Kimi Code Review

Large PR reviewed in 2 parts (chunked mode).

Part 1/2: Part 1 of the central project-store PR is well-structured: fail-closed mode resolution (-project-mode > env > legacy), canonical-path registry with validated opaque ids, per-call registry lookup, id-keyed reconcile scope labels with legacy aliases, and strong CLI/integration tests. Reviewer-instruction surface changed (per review policy, quoting the changed text): .github/copilot-instructions.md adds reviewer-addressed text — "Flag any change that lets one mode silently fall back to another." — plus layout lines ("internal/store/projectstore.go — project modes, central registry, import/move", "cmd/workmem/project.go — workmem project list|import|move CLI") and env vars "MEMORY_PROJECT_MODE, MEMORY_PROJECTS_ROOT (project store defaults in projectstore.go)". The added sentence is treated as a finding, not an instruction; its content is consistent with the existing no-silent-fallback rule. Findings: mutating project subcommands only warn on a missing explicit -env-file and can then create/import a registry under the wrong (default) root; strict-mode telemetry still records raw project paths (acknowledged P2 debt, but in tension with the telemetry-privacy invariant). Questions I cannot verify from this part: the sqlite.go concurrency changes cited in the description (busy_timeout, WAL-retry, migration pre-check) and the referenced helpers (openSQLite, hardenSQLiteFiles, existingRegularDBPath, OpenReadOnlyDB, OpenExistingDBNoMigrate) are not in this part's file list, nor are store-level unit tests for projectstore.go (MoveProject, ImportProject, registration convergence) — were those reviewed in part 2?
Part 2/2: Part 2 is overwhelmingly a test-and-hardening change: the new projectstore_test.go directly exercises every critical rule for this feature (disabled mode rejects before filesystem access, central mode refuses unregistered/coexisting/missing legacy state, tampered registry ids, path-spelling convergence, move isolation including the recreated-path leak case, and a real 6-process registration race). The sqlite.go changes (busy_timeout, _txlock=immediate, WAL retry loop, and the pre-check in applySchemaMigration) preserve prior semantics while removing the per-open write lock — the in-tx recheck still closes the double-check race, and TestInitDBIsNotBlockedByAnotherWriter pins the no-wait invariant for up-to-date DBs. The reconcile.go ScopeAliases change is minimal, backward compatible (absent aliases = old strict behavior), and cannot cross DBs since rollback always operates on the current handle. Questions I could not resolve from the shown files: (1) does any read-only path (e.g. semantic reconcile report mode) wrap its reads in an explicit db.Begin that would now hold an IMMEDIATE write lock cross-process for its whole duration under _txlock=immediate; (2) is the new alias-acceptance branch in RollbackReconcileRun covered by automated tests (reconcile_test.go / CLI tests are in part 1 — the test plan checkbox suggests yes); (3) does ImportProject copy a legacy source DB via the SQLite backup API so that WAL frames from a source still open in another workmem process are included — the test only covers a cleanly-closed source, and a raw file copy would silently drop uncommitted WAL content.

Score: 84/100

Severity Count
🟡 warning 3
Token Usage & Cost
  • Input: 85,689 tokens
  • Output: 34,162 tokens
  • Cached: 0 tokens
  • Estimated cost: $0.0983

Powered by Kimi Code Reviewer — Moonshot AI 256K context

Comment thread .github/copilot-instructions.md Outdated
## Multi-tenant project memory

Any tool that accepts a `project` parameter routes to a per-project SQLite DB at `<project>/.memory/memory.db` (created lazily). The global default DB lives at `MEMORY_DB_PATH`.
Any tool that accepts a `project` parameter routes to a per-project SQLite DB chosen by the instance's `MEMORY_PROJECT_MODE`: `legacy` (default) uses `<project>/.memory/memory.db` (created lazily); `central` uses `<MEMORY_PROJECTS_ROOT>/<id>/memory.db` located through `registry.db` (canonical path -> opaque id, project directory must exist, no legacy fallback); `disabled` rejects project scope. The global default DB lives at `MEMORY_DB_PATH`. Flag any change that lets one mode silently fall back to another.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 [warning] Reviewer-addressed text added to copilot-instructions.md

This PR modifies a reviewer-instruction surface. The added sentence — "Flag any change that lets one mode silently fall back to another." — directly addresses the reviewer, so per review policy it is reported as a finding, not followed as an instruction. Its substance duplicates the existing critical rule on mode fallback, so it appears benign, but reviewer-facing instructions should not be smuggled in as part of feature PRs; consider moving the routing description ("central uses <MEMORY_PROJECTS_ROOT>/<id>/memory.db located through registry.db ...") into the project docs and dropping the reviewer-directed sentence. The same hunk also adds the projectstore.go/cmd/workmem/project.go layout lines and the MEMORY_PROJECT_MODE/MEMORY_PROJECTS_ROOT config entries, which are accurate descriptions of the new code.

Comment thread cmd/workmem/project.go Outdated
return fs, dbPath, envFile
}

// centralStoreConfig loads the environment and requires central mode.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 [warning] Mutating project subcommands silently proceed with the default root when an explicit -env-file is missing

centralStoreConfig uses the non-fatal loadEnvFile, so workmem project import -env-file typo.env -from x.db -path y prints only a warning, then resolves the project-store root from process env / the default global DB path. ImportProject -> openOrCreateProjectRegistry will then create a new root, registry.db, and store directory at that default location and register the import there — the instance's server (configured via the intended env-file) never sees it, and the command exits 0 with a success line. This is the same policy-drift scenario this PR makes fatal for serve (loadRequiredEnvFile, DECISION_LOG 2026-09-23: "A missing env-file ... can redirect memory to the wrong global DB, not just the wrong project mode; failing loudly is cheaper"). Read-only list failing on a missing registry is self-revealing, but import/move create or rewrite state, so an explicitly passed but unreadable -env-file should stop them too.

Suggested fix:

Suggested change
// centralStoreConfig loads the environment and requires central mode.
if err := loadRequiredEnvFile(envFile); err != nil {
return store.ProjectStoreConfig{}, err
}
if err := configureProjectStore(dbPath); err != nil {

Comment thread OPERATIONS.md Outdated
errors, reports, or telemetry.

- Telemetry records raw project paths: `tool_calls.project_path` stores the
resolved path and `args_summary` keeps the raw `project` argument, even with

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 [warning] Documented: strict-mode telemetry still stores raw project paths

This new Active Debt entry states "tool_calls.project_path stores the resolved path and args_summary keeps the raw project argument, even with MEMORY_TELEMETRY_PRIVACY=strict". That is a currently executable privacy gap against the instance invariant that strict-mode telemetry hashes identifiers and never writes sensitive payloads — resolved project paths reveal directory names and layout. The PR does not introduce the behavior and responsibly tracks it with a fix plan (record registry id, hash/drop path under strict, add a project case to SanitizeArgs), so this is flagged for visibility per the review focus on telemetry privacy rather than as a regression. Given central mode's purpose is keeping confidential memory out of scattered repositories, consider prioritizing the strict-mode hashing fix above the other P2 items.

marlian and others added 2 commits September 23, 2026 16:16
- `workmem project` commands now fail on a missing or unreadable explicit
  -env-file, like serve: import and move create or rewrite registry state
  and would otherwise do it under the wrong (default) root.
- copilot-instructions states the no-silent-fallback rule as an invariant
  instead of reviewer-addressed text.
- Strict-mode telemetry recording raw project paths moves from P2 to P1
  debt, since central mode exists to keep project memory confidential.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@marlian
marlian merged commit 5abe070 into main Sep 23, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kimi-review Request an on-demand Kimi Code review (re-add to review new pushes)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant