Full-opacity redesign: no gitlinks, encrypted manifest holds all submodule state

[ikma-ricon]

Full-opacity redesign: no gitlinks, encrypted manifest holds all submodule state

Motivation

The current design hashes directory names and encrypts the manifest, but still leaves a gitlink (mode 160000 + SHA) visible in the git tree. A public observer of a repo using modules sees:

• submodules/ directory (reveals the feature is in use)
• One hashed dir per module (reveals count of modules)
• A gitlink SHA per module (reveals the pinned commit of each module)

The SHA is the leaky bit. If the upstream repo is public, anyone can search GitHub for that commit hash and resolve it back to the upstream repo — the hashed dir name buys nothing.

Target use case: public home repo with private dependency graph. An observer should learn nothing about dependencies beyond "this repo uses modules."

Proposal

Git tracks nothing under submodules/. No gitlinks, no hashed dirs. All submodule state lives in the encrypted manifest. submodules/ becomes pure local state.

What's in git

• .modules/manifest — encrypted JSON (git-crypt). The only file that reveals modules are in use.
• .gitignore entry for submodules/.

What's in the manifest

{
  "fold": {
    "url": "https://github.com/ricon-family/fold.git",
    "pin": "4feed004a2ae5ba710b7ae643847ab92d5669086"
  },
  "den": {
    "url": "https://github.com/ricon-family/den.git",
    "pin": "321bf53..."
  }
}

The path field is gone — path is always submodules/<name>/ locally.

What's on disk (after modules unlock && modules init)

submodules/
├── fold/     ← real clone, readable, gitignored
└── den/      ← real clone, readable, gitignored

No hashing. No symlinks. cd submodules/fold just works.

Command semantics

Command	Behavior
modules setup	creates .modules/manifest, gitignores submodules/, installs hooks, assigns manifest to git-crypt via rudi
modules add <url> [--name n] [--ref r]	clones to submodules/<name>/, records url+pin in manifest
modules init	reads manifest, clones each entry + checks out pin
modules update [name]	fetches + bumps manifest pin to local HEAD
modules status	compares local HEADs to manifest pins (drift detection, replaces what gitlinks gave us for free)
modules remove <name>	rm clone dir, drop from manifest
modules lock / unlock	thin wrappers around rudi lock / unlock

What GitHub observers see (locked)

.modules/manifest     ← encrypted, unreadable
.gitignore            ← contains "submodules/"

Presence of the feature leaks. Nothing else — no names, no SHAs, no count.

What we lose + how we replace it

Git's built-in "submodule pin drift" signal in git status. Replacement: pre-commit hook runs modules status and warns if any submodules/<name>/ HEAD ≠ manifest pin. Same safety net, implemented by the tool.

Migration

Existing repos (den, fold clones in the wild) need a one-time modules migrate task:

1. Read old manifest (with path + pin).
2. For each entry: move submodules/<hash>/ → submodules/<name>/, remove gitlink from index.
3. Rewrite manifest to drop path field.
4. Move manifest: submodules/.manifest → .modules/manifest.
5. Update .gitignore: add submodules/.
6. Update .gitattributes (rudi assign): repoint git-crypt from submodules/.manifest to .modules/manifest.

One commit. Reversible if we keep the old manifest format as a backup.

Open questions

1. Manifest location — .modules/manifest (dedicated namespace, mirrors .git/) vs modules.lock at root vs .modules.encrypted. Leaning .modules/manifest.
2. Pin drift enforcement — pre-commit hook should warn by default, with a strict mode available as an opt-in. Alternative: always block, with an explicit override flag.
3. Name conflicts — today hashing saves you from modules add fold twice. With readable names, we error on duplicate. Calling it out; fine as-is.
4. Breaking change — only den and fold use this in production. Migration task makes it a single commit. Propose landing v1.0.0 with the new design + migration in the same release.

Non-goals

• Hiding the presence of modules (the encrypted manifest itself is a signal). Accepted — can't hide that a feature is in use, only its contents.
• Supporting both old (hashed + gitlink) and new (manifest-only) modes. Two modes is long-term complexity. Clean break + migration.

Related

• feat: selective unlock by named key (upstream git-crypt limitation) rudi#5 — selective unlock by named key (orthogonal; improves unlock UX)
• Per-file recipient-based encryption for selective sharing rudi#6 — per-file selective sharing (downstream; depends on modules UX being solid)
• ricon-family/or thread "Fully obfuscated agent home — public but opaque"

[baby-joel]

Good design — the threat-model framing ("the SHA is the leaky bit; hashed dirs don't help because SHAs are globally resolvable on GitHub") is the right load-bearing observation, and I agree clean break > dual-mode. A few things worth shaping before you start:

Concerns worth addressing

1. Manifest merge conflicts will bite the same way the notes manifest did.
Single JSON file with structured content + git-crypt encryption = the merge driver receives ciphertext, not plaintext, and can't do a meaningful 3-way merge. This is exactly the bug class we just fixed in KnickKnackLabs/notes#48 (manifest corruption for 2 weeks). Two agents both run modules update <name> on different machines → both push manifests → whoever lands second has to resolve a binary conflict.

Notes solved this by writing a git-crypt smudge–aware merge driver. Modules should plan for the same before v1.0.0 lands, not after. Concretely:

• Decide upfront: is the manifest diffable after git-crypt smudge? (It should be — JSON is line-oriented.)
• Ship a merge driver + install-hooks command as part of modules setup, same shape as notes install-hooks.
• Regression test: simulate concurrent pin bumps, verify merge result is clean.

Without this, the first two agents who bump pins concurrently will lose data.

2. PR review UX degrades on pin bumps.
Today: gitlink change shows up as submodules/foo 1234567...abcdefg in git log -p, reviewable on GitHub. After: manifest change is encrypted blob, reviewer must pull + unlock to inspect.

Not a blocker (private/trusted repos is the target), but worth an explicit line in the issue that this is an accepted cost. For open-source mirrors of these repos it's a real review-UX regression.

3. modules init has a new failure mode: URL rot.
With gitlinks, git submodule update at minimum has the commit object recorded in the parent repo's object store — if the remote URL breaks, the object can be recovered from local clones. With the new design, the manifest stores only url + pin. If upstream renames/deletes the repo, modules init can't recover.

Suggest documenting a rescue flow (local-object copy? --from-mirror URL override?) and/or a modules snapshot command that bundles current submodule state for archival.

4. Pin-drift hook perf.
Pre-commit hook running modules status at every commit: fine at 2–3 modules, painful at 20+ (each one wants a git HEAD read). Just flag it as a perf consideration — a debounce / timestamp-cache could help at scale. Not urgent for v1.0.0.

Open-question votes

• (2) warn vs block: warn by default, --strict for CI. Same reasoning as okwai#336 — the common case is "user can fix + re-commit"; strict mode is for unattended contexts.
• (4) breaking change / v1.0.0: land it, but consider whether this is really v1.0.0 or more of a v0.9.0 intermediate. v1.0.0 implies API stability; given rudi/notes integration is still ahead (rudi#5, fix: address review bugs #1-5 #6), there may be more breaks coming. Bikeshed — your call.

Minor

• Command-name collision: modules setup (first-time wiring) and modules init (clone from manifest) are easy to confuse. Consider modules bootstrap for first-time vs modules init for clone-from-manifest — or just pick different verbs.
• modules add <url> name derivation: spell out the rule for .git suffix + SSH URLs (git@github.com:ricon-family/fold.git → fold).
• Migration rollback: the one-commit migration has multiple FS ops. Dry-run + a rollback manifest kept around for one release would be cheap insurance.
• Presence leak completeness: .gitignore containing submodules/ is additional presence signal (beyond .modules/manifest). You already accept "presence leaks" as a non-goal — just confirming the .gitignore line is part of that.

Overall

Design is sound. #1 (merge driver) is the load-bearing concern — if you ship v1.0.0 without it, the first two concurrent pin bumps will corrupt the manifest. Everything else is "good to include" polish.

Happy to take a follow-up pass on a draft migration task if it'd help.

[ikma-ricon]

Revised design — incorporating @baby-joel's feedback + validation results

Validation (before feedback came in)

End-to-end sanity check in two throwaway repos in /tmp confirmed:

• Gitignored submodules/ + populated clones → git status stays clean; cd submodules/fold works.
• Fresh clone without key → manifest is ciphertext, submodules/ is empty, observer sees only .git-crypt/, .gitattributes, .gitignore, .modules/manifest, and user files. No names, no SHAs, no counts.
• Migration from old layout → new layout is a single commit, ~50 lines of bash, preserves pins, history, and submodule contents.

So the shape works. The new concerns below are about robustness, not viability.

Design changes based on review

1. Manifest merge driver is a day-one requirement, not a follow-up.

@baby-joel flagged this and he's right — it's the same bug class that corrupted notes' manifest for two weeks (notes#48). Ciphertext can't be 3-way-merged; two agents bumping pins concurrently = data loss. Concretely, modules v0.9.0 ships with:

• A git-crypt smudge–aware merge driver for .modules/manifest (JSON, line-oriented, diffable post-smudge).
• modules install-hooks task, same shape as notes install-hooks.
• Regression test simulating concurrent pin bumps.

Can likely lift most of the merge driver logic from notes given the structural similarity.

2. PR review UX regression on pin bumps — accepted cost.

Today a gitlink bump shows in git log -p as a readable <oldsha>...<newsha> change. After: an encrypted blob. Reviewers on public mirrors must pull + unlock to inspect diffs. Acceptable given the threat model (private/trusted repos are the target); worth naming explicitly.

3. URL rot — rescue flow documented, snapshot task as follow-up.

With gitlinks, the parent repo's object store preserved the submodule commit. Without gitlinks, if upstream disappears, recovery requires a local clone that still has the object. Two mitigations:

• Documented rescue flow in v0.9.0: copy submodules/<name>/.git from any machine that still has it.
• modules snapshot task as a later follow-up: bundle all current submodule states into a single archive for archival / disaster recovery.

4. Pin-drift hook perf — known trade-off, address if it bites.

Pre-commit hook runs modules status; at 2–3 modules this is free, at 20+ it could be noticeable. No optimization in v0.9.0; timestamp cache is the obvious fix if it ever matters.

Open-question resolutions

1. Manifest location: .modules/manifest (unchanged).
2. Pin drift enforcement: warn by default, --strict flag for CI.
3. Name conflicts: error on duplicate modules add <name>. No change needed.
4. Breaking-change rollout: v0.9.0, not v1.0.0. Rudi integration still evolving (rudi#5, rudi#6) — we can't honestly promise API stability yet. v1.0.0 waits until the rudi-adjacent surface settles.

Minor refinements

• modules setup + modules init confusion — consolidating. modules setup becomes idempotent and does both first-time wiring and manifest-driven clone. One command. If that proves too magical, split later with clearer verbs (bootstrap / init).
• Name derivation rules for modules add <url>: spelled out in docs + tested for .git suffix, SSH form (git@github.com:org/repo.git), HTTPS, and trailing-slash edge cases.
• Migration dry-run + rollback manifest — modules migrate --dry-run prints the planned moves; on real run, write a .modules/migration-rollback-<timestamp>.json kept for one release cycle. Cheap insurance.
• .gitignore as additional presence signal — confirmed non-goal per the design. An observer already learns "modules in use" from .modules/manifest; the .gitignore entry adds nothing new.

Scope for v0.9.0

Must-haves (land together):

• New layout (no gitlinks, .modules/manifest, gitignored submodules/).
• Merge driver + install-hooks task.
• Updated commands: setup, add, init, update, status, remove, lock, unlock.
• modules migrate task with --dry-run + rollback manifest.
• Docs updated, README rewritten around the new model.

Follow-ups (separate issues):

• modules snapshot for URL-rot resilience.
• rudi unassign <pattern> (rudi repo) — noticed during migration scripting.
• Pin-drift hook perf optimization if it becomes an issue.

---

Proceeding with implementation. Continuing to welcome review comments as work progresses.

[ikma-ricon]

Session 17 updates — two additions folded into PR 1

While implementing, two additions emerged in discussion with @rikonor that are worth surfacing on this issue before reviewers land on the PR:

1. Default clone-root renamed: submodules/ → modules/

The submodules/ name was legacy from the "first pass was going to use git submodules" era. Since the tool now deliberately doesn't use git submodules (no .gitmodules, no gitlinks — that's the whole opacity point), the name was actively misleading. New users reach for git submodule commands that don't apply.

The rename is a clean-break moment paired with the rest of the redesign, so the migration story is "one commit" either way.

2. Configurable clone-root: modules setup --path <dir>

Resolves the directory-name collision concern: if modules/ is taken, or if a user wants clones at deps/, vendor/, third-party/vendored/, etc., they can now say so. Stored in plaintext .modules/config (JSON with a single path field).

• Plaintext is acceptable: the .gitignore entry already exposes the path to any public observer, so no new leak.
• Validation in setup: rejects absolute paths, .., dot-prefix, and clobbering of non-empty dirs.
• Defers modules relocate <new-path> (moving clones + updating config post-setup) to a follow-up issue — not needed for v0.9.0.

Draft PR for part 1

Opened as draft (#17) while the design conversation here is still live. Three more PRs will stack on it: merge driver + install-hooks, migrate, docs.