feat: txstore, resolver, cascade, git-remote-gib

[shruggr]

Summary

Greenfield WS-C on a new tree (prototype stays on archive/prototype).

• File-backed txstore: get/put signed bytes, txid verified by rehash, sharded under $GIB_HOME/txstore.
• Resolver: B + ord payloads, ordfs/dir walk (depth 8, . / index.html default), ordfs/patch chain apply. Chain/txstore only — no ORDFS HTTP.
• Cascade planner (planCommit): genesis leaves + dirs; unchanged files cited, never patched.
• Commit-token PushDrop fields ["gib", origin, branch, root, pubkey]; head is PushDrop prefix + git-commit inscription.
• git-remote-gib: capabilities/list/fetch/push. Genesis push publishes content, validation-gates the resolved tree against the commit object, then seals the head. Pending-upload cache under $GIB_HOME/pending.

Test plan

• bun test (17 tests: txstore, resolver, cascade, git objects, gitread, push genesis+validate, token, recovery, pending, helper protocol)

VCDIFF on-chain profile documented in docs/plans/ordfs-formats.html.

[shruggr]

Reviewed at 566fb77 with the branch checked out locally. I ran the suite (17/17 pass), bunx tsc --noEmit, and targeted repro scripts for every claim below — the bug findings are empirically demonstrated, not code-read guesses.

What's solid: txstore put verifies txid by rehash; resolver patch-chain cycle guard; validation gates the head seal; the helper's list/push/ok/error shapes match gitremote-helpers(7) (including the loose-object fetch contract — "writing the necessary objects to the database"); genesis push works end-to-end; the four vendored src/ordfs/* files are byte-identical to the 1sat-sdk PR (diffed); spec updates in ordfs-formats.html (txid byte order, vcdiff profile) are exactly the right clarifications.

But two critical bugs mean every push after genesis currently fails, and fetch cannot work from a cold machine. The 17 tests pass because they exercise the genesis path and unit-level pieces only — push.test.ts uses origin: 'new' and localPublisher (which ignores spend), and no test covers fetch/importCommit, subdirectories in cascade, or empty files.

Blockers (inline):

1. cascade.ts — untouched subdirectories are silently dropped from the parent manifest (verified: b/y disappears from the new root when only a/x changes). Nested incremental pushes fail validation after the content tx is broadcast.
2. publish.ts — publishHead unlocks the previous head with the new token's keyID; the head was locked at the old root's keyID. Verified by key derivation comparison. Every incremental push produces an unspendable input.

High (inline):
3. Cold start: advertise throws missing tx when the head tx isn't in txstore, and nothing anywhere in the tree fetches txs from the chain — git fetch/ls-remote on a fresh machine kills the helper.
4. Empty files (.gitkeep etc.) can't round-trip: an empty B/ord payload decodes to undefined → push fails at validation, after publishing.

Medium (inline): unbounded collectTree recursion (demonstrated event-loop-starving hang); content broadcast before validation; silent head bifurcation when loadSpend misses; tsc fails under the repo's own tsconfig (81 errors); option capability vs push-batch option lines.

Nits (no inline):

• push.ts:56 — remoteSha() re-advertises and compares the result to itself; dead check plus a second full wallet round-trip.
• validate.ts:21-24 — the commit-hash check is a tautology (got.commit and expectCommit are both gitHash('commit', commitBytes)).
• publish-local.ts:7 — hardcoded new PrivateKey(4242) default in src/; require an explicit wallet argument.
• git.ts:38-42 — encodeTree sorts with UTF-16 string comparison; git sorts tree entries by UTF-8 bytes. Names mixing astral chars (emoji) and U+E000..U+FFFF codepoints order differently → tree-hash divergence. A byte comparator (like dir.ts compareBytes) fixes it.
• txstore.ts:36 — writeFile is not atomic; a crash mid-write leaves a corrupt entry that get serves unverified. temp+rename closes it.
• recovery.ts — recoverPush has no caller yet (main.ts is doctor/put); fine as plan-level groundwork, just noting.
• Once the 1sat-sdk PR ships, note the sync obligation for the vendored ordfs/* copies (or switch to the dependency).

The two blockers are small, contained fixes (cite old dir outpoints; unlock with the prev token's keyID) — the architecture around them looks right.

[shruggr]

Critical — untouched subdirectories are dropped from the published tree (data loss).

This line skips untouched non-root dirs, and then line 126 (if (v === undefined) continue) silently omits them from the parent manifest — nothing ever cites the old dir output. fileVout citations only rescue files whose parent dir is re-emitted.

Verified with a repro against this branch: genesis {a/x, b/y} → next commit changing only a/x (with prevRoot + store, full file list passed in):

[1] cascade root entries after changing only a/x: [ "a" ]
[1] genesis tree files: [ "a/x", "b/y" ]

b/y is gone from the new root manifest. In the push flow this surfaces as validateRoot failing (resolved tree != commit tree) — but only after publishContent has already broadcast, so every nested incremental push burns a tx and fails. Only genesis and flat (root-level files only) repos work; both cascade tests stay green because neither covers an untouched subdir.

The spec (ordfs-formats.html) describes the intended behavior: "manifest cascade = full re-inscribe of touched dirs, one new output per directory, parents citing child outpoints." The dir format already supports this — a dir entry with REFTYPE 1 pointing at the previous dir-manifest output resolves correctly (collectTree maps that manifest's same-tx children against the old txid). The missing piece: collectTree flattens away dir outpoints, so planCommit can't cite them. Return per-directory outpoints alongside FileEntry[] (or walk the prev manifests here) and emit {kind: 'outpoint', ...} refs for untouched child dirs instead of continue.

[shruggr]

Critical — this unlocks the previous head with the wrong key.

opts.token is the new token, so gibKeyId(opts.token.root) is the new commit's root outpoint. But the input being spent (opts.spend.outpoint) is the previous head, which sealCommitLock locked with a pubkey derived from gibKeyId(prevToken.root) (seal.ts:16). The root outpoint changes on every commit (new content tx → new txid), so the keyID changes on every commit, and the createSignature inside unlock.sign derives a different key than the one in the locking script → OP_CHECKSIG can never pass.

Verified by derivation on this branch:

head1 locking pubkey == key(old root): true
head1 locking pubkey == key(new root) [what publishHead unlock uses]: false

So every non-genesis push produces an invalid spend (rejected by the wallet host's script verification at signAction, or on-chain if it got that far). burnHead gets this right — it takes keyID: prev.root from the caller (push.ts:124). The fix is the same pattern: use the previous token's root here — either pass it through spend from pushLine (which has prev.root), or decodeCommitToken(src.lockingScript) right here since the source output is already loaded at line 103.

Untested today because push.test.ts only covers genesis (origin: 'new', no spend) and localPublisher.publishHead ignores spend entirely. A second-push test against walletPublisher with a wallet that verifies scripts would have caught this.

[shruggr]

High — cold start is impossible: nothing in the tree ever fetches a tx from the chain.

commitShaFromHead → loadTx(store, ...) requires the head tx to already be in txstore. On a fresh machine — wallet synced (so listOutputs returns the tokens), txstore empty — this throws missing tx <headTxid>. Verified with a fake wallet returning a valid sealed token and an empty memStore:

[4] advertise THROWS on cold start: missing tx ffff...ffff

And runHelper doesn't catch, so git ls-remote / git fetch gib://… kills the helper process mid-protocol. The PR body says "Chain/txstore only — no ORDFS HTTP", but the chain side doesn't exist yet: loadTx is store-only and there is no ARC/ORDFS fetch anywhere in src/. Fetch therefore only works on the machine that pushed (or after manual gib put of every tx in the tree).

Suggestions, roughly in order of payoff:

• The head tx is wallet-owned, so it's obtainable without new infrastructure: listOutputs({..., include: 'entire transactions'}) → Transaction.fromBEEF → toBinary → store.put. That bootstraps advertise + importCommit's head read.
• Content/patch txs are 0-sat and not wallet-owned — those need a by-txid chain source (ARC raw-tx endpoint, or ORDFS HTTP later). Worth an explicit TxStore decorator that falls back to chain fetch, so loadTx gains it everywhere at once.
• Alternatively/additionally: put the commit sha in the token fields (currently ["gib", origin, branch, root, pubkey]). Then advertise/list works from wallet data alone with no txstore dependency, and it also gives a deterministic tie-breaker for the multi-head case.
• Independently: wrap the list/fetch handlers in runHelper in try/catch and emit error … / die gracefully instead of an unhandled crash.

[shruggr]

High — empty files can't round-trip, and the failure lands after the content tx is broadcast.

Script.writeBin([]) emits OP_0 with data: undefined (@bsv/sdk Script.js:297-302), so for a zero-byte body !data is true here and decodeB returns undefined; decodeOrd has the same hole at line 64 (content.length === 0 → undefined). Verified:

[3] payloadFromScript(B, empty body): undefined

A repo containing an empty file (.gitkeep, empty __init__.py, …) plans a B output with an empty body, publishContent broadcasts it, and then validateRoot → collectTree → resolveOutpoint throws no content at <outpoint> — push fails after publishing, and deterministically fails again on every retry (new orphan tx each time).

Two ways out — pick one deliberately:

• Accept empty payloads: treat an OP_0/zero-length data push in the body position as new Uint8Array(0) in both decoders (this is the interop-friendly reading; note ord proper does allow empty bodies), and make sure cascade/validate are happy with zero-byte blobs (git's empty blob e69de29… must hash back).
• Or reject empty files at planCommit time with a clear error before anything is published.

Either way this needs a test — an empty file in a pushed repo is very common.

[shruggr]

Medium — collectTree has no depth cap or cycle guard; a self-referencing dir manifest hangs the process forever.

walkDir enforces MAX_DIRECTORY_DEPTH = 8 and resolveOutpoint guards patch cycles, but collectTree recurses unboundedly. A dir entry with isDir: true referencing its own output is trivially constructible — REFTYPE 0 with vout = the manifest's own index passes dirEncode validation. Verified: collectTree on such a manifest never returns and never stack-overflows — the async recursion resolves every await immediately, so the microtask queue starves the event loop. Even an in-process setTimeout watchdog never fired; only an external kill stopped it:

[5] collectTree self-loop: (process hung; killed by external timeout)

This runs on semi-trusted chain data in three places (fetch/importCommit, validateRoot, planCommit against prevRoot), so any client following a malicious or corrupted head wedges permanently. Fix: thread a depth counter (cap 8, matching walkDir and the gateway spec) plus a seen set of visited dir outpoints, mirroring the patch-cycle guard in resolveOutpoint.

[shruggr]

Medium — content is broadcast before validation, so every validation failure leaves an orphan tx (and burned fees).

publishContent runs with signAndProcess: true (publish.ts:60), and validateRoot only runs at line 75. Combined with the cascade bug (untouched subdirs dropped) and the empty-file hole, the common failure mode is: tx broadcast → validation throws → push errors → retry creates another content tx (new funding inputs → new txid → new orphan).

The tree validation doesn't actually need the on-chain tx: everything validateRoot checks is derivable from plan.outputs locally (vout = index into the plan, root = plan.rootIndex; only the txid is unknown pre-signing, and same-tx refs resolve by index). Suggest validating the materialized tree against the commit object before publishContent, keeping the post-publish validateRoot as a belt-and-braces re-check if you want it. That converts "publish garbage, fail, repeat" into "fail locally, publish nothing".

[shruggr]

Medium — when prev exists but loadSpend returns undefined, this silently mints a second live head for the same origin+branch.

loadSpend can miss even when currentToken found the token: it lists the whole gib basket with limit: 100 and no tag filter (so a wallet with >100 gib outputs can paginate past the target), and any wallet/BEEF hiccup returns undefined. The flow then proceeds with spend: undefined → publishHead takes the no-spend path → the old head token is never spent → two unspent heads with identical origin:/branch: tags.

Downstream that's ambiguity with no tie-breaker: advertise emits duplicate refs/heads/<branch> lines (git's behavior on duplicate ref advertisements is not something you want to depend on), and currentToken/the fetch handler's limit: 1 pick arbitrarily. Suggest: hard-fail the push when prev && !spend (that's the recovery scenario — surface it as such), filter loadSpend's listOutputs by the origin+branch tags, and dedupe/pick-latest in advertise (a commit sha or sequence field in the token — see the advertise comment — would give a deterministic rule).

[shruggr]

Medium — bunx tsc --noEmit fails under this tsconfig: 81 errors.

75 are TS5097 (.ts-extension imports) — add "allowImportingTsExtensions": true here (legal since noEmit is set) and they all go away; bun runs the code fine today, but the repo as shipped doesn't typecheck, which is how the other 6 slipped in:

• src/content.ts:52 and src/token.ts:37-38 — chunks[i]?.data != null doesn't narrow chunks[i].data on the following line (number[] | undefined not assignable). Capture const c = chunks[i + 2] and check c?.data once.
• src/publish-local.ts:20, test/push.test.ts:38, test/token.test.ts:16 — ProtoWallet doesn't satisfy WalletInterface (missing createAction/signAction/…). sealCommitLock only needs getPublicKey + createSignature; type its wallet parameter as Pick<WalletInterface, 'getPublicKey' | 'createSignature'> and the callers typecheck honestly.

Worth adding tsc --noEmit to the test script so this stays green.

[shruggr]

Low — advertising option while treating in-batch option lines as push specs.

Per gitremote-helpers(7), option lines don't only arrive standalone: "Zero or more protocol options may be entered after the last push command, before the batch's terminating blank line." Since the helper advertises option, git may send e.g. option verbosity 0 / option progress … inside the push batch (git push --quiet, git push -o …), and readUntilBlank at line 74 will hand those to pushLine → parsePushLine throws bad push spec → a spurious error <garbage-dst> line for a push that may have succeeded.

Simplest fix: drop option from the capabilities string — git then refuses push options up front with a clear message and never sends option lines. (Keeping it and responding unsupported to standalone ones is fine, but you'd also need to filter option … lines out of the push/fetch batches.)

[shruggr]

Low — repeated tag-0 chunks overwrite instead of append.

Ord-conformant writers split large bodies across multiple tag-0 pushes and readers concatenate them; this decoder keeps only the last chunk (content = data). Gib's own writer (appendOrdEnvelope) always emits a single body push so internal data is safe, but payloadFromScript is the resolver's general ord path — foreign ord inscriptions with chunked bodies would silently resolve to their final chunk. content = concat(content, data) (or a bodyParts array joined at the end) makes this ord-correct for free.

[shruggr]

Review (verdict: needs changes)

Reviewed at 566fb77 against gib-cli.html / gib-token.html / ordfs-formats.html, the SDK on feat/ordfs-patch, and the 1sat-stack gateway decoders. The genesis path is solid: I ran a real git push gib://new main through the helper against a stub wallet and it published content, validated the resolved tree against the commit object, and sealed the head. The head script decodes with the stack's gib decoder (fields as txid_vout strings, identity as hex, inscription as suffix), the B outputs decode with the bitcom parser, and the byte-order fix in 566fb77 is correct: dirEncode/patchEncode output fed to go-sdk NewOutpointFromBytes recovers the right txid and vout. All 17 tests pass.

Everything after genesis breaks, and none of it is covered by tests. Each item below was reproduced, not inferred.

Critical

1. Token spend is signed under the new root's keyID. src/publish.ts:105-108 calls unlock(GIB_PROTOCOL, gibKeyId(opts.token.root), …) where opts.token is the new head. The coin being spent was locked with keyID = previous root ("old coin unlocks under the old root's keyID"). burnHead gets this right (src/push.ts:124, keyID: prev.root). With Spend.validate() in the stub's signAction, the unlocking script fails ("top stack element must be truthy"); with the keyID taken from the spent output's own field 3 it passes. On mainnet every non-genesis push would be rejected after the content tx has already been broadcast, orphaning it. Fix: derive the keyID from decodeCommitToken(src.lockingScript).root, or pass prev.root through publishHead as burnHead does.

2. Cascade drops untouched subdirectories. src/cascade.ts:118-127 skips dirs not in touchedDirs, so they get no dirVout, and the parent's if (v === undefined) continue silently omits them. Nothing cites an unchanged directory's previous outpoint (collectTree returns files only). Repro: commit 2 changes only README.md → error refs/heads/main validation: resolved tree 890e3bf… != commit tree f8c2372… because src/ vanished. Same root cause: a deleted file never marks its parent touched (touchedDirs is fed only by changed/new files, lines 68-73), and an exec-bit-only change with identical bytes is never re-inscribed (lines 60-67). Fix: record each directory's outpoint in the previous-tree walk, compute touched bottom-up (any child added, removed, changed in bytes or mode, or any touched child dir), and cite untouched dirs as {kind:'outpoint'} refs.

3. list is empty against a compliant wallet. src/remote/advertise.ts:24-30 calls listOutputs with include: 'locking scripts' but never includeTags: true, then requires o.tags to contain the branch tag at line 41-42. includeTags is a separate flag in BRC-100 and wallet-toolbox omits tags unless asked. Repro: git ls-remote prints nothing, git clone yields an empty repository. Knock-on: in pushLine (src/push.ts:49-59) current is undefined so the fast-forward gate is skipped while currentToken still finds the coin, so non-ff pushes silently rewind the branch. Fix: includeTags: true, or drop the tag check since the token field already carries the branch.

4. Fetch imports only the tip commit. src/remote/helper.ts:114-130 importCommit writes one commit and its tree. Clone after three pushes fails with fatal: remote did not send all necessary objects. Fetch has to walk the head's spend chain backwards (the previous head is the token input of each head tx) importing commit and tree until it hits objects already present. Also fetch requires opts.wallet (line 50) and advertise is wallet-only, so fetching a foreign origin by walk-from-outpoint is not implemented, contrary to the PR body.

High

5. Empty files cannot be pushed. src/content.ts:64 returns undefined for zero-length content and :82-84 rejects an OP_0 push with no data. Repro: adding empty.txt → remote rejected … no content at …_0. __init__.py and .gitkeep are common. Treat OP_0 / zero-length pushes as an empty payload in both decoders.

6. Genesis never reports the new origin. pushLine returns origin but helper.ts:94 writes only ok <dst>. The remote stays gib://new, and push.ts:62,134 treat new as no-prev, so the next push mints a second unrelated genesis. Print gib://<origin> on genesis and refuse a second push to gib://new when a push:<sha> label or pending record already exists.

7. No batching or size limits. src/publish.ts:50-62 puts the whole repo in one createAction; more than 256 outputs throws DirFormatError: same-tx vout … out of range from dirEncode, so a repo of roughly 250+ files and dirs cannot push. Every blob is read via its own git cat-file subprocess (src/gitread.ts:65). Split into multiple content txs citing earlier ones by full outpoint, use git cat-file --batch, and at minimum enforce a byte cap with a clear error.

8. Pending cache is write-only. savePending (push.ts:71) is never read by any push path; recoverPush and loadPending are referenced only by tests, and main.ts has only doctor and put. After a failed push 2, push 3 republished fresh content. Consult loadPending(sha) on push, skip to validation/seal when the content txid is already in the txstore, and wire gib recover (or do it inside the helper).

Medium

9. Vendored SDK code. src/ordfs/{dir,patch,vcdiff,outpoint}.ts are byte-identical copies of 1sat-sdk/packages/actions/src/ordfs/* (only .js→.ts imports differ) and seal.ts hand-rolls what the SDK pushdrop helpers provide. The byte-order fix already had to land in both repos within the hour. Depend on @1sat/actions (git ref until published) and delete src/ordfs/.
10. /tmp scratch git dirs leak on every push (push.ts:74 mkdtemp never removed). Objects are content-addressed; write into the real gitDir or rm in a finally.
11. txstore writes are non-atomic and reads unverified (src/txstore.ts:19-36). Write to path.tmp then rename; verify the txid on read as the spec says.
12. collectTree has no depth limit or cycle guard (src/tree.ts:16-55); a same-tx entry pointing at its own vout recurses forever, and patch recursion in resolver.ts:57-63 is unbounded. Foreign manifests are untrusted input.
13. Tautological old-sha check. push.ts:56-58 compares advertise() with remoteSha(), which calls advertise() again, so the mismatch branch can never fire.
14. loadSpend lists 100 untagged outputs and hopes (push.ts:160-168). Filter by origin and branch tags with limit: 1.
15. tsc fails and there is no typecheck script. tsconfig.json lacks allowImportingTsExtensions, plus real errors at src/content.ts:52 and src/publish-local.ts:20 (ProtoWallet is not a WalletInterface). Bun runs it regardless.
16. Uncaught throws crash the helper (helper.ts:50,55,67) so git prints "remote helper died" with a stack trace. Catch, report on stderr, exit 1.

Low

• Tree-entry sort uses JS string comparison (src/git.ts:38-42); git compares raw bytes. Differs for names mixing U+E000–U+FFFF and astral chars.
• Non-UTF-8 paths are mangled by Response.text() (gitread.ts:57-62); submodule entries are silently skipped (:64). Reject submodules explicitly.
• Depth > 8 pushes fine but the gateway (maxDirectoryDepth = 8) cannot serve it; add a writer-side check.
• No HEAD symref advertised, so fresh clones warn about a nonexistent remote HEAD (known open item).
• src/publish-local.ts is test-only (hard-coded key) but lives in src/. Dead code: resolver.ts parseRoot, publish.ts export { LockingScript }, pending.ts listPending.
• advertise() runs two to three times per push; cascade.ts:135 opts.files.find inside the entry loop is O(n²).
• No golden-vector test for the outpoint wire order (the SDK has one); it could regress silently here.

Tests

17/17 pass. Not tested, and each gap hides one of the bugs above: non-genesis push (patch path, dir citation, deletes, mode-only change), token spend signature validity, includeTags, any fetch or clone, force push, branch delete, multi-hop patch chains, empty files, >256 outputs, helper protocol against real git. The stub-wallet harness that surfaced items 1 to 6 was about 80 lines (persist outputs to JSON, Spend.validate() in signAction, a PATH shim for git-remote-gib); landing something like it under test/e2e/ would have caught all of them.

🤖 Generated with Claude Code