Skip to content

Storage schema versioning makes additive changes as expensive as breaking ones #85

Description

@nnennandukwe

Summary

Adding one value to an enum forced a full storage schema bump, a table rebuild, and a breaking change to every
pinned consumer. The change was purely additive and backward compatible. The versioning system could not
express that, so it charged the full price of a breaking migration.

Found while implementing #78, which adds a setup_failed gate result.

What one additive change actually cost

gate_receipts.result carries a CHECK constraint enumerating the allowed results
(src/adapters/fs/sqlite-store.ts:2076). Admitting one more value required:

  1. A table rebuild, because SQLite cannot ALTER a CHECK constraint. That means dropping and recreating the
    three append-only immutability triggers on the table that holds signed evidence.
  2. CURRENT_SCHEMA_VERSION 7 -> 8.
  3. Every existing repository reporting migration_required until an operator runs threadloop init,
    including the consumer pilot's, mid-pilot.
  4. data.lifecycle.storage_schema_version = 7 in .agents/skills/threadloop-runner/SKILL.md, which fails
    closed on exact equality, so every consumer pinned to a reviewed skill snapshot breaks.
  5. Eight hardcoded '7' assertions across four test files, plus a "rejects a newer schema" fixture that had
    to move from 8 to 9 because its sentinel collided with the new current version.

Nothing about a wider result enum changes lifecycle semantics. A consumer pinned to storage version 7 has no
reason to care. It breaks anyway.

Root causes

One global integer for unrelated concerns. A widened enum on gate_receipts, a new lifecycle state, and a
destructive column rewrite all move the same number. Consumers cannot tell which happened, so they must treat
every bump as breaking.

Exact-equality pinning. Consumers assert storage_schema_version = 7. Any bump is a break by
construction, so there is no way to ship a forward-compatible change.

A domain allowlist duplicated into the least alterable place. GATE_RECEIPT_RESULTS
(src/domain/proof.ts:4) is already the authority on valid results, and validateProofPlan already rejects
anything else before persistence. The SQL CHECK restates that list in the one location SQLite makes
practically immutable. The second copy bought no safety the domain did not already provide, and cost a
migration.

The version count is growing independently. protocol, proofPlan, sessionNext, signedReviewReceipt,
auditEvent, and handoff are each separately versioned and separately pinned, plus the storage version.
Each new contract multiplies the pin surface.

Existing precedent that already works better

The codebase already has additive migration that does not bump anything:
ensureSessionHeartbeatColumns, ensureTaskIssueRefColumn, and ensureTaskLifecycleColumns add columns
idempotently inside runPendingMigrations. There are also shape assertions -- assertProofSchemaShape,
assertTransitionSchemaShape, assertSignedReceiptSchemaShape -- that verify the database has what the code
needs rather than trusting an integer.

The shape-assertion approach is the one to generalize. It answers the question that actually matters, "can
this code operate on this database", instead of the proxy question, "does this integer match".

Options

  1. Split additive from breaking. Keep the integer for destructive or semantic migrations only. Additive,
    backward-compatible changes -- new enum value, new nullable column, new index -- are applied idempotently
    and do not bump it. Requires a stated rule for which is which, enforced in review.
  2. Capability descriptors instead of an ordinal. Publish a set of capability names the storage supports and
    have consumers assert the capabilities they use. A consumer that never reads gate results is unaffected by
    a widened result enum.
  3. Minimum-version pinning plus capabilities. Consumers assert storage_schema_version >= N rather than
    equality, with capabilities covering anything finer. Removes the exact-match break for every additive
    change.
  4. Stop restating domain enums in SQL. Let the domain own value domains and keep SQL constraints to
    structural invariants that cannot drift, such as length(sha256) = 64 and NOT NULL. This specific change
    would then have required no migration at all.
  5. Generalize shape assertions. Derive the required shape from a declarative description and assert it at
    startup, so the version becomes advisory rather than a gate.

Options 4 and 3 together would have made #78 a zero-migration change. Option 1 is the smallest step that
stops the ratchet.

Acceptance criteria

  • A documented rule distinguishing additive from breaking storage changes, with the additive path requiring no
    version bump and no operator migration.
  • Consumers can express what they depend on without pinning an exact storage version.
  • Adding a value to a domain enum does not require a table rebuild.
  • The number of independently pinned version integers is reduced, or their pinning is made non-breaking for
    additive changes.
  • The threadloop-runner skill's pinning strategy is updated so an additive storage change does not invalidate
    reviewed snapshots.
  • Existing repositories keep working, and prior recorded evidence keeps its current meaning.

Not in scope

The #78 migration itself. That one is already landing the expensive way, because the cheaper way does not
exist yet. This issue is about making the next one cheap.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions