Add allorad debug-store commands for multistore recovery

[kihahu]

Summary

• An interrupted store migration (e.g. the fast-node "Upgrading IAVL storage" step killed mid-flight) can leave the root multistore's s/latest version ahead of a lagging store.
• The stock rollback command (wired via server.AddCommands → server.NewRollbackCmd) constructs the app with loadLatest=true, so it panics in LoadLatestVersion before it can ever call RollbackToVersion. This leaves an affected node unrecoverable via any built-in tool.
• Adds allorad debug-store {inspect, check-tip, force-rollback}:
  • inspect — read-only dump of s/latest and per-store commit-info at the tip and tip-1.
  • check-tip — builds the app with loadLatest=false (so store construction succeeds regardless of tip consistency) and attempts to load all stores at the latest version, without writing anything.
  • force-rollback --to N — same load-and-validate probe at a target version; only writes (deletes versions above target, rewrites s/latest) when passed --yes.
• Uses the app's real declared store keys (via app.NewAlloraApp(..., loadLatest=false, ...) and its CommitMultiStore()) rather than inferring stores from on-disk commit-info, so it stays correct if store types/keys change in the future.

Alternative considered

Hardening the existing rollback command itself (constructing the app with loadLatest=false and auto-detecting/loading latest-1 when the tip is inconsistent) would be a cleaner long-term fix and could make this command unnecessary. Open to that direction instead if maintainers prefer it — happy to discuss.

Test plan

• go build ./...
• golangci-lint run ./cmd/allorad/... — no issues
• allorad debug-store --help / subcommand --help output reviewed
• allorad debug-store inspect --home <empty home> runs without panicking on a home with no application state
• Exercise check-tip / force-rollback against a real interrupted-migration data dir (not available in this environment) before relying on it in a live incident

---

Summary by cubic

Adds allorad debug-store commands to inspect and safely recover a multistore after an interrupted migration, avoiding the rollback panic by loading the app with loadLatest=false. Enables dry-run validation and a gated rollback to a target version.

• New Features
  • allorad debug-store inspect — show s/latest and per-store commit-info at tip and tip-1.
  • allorad debug-store check-tip — build with loadLatest=false and attempt to load all stores at the latest version (read-only).
  • allorad debug-store force-rollback --to N [--yes] — validate target version loads; with --yes, delete versions above N and set s/latest=N.
  • Uses the app’s real store keys via CommitMultiStore() for correctness across store changes.

^{Written for commit 3a99796. Summary will update on new commits.}

[cubic-dev-ai]

2 issues found across 2 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="cmd/allorad/cmd/debug_store.go">

<violation number="1" location="cmd/allorad/cmd/debug_store.go:60">
P1: Debug-store commands do not inspect the node's configured application DB backend, so they can probe or modify the wrong database when the node uses PebbleDB or RocksDB. Reuse the server command's configured Viper/AppOptions when selecting the backend.</violation>

<violation number="2" location="cmd/allorad/cmd/debug_store.go:119">
P2: The advertised read-only probes write outside the selected home and do not see pending upgrade metadata because the app is built without the command's home in its AppOptions. Pass the real command AppOptions/home into `NewAlloraApp` before constructing the probe.</violation>
</file>

Architecture diagram

sequenceDiagram
    participant CLI as CLI (allorad)
    participant Cmd as debug-store Command
    participant DB as Application DB (LevelDB/GoLevelDB)
    participant App as AlloraApp
    participant Store as rootmulti.Store (CommitMultiStore)
    participant Log as Logger

    Note over CLI,Log: debug-store inspect — Read-Only Diagnostic

    CLI->>Cmd: debug-store inspect --home <dir>
    Cmd->>DB: openApplicationDB(home)
    DB-->>Cmd: db instance
    Cmd->>DB: rootmulti.GetLatestVersion(db)
    DB-->>Cmd: latest version (V)
    Cmd->>CMD: Print "s/latest = V"
    loop For version = V, V-1
        Cmd->>DB: commitInfoAt(db, version)
        DB->>DB: GET s/{version}
        alt Entry exists
            DB-->>Cmd: CommitInfo (unmarshaled)
            Cmd->>Cmd: Print store names, commit versions
        else Entry missing
            DB-->>Cmd: nil
            Cmd->>Cmd: Print error for this version
        end
    end

    Note over CLI,Log: debug-store check-tip — Load Validation (No Write)

    CLI->>Cmd: debug-store check-tip --home <dir>
    Cmd->>DB: openApplicationDB(home)
    DB-->>Cmd: db instance
    Cmd->>DB: rootmulti.GetLatestVersion(db)
    DB-->>Cmd: latest version (V)
    Cmd->>App: buildUnloadedApp(logger, db)
    App->>App: NewAlloraApp(logger, db, nil, false, viper)
    App-->>Cmd: App instance
    Cmd->>App: CommitMultiStore()
    App-->>Cmd: Store instance (type *rootmulti.Store)
    Cmd->>Store: LoadVersion(V)
    alt Load succeeds
        Store->>DB: Load each store at version V
        DB-->>Store: Store data
        Store-->>Cmd: nil (success)
        Cmd->>Cmd: Print "load OK"
    else Load fails (inconsistent tip)
        Store-->>Cmd: error
        Cmd->>Cmd: Print error message
    end

    Note over CLI,Log: debug-store force-rollback — Validate + Optional Write

    CLI->>Cmd: debug-store force-rollback --to N [--yes]
    Cmd->>DB: openApplicationDB(home)
    DB-->>Cmd: db instance
    Cmd->>DB: rootmulti.GetLatestVersion(db)
    DB-->>Cmd: latest version (V)
    Cmd->>Cmd: Validate 0 < N < V
    alt Invalid target
        Cmd-->>CLI: Error: target out of range
    end
    Cmd->>App: buildUnloadedApp(logger, db)
    App->>App: NewAlloraApp(logger, db, nil, false, viper)
    App-->>Cmd: App instance
    Cmd->>App: CommitMultiStore()
    App-->>Cmd: Store instance (type *rootmulti.Store)
    Cmd->>Store: LoadVersion(N)
    alt Load fails
        Store-->>Cmd: error
        Cmd-->>CLI: Error: LoadVersion failed
    else Load succeeds
        Store-->>Cmd: nil
        alt --yes flag NOT set (dry run)
            Cmd->>Cmd: Print "dry run — re-run with --yes"
            Cmd-->>CLI: exit 0
        else --yes flag set (perform rollback)
            Cmd->>Store: RollbackToVersion(N)
            Store->>DB: Delete versions > N
            Store->>DB: Rewrite s/latest = N
            DB-->>Store: Done
            Store-->>Cmd: nil
            Cmd->>DB: rootmulti.GetLatestVersion(db)
            DB-->>Cmd: N
            Cmd->>Cmd: Print "s/latest is now N"
            Cmd-->>CLI: exit 0
        end
    end

Loading

<sub>Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic</sub>

[cubic-dev-ai]

P1: Debug-store commands do not inspect the node's configured application DB backend, so they can probe or modify the wrong database when the node uses PebbleDB or RocksDB. Reuse the server command's configured Viper/AppOptions when selecting the backend.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At cmd/allorad/cmd/debug_store.go, line 60:

<comment>Debug-store commands do not inspect the node's configured application DB backend, so they can probe or modify the wrong database when the node uses PebbleDB or RocksDB. Reuse the server command's configured Viper/AppOptions when selecting the backend.</comment>

<file context>
@@ -0,0 +1,233 @@
+
+func openApplicationDB(home string) (dbm.DB, error) {
+	vp := viper.New()
+	backend := server.GetAppDBBackend(vp)
+	return dbm.NewDB("application", backend, filepath.Join(home, "data"))
+}
</file context>

[cubic-dev-ai]

P2: The advertised read-only probes write outside the selected home and do not see pending upgrade metadata because the app is built without the command's home in its AppOptions. Pass the real command AppOptions/home into NewAlloraApp before constructing the probe.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At cmd/allorad/cmd/debug_store.go, line 119:

<comment>The advertised read-only probes write outside the selected home and do not see pending upgrade metadata because the app is built without the command's home in its AppOptions. Pass the real command AppOptions/home into `NewAlloraApp` before constructing the probe.</comment>

<file context>
@@ -0,0 +1,233 @@
+// the caller decides which version to attempt loading via the returned CommitMultiStore.
+func buildUnloadedApp(logger log.Logger, db dbm.DB) (*rootmulti.Store, error) {
+	vp := viper.New()
+	alloraApp, err := app.NewAlloraApp(logger, db, nil, false, vp)
+	if err != nil {
+		return nil, err
</file context>