Warm-standby promotion skips catalog log replay for an in-flight DDL → schema op orphaned cluster-wide #499

When a warm standby is promoted while a global DDL is in flight, the catalog log-replay step exits early because the catalog cc-entry already exists in the warm cache, so the in-flight schema op is never recovered: no write-lock restore, no CreateSchemaRecoveryTx. The DDL is left half-applied — locks and dirty schema stranded on the other node groups, and the schema log never cleaned.

Evidence

CatalogCcMap::Execute(ReplayLogCc) (tx_service/include/cc/catalog_cc_map.h:1309-1319):

if (shard_->core_id_ == 0) {
    CatalogKey table_key(table_name);
    Iterator it = Find(table_key);
    CcEntry<CatalogKey, CatalogRecord, true, false> *cce = it->second;
    if (cce != nullptr) {        // warm standby ALWAYS has the catalog cce cached
        req.SetFinish();
        return true;             // <-- skips the entire schema-op replay below
    }
}

The code below this guard is what restores a PrepareSchema/PrepareData-stage write lock and spawns the schema-recovery tx. On a cold-start leader the cce is absent so replay runs; on a promoted warm standby the cce is present (the standby has been receiving catalog state), so replay is skipped entirely.

Scenario

1. Cluster runs CREATE/ALTER/DROP on table T; coordinator NG writes the PrepareSchema log; participant NGs hold catalog write locks / dirty schema.
2. The coordinator NG's leader fails; a warm standby is promoted.
3. The new leader replays the catalog log, but core-0's early-exit fires (cce exists) → the schema op is not recovered.
4. Participant NGs keep their catalog write locks and dirty schema forever; the DDL never commits or rolls back; new DML on T is blocked cluster-wide; the schema log is never cleaned.

Fix: on warm-standby promotion, the replay path must reconcile the in-flight schema op (restore the lock / run the recovery tx) even when the catalog cce is already cached — e.g. gate on schema-op stage/version rather than cce presence.

Found during a code audit (PR #493). Verified against source at the cited lines.

🤖 Found with Claude Code