DROP/TRUNCATE retry KV and kickout stages forever while holding cluster-wide catalog write locks → indefinite per-table DML outage #503

DROP/TRUNCATE acquire catalog write-all locks (every NG, every shard) and then run KV-store and cache-kickout stages that retry forever while this node is leader, with no bound and no lock release. If the underlying stage cannot make progress (KV store unreachable, a wedged shard, a participant that never acks), the DDL holds the cluster-wide catalog write locks indefinitely → every new DML on that table is blocked across the whole cluster for the duration of the outage.

Evidence

UpsertTableOp::Forward, truncate clean stage (tx_service/src/tx_operation.cpp:3185-3204):

if (clean_ccm_op_.hd_result_.IsError()) {
    if (txm->CheckLeaderTerm()) {
        LOG(ERROR) << "... failed to truncate table ... keep retrying";
        op_ = &clean_ccm_op_;
        txm->PushOperation(&clean_ccm_op_);
        txm->Process(clean_ccm_op_);   // unbounded retry, locks still held
    } else { ForceToFinish(txm); }
    return;
}

The KV-write stage (tx_operation.cpp:2761-2771) has the same keep-retrying-while-leader shape. The catalog write locks taken earlier (CatalogAcquireAllOp) are not downgraded/released during these retries.

By contrast CREATE/UPDATE perform their KV work under write intents (not full write locks), so they don't black out DML the same way — the exposure is specific to the lock-holding DROP/TRUNCATE stages.

Impact

A transient KV/WAL outage or one slow participant during a DROP/TRUNCATE turns into an indefinite cluster-wide DML outage for that table, lasting until the stage finally succeeds or leadership changes.

Fix: bound the retries / add backoff with a failure path that releases the catalog locks (abort the DDL and surface the error) instead of holding write-all locks across an unbounded retry loop.

Found during a code audit (PR #493). Verified against source at the cited lines.

🤖 Found with Claude Code