driver/kubernetes: same first-connection race as #4021 — Dial() returns before SPDY stream is ready

[samifruit514]

The kubernetes driver has the structural bug #4021 identified in the docker-container driver: Dial() returns synchronously while the transport underneath isn't ready, so the failure surfaces at the first gRPC RPC. The retry added by #3493 wraps Dial(), so it doesn't cover this path.

Symptom is #2668's error, on buildx v0.36.1 (which includes #3493):

ERROR: unable to upgrade connection: error dialing backend:
       remote error: tls: internal error
ERROR: failed to build: listing workers for Build: failed to list workers:
       Unavailable: connection error: desc = "transport: failed to write
       client preface: ..."

Why #3493 doesn't catch it

ExecConn runs StreamWithContext in a goroutine and returns success immediately:

go func() {
    serr := exec.StreamWithContext(ctx, ...)
    if serr != nil && serr != context.Canceled {
        logrus.Error(serr)   // only logged
    }
}()
return kc, nil               // always success

The apiserver→kubelet TLS handshake happens inside that goroutine. If it fails (kubelet-serving CSR not yet signed on a fresh node, etc.), the goroutine logs and closes the pipe, but ExecConn has already returned success. tryWithBackoff sees no error. gRPC's later Write(clientPreface) on the broken conn is what surfaces tls: internal error, outside the retry scope.

Same shape as #4021: dial succeeds, transport isn't ready, first RPC fails, no retry.

Evidence

Hit on EKS 1.35 during a CA scale-up. Buildkit pods landed on freshly-joined nodes. buildx create --bootstrap succeeded; buildx build failed 259 ms later — shorter than tryWithBackoff's first 500 ms backoff, consistent with the retry not firing.

Cluster's kubelet-serving CSR sign latency during the scale-up: 15–33s, which is the window during which the apiserver→kubelet TLS handshake fails.

Suggested fix

Mirror #4027: probe readiness before returning from Dial(), so async setup failures surface synchronously and tryWithBackoff can retry them.

Happy to send a PR if that direction looks right.

Environment

• buildx v0.36.1, moby/buildkit v0.32.2 (vendored)
• EKS 1.35, serverTLSBootstrap: true, eks:certificate-controller as CSR signer