Error during build using corporate proxy

[egorov-denis]

We are using corporate proxy and all our outside traffic goes only through a proxy. I can't connect to internet without a proxy.

I'm trying to build new image based on another image.

Dockerfile looks like:

FROM nginx:1.13-alpine
COPY conf /etc/nginx/conf.d/default.conf

The error message:

[+] Building 30.2s (3/3) FINISHED                                                                                                                                                                                             docker:default
 => [internal] load build definition from Dockerfile                                                                                                                                                                                    0.0s
 => => transferring dockerfile: 101B                                                                                                                                                                                                    0.0s
 => ERROR [internal] load metadata for docker.io/library/nginx:1.13-alpine                                                                                                                                                             30.1s
 => [auth] library/nginx:pull token for registry-1.docker.io                                                                                                                                                                            0.0s
------
 > [internal] load metadata for docker.io/library/nginx:1.13-alpine:
------
Dockerfile:1
--------------------
   1 | >>> FROM nginx:1.13-alpine
   2 |     COPY conf /etc/nginx/conf.d/default.conf
--------------------
ERROR: failed to solve: DeadlineExceeded: DeadlineExceeded: DeadlineExceeded: nginx:1.13-alpine: failed to authorize: DeadlineExceeded: failed to fetch oauth token: Post "https://auth.docker.io/token": dial tcp 54.227.20.253:443: i/o timeout

But another Dockerfile works well:

FROM mcr.microsoft.com/dotnet/aspnet:6.0
COPY conf /etc/nginx/conf.d/default.conf

Also manually pulling image works well. A command for manually pulling:

docker pull nginx:1.13-alpine

A proxy is configured in Docker (desktop) for both http and https connections and also in windows settings.

I've tried to use debug command docker --debug build . and it returns:

Dockerfile:1
--------------------
   1 | >>> FROM nginx:1.13-alpine
   2 |     COPY conf /etc/nginx/conf.d/default.conf
--------------------
ERROR: failed to solve: DeadlineExceeded: DeadlineExceeded: DeadlineExceeded: nginx:1.13-alpine: failed to authorize: DeadlineExceeded: failed to fetch oauth token: Post "https://auth.docker.io/token": dial tcp 54.227.20.253:443: i/o timeout
23236 v0.12.1-desktop.4 C:\Program Files\Docker\cli-plugins\docker-buildx.exe --debug buildx build .
github.com/moby/buildkit/session/auth/authprovider.(*authProvider).FetchToken.func2
        github.com/moby/buildkit@v0.12.0/session/auth/authprovider/authprovider.go:100
github.com/moby/buildkit/session/auth/authprovider.(*authProvider).FetchToken
        github.com/moby/buildkit@v0.12.0/session/auth/authprovider/authprovider.go:124
github.com/moby/buildkit/session/auth._Auth_FetchToken_Handler.func1
        github.com/moby/buildkit@v0.12.0/session/auth/auth.pb.go:954
github.com/moby/buildkit/session.NewSession.ChainUnaryServer.func3.1.1
        github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1
        go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@v0.40.0/interceptor.go:342
github.com/moby/buildkit/session.NewSession.filterServer.func1
        github.com/moby/buildkit@v0.12.0/session/session.go:177
github.com/moby/buildkit/session.NewSession.ChainUnaryServer.func3.1.1
        github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25
github.com/moby/buildkit/session.NewSession.ChainUnaryServer.func3
        github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:34
github.com/moby/buildkit/session/auth._Auth_FetchToken_Handler
        github.com/moby/buildkit@v0.12.0/session/auth/auth.pb.go:956
google.golang.org/grpc.(*Server).processUnaryRPC
        google.golang.org/grpc@v1.53.0/server.go:1336
google.golang.org/grpc.(*Server).handleStream
        google.golang.org/grpc@v1.53.0/server.go:1704
google.golang.org/grpc.(*Server).serveStreams.func1.2
        google.golang.org/grpc@v1.53.0/server.go:965
runtime.goexit
        runtime/asm_amd64.s:1650

183  /usr/local/bin/dockerd --containerd /run/containerd/containerd.sock --pidfile /run/desktop/docker.pid --swarm-default-advertise-addr=eth0 --host-gateway-ip 192.168.65.254
github.com/moby/buildkit/session.grpcClientConn.ChainUnaryClient.func6.1.1
        /root/build-deb/engine/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:72
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryClientInterceptor.func1
        /root/build-deb/engine/vendor/go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc/interceptor.go:107
github.com/moby/buildkit/session.grpcClientConn.filterClient.func4
        /root/build-deb/engine/vendor/github.com/moby/buildkit/session/session.go:186
github.com/moby/buildkit/session.grpcClientConn.ChainUnaryClient.func6.1.1
        /root/build-deb/engine/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:72
github.com/moby/buildkit/session.grpcClientConn.ChainUnaryClient.func6
        /root/build-deb/engine/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:81
google.golang.org/grpc.(*ClientConn).Invoke
        /root/build-deb/engine/vendor/google.golang.org/grpc/call.go:35
github.com/moby/buildkit/session/auth.(*authClient).FetchToken
        /root/build-deb/engine/vendor/github.com/moby/buildkit/session/auth/auth.pb.go:869
github.com/moby/buildkit/session/auth.FetchToken.func1
        /root/build-deb/engine/vendor/github.com/moby/buildkit/session/auth/auth.go:59
github.com/moby/buildkit/session.(*Manager).Any
        /root/build-deb/engine/vendor/github.com/moby/buildkit/session/group.go:82
github.com/moby/buildkit/session/auth.FetchToken
        /root/build-deb/engine/vendor/github.com/moby/buildkit/session/auth/auth.go:56
github.com/moby/buildkit/util/resolver.(*authHandler).fetchToken
        /root/build-deb/engine/vendor/github.com/moby/buildkit/util/resolver/authorizer.go:341
github.com/moby/buildkit/util/resolver.(*authHandler).doBearerAuth.func1
        /root/build-deb/engine/vendor/github.com/moby/buildkit/util/resolver/authorizer.go:307
github.com/moby/buildkit/util/flightcontrol.(*call[...]).run
        /root/build-deb/engine/vendor/github.com/moby/buildkit/util/flightcontrol/flightcontrol.go:121
sync.(*Once).doSlow
        /usr/local/go/src/sync/once.go:74
sync.(*Once).Do
        /usr/local/go/src/sync/once.go:65
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1650

183  /usr/local/bin/dockerd --containerd /run/containerd/containerd.sock --pidfile /run/desktop/docker.pid --swarm-default-advertise-addr=eth0 --host-gateway-ip 192.168.65.254
github.com/moby/buildkit/frontend/dockerfile/dockerfile2llb.toDispatchState.toDispatchState.func3.func4
        /root/build-deb/engine/vendor/github.com/moby/buildkit/frontend/dockerfile/dockerfile2llb/convert.go:413
golang.org/x/sync/errgroup.(*Group).Go.func1
        /root/build-deb/engine/vendor/golang.org/x/sync/errgroup/errgroup.go:75
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1650

23236 v0.12.1-desktop.4 C:\Program Files\Docker\cli-plugins\docker-buildx.exe --debug buildx build .
google.golang.org/grpc.getChainUnaryInvoker.func1
        google.golang.org/grpc@v1.53.0/clientconn.go:361
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryClientInterceptor.func1
        go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@v0.40.0/interceptor.go:99
github.com/moby/buildkit/client.New.filterInterceptor.func6
        github.com/moby/buildkit@v0.12.0/client/client.go:403
google.golang.org/grpc.DialContext.chainUnaryClientInterceptors.func3
        google.golang.org/grpc@v1.53.0/clientconn.go:349
google.golang.org/grpc.(*ClientConn).Invoke
        google.golang.org/grpc@v1.53.0/call.go:35
github.com/moby/buildkit/frontend/gateway/pb.(*lLBBridgeClient).Solve
        github.com/moby/buildkit@v0.12.0/frontend/gateway/pb/gateway.pb.go:2875
github.com/moby/buildkit/client.(*gatewayClientForBuild).Solve
        github.com/moby/buildkit@v0.12.0/client/build.go:89
github.com/moby/buildkit/frontend/gateway/grpcclient.(*grpcClient).Solve
        github.com/moby/buildkit@v0.12.0/frontend/gateway/grpcclient/client.go:411
github.com/docker/buildx/build.BuildWithResultHandler.func3.1.2
        github.com/docker/buildx/build/build.go:950
github.com/moby/buildkit/frontend/gateway/grpcclient.(*grpcClient).Run
        github.com/moby/buildkit@v0.12.0/frontend/gateway/grpcclient/client.go:214
github.com/moby/buildkit/client.(*Client).Build.func2
        github.com/moby/buildkit@v0.12.0/client/build.go:59
github.com/moby/buildkit/client.(*Client).solve.func3
        github.com/moby/buildkit@v0.12.0/client/solve.go:283
golang.org/x/sync/errgroup.(*Group).Go.func1
        golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75
runtime.goexit
        runtime/asm_amd64.s:1650

183  /usr/local/bin/dockerd --containerd /run/containerd/containerd.sock --pidfile /run/desktop/docker.pid --swarm-default-advertise-addr=eth0 --host-gateway-ip 192.168.65.254
github.com/moby/buildkit/frontend/gateway.(*llbBridgeForwarder).Return
        /root/build-deb/engine/vendor/github.com/moby/buildkit/frontend/gateway/gateway.go:923
github.com/moby/buildkit/control/gateway.(*GatewayForwarder).Return
        /root/build-deb/engine/vendor/github.com/moby/buildkit/control/gateway/gateway.go:135
github.com/moby/buildkit/frontend/gateway/pb._LLBBridge_Return_Handler.func1
        /root/build-deb/engine/vendor/github.com/moby/buildkit/frontend/gateway/pb/gateway.pb.go:3216
github.com/docker/docker/api/server/router/grpc.NewRouter.ChainUnaryServer.func1.1.1
        /root/build-deb/engine/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:25
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1
        /root/build-deb/engine/vendor/go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc/interceptor.go:376
github.com/docker/docker/api/server/router/grpc.unaryInterceptor.func1
        /root/build-deb/engine/api/server/router/grpc/grpc.go:58
github.com/docker/docker/api/server/router/grpc.NewRouter.ChainUnaryServer.func1.1.1
        /root/build-deb/engine/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:25
github.com/docker/docker/api/server/router/grpc.NewRouter.ChainUnaryServer.func1
        /root/build-deb/engine/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:34
github.com/moby/buildkit/frontend/gateway/pb._LLBBridge_Return_Handler
        /root/build-deb/engine/vendor/github.com/moby/buildkit/frontend/gateway/pb/gateway.pb.go:3218
google.golang.org/grpc.(*Server).processUnaryRPC
        /root/build-deb/engine/vendor/google.golang.org/grpc/server.go:1374
google.golang.org/grpc.(*Server).handleStream
        /root/build-deb/engine/vendor/google.golang.org/grpc/server.go:1751
google.golang.org/grpc.(*Server).serveStreams.func1.1
        /root/build-deb/engine/vendor/google.golang.org/grpc/server.go:986
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1650

23236 v0.12.1-desktop.4 C:\Program Files\Docker\cli-plugins\docker-buildx.exe --debug buildx build .
google.golang.org/grpc.getChainUnaryInvoker.func1
        google.golang.org/grpc@v1.53.0/clientconn.go:361
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryClientInterceptor.func1
        go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@v0.40.0/interceptor.go:99
github.com/moby/buildkit/client.New.filterInterceptor.func6
        github.com/moby/buildkit@v0.12.0/client/client.go:403
google.golang.org/grpc.DialContext.chainUnaryClientInterceptors.func3
        google.golang.org/grpc@v1.53.0/clientconn.go:349
google.golang.org/grpc.(*ClientConn).Invoke
        google.golang.org/grpc@v1.53.0/call.go:35
github.com/moby/buildkit/api/services/control.(*controlClient).Solve
        github.com/moby/buildkit@v0.12.0/api/services/control/control.pb.go:2208
github.com/moby/buildkit/client.(*Client).solve.func2
        github.com/moby/buildkit@v0.12.0/client/solve.go:258
golang.org/x/sync/errgroup.(*Group).Go.func1
        golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75
runtime.goexit
        runtime/asm_amd64.s:1650

23236 v0.12.1-desktop.4 C:\Program Files\Docker\cli-plugins\docker-buildx.exe --debug buildx build .
github.com/moby/buildkit/client.(*Client).solve.func2
        github.com/moby/buildkit@v0.12.0/client/solve.go:273
golang.org/x/sync/errgroup.(*Group).Go.func1
        golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75
runtime.goexit
        runtime/asm_amd64.s:1650

I tried to run command RUN curl -x "<mycorporateproxy>" "some url" -k using docker build in another Dockerfile and it works. It means that corporate proxy works well.
I think that command docker build . ignores corporate proxy on Windows.

OS: Windows 10 2022 H2
Docker desktop: v4.28.0

[profnandaa]

Cross-linking #4643

[TBBle]

This one's not really area/windows, since this is Docker Desktop's WSL2 environment. And yeah, probably the same as #4643, although in this case it's the BuildKit embedded in buildx, and that one is seeing the issue in buildkitd in a buildx docker-container builder instance.

Edit: Ooops, technically it is area/windows, since it's a Windows build of buildx trying to fetch the token on the client side, not the buildkitd in WSL2, that's hitting the failure.

Edit again: Another oops, this might be unrelated to #4643, as that's shelled out to a command-line, and we explicitly provide an env that doesn't include http proxy info. This issue lies in our calls to the http client, and I noticed we changed HTTP clients between 0.12.1 and 0.13, so it might already be solved, but will need a version of buildx that implements 0.13 to re-test the repro case here.

---

Edit: @amyboose: Docker BuildX 0.13.0 is out and includes BuildKit 0.13.0. Please retest with that release, as it may have solved the issue due to a change in HTTP client used for the failing operation.

Also, I just noticed docker/buildx#1487 (comment) and docker/buildx#1979 which notes that BuildX doesn't currently get the Docker Desktop proxy configuration. So is the HTTP_PROXY env-var set in the session where you're running BuildX? If not, try setting it and see if that resolves the issue. I expect this would affect other command-line tools like curl (outside the container) but it's possible curl can read other sources of proxy info that maybe Go's HTTP client cannot.

[egorov-denis]

I've created buildkitd v13 using docker:

docker run --rm --privileged -d --name buildkit moby/buildkit

buildctl is located on Windows machine.
My command to build image:

buildctl build --frontend=dockerfile.v0 --local context=. --local dockerfile=.

Result:

[+] Building 0.4s (2/2) FINISHED
 => [internal] load build definition from Dockerfile                                                                                                                                                                                    0.0s
 => => transferring dockerfile: 101B                                                                                                                                                                                                    0.0s
 => ERROR [internal] load metadata for docker.io/library/nginx:1.13-alpine                                                                                                                                                              0.3s
------
 > [internal] load metadata for docker.io/library/nginx:1.13-alpine:
------
Dockerfile:1
--------------------
   1 | >>> FROM nginx:1.13-alpine
   2 |     COPY conf /etc/nginx/conf.d/default.conf
--------------------
error: failed to solve: nginx:1.13-alpine: failed to resolve source metadata for docker.io/library/nginx:1.13-alpine: failed to do request: Head "https://registry-1.docker.io/v2/library/nginx/manifests/1.13-alpine": tls: failed to verify certificate: x509: certificate signed by unknown authority

Next time:

[+] Building 30.1s (3/3) FINISHED
 => [internal] load build definition from Dockerfile                                                                                                                                                                                    0.0s
 => => transferring dockerfile: 101B                                                                                                                                                                                                    0.0s
 => ERROR [internal] load metadata for docker.io/library/nginx:1.13-alpine                                                                                                                                                             30.1s
 => [auth] library/nginx:pull token for registry-1.docker.io                                                                                                                                                                            0.0s
------
 > [internal] load metadata for docker.io/library/nginx:1.13-alpine:
------
Dockerfile:1
--------------------
   1 | >>> FROM nginx:1.13-alpine
   2 |     COPY conf /etc/nginx/conf.d/default.conf
--------------------
error: failed to solve: nginx:1.13-alpine: failed to resolve source metadata for docker.io/library/nginx:1.13-alpine: failed to authorize: failed to fetch oauth token: Post "https://auth.docker.io/token": dial tcp 54.236.113.205:443: i/o timeout

After using debug:

time="2024-03-07T14:07:42+05:00" level=debug msg="commandconn: starting docker with [exec -i buildkit buildctl dial-stdio]"
time="2024-03-07T14:07:42+05:00" level=debug msg="serving grpc connection" spanID=86eca0f6028438bc traceID=2ee034080aedf0c2ecb2727962aab3b7
[+] Building 30.1s (3/3)
[+] Building 30.2s (3/3) FINISHED
 => [internal] load build definition from Dockerfile                                                                                                                                                                                    0.0s
 => => transferring dockerfile: 101B                                                                                                                                                                                                    0.0s
 => ERROR [internal] load metadata for docker.io/library/nginx:1.13-alpine                                                                                                                                                             30.1s
 => [auth] library/nginx:pull token for registry-1.docker.io                                                                                                                                                                            0.0s
------
 > [internal] load metadata for docker.io/library/nginx:1.13-alpine:
------
time="2024-03-07T14:08:12+05:00" level=debug msg="commandconn (docker):write /dev/stdout: The pipe is being closed.\n"
Dockerfile:1
--------------------
   1 | >>> FROM nginx:1.13-alpine
   2 |     COPY conf /etc/nginx/conf.d/default.conf
--------------------
error: failed to solve: DeadlineExceeded: DeadlineExceeded: DeadlineExceeded: nginx:1.13-alpine: failed to resolve source metadata for docker.io/library/nginx:1.13-alpine: failed to authorize: DeadlineExceeded: failed to fetch oauth token: Post "https://auth.docker.io/token": dial tcp 54.236.113.205:443: i/o timeout
8600 v0.13.0 C:\Users\DVEgorov10\Downloads\buildkit-v0.13.0.windows-amd64\bin\buildctl.exe --debug build --frontend=dockerfile.v0 --local context=. --local dockerfile=.
github.com/moby/buildkit/session/auth/authprovider.(*authProvider).FetchToken.func2
        /src/session/auth/authprovider/authprovider.go:109
github.com/moby/buildkit/session/auth/authprovider.(*authProvider).FetchToken
        /src/session/auth/authprovider/authprovider.go:133
github.com/moby/buildkit/session/auth._Auth_FetchToken_Handler.func1
        /src/session/auth/auth.pb.go:954
github.com/moby/buildkit/session.NewSession.ChainUnaryServer.func3.1.1
        /src/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:25
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1
        /src/vendor/go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc/interceptor.go:326
github.com/moby/buildkit/session.NewSession.filterServer.func1
        /src/session/session.go:177
github.com/moby/buildkit/session.NewSession.ChainUnaryServer.func3.1.1
        /src/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:25
github.com/moby/buildkit/session.NewSession.ChainUnaryServer.func3
        /src/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:34
github.com/moby/buildkit/session/auth._Auth_FetchToken_Handler
        /src/session/auth/auth.pb.go:956
google.golang.org/grpc.(*Server).processUnaryRPC
        /src/vendor/google.golang.org/grpc/server.go:1343
google.golang.org/grpc.(*Server).handleStream
        /src/vendor/google.golang.org/grpc/server.go:1737
google.golang.org/grpc.(*Server).serveStreams.func1.1
        /src/vendor/google.golang.org/grpc/server.go:986
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1650

1 v0.13.0 buildkitd
github.com/moby/buildkit/session.grpcClientConn.ChainUnaryClient.func8.1.1
        /src/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:72
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryClientInterceptor.func1
        /src/vendor/go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc/interceptor.go:110
github.com/moby/buildkit/session.grpcClientConn.filterClient.func6
        /src/session/session.go:186
github.com/moby/buildkit/session.grpcClientConn.ChainUnaryClient.func8.1.1
        /src/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:72
github.com/moby/buildkit/session.grpcClientConn.ChainUnaryClient.func8
        /src/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:81
google.golang.org/grpc.(*ClientConn).Invoke
        /src/vendor/google.golang.org/grpc/call.go:35
github.com/moby/buildkit/session/auth.(*authClient).FetchToken
        /src/session/auth/auth.pb.go:869
github.com/moby/buildkit/session/auth.FetchToken.func1
        /src/session/auth/auth.go:59
github.com/moby/buildkit/session.(*Manager).Any
        /src/session/group.go:83
github.com/moby/buildkit/session/auth.FetchToken
        /src/session/auth/auth.go:56
github.com/moby/buildkit/util/resolver.(*authHandler).fetchToken
        /src/util/resolver/authorizer.go:341
github.com/moby/buildkit/util/resolver.(*authHandler).doBearerAuth.func1
        /src/util/resolver/authorizer.go:307
github.com/moby/buildkit/util/flightcontrol.(*call[...]).run
        /src/util/flightcontrol/flightcontrol.go:121
sync.(*Once).doSlow
        /usr/local/go/src/sync/once.go:74
sync.(*Once).Do
        /usr/local/go/src/sync/once.go:65
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1650

1 v0.13.0 buildkitd
github.com/moby/buildkit/client/llb/sourceresolver.(*imageMetaResolver).ResolveImageConfig
        /src/client/llb/sourceresolver/imageresolver.go:51
github.com/moby/buildkit/frontend/gateway/forwarder.(*BridgeClient).ResolveImageConfig
        /src/frontend/gateway/forwarder/forward.go:101
github.com/moby/buildkit/frontend/dockerfile/dockerfile2llb.toDispatchState.toDispatchState.func3.func4
        /src/frontend/dockerfile/dockerfile2llb/convert.go:429
golang.org/x/sync/errgroup.(*Group).Go.func1
        /src/vendor/golang.org/x/sync/errgroup/errgroup.go:75
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1650

1 v0.13.0 buildkitd
github.com/moby/buildkit/frontend/dockerfile/dockerfile2llb.toDispatchState.toDispatchState.func3.func4
        /src/frontend/dockerfile/dockerfile2llb/convert.go:437
golang.org/x/sync/errgroup.(*Group).Go.func1
        /src/vendor/golang.org/x/sync/errgroup/errgroup.go:75
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1650

8600 v0.13.0 C:\Users\DVEgorov10\Downloads\buildkit-v0.13.0.windows-amd64\bin\buildctl.exe --debug build --frontend=dockerfile.v0 --local context=. --local dockerfile=.
google.golang.org/grpc.getChainUnaryInvoker.func1
        /src/vendor/google.golang.org/grpc/clientconn.go:519
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryClientInterceptor.func1
        /src/vendor/go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc/interceptor.go:110
github.com/moby/buildkit/client.New.filterInterceptor.func5
        /src/client/client.go:387
google.golang.org/grpc.DialContext.chainUnaryClientInterceptors.func3
        /src/vendor/google.golang.org/grpc/clientconn.go:507
google.golang.org/grpc.(*ClientConn).Invoke
        /src/vendor/google.golang.org/grpc/call.go:35
github.com/moby/buildkit/frontend/gateway/pb.(*lLBBridgeClient).Solve
        /src/frontend/gateway/pb/gateway.pb.go:3078
github.com/moby/buildkit/client.(*gatewayClientForBuild).Solve
        /src/client/build.go:94
github.com/moby/buildkit/frontend/gateway/grpcclient.(*grpcClient).Solve
        /src/frontend/gateway/grpcclient/client.go:415
main.buildAction.func5.2
        /src/cmd/buildctl/build.go:378
github.com/moby/buildkit/frontend/gateway/grpcclient.(*grpcClient).Run
        /src/frontend/gateway/grpcclient/client.go:218
github.com/moby/buildkit/client.(*Client).Build.func2
        /src/client/build.go:59
github.com/moby/buildkit/client.(*Client).solve.func3
        /src/client/solve.go:300
golang.org/x/sync/errgroup.(*Group).Go.func1
        /src/vendor/golang.org/x/sync/errgroup/errgroup.go:75
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1650

1 v0.13.0 buildkitd
github.com/moby/buildkit/frontend/gateway.(*llbBridgeForwarder).Return
        /src/frontend/gateway/gateway.go:975
github.com/moby/buildkit/control/gateway.(*GatewayForwarder).Return
        /src/control/gateway/gateway.go:146
github.com/moby/buildkit/frontend/gateway/pb._LLBBridge_Return_Handler.func1
        /src/frontend/gateway/pb/gateway.pb.go:3441
main.main.func3.ChainUnaryServer.func2.1.1
        /src/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:25
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1
        /src/vendor/go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc/interceptor.go:326
main.unaryInterceptor.func1
        /src/cmd/buildkitd/main.go:686
main.main.func3.ChainUnaryServer.func2.1.1
        /src/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:25
main.main.func3.ChainUnaryServer.func2
        /src/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:34
github.com/moby/buildkit/frontend/gateway/pb._LLBBridge_Return_Handler
        /src/frontend/gateway/pb/gateway.pb.go:3443
google.golang.org/grpc.(*Server).processUnaryRPC
        /src/vendor/google.golang.org/grpc/server.go:1343
google.golang.org/grpc.(*Server).handleStream
        /src/vendor/google.golang.org/grpc/server.go:1737
google.golang.org/grpc.(*Server).serveStreams.func1.1
        /src/vendor/google.golang.org/grpc/server.go:986
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1650

8600 v0.13.0 C:\Users\DVEgorov10\Downloads\buildkit-v0.13.0.windows-amd64\bin\buildctl.exe --debug build --frontend=dockerfile.v0 --local context=. --local dockerfile=.
google.golang.org/grpc.getChainUnaryInvoker.func1
        /src/vendor/google.golang.org/grpc/clientconn.go:519
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryClientInterceptor.func1
        /src/vendor/go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc/interceptor.go:110
github.com/moby/buildkit/client.New.filterInterceptor.func5
        /src/client/client.go:387
google.golang.org/grpc.DialContext.chainUnaryClientInterceptors.func3
        /src/vendor/google.golang.org/grpc/clientconn.go:507
google.golang.org/grpc.(*ClientConn).Invoke
        /src/vendor/google.golang.org/grpc/call.go:35
github.com/moby/buildkit/api/services/control.(*controlClient).Solve
        /src/api/services/control/control.pb.go:2234
github.com/moby/buildkit/client.(*Client).solve.func2
        /src/client/solve.go:274
golang.org/x/sync/errgroup.(*Group).Go.func1
        /src/vendor/golang.org/x/sync/errgroup/errgroup.go:75
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1650

8600 v0.13.0 C:\Users\DVEgorov10\Downloads\buildkit-v0.13.0.windows-amd64\bin\buildctl.exe --debug build --frontend=dockerfile.v0 --local context=. --local dockerfile=.
github.com/moby/buildkit/client.(*Client).solve.func2
        /src/client/solve.go:290
golang.org/x/sync/errgroup.(*Group).Go.func1
        /src/vendor/golang.org/x/sync/errgroup/errgroup.go:75
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1650

[TBBle]

Try running the buildkit daemon with

docker run --rm --privileged -d --name buildkit --env "HTTP_PROXY=<YourProxyDetailsHere>" --env "HTTPS_PROXY=<YourProxyDetailsHere>" moby/buildkit

or if you already have the HTTP_PROXY and/or HTTPS_PROXY env-vars set in the relevant shell,

docker run --rm --privileged -d --name buildkit --env HTTP_PROXY --env HTTPS_PROXY moby/buildkit

[egorov-denis]

Try running the buildkit daemon with

docker run --rm --privileged -d --name buildkit --env "HTTP_PROXY=<YourProxyDetailsHere>" --env "HTTPS_PROXY=<YourProxyDetailsHere>" moby/buildkit

or if you already have the HTTP_PROXY and/or HTTPS_PROXY env-vars set in the relevant shell,

docker run --rm --privileged -d --name buildkit --env HTTP_PROXY --env HTTPS_PROXY moby/buildkit

The same result. It's not the error of only my PC. Other computers also has the same error message. I think it can be the error on SSL sertificate.

I've tested a command

docker run -it mcr.microsoft.com/dotnet/aspnet:8.0 openssl s_client -connect auth.docker.io:443

and it returns result:

CONNECTED(00000003)
depth=1 O = AO Kaspersky Lab, CN = Kaspersky Endpoint Security Personal Certification Authority
verify error:num=19:self-signed certificate in certificate chain
verify return:1
depth=1 O = AO Kaspersky Lab, CN = Kaspersky Endpoint Security Personal Certification Authority
verify return:1
depth=0 CN = *.docker.com
verify return:1
---
Certificate chain
 0 s:CN = *.docker.com
   i:O = AO Kaspersky Lab, CN = Kaspersky Endpoint Security Personal Certification Authority
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Sep  7 08:47:15 2023 GMT; NotAfter: Sep  5 08:47:15 2024 GMT
 1 s:O = AO Kaspersky Lab, CN = Kaspersky Endpoint Security Personal Certification Authority
   i:O = AO Kaspersky Lab, CN = Kaspersky Endpoint Security Personal Certification Authority
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Mar 10 08:13:42 2014 GMT; NotAfter: Mar  5 08:13:42 2034 GMT
---
Server certificate
-----BEGIN CERTIFICATE-----
some_Value
-----END CERTIFICATE-----
subject=CN = *.docker.com
issuer=O = AO Kaspersky Lab, CN = Kaspersky Endpoint Security Personal Certification Authority
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 10785 bytes and written 384 bytes
Verification error: self-signed certificate in certificate chain
---
New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 19 (self-signed certificate in certificate chain)
---
DONE

[TBBle]

That error shows it's not getting or using the proxy settings, and is instead being transparently redirected to your corporate firewall (or possibly the local computer's firewall), which relies on having the firewall's own certificate trusted by the computer as a variant of a man-in-the-middle attack. The SSL setup in side the container does not trust this certificate of course, so it fails as you see.

The real docker.com certificate is not signed by Kaspersky's firewall product, but by Amazon.

So, try:

docker run -it mcr.microsoft.com/dotnet/aspnet:8.0 openssl s_client -connect auth.docker.io:443 -proxy <YourProxyDetailsHere>

and if that still doesn't work, then I think <YourProxyDetailsHere> is incorrect in some way.

Once that's working, you can test with curl in a container, passing the --env "HTTP_PROXY=<YourProxyDetailsHere>" --env "HTTPS_PROXY=<YourProxyDetailsHere>" parameters to docker run.

Once that is working, there's no reason I'm aware of that

docker run --rm --privileged -d --name buildkit --env "HTTP_PROXY=<YourProxyDetailsHere>" --env "HTTPS_PROXY=<YourProxyDetailsHere>" moby/buildkit

would not work, but if it still fails, then we've possibly identified a real bug somewhere, which will probably be a hassle to reproduce and diagnose from outside a corporately-firewalled network. >_< But we can cross that bridge when we come to it.

[egorov-denis]

Recently my organization disabled firewall for .docker.com host.
The error from header is still exists.

Executing docker with openssl on auth.docker.io:

docker run -it --rm mcr.microsoft.com/dotnet/aspnet:8.0 openssl s_client -connect auth.docker.io:443
docker run -it --rm mcr.microsoft.com/dotnet/aspnet:8.0 openssl s_client -proxy my_proxy_here -connect auth.docker.io:443

Result for both:

depth=2 C = US, O = Amazon, CN = Amazon Root CA 1
verify return:1
depth=1 C = US, O = Amazon, CN = Amazon RSA 2048 M01
verify return:1
depth=0 CN = *.docker.com
verify return:1
---
Certificate chain
 0 s:CN = *.docker.com
   i:C = US, O = Amazon, CN = Amazon RSA 2048 M01
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Oct  2 00:00:00 2023 GMT; NotAfter: Oct 31 23:59:59 2024 GMT
 1 s:C = US, O = Amazon, CN = Amazon RSA 2048 M01
   i:C = US, O = Amazon, CN = Amazon Root CA 1
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Aug 23 22:21:28 2022 GMT; NotAfter: Aug 23 22:21:28 2030 GMT
 2 s:C = US, O = Amazon, CN = Amazon Root CA 1
   i:C = US, ST = Arizona, L = Scottsdale, O = "Starfield Technologies, Inc.", CN = Starfield Services Root Certificate Authority - G2
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: May 25 12:00:00 2015 GMT; NotAfter: Dec 31 01:00:00 2037 GMT
 3 s:C = US, ST = Arizona, L = Scottsdale, O = "Starfield Technologies, Inc.", CN = Starfield Services Root Certificate Authority - G2
   i:C = US, O = "Starfield Technologies, Inc.", OU = Starfield Class 2 Certification Authority
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Sep  2 00:00:00 2009 GMT; NotAfter: Jun 28 17:39:16 2034 GMT
---
Server certificate
-----BEGIN CERTIFICATE-----
some data
-----END CERTIFICATE-----
subject=CN = *.docker.com
issuer=C = US, O = Amazon, CN = Amazon RSA 2048 M01
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 5693 bytes and written 453 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)

Executing docker with openssl on reddit:

docker run -it --rm mcr.microsoft.com/dotnet/aspnet:8.0 openssl s_client -connect reddit.com:443

Result:

CONNECTED(00000003)
depth=1 O = AO Kaspersky Lab, CN = Kaspersky Endpoint Security Personal Certification Authority
verify error:num=19:self-signed certificate in certificate chain
verify return:1
depth=1 O = AO Kaspersky Lab, CN = Kaspersky Endpoint Security Personal Certification Authority
verify return:1
depth=0 C = US, ST = California, L = SAN FRANCISCO, O = "REDDIT, INC.", CN = *.reddit.com
verify return:1
---
Certificate chain
 0 s:C = US, ST = California, L = SAN FRANCISCO, O = "REDDIT, INC.", CN = *.reddit.com
   i:O = AO Kaspersky Lab, CN = Kaspersky Endpoint Security Personal Certification Authority
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Oct  6 03:31:43 2023 GMT; NotAfter: Oct  4 03:31:43 2024 GMT
 1 s:O = AO Kaspersky Lab, CN = Kaspersky Endpoint Security Personal Certification Authority
   i:O = AO Kaspersky Lab, CN = Kaspersky Endpoint Security Personal Certification Authority
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Mar 10 08:13:42 2014 GMT; NotAfter: Mar  5 08:13:42 2034 GMT
---
Server certificate
some data
-----END CERTIFICATE-----
subject=C = US, ST = California, L = SAN FRANCISCO, O = "REDDIT, INC.", CN = *.reddit.com
issuer=O = AO Kaspersky Lab, CN = Kaspersky Endpoint Security Personal Certification Authority
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 7740 bytes and written 380 bytes
Verification error: self-signed certificate in certificate chain
---
New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 19 (self-signed certificate in certificate chain)
---

Kaspersky (firewall) is still exists.

Executing docker build:

docker build . --build-arg HTTP_PROXY=http://my_proxy_here --build-arg HTTPS_PROXY=http://my_proxy_here

Result:

ERROR: failed to solve: DeadlineExceeded: DeadlineExceeded: DeadlineExceeded: nginx:1.13-alpine: failed to authorize: DeadlineExceeded: failed to fetch oauth token: Post "https://auth.docker.io/token": dial tcp 54.236.113.205:443: i/o timeout

Executing curl with no proxy:

docker run -it --rm --env "HTTP_PROXY=http://my_proxy_here" --env "HTTPS_PROXY=http://my_proxy_here" mcr.microsoft.com/dotnet/aspnet:8.0 /bin/bash -c "apt update && apt upgrade -y && apt install curl -y && curl https://reddit.com/"
docker run -it --rm --env "HTTP_PROXY=http://my_proxy_here" --env "HTTPS_PROXY=http://my_proxy_here" mcr.microsoft.com/dotnet/aspnet:8.0 /bin/bash -c "apt update && apt upgrade -y && apt install curl -y && curl https://reqbin.com/echo"

Result:

curl: (60) SSL certificate problem: self-signed certificate in certificate chain
More details here: https://curl.se/docs/sslcerts.html
curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

Executing curl with auth.docker.io and no proxy:

docker run -it --rm --env "HTTP_PROXY=http://my_proxy_here" --env "HTTPS_PROXY=http://my_proxy_here" mcr.microsoft.com/dotnet/aspnet:8.0 /bin/bash -c "apt update && apt upgrade -y && apt install curl -y && curl https://auth.docker.io/token"

Result: success request

Executing curl with proxy on context.reverso.net:

docker run -it --rm --env "HTTP_PROXY=http://my_proxy_here" --env "HTTPS_PROXY=http://my_proxy_here" mcr.microsoft.com/dotnet/aspnet:8.0 /bin/bash -c "apt update && apt upgrade -y && apt install curl -y && curl -x http://my_proxy_here https://context.reverso.net/"

Result:

curl: (60) SSL certificate problem: self-signed certificate in certificate chain
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

Most of the time curl -x works good.

Executing buildctl:

docker run --rm --privileged -d --name buildkit --env "HTTP_PROXY=http://my_proxy_here" --env "HTTPS_PROXY=http://my_proxy_here" moby/buildkit
buildctl build --frontend=dockerfile.v0 --local context=. --local dockerfile=.

Result:

[+] Building 31.0s (3/3) FINISHED
 => [internal] load build definition from Dockerfile                                                                                                                                                                                    0.1s
 => => transferring dockerfile: 158B                                                                                                                                                                                                    0.0s
 => ERROR [internal] load metadata for docker.io/library/nginx:1.13-alpine                                                                                                                                                             30.8s
 => [auth] library/nginx:pull token for registry-1.docker.io                                                                                                                                                                            0.0s
------
 > [internal] load metadata for docker.io/library/nginx:1.13-alpine:
------
Dockerfile:1
--------------------
   1 | >>> FROM nginx:1.13-alpine
   2 |     RUN openssl verify /usr/share/ca-certificates/nuget.crt
   3 |     COPY conf /etc/nginx/conf.d/default.conf
--------------------
error: failed to solve: DeadlineExceeded: DeadlineExceeded: DeadlineExceeded: nginx:1.13-alpine: failed to resolve source metadata for docker.io/library/nginx:1.13-alpine: failed to authorize: DeadlineExceeded: failed to fetch oauth token: Post "https://auth.docker.io/token": dial tcp 54.198.86.24:443: i/o timeout

I think that http client ignores proxy when build image based on another image

[brecode]

I am facing the exact same issue when running behind a corporate proxy. Everything else works fine but build doesn't honor proxy settings

[egorov-denis]

@TBBle Can you look at my last comment, please?

[kunZooberg]

It looks like a discussion started from 2019, may I know if there is some inner reason why we don't support it yet?
#971

[tonistiigi]

(This should really be tracked at docker/buildx)

Looks like the issue is that the client side request to auth.docker.io is blocked, presumably because the system is configured to only allow it through proxy. Note that this request is different as client is sending it directly from buildx binary while for example pulling the layers happens between registry and buildkit daemon (inside a VM). This is so that your credentials are not shared with buildkit and only the short-lived token is sent there.

Options:

• Client-side token fetching can be disabled by defining BUILDKIT_NO_CLIENT_TOKEN=1. Obviously, this does reduce security a bit because your password is sent to buildkit so that it can authenticate for you. This assumes that the proxy is either not needed or already configured for the VM requests.
• You can define HTTP_PROXY=... docker buildx ... when you run the buildx command (you can also configure your system so it is added automatically. Buildx will pick up such variables from the environment if they exist.

There is also old open PR docker/buildx#1487 to allow this case to be configured via config file but have not managed to get it merged yet.