host-gateway no longer supported in docker build

[zamlz]

Description

Hi all, I've noticed that once I upgraded my docker to the new version I'm unable to use host-gateway during the build process.

I get the following error when using the --add-host=host.docker.internal:host-gateway argument for docker build.

[+] Building 0.0s (0/0)
 ERROR: failed to solve: failed to parse additional hosts: failed to parse IP host-gateway

This used to work before the upgrade and what's interesting is that the docker run command and even docker comnpose still support the argument.

Reproduce

1. Run a docker build command with the --add-host=host.docker.internal:host-gateway

For example

docker build --add-host=host.docker.internal:host-gateway .

Expected behavior

docker build is able to use the host-gateway special string to bridge the host network with the containers network.

docker version

Client: Docker Engine - Community
  Version:           23.0.0
  API version:       1.42
  Go version:        go1.19.5
  Git commit:        e92dd87
  Built:             Wed Feb  1 17:49:08 2023
  OS/Arch:           linux/amd64
  Context:           default
 
 Server: Docker Engine - Community
  Engine:
   Version:          23.0.0
   API version:      1.42 (minimum version 1.12)
   Go version:       go1.19.5
   Git commit:       d7573ab
   Built:            Wed Feb  1 17:49:08 2023
   OS/Arch:          linux/amd64
   Experimental:     false
  containerd:
   Version:          1.6.16
   GitCommit:        31aa4358a36870b21a992d3ad2bef29e1d693bec
  runc:
   Version:          1.1.4
   GitCommit:        v1.1.4-0-g5fd4c4d
  docker-init:
   Version:          0.19.0
   GitCommit:        de40ad0

docker info

Client:
  Context:    default
  Debug Mode: false
  Plugins:
   buildx: Docker Buildx (Docker Inc.)
     Version:  v0.10.2
     Path:     /usr/libexec/docker/cli-plugins/docker-buildx
   compose: Docker Compose (Docker Inc.)
     Version:  v2.15.1
     Path:     /usr/libexec/docker/cli-plugins/docker-compose
   scan: Docker Scan (Docker Inc.)
     Version:  v0.23.0
     Path:     /usr/libexec/docker/cli-plugins/docker-scan
 
 Server:
  Containers: 9
   Running: 7
   Paused: 0
   Stopped: 2
  Images: 115
  Server Version: 23.0.0
  Storage Driver: overlay2
   Backing Filesystem: extfs
   Supports d_type: true
   Using metacopy: false
   Native Overlay Diff: true
   userxattr: false
  Logging Driver: json-file
  Cgroup Driver: cgroupfs
  Cgroup Version: 1
  Plugins:
   Volume: local
   Network: bridge host ipvlan macvlan null overlay
   Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
  Swarm: inactive
  Runtimes: io.containerd.runc.v2 runc
  Default Runtime: runc
  Init Binary: docker-init
  containerd version: 31aa4358a36870b21a992d3ad2bef29e1d693bec
  runc version: v1.1.4-0-g5fd4c4d
  init version: de40ad0
  Security Options:
   apparmor
   seccomp
    Profile: builtin
  Kernel Version: 5.4.0-121-generic
  Operating System: Ubuntu 20.04.5 LTS
  OSType: linux
  Architecture: x86_64
  CPUs: 4
  Total Memory: 3.666GiB
  Name: dev-testing-amlesh
  ID: HYQS:EF3Y:2IJR:XJHT:IPNI:2M3H:SPSX:3UBG:7OFQ:SE77:7U7J:WHYU
  Docker Root Dir: /var/lib/docker
  Debug Mode: false
  Registry: https://index.docker.io/v1/
  Experimental: false
  Insecure Registries:
   127.0.0.0/8
  Live Restore Enabled: false
 
 WARNING: No swap limit support

Additional Info

No response

[bsousaa]

There are changes in this area that could impact this functionality:

• [23.0 backport] builder: handle host-gateway with extra hosts moby/moby#44574
• builder: handle host-gateway with extra hosts moby/moby#44537

cc @crazy-max

[zamlz]

Thanks @bsousaa for sharing those two PRs. I'll read into those and see if explains why this functionality doesn't work anymore or if it's being added back in! While I go through that, could I get some clarification on what the moby/moby project is really and how it relates to docker?

[zamlz]

@bsousaa finished reviewing those two PR threads. Unless I'm misunderstanding something, it looks like both of them are adding the functionality I'm describing that appears to be missing my docker build command (one is a backport for the newer version).

// If the IP Address is a "host-gateway", replace this value with the
// IP address stored in the daemon level HostGatewayIP config variable.
if ip == opts.HostGatewayName {
    gateway := hostGatewayIP.String()
    if gateway == "" {
        return "", fmt.Errorf("unable to derive the IP value for host-gateway")
    }
    ip = gateway
}

[thaJeztah]

Yes, those pull requests were meant to fix issues around this (and are part of the 23.0 release). It's possible though that other changes are still needed to either BuildKit, or buildx (which provides the client for docker build), or the Dockefile front-end (parser).

Could I get some clarification on what the moby/moby project is really and how it relates to docker?

Docker uses a client/server model with an API. Docker Consists of various components. On a Linux installation of Docker Engine;

• The docker/cli (this repository) repository contains the source code for the command-line client (docker). There's additional plugins installed that handle specific parts of the CLI (docker compose, docker build and docker buildx).
• The moby/moby repository contains the source-code used to build the Docker Engine (the dockerd "daemon"), which provides the API, and contains the server-side components to run your containers, pull images, and build images.
• For building images, the dockerd daemon contains the "builder", which is code embedded from the moby/buildkit repository.
• In addition, there's further "lower level" components, such as containerd (OCI intermediate runtime), and runc (OCI runtime)

All of the above is Open Source. If you run Docker Desktop, there's additional components and features, some of which are not open source, which includes a VM and LinuxKit Linux distribution (to allow you to run Linux containers on macOS and Windows, and provide additional security boundaries), but also includes a GUI / Dashboard, Management tools, Extensions, etc.

[zamlz]

Thank you @thaJeztah for the explanation. That all makes sense.

That is unfortunate that this feature is broken at the moment. Can we confirm that this is a bug that is effecting multiple people and not just me? This will help me decide whether I should focus on debugging my system/package configuration on ubuntu or into looking for a temporary workaround.

Your work is much appreciated!

[crazy-max]

@bsousaa finished reviewing those two PR threads. Unless I'm misunderstanding something, it looks like both of them are adding the functionality I'm describing that appears to be missing my docker build command (one is a backport for the newer version).

// If the IP Address is a "host-gateway", replace this value with the
// IP address stored in the daemon level HostGatewayIP config variable.
if ip == opts.HostGatewayName {
    gateway := hostGatewayIP.String()
    if gateway == "" {
        return "", fmt.Errorf("unable to derive the IP value for host-gateway")
    }
    ip = gateway
}

There's something fishy. You're on Docker 23.0.0 and you're using Buildx 0.10.2 so it should work as intended.

Can you show me the output of docker build --help and also docker buildx ls please?

[zamlz]

Thanks @crazy-max for taking a look at this!

Here is the output of docker build --help

Usage:  docker buildx build [OPTIONS] PATH | URL | -

Start a build

Aliases:
  docker buildx build, docker buildx b

Options:
      --add-host strings              Add a custom host-to-IP mapping (format: "host:ip")
      --allow strings                 Allow extra privileged entitlement (e.g., "network.host", "security.insecure")
      --attest stringArray            Attestation parameters (format: "type=sbom,generator=image")
      --build-arg stringArray         Set build-time variables
      --build-context stringArray     Additional build contexts (e.g., name=path)
      --builder string                Override the configured builder instance (default "default")
      --cache-from stringArray        External cache sources (e.g., "user/app:cache", "type=local,src=path/to/dir")
      --cache-to stringArray          Cache export destinations (e.g., "user/app:cache", "type=local,dest=path/to/dir")
      --cgroup-parent string          Optional parent cgroup for the container
  -f, --file string                   Name of the Dockerfile (default: "PATH/Dockerfile")
      --iidfile string                Write the image ID to the file
      --label stringArray             Set metadata for an image
      --load                          Shorthand for "--output=type=docker"
      --metadata-file string          Write build result metadata to the file
      --network string                Set the networking mode for the "RUN" instructions during build (default "default")
      --no-cache                      Do not use cache when building the image
      --no-cache-filter stringArray   Do not cache specified stages
  -o, --output stringArray            Output destination (format: "type=local,dest=path")
      --platform stringArray          Set target platform for build
      --progress string               Set type of progress output ("auto", "plain", "tty"). Use plain to show container output (default "auto")
      --provenance string             Shortand for "--attest=type=provenance"
      --pull                          Always attempt to pull all referenced images
      --push                          Shorthand for "--output=type=registry"
  -q, --quiet                         Suppress the build output and print image ID on success
      --sbom string                   Shorthand for "--attest=type=sbom"
      --secret stringArray            Secret to expose to the build (format: "id=mysecret[,src=/local/secret]")
      --shm-size bytes                Size of "/dev/shm"
      --ssh stringArray               SSH agent socket or keys to expose to the build (format: "default|<id>[=<socket>|<key>[,<key>]]")
  -t, --tag stringArray               Name and optionally a tag (format: "name:tag")
      --target string                 Set the target build stage to build
      --ulimit ulimit                 Ulimit options (default [])

And here is the output of docker buildx ls

NAME/NODE DRIVER/ENDPOINT STATUS  BUILDKIT PLATFORMS
default * docker
  default default         running 23.0.0   linux/amd64, linux/amd64/v2, linux/386

[crazy-max]

Ok it seems to be a bug, I was able to repro on current 23.0.1:

$ docker build --add-host=host.docker.internal:host-gateway .
[+] Building 0.0s (0/0)                                                                                                                                                                                                                       
ERROR: failed to solve: failed to parse additional hosts: failed to parse IP host-gateway

I will take a look.

[crazy-max]

It seems #1446 and moby/moby#44537 are not enough. The error failed to parse additional hosts comes from BuildKit when build is forwarded through the gateway: https://github.com/moby/buildkit/blob/v0.10/frontend/dockerfile/builder/build.go#L122.

[dboreham]

Also affected by this. Presumably there is no test coverage for this feature, since it doesn't work in 23.0.0 and 23.0.1?

[neersighted]

@zamlz There were some improvements here with BuildKit 0.11 (shipped in v24.0.y) -- could you please try reproducing again on e.g. 24.0.1?

[dboreham]

fwiw, problem is still present in 24.0.1 for me:

$ docker version
Client: Docker Engine - Community
 Version:           24.0.1
 API version:       1.43
 Go version:        go1.20.4
 Git commit:        6802122
 Built:             Fri May 19 18:06:21 2023
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          24.0.1
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.4
  Git commit:       463850e
  Built:            Fri May 19 18:06:21 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.21
  GitCommit:        3dce8eb055cbb6872793272b4f20ed16117344f8
 runc:
  Version:          1.1.7
  GitCommit:        v1.1.7-0-g860f061
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Then:

ERROR: failed to solve: failed to parse additional hosts: failed to parse IP host-gateway