On a host with no IPv6 capabilities the plugin crashes when blocking the IMDS endpoint

[racingjellyfish]

Summary

Since v1.69.0 of the ECS agent the plugin crashes, with the latest version, 1.77.0, the service can start a task, but it cannot connect to the network. As per AWS recommendations we're using awsvpc mode and blocking the IMDS endpoint, which appears to be the cause of the issue.

Description

We see an error in the logs, see below, when the plugin attempts to add the blackhole route for the IPv6 address of the IMDS, this line in the code appears to be what's failing.

The AMI that we use for the EC2 instance does not have IPv6 enabled for security reasons.

Expected Behavior

The plugin handles the resulting error gracefully, or provides some means of disabling the IPv6 support to avoid the error.

Observed Behavior

The plugin crashes and fails to finish configuring the network routes for the task, e.g. the default route is missing from the task route table, etc.

Route table with latest agent and ECS_AWSVPC_BLOCK_IMDS=true:

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
169.254.169.254 *               255.255.255.255 UH    0      0        0 *
169.254.170.2   169.254.172.1   255.255.255.255 UGH   0      0        0 ecs-eth0
169.254.172.1   169.254.172.1   255.255.255.255 UGH   0      0        0 ecs-eth0

Route table with latest agent and ECS_AWSVPC_BLOCK_IMDS=false:

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         ip-10-116-128-1 0.0.0.0         UG    0      0        0 eth0
10.116.128.128  *               255.255.255.224 U     0      0        0 eth0
169.254.170.2   169.254.172.1   255.255.255.255 UGH   0      0        0 ecs-eth0
169.254.172.1   169.254.172.1   255.255.255.255 UGH   0      0        0 ecs-eth0

Route table with v1.68.2 of the agent and ECS_AWSVPC_BLOCK_IMDS=true:

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         ip-10-116-128-1 0.0.0.0         UG    0      0        0 eth0
10.116.128.160  *               255.255.255.224 U     0      0        0 eth0
169.254.169.254 *               255.255.255.255 UH    0      0        0 *
169.254.170.2   169.254.172.1   255.255.255.255 UGH   0      0        0 ecs-eth0
169.254.172.1   169.254.172.1   255.255.255.255 UGH   0      0        0 ecs-eth0

Environment Details

docker info:

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc., v0.0.0+unknown)

Server:
 Containers: 3
  Running: 3
  Paused: 0
  Stopped: 0
 Images: 3
 Server Version: 20.10.25
 Storage Driver: overlay2
  Backing Filesystem: xfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 1e1ea6e986c6c86565bc33d52e34b81b3e2bc71f
 runc version: f19387a6bec4944c770f7668ab51c4348d9c2f38
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.10.197-186.748.amzn2.x86_64
 Operating System: Amazon Linux 2
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 15.43GiB
 Name: ip-X-X-X-X.eu-west-2.compute.internal
 ID: U742:Y4OW:URO4:EH6C:LBN5:5HMC:HHOP:LWGQ:6RIV:FTJY:KJW3:QBBQ
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

[racingjellyfish]

FYI - We're using CIS recommended way of setting things up when hardening the base AMI and therefore disable IPv6 by adding ipv6.disable=1 to kernel options in /etc/default/grub, if that helps with debugging and/or fixing the issue?