chunkpull

Pull OCI/Docker images through HTTP proxies that cap per-connection transfer size. In some restricted networks the Docker daemon can't reach the registry at all, or an intercepting proxy resets any single download that exceeds a size/time limit — so docker pull, crane pull, and skopeo copy all fail on large image layers even though small requests succeed.

chunkpull downloads every blob in small HTTP Range chunks and writes a docker load-able tarball, so no single connection ever exceeds the limit.

Install

go install github.com/vic322/chunkpull@latest

Usage

# Write a tarball, then load it into the local daemon.
export HTTPS_PROXY=http://proxy.example:8080
chunkpull -o wp.tar wordpress:6.8-php8.3-apache
docker load -i wp.tar

# Or pull several images and load them in one step.
chunkpull -load -o images.tar wordpress:6.8-php8.3-apache mysql:8.0

# Smaller chunks for a stricter proxy; pick a platform.
chunkpull -chunk $((16*1024*1024)) -arch arm64 alpine:3.20

Flag	Default	Meaning
-o	images.tar	Output docker-archive tarball
-load	false	Run docker load on the tarball afterwards
-chunk	33554432	Chunk size in bytes (must stay under the cap)
-os	linux	Target OS
-arch	host arch	Target architecture
-retries	8	Retries per chunk

Proxy settings come from HTTPS_PROXY / HTTP_PROXY / NO_PROXY.

How it works

chunkpull wraps the registry HTTP transport. For every plain GET, the first request is sent as Range: bytes=0-<chunk-1>:

• 206 Partial Content with a body larger than one chunk → the rest is fetched as sequential ranges and reassembled into one 200 OK stream.
• 307/302 redirect (registry → CDN) → returned untouched; the client follows it and the CDN GET re-enters the same logic.
• plain 200 (server ignores ranges) → passed through unchanged.

The decision is based only on the server advertising range support, so it works against any registry/CDN without hardcoding hostnames.

Limitations

• The registry/CDN must support Range requests for the layer blobs (Docker Hub, GHCR, ECR, and most CDNs do).
• This is a workaround for a broken network path. If you control the environment, a pull-through cache registry (e.g. registry:2 in proxy mode, Harbor, Zot) placed where egress is open is the sturdier fix.

License

MIT