Skip to content

OverlayBD cold start of public image fails without Docker credential file #128

Description

@yingdi-shan

Affected component

OverlayBD / image resolution

AgentENV version

  • Version or commit: Reproduced on 8f028b1; still present by source inspection at 990eb0daea1cac3f49b4dd083467a90f49f8278e
  • Local modifications: none in the affected code

Host environment

  • OS: Ubuntu 24.04.4 LTS
  • Kernel (uname -a): Linux 6.17.0-41-generic x86_64
  • Architecture: x86_64
  • Bare metal, VM, or cloud: KVM virtual machine, Docker Compose deployment
  • ublk support: ublk_drv loaded
  • CPU: Intel Xeon Gold 6240R
  • Memory: approximately 31 GiB
  • Storage/filesystem: Docker volumes; underlying filesystem not collected

Operation and configuration

# agentenv-a and agentenv-b do not contain:
#   ~/.docker/config.json
#   $DOCKER_CONFIG/config.json
#   /opt/overlaybd/cred.json

aenv start --cold overlaybd/ubuntu:22.04_obd

The generated OverlayBD global configuration contains:

{
  "credentialConfig": {
    "mode": "",
    "path": "",
    "timeout": 1
  }
}

Steps to reproduce

  1. Start the standard Docker Compose deployment without mounting a Docker credential file into the runtime containers.
  2. Authenticate the CLI against the gateway.
  3. Run aenv start --cold overlaybd/ubuntu:22.04_obd.
  4. Observe that image resolution succeeds but creation of the OverlayBD runtime device fails.

Expected behavior

The public OverlayBD-native image should be accessed anonymously when no Docker credential file is configured, and the sandbox should start successfully.

Actual behavior

Image resolution succeeds, but the ublk daemon fails before Firecracker starts:

create user image overlaybd runtime device: create overlaybd runtime device via daemon:
daemon: create overlaybd runtime device failed: materialize overlaybd runtime:
resolve overlaybd base virtual size: open overlaybd image config:
failed to open lower layer 0: read credential file:
No such file or directory (os error 2)

Logs and diagnostics

2026-07-31T12:00:27Z WARN regctl failed; retrying
  failed to request auth token: EOF

2026-07-31T12:00:34Z INFO source image is overlaybd-native;
  skipping blob download; runtime registryfs_v2 will fetch on demand

2026-07-31T12:00:34Z INFO image resolved to overlaybd config
  image=docker.io/overlaybd/ubuntu:22.04_obd

2026-07-31T12:00:35Z WARN failed to start sandbox
  failed to open lower layer 0: read credential file:
  No such file or directory (os error 2)

The initial Docker Hub EOF is retried successfully and is not the fatal error.

Reproduction frequency

Always in the tested Docker Compose configuration when no credential file is present.

Regression information

  • Last known good: unknown
  • First known bad: present in 8f028b1, the initial open-source release

Additional context

write_generated_overlaybd_global_config intentionally disables credential lookup when no Docker config is detected. However:

  1. GlobalConfig::default() assigns the legacy credentialFilePath value /opt/overlaybd/cred.json.
  2. Serde supplies that default because the generated configuration omits the legacy field.
  3. normalize_compat_fields() sees an empty modern credential mode and a non-empty legacy path, then enables file credentials.
  4. registryfs_v2 tries to read /opt/overlaybd/cred.json, which does not exist.

A permanent fix should ensure an explicitly disabled credentialConfig is not overridden by a defaulted legacy field. The generated configuration could explicitly set "credentialFilePath": "", or deserialization could distinguish an omitted legacy field from one explicitly configured.

A regression test should generate and reload the global config without Docker credentials, then assert that the effective credential mode remains disabled.

Temporary workaround: create /opt/overlaybd/cred.json containing:

{"auths":{}}

Pre-submission checklist

  • I searched existing open and closed issues and did not find a duplicate.
  • I provided a minimal reproduction.
  • I removed credentials, tokens, private image references, and sensitive information.
  • This is not a security vulnerability.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions