Skip to content

Lock the container image instead of reactive version pins #2

Description

@blzee-maker

Problem

Getting the worker to boot took four rounds of reactive pinning, each found by
reading a traceback from a failed deploy:

  1. protobuf too old for Modal's injected client → container crash-loop
  2. huggingface_hub 1.x dropped proxies/resume_download, breaking
    sam-audio's BaseModel._from_pretrained
  3. Pinning hub <1.0 then broke transformers 5.x, which imports
    is_offline_mode and needs hub >=1.0
  4. Resolved by holding transformers in the 4.x line

The image now carries three hand-written constraints:

.pip_install("protobuf>=5.27,<7", "huggingface_hub>=0.26,<1.0", "transformers>=4.54,<5")

Why this is fragile

These are floating ranges layered over sam-audio's own floating ranges.
sam-audio asks for transformers>=4.54 with no ceiling and says nothing about
huggingface_hub. It also pulls four git dependencies — dacvae, imagebind,
laion-clap, perception-models — each unpinned and tracking a moving branch.

A rebuild months from now resolves differently and breaks in a new way. The
build is not reproducible, and nothing detects that until a deploy fails.

The real fix

Compile a fully-resolved lockfile and build from it:

uv pip compile modal-requirements.in -o modal-requirements.txt --generate-hashes

Then .pip_install_from_requirements("modal-requirements.txt"). Every version
is explicit and hash-checked; upgrades become a deliberate recompile with a
reviewable diff, not a surprise at deploy time.

Git dependencies need pinning to commit SHAs, not branches.

Done when

Two builds a month apart produce identical resolved versions, and a dependency
change shows up as a lockfile diff in review.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    infraBuild, deploy and runtime plumbingtech-debtShipped a workaround; the real fix is still open

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions