Skip to content

[26.1] Add namespace stripping to CachedExplicitSingularityContainerResolver - #22878

Merged
mvdbeek merged 1 commit into
galaxyproject:release_26.1from
mvdbeek:fix-cached-explicit-singularity-namespace
Jun 12, 2026
Merged

[26.1] Add namespace stripping to CachedExplicitSingularityContainerResolver#22878
mvdbeek merged 1 commit into
galaxyproject:release_26.1from
mvdbeek:fix-cached-explicit-singularity-namespace

Conversation

@mvdbeek

@mvdbeek mvdbeek commented Jun 10, 2026

Copy link
Copy Markdown
Member

Add an optional namespace config parameter that strips docker://quay.io/{namespace}/ from image identifiers when constructing the local cache path, producing flat filenames compatible with CVMFS-hosted biocontainer caches (e.g. /cvmfs/singularity.galaxyproject.org/all/).

Mirrors the same stripping already done in pull_mulled_singularity_command.

How to test the changes?

(Select all options that apply)

  • I've included appropriate automated tests.
  • This is a refactoring of components with existing test coverage.
  • Instructions for manual testing are as follows:
    1. [add testing steps and prerequisites here if you didn't write automated tests covering all your changes]

License

  • I agree to license these and all my past contributions to the core galaxy codebase under the MIT license.

@natefoo

natefoo commented Jun 10, 2026

Copy link
Copy Markdown
Member

Will this not break existing caches?

@mvdbeek

mvdbeek commented Jun 10, 2026

Copy link
Copy Markdown
Member Author

The core of the logic is

if self.namespace:
    prefix = f"docker://quay.io/{self.namespace}/"
    if image_id.startswith(prefix):
        image_id = image_id[len(prefix):]

namespace is an option you can configure (so you could put non-biocontainers on a different path).
It should not have an effect on any other caches and even the stripping is not enforced.

@natefoo

natefoo commented Jun 10, 2026

Copy link
Copy Markdown
Member

Ah yeah, no existing caches would have namespace configured.

@mvdbeek
mvdbeek force-pushed the fix-cached-explicit-singularity-namespace branch from 8813ad7 to b0a6364 Compare June 10, 2026 18:40
@natefoo

natefoo commented Jun 10, 2026

Copy link
Copy Markdown
Member

So this also doesn't use configurable cachers (i.e. CacheDirectory from mulled) meaning there's going to be a dir listing every time - this is prohibitively expensive, hence why the dir_mtime cacher exists. Could we refactor and use it here? Otherwise I don't think I can configure this in production.

FWIW I already did that in #20447.

@mvdbeek
mvdbeek force-pushed the fix-cached-explicit-singularity-namespace branch from b0a6364 to 4391493 Compare June 10, 2026 20:29
Comment thread lib/galaxy/tool_util/deps/container_resolvers/mulled.py Outdated
Add an optional `namespace` config parameter that strips
`docker://quay.io/{namespace}/` from image identifiers when constructing
the local cache path, producing flat filenames compatible with
CVMFS-hosted biocontainer caches (e.g. /cvmfs/singularity.galaxyproject.org/all/).

Mirrors the same stripping already done in pull_mulled_singularity_command.
@mvdbeek
mvdbeek force-pushed the fix-cached-explicit-singularity-namespace branch from fbecda0 to 329e36b Compare June 11, 2026 13:57
@natefoo

natefoo commented Jun 11, 2026

Copy link
Copy Markdown
Member

As an aside, very hard to tell what changes with each iteration. For someone watching and reviewing it'd be easier if Claude pushed new commits and then just squashed at the end when we're happy.

pull Bot pushed a commit to AKJUS/usegalaxy-playbook that referenced this pull request Jun 11, 2026
Uses galaxyproject/galaxy#22878 to resolve explicit docker://quay.io/biocontainers/
container specs against /cvmfs/singularity.galaxyproject.org/all before falling
back to pulling. Applied to all destinations that mount singularity.galaxyproject.org
(cyclone, jetstream2, tacc_hpc, bridges2, expanse, anvil, stampede3_nvdimm).
@mvdbeek
mvdbeek merged commit 42f1ef7 into galaxyproject:release_26.1 Jun 12, 2026
64 of 73 checks passed
@github-project-automation github-project-automation Bot moved this from Needs Review to Done in Galaxy Dev - weeklies Jun 12, 2026
@github-actions

Copy link
Copy Markdown

This PR was merged without a "kind/" label, please correct.

@nsoranzo
nsoranzo deleted the fix-cached-explicit-singularity-namespace branch June 12, 2026 14:16
@nsoranzo nsoranzo removed this from the 26.2 milestone Jun 12, 2026
@nsoranzo nsoranzo added this to the 26.1 milestone Jun 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Development

Successfully merging this pull request may close these issues.

3 participants