Pipeline download feature does not appear to handle Docker image conversion correctly

[scwatts]

Overview

Recent changes to the nf-core pipelines download feature appear to have introduced a bug during the conversion of Docker images to Singularity SIF files, and impacts the ability to successfully run the download_pipeline.yml GH Action workflow jobs

The cause is likely related to different output filepaths / filenames of the converted Docker images:

previously: nf-core-gridss-2.13.2--1.img
now:        nf-core-gridss-2.13.2--1.img/tempfile

In the context of the relevant GH Action workflow, the 'Download the pipeline' step completes successfully but the 'Run the downloaded pipeline' step fails with the following error:

  INFO:    Environment variable SINGULARITYENV_NXF_TASK_WORKDIR is set, but APPTAINERENV_NXF_TASK_WORKDIR is preferred
  INFO:    Environment variable SINGULARITYENV_NXF_DEBUG is set, but APPTAINERENV_NXF_DEBUG is preferred
  WARNING: passwd file doesn't exist in container, not updating
  WARNING: group file doesn't exist in container, not updating
  WARNING: Skipping mount /tmp [tmp]: /tmp doesn't exist in container
  WARNING: Skipping mount /var/tmp [tmp]: /var/tmp doesn't exist in container
  FATAL:   stat /bin/bash: no such file or directory

I expect this is because the singularity command isn't receiving the actual SIF file but rather the directory containing the SIF file, which is now named tempfile

Additional information

Example GH Actions runs

To help you debug, I have two useful GH Action workflow runs:

• an example of the above failure in oncoanalyser with the latest nf-core/tools dev codebase (link), and
• the same oncoanalyser version but with the nf-core/tools codebase prior to the download refactor (link)

Minimal, locally reproducible example

I have also put together a minimal reproducible example that you can use to debug locally:

# Get software
conda create -y -n run nextflow==25.04.* python==3.13.* apptainer==1.3.* gcc clang gxx clangxx
conda activate run

python3 -m venv venv/
. venv/bin/activate

python -m pip install --upgrade pip

# NOTE(SW): pinned to latest commit on `dev` at time of writting
#pip install git+https://github.com/nf-core/tools.git@dev
pip install git+https://github.com/nf-core/tools.git@e98ae9b


# Download images
mkdir -p ./singularity_container_images

export NXF_SINGULARITY_CACHEDIR=./singularity_container_images

nf-core pipelines download nf-core/oncoanalyser \
  --revision nf-core-tools-download-mre \
  --outdir ./oncoanalyser \
  --compress "none" \
  --container-system 'singularity' \
  --container-library "quay.io" -l "docker.io" -l "community.wave.seqera.io/library/" \
  --container-cache-utilisation 'amend' \
  --download-configuration 'yes'


# Run test, trigger error
export NXF_SINGULARITY_CACHEDIR=./singularity_container_images
export NXF_SINGULARITY_HOME_MOUNT=true

nextflow run ./oncoanalyser/nf_core_tools_download_mre \
  -profile test,singularity \
  --outdir ./results

[scwatts]

Let me know if there is any further context or information that might be helpful to provide

[mirpedrol]

Hi @scwatts and @johandahlberg, thanks for reporting this issue.
I have tried your commands to reproduce the error, running them on GitPod.
My pipeline run succeeds; it runs fastqc and multiqc. Is this expected, or am I missing something?

[JulianFlesch]

I also wasn't able to reproduce this error.

[JulianFlesch]

After download, singularity images seem to be successfully stored in ./singularity_container_images which is set as NXF_SINGULARITY_CACHEDIR. Pipeline files also see to be present.

Interestingly, the nextflow command in our workflow to test run tries to use the test_stub profile. This is not defined in the TEMPLATE and also neither in oncoanalyser nor pixelator.

Here is the nextflow command run:

nextflow run ./oncoanalyser/$( sed 's/\W/_/g' <<< dev) -stub -profile test_stub,singularity --outdir ./results
  shell: /usr/bin/bash -e {0}
  env:
    NXF_ANSI_LOG: false
    JAVA_HOME: /opt/hostedtoolcache/Java_Zulu_jdk/17.0.16-8/x64
    JAVA_HOME_17_X64: /opt/hostedtoolcache/Java_Zulu_jdk/17.0.16-8/x64
    CAPSULE_LOG: none
    pythonLocation: /opt/hostedtoolcache/Python/3.13.7/x64
    PKG_CONFIG_PATH: /opt/hostedtoolcache/Python/3.13.7/x64/lib/pkgconfig
    Python_ROOT_DIR: /opt/hostedtoolcache/Python/3.13.7/x64
    Python2_ROOT_DIR: /opt/hostedtoolcache/Python/3.13.7/x64
    Python3_ROOT_DIR: /opt/hostedtoolcache/Python/3.13.7/x64
    LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.13.7/x64/lib
    NXF_SINGULARITY_CACHEDIR: ./singularity_container_images
    NXF_SINGULARITY_HOME_MOUNT: true

---

Update:

The workflow that was run with the release seems to vary from the workflow on the oncoanalyser TEMPLATE branch and the TEMPLATE in nf-core tools (Introduced in this merge
nf-core/oncoanalyser@ced6beb)

This is not the case for the pixelator pipeline.

[scwatts]

Thanks for the feedback, I've now containerised the MRE so hopefully that helps you reproduce the error

Containerised MRE

Command

image=docker.io/scwatts/nfcoretools-download-debug:ubuntu-24.04
docker run --privileged ${image} /usr/local/bin/run.sh 2>&1 | tee log.ubuntu-24.04.txt

Log excerpt

  INFO:    Environment variable SINGULARITYENV_NXF_TASK_WORKDIR is set, but APPTAINERENV_NXF_TASK_WORKDIR is preferred
  INFO:    Environment variable SINGULARITYENV_NXF_DEBUG is set, but APPTAINERENV_NXF_DEBUG is preferred
  WARNING: passwd file doesn't exist in container, not updating
  WARNING: group file doesn't exist in container, not updating
  WARNING: Skipping mount /tmp [tmp]: /tmp doesn't exist in container
  WARNING: Skipping mount /var/tmp [tmp]: /var/tmp doesn't exist in container
  FATAL:   stat /bin/bash: no such file or directory

Notes

• the MRE uses nf-core tools 3.3.2 template where the FASTQC module is modified to have only a Docker image available
• in order for nf-core tools to access the MRE, it is placed in nf-core/oncoanalyser (nf-core-tools-download-mre)
• given the difficulty replicating the error there may be some interaction with the underlying system
• containerisation has been done using the same Linux distribution, Ubuntu 24.04 (20250824), as the GH Action runner

Additional files

Dockerfile [nfcoretools-download-debug:ubuntu-24.04] (click to show)

FROM docker.io/ubuntu:24.04

RUN \
  apt update && \
  apt install -y curl systemd

RUN \
  curl -L "https://repo.anaconda.com/miniconda/Miniconda3-py310_25.5.1-0-Linux-$(uname -m).sh" > ~/miniconda.sh && \
  /bin/bash ~/miniconda.sh -b -p /opt/miniconda/ && \
  /opt/miniconda/bin/conda init

ENV PATH="/opt/miniconda/bin/:${PATH}"

RUN \
  conda config --prepend channels bioconda && \
  conda config --prepend channels conda-forge && \
  conda config --system --remove channels defaults && \
  conda config --remove channels defaults

RUN \
    echo -e > ~/.condarc '\
solver: libmamba\n\
channels:\n\
    - conda-forge\n\
    - bioconda'

RUN \
    conda create -y -p /env/ \
        'python==3.13.*' \
        'apptainer==1.3.*'  \
        'nextflow==25.04.*' \
        'tree' \
        'git' \
        'gcc' \
        'gxx' \
        'clang' \
        'clangxx'

RUN \
    conda clean -yaf

ENV PATH="/env/bin/:${PATH}"
ENV LD_LIBRARY_PATH="/env/lib/:${LD_LIBRARY_PATH}"

# Install Python dependencies
# NOTE(SW): pinned to latest commit on `dev` at time of writting
RUN \
  pip install git+https://github.com/nf-core/tools.git@e98ae9b

COPY run.sh /usr/local/bin/

/usr/local/bin/run.sh (click to show)

#!/usr/bin/env bash
set -exuo pipefail

# Download images
mkdir -p ./singularity_container_images

export NXF_SINGULARITY_CACHEDIR=./singularity_container_images

nf-core pipelines download nf-core/oncoanalyser \
  --revision nf-core-tools-download-mre \
  --outdir ./oncoanalyser \
  --compress "none" \
  --container-system 'singularity' \
  --container-library "quay.io" -l "docker.io" -l "community.wave.seqera.io/library/" \
  --container-cache-utilisation 'amend' \
  --download-configuration 'yes'


# List downloaded images
tree ${NXF_SINGULARITY_CACHEDIR};


# Run pipeline with downloaded images
export NXF_SINGULARITY_CACHEDIR=./singularity_container_images
export NXF_SINGULARITY_HOME_MOUNT=true

nextflow run ./oncoanalyser/nf_core_tools_download_mre \
  -ansi-log false \
  -profile test,singularity \
  --outdir ./results

Full command log (click to show)

Unable to find image 'scwatts/nfcoretools-download-debug:ubuntu-24.04' locally
ubuntu-24.04: Pulling from scwatts/nfcoretools-download-debug
76249c7cd503: Already exists
03ed72dc0e15: Pulling fs layer
191d92a37ba0: Pulling fs layer
cb87505370b5: Pulling fs layer
d9a6b87844fa: Pulling fs layer
7957a7bf9064: Pulling fs layer
237b1a0a61cc: Pulling fs layer
9da8e7fe5375: Pulling fs layer
43646debfe3b: Pulling fs layer
d9a6b87844fa: Waiting
7957a7bf9064: Waiting
237b1a0a61cc: Waiting
43646debfe3b: Waiting
9da8e7fe5375: Waiting
cb87505370b5: Verifying Checksum
cb87505370b5: Download complete
d9a6b87844fa: Download complete
03ed72dc0e15: Verifying Checksum
03ed72dc0e15: Download complete
237b1a0a61cc: Verifying Checksum
237b1a0a61cc: Download complete
03ed72dc0e15: Pull complete
9da8e7fe5375: Verifying Checksum
9da8e7fe5375: Download complete
43646debfe3b: Verifying Checksum
43646debfe3b: Download complete
191d92a37ba0: Verifying Checksum
191d92a37ba0: Download complete
191d92a37ba0: Pull complete
cb87505370b5: Pull complete
d9a6b87844fa: Pull complete
7957a7bf9064: Verifying Checksum
7957a7bf9064: Download complete
7957a7bf9064: Pull complete
237b1a0a61cc: Pull complete
9da8e7fe5375: Pull complete
43646debfe3b: Pull complete
Digest: sha256:97b89728d78e262cdc7ef106ca35ef8de212cbaa8db12d138f119267cfaaa067
Status: Downloaded newer image for scwatts/nfcoretools-download-debug:ubuntu-24.04
+ mkdir -p ./singularity_container_images
+ export NXF_SINGULARITY_CACHEDIR=./singularity_container_images
+ NXF_SINGULARITY_CACHEDIR=./singularity_container_images
+ nf-core pipelines download nf-core/oncoanalyser --revision nf-core-tools-download-mre --outdir ./oncoanalyser --compress none --container-system singularity --container-library quay.io -l docker.io -l community.wave.seqera.io/library/ --container-cache-utilisation amend --download-configuration yes


                                          ,--./,-.
          ___     __   __   __   ___     /,-._.--~\ 
    |\ | |__  __ /  ` /  \ |__) |__         }  {
    | \| |       \__, \__/ |  \ |___     \`-._,-`-,
                                          `._,._,'

    nf-core/tools version 3.4.0.dev0 - https://nf-co.re


WARNING  Could not find GitHub authentication token. Some API requests may fail.
INFO     Detected Nextflow version 25.04.6                                      
INFO     Saving 'nf-core/oncoanalyser'                                          
         Pipeline revision: 'nf-core-tools-download-mre'                        
         Use containers: 'singularity'                                          
         Container library: 'quay.io, docker.io,                                
         community.wave.seqera.io/library/'                                     
         Using NXF_SINGULARITY_CACHEDIR': ./singularity_container_images'       
         Output directory: 'oncoanalyser'                                       
         Include default institutional configuration: 'True'                    
INFO     Downloading workflow                                                   
INFO     Downloading centralised configs from GitHub                            
INFO     Downloading workflow files from GitHub                                 
INFO     Fetching container names for workflow revision                         
         nf-core-tools-download-mre using nextflow inspect. This might take a   
         while.                                                                 
INFO     Processing workflow revision nf-core-tools-download-mre, found 2       
         container images in total.                                             

+ tree ./singularity_container_images
./singularity_container_images
|-- biocontainers-fastqc-0.12.1--hdfd78af_0.img
|   `-- tempfile
|-- community-cr-prod.seqera.io-docker-registry-v2-biocontainers-fastqc-0.12.1--hdfd78af_0.img -> biocontainers-fastqc-0.12.1--hdfd78af_0.img
|-- community-cr-prod.seqera.io-docker-registry-v2-multiqc-1.29--pyhdfd78af_0.img -> multiqc-1.29--pyhdfd78af_0.img
|-- community.wave.seqera.io-library--biocontainers-fastqc-0.12.1--hdfd78af_0.img -> biocontainers-fastqc-0.12.1--hdfd78af_0.img
|-- community.wave.seqera.io-library--multiqc-1.29--pyhdfd78af_0.img -> multiqc-1.29--pyhdfd78af_0.img
|-- community.wave.seqera.io-library-biocontainers-fastqc-0.12.1--hdfd78af_0.img -> biocontainers-fastqc-0.12.1--hdfd78af_0.img
|-- community.wave.seqera.io-library-multiqc-1.29--pyhdfd78af_0.img -> multiqc-1.29--pyhdfd78af_0.img
|-- depot.galaxyproject.org-singularity-biocontainers-fastqc-0.12.1--hdfd78af_0.img -> biocontainers-fastqc-0.12.1--hdfd78af_0.img
|-- depot.galaxyproject.org-singularity-multiqc-1.29--pyhdfd78af_0.img -> multiqc-1.29--pyhdfd78af_0.img
|-- docker.io-biocontainers-fastqc-0.12.1--hdfd78af_0.img -> biocontainers-fastqc-0.12.1--hdfd78af_0.img
|-- docker.io-multiqc-1.29--pyhdfd78af_0.img -> multiqc-1.29--pyhdfd78af_0.img
|-- multiqc-1.29--pyhdfd78af_0.img
|-- quay.io-biocontainers-fastqc-0.12.1--hdfd78af_0.img -> biocontainers-fastqc-0.12.1--hdfd78af_0.img
`-- quay.io-multiqc-1.29--pyhdfd78af_0.img -> multiqc-1.29--pyhdfd78af_0.img

8 directories, 8 files
+ export NXF_SINGULARITY_CACHEDIR=./singularity_container_images
+ NXF_SINGULARITY_CACHEDIR=./singularity_container_images
+ export NXF_SINGULARITY_HOME_MOUNT=true
+ NXF_SINGULARITY_HOME_MOUNT=true
+ nextflow run ./oncoanalyser/nf_core_tools_download_mre -ansi-log false -profile test,singularity --outdir ./results
N E X T F L O W  ~  version 25.04.6
Launching `./oncoanalyser/nf_core_tools_download_mre/main.nf` [compassionate_panini] DSL2 - revision: 7fee41ef91

------------------------------------------------------
                                        ,--./,-.
        ___     __   __   __   ___     /,-._.--~'
  |\ | |__  __ /  ` /  \ |__) |__         }  {
  | \| |       \__, \__/ |  \ |___     \`-._,-`-,
                                        `._,._,'
  nf-core/oncoanalyser 1.0.0dev
------------------------------------------------------
�[1mInput/output options�[0m
  �[0;34minput                     : �[0;32mhttps://raw.githubusercontent.com/nf-core/test-datasets/viralrecon/samplesheet/samplesheet_test_illumina_amplicon.csv�[0m
  �[0;34moutdir                    : �[0;32m./results�[0m

�[1mReference genome options�[0m
  �[0;34mgenome                    : �[0;32mR64-1-1�[0m
  �[0;34mfasta                     : �[0;32ms3://ngi-igenomes/igenomes//Saccharomyces_cerevisiae/Ensembl/R64-1-1/Sequence/WholeGenomeFasta/genome.fa�[0m

�[1mInstitutional config options�[0m
  �[0;34mcustom_config_base        : �[0;32m/oncoanalyser/nf_core_tools_download_mre/../configs/�[0m
  �[0;34mconfig_profile_name       : �[0;32mTest profile�[0m
  �[0;34mconfig_profile_description: �[0;32mMinimal test dataset to check pipeline function�[0m

�[1mGeneric options�[0m
  �[0;34mtrace_report_suffix       : �[0;32m2025-09-03_03-21-48�[0m

�[1mCore Nextflow options�[0m
  �[0;34mrunName                   : �[0;32mcompassionate_panini�[0m
  �[0;34mcontainerEngine           : �[0;32msingularity�[0m
  �[0;34mlaunchDir                 : �[0;32m/�[0m
  �[0;34mworkDir                   : �[0;32m/work�[0m
  �[0;34mprojectDir                : �[0;32m/oncoanalyser/nf_core_tools_download_mre�[0m
  �[0;34muserName                  : �[0;32mroot�[0m
  �[0;34mprofile                   : �[0;32mtest,singularity�[0m
  �[0;34mconfigFiles               : �[0;32m/oncoanalyser/nf_core_tools_download_mre/nextflow.config�[0m

!! Only displaying parameters that differ from the pipeline defaults !!
-�[2m----------------------------------------------------�[0m-
* The nf-core framework
    https://doi.org/10.1038/s41587-020-0439-x

* Software dependencies
    https://github.com/nf-core/oncoanalyser/blob/master/CITATIONS.md

[0f/d74fa9] Submitted process > NFCORE_ONCOANALYSER:ONCOANALYSER:FASTQC (SAMPLE2_PE)
[e5/0d29b5] Submitted process > NFCORE_ONCOANALYSER:ONCOANALYSER:FASTQC (SAMPLE3_SE)
ERROR ~ Error executing process > 'NFCORE_ONCOANALYSER:ONCOANALYSER:FASTQC (SAMPLE2_PE)'

Caused by:
  Process `NFCORE_ONCOANALYSER:ONCOANALYSER:FASTQC (SAMPLE2_PE)` terminated with an error exit status (255)


Command executed:

  printf "%s %s\n" sample2_R1.fastq.gz SAMPLE2_PE_1.gz sample2_R2.fastq.gz SAMPLE2_PE_2.gz | while read old_name new_name; do
      [ -f "${new_name}" ] || ln -s $old_name $new_name
  done
  
  fastqc \
      --quiet \
      --threads 4 \
      --memory 3840 \
      SAMPLE2_PE_1.gz SAMPLE2_PE_2.gz
  
  cat <<-END_VERSIONS > versions.yml
  "NFCORE_ONCOANALYSER:ONCOANALYSER:FASTQC":
      fastqc: $( fastqc --version | sed '/FastQC v/!d; s/.*v//' )
  END_VERSIONS

Command exit status:
  255

Command output:
  (empty)

Command error:
  INFO:    Environment variable SINGULARITYENV_NXF_TASK_WORKDIR is set, but APPTAINERENV_NXF_TASK_WORKDIR is preferred
  INFO:    Environment variable SINGULARITYENV_NXF_DEBUG is set, but APPTAINERENV_NXF_DEBUG is preferred
  WARNING: passwd file doesn't exist in container, not updating
  WARNING: group file doesn't exist in container, not updating
  WARNING: Skipping mount /tmp [tmp]: /tmp doesn't exist in container
  WARNING: Skipping mount /var/tmp [tmp]: /var/tmp doesn't exist in container
  FATAL:   stat /bin/bash: no such file or directory

Work dir:
  /work/0f/d74fa9f95927b20d0e9950d61ee151

Container:
  /./singularity_container_images/quay.io-biocontainers-fastqc-0.12.1--hdfd78af_0.img

Tip: view the complete command output by changing to the process work dir and entering the command `cat .command.out`

 -- Check '.nextflow.log' file for details
Execution cancelled -- Finishing pending tasks before exit
ERROR ~ Pipeline failed. Please refer to troubleshooting docs: https://nf-co.re/docs/usage/troubleshooting

 -- Check '.nextflow.log' file for details
-�[0;35m[nf-core/oncoanalyser]�[0;31m Pipeline completed with errors�[0m-

Details

[JulianFlesch]

The above PR solves the issue you describe.
nf-core pipelines download doesn't properly clean up after the temporary files it creates when using singularity pull internally. This should be fixed now ✅

However, your MRE then runs into another issue with the multiqc process.
I think this is due to wrong file permissions of some resulting image files and related to this #3730

(base) root@6544c2b9cc46:/usr/local/bin# ./run.sh 
+ mkdir -p ./singularity_container_images
+ export NXF_SINGULARITY_CACHEDIR=./singularity_container_images
+ NXF_SINGULARITY_CACHEDIR=./singularity_container_images
+ nf-core pipelines download nf-core/oncoanalyser --revision nf-core-tools-download-mre --outdir ./oncoanalyser --compress none --container-system singularity --container-library quay.io -l docker.io -l community.wave.seqera.io/library/ --container-cache-utilisation amend --download-configuration yes


                                          ,--./,-.
          ___     __   __   __   ___     /,-._.--~\ 
    |\ | |__  __ /  ` /  \ |__) |__         }  {
    | \| |       \__, \__/ |  \ |___     \`-._,-`-,
                                          `._,._,'

    nf-core/tools version 3.4.0.dev0 - https://nf-co.re


WARNING  Could not find GitHub authentication token. Some API requests may fail.                                                                                                                                           
INFO     Detected Nextflow version 25.04.6                                                                                                                                                                                 
INFO     Saving 'nf-core/oncoanalyser'                                                                                                                                                                                     
         Pipeline revision: 'nf-core-tools-download-mre'                                                                                                                                                                   
         Use containers: 'singularity'                                                                                                                                                                                     
         Container library: 'quay.io, docker.io, community.wave.seqera.io/library/'                                                                                                                                        
         Using NXF_SINGULARITY_CACHEDIR': ./singularity_container_images'                                                                                                                                                  
         Output directory: 'oncoanalyser'                                                                                                                                                                                  
         Include default institutional configuration: 'True'                                                                                                                                                               
INFO     Downloading workflow                                                                                                                                                                                              
INFO     Downloading centralised configs from GitHub                                                                                                                                                                       
INFO     Downloading workflow files from GitHub                                                                                                                                                                            
INFO     Fetching container names for workflow revision nf-core-tools-download-mre using nextflow inspect. This might take a while.                                                                                        
INFO     Processing workflow revision nf-core-tools-download-mre, found 2 container images in total.                                                                                                                       

+ tree ./singularity_container_images
./singularity_container_images
|-- biocontainers-fastqc-0.12.1--hdfd78af_0.img
|-- community-cr-prod.seqera.io-docker-registry-v2-biocontainers-fastqc-0.12.1--hdfd78af_0.img -> biocontainers-fastqc-0.12.1--hdfd78af_0.img
|-- community-cr-prod.seqera.io-docker-registry-v2-multiqc-1.29--pyhdfd78af_0.img -> multiqc-1.29--pyhdfd78af_0.img
|-- community.wave.seqera.io-library--biocontainers-fastqc-0.12.1--hdfd78af_0.img -> biocontainers-fastqc-0.12.1--hdfd78af_0.img
|-- community.wave.seqera.io-library--multiqc-1.29--pyhdfd78af_0.img -> multiqc-1.29--pyhdfd78af_0.img
|-- community.wave.seqera.io-library-biocontainers-fastqc-0.12.1--hdfd78af_0.img -> biocontainers-fastqc-0.12.1--hdfd78af_0.img
|-- community.wave.seqera.io-library-multiqc-1.29--pyhdfd78af_0.img -> multiqc-1.29--pyhdfd78af_0.img
|-- depot.galaxyproject.org-singularity-biocontainers-fastqc-0.12.1--hdfd78af_0.img -> biocontainers-fastqc-0.12.1--hdfd78af_0.img
|-- depot.galaxyproject.org-singularity-multiqc-1.29--pyhdfd78af_0.img -> multiqc-1.29--pyhdfd78af_0.img
|-- docker.io-biocontainers-fastqc-0.12.1--hdfd78af_0.img -> biocontainers-fastqc-0.12.1--hdfd78af_0.img
|-- docker.io-multiqc-1.29--pyhdfd78af_0.img -> multiqc-1.29--pyhdfd78af_0.img
|-- multiqc-1.29--pyhdfd78af_0.img
|-- quay.io-biocontainers-fastqc-0.12.1--hdfd78af_0.img -> biocontainers-fastqc-0.12.1--hdfd78af_0.img
`-- quay.io-multiqc-1.29--pyhdfd78af_0.img -> multiqc-1.29--pyhdfd78af_0.img

1 directory, 14 files
+ export NXF_SINGULARITY_CACHEDIR=./singularity_container_images
+ NXF_SINGULARITY_CACHEDIR=./singularity_container_images
+ export NXF_SINGULARITY_HOME_MOUNT=true
+ NXF_SINGULARITY_HOME_MOUNT=true
+ nextflow run ./oncoanalyser/nf_core_tools_download_mre -ansi-log false -profile test,singularity --outdir ./results
N E X T F L O W  ~  version 25.04.6
Launching `./oncoanalyser/nf_core_tools_download_mre/main.nf` [wise_heisenberg] DSL2 - revision: 7fee41ef91

------------------------------------------------------
                                        ,--./,-.
        ___     __   __   __   ___     /,-._.--~'
  |\ | |__  __ /  ` /  \ |__) |__         }  {
  | \| |       \__, \__/ |  \ |___     \`-._,-`-,
                                        `._,._,'
  nf-core/oncoanalyser 1.0.0dev
------------------------------------------------------
Input/output options
  input                     : https://raw.githubusercontent.com/nf-core/test-datasets/viralrecon/samplesheet/samplesheet_test_illumina_amplicon.csv
  outdir                    : ./results

Reference genome options
  genome                    : R64-1-1
  fasta                     : s3://ngi-igenomes/igenomes//Saccharomyces_cerevisiae/Ensembl/R64-1-1/Sequence/WholeGenomeFasta/genome.fa

Institutional config options
  custom_config_base        : /usr/local/bin/oncoanalyser/nf_core_tools_download_mre/../configs/
  config_profile_name       : Test profile
  config_profile_description: Minimal test dataset to check pipeline function

Generic options
  trace_report_suffix       : 2025-09-08_15-55-41

Core Nextflow options
  runName                   : wise_heisenberg
  containerEngine           : singularity
  launchDir                 : /usr/local/bin
  workDir                   : /usr/local/bin/work
  projectDir                : /usr/local/bin/oncoanalyser/nf_core_tools_download_mre
  userName                  : root
  profile                   : test,singularity
  configFiles               : /usr/local/bin/oncoanalyser/nf_core_tools_download_mre/nextflow.config

!! Only displaying parameters that differ from the pipeline defaults !!
------------------------------------------------------
* The nf-core framework
    https://doi.org/10.1038/s41587-020-0439-x

* Software dependencies
    https://github.com/nf-core/oncoanalyser/blob/master/CITATIONS.md

[36/604551] Submitted process > NFCORE_ONCOANALYSER:ONCOANALYSER:FASTQC (SAMPLE3_SE)
[ea/46a22c] Submitted process > NFCORE_ONCOANALYSER:ONCOANALYSER:FASTQC (SAMPLE2_PE)
[31/318c09] Submitted process > NFCORE_ONCOANALYSER:ONCOANALYSER:FASTQC (SAMPLE1_PE)
[10/6be6e7] Submitted process > NFCORE_ONCOANALYSER:ONCOANALYSER:MULTIQC
ERROR ~ Error executing process > 'NFCORE_ONCOANALYSER:ONCOANALYSER:MULTIQC'

Caused by:
  Process `NFCORE_ONCOANALYSER:ONCOANALYSER:MULTIQC` terminated with an error exit status (127)


Command executed:

  multiqc \
      --force \
       \
      --config multiqc_config.yml \
       \
       \
       \
       \
       \
      .
  
  cat <<-END_VERSIONS > versions.yml
  "NFCORE_ONCOANALYSER:ONCOANALYSER:MULTIQC":
      multiqc: $( multiqc --version | sed -e "s/multiqc, version //g" )
  END_VERSIONS

Command exit status:
  127

Command output:
  (empty)

Command error:
  INFO:    Environment variable SINGULARITYENV_NXF_TASK_WORKDIR is set, but APPTAINERENV_NXF_TASK_WORKDIR is preferred
  INFO:    Environment variable SINGULARITYENV_NXF_DEBUG is set, but APPTAINERENV_NXF_DEBUG is preferred
  .command.sh: line 3: multiqc: command not found

Work dir:
  /usr/local/bin/work/10/6be6e7d520eeb85ebc81cde6fa2002

Container:
  /usr/local/bin/./singularity_container_images/depot.galaxyproject.org-singularity-multiqc-1.29--pyhdfd78af_0.img

Tip: when you have fixed the problem you can continue the execution adding the option `-resume` to the run command line

 -- Check '.nextflow.log' file for details
Execution cancelled -- Finishing pending tasks before exit
ERROR ~ Pipeline failed. Please refer to troubleshooting docs: https://nf-co.re/docs/usage/troubleshooting

 -- Check '.nextflow.log' file for details
-[nf-core/oncoanalyser] Pipeline completed with errors-

[scwatts]

Thank you so much! I have been able to successfully run the download command and subsequent Nextflow test run using your issues/3742-download branch at 1f1c346 for the following:

• oncoanalyser 2.2.0 GH Action workflow (run: #89; branch: check-tools-download--fix)
• updated MRE (see details below)

Additional details

Commands

image=docker.io/scwatts/nfcoretools-download-debug:ubuntu-24.04--fix
docker run --privileged ${image} /usr/local/bin/run.sh 2>&1 | tee log.ubuntu-24.04.txt

log.ubuntu-24.04.txt (click to show)

Unable to find image 'scwatts/nfcoretools-download-debug:ubuntu-24.04--fix' locally
ubuntu-24.04--fix: Pulling from scwatts/nfcoretools-download-debug
76249c7cd503: Already exists
097cf687f4f0: Already exists
35395d6aeec0: Already exists
a8f9393fd839: Already exists
998104be59b2: Already exists
611e71450c89: Already exists
a93cf2c86c51: Already exists
21c79a54d09f: Already exists
dea8b595c6c4: Already exists
Digest: sha256:5993cb81aa2196f66c6b5d69265873c2689c8159d249e36ebee8d61bd8380003
Status: Downloaded newer image for scwatts/nfcoretools-download-debug:ubuntu-24.04--fix
+ mkdir -p ./singularity_container_images
+ export NXF_SINGULARITY_CACHEDIR=./singularity_container_images
+ NXF_SINGULARITY_CACHEDIR=./singularity_container_images
+ nf-core pipelines download nf-core/oncoanalyser --revision nf-core-tools-download-mre --outdir ./oncoanalyser --compress none --container-system singularity --container-library quay.io -l docker.io -l community.wave.seqera.io/library/ --container-cache-utilisation amend --download-configuration yes


                                          ,--./,-.
          ___     __   __   __   ___     /,-._.--~\ 
    |\ | |__  __ /  ` /  \ |__) |__         }  {
    | \| |       \__, \__/ |  \ |___     \`-._,-`-,
                                          `._,._,'

    nf-core/tools version 3.4.0.dev0 - https://nf-co.re


WARNING  Could not find GitHub authentication token. Some API requests may fail.
INFO     Detected Nextflow version 25.04.7                                      
INFO     Saving 'nf-core/oncoanalyser'                                          
         Pipeline revision: 'nf-core-tools-download-mre'                        
         Use containers: 'singularity'                                          
         Container library: 'quay.io, docker.io,                                
         community.wave.seqera.io/library/'                                     
         Using NXF_SINGULARITY_CACHEDIR': ./singularity_container_images'       
         Output directory: 'oncoanalyser'                                       
         Include default institutional configuration: 'True'                    
INFO     Downloading workflow                                                   
INFO     Downloading centralised configs from GitHub                            
INFO     Downloading workflow files from GitHub                                 
INFO     Fetching container names for workflow revision                         
         nf-core-tools-download-mre using nextflow inspect. This might take a   
         while.                                                                 
INFO     Processing workflow revision nf-core-tools-download-mre, found 2       
         container images in total.                                             

+ tree ./singularity_container_images
./singularity_container_images
|-- biocontainers-fastqc-0.12.1--hdfd78af_0.img
|-- community-cr-prod.seqera.io-docker-registry-v2-biocontainers-fastqc-0.12.1--hdfd78af_0.img -> biocontainers-fastqc-0.12.1--hdfd78af_0.img
|-- community-cr-prod.seqera.io-docker-registry-v2-multiqc-1.29--pyhdfd78af_0.img -> multiqc-1.29--pyhdfd78af_0.img
|-- community.wave.seqera.io-library--biocontainers-fastqc-0.12.1--hdfd78af_0.img -> biocontainers-fastqc-0.12.1--hdfd78af_0.img
|-- community.wave.seqera.io-library--multiqc-1.29--pyhdfd78af_0.img -> multiqc-1.29--pyhdfd78af_0.img
|-- community.wave.seqera.io-library-biocontainers-fastqc-0.12.1--hdfd78af_0.img -> biocontainers-fastqc-0.12.1--hdfd78af_0.img
|-- community.wave.seqera.io-library-multiqc-1.29--pyhdfd78af_0.img -> multiqc-1.29--pyhdfd78af_0.img
|-- depot.galaxyproject.org-singularity-biocontainers-fastqc-0.12.1--hdfd78af_0.img -> biocontainers-fastqc-0.12.1--hdfd78af_0.img
|-- depot.galaxyproject.org-singularity-multiqc-1.29--pyhdfd78af_0.img -> multiqc-1.29--pyhdfd78af_0.img
|-- docker.io-biocontainers-fastqc-0.12.1--hdfd78af_0.img -> biocontainers-fastqc-0.12.1--hdfd78af_0.img
|-- docker.io-multiqc-1.29--pyhdfd78af_0.img -> multiqc-1.29--pyhdfd78af_0.img
|-- multiqc-1.29--pyhdfd78af_0.img
|-- quay.io-biocontainers-fastqc-0.12.1--hdfd78af_0.img -> biocontainers-fastqc-0.12.1--hdfd78af_0.img
`-- quay.io-multiqc-1.29--pyhdfd78af_0.img -> multiqc-1.29--pyhdfd78af_0.img

1 directory, 14 files
+ export NXF_SINGULARITY_CACHEDIR=./singularity_container_images
+ NXF_SINGULARITY_CACHEDIR=./singularity_container_images
+ export NXF_SINGULARITY_HOME_MOUNT=true
+ NXF_SINGULARITY_HOME_MOUNT=true
+ nextflow run ./oncoanalyser/nf_core_tools_download_mre -ansi-log false -profile test,singularity --outdir ./results
N E X T F L O W  ~  version 25.04.7
Launching `./oncoanalyser/nf_core_tools_download_mre/main.nf` [cheeky_bernard] DSL2 - revision: 7fee41ef91

------------------------------------------------------
                                        ,--./,-.
        ___     __   __   __   ___     /,-._.--~'
  |\ | |__  __ /  ` /  \ |__) |__         }  {
  | \| |       \__, \__/ |  \ |___     \`-._,-`-,
                                        `._,._,'
  nf-core/oncoanalyser 1.0.0dev
------------------------------------------------------
�[1mInput/output options�[0m
  �[0;34minput                     : �[0;32mhttps://raw.githubusercontent.com/nf-core/test-datasets/viralrecon/samplesheet/samplesheet_test_illumina_amplicon.csv�[0m
  �[0;34moutdir                    : �[0;32m./results�[0m

�[1mReference genome options�[0m
  �[0;34mgenome                    : �[0;32mR64-1-1�[0m
  �[0;34mfasta                     : �[0;32ms3://ngi-igenomes/igenomes//Saccharomyces_cerevisiae/Ensembl/R64-1-1/Sequence/WholeGenomeFasta/genome.fa�[0m

�[1mInstitutional config options�[0m
  �[0;34mcustom_config_base        : �[0;32m/oncoanalyser/nf_core_tools_download_mre/../configs/�[0m
  �[0;34mconfig_profile_name       : �[0;32mTest profile�[0m
  �[0;34mconfig_profile_description: �[0;32mMinimal test dataset to check pipeline function�[0m

�[1mGeneric options�[0m
  �[0;34mtrace_report_suffix       : �[0;32m2025-09-09_01-48-45�[0m

�[1mCore Nextflow options�[0m
  �[0;34mrunName                   : �[0;32mcheeky_bernard�[0m
  �[0;34mcontainerEngine           : �[0;32msingularity�[0m
  �[0;34mlaunchDir                 : �[0;32m/�[0m
  �[0;34mworkDir                   : �[0;32m/work�[0m
  �[0;34mprojectDir                : �[0;32m/oncoanalyser/nf_core_tools_download_mre�[0m
  �[0;34muserName                  : �[0;32mroot�[0m
  �[0;34mprofile                   : �[0;32mtest,singularity�[0m
  �[0;34mconfigFiles               : �[0;32m/oncoanalyser/nf_core_tools_download_mre/nextflow.config�[0m

!! Only displaying parameters that differ from the pipeline defaults !!
-�[2m----------------------------------------------------�[0m-
* The nf-core framework
    https://doi.org/10.1038/s41587-020-0439-x

* Software dependencies
    https://github.com/nf-core/oncoanalyser/blob/master/CITATIONS.md

[d2/a78e99] Submitted process > NFCORE_ONCOANALYSER:ONCOANALYSER:FASTQC (SAMPLE1_PE)
[9a/7e1f8d] Submitted process > NFCORE_ONCOANALYSER:ONCOANALYSER:FASTQC (SAMPLE3_SE)
[21/a854f3] Submitted process > NFCORE_ONCOANALYSER:ONCOANALYSER:FASTQC (SAMPLE2_PE)
[b1/dacda0] Submitted process > NFCORE_ONCOANALYSER:ONCOANALYSER:MULTIQC
-�[0;35m[nf-core/oncoanalyser]�[0;32m Pipeline completed successfully�[0m-

run.sh (click to show)

#!/usr/bin/env bash
set -exuo pipefail

# Download images
mkdir -p ./singularity_container_images

export NXF_SINGULARITY_CACHEDIR=./singularity_container_images

nf-core pipelines download nf-core/oncoanalyser \
  --revision nf-core-tools-download-mre \
  --outdir ./oncoanalyser \
  --compress "none" \
  --container-system 'singularity' \
  --container-library "quay.io" -l "docker.io" -l "community.wave.seqera.io/library/" \
  --container-cache-utilisation 'amend' \
  --download-configuration 'yes'


# List downloaded images
tree ${NXF_SINGULARITY_CACHEDIR};


# Run pipeline with downloaded images
export NXF_SINGULARITY_CACHEDIR=./singularity_container_images
export NXF_SINGULARITY_HOME_MOUNT=true

nextflow run ./oncoanalyser/nf_core_tools_download_mre \
  -ansi-log false \
  -profile test,singularity \
  --outdir ./results

Dockerfile (click to show)

FROM docker.io/ubuntu:24.04

RUN \
  apt update && \
  apt install -y curl systemd

RUN \
  curl -L "https://repo.anaconda.com/miniconda/Miniconda3-py310_25.5.1-0-Linux-$(uname -m).sh" > ~/miniconda.sh && \
  /bin/bash ~/miniconda.sh -b -p /opt/miniconda/ && \
  /opt/miniconda/bin/conda init

ENV PATH="/opt/miniconda/bin/:${PATH}"

RUN \
  conda config --prepend channels bioconda && \
  conda config --prepend channels conda-forge && \
  conda config --system --remove channels defaults && \
  conda config --remove channels defaults

RUN \
    echo -e > ~/.condarc '\
solver: libmamba\n\
channels:\n\
    - conda-forge\n\
    - bioconda'

RUN \
    conda create -y -p /env/ \
        'python==3.13.*' \
        'apptainer==1.3.*'  \
        'nextflow==25.04.*' \
        'tree' \
        'git' \
        'gcc' \
        'gxx' \
        'clang' \
        'clangxx'

RUN \
    conda clean -yaf

ENV PATH="/env/bin/:${PATH}"
ENV LD_LIBRARY_PATH="/env/lib/:${LD_LIBRARY_PATH}"

# Install Python dependencies
# NOTE(SW): pinned to latest commit on `issues/3742-download` of `JulianFlesch/nf-core-tools` at time of writting
RUN \
  pip install git+https://github.com/JulianFlesch/nf-core-tools.git@1f1c346

COPY run.sh /usr/local/bin/

[JulianFlesch]

Glad it works 👍

Can confirm that it now works for me, too in codespaces.

[JulianFlesch]

Closing since the current download workflow uses the dev branch of tools and this was merged to dev in #3749