On CESGA FT3 I have these three places (there are others) to use for installing the software and collecting traces/logs.
$HOME, ~ is my user dir on a partition on LustreFS, with a small quota.
$STORE is my user dir on a partition on NFS, not super fast, but an OK size/inodes quota.
$LUSTRE is my user dir on a partition on LustreFS, faster and with a larger quota for size/inodes.
My Python is installed in $STORE. This is where I keep micromamba environments. I have python 3.13 there. (During classes we had to use $STORE for some exercises with Python/AI/torch/etc., that's why it's this way.)
StreamFlow is installed in $LUSTRE. The conformance tests are launched from $LUSTRE, but the venv is using the Python from NFS. I'm using the node-singularity branch #1054 .
This is the script to reproduce.
It's using the archive created automatically during the execution, so the hash will vary randomly.
It fails with this error.
long log, with --debug
(cwl-conformance-venv) (python-3.13) [curso348@login210-18 streamflow]$ streamflow run --debug repro.yml
2026-05-05 22:12:56.610 INFO Processing workflow 50271fae-f1ae-4920-9112-f85c03b4c62d
2026-05-05 22:12:56.610 INFO Building workflow execution plan
2026-05-05 22:12:56.667 DEBUG Translating Workflow /
2026-05-05 22:12:56.668 DEBUG Translating WorkflowStep /rev
2026-05-05 22:12:56.678 DEBUG Translating CommandLineTool /rev
2026-05-05 22:12:56.679 DEBUG Translating WorkflowStep /sorted
2026-05-05 22:12:56.689 DEBUG Translating CommandLineTool /sorted
2026-05-05 22:12:56.728 INFO COMPLETED building of workflow execution plan
2026-05-05 22:12:56.728 INFO EXECUTING workflow 50271fae-f1ae-4920-9112-f85c03b4c62d
2026-05-05 22:12:56.799 DEBUG Step /reverse_sort-wf-default-transformer-injector received inputs ['0']
2026-05-05 22:12:56.799 DEBUG Step /input-injector received inputs ['0']
2026-05-05 22:12:56.800 DEBUG Retrieving available locations for job /reverse_sort-wf-default-transformer-injector/0 on LOCAL.
2026-05-05 22:12:56.800 DEBUG Available locations for job /reverse_sort-wf-default-transformer-injector/0 on LOCAL are ['LOCAL'].
2026-05-05 22:12:56.800 DEBUG Job /reverse_sort-wf-default-transformer-injector/0 allocated locally
2026-05-05 22:12:56.800 DEBUG Retrieving available locations for job /input-injector/0 on LOCAL.
2026-05-05 22:12:56.801 DEBUG Available locations for job /input-injector/0 on LOCAL are ['LOCAL'].
2026-05-05 22:12:56.801 DEBUG Job /input-injector/0 allocated locally
2026-05-05 22:12:56.802 DEBUG COMPLETED Step deploy/LOCAL
2026-05-05 22:12:56.802 DEBUG COMPLETED Step deploy/a0496f3e-89bd-4a8d-93b0-d0a03750ed3a
2026-05-05 22:12:56.802 DEBUG COMPLETED Step deploy/35592cae-1c01-4642-b6e0-a0216b12644c
2026-05-05 22:12:56.802 DEBUG Job /reverse_sort-wf-default-transformer-injector/0 changed status to RUNNING
2026-05-05 22:12:56.802 DEBUG Job /input-injector/0 changed status to RUNNING
2026-05-05 22:12:56.803 DEBUG COMPLETED Step /reverse_sort-wf-default-transformer-injector/schedule
2026-05-05 22:12:56.803 DEBUG COMPLETED Step /input-injector/schedule
2026-05-05 22:12:56.804 DEBUG Job /reverse_sort-wf-default-transformer-injector/0 changed status to COMPLETED
2026-05-05 22:12:56.804 DEBUG Step /reverse_sort-wf-default-transformer-injector received termination token with Status COMPLETED
2026-05-05 22:12:56.804 DEBUG Job /input-injector/0 changed status to COMPLETED
2026-05-05 22:12:56.804 DEBUG Step /reverse_sort-wf-default-transformer received inputs ['0']
2026-05-05 22:12:56.804 DEBUG Step /input-token-transformer received inputs ['0']
2026-05-05 22:12:56.804 DEBUG Processing /mnt/lustre/scratch/nlsas/home/ulc/cursos/curso348/streamflow/common-workflow-language-1c1f122f780075d910fdfdea7e15e46eef3c078d/v1.0/v1.0/whale.txt on location LOCAL/LOCAL.
2026-05-05 22:12:56.804 DEBUG Step /reverse_sort-wf-default-transformer received inputs ['0']
2026-05-05 22:12:56.805 DEBUG Step /input-injector received termination token with Status COMPLETED
2026-05-05 22:12:56.805 DEBUG Step /reverse_sort-wf-default-transformer received inputs ['0']
2026-05-05 22:12:56.805 DEBUG COMPLETED Step /reverse_sort-wf-default-transformer-injector
2026-05-05 22:12:56.805 DEBUG COMPLETED Step /input-injector
2026-05-05 22:12:56.805 DEBUG Step /rev/input-token-transformer received inputs ['0']
2026-05-05 22:12:56.805 DEBUG Processing /mnt/lustre/scratch/nlsas/home/ulc/cursos/curso348/streamflow/common-workflow-language-1c1f122f780075d910fdfdea7e15e46eef3c078d/v1.0/v1.0/whale.txt on location LOCAL/LOCAL.
2026-05-05 22:12:56.805 DEBUG Step /input-token-transformer received termination token with Status COMPLETED
2026-05-05 22:12:56.806 DEBUG Step /reverse_sort-token-transformer received inputs ['0']
2026-05-05 22:12:56.806 DEBUG Step /reverse_sort-wf-default-transformer received termination token with Status COMPLETED
2026-05-05 22:12:56.806 DEBUG COMPLETED Step /input-token-transformer
2026-05-05 22:12:56.806 DEBUG COMPLETED Step /reverse_sort-wf-default-transformer
2026-05-05 22:12:56.806 DEBUG Step /reverse_sort-token-transformer received termination token with Status COMPLETED
2026-05-05 22:12:56.806 DEBUG Step /rev/transfer/input received inputs ['0']
2026-05-05 22:12:56.806 DEBUG Step /rev/schedule received inputs ['0']
2026-05-05 22:12:56.807 DEBUG Step /rev/input-token-transformer received termination token with Status COMPLETED
2026-05-05 22:12:56.807 DEBUG Retrieving available locations for job /rev/0 on a0496f3e-89bd-4a8d-93b0-d0a03750ed3a/debian:stretch-slim.
2026-05-05 22:12:56.807 INFO DEPLOYING a0496f3e-89bd-4a8d-93b0-d0a03750ed3a
2026-05-05 22:12:56.807 DEBUG EXECUTING command singularity --version 2>&1 on LOCAL/LOCAL
2026-05-05 22:12:56.809 DEBUG COMPLETED Step /reverse_sort-token-transformer
2026-05-05 22:12:56.809 DEBUG COMPLETED Step /rev/input-token-transformer
2026-05-05 22:12:56.842 DEBUG Using singularity-ce version 4.3.3.
2026-05-05 22:12:56.842 DEBUG EXECUTING command singularity instance start --bind "/tmp/streamflow:/tmp/streamflow" --cleanenv --contain --net --network "none" docker://debian:stretch-slim 92875428-58f7-47bc-9c37-366bfe8e81e5 2>&1 on LOCAL/LOCAL
2026-05-05 22:12:59.408 DEBUG EXECUTING command singularity instance list --json 2>&1 on LOCAL/LOCAL
2026-05-05 22:12:59.454 DEBUG EXECUTING command nproc 2>&1 on SubprocessShell with command singularity exec instance://92875428-58f7-47bc-9c37-366bfe8e81e5 sh
2026-05-05 22:12:59.512 DEBUG EXECUTING command cat /proc/meminfo | grep MemTotal | awk '{print $2}' 2>&1 on SubprocessShell with command singularity exec instance://92875428-58f7-47bc-9c37-366bfe8e81e5 sh
2026-05-05 22:12:59.514 DEBUG EXECUTING command cat /proc/1/mountinfo 2>&1 on SubprocessShell with command singularity exec instance://92875428-58f7-47bc-9c37-366bfe8e81e5 sh
2026-05-05 22:12:59.517 DEBUG Container binds: {'/tmp/streamflow': '/tmp/streamflow', '/.singularity.d': '/tmp/rootfs-632486643/root/.singularity.d', '/bin': '/tmp/rootfs-632486643/root/bin', '/boot': '/tmp/rootfs-632486643/root/boot', '/dev': '/tmp/rootfs-632486643/root/dev', '/etc': '/tmp/rootfs-632486643/root/etc', '/lib': '/tmp/rootfs-632486643/root/lib', '/lib64': '/tmp/rootfs-632486643/root/lib64', '/media': '/tmp/rootfs-632486643/root/media', '/mnt': '/tmp/rootfs-632486643/root/mnt', '/opt': '/tmp/rootfs-632486643/root/opt', '/proc': '/tmp/rootfs-632486643/root/proc', '/root': '/tmp/rootfs-632486643/root/root', '/run': '/tmp/rootfs-632486643/root/run', '/sbin': '/tmp/rootfs-632486643/root/sbin', '/srv': '/tmp/rootfs-632486643/root/srv', '/sys': '/tmp/rootfs-632486643/root/sys', '/usr': '/tmp/rootfs-632486643/root/usr', '/var': '/tmp/rootfs-632486643/root/var', '/usr/share/zoneinfo/UCT': '/usr/share/zoneinfo/Europe/Madrid', '/scratch': '/scratch', '/mnt/netapp2': '/', '/mnt/netapp2/Home_FT2': '/Home_FT2', '/mnt/netapp2/Store_uni': '/Store_uni', '/mnt/netapp2/Store_emcol': '/Store_emcol', '/mnt/netapp2/Store_uscfmgmm': '/Store_uscfmgmm', '/mnt/netapp2/Store_xumetssp': '/Store_xumetssp', '/mnt/netapp2/Store_JAL': '/Store_JAL', '/mnt/netapp2/Store_JAL/ngs/DRAGEN': '/Store_JAL/ngs/DRAGEN', '/mnt/netapp2/Store_ulcfcefb': '/Store_ulcfcefb', '/mnt/netapp2/Store_uvifaaff': '/Store_uvifaaff', '/mnt/netapp2/uscmg_aplic': '/uscmg_aplic', '/mnt/netapp2/Store_cseygfla': '/Store_cseygfla', '/mnt/netapp2/Store_EMM1': '/Store_EMM1', '/mnt/netapp2/Store_cseyglmf': '/Store_cseyglmf', '/mnt/netapp2/Store_jcfernandez': '/Store_jcfernandez', '/mnt/netapp2/Store_pouro': '/Store_pouro', '/mnt/netapp2/Store_uscge': '/Store_uscge', '/mnt/netapp2/Store_csgpomdm': '/Store_csgpomdm', '/mnt/netapp2/Store_NPQ': '/Store_NPQ', '/mnt/netapp2/Store_uscfajcs': '/Store_uscfajcs', '/mnt/netapp2/Store_FPGMX': '/Store_FPGMX', '/mnt/netapp2/COM_SAS': '/COM_SAS', '/mnt/netapp2/Store_Tragaldabas': '/Store_Tragaldabas', '/mnt/netapp2/Store_IEO': '/Store_IEO', '/mnt/netapp2/Fibratic': '/Fibratic', '/mnt/netapp2/Store_bihcxamo': '/Store_bihcxamo', '/mnt/netapp2/Store_uscbopdt': '/Store_uscbopdt', '/mnt/netapp2/epicovigal': '/epicovigal', '/mnt/netapp2/mobilegenomes': '/mobilegenomes', '/mnt/netapp2/Store_ulcbahnf': '/Store_ulcbahnf', '/mnt/netapp2/posadalab2': '/posadalab2', '/mnt/netapp2/Store_uscciaep': '/Store_uscciaep', '/mnt/netapp2/Store_csbbeims': '/Store_csbbeims', '/mnt/netapp2/Store_usczanpp': '/Store_usczanpp', '/mnt/netapp2/scubacancers_wp': '/scubacancers_wp', '/mnt/netapp2/Store_uvibglta': '/Store_uvibglta', '/mnt/netapp2/Store_csiaafpr': '/Store_csiaafpr', '/mnt/netapp2/Store_RADprecise': '/Store_RADprecise', '/mnt/netapp2/Store_csdvecpm': '/Store_csdvecpm', '/mnt/netapp2/bio_databases': '/bio_databases', '/mnt/netapp2/Store_csbepffa': '/Store_csbepffa', '/mnt/netapp2/Store_cscyemfs': '/Store_cscyemfs', '/mnt/netapp2/Exoma_OA': '/Exoma_OA', '/mnt/netapp2/Store_CERAINE2': '/Store_CERAINE2', '/mnt/netapp2/RES_uscmg': '/RES_uscmg', '/mnt/netapp2/Store_Componc_tmp': '/Store_Componc_tmp', '/mnt/netapp2/Store_Logan50': '/Store_Logan50', '/mnt/netapp2/Store_Omica': '/Store_Omica', '/mnt/netapp2/Store_Omica2': '/Store_Omica2', '/mnt/netapp2/Store_Pato': '/Store_Pato', '/mnt/netapp2/Store_SuPepMem': '/Store_SuPepMem', '/mnt/netapp2/Store_TGU_global': '/Store_TGU_global', '/mnt/netapp2/Store_csbbeavg': '/Store_csbbeavg', '/mnt/netapp2/Store_csbepaal': '/Store_csbepaal', '/mnt/netapp2/Store_csbeppgd': '/Store_csbeppgd', '/mnt/netapp2/Store_csbivarg': '/Store_csbivarg', '/mnt/netapp2/Store_csbye': '/Store_csbye', '/mnt/netapp2/Store_csbyegim': '/Store_csbyegim', '/mnt/netapp2/Store_csdvepfm': '/Store_csdvepfm', '/mnt/netapp2/Store_csdvemgr': '/Store_csdvemgr', '/mnt/netapp2/Store_csebdjgl': '/Store_csebdjgl', '/mnt/netapp2/Store_csfammlc': '/Store_csfammlc', '/mnt/netapp2/Store_csgcyfde': '/Store_csgcyfde', '/mnt/netapp2/Store_csgcyjgp': '/Store_csgcyjgp', '/mnt/netapp2/Store_csgfyaam': '/Store_csgfyaam', '/mnt/netapp2/Store_csgfyirt': '/Store_csgfyirt', '/mnt/netapp2/Store_mncn_peces': '/Store_mncn_peces', '/mnt/netapp2/Store_strauss': '/Store_strauss', '/mnt/netapp2/Store_otfmxavg': '/Store_otfmxavg', '/mnt/netapp2/Store_ulcctfni': '/Store_ulcctfni', '/mnt/netapp2/Store_ulciiska': '/Store_ulciiska', '/mnt/netapp2/Store_ulcfcggf': '/Store_ulcfcggf', '/mnt/netapp2/Store_ulcqfjrg': '/Store_ulcqfjrg', '/mnt/netapp2/Store_usccqjba': '/Store_usccqjba', '/mnt/netapp2/Store_usccqjmo': '/Store_usccqjmo', '/mnt/netapp2/Store_usccqpar': '/Store_usccqpar', '/mnt/netapp2/Store_uscfafsl': '/Store_uscfafsl', '/mnt/netapp2/Store_uscgepcp': '/Store_uscgepcp', '/mnt/netapp2/Store_uscmgpos': '/Store_uscmgpos', '/mnt/netapp2/Store_uscqojgg': '/Store_uscqojgg', '/mnt/netapp2/Store_uvibemva': '/Store_uvibemva', '/mnt/netapp2/Store_uvibedfg': '/Store_uvibedfg', '/mnt/netapp2/Store_uvicmepl': '/Store_uvicmepl', '/mnt/netapp2/Store_uvibgavc': '/Store_uvibgavc', '/mnt/netapp2/Store_uvibfnrc': '/Store_uvibfnrc', '/mnt/netapp2/Store_uviesmsl': '/Store_uviesmsl', '/mnt/netapp2/Store_uvifibap': '/Store_uvifibap', '/mnt/netapp2/bio_databases_ssd': '/bio_databases_ssd', '/mnt/netapp2/genomas_enfermedad': '/genomas_enfermedad', '/mnt/netapp1': '/', '/mnt/netapp1/Optcesga_FT2_RHEL7': '/Optcesga_FT2_RHEL7', '/mnt/netapp1/Optcesga_FT2_RHEL7/2025/arch/intel/icelake/easybuild/software/Core/singularity/4.3.3/var/singularity/mnt/session/rootfs': '/tmp/rootfs-632486643/root', '/mnt/netapp1/Optcesga_FT2_RHEL7/2025/arch/intel/icelake/easybuild/software/Core/singularity/4.3.3/var/singularity/mnt/session/underlay/.singularity.d': '/tmp/rootfs-632486643/root/.singularity.d', '/mnt/netapp1/Optcesga_FT2_RHEL7/2025/arch/intel/icelake/easybuild/software/Core/singularity/4.3.3/var/singularity/mnt/session/underlay/bin': '/tmp/rootfs-632486643/root/bin', '/mnt/netapp1/Optcesga_FT2_RHEL7/2025/arch/intel/icelake/easybuild/software/Core/singularity/4.3.3/var/singularity/mnt/session/underlay/boot': '/tmp/rootfs-632486643/root/boot', '/mnt/netapp1/Optcesga_FT2_RHEL7/2025/arch/intel/icelake/easybuild/software/Core/singularity/4.3.3/var/singularity/mnt/session/underlay/dev': '/tmp/rootfs-632486643/root/dev', '/mnt/netapp1/Optcesga_FT2_RHEL7/2025/arch/intel/icelake/easybuild/software/Core/singularity/4.3.3/var/singularity/mnt/session/underlay/etc': '/tmp/rootfs-632486643/root/etc', '/mnt/netapp1/Optcesga_FT2_RHEL7/2025/arch/intel/icelake/easybuild/software/Core/singularity/4.3.3/var/singularity/mnt/session/underlay/lib': '/tmp/rootfs-632486643/root/lib', '/mnt/netapp1/Optcesga_FT2_RHEL7/2025/arch/intel/icelake/easybuild/software/Core/singularity/4.3.3/var/singularity/mnt/session/underlay/lib64': '/tmp/rootfs-632486643/root/lib64', '/mnt/netapp1/Optcesga_FT2_RHEL7/2025/arch/intel/icelake/easybuild/software/Core/singularity/4.3.3/var/singularity/mnt/session/underlay/media': '/tmp/rootfs-632486643/root/media', '/mnt/netapp1/Optcesga_FT2_RHEL7/2025/arch/intel/icelake/easybuild/software/Core/singularity/4.3.3/var/singularity/mnt/session/underlay/mnt': '/tmp/rootfs-632486643/root/mnt', '/mnt/netapp1/Optcesga_FT2_RHEL7/2025/arch/intel/icelake/easybuild/software/Core/singularity/4.3.3/var/singularity/mnt/session/underlay/opt': '/tmp/rootfs-632486643/root/opt', '/mnt/netapp1/Optcesga_FT2_RHEL7/2025/arch/intel/icelake/easybuild/software/Core/singularity/4.3.3/var/singularity/mnt/session/underlay/proc': '/tmp/rootfs-632486643/root/proc', '/mnt/netapp1/Optcesga_FT2_RHEL7/2025/arch/intel/icelake/easybuild/software/Core/singularity/4.3.3/var/singularity/mnt/session/underlay/root': '/tmp/rootfs-632486643/root/root', '/mnt/netapp1/Optcesga_FT2_RHEL7/2025/arch/intel/icelake/easybuild/software/Core/singularity/4.3.3/var/singularity/mnt/session/underlay/run': '/tmp/rootfs-632486643/root/run', '/mnt/netapp1/Optcesga_FT2_RHEL7/2025/arch/intel/icelake/easybuild/software/Core/singularity/4.3.3/var/singularity/mnt/session/underlay/sbin': '/tmp/rootfs-632486643/root/sbin', '/mnt/netapp1/Optcesga_FT2_RHEL7/2025/arch/intel/icelake/easybuild/software/Core/singularity/4.3.3/var/singularity/mnt/session/underlay/srv': '/tmp/rootfs-632486643/root/srv', '/mnt/netapp1/Optcesga_FT2_RHEL7/2025/arch/intel/icelake/easybuild/software/Core/singularity/4.3.3/var/singularity/mnt/session/underlay/sys': '/tmp/rootfs-632486643/root/sys', '/mnt/netapp1/Optcesga_FT2_RHEL7/2025/arch/intel/icelake/easybuild/software/Core/singularity/4.3.3/var/singularity/mnt/session/underlay/usr': '/tmp/rootfs-632486643/root/usr', '/mnt/netapp1/Optcesga_FT2_RHEL7/2025/arch/intel/icelake/easybuild/software/Core/singularity/4.3.3/var/singularity/mnt/session/underlay/var': '/tmp/rootfs-632486643/root/var', '/mnt/netapp1/Optcesga_FT2_RHEL7/2025/arch/intel/icelake/easybuild/software/Core/singularity/4.3.3/var/singularity/mnt/session/final/.singularity.d': '/tmp/rootfs-632486643/root/.singularity.d', '/mnt/netapp1/Optcesga_FT2_RHEL7/2025/arch/intel/icelake/easybuild/software/Core/singularity/4.3.3/var/singularity/mnt/session/final/bin': '/tmp/rootfs-632486643/root/bin', '/mnt/netapp1/Optcesga_FT2_RHEL7/2025/arch/intel/icelake/easybuild/software/Core/singularity/4.3.3/var/singularity/mnt/session/final/boot': '/tmp/rootfs-632486643/root/boot', '/mnt/netapp1/Optcesga_FT2_RHEL7/2025/arch/intel/icelake/easybuild/software/Core/singularity/4.3.3/var/singularity/mnt/session/final/dev': '/tmp/rootfs-632486643/root/dev', '/mnt/netapp1/Optcesga_FT2_RHEL7/2025/arch/intel/icelake/easybuild/software/Core/singularity/4.3.3/var/singularity/mnt/session/final/etc': '/tmp/rootfs-632486643/root/etc', '/mnt/netapp1/Optcesga_FT2_RHEL7/2025/arch/intel/icelake/easybuild/software/Core/singularity/4.3.3/var/singularity/mnt/session/final/lib': '/tmp/rootfs-632486643/root/lib', '/mnt/netapp1/Optcesga_FT2_RHEL7/2025/arch/intel/icelake/easybuild/software/Core/singularity/4.3.3/var/singularity/mnt/session/final/lib64': '/tmp/rootfs-632486643/root/lib64', '/mnt/netapp1/Optcesga_FT2_RHEL7/2025/arch/intel/icelake/easybuild/software/Core/singularity/4.3.3/var/singularity/mnt/session/final/media': '/tmp/rootfs-632486643/root/media', '/mnt/netapp1/Optcesga_FT2_RHEL7/2025/arch/intel/icelake/easybuild/software/Core/singularity/4.3.3/var/singularity/mnt/session/final/mnt': '/tmp/rootfs-632486643/root/mnt', '/mnt/netapp1/Optcesga_FT2_RHEL7/2025/arch/intel/icelake/easybuild/software/Core/singularity/4.3.3/var/singularity/mnt/session/final/opt': '/tmp/rootfs-632486643/root/opt', '/mnt/netapp1/Optcesga_FT2_RHEL7/2025/arch/intel/icelake/easybuild/software/Core/singularity/4.3.3/var/singularity/mnt/session/final/proc': '/tmp/rootfs-632486643/root/proc', '/mnt/netapp1/Optcesga_FT2_RHEL7/2025/arch/intel/icelake/easybuild/software/Core/singularity/4.3.3/var/singularity/mnt/session/final/root': '/tmp/rootfs-632486643/root/root', '/mnt/netapp1/Optcesga_FT2_RHEL7/2025/arch/intel/icelake/easybuild/software/Core/singularity/4.3.3/var/singularity/mnt/session/final/run': '/tmp/rootfs-632486643/root/run', '/mnt/netapp1/Optcesga_FT2_RHEL7/2025/arch/intel/icelake/easybuild/software/Core/singularity/4.3.3/var/singularity/mnt/session/final/sbin': '/tmp/rootfs-632486643/root/sbin', '/mnt/netapp1/Optcesga_FT2_RHEL7/2025/arch/intel/icelake/easybuild/software/Core/singularity/4.3.3/var/singularity/mnt/session/final/srv': '/tmp/rootfs-632486643/root/srv', '/mnt/netapp1/Optcesga_FT2_RHEL7/2025/arch/intel/icelake/easybuild/software/Core/singularity/4.3.3/var/singularity/mnt/session/final/sys': '/tmp/rootfs-632486643/root/sys', '/mnt/netapp1/Optcesga_FT2_RHEL7/2025/arch/intel/icelake/easybuild/software/Core/singularity/4.3.3/var/singularity/mnt/session/final/usr': '/tmp/rootfs-632486643/root/usr', '/mnt/netapp1/Optcesga_FT2_RHEL7/2025/arch/intel/icelake/easybuild/software/Core/singularity/4.3.3/var/singularity/mnt/session/final/usr/share/zoneinfo/UCT': '/usr/share/zoneinfo/Europe/Madrid', '/mnt/netapp1/Optcesga_FT2_RHEL7/2025/arch/intel/icelake/easybuild/software/Core/singularity/4.3.3/var/singularity/mnt/session/final/var': '/tmp/rootfs-632486643/root/var', '/mnt/netapp1/Optcesga_FT2_RHEL7/2025/arch/intel/icelake/easybuild/software/Core/singularity/4.3.3/var/singularity/mnt/session/final/tmp/streamflow': '/tmp/streamflow', '/mnt/netapp1/Optcesga_FT2_RHEL7/2025/arch/intel/icelake/easybuild/software/Core/singularity/4.3.3/var/singularity/mnt/session/final/scratch': '/scratch', '/mnt/netapp1/Store_Puertos': '/Store_Puertos', '/mnt/netapp1/Store_CESGA': '/Store_CESGA', '/mnt/netapp1/Store_RES': '/Store_RES', '/mnt/netapp1/Store_chumxrcg': '/Store_chumxrcg', '/mnt/netapp1/Store_CSIC': '/Store_CSIC', '/mnt/netapp1/Store_ulcticfl': '/Store_ulcticfl', '/mnt/netapp1/Store_Seaclim': '/Store_Seaclim', '/mnt/netapp1/Store_Pendeco': '/Store_Pendeco', '/mnt/netapp1/share_bioinf': '/share_bioinf', '/mnt/netapp1/Store_xumetcoc': '/Store_xumetcoc', '/mnt/netapp1/Store_Sirena': '/Store_Sirena', '/mnt/netapp1/Store_olamur': '/Store_olamur', '/mnt/netapp1/Store_uvifimdc': '/Store_uvifimdc', '/mnt/netapp1/Store_csbyejso': '/Store_csbyejso', '/mnt/netapp1/Store_ulcbagsf': '/Store_ulcbagsf', '/mnt/netapp1/Store_csbyestj': '/Store_csbyestj', '/mnt/netapp1/Store_Lagrima': '/Store_Lagrima', '/mnt/netapp1/Store_climate_ore_group': '/Store_climate_ore_group', '/mnt/netapp1/Store_ulcctaan': '/Store_ulcctaan', '/mnt/netapp1/Store_csbyerfg': '/Store_csbyerfg', '/mnt/netapp1/Store_EVOKE': '/Store_EVOKE', '/mnt/netapp1/Store_argos': '/Store_argos', '/mnt/netapp1/Store_SpongesMicrobiome': '/Store_SpongesMicrobiome', '/mnt/netapp1/Store_csbyemca': '/Store_csbyemca', '/mnt/netapp1/COM_NLSAS': '/COM_NLSAS', '/mnt/netapp1/ECOMT': '/ECOMT', '/mnt/netapp1/Home_BD': '/Home_BD', '/mnt/netapp1/Proxecto_NOS': '/Proxecto_NOS', '/mnt/netapp1/Proyectos_CESGA': '/Proyectos_CESGA', '/mnt/netapp1/RES_SuPepMem': '/RES_SuPepMem', '/mnt/netapp1/Store_CERAINE': '/Store_CERAINE', '/mnt/netapp1/Store_ChiralPro': '/Store_ChiralPro', '/mnt/netapp1/Dataspace': '/Dataspace', '/mnt/netapp1/Store_Cunqa': '/Store_Cunqa', '/mnt/netapp1/Store_Grheco_xen': '/Store_Grheco_xen', '/mnt/netapp1/Store_IMIB': '/Store_IMIB', '/mnt/netapp1/Store_Meteo2': '/Store_Meteo2', '/mnt/netapp1/Store_Lung_CTC': '/Store_Lung_CTC', '/mnt/netapp1/Store_Microbioma': '/Store_Microbioma', '/mnt/netapp1/Store_PPMCAP': '/Store_PPMCAP', '/mnt/netapp1/Store_STCshared': '/Store_STCshared', '/mnt/netapp1/Store_aphanius': '/Store_aphanius', '/mnt/netapp1/Store_biomexp': '/Store_biomexp', '/mnt/netapp1/Store_SnailEvoLab': '/Store_SnailEvoLab', '/mnt/netapp1/Store_csbepgnf': '/Store_csbepgnf', '/mnt/netapp1/Store_csbeeelp': '/Store_csbeeelp', '/mnt/netapp1/Store_csbepjpb': '/Store_csbepjpb', '/mnt/netapp1/Store_csbiybor': '/Store_csbiybor', '/mnt/netapp1/Store_csbepisa': '/Store_csbepisa', '/mnt/netapp1/Store_csbeprri': '/Store_csbeprri', '/mnt/netapp1/Store_csbyecvc': '/Store_csbyecvc', '/mnt/netapp1/Store_csbyepta': '/Store_csbyepta', '/mnt/netapp1/Store_csbyemes': '/Store_csbyemes', '/mnt/netapp1/Store_cscdiica': '/Store_cscdiica', '/mnt/netapp1/Store_cscenacs': '/Store_cscenacs', '/mnt/netapp1/Store_cseeiapc': '/Store_cseeiapc', '/mnt/netapp1/Store_csdvejpd': '/Store_csdvejpd', '/mnt/netapp1/Store_csdblagr': '/Store_csdblagr', '/mnt/netapp1/Store_cseygsgo': '/Store_cseygsgo', '/mnt/netapp1/Store_cslordgl': '/Store_cslordgl', '/mnt/netapp1/Store_cspdalcs': '/Store_cspdalcs', '/mnt/netapp1/Store_cstoasgh': '/Store_cstoasgh', '/mnt/netapp1/Store_csgcyjgp3': '/Store_csgcyjgp3', '/mnt/netapp1/Store_eDNA': '/Store_eDNA', '/mnt/netapp1/Store_dlab': '/Store_dlab', '/mnt/netapp1/Store_microbiota': '/Store_microbiota', '/mnt/netapp1/Store_next_data': '/Store_next_data', '/mnt/netapp1/Store_iadt': '/Store_iadt', '/mnt/netapp1/Store_gradiant': '/Store_gradiant', '/mnt/netapp1/Store_otomtrll': '/Store_otomtrll', '/mnt/netapp1/Store_rcc_bl': '/Store_rcc_bl', '/mnt/netapp1/Store_othcxtch': '/Store_othcxtch', '/mnt/netapp1/Store_otuicacc': '/Store_otuicacc', '/mnt/netapp1/Store_ulcbmavs': '/Store_ulcbmavs', '/mnt/netapp1/Store_ulciaaaa': '/Store_ulciaaaa', '/mnt/netapp1/Store_seresco': '/Store_seresco', '/mnt/netapp1/Store_usccimyg': '/Store_usccimyg', '/mnt/netapp1/Store_ulcesepg': '/Store_ulcesepg', '/mnt/netapp1/Store_usccisss': '/Store_usccisss', '/mnt/netapp1/Store_usccqaan': '/Store_usccqaan', '/mnt/netapp1/Store_uscgbmrf': '/Store_uscgbmrf', '/mnt/netapp1/Store_uscafvpc': '/Store_uscafvpc', '/mnt/netapp1/Store_uscgejfl': '/Store_uscgejfl', '/mnt/netapp1/Store_uscpcsbt': '/Store_uscpcsbt', '/mnt/netapp1/Store_uscqorgf': '/Store_uscqorgf', '/mnt/netapp1/Store_uscrdjrr': '/Store_uscrdjrr', '/mnt/netapp1/Store_uvibcspc': '/Store_uvibcspc', '/mnt/netapp1/Store_uscmlmdl': '/Store_uscmlmdl', '/mnt/netapp1/Store_uvicmvzo': '/Store_uvicmvzo', '/mnt/netapp1/Store_uvicmlgv': '/Store_uvicmlgv', '/mnt/netapp1/Store_uvifijcn': '/Store_uvifijcn', '/mnt/netapp1/bio_databases_ssd2': '/bio_databases_ssd2', '/mnt/netapp1/genomes_shared': '/genomes_shared', '/mnt/netapp1/fs_hadoop': '/fs_hadoop', '/mnt/netapp1/genomes_tmp': '/genomes_tmp', '/mnt/netapp1/posadalab': '/posadalab', '/mnt/netapp1/interaction': '/interaction', '/mnt/netapp1/scubacancers': '/scubacancers', '/mnt/netapp1/quantumcesga': '/quantumcesga', '/mnt/netapp1/share_mugg': '/share_mugg', '/mnt/netapp1/translational_oncology': '/translational_oncology', '/mnt/lustre/hsm': '/mnt/lustre/hsm/', '/mnt/lustre/scratch': '/mnt/lustre/scratch/'}
2026-05-05 22:12:59.517 DEBUG EXECUTING command df -aT | tail -n +2 | awk 'NF == 1 {device = $1; getline; $0 = device $0} {print $7, $2, $5}' 2>&1 on SubprocessShell with command singularity exec instance://92875428-58f7-47bc-9c37-366bfe8e81e5 sh
2026-05-05 22:12:59.582 INFO COMPLETED deployment of a0496f3e-89bd-4a8d-93b0-d0a03750ed3a
2026-05-05 22:12:59.584 DEBUG Available locations for job /rev/0 on a0496f3e-89bd-4a8d-93b0-d0a03750ed3a/debian:stretch-slim are ['92875428-58f7-47bc-9c37-366bfe8e81e5'].
2026-05-05 22:12:59.584 DEBUG Job /rev/0 allocated on location a0496f3e-89bd-4a8d-93b0-d0a03750ed3a/92875428-58f7-47bc-9c37-366bfe8e81e5
2026-05-05 22:12:59.584 DEBUG EXECUTING command test -e /tmp/streamflow/ec158ac0-36c0-49ac-bffb-a7b84485d60a && readlink -f /tmp/streamflow/ec158ac0-36c0-49ac-bffb-a7b84485d60a 2>&1 on SubprocessShell with command singularity exec instance://92875428-58f7-47bc-9c37-366bfe8e81e5 sh
2026-05-05 22:12:59.585 DEBUG EXECUTING command test -e /tmp/streamflow/3167764c-2a4e-4364-b1fd-8d9eb8b52064 && readlink -f /tmp/streamflow/3167764c-2a4e-4364-b1fd-8d9eb8b52064 2>&1 on SubprocessShell with command singularity exec instance://92875428-58f7-47bc-9c37-366bfe8e81e5 sh
2026-05-05 22:12:59.585 DEBUG EXECUTING command test -e /tmp/streamflow/a419bebf-7a55-4701-b683-7b18064e7595 && readlink -f /tmp/streamflow/a419bebf-7a55-4701-b683-7b18064e7595 2>&1 on SubprocessShell with command singularity exec instance://92875428-58f7-47bc-9c37-366bfe8e81e5 sh
2026-05-05 22:12:59.585 DEBUG EXECUTING command mkdir -m 777 -p /tmp/streamflow/ec158ac0-36c0-49ac-bffb-a7b84485d60a 2>&1 on SubprocessShell with command singularity exec instance://92875428-58f7-47bc-9c37-366bfe8e81e5 sh
2026-05-05 22:12:59.586 DEBUG EXECUTING command mkdir -m 777 -p /tmp/streamflow/3167764c-2a4e-4364-b1fd-8d9eb8b52064 2>&1 on SubprocessShell with command singularity exec instance://92875428-58f7-47bc-9c37-366bfe8e81e5 sh
2026-05-05 22:12:59.587 DEBUG EXECUTING command mkdir -m 777 -p /tmp/streamflow/a419bebf-7a55-4701-b683-7b18064e7595 2>&1 on SubprocessShell with command singularity exec instance://92875428-58f7-47bc-9c37-366bfe8e81e5 sh
2026-05-05 22:12:59.588 DEBUG EXECUTING command test -e /tmp/streamflow/ec158ac0-36c0-49ac-bffb-a7b84485d60a && readlink -f /tmp/streamflow/ec158ac0-36c0-49ac-bffb-a7b84485d60a 2>&1 on SubprocessShell with command singularity exec instance://92875428-58f7-47bc-9c37-366bfe8e81e5 sh
2026-05-05 22:12:59.588 DEBUG EXECUTING command test -e /tmp/streamflow/3167764c-2a4e-4364-b1fd-8d9eb8b52064 && readlink -f /tmp/streamflow/3167764c-2a4e-4364-b1fd-8d9eb8b52064 2>&1 on SubprocessShell with command singularity exec instance://92875428-58f7-47bc-9c37-366bfe8e81e5 sh
2026-05-05 22:12:59.589 DEBUG EXECUTING command test -e /tmp/streamflow/a419bebf-7a55-4701-b683-7b18064e7595 && readlink -f /tmp/streamflow/a419bebf-7a55-4701-b683-7b18064e7595 2>&1 on SubprocessShell with command singularity exec instance://92875428-58f7-47bc-9c37-366bfe8e81e5 sh
2026-05-05 22:12:59.589 DEBUG EXECUTING command test -e /tmp/streamflow/ec158ac0-36c0-49ac-bffb-a7b84485d60a && readlink -f /tmp/streamflow/ec158ac0-36c0-49ac-bffb-a7b84485d60a 2>&1 on SubprocessShell with command singularity exec instance://92875428-58f7-47bc-9c37-366bfe8e81e5 sh
2026-05-05 22:12:59.590 DEBUG EXECUTING command test -e /tmp/streamflow/3167764c-2a4e-4364-b1fd-8d9eb8b52064 && readlink -f /tmp/streamflow/3167764c-2a4e-4364-b1fd-8d9eb8b52064 2>&1 on SubprocessShell with command singularity exec instance://92875428-58f7-47bc-9c37-366bfe8e81e5 sh
2026-05-05 22:12:59.591 DEBUG EXECUTING command test -e /tmp/streamflow/a419bebf-7a55-4701-b683-7b18064e7595 && readlink -f /tmp/streamflow/a419bebf-7a55-4701-b683-7b18064e7595 2>&1 on SubprocessShell with command singularity exec instance://92875428-58f7-47bc-9c37-366bfe8e81e5 sh
2026-05-05 22:12:59.593 DEBUG Step /rev/schedule received termination token with Status COMPLETED
2026-05-05 22:12:59.594 DEBUG EXECUTING command test -d "/tmp/streamflow/ec158ac0-36c0-49ac-bffb-a7b84485d60a/a5d79ed3-1a94-4fc2-a6dd-2c76c7abd798/whale.txt" 2>&1 on SubprocessShell with command singularity exec instance://92875428-58f7-47bc-9c37-366bfe8e81e5 sh
2026-05-05 22:12:59.594 DEBUG COMPLETED Step /rev/schedule
2026-05-05 22:12:59.594 DEBUG Extracted effective locations 92875428-58f7-47bc-9c37-366bfe8e81e5 from initial locations 92875428-58f7-47bc-9c37-366bfe8e81e5
2026-05-05 22:12:59.594 INFO COPYING from /mnt/netapp2/mnt/lustre/scratch/nlsas/home/ulc/cursos/curso348/streamflow/common-workflow-language-1c1f122f780075d910fdfdea7e15e46eef3c078d/v1.0/v1.0/whale.txt to /tmp/streamflow/ec158ac0-36c0-49ac-bffb-a7b84485d60a/a5d79ed3-1a94-4fc2-a6dd-2c76c7abd798/whale.txt on location a0496f3e-89bd-4a8d-93b0-d0a03750ed3a/92875428-58f7-47bc-9c37-366bfe8e81e5
2026-05-05 22:12:59.594 DEBUG EXECUTING command ln -snf /mnt/netapp2/mnt/lustre/scratch/nlsas/home/ulc/cursos/curso348/streamflow/common-workflow-language-1c1f122f780075d910fdfdea7e15e46eef3c078d/v1.0/v1.0/whale.txt /tmp/streamflow/ec158ac0-36c0-49ac-bffb-a7b84485d60a/a5d79ed3-1a94-4fc2-a6dd-2c76c7abd798/whale.txt 2>&1 on SubprocessShell with command singularity exec instance://92875428-58f7-47bc-9c37-366bfe8e81e5 sh
2026-05-05 22:12:59.595 INFO COMPLETED copy from /mnt/netapp2/mnt/lustre/scratch/nlsas/home/ulc/cursos/curso348/streamflow/common-workflow-language-1c1f122f780075d910fdfdea7e15e46eef3c078d/v1.0/v1.0/whale.txt to /tmp/streamflow/ec158ac0-36c0-49ac-bffb-a7b84485d60a/a5d79ed3-1a94-4fc2-a6dd-2c76c7abd798/whale.txt on location a0496f3e-89bd-4a8d-93b0-d0a03750ed3a/92875428-58f7-47bc-9c37-366bfe8e81e5
2026-05-05 22:12:59.595 DEBUG EXECUTING command test -L /tmp/streamflow/ec158ac0-36c0-49ac-bffb-a7b84485d60a/a5d79ed3-1a94-4fc2-a6dd-2c76c7abd798/whale.txt 2>&1 on SubprocessShell with command singularity exec instance://92875428-58f7-47bc-9c37-366bfe8e81e5 sh
2026-05-05 22:12:59.595 DEBUG Step /rev received inputs ['0']
2026-05-05 22:12:59.595 DEBUG Step /rev/transfer/input received termination token with Status COMPLETED
2026-05-05 22:12:59.596 DEBUG COMPLETED Step /rev/transfer/input
2026-05-05 22:12:59.596 DEBUG Job /rev/0 started
2026-05-05 22:12:59.596 DEBUG Job /rev/0 changed status to RUNNING
2026-05-05 22:12:59.596 DEBUG Job /rev/0 inputs: {
"input": {
"basename": "whale.txt",
"checksum": "sha1$327fc7aedf4f6b69a42a7c8b808dc5a7aff61376",
"class": "File",
"dirname": "/tmp/streamflow/ec158ac0-36c0-49ac-bffb-a7b84485d60a/a5d79ed3-1a94-4fc2-a6dd-2c76c7abd798",
"location": "file:///tmp/streamflow/ec158ac0-36c0-49ac-bffb-a7b84485d60a/a5d79ed3-1a94-4fc2-a6dd-2c76c7abd798/whale.txt",
"nameext": ".txt",
"nameroot": "whale",
"path": "/tmp/streamflow/ec158ac0-36c0-49ac-bffb-a7b84485d60a/a5d79ed3-1a94-4fc2-a6dd-2c76c7abd798/whale.txt",
"size": 1111
}
}
2026-05-05 22:12:59.596 INFO EXECUTING step /rev (job /rev/0) on location a0496f3e-89bd-4a8d-93b0-d0a03750ed3a/92875428-58f7-47bc-9c37-366bfe8e81e5 into directory /tmp/streamflow/3167764c-2a4e-4364-b1fd-8d9eb8b52064:
rev
/tmp/streamflow/ec158ac0-36c0-49ac-bffb-a7b84485d60a/a5d79ed3-1a94-4fc2-a6dd-2c76c7abd798/whale.txt
2026-05-05 22:12:59.596 DEBUG Step /rev received termination token with Status COMPLETED
2026-05-05 22:12:59.596 DEBUG EXECUTING command cd /tmp/streamflow/3167764c-2a4e-4364-b1fd-8d9eb8b52064 && export HOME="/tmp/streamflow/3167764c-2a4e-4364-b1fd-8d9eb8b52064" && export TMPDIR="/tmp/streamflow/a419bebf-7a55-4701-b683-7b18064e7595" && rev /tmp/streamflow/ec158ac0-36c0-49ac-bffb-a7b84485d60a/a5d79ed3-1a94-4fc2-a6dd-2c76c7abd798/whale.txt > output.txt 2>&1 on a0496f3e-89bd-4a8d-93b0-d0a03750ed3a/92875428-58f7-47bc-9c37-366bfe8e81e5 for job /rev/0
2026-05-05 22:12:59.596 DEBUG EXECUTING command singularity exec --cleanenv --contain instance://92875428-58f7-47bc-9c37-366bfe8e81e5 sh -c 'cd /tmp/streamflow/3167764c-2a4e-4364-b1fd-8d9eb8b52064 && export HOME="/tmp/streamflow/3167764c-2a4e-4364-b1fd-8d9eb8b52064" && export TMPDIR="/tmp/streamflow/a419bebf-7a55-4701-b683-7b18064e7595" && rev /tmp/streamflow/ec158ac0-36c0-49ac-bffb-a7b84485d60a/a5d79ed3-1a94-4fc2-a6dd-2c76c7abd798/whale.txt > output.txt 2>&1' 2>&1 on LOCAL/LOCAL for job /rev/0
2026-05-05 22:12:59.661 ERROR FAILED Job /rev/0 with error:
Traceback (most recent call last):
File "/mnt/lustre/scratch/nlsas/home/ulc/cursos/curso348/streamflow/cwl-conformance-venv/lib/python3.13/site-packages/streamflow/core/recovery.py", line 42, in wrapper
await func(*args, **kwargs)
File "/mnt/lustre/scratch/nlsas/home/ulc/cursos/curso348/streamflow/cwl-conformance-venv/lib/python3.13/site-packages/streamflow/workflow/step.py", line 743, in _execute_command
raise WorkflowExecutionException(
f"FAILED Job {job.name} with error:\n\t{command_output.value}"
)
streamflow.core.exception.WorkflowExecutionException: FAILED Job /rev/0 with error:
2026-05-05 22:12:59.662 WARNING Job /rev/0 failure can not be recovered. Failure manager is not enabled.
2026-05-05 22:12:59.662 ERROR FAILED Job /rev/0 with error:
2026-05-05 22:12:59.663 DEBUG Job /rev/0 changed status to FAILED
2026-05-05 22:12:59.663 DEBUG EXECUTING command find -L "/tmp/streamflow/3167764c-2a4e-4364-b1fd-8d9eb8b52064" -type f -exec ls -ln {} + | awk 'BEGIN {sum=0} {sum+=$5} END {print sum}'; 2>&1 on SubprocessShell with command singularity exec instance://92875428-58f7-47bc-9c37-366bfe8e81e5 sh
2026-05-05 22:12:59.665 DEBUG EXECUTING command find -L "/tmp/streamflow/a419bebf-7a55-4701-b683-7b18064e7595" -type f -exec ls -ln {} + | awk 'BEGIN {sum=0} {sum+=$5} END {print sum}'; 2>&1 on SubprocessShell with command singularity exec instance://92875428-58f7-47bc-9c37-366bfe8e81e5 sh
2026-05-05 22:12:59.666 DEBUG FAILED Job /rev/0 terminated
2026-05-05 22:12:59.667 DEBUG Step /sorted/reverse-token-transformer received termination token with Status FAILED
2026-05-05 22:12:59.667 DEBUG Step /sorted/input-token-transformer received termination token with Status FAILED
2026-05-05 22:12:59.667 DEBUG Step /sorted/transfer/reverse received termination token with Status FAILED
2026-05-05 22:12:59.667 DEBUG Step /sorted/transfer/input received termination token with Status FAILED
2026-05-05 22:12:59.667 DEBUG Step /sorted/schedule received termination token with Status FAILED
2026-05-05 22:12:59.667 DEBUG Step /sorted received termination token with Status FAILED
2026-05-05 22:12:59.667 DEBUG Step /output-collector-transformer received termination token with Status FAILED
2026-05-05 22:12:59.667 DEBUG Step /output-collector received termination token with Status FAILED
2026-05-05 22:12:59.667 DEBUG Step /output-collector/schedule received termination token with Status FAILED
2026-05-05 22:12:59.668 INFO FAILED Step /rev
2026-05-05 22:12:59.668 DEBUG FAILED Step /sorted/reverse-token-transformer
2026-05-05 22:12:59.668 DEBUG FAILED Step /sorted/input-token-transformer
2026-05-05 22:12:59.668 DEBUG FAILED Step /sorted/schedule
2026-05-05 22:12:59.668 DEBUG FAILED Step /sorted/transfer/reverse
2026-05-05 22:12:59.668 DEBUG FAILED Step /sorted/transfer/input
2026-05-05 22:12:59.668 INFO FAILED Step /sorted
2026-05-05 22:12:59.668 DEBUG FAILED Step /output-collector-transformer
2026-05-05 22:12:59.668 DEBUG FAILED Step /output-collector/schedule
2026-05-05 22:12:59.668 DEBUG FAILED Step /output-collector
2026-05-05 22:12:59.668 INFO UNDEPLOYING a0496f3e-89bd-4a8d-93b0-d0a03750ed3a
2026-05-05 22:12:59.669 DEBUG EXECUTING command singularity instance stop 92875428-58f7-47bc-9c37-366bfe8e81e5 2>&1 on LOCAL/LOCAL
2026-05-05 22:12:59.670 INFO UNDEPLOYING 35592cae-1c01-4642-b6e0-a0216b12644c
2026-05-05 22:12:59.670 INFO COMPLETED undeployment of 35592cae-1c01-4642-b6e0-a0216b12644c
2026-05-05 22:12:59.804 INFO COMPLETED undeployment of a0496f3e-89bd-4a8d-93b0-d0a03750ed3a
2026-05-05 22:12:59.817 ERROR FAILED Workflow execution
Traceback (most recent call last):
File "/mnt/lustre/scratch/nlsas/home/ulc/cursos/curso348/streamflow/cwl-conformance-venv/lib/python3.13/site-packages/streamflow/main.py", line 281, in main
asyncio.run(_async_run(parsed_args))
~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/netapp2/Store_uni/home/ulc/cursos/curso348/micromamba/envs/python-3.13/lib/python3.13/asyncio/runners.py", line 195, in run
return runner.run(main)
~~~~~~~~~~^^^^^^
File "/mnt/netapp2/Store_uni/home/ulc/cursos/curso348/micromamba/envs/python-3.13/lib/python3.13/asyncio/runners.py", line 118, in run
return self._loop.run_until_complete(task)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
File "/mnt/netapp2/Store_uni/home/ulc/cursos/curso348/micromamba/envs/python-3.13/lib/python3.13/asyncio/base_events.py", line 725, in run_until_complete
return future.result()
~~~~~~~~~~~~~^^
File "/mnt/lustre/scratch/nlsas/home/ulc/cursos/curso348/streamflow/cwl-conformance-venv/lib/python3.13/site-packages/streamflow/main.py", line 187, in _async_run
await asyncio.gather(*workflow_tasks)
File "/mnt/lustre/scratch/nlsas/home/ulc/cursos/curso348/streamflow/cwl-conformance-venv/lib/python3.13/site-packages/streamflow/cwl/main.py", line 93, in main
output_tokens = await executor.run()
^^^^^^^^^^^^^^^^^^^^
File "/mnt/lustre/scratch/nlsas/home/ulc/cursos/curso348/streamflow/cwl-conformance-venv/lib/python3.13/site-packages/streamflow/workflow/executor.py", line 175, in run
raise WorkflowExecutionException("FAILED Workflow execution")
streamflow.core.exception.WorkflowExecutionException: FAILED Workflow execution
Which is not very helpful.
So my guess now is that StreamFlow (or a part of it, some lib) is resolving the file based on the location of Python, and not the location where StreamFlow was launched. Causing StreamFlow to fail in environments where for some reason Python & StreamFlow reside on different partitions/filesystems... I think!
Hi,
On CESGA FT3 I have these three places (there are others) to use for installing the software and collecting traces/logs.
$HOME, ~ is my user dir on a partition on LustreFS, with a small quota.
$STORE is my user dir on a partition on NFS, not super fast, but an OK size/inodes quota.
$LUSTRE is my user dir on a partition on LustreFS, faster and with a larger quota for size/inodes.
My Python is installed in $STORE. This is where I keep micromamba environments. I have python 3.13 there. (During classes we had to use $STORE for some exercises with Python/AI/torch/etc., that's why it's this way.)
StreamFlow is installed in $LUSTRE. The conformance tests are launched from $LUSTRE, but the venv is using the Python from NFS. I'm using the node-singularity branch #1054 .
This is the script to reproduce.
It's using the archive created automatically during the execution, so the hash will vary randomly.
It fails with this error.
long log, with --debug
(cwl-conformance-venv) (python-3.13) [curso348@login210-18 streamflow]$ streamflow run --debug repro.yml
2026-05-05 22:12:56.610 INFO Processing workflow 50271fae-f1ae-4920-9112-f85c03b4c62d
2026-05-05 22:12:56.610 INFO Building workflow execution plan
2026-05-05 22:12:56.667 DEBUG Translating Workflow /
2026-05-05 22:12:56.668 DEBUG Translating WorkflowStep /rev
2026-05-05 22:12:56.678 DEBUG Translating CommandLineTool /rev
2026-05-05 22:12:56.679 DEBUG Translating WorkflowStep /sorted
2026-05-05 22:12:56.689 DEBUG Translating CommandLineTool /sorted
2026-05-05 22:12:56.728 INFO COMPLETED building of workflow execution plan
2026-05-05 22:12:56.728 INFO EXECUTING workflow 50271fae-f1ae-4920-9112-f85c03b4c62d
2026-05-05 22:12:56.799 DEBUG Step /reverse_sort-wf-default-transformer-injector received inputs ['0']
2026-05-05 22:12:56.799 DEBUG Step /input-injector received inputs ['0']
2026-05-05 22:12:56.800 DEBUG Retrieving available locations for job /reverse_sort-wf-default-transformer-injector/0 on LOCAL.
2026-05-05 22:12:56.800 DEBUG Available locations for job /reverse_sort-wf-default-transformer-injector/0 on LOCAL are ['LOCAL'].
2026-05-05 22:12:56.800 DEBUG Job /reverse_sort-wf-default-transformer-injector/0 allocated locally
2026-05-05 22:12:56.800 DEBUG Retrieving available locations for job /input-injector/0 on LOCAL.
2026-05-05 22:12:56.801 DEBUG Available locations for job /input-injector/0 on LOCAL are ['LOCAL'].
2026-05-05 22:12:56.801 DEBUG Job /input-injector/0 allocated locally
2026-05-05 22:12:56.802 DEBUG COMPLETED Step deploy/LOCAL
2026-05-05 22:12:56.802 DEBUG COMPLETED Step deploy/a0496f3e-89bd-4a8d-93b0-d0a03750ed3a
2026-05-05 22:12:56.802 DEBUG COMPLETED Step deploy/35592cae-1c01-4642-b6e0-a0216b12644c
2026-05-05 22:12:56.802 DEBUG Job /reverse_sort-wf-default-transformer-injector/0 changed status to RUNNING
2026-05-05 22:12:56.802 DEBUG Job /input-injector/0 changed status to RUNNING
2026-05-05 22:12:56.803 DEBUG COMPLETED Step /reverse_sort-wf-default-transformer-injector/schedule
2026-05-05 22:12:56.803 DEBUG COMPLETED Step /input-injector/schedule
2026-05-05 22:12:56.804 DEBUG Job /reverse_sort-wf-default-transformer-injector/0 changed status to COMPLETED
2026-05-05 22:12:56.804 DEBUG Step /reverse_sort-wf-default-transformer-injector received termination token with Status COMPLETED
2026-05-05 22:12:56.804 DEBUG Job /input-injector/0 changed status to COMPLETED
2026-05-05 22:12:56.804 DEBUG Step /reverse_sort-wf-default-transformer received inputs ['0']
2026-05-05 22:12:56.804 DEBUG Step /input-token-transformer received inputs ['0']
2026-05-05 22:12:56.804 DEBUG Processing /mnt/lustre/scratch/nlsas/home/ulc/cursos/curso348/streamflow/common-workflow-language-1c1f122f780075d910fdfdea7e15e46eef3c078d/v1.0/v1.0/whale.txt on location LOCAL/LOCAL.
2026-05-05 22:12:56.804 DEBUG Step /reverse_sort-wf-default-transformer received inputs ['0']
2026-05-05 22:12:56.805 DEBUG Step /input-injector received termination token with Status COMPLETED
2026-05-05 22:12:56.805 DEBUG Step /reverse_sort-wf-default-transformer received inputs ['0']
2026-05-05 22:12:56.805 DEBUG COMPLETED Step /reverse_sort-wf-default-transformer-injector
2026-05-05 22:12:56.805 DEBUG COMPLETED Step /input-injector
2026-05-05 22:12:56.805 DEBUG Step /rev/input-token-transformer received inputs ['0']
2026-05-05 22:12:56.805 DEBUG Processing /mnt/lustre/scratch/nlsas/home/ulc/cursos/curso348/streamflow/common-workflow-language-1c1f122f780075d910fdfdea7e15e46eef3c078d/v1.0/v1.0/whale.txt on location LOCAL/LOCAL.
2026-05-05 22:12:56.805 DEBUG Step /input-token-transformer received termination token with Status COMPLETED
2026-05-05 22:12:56.806 DEBUG Step /reverse_sort-token-transformer received inputs ['0']
2026-05-05 22:12:56.806 DEBUG Step /reverse_sort-wf-default-transformer received termination token with Status COMPLETED
2026-05-05 22:12:56.806 DEBUG COMPLETED Step /input-token-transformer
2026-05-05 22:12:56.806 DEBUG COMPLETED Step /reverse_sort-wf-default-transformer
2026-05-05 22:12:56.806 DEBUG Step /reverse_sort-token-transformer received termination token with Status COMPLETED
2026-05-05 22:12:56.806 DEBUG Step /rev/transfer/input received inputs ['0']
2026-05-05 22:12:56.806 DEBUG Step /rev/schedule received inputs ['0']
2026-05-05 22:12:56.807 DEBUG Step /rev/input-token-transformer received termination token with Status COMPLETED
2026-05-05 22:12:56.807 DEBUG Retrieving available locations for job /rev/0 on a0496f3e-89bd-4a8d-93b0-d0a03750ed3a/debian:stretch-slim.
2026-05-05 22:12:56.807 INFO DEPLOYING a0496f3e-89bd-4a8d-93b0-d0a03750ed3a
2026-05-05 22:12:56.807 DEBUG EXECUTING command singularity --version 2>&1 on LOCAL/LOCAL
2026-05-05 22:12:56.809 DEBUG COMPLETED Step /reverse_sort-token-transformer
2026-05-05 22:12:56.809 DEBUG COMPLETED Step /rev/input-token-transformer
2026-05-05 22:12:56.842 DEBUG Using singularity-ce version 4.3.3.
2026-05-05 22:12:56.842 DEBUG EXECUTING command singularity instance start --bind "/tmp/streamflow:/tmp/streamflow" --cleanenv --contain --net --network "none" docker://debian:stretch-slim 92875428-58f7-47bc-9c37-366bfe8e81e5 2>&1 on LOCAL/LOCAL
2026-05-05 22:12:59.408 DEBUG EXECUTING command singularity instance list --json 2>&1 on LOCAL/LOCAL
2026-05-05 22:12:59.454 DEBUG EXECUTING command nproc 2>&1 on SubprocessShell with command
singularity exec instance://92875428-58f7-47bc-9c37-366bfe8e81e5 sh2026-05-05 22:12:59.512 DEBUG EXECUTING command cat /proc/meminfo | grep MemTotal | awk '{print $2}' 2>&1 on SubprocessShell with command
singularity exec instance://92875428-58f7-47bc-9c37-366bfe8e81e5 sh2026-05-05 22:12:59.514 DEBUG EXECUTING command cat /proc/1/mountinfo 2>&1 on SubprocessShell with command
singularity exec instance://92875428-58f7-47bc-9c37-366bfe8e81e5 sh2026-05-05 22:12:59.517 DEBUG Container binds: {'/tmp/streamflow': '/tmp/streamflow', '/.singularity.d': '/tmp/rootfs-632486643/root/.singularity.d', '/bin': '/tmp/rootfs-632486643/root/bin', '/boot': '/tmp/rootfs-632486643/root/boot', '/dev': '/tmp/rootfs-632486643/root/dev', '/etc': '/tmp/rootfs-632486643/root/etc', '/lib': '/tmp/rootfs-632486643/root/lib', '/lib64': '/tmp/rootfs-632486643/root/lib64', '/media': '/tmp/rootfs-632486643/root/media', '/mnt': '/tmp/rootfs-632486643/root/mnt', '/opt': '/tmp/rootfs-632486643/root/opt', '/proc': '/tmp/rootfs-632486643/root/proc', '/root': '/tmp/rootfs-632486643/root/root', '/run': '/tmp/rootfs-632486643/root/run', '/sbin': '/tmp/rootfs-632486643/root/sbin', '/srv': '/tmp/rootfs-632486643/root/srv', '/sys': '/tmp/rootfs-632486643/root/sys', '/usr': '/tmp/rootfs-632486643/root/usr', '/var': '/tmp/rootfs-632486643/root/var', '/usr/share/zoneinfo/UCT': '/usr/share/zoneinfo/Europe/Madrid', '/scratch': '/scratch', '/mnt/netapp2': '/', '/mnt/netapp2/Home_FT2': '/Home_FT2', '/mnt/netapp2/Store_uni': '/Store_uni', '/mnt/netapp2/Store_emcol': '/Store_emcol', '/mnt/netapp2/Store_uscfmgmm': '/Store_uscfmgmm', '/mnt/netapp2/Store_xumetssp': '/Store_xumetssp', '/mnt/netapp2/Store_JAL': '/Store_JAL', '/mnt/netapp2/Store_JAL/ngs/DRAGEN': '/Store_JAL/ngs/DRAGEN', '/mnt/netapp2/Store_ulcfcefb': '/Store_ulcfcefb', '/mnt/netapp2/Store_uvifaaff': '/Store_uvifaaff', '/mnt/netapp2/uscmg_aplic': '/uscmg_aplic', '/mnt/netapp2/Store_cseygfla': '/Store_cseygfla', '/mnt/netapp2/Store_EMM1': '/Store_EMM1', '/mnt/netapp2/Store_cseyglmf': '/Store_cseyglmf', '/mnt/netapp2/Store_jcfernandez': '/Store_jcfernandez', '/mnt/netapp2/Store_pouro': '/Store_pouro', '/mnt/netapp2/Store_uscge': '/Store_uscge', '/mnt/netapp2/Store_csgpomdm': '/Store_csgpomdm', '/mnt/netapp2/Store_NPQ': '/Store_NPQ', '/mnt/netapp2/Store_uscfajcs': '/Store_uscfajcs', '/mnt/netapp2/Store_FPGMX': '/Store_FPGMX', '/mnt/netapp2/COM_SAS': '/COM_SAS', '/mnt/netapp2/Store_Tragaldabas': '/Store_Tragaldabas', '/mnt/netapp2/Store_IEO': '/Store_IEO', '/mnt/netapp2/Fibratic': '/Fibratic', '/mnt/netapp2/Store_bihcxamo': '/Store_bihcxamo', '/mnt/netapp2/Store_uscbopdt': '/Store_uscbopdt', '/mnt/netapp2/epicovigal': '/epicovigal', '/mnt/netapp2/mobilegenomes': '/mobilegenomes', '/mnt/netapp2/Store_ulcbahnf': '/Store_ulcbahnf', '/mnt/netapp2/posadalab2': '/posadalab2', '/mnt/netapp2/Store_uscciaep': '/Store_uscciaep', '/mnt/netapp2/Store_csbbeims': '/Store_csbbeims', '/mnt/netapp2/Store_usczanpp': '/Store_usczanpp', '/mnt/netapp2/scubacancers_wp': '/scubacancers_wp', '/mnt/netapp2/Store_uvibglta': '/Store_uvibglta', '/mnt/netapp2/Store_csiaafpr': '/Store_csiaafpr', '/mnt/netapp2/Store_RADprecise': '/Store_RADprecise', '/mnt/netapp2/Store_csdvecpm': '/Store_csdvecpm', '/mnt/netapp2/bio_databases': '/bio_databases', '/mnt/netapp2/Store_csbepffa': '/Store_csbepffa', '/mnt/netapp2/Store_cscyemfs': '/Store_cscyemfs', '/mnt/netapp2/Exoma_OA': '/Exoma_OA', '/mnt/netapp2/Store_CERAINE2': '/Store_CERAINE2', '/mnt/netapp2/RES_uscmg': '/RES_uscmg', '/mnt/netapp2/Store_Componc_tmp': '/Store_Componc_tmp', '/mnt/netapp2/Store_Logan50': '/Store_Logan50', '/mnt/netapp2/Store_Omica': '/Store_Omica', '/mnt/netapp2/Store_Omica2': '/Store_Omica2', '/mnt/netapp2/Store_Pato': '/Store_Pato', '/mnt/netapp2/Store_SuPepMem': '/Store_SuPepMem', '/mnt/netapp2/Store_TGU_global': '/Store_TGU_global', '/mnt/netapp2/Store_csbbeavg': '/Store_csbbeavg', '/mnt/netapp2/Store_csbepaal': '/Store_csbepaal', '/mnt/netapp2/Store_csbeppgd': '/Store_csbeppgd', '/mnt/netapp2/Store_csbivarg': '/Store_csbivarg', '/mnt/netapp2/Store_csbye': '/Store_csbye', '/mnt/netapp2/Store_csbyegim': '/Store_csbyegim', '/mnt/netapp2/Store_csdvepfm': '/Store_csdvepfm', '/mnt/netapp2/Store_csdvemgr': '/Store_csdvemgr', '/mnt/netapp2/Store_csebdjgl': '/Store_csebdjgl', '/mnt/netapp2/Store_csfammlc': '/Store_csfammlc', '/mnt/netapp2/Store_csgcyfde': '/Store_csgcyfde', '/mnt/netapp2/Store_csgcyjgp': '/Store_csgcyjgp', '/mnt/netapp2/Store_csgfyaam': '/Store_csgfyaam', '/mnt/netapp2/Store_csgfyirt': '/Store_csgfyirt', '/mnt/netapp2/Store_mncn_peces': '/Store_mncn_peces', '/mnt/netapp2/Store_strauss': '/Store_strauss', '/mnt/netapp2/Store_otfmxavg': '/Store_otfmxavg', '/mnt/netapp2/Store_ulcctfni': '/Store_ulcctfni', '/mnt/netapp2/Store_ulciiska': '/Store_ulciiska', '/mnt/netapp2/Store_ulcfcggf': '/Store_ulcfcggf', '/mnt/netapp2/Store_ulcqfjrg': '/Store_ulcqfjrg', '/mnt/netapp2/Store_usccqjba': '/Store_usccqjba', '/mnt/netapp2/Store_usccqjmo': '/Store_usccqjmo', '/mnt/netapp2/Store_usccqpar': '/Store_usccqpar', '/mnt/netapp2/Store_uscfafsl': '/Store_uscfafsl', '/mnt/netapp2/Store_uscgepcp': '/Store_uscgepcp', '/mnt/netapp2/Store_uscmgpos': '/Store_uscmgpos', '/mnt/netapp2/Store_uscqojgg': '/Store_uscqojgg', '/mnt/netapp2/Store_uvibemva': '/Store_uvibemva', '/mnt/netapp2/Store_uvibedfg': '/Store_uvibedfg', '/mnt/netapp2/Store_uvicmepl': '/Store_uvicmepl', '/mnt/netapp2/Store_uvibgavc': '/Store_uvibgavc', '/mnt/netapp2/Store_uvibfnrc': '/Store_uvibfnrc', '/mnt/netapp2/Store_uviesmsl': '/Store_uviesmsl', '/mnt/netapp2/Store_uvifibap': '/Store_uvifibap', '/mnt/netapp2/bio_databases_ssd': '/bio_databases_ssd', '/mnt/netapp2/genomas_enfermedad': '/genomas_enfermedad', '/mnt/netapp1': '/', '/mnt/netapp1/Optcesga_FT2_RHEL7': '/Optcesga_FT2_RHEL7', '/mnt/netapp1/Optcesga_FT2_RHEL7/2025/arch/intel/icelake/easybuild/software/Core/singularity/4.3.3/var/singularity/mnt/session/rootfs': '/tmp/rootfs-632486643/root', '/mnt/netapp1/Optcesga_FT2_RHEL7/2025/arch/intel/icelake/easybuild/software/Core/singularity/4.3.3/var/singularity/mnt/session/underlay/.singularity.d': '/tmp/rootfs-632486643/root/.singularity.d', '/mnt/netapp1/Optcesga_FT2_RHEL7/2025/arch/intel/icelake/easybuild/software/Core/singularity/4.3.3/var/singularity/mnt/session/underlay/bin': '/tmp/rootfs-632486643/root/bin', '/mnt/netapp1/Optcesga_FT2_RHEL7/2025/arch/intel/icelake/easybuild/software/Core/singularity/4.3.3/var/singularity/mnt/session/underlay/boot': '/tmp/rootfs-632486643/root/boot', '/mnt/netapp1/Optcesga_FT2_RHEL7/2025/arch/intel/icelake/easybuild/software/Core/singularity/4.3.3/var/singularity/mnt/session/underlay/dev': '/tmp/rootfs-632486643/root/dev', '/mnt/netapp1/Optcesga_FT2_RHEL7/2025/arch/intel/icelake/easybuild/software/Core/singularity/4.3.3/var/singularity/mnt/session/underlay/etc': '/tmp/rootfs-632486643/root/etc', '/mnt/netapp1/Optcesga_FT2_RHEL7/2025/arch/intel/icelake/easybuild/software/Core/singularity/4.3.3/var/singularity/mnt/session/underlay/lib': '/tmp/rootfs-632486643/root/lib', '/mnt/netapp1/Optcesga_FT2_RHEL7/2025/arch/intel/icelake/easybuild/software/Core/singularity/4.3.3/var/singularity/mnt/session/underlay/lib64': '/tmp/rootfs-632486643/root/lib64', '/mnt/netapp1/Optcesga_FT2_RHEL7/2025/arch/intel/icelake/easybuild/software/Core/singularity/4.3.3/var/singularity/mnt/session/underlay/media': '/tmp/rootfs-632486643/root/media', '/mnt/netapp1/Optcesga_FT2_RHEL7/2025/arch/intel/icelake/easybuild/software/Core/singularity/4.3.3/var/singularity/mnt/session/underlay/mnt': '/tmp/rootfs-632486643/root/mnt', '/mnt/netapp1/Optcesga_FT2_RHEL7/2025/arch/intel/icelake/easybuild/software/Core/singularity/4.3.3/var/singularity/mnt/session/underlay/opt': '/tmp/rootfs-632486643/root/opt', '/mnt/netapp1/Optcesga_FT2_RHEL7/2025/arch/intel/icelake/easybuild/software/Core/singularity/4.3.3/var/singularity/mnt/session/underlay/proc': '/tmp/rootfs-632486643/root/proc', '/mnt/netapp1/Optcesga_FT2_RHEL7/2025/arch/intel/icelake/easybuild/software/Core/singularity/4.3.3/var/singularity/mnt/session/underlay/root': '/tmp/rootfs-632486643/root/root', '/mnt/netapp1/Optcesga_FT2_RHEL7/2025/arch/intel/icelake/easybuild/software/Core/singularity/4.3.3/var/singularity/mnt/session/underlay/run': '/tmp/rootfs-632486643/root/run', '/mnt/netapp1/Optcesga_FT2_RHEL7/2025/arch/intel/icelake/easybuild/software/Core/singularity/4.3.3/var/singularity/mnt/session/underlay/sbin': '/tmp/rootfs-632486643/root/sbin', '/mnt/netapp1/Optcesga_FT2_RHEL7/2025/arch/intel/icelake/easybuild/software/Core/singularity/4.3.3/var/singularity/mnt/session/underlay/srv': '/tmp/rootfs-632486643/root/srv', '/mnt/netapp1/Optcesga_FT2_RHEL7/2025/arch/intel/icelake/easybuild/software/Core/singularity/4.3.3/var/singularity/mnt/session/underlay/sys': '/tmp/rootfs-632486643/root/sys', '/mnt/netapp1/Optcesga_FT2_RHEL7/2025/arch/intel/icelake/easybuild/software/Core/singularity/4.3.3/var/singularity/mnt/session/underlay/usr': '/tmp/rootfs-632486643/root/usr', '/mnt/netapp1/Optcesga_FT2_RHEL7/2025/arch/intel/icelake/easybuild/software/Core/singularity/4.3.3/var/singularity/mnt/session/underlay/var': '/tmp/rootfs-632486643/root/var', '/mnt/netapp1/Optcesga_FT2_RHEL7/2025/arch/intel/icelake/easybuild/software/Core/singularity/4.3.3/var/singularity/mnt/session/final/.singularity.d': '/tmp/rootfs-632486643/root/.singularity.d', '/mnt/netapp1/Optcesga_FT2_RHEL7/2025/arch/intel/icelake/easybuild/software/Core/singularity/4.3.3/var/singularity/mnt/session/final/bin': '/tmp/rootfs-632486643/root/bin', '/mnt/netapp1/Optcesga_FT2_RHEL7/2025/arch/intel/icelake/easybuild/software/Core/singularity/4.3.3/var/singularity/mnt/session/final/boot': '/tmp/rootfs-632486643/root/boot', '/mnt/netapp1/Optcesga_FT2_RHEL7/2025/arch/intel/icelake/easybuild/software/Core/singularity/4.3.3/var/singularity/mnt/session/final/dev': '/tmp/rootfs-632486643/root/dev', '/mnt/netapp1/Optcesga_FT2_RHEL7/2025/arch/intel/icelake/easybuild/software/Core/singularity/4.3.3/var/singularity/mnt/session/final/etc': '/tmp/rootfs-632486643/root/etc', '/mnt/netapp1/Optcesga_FT2_RHEL7/2025/arch/intel/icelake/easybuild/software/Core/singularity/4.3.3/var/singularity/mnt/session/final/lib': '/tmp/rootfs-632486643/root/lib', '/mnt/netapp1/Optcesga_FT2_RHEL7/2025/arch/intel/icelake/easybuild/software/Core/singularity/4.3.3/var/singularity/mnt/session/final/lib64': '/tmp/rootfs-632486643/root/lib64', '/mnt/netapp1/Optcesga_FT2_RHEL7/2025/arch/intel/icelake/easybuild/software/Core/singularity/4.3.3/var/singularity/mnt/session/final/media': '/tmp/rootfs-632486643/root/media', '/mnt/netapp1/Optcesga_FT2_RHEL7/2025/arch/intel/icelake/easybuild/software/Core/singularity/4.3.3/var/singularity/mnt/session/final/mnt': '/tmp/rootfs-632486643/root/mnt', '/mnt/netapp1/Optcesga_FT2_RHEL7/2025/arch/intel/icelake/easybuild/software/Core/singularity/4.3.3/var/singularity/mnt/session/final/opt': '/tmp/rootfs-632486643/root/opt', '/mnt/netapp1/Optcesga_FT2_RHEL7/2025/arch/intel/icelake/easybuild/software/Core/singularity/4.3.3/var/singularity/mnt/session/final/proc': '/tmp/rootfs-632486643/root/proc', '/mnt/netapp1/Optcesga_FT2_RHEL7/2025/arch/intel/icelake/easybuild/software/Core/singularity/4.3.3/var/singularity/mnt/session/final/root': '/tmp/rootfs-632486643/root/root', '/mnt/netapp1/Optcesga_FT2_RHEL7/2025/arch/intel/icelake/easybuild/software/Core/singularity/4.3.3/var/singularity/mnt/session/final/run': '/tmp/rootfs-632486643/root/run', '/mnt/netapp1/Optcesga_FT2_RHEL7/2025/arch/intel/icelake/easybuild/software/Core/singularity/4.3.3/var/singularity/mnt/session/final/sbin': '/tmp/rootfs-632486643/root/sbin', '/mnt/netapp1/Optcesga_FT2_RHEL7/2025/arch/intel/icelake/easybuild/software/Core/singularity/4.3.3/var/singularity/mnt/session/final/srv': '/tmp/rootfs-632486643/root/srv', '/mnt/netapp1/Optcesga_FT2_RHEL7/2025/arch/intel/icelake/easybuild/software/Core/singularity/4.3.3/var/singularity/mnt/session/final/sys': '/tmp/rootfs-632486643/root/sys', '/mnt/netapp1/Optcesga_FT2_RHEL7/2025/arch/intel/icelake/easybuild/software/Core/singularity/4.3.3/var/singularity/mnt/session/final/usr': '/tmp/rootfs-632486643/root/usr', '/mnt/netapp1/Optcesga_FT2_RHEL7/2025/arch/intel/icelake/easybuild/software/Core/singularity/4.3.3/var/singularity/mnt/session/final/usr/share/zoneinfo/UCT': '/usr/share/zoneinfo/Europe/Madrid', '/mnt/netapp1/Optcesga_FT2_RHEL7/2025/arch/intel/icelake/easybuild/software/Core/singularity/4.3.3/var/singularity/mnt/session/final/var': '/tmp/rootfs-632486643/root/var', '/mnt/netapp1/Optcesga_FT2_RHEL7/2025/arch/intel/icelake/easybuild/software/Core/singularity/4.3.3/var/singularity/mnt/session/final/tmp/streamflow': '/tmp/streamflow', '/mnt/netapp1/Optcesga_FT2_RHEL7/2025/arch/intel/icelake/easybuild/software/Core/singularity/4.3.3/var/singularity/mnt/session/final/scratch': '/scratch', '/mnt/netapp1/Store_Puertos': '/Store_Puertos', '/mnt/netapp1/Store_CESGA': '/Store_CESGA', '/mnt/netapp1/Store_RES': '/Store_RES', '/mnt/netapp1/Store_chumxrcg': '/Store_chumxrcg', '/mnt/netapp1/Store_CSIC': '/Store_CSIC', '/mnt/netapp1/Store_ulcticfl': '/Store_ulcticfl', '/mnt/netapp1/Store_Seaclim': '/Store_Seaclim', '/mnt/netapp1/Store_Pendeco': '/Store_Pendeco', '/mnt/netapp1/share_bioinf': '/share_bioinf', '/mnt/netapp1/Store_xumetcoc': '/Store_xumetcoc', '/mnt/netapp1/Store_Sirena': '/Store_Sirena', '/mnt/netapp1/Store_olamur': '/Store_olamur', '/mnt/netapp1/Store_uvifimdc': '/Store_uvifimdc', '/mnt/netapp1/Store_csbyejso': '/Store_csbyejso', '/mnt/netapp1/Store_ulcbagsf': '/Store_ulcbagsf', '/mnt/netapp1/Store_csbyestj': '/Store_csbyestj', '/mnt/netapp1/Store_Lagrima': '/Store_Lagrima', '/mnt/netapp1/Store_climate_ore_group': '/Store_climate_ore_group', '/mnt/netapp1/Store_ulcctaan': '/Store_ulcctaan', '/mnt/netapp1/Store_csbyerfg': '/Store_csbyerfg', '/mnt/netapp1/Store_EVOKE': '/Store_EVOKE', '/mnt/netapp1/Store_argos': '/Store_argos', '/mnt/netapp1/Store_SpongesMicrobiome': '/Store_SpongesMicrobiome', '/mnt/netapp1/Store_csbyemca': '/Store_csbyemca', '/mnt/netapp1/COM_NLSAS': '/COM_NLSAS', '/mnt/netapp1/ECOMT': '/ECOMT', '/mnt/netapp1/Home_BD': '/Home_BD', '/mnt/netapp1/Proxecto_NOS': '/Proxecto_NOS', '/mnt/netapp1/Proyectos_CESGA': '/Proyectos_CESGA', '/mnt/netapp1/RES_SuPepMem': '/RES_SuPepMem', '/mnt/netapp1/Store_CERAINE': '/Store_CERAINE', '/mnt/netapp1/Store_ChiralPro': '/Store_ChiralPro', '/mnt/netapp1/Dataspace': '/Dataspace', '/mnt/netapp1/Store_Cunqa': '/Store_Cunqa', '/mnt/netapp1/Store_Grheco_xen': '/Store_Grheco_xen', '/mnt/netapp1/Store_IMIB': '/Store_IMIB', '/mnt/netapp1/Store_Meteo2': '/Store_Meteo2', '/mnt/netapp1/Store_Lung_CTC': '/Store_Lung_CTC', '/mnt/netapp1/Store_Microbioma': '/Store_Microbioma', '/mnt/netapp1/Store_PPMCAP': '/Store_PPMCAP', '/mnt/netapp1/Store_STCshared': '/Store_STCshared', '/mnt/netapp1/Store_aphanius': '/Store_aphanius', '/mnt/netapp1/Store_biomexp': '/Store_biomexp', '/mnt/netapp1/Store_SnailEvoLab': '/Store_SnailEvoLab', '/mnt/netapp1/Store_csbepgnf': '/Store_csbepgnf', '/mnt/netapp1/Store_csbeeelp': '/Store_csbeeelp', '/mnt/netapp1/Store_csbepjpb': '/Store_csbepjpb', '/mnt/netapp1/Store_csbiybor': '/Store_csbiybor', '/mnt/netapp1/Store_csbepisa': '/Store_csbepisa', '/mnt/netapp1/Store_csbeprri': '/Store_csbeprri', '/mnt/netapp1/Store_csbyecvc': '/Store_csbyecvc', '/mnt/netapp1/Store_csbyepta': '/Store_csbyepta', '/mnt/netapp1/Store_csbyemes': '/Store_csbyemes', '/mnt/netapp1/Store_cscdiica': '/Store_cscdiica', '/mnt/netapp1/Store_cscenacs': '/Store_cscenacs', '/mnt/netapp1/Store_cseeiapc': '/Store_cseeiapc', '/mnt/netapp1/Store_csdvejpd': '/Store_csdvejpd', '/mnt/netapp1/Store_csdblagr': '/Store_csdblagr', '/mnt/netapp1/Store_cseygsgo': '/Store_cseygsgo', '/mnt/netapp1/Store_cslordgl': '/Store_cslordgl', '/mnt/netapp1/Store_cspdalcs': '/Store_cspdalcs', '/mnt/netapp1/Store_cstoasgh': '/Store_cstoasgh', '/mnt/netapp1/Store_csgcyjgp3': '/Store_csgcyjgp3', '/mnt/netapp1/Store_eDNA': '/Store_eDNA', '/mnt/netapp1/Store_dlab': '/Store_dlab', '/mnt/netapp1/Store_microbiota': '/Store_microbiota', '/mnt/netapp1/Store_next_data': '/Store_next_data', '/mnt/netapp1/Store_iadt': '/Store_iadt', '/mnt/netapp1/Store_gradiant': '/Store_gradiant', '/mnt/netapp1/Store_otomtrll': '/Store_otomtrll', '/mnt/netapp1/Store_rcc_bl': '/Store_rcc_bl', '/mnt/netapp1/Store_othcxtch': '/Store_othcxtch', '/mnt/netapp1/Store_otuicacc': '/Store_otuicacc', '/mnt/netapp1/Store_ulcbmavs': '/Store_ulcbmavs', '/mnt/netapp1/Store_ulciaaaa': '/Store_ulciaaaa', '/mnt/netapp1/Store_seresco': '/Store_seresco', '/mnt/netapp1/Store_usccimyg': '/Store_usccimyg', '/mnt/netapp1/Store_ulcesepg': '/Store_ulcesepg', '/mnt/netapp1/Store_usccisss': '/Store_usccisss', '/mnt/netapp1/Store_usccqaan': '/Store_usccqaan', '/mnt/netapp1/Store_uscgbmrf': '/Store_uscgbmrf', '/mnt/netapp1/Store_uscafvpc': '/Store_uscafvpc', '/mnt/netapp1/Store_uscgejfl': '/Store_uscgejfl', '/mnt/netapp1/Store_uscpcsbt': '/Store_uscpcsbt', '/mnt/netapp1/Store_uscqorgf': '/Store_uscqorgf', '/mnt/netapp1/Store_uscrdjrr': '/Store_uscrdjrr', '/mnt/netapp1/Store_uvibcspc': '/Store_uvibcspc', '/mnt/netapp1/Store_uscmlmdl': '/Store_uscmlmdl', '/mnt/netapp1/Store_uvicmvzo': '/Store_uvicmvzo', '/mnt/netapp1/Store_uvicmlgv': '/Store_uvicmlgv', '/mnt/netapp1/Store_uvifijcn': '/Store_uvifijcn', '/mnt/netapp1/bio_databases_ssd2': '/bio_databases_ssd2', '/mnt/netapp1/genomes_shared': '/genomes_shared', '/mnt/netapp1/fs_hadoop': '/fs_hadoop', '/mnt/netapp1/genomes_tmp': '/genomes_tmp', '/mnt/netapp1/posadalab': '/posadalab', '/mnt/netapp1/interaction': '/interaction', '/mnt/netapp1/scubacancers': '/scubacancers', '/mnt/netapp1/quantumcesga': '/quantumcesga', '/mnt/netapp1/share_mugg': '/share_mugg', '/mnt/netapp1/translational_oncology': '/translational_oncology', '/mnt/lustre/hsm': '/mnt/lustre/hsm/', '/mnt/lustre/scratch': '/mnt/lustre/scratch/'}
2026-05-05 22:12:59.517 DEBUG EXECUTING command df -aT | tail -n +2 | awk 'NF == 1 {device = $1; getline; $0 = device $0} {print $7, $2, $5}' 2>&1 on SubprocessShell with command
singularity exec instance://92875428-58f7-47bc-9c37-366bfe8e81e5 sh2026-05-05 22:12:59.582 INFO COMPLETED deployment of a0496f3e-89bd-4a8d-93b0-d0a03750ed3a
2026-05-05 22:12:59.584 DEBUG Available locations for job /rev/0 on a0496f3e-89bd-4a8d-93b0-d0a03750ed3a/debian:stretch-slim are ['92875428-58f7-47bc-9c37-366bfe8e81e5'].
2026-05-05 22:12:59.584 DEBUG Job /rev/0 allocated on location a0496f3e-89bd-4a8d-93b0-d0a03750ed3a/92875428-58f7-47bc-9c37-366bfe8e81e5
2026-05-05 22:12:59.584 DEBUG EXECUTING command test -e /tmp/streamflow/ec158ac0-36c0-49ac-bffb-a7b84485d60a && readlink -f /tmp/streamflow/ec158ac0-36c0-49ac-bffb-a7b84485d60a 2>&1 on SubprocessShell with command
singularity exec instance://92875428-58f7-47bc-9c37-366bfe8e81e5 sh2026-05-05 22:12:59.585 DEBUG EXECUTING command test -e /tmp/streamflow/3167764c-2a4e-4364-b1fd-8d9eb8b52064 && readlink -f /tmp/streamflow/3167764c-2a4e-4364-b1fd-8d9eb8b52064 2>&1 on SubprocessShell with command
singularity exec instance://92875428-58f7-47bc-9c37-366bfe8e81e5 sh2026-05-05 22:12:59.585 DEBUG EXECUTING command test -e /tmp/streamflow/a419bebf-7a55-4701-b683-7b18064e7595 && readlink -f /tmp/streamflow/a419bebf-7a55-4701-b683-7b18064e7595 2>&1 on SubprocessShell with command
singularity exec instance://92875428-58f7-47bc-9c37-366bfe8e81e5 sh2026-05-05 22:12:59.585 DEBUG EXECUTING command mkdir -m 777 -p /tmp/streamflow/ec158ac0-36c0-49ac-bffb-a7b84485d60a 2>&1 on SubprocessShell with command
singularity exec instance://92875428-58f7-47bc-9c37-366bfe8e81e5 sh2026-05-05 22:12:59.586 DEBUG EXECUTING command mkdir -m 777 -p /tmp/streamflow/3167764c-2a4e-4364-b1fd-8d9eb8b52064 2>&1 on SubprocessShell with command
singularity exec instance://92875428-58f7-47bc-9c37-366bfe8e81e5 sh2026-05-05 22:12:59.587 DEBUG EXECUTING command mkdir -m 777 -p /tmp/streamflow/a419bebf-7a55-4701-b683-7b18064e7595 2>&1 on SubprocessShell with command
singularity exec instance://92875428-58f7-47bc-9c37-366bfe8e81e5 sh2026-05-05 22:12:59.588 DEBUG EXECUTING command test -e /tmp/streamflow/ec158ac0-36c0-49ac-bffb-a7b84485d60a && readlink -f /tmp/streamflow/ec158ac0-36c0-49ac-bffb-a7b84485d60a 2>&1 on SubprocessShell with command
singularity exec instance://92875428-58f7-47bc-9c37-366bfe8e81e5 sh2026-05-05 22:12:59.588 DEBUG EXECUTING command test -e /tmp/streamflow/3167764c-2a4e-4364-b1fd-8d9eb8b52064 && readlink -f /tmp/streamflow/3167764c-2a4e-4364-b1fd-8d9eb8b52064 2>&1 on SubprocessShell with command
singularity exec instance://92875428-58f7-47bc-9c37-366bfe8e81e5 sh2026-05-05 22:12:59.589 DEBUG EXECUTING command test -e /tmp/streamflow/a419bebf-7a55-4701-b683-7b18064e7595 && readlink -f /tmp/streamflow/a419bebf-7a55-4701-b683-7b18064e7595 2>&1 on SubprocessShell with command
singularity exec instance://92875428-58f7-47bc-9c37-366bfe8e81e5 sh2026-05-05 22:12:59.589 DEBUG EXECUTING command test -e /tmp/streamflow/ec158ac0-36c0-49ac-bffb-a7b84485d60a && readlink -f /tmp/streamflow/ec158ac0-36c0-49ac-bffb-a7b84485d60a 2>&1 on SubprocessShell with command
singularity exec instance://92875428-58f7-47bc-9c37-366bfe8e81e5 sh2026-05-05 22:12:59.590 DEBUG EXECUTING command test -e /tmp/streamflow/3167764c-2a4e-4364-b1fd-8d9eb8b52064 && readlink -f /tmp/streamflow/3167764c-2a4e-4364-b1fd-8d9eb8b52064 2>&1 on SubprocessShell with command
singularity exec instance://92875428-58f7-47bc-9c37-366bfe8e81e5 sh2026-05-05 22:12:59.591 DEBUG EXECUTING command test -e /tmp/streamflow/a419bebf-7a55-4701-b683-7b18064e7595 && readlink -f /tmp/streamflow/a419bebf-7a55-4701-b683-7b18064e7595 2>&1 on SubprocessShell with command
singularity exec instance://92875428-58f7-47bc-9c37-366bfe8e81e5 sh2026-05-05 22:12:59.593 DEBUG Step /rev/schedule received termination token with Status COMPLETED
2026-05-05 22:12:59.594 DEBUG EXECUTING command test -d "/tmp/streamflow/ec158ac0-36c0-49ac-bffb-a7b84485d60a/a5d79ed3-1a94-4fc2-a6dd-2c76c7abd798/whale.txt" 2>&1 on SubprocessShell with command
singularity exec instance://92875428-58f7-47bc-9c37-366bfe8e81e5 sh2026-05-05 22:12:59.594 DEBUG COMPLETED Step /rev/schedule
2026-05-05 22:12:59.594 DEBUG Extracted effective locations 92875428-58f7-47bc-9c37-366bfe8e81e5 from initial locations 92875428-58f7-47bc-9c37-366bfe8e81e5
2026-05-05 22:12:59.594 INFO COPYING from /mnt/netapp2/mnt/lustre/scratch/nlsas/home/ulc/cursos/curso348/streamflow/common-workflow-language-1c1f122f780075d910fdfdea7e15e46eef3c078d/v1.0/v1.0/whale.txt to /tmp/streamflow/ec158ac0-36c0-49ac-bffb-a7b84485d60a/a5d79ed3-1a94-4fc2-a6dd-2c76c7abd798/whale.txt on location a0496f3e-89bd-4a8d-93b0-d0a03750ed3a/92875428-58f7-47bc-9c37-366bfe8e81e5
2026-05-05 22:12:59.594 DEBUG EXECUTING command ln -snf /mnt/netapp2/mnt/lustre/scratch/nlsas/home/ulc/cursos/curso348/streamflow/common-workflow-language-1c1f122f780075d910fdfdea7e15e46eef3c078d/v1.0/v1.0/whale.txt /tmp/streamflow/ec158ac0-36c0-49ac-bffb-a7b84485d60a/a5d79ed3-1a94-4fc2-a6dd-2c76c7abd798/whale.txt 2>&1 on SubprocessShell with command
singularity exec instance://92875428-58f7-47bc-9c37-366bfe8e81e5 sh2026-05-05 22:12:59.595 INFO COMPLETED copy from /mnt/netapp2/mnt/lustre/scratch/nlsas/home/ulc/cursos/curso348/streamflow/common-workflow-language-1c1f122f780075d910fdfdea7e15e46eef3c078d/v1.0/v1.0/whale.txt to /tmp/streamflow/ec158ac0-36c0-49ac-bffb-a7b84485d60a/a5d79ed3-1a94-4fc2-a6dd-2c76c7abd798/whale.txt on location a0496f3e-89bd-4a8d-93b0-d0a03750ed3a/92875428-58f7-47bc-9c37-366bfe8e81e5
2026-05-05 22:12:59.595 DEBUG EXECUTING command test -L /tmp/streamflow/ec158ac0-36c0-49ac-bffb-a7b84485d60a/a5d79ed3-1a94-4fc2-a6dd-2c76c7abd798/whale.txt 2>&1 on SubprocessShell with command
singularity exec instance://92875428-58f7-47bc-9c37-366bfe8e81e5 sh2026-05-05 22:12:59.595 DEBUG Step /rev received inputs ['0']
2026-05-05 22:12:59.595 DEBUG Step /rev/transfer/input received termination token with Status COMPLETED
2026-05-05 22:12:59.596 DEBUG COMPLETED Step /rev/transfer/input
2026-05-05 22:12:59.596 DEBUG Job /rev/0 started
2026-05-05 22:12:59.596 DEBUG Job /rev/0 changed status to RUNNING
2026-05-05 22:12:59.596 DEBUG Job /rev/0 inputs: {
"input": {
"basename": "whale.txt",
"checksum": "sha1$327fc7aedf4f6b69a42a7c8b808dc5a7aff61376",
"class": "File",
"dirname": "/tmp/streamflow/ec158ac0-36c0-49ac-bffb-a7b84485d60a/a5d79ed3-1a94-4fc2-a6dd-2c76c7abd798",
"location": "file:///tmp/streamflow/ec158ac0-36c0-49ac-bffb-a7b84485d60a/a5d79ed3-1a94-4fc2-a6dd-2c76c7abd798/whale.txt",
"nameext": ".txt",
"nameroot": "whale",
"path": "/tmp/streamflow/ec158ac0-36c0-49ac-bffb-a7b84485d60a/a5d79ed3-1a94-4fc2-a6dd-2c76c7abd798/whale.txt",
"size": 1111
}
}
2026-05-05 22:12:59.596 INFO EXECUTING step /rev (job /rev/0) on location a0496f3e-89bd-4a8d-93b0-d0a03750ed3a/92875428-58f7-47bc-9c37-366bfe8e81e5 into directory /tmp/streamflow/3167764c-2a4e-4364-b1fd-8d9eb8b52064:
rev
/tmp/streamflow/ec158ac0-36c0-49ac-bffb-a7b84485d60a/a5d79ed3-1a94-4fc2-a6dd-2c76c7abd798/whale.txt
2026-05-05 22:12:59.596 DEBUG Step /rev received termination token with Status COMPLETED
2026-05-05 22:12:59.596 DEBUG EXECUTING command cd /tmp/streamflow/3167764c-2a4e-4364-b1fd-8d9eb8b52064 && export HOME="/tmp/streamflow/3167764c-2a4e-4364-b1fd-8d9eb8b52064" && export TMPDIR="/tmp/streamflow/a419bebf-7a55-4701-b683-7b18064e7595" && rev /tmp/streamflow/ec158ac0-36c0-49ac-bffb-a7b84485d60a/a5d79ed3-1a94-4fc2-a6dd-2c76c7abd798/whale.txt > output.txt 2>&1 on a0496f3e-89bd-4a8d-93b0-d0a03750ed3a/92875428-58f7-47bc-9c37-366bfe8e81e5 for job /rev/0
2026-05-05 22:12:59.596 DEBUG EXECUTING command singularity exec --cleanenv --contain instance://92875428-58f7-47bc-9c37-366bfe8e81e5 sh -c 'cd /tmp/streamflow/3167764c-2a4e-4364-b1fd-8d9eb8b52064 && export HOME="/tmp/streamflow/3167764c-2a4e-4364-b1fd-8d9eb8b52064" && export TMPDIR="/tmp/streamflow/a419bebf-7a55-4701-b683-7b18064e7595" && rev /tmp/streamflow/ec158ac0-36c0-49ac-bffb-a7b84485d60a/a5d79ed3-1a94-4fc2-a6dd-2c76c7abd798/whale.txt > output.txt 2>&1' 2>&1 on LOCAL/LOCAL for job /rev/0
2026-05-05 22:12:59.661 ERROR FAILED Job /rev/0 with error:
Traceback (most recent call last):
File "/mnt/lustre/scratch/nlsas/home/ulc/cursos/curso348/streamflow/cwl-conformance-venv/lib/python3.13/site-packages/streamflow/core/recovery.py", line 42, in wrapper
await func(*args, **kwargs)
File "/mnt/lustre/scratch/nlsas/home/ulc/cursos/curso348/streamflow/cwl-conformance-venv/lib/python3.13/site-packages/streamflow/workflow/step.py", line 743, in _execute_command
raise WorkflowExecutionException(
f"FAILED Job {job.name} with error:\n\t{command_output.value}"
)
streamflow.core.exception.WorkflowExecutionException: FAILED Job /rev/0 with error:
2026-05-05 22:12:59.662 WARNING Job /rev/0 failure can not be recovered. Failure manager is not enabled.
2026-05-05 22:12:59.662 ERROR FAILED Job /rev/0 with error:
2026-05-05 22:12:59.663 DEBUG Job /rev/0 changed status to FAILED
2026-05-05 22:12:59.663 DEBUG EXECUTING command find -L "/tmp/streamflow/3167764c-2a4e-4364-b1fd-8d9eb8b52064" -type f -exec ls -ln {} + | awk 'BEGIN {sum=0} {sum+=$5} END {print sum}'; 2>&1 on SubprocessShell with command
singularity exec instance://92875428-58f7-47bc-9c37-366bfe8e81e5 sh2026-05-05 22:12:59.665 DEBUG EXECUTING command find -L "/tmp/streamflow/a419bebf-7a55-4701-b683-7b18064e7595" -type f -exec ls -ln {} + | awk 'BEGIN {sum=0} {sum+=$5} END {print sum}'; 2>&1 on SubprocessShell with command
singularity exec instance://92875428-58f7-47bc-9c37-366bfe8e81e5 sh2026-05-05 22:12:59.666 DEBUG FAILED Job /rev/0 terminated
2026-05-05 22:12:59.667 DEBUG Step /sorted/reverse-token-transformer received termination token with Status FAILED
2026-05-05 22:12:59.667 DEBUG Step /sorted/input-token-transformer received termination token with Status FAILED
2026-05-05 22:12:59.667 DEBUG Step /sorted/transfer/reverse received termination token with Status FAILED
2026-05-05 22:12:59.667 DEBUG Step /sorted/transfer/input received termination token with Status FAILED
2026-05-05 22:12:59.667 DEBUG Step /sorted/schedule received termination token with Status FAILED
2026-05-05 22:12:59.667 DEBUG Step /sorted received termination token with Status FAILED
2026-05-05 22:12:59.667 DEBUG Step /output-collector-transformer received termination token with Status FAILED
2026-05-05 22:12:59.667 DEBUG Step /output-collector received termination token with Status FAILED
2026-05-05 22:12:59.667 DEBUG Step /output-collector/schedule received termination token with Status FAILED
2026-05-05 22:12:59.668 INFO FAILED Step /rev
2026-05-05 22:12:59.668 DEBUG FAILED Step /sorted/reverse-token-transformer
2026-05-05 22:12:59.668 DEBUG FAILED Step /sorted/input-token-transformer
2026-05-05 22:12:59.668 DEBUG FAILED Step /sorted/schedule
2026-05-05 22:12:59.668 DEBUG FAILED Step /sorted/transfer/reverse
2026-05-05 22:12:59.668 DEBUG FAILED Step /sorted/transfer/input
2026-05-05 22:12:59.668 INFO FAILED Step /sorted
2026-05-05 22:12:59.668 DEBUG FAILED Step /output-collector-transformer
2026-05-05 22:12:59.668 DEBUG FAILED Step /output-collector/schedule
2026-05-05 22:12:59.668 DEBUG FAILED Step /output-collector
2026-05-05 22:12:59.668 INFO UNDEPLOYING a0496f3e-89bd-4a8d-93b0-d0a03750ed3a
2026-05-05 22:12:59.669 DEBUG EXECUTING command singularity instance stop 92875428-58f7-47bc-9c37-366bfe8e81e5 2>&1 on LOCAL/LOCAL
2026-05-05 22:12:59.670 INFO UNDEPLOYING 35592cae-1c01-4642-b6e0-a0216b12644c
2026-05-05 22:12:59.670 INFO COMPLETED undeployment of 35592cae-1c01-4642-b6e0-a0216b12644c
2026-05-05 22:12:59.804 INFO COMPLETED undeployment of a0496f3e-89bd-4a8d-93b0-d0a03750ed3a
2026-05-05 22:12:59.817 ERROR FAILED Workflow execution
Traceback (most recent call last):
File "/mnt/lustre/scratch/nlsas/home/ulc/cursos/curso348/streamflow/cwl-conformance-venv/lib/python3.13/site-packages/streamflow/main.py", line 281, in main
asyncio.run(_async_run(parsed_args))
~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/netapp2/Store_uni/home/ulc/cursos/curso348/micromamba/envs/python-3.13/lib/python3.13/asyncio/runners.py", line 195, in run
return runner.run(main)
~~~~~~~~~~^^^^^^
File "/mnt/netapp2/Store_uni/home/ulc/cursos/curso348/micromamba/envs/python-3.13/lib/python3.13/asyncio/runners.py", line 118, in run
return self._loop.run_until_complete(task)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
File "/mnt/netapp2/Store_uni/home/ulc/cursos/curso348/micromamba/envs/python-3.13/lib/python3.13/asyncio/base_events.py", line 725, in run_until_complete
return future.result()
~~~~~~~~~~~~~^^
File "/mnt/lustre/scratch/nlsas/home/ulc/cursos/curso348/streamflow/cwl-conformance-venv/lib/python3.13/site-packages/streamflow/main.py", line 187, in _async_run
await asyncio.gather(*workflow_tasks)
File "/mnt/lustre/scratch/nlsas/home/ulc/cursos/curso348/streamflow/cwl-conformance-venv/lib/python3.13/site-packages/streamflow/cwl/main.py", line 93, in main
output_tokens = await executor.run()
^^^^^^^^^^^^^^^^^^^^
File "/mnt/lustre/scratch/nlsas/home/ulc/cursos/curso348/streamflow/cwl-conformance-venv/lib/python3.13/site-packages/streamflow/workflow/executor.py", line 175, in run
raise WorkflowExecutionException("FAILED Workflow execution")
streamflow.core.exception.WorkflowExecutionException: FAILED Workflow execution
Which is not very helpful.
But when I inspected the files created, I can see that the
whale.txtfile in/tmp/streamflow/$HASH/whale.txtis pointing to a non-existing location on NFS. If I copy the partstreamflow/cwl-conformance-tests/onward and replace by the $LUSTRE location, the filewhale.txtexists there.So my guess now is that StreamFlow (or a part of it, some lib) is resolving the file based on the location of Python, and not the location where StreamFlow was launched. Causing StreamFlow to fail in environments where for some reason Python & StreamFlow reside on different partitions/filesystems... I think!