Pyxis not working with Slurm's Task Prolog

[mwessman-amd]

We are using Slurm's Task Prolog in our cluster for some automatic environment variable setting, but I recently noticed that there seems to be an integration issue between Pyxis and using task prologs. Regular epilogs and prologs work completely fine without issues, but for some reason the task_prolog.sh script breaks Pyxis.

$ srun -t 1:00:00 --gpus 1 --cpus-per-gpu 8 --mem-per-cpu 8G --container-image=hello-world:latest echo 1
error: run_command: slurm task_prolog can not be executed (/etc/slurm/task_prolog.sh) No such file or directory
error: slurm task_prolog did not exit normally. reason: Run command failed - configuration error
error: TaskProlog failed status=1
srun: error: task 0: Exited with exit code 1

A simple workaround for this would be to simply add --container-mounts=/etc/slurm/task_prolog.sh:/etc/slurm/task_prolog.sh to the srun command above. But I'm more curious about why the container tries to run the task_prolog script inside the container without even trying to mount it first?
If it's smart enough to try to run it automatically inside the container, shouldn't it also mount the file automatically if it detects that a task_prolog.sh file exists?

I am currently running these versions:
Slurm 25.11.2
Enroot 4.1.0
Pyxis 0.22.0

[flx42]

pyxis doesn't execute the TaskProlog, slurmd does.
But you can easily inject the mount for all containers, by creating an enroot mount file like this:

echo '/etc/slurm/task_prolog.sh' | sudo tee /etc/enroot/mounts.d/task_prolog.fstab