Different SLURM queues for CPU and GPU steps

[lbeltrame]

Hello,

this is more related to the Snakemake workflow, but since there are discussions enabled here, I'd rather post in this forum rather than open an issue for what it is essentially a question.

At my current workplace we run AF in two separate steps and two different queues on our HPC cluster, because CPU-only nodes are far more than GPU nodes, and so we free up resources by running CPU-only steps there. I'm not very familiar with Snakemake (last time I used it was in 2019, and I guess things changed a bit...), but can (similarly to one can do with the two separate scripts) one direct the CPU-bound steps (what the first script does) to one SLURM queue, and the GPU-bound (what the second script does) to another?

Thanks a lot. Apologies if it's unclear!

[DimaMolod]

Hi @lbeltrame,

yes, it's the default behaviour. Snakemake submits every rule as its own SLURM job, so each step can have its own partition. Only the GPU rule, structure_inference, pins one from slurm_partition in config/config.yaml:

slurm_partition: "gpu-el8"   # only structure_inference uses this

The CPU rules (create_features, i.e. what your first script does, plus the small download/symlink/analysis ones) don't set a partition, so they go to your cluster's default one. If that isn't the CPU queue you want, name it in the Snakemake profile config/profiles/slurm/config.yaml:

default-resources:
  - slurm_partition=cpu-el8   # <- your CPU queue

Rule-level resources win over default-resources, so inference still goes to the GPU partition. For per-rule control there's also set-resources (commented example in that same file), which overrides both:

set-resources:
  - create_features:slurm_partition=cpu-el8
  - structure_inference:slurm_partition=gpu-el8

Same works for qos, mem_mb, runtime, etc.

Bonus over the two-script setup: each prediction only depends on its own features, not on all of them, so GPU jobs start going out while the remaining MSAs are still running on the CPU nodes.

Hope that helps!

[lbeltrame]

Hello @DimaMolod!

This confirms what I've seen at a very cursory glance. This is actually a huge plus when evaluating deployment! Thanks!