Set $GALAXY_MEMORY_GB and allow for lowering values by an overhead.

[natefoo]

Fixes #15952.

How to test the changes?

(Select all options that apply)

• I've included appropriate automated tests.
• This is a refactoring of components with existing test coverage.
• Instructions for manual testing are as follows:
  1. Using Slurm you can verify that $GALAXY_MEMORY_GB is set properly using metrics or inspecting the tool_script.sh
  2. Using the local runner you can set $GALAXY_MEMORY_MB as an env var and the others should be set accordingly.
  3. Test the overhead and floor vars the same way, this simple tool is helpful:

     <tool id="vartest" name="vartest" version="0.0.1" profile="16.04">
         <command><![CDATA[
             echo "$echo" >  "$output1"
             ; declare -p GALAXY_MEMORY_MB >> "$output1" 2>&1 || true
             ; declare -p GALAXY_MEMORY_MB_PER_SLOT >> "$output1" 2>&1 || true
             ; declare -p GALAXY_MEMORY_GB >> "$output1" 2>&1 || true
             ; declare -p GALAXY_MEMORY_GB_PER_SLOT >> "$output1" 2>&1 || true
             ; declare -p GALAXY_MEMORY_MB_OVERHEAD >> "$output1" 2>&1 || true
             ; declare -p GALAXY_MEMORY_MB_FLOOR >> "$output1" 2>&1 || true
         ]]></command>
         <inputs>
             <param name="echo" type="text" title="Text to echo"/>
         </inputs>
         <outputs>
             <data name="output1" format="txt" />
         </outputs>
     </tool>

License

• I agree to license these and all my past contributions to the core galaxy codebase under the MIT license.

[bernt-matthias]

Wondering if am additive or multiplicative overhead is better (or both). In my setup I use -Xmx$JAVAMEM where $GALAXY_MEMORY_MB*2/3 seemed to be a value that always works for me, but I never tested with additive overheads.

[bernt-matthias]

This will only work for SLURM jobs (for which we only set GALAXY_MEMORY_MB so far) for SGE jobs we only have GALAXY_MEMORY_MB_PER_SLOT.

[natefoo]

I'll reorder - I originally had it below the next bit where we set $GALAXY_MEMORY_MB from $GALAXY_MEMORY_MB_PER_SLOT, but moved it up so $GALAXY_MEMORY_MB_PER_SLOT would take the overhead into account. I can move it back down and then just recalculate $GALAXY_MEMORY_MB_PER_SLOT accordingly.

[bernt-matthias]

For GALAXY_SLOTS>1 this would mean that the total memory is larger then the floor value? Why are you checking for greater 0 anyway (Also in the previous if) -- is it for accomodating rounding problems?

[natefoo]

Yeah, the idea here is if $(($GALAXY_MEMORY_GB / $GALAXY_SLOTS)) is less than 1, the division results in 0, so this ensures $GALAXY_MEMORY_GB_PER_SLOT is not set to 0, since I imagine most tools are not going to react well to that. I admit that setting it to 1 is also wrong, but I don't have a better solution for this case.

[natefoo]

So I did think about additive vs. multiplicative beforehand and the issue I saw with that is that at large memory allocations you waste a lot of memory with multiplicative. I think it's more often the case that you want to leave 1-2 GB overhead than you want to leave 25%, especially when you're giving a job 1 TB.

[mvdbeek]

@natefoo Given @bernt-matthias comments and you wanting to re-order (#21735 (comment)) i'll move the milestone, but if you have time to update this we can