Skip to content

Set $GALAXY_MEMORY_GB and allow for lowering values by an overhead. #21735

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
natefoo wants to merge 2 commits into
base: dev
Choose a base branch
from
+41 −0
Draft

Set $GALAXY_MEMORY_GB and allow for lowering values by an overhead. #21735

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
473006f
Set `$GALAXY_MEMORY_GB` and allow for lowering values by an overhead.
natefoo Feb 3, 2026
d057c7c
Document new env vars
natefoo Feb 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 24 additions & 0 deletions lib/galaxy/config/sample/job_conf.sample.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -737,6 +737,30 @@ execution:
# clusters. These don't need to be available on the Galaxy server
# itself.

# Under most runners, Galaxy will set the `$GALAXY_SLOTS` environment variable in the
# tool execution environment to the number of cores allocated to the job by the
# scheduler. Tools can use this to control how many threads or processes they start.
#
# Under some runners, most notably Slurm and SGE, Galaxy will set environment
# variables related to the amount of memory allocated to the job by the scheduler, if
# possible. These are `$GALAXY_MEMORY_MB`, `$GALAXY_MEMORY_GB`,
# `$GALAXY_MEMORY_MB_PER_SLOT`, and `$GALAXY_MEMORY_GB_PER_SLOT`. You can override
# these on a per-environment basis as shown below.
#
# Additionally, you can specify two variables that will indicate a tool should use
# less memory than allocated. Typically some overhead is required or else the Linux
# OOM Killer will simply kill the job's processes if they exceed they amount of memory
# allocated by the scheduler.
job_vars_example:
runner: drmaa
env:
# Sets `$GALAXY_MEMORY_MB` to 2GB less than what has been allocated
- name: GALAXY_MEMORY_MB_OVERHEAD
value: "2048"
# Floor value for `$GALAXY_MEMORY_MB` if (allocated - overhead < floor)
- name: GALAXY_MEMORY_MB_FLOOR
value: "1024"

# Following three environments demonstrate setting up per-job temp directory handling.
# In these cases TEMP, TMP, and TMPDIR will be set for each job dispatched to these
# environments.
Expand Down
17 changes: 17 additions & 0 deletions lib/galaxy/jobs/runners/util/job_script/MEMORY_STATEMENT_TEMPLATE.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,10 +5,27 @@ if [ -n "$SGE_HGR_h_vmem" ]; then
GALAXY_MEMORY_MB_PER_SLOT=`echo "$SGE_HGR_h_vmem" | sed 's/G$/ * 1024/' | bc | cut -d"." -f1` 2>$metadata_directory/memory_statement.log
fi

if [ -n "$GALAXY_MEMORY_MB" -a -n "${GALAXY_MEMORY_MB_OVERHEAD:-}" ]; then
GALAXY_MEMORY_MB=$(($GALAXY_MEMORY_MB - GALAXY_MEMORY_MB_OVERHEAD))

@bernt-matthias bernt-matthias Feb 5, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will only work for SLURM jobs (for which we only set GALAXY_MEMORY_MB so far) for SGE jobs we only have GALAXY_MEMORY_MB_PER_SLOT.

@natefoo natefoo Feb 5, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll reorder - I originally had it below the next bit where we set $GALAXY_MEMORY_MB from $GALAXY_MEMORY_MB_PER_SLOT, but moved it up so $GALAXY_MEMORY_MB_PER_SLOT would take the overhead into account. I can move it back down and then just recalculate $GALAXY_MEMORY_MB_PER_SLOT accordingly.

[ "$GALAXY_MEMORY_MB" -gt "${GALAXY_MEMORY_MB_FLOOR:=256}" ] || GALAXY_MEMORY_MB=$GALAXY_MEMORY_MB_FLOOR
fi

if [ -z "$GALAXY_MEMORY_MB_PER_SLOT" -a -n "$GALAXY_MEMORY_MB" ]; then
GALAXY_MEMORY_MB_PER_SLOT=$(($GALAXY_MEMORY_MB / $GALAXY_SLOTS))
elif [ -z "$GALAXY_MEMORY_MB" -a -n "$GALAXY_MEMORY_MB_PER_SLOT" ]; then
GALAXY_MEMORY_MB=$(($GALAXY_MEMORY_MB_PER_SLOT * $GALAXY_SLOTS))
fi

if [ -n "$GALAXY_MEMORY_MB" -a -z "$GALAXY_MEMORY_GB" ]; then
GALAXY_MEMORY_GB=$(($GALAXY_MEMORY_MB / 1024))
[ "$GALAXY_MEMORY_GB" -gt 0 ] || GALAXY_MEMORY_GB=1
fi
if [ -n "$GALAXY_MEMORY_GB" -a -z "$GALAXY_MEMORY_GB_PER_SLOT" ]; then
GALAXY_MEMORY_GB_PER_SLOT=$(($GALAXY_MEMORY_GB / $GALAXY_SLOTS))
[ "$GALAXY_MEMORY_GB_PER_SLOT" -gt 0 ] || GALAXY_MEMORY_GB_PER_SLOT=1

@bernt-matthias bernt-matthias Feb 5, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For GALAXY_SLOTS>1 this would mean that the total memory is larger then the floor value? Why are you checking for greater 0 anyway (Also in the previous if) -- is it for accomodating rounding problems?

@natefoo natefoo Feb 5, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, the idea here is if $(($GALAXY_MEMORY_GB / $GALAXY_SLOTS)) is less than 1, the division results in 0, so this ensures $GALAXY_MEMORY_GB_PER_SLOT is not set to 0, since I imagine most tools are not going to react well to that. I admit that setting it to 1 is also wrong, but I don't have a better solution for this case.

fi

[ "${GALAXY_MEMORY_MB--1}" -gt 0 ] 2>>$metadata_directory/memory_statement.log && export GALAXY_MEMORY_MB || unset GALAXY_MEMORY_MB
[ "${GALAXY_MEMORY_MB_PER_SLOT--1}" -gt 0 ] 2>>$metadata_directory/memory_statement.log && export GALAXY_MEMORY_MB_PER_SLOT || unset GALAXY_MEMORY_MB_PER_SLOT
[ "${GALAXY_MEMORY_GB--1}" -gt 0 ] 2>>$metadata_directory/memory_statement.log && export GALAXY_MEMORY_GB || unset GALAXY_MEMORY_GB
[ "${GALAXY_MEMORY_GB_PER_SLOT--1}" -gt 0 ] 2>>$metadata_directory/memory_statement.log && export GALAXY_MEMORY_GB_PER_SLOT || unset GALAXY_MEMORY_GB_PER_SLOT
Loading