Repository navigation
Allocating Java memory for OSMOSE experiments
This page explains how to allocate memory to the Java Virtual Machine (JVM) when running different types of OSMOSE experiments.
Requesting memory from a computing system and allocating memory to Java are two different things.
When running OSMOSE on a cluster, the job scheduler reserves resources for the job. Depending on the system, this may be done with Slurm, PBS, or another scheduler. For example, the job may request 128 GB or 160 GB of RAM from the node.
However, Java will not automatically use all the memory reserved for the job. The maximum Java heap is controlled separately through the -Xmx option. For example:
options = " -Xms1g -Xmx3g"This limits the Java heap to 3 GB (and starts with 1 GB), independently of how much memory was requested from the cluster.
If -Xmx is too small, OSMOSE can fail with:
java.lang.OutOfMemoryError: Java heap space
even when the job itself requested enough memory.
In calibration experiments, parallelisation is often controlled outside Java, by R. In that case, R launches several independent OSMOSE runs, each with its own JVM.
Because many JVMs may run in parallel, each JVM should receive only a limited amount of memory. A typical setting is:
options = "-Xmx3g -Xms1g"This is appropriate when each Java process runs one relatively small OSMOSE simulation, and the external calibration workflow manages the parallel execution of many such processes.
For long OSMOSE simulations with several replicates, parallelisation may instead be handled directly by Java/OSMOSE. In this case, one main JVM controls the full experiment and manages the internal parallelisation across simulations.
The JVM should therefore receive access to most of the memory reserved for the job. For example:
options = "-Xms4g -Xmx110g"The value of -Xmx should remain below the total memory requested from the computing system, leaving some headroom for non-heap memory, native libraries, NetCDF buffers, threads, and the operating system.
Use smaller Java heaps when the parallelisation is controlled externally and many JVMs are launched:
options = " -Xms1g -Xmx3g"Use larger Java heaps when Java/OSMOSE controls the parallelisation within a single JVM:
options = "-Xms4g -Xmx110g"or higher, depending on the memory reserved for the job.
A common symptom of insufficient Java heap memory is:
java.lang.OutOfMemoryError: Java heap space
If this happens, first check whether the job failed because the computing system ran out of memory, or because Java was limited by its own heap size.
If the job requested enough memory but Java was launched with a small value such as:
options = " -Xms1g -Xmx3g"then the problem is likely the JVM memory limit, not the cluster memory allocation.
When diagnosing memory problems, check two levels:
- the memory requested from the computing system or job scheduler;
- the Java heap size defined through
-Xmx.
The memory requested from the scheduler defines how much memory the job is allowed to use on the node. The -Xmx option defines how much of that memory the Java heap is allowed to use.
For long OSMOSE simulations, these two values should be consistent. For example, if the job requests around 128 GB of memory, a Java heap around 100–110 GB may be reasonable.
The exact value should be adjusted based on the observed peak memory use of the experiment. A small test locally, with lower replicates and reduced number of years could help to estimate the actual amount of memory that is needed.
When OSMOSE is launched from R, Java options can be passed directly through the options argument of run_osmose(). This is where JVM memory settings such as -Xms and -Xmx should be defined. For example:
options = "-Xms4g -Xmx110g"
run_osmose(
input = configFile,
osmose = jarFile,
output = outputDir,
options = options,
verbose = TRUE
)The value passed to options should be adapted to the type of experiment and to the memory available for the job.