When Running for higher experiment config e.g. (2k samples and 66k locations), I see a RAM spike which causes an OOM.
I have narrowed it down to np.multiply.outer in the Ice sheet and glacier modules (see L 71-74). With larger chunks (>500), this creates a very large matrix that has to be held in memory and causes the OOM.
A suggested fix is to chunk samples too:
sample_chunk = 200
gis_samps = da.from_array(gis_samps, chunks=(sample_chunk, nyears))
gissl = gis_samps[:, :, None] * gisfp[None, None, :]
wais_samps = da.from_array(wais_samps, chunks=(sample_chunk, nyears))
eais_samps = da.from_array(eais_samps, chunks=(sample_chunk, nyears))
waissl = wais_samps[:, :, None] * waisfp[None, None, :]
eaissl = eais_samps[:, :, None] * eaisfp[None, None, :]
The speed of your FACTS experiment will depend on a combination of chunksize and sample chunks.
In this configuration, you can replicate AR6 ( >2.5TB).
When Running for higher experiment config e.g. (2k samples and 66k locations), I see a RAM spike which causes an OOM.
I have narrowed it down to
np.multiply.outerin the Ice sheet and glacier modules (see L 71-74). With larger chunks (>500), this creates a very large matrix that has to be held in memory and causes the OOM.A suggested fix is to chunk samples too:
The speed of your FACTS experiment will depend on a combination of chunksize and sample chunks.
In this configuration, you can replicate AR6 ( >2.5TB).