CORDEX-CMIP6 storage requirements estimation

[jesusff]

In order to secure the archival of CMIP6-driven simulations in ESGF, we first need to estimate the volume of data that we will be producing. An accurate estimate is difficult due to the dependence of the compression rates on the different variables and even on the different models as the effective model resolution affects compression.

A very rough estimate should, in principle, be easily extracted by combining the information in:

• https://github.com/WCRP-CORDEX/domain-tables/blob/main/CORDEX-CMIP5_rotated_grids.csv (nlon x nlat)
• https://github.com/WCRP-CORDEX/simulation-status/blob/main/CMIP6_downscaling_plans.csv (number of simulations planned and length -- depending on eval/hist/ssp)
• https://github.com/WCRP-CORDEX/data-request-table/blob/main/data-request/dreq_default.csv (number and frequency of variables)

[jesusff]

This is a first estimate, only for CORE variables and with an assumed average 60% compression rate wrt to raw binary float precision. This value can be adjusted (e.g. by comparing with real processed output).

ic| simulation_count: experiment  evaluation  historical  ssp119  ssp126  ssp245  ssp370  ssp585
                      domain
                      AFR-25               0           0       0       0       0       0       0
                      ANT-12               3           4       0       0       0       4       0
                      ARC-12               4           6       0       0       0       6       0
                      AUS-20i              4          34       0      34      20      34       4
                      CAM-12               1           8       0       8       8       0       8
                      CAS-12               0           2       0       2       0       2       2
                      EAS-25               0           5       0       5       5       5       5
                      EUR-12              19          59       9      58      18      55      21
                      MED-12              10           9       0       4       1       6       3
                      MED-25               1           1       0       0       0       0       0
                      MENA-25              1           1       0       1       1       1       1
                      NAM-12               1           8       0       4       4       8       1
                      NAM-25               1          15       0       0       0      15       0
                      SAM-25               1           1       0       0       0       1       0
                      SEA-12               1           3       0       0       0       3       0
                      SEA-25               3          13      10      13       3      13       3
                      WAS-25               2           5       0       1       4       2       4
ic| variable_count: priority   CORE  TIER1  TIER2
                    frequency
                    1hr          13     30      7
                    6hr           0     71     51
                    day          15    105     63
                    fx            2      0      7
                    mon          15    105     64
ic| variable_records_per_yr: priority     CORE   TIER1  TIER2
                             frequency
                             1hr        113880  262800  61320
                             6hr             0  103660  74460
                             day          5475   38325  22995
                             fx              0       0      0
                             mon           180    1260    768
ic| size_TB: experiment  evaluation  historical  ssp119  ssp126  ssp245  ssp370  ssp585
             domain
             AFR-25             0.0         0.0     0.0     0.0     0.0     0.0     0.0
             ANT-12             6.8        12.0     0.0     0.0     0.0    19.1     0.0
             ARC-12            11.6        22.9     0.0     0.0     0.0    36.5     0.0
             AUS-20i            4.9        54.4     0.0    86.6    50.9    86.6    10.2
             CAM-12             4.5        47.1     0.0    74.9    74.9     0.0    74.9
             CAS-12             0.0         7.6     0.0    12.1     0.0    12.1    12.1
             EAS-25             0.0         7.7     0.0    12.3    12.3    12.3    12.3
             EUR-12            39.0       159.7    38.8   250.0    77.6   237.0    90.5
             MED-12            11.6        13.8     0.0     9.7     2.4    14.6     7.3
             MED-25             0.3         0.4     0.0     0.0     0.0     0.0     0.0
             MENA-25            1.3         1.7     0.0     2.7     2.7     2.7     2.7
             NAM-12             3.8        40.0     0.0    31.8    31.8    63.6     8.0
             NAM-25             0.9        18.7     0.0     0.0     0.0    29.8     0.0
             SAM-25             1.1         1.5     0.0     0.0     0.0     2.4     0.0
             SEA-12             2.4         9.5     0.0     0.0     0.0    15.2     0.0
             SEA-25             1.8        10.3    12.6    16.4     3.8    16.4     3.8
             WAS-25             2.4         7.8     0.0     2.5     9.9     5.0     9.9
ic| size_TB.T.sum(): domain
                     AFR-25       0.0
                     ANT-12      37.9
                     ARC-12      71.0
                     AUS-20i    293.6
                     CAM-12     276.3
                     CAS-12      43.9
                     EAS-25      56.9
                     EUR-12     892.6
                     MED-12      59.4
                     MED-25       0.7
                     MENA-25     13.8
                     NAM-12     179.0
                     NAM-25      49.4
                     SAM-25       5.0
                     SEA-12      27.1
                     SEA-25      65.1
                     WAS-25      37.5
                     dtype: float64
/!\ Considering just ['CORE'] vars.)
Total CORDEX-CMIP6 estimated size is: 2109 TB

[gnikulin]

I think such an approach should work as an basic estimate. RCMs with non rotated grids usually provide larger domains but some RCM groups may provide only a subset of the Tier1 and 2 variables.

Are all simulations for EUR-12 only 892.6 Tb ?

[jesusff]

Yes, but this is only CORE variables and only those simulations planned so far in this repo. I paste here the summary including all tiers:

ic| simulation_count: experiment  evaluation  historical  ssp119  ssp126  ssp245  ssp370  ssp585
                      domain                                                                    
                      AFR-25               0           0       0       0       0       0       0
                      ANT-12               3           4       0       0       0       4       0
                      ARC-12               4           6       0       0       0       6       0
                      AUS-20i              4          34       0      34      20      34       4
                      CAM-12               1           8       0       8       8       0       8
                      CAS-12               0           2       0       2       0       2       2
                      EAS-25               0           5       0       5       5       5       5
                      EUR-12              19          59       9      58      18      55      21
                      MED-12              10           9       0       4       1       6       3
                      MED-25               1           1       0       0       0       0       0
                      MENA-25              1           1       0       1       1       1       1
                      NAM-12               1           8       0       4       4       8       1
                      NAM-25               1          15       0       0       0      15       0
                      SAM-25               1           1       0       0       0       1       0
                      SEA-12               1           3       0       0       0       3       0
                      SEA-25               3          13      10      13       3      13       3
                      WAS-25               2           5       0       1       4       2       4
ic| variable_count: priority   CORE  TIER1  TIER2
                    frequency                    
                    1hr          13     30      7
                    6hr           0     71     51
                    day          15    105     63
                    fx            2      0      7
                    mon          15    105     64
ic| variable_records_per_yr: priority     CORE   TIER1  TIER2
                             frequency                       
                             1hr        113880  262800  61320
                             6hr             0  103660  74460
                             day          5475   38325  22995
                             fx              0       0      0
                             mon           180    1260    768

/!\ Considering just ['CORE', 'TIER1', 'TIER2'] vars.)

ic| size_TB: experiment  evaluation  historical  ssp119  ssp126  ssp245  ssp370  ssp585
             domain                                                                    
             AFR-25             0.0         0.0     0.0     0.0     0.0     0.0     0.0
             ANT-12            39.2        68.9     0.0     0.0     0.0   109.7     0.0
             ARC-12            66.6       131.5     0.0     0.0     0.0   209.4     0.0
             AUS-20i           27.8       311.6     0.0   496.2   291.9   496.2    58.4
             CAM-12            25.6       269.7     0.0   429.5   429.5     0.0   429.5
             CAS-12             0.0        43.5     0.0    69.2     0.0    69.2    69.2
             EAS-25             0.0        44.1     0.0    70.3    70.3    70.3    70.3
             EUR-12           223.8       915.1   222.3  1432.7   444.6  1358.6   518.8
             MED-12            66.6        78.9     0.0    55.9    14.0    83.8    41.9
             MED-25             1.7         2.2     0.0     0.0     0.0     0.0     0.0
             MENA-25            7.4         9.7     0.0    15.5    15.5    15.5    15.5
             NAM-12            21.7       229.0     0.0   182.4   182.4   364.7    45.6
             NAM-25             5.4       107.3     0.0     0.0     0.0   171.0     0.0
             SAM-25             6.6         8.7     0.0     0.0     0.0    13.8     0.0
             SEA-12            13.8        54.6     0.0     0.0     0.0    86.9     0.0
             SEA-25            10.4        59.1    72.4    94.2    21.7    94.2    21.7
             WAS-25            13.5        44.6     0.0    14.2    56.8    28.4    56.8
ic| size_TB.T.sum(): domain
                     AFR-25        0.0
                     ANT-12      217.8
                     ARC-12      407.5
                     AUS-20i    1682.1
                     CAM-12     1583.8
                     CAS-12      251.1
                     EAS-25      325.3
                     EUR-12     5115.9
                     MED-12      341.1
                     MED-25        3.9
                     MENA-25      79.1
                     NAM-12     1025.8
                     NAM-25      283.7
                     SAM-25       29.1
                     SEA-12      155.3
                     SEA-25      373.7
                     WAS-25      214.3
                     dtype: float64

Total CORDEX-CMIP6 estimated size is: 12090 TB

[gnikulin]

Now it looks reasonable :-) as we need an estimate for all CORE, Tier1 and 2 variables.

[gnikulin]

Although I don't expect many variables from Tier2.

[jesusff]

Ok, this is the remaining one 😉 : CORE + Tier1

ic| simulation_count: experiment  evaluation  historical  ssp119  ssp126  ssp245  ssp370  ssp585
                      domain                                                                    
                      AFR-25               0           0       0       0       0       0       0
                      ANT-12               3           4       0       0       0       4       0
                      ARC-12               4           6       0       0       0       6       0
                      AUS-20i              4          34       0      34      20      34       4
                      CAM-12               1           8       0       8       8       0       8
                      CAS-12               0           2       0       2       0       2       2
                      EAS-25               0           5       0       5       5       5       5
                      EUR-12              19          59       9      58      18      55      21
                      MED-12              10           9       0       4       1       6       3
                      MED-25               1           1       0       0       0       0       0
                      MENA-25              1           1       0       1       1       1       1
                      NAM-12               1           8       0       4       4       8       1
                      NAM-25               1          15       0       0       0      15       0
                      SAM-25               1           1       0       0       0       1       0
                      SEA-12               1           3       0       0       0       3       0
                      SEA-25               3          13      10      13       3      13       3
                      WAS-25               2           5       0       1       4       2       4
ic| variable_count: priority   CORE  TIER1  TIER2
                    frequency                    
                    1hr          13     30      7
                    6hr           0     71     51
                    day          15    105     63
                    fx            2      0      7
                    mon          15    105     64
ic| variable_records_per_yr: priority     CORE   TIER1  TIER2
                             frequency                       
                             1hr        113880  262800  61320
                             6hr             0  103660  74460
                             day          5475   38325  22995
                             fx              0       0      0
                             mon           180    1260    768

/!\ Considering just ['CORE', 'TIER1'] vars.

ic| size_TB: experiment  evaluation  historical  ssp119  ssp126  ssp245  ssp370  ssp585
             domain                                                                    
             AFR-25             0.0         0.0     0.0     0.0     0.0     0.0     0.0
             ANT-12            30.1        52.9     0.0     0.0     0.0    84.2     0.0
             ARC-12            51.1       100.9     0.0     0.0     0.0   160.7     0.0
             AUS-20i           21.3       239.0     0.0   380.6   223.9   380.6    44.8
             CAM-12            19.6       206.9     0.0   329.5   329.5     0.0   329.5
             CAS-12             0.0        33.3     0.0    53.1     0.0    53.1    53.1
             EAS-25             0.0        33.9     0.0    53.9    53.9    53.9    53.9
             EUR-12           171.7       702.0   170.6  1099.1   341.1  1042.3   398.0
             MED-12            51.1        60.6     0.0    42.9    10.7    64.3    32.1
             MED-25             1.3         1.7     0.0     0.0     0.0     0.0     0.0
             MENA-25            5.7         7.5     0.0    11.9    11.9    11.9    11.9
             NAM-12            16.7       175.7     0.0   139.9   139.9   279.8    35.0
             NAM-25             4.2        82.4     0.0     0.0     0.0   131.2     0.0
             SAM-25             5.0         6.6     0.0     0.0     0.0    10.6     0.0
             SEA-12            10.6        41.9     0.0     0.0     0.0    66.7     0.0
             SEA-25             7.9        45.4    55.6    72.2    16.7    72.2    16.7
             WAS-25            10.4        34.2     0.0    10.9    43.5    21.8    43.5
ic| size_TB.T.sum(): domain
                     AFR-25        0.0
                     ANT-12      167.2
                     ARC-12      312.7
                     AUS-20i    1290.2
                     CAM-12     1215.0
                     CAS-12      192.6
                     EAS-25      249.5
                     EUR-12     3924.8
                     MED-12      261.7
                     MED-25        3.0
                     MENA-25      60.8
                     NAM-12      787.0
                     NAM-25      217.8
                     SAM-25       22.2
                     SEA-12      119.2
                     SEA-25      286.7
                     WAS-25      164.3
                     dtype: float64
Total CORDEX-CMIP6 estimated size is: 9275 TB

[jesusff]

Just for the record: some reasonable approximations are in place until all domains are defined in https://github.com/WCRP-CORDEX/domain-tables/blob/main/CORDEX-CMIP5_rotated_grids.csv

simulation-status/storage_estimate.py

Lines 30 to 34 in bc8c5a3

	# Some fixes for missing domains
	ngridcells['AUS-20i'] = ngridcells['AUS-25'] #!!
	ngridcells['MENA-25'] = ngridcells['MNA-25']
	ngridcells['MED-25'] = ngridcells['MED-12']/4
	ngridcells['SEA-12'] = ngridcells['SEA-25']*4

[gnikulin]

I think 10 PB sounds like an reasonable estimate. In any case, it's impossible to provide exact size :-)

[jesusff]

I updated the estimates for CORE+TIER1, but ignoring ESD plans, as they are expected to provide much fewer variables and at maximum daily resolution. They were inflating the estimate for CAM-12 and EUR-12. The estimate is about 8 PB now. Note that this is only the simulations currently reported (i.e. does not include the upcoming CORDEX-CORE effort). The estimate for the simulations which are already completed or running (i.e. not just plans) is about 5 PB.

ic| simulation_count: experiment  evaluation  historical  ssp119  ssp126  ssp245  ssp370  ssp585
                      domain                                                                    
                      AFR-25               0           0       0       0       0       0       0
                      ANT-12               4           6       0       0       0       6       0
                      ARC-12               5           8       0       0       0       8       0
                      AUS-20i              5          35       0      34      20      35       4
                      CAM-25               1           1       0       0       0       0       1
                      CAS-12               0           2       0       2       0       2       2
                      EAS-25               0           5       0       5       5       5       5
                      EUR-12              21          53       4      52      12      50      14
                      MED-12              10          13       0       3       1       8       3
                      MED-25               1           1       0       0       0       0       0
                      MENA-25              1           1       0       1       1       1       1
                      NAM-12               1           8       0       4       4       8       1
                      NAM-25               2          15       0       0       0      15       0
                      SAM-25               1           1       0       0       0       1       0
                      SEA-12               1           3       0       0       0       3       0
                      SEA-25               4          25       6      15       9      21       9
                      WAS-25               2           5       0       1       4       2       4
ic| variable_count: priority   CORE  TIER1  TIER2
                    frequency                    
                    1hr          13     30      7
                    6hr           0     71     51
                    day          15    105     63
                    fx            2      0      7
                    mon          15    104     64
ic| variable_records_per_yr: priority     CORE   TIER1  TIER2
                             frequency                       
                             1hr        113880  262800  61320
                             6hr             0  103660  74460
                             day          5475   38325  22995
                             fx              0       0      0
                             mon           180    1248    768
ic| size_TB: experiment  evaluation  historical  ssp119  ssp126  ssp245  ssp370  ssp585
             domain                                                                    
             AFR-25             0.0         0.0     0.0     0.0     0.0     0.0     0.0
             ANT-12            40.1        79.3     0.0     0.0     0.0   126.3     0.0
             ARC-12            63.8       134.5     0.0     0.0     0.0   214.2     0.0
             AUS-20i           26.7       246.0     0.0   380.6   223.9   391.8    44.8
             CAM-25             4.9         6.5     0.0     0.0     0.0     0.0    10.3
             CAS-12             0.0        33.3     0.0    53.1     0.0    53.1    53.1
             EAS-25             0.0        33.9     0.0    53.9    53.9    53.9    53.9
             EUR-12           189.7       630.6    75.8   985.4   227.4   947.5   265.3
             MED-12            51.1        87.5     0.0    32.1    10.7    85.7    32.1
             MED-25             1.3         1.7     0.0     0.0     0.0     0.0     0.0
             MENA-25            5.7         7.5     0.0    11.9    11.9    11.9    11.9
             NAM-12            16.7       175.7     0.0   139.9   139.9   279.8    35.0
             NAM-25             8.3        82.3     0.0     0.0     0.0   131.1     0.0
             SAM-25             5.0         6.6     0.0     0.0     0.0    10.6     0.0
             SEA-12            10.6        41.9     0.0     0.0     0.0    66.7     0.0
             SEA-25            10.6        87.2    33.3    83.3    50.0   116.7    50.0
             WAS-25            10.4        34.2     0.0    10.9    43.5    21.8    43.5
ic| size_TB.T.sum(): domain
                     AFR-25        0.0
                     ANT-12      245.7
                     ARC-12      412.5
                     AUS-20i    1313.8
                     CAM-25       21.7
                     CAS-12      192.6
                     EAS-25      249.5
                     EUR-12     3321.7
                     MED-12      299.2
                     MED-25        3.0
                     MENA-25      60.8
                     NAM-12      787.0
                     NAM-25      221.7
                     SAM-25       22.2
                     SEA-12      119.2
                     SEA-25      431.1
                     WAS-25      164.3
                     dtype: float64
/!\ Considering just ['CORE', 'TIER1'] vars.)
Total CORDEX-CMIP6 estimated size is: 7866 TB

[jesusff]

I provide a new update on this rough estimates with the new simulations that are planned (e.g. for CORDEX-CORE). Core+Tier1 variables reach now 11 PB. Considering only simulations completed and running (i.e. expected in the next few months) Core+Tier1 variables take 8.3 PB (2 PB the Core variables alone).

ping @aulemahal @bthrasher

ic| simulation_count: driving_experiment_id  evaluation  historical  ssp119  ...  ssp245  ssp370  ssp585
                      domain_id                                              ...                        
                      AFR-12                          1           3       0  ...       0       3       0
                      AFR-18                          1           3       0  ...       0       3       0
                      ANT-12                          4           6       0  ...       0       6       0
                      ARC-12                          6          10       0  ...       0      10       0
                      AUS-12                          1           3       0  ...       0       3       0
                      AUS-20i                         5          35       0  ...      20      35       4
                      CAM-12                          2           3       0  ...       0       3       0
                      CAM-25                          1           1       0  ...       0       0       1
                      CAS-12                          0           2       0  ...       0       2       2
                      EAS-12                          1           3       0  ...       0       3       0
                      EAS-25                          0           5       0  ...       5       5       5
                      EUR-12                         26          52       1  ...      11      45      14
                      MED-12                         11          13       0  ...       1       9       3
                      MED-25                          1           1       0  ...       0       0       0
                      MENA-25                         1           1       0  ...       1       1       1
                      NAM-12                          3          15       0  ...       7      14       2
                      NAM-25                          1          21       0  ...      21      21       0
                      SAM-12                          2           6       0  ...       0       6       0
                      SAM-25                          2           3       0  ...       1       1       2
                      SEA-12                          2           6       0  ...       0       6       0
                      SEA-25                          5          26       6  ...      13      21      11
                      WAS-12                          1           3       0  ...       0       3       0
                      WAS-18                          1           3       0  ...       0       3       0
                      WAS-25                          1           3       0  ...       3       0       3
                      
                      [24 rows x 7 columns]
ic| variable_count: priority   CORE  TIER1  TIER2
                    frequency                    
                    1hr          13     30      7
                    6hr           0     71     51
                    day          15    105     63
                    fx            2      0      7
                    mon          15    104     64
ic| variable_records_per_yr: priority     CORE   TIER1  TIER2
                             frequency                       
                             1hr        113880  262800  61320
                             6hr             0  103660  74460
                             day          5475   38325  22995
                             fx              0       0      0
                             mon           180    1248    768
ic| size_TB: driving_experiment_id  evaluation  historical  ssp119  ...  ssp245  ssp370  ssp585
             domain_id                                              ...                        
             AFR-12                       32.3       127.5     0.0  ...     0.0   203.0     0.0
             AFR-18                       16.1        63.7     0.0  ...     0.0   101.5     0.0
             ANT-12                       40.1        79.3     0.0  ...     0.0   126.3     0.0
             ARC-12                       76.6       168.1     0.0  ...     0.0   267.8     0.0
             AUS-12                       21.3        84.4     0.0  ...     0.0   134.3     0.0
             AUS-20i                      26.7       246.0     0.0  ...   223.9   391.8    44.8
             CAM-12                       39.3        77.6     0.0  ...     0.0   123.6     0.0
             CAM-25                        4.9         6.5     0.0  ...     0.0     0.0    10.3
             CAS-12                        0.0        33.3     0.0  ...     0.0    53.1    53.1
             EAS-12                       28.1       110.8     0.0  ...     0.0   176.5     0.0
             EAS-25                        0.0        33.9     0.0  ...    53.9    53.9    53.9
             EUR-12                      234.9       618.7    18.9  ...   208.4   852.7   265.3
             MED-12                       56.2        87.5     0.0  ...    10.7    96.4    32.1
             MED-25                        1.3         1.7     0.0  ...     0.0     0.0     0.0
             MENA-25                       5.7         7.5     0.0  ...    11.9    11.9    11.9
             NAM-12                       50.0       329.4     0.0  ...   244.8   489.6    69.9
             NAM-25                        4.2       115.3     0.0  ...   183.6   183.6     0.0
             SAM-12                       40.3       159.4     0.0  ...     0.0   253.9     0.0
             SAM-25                       10.1        19.9     0.0  ...    10.6    10.6    21.2
             SEA-12                       21.2        83.7     0.0  ...     0.0   133.3     0.0
             SEA-25                       13.2        90.7    33.3  ...    72.2   116.7    61.1
             WAS-12                       20.8        82.0     0.0  ...     0.0   130.6     0.0
             WAS-18                       10.4        41.0     0.0  ...     0.0    65.3     0.0
             WAS-25                        5.2        20.5     0.0  ...    32.7     0.0    32.7
             
             [24 rows x 7 columns]
ic| size_TB.T.sum(): domain_id
                     AFR-12      362.8
                     AFR-18      181.3
                     ANT-12      245.7
                     ARC-12      512.5
                     AUS-12      240.0
                     AUS-20i    1313.8
                     CAM-12      240.5
                     CAM-25       21.7
                     CAS-12      192.6
                     EAS-12      315.4
                     EAS-25      249.5
                     EUR-12     3089.5
                     MED-12      325.8
                     MED-25        3.0
                     MENA-25      60.8
                     NAM-12     1323.6
                     NAM-25      670.3
                     SAM-12      453.6
                     SAM-25       72.4
                     SEA-12      238.2
                     SEA-25      492.8
                     WAS-12      233.4
                     WAS-18      116.7
                     WAS-25       91.1
                     dtype: float64
/!\ Considering ['CORE', 'TIER1'] variables
/!\ Considering status ['published', 'completed', 'running', 'planned']
Total CORDEX-CMIP6 estimated size is: 11047 TB