Revise mars model by cosunae · Pull Request #2 · cosunae/eccodes-cosmo-resources

cosunae · 2023-09-05T08:03:12Z

No description provided.

Its main purpose is to be able to work with COSMO-1E, COSMO-2E, ICON-CH1-EPS, ICON-CH2-EPS, and KENDA-I1 data: - define stream by typeOfGeneratingProcess and backgroundProcess - define model by generatingProcessIdentifier (and additional keys for the COSMO/KENDA-1 case, since KENDA-1 uses the same generatingProcessIdentifier as COSMO-1E) - define type by productDefinitionTemplateNumber (and further, template-dependent keys, if necessary)

…ocessIdentifier for KENDA-CH1

victoria-cherkas · 2023-09-05T14:51:09Z

just some remarks on using this branch for the GRIB_DEFINITION_PATH for archiving the single COSMO run to FDB:

using this schema:
(needed to add levelist and expver to avoid FDB error, others are all the same as you've defined in schema_keys)

[ date, time, stream, class, expver?, model, type
    [ param, number?, quantile?, levtype
        [ step, level?, levelist? ]]]

no number, quantile, level were extracted in any keys.
values of levtype and stream contained unknowns:


    "levtype": [
        "pl",
        "sfc",
        "unknown"
    ],

    "stream": [
        "enda",
        "enfo",
        "unknown"
    ],

cosunae · 2023-09-05T14:58:41Z

Thanks @victoria-cherkas
could you debug the cases for which levtype is not defined ? We need to add a mapping for each pair of typeOfFirstFixedSurface, typeOfSecondFixedSurface to the corresponding levtype, and COSMO definitions are adding several new combinations.
Also which records do not define the stream? i.e. typeOfGeneratingProcess

cosunae · 2023-09-05T15:03:03Z

Also can you give more details concerning 1. ? On which data number and level are not defined ?
I tested grib_ls -p number,level /project/s83c/rz+/mars/COSMO-1E/1h/ml_sl/010/lfff00000000 and they seem to be ok

victoria-cherkas · 2023-09-05T15:10:45Z

For example printing the mars keys with -m we see there is no number, or level :

grib_ls -m /opr/vcherkas/COSMO-1E/23020103_409/000/lfff01150000

but those MARS keys share their names with GRIB keys perhaps which is why you see them with grib_ls -p number,level (?)

victoria-cherkas · 2023-09-05T15:18:26Z

The stream is only 'unknown' for some records in laf or laf*_upscale_in files
Can debug further

petrabaumann · 2023-09-05T15:54:30Z

At least I had checked the model with the COSMO-1E data in /project/s83c/rz+/mars and the script, before I pushed the branch, because my goal was to make it work at least for the COSMO forecasts. KENDA is another subject.

victoria-cherkas · 2023-09-06T07:06:58Z

I see, I was just trying to ingest everything in /opr/vcherkas/COSMO-1E/23020103_409.

petrabaumann · 2023-09-06T07:19:22Z

Ah, then it is clear, why there are records with unknown stream - the directories there als contain the initial condition, which contains data that is produced by the LETKF, and the First Guess. I have not yet added any rule for the First Guess to the scheme. Maybe we can just omit the laf-files, and use only forecast data for this PoC. The KENDA case can be tested separately.

petrabaumann · 2023-09-06T07:20:13Z

So maybe the missing level types also occurred in the KENDA data.

victoria-cherkas · 2023-09-06T07:21:06Z

Sounds good to me, I will omit the laf files, and retry

victoria-cherkas · 2023-09-06T12:16:22Z

Ok so apologies for all the confusion - now I have sorted my environment, there are no unknowns!

The only issue I see is that no mars keys are defined for number, quantile, and level in eccodes or eccodes_cosmo_resources. I believe these are GRIB keys and this is why FDB never uses them to index because it can't find them. Are you sure _get_codes_key in check_mars_model.py is only getting MARS keys?

{
    "date": [
        "20230201"
    ],
    "time": [
        "0300"
    ],
    "class": [
        "od"
    ],
    "stream": [
        "enfo"
    ],
    "type": [
        "ememb"
    ],
    "model": [
        "COSMO-1E"
    ],
    "expver": [
        "1"
    ],
    "step": [
        "0", "1", "10", "11", "12", "13", "14", "15", "16", "17", "18", "19", "2", "20", "21", "22", "23", "24", "25", "26", "27", "28", "29", "3", "30", "31", "32", "33", "34", "35", "36", "37", "38", "39", "4", "40", "41", "42", "43", "44", "45", "5", "6", "7", "8", "9"
    ],
    "levelist": [
        "0", "1", "10", "100", "1000", "11", "12", "13", "14", "15", "150", "16", "17", "18", "19", "2", "20", "200", "21", "22", "23", "24", "25", "250", "26", "27", "28", "29", "3", "30", "300", "31", "32", "33", "34", "35", "350", "36", "37", "38", "39", "4", "40", "400", "41", "42", "43", "44", "45", "450", "46", "47", "48", "49", "5", "50", "500", "51", "52", "53", "54", "55", "550", "56", "57", "58", "59", "6", "60", "600", "61", "62", "63", "64", "65", "650", "66", "67", "68", "69", "7", "70", "700", "71", "72", "73", "74", "75", "750", "76", "77", "78", "79", "8", "80", "800", "81", "850", "9", "900", "925", "950", "975"
    ],
    "levtype": [
        "ml", "pl", "sfc"
    ],
    "param": [
        "500000", "500001", "500002", "500003", "500004", "500006", "500007", "500008", "500010", "500011", "500014", "500015", "500016", "500017", "500027", "500028", "500029", "500030", "500031", "500032", "500034", "500035", "500037", "500038", "500041", "500044", "500046", "500047", "500048", "500049", "500050", "500051", "500053", "500054", "500055", "500056", "500065", "500078", "500080", "500086", "500087", "500088", "500089", "500098", "500100", "500101", "500102", "500103", "500106", "500107", "500108", "500117", "500127", "500128", "500132", "500133", "500134", "500143", "500145", "500146", "500147", "500148", "500149", "500150", "500151", "500152", "500153", "500154", "500158", "500164", "500166", "500167", "500169", "500170", "500171", "500173", "500174", "500175", "500200", "500204", "500205", "500206", "500207", "500208", "500209", "500217", "500218", "500235", "500236", "500237", "500239", "500240", "500241", "500304", "500480", "500481", "500482", "500490", "500491", "500584", "502308", "502757", "503135", "503142", "503154", "503155", "503167", "503186", "503187", "503197", "503204", "503206", "503207", "503299", "503300", "503301", "503302", "503303", "503307", "503375", "503380", "503424", "503425", "503555", "503556", "503557", "503636", "503640"
    ]
}

petrabaumann · 2023-09-06T12:45:08Z

At least number should be a valid MARS key, see for example this sample request for the perturbed forecasts of ECMWF ENS:
retrieve,
class=od,
date=2023-09-06,
expver=1,
levtype=sfc,
number=3,
param=168.128,
step=1,
stream=enfo,
time=00:00:00,
type=pf,
target="output"

I, however, set the self-defined type=cpf', but use stream='enfo', which is also defined by ECMWF. Maybe there is some logic requiring the usage of type='pf' to get the number?

petrabaumann · 2023-09-06T12:54:31Z

It looks as if MARS uses the key 'levelist' instead of 'level'.

petrabaumann · 2023-09-06T12:55:40Z

And actually, the key'level' is not on this list: https://confluence.ecmwf.int/display/UDOC/Keywords+in+MARS+and+Dissemination+requests

victoria-cherkas · 2023-09-06T12:58:29Z

Exactly levelist rather than level.

It seems that number is defined for only some productionStatusOfProcessedData (eccodes/definitions/grib2/template.4.eps.def). productionStatusOfProcessedData=2 in the data I ingested, I believe.

#  Rules for TIGGE, S2S, UERRA and CRRA
if (productionStatusOfProcessedData == 4 ||
    productionStatusOfProcessedData == 5 ||
    productionStatusOfProcessedData == 6 ||
    productionStatusOfProcessedData == 7 ||
    productionStatusOfProcessedData == 8 ||
    productionStatusOfProcessedData == 9 ||
    productionStatusOfProcessedData == 10||
    productionStatusOfProcessedData == 11)
{
    alias mars.number=perturbationNumber;
}

petrabaumann · 2023-09-06T13:00:39Z

The keyword quantile, however, exists, although it is not in the list, e.g. this request for ENS extended range data:
retrieve,
class=od,
date=2023-09-03,
expver=1,
levtype=sfc,
param=167.131,
quantile=1:5,
step=24-192,
stream=eefo,
time=00:00:00,
type=pd,
target="output"

FDB expects at retrieval time that expver comes in a canonical way (4 chars) instead of 1 integer. However at write time, it does not imposes checks nor transform, it will take grib encoding directly. This generates inconsistency between read and write.

set number

expver as string

victoria-cherkas · 2023-10-05T10:04:53Z

@cosunae out of curiosity, how did you arrive at the various levels in the marsLevelConcept file? Did you find these defined somewhere else already?

Fix duplicate class definition

…#8) * Define mars.step by stepRange for stream enfo, type ememb * Add mars aliases for stream enfo, types emean, and estdv * Remove centre-specific definitions from local.78.def

add echoTopInDBZ in grib2/marsLevtypeConcept.def

cosunae and others added 8 commits July 19, 2023 08:32

add mars concepts for cosmo

7d255aa

adding more typeOfLevel levels

7c9f594

adding options and possibility to traverse files in directories

78a5b21

transient->constant

fabe2c2

add an optional key syntax

1ac9878

scripts: check some additional MARS keys in check_mars_model.py

771b76c

definitions/grib2/local.78.def: add the correct name and generatingPr…

63c2a21

…ocessIdentifier for KENDA-CH1

cosunae and others added 6 commits September 15, 2023 10:17

canonical expver for FDB

b03d952

set number

b21e8dc

adding a mars mapping for enfo.ememb

088b8e9

expver as string

f13fbad

Merge pull request #4 from cosunae/mars_number

009ab8e

set number

Merge pull request #5 from cosunae/expver

9807e37

expver as string

victoria-cherkas and others added 8 commits November 10, 2023 13:40

Update local.78.def remove transient class

2554b2b

Merge pull request #7 from cosunae/class-duplication-error

1b6a288

Fix duplicate class definition

Re-arrange inclusion of local MARS concepts, and update MARS mappings (…

07a9785

…#8) * Define mars.step by stepRange for stream enfo, type ememb * Add mars aliases for stream enfo, types emean, and estdv * Remove centre-specific definitions from local.78.def

add echoTopInDBZ in grib2/marsLevtypeConcept.def

a4a95fc

Merge pull request #10 from cosunae/echotopDBZmars

c797d55

add echoTopInDBZ in grib2/marsLevtypeConcept.def

fix step-> step range

74a1c4a

fix step-> step range

7226220

remove the alias step-> stepRange

459e3d3

Conversation

cosunae commented Sep 5, 2023

Uh oh!

victoria-cherkas commented Sep 5, 2023

Uh oh!

cosunae commented Sep 5, 2023

Uh oh!

cosunae commented Sep 5, 2023

Uh oh!

victoria-cherkas commented Sep 5, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

victoria-cherkas commented Sep 5, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

petrabaumann commented Sep 5, 2023

Uh oh!

victoria-cherkas commented Sep 6, 2023

Uh oh!

petrabaumann commented Sep 6, 2023

Uh oh!

petrabaumann commented Sep 6, 2023

Uh oh!

victoria-cherkas commented Sep 6, 2023

Uh oh!

victoria-cherkas commented Sep 6, 2023

Uh oh!

petrabaumann commented Sep 6, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

petrabaumann commented Sep 6, 2023

Uh oh!

petrabaumann commented Sep 6, 2023

Uh oh!

victoria-cherkas commented Sep 6, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

petrabaumann commented Sep 6, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

victoria-cherkas commented Oct 5, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

victoria-cherkas commented Sep 5, 2023 •

edited

Loading

victoria-cherkas commented Sep 5, 2023 •

edited

Loading

petrabaumann commented Sep 6, 2023 •

edited

Loading

victoria-cherkas commented Sep 6, 2023 •

edited

Loading

petrabaumann commented Sep 6, 2023 •

edited

Loading