CORA TS - failed to process query with the filter: "use_specific_dataset"

[geofrizz]

With the API it's possible to download the metadata for a specific dataset id

e.g.: https://beacon-cora-ts.maris.nl/api/datasets/dataset-metadata/64282

{
"id": 64282,
"name": "datasets/2018/CO_DMQCGL01_20180520_TS_MO.nc",
"tensors": {
"CONFIG_MISSION_NUMBER": {
"name": "CONFIG_MISSION_NUMBER",
"tensor_range": {
...

If you try to download the data with the following query:

  "query_parameters": [
    {
      "column_name": "TEMP",
      "alias": "Temperature [celsius]"
    },
    {
      "column_name": "JULD",
      "alias": "Julian Date"
    },
    {
      "column_name": "DEPH",
      "alias": "Depth [meter]"
    },
    {
      "column_name": "LATITUDE",
      "alias": "Latitude [degrees_north]"
    },
    {
      "column_name": "LONGITUDE",
      "alias": "Longitude [degrees_east]"
    },
    {
      "column_name": "PLATFORM_NUMBER",
      "alias": "PLATFORM_NUMBER"
    }
  ],
  "filters": [
    {
        "use_specific_datasets": [ 64282 ]
    },
  ],
  "output": {
    "format": "netcdf"
  }
}

The result message is:

"Failed to process query: Failed to process query: Memory limit exceeded. Limit: 17179869184."

Probably the filter doesn't work because the original netcdf file from copernicus (CO_DMQCGL01_20180520_TS_MO.nc) is only 270.2 KB and I think that doesn't exceed the limit !! :-)

I used the filter in the wrong manner ?

[robinskil]

Hi,

Thanks for the detailed explanation.
I can confirm that it indeed shouldn't fail. I will have to check if it is something with the filter. (It could be that the specific datasets filters is not correctly being applied somehow by Beacon)

Cheers,
Robin

[geofrizz]

Hi Robin,
today I can add other details, I apply the follow query:

  "query_parameters": [
    {
      "column_name": "TEMP",
      "alias": "Temperature [celsius]"
    },
    {
      "column_name": "JULD",
      "alias": "Julian Date"
    },
    {
      "column_name": "DEPH",
      "alias": "Depth [meter]"
    },
    {
      "column_name": "LATITUDE",
      "alias": "Latitude [degrees_north]"
    },
    {
      "column_name": "LONGITUDE",
      "alias": "Longitude [degrees_east]"
    },
    {
      "column_name": "PLATFORM_NUMBER",
      "alias": "PLATFORM_NUMBER"
    }
  ],
  "filters": [
    {
        "use_specific_datasets": [ 64282 ]
    },
    {
        "for_query_parameter": "Longitude [degrees_east]",
        "min": 2.50,
        "max": 10.0
    },
    {
        "for_query_parameter": "Latitude [degrees_north]",
        "min": 41.5,
        "max": 44.5
    },
    {
        "for_query_parameter": "Depth [meter]",
        "min": 0,
        "max": 5000,
    },
    {
        "for_query_parameter": "Julian Date",
        "min": 24976,
        "max": 24976.999305555557,
    }
  ],
  "output": {
    "format": "netcdf"
  }
}

I tried to recreate the same area with a bbox and datetime filter (latitude & longitude, juld).

Two datasets in the result (dataset_id: 64284, 64282)

Some problem with the depth: my request 0 - 5000, the answer is 0 - 2316.0; the metadata report 0 - 2565.468017578125

I checked the Platform_Number, also in this case there are some inconsistencies, in the beacon extraction are present some new Platform_Number (this is correct, there are 2 dataset_id) but some Platform_Number present in copernicus are missing:

BEACON ,COPERNICUS
,
FGTO ,none
FNCM ,none
none ,EXIN0002
EXIN0004 ,EXIN0004
none ,IF000584
6100001 ,6100001
6100002 ,6100002
6100021 ,6100021
6100022 ,6100022
6100188 ,6100188
6100189 ,none
6100190 ,6100190
6100191 ,6100191
6100284 ,6100284
none ,6100289
6100294 ,6100294
6100295 ,6100295
6100431 ,6100431
6800418 ,6800418
6801015 ,6801015

At the end the amount of data are really small: 3425 rows (beacon) vs 407680 rows (copernicus)

Copernicus data:

beacon data:

I hope that these new details can help you!

Ciao,
Paolo

[robinskil]

Hi Paolo,

I will take a look at the Platform_Number differences. As soon as i know more, i will let you know.

Regarding the number of rows returned for the query, it could be because when filtering on a column containing fill values, beacon will automatically remove these records as well. So when filtering on the depth columns, it would remove records where the depth value is a fill value.

Cheers,
Robin

[robinskil]

Hi Paolo,

I've quickly checked, if you remove the depth filter, you will get back the correct Platform_Number records. It could be that the timeseries for these platforms, don't have any valid depth values (only fill values) and thus would be removed when applying a depth filter.

Cheers,
Robin