Consideration of properties.datetime in duplicate checking

[josusky]

I am not sure if this is the right place to mention the issue, but it was raised during testing so let's start here. The current test for Global Cache expects that a a notification with the same pubtime but different datetime is considered a duplicate. I think that this is not in line with the WIS Manual on WIS, APPENDIX E: WIS2 NOTIFICATION MESSAGE. The section "1.7 Properties / Publication time" contains except other the following statement:

Ensuring pubtime is properly managed for updates and deletions is important for data and metadata download workflows. For example, an out-of-date pubtime can lead to errors for clients when managing updates or deletions in their local copies of data. An update with newer pubtime and identical datetime indicates a newer version of the data or metadata.

Moreover the section "1.11 Properties / Temporal description" makes it even clearer:

The datetime property identifies the date and time of the data (for example, when a measurement was observed). When a data or metadata is updated or deleted, this value should identify the original data or metadata, which can be significantly different from the current time.

My conclusion is, that a notification about a data instance can be a duplicate only if it has the same datetime and the same or older pubtime.

[josusky]

@amilan17 @hhaddouch Could you please assign this task to someone? I do not seem to have such right in this project. I think that the correct duplicate checking is important for WIS2 to GTS testing.

[kaiwirt]

In our case we only consider data_id and pubtime when checking for duplicates. If there is an "update" link in the message, then we only check for pubtime.

[josusky]

WMO publication 1060 vol II (aka manual on WIS 2.0), in chapter "1.11 Properties/Temporal description" states:

The datetime property identifies the date and time of the data (for example, when a
measurement was observed). When data or metadata are updated or deleted, this value should
identify the original date/time of the data or metadata, which can be significantly different from
the current time.

this is also mentioned in "1.7 Properties/Publication time":

An update with a newer pubtime and identical datetime indicates a newer version of the data or metadata.

In other words, the data_id is not necessary unique without combining it with the datetime. I know that usually the datetime is encoded in the data_id but that is strictly speaking not required.

[kaiwirt]

The requirement is, that data_id is used to uniquely identify a data set and is to be used when checking for duplicates. Same data_id in two messages -> duplicate. That is, unless the WNM contains an "update" link.

[6a6d74]

A data-object has a unique data_id. The datetime is an intrinsic part of what makes the data-object unique, i.e., datetime values 1-hour apart would indicate two individual observations. For that reason, I believe that it is sufficient to only check data_id for uniqueness of the data-object. Only if the WNM contains a link-object with rel=update should a WNM with a previously seen data_id be processed.

@kaiwirt - I think this is in-line with what you wrote above.

@josusky - are you content to close this issue?