WIS2 GDC harvesting from remote catalogues

[tomkralidis]

The WIS2 discovery/metadata publication and transmission workflow can be summarized as:

• metadata are treated the same as data
• a WNM of a WCMP2 record is published by a WIS2 Node and is acted upon by the WIS2 Global Discovery Catalogue accordingly
  • definition
  • technical considerations

Numerous data catalogues already exist with copious metadata in various Earth system disciplines (e.g. ODIS for ocean, GCW for cryosphere, WHOS for hydrology, etc.). Providing WIS2 discovery of data from these catalogues provides value for WIS2.

To comply w/ WIS2, these catalogues would need to:

1. be registered as a WIS2 Node (along w/ a centre identifier)
2. emit WCMP2 as a representation
3. make available (or leverage) an MQTT broker
4. publish a WNM of WCMP2 on the .../metadata topic

Item 2 is likely the least prohibitive for these catalogues. If we want to make more data discoverable on WIS2, are there alternate pathways?

Options:

1. keep the same architecture and requirements as current
2. provide a "mediator" type WIS2 Node that will (on reference of a WCMP2 via URL), publish the metadata record via MQTT && WTH/WNM
3. enable GDCs to "harvest" WCMP2 from "certified" data catalogues, by aggregating from these catalogues, ensuring the canonical representation dereferences at source
4. no aggregation; real-time distributed search (accurate but high risk of network brittleness/latency, cumbersome to combine result sets, hence not recommended)
5. other options?

[6a6d74]

@tomkralidis - which domain is providing the driver for this? Cryo, hydro, ocean ... etc.

Would there be support from the "other side" to get something implemented?

[steingod]

From the cryo perspective this would be a welcome approach, and we would provide records through pycsw from GCW.

[golfvert]

When catalogue X is integrated/merged into the WIS2 catalogue, what will it bring to users ?
Another benefit of WIS2 is the standard announcement of new data/products accessible for download, are we also aiming at this next step of "integration" ?

[tomkralidis]

Related item (in OGC API - Records): opengeospatial/ogcapi-records#48

[tomkralidis]

Update (via discussions with @steingod)

-pathway for GCW to provide their catalogue endpoint (with 97 collections in WIS2 data granularity) for harvesting. Harvesting would be a preferred approach (thus reducing GCW infrastructure requirements for MQTT provisioning), although a GCW arrangement with an existing MQTT provider would work as well

• content
  • metadata crosswalk exercise available at https://github.com/metno/mmd/blob/master/crosswalks/MMD-WCMP2_schema_mapping.adoc
  • pygeometa tool updated to support transformation of GCW internal metadata format (MMD) to WCMP2 implemented in add support for MMD (import) geopython/pygeometa#347
• GCW would be required to submit paperwork for DCPC status (int-gcw ?). @tomkralidis to follow up with @steingod offline

The core matter here is the value proposition of metadata harvesting into WIS2 GDCs with or without event driven data notifications via MQTT. Given GCW datasets are a hybrid between local and distributed endpoints and access mechanisms, is there value of a WIS2 discovery of GCW datasets and being able to assess/access without an MQTT capability? I would say yes, as WIS2 already supports non-realtime data publication as well.

IMO GCW using a shared MQTT capability while publishing WCMP2 to origin/a/wis2/int-gcw/metadata would be the shortest path here for initial implementation of GCW <-> WIS2.