Refine subscription and processing steps for Sensor Centre

[golfvert]

Now that all GC include the properties.global-cache, the text can be amended. And based on the suggestion from ET-WISOP on May 18th, add a check of the provided link.

[kaiwirt]

@golfvert

I would suggest, that the sensor centre is asked to store and make available the missed data_ids. This could be done for example by a monitor message destined to the global cache or by providing a means to download a list of data ids for a given global cache.

Without data id there is no way for a Global Cache to investigate issues because the metrics are telling the cache only "You are missing something" without specifying what.

[golfvert]

My implementation of the Sensor Centre already does that. It creates an hourly message on monitor topic with a link to a file with all data_id seen over the last hour (in fact the full WNM from origin) and then for each global cache a timestamp when the data_id was republished by the cache and, if missed a 999....999 instead. An example of such entry:

{
  "level": "info",
  "message": {
    "centre-id": "uk-metoffice-wmc",
    "cn-cma-global-cache": "1779206520382",
    "data-metoffice-noaa-global-cache": "9999999999999",
    "de-dwd-global-cache": "1779206567012",
    "fr-meteofrance-global-cache": "1779206526907",
    "jp-jma-global-cache": "1779206558850",
    "kr-kma-global-cache": "9999999999999",
    "orig": "1779206511921",
    "sa-ncm-global-cache": "1779206845062",
    "wnm": {
      "conformsTo": [
        "http://wis.wmo.int/spec/wnm/1/conf/core"
      ],
      "geometry": {
        "coordinates": [
          [
            [
              -50,
              0
            ],
            [
              70,
              0
            ],
            [
              70,
              -90
            ],
            [
              -50,
              -90
            ],
            [
              -50,
              0
            ]
          ]
        ],
        "type": "Polygon"
      },
      "id": "ae2edc0d-55b3-4aef-8a2b-0843ca9c7a88",
      "links": [
        {
          "href": "https://uk-metoffice-global-deterministic-nwp-wis2.s3.amazonaws.com/data/grib2/20260519T120000Z/weather/prediction/forecast/medium-range/deterministic/global/20260519T120000Z.global.forecast.-50_-90_70_0.2026-05-20T15_00_00Z.temperature.20000.grib2",
          "length": 49192,
          "rel": "canonical",
          "type": "application/grib2"
        }
      ],
      "properties": {
        "data_id": "global/temperature/forecast/20000/T+27/20260519T120000Z.global.forecast.-50_-90_70_0.2026-05-20T15_00_00Z.temperature.20000.grib2/20260519T160148Z.1107",
        "datetime": "2026-05-19T12:00:00Z",
        "diagnostic": "Temperature",
        "integrity": {
          "method": "sha512",
          "value": "4HLqxK+YE8TbPauL3J4ij7JPFQ36xxYn4/JaTNMhHMIKy4Rby0LRms38A+uW+WmaaG5cWs9l0nSFTvBwzRqqmg=="
        },
        "metadata_id": "urn:wmo:md:uk-metoffice-wmc:weather.prediction.forecast.global",
        "model_run_hour": "12",
        "pubtime": "2026-05-19T16:01:48Z",
        "timestep": "T+27"
      },
      "type": "Feature"
    },
    "wth": "origin/a/wis2/uk-metoffice-wmc/data/core/weather/prediction/forecast/medium-range/deterministic/global"
  },
  "timestamp": "2026-05-19T16:32:27Z"
}

Should this be also part of the spec ? I was seeing this as a (nice) bonus and not a must.

[kaiwirt]

Yes, i know that your Sensor Centre does that. But i think it should be part of the Spec. At least we need someone who does that if we want to solve the problems.

Alternatively i can only think of using the Global Replay and verify if all data ids are on the cache

[golfvert]

OK. I'll add the spec for the "log file" to be provided.

[kaiwirt]

ok