Guidelines for graph rankings

[fjjulien]

Here are a few considerations for assigning a ranking to a given graph, for the purpose of syncing data to Artsdata's core graph or determining which data source is passed on to a data consumer:

Primary vs secondary source

• Primary source: the data was generated by the party themselves or their official representative (example: I Want To Showcase, an agents’ artists directory, a presenter’s website).
• Secondary source: the data was generated by a third party not directly related to the entity or authorized to act on their behalf (examples: Arts Touring Connector, an artist listing their tour performances [the presenter is the authority for event data], a cultural calendar).
  • Note: A given graph may be a primary source for a given class of entities and a secondary source for other classes of entities (see "Differentiated ranking", below)

Presence of an open data policy and consent

If the user consented to release the data as open data, this implies an understanding that the data will be reused, and therefore an assumption that the user exercised greater care in data population than if they were simply populating one-time disposable data.

Presence of data quality processes

• Does the source have automated data validation processes in place (for example, does it check that a URL resolves or does it accept any text string)?
• Does the source have a dedicated person responsible for manually checking and validating data entries?
• Does the source have error reporting mechanisms?

Differentiated ranking by class or by property

Ideally, a system should consider each source not as simple, monolithic artifact, but as a complex artifact with multiple facets. Indeed, a given source may have a higher level of authority for entities of a given class or a for a given property.

• For example, the CAPACOA membership directory is a very reliable and granular source of information on type Organization entities. However, it is a slightly less reliable source for Person and Place entities.
• For example, a ticketing service should be ranked higher on offers data than the official organizer's website. Or a given calendar might have more exhaustive data about nested performer entities.

If you think of additional criterias or considerations, please add them below.

[fjjulien]

Ideas for class-based ranking

Granularity of the data model

• How many mapped properties does the source have for this given class?
  • A higher level of granularity should be preferred for performance purposes, and also because it is an indication that the class is a core class in the source's data model.
  • In the case of event entities, rather than simply counting the number of properties, we should use the structured data score.
  • Question: Should a property that cannot be mapped to the Artsdata data model or whose values cannot be mapped to a controlled vocabulary be counted for ranking purpose?

Interoperability of the data model

• In addition to granularity, we should consider to what extent the source's data model follow domain standards.
  • Question: If we rank the "granularity" criteria based on the number of mapped properties, the mapping process already involves an assessment of the data model's interoperability. Do we need to assess interoperability as a distinct ranking criteria?

Data quality

• Are there constraints violations or other known data quality issues (or asked in a positive way: Is the dataset exempt of constraints violations and known data quality issues)?

[dlh28]

I don't think it makes sense to rank a property that cannot be mapped to the Artsdata model. If we included the property, it would assign a source a higher overall rank to the source, but the data would have the exact same quality in the knowledge graph as data from another source with the exact same properties that "show up" in Artsdata, but with a lower overall rank because they lack the properties that aren't even present in our model.

I also don't think we need to assess interoperability as a distinct criteria. If it's not interoperable, then a default score of zero would suffice.

[fjjulien]

Thank you, Dessa.
As I read your response, I wonder why I even asked the questions in the first place. If a property isn't interoperable or can't be mapped, it won't be loaded to Artsdata, and it will therefore not taken into account in the computing of the ranking score.

[dlh28]

In terms of secondary source data, we could also consider whether the data was first entered by the presenter (through a smart form on the third party's website, for example, or through some sort of automated data flow from the presenter's website) or whether it was completely entered manually or scraped by the third party without any input at all from the presenter.

On a more micro level, we could also consider whether or not the source yields consistent errors across their data, indicating a systemic problem (ex: the wrong timezone value in their event structured data; automatically assigning all artists in a directory @ type Person when the directory contains a mixture of solo artists, groups and companies).

[fjjulien]

In terms of secondary source data, we could also consider whether the data was first entered by the presenter (through a smart form on the third party's website, for example, or through some sort of automated data flow from the presenter's website) or whether it was completely entered manually or scraped by the third party without any input at all from the presenter.

I agree. That's what I attempted to allude to with the "Primary vs secondary source" criteria. But you bring an important nuance: in the case of a website scraped by a third-party, someone ranking the graph could interpret that the data is "generated by the party themselves", but the scraped data is in fact a copy of the primary source data, so it is in fact a secondary source. We'll have to clarify this in the guidelines.

[fjjulien]

Further to the previous comment, the notion of data recency could also be considered. One could trust that an official presenter's site has the most up-to-date event information, while a listing site scraping event data could have outdated data.

Is data recency always correlated with the primary/secondary nature of the source? If so, then it doesn't need to be assessed as a distinct criteria.

[fjjulien]

On a more micro level, we could also consider whether or not the source yields consistent errors across their data, indicating a systemic problem (ex: the wrong timezone value in their event structured data; automatically assigning all artists in a directory @ type Person when the directory contains a mixture of solo artists, groups and companies).

I agree.

That's what I was trying to get to with my question "Are there constraints violations or other known data quality issues?". A data source known to have a systemic problem should be ranked accordingly.

Should we rank this on a positive scale or should we allow for negative scoring? We'll have to consider this when we define the scales and the algorthimic formula for computing the score.

[fjjulien]

A draft ranking algorithm is described in this spreadsheet and this companion document.