Promote relation-level F1 to a first-class metric

[Colinho22]

Motivation

Structural correctness of a relational diagram depends as much on relations as on entities, and relation extraction is the harder subtask. Primary scoring currently runs on entity F1, which likely compresses differences between strategies and does not measure the part of the output most likely to be wrong.

Proposal

• Compute relation-level precision, recall, and F1 through the same matching infrastructure used for entities.
• Define and document a matching rule: exact triple match versus endpoint match with relaxed labelling, and whether cardinality is scored.
• Add a combined structural score, and make the primary metric config-selectable rather than hardcoded.
• Re-score existing runs so old and new metrics are comparable on identical data.

Done when:

• Relation F1 reported for every run alongside entity F1
• Matching rule documented with worked examples of a match and a near-miss
• Existing runs re-scored with no model re-invocation

Scope

• In scope: relation scoring, matching-rule definition, combined score, config-selectable primary metric, re-scoring of stored outputs.
• Out of scope: new ground-truth data; changing ground-truth provenance; graph-edit-distance or other whole-graph similarity measures.

Open questions

• Should a relation whose endpoints are correct but whose label is wrong score as partial credit, or zero? Partial credit is more informative but complicates the F1 interpretation.
• Does a relation count as missed when one endpoint entity was itself never extracted, or is that already captured by entity recall? Risk of double-penalising the same failure.
• Weighting for the combined score: equal, or weighted toward relations given their difficulty?

Related

• Blocks Add equivalence testing (TOST) to the analysis pipeline #92 — TOST should run against the settled primary metric.