Motivation
Two diagram representation formats are supported, but format is an implicit choice rather than a controlled variable. Their effects on quality, interpretability, and failure behaviour are unmeasured, so there is no basis for recommending one.
Proposal
- Make format a first-class factor in run configuration, recorded in telemetry.
- Score identical tasks under both formats and report the paired delta.
- Split the failure-cause breakdown by format — a format that is harder to emit validly is a different problem from one that is harder to emit accurately.
Done when:
Scope
- In scope: format as a configuration factor, paired comparison, format-split failure analysis.
- Out of scope: adding a third format; human-subject evaluation of interpretability.
Open questions
- Are the two formats expressively equivalent? If one cannot represent a construct the other can, part of the measured quality gap is a representation limit rather than a model limit, and that needs separating.
- Does each format need its own ground truth, or can one be mechanically converted to the other? Conversion is cheaper but may inject artefacts.
- Should format interact with strategy in the analysis, or be treated as an additive factor?
Related
Motivation
Two diagram representation formats are supported, but format is an implicit choice rather than a controlled variable. Their effects on quality, interpretability, and failure behaviour are unmeasured, so there is no basis for recommending one.
Proposal
Done when:
Scope
Open questions
Related