| Agreement Measurement | Agreement Catastrophy | Annotation Errors | Agreement Statistics | in-the-loop Machine Learning Annotation/ Agreement |
|---|---|---|---|---|
| Inter-Annotator-Agreement | Task complexity | Ambiguity (data) | Subsample (Overlaps) | Uncertainity: Confidence, judgement distributions, annotator confusion matrices |
| Intra-Annotator-Agreement | Task subjectivity | Variation (data, across documents) | Whole dataset | Active Learning |
| Whole dataset (ReproGen) | label type | Uncertainity (annotators' lack of knowledge) | ||
| Data type | Error (annotations) | |||
| Annotators (variable mostly considered in research) | Bias (comeplete annotation scheme) | |||
| Training | ||||
| Measurement |