- N. Ide and J. Pustejovsky, Handbook of Linguistic Annotation. Springer, 2017.
- J. Pustejovsky and A. Stubbs, Natural Language Annotation for Machine Learning: A Guide to Corpus-Building for Applications. O’Reilly Media, Inc., 2012.
- M. A. Finlayson and T. Erjavec, ‘Overview of Annotation Creation: Processes & Tools’, arXiv:1602.05753 [cs], Feb. 2016. Available: http://arxiv.org/abs/1602.05753
- Best practices: https://www.techatbloomberg.com/blog/bloombergs-global- data-cto-data-science-teams-publish-best-practices-for- data-annotation-projects/
Tried with free with educational license:
- Lightag:
- many annotators
- easy set-up
- text segements, whole text posts
- progress tracking (IAA; annotator matrices)
- guidelines upload
- annotation allocation
- Labelbox
- many annotators
- whole text labeling, hierarchical labeling possible
- comment function
- progress tracking (procentual annotator tracking, time of labeling)
- data upload, download sometimes converting errors but works put most of the times (2019)
Others:
- GATE
- Hearttex
- Label Studio
- Open NLP