Hi all,
I noticed that the evaluation script scripts/evaluation/test_joint_object.sh uses the argument --keep_events 1.
This setting only loads M2E2 sentences containing at least one event, which means the majority of test sentences without any events are ignored. When I run the code with --keep_events 1, I obtain better performance in terms of textual event mentions than reported in the paper (P: 45.6, R: 72.0, F1: 55.8).
However, when I switch to --keep_events 0, performance drops significantly due to obvious precision issues (P: 14.4, R: 72.0, F1: 24.0).
What is the intuition behind dropping sentences without ground-truth events and which setup was used for the reported results? I’d be happy to discuss further if anyone can clarify this.
Hi all,
I noticed that the evaluation script
scripts/evaluation/test_joint_object.shuses the argument --keep_events 1.This setting only loads M2E2 sentences containing at least one event, which means the majority of test sentences without any events are ignored. When I run the code with --keep_events 1, I obtain better performance in terms of textual event mentions than reported in the paper (P: 45.6, R: 72.0, F1: 55.8).
However, when I switch to --keep_events 0, performance drops significantly due to obvious precision issues (P: 14.4, R: 72.0, F1: 24.0).
What is the intuition behind dropping sentences without ground-truth events and which setup was used for the reported results? I’d be happy to discuss further if anyone can clarify this.