check replication file replication-package.ipynb
Performance comparison of various language models :label: tbl-exp-classwise
| Models | CONFERENCE | DATASET | EVALMETRIC | LICENSE | ONTOLOGY | PROGLANG | PROJECT | PUBLICATION | SOFTWARE | WORKSHOP | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| F1-P. | Mistral-7B | 29.08 | 30.85 | 6.49 | 69.50 | 45.98 | 10.13 | 33.27 | 37.39 | 10.99 | 20.48 |
| Llama3.1-8B | 62.61 | 40.07 | 34.29 | 77.33 | 33.90 | 9.93 | 49.62 | 55.84 | 29.96 | 0.00 | |
| Deepseek-Chat | 60.62 | 50.98 | 37.07 | 79.68 | 33.90 | 29.97 | 38.10 | 70.42 | 28.50 | 5.29 | |
| Gemini-2.0-Flash | 77.38 | 41.70 | 28.28 | 75.61 | 45.98 | 34.41 | 53.72 | 61.07 | 15.18 | 12.99 | |
| GPT-4o-mini | 75.48 | 43.86 | 19.76 | 87.34 | 23.14 | 18.82 | 56.23 | 63.51 | 22.20 | 20.68 | |
| SciBERT | 78.88 | 69.94 | 60.18 | 86.62 | 79.43 | 82.14 | 38.21 | 90.16 | 77.25 | 0.01 | |
| RoBERTa-Base | 82.29 | 66.04 | 53.88 | 84.76 | 0.00 | 82.72 | 47.36 | 86.99 | 69.66 | 13.27 | |
| F1-E. | Mistral-7B | 17.59 | 21.51 | 6.49 | 66.97 | 13.89 | 6.70 | 4.66 | 32.05 | 6.46 | 20.48 |
| Llama3.1-8B | 33.10 | 30.85 | 33.37 | 72.40 | 18.52 | 8.34 | 49.62 | 48.55 | 22.41 | 0.00 | |
| Deepseek-Chat | 36.68 | 45.30 | 33.94 | 77.73 | 18.52 | 27.37 | 38.10 | 64.95 | 23.12 | 0.00 | |
| Gemini-2.0-Flash | 47.63 | 38.03 | 26.97 | 71.12 | 13.89 | 31.69 | 53.72 | 58.99 | 13.48 | 5.41 | |
| GPT-4o-mini | 42.44 | 37.11 | 19.76 | 86.15 | 0.00 | 13.75 | 48.21 | 54.14 | 18.68 | 20.68 | |
| SciBERT | 58.38 | 63.94 | 44.78 | 51.87 | 73.13 | 70.99 | 34.22 | 82.12 | 72.46 | 0.00 | |
| RoBERTa-Base | 35.84 | 58.95 | 35.01 | 78.38 | 0.00 | 79.96 | 33.26 | 78.11 | 64.48 | 0.00 |