Skip to content

Fix annotation issues in complete_lsat_data/test_ar.json#4

Open
zyni2001 wants to merge 1 commit into
zhongwanjun:mainfrom
zyni2001:fix/ar-lsat-test-ar-labels
Open

Fix annotation issues in complete_lsat_data/test_ar.json#4
zyni2001 wants to merge 1 commit into
zhongwanjun:mainfrom
zyni2001:fix/ar-lsat-test-ar-labels

Conversation

@zyni2001

@zyni2001 zyni2001 commented Mar 6, 2026

Copy link
Copy Markdown

This PR fixes a small set of annotation issues in complete_lsat_data/test_ar.json.

Changes:

  • remove 201510_3-G_3_19
    • reason: the question/options do not match the context
  • relabel 8 items:
    • 201310_3-G_3_14: 3 -> 2
    • 201612_3-G_1_3: 1 -> 3
    • 201612_3-G_1_4: 0 -> 4
    • 201612_3-G_1_5: 4 -> 1
    • 201612_3-G_2_6: 2 -> 3
    • 201612_3-G_2_8: 0 -> 1
    • 201612_3-G_3_12: 2 -> 0
    • 201612_3-G_3_17: 3 -> 4

These fixes come from manual verification while auditing shared failure cases on the AR-LSAT test set.

Impact:

  • original test set size: 230
  • corrected test set size: 229

Note: complete_lsat_data/test_ar.json is stored upstream as a single-line minified JSON file, so GitHub shows the patch as one replaced line even though the intended content changes are only 8 relabels and 1 removed sample.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant