Skip to content

README.md

Latest commit

 

History

History
63 lines (47 loc) · 3.23 KB

README.md

File metadata and controls

63 lines (47 loc) · 3.23 KB
Raw

CORE Data

Directory Structure

data/
├── train.parquet              # RL training set in verl format (1,110 concept quizzes)
├── val.parquet                # Validation set in verl format
└── quizzes/
    └── concept_quizzes.jsonl          # 1,110 validated concept quizzes (with concept text)

Note on Textbook Data

The concept definitions, illustrative examples, and exercises in our paper are curated from Advanced Algebra (3rd Edition, Yao & Xie, Fudan University Press, 2015). Due to copyright restrictions on the source textbook, we do not release these materials separately. We release only the AI-generated concept quizzes, which include associated concept text as part of the quiz context. Please refer to Section 3.2 of the paper for details on the textbook corpus.

For users who wish to reproduce the SFT baseline, the released quiz data (with embedded concept text) is sufficient.

Data Format

Training Data (train.parquet, val.parquet)

Parquet files for verl RL training, with columns:

  • data_source: Source identifier (math_mcq)
  • prompt: Formatted prompt as a list of message dicts (system + user)
  • ability: Task type (math)
  • reward_model: Ground truth and evaluation configuration

Concept Quizzes (quizzes/concept_quizzes.jsonl)

The main training dataset — 1,110 concept-aligned quizzes generated by Qwen2.5-72B-Instruct and validated by GPT-4o. Each line is a JSON object:

{
  "concept_title": "Definition of Determinant: Minor",
  "question_id": 1,
  "original_question": "When calculating the determinant of a matrix, ...",
  "enhanced_question": "Related Concept: ... Question: ...",
  "question": "<duplicate of enhanced_question (legacy field)>",
  "options": ["A. ...", "B. ...", "C. ...", "D. ..."],
  "answer": "B. ...",
  "explanation": "The cofactor is the minor with a sign change ...",
  "tags": ["<topic tag>", "..."],
  "related_concept": "<concept definition text used for CORE-CR/KL>"
}

Key fields:

  • original_question: The quiz question without concept context
  • enhanced_question: The quiz question with concept title prepended (used for concept injection in CORE-CR/KL); question is a legacy duplicate of this field
  • answer: The correct option — the letter, usually followed by the option text (e.g., "B. 2")
  • tags: Topic tags for the quiz
  • related_concept: The full concept definition text (used during concept-guided rollouts)
  • explanation: Step-by-step solution reasoning

Robust Evaluation

The Robust Evaluation protocol (Table 1 in the paper) generates three option-permuted variants of each question; a model is considered to have robustly solved a problem only if it correctly answers the original question and all permuted variants. The permutation and scoring scripts are not part of this release — see Section 3.3.2 of the paper for the protocol details.

Data Provenance

  • Concept quizzes: Generated by Qwen2.5-72B-Instruct, validated by GPT-4o (6-dimension evaluation). See Section 3.3.2 of the paper.
  • Quiz generation and validation prompts: See scripts/data/README.md for the complete prompt templates.