StatLLM: A Dataset for Evaluating the Performance of Large Language Models in Statistical Analysis

This repository accompanies the paper: StatLLM: A Dataset for Evaluating the Performance of Large Language Models in Statistical Analysis.

Repository Structure

This repository is organized into three main directories: Statistical_Analysis_Tasks, LLM_Generated_SAS_Code, and Human_Evaluation_Scores. Each directory contains well-structured subdirectories to facilitate access to datasets, problem descriptions, and SAS code.

1. Statistical Analysis Tasks

This directory contains all essential resources required for statistical problem-solving. Summary.csv Serves as a metadata file mapping each statistical task to corresponding DataDescription, DataSet, SAS code solutions and SAS code for data reading.

• ProblemDescription/: Provides detailed statistical task statements.
• DataDescription/: Contains detailed dataset descriptions, including metadata and variable definitions.
• TaskDatasets/: Includes raw dataset files in CSV format.
• Humanverified_SAScode/: Stores manually verified SAS solutions for each problem.
• DataReading_SASCode/: Includes SAS scripts designed to load and preprocess datasets.

2. LLM-Generated SAS Code

This directory stores SAS code generated by different large language models (LLMs): GPT35, GPT4, and Llama. Each folder contains SAS scripts generated by the respective model.

• SAS_code_only/: Contains the SAS code generated from LLM.

3. Human Evaluation Scores

This directory contains the three group scores (CodeQuality_Score, CodeExecutability_Score, CodeOutput_Score) along with the Total_score for the three LLMs, evaluated across 207 statistical tasks. These scores are derived based on the SAS code found in the LLM-generated SAS Code/SAS_code_only folder. The corresponding task IDs are documented in Summary.csv.

4. NLPMetrics

This directory contains the implementation and results of NLP metric analyses presented in Section "Usage Illustrations" of the paper. The notebook NLPScoreCalculation.ipynb computes a range of NLP metrics (e.g., BLEU), with the results stored in NLPScores.csv. The notebook CorrNLPvsHumaneval.ipynb evaluates the correlation between these metric scores and human evaluation ratings, while MLmodel.ipynb develops machine learning models to predict human ratings from NLP metric scores and further examines their correlation structure.

How to use

To obtain the SAS results, users must run the corresponding DataReading_SASCode file along with either the Humanverified_SAScode or the SAS_code_only file from the LLM-generated outputs (refer to Summary.csv). For example, if solving the problem PD0007 with dataset DS0003, the user should first run DataReading_SASCode/DR0003.txt to properly import the dataset into SAS. After successfully loading the data, the user can then execute either Humanverified_SAScode/SC0007.txt (if using the manually verified solution) or LLM_Generated_SAS_Code/GPT4/SAS_code_only/sas_query7.txt (if testing the LLM-generated code).

Purpose

This repository allows users to efficiently compare human-verified SAS code with LLM-generated solutions. It supports systematic evaluation of LLM performance in statistical programming tasks, ensuring reproducibility and structured analysis.

For any inquiries or contributions, please refer to the documentation or contact the repository maintainer.