Perc&Act Sofia and Sissel Github repository for their exam in Perception and Action. Data files and scripts are saved in their own folders, where we call upon them throughout. This repository is structured in the steps of our workflow
In this step, we combine ExtractedFeatures and participant_stats_csv from Data/Orignial, creating combined_data.csv. We then subtract the most irrelevant features, compute binary demographic variables, and compute word length and frequency scores. The final output in Data/Original is data_final_full-csv
data_final_full_csv is used to to explore possibilities for different age group splits, levels of aggregation (word/sentence/speech), and down/upsampling methods. Outputs in the folders Data/Age_group_splits and Data/Balanced
csv file for sentence-level is used to explore features further. We extract important features and compute interactions using RandomForests and SHAP, and testing the models throughout on a SVM. Top 10 interactions are saved in Data/Original/Post_Feature_Engineering/Top10Interactions. We find the final dataframes for the models, resulting in different X_train and X_test matrices in Data/Original/Post_Feature_Engineering/X_y_values/X_train_test. Additionally, we save test_data with all information (SubjectID and age) in df_test_all_dropped_columns so we're able to explore misclassified participants in the future! One y_train and y_test is saved in Data/Original/Post_Feature_Engineering/X_y_values/y_vectors. Thus, all the data is ready for...