Pandas fails to read some run sheets

[tashrifbillah]

In rc-predict-lochness:

AD00051.Prescient.Run_sheet_eeg_1.csv reading is successful but AD00067.Prescient.Run_sheet_eeg_1.csv fails

File ~/miniconda3/lib/python3.8/site-packages/pandas/_libs/parsers.pyx:805, in pandas._libs.parsers.TextReader.read_low_memory()

File ~/miniconda3/lib/python3.8/site-packages/pandas/_libs/parsers.pyx:861, in pandas._libs.parsers.TextReader._read_rows()

File ~/miniconda3/lib/python3.8/site-packages/pandas/_libs/parsers.pyx:847, in pandas._libs.parsers.TextReader._tokenize_rows()

File ~/miniconda3/lib/python3.8/site-packages/pandas/_libs/parsers.pyx:1960, in pandas._libs.parsers.raise_parser_error()

ParserError: Error tokenizing data. C error: out of memory

[tashrifbillah]

Solved by on_bad_lines='skip':

eeg-qc-dash/app.py

Line 300 in 8de159b

run_sheet_df= pd.read_csv(r, on_bad_lines='skip')

[tashrifbillah]

Another solution is to use:
https://github.com/AMP-SCZ/utility/blob/eccb9f842641f49739011d082c7e960dc02918e1/gen_dict_diff.py#L10-L14

pd.read_csv(sys.argv[2], encoding='ISO-8859-1')