CSV Import Mismatch

[simon-leech]

Technical Problem Statement - When importing a CSV, if not all rows have the same number of values in the columns, data is inserted to the wrong row.
Customer Problem Statement - As a user, when I import a CSV - I expect my data to be assigned to the correct record once imported.

Uploading a CSV with rows of different lengths should infill missing values to preserve data structure.
Uploading a CSV with all rows of the same length should preserve data structure.
Example Data
In this example, the type and name are missing from row 2. The name is missing from row 3. The type is missing from row 4.
id,type,lat,long,name
1,cafe,51.5074,-0.1278,Central Cafe
2,,51.5154,-0.0928,
3,park,51.5312,-0.1569,
4,,51.5033,-0.1195,London Eye

Fix in the framework
Issue is in : /utils/csvUpload
chunkRows is skipping missing columns
In your chunkRows function, you are iterating over the parsed fields of the row using for (const indx in fields).
splitRowIntoFields drops trailing empty columns
At the very end of your splitRowIntoFields function, you have this check:
If a row is missing values at the end (for example, it only has 1 or 3 fields instead of the expected 4), this loop finishes early. The columns that didn't get a value parsed do not get anything pushed to their arrays for that row.

Current Workaround
The current workaround ‘solution’ - is to ensure that every value in the CSV has a value, we are typically using a ‘-’ to represent null data. Then in the import function on the db, converting this back to actual null.

[dbauszus-glx]

This should be fairly easy to resolve. The utility module utils/csvUpload knows how many fields are in a dataset. The method splitRowIntoFields which parses each row and returns the fields can check whether the fields has the same length before returning. And returning nothing otherwise while logging a warning for the row with mismatched length.