Explanation of problem: Since we receive some permit files in .xlsx or .csv format, data values have the potential to exceed our FWF limits. For example, we received "NEW PROVIDENCE" in the zip field, which has a length of 14 despite the limit being 10 for a correct zip code, e.g. 20708-4011. This error is captured by proof() but persists to the data being written out to the processed data folder. Values that are too long cause errors when loading to the database.
Change needed: Stop function and return message for values that exceed FWF field length limits. Manually find and fix values before writing the output. Field lengths are defined in read_hip() here:
|
fwf_widths(c(1, 15, 1, 20, 3, 60, 20, 2, 10, 10, 10, |
... but it may be beneficial to move these to the constants.R file now.
Explanation of problem: Since we receive some permit files in
.xlsxor.csvformat, data values have the potential to exceed our FWF limits. For example, we received"NEW PROVIDENCE"in thezipfield, which has a length of14despite the limit being10for a correct zip code, e.g.20708-4011. This error is captured byproof()but persists to the data being written out to the processed data folder. Values that are too long cause errors when loading to the database.Change needed: Stop function and return message for values that exceed FWF field length limits. Manually find and fix values before writing the output. Field lengths are defined in
read_hip()here:migbirdHIP/R/read_hip.R
Line 107 in 9165b80
... but it may be beneficial to move these to the
constants.Rfile now.