206 optimize preprocess cdc function for speed

[truenomad]

In this PR I managed to improve the speed of preprocess_cdc() by applying several refactorings. Most importantly, I moved st_make_valid() out of Step 2 and into process_spatial(), so validity checks only run once during preprocessing of new shapefiles.

As a result, the function now runs in just under 10, this for global POLIS data. The remaining bottlenecks are in Steps 2 and 5, where we load, save, and bind the large AFP datasets. These steps should speed up considerably if we decide to drop unnecessary raw columns and convert strings to factors, as we discussed recently @mcuadera.

I also cleaned up console output so that preprocess_cdc() produces concise, informative CLI messages rather than verbose logs or warnings.

Closes #206

[mcuadera]

Hey Mo, added my feedback. For f.pre.st.sample.01(), we really need to make sure that we have the ability to filter for shapes that we were unable to make valid. This means modifying process_spatial() so that the outputs include a logical column indicating whether we were successful in making a shape valid. Maybe we can this "is_valid"

[mcuadera]

Another idea to increase the speed is to write R data objects in parallel. Need to look at points in preprocessing where we're sequentially writing to disk or to EDAV, and then using future.

[truenomad]

Another idea to increase the speed is to write R data objects in parallel. Need to look at points in preprocessing where we're sequentially writing to disk or to EDAV, and then using future.

Yes, I think this can work. I haven't done this before, so look to your guidance.

Also, using qs2 and dropping already cleaned cols will make the key difference!

[mcuadera]

Another idea to increase the speed is to write R data objects in parallel. Need to look at points in preprocessing where we're sequentially writing to disk or to EDAV, and then using future.

Yes, I think this can work. I haven't done this before, so look to your guidance.

Also, using qs2 and dropping already cleaned cols will make the key difference!

Indeed, and qs2 has the rds_to_qs() function as well that we could potentially use. Thinking about when we pull data fom POLIS, which come as RDS as of now (maybe there should be an output type param for that as well)

[truenomad]

@mcuadera actually, looking into this more closely, it might be easier to simply specify the output_format directly within get_polis_data, since importing raw data contributes to preprocessing speed. We’ve already done the heavy lifting, tidypolis_io can read and write qs2 files. We just need to make sure output_format is specified in the function, which should be straightforward.

Let me try that now!

Update I have been able to update get_polis_data as well s1_prep_polis_tables. In s1_prep_polis_tables I have separate output_format from raw_output_format so that we the saving format and raw polis data format are different. I hope this is within scope!

[mcuadera]

oof...tried to merge dev into this branch and encountered too many conflicts. I think I will be cherry picking from this branch into a newer branch just to make sure that I don't inadvertently do a bad merge