Problem with creating DECam_SWARP2/*.in files

[SeanPoints]

in branch dev commit 4a96b1

We are getting an error using Swarp.py -all -bsub

For some fields and tiles, the SWARP2 stage (e.g., Swarp.py -bsub LMC_c42 T16), I get the following:
***Swarp: Beginning LMC_c42_T16.r.run (6 of 6)
***Swarp: Writting last portion of swarp outputs
----- SWarp 2.41.5 started on 2025-02-17 at 19:18:04 with 16 threads

Error: /home/points/data/DECam/workspace50/DECam_PREP2/LMC_c42/T16/c4d_211112_073303_ooi_r_req_S29.fits not found

This is causing Swarp to not produce mosaicked images for the filter.

In the example above, the corresponding *.in file is: LMC_c42_T16.r.in
I go to the /home/points/data/DECam/workspace50/DECam_PREP2/LMC_c42/T16/ directory

ls 073303fits
ls: cannot access '073303': No such file of directory

So, the inputs in the DECam_SWARP2 *.in files are referencing data that are not in the DECam_PREP2 directory.

A test field where it fails for me is LMC_c42 T16 on the r images.

[kslong]

Just so I can duplicate this, Sean, are you using the -use_all_data option in your build up to this point.

[SeanPoints]

Yes, using the -use_all_data -S7 -seeing_max 1.5 flags

[SeanPoints]

Just as a further aside, I think that the *.in files are being created from files that are located in DECam_PREP/LMC_c42/T16 and not from files that are in DECam_PREP2/LMC_c42/T16. If you run the entire LMC_c42/T16 file, the only SWARP2 stage that was successful (for me) was LMC_c42_T16.N662_s.fits. If I look at the number of files in
DECam_PREP/LMC_c42/T16 and the number of files in DECam_PREP2/LMC_c42/T16 I get 567 and 561 respectively.

[kslong]

OK. I have duplicated the error. I am in a position at least to try to understand it.

[kslong]

I am still not sure what the problem is, but the problem for the file begins in BackStats. The file in question has only one overlap with another file, indicated in the overlap file, and does not appear in the output xxx file that is created by Backstats. Most of the other files have multiple overlaps.

[SeanPoints]

Good to know.

[kslong]

OK, so the problem is that the one overlap failed for the reason below, and for some reason, we continued with the assumption this file should be included

Backstats: Failed with 0 non_zero < 100 pixels for files c4d_211112_071914_ooi_N673_req_S29.fits and c4d_211112_073433_ooi_N673_req_S29.fits

So I just need to understand why we did not discard this file from further consideration

[kslong]

[kslong]

The figure above shows the underlying issue. It's a two panel ds9 image on the with a locked WCS, with the entire tile image displayed. The (2) files that were causing problems do not overlap with the rest of the image. This is not really something envisaged. The reason the file was not processed was that since it did not really overlap with any other exposure, it was not included in the images that were to be background subtracted.

I am adding some warnings in BackStats when this happens and a check that a file actually exists where it is supposed to be in SwarpSetup. This will be somewhat of a bandaid. A better approach would be to record this information in one of the Tables for the 'tile".

But there is a more fundamental issue in the general case of what could happen. The assumption in kred is that all of the fields we are trying to combine form a single "island" topologically. We do not make allowances for the possibility that there could be two or more disconnected "islands". There are undoubtedly ways to identify when this is the case, but it does not seem worthwhile to pursue that at this point.

[kslong]

@SeanPoints I believe this can be closed, but you should confirm that this is the case.