FileNotFoundError in renamed_files plugin when check_sunlight_coverage has removed one or more products and the new satpy geoninjotiff writer is used

[gerritholl]

Describe the bug

At night, the save_datasets fails with FileNotFoundError when the renamed_files context manager tries to copy files back in a setup that is using staging_zone and check_sunlight_coverage.

I'm not sure what causes it yet. I suspect it is related to check_sunlight_coverage removing datasets but renamed_files still expecting them, as it seems to happen only after the check_sunlight_coverage has removed datasets, although that doesn't explain everything.

To Reproduce

MCVE pending.

Expected behavior

No FileNotFoundErrors.

Actual results

Example traceback with some context from trollflow2.log:

[2021-12-22 09:54:13,815 INFO     trollflow2.plugins] Not enough sunlight coverage for product 'dwd_VIS006', removed.
[2021-12-22 09:54:13,820 INFO     trollflow2.plugins] Not enough sunlight coverage for product 'dwd_VIS008', removed.
[2021-12-22 09:54:13,824 INFO     trollflow2.plugins] Not enough sunlight coverage for product 'dwd_NIR016', removed.
[2021-12-22 09:54:13,862 INFO     trollflow2.plugins] Not enough sunlight coverage for product 'overview', removed.
[2021-12-22 09:54:13,866 INFO     trollflow2.plugins] Not enough sunlight coverage for product 'night_microphysics', removed.
[2021-12-22 09:54:13,870 INFO     trollflow2.plugins] Not enough sunlight coverage for product 'day_microphysics', removed.
[2021-12-22 09:54:13,884 INFO     trollflow2.plugins] Not enough sunlight coverage for product 'green_snow', removed.
[2021-12-22 09:54:13,888 INFO     trollflow2.plugins] Not enough sunlight coverage for product 'natural_color', removed.
(...)
[2021-12-22 09:58:23,633 INFO     pyninjotiff.ninjotiff] Successfully created a NinJo tiff file: '/data/pytroll/TMP/staging/Metop-B-green_snow-nqeuro1km-20211222_0924-20211222_0947-ninjotiff.tif'
[2021-12-22 09:58:40,516 INFO     pyninjotiff.ninjotiff] Successfully created a NinJo tiff file: '/data/pytroll/TMP/staging/Metop-B-cloudtop-nqeuro1km-20211222_0924-20211222_0947-ninjotiff.tif'
[2021-12-22 09:58:51,481 INFO     pyninjotiff.ninjotiff] Successfully created a NinJo tiff file: '/data/pytroll/TMP/staging/Metop-B-day_microphysics-nqeuro1km-20211222_0924-20211222_0947-ninjotiff.tif'
[2021-12-22 09:58:52,200 INFO     pyninjotiff.ninjotiff] Successfully created a NinJo tiff file: '/data/pytroll/TMP/staging/Metop-B-overview-nqeuro1km-20211222_0924-20211222_0947-ninjotiff.tif'
[2021-12-22 09:58:54,109 INFO     pyninjotiff.ninjotiff] Successfully created a NinJo tiff file: '/data/pytroll/TMP/staging/Metop-B-night_microphysics-nqeuro1km-20211222_0924-20211222_0947-ninjotiff.tif'
[2021-12-22 09:58:58,795 INFO     pyninjotiff.ninjotiff] Successfully created a NinJo tiff file: '/data/pytroll/TMP/staging/Metop-B-natural_color-nqeuro1km-20211222_0924-20211222_0947-ninjotiff.tif'
Traceback (most recent call last):
  File "/opt/pytroll/pytroll_inst/mambaforge/envs/py39/lib/python3.9/site-packages/trollflow2/launcher.py", line 373, in process
    cwrk.pop('fun')(job, **cwrk)
  File "/opt/pytroll/pytroll_inst/mambaforge/envs/py39/lib/python3.9/site-packages/trollflow2/plugins/__init__.py", line 285, in save_datasets
    compute_writer_results(objs)
  File "/opt/pytroll/pytroll_inst/mambaforge/envs/py39/lib/python3.9/contextlib.py", line 126, in __exit__
    next(self.gen)
  File "/opt/pytroll/pytroll_inst/mambaforge/envs/py39/lib/python3.9/site-packages/trollflow2/plugins/__init__.py", line 250, in renamed_files
    os.rename(tmp_name, actual_name)
FileNotFoundError: [Errno 2] No such file or directory: '/data/pytroll/TMP/staging/Metop-B-VIS006-nqeuro1km-20211222_0924-20211222_0947-geotiff.tif' -> '/data/pytroll/OUTBOXES/ninjo/Metop-B-VIS006-nqeuro1km-20211222_0924-20211222_0947-geotiff.tif'

Oddly, it has created a ninjotiff file, but then can't find the corresponding geotiff file. This geotiff file uses the new geoninjotiff writer from Satpy. I have not seen the problem using the old geotiff or ninjotiff writers.

further instances — I don't see any in the middle of the day, although by 09:30 it should be light!

FileNotFoundError: [Errno 2] No such file or directory: '/data/pytroll/TMP/staging/Metop-C-IR037-nqeuro1km-20211221_1718-20211221_1736-geotiff.tif' -> '/data/pytroll/OUTBOXES/ninjo/Metop-C-IR037-nqeuro1km-20211221_1718-20211221_1736-geotiff.tif'
FileNotFoundError: [Errno 2] No such file or directory: '/data/pytroll/TMP/staging/Metop-B-IR037-nqeuro1km-20211221_1802-20211221_1813-geotiff.tif' -> '/data/pytroll/OUTBOXES/ninjo/Metop-B-IR037-nqeuro1km-20211221_1802-20211221_1813-geotiff.tif'
FileNotFoundError: [Errno 2] No such file or directory: '/data/pytroll/TMP/staging/NOAA-19-IR037-nqeuro1km-20211221_1834-20211221_1850-geotiff.tif' -> '/data/pytroll/OUTBOXES/ninjo/NOAA-19-IR037-nqeuro1km-20211221_1834-20211221_1850-geotiff.tif'
FileNotFoundError: [Errno 2] No such file or directory: '/data/pytroll/TMP/staging/NOAA-19-IR037-nqeuro1km-20211221_1836-20211221_1857-geotiff.tif' -> '/data/pytroll/OUTBOXES/ninjo/NOAA-19-IR037-nqeuro1km-20211221_1836-20211221_1857-geotiff.tif'
FileNotFoundError: [Errno 2] No such file or directory: '/data/pytroll/TMP/staging/Metop-C-IR037-nqeuro1km-20211221_1850-20211221_1915-geotiff.tif' -> '/data/pytroll/OUTBOXES/ninjo/Metop-C-IR037-nqeuro1km-20211221_1850-20211221_1915-geotiff.tif'
FileNotFoundError: [Errno 2] No such file or directory: '/data/pytroll/TMP/staging/Metop-B-IR037-nqeuro1km-20211221_1939-20211221_2002-geotiff.tif' -> '/data/pytroll/OUTBOXES/ninjo/Metop-B-IR037-nqeuro1km-20211221_1939-20211221_2002-geotiff.tif'
FileNotFoundError: [Errno 2] No such file or directory: '/data/pytroll/TMP/staging/NOAA-19-IR037-nqeuro1km-20211221_2019-20211221_2029-geotiff.tif' -> '/data/pytroll/OUTBOXES/ninjo/NOAA-19-IR037-nqeuro1km-20211221_2019-20211221_2029-geotiff.tif'
FileNotFoundError: [Errno 2] No such file or directory: '/data/pytroll/TMP/staging/Metop-C-IR037-nqeuro1km-20211221_2032-20211221_2054-geotiff.tif' -> '/data/pytroll/OUTBOXES/ninjo/Metop-C-IR037-nqeuro1km-20211221_2032-20211221_2054-geotiff.tif'
FileNotFoundError: [Errno 2] No such file or directory: '/data/pytroll/TMP/staging/Metop-B-IR037-nqeuro1km-20211221_2121-20211221_2142-geotiff.tif' -> '/data/pytroll/OUTBOXES/ninjo/Metop-B-IR037-nqeuro1km-20211221_2121-20211221_2142-geotiff.tif'
FileNotFoundError: [Errno 2] No such file or directory: '/data/pytroll/TMP/staging/NOAA-19-IR037-nqeuro1km-20211221_2159-20211221_2220-geotiff.tif' -> '/data/pytroll/OUTBOXES/ninjo/NOAA-19-IR037-nqeuro1km-20211221_2159-20211221_2220-geotiff.tif'
FileNotFoundError: [Errno 2] No such file or directory: '/data/pytroll/TMP/staging/NOAA-18-IR037-nqeuro1km-20211221_2226-20211221_2238-geotiff.tif' -> '/data/pytroll/OUTBOXES/ninjo/NOAA-18-IR037-nqeuro1km-20211221_2226-20211221_2238-geotiff.tif'
FileNotFoundError: [Errno 2] No such file or directory: '/data/pytroll/TMP/staging/Metop-C-IR037-nqeuro1km-20211221_2213-20211221_2235-geotiff.tif' -> '/data/pytroll/OUTBOXES/ninjo/Metop-C-IR037-nqeuro1km-20211221_2213-20211221_2235-geotiff.tif'
FileNotFoundError: [Errno 2] No such file or directory: '/data/pytroll/TMP/staging/Metop-B-IR037-nqeuro1km-20211221_2301-20211221_2323-geotiff.tif' -> '/data/pytroll/OUTBOXES/ninjo/Metop-B-IR037-nqeuro1km-20211221_2301-20211221_2323-geotiff.tif'
FileNotFoundError: [Errno 2] No such file or directory: '/data/pytroll/TMP/staging/Metop-C-IR037-nqeuro1km-20211221_2353-20211222_0016-geotiff.tif' -> '/data/pytroll/OUTBOXES/ninjo/Metop-C-IR037-nqeuro1km-20211221_2353-20211222_0016-geotiff.tif'
FileNotFoundError: [Errno 2] No such file or directory: '/data/pytroll/TMP/staging/NOAA-19-IR037-nqeuro1km-20211222_0511-20211222_0522-geotiff.tif' -> '/data/pytroll/OUTBOXES/ninjo/NOAA-19-IR037-nqeuro1km-20211222_0511-20211222_0522-geotiff.tif'
FileNotFoundError: [Errno 2] No such file or directory: '/data/pytroll/TMP/staging/NOAA-19-VIS006-nqeuro1km-20211222_0651-20211222_0707-geotiff.tif' -> '/data/pytroll/OUTBOXES/ninjo/NOAA-19-VIS006-nqeuro1km-20211222_0651-20211222_0707-geotiff.tif'
FileNotFoundError: [Errno 2] No such file or directory: '/data/pytroll/TMP/staging/NOAA-19-VIS006-nqeuro1km-20211222_0644-20211222_0706-geotiff.tif' -> '/data/pytroll/OUTBOXES/ninjo/NOAA-19-VIS006-nqeuro1km-20211222_0644-20211222_0706-geotiff.tif'
FileNotFoundError: [Errno 2] No such file or directory: '/data/pytroll/TMP/staging/Metop-B-VIS006-nqeuro1km-20211222_0924-20211222_0947-geotiff.tif' -> '/data/pytroll/OUTBOXES/ninjo/Metop-B-VIS006-nqeuro1km-20211222_0924-20211222_0947-geotiff.tif'

Environment Info:

• OS: CentOS Linux 7
• Trollflow2 Version: 0.12.1.dev2
• Satpy Version: 0.33.2.dev11+g08701f5b
• Pyresample Version: 1.22.3

Additional context

I might have broken something myself.

More investigation is needed.