Skip to content

trollflow2 falsely claims all files generated successfully #156

Description

@gerritholl

Describe the bug

I have three products in my product list. Two are successfully produced and one fails to be produced. In the logfile, trollflow2 reports: "all 2 files produced nominally". It would rather be more useful if it noted that two files were produced nominally and one failed.

To Reproduce

product_list:
  output_dir:
    /data/pytroll/OUTBOXES/generic/
  subscribe_topics:
    - /set/fci/fdss
  reader:
    - fci_l1c_nc
  mask_area: False
  delay_composites: True
  use_tmp_file: False
  staging_zone: /data/pytroll/TMP/staging
  cache_dir: /data/pytroll/TMP/cache
  resampler: null
  radius_of_influence: 60000
  scene_load_kwargs:
    upper_right_corner:
      NE
  areas:
    null:
      areaname: fulldisc
      products:

        'dwd_vis04':
          productname: dwd_vis04
          formats: &config
            - writer: geotiff
              fname_pattern: "{platform_name}-{productname}-{areaname}-{start_time:%Y%m%d_%H%M}-{end_time:%H%M}-devel.tif"
              nbits: 8
              tiled: True
              blockxsize: 512
              blockysize: 512
              compress: lzw
              predictor: 2
              fill_value: 0

        'true_color_raw':
          productname: true_color_raw
          formats: *config

        'convection':
          productname: convection
          formats: *config

workers:
  - fun: !!python/name:trollflow2.plugins.create_scene
  - fun: !!python/name:trollflow2.plugins.load_composites
  - fun: !!python/name:trollflow2.plugins.resample
  - fun: !!python/name:trollflow2.plugins.save_datasets

running with:

satpy_launcher.py -m /tmp/test-message test-pl.yaml

Expected behavior

I expect one composite to fail: the convection composite requires a mixture of 1 km and 2 km channels. I am not resampling in the example playlist. Indeed, Satpy logs a warning message as expected:

[WARNING: 2022-07-29 14:10:56 : satpy.scene] The following datasets were not created and may require resampling to be generated: DataID(name='convection')

I expect that trollflow2 does not claim that all files were produced nominally.

Actual results

trollflow2 claims that all files were produced nominally:

[WARNING: 2022-07-29 14:10:57 : trollflow2.plugins] Skipping convection: "No dataset matching 'DataQuery(name='convection')' found"
/opt/pytroll/pytroll_inst/mambaforge/envs/pytroll-py310/lib/python3.10/site-packages/dask/core.py:119: RuntimeWarning: invalid value encountered in log
  return func(*(_execute_task(a, cache) for a in args))
[DEBUG: 2022-07-29 14:11:14 : trollflow2.launcher] Cleaning up
[INFO: 2022-07-29 14:11:15 : trollflow2.launcher] All 2 files produced nominally in 0:00:35.071999

Environment Info:

  • Trollflow2 Version: 0.13.5.dev30+gb9400c3
  • Satpy Version: 0.36.1.dev192+g15dbf3db
  • Pyresample Version: 1.24.1

Additional context
Add any other context about the problem here.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions