When an attribute is missing, error message does not always say which one

[corviday]

While indexing a file:

2018-11-16 13:20:15 INFO: Processing file: /storage/data/projects/comp_support/climate_explorer_data_prep/climatological_means/return_periods/pr_RP20_BCCAQ2_PRISM_CanESM2_rcp85_r1i1p1_2041-2070.nc
2018-11-16 13:20:15 ERROR: Traceback (most recent call last):
  File "/home/lzeman/Code/modelmeta-newdb/modelmeta/mm_cataloguer/index_netcdf.py", line 1354, in index_netcdf_file
    data_file = find_update_or_insert_cf_file(session, cf)
  File "/home/lzeman/Code/modelmeta-newdb/modelmeta/mm_cataloguer/index_netcdf.py", line 1244, in find_update_or_insert_cf_file
    find_data_file_by_id_hash_filename(sesh, cf)
  File "/home/lzeman/Code/modelmeta-newdb/modelmeta/mm_cataloguer/index_netcdf.py", line 1085, in find_data_file_by_id_hash_filename
    q = sesh.query(DataFile).filter(DataFile.unique_id == cf.unique_id)
  File "netCDF4/_netCDF4.pyx", line 2450, in netCDF4._netCDF4.Dataset.__getattr__
  File "netCDF4/_netCDF4.pyx", line 2396, in netCDF4._netCDF4.Dataset.getncattr
  File "netCDF4/_netCDF4.pyx", line 1110, in netCDF4._netCDF4._get_att
  File "netCDF4/_netCDF4.pyx", line 1556, in netCDF4._netCDF4._ensure_nc_success
AttributeError: NetCDF: Attribute not found

Error seems to be somewhere in the unique_id generation, but it's very difficult to trace. An error message that just told me which attribute was missing without me having to dig through code and make wild guesses would be nice.

[corviday]

Relatedly: sometimes an error says ensemble_member is missing instead of initialization_method, physics_version, or realization. A little more specificity might be helpful.

[nikola-rados]

An example of this issue:

2019-02-21 15:55:25 INFO: DRY RUN
2019-02-21 15:55:25 INFO: 
2019-02-21 15:55:25 INFO: File: ../../data/tiny_test_data.nc
2019-02-21 15:55:25 INFO: climo_periods: dict_keys(['6190'])
2019-02-21 15:55:25 INFO: project: CMIP5
2019-02-21 15:55:25 INFO: institution: PCIC
2019-02-21 15:55:25 INFO: model: DEMO
2019-02-21 15:55:25 INFO: emissions: DEMO
2019-02-21 15:55:25 INFO: run: CFAttributeError: Expected file to contain attribute 'ensemble_member', but no such attribute exists
2019-02-21 15:55:25 INFO: dependent_varnames: ['tasmax']
2019-02-21 15:55:25 INFO: time_resolution: daily
2019-02-21 15:55:25 INFO: is_multi_year_mean: False

[rod-glover]

@corviday and @nikola-rados you are quite right, and I put (apparently inadequate) effort into catching these errors (from netCDF4) and turning them into something informative.

[corviday]

Most missing attribute errors are caught. I think all individual attributes have good error messages. I think there are just a couple of weird cases around the composite attributes like ensemble_member and cmor_file_name. I haven't had time to look into it in any detail - this is sometimes an annoyance, but does not stop the script from functioning - but I think it's really just composites.