Skip to content

cildataconverter.py needs to gracefully handle invalid zip files #2

@coleslaw481

Description

@coleslaw481

Example of failure:

2017-10-09 14:23:37,515 DEBUG cildata_util.cildataconverter 25696.tif
2017-10-09 14:23:37,630 ERROR cildata_util.dbutil 25696.tif did not have a successful download. Skipping...
2017-10-09 14:23:37,630 DEBUG cildata_util.cildataconverter Making backup of /home/churas/tasks/cil_downloader/converted_images/images/25696/25696.json
2017-10-09 14:23:37,690 DEBUG cildata_util.cildataconverter Entry file name: 25696.tif and cdf file name: 25696.tif
2017-10-09 14:23:37,690 INFO cildata_util.cildataconverter Updating 25696.tif
2017-10-09 14:23:37,690 DEBUG cildata_util.cildataconverter Appending to list
2017-10-09 14:23:37,690 DEBUG cildata_util.cildataconverter Entry file name: 25696.jpg and cdf file name: 25696.tif
2017-10-09 14:23:37,691 DEBUG cildata_util.cildataconverter Entry file name: 25696.raw and cdf file name: 25696.tif
2017-10-09 14:23:37,693 DEBUG cildata_util.dbutil Writing out json file to /home/churas/tasks/cil_downloader/converted_images/images/25696/25696.json
2017-10-09 14:23:37,693 DEBUG cildata_util.cildataconverter 25696.jpg
2017-10-09 14:23:37,693 DEBUG cildata_util.cildataconverter Making backup of /home/churas/tasks/cil_downloader/converted_images/images/25696/25696.json
2017-10-09 14:23:37,696 DEBUG cildata_util.cildataconverter Entry file name: 25696.tif and cdf file name: 25696.jpg
2017-10-09 14:23:37,696 DEBUG cildata_util.cildataconverter Entry file name: 25696.jpg and cdf file name: 25696.jpg
2017-10-09 14:23:37,696 INFO cildata_util.cildataconverter Updating 25696.jpg
2017-10-09 14:23:37,696 DEBUG cildata_util.cildataconverter Appending to list
2017-10-09 14:23:37,696 DEBUG cildata_util.cildataconverter Entry file name: 25696.raw and cdf file name: 25696.jpg
2017-10-09 14:23:37,699 DEBUG cildata_util.dbutil Writing out json file to /home/churas/tasks/cil_downloader/converted_images/images/25696/25696.json
2017-10-09 14:23:37,724 DEBUG cildata_util.cildataconverter 25696.raw
2017-10-09 14:23:37,847 DEBUG cildata_util.dbutil Making copy of /home/churas/tasks/cil_downloader/converted_images/images/25696/25696.raw naming it /home/churas/tasks/cil_downloader/converted_images/images/25696/25696.zip
2017-10-09 14:26:34,972 ERROR cildata_util.cildataconverter Caught fatal exception
Traceback (most recent call last):
File "/usr/bin/cildataconverter.py", line 154, in main
return _convert_data(theargs)
File "/usr/bin/cildataconverter.py", line 107, in _convert_data
newcdfs = converter.convert(cdf, base_dir)
File "/usr/lib/python2.7/site-packages/cildata_util/dbutil.py", line 839, in convert
return self._convert_image(cdf, cdf_dir)
File "/usr/lib/python2.7/site-packages/cildata_util/dbutil.py", line 878, in _convert_image
extracted_cdfs = self._extract_image_from_zip(cdf, cdf_dir)
File "/usr/lib/python2.7/site-packages/cildata_util/dbutil.py", line 906, in _extract_image_from_zip
extracted_file = zf.extract(zentry, path=tmpdir)
File "/usr/lib64/python2.7/zipfile.py", line 1024, in extract
return self._extract_member(member, path, pwd)
File "/usr/lib64/python2.7/zipfile.py", line 1078, in _extract_member
with self.open(member, pwd=pwd) as source,
File "/usr/lib64/python2.7/zipfile.py", line 967, in open
raise BadZipfile("Bad magic number for file header")
BadZipfile: Bad magic number for file header

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions