Add corrupted file check to pipeline

[dunnkers]

We might want to check for corrupted files, in order to identify possible pipeline congestion. Validation code:

    for i in range(len(sv.files)):
        try:
            im=sv[i]
        except Exception:
            print('rm',sv.files[i])

We could add this after caching, or when transforming. Here for example, where we first actually read the images: (if an image is corrupted an error will first show up here.)

bsc-thesis/src/transform.py

Lines 144 to 151 in 312da72

	# Convert image collections to array
	print(' Converting image collections to arrays...')
	start = time()
	gt_arr = np.array(gt)
	sv_arr = np.array(sv)
	usv_arr = np.array(usv)
	end = time()
	print(' Converted to arrays in {}'.format(timedelta(seconds=end - start)))