Alter the Bulk Data Service to make the uploading of ZIPs atomic (to solve the corrupted ZIP downloads issue)

[simon-20]

Brief Description
It has happened twice since the Dashboard work began. One occurrence was on Oct 9th around 21:13.

The unzip program issues the following warning:

Archive:  iati-zip-bad.zip
   creating: iati-data/datasets/
  inflating: iati-data/datasets-full.json  
error: invalid zip file with overlapped components (possible zip bomb)

If you force unzip to unzip it anyway, then the XML files (which should be in subdirectories according to their reporting_org are all extracted into the root of the ZIP, and lots of them are zero length.

Severity
Medium

[Bjwebb]

I was seeing this issue again, on the 30th, after the fix was merged.

The zip file was deleted as part of the following days updates, so I don't have it to share. I did notice this in the docs though:

2025-10-30 22:00:14 (78.6 MB/s) - Connection closed at byte 62914560. Retrying.

--2025-10-30 22:00:15--  (try: 2)  https://dev-bulk-data.iatistandard.org/iati-data.zip
Connecting to dev-bulk-data.iatistandard.org (dev-bulk-data.iatistandard.org)|13.107.246.64|:443... connected.
HTTP request sent, awaiting response... 206 Partial Content
Length: 797399220 (760M), 734484660 (700M) remaining [zip]
Saving to: ‘iati-data.zip’

        [ skipping 61400K ]

Which suggests wget is maybe merging 2 different zip files, if it tries to download whilst the zip is updated. I hadn't previously looked into this, as I'd assumed the zip file would be more broken in this case.

[simon-20]

@Bjwebb ,

Here are the BDS dev logs for that period:

2025-10-30T21:53:21,006Z - bds - INFO - dataset id: 7b221cc0-3708-45b9-8f47-2d3388b98940 - Downloading	
2025-10-30T21:55:08,833Z - bds - INFO - Zipping 12309 datasets.	
2025-10-30T21:58:12,754Z - bds - INFO - Verifying ZIP file /tmp/bulk-data-service-zip-1/iati-data.zip	
2025-10-30T21:59:53,779Z - bds - INFO - Uploading Bulk Data Service ZIP to Azure with filename: iati-data.zip.	
2025-10-30T22:02:38,876Z - bds - INFO - Zipping 13191 datasets.	
2025-10-30T22:05:52,881Z - bds - INFO - Verifying ZIP file /tmp/bulk-data-service-zip-2/code-for-iati-data-download.zip	
2025-10-30T22:07:38,726Z - bds - INFO - Uploading Code for IATI ZIP to Azure with filename: code-for-iati-data-download.zip.	
2025-10-30T22:08:02,703Z - bds - INFO - Zipper run finished in 0:14:44.099360.	
2025-10-30T22:08:02,708Z - bds - INFO - Pausing for 15 mins	
2025-10-30T22:23:02,709Z - bds - INFO - Checker starting run  

So the time that the Dashboard downloaded the ZIP was a time which the BDS was uploading a new ZIP file to Azure.

Azure's blob storage upload is atomic.

The cause of this issue is our use of Azure Front Door CDN (we were forced to use a CDN by Azure because Azure blob storage can't be served from a custom domain unless you use a CDN).

I guess Front Door maybe drops the connection if the file is overwritten during a download, then wget starts a partial download, which grabs part of a different file.

The proper way to deal with this is to setup the BDS ZIP URLs as redirects which point to a versioned copies of the ZIP files, with the redirects being updated each time the file is re-uploaded.

Azure recommend doing this with Front Door plus an Azure serverless Functions app, to read the latest version and then stream the download from blob storage. But if Azure Functions can be allocated to custom domains, we could just get rid of Front Door entirely - we only used it because blob storage can't use custom domains over https without using the Azure CDN.

[Bjwebb]

On the dashboard, I'm currently retrying the download if the unzip fails https://github.com/IATI/deploy/commit/fae7a1d606c591a78cd3b01562622a742f708cf5