Revise PDF reports creation

[matamadio]

The current pdf creation system stores can be improved.
See as example: https://int.thinkhazard.org/en/report/FRA-france/FL

• Storage: the current system keeps pdf files saved for 10 years into S3 bucket - @stufraser1 is it necessary? They don't seem to update when the user download them, i.e. old version with mismatching scores. Suggest to re-create pdf on the fly on user request.
• Figures: check basemap data: some are not printed correctly
  
• Data source: missing hazard layer description as from the page.
  
• Layout: random blank pages

[arnaud-morvan]

Hello @pzwsk

I'm still waiting for your response concerning output PDF permanent storage.

For memory, considered options are:

• Replace, in PDF file storage, data publication date by PDF resquest date, this is the more concervative option, PDF still get stored in S3 bucket, S3 bucket might grow faster, but this should not be a big issue.

• Do not store PDF anymore in persistent storage (S3 bucket), generate them for each request without storing them, this may have an impact on GFDRR responsability in case of later conflict (this is why we asked your validation).

Best regards

[stufraser1]

I've checked (using WB internal copilot) for guidelines around document retention within WB. There is no definitive guidance, or defined required retention period, or even commonly used retention period in the searched guidelines/rules.

Given how the World Bank frameworks work:
A hazard report (e.g. risk assessment, environmental or disaster report) would be treated as an operational/project record
Its retention period will be set by:

• the applicable retention schedule (not publicly detailed in the documents I found)
• the project lifecycle + audit requirements
• any legal / compliance obligations.

In practice, this usually means:

• kept at least for the duration of the project + audit window
• potentially longer if needed for accountability, litigation, or institutional memory
  But no explicit number of years is stated in the World Bank’s general public guidelines.

Technical hazard/risk assessment outputs (e.g., hazard model report, flood risk profile) are usually treated as project operational records / analytical deliverables (often tied to lending/ASA project records). Your own email thread shows technical deliverables and data-management concerns (storage, licensing, delivery), but again not retention schedule categories.

The conclusion is that Records must be kept “as long as required” based on legal, operational, and audit needs.

I would therefore say we DO need to retain the PDFs to support potential legal, operational, and audit needs. Ten years is probably a conservative duration. It could be shortened, but I have no basis with which to suggest a specific number of years, so recommend until we can understnad a minimum duration, we keep it at 10.

We can ask legal team for more specific guidance on document retention, but if we're talking 100 of MBs overall, then it may not be worth it. What is the size of PDF, and how many have / what total size have we stored so far?

[matamadio]

I wonder whether other WB tools apply this data retention policy.
Also wondering, wouldn't be the responsability of the user/project who is getting the data from thinkhazard to save a copy of report generated at the time of inspection?

Anyway, the printing of pdf from page should not overlap with historical archive:

• "Download pdf" should always return the same content shown by the page.
• Retrieval of old prints from S3 should go through backup access request

IMHO

[stufraser1]

I wonder whether other WB tools apply this data retention policy.

I also don't know.

wouldn't it be the responsibility of the user/project who is getting the data from thinkhazard to save a copy of report generated at the time of inspection?

It could be - and TH is generally one piece of information used in project preparation, not the sole piece of information. My experience is that people are using the information through the Operations Portal anyway, not direct where you can download PDF.

"Download pdf" should always return the same content shown by the page.

Absolutely, yes.

I don't know of a single case where someone has had to justify a decision based on information from TH, and needed to retrieve past information from a pdf (we've not had a request for an old pdf), so I acknowledge the view to keep the PDFs is a conservative one.

Considering the above a bit more: IF we have a strong record of hazard levels available for every hazard and ADM unit, and can demonstrate the period of time in which those values were valid / timestamp of when they change, then that should be enough of a backup if someone wanted to retrive historical hazard levels. In that case, I think we already justify / meet the expectation that Records must be kept “as long as required” based on legal, operational, and audit needs, without storing PDFs.

[matamadio]

Btw we could also simply comply with that by having old scores hosted at the TH page in DDH.

[stufraser1]

Lets host only the latest version available there, to avoid the risk someone takes and uses an old version by mistake. Metadata can include a comment though, that previous versions of rankings are available on request.

[matamadio]

We briefly discussed the item now, and considering no recorded requests of access to legacy scores since tool publication we think it is safe enough to disable the storage of pdf completely.
We can still provide the legacy scores in form of excel table if someone needs them.

[arnaud-morvan]

Ok so I proposed a pull request to completely remove S3 storage for PDF files: #1063

[matamadio]

Approved-Merged #1063, closing.