Harden WAF ETL pipeline

[btylerburton]

User Story

In order to harvest WAF sources effectively and at scale, datagovteam would like to harden the current WAF ETL pipeline.

Acceptance Criteria

[ACs should be clearly demoable/verifiable whenever possible. Try specifying them using BDD.]

• GIVEN [a contextual precondition]
  [AND optionally another precondition]
  WHEN [a triggering event] happens
  THEN [a verifiable outcome]
  [AND optionally another verifiable outcome]

Background

[Any helpful contextual notes or links to artifacts/evidence, if needed]

Security Considerations (required)

[Any security concerns that might be implicated in the change. "None" is OK, just be explicit here!]

Sketch

• add record partition logic into harvesting logic repo
• benchmark and report metrics on traversal and download ( how many files vs how long it took ). total processing time.
• get number of WAF harvest sources
• consider implementing download xml inside traversal instead of separate function depending if performance impact is noticeable

[rshewitt]

noaa waf

[rshewitt]

processing reached 12 hours for the noaa waf so i stopped it ( the conclusion being...it's gonna take awhile ). I duplicated our waf test but added a new fixture with an updated url. I didn't commit anything. considering how long it was running, I didn't see the benefit of knowing exactly how much longer it would take. the bottleneck is requesting/downloading the documents. requesting the initial page, parsing it with beautifulsoup, and getting a list of all the anchors with a populated href attr took 46 seconds ( this is our waf traversal function ).

conclusion of test

• how long did the process we control take? 46 seconds
• how long did the process we don't control take? a long time

[rshewitt]

json with list of all waf urls
waf_sources.json

[rshewitt]

pausing on this. more discussion on waf needed.