Skip to content

feat: parallelize FDSN provider station queries per box - #16

Merged
jsteinberg34 merged 5 commits into
mainfrom
parallelize-station-queries
Jul 24, 2026
Merged

feat: parallelize FDSN provider station queries per box#16
jsteinberg34 merged 5 commits into
mainfrom
parallelize-station-queries

Conversation

@jsteinberg34

Copy link
Copy Markdown
Owner

Summary

  • Replaces the sequential for provider in providers loop in download_boxes() with a ThreadPoolExecutor so all FDSN provider station queries for a given box fire concurrently
  • Extracts _query_provider as a module-level worker function
  • Bumps version to 0.2.1 and checks off the roadmap item in README

Background

For each download box, the old code queried each FDSN provider (EARTHSCOPE, SCEDC, etc.) sequentially with blocking client.get_stations() HTTP calls. With N providers, the station-query phase cost N × latency per box — pure network idle time, since providers are completely independent servers with no data dependency between them.

The new approach fires all queries concurrently, reducing per-box station-query time from N × latency to ~latency. MassDownloader (which dominates total runtime) is untouched — it already parallelises internally and runs after all queries complete.

Implementation notes

  • max_workers=max(1, len(clients)) guards against the edge case where all providers fail to initialise and clients is empty — ThreadPoolExecutor(max_workers=0) raises ValueError
  • Each provider uses its own Client instance initialised before the box loop, so there is no shared mutable state between threads
  • Each worker writes to a unique path ({provider_name}_stations.xml), so no file contention
  • Error handling is preserved: FDSNNoDataException → skip silently, any other exception → log and skip

Test plan

  • 126/126 existing tests pass with no changes
  • test_download_boxes_tolerates_get_stations_errors[nodata]FDSNNoDataException path
  • test_download_boxes_tolerates_get_stations_errors[generic] — generic exception path
  • test_download_boxes_tolerates_client_init_failure — empty clients dict edge case

@jsteinberg34 jsteinberg34 self-assigned this Jul 24, 2026
@jsteinberg34 jsteinberg34 added the enhancement New feature or request label Jul 24, 2026
@codecov

codecov Bot commented Jul 24, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 72.72727% with 6 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
src/groundtrack/download.py 71.42% 3 Missing and 3 partials ⚠️

📢 Thoughts on this report? Let us know!

@jsteinberg34
jsteinberg34 merged commit 63a9737 into main Jul 24, 2026
4 of 5 checks passed
@jsteinberg34
jsteinberg34 deleted the parallelize-station-queries branch July 24, 2026 04:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant