Skip to content

fix: offload SDMX response parsing from the event loop - #503

Open
ypldan wants to merge 4 commits into
developmentfrom
perf/500-sdmx-parse-off-loop
Open

ypldan wants to merge 4 commits into
developmentfrom
perf/500-sdmx-parse-off-loop

Conversation

@ypldan

@ypldan ypldan commented Jul 2, 2026

Copy link
Copy Markdown

Applicable issues

Description of changes

SDMX responses were parsed synchronously on the event loop; large data messages take seconds of CPU-bound parsing and stall every concurrent request. Both parse paths now run in a worker thread via asyncio.to_thread, mirroring the existing _data_msg_to_dataframe offload in dataset.py. Threaded XML parses in AsyncSdmxClient all run with structure=None and build private structures, so they stay fully parallel; the proxy v30 data parse carries a shared DataStructureDefinition that the reader attaches to the message and mutates while parsing (msg.structure...getdefault(...)), so dsd-bearing parses are serialized with a module-level threading.Lock.

  • Run _parse_response via asyncio.to_thread in AsyncSdmxClient._fetch
  • Run the StatGptSdmxProxyDataReader convert via asyncio.to_thread in AsyncStatGptSdmxProxyClient._proxy_data
  • Add _PARSE_DSD_LOCK guarding reader.convert when a dsd is passed (no _parse_response caller currently does; the proxy data path does)
  • Add unit tests asserting a slow in-thread parse does not block a concurrent coroutine (XML client and proxy client)

Checklist

By submitting this pull request, I confirm that my contribution is made under the terms of the MIT license.

@ypldan ypldan changed the title perf: offload SDMX response parsing from the event loop fix: offload SDMX response parsing from the event loop Jul 2, 2026
@ypldan ypldan changed the title fix: offload SDMX response parsing from the event loop perf: offload SDMX response parsing from the event loop Jul 2, 2026
@ypldan ypldan changed the title perf: offload SDMX response parsing from the event loop fix: offload SDMX response parsing from the event loop Jul 2, 2026
@ypldan ypldan self-assigned this Jul 3, 2026
@ypldan
ypldan marked this pull request as ready for review July 3, 2026 06:11
@ypldan
ypldan requested a review from Fedir-Yatsenko as a code owner July 3, 2026 06:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Offload SDMX response parsing from the event loop

1 participant