Skip to content

Can't tiled .export from DatasetClient with different shaped data #879

@evvaaaa

Description

@evvaaaa

In my client I am trying to export data

In [50]: db["a638fd91-28fc-4378-9bca-9170e35563d5"]["primary"]["data"]["pfy_bruker_mca_spectrum"]
Out[50]: <ArrayClient shape=(81, 1024) chunks=((81,), (1024,)) dtype=float64 dims=('time', 'dim_0')>

In [51]: db["a638fd91-28fc-4378-9bca-9170e35563d5"]["primary"]["data"]["pfy_bruker_mca_roi0_count"]
Out[51]: <ArrayClient shape=(81,) chunks=((81,)) dtype=float64 dims=('time',)>

into a csv.

I run

In [53]: db["a638fd91-28fc-4378-9bca-9170e35563d5"]["primary"]["data"].export("meh/meh.spec")

or

In [54]: db["a638fd91-28fc-4378-9bca-9170e35563d5"]["primary"]["data"].export("meh/meh.csv")

and get internal server errors

Scheduled retry in 0.37 seconds due to HTTPStatusError("Server error '500 Internal Server Error' for url '<....>/api/v1/container/full/a638fd91-28fc-4378-9bca-9170e35563d5/primary/data?format=spec'\nFor more information, server admin can search server logs for correlation ID 6ff861b57ceeefe5.") (attempt 1)
Scheduled retry in 0.54 seconds due to HTTPStatusError("Server error '500 Internal Server Error' for url '<....>/api/v1/container/full/a638fd91-28fc-4378-9bca-9170e35563d5/primary/data?format=spec'\nFor more information, server admin can search server logs for correlation ID 02ffcfd49f7c9d3c.") (attempt 2)

of the form

  File "/home/bluesky/.venv/lib/python3.11/site-packages/fastapi/routing.py", line 430, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bluesky/.venv/lib/python3.11/site-packages/fastapi/routing.py", line 316, in run_endpoint_function
    return await dependant.call(**values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bluesky/.venv/lib/python3.11/site-packages/tiled/server/router.py", line 1077, in get_container_full
    return await container_full(
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/bluesky/.venv/lib/python3.11/site-packages/tiled/server/router.py", line 1147, in container_full
    data = await ensure_awaitable(entry.read, fields=field)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bluesky/.venv/lib/python3.11/site-packages/tiled/utils.py", line 742, in ensure_awaitable
    return await anyio.to_thread.run_sync(functools.partial(func, **kwargs), *args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bluesky/.venv/lib/python3.11/site-packages/anyio/to_thread.py", line 61, in run_sync
    return await get_async_backend().run_sync_in_worker_thread(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bluesky/.venv/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 2525, in run_sync_in_worker_thread
    return await future
           ^^^^^^^^^^^^
  File "/home/bluesky/.venv/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 986, in run
    result = context.run(func, *args)
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bluesky/.venv/lib/python3.11/site-packages/databroker/mongo_normalized.py", line 741, in read
    columns.update(self.get_columns(keys_to_fetch, slices=None))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bluesky/.venv/lib/python3.11/site-packages/databroker/mongo_normalized.py", line 882, in get_columns
    to_stack = self._inner_get_columns(tuple(keys), min_seq_num, max_seq_num)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bluesky/.venv/lib/python3.11/site-packages/databroker/mongo_normalized.py", line 997, in _inner_get_columns
    populate_columns(tuple(scalars), min_, max_)
  File "/home/bluesky/.venv/lib/python3.11/site-packages/databroker/mongo_normalized.py", line 954, in populate_columns
    validated_column = list(
                       ^^^^^
  File "/home/bluesky/.venv/lib/python3.11/site-packages/databroker/mongo_normalized.py", line 956, in <lambda>
    lambda item: self.validate_shape(key, numpy.asarray(item), expected_shape)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bluesky/.venv/lib/python3.11/site-packages/databroker/mongo_normalized.py", line 2138, in default_validate_shape
    raise BadShapeMetadata(
databroker.mongo_normalized.BadShapeMetadata: For data key pfy_bruker_mca_roi0_count shape () does not match expected shape (1024,).

From this, I think databroker is mixing up rows with columns.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions