Deprecated distutils import from upstream dependency in v1.2.5

[ravngr]

Expected Behavior

The bluesky tutorial should be completeable on an up-to-date install of Python 3.12.

Current Behavior

Loading the databroker module raises a ModuleNotFoundError due to an import of the distutils in an upstream dependency intake.

Full Traceback

---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
Cell In[3], line 2
      1 from databroker import Broker
----> 2 db = Broker.named('temp')
      4 # Insert all metadata/data captured into db.
      5 RE.subscribe(db.insert)

File c:\\Users\\User\\repo\\.venv\\Lib\\site-packages\\databroker\\v1.py:256, in Broker.named(cls, name, auto_register)
    225 \"\"\"
    226 Create a new Broker instance using a configuration file with this name.
    227 
   (...)
    253 db : Broker
    254 \"\"\"
    255 if name == 'temp':
--> 256     return temp()
    257 else:
    258     try:

File c:\\Users\\User\\repo\\.venv\\Lib\\site-packages\\databroker\\v1.py:39, in temp()
     38 def temp():
---> 39     from .v2 import temp
     40     catalog = temp()
     41     return Broker(catalog)

File c:\\Users\\User\\repo\\.venv\\Lib\\site-packages\\databroker\\v2.py:5
      2 import importlib
      3 import tempfile
----> 5 from .core import parse_handler_registry, discover_handlers, parse_transforms
      6 from intake.catalog import Catalog
      7 from event_model import DuplicateHandler

File c:\\Users\\User\\repo\\.venv\\Lib\\site-packages\\databroker\\core.py:17
     15 import intake.catalog.base
     16 import intake.catalog.local
---> 17 import intake.container.base
     18 from intake.compat import unpack_kwargs
     19 import msgpack

File c:\\Users\\User\\repo\\.venv\\Lib\\site-packages\\intake\\container\\__init__.py:8
      1 #-----------------------------------------------------------------------------
      2 # Copyright (c) 2012 - 2018, Anaconda, Inc. and Intake contributors
      3 # All rights reserved.
      4 #
      5 # The full license is in the LICENSE file, distributed with this software.
      6 #-----------------------------------------------------------------------------
----> 8 from .dataframe import RemoteDataFrame
      9 from .ndarray import RemoteArray
     10 from .semistructured import RemoteSequenceSource

File c:\\Users\\User\\repo\\.venv\\Lib\\site-packages\\intake\\container\\dataframe.py:7
      1 #-----------------------------------------------------------------------------
      2 # Copyright (c) 2012 - 2018, Anaconda, Inc. and Intake contributors
      3 # All rights reserved.
      4 #
      5 # The full license is in the LICENSE file, distributed with this software.
      6 #-----------------------------------------------------------------------------
----> 7 from distutils.version import LooseVersion
     10 from intake.source.base import Schema, DataSource
     11 from .base import RemoteSource, get_partition

ModuleNotFoundError: No module named 'distutils'

Possible Solution

• Call out incompatibility with Python 3.12 in tutorial/documentation. I'll likely roll-back to 3.11 for the time being.
• Update dependency in v1.2.5 to intake >= 0.6.7
• Wait for databroker v2.

Steps to Reproduce (for bugs)

1. Install Python 3.12.
2. Follow the bluesky tutorial to the Prepare Data Storage step.

from databroker import Broker
db = Broker.named('temp')

Context

databroker v1.2.5 requires intake >= 0.5.5, <= 0.6.4 though this appears to have been removed in the v2 betas (sorry not familiar with the current build system so wasn't able to check). Dependency on distutils is removed in intake >= 0.6.7 (before, after). I haven't tested a build of the v1 branch with an updated dependency.

Your Environment

Python 3.12 virtual environment on Windows 11 with bluesky and other packages installed via pip.

pip freeze

anyio==4.2.0
appdirs==1.4.4
argon2-cffi==23.1.0
argon2-cffi-bindings==21.2.0
arrow==1.3.0
asciitree==0.3.3
asttokens==2.4.1
async-lru==2.0.4
attrs==23.2.0
Babel==2.14.0
beautifulsoup4==4.12.2
bleach==6.1.0
bluesky==1.12.0
bluesky-live==0.0.8
boltons==23.1.1
cachetools==5.3.2
certifi==2023.11.17
cffi==1.16.0
charset-normalizer==3.3.2
click==8.1.7
cloudpickle==3.0.0
colorama==0.4.6
comm==0.2.1
contourpy==1.2.0
cycler==0.12.1
dask==2023.12.1
databroker==1.2.5
debugpy==1.8.0
decorator==5.1.1
defusedxml==0.7.1
dnspython==2.4.2
doct==1.1.0
entrypoints==0.4
event-model==1.19.9
executing==2.0.1
fasteners==0.19
fastjsonschema==2.19.1
fonttools==4.47.0
fqdn==1.5.1
fsspec==2023.12.2
HeapDict==1.0.1
historydict==1.2.6
humanize==4.9.0
idna==3.6
imageio==2.33.1
importlib-metadata==7.0.1
importlib-resources==6.1.1
intake==0.6.4
ipykernel==6.28.0
ipython==8.20.0
isoduration==20.11.0
jedi==0.19.1
Jinja2==3.1.2
json5==0.9.14
jsonpointer==2.4
jsonschema==4.20.0
jsonschema-specifications==2023.12.1
jupyter-events==0.9.0
jupyter-lsp==2.2.1
jupyter_client==8.6.0
jupyter_core==5.7.1
jupyter_server==2.12.3
jupyter_server_terminals==0.5.1
jupyterlab==4.0.10
jupyterlab_pygments==0.3.0
jupyterlab_server==2.25.2
kiwisolver==1.4.5
locket==1.0.0
MarkupSafe==2.1.3
matplotlib==3.8.2
matplotlib-inline==0.1.6
mistune==3.0.2
mongoquery==1.4.2
msgpack==1.0.7
msgpack-numpy==0.4.8
nbclient==0.9.0
nbconvert==7.14.0
nbformat==5.9.2
nest-asyncio==1.5.8
networkx==3.2.1
notebook_shim==0.2.3
numcodecs==0.12.1
numpy==1.26.3
ophyd==1.9.0
overrides==7.4.0
packaging==23.2
pandas==2.1.4
pandocfilters==1.5.0
parso==0.8.3
partd==1.4.1
pillow==10.2.0
PIMS==0.6.1
Pint==0.23
pipdeptree==2.13.1
platformdirs==4.1.0
prettytable==3.9.0
prometheus-client==0.19.0
prompt-toolkit==3.0.43
psutil==5.9.7
pure-eval==0.2.2
pycparser==2.21
Pygments==2.17.2
pymongo==4.6.1
pyparsing==3.1.1
python-dateutil==2.8.2
python-json-logger==2.0.7
pytz==2023.3.post1
pywin32==306
pywinpty==2.0.12
PyYAML==6.0.1
pyzmq==25.1.2
referencing==0.32.1
requests==2.31.0
rfc3339-validator==0.1.4
rfc3986-validator==0.1.1
rpds-py==0.16.2
Send2Trash==1.8.2
six==1.16.0
slicerator==1.1.0
sniffio==1.3.0
soupsieve==2.5
stack-data==0.6.3
suitcase-mongo==0.4.0
suitcase-msgpack==0.3.0
suitcase-utils==0.5.4
super-state-machine==2.0.2
terminado==0.18.0
tifffile==2023.12.9
tinycss2==1.2.1
toolz==0.12.0
tornado==6.4
tqdm==4.66.1
traitlets==5.14.1
types-python-dateutil==2.8.19.20240106
typing_extensions==4.9.0
tzdata==2023.4
tzlocal==5.2
uri-template==1.3.0
urllib3==2.1.0
wcwidth==0.2.13
webcolors==1.13
webencodings==0.5.1
websocket-client==1.7.0
xarray==2023.12.0
zarr==2.16.1
zict==2.2.0
zipp==3.17.0

[tacaswell]

We have tightly pinned intake because we were running into frequent small changes to the API. In databroker v2 we completely refactored the dependency on intake out.

Going back to 3.11 is likely the fastest option.

we do not appear to have a 1.2.x bug-fix branch at the moment, on my fork I pushed a branch adjusts the dependency, but it does not look like tests are passing : https://github.com/tacaswell/databroker/actions/runs/7475146754/job/20342717390