Permission errors from ucd.tl.base

[madscientistca]

When I installed the latest package and registered and got my updated API key, in running:

temp_ucd_list.append(ucd.tl.base(temp_matrix, split=True, sort=False))

on a windows 11 machine, where I am running visual studio 2022, have a conda environment set up for this, and where I am setting the temp directory as:

tempfile.tempdir = "D:/Dropbox/appdata/temp"

where I have full control (so no permissions issues), I get the following permissions error:

test_ucd_list.append(ucd.tl.base(test_matrix, split=True, sort=False))

D:\Dropbox\data\UCD_Test\Python\ucd_env\lib\site-packages\anndata_core\anndata.py:117: ImplicitModificationWarning: Transforming to str index.
warnings.warn("Transforming to str index.", ImplicitModificationWarning)
2023-05-08 08:39:57,300|[UCD]|INFO: Starting UCDeconvolveBASE Run. | Timer Started.

Preprocessing Dataset | 0% (0 of 1) | | Elapsed Time: 0:00:00 ETA: --:--:--
Preprocessing Dataset | 100% (1 of 1) |##| Elapsed Time: 0:00:00 Time: 0:00:00
2023-05-08 08:40:02,039|[UCD]|INFO: Uploading Data | Timer Started.
2023-05-08 08:40:04,841|[UCD]|INFO: Upload Complete | Elapsed Time: 2.802 (s)

Waiting For Submission : UNKNOWN | Queue Size : 0 | / |#| 0 Elapsed Time: 0:00:00
Waiting For Submission : UNKNOWN | Queue Size : 0 | - |#| 1 Elapsed Time: 0:00:01
Waiting For Submission : UNKNOWN | Queue Size : 0 | \ |#| 2 Elapsed Time: 0:00:03
Waiting For Submission : QUEUED | Queue Size : 1 | | |#| 3 Elapsed Time: 0:00:05
Waiting For Submission : RUNNING | Queue Size : 1 | / |#| 3 Elapsed Time: 0:00:05
Waiting For Submission : RUNNING | Queue Size : 1 | | |#| 3 Elapsed Time: 0:00:05

Waiting For Completion | 0% (0 of 76) || Elapsed Time: 0:00:00 ETA: --:--:--
Waiting For Completion | 0% (0 of 76) || Elapsed Time: 0:00:01 ETA: --:--:--
Waiting For Completion | 0% (0 of 76) || Elapsed Time: 0:00:03 ETA: --:--:--
Waiting For Completion | 100% (76 of 76) || Elapsed Time: 0:00:05 ETA: 00:00:00
Waiting For Completion | 100% (76 of 76) || Elapsed Time: 0:00:07 ETA: 00:00:00
Waiting For Completion | 100% (76 of 76) || Elapsed Time: 0:00:07 Time: 0:00:07
2023-05-08 08:40:20,105|[UCD]|INFO: Download Results | Timer Started.
2023-05-08 08:40:21,022|[UCD]|INFO: Download Complete | Elapsed Time: 0.916 (s)
2023-05-08 08:40:21,022|[UCD]|INFO: Run Complete | Elapsed Time: 23.722 (s)
Traceback (most recent call last):
File "", line 1, in
File "D:\Dropbox\data\UCD_Test\Python\ucd_env\lib\site-packages\ucdeconvolve_tools_base.py", line 160, in deconvolve_base
ucdapiutils.download_and_attach_results(
File "D:\Dropbox\data\UCD_Test\Python\ucd_env\lib\site-packages\ucdeconvolve_api_apiutils.py", line 274, in download_and_attach_results
with h5py.File(temp_file.name) as hfile:
File "D:\Dropbox\data\UCD_Test\Python\ucd_env\lib\site-packages\h5py_hl\files.py", line 567, in init
fid = make_fid(name, mode, userblock_size, fapl, fcpl, swmr=swmr)
File "D:\Dropbox\data\UCD_Test\Python\ucd_env\lib\site-packages\h5py_hl\files.py", line 231, in make_fid
fid = h5f.open(name, flags, fapl=fapl)
File "h5py_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
File "h5py_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
File "h5py\h5f.pyx", line 106, in h5py.h5f.open
PermissionError: [Errno 13] Unable to open file (unable to open file: name = 'D:\Dropbox\appdata\temp\tmp9eay61jo', errno = 13, error message = 'Permission denied', flags = 0, o_flags = 0)

where I note the conda environment is set up in D:\Dropbox\data\UCD_Test\Python\ucd_env

Not sure if this has been seen before. So posting in case there is some permission conflict that happens in windows. In addition to setting up an isolated environment, I shut off all antivirus software and shut down all firewalls. The permissions also in the directories as indicated are appropriately set. So can't figure it out. This was not an issue in the previous version I had been using of UCD.

[madscientistca]

Okay, well, I hacked this to make it work. The problem was the download_and_attach_results function in the _apiutils.py file, where a temporary file is created using tempfile.NamedTemporaryFile, which opens the file and provides a context manager for working with the file. When this is done within the context of a with statement, the temporary file is automatically deleted when the block is exited. However, it appears on Windows 11, at least 3 different computers I tested that are maintained independently, but all running Windows 11, that is an issue when another function tries to access the temporary file while it is still open within the NamedTemporaryFile() context. Windows generally mandatory file locking.

To work around this I changed the way in which the temporary files are created and pass into this function (and the parent functions that call this function -- I only found one other function that called this function, the base function) a temporary directory I set in my main code. In case useful to others, the modified download_and_attach_results function to run on Windows systems having this problem is:

def download_and_attach_results(
adata : anndata.AnnData,
download_url : str,
runtype : str,
key_added : str,
additional_runinfo : Optional[dict] = dict(),
temp_directory : Optional[str] = None,
) -> str:
"""\

Download Result

Downloads results at attaches them to the anndata object
depending on the type of run.

Params
-------
adata
    Annotated dataset to save results to
download_url
    A valid signed URL to download results content to
runtype
    The type of run which changes how we save data.
key_added
    Key to use
    
Returns
-------
adata : anndata.AnnData
    An annotated dataset with saved results

"""
# Make a GET request to the signed URL to download the file
response = requests.get(download_url)

# Write the downloaded content to a temporary file
# Create a temporary file
#
fd, temp_file_path = tempfile.mkstemp(dir=temp_directory)

# Write the content to the temporary file
with os.fdopen(fd, 'wb') as temp_file:
    temp_file.write(response.content)
    
    # Save function for base
    if runtype == 'base':
        
        # Get keys from file
        with h5py.File(temp_file_path, 'r') as hfile:
            keys = list(hfile.keys())

        # Iterate each results key and load
        preds = dict()
        for key in keys:
            preds[key] = pd.read_hdf(temp_file_path, key=key)

        # Attach to anndata
        append_predictions_to_anndata(preds, adata,
                                    key_added=key_added,
                                    additional_runinfo=additional_runinfo)

    if runtype == 'select':

        # Get keys from file
        with h5py.File(temp_file.name) as hfile:
            keys = list(hfile.keys())

        # Iterate each results key and load
        preds = dict()
        for key in keys:
            preds[key] = pd.read_hdf(temp_file.name, key = key)

        # attach to anndata
        append_predictions_to_anndata(preds, adata, 
            key_added = key_added, 
            additional_runinfo = additional_runinfo)
        
    if runtype == 'explain':

        # Load explanations file as sparse CSR matrix
        explanations = _read_csr_matrix_from_h5(temp_file_path)

        # Don't attach here, return directly?
        #adata.obsm[key_added] = explanations

        # attach runinfo
        adata.uns[key_added] = dict(runinfo=additional_runinfo)

        # We need to fix this later
        return explanations
    
# Clean up by deleting the temporary file
os.remove(temp_file_path) 
    
# return anndata
return adata