save() crashes after simple_aggregate()

[EitanGronich]

Hi,

Working solely based on your tutorial, it seems using .save() after simple_aggregate crashes and creates an unusable .pxl file.

Minimal code:

from pathlib import Path
import pixelator
from pixelator import simple_aggregate


DATA_DIR = Path('./datasets')
FILENAMES = [
    "Sample05_V2_PBMC_r1.layout.dataset.pxl",
    "Sample06_V2_PBMC_r2.layout.dataset.pxl",
]

COMBINED_FILENAME = "combined_resting_data.pxl"

combined_path = DATA_DIR / COMBINED_FILENAME
baseurl = "https://pixelgen-technologies-datasets.s3.eu-north-1.amazonaws.com/mpx-datasets/pixelator/0.18.x/technote-v1-vs-v2-immunology-II"
for filename in FILENAMES:
    !curl -L -O -C - --create-dirs --output-dir {DATA_DIR} "{baseurl}/{filename}"

pg_data = pixelator.simple_aggregate(
    ["resting_r1", "resting_r2", ], [pixelator.read(DATA_DIR / filename) for filename in FILENAMES]
)
pg_data.save(combined_path, force_overwrite=True)

Result:

ComputeError: ValueError: Attempt to use ZIP archive that was already closed

I'm using python 3.10.16 and pixelator 0.19. Attached is environment.txt (the .yml file I wrote to create the conda environment, converted to .txt so I can upload it here) and printed_environment.txt (same but created by conda env export so this is the exact environment I'm running on).
Platform is Red Hat Enterprise Linux 9.1 (Plow) Linux 5.14.0-162.6.1.el9_1.x86_64,
but I had the exact same issue on Ubuntu 24.04.1 LTS Linux 5.15.167.4-microsoft-standard-WSL2.

Any clue?

Thanks!

environment.txt
printed_environment.txt

[johandahlberg]

Hi @EitanGronich!

Thank you for reporting this. While we work on an actual fix for this I just wanted to let you know that you can work around it by making sure that the PixelDataset instances are a list instead of a list expression. I'm attaching an example below:

from pathlib import Path
from pixelator import read, simple_aggregate


DATA_DIR = Path("./datasets")
FILENAMES = [
    "Sample05_V2_PBMC_r1.layout.dataset.pxl",
    "Sample06_V2_PBMC_r2.layout.dataset.pxl",
]

COMBINED_FILENAME = "combined_resting_data.pxl"

combined_path = DATA_DIR / COMBINED_FILENAME

datasets = list([read(DATA_DIR / filename) for filename in FILENAMES])
pg_data = simple_aggregate(
    [
        "resting_r1",
        "resting_r2",
    ],
    datasets,
)
pg_data.save(combined_path, force_overwrite=True)

I'll post an update on this issue once we have fixed this bug.

[EitanGronich]

Thanks, I'll add that I found out this problem occurs anytime a reference to a pxl object is lost. I'm guessing this occurs since the inner data structures aren't actually accessible to the new reference (the ZIP files are closed, to judge by the error descriptions). For instance, when reusing a variable name for filtering:
pg_data = pg_data.filter(components=components_to_keep)
There is trouble down the line with pg_data (for instance save() crashes with Attempt to use ZIP archive that was already closed)

[johandahlberg]

Yes. You are absolutely right about that.

At least part of the problem comes from this:

pixelator/src/pixelator/pixeldataset/datastores.py

Line 361 in 34eb6f8

self.close()

where we close the file handle when the underlying datastore class goes out of context and is cleaned up by the garbage collector.

We are looking into some different solutions to this, however it might take us some time to implement especially with the holidays coming up now.

I can also mention that you can get around this problem by removing the precomputed_layouts since they are the culprit in this drama. For example this works fine (note that I also skip the edgelist here to speed things up a bit):

from pathlib import Path
from pixelator import read, simple_aggregate


DATA_DIR = Path("./datasets")
FILENAMES = [
    "Sample05_V2_PBMC_r1.layout.dataset.pxl",
    "Sample06_V2_PBMC_r2.layout.dataset.pxl",
]

COMBINED_FILENAME = "combined_resting_data.pxl"

combined_path = DATA_DIR / COMBINED_FILENAME

pg_data = simple_aggregate(
    [
        "resting_r1",
        "resting_r2",
    ],
    [read(DATA_DIR / filename) for filename in FILENAMES],
    ignore_edgelists=True,
)
pg_data.precomputed_layouts = None
pg_data.save(combined_path, force_overwrite=True)