Consider instantiating a new PartitionedCellRepresentation object for each worker in the DataLoader using the worker_init_fn argument. That way, each worker has a different handle on the AnnData backed object and are not changing the same mutable state of a single object but rather change state of their own individual dataset objects.
This may not work, since it may or may not be possible to open multiple file handles for a backed AnnData (have to check)...
Consider instantiating a new
PartitionedCellRepresentationobject for each worker in theDataLoaderusing theworker_init_fnargument. That way, each worker has a different handle on theAnnDatabacked object and are not changing the same mutable state of a single object but rather change state of their own individual dataset objects.This may not work, since it may or may not be possible to open multiple file handles for a backed AnnData (have to check)...