More flexible cell hold out

[mbabadi]

Cell hold out table

column_A,column_B,column_C, …
X1,Y1,Z1, …
X2,Y2,Z2, …
X3,Y3,Z3, …
…

Example:

dataset_id,donor_id
X1,Y1
X2,Y2
X3,Y3

Let’s say we have n rows in the “cell hold out table”.

We want the following features:

• Feature 1
  Make a shuffled extract, excluding all cells matching any row of the cell hold out table

• Feature 2
  Make an extract (shuffled or not) of cells matching each row of the table separately

  • row_1.h5ad
  • row_2.h5ad
  • …
  • row_n.h5ad

[fedorgrab]

@mbabadi

Feature 1
We can hold out any data by simply concatenating columns and we agreed on proceeding with that approach for now as it requires no additional development.

Feature 2
This PR makes it possible to generate extracts where each extract bin corresponds to a specific feature value. For example, by using a concatenated value of donor and dataset ID, we can create extracts where each extract file corresponds to a particular value of the donor-dataset concatenated column. However, this results in extract files of varying lengths. It is unclear how well this will work with data loaders and prediction scripts.
I marketed it as a draft PR until we can be sure this is what we want from this feature.