Big dataset issue

[Diennguyen8290]

Hi,

Many thanks for creating this tool.

Please may I ask when I work with a big dataset set (say about 100k cells), can I divide the dataset into several smaller subsets and work on those subsets, subsequently merge the annotated cell metadata of those subsets and assign to the original dataset?

I came up with this question because litererally I can not run on the big dataset. The error message is as follow:

"Error in asMethod(object) :
Cholmod error 'problem too large' at file ../Core/cholmod_dense.c, line 105
Calls: run_scmap_seurat ... scmapCell -> as.matrix -> as.matrix.Matrix -> as -> asMethod
Execution halted"

Or please could you advise me any other solutions?

Many thanks.

Regards,
Dien

[mhemberg]

Hi Dien,

Thanks for your comment. When we developed scmap we did not have access to such large datasets, so it was never tested in such conditions. Hence, I have not come across this issue before. It seems as if you are using scmapCell - do you have the same problem with scmapCluster? Also, it is not clear from your message at what step the error occurs, so could you provide some more context?

Best regards,
Martin

[torgheh]

Hi,
Same error here with a dataset of 120,000 cells and when calling selectFeatures.

> sce.SS
class: SingleCellExperiment 
dim: 21087 120583 
metadata(0):
assays(2): counts logcounts
rownames(21087): Sox17 Mrpl15 ... 1700109K24Rik Gm10556
rowData names(1): feature_symbol
colnames(120583): AAACCTGAGGCAATTA-1_1_1 AAACCTGGTAGATTAG-1_1_1 ... TTTGTCATCTTGTACT.4_32
  TTTGTCATCTTTACGT.4_32
colData names(6): nCount_RNA nFeature_RNA ... Tissue ident
reducedDimNames(3): PCA HARMONY UMAP
mainExpName: RNA
altExpNames(0):
> rowData(sce.SS)$feature_symbol <- rownames(sce.SS)
> sce <- selectFeatures(sce.SS, suppress_plot = T, n_features = 500)
Error in asMethod(object) : 
  Cholmod error 'problem too large' at file ../Core/cholmod_dense.c, line 102

[fyh18]

Hi,

I came up with the same question when calling selectFeatures with a dataset of about 150k cells.

It seems like the function tried to convert the sparse matrix into a dense object so that it became too large to handle with.

the error message is as follow:

Error in asMethod(object): Cholmod error 'problem too large' at file ../Core/cholmod_dense.c, line 102
Traceback:

1. selectFeatures(trainset.sce, suppress_plot = TRUE)
2. selectFeatures(trainset.sce, suppress_plot = TRUE)
3. linearModel(object, n_features)
4. as.matrix(logcounts(object))
5. as.matrix.Matrix(logcounts(object))
6. as(x, "matrix")
7. asMethod(object)

[zyh4482]

Is there a solution?

[wakelin-g]

A potential workaround for people might be here which worked for me on a node with 128G of RAM (this worked the second time I tried it, as the first time my R process was killed for exceeding the memory allocations). I had ~180k cells, so take it as you will.

Best,
GW

[Diennguyen8290]

A potential workaround for people might be here which worked for me on a node with 128G of RAM (this worked the second time I tried it, as the first time my R process was killed for exceeding the memory allocations). I had ~180k cells, so take it as you will.

Best, GW

Glad you found it works for your case. Actually, I worked on HPC with a memory of 200G, so this should not be a problem to my case. I eventually ended up using Scanpy to move on but still following this case.

[wakelin-g]

A potential workaround for people might be here which worked for me on a node with 128G of RAM (this worked the second time I tried it, as the first time my R process was killed for exceeding the memory allocations). I had ~180k cells, so take it as you will.
Best, GW

Glad you found it works for your case. Actually, I worked on HPC with a memory of 200G, so this should not be a problem to my case. I eventually ended up using Scanpy to move on but still following this case.

I think the cholmod error can result from your matrix exceeding the maximum allowed data dimensions by the conversion tool used by Matrix package, which would be independent of the amount of RAM you have. The possible solution that I linked provides an alternative way to convert a matrix from sparse to dense that bypasses this problem. If you re-tried with this way, you probably wouldn't get the cholmod error again with a 200G HPC.

Best,
GW