Memory problem?

[kokyriakidis]

Hi,

I tried transabyss (k32) on my university cluster on 40gb RNASeq data, following your short tutorial, and it is stuck because it used all 384gb of the node and also swaped! I think it is really strange to require so much memory! Do you have any thoughts?

[sjackman]

Use ntCard to count the number of distinct 32-mers in the data set. With that value in hand, we can predict the memory usage of ABySS.
https://github.com/bcgsc/ntCard

[kmnip]

@kokyriakidis
Since you are using a small k-mer size (ie. 32), I would recommend recompiling ABySS with a smaller max kmer size. The default is 128, which is 4 times as much memory than needed.

Reconfigure from source with this command and then re-compile:

./configure --enable-maxk=32

By the way, Trans-ABySS is actually not using the Bloom filter de bruijn graph implementation of ABySS.

[sjackman]

The easiest way to install ABySS with a different value of maxk is to use Linuxbrew.
Install Linuxbrew https://linuxbrew.sh

brew tap brewsci/bio
brew edit abyss
# Change --enable-maxk=128 to --enable-maxk=32
brew install -s abyss

[kokyriakidis]

So let me ask you something. Let’s say I compile abyss with maxk=512 in a machine and in an other machine I compile abyss with kmax=90. If I run the same commands with k=32 and with the same input files, am I gonna have different memory usage results?

[sjackman]

Yes, --enable-maxk=512 will use 16 times the memory as --enable-maxk=32.

[kokyriakidis]

Ok I will try it and I will tell you my results. Keep it open.

One more question. If I iterate from 32 to 90 by 2 and in a second machine (with the same maxk) iterate from 32 to 90 by 1, will I have the same result at k=40 for example? Or the number of previous iterations have an impact on the result?

[kmnip]

If you assemble at k=40 on different machines:

• the peak memory usage would be the same
• the set of assembled sequences should be the same

However, the merged assembly of 32-90:1 would not be the same as the merged assembly of 32-90:2.

[kokyriakidis]

So what's the best procedure? With step 1?

[kokyriakidis]

Ok this is the kmer frequency

[kyriakidk@ui3 kyriakidk]$ more freq_k32.hist
F1      31958617028
F0      3094977367
f1      2589161085
f2      182448062
f3      78785531
f4      47139273
f5      28384407
f6      21130475
f7      14353281
f8      11746673
f9      8502788
f10     7583184
f11     5869035
f12     5228492
f13     4310735
f14     3841182
f15     3372548
f16     2958430
f17     2796366
f18     2497192
f19     2220842
f20     2071655
f21     1920211
f22     1785256
f23     1641820
f24     1548139
f25     1453978
f26     1348387
f27     1287416
f28     1256055
f29     1207811
f30     1029205
f31     1052726
f32     1006082
f33     937207
f34     896512
f35     831747
f36     820010
f37     778788
f38     790625
f39     767795
f40     706938
f41     680623
f42     638476
f43     624427
f44     645331
f45     595467
f46     576271
f47     500900
f48     525466
f49     479743
f50     493656
f51     492538
f52     454246
f53     470155
f54     446223
f55     453706
f56     422592
f57     386648
f58     398407
f59     372447
f60     391344
f61     351841
f62     346001
f63     325377
f64     370122
f65     300342
f66     326872
f67     326668
f68     296638
f69     267930
f70     300329
f71     278309
f72     291972
f73     279434
f74     258838
f75     248687
f76     258123
f77     238375
f78     253063
f79     246729
f80     256107
f81     198032
f82     233786
f83     258345
f84     230144
f85     219002
f86     231547
f87     229372
f88     204536
f89     196487
f90     188297
f91     214279
f92     161232
f93     186575
f94     185336
f95     206056
f96     165476
f97     168952
f98     163766
f99     147252
f100    159844
f101    170098
f102    142040
f103    161975
f104    172162
f105    144098
f106    129856
f107    160040
f108    146310
f109    152640
f110    132910
f111    158984
f112    152538
f113    119448
f114    131152
f115    127958
f116    142488
f117    126822
f118    119713
f119    148786
f120    110203
f121    131263
f122    133157
f123    117599
f124    141562
f125    132026
f126    115533
f127    122940
f128    114598
f129    120890
f130    131194
f131    98992
f132    124142
f133    108392
f134    124064
f135    110463
f136    95049
f137    119020
f138    113630
f139    92958
f140    130435
f141    108349
f142    100252
f143    107586
f144    96134
f145    100383
f146    113814
f147    112652
f148    87800
f149    82854
f150    105700
f151    99282
f152    101402
f153    82743
f154    102594
f155    82743
f156    83957
f157    80850
f158    73634
f159    81992
f160    81920
f161    87097
f162    90155
f163    81838
f164    79842
f165    100577
f166    69317
f167    80990
f168    79862
f169    66408
f170    67571
f171    62393
f172    76943
f173    95453
f174    90094
f175    72524
f176    79936
f177    75734
f178    76808
f179    62302
f180    65546
f181    75882
f182    69574
f183    67557
f184    80005
f185    70570
f186    71687
f187    64431
f188    72785
f189    64427
f190    74858
f191    56124
f192    57331
f193    78043
f194    55068
f195    66664
f196    57246
f197    62509
f198    64537
f199    81085
f200    65394

The command was:

./ntCard/ntcard -k32  -t10 /mnt/denovo_a/home/kyriakidk/FASTQ_FILES/ERR522068_1.fastq.gz /mnt/denovo_a/home/kyriakidk/FASTQ_FILES/ERR522068_2.fastq.gz

How do I interpret it?

[kmnip]

@kokyriakidis
Trans-ABySS does not use the Bloom filter de bruijn graph. So, running ntCard would not help with your memory issue at all. Please recompile ABySS with a smaller max kmer size as I had suggest earlier. Thank you.

[sjackman]

I had asked @kokyriakidis to report the output of ntCard to predict the memory usage of ABySS.

F0      3094977367

There are three billion distinct k-mers. The memory usage of ABySS is…

(8 + maxk / 4) · F₀
maxk = 512: (8 + 512 / 4) · 3e9 = 408 GB of RAM
maxk = 128: (8 + 128 / 4) · 3e9 = 120 GB of RAM
maxk = 32: (8 + 32 / 4) · 3e9 = 48 GB of RAM

@benvvalk I added an entry to ABySS's FAQ: https://github.com/bcgsc/abyss/wiki/ABySS-Users-FAQ#6-how-much-memory-does-abyss-use