Error when using option "--skip-completed-runs" at factorization step

[Tang-YH]

Hi,

Thank you for the amazing tool!

I am encountering an error when I am running the latest version of cnmf from the command line. Specifically, when I use the option --skip-completed-runs I get the following error:

Traceback (most recent call last):
  File "~/opt/miniconda3/envs/cnmf/lib/python3.11/site-packages/pandas/core/indexes/base.py", line 3790, in get_loc
    return self._engine.get_loc(casted_key)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "index.pyx", line 152, in pandas._libs.index.IndexEngine.get_loc
  File "index.pyx", line 181, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 7080, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 7088, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'completed'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "~/opt/miniconda3/envs/cnmf/bin/cnmf", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "~/opt/miniconda3/envs/cnmf/lib/python3.11/site-packages/cnmf/cnmf.py", line 1212, in main
    cnmf_obj.factorize(worker_i=args.worker_index, total_workers=args.total_workers,
  File "~/opt/miniconda3/envs/cnmf/lib/python3.11/site-packages/cnmf/cnmf.py", line 666, in factorize
    jobs_for_this_worker = worker_filter(run_params.index[run_params['completed']==False],
                                                          ~~~~~~~~~~^^^^^^^^^^^^^
  File "~/opt/miniconda3/envs/cnmf/lib/python3.11/site-packages/pandas/core/frame.py", line 3896, in __getitem__
    indexer = self.columns.get_loc(key)
              ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/opt/miniconda3/envs/cnmf/lib/python3.11/site-packages/pandas/core/indexes/base.py", line 3797, in get_loc
    raise KeyError(key) from err
KeyError: 'completed'

I have previously rerun the "cnmf prepare" step. The factorization step runs without an error if I remove the parameter --skip-completed-runs.

Thank you for your time!

[dylkot]

Hi there, can you provide your full sequence of commands please?

[Tang-YH]

Of course, here is the full series of my commands:

cnmf prepare --output-dir output.32/ --name ${sample}_cnmf -c sample_counts/${sample}.h5ad -k 32 --n-iter 100 --seed 42 --genes-file ../gene_list.txt
for workerindex in `seq 0 7`
do 
  cnmf factorize --output-dir output.32/ --name ${sample}_cnmf --total-workers 8 --worker-index $workerindex &
done

One of the factorization steps was interrupted due to time limits, so I then reran:

cnmf prepare --output-dir output.32/ --name ${sample}_cnmf -c sample_counts/${sample}.h5ad -k 32 --n-iter 100 --seed 42 --genes-file ../gene_list.txt
cnmf factorize --output-dir output.32/ --name ${sample}_cnmf --total-workers 8 --worker-index 3 --skip-completed-runs
cnmf combine --output-dir output.32/ --name ${sample}_cnmf
cnmf consensus --output-dir output.32/ --name ${sample}_cnmf --components 32 --local-density-threshold 0.5 --show-clustering

which failed with the error above.

After removing the parameter --skip-completed-runs everything completed successfully.

[dylkot]

Hey @Tang-YH, thanks. It is the

cnmf factorize --output-dir output.32/ --name ${sample}_cnmf --total-workers 8 --worker-index 3 --skip-completed-runs

step that is failing? It is weird that the "completed" column isn't the replicate file. If you can open the run in python you can confirm this like so:

from cnmf import cNMF, load_df_from_npz
cnmf_obj = cNMF(output_dir="output.32", name="${sample}_cnmf")
load_df_from_npz(cnmf_obj.paths['nmf_replicate_parameters'])

I also wonder if you should just do --total-workers 1 --worker-index 0 since most of the jobs should be complete so you shouldn't need multiple workers. Like I wonder if all the jobs from --worker-index 3 completed.