multiple runs fail in a boostrap

[yanxiaoyu2001]

Hello,

I am running a boostrap job using a multicore computer. Somehow, some runs failed with the following error message that "It seems like Fortran compilation by NONMEM's nmfe script failed. Cannot start NONMEM. Go to the NM_run192 subdirectory and run psn.mod with NONMEM's nmfe script to diagnose the problem. ". I did retest the run using nmfe command in the NONMEM run folder and the run was excuted fine. A screenshot showing the error message is attached. Your help is much appreciated!

Best regards,
Xiaoyu

[rikardn]

Only some runs fail in the bootstrap and not all? The linker (ld) is complaining that some symbols are missing when linking all the Fortran object files. Check that the nmfe you tested outside of PsN is from the same version of NONMEM as in your psn.conf. Are you running on a cluster or on your local machine? If you are using the -nodes option try using -threads instead. If that works it could be a problem with the MPI parallelization.

[yanxiaoyu2001]

Dear Rikard,

Many thanks for the help! Only some runs fail, not all. The PsN version uses the same NM as nmfe. The job was on a local machine with an Intel Core i5-14500 processor, which has 14 cores and 20 logical processors. I did use the -nodes and -threads option as shown below:
\bootstrap run6.mod -samples=500 -seed=12345 -parafile=C:\nm76g64\run\fpiwini8.pnm -nodes=6 -threads=6 -clean=0

I did rerun the boostrap using the following
\bootstrap run6.mod -samples=500 -seed=12345 -threads=6 -clean=0

However, I got the same issue as shown the screenshot. Another observation is that the run4 was failed for the second boostrap but not the first bootstrap. I would think they are identical given the same seed number.

Given the boostrap is really time consuming, I wonder if there is a work around, e.g. rerun all failed run manually, replace files in the boostrap folder, then trick the PsN to calculate statistics?

Thanks,
Xiaoyu

[rikardn]

Some thoughts:

1. Note that using -threads and -nodes at the same time would mean that the number of cores used will be the product, e.g. in your case 6*6 = 36
2. Differences between parallel runs and non-parallel should be the same. Check if run4 in the first run used the same dataset as in the second.
3. There is an option -summarize to bootstrap that could do the post processing, but it requires the raw_results file to be updated for the failed runs.
4. In any case it is surprising that some models crash in this way in bootstrap, since all models are exactly the same except for the dataset. I don't see what could cause this.

[yanxiaoyu2001]

Dear Rikard,

1. I was trying to push the CPU to a higher % usage. Somehow, it was still ~60%. I guess there is some mechanism for protection.
2. I double-checked the data using notepad++ compare function. The dataset was identical.
3. Could you explain more about how to do this? I will try this..

Thanks a lot!

Best,
Xiaoyu

[rikardn]

2. This is also a bit strange then.
3. The summarize option will recalculate the bootstrap_results using the raw_results file. The purpose of the option is to be able to change the inclusion criteria (options starting with skip), but it could be used to recalculate. What you need to do is change the rows in the raw_results file to reflect the new results after your fixed runs. If you have many models that failed, this can be a bit tedious.

[yanxiaoyu2001]

Dear Rikard,

Many thanks for the information! I will try the summarize function and let you know how it goes.

Best regards,
Xiaoyu

[yanxiaoyu2001]

Dear Rikard,

I remove all the failed run rows from the raw_result.csv and then rerun the bootstrap using the same command. Then PsN reran all the failed runs and summarized the results in the same folder. I have to do this a few times to make sure all failed runs were executed, because each time there were still some failed runs.

Thanks,
Xiaoyu

[rikardn]

I am not sure I understand. I was thinking to rerun the failed models (from the m1 dir) outside of the bootstrap and then update the raw_results of the bootstrap and then run the bootstrap in the same dir with summarize. This shouldn't start any new runs in the bootstrap.

[yanxiaoyu2001]

After executing the same command with the same seed number, PsN generates the same number of data files and control streams. Somehow, only the deleted run No. was reruned. I forgot to mention that I deleted the failed run folder as well. Hope this clarifies things a little.