Executing the igneous mesh forge command, the program did not make any progress

[Ahusbj]

Question 1: After following the command in the Figure 1 for a period of time, the program page will be stuck on the progress of Figure 2 for more than 10 hours without any change. At this time, the CPU running rate is 0, but the memory usage is high. What is the reason for this and how can it be resolved.
Question 2: When I further execute "igneous mesh merge sharded", the program often takes a long time to execute. Is there any way to speed it up?
Thanks

[william-silversmith]

Hi @Ahusbj, thanks for writing in!

Question 1: After following the command in the Figure 1 for a period of time, the program page will be stuck on the progress of Figure 2 for more than 10 hours without any change. At this time, the CPU running rate is 0, but the memory usage is high.

When the CPU usage is low, this is usually due to some issue with IO as the meshing process is fairly CPU intensive. Do you notice this period of low CPU before or after a period of high CPU? Is the filesystem you are writing to a network mount? Does it have any performance issues?

Sometimes parallel executing can be funky, so see if setting -p 1 helps. You can run this against the queue without deleting it and repopulating it.

Question 2: When I further execute "igneous mesh merge sharded", the program often takes a long time to execute. Is there any way to speed it up?

Yes, there are several steps that can possibly take some time, but the most common first issue people run into is efficient use of the spatial index. In order to generate sharded files, it is necessary to find which fragment files generated during forging have the right pieces from across the entire dataset.

In order to make this search efficient, it is necessary to create a database with an index. I presume you are using a local or networked filesystem to do your work, in which case the simplest approach is to use sqlite, which writes a database into a single file.

Use this command to generate the database.

igneous mesh spatial-index db $OUTPUT_DIR sqlite://index.db

Then provide the database to your tasks.

igneous mesh merge-sharded ... --spatial-index-db /path/to/index.db

Let me know if this helps!

[Ahusbj]

Thank you very much for your reply!

Regarding Question 1:

(1) My program initially had high CPU usage, but after running for a while, the utilization gradually decreased to 0, resulting in a state where there was no progress for half a day. The relevant content is written in the server's storage path, so performance shouldn't be a problem. I wonder if this information is helpful.

(2) Regarding the impact of the number of concurrent connections you mentioned, I set it to -p 1, but found that the speed was very slow. I tried setting -p to 50, wondering if this could reduce concurrency and thus avoid the program becoming unresponsive.

(3) After reducing the number of concurrent connections, I found that the program was throwing an error, which I happened to see, as shown in Figure 1. Is this one of the causes? Previously, with a high number of concurrent connections, the error message displayed on the command line might have been overwritten, so I didn't notice it.

Regarding Question 2:

Thank you very much for your reply. I have followed your suggestions and made the necessary modifications. The revised commands are shown in Figure 2. The commands are executed sequentially. If there are any errors in the commands, please point them out. Thank you.

[william-silversmith]

My program initially had high CPU usage, but after running for a while, the utilization gradually decreased to 0

Ah, I know what happened. One of the processes encountered an error and it killed all of them. The -p 80 parallelization technique was added as a convenience, it is not robust and tends to hide error messages. Igneous was designed to run as independent processes (-p 1) using e.g. SLURM (with the appropriate flags for respawning a dead process) or Kubernetes with 1 process per a CPU. The error you encountered looks like a filesystem error, possibly due to contention. Most likely respawning a process would fix it.