Enabling Distributed Processing

[markmac99]

An upgrade to the RMS solver, CorrelateRMS, to enable distributed processing across multiple servers. To assist with this the first step has been split into two, one to create candidates and one to perform initial simple solutions.

This PR also replaces the JSON database with SQLite which is necessary to enable distributed processing and also brings performance benefits. The PR adds two new commandline arguments, one to control how much data to retain in the databases, and one to post-fix the log name with the phase name eg correlate_rms_20260101_12345_cands.log. This is to ensure each phase's logfile is uniquely named and to make it easier to monitor and debug.

[satmonkey]

Mark, just a question, is the rebound import needed for wmpl.Utils.Dcriteria?
I keep getting this when trying to import this single function:
OSError: librebound.cpython-310-x86_64-linux-gnu.so: cannot open shared object file: No such file or directory
If not, would it be able to make the dependency optional?

[markmac99]

Mark, just a question, is the rebound import needed for wmpl.Utils.Dcriteria?

I didn't actually add the Rebound functionality. The "rebound" functionality was added by @tamidodo and Denis last year.

However, i just checked and Dcriteria.py has no dependency on rebound, so i am not sure how the error is arising for you.
i am able to import calcDSH without any issues,

>>> from wmpl.Utils.Dcriteria import calcDSH
>>>

One possibility is that at some point in the past you "installed" the library by running python setup.py install. This would have put a precompiled copy in the lib/python3.10/site-packagesWesternMeteorPyLib directory in your virtualenv, and this is likely to be out of date. There might then be dependencies which were since removed. Python will prefer to read from any copy in site-packages before reading files in the PYTHONPATH or local tree, so this can definirely cause issues.

I now routinely delete the copy in site-packages to avoid issues. WMPL runs just fine "not installed", once you've obtained the required datafiles which will be in the share folder. I keep meaning to write a standalone function to collect the files (orbit, shower, and ephemerides) without doing the whole install.

ps I also think that the REBOUND module should be renamed to avoid the clash with the pypi module and Linux library of the same name.

[satmonkey]

fine, the install by setup.py was the cause, now the issue is gone.
Thanks!