Repository navigation
Conversation
This commit updates UserCtrl to allow a new final, optional, argument -P. This argument takes in a character, [file_prefix]. When passed to ModEM, ModEM will set `ctrl % storeSolnsInFile` to `.true.` and will set `ctrl % prefix` to [file_prefix]. This commit also updates the `search_args` do loop in `parseArgs` to allow it to better handle additional final optional arguments. It should be easy now to expand it to allow additional optional arguments. Just ensure you set `optArgSize` correctly. For single, standalone options without an argument, `optArgSize` should be 1, options with an argument should set `optArgSize` to 2. This commit also updates the user guide with the details of this new optional argument for inversion.
This commits creates three new routines in SolnSpace that allow the reading and writing of solnVector_t types. The routines will read and write the active transmitter and the active polarization of the passed in solnVector_t. These routines will be used to save partial solutions, thus, perhaps they are a bit limited in their interface. It may be appropriate to have a routine that takes in desired iTx and polarization and allocates an solnVector_t, rather than requiring it be already allocated and created. I'll think about that, but for now this should be sufficient.
This commit creates an ModEM_mpi_context_t type which will be used to store and pass around MPI communicators to the EsolnManager. If it works well, perhaps in future merges it can be used to in other areas of ModEM. As of this commit, the ModEM_mpi_context_t is not used, but will be in a future commit.
This commit allows `solnVectors_t` and `cvectors_t` to be created as 'place holders'. These 'place holder' `solnVectors_t` and `cvectors_t` will only contain the information on that electrical solution and wont contain any data. In conjunction with the `ESolnmManager` this will allow the main task to *not* allocated eAll, saving a bit of unnecessary memory.
This commit simply adds the ESolnManager to modem code base and adds it to the CMakeLists.txt. The ESolnManager will serve as a manager of electric and adjoint fields for the main task and the worker tasks. Depending on the arguments passed in, it will either have the worker tasks communicate the electric and adjoin fields via writing and reading them to disk, or it will send the fields to and from the main task using MPI (the current ModEM implementation). Perhaps in a future commit the ESolnManager will be responsible for allocating and deallocating the solnVectors_t.
This commit adds error checking when reading and writing cvectors. Now that these routines will be used more often, it will be beneficial to ensure that these routines complete successfully.
This commit adds in a 'trial' logical to the define_worker_job type. This logical will be used in future commits to signify to worker tasks that a trial is being calculated. This will allow the EsolnManager the ability to save files with an appropriate filename (e.g. labeling them as 'trial').
This commit adds in the PQMult and DataResp Master and worker jobs. These routines will be used with the ESolnManager implementation. It also udpates create_data_vec_place_holder and associated routines so that they can take a range of data instead of just one.
This commit initializes the ModEM context in Constructor_MPI.
This commit adds a new flag to func and to gradient. The flag in func can be used to signify to the worker tasks that a trial forward is being calculated. The worker_job in the future will use this to write out it's electric field to a file with the appropriate prefix. When we use the initial guess in the line search we will also need to tell the JMultT calculations to read the trial electric field, rather than the computations of the line search. Thus, we need to tell the gradient, and JmultT to use the starting guess. Worker jobs will then be able to know if they should read from the trial file or not.
This commit finally uses the ESolnManager into ModEM. As of this commit it generates bit-for-bit results when running with or without saving the electric field solutions.
This commit updates the Main_MPI code that was added in previous commits to allow the new MPI code associated with the ESolnManager to work with SP2 FG. To do this, it updates the DataResp and the PQMult to not send their MPI send or receives on the MPI_COMM_WORLD but on either comm_world or comm_leader depending on the para_method. For both the DATARESP and the JQMULT, it performs both by sending the work to the leader tasks. Because these tasks are rather simple and fast, the leader tasks do not send the worker tasks anything. When running with FG on equal processor counts with or without -P gives bit-for-bit result. It appears that the running with FG with different processor counts does not give bit-for-bit results, this may be an issue before this task. Running without FG at different processor counts gives the same results, but there are difference when running with and without FG. I'll open and investigate that issue further.
The last commit to allow SP2 to work broke the DataResp when using 2, 3 , and 4 procs, and potentially others. The culprit was using size_leader instead of using size_leader - 1 in places.
This commit responds to one of Hao's comments on the PR. It moves the DataResp logic into three new functions. These are: * Master_job_DataResp - Interface function that decides what function to call below if ESolnManager % EsMgr_save_in_file is true or false. * Master_job_DataResp_main_only - The originally way to calculate dataresp. Only the main task calculates it. * Master_job_DataREsp_IO - The workers compute the dataresp for a range of transmitters and read the electric fields from files.
This commit separates out the PQMult logic into three new functions: * Master_job_PQMult - Interface function that decides what function to call if ESoln_save_in_file is either .true. or .false. * Master_job_PQMult_main_only - The original way that PQMult was calculated. Is only calculates on the main task. * Master_job_PQMult_IO - The workers compute PQMult by reading in the electric fields and the adjoint fields.
This commit updates ModEM to allow the ability for labels to be passed to the esoln manager from either NLCG or LBFGS. First, this commit updates the ESolnManager so that it EsMgr_save and EsMgr_get take in a label, a character string to label the current line search equation, and a field type, to label either JmultT or electric fields (FWD) solnVectors. The worker_job_task is updated to pass a new variable called label, which can be used by worker tasks to correctly label their calculations (specifically with the EsolnManager). The Main_MPI routines, the INVCore routines are updated with pass through labels, and then the line searches of both LBFGS and NLCG have been updated to use a label when calling either func or gradient. The labels represent the location of the line search, e.g. the trial, the minimum etc. In this commit I have labeled these as F_1, for the trial, and F, for the non trial. Depending on the line search, we label our forwards calculations as F_1 or F and then use these in the gradient calculation later on. These labels allow the ESolnManager to write out its fields accordingly.
This commit updates distribute taskes so that PQMULT job can be passed to it.
This commit only builds the ESolnManager if building MPI. In the future, we might want the EsolnManager to function in serial, but that might be better for a different pull request.
Member
Author
|
Open on accident. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is a cleaned up pull request of #16 and now pointed to the develop branch.
This pull requests implements the optional ability to run inversions, forwards and JMultT with the ability for worker tasks to save their electric and ad-joint fields to disk, rather then sending them between the master task.
If the optional -P argument is passed on the command line, with a prefix string: -P esoln, ModEM will write out the electric fields to disk when computing the forward or adjoint solutions.
When running an inversion with NLCG, the worker tasks will read in the electric field that matches the transmitter they have been told to compute adjoint solution of rather then getting the electric field from the master task.
Note: When running the NLCG inversion with -P, worker tasks do not use unique filenames between iterations, thus when a worker tasks writes out the electric or adjoint field for iteration 2, it will overwrite the files that were written in iteration 1.
These changes now work with the SP2 Fined-Grained. Running with the same number of processors with and without saving the electric field/adjoint solutions gives bit-for-bit results. However, it appears the FG does not produce bit-for-bit results on different processor counts even before this change. It also looks like the FG produces different results when comparing SP2 with and without SP2.