NERSC reproduction, doc, and cleaned up config

[BenCarter44]

I was able to reproduce the simulation + training + model publishing on NERSC.
I ran into a few errors along the way, so this PR covers the things I did to get it working. See REPRODUCE.md for a step-by-step intro for how to get it running at NERSC.

I also cleaned up the configuration system. I changed the config in utils/coordinator.py to reference the config.sh so that all the configuration variables are in the same place.

I am going to rerun one last time from scratch to make sure these changes are good. Would love any feedback.

[BenCarter44]

Doing one final end-to-end test, but should be good.

[BenCarter44]

Successful end-to-end test. It however needed more than one hour to finish (using only CPU).

[dthain]

How much of that was execution time, and how much waiting time?

[BenCarter44]

Hello Dr. Thain, just saw this comment.
I don't have the exact numbers, would have to rerun it. Note, I am not using the GPU for training. It is using CPU only for now.

From what I recall:

• Each simulation takes about 3 minutes.
• All three models training in parallel take under 30 minutes.
• There was some non-trivial time for data transfer / setup.

I originally had requested 72 nodes for 1hr, but after checking on it later, I noticed that workflow progress froze. I realized that there were no work_queue workers remaining. Relaunching a new request for more nodes and work_queue workers allowed for the makeflow progress to continue. There were no errors, and makeflow recovered perfectly.

I set the time limit to 2hrs now which is more than plenty.

[BenCarter44]

Also, this branch is for packaging up xGFabric to run on NERSC only. It may have broke things if running at other sites.

The purpose of this branch is more of a drop-in "download-and-run" on NERSC.