Wrap fits in MPI

[mdekauwe]

Should be trivial to do - wrap the fitting logic within the MPI stuff. Something like the below...i.e. instead of breaking up file list between cores, we just send curves to different cores

• note we will need to put all the file writing into a pandas df and write it all at the end.
• this does lead to a dependance on the pandas lib, but it would be optional and I'd prefer this.

def mpi_wrapper(flist):

    # setup multiprocessor stuff
    num_processors = mp.cpu_count()
    chunk_size = int(np.ceil(len(flist) / float(num_processors)))
    pool = mp.Pool(processes=num_processors)
    queue = mp.Queue() # define an output queue

    # break up the files list equally between prcoessors, of course it won't
    # quite fit eqaully so account for this
    processes = []
    for i in xrange(num_processors):

        start = chunk_size * i
        end = chunk_size * (i + 1)
        if end > len(flist):
            end = len(flist)

        # setup a list of processes that we want to run
        p = mp.Process(target=worker,
                       args=(queue, flist[start:end]))
        processes.append(p)

    # Run processes
    for p in processes:
        p.start()

    # OS pipes are not infinitely long, so the process queue can get blocked
    # when using the put command - a "deadlock"
    # The following logic gets around this...

    # Get process results from the output queue
    #results = [queue.get() for p in processes]
    results = []
    while True:
        while not queue.empty():
           results.append(queue.get())
        if not any(p.is_alive() for p in processes):
            break

    # Not entirely clear if this bit is still needed, or if the above covers it. Doesn't seem to 
    # do any harm...
    #
    # Exit the completed processes
    for p in processes:
        p.join()

    return results

def worker(output, flist):

    for fname in flist:
        # blah

    output.put(df_out)

'''