file structure

[ris130]

Thank you for developing and releasing this tool. I am currently using weighted ensemble in a project (using the westpa implementation) and I think that your method could solve some problems. However, I am not sure how to generate the fixed_wt_cycle_based_dictionary, wt_cycle_based_dictionary, and M_stored_w2_counts_lig50_lag100.pkl files, or what their structure is.
Could you please provide some example files for reference?
Best,
Riccardo

[SamikBose]

Hi Riccardo,

Sorry for the delayed response!

At this point the codes and the file formats are directed towards building MSMs from the WE data from our own software wepy (https://github.com/ADicksonLab/wepy). But I will be interested to tweak the codes for WESTPA implementation. The method is independent of the underlying WE algorithm but the dependence come from the different ways the resampling information are weights are stored. We can discuss a WESTPA implementation over email if you want to collaborate. (I have just replied to your email too)

As per the questions:
(1) fixed_wt_cycle_based_dictionary:
Here, we go over all the cycles one by one to get rid of the any merging effect on the weights post that cycle. So, if we are in cycle A, all the cycles till A (cyc_idx = 0 to A-1) will have same weights directly from WE simulation but all the cycles from A to Num_cyc will have 'fixed' or modified weights where the effect of merging will be removed (no accumulation of wts). This is instrumental to build the analytically correct counts matrix and missing transition matrix (M) in equation 2 here: https://pubs.acs.org/doi/epdf/10.1021/acs.jctc.4c01141. So for each cycle index A, we have a modified weight array which has shape of [num_walker X num_cyc] whose first A columns are directly from WE simulation weights and next A to num_cyc cols are where the merging events are removed. Now, we have total num_cyc cycles in our WE data anyway. So if modify/fix weights at each cycle the overall file structure can have a 3-D array (num_cyc, num_walker, num_cyc) shape or more conveniently we can use a dictionary (fixed_wt_cycle_based_dictionary), whose 'keys' are each cycle index from where the subsequent cycle wts are fixed. The 'values' corresponding to each key is a np.array of [nun_walkers, num_cycles] shape, containing the modified weights. You can check this section here, Line #216 onwards in the code:

` print('Building the merge fixed weights...')
wt_dict_merge_fixed = {cyc: np.zeros((n_walkers, n_cycles)) for cyc in range(n_cycles)}

    for cyc in range(n_cycles):
        test_arr = np.zeros((n_walkers, n_cycles))
        # cycles prior to the current cycle: append weights from h5 directly
        test_arr[:, 0:cyc+1] = WE_weights[:,0:cyc+1]
        # move forward in cycle (we correct the wts from here on...)
        for cycle_idx in range(cyc, n_cycles-1):  # refer to the wepy flowchart for the range.
            current_cyc_wt = test_arr[:,cycle_idx]    # store the current cycle wts.
            cloning_record = cl_dict[cycle_idx]   # check the cloning record.
            next_cycle = cycle_idx+1
            test_arr[:, next_cycle] = current_cyc_wt  # pre-load the next cycle weights with current wts
            # check for cloning
            if cloning_record:
                # check the format of the cloning_record; each key is a parent walker (clonee) and corresponding list
                # are the walkers into which it got cloned.
                # these are the walkers where the parent walker have cloned      
                for cloning_parent, cloned_walker_list in cloning_record.items():
                    # distributed weights (add one to include the parent walkers as well)
                    split_wt = current_cyc_wt[cloning_parent]/(len(cloned_walker_list[0])+1)  # distributed weights
                    # update weight of the parent walker
                    test_arr[cloning_parent,next_cycle] = split_wt
                    # update weight of the squashed walkers
                    test_arr[cloned_walker_list, next_cycle] = split_wt     # update weight of the squashed walkers
        # for a particular 'key' cycles add the corresponding (n_walker, n_cycles) dimension of array
        wt_dict_merge_fixed[cyc] = test_arr
        # print('after loop test arr:', test_arr)
    save_file(wt_dict_merge_fixed, wt_fixed_path)

`

The function

MBC_MSM/build_mbc_counts_mat.py

Line 187 in 539358c

def load_fixed_weights(wt_fixed_path, WE_weights, cl_dict, n_walkers, n_cycles):

actually calculates the fixed wts.

(2) wt_cycle_based_dictionary: I am not exactly sure which one are you talking about but I am assuming you are talking about something like this (

MBC_MSM/build_mbc_counts_mat.py

Line 503 in 539358c

we_wts_path_name = input_dict['we_wts']

) maybe? In that case, this is just the WE weights. Nothing much. Use it in whatever structure you are comfortable with but ensure the indices are correct.

(3) M files: This contains all the missing transitions (in a dictionary format again) that we have 'lost' due to merging events in WE simulation, when we want to build an MSM from WE data. Lets say the lag time of the MSM is 'tau' and number of states is 'n_clust'. The code for estimating the M dictionary from wepy data is here (and following 30 lines):

MBC_MSM/build_mbc_counts_mat.py

Line 445 in 539358c

if Calc_M == True:

. Again, since I am not very familiar to the WESTPA storage of resampling information, I can not really modify the code for you but if you want my help in rewriting this part for you, I am definitely ready to assist! The more detailed information about M can be found in our JCTC paper (pages 1809-1810 in the pdf).

With regards,
-Samik