Line 935 Error - KeyError: 'global_x_train_shape'

[Likelyt]

Level-Set-Boosting/LSBoost.py

Line 935 in 30643e9

x_train_shared = np.ndarray(globals()['global_x_train_shape'], dtype = np.float64, buffer=globals()['global_x_train_shm'].buf)

I tried to locally run it over Mac of ipynb file. However, it shows that "KeyError: 'global_x_train_shape'" in line 935 of LSBoost.py file. Do you know how to solve it?

Thank!

[Declancharrison]

Hm. That probably means the shared memory array failed to initialize. Can you post the whole error code? Also, did this occur while fitting the model or predicting with a fit model?

[Likelyt]

When I run the "LSBoostReg.fit(x_train[:10000], y_train[:10000])". It shows this:

LS: 0%| 0/100 ?, Training Error=0.0496468
Memory released!

RemoteTraceback Traceback (most recent call last)
RemoteTraceback:
"""
Traceback (most recent call last):
File "/Users/liyt/opt/anaconda3/envs/PE/lib/python3.8/multiprocessing/pool.py", line 125, in worker
result = (True, func(*args, **kwds))
File "/Users/liyt/opt/anaconda3/envs/PE/lib/python3.8/multiprocessing/pool.py", line 51, in starmapstar
return list(itertools.starmap(args[0], args[1]))
File "/Users/liyt/Documents/2023Spring/BIC/multi-calibration/code/Level-Set-Boosting-main/LSBoost.py", line 935, in worker
x_train_shared = np.ndarray(globals()['global_x_train_shape'], dtype = np.float64, buffer=globals()['global_x_train_shm'].buf)
KeyError: 'global_x_train_shape'
"""

The above exception was the direct cause of the following exception:

KeyError Traceback (most recent call last)
/var/folders/kx/qstn8wln7mv3jp6yrn13wff00000gn/T/ipykernel_72960/411476469.py in
2 # y_train = y_train[:1000], (437816,)
3 # use a subset of the data to speed up the process
----> 4 LSBoostReg.fit(x_train[:10000], y_train[:10000])

~/Documents/2023Spring/BIC/multi-calibration/code/Level-Set-Boosting-main/LSBoost.py in fit(self, x_train, y_train)
702 # rounds 1 and 2 updates
703 elif ((i == 0) or (i == 1)):
--> 704 self.update()
705 continue
706

~/Documents/2023Spring/BIC/multi-calibration/code/Level-Set-Boosting-main/LSBoost.py in update(self)
228
229 # fit weak learners in parallel over non-empty level sets
--> 230 result = executor.starmap(worker, zip(occupied_level_sets, repeat(self.round_count)))
231
232 # wait for all computations to terminate and close executor

~/opt/anaconda3/envs/PE/lib/python3.8/multiprocessing/pool.py in starmap(self, func, iterable, chunksize)
370 func and (a, b) becomes func(a, b).
371 '''
--> 372 return self._map_async(func, iterable, starmapstar, chunksize).get()
373
374 def starmap_async(self, func, iterable, chunksize=None, callback=None,

~/opt/anaconda3/envs/PE/lib/python3.8/multiprocessing/pool.py in get(self, timeout)
769 return self._value
770 else:
--> 771 raise self._value
772
773 def _set(self, i, obj):

KeyError: 'global_x_train_shape'

[Declancharrison]

It doesn't seem to be mapping the data into a shared memory region. Can you try running the following in a cell:

def create_shared_memory_nparray(data, name, create = False):
'''
Creates array in shared memory for parallel processing.
'''
d_size = np.dtype(np.float64).itemsize * np.prod(data.shape)
try:
shm = shared_memory.SharedMemory(create=create, size=d_size, name=f'{name}_buf')
except:
shm = shared_memory.SharedMemory(create=False, size=d_size, name=f'{name}_buf')
return shm

dst = np.ndarray(shape=data.shape, dtype=np.float64, buffer=shm.buf)
dst[:] = data[:]
globals()[f'global_{name}shm'] = shm
globals()[f'shms'].append(name)
globals()[f'global{name}_shape'] = data.shape
return shm

initialize shared data list

globals()[f'shms'] = []

share training data

create_shared_memory_nparray(x_train,'x_train', True)
create_shared_memory_nparray(y_train,'y_train', True)

try to access shared data

x_train_shared = np.ndarray(globals()['global_x_train_shape'], dtype = np.float64, buffer=globals()['global_x_train_shm'].buf)

[Likelyt]

It seems to give me the same error.

KeyError Traceback (most recent call last)
/var/folders/kx/qstn8wln7mv3jp6yrn13wff00000gn/T/ipykernel_2668/3331667428.py in
23 create_shared_memory_nparray(y_train,'y_train', True)
24
---> 25 x_train_shared = np.ndarray(globals()['global_x_train_shape'], dtype = np.float64, buffer=globals()['global_x_train_shm'].buf)

KeyError: 'global_x_train_shape'

from multiprocessing import shared_memory
def create_shared_memory_nparray(data, name, create = False):
    '''
    Creates array in shared memory for parallel processing.
    '''
    d_size = np.dtype(np.float64).itemsize * np.prod(data.shape)
    try:
        shm = shared_memory.SharedMemory(create=create, size=d_size, name=f'{name}_buf')
    except:
        shm = shared_memory.SharedMemory(create=False, size=d_size, name=f'{name}_buf')
        return shm

    dst = np.ndarray(shape=data.shape, dtype=np.float64, buffer=shm.buf)
    dst[:] = data[:]
    globals()[f'global_{name}shm'] = shm
    globals()[f'shms'].append(name)
    globals()[f'global{name}_shape'] = data.shape
    return shm

globals()[f'shms'] = []

create_shared_memory_nparray(x_train,'x_train', True)
create_shared_memory_nparray(y_train,'y_train', True)

x_train_shared = np.ndarray(globals()['global_x_train_shape'], dtype = np.float64, buffer=globals()['global_x_train_shm'].buf)

[Declancharrison]

You can try inserting the following into the top of the LSBoost.py file after importing multiprocessing: multiprocessing.set_start_method("fork"). This is a MacOS specific issue, and using 'fork' may cause the program to crash depending on if threading is used. It's worth a shot, and if not I may create a non-parallel version for MacOS or rework the parallel use cases.

[Likelyt]

Hi, I tried the code in the linux system and it works. It seems iOS system is not compatible with the code. Additionally, the graph function has error. Not sure whether the package is obsolete or not. However, it does not affect the whole project. Thanks!

[nastaran75]

I am getting the same error and adding multiprocessing.set_start_method("fork") does not seem to resolve it. Any further insights on what can be done to resolve this issue? Many thanks!