Problem in running main.py

[denabazazian]

I have this error when I am running the python3 main.py InceptionTime :

iter 0
dataset_name: Coffee
Already_done /media/Disk1/projects/Time_series/data/results/inception/TSC/ Coffee
dataset_name: Meat
Already_done /media/cttc/Disk1/projects/Time_series/data/results/inception/TSC/ Meat
iter 1
dataset_name: Coffee
Already_done /media/Disk1/projects/Time_series/data/results/inception/TSC_itr_1/ Coffee
dataset_name: Meat
Already_done /media/cttc/Disk1/projects/Time_series/data/results/inception/TSC_itr_1/ Meat
iter 2
dataset_name: Coffee
Already_done /media/Disk1/projects/Time_series/data/results/inception/TSC_itr_2/ Coffee
dataset_name: Meat
Already_done /media/cttc/Disk1/projects/Time_series/data/results/inception/TSC_itr_2/ Meat
iter 3
dataset_name: Coffee
Already_done /media/Disk1/projects/Time_series/data/results/inception/TSC_itr_3/ Coffee
dataset_name: Meat
Already_done /media/cttc/Disk1/projects/Time_series/data/results/inception/TSC_itr_3/ Meat
iter 4
dataset_name: Coffee
Using TensorFlow backend.
2020-02-18 11:55:41.451561: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
2020-02-18 11:55:41.553535: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:964] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-02-18 11:55:41.553936: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 0 with properties:
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545
pciBusID: 0000:0a:00.0
totalMemory: 10.76GiB freeMemory: 10.32GiB
2020-02-18 11:55:41.553949: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0
2020-02-18 11:55:41.742036: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-02-18 11:55:41.742067: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988] 0
2020-02-18 11:55:41.742072: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0: N
2020-02-18 11:55:41.742141: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9960 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080 Ti, pci bus id: 0000:0a:00.0, compute capability: 7.5)
Segmentation fault (core dumped)

I have 10.32GiB free memory on the GPU and I suppose this error didn't raise due to the lack of memory space since it would occupy 9960 MB in memory.
In any case I also tried to reduce the data size based on this line but it was not very clear about how can we manage it if the data is too big.
I have also tried to run it just with 'Coffee' by editing this line to UNIVARIATE_DATASET_NAMES = ['Coffee'], but the memory is still 9960 MB and I have the same error.
I have also tried to run python3 main.py InceptionTime_xp, but the same error happened.
I would appreciate it if you could provide any clue that why this error happens and how can I solve it.

[hfawaz]

Thanks for sharing the issue.
I am not sure if such problem is due to using another version of tensorflow.
I will be updating the repository soon to make sure we are using tensorflow 2.1
Stay tuned for the update.

[denabazazian]

Thanks for your quick reply @hfawaz .
I have installed the same tensorflow version (1.12.0) as you mentioned in requirements. I don't think if the problem is related to the TF version.

[santiagoecheverri94]

Hey! Joining the conversation since I also ran into this problem recently. Fortunately, I was able to overcome it by creating the following conda environment in Linux via this environment.yml file.

# Conda environment file. Tested effectively on Nvidia T4 on Linux. Did not work on A100.
# Ensure `export TF_FORCE_GPU_ALLOW_GROWTH="true"` in your env before running `main.py`
name: InceptionTime
channels:
    - defaults
    - conda-forge
    - anaconda
dependencies:
    - python=3.6
    - keras=2.2.4
    - tensorflow-gpu=1.12.0
    - pandas==0.23.0
    - protobuf==3.6.1
    - pip
    - pip:
        - sklearn==0.0
        - matplotlib==1.5.3

@hfawaz , if you think this could be helpful to others, I'd be happy to open a small pull request for this. Please let me know if you'd think it worthwhile. Also, please allow me the opportunity to thank you for all your work in deep learning and your generosity with your knowledge. All the best to you.