(raylet) [2024-08-25 19:21:02,241 E 1111013 1111013] (raylet) node_manager.cc:3064: 12 Workers (tasks / actors) killed due to memory pressure (OOM), 0 Workers crashed due to other reasons at node (ID: b8aad7ac53c083fcae4851424e0362cec4a21916876b69bb1722ed29, IP: 172.31.105.218) over the last time period. To see more information about the Workers killed on this node, use ray logs raylet.out -ip 172.31.105.218
(raylet)
(raylet) Refer to the documentation on how to address the out of memory issue: https://docs.ray.io/en/latest/ray-core/scheduling/ray-oom-prevention.html. Consider provisioning more memory on this node or reducing task parallelism by requesting more CPUs per task. To adjust the kill threshold, set the environment variable RAY_memory_usage_threshold when starting Ray. To disable worker killing, set the environment variable RAY_memory_monitor_refresh_ms to zero.
(raylet) [2024-08-25 19:21:02,241 E 1111013 1111013] (raylet) node_manager.cc:3064: 12 Workers (tasks / actors) killed due to memory pressure (OOM), 0 Workers crashed due to other reasons at node (ID: b8aad7ac53c083fcae4851424e0362cec4a21916876b69bb1722ed29, IP: 172.31.105.218) over the last time period. To see more information about the Workers killed on this node, use
ray logs raylet.out -ip 172.31.105.218(raylet)
(raylet) Refer to the documentation on how to address the out of memory issue: https://docs.ray.io/en/latest/ray-core/scheduling/ray-oom-prevention.html. Consider provisioning more memory on this node or reducing task parallelism by requesting more CPUs per task. To adjust the kill threshold, set the environment variable
RAY_memory_usage_thresholdwhen starting Ray. To disable worker killing, set the environment variableRAY_memory_monitor_refresh_msto zero.