minicord

[panda1100]

I'm trying to run minicord and I got following error. Any help is greatly appreciated.

My environment is following:

• python==3.9.14
• tornado==5.1.1
• requests==2.28.2

python3 -i run.py
INFO:run:Starting coordinator service on port 8888
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib64/python3.9/threading.py", line 980, in _bootstrap_inner
    self.run()
  File "/usr/lib64/python3.9/threading.py", line 917, in run
    self._target(*self._args, **self._kwargs)
  File "/root/python/minicoord/coord_service.py", line 54, in run
>>>     self.listen(self.port)
  File "/usr/local/lib64/python3.9/site-packages/tornado/web.py", line 2042, in listen
    server.listen(port, address)
  File "/usr/local/lib64/python3.9/site-packages/tornado/tcpserver.py", line 144, in listen
    self.add_sockets(sockets)
  File "/usr/local/lib64/python3.9/site-packages/tornado/tcpserver.py", line 157, in add_sockets
    self._handlers[sock.fileno()] = add_accept_handler(
  File "/usr/local/lib64/python3.9/site-packages/tornado/netutil.py", line 229, in add_accept_handler
    io_loop = IOLoop.current()
  File "/usr/local/lib64/python3.9/site-packages/tornado/ioloop.py", line 282, in current
    loop = asyncio.get_event_loop()
  File "/usr/lib64/python3.9/asyncio/events.py", line 642, in get_event_loop
    raise RuntimeError('There is no current event loop in thread %r.'
RuntimeError: There is no current event loop in thread 'Thread-1'.

[panda1100]

Under following environment it works by the way.

• python==3.6.8
• tornado==4.5.3
• requests==2.14.2

[panda1100]

More follow up

• python==3.9.14
• tornado==4.5.3
• requests==2.28.2

on Rocky Linux 9 works.

https://docs.openmoonray.org/developer-reference/arras/distributed-arras/#coordinator

[panda1100]

Coordinator <--> Node connection error

Error on Coordinator

python3 -i /installs/openmoonray/python/minicoord/run.py                                         
INFO:run:Starting coordinator service on port 8888
>>> ERROR:tornado.application:Uncaught exception POST /coordinator/1/nodes (::1)                            
HTTPServerRequest(protocol='http', host='localhost:8888', method='POST', uri='/coordinator/1/nodes', version='HTTP/1.1', remote_ip='::1', headers={'Host': 'localhost:8888', 'Accept': '*/*', 'Content-Type': 'application/json', 'User-Agent': 'Node Service', 'Content-Length': '1918'})                                            
Traceback (most recent call last):
  File "/usr/local/lib64/python3.9/site-packages/tornado/web.py", line 1510, in _execute                    
    result = method(*self.path_args, **self.path_kwargs)                                                    
  File "/installs/openmoonray/python/minicoord/coord_service.py", line 81, in post                          
    req = req.rstrip('\x00')
TypeError: a bytes-like object is required, not 'str'
ERROR:tornado.access:500 POST /coordinator/1/nodes (::1) 1.83ms

ERROR on Node

arras4_node --coordinator-host localhost --coordinator-port 8888 -l 5 --no-consul                
2023-04-09T09:36:10.513203 I node[4972]:main: Initializing Node ID dc4baacd-fced-4f20-a69b-30954a697347     
2023-04-09T09:36:10.513312 I node[4972]:main: Node memory 1073741824 bytes, computation memory 505383370752 bytes, total physical 506457112576 bytes
2023-04-09T09:36:10.515485 I node[4972]:main: 59 cores available for computations                           
2023-04-09T09:36:10.566280 D node[4972]:main: Control groups disabled                                       
2023-04-09T09:36:10.566312 D node[4972]:main: Current fd limit at 1024, setting to max of 262144            
2023-04-09T09:36:10.566444 I node[4972]:main: Node Address 10.0.0.97 [instance-20230316-0837]               
2023-04-09T09:36:10.566462 I node[4972]:main: Node using Coordinator URL http://localhost:8888/coordinator/1
2023-04-09T09:36:10.566555 D node[4972]:main: [8ef4ebd1-173d-4b4b-a48b-db9b21ff2a80]: Spawning: arras4_router '--nodeid' 'dc4baacd-fced-4f20-a69b-30954a697347' '--ipcName' '/tmp/arrasnodeipc-dc4baacd-fced-4f20-a69b-30954a697347' '-l' '5' '--athena-env' 'prod' '--athena-host' 'localhost' '--athena-port' '514'                 
( cwd='/home/opc'; Mb=0 Cores=0)
2023-04-09T09:36:10.566781 D node[4972]:main: [8ef4ebd1-173d-4b4b-a48b-db9b21ff2a80]: Spawned: arras4_router PID: 4999
2023-04-09T09:36:11.567067 D node[4972]:main: Retrying router connect (1)                                   
2023-04-09T09:36:11.567444 D node[4972]:handler: Received from router: RouterInfoMessage                    
2023-04-09T09:36:11.667677 I node[4972]:main: NodeService listening on HTTP port 46997                      
2023-04-09T09:36:11.681806 I node[4972]:main: Registering Node ID dc4baacd-fced-4f20-a69b-30954a697347 with Coordinator
2023-04-09T09:36:11.684813 E node[4972]:main: {NodeRegisterFail} (NodeService) Node Registration ('POST http://localhost:8888/coordinator/1/nodes') returned unacceptable status code 500(response body: '<html><title>500: Internal Server Error</title><body>500: Internal Server Error</body></html>')                             
2023-04-09T09:36:11.684900 E node[4972]:main: {NodeInitError} Error during initialization of Node : Node registration failed
pure virtual method called
terminate called without an active exception
2023-04-09 09:36:11,819 W default[4999]: Service EP receiveThread: service disconnected from node           
Aborted (core dumped)

[panda1100]

Coordinator works with python==2.7.18
Coordinator doesn't work with python==3.9.14

[tstraus13]

So I am always getting "Insufficient Resources available to fill this request". I am not sure where/why its failing. I have 3 nodes with 8 cores and 16GB RAM. I am using the sample session from the documentation. Each of the nodes are able to connect/register but as soon as I attempt to start a render I get that message. Any help is appreciated. Thanks.