Server DB connection overload

[sambles]

Reported that the platform running with postgres:16.3 is overload its DB number of connections and causing run failures.
Happens intermittent and during different parts of the execution.

Needs investigating and reproducing

Example Trace

server-db-1           | 2026-01-23 08:36:40.858 UTC [445] FATAL:  sorry, too many clients already
server-1              | ERROR 2026-01-23 08:36:40,863 log 233 140468336144512 Internal Server Error: /v2/analyses/8/
server-1              | Traceback (most recent call last):
server-1              |   File "/home/server/.local/lib/python3.12/site-packages/django/db/backends/base/base.py", line 279, in ensure_connection
server-1              |     self.connect()
server-1              |   File "/home/server/.local/lib/python3.12/site-packages/django/utils/asyncio.py", line 26, in inner
server-1              |     return func(*args, **kwargs)
server-1              |            ^^^^^^^^^^^^^^^^^^^^^
server-1              |   File "/home/server/.local/lib/python3.12/site-packages/django/db/backends/base/base.py", line 256, in connect
server-1              |     self.connection = self.get_new_connection(conn_params)
server-1              |                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
server-1              |   File "/home/server/.local/lib/python3.12/site-packages/django/utils/asyncio.py", line 26, in inner
server-1              |     return func(*args, **kwargs)
server-1              |            ^^^^^^^^^^^^^^^^^^^^^
server-1              |   File "/home/server/.local/lib/python3.12/site-packages/django/db/backends/postgresql/base.py", line 332, in get_new_connection
server-1              |     connection = self.Database.connect(**conn_params)
server-1              |                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
server-1              |   File "/home/server/.local/lib/python3.12/site-packages/psycopg/connection.py", line 118, in connect
server-1              |     raise last_ex.with_traceback(None)
server-1              | psycopg.OperationalError: connection failed: connection to server at "172.18.0.3", port 5432 failed: FATAL:  sorry, too many clients already
server-1              | 
server-1              | The above exception was the direct cause of the following exception:
server-1              | 
server-1              | Traceback (most recent call last):
server-1              |   File "/home/server/.local/lib/python3.12/site-packages/django/core/handlers/exception.py", line 55, in inner
server-1              |     response = get_response(request)
server-1              |                ^^^^^^^^^^^^^^^^^^^^^
server-1              |   File "/home/server/.local/lib/python3.12/site-packages/django/core/handlers/base.py", line 197, in _get_response
server-1              |     response = wrapped_callback(request, *callback_args, **callback_kwargs)
server-1              |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
server-1              |   File "/home/server/.local/lib/python3.12/site-packages/django/views/decorators/csrf.py", line 65, in _view_wrapper
server-1              |     return view_func(request, *args, **kwargs)
server-1              |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
server-1              |   File "/home/server/.local/lib/python3.12/site-packages/rest_framework/viewsets.py", line 125, in view
server-1              |     return self.dispatch(request, *args, **kwargs)
server-1              |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
server-1              |   File "/home/server/.local/lib/python3.12/site-packages/rest_framework/views.py", line 509, in dispatch
server-1              |     response = self.handle_exception(exc)
server-1              |                ^^^^^^^^^^^^^^^^^^^^^^^^^^
server-1              |   File "/home/server/.local/lib/python3.12/site-packages/rest_framework/views.py", line 469, in handle_exception
server-1              |     self.raise_uncaught_exception(exc)
server-1              |   File "/home/server/.local/lib/python3.12/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
server-1              |     raise exc
server-1              |   File "/home/server/.local/lib/python3.12/site-packages/rest_framework/views.py", line 497, in dispatch
server-1              |     self.initial(request, *args, **kwargs)
server-1              |   File "/home/server/.local/lib/python3.12/site-packages/rest_framework/views.py", line 414, in initial
server-1              |     self.perform_authentication(request)
server-1              |   File "/home/server/.local/lib/python3.12/site-packages/rest_framework/views.py", line 324, in perform_authentication
server-1              |     request.user
server-1              |   File "/home/server/.local/lib/python3.12/site-packages/rest_framework/request.py", line 227, in user
server-1              |     self._authenticate()
server-1              |   File "/home/server/.local/lib/python3.12/site-packages/rest_framework/request.py", line 380, in _authenticate
server-1              |     user_auth_tuple = authenticator.authenticate(self)
server-1              |                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
server-1              |   File "/home/server/.local/lib/python3.12/site-packages/rest_framework_simplejwt/authentication.py", line 51, in authenticate
server-1              |     return self.get_user(validated_token), validated_token
server-1              |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
server-1              |   File "/home/server/.local/lib/python3.12/site-packages/rest_framework_simplejwt/authentication.py", line 132, in get_user
server-1              |     user = self.user_model.objects.get(**{api_settings.USER_ID_FIELD: user_id})
server-1              |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
server-1              |   File "/home/server/.local/lib/python3.12/site-packages/django/db/models/manager.py", line 87, in manager_method
server-1              |     return getattr(self.get_queryset(), name)(*args, **kwargs)
server-1              |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
server-1              |   File "/home/server/.local/lib/python3.12/site-packages/django/db/models/query.py", line 631, in get
server-1              |     num = len(clone)
server-1              |           ^^^^^^^^^^
server-1              |   File "/home/server/.local/lib/python3.12/site-packages/django/db/models/query.py", line 368, in __len__
server-1              |     self._fetch_all()
server-1              |   File "/home/server/.local/lib/python3.12/site-packages/django/db/models/query.py", line 1954, in _fetch_all
server-1              |     self._result_cache = list(self._iterable_class(self))
server-1              |                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
server-1              |   File "/home/server/.local/lib/python3.12/site-packages/django/db/models/query.py", line 93, in __iter__
server-1              |     results = compiler.execute_sql(
server-1              |               ^^^^^^^^^^^^^^^^^^^^^
server-1              |   File "/home/server/.local/lib/python3.12/site-packages/django/db/models/sql/compiler.py", line 1621, in execute_sql
server-1              |     cursor = self.connection.cursor()
server-1              |              ^^^^^^^^^^^^^^^^^^^^^^^^
server-1              |   File "/home/server/.local/lib/python3.12/site-packages/django/utils/asyncio.py", line 26, in inner
server-1              |     return func(*args, **kwargs)
server-1              |            ^^^^^^^^^^^^^^^^^^^^^
server-1              |   File "/home/server/.local/lib/python3.12/site-packages/django/db/backends/base/base.py", line 320, in cursor
server-1              |     return self._cursor()
server-1              |            ^^^^^^^^^^^^^^
server-1              |   File "/home/server/.local/lib/python3.12/site-packages/django/db/backends/base/base.py", line 296, in _cursor
server-1              |     self.ensure_connection()
server-1              |   File "/home/server/.local/lib/python3.12/site-packages/django/utils/asyncio.py", line 26, in inner
server-1              |     return func(*args, **kwargs)
server-1              |            ^^^^^^^^^^^^^^^^^^^^^
server-1              |   File "/home/server/.local/lib/python3.12/site-packages/django/db/backends/base/base.py", line 278, in ensure_connection
server-1              |     with self.wrap_database_errors:
server-1              |   File "/home/server/.local/lib/python3.12/site-packages/django/db/utils.py", line 91, in __exit__
server-1              |     raise dj_exc_value.with_traceback(traceback) from exc_value
server-1              |   File "/home/server/.local/lib/python3.12/site-packages/django/db/backends/base/base.py", line 279, in ensure_connection
server-1              |     self.connect()
server-1              |   File "/home/server/.local/lib/python3.12/site-packages/django/utils/asyncio.py", line 26, in inner
server-1              |     return func(*args, **kwargs)
server-1              |            ^^^^^^^^^^^^^^^^^^^^^
server-1              |   File "/home/server/.local/lib/python3.12/site-packages/django/db/backends/base/base.py", line 256, in connect
server-1              |     self.connection = self.get_new_connection(conn_params)
server-1              |                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
server-1              |   File "/home/server/.local/lib/python3.12/site-packages/django/utils/asyncio.py", line 26, in inner
server-1              |     return func(*args, **kwargs)
server-1              |            ^^^^^^^^^^^^^^^^^^^^^
server-1              |   File "/home/server/.local/lib/python3.12/site-packages/django/db/backends/postgresql/base.py", line 332, in get_new_connection
server-1              |     connection = self.Database.connect(**conn_params)
server-1              |                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
server-1              |   File "/home/server/.local/lib/python3.12/site-packages/psycopg/connection.py", line 118, in connect
server-1              |     raise last_ex.with_traceback(None)
server-1              | django.db.utils.OperationalError: connection failed: connection to server at "172.18.0.3", port 5432 failed: FATAL:  sorry, too many clients already
server-db-1           | 2026-01-23 08:36:40.882 UTC [446] FATAL:  sorry, too many clients already

[sambles]

from @carlfischerjba

OasisPlatform/wsgi/run-wsgi.sh

Line 7 in 695e696

WORKERS=$(nproc --all)

The problem is from here, inside the container the incorrect number of cores is retruned, which sets gunicorn to a high number of workers

~/$ nproc --all
8
 ~/$ docker run --rm --entrypoint '' coreoasis/api_server:2.4.11 nproc --all
240

[carlfischerjba]

Seems to be a problem with Xeon Gold processors only.