[Bug]: Prisma reconnection failed

[xiaocongji]

Check for existing issues

• I have searched the existing issues and checked that my issue is not a duplicate.

What happened?

We've been experiencing periodic instability in our LiteLLM proxy pod. The issue manifests as the Prisma query engine process crashing:

13:50:30 - LiteLLM Proxy:WARNING: utils.py:4178 - Attempting Prisma DB reconnect. reason=db_health_watchdog_connection_error
13:50:30 - LiteLLM Proxy:ERROR: utils.py:3918 - prisma-query-engine PID 71 exited (waitpid thread); triggering reconnect.

Once the query engine crashes, all subsequent database calls fail immediately:

prisma.errors.ClientNotConnectedError: Client is not connected to the query engine, you must call `connect()` before attempting to query data.

LiteLLM detects the persistent failures and performs a clean shutdown, triggering a Kubernetes pod restart. This restart cycle repeated 4 times before the reconnect finally succeeded and the pod reached a stable state on the 5th attempt.

We have two questions:

1. Is this a known bug in LiteLLM or Prisma, or is this expected behavior under certain conditions?
2. Are there any configuration changes (e.g., connection pool tuning, reconnect retry settings, keepalive parameters) we can apply to either prevent the query engine from crashing or make the reconnect more resilient so it succeeds on the first attempt rather than requiring multiple pod restarts?

---

Steps to Reproduce

The issue happens intermittently, not easy to reproduce

Relevant log output

What part of LiteLLM is this about?

Proxy

What LiteLLM version are you on ?

1.83.10-stable

Twitter / LinkedIn details

No response

[RocketRider]

I am facing the same issue on v1.83.14 with the addtional issue that I am using the "separateHealthApp" and therefore the issue does not get caught by kubernetes

05:09:19 - LiteLLM Proxy:ERROR: proxy_server.py:4912 - litellm.proxy_server.py::add_deployment() - Error getting new models from DB - Client is not connected to the query engine, you must call `connect()` before attempting to query data.
Traceback (most recent call last):
  File "/app/litellm/proxy/proxy_server.py", line 4910, in _get_models_from_db
    new_models = await prisma_client.db.litellm_proxymodeltable.find_many()
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.13/site-packages/prisma/actions.py", line 2539, in find_many
    resp = await self._client._execute(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<11 lines>...
    )
    ^
  File "/app/.venv/lib/python3.13/site-packages/prisma/client.py", line 663, in _execute
    return await self._engine.query(builder.build(), tx_id=self._tx_id)
                 ^^^^^^^^^^^^
  File "/app/.venv/lib/python3.13/site-packages/prisma/client.py", line 701, in _engine
    raise errors.ClientNotConnectedError()
prisma.errors.ClientNotConnectedError: Client is not connected to the query engine, you must call `connect()` before attempting to query data.
05:09:19 - LiteLLM Proxy:ERROR: utils.py:2721 - LiteLLM Prisma Client Exception get_generic_data: Client is not connected to the query engine, you must call `connect()` before attempting to query data.
05:09:19 - LiteLLM Proxy:ERROR: utils.py:2721 - LiteLLM Prisma Client Exception get_generic_data: Client is not connected to the query engine, you must call `connect()` before attempting to query data.
05:09:19 - LiteLLM Proxy:ERROR: utils.py:2721 - LiteLLM Prisma Client Exception get_generic_data: Client is not connected to the query engine, you must call `connect()` before attempting to query data.
05:09:19 - LiteLLM Proxy:ERROR: utils.py:2721 - LiteLLM Prisma Client Exception get_generic_data: Client is not connected to the query engine, you must call `connect()` before attempting to query data.
05:09:19 - LiteLLM Proxy:WARNING: proxy_server.py:4228 - Failed to load config in _update_llm_router: Client is not connected to the query engine, you must call `connect()` before attempting to query data.. Proceeding with model loading using cached/empty config.
05:09:19 - LiteLLM Proxy:ERROR: proxy_server.py:4957 - litellm.proxy.proxy_server.py::ProxyConfig:add_deployment - Client is not connected to the query engine, you must call `connect()` before attempting to query data.
Traceback (most recent call last):
  File "/app/litellm/proxy/proxy_server.py", line 4939, in add_deployment
    await self._update_llm_router(
        new_models=new_models, proxy_logging_obj=proxy_logging_obj
    )
  File "/app/litellm/proxy/proxy_server.py", line 4277, in _update_llm_router
    await self._add_router_settings_from_db_config(
        config_data=config_data, llm_router=llm_router, prisma_client=prisma_client
    )
  File "/app/litellm/proxy/proxy_server.py", line 4490, in _add_router_settings_from_db_config
    db_router_settings = await prisma_client.db.litellm_config.find_first(
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        where={"param_name": "router_settings"}
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/app/.venv/lib/python3.13/site-packages/prisma/actions.py", line 21071, in find_first
    resp = await self._client._execute(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<10 lines>...
    )
    ^
  File "/app/.venv/lib/python3.13/site-packages/prisma/client.py", line 663, in _execute
    return await self._engine.query(builder.build(), tx_id=self._tx_id)
                 ^^^^^^^^^^^^
  File "/app/.venv/lib/python3.13/site-packages/prisma/client.py", line 701, in _engine
    raise errors.ClientNotConnectedError()
prisma.errors.ClientNotConnectedError: Client is not connected to the query engine, you must call `connect()` before attempting to query data.
05:09:19 - LiteLLM Proxy:ERROR: proxy_server.py:5725 - litellm.proxy_server.py::get_credentials() - Error getting credentials from DB - Client is not connected to the query engine, you must call `connect()` before attempting to query data.
Traceback (most recent call last):
  File "/app/litellm/proxy/proxy_server.py", line 5716, in get_credentials
    credentials = await prisma_client.db.litellm_credentialstable.find_many()
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.13/site-packages/prisma/actions.py", line 1501, in find_many
    resp = await self._client._execute(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<11 lines>...
    )
    ^
  File "/app/.venv/lib/python3.13/site-packages/prisma/client.py", line 663, in _execute
    return await self._engine.query(builder.build(), tx_id=self._tx_id)
                 ^^^^^^^^^^^^
  File "/app/.venv/lib/python3.13/site-packages/prisma/client.py", line 701, in _engine
    raise errors.ClientNotConnectedError()
prisma.errors.ClientNotConnectedError: Client is not connected to the query engine, you must call `connect()` before attempting to query data.
INFO:     169.254.3.1:57814 - "GET /health/liveliness HTTP/1.1" 200 OK
INFO:     169.254.3.1:57830 - "GET /health/readiness HTTP/1.1" 200 OK
05:09:24 - LiteLLM Proxy:WARNING: utils.py:4176 - Escalating to heavy reconnect after 1605 consecutive failures. reason=db_health_watchdog_connection_error
05:09:24 - LiteLLM Proxy:WARNING: utils.py:4183 - Attempting Prisma DB reconnect. reason=db_health_watchdog_connection_error
05:09:24 - LiteLLM Proxy:WARNING: utils.py:4112 - prisma-query-engine PID 0 is dead; reconnecting.
05:09:24 - LiteLLM Proxy:ERROR: utils.py:4197 - Prisma DB reconnect failed (1606 consecutive). reason=db_health_watchdog_connection_error error=Client is not connected to the query engine, you must call `connect()` before attempting to query data.
INFO:     169.254.3.1:46462 - "GET /health/liveliness HTTP/1.1" 200 OK
INFO:     169.254.3.1:46464 - "GET /health/readiness HTTP/1.1" 200 OK
INFO:     169.254.3.1:32954 - "GET /health/liveliness HTTP/1.1" 200 OK
INFO:     169.254.3.1:32970 - "GET /health/readiness HTTP/1.1" 200 OK
05:09:44 - LiteLLM Proxy:WARNING: utils.py:4176 - Escalating to heavy reconnect after 1606 consecutive failures. reason=auth_get_key_object_lookup_failure
05:09:44 - LiteLLM Proxy:WARNING: utils.py:4183 - Attempting Prisma DB reconnect. reason=auth_get_key_object_lookup_failure
05:09:44 - LiteLLM Proxy:WARNING: utils.py:4112 - prisma-query-engine PID 0 is dead; reconnecting.
05:09:44 - LiteLLM Proxy:ERROR: utils.py:4197 - Prisma DB reconnect failed (1607 consecutive). reason=auth_get_key_object_lookup_failure error=Client is not connected to the query engine, you must call `connect()` before attempting to query data.
05:09:44 - LiteLLM Proxy:ERROR: auth_exception_handler.py:95 - litellm.proxy.proxy_server.user_api_key_auth(): Exception occured - Client is not connected to the query engine, you must call `connect()` before attempting to query data.
Requester IP Address:10.1.11.141
Traceback (most recent call last):
  File "/app/litellm/proxy/auth/user_api_key_auth.py", line 1194, in _user_api_key_auth_builder
    valid_token = await get_key_object(
                  ^^^^^^^^^^^^^^^^^^^^^
    ...<5 lines>...
    )
    ^
  File "/app/litellm/proxy/db/log_db_metrics.py", line 99, in wrapper
    raise e
  File "/app/litellm/proxy/db/log_db_metrics.py", line 42, in wrapper
    result = await func(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/litellm/proxy/auth/auth_checks.py", line 2327, in get_key_object
    _valid_token: Optional[BaseModel] = await _fetch_key_object_from_db_with_reconnect(
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<4 lines>...
    )
    ^
  File "/app/litellm/proxy/auth/auth_checks.py", line 2231, in _fetch_key_object_from_db_with_reconnect
    return await prisma_client.get_data(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<4 lines>...
    )
    ^
  File "/app/.venv/lib/python3.13/site-packages/backoff/_async.py", line 151, in retry
    ret = await target(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/litellm/proxy/db/log_db_metrics.py", line 99, in wrapper
    raise e
  File "/app/litellm/proxy/db/log_db_metrics.py", line 42, in wrapper
    result = await func(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/litellm/proxy/utils.py", line 3200, in get_data
    raise e
  File "/app/litellm/proxy/utils.py", line 3112, in get_data
    response = await self._query_first_with_cached_plan_fallback(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        sql_query, hashed_token
        ^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/app/litellm/proxy/utils.py", line 2757, in _query_first_with_cached_plan_fallback
    return await self.db.query_first(sql_query, *args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.13/site-packages/prisma/client.py", line 524, in query_first
    results = await self.query_raw(query, *args)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.13/site-packages/prisma/client.py", line 559, in query_raw
    resp = await self._execute(
           ^^^^^^^^^^^^^^^^^^^^
    ...<6 lines>...
    )
    ^
  File "/app/.venv/lib/python3.13/site-packages/prisma/client.py", line 663, in _execute
    return await self._engine.query(builder.build(), tx_id=self._tx_id)
                 ^^^^^^^^^^^^
  File "/app/.venv/lib/python3.13/site-packages/prisma/client.py", line 701, in _engine
    raise errors.ClientNotConnectedError()
prisma.errors.ClientNotConnectedError: Client is not connected to the query engine, you must call `connect()` before attempting to query data.
INFO:     10.1.11.141:41662 - "GET /litellm/spend/logs/ui?start_date=2026-05-04+17%3A11%3A00&end_date=2026-05-06+05%3A09%3A43&page=1&page_size=50&sort_by=startTime&sort_order=desc HTTP/1.1" 401 Unauthorized
05:09:49 - LiteLLM Proxy:ERROR: proxy_server.py:4912 - litellm.proxy_server.py::add_deployment() - Error getting new models from DB - Client is not connected to the query engine, you must call `connect()` before attempting to query data.
Traceback (most recent call last):
  File "/app/litellm/proxy/proxy_server.py", line 4910, in _get_models_from_db
    new_models = await prisma_client.db.litellm_proxymodeltable.find_many()
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.13/site-packages/prisma/actions.py", line 2539, in find_many
    resp = await self._client._execute(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<11 lines>...
    )
    ^
  File "/app/.venv/lib/python3.13/site-packages/prisma/client.py", line 663, in _execute
    return await self._engine.query(builder.build(), tx_id=self._tx_id)
                 ^^^^^^^^^^^^
  File "/app/.venv/lib/python3.13/site-packages/prisma/client.py", line 701, in _engine
    raise errors.ClientNotConnectedError()
prisma.errors.ClientNotConnectedError: Client is not connected to the query engine, you must call `connect()` before attempting to query data.
05:09:49 - LiteLLM Proxy:ERROR: utils.py:2721 - LiteLLM Prisma Client Exception get_generic_data: Client is not connected to the query engine, you must call `connect()` before attempting to query data.
05:09:49 - LiteLLM Proxy:ERROR: utils.py:2721 - LiteLLM Prisma Client Exception get_generic_data: Client is not connected to the query engine, you must call `connect()` before attempting to query data.
05:09:49 - LiteLLM Proxy:ERROR: utils.py:2721 - LiteLLM Prisma Client Exception get_generic_data: Client is not connected to the query engine, you must call `connect()` before attempting to query data.
05:09:49 - LiteLLM Proxy:ERROR: utils.py:2721 - LiteLLM Prisma Client Exception get_generic_data: Client is not connected to the query engine, you must call `connect()` before attempting to query data.
05:09:49 - LiteLLM Proxy:WARNING: proxy_server.py:4228 - Failed to load config in _update_llm_router: Client is not connected to the query engine, you must call `connect()` before attempting to query data.. Proceeding with model loading using cached/empty config.
05:09:49 - LiteLLM Proxy:ERROR: proxy_server.py:4957 - litellm.proxy.proxy_server.py::ProxyConfig:add_deployment - Client is not connected to the query engine, you must call `connect()` before attempting to query data.
Traceback (most recent call last):
  File "/app/litellm/proxy/proxy_server.py", line 4939, in add_deployment
    await self._update_llm_router(
        new_models=new_models, proxy_logging_obj=proxy_logging_obj
    )
  File "/app/litellm/proxy/proxy_server.py", line 4277, in _update_llm_router
    await self._add_router_settings_from_db_config(
        config_data=config_data, llm_router=llm_router, prisma_client=prisma_client
    )
  File "/app/litellm/proxy/proxy_server.py", line 4490, in _add_router_settings_from_db_config
    db_router_settings = await prisma_client.db.litellm_config.find_first(
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        where={"param_name": "router_settings"}
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/app/.venv/lib/python3.13/site-packages/prisma/actions.py", line 21071, in find_first
    resp = await self._client._execute(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<10 lines>...
    )
    ^
  File "/app/.venv/lib/python3.13/site-packages/prisma/client.py", line 663, in _execute
    return await self._engine.query(builder.build(), tx_id=self._tx_id)
                 ^^^^^^^^^^^^
  File "/app/.venv/lib/python3.13/site-packages/prisma/client.py", line 701, in _engine
    raise errors.ClientNotConnectedError()
prisma.errors.ClientNotConnectedError: Client is not connected to the query engine, you must call `connect()` before attempting to query data.
05:09:49 - LiteLLM Proxy:ERROR: proxy_server.py:5725 - litellm.proxy_server.py::get_credentials() - Error getting credentials from DB - Client is not connected to the query engine, you must call `connect()` before attempting to query data.
Traceback (most recent call last):
  File "/app/litellm/proxy/proxy_server.py", line 5716, in get_credentials
    credentials = await prisma_client.db.litellm_credentialstable.find_many()
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.13/site-packages/prisma/actions.py", line 1501, in find_many
    resp = await self._client._execute(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<11 lines>...
    )
    ^
  File "/app/.venv/lib/python3.13/site-packages/prisma/client.py", line 663, in _execute
    return await self._engine.query(builder.build(), tx_id=self._tx_id)
                 ^^^^^^^^^^^^
  File "/app/.venv/lib/python3.13/site-packages/prisma/client.py", line 701, in _engine
    raise errors.ClientNotConnectedError()
prisma.errors.ClientNotConnectedError: Client is not connected to the query engine, you must call `connect()` before attempting to query data.
INFO:     169.254.3.1:55744 - "GET /health/liveliness HTTP/1.1" 200 OK
INFO:     169.254.3.1:55760 - "GET /health/readiness HTTP/1.1" 200 OK

A manual restart of the litellm pod worked for me, but this is no solution...

[krrish-berri-2]

this isn't a bug on litellm. it looks like your db is unresponsive.

.connect() is litellm trying to connect to your db and being unable to do so.

[krrish-berri-2]

Let me know if i'm missing anything here.

cc: @yuneng-berri who's overlooking the prisma errors.

[RocketRider]

For me the db was definitely responsive and working as expected.
One after the other litellm got stuck in this loop.
After restarting a litellm pod it reconnected perfectly.

[xiaocongji]

@krrish-berri-2 For me the same, we use aws postgres, I have checked the health status of the db, no downgraded. Also after a couple try, litellm can connect to the db

[thiago-juro]

We saw the same error today. There wasn't anything wrong with database..

[juris]

Same here. Nothing wrong with the DB. Two LiteLLM ghcr.io/berriai/litellm:v1.83.10-stable containers lost connection and got stuck in an infinite reconnection loop consuming 100% CPU. Restart was the only option.

[harish-berri]

Hi,

I’d suggest making the Prisma health watchdog a lot less aggressive while we work on a proper fix.

My guess here is that, the watchdog’s SELECT 1 check is failing, LiteLLM then tries to reconnect Prisma, and that reconnect path recreates the Prisma query engine. In your case it looks like this can repeat without the worker fully recovering.

Could you try the following env changes?

PRISMA_HEALTH_WATCHDOG_INTERVAL_SECONDS: "120"
PRISMA_HEALTH_WATCHDOG_PROBE_TIMEOUT_SECONDS: "15"
PRISMA_WATCHDOG_RECONNECT_TIMEOUT_SECONDS: "60"
PRISMA_RECONNECT_COOLDOWN_SECONDS: "120"Rationale here is to make the watchdog checks less aggressive so that in case of any transient failures, the query engine is not torn down and reconstructed.

If possible, I’d also recommend running NUM_WORKERS=1 per pod and scaling out with HPA instead. Each worker has its own Prisma engine and DB connection pool, so this makes failures easier for Kubernetes to isolate and replace.

I wouldn’t disable the watchdog yet unless this still happens with the above settings, since disabling it means we may take longer to notice a genuinely broken DB connection.

From my investigation: Prisma’s client runs as a separate process by design, and LiteLLM tries to manage and recover that process. When recovery fails, the proxy keeps running but database operations fail with connection errors.

[fmanigrasso]

Facing the exact same issue in production. The DB is healthy and responsive (Aurora) - this is purely a LiteLLM/Prisma reconnection loop problem. The only resolution we've found is a manual pod restart, which is not viable long-term.

This is a major blocker for adopting LiteLLM at scale. We'd really appreciate a proper fix rather than just tuning the watchdog intervals. Happy to provide additional logs or help test a patch if that's useful.

[tanmay2k]

@fmanigrasso +1 facing the same issue.