Skip to content

Usage-stats job loads connections without their password, so every run fails ClickHouse auth #3612

Closed

Description

@ahmedriza

What happened?

Every 4 hours the API logs one AUTHENTICATION_FAILED per saved connection that has a password, and ClickHouse logs a matching WRONG_PASSWORD for the connection's user from the HyperDX host:

[ERROR][@clickhouse/client][Connection] Query: HTTP request error.
...
Caused by: ClickHouseError: <user>: Authentication failed: password is incorrect, or there is no user with such name.
  code: '516',
  type: 'AUTHENTICATION_FAILED'

The same connections work everywhere else in HyperDX (search, dashboards, alerts), with the same stored password.

Cause: getClickhouseTableSize() in packages/api/src/tasks/usageStats.ts loads connections with

const connections = await Connection.find();

but the password field in packages/api/src/models/connection.ts is declared with select: false, so connection.password is undefined when the job builds its ClickhouseClient. The query on system.parts is sent with an empty password and rejected.

Expected: the job reads the password, as other code paths do via getConnectionById(..., selectPassword = true), e.g.

const connections = await Connection.find().select('+password');

(The usage stats the job reports are presumably empty or incomplete as a result, since the failure is swallowed by its catch.)

Steps to reproduce

Reproduces on play.hyperdx.io? didn't try (it needs a password-protected ClickHouse user and access to the API logs)

  1. Run HyperDX with USAGE_STATS_ENABLED unset (it defaults to on).
  2. Add a connection whose ClickHouse user has a non-empty password, and at least one source on it.
  3. Restart the API (the job runs at startup, then every 4h) and watch the API logs, or ClickHouse's system.text_log, for the auth failure.

How are you running HyperDX?

HyperDX-only image docker.hyperdx.io/hyperdx/hyperdx:2.40.0 under podman compose, with its own MongoDB and a self-managed ClickHouse 26.9. The code is unchanged on main as of commit 631002374 (2026-11-10).

Where does it show up?

API (background task tasks/usageStats).

Debug info

N/A (API background task).

Logs

See above. Workaround: set USAGE_STATS_ENABLED=false.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Needs FeedbackUnder discussion; not ready for a PR yetbugSomething isn't workingexternal

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions