Skip to content

minidb query-store fails permanently on filesystems without hardlink support (link() EPERM), breaking kimi web #3535

Description

@CattieWish

Environment

  • OS: HarmonyOS (HongMeng Kernel 1.13.0, aarch64) — filesystem is hmdfs/tmpfs under /storage, which does not support hard links: link(2) always fails with EPERM
  • Node: v26.7.0
  • kimi-code: 0.40.1 (worked fine on 0.39.1)

Symptom

After upgrading to 0.40.1, kimi web starts and serves the UI, but agent turns stall mid-stream: subagent output freezes, quota/plugin info in the UI stops refreshing, and no requests reach the API. The TUI keeps working in the same environment.
The log is flooded (~20 entries/sec, 120k+ occurrences) with:

WARN failed to flush session index mirror chunk pending=1 failures=120918
  error="Error: EPERM: operation not permitted, link
  '.../cache/query-store/cluster.meta.json.tmp-64164' ->
  '.../cache/query-store/cluster.meta.json'"
WARN session index mirror giving up until the next record; reconciliation will heal
WARN session index read model degraded; serving authoritative reads reason="prepare failed"

Root cause

The new minidb query-store uses fs.promises.link(tmp, path) for atomic create-if-absent in several places and only tolerates EEXIST:

  • Topology.open (cluster/utils): writeFile(tmp) + link(tmp, cluster.meta.json) — throws on EPERM, so the cluster meta file is never created and every open attempt fails forever (even after the file exists, since link() fails with EPERM rather than EEXIST on such filesystems).
  • Lockfile tryCreate and snapshot generation use the same pattern.
    On filesystems without hardlink support this makes the whole query-store unusable, floods the log, and wedges the web server's event/session pipeline.

Suggested fix

Fall back when link() fails with EPERM (or ENOSYS), e.g.:

try {
  await fs.promises.link(tmp, dst);
} catch (e) {
  if (e.code === 'EPERM' || e.code === 'ENOSYS') {
    // copyFile with COPYFILE_EXCL gives the same atomic create-if-absent
    // semantics and throws EEXIST when the destination exists
    await fs.promises.copyFile(tmp, dst, fs.constants.COPYFILE_EXCL);
  } else throw e;
}

Workaround confirmed working

Preloading a small Node shim (NODE_OPTIONS=--import ...) that patches fs.promises.link with the copyFile fallback above fixes 0.40.1 completely on this machine: the query-store initializes, the log flood stops, and kimi web works normally. Setting KIMI_CODE_EXPERIMENTAL_PERSISTENCE_MINIDB_READMODEL=0 also avoids the issue by disabling the feature.
(That's how kimi code fix itself anyway.)

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions