Repository navigation
Flaky functional tests: a node briefly misses a just-created index or collection #3121
Description
Problem
Functional scenarios occasionally fail because a node does not see an index or a collection that was just created (or still sees one that was just reset), as if the cluster-wide index cache were briefly out of sync between nodes. Three occurrences since the first 2.61 beta, each passing on rerun:
| Date | Run / job | Scenario | Failure |
|---|---|---|---|
| 2026-10-30 | 41084946612 — ES9 driver (http) | features/IndexController.feature:48 "List indexes" |
Given an index "nyc-open-data" → PreconditionError: A public index named "nyc-open-data" already exists |
| 2026-11-02 | 41410081963 — http, Node 24, ES8 (#3119) | features/CollectionController.feature:171 "Create a new collection with mappings" |
I should see the collection "nyc-open-data":"green-taxi" → expected Array [] to contain 'green-taxi' |
| 2026-11-02 | 41416369508 — legacy:http, Node 24, ES8 (#3120) | features-legacy/kuzzle.feature:56 "Update collection mapping: dynamic field" |
updateMapping right after the collection creation → 412 services.storage.unknown_collection |
The functional tests send requests through nginx to a 3-node cluster, so consecutive requests of one scenario can hit different nodes.
To check before 2.61.0
- Whether this is new in 2.61: fix(storage): replace the index cache on reload instead of merging into it #3098 (fixes Index cache keeps an index a peer deleted before this node joined the cluster #3040) changed how a node loads its index cache (it now replaces it with what the storage holds). A load racing a concurrent create /
admin:resetDatabasecould drop an entry another node just added. Reruns were already frequent on 2026-10-22–24 (before 2.61), but their causes were not recorded. - Whether
collection:create/index:createreturn before every node's cache is updated (cluster sync is asynchronous), which would make these scenarios inherently racy rather than the code wrong.
Outcome expected: either a fix (if a real cache inconsistency), or the scenarios made robust to the asynchronous sync, documented here.
Activity
rolljee commentedon Nov 2, 2026
Analysis
Not a 2.61 regression. The index cache is synchronized across nodes by fire-and-forget broadcasts:
ClientAdapter.createCollectionupdates the local cache and emitscore:storage:collection:create:after,ClusterNode.onCollectionAddedpublishesAddCollectionwithout awaiting it (Publisher.send), and the API response goes back right away. A peer applies it when its subscriber gets to it (Subscriber.handleCollectionAddition). Same for index creation andindex:mDelete(admin:resetDatabase). Nothing in this path changed in 2.61.#3098's
replaceIndexesis not involved: it only runs on a reload (cluster join,kuzzle:state:live,admin:refreshIndexCache). On the two 2026-11-02 failures (attempt 1 of runs 41410081963 and 41416369508), all three nodes were ready before cucumber started, the failures came 23 s and 94 s later, and no reload ran (refreshIndexCacheis only called byfeatures/CollectionLock.feature, the legacy suite never calls it). No cluster warning in the node logs either.The functional tests go through nginx, which spreads consecutive requests of one scenario across the 3 nodes:
collection:createanswered by node A, thencollection:list/updateMapping/index:createserved by node B before B has applied A's broadcast. The window is a few milliseconds, so it shows up only under CI load. A client behind a load balancer can hit the same window in production.(Side note, found while reading
replaceIndexes: a collection removed then re-created while a reload reads storage is judged unchanged byreconcileCollections(cached before and after) and takes the stored state, which may predate the re-creation. Only during a reload, so not the cause here; to keep in mind.)Options
chore(release): 2.61.0-beta.4 [skip ci]
kuzzlebot commentedon Nov 2, 2026
🎉 This issue has been resolved in version 2.61.0-beta.4 🎉
The release is available on:
Your semantic-release bot 📦🚀
chore(release): 2.61.0-beta.6 [skip ci]