From a8c49d9cfb439aa03f6d68f738a0ae8557cd07f1 Mon Sep 17 00:00:00 2001 From: Holmes Wilson Date: Thu, 21 May 2026 16:48:09 -0400 Subject: [PATCH] test(backend): add xfail repro for leaveCommunity teardown hang (#3243) leaveCommunity() awaits storageService.clean() directly. That teardown intermittently deadlocks in OrbitDB/IPFS channel-store cleanup, so leaveCommunity never reaches openSocket() and the Android app is stranded on "Starting backend" forever. Adds an it.failing (xfail) regression test that injects the failure (a storageService.clean() that never resolves) and asserts leaveCommunity still completes, plus docs/leave-community-teardown-hang.md with the full investigation - including why a naive timeout-based recovery does not work (the deadlocked teardown survives as a zombie and poisons the next storage operation). Does not fix the bug; see #3243. Co-Authored-By: Claude Opus 4.7 (1M context) --- .../docs/leave-community-teardown-hang.md | 120 ++++++++++++++++++ .../connections-manager.service.spec.ts | 39 ++++++ 2 files changed, 159 insertions(+) create mode 100644 packages/backend/docs/leave-community-teardown-hang.md diff --git a/packages/backend/docs/leave-community-teardown-hang.md b/packages/backend/docs/leave-community-teardown-hang.md new file mode 100644 index 0000000000..14c8a8d0b5 --- /dev/null +++ b/packages/backend/docs/leave-community-teardown-hang.md @@ -0,0 +1,120 @@ +# leaveCommunity teardown hang + +Tracking issue: [#3243](https://github.com/TryQuiet/quiet/issues/3243) + +This document records what was learned investigating an Android bug where Quiet +hangs on the **"Starting backend"** screen after leaving a community. It is +written for whoever picks up the real fix — the bug is **not fixed** by the PR +that adds this document; the PR only adds an `it.failing` (xfail) regression +test that reproduces it. + +## Symptom + +On Android, leaving a community — especially **soon after joining** — sometimes +strands the app on "Starting backend" forever. If you then try to rejoin, it +strands instead on "Connecting process started". Only force-stopping the app +recovers it. + +## Root cause + +`ConnectionsManagerService.leaveCommunity()` runs, in order: + +``` +closeAllServices() // closes the backend socket, stops libp2p, closes localDb +storageService.clean() // tears down OrbitDB stores + IPFS/helia +libp2pService.cleanDatastore() / closeDatastore() +storageService.purgeData() +tor.resetHiddenServices() +resetState() // communityState -> DEFAULT +localDbService.open() +openSocket() // reopens the backend socket +qssService.resume() +``` + +`storageService.clean()` **intermittently deadlocks** while cleaning a channel's +OrbitDB store. The last backend log line is: + +``` +INFO backend:storage:channels:channelStore:general Cleaning channel store general_ general +``` + +…and then the backend goes silent. Because `leaveCommunity()` `await`s +`clean()` directly and never returns: + +- It never reaches `openSocket()`. The backend socket — already closed by + `closeAllServices()` earlier in the same method — is **never reopened**, so the + frontend can't reach the backend and shows "Starting backend" indefinitely. +- `communityState` is left stuck at `LAUNCHED`. + +The same `storageService.clean()` is also called on the join path +(`joinCommunity()` → `erasePreviousCommunityArtifacts()`), so a rejoin after a +wedged leave hangs the same way ("Connecting process started"). + +The deadlock itself is **intermittent and timing-sensitive** — observed when +leaving a community shortly after joining it (tearing storage down before it has +settled). On the same device, one leave completes in ~2 s and the next one +deadlocks. The deadlock lives in OrbitDB / helia channel-store teardown; its +exact mechanism was not root-caused here. + +## Why the obvious fix does not work + +The tempting fix is to bound the teardown with a timeout and continue: + +```ts +try { + await withTimeout(teardown(), 30_000) +} finally { + await restoreApp() // resetState + localDbService.open + openSocket + qssService.resume +} +``` + +This was implemented and tested on device. It **does** un-stick the leave: after +the timeout the socket reopens and the app returns to the join screen. But it is +**not a real fix**: + +- `withTimeout` does not cancel the underlying promise. The deadlocked + `storageService.clean()` keeps running as a detached **zombie**, still holding + OrbitDB / IPFS instances, file handles and LevelDB locks. +- The next `joinCommunity()` calls `storageService.clean()` again (via + `erasePreviousCommunityArtifacts()`), on the same poisoned storage layer — and + hangs again. On device this reproduced exactly: leave "recovered" after the + timeout, then the rejoin stuck on "Connecting process started". +- The timeout value is a red herring. A genuine deadlock never completes, so + **any** timeout abandons a poisoning zombie. (A short timeout is worse still: + it also trips on merely-slow teardowns of large communities, where the + abandoned teardown then races the next join over the same OrbitDB/IPFS/fs + state.) + +In other words: **you cannot recover from a deadlocked storage layer in-process.** +The deadlocked instances and locks survive the timeout. The only thing that +reliably clears them is a fresh process — which is exactly what force-stopping +the app did. + +## Real fix directions + +1. **Fix the deadlock at the source** — determine why OrbitDB / helia + channel-store teardown in `storageService.clean()` deadlocks, and make it + hang-proof. This is the proper fix. +2. **If teardown cannot be made hang-proof, restart the backend process.** + Detect the hang and cleanly restart the nodejs-mobile backend (automating the + force-stop). A fresh process has no zombie. This needs the mobile host to + respawn the node process after exit. +3. **Decouple the backend socket lifecycle from community teardown**, so the UI + is never stranded on "Starting backend" even when teardown is slow or stuck. + +## Investigation notes / gotchas + +- **Logcat filtering**: the first capture used `adb logcat | grep com.quietmobile`, + which silently dropped every backend log line that did not contain a filesystem + path — making the backend look like it had gone silent right after libp2p + datastore init, and leading to an initial misdiagnosis. Always capture backend + logs by tag: `adb logcat -s NODEJS-MOBILE`. +- **Swiping the app away does not restart the backend.** A foreground service + keeps the nodejs-mobile process alive; reopening the app sends a `wake` that is + a no-op when the backend was never hibernated. Only a real force-stop restarts + the backend. +- Reproducing the deadlock deterministically in a test is not practical (it is a + race in OrbitDB/helia). The xfail test instead injects the failure — a + `storageService.clean()` that never resolves — and asserts `leaveCommunity()` + still completes. See `connections-manager.service.spec.ts` + ("leaveCommunity completes even when storage teardown hangs"). diff --git a/packages/backend/src/nest/connections-manager/connections-manager.service.spec.ts b/packages/backend/src/nest/connections-manager/connections-manager.service.spec.ts index 40b8c8001e..4073a61852 100644 --- a/packages/backend/src/nest/connections-manager/connections-manager.service.spec.ts +++ b/packages/backend/src/nest/connections-manager/connections-manager.service.spec.ts @@ -17,6 +17,7 @@ import { createLibp2pAddress, validInvitationDatav1 } from '@quiet/common' import { createLogger } from '../common/logger' import { SigChainService } from '../auth/sigchain.service' import { StorageModule } from '../storage/storage.module' +import { StorageService } from '../storage/storage.service' import { QSSService } from '../qss/qss.service' import { QSSSyncManager } from '../qss/qss-sync-manager.service' import { Libp2pEvents } from '../libp2p/libp2p.types' @@ -175,6 +176,44 @@ describe('ConnectionsManagerService', () => { expect(launchSpy).toHaveBeenCalledWith(community) }) + /** + * Reproduces the leaveCommunity teardown deadlock. + * See docs/leave-community-teardown-hang.md and issue #3243. + * + * leaveCommunity() awaits storageService.clean() directly. When that teardown + * deadlocks - as OrbitDB/IPFS channel-store cleanup intermittently does on + * device - leaveCommunity() never resolves, so it never reaches openSocket(): + * the backend socket stays closed and the app is stranded on "Starting + * backend" forever. + * + * Marked `it.failing` (xfail): the assertion below fails today because + * leaveCommunity hangs, which is how `it.failing` reports the test as passing. + * Once leaveCommunity is made resilient to a hung teardown, this test will + * genuinely pass and `it.failing` will fail - the signal to drop `.failing`. + */ + it.failing('leaveCommunity completes even when storage teardown hangs', async () => { + await localDbService.setCommunity(community) + await localDbService.setCurrentCommunityId(community.id) + + const storageService = await module.resolve(StorageService) + // Simulate the on-device OrbitDB/IPFS channel-store teardown deadlock. + const neverResolves = new Promise(() => { + // intentionally never settles + }) + jest.spyOn(storageService, 'clean').mockReturnValue(neverResolves) + // Isolate the test to the storage teardown hang. + jest.spyOn(connectionsManagerService, 'closeAllServices').mockResolvedValue() + jest.spyOn(qpsService, 'tombstoneCurrentUserNotificationTokens').mockResolvedValue(true) + + // leaveCommunity must not hang forever when teardown is stuck. + const TIMED_OUT = Symbol('timed-out') + const outcome = await Promise.race([ + connectionsManagerService.leaveCommunity(), + new Promise(resolve => setTimeout(() => resolve(TIMED_OUT), 2_000)), + ]) + expect(outcome).not.toBe(TIMED_OUT) + }) + it('pauses and resumes qss alongside the mobile lifecycle', async () => { const closeSocketSpy = jest.spyOn(connectionsManagerService, 'closeSocket').mockResolvedValue() const openSocketSpy = jest.spyOn(connectionsManagerService, 'openSocket').mockResolvedValue()