Skip to content

Stop obsolete folder counts during directory traversal - #119

Merged
dhh merged 3 commits into
omacom:mainfrom
k-kostin:codex/cancel-folder-counts
Nov 16, 2026
Merged

dhh merged 3 commits into
omacom:mainfrom
k-kostin:codex/cancel-folder-counts

Conversation

@k-kostin

Copy link
Copy Markdown
Contributor

Summary

Leaving a folder invalidates its count job, but a count already running still enumerates the entire child directory. Pass the existing generation predicate into that loop so navigation, refresh or model destruction can stop obsolete work.

invalidate generation → stop counting → return unknown (-1), never a partial total

Include hidden entries in the iterator, then exclude them from the total when hidden files are off. Otherwise Qt can filter a long hidden run internally without yielding to the cancellation check. Linux hidden-name semantics and exclusion of . / .. are preserved.

Based directly on main; no dependency on #67, #117 or #118. Complements #116's pool separation by avoiding wasted counting itself. No dependency or thread-pool change.

Evidence

  • A real FolderModel with a delayed synthetic directory continued reading 981 entries after navigation on the unchanged base. The final patch reads 0–1 additional visible entries, or 0 hidden entries, in the measured runs. No partial counter is published.
  • 50 controlled runs, five alternating pairs per scenario, cover navigation, refresh, destruction and hidden directories. Navigation latency in the one-worker harness is 543 → 10 ms median. This deliberately delays real readdir64() by 500 µs per entry; it measures queue contention, not ordinary SSD navigation. Give folder listings their own thread pool #116 may remove that listing contention independently.
  • Release build, focused tests and git diff --check pass. Final complete local suite: 197 passed, 0 failed, using Qt image plugins matching the CI dependency set. Regression covers cancellation during enumeration, before starting, empty/hidden directories and a complete retry. Independent review found no remaining issues.
  • One earlier full run failed the plugin process-cleanup test. That test passed standalone on both heads, followed by a passing full pair (base 196/0, candidate 197/0) and another full pass after the hidden-entry fix. No plugin code was changed.
  • Reapplying the change to the current Open and sort large folders faster #26/Give folder listings their own thread pool #116 heads preserves their changes and passes syntax compilation. Their surrounding edits cause patch-context conflicts; full combined-branch checks remain due after any rebase.

Merge Danger

Door: two-way.

Blast Radius: folder counters. Cancellation is cooperative between iterator advances and cannot interrupt an already blocked filesystem call. The previewer's separate folder-summary path is unchanged.

@greptile-apps

greptile-apps Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The PR appears safe to merge with the explicitly accepted delay when Qt skips hidden entries.

Summary

Adds a stop check inside FolderModel::itemCount() and passes the existing generation check from countFolders().

  • Folder counts stop when their listing becomes obsolete.

Reviews (2) · Last reviewed commit: "Check for a stale count in the existing ..." · Reviewed by Greptile

@omarchybot omarchybot added bug Something isn't working verified ready labels Nov 15, 2026
@omarchybot

Copy link
Copy Markdown
Contributor

Reviewed at 1a7e72d by Claude Opus 5.5, with Codex Medium and muse-wonder (via pi, thinking high) as second opinions. Nothing found to fix, and nothing was pushed.

The bug is real. On a disposable Arch worker, a model loaded a folder of four subfolders holding 400,000 files each and was sent elsewhere 30 ms into counting. On main (bb94499) the thread pool stayed busy 112–160 ms after leaving, and strace saw 330–339 more getdents64 reads (about 10.7 MB of directory entries) for a count whose result is thrown away; deleting the model mid-count behaved the same. On this head both cases went idle within 0–3 ms after one or two reads. Coming back to the folder, every subfolder was counted in full (400,000 each) on both.

The cancellation holds up. The predicate is checked before each iterator advance; a -1 only ever comes from an overtaken generation, and the finished handler drops those before touching m_entries or the listing cache, so neither a stale nor a partial count is written. The task captures only copies and the shared atomic, so destroying the model mid-count touches nothing it owns. Counting hidden entries by their leading dot gives the same totals as QDir::Hidden on Linux, with ./.., sockets and broken links treated as before.

Tests. bin/test on the worker: 197 passed on this head, 202 on main, and 203 with this head merged into main. Taking out the in-loop check, or filtering hidden entries inside Qt again, each makes folderCountStopsInsideTheFolderAndCanBeRetried fail, so the test is not vacuous. GitHub's own Test workflow has not run on this pull request: it is waiting for a maintainer to approve it for a first-time contributor.

Second opinions. Both found no defect and agreed with the above; they can read this review's notes, so that agreement is not guaranteed to be independent. muse-wonder pointed out that the previewer's summarize() still lists a whole folder after the selection moves on; that is older than this change and a separate follow-up, not something this pull request needs.

Related. #116 (separate pools for listings and counts) is complementary: it keeps counts from holding up listings, this stops the counts themselves, and with only two counting threads there this matters more. The two merge together without conflict. #92 and #89 change the same itemCount() call and will conflict textually with this one, whichever lands second.

Waiting on the maintainer.

The maintainer asked for the simplest version. The stop check sits in the loop itemCount() already had, as list() does, and hidden entries are filtered by Qt as before. Its only caller already checks before each folder and drops an overtaken result, so the checks before and after the loop add nothing, and one assertion in the existing count test covers stopping mid-folder. The price is that a long run of hidden entries, with hidden files off, is stepped over inside Qt before the check runs again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Codex Medium <noreply@openai.com>
@omarchybot

Copy link
Copy Markdown
Contributor

The maintainer asked for this pull request in its simplest form, so I pushed bf301a1 on top of your branch, @k-kostin. It keeps your fix and drops what the one caller doesn't need. The change is now +13/-6 (src +10/-6, tests +3) instead of +49/-7.

What changed, and why

  • The stop check sits in the loop itemCount() already had, the way list() checks its own. Hidden files are filtered by Qt again (entryFilters(showHidden)), instead of being read and dropped by name.
  • The checks before and after the loop are gone. countFolders() already checks overtaken() before each folder, and its finished handler drops any result from an overtaken generation before it reaches m_entries or the listing cache. A stale count returning -1, a partial total or a full total makes no difference to what the user sees.
  • The test is one assertion in folderSummaryCountsAndPeeks. A predicate that turns true on its second call gives -1 for a four-entry folder. The assertion fails if the in-loop check is removed, and also if the check sits only before the loop.

What this gives up. With hidden files off, a long run of hidden entries inside one subfolder is stepped over inside Qt before the check runs again. Cancellation can lag across that run. The wasted work is limited to that run, and no wrong count is ever shown. Your description still describes the hidden-entry handling and the larger test, so read this comment for the current state.

Verified at bf301a1 on a disposable Arch worker. A model loaded a folder of four subfolders holding 400,000 files each and moved away 30 ms into counting:

  • On main (41e9f41), the thread pool stayed busy 97–134 ms after leaving, and strace saw 311–326 more getdents64 reads (about 10.4 MB).
  • On this head, the pool went idle within 0–4 ms, after one to four reads.
  • Deleting the model mid-count behaved the same way on both.
  • Coming back, every folder was counted in full.

bin/test: 196 passed on this head, 202 on main, and 202 with this head merged into main.

Second opinions. Codex Medium and muse-wonder (via pi, thinking high) reviewed it twice. Both said nothing more is worth removing:

  • The predicate parameter matches list() and search().
  • The outer break still saves opening every remaining folder.
  • -1 costs nothing and keeps the helper's contract.

Both named the hidden-entry lag above as the only case given up. Their one finding was that the header comment overpromised for an empty folder, and it now says "stopped partway". They can read this review's notes, so their agreement is not guaranteed to be independent.

GitHub's Test workflow has still not run here, because it is waiting for a maintainer to approve it for a first-time contributor. Waiting on the maintainer.

@dhh
dhh merged commit bed490b into omacom:main Nov 16, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working verified

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants