Skip to content

feat: log each page's memory, and when it would be rebuilt - #36

Merged
nicojan merged 1 commit into
nicojan:mainfrom
marcioviniciusspiridigliozzi-dot:fix/recycle-growing-pages
Oct 18, 2026
Merged

nicojan merged 1 commit into
nicojan:mainfrom
marcioviniciusspiridigliozzi-dot:fix/recycle-growing-pages

Conversation

@marcioviniciusspiridigliozzi-dot

@marcioviniciusspiridigliozzi-dot marcioviniciusspiridigliozzi-dot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Reshaped after review: this PR now only measures and logs; nothing is rebuilt. See the latest comment for what changed. The original description follows for context.

Hi Nico, and thank you for Chorus. I use it every day for Slack, Teams and WhatsApp, and it has been lovely to watch it grow. This PR comes out of one afternoon of poking at it on my own machine. I have tried to set down everything we looked at, including the parts that ended with no code, in case any of it is useful to you. Please treat all of it as a suggestion. You know this code far better than I do, and I will gladly change or drop anything you would rather have done another way.

What this changes

Chat services never hibernate, which is right for notifications. But it also means nothing ever makes their pages let go of what their scripts build up, so a page that grows keeps growing for as long as Chorus runs.

With this PR, every five minutes Chorus looks at how much memory each service's page is using. If a page you are not looking at has grown past both 1.5 GB and twice the size it settled at, Chorus tears it down and loads it again straight away. Because it comes back at once, a chat service keeps posting notifications. The reload is skipped when the service:

  • is on screen, or was left less than ten minutes ago (so a half-typed message is never lost under you)
  • is pinned, or set to Keep Loaded
  • is using the camera or microphone, playing sound, or in a call

The call probe is a suspension point, so recycleIfStillOversized reads every guard again after it, the same way hibernateIfStillIdle does. No schema change, no new setting.

What I measured

On 2026-10-01, with Chorus 1.5.25 on macOS 27.0.1, one Chorus WebContent process went:

time footprint
18:00 154 MB
18:28 1.6 GB
18:30 1.8–2.2 GB, rising and falling with GC

footprint -p put 1.5 GB of it in WebKit Malloc, plus 41 MB of WebAssembly memory. I could not prove which service it held, because the WebKit logs hide URLs in a release build. I am fairly sure it is WhatsApp Web: it is the WebAssembly user among my six services, and the Teams process had just been rebuilt. The WhatsApp app sat at about 460 MB on the same machine.

I saw that OPEN-ITEMS.md reads webcontent_mb as showing no trend. I think that reading is right about the sum. The sum swings by gigabytes as pages come and go, and that swing hides one page that keeps climbing. Read one process at a time, the climb shows. I have added a short note there rather than changing your analysis.

How it works

  • WebViewPool.webContentFootprint(for:) gets the page's process id from _webProcessIdentifier and reads its physical footprint with proc_pid_rusage, which is the number Activity Monitor shows. _webProcessIdentifier is SPI, so it is probed first, the way DataStoreManager treats its own SPI. If a future macOS drops it, nothing is ever recycled, which is how Chorus behaves today.
  • Each page gets a baseline, its footprint once it is five minutes old.
  • AppState.startMemoryRecycleTimer runs the sweep. A recycled page goes through teardownWebView and then the normal preload path, so onServicePreloaded restarts its background polling as usual.

Two questions worth answering

While I was building this, the person I was working with raised two good questions, and the second one changed the design.

Does the check itself cost anything? Very little. Each pass makes one proc_pid_rusage call per live page, every five minutes. The JavaScript call probe only runs for a page that has already crossed the limit.

Could a page that just needs a lot of memory get reloaded in a loop? With a fixed 1.5 GB limit, yes. A page that needs 1.6 GB just to run would be rebuilt every ten minutes, forever. That is why the limit is relative to each page's own baseline. A fresh load settles near its baseline, so the rebuild cannot trip the rule again on its own. Only real growth can. The test pins this case ("big but steady stays"). It also caught a bug in my first version: baseline × 2 could overflow and wrap to a small number, which would have recycled the page. That case is now handled and tested.

Testing

  • xcodebuild test (Debug, as CLAUDE.md asks): 300 tests, 0 failures, 1 skipped, as before. One new test, testRecycleTakesOnlyAPageThatKeptGrowingAndNobodyIsUsing, covers the rule.
  • By hand, with the limits turned down on purpose (floor 1 MB, growth factor 0, settle 20 s, idle 30 s, sweep 30 s) in a Debug build against Chorus-debug: five background services were recycled in one pass. Each old WebContent process exited, a new one came up and loaded back to its earlier size (84–938 MB), and the service on screen was left alone.
  • With the growth factor back at 2, five minutes of the same services stayed within a few MB of their baselines (79 vs 85, 188 vs 190, and so on), and none was recycled.

What I have not tested: a signed-in WhatsApp over many hours on a release build, which is the case this exists for. The services in my Debug store were not signed in. I would not call it proven until it has run that way, and I would be glad to run it on my machine and report back if that helps. I could also only check macOS 27. I am counting on CI for 14 and 15.

Something unrelated that came up: Teams sign-in

This needs no code, but it might save you a report. Teams showed its "Precisamos que você entre novamente" ("we need you to sign in again") banner, and clicking Entrar (Sign in) did nothing. I checked whether my earlier SSO fix (#17) had stopped working on macOS 27. It has not: no tracking-prevention warnings in the log. A Debug build with a fresh data store signed in to Teams cleanly in one window: teams.cloud.microsoft → login.microsoftonline.com → kmsi → back with #code=.

In the release app, the click did reach the page. It set off a run of same-site redirects that ended back on the same screen, and WebKit dropped clicks while they ran (handleMouseEvent ... possibly stuck?). Deleting the Teams service and adding it again fixed it. So the stored data for that service had gone bad, most likely from the time before #17. I did not check whether "Clear session (log out)" on its own would have been enough.

On the writing

CLAUDE.md asks that public text pass the humanizer check. That tool was not available in my session, so the CHANGELOG line and the OPEN-ITEMS note have only been through Orwell's rules by hand. I am sorry about that. Please tell me if anything reads wrong and I will fix it.

Thank you again for your time, and for Chorus.

🤖 Generated with Claude Code

@nicojan nicojan left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for this, and for writing down everything you looked at, the dead ends included. You're right that the sum of webcontent_mb hides one page that climbs, and reading each process on its own is the better measure. Each service has its own data store, so a footprint belongs to one service, and the SPI fallback is safe. I can't merge it yet, though. Most of what follows comes from main moving on since your base.

Needs fixing

  1. The ten-minute guard counts from when you opened the service, not from when you left it. lastAccessTimes is only set on activation. Spend half an hour in WhatsApp, switch to Slack, and the next pass can rebuild WhatsApp seconds later. Please record when a service is left and measure from that, and add a test that runs activate, switch away, sweep.
  2. A rebuild closes every open tab. Main added tabs in a service's card, and teardownWebView now calls closeAllTabs() without the save step. Every hibernation path skips a service with hasOpenTabs(id). Please add the same check before and after the await. Tabs also run in the same process as the page, so a big tab inflates the very number that triggers the rebuild.
  3. WhatsApp may come back signed out. The comment above the visibility override in UserScriptManager says WhatsApp saves its session when the page goes hidden. Quit and tab close both run quitReleaseJS and wait before tearing down. The rebuild doesn't. Your by-hand test used services that weren't signed in, so it couldn't show this.
  4. The page comes back at its home address. preload loads instance.url, so the open chat or channel is lost. Please keep webView.url and load that.
  5. It can still loop. The limit is max(1.5 GB, 2 × baseline), so below a 768 MB baseline the floor decides alone. A heavy client sampled early at 700 MB that settles at 1.6 GB gets rebuilt every pass. Carrying the old footprint over as the new baseline, or adding a backoff per service, would stop that.

What I'd suggest

The evidence is one process on one evening on macOS 27, and the service is a guess. I'd rather ship the measurement first: log each page's footprint, and log when it would have rebuilt one, for a release. That would also test my own finding that memory levels off, which you rightly question. If the logs show the climb, turn the rebuild on with the fixes above, one service per pass, skipping while offline and just after wake. I'd gladly take a PR in that shape, and a release-build run against a signed-in WhatsApp would help a lot.

The CHANGELOG states the WhatsApp figures as fact, while the internal note says the service was inferred. The old "no trend" lines in OPEN-ITEMS.md also need marking as superseded. I can do both when it lands.

On Teams: thanks for checking that #17 still holds. Deleting and re-adding the service is a useful fix to know about.

Measure first, act later, as the review of nicojan#36 asked. Every five
minutes Chorus logs each page's footprint, read with proc_pid_rusage on
the pid from _webProcessIdentifier (SPI, probed), under the service's
catalog id. It also logs when a page would be rebuilt: past both 1.5 GB
and twice its baseline, out of sight for ten minutes, and not on
screen, pinned, Keep Loaded, capturing, audible or holding tabs.
Nothing is rebuilt.

From the review:
- Out of sight is counted from when the user left the service, not from
  when they opened it. ServiceVisits records the switch, and a test runs
  activate, switch away, sweep.
- A service with tabs open never counts, since a rebuild closes them.
- A page that counts carries its footprint over as its new baseline, so
  a heavy page sampled early counts once, not on every pass.
- The rebuild itself, and with it the quitReleaseJS save and the
  restore of webView.url, waits for the logs. OPEN-ITEMS lists what
  turning it on needs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@marcioviniciusspiridigliozzi-dot marcioviniciusspiridigliozzi-dot changed the title fix: rebuild a background page that keeps growing, to give its memory back feat: log each page's memory, and when it would be rebuilt Oct 6, 2026
@marcioviniciusspiridigliozzi-dot

Copy link
Copy Markdown
Contributor Author

Thank you, Nico, for such a careful review. You were right on every point, and the measure-first idea is better than what I sent. I have reshaped the PR to match it, rebased on 1.5.27. The conflict is gone, and the CHANGELOG change is gone with it.

What the PR does now. It only measures. Every five minutes Chorus logs each page's footprint under the service's catalog id, so a reading says whatsapp instead of leaving us to guess. When a page would be rebuilt, it logs that too. Nothing is rebuilt. Two lines from a Debug run, with the interval shortened just for the check:

Memory watch: teams 956 MB, baseline 956 MB, out of sight 28 s, tabs false
Memory watch: gmail 106 MB, baseline 106 MB, out of sight 0 s, tabs false

To read it back: log show --predicate 'subsystem == "com.nicojan.Chorus" AND eventMessage BEGINSWITH "Memory watch"'. I hope it also helps check your finding that memory levels off.

Your five points:

  1. The ten-minute guard. Fixed. ServiceVisits records the moment a service is left, whether for another service or for a Mac app, and out of sight counts from then. testOutOfSightCountsFromLeavingAServiceNotFromOpeningIt runs your sequence: half an hour in WhatsApp, a switch to Slack, then a sweep five seconds later. WhatsApp counts as five seconds out of sight and never counts as grown.
  2. Tabs. A service with open tabs never counts. I also noted in the doc comment that tabs share the page's process and add to its footprint, and the log shows tabs true/false on every line so a reader can tell.
  3. WhatsApp signing out and 4. the home address: the rebuild is no longer in this PR. I wrote both into OPEN-ITEMS.md as conditions for turning it on: run the quitReleaseJS save and wait, as quit and tab close do, and reload webView.url. Your other conditions are there too: one service per pass, skip while offline and just after wake. Thank you for catching the session one. My by-hand test could never have shown it.
  4. The loop under a 768 MB baseline. When a page counts, its footprint becomes its new baseline, standing in for the rebuild. Your example, a page sampled at 700 MB that settles at 1.6 GB, now counts once and stays quiet until it doubles again. testMemoryWatchDoesNotCountAHeavyPageSampledEarlyOverAndOver pins it.

On the docs. The internal note now says plainly that the service is a guess from one evening on macOS 27. The old "no trend" line is marked as superseded for single pages, with a pointer to the new note.

319 tests pass locally (Debug), including the three new ones. I will run the release build with a signed-in WhatsApp over a working day once this is in, and report what the logs show.

Thank you again for your time and patience.

@nicojan
nicojan merged commit c15743d into nicojan:main Oct 18, 2026
2 checks passed
@nicojan

nicojan commented Oct 18, 2026

Copy link
Copy Markdown
Owner

Merged, thank you. You fixed all five points, and the log-only shape is what I hoped for. I ran it on a Debug build tonight: after five minutes it wrote one line per page, and the pids matched the WebContent processes Chorus had started.

I added a small commit on top (8221c77) rather than send it back again:

  • Each line now carries the WebContent pid. With two Gmail or two WhatsApp accounts the catalog id alone can't tell their curves apart, and a new pid shows when a process was swapped or crashed, so a drop doesn't read as the page shrinking.
  • The baseline step now lives in WebViewPool.memoryWatchPass, and the second test runs a series of passes through it. Before, the test fed wouldRebuild a baseline by hand, so deleting the carry-over line left it green. I checked that the new one goes red.
  • A real rebuild would start the page small again and count sooner than the new baseline lets it, so I reworded the comment and the note to call the "would rebuild" count a floor. The five-minute series is what shows how often.
  • A catalog id Chorus doesn't know, say from an imported setup, logs as custom.

It ships in the next release. When you run it against a signed-in WhatsApp over a day, please post what the log shows here or in a new issue. Read it with /usr/bin/log show, since in zsh a plain log is a shell builtin and fails.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants