Optimize inbox scanning for very large inboxes

[anna-st-40]

Apparently, for very large inboxes (allegedly ~500k), fetching freezes.

Make a way to allow emails to be loaded in NOT all at once.

[kmgrime]

Hi,

I only have 28k emails and have the same problem. :)

[Vamsiindugu]

Hi @kmgrime,

Thanks for reporting your experience.

I'm the developer who worked on this issue. I can confirm this problem is now resolved in pull request #72, which was just merged into the main branch.

The fix moves away from loading all emails at once and implements a progressive scanning system, which resolves the freezing you're seeing. This will be available in the next official release of the extension.

[kmgrime]

The fix moves away from loading all emails at once and implements a progressive scanning system, which resolves the freezing you're seeing. This will be available in the next official release of the extension.

Hi @Vamsiindugu,

Great! Cant wait to try that release, could consider beta or rc so it could be tested before you release it as stable.

Thanks for replying, the addon is helpful to keep the inbox clean 👍🏽

[andyg2]

I have 28,352 emails and it got to about 22k before I got this message.

<div class="vX UC" style=""><div class="J-J5-Ji"><div class="UD"></div><div class="vh"><div class="J-J5-Ji"><div class="vZ cNDVKe wTsMFb"><span class="v1">Oops… the system encountered a problem (#2008)</span> - Retrying in 59s…</div></div><div class="J-J5-Ji"><div class="v0"><span id=":e1w" class="v2" tabindex="0" role="link">Retry now</span></div></div></div><div class="UB"></div></div></div>

Other hints for what might be going wrong.

• If I open Gmail in a second tab, switch to All Mail and cycle thorough them manually there's no error.
• If I reload the tab with InboxWhiz then I'm back to No senders yet - Load senders to get started.

Possible fix methodology:

Perhaps saving the list to localStorage after each batch and then deleting it on completion would allow a reloaded InboxWhiz to pick up that data and skip the scanning for senders. Just an idea.

A UX idea would be to have finish scan button to break the scan loop, or have a date input to limit how far back a user wants to go, for example I don't think many people will need to go back to 2005 and that might also terminate the finding senders loop before the error occurs.

This is such a nice extension.

[aioue]

I also just got to around low 20k before it stopped

[sancarn]

Honestly I think this is a rate limit issue. I'm hitting this on 8.9k emails after I've reloaded the extension a few times...

[heltonteixeira]

After the first 20 pages of emails the requests are being blocked by google, and all senders are lost after this.
What if it save senders after each page, so we don't lose the already fetched senders.

[anna-st-40]

Today's newest release (2.1.0) now saves emails after each page, and you can stop the scan at any time (such as when it freezes) to show the emails loaded in so far.

[jlcvp]

Thanks for the release @anna-st-40.

Seems like Google is not happy when you load dozens of pages and there are some rate limits being applied from the backend to their own frontend as well. In my tests they vary a little bit, from 40 ~ 50 pages to as low as 5 ~ 8 pages before hitting the VERY delayed yellow "loading..." message between pages (5~30 seconds per page and increasing as we try to push harder)

Maybe we can get a reasonable timeout on the scanner so we know we've hit that hard throttler and show some informative text to the users blaming google and saying that we've managed to load at least X e-mails successfully... Or something like that. I'm terrible with UX.

[rafaelcomim]

im experiencing the same issue even with the 2.1.0 version.
i've had something around 38k e-mails, and in this scenario the load stoped at aproxamtely 20k.
at first i though that the limitation could be in the e-mail count, so i deleted some old messages, but even deleting 18k messages seems not to ease the load. Still freezes around the time gmail loading yellow message start to show up. Looking at the console, it seems that the app doesn't aknowledge the successfull advance to a new page that takes too long to load, because there is no log of error, it is just hanging.

I've tried to use the stop button to get around the problem, to see what was already processed, but it seems not to work in my case. If i click the stop button, it just goes back to the initial state of the app.

i really dont want to get in the way here, but here are my two cents:
i dont know if it is really a limitation as said above, or if it is some implementation detail related to the slow load.
maybe we can get to work even with slow loads if we could think on the possibility of applying some easy filters on the search instead of scanning all e-mails at once.
Of course that if we can get the stop button working as intended it will mitigate the problem as well, but it will not really solve it.

i hope it helps :)

[aioue]

Reporting fixed. Success with a very large inbox: