Optimize Frappe Gmail Thread Syncing

[shivamsn97]

We need to optimize the syncing of Gmail Threads into ERP, and improve handling of Rate Limit errors.

1. Disable syncing of CHAT and DRAFT label:

• We found out that if someone enables the CHAT label for syncing, it returns an 404 error. So we need to hide the CHAT label from the child table.
• In some threads, if there is a DRAFT message, it leads to an error.

2. Optimize requests to Google API:

• We're implementing a request batching mechanism to send requests to Google API. (Ref: https://stackoverflow.com/questions/67658283/how-do-i-send-a-batch-request-to-the-gmail-api-in-python). This will allow us to send multiple queries in one request.

3. Reduce Rate-limit Errors:

• For this, we are considering executing each batch request in a separate RQ Job. This will divide our syncing process in smaller jobs, thus enabling a more robust rate-limit handling. In the event of an rate limit error, we can use Redis to store the value of Retry-After, and allow the next batch request to be processed after that only.

4. Limit number of threads to Sync:

• In my case, I was able to sync around 750 threads in a time of 6 minutes. Should we implement a limit on how many maximum threads we can sync per label? If yes, we can store the Hard Limit value in Google Settings doctype under the Gmail Threads section.

cc @KanchanChauhan @niraj2477 @dhsathiya

[SachinSingh008]

Hi @shivamsn97, just checking in — are you still working on this issue?
If you’re busy or it’s on hold, I’d be happy to help or take it over.
Let me know what works best for you. 👍