Storage for a multithreaded system?

[bedrockjohan]

Hello! We've been using this client in a Django application environment for quite some time now and several times we've had the issue that the connection was lost and we had to login again. This shouldn't happen as there is an automatic refresh token but the Exact help desk did not know what could be the cause of this issue.

Now I read in your latest release that this IniStorage which is used by default might be the cause of this issue.

We have a Django web application but also several management commands and Celery tasks which all use this client using the same storage.

Your suggestion is to create a custom storage and that it's really easy to do and indeed, the example in exactonline.storage.base looks easy but how can we implement a self.fetch_stuff_from_somewhere(...) which is then threadsafe and won't cause this issue?

Am I correct in assuming that the [transient] settings (access_token, refresh_token, etc.) which are now stored in the ini should actually be unique per thread or application in our case?

My first thought was to keep the [transient] settings in memory (as properties on the storage object itself) while the other settings could be read from a persistent storage. This way the Django application will get its own transient settings and the Celery tasks will create their own personal storage each time they are run.

Do you have any suggestions on how to implement a storage like this?
Are there any downsides of creating a ExactApi instance with a "fresh" storage (without any of the transient settings) each time? Some tasks could be doing this several times an hour.
Have you ever implemented something like this yourself which could serve as an example?

I hope you can point me in the right direction!

Kind regards

[wdoekes]

None of these questions should really be directed here.

Do you want to know how OAuth works? Please refer to OAuth documentation on the internet.

Do you want to know whether you can keep multiple access/refresh tokens alive at the same time (keeping most transient settings in memory)? Ask the ExactOnline API support desk. It helps if you checked the OAuth documentation first and have a grasp of what the tokens do (client_secret, code, access_token, refresh_token). They will know whether you're allowed to diverge multiple sets of refresh/access-tokens.

Do you (instead) want to synchronize tokens over multiple applications? Use a distributed locking storage. This wholly depends on your setup. Shared memory with mutexes? Local file locks? Redis?

I suggest you look at what changes in the IniFile when you do a lookup, and then go back to a previous version (of access/refresh tokens). If using previous values works, then it would appear like you're allowed to branch the tokens and you can use multiple sets. If not, then you're stuck using a global lock. (Or you could try making the token refresh a separate one-instance job.)

So many options. But not really for me to answer..

I'll be looking forward to your writeup of the options you considered and the final choices you make.

[bedrockjohan]

@wdoekes Thanks for your response! I was indeed hoping you knew if it was possible to keep multiple access/refresh tokens.

The https://support.exactonline.com/community/s/knowledge-base#All-All-DNO-Content-oauth-eol-oauth-devstep4 say:

You now have a new access token that is valid for 10 minutes, and a new refresh token. You can use your new refresh token to receive a new access token. Your old refresh token is no longer valid.
When your access token has expired you have to use the latest refresh token to obtain a new access token.

Reading this tells me that there can be only 1 token.

I was thinking way too complicated. The only thing the storage needs to do is to return the current value of a setting. The IniStorage returns the value from when the file was first read. When another process updates the refresh token the current app tries to use the old one causing it to fail.

I think i know what to do now. I'm going to try to use the cache to store the transient values which then should be shared between all apps using the client.

Thanks