Feature: cross-device synchronization

[PanJohnny]

I want to create a synchronization layer for Daily You! The main goal is to be able directly, in app, upload a backup to a storage provider (e.g. dropbox). This backup could be than downloaded to another device, changes compared and data merged.

A similar feature was requested in #321

Key concepts

• let user pick a provider (to be thought of, because some APIs are just pain), can also add a self hosted option – but that would need to be a separate project and a ton of work, so scrap that and instead don't create anything proprietary and single-use
• automatically encrypt the data, with the key stored on device - symetric encryption. This may be an issue due to the danger of user losing the key and then not being to download their backup.
• to add another device one will to set a provider and then add the key from another device (e.g. scanning QR, sending link 🤷‍♂️)
• resolving conflicts may be done in three ways: respect latest edit / create a dedicated view / yolo it and just say that one of the versions is the right one
• API access: oauth with the provider, store secrets and let user do provider specific configuration
• syncing: could encrypt the whole sqllite db and upload it, or do something similar to how git handles data, but there is not a need to preserve commit history

I am opening this pull request as I would love to receive feedback on the idea, to shape it into something or drop it entirely. Direct P2P sync is possible, although hard and tricky to implement.

[PanJohnny]

BTW I am not an android nor flutter developer, so you may need to cleanup some mess if I make this a complete feature. I am more keen on implementing the synchronization logic than doing frontend, config and secret storage shenanigans. Help appreciated :)

[Demizo]

Awesome! Just to organize my thoughts, let me lay out where Daily You currently is with respect to these features and where it could go:

• Currently, Daily You supports setting external folder locations for logs and photos.
  • This is a very basic syncing feature since the app actually has an internal copy and simply syncs up with the external copy when the app launches. This is not an intelligent sync, it just takes the newer database of the two.
    • For better conflict resolution options, the app would need to perform conflict resolution on the SQLite database, that seems to be a large effort.
• A first step I see is to allow for other external storage APIs (like WebDAV, see Support webdav backup #103). This could sync with the API in the same way it syncs with the external folder. Basically, everything could logically fit under the concept of "external location" regardless of whether it is a folder or cloud provider.
• After this, optional encryption for external files could be added. This could apply to any external location, folder or cloud provider.
• Finally, it would be nice to explore improvements that would allow for easy conflict resolution so that working on your logs from multiple devices (i.e. Daily You Desktop) becomes more viable.

Thoughts?

[PanJohnny]

Thanks for sharing your thoughts. I have added simple WebDav support and a way to create and register new synchronization providers. This is extremely WIP but I got uploading and downloading from remote working. Conflict resolution is not yet implemented in any sense. Only the database is uploaded.

I will look into conflict resolution next. I think it is upmost necessary with adding remotes. Sorry for the rough state, I am new to dart.

To test out WebDav, you can spin up server easily with docker: docker run --rm -p 8080:8080 -v ./data:/data -e RCLONE_USER=username -e RCLONE_PASS=password rclone/rclone:1.71.2 serve webdav /data --addr :8080 --baseurl '/webdav'

Regarding encryption - I believe it should be opt-out as this is a diary app and online storage is often shared amongst more people.

See synchronization_provider and webdav_provider for the rough sketch.

[PanJohnny]

And I forgot to mention that I agree that this should be better baked in, as external database location currently is.