Reflections on local change-preserving sync

[johannrichard]

As noted elsewhere in PR's, I've been mulling a sync that preserves (or merges) local changes to notes for some time.

Obviously the main driver for a merge-sync is the fact that I might want to modify a highlight or its note locally, and avoid overwriting it in case I change the highlights remotely as well.

This is an attempt to structure the potential merge-sync approaches, incl. hurdles, drawbacks, and strategies, in a way that allows at least a discussion.

General hurdles and drawbacks

• Markdown merge: Markdown has some features that make it more difficult to merge changes. For one, a lot of blocks are rendered as very long lines. Whitespace has semantic meaning w.r.t. formatting, omitting line breaks between certain blocks can change or break the display. Also, since the user might change the templates between syncs, the resulting Markdown might look completely different and a merge might fail.
• Identifying local changes vs. remote changes: this is a tricky one. To be able to merge sensibly, we must know which version changed last (local or remote) in comparison to the last successful sync of a file, and ideally, we can even compare with the original (i.e. last synched and merged remote version). This requires full three-way merge, for Markdown.
• Atomicity of the sync: because Markdown (re-)formatting by Obsidian or plugins might change the structure of the local file when compared to the original rendering from the template, a global approach to comparison and merging might create lots of sync, diff, and merge issues, with potentially catastrophic or impractical results. Ideally, there would be some way to ensure atomicity in terms of diffing and matching as well as processing (see strategies for what I think would be possible).
• Flagging changes: there's no native way of flagging changes in Markdown. It also raises questions on how to flag changes to the user.

Strategies

• Markdown diff: This requires a diff-and-patch (or merge) approach that is semantic, targeting word-by-word approaches, and considers specificities of Markdown. This should be doable by building e.g. on the code from dubdiff which implements a Markdown tokenizer for the 'Diff' library.
• Local vs remote changes: proper diff and merge of local and remote changes requires that we keep track of the "original" (last sync result from Readwise), which is possible, but in the minimum adds a local cache (JSON file), thus slightly increasing the size of the Vault, and adding some complexity if we want to incorporate the result of the merge.
• Atomicity - Frontmatter vs. Content: this is actually a relatively easy one. Obsidian has APIs to separate frontmatter from content programmatically. If frontmatter is present, we could at the very least allow the user to choose to keep the frontmatter and "just" overwrite the contents when syncing changes from Readwise. That would allow things like persistent linking to additional files or keeping other metadata via frontmatter fields.
• Atomicity - Header, Notes, and Highlights: this is trickier. Using block ids for the different parts (Header, and individual highlights) when composing the rendered file, we might be able to leverage Obsidians APIs to separate the local file and then compare "block by block" (e.g. for each highlight). A different strategy would be to save each highlight as a separate file and link to the parent via ![[Inclusion]]. The practicality of this seems at least questionable, even though have a file per quote might make it simpler to integrate highlights and notes atomically into other files. Issues with reformatted Markdown might prevail with both approaches.
• Flagging and dealing with conflicting changes: Visually, the "best" one can do is using HTML tags to mark insertions and deletions. However, this looks ugly, and also, as a consequence of the markings, what to do on a subsequent sync if the markers have not been removed). Also, removing HTML tags in Obsidian is not so simple and any attempt at providing some assistance to the user will increase the complexity of the plugin. Still, it might be possible, e.g. by using tags in frontmatter and/or marker detection when re-synching.

[TfTHacker]

While I love the idea and approaches you document here, I provide you with an additional opinion. I would make your tool the best one-way sync tool, and let Obsidian do its job. Users can combine easily Readwise highlights into their notes using linking or embedding.

[johannrichard]

Thanks @TfTHacker for your view. As you write, and looking forward to the atomizer release, it should indeed become simpler to combine local notes and remote highlights.

[TfTHacker]

Thank you for adding this great feature. I am surprised it has minimal impact on my vault with all the additional files. It also works very nicely with Bases!