Indexes for interoperability?

[jg10-mastodon-social]

This is more of a discussion than a feature request.

Currently because a task list is a container, it gets rendered generically, e.g.

In the past I've suggested adding a subclass to the container so that a custom view could be used. While that might still be useful, I now have another suggestion.

Solid Focus could encourage further interoperability by providing one or more default indexes that are automatically kept updated on sync.
These would provide a friendly way in to tasks in generic data browsers and for power users.
It would also make it easier for other apps to work with task lists without necessarily downloading all data.

Other apps could obviously manage indexes themselves - this is more of an opportunity for Solid Focus to define an additional interface and increase the chances of serendipity rather than being necessary for interoperability.

[jg10-mastodon-social]

I'm experimenting with a PodOS-tailored index, which I'm currently regenerating daily on the server with a Cron job.

It indexes just task titles.

• Out of the box, PodOS then shows the list of task names, which makes the Solid Focus data usable in the generic browser
• Adding the index to the PodOS label index allows for quick navigating to the task list
• Adding the index document to the list of PodOS label indexes allows search across all notes

</tasks/index.ttl#it> a schema:ItemList;
rdfs:label "Focus tasks";
schema:itemListElement </tasks/main/43986ac6-9aeb-4a87-939a-d07450c939a2#it>.

</tasks/main/43986ac6-9aeb-4a87-939a-d07450c939a2#it> rdfs:label "my task title";
rdfs:comment "Focus task".

• I use schema:ItemList to increase the chances of interoperable serendipity.
• I use rdfs:label specifically because that's what PodOS currently uses for its search index. Perhaps in future PodOS could support schema:name as well.
• In this index I only include tasks that are not marked as complete.
• The RDFS comment is added because it shows up as a description on PodOS search

I think the resulting index is fairly generic and is easy to manage if it's treated as a read-only derived resource.

[NoelDeMartin]

My general opinion of all these indexes is that they shouldn't be necessary, and in particular the use-case of working with tasks without downloading all the documents is something that has been discussed for years. Ideally, this is something that the servers should handle.

But seeing that that's not happening anytime soon, I can see why we could do some indexing ourselves... The problem here, though, is that I don't think doing this type of index would change anything in Solid Focus. Being a local-first app, I always download all the data anyways, so it's not necessary to use any indexes. If anything, I may have some local indexes (which I sort of have in IndexedDB).

The main reason why I'm very reticent to doing this, besides the fact that it doesn't enable any feature in the app, is that this type of index could be very fragile, and it would add a lot of overhead in the app to make sure that the index is up to date. I prefer to have a single source of truth, which is the container itself. And when I create/remove a task, I don't have to think about syncing anything else (it's already complicated enough with all the CRDTs 😅).

I actually think this is the perfect use-case for another service, such as PodOS, that connects to a POD and enriches the data somehow (making indexes or whatever). If I were to do that nowadays, I think the ideal solution would be to augment the type index with individual instances, not just the containers. But that can be a problem when a user has a lot of data, because the type index may blow up in size.

[jg10-mastodon-social]

Thanks for the response! I understand your position on this, and will add some clarification for the sake of discussion rather than to convince you to change your mind!

My general opinion of all these indexes is that they shouldn't be necessary, and in particular the use-case of working with tasks without downloading all the documents is something that has been discussed for years. Ideally, this is something that the servers should handle.

The utility I see is precisely to pull out pieces of information that are useful in aggregate without having to download everything. From this perspective creating an index is a deliberate act rather than something implemented by default by a server.
I agree that in the case of Solid Focus, the task list overview includes not just name, but also completion status, importance and existence of a note. This is then enough information that downloading all documents is already worthwhile.

The main reason why I'm very reticent to doing this, besides the fact that it doesn't enable any feature in the app, is that this type of index could be very fragile, and it would add a lot of overhead in the app to make sure that the index is up to date. I prefer to have a single source of truth, which is the container itself. And when I create/remove a task, I don't have to think about syncing anything else (it's already complicated enough with all the CRDTs 😅).

I agree we don't want index creation to cause fragility or incur substantial overhead.
I think it's important that it be a read-only derived resource with a clear and efficient updating process.
To use the example of Solid Focus, PUTting a list of tasks at the end of the sync would likely be efficient enough, though would be limited in scale.
The expensive part is reading all the data, which is already done as part of the sync.

If I were to do that nowadays, I think the ideal solution would be to augment the type index with individual instances, not just the containers. But that can be a problem when a user has a lot of data, because the type index may blow up in size.

This is quite different to how I see this index. The type index is primarily about discovery, whereas what I am describing is primarily about layered interoperability - having additional read-only views of the underlying resources that lower the entry barrier to interacting with the data.