Spec: Change metadata of data set / piece post creation

[juliangruber]

Currently, you set data set / piece metadata on creation, and can't change it afterwards. If you need to change it, you end the existing deals and re-upload with changed metadata. That is cumbersome.

Use cases require changing meta data, e.g.: I would like to enable CDN for a data set after I have created it.

Evaluate, whether changing metadata is the right way (could be very complex) or what the alternatives are:

• keep metadata constant / fixed and rather change the abstractions around it so that enabling CDN doesn't mean changing metadata.
• build tooling that handles the migration of pieces to new data sets with updated metadata (could even live directly in curio, since the data is there)

Changing metadata could be complex, because it affects the ongoing payment rate. Tools need to be notified of the changes (through contract events). Tooling built around filecoin services is built on the assumption that metadata is constant.

The spec should have a proposal not just for how to change the smart contracts, but

• how to upgrade them live
• how to update related tooling
  • filozone/synapse-sdk
  • filecoin-project/curio
  • filecoin-project/filecoin-pin
  • filbeam/worker
  • filbeam/contracts

[juliangruber]

Questions that need to be answered before a spec can be drafted:

• do we need arbitrary metadata updates, or only the specific use case of enabling CDN after upload?
• prior question also informs whether we need data set or also piece level metadata updating
• authorization + lockup/refund + timing details

[rvagg]

Posted on Slack:

My opinions on metadata have shifted quite a bit since we’ve been actively using the system; in fact the next step with metadata for FWSS v1.5 is to remove it entirely from state, leaving only the withCDN special-casing state. Metadata then moves to events. In that world, we could then imagine metadata as immutable events over the lifetime of a data set - initial metadata, then “updated” metadata that a user might submit later on.

@juliangruber your issue seems to be focused on CDN though, are there other use cases for editing metadata after the fact that you can imagine? I think we should separate out CDN from everything else and stop thinking about it purely in metadata terms—only that we’re using metadata as a transport to convey some important contract state. We could consider:

• Dedicated APIs to interact with CDN state - turn it on and off for a data set, in this world, metadata does the initial job but later on you can submit a message to enable or disable it, not in the “metadata” frame, just pure “{enable,disable}CDN()” type APIs.
• A metadata “update” API that just emits new metadata events. We currently use DataSetCreated to convey metadata, we may either keep that for initial metadata or change it to some new DataSetMetadata event, which could be used for initial and later updates.
• Are these client-only APIs or do we need to plumb Curio in here? It’d be nice to keep these client-only and avoid Curio entirely since these are “preen my data set” interactions, but it then does require FIL and is less like a last-resort style that we’ve tried to reserve for the non-SP mediated interactions (essentially we’re just down to the client-unilateral termination case which mainly exists to deal with uncooperative SPs). We might be able to make a case that this needs to go through Curio because it wants to know about this state - it may eventually care about the CDN flag because it may change retrieval behaviour or settlement behaviour - there’s also the IPNI metadata which I guess we could consider turning on and off with the SP, that would be … interesting.

[juliangruber]

@juliangruber your issue seems to be focused on CDN though, are there other use cases for editing metadata after the fact that you can imagine?

Not confirmed, no. I can theorize that withIPFSIndexing can also be flipped, and that users may want to update metadata keys they invent themselves.

I think we should separate out CDN from everything else and stop thinking about it purely in metadata terms—only that we’re using metadata as a transport to convey some important contract state. We could consider:[...]

I think you were right in asking the bigger picture question first:

My opinions on metadata have shifted quite a bit since we’ve been actively using the system; in fact the next step with metadata for FWSS v1.5 is to remove it entirely from state, leaving only the withCDN special-casing state.

Iiuc, metadata was also requested for users to able to tag their data, e.g. add a "description", so explorers can be more useful in helping make sense of the data (which seems not to be happening yet).

What @hugomrdias suggested in Slack resonated with me as well: We're using metadata for "feature flags", whether to enable CDN or not for a piece / data set isn't really metadata.

[juliangruber]

I think what we would like to have eventually is:

• arbitrary string metadata, that can be edited post creation
• enable / disable services on top of piece / data set, independent of the creation event

The harder question is how to get there, and when the right time for it is, given that's bigger changes.

[juliangruber]

Asking @jennijuju whether we need this for CDN still, given Storacha migration tooling enables CDN by default: https://github.com/storacha/upload-service/blob/ce3168640a3cfd5b47000ea15d05e0ee2204bba3/packages/filecoin-pin-migration/src/constants.js#L11 (https://filecoinproject.slack.com/archives/C09BCHQC924/p1782123714457259)