Proposal: descriptor: "dependency" property

[AkihiroSuda]

A blob referenced via descriptor MAY be a fine-graind CAS archive. e.g. continuity manifest. (WIP: #577)

This might make it difficult to implement "gc" for blob store, because the actual referenced blob set cannot be assured just by parsing OCI descriptors.

So I suggest adding a new property named dependency to the descriptor spec.

{
  "mediaType": "application/vnd.continuity.manifest.v0+json",
  "digest": "sha256:deadbeef",
  "size": 424242,
  "dependency": {
    "type": "media-specific"
  },
}

When dependency.type is media-specific, implementations can skip pruning unless it is aware of the corresponding mediaType.

Spec Proposal (for v1.0 if possible)

• dependency.type string

  This OPTIONAL property specifies the type of the content set that targeted content refers.
  Values SHOULD be one of the followings:

  • ``(empty): the set is empty
  • media-specific: the set is specific to the media type of the targeted content

Implementor's Note
Implementations MAY compute unreferenced blob set so as to prune them.
When implementations encouters a descpritor of which dependency.type is media-specific and the implementation is not aware of the corresponding media type, or dependency.type is set to unknown value, the implementation SHOULD NOT try to compute unreferenced blob set.

Post-v1

e.g. list (can be huge)

"dependency": {
  "type": "list",
  "list": [
    "sha256:f00",
    "sha256:baa",
  ]
}

e.g. list as a text file blob

"dependency": {
  "type": "blob",
  "blob": {
    "mediaType": "application/vnd.oci.dependency.v2+text",
    "digest": "sha256:ba2",
    "size": 4242
  }
}

[wking]

On Tue, Apr 25, 2017 at 09:30:00PM -0700, Akihiro Suda wrote: This might make it difficult to implement "gc" for blob store, because the actual referenced blob set cannot be assured just by parsing OCI descriptors.

Currently umoci leans on the SHOULD for references [1] to walk the Merkle DAG [2] for garbage collection and such. That covers this use case for all blobs that: 1. Are parseable as JSON. 2. Follow the spec's SHOULD and use descriptors for referencing dependencies. 3. Don't use things that can be parsed into descriptors for purposes other than referencing dependencies. 3 is a weak requirement, because keeping an unneeded blob is less troublesome than pruning a needed blob. Those requirements seem like a low enough bar that I'm not sure we need the flag you're proposing to mark when they're violated. Do you have a use case in mind that violates 1 or 2? [1]: https://github.com/opencontainers/image-spec/blame/v1.0.0-rc5/descriptor.md#L10 [2]: https://github.com/openSUSE/umoci/blob/v0.2.1/oci/casext/walk.go#L34-L43

[stevvooe]

To implement full gc, you really need to have knowledge of all the stored formats. It is a little unfortunate, but not entirely impossible implement in a DAG visitor model. In containerd, this is as simple as adding a handler that can de-serialize when encountering these types.

The main issue with this proposal is that you allow a single item to poison the garbage collector. To be clear, when it encounters any media-specific type, it must assume that the target may reference anything, allow remotes to shut down garbage collection. This is not a very nice property.

If we did add such functionality, I would think it would look more like this:

{
"mediaType": "application/unknown",
"digest": "sha256...",
"size": 1234,
"references": [{
    "mediaType": "application/vnd.oci.dependency.v2+text",
    "digest": "sha256:ba2",
    "size": 4242
  }, ...]
}

The references would provide the guarantee that all child objects of the target would appear in references. This would handle cases where we have deep trees of objects or opaquely referenced material.

Either way, I am not sure GC is part of the scope of the OCI specification. There are lots of cases where runtime references and policy must come together to operate correctly on the referenced set of objects, which is application specific.

[AkihiroSuda]

How to explicitly describe that the blob does not depend on anything?
IIUC an empty references cannot be used for that purpose, because it would means "media-specific"

[stevvooe]

How to explicitly describe that the blob does not depend on anything?
IIUC an empty references cannot be used for that purpose, because it would means "media-specific"

I'm not really sure either of these classifications will work in practice. They are effectively de-normalizations of existing "facts". The main issue with this approach is that it allows remote data structures to influence the behavior of garbage collection, which provides an opportunity for malicious behavior to be introduced.

I'll reiterate this, as it may have been lost in my original comment:

To be clear, when it encounters any media-specific type, it must assume that the target may reference anything, allow remotes to shut down garbage collection. This is not a very nice property.

The only solution for this to work correctly in all cases is to have first-hand knowledge of the data formats. The media-specific type would almost always force the system to turn off GC.

[vbatts]

it seems to me, this could be done with the discussed "references API"

[sudo-bmitch]

What are the scenarios where an unknown manifest media type would be encountered? For registries, they almost universally reject a manifest with an unknown media type, and validating registries require all child manifests to be pushed before the index is pushed. For runtimes, if an unknown manifest is contained in the pulled index, I would expect that to be ignored, and all referenced content from that descriptor does not need to be pulled/imported/preserved. If this is tooling working with an OCI Image Layout, I think a good best practice is to abort any GC code if an unknown manifest is encountered, but since the spec doesn't specify GC behaviors, there's nothing for us to do here.

Given all these behaviors, are there scenarios or use cases that would be resolved by a new field in the descriptor, or is this safe to close?

[tianon]

(Is this a duplicate of #102?)