Use the new ocidir-rs archive support#315
Conversation
Latest ocidir-rs has the oci-archive support. Unfortunately it also updates the oci-spec dependency to 0.10, so we need to bump oci-spec to 0.10, and containers-image-proxy to latest main commit to sync the oci-spec version everywhere. This is marked WIP because these should really get releases and we should bump dependencies to that release version. Signed-off-by: Alexander Larsson <alexl@redhat.com>
This will be useful later when we start using OciDir for this, as it needs the full descriptor, including the size. Signed-off-by: Alexander Larsson <alexl@redhat.com> Assisted-by: Claude Code (Opus 4.6)
This is similar to tokio::fs::File, but just converts a sync Read to an AsyncRead, doing each actual read in spawn_blocking. We will need this to handle async reading from the OciDir reader we get from a oci-archive, as that is not a fs::File. Signed-off-by: Alexander Larsson <alexl@redhat.com> Assisted-by: Claude Code (Opus 4.6)
This applies to both pulling deltas and regular images and uses OciArchive::open() if the source is a file. Also, we have to convert the code to be more generic on the source data stream, as it is no longer always fs::File. We use the new BlockingReader helper to convert the sync Read to an async stream. Signed-off-by: Alexander Larsson <alexl@redhat.com>
|
Looks sane offhand, but let's get releases of the dependent crates out. |
|
@cgwalters I really thought there would be something like the BlockingReader helper somewhere already, but I was unable to find it... |
| const READ_BUF_SIZE: usize = 128 * 1024; | ||
|
|
||
| /// Adapts a synchronous `Read` into a `tokio::io::AsyncRead` by dispatching | ||
| /// reads to [`tokio::task::spawn_blocking`], similar to `tokio::fs::File`. |
There was a problem hiding this comment.
This one is interesting; way back in the past I hit this in the other direction, see https://users.rust-lang.org/t/best-practices-for-bridging-async-and-sync-particularly-read-write/58994 and we ended up adding some tokio helpers.
This case is the inverse though.
I think implementing the low-level poll_read API requires a lot of thought about correctness.
In this case, instead of a spawn_blocking per read call, what I think would be more efficient is basically having a pipe:
- spawn exactly one spawn_blocking task that uses sync IO and writes to a channel (stream)
- A stream of bytes can be converted to AsyncRead https://docs.rs/tokio-stream/latest/tokio_stream/#conversion-to-and-from-asyncreadasyncwrite
There was a problem hiding this comment.
Ok, I did something like this, as a separate commit so you can look at it. For the final version I'll delete the BlockingReader commit and squash this into the other commit.
We now create a single blocking thread and move the entire stream to it, we then use tokio::io::duplex() stream that we copy the read to, and a SyncIoBridge to create an AsyncRead from this stream. Signed-off-by: Alexander Larsson <alexl@redhat.com>
Signed-off-by: Alexander Larsson <alexl@redhat.com>
alexlarsson
force-pushed
the
delta-archive
branch
from
d221d9c to
a9577de
Compare
2 participants
With this we can pull (both regular and delta) images directly from an oci-archive without using the container image proxy.
NOTE: This is draft, because the first commit updates some dependencies to random commits. We really need a new release of oci-dir and containers-image-proxy for this to be clean.