Support Virtiofs (New filesystem type based on FUSE)

[stepancheg]

For heavy containers running full linux (like cloud-hypervisor) there's an option to provide filesystem using virtiofs protocol (which is FUSE where the server is running on the host).

Would be helpful if gVisor also supported that.

Applications are obvious: a container may want to access large filesystems without downloading it on the host (as both root and non-root filesystems).

(We actually only need readonly access with overlay, which is probably much easier to make work, but I guess others may want full rw access.)

Currently fuse can be mounted on the host and exported to gVisor as regular host filesystem, but it has drawbacks:

• larger overhead
• less secure
• easier to overwhelm host kernel
• bugs depend host kernel version

There's document describing gVisor FUSE work in progress (fuse.md), but it only mentions fuse server running in the sandbox, but not on the host.

[manninglucas]

I'm not sure how feasible this is to implement in gVisor. FUSE servers require opening the kernel's /dev/fuse and passing that FD to the FUSE filesystem via mount(2) syscall as a mount option. A FUSE FD from the gVisor kernel (aka sentry) doesn't logically translate to any kind of file descriptor on the host system. There isn't any way I know of to associate a sandbox FD with a host FD at runtime such that you to would be able to read/write to the host and have it be reflected in the sandbox.

One way I could imagine this working is if you setup some kind of proxy script inside the sandbox that reads from the sentry's /dev/fuse FD, forwards those requests to your host space FUSE server, then reads responses from your server back to gVisor's /dev/fuse FD. This is a brittle setup but may work.

For what it's worth, we've seen users running FUSE servers inside the sandbox with general success, so if you're able to swing that approach that may be your best bet.

[stepancheg]

I was thinking about setup like this:

  "mounts": [
    {
      "destination": "/artifacts",
      "type": "virtiofs",
      "source": "tag1"
    }
  ],
  "annotations": {
     "dev.gvisor.root.virtiofs": "tag2",
   }

And then invoke gvisor like this:

runsc --fs=tag=tag1,socket=/var/artifacts.sock --fs=tag=tag2,socket=/var/rootfs.sock --fs=tag=tag3,socket=/var/somethingelse.sock

(This would be similar to --fs arg of cloud-hypervisor).

(sock files here are provided by tools like virtiofsd except virtiofsd itself would be useless here because we already have gofer).

And with this setup, rootfs and /artifacts directory would be mounted, but also following mount command would also work, like in regular linux running inside cloud-hypervisor:

mount -t virtiofs tag3 /somethingelse

Fuse/virtiofs works here in userspace, without host kernel fuse (it uses unix sockets).

There are no file descriptors exposed to processes in the container, only premounted filesystems or virtiofs tags.

[manninglucas]

I see, according to their design doc virtiofs is a new filesystem type that just uses FUSE protocol messages to communicate. I was confused. You don't want to use gVisor's existing FUSE implementation for this, but rather extend gVisor to support virtiofs.

For anyone looking to implement this, looking at how our FUSE implementation is set up is a good start. From there you'll have to add support for the flags that @stepancheg described and pipe the configuration into the sandbox.