A Technical Overview of virtio-wayland

If you’re just looking for a plug-and-play virtio_wl demo you can try out right now, you might want to skip to the Demo section.

virtio_wl (seemingly more commonly called virtio-wayland, which I used in the title because it’s a bit clearer, and also previously called virtwl) is a transport for the Wayland protocol over virtio, the standard interface for communication with KVM virtual machines. It’s interesting because it means that applications running in separate VMs can appear as part of a single desktop environment, while still remaining isolated from each other because of Wayland’s security-focused design.

virtio_wl was developed by Google for Chrome/Chromium OS (Chromium OS is the name for the open source operating system; Chrome OS is the name given to the operating system you get on Chromebook hardware, which has a small(?) amount of proprietary modifications by Google). Chromium OS allows non-Chromium applications to be run in Debian-based VMs and be integrated into Chromium OS’s Chromium-based desktop environment.

Despite being an extremely promising technology, I’m not aware of any use of virtio_wl outside Chromium OS. In fact, as far as I know, I’m the first person to use it in such a way. I’m interested in virtio_wl for Spectrum, the compartmentalized operating system I’m developing.

In this article, I want to give a general overview of all the required components to make virtio_wl work. I’m not going to provide step-by-step instructions for building each component, in part because it will be different on every system, but largely because you won’t end up with anything particularly useful out of the end. In its current state, virtio_wl outside of Chromium OS is really only useful to people who want to build on top of it, and if you’re doing that, you probably know better than me how to build it with your choice of packaging system, etc. I will, however, provide notes on the build process, and try to document anything surprising or confusing. Additionally, I will provide the virtio_wl demo that I’ve been working with, if you just want to try it out and see how well it works.

Minijail

Minijail is a sandboxing program that is used in Chromium OS to lock down virtio devices with namespaces and seccomp to limit damage potential in the event of a VM escape. It’s also used in Android.

Minijail is a nice easy build — just Make. I’m surprised it’s not more widely packaged because it seems to be a fairly general purpose tool, but at the time of writing it’s only in Gentoo (and derivatives), Void, Nixpkgs, and the AUR.

There also appears to be ongoing work to build Minijail with Cargo, despite it being written in C, so it can be automatically built as a dependency of crosvm rather than having to be built and installed separately. This means that at some point in the future this step of manually building Minijail may not be required any more.

crosvm

crosvm is Chromium OS’s Virtual Machine Monitor (VMM). It is the only VMM that supports virtio_wl, so it’s what we’ll be using. crosvm has some other interesting things going for it — it’s written in Rust, a memory-safe language, and was the base for Firecracker, which is a VMM with a marketing budget and a built-in HTTP server. crosvm and Firecracker are not general purpose VMMs, but are designed to do the bare minimum to virtualize a Linux kernel, which gives them performance and code size benefits over general purpose VMMs like QEMU. This is what the Firecracker marketing department calls a “micro VM”.

Unlike most of Chromium OS, which is all integrated into a single Git repository and build system, crosvm is kept separate, and is almost a pretty standard Rust project.

The one unusual thing about building crosvm is that it expects to be in a Chromium OS tree. Specifically, it expects to be able to access the chromiumos/third_party/adhd tree at ../../third_party/adhd. This tree seems to be a collection of various audio bits for Chromium OS, most of which does not appear to actually be “third party”. (The Chromium OS codebase is in general not very well organised.) crosvm uses the audio_streams and cras libraries. The paths to these libraries are defined in Cargo.toml. You could either just create the directory layout it expects, or try patching that. I chose the former.

crosvm also depends on libminijail0, so you’ll need to make that available too.

One more thing to say about crosvm: most of Chromium OS is very poorly documented, but the crosvm source code actually has quite good documentation. I assume this is because of Rust’s amazing documentation culture. This doesn’t help you much with using crosvm, but if you want to hack on it it’s invaluable. I host a copy of the HTML version of crosvm’s documentation, because it isn’t online anywhere else as far as I can tell — you’d need to generate it locally.

Linux

The virtio_wl kernel module lives in the Chromium OS kernel tree. The easiest way to use it is to just build a Chromium OS kernel to use in the VM. You could also just extract the virtio_wl driver and add it to an upstream kernel. I have successfully used both approaches.

The Chromium OS kernel builds just like the upstream one, so I won’t go into detail here. Make sure to enable CONFIG_VIRTIO_WL.

vm_protos

vm_protos is a collection of Protocol Buffer proto files used across the VM-related components of Chromium OS. It’s also our first encounter with platform2, which contains more than a hundred Chromium OS components, integrated into a single build system, common-mk.

Since vm_protos is just a small collection of proto files, it would be easy enough to build them manually, but since the next step also requires common-mk, we might as well get it working now.

common-mk is built on top of the GN build system. GN stands for “Generate Ninja”, which is exactly what it does. So, to build anything with GN, you run gn gen to generate some Ninja files into a build directory, and then run ninja inside that directory to do the build.

I’ve found that common-mk fails to work for me in some confusing ways, in that I don’t understand how the errors I’m seeing can’t also be a problem for Google. My best guess is that they have an internal version of GN that works slightly differently. I’ve encountered some other stuff that only makes sense this way too, e.g. -Xclang-only and -Xgcc-only flags that are interpreted by some sort of wrapper they have around GCC and Clang. Anyway, to get common-mk to evaluate using the version of GN available to you and me, you’ll need this patch I wrote.

As an aside, if when you’re reading this any of the patches I’ve linked to don’t apply any more, or you’re having trouble with the components I’ve mentioned here, try looking in the master branch of my Nixpkgs tree. At the time of writing, all Chromium OS components live in pkgs/os-specific/linux/chromium-os. find pkgs/os-specific/linux/chromium-os -name '*.patch' will show you all the patches I’m applying to Chromium OS components. Not all of these will be generally useful — some of it will be Spectrum-specific modifications, but a lot of it probably will be. It’s possible I’ve had the same issue you’re having now and solved it already. I’ve used permalinks in this post so that the links don’t break in future, but the flipside of that is that the linked content will be frozen to the moment in time I publish this post.

common-mk requires a lot of GN arguments (set using --args). Here’s the ones I found needed to be set for vm_protos, and what I set them to:

Name	Value
`ar`	`"ar"`
`cc`	`"cc"`
`cxx`	`"cxx"`
`libdir`	the prefix of my vm_tools package
`pkg_config`	`"pkg_config"`
`platform2_root`	`"."`
`platform_subdir`	`"vm_tools/proto"`
`use`	`{cros_host=false fuzzer=false profiling=false tcmalloc=false}`

Sommelier

Sommelier is the bridge between the virtio_wl kernel module and a Wayland client. It implements the interface of a Wayland compositor, giving unmodified Wayland clients an interface they know how to speak, and then forwarding that over the virtio_wl device. This could have been implemented as part of the kernel module, but it was apparently easier to do it in userspace. (A kernelspace Wayland compositor, even just a forwarding one, sounds like a bit much, anyway!)

Sommelier is the most awkward to build of all the component I’ll describe here. The BUILD.gn file for Sommelier attempts to build two demo programs (x11_demo and wayland_demo), and both of these depend on two Chromium OS libraries, libbrillo and libchrome (also confusingly known as libbase). These are possible to build outside of Chromium OS, but libbrillo needs five patches and they have a lot of their own dependencies (including two different versions of ModemManager as far as I can tell). So rather than doing that, it makes much more sense to not build the demos. Sadly, Sommelier’s BUILD.gn does not support that, so you’ll need my patch for that. You’ll also need another patch to fix some broken GN code similar to the problems with common-mk described earlier. Finally, Sommelier’s implementations of some Wayland protocols are quite outdated, to the point where most Wayland clients I tried just wouldn’t work with it. This is largely due to Sommelier still using the unstable xdg-shell protocol, rather than the newer standardized one. puck was kind enough to contribute a final patch to upgrade this, and with this I was able to run most Wayland clients (but not all, because there are still other outdated protocols).

After all that patching, Sommelier can be built. vm_protos will need to be discoverable through pkg-config. I used these GN arguments.

Name	Value
`ar`	`"ar"`
`cc`	`"cc"`
`cxx`	`"cxx"`
`libdir`	the prefix of my sommelier package
`pkg_config`	`"pkg_config"`
`platform2_root`	`"."`
`platform_subdir`	`"vm_tools/proto"`
`use`	`{amd64=true arm=false cros_host=false profiling=false tcmalloc=false}`
`use_demos`	`false`

The values of use.amd64 and use.arm are of course specific to the platform I was building on, and use_demos is the flag added by my patch.

There are several ways to use Sommelier, documented in its README. The simplest is to use sommelier as a wrapper program, where you run, e.g. sommelier weston-terminal, and Sommelier starts up and spawns the client, passing its Wayland socket to the child.

Putting it all together

crosvm run --help gives a decent overview of available crosvm options. Make sure to give it a Wayland socket, and a virtio_wl-enabled kernel image. Once you have a VM running, check that crosvm is exposing the Wayland socket to the VM, and that the VM kernel is picking it up, by checking that the VM has a device at /dev/wl0. Run Sommelier as a wrapper as described above, and a window should appear on the host system’s compositor.

Demo

If you just want to try virtio_wl out and see how it works, and how it feels to use, you can run my self-contained reproducible demo. It should just work.

Assuming you have Nix installed, and are running this inside a Wayland compositor (e.g. in a graphical terminal emulator):

git clone https://spectrum-os.org/git/nixpkgs
cd nixpkgs
git checkout 3fb8d00d089718abbdee679ad1a21dadb2428562
$(nix-build start-vm.nix)

This will build all of the dependencies (if necessary), and then start two VMs. One will display a Weston Terminal window, and the other will share its root file system with the first one over 9p through a TAP device on the host computer. (This is to demonstrate inter-VM networking, an area I’m currently working on. I’d like to get this working with guest-to-guest virtio, so the host’s network stack isn’t involved at all.) If you cd to /mnt in the terminal window, you’ll find the file system of the second VM.

It assumes that the 10.0.103.0/24 and 10.0.104.0/24 ranges are available on the host for addresses for the TAP device.

Summary

For a complete virtio_wl setup, a hypervisor (e.g. crosvm) needs to expose the host’s Wayland socket to the guest kernel over virtio. A kernel driver (virtio_wl) in the guest exposes the virtio device to userspace, and then a userspace application in the guest acts as a forwarding Wayland compositor, interfacing between a Wayland client and the virtio device.

Despite being officially supported by Google only for running Android apps on Chrome OS, virtio_wl works surprisingly well as a general-purpose mechanism for using a shared windowing system with applications running in multiple isolated environments. The biggest challenge is that Sommelier needs to be updated to speak more modern Wayland protocols, but this is an achievable task even for a small project. Indeed, I intend for the Spectrum project to create further progress in this area.

Finally, I’d like to thank my sponsors for making it possible for me to take a few days out of working on Spectrum to write this article. Their support means that I can afford the time to take this knowledge that I’ve acquired, and write it down so that it can be useful to more people and projects than just me and mine. Information on virtio_wl was very difficult for me to find, and a lot of it had to be acquired from digging through the code and trial and error. I hope that this article can help make possible more uses of this extremely promising technology outside of Chromium OS. Please consider supporting my work through GitHub Sponsors (which will match your donation) or Liberapay to enable me to produce more content like this as I work to make compartmentalized computing simpler and more usable.