Introduction

As I start writing this, it has been one day since I arrived back from CCCamp, hoping for a couple of days of sleep, and then a start to work on Spectrum in earnest. My plans for sleep today, aside from a brief intermission to attend a protest against asylum seekers in Glasgow being evicted by lock change before they had any chance to appeal (a very important issue I encourage you to read about further), were rudely interrupted by the British Prime Minister announcing that he is going to suspend Parliament in order to force through a no-deal Brexit. So a couple of hours after I got home, it was back out onto the streets to attend another protest against that. The Edinburgh protest was extremely disappointing. We marched to Parliament, but then nobody seemed to have any ideas for further action, and the whole thing just sort of fell apart after that. I gather the resistance in other places are doing better, though. I’m not holding out hope for them being able to change anything, but it still feels important to try to do something, powerless as we are.

Notwithstanding what it would do to the other 70 million people in the UK, this would be a disaster to me personally. My work and life depends on being able to spend lots of time in continental Europe, and my partner and I spent a lot of time today worrying about our future, and seriously considering leaving the UK in the near future.

And so, it’s rather bittersweet that against this backdrop of potentially having to take drastic measures to preserve my ability to remain part of the European hacker community, I describe some very important outcomes for Spectrum that came out of Camp.

Background

I was invited to the NixOS tent for a workshop that had been arranged about integrating Nix with Qubes, seeing as that was rather closely related to my work. At the start of the session, I was given an opportunity to talk briefly about my plans for Spectrum. After that, the workshop proper began, with a focus on using Nix inside Qubes guest VMs. Having spent quite a lot of time thinking about exactly this, I was able to contribute quite a lot to the conversation. There’s a lot that needs to be worked out, because the Qubes model as things stand is very focused on the idea of traditional package managers, where packages have to be installed ahead of time, and are globally accessible and can make arbitrary changes to a system. I’m rooting for the people leading this effort, because using Nix with Qubes would make running a Qubes system significantly more maintainable. Notes from the session are available, although they don’t appear to be very detailed.

One of the Qubes people was wearing a Qubes T-shirt, which just had a Qubes logo and the Qubes Master Signing Key fingerprint below it. I love that they’ve managed to make this part of their brand identity. As a Qubes-curious person a couple of years ago, it was great to be able to verify their key from sources available online that would be difficult to forge, like videos of talks (which had the fingerprint on every slide) or profiles on independent websites.

I’d hoped that maybe I’d be able to ask advice from the two Qubes developers at the session, but after the conversation was over one had to head off, and the other was keen to dive right into helping people get started on the implementation. A few hours later, though, I saw an IRC message telling me that a Qubes person was keen to meet me. Shortly afterward, I walked into the NixOS tent.

Almost everybody in the tent reacted to my entrance. “Oh, Alyssa is here!”, “Alyssa, you don’t know what you’ve just walked into”, “Shall we form a circle?”. This was going to be a lot more than the low-key conversation with a Qubes developer I’d expected.

As it turned out, everybody was interested in the conversation, and they all wanted to be a part of it. I think there were maybe eight of us? And everybody wanted to talk about my thing!

What I learned

  • Despite originally planning on them being the only method of compartmentalization, there’s probably little benefit to using containers. KVM-based virtualization gives us various tools we can use to decrease the overhead of VMs beyond what Xen affords. For example, it’s possible that in future, five VMs all running Chromium could use the same copy of Chromium in hypervisor kernel memory through DAX to a shared read-only virtual storage device, which would mean that Qubes’s huge memory requirements wouldn’t apply to us. All the innovation around DAX etc. is happening in Linux, and so KVM is really the place to be if we want to take advantage of new developments like this.

  • KVM is definitely a better choice than Xen. Qubes continues to struggle with difficulty doing hardware acceleration because of its Xen base (and the tight coupling it has with it). With KVM, we can also support a much wider range of hardware. Qubes is pretty tied to x86, which is arguably the worst platform there is in terms of freedom and auditability. (And nobody has been more vocal about this than the creator of Qubes herself.)

  • We should use crosvm (from ChromiumOS) instead of QEMU. It has a much lower attack surface, and has interesting features like virtio-wayland that could be useful to us.

  • There are very interesting possibilities regarding the use of Genode for the host system. This isn’t something I have the knowledge or capacity to do, but I can structure the system to make it an easy port for somebody else to do in future. (I may be able to arrange funding for this.) In particular, if I heavily structure software that runs on the host around dependency injection, it should map very well to a capability system. Interestingly, there has already been work on building Genode systems with Nix.

  • KVM’s hardware isolation story is very different to Xen’s. Some parts, like graphics acceleration, should work a lot better, but others are not there yet and may require kernel work before full isolation can be achieved. Again, I am probably not the person to address these shortcomings, but they almost certainly could be solved with the right person working on them and some further funding.

  • There needs to be a way to inject additional software into a running isolation unit. I was sceptical of this, but was eventually convinced by the use case of needing to add debug symbols and tools to a running VM to troubleshoot some software. We spent a while talking about this, but realised that Nix’s model makes parts of this rather easy. There’s no need for application units to be running any sort of package manager, so we just need to expose extra store paths. How best to do this remains an open question, but the Nix store being read-only means that a lot of security concerns around doing this at the filesystem level are less troublesome. We were able to get some valuable perspective from Michael Stapelberg (of i3 fame), who has recently been working on optimising package manager performance with distri. (As an aside, Michael wasn’t part of the conversation initially, but because he was at Camp, somebody was able to fetch him and rope him in when it became clear how valuable his input would be. I love that this was possible, and it demonstrates just how valuable the events by the CCC and others are to our community.)

  • While we should definitely push for fully reproducible builds in Nixpkgs as much as possible, in the meantime we’re not quite there yet. This means that there’s still a need for trustworthy build hardware, and even though most people will likely run Spectrum on x86, there’s no real reason to use x86 to build it when there are other options available. So, I plan to buy some of Raptor Computing Systems’ POWER9 hardware, which has free, auditable software right down to the CPU microcode, and use it to build packages for PowerPC and x86 alike. I will endeavour to have Spectrum support PowerPC as a first-class citizen in addition to x86, providing trustworthy secure computing on modern hardware for the first time.

Overall, the conversation was hugely inspiring. It was amazing to realise I already have a community who are invested in my work and want to see it succeed. That a Qubes developer was interested was a big help with some of the imposter syndrome I’d been feeling, and it was great to finally get the constructive criticism I’d been searching for for over a year. While “this is great, I can’t wait to use it” is extremely motivating, and especially useful to encourage me to apply for the grant, there comes a point where lack of criticism becomes an issue, as it’s very difficult to figure out whether an idea is good or not without somebody to tell you which parts need to be improved.

It was also good to identify scope for large projects that other people could take on in relation to Spectrum. My impression is that there may well be funding available to make these things happen, especially if they would have benefit to other projects independent of Spectrum (like, say, Linux or Genode). I’d love to have other people on board in such a big way of pushing Spectrum forward, and I’ll definitely be doing what I can to accommodate this when the time feels right.

Realising that I already have a community made it clear that we really needed communication channels immediately. We’ve definitely outgrown #nixos-chat. I followed up with Freenode about an inquiry into official group registration I made a few weeks ago, and as a result we now have #spectrum, which at the time of writing has 16 members (one of which is a logging bot kindly provided by samueldr)! There’s also quite a clear need for longer-form communication, and so setting up a Spectrum mailing list is my immediate priority.

Next steps for me

  • Submit a project plan to NLnet. This is a necessary step before I can receive any of my grant money, and it’s been on my todo list for a couple of weeks. I got it sent off a few hours before publishing this post.

  • Set up more project infrastructure. The IRC channel is a good start, but we really need a mailing list, and subsequently repository hosting and a better website that links to all this stuff. This is the first subtask in my project plan, and I’ll be jumping right into it next.

Next steps for you, a person interested in Spectrum

  • Join #spectrum on Freenode. There’s already interesting discussion there, even when I’m not part of the conversation. In these early stages, the best thing you can do to help Spectrum succeed is to join the conversations and help keep the community we already have engaged.

  • Keep your eyes posted for the announcement of the Spectrum mailing list. I’ll post an IRC message, and update this post, and then at some point the Spectrum website. Alternatively, if you want to make sure you’re added to the list when it launches, send me mail or an IRC message and I’ll make sure to subscribe you.

Mailing lists are now available.

Conclusion

It’s really happening! I can’t believe how much momentum there already is behind Spectrum, and how many people are invested in its success. At this point, I really feel like I’m the limiting factor – I’ve barely written any code yet, and there’s still so much to do! All I can do is ask you to stay engaged in #spectrum and be patient.

All of this Community Management™ is new to me, so I hope you can bear with me while I figure it out. Stay engaged, get involved, let me know when I make mistakes, and we’re going to make something amazing together.

I’d like to thank everybody who was part of putting this conversation together, especially adisbladis, who arranged the conversation, and Jean-Philippe Ouellet, the Qubes person.

Changelog

Belatedly update with mailing list link.

  • Thank adisbladis and Jean-Phillipe by name.
  • Some technical clarifications and corrections (shown inline), thanks to Jean-Phillipe.