Concerning LXD, snaps, and Debian
I’ve been a long-time user of LXC, going all the way back to 2014. It’s come a long way and is a great containerization technology. I use it extensively in my projects and regularly create and destroy containers as needed. The primary Linux distro I run is Debian stable, and I do my best to not pollute it with backports or non-Debian packages. This means I usually trail behind the cutting edge, but in return I have very stable systems. The first official release of Debian to ship lxc was jessie.
This post will cover my experience getting LXD 3.0 LTS running on a Debian buster system, as well as highlight concerns I have with the current path upstream LXD is taking for its packaging and distribution.
LXC to LXD
While the first stable release of LXD was in 2016, I didn’t do much with it until this past fall. The various (legacy)
lxc-* commands continue to work fine, and over the past few years I had built up quite a collection of local scripts and auto-config tooling around the underlying LXC that provided everything I needed for my container usage.
However, I knew I needed to rebuild one of my physical hardware servers due to some hardware upgrades, and thought that would make an excellent opportunity to try out LXD as well. Unfortunately LXD still isn’t included in Debian, but work is being done to get it included in the next stable release. LXD has a really cool online demo that I was able to play with and verify that pretty much all of my wrappers around LXC are no longer needed. Additionally, I’m running the 3.0 LTS release, as I don’t want to be updating every couple of months.
The currently recommended way of installing LXD is to use snaps. This is a package manager designed to distribute self-contained Linux applications, similar to AppImage and Flatpak. Rather than a traditional deb or RPM package, a snap contains all its dependencies and is by default very limited in what resources it can access. This is a plus for security, and can also make distributing an application across multiple Linux distros much easier.
I setup a test system to try out
snapd and LXD before rebuilding my server. Debian has
snapd already packaged, and it was pretty simple to get installed, followed by the snap of LXD. Things worked well, and in testing over Thanksgiving weekend things looked promising. After achieving basic proof-of-concept usage of LXD, I started testing more complex setups that mimicked my actual usage of LXC. This is where I found some issues with the snap-approach of installing a service.
While snaps are pretty cool, there are three reasons why I ultimately decided against installing LXD via the snap route:
The most significant reason, and what seems to me (and others) to be a major issue with
snapdis that there is no way to disable automatic updates of installed snaps. Don’t believe me? Just do a quick Google search for “snap disable updates”. This means that, at any arbitrary time, an update to an installed snap could be released, which will then cause the service, such as lxd, to restart. There are documented cases where this causes massive, unexpected downtime.
snapdis advertised as being ready for production use, but until disabling auto-updates is supported, I won’t use it.
The directory structure used by snaps seems a little convoluted. Plus, each snap will have a bindmount setup. While not every Linux distro strictly follows the Filesystem Hierarchy Standard, I do appreciate software that follows the principle of least surprise.
Lastly, and this is purely my own opinion,
snapdis too much of a Canonical project. There doesn’t seem to be widespread adoption of snaps beyond Ubuntu, and a lot of
snapd’s capabilities seem to work best on Ubuntu. I can’t complain too much, as Canonical is paying the main developers who work on
snapd. I just hope it doesn’t end up like the numerous Canonical projects that get dropped after a few years.
LXD on Debian
snapd was ruled out, I had to turn back to a traditional deb-based install of LXD. Currently, it’s not possible to build LXD on Debian buster due to missing golang dependencies, but there are packages for Ubuntu Bionic (18.04LTS). The stars are aligned between this release and buster, which makes it relatively easy to use bionic packages on a Debian system.
liblxc1_3.0.3-0ubuntu1~18.04.1_amd64.deb liblxc-common_3.0.3-0ubuntu1~18.04.1_amd64.deb lxcfs_3.0.3-0ubuntu1~18.04.1_amd64.deb lxd_3.0.3-0ubuntu1~18.04.1_amd64.deb lxd-client_3.0.3-0ubuntu1~18.04.1_amd64.deb lxd-tools_3.0.3-0ubuntu1~18.04.1_amd64.deb python3-lxc_3.0.1-0ubuntu2_amd64.deb (from Ubuntu disco to match python3.7 in Debian buster)
Warning! FrankenDebian ahead!
Obligatory warning: This flirts with FrankenDebian. If something breaks, you’ll be on your own. Your system will be in an unsupported state which will be significantly different from a standard install. You’ve been warned.
Updating and building LXD
After installing these packages from bionic, I had a working LXD on Debian buster. (I did have to manually pin the versions after install with
apt-mark hold liblxc1 python3-lxc lxcfs to prevent apt from trying to install the versions in Debian that have a newer epoch.) Things were looking up, except that services such as MariaDB couldn’t start within containers and were erroring out with a status of
(code=exited, status=226/NAMESPACE) in the systemd logs. A quick Google search found Debian bug #918839, from which it looked like this was an issue with a newer version of systemd and AppArmour. Luckily this was fixed in LXD version 3.0.4.
But 3.0.4 hasn’t been uploaded to bionic, and there’s a languishing bug on launchpad about this. The upstream project seems to only be releasing updated snaps now, and no longer updating debs.
Luckily, this being an open source project, we can build updated packages ourselves! As I’m very comfortable rebuilding debs, this didn’t seem too daunting. Briefly:
- Fetch the source packages with
apt-get source <pkg>(https://wiki.debian.org/Packaging/SourcePackage)
- For each version, get the current LTS release with
uscan --download-version 3.0.4(https://www.debian.org/doc/manuals/maint-guide/update.en.html#newupstream)
- For the python3-lxc package, also need to edit
debian/rulesto use python3.7 (https://wiki.debian.org/Python/Pybuild)
- Build the new package with
dpkg-buildpackage -us -uc(https://www.debian.org/doc/manuals/maint-guide/build.en.html#completebuild)
- Copy newly built deb packages and install the updated LXC, lxcfs, and LXD packages
After this was done, I had a fully functioning LXD install on Debian buster. Life is good.
LXC is cool; LXD is even cooler. The focus by the upstream developers on only releasing snaps of new versions is concerning, as
snapd is clearly not ready for use in “production” environments. However, since it’s an open source project, it’s easy to update and rebuild debs to get the latest and greatest. I’ve now got LXD 3.0 running on Debian buster, and it is indeed nicer than my wrappers. :)