Posted on June 10, 2022 by Richard GoulterTags:

As with anyone who’s been using git for long enough, I’m familiar enough with SSH keys (Even to the point where I don’t really need to follow the GitHub docs for generating a new key.

My developer journey hasn’t led me to build the same intuition for GPG keys.

So, when I see fancy wiki pages and guids, such as Debian wiki’s page on subkeys, on offline master keys, or on an airgapped master key, it sounds really fancy; but I’m left curious to get an intuition for what’s going on.

I hope to use this blogpost illustrate enough of the details to build an intuition.

Public Key Cryptography

The Wikipedia page on Public Key cryptography has some nice diagrams.

This itself should still be relatively straightforward, comparing with how SSH keys get used with git forges like GitHub, and with SSHing into servers:

• You have a key pair with a private key and public key,

• you keep the private key private,

• you share the public key with the servers you want to access to.

  • Having the private key gives access. And this is ‘secure’ because only the user with the private key which matches the public keys will be granted access.

So it makes sense: Encrypting;

• You encrypt something for someone else using their public key.

  • Everyone has their public key, so everyone can send them something.

  • Only they have their private key, so only they can read the message.

Or signing:

• You sign something with your private key.

  • You’re the only person who has your private key, so only you can sign something.

  • Everyone has your public key, so everyone else can verify you signed it.

The Rote GPG Commands

So far, we can just label everything above with “GPG” and it still makes sense:

e.g. Key generation could be done by invoking a command like:

gpg --quick-generate-key "Richard Goulter (NixOS Desktop) <richard.goulter@rgoulter.com>"

e.g. exporting a key, to copy the public part to GitHub or whatever could be done by invoking a command like:

gpg --export --armor "Richard Goulter (NixOS Desktop)"

e.g. encrypting/decrypting a message could be done by invoking a command like:

gpg --encrypt --recipient "My Colleague" document.txt

and the recipient could decrypt it with:

gpg --decrypt document.txt.gpg

e.g. signing could be done by invoking a command like:

gpg --clearsign file.txt

or

gpg --detach-sign file.txt

and the signature could be verified with a command like:

gpg --verify file.txt.asc

or

gpg --verify file.txt.sig

What Starts to Confuse Me: Subkeys

Okay, but having created a key, then list the keys:

gpg --list-keys

outputs something like:

pub   rsa4096 2020-10-05 [SC] [expires: 2022-08-01]
      D9463A85F093811B887DE4B03229F4BB09CC8A22
uid           [ultimate] Richard Goulter (NixOS Desktop) 
sub   rsa4096 2020-10-05 [E] [expires: 2022-08-01]

and secret keys:

gpg --list-secret-keys

outputs something like:

sec   rsa4096 2020-10-05 [SC] [expires: 2022-08-01]
      D9463A85F093811B887DE4B03229F4BB09CC8A22
uid           [ultimate] Richard Goulter (NixOS Desktop) 
ssb   rsa4096 2020-10-05 [E] [expires: 2022-08-01]

I can see the only difference is pub, sub, vs sec and ssb. What?

What’s [SC], [E]?

And if you enable some recommended options in ~/.gnupg/gpg.conf:

keyid-format LONG
with-fingerprint
with-keygrip

re-running:

gpg --list-keys

outputs:

pub   rsa4096/3229F4BB09CC8A22 2020-10-05 [SC] [expires: 2022-08-01]
      Key fingerprint = D946 3A85 F093 811B 887D  E4B0 3229 F4BB 09CC 8A22
      Keygrip = 480A928A7AA17A693FDF4F4E9DEAEBEE1B778FB7
uid                 [ultimate] Richard Goulter (NixOS Desktop) 
sub   rsa4096/15BAE7FB6F3DA787 2020-10-05 [E] [expires: 2022-08-01]
      Keygrip = A996CF5366A2FFCA11EE958FBA3AC4BDFDB6AC76

I feel like this one I’m allowed to be confused by.

Long Id?

Fingerprint??

Keygrip???

A Key Consisting of Multiple Keys

Uh. Hopefully it’s not too late in this blogpost to mention it.But “key” gets used in multiple ways e.g. “running gpg --quick-generate-key generates a GPG ‘key’”, and in “a GPG ‘key’ consists of at least one key pair”.

Generally, context is often enough to disambiguate these. But given that GPG is confusing enough as it is, it’s worth mentioning the ambiguity.

Taking a look at the terminology GPG’s manual uses: https://www.gnupg.org/gph/en/manual.html#CONCEPTS

GnuPG uses a somewhat more sophisticated scheme in which a user has a primary keypair and then zero or more additional subordinate keypairs. The primary and subordinate keypairs are bundled to facilitate key management and the bundle can often be considered simply as one keypair.

This excerpt refers to the keys being ‘bundled’, and that the bundle can often be considered as a keypair.

The GnuPG FAQ has an answer which better describes the ambiguity. https://www.gnupg.org/faq/gnupg-faq.html#define_key

The word ‘key’ is unfortunately ambiguous. It can either refer to the mathematical structures that allow encryption, decryption, signing and verification to occur, or to the rather large blobs of data that contain those mathematical structures as well as information about the person associated with it, additional subkeys, and so forth.

With respect to the large blobs of data, it is preferable to call them ‘certificates’, so that the word ‘key’ may be unambiguously recognized as meaning just the mathematical structures. Unfortunately, this is a custom that seems to be honored mostly in the breach.

In this blogpost, I’ll try to use the word “key” in scare quotes to indicate “bundle of key pairs”.

Other GPG Features/Complications

So while I feel like the idea of “a key pair, with a private part and a public part” is understandable, there are other features of GPG “keys” which make things more complicated:

GPG “keys” can be set with an expiry date.(This is useful as a way of indicating you’ve still got access to the private key).

GPG “keys” can have multiple “User IDs” (Name + email). Photos can be included in the GPG “key”.

GPG “keys” can be signed. Or more specifically: the keys and UIDs can be signed.– I don’t find this intuitive. Sure, Bruce Wayne and Batman are different identities, but I don’t see why they’d want to use the same key.

GPG “keys” can be revoked. That is, an indication that other subkeys or UIDs or signatures should not be trusted.– I didn’t find this intuitive.

GPG “keys” can have some level of trust.– This also seems unintuitive to me.

Why Subkeys are Subkeys

I found this StackOverflow answer about why different keys are used for encryption vs signing illuminating:

If you look into the details of the math of public-key encryption, you will discover that signing and decrypting are actually identical operations. Thus in a naïve implementation it is possible to trick somebody into decrypting a message by asking them to sign it.

i.e. it would be bad a idea to use the same (sub)key for both signing and encryption. So, GPG uses a separate subkey for at least encryption.

This answer also explains what the [SC] and [E] refer to:

E = encrypt/decrypt (decrypt a message you received encrypted for you to read)S = sign (sign data. For example a file or to send signed e-mail)C = certify (sign another key, establishing a trust-relation)A = authentication (log in to SSH with a PGP key; this is relatively new usage)

The Wikipedia page about RSA gives an example of using RSA keys, which is useful for understanding “signing and decrypting are identical operations”.

The GPG User Guide (Section “Key Management”, subsection “Key integrity”) sort of does explain the “master” and “subordinate” part:

Using digital signatures is a solution to this problem. When data is signed by a private key, the corresponding public key is bound to the signed data. In other words, only the corresponding public key can be used to verify the signature and ensure that the data has not been modified. A public key can be protected from tampering by using its corresponding private master key to sign the public key components and user IDs, thus binding the components to the public master key. Signing public key components with the corresponding private master signing key is called self-signing, and a public key that has self-signed user IDs bound to it is called a certificate.

The ‘self-signing’ of sub-keys by the master key is a detail which wasn’t clear to me.

“Subordinate” is a surprising word here.Even the absolutely excellent post “anatomy of a gpg key” expands “subkey” to “sub-component of another key”, not “subordinate key”.

I further feel vindication in that the website https://gpg.wtf/ also discusses ‘what a subkey is’ in its “things that are confusing about gpg”.

The “master key signs the subkeys” can be seen by running a command like:

gpg --check-signatures 0x3229F4BB09CC8A22

which outputs something like:

pub   rsa4096/0x3229F4BB09CC8A22 2020-10-05 [SC] [expires: 2022-08-01]
      Key fingerprint = D946 3A85 F093 811B 887D  E4B0 3229 F4BB 09CC 8A22
      Keygrip = 480A928A7AA17A693FDF4F4E9DEAEBEE1B778FB7
uid                   [ultimate] Richard Goulter (NixOS Desktop) 
sig!3        0x3229F4BB09CC8A22 2022-05-03  Richard Goulter (NixOS Desktop) 
sub   rsa4096/0x15BAE7FB6F3DA787 2020-10-05 [E] [expires: 2022-08-01]
      Keygrip = A996CF5366A2FFCA11EE958FBA3AC4BDFDB6AC76
sig!         0x3229F4BB09CC8A22 2022-05-03  Richard Goulter (NixOS Desktop) 
gpg: 2 good signatures

i.e. The GPG “key” has:

• the public master key,

• a UID which is signed with the [SC] key,

• a subordiate key, which is intended for [E] usage, as signed with the [SC] key.

Illustrating the Rote GPG Operations

Going back over the rote commands again, now with a diagrams to illustrate subkeys at work:

Key Generation

could be done by invoking a command like:

gpg --quick-generate-key "Richard Goulter (NixOS Desktop) <richard.goulter@rgoulter.com>"

Export Public Key

Exporting the key to copy the public part to GitHub or whatever:

could be done by invoking a command like:

gpg --export --armor "Richard Goulter (NixOS Desktop)"

Encrypting/Decrypting a Message

could be done by invoking a command like:

gpg --encrypt --recipient "My Colleague" document.txt

and the recipient could decrypt it with:

gpg --decrypt document.txt.gpg

Signing

could be done by invoking a command like:

gpg --clearsign file.txt

or

gpg --detach-sign file.txt

and the signature could be verified with a command like:

gpg --verify file.txt.asc

or

gpg --verify file.txt.sig

GPG for SSH

Again, coming from the mindset of “SSH keys for pushing to git repositories, gpg for signing stuff”, hearing “GPG ‘keys’ can be used for SSH too” sounds like wizardry.

This is also where “keygrip” comes into use. As this blogpost discussing GPG fingerprint vs SSH key fingerprint vs GPG keygrip explains, the keygrip is a way of identifying the key regardless of protocol/usage.

Now that I have a better understanding of subkeys, it seems the main points for “using GPG for SSH” involves two things to understand:

• Using a suitably configured gpg-agent instead of ssh-agent, by pointing the SSH_AUTH_SOCK and SSH_AGENT_PID to GPG.

• Having a subkey with [A] capability.

Visualisation of the Offline Master Key Setup

With the above in mind, a summary of the result of these tutorials for setting up master keys could be visualised as follows:

Which seems less complex than all the confusion I had over it made it out to be.

As the Debian wiki article pages discuss, because the private master key is used to self-sign the subkeys in the GPG “key”, it can then be used to revoke the other subkeys.

Yubikey as a GPG Smart Card

Yubikeys can then allow a nicer User Experience for using GPG keys. e.g. the Yubikey supports 3 slots for private keys: for signing, encryption, or authentication.

e.g. Yubico’s PGP walkthrough.

Newer post Older post

The outdated and historical nature of the Filesystem Hierarchy Standard means traditional Linux distributions have to go to great lengths to separate “user data” from “system data.”

NixOS’s filesystem architecture does cleanly separate user data from system data, and has a much easier job to do.

Traditional Linuxes

Because FHS mixes these two concerns across the entire hierarchy, splitting these concerns requires identifying every point across dozens of directories where the data is the system’s or the user’s. When adding ZFS to the mix, the installers typically have to create over a dozen datasets to accomplish this.

For example, Ubuntu’s upcoming ZFS support creates 16 datasets:

rpool/
├── ROOT
│   └── ubuntu_lwmk7c
│       ├── log
│       ├── mail
│       ├── snap
│       ├── spool
│       ├── srv
│       ├── usr
│       │   └── local
│       ├── var
│       │   ├── games
│       │   └── lib
│       │       ├── AccountServices
│       │       ├── apt
│       │       ├── dpkg
│       │       └── NetworkManager
│       └── www
└── USERDATA

Going through the great pains of separating this data comes with significant advantages: a recursive snapshot at any point in the tree will create an atomic, point-in-time snapshot of every dataset below.

This means in order to create a consistent snapshot of the system data, an administrator would only need to take a recursive snapshot at ROOT. The same is true for user data: take a recursive snapshot of USERDATA and all user data is saved.

NixOS

Because Nix stores all of its build products in /nix/store, NixOS doesn’t mingle these two concerns. NixOS’s runtime system, installed packages, and rollback targets are all stored in /nix.

User data is not.

This removes the entire complicated tree of datasets to facilitate FHS, and leaves us with only a few needed datasets.

Datasets

Design for the atomic, recursive snapshots when laying out the datasets.

In particular, I don’t back up the /nix directory. This entire directory can always be rebuilt later from the system’s configuration.nix, and isn’t worth the space.

One way to model this might be splitting up the data into three top-level datasets:

tank/
├── local
│   └── nix
├── system
│   └── root
└── user
    └── home

In tank/local, I would store datasets that should almost never be snapshotted or backed up. tank/system would store data that I would want periodic snapshots for. Most importantly, tank/user would contain data I want regular snapshots and backups for, with a long retention policy.

From here, you could add a ZFS dataset per user:

tank/
├── local
│   └── nix
├── system
│   └── root
└── user
    └── home
        ├── grahamc
        └── gustav

Or a separate dataset for /var:

tank/
├── local
│   └── nix
├── system
│   ├── var
│   └── root
└── user

Importantly, this gives you three buckets for independent and regular snapshots.

The important part is having /nix under its own top-level dataset. This makes it a “cousin” to the data you do want backup coverage on, making it easier to take deep, recursive snapshots atomically.

Properties

• Enable compression with compression=on. Specifying on instead of lz4 or another specific algorithm will always pick the best available compression algorithm.
• The dataset containing journald’s logs (where /var lives) should have xattr=sa and acltype=posixacl set to allow regular users to read their journal.
• Nix doesn’t use atime, so atime=off on the /nix dataset is fine.
• NixOS requires (as of 2020-04-11) mountpoint=legacy for all datasets. NixOS does not yet have tooling to require implicitly created ZFS mounts to settle before booting, and mountpoint=legacy plus explicit mount points in hardware-configuration.nix will ensure all your datasets are mounted at the right time.

I don’t know how to pick ashift, and usually just allow ZFS to guess on my behalf.

Partitioning

I only create two partitions:

1. /boot formatted vfat for EFI, or ext4 for BIOS
2. The ZFS dataset partition.

There are spooky articles saying only give ZFS entire disks. The truth is, you shouldn’t split a disk into two active partitions. Splitting the disk this way is just fine, since /boot is rarely read or written.

Note: If you do partition the disk, make sure you set the disk’s scheduler to none. ZFS takes this step automatically if it does control the entire disk.

On NixOS, you an set your scheduler to none via:

{ boot.kernelParams = [ "elevator=none" ]; }

NixOS’s clean separation of concerns reduces the amount of complexity we need to track when considering and planning our datasets. This gives us flexibility later, and enables some superpowers like erasing my computer on every boot, which I’ll write about on Monday.

What a treat it is to review the Framework laptop a few months before I’ll be buying my own.

The Framework promises to be an powerful, high-end ultra thin laptop with the stand-out feature of easy repairs and upgrades. I think they’ve done it.

Let’s install NixOS.

Live Media

You’ll need an install image with Linux 5.13 or newer to have a working Wi-Fi card.

This presents a challenge: the ISOs provided by https://nixos.org today use Linux 5.10.

If you’re comfortable installing NixOS without a GUI, you can fetch a minimal 21.05 ISO with the latest kernel from Hydra.

Optionally, Building Our Own Live Media

Building your own customized install media is straightforward. You’ll need a Linux machine with Nix.

First, write the following in to a file named custom-media.nix:

{ pkgs, modulesPath, ... }: {
    imports = [
        "${modulesPath}/installer/cd-dvd/installation-cd-graphical-gnome.nix"
    ];
    boot.kernelPackages = pkgs.linuxPackages_latest;
}

Then enter a nix-shell with nixos-generators and build the media:

$nix-shell -p nixos-generators
nix-shell$nixos-generate -I nixpkgs=channel:nixos-unstable --format iso --configuration ./custom-media.nix 
unpacking 'https://nixos.org/channels/nixos-unstable/nixexprs.tar.xz'...
these derivations will be built:
  ...snip...
/nix/store/gnnbjvd916yh1f4svbgrssq94550pbxl-nixos-21.11pre304626.8ecc61c91a5-x86_64-linux.iso/iso/nixos-21.11pre304626.8ecc61c91a5-x86_64-linux.iso

Then copy the ISO to my USB disk, which is called /dev/sda.

Note: the sync at the end is critical.

$ sudo cp /nix/store/gnnbjvd916yh1f4svbgrssq94550pbxl-nixos-21.11pre304626.8ecc61c91a5-x86_64-linux.iso/iso/nixos-21.11pre304626.8ecc61c91a5-x86_64-linux.iso /dev/sda
$ sudo sync

Disabling Secure Boot

1. Reboot
2. Enter the Firmware Configuration (Fn + F2)
3. Navigate to the Security tab
4. Select Secure Boot
5. Select Enforce Secure Boot
6. Select Disabled
7. Save and reboot with Fn + F10

Note: Secure Boot will prevent unsigned software from running. NixOS does not support Secure Boot today.

Booting Your Media

It appears the machine can boot from any USB port.

1. Reboot
2. Enter the Boot Manager (Fn + F12)
3. Select your USB disk

Installing NixOS

Follow the standard NixOS installation instructions.

After running nixos-generate-config, edit /mnt/etc/nixos/configuration.nix and add a few lines:

• boot.kernelPackages = pkgs.linuxPackages_latest; - for WiFi support
• services.fprintd.enable = true; for fingerprint support

Continue the installation procedure.

Seriously. That’s it.

The review unit they shipped me is pre-release hardware, but well specced out and lovely:

• 11th Gen Intel i7-1185G7, up to 4.8GHz
• 16G of RAM
• Intel(R) Wi-Fi 6 AX210 160MHz, REV=0x420
• 2 USB-C ports, a mini-display port, and a Micro SD slot

Its performance seems quite good, easily good enough for my day to day work. It has no problems driving my 38” ultrawide display.

The machine is silent except when I’m compiling, and it has never felt hot. After a long compilation some areas near the hinge are noticably warm, but the hand rests are still cool.

The keyboard and trackpad feel, to me, perfect. The keys feel crisp but not heavy. The trackpad feels large and precise. That said, click is a bit too clicky.

Hardware switches for the audio and camera are nice, but a bit hard to toggle. The switches are very small and smooth, making it difficult to move. The switches are part of the replacable bezel, and this may be easily fixed.

The power brick is UL Energy Verified and just slightly larger than the Dell XPS 9300 brick.

The Intel Wi-Fi 6 AX210 has integration issues with Linux. Sometimes the wifi firmware crashes on startup, but it seems to be no more frequent than 50/50. Once it is booted, the wifi is rock solid.

QR Codes on Everything

Everything has a QR code as a link to instructions. And I mean everything: down to the cable connecting the wall to the power brick.

However, many of them don’t actually lead anywhere. I wasn’t too surprised to see the power cable’s QR code was 404ing, but a bit surprised to see the link for RAM replacement 404ing. I am sure this is a work in progress and these QR codes will all work.

Relative to the XPS 9300

I want the smallest, lightest laptop I can buy, and the Framework is a strong contender here.

This laptop is essentially the same width and thickness: within a milimeter or two. The framework comes in at just 60 grams heavier than the XPS (1330g vs. 1270g.)

One major improvement over the 9300: The fingerprint sensor is supported by recent versions of fprintd out of the box, and no TOD patches are required. The sensor is fast and reliable, whereas the Dell XPS 9300’s TOD Goodix drivers are nearly worthless on Linux.

Some aspects of the Framework are a bit more “industrial” feeling:

• The cable from the brick to the wall looks like a desktop PC’s cable. Thick, shiny plastic. compared to the more polished, polished, on-brand look of the Dell cable.
• The rubber feet are chunky, and not super sticky.
• Opening the lid is not super easy like on the XPS.

Overall, I am extremely satisfied by Framework. The team seems great, and their values appear to match my own. I’ll be pulling out my wallet later this year when my current laptop isn’t so new. I think they’ve fulfilled their promise, and have created a refreshing machine: an ultralight which is servicable.

Yes, please.

grahamc’s “Erase your darlings” blog post is an amazing example of what a snapshotting filesystems (zfs) combined with an immutable, infrastructue-as-code OS (NixOS) can achieve. To summarize the post, grahamc demonstrates how to erase the root partition at boot while opting in to state by getting NixOS to symlink stuff to a dedicated partition. This restores the machine to a clean state on every boot, preserving the “new computer smell”.

I believe the main selling point of this concept of opt-in state is that it makes it dead simple to keep track of ephemeral machine state (everything not explicitly specified by your NixOS configuration) and enforces elimination of Configuration Drift. While the benefits of this are clear for servers, this also works pretty well with workstations and laptops, where you gradually accumulate junk in /etc and /var which you never can be completely confident in deleting.¹

Here are some notes on how to reproduce the setup with an encrypted² btrfs root, along with a few tips for a nicer laptop experience. The instructions for encrypted btrfs root are heavily based on this blog post.

Making a Live USB

The laptop I’m currently using is a Dell XPS-13 2-in-1 (7390) with a fair number of issues running on Linux, some of which interferes with boot. Fortunately, most of these have been fixed in newer kernels, but the default installation ISO ships an older kernel version, so we need a custom ISO. Building an ISO with a custom configuration for NixOS is shockingly simple; following the instructions on the wiki:

# iso.nix
{ config, pkgs, ... }:
{
  imports = [
    # installation-cd-graphical-plasma5-new-kernel.nix uses pkgs.linuxPackages_latest
    # instead of the default kernel.
    <nixpkgs/nixos/modules/installer/cd-dvd/installation-cd-graphical-plasma5-new-kernel.nix>
    <nixpkgs/nixos/modules/installer/cd-dvd/channel.nix>
  ];
  hardware.enableAllFirmware = true;
  nixpkgs.config.allowUnfree = true;
  environment.systemPackages = with pkgs; [
    wget
    vim
    git
    tmux
    gparted
    nix-prefetch-scripts
  ];
}

The image can be built with

nix-build '<nixpkgs/nixos>' -A config.system.build.isoImage -I nixos-config=iso.nix

Then, we write the ISO to a USB stick like so:

sudo dd if=./result/iso/nixos-20.03....-x86_64-linux.iso of=/dev/<usb device> bs=1M status=progress

NixOS Installation

Once we’ve booted into a graphical session, we need to partition the disk. We’ll refer to the whole disk as $DISK (/dev/nvme0n1 in my case), and we need three partitions. The EFI partition, swap³, and the rest of the disk for btrfs to use, which we’ll respecively refer to as "$DISK"p1, "$DISK"p2, and "$DISK"p3.

Btrfs doesn’t natively support encryption, so we’ll be using dm-crypt to transparently encrypt the partition, which would be available at /dev/mapper/enc after running these commands:

cryptsetup --verify-passphrase -v luksFormat "$DISK"p3
cryptsetup open "$DISK"p3 enc

We can then format each partition as needed:

mkfs.vfat -n boot "$DISK"p1
mkswap "$DISK"p2
swapon "$DISK"p2
mkfs.btrfs /dev/mapper/enc

Now we have a btrfs volume, we need to decide on how to structure our subvolumes. We want to split our data into a number of subvolumes to keep track of a few things:

• root: The subvolume for /, which will be cleared on every boot.
• home: The subvolume for /home, which should be backed up.
• nix: The subvolume for /nix, which needs to be persistent but is not worth backing up, as it’s trivial to reconstruct.
• persist: The subvolume for /persist, containing system state which should be persistent across reboots and possibly backed up.
• log: The subvolume for /var/log. I’m not so interested in backing up logs but I want them to be preserved across reboots, so I’m dedicating a subvolume to logs rather than using the persist subvolume.

Somewhat arbitrarily, we’ll go with the “Flat” layout as described in the btrfs wiki, and create our subvolumes accordingly.

mount -t btrfs /dev/mapper/enc /mnt
# We first create the subvolumes outlined above:
btrfs subvolume create /mnt/root
btrfs subvolume create /mnt/home
btrfs subvolume create /mnt/nix
btrfs subvolume create /mnt/persist
btrfs subvolume create /mnt/log
# We then take an empty *readonly* snapshot of the root subvolume,
# which we'll eventually rollback to on every boot.
btrfs subvolume snapshot -r /mnt/root /mnt/root-blank
umount /mnt

Once we’ve created the subvolumes, we mount them with the options that we want. Here, we’re using Zstandard compression along with the noatime option.

mount -o subvol=root,compress=zstd,noatime /dev/mapper/enc /mnt
mkdir /mnt/home
mount -o subvol=home,compress=zstd,noatime /dev/mapper/enc /mnt/home
mkdir /mnt/nix
mount -o subvol=nix,compress=zstd,noatime /dev/mapper/enc /mnt/nix
mkdir /mnt/persist
mount -o subvol=persist,compress=zstd,noatime /dev/mapper/enc /mnt/persist
mkdir -p /mnt/var/log
mount -o subvol=log,compress=zstd,noatime /dev/mapper/enc /mnt/var/log
# don't forget this!
mkdir /mnt/boot
mount "$DISK"p1 /mnt/boot

Then, let NixOS figure out the config.

nixos-generate-config --root /mnt

This should result with /mnt/etc/nixos/hardware-configuration.nix looking something like this:

# Do not modify this file!  It was generated by ‘nixos-generate-config’
# and may be overwritten by future invocations.  Please make changes
# to /etc/nixos/configuration.nix instead.
{ config, lib, pkgs, ... }:
{
  imports =
    [ <nixpkgs/nixos/modules/installer/scan/not-detected.nix>
    ];
  boot.initrd.availableKernelModules = [ "xhci_pci" "nvme" "usb_storage" "sd_mod" "rtsx_pci_sdmmc" ];
  boot.initrd.kernelModules = [ ];
  boot.kernelModules = [ "kvm-intel" ];
  boot.extraModulePackages = [ ];
  fileSystems."/" =
    { device = "/dev/disk/by-uuid/f73c53b7-ae6c-4240-89c3-511ad918edcc";
      fsType = "btrfs";
      options = [ "subvol=root" "compress=zstd" "noatime" ];
    };
  boot.initrd.luks.devices."enc".device = "/dev/disk/by-uuid/050db9bf-0741-4150-8cf8-d6ec12735d4c";
  fileSystems."/home" =
    { device = "/dev/disk/by-uuid/f73c53b7-ae6c-4240-89c3-511ad918edcc";
      fsType = "btrfs";
      options = [ "subvol=home" "compress=zstd" "noatime" ];
    };
  fileSystems."/nix" =
    { device = "/dev/disk/by-uuid/f73c53b7-ae6c-4240-89c3-511ad918edcc";
      fsType = "btrfs";
      options = [ "subvol=nix" "compress=zstd" "noatime" ];
    };
  fileSystems."/var/log" =
    { device = "/dev/disk/by-uuid/f73c53b7-ae6c-4240-89c3-511ad918edcc";
      fsType = "btrfs";
      options = [ "subvol=log" "compress=zstd" "noatime" ];
    };
  fileSystems."/persist" =
    { device = "/dev/disk/by-uuid/f73c53b7-ae6c-4240-89c3-511ad918edcc";
      fsType = "btrfs";
      options = [ "subvol=persist" "compress=zstd" "noatime" ];
    };
  fileSystems."/boot" =
    { device = "/dev/disk/by-uuid/8CE7-3C76";
      fsType = "vfat";
    };
  swapDevices =
    [ { device = "/dev/disk/by-uuid/5b1b6659-14ab-497f-a788-5518c25e7ec8"; }
    ];
  nix.maxJobs = lib.mkDefault 8;
  powerManagement.cpuFreqGovernor = lib.mkDefault "powersave";
  # High-DPI console
  console.font = lib.mkDefault "${pkgs.terminus_font}/share/consolefonts/ter-u28n.psf.gz";
}

Make sure that this is what you want, and adjust options as necessary. Note that in order to correctly persist /var/log, the log subvolume needs to be mounted early enough in the boot process. To do this, we need to add neededForBoot = true; so the entry will look like this:

  fileSystems."/var/log" =
    { device = "/dev/disk/by-uuid/f73c53b7-ae6c-4240-89c3-511ad918edcc";
      fsType = "btrfs";
      options = [ "subvol=log" "compress=zstd" "noatime" ];
      neededForBoot = true;
    };

Although it’s possible to customize /etc/nixos/configuration.nix at this point to set up all the things you need in one fell swoop, I recommend starting out with a reletively minimal config to make sure everything works ok. I went with something like this, with a user called delta:

{ config, pkgs, ... }:
{
  imports =
    [ # Include the results of the hardware scan.
      ./hardware-configuration.nix
    ];
  boot.kernelPackages = pkgs.linuxPackages_latest;
  boot.supportedFilesystems = [ "btrfs" ];
  hardware.enableAllFirmware = true;
  nixpkgs.config.allowUnfree = true;
  # Use the systemd-boot EFI boot loader.
  boot.loader.systemd-boot.enable = true;
  boot.loader.efi.canTouchEfiVariables = true;
  networking.hostName = "apollo"; # Define your hostname.
  networking.networkmanager.enable = true;
  # Enable the X11 windowing system.
  services.xserver.enable = true;
  # Enable the KDE Desktop Environment.
  services.xserver.displayManager.sddm.enable = true;
  services.xserver.desktopManager.plasma5.enable = true;
  # Define a user account. Don't forget to set a password with ‘passwd’.
  users.users.delta = {
    isNormalUser = true;
    extraGroups = [ "wheel" ]; # Enable ‘sudo’ for the user.
  };
  system.stateVersion = "20.03";
}

Take a deep breath.

nixos-install
reboot

If all goes well, we’ll be prompted for the passphrase for $DISK entered earlier, then we’ll see the greeter for the KDE Desktop Environment. Swith to another tty with Ctrl+Alt+F1, login as root, passwd delta to set your password, and switch back to KDE with Ctrl+Alt+F7. Once you’re logged in, you can continue to tweak your NixOS configuration as you want. However, I generally recommend keeping enabled services at a minimum, and setting up opt-in state first.

Now that we’re comfortable in our desktop environment of choice (mine is XMonad), we can move onto the opt-in state setup. First, we need to find out what state exists in the first place. Seeing what has changed since we took the blank snapshot seems like a good way to do this.

Taking a diff between the root subvolume and the root-blank subvolume (in btrfs, snapshots are just subvolumes) can be done with a script based off of the answers to this serverfault question.

#!/usr/bin/env bash
# fs-diff.sh
set -euo pipefail
OLD_TRANSID=$(sudo btrfs subvolume find-new /mnt/root-blank 9999999)
OLD_TRANSID=${OLD_TRANSID#transid marker was }
sudo btrfs subvolume find-new "/mnt/root" "$OLD_TRANSID" |
sed '$d' |
cut -f17- -d' ' |
sort |
uniq |
while read path; do
  path="/$path"
  if [ -L "$path" ]; then
    : # The path is a symbolic link, so is probably handled by NixOS already
  elif [ -d "$path" ]; then
    : # The path is a directory, ignore
  else
    echo "$path"
  fi
done

Then, all it takes to find out which files now exist in the root subvolume is:

sudo mkdir /mnt
sudo mount -o subvol=/ /dev/mapper/enc /mnt
./fs-diff.sh

This may show a surprisingly small list of files, or possible something fairly lengthy, depending on your configuration. We’ll first tackle NetworkManager, so we don’t have to re-type passwords to Wi-Fi access points after every reboot. While grahamc’s original blog post suggests that simply persisting /etc/NetworkManager/system-connections by moving it to somewhere in /persist and creating a symlink is enough, this was not enough to get it to work on my XMonad setup. I ended up with something like this, symlinking a few files in /var/lib/NetworkManager as well.

  environment.etc = {
    "NetworkManager/system-connections".source = "/persist/etc/NetworkManager/system-connections";
  };
  systemd.tmpfiles.rules = [
    "L /var/lib/NetworkManager/secret_key - - - - /persist/var/lib/NetworkManager/secret_key"
    "L /var/lib/NetworkManager/seen-bssids - - - - /persist/var/lib/NetworkManager/seen-bssids"
    "L /var/lib/NetworkManager/timestamps - - - - /persist/var/lib/NetworkManager/timestamps"
  ];

Now you might have noticed that the NixOS configuration itself lives in /etc/nixos/, which will be deleted if left there. After adding a few things, I ended up with a configuration like this.

 environment.etc = {
    nixos.source = "/persist/etc/nixos";
    "NetworkManager/system-connections".source = "/persist/etc/NetworkManager/system-connections";
    adjtime.source = "/persist/etc/adjtime";
    NIXOS.source = "/persist/etc/NIXOS";
    machine-id.source = "/persist/etc/machine-id";
  };
  systemd.tmpfiles.rules = [
    "L /var/lib/NetworkManager/secret_key - - - - /persist/var/lib/NetworkManager/secret_key"
    "L /var/lib/NetworkManager/seen-bssids - - - - /persist/var/lib/NetworkManager/seen-bssids"
    "L /var/lib/NetworkManager/timestamps - - - - /persist/var/lib/NetworkManager/timestamps"
  ];
  security.sudo.extraConfig = ''
    # rollback results in sudo lectures after each reboot
    Defaults lecture = never
  '';

Rolling back the root subvolume is a little bit involved when compared to zfs, but can be achieved with this config.

  # Note `lib.mkBefore` is used instead of `lib.mkAfter` here.
  boot.initrd.postDeviceCommands = pkgs.lib.mkBefore ''
    mkdir -p /mnt
    # We first mount the btrfs root to /mnt
    # so we can manipulate btrfs subvolumes.
    mount -o subvol=/ /dev/mapper/enc /mnt
    # While we're tempted to just delete /root and create
    # a new snapshot from /root-blank, /root is already
    # populated at this point with a number of subvolumes,
    # which makes `btrfs subvolume delete` fail.
    # So, we remove them first.
    #
    # /root contains subvolumes:
    # - /root/var/lib/portables
    # - /root/var/lib/machines
    #
    # I suspect these are related to systemd-nspawn, but
    # since I don't use it I'm not 100% sure.
    # Anyhow, deleting these subvolumes hasn't resulted
    # in any issues so far, except for fairly
    # benign-looking errors from systemd-tmpfiles.
    btrfs subvolume list -o /mnt/root |
    cut -f9 -d' ' |
    while read subvolume; do
      echo "deleting /$subvolume subvolume..."
      btrfs subvolume delete "/mnt/$subvolume"
    done &&
    echo "deleting /root subvolume..." &&
    btrfs subvolume delete /mnt/root
    echo "restoring blank /root subvolume..."
    btrfs subvolume snapshot /mnt/root-blank /mnt/root
    # Once we're done rolling back to a blank snapshot,
    # we can unmount /mnt and continue on the boot process.
    umount /mnt
  '';

While NixOS will take care of creating the specified symlinks, we need to move the relevant file and directories to where the symlinks are pointing at after running sudo nixos-rebuild boot and before rebooting.

sudo nixos-rebuild boot
sudo mkdir -p /persist/etc/NetworkManager
sudo cp -r {,/persist}/etc/NetworkManager/system-connections
sudo mkdir -p /persist/var/lib/NetworkManager
sudo cp /var/lib/NetworkManager/{secret_key,seen-bssids,timestamps} /persist/var/lib/NetworkManager/
sudo cp {,/persist}/etc/nixos
sudo cp {,/persist}/etc/adjtime
sudo cp {,/persist}/etc/NIXOS

Before rebooting, make sure that your user credentials are appropriately handled. Be especially careful⁴ when setting users.mutableUsers to false and using users.extraUsers.<name?>.passwordFile, as these settings are some of the few in NixOS which can lock you out across NixOS configurations and require non-trivial recovery work or a reinstall. If you want declerative user management, I recommend using users.extraUsers.<name?>.hashedPasswords, but this has it’s own downsides as well.⁵

Take another deep breath.

reboot

If something goes wrong and /mnt/root isn’t deleted, btrfs subvolume snapshot /mnt/root-blank /mnt/root will just create a snapshot under /mnt/root, so a quick hack to check if rolling back failed without consulting journalctl -b is to see if /mnt/root/root-blank exists.⁶

As much as Nix and NixOS are attractive for everyday use, sometimes the time it takes to get some language or package running on NixOS just doesn’t seem worth it. That’s when container runtimes like Docker and LXD can help. These tools can act as an escape hatch to get some software working quickly on your machine.

Here, we’ll go through the workflow for getting NixOS services to work with opt-in state, with Docker and LXD as examples.

First, let’s get Docker and LXD running to inspect what kind of state they have. Thanks to NixOS, this is a just a few lines of configuration.

  virtualisation = {
     docker.enable = true;
    lxd = {
      enable = true;
      recommendedSysctlSettings = true;
    };
  };

sudo nixos-rebuild switch

This will install, set up, and start both Docker and LXD on our machine. With fs-diff.sh we can see a few relevant files and directories show up.

/etc/docker/key.json
...
/var/lib/docker/...
/var/lib/lxd/...

Some quick googling tells us that /etc/docker/key.json is generated on every boot, so it seems like we don’t need to keep this around. On the other hand, /var/lib/docker and /var/lib/lxd seem important, so let’s adjust our config accordingly.

  systemd.tmpfiles.rules = [
    "L /var/lib/NetworkManager/secret_key - - - - /persist/var/lib/NetworkManager/secret_key"
    "L /var/lib/NetworkManager/seen-bssids - - - - /persist/var/lib/NetworkManager/seen-bssids"
    "L /var/lib/NetworkManager/timestamps - - - - /persist/var/lib/NetworkManager/timestamps"
    "L /var/lib/lxd - - - - /persist/var/lib/lxd"
    "L /var/lib/docker - - - - /persist/var/lib/docker"
  ];

Now, stop the two services and copy over the directories.

sudo mkdir -p /persist/var/lib/
sudo systemctl stop lxd
sudo cp -r {,/persist}/var/lib/lxd
sudo systemctl stop docker
sudo cp -r {,/persist}/var/lib/docker
sudo nixos-rebuild boot
reboot

If all goes well, running the fs-diff.sh after reboot shouldn’t show persisted directories /var/lib/lxd and /var/lib/docker since they should be symlinks which are created during the boot process.

Docker should work without any problems at this point, but we LXD needs some additional configuration. LXD requires a storage pool to operate, so we create a subvolume for LXD, and mount it in /persist.

sudo mount -o subvol=/ /mnt
sudo btrfs subvolume create /mnt/lxd
sudo umount /mnt
sudo mkdir /persist/lxd
sudo mount -o subvol=lxd /dev/mapper/enc /persist/lxd

Once the subvolume is ready, we run lxd init and answer the questions in the following manner.

$ lxd init
Would you like to use LXD clustering? (yes/no) [default=no]: no
Do you want to configure a new storage pool? (yes/no) [default=yes]:
Name of the new storage pool [default=default]:
Name of the storage backend to use (btrfs, dir, lvm) [default=btrfs]:
Would you like to create a new btrfs subvolume under /var/lib/lxd? (yes/no) [default=yes]: no
Create a new BTRFS pool? (yes/no) [default=yes]: no
Name of the existing BTRFS pool or dataset: /persist/lxd
Would you like to connect to a MAAS server? (yes/no) [default=no]:
Would you like to create a new local network bridge? (yes/no) [default=yes]:
What should the new bridge be called? [default=lxdbr0]:
What IPv4 address should be used? (CIDR subnet notation, “auto” or “none”) [default=auto]:
What IPv6 address should be used? (CIDR subnet notation, “auto” or “none”) [default=auto]:
Would you like LXD to be available over the network? (yes/no) [default=no]:
Would you like stale cached images to be updated automatically? (yes/no) [default=yes]
Would you like a YAML "lxd init" preseed to be printed? (yes/no) [default=no]:

Remember to add the relevant information to /etc/nixos/hardware-configuration.nix so NixOS will mount the subvolume where LXD expects (i.e. /persist/lxd).

  fileSystems."/persist/lxd" =
    { device = "/dev/disk/by-uuid/f73c53b7-ae6c-4240-89c3-511ad918edcc";
      fsType = "btrfs";
      options = [ "subvol=lxd" "compress=zstd" "noatime" ];
    };

EDIT 2020-01-26: Added persistence for /etc/machine-id, which fixes an issue where journalctl fails to find logs from past boots, among various others. Thanks j-hui for pointing this out!

Thanks to cannorin and __pandaman64__ for comments and suggestions.