Die rechtsextreme AfD hat am letzten Wochenenden ihren Spitzenkandidaten für die Europawahl gekürt: Es wurde Maximilian Krah. Auch wenn er schon 2019 über AfD-Listenplatz 3 ins Europaparlament einzog und dort seitdem in aller Regel einfach gegen alles abstimmt, ist er zuletzt eher unter dem Radar geblieben. Dabei ist sogar vom Verfassungsschutz bestätigt worden, dass er verfassungsfeindliche Positionen vertritt. Haben wir uns schon so sehr daran gewöhnt, demokratische Institutionen von glasklaren Rechtsextremen aushöhlen zu lassen, dass das kein Skandal mehr darstellt? Oder ist es der breiten Öffentlichkeit vielleicht wirklich noch nicht bekannt, wie gefährlich Krah ist?
Wir haben mal nachgeschaut und die wichtigsten Dinge zusammengefasst, die Du über Krah wissen solltest. Spoiler: Selbst wenn unser kleinster gemeinsamer Nenner nur ist, dass wir die Taliban ablehnen – yep, dann wird Dir Krah nicht gefallen.
Ja, das ist tatsächlich passiert. Krah war zu Gast im Podcast beim rechtsextremen Chefideologen Götz Kubitschek (dazu später mehr): In der Diskussion solidarisiert er sich tatsächlich mit den Taliban, die im Sommer 2021 die Macht übernommen hatten, laut Krah, nachdem die US-Botschaft in Kabul auf Twitter über den Pridemonth informiert habe. Dass die Machtübernahme im Zusammenhang mit dem Pridemonth stünde, ist natürlich eine unsinnige Behauptung und zeugt von mangelndem Hintergrundwissen über den Konflikt. Auf Nachfrage wird er es sicherlich als „Ironie“ oder ähnliches entschuldigen, doch die unterschwellige Botschaft kommt bei den Fans an. Der Einmarsch der Taliban im Kabul sei „die einzig richtige Antwort auf den Pridemonth“, sagte Krah. Im Zweifel lieber die Taliban als queere Menschen – das ist die AfD. Selbst eine von al-Quaida unterwanderte Diktatur klingt für den AfD-Spitzenkandidaten Krah also offenbar besser als das deutsche Grundgesetz.
Am Rande bemerkt: Eine Zusammenarbeit von Rechtsextremen und islamistischen Extremisten ist gar nicht so abwegig, wie es vielleicht auf den ersten Blick scheint. Sie haben sehr viel gemeinsam. Belltower News haben eine ausführliche Recherche zu diesem Thema veröffentlicht.
Maximilian Krah gibt sich rechts – und damit meint er nicht konservativ oder patriotisch, nicht mal „nur“ rechtsradikal. Der AfD-Spitzenkandidat sucht sich zielgerichtet diejenigen Manipulationsportale, die am weitesten rechts außen stehen. So ist er beispielsweise gern Gast bei Götz Kubitschek, dem rechtsextremen Verleger, der so etwas wie die graue Eminenz des modernen deutschen Rechtsextremismus ist. Wir haben schon 2019 über ihn geschrieben.
Krah trat beispielsweise in Kubitscheks Podcast auf, zuletzt am 24.06. Er war Mitte Juli auch für ein rechtes „Sommerfest“ bei Kubitschek in Thüringen am Start. Natürlich ist dem AfD-Spitzenkandidaten Krah dabei bewusst, dass Kubitscheks „Institut für Staatspolitik“ seit 2021 als gesichert rechtsextrem und verfassungsfeindlich eingestuft ist. Krah geht dort nicht trotzdem hin – sondern gerade deswegen. Da gehört er hin.
Mit dem Antisemiten und Rassisten Steve Bannon, ehemaliger Trump-Berater, hat sich Maximilian Krah auch im vergangenen Jahr laut seiner eigenen Facebook-Seite getroffen:
Screenshot Facebook
Auch für Éric Zemmour hat Maximilian Krah Sympathien – dem französischen Rechtsextremen, der dann selbst der AfD zu rechts war. Die AfD-Fraktion im Europaparlament suspendierte Krah für sechs Monate, nachdem er Werbung für eine Zemmour nahestehende Politikerin gemacht hatte. Ja, Krah war sogar einst zu rechtsextrem für die AfD. Heute nicht mehr, du weißt Bescheid.
Auch Antisemitismus ist im Repertoire des AfD-Spitzenkandidaten. Das ist wenig überraschend, hat doch eine Analyse des American Jewish Committee schon 2021 ergeben, dass Antisemitismus Teil des „programmatischen Kerns“ der AfD sei. Deswegen hatte Krah in seiner bisherigen Amtszeit als Europa-Abgeordneter auch absolut kein Problem damit, Guillaume Pradoura in seinem Büro anzustellen. Der ist Mitbegründer der „neurechten“ und rechtsextremen Identitären Bewegung und hatte zuvor seinen Job verloren, weil er es offenbar sehr witzig fand, eine antisemitische Karikatur aus der NS-Zeit nachzuahmen. Yep, das ist der Punkt in der Geschichte, an dem wir mittlerweile angekommen sind.
Der Verfassungsschutz dokumentierte auch weitere explizit rassistische Aussagen Krahs. Die umfassen zum Beispiel den rechtsextremen talking point der „orientalischen Landnahme„, gepaart mit dem faktenfernen, pauschalen Vorwurf des kollektiven sexuellen Missbrauchs. Und wie so ziemlich alle AfD-Politiker:innen, die etwas auf sich halten, denkt er sich natürlich auch immer wieder irgendwelche angeblichen „Fakten“ aus, um seine rassistische Hetze zu untermauern, die sich regelmäßig als komplett falsch herausstellen. Auch wir haben schon seine dreisten Lügen gefaktencheckt:
Übrigens: Wenn Du dachtest, dass Rückgriffe auf die NS-Zeiten eine Seltenheit oder Zufall sind, dann müssen wir Dich leider enttäuschen. Noch ein Beispiel ist Krahs Forderung, Menschen in Schützenvereinen Polizeiuniformen in den Schrank zu hängen und sie wie eine mobile Einsatztruppe bereitzuhalten, falls die AfD mal wieder wo für Ordnung sorgen (sprich: Menschen jagen) möchte. Falls das für Dich verdächtig nach der NS-Hilfspolizei aus der Frühphase des NS-Reiches klingt, dann bist Du mit dieser Einschätzung nicht allein.
In einem Vortrag über das „Volk“, die „Volkssouveränität“ und die Verfassung machte er 2019 ein überraschend offenes Geständnis. Dort erklärte er die Dogmatik der AfD. Und hat nachvollziehbar erklärt, dass seine Partei im Konflikt mit der Verfassung der Bundesrepublik Deutschland steht. Das habe zu tun mit der NS-„Blut und Boden“-Ideologie der AfD, versteckt hinter Euphemismen von „Volkssouveränität“ und dem Ethnopluralismus.
Krah bezieht sich ganz offen auf den NS-Juristen Carl Schmitt, der 1941 das Buch „Völkerrechtliche Großraumordnung mit Interventionsverbot für raumfremde Mächte“ veröffentlicht hat. Das war nicht das erste Mal, dass sich Krah auf den Nazi bezieht.
Krah erklärt, dass es neben dem deutschen „Staatsvolk“ (die Bevölkerung Deutschland nach dem Grundgesetz) ein deutsches ETHNISCHES VOLK gäbe, das als „wahrer Souverän“ das Recht habe, sich neu zu verfassen. Das heißt also, unter Berufung auf buchstäblicher Nazi-Propaganda möchte die AfD ein Konzept des „Volkes“ definieren, das dem Grundgesetz widerspricht.
Mit dieser auf Nazi-Ideologie basierender Definition des „ethnischen deutschen Volkes“, das wie erwähnt dem Grundgesetz widerspricht, erklärt er ganz offen, dass diese Gruppe das Recht habe, das Grundgesetz außer Kraft zu setzen. Die Stelle erklärt er hier ganz genau. Der Jurist Peter Nagel stellt fest: „Nach Ansicht der AfD steht der ethnische Deutsche also über dem deutschen Staatsbürger des Grundgesetzes. Er kann ihn entrechten.“ Kurz: Krah gibt offen zu, dass er eine verfassungsfeindliche Ideologie hat, die sich offenbar auch auf die NS-Ideologie stützt. Muss man mehr dazu sagen?
Der neue AfD-Vorsitzende ist Sinnbild einer absurden kognitiven Dissonanz, die in der rechtsextremen Partei allgegenwärtig ist. Einerseits gibt man sich „patriotisch“, mit dem Deutschlandfähnchen stolz am Revers. Gleichzeitig wollen Krah und die anderen führenden Köpfe des völkischen „Flügels“ außenpolitisch so ziemlich ALLES tun, um Deutschland von Diktaturen und Autokratien abhängig zu machen und zu schwächen. Die „Stimme Moskaus“ bei der Arbeit. Aber ehrlich: Hören die sich eigentlich selbst noch zu?
Zuerst einmal ist da das Thema EU-Austritt. Ein Parteitag in Riesa im Juni vergangenen Jahres war deswegen sogar abgebrochen worden. Der faschistische Führer des völkischen Flügels Björn Höcke wollte raus aus der EU, näher ran an Russland, die Delegierten gingen letztlich im Streit auseinander. Auch Maximilian Krah ist bekannt dafür, dass er enge Verbindungen zur russischen Elite hat. Krahs außenpolitische Richtung ist also klar. Weg von dem europäischen Binnenmarkt, mit dem das Exportland Deutschland mehr als 50 % seiner Exporte bestreitet, und hin in die Abhängigkeit von einem autokratischen Regime, das einen völkerrechtswidrigen Angriffskrieg führt im verzweifelten Versuch, die Länder Europas nach seiner Pfeife tanzen zu lassen. Vielleicht ist Krah ja wirklich stolzer Verteidiger nationaler Interessen – nur vielleicht nicht der deutschen….
Während Russland in den letzten Monaten aus gegebenem Anlass ein heiß diskutiertes Thema war, blieben die Verbindungen der AfD zur China bislang noch etwas unter dem Radar. Tatsächlich gönnt sich aber auch der Europa-Spitzenkandidat Krah beispielsweise mal eine Reise nach China auf Kosten des staatlichen Ölkonzerns China Petroleum und von Huawei. Entsprechend fleißig war er dann auch darin, sich der Propaganda der Kommunistischen Partei Chinas zur Verfügung zu stellen.
Übrigens: Wenn Du mehr über die weitreichenden Verbindungen zwischen den Autokratien China und Russland sowie den Einfluss Chinas auf den Krieg Russlands in der Ukraine wissen willst, können wir Dir ein nagelneues VVP-Spezial-Video ans Herz legen:
Nicht nur ist der AfD-„Patriotismus“ erlogen und ein rhetorischer Trick, um Stimmen zu erhaschen. Die rechtsextreme Partei – und ganz besonders ihr neuer Europa-Spitzenkandidat Krah – ist sogar genau das Gegenteil einer patriotischen Partei. Sie möchte die Souveränität Deutschland untergraben und den Autokratien Russland und China unterstellen. Noch mal daran erinnern: Die AfD freut sich, wenn es Deutschland schlecht geht:
In der Demokratie sollte es immer ein Skandal sein, wenn ein Rechtsextremist wie Maximilian Krah Macht erhält. Denn an ihm zeigt sich, dass die AfD längst nicht mehr da ist, wo manche Medien sie immer noch verorten. Auch wenn es sicherlich gut gemeint ist, schreiben Schlagzeilen wie „AfD: Rechtspopulismus auf dem Vormarsch in Deutschland“ der Realität hinterher. Die AfD ist schon lange beim offenen Rechtsextremismus angekommen.
Das Feld „Rechtspopulismus“ beginnt bereits eher die Merz-Union zu übernehmen, aber deren rechter Kulturkampf ist eine andere Geschichte. Die AfD ist eine Partei, in der immer mehr ein ganz bestimmter, rechtsextremer Typ Politiker die Macht übernimmt. Einer wie Maximilian Krah, der seinen Extremismus, Antisemitismus, seinen offenen Rassismus und seine Ablehnung des Grundgesetzes und damit der Demokratie in Deutschland offen zur Schau stellt. Wir sind über den Punkt hinweg, an dem AfD-Politiker:innen noch „mit der Maus ausgerutscht“ sind, wenn sie menschenverachtende Dinge gesagt und gepostet haben.
Mittlerweile sind wir an dem Punkt, wo die AfD sich ganz offen extremistisch, rassistisch und antisemitisch zeigt. Sie machen keinen Hehl mehr aus ihren verfassungsfeindlichen und zerstörerischen Absichten. Und sie müssen es offenbar auch nicht, da rechter Kulturkampf gerade mainstream geworden ist. Alle Demokrat:innen müssen die Machtübernahme dieser Partei nun verhindern. Unsere wehrhafte Demokratie muss die AfD mit allen ihr zur Verfügung stehenden Mitteln bekämpfen. Wir sind nicht mehr in der komfortablen Lage, „Wehret den Anfängen!“ zu sagen. Wir sind mittendrin im nächsten Versuch einer faschistischen Machtübernahme in Deutschland. Das sollte allen bewusst werden, denen es an der Demokratie gelegen ist. Die Uhr tickt: Nächsten Herbst sind Landtagswahlen in Brandenburg, Thüringen und Sachsen. Dazu brauchen wir dringend auch Konservative, die die Demokratie verteidigen – und nicht der AfD den Weg bereiten.
If you reach this page, you may be interested into this new category of Linux distributions labeled "immutable".
In this category, one can find by age (oldest → youngest) NixOS, Guix, Endless OS, Fedora Silverblue, OpenSUSE MicroOS, Vanilla OS and many new to come.
I will give examples of immutability implementation, then detail my thoughts about immutability, and why I think this naming can be misleading. I spent a few months running all of those distributions on my main computers (NAS, Gaming, laptop, workstation) to be able to write this text.
The word immutability itself refers to an object that can't change.
However, when it comes to an immutable operating system, the definition immediately become vague. What would be an operating system that can't change? What would you be supposed to do with it?
We could say that a Linux LIVE-CD is immutable, because every time you boot it, you get the exact same programs running, and you can't change anything as the disk media is read only. But while the LIVE-CD is running, you can make changes to it, you can create files and directories, install packages, it's not stuck in an immutable state.
Unfortunately, this example was nice but the immutability approach by those Linux distribution is totally different, so we need to think a bit further.
There are three common principles in these systems:
system upgrades aren't done on the live system
packages changes are applied on the next boot
you can roll back a change
Depending on the implementation, a system may offer more features. But this list is what a Linux distribution should have to be labelled "immutable" at the moment.
Now we found what are the minimum requirements to be called immutable, let's go through each implementation, by their order of appearance.
In this section, I'm mixing NixOS and Guix as they both rely on the same implementation. NixOS is based on Nix (first appearance in 2003), which has been forked into early 2010s into the Guix package manager to be 100% libre, which gave birth to an eponym operating system also 100% free.
These two systems are really different than a traditional Unix like system we are used to, and immutability is a main principle. To make it quick, they are based on their package manager (being Nix or Guix) that contains every package or built file into a special read-only directory (where only the package manager can write) where each package has its own unique entry, and the operating system itself is a byproduct of the package manager.
What does that imply? If the operating system is built, this is because it's made of source code, you literally describe what you want your system to be in a declarative way. You have to list users, their shells, installed packages, running services and their configurations, partitions to mount with which options etc... Fortunately, it's made a lot easier by the use of modules which provide sane defaults, so if you create a user, you don't have to specify its UID, GID, shell, home etc...
So, as the system is built and stored in the special read-only directory, all your system is derived from that (using symbolic links), so all the files handled by the package manager are read-only. A concrete example is that /etc/fstab or /bin/sh ARE read-only, if you want to make a change in those, you have to do it through the package manager.
I'm not going into details, because this store based package manager is really different than everything else but:
you can switch between two configurations on the fly as it's just a symlink dance to go from a configuration to another
you can select your configuration at boot time, so you can roll back to a previous version if something is wrong
you can't make change to a package file or system file as they are read only
the mount points except the special store directory are all mutable, so you can write changes in /home or /etc or /var etc... You can remove the system symlinks by a modified version, but you can't modify the symlink source itself.
This is the immutability as seen through the Nix lens.
I've spent a few years running NixOS systems, this is really a blast for me, and the best "immutable" implementation around, but unfortunately it's too different, so its adoption rate is very low, despite all the benefits.
While this one is not the oldest immutable OS around, it's the first one to be released for the average user, while NixOS and Guix are older but for a niche user category. The company behind Endless OS is trying to offer a solid and reliable system, free and open source, that can works without Internet, to be used in countries with a low Internet / powergrid coverage. They even provide a version with "offline internet included" containing Wikipedia dumps, class lessons and many things to make a computer useful while offline (I love their work).
Endless OS is based on Debian, but uses the OSTree tool to make it immutable. OSTree allows you to manage a core system image, and add layers on top of it, think of packages as layers. But it can also prepare a new system image for the next boot.
With OSTree, you can apply package changes in a new version of the system that will be available at next boot, and revert to a previous version at boot time.
The partitions are mounted writable, except for /usr, the land of packages handled by OSTree, which is mounted read-only. There are no rollbacks possible for /etc.
Programs meant to be for the user (not the packages to be used by the system like grub, X display or drivers) are installed from Flatpak (which also uses OSTree, but unrelated to the system), this avoids the need to reboot each time you install a new package.
My experience with Endless OS is mixed, it is an excellent and solid operating system, it's working well, never failed, but I'm just not the target audience. They provide a modified GNOME desktop that looks like a smartphone menu, because this is what most non-tech users are comfortable with (but I hate it). And installing DevOps tools isn't practical but not impossible, so I keep Endless OS for my multimedia netbook and I really enjoy it.
This linux distribution is the long descendant of Project Atomic, an old initiative to make Fedora / CentOS/ RHEL immutable. It's now part of the Fedora releases along with Fedora Workstation.
Fedora Silverblue is also using OSTree, but with a twist. It's using rpm-OSTree, a tool built on top of OSTree to let your RPM packages apply the changes through OSTree.
The system consists of a single core image for the release, let's say fedora-38, and for each package installed, a new layer is added on top of the core. At anytime, you can list all the layers to know what packages have been installed on top of the core, if you remove a package, the whole stack is generated again (which is terribly SLOW) without the package, there is absolutely no leftover after a package removal.
On boot, you can choose an older version of the system, in case something broke after an upgrade. If you install a package, you need to reboot to have it available as the change isn't applied on the current booted system, however rpm-OSTree received a nice upgrade, you can temporarily merge the changes of the next boot into the live system (using a tmpfs overlay) to use the changes.
The mountpount management is a bit different, everything is read-only except /etc/, /root and /var, but your home directory is by default in /var/home which sometimes breaks expectations. There are no rollbacks possible for /etc.
As installing a new package is slow due to rpm-OSTree and requires a reboot to be fully usable (the live change back port store the extra changes in memory), they recommend to use Flatpak for programs, or toolbox, some kind of wrapper that create a rootless fedora container where you can install packages and use it in your terminal. toolbox is meant to provide development libraries or tool you wouldn't have in Flatpak, but that you wouldn't want to install in your base Fedora system.
My experience with Fedora Silverblue has been quite good, it's stable, the updates are smooth even if they are slow. toolbox was working fine despite I don't find this practical.
This spin of OpenSUSE Tumbleweed (rolling-release OpenSUSE) features immutability, but with its own implementation. The idea of MicroOS is really simple, the whole system except a few directories like /home or /var lives on a btrfs snapshot, if you want to make a change to the system, the current snapshot is forked into a new snapshot, and the changes are applied there, ready for the next boot.
What's interesting here is that /etc IS part of the snapshots, and can be roll backed, which wasn't possible in the OSTree based systems. It's also possible to make changes to any file of the file system (in a new snapshot, not the live one) using a shell, which can be very practical for injecting files to solve a driver issue. The downside it's not guaranteed that your system is "pure" if you start making changes, because they won't be tracked, the snapshots are just numbered, and you don't know what changes were made in each of them.
Changes must be done through the command transactional-update which do all the snapshot work for you, and you could either manipulate package by adding/removing a package, or just start a shell in the new snapshot to make all the changes you want. I said /etc is part of the snapshots, it's true, but it's never read-only, so you could make a change live in /etc, then create a new snapshot, the change would be immediately inherited. This can create troubles if you roll back to a previous state after an upgrade if you also made changes to /etc just before.
The default approach of MicroOS is disturbing at first, a reboot is planned every day after a system update, this is because it's a rolling-release system and there are updates every day, and you won't benefit from them until you reboot. While you can disable this automatic reboot, it makes sense to use the newest packages anyway, so it's something to consider if you plan to use MicroOS.
There is currently no way to apply the changes into the live system (like Silverblue is offering), it's still experimental, but I'm confident this will be doable soon. As such, it's recommended to use distrobox to use rootless containers of various distributions to install your favorite tools for your users, instead of using the base system packages. I don't really like this because this adds maintenance, and I often had issues of distrobox refusing to start a container after a reboot, I had to destroy and recreate it entirely to solve.
My experience with OpenSUSE MicroOS has been wonderful, it's in dual-boot with OpenBSD on my main laptop, it's my Linux Gaming OS, and it's also my NAS operating system, so I don't have to care about updates. I like that the snapshots system doesn't restrict me, while OSTree systems just doesn't allow you to make changes without installing a package.
Finally, the really new (but mature enough to be usable) system in the immutable family is Vanilla OS based on Ubuntu (but soon on Debian), using ABroot for immutability. With Vanilla OS, we have another implementation that really differs from what we saw above.
ABroot named is well thought, the idea is to have a root partition A, another root partition B, and a partition for persistent data like /home or /var.
Here is the boot dance done by ABroot:
first boot is done on A, it's mounted in read-only
changes to the system like new packages or file changes in /etc are done on B (and can be applied live using a tmpfs overlay)
upon reboot, if previous boot was A, you boot on B, then if the boot is successful, ABroot scan for all the changes between A and B, and apply all the changes from B to A
when you are using your system, until you make a change, A and B are always identical
This implementation has downsides, you can only roll back a change until you boot on the new version, then the changes are also applied on the previous boot, and you can't roll back. This implementation mostly protects you from a failing upgrade, or if you made changes and tried them live, but you prefer to rollback.
Vanilla OS features the package manager apx, written by distrobox author. That's for sure an interesting piece of software, allowing your non-root user to install packages from many distributions (arch linux, fedora, ubuntu, nix, etc...) and integrates them into the system as if they were installed locally. I suppose it's some kind of layer on top of distrobox.
My experience wasn't very good, I didn't find ABroot to be really useful, and the version 22.10 I tried was using an old Ubuntu LTS release which didn't make my gaming computer really happy. The overall state of Vanilla OS, ABroot and apx is that they are young, I think it can become a great distribution, but it still has some rough edges.
I don't want to go much into details, but here is the short version: you can use Alpine Linux installer as a base system to boot from, and create tarballs of "saved configurations" that are automatically applied upon boot (it's just tarred directories and some automation to install packages). At every boot, everything is untarred again, and packages are installed again (you should use an apk cache directory), everything in live memory, fully writable.
What does this achieve? You always start from a clean state, changes are applied on top of it at every boot, you can roll back the changes and start fresh again. Immutability as we defined above here isn't achieved because changes are applied on the base system, but it's quite close to fulfill (my own) requirements.
I've been using it a few days only, not as my main system, and it requires a very good understanding of what you are doing because the system is fully in memory, and you need to take care about what you want to save/restore, which can create big archives.
On top of that, it's poorly documented.
Now I gave some details about all the major immutable systems (Linux based) around, I think it's time to list the real pros and cons I found from my experimentation.
configuration management tool (ansible, salt, puppet etc..) integrate VERY badly, they received updates to know how to apply package changes, but you will mostly hit walls if you want to manage those like regular systems.
having to reboot after a change is annoying (except for NixOS and Guix which don't require rebooting for each change).
OSTree based systems aren't flexible, my netbook requires some extra files in alsa directories to get sound (fortunately Endless OS have them!), you just can't add the files without making a package deploying them.
blind rollbacks, it's hard to figure what was done in each version of the system, so when you roll back it's hard to figure what you are doing exactly.
it can be hard to install programs like Nix/Guix which require a directory at the root of the file system, or install non-packaged software system-wide (this is often bad practice, but sometimes a necessary evil).
immutability is a lie, many parts of the systems are mutable, although I don't know how to describe this family with a different word (transactional something?).
immutable doesn't imply stateless.
NixOS / Guix are doing it right in my opinion, you can track your whole system through a reliable package manager, and you can use a version control system on the sources, it has the right philosophy from the ground up.
immutability is often associated with security benefits, I don't understand why. If someone obtains root access on your system, they can still manipulate the live system and have fun with the /boot partition, nothing prevent them to install a backdoor for the next boot.
immutability requires discipline and maintenance, because you have to care about the versioning, you have extra programs like apx / distrobox / devbox that must be updated in parallel of the system (while this is all integrated into NixOS/Guix).
Immutable operating systems are making the news in our small community of open source systems, but behind this word lies many implementations with different use cases. The word immutable certainly creates expectations from users, but it's really nothing more than transactional updates for your operating system, and I'm happy we can have this feature now.
But transactional updates aren't new, I think it started a with Solaris and ZFS allowing you to select a system snapshot at boot time, then I'm quite sure FreeBSD implemented this a decade ago, and it turns out that on any linux distribution with regular btrfs snapshots you could select a snapshot at boot time.
In the end, what's REALLY new is the ability to apply a transactional change on a non-live environment, integrates this into the bootloader, and give the user the tooling to handle this easily.
I recommend reading the blog post "“Immutable” → reprovisionable, anti-hysteresis" by Colin Walters.
Your SSH keys are secrets (the private part). So the best way to store them is to handle them like your other secrets (passwords, API keys, etc.) in your personal secret manager.
I assume that you already have one because it’s a very basic security best practice, applicable for your staff and organization secrets. That allows you to use a complex, unique and random password every time you need one.
There are many options to manage your secrets on your favorite desktop. You could use Dashlane, 1Password, or Bitwarden. Personally, I have been using KeePassXC for many years, and I’m very happy with it because there is an option only available (as far as I know) on it: manage your SSH keys with it. We will cover this part in a few words.
Even if it’s not the purpose of this article, a few things I really like with KeePassXC over other solutions:
KeePassXC is Open Source and well maintained
Support for MacOS X (even M1) and the ability to unlock the database with my fingerprint
I store my secret database on a Google Drive that allows me to access my secrets everywhere, even on my phone with Keepass2Android
Integration with Firefox works very well and allows me to quickly log in to most of the websites I use daily (it’s not 100% but close too)
Everything is encrypted inside the database, even other information not secret (login, Website URL, my SSH keys, added files, etc.)
Versioning of secret is available
KeepassXC
Why use KeePassXC to manage your SSH keys on your SSH agent?
Since you can protect your SSH keys with a passphrase, why is it better to store them inside KeePassXC?
If your SSH keys are protected with a passphrase, you have to enter your passphrase every time you need to use them. When you are doing this a hundred times in your daily routines, it’s very painful, and you are losing a lot of time (who said it’s toil?). Also, every key should have a different password. So you store your passwords in a password manager, like KeePassXC, and copy and paste your secrets all day long.
To avoid this nightmare, some people store their keys AND the passphrase within SSH Agent. Since it’s a good and widely used solution, I don’t want to have my keys loaded into SSH agent.
Here comes this awesome feature from KeePassXC that allows you to load and unload your keys into your SSH agent when you unlock or lock your secret database. In addition, because you can handle how the application is keeping your database open, you can also control how you maintain your keys available in SSH Agent. It means, for example, that when you lock your session, your keys are unloaded from SSH Agent and will only be available the next time you unlock your database. And ALL the keys will be loaded/unloaded this way. So, with only one action, all my keys are available (or unavailable). Combined with fingerprint unlock, it’s never an issue, and I save a lot of time every day for more interesting things than entering a passphrase for my SSH keys in addition to keeping my keys secret.
Installing KeePassXC
If you don’t already have KeePassXC on your desktop, here is a quick installation guide. If you already have it, go to the next part.
To install KeepassXC, go to the download page and choose the appropriate version regarding your operating system.
Now the first thing to do is to create your secret database. Click on “Create new database” and fill the form.
Then, click on “Continue” to choose the encryption parameters for your secret database.
Default are goods, but you can customize the encryption algorithm in the “Advanced Settings”. For example, you can use ChaChat20 256-bit for encryption.
Keep in mind that all encryption algorithms available are very good choices. If you are not an expert in the mathematical differences between them, all options are good enough regarding today's security standards.
Once you have chosen the settings that meet your requirements, click on continue and enter your secret database password.
⚠ This is one of the most important passwords you have to create and memorize. With a secret manager, you have only one password to remind. So decide carefully regarding current best practices.
Here are a few guidelines :
Length is more significant than complexity because length is a major factor to increase the time to crack the password. The NIST recommendation is 8 characters minimum, but I recommend for your master password at least 16 characters (remember that you have only one password to remind now!)
Use both alphabet letters, uppercase and lowercase, and numbers
Add one or more special characters, like ! - ; .
A common usage is to use a phrase that you can easily memorize, for example: The Cake Is a Lie !42
Click on “Done” to finally create your password database.
Store your SSH keys and load them into your SSH agent
Since you have your first database, or if you already use KeePassXC, it’s time to load your SSH keys into your database and allow KeePassXC to load and unload them into your SSH agent.
Assuming we have an ed25519 key pair :
ssh-keygen -t ed25519 -C "bastion production"
Generating public/private ed25519 key pair.
Enter file in which to save the key (/Users/padok/.ssh/id_ed25519):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /Users/padok/.ssh/id_ed25519
Your public key has been saved in /Users/padok/.ssh/id_ed25519.pub
The key fingerprint is:
SHA256:pPcl+pI2TTkzIL6psB/13wwxQLOA9jZp1+A/E+sHI3Q bastion production
The key's randomart image is:
+--[ED25519 256]--+
| .. o |
| o o.o |
| . . o+o |
| Bo=.E |
| +o=S+++. |
| ..o.o&= |
| . . oo*oO |
| o .o =oo+. |
| ..o. . oo.o |
+----[SHA256]-----+
Now, open your secret database and add a new entry.
Once you have named your new secret (“bastion production ssh key” in this example), go to the Advanced menu on the left.
Click on “Add” to add your private SSH key.
Your file will appear in the “Attachments” part.
One important thing here is that your private key file is now stored inside your database! That means you can safely delete it from your computer!
Next, go to the “SSH Agent” menu on the left.
Select the two first options:
Add key to agent when the database is opened/unlocked
Remove key from agent when the database is closed/locked
Depending on your usage and your security needs, you could also unload your key after several seconds. It’s up to you.
Also, select your file previously imported in the Attachment part (from the drop-down menu). You will see that your private key file is available, and once selected, we could see some information, including the Public key (available for copy!).
Finish the creation of your secret by clicking on “Ok”. And that’s all you need to have your keys loaded into your SSH Agent!
Lock and unlock your database to see that the key is now available in your agent:
ssh-add -l
256 SHA256:pPcl+pI2TTkzIL6psB/13wwxQLOA9jZp1+A/E+sHI3Q bastion production (ED25519)
Yes! It’s your key (mine in this example …).
Bonus tips for MacOS X users
By default, KeePassXC prevents recordings and screenshots of the application window on Windows and macOS.
As said in the documentation, you can’t take a screenshot of KeePassXC. It’s a good security feature. But if one day you need to do it (like, when writing an article about this awesome tool), we will have to bypass this option. The manual says to launch the application with --allow-screen capture parameter. But how to do it on MacOSX? Here we go:
open -a KeePassXC.app /dev/null --args --allow-screencapture
Conclusion
I hope this article convinces you that KeePassXC is a great piece of software and can do more than securely store your passwords: it could help you in your daily routine by keeping your SSH key secret and save time when you use them.
If you have any questions, or if you think I missed something with my KeePassXC / SSH Agent usage, feel free to contact me, I will be happy to continue the conversation.
