Reproducibility for a fleet, why NixOS?

Selected music: 1000 Airplanes on the Roof by Philip Glass.

In this post, we explore what is NixOS and how to use it to make a hardened OS.

NixOS in deep

NixOS is a Linux distribution that treats an operating system like code. First, NixOS is declarative; we write a configuration that describes exactly how we want our system to look. NixOS is binary: either the installation succeeded, or it failed and nothing happened. Because the system is built from a text file, we can take that one file to a brand-new computer, run the installer, and get an identical twin of our original machine in minutes.

We will focus solely on the production logic, which is more comprehensive because it involves very strict security policies. Here is an extract:

{
  lib,
  config,
  pkgs,
  ...
}: {
  ringil.env.mode = "prod";
  users.allowNoPasswordLogin = lib.mkForce true;

  services.openssh.enable = false;
  users.users.root.hashedPassword = lib.mkForce "!";

  services.journald.extraConfig = ''
    Storage=volatile
    RuntimeMaxUse=50M
  '';

  boot.kernelParams = ["quiet" "loglevel=0" "console=tty0"];
  systemd.services."serial-getty@ttyS0".enable = false;

  services.udisks2.enable = false;
  services.xserver.enable = false;
  documentation.enable = false;
  documentation.nixos.enable = false;
}

In this code, we block all user access to the OS… in other words, no one –not even us–will be able to access the shell or the files. And Nix doesn’t like that. By default, it prevents anyone from creating such an OS. That’s why we are using lib.mkForce; that tells Nix that this is exactly the logic we’re looking for. We also disables documentation (documentation.enable), and logs are now stored in memory rather than on disk. So if someone intercepts the drone, they won’t be able to determine where it came from, nor will they know what the drone is used for.

In fact, at the same time, we need to enable disk encryption on the drone and, most importantly, make sure that no one tries to tamper with the OS during startup (or before!). For this, we’ll use lanzaboote and disko.

• disko allows us to describe our entire partitioning scheme (GPT, LUKS, Btrfs, ZFS, etc.). We’ll mainly use LUKS and ZFS here.
• lanzaboote allows us to use our own security keys to sign our NixOS kernel –partial thanks to Microsoft.

First, with disko we enable LUKS on nvme0n1 (NVMe) or mmcblk0 (eMMC). Generally, UEFI is stored on the eMMC, so there is no need to encrypt it. However, the NVMe SSD must be encrypted to prevent access to AI models, compiled code (binary), Linux files, and so on.

Then, we’ll sign our kernel. Here is how:

{
  lib,
  config,
  pkgs,
  ...
}: let
  cfg = config.ringil.env;
  isProd = cfg.mode == "prod";
in {
  boot.loader.systemd-boot.enable = lib.mkIf (!isProd) (lib.mkDefault true);

  boot.loader.efi.canTouchEfiVariables = true;
  boot.loader.timeout = lib.mkIf isProd 0;

  boot.lanzaboote = lib.mkIf isProd {
    enable = true;
    # Key must be generated using `sbctl create-keys`
    pkiBundle = "/etc/secure/keys";
  };

  boot.loader.systemd-boot.editor = lib.mkIf isProd false;
  boot.initrd.systemd.emergencyAccess = lib.mkIf isProd false;
}

In a production, if the keys have not been generated, NixOS will not be able to build because lanzaboote will not be able to sign. So we need to generate the keys beforehand. There are two options: launch the development environment first, which isn’t as secure–SSH access, root access, etc.–or generate the keys on our machine and transfer them to the Jetson.

In short, if we follow these security measures to the letter, we end up with a safe to which even we no longer have the key (especially if we generated the Secure Boot key pair on the Jetson). But everything is not lost.

The only weakness: “tout oublier”

If an attacker wants to get $200 for free, they can easily enter UEFI recovery mode and disable Secure Boot and NixOS. Although he doesn’t retrieve any data and can’t determine what the drone was used for, whether any information was transmitted, etc., he can still profit from capturing it… provided, of course, that the drone hasn’t been destroyed.

BUT, we can prevent the enemy from earning $200. All we have to do is set a Supervisor Password before launching the drone… This requires first logging in, repeatedly pressing a key, and accessing the Jetson’s UEFI. But in those cases, if we lose the password, we also lose the Jetson. It’s a double-edged sword. It’s usually not worth.

Another weakness: the Cold Boot Attack

It is difficult to implement, but it is possible. The process involves cooling the components (usually the RAM) until they freeze. This then allows the data to be copied to another drive, thanks to data persistence… and, in most cases, provides access to sensitive information such as the LUKS decryption key.

Finally, another vulnerability is sniffing. This involves connecting an electronic component and reading the data passing between two components on a board. If the TPM communicates with the motherboard via an LPC or SPI bus, an attacker could easily intercept the key.

That’s why we minimize log data… nothing is stored on disk, so there’s no information to extract. And Rust, with its most aggressive optimizations, makes it difficult to read the compiled code.

Why is this useful for a swarm?

Let’s say we have a fleet of 100 drones, each one based on the NVIDIA Jetson and Holybro S500. If we need to patch a vulnerability or change a parameter, we don’t want to SSH into each unit and risk configuration drift. With NixOS, we push one update to our configuration repository, and every drone receives the exact same upgrade-securely and reproducibly.

In this regard, there are several possible solutions. Either retrieve all the drones and reflash them, or create a script that connects to the server (via a tunnel) at firm to download the update (which must then use the same Secure Boot keys).