varl's nixtapes vol.2

9 Aug 2023

Viktor Varland (varl)

TL;DR

Now that we have a working nix installation and understand how the nix utilities (nix-* commands) work at a basic level, we can integrate nix more deeply into our user environment to manage packages.

For MacOS we will end up with a decent replacement for Homebrew, and for Linux, we will have a package manager that is independent from the system package manager that we can use to manage packages for our user.

volume 2: integrating nix with the environment

Prerequisites

A working nix installation, see vol.1

Nix Channels

A new concept has appeared !

To find Nix packages, there are different channels that exist under the Git repository nixpkgs.

There are stable and unstable, large and small, channels that have different use cases.

I like to be able to lock my channel to a specific commit, so I clone nixpkgs to ~/.nix-defexpr so I can manipulate it at will by switch branches, updating, making local changes to try out (read: break) various things.^[1]

All in all, I find it handy, so I would recommend doing the same:

rm -rd ~/.nix-defexpr
git clone --depth=1 https://github.com/NixOS/nixpkgs.git ~/.nix-defexpr

To make it stick immediately (we will make it more permanent later) run:

export NIX_PATH="nixpkgs=$HOME/.nix-defexpr"

Channels can be listed with:

nix-channel --list

nix-channel also provides --add, --update, and --remove commands.

I would recommend removing all existing channels if you go along with having a local clone of nixpkgs.

nix-channel --remove {channel-alias}

It is by far the simplest way, so until you need multiple channels, sticking with one is the most predictable. Each channel also has the concept of generations, and it is possible to rollback an --update to a previous generation.

Quite powerful, and overkill for my ends. Onwards !

Configure user packages

A promise I made is declarative package management, and that boils down being able to state what packages we want, and have the package manager make sure that we have those in our environment.

Let's take a look at how to configure at the user level, as opposed to system, and project, level.

If you don't have the file ~/.config/nixpkgs/config.nix we need to create it:

mkdir -p ~/.config/nixpkgs
touch ~/.config/nixpkgs/config.nix

Open the config.nix file in an editor and let's dip into Nix-the-language.

Paste the following into the file:

pkgs : {
    allowUnfree = true;
    packageOverrides = pkgs: with pkgs; rec {
        nixUserProfile = writeText "nix-user-profile" ''
            export PATH=$HOME/.nix-profile/bin:/nix/var/nix/profiles/default/bin:/sbin:/bin:/usr/sbin:/usr/bin
            export NIX_PATH="nixpkgs=$HOME/.nix-defexpr"
        '';
        userBasePkgs = pkgs.buildEnv {
            name = "user-base";
            paths = [
                (runCommand "profile" {} ''
                    mkdir -p $out/etc/profile.d
                    cp ${nixUserProfile} $out/etc/profile.d/nix-user-profile.sh
                '')
            ];
            pathsToLink = [
                "/share"
                "/bin"
                "/etc"
            ];
            extraOutputsToInstall = [ "man" "doc" ];
        };
    };
}

I'm not going to go into the semantics of the language, but rather talk a bit about what we are doing and why it is helpful.^[2]

allowUnfree determines if we are able to use non-free, as in proprietrary, packages from nixpkgs. By default only free software is allowed to install, and your mileage will vary if you need non-free software or not. I do, so I have it set to true.

packageOverrides is where we will define our own packages to accomplish a one-shot command to install all the packages we want in our environment.

The first package we define is nixUserProfile which uses a writeText helper function to build a package that consists of a file with the text we have defined.

The output file will have the contents:

export PATH=$HOME/.nix-profile/bin:/nix/var/nix/profiles/default/bin:/sbin:/bin:/usr/sbin:/usr/bin
export NIX_PATH="nixpkgs=$HOME/.nix-defexpr"

This does two things.

First, it sets up our PATH variable to include the bin folders that contain our nix package binaries. We want them first in PATH to ensure that the nix packages are used before the system packages that may already exist on the system.

Second, it sets the NIX_PATH to the ~/.nix-defexpr folder where we cloned the nixpkgs repo to. I mentioned that we did a one-off export to set this, and this is the more permanent fix to make sure it sticks across reboots and all interactive shells we use.

However. This simply builds a package that consists of a file, but it doesn't run it.

The next package we define is our userBasePkgs, and here is a trimmed version of the above to make it easier to deconstruct.

userBasePkgs = pkgs.buildEnv {
    name = "user-base";
    paths = [
        (runCommand "profile" {} ''
            mkdir -p $out/etc/profile.d
            cp ${nixUserProfile} $out/etc/profile.d/nix-user-profile.sh
        '')
    ];
};

We have to give it a name, so we can refer to it, and I chose user-base. The convention will make more sense down the line.

In paths it is a bit murkier. Right off the bat we invoke another nix helper function to run a command to trigger a side effect on our system.

(runCommand "profile" {} ''
    mkdir -p $out/etc/profile.d
    cp ${nixUserProfile} $out/etc/profile.d/nix-user-profile.sh
'')

$out refers to our ~/.nix-profile folder, so it creates the folder ~/.nix-profile/etc/profile.d directory and copies the output from the nixUserProfile package we built before to it.

Now that we have a package that has a build output, we need to install the package to reflect the change to our environment.

Install `user-base` with `nix-env`

Now that we have our user-base package, let us install it and continue hooking up our user environment with the nix user environment.

nix-env --install --remove-all user-base

Or, short form:

nix-env -ir user-base

This evaluates our nix package definition in ~/.config/nixpkgs/config.nix and builds our derivation, and installs it into our environment. We didn't define any additional software yet so we can check if ~/.nix-profile/etc/profile.d/nix-user-profile.sh contains the two lines we expect:

cat ~/.nix-profile/etc/profile.d/nix-user-profile.sh

export PATH=$HOME/.nix-profile/bin:/nix/var/nix/profiles/default/bin:/sbin:/bin:/usr/sbin:/usr/bin
export NIX_PATH="nixpkgs=$HOME/.nix-defexpr"

Nothing in our shell knows this file exists though, so we need to wire up a few more things.

Loading scripts in `~/.nix-profile/etc/profile.d`

Add this chunk to ~/.zprofile (zsh), ~/.bash_profile (bash), or ~/.profile (bash, sh).

It loads scripts in ~/.nix-profile/etc/profile.d, and this will handle our script as well, and ensure that we have NIX_PATH and PATH setup correctly.

if [ -d $HOME/.nix-profile/etc/profile.d ]; then
  for i in $HOME/.nix-profile/etc/profile.d/*.sh; do
    if [ -r $i ]; then
      . $i
    fi
  done
fi

Source the profile file, or start a new shell, and then test it:

echo $PATH
/home/varl/.nix-profile/bin:/nix/var/nix/profiles/default/bin:/sbin:/bin:/usr/sbin:/usr/bin

echo $NIX_PATH
nixpkgs=/home/varl/.nix-defexpr

Managing packages

I promised package management for this volume, and it shall be done. We have the right amount of environment integration to make it all work.

Back in the ~/.config/nixpkgs/config.nix file, let's extend it to cover:

user-base: packages that are cross-platform and I want available always, both on mac and linux.
user-linux: linux specific packages that do not exist, or I don't want, on MacOS.
user-macos: macos specific packages that i don't want/need on other OSes.

pkgs : {
    allowUnfree = true;
    packageOverrides = pkgs: with pkgs; rec {
        nixUserProfile = writeText "nix-user-profile" ''
            export PATH=$HOME/.nix-profile/bin:/nix/var/nix/profiles/default/bin:/sbin:/bin:/usr/sbin:/usr/bin
            export NIX_PATH="nixpkgs=$HOME/.nix-defexpr"
        '';
        userBasePkgs = pkgs.buildEnv {
            name = "user-base";
            paths = [
                (runCommand "profile" {} ''
                    mkdir -p $out/etc/profile.d
                    cp ${nixUserProfile} $out/etc/profile.d/nix-user-profile.sh
                '')
                fzf
                fd
                ripgrep
                vim
                rsync
                gnupg
                curl
                wget
            ];
            pathsToLink = [
                "/share"
                "/bin"
                "/etc"
            ];
            extraOutputsToInstall = [ "man" "doc" ];
        };

        userLinuxPkgs = pkgs.buildEnv {
            name = "user-linux";
            paths = [
                userBasePkgs
                bitwarden
                bitwarden-cli
                signal-desktop
                xcolor
            ];
            pathsToLink = [] ++ (userBasePkgs.pathsToLink or []);
            extraOutputsToInstall = []
                ++ (userBasePkgs.extraOutputsToInstall or []);
        };

        userMacOSPkgs = pkgs.buildEnv {
            name = "user-macos";
            paths = [
                userBasePkgs
                bash
                zsh
                zsh-completions
                alacritty
                karabiner-elements
                coreutils
                podman
                podman-compose
            ];
            pathsToLink = [
                "/Applications"
            ] ++ (userBasePkgs.pathsToLink or []);
            extraOutputsToInstall = []
                ++ (userBasePkgs.extraOutputsToInstall or []);
        };
    };
}

Key here is to notice that we defined two more packages, userLinuxPkgs and userMacOSPkgs^[3], and the first item in the paths list is userBasePkgs.

This is there so that when we install user-linux or user-macos, it installs the user-base package which includes any common packages that we might have. We definitely have one in common, the package that creates the script to update our environment, but the list can be as long or short as we want it.

On Linux I would run:

nix-env -ir user-linux

And on MacOS I would run:

nix-env -ir user-macos

nix-env would then rebuild my environment and install any packages defined in ~/.config/nixpkgs/config.nix.

To remove packages, we just delete them from the config.nix file, and re-run the relevant nix-env -ir command.

Finding package names to use in `config.nix`

Since we can pin our nixpkgs to a commit, it is nice to avoid some versions on packages in config.nix.

When we search with nix-env -qa vim, we would get back a string like vim-9.0.1642. We can add that to the paths list, but when there is a new version it would stop working.

Instead we can search for packages with the -P flag to find the unambiguous attribute path.

nix-env -qaP vim

vim vim-9.0.1642

The first column, vim, is the path we can use to refer to the vim package without the version.

For NodeJS it can look like:

nix-env -qaP nodejs
nodejs-14_x         nodejs-14.21.3
nodejs_14           nodejs-14.21.3
nodejs-16_x         nodejs-16.20.1
nodejs_16           nodejs-16.20.1
elmPackages.nodejs  nodejs-18.17.0
nodejs-18_x         nodejs-18.17.0
nodejs_18           nodejs-18.17.0
nodejs_20           nodejs-20.5.0

Here we could opt for nodejs_18 to get the latest 18.x version.

When looking for multiple packages, I would recommend doing something like this:

nix-env -qaP | fzf

nix-env -qa is a very slow command that loads all the packages before searching according to a pattern, so loading the packages once, then piping them into fzf provides a nice and fast search interface over all the packages in nixpkgs.

Where are we now ?

We have set up our NIX_PATH to refer to a local clone of nixpkgs that we use to pin software versions.

We have also configured our PATH to include directories where nix places binaries that we want to take precedence.

We did so by leveraging a custom package that generates a script to update our environment.

We still needed to manually wire up our shell to actually run the scripts when the shell starts, and we did that in our .profile file (different for sh/bash/zsh).

We then added more packages to our config.nix and looked at how to manage packages in two ways: 1) common packages that are cross-platform and 2) packages that are specific to an operating system (mac/linux).

We can install all the common + operating system specific packages with a single command using nested package evaluation in Nix the language.

And finally we took a look at how to search for packages and refer to them in config.nix.

All this means that you can declare packages in config.nix, and move that file across computers. It will work the same regardless of machine. Put it in a Git repo and keep your user environments in sync.

/v.

I use the master branch, which is unstable. For my usage I haven't had any real breakages. Simply switch branch to swap the channel. Simply git switch nixos-23.05 to use the stable branch. ↩︎
You can dive into the language yourself if you wish: https://nixos.org/manual/nix/stable/language/index.html ↩︎
Notice that the pathsToLink for userMacOSPkgs includes /Applications. This is an attempt to link the package to the /Applications to allow e.g. Spotlight to find the application we installed with Nix. ↩︎

TL;DR