TL;DR

Let us cut into the meat of Nix and finally take a look at how we can use what we tediously went through vol.1 and vol.2 for. Declarative and isolated development environments.

volume 3: declarative developer environments

Prerequisites

• Again we need a working nix installation, see vol.1
• We also need Nix integrated with the user environment, see vol.2

Context

Why do we want declarative development environments ?

To be fair, once an development environment has been set up, it mostly chugs along just fine, and will do so for some time.

Eventually the developer needs to update something, for example NodeJS to another version. She can choose to do so globally, forcing all the apps she works on to run on the global version provided. That might not be ideal, so there are tools that solve this problem by allowing her to install multiple versions of Node and either set one for a specific project, or manually switch between them.

There are many, many, tools that do this (to list a few):

• Polyglot: asdf
• NodeJS: volta, fnm, n, nvm
• Python: pyenv, venv
• Rust: r, rustup, zrsvm, rsvm
• Go: gvm, g

Nix however, can replace all of these. It circumvents the problem by changing the paradigm. What if all versions can be installed side-by-side, making it so that there is no truly global version at all ?

There may be a default version, sure, but with declarative environments we can switch between them automatically, or at will. And once we leave the environment where we loaded the tool, it is removed, so it only exists in the context we want it in.

By using a declarative style, we tell the system what state we want it to be in, and the system moves to that state. And we can do that at any granularity: coarse (all of our development), or fine (just a single project).

On with the show.

Our first development environment

Having an organised folder structure will have an immense impact on how reusable how environments will be, and allow for tricks like nesting developer environments. I expect most developers have a folder that contains the projects that they work on, and for now that is perfectly fine.

If you don't have something like that though, and dump all projects in your $HOME ... May I suggest to create a ~/dev folder, at least for the purpose of this experiment.

Jump into ~/dev and create a new file called shell.nix. This is where we will declare what our environment should be:

Starting with some basics, let's add NodeJS 18, Python 3, Go 1.20, and Java 17.

with (import <nixpkgs> {});

mkShell {
  buildInputs = [
    nodejs-18_x
    python3
    python3.pkgs.pip
    go_1_20
    maven
    jdk17
  ];

  shellHook = ''
  '';
}

Save and exit the file, and type nix-shell. After a bit of downloading and plumbing, you will be dropped into a shell environment that may change your prompt:

~/dev% nix-shell

[nix-shell:~/dev]$

In the shell we can see that we indeed have an environment with the tools we declared^[1]:

[nix-shell:~/dev]$ node --version
v18.17.1

[nix-shell:~/dev]$ go version
go version go1.20.7 linux/amd64

[nix-shell:~/dev]$ javac -version
javac 17.0.7

[nix-shell:~/dev]$ python3 --version
Python 3.10.12

We can also make sure we are in fact using the versions provided by Nix:

[nix-shell:~/dev]$ which python3
/nix/store/wlxpsdzfvdanfzh704qmgyzb42qvy4fr-python3-3.10.12/bin/python3

[nix-shell:~/dev]$ which node
/nix/store/5my366b5ws4mnmycnynrwiwmz93zj37b-nodejs-18.17.1/bin/node

[nix-shell:~/dev]$ which go
/nix/store/54qksp8q7lwwhpjyxqbhzvrmn883gkpy-go-1.20.7/bin/go

[nix-shell:~/dev]$ which javac
/nix/store/l4plrvlxi46fiy65sz75x7850wx0nhhw-openjdk-17.0.7+7/bin/javac

Exit the nix-shell environment, and poof. The tools are either gone or reverted to whatever I had before:

~/dev% which javac
javac not found

That's a neat trick, but we don't want to type nix-shell, and we absolutely don't want to change our prompt and/or drop into bash when we are using zsh in our main environment. We want some things to blend between environments.

Enter: direnv and nix-direnv

• https://github.com/direnv/direnv
• https://github.com/nix-community/nix-direnv

These two tools will do the heavy lifting for us. direnv should be installed in our user environment that we set up in vol.2.

So jumping back to ~/.config/nixpkgs/config.nix, we can add it to our userBasePkgs:

pkgs : {
    allowUnfree = true;
    packageOverrides = pkgs: with pkgs; rec {
        nixUserProfile = writeText "nix-user-profile" ''
            export PATH=$HOME/.nix-profile/bin:/nix/var/nix/profiles/default/bin:/sbin:/bin:/usr/sbin:/usr/bin
            export NIX_PATH="nixpkgs=$HOME/.nix-defexpr"
        '';
        userBasePkgs = pkgs.buildEnv {
            name = "user-base";
            paths = [
                (runCommand "profile" {} ''
                    mkdir -p $out/etc/profile.d
                    cp ${nixUserProfile} $out/etc/profile.d/nix-user-profile.sh
                '')
+               direnv
            ];
            pathsToLink = [
                "/share"
                "/bin"
                "/etc"
            ];
            extraOutputsToInstall = [ "man" "doc" ];
        };
    };
}

And we instruct Nix to set up our user environment with nix-env -ir user-base.

Now direnv should be available:

$ which direnv
/nix/store/h77a0hqm3jcfqq7fgs310rf5l9w9g66y-direnv-2.32.3/bin/direnv

Last installation step for direnv is to hook it into our environment. In .zshrc or .bashrc add the line for your shell^[2]:

# for zsh
eval "$(direnv hook zsh)"

# for bash
eval "$(direnv hook bash)"

Now, create a .envrc in your ~/dev folder, making it a sibling with shell.nix:

~/dev% tree -L 1
.
├── shell.nix
└── .envrc

Add this content to .envrc:

use nix

Save and quit, and then run direnv allow in the ~/dev folder to allow direnv to load the file automatically.

It never runs any .envrc that isn't explicitly allowed, and if the .envrc file changes, you need to manually allow it again. This is a safeguard in case someone for example sneaks an .envrc file into version control.

Upon running direnv allow, it will immediately source the .envrc file and load the environment as specified. It will spit out something like this:

~/dev% direnv allow

direnv: loading ~/dev/.envrc
direnv: using nix
direnv: export +AR +AS +CC +CONFIG_SHELL +CXX +DETERMINISTIC_BUILD
+HOST_PATH +IN_NIX_SHELL +JAVA_HOME +LD +NIX_BINTOOLS
+NIX_BINTOOLS_WRAPPER_TARGET_HOST_x86_64_unknown_linux_gnu
+NIX_BUILD_CORES +NIX_BUILD_TOP +NIX_CC
+NIX_CC_WRAPPER_TARGET_HOST_x86_64_unknown_linux_gnu +NIX_CFLAGS_COMPILE
+NIX_ENFORCE_NO_NATIVE +NIX_HARDENING_ENABLE +NIX_LDFLAGS +NIX_STORE +NM
+NODE_PATH +OBJCOPY +OBJDUMP +PYTHONHASHSEED +PYTHONNOUSERSITE
+PYTHONPATH +RANLIB +READELF +SIZE +SOURCE_DATE_EPOCH +STRINGS +STRIP
+TEMP +TEMPDIR +TMP +TMPDIR +XDG_DATA_DIRS +_PYTHON_HOST_PLATFORM
+_PYTHON_SYSCONFIGDATA_NAME +__structuredAttrs +buildInputs +buildPhase
+builder +cmakeFlags +configureFlags +depsBuildBuild
+depsBuildBuildPropagated +depsBuildTarget +depsBuildTargetPropagated
+depsHostHost +depsHostHostPropagated +depsTargetTarget
+depsTargetTargetPropagated +doCheck +doInstallCheck +mesonFlags +name
+nativeBuildInputs +out +outputs +patches +phases +preferLocalBuild
+propagatedBuildInputs +propagatedNativeBuildInputs +shell +shellHook
+stdenv +strictDeps +system ~PATH

~/dev%

If you have new packages, it can take some time for it to download them and put them in the /nix store. Hang tight. Subsequent loads will be faster, and through the power of caching, we can make them even faster.

Any changes to shell.nix will be picked up and trigger direnv to load our changed environment.

At this point we could call it a day, but it's a bit too slow, so let's add in an extension to direnv called nix-direnv that caches our development environments, making them a lot faster to load, after the cache is populated.

There are a few ways to install nix-direnv, either by adding it to our user configuration^[3], or by using a directive in .envrc to load it for us (also cached, so not done each time):

In .envrc:

+if ! has nix_direnv_version || ! nix_direnv_version 2.3.0; then
+  source_url "https://raw.githubusercontent.com/nix-community/nix-direnv/2.3.0/direnvrc" "sha256-Dmd+j63L84wuzgyjITIfSxSD57Tx7v51DMxVZOsiUD8="
+fi
use nix

It does a SHA validation to make sure that what it downloads matches what we think it does, which is a reasonable thing to do when loading scripts from the internet.

On saving and quitting our editor, direnv will angrily state:

direnv: error /home/varl/dev/.envrc is blocked. Run `direnv allow` to approve its content

After another direnv allow, it will use cached developer environments, speeding up the time it takes to cd into a directory and setup the environment according to shell.nix by a lot.

We can verify that by looking for lines like:

direnv: loading ~/dev/.envrc
direnv: loading https://raw.githubusercontent.com/nix-community/nix-direnv/2.3.0/direnvrc (sha256-Dmd+j63L84wuzgyjITIfSxSD57Tx7v51DMxVZOsiUD8=)
direnv: using nix
+direnv: nix-direnv: renewed cache

And:

direnv: loading ~/dev/.envrc
direnv: loading https://raw.githubusercontent.com/nix-community/nix-direnv/2.3.0/direnvrc (sha256-Dmd+j63L84wuzgyjITIfSxSD57Tx7v51DMxVZOsiUD8=)
direnv: using nix
+direnv: nix-direnv: using cached dev shell

Shell hooks

In our shell.nix above, notice that there is a shellHook^[4] directive that was empty. Let's see what we can use it for:

with (import <nixpkgs> {});

mkShell {
  buildInputs = [
    nodejs-18_x
    python3
    python3.pkgs.pip
    go_1_20
    maven
    jdk17
  ];

+  shellHook = ''
+  '';
}

shellHook is run after the setup, so we can use it for initialisation that is specific for our nix-shell. For example, we can export environment variables to change things according to our context.

We can use a specific browser profile to open links, a few examples I use is that we can change the GOPATH and NODE_PATH, or autoload Python venvs.

with (import <nixpkgs> {});

mkShell {
  buildInputs = [
    nodejs-18_x
    python3
    python3.pkgs.pip
    go_1_20
    maven
    jdk17
  ];

  shellHook = ''
+    # setup node
+    mkdir -p .nix-node
+    export NODE_PATH=$PWD/.nix-node
+    export NPM_CONFIG_PREFIX=$PWD/.nix-node
+
+    # setup python
+    mkdir -p .nix-python/venvs
+    xport SOURCE_DATE_EPOCH="$(date +%s)"
+    export VIRTUAL_ENV_DISABLE_PROMPT=1
+    export PIPENV_VERBOSITY=-1
+
+    if [ -f $PWD/.nix-python/venv/bin/activate ]; then
+      source $PWD/.nix-python/venv/bin/activate
+    fi
+
+    # setup go
+    mkdir -p .nix-go/{bin,pkg,src,cache}
+    touch .nix-go/env
+    export GOPATH=$PWD/.nix-go
+    export GOCACHE=$PWD/.nix-go/cache
+    export GOENV=$PWD/.nix-go/env
  '';
}

We could also do these things in .envrc (more or less), as both direnv and nix-shell ensures that it also unloads any environment variables we set up when we leave a directory, so they only exist in the context we are in.

Going a step further

While what we have now is perfectly useful, the ~/dev/shell.nix approach is not what I actually use.

My setup looks more like:

~/dev% tree -L 2
.
├── work
│   ├── apps
│   │   ├── .envrc
│   │   └── shell.nix
│   ├── .envrc
│   └── shell.nix
├── forks
│   ├── .envrc
│   └── shell.nix
├── personal
│   ├── .envrc
│   └── shell.nix
├── .envrc
└── shell.nix

With our current setup, each environment would fully overwrite the previous (so no nesting of environments, yet).

Nested developer environments

Luckily, there is a handy feature in direnv that allows us to solve this by adding a single line to our shell.nix that we want to instruct to inherit from the parent shell.nix.

If we want our ~/dev/work environment to inherit from the parent ~/dev, we put the following in ~/dev/work/.envrc:

if ! has nix_direnv_version || ! nix_direnv_version 2.3.0; then
  source_url "https://raw.githubusercontent.com/nix-community/nix-direnv/2.3.0/direnvrc" "sha256-Dmd+j63L84wuzgyjITIfSxSD57Tx7v51DMxVZOsiUD8="
fi
+source_up
use nix

Now we can specify tools that are valid across all contexts in ~/dev/shell.nix:

with (import <nixpkgs> {});

mkShell {
  buildInputs = [
    yq
    jq
    gh
    shellcheck
    mr
    tokei
  ];

  shellHook = ''
    # empty so far
  '';
}

And get more specific, so for example for ~/dev/work/shell.nix:

with (import <nixpkgs> {});

mkShell {
  buildInputs = [
    nodejs-18_x
    python3
    python3.pkgs.pip
    python3.pkgs.pex
    kubectl
    kustomize
    (google-cloud-sdk.withExtraComponents [ google-cloud-sdk.components.gke-gcloud-auth-plugin ])
    go_1_20
    cloudflared
    glibcLocales
    maven
    jdk17
  ];

  shellHook = ''
    # ... truncated ...
  '';
}

For specific projects we can use nesting as an override feature, so we can swap nodejs-18_x for nodejs-20_x in specific projects, for example.

I think that's about it for vol.3.

/v.