Nix Packaging Quickstart Guide

Creating a simple NixOS configuration is quite easy - just throw some options into configuration.nix and you're done. However, when you inevitably want to go beyond what's provided in nixpkgs, you face a challenge - what lies beyond? How can I do what nixpkgs usually does for me? There are many reference documents covering the very basic, low-level parts of Nix - but how do I combine them into something high-level?

One of the significant first hurdles I've encountered is packaging. So, in this post I'll attempt to explain the basics of NixOS packaging (you will inevitably have to read parts of the nixpkgs manual, but this article should provide a starting point).

Nix Language

Let's start with the basics - there are some parts of the Nix language you may not know.

First, functions. Those familiar with lambda calculus will find the way they work in Nix familiar. Functions are defined with <function argument>: <function body> (indeed, argument, not arguments) and called with <function> <argument>. For example, builtins.isString "abcd" will evaluate to true, and (x: builtins.isString x || builtins.isNull x) 5 will evaluate to false. A function may return another function, allowing you to pass another argument, for example a: b: a + b is a function that adds 2 numbers (it can be called as func 5 6). If you forget an argument (func 5), you will get a function instead of your expected value, so whenever see an error "expected X, got function", this is probably the case.

Additionally, there's special syntax for functions that receive an attrset argument (attribute sets are key-value dictionaries/maps, like javascript objects): for example, { a, b }: a + b is a function that takes an attrset with attrs a and b and returns their sum. It can be called with func { a = 10; b = 20; }. By default, any unknown attrs will trigger an error (so you can't do func { a = 10; b = 20; c = 30; }), but you can ignore them by adding ... after the last argument like { a, b, ... }.

If you want some attrs to be optional, there's syntax for that too - attr ? default value, for example ({ a ? 1, b ? 2 }: a + b) { b = 5; } will evaluate to 6.

Then there's the import expression. What import expression does is take a path - say, import ./file.nix, and executes the code in that Nix file. If the path you provided is a directory, it executes default.nix in that directory. If it isn't a path, it gets converted to a path.

Additionally, some imports may be defined externally. For example, by calling import <nixpkgs>, you import your system's version of nixpkgs.

You much more often see import <nixpkgs> { }. This is because nixpkgs's default.nix file defines a function that takes a bunch of optional attrs. To do anything useful with nixpkgs, you have to call that function first.

(side note - in flakes, nixpkgs.legacyPackages.${system} is syntactic sugar for import nixpkgs { system = ...; })

Package Structure

So, how do we create our first package?

First, find a similar package in the same language using the same build system (you can use search.nixos.org for easily finding a package's nix source code). You might have to find multiple packages. Hopefully, it won't be that long or complicated (Gradle, I'm looking at you...). For our purpose, let's assume your application is a C package, as that's essentially the default - but check the documentation of nixpkgs for more info on each language. This post's goal is to explain the general process, not specific details.

Here's a sample C++ library, here's a sample Python package, here's a sample Rust package, chosen completely at random.

When looking at the source, you will see that the package source is all in a separate directory (with default.nix being the "entry point"). Technically this doesn't have to be the case, but it's very convenient in case the package has additional files required for building using Nix - for example, if a patch has to be applied, patches may get put in the same directory.

default.nix

Let's read the first few lines in the C package I linked above:

{ lib
, stdenvNoCC
, fetchFromGitHub
}:

stdenvNoCC.mkDerivation rec {

As you can see, default.nix defines a function. This function returns a derivation (what's a derivation? read on), the first few lines define the function's arguments. Where do the arguments come from?

In the past, they were explicitly passed when import'ing the package. Even now, some arguments may be directly supplied by the user - for example, when override'ing the package, you simply change those inputs. However, most of them are usually provided by pkgs.callPackage. It imports a file as a function, checks which attributes were supplied by the user, and for every missing attribute it automatically takes it from pkgs, and then makes the result override'able using lib.mkOverridable.

And, indeed, this is how it's done in all-packages.nix:

popl = callPackage ../development/libraries/popl { };

One of the inputs was lib - this is the nixpkgs standard library, it provides many helpful functions that aren't built into Nix. You can find a reference in the nixpkgs manual, but I usually use this page as it provides reference for built-in Nix functions as well. Additionally, using a language server like nil can help you write Nix code in your editor of choice.

The function in default.nix returns a derivation. A derivation is something that calls some commands and takes some files, and produces some other files in Nix store. Most inputs will be provided to the commands as environment variables, for example $out refers to where Nix wants you to put your derivation's output.

Derivation Structure

For C apps and libs as well as most build systems for other languages, stdenv.mkDerivation is used - it creates a derivation that has access to a C compiler and various Unix-y utilities. However if you don't need a C compiler, you should use stdenvNoCC.mkDerivation. Overall, what you have to call will depend on the language you're packaging for (rustPlatform.buildRustPackage, python3Packages.buildPythonPackage, python3Packages.buildPythonApplication, etc)

Let's continue reading the source code:

stdenvNoCC.mkDerivation rec {
  pname = "popl";
  version = "1.3.0";

  src = fetchFromGitHub {
    owner = "badaix";
    repo = pname;
    rev = "v${version}";
    hash = "sha256-AkqFRPK0tVdalL+iyMou0LIUkPkFnYYdSqwEbFbgzqI=";
  };

(rec in this case means this is a recursive attrset - an attrset that can refer to its own keys. rec { a = 5; b = a; } specifies an attrset with a and b set to the same value.)

The first few attrs provided to mkDerivation are pname, version and src - package name, version and source code. Since the derivation must (well, should - there are still ways to break it) be reproducible, as reproducibility is one of Nix's goals, it must either have a known output hash or not depend on the outside world, and in most cases the output depends on the target platform and dependency versions so you can only provide input hashes, not output hashes. This means our derivation can't access the outside world, including the internet - that's why source code has to be fetched before the derivation is ran. The source code is fetched with a "fetcher" (in this case fetchFromGitHub, see nixpkgs manual for a list of those) - fetchers are derivations that download source code. Of course, since they're derivations, to download stuff from the internet they must have a known output hash to make sure everyone gets the same source code, and you have to specify the hash as a fetcher argument. Alternatively, if you have the code locally, you may skip using a fetcher and set src to the path to the source code directory (use a relative path to ensure purity).

Next you usually have buildInputs and nativeBuildInputs, but the C++ library I linked doesn't have them, as it's a header-only library. The former are packages that will be used by the package when it is already built - for example, C libraries. The latter are used for building the package, not for running it, and must therefore runnable on the build system, but not necessarily the target system.

There are also propagatedBuildInputs, which mean that any dependency of this package must also depend on those (most often used for Python packages), and propagatedNativeBuildInputs, and some other input types - once again, read the manual if you want to learn more.

The Python package I linked does have propagatedBuildInputs:

  propagatedBuildInputs = [
    numpy
  ];

If you want to know what inputs the package needs, either try running the build until it complains about a missing dependency, or consult the package documentation for a dependency list. Additionally, if it complains about specific files missing and you don't know where they can be found, nix-locate is very helpful for locating specific files across nixpkgs - nix-locate bin/sway to find a binary called sway, nix-locate gtk+-2.0.pc to find a package that provides gtk+-2.0.pc. Sometimes a file may be copied into multiple packages - in that case you're going to have to find what the "source of truth" for the file is. Other times it's straightforward as only one package provides it.

Reading on the C++ derivation source:

  dontConfigure = true;
  dontBuild = true;
  dontFixup = true;

  installPhase = ''
    runHook preInstall

    install -Dm644 $src/include/popl.hpp $out/include/popl.hpp

    runHook postInstall
  '';

Here we have some environment variables. Well, in fact src was an environment variable as well, but without reading mkDerivation's source code, we can't really tell what is an environment variable, what is an argument read by mkDerivation, and what is both at the same time - and finally the package's "phases" - the shell code to be executed during the build process.

There are many phases to building a package - unpackPhase, patchPhase, buildPhase, installPhase, fixupPhase, etc. The most important ones are build phase, which takes care of building the package, and install phase, which moves the build results to $out. Build phase generally starts in the root of the unpacked and patched sources. The phases may reference some environment variables, and as the stdenv builder is a magic combination of shell scripts (hooks) referencing various environment variables, you simply have to deal with it.

The default configure phase looks for ./configure and executes it if necessary. However, it also checks for a variable called dontConfigure, and since we set it to 1 (yes, true gets converted to 1, false gets converted to an empty string), it doesn't execute anything. The same applies for the build phase (which would've tried calling something like make).

The default behavior of the different phases may change depending on native build inputs if they introduce "hooks" - additional code that runs during the build! For example, if you add meson, a meson hook which uses meson will automatically be added for configuring the package. All native build inputs that provide hooks automatically alter the way the package is built. For example, autoPatchelfHook automatically patches binaries, rewriting the .so files' paths to ones in Nix store. As you can see above, there is usually a way to disable the hook with an environment variable if you only want the package itself, but not the hook.

If you are confused what derivation arguments you need to change, read nixpkgs manual, or, worse, the shell scripts themselves - start with mkDerivation's setup.sh - note that you can override any function from that file by passing shell commands to mkDerivation - and continue with any "hooks" defined by the native build inputs, like cmake.

Additionally, you may make the derivation configurable by making one of derivation's arguments provided by the user. For example, if you add enableFeatureX ? false and use that flag in generating the derivation (for example, including or not including a certain input; adding build flags; etc), users can call package.override { enableFeatureX = true; } or callPackage ./package { enableFeatureX = true; }.

A good thing to get a quick start on writing your own package is using nix-init to get a quick draft. nix-init won't create a fully working derivation in most cases, but it will help you to get started, using the required fetcher, filling the source hash and package metadata, guessing the language the package is written in.

Running the Derivation

So, let's assume you've written the derivation and want to test it. How do you do it?

Use the aforementioned callPackage with any Nix tooling!

For example, in your system config:

pkgs.callPackage ./package.nix {
  # here you can pass something to the package function
  # everything you haven't defined will be filled in from pkgs
  # passing something here is roughly the same as calling .override on the result
}

or using nix-build:

$ nix-build -E "(import<nixpkgs>{}).callPackage ./package.nix {}"
...
$ # you will now have a symlink "result" in the working directory with the package's files

or nix-shell:

$ nix-shell -p "(import<nixpkgs>{}).callPackage ./package.nix {}"

Or if you have added a package to nixpkgs and want to test it before opening a merge request:

.../nixpkgs $ nix-build  "(import ./. {}).your-package"
.../nixpkgs $ # or using the new nix command
.../nixpkgs $ nix build .#your-package

Finally, for lack of a better place to put it - while package.override { ... } overrides the package function's inputs, package.overrideAttrs (oldAttrs: { ... }) overrides whatever you pass to mkDerivation. For example, when you need an old version of a program, you can override src. This, too, is covered in the nixpkgs manual.

Packaging is an immensely complex topic, and nixpkgs has an incredible amount of code dedicated to it even if you exclude the final packages' code and only include the basic code for interacting with build systems, but hopefully this covers enough for you to understand the basics of how to do it in Nix in a way not covered by the reference manual!

Let me end this with an operator cheatsheet, since you're probably new to Nix as a language if you're reading this:

a?b - returns true if a has an attr named b
a.b or c - returns a.b if a has an attr b, otherwise returns c. Not to be confused with a.b || c
a // b - takes b, and fills in attributes it *doesn't* already have from a. Alternative way to say it - it updates a with attrs from b
a ++ b - concatenates (appends) lists a and b. Not to be confused with a + b for strings, paths and numbers
a -> b - logical implication, same as !a || b

More parts of the language you may not know about are covered in Nix Language Reference. Unlike the nixpkgs manual or Nix-the-Package-Manager reference, it's quite small, so I recommend everyone to read it.