Getting your Haskell executable statically linked with Nix

I have been making my products statically linked over the past few days. This post presents why and how to statically link your Haskell executables and collects a mapping from obscure error to unexpected fixes.

This work would not have been possible without the many-year-long effort of people like nh2. This issue on GitHub is a good summary of what went into making this possible.

Why statically link?

The issue has a section called "Why is static linking desirable?" In short:

Statically linked executables are easier to deploy because you don't need to worry about linked libraries being deployed.
Statically linked executables start faster because they don't need to look through the file system for the libraries they link against.

Why is this so complicated?

Libc

If you don't know what this means, you have probably been (dynamically) linking your executables against glibc. Statically linking against glibc is discouraged, so we will have to statically link against musl instead. This includes building ghc against musl and involves using pkgs.pkgsMusl.

Template Haskell

In order to statically link an executable that uses Template Haskell at compile-time, the GHC RTS has to be built in the same way as the code you are compiling. This means that we need to tell ghc to produce relocatedStaticLibs in order to use Template Haskell.

Non-Haskell dependencies

Nix is excellent for providing non-Haskell dependencies, but the nixpkgs infrastructure has a dontDisableStatic flag that is false by default. (Confusing, right?! Couldn't they just have called it enableStatic = false?!) So for a bunch of non-Haskell dependencies, we'll have to override them with .overrideAttrs (old: { dontDisableStatic = true; }).

Getting started with your own project

This blog post assumes you use the nixpkgs infrastructure for defining the nix build of your Haskell package and you can build it with some nix build invocation. You will see that it is dynamically linked against a few libraries:

$ ldd ./result/bin/foo-bar
linux-vdso.so.1 (0x00007fff0e5d8000)
libm.so.6 => /nix/store/qn3ggz5sf3hkjs2c797xf7nan3amdxmp-glibc-2.38-27/lib/libm.so.6 (0x00007fc9be0ec000)
libgmp.so.10 => /nix/store/s3s7gv33p88kzbgki2bprg2a1nc7jnf8-gmp-with-cxx-6.3.0/lib/libgmp.so.10 (0x00007fc9be049000)
libc.so.6 => /nix/store/qn3ggz5sf3hkjs2c797xf7nan3amdxmp-glibc-2.38-27/lib/libc.so.6 (0x00007fc9bde61000)
librt.so.1 => /nix/store/qn3ggz5sf3hkjs2c797xf7nan3amdxmp-glibc-2.38-27/lib/librt.so.1 (0x00007fc9bde5c000)
libdl.so.2 => /nix/store/qn3ggz5sf3hkjs2c797xf7nan3amdxmp-glibc-2.38-27/lib/libdl.so.2 (0x00007fc9bde57000)
libffi.so.8 => /nix/store/9kd4bc8fpclpvf1vdwlbila71svyb6w1-libffi-3.4.4/lib/libffi.so.8 (0x00007fc9bde44000)
/nix/store/qn3ggz5sf3hkjs2c797xf7nan3amdxmp-glibc-2.38-27/lib/ld-linux-x86-64.so.2 => /nix/store/qn3ggz5sf3hkjs2c797xf7nan3amdxmp-glibc-2.38-27/lib64/ld-linux-x86-64.so.2 (0x00007fc9be1ce000)

What we will be aiming for, is to see output like this:

$ ldd ./result/bin/foo-bar
	not a dynamic executable

Fail with a static check

You can add a check to your build to test whether your binaries are now statically linked. This part is optional, but helps you future-proof your build against accidentally doing your own work.

overrideCabal fooBar (old: {
  postInstall = (old.postInstall or "") + ''
    for b in $out/bin/*
    do
      if ldd "$b"
      then
        echo "ldd succeeded on $b, which may mean that it is not statically linked"
        exit 1
      fi
    done
  '';
})

Linking against `musl`

The first step will be to use pkgs.pkgsMusl instead of just pkgs. This is "version" of nixpkgs that uses musl instead of glibc.

So your package would be pkgs.pkgsMusl.haskellPackages.foo-bar instead of pkgs.haskellPackages.foo-bar. If we build it, we'll see that it is indeed (dynamically) linked against musl instead of glibc:

$ ldd ./result/bin/foo-bar
[...]
	libc.so => /nix/store/rhpfpswa12l0hdipy9r26j844lp9pp8g-musl-1.2.3/lib/libc.so (0x00007f879edd6000)
[...]

Statically linking

The nixpkgs infrastructure lets you use pkgs.haskell.lib.overrideCabal on your packages to change build settings. We will want to override configure flags, which looks like this:

overrideCabal fooBar (old: {
  configureFlags = (old.configureFlags or [ ]) ++ [
    "new configure flags go here"
  ];
})

The first important flag to add is "--ghc-option=-optl=-static", which tells Cabal to tell ghc to tell the linker to link statically.

overrideCabal fooBar (old: {
  configureFlags = (old.configureFlags or [ ]) ++ [
    "--ghc-option=-optl=-static"
  ];
})

But if you do only that, you'll start seeing obscure errors:

foo-bar-cli> /nix/store/cjcp2ssfn2ng849nqvbyb378d0xrkl01-binutils-2.40/bin/ld: /nix/store/zpk49i784kgwfp8d86rv300aqnd1klyh-gcc-12.3.0/lib/gcc/x86_64-unknown-linux-musl/12.3.0/crtbeginT.o: relocation R_X86_64_32 against hidden symbol `__TMC_END__' can not be used when making a shared object
foo-bar-cli> /nix/store/cjcp2ssfn2ng849nqvbyb378d0xrkl01-binutils-2.40/bin/ld: failed to set dynamic section sizes: bad value
foo-bar-cli> collect2: error: ld returned 1 exit status
foo-bar-cli> `cc' failed in phase `Linker'. (Exit code: 1)

You may have to scroll to the right to read all of that. This is the important piece:

relocation R_X86_64_32 against hidden symbol `__TMC_END__' can not be used when making a shared object

We're still trying to make a shared executable (and library), so we'll also have to add these to our override to fix that error:

overrideCabal fooBar (old: {
  enableSharedExecutables = false;
  enableSharedLibraries = false;
})

Enable static linking for dependencies

Next we get more comprehensible errors:

foo-bar-cli> [6 of 6] Linking dist/build/foo-bar-cli-test/foo-bar-cli-test
foo-bar-cli> /nix/store/cjcp2ssfn2ng849nqvbyb378d0xrkl01-binutils-2.40/bin/ld: cannot find -lgmp: No such file or directory
foo-bar-cli> /nix/store/cjcp2ssfn2ng849nqvbyb378d0xrkl01-binutils-2.40/bin/ld: cannot find -lffi: No such file or directory
foo-bar-cli> collect2: error: ld returned 1 exit status
foo-bar-cli> ghc: `cc' failed in phase `Linker'. (Exit code: 1)

Again, scroll to the right to find:

ld: cannot find -lgmp: No such file or directory
ld: cannot find -lffi: No such file or directory

The linker cannot find gmp.a or ffi.a because building those is turned off by default in nixpkgs.

We can override those in a nixpkgs override, but that will cause a lot of recompilation:

gmp6 = prev.gmp6.override { withStatic = true; };
libffi = prev.libffi.overrideAttrs (old: { dontDisableStatic = true; })

So instead we make them available via some more configure flags:

overrideCabal fooBar (old: {
  configureFlags = (old.configureFlags or [ ]) ++ [
    "--ghc-option=-optl=-static"
    "--extra-lib-dirs=${final.gmp6.override { withStatic = true; }}/lib"
    "--extra-lib-dirs=${final.libffi.overrideAttrs (old: { dontDisableStatic = true; })}/lib"
  ];
})

Indeed, if we have a look at the paths that are passed in now, we see that there are .a files in there:

$ tree /nix/store/iqr454kg405rvhlfqji5liqjpdnaw3d8-libffi-3.4.4/lib               44ms
/nix/store/iqr454kg405rvhlfqji5liqjpdnaw3d8-libffi-3.4.4/lib
├── libffi.a
├── libffi.la
├── libffi.so -> libffi.so.8.1.2
├── libffi.so.8 -> libffi.so.8.1.2
└── libffi.so.8.1.2

If we have another look at the executable, we'll see that it's statically linked:

ldd ./result/bin/foo-bar
	not a dynamic executable

Yay! Except, that was the happiest path out there. Your real projects will probably be rather more complicated than my little foo-bar example executable.

The rest of this post describes errors that you might run into, and how you can go about fixing them. If you run into any errors that aren't listed here, feel free to send them to me and I'll add them.

Common errors

Zlib

If you see

ld: cannot find -lz: No such file or directory

You may need to pass in the .static version of zlib:

overrideCabal fooBar (old: {
  configureFlags = (old.configureFlags or [ ]) ++ [
    "--ghc-option=-optl=-static"
    "--extra-lib-dirs=${final.zlib.static}/lib"
  ];
})

Terminfo

If you see

ld: cannot find -ltinfo: No such file or directory

You may need to pass in the enableStatic = true version of ncurses:

overrideCabal fooBar (old: {
  configureFlags = (old.configureFlags or [ ]) ++ [
    "--ghc-option=-optl=-static"
    "--extra-lib-dirs=${final.ncurses.override { enableStatic = true; }}/lib"
  ];
})

However, this is not enough (yet). To be able to move the resulting executable onto another machine and have it find the terminfo database, we also need to pass in terminfo dirs:

overrideCabal fooBar (old: {
  configureFlags = (old.configureFlags or [ ]) ++
    let
      # Until https://github.com/NixOS/nixpkgs/pull/311411
      terminfoDirs = final.lib.concatStringsSep ":" [
        "/etc/terminfo" # Debian, Fedora, Gentoo
        "/lib/terminfo" # Debian
        "/usr/share/terminfo" # upstream default, probably all FHS-based distros
        "/run/current-system/sw/share/terminfo" # NixOS
      ];
      staticNcurses = (
        (final.ncurses.override {
          enableStatic = true;
        })
      ).overrideAttrs
        (old: {
          configureFlags = (old.configureFlags or [ ]) ++ [
            "--with-terminfo-dirs=${terminfoDirs}"
          ];
        });
    in [
    "--ghc-option=-optl=-static"
    "--extra-lib-dirs=${final.ncurses.override { enableStatic = true; }}/lib"
  ];
})

Sqlite

If you see

ld: cannot find -lsqlite3: No such file or directory

You may think you need to pass in the dontDisableStatic = true version of sqlite using --extra-lib-dirs. However, you'll just see the same error if you do that. What you need to do instead is to override sqlite globally:

sqlite = prev.sqlite.overrideAttrs (old: { dontDisableStatic = true; })

Text ICU

If you see this error:

ld: cannot find -licuuc: No such file or directory
ld: cannot find -licui18n: No such file or directory
ld: cannot find -licuuc: No such file or directory
ld: cannot find -licudata: No such file or directory

You'll need a version of icu with static linking enabled:

let
  # Until https://github.com/NixOS/nixpkgs/pull/304772
  icuWithStatic = final.icu.overrideAttrs
    (old: {
      dontDisableStatic = true;
      configureFlags = (old.configureFlags or "") ++ [ "--enable-static" ];
      outputs = old.outputs ++ [ "static" ];
      postInstall = ''                   
        mkdir -p $static/lib             
        mv -v lib/*.a $static/lib        
      '' + (old.postInstall or "");
    });
in

... and mention it to cabal:

overrideCabal fooBar (old: {
  configureFlags = (old.configureFlags or [ ]) ++ [
    "--ghc-option=-optl=-static"
    "--extra-lib-dirs=${icuWithStatic.static}/lib"
    "--ghc-option=-optl=-licui18n"
    "--ghc-option=-optl=-licuio"
    "--ghc-option=-optl=-licuuc"
    "--ghc-option=-optl=-licudata"
    "--ghc-option=-optl=-ldl"
    "--ghc-option=-optl=-lm"
    "--ghc-option=-optl=-lstdc++"
  ];
})

Thank you 4e6 for figuring this out years ago!

Webp

If you see this error:

ld: cannot find -lwebp: No such file or directory

You may need the dontDisableStatic = true version of libwebp.

If you see this error:

ld: (.text+0xcf6): undefined reference to `SharpYuvGetConversionMatrix'
ld: (.text+0xd41): undefined reference to `SharpYuvConvert'

You'll also need -lsharpyuv:

overrideCabal fooBar (old: {
  configureFlags = (old.configureFlags or [ ]) ++ [
    "--ghc-option=-optl=-static"
    "--extra-lib-dirs=${final.libwebp.overrideAttrs (old: { dontDisableStatic = true; })}/lib"
    "--ghc-option=-optl=-lsharpyuv"
  ];
})

Elf references

If you see errors like this:

ld: (.text+0x911): undefined reference to `gelf_getehdr'
ld: (.text+0x934): undefined reference to `elf_getshdrstrndx'
ld: (.text+0x977): undefined reference to `elf_nextscn'
ld: (.text+0x98e): undefined reference to `gelf_getshdr'
ld: (.text+0x9a5): undefined reference to `elf_strptr'
ld: (.text+0x9f3): undefined reference to `elf_rawdata'
ld: (.text+0xa44): undefined reference to `elf_rawdata'
ld: (.text+0xb69): undefined reference to `elf_getphdrnum'
ld: (.text+0xba4): undefined reference to `gelf_getphdr'
ld: (.text+0xbf4): undefined reference to `elf_getdata_rawchunk'
ld: (.text+0xcdf): undefined reference to `elf_rawfile'
ld: (.text+0xd0f): undefined reference to `elf_getdata_rawchunk'

you need the same fix as in this next section:

Template Haskell

If you see this error again, double-check if you are using Template Haskell. (Remember, this is the same as the first error we encountered.)

relocation R_X86_64_32 against hidden symbol `__TMC_END__' can not be used when making a shared object
ld: failed to set dynamic section sizes: bad value

If so, you need to override ghc in the haskellPackages part of your overlay:

let
  fixGHC = pkg: pkg.override {    
    enableRelocatedStaticLibs = true;
    enableShared = false;
    enableDwarf = false;
  };
in {
 ghc = fixGHC super.ghc;  
 buildHaskellPackages = old.buildHaskellPackages.override (oldBuildHaskellPackages: {
   ghc = fixGHC oldBuildHaskellPackages.ghc;                                     
 });
}

Webdriver tests

Building big projects like Firefox or Chromium can prove difficult. For this reason, I've not figured out how to run webdriver tests in a pkgsMusl context. You may be able to pass in a regular (non-pkgsMusl) version of the browsers instead.

Conclusion

There is a very good likelihood that it is entirely possible to statically link your Haskell executable these days. The work that nh2 et al. have done on Fully static Haskell executables has made this all possible.

Getting your Haskell executable statically linked with Nix

Why statically link?

Why is this so complicated?

Libc

Template Haskell

Non-Haskell dependencies

Getting started with your own project

Fail with a static check

Linking against musl

Statically linking

Enable static linking for dependencies

Common errors

Zlib

Terminfo

Sqlite

Text ICU

Webp

Elf references

Template Haskell

Webdriver tests

Conclusion

Linking against `musl`