CS SYD - The ci.nix pattern

Having set up CI for dozens of repositories, Nix and otherwise, I think I've stumbled upon some best practices. The following describes a pattern that includes all those practices.

In a nutshell

Have a ci.nix file in the root of your repository that evaluates to an attribute set.
Have developers run nix-build ci.nix before they submit a pull request.
Run nix-build ci.nix on CI as your CI
Optionally: Run nix-build ci.nix -A attr for each attr in the attribute set to speed up CI if the jobs are fairly independent.

Why it works

Setting up CI is a nightmare because of how long the feedback loop is. You can only know if your setup works after you've made a commit, pushed, and waited for the slow CI machines to queue and run your job. Consequently, the best CI setups are the ones where the part that is unique to the CI machine is the smallest.

Using Nix, the only thing the CI needs to know about is how to run nix-build. This way, as much as possible is shared between a dev machine and a CI machine. (There is still a possibility that nix-build ci.nix succeeds locally but fails remotely, but these instances are luckily rarer than with other CI setup.) As a result, setting up CI for this pattern is much easier than usual.

There are some nice side-effect of this pattern:

Developers can check whether CI will (probably) pass before they have to push any commits.
CI results can be cached easily using a nix cache like cachix.
To push CI through quickly, a developer can push the result of bullding ci.nix to their cache so that CI doesn't even have to build anything.

Smos' example for Haskell

The Smos CI uses the ci.nix pattern. The ci.nix file evaluates to 7 different attributes which correspond to 3 different CI jobs.

The pre-commit check checks that the pre-commit hooks have been run.
The normal build makes sure that all builds succeed and tests pass.
The static build does the same for statically linked executables.
The NixOS module test ensures that the nixos and nix-home-manager modules work together nicely in the ultimate integration test.