How to put your /nix directory on a separate device

Date 2019-09-14

In this Tutorial, we explain why and how to put your /nix directory on a separate drive than the one where / is located. This might seem simple to some, but there are some caveats and there is no abundance of clear guides on using nix.

Why

If you've set up your machine like I have, it looks something like this:

$ lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
nvme0n1     259:0    0   477G  0 disk 
├─nvme0n1p1 259:1    0   550M  0 part /boot
├─nvme0n1p2 259:2    0   128G  0 part /
├─nvme0n1p3 259:3    0    32G  0 part [SWAP]
└─nvme0n1p4 259:4    0 316.4G  0 part /home

You have a /boot partition because you use UEFI, a root (/) partition that is too small (or too big because you've run into a problem with having too small of a root partition before), a /home partition and optionally a swap partition.

This means that your /nix directory resides on the root (/) partition. Because your /nix directory tends to become bigger as time goes on, and any space on your root (/) partition cannot be used for your /home partition, you may want to put the /nix directory on its own partition.

How

Disclaimer: I use Arch Linux, adapt the guide for your own setup as needed.

In my case, I added an extra HDD to my setup when I built my computer, for extra storage. This means that my setup looks something like this:

$ lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda           8:0    0   1.8T  0 disk 
└─sda1        8:1    0   1.8T  0 part /nix
nvme0n1     259:0    0   477G  0 disk 
├─nvme0n1p1 259:1    0   550M  0 part /boot
├─nvme0n1p2 259:2    0   128G  0 part /
├─nvme0n1p3 259:3    0    32G  0 part [SWAP]
└─nvme0n1p4 259:4    0 316.4G  0 part /home

I have a nice big HDD at /dev/sda and partitioned it with a single partition: /dev/sda1. I will explain how I put my /nix directory on it, but if you are following along, make sure that you choose the right device and partition for yourself. On many setups, /dev/sda is the device that hosts your operating system, and you do not want to mess that up.

First we need to choose a filesystem for the partition. I chose ext4:

mkfs.ext4 /dev/sda1

Next, we will want to mount it somewhere else, like at /media/hdda so we can prepare the drive.

mkdir /media/hdda
mount /dev/sda1 /media/hdda

We can't just mount /dev/sda1 at /nix because then the files that are currently in /nix will be hidden and not put on the partition.

We want to copy our entire /nix directory into /media/hdda so that the two directories are equal. First we need to stop the nix daemon to make sure that the nix database is flushed:

systemctl stop nix-daemon.socket

Then we can copy the current /nix directory. We need to make sure that the symlinks are preserved with --links and that directories are copied recursively with --recursive.

rsync --recursive --links /nix/ /media/hdda/ # Don't forget the trailing slashes

Now check that /media/hdda has the contents of your /nix directory and then unmount that device.

umount /media/hdda

Now comes the dangerous part. We will remove the current /nix directory to prepare it for mounting:

rm -rf /nix

Then we have to remake it to be able to mount anything on it, and mount it:

mkdir /nix
mount /dev/sda1 /nix

Now the /nix directory should be mounted correctly.

You might see some warnings like these:

WARNING: bad ownership on /nix/var/nix/profiles/per-user/syd
WARNING: bad ownership on /nix/var/nix/gcroots/per-user/syd
Nix: WARNING: bad ownership on /nix/var/nix/profiles/per-user/syd, should be 1000
Nix: WARNING: bad ownership on /nix/var/nix/gcroots/per-user/syd, should be 1000
Nix: WARNING: bad ownership on /nix/var/nix/profiles/per-user/syd, should be 1000
Nix: WARNING: bad ownership on /nix/var/nix/gcroots/per-user/syd, should be 1000

To fix them, we have to re-do some ownership:

sudo chown -R 1000 /nix/var/nix/gcroots/per-user/syd /nix/var/nix/profiles/per-user/syd
# Do that for each of the users that you get warnings for.

The last step is to make sure that this mounting happens automatically on boot. First we need to figure out what the UUID of the device is:

lsblk -fs
NAME      FSTYPE LABEL UUID                                 FSAVAIL FSUSE% MOUNTPOINT
sda1      ext4        273e9ea9-ade1-4df3-8358-94a3e9c91fae    1.7T     7% /nix
└─sda                                                                      
...

Then we put the following line in /etc/fstab:

# /dev/sda1
UUID=273e9ea9-ade1-4df3-8358-94a3e9c91fae	/nix     	ext4      	rw,relatime	0 2

Now reboot to try it out. If anything goes wrong during boot, just comment out this line and reboot. You can then run journalctl -u nix.mount to figure out what went wrong. Most likely you'll have messed up something in the line you added, like the UUID or the filesystem that you used.

Previous
Hacktoberfest

Looking for a lead engineer?

Hire me
Next
Millismos: Writing a simple forest-editor with brick.