Raspberry Pi 4
If you’re worried about destroying your SD Cards, need some more space, or just want to learn something new, I’m going to show you how to use an NFS root for the Raspberry Pi 4.
When you use an NFS Root with your Raspberry Pi, it stores the entire root filesystem on a remote NFS export (think of it as a network filesystem share). This means you’ll have as much space as the NFS export, and you’ll probably see way faster performance since it’ll be running at 1Gb/sec instead of the speed of the SD Card.
This also protects your SD card, as the majority of the reading and writing is performed on the physical storage of the NFS export, instead of the SD card in the Pi which has limited reads and writes.
To get started, you’ll need:
There are multiple ways to do this, but I’m providing instructions on the easiest way it was for me to do this with the resources I had immediately available.
To boot your Raspberry Pi 4 from an NFS root, multiple steps are involved. Below you’ll find the summary, and further down you’ll find the full instructions. You can click on an item below to go directly to the section.
The process:
See below for the individual instructions for each step.
First, we need to write the SD Card Linux image to your SD card. You’ll need to know which device your SD card will appear to your computer. In my case it was /dev/sdb, make sure you verify the right device or you could damage your current Linux install.
dd if=imagename.img of=/dev/sdb bs=4M
You now have an SD Card Linux install for your Raspberry Pi. We will later modify and then copy this to the NFS root and boot SD card.
In this step, we’re going to create a bootable SD card that contains the Linux kernel and other needed files for the Raspberry Pi to boot.
This card will be installed in the Pi, load the kernel, and then kick off the boot process to load the NFS root.
I previously created a post to create a boot partition layout for a Raspberry Pi. Please follow those instructions to complete this step.
Later on in this guide, you’ll be copying the boot partition from the SD Card Linux image, on to this newly created boot SD Card for the NFS Root.
There’s a few things we have to do to prep the Ubuntu or Raspbian Linux install to be usable as an NFS Root.
apt install nfs-common
iface eth0 inet manual
For step 6, we need to modify the /etc/fstab entry for the root fs. It is different depending on whether you’re using Ubuntu or Raspbian.
For Raspbian, your /etc/fstab should look like this:
proc /proc proc defaults 0 0 LABEL=boot /boot vfat defaults 0 2 NFS-SERVER-IP:/nfs-export/PI-Raspbian / nfs defaults 0 0
For Ubuntu, your /etc/fstab should look like this:
LABEL=system-boot /boot/firmware vfat defaults 0 2 /dev/nfs / nfs defaults 0 0
After you do this, the Linux SD image may not boot again if directly installed in the Raspberry Pi, so make sure you’ve made the proper modifications before powering it down.
In my case I used a Synology DS1813+ as an NFS server to host my Raspberry Pi NFS root images. But you can use any Linux server to host it.
If you’re using a synology disk station, create a shared folder, disable the recycling bin, leave everything else default. Head over to the “NFS Permissions” tab and create an ACL entry for your PI and workstations. You can also add a network segment for your entire network (ex. 192.168.0.0/24″) instead of specifying individual IPs.
Once you create an entry, it’ll look like this. Note the “Mount path” in the lower part of the window.
Now, if you’re using a standard Linux server the steps are different.
apt install nfs-kernel-server
mkdir /nfs-export/
mkdir /nfs-export/PI-ImageName
/nfs-export/PI-ImageName IPorNetworkRange(rw,no_root_squash,async,insecure)
exportfs -ra
Take note of the mount point and/or NFS export path, as this is the directory your Raspberry Pi will need to mount to access it’s NFS root. This is also the directory you will be copying your SD Card Linux install root FS to.
When you’re ready to copy your SD Card Linux install to your NFS Export, you’ll need to do the following. In my case I’ll be using an Ubuntu desktop computer to perform these steps.
When I insert the SD Card containing the Raspberry Pi Linux image, it appeared as /dev/sdb on my system. Please make sure you are using your proper device names to avoid using the wrong one to avoid writing or using the wrong disk.
Instructions to copy the root fs from the SD card to the NFS root export:
mount /dev/sdb2 old/
mount IPADDRESS:/nfs-export/PI-ImageName nfs/
rsync -avxHAXS --numeric-ids --info=progress2 --progress old/ nfs/
umount old/
umount nfs/
Once this is complete, your OS root is now copied to the NFS root.
First we have to copy the boot partition from the SD Card Linux install to the boot SD card, then we need to modify the contents of the new boot SD card.
Top copy the boot files, follow these instructions.
mount /dev/sdb1 old/
mount /dev/sdc1 new/
rsync -avxHAXS --numeric-ids --info=progress2 --progress old/ new/
umount old/
umount new/
Now there are few steps we have to take to make to the boot SD card boot to an NFS Root.
We have to make a modification to the PI boot command. It is different depending on which Linux image (Ubuntu or Raspbian) you’re using.
First, insert the boot SD card, and mount it to a temporary directory.
mount /dev/sdc1 new/
If you’re running Ubuntu, your existing nobtcmd.txt should look like this:
dwc_otg.lpm_enable=0 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline rootwait
We’ll modify and replace some text to make it look like this. Don’t forget to change the command to reflect your IP and directory:
dwc_otg.lpm_enable=0 console=tty1 root=/dev/nfs nfsroot=IPADDRESS:/nfs-export/PI-Ubuntu,tcp,rw ip=dhcp rootfstype=nfs elevator=deadline rootwait
For Raspbian, your existing cmdline.txt should look like this:
console=serial0,115200 console=tty1 root=PARTUUID=97709164-02 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait
We’ll modify and replace some text to make it look like this. Don’t forget to change the command to reflect your IP and directory:
console=serial0,115200 console=tty1 root=/dev/nfs nfsroot=IPADDRESS:/nfs-export/PI-Raspbian,tcp rw vers=3 ip=dhcp rootfstype=nfs elevator=deadline rootwait
Once you make the modifications, save the file and unmount the SD card.
Your SD card is now ready to boot.
At this point, insert the boot SD Card in your Raspberry Pi and attempt to boot. All should be working now and it should boot and use the NFS root!
If you’re having issues, if the boot process stalls, or something doesn’t work right, look back and confirm you followed all the steps above properly.
You’re now complete and have a fully working NFS root for your Raspberry Pi. You’ll no longer worry about storage, have high speed access to it, and you’ll have some new skills!
And don’t forget to check out these Handy Tips, Tricks, and Commands for the Raspberry Pi 4!
While most of us frequently deploy new ESXi hosts, a question and task not oftenly discussed is how to properly decommission a VMware ESXi host. Some might be surprised to… Read More
This guide will outline the instructions to Disable the VMware Horizon Session Bar. These instructions can be used to disable the Horizon Session Bar (also known as the Horizon Client… Read More
Normally, any VMs that are NVIDIA vGPU enabled have to be manually migrated with manual vMotion if a host is placed in to maintenance mode, to evacuate the host. While… Read More
You may experience GPU issues with the VMware Horizon Indirect Display Driver in your environment when using 3rd party applications which incorrectly utilize the incorrect display adapter. This results with… Read More
Today we're going to cover a powerful little NAS being used with VMware; the Synology DS923+ VMware vSphere Use case and Configuration. This little (but powerful) NAS is perfect for… Read More
Today we'll go over how to install the vSphere vCenter Root Certificate on your client system. Certificates are designed to verify the identity of the systems, software, and/or resources we… Read More
View Comments
Thanks for your information.
I can boot 'nfs root filesystem' w/ my Raspberry Pi 3B+.
I will try to do no-SD card boot using tftp next.
Best Regards,
Hi Hisaaki,
Glad to hear it works! Feel free to post your findings once you try the TFTP boot! :)
Cheers,
Stephen
Hi,
trying this on a RPi 2, I found that the cmdline for raspbian needs to be modified:
the vers=3 need to be appended as an nfsroot= option like tcp, like this:
nfsroot=:/path/to/nfs/root,tcp,vers=3
Otherwise, the kernel panics (unable to mount root).
The rest works fine, thanks for the writeup.
Hi, was able to set up NFS root this way, Pi boots, however apt update fails with following message:
pi@raspberrypi:~ $ sudo apt-get update
Get:1 http://archive.raspberrypi.org/debian buster InRelease [32.8 kB]
Get:2 http://raspbian.raspberrypi.org/raspbian buster InRelease [15.0 kB]
Reading package lists... Done
E: Unable to determine file size for fd 9 - fstat (5: Input/output error)
E: Problem closing the file - close (116: Stale file handle)
Hi Adam,
It sounds like there may be a permission issue.
This is either due to the NFS server, an incorrect rootfs copy, or incorrect mount.
I'd check your config and try again.
Cheers
Hi Stephen,
Indeed, I use QNAP and raspberry was mounting NFS version 1 by default. I solved it with adding vers=3 at the end of nfsroot parameter, hence
nfsroot=IP_ADDRESS:/path,tcp,vers=3
Thanks for the quide and help.
Cheers!
Thank You! I've learned few things today. Now I understand that it's all about root folder being moved to NFS share and the boot remaing on the SD.
Your solution is more convinient for those like me who cannot install dnsmasq/tftp on the NFS server.
The only thing I need right now is to find cheap SDs
Hi Mr. Stephen, First of all, thanks for your time and effort. This is quite an amazing tutorial for newbies like me :)
Although the tutorial is detailed sufficiently, I've faced some issues while trying to apply it on my Raspberry Pi 4. Here are some points I want to add:
When I edit the /etc/exports file and apply the changes via
exportfs -racommand, an error occurs. The output is as follows:The solution I found is to replace the appended line with the following:
This way, I was able to run the
exportfs -racommand without any problem.I also used restarted the nfs-kernel-server using the command below.The cmdline.txt file which includes initial kernel arguments needs to be changed a bit. I couldn't achieve to bring up my board until I change the nfsroot parameter as follows
I guess that the problem was about the
nfsversthing. You may want to enlighten us about it.I also added nfsrootdebug option to kernel parameter to further inspect the debug information about the process.
Thanks again for your time and effort. The tutorial was absolutely beneficial!
Greetings from Turkey, Caglayan
Hi Caglayan DOKME,
Thanks for posting and leaving the comment! I'm sure it will help someone else. It's normal to have to tweak the version numbers around depending on what distribution is being used, the NFS version, etc.
Glad you got it working!
Cheers,
Stephen
How did you configure Linux ACLs via NFS? I am not referring to the shared folder permissions that you are referring to as an ACL, I am referring to POSIX-compliant ACLs that allow the kernel to write to /var/log/journal, or use tools like dmesg. I get errors like:
>dmesg: read kernel buffer failed: Operation not permitted
or:
>"Failed to set ACL on /var/log/journal/UUID/user-1000.journal, ignoring: Operation not supported."
This guide seems not to acknowledge the following issues regarding Linux ACLs, and Synology's ACL setup (As of July 2022, but seem to pre-date this article)
* rsync being broken in Ubuntu/Debian for this very issue: https://unix.stackexchange.com/questions/12203/rsync-failed-to-set-permissions-on-error-with-rsync-a-or-p-option
* Synology not supporting Linux ACLs over NFS (v3/v4/v4.1):
https://community.synology.com/enu/forum/1/post/150225
https://community.synology.com/enu/forum/1/post/149472
The rpi4 boots to the root fs via NFS, but without logging to the journal, or having random sys utils not being able to run, it's too unsafe to run any type of workload over NFS. I'd love to understand what you did here, because I feel like there's a gap in documentation from this article.
Hi Chris,
The NFS export has to be squashed as root. This allows the connecting system attempting an NFS root to give full access to the kernel, etc... From that point on, the users only have access to what they would normally.
Cheers,
Stephen
Update Wrong stack link to rsync issue, correct one is here: https://serverfault.com/questions/1059618/rsync-generator-failed-to-set-permissions-operation-not-supported-95
Hi Stephen
The synology screenshot in the tutorial shows no mapping for squash, is that an error? Nevertheless, the synology shared folders do not have "squash as root" as an option, rather to map root to admin or guest. In all cases, I still can't get proper ACL support in the filesystem. I suspect others are hitting this issue, and are unaware of it, as the boot errors regarding ACLs are consistent during init.
I was able to loopback mount an img file I created with some success, but there is some very strange behavior in which (I believe) the NFS page cache refuses to acknowledge multiple discrete mounts, causing (in my case, 6 raspberry pis) to all route to the same NFS mount, even though each pi is reporting that it is using separate, unique NFS mounts, both in each Pi's fstab, and cmdline.txt. This also results in incredibly poor performance and is generally not recommended (see: https://www.suse.com/support/kb/doc/?id=000017812)
At this point, I'm ready to give up on this endeavor, sadly. I spent a couple of months working/resolving corner cases with Unifi equipment to try PXE/TFTP booting; the boot partition + NFS root filesystem was my backup. Looks like my choices are to roll my own NFS server (no thanks), or USB mount an SSD or two, which is orthogonal to the whole concept + benefit of networked filesystems.
Appreciate the help,
Christian
I agree with Chris about NSFv4 ACLs, the nfs4-acl-tools may work on other Linux systems, but as far as I’ve experimented with them, not on Raspberry Pi OS built off Debian Bullseye. If you have a share with either a Raspberry Pi server, or something else (In my experiments a Solaris 11.4 machine), the tools can’t change the NSFv4 ACLs on the client. I don’t fully understand what you’re doing here by putting the OS on an NFS server, I mean it looks and sounds great, but are you forcing the OS to be of file system type NFS, rather than EXT4? I couldn’t get the nfs4-acl-tools to work when the server was an NFS file system (Solaris 11.4). Perhaps many people are not that interested in levying NSFv4 ACLs on their files and directories, I don’t know. But if you needed big iron servers, maybe that would be important to you. I’m only toying around on a couple three Pi’s.