Foreword

Recent improvements in the Raspberry Pi EEPROM has made it possible to boot them completely online, this method currently requires access to the Pi and human interaction. What I’m describing here is a way to image Pis with zero access or interaction required. It uses the network booting mechanism which has been available in the EEPROM for a while. You’ll be right at home if you know PXE. It is meant for tighter controlled local networks, not distributing images online.

In the interest of clarity, I’ll focus on the bare minimum necessary to the imaging process. Where I use this professionally, it is wrapped with more logic to have fleets of Pis detect when new images are published, and report to Slack when they go for reimaging.

What you Need

Simply 2 Pis on the same network. One is the Netboot Server providing images to other Pis on the network, the other is the Pi to image which will ask the Netboot Server for an image and write it to its own SD card. The Netboot Server does not have to be another Pi, it requires not special Pi sauce, but I like to throw Pis at all my hosting needs. The Pi to image on the other hand does need to be a Pi as well, we are imaging Pis here.

Setting up the NetBoot Server

The server can be any recent vanilla RaspiOS. Copy the following script to it under /home/pi/script.sh

#!/bin/bash

# ascii text from https://patorjk.com/software/taag/#p=display&f=Standard

if [ -z "$1" ]
then
    echo "ERROR: I need one argument: the raspi image file"
    exit 1
fi

if [ ! -f "$1" ]; then
	echo "ERROR: Image file $1 doesn't exist"
	exit 1
fi

echo "> installing needed packages"
apt-get update > /dev/null
apt-get install -y bc nfs-kernel-server tftpd-hpa apache2 php php-curl php-xml libapache2-mod-php ipcalc curl wget screen lsof > /dev/null

myip=`ifconfig | grep -Eo 'inet (addr:)?([0-9]*\.){3}[0-9]*' | grep -Eo '([0-9]*\.){3}[0-9]*' | grep -v '127.0.0.1'`
mymask=`ifconfig | grep $myip | grep -Eo 'netmask (addr:)?([0-9]*\.){3}[0-9]*' | grep -Eo '([0-9]*\.){3}[0-9]*' | grep -v '127.0.0.1'`
mynetwork=`ipcalc $myip/$mymask | grep "Network:" | sed 's/\s\{1,\}/ /g' | cut -d' ' -f2`
echo "> my ip: $myip"
echo "> my network: $mynetwork"

echo "> starting services"
echo ">   tftp"
service tftpd-hpa start
echo ">   nfs"
service nfs-kernel-server start
echo ">   web"
service apache2 start

echo "> clean slate"
rm -rf /srv/tftp/*
rm -rf /srv/nfs/*
rm -rf /var/www/html/*

echo "> installing web components"
echo ">   bootstrap.sh"
#  _                 _       _                        _
# | |__   ___   ___ | |_ ___| |_ _ __ __ _ _ __   ___| |__
# | '_ \ / _ \ / _ \| __/ __| __| '__/ _` | '_ \ / __| '_ \
# | |_) | (_) | (_) | |_\__ \ |_| | | (_| | |_) |\__ \ | | |
# |_.__/ \___/ \___/ \__|___/\__|_|  \__,_| .__(_)___/_| |_|
#                                         |_|
cat << EOF > /var/www/html/bootstrap.sh
disk_count=\`lsblk -d -l -p -n | cut -d' ' -f1 | wc -l\`
if [ \$disk_count -ne 1 ]
then
	echo "ERROR: I need to have exactly 1 disk to write to"
	exit 1
fi

echo "> writing image to disk"
disk=\`lsblk -d -l -p -n | cut -d' ' -f1\`
dd if=/tmp/img of=\$disk bs=1M status=progress

if [ \$? -ne 0 ]
then
	echo "> ERROR: writing image to disk failed"
	echo ">   rebooting in 300 seconds"
	sleep 300
	reboot
	exit 1
fi

echo "> rebooting in 10 seconds"
sleep 10
wget -qO- http://${myip}/reboot.php &> /dev/null
reboot
EOF


#  _           _                   _
# (_)_ __   __| | _____  __  _ __ | |__  _ __
# | | '_ \ / _` |/ _ \ \/ / | '_ \| '_ \| '_ \
# | | | | | (_| |  __/>  < _| |_) | | | | |_) |
# |_|_| |_|\__,_|\___/_/\_(_) .__/|_| |_| .__/
#                           |_|         |_|
#
echo ">   index.php"
cat << EOF > /var/www/html/index.php
<?php

echo "Hi!" ;
exit( 0 ) ;

?>
EOF


#  _                         _
# (_)_ __ ___   __ _   _ __ | |__  _ __
# | | '_ ` _ \ / _` | | '_ \| '_ \| '_ \
# | | | | | | | (_| |_| |_) | | | | |_) |
# |_|_| |_| |_|\__, (_) .__/|_| |_| .__/
#              |___/  |_|         |_|
echo ">   img.php"
cat << EOF > /var/www/html/img.php
<?php

header( "Cache-Control: no-store, no-cache, must-revalidate" ) ;
header( "Cache-Control: post-check=0, pre-check=0", false ) ;
header( "Pragma: no-cache" ) ;
header( "Expires: ".gmdate("D, d M Y H:i:s", mktime(date("H")+2, date("i"), date("s"), date("m"), date("d"), date("Y")))." GMT" ) ;
header( "Last-Modified: ".gmdate("D, d M Y H:i:s")." GMT" ) ;
header( "Content-Type: application/octet-stream" ) ;
header( "Content-Length: ".(string)(filesize("/var/img")) ) ;
header( "Content-Transfer-Encoding: binary\n" ) ;

\$handle = fopen( "/var/img", "rb" ) ;
if( \$handle===false ) {
    echo "can't read image" ;
    exit( 1 ) ;
}

while( !feof(\$handle) ) {
    echo fread( \$handle, 8192 ) ;
}
fclose( \$handle ) ;

?>
EOF


#           _                 _           _
#  _ __ ___| |__   ___   ___ | |_   _ __ | |__  _ __
# | '__/ _ \ '_ \ / _ \ / _ \| __| | '_ \| '_ \| '_ \
# | | |  __/ |_) | (_) | (_) | |_ _| |_) | | | | |_) |
# |_|  \___|_.__/ \___/ \___/ \__(_) .__/|_| |_| .__/
#                                  |_|         |_|
echo ">   reboot.php"
cat << EOF > /var/www/html/reboot.php
<?php

echo shell_exec( "sudo /usr/local/bin/enable_eeprom_sdboot.sh 2>&1; screen -S disable_eeprom_sdboot -dm sh -c \"sleep 240; sudo /usr/local/bin/disable_eeprom_sdboot.sh\"" ) ;

?>
EOF

echo "> dissecting image file"
offset=`fdisk -l $1 | grep Linux | tr -s ' ' | tr '\t' ' ' | cut -d' ' -f2`
real_linux_offset=`echo "$offset*512" | bc`
offset=`fdisk -l $1 | grep W95 | tr -s ' ' | tr '\t' ' ' | cut -d' ' -f2`
real_boot_offset=`echo "$offset*512" | bc`

mkdir /tmp/boot 2>/dev/null
mount -o loop,offset=$real_boot_offset $1 /tmp/boot
rm -rf /srv/tftp/*
cp -rpf /tmp/boot/* /srv/tftp/
umount /tmp/boot

mkdir /tmp/root 2>/dev/null
mount -o loop,offset=$real_linux_offset $1 /tmp/root
mkdir /srv/nfs 2>/dev/null
cp -rpf /tmp/root/* /srv/nfs/

echo "> getting sdboot eeprom ready"
cat << EOF > /tmp/eeprom_config.sdboot
[all]
DHCP_TIMEOUT=45000
DHCP_REQ_TIMEOUT=4000
TFTP_FILE_TIMEOUT=30000
TFTP_IP=${myip}
TFTP_PREFIX=1
TFTP_PREFIX_STR=
BOOT_ORDER=0xf21
ENABLE_SELF_UPDATE=1
SD_BOOT_MAX_RETRIES=3
NET_BOOT_MAX_RETRIES=2
EOF

latest_eeprom=`ls -1 /tmp/root/lib/firmware/raspberrypi/bootloader/stable/pieeprom-*.bin | sort -u | tail -1`
cp $latest_eeprom /tmp/pieeprom.bin
rpi-eeprom-config --out /tmp/pieeprom-out.bin --config /tmp/eeprom_config.sdboot /tmp/pieeprom.bin && rpi-eeprom-update -d -f /tmp/pieeprom-out.bin
mv /boot/pieeprom.sig /srv/tftp/pieeprom.sig.inert
mv /boot/pieeprom.upd /srv/tftp/pieeprom.upd.inert
mv /boot/recovery.bin /srv/tftp/recovery.bin.inert

cat << EOF > /usr/local/bin/enable_eeprom_sdboot.sh
sed -i "s/ts:.*/ts: \`date +%s\`/g" /srv/tftp/pieeprom.sig.inert
mv /srv/tftp/pieeprom.sig.inert /srv/tftp/pieeprom.sig
mv /srv/tftp/pieeprom.upd.inert /srv/tftp/pieeprom.upd
mv /srv/tftp/recovery.bin.inert /srv/tftp/recovery.bin
EOF
chmod 755 /usr/local/bin/enable_eeprom_sdboot.sh

cat << EOF > /usr/local/bin/disable_eeprom_sdboot.sh
mv /srv/tftp/pieeprom.sig /srv/tftp/pieeprom.sig.inert
mv /srv/tftp/pieeprom.upd /srv/tftp/pieeprom.upd.inert
mv /srv/tftp/recovery.bin /srv/tftp/recovery.bin.inert
EOF
chmod 755 /usr/local/bin/disable_eeprom_sdboot.sh

cat << EOF > /etc/sudoers.d/010_www-data_eeprom
www-data	ALL=(ALL) NOPASSWD: /usr/local/bin/enable_eeprom_sdboot.sh
www-data	ALL=(ALL) NOPASSWD: /usr/local/bin/disable_eeprom_sdboot.sh
EOF

umount /tmp/root

echo "/srv/nfs    $mynetwork(rw,sync,no_subtree_check,no_root_squash)" > /etc/exports
exportfs -rav

echo "console=serial0,115200 console=tty1 root=/dev/nfs nfsroot="${myip}":/srv/nfs,nfsvers=3 ip=dhcp rw elevator=deadline fsck.repair=yes rootwait" > /srv/tftp/cmdline.txt

echo "proc /proc proc defaults 0 0" > /srv/nfs/etc/fstab

echo "> deploying image"
if [ ! -f /var/img ]; then
	cp -f $1 /var/img
else
	cp -f $1 /var/img.new
	if [ "`lsof | grep "/var/img" | wc -l`" -ne 0 ]; then
		echo "> waiting for file handle release on /var/img"
		while [ "`lsof | grep "/var/img" | wc -l`" -ne 0 ]; do
			echo -n "."
			sleep 1
		done
		echo ""
	fi
	mv /var/img /var/img.old
	mv /var/img.new /var/img
	rm /var/img.old
fi

echo "> adding web bootstrap"
echo '#!/bin/sh -e' > /srv/nfs/etc/rc.local
echo "echo \">   disabling cron\"" >> /srv/nfs/etc/rc.local
# I've seen this command hang several times before
echo "/usr/bin/timeout --kill-after=10s 5s /usr/sbin/service cron stop || :" >> /srv/nfs/etc/rc.local
echo "echo \">   sleeping until we have connectivity\"" >> /srv/nfs/etc/rc.local
echo "bash -c 'count=0; res=1; while [ \$res -ne 0 -a \$count -lt 120 ]; do echo "."; wget -q --spider http://${myip}; res=\$?; sleep 1; count=\$((count+1)); done'" >> /srv/nfs/etc/rc.local
echo "echo \">   retrieving imaging image\"" >> /srv/nfs/etc/rc.local
echo "curl -s http://${myip}/img.php --output /tmp/img" >> /srv/nfs/etc/rc.local
echo "echo \">   launching bootstraping process\"" >> /srv/nfs/etc/rc.local
echo "curl -s http://${myip}/bootstrap.sh --output /tmp/bootstrap.sh" >> /srv/nfs/etc/rc.local
echo "bash /tmp/bootstrap.sh" >> /srv/nfs/etc/rc.local

echo "> all done"

Make is executable with:

chmod 755 /home/pi/script.sh

Download a RaspiOS image, the one you want to serve out to the Pi to image, and copy it on the server under /home/pi.

Run the script with the RaspiOS image as the argument:

sudo /home/pi/script.sh /home/pi/2022-01-28-raspios-bullseye-armhf-lite.img

This can take a while depending on the speed of your SD card. Note that our NetBoot Server‘s IP is 192.168.1.116.

Sending the Pi to image for reimaging

With your server ready, you can instruct the Pi to image to go reimage itself by asking it to network boot from 192.168.1.116. The following script will update its EEPROM to this effect:

#!/bin/bash

cat << EOF > /tmp/eeprom_config.netboot
[all]
DHCP_TIMEOUT=45000
DHCP_REQ_TIMEOUT=4000
TFTP_FILE_TIMEOUT=30000
TFTP_IP=192.168.1.116
TFTP_PREFIX=1
TFTP_PREFIX_STR=
BOOT_ORDER=0xf12
ENABLE_SELF_UPDATE=1
SD_BOOT_MAX_RETRIES=3
NET_BOOT_MAX_RETRIES=2
EOF

/usr/bin/rpi-eeprom-config --apply /tmp/eeprom_config.netboot
sleep 1
/usr/sbin/reboot

The 192.168.1.116 IP is the only thing you need to change. Copy it to the Pi to image as reimage.sh for example, make it executable and run it.

Working Principle

The script on the NetBoot Server grabs a RaspiOS image and serves it via NFS. It also runs a TFTP server pointing to the NFS share. You could run your Pis entirely off the network and disregard the SD card, however I think it makes more sense for most purposes to simply burn that image locally on the Pi’s SD card and run off of it. The image has added instructions in /etc/rc.local for doing just that, retrieving the original RaspiOS image via HTTP, and burning it to the SD card. Having burnt it, it reboots, but right before it tells the NetBoot Server that it is rebooting. This is because for a brief moment, the NetBoot Server will have to catch that Pi rebooting and ask it to go back to booting from its newly burnt SD card.

You might see errors while the Pi is reimaging, this is because we are serving via NFS a full OS that doesn’t like some of its filesystems being read-only. And that is fine, we only need enough kernel to run wget, dd and reboot. A better alternative might be to serve via NFS a more well purposed OS, but in case I prefer serving the very OS we are imaging, because it’s more expedient, but also because it gives me access to the very EEPROM binaries which are critical to the process.

There you Have it

The building blocks to set up a more seamless and better integrated reimaging process :).

I currently use this for a fleet of 70 Pis to great effect. Of course I image them with a custom RaspiOS build which on top of serving professional needs, is setup to check with the NetBoot server every 15 minutes. I’ll talk about building custom RaspiOS images in a different post. I can simply publish a new image there and the whole fleet will go in for reimaging. When it does so, I have the Pis sleep a random amount between 0 and 48 hours to avoid having a huge wave of traffic, but also to avoid breaking everything at the same time. This give me time to halt the process if the Pis are coming out of reimaging wrong. It’s also particularly relevant as the NetBoot server can have concurrency issues with Pis imaging at the same time. None that will create real issues, but there is a chicken-and-egg issue with the way Pis update their EEPROM to be instructed to boot from their SD card once they have booted from the network and reimaged themselves. They need to be told by the NetBoot server itself, and so it needs to switch into serving instructions for SD Booting for a few minutes while a newly imaged Pi reboots. And so if a Pi shows up to be imaged at that time, it’ll reboot into its SD card having done nothing. This issue seems to exist in other scenarios such as USB booting, it seems to be a shortcoming of how EEPROMS are updated.