Network Booting Tutorial

FLnelson · April 16, 2022, 11:58pm

Have you wondered about that weird last boot option on the boot loader page? Wife getting on you about the server in the house and need to move to a datacenter? Well it’s time to learn about network booting.

Network booting ditches your standard boot USB with a local server. This TFTP server delivers your boot files to your 3 nodes. This can be useful in bigger home farms, but is all but mandatory in a datacenter setup.

Network boot setup is quite easy and is centered about configuring a TFTP server. There are essentially 2 options for this, a small dedicated server such as a raspberry pi, or piggybacking on your pfsense or opnsense router. I would recommend the latter as it eliminates another piece of equipment and is probably more reliable.

You should familiarize yourself with this link.

github.com

threefoldtech/zos/blob/main/docs/network/setup_farm_network.md

# ZOSv2 network considerations

Running ZOS on a node is just a matter of booting it with a USB stick, or with a dhcp/bootp/tftp server with the right configuration so that the node can start the OS.
Once it starts booting, the OS detects the NICs, and starts the network configuration. A Node can only continue it's boot process till the end when it effectively has received an IP address and a route to the Internet. Without that, the Node will  retry indefinitely to obtain Internet access and not finish it's startup.

So a Node needs to be connected to a __wired__ network, providing a dhcp server and a default gateway to the Internet, be it NATed or plainly on the public network, where any route to the Internet, be it IPv4 or IPv6 or both is sufficient.

For a node to have that ability to host user networks, we **strongly** advise to have a working IPv6 setup, as that is the primary IP stack we're using for the User Network's Mesh to function.

## Running ZOS (v2) at home

Running a ZOS Node at home is plain simple. Connect it to your router, plug it in the network, insert the preconfigured USB stick containing the bootloader and the `farmer_id`, power it on.
You will then see it appear in the [Cockpit](https://cockpit.testnet.grid.tf/capacity), under your farm.

## Running ZOS (v2) in a multi-node farm in a DC

Multi-Node Farms, where a farmer wants to host the nodes in a data centre, have basically the same simplicity, but the nodes can boot from a boot server that provides for DHCP, and also delivers the iPXE image to load, without the need for a USB stick in every Node.

A boot server is not really necessary, but it helps ;-). That server has a list of the MAC addresses of the nodes, and delivers the bootloader over PXE. The farmer is responsible to set-up the network, and configure the boot server.

This file has been truncated. show original

Setting Up Your Router to Allow Network Booting

These steps are for OPNsense, PFsense may differ. These set are required regardless of where you have your TFTP server.

Services>DHCPv4>LAN>Network Booting

Check “Enable Network Booting”

Enter the IP address of your TFTP server under “Set next-server IP” This may be the router’s IP or whatever device you are booting from.

Enter “pxelinux.0” under Set default bios filename

Ignore the TFTP Server settings.

TFTP server setup on a debian machine such as Ubuntu or Raspberry Pi

apt-get update

apt-get install tftpd-hpa

cd /srv/tftp/

wget http://ftp.nl.debian.org/debian/dists/buster/main/installer-amd64/current/images/netboot/netboot.tar.gz

wget http://ftp.nl.debian.org/debian/dists/buster/main/installer-amd64/current/images/netboot/pxelinux.0

wget https://bootstrap.grid.tf/krn/prod/ --no-check-certificate

mv ipxe-prod.lkrn

tar -xvzf netboot.tar.gz

rm version.info netboot.tar.gz

rm pxelinux.cfg/default

chmod 777 /srv/tftp/pxelinux.cfg (optional if next step fails)

echo ‘default ipxe-prod.lkrn’ >> pxelinux.cfg/default

TFTP Server on a OPNsense router

(I have not used on PFsense, steps are probably similar, but the directory or other small things may differ)

The first step is to download the TFTP server plugin. Go to system>firmware>Status and check for updates, follow prompts to install. Then click the Plugins tab and search for tftp, install os-tftp. Once that is installed go to Services>TFTP (you may need to refresh page). Check the Enable box and input your router ip (normally 192.168.1.1). Click save.

Turn on ssh for your router. In OPNsense it is System>Settings>Administration. Then check the Enable, root login, and password login. Hop over to Putty and connect to your router, normally 192.168.1.1. Login as root and input your password. Hit 8 to enter the shell.

In OPNsense the tftp directory is /usr/local/tftp

cd /usr/local

mkdir tftp

cd ./tftp

fetch http://ftp.nl.debian.org/debian/dists/buster/main/installer-amd64/current/images/netboot/netboot.tar.gz

fetch http://ftp.nl.debian.org/debian/dists/buster/main/installer-amd64/current/images/netboot/pxelinux.0

fetch https://bootstrap.grid.tf/krn/prod/

mv ipxe-prod.lkrn

tar -xvzf netboot.tar.gz

rm version.info netboot.tar.gz

rm pxelinux.cfg/default

echo ‘default ipxe-prod.lkrn’ >> pxelinux.cfg/default

You can get out of shell by entering exit or just closing the window.

3Node Setup

Set the server to BIOS boot and put PXE or network boot as the first choice. At least on Dell machines, make sure you have the network cable in plug 1 or it won’t boot.

Most credit goes to Micheal who sent the linux commands to me, Scott who sent it to him, and some mystery person who sent it to scott. I managed to figure out the server and router settings somehow and adapted the commands to ubuntu and OPNsense.

colossus · April 16, 2022, 8:51pm

Excellent work, Nelson!

Mik · April 16, 2022, 10:47pm

Amazing! That’s literally a next level guide here!!

Let the Data Centers know about Threefold.

Great work.

FLnelson · April 16, 2022, 11:59pm

Video tutorial coming soon once new webcam shows up. Time is money and syncing up gopro footage to screen capture is no fun.

randynho · April 17, 2022, 9:31am

Thanks a lot! That helped me a lot

parkers · May 27, 2022, 8:32pm

Wanted to add in, openwrt can do this aswell with the package “pxe-boot” downloadable in the software section

TheCaptain · June 29, 2022, 2:44pm

These are the steps required to enable PXE booting on pfSense. This guide assumes you’ll be using the router as your PXE server; pfSense allows boot file uploads directly from its web GUI.

Log into your pfSense instance, go to System>Package Manager, search and add ‘tftpd’ package under ‘Available Packages’ tab
Go to Services>TFTP Server, under ‘Settings’ tab check enable and enter the router IP in TFTP Server Bind IP field
Switch to ‘Files’ tab under Services>TFTP Server and upload your ‘ipxe-prod.efi’ file acquired from https://v3.bootstrap.grid.tf/ (second option labeled ‘EFI Kernel’)
Go to Services>DHCP Server, under ‘Other Options’ section click Display Advance next to ‘TFTP’ and enter router IP, then click Display Advance next to ‘Network Booting’, check enable, enter router IP in Next Server field and enter ipxe-prod.efi in Default BIOS file name field

You’re done. You’ll want to ensure your clients are configured with boot priority set as ipv4 in first spot and my experience with HP mini machines says you’ll need to disable secure boot and enable legacy boot within BIOS.

Mik · June 29, 2022, 3:22pm

That’s an amazing and short guide @TheCaptain

Thanks for sharing.