Bdtyktl

For my personal minicluster, I successfully setup an Ubuntu server 16.04 LTS PXE network booting system, however, in order to take advantage of the 4-port NICs I have, I want to setup Ethernet bonding.

I have found no relevant resources to Linux kernel version 4.4.0-21-generic, older solutions are ineffective.

This is a custom solution, a hack really, that I have managed to get working.

In my NFS that the PXE systems are supposed to boot from, Ensure the /etc/initramfs-tools/initramfs.conf to have no DEVICE set, and modules equal to most.

From there, you're going to want to add a file at /etc/initramfs-tools/scripts/nfs-premount/00_bond, ensure it has 0755 permissions, that is it is executable.

00_bond contents:

#!/bin/sh

ip link add bond0 type bond mode 802.3ad

echo "+eth0" > /sys/class/net/bond0/bonding/slaves

echo "+eth1" > /sys/class/net/bond0/bonding/slaves

echo "+eth2" > /sys/class/net/bond0/bonding/slaves

modprobe af_packet

ipconfig -c dhcp -d bond0

This script creates a bond within the initramfs, manually adds the slaves, as we do not have the tools to do that (they could be added to the initramfs, but unnecessary). Add/remove/rename interfaces as needed.

Finally, we load the af_packet kernel module, and initialize DHCP on our bonded interface. ipconfig is a program provided by klibc, not to be confused with ifconfig.

The last configuration of our initramfs comes in at /etc/initramfs-tools/modules, where we want to add the bonding module to our initramfs. Simply add a line bonding to the end of the file, it is generally empty by default.

You can update the initramfs (make sure you are chrooted or in another system booted into the NFS partition) by running update-initramfs -u. Once this is complete, make sure to update any usage of it from your TFTP server so that your new initramfs is booted remotely.

In DHCP, since we have bypassed DHCP configuration of the network interfaces (assuming pxelinux), and are dependent on the ethX NIC name format, we want to change some kernel boot parameters. You can change them in the script, however, I find it more reliable to use the old naming scheme, as this is a case where if a NIC changes, it's best we have drop-in replacement naming -- an issue I have run into before.

These are the kernel boot parameters I used for version 4.4.0-21-generic, as entered in my PXELinux configuration:

APPEND root=/dev/nfs initrd=initrd.img-4.4.0-21-generic nfsroot=192.168.0.1:/nfs net.ifnames=0 biosdevname=0 ip=none rw

net.ifnames=0 biosdevname=0 disables the new NIC naming scheme.

ip=none disables any default network configuration. I had no issues with ip=dhcp, but it is unnecessary as it will terminate when it sees a network already in place.

The rest are standard NFS mounting parameters.

Once all of this is complete, your NFS server is running, accessible, and a proper Ubuntu install, your DHCP server is prepared, then you should be ready to boot into Ubuntu with an Ethernet bonded NFS root system, from PXE. The mode I chose, 802.3ad, provides speed and redundancy from disconnections, while there are other modes available that can provide stronger redundancy in exchange for speed. This is particularly useful for this use case, as when we get a sudden disconnection, it's akin to having the root drive pulled mid-operation.

It should be noted that none of the standard Ubuntu networking will have been configured (i.e. /etc/network/interfaces), and as such, tools like ifup and ifdown may not work until that is setup. Of course, you should be prepared to power cycle if you were to run such a command, unless you boot a persistent ramdisk with a NFS as a non-root, to maintain minimal functionality.