Here’s is some information that took me a good while to gather.
With the igb driver in FreeBSD, the mbuf cluster size needed is a mathematical formula involving the number of CPUs & the desired MTU. Unfortunately, it is currently hard set. On enterprise machines with many cores and higher MTUs, it is quite easy to reach this set limit. It will express itself with the following error message after an ifconfig:
igb0: Could not setup receive structures
This limit can be overridden with the following in /etc/sysctl.conf
kern.ipc.nmbclusters=131072 kern.ipc.nmbjumbo9=38400
These are the value that worked for 16 cores & an MTU of 9000.
While we’re at it, it took me a while to nail the exact syntax require for NIC bonding so here it is:
/etc/rc.conf
if_lagg_load="YES" ifconfig_igb0="mtu 9000 UP" ifconfig_igb1="mtu 9000 UP" cloned_interfaces="lagg0" ifconfig_lagg0="laggproto failover laggport igb0 laggport igb1 192.168.0.123 netmask 255.255.255.0"
As far as I can tell, capitalization matters…