Introduction
After several months, it's finally time to post an update about my home lab router setup.
Today, we are announcing that the Home+Lab version of pfSense Plus, the commercial fork of the popular open-source firewall pfSense, is no longer available for free download. The decision to stop offering the Home+Lab version of pfSense Plus was made in order to align Netgate’s business model to better serve our worldwide customer base and partners while continuing to invest in the development and support of the product. – Netgate, October 26, 2023
Unfortunately, Netgate has also discontinued the Home+Lab licensing for TNSR. This means I would need to purchase an expensive subscription to the software in order to continue receiving updates.
As time passed, I started encountering issues with my setup. These problems were caused both by configuration mistakes on my end, the ProSafe MS510TX switch, and other various pieces for the link. The ProSafe began experiencing throughput issues, and at times, upload speeds seemed randomly limited to 2Gbps, requiring a reboot of the switch to resolve the issue, and sadly I was already on the latest firmware.
Needing some alternatives, I tried the TP-Link TL-SX3008F as a replacement, but encountered similar issues with upload throughput when using a copper to fiber module. The 5GbE port on the AT&T gateway sadly can't negotiate at 10GbE which requires a device between it and TNSR to get the link to the correct speed. Despite trying numerous adapters that supposedly support 5GbE on the copper side, and experimenting with different configurations, I was unable to resolve the issue.
For a temporary fix, I swapped out the X520 for an AQC107 based NIC that supports the correct speeds, installed TSNR under VMWare ESXi 8 using VMXNET3 adapters, which allowed my incoming WAN to work at 5GbE. However, this was not a permanent solution; I knew I needed something more reliable and simpler. After an unreasonable amount of time, research and planning, I decided to try VyOS bare metal. Having fewer devices between the AT&T gateway and my router is ideal, and introduces fewer places for a configuration mistake, or an unseen incompatibility to occur. Ideally, if VyOs supports the AQC107 NIC properly, this would be great and I could eliminate any extra hops and keep things simple.
The Router
Compared to the last post, I've not changed much. The current hardware is as follows:
Server: Dell PowerEdge R330
CPU: Intel Xeon E3-1285 v6 (4.1GHz, 4.5GHz Boost), 4C/8T
RAM: 32GB DDR4 3200MHz ECC (x2 16GB)
SSD: Crucial MX500 500GB SSD
PSU: (x2) 500W EPP PSUs
NIC 1: TP-Link TX401 (AQC107) - WAN
NIC 2: Intel X710-T4L (Intel ARK) - LANs
The two main changes here are the CPU and the first NIC. I was able to find the new chip for a very good deal. Since I am leaving behind the all-mighty DPDK in TNSR, I figured I'd want to ensure the CPU had as much juice as I could have on this server for routing. The E3-1285 v6 is basically an i7-7700k, and you can find them for under 250 on eBay if you search a bit. You can still get away with the E3-1270 v6 for this task if your NICs have good offloading support.
Previously, I had replaced the X520 with the TP-Link TX401 (AQC107). The drivers for it in ESXi allowed the link to operate at 5GbE at the top, but also at 10GbE with the VMXNET3 adapters in the VM for TNSR to use. I was very interested to see if VyOS allowed the card to operate at it's rated speeds bare metal, so I left it in.
For now, I will be testing the AQC107 NIC as my WAN, and the Intel X710-T4L card as my LANs, each port a different subnet.
Installing VyOS
For testing, I went ahead and downloaded the latest rolling release, which I may stay on for a while depending on how things go. If I need to upgrade due to any bugs, they make it very simple.
Installation was very straight forward. You pick the user you want to sign in with, which drive to install to, and so on. Once it was installed, I followed the simple config guide here, with some changes to get my home's IoT network, WAN, and management network going so I could SSH in and configure from my desk.
Initial Configuration
On boot, the first thing to do was to make sure the AQC107 NIC was working. Without needing to install any drivers, the NIC was recognized and working. I am not sure if this is the case on the LTS releases, which may require you to install drivers from Marvel for the AQC107.
My main concern is if the card would be able to negotiate at 5GbE natively with my AT&T gateway, since on bare metal TNSR, I would not be able to use 5GbE speeds, only 1GbE and 10GbE.
set interfaces ethernet <interface> speed <auto | 10 | 100 | 1000 | 2500 | 5000 | 10000 | 25000 | 40000 | 50000 | 100000>
set interfaces ethernet <interface> duplex <auto | full | half>The above commands let you set either the speed or duplex of the card, pressing tab after "speed" for the desired WAN interface netted me these options:
It's funny that I have options that are not supported by the card, but I digress. The port on the AT&T gateway is set to force 5GbE, so ideally, I leave the interface to Auto to get what I need, and I did! The AQC107 worked at the right speeds, flawlessly.
I spent some time configuring my LANs, DHCP, firewall rules, etc to get my network back up to snuff. The quick start guide was very thorough at explaining the setup process, I feel that any novice with a little but of networking knowledge would be able to get this going pretty quickly.
Performance Tuning & Speed Test
Before adjusting any settings, I ran a few speed tests, seeing an average of 3Gbps up and 3.5Gbps down. VyOS accommodates a broad spectrum of hardware, so there's no one-size-fits-all performance configuration for every setup.
set system option performance < throughput | latency >The first thing I did was above, setting the system profile to performance. This page describes the two options and which may work best for you.
Next, I enabled offloading for my NICs, and followed some advice on this guide. After about 20 minutes of tuning, I was able to see the results I had hoped for, with roughly the same results across all of my servers.
A Few Considerations
I haven't tested 5GbE functionality with the Intel X710-T4L on VyOS yet. It faced a similar problem like my AQC107 did, negotiating only at 1GbE or 10GbE under TNSR bare metal. I plan to test this soon, and if the results are promising, I might replace the AQC107 with another X710-T4L, provided I can find one at a reasonable price, they are still somewhat expensive. Having three more ports on the machine would be nice, and using only industry proven NICs would be ideal for reliability. I've never had any issues with the AQC107, but time will tell.
In the future, if AT&T eventually offers 10Gbps to the home without a dedicated line, it will be interesting to see how this setup handles that kind of bandwidth. I would imagine I'd need a little more oompf with my CPU performance to get to that point, but we'll see when and if that day comes.
Final Thoughts
For now, after configuring VyOS to best suit my network and firewall needs over the last week, I can say I am quite happy with the results. If you are curious to learn more about VyOS, check this video here. It's an older video, but quite accurate and informative. I will post updates if anything significant changes.
