My darling fiancé enjoys the speed offered by TNSR, as evidence by his article about it. I enjoy the benefits of using a firewall with tons of other rich features like OPNsense or pfSense. Our ATT gateway doesn't have an extra 5 Gb port so I can't just run another cable to my firewall. I don't like dealing with double NAT either, so I had to get creative.
The challenge
We are not a business, we just have a hyper-fixation on technology and enjoy owning the permanent headache that is a home-lab environment. As a byproduct of that we have to shoehorn what we want and need into whatever hardware we can get. As mentioned in our post about TNSR we were seeing issues caused by TNSR using an outdated version of DPDK. Our network cards were supported by the drivers installed with TNSR out of the box, however only partially. We could only get them to negotiate at 1 Gbps or 10 Gbps, not at 5 Gbps. This is an issue because our ATT gateway only has a 5 Gb port and three 1 Gb ports. If we're paying for a 5 Gbps fiber internet connection, we're using all 5 Gbps.
Going back again to the TNSR post, we actually had to redo our network infrastructure entirely. The post has a solution that didn't work for us thanks to a hidden issue with the NetGear multi-gig switch. We tried grabbing a fairly inexpensive fiber switch that promised high throughput but the same performance issues came up again and again no matter what we did.
At the end of the day, we just virtualized TNSR even though it's the only thing running on the R330. ESXi has the drivers we need for a different set of network cards that we already own that can handle the 5 Gbps speeds we so desire and VMXNET3 drivers will negotiate with TNSR at 10 Gbps, so it was the next best solution. I would have loved to put my firewall on there too but the R330 just doesn't have the oomph to handle it, so my firewall lives as a VM on my server. This makes it complicated.
VMWare, routing, and bridging networks
TNSR being virtualized is not ideal. It doesn't effect the routing speed in any perceivable way but it does effect the network architecture of our environment significantly. In order to make it work we had to create port groups with specific vLAN tags within ESXi on the R330, then replicate those port groups on our other servers.
As far as TNSR or OPNsense are concerned, these are regular interfaces. We do not have any vLAN configuration in TNSR and I did not set up a vLAN configuration within OPNsense. We just have to make these port groups within ESXi to give TNSR and OPNsense network interfaces to use.
In order for bridging to work while using OPNsense (or pfSense or likely any software router/firewall that can do so), I had to configure security settings for the vSwitch to allow the switch to run in Promiscuous Mode and accept Forged Transmits. If OPNsense was installed bare-metal this wouldn't be necessary at all, it's just a byproduct of having to virtualize everything. This must also be repeated on the vSwitch used by the interfaces "behind" the firewall.
Once this is done, we're ready to set up OPNsense as a layer 2 firewall! If you're following along with hopes of doing this yourself you rely on the documentation for OPNsense to configure the firewall itself. If you want more details about how I made this work, follow along the short guide below!
Configuring ESXi to allow OPNsense to act as a Layer 2 Firewall
General Requirements:
- Router or gateway outside of the virtual environment.
- It can be inside the same ESXi environment but it will require a different configuration than what is shown here. It's similar enough that if you can follow this guide, you can make it work.
- OPNsense installed or ready to install in ESXi.
- NIC with multiple ports.
- (optional) vLANs configured from router or gateway.
- Management (MGT) - vLAN tag 0 / untagged
- vLAN10 (LAN) - vLAN tag 10
- vLAN20 (DMZ) - vLAN tag 20
Step-by-step:
- Create or modify an existing vSwitch in ESXi for the incoming connection from your router or gateway. Use the following options:
- Enable promiscuous mode
- Allow forged transmits
- Create or modify an existing vSwitch in ESXi for the outgoing connection from your firewall. This connection doesn't need to be utilized by a physical cable, it can be entirely virtual. Use the same options as before.
- Create port groups in ESXi for inbound connections from your router. If using vLANs ensure that the port group vLAN ID matches the tag correctly.
- Create port groups in ESXi for outbound connections from your firewall. This is my configuration as an example.
- Assign port groups as network adapters to the OPNsense VM.
- Install and configure OPNsense as normal, then follow their documentation on setting up the layer 2 / transparent firewall. In my case, I bridged "shichi - LAN" and "Firewall - LAN" as well as "shichi - DMZ" and "Firewall - DMZ". A more standard method is to use "WAN" and "LAN". It will work as long as one side can reach the internet and the other side is a local network you want to protect behind a firewall.
