The post title is a bit of a mouthful but a necessary one – this is a very specific scenario. Using pfSense as a dedicated gateway/firewall on the OVH-network was already not a simple situation; doing it inside a KVM (with Proxmox VE as wrapper which will also host LXC’s on the ‘LAN’) adds another layer of complexity. This guide was pieced together from various tidbits all over the web and my own experimentation so I thought it a good idea to do a solid write-up on it – if not for posterity then for my own future reference.

 

Preface

If you don’t care about the background or aims, skip this section.

I originally had a Proxmox VE server running several LXC containers which was also acting as a gateway and firewall itself thanks to FireHOL (a very nice iptables rule generator). FireHOL was really great to use and I had no problem with it, but it isn’t as powerful as a purpose-built gateway distribution. I also wanted something I could easily manage instead of editing config files over SSH whenever I wanted a temporary rule change. FireHOL could, however, be useful as another layer of defense – especially for sensitive systems. I’ll probably do an  guide on that later. Regardless, the act of having a firewall/gateway on the same machine as Proxmox itself is obviously not a permanent solution so I decided to create a dedicated VM to act as a gateway.

Despite all my other Proxmox machines being LXC’s (reduced overhead and increased density compared to hypervisors) I opted for a KVM setup. The fact that pfSense is FreeBSD-based and therefore not compatible with LXC was actually of secondary concern; I was considering ClearOS, IPCop and some other gateway/firewall distro’s that could of worked as an LXC. I decided on KVM over LXC in this case to provide maximum isolation from the Proxmox VE host.

 

Before you start…

The Proxmox VE host will remain directly connected to the OVH gateway via vmbr0. You can use it as an entry-point for an SSH tunnel when you need to work on the pfSense gateway. Consider using iptables to block everything except SSH access to the PVE host.
It is possible for LXC guests to bridge directly to vmbr0 and completely bypass the pfSense firewall/LAN if desired. Simply modify the LXC networking configuration and give it the external IP, and a regenerated and unique Virtual MAC assigned to that IP.
Take note of your OVH gateway address. It is always the main IP of your server with the fourth octet replaced with 254, e.g. main IP 1.2.3.4 has a gateway of 1.2.3.254.

 

Prerequisites

The following assumptions will be made about your situation. They are exemplified values but will be referenced as such throughout the guide – be sure to substitute with your own configuration where necessary:

Basic network diagram

  1. You’re running Proxmox VE 5 (Debian 9 based) on an OVH-family (OVH/So You Start/Kimsufi) server;
  2. Your PVE host has a vmbr0 bridge with external IP of 1.2.3.4 assigned. It already has fully working internet access in both directions (i.e. has the Gateway of 1.2.3.254);
  3. PVE has a vmbr2 bridge with IP of 192.168.1.1 and subnet 255.255.0.0 – this corresponds to the LAN network that pfSense will be the gateway of. The third octet will correspond to the PVE node and the fourth octet matches the specific CTID/VMID on that node;
  4. An extra fail-over external IP 5.6.7.8 provided by OVH will be dedicated to the pfSense gateway;
  5. An extra fail-over external IP 9.10.11.12 will be assigned (through NAT) to our LXC machine called www. It has a CTID of 108 and exists on the same (first) PVE node, thus will have an internal address of 192.168.1.108 bridged to vmbr2.
  6. Both the extra fail-over IP’s provided by OVH have the same Virtual MAC assigned to them. 
    It is important that all external IP’s you want to serve behind the pfSense gateway share the same Virtual MAC as the gateway itself!
  7. You know how to navigate text-mode setup screens, edit XML files, and other such things a sysadmin should be able to do in their sleep.

 

Creating the KVM

  1. Head on over to the pfSense download page and grab a link to the latest x64 ISO, jump into SSH on your Proxmox VE host and wget it to your template ISO folder:
cd /var/lib/vz/template/iso
wget https://nyifiles.pfsense.org/mirror/downloads/pfSense-CE-2.3.4-RELEASE-amd64.iso.gz
  1. Login to Proxmox VE and Create VM. For the purposes of this guide:
    1. VMID will be 254;
    2. VM name will be gateway;
  2. OS tab:
    1. SelectOther OS types;
  3. CD/DVD tab:
    1. Select the ISO we previously downloaded;
  4. Hard Disk tab:
    1. Set Bus/Device to SCSI;
      This is actually VirtIO SCSI which is generally the best choice, for those OS’s that support it – FreeBSD indeed does;
    2. Size doesn’t need to be a lot, I chose 10GB as a start since I’d like to look into using squid cache or something, but you should be able to get away with 4GB;
    3. Cache should be kept on Default (No cache) in my opinion, though you’re welcome to read the documentation and decide for yourself;
    4. Check the Discard option if this storage is thinly provisioned (tip – it should be! I might do a write up sometime about how to convert to it);
  5. CPU tab:
    1. This is completely dependent on your hardware and you’ll need to measure performance to get an idea. It’s easy to just assign more cores later if you need it, start with at least two cores though;
    2. Type should be kept on Default (kvm64) – works fine on anything that isn’t a dinosaur;
  6. Memory tab:
    1. Again this is hardware and load dependent, I set mine to a dedicated 1GB. Note that pfSense recommend a minimum of 1024MB;
  7. Network tab – We will setup the WAN interface now and add LAN afterwards:
    1. Set Bridge to vmbr0 (our preconfigured WAN interface);
    2. Set Model to VirtIO (paravirtualized)note that this requires an additional fix later due to an outstanding FreeBSD issue;
    3. Set MAC Address to the Virtual MAC you’ve preconfigured in OVH (see point #4 in Prerequisites).
  8. Now that the KVM instance is created, we need to add another Network interface for the WAN:
    1. Select the VMID in the left pane > Hardware tab > Add > Network Device;
    2. Set Bridge to vmbr2;
    3. Set Model to VirtIO again;
    4. MAC Address doesn’t matter this time – it can be randomly generated.

Ready to install! 🤓

 

Installing pfSense

Power up the VM and switch to the “Console” tab – ore alternative, select Console from the pulldown menu at the top bar for a popout console window. We’ll immediately get a VNC display of the CD booting (probably already at a menu).

  1. Either wait for timeouts to boot default, or select Multi User (default) followed by Installation – pretty self-explainatory;
  2. Configure Console screen – No need to change anything here, just select Accept;
  3. Select Task screen – Let’s select Custom Install, because why not;
  4. Next few screens are just selecting the disk and formatting, do and accept all. Feel free to chuckle at the Select Geometry screen;
  5. Partition Disk screen can be skipped – it will just make a FreeBSD partition covering the whole disk automatically;
  6. Install bootblock(s) can be left at default, just Accept;
  7. At Select Subpartitions screen I reduced my swap significantly. For my 1GB VM with 10GB disk, I set swap to 500M. I’ll keep an eye on memory use.
  8. Select Standard Kernel when asked. It would be nice if we had something in-between to reduce interrupts, but I digress;

Installed! The system will reboot automatically and start initial text setup.

 

Initial Configuration

Upon rebooting, you’ll have to do some minor setup before you can jump into the Web GUI. We’re on VNC so unfortunately there’s no clipboard pasting here.

  1. VLAN’s are a no;
  2. Enter the WAN interface should be vtnet0 if you followed along closely, corresponding to the first NIC;
  3. Enter the LAN interface should be vtnet1, similarly to above;
  4. Enter the Optional 1 interface should be blank, we’re done;
  5. Confirm with y and take note that the process will freeze on Configuring WAN interface... for a minute or two, or at least it did for me. I imagine it’s trying to get a DHCP ack from OVH, but it eventually times-out;

    pfSense CLI Menu

  6. Now you should be presented with the main menu. Observe the interfaces info underneath the menu header (picture on right) – we just need to set the right LAN IP so we can access the web GUI via SSH tunnel. So go ahead and press 2 for Set interface(s) IP address;
  7. Press 2 again to assign the LAN; then enter the IP address we decided on in prerequisites – for this case it’s 192.168.1.254 with a subnet CIDR of 16;
  8. Leave the LAN Gateway empty; assign an IPv6 address at your preference;
  9. DHCP is a no – all of our LAN machines will have static IP’s;
  10. Revert to HTTP is your prerogative, I recommend no (just remember to tunnel the 443 – i.e. HTTPS – port);

Now you should be able to access the web GUI over an SSH tunnel on HTTPS (or HTTP if you opted for it).

 

Initial GUI config

Upon visiting the GUI you’ll need to login with default credentials: username of admin and password of pfsense.  then go through the setup wizard.

  1. Hostname and Domain are self-explanatory. You will need these later for setting LAN DNS. DNS Server entries should probably point to the OVH DNS, at least the primary one, which is 213.186.33.99. For secondary, I opted to use Google’s 8.8.8.8. The Override DNS can be left checked, doesn’t matter since we don’t use DHCP or PPP on WAN;
  2. Time server hostname can be whatever you like, but I opted to use pfsense’s own – replace the entry with 0.pfsense.pool.ntp.org 1.pfsense.pool.ntp.org 2.pfsense.pool.ntp.org 3.pfsense.pool.ntp.org exactly (i.e. space separated list). Timezone is self-explanatory;
  3. WAN Interface should be set to static. Set the IP Address to the IP that OVH has assigned to this MAC, for this example it was 5.6.7.8. Ensure Subnet Mask is on 32. Leave Upstream Gateway BLANK. Finally, at the very bottom, uncheck Block private networks from entering via WAN. (NB: I left this on, confirm later);
  4. The next page, Configure LAN Interface, was already done on the VNC console so we can skip it;
  5. Setup an ultra-secure admin password – especially if you’re doing something silly like opening up the web GUI to WAN later.

Now the initial config is done, we need to do one more thing:

  1. Browse to System > Advanced and the Networking tab;
  2. Look for Disable hardware checksum offload and check it. This is a work-around for a FreeBSD VirtIO Network driver issue;
  3. Press Save, the go to Diagnostics > Reboot.

 

Gateway setup

Next step is to get internet access working for the gateway.

  1. Go to System > Routing and you should be on the Gateways tab. If there is a Gateway present (I had WAN_DHCP6), press the Edit icon and disable it (first option). I haven’t bothered to figure out IPv6 yet;
  2. Ensure Interface is WAN and Address Family is IPv4. Set the Name to whatever you like, though do note that this is a ‘dummy’ entry so I named mine OVH_DUMMY. Leave Gateway as BLANK, check both Default Gateway and Disable Gateway Monitoring. Add a Description if you like – I entered Dummy gateway for OVH default route. Save then Apply Changes;
  3. Now go to the Static Routes tab and press Add. Set Destination Network to [YOUR_GATEWAY_IP]/32, e.g. 1.2.3.254/32. Ensure the Gateway selected is the dummy entry we created just before, e.g. OVH_DUMMY. Enter whatever Description you like, e.g. OVH gateway. Save then Apply Changes;
Note that this Gateway can not be selected under Interface configuration, it will simply not appear in the list. This is nothing to worry about.

Now the fancy part. We need to add that route as the default via a startup shell command. We do this by manually adding it to pfSense’s xml config. This is their documented way so is probably the best (i.e. should persist through updates).

  1. Go to Diagnostics > Backup & Restore;
  2. Change Backup area to System, then Download;
  3. Edit the downloaded XML file in your favorite Unix-EOL-friendly text editor. At the end of the <system> element, which should be after the <disablechecksumoffloading></disablechecksumoffloading> element, add a new shellcmd…
    <shellcmd>route add default 1.2.3.254</shellcmd>

    … where 1.2.3.254 is your actual Gateway IP. Save the XML;

  4. On the Backup & Restore screen, change Restore area to System, browse to the XML we just edited, and Restore;
  5. Go to Diagnostics > Reboot.

After rebooting, the Gateway should now have working internet access. You might have an Update Available listed on the admin dashboard now. If not, ping google.com from the VNC and it should work – might be a good idea anyway just to confirm that DNS is also working.

 

NAT setup

We’re going to use 1:1 NAT instead of routing because it allows ARP’s towards our LAN clients to correctly resolve to their external IP.

First, we’ll reconfigure and start an LXC container, as mentioned earlier this LXC is called www. Add/edit the LXC’s eth0 interface:

  1. Set the Bridge to vmbr2, this is our previously-created cluster LAN bridge;
  2. Set the IPv4/CIDR to 192.168.1.108/24, or whatever scheme you like. As mentioned before, my setup has third octet as Node number and fourth as CTID;
  3. Set the Gateway to 192.168.1.254, as we set to the pfSense LAN interface earlier.
  4. Also update the DNS. In the DNS tab, set the DNS domain (it should match the same domain you set in pfSense web setup wizard) and the DNS server to 192.168.1.254 (the pfSense gateway IP).

Start that LXC and jump into it’s shell via pct enter 108 command (or similar) – you should be able to ping 192.168.1.254 immediately (default pfSense LAN firewall rules are to allow everything from LAN).

Next we add a Virtual IP to get the WAN to respond to our LAN’s external IP’s on the WAN side:

  1. Go to Firewall > Virtual IPs and press Add;
  2. Set Type to IP Alias and Interface to WAN;
  3. Set Address(es) to 9.10.11.12/32 as taken from our OVH control panel which we want assigned to the new LXC container (remember – the additional IP’s must have the same MAC as the Gateway IP itself);
  4. Enter a meaningful Description, e.g. www, then Save and Apply.

Now we add the actual 1:1 NAT rule:

  1. Go to Firewall > NAT and the 1:1 tab, press either of the Add buttons (FYI, they’re Add to Top and Add to Bottom);
  2. Set Interface to WAN, set External subnet IP to 9.10.11.12 (or whatever you had designated) and set Internal IP to Single Host with IP 192.168.1.108 (or whatever you set in the LXC’s eth0 config). Enable the NAT Reflection option if you want it.
  3. Enter a meaningful Description, e.g. www, then Save and Apply.

That’s it! Your LXC container should be able to ping google.com, apt update, and so on.

Note that pfSense firewall is, by default, “allow all” on outbound and “allow none” on inbound. Firewall rules are beyond the scope of this article – there is plenty of help already out there for this.
Final tips:

  • Use Firewall > Alias to create Host(s) aliases so you can reference LAN machines by a friendly name instead of IP in your firewall rules.
  • Use pfBlockerNG and FireHOL IP lists for an automatically-updating nasties-blocker! Check out this great guide (link).

 

References