Highly Available PFsense Cluster On Proxmox - Walkthrough
Before kubeadm and different Kubernetes (aka k8s) installation tools came up. The community decided to not get involved with the underlying resources. Therefore k8s installers only focused on bootstrapping clusters on present nodes and you had to figure out the topology yourself, secure the underlying compute resources and so on..
Even though that set security a milestone back, i think it was a wise decision to make. The system was complex enough for them to care about securing it, also there's the vision of the cloud. Where developers don't have to focus on provisioning k8s instances as much as providing solutions to real world problems.
It seems logical like we established a de-facto in the Emailing system, Linux and k8s combined set a new era, a new way of looking at hardware and we should embrace it.
For people like me working with on premise clusters, the attack surface is huge and unheard off, so we battle our way through it and have to convince our bosses with the importance of securing this single point of failure global API.
So the whole traffic passes through the load balancer, It made sense to use PFsense as a gateway since it has a built in HAproxy.
Many companies especially in my home country, use PFsense to protect their internal networks, Since it is free of charge and provide enterprise grade features.
Here's a quick cool fact, PFsense took its name from the packet filtering software named "pf" as it was making sense of it. That's dope!
PFsense Allows for all sorts of restrictions and monitoring from a single dashboard written in PHP.
You can have things like VPN access, install certificates, have Virtual IPs for your internal nodes, act as a router and loadbalance traffic as it goes through it, have things like snort and apply intrusion detection and prevention. A complete list of PFsense features can be found here.
Your costumers would be impressed when they ask to see the product and you tell them that they have to install openVPN to get access.
As you can see we have two firewalls accessible from the outside world using failover IPs.
If we would simulate the fail-over feature by going to maintenance mode under status > CARP (failover), the roles would be switched in a fraction of a second.
Not everything would be synced from the master to the backed firewall as illustrated in this figure.
This is really useful for zero downtime when you have hardware failures, performing maintenance, rebooting the firewall and so on..
In this tutorial we'll use a single Proxmox server, which doesn't cover disaster recovery strategies. A typical one would involve using the Proxmox HA feature and have two Firewalls on two separate physical nodes joined as a cluster using things like vracks. If the whole server goes down, the backup firewall would take the lead while the other is being migrated automatically to a different node.
As i always say, do not reinvent the wheel trying to impress people. If someone made it work, give them credit and use their material. Follow the guides i linked in the references section and you'll end up with something like this:
I used The Live Kali linux OS as a test machine since it comes packed with all the networking tools to debug any problem.
For this setup you'll have three network interfaces, a WAN facing the internet, a LAN for the internal network and a SYNC interface to enable the two firewalls to communicate.
On the LAN interface, you'll create a shared virtual IP that all VMs can use as a gateway and a dns server.
With that, all your traffic comes and goes through PFsense, the IP is virtual, so the fact that the Master node goes down isn't an issue since it's abstracted and irrelevant to the VMs who will still be using the same IP.
- The Pfsense Book. pdf version
Example:
cd /var/lib/vz/template/iso
wget https://cdimage.kali.org/kali-2020.1b/kali-linux-2020.1b-live-amd64.iso
- If you're planing on using HAproxy on pfsense consider using more CPU and RAM than described in Internetz tutorial. The disk space isn't that important though. How To
There's a lot more to cover and optimize, up to the application level.
It is worth noting that security isn't always technical, the weakest link in the chain is the human element. You can have an armored door, yet if the one holding the key is not well informed, your efforts are in vain.
Therefore the first step to secure a system, is adapting a secure mindset.
Even though that set security a milestone back, i think it was a wise decision to make. The system was complex enough for them to care about securing it, also there's the vision of the cloud. Where developers don't have to focus on provisioning k8s instances as much as providing solutions to real world problems.
It seems logical like we established a de-facto in the Emailing system, Linux and k8s combined set a new era, a new way of looking at hardware and we should embrace it.
For people like me working with on premise clusters, the attack surface is huge and unheard off, so we battle our way through it and have to convince our bosses with the importance of securing this single point of failure global API.
The reason behind using PFsense
We use Proxmox hypervisor in production, on top of it we have 3 master nodes and 5 workers behind an HAproxy that acts as a loadbalancer.So the whole traffic passes through the load balancer, It made sense to use PFsense as a gateway since it has a built in HAproxy.
Many companies especially in my home country, use PFsense to protect their internal networks, Since it is free of charge and provide enterprise grade features.
Here's a quick cool fact, PFsense took its name from the packet filtering software named "pf" as it was making sense of it. That's dope!
PFsense Allows for all sorts of restrictions and monitoring from a single dashboard written in PHP.
You can have things like VPN access, install certificates, have Virtual IPs for your internal nodes, act as a router and loadbalance traffic as it goes through it, have things like snort and apply intrusion detection and prevention. A complete list of PFsense features can be found here.
Your costumers would be impressed when they ask to see the product and you tell them that they have to install openVPN to get access.
High Availability Synchronization
What this basically mean is that we'll have a main PFsense instance receiving all traffic, and a backup instance that sends heart beats and healthchecks to the master, ready to take the lead if it went offline. Allowing the elimination of the firewall as a single point of failure.As you can see we have two firewalls accessible from the outside world using failover IPs.
If we would simulate the fail-over feature by going to maintenance mode under status > CARP (failover), the roles would be switched in a fraction of a second.
Not everything would be synced from the master to the backed firewall as illustrated in this figure.
This is really useful for zero downtime when you have hardware failures, performing maintenance, rebooting the firewall and so on..
In this tutorial we'll use a single Proxmox server, which doesn't cover disaster recovery strategies. A typical one would involve using the Proxmox HA feature and have two Firewalls on two separate physical nodes joined as a cluster using things like vracks. If the whole server goes down, the backup firewall would take the lead while the other is being migrated automatically to a different node. How To
The setup requires a proxmox server and two failover IPs. Each Firewall will get its own public IP and we have one for the hypervisor. Failover IPs are cheap, i used an OVH dedicated server.As i always say, do not reinvent the wheel trying to impress people. If someone made it work, give them credit and use their material. Follow the guides i linked in the references section and you'll end up with something like this:
I used The Live Kali linux OS as a test machine since it comes packed with all the networking tools to debug any problem.
For this setup you'll have three network interfaces, a WAN facing the internet, a LAN for the internal network and a SYNC interface to enable the two firewalls to communicate.
On the LAN interface, you'll create a shared virtual IP that all VMs can use as a gateway and a dns server.
With that, all your traffic comes and goes through PFsense, the IP is virtual, so the fact that the Master node goes down isn't an issue since it's abstracted and irrelevant to the VMs who will still be using the same IP.
References:
- Pfsense 101 by Internetz.me- The Pfsense Book. pdf version
Tips:
- If you're using a hosted server like myself, don't upload ISOs from the dashboard as that will eat a lot of bandwidth. What you can do instead, is ssh into proxmox, position yourself in the ISOs directory and wget the desired image.Example:
cd /var/lib/vz/template/iso
wget https://cdimage.kali.org/kali-2020.1b/kali-linux-2020.1b-live-amd64.iso
- If you're planing on using HAproxy on pfsense consider using more CPU and RAM than described in Internetz tutorial. The disk space isn't that important though. How To
Conclusion
Securing access to your infrastructure is a primary step towards security in depth. Which is a principle that involves integrating security in all layers of your environment.There's a lot more to cover and optimize, up to the application level.
It is worth noting that security isn't always technical, the weakest link in the chain is the human element. You can have an armored door, yet if the one holding the key is not well informed, your efforts are in vain.
Therefore the first step to secure a system, is adapting a secure mindset.
Comments
Post a Comment