4 Nodes Kubernetes Cluster Behind A PFsense VM On Proxmox Hypervisor
Securing a Kubernetes cluster can be overwhelming especially for companies and individuals who have their own infrastructure.
For startups or people with no prior knowledge on k8s, i would recommend using a managed service by a cloud provider for your production services at least.
On the other hand for people like me who like to have control and break things over and over again to understand them, we tend to build things on our own.
After a month of trial and error, this is what i settled on. I don't consider this design production ready yet. Regardless, i made extensive research that made me stop looking further.
The default PFsense installation alone can give you a huge advantage, on its own. Yet you can still install other packages with few clicks, which are periodically patched and updated by Netgate, the company behind PFsense.
The choice for PFsense brings a lot of potential to the table. For example using the built in HAproxy i can have multiple masters for High Availability and split traffic between them. Or maybe multiple clusters.
I setup a DHCP server, allowing my virtual machines to request IPs from PFsense.
Kubernetes design, relies on a flat network. Which made sense to use static dhcp leases for the nodes. I Reduced the DHCP range to allow Dynamic and Static leases. That way other Virtual Machines can still use DHCP and coexist with the cluster without any problem.
You can further segment your network by creating a DMZ or a different gateway for the LAN interface.
That way attackers who gained access to one of your other virtual machines, can't go further.
Since i used an OVH server, i used what they call Fail-Over IPs.These IPs are referenced by their Mac addresses inside OVH network.
I linked the WAN interface of PFsense with the first NIC through a linux bridge and the LAN interface with the second one.
All VMs share a Linux bridge "vmbr1" to be able to communicate with one another.
Then i made all Virtual machines use PFsense as a gateway on a different network.
I like to use Debian whenever i had a chance. It is very lightweight and one of the most stable Linux distributions out-there.
I only installed the base operating system and open-ssh server.
Since we're learning, we'll go with a basic setup then secure it on our own.
Here's a great and straightforward tutorial. No need to look any further.
To allow external access to the cluster i added a NAT rule in pfsense using aliases.
One thing to note is that you should add your PFsense public IP to the kubeadm command to allow external access. Otherwise you won't be able to use the kubectl command from your Laptop at home.
kubeadm init --apiserver-cert-extra-sans=<PFSENSE_PUBLIC_IP>
Now I can access the cluster from anywhere.
kube-bench is a Go application that checks whether Kubernetes is deployed securely by running the checks documented in the CIS Kubernetes Benchmark.
The first time i installed it on the master node i had some problems to fix.
The tool gives you remediation steps that you can follow. From setting up policies for pods, configuring audit logs or enabling TLS between nodes.
So i would typically enable ssh access on a different port every time i needed to debug something then disable it. Thus reducing the attack surface.
I also disabled root login, and allowed the k8s user certain privileges only.
Once again, you should not consider this setup production ready. There are many other areas to cover.
Till next time stay safe.
For startups or people with no prior knowledge on k8s, i would recommend using a managed service by a cloud provider for your production services at least.
On the other hand for people like me who like to have control and break things over and over again to understand them, we tend to build things on our own.
After a month of trial and error, this is what i settled on. I don't consider this design production ready yet. Regardless, i made extensive research that made me stop looking further.
Why PFsense?
The pfSense project is a free network firewall distribution, based on the FreeBSD operating system with a custom kernel and including third party free software packages for additional functionality. Shipped with a nice dashboard that makes it easier and more convenient to apply rules, update configuration,or monitor all traffic... |
| Monitoring Dashboard |
 |
| PFsense Package Manager |
The choice for PFsense brings a lot of potential to the table. For example using the built in HAproxy i can have multiple masters for High Availability and split traffic between them. Or maybe multiple clusters.
I setup a DHCP server, allowing my virtual machines to request IPs from PFsense.
Kubernetes design, relies on a flat network. Which made sense to use static dhcp leases for the nodes. I Reduced the DHCP range to allow Dynamic and Static leases. That way other Virtual Machines can still use DHCP and coexist with the cluster without any problem.
You can further segment your network by creating a DMZ or a different gateway for the LAN interface.
That way attackers who gained access to one of your other virtual machines, can't go further.
Configuring The Network
When it comes to configuring the network, Dan Ford a fellow researcher made a great article on how to setup PFsense on Proxmox. I made some tweaks to be able to use a public IP for PFsense.Since i used an OVH server, i used what they call Fail-Over IPs.These IPs are referenced by their Mac addresses inside OVH network.
I linked the WAN interface of PFsense with the first NIC through a linux bridge and the LAN interface with the second one.
Then i made all Virtual machines use PFsense as a gateway on a different network.
Choosing an Operating System
When choosing an OS my requirements where simple, "stable". I didn't want any preinstalled packages, user interfaces, or any of sort. Just a base operating system with long term support.I like to use Debian whenever i had a chance. It is very lightweight and one of the most stable Linux distributions out-there.
I only installed the base operating system and open-ssh server.
Installing Kubernetes
There are million ways to install k8s since it is a distributed and highly customizable system.Since we're learning, we'll go with a basic setup then secure it on our own.
Here's a great and straightforward tutorial. No need to look any further.
To allow external access to the cluster i added a NAT rule in pfsense using aliases.
One thing to note is that you should add your PFsense public IP to the kubeadm command to allow external access. Otherwise you won't be able to use the kubectl command from your Laptop at home.
kubeadm init --apiserver-cert-extra-sans=<PFSENSE_PUBLIC_IP>
Now I can access the cluster from anywhere.
kube-bench is a Go application that checks whether Kubernetes is deployed securely by running the checks documented in the CIS Kubernetes Benchmark.
The first time i installed it on the master node i had some problems to fix.
The tool gives you remediation steps that you can follow. From setting up policies for pods, configuring audit logs or enabling TLS between nodes.
SSH Access
The cool thing about this setup is that i can enable/disable ssh access from the dashboard, use a Virtual Private Network, etc..
So i would typically enable ssh access on a different port every time i needed to debug something then disable it. Thus reducing the attack surface.
I also disabled root login, and allowed the k8s user certain privileges only.
Final Word
I hope this article was beneficial for anyone looking to setup a development environment for his team or experiment with Kubernetes in a secure fashion.Once again, you should not consider this setup production ready. There are many other areas to cover.
Till next time stay safe.
Great information. Thanks for sharing
ReplyDeleteSelenium Training in Bangalore