Using Cloudflare Tunnels to Securely Expose Kubernetes Services

Image source: https://www.cloudflare.com/products/tunnel/

Cloudflare Tunnel (previously known as Argo Tunnel) is a tool that allows a private and secure connection between your web server and Cloudflare infrastructure. If you are not familiar with Cloudflare, I suggest you check out their website as they offer a ton of services, the most important of which is their CDN network and web service protection (DDoS protection, etc.). For this tutorial to work, you need to use Cloudflare as your DNS server. This will allow them to control how traffic gets routed for your domain.

In this tutorial, I will show you how to set up a Cloudflare tunnel to expose Kubernetes services securely over the internet. If any of the words I just mentioned didn’t make sense to you, keep on reading, I promise I will do my best to explain them. I just assume you know what Kubernetes is. If you are unfamiliar with Kubernetes, do a quick google search and then use my tutorial to set up your cluster in a few minutes on a VM and you should be able to follow along.

Now the big question is: why would you want to do this? Or who would benefit from this? It might not seem very clear at first, but it enables a ton of capabilities, the most important of which is security. I personally used Cloudflare tunnels for 3 purposes: 1) Expose services from clusters that don’t have static IP and/or are sitting behind a NAT (my home lab); 2) Protect running web servers from direct attack; 3) Leverage Cloudflare Access Zero Trust services to add an additional layer of security to sensitive services.

This tutorial is a part of my personal growth to improve the security of the infrastructure I am using to host my projects and self-hosted services. I am now running about 20 services on my own infrastructure, and as time goes by I am becoming more conscious about the security of these services. I also wanted to point out that if you are running a managed Kubernetes service (e.g., from AWS or GCP) you probably run your services behind managed load balancers and services like Cloud Armor and most of these use cases won’t apply to you, but you are welcome to continue reading.

The Problem

Let’s dissect the problem we are trying to solve here in a bit more detail.

1. Clusters running behind NAT

One way I managed to stay sane during the pandemic was to create my personal home lab where I host services like Home Assistant to support smart devices in my home. Before Cloudflare Tunnels, to allow remote access to these services you would have to set up a dynamic DNS (using services like Duck DNS) that points a domain to your home IP and expose specific ports on your home firewall (typically using port forwarding capabilities of your modem if your provider allows you to).

Now, this brings out a few issues. First, you have made your home IP public on the internet, and from a security point of view, we want to protect our privacy in any way possible. Second, you are allowing traffic to enter your home network, which makes me uncomfortable. In addition, this might not even be possible for many internet service providers as they won’t allow you to configure port forwarding at all. Cloudflare Tunnel solves this by punching out a tunnel connection to Cloudflare servers. When a request hits their servers for your service, they will route that traffic through this tunnel and securely into your infrastructure.

2. Protect running web servers from direct attack

image source: https://www.cloudflare.com/products/tunnel/

Let’s assume you are hosting example.com from your virtual machine with IP 1.2.3.4 that you purchased from a cloud vendor. You probably have a DNS A-Record pointing your domain to 1.2.3.4. However, you probably have SSH and many more services running on your virtual machine as well. In a perfect world, you have a properly configured SSH agent and firewall at all times and there are no security bugs in any of the services that you use. But we don’t live in a perfect world, and in case you expose any services publicly by mistake or use bad SSH configurations, the attackers know your VM’s IP address.

This is solved here by forwarding all traffic to Cloudflare servers and they will route the traffic to the Cloudflare tunnel agent running on your VM. There is no need for you to expose the IP of your VM. In fact, you don’t even have to allow any traffic through your firewall.

3. Use Cloudflare Access

image source: https://www.cloudflare.com/products/tunnel/

As I mentioned, I self-host many web applications, some of which hold rather sensitive data. I initially exposed these services with Nginx basic authentication (in the load balancer) and a password (in the application). But as we know, basic authentication is not secure and I wanted to replace this with a better alternative that uses identity providers like GitHub or Google to use the services. This is when I came across Cloudflare Access, their hosted Zero Trust security services that allow you to add several rules to limit access to services running in your infrastructure. You could initially have your traffic proxied through Cloudflare:

And this would work perfectly, traffic for secret.nima-dev.com would be routed to Cloudflare and they would apply the security rules and require authentication for the protected endpoints. However, for this to work, you need to allow HTTP/HTTPS traffic in your firewall, anyone can send a direct request to your server and bypass Cloudflare authentication altogether. When using Cloudflare Tunnel, you don’t need to have any ingress rules for the protected service. Traffic is securely tunnelled to the agent running in the cluster and then is routed to your service. This also allows me to expose unsecured applications (like Homer dashboard) to the internet securely and with a few clicks in my Cloudflare Teams dashboard.

Now that we know why we might want to use Cloudflare tunnels, let’s see how you can set it up for your own cluster.

Installation Process

In a previous post, I went over the process to create a K3S cluster on a virtual machine that you can purchase from any cloud vendor (or host yourself). Here, I assume that you have a functional Kubernetes cluster and you have a basic understanding of its terminology (deployment, service, ingress, etc.). Now that we are ready, let’s create a tunnel to securely expose a service named web in the default namespace. As a result, internally (from within the cluster), we can refer to this service as web.default.svc.cluster.local(the general pattern is my-service.my-namespace.svc.cluster.local). If you don’t know about Kubernetes DNS for Services, check this page out. We will now deploy a tunnel to route traffic to this service. The process can be done in two steps: configuring the tunnel and deploying it to Kubernetes.

Cloudflare Tunnel Configuration

To configure the Kubernetes deployment, we will need the tunnel agent’s private key stored in a file named cert.pem, the tunnel’s info stored in a file named tunnel.json, and a configuration file stored in a file named config.yml. To get these, you will need to ssh into your VM and follow the Cloudflare Tunnel Getting Started guide. The process is rather straightforward, so I won’t go into its details here, but here is the summary:

[embed]

After this process, you have logged in (generates cert.pem) and created the tunnel (generates the tunnel JSON file). You have also created the DNS rule to forward traffic to your Cloudflare Tunnel, you can verify that by going to your Cloudflare dashboard. There should be a new DNS CNAME record routing your hostname (e.g., secure.nima-dev.com) to TUNNEL_UUID.cfargotunnel.com that is proxied through Cloudflare. Now you need to create your configuration config.yml file. This file tells the tunnel where each request should be routed and where the tunnel JSON file is located. The following configuration file would work for our example:

[embed]

For more complicated configurations you can go to the Cloudflare documentation. Now that we have all files that we need, it is time to gather them and create the Kubernetes deployment. If you take a look at the ~/.cloudflared folder in the VM, you should now have cert.pem and TUNNEL_UUID.json files ready. We have also created our config.yml.

Kubernetes Deployment

Now, that we have everything ready to go, let’s prepare our Kubernetes deployment. Create the following folder structure:

.
├── config
│   ├── cert.pem
│   └── tunnel.json
├── configmap.yml
└── deployment.yml

The cert.pem and tunnel.json should come from the previous step. The configmap.yml includes the configuration, it should be something like the following:

[embed]

The deployment.yml should be something like the following. Just make sure to replace the $CLOUDFLARE_TUNNEL_NAME with the tunnel name that you used:

[embed]

Now that everything is ready to go, let’s deploy this to our Kubernetes cluster:

[embed]

After a couple of minutes, you should see something like this in the logs:

2022-01-22T19:17:40Z INF Connection XXXXXXXXX registered connIndex=0 location=AMS
2022-01-22T19:17:40Z INF Connection XXXXXXXXX registered connIndex=1 location=FRA
2022-01-22T19:17:41Z INF Connection XXXXXXXXX registered connIndex=2 location=AMS
2022-01-22T19:17:43Z INF Connection XXXXXXXXX registered connIndex=3 location=FRA

This means that the deployment has been successful and everything should be working. You can now visit the hostname you specified to see the end result.

Docker Image

In our deployment, I used my own docker image for Cloudflare. Try to update the image tag in deployment.yml every now and then to use the latest version. Also, know that you could use the cloudflared official image with little tweaks, but I created my own because the official image didn’t support ARM architecture and I wanted to also run this on my raspberry pi.

Conclusion

In this tutorial, you learned how to expose your Kubernetes services securely to the internet using Cloudflare Tunnels. From there, there is a lot you can do with Cloudfare services most of which include very generous free tiers. Personally, I really enjoyed the peace of mind and simple authentication managed by Cloudflare for my deployments. If you like to see tutorials like this about Cloudflare Access to add authentication for these services, let me know in the comments.

About Me

I am a Ph.D. candidate at the University of Alberta and a visiting researcher and a part-time Instructor at York University. Day-in day-out I research serverless computing platforms, trying to find ways to improve their performance, reliability, energy consumption, etc., using analytical or data-driven methods (fancy words for “I either use mathematics or machine learning to model serverless computing platforms”). What I wrote here is the result of my insight into some of the serverless computing platforms that I have worked with during my research and a brief compilation of their documentation regarding their autoscaling patterns.

In case you want to know more about me, check out my website.