In one of my projects I wanted to use an ubuntu VM for hosting my databases, rather than using a managed service. It's easy enough to expose a port to the database on the VM and use it in a Kubernetes cluster. However, I have a bit of paranoia regarding open ports on the internet, I always prefer leaving those ports closed on the firewall in the VM and connecting using security groups inside the VPC. But with this project I had a bit of a problem, my VM was in a different cloud provider than the cluster. 🤔

Private Connections with Twingate

The solution for connecting these two cloud providers is twingate. Twingate is a product that can be used to connect using Zero Trust principles, this is great news if you are paranoid like me. Earlier this year I started using this product at work, it was a drop in replacement for an OpenVPN appliance. Our team is small and we were able to get going quickly with the tool using the Kubernetes connector within our VPC. I liked it so much that I decided to start using it on my personal projects, first I connected to my VM's local network and used it behind the firewall as I was developing. I deployed my application stack to Kubernetes and at first left my database exposed over the internet (I won't tell you which ports I used), but I still couldn't sleep well at night.

Upon further research I found some documentation on twingate for creating machine to machine connections in Kubernetes with a headless client. The article gives a few examples, one of which makes use of the twingate/client docker container. It isn't exactly what I was trying to accomplish though. I want to expose a number of database ports to anything within the cluster so my entire stack could use it, using nginx as a proxy for TCP connections I was able to do that. Below are the steps I took to set up this private cross cloud network.

1. Create Your Twingate Network and Service Account

You must have an existing twingate network set up in your VM and a service account with access to that network. I won't go into detail how to set up a VM or your service account but here are the links: creating a connector on your VM and setting up a service account.

You will need to save the service account key for use in the cluster in a secret, get the service account account key you set up above and save it to a secret. The name of the secret I used below is twingate-databases-for-k8s.

Side note, there are other configurations that will work but the setup below will proxy to a domain that I set up for the VM, tweak to match your own network.

2. Deploying the Headless Client

Below is the full deployment file I used with an example of postgres, mongodb, and redis. It includes an nginx configuration, a deployment of the twingate client and the nginx sidecar for networking, and a service that handles networking in the cluster.

In a nutshell, this Kubernetes manifest enables any service in the cluster to access the database using the following domain: db-example-vm-com.

The first manifest is an nginx config map, you'll see that the nginx config uses twingate DNS resolvers that are local to the twingate client container, this is a key requirement to set up the network correctly. The configuration uses a different upstream server for each database, just to be explicit. Modify this according to your specific network set up.

Skipping to the bottom of the file we have a service that sets up networking for the rest of the cluster, mirroring the ports in the nginx configuration file. This is a crucial step as well:

Finally, the twingate client and the nginx proxy. The manitfest below combines the service account credentials, mounted into the container at /etc/twingate/service_key.json, with the client and the nginx container sidecar for networking to the cluster. This is an adapted version of the documentation provided by twingate, except in my example the twingate client is the main container and I think of nginx as the sidecar because I'm just using it for proxying to the twingate network.

Conclusion

Now you've seen how to set up networking between a VM and your Kubernetes cluster across cloud providers. If you're paranoid like me you will sleep better knowing that database ports are not exposed to the internet! 😇