I recently spent a Saturday setting up K3S, a lightweight Kubernetes distribution. I repurposed a few old computers turning them into agents with 500 gigabytes each of hard drive space and needed to make the space available to containers in the cluster. To make this happen I used Longhorn, a distributed block storage system for Kubernetes. With the exception of one gotcha, the set up was really simple. Below is a quick guide on how to set it up and attach a volume to resources in your cluster.
Disclaimer, my setup in this cluster is simple because it is just a home lab cluster. I don't have need for backups, although redundancy is kind of baked into Longhorn. The first time I created a volume and attached to a deployment it tried to reserve 3 times the disk space I requested, by default it creates 3 replicas! Below I'm going to create a custom storage class in Kubernetes that creates just 1 replica, but in a production environment that redundancy would be very useful and I love that Longhorn defaults with so many replicas on a volume.
Prerequisites
- Understanding of kubectl, Kubernetes' cli tool
- You need a Kubernetes cluster, not necessarily K3S, but I recommend it for experimentation
- An option could be to spin up a few ubuntu VMs in DigitalOcean with K3S and volumes to follow along
- On your cluster you need one or more ubuntu agents with disk space available on them
- General understanding of how storage in Kubernetes works
- An understanding of helm and how to use it
Step 1 - Install Longhorn via Helm
Longhorn is available as a helm chart, longhorn/longhorn, I didn't make any modifications in my cluster but of course you may need to depending on your specific needs. Make sure your kubectl context is set to the cluster you want to do this with, and run the following commands:
I used the namespace provided by the helm chart documentation also, longhorn-system, this can be changed of course.
Note: If you'd like to have the UI available via an ingress, you can modify the helm installation with ingress settings, your custom settings might look something like the yaml below, just run the command above with -f <path to file with the settings below> argument appended:
But that isn't a necessary step, everything after this will still work correctly.
Step 2 - The Gotcha, iSCSI
If you're anything like me you just try things. The first time I installed the helm chart I ran into some networking issues (the agents couldn't reach the control plane), but that was a K3S networking issue I had to work through. The second time one of the Longhorn pods wasn't able to start, because I skipped the rather long Quick Installation Guide (which I should have at least skimmed 😅). In the guide, it states you need the open-iscsi package installed on your linux node. I'm using previously installed and upgraded ubuntu servers, so I didn't have this dependency on either node.
To fix the failing pod, run this command on each node:
Now go restart the container and the helm charts deployment shows green in every pod.
Step 3 - Create a Storage Class
So, by default the helm chart installs a storage class named longhorn. And you can use this for PersistentVolumeClaims, but as I mentioned above, this creates 3 replicas of the data in your volume. If that's ok with you, skip the following steps and move forward with your project. I don't want this redundancy in my home lab right now, so I'm going to create another storage class named longhorn-one-replica, using this configuration:
Now, you should be able to create a PersistentVolumeClaim using this storage class:
Conclusion
And now you know how to set up Longhorn, we walked through helm chart installation instructions (with optional ingress settings), debugged the open-iscsi dependency together, and (optionally) created a new storage class to remove redundant replicas. This has been a useful add-on in my home cluster already and a worthwhile set up! Happy coding 🧑💻