Authentication and user authorization is important and Kubernetes offers a few strategies. A few weeks ago I posted about Keycloak on Kubernetes, which is an open source Identity and Access Management tool that can be used as an external integration with OpenID Connect for your cluster. This is how you can use your Keycloak as an identity provider for K3s.
Prerequisites
- An understanding of OpenID Connect
- An understanding of Kubeconfig files
- You need an existing K3s cluster to follow along, an option could be to spin up a few ubuntu VMs in DigitalOcean and then run the super simple quick start guide here
- You need Keycloak somewhere, I wrote a guide on installing Keycloak on Kubernetes (in this case I actually installed Keycloak on the K3s instance that I'm integrating with)
- You need a cli tool for logging in with Kubectl called oidc-login
A Bit About Why
When you're working in a larger development team you have members with different roles. To name a few, these can be developers, QA, DevOps engineers, leadership, SREs, or release management. Each of these team members may need access at some level to the cluster and using an external integration like Keycloak can help manage who gets the access they need, even if it's just read only access.
In a managed Kubernetes service in the cloud such as DOKS (Digital Ocean Kubernetes), roles within the cloud provider can be associated with roles in the cluster (read more here in DigitalOcean's official documentation). So any existing roles you have set up for your team in the provider nicely translates over to the cluster with some configuration in Kubernetes and this is the case in other cloud providers as well.
But if you've taken a self-managed approach to Kubernetes and are working with K3s, you don't have the built-in IAM that comes along with the cloud provider. This is why it can be useful to have a Keycloak instance for your IAM, where you can easily manage users access and change user access level when you need to. And the good news is it's really not that difficult to set up!
Step 1 - Configure Keycloak Client
We need to work with Keycloak first, follow the steps below to create a client that can be used by K3s:
1. Log in to your Keycloak instance
2. Create a realm for your K3s instance, I will name mine k3s-cluster. Keep enabled checked
3. In the navigation under your realm click Clients, and click the Create Client button
4. Give your client a name and click next, I will name mine cluster-user
5. In the next form turn on Client authentication (leave authorization off) and make sure Standard flow and Direct access grants are checked, click next
6. Add these two valid redirect URIs: "http://localhost:8000/*" and "http://localhost:18000/*". These are uris that Kubernetes will use as callbacks from the OIDC flow
7. Also, the web origins field should have a value "+" (the plus character means to allow all origins from the redirect uris above)
8. Then click save
After you have created the client, navigate to the Credentials tab and copy the Client secret, save this somewhere you can share with team members (like a password manager). Alternatively, if you are behind a firewall or using Twingate (like me), you can actually turn off Client authentication in step 5, then you won't have to pass around this secret.
Step 2 - Create a Group Membership Mapper in Keycloak
K3s needs a way to associate users in Keycloak and to do that we will pass groups along with the token generated from the OIDC flow. Follow these steps to do that:
1. Open the "Client scopes" tab in the newly created client (cluster-user, or whatever you named it)
2. Click on "cluster-user-dedicated" Assigned client scope table ("cluster-user-dedicated" will be different for you if used a different client name)
3. In the mappers tab, click "Configure a new mapper" then select "Group Membership"
4. Give the Name field the value "groups", this will identify what the user is allowed to do in our policy in K3s
5. Give the Token Claim Name field the same value, "groups", this will be what is in the decoded token
6. Toggle Full group path off
7. Leave everything else default
8. Click save
Now if you click navigate to Groups in your new realm (k3s-cluster), you can create a new group and associate it with any user you create. Here are links to official Keycloak documentation for creating users and creating groups.
Step 3 - Update Your K3s Installation for OIDC
The next step is to update your K3s installation to use your new identity provider. If you haven't made any customizations to your K3s instance, you can shell into the master node for your K3s cluster and run the following command:
If you have made modifications, open the file "/etc/systemd/system/k3s.service" on your master node and append those to the K3s server command with the other options in the "ExecStart" property and restart with "sudo systemctl restart k3s.service". For instance, I already disabled traefik in favor of nignx for ingress, so mine looks like this:
Note: Because this was all local, at this point I had created a self-signed certificate for Keycloak with a certificate authority created with cert-manager. I had to trust my certificate in the K3s master node in order to properly authenticate my user below. Here is a link to an article where I created this same certificate authority in my K3s cluster.
Step 4 - Create Cluster Role Binding
For each role you intend to create in your cluster you need to create a cluster role binding to associate the Keycloak user group in K3s, for example I created the "admins" group so I need to run this command in my cluster so someone with that group can be properly authenticated and authorized:
Step 5 - Authenticate Locally
Now we need to authenticate, you will need to have created a user and created a group. I created a group named "admins" and gave it a user named "benjaminknox". To authenticate locally you need to add the following to your kubeconfig file, generally this is at your home path in the file "~/.kube/config" (make sure to replace values in the configuration to match yours):
Note: This requires the secret from the Keycloak client we created earlier in Step 2 if you are not using a public client.
Now using the context we just added, you should be able to run "kubectl config use-context k3s-cluster-oidc-context" to switch to the context locally. When you run a command like "kubectl get nodes" Keycloak should open and present the user with a login screen.
And that is it! This is great because we can create groups of Keycloak users with specific permissions within our cluster and Kubernetes has really fine grained access control, read more on their documentation.
Conclusion
So let's recap a bit: when you are running a self-managed K3s cluster you don't really have an IAM solution available to you like you would in a managed Kubernetes service in a cloud provider, so we integrated K3s with Keycloak as an external OIDC provider for IAM. Then we authenticated locally as an "admin" user. Very cool! 🔑