This week I chose to upgrade cert-manager (an automated SSL manager for your domains) on an AWS Elastic Kubernetes Service cluster using Fargate nodes. These clusters were the first I had ever built on Fargate, but not my first for cert-manager, turns out they both use port 10250. Fargate nodes kubelet and cert-manager's webhook pod conflict. But there's a fix.

First a disclaimer, this is a pretty low risk update to the cluster depending on when you perform the operation. Cert-manager saves the generated SSL certificates to a Kubernetes secret, so as long as your current cert-manager job is not renewing not much will break. Cert-manager runs in the background, and if you're using Let's Encrypt certificates renews every 60 days.

If you're unfamiliar with Fargate, it is a serverless compute engine in AWS. For Elastic Kubernetes Service (EKS) it's an alternative to having servers as agent nodes that can have a multiple individual workloads scheduled, when in use each individual workload has its own dedicated "Fargate node" that has enough resources provisioned when scheduled.

There's an important bit of information about Fargate nodes to mention now, they have a forwarded port for the kubelet (the primary "node agent" that runs on each node) on port 10250. This is how each provisioned node and the cluster control plane are connected and this is the conflicting port for cert-manager webhook.

Cert-manager also relies on a webhook the API server calls on port 10250, it's basically the gatekeeper for issuing a cert. But in a Fargate cluster, the kubelet is reached instead of the webhook and cert-manager cannot complete its process.

The fix is to configure cert-manager to use a different port so I went with 10260.

Without the contextual information above I, about a year ago, in my EKS cluster installed cert-manager with the default kubernetes manifest they provide. I found the problem and fixed it by reconfiguring cert-manager (in retrospect I recall the struggle to find a solution). Some of my present troubles were caused by not using the declarative helm chart for installation, had the me of a year ago used it, I would have had the fix in code and saved to a git repo. But I didn't at the time and having since lost all the context, the present me updated cert manager in my cluster using the same default method:

Then I ran into this confusing error (cmctl is the cli utility provided by cert-manager):

And here's the fix I applied in my EKS cluster to get things working again:

You can see above that I first delete the old webhook (which is re-deployed by the update) then I replace the ports with our non-conflicted 10260. Now certs renew without a problem.

And if you wanted to use helm so you can remember in a year:

A port change like this is a small detail and easy to forget, but I think my own takeaway with is use helm and save to a repository, or at least document well. Happy coding!