Your Kubernetes OIDC issuer is just two static files

How we brought on-prem Kubernetes clusters into an AKS-based GitOps platform without rebuilding a control plane every time we created one.

One platform, two kinds of hardware

Our developer platform runs on managed Kubernetes in the cloud, on AKS, and it is driven by GitOps. A team describes a service in a registry entry, Argo CD picks it up, and the service appears in the right namespace on the right cluster. Nobody files a ticket, and nobody runs kubectl by hand.

Not everything belongs in the cloud, though. Some workloads need to sit close to machines on the shop floor, some data is not allowed to leave the building, and some things are simply cheaper on hardware we have already paid for. So we also run Kubernetes clusters in our own data centre, on vSphere. Early on we decided those clusters would not be a second platform with its own rules. They join the existing one.

That word "join" carries most of the design. What we are after is roaming: a service should be able to run on a cloud cluster today and an on-prem cluster tomorrow, chosen for cost, capacity, latency or a maintenance window, without the team that owns the service changing anything about it. Same registry entry, same chart, same secrets, same access to the same resources. Placement becomes a scheduling decision that a platform engineer, or eventually a controller, can make and reverse.

Roaming only works if identity is the same on both sides. If the cloud copy of a service authenticates with a federated identity while the on-prem copy authenticates with a client secret sitting in a Kubernetes Secret, then they are not the same service. You have two deployment variants, two rotation stories, two RBAC models, and moving between clusters turns into a small migration project. The moment identity diverges, roaming stops being scheduling and becomes porting.

So the requirement was one identity source for every cluster, wherever the hardware happens to be. RBAC and secret access both hang off it: the same Entra identity, the same role assignments, the same path to the same key vault. In practice that means projecting the on-prem clusters into the cloud identity plane, which is what a connected-cluster agent is for. We use Azure Arc. It gives an on-prem cluster a presence in the cloud control plane, so it can carry the same identity and policy constructs as a managed cluster. Arc is not there to run the cluster. It is there so that a service does not have to care which cluster it landed on.

Clusters are built by a pipeline, not by people

The other half of the platform promise is that clusters themselves are cheap to create and cheap to throw away. On-prem, that is Cluster API: a management cluster holds the Cluster, KubeadmControlPlane and MachineDeployment objects, and the vSphere provider clones VMs from a template and runs kubeadm on them. Provisioning a cluster is a workflow run with a cluster name as its input.

This matters more than it first sounds. If creating a cluster needs an engineer to sit and watch it, the cluster quietly turns into a pet. You stop rebuilding it, you stop testing the rebuild, configuration drift accumulates, and the disaster recovery plan becomes a document instead of a procedure. Unattended creation is what keeps clusters disposable, and disposable clusters are what make roaming credible in the first place.

Two requirements, then. Every cluster joins the same identity plane, and every cluster can be created from nothing by a single automated run.

Where the pieces sit. The provisioning workflow drives a Cluster API management plane in the cloud, which builds virtual machines on premises that become a workload cluster. Argo CD deploys to it, Arc projects it into the cloud identity plane, and the cluster's OIDC documents are published to a public endpoint the identity provider can read.

Those two turned out to fight each other, and the fight was about the OIDC issuer.

What federation actually asks of you

The mechanism that gives a pod a cloud identity without a stored secret is workload identity. A pod asks its own cluster for a short-lived token, hands that token to the cloud identity provider, and gets a real access token back. This is what we wanted on both sides of the platform, because it is the only way a service keeps the same identity wiring whether it lands on AKS or on our own hardware.

Strip away the product names, and it needs three things.

Your cluster has to stamp its tokens with an issuer identity, which is just a URL. The API server puts it in the iss claim of every service account token it signs.

That URL has to serve two documents, and the cloud identity provider has to be able to reach them:

https://<issuer>/.well-known/openid-configuration
https://<issuer>/openid/v1/jwks

Finally, you register a federated credential on the cloud side saying "I trust tokens from this issuer, for this exact service account, and I will exchange them for access to this identity."

That's the whole contract. Keep it in mind, because the shape of it is what eventually solved our problem.

The path we tried first

Arc has a feature for exactly this. You enable an OIDC issuer on the connected cluster, and the cloud hosts the discovery documents for you, backed by your cluster's own signing keys. Publicly reachable, managed, nothing for us to run or monitor. We already needed Arc for the identity plane, so using the issuer it came with looked like getting something for free. We took it.

The problem is ordering, and it took us a while to see it clearly.

To attach a cluster, the cluster has to exist and be healthy. The issuer URL is generated during or after that attachment, and it contains a generated identifier, so you cannot predict it. You only learn your issuer URL once the cluster is already running.

But --service-account-issuer is an API server flag. It has to be set to that URL, and on a Cluster API cluster the API server flags live in the KubeadmControlPlane spec. Editing that spec does not restart a process. It triggers a rolling replacement of every control plane machine.

It gets worse, because you cannot flip the issuer in one move. Tokens already in flight were signed under the old issuer, and controllers holding them will fail the moment the API server stops recognising it. The safe sequence is two phases: first make the API server accept both issuers with the old one primary, then make it accept both with the new one primary. Two edits to the control plane spec. Two full rollouts.

On vSphere, one control plane rollout means cloning a VM, booting it, joining it with kubeadm, waiting for etcd to accept the new member, waiting for the old member to leave cleanly, and repeating for each node. With three control plane nodes, two phases is six machine replacements. When everything worked it took most of an afternoon. When it didn't, and it often didn't, we were debugging a half-migrated cluster where some components authenticated fine and others were rejected, while etcd quorum sat somewhere in the middle of the operation.

This is where the two requirements collided head-on. Our provisioning workflow is supposed to be a single run: render the Cluster API objects, wait for the API server, install the CNI, write the cluster access contract, register the cluster with Argo CD, finish. The Arc issuer forced a second act that could not live inside that run, because it depended on a value that did not exist until the first act had already completed.

We automated the second act anyway. The result was a workflow that built a control plane and then immediately started tearing down the control plane it had just built, with a two-phase migration in the middle and no safe point to resume from if a machine got stuck. On paper, cluster creation was automated. In practice somebody watched every run, and when it went wrong they were recovering a cluster that was minutes old.

The part that finally pushed us to change direction was not the flakiness. It was that we had accepted a design where joining the identity plane cost us unattended provisioning. Those were the two things the platform was built on, and we were trading one for the other.

The thing we should have noticed sooner

We spent a while trying to make the rollout more reliable. Better waits, better retry logic, better detection of stuck machines. Then someone asked a more useful question: what does the identity provider actually fetch from that issuer URL?

Two JSON documents. That's it.

The discovery document is a few lines of metadata. The JWKS is the public half of the key pair the API server uses to sign service account tokens. Neither is dynamic. The identity provider never talks to your API server, never opens a connection into your network, and has no idea whether your cluster is even running. It reads two static files over HTTPS and uses the public key to verify a signature on a token the pod handed it directly.

Once you see it that way, the constraint inverts. The issuer URL is not something a platform grants you. It is a name you choose, and the only requirements are that you can serve two files at that name and that your API server signs tokens claiming it.

We had been treating an interface as a service.

Choosing the URL first

So we turned the order around. The issuer URL is now declared in our cluster inventory before the cluster exists. That inventory is an HCL file read by OpenTofu, the open source Terraform fork we run:

access = {
  server          = "https://10.10.0.1:6443"
  oidc_issuer_url = "https://oidc.example.org/clusters/cluster-a"
}

Provisioning renders that value straight into the KubeadmControlPlane that Cluster API builds the cluster from, so the very first API server that ever starts already has its permanent identity:

apiServer:
  extraArgs:
    - name: service-account-issuer
      value: https://oidc.example.org/clusters/cluster-a
    - name: service-account-jwks-uri
      value: https://oidc.example.org/clusters/cluster-a/openid/v1/jwks
    - name: service-account-signing-key-file
      value: /etc/kubernetes/pki/sa.key

The service-account-jwks-uri flag is easy to overlook and matters a lot. Without it, the cluster's own discovery document advertises the API server's internal address as the place to fetch keys, and the document you publish will contradict itself.

No mutation after the fact, no phased migration, no rollout. The cluster is born with the identity it keeps for life.

Publishing the two documents

The API server sits on a private network and is not reachable from the internet, which is exactly how we want it. The identity provider is on the internet. So the last step of provisioning copies the documents out to somewhere public.

Read them from the fresh cluster:

openid_configuration="$(kubectl get --raw /.well-known/openid-configuration)"
jwks="$(kubectl get --raw /openid/v1/jwks)"

Rewrite the self-references to the public URL:

openid_configuration="$(jq \
  --arg issuer   "https://oidc.example.org/clusters/cluster-a" \
  --arg jwks_uri "https://oidc.example.org/clusters/cluster-a/openid/v1/jwks" \
  '.issuer = $issuer | .jwks_uri = $jwks_uri' <<<"$openid_configuration")"

That rewrite is not cosmetic. The identity provider compares the issuer field in the document against the URL it requested and against the iss claim in the token. If any of the three disagree, the exchange is refused.

Then we serve them. Both documents go into a ConfigMap on a cluster that already has public ingress, mounted into a plain nginx pod. The file layout mirrors the URL path, so a single static root serves every cluster we have:

/usr/share/nginx/html/clusters/cluster-a/.well-known/openid-configuration
/usr/share/nginx/html/clusters/cluster-a/openid/v1/jwks

One hostname, oidc.example.org, with a route per cluster on the path prefix /clusters/<name>. Onboarding another cluster adds a path prefix. It does not add infrastructure.

This part surprises people, so I will say it plainly: none of this is secret. A JWKS contains public keys. The discovery document is metadata. Serving both from an unauthenticated nginx pod is the correct design and not a shortcut. The private signing key never leaves the control plane nodes.

Connecting it to the cloud

With the issuer stable and reachable, the federated credential is ordinary OpenTofu against the Azure provider:

resource "azurerm_federated_identity_credential" "external_dns" {
  name                = "fic-cluster-a-external-dns"
  resource_group_name = var.resource_group_name
  parent_id           = azurerm_user_assigned_identity.external_dns.id
  issuer              = "https://oidc.example.org/clusters/cluster-a"
  subject             = "system:serviceaccount:external-dns:external-dns"
  audience            = ["api://AzureADTokenExchange"]
}

Inside the cluster we install the upstream workload identity webhook ourselves. It injects the projected token and the environment variables the cloud SDKs look for. We had previously been using the version Arc installs, and switching that off in favour of the upstream chart was part of the same change.

Arc itself stays exactly where it was, and that is deliberate. It still projects the cluster into the cloud control plane, still backs the inventory and policy view, and is still the reason a service gets the same role assignments on-prem that it gets on AKS. Everything in the first half of this article still depends on it. What Arc no longer does is own the signing identity of the cluster. We took one responsibility away from it, the single one that sat on the provisioning critical path, and left the rest untouched.

The runtime path is short. A pod asks its own API server for a token. It presents that token to the identity provider. The provider fetches our two files, checks the signature against the published key, checks that the issuer and the service account subject match a registered credential, and returns an access token. The only thing crossing the internet on the inbound side is a request for two static documents.

This is the point where the platform goal and the plumbing finally meet. A service is defined once and gets the same identity treatment wherever it is scheduled. Its service account name is the subject of a federated credential on AKS and on-prem alike, its role assignments are the same role assignments, and it reads secrets from the same place by the same mechanism. The only thing that differs between the two is which issuer signed the token, and nothing in the service description mentions that. Moving a workload is a placement decision again, which is what we wanted from the beginning. Meanwhile a fresh on-prem cluster is one workflow run, start to finish, with nobody watching it.

What it cost us

I would rather read an honest account than a triumphant one, so here is the other side.

You now own an endpoint. If oidc.example.org is unreachable, no on-prem workload can obtain a new cloud token. Access tokens already issued keep working until they expire, so it is not an instant outage, but it is a dependency you did not have before. Put it behind the same ingress and monitoring as your other public services and treat it as production, because it is.

Key rotation becomes your job. What you publish is a snapshot of the JWKS taken at provisioning time. If the control plane's service account signing keys are ever regenerated, the published keys go stale and every federation on that cluster breaks. The failure appears on the cloud side as a signature validation error, which is nobody's first place to look. If you build this, add a scheduled job that re-reads the live JWKS and republishes when it changes, and build it at the same time as the rest. We treated it as a follow-up, and it is the one decision I would take back.

Someone has to understand it. A managed issuer is a checkbox in a portal. This is a script, a ConfigMap and an ingress route. Write down where those documents come from and what regenerates them, or the next engineer will find an nginx pod serving JSON with no obvious owner and quite reasonably assume it is safe to delete.

Would I do it again

For this shape of problem, yes, without hesitating.

The deciding factor was never self-hosted versus managed. It was when the issuer URL becomes knowable. Any issuer that only tells you its URL after the cluster is running forces you to change the API server's identity after the fact, and on a declaratively managed control plane that means replacing machines. If the URL is known before you run the first command, that entire class of work disappears. On a managed cloud cluster the provider hands you the issuer at creation time, which is why nobody hits this there. On your own hardware, with an issuer bolted on afterwards, you hit it immediately.

The wider lesson is smaller and more embarrassing than the technical one. We assumed the OIDC issuer was a service, so we went looking for a service that would give us one, and then spent weeks making a bad ordering work. It is a contract: a URL, two JSON documents, and a signing key you already have. Reading the specification instead of the product page would have saved us all of those weeks.