In this post, we’ll take a look at a new change to Workload Identity Federation on GKE that can reduce the amount of configuration and overhead required for IAM resources, and see it in action with cert-manager using Cloud DNS.
GKE Workload Identity enables a Kubernetes Service Account (KSA) to authenticate with Google Cloud APIs without needing to manage keys or credentials.
By using this feature, the KSA’s token is used to verify its identity and receive a Google Cloud Service Account (GSA) access token which can be used to authenticate and access Google Cloud APIs.
As recommended in the CIS GKE Benchmarks v1.5.0 (5.2.2), users should: ‘Prefer using dedicated Google Cloud Service Accounts and Workload Identity’.
This mitigates against generating, storing and rotating Google Service Account Keys, and adheres to the 5.2.1 recommendation of ‘Ensure GKE clusters are not running using the Compute Engine default service account’ which by default has overly permissive access.
Previously, Workload Identity on GKE used Service Account Impersonation to grant Kubernetes Service Accounts access to Google Cloud APIs by impersonating Google Cloud Service Accounts and inheriting their associated IAM permissions.
Now, Workload Identity Federation for GKE allows Kubernetes Service Accounts to be referenced directly using a principal identifier in IAM Policies without using impersonation.
Before:
$ gcloud iam service-accounts create my-gsa
$ gcloud projects add-iam-policy-binding jetstack-paul --member “serviceAccount:my-gsa@jetstack-paul.iam.gserviceaccount.com” --role “roles/viewer”
$ gcloud iam service-accounts add-iam-policy-binding my-gsa@jetstack-paul.iam.gserviceaccount.com --role roles/iam.workloadIdentityUser --member “serviceAccount:jetstack-paul.svc.id.goog[default/my-ksa]”
$ kubectl create serviceaccount my-ksa
$ kubectl annotate serviceaccount my-ksa iam.gke.io/gcp-service-account=my-gsa@jetstack-paul.iam.gserviceaccount.comAfter:
$ gcloud projects add-iam-policy-binding jetstack-paul --role=roles/viewer --member=principal://iam.googleapis.com/projects/993897508389/locations/global/workloadIdentityPools/jetstack-paul.svc.id.goog/subject/ns/default/sa/my-ksa \
--condition=NoneThe benefits of removing the need to impersonate Google Service Accounts are:
- Fewer IAM Policy bindings to manage
- Previously, each KSA required an IAM Policy Binding to the GSA that granted the
workloadIdentityUserIAM Role to impersonate
- Previously, each KSA required an IAM Policy Binding to the GSA that granted the
- No superfluous GSA
- There is no longer the need for GSAs to impersonate as IAM policy bindings can name KSAs as principals
- No more annotating Kubernetes Service Accounts with the Google Service Account to impersonate
- This can be especially painful when setting annotations for resources in templates, or consuming public templates that don’t support annotations in their inputs
Let’s look at a concrete example of where GKE Workload Identity Federation can come in useful for everyday applications.
To use GKE Workload Identity Federation, create a GKE cluster with a Workload Pool. This is created by default on Autopilot clusters.
$ gcloud container clusters create example \
--location=europe-west2-a \
--workload-pool=jetstack-paul.svc.id.googNext, let’s configure a Kubernetes Service Account to use with cert-manager and Google Cloud DNS.
Creating an IAM Policy Binding can grant a Kubernetes Service Account principal the desired Google Cloud IAM permissions. Previously, these permissions would have been granted to a Google Service Account which the Kubernetes Service Account would impersonate.
$ gcloud projects add-iam-policy-binding project/jetstack-paul \
--role=roles/dns.admin \
--member=principal://iam.googleapis.com/projects/0123456789012/locations/global/workloadIdentityPools/jetstack-paul.svc.id.goog/subject/ns/cert-manager/sa/cert-manager \
--condition=None
When using principal identifiers in IAM Policy Bindings, the Workload Identity Pool is shared across all clusters within a project. This means that if two clusters in the same Workload Identity Pool both contain a Kubernetes Service Account with the same name (in the same namespace), the principal identifier will match both Service Accounts and therefore they will each be granted the same permissions.
This identity sameness stems from there only being a single Workload Identity Pool per Google Cloud Project (at the time of writing) which includes all GKE clusters and therefore all workload identities in these clusters. IAM Policy Conditions can be used to restrict a policy to a specific principal, however, to isolate workload identities then separate Google Cloud Projects and Workload Identity Pools should be used to separate principals across clusters.
With the Kubernetes Service Account now having the necessary IAM Permissions, cert-manager can be deployed using the Kubernetes Service account which will be issued with a token with authorization to access the required Cloud DNS APIs.
helm upgrade --install cert-manager jetstack/cert-manager \
--namespace cert-manager \
--create-namespace \
--set installCRDs=true \
--set global.leaderElection.namespace=cert-manager \
--set extraArgs={--issuer-ambient-credentials}As normal, an Issuer can be deployed that uses Google CloudDNS to solve DNS01 ACME challenges for requested Certificates.
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: cloud-dns
spec:
acme:
email: paul.jones@jetstack.io
server: https://acme-staging-v02.api.letsencrypt.org/directory
privateKeySecretRef:
name: issuer-account-key
solvers:
- dns01:
cloudDNS:
project: jetstack-paul
–--
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: example-com
spec:
secretName: example-com-tls
issuerRef:
name: cloud-dns
dnsNames:
- example.paul-gcp.jetstacker.netAll examples can be found in this repo: https://github.com/paulwilljones/gke-cert-manager-wi-fed
As we have seen, workloads in GKE can now access Google Cloud APIs by having a single IAM policy that directly references the Kubernetes Service Account as a principal.
To create this IAM allow policy, we used gcloud but it can easily be done with any IaC tool that manages Google Cloud resources. See these examples for how to administer Workload Identity Federation on GKE:
Terraform
resource "google_project_iam_custom_role" "cert_manager" {
project = data.google_project.project.project_id
role_id = "certmanagertf"
title = "Cert Manager" permissions = ["dns.resourceRecordSets.create", "dns.resourceRecordSets.list", "dns.resourceRecordSets.get", "dns.resourceRecordSets.update", "dns.resourceRecordSets.delete", "dns.changes.get", "dns.changes.create", "dns.changes.list", "dns.managedZones.list"]
}
resource "google_project_iam_member" "cert_manager" {
project = data.google_project.project.project_id
role = google_project_iam_custom_role.cert_manager.name
member = "principal://iam.googleapis.com/projects/${data.google_project.project.number}/locations/global/workloadIdentityPools/${data.google_project.project.project_id}.svc.id.goog/subject/ns/cert-manager/sa/cert-manager"
}https://github.com/paulwilljones/gke-cert-manager-wi-fed/tree/develop/terraform
Config Connector
apiVersion: iam.cnrm.cloud.google.com/v1beta1
kind: IAMCustomRole
metadata:
annotations:
cnrm.cloud.google.com/project-id: jetstack-paul
name: certmanagerkcc #intentional naming to comply with IAM naming
spec:
title: Cert Manager
resourceID: certmanagerkcc
permissions:
- dns.resourceRecordSets.create
- dns.resourceRecordSets.list
- dns.resourceRecordSets.get
- dns.resourceRecordSets.update
- dns.resourceRecordSets.delete
- dns.changes.get
- dns.changes.create
- dns.changes.list
- dns.managedZones.list
stage: GA
---
apiVersion: iam.cnrm.cloud.google.com/v1beta1
kind: IAMPolicyMember
metadata:
name: cert-manager-kcc
namespace: cert-manager
annotations:
cnrm.cloud.google.com/project-id: jetstack-paul
spec:
member: principal://iam.googleapis.com/projects/993897508389/locations/global/workloadIdentityPools/jetstack-paul.svc.id.goog/subject/ns/cert-manger/sa/cert-manager
role: projects/jetstack-paul/roles/certmanagerkcc
resourceRef:
kind: Project
external: projects/jetstack-paulhttps://github.com/paulwilljones/gke-cert-manager-wi-fed/tree/develop/kcc
Crossplane
apiVersion: cloudplatform.gcp.upbound.io/v1beta1
kind: ProjectIAMCustomRole
metadata:
name: certmanagerxp
spec:
forProvider:
permissions:
- dns.resourceRecordSets.create
- dns.resourceRecordSets.list
- dns.resourceRecordSets.get
- dns.resourceRecordSets.update
- dns.resourceRecordSets.delete
- dns.changes.get
- dns.changes.create
- dns.changes.list
- dns.managedZones.list
title: Cert Manager (Crossplane)
–--
apiVersion: cloudplatform.gcp.upbound.io/v1beta1
kind: ProjectIAMMember
metadata:
name: cert-manager-xp
namespace: cert-manager
spec:
forProvider:
project: jetstack-paul
member: principal://iam.googleapis.com/projects/993897508389/locations/global/workloadIdentityPools/jetstack-paul.svc.id.goog/subject/ns/cert-manager/sa/cert-manager
role: projects/jetstack-paul/roles/certmanagerxphttps://github.com/paulwilljones/gke-cert-manager-wi-fed/tree/develop/crossplane
Pulumi
cert_manager_role = gcp.projects.IAMCustomRole(
"cert-manager",
project=my_project.projects[0].project_id,
role_id="certmanagerpulumi",
title="Cert Manager (Pulumi)",
permissions=[
"dns.resourceRecordSets.create",
"dns.resourceRecordSets.list",
"dns.resourceRecordSets.get",
"dns.resourceRecordSets.update",
"dns.resourceRecordSets.delete",
"dns.changes.get",
"dns.changes.create",
"dns.changes.list",
"dns.managedZones.list",
],
)
project = gcp.projects.IAMMember(
"project",
project=my_project.projects[0].project_id,
role=cert_manager_role.name,
member=f"principal://iam.googleapis.com/projects/{my_project.projects[0].number}/locations/global/workloadIdentityPools/{my_project.projects[0].project_id}.svc.id.goog/subject/ns/cert-manager/sa/cert-manager",
)
manager = CertManager(
"cert-manager",
install_crds=True,
helm_options=ReleaseArgs(namespace="cert-manager", create_namespace=True),
extra_args=["--issuer-ambient-credentials"],
global_=CertManagerGlobalArgs(
leader_election=CertManagerGlobalLeaderElectionArgs(namespace="cert-manager")
)
)https://github.com/paulwilljones/gke-cert-manager-wi-fed/tree/develop/pulumi
For more information on GKE Workload Identity Federation, see the documentation here.
See more of our blogs for the latest ways to manage infrastructure using Terraform in Google Cloud and how to use GKE Config Controller for cluster bootstrapping.
At Jetstack Consult we’re often helping customers adopt and mature their cloud-native and Kubernetes offerings. If you’re interested in discussing how GKE and platform engineering can help your organisation, get in touch and see how we can work together.
Paul is a Google Cloud Certified Fellow with a focus on application modernisation, Kubernetes administration and complementing managed services with open-source solutions. Find him on Twitter & LinkedIn.