Accessing GKE endpoints using DNS endpoints and Workload Identity Federation from GitHub Actions
TL;DR
- Google Kubernetes Engine (GKE) clusters with internal IP endpoints can be accessed externally by using the DNS-based endpoint, no longer requiring bastion hosts or VPNs for external connectivity
- Workload Identity Federation can use OIDC providers (like GitHub) to authenticate principals outside of Google Cloud without needing Service Account impersonation
- Use the get-gke-credentials GitHub Action to access GKE clusters using the DNS-based endpoint
- GitHub Actions does not need self-hosted GitHub runners when accessing GKE clusters by the DNS-based endpoint
When using GitHub Actions to deploy and manage GKE in Google Cloud, then users either need to create a public GKE endpoint or provide a publicly accessible mechanism for GitHub to connect to a private GKE cluster endpoint, usually through self-hosted GitHub runners. This blog will discuss how to provide access to private GKE cluster endpoints from GitHub Actions, as well as using transparent authentication between GitHub and Google Cloud using Workload Identity Federation and OIDC.
To Host or Not to Host
One of the main benefits of using GKE DNS-based endpoints is the ability to connect to the cluster's private endpoint externally, thus removing the need for a public IP address for the cluster’s control plane. See our previous blog for more information on GKE DNS-based endpoints.
Normally, this would require a VPN or bastion to reach the internal IP, or just give the cluster endpoint a public IP. Instead, using the cluster’s unique FQDN provided by the DNS-based endpoint routes requests through Google Cloud APIs and requires Cloud IAM permissions before requests are sent to the internal GKE endpoint.
When using deployment pipelines like GitHub Actions, this poses a challenge as the GKE internal IP is not routable from GitHub. Typically, the workaround is to self-host GitHub Runners in Google Cloud that are publicly accessible and can then connect to the internal IP of the GKE cluster.
This isn’t optimal, as it:
- Introduces a publicly accessible attack vector into your infrastructure
- Adds operating costs to run the cloud resources
- Creates operational overhead to maintain and administer the runners
Instead, using GKE DNS-based endpoints allows for GitHub Actions to reach private GKE cluster endpoints without deploying self-hosted GitHub runners in Google Cloud.
Rather than using a public IP endpoint that routes directly to the Kubernetes API Server, using the GKE cluster’s FQDN also adds an additional layer of authorization. As the connection uses a Google Cloud API, Cloud IAM permissions are required before users reach the Kubernetes control plane.
Who Do You Think You Are
When using GitHub Actions to authenticate with Google Cloud, a useful feature to manage credentials is Workload Identity Federation.
This allows for identities outside of Google Cloud to authenticate with Cloud IAM and be authorized to use Google Cloud resources. Specifically, Workload Identity Pools can be configured to use OIDC providers which can then authenticate federated identities (principals) to which IAM permissions can be granted.
Principals outside of Google Cloud can be configured to impersonate Google Cloud Service Accounts, however, Principals can also be used directly to access Google Cloud Resources.
This has the benefits of:
- Not creating Service Account keys for pipelines to use for authentication
- Short-lived tokens reduce the chances of being compromised and exploited
- Federated Identities from GitHub Actions can be scoped to specific subjects, such as the repository or branch the Workflow is running as
Combining these capabilities, GitHub Actions can be used to transparently authenticate with Google Cloud and connect to private GKE cluster endpoints without needing internal network access.
To put this into practice, we’ll see GitHub Actions connecting to a private GKE cluster endpoint using the FQDN. To access the GKE cluster, the GitHub Actions principal must be authenticated and authorized to use the DNS-based endpoint (requiring the container.clusters.connect IAM permission). Once verified, GitHub Actions can then access the cluster which is otherwise not publicly accessible.
Apply, Authenticate, Action
When creating a GKE cluster, IP-based and DNS-based endpoints can be created to access the Kubernetes control plane. As we’ve mentioned, DNS-endpoints provide a unique FQDN that resolves to a Google Cloud API where Cloud IAM permissions are required before reaching the GKE control plane. Alternatively, IP-based endpoints route directly to the GKE control plane and can be publicly or privately routable.
If either a private and/or public IP is allocated to the GKE control plane, it is recommended to set master-authorized-networks to restrict access to the control plane by whitelisting IP ranges that can reach the IP-based endpoint.
The recommended approach however is to disable IP-based endpoints and just use DNS-based. This provides a globally resolvable FQDN that requires Cloud IAM permissions to access. This is preferable to using IP-based endpoints, as it doesn’t require a public IP and whitelisted IP ranges, or a private IP with a VPN or bastion host for external access.
In this example, a GKE Autopilot cluster with a DNS-based endpoint will be created that does not have a public or private IP-based endpoint. This means the only way to access the GKE cluster outside of Google Cloud is by using the FQDN.
Note the flags --no-enable-ip-access which disables IP-based endpoints, and --enable-dns-access which creates an FQDN as the cluster’s DNS-based endpoint.
gcloud container clusters create-auto gke-priv-dns \
--enable-dns-access \
--no-enable-ip-access \
--enable-private-nodes \
--location europe-west2
With the GKE cluster ready, the Workload Identity Federation resources need to be set up to allow GitHub Actions Principals to authenticate with Google Cloud.
As mentioned, OIDC Providers can be used with Workload Identity Federation to allow Principals to authenticate with Google Cloud and be granted IAM permissions. Federated identities can impersonate Google Cloud Service Accounts or they can be granted permissions directly.
To federate GitHub Actions identities, a Workload Identity Pool and Workload Identity Pool Provider are required to validate GitHub Actions OIDC tokens and manage principals’ IAM permissions.
Within the Workload Identity Pool Provider configuration, claims from the OIDC token are mapped to Google Security Token Service token attributes.
Attribute Conditions provide the ability to restrict which principals and claims can authenticate as federated identities. For public OIDC providers like GitHub, this is important to define as it specifies which repository and branch can authenticate with the Workload Identity Pool and receive the Google Cloud permissions associated with the principal.
gcloud
gcloud iam workload-identity-pools create "github-actions" \
--location="global"
gcloud iam workload-identity-pools providers create-oidc "github-provider" \
--location="global" \
--workload-identity-pool="github" \
--attribute-mapping="google.subject=assertion.sub,attribute.repository=assertion.repository,attribute.actor=assertion.actor,attribute.aud=assertion.aud" \
--attribute-condition="assertion.repository_owner_id==\"4161923\" && attribute.repository==\"paulwilljones/gke-dns-based\" && assertion.ref==\"refs/heads/main\" && assertion.ref_type==\"branch\"" \
--issuer-uri="https://token.actions.githubusercontent.com"Terraform
resource "google_iam_workload_identity_pool" "github_actions" {
workload_identity_pool_id = "github-actions"
}
resource "google_iam_workload_identity_pool_provider" "github_provider" {
workload_identity_pool_id = google_iam_workload_identity_pool.github_actions.workload_identity_pool_id
workload_identity_pool_provider_id = "github-actions"
disabled = false
attribute_condition = <<EOT
assertion.repository_owner_id == "4161923" &&
attribute.repository == "paulwilljones/gke-dns-based" &&
assertion.ref == "refs/heads/main" &&
assertion.ref_type == "branch"
EOT
attribute_mapping = {
"google.subject" = "assertion.sub"
"attribute.actor" = "assertion.actor"
"attribute.aud" = "assertion.aud"
"attribute.repository" = "assertion.repository"
}
oidc {
issuer_uri = "https://token.actions.githubusercontent.com"
}
}
Once identities have been federated between GitHub and Google Cloud, they can be granted IAM roles. A key benefit here is that IAM roles can be granted directly to the federated identity in GitHub, so the principal (or principalSet) can directly access Google Cloud resources instead of needing to impersonate a Service Account.
Next, an IAM Policy Binding is created to grant the principal in GitHub Actions access to GKE clusters in Google Cloud. See how the identity is not a Service Account, but the federated identity itself. This will allow the GitHub Action to retrieve credentials for the GKE cluster and send commands to the GKE endpoint without impersonating a Service Account.
gcloud projects add-iam-policy-binding jetstack-paul --member="principal://iam.googleapis.com/projects/993897508389/locations/global/workloadIdentityPools/github/subject/repo:paulwilljones/gke-dns-based:ref:refs/heads/main" --role="roles/container.developer" --condition=Noneresource "google_project_iam_member" "workload_identity" {
project = data.google_project.project.id
role = "roles/container.developer"
member = "principal://iam.googleapis.com/${google_iam_workload_identity_pool.github_actions.name}/subject/repo:paulwilljones/gke-dns-based:ref:refs/heads/main"
}
Now that the IAM resources are configured to use Workload Identity Federation between GitHub and Google Cloud, let’s access a GKE cluster using the DNS-based endpoint with a GitHub federated identity.
GitHub Actions
To authenticate with Google Cloud from GitHub Actions, the auth GitHub Action can use Workload Identity Federation to exchange an OIDC token for Google Cloud credentials.
Once authenticated with Google Cloud, credentials can be requested for GKE clusters using the get-gke-credentials GitHub Action. Since v2.3.0, the option to use the DNS endpoint has been added when requesting GKE credentials.
What this means is that requests will be sent to the cluster’s FQDN, not the IP-based endpoint. This is significant, as it means the cluster does not need a public IP endpoint and can be private and only routable internally within Google Cloud.
Outside of Google Cloud requests can be sent to the FQDN, which are then authorized by Cloud IAM before being routed to the internal GKE endpoint.
This GitHub Actions Workflow shows how to authenticate with the Workload Identity Pool using OIDC and retrieve GKE credentials using the DNS-based endpoint. Then, requests can be sent to GKE from GitHub Actions using the FQDN using the federated identity.
name: GKE DNS-based endpoint
on: workflow_dispatch
jobs:
GKE-DNS-endpoint:
runs-on: ubuntu-latest
permissions:
contents: 'read'
id-token: 'write'
steps:
- id: 'auth'
uses: 'google-github-actions/auth@v2'
with:
project_id: 'jetstack-paul'
workload_identity_provider: 'projects/993897508389/locations/global/workloadIdentityPools/github-actions/providers/github-actions'
- id: 'get-credentials'
uses: 'google-github-actions/get-gke-credentials@v2'
with:
cluster_name: 'gke-priv-dns'
location: 'europe-west2'
use_dns_based_endpoint: 'true'
- id: 'whoami'
run: 'kubectl auth whoami'
A run of the Workflow shows:
- GitHub Actions authenticating with Google Cloud using the OIDC provider and Workload Identity Federation
- GKE credentials are retrieved using the DNS-endpoint
- The federated identity from GitHub Actions sends requests to GKE
- The endpoint for the GKE cluster is the FQDN
- The GKE cluster only has an internal IP address and the FQDN for the endpoint
https://github.com/paulwilljones/gke-dns-based/actions/runs/12182303392/job/33980981632
Screenshot 2024-12-05 at 15.00.05.png
EOF
Combining Workload Identity Federation and GKE DNS-based endpoints enables external access from GitHub Actions to internal GKE endpoints without requiring self-hosted GitHub runners for private network connectivity.
This has many security, operational and cost benefits, by mitigating the publicly accessible endpoints in Google Cloud, removing the management and maintenance of self-hosted GitHub runners and the cost of compute resources to run them.
Using the GKE DNS-endpoints fundamentally changes the architecture of connecting to GKE clusters, removing the need for public IP endpoints and the need for internal network connections through VPNs or bastion hosts.
If you are interested in knowing more about GKE or want to know how to optimise your existing Google Cloud infrastructure, contact Jetstack Consult to learn more about our Google Cloud services.