Developer enablement is being accelerated and simplified through the adoption of internal development platforms (IDP) and the re-emergence of software ‘stacks’. Bringing together highly cohesive, loosely coupled components that provide a comprehensive set of capabilities has always been part of solutions engineering, from the LAMP stack to today’s incarnations that include the CNOE and BACK stacks.
Together, self-service IDPs and modern, cloud-native stacks can enable developers to be more productive through out-of-the-box platforms that are secure, automated, and centrally managed with decentralised operations.
To demonstrate this concept, we’ll look at how each component can add value to the user experience when provisioning and bootstrapping development platforms. We’ll walk through how self-service using Port can integrate with GitHub Actions to invoke a workflow that leverages GKE Enterprise for GitOps, policy enforcement and resource lifecycle management.
Stacks are back
With the abundance of open-source projects across the CNCF Landscape and Kubernetes ecosystem, there are both difficulties and opportunities when composing platforms. With such a vast array of options, it can be challenging to identify the right set of components that are both compatible and deliver the desired functionality. This is where stacks can offer an opinionated set of applications that provides a reference implementation when building platform solutions.
Within Kubernetes, consensus is building around what the de facto components are for certain capabilities, particularly in areas like policy enforcement and GitOps. Also, there is a growing desire to consolidate cloud resource management into Kubernetes, where resources are expressed through Custom Resources and controllers reconcile the desired state against cloud provider APIs.
Projects such as ArgoCD and FluxCD, Gatekeeper and Kyverno, and Crossplane are typically the go-to projects for such features, as such they form most of the CNOE and BACK stacks mentioned above. As CNCF projects, they benefit from community support and feedback, with some having their own dedicated conferences due to their growing ecosystems.
There is a tradeoff though. As with all open-source projects, there is an amount of operational overhead that comes with self-hosting these applications. Deployments, configuration, integration, upgrades, high availability and security are just some of the aspects of maintaining platform components across fleets of clusters in an organisation. Often as the scale increases, so does the operational toil and risk to the business when incidents occur.
Fundamentally, this is part of the balance between buy versus build, and such managed services provide an option for organisations and administrators to offset the shared responsibility of running platforms at scale.
This is where GKE Enterprise can offer an alternative approach to this tradeoff, by providing a set of software capabilities that are managed by Google Cloud and reduce the operational responsibilities of cluster administrators and stakeholders.
GKE Enterprise
GKE Enterprise is a set of fleet management tools for Google Kubernetes Engine (GKE) that orchestrate the lifecycle, compliance and delivery of clusters and platforms in Google Cloud.
Made up of Config Sync, Policy Controller and Config Connector, GKE Enterprise is a set of managed components that are configured through the GKE Fleet API (therefore supported by Terraform, Crossplane, and Pulumi) that can be incorporated into cluster provisioning for a fully automated bootstrapped experience without the need for post-provisioning deployments or configuration.
Config Sync
Config Sync is a GitOps controller that reconciles your desired state as defined in source control. As mentioned, Config Sync is configurable through the GKE Fleet API and therefore can be configured during cluster provisioning. This means that as a GKE cluster is being provisioned, Config Sync can be deployed and sync a target repository before the cluster has been made available to tenants. This is extremely compelling, as it means that policies and cluster add-ons can be deployed by Config Sync as part of cluster provisioning, therefore no one is interacting with the cluster before it is fully compliant, secured and functional.
Policy Controller
To ensure that policies are enforced that follow the standards set by the organisation and compliance requirements, Policy Controller is a managed Gatekeeper deployment that uses Constraints to validate resources during admission.
Config Connector
Google Cloud resources can be managed through Config Connector (KCC) and the corresponding Kubernetes resources. This allows for applications to be created along with the external dependencies in Google Cloud they need, together in the same set of Kubernetes manifests. Config Connector reconciles the desired state as defined in Kubernetes with the observed state in Google Cloud, centralising the management of cloud resources through GKE.
On the other hand, Config Connector can only manage Google Cloud resources, so managing other external dependencies in third parties or SaaS products will either have to be handled by Crossplane or Operators with Custom Resources.
Config Controller
A Config Controller instance is a GKE cluster that hosts managed instances of Config Sync, Config Connector and Policy Controller. Config Controller is designed to be a 'hub' cluster, which orchestrates the desired state of your Google Cloud environments through KCC resources. As such, there are managed Constraints deployed as part of Config Controller to limit the cluster's ability to run arbitrary workloads, and instead limit its scope to being a managed instance of GKE Enterprise components.
Config Controller becomes the control plane for your organization’s desired state. By expressing your cloud resources through Kubernetes resources, it becomes the responsibility of Google’s controllers to reconcile that desired state. This consolidates the organization’s state into one place (as opposed to disparate state files or imperative workflows), and being a managed service, its availability, updates and support are provided by Google Cloud.
Through GKE Enterprise, cluster provisioning, configuration and bootstrapping are fully automated and simplified to reduce the complexity and steps required before being able to onboard users.
Now, these tools can be made available to users to drive the self-service of platforms and provide a user experience that enables developers in a way that is compliant, optimised and secure.
We’re going to take a look at how GKE Enterprise can be integrated into self-service IDPs to empower users to consume development platforms, and how organisations can benefit from GKE Enterprise for consistency and compliance, as well as managing their cloud estate.
Developer Self-service
These stacks start with the user interface presented through an internal developer platform (IDP). IDPs centralise the developer experience and provide an interface that drives automation and self-service for users to interact with services that are made available in accordance with organizational and industry standards. Streamlining and simplifying how developers can request resources, as well as interface with infrastructure and applications enhances productivity and reduces the cognitive overhead.
Port
Port is an IDP that enables self-service and cataloguing of entities within Kubernetes and cloud environments.
Port’s Self-service Actions can integrate with backends to trigger events upon invocation using the inputs provided by the user.
For more on Port’s self-service actions see their demo.
In this example, we use Port as an interface that users can interact with to request resources due to its free, hosted tier and simple configuration (also supported in Terraform). There are many other IDP solutions available with varying levels of features and management overhead, however, the concepts in this example can be implemented with alternatives, such as Backstage.
The Port Actions in this example will be triggering GitHub Actions Workflows, so the user form will merely be taking inputs which will be passed to the GitHub Workflow in the invocation payload.
The two self-service actions in this example correspond to different components being provisioned in GKE. These self-service actions are designed to be lightweight for the user to request, and opinionated in how the downstream resources are configured and provisioned.
User Experience
Create a Config Controller hub cluster
In this example, a Port Action triggers a GitHub Workflow that creates a Config Controller instance. This hub cluster will host our KCC resources, so depending on how you want to organise what resources are managed within a hub cluster, there typically isn't a requirement to host multiple or many Config Controller clusters.
At the time of writing, Config Controller instances can be deployed using gcloud or through a Config Connector resource (ConfigControllerInstance). The lifecycle of Config Controller instances should be managed carefully given their centrality when managing other cloud resources through Config Connector resources. To ensure that Config Controller doesn’t become a single point of failure, consider using a deletion policy that doesn’t delete the corresponding Google Cloud resource upon the deletion of its Config Connector resource (https://cloud.google.com/config-connector/docs/how-to/managing-deleting-resources#keeping_resources_after_deletion).
Config Controller instances can be hosted on GKE Standard and Autopilot clusters. For Autopilot, the --full-managementflag is passed to gcloud anthos config controller create.
As part of Config Controller, Config Sync is already installed and just needs to be configured. This is done by deploying a RootSync resource that encapsulates the configuration and status of the reconciliation.
A RootSync is deployed during the GitHub Workflow to configure Config Sync that pulls Config Connector resources from a directory in the repo which are added and removed by GitHub Actions.
These Config Connector resources include a ContainerCluster and ComputeNetwork which the controller will use to provision a new GKE cluster and VPC. In addition to these, a couple of GKE Hub resources are deployed (GKEHubMembership and GKEHubFeatureMembership) which registers the cluster and configures Config Sync. This creates a powerful pattern for bootstrapping clusters with all the necessary policies and components required for provisioning.
Create a GKE development environment
With Config Controller created, this will be the hub cluster that will host the Config Connector resources, describing the desired state of our cloud resources. As such, all GKE clusters that are created will be Config Connector resources (ContainerClusters) that are then reconciled by the Config Connector controller and created in Google Cloud.
The Self-service Action in Port will provide an interface for users to request a GKE cluster by providing inputs that will invoke a GitHub Action.
The GitHub Action simply renders the necessary Config Connector resources and creates a PR against the repository which can be approved by an administrator. This approval serves as the validation for the requested cluster to be provisioned, as well as the dependent resources. Once merged, Config Sync will pull the new manifests from the Git repository and Config Connector will begin to reconcile the new resources.
Once Config Connector has reconciled the resources and the spoke clusters are provisioned, we again configure Config Sync now in the spoke clusters to pull different RootSync resources which each manage a separate repository sync.
In these, we are pulling Policy Controller Constraints that are bundled to form different sets of policies that correspond to different industry standards for security and best practices (see Policy Controller bundles for more).
$ kubectl get rootsync -A
NAMESPACE NAME RENDERINGCOMMIT RENDERINGERRORCOUNT SOURCECOMMIT SOURCEERRORCOUNT SYNCCOMMIT SYNCERRORCOUNT
config-management-system cis-gke-v1-5-0 956cc41f9f797e542eca2b35ebfd8953f72a78ca 956cc41f9f797e542eca2b35ebfd8953f72a78ca 956cc41f9f797e542eca2b35ebfd8953f72a78ca
config-management-system cis-k8s-v1-7-1 956cc41f9f797e542eca2b35ebfd8953f72a78ca 956cc41f9f797e542eca2b35ebfd8953f72a78ca 956cc41f9f797e542eca2b35ebfd8953f72a78ca
config-management-system cost-reliability-v2023 956cc41f9f797e542eca2b35ebfd8953f72a78ca 956cc41f9f797e542eca2b35ebfd8953f72a78ca 956cc41f9f797e542eca2b35ebfd8953f72a78ca
config-management-system nsa-cisa-k8s-v1-2 956cc41f9f797e542eca2b35ebfd8953f72a78ca 956cc41f9f797e542eca2b35ebfd8953f72a78ca 956cc41f9f797e542eca2b35ebfd8953f72a78ca
config-management-system pss-restricted-v2022 956cc41f9f797e542eca2b35ebfd8953f72a78ca 956cc41f9f797e542eca2b35ebfd8953f72a78ca 956cc41f9f797e542eca2b35ebfd8953f72a78ca
config-management-system root-sync 35f730107c43568d3a7ada364236f6dca2502016 35f730107c43568d3a7ada364236f6dca2502016 35f730107c43568d3a7ada364236f6dca2502016
This ensures that the newly provisioned spoke clusters that will serve as the user’s development environment are compliant out of the box. The cluster has been provisioned and Config Sync has pulled the required policies for Policy Controller to enforce before anyone has been granted access to the cluster.
This simplifies provisioning by using GitOps to pull the required policies across the clusters in a Fleet, which ensures compliance that can then be shown in the Google Cloud Console for a single pane of glass facilitating holistic visibility.
Wrap Up
For the full example, see https://github.com/paulwilljones/gke-enterprise-self-service.
As we have seen, self-service through IDPs can enable users to request and gain access to the resources they need in an automated, compliant and standardised way.
GKE Enterprise offers a set of managed software components that enable this automation, compliance and standardisation with the added benefits of reducing the operational toil and cognitive overhead of maintaining such components in-house which only increases with scale.
When our experts are your experts, you can make the most of Kubernetes
At Jetstack Consult we’re often helping customers adopt and mature their cloud-native and Kubernetes offerings. If you’re interested in discussing how GKE and platform engineering can help your organisation, get in touch and see how we can work together.
Paul is a Google Cloud Certified Fellow with a focus on application modernisation, Kubernetes administration and complementing managed services with open-source solutions. Find him on X & LinkedIn.