Introduction to Kubernetes Kustomize
Kubernetes Kustomize iis an increasingly popular tool for managing Kubernetes manifests. Rather than using templates, as Helm does, Kustomize works by building on existing manifests. Using this pattern it provides various features including resource namespacing, modification of metadata, and generation of Kubernetes Secrets - all without editing the source manifests.
This blog post was originally published in April 2019, but as it continues to be relevant has been updated in February 2022 to benefit from my increased experience working with Kustomize and to reflect changes to the tool itself.
To start using Kubernetes Kustomize you just need one or more Kubernetes manifests and a Kustomization file called
kustomization.yaml. The Kustomization file is itself a manifest, which specifies a list of resources, patches to apply, and various other options.
Here’s an example of a minimal Kustomization to manage a Namespace and Deployment:
Kustomize offers much more functionality than this though! This post will summarize some of Kustomize’s features, then walk through a demo to show them in action.
Manifests that give complete Kubertnetes resources are listed in the Kustomization as resources, as seen in the example in the introduction.
Partial manifests can also be used as strateic merge patches, where they are combined with existing resources with matching metadata to add or edit fields. This can be a convenient way to add large blocks of YAML to a resource, as the patch file just looks like a normal YAML manifest.
JSON patches can also be used in a similar way, but with a more concise syntax that’s better suited to small changes.
To modify a kustomization further, or create variants of it, Kustomize uses overlays. Overlays also contain a
kustomization.yaml file and can include new resource manifests, or patches for existing resources.
An overlay Kustomization must specify one or more bases. These bases must be directories containing Kustomizations which the overlay can build on. Overlays can also specify other overlays as bases, allowing them to be stacked.
Standalone Or Kubectl
For many older versions of Kubectl the integrated Kustomize version was not updated and fell behind the standalone version. This caused some confusion to Kubectl users as newer Kustomize features were missing.
From Kubectl version 1.21 the integrated Kustomize version was updated and has been kept up to date in following versions. It’s still slightly behind the standalone Kustomize version though, so this is something to watch when using new features!
The main useful functionality of Kustomize is rendering Kustomizations and applying the resulting manifests to clusters.
A Kustomization, for example in the
./my-kustomization/ directory, can be rendered and output like so:
The same Kustomization can be applied to a cluster like this:
Throughout the rest of this guide the Kubectl integrated Kustomize method will be used for convenience, but both should work and have the same effect.
All the Kustomize files can be found in a GitHub repository, so you can follow along and try the demo yourself to see what Kustomize can do! Please raise an issue if you have any problems with the examples. It’s recommended you clone the repository for easy access to all the files. All example commands assume you are in the repository directory.
To complete the demo yourself you will need access to a Kubernetes cluster which can create
LoadBalancer type Services with public IP addresses to deploy the Helloweb app, as well as a domain (or subdomain) that you can point at your Helloweb deployment (to get a certificate). It’s recommended that a newly provisioned cluster without anything else running is used, for example a small GKE cluster which can be destroyed once the demo is complete.
You will also need Kubectl version 1.21 or newer installed for up to date Kustomize features. Your Kubectl version should also be no more than one minor version older or newer than the version of the cluster you are using to meet the version skew policy.
1. Helloweb Base
The initial Helloweb manifests can be found in the
bases/helloweb/ directory. The resources in this directory could be applied to a cluster as they are, however a simple
kustomization.yaml is used to add
commonLabels and set a
Display the Kustomize output to see the modified resources:
Note the addition of a Namespace and labels to the resources'
Deploy the Kustomization to your cluster
Look at the
helloweb namespace to verify a Deployment, Service, and Ingress has been created.
The Ingress won’t have an IP address yet because we need to deploy our NGINX ingress controller.
2. NGINX Ingress Controller Base
The NGINX ingress controller Kustomize base can be found in
Note that the Kustomization refers to a remote manifest using
https://. Kustomize is able to load manifests over HTTP and HTTPS and work with their resources in the same way it can with local manifest files.
Apply the Kustomization to install NGINX ingress:
Verify that the controller is running:
While using remote manifests is a great way to get started quickly it’s not recommended for ongoing use. The best practise would be to bring in the provided manifests and store them alongside your other manifests, for example in your Git repository. This removes the risk of transient errors, such as sites being down, causing the remote manifest to be temporarily unavailable.
Bringing in provided manifests also gives better visibility of what is being applied to your cluster and allows changes to the provided manifests to be tracked between versions. A good simple workflow for this when storing manifests in Git would be to bring in each new provided manifest version using a pull request, so it can be reviewed and ideally checked by whatever automated tooling is used to validate all other changes.
Using Kustomize to fetch arbitrary manifests from a URL is a potential vulnerability, for example if someone malicious gains control of the where that URL points or what manifests it provides.
Fix this by fetching the cert-manager manifests and changing the Kustomization to reference the local file rather than the remote URL:
ingress-nginx.yaml manifests can be added to Git with your other manifests. After these changes the Kustomization should look like this:
Apply the Kustomization again, there should be no changes as the manifests are the same:
3. Helloweb DNS
Check the helloweb Ingress we deployed before, it should now have been allocated an IP address:
If no IP address has been allocated wait a few minutes and try again.
Once there is an IP address specified confirm the app is accessible:
It should display the message
Hello, world!. You can also enter this into a web browser to see the same message.
Now you must point your domain at this Ingress IP address. Wait for the DNS record to propagate and verify that you can access the page using the domain you have configured:
This should display the same
Hello, world! message.
4. cert-manger Base
So far we have just been conecting to Helloweb using plain unsecured HTTP. To secure the connection and enable use of HTTPS we need a certificate, which is where cert-manager comes in! The cert-manager Kustomize base can be found in
As with the NGINX ingress controller Kustomization fetch the cert-manager manifests:
Apply the Kustomization to install cert-manager:
Then verify that the cert-manager Deployments are running:
5. Kustomize Image Changes
Another example of how Kustomize can help to improve your security is by changing the container image name in manifests.
It’s best practice when using externally supplied container images to scan them for CVEs before copying them to your own secure container registry. Some vulnerabilities in the image may be acceptable, especially if they are deemed low risk, however it’s important to be aware of them. If there are higher risk vulnerabilities these should be addressed, ideally by getting a patched version of the image from the supplier, or alternatively by remediating it yourself with your own patch or other workarounds.
Once the images are in your own secure container registry the cluster should be configured to only allow pulling container images from the secure registry, for example using a Gatekeeper policy. Use of your own secure registry also eliminates possible transient errors, such as Quay.io downtime making cert-manager images unavailable.
To change the image names in your manifests add an
images section to your Kustomization by running the following:
This example shows how you could change the manifests to use images in a Google Container Registry in a project called
my-scanned-registry, and even change the tag of the
cert-manager-webhook image to a custom tag that may include a patch you have added.
Output the manifests to confirm the images have changed:
Actually scanning the container images and setting up a secure registry is beyond the scope of this demo, so don’t actually apply this to your cluster (unless you also want to try setting this up yourself).
Revert any changes to the cert-manager Kustomization before proceeding:
6. Helloweb Cert Overlay
Now that the Helloweb app and cert-manager are both deployed we can secure the Helloweb app with a certificate. The Helloweb Cert Kustomization, found in
./overlays/helloweb-cert/, is referred to as an overlay. It shows how Kustomize can be used to enhance sets of manifests, making it easy to keep the simple manifests in the
bases separate from the modifications required for it to work with cert-manager.
Note that the base specification is the only way that manifests and other files ‘above’ the current directory can be used. Kustomize allows for subdirectories and does not enforce any specific structure, but it does not allow resources to be used from directories ‘up’ from it. This is enforced for security reasons, for example to prevent a
kustomization.yaml from pulling private information from elsewhere on the filesystem.
There are no
resources specified in this Kustomization as it does not add any new manifests. Everything specified in the base Kustomization’s
resources are included in the overlay.
patchesStrategicMerge function is used to apply a strategic merge style patch. Here this is used to modify the base Ingress, adding the TLS specification with the Secret that will be created by cert-manager. The
ingress.yaml referenced is a partial YAML file which is then applied on top of the resource to patch.
The top section, giving the
name is needed to match the patch with the resource it should be applied to. New items can then be specified in an additive way: So the
spec definition will match the existing definition of the same name, but the
tls key is different so this will be added without overwriting the existing
patchesJSON6902 specifies another patch, this time using the JSON patch style. This is used to set the correct hostname value in the Ingress. Run the following to create the patch, setting
MY_HOST to the domain you pointed at the Helloweb application earlier:
This Kustomization also features a
configurations specification which points to a YAML file telling Kustomize how to recognise when resources are named in cert-manager’s custom resources. For example, a Certificate definition references an Issuer. If Kustomize adds a name prefix to all resources, then it also needs to update references to those resources in other manifests.
Apply this Kustomization to your cluster. As the resources have the same name and namespaces as the ones deployed previously they will be modified in place.
You may have noticed that we are still missing a cert-manager Issuer resource to actually create the Certificate we have requested with the Ingress annotation. As Issuers can be configured in different ways two further overlays are used to build on top of this overlay.
7. Helloweb Self-Signed Cert Overlay
This overlay further builds on top of the previous overlay to add the missing cert-manager Issuer and supporting configuration. It can be found in the
Issuers often need to be configured differently in different environments, so Kustomize is a great way to manage these variations separately while leaving common cert-manager resources in a shared base: The full reason for separating the Issuer is explored further in the next section.
This Kustomization adds an Issuer under
resources and also uses a
secretGenerator to create a Kubernetes Secret from some certificate and key files which we will provide. This Secret will then be used by the cert-manager Issuer as the root CA for generating a certificate for the Helloweb application.
This could be used to integrate with certificate provider solutions that don’t directly integrate with cert-manager, or in other cases where you need to bring files into the cluster. Keep in mind that secrets, such as keys, should never be checked into Git (unless they are encrypted using something like SOPS or git-crypt). These secret files should only be available unencrypted at run-time, for example after being pulled from a secret store before the Kustomization is applied to a cluster by a pipeline.
For this example use
openssl to generate a certificate and key file to use:
Note that on MacOS the default version of
openssl does not include the
v3_ca extension. Install a newer version with Brew or use the workaround described in this GitHub issue on the cert-manager repository. Alternatively just copy the example files from
The Kustomization also sets some
generatorOptions. By default, the Secret and ConfigMap generators will append the resource name with a hash of the files, however for the cert-manager Issuer to locate the secret we’re creating it needs to have a predictable name, so we will
disableNameSuffixHash. There are additional generation options available, such as labeling and annotating generated resources.
Kustomize also includes a configMapGenerator which works in a similar way to create Kubernetes ConfigMaps.
Note that the
commonLabels are specified again for this Kustomization, with the same labels as the Helloweb base. Overlays don’t inherit the labels and namespace set in their bases, so new resources that are added or generated won’t have these set unless they are specified again.
Apply the Kustomization to your cluster. As it’s set to use the same namespace as before it will just add the Secret and Issuer alongside the existing resources:
After a short time cert-manager should now generate a Certificate for the Helloweb application. Check this is successful with:
Visit the domain you pointed at the Helloweb application in a browser to verify that it’s working with HTTPS. You will see a
NET::ERR_CERT_INVALID warning when accessing the page as the certificate is self-signed so won’t be recognised by your browser. You have to accept this warning to continue, which can be particularly awkward in Chrome.
8. Kustomization Design
The next Kustomization we will look at is the Helloweb Let’s Encrypt cert overlay which builds on the Helloweb cert Kustomization. Rather than continuing to build on the previous Helloweb self-signed cert overlay this is a separate ‘fork’ overlay.
In general Kustomize is only additive and removal of resources is a specifically eschewed feature. This is because of the additional complexity and inconsistencies that removal can cause when layering Kustomizations.
Patching can be used to set parts of a manifest to
null if they are no longer required. For example, the patch below could be used to modify an Issuer to remove the
ca configuration and instead use
It is possible to completely remove a resource using the strategic merge patch’s
$patch: delete directive. This can be useful if working with off the shelf configuration where it’s desirable to keep the provided Kustomization or plain manifests consistent with the upstream version to make upgrades easier, but you don’t want some of the resources it creates:
This should only be used where required, and it’s important to keep in mind the possible issues it can cause. For this demo rather than awkwardly try to remove a whole generated Secret it is better to keep the two Kustomizations separate.
This is a key design difference compared to a templating tool, like Helm, which may allow use of some resources to be turned on or off. For some use cases it can be limiting, but also makes the output of Kustomize more predictable. This workflow is important to keep in mind when creating Kustomization bases and overlays.
Before applying the Helloweb LetEncrypt cert overlay we also need to do some manual cleanup. This is another key difference compared to Helm: Kustomize does not try to manage resources.
An entire Kustomization can be removed with
kubectl delete -k, however this will also delete resources from the base, so in this case it’s easier to just manually prune the Secret resource that we don’t need:
9. Helloweb Let’s Encrypt Cert Overlay
Now we can move on to looking at the Helloweb Let’s Encrypt cert Kustomization in the
Like the previous Kustomization, this adds a cert-manager Issuer under
resources. The new Issuer uses Let’s Encrypt to sign a certificate for the Helloweb application:
When we apply this Kustomization the Issuer will replace the existing
helloweb-issuer from the previous Kustomization. Note that this is different from patching with Kustomize, so the Issuer will no longer have a
spec.ca section as it did before.
We also need to create a patch to set an email address on the Issuer:
Apply the Kustomization:
Now delete the previous certificate Secret to prompt cert-manager to fetch a new certificate from Let’s Encrypt:
Check that a new certificate has been issued:
Once the new certificate has been issued and picked up by the NGINX ingress controller when visiting the Helloweb page there should be no warning, and you should see a green padlock in your browser’s URL bar to indicate the site has a valid certificate!
Finally, we look at bringing all of this together in different environments: This is not a standard Kustomize concept, but is a helpful way to group other Kustomizations together in a simple, single resource that can apply them all.
In this demo there are
prod environments specified, designed to represent what would be deployed into development and production clusters respectively. Using a Kustomization to group other Kustomizations for different environments is just one way this approach could be used, but there are many others, for example types like
workload, or tiers such as
The development environment can be found in
The development environment can be found in
Note the use of the Helloweb cert self-signed Kustomization for development and Let’s Encrypt for production.
These groupings are not so valuable with only a small number of Kustomizations, but most clusters will have many more components deployed. It can become challenging to manage everything, especially when different clusters require different Kustomizations.
Now the demo is complete the production environment Kustomization can be used to delete everything from the cluster:
Kustomize is a very flexible and useful tool. It’s layered approach to managing manifests makes is very well suited to managing your own variations of other peoples work, and to managing variations across environments, without requiring duplication.
As this demo shows, Kustomize makes it easy to start with a minimal app deployment, as is often provided with an app, then modify and build on this to suit your own requirements. It can also be used to separate parts of a deployment, so they can be more easily worked on by different teams. All of these modifications are expressed explicitly, making it clear what the changes are, and allowing them to be put under version control.
When I started using Kustomize it was hard to get used to the design differences compared to templating tools, like Helm, which I was used to. At the same time there is an elegance to Kustomize’s simpler workflow: Many Helm charts become very complex with many settings to turn on or off different components. The template YAML itself is often difficult to read due to the number of variables and conditional blocks included.
There is a case for working with simpler, easier to audit manifests, and only applying the changes required for each use case or environment. While human review of manifests should not be the only check for bad configuration it is sometimes required, particularly when debugging, and Kustomize’s approach can make it easier to do this.
It’s possible to use both Kustomize and Helm, and indeed combine them, to get the best of both! But that’s a subject for a future blog post! In the meantime, if this work as well as other work we do around Kubernetes tooling and open source development is interesting for your own operations, please reach out and let us know how we can help.