Kubernetes Kustomize for Manifests in Cert-Manager

Introduction to Kubernetes Kustomize

Kubernetes Kustomize iis an increasingly popular tool for managing Kubernetes manifests. Rather than using templates, as Helm does, Kustomize works by building on existing manifests. Using this pattern it provides various features including resource namespacing, modification of metadata, and generation of Kubernetes Secrets - all without editing the source manifests.

Update

This blog post was originally published in April 2019, but as it continues to be relevant has been updated in February 2022 to benefit from my increased experience working with Kustomize and to reflect changes to the tool itself.

Getting Started

To start using Kubernetes Kustomize you just need one or more Kubernetes manifests and a Kustomization file called kustomization.yaml. The Kustomization file is itself a manifest, which specifies a list of resources, patches to apply, and various other options.

Here’s an example of a minimal Kustomization to manage a Namespace and Deployment:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- namespace.yaml
- deployment.yaml

Kustomize offers much more functionality than this though! This post will summarize some of Kustomize’s features, then walk through a demo to show them in action.

Patches

Manifests that give complete Kubertnetes resources are listed in the Kustomization as resources, as seen in the example in the introduction.

Partial manifests can also be used as strateic merge patches, where they are combined with existing resources with matching metadata to add or edit fields. This can be a convenient way to add large blocks of YAML to a resource, as the patch file just looks like a normal YAML manifest.

JSON patches can also be used in a similar way, but with a more concise syntax that’s better suited to small changes.

Overlays

To modify a kustomization further, or create variants of it, Kustomize uses overlays. Overlays also contain a kustomization.yaml file and can include new resource manifests, or patches for existing resources.

An overlay Kustomization must specify one or more bases. These bases must be directories containing Kustomizations which the overlay can build on. Overlays can also specify other overlays as bases, allowing them to be stacked.

Standalone Or Kubectl

Kustomize is released both as a standalone binary and, since version 1.14 onwards, as a Kubectl integration.

For many older versions of Kubectl the integrated Kustomize version was not updated and fell behind the standalone version. This caused some confusion to Kubectl users as newer Kustomize features were missing.

From Kubectl version 1.21 the integrated Kustomize version was updated and has been kept up to date in following versions. It’s still slightly behind the standalone Kustomize version though, so this is something to watch when using new features!

Usage

The main useful functionality of Kustomize is rendering Kustomizations and applying the resulting manifests to clusters.

A Kustomization, for example in the ./my-kustomization/ directory, can be rendered and output like so:

# For standalone Kustomize
kustomize build ./my-kustomization/

# For Kubectl integrated Kustomize
kubectl kustomize ./my-kustomization/

The same Kustomization can be applied to a cluster like this:

# For standalone Kustomize pipe into Kubectl
kustomize build ./my-kustomization/ | kubectl apply -f -

# For Kubectl integrated Kustomize
kubectl apply -k ./my-kustomization/

Throughout the rest of this guide the Kubectl integrated Kustomize method will be used for convenience, but both should work and have the same effect.

Cloud Native Consulting Services

When our experts are your experts, you can make the most of Kubernetes

Find an expert

Demo

To demonstrate how Kustomize can be used we will deploy and certify a simple web application called Helloweb in multiple different ways, using cert-manager to provision Certificates.

All the Kustomize files can be found in a GitHub repository, so you can follow along and try the demo yourself to see what Kustomize can do! Please raise an issue if you have any problems with the examples. It’s recommended you clone the repository for easy access to all the files. All example commands assume you are in the repository directory.

git clone https://github.com/jetstack/kustomize-cert-manager-demo.git
cd ./kustomize-cert-manager-demo/

To complete the demo yourself you will need access to a Kubernetes cluster which can create LoadBalancer type Services with public IP addresses to deploy the Helloweb app, as well as a domain (or subdomain) that you can point at your Helloweb deployment (to get a certificate). It’s recommended that a newly provisioned cluster without anything else running is used, for example a small GKE cluster which can be destroyed once the demo is complete.

You will also need Kubectl version 1.21 or newer installed for up to date Kustomize features. Your Kubectl version should also be no more than one minor version older or newer than the version of the cluster you are using to meet the version skew policy.

1. Helloweb Base

The initial Helloweb manifests can be found in the bases/helloweb/ directory. The resources in this directory could be applied to a cluster as they are, however a simple kustomization.yaml is used to add commonLabels and set a namespace:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

commonLabels:
  app: helloweb
  demo: kustomize-cert-manager

namespace: helloweb

resources:
- namespace.yaml
- deployment.yaml
- service.yaml
- ingress.yaml

Display the Kustomize output to see the modified resources:

kubectl kustomize ./bases/helloweb/

Note the addition of a Namespace and labels to the resources' metadata.

Deploy the Kustomization to your cluster

kubectl apply -k ./bases/helloweb/

Look at the helloweb namespace to verify a Deployment, Service, and Ingress has been created.

kubectl get -n helloweb Deployment
kubectl get -n helloweb Service
kubectl get -n helloweb Ingress

The Ingress won’t have an IP address yet because we need to deploy our NGINX ingress controller.

2. NGINX Ingress Controller Base

The NGINX ingress controller Kustomize base can be found in bases/ingress-nginx/.

Note that the Kustomization refers to a remote manifest using https://. Kustomize is able to load manifests over HTTP and HTTPS and work with their resources in the same way it can with local manifest files.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.1.1/deploy/static/provider/cloud/deploy.yaml

Apply the Kustomization to install NGINX ingress:

kubectl apply -k bases/ingress-nginx/

Verify that the controller is running:

kubectl get -n ingress-nginx Deployment

While using remote manifests is a great way to get started quickly it’s not recommended for ongoing use. The best practise would be to bring in the provided manifests and store them alongside your other manifests, for example in your Git repository. This removes the risk of transient errors, such as sites being down, causing the remote manifest to be temporarily unavailable.

Bringing in provided manifests also gives better visibility of what is being applied to your cluster and allows changes to the provided manifests to be tracked between versions. A good simple workflow for this when storing manifests in Git would be to bring in each new provided manifest version using a pull request, so it can be reviewed and ideally checked by whatever automated tooling is used to validate all other changes.

Using Kustomize to fetch arbitrary manifests from a URL is a potential vulnerability, for example if someone malicious gains control of the where that URL points or what manifests it provides.

Fix this by fetching the cert-manager manifests and changing the Kustomization to reference the local file rather than the remote URL:

wget --output-document=./bases/ingress-nginx/ingress-nginx.yaml https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.1.1/deploy/static/provider/cloud/deploy.yaml
sed -i.bak "s|https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.1.1/deploy/static/provider/cloud/deploy.yaml|ingress-nginx.yaml|" ./bases/ingress-nginx/kustomization.yaml && rm ./bases/ingress-nginx/kustomization.yaml.bak

The downloaded ingress-nginx.yaml manifests can be added to Git with your other manifests. After these changes the Kustomization should look like this:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - ingress-nginx.yaml

Apply the Kustomization again, there should be no changes as the manifests are the same:

kubectl apply -k bases/ingress-nginx/

3. Helloweb DNS

Check the helloweb Ingress we deployed before, it should now have been allocated an IP address:

kubectl get -n helloweb Ingress

If no IP address has been allocated wait a few minutes and try again.

Once there is an IP address specified confirm the app is accessible:

IP_ADDRESS=$(kubectl get -n helloweb Ingress helloweb-ingress -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
curl ${IP_ADDRESS}/helloweb

It should display the message Hello, world!. You can also enter this into a web browser to see the same message.

Now you must point your domain at this Ingress IP address. Wait for the DNS record to propagate and verify that you can access the page using the domain you have configured:

MY_HOST="my-domain.io" # Set the domain you used here
curl ${MY_HOST}/helloweb

This should display the same Hello, world! message.

4. cert-manger Base

So far we have just been conecting to Helloweb using plain unsecured HTTP. To secure the connection and enable use of HTTPS we need a certificate, which is where cert-manager comes in! The cert-manager Kustomize base can be found in bases/cert-manager/.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
- cert-manager.yaml

As with the NGINX ingress controller Kustomization fetch the cert-manager manifests:

wget --output-document=./bases/cert-manager/cert-manager.yaml https://github.com/cert-manager/cert-manager/releases/download/v1.7.1/cert-manager.yaml

Apply the Kustomization to install cert-manager:

kubectl apply -k ./bases/cert-manager/

Then verify that the cert-manager Deployments are running:

kubectl get -n cert-manager Deployment

5. Kustomize Image Changes

Another example of how Kustomize can help to improve your security is by changing the container image name in manifests.

It’s best practice when using externally supplied container images to scan them for CVEs before copying them to your own secure container registry. Some vulnerabilities in the image may be acceptable, especially if they are deemed low risk, however it’s important to be aware of them. If there are higher risk vulnerabilities these should be addressed, ideally by getting a patched version of the image from the supplier, or alternatively by remediating it yourself with your own patch or other workarounds.

Once the images are in your own secure container registry the cluster should be configured to only allow pulling container images from the secure registry, for example using a Gatekeeper policy. Use of your own secure registry also eliminates possible transient errors, such as Quay.io downtime making cert-manager images unavailable.

To change the image names in your manifests add an images section to your Kustomization by running the following:

cat <<EOF >> ./bases/cert-manager/kustomization.yaml

images:
- name: "quay.io/jetstack/cert-manager-cainjector"
  newName: "gcr.io/my-scanned-registry/cert-manager-cainjector"
- name: "quay.io/jetstack/cert-manager-controller"
  newName: "gcr.io/my-scanned-registry/cert-manager-controller"
- name: "quay.io/jetstack/cert-manager-webhook"
  newName: "gcr.io/my-scanned-registry/cert-manager-webhook"
  newTag: "v1.6.1-my-custom-patch"
EOF

This example shows how you could change the manifests to use images in a Google Container Registry in a project called my-scanned-registry, and even change the tag of the cert-manager-webhook image to a custom tag that may include a patch you have added.

Output the manifests to confirm the images have changed:

kubectl kustomize ./bases/cert-manager/ | grep "image:"

Actually scanning the container images and setting up a secure registry is beyond the scope of this demo, so don’t actually apply this to your cluster (unless you also want to try setting this up yourself).

Revert any changes to the cert-manager Kustomization before proceeding:

git checkout -- ./bases/cert-manager/kustomization.yaml

6. Helloweb Cert Overlay

Now that the Helloweb app and cert-manager are both deployed we can secure the Helloweb app with a certificate. The Helloweb Cert Kustomization, found in ./overlays/helloweb-cert/, is referred to as an overlay. It shows how Kustomize can be used to enhance sets of manifests, making it easy to keep the simple manifests in the bases separate from the modifications required for it to work with cert-manager.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

bases:
  - ../../bases/helloweb

namespace: helloweb

patchesStrategicMerge:
  - ingress.yaml

# Create this patch to set the Ingress domain
patchesJSON6902:
  - path: ingress_patch.json
    target:
      group: networking.k8s.io
      version: v1
      kind: Ingress
      name: helloweb-ingress

configurations:
  - cert-manager-configuration.yaml

Note that the base specification is the only way that manifests and other files ‘above’ the current directory can be used. Kustomize allows for subdirectories and does not enforce any specific structure, but it does not allow resources to be used from directories ‘up’ from it. This is enforced for security reasons, for example to prevent a kustomization.yaml from pulling private information from elsewhere on the filesystem.

There are no resources specified in this Kustomization as it does not add any new manifests. Everything specified in the base Kustomization’s resources are included in the overlay.

The patchesStrategicMerge function is used to apply a strategic merge style patch. Here this is used to modify the base Ingress, adding the TLS specification with the Secret that will be created by cert-manager. The ingress.yaml referenced is a partial YAML file which is then applied on top of the resource to patch.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: helloweb-ingress
  annotations:
    cert-manager.io/issuer: helloweb-issuer
spec:
  tls:
    - hosts: [] # hosts will be added in another patch
      secretName: helloweb-cert

The top section, giving the apiVersion, kind, and name is needed to match the patch with the resource it should be applied to. New items can then be specified in an additive way: So the spec definition will match the existing definition of the same name, but the tls key is different so this will be added without overwriting the existing rules block.

Next the patchesJSON6902 specifies another patch, this time using the JSON patch style. This is used to set the correct hostname value in the Ingress. Run the following to create the patch, setting MY_HOST to the domain you pointed at the Helloweb application earlier:

MY_HOST="my-domain.io" # Set the domain you used here
cat <<EOF > ./overlays/helloweb-cert/ingress_patch.json
[
    {
        "op": "add",
        "path": "/spec/tls/0/hosts/0",
        "value": "${MY_HOST}"
    },
    {
        "op": "add",
        "path": "/spec/rules/0/host",
        "value": "${MY_HOST}"
    }
]
EOF

This Kustomization also features a configurations specification which points to a YAML file telling Kustomize how to recognise when resources are named in cert-manager’s custom resources. For example, a Certificate definition references an Issuer. If Kustomize adds a name prefix to all resources, then it also needs to update references to those resources in other manifests.

Apply this Kustomization to your cluster. As the resources have the same name and namespaces as the ones deployed previously they will be modified in place.

kubectl apply -k ./overlays/helloweb-cert/

You may have noticed that we are still missing a cert-manager Issuer resource to actually create the Certificate we have requested with the Ingress annotation. As Issuers can be configured in different ways two further overlays are used to build on top of this overlay.

7. Helloweb Self-Signed Cert Overlay

This overlay further builds on top of the previous overlay to add the missing cert-manager Issuer and supporting configuration. It can be found in the ./overlays/helloweb-cert-self-signed/ directory.

Issuers often need to be configured differently in different environments, so Kustomize is a great way to manage these variations separately while leaving common cert-manager resources in a shared base: The full reason for separating the Issuer is explored further in the next section.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

bases:
  - ../helloweb-cert

namespace: helloweb

commonLabels:
  app: helloweb
  demo: kustomize-cert-manager

secretGenerator:
  - name: helloweb-ca
    files:
      - secrets/tls.crt
      - secrets/tls.key
    type: "kubernetes.io/tls"

generatorOptions:
  disableNameSuffixHash: true

resources:
  - issuer.yaml

This Kustomization adds an Issuer under resources and also uses a secretGenerator to create a Kubernetes Secret from some certificate and key files which we will provide. This Secret will then be used by the cert-manager Issuer as the root CA for generating a certificate for the Helloweb application.

This could be used to integrate with certificate provider solutions that don’t directly integrate with cert-manager, or in other cases where you need to bring files into the cluster. Keep in mind that secrets, such as keys, should never be checked into Git (unless they are encrypted using something like SOPS or git-crypt). These secret files should only be available unencrypted at run-time, for example after being pulled from a secret store before the Kustomization is applied to a cluster by a pipeline.

For this example use openssl to generate a certificate and key file to use:

mkdir -p ./overlays/helloweb-cert-self-signed/secrets/
openssl req -x509 -nodes -newkey rsa:4096 -sha256 -days 365 -extensions v3_ca \
  -subj "/O=Kustomize cert-manager Demo" \
  -keyout ./overlays/helloweb-cert-self-signed/secrets/tls.key \
  -out ./overlays/helloweb-cert-self-signed/secrets/tls.crt

Note that on MacOS the default version of openssl does not include the v3_ca extension. Install a newer version with Brew or use the workaround described in this GitHub issue on the cert-manager repository. Alternatively just copy the example files from ./overlays/helloweb-cert-self-signed/example-secrets/.

The Kustomization also sets some generatorOptions. By default, the Secret and ConfigMap generators will append the resource name with a hash of the files, however for the cert-manager Issuer to locate the secret we’re creating it needs to have a predictable name, so we will disableNameSuffixHash. There are additional generation options available, such as labeling and annotating generated resources.

Kustomize also includes a configMapGenerator which works in a similar way to create Kubernetes ConfigMaps.

Note that the namespace and commonLabels are specified again for this Kustomization, with the same labels as the Helloweb base. Overlays don’t inherit the labels and namespace set in their bases, so new resources that are added or generated won’t have these set unless they are specified again.

Apply the Kustomization to your cluster. As it’s set to use the same namespace as before it will just add the Secret and Issuer alongside the existing resources:

kubectl apply -k ./overlays/helloweb-cert-self-signed/

After a short time cert-manager should now generate a Certificate for the Helloweb application. Check this is successful with:

kubectl describe -n helloweb Certificate helloweb-cert

Visit the domain you pointed at the Helloweb application in a browser to verify that it’s working with HTTPS. You will see a NET::ERR_CERT_INVALID warning when accessing the page as the certificate is self-signed so won’t be recognised by your browser. You have to accept this warning to continue, which can be particularly awkward in Chrome.

8. Kustomization Design

The next Kustomization we will look at is the Helloweb Let’s Encrypt cert overlay which builds on the Helloweb cert Kustomization. Rather than continuing to build on the previous Helloweb self-signed cert overlay this is a separate ‘fork’ overlay.

In general Kustomize is only additive and removal of resources is a specifically eschewed feature. This is because of the additional complexity and inconsistencies that removal can cause when layering Kustomizations.

Patching can be used to set parts of a manifest to null if they are no longer required. For example, the patch below could be used to modify an Issuer to remove the ca configuration and instead use selfSigned:

apiVersion: certmanager.k8s.io/v1
kind: Issuer
metadata:
  name: issuer
spec:
  ca: null
  selfSigned: {}

It is possible to completely remove a resource using the strategic merge patch’s $patch: delete directive. This can be useful if working with off the shelf configuration where it’s desirable to keep the provided Kustomization or plain manifests consistent with the upstream version to make upgrades easier, but you don’t want some of the resources it creates:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
- provided-manifest.yaml

patchStrategicMerge:
- $patch: delete
  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: unnecessary-deployment

This should only be used where required, and it’s important to keep in mind the possible issues it can cause. For this demo rather than awkwardly try to remove a whole generated Secret it is better to keep the two Kustomizations separate.

This is a key design difference compared to a templating tool, like Helm, which may allow use of some resources to be turned on or off. For some use cases it can be limiting, but also makes the output of Kustomize more predictable. This workflow is important to keep in mind when creating Kustomization bases and overlays.

Before applying the Helloweb LetEncrypt cert overlay we also need to do some manual cleanup. This is another key difference compared to Helm: Kustomize does not try to manage resources.

An entire Kustomization can be removed with kubectl delete -k, however this will also delete resources from the base, so in this case it’s easier to just manually prune the Secret resource that we don’t need:

kubectl delete -n helloweb Secret helloweb-ca

9. Helloweb Let’s Encrypt Cert Overlay

Now we can move on to looking at the Helloweb Let’s Encrypt cert Kustomization in the ./overlays/helloweb-cert-letsencrypt/ directory.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

bases:
  - ../helloweb-cert

namespace: helloweb

commonLabels:
  app: helloweb
  demo: kustomize-cert-manager

resources:
  - issuer.yaml

# Create this patch to set the Issuer email
patchesJSON6902:
  - path: issuer_patch.json
    target:
      group: cert-manager.io
      version: v1
      kind: Issuer
      name: helloweb-issuer

Like the previous Kustomization, this adds a cert-manager Issuer under resources. The new Issuer uses Let’s Encrypt to sign a certificate for the Helloweb application:

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: helloweb-issuer
spec:
  acme:
    server: "https://acme-v02.api.letsencrypt.org/directory"
    privateKeySecretRef:
      name: issuer-letsencrypt-account-key
    solvers:
      - http01:
          ingress:
            name: helloweb-ingress

When we apply this Kustomization the Issuer will replace the existing helloweb-issuer from the previous Kustomization. Note that this is different from patching with Kustomize, so the Issuer will no longer have a spec.ca section as it did before.

We also need to create a patch to set an email address on the Issuer:

EMAIL="user@example.com" # Set your email address here
cat <<EOF > ./overlays/helloweb-cert-letsencrypt/issuer_patch.json
[
    {
        "op": "add",
        "path": "/spec/acme/email",
        "value": "${EMAIL}"
    }
]
EOF

Apply the Kustomization:

kubectl apply -k ./overlays/helloweb-cert-letsencrypt/

Now delete the previous certificate Secret to prompt cert-manager to fetch a new certificate from Let’s Encrypt:

kubectl delete -n helloweb Secret helloweb-cert

Check that a new certificate has been issued:

kubectl describe -n helloweb Certificate helloweb-cert

Once the new certificate has been issued and picked up by the NGINX ingress controller when visiting the Helloweb page there should be no warning, and you should see a green padlock in your browser’s URL bar to indicate the site has a valid certificate!

10. Environments

Finally, we look at bringing all of this together in different environments: This is not a standard Kustomize concept, but is a helpful way to group other Kustomizations together in a simple, single resource that can apply them all.

In this demo there are dev and prod environments specified, designed to represent what would be deployed into development and production clusters respectively. Using a Kustomization to group other Kustomizations for different environments is just one way this approach could be used, but there are many others, for example types like management and workload, or tiers such as frontend and backend.

The development environment can be found in ./environments/dev/:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

bases:
  - ../../bases/ingress-nginx
  - ../../bases/cert-manager
  - ../../overlays/helloweb-cert-selfsigned

The development environment can be found in ./environments/prod/:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

bases:
  - ../../bases/ingress-nginx
  - ../../bases/cert-manager
  - ../../overlays/helloweb-cert-letsencrypt

Note the use of the Helloweb cert self-signed Kustomization for development and Let’s Encrypt for production.

These groupings are not so valuable with only a small number of Kustomizations, but most clusters will have many more components deployed. It can become challenging to manage everything, especially when different clusters require different Kustomizations.

Now the demo is complete the production environment Kustomization can be used to delete everything from the cluster:

kubectl delete -k ./environments/prod/

Conclusion

Kustomize is a very flexible and useful tool. It’s layered approach to managing manifests makes is very well suited to managing your own variations of other peoples work, and to managing variations across environments, without requiring duplication.

As this demo shows, Kustomize makes it easy to start with a minimal app deployment, as is often provided with an app, then modify and build on this to suit your own requirements. It can also be used to separate parts of a deployment, so they can be more easily worked on by different teams. All of these modifications are expressed explicitly, making it clear what the changes are, and allowing them to be put under version control.

When I started using Kustomize it was hard to get used to the design differences compared to templating tools, like Helm, which I was used to. At the same time there is an elegance to Kustomize’s simpler workflow: Many Helm charts become very complex with many settings to turn on or off different components. The template YAML itself is often difficult to read due to the number of variables and conditional blocks included.

There is a case for working with simpler, easier to audit manifests, and only applying the changes required for each use case or environment. While human review of manifests should not be the only check for bad configuration it is sometimes required, particularly when debugging, and Kustomize’s approach can make it easier to do this.

It’s possible to use both Kustomize and Helm, and indeed combine them, to get the best of both! But that’s a subject for a future blog post! In the meantime, if this work as well as other work we do around Kubernetes tooling and open source development is interesting for your own operations, please reach out and let us know how we can help.

Kubernetes Subscriptions

Partner with experts to verify, maintain and optimize your Kubernetes production environment

Browse subscriptions

Using Kustomize for Per-environment Deployment of cert-manager Resources