You’ve probably heard of cert-manager - the easiest way to securely issue and renew TLS certificates in Kubernetes. cert-manager started its life at Jetstack and, now that we are part of Venafi, we’re proud of how far it’s come. In just the past few months, we’ve achieved the incredible milestones of becoming a CNCF Incubating project while also reaching 10,000 GitHub Stars!
Retrieving signed certificates is cert-manager’s job, and an important problem to solve - but it’s not the only problem when it comes to TLS. In fact, getting a certificate is just the start.
We also need to address trust; specifically, we need clients to be able to answer an important question:
“How do I decide to trust the certificate(s) that the server sent to me?”
Since you’re reading these words, you clearly decided to trust the certificate chain for this website and that choice probably wasn’t a big deal to you. Your device very likely came with all the things needed to make a reasonable decision about trust for this website.
Usually that takes the form of a trust store on your device, which is also known by several other names including “trust anchor set”, “CA certificate bundle” or “trust bundle”. The name doesn’t matter though - what matters is that it’s a list of certificates which are ultimately trusted when verifying other certificates.
But it’s not always that simple! With Kubernetes the question of TLS trust is just as important and yet can also be considerably more complicated than using a browser.
Let’s dig into TLS trust in cloud native environments and then explore some solutions to the problems we face with Kubernetes workloads!
Kubernetes and trust
The reason that Kubernetes can be a little more tricky lies in the way that trusted certificates are distributed. In many clusters today, trusted CA certificates are baked into containers at build time after being taken from a base image. Usually the base image is a Linux distribution, like Debian or Alpine.
This approach is simple and it clearly works well enough, given how common it is in clusters around the world today.
The simplicity of this approach does have a downside as you can’t update your trust store without rebuilding your containers. That means that the time to update your trust store is the time it takes to get a container through the CI pipeline, build and tag a new version and then to deploy the new container.
It also means that if you roll back to an older version of your container, you’ll also get the older trust store. Plus, if you need to include private CAs, the problem just gets harder. Linux distributions generally allow for private CAs to be added to the trust store but this tends to require running a tool either in CI or at container startup.
An alternative option is for developer teams to change their application code so it points to a specific list of CAs, but that can introduce issues with making requests to public services and in any case requires a reasonably in-depth understanding of how TLS trust works. It’s all too easy to trust the wrong certificate which will cause either security issues or outages down the road.
Whichever way developers end up using your private CAs, you’re still left with the issue of distributing the additional trusted certificates to your CI or runtime environments and ensuring that they’re kept up to date. The more places your CAs are distributed, the harder it is to track what’s being used - and to check if it’s being used correctly.
In short, trust store updates in Kubernetes today are usually slow, brittle, and difficult in a world where we need to be prepared to be able to update everything quickly if there’s a vulnerability. Since so much modern security relies on TLS, trust store updates are all the more critical.
trust-manager to the rescue!
We can handle this better. Trust bundles can actually be decoupled from containers so they can be handled at runtime.
That’s why the cert-manager team - with the support of Venafi experts from Jetstack - have created trust-manager, which has just reached version v0.4.0 with some huge new improvements!
The idea behind trust-manager is simple: define a list of sources which can be used to fetch trusted certificates, and trust-manager will watch those sources and dynamically create and maintain a “target” containing the concatenated certificates from each defined source. That target in turn can be mounted and used directly by your workloads.
trust-manager provides a new Bundle resource which enables trust-bundle management in a Kubernetes-native way, much like what cert-manager’s Certificate resource did for obtaining TLS certificates. Here’s what a Bundle looks like:
apiVersion: trust.cert-manager.io/v1alpha1
kind: Bundle
metadata:
name: mybundle
spec:
sources:
- useDefaultCAs: true
target:
configMap:
key: "trusted-certs.pem"It’s that simple! This Bundle will create ConfigMap resources containing a complete trust store which users can immediately use by mounting the ConfigMap into pods at runtime.
If there’s a security issue which requires an update to trusted certificates, trust-manager means that you don’t need to panic to rebuild every container image - you just need to update your Bundle and then redeploy your containers to pick up the changes.
Plus, the problem of rollback is solved too since using trust-manager will safely roll back your application version without rolling back your trust store!
Trust-manager for Supercharged private PKI
trust-manager also helps with private PKI! In the example below, we include a private CA stored in a Kubernetes Secret. trust-manager will both validate the certificates to ensure they are standards compliant, and seamlessly include certificates into the target.
apiVersion: trust.cert-manager.io/v1alpha1
kind: Bundle
metadata:
name: mybundle
spec:
sources:
- useDefaultCAs: true
# could also use a ConfigMap or specify a certificate inline
- secret:
name: "my-org-ca-secret"
key: "ca.pem"
target:
configMap:
key: "trusted-certs.pem"Rather than having to get your list of trusted CAs into every team’s CI/CD pipeline so it can be built into containers, trust-manager allows you to distribute the CAs at runtime instead. This makes it easier to maintain different trust stores within test environments versus production, which can enable you to use private CAs for issuing testing certificates and publicly trusted certificates in production.
And by combining your organization's trust store with the public bundle, trust-manager enables a single trust store which can be used by most applications without code changes, thereby eliminating the toil of developers needing to change their code to point at custom CA locations on disk.
Last, if a root CA needs to be removed - either because of compromise or a scheduled rotation - trust-manager’s faster update cycle enables much quicker turnaround allowing you to cut off access from old or insecure CAs, which will reduce your mean time to resolve (MTTR) for certificate-related incidents.
Built for the future
Our experience with cert-manager is that a lot of users are slow to update to new major versions; while the open source cert-manager project has a well-defined support policy, that policy alone isn’t a motivation for many users to upgrade since the tool “just works”.
If you’ve made it this far, we hope it’s OK to do a little plug!
If you find that your organization - like many others - can be slow to update to officially supported versions of cert-manager then Venafi’s TLS Protect for Kubernetes will give you visibility and consistency with the versions you’re actually running in your production clusters.
Venafi TLS Protect for Kubernetes provides visibility of all certificates in Kubernetes clusters to help you better manage risk and quickly remediate certificate misconfigurations. If you have cert-manager already running in a cluster, you can sign up () and connect your first cluster for free.
Plus, Venafi can help your organization fully operationalize cert-manager across a large number of clusters with 24x7 support to give you complete peace of mind. If you’re running cert-manager in production across multiple clusters on infrastructure that is scaling fast, it’s worth checking out TLS Protect for Kubernetes!
We obviously want trust-manager to be just as reliable as cert-manager but it’s also critically important that users are able to update the publicly trusted certificate bundle we package with trust-manager as soon as an update is released, without needing to worry about bringing in any potentially unrelated changes which haven’t yet been tested in your clusters.
As such, we’ve designed trust-manager such that users should - by default - be able to choose to update the certificate bundle while running any version of trust-manager from v0.4.0 onwards. The certificate package is bundled into a simple auxiliary container which can be replaced as needed.
That means that no matter what version of trust-manager you’re running, you should be able to update your certificate bundle independently which is crucial for responding quickly to security incidents and ensuring that updates run smoothly.
And since updating your default certificate package is as simple as pointing your trust-manager deployment at a new container image, it has the side effect of making it very simple to run trust-manager in highly restricted or air-gapped environments since no external network requests are required at runtime to be able to update!
Give it a go!
We’re really excited by the possibilities that trust-manager can unlock, and we’ve got an ambitious roadmap including supporting more certificate formats (such as JKS) and a focus on improving the UX for consuming certificate bundles from ConfigMap resources.
Install trust-manager today - there’s a full quick-start demo in the documentation, and we’ve also written up some guides for securely updating trust-manager and avoiding common TLS trust gotchas, too!
We’re more than happy to talk trust! Please reach out to us if you’ve got any questions (and if you made it to the end, maybe leave trust-manager a star on GitHub)!