Many companies and organizations, especially large companies, are adopting Kubernetes using hybrid clouds. And hybrid clouds are effectively using a combination of public cloud infrastructure from cloud service providers as well as on-premises data center infrastructure. When using a hybrid cloud for Kubernetes application development environments, you’ll need flexibility, consistency, and control in the way you secure identities for Kubernetes workloads. To accomplish that, you can't really rely solely on solutions from a single cloud provider to give you that flexibility for all environments.
From a security standpoint, having a consistent way to properly manage all machine identities across all environments is key. And a strong consequence of using Kubernetes for development is that the volume of certificates being used in production environment starts to grow very fast. As developer’s do what they need to do for fast automation, it's not uncommon to see hundreds, if not thousands of certificates in a single cluster—none of which will have been issued with the validation of the security team.
We can assume that adopting Kubernetes will only become stronger and development teams will create more and more clusters. If InfoSec is not able to maintain a certain level of validation, their task to manage threat prevention and eliminate data breaches becomes more and more complex and challenging. This challenge is further complicated by developers, who are focused on making the choices that allow them to run their automation in the fastest way possible.
Fortunately, the developer choice for machine identity management is cert-manager, which has quickly become the facto cloud native open source solution for workload security. As the creator of cert-manager, Jetstack, now part of Venafi, we have some very good data points in terms of how well cert-manager is being adopted. Based on research that we did across the community, we see that developers are saying they use cert-manager more than 80% of the time when they spin up and bootstrap a new cluster. So that means that each time a new cluster is created automatically, they're using cert-manager to instantly automate, issue and renew X 509 certificates for Kubernetes workloads.
In the same vein, over 80% believe that as Kubernetes infrastructure grows, cert-manager will be a vital part of that growth. So, all of this underlines the popularity, the credibility and the success that cert-manager has got with developer teams and the impact that it has on the ecosystem. Another indicator of the success of cert-manager is that it is very widely adopted within the CNCF. As of October last year, it was donated to the CNCF by Venafi—who continues to be the primary maintainer of the project in partnership with the CNCF. And it's now at incubation status. cert-manager also has 10,000 GitHub stars which, if you know a bit about GitHub projects within a cloud native context, is a very high level of acknowledgment. And as organizations invest more and more in Kubernetes and continue to see the growth and value of open source, cert-manager will only become stronger as Kubernetes grows.
But that growth is not without its challenges. There are a range of security vulnerabilities that the developer teams recognize in their use of cert-manager, and they need to communicate those back to security teams.
One example is security drift on cert-manager tooling. Because cert-manager updates every quarter, it's very easy to start to see drift in terms of different versions of cert-manager, especially when looking at multi-cluster infrastructure. So not being able to manage an operational cert-manager with a level of support is something of concern.
Also, developers can easily create sub CAs without validating their trust in a way that security team would. That makes it relatively easy for developers to introduce self-signed certificates, which are very much a certificate misconfiguration. They are not best practice when it comes to being able to run Kubernetes production environments. And much of the feedback that we get from developers suggests that all too often self-signed certificates are found in production clusters. These self-signed certificates can create security vulnerabilities that quickly surface when looking at how to properly manage and secure machine identities in Kubernetes environments.
Three areas where policy and governance can mitigate risk
There are three scenarios that communicate the value of policy and how to control the governance of certificates inside clusters.
1. Stop unwanted certificates from being deployed in Kubernetes clusters
Sadly, untrusted certificates being deployed into Kubernetes clusters is a very common occurrence and it is absolutely not best practice when it comes to security in Kubernetes.
TLS Protect for Kubernetes can bring your cloud and non-cloud infrastructure in one place with one view and one configuration and policy. So, you can avoid using access code credentials that are stored inside your Kubernetes clusters as secrets or providing a user ID and password, which is not really recommended for connecting with your Venafi platform.
The advantage you get is that your Infosec team can maintain control of how to enable connection to the Venafi platform, which is either by using OAuth and Open ID connect or external tools like Vault or other secret providers to make it happen. In any case, no passwords or no tokens are stored inside of your Kubernetes cluster, so there is no security vulnerability of it being exposable.
2. Extend security policies to validate certificate issuance for platform teams
Platform teams are already focused on doing the main areas of operation that they are focused on. We know that security isn't always the primary concern for developers. So, what they want are policies that allow them to operate the platforms in the way that they want to effectively around automation. For this to be able to work for platform teams, the best solutions need to be integrated with existing open source tooling that's already working for developers.
Currently, with cert-manager a CA issuer is available as part of your open source. However, if you have TLS Protect for Kubernetes, you will see more benefits from using the enhanced issuer. First, none of the details for the connection are stored anywhere except centrally through a JSON Web Token. Also, with a better integration between Venafi and cert-manager, you will have more detailed error handling in terms of problems that are encountered with certificate issuance. And you have the option to do auto retry for any failures in certificate integration, as well as better governance structures that can be put around it.
You can also define the domain that you need to use, which means that any other domain requests or common name requests for certificates which don't meet this criterion will be automatically rejected.
3. Ensure better audit controls and governance for certificate issuance in Kubernetes clusters
If you use Venafi as your certificate authority, any certificates that are issued are not just part of your Kubernetes environment, but also in the inventory of your Venafi platform, which means you are able to see the status of those certificates. That means, you are immediately bringing in audit, controls and governance on top of it.
And if you change your policies in the future, you will immediately see which particular policies or certificates would be at risk be able to remediate that risk. But what if your development teams say that issuing certificates with Venafi is too restricted an environment to work for them? They want to work faster and they want to use self-signed certificates and they want you to approve that. Your InfoSec team may decide they want to approve it, but at the same time they want to enforce corporate policies on using the right domain names and certificate credentials, etc.
To do that, you would set up a policy where the selector is much wider. Any issuer that you use within the cluster, would apply the same policy. Non-compliant certificate requests will be denied by the approval controller. However, if you set up the self-signed issuer, but with the right certificate credentials, developers will see that certificate request approved.
Interested in learning more about how TLS Protect for Kubernetes will help you enforce policies in a way that makes your developers jobs easier? Sign up for a free trial.