cert-manager has become the de facto solution for managing X.509 certificates for applications running in Kubernetes. The project, now part of the CNCF Sandbox, was built with flexibility and extensibility in mind. It allows operators to use Certificates and Issuers as first-class citizens in the API and enable almost any use case in which an automated certificate is required.
As Kubernetes adoption grows, there is now increased interest in service mesh to securely connect workloads, as well as provide dynamic configuration of traffic routing and consistent observability. Services meshes typically use X.509 certificates for mutual TLS (mTLS), with the identity encoded using SPIFFE.
With a range of supported certificate issuers, there has been interest from the community to see Istio integrated with cert-manager to provide signing capabilities for workloads. Working with the Security WG in the Istio community, as well as a number of our customers, the cert-manager team at Jetstack have built an integration that enables cert-manager to sign workload certificates in an Istio service mesh.
Rise of the service mesh
Istio is a popular, fully-featured service mesh; it has a rich set of configurations for traffic routing, policy control, and observability. Like most service meshes, the data plane is powered by Envoy, and each workload running in the mesh will have proxies that intercept all traffic, and make routing decisions based upon the mesh configuration as a whole.
When it comes to managing certificates, service meshes typically ship with a limited number of Certificate Authority (CA) choices, those being either self signed, perhaps Vault, and sometimes an integration with a cloud provider solution. This imposes limitations on how and where your certificates get signed. These limitations can be problematic for larger enterprises where there are requirements for using particular CAs, auditing and visibility of these signed certificates, as well as the headaches that come with changing the Certificate Authority already used in production.
We’ve gone deeper into Istio OIDC Authentication here.
Zero Trust with cert-manager, Istio and Kubernetes
Why use cert-manager?
cert-manager is commonly used to secure ingresses, in conjunction with an Ingress controller (NGINX, for example) and a public CA such as Let’s Encrypt. You can use cert-manager with Istio today to secure ingress using the Istio Gateway (guide), but up until now it’s not been straightforward to use for issuance and renewal of workload certificates.
cert-manager was always built to be CA-agnostic and enable integration with both public and private CAs. The project and wider ecosystem has a wide choice of public and private CA providers available for certificate signing. There are internal issuers core to the project, including HashiCorp Vault, and a fast-growing set of External Issuers that should cover many bases, including cloud provider PKI services. It’s also simple enough to build on cert-manager and develop your own external issuer if you’ve a more bespoke requirement.
A further benefit of using cert-manager is its centralised API and consistent interface. This means that swapping out an issuer, say to use a different CA provider, is fairly trivial. This operation only requires a change to the Issuer configuration and issuer that the mesh is configured to use - the rest is all handled by cert-manager.
By using cert-manager as the single tool for managing certificates in the cluster, operators are able to have complete visibility of the machine identities with a single pane of glass, for auditing, policy and observability purposes (and if that sounds useful, check out our Jetstack Secure service - it’s free to get started).
Machine identity lifecycle in an Istio service mesh
When a workload starts, along with its sidecar, it needs to identify itself to the mesh control plane, and retrieve a signed certificate in order to begin sending traffic over the network. A helper executable called the istio-agent is responsible for retrieving this signed certificate, and delivering it to Envoy to use. This involves sending a certificate signing request to the control plane component of Istio (istiod), along with the pod’s Service Account token to securely identity itself. Once istiod has validated the request, it will in the default case, sign and send back a certificate trusted by the mesh that Envoy can begin to serve and send traffic.
How we’ve integrated cert-manager
The cert-manager integration with Istio involves running a separate component to istiod. The cert-manager-istio-agent processes certificate requests from istio-agents in the mesh. The agent will verify the identity, using the incoming Service Account token and ensure the certificate signing request matches that identity, kicking off a CertificateRequest flow with cert-manager. This involves cert-manager watching for the CertificateRequest created, and signing it using the Issuer installed and referenced on the request. Once signed, the certificate is returned back to the istio-agent to facilitate mTLS in the mesh.
This agent is responsible for distributing the trust chain of the configured Issuer, along with signing certificates for all workloads running in the cluster.
Using the istio-csr project, you can integrate any of cert-manager’s Issuers and have this provide certificate signing for workloads in the mesh. With more External Issuers, such as the Google CAS Issuer, being added to the cert-manager ecosystem all the time, it allows certificates in all areas of the cluster be signed from any number of centralised CAs that best suits your needs, whilst ensuring the portability that the cert-manager API provides.
Not using Istio?
In the CNCF, the Linkerd project already provides cert-manager integration. Jetstack has also integrated Open Service Mesh (OSM), a new service mesh from Microsoft in the CNCF Sandbox - (see this video for a talk featuring a demo).
If you are interested in trying out the new Istio cert-manager integration, you can find the repository and documentation on GitHub here. Let us know what you think! We’re here to help should you want to discuss further integration and find out more about Jetstack Secure for cert-manager.
Cover every cluster with ease and efficiency.
Related posts