At Venafi, we provide the foundations for zero trust security by protecting the PKI that underpins all mTLS traffic. This ties in with an upsurge in mTLS usage due to the widespread adoption of cloud-native software development. Using Kubernetes, a Certificate Authority (CA) is required to secure critical components within the Kubernetes system, such as the Kubernetes APIService, etcd, kubelets, and front-proxies. Thanks to the managed Kubernetes services, the management of the CAs used for the critical components has been abstracted, so it is up to the internal teams to manage these. As a general rule, it is up to the cloud provider to secure them because it is essential the CAs are protected.
Service meshes, which are now commonly used to secure inter-service communication, have become an exception to this rule. Like a standard Kubernetes environment, a service mesh requires a CA to cryptographically sign workload identities that are used as a basis to secure all network traffic. However, internal teams deploy many service meshes using unmanaged, self-signed CAs. A typical setup stores the CA in a Kubernetes Secret. For example, Istio generates a self-signed CA with a default lifespan of 10 years.
Persisting with a CA operation with an overly long lifetime and no basis to manage automated rotations does not align with security best practices. It can prevent organizations from implementing zero trust due to the heightened risk of the CA being compromised. This reason is precisely why Venafi has developed Venafi Firefly. Firefly introduces simplified lifecycle management of these CAs, replacing the (self-signed) CA with a managed operation—automating the use of shorter-lived intermediate CAs (a few days instead of years) that chain back to your enterprise root CA. Using shorter-lived intermediate CAs allows the service mesh to operate using fully trusted CAs in the same way you rely on your cloud service provider to manage the CA system for the other Kubernetes components securely. Furthermore, when transitioning to a multi-cluster mesh operation using mTLS, Firefly’s inherent decentralized architecture will allow trust to be distributed to where it is required in the mesh.
Firefly is designed to simplify the lives of platform engineers by replacing unmanaged CAs while providing the crypto teams with the necessary security governance. Protecting PKI is crucial, as the CA of your service mesh could become the weakest point in your Zero Trust Network Architecture.
Storing your CA in a Kubernetes Secret is not the safest option due to ineffective governance of access controls to Kubernetes Secrets. We see many examples where too many users and applications have access to cluster-wide secrets. Hence, we recommend the usage of ephemeral CAs that are stored in memory or HSM. In the latest Red Hat Kubernetes Security Report, 26% of the respondents believe inadequate access controls are considered a high-risk security issue for their company. A managed, ephemeral CA operation will ensure these genuine concerns will not apply to your PKI.
Besides over-privileged access to Kubernetes Secrets, we have also recently seen an increase in the opportunity for cyber threats, such as CVEs like the ingress-nginx, where an attacker or rogue developer could run arbitrary commands and access the secret data to which that component has access (and includes ALL Kubernetes Secrets). On top of this, there are heightened chances of compromise in an open-source component, such as what happened with XZ utils.
We need to remove these opportunities for increased threat, and there are ways in Kubernetes to prevent these things from happening, like having good governance on RBAC, preventing direct access to the internet, and many others. However, it is recommended to have defense-in-depth and reduce the attack surface even more.
How might a potential threat to your service mesh come about, meaning the CA used for your service mesh is compromised? A CA compromise means an attacker can generate an identity for ANY workload in your cluster. An identity in a service mesh is nothing more than an X.509 certificate with a SPIFFE ID encoded. It is comparable to getting your hands on a bunch of blank passports and being able to put whatever name you want on them.
If an attacker can generate an identity for any workload in your clusters, this will circumvent any authorization rules that have been set up, allowing the attacker to move laterally undetected within your zero-trust network.
We see this type of threat as potentially devastating to any operation that is using a service mesh with CAs stored in Kubernetes Secrets. Check out the following video, which demonstrates a compromise of a self-signed CA in a Kubernetes cluster and using it to talk to applications it shouldn’t be able to talk to.
Machine Identity Security Summit 2024
Help us forge a new era of cybersecurity
☕ We're spilling all the machine identiTEA Oct. 1-3, but these insights are too valuable to just toss in the harbor! Browse the agenda and register now.