Internet security and identity often focuses on TLS, which underpins the majority of encrypted communication and machine identity on the internet today. For Venafi experts from Jetstack, our open source work is at the heart of cloud native workload security. Modern application services rely on our work to prove their identity, trust each other, and allow encrypted communication.
Just as important, and often overlooked, is how a client trusts the certificate in the first place. Most clients do this by trusting a list of “certificate authorities”. As long as the certificate presented by a service is signed by one of those certificate authorities it is trusted.
Right now if you’re reading this on venafi.com, you can check in your browser and see which certificate authority signed our website certificate. (Click the “padlock” icon in the address bar in most browsers). Browsers and operating systems trust hundreds of certificate authorities, because a user might want to securely connect to any number of websites and services, which may chose to use any number of certificate authorities.
While certificate authority key compromise is rare, it does happen. A compromised CA would allow an attacker to issue valid certificates for any domain, including internal ones, and be trusted by your application.
Operating system and web browser vendors are quick to push updates to certificate authority stores, and any breach would be remedied quickly for users. But is the same true of your containers?
Containers and certificate authorities
When we build containers for our applications, it’s standard practice to do this:
This pulls in the ca-certificates package from the Alpine Linux repositories. This package is the entire contents of the Mozilla standard certificate authority package, with hundreds of CAs. This isn’t just an Alpine Linux concern either, exactly the same thing happens in Debian, Ubuntu, and even Google’s Distroless container images.
If one were to be compromised, how would you remediate it? Could you rebuild and redeploy every container in your infrastructure? How would you handle third-party containers? Would you even be able to tell if a compromised certificate authority was in use?
Introducing Paranoia
Jetstack Paranoia is a new tool to help address these problems and more. It operates on container images, and can tell you what certificate authorities are present inside the container.
Paranoia can even find certificates inside text files and binaries, that may be used by the application at runtime.
We also support JSON output, in case you want to further inspect the information programmatically.
With this visibility it’s now possible to see what certificate authorities your applications bundle, and what they might be relying on.
Paranoia can also be run in your build pipeline, and can be used to verify in a CI step that your container image does or does not contain certain certificates — useful if you’re intending to distribute internal CAs, or want to narrow down to a subset of certificate authorities for your application.
With a simple config file
We can use the validate command on our image
Finally, Paranoia’s inspect command can give you a quick view of any ‘unusual’ output, including expired certificate authorities.
Of course, any of these commands work on local images too
For the full suite of functionality to help manage your container image trust stores, see the Paranoia documentation on GitHub. Paranoia is fully open source under the Apache 2.0 License, and contributions are welcome!
Please note that the images in our examples were picked to demonstrate the functionality of Paranoia and their usage here is not indicative of their security.
The future of trust stores
My team members and I have spent a lot of time thinking about machine identity and trust, and ultimately we believe that certificate authorities are best outside of a container. For that we’ve worked with the community to build cert-manager’s trust-manager, a way to distribute trust bundles at runtime. In trust-manager you maintain your desired certificate authorities at the cluster level, and trust-manager provides them to the container at runtime — just like cert-manager does for server certificates. Jetstack Paranoia is a complementary technology, so you can understand and manage your trust bundles while considering if trust-manager is right for your application.
You can install Paranoia today with go install, it’s fully open source on GitHub, and we hope to make it more available in the coming weeks.
Cover photo by Kreeson Naraidoo on Unsplash
