Internet security and identity often focuses on TLS, which underpins the majority of encrypted communication and machine identity on the internet today. For Venafi experts from Jetstack, our open source work is at the heart of cloud native workload security. Modern application services rely on our work to prove their identity, trust each other, and allow encrypted communication.
Just as important, and often overlooked, is how a client trusts the certificate in the first place. Most clients do this by trusting a list of “certificate authorities”. As long as the certificate presented by a service is signed by one of those certificate authorities it is trusted.
Right now if you’re reading this on venafi.com, you can check in your browser and see which certificate authority signed our website certificate. (Click the “padlock” icon in the address bar in most browsers). Browsers and operating systems trust hundreds of certificate authorities, because a user might want to securely connect to any number of websites and services, which may chose to use any number of certificate authorities.
While certificate authority key compromise is rare, it does happen. A compromised CA would allow an attacker to issue valid certificates for any domain, including internal ones, and be trusted by your application.
Operating system and web browser vendors are quick to push updates to certificate authority stores, and any breach would be remedied quickly for users. But is the same true of your containers?
Containers and certificate authorities
When we build containers for our applications, it’s standard practice to do this:
FROM alpine:latest
RUN apk add --no-cache ca-certificatesThis pulls in the ca-certificates package from the Alpine Linux repositories. This package is the entire contents of the Mozilla standard certificate authority package, with hundreds of CAs. This isn’t just an Alpine Linux concern either, exactly the same thing happens in Debian, Ubuntu, and even Google’s Distroless container images.
If one were to be compromised, how would you remediate it? Could you rebuild and redeploy every container in your infrastructure? How would you handle third-party containers? Would you even be able to tell if a compromised certificate authority was in use?
Introducing Paranoia
Jetstack Paranoia is a new tool to help address these problems and more. It operates on container images, and can tell you what certificate authorities are present inside the container.
$ paranoia export postgres:latest
File Location Subject
/etc/ssl/certs/ssl-cert-snakeoil.pem CN=dae48bee9747
/usr/share/gnupg/sks-keyservers.netCA.pem CN=sks-keyservers.net CA,O=sks-keyservers.net CA,ST=Oslo,C=NO
Found 2 certificates
File Location Parser Reason
/usr/bin/pg_createcluster pem found start of PEM encoded certificate, but could not find end
/usr/lib/x86_64-linux-gnu/libgnutls.so.30.29.1 pem found start of PEM encoded certificate, but could not find end
Found 2 partial certificatesParanoia can even find certificates inside text files and binaries, that may be used by the application at runtime.
We also support JSON output, in case you want to further inspect the information programmatically.
$ paranoia export --output json python:3 | jq '.certificates[].fingerprintSHA256' | head -n 5
"ebd41040e4bb3ec742c9e381d31ef2a41a48b6685c96e7cef3c1df6cd4331c99"
"6dc47172e01cbcb0bf62580d895fe2b8ac9ad4f873801e0c10b9c837d21eb177"
"16af57a9f676b0ab126095aa5ebadef22ab31119d644ac95cd4b93dbf3f26aeb"
"73c176434f1bc6d5adf45b0e76e727287c8de57616c1e6e6141a2b2cbc7d8e4c"
"d7a7a0fb5d7e2731d771e9484ebcdef71d5f0c3e0a2948782bc83ee0ea699ef4"With this visibility it’s now possible to see what certificate authorities your applications bundle, and what they might be relying on.
Paranoia can also be run in your build pipeline, and can be used to verify in a CI step that your container image does or does not contain certain certificates — useful if you’re intending to distribute internal CAs, or want to narrow down to a subset of certificate authorities for your application.
With a simple config file
version: "1"
require:
- fingerprints:
sha256: 4348A0E9444C78CB265E058D5E8944B4D84F9662BD26DB257F8934A443C70161
comment: "DigiCert Global Root, required for fetching Mozilla CA list"We can use the validate command on our image
$ paranoia validate ubuntu:latest
Validating certificates with 1 allowed, 0 forbidden, and 1 required certificates, in strict mode
Scanned 0 certificates in image ubuntu:latest, found issues.
Certificate with SHA256 4348A0E9444C78CB265E058D5E8944B4D84F9662BD26DB257F8934A443C70161 was required, but was not found Comment: DigiCert Global Root, required for fetching Mozilla CA listFinally, Paranoia’s inspect command can give you a quick view of any ‘unusual’ output, including expired certificate authorities.
$ paranoia inspect golang:1.19
Certificate O=Acme Co
┗ 🚨 expired ( expired on 2020-12-29T15:04:05Z, 1 year 41 weeks since expiry)
Certificate CN=GlobalSign,OU=GlobalSign Root CA - R2,O=GlobalSign
┣ 🚨 expired ( expired on 2021-12-15T08:00:00Z, 43 weeks 1 day since expiry)
┗ 🚨 removed from Mozilla trust store, comments: Ownership transferred to GTS:
https://bug1325532.bmoattachments.org/attachment.cgi?id=8844281
Certificate O=Acme Co
┗ 🚨 expired ( expired on 2018-10-20T19:43:06Z, 3 years 51 weeks since expiry)
Certificate O=Acme Co
┗ 🚨 expired ( expired on 2018-10-20T19:43:06Z, 3 years 51 weeks since expiry)Of course, any of these commands work on local images too
docker save my-local-image | paranoia export -For the full suite of functionality to help manage your container image trust stores, see the Paranoia documentation on GitHub. Paranoia is fully open source under the Apache 2.0 License, and contributions are welcome!
Please note that the images in our examples were picked to demonstrate the functionality of Paranoia and their usage here is not indicative of their security.
The future of trust stores
My team members and I have spent a lot of time thinking about machine identity and trust, and ultimately we believe that certificate authorities are best outside of a container. For that we’ve worked with the community to build cert-manager’s trust-manager, a way to distribute trust bundles at runtime. In trust-manager you maintain your desired certificate authorities at the cluster level, and trust-manager provides them to the container at runtime — just like cert-manager does for server certificates. Jetstack Paranoia is a complementary technology, so you can understand and manage your trust bundles while considering if trust-manager is right for your application.
You can install Paranoia today with go install, it’s fully open source on GitHub, and we hope to make it more available in the coming weeks.
go install github.com/jetstack/paranoia@latestCover photo by Kreeson Naraidoo on Unsplash