Many security teams are already familiar with signing code for traditional applications, executables, software artifacts, documents, and so on. However, they may be less familiar with ways that you sign code in some of the newer areas, especially as we start to containerize microservices and applications. I’ve done some research in terms of understanding what code signing is, what it was, and how did we started sign containers. Based on those learnings, I presented some of the new techniques for signing container images at Machine Identity Management Summit 2023. Here’s a summary of that presentation.
So, everyone's been hearing this mantra of stopping unauthorized code—all the way from protecting the signing keys to the other half, which is the verification. For containers, we need to be able to validate that the code has an identity and then enforce that at runtime. Container image signing involves digitally signing container images to verify their authenticity and integrity. This process is vital for preventing unauthorized code from infiltrating the cluster, ensuring that only trusted images are deployed.
So all these different points of that we need to continue to look at that are really moving the needle, maturing this area that's a lot less explored. But there's definitely a lot of innovation happening there. So everyone has probably heard of the Open Source Security Foundation (OpenSSF), which has been a great vehicle for really improving and increasing adoption in the open source space. Cosign is the newer one. It's been around for a few years.
If you're building code or building container apps, you have probably played with Podman or Skopeo. We had a customer recently that that had been traditionally signing with Skopeo, which uses GPG keys to sign. The challenge they saw was in some of the less mature areas, especially with things like where does a signature go after signing? Where do you store that? And how do you use that process to inform or create policy? So we worked with them on a complete reset and helped them focus on some of the newer utilities like Cosign.
Speaking of Cosign, if you're familiar with some of the early stages of container signing, then there was Notary V1 that's being replaced with The Notary Project notation —which has taken a renewed approach, very similar to Cosign in the sense of attaching the signatures to the images stored in these secure registries.
I would say right now Cosign is probably the most popular choice now for signing and validating container image signatures. It does, without going into some of the details, it does attach a signed image tag to the container. So, this is going to be important because I'll be talking to you about what about the other side. So, what about the policy enforcement side?
When I attempt to run a specific application, the runtime has to be able to enforce that. So obviously being able to support standard Open Container Initiative (OCI) is going to be going to be critical. I wouldn't say it's absolutely necessary, especially if you've already played with Sigstore: they have Rekor for transparency blogs for digital signatures.
Some of the challenges that I ran into was how do you integrate with third-party systems, like key management systems (KMS) such as GCP or HashiCorp Vault. So, you may have to deal with limited key management frameworks.
Venafi is in a good spot to help with challenge because we can integrate with PKCS#11 and there are some configuration tweaks that make it more enterprise ready. Think of Cosign sitting on your developer workstation with the registry pulling the image down, interacting with the Venafi client. Here’s how that would work: Venafi would take the digest of the image manifest and then take the hash, assign it, and return it back. And utilities such as Cosign are then going to include that in the registry. So in all these interactions, whether you're signing them to containers, Windows apps, or PowerShell scripts, the keys are never being exposed.
Stop unauthorized code: Strategies to secure your software supply chain
So a quick segment on the latest from the Notary Project, I'm not sure how many people have heard of Notation. It’s had kind of a slower start, but I think they've spent they spent a lot of their time on generating the specifications. A lot of good work has gone into creating the plug in framework. Thankfully it's not based on Notary V1, which if you're familiar with that old system was a great idea. It just wasn't ready for prime time, especially for enterprises. Of course it has to be OCI registry compliant.
Notation from the Notary Project was really strong from the beginning, designed with a plug in framework which allowed companies like Venafi to create a plug in that you can download today and add to your pipelines to start signing and verifying. So, this is a very, comprehensive specification which they spent a lot of time on. And of course because of that upfront work, there is limited adoption right now. But if you're asking me, I would prefer Notation because it's more enterprise ready.
So key management is going to be important and trust store policy management is also going to be more significant. Cosign is kind of lacking that in that area. Notation enforces policies that prevent you from using a certificate that has, for example, an EKU for a web server or for a client. So, I think it's in, in a good spot there. Most of the admission controllers support policy enforcement now with Notation and our partner Nirmata is kind of pushing the envelope on that.
Signature discovery is going to be important because as you start to harden your registries and you're going to be building and signing secure containers, there's got to be a good way for you to get that observability and that visibility into what containers were signed. For example, you’ll want to know which ones were signed using a self-signed certificate.
And other things that enterprises are probably looking at time stamping. Probably everyone that's doing some form of code signing is probably at least on the traditional executables or PowerShell scripts are probably time stamping.
Additionally, Notation has the ability to introduce custom metadata, such environment information and then revocation. Notation with the Venafi plugin supports revocation. So we can actually check and Venafi provides that attestation as that source of truth for verifying that that certificate is valid.
So, what about policy enforcement? Once you have all the artifacts signed and they're all compatible, it really makes it easy to standardize. And admission controllers are a pretty popular way to be able to admit code mutate in other areas. But let me share an example from the from Kyverno: if you want to admit an application, you can write specific policy that says, “only these signed containers can be admitted”.
So initially, Connaisseur and Kyverno integrated added support. OPA Gatekeeper Kubewarden is another popular approach. Of course, a lot of them support Cosign just because of its adoption rate. But there are also those that continue to support older versions of Notary.
And just kind of a quick shout out to the Venafi Nirmata solution. Out-of-the-box today, you can actually leverage the Venafi platform to sign container images and then push those secure images to the to the hardened registry ideally. And then leverage the Nirmata policy manager in Kyverno to be able to enforce that.
Get Fast, Easy, and Secure Enterprise-Grade Code Signing With Venafi!
Conclusion
Venafi CodeSign Protect provides a strong foundation for the effort for blocking malicious code: signing for the images in these environments, as well as signing of artifacts that you want to attach to images for additional checks. These artifacts could be anything from software build materials, vulnerability scan reports, or other critical metadata such as provenance information such as the build system information. That means you might need to support hundreds of parallel builds running, with each one getting its own identity. To do that, you simply run a client in your CI/CD pipelines which retrieves these machine identities, gets the right certificate, signs your images, signs metadata, and then pushes it back.
Third-party solutions, such as Nirmata, provide policy enforcement that you can distribute across your clusters before malicious code gets to production. Our solution verifies signed container images during Kubernetes admission controls and in-cluster background scans via policies. The policy is delivered as a Kubernetes resource through native APIs, which the DevOps teams easily manage.
Once the policies are deployed within the cluster, if you try to run an image which is not signed it will immediately be blocked. However, if you run a signed and verified image it will be allowed. Nirmata also collects a lot of data on each one of these events from your clusters. So you have full observability in addition to the automation provided itself.