Venafi experts from Jetstack have been working with a number of customers, building container assessment pipelines that enable them to understand the security profile of the images in use within their organization. During these engagements, one thing we’ve had to be consistently mindful of is how we handle multi-architecture images.
Multi-arch images allow you to use one image reference to run the same container on different CPU architectures. This simplifies configuration but obfuscates the fact that multi-arch images are simply a collection of distinct images, each with the potential to have their own set of components, vulnerabilities and licenses.
If you aren’t careful, you can end up associating vulnerability reports and other security-related information with the wrong platform.
This post explores this in more detail and suggests ways that you can appropriately assign metadata to multi-arch images.
How multi-arch images work
First, let’s see how multi-architecture images work under the hood.
Each individual container image in a registry is represented by a manifest, which is a JSON document that describes the configuration for running the container and its layers.
For instance, here’s the manifest for cert-manager-controller’s
A multi-arch image, on the other hand, is represented by a ‘manifest list’, a different type of JSON document that lists individual manifests and the platforms they support.
Generally, friendly tags like
quay.io/jetstack/cert-manager-controller:v1.9.1 are associated with a manifest list. When your
docker client pulls this tag, it uses the manifest list to select the right manifest for your current platform and then uses that manifest to figure out which layers to pull.
This makes it possible to use the exact same reference across platforms, while still receiving the image specific to the platform you’re currently running on.
Manifests in a manifest list are not the same
It’s important to reiterate that manifests in a manifest list are completely distinct from each other. Usually a manifest list contains platform-specific variants of the same piece of software. However, there’s nothing that guarantees that.
Even if the manifests in a manifest list are related, each platform-specific image can have its own set of dependencies, vulnerabilities and licenses.
You need to be aware of this when implementing supply chain security tooling. If you aren’t careful, your assessment of an image for one platform can be incorrectly applied to another.
Here’s a concrete example.
In general, when security scanning tools target multi-arch images they will do one of a few things:
- Resolve the image to the platform the scanner is running on
- Target the image provided to them by the Docker daemon
- Default to the most common platform,
You need to account for this when you’re putting pipelines together. It’s an easy mistake to make to scan an image on one platform and then use the results to green light deployment to another.
For instance, take this example script, which we will assume runs on a
Here we’re building a multi-arch image for the
linux/arm64 platforms, but when
trivy scans it, it resolves the image to the platform it’s running on (
linux/amd64) and therefore the vulnerability report it produces only strictly applies to that particular platform.
When we attest the image, we attach the attestation to the manifest list. This is a problem because now we have a vulnerability report for one image associated, implicitly, with all the images in the list.
When someone running on
linux/arm64 runs a
cosign download attestation on
repository.example.com/myimage:tag they’re going to get the report for
linux/amd64 back. This could be missing vulnerabilities that affect the
linux/arm64 image, or include false positives that don’t apply to it.
Fixing the example
You could improve the example by scanning each platform individually.
Now we have a vulnerability attestation attached to each individual manifest, that applies specifically to that manifest.
You can download the attestation by resolving the image to your target platform and then running
cosign download attestation.
This is less user friendly than it could be. Ideally
cosign would support the option to drill down to your intended platform, without requiring
Other kinds of metadata
Vulnerability reports are generally platform-specific. However, some metadata will apply equally to all images in a manifest list.
One example would be a SLSA provenance attestation. Provided the manifest list and the images were all generated from the same source by the same build command and builder, it would be acceptable to attach the provenance to the manifest list.
If you retrieve the SBOM from the manifest list, the
packages field only lists the child manifests.
But the SBOM for a specific manifest lists the specific dependencies for that platform.
This is good because each SBOM is an accurate reflection of the object it’s attached to. There’s less chance of mistakenly associating a dependency on one platform with another.
The image for
ko is built by
ko itself. So if you use
ko to build your Go containers then you should get the same.
Multi-arch images are implemented explicitly so that users don’t need to think too hard about the platforms they’re running on. This is user friendly but can get you into trouble in the context of security.
When building multi-arch images, ensure that any metadata associated with a manifest list applies equally to all the manifests it describes. Otherwise, metadata should probably be associated individually with platform-specific manifests.
When consuming multi-arch images, ensure that you’re assessing the correct images for the platforms you’re deploying to.
Get in touch
If you need help improving the overall security posture of your organization's container image pipelines, we can help. We have deep expertise and experience working with clients across all areas of software supply chain security.
Contact us directly to discuss how we can help.