By design, malware is designed to ultimately get executed on your enterprise’s computing devices. While malware scanners can be helpful in scanning commercial software packages, how can you prevent malware from getting inserted into software that you develop?
I started my career in the 1990s working for the leading vendor of antivirus and computer security software. My job title at the time was probably the coolest of my career—Buildmaster. It was my responsibility to take the source code from the engineering teams, compile it and deliver an installable product for QA team to test. Once qualified and approved, I released it for our customers to install. I was young and naïve with no real understanding of all the things that could go wrong during this process. The recent SUNBURST hack demonstrates a perfect example of what could go wrong. But before we get into SUNBURST, a quick story about my time as Buildmaster.
During my time as Buildmaster, our company acquired a competing company and its products. As part of the acquisition, it became my responsibility to take over the building and delivering of their products while we integrated our product lines. The build system I had in place was designed to be accessible and easy. It had a client where developers could request builds and integrations, and then designate where those builds occurred, which simplified providing builds for testing.
When I arrived at the offices of our newly acquired company to learn about their build system—I quickly realized they took the responsibility and risks associated with building software much more seriously. Their production build system was completely sandboxed. No outside network access—not to the source control system, not to the developers, nothing. Any source code that was brought in was scanned across all competing AV engines to ensure nothing known was present. After builds were complete, the CDs that were generated were also scanned with all AV products to ensure nothing known was present. Their organization elevated the responsibility of security and risk of their software builds, sacrificing convenience and speed to ensure they delivered the best product possible.
You may be asking yourself at this point, what does this have to do with SUNBURST? Well, recently SolarWinds’ customers were significantly impacted because they trusted the software that SolarWinds released. They trusted that when SolarWinds built their software that it would be free of malicious code. SolarWind’s customers had implicit confidence in the signed software they received; they believed that it was free of viruses and malicious code because it had not been modified since SolarWinds built and delivered it to them. For a great writeup on what happened in the breach, go here. Signed code that was supposed to protect their customers failed them.
How could code signing have prevented SUNBURST and ensured the delivery of software products that were free of malicious code and malware?
If you have read the analysis of the SUNBURST attack, you are aware that malware was installed in the build environment. That malware modified source code as the build was in progress and then cleaned up evidence of that modification once the build was complete.
I personally do not believe that a completely sandboxed environment, one with no network access, is realistic in today’s enterprises that are striving for digital transformation. So, if we can’t have a completely offline system building software, how can we better protect our ‘always online’ continuous build and delivery pipelines?
Let us start with the environment that builds the software. If SolarWinds had been locking down their build pipeline, and only allowing software packages specifically approved for installation in their build delivery pipeline and enforced only software that was signed from the vendors that are part of the build system, the malware that was installed would not have been able to run. There is no excuse not to have tighter controls over a build server. This is a very high value asset, and a big target for someone with malicious intentions.
In any software build system, compiler versions, libraries, source code are scrutinized heavily. This is not a workstation that should have random packages of software installed to meet productivity needs. There is no reason a very static list of approved binaries with valid signatures couldn’t be checked before being allowed to execute. Ironically, with code signing, signing code is the first step. The critical component is to put enforcement in place to check those signatures and only run trusted code. For more information on this, see my other post on “Why Even Sign Code”.
As with any breach there is no single silver bullet that would have prevented the SolarWinds breach and subsequent modification of source code. The layers-of-security / defense-in-depth approach always applies when talking about overall system security. From the network layer, to individuals that have access to source code, to vetting the source code to delivering a product that you can prove came from your trusted build system; it all needs to be audited, scrutinized, and maintained with all risks in mind. The speed of DevOps and CI/CD pipelines have already moved the problem beyond human speed and human control, you cannot rely on manual processes and checklist to ensure your running the right software and no unauthorized changes have been made.
Since we have specifically been talking about how signed code could have helped prevent the SUNBURST malware, here are some highlights showing how Venafi provides Visibility, Intelligence, Automation, and Control around the code signing process in your software delivery pipeline and can help you deliver faster with more security:
- Centralized key protection
Venafi centralizes storage of your private keys. This means you will not have to worry about unauthorized copies or locations of signing keys in your enterprise. Venafi supports all the major HSM vendors as well so you can use them with ease as part of your signing process.
- Approval of who can sign where
There is an approval workflow so you can configure which accounts can sign, and even on which systems they are approved to request signing operations from on the centrally stored keys. This approval can be updated, revoked at any time giving you complete control of all signing operations.
- Approval at time of signing (approve what can be signed)
Once you have approved a user and a system to do signing operations, what are they actually signing? Venafi provides an approval process that can allow an administrator to inspect the signing request before approving. This allows both the ability to approve who can sign, but also approval at time of signing.
- Ability to control signing, even in a highly automated CI/CD pipeline
All approval and signing operations can be assigned to systems accounts and inspected and scheduled. This allows the enterprise to get the automation they need to deliver on time, but with the controls they need to prove they are in control of the signing process.
- High speed supports all your environments
Even though keys are centralized, the signing operations happen in a “hybrid” environment. This means that signing tools on end systems can request signing with the centralized key. Because of the hybrid architecture, only a hash of what is being signed is sent across the network, with signing speed is nearly identical to the speed of a signing operation with a local key t. Venafi also supports all the standard industry signing tools that are present on Windows, Linux and Mac to ensure anything you signed can be controlled and monitored.
- Auditing and reporting
Being able to audit and prove that you are following policy and best practice is critical when using code signing in your software delivery pipeline. Venafi has the built in reports you need to show signing operation activity as well as approval operations. The data the Venafi Platform has on your code signing process can also be delivered out to any of the industry standard SIEM platforms giving you the single pain of glass security reporting that make your security operations teams jobs easier.
The attack on SolarWinds marks a new generation of supply chain attacks. While supply chain attacks are not novel, this attack is different in the level of sophistication, stealth and patience. What enabled the adversaries to gain initial access and keep low profile as long as possible throughout the entire attack lifecycle was the leverage of legitimate and trusted machine identities. Using their presence in the SolarWinds network, the group was able to compromise the code signing system and force it to sign a malicious code that was delivered through the legitimate software release management system to around 18k SolarWinds customers around the globe.