I've told the “DevSecOps walk into a bar” joke to illustrate the need for a shift-left approach to software security before now. Long story short: developer buys a beer, tells barman operations is paying while security says they saw that coming but nobody asked. Yeah, I know, but even less funny is when attackers start shifting left as well.
Venafi's Eddie Glenn has described this new generation of shift-left cyberattacks as being perpetrated by "sophisticated software developers that target build infrastructure." I’ll raise a red flag when I see the word “sophisticated” used in relation to an attack, but they are anything but. However, even if these threat actors are not gifted with the dictionary definition of ‘a wide-ranging knowledge’ I’ll gladly agree that those targeting the integrity of software build pipelines are a clear and present danger to enterprise security. Which leaves me wondering if 15 new punchlines to my attempt at humour might just help?
Owning software assurance within your organization is not optional
Software supply chain security has been revealed as wanting over the last two years, as illustrated by three high-profile attacks in particular: SolarWinds, Codecov and Kaseya. The European Union Agency for Cybersecurity (ENISA) has seen such supply chain attacks increase "in number and sophistication" across the last two years, warning that the vector poses "an increasing risk for organisations."
Yet, if a recent Venafi survey is anything to go by there remains a concerning lack of security self-awareness when it comes to addressing that risk. This survey, which included responses by executives with both security and software development responsibility, reassuringly found 97% agreeing that software providers should improve build and code-signing security, with 96% going as far as to suggest the integrity of update code should be guaranteed by those providers. My positivity soon enough took a nose-dive, however, when the same polling saw 55% saying the SolarWinds attack had "little or no impact" on their concerns when purchasing software and 69% admitting they had not increased the number of security assurance questions they asked of software providers.
Disconnects always concern me, and they should concern you as well: admitting awareness of risk without being willing to work to prevent it is both unacceptable and self-harming. Not least because the same old siloed approach appears to be fuelling this mitigation stagnation: there was nearly a 50/50 split between thinking IT security and development teams should be responsible for making it happen.
Engineer the shift-left attacker out of the software supply chain equation
The secure software pipeline assurance disconnect worryingly, it seems to me, goes further than just not grasping the responsibility nettle. It also encompasses a failure to properly understand the how as well as the who. Bridging this understanding gap is vital if an organisation is going to be able to properly address the shift-left attack threat vector. Which is where Venafi, Veracode, Sophos, CloudBees and GitLab have raised their collaborative open-source project hands with a codesigning blueprint.
The 'blueprint for building modern, secure software development pipelines', to be precise, is a community-supported project that aims to achieve what it says on the virtual tin. Aimed at both engineers and security teams alike, the blueprint exists to define a vendor-neutral 'map' of diverse controls that can both reduce risk while still staying aligned with the agile development pipelines expected today. Yes, it's another shift-left framework to enable secure continuous integration and continuous deployment. However, it's also focussed on a very specific goal: "to minimize the possibility of supply chain attacks." In other words, it exists to effectively engineer out the shift-left attacker and lower the real-world risk of supply chain tampering. It does this using a pragmatic approach that fits the development pipelines of today and does so by the application of 15 controls that make securely signing code a standardised reality.
My take on the 15 secure software controls that can reduce supply chain attack risk
- Restrict admin access to CI/CD tools and secure, both at host and application-layer, using multi-factor authentication. Not doing this leave the remaining controls open to modification by shift-left threat actors and so undermine the integrity of the entire process.
- Don't accept commits unless signed with a developer GnuPG (GPG) key so as to make life as difficult as possible for the would-be attacker.
- Attackers love automated build systems, so make the 'compromised keys' attack window as brief as possible by ensuring automation access keys expire periodically and automatically.
- Did I mention that attackers love automated build systems? Don't allow them to bypass the standard source code review process. Instead, apply the principle of least privilege and ensure CI systems have read-only access to those source code repositories.
- Blunt the attack risk from modern software dependency managers by only allowing connections to authorized registries and so restricting malicious packages from accessing the pipeline. Using public repositories raises the attack stakes as threat actors will do whatever they can to introduce nefarious packages into builds.
- Identify high-severity or critical vulnerabilities using static application security testing (SAST) or software composition analysis (SCA) and break the build if found. Don't risk introducing insecure code into the pipeline but do ensure defined and documented vulnerability exception management is in place and apply the control often to reduce remediation costs through early vulnerability identification.
- Segregate repositories for storage of development, stage and production artifacts to ensure a chain of control can be maintained. Storing artifacts like this at every stage of the build pipeline can help enforce immutability. Segregation allows role-based access controls to be applied.
- And talking of artifacts, always validate digests against the relevant artifact in the repository to prevent compromised deployments.
- Requiring two code reviews at the time of pull-requests, together with an 'all tests' passing build, is not optional. Nor, for that matter, is requiring one of the reviewers to be a developer who owns the code. Doing so ensures commits without competent human oversight cannot be made.
- Ensure visibility, and traceability, by requiring all artifacts in higher repositories to be signed. C'mon, preventing untrusted binary deployment to customer environments should be a given, right?
- See control 6? Now apply it to available container images. The same reasoning applies, especially given that such images could have open-source and configuration vulnerabilities. Testing with a relevant security tool in the pipeline before deployment is another given.
- When deploying artifacts from the internal registry, always validate signatures and digests to ensure the one you previously tested is the one that's being deployed. Think the worst of code to ensure the best.
- Scan all the things. Well, all the pre-production controls in production anyway. Such deployed image validation should ensure prior controls were followed.
- Because Kubernetes is responsible for deploying application containers into production, validate the Kubernetes resource manifests. This container orchestration layer is your last line of defence, you need to be sure they have not been tampered with.
- And finally, remember shift-left attackers accessing build environments can bypass earlier build pipeline controls: ensure build environments are both transitory and immutable. Require a fresh environment for every build by defining them in code with automated creation and teardown. Oh, and don't enable interactive logins on build hosts!
It’s time to grasp the secure enabler nettle
Open-source components have been a great enabler for organizations moving ahead with digital transformation processes. They have also, sadly but perhaps inevitably, been just as enabling for threat actors shifting left to hit more potential victims by targeting the software build pipeline. High-profile software supply chain attacks have already illustrated how impactful this tactic can be, and with security standards sorely lacking here I suspect that I, and other business and cybersecurity journalists, will have more headlines to write in the coming months and years.
Unless, that is, developers grasp the security nettle through a process of 'engineered in' protection throughout the software pipeline. It requires people to get talking, not just already time-constrained developers but also open-source repository managers, security teams and those IT executives with their hands on the purse-strings and the go-ahead button. Only time will tell if this blueprint is the ultimate solution, but by building code verification and vulnerability scanning into the workflow, without significantly slowing it down, it's certainly a good start.
- CISA Advisory on Conti Ransomware Warns of Increased Attacks [Is Code Signing the Answer?]
- SUNBURST—Code Signing Was a Problem; It Should Have Been the Solution
- Linux Foundation Launches sigstore to Combat Open-Source Supply Chain Attacks
- SolarWinds: Should Security Live in InfoSec or DevOps? [Ask the Experts]