The Disconnect between DevOps and Security
Modern application development teams need to move fast to develop products that are customer-centric and in line with market needs. The problem is that the DevOps teams’ speed is often far greater than the security teams’ operational tempo. As such, developers may view Information Security as a barrier rather than an enabler. Plus, they have a vested interest in staying within their “siloed” toolchain.
Secure software development requires following the following policies during application development and testing:
- Production data is not to be used in a test environment, mostly due to lack of security and integrity controls
- Developers are prohibited from accessing production systems unless their role explicitly requires it (this may vary based on team and organization size)
The above policies indicate that development and testing environments should be separated from production. When it comes to using different certificates based on the environment, there are excellent reasons for doing this. Unfortunately, existing security processes often do not enable developers to get the certificates they need at the speed they require it.
Security processes for SSL/TLS certificates are often antiquated and require manual steps such as submitting a ticket. As a result, developers look for an easier way, often using certificates without the security team’s stamp of approval. Instead, they create their own security infrastructure, issuing certificates using a combination of OpenSSL, secrets management tools, DevOps platforms and scripts. They can then issue certificates as fast as they need them, but that comes with security risks.
The hard truth is that developers are not famous for following corporate security policy. As the book titled “The New Kingmakers, How Developers Conquered the World,” developers now have access to open source and cloud computing which enables them to be largely self-sufficient.
In addition, development teams are focused on rapid SLAs, continuous delivery and compressed timelines. By leveraging scripts or readily available options within their DevOps platform, their certificate management processes often are not compliant with the enterprise-wide security policy. By doing this, developers may inadvertently create a false sense of trust in unauthorized certificates, whether they are issuing self-signed certificates or spinning up their own internal certificate authorities (CAs) that are outside of corporate security policy.
The current state of certificate provisioning within modern application environments also poses a variety of challenges:
- Certificate provisioning is not embedded into software-defined infrastructure
- Development teams are unable to easily comply with policy
- Information security, risk and governance teams have little visibility into certificates
- Certificate-related outages continue to plague organizations
The Dangers of Using “Test” Certificates in Production
Visibility, accountability and ownership are the topmost challenges when it comes to managing certificates across the full software development life cycle. Most organizations underestimate the number of active certificates in use on their systems. For example, one major retail organization estimated it had 5,000 active certificates, but in reality, it had 20,000! Even more alarming, more than 5,000 certificates had no owners, and no one knew what they did, what they allowed, and who was responsible for them. This lack of insight can easily occur when organizations use self-signed certificates issued outside security-controlled processes.
Remember the Heartbleed vulnerability from 2014? That flaw showed exactly how organizations blindly trust keys and certificates and how easily cybercriminals can exploit a trust-related vulnerability.
The underlying assumption is that “test” certificates are to be used only in development environments and that they should be replaced by corporate-compliant certificates when the application moves to production. But this does not always happen according to plan. As a result, organizations are exposed to a veritable avalanche of certificates that they are not aware of and they cannot discover.
According to a Venafi study, 62% of mature DevOps organizations consistently replace development and test certificates with production certificates when code rolls into production. In organizations that are just adopting DevOps practices, only a 36% of the respondents follow this critical best practice. Without changing certificates, there is no way to distinguish between the identities of trusted machines that are safe to place in production and untested machines that should remain in development environments.
Unauthorized CAs raise another issue. Every time developers set up an internal CA, they are distributing that CA as a root of trust within the organization. Accordingly, they create a set of applications that are configured to “trust” this CA and the certificates that it issues. Without the proper visibility into your application trust stores, it’s very difficult to remediate this issue. Even if you re-issue these certificates under corporate policy, you’re still left with the challenge of eradicating the unauthorized CAs from your trust store.
How Do You Solve this Problem?
Here’s the dilemma. Do you force your application development teams to follow security policies for certificates and jeopardize their SLA? Or do you allow them to create self-signed or unauthorized subordinate CA certificates to move as fast as possible and boost your population rogue certificates? The answer is neither. Ultimately, you need to provide development teams a safer process that gives you visibility into all certificates, while also providing developers easy access to certificates from within their native DevOps toolchain.
By “shifting security left” into the continuous integration/continuous delivery (CI/CD) pipeline, your security team can prevent vulnerabilities early in the development process. In “shifting security right,” you can enable your certificate PKI team to provide as many machine identities as the development team needs, when and where they need them, across every environment, from development to production. This security shift can be achieved effectively through automation using a platform that integrates with the tools in use. Proactive InfoSec and development teams know that the security policy should be treated the same as any application code—it should be defined as code, versioned as code and tested as code.
Taking a programmatic approach allows for “baking in” compliant and trusted certificates into CI/CD pipelines. This is the gold standard and should be the blueprint to managing and protecting machine identities in modern applications. If this is done, developers can continue to use the tools they love, while security can ensure compliance and audit against it.
How much do you know about the certificates used in your containers, microservices, and other infrastructure-as-code?
- DevOps and the Proliferation of Secrets
- Accelerate DevOps by Offering a Certificate Service for CI/CD Pipelines
- How Can You Be More Successful in a Compliance Conversation with DevSecOps?
- X.509 Certificate Issuance: Too Slow for DevOps?