At the 4th annual DevOps Enterprise Summit (October 28-30) in Las Vegas, Nevada, Venafi conducted a survey of attendees. According to the survey, 55% of respondents stated that their organization experienced a certificate-related outage in the past 12 months. In addition, 75% stated they are concerned that corporate certificate issuance policies slow down development.
Additional findings from the survey include:
- Forty-eight percent are confident that developers always request certificates through authorized channels.
- Thirty-nine percent believe that developers should be able to circumvent corporate certificate issuance policies to meet SLAs.
- Seventy-five percent are unfamiliar with the Securing Web Transactions NIST 1800-16 Practice Guide (Draft)
With DevOps, automation immediately comes to mind. With high levels of automation, why are certificate-related outages still an issue? And, why isn’t policy enforced as code? While at the DevOps Enterprise Summit, Venafi spoke with many IT Managers, DevOps Engineers, Tech Leads, and even C-level executives about their current processes for SSL/TLS certificates. After many in-depth conversations about current practices, several themes surfaced that explain the survey results in vivid detail.
CIO Study: Automation Vital to Address Shorter Lifespans and Massive Growth of TLS/SSL Certificates
Theme 1: Public-trust certificate issuance and installation often issued via ticketing systems
Publicly-trusted certificates—such as those issued by certificate authorities such as Entrust, DigiCert, Comodo and GlobalSign—still tend to be requested manually, via ticketing systems such as ServiceNow. Often, the process goes like this: DevOps teams submit a ticket, wait for security/PKI teams to issue the certificate, download the certificate and then provide it to their infrastructure team (e.g. F5 administrators) so that it can be installed on the load balancer fronting the application. The DevOps team then also installs the same certificate on the appropriate server for the application.
After speaking with a network operations manager who manages F5 load balancers at a bank, it became clear why this could result in certificate-related outages on critical business applications. The F5 BIG-IQ administrators have visibility to upcoming certificate expiries but often do not know who owns the application. To make matters worse, application owners often do not track these certificates closely enough. As a result, once the certificate expires, this causes an application outage with P1 level urgency for operations teams, security teams and application owners. This was a common theme heard over and over at the DevOps Enterprise Summit this year.
Theme 2: Kitchen-sink solutions for certificates are all too common
DevOps has redefined how software is brought to market. Many tools now exist to automate pretty much anything, whether that’s pushing code, checking for vulnerabilities or testing code. But, certificate processes are often an area that hasn’t been tackled yet. Instead, today there’s a veritable kitchen-sink of solutions that vary across environments. These can include:
- Using OpenSSL for self-signed certificates
- Directly coding against certificate authority APIs
- Using ACME for requesting Let’s Encrypt certificates
- Leveraging things like Amazon Certificate Manager or built-in secrets tools such as Ansible Vault, Pivotal CredHub and Kubernetes secrets
The result? Developers are spending (wasting!) a ton of time coding custom solutions rather than asking security for help—or demanding a standardized certificate service. Although certificates are increasingly important since they serve as machine identities, they are often seen by developers as an afterthought. But you may ask why machine identities are so important in the world of modern applications built using DevOps practices and tools?
The truth is, machines have evolved over the past decade. Machines today are comprised of containers, infrastructure-as-code, virtual machines, microservices, serverless, and also include traditional infrastructure. Because many organizations employ a hybrid cloud strategy, machines are running on-premises and across multiple clouds. Consequently, modern applications often need to communicate across clouds or from a cloud to on-premises. For that to occur securely, all machine-to-machine communications have to be encrypted using machine identities that are trusted and known to the security organization.
For that reason, machine identity management is critical to enable digital transformation. So, why do most modern applications deployed using CI/CD pipelines not have an automated, standardized and policy-driven workflow for certificates? Could be it that…
- Security teams simply don’t have a way to help application development teams?
- DevOps teams are moving too fast for security and want to use their existing tools wherever possible?
- Development and operations teams are still not utilizing the same processes and tools for a given application?
The answer is: all of the above. But, let’s address the elephant in the room.
Theme 3: Security policy is perceived as slowing down development
If we really look at what’s happening, there are good, but not great, reasons why so many of the certificates used in modern applications are not compliant with security policy. These reasons may include using certificates that come from unauthorized certificate authorities, deploying weak certificates or wildcard certificates or trusting certificates that are unknown to the security organization.
The survey results make this very clear. It is apparent that DevOps practitioners view corporate security policies as slowing them down. At the same time, more than a third believe that speed should come first and that developers should be able to circumvent corporate certificate issuance policies to meet SLAs.
Theme 4: Regulatory guidance often not top of mind for DevOps practitioners
In addition, it takes time for new guidance from organizations like the National Institute of Standards and Technology (NIST) to gain mindshare from anyone outside of security. In our survey 75% of respondents stated that they were unfamiliar with the Securing Web Transactions NIST 1800-16 Practice Guide (Draft) which states, “As organizations push for more rapid and efficient deployment of business applications, many DevOps teams deploy certificates without coordination with the Certificate Services team. This can result in certificates for mission-critical applications not being tracked. This can be particularly problematic if bugs in DevOps programs/scripts cause certificates to be improperly deployed or updated. In addition, as DevOps teams adopt newer frameworks and tools, it is important to continue to monitor certificates and applications deployed and maintained by older DevOps frameworks and tools.”
The NIST guidance also states that security teams should maintain an up-to-date inventory of all deployed certificates and that this inventory should include information on each certificate such as the:
- Subject Distinguished Name
- Subject Alternative Name
- key length
- expiration date
- validity period
- group responsible for the DevOps technology used to deploy the certificate (if the certificate was deployed via DevOps technology), as well as many other fields.
As you can see, DevOps and security are currently at odds. What gives? Speed or security? It’s a tight race. As you can see 39% of respondents stated that speed should take priority.
Is there a better way forward? Yes, it requires a multi-faceted approach to automation
Because the application development lifecycle is moving at a faster pace than ever, security teams who used to leverage periodic or manual processes have to get involved much earlier in the lifecycle. This will allow them to find and fix certificate issues in partnership with the network operations and application development teams before they ever make their way into production. To do so, they must invest in automation, otherwise they can’t keep up with the requirement for speed—especially if they are using manual processes or policy non-compliant kitchen-sink approaches.
Early in this blog, we discussed the challenge of certificate-related outages. To better serve their customers, F5 Networks has developed an integration with Venafi to support automation of certificates. With BIG-IQ and Venafi, it’s now possible to automate and orchestrate keys and certificates to secure the lifecycle of your machine identities across all F5 BIG-IPs. This integration allows you to use a standard, compliant certificate-creation policy while also ensuring strong security. All in all, this helps you keep outages at bay on business-critical infrastructure and empowers teams to deploy new applications faster. But, this only addresses a small fraction of the pain felt today. Security teams must do more to bake in policy as code when it comes to machine identities.
Security teams must look at what tools developers are using and embed their policies seamlessly into the existing toolchain, using code. By delivering a standardized set of consumable services for autonomous application development teams, security relieves DevOps of the burden of creating their own security infrastructure (aka the kitchen sink) and makes it easy for them to comply with corporate machine identity policies so they can ultimately, move faster more securely.
Taking a programmatic approach allows for “baking in” trusted certificates into DevOps continuous integration/continuous delivery (CI/CD) pipelines is the new gold standard. This can be easily accomplished by using a standardized REST API that’s integrated into DevOps tools to abstract away details regarding certificate issuers. If this is done within the existing DevOps toolchain where developers can continue to use their existing tools, you establish a whole new paradigm for injecting machine identity management into DevOps workflows.
Proactive InfoSec and development teams know that the security policy should be treated the same as any application code—it should be defined as code, versioned as code and tested as code. Here’s to next year where we hope to report that DevOps, operations and security teams will have come together to solve this difficult challenges with eloquent solutions such as the Venafi Control Plane for Machine Identities and TLS Protect Cloud.
Get a 30 Day Free Trial of TLS Protect Cloud, Automated Certificate Management.
Related posts