An outage that is triggered by an expired certificate is usually a wake-up call for an organization to start paying attention to machine identities. It reminds them that there are machine identities within their organization that they can’t account for or that they may not fully control. And that’s the point where most organizations realize they have a problem. And that problem goes beyond the potential loss of revenue and drain on productivity. If they don’t know where their certificates are, then the organization won’t know what those machine identities are granting access to and who’s using them. Worst case scenario is that cyber criminals are misusing unprotected machine identities.
But let’s stick to outages for now.
There are still major organizations who keep experiencing certificate-related outages. But they are taking the stance that outages are just a cost of business. Their contention is that fixing periodic outages and dealing with the aftermath is cheaper than investing what it would take to tightly control all of their machine identities.
Here’s a case that’s not all that uncommon. One of my colleagues was speaking with a customer that uses at least 100,000 certificates and they are only managing about a quarter of those. But it’s the other three quarters that will continue to plague them. In my opinion, that is a bit short sighted. And I’ll tell you why.
"a customer used at least 100,000 certificates and they are only managing about a quarter of those"
First of all, I would estimate that a major outage costs a large organization about a million dollars, including lost revenue, productivity downtime and diverted IT resources. And honestly, I'm probably being too conservative. In the 2019 Cost of a Data Breach report by Ponemon Institute reported that the average total cost of a data breach in the U.S. for the companies studied has grown to $8.19 million in 2019. That's a big jump from the from the $3.54 million reported when the first Cost of a Data Breach report was issued in 2006! Many outages could cost much more than that.
The Ericsson outage cost over £100 million
Examples we see reported in the news show extremely high outage costs as well. We recently sponsored a CIO study on certificate-related outages and one stat included in the report was the likely cost of the Ericsson/O2 outage in December 2018 which was caused by an expired certificate. That outage left 32 million customers without access to mobile phone and data services and may end up costing over £100 million. So, a platform for machine identity management would seem to be pretty easy to justify from a purely cost of purchase standpoint. But that’s not what’s holding up many companies. It’s the expertise and staffing that they need to extend that support across the enterprise.
Often, when people calculate the cost of managing PKI, they base it on the two or three people on the PKI team. But it's just not realistic to have three people managing a certificate inventory on a spreadsheet, which is what most people think it is. They mistakenly assume, "I'll just add another person to solve the outage problem, it's not a big deal."
This may seem a bit far-fetched, but we spoke with an organization just the other day who supports an outsourced staff of about 15 who are managing their certificates using a spreadsheet. Yeah. And these poor folks get the blame any time there’s an outage or a security event. It's not their fault. And they wonder why the offshore PKI staff can't identify the business owner. Well, of course they can't, because the business owner left six months ago and no one put that on the spreadsheet, because they didn't know about it.
The spreadsheet method: not working
This leads us to an even bigger cost factor: the responsibility of managing certificates extends across an enterprise of untrained and ill prepared certificate owners. Even those who have been working with PKI security for years may have to stop and think about it if they were asked to renew a certificate. Depending on how long it’s been, it might be easy or might require a consult with Google.
Can you imagine what happens if people that have never been exposed to certificates, are told to generate a CSR? Most likely, they do that "How do I generate a CSR?" Google search. And even when they got the steps to follow, they may not fully understand the process. So, they may go and find the old certificates and copy and paste them because they don’t know any better. But chances are that they won't renew their keys, and then they get rejected. And amount of wasted effort ends up in just hours and hours of grief and frustration all over the business. And that adds up to costs that are not being counted. Times that effort by hundreds of systems administrators and line-of-business owners and you see the scope of the problem.
"...it's time to get back to the basics and invest in managing machine identities, not obscure scenarios"
The simple issue is that we don't spend enough money trying to solve the everyday problems. Yet, in many cases, we’re spending millions preventing against obscure security scenarios that, for whatever reason, have become high profile. It’s time to get back to the basics and invest in managing the machine identities that keep our businesses up and running with secure machine-to-machine connections and communications.
The CIO Study: Certificate-Related Outages Continue to Plague Organizations mentioned earlier shares survey data from 500 CIOs from five countries on the frequency and scale of outages as well as a list of best practices you can use to address them. It's available for download now from the Venafi website.
- Venafi Study: Are Financial Service Organizations More Likely to Suffer Certificate-Related Outages?
- Majority of Businesses Still Experience Outages: Are You Protecting Your Certificates?
- GAO Report: Expired Certificate Allowed Extended Exfiltration
- How Big Is Your Risk of Certificate-based Outages?