Certificate-related outages rank among the most embarrassing and avoidable website disasters. Especially for large, well respected companies like Microsoft and Cisco, who recently experienced such outages. But with the rapid growth of certificates and shortened certificate lifespans, even the most security conscious organizations still have trouble preventing outages caused by expired certificates.
In a recent survey, CIOs estimated that the number of certificates they use in their organizations would more than triple to around 1 million machine identities by 2024. No surprise that outages are increasing as well. 80% of companies that report outages have a minimum of three per year, 55% have 12 or more per year and a shocking 25% have weekly outages (52+ per year).
In the past few months, tech giants such as Microsoft, Cisco and others have fallen prey to security certificate-based outages. Read more about what happened, speculate on the factors behind these lapses in certificate management, and how Venafi can help you avoid similar complications in your own internet offerings.
Expired certificate at Cisco took down vEdge SD-WAN
When a security certificate expired on the TPM chip of Cisco vEdge Routers, apparently both Cisco and its customers were caught by surprise. Cisco defined the problem as “an expired certificate affecting control plane connections, which eventually impacts data plane connections resulting in loss of service.” The Cisco bulletin continued warned of impact to data plane connections and SD-WAN downtime.
Ultimately, the certificate-related outage left vEdge based SD-WAN customers waiting to see if their WAN would evaporate while looking at ways to rearchitect their WAN to maintain connectivity.
Expired certificate causes failure in Microsoft WinGet package manager
Back in February, users of Microsoft's WinGet package manager began complaining that the popular tool was not working, with many seeing "InternetOpenUrl() failed" errors. It did not take long for affected users to pinpoint the problem as an expired security certificate.
Soon after, Bleeping Computer reported that Microsoft's WinGet package manager was “having problems installing or upgrading packages after WinGet CDN's SSL/TLS certificate expired.” This proved to be a rather embarrassing error for Microsoft as they faced the aftermath of forgetting to do something as basic as renewing the SSL certificate for WinGet.
Expired certificate causes outage at Starlink
Back in April, Elon Musk’s Starlink went down for several hours over an expired digital certificate. Starlink is the satellite internet constellation operated by SpaceX. Musk himself tweeted that the severe downtime was “caused by expired ground station cert.”
When a certificate expires, the authenticity of a service like Starlink cannot be verified, and systems that rely on it will stop working. That means that machines in that constellation will no longer be able to connect or communicate. In the case of Starlink, the impact of that would be beyond global.
API certificate expiration debilitates Real Debrid
In March, download manager Real Debrid users complained that the tool was down, not working or connecting with any service. Real Debrid users were also unable to connect the tool with supported services or add-ons. It soon became apparent that outage was caused by the expiration of the Real Debrid’s SSL/TLS certificate.
Commentator Jean Leon lamented that “The reason why the company forgot something as important as renewing its SSL certificate is still unknown.” However, the issue was eventually resolved and the team at Real Debrid Team explained that “a delay while reissuing the renewal of the SSL certificate for their web API was the reason for the outage.”
Why is effective certificate management so critical
Expired certificates not only cause outages but can also act as the gateway for criminals to infiltrate corporate networks, notes Pratik Savla, Lead Security Engineer at Venafi.
“Not only can expired certificates cause unplanned system or service outages as has been seen several times over in different incidents, but what is not equally well-known is that they can also open the door through which malicious actors can find entry into one’s environment,” Savla said.
Proper and timely renewal of expired certificates is key to mitigating man-in-the-middle attacks, according to Savla. “The first step is to make sure that you develop and continuously update a detailed certificate inventory. Next, expiry notifications should be setup to ensure it reaches the right owners ahead of time. This includes a set period starting at least a month before the expiry date for non-critical systems and starting with at least two months before the expiry date for systems deemed critical.”