Month end is generally a busy time for financial institutions—especially for payroll. End of week is much the same. And end of quarter is even busier as organizations wrap up their fiscal business. Apparently, the end of the month is also when many certificates are set to expire. When all of those events happen simultaneously, it’s a recipe for panic and frustration—as Bank of Ireland learned last month.
On Friday June 30, 2023, an expired certificate triggered an outage at the Bank of Ireland. The outages prevented online and mobile customers from accessing their accounts and services on one of the busiest days of the month.
The bank initially acknowledged the outage, stating that they were “aware some customers using our mobile app and 365Online are currently experiencing difficulties” and that they were “working to fix this issue as quickly as possible and apologise for any inconvenience caused”.
Services were down for most of the day. And the impact of this was so wide that the bank had to take immediate action to extend business hours, including weekend openings, and divert resources to its customer support team.
CIO Study: Financial Services Organizations Face Escalating Certificate Outages with Massive Growth in Machine Identities
Kevin Bocek, VP Ecosystem and Community at Venafi, commented in Continuity Central: says that the whole issue was completely avoidable:
“Having an outage of this scale on the last Friday of the month is a disaster for a bank. Customers have been unable to pay wages, bills have gone unpaid, it’s chaos. The operational costs of having to open over 90 branches and add additional staff to customer services will come with a hefty price tag, but the long term impact on trust and reputation could be even worse. And it appears to have been totally avoidable, as it was due to an expired SSL machine identity.”
The Irish Times cataloged a number of comments that unhappy customers shared on social media. One customer ranted, “Absolute amateurs. Been trying to log on for nearly three hours now. Imagine you’re a client with bills to pay, urgent monies to transfer out, and you cannot access your account because Bank of Ireland cannot run a stable or modern banking platform.”
Another wondered, “Any information on the issue with everything related to BOI today? Can’t access the app, either on mobile or laptop. Can’t make any payments with my account online either. What’s going on?”
So how does something like this happen to a major bank? “It all comes down to machine identity management, notes Bocek. “SSL machine identities authenticate that a website is trusted, enabling secure, encrypted communication with other services. They are issued with a set lifespan, if they are not renewed and left to expire, the machines can no longer authenticate one another and therefore cut the connection causing an outage. The challenge for organizations when this happens is that if you do not have control and visibility of all the identities in use across the organization it can be like finding a needle in a haystack.”
The best way to prevent outages is to be proactive. Look for an automated solution that helps you discover all your certificates and then monitors those encryption assets for vulnerabilities and signs of misuse. Bocek recommends that, “Organizations can have hundreds of thousands of machine identities. Managing them manually is an impossible task – particularly with the growing use of cloud native technologies spurring a rapid escalation in the use of machine identities. Yet by having a control plane to automate the management of machine identities throughout their lifecycle, you can eliminate the risk of outages of this kind.”
Venafi has been helping the world’s largest organizations prevent certificate outages for over a decade. See our full list of steps you should follow to eliminate certificate outages.
Are you concerned your apps will be impacted by an outage?
VIA Venafi: 8 Steps to Stopping Certificate-Related Outages
Related posts