Managing machine identities, such as SSL/TLS certificates is boring, right? It’s not inspiring work and it’s easily overlooked or forgotten in the day to day onslaught of changes and incidents in a typically enterprise technology department. And they seem like such little things… but when certificates go bad, well, life can turn pretty dark. Here are some real-life nightmares that happened as the result of mismanagement of machine identities.
1. Expired certificates delayed breach detection
The notorious breach at Equifax—talk about reputational damage, right? Nearly 150 million customer records stolen including date of birth and social security numbers. That’s a lot of people having sleepless nights about ID fraud thanks to an error somewhere in Equifax’s approach to cybersecurity. Whilst the initial attack was performed via a Struts vulnerability (a common one I still frequently see during application scanning), the detection of the breach took 76 days. The reason it took 76 days to detect: misconfiguration of the device inspecting encrypted traffic on the network. The reason for the misconfiguration of the device: a digital certificate that had expired ten months previously.
2. Mobile service goes dark when certificate expires
One day late last year, tens of millions of people in the UK couldn’t use their Ericsson mobile phones (I was one!). It’s tempting to laugh at the modern world’s insane levels of attachment to our smart little digital friends but what if you were having a life crisis that day, weren’t near any Wi-Fi, and needed help urgently from a friend or family? What if you were doing business on the move? Regardless, this is the service Ericsson committed to their customers to provide so simple inconvenience does count: it was a black day indeed for all involved. The reason for all this disruption: an expired certificate in some of the software used to manage the network.
3. Vote of no-security for political website certificates
Sticking with the UK for a minute (since I’m a Brit), imagine this: your party’s in power, you’re trying to deal with this fairly tempestuous thing called Brexit and reshuffling your cabinet. And your website is down. That’s just embarrassing, right? I can’t claim that I spend an awful lot of time on the UK Conservative Party’s website, but I imagine they get a decent amount of traffic and there were some disappointed people that day. Oh, and how did visitors know it was down? Because they got that screen—you know, the big scary one that shouts, “Your connection is not secure”, lets you know the website owner has not configured their site properly and that the browser will not let you connect to the website for fear of you losing your data. And tells you the reason why: the site is using an invalid security certificate which is invalid because it expired. Hashtag awks.
4. No privacy for 23,000 compromised private keys
So, that was embarrassing for the Tories, but, you know, tech’s not their core business. What if your job is to provide SSL keys and you email 23,000 private keys to DigiCert thus compromising that many websites and customers? Yeuch. This happened because the keys were archived so they were available to Trustico’s CEO, not stored and isolated under the customer’s control. Have there been any breaches as a result? Who knows? This one got intense as customers took to Twitter to share and question the emails DigiCert had sent blaming Trustico and things took a turn towards legal action for defamation. That smells expensive, right? As Kevin Beaumont explained:
“A big element here is Trustico emailed 23,000 private certificates to another company. The whole certificate model is based around only the certificate owner—the customer—having the private key. So Trustico shouldn’t have even had the keys. It suggests the certificates have been exposed security-wise for some time, which defeats their purpose.”
Never hand over your keys. Simple.
5. Expired certificate means NoGo for PoGo
Last year, Pokemon Go, the hugely popular game known to those in the know as PoGo was down for a stress-inducing thirty minutes last year due to an expired SSL certificate. Who knows what those poor gamers did instead for that half hour? More concerning though, is the role certificates play when hackers performed a Man in the Middle (MITM) attack and intercepted the user data. The developers of PoGo, Niantic, didn’t make their API public, but some people hacked it anyway, and then created programs posing as users. Here’s what happened next, according to Abraham Lin:
“The dozens upon dozens of apps that sprung out following the unofficial API breakthrough involved each using thousands of these fake users (or bots) asking the Niantic servers for nearby Pokemon. Placed within a specific, equal distance from each other all over any city, and refreshing the server calls made by each bot every 15/30 min to check for new Pokemon, these apps were able to generate large scale, comprehensive maps of Pokemon locations over hundreds of cities across the world, at the cost of millions (or more?) calls to Niantic servers each minute. The veritable stampedes of people running ten city blocks for a Lapras or Snorlax was really only made possible by the existence and use of these maps.”
So, back to the certificate’s role in the human stampede: PoGo was using SSL—but it wasn’t correctly pinned. The MITM attack was possible because of this; when Niantic added certificate pinning that required a certificate to originate from the PoGo app directly to be recognized by the servers, those hackers whose certificates were only signed by a certificate authority (CA)—and not the PoGo app itself—were now out of the loop. Problem solved!
6. Social networking misses connections when certificate expires
Those pesky expired certificates, huh? It’s tough to keep track of them as already shown, but here’s another example from one of the most popular global social networks, largely populated by us in the tech industry, LinkedIn. Millions couldn’t access the website and plenty were left with an insecure connection.
Security researcher Scott Helme, you can find out more about him here, on LinkedIn (the irony!) pointed out:
"This is a good indicator that they may still be renewing and replacing these certificates manually and not using an automated process."
Note the word ‘still’ in that sentence—the learning? If you are not yet automating your certificate management, you should be.
7. VPN stops working when certificate expires
And another one, this time from tech giant Cisco, who should know better right? Symptom: VPN stops working. Cause: an expired SSL certificate.
One teeny certificate has the ability to compromise your cryptographic posture. As Richard Chirgwin at the Register put it: “Well that's one way to secure systems: deny new trustpoints.”
8. Internet company fails to replace certificates after breach
Talking of “cryptographic posture” (cool phrase, huh?) this next sorry tale isn’t directly connected to certificate management, but shows just how it’s not that difficult for an actor to make some fairly valid assumptions about your approach and commitment to cybersecurity based on the way you treat certificates.
So, Yahoo, yeah, they still exist, just. They are called Altaba now. Starting back in 2013 they lost three billion customer records in two breaches—they didn’t report them until 2016. They were still paying for that misdemeanor as recently as October last year, when a new settlement ordered the company to pay $50 million in damages, cover attorneys' fees up to $35 million and provide free credit-monitoring services to victims of the breach. They’d already paid a $35 million penalty to the Securities and Exchange Commission (SEC) for failing to notify customers of the breach in a timely fashion earlier in 2018. So, er, that was painful.
Venafi’s Labs team took it upon themselves to have a closer look at the cryptographic posture (I just can’t stop saying it) at yahoo.com. They found evidence that some certificates had not been replaced since the breach, that a very low number of certificates had been replaced in the last 90 days, that some certificates were using the highly vulnerable MD5—basically a whole load of stuff that led Hari Nair, director of product management at Venafi, to conclude:
“Any one of these cryptographic issues would leave an organization extremely vulnerable to attacks on encrypted communication and authentication. Collectively, they pose serious questions about whether Yahoo! has the visibility and technology necessary to protect encrypted communications and ensure its customers privacy. Our team has been working on a major research project that led us to believe that there is usually a high degree of correlation between weak cryptographic controls and overall cybersecurity posture.”
9. Certificate authority mistakenly issues fraudulent certificate
This is a story about being famous for the wrong reason. A simple mistake was made at certificate authority, TurkTrust, when they issued two intermediate SSL certificates. One was immediately revoked, the other was not identified and went into service in December 2012 when the certificate owner, EGO, switched on SSL keybridging in its web proxy.
A few days later a Google Chrome user got a warning about an unexpected certificate claiming to represent a google.com web property thanks to Chrome’s feature, public key pinning, which equips the browser with both a list of presumed-good root CAs and a list of known-good Google SSL certificates. And so the world went wild with the imagined potential repercussions of an intermediate certificate being in the wrong hands and crazy conspiracy theories. Nothing, to my knowledge, actually happened in terms of the error leading to a breach but it certainly gave Turkey some airtime.
10. Certificate authority forced to revoke 9,000 certificates
And finally, GoDaddy, you, dear reader, probably have a domain or two registered with these guys yourself. I know I do. They are one of the world’s leading CAs and domain registrars after all. They had a defect in their domain registration system that, when they discovered it (after five months) meant that they revoked 9,000 SSL certificates affecting over 6,000 customers. Whilst that’s only 2% of their customers, and GoDaddy point out that they have issued over ten million certificates in the previous 13 years, that’s still quite a large group of people with some annoying and to them, superfluous admin to perform. Not to mention the overhead to GoDaddy themselves to resolve.
The bug meant the system could have been validating certificates when it shouldn’t have done, potentially opening a door for inquisitive hackers. Whilst it looks like nobody found and exploited the problem in the 5 months the machines were being issued certificates to validate machine identities, is it possible that sometimes we never know?
So, there you have it; when machine identities go bad - when they are neglected or mismanaged –the consequences can be literally catastrophic. There is no doubt that the job of managing them properly is complex and can be time-consuming (not to mention uninspiring). There are solutions that can help organisations automate certificate enrollment, installation, deployment, renewal, and revocation to assure machine identity management. Take a look at Venafi.