I was recently speaking with a large organization that had discovered that it had almost 2 million SSH key pairs. Yup. That’s about a million distinct keys. In other words, they had a lot of machine identities to manage and protect. But that’s not even the whole of it. I suspect that when they complete their discovery, they will have identified at least twice that number. How, you might ask, did this type of SSH sprawl proliferate to this point? My answer is quite easily.
I think that it's fair to say that in most organizations SSH keys are just being created to solve a particular problem. And when that problem is solved, administrators quickly move on to solve the next problem. The good news is that you get a lot of problems solved. But the bad news is that you may also get a lot of stray keys floating around.
At some point, you need to start answering questions, like “How many of those host keys are we actually using?” and “Do we still have user keys for people who don't work here anymore?” and “How can we find keys that have just been sort of lost?” But many organizations are not prepared for the aftermath of such investigations. So they just never begin.
One organization went so far as to tell me, "We are not interested in doing discovery." These individuals are (quite legitimately) concerned that if they find something out of place, they will 1) be blamed for it and/or 2) have to drop everything and fix it. In other words, "If we don't know about it, and we can prove that we didn't know about it, then it's not our problem.”
It’s not that they are ignorant about the security implications of SSH key sprawl, it’s just that their plates are full of other security concerns that all require more or less immediate action. They simply have too much to do and they need to constantly prioritize the urgent from the important. It's like shoveling snow in a blizzard. It’s easy to ignore the snow that’s already been piling up for weeks. That’s not an emergency. At the moment, you're just trying to clear a path to the door.
Of course, they would like to be thinking about proactively solving problems before they arise, but they almost never have the luxury of doing that. Most often, organizations only fix things because they have to fix them, because it's broken and somebody's shouting at them to fix it. If somebody's not shouting at them to fix it, it can wait because there are other things other people are shouting at them to fix.
So, it’s understandable that many say, "We know what we've got, and we know there are problems with it, but we're just going to leave well alone, because we've got a lot of other things to think about." And every now and then they realize that they have been leaving something alone for three years, and they'll say, "My Gosh, it's going to be a problem to change that. Let's leave it alone for another three years." After those second three years, they've actually got more than double the problem than they would have had if they'd just refreshed earlier.
Sometimes they realize that with a rude awakening. In the worst-case scenario, they become aware of a problem when they trace an initial attack vector back that leverages an unmanaged SSH key. A few years ago, if you'd suggested SSH as an attack vector, folks would say, "Don't be silly. That's way too farfetched. Nobody is going to go and look for people who've stopped working for a company and try and break into their machines in the hope that they might find keys that will give them access to their old employer." Au contraire. That's exactly what attackers will do.
SSH Machine Identity Management for Dummies
Even then, it’s not that easy to address. Organizations may recognize they have a problem and they may realize they need to address it, but there’s always the danger that attempting to fix it could make it worse. It’s not an easy task to map SSH key access. And deleting or updating these keys can have unexpected results. Many are worried that looking on every machine will just open up a can of worms.
No one wants to have fingers pointed at them and hear, “You were looking for stuff, and now it doesn't work exactly the way it’s supposed to.” The assumption when irregularities are found is that the person who starts looking for them and finds a problem should have known about it in the first place.
You can sort of understand where this idea comes from. The last person to do anything unexpected on a machine is the first person we will look at when we discover a problem. So hats off to the brave souls who commit to doing the right thing for the organizations, regardless of the potential personal repercussions.
Our job at Venafi is to make this process as painless as possible. We help organizations map their SSH exposure and put into place policies that minimize it. So organizations can automate the rotation of SSH keys on a regular basis, without the finger pointing and tongue lashing.
How well are you protecting your SSH keys?
Get a FREE & Confidential SSH Risk Assessment from Venafi!
Related posts