![What Is Your DevSecOps Manifesto? [Interview with Larry Maccherone] - cover graphic](/_next/image/?url=https%3A%2F%2Fcdn.venafi.com%2F994513b8-133f-0003-9fb3-9cbe4b61ffeb%2Fd6fa7c56-e39d-4128-b177-1072e31fd6fc%2Faid_field_img__b779db13a5a82cdc65f9ab04e8ed69ea.png%3Fw%3D864%26h%3D370%26fm%3Dwebp%26q%3D75%26fit%3Dcrop&w=3840&q=75&dpl=dpl_3EAuLccmwaq4KZ9GcZftJq5NbwLA){kind=small}
Over the past few months, I’ve been speaking with leaders in DevSecOps to gain perspective on how to make security successful in DevOps environments. This time, my DevSecOps conversation is with Larry Maccherone, the DevSecOps Transformation Senior Director at entertainment giant, Comcast. In our first conversation, we spoke about his DevSecOps manifesto.
What was the epiphany that led to you developing the trust algorithm?
Larry: About three weeks after I started at Comcast, I realized that any behavior transformation that we wanted to achieve within the development organization would be impossible with the current lack of trust between devs and security. I mentioned this to one of my bosses and he responded, "I understand that's a problem but it's not as if trust is a formula." My response was, "Well... maybe it is..." I then got up from my chair and wrote the trust formula from Charlie Green at TrustedAdvisor on the wall. We then had a great conversation about how to optimize the terms in that formula for our context.
You published a DevSecOps manifesto. What are your hopes for this? Who should read it and what kind of change does it intend to trigger?
Larry: I have a four-part framework for accomplishing change within a developer organization:
- Win their hearts and minds. The DevSecOps manifesto was my first attempt to convince developers that this is not a normal security initiative. It starts by stating what we value. Every statement is "<some newer way of thinking> more than <some more traditional security way of thinking>." The point is differentiating it from the past so they don't immediately tune out as something they've heard before.
Later, I introduced something else called "The Pledge" which was even more effective at accomplishing this. It states that security teams trust engineering teams to want to do the right thing and recognizes that engineering teams are closer to the business context and will make trade-off decisions between security and other risks. The pledge is to provide information and advice to inform decisions and lower the cost and effort of investment in developer security tools or practices. It recognizes that we understand that we are no longer gatekeepers, but toolsmiths and advisors.
- Create a shallow and obvious path for improvement where the next step is always visible. We do this with a facilitated self-assessment and a series of 90-day plans.
- Engage executives to sponsor the effort and give teams encouragement to make the changes that they need to make. We do this with visualization.
- Reinforcement with community and gamification:
- Create a security guild made up not of security specialists but product development engineers
- Set up vibrant Slack channels where members do most of the responding rather than being assigned monitors
- Hold a quarterly off-site event for the guild
- Empower defacto champions in the organization
- Set up a belt system for gaining expertise
- Practice adoption challenges and capture the flag events
- Etc.
How do you think the manifesto may evolve over time?
Larry: We've already added one new bullet since it was published, "Adopt a few key practices deeply and universally more than a comprehensive set poorly and sporadically." I'll take suggestions for further improvement.
You are directing a major transformation in an enormous (global) enterprise. What are the biggest challenges you face day-to-day evolving towards your vision?
Larry: From a DevSecOps perspective, the way security teams have been engaging with development teams has not been fundamentally conducive to trust. They are polite in a corporate, political sense, acknowledging each other’s work, then they ignore what the opposite party is asking for from them. Behind each other's backs, the developers are essentially saying about the security folks, "Those security people, they just don't get what we do, they don't understand. They're just trying to force mandates on us, and this too shall pass just like every other movement that's been tried to be imposed upon us externally. We can just ride it out, and we will just ride it out." And then the security folks are saying, "Those developers, they're just putting poor quality code out there that's going to get us hacked. They're lazy and they don't care."
With those two mindsets, collaboration and real improvement is impossible. So, it's critical that I get the organization past it. One side is relatively easy. If you want to change the way someone responds to you, start by changing your own behavior. So, it all starts with the way our security people interact with development teams. I'm constantly trying to coach empathy into the security operation. Security folks know their stuff. They have good reasons for what they are asking developers to do, but they're not very good at explaining it and communicating it in a way that actually gets developers to care.
Let me give you an example. One of the leaders in the security organization complained about how non-response a particular development team was being. I asked him to explain the situation to me so he just pulled up an ongoing 10+ reply email chain and had me read it. I very quickly identified the problem. What he had done was essentially repeat his original request over and over again without showing he even read and understood what they wrote in their first response where they explained why they didn't think this was a significant risk. In fact, he had read and understood their comments, but he just never communicated that back to the team. We quickly composed his next response in the form, "I understand <repeat back to them their reasoning>. I understand how <some expression of their feelings>. However, even considering all that, I believe it's a more significant risk than you realize because <short reasons of things they probably hadn't thought of>.” This is similar to the technique that customer support folks are coached to follow when dealing with upset customers.
OK, that's how I coach the security organization. What do I do with the development teams?
I want development teams to own security. Like the agile movement caused the engineering team to take quality into the fold, as well as part of product management, then the DevOps movement brings IT Operations into the fold, and now DevSecOps takes security into the fold. DevOps is a fertile ground for the concept of development teams owning the problem. Times have changed; fifteen years ago we had teams that would literally throw a product over the wall to QA, who would spend a month testing it before it got shipped to production.
We don't have that any more. Teams that write code deploy their own stuff to production, and IT Operations are experts and specialists and advisors and toolsmiths, rather than the folks that actually do the brunt work of operating it. Security needs to do the same thing. As a result, I think the engineering organization is very ready for this ownership message. All they really need is knowledge and tools. So, our job is to just provide that.
Will it upset the security group? Will they feel like they're losing their power?
Larry: Just like it upset all the QA people when the agile movement said, "Let's disband the QA department." That is what’s happening and they're not happy about it. ‘Who Moved My Cheese’ is really what I want security people to be thinking about. The cheese is moving whether you are ready for it or not, and fighting me and being angry at me and telling me I'm not right is just not true. It's going to happen. You're better off getting ready for it and doing something about it now than just continuing to resist it. There are a lot of QA people that can't find good paying jobs now, because they never upped their skills to write automation. And there are very few jobs for pure manual testing with hand on mouse QA left anymore. The QA folks that can't write automated tests are mostly out of a job.
Does this relate to wanting multi-functional or T, Pi or Comb-shaped people?
Larry: That is where it's going. The thing that is really critical is that horizontal line in that Pi or that T. If you don't have coding as one element of that horizontal line, you're pretty much going to be irrelevant, whether you're an ops person, a QA person, or a security person. If all you can do is click buttons on a screen, you're going to be out of a job.
What do the security and engineering teams need to do to address the cybersecurity skills shortage constraint and make sure everybody's got a security stick in their Comb or Pi?
Larry: The key is to spread the problem around. And stop hiring people with security specialist acronyms after their name and no coding experience. When I hire here at Comcast, I drive the HR folks nuts and they drive me nuts, because they reject the resumes I want to talk to and they pass through the ones I don't want to talk to. I want a coder first, and if the applicant has some security knowledge, that's great, but if not I can teach them the security things. But it's much, much, much harder to teach them the mindset of a developer, and if they've never had that mindset, if they aren't actively interested in that, they're going to have stale skills at best. Give me coders and I'll teach them the security stuff. Hire coders into security roles and spread them throughout the security organization. Don't just hire people with security skills.
In part 2 of this interview series, Larry Maccherone shares his vision for DevSecOps in 2020 and beyond.
Machine Identity Security Architecture
Related posts
- DevSecOps, SecDevOps, or RainbowMonkeyUnicornPony? [Interview with DJ Schleen]
- Why Manually Managing Machine Identities Destroys DevOps Automation
- DevOps: Bridging the Gap Between DevOps Security and Agility
- How Vulnerable Are DevOps Certificates? New Study Reveals Weak Use of Cryptographic Security in DevOps