This is part four of the blog series, CALMS for DevSecOps, and this time I am looking at the ‘M’:
MEASUREMENT
In the last post I looked at DevSecOps through the lens of lean and concluded that the key metric for lean, in the context of flow, is cycle time. Essentially, we want value outcomes in our customers’ hands as fast as possible and this is the key element of a DevOps business case whilst at the same time not compromising system integrity: throughput AND stability.
In the DevSecOps pattern we want security embedded in the value stream flow. That means:
● Security requirements (NFRs) are built into the development backlog
● Zero time is spent waiting for security approvals or testing during development
● Security tests are embedded into CI/CD pipelines
● Security testing requires no touch and no wait time
● IT Ops Security actions are available as governed self-service to developers
Following this pattern means security doesn’t represent waste in the value stream, so the cycle time is optimized from this perspective. But what other metrics are of interest to DevSecOps? Peter Drucker said: “If you can’t measure it, you can’t improve it.” and if DevOps is nothing else it’s a set of patterns for ways of working that improve organizational performance. So how do we measure the efficacy of DevSecOps and the delivery of value? How do we determine when DevSecOps is no longer a ‘thing’?
In a previous blog post, ‘Making A BusinessCase for Machine Identity Intelligence’, I explored a number of metrics which broadly fall into two categories:
- Cost of breaches
- Cost of avoiding breaches
The cost of breaches is an interesting one, and difficult to get from a system. It’s a bit like cost of outage, in that not all web applications are transactional so it’s not as simple as saying the application was down for an hour and it transacts $1M per hour so we lost $1M. I recently facilitated a DevSecOps workshop at the Institute of Physics in the UK. Whilst you can apply and pay for your membership to the institute on the website, that’s not its core purpose. The institute is there to support their organizational purpose to gather, inspire, guide, represent and celebrate all who share a passion for physics and ensure that physics delivers on its exceptional potential to benefit society.
Similarly, I work with a bank with whom you can apply for a credit line through the website, but their governance, from a vulnerability perspective, demands that credit is only approved following a telephone conversation. The core of the transactions they do online is around collecting payments—if that goes down, it just picks up where it left off later.
The cost of a breach isn’t just system downtime—and many types of breach don’t require any downtime anyway. When a threat agent succeeds in getting into your system, stealing your customer data and you find out nine months later, that’s not your problem. Your problem is now the clean-up operation, your reputational damage and any fines (see in particular GDPR) coming your way. Oh, and if anyone’s going to jail (that CYA ‘sign this to say you know the risk’ behavior is not so clever now, right?).
So what’s the cost of avoiding the breach? It’s a significant investment in people and systems and this is where measuring your DevSecOps improvements comes in useful. Here are some examples:
- Time saved from avoiding security approvals - security is involved in process and the autonomous team has gained required knowledge and automated tests
- Time saved from managing certificates using a machine identity protection solution
- Time saved from IT Ops doing firewall changes because product teams are serving themselves, securely
Remember: time saved ultimately means value outcome received sooner.
In summary, DevSecOps measurements seek to do three things:
- Show no additional time/cost in the value stream flow for security activities
- Avoid all breaches or have near instant remediation
- Optimise and automate all security activities through the end-to-end pipeline
When those three things are achieved, we don’t need to talk about DevSecOps anymore. It’s just DevOps, or even better, just the way of working.

Machine Identity Security Architecture
Related posts