This weekend, I got a call from Josh Corman who wanted to talk with me about some comments I made on the Southern Fried Security Podcast a few weeks ago when Cindy Valladares and I were invited to discuss the 2012 State of Risk Based Security Management report. Since my comments apparently came across different from the way I intended, I thought I’d elaborate on my thoughts here.
Metrics vs. Tracking Indicators
One of the comments I made on the podcast was that some of the things we love to track are, in fact, terrible metrics. A specific example I cited was tracking the number of vulnerabilities in your environment. I still stand by my claim that this is a terrible metric, since it is not something you can control directly – vulnerabilities are within your area of concern, but for the most part are outside your area of control.
Unfortunately, it seems I came across as saying we shouldn’t track the number of vulnerabilities in our environments – that was definitely not my intent. I was trying to make the point that I don’t believe anyone’s bonus or performance rating should be based on the number of vulnerabilities in their environment – that’s a losing proposition.
In my view, Metrics are things you can be measured against and represent things you can directly influence. Indicators (especially Tracking Indicators) are data that inform your decisions, and typically report on things that are relevant to you but not within your direct area of control.
Tut-tut – it looks like rain
In hopes of illustrating this concept, I want to use a non-technical story. Every day I look at the weather forecast (a classic Tracking Indicator), and I adjust my attire and weather “countermeasures” accordingly. Umbrellas, warm clothing, hats, and things like that are among my available countermeasures. I may also adjust my actions based on the forecast – for example, deciding to cut my grass today because it is going to rain tomorrow.
In this model, if I were going to use Metrics for my performance, I wouldn’t want to be measured on the number of rainy days, heat waves, or anything like that – those things are outside my control. Instead, I’d want to be measured on something I can influence – for example, the percentage or days on which I dressed appropriately for the weather or the percentage of times I cut my grass once a week during the summer months.
To apply that metaphor to information security, I still need to know the number and nature of vulnerabilities in my network, as that information will guide my use of countermeasures and impact my actions – much like a weather forecast – but I don’t want to be measured on the number of vulnerabilities. Instead, I’d want to be measured on things like the percentage of times I patched critical, patchable vulnerabilities within 72 hours; or how well I have hardened my configurations to mitigate the risk of known vulnerabilities.
Take a look at the things you measure. In particular, take a look at the things that become part of your bonus calculations or your performance reviews. Which of them just “happen to you,” and which of them are within your control enough that you can influence their outcomes?
If you are currently being measured against things that feel more like Tracking Indicators (like a weather forecast), then it’s time to renegotiate your Metrics.