This week I attended the Gartner Security & Risk Management Summit in Washington, D.C. I attended a lot of very good sessions, but the one that left the biggest mark on me was a session called “Metrics That Matter,” delivered by Jeffrey Wheatman.
I went to this session because I’ve had a lot of conversations with information security executives this year, and a common question is “What should I really be measuring?,” or they make comments like “I report on a lot of things, but I am not sure what the top security indicators are that I should roll up to my executive team.”
Wheatman shared a really good list of “Five characteristics of effective metrics,” and I think it is a good litmus test for our metrics (security or otherwise). I’ll paraphrase some of my session notes so you can get a feel for this:
- Effective metrics must support the business’s goals, and the connection to those goals should be clear.
- Effective metrics must be controllable. (In other words, don’t report on the number of vulnerabilities in your environment, since you can’t control that. Instead, report on the % of “Critical” systems patched within 72 hours, which you can control)
- Effective metrics must be quantitative.
- Effective metrics must be easy to collect and analyze. (Wheatman says “If it takes 3 weeks to gather data that you report on monthly, you should find an easier metric to track.”)
- Effective metrics are subject to trending. (Tracking progress and setting targets is vital to get people to pay attention)
This set of guidelines really resonated with me, and I am going to run my metrics through this regimen to make my own metrics better. If you’re a Gartner client, there is a detailed research report from Wheatman on this topic, and I suggest you grab a copy.
The other thing I’ve noticed is that there seems to be a gap out here in the real world in terms of effective security metrics that are “universal” and also meet these criteria. So, I’m on a quest to find and/or establish some good ones that transcend company boundaries.
If you’ve been reading my posts here you know I’d like your help. If you have either a) good metrics that are working; b) vexing metrics problems you’d like to collaborate on; I would love to hear from you. Drop me a line at “dm at tripwire.com” and let me know what’s on your mind.