In my post on Measuring and Reporting on Vulnerability Risk, I talked about how rankings and categories make for some easy to understand graphs, but they tend to fail at meaningfully measuring progress over time.
It’s tempting to use the standard output of your information security products as the basis for tracking progress, but counting the numbers of highs, mediums and lows simply isn’t an accurate a representation overall progress.
These kinds of operational metrics, such as vulnerability counts, are attractive as a means of measuring progress, or more importantly for communicating progress, because they’re intuitive; If we focus on patching vulnerabilities, then the vulnerability count should go down.
There are, however, a number of important considerations to take into account when measuring progress in vulnerability management specifically, or information security in general. Here are 5 tips for improving your ability to measure progress and communicate effectively in information security.
1. Use Metrics that Matter
In order to communicate progress, you need a destination. Very simply, ‘security’ is not a destination by itself. We’ve all heard that phrase that security is a process, not a destination. It’s true, but it’s misleading.
You’re not going to suddenly arrive at ‘security,’ no doubt, but that doesn’t mean information security shouldn’t have objectives and measure progress against them. Start with objectives, and build metrics to measure progress against them. If you’re not sure what your objectives are, then that’s a good indicator that you won’t achieve them.
An outside framework, like the SANS 20 Critical Security Controls, can help give structure to your efforts. In fact, SANS actually provides metrics you can use. Information Security has a tendency to be myopic about risk. When you talk about risk at the business level, risk and opportunity are a pair.
Too often, we pick metrics in information security that inherently foster fear, uncertainty and doubt. Measuring things like vulnerability counts, number of attacks, or viruses blocked are very effective at ensuring information security remains irrelevant to the business.
What are the metrics that matter to the business? That depends on *your* business, and perhaps more important, on the objectives of your information security organization.
2. Measure Performance, Not Activity
All metrics are not the same; there are different types, such as state, operational and performance. When aiming to track progress in information security, make sure you’re using performance metrics. State metrics measure some kind of fact, like anti-virus signature versions or scan progress.
Operational metrics measure an activity; this is where the suspect ‘counts’ tend to show up, such as SPAM blocked and vulnerabilities found. Performance metrics measure against a goal or objective. These types of metrics are what measure progress, and that’s what you want to communicate outside of information security.
3. Understand that Vendors Are Generalists
As a vendor, our objective is to solve problems by selling product. In order to do that effectively, we need to solve a problem that lots of people are willing to pay for. By definition, that means we’re generalists, rather than specialists, when it comes to your business.
When specificity is required, successful product vendors add flexibility, not features. That means you can’t blindly rely on the product to produce the results you require to measure performance. Vendors should be very good at operational and state metrics, but performance is specific to your objectives (see “Measure to Objectives”)
4. Plan for Scale
We all know that you can’t protect what you don’t know about. An effective measurement of performance includes some indication of the unknown unknowns. If you’re running a vulnerability management project, then you have to include comprehensiveness of coverage (are you scanning all your assets?).
If you’re auditing for policy compliance, you need to articulate what percentage of the relevant assets are being measured. If you leave this aspect of program performance out, you are simply providing misleading results.
5. Get Roll Ups Right
Individual key performance indicators aren’t enough, or more accurately, they’re too much. You may have KPIs to measure many aspects of a project, and many projects running in parallel, but you get one or two slides at best to represent performance to non-security executives.
Inside information security you don’t have the time to review every KPI for every group. You need a series of tiered roll-ups that represent abstracted performance to objectives, with the ability to drill into problem areas and get at the underlying KPIs.
On the surface, this may seem straightforward, but it gets tricky when you start trying to combine different types of metrics into a single roll up. Consider how you can normalize the metrics into a roll-up by measuring distance from a target.
It can be challenging to change any organization, but when you start measuring performance to objectives , you start to behave more like the business, which ultimately fosters more effective communication within and outside of Information Security.
- Brian Martin on Why Vulnerability Statistics Suck
- Vulnerabilities: It’s Time to Review Your ReviewBoard
- What is Vulnerability Management Anyway?
- Your Enterprise Vulnerability Management Reality Check
P.S. Have you met John Powers, supernatural CISO?
Title image courtesy of ShutterStock