Skip to content ↓ | Skip to navigation ↓

On Tuesday, August 7, I attended Metricon 7.0 in Seattle and participated in a three-member panel titled, “Rules of the road for useful security metrics.”   The questions posed to the panel were: How do you define security metrics?  What, in your opinion, is the purpose of having security metrics?  Can we hope to develop standard ways for measuring security?  And, what differentiates useful from useless security metrics?  Joining me on the panel were Anoop Singhal (NIST) and Jason Leuenberger (E&Y; @securitah).   The entire day revolved around the notions of defining security metrics, understanding the purpose of security metrics, and what makes such metrics useful, and this panel was no exception.

How do you define security metrics?

We all seemed to agree that security metrics are objective measures of the effectiveness of your security controls and/or processes.  Surprisingly, no one brought up the quantitative vs. qualitative debate at this point, though we did discuss it throughout the day.

What is the purpose of having security metrics?

For this question, Anoop had a really good point to make.  Yes, we all know the old addage “you can’t manage what you don’t measure,” but we must realize also that what you measure must answer the questions you ask.

Can we hope to develop standard ways for measuring security?

Here the panel seemed to diverge somewhat, but for good reason.  Some of us would look at things from a low level, while others of us would look at things from a higher level.  The lower the level your perspective, the more likely it is that you think standard ways for measuring security can be found and vice versa.

What differentiates useful from useless security metrics?

We all seemed to agree that your metrics have to measure what matters, and that what matters should help you be proactive rather than reactive.  Put another way, a useful metric tells you how you are doing relative to what you were doing before.

Overall, the panel was more or less a discussion between three people at the front of the room, the moderator (Anton Chuvakin, @anton_chuvakin) and the audience of about 30 participants.  This is really how the entire day went, and that’s what I liked most about Metricon – the interaction and diverse points of view.

Two presentations were particularly interesting to me.  The first was by David Severski (@dseverski), Information Security Manager at Seattle Children’s Hospital, and the second was by Angela Gunn (@agunn) and Jon Espenschied of Microsoft.

David’s presentation demonstrated some metrics he put together in a dashboard he shares with his boss (the CISO).  David claims that his CISO loves what he has done so far, and he’s done it all himself.  He started small, and that was really the point of his presentation.  Start small and spend time on the problem statement, because this is where most people fail.  They jump right in to metrics without considering what questions they really want to answer.  The Vulnerability Management Dashboard he creates for the CISO is based on data from several sources: Network configurations, logical topology, the National Vulnerability Database, and scan information.  The tools he used?  RedSeal, Tableau, and PowerShell.  The dashboard itself contains:

  • Risk Change: previously open, currently open, changed state
  • The most common vulnerable applications
  • Vulnerabilities exceeding SLA (SLA = 45 days for critical vulnerabilities)
  • Days open with criticality
That’s the dashboard, but for vulnerabilities the metric they use is a single, two-color bar representing all unique hosts.  One color represents those which are compliant with SLA and the other represents those which are not.  A simple graphic telling a simple story relevant to the operation of their business, which, as David pointing out more than once (indicating that he understands the business mission), is to heal sick kids.  At this point, he hasn’t any trend data, but that’s next on his list, and I’m sure he’ll do well.  The downside?  It takes several hours each month to get the data, which is presented on a trimester basis.

The Microsoft folks are doing very interesting work around advanced threat identification and detection based on the notion of  “threat sequences.”   They have been successful in identifying and characterizing things like Grafix Softech, Buckshot Yankee, Stuxnet, and others – and I think most of us realized that they have a massive dataset to analyze.  They’ve taken to two sets of vocabulary that are, in my opinion, perfect to describe this space.

The first is genomics.  If I have this right, the premise of threat sequences is essentially that we can, by looking at the characteristics of an attack (the lifecycle they use is described below) construct a genome for the attack.  Once that genome is understood, we can then study the evidence for a given attack sequence to validate, which is a phoneme.  The way they described this relationship is that a given genome might indicate a person has brown hair, and the phenome is the brown hair (it’s that observation substantiating the hypothesis made by the genome).

The second vocabulary they use stems from a military attack lifecycle:: Recon, commence, entry, foothold, lateral movement, acquire control, acquire target, implement/execute, conceal and maintain, withdraw.  It’s safe to infer from this lifecycle that they’re interested in advanced attacks – the APT and Nation State level attacks.    Using this approach they claim to have been able to make some interesting inferences:

  • Persistence after successful entry correlates with different targeting.  If the attack moves through the lifecycle quickly, it’s targeted and the attackers knew what they were doing.
  • Fast, successful attacks are usually preceded by extensive recon.
  • Operational sophistication does not correlate with applied technical sophistication.
  • Historical similarities between attacks are useful for pattern matching.

And, that’s really what they’re after – patterns of attacks.  Only, they’re doing it in a much more advanced way.  And, they’re quite observant.  Many in the press and blogs were really impressed by the fact that Stuxnet took advantage of zero-day vulnerabilities.  What impressed these two presenters more, and I agree with them emphatically, is that they took advantage of 64 out of 64 vulnerabilities present in the targeted systems. That’s simply impressive.  I’ve included some photos I took of their graphs at the end of this post – hopefully they’re good enough.


So, getting back to the point of the overall day, which is security metrics.  They come in all flavors, shapes and sizes, but there are some commonalities to them. There were definitely some key takeaways we reviewed at the end of the day (led by Anton):

  • Culture matters: This concerns corporate culture where a culture that is evidence-based will be more successful more quickly than a culture that is not. If your’e not in an evidence-based culture, your road will be longer, but not impossible.  Know your stakeholders well, and really understand how you can manage them (and each may need to be managed differently).
  • Goals matter: This is really something that can be called something like “understanding the business.”  Too often security metrics are invented that don’t really meet the goals of the business or the stakeholders, and they fail.
  • Measure what you’re told: A consequence to understanding the goals of the measurement, you need to ensure that your metric is really measuring that which you intend, but you should ensure that what you’re being told is correct!  If what you’re told is wrong, then you end up with a garbage in, garbage out situation, which is unhelpful.
  • Adopt GQM: Identify your goal, form the right question(s), then define your metric.  This was created in a different industry (software quality, I believe), but it’s very applicable in this space as well.  Those participants having experience defining security metrics in this manner appeared to be successful.
  • Accountability and “metricophobia”: Metrics should hold some one or group accountable, but this gives rise to a potential fear of metrics for obvious reasons.  After all, measuring something means accountability in most cases, and some people would rather not be accountable, right?
  • Available data drives metrics: Start with the data you have available and grow from there.  But, don’t fall into the trap where you believe the only questions you can answer are those supported by your data!  Consider what you have as a starting point, and you should grow your questions as needed (see Goals Matter above), and harvest more data as appropriate.
  • A-GQM: A corollary to GQM, this adds “audience,” and is aligned with understanding, and managing, your stakeholders.
  • Other takeaways:
    • Process clarity – understand the process to get the right metrics (Lance Hayden was in attendance and he was emphatic on this point – it’s not just about the metric, but how you get there.  I’ll be reading Lance’s book in no small part due to David’s recommendation, but also because I found him to be on target, knowledgable, and practical.
    • Good is better than perfect – If you strive to be perfect, you’re going to fail.  Get something good out there, refine it until it’s good enough, then move on.
    • Action, status, and deadline measures are more useful than environment metrics.  Quantities of vulnerabilities are not as important as your patch rate or how many are left unpatched contrary to SLA, for example.  This belies the overarching problem when considering the portability of security metrics.  Measuring actions, status, and deadline measures are different from one organization to the next.  I suggested that we should consider a set of key characteristics – an abstraction of sorts – for security metrics once we get enough of them to help this cause.
    • Cause and effect in security is fuzzy, which leads to inconsistent metrics between organizations.

Most of the attendees subscribe to the mailing list, which I recommend doing if you’re at all associated with the creation, collection, maintenance, and/or consumption of security metrics.

Featured Image Courtesy of Johan G under the Creative Commons Attribution 2.0 license.