Well, Metricon 8 came and went along with RSA. This con was a bit different – not even a conference, but a working session. Pete Lindstrom (@spiresec) did a fine job along with Andrew Jaquith (@arj) and Jennifer Bayuk, to bring this Metricon together and see it through. We saw a few presentations, but the meat of the day was working in breakout groups exploring scenarios in the following domains: Risk, Compliance, Incident, process, Vulnerability, Application Security, and Malware. I joined the Risk table, which was facilitated by Bob Rudis (@hrbrmstr). Others at the table included Jack Jones (@JonesFAIRiq), Jay Jacobs (@jayjacobs), Doug Foster, Chris Berry, Michael Makstman, Paula Hant, and David F. Severski (@dseverski).
While we didn’t come up with any Thou Shalt metrics, we did explore more than a few scenarios between all the groups – I’d say at least 12. The two scenarios the Risk group worked on, to give you an idea, included: Are we going to support platform X in our BYOD program, and should we outsource our Customer Support Representative business process?
Here’s what I can draw from the day (these are my conclusions, not necessarily those of my table mates):
- We have terminology problems in this subdomain as with others. I heard a lot of phrases such as, “label it what you want,” or “pick your phrase,” and so on. Without additional data, I’m inferring that we have specific concepts in mind, but we are unable to articulate them in a common manner. If we’re going to eventually come to some degree of measuring that can be automated in any way, we’re going to need to work on changing this.
- “The SIG is where they lie, er, wordsmith.” This is a very telling phrase. We want to minimize loss, which includes maximizing our process efficiency as we implement and assess our controls. The way many organizations start out assessing third parties is by leveraging Standard Information Gathering (SIG) techniques, which come from BITS. The problem is that both parties to the SIG are wasting resources if “trust but verify” is the mode of operation, and it usually is. It seems that a basic assessment of business process security properties is something that could benefit from automation, though not likely ever be fully automatic. The truth is that platform providers and tool vendors will ultimately need to provide appropriate interfaces for data collection, and the collected data needs to be available in a reasonable way to parties interested in understanding our security baseline and comparing it to their own.
- There seems to be a positive correlation between articulation of concise metrics and understanding of the process being measured. Vulnerability, Malware, Incident, and (though to a somewhat lesser degree) Application Security all seemed to come up with metrics that were concise, easily measured, and easily understood. I would argue that these domains all have well-known, understood process details on the ground. Perhaps a corollary to this is that the more abstract you are about what you’re trying to measure, the more difficult it is to define concise metrics.
All the notes and outcomes of the days sessions will be posted to the SecurityMetrics.org site in due time. If you have a chance, stop by there, check it out, and get engaged – especially if you’re someone who depends on security metrics for any part of your daily work.