Best Practices Resources

SC Magazine: Last Word Column

Gene Kim, CTO, Tripwire, Inc.

Title

Tug Of War: Information Security vs. IT Operations

Article

An often puzzling conflict in information technology management today is the continual tug of war between security and IT operations for control of the corporate information security roadmap and its execution. Over the past ten years, the most successful and secure IT operations I've seen are those where information security and operations are partners, working together to solve common business objectives. By studying these organizations, it's clear that what is good for security is good for operations, and vice versa.

I recently had the honor of co-chairing the SANS Auditable Security Controls That Work workshop, where best in class organizations discussed what management models really work. This extended research I've been doing with the Software Engineering Institute, IT Process Institute, and for me, further confirmed a significant key finding: organizations that had the most repeatable operational processes with the earliest integration of security into the IT lifecycle not only achieved the best service levels and security, but maintained some of the lowest cost structures around.

In other words, the organizations that were most efficient were also the most secure! These best in class organizations had the lowest mean time to repair, the highest mean time between failures, most predictable security incident remediation process, and also the highest number servers being managed by each system administrator.

It may not surprise long-time data center practitioners to hear that the underlying reasons have far less to do with technology or people, but instead have everything to do with process.

In the figure above, the best in class organizations are at the upper-right hand quadrant, with the best defensive postures and server/sysadmin ratios higher than 100:1. According to META Group and Gartner Group, the IT industry average server/sysadmin ratios around 25:1 to 35:1, which matches our own benchmarking.

What can we learn by comparing best in class organizations to the merely average? In the parlance of the Software Engineering Institute, the chasm between best in class and average organizations is referred to as a "knowledge gap." What are the best in class organizations doing that makes them so much better? What are the critical success factors for information security practitioners?

First, security and operations in best in class organizations work together to solve common objectives. Compare this with the often adversarial relationship commonly found between security and operations, where finger-pointing and turf-wars are the norm. Examples include situations where security crafts brilliant infosec policies, only to have them completely ignored by operations. Worse, in an effort to fix the problem, security may resort to having to make changes to operations infrastructure, only to have a failed patch deployment bring the infrastructure to its knees. The bottom line is that when security and operations work well together it's good for the business, both increasing service levels and decreasing costs.

Second, best in class organizations all share some common process characteristics: a prevailing culture of change management, rigorous configuration management practices, and a heavy reliance on release management. Release management provides a repeatable process to provision infrastructure in a known, good state, and provides an acceptance mechanism for security and operations to ensure that nothing gets deployed without all the stakeholders' approvals. Change management processes ensure that all post-deployment changes are reviewed, documented, and scheduled to minimize risk to the business. Rigorous configuration audits ensure that no infrastructure mysteriously develop "personalities" after deployment, which cause undesired and potentially dangerous variance.

Third, best in class organizations manage by fact, as opposed to merely managing by belief. They verify and measure whether the processes described above are all effective, and destroy any variances. Do all deployed infrastructure match a known, good build? Can all changes be mapped to a valid business reason? Is causality being used in the problem management processes? Is operations responsible for making all changes, with security working as coach and consultant? When these practices are being followed, the amount of firefighting decreases, the functional roles of operations and security become more clearly defined, and the entire business becomes more competitive.

From our research, it appears that when these things are being done, each stakeholder does what they do best. Operations keeps the factory floor humming. Security acts as coach and consultant to keep risks and controls in balance. And Management makes the final call on whether that balance serves the business needs.

How can security help operations create repeatable and verifiable processes? ITIL (IT Infrastructure Library) best practices are a rapidly growing movement in IT operations, and provides an ideal ecosystem for security and operations to collaborate as partners. The reason? It stresses the value of change and configuration management, release management, and problem resolution processes - the three key process areas that most impact security. When these repeatable processes exist, security practitioners can easily instrument controls, to help manage by fact, instead of merely by belief.

How long will the current tug-of-war between operations and security last? Who knows - but now we know the best business outcome is for both operations and security to win.

Back to Resources page