Security at the Speed of DevOps

DevOps and traditional security seem to be at odds with one other. But it doesn’t have to be that way. You can make security a part of your DevOps process without sacrificing agility or security. First, let's define what DevOps is. Let's then look at how it combines with security to create DevSecOps.

DevOps: A Working Definition

So, what do we mean by “DevOps”? The name itself implies a combination of “development” and “operations,” but it involves a lot more than just sticking two departments together under one umbrella. It is a culture and process that has a lot in common with Agile, only even more extreme in some ways. Instead of a release schedule measured in months or weeks, a DevOps team may release a new version 10, 50, 100, or more times every single day, with the developers that write the code deploying their own code directly to production. This is made possible by automating every step of the release pipeline. DevOps teams rely on a variety of tools to help them deploy code faster, and in many cases, they write or extend these tools themselves. Continuous integration tools like Jenkins ensure that every code change results in a completely new product build. Various unit tests and acceptance tests can be run against the new build to verify that no regressions exist in the new build that would cause problems in production. Configuration management tools like Puppet and Chef allow you to define your server infrastructure as code, so that new servers can be provisioned in minutes instead of days. Performance statistics and the results of experimental A/B tests on users are used as feedback for the next round of improvements, and the cycle begins all over again.

Image

To a traditional IT operations team, the idea of developers deploying their code directly to production generally causes shock and horror. One of the slogans of the DevOps movement is to “move fast and break things.” Breaking things is exactly what most IT operations teams spend their time trying to prevent. So, why would a company move to a DevOps approach?

Benefits of DevOps

One of the most obvious benefits is that your features make it to market faster. Rather than waiting for a scheduled release six months in advance, a new feature can be deployed the day that development on it is finished. So with all else being equal, a company using a DevOps approach could always be six months ahead of a competitor that is not. After all, nothing makes a customer happier than having more features faster, and most developers are happier working with the latest tools and techniques anyway. According to the Puppet 2017 State of DevOps Report, the highest performing DevOps teams surveyed had a rate of failures caused by changes five times lower than the average. And when there was a failure resulting in downtime, the teams were 96 times faster at recovering from that failure. It seems counterintuitive, but moving fast and breaking things eventually leads to a lot less failure and downtime.

Coupling DevOps and Security

So, how does security fit into all this? You may have noticed that there isn’t a “security” stage in the DevOps process, and it isn’t one of the guiding principles or techniques usually mentioned in the context of DevOps. If you think that the idea of DevOps would give your IT operations team heartburn, try talking to your IT security team about it. (You might want to make sure they’re sitting down first and that there are no sharp objects nearby.) A DevOps workflow does not leave room for the extended planning and auditing that normally goes along with security operations. You cannot deploy 50 times a day if you have to wait for 50 meetings of a change advisory board to approve the change. Stopping for approval or auditing breaks the cycle, and it prevents the rapid improvement that leads to all of the positive effects of DevOps. In a 2017 DevSecOps Community survey, more than half of respondents either somewhat or strongly agreed that “security is an inhibitor to DevOps agility.” So, if we can’t have both security and DevOps agility, it seems like we have to give one of them up. We can either have our rapid DevOps improvement and give up security considerations, or we can refuse to compromise our security standards but miss out on all the benefits of DevOps.

Image

It really isn’t sustainable to just leave security out of the picture completely, but if security doesn’t allow for DevOps agility, then we’re wasting our time talking about it. There has been a movement to make security a more integral part of DevOps rather than being in conflict with it. People have described this idea as SecDevOps, DevSecOps and DevOpsSec. There’s also a “rugged DevOps” movement, which can’t really decide if it’s the same thing or not. For this blog, I’m using the extremely scientific method of picking the term that has the most Google hits, so I’ll continue to call it “DevSecOps.”

Image

The DevSecOps answer is that security fits in everywhere. Rather than make security a release gateway or something that only happens in later stages of a release, security is integrated into every part of the workflow. This is also referred to as “shifting security to the left.” That would make more sense if my pipeline diagram below was a straight line rather than a loop, but the idea is that security is getting pushed into earlier stages of a release pipeline, all the way back to planning.

Image

The DevOps way of automating everything is a great fit for security. Instead of automated tests that only cover functionality, they can also involve security tools to handle security testing before anything goes to production. Security can even be a part of deploying new infrastructure. Configuration tools like Puppet and Chef that allow you to define your infrastructure as code also allow you to define the secure configuration of that infrastructure as code, as well. A DevSecOps workflow for infrastructure could involve writing the code to define a new server, building a new virtual machine of that server, testing the virtual machine for vulnerabilities and configuration failures, updating the code to correct any failures, and repeating the process until you can eventually deploy an already-secure server to production with high confidence and continue to monitor it for any drift from its original security posture. In addition to using DevSecOps to build secure services and application, the DevOps process itself needs to be secured. A compromise of a git repository, Jenkins server, or Puppet master could make all of your other security precautions meaningless. These critical pieces of DevOps infrastructure need to be locked down and monitored as closely as your company’s Active Directory server. Consider the following: A configuration management tool like Puppet generally has full root access to anything using it. If someone were able to modify Puppet’s configuration scripts, they would be able to make arbitrary changes as root across your entire environment. They could look into changing a Puppet master, which has a code directory containing all of its manifests and modules. That directory needs to be monitored for any unauthorized changes. Chef has a similar directory called the bookshelf that is used to store all of the cookbooks for the server. It’s also important to avoid hardcoding keys and passwords into DevOps scripts. These scripts generally exist in more than one place, like on all of your developer workstations. Having the password or API key to a highly privileged account in plaintext and spread out across your environment is a big risk. To address this issue, Puppet does have a built-in tool called Hiera to separate out code from important configuration data. Using Hiera, scripts can be freely distributed and checked in to source control without any passwords, and the secrets themselves can exist only in a configuration file on the Puppet master. For even more protection, the hiera-eyaml add-on allows for encrypting values in configuration files, so they are only decrypted in memory when needed. If you use Chef, it has data bags and encrypted data bags. There’s also a secret management tool called Vault by HashiCorp that is more generic if you’re using something other than Puppet or Chef. You can further mitigate the risk of leaked credentials by making sure that the accounts have only the privileges they need. You usually do not need services to have administrator account access for a cloud environment or database server.

How Tripwire Can Help

If you are going to implement DevSecOps, then Tripwire would like to be part of your solution. An important part of DevSecOps tools is the ability to use product APIs to be able to automate your security assessments as part of your workflow. Many of Tripwire's products also have command-line tools which can make it easier to interact with the APIs instead of writing your own tools. You can also engage our professional services team to help if you don’t want to do all of the integrations yourself. Tripwire Enterprise can be used to monitor your critical DevOps infrastructure servers and evaluate them for compliance with hardened security standards. It can also monitor critical files like the scripts on Puppet and Chef servers. Tripwire Enterprise and Tripwire IP360 can be used to evaluate the security posture of systems before they go to production and continue to monitor them for deviations after. At the same time, Tripwire Log Center can be used to alert on suspicious security events, like a terminated employee logging in and modifying files. Another specific way that Tripwire integrates with DevSecOps is through Puppet and Chef. We have Puppet modules and Chef cookbooks for deploying and managing Tripwire Enterprise agents. These can allow you to install, configure, or upgrade agents automatically across your entire infrastructure at once. They can manage both the legacy Java-based Tripwire Enterprise agent and the next generation Axon agent; they can even help migrate your environment from one kind of agent to another. They are available now on the Puppet Forge and Chef Supermarket, and the source code is also available on GitHub under an open-source license.

Recapping DevSecOps

There are a lot of advantages to taking a DevOps approach, from faster time to market to more flexibility and resiliency. By integrating security into every stage of DevOps from the beginning, it’s possible to get the advantages of DevOps without sacrificing your organization’s security. If you have a favorite DevOps tool that you would like to see Tripwire integrate with better, please let us know in the comments. If you’re already using DevOps tools, I hope this gave you some ideas of how Tripwire can work with your tools and process. Also, Tripwire is hosting a special webcast on August 21 titled, "Leading a DevOps Transformation". Join us and guest presenters to learn how to help your organization achieve higher levels of performance whilst ensuring security is a continuous aspect of the process. You can register here or click on the image below!

Image