In early August, I will be leading a couple of sessions at the Community College Cyber Summit about cyber security fundamentals. I've also been spending time working with my amazing colleagues thinking about DevOps. Spending so much time going back and forth from "back to basics" and "the future of development" had me thinking that securing DevOps is really Back to the Future. This post takes a different angle on securing DevOps by looking at DevOps in exactly the same way we look at cyber risk in traditional environments. It turns out the same principles and practices apply and can be implemented without getting in the way of DevOps agility. After all, one of the main advantages of DevOps is rapid development and deployment; anything that gets in the way of that could be seen as an impediment. So let's get back to the future with those basics.
Risk Reduction in DevOps Practices
Risk is the likelihood that something bad will happen and result in a loss to the organization. There are a lot of things that can go wrong when developing software, some more likely than others, and each has more or less impact. Security controls reduce risk -- both the likelihood and impact of something going wrong -- but controls come at a price, and that price is often speed. It's easy to assume that a philosophy that values speed and rapid iteration will come into conflict with one that seeks to limit risk. Speed isn't the only tenet of DevOps, however. No developer wants to introduce bad code or take down their cloud environment. It's common to see plenty of risk reduction in DevOps practices; for instance, quality gates are a standard DevOps practice -- automated unit and integration tests are injected into the pipeline and serve as a stop point when there are failures. In the same way, security is simply another quality gate. Poorly executing code could impact the availability of a service as much as a bad actor exploiting a vulnerability. The end state is the same even though mitigation is implemented differently.
People, Process, Technology
Security is comprised of much more than just technology. It's tempting to spend time looking at the shiny tools and talking about clever exploits, but we can't neglect the two other legs of the security stool: people and process. With DevOps especially, people and process may be the most important legs! DevOps isn't simply about the tools; it's also about the collaboration, the planning, and the thought which go into building the delivery pipeline. It may seem ironic that people are so important when one of the benefits of DevOps is taking people OUT of the process, yet people are still the ones writing and committing the code, kicking off builds, (or writing the triggers to do so) and determining what is allowed to pass the quality gates. Access control, code reviews, knowledge sharing and communication are all important 'people' aspects of security and risk mitigation and an integral part of DevOps. When it comes to process, the security benefits to DevOps are easy to spot. I've already mentioned quality gates -- a critical process component to mitigating risk -- and other people-led processes like code reviews, access reviews, change management and responding to alerts all contribute to the security of a DevOps pipeline. The pipeline itself is a process. If well-designed, it can mitigate risk and prevent attacks from being successful by securing containers and applications before they are deployed. A good process can even allow for rapid recovery from accidental or malicious loss by building in detective controls that respond to systems which fall into an undesirable state. Planning for rollback or replacement when a potential loss event or suspicious activity is detected ensures that loss is minimized and that a system can quickly return to a known good state. The last leg of the stool, technology, enables the people and processes to function efficiently, and it allows for continuous, consistent security practices. Human code and security reviews are necessary because there is no replacement for the critical analysis people can perform. That doesn't mean we want to manually inspect code and containers for vulnerabilities before deployment, nor is it really feasible during runtime. Tools to do static code analysis, vulnerability assessment and log management and monitoring are all excellent ways to mitigate risk and should be a consideration for any DevOps pipeline. The DevOps tools themselves should fall under consideration for security. Who can access and deploy your source code, manage and execute your continuous integrations, administer your continuous integration and deployment tools and access your stored secrets must be considered when thinking about DevOps security.
CIA: Confidentiality, Integrity, and Availability
Another fundamental security concept that applies directly to DevOps is the CIA triad -- confidentiality, integrity and availability. Here are a few questions you may ask for each element of the triad.
Confidentiality -- Does your application or pipeline contain confidential information? How are you securing secrets, access keys, or personally identifiable information (PII)? Speaking of PII, what about GDPR and other privacy regulations to which you may have to comply? Integrity -- How are you ensuring file and processing integrity throughout the DevOps lifecycle? Are you sure critical files haven't been modified along the way? What about when the application, container or virtual machine is deployed and running? Have you analyzed the cloud environment itself to ensure the integrity of the configurations of the underlying components? Availability -- How are you ensuring your pipeline is stable and accessible so that you can realize continuous delivery? What could cause instability internally and disrupt your service delivery? What are the threats externally that could cause a denial of service or diminish access to your application? The CIA triad provides a great framework for threat modeling, and questions like the ones above can help inform and prioritize the security controls you implement in your CI/CD pipeline. A formal threat model identifies the assets in the system, the threats to those assets, the impact if there is any loss in confidentiality, integrity or availability and any mitigating controls in place. Applying this process to your DevOps pipeline will raise any potential areas of concern and guide action around mitigating risk to your organization.
Detect and Prevent
An ounce of prevention is worth a pound of security cure. Nobody wants a 3:00 AM phone call; doing the right things to protect your infrastructure, application and pipeline upfront is a much better option than having to implement your rapid response and recovery procedures. Preventing security breaches is as much a DevOps concern as a traditional IT one. In fact, DevOps is especially suited for implementing a zero-trust model that emphasizes preventing unauthorized access at every point in the process. (See also; Zero Trust Is a Key to DevOps Security, Simplify DevSecOps with a Zero Trust Approach.) When reviewing your threat model and considering the controls you want to implement, look for ways to thwart attacks before they start. If an attacker can't get a foothold, the attack can't progress. Build that prevention into every aspect you can to lock down your environment. As important as prevention is, however, it isn't enough. Relying solely on prevention is a recipe for cyber-security disaster. Even the best efforts at prevention can fail; new vulnerabilities can be discovered and exploited, something can be overlooked in your prevention strategy, or an insider threat could arise. When prevention fails, detection is necessary. Looking for anomalies, unexpected changes and unauthorized access is a must-have in the security toolbox. As with prevention, DevOps is uniquely positioned to take advantage of detective controls. In traditional IT environments, detection tends to be a runtime concern, with DevOps, detection, correction and prevention can be wrapped into the pipeline itself before anything hits the production environment. This doesn't mean detective controls can be left out of the runtime. That would be a huge mistake, but it means security can be built-in and more efficient than in more traditional IT environments. Another aspect to consider is that the tooling pipeline itself likely resembles traditional IT and the source control repositories, continuous integration and build servers and applications that manage secrets must all be secured. Detecting or preventing a breach in the runtime environment doesn't help if the breach already occurred upstream. One last point on this topic. I mentioned above that detection, correction and prevention can all be built into the DevOps pipeline pre-runtime. In my view, detecting vulnerabilities, insecure configurations, unauthorized libraries or applications or test data all fall under the umbrella of detection. Quality gates that prevent builds from moving up the pipeline if they fail any of those security tests is a great preventive control and forces a risk assessment at the very least. If the DevOps team has a policy of correcting the quality gaps before releasing the build, the corrective control functions as an excellent way of ensuring only the most secure builds are deployed.
Back to the Future with DevOps
Wrapping things up, DevOps may be the future of software development and IT operations, but that doesn't mean security fundamentals don't apply. It's easy to fall into the trap of thinking that DevOps is a completely different beast and that all those old ways of doing things don't apply. This trap is really a half-truth looking to lull you into complacency. It's very likely the old ways of how we do security may not work, but the what and why are still very much operational. Like DevOps itself, security just needs to be adapted to new tools and processes. The underlying fundamentals still apply; it's up to us to find the best ways to implement those fundamentals.
Mastering Security Configuration Management
Master Security Configuration Management with Tripwire's guide on best practices. This resource explores SCM's role in modern cybersecurity, reducing the attack surface, and achieving compliance with regulations. Gain practical insights for using SCM effectively in various environments.
