Image
What is GitOps?
GitOps is a new development paradigm that promises to achieve simple, continuous deployment for cloud-native applications. Git is described as:“an open source version control system—to infrastructure configuration. Git is used to version and store the necessary infrastructure configuration files,often in a Git repository, such as GitHub, GitLab, or a private Git repository.”
This new model provides a developer-centric experience, allowing development teams to operate and deploy infrastructure using tools they are already familiar with, such as continuous deployment tools, and with no assistance from IT. The core idea of GitOps is to have a Git repository that always contains a declarative configuration which represents a desired state of the production environment. An automated process monitors the Git repository and matches the production environment to this desired state. To deploy a new application or update an existing one, developers simply update the repository, and automated processes take care of everything. GitOps tools and work processes are growing in popularity. The GitOps model shows great promise because it can improve software reliability, reduce the danger of “configuration drift”, makes it easier to manage complex containerized environments, and improves auditability and security of production environments.What is Digital Forensics and DFIR?
Digital forensics is a branch of forensic science that focuses on recovery and investigation of evidence found on digital devices. As society increasingly relies on computer systems and cloud computing, digital forensics has become an important aspect of law enforcement, and a critical operational process for cybersecurity. Digital Forensics and Incident Response (DFIR) is a specialized cybersecurity profession traditionally associated with a Computer Security Incident Response Team (CSIRT). DFIR is a set of practices and tools that enable rapid investigation and response to cyber attacks. DFIR relies on artifacts found in file systems, operating systems, information system hardware, and other sources of evidence to reconstruct a crime. Open source digital forensics tools make it possible to perform remote forensic triage to identify the root cause of an incident. Complete DFIR platforms are also available, which can manage the entire process of incident detection, investigation, and response from end to end.How GitOps Affects DevOps Security and Incident Response
DevOps teams have long been aware of security and are tightening cooperation with security experts in their organizations. However, GitOps can take this cooperation to the next level, creating an integrated security process that is “baked into” development pipelines.Shifting Security Left
With GitOps, everything, including security, can be handled in code. All configuration and security policies are declarative, keeping everything in a version control system. This allows all changes to be entered into an automated pipeline to validate, deploy, and monitor them. GitOps is not simply an efficient way to manage infrastructure. It also provides a strategy for shifting security to the left, capturing required changes to the desired state – such as security issues, bugs, and vulnerabilities – earlier in the development cycle. GitOps makes it easy to fix security-related bugs and instantly redeploy affected environments or applications.Faster Incident Response
GitOps speeds up pipeline changes. If your pipeline is compromised, GitOps enables a quick response to fix security issues. You can quickly respond to incidents or vulnerabilities by rolling back to a previous security configuration, or adding a patch or fix to instantly deploy a new version. Storing an infrastructure as code helps to quickly identify specific lines of code in a repository that are related to a vulnerability. It helps investigators to quickly assess the scale of an incident, recover faster, and mitigate the damage caused by cyberattacks.Best Practices for DFIR in a GitOps Environment
In order to form the nexus between DFIR and GitOps, a few adjustments need to be undertaken.Integrate Alerting and DFIR Tools
Security teams need direct access to forensic data they can use to prioritize alerts and triage incidents. Therefore, DFIR platforms, security monitoring, and alerting tools should be integrated with incident management tools. This allows security alerts to go directly to the existing tools and workflows used by your team, without switching tools to see what's going on. Create an audit trail that captures responses to each alert. This provides visibility and accountability and helps improve response processes. Any action taken by a security tool should be visible in the associated collaboration tool. This allows you to see who or what system took action when and where.Provide Actionable Feedback
When security issues are discovered, blocking a build or deployment without clear guidance can cause friction between teams. DevOps teams are trying to push updates as quickly as possible and will view this as a nuisance. Therefore, it is important to keep the developer in mind when including and implementing guardrails. Ensure that developers understand the criteria for blocking a build or deployment. Ensure that the system does not only block a build but also provides meaningful context for developers, for example, what security issue exists, how it affects the code, and what is needed to remediate it.Prevent Cloud Drift
Any changes to the cloud environment should be included in Infrastructure as Code (IaC) templates stored in the central Git repository, which is the source of truth for the GitOps environment. These templates should be synchronized with the cloud environment through reconciliation loops and the use of immutable compute instances. In a pure GitOps model, a controller component constantly checks the desired state in the IaC template against the actual state of the production resource, and redeploys resources if any deviation is detected. However, in real life this is often not the case. Compute resources are not always immutable, and it is not always possible to tear down and replace them at will. In many cases, maintenance work will cause drift through manual configuration changes and alteration of resources. Be aware of cloud drift and make sure there is full visibility and monitoring to assure accountability. Once drift is identified, undo these changes as soon as possible or add the changes to the IaC template and push it through the pipeline. Keep in mind that IaC templates pose one of the biggest security risks to DevOps environments and any change could potentially be malicious.Track Progress Over Time
To track the progress of a GitOps-based security implementation, catalogue the number of existing bad configurations identified and patched, as well as the number of newly introduced bad configurations. Both numbers should decline over time. Another valuable metric is the time it takes to patch identified bad configurations. Initially, this could be hours or days, and as the GitOps setup matures, it can reduce to minutes or seconds. Having a baseline of security metrics is also important because it can help teams spot abnormalities. For example, if there is a sudden jump in the number of misconfigurations, it should be immediately clear to teams that an anomaly has occurred. Developers can investigate the environment holistically, see if new or existing deployments are violating policies, and resolve the root cause of the issue.GitOps is a new way of working and requires a different way of looking at digital forensics, incident response, and other security disciplines. In the past, incident responders were very busy figuring out what was happening in the environment while piecing together data points. A new generation of technologies, such as Security Orchestration and Automation (SOAR) and eXtended Detection and Response (XDR), have helped to automate this process. But the underlying difficulty remained; the need to coordinate multiple moving parts and different teams to achieve a security outcome. With a correctly implemented GitOps model, there is a single source of truth that captures the entire environment. Vulnerabilities, insecure configurations, or malicious changes created to resources, can be instantly detected either by identifying a change in the central Git repository, or by detecting “drift” in the environment. In either case, it is easy to roll back to previous secure configuration or roll forward to deploy a new patch. This can literally happen in seconds. This is a completely new ballgame for DevSecOps organizations. It makes security issues immediately visible, even without complex tooling, and enables instant remediation. There is tremendous potential for improving the security of DevOps pipelines, and you can be among the first to seize it.Conclusion
Image
