Root Cause Analysis for Deployment Failures

Image

Root Cause Analysis (RCA) is a technique used to identify the underlying reasons for a problem, with the aim of trying to prevent it from recurring in the future. It is often used in change management processes to help identify the source of any issues that arise following any modifications to a system or process.

RCA is something Tripwire Enterprise is often deeply tied into too. Before we talk too much about the tooling though, it’s worth setting the scene for RCA, and exploring some common processes involved in the analysis and where it “kicks in” when something goes wrong.

What is RCA?

When a change is implemented and issues arise, the first step is to contain and resolve the immediate problem. This is typically tied to familiar infrastructure support processes, often categorized as “incidents”, to describe the failure and remediation workflows required to address any immediate faults as much as possible. Once the issue has been resolved, a root cause analysis can be conducted to identify the underlying cause or specifics that led to the incident. The root cause analysis process typically involves the following steps:

1. Identify the problem – to clearly define the problem that needs to be investigated.
2. Collect data - including when and how it occurred, what systems and processes were affected, and any other relevant information.
3. Analyze the data - to identify the underlying cause of the problem.
4. Identify the root cause of the problem - which may be a process issue, a software or hardware problem, or a human error.
5. Develop and implement corrective actions - Once the root cause has been identified, develop and implement corrective actions to prevent the problem from recurring.
6. Monitor the system - to ensure that the corrective actions are effective and that the problem does not recur.

By conducting root cause analysis as part of the change management process, organizations can identify the source of issues, take corrective action, and prevent similar issues from occurring in the future.

Tripwire Enterprise and Root Cause Analysis

With that in mind, many of you with experience with Tripwire Enterprise’s (TE) File Integrity Monitoring (FIM) functionality can already see how TE may be valuable tool in the root cause analysis process. TE provides a detailed record of changes made to files and systems, which can help identify the root cause of a problem or issue.

In the context of change management processes, FIM can help identify when unauthorized changes have been made to a system or file, and can provide information on what specific changes were made and by whom. This information can be critical, as it allows investigators to isolate the change that caused the problem. A “snapshot in time”, both before and after the incident, can massively help with understanding potential causes.

On top of this, by allowing for comparison not just between points in time, but between systems in your infrastructure, it’s possible to identify discrepancies in configurations that might otherwise be difficult to assess. For example, whilst it’s easy to assume that a patch deployment was completed successfully and consistently, FIM allows you to audit whether that was indeed the case.

Furthermore, TE’s FIM and Security Configuration Management (SCM) can work in tandem to help  identify other potential issues, such as configuration drift or unauthorized access that may also have contributed to the root cause of a problem or outage. This complete record of changes made to files and systems enables investigators to more easily identify potential causes and trace the history of any issues that arose. Overall, FIM can enable organizations to take corrective action to prevent similar issues from occurring in the future, and improve their change management processes.

RCA may seem like a luxury to some organisations who find themselves constantly “fighting fires”, but as we’ve long known it’s in prevention that we can often be the most successful at tackling issues, and having the right tools to support your analysis can ensure you save time and effort as well as prevent future outages. That is a very worthwhile investment for any business.