Detection of change is easy… There, I said it. Anyone can do it. One thousand monkeys with keyboards can pound out scripts to detect change. What is not so easy, what the monkeys can’t do, is reconcile change. Even worse, it’s usually the monkeys who make the changes that bring everything crashing down around your knees.
It’s the reconciliation of change that most organizations have the most trouble with. What was the change? When was it made? Who made it? Was it authorized?
Welcome to change management.
These are all questions that most folks struggle with. In the early days of tech, it was a Wild Wild West mentality where speed was of the essence. If something needed to be done, the admins would just reach their little monkey hands into the server and do what needed to be done… To hell with the consequences.
When things worked out, the admin was the hero. When they didn’t, the admin would quietly hide behind the anonymity of being one of a thousand monkeys. Sometimes, if something broke, the hero would emerge from the smoking carcass of the server with the fix in hand. However, one of my old CTOs had a saying:
“The person who saved the ship is usually the person who started sinking it in the first place.”
Now, in many organizations, some processes exist for change management – not because they want to but because they have to. Some groups/governments/industries say they need it. PCI DSS, SOX and NERC/CIP are all examples of producing standards that tell you that you need to be able to detect change. Hell, I made a career selling software that allowed them to check that little box.
But what I have seen over the course of that career is the following: detecting change is easy, but reconciling change is hard.
DevOps and the containerization of apps is bringing things back full-circle to the days of the wild west. Developers can now make and push changes on the fly in production again. In fact, it’s highly encouraged. The dream of making servers and apps like fuses has been achieved.
The potential for misconfigured or vulnerable apps to be published has increased exponentially, with much of it being automated using DevOps tools.
With that being said, we have finally arrived at the reason why we are here: the need for an effective change management program. A process through which changes are made in a safe manner with the ability to back out of them when things go wrong.
Well, what does it take to develop and grow this mythical beast? Unfortunately, the answer for my ADHD friends is not a short one. I will labor to provide you a Cliff Notes version, but I would recommend picking up a copy of The Visible Ops Hand Book to further your education on this critical matter.
In any case, here are some of the key characteristics of organizations who have developed an effective change management system:
1. Tone at the top
There has to be management level or higher buy-in to the idea that unauthorized changes are bad. This buy-in cannot be just from the IT managers; it must also come from the business owners. Additionally, there need to be consequences when the process is not followed.
2. Electrified fence
Going back to the Wild West Days, if something broke, the first response was to start hammering the keyboard until something happened, good or bad. But things have changed nowadays. In successful organizations, the first step is to understand what the issue is and to determine what happened and what services are affected before turning the monkeys loose.
3. Identified fragile artifacts
Risk classification is yet another topic that can be explored in depth, but the gist is this: you need to know what you have and which assets are critical to the organization. It is surprising to know how many large companies have a tenuous grasp on either of these.
4. Have a process
It doesn’t matter if it’s built on a spreadsheet or some fancy cloud-based application – you need to have some process that basically says, “Before anyone touches that server over there, these are the steps to be followed.”
The process itself does not need to be complicated. These controls should not be so onerous that the monkeys complain and try to figure out how to get around them. Interestingly enough, the implementation of effective controls allows for more changes to occur.
Another fun little saying from my old CTO:
“Do you know what the brakes on a car are designed to do? They are not there to stop the car but to allow it to go faster safely.”
So, what might your change management system have in place?
- An easily understood series of steps. A recipe. A playbook. Call it what you want, but it needs to be repeatable and not overly complicated. Make sure to build in an emergency break/fix plan, as well.
- An effective system of record that is accessible. Something automated is preferable, but again, a spreadsheet on a SharePoint server is better than nothing at all. It also certainly beats shouting across the cubicles: “HEY, I’M ABOUT TO MAKE SOME CHANGES TO THAT SERVER!”
- Identify who the stakeholders are. There should be a mix of technical and non-technical folks involved who have a vested interest in the process. Segregation of duties will be a key component here. The folks making the changes should not be the ones approving them.
- Reporting is a huge issue. Not just deep detailed byte-level changes, mind you. No business unit owner will understand that. This is why service-level reporting is also critical. If three servers and a database make up a critical application, have reporting that allows you to drill down to the individual servers but present higher level reports around the service itself to which the business owner can relate.
- Methods to detect change and map them back to the process. Auditors love automation. Anything that takes the monkeys out of the equation lowers the chances for mistakes. This is usually why scripts and/or relying on logging to detect changes does not scale beyond a few servers very well. Purpose-built applications that integrate directly with your change process software reduces the opportunity for monkey-based errors.
- Understand that authorized change is not necessarily good change. Just because the monkey has an authorized change request to install and enable telnet on a server doesn’t make it a good change. You might as well plug the server into the wall at the local Starbucks with the password taped to the monitor.
These are just a few of the basic things to put into place to begin the journey towards an effective change management program.
There are certainly more concepts to be discussed along the way. Just wait until security gets involved. That will be a whole different book.