In the previous post, I talked about a Twitter contest I was running to answer the following question, with a Visible Ops book as a prize going to the best answer:
“When is it acceptable to patch the QA environment ahead of the production environment?”
If you believe that the goal of QA is to test that application code will operate properly with the production databases and OSes, then the answer is… Drum roll, please…
When the QA systems are patched ahead of the production environments, several undesirable things will likely result:
- The code fails to function as designed in QA, because of differences in the OS, libraries, databases, etc. When this occurs, QA must figure out whether it is a code failure or an environment failure. Either way, QA is spending more time on non-productive work, most likely slipping the release schedule or forcing QA to skip tests to keep the schedule.
- This situation is worse than the first. Here, the code works in QA, and then the code is then deployed into production, which then fails spectacularly. Now the problem isn’t that the QA schedule is slipping. Now the problem is that a potentially mission-critical service is down, and we have a potential Sev 1 outage, requiring the best Ops, QA and Development people to figure out how to restore service.
In the second scenario, it appeared that QA did all the right things, but the deployment still failed.
My long-time collaborator Kevin Behr noted, “It sure does seem like the entire point of QA is inextricably linked to production though?”
So, we have two winners. The first winner is Jonathan Katz (Twitter handle @katzmandu), who wrote “[You should] have OS patches in-sync the release cycle; as you promote code you test the code + OS + app together.” Perfect!
The second winner is @sabletek, who had a long answer that I won’t quote exactly. But, what I believe he was stating very precisely is that you can patch QA early, if you are weighing the risk of patching production vs. delaying the patch to the next release and actively managing the QA vs. production variance risk. He notes, “I’m making some large inherent assumptions regarding not only QA but patch management as well.”
One honorable mention, because he pointed out a very good wording error in my question. Nicholas Weaver (@lynxbat) writes, “Wouldn’t the answer be “always”? I mean, you have to QA the patch…”
Crap, yes, that is also true. If you’re not testing application functionality, then yes, you should always deploy in QA before production. But, I think @katzmandu gives the more complete answer: in order to predict outcomes, all the components have to be tested together.
Okay, enough theory. To see what it feels like when this happens in a real life environment, see the next post!
Questions or comments? Feel free to send me a note on Twitter! I’m @RealGeneKim.