I was just reading an interesting Forbes article on IT belt-tightening, and they talk about a number of measures you can take to reduce operating costs.  It’s no surprise that they mention virtualization, and I wasn’t surprised to hear that inertia (i.e. doing nothing) is still a big obstacle.

“To be sure, virtualization significantly increases complexity. The 3,000-server reduction example took three years to implement and required highly skilled workers who weren’t easy to find, and failure to architect applications and storage correctly can lead to a dramatic reduction in availability. Given this risk and lack of time, many IT professionals cannot give virtualization the effort it requires to do well–so they end up doing nothing.”

I guess it’s no surprise that this phenomenon is playing out in IT - after all, how many things are on your “honey do” or “wanna do” list at home that never get done because it’s easier to put them off?

So, my question is this: what effect will the economic funk that’s happening now have on the virtualization market?  A couple of likely scenarios:

1. Organizations will use uncertainty as a reason to continue to put off infrastructure reduction projects, causing the virtualization market to flatten or decline.
2. Organizations will use the uncertainty to drive economy and centralization through infrastructure reduction projects, thereby feeding the virtualization market.

Which do you think will happen?  Or is there another option I’m missing?

These days, there’s all sorts of buzz about clouds, virtualized services, and the like.  For example, Amazon is making a big deal about their “Elastic Compute Cloud,”, Sun continues to talk about their Utility Computing model, and so forth.  You can add SaaS, Grid Computing, Virtual Datacenter providers, application outsourcing, and more to that list.

The truth is, we’ve had what’s being called “cloud computing” for quite a while.  Blogs, online document sharing, all of your free email accounts, the web-based maps you use, etc. all qualify as applications that run “in the cloud.”

Also, a growing number of companies are using online services to replace what were (or would have been in the olde days) in-house applications.  Examples include Salesforce.com, Siebel On Demand, Compli, and other such services.

So what’s with all the hype about cloud computing now?  Well, we’re in the midst of an industry cycle that mirrored the US election cycle we just completed:  hope is high, and everyone is looking for Change.

In the case of cloud computing, people are looking to reduce costs while creating a centralized (yet location-independent) way to access shared data.  The side benefit of this is they can often reduce datacenter space, power, and cooling (or at least they believe they can).

Sounds pretty good, eh?  Well - not so fast.  Remember:  Hope is not a strategy, and Trust is not a control.  Even though you can get that compelling Change you’re hankering for, you can’t outsource accountability.  If you’re Management, you still need to ensure the health of your business. That means understanding some basic aspects of any cloud-provided service.  For example:

• Is the cloud provider a viable, stable business?  What assurance and protection will you have as a customer?
• What’s the data provider’s risk & security model?  Don’t forget - even if “they” host your stuff, Management is still accountable for protecting company and customer data.
• What’s your fallback plan?  If something goes wrong, or you lose connectivity to your cloud-based service, what will you do?  Keep in mind you may need to deal with this on both a short- and long-term basis, depending on the nature of the interruption.
  • Note:  this should lead you, logically, to ask your provider what their fallback plans - and commitments - are.

The list could go on, but I hope you get the point.  Clouds are a lot like the Application Service Provider (ASP) model that was so prolific just prior to the dot-com crash around 2000, and a lot of those providers aren’t around.  The benefits are there for sure - just make certain you know what you’re getting into, and whether you can manage the risks.

Like so many bloggers I am here to talk about lurking issues with virtualization.  However, I’m not here to talk about the technical details of items like promiscuous mode on a vSwitch or setting proper permissions in Virtual Center (which by the way these are both important).  I’m here to talk about a more fundamental issue that plagues many IT shops–proper process around change management.

I talk with many customers today about their change management process and the vast majority still struggle to effectively manage change.  They all have a change management process but they have varying degrees of success with it.  Most customers enter change information into a change management system but for the most part the data entered is inconsistent.  If I ask a few probing questions like can you tell about all the changes that took place to a particular asset or how do you know a ticket was completed successfully they generally do not have this information. 

One of my favorite examples of a poor process was a very large customer who used an enterprise ticketing system yet could really not track anything of relevance with this system.  It was not the change management system’s issue because it had all the bells and whistles.  It was the customer who had put a terrible work flow around this system.  I sat with the customer to understand their existing procedure for submitting a ticket and was astounded to see that when it was time to fill in the asset name/group that was getting the change they used a drop down to select “Production.”  This is an organization that had 2000+ production servers so their filter was just to say production versus staging or QA.  Suffice it to say this left some room for improvement.

I’d like to say this was an anomaly but it really is not.  In many cases when I sit with a customer to understand their change process I see similar results.  They do not enter the asset’s name or instead put it in a giant memo field so this obviously makes it impossible to report upon.  They may or may not document what exactly is going to change or keep it very generic like “Updating the Firewall Rules.”  Well, if I need to roll back a change that definitely tells me what I need to know!

So why is this relevant to virtualization?  Obviously, this is a process framework that should not only govern the physical world but also the virtual world.   So, if an organization manages changes poorly today this problem will grow by a large factor.  It so much easier to make changes in the virtual world and these changes often go under the radar.  Since you can easily add a brand new machine or even add a new network with a few mouse clicks most people do this without the proper authorization.  This leads to undocumented changes but even when people do document the changes they put in the bare minimum that is required for the change system.

One of the greatest features of virtualization is that it runs at lighting speed and you can react to the business’ needs faster than ever possible.  However, in this case its greatest strength could become its greatest weakness if it paired together with an organization that does not have a well defined and mature change process.

Just catching up on my virtualization news feeds after a week of heavy travel, and I noticed that there are a lot of interesting things going on - mostly involving mixed messages:

1. A well-known research firm, IDC is apparently providing some numbers around virtualization adoption that aren’t sitting well with VMware.  They think IDC is giving Microsoft too much credit for gaining adoption / market share.  IDC says Microsoft’s Hyper-V accounted for 23 percent of new shipments.
2. VMware announced strong earnings for the 3rd quarter, and are forecasting heavy growth in 2009.  However, VMware is freezing hiring for the foreseeable future in spite of that expected growth.  The article also says they are doing some sort of “realignment” (aka “reorg”).

I think this will be very interesting as it unfolds.  Some things I think will make it more interesting:

• Many IT organizations are deploying (and will continue to deploy) hypervisors from multiple vendors (VMware, Microsoft, and Xen most commonly, but virtualization from Sun and IBM are already common in our customer base).  This will make the counting more difficult, and will make for some interesting posturing from the virtualization vendors.
• The economy is anybody’s guess, but with virtualization technology it’s trickier to figure out what to do.  Companies will be more frugal with their dollars, but the ROI on virtualization may make it less likely to get cut.
• To make it even more tricky, organizations seem to be deploying virtual infrastructure ahead of their ability to manage it effectively.  That may mean they need to spend money on management tools to attain the visibility, capability, and integration required to realize virtualization’s ROI.  So, how good are you at convincing management they need to spend money to save money?

Watching the vendor landscape will be interesting, as well.  Obviously, the big virtualization vendors have plenty of cash in the bank and will make it through the storm in some fashion.  However, the “upstarts & startups” in the field will have more of a challenge to deal with.  I fully expect many small vendors to fail or get bought up for pennies on the dollar.   It’s times like these when I’m glad I’m in an established, growing, and profitable company.

What do you think?  How (if at all) has your outlook, strategy, budget, etc. changed in recent weeks?

I’ve heard this twice in the last three weeks!  The IT group could return millions of dollars to the bottom line, if only they could win the argument with Finance to buy new servers and allocate the four man-months to complete the infrastructure replacement project.

• IT guy: “You know that virtualization project?  Well, we need to replace 120 servers which are about seven years old, and incidentally, no longer under maintenance.  And we’re four versions behind on the operating system.”
• Finance guy: “Are those servers paid for?”
• IT guy: “Yes.”
• Finance guy: “And they’re fully depreciated?”
• IT guy: “Yes.”
• Finance guy: “So, they’re free.  Request denied.  Keep running them until they fail!”

Of course, they already are failing (at random times), causing urgent firefighting, taking even more time away from other urgent infrastructure improvement projects.

And thus, we keep swirling down…  Even though we purchased the VMware licenses.

Does this resonate with anyone else out there?

Image courtesy: John angry/Oboyah Za3lan

With the announcement of Tripwire Enterprise 7.5V, like new parents, we are eager to show it off and smile lovingly at our new little baby. With this in mind I thought I’d spend a few posts showing some of the new functionality that makes TE 7.5V so great. 

My first post is to give some detail on our new role based Home Pages. This functionality allows customers to create their own widget based pages that give them the information that is important to them without having to see or understand the underlying technology. With this being a Virtualization security blog I’ve taken an example of what a VI admin or a member of the security team might want to have as a Home Page. Each widget is customizable to show the key pieces of information you are interested in. In the example to the right I have four widgets that I’ve created for a “Virtual Infrastructure” Home Page (click to enlarge).

The first widget “Virtual Infrastructure Security” contains scorecards for configuration assessment against security best practices for the Hypervisors, vSwitches and workloads contained in the monitored Virtual Center. These reports are clickable to get more information. If I click into the vSwitch VMware Hardening report I can see the different vSwitches I am monitoring and whether they are compliant to the best practices laid down by VMware or not.

In the vSwitch VMware Hardening report, we can see that vSwitch1 failing some of the tests that we’ve designated to run on that switch. These failures are due to both the MAC address changes and Forged Transmits not being set to reject. Drilling further allows us to see the actual settings and also how to remedy the issue if desired. This remediation advice is included in the policies to help customers not only find out where they are failing on security parameters but also how to fix it. This remediation advice can be edited to suit different environments or processes and could even be used to create a change ticket and be included as a run book of actions that need to take place by other admin staff.

 Moving to the right of the home page we have another dashboard widget called “VI PCI Compliance” which allows us to see how well the environment is configured against VISA’s PCI policies. Read this post to see why Mike Lohr so eloquently pointed out that the virtual infrastructure should be compliant to PCI. Any graphical report could be added to these dashboard widgets, for example guest OS complaince to CIS, NIST, DISA or ISO27001.

Directly below that widget we have an alert widget that is configured to show any alerts on the virtual infrastructure which includes new guests or hosts being added and changes to the infrastructure that reduces or increases the compliance to the security policies. This is a great alert for a VI admin to use to see who is adding new guests to the Hypervisors to control VMSprawl.

Below that alert widget we also have a “Report Repository” widget that allows us to add reports an admin might want front and center. A very useful report, that I’ll discuss in a future post, is the unmonitored guest systems and hypervisors which allows us to show any guest or hypervisor that have been deployed without security monitoring.

As you can hopefully see, this new piece of functionality we really help staff gain control of their virtualized environment. My next post will be about how Tripwire can automate discovery of all virtual assets.

In today’s tougher spending climate, I’ve found that virtualization is one of the few IT projects that don’t have to do with compliance that are being funded these days.  They often hinge upon reducing operating expenses and increasing computing densities, but I wonder how many of these projects will deliver on the goals of real reduction in operating costs…

Image courtesy: John angry/Oboyah Za3lan

I got a lot of input on my last post about moving live VM’s around, so I thought I’d do two things:

1. Clarify the point I was on about;
2. Give an example of what I mean by a technical feature that has - in itself - business value.

Business value: It’s about results, not just activity

Some people thought I was dissing VMware’s VMotion in my previous post.  I wasn’t - in fact, we use it in our lab environment and it is very cool and powerful.  Rather, the point of my post was that faster movement of VM’s, in itself, may not solve any high-level business problems for at some customers.  Therefore it may not even make the list of features that drive “the decision” when choosing a virtualization platform for the business.

Organizations should start with a top-down analysis of the business problem they wish to solve, detail the business requirements / expectations associated with the business problem, then implement processes and technologies that best support their requirements and measure the results vs. their expectations. 

The trouble with this approach is that organizations often shy away from dealing with the process changes required to really solve their problems, since that requires getting people to change how they work. 

Getting people to change can be hard and painful, so organizations often avoid it.  Capital expenditures are much easier to justify than organizational change initiatives, aren’t they?

What’s the consequence?  Organizations often turn to a hot feature in a technology, hoping that it will solve or mask their business problem.

Here’s a story about that:  A couple of years ago I worked with a large financal institution whose VP of Ops told me, “I can hide most of my unplanned downtime just by overinvesting in capacity and failover.” 

He still had availability problems - it was just harder for people to notice because he was masking the symptoms.

So be careful when you’re selecting technology, and don’t get distracted by features for features’ sake - make sure the feature really solves the business’s problem.

And, of course: if VMware’s VMotion and/or DRS solve or reduce your business problem, then that’s good thing.

Technology can tear down walls

Now, let me give you an example of a technical capability I learned more about at the recent VMworld 2008 coference that I believe does add business value, all on its own:  the Cisco Nexus 1000V distributed virtual switch.  What is that, and why do I think it delivers business value? 

This technology will provide a seamless integration of Cisco’s security, policy enforcement, automated provisioning and diagnostics features into dynamic VMware environments.  More importantly, it will allow an enterprise’s certified Cisco experts to manage virtual network infrastructure using the same commands, tools, concepts, etc. they already use for physical Cisco switches.

Where is the business value in this?  It provides a way to get Cisco network engineers engaged in the management of virtual networking without forcing them to get new training, use new tools, or significantly alter their workflow.  More buy-in with lower effort == better business value.

This kind of integration is a huge step in enabling faster, more secure scaling of virtual infrastructure because it eliminates a lot of the “hard and painful” aspects of this kind of business change.

Bravo, VMware and Cisco for making it easier for more people to get more business value out of your products, and capitalize on their past investment in skills development!

I guess I really am a blue-collar type of guy…  Certainly, there were many interesting technologies previewed at VMworld, including some mind-blowing features and capabilities that allow transparent failover and transactional integrity using pedestrian hardware and software. Pretty astounding stuff!

But, to be honest, my favorite talk was given not by the technologists and marketers, but by two folks from the IT Operations group from VMware. Tayloe Stansbury and Drew Kramer gave a fantastic talk called, “How VMWare Virtualizes Its Own IT Environment.”

Tayloe and Drew talked about delivering some of the most mission critical IT services for VMware, including the financial reporting systems and email, to VMware through periods of incredible growth, resulting in pressures to increase capacity and provide an infrastructure more resilient to failures.

I wish they had talked more about the change and configuration processes required to achieve their stabilization of service levels, which mostly likely had far more to do with their results than virtualization. Some of their interesting lessons learned included (Gene’s commentary in parentheses):

• The real lurking problem of virtualization sprawl (in the absence of good preventive and detection controls, beyond “hey, why is my ESX cluster running slowly”)
• The not-insignificant upfront investment required to really architect an IT service correctly for virtualization (they gave an example of their Microsoft Exchange environment: 1 quarter to archtect, 1quarter to implement)
• That the uptime risks aren’t from the technology, but from the daily operational tasks (only 1 ESX server crash in the last two years, but I’m sure they’ve had plenty of IT service interruptions not caused by ESX)
• Monitoring VMs for low usage, which make them candidates for decommisioning (apparently, like reclaiming bodies for food in the movie Silent Running.)

For those of you who want to study their slides, as I do, look out for presentation TA3808, which will hopefully be posted on the VMworld website soon.

Great job, Tayloe and Drew!  You make hard working IT operations folks everywhere proud!  :-)

Okay, guys, what was your favorite VMworld session?

A few days ago, I wrote about how Microsoft and VMware were reaching feature parity on the ability to move “live” VM’s from one host to another, and that the real test for virtualization was who could deliver the best value and return to the business.

Well, it turns out Microsoft’s Live Migration feature won’t be available unti 2010, so we have a while to wait.  My first reaction was, “well, that sucks for Microsoft.”  However, as I thought more about it, I don’t think it’s such a big deal after all.

Motion, motion, who’s got the motion?

From a feature perspective, VMware can move a VM from one host to another in a few milliseconds - if you satisfy a lot of very specific technical requirements for near-identical hardware. 

Microsoft’s current “Quick Migration” capability suspends a VM, moves it to a new host, and resumes the session about 6 seconds later.  The underlying hardware doesn’t have to be near-identical.

Is Microsoft’s Quick Migration fast enough?  Depends on the application.  In some use cases, 6 seconds is too slow, but I’d venture that 6 seconds is fast enough for most of the apps the typical IT department manages.  And, in any case, it is significantly better than the physical world. 

The bottom line:  I don’t doubt this (very) technical feature will be a deciding factor for certain availability-critical applications & systems, but I think Quick Migration will be sufficient for most circumstances.

Business value trumps technical coolness

So - that means we’re back to a battle over delivering strong business value.  Today, there are a lot of arguments floating around about whose virtualization platform is cheaper, better, etc.

The cost of the platform is not the primary factor in the equation, in my opinion - it’s the ease of incorporating the platform into your IT operations in a beneficial, effective way.  And when it comes to costs, management an operational costs will trump the cost of your virtualization platform in no time at all.

In this area, I (currently) give Microsoft the edge because they can (already) manage physical and virtual infrastructure, and have broader support for cross-silo IT operations and application management than any other virtualization vendor.

What do you think?  Is a “one stop” solution like Microsoft better for your business than what you get from a virtualization specialist like VMware?  Would love to hear your perspective.