One of the MANY challenges we faced while trying to implement a brand-spanking-new virtualization infrastructure was to put together some type of guidance for our server admin teams to help them determine if existing systems would “live” happily in our new environment.

After reading recommendations from VMware, digging through Microsoft Management Packs for Operations Manager, etc., it didn’t seem that any of those really fit our needs nor did they make much sense.

So we happen to use Operations Manager to collect performance data on our Windows-based servers, but there is a TON of data.

We also had to make sure that the tools we were using (Operations Manager) didn’t aggregate our data.  We NEEDED to have granular data for our entire 3 month look-back period.  We just didn’t like the idea of averaging averages, so we have data points every 5 or 15 minutes (depending on the counter) for 90 days.  One of the early reports we developed let admins specify the peak usage times so that the performance analysis could be tailored for specific systems.

Now we have all of this data but how do we look at it?  How do we know what’s important?  What recommendations from VMware and MS do we use and which ones do we throw away?

Several of the deficiencies we saw with most tools are:

1. They average entire periods of time (a day, a week, a month) with no regard for peak usage times
2. They use aggregated data (kind of like above)
3. They assume that “CPU %” means the same thing for all systems

So now we have to look only at peak times for specific servers.  Wow..that can be a VERY daunting task unless you make SOME assumptions.  That’s exactly what we did.

We know that not ALL systems will have the same usage patterns, but generally our systems will be the busiest during normal business hours at our corporate data center.  So, that would be Monday – Friday, 8am-5pm CST.  Well, we also have people using our systems on the east and west coasts, so if we move our “peak” time to 7am-7pm CST, we should cover most of the usage we wish to capture.

You  may be asking why not just take all data points and be done with it… I know *WE* did.

It would certainly have been easier, but far less accurate.  To demonstrate this, think about this:

• A system has a CPU utilization of 75% all the time during peak usage times
• This same system has a CPU utilization of 10% during off-peak times
• This system has a CPU utilization of 10% during the ENTIRE weekend

Okay, so if we take the mean average (the normal practice) we get something in the range of…well.. I am not gonna do the math, but it is WAY less than 75%.  We THEN thought, well… why not just take the highest or maximum measurement for the entire period and plan for that?  That doesn’t work out so well either if you have systems that don’t do ANYTHING, then a virus scan kicks off and pegs the CPU for 3o minutes.  This greatly throws our performance footprint off.  So now we have to find a happy sweet-spot for evaluating performance.

After mulling through all of our options and thinking about what types of data were REALLY helpful in this process, we decided to do a “Top 20% Average”.  I know it sounds a little strange, but it is pretty straight-forward. How did we come up with this number?  Well, we had to pick a number and threw some others around and this one seems to work, so we stuck with it.  The calculation is pretty simple:

1. Take all of your data points
2. Sort them High to Low
3. Take the top 20% of your values and average them

This gives us a nice look at what our systems do when they are operating at or near their highest loads.  The benefit of having a Top 20% Average and its normal average is that the greater the difference between the two, the “more spiky” the performance is.  As the numbers get closer and closer we can see that our systems are more consistently busy during our peak usage time.

The two most important things we need to be looking at are CPU and Memory utilization since those are often the most limiting resources in our environment.  Of less importance to us were Disk IO and Network IO because we were really looking at “low hanging fruit” types of systems and NORMALLY if a system is disk or network heavy, CPU utilization will also shoot up.

So now we have this notion of Top 20% Average (T20 Average) and Average, but what are we actually analyzing?

Not So Normal CPU Usage

Well, we can’t just look at CPU Utilization Percent!  Why, you may ask? Well CLEARLY the following are not equivalent:

• 1 CPU – 700MHz system running 75% utilization
• 4 CPU – 2.2 GHz system running 20% utilization

Many tools used to evaluate this type of data look at Total CPU Percent Utilization.  This would mean our 1 CPU system has a larger processor footprint than our 4 CPU system.  This doesn’t really make sense.

One of the interesting strategies VMware presents in some of their documentation is this idea of a “normalized” CPU.   Basically this is kind of calculating how many megahertz  a system uses instead of using a percentage (which is relative to how many horses are under the hood).  While this isn’t entirely accurate, it does help us create high-water marks for a normalized calculation.

In the above example, our 1CPU system would have a normalized CPU usage of :

• 1 CPU x 700MHz  x .75 = 525MHz

Our 4CPU system would normalize to :

• 4 CPU x 2200MHz x .20 = 1760MHz

It is MUCH easier to compare the 2 of them now and to see which one will probably have a larger processor footprint.  I do realize that it doesn’t really take into account performance of multi-threaded applications and such, but it does a pretty good job.

So, how do we get all that info to calculate it? Well, we also happen to have SMS or SCCM available to collect configuration information about our systems.  This enables us to create some custom database views that contained system information regarding CPU Speeds, number of CPUs,  Hyperthreading configuration, etc.  We needed to have info on Hyper-Threading enabled hardware because Windows actually reports those “HT” processors as physical processors.  Unfortunately a  HT processor doesn’t really have the same performance as a TRUE physical processor.

To get around this, we figure a HT Processor is about a half of a physical processor.  So if our 4CPU system above is really a 2CPU system with HT, it would look like this:

• 2 CPU x 2200MHz x .20 = 880MHz
• 2 CPU x .5 (because they are HT)  x 2200MHz x .20 = 440MHz
• Total Normalized CPU usage is 1320MHz

I hope I did an okay job of explaining it.  If not, please let me know.

Memory and Commitment

Memory is VERY easy to evaluate compared to CPU numbers as it is already normalized (kinda).  We will use the same methodologies for memory as we did for CPU with respect to Average Values and peak times.  When looking at memory counters, we are interested in finding out how much memory our systems are using, not how much is free, or the percent free or anything like that.

As Microsoft calls it, we need the “Committed Bytes in Use” counter.   I don’t think Operations Manager collects this info by default, so you may need to start collecting it to do this type of analysis.

Other than using the right counter it is pretty much the same as CPU.

Disk and Network Stuff

While we DO care about network and disk throughput, they typically will be weighted much less than CPU and memory utilization when looking at the overall analysis.  If you are looking at virtualizing some beefy SQL or Exchange systems, then these will become MUCH more important, but probably still following memory and cpu.

We use the same Average calculations as we do for CPU and Memory, but we will look at the disk counter “Disk Bytes per Second”  for this analysis and
“Total Bytes per second”(I think) for the network counter.  It MAY be important to eliminate loopback adapters in the query for network data, but most of the time our data comes back with numbers that are WAY below our thresholds for candidacy on even relatively busy systems.

Whew!!!

So, I made it through this in one fell swoop.  I am SURE that it isn’t written perfectly, but I wanted to get my thoughts down on this topic.  If you have any questions about it or recommendations….. PLEASE comment and open a dialogue.

Thanks for making it through this… and good luck.

– Mark