One of the MANY challenges we faced while trying to implement a brand-spanking-new virtualization infrastructure was to put together some type of guidance for our server admin teams to help them determine if existing systems would “live” happily in our new environment.

After reading recommendations from VMware, digging through Microsoft Management Packs for Operations Manager, etc., it didn’t seem that any of those really fit our needs nor did they make much sense.

So we happen to use Operations Manager to collect performance data on our Windows-based servers, but there is a TON of data.

We also had to make sure that the tools we were using (Operations Manager) didn’t aggregate our data.  We NEEDED to have granular data for our entire 3 month look-back period.  We just didn’t like the idea of averaging averages, so we have data points every 5 or 15 minutes (depending on the counter) for 90 days.  One of the early reports we developed let admins specify the peak usage times so that the performance analysis could be tailored for specific systems.

Now we have all of this data but how do we look at it?  How do we know what’s important?  What recommendations from VMware and MS do we use and which ones do we throw away?

Several of the deficiencies we saw with most tools are:

1. They average entire periods of time (a day, a week, a month) with no regard for peak usage times
2. They use aggregated data (kind of like above)
3. They assume that “CPU %” means the same thing for all systems

So now we have to look only at peak times for specific servers.  Wow..that can be a VERY daunting task unless you make SOME assumptions.  That’s exactly what we did.

We know that not ALL systems will have the same usage patterns, but generally our systems will be the busiest during normal business hours at our corporate data center.  So, that would be Monday – Friday, 8am-5pm CST.  Well, we also have people using our systems on the east and west coasts, so if we move our “peak” time to 7am-7pm CST, we should cover most of the usage we wish to capture.

You  may be asking why not just take all data points and be done with it… I know *WE* did.

It would certainly have been easier, but far less accurate.  To demonstrate this, think about this:

• A system has a CPU utilization of 75% all the time during peak usage times
• This same system has a CPU utilization of 10% during off-peak times
• This system has a CPU utilization of 10% during the ENTIRE weekend

Okay, so if we take the mean average (the normal practice) we get something in the range of…well.. I am not gonna do the math, but it is WAY less than 75%.  We THEN thought, well… why not just take the highest or maximum measurement for the entire period and plan for that?  That doesn’t work out so well either if you have systems that don’t do ANYTHING, then a virus scan kicks off and pegs the CPU for 3o minutes.  This greatly throws our performance footprint off.  So now we have to find a happy sweet-spot for evaluating performance.

After mulling through all of our options and thinking about what types of data were REALLY helpful in this process, we decided to do a “Top 20% Average”.  I know it sounds a little strange, but it is pretty straight-forward. How did we come up with this number?  Well, we had to pick a number and threw some others around and this one seems to work, so we stuck with it.  The calculation is pretty simple:

1. Take all of your data points
2. Sort them High to Low
3. Take the top 20% of your values and average them

This gives us a nice look at what our systems do when they are operating at or near their highest loads.  The benefit of having a Top 20% Average and its normal average is that the greater the difference between the two, the “more spiky” the performance is.  As the numbers get closer and closer we can see that our systems are more consistently busy during our peak usage time.

The two most important things we need to be looking at are CPU and Memory utilization since those are often the most limiting resources in our environment.  Of less importance to us were Disk IO and Network IO because we were really looking at “low hanging fruit” types of systems and NORMALLY if a system is disk or network heavy, CPU utilization will also shoot up.

So now we have this notion of Top 20% Average (T20 Average) and Average, but what are we actually analyzing?

Not So Normal CPU Usage

Well, we can’t just look at CPU Utilization Percent!  Why, you may ask? Well CLEARLY the following are not equivalent:

• 1 CPU – 700MHz system running 75% utilization
• 4 CPU – 2.2 GHz system running 20% utilization

Many tools used to evaluate this type of data look at Total CPU Percent Utilization.  This would mean our 1 CPU system has a larger processor footprint than our 4 CPU system.  This doesn’t really make sense.

One of the interesting strategies VMware presents in some of their documentation is this idea of a “normalized” CPU.   Basically this is kind of calculating how many megahertz  a system uses instead of using a percentage (which is relative to how many horses are under the hood).  While this isn’t entirely accurate, it does help us create high-water marks for a normalized calculation.

In the above example, our 1CPU system would have a normalized CPU usage of :

• 1 CPU x 700MHz  x .75 = 525MHz

Our 4CPU system would normalize to :

• 4 CPU x 2200MHz x .20 = 1760MHz

It is MUCH easier to compare the 2 of them now and to see which one will probably have a larger processor footprint.  I do realize that it doesn’t really take into account performance of multi-threaded applications and such, but it does a pretty good job.

So, how do we get all that info to calculate it? Well, we also happen to have SMS or SCCM available to collect configuration information about our systems.  This enables us to create some custom database views that contained system information regarding CPU Speeds, number of CPUs,  Hyperthreading configuration, etc.  We needed to have info on Hyper-Threading enabled hardware because Windows actually reports those “HT” processors as physical processors.  Unfortunately a  HT processor doesn’t really have the same performance as a TRUE physical processor.

To get around this, we figure a HT Processor is about a half of a physical processor.  So if our 4CPU system above is really a 2CPU system with HT, it would look like this:

• 2 CPU x 2200MHz x .20 = 880MHz
• 2 CPU x .5 (because they are HT)  x 2200MHz x .20 = 440MHz
• Total Normalized CPU usage is 1320MHz

I hope I did an okay job of explaining it.  If not, please let me know.

Memory and Commitment

Memory is VERY easy to evaluate compared to CPU numbers as it is already normalized (kinda).  We will use the same methodologies for memory as we did for CPU with respect to Average Values and peak times.  When looking at memory counters, we are interested in finding out how much memory our systems are using, not how much is free, or the percent free or anything like that.

As Microsoft calls it, we need the “Committed Bytes in Use” counter.   I don’t think Operations Manager collects this info by default, so you may need to start collecting it to do this type of analysis.

Other than using the right counter it is pretty much the same as CPU.

Disk and Network Stuff

While we DO care about network and disk throughput, they typically will be weighted much less than CPU and memory utilization when looking at the overall analysis.  If you are looking at virtualizing some beefy SQL or Exchange systems, then these will become MUCH more important, but probably still following memory and cpu.

We use the same Average calculations as we do for CPU and Memory, but we will look at the disk counter “Disk Bytes per Second”  for this analysis and
“Total Bytes per second”(I think) for the network counter.  It MAY be important to eliminate loopback adapters in the query for network data, but most of the time our data comes back with numbers that are WAY below our thresholds for candidacy on even relatively busy systems.

Whew!!!

So, I made it through this in one fell swoop.  I am SURE that it isn’t written perfectly, but I wanted to get my thoughts down on this topic.  If you have any questions about it or recommendations….. PLEASE comment and open a dialogue.

Thanks for making it through this… and good luck.

– Mark

### Written by Mark A. Weaver
##  Date: 7/27/2008
##  Version: 1.0
##  Website: www.vmweaver.com
##
##  Call this function and pass in -ServerName  -ClusterName
##  Output should be an object containing the information
##
##  Feel free to modify as needed to suit your needs, but please keep this header
##
##  Thanks  -- Mark
 
function Get-HACapacity(
[string]$ServerName,
[string]$ClusterName)
{
		if (($ServerName -ne "") -and ($ClusterName -ne ""))
	{
		# These booleans tell me if I am using the VMware default memory and cpu reservations for the cluster.
			$DASMemDefault = $True
			$DASCPUDefault = $True
 
		# The following numbers are derived from VMware published numbers for memory overhead.
		# I have dropped them into arrays using the number of vCPUs as an index to get the correct constant.
		# This is why you will notice only [1], [2], and [4] have non-zero values
		# These constants are used later on when calculating Memory Reserve.
			$MemConst32 = 0, 3.262, 5.769059, 0, 6.77933
			$MemConst64 = 0, 3.2678, 5.79251, 0, 6.82622
			$MemBase32 = 0, 87.56, 108.73, 0, 146.75
			$MemBase64 = 0, 107.54, 146.41, 0, 219.82
 
		# Initialize some Variables
			$MaxMemRes = 0
			$MacNumCPU = 0
			$MaxCPUResVM = ""
			$VMCount = 0
 
		# define default memory and cpu reservation
			$DASMinMHz = 256
			$DASMinMemory = 256
 
			$viServerName = $ServerName
			$viClusterName = $ClusterName
 
		# Connect to the VirtualCenter Server and get some info
			$viServer = Connect-VIServer $viServerName
			$viCluster = get-cluster $viClusterName
			$viHosts = get-vmhost -location $viCluster
			$viClusterV = get-view $viCluster.ID
 
		# Get the "Resources" Resource Pool from the cluster.
		# This gives us the Reservation Pools for Memory and CPU
			$viResGroup = Get-ResourcePool -Name "Resources" -Location $viCluster
			$viCPURes = $viResGroup.CpuReservationMHz
			$viMemRes = $viResGroup.MemReservationMB
			$viHostCount = $viClusterV.Summary.NumHosts
 
		# Get HA cluster configuration information
			$viHostFailures = $viClusterV.Configuration.DasConfig.FailoverLevel
 
		# Get a list of options that may be configured at the clusters level
		# We are looking for whether or not the default memory and cpu
		#  reservations have been overridden
			$viDASOptions = $viClusterV.Configuration.DASConfig.Option
			$viVMs = get-vm -Location $viCluster
 
		# Is Adminisssion Control enabled on the cluster?
			$viClusterControl = $viClusterV.Configuration.DASConfig.AdmissionControlEnabled	
 
		# See if das.vmMemoryMinMB key is defined and grab its value
		# See if das.vmCpuMinMHZ key is defined and grab its value
			if ($viDASoptions.Count -ne 0)
			{
				foreach ($viDASOption in $viDASOptions)
				{
					if ($viDASOption.Key -eq "das.vmMemoryMinMB")
					{
						$DASMemDefault = $False
					$DASMinMemory = $viDASOption.Value }
 
					if ($viDASOption.Key -eq "das.vmCpuMinMHz")
					{
						$DASCPUDefault = $False
					$DASMinMHz = $viDASOption.Value }
				}
		}
 
		# Let's go through every VM and see what the maximum CPU and Memory reservation is.
		# We will also get a count of powered on VMs.
		# When we hit a maximum reservation, save the machine name that set that maximum
			foreach ($viVM in $viVMs)
			{
				$NumCPU = $viVm.NumCPU
				$VMMem = $viVm.MemoryMB
				$MemRes = 0
 
				if ($viVM.PowerState -eq "PoweredOn")
				{
					$VMCount += 1
			}
 
			# Get the VM-view and determine if the current guest CPU or memory reservations configured
			$vmView = get-view $viVM.ID
			$vmViewCPURes = $vmView.ResourceConfig.CpuAllocation.Reservation
			$vmViewMemRes = $vmView.ResourceConfig.MemoryAllocation.Reservation
 
			# If no reservations are set at the VM level, calculate the memory reservation.
			if ($vmViewMemRes -eq 0)
			{
				if ($VMMem -le 256)
				{
					$MemRes = $MemConst64[$NumCpu] + $MemBase64[$NumCPU]
				}
				else
				{
					if ((($viVM.Guest.OSFullName | Select-String "64-bit").Matches.Count) -ge 1)
					{
						$MemRes = ($VMMem / 256) * $MemConst64[$NumCPU] + $MemBase64[$NumCPU]
					}
					else
					{
						$MemRes = ($VMMem / 256) * $MemConst32[$NumCPU] + $MemBase32[$NumCPU]
					}
				}
 
				$MemRes += $DASMinMemory
			}																			
 
			else
			{
				$MemRes = $vmViewMemRes 									
 
			}
 
			#Figure out if the current VM holds the highest reservation so far
 
				if ($vmViewCPURes -gt $DASMinMHz)
				{
					$DASMinMHz = $vmViewCPURes
					$MaxCPUResVM = $viVM.Name
				}			
 
				if ($MemRes -gt $MaxMemRes)
				{
					$MaxMemRes = $MemRes
					$MaxMemResVM = $viVM.Name
				}
 
				if ($NumCPU -gt $MaxNumCPU)
				{
					$MaxNumCPU = $NumCPU
					$MaxCPUNumVM = $viVM.Name
				}
 
			}
 
			if ($MaxCPUResVM -eq "") { $MaxCPUResVM = $MaxCPUNumVM }			
 
		$MaxCPURes = $MaxNumCPU * $DASMinMHz
 
		# Calculate the VM Capacity for the cluster based on memory and cpu reservations.
			$ClusterVMCapacityMEM = [Math]::Truncate(((($viMemRes / $MaxMemRes) * ( $viHostCount - $viHostFailures )) / $viHostCount))
			$ClusterVMCapacityCPU = [Math]::Truncate(((($viCPURes / $MaxCPURes) * ( $viHostCount - $viHostFailures )) / $viHostCount))
 
			if ($ClusterVMCapacityMEM -lt $ClusterVMCapacityCPU)
			{
				$ClusterVMCapacity = $ClusterVMCapacityMEM
			}
			else
			{
				$ClusterVMCapacity = $ClusterVMCapacityCPU
		}
 
		# Create an object to return
			$CPUObj = New-Object System.Object
			$CPUObj | Add-Member -type NoteProperty -name ClusterCPURes -value $viCPURes
			$CPUObj | Add-Member -type NoteProperty -name DefaultCPURes -value $DASCPUDefault
			$CPUObj | Add-Member -type NoteProperty -name MinCPURes -value $DASMinMHz
		   $CPUObj | Add-Member -type NoteProperty -name MaxCPUNumVM -value $MaxCPUNumVM
			$CPUObj | Add-Member -type NoteProperty -name MaxCPURes -value $MaxCPURes
			$CPUObj | Add-Member -type NoteProperty -name MaxCPUResVM -value $MaxCPUResVM
			$CPUObj | Add-Member -type NoteProperty -name MaxCPUs -value $MaxNumCPU
			$CPUObj | Add-Member -type NoteProperty -name VMCapacityCPU -value $ClusterVMCapacityCPU
 
			$MemObj = New-Object System.Object
			$MemObj | Add-Member -type NoteProperty -name ClusterMemRes -value $viMemRes
			$MemObj | Add-Member -type NoteProperty -name DefaultMemRes -value $DASMemDefault
			$MemObj | Add-Member -type NoteProperty -name MinMemRes -value $DASMinMemory
			$MemObj | Add-Member -type NoteProperty -name MaxMemRes -value $MaxMemRes
			$MemObj | Add-Member -type NoteProperty -name MaxMemResVM -value $MaxMemResVM
			$MemObj | Add-Member -type NoteProperty -name VMCapacityMem -value $ClusterVMCapacityMEM
 
			$OutObj = New-Object System.Object
			$OutObj | Add-Member -type NoteProperty -name AdmissionControl -value $viClusterControl
			$OutObj | Add-Member -type NoteProperty -name CPU -value $CPUObj
			$OutObj | Add-Member -type NoteProperty -name FailoverHosts -value $viHostFailures
			$OutObj | Add-Member -type NoteProperty -name HostCount -value $viHostCount
			$OutObj | Add-Member -type NoteProperty -name Memory -value $MemObj
			$OutObj | Add-Member -type NoteProperty -name RunningVMs -value $VMCount
			$OutObj | Add-Member -type NoteProperty -name VIServer -value $viServerName
			$OutObj | Add-Member -type NoteProperty -name VICluster -value $viClusterName
			$OutObj | Add-Member -type NoteProperty -name VMCapacity -value $ClusterVMCapacity	
 
			return($outObj)
	}
	else
	{
		# Write usage info
		Write-Host ("")
		Write-Host ("-------------------------------------")
		Write-Host ("Get-HACapacity.ps1 Usage:")
		Write-Host( "You must specify the following parameters: ")
		Write-Host ("     '-ServerName '  where  is the name of the VirtualCenter Server")
		Write-Host("     '-ClusterName '  where  is the name of the cluster to query")
		Write-Host ("")
		}
}

Let me know if you have any problems with it or would like more info..

– Mark