A co-worker asked me the other day how to get the version of an ESX host from the host itself (the service console). I told him to look at /etc/vmware-release, but he said that that wasn’t enough information. The next day another co-worker asked if it is possible to tell the difference between a vCenter server and an ESX host without authenticating to either. I said, sure, the RetrieveServiceContent method does not require authentication. Then it occurred to me you can use the same thing locally on an ESX host to determine the exact version of the host.

Since ESX <4 hosts do not have cURL on them, it was up to Perl to do the job. Here is the Perl that you can execute from the service console that will print out the full version of the ESX server you are executing from:

use strict;
use LWP::UserAgent;
use HTTP::Request::Common; 

my $userAgent = LWP::UserAgent->new(agent => 'perl post');
my $message = '
<?xml version="1.0" encoding="UTF-8"?>
<soapenv:envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
	<soapenv:body>
		<retrieveservicecontent xmlns="urn:vim25">
			<_this type="ServiceInstance">ServiceInstance
		</_this>
	</retrieveservicecontent>
</soapenv:body>
</soapenv:envelope>
';

my $response = $userAgent->request( POST 'http://localhost/sdk/webService',
									Content_Type => 'text/xml',
									Content => $message); 

if ($response->content =~ /<fullname>([^<].*)<\/fullName>/)
{
	print $1;
}

And for your convenience, here is the same Perl properly escaped so that it can execute on the command line in a single command:

perl -e "use strict; use LWP::UserAgent; use HTTP::Request::Common; my \$userAgent = LWP::UserAgent->new(agent => 'perl post'); my \$message = '<?xml version=\"1.0\" encoding=\"UTF-8\"?><soapenv:Envelope xmlns:soapenv=\"http://schemas.xmlsoap.org/soap/envelope/\" xmlns:xsd=\"http://www.w3.org/2001/XMLSchema\" xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\"><soapenv:Body><RetrieveServiceContent xmlns=\"urn:vim25\"><_this type=\"ServiceInstance\">ServiceInstance</_this></RetrieveServiceContent></soapenv:Body></soapenv:Envelope>'; my \$response = \$userAgent->request(POST 'http://localhost/sdk/webService', Content_Type => 'text/xml', Content => \$message); if (\$response->content =~ /<fullName>([^<].*)<\/fullName>/) {print \$1;}"

Hope this helps!

Update 1

Before someone point out the obvious, I did not display the result of the script. It will look something like this: VMware ESX Server 3.5.0 build-64607

VMWare ESX server.. interesting working late one night we experienced corruption of one of our vmx files for a running vmware image. Whilst the server continued running without a glitch, we could not stop or restart and console on to it, with this damaged vmx file. A bit of ‘googling’ and we turned up fantastic article below that helped us confirm our thinking and fix it in a bout five minutes flat.

• VMware Vitrual Machine .vmx files recovery.. great article.

So we had a massive maintenance on one of our ESX clusters. The maintenance entailed the complete shutdown of all VMs and hosts in the cluster. The entire task went relatively fine, with the exception of the usual stuff like:

• HA configuration failed to apply to most of the hosts when they were powered on. Fix for this issue is straight forward: Disable HA for the cluster and re-enable it.

The upside to disabling HA when having to power-on multiple VMs (350+) is that they power-on faster, as the VMs do not have to go through HA admission control and you will of course receive no HA related errors

• Job queuing in VI causing VMs to take some time to power-on

It appears to me that VI just throws all jobs in a pool and try to get to them as fast as possible. I would like the option to tell it to process a set amount of jobs at a time. This can of course be achieved with the power of shell, but if anybody knows how it can be done in the GUI, let me know.

But what about the invalid VM?

Well, one of the machines came up with the name in italics and with an (invalid) appended to the name… Of course the GUI did not provide much help. When attempting to power on the machine, the only message I got was a “not allowed in the the current state”

Directed by this post: http://itsupportjournal.com/2008/12/09/fix-invalid-guest-on-virtual-center/ I started checking my .vmx file for errors and found.. none. As it turns out, my issue was in the extended config file (.vmxf)

This is what was in there:

I started scratching my head, as it was obvious that the invalid came from this file, but how to create a new one, when the .vmxf file contains a unique identifier type string amongst others:

<?xml version="1.0"?>
<Foundry>
<VM>
<VMId type="string">52 fc dd 7e 09 fa ac 07-46 87 01 ad 28 e5 ca 98</VMId>
<ClientMetaData>
<clientMetaDataAttributes/>
<HistoryEventList/></ClientMetaData>
<vmxPathName type="string">vmname.vmx</vmxPathName></VM></Foundry>

Well sometimes the easy solution is the right one (love when that happens

1. Unregister the invalid VM in VI
2. Open the datastore browser and browse to the folder containing the VM
3. Rename the original .vmxf file.
4. Register the VM in VI from the datastore browser (right-click –> Add to inventory –> Step through the wizard)
5. A new and proper .vmxf file is generated
6. Power-On the VM

Do let me know if there is any way of “hand creating” the extended config file. Not that it is very useful if it can be automatically generated, but it would be cool to be able to do it…

/theAdminGuy

We all know how long and annoying the deletion of ancient snapshots can be. Even worse when vcenter times-out and you are literally flying blind.

The command bellow gives you real-time view of the VMDK/snapshot files and you can watch them shrinking.

watch "ls -Ghtu –full-time *.vmdk" –d

Have Fun

This post is a writeup of a project for a master’s class in Decision Support Systems at Murray State.  This is my first dive into VMware PowerCLI aside from some one shots. All feedback is welcome.

Our Problems

Problem 1: Servers are not being virtualized due to a decentralized procurement process

A decentralized server procurement process presents many problems to an organization. There are many gains with standardizing OS/hardware platforms.

Problem 2: Servers are not being virtualized because knowledge is required to make “Virtualize/Don’t Virtualze” decision

The benefits of server virtualization are easy to explain and are a part of our culture. However, the organization has not adopted a “virtualize first” mentality. There is still a lack of stakeholder understanding with regards to virtualization.

Due to lack of knowledge, ROI is not maximized. This knowledge exists in two places – the virtual infrastructure itself and as tacit knowledge with the VMware administrator.

PROBLEM ANALYSIS

Problem 1: Servers are not being virtualized due to a decentralized procurement process
This problem is outside of the scope of the CIS645 class. We’re working on it.

Problem 2: Servers are not being virtualized because knowledge is required to make “Virtualize/Don’t Virtualze” decision

Problem 2 has two major parts.

CAPACITY – CAN OUR VIRTUAL INFRASTRUCTURE SUPPORT THIS APPLICATION?

This question has historically been answered heuristically with ball park figures. Manually gathering current storage and RAM capacity data too time consuming.

CANDIDACY – BASED ON SYSTEM REQUIREMENTS AND INDUSTRY KNOWLEDGE, IS VIRTUALIZATION SUITABLE FOR THIS APPLICATION?

This is the harder question. Typically you’ll hear consultants say “it depends”. Answering this question usually involves a phone call with the VMware administrator. The conversation is series of questions from the administrator to the stakeholder.

RECOMMENDATION

When the two questions have been answered, a recommendation of Virtualize/Don’t Virtualize is made. If a Virtualize decision is made, the VMware administrator must find the optimal storage unit to deploy to and coordinate the deployment with the stakeholder.

SOLUTION DESIGN

USER INTERFACE

The users of this system are already familiar with Excel and would prefer to utilize Excel’s familiarity and What-If scenario planning.

What if we added another 2TB of storage?
What if we upgraded our RAM?
What if we didn’t have to have the license dongle?

Excel quickly enables these questions to be answered. A normal ‘GUI’ application would take more time to develop and would not invite queries of an ad-hoc nature.

CAPACITY

Capacity data resides at several levels: the virtual machine itself, the host, and the data store. The data is put into Excel using VMware’s PowerCLI. PowerCLI is a Windows PowerShell snap-in that integrates with any VMware Virtual Infrastructure. Windows PowerShell also integrates nicely with Excel.
Here are the steps to capacity gathering with the VMware Expert System:

• Open the Excel Spreadsheet
• Clear previously gathered data
• Connect to a vCenter Server
• Gather datastore information
• Gather host information
• Gather virtual machine information
• Write values to ‘Capacity’ Worksheet
• Write values to ‘New Virtual Machine’ Worksheet
• Save Excel Spreadsheet
• Clean up and quit Excel

CANDIDACY

The user of the VMware Expert System will answer a series of questions to determine system candidacy. Through knowledge capture, the conversation with the VMware Administrator does not need to take place. The knowledge is generally accepted by a community of VMware experts.

RECOMMENDATION

After answering the capacity and candidacy questions, the user receives a final recommendation. The recommendation is only “Virtualize” if capacity is available and candidacy is met.

The interface also displays reasons why a machine is not suitable for virtualization to enable What-If analysis.

DECISION TREE

Modified from VI:OPS P2V Decision Tree

RUNNING THE VMWARE EXPERT SYSTEM

PREREQUISITES

• VMware PowerCLI 4.0 or better
  • Installation guide: http://www.vmware.com/support/developer/windowstoolkit/wintk40/doc/viwin_install.pdf

STEPS TO RUN THE VMWARE EXPERT SYSTEM

• Download and extract vmware-expert-system.zip
• Rename launch.tab to launch.bat
• Edit launch.bat, line 2
  • Substitute your path to updatespreadsheet.ps1 where you see “C:\users\%username%\Documents\cis645\Project\vmware_expert_system\updatespreadsheet.ps1″, make sure the path is in quotation marks
• Edit updatespreadsheet.ps1, line 11
  • Substitute your path to vmware_expert_system.xlsm where you see “C:\users\%username%\Documents\cis645\Project\vmware_expert_system\vwmare_expert_system.xlsm”, make sure the path is in quotation marks
• Run click ‘launch.bat’
• A screen similar to this will appear:
  
• Launch the spreadsheet “vmware_expert_system.xlsm” and enable macros
• Enter system requirements
• Press “Send Work Order”

EXAMPLE SYSTEM: NEW WEB SERVER

• Enter the hostname: newwebserver
  • The hostname must not be already existing and must be a valid hostname (“The Internet Engineering Task Force (IETF)”)
• Enter a functional contact: Andy Hill
• Enter a staff contact: Andy Hill
• Select an Operating System: Windows Server 2003
• Enter a storage requirement: 20 GB
  • The minimum storage requirement must be >8 GB and less than the size of a maximum single disk
• Enter a RAM requirement: 1024 MB
  • The minimum RAM requirement is 256MB and must be less than one host and still tolerant of a host failure
• Number of Processors: 1
  • Must be numeric, greater than or equal to 1, less than or equal to 4
• Number of NICs: 1
  • Must be numeric, greater than or equal to 1, less than or equal to 4
• Average CPU utilization: 5%
  • Must be numeric, between 0 and 1, if 4 processors are used average utilization cannot exceed 50%
• Average RAM utilization: 256 MB
  • Must not exceed 8GB
• Average NIC utilization: 1 MBps
  • Must not exceed 100MBps
• Maximum Disk IO: 10 MBps
  • Must not exceed 100MBps
• Answer TRUE/FALSE to the following hardware components
  • Modems: FALSE
  • Fax Cards: FALSE
  • License Dongles: FALSE
  • Security Dongles: FALSE
  • Hardware Encryption: FALSE
• Answer TRUE/FALSE to Vendor Support: TRUE
• Recommendation: Virtualize!

ADD NEW SUPPORTED GUEST OS

VMware’s Guest OS Compatibility Guide (“VMware, Inc.”) is exhaustive and does not line up with Murray State University’s environment. The drop-down list is populated from a hidden worksheet within Excel. For our environment, we limited this drop down to Guests OSes which have regularly maintained templates.

To add, delete, or change an entry in the operating system list follow these steps:

1. Toward the bottom of Excel, right click the current worksheet
2. From the context menu, select “Unhide…”
   
3. From the Unhide Window, Select ‘Supported Guest Operating Systems’ and press OK
   
4. Navigate to the ‘Supported Guest Operating Systems’ Worksheet. Make changes Column A. Only changes in Column A will be reflected in the spreadsheet. Save your changes.

Future Considerations

# VMware Expert System Capacity Gathering
# v0.2
# by Andy Hill
# http://virtualandy.wordpress.com

# gathering data for VMware capacity
$viserver = Read-Host "Enter a vCenter server";
Write-Host "Gathering Excel data...1/8"

$excel = new-object -comobject Excel.Application
# Edit this value to the location of your vmware_expert_system.xlsm
$excelfile = $excel.workbooks.open("C:\Users\andy.hill\Documents\cis645\Project\vmware_expert_system\vmware_expert_system.xlsm")
$worksheet = $excelfile.worksheets.item(3) # Select Capacity Worksheet

Write-Host "Clearing existing capacity data...2/8"
# Clear existing data
$worksheet.Range("A5:N65000").Clear() | out-null
$worksheet.cells.item(1,2) = $viserver

Write-Host "Connecting to $viserver, this may take a moment...3/8"
connect-viserver $viserver -erroraction stop -WarningAction SilentlyContinue | out-null

# datastore information
Write-Host "Gathering disk information...4/8"
$i = 5
$disks = get-datastore
foreach($disk in $disks) {
	$worksheet.cells.item($i, 1) = $disk.name;
	$worksheet.cells.item($i, 2) = $disk.freespaceMB;
	$worksheet.cells.item($i, 3) = $disk.capacityMB;
	$i++;
}
$disk_count = $i;

$i = 5
Write-Host "Gathering host information...5/8"
# host information
Get-VMHost | %{Get-View $_.ID} | %{
	$esx = "" | select Name, NumCpuPackages, NumCpuCores, Hz, Memory
	$esx.NumCpuPackages = $_.Hardware.CpuInfo.NumCpuPackages
	$esx.NumCpuCores = $_.Hardware.CpuInfo.NumCpuCores
	$esx.Hz = $_.Hardware.CpuInfo.Hz
	$esx.Memory = $_.Hardware.MemorySize
	$esx.Name = $_.Name

	$worksheet.cells.item($i, 6) = $esx.Name
	$worksheet.cells.item($i, 7) = $esx.NumCpuPackages
	$worksheet.cells.item($i,  = $esx.NumCpuCores
	$worksheet.cells.item($i, 9) = $esx.hz / 1000 / 1000
	$worksheet.cells.item($i, 10) = $esx.memory / 1024 / 1024;
	$i++;
}
$host_count = $i;

# vm information
$i = 5
Write-Host "Gathering virtual machine information...6/8"

get-vm | % {
	$vm = "" | select name, MemoryMB
	$worksheet.cells.item($i, 13) = $_.Name
	$worksheet.cells.item($i, 14) = $_.MemoryMB
	$i++;
}

# Create the totals and amount utilized
$worksheet.cells.item(($i+1),13) = "Total"
$worksheet.cells.item(($i+1),14) = "=sum(N6:N" + $i + ")"
$vm_count = $i;

Write-Host "Writing values to Excel Spreadsheet...7/8"
#add some formatting
$worksheet.cells.item(($disk_count + 2), 1) = "Datastore with most free space";
$worksheet.cells.item(($disk_count + 3), 1) = "Memory (MB) Available";
$worksheet.cells.item(($disk_count + 4), 1) = "Memory Utilization %";
$worksheet.cells.item(($disk_count + 5), 1) = "Storage Available (GB)";
$worksheet.cells.item(($disk_count + 6), 1) = "Storage Utilization %";
$worksheet.cells.item(($disk_count + 7), 1) = "Most Storage Available on a datastore (GB)";

# add the formulas
$worksheet.cells.item(($disk_count + 2), 2) = "=INDEX((A5:A" + $disk_count + "),MATCH(MAX(B5:B" + $disk_count + "),B5:B" + $disk_count + ",0))";
$worksheet.cells.item(($disk_count + 3), 2) = "=SUM(J5:J" + $host_count + ") - N" + ($vm_count+1);
$worksheet.cells.item(($disk_count + 4), 2) = "=N" + ($vm_count+1) + "/SUM(J5:J" + ($host_count-1) + ")"; # n-1 hosts for HA failover
$worksheet.cells.item(($disk_count + 5), 2) = "=SUM(B5:B" + $disk_count + ")/1024";
$worksheet.cells.item(($disk_count + 6), 2) = "=1-SUM(B5:B" + $disk_count + ")/SUM(C5:C" + $disk_count + ")";
$worksheet.cells.item(($disk_count + 7), 2) = "=INDEX((B5:B" + $disk_count + "),MATCH(MAX(B5:B" + $disk_count + "),B5:B" + $disk_count + ",0))/1024";

Write-Host "Saving Excel Spreadsheet...8/8";
# Select main worksheet
$worksheet = $excelfile.worksheets.item(1);
# Update the 'new virtual machine' worksheet with capacity data
$worksheet.cells.item(8,4) = "=Capacity!B" + ($disk_count + 5) + "-'New Virtual Machine'!B8";
$worksheet.cells.item(8,7) = "=MAX(Capacity!B5:" + "B" + ($disk_count - 1) + ")/1024";
$worksheet.cells.item(9,4) = "=(Capacity!B" + ($disk_count +3) + ")/1024";
$worksheet.cells.item(29,2) = "=Capacity!B" + ($disk_count + 2);
$excel.activeworkbook.save();
$excel.quit();

Recently I’m involved in a design review of a new vSphere environment, the customer is considering implementing ESX4i. One of their wishes is to use the software iSCSI initiator in combination with Jumbo frames. Unfortunately this is (still) an unsupported configuration. Jumbo frames are supported on ESX4 and ESX 4i. ESX4 supports jumbo frames on all network ports, but ESX4i only supports it on VM Machine Network interfaces. Page 54 of the ESXi Configuration Guide Update 1 ESXi 4.0 guide lists the following:

Jumbo frames are not supported for VMkernel networking interfaces in ESXi

If Software iSCSI initiator and Jumbo frames are required, you are oblidged to use ESX 4.0 at this moment.

ESXi Configuration Guide Update 1 ESXi 4.0 guide
http://www.vmware.com/pdf/vsphere4/r40_u1/vsp_40_u1_esxi_server_config.pdf

Update! See new blog post about ESX4i and jumbo frames support
http://frankdenneman.wordpress.com/2009/12/19/esx-4i-support-jumbo-frames/

I have a customer who wants to set memory reservation on a large scale. Instead of using resource pools they were thinking of setting reservations on VM level to get a guaranteed performance level for every VM. Due to memory management on different levels, using such a setting will not get the expected results. Setting aside the question if it’s smart to use memory reservation on ALL VM’s, it raises the question what kind of impact setting memory reservation has on the virtual infrastructure, how ESX memory management handles memory reservation and even more important; how a proper memory reservation can be set.

Key elements of the memory system
Before looking at reservations, let’s take a look what elements are involved. There are three memory layers in the virtual infrastructure:

• Guest OS virtual memory – Virtual Page Number (VPN)
• Guest OS physical memory – Physical Page Number (PPN)
• ESX machine memory – Machine Page Number (MPN)

The OS inside the guest maps virtual memory ( VPN) to physical memory(PPN). The Virtual Machine Monitor (VMM) maps the PPN to machine memory (MPN). The focus of this article is on mapping physical page numbers (PPN) to Machine Page Number (MPN).

Impact of memory management on the VM
Memory reservations guarantee that physical memory pages are backed by machine memory pages all the time, whether the ESX server is under memory pressure or not.
Opposite of memory reservations are limits. When a limit is configured, the memory between the limit and the configured memory will never be backed by machine memory; it could either be reclaimed by the balloon driver or swapped even if enough free memory is available in the ESX sever.
Next to reservations and limits, shares play an important factor in memory management of the VM. Unlike memory reservation, shares are only of interest when contention occurs.

The availability of memory between memory reservation and configured memory depends on the entitled shares compared to the total shares allocated to all the VMs on the ESX server.
This means that the virtual machine with the most shares can have its memory backed by physical pages. For the sake of simplicity, the vast subject of resource allocation based on the proportional share system will not be addressed in this article.

One might choose to set the memory reservation equal to the configured memory, this will guarantee the VM the best performance all of the time. But using this “policy” will have its impact on the environment.

Admission Control
Configuring memory reservation has impact on admission control . There are three levels of admission control;

• Host
• High Availability
• Distributed Resource Scheduler

Host level
When a VM is powered on, admission control checks the amount of available unreserved CPU and memory resources. If ESX cannot guarantee the memory reservation and the memory overhead of the VM, the VM is not powered on. VM memory overhead is based on Guest OS, amount of CPUs and configured memory, for more information about memory overhead review the Resource management guide.

HA and DRS
Admission control also exist at HA and DRS level. HA admission control uses the configured memory reservation as a part of the calculation of the cluster slot size.The amount of slots available equals the amount of VM’s that can run inside the cluster. To find out more about slot sizes, read the HA deepdive article of Duncan Epping. DRS admission control ignores memory reservation, but uses the configured memory of the VM for its calculations. To learn more about DRS and its algorithms read the DRS deepdive article at yellow-bricks.com

Virtual Machine Swapfile
Configuring memory reservation will have impact on the size of the VM swapfile; the swapfile is (usually) stored in the home directory of the VM. The virtual machine swapfile is created when the VM starts. The size of the swapfile is calculated as follows:

Configured memory – memory reservation = size swap file

Configured memory is the amount of “physical” memory seen by guest OS. For example; configured memory of VM is 2048MB – memory reservation of 1024MB = Swapfile size = 1024MB.

ESX use the memory reservation setting when calculating the VM swapfile because reserved memory will be backed by machine memory all the time. The difference between the configured memory and memory reservation is eligible for memory reclamation.

Reclaiming Memory
Let’s focus a bit more on reclaiming. Reclaiming of memory is done by ballooning or swapping. But when will ESX start to balloon or swap? ESX analyzes its memory state. The VMkernel will try to keep 6% free (Mem.minfreepct) of its memory. (physical memory-service console memory)
When free memory is greater or equal than 6%, the VMkernel is in a HIGH free memory state. In a high free memory state, the ESX host considers itself not under memory pressure and will not reclaim memory in addition to the default active Transparent Page sharing process.
When available free memory drops below 6% the VMkernel will use several memory reclamation techniques. The VMkernel decides which reclamation technique to use depending on its threshold. ESX uses four thresholds high (6%), soft (4%) hard (2%) and low (1%). In the soft state (4% memory free) ESX prefers to use ballooning, if free system memory keeps on dropping and ESX will reach the Hard state (2% memory free) it will start to swap to disk. ESX will start to actively reclaim memory when it’s running out of free memory, but be aware that free memory does not automatically equal active memory.

Memory reservation technique
Let’s get back to memory reservation .How does ESX handle memory reservation? Page 17 of the Resource Management Guide states the following:

Memory Reservation
If a virtual machine has a memory reservation but has not yet accessed its full reservation, the unused memory can be reallocated to other virtual machines.

Memory Reservation Used
Used for powered‐on virtual machines, the system reserves memory resources according to each virtual machine’s reservation setting and overhead. After a virtual machine has accessed its full reservation, ESX Server allows the virtual machine to retain this much
memory, and will not reclaim it, even if the virtual machine becomes idle and stops accessing memory.

To recap the info stated in the Resource Management Guide, when a VM hits its full reservation, ESX will never reclaim that amount of reserved memory even if the machine idles and drops below its guaranteed reservation. It cannot reallocate that machine memory to other virtual machines.

Full reservation
But when will a VM hit its full reservation exactly? Popular belief is that the VM will hit full reservation when a VM is pushing workloads, but that is not entirely true. It also depends on the Guest OS being used by the VM. Linux plays rather well with others, when Linux boots it only addresses the memory pages it needs. This gives ESX the ability to reallocate memory to other machines. After its application or OS generates load, the Linux VM can hit its full reservation. Windows on the other hand zeroes all of its memory during boot, which results in hitting the full reservation during boot time.

Full reservation and admission control
This behavior will have impact on admission control. Admission control on the ESX server checks the amount of available unreserved CPU and memory resources. Because Windows will hit its full reservation at startup, ESX cannot reallocate this memory to other VMs, hereby diminishing the amount of available unreserved memory resources and therefore restricting the capacity of VM placement on the ESX server. But memory reclamation, especially TPS will help in this scenario, TPS (transparent page sharing) reduces redundant multiple guest pages by mapping them to a single machine memory page. Because memory reservation “lives” at machine memory level and not at virtual machine physical level, TPS will reduce the amount of reserved machine memory pages, memory pages that admission controls check when starting a VM.

Transparant Page Sharing
TPS cannot collapse pages immediately when starting a VM in ESX 3.5. TPS is a process in the VMkernel; it runs in the background and searches for redundant pages. Default TPS will have a cycle of 60 minutes (Mem.ShareScanTime) to scan a VM for page sharing opportunities. The speed of TPS mostly depends on the load and specs of the Server. Default TPS will scan 4MB/sec per 1 GHz. (Mem.ShareScanGHz). Slow CPU equals slow TPS process. (But it’s not a secret that a slow CPU will offer less performance that a fast CPU.) TPS defaults can be altered, but it is advised to keep to the default.TPS cannot collapse pages immediately when starting a VM in ESX 3.5. VMware optimized memory management in ESX 4; pages which Windows initially zeroes will be page-shared by TPS immediately.

TPS and large pages
One caveat, TPS will not collapse large pages when the ESX server is not under memory pressure. ESX will back large pages with machine memory, but installs page sharing hints. When memory pressure occurs, the large page will be broken down and TPS can do it’s magic. More info on Large pages and ESX can be found at Yellow Bricks. http://www.yellow-bricks.com/2009/05/31/nehalem-cpu-and-tps-on-vsphere/

Use resource pools
Setting memory reservation has impact on the VM itself and its surroundings. Setting reservation per VM is not best practice; it is advised to create resource pools instead of per VM reservations. Setting reservations on a granular level leads to increased administrative and operational overhead. But when the situation demands to use per VM reservation, in which way can a reservation be set to guarantee as much performance as possible without wasting physical memory and with as less impact as possible. The answer: set reservation equal to the average Guest Memory Usage of the VMs.

Guest Memory Usage
Guest Memory Usage shows the active memory use of the VM. Which memory is considered active memory? If a memory page is accessed in mem.sampleperiod (60sec), it is considered active. To accomplish this you need to monitor each VM, but this is where vCenter comes to the rescue. vCenter logs performance data and does this for a period of time. The problem is that the counters average-, minimum and maximum active memory data is not captured on the default vCenter statistics. vCenter logging level needs to upgraded to a minimum level of 4. After setting the new level, vCenter starts to log the data. Changing the statistic setting can be done by Administration > VirtualCenter Management Server Configuration > Statistics.

To display the average active memory of the VM, open the performance tab of the VM and change chart options, select memory

Select the counters consumed memory and average-, minimum- and maximum active memory. The performance chart of most VMs will show these values close to each other. As a rule the average active memory figure can be used as input for the memory reservation setting, but sometimes the SLA of the VM will determine that it’s better to use the maximum active memory usage.

Consumed memory is the amount of host memory that is being used to back guest memory. The images shows that memory consumed slowly decreases.
The active memory use does not change that much during the monitored 24 hours. By setting the reservation equal to the maximum average active memory value, enough physical pages will be backed to meet the VM’s requests.

My advice

While memory reservation is an excellent mechanism to guarantee memory performance levels of a virtual machine, setting memory reservation will have a positive impact on the virtual machine itself and can have a negative impact on its surroundings.

Memory reservation will ensure that virtual machine memory will be backed by physical memory (MPN) of the ESX host server. Once the VM hit its full reservation the VMkernel will not reclaim this memory, this will reduce the unreserved memory pool. This memory pool is used by admission control, admission control will power up a VM machine only if it can ensure the VMs resource request. The combination of admission control and the restraint of not able to allocate reserved memory to other VMs can lead to a reduced consolidation ratio.

Setting reservations on a granular level leads to increased administrative and operational overhead and is not best practice. It is advised to create resource pools instead of per VM reservations. But if a reservation must be set, use the real time counters of VMware vCenter and monitor the average active memory usage. Using average active memory as input for memory reservation will guarantee performance for most of its resource requests.

I recommend reading the following whitepapers and documentation;
Carl A. Waldspurger. Memory Resource Management in VMware ESX Server: http://waldspurger.org/carl/papers/esx-mem-osdi02.pdf
Understanding Memory Resource Management in VMware ESX: http://www.vmware.com/files/pdf/perf-vsphere-memory_management.pdf
Description of other interesting memory performance counters can be found here http://www.vmware.com/support/developer/vc-sdk/visdk25pubs/ReferenceGuide/mem.html
Software and Hardware Techniques for x86 Virtualization: http://www.vmware.com/files/pdf/software_hardware_tech_x86_virt.pdf

This is a contract to hire position in Houston, TX.  Length of the contract term is anticipated to be 6 months or less.

JOB DESCRIPTION

Position Summary:

This position is responsible for driving the technical architecture for our  IT Operations Customers.  The resource must have technical prowess in many areas of technology including storage, servers, and virtualization.  This resource must have vast experiences in cross data center migrations.

Responsibilities:

• Create and define technical landscapes and architecture for large enterprise systems
• The ability to manage multiple technical projects with local and remote colleagues domestically and internationally.
• Serves as single point of contact for other project managers, functional team leads and managers.  Must interface with upper management.
• Must also interact with the customer directly to obtain technical details and requirements.
• Consistently display model leadership qualities for team constantly enhancing performance and driving efficiency throughout the team.
• Supports and adheres to management of contractual agreements in compliance with company policies.
• Define, manage, and comply with standards and have them deployed to all customers if possible.
• Create a technical direction based on customer and internal direction.
• Aligns on-site team with our central resources and other regional team leads to foster best practice, collaboration and mutual support.
• Up to 30% travel visiting client sites and datacenter locations

Business Skills and Knowledge:

• Ability to direct and coordinate efforts and actions across different functional service lines both domestically and internationally including virtualized teams.
• Exceptional verbal and written communications skills; expertise in setting and managing customer expectations
• Strong teamwork and interpersonal skills at all management levels
• Strong knowledge and understanding of business needs, with the ability to establish and maintain a high level of customer trust and confidence in the functional teams
• Ability to translate business needs into technical requirements
• Solid project management skills with ability to multitask and manage multiple small to large projects in a cross functional environment
• Best Practices experience or certification desired (ISO, ITIL, COBIT, Six Sigma), including regulatory compliances (SOX, SAS, HIPAA)
• Proven written and oral communications and presentation skills
• Experience with Disaster Recovery procedures
• Ability to successfully manage customer relationships, uncover needs and provide solutions support
• Adhere to all corporate processes
• Maintaining vendor relationships and staying abreast of best practices and new technologies
• Ability to consult on best practices and solution options

Technical Qualifications:

Minimum of 10 years of relevant job experience required to date.  Experience should include exposure to large systems and data center migrations with a large exposure to heterogeneous environments.  Holding multiple current technical certifications in storage and virtualization is a plus.

• Expert level expertise on VMWare, ESX and the supporting tools including Virtual Center, SRM, HA\DRS, Capacity planner.
• The ability to design large scale virtual environments and migration strategies.
• Experience with storage virtualization, data migration solutions and best practices
• Experience with NetApp devices in multiprotocol environments
• Extensive NetApp experience including all Data OnTAP utilities and software
• Knowledge of Sun StorageTek, HDS and HP, EMC and IBM storage solutions
• Experience in architecture of Storage networks with all storage protocols
• Experience with all major operating systems, and the administration fundamentals
• Knowledge of Network (LAN/WAN) technologies and IT Security standards.

Contact me if you are interested. Questions can be posed as a comment below or directly to me.

Since my blog about Understanding HP Flex-10 Mappings with VMware ESX/vSphere is quite a big hit (seeing the page views per day) I decided to also write about the testing scenario’s which should all be walked through before taking a design as this into production.

In my blog I stated:

Last word of advice: while implementing a technical environment like this it’s crucial to test every possible failure, from single ESX Host to all the separate components. I’ve wrote very detailed documents about it

So let’s take a look at these testing scenario’s which can be divided into three main subjects:

• Hardware (ex. power redundancy)
• Connectivity and failover within the hardware (This is Virtual Connect in my design but could also be normal (SAN)switch configurations, this is depending on the modules that are present in the enclosure.)
• Connectivity and failover within the OS (vSphere Configuration)

As a short introduction: I’ve have been working with HP c-Class components ever since the first c7000 enclosure was placed in the Netherlands. In this time I’ve seen many HP c-Class implementations were people just rely on the fact that “everything is redundant” and thus assume that it simply works. Like Travis Dane (Under Siege 2) said: Did you see the body? Assumption is the mother of all F*CK UPS!
My statement is clear, it isn’t working until you’ve seen the behavior in failure scenario yourself.

Hardware

So what hardware should we all test in the c-Class enclosure?

• Power Reduncancy, test what happens if either 3 power supplies on a single phase enclosure or 1 power line feed on a three phase enclosure fails;
• Fans, just randomly pull out some fans in the enclosure and see if this is noticed by the OA; (don’t pull out to many fans, keep it real. There’s no chance that 8 fans fail simultaneously in a 10 fan enclosure);
• Onboard Administrator Redundancy, does the second OA take over full functionality when the first one fails? (Pull it out of the sleeve to test this, there’s no power off function on this component);
• Onboard Administrator Link Loss Failover (if configured);
• Redundancy on the Interconnect Modules (I will describe this later on in detail since this also covers the Connectivity and failover behavior in hardware and OS).
• Verify that all the HP component firmwares are compatibly with eachother, see the BladeSystem Matrix

For all subjects above it’s important to verify that enclosure alerting is working via either alert mails and/or SNMP traps.

Connectivity and failover within the hardware

So now let’s look at the Virtual Connect Configuration I got in place:

In the image above you can see that only Interconnect Bay 5 and 6 have external connections. So you could conclude that these 2 Interconnect’s are the only important ones for testing a failover on hardware level. This is not true and I’ll show you why in the next picture.
The red lines indicate 10 Gb connections between all individual Interconnect Bay’s, which all tied together form the “Virtual Connect Domain”.
The horizontal lines are the X0 ports which are internally connected by the c7000 backplane. The vertical and diagonal lines are 0.5 meter CX4 cables. (Note that since IC1 and 2 and IC 5 and 6 are Flex-10 modules, they are horizontal linked with 2 links  (20 Gb) as designed by HP)

So imagine a packet coming from A (Onboard Port 2) which needs to get out at point B. By powering down Interconnect Bay 1 this packet can only find its way to B via the diagonal CX4 cable. Knowing this it’s obvious that you should test a power down of every individual Interconnect Module.

Powering down a Interconnect Module can be done from the Onboard Administrator, this isn’t a graceful shutdown and thus a good test.

Testing the powering off of IC Modules is in fact a double test, first of all you are testing the Virtual Connect Domain behavior and secondly the failovers from within the ESX (I will dive into this subject later on).
Please make sure that you’ve opened up a ping -t to different IP addresses in the enclosure (ex. to VM’s and Service Console) to get a view on packet losses and re-established connections.

Word of advice: my experiences are that the failovers mostly work fine and re-enabling an IC module causes the real problems. Please write down what exactly happens, count ping losses and report them to the network team since most of the time this is caused by network misconfiguration.
To give you an example, I had scenario’s in were the failover went fine but re-enabling the IC module caused the network to send a topology change which blocked the whole network for a small period of time.

So, after testing IC1,2,3 and 4 we reach up to IC5 and 6 which differ from the rest since they have active links to the outside.

When powering down for example IC5, Virtual Connect Manager has to failover the active link to IC6.

To verify the Shared Uplink Set (SUS) failover behavior you just take a look at the pings to the components that are using that specific SUS and you can also view Virtual Connect Manager itself:

So now we’ve tested to power down and power on all the IC modules let’s take a look at the last main subject.

Connectivity and failover within the OS

These tests all start again with powering down and powering on IC modules.  Let’s take a look at the exact steps

Powering down IC1 causes the downlink to Onboard NIC Port 1 to fail (since this is hardwired via the c7000 backplane) leaving Onboard NIC Port 1 with no connection.

Since Onboard NIC Port 1 is divided into FlexNics (as described here) these FlexNics will all fail as illustrated in the image below.
So the failure of IC1 causes vmnic0, vmnic2, vmnic4 and vmnic6 to fail from ESX.

From vSwitch perspective this looks like this:

As designed this shouldn’t cause anything in ESX to fail since all the vSwitches are still being served by the other NIC port. Just test this to be sure!
The example above applies to all the IC modules since all the NIC’s are configured to be used from within vSphere.

Next to the failover behavior of the vSwitches the following subjects also need to be tested:

• VMware VMotion to every host in the Cluster;
• Test High Availability (power down a host, or see this blog from Maish Saidel-Keesing which explains how to disable a specifc vmknic);
• In my design I also tested what happens if I powered down a complete enclosure (which in fact meants that 2 of the 4 ESX Hosts will get unavailable);
• If applicable test the RAID from the Physical Server hosting ESX.

So hopefully I gave you some helpful hints in this blog, happy testing!

Coming from mostly HP EVA environments I got used to identify a Volume/LUN by there LUN number which was a real unique identifier. I could always ”talk” LUN number and be sure that it was understood and unique.
So ever since I’ve been working with our LeftHand environment I disliked the way that every Volume/LUN is marked with LUN 0.

Goal of this blog is to show you how you can easily rename a published LeftHand Volume to something that is easy to recognize and is unique.

As you can see in the image below the SCSI target number (T) seems to be the unique identifier  instead of the LUN number.
Well, fair enough you might, just switch to T number instead of the L number.  Unfortunately the SCSI target number can vary on the ESX hosts so it isn’t a static number that uniquely identifies a LUN!

VMHBAx:C:T:L:(P)
x=SCSI Adapter Number
C= Controller
T= Target
L= LUN Number
P= Partition (if available)

The only real unique identifier in fact is a very long string, which can be compared to a WWN from FC devices.

So what’s there to dislike more? We are using many RDM’s and it takes a long time to identify the correct RDM with the LeftHand Volume.
First I needed to link either the “Identfier” or the “Target”-number to the actual target

Step 1: Select Paths and link the LeftHand volume name to the target number

Alternative to Step 1: Select Paths, right mouse-click the title bar, select Devices and link the LeftHand volume name to the Device which also contains the “Identifier” (this is a lot of scrolling work!)

Step 2: Go back to “Devices” and link the Target number to the “Identifier” (if you’ve done the alternative step above you already have the “Identifier” linked to the correct LeftHand volume)

Step 3: Add the new RDM to the VM and either identify the correct LeftHand volume by:
- Target number (if the VM is on the same ESX Host as where the ‘Target number to LeftHand volume’ is checked; Remember that the Target number isn’t a real unique identifier in the ESX Cluster!)
Or
- The long Identify string.

Now this procedure could be easy when adding 1 RDM but in my case I had situations in where I needed to add 20 RDM’s in the same time so I needed to get a pen and paper and make an overview.
Lately (too late IMHO ;) I discovered that the Volume Name can be renamed, making the new procedure as followes:

Step 1: Select Paths and link the LeftHand volume name to the target number

Step 2: Go back to Devices, select the correct Target number, right mouse click the entry and select rename. It’s advisable to keep the LeftHand volume name the same as the new VMware Name.

Step 3: Add the new RDM to the VM and easy identify now

As a last remark I like to note that this ”identify issue” doesn’t really occur while making new VMFS datastores.
Whenever walking through the wizard “Add Storage” the wizard ask you to select a Disk/LUN. Within this window (see image below) there’s a “Path ID”-property field in which you can directly see the LeftHand Volume name. This “Path ID”-property isn’t availabe in the wizard that lets you connect a RDM.

My new DroboPro arrived this week and so far I’m less than impressed.  As you know, I already have had a generation 1 Drobo for some time.  It has been a great device, doing exactly what I wanted.  After winning a second standard Drobo at Tech Field Day, I paid for the upgrade to the DroboPro in anticipation of a bigger and better machine.  Here’s what I’ve managed to achieve and determine so far.

The device itself is very different.  First of all, the DroboPro accomodates up to 8 drives and so is orientated horizontally (otherwise it would fall over).  At the back, the cabling supports Ethernet, USB and Firewire.  Unfortunately the plugs are recessed and consequently removing the Ethernet cable is a pain as you can’t easily squeeze the retaining lug that releases the cable.  The power supply is now integrated into the unit and there’s no separate PSU brick.  This could present a problem if the power supply fails, as it isn’t obvious how (or even if) the integrated power supply could be replaced.  I expect that power is now integrated to make the DroboPro more datacentre friendly.  One last comment; the device is considerably more more noisy than its smaller brother, presumably because it has to cool twice as many hard drives.

• USB and iSCSI Configuration – it isn’t clear whether the Drobo Dashboard can configure the DroboPro over either the USB connection or the iSCSI connection or whether USB connectivity needs to be retained for future configuration tasks.  Be aware – unplugging the USB cable seems to cause the Drobo to immediately reboot.
• Reduced Management – if the DroboPro is to be connected to an ESX server, then the documentation indicates the USB cable should be removed.  What’s then not clear is how the  ongoing status of the device can be managed.  For example, at this point, where should the Drobo Dashboard be run from?  The lack of a permanent management console means downtime to check the status of the device, to see the current occupancy levels and the effect of adding in additional capacity.
• Lack of detailed Information – As yet, I haven’t managed to work out where or whether there are details provided on the utilisation of each configured LUN.  The DroboPro allows for up to 16x 16TB LUNs, which is a huge volume of data (although ESX only allows iSCSI LUNs up to 2TB).  I’d like to be able to work out which of those 16 volumes consumes the most physical compared to logical space, especially with ESX where the Drobo’s intelligent file system aware technology currently doesn’t apply.
• Multi-Drobo Management.  I have more than one Drobo device now (DroboPro, DroboShare & Drobo).  The dashboard doesn’t appear to cater for multiple devices managed from the same machine.

Got this from our TAM today:

Not sure if you’ve downloaded Update 1 yet, there’s an issue that was discovered last night, please see below.

Please postpone deployment of Update 1 until further notice.

Who is affected:
Customers who are using VMware vSphere ESX 4.0 and are wanting to install update 1, via VMware Update Manager (VUM).

Problem description:
Upgrading ESX 4.0 to 4.0 U1 using VUM fails or times out. Rebooting host results in a PSOD and the host has to be re-installed.

Workaround:
Please disable any 3rd party hardware monitoring agents running on the ESX 4.0 server before applying the update.

For customers that have already run into this issue and have installed the update but have not rebooted, they must contact Support.

The 3rd party agents can be enabled again after the upgrade is completed.

ESX 4.0 U1 is no longer available for download.

VMware is investigating this issue.

For additional details, see Upgrading ESX 4.0 to 4.0 U1 using Update Manager fails or times out and rebooting the host results in a purple diagnostics screen (1016070).

An error message is reported in the VMware Infrastructure Client

‘VMware ESX Server unrecoverable error: (vcpu-0). Could not reserve memory for vmm64. Are you sure your VM specifiEs a 64 bit guest 0S?’

This error is caused, in my case inadvertently when the 64-bit version of Solaris is installed into a 32-bit defined VMware Virtual Machine. By default the Solaris 10 installation iso comes with both 32-bit and 64-bit versions. During the installation process if it detects a 64-bit compatible platform it will install the 64-bit version unprompted. This may be a problem later if your ESX servers are not all 64-bit compatible and therefore would not be able to accept a 64-bit VM migration or if you intend to install a 32-bit version of an application such as Oracle into the OS.

If 64-bit is not a problem then edit the settings of the VM when it is powered down and select the 64-bit version as the Guest Operating System. This should now work when powered up.

To revert to the 32-bit version, first configure the Guest OS as (64-bit) see above. Power on the VM in 64-bit mode and you now need to force the Solaris OS to boot in 32-bit mode.

To do this run the following command with superuser privileges from a command line prompt within the guest Solaris VM:
eeprom boot-file=kernel/unix

Shut down the Solaris VM, edit the VM settings and then reconfigure the Guest OS as Sun Solaris 10 (32-bit). Power on the VM and both the VM and OS should be running as 32-bit regardless of the host hardware.

So a while back I made a post about some new best practices for vSphere based on some new features that were available. One of the ugly points at the time was that while the new PVSCSI controller was awesome, you could only use it for additional disks.

Well I’m thankful to report that as of vSphere Update 1, you can boot from a PVSCSI controller so there shouldn’t be a necessity to add a second controller just when you want to take advantage of this new IO device.

Coincidentally, one of the other points in that article was about Thin Provisioning, but a recent whitepaper from VMware has pretty much alleviated all my fears on that front.

Looks like VMware is already preparing for the launch of Intel’s newest chip. I noticed the following in vCenter 4 Update 1’s EVC Cluster options:

Good to note this means that there is sufficient changes ahead to require a different EVC mode. That means for those of you installing Nehalem-based clusters today, if you plan on adding Westmere cores in the future, you’ll have to turn EVC on in Core i7 mode.

VMware lanserte i dag ESX 4.0 Update 1 og vCenter 4.0 Update 1.

Noen av nyhetene er:

• VMware View 4.0 support
• Windows 7 and Windows 2008 R2 support
• Enhanced Clustering Support for Microsoft Windows
• Enhanced VMware Paravirtualized SCSI Support
• Improved vNetwork Distributed Switch Performance
• Increase in vCPU per Core Limit
• Enablement of Intel Xeon Processor 3400 Series

Full ESX 4.0 U1 Release Notes
Full vCenter 4.0 U1 Release Notes

Para poder realizar pruebas y trastear un poco vamos a describir los pasos necesarios para poder crear una máquina virtual en VMware Workstation y poder ejecutar en ella VMware ESXi. Como es de suponer el rendimiento obtenido no va a ser bueno, pero esto no es lo que queremos conseguir, sino el probar el funcionamiento y ver el software. Los requerimientos para poder realizar las pruebas son:

• VMware Workstation 6.5 o superior
• VMware ESXi 3.5 o VMware ESXi 4.0

Los pasos a seguir son los siguientes, accedemos al menú para crear una nueva máquina virtual y seguimos el asistente:

En la ventana inicial seleccionamos “custom(advanced)”

Seleccionamos la versión de la máquina virtual como “Workstation 6.5″

De momento indicamos que instalaremos el Sistema Operativo más adelante

Como sistema operativo seleccionamos “Linux” y como versión, por ejemplo, “Red Hat Enterprise Linux 4 64-bit” (esta opción no es determinante a la hora de crear la máquina virtual)

Le damos un nombre a la máquina virtual, y seleccionamos la ruta donde se van a guardar los archivos

Seleccionamos el número de procesadores

Indicamos la cantidad de memoria de la máquina virtual. Si la versión de ESXi a instalar es la 4.0, tendremos que indicar un mínimo de 2048MB de memoria

Seleccionamos el tipo de conexión de la tarjeta de red

Dejamos la conexión de SCSI en la opción recomendada “LSI Logic”

Indicamos que queremos crear un nuevo disco duro

En el tipo de disco seleccionamos SCSI

Indicamos el tamaño del disco, con 1GB tenemos suficiente. Podemos marcar la opción para que se cree el archivo con el tamaño ya definido en lugar de crecer conforme lo utilizamos

Dejamos el nombre del archivo del disco por defecto

Ya hemos terminado de definir los elementos principales del disco y se nos muestra un resumen. Antes de Terminar pinchamos en el botón “Customize Hardware…”

Se nos muestran los distintos componentes de la máquina virtual

Eliminamos los siguientes componentes:

• Floppy
• USB Controller
• Sound Card

Indicamos la ruta de la ISO con la imagen de instalación de VMware ESXi

En el componente “Display” desmarcamos la opción “Accelerate 3d Graphics (DirectX 9.0c)”

En el componente “Processors” seleccionamos en el campo “Preferred Mode” la opción “Intel-VT-x or AMD-V”

Pinchamos en OK y terminamos la configuración de los componentes

Esperamos a que se cree el archivo del disco, que es lo que le costará más tiempo

Finalmente, editamos el archivo .VMX de la máquina virtual que hemos creado y añadimos la entrada “monitor_control.restrict_backdoor=”TRUE”"

Una vez creada la máquina virtual procederemos a la instalación de VMware ESXi

 La siguiente tabla compara VMware ESX / ESXi 3.5 conMicrosoft Hyper-V y el hipervisor muestra cómo la tecnología Hyper-V se presentan importantes limitaciones en todos los aspectos críticos de un hipervisor:

Traduccion Automatica desde Web de VMware.

As I have mentioned numerous times I am a big fan of The VESI and Alan Renouf’s powershell blog, well now Alan has released his powerpack for The VESI which means every VMware admin can have the power of Alan Renouf in their toolkit. Alan has taken his scripts and incorporated them in the VESI powerpack which means they can now be ran simply through the VESI GUI, saving you even more time. Due to current work commitments I haven’t had a lot of time to have a play with the powerpack yet but from the little I have seen it’s fantastic!

More information on The VESI can be found here

http://www.thevesi.org

and Alan’s powerpack can be found here

http://www.virtu-al.net/2009/11/02/virtu-al-vesi-powergui-powerpack/

If you like what you see and it starts saving you time consider a donation to Alan’s home test environment that will help us all in the long run!

http://www.virtu-al.net/likemyscripts/

I’ve written this blog as an add-on to Frank Denneman’s blog about Flex-10 which you can find over here.
Goal of this blog is to get a clear vision about the Flex-10 port mappings that HP uses to facilitate their blades with NIC’s, with the special focus towards VMware ESX/vSphere.

The discussed sample in this blog could also be used for a real production environment (in fact, it is ;)

First we start of looking at the “NIC to Interconnect“-mappings. These are pretty straight forward and should be known to all HP c-Class Administrators.
In our example we use HP BL460 G6 blades with 4 Flex-10 NIC’s (two onboard and two provided via a Dual Port Mezzanine Card)

Please note that the connections that are drawn below are hardwired connections on the Backplane of the HP c7000 Enclosure.

 (The reason that we use Mezzanine Slot 2 instead of Slot 1 is due to the fact that we have other servers in the enclosure as well that already have a connection via Mezzanine Slot 1)

So, our VMware vSphere Host is physically equipped with 4 10GB NIC’s so you would expect to see 4 vmnic’s in ESX right?….. Wrong!
The HP Virtual Connect Domain virtualizes each 10GB NIC and creates 4 FlexNics for it. After doing some math ;) we can conclude that we will get  16 vmnic’s in our ESX Host.

The image below shows us that we get 4 FlexNics per Port and how these FlexNics correspond to a vmnic from within ESX.

So in the image above we see that for example Port 1 from the Onboard Adapter is divided into 4 FlexNics: 1A, 1B, 1C and 1D.
PCI numbering (and thus the order in which the vmnic’s are numbered within ESX) is based on 1A (onboard), 2A (onboard), 1B (onboard), 2B (onboard), 1C (Onboard), 2C (Onboard) etc.

Notice that the first 8 vmnic’s are from the Onboard Card and the second 8 vmnic’s are from the Mezzanine Card.

From within the HP Virtual Connect Manager we can divide the available 10 GB speed over those 4 FlexNics, for example we can give 1A (vmnic0) 1GB, 1B (vmnic2) 7GB, 1C (vmnic4) 1GB which will leave us with 1GB to give out for 1D (vmnic6).

Since vSphere has much better iSCSI performance than ESX 3.5  did, we decided to use the full 10GB bandwidth to connect the LeftHand iSCSI storage. Technically this means that we give 1 FlexNic 10GB which leaves us with 0GB to share among the other 3 FlexNics remaining (per port).

 The image below shows how the technical design looks now:

From a Virtual Connect Manager perspective we used the following settings in the attached Server Profile (see image below)

Pleaste note that we defined all 16 NIC’s and left 6 of them “Unassigned”.

The “Unassigned”-ones are the FlexNics from Mezzanine Slot 2 which didn’t got any bandwidth assigned to them as you can see in the “Allocated Bandwidth”-column.
So for iSCSI we selected MZ2:1-A en MZ2:2-A as the 2 links with 10 GB allocated, leaving 0GB for MZ2:1-B, MZ2:2-B etc etc.

The final picture from vSwitch perspective looks like this, where we separated:

-Service Console (1GB – vSwitch0)
-VMotion (7GB – vSwitch1)
-Fault Tolerance (1GB – vSwitch2)
-VM Network’s (1GB – vSwitch3)

 And gave the full 10GB to the iSCSI Storage. (vSwitch4)

Please note that the above design contains two single point of failure’s, whenever the Onboard NIC fails my whole front-end fails (same story whenever the Mezzanine Card fails, in that case my whole storage will be lost.)
Customer constraints however kept me from doing it the way displayed in the image below (which obviously is technically the best way). In the image below we also cover hardware failure from either the Onboard or Mezzanine Card.

So now that I’ve explained the mappings from Virtual Connect (FlexNics) towards ESX (vmnics) lets take a look at the rest of the Virtual Connect Domain configuration.

There are two Shared Uplink Sets (SUS) created:

-         FRONTEND which controls the COS, VMotion, VM Networks and Fault Tolerance;
-         STORAGE which controls the physically separated iSCSI Storage LAN.

The “FRONTEND”-SUS is connected via 4 10GB connections towards two Cisco 6509. (20GB Active/20GB Passive)
The “STORAGE”-SUS is connected via 4 10GB connections towards two Cisco Nexus 5000’s (20GB Active/20GB Passive)

Word of advice: It’s recommended to use Portfast-settings on the endpoints of the Shared Uplink Set-connections. While doing failover tests we noticed that our networking department didn’t turned on Portfast as we had requested which resulted in spanning tree kicking in whenever we powered on a Virtual Connect Module.

Word of advice: Next issue we ran into where some CRC errors in the Virtual Connect Statistics (while the Cisco’s didn’t register any CRC errors). These errors disappeared when we defined the Shared Uplink Sets as 10GB static speed instead of “auto”.

Last word of advice: while implementing a technical environment like this it’s crucial to test every possible failure, from single ESX Host to all the separate components. I’ve wrote very detailed documents about it and it helped us discover a very strange technical problem which I’m currently investigating.

Update 04-12-2009: Since this post is one of my top articles I decided to write more about the extensive testing, read all about it here

For those who are interested I’ll explain the strange technical problem to close off of this blog:

Whenever a Virtual Connect Module fails, the downlinks towards ESX will fail as well (since these are hardwired via c7000 Backplane). So for example, whenever the module from Interconnect Bay 1 fails, vmnic0, 2,4 and 6 will fail as well, causing a failover to be initiated from within ESX (whenever configured correctly obviously

When the module is powered on again the vSwitch uplinks will be restored. This is the case for Interconnect Bay 1 and 2 but this isn’t the case for Interconnect Bay 5 and 6. Whenever we simulate a device failure on Interconnect Bay 5 or 6 we obviously lose our corresponding vmnic’s connections but they won’t come back online when we power on the Interconnect unit again. Currently the only way to get our connection back is to reboot the whole ESX Host. I’m currently working on firmwares as it seems that this will resolve my issue. I’ll keep you guys posted.

Problem update 02-10-2009
For the last weeks I’ve been swapping e-mails with VMware Support since updating the HP firmwares (Virtual Connect and all the other components)  didn’t solve the problem.

First of all VMware gave me an alternate Broadcom bnx2x driver which unfortunately didn’t solve the problem. Next step was to unload the bnx2x module and start it again in debugging mode so they could get some more information than just the vmkernel that gets flooded with cpu7:4215)<3>bnx2x: vmnic10 NIC Link is Down entries.

So I’ve enabled debugging mode on the bnx2x module, rebooted the Interconnect Module again and remarkably my vmnic connection got restored this time! I did the test again without the “enable debugging”-mode and my connection never gets restored. Very odd!

I passed out the new details towards VMware and currently awaiting there response. To be continued…

OK, so this is one of my pet-hates.

The subject of Network Load Balancing comes up all the time when I’m visiting clients and it’s always a cause of some pain.

James’ article on Spotting The Red Flags made me think about what triggers the alarm bells ringing in the network world.

For me, if often starts with the Switch configuration, when I see something like this:

interface GigabitEthernet0/1
description ESX Server NIC0
switchport mode access
channel-group 1 mode on

Some of you might be thinking, well, there’s nothing wrong with that. In fact you’d be right. “Technically” there’s nothing wrong with that IF the vSwitch on the ESX Server is using the right load-balancing method.

BUT, it’s a red flag, because 9 times out of 10, the vSwitch configuration is using the default ‘Route based on originating port ID’ method.

And listen closely folks, as I’m not going to say it again, this method DOES NOT REQUIRE SWTICH CONFIGURATION. (Sorry for shouting, but it really, really bugs me).

There is only one ESX vSwitch load-balancing method which requires switch-side configuration and that’s ‘Route based on IP hash’.

From page 7 of the ESX Configuration Guide:

“NOTE: IP-based teaming requires that the physical switch be configured with etherchannel. For all other options, etherchannel should be disabled.”

I get even more scared when I see this:

interface GigabitEthernet0/1
description ESX Server NIC0
switchport mode access
channel-group 1 mode [auto|desirable|active|passive]

This is because any of those channel-group modes imply the use of PAgP or LACP. Let’s put an end to this debate once and for all – “VMware ESX Server does not support dynamic PAgP or LCAP.”

These dynamic link aggregation protocol are absolutely not supported on ESX, so don’t use them on the switch-side!

If you’re an ESX Admin or a Network Admin, please get together and make sure that you’re both on the same page. Time and time again, we find that there’s a disconnect between what each team thinks the other is doing.

I would strongly encourage everyone to read Ken Clien’s series of Blogs called ‘The Great vSwitch Debate‘. Especially if you’re new to the world of VMware and ESX. Part 3 focuses on the different Load Balancing methods, so check it out.

For the Network folks, this is a great reference:

VMware Infrastructure 3 in a Cisco Network Environment

For the ESX folks, these are also handy:

ESX Server host requirements for link aggregation

Sample configuration of EtherChannel / Link aggregation with ESX 3.x and Cisco/HP switches

Have a read, build some understanding and stop getting it wrong! (Please…)