Weathervane, a benchmarking tool for…

Weathervane, a benchmarking tool for virtualized infrastructure and clouds – now open source!

Weathervane, a benchmarking tool for…

Weathervane is a performance benchmarking tool developed at VMware. It lets you assess the performance of your virtualized or cloud environment by driving a load against a realistic application and capturing relevant performance metrics. You might use it to compare the performance characteristics of two different environments, or to understand the performance impact of some change in an existing environment.

VMware Social Media Advocacy

VM Delta Migration Procedure

A while back we were charged with moving VMs to a new data center while also keeping downtime to a minimum.  My team and I came up with a VM Delta Migration process to move a delta of the VM (basically the snapshot) so that we could keep the downtime short.  The basic process was to take a snapshot, copy the VM to external media, and power it on.  Then that media was shipped to the new DC to import.  Once imported and ready, we shut down the VM again, SFTP the snapshot files, imported those into the new VM folder and powered on the VM.  Once the VM was powered on and verified working, we were able to remove the snapshot.  I’ve documented the process below for anyone that may be wanting to do something similar.

This article details the steps taken to perform the migration of a large VM in multiple parts – Part 1 is a bulk data copy, sent via physical media for large files. Part 2 is an incremental copy, to allow us to keep the VM available during this window. When the VM is imported at its new home, both parts should be combined.

Step 1:

Power off the VM, and create a snapshot.

Create Snapshot

Step 2:

Browse to the datastore that the VM is located in, and copy all files in the folder to the bulk storage destination. – Delete the VMWare.log files from the destination.

Browse Datastore

Step 3:

Power the VM back on, and ship the physical media over to the new location.

Step 4:

Once the media has been received, power the VM off again, and copy the following files over to the SFTP server:

  • The VMX file
  • The NVRAM file
  • The 000001.vmdk – Snapshot file
  • The –delta.vmdk – Snapshot deltas

Step 5:

At the new data center, copy the files from step 4 to the physical media from step 2. Overwrite any files that are duplicates.

Step 6:

Add all files from the physical media to a datastore, and import the VM using “Add to Inventory” on the .VMX file.

Step 7:

Power the VM online, and once everything is confirmed working, delete the snapshot.


I hope this helps anyone else needing a process to perform a migration of VMs between data centers while keeping downtime to a minimum.

First vSphere Client (HTML5) update in vSphere…

First vSphere Client (HTML5) update in vSphere 6.5.0b! [VMware vSphere Blog]

First vSphere Client (HTML5) update in vSphere…

With the release of vSphere 6.5.0b, we are proud to announce the first update to the vSphere Client!

VMware Social Media Advocacy

Released: vCenter and ESXi 6.0 Update 3 –…

Released: vCenter and ESXi 6.0 Update 3 – What’s in It for Service Providers — via VIRTUALIZATION IS LIFE!

Released: vCenter and ESXi 6.0 Update 3 –…

Last month I wrote a blog post on upgrading vCenter 5.5 to 6.0 Update 2 and during the course of writing that blog post I conducted a survey on which version of vSphere most people where seeing out in the wild…overwhelmingly vSphere 6.0 was the most popular version with 5.5 second and 6.5 lagging in adoption for the moment.

VMware Social Media Advocacy

Disable the “This host currently has no management network redundancy” message

Let’s go over how to disable the “This host currently has no management network redundancy” message.  It’s annoying and we can get rid of the yellow triangles that show on the hosts due to this message.  And I know, you “should” have redundancy on your management network but we’re just not worried about it.  Our hosts are in our building and not at a co-lo so we have constant access to them in the event something happens and we need access.

Management Network Redundancy WarningSince we don’t care about this warning, I wanted to hide it.  This way we can see if there are actual errors on the host and not some warning about network redundancy.  The fix is done with an advanced option in the cluster properties. In the cluster properties, under vSphere HA, select Advanced Options.  Then add an option named das.ignoreRedundantNetWarning and set the Value to true.

ignoreRedundantNetWarningAnd that’s it! Once the option is in, go to each host and reconfigure for vSphere HA.  The warning will then disappear and your vCenter will look clean again.

SRM 5.8 Plugin does not display in the vSphere Web Client

Today when logging into the vSphere Web Client to document the SRM testing process, I noticed that the SRM plugin did not show on the home screen.  However, when logging into the protected site I noticed that it was there.  Here are my troubleshooting steps:

1. Logged into the SRM server and noticed the service was not running.  I started the service and tried logging into the web client, but the plugin was still not showing.

2. I then rebooted the SRM box and logged back into the web client.  Still no plugin.

3. I restarted the vCenter service and web managementservices on the vCenter box. Still no plugin.

4. Finally, I restarted the web client service on the vcenter box. Logged into the web client, and voila! Plugin was showing.

The root cause is that the SRM service must be running.  If it is not, start the service and then restart the web client service on the vCenter server.

Update Manager Error: sysimage.fault.SSLCertificateError

I had to redeploy a vCenter Server Appliance recently and got an error when opening vCenter: sysimage.fault.SSLCertificateError.

sysimage.fault.SSLCertificateErrorThis is caused by the certificate on the vCenter Server changing.  This causes the Upgrade Manager needing to be re-registered with the vCenter Server.  Luckily, VMware provides a utility to do just that.  Go to your vCenter Upgrade Manager Server (The appliance does not include this, so it will typically be installed on a separate Windows Server).

Go to “C:\Program Files (x86)\VMware\Infrastructure\Update Manager” and open the file “VMwareUpdateManagerUtility”.

VMwareUpdateManagerUtilityNext, enter the credentials for your vCenter Server and hit Login.

UtilityLoginOnce the window opens, you’ll want to click “Re-register to vCenter Server”. This brings up another login screen.  You can use the same credentials you did to open the utility here as well.

Re-register to vCenter ServerFinally click apply and you will receive a notification that you need to restart the VMware vSphere Update Manager in order for the settings to take affect.

RestartUpdateManagerServiceRestart the service and open the vSphere Client again.  The error will be gone and you’ll be able to use Update Manager again.


Lost Path Redundancy to Storage Device

After installing 3 new hosts, I kept getting errors for Storage Connectivity stating “Lost path redundancy to storage device naa…….”.  We had 2 fibre cards and one of the paths was being marked as down.  I spent a couple weeks troubleshooting and trying different path selection techniques.  Still, we would randomly get alerts that the redundant path has gone down.  The only fix was to reboot the host, as not even a rescan would bring the path back up.

So after some trial and error, I found a solution.  The RCA isn’t necessarily complete yet, but I believe it was a problem with the fibre switch having an outdated firmware and us using new fibre cards in our hosts.  When using the path selection of Fixed, it would randomly pick an hba to use for each datastore.  Some datastores would use path 2 and some would use path 4.

The solution I came up with was to manually set the preferred path on each datastore (we have about 40, so it was no easy task).  You go into your host configuration, choose storage, pick a datastore and go into properties.  Inside this window, select manage paths from the bottom right and you should see your HBA’s listed.  There is a column marked Preferred with an asterisk showing which hba to prefer for the datastore (see the image below).  I went through and manually set the preferred path to be hba2 instead of letting vmware pick the path. The path selection is persistent across reboot as well when setting it manually.

storage path selectionSince manually setting the preferred path, the hosts have been stable and we have not gotten any more errors about path redundancy.  This is pretty much a band aid fix but at least we are not rebooting hosts 2-3 times per week.

ESXi Hosts Disconnecting Randomly

A recent issue we experienced was seeing hosts disconnecting from vCenter and reconnecting.  The host would drop and randomly come back for about an hour or more.  The VM’s never saw any issues nor was there any type of outage.  It was that vCenter could no longer see the host.

After quite a bit of troubleshooting, I started digging around in the vCenter Server Settings (Administration > vCenter Server Settings).  In this menu, there is a tab for Runtime settings.  I noticed that we only had the vCenter Server Name filled in and not the vCenter Server Managed IP. The window looks as follows:

vCenter Runtime SettingsAfter completing all the fields in this window, the hosts magically all reconnected and have not dropped again.  This is due to the fact that the hosts use these settings to check in with the vCenter box and they let the host know who it’s being managed by.  As you can guess, if the host doesn’t know who’s managing it, it doesn’t know who to check in with.

The more curious issue was that this field hadn’t even been filled out, but didn’t start immediately.  Which made troubleshooting more difficult and made us all panic as we started getting numerous alerts for hosts dropping.

As best practice, whether you only have 1 vCenter server, is to fill out all these fields and enure they are correct.  Especially if you want the host to check in with the correct vCenter server and you don’t want the heart attack of seeing numerous hosts suddenly disconnecting from vCenter.