MTU Mismatch and Slow I/O

After a month or two of troubleshooting some storage issues we have been having with our NetApp system, we dug up an interesting piece of information.  When reviewing the MTU size on the host and on the NetApp, we noticed that the host was set for 1500 and the NetApp interface was set at 9000.  Whoops!

Before troubleshooting, we were seeing I/O at a rate of about 2500 IOPS to the NetApp system. However, when making the MTU change to match on both the ESXi host and the NetApp, we saw IOPS jump to close to 10,000.  Just a quick breakdown of what was happening here:

  1. The host would send data with an MTU of 1500 to the NetApp.
  2. The NetApp would retrieve the data and try to send it back at 9000
  3. It would fail from the switch stating it could only accept 1500
  4. The NetApp would then have to translate the data down to 1500

Basically, we were doubling the time it took to return the data back to the host and in turn to the guest VM.  The slow I/O was due to the translation time on the NetApp to get the proper data back to the host.  The switch interface was also set at 1500 and was rejecting the traffic.

Word to the wise: Always double check MTU settings and ensure it is the same through the entire path back to the host.  Just another one of those things to have in your back pocket when troubleshooting.

Script to Add Multiple NFS Datastores to an ESXi Host

I am sure I’m not the first admin that has needed to add an NFS datastore to multiple hosts, and usually it’s multiple datastores that are needed as well. Normally, I would go to each host, and add the storage manually via the vSphere Client.  But after doing this for quite some time, I decided I needed a better way to get this task done.  Scripting!

I have recently started to use PowerCLI to automate many of my daily tasks (Hint: more scripts to come) and decided to share the one I’ve used the most first.  The script takes an input after running, asking you to input your host name.  Inside the script, you will set the datastores you want to add to the host.  Those are the only changes that are needed when being run.  This script has saved me a great deal of man hours and some headaches.  A great advantage to scripting is that you avoid those sneaky spelling mistakes or clicking the wrong button.

The script is below here (The bold parts are what need to be changed by you):

$VMHost = (Read-Host “Please enter the name of your ESX Host you wish to configure”)

$shellObject = new-object -comobject wscript.shell
$intAnswer = $shellObject.popup(“Do you want to add the datastores?”, 0,”Add datastores – remember you must have added the hosts on the storage”,4)
If ($intAnswer -eq 6) {
Write “Creating Datastores on $VMHost…”
New-Datastore -Nfs -VMHost $VMHost -Name DatastoreName1 -Path /vol/DatastoreName1 -NfsHost 192.168.255.251
New-Datastore -Nfs -VMHost $VMHost -Name DatastoreName2 -Path /vol/DatastoreName2 -NfsHost 192.168.255.251
New-Datastore -Nfs -VMHost $VMHost -Name DatastoreName3 -Path /vol/DatastoreName3 -NfsHost 192.168.255.251
} else {
Write “Skipping Datastores on $VMHost…”
}

When running the script, you will see the following output asking you to put in your hostname:

addnfsdatastore

You will then see a successful message for each datastore added to the host. You must remember to add the host to the NFS Export share on the storage itself before completing this step.

Script courtesy of VMware PowerCLI Blog

VMware releases vCenter Server 5.1 Update 1a

vmware_logo

VMware released vCenter Server 5.1 Update 1a on May 22nd in reply to a known issue that users were seeing related to logging into vCenter.  Users that had a large number of AD groups in their environment would see an error when logging in using SSO.  This issue has been fixed in Update 1a and some other improvements were snuck in as well:

  • vCenter server can now run on Windows Server 2012
  • vCenter not supports SQL Server 2008 R2 and SQL Server 2012
  • You can now customize the following Guest OS’s: Windows 8, Windows Server 2012, RHEL 5.9 and Ubuntu 12.04
  • The vRAM limit of 192GB has been removed
  • And other bug fixes

The upgrade and installation process is the same as previous releases.  You can do an in place upgrade from 4.x and up or you can always do a new install.  And remember that all the new features within vCenter can only be accessed through the web client, so ditch the desktop client and fully utilize your vCenter.

You can read the full version of VMware’s release notes for vCenter server 5.1u1a here.