Unable to remove a datastore from vCenter Server Inventory

I recently had an issue where I was unable to remove a datastore from the vCenter Server Inventory.  The datastore was grayed out and when right-clicking, had no options.  After some digging and some research in SQL, I found a way to manually do this in the vCenter database.  Every datastore is given a unique ID and can be found and removed inside of the database.

Warning: Always make a SQL backup before attempting any manual database changes.  You never know when things might break and you need to restore.

So here we go:

  1. Stop the vCenter Server Service
  2. Open SQL Management Studio
  3. Run the following against your vCenter Server database (This will give you the datastore ID):

select ID from VPX_ENTITY where name = ‘datastore_name’

  1. Now we have the ID and can remove it from the database
  2. Run the following 3 queries individually (Using the ID we got from the previous query):

delete from VPX_DS_ASSIGNMENT where DS_ID=ID;
delete from VPX_VM_DS_SPACE where DS_ID=
ID;
delete from VPX_DATASTORE where ID=
ID;

  1. Finally, run the following:

delete from VPX_ENTITY where ID=ID;

If you want to verify that everything went correctly, you can run the following:

select * from VPX_DS_ASSIGNMENT where DS_ID=ID;
select * from VPX_VM_DS_SPACE where DS_ID=ID;
select * from VPX_DATASTORE where ID=ID;
select * from VPX_ENTITY where ID=ID;

Now you’ve removed the datastore from the database and can start the vCenter Server Service again. If you don’t see that it has been removed, a reboot may help. I rebooted my server just to be on the safe side.

You can check out this VMware KB for more info.

ESXi Hosts Disconnecting Randomly

A recent issue we experienced was seeing hosts disconnecting from vCenter and reconnecting.  The host would drop and randomly come back for about an hour or more.  The VM’s never saw any issues nor was there any type of outage.  It was that vCenter could no longer see the host.

After quite a bit of troubleshooting, I started digging around in the vCenter Server Settings (Administration > vCenter Server Settings).  In this menu, there is a tab for Runtime settings.  I noticed that we only had the vCenter Server Name filled in and not the vCenter Server Managed IP. The window looks as follows:

vCenter Runtime SettingsAfter completing all the fields in this window, the hosts magically all reconnected and have not dropped again.  This is due to the fact that the hosts use these settings to check in with the vCenter box and they let the host know who it’s being managed by.  As you can guess, if the host doesn’t know who’s managing it, it doesn’t know who to check in with.

The more curious issue was that this field hadn’t even been filled out, but didn’t start immediately.  Which made troubleshooting more difficult and made us all panic as we started getting numerous alerts for hosts dropping.

As best practice, whether you only have 1 vCenter server, is to fill out all these fields and enure they are correct.  Especially if you want the host to check in with the correct vCenter server and you don’t want the heart attack of seeing numerous hosts suddenly disconnecting from vCenter.

Datastore not visible after upgrading to ESXi 5

After upgrading my dev datacenter and rebooting the first ESXi 5 host, I realized that one of my fiber datastores was missing.  The path to the datastore was still visible to the host under the HBA, but it was not showing as an available datastore in the storage view.  Upon investigation, the datastore had been tagged as a snapshot datastore and was not mounting properly to the host.  This can be found by running the following:

esxcli storage vmfs snapshot list

You will see an output similar to:

<UDID>

   Volume Name: <VOLUME_NAME>

   VMFS UUID: <UDID>

   Can mount: true

   Reason for un-mountability:

   Can resignature: false

   Reason for non-resignaturability: the volume is being actively used

   Unresolved Extent Count: 2

Next, I had to force mount the datastore in CLI by first changing to “/var/log” and running:

esxcli storage vmfs snapshot mount -u <UUID> -l <VOLUME_NAME>

The command will be persistent across reboots.  If you would like to make it non-persistent then you will need to add “-n” to your command.  Once it is run, check your host and the datastore should be showing as an available datastore again.  No reboot needed and the change takes affect immediately.

You can also mount the datastore using the vSphere client as well by following the below steps:

  1. Go to your host in question
  2. On the storage tab, click add storage
  3. Choose disk/LUN
  4. Find the LUN that is missing. If it is not shown, you will need to use the above steps to mount using CLI
  5. Under mount options, choose “Keep Existing Signature” to mount persistent across reboots
  6. Click through to finish

There are a few caveats to force mounting a datastore though.  The datastore can only be mounted if it doesn’t already exist with a unique UDID.  If you choose to use the client to force mount the datastore, it cannot be mounted to other hosts in the same datacenter.  You will need to use the CLI steps posted above to mount to other hosts.

For more information about this issue and steps to fix in ESX/ESXi 4 and 3.5, you can find the VMware KB here.

VMware releases vCenter Server 5.1 Update 1a

vmware_logo

VMware released vCenter Server 5.1 Update 1a on May 22nd in reply to a known issue that users were seeing related to logging into vCenter.  Users that had a large number of AD groups in their environment would see an error when logging in using SSO.  This issue has been fixed in Update 1a and some other improvements were snuck in as well:

  • vCenter server can now run on Windows Server 2012
  • vCenter not supports SQL Server 2008 R2 and SQL Server 2012
  • You can now customize the following Guest OS’s: Windows 8, Windows Server 2012, RHEL 5.9 and Ubuntu 12.04
  • The vRAM limit of 192GB has been removed
  • And other bug fixes

The upgrade and installation process is the same as previous releases.  You can do an in place upgrade from 4.x and up or you can always do a new install.  And remember that all the new features within vCenter can only be accessed through the web client, so ditch the desktop client and fully utilize your vCenter.

You can read the full version of VMware’s release notes for vCenter server 5.1u1a here.