When installing VMware vCenter Server 5.1, I came accross this issue after I had uninstalled an instance of vCenter running on the machine.

Symptom: When installing vCenter Server 5.1, you are presented with a dialog box saying Creation of instance VMwareVCMSDS failed: The name 'VMwareVCMSDS' is already in use as an AD LDS instance name; and are presented with options to force install or cancel.  I didn't just keep blowing through the installer as I felt it would probably bite me in the butt later.  Instead I decided to figure out what the problem was and how to make the vCenter installer happy.

As the dialog box says, you can find information inside of the jointool.log and status.txt files stored inside of your user's TEMP directory during installation (which can probably be found here: C:\Users\myusersdirectory\AppData\Local\Temp).

Here is what my log files showed:

jointool.log

[2013-01-08 10:46:53,833 com.vmware.vim.jointool] Failed to find instance config at: "C:\ProgramData\VMware\VMware VirtualCenter\instance.cfg", assuming defaults
[2013-01-08 10:46:53,833 com.vmware.vim.jointool] Storage directory not found in instance.cfg.
[2013-01-08 10:46:54,223 com.vmware.vim.jointool] Skipping DB init as we are attempting to initialize without force/recovery/upgrade.
[2013-01-08 10:46:54,223 com.vmware.vim.jointool] JoinTool started
[2013-01-08 10:46:54,223 com.vmware.vim.jointool] Storage directory for LDAP instance: C:\ProgramData\VMware\VMware VirtualCenter\\VMwareVCMSDS
[2013-01-08 10:46:54,223 com.vmware.vim.jointool] Operation Mode: initialize
[2013-01-08 10:46:54,254 com.vmware.vim.jointool] Creating directory services instance VMwareVCMSDS
[2013-01-08 10:46:54,254 com.vmware.vim.jointool] LDAP port = 389
[2013-01-08 10:46:54,254 com.vmware.vim.jointool] Base DN = dc=virtualcenter,dc=vmware,dc=int
[2013-01-08 10:46:54,254 com.vmware.vim.jointool] Storage dir = C:\ProgramData\VMware\VMware VirtualCenter\VMwareVCMSDS
[2013-01-08 10:46:56,080 com.vmware.vim.jointool] Operation "Create standalone instance VMwareVCMSDS" failed: : Action: Create Standalone Instance
Action: Creation of standalone instance
Action: Create Instance
Problem: Creation of instance VMwareVCMSDS failed: The name 'VMwareVCMSDS' is already in use as an AD LDS instance name. Type a different instance name.

 

[2013-01-08 10:46:56,080 com.vmware.vim.jointool] Recovering from failed Operation "Create standalone instance VMwareVCMSDS"

[2013-01-08 10:46:56,080 com.vmware.vim.jointool] Recovery successful

[2013-01-08 10:46:56,080 com.vmware.vim.jointool] Execution error.

status.txt

-----------------
Operation "Create standalone instance VMwareVCMSDS" failed:
Action: Create Standalone Instance
Action: Creation of standalone instance
Action: Create Instance
Problem: Creation of instance VMwareVCMSDS failed: The name 'VMwareVCMSDS' is already in use as an AD LDS instance name. Type a different instance name.

 

-----------------
Recovering from failed Operation "Create standalone instance VMwareVCMSDS"

-----------------
Recovery successful

-----------------
Execution error.

Resolution:
To fix this issue, I ended up navigating to C:\ProgramData\VMware and removing the VMware VirtualCenter folder (you could move it to a different directory to save as a backup just in case).

Now, rerun the installer and a clean copy of vCenter should install fine.

Notes: If you are prestaging your SSL certificates prior to installation, go ahead and recreate the folder structure of C:\ProgramData\VMware\VMware VirtualCenter\ssl and place your rui.crt, rui.key, and rui.pfx files in there.