Yearly Archives: 2013

Stacking with the Dell PowerConnect 5548's

This evening I had the pleasure of moving our switches from normal trunking to use the stacking ports.  By this, I mean we are now using HDMI cables to achieve 10Gbps uplinks between switches, managing one "super switch" (all of the ports are controlled by the master switch), and also provide redundancy by moving to a ring setup via the stacking ports.

How do you stack with the Dell PowerConnect 5548's?

You will connect a HDMI cable from the first switch's primary port (left HDMI port) to the second switch's secondary port (right HDMI port).  You will do this for all of your switches.  Once you have them all connected, you can optionally connect the last switch's secondary port to the first switch's primary port.  This will get you into a ring topology, which will provide some redundancy. Below you can see an image from the Dell manual using this topology.

Dell Stacking

What is the difference between a ring and stack?

From what I gather, the only difference is that in the stack you are simply daisy-chaining each of the switches together.  If one fails in the middle of your stack, you are kind of SOL.  By connecting the last switch to the first switch, you have a "ring" setup, which will provide redundancy in the event one switch fails.

What is the process to adding the switches?

When you add a switch to your stack, the new switch will automatically download the configuration from the "master" switch.  According to the manual, the best practice is to setup your master, and have the rest of your switches unplugged.  Once you have your first switch setup, connect the HDMI cable to the second switch and then power it on.  When you power on the second switch, make sure you are consoled in.  The switch will ask you to press enter to get into the menu.  Hit enter and select the stacking options.  Inside of the sub-menu, set what number the switch will be in the stack and then hit the ESC key to continue the switch booting process.  You should notice that your switch's are now being stacked.  Once that is done, I would recommend logging into the web gui of the primary switch and ensuring that the stack number of the switch remains persistant in the event of a power outage/any other disruption.

What HDMI cable did I use?

This was interesting as no one really recommended any cables to use for this.  As long as the cable was rated for 10.2Gbps or higher, it said we were good to go.  I checked with Dell to see what they sell, but apparently you can only order the stacking cables (HDMI cables) when you purchase the switch.  In-turn, I ended up going with the following HDMI cables from monoprice.com: http://www.monoprice.com/products/product.asp?c_id=102&cp_id=10240&cs_id=1024004&p_id=4963&seq=1&format=2

Where can I find more info?

Here is a link to the Dell manual for the PowerConnect 5548's.  While the stacking chapter is only a couple pages, it is definitely worth a read to understand what is going on as well as see a couple of recommended practices (the stacking info starts on page 9): ftp://ftp.dell.com/Manuals/all-products/esuprt_ser_stor_net/esuprt_powerconnect/powerconnect-5548p_Setup%20Guide_zh-cn.pdf

How do I remove an iSCSI Software Adapter in VMware?

While messing with some options in my lab, I noticed that once I added an iSCSI software Adapter, every option to remove it is greyed out.

Crazy enough, the only way to remove an iSCSI adapter in VMware is to:

  1. Right click->Properties on the adapter
  2. Click on the Configure... button
  3. Uncheck Enabled under Status in the General Properties dialog box
  4. Restart the ESXi host (yep, you literally have to restart the whole box)

Upon restart, the adapter will be removed automatically.  If anyone finds another solution, please drop a comment below and let me know 🙂

Adding a host to vCenter - Datacenter.QueryConnectionInfo Error

Symptom: When adding a host to your vCenter Datacenter, you receive the following error:

Permission to perform this operation was denied. You do not hold privilege "System > View" on folder""

Error Stack
Call "Datacenter.QueryConnectionInfo" for object "DATACENTERNAME" on vCenter Server "myhost.mydomain" failed.
VMware - Error Datacenter.QueryConnectionInfo

Resolution: Make sure you have not enabled Lockdown mode on the host. To change the setting, use the direct console user interface (DCUI), and press F2. Login with your credentials and arrow down to Configure Lockdown Mode and hit enter. Make sure this option is disabled (unchecked) when the prompt opens. With Lockdown Mode disabled, try adding the host to vCenter.

Additionally, make sure that you can fully resolve the DNS name to the actual ESXi host.  If the DNS entry does not exist or points to an invalid ESXi host, the warning will be prompted as well.

If you still have issues, you can try restarting the Management Agents using the troubleshooting menu. Use the DCUI, press F2, login with your credentials, and arrow down to Troubleshooting Options and press Enter. Arrow down to Restart Management Agents and hit enter. This will restart some of the ESXi services on the host and should allow it to connect to vCenter.

VMware vCenter Server 5.1 Install Error - "Create standalone instance VMwareVCMSDS" failed

When installing VMware vCenter Server 5.1, I came accross this issue after I had uninstalled an instance of vCenter running on the machine.

Symptom: When installing vCenter Server 5.1, you are presented with a dialog box saying Creation of instance VMwareVCMSDS failed: The name 'VMwareVCMSDS' is already in use as an AD LDS instance name; and are presented with options to force install or cancel.  I didn't just keep blowing through the installer as I felt it would probably bite me in the butt later.  Instead I decided to figure out what the problem was and how to make the vCenter installer happy.

As the dialog box says, you can find information inside of the jointool.log and status.txt files stored inside of your user's TEMP directory during installation (which can probably be found here: C:\Users\myusersdirectory\AppData\Local\Temp).

Here is what my log files showed:

jointool.log

[2013-01-08 10:46:53,833 com.vmware.vim.jointool] Failed to find instance config at: "C:\ProgramData\VMware\VMware VirtualCenter\instance.cfg", assuming defaults
[2013-01-08 10:46:53,833 com.vmware.vim.jointool] Storage directory not found in instance.cfg.
[2013-01-08 10:46:54,223 com.vmware.vim.jointool] Skipping DB init as we are attempting to initialize without force/recovery/upgrade.
[2013-01-08 10:46:54,223 com.vmware.vim.jointool] JoinTool started
[2013-01-08 10:46:54,223 com.vmware.vim.jointool] Storage directory for LDAP instance: C:\ProgramData\VMware\VMware VirtualCenter\\VMwareVCMSDS
[2013-01-08 10:46:54,223 com.vmware.vim.jointool] Operation Mode: initialize
[2013-01-08 10:46:54,254 com.vmware.vim.jointool] Creating directory services instance VMwareVCMSDS
[2013-01-08 10:46:54,254 com.vmware.vim.jointool] LDAP port = 389
[2013-01-08 10:46:54,254 com.vmware.vim.jointool] Base DN = dc=virtualcenter,dc=vmware,dc=int
[2013-01-08 10:46:54,254 com.vmware.vim.jointool] Storage dir = C:\ProgramData\VMware\VMware VirtualCenter\VMwareVCMSDS
[2013-01-08 10:46:56,080 com.vmware.vim.jointool] Operation "Create standalone instance VMwareVCMSDS" failed: : Action: Create Standalone Instance
Action: Creation of standalone instance
Action: Create Instance
Problem: Creation of instance VMwareVCMSDS failed: The name 'VMwareVCMSDS' is already in use as an AD LDS instance name. Type a different instance name.

 

[2013-01-08 10:46:56,080 com.vmware.vim.jointool] Recovering from failed Operation "Create standalone instance VMwareVCMSDS"

[2013-01-08 10:46:56,080 com.vmware.vim.jointool] Recovery successful

[2013-01-08 10:46:56,080 com.vmware.vim.jointool] Execution error.

status.txt

-----------------
Operation "Create standalone instance VMwareVCMSDS" failed:
Action: Create Standalone Instance
Action: Creation of standalone instance
Action: Create Instance
Problem: Creation of instance VMwareVCMSDS failed: The name 'VMwareVCMSDS' is already in use as an AD LDS instance name. Type a different instance name.

 

-----------------
Recovering from failed Operation "Create standalone instance VMwareVCMSDS"

-----------------
Recovery successful

-----------------
Execution error.

Resolution:
To fix this issue, I ended up navigating to C:\ProgramData\VMware and removing the VMware VirtualCenter folder (you could move it to a different directory to save as a backup just in case).

Now, rerun the installer and a clean copy of vCenter should install fine.

Notes: If you are prestaging your SSL certificates prior to installation, go ahead and recreate the folder structure of C:\ProgramData\VMware\VMware VirtualCenter\ssl and place your rui.crt, rui.key, and rui.pfx files in there.

How do I setup msSQL to use SSL?

Want to encrypt your msSQL traffic?  Here is how to do it.

  1. Request/Install a certificate in the Windows Certificate store
    1. If you are on a domain with a certificate authority, you can do this by clicking Start->run->mmc
    2. Click File->Add/Remove Snap-in
    3. Select Certificates, click the Add button, select Computer account, click OK, click Finish the wizard.
    4. Expand Certificates (Local Computer) and navigate to Personal->Certificates
    5. Right click All Tasks -> Import... or Request New Certificate (depending on what you want to do)
  2. Once you have finished installing the certificate, click Start->All Programs->Microsoft SQL Server 2008 R2->Configuration Tools->SQL Server Configuration Manager (Launch SQL Server Configuration Manager)
  3. Expand SQL Server Network Configuration
  4. Right click on "Protocols for MSSQLSERVER" (or whatever your instance name is on the left side) and click Properties
  5. On the Flags tab, you can optionally set "Force Encyrption" to Yes, which will make your msSQL server only allow connections that are secure.  You may skip this step if you don't want to do this.
  6. Click on the Certificate tab.
  7. Select your certificate that you installed in Step 1 in the Certificate dropdown box.
  8. Click OK
  9. Click on SQL Server Services
  10. Right click on the SQL Server (MSSQLSERVER) service and click Restart (MSSQLSERVER==your instance name)

That's all that's to it.  Note, if you receive an error that the service cannot run (I forgot what the original error was), try disabling the VIA Client Protocol underneath SQL Native Client 10.0 Configuration inside of SQL Server Configuration Manager.

As a reference, you can find the official Microsoft KB article on how to do this here.

vCenter 5.1 - vSphere Web Client - Log Browser - Unauthorized access ' faultDetail:'null' Error

Symptom: When you click on the Log Browser link from the vSphere Web Client, you receive the following error:

faultCode:Server.Processing faultString:'javax.servlet.ServletException : java.lang.Exception: https://MYVCENTER.MYDOMAIN.local:12443/vmwb/logbrowser: Unauthorized access ' faultDetail:'null'

Resolution: You need to replace the SSL certificate for the Log Browser service with a valid one. Assuming you have applied a valid certificate on your SSO instance, Web Client, and have done the necessary steps to generate the rui.pfx, rui.key, and rui.crt files, here are the steps to apply the certificate:

  1. Stop the VMware Log Browser service
  2. Navigate to C:\Program Files\VMware\Infrastructure\vSphereWebClient\logbrowser\conf via Windows Explorer
  3. Backup the rui.crt, rui.key, rui.pfx to your Desktop (or some other folder incase we have to roll-back)
  4. Replace the rui.crt, rui.key, rui.pfx files with the ones you have created.
  5. Next, we need to create a new java keystore with the chain trust for our certificate
    1. Open up an elevated command prompt
    2. Change directories to the VMware JRE
      1. cd /d C:\Program Files\VMware\Infrastructure\jre\bin
    3. Generate the new keystore (Do not change the testpassword or changeit password) (Change the Paths to point to your pfx certificate and the destination path to output the java Keystore)
      1. keytool -v -importkeystore -srckeystore C:\PATHTOYOURSSOCERTPFXFILE\rui.pfx -srcstoretype pkcs12 -srcstorepass testpassword -srcalias rui -destkeystore C:\OUTPUTPATHYOUKNOWMAYBEYOURDESKTOP\rui.jks -deststoretype JKS -deststorepass changeit -destkeypass changeit
    4. Copy the C:\OUTPUTPATHYOUKNOWMAYBEYOURDESKTOP\rui.jks to:
      1.  C:\Program Files\VMware\Infrastructure\SSOServer\Security\
    5. Login to your vSphere Web Client with the admin@System-domain account
    6. Navigate to Administration > Sign-On and Discovery > Configuration
    7. Click on the STS Certificate tab and the click Edit
    8. Select the rui.jks file from C:\Program Files\VMware\Infrastructure\SSOServer\Security\
    9. When prompted for a keystore password enter changeit
    10. Click on the rui line to highlight it, then click OK
    11. Enter changeit again for the password
    12. Acknowledge the dialog box that says you need to restart the server in order for the changes to take effect.
      1. Note, if you receive the following error below, that means you have not fully established a certificate trust.  If you have an intermediate certificate, you will need to put all of those certificates inside of your pfx file (see my notes at the end).
        1. The last operation failed for the entity with the following error message.An error ocurred while updating server configuration

    13. Reboot your server
  6. Log back into your vSphere Web Client
  7.  Click on the Log Browser link and verify the error has been resolved

----Notes for people with tiered PKI environments----
If you have intermediate certificates, when you generate the pfx file you HAVE to import the entire chain into the pfx, otherwise the certificate will not be imported when using the vSphere Web Client.  To include the chain in your pfx file, use the following command:

openssl.exe pkcs12 -export -in C:\PATHTOMYSSOCERT\rui.crt -inkey C:\PATHTOMYSSOPRIVATEKEY\rui.key -name "rui" -chain -CAfile C:\PATHTOCACHAIN\certs.pem -passout pass:testpassword -out C:\PATHTOOUTPUTPFXFILE\rui.pfx

Notice the certs.pem file I have created.  This includes the public keys from each intermediate cert, and then followed by the root certificate.  I.e. Your file will look something like this:

-----BEGIN CERTIFICATE-----
INTERMEDIATECERTIFICATEBASE64STUFFHERE.crt
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
ROOTCERTIFICATEBASE64STUFFHERE.crt
-----END CERTIFICATE-----

Where the first BEGIN CERTIFICATE would be your intermediate certificate, and the second certificate your root certificate.  If you have multiple intermediate certificates, always put them before the root.  The root certificate should be the last in the pem file.

The official VMware KB article on how to do this can be found here (note, it does not cover the pfx file creation for tiered PKI environments): http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2037927

VMware vCenter 5.1 - Preappend domain name

After upgrading from vCenter 5.0 to 5.1 (and installing the new Single-sign On service), you may have noticed that you now have to use the DOMAIN\Username format when logging into your vCenter instance. Personally, I find this very agrevating; luckily, there is an easy way to have vCenter just figure out what domain you are on.

In order to get this job done, this guide assumes you have the vCenter Web Client installed (I don't believe this can be done from the vSphere Client for Windows -- if there is a way, please leave a comment below with the instructions, so I can add them here).

  1. Navigate to your vCenter Web Client
  2. Login using your admin@System-Domain account
  3. On the left side, click on Administration
  4. Underneath Sign-On and Discovery, click on Configuration
  5. Click on the Add Identity button underneath the identity sources tab (it looks like a green + sign).
  6. Enter in your domain info if it isn't already there, test the connection, and click OK.
  7. Select your domain from the Identity Sources list and click on the Add to default domains button (Looks like a globe with an arrow on it)
  8. Click the Save button in the Default Domains list below the Identity Sources list.

That should do it.  Try logging in without the domain appended and it should be good to go 🙂

VMware VirtualCenter Server Service - service-specific error code 2

If you receive the following error:

Windows could not start the VMware VirtualCenter Server on Local Computer. For more information, review the System Event Log. If this is a non-Microsoft service, contact the service vendor, and refer to service-specific error code 2.

Here is how to figure out what is going on (as you probably found out, event viewer is useless giving you the following info:

The description for Event ID 1000 from source VMware VirtualCenter Server cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

If the event originated on another computer, the display information had to be saved with the event.

The following information was included with the event:

Failed to intialize VMware VirtualCenter. Shutting down...

the message resource is present but the message is not found in the string/message table

To figure out what is going on, open up the vpxd-1.log file (mine was in the following directory: C:\ProgramData\VMware\VMware VirtualCenter\Logs)

Towards the bottom, you should see the error. In my case, I found the following error:

2013-01-05T13:23:54.267-06:00 [01912 error 'Default'] SSLStreamImpl::DoClientHandshake (000000000a01aec0) SSL_connect failed. Dumping SSL error queue:
2013-01-05T13:23:54.267-06:00 [01912 error 'Default'] [0] error:0407006A:rsa routines:RSA_padding_check_PKCS1_type_1:block type is not 01
2013-01-05T13:23:54.267-06:00 [01912 error 'Default'] [1] error:04067072:rsa routines:RSA_EAY_PUBLIC_DECRYPT:padding check failed
2013-01-05T13:23:54.267-06:00 [01912 error 'Default'] [2] error:0D0C5006:asn1 encoding routines:ASN1_item_verify:EVP lib
2013-01-05T13:23:54.267-06:00 [01912 error 'Default'] [3] error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
2013-01-05T13:23:54.267-06:00 [01912 error 'HttpConnectionPool-000001'] [ConnectComplete] Connect failed to ; cnx: (null), error: class Vmacore::Ssl::SSLVerifyException(SSL Exception: Verification parameters:
--> PeerThumbprint: 45:9E:FA:36:44:E1:51:9D:03:D6:FA:A3:56:DE:01:91:0D:93:78:C8
--> ExpectedThumbprint:
--> ExpectedPeerName: MYHOST.MYDOMAIN.local
--> The remote host certificate has these problems:
-->
--> * certificate signature failure)
2013-01-05T13:23:54.267-06:00 [01152 error '[SSO][SsoFactory_CreateFacade]'] Unable to create SSO facade: SSL Exception: Verification parameters:
--> PeerThumbprint: 45:9E:FA:36:44:E1:51:9D:03:D6:FA:A3:56:DE:01:91:0D:93:78:C8
--> ExpectedThumbprint:
--> ExpectedPeerName: MYHOST.MYDOMAIN.local
--> The remote host certificate has these problems:
-->
--> * certificate signature failure.
2013-01-05T13:23:54.267-06:00 [01152 error 'vpxdvpxdMain'] [Vpxd::ServerApp::Init] Init failed: Vpx::Common::Sso::SsoFactory_CreateFacade(sslContext, ssoFacadeConstPtr)
--> Backtrace:
--> backtrace[00] rip 000000018018a8ca
--> backtrace[01] rip 0000000180102f28
--> backtrace[02] rip 000000018010423e
--> backtrace[03] rip 000000018008e00b
--> backtrace[04] rip 0000000000405c2c
--> backtrace[05] rip 0000000000426512
--> backtrace[06] rip 000000013ffd0701
--> backtrace[07] rip 000000013ffca51c
--> backtrace[08] rip 00000001401ec92b
--> backtrace[09] rip 000007fefefba82d
--> backtrace[10] rip 000000007765652d
--> backtrace[11] rip 000000007788c521
-->
2013-01-05T13:23:54.267-06:00 [01152 warning 'VpxProfiler'] ServerApp::Init [TotalTime] took 6505 ms
2013-01-05T13:23:54.267-06:00 [01152 error 'Default'] Failed to intialize VMware VirtualCenter. Shutting down...
2013-01-05T13:23:54.267-06:00 [01152 info 'vpxdvpxdSupportManager'] Wrote uptime information
2013-01-05T13:23:59.774-06:00 [01312 warning 'VpxProfiler' opID=SWI-184910ec] VpxUtil_InvokeWithOpId [TotalTime] took 12012 ms
2013-01-05T13:24:11.786-06:00 [01312 warning 'VpxProfiler' opID=SWI-82674fff] VpxUtil_InvokeWithOpId [TotalTime] took 12012 ms
2013-01-05T13:24:23.799-06:00 [01312 warning 'VpxProfiler' opID=SWI-699faf4] VpxUtil_InvokeWithOpId [TotalTime] took 12012 ms
2013-01-05T13:24:35.811-06:00 [01312 warning 'VpxProfiler' opID=SWI-6e861832] VpxUtil_InvokeWithOpId [TotalTime] took 12012 ms
2013-01-05T13:24:47.823-06:00 [01312 warning 'VpxProfiler' opID=SWI-f1537246] VpxUtil_InvokeWithOpId [TotalTime] took 12012 ms
2013-01-05T13:24:48.057-06:00 [01152 info 'Default'] Forcing shutdown of VMware VirtualCenter now

As you can see from the error above, we are having some SSL issues. To fix this issue, you need to replace the SSL certificate for the SSO service with a trusted certificate (either by your own internal CA or an external one).

How do I install vCenter 5.1

Check out Derek Seaman's multi-step blog post. It is the most up-to-date guide with tons of information to get you going. It also refer's to many of the installation issues that were found in previous versions of vCenter. As I write this, the blog has already begun making notes for 5.1a instead of 5.1 GA. At this time, 5.1B is out (which I would recommend you install), but you should be able to get through the installation just fine.

http://derek858.blogspot.com/2012/09/vmware-vcenter-51-installation-part-1.html

And as a reference, I would recommend looking over the release notes for 5.1  There is a plethora of info you may want to know as a heads up, prior to the installation.

https://www.vmware.com/support/vsphere5/doc/vsphere-vcenter-server-510b-release-notes.html

https://www.vmware.com/support/vsphere5/doc/vsphere-vcenter-server-510a-release-notes.html

 

Good luck!