vSphere Web Client isn’t accessible after vCenter Server 6.* upgrade – error 503 Service Unavailable

vSphere Web Client isn’t accessible after vCenter Server 6. upgrade logo

In my previous post vCenter Server 6.* Upgrade I described process of upgrading vCenter Server 6.*. Right after upgrade I noticed that I can’t connect to vCenter Server using vSphere Web Client. It was a bit odd behaviour but I wanted to fix it.

Error I saw in browser was the following:

503 Service Unavailable (Failed to connect to endpoint: [class Vmacore::Http::LocalServiceSpec:00000000025053F0] _serverNamespace = /vsphere-client _isRedirect = false _port = 9090)

Here is the wirgo.log located in C:\ProgramData\VMware\vCenterServer\logs\vsphere-client

STATUS | wrapper  | 2016/03/01 12:46:28 | --< Wrapper Started as Service
STATUS | wrapper  | 2016/03/01 12:46:28 | Java Service Wrapper Professional Edition 64-bit 3.5.24
STATUS | wrapper  | 2016/03/01 12:46:28 |   Copyright (C) 1999-2014 Tanuki Software, Ltd. All Rights Reserved.
STATUS | wrapper  | 2016/03/01 12:46:28 |     http://wrapper.tanukisoftware.com
STATUS | wrapper  | 2016/03/01 12:46:28 |   Licensed to VMware Global, Inc. for VMware vSphere Web Client (custom)
STATUS | wrapper  | 2016/03/01 12:46:28 | 
INFO   | wrapper  | 2016/03/01 12:46:28 | Event Command 'wrapper_start': Command line: "C:\Program Files\VMware\vCenter Server\visl-integration\usr\sbin\cloudvm-ram-size.bat" -J vsphere-client -O "C:\ProgramData\VMware\vCenterServer\cfg/java/vsphere-client_jvm.conf"
INFO   | wrapper  | 2016/03/01 12:46:28 | Event Command 'wrapper_start': Command launched (pid: 5908), blocking for up to 3600 seconds...
INFO   | wrapper  | 2016/03/01 12:46:28 | Event Command 'wrapper_start': Command completed with exit code: 0
INFO   | wrapper  | 2016/03/01 12:46:28 | Event Command 'wrapper_start': Command completed with exit code: 0  Continuing.
STATUS | wrapper  | 2016/03/01 12:46:29 | Launching a JVM...
INFO   | jvm 1    | 2016/03/01 12:46:29 | WrapperManager: Initializing...
INFO   | jvm 1    | 2016/03/01 12:46:29 | VKS keystore loaded
INFO   | jvm 1    | 2016/03/01 12:46:29 | Deleted the old JKS file.
INFO   | jvm 1    | 2016/03/01 12:46:29 | JKS keystore populated successfully.
INFO   | jvm 1    | 2016/03/01 12:46:29 | JKS keystore written to file system successfully.
INFO   | jvm 1    | 2016/03/01 12:46:29 | Tomcat config file populated successfully.
INFO   | jvm 1    | 2016/03/01 12:46:32 | [2016-03-01T12:46:32.176+01:00] [INFO ] startup-tracker                Kernel starting. 
INFO   | jvm 1    | 2016/03/01 12:46:32 | [2016-03-01T12:46:32.554+01:00] [ERROR] kernel-dm-4                    Application context creation failure for bundle 'org.eclipse.virgo.kernel.services' version '3.6.3.RELEASE'. org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'kernelWorkArea' defined in URL [bundleentry://34.fwk649660992/META-INF/spring/io-context.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [org.eclipse.virgo.kernel.services.work.StandardWorkArea]: Constructor threw exception; nested exception is org.eclipse.virgo.util.io.FatalIOException: Unable to create directory C:\PROGRA~1\VMware\VCENTE~1\WEBCLI~1\server\work\services
INFO   | jvm 1    | 2016/03/01 12:46:32 |   at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:288)
INFO   | jvm 1    | 2016/03/01 12:46:32 |   at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1035)
INFO   | jvm 1    | 2016/03/01 12:46:32 |   at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:939)
INFO   | jvm 1    | 2016/03/01 12:46:32 |   at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:485)
INFO   | jvm 1    | 2016/03/01 12:46:32 |   at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
INFO   | jvm 1    | 2016/03/01 12:46:32 |   at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:294)
INFO   | jvm 1    | 2016/03/01 12:46:32 |   at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:225)
INFO   | jvm 1    | 2016/03/01 12:46:32 |   at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:291)
INFO   | jvm 1    | 2016/03/01 12:46:32 |   at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:193)
INFO   | jvm 1    | 2016/03/01 12:46:32 |   at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:585)
INFO   | jvm 1    | 2016/03/01 12:46:32 |   at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:913)
INFO   | jvm 1    | 2016/03/01 12:46:32 |   at org.eclipse.gemini.blueprint.context.support.AbstractDelegatedExecutionApplicationContext.access$1600(AbstractDelegatedExecutionApplicationContext.java:60)
INFO   | jvm 1    | 2016/03/01 12:46:32 |   at org.eclipse.gemini.blueprint.context.support.AbstractDelegatedExecutionApplicationContext$4.run(AbstractDelegatedExecutionApplicationContext.java:325)
INFO   | jvm 1    | 2016/03/01 12:46:32 |   at org.eclipse.gemini.blueprint.util.internal.PrivilegedUtils.executeWithCustomTCCL(PrivilegedUtils.java:85)
INFO   | jvm 1    | 2016/03/01 12:46:32 |   at org.eclipse.gemini.blueprint.context.support.AbstractDelegatedExecutionApplicationContext.completeRefresh(AbstractDelegatedExecutionApplicationContext.java:290)
INFO   | jvm 1    | 2016/03/01 12:46:32 |   at org.eclipse.gemini.blueprint.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor$CompleteRefreshTask.run(DependencyWaiterApplicationContextExecutor.java:137)
INFO   | jvm 1    | 2016/03/01 12:46:32 |   at org.eclipse.virgo.kernel.agent.dm.ContextPropagatingTaskExecutor$2.run(ContextPropagatingTaskExecutor.java:95)
INFO   | jvm 1    | 2016/03/01 12:46:32 |   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
INFO   | jvm 1    | 2016/03/01 12:46:32 |   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
INFO   | jvm 1    | 2016/03/01 12:46:32 |   at java.lang.Thread.run(Thread.java:745)
INFO   | jvm 1    | 2016/03/01 12:46:32 | Caused by: org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [org.eclipse.virgo.kernel.services.work.StandardWorkArea]: Constructor threw exception; nested exception is org.eclipse.virgo.util.io.FatalIOException: Unable to create directory C:\PROGRA~1\VMware\VCENTE~1\WEBCLI~1\server\work\services
INFO   | jvm 1    | 2016/03/01 12:46:32 |   at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:162)
INFO   | jvm 1    | 2016/03/01 12:46:32 |   at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:110)
INFO   | jvm 1    | 2016/03/01 12:46:32 |   at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:280)
INFO   | jvm 1    | 2016/03/01 12:46:32 |   ... 19 common frames omitted
INFO   | jvm 1    | 2016/03/01 12:46:32 | Caused by: org.eclipse.virgo.util.io.FatalIOException: Unable to create directory C:\PROGRA~1\VMware\VCENTE~1\WEBCLI~1\server\work\services
INFO   | jvm 1    | 2016/03/01 12:46:32 |   at org.eclipse.virgo.util.io.PathReference.createDirectory(PathReference.java:382)
INFO   | jvm 1    | 2016/03/01 12:46:32 |   at org.eclipse.virgo.kernel.services.work.StandardWorkArea.(StandardWorkArea.java:46)
INFO   | jvm 1    | 2016/03/01 12:46:32 |   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
INFO   | jvm 1    | 2016/03/01 12:46:32 |   at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
INFO   | jvm 1    | 2016/03/01 12:46:32 |   at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
INFO   | jvm 1    | 2016/03/01 12:46:32 |   at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
INFO   | jvm 1    | 2016/03/01 12:46:32 |   at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:147)
INFO   | jvm 1    | 2016/03/01 12:46:32 |   ... 21 common frames omitted

In the logs there is one entry that is very interesting.

NFO   | jvm 1    | 2016/03/01 12:46:32 | Caused by: org.eclipse.virgo.util.io.FatalIOException: Unable to create directory C:\PROGRA~1\VMware\VCENTE~1\WEBCLI~1\server\work\services

vSphere Web Client was running as NT Service\vspherewebclientsvc and it didn’t have correct permissions in C:\Program Files\VMware\vCenter Server\WebClient\server\work directory. Adding NT Service with full rights access to folder didn’t help as well. I changed it to main domain account (it could by any account with rights logon as a service and local admin rights on vCenter Server) and restarted vSphere Web Client.

STATUS | wrapper  | 2016/03/01 13:05:48 | --< Wrapper Started as Service
STATUS | wrapper  | 2016/03/01 13:05:48 | Java Service Wrapper Professional Edition 64-bit 3.5.24
STATUS | wrapper  | 2016/03/01 13:05:48 |   Copyright (C) 1999-2014 Tanuki Software, Ltd. All Rights Reserved.
STATUS | wrapper  | 2016/03/01 13:05:48 |     http://wrapper.tanukisoftware.com
STATUS | wrapper  | 2016/03/01 13:05:48 |   Licensed to VMware Global, Inc. for VMware vSphere Web Client (custom)
STATUS | wrapper  | 2016/03/01 13:05:48 | 
INFO   | wrapper  | 2016/03/01 13:05:48 | Event Command 'wrapper_start': Command line: "C:\Program Files\VMware\vCenter Server\visl-integration\usr\sbin\cloudvm-ram-size.bat" -J vsphere-client -O "C:\ProgramData\VMware\vCenterServer\cfg/java/vsphere-client_jvm.conf"
INFO   | wrapper  | 2016/03/01 13:05:48 | Event Command 'wrapper_start': Command launched (pid: 9836), blocking for up to 3600 seconds...
INFO   | wrapper  | 2016/03/01 13:05:48 | Event Command 'wrapper_start': Command completed with exit code: 0
INFO   | wrapper  | 2016/03/01 13:05:48 | Event Command 'wrapper_start': Command completed with exit code: 0  Continuing.
STATUS | wrapper  | 2016/03/01 13:05:48 | Launching a JVM...
INFO   | jvm 1    | 2016/03/01 13:05:48 | WrapperManager: Initializing...
INFO   | jvm 1    | 2016/03/01 13:05:48 | WrapperManager: Port 31,550 already in use, using port 31,551 instead.
INFO   | jvm 1    | 2016/03/01 13:05:49 | VKS keystore loaded
INFO   | jvm 1    | 2016/03/01 13:05:49 | Deleted the old JKS file.
INFO   | jvm 1    | 2016/03/01 13:05:49 | JKS keystore populated successfully.
INFO   | jvm 1    | 2016/03/01 13:05:49 | JKS keystore written to file system successfully.
INFO   | jvm 1    | 2016/03/01 13:05:49 | Tomcat config file populated successfully.
INFO   | jvm 1    | 2016/03/01 13:05:51 | [2016-03-01T13:05:51.798+01:00] [INFO ] startup-tracker               <KE0001I< Kernel starting.
STATUS | wrapperm | 2016/03/01 13:05:55 | The VMware vSphere Web Client Service is installed.
STATUS | wrapperm | 2016/03/01 13:05:55 |   Start Type: Automatic
STATUS | wrapperm | 2016/03/01 13:05:55 |   Interactive: No
STATUS | wrapperm | 2016/03/01 13:05:55 |   Running: Yes
INFO   | jvm 1    | 2016/03/01 13:05:57 | [2016-03-01T13:05:57.668+01:00] [INFO ] startup-tracker               <KE0002I< Kernel started. 
INFO   | jvm 1    | 2016/03/01 13:05:58 | [2016-03-01T13:05:58.014+01:00] [INFO ] system-artifacts              <DE0000I< Installing plan 'org.eclipse.virgo.kernel.userregion.blueprint' version '3.6.3'. 
INFO   | jvm 1    | 2016/03/01 13:05:58 | [2016-03-01T13:05:58.209+01:00] [INFO ] system-artifacts              <DE0000I< Installing bundle 'org.eclipse.gemini.blueprint.core' version '1.0.2.RELEASE'. 
INFO   | jvm 1    | 2016/03/01 13:05:58 | [2016-03-01T13:05:58.267+01:00] [INFO ] system-artifacts              <DE0000I< Installing bundle 'org.eclipse.gemini.blueprint.extender' version '1.0.2.RELEASE'. 
INFO   | jvm 1    | 2016/03/01 13:05:58 | [2016-03-01T13:05:58.283+01:00] [INFO ] system-artifacts              <DE0000I< Installing bundle 'org.eclipse.gemini.blueprint.io' version '1.0.2.RELEASE'. 
INFO   | jvm 1    | 2016/03/01 13:05:58 | [2016-03-01T13:05:58.294+01:00] [INFO ] system-artifacts              <DE0000I< Installing bundle 'org.eclipse.virgo.kernel.agent.dm' version '3.6.3.RELEASE'. 
INFO   | jvm 1    | 2016/03/01 13:05:58 | [2016-03-01T13:05:58.302+01:00] [INFO ] system-artifacts              <DE0000I< Installing bundle 'org.eclipse.virgo.kernel.deployer.dm' version '3.6.3.RELEASE'. 
INFO   | jvm 1    | 2016/03/01 13:05:58 | [2016-03-01T13:05:58.337+01:00] [INFO ] system-artifacts              <DE0000I< Installing bundle 'org.eclipse.equinox.ds' version '1.4.0.v20120112-1400'. 
INFO   | jvm 1    | 2016/03/01 13:05:58 | [2016-03-01T13:05:58.368+01:00] [INFO ] system-artifacts              <DE0000I< Installing bundle 'org.eclipse.equinox.util' version '1.0.300.v20111010-1614'. 
INFO   | jvm 1    | 2016/03/01 13:05:58 | [2016-03-01T13:05:58.373+01:00] [INFO ] system-artifacts              <DE0000I< Installing configuration 'osgi.console' version '0.0.0'. 
INFO   | jvm 1    | 2016/03/01 13:06:00 | [2016-03-01T13:06:00.284+01:00] [INFO ] system-artifacts              <DE0001I< Installed bundle 'org.eclipse.gemini.blueprint.core' version '1.0.2.RELEASE'. 
INFO   | jvm 1    | 2016/03/01 13:06:00 | [2016-03-01T13:06:00.285+01:00] [INFO ] system-artifacts              <DE0001I< Installed bundle 'org.eclipse.gemini.blueprint.extender' version '1.0.2.RELEASE'. 
INFO   | jvm 1    | 2016/03/01 13:06:00 | [2016-03-01T13:06:00.286+01:00] [INFO ] system-artifacts              <DE0001I< Installed bundle 'org.eclipse.gemini.blueprint.io' version '1.0.2.RELEASE'. 
INFO   | jvm 1    | 2016/03/01 13:06:00 | [2016-03-01T13:06:00.287+01:00] [INFO ] system-artifacts              <DE0001I< Installed bundle 'org.eclipse.virgo.kernel.agent.dm' version '3.6.3.RELEASE'. 
INFO   | jvm 1    | 2016/03/01 13:06:00 | [2016-03-01T13:06:00.293+01:00] [INFO ] system-artifacts              <DE0001I< Installed bundle 'org.eclipse.virgo.kernel.deployer.dm' version '3.6.3.RELEASE'. 
INFO   | jvm 1    | 2016/03/01 13:06:00 | [2016-03-01T13:06:00.294+01:00] [INFO ] system-artifacts              <DE0001I< Installed bundle 'org.eclipse.equinox.ds' version '1.4.0.v20120112-1400'. 
INFO   | jvm 1    | 2016/03/01 13:06:00 | [2016-03-01T13:06:00.295+01:00] [INFO ] system-artifacts              <DE0001I< Installed bundle 'org.eclipse.equinox.util' version '1.0.300.v20111010-1614'. 
INFO   | jvm 1    | 2016/03/01 13:06:00 | [2016-03-01T13:06:00.296+01:00] [INFO ] system-artifacts              <DE0001I< Installed configuration 'osgi.console' version '0.0.0'. 
INFO   | jvm 1    | 2016/03/01 13:06:00 | [2016-03-01T13:06:00.296+01:00] [INFO ] system-artifacts              <DE0001I< Installed plan 'org.eclipse.virgo.kernel.userregion.blueprint' version '3.6.3'. 
INFO   | jvm 1    | 2016/03/01 13:06:00 | [2016-03-01T13:06:00.318+01:00] [INFO ] system-artifacts              <DE0004I< Starting plan 'org.eclipse.virgo.kernel.userregion.blueprint' version '3.6.3'. 
INFO   | jvm 1    | 2016/03/01 13:06:00 | [2016-03-01T13:06:00.323+01:00] [INFO ] system-artifacts              <DE0004I< Starting bundle 'org.eclipse.gemini.blueprint.core' version '1.0.2.RELEASE'. 
INFO   | jvm 1    | 2016/03/01 13:06:00 | [2016-03-01T13:06:00.325+01:00] [INFO ] system-artifacts              <DE0004I< Starting bundle 'org.eclipse.gemini.blueprint.extender' version '1.0.2.RELEASE'. 
INFO   | jvm 1    | 2016/03/01 13:06:00 | [2016-03-01T13:06:00.586+01:00] [INFO ] start-signalling-1            <DE0005I< Started bundle 'org.eclipse.gemini.blueprint.core' version '1.0.2.RELEASE'. 
INFO   | jvm 1    | 2016/03/01 13:06:00 | [2016-03-01T13:06:00.831+01:00] [INFO ] system-artifacts              <DE0004I< Starting bundle 'org.eclipse.gemini.blueprint.io' version '1.0.2.RELEASE'. 
INFO   | jvm 1    | 2016/03/01 13:06:00 | [2016-03-01T13:06:00.833+01:00] [INFO ] system-artifacts              <DE0004I< Starting bundle 'org.eclipse.virgo.kernel.agent.dm' version '3.6.3.RELEASE'. 
INFO   | jvm 1    | 2016/03/01 13:06:00 | [2016-03-01T13:06:00.837+01:00] [INFO ] start-signalling-2            <DE0005I< Started bundle 'org.eclipse.gemini.blueprint.io' version '1.0.2.RELEASE'. 
INFO   | jvm 1    | 2016/03/01 13:06:00 | [2016-03-01T13:06:00.839+01:00] [INFO ] system-artifacts              <DE0004I< Starting bundle 'org.eclipse.virgo.kernel.deployer.dm' version '3.6.3.RELEASE'. 
INFO   | jvm 1    | 2016/03/01 13:06:00 | [2016-03-01T13:06:00.841+01:00] [INFO ] start-signalling-1            <DE0005I< Started bundle 'org.eclipse.gemini.blueprint.extender' version '1.0.2.RELEASE'. 
INFO   | jvm 1    | 2016/03/01 13:06:00 | [2016-03-01T13:06:00.842+01:00] [INFO ] start-signalling-3            <DE0005I< Started bundle 'org.eclipse.virgo.kernel.agent.dm' version '3.6.3.RELEASE'. 
INFO   | jvm 1    | 2016/03/01 13:06:00 | [2016-03-01T13:06:00.845+01:00] [INFO ] system-artifacts              <DE0004I< Starting bundle 'org.eclipse.equinox.ds' version '1.4.0.v20120112-1400'. 
INFO   | jvm 1    | 2016/03/01 13:06:01 | [2016-03-01T13:06:01.003+01:00] [INFO ] system-artifacts              <DE0004I< Starting bundle 'org.eclipse.equinox.util' version '1.0.300.v20111010-1614'. 
INFO   | jvm 1    | 2016/03/01 13:06:01 | [2016-03-01T13:06:01.009+01:00] [INFO ] start-signalling-3            <DE0005I< Started bundle 'org.eclipse.equinox.ds' version '1.4.0.v20120112-1400'. 
INFO   | jvm 1    | 2016/03/01 13:06:01 | [2016-03-01T13:06:01.029+01:00] [INFO ] system-artifacts              <DE0004I< Starting configuration 'osgi.console' version '0.0.0'. 
INFO   | jvm 1    | 2016/03/01 13:06:01 | [2016-03-01T13:06:01.035+01:00] [INFO ] start-signalling-3            <DE0005I< Started bundle 'org.eclipse.equinox.util' version '1.0.300.v20111010-1614'. 
INFO   | jvm 1    | 2016/03/01 13:06:01 | [2016-03-01T13:06:01.038+01:00] [INFO ] system-artifacts              <DE0005I< Started configuration 'osgi.console' version '0.0.0'. 
INFO   | jvm 1    | 2016/03/01 13:06:01 | [2016-03-01T13:06:01.085+01:00] [INFO ] start-signalling-3            <DE0005I< Started bundle 'org.eclipse.virgo.kernel.deployer.dm' version '3.6.3.RELEASE'. 
INFO   | jvm 1    | 2016/03/01 13:06:01 | [2016-03-01T13:06:01.086+01:00] [INFO ] start-signalling-3            <DE0005I< Started plan 'org.eclipse.virgo.kernel.userregion.blueprint' version '3.6.3'. 
INFO   | jvm 1    | 2016/03/01 13:06:01 | [2016-03-01T13:06:01.495+01:00] [INFO ] system-artifacts              <DE0000I< Installing plan 'org.eclipse.virgo.web.tomcat' version '3.6.3'. 
INFO   | jvm 1    | 2016/03/01 13:06:01 | [2016-03-01T13:06:01.499+01:00] [INFO ] system-artifacts              <DE0000I< Installing configuration 'org.eclipse.virgo.web' version '0.0.0'. 
INFO   | jvm 1    | 2016/03/01 13:06:01 | [2016-03-01T13:06:01.503+01:00] [INFO ] system-artifacts              <DE0000I< Installing bundle 'org.eclipse.virgo.web.spring.integration' version '3.6.3.RELEASE'.

Finally it started without any problems. I hope this might help someone else who might help same problem with vSphere Web Client.