Hi all
I have a problem with EAM. We have been using NSX guest introspection and it started playing up recently, which attracted our attention.
In vSphere Web Client, under Administration\vSphere ESX Agent Manager \ Solution I get the following error:
503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http16LocalServiceSpecE:0x7fdc140fea00] _serverNamespace = /eam _isRedirect = false _port = 15005)
On VCSA, the VMware-eam service is not running. Starting it results in an error, the details of which are found in /var/log/VMware-eam/wrapper.log
STATUS | wrapper | 2017/09/06 15:45:55 | --> Wrapper Started as Daemon
STATUS | wrapper | 2017/09/06 15:45:55 | Java Service Wrapper Professional Edition 64-bit 3.5.24
STATUS | wrapper | 2017/09/06 15:45:55 | Copyright (C) 1999-2014 Tanuki Software, Ltd. All Rights Reserved.
STATUS | wrapper | 2017/09/06 15:45:55 | http://wrapper.tanukisoftware.com
STATUS | wrapper | 2017/09/06 15:45:55 | Licensed to VMware Global, Inc. for VMware vCenter ESX Agent Manager
STATUS | wrapper | 2017/09/06 15:45:55 |
INFO | wrapper | 2017/09/06 15:45:55 | Event Command 'wrapper_start': Command line[0] : /usr/sbin/cloudvm-ram-size
INFO | wrapper | 2017/09/06 15:45:55 | Event Command 'wrapper_start': Command line[1] : -J
INFO | wrapper | 2017/09/06 15:45:55 | Event Command 'wrapper_start': Command line[2] : vmware-eam
INFO | wrapper | 2017/09/06 15:45:55 | Event Command 'wrapper_start': Command line[3] : -O
INFO | wrapper | 2017/09/06 15:45:55 | Event Command 'wrapper_start': Command line[4] : /etc/vmware/java/vmware-eam_jvm.conf
INFO | wrapper | 2017/09/06 15:45:55 | Event Command 'wrapper_start': Command launched (pid: 23310), blocking for up to 3600 seconds...
INFO | wrapper | 2017/09/06 15:45:56 | Event Command 'wrapper_start': Command completed with exit code: 0
INFO | wrapper | 2017/09/06 15:45:56 | Event Command 'wrapper_start': Command completed with exit code: 0 Continuing.
INFO | wrapper | 2017/09/06 15:45:56 | Event Command 'jvm_prelaunch': Command line[0] : /opt/vmware/bin/python
INFO | wrapper | 2017/09/06 15:45:56 | Event Command 'jvm_prelaunch': Command line[1] : ../wrapper/bin/update_pnid.py
INFO | wrapper | 2017/09/06 15:45:56 | Event Command 'jvm_prelaunch': Command line[2] :
INFO | wrapper | 2017/09/06 15:45:56 | Event Command 'jvm_prelaunch': Command line[3] :
INFO | wrapper | 2017/09/06 15:45:56 | Event Command 'jvm_prelaunch': Command line[4] :
INFO | wrapper | 2017/09/06 15:45:56 | Event Command 'jvm_prelaunch': Command launched (pid: 23313), blocking for up to 15 seconds...
INFO | wrapper | 2017/09/06 15:45:56 | Event Command 'jvm_prelaunch': Command completed with exit code: 1
INFO | wrapper | 2017/09/06 15:45:56 | Event Command 'jvm_prelaunch': Command completed with exit code: 1 Continuing.
STATUS | wrapper | 2017/09/06 15:45:56 | Launching a JVM...
INFO | jvm 1 | 2017/09/06 15:45:56 | Picked up JAVA_TOOL_OPTIONS: -Xms32M -Xmx128M
INFO | jvm 1 | 2017/09/06 15:45:56 | WrapperManager: Initializing...
INFO | jvm 1 | 2017/09/06 15:45:57 | WrapperSimpleApp:
INFO | jvm 1 | 2017/09/06 15:45:57 | WrapperSimpleApp Error: Encountered an error running main:
INFO | jvm 1 | 2017/09/06 15:45:57 | WrapperSimpleApp Error: org.springframework.beans.factory.BeanDefinitionStoreException: Invalid bean definition with name 'esxAgentMgr' defined in class path re source [eam-server-beans.xml]: Could not resolve placeholder 'eam.hostd_restart_timeout'
INFO | jvm 1 | 2017/09/06 15:45:57 | WrapperSimpleApp Error: at org.springframework.beans.factory.config.PropertyPlaceholderConfigurer.processProperties(PropertyPlaceholderConfigurer.java:268 )
INFO | jvm 1 | 2017/09/06 15:45:57 | WrapperSimpleApp Error: at org.springframework.beans.factory.config.PropertyResourceConfigurer.postProcessBeanFactory(PropertyResourceConfigurer.java:75)
INFO | jvm 1 | 2017/09/06 15:45:57 | WrapperSimpleApp Error: at org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors(AbstractApplicationContext.java: 554)
INFO | jvm 1 | 2017/09/06 15:45:57 | WrapperSimpleApp Error: at org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors(AbstractApplicationContext.java: 528)
INFO | jvm 1 | 2017/09/06 15:45:57 | WrapperSimpleApp Error: at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:363)
INFO | jvm 1 | 2017/09/06 15:45:57 | WrapperSimpleApp Error: at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:139)
INFO | jvm 1 | 2017/09/06 15:45:57 | WrapperSimpleApp Error: at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:83)
INFO | jvm 1 | 2017/09/06 15:45:57 | WrapperSimpleApp Error: at com.vmware.eam.util.Main.main(Main.java:50)
INFO | jvm 1 | 2017/09/06 15:45:57 | WrapperSimpleApp Error: at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
INFO | jvm 1 | 2017/09/06 15:45:57 | WrapperSimpleApp Error: at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
INFO | jvm 1 | 2017/09/06 15:45:57 | WrapperSimpleApp Error: at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
INFO | jvm 1 | 2017/09/06 15:45:57 | WrapperSimpleApp Error: at java.lang.reflect.Method.invoke(Unknown Source)
INFO | jvm 1 | 2017/09/06 15:45:57 | WrapperSimpleApp Error: at org.tanukisoftware.wrapper.WrapperSimpleApp.run(WrapperSimpleApp.java:290)
INFO | jvm 1 | 2017/09/06 15:45:57 | WrapperSimpleApp Error: at java.lang.Thread.run(Unknown Source)
STATUS | wrapper | 2017/09/06 15:45:59 | <-- Wrapper Stopped
So following my nose, a potentially relevant snip from the aforementioned eam-server-beans.xml found in /usr/lib/VMware-eam/conf is:
<!-- ESX Agent Manager beans -->
<bean id="esxAgentMgr"
class="com.vmware.eam.EsxAgentManagerImpl"
lazy-init="false"
depends-on="springContext,
vcInventory,
vcListener,
httpNfcLeaseTimeoutHandler,
nameManager,
vcDatastoreManager,
vcEventManager,
vcKeyValueStore,
l10n">
<constructor-arg ref="adapterServer" />
<constructor-arg value="${vc.proxy.host}" />
<constructor-arg value="${vc.proxy.port}" />
<constructor-arg value="${vc.tunnelSdkUri}" />
<constructor-arg ref="clientAuthenticator" />
<constructor-arg value="${eam.clear_db_on_startup}" />
<constructor-arg value="${eam.support_linked_clone}" />
<constructor-arg value="${eam.debug_ref_count}" />
<constructor-arg value="${vum.integration}" />
<constructor-arg value="${Net.DVFilterBindIpAddress}" />
<constructor-arg value="${Net.TrafficFilterIpAddress}" />
<constructor-arg value="${Net.DVFilterVswifIpAddress}" />
<constructor-arg ref="commonMapStore" />
<constructor-arg ref="vpxdConnection" />
<constructor-arg ref="vumConnection" />
<constructor-arg ref="nameManager" />
<constructor-arg ref="vibPackageDepots" />
<property name="hostdRestartTimeout" value="${eam.hostd_restart_timeout}" />
</bean>
I'm struggling as I need to get my NSX piece back up and running (ironically so I can gracefully remove it), but believe I need the EAM to be working properly first.
Any ideas before I open a ticket?
Thanks.
vDave