We have 8 VDP servers running in our environment and about 10 days ago they all stopped working. We haven't made changes to the firewalls nor have we made changes to vCenter but when I tail this log /space/avamar/var/vdr/server_logs/vdr-server.log I see the following errors:
2013-05-07 11:04:54,026 ERROR [com.emc.vdr.server.VDRServer$1]-service.AdapterUtils: ServiceInstance failed to establish connection
org.apache.axis2.AxisFault: Connection refused
at org.apache.axis2.AxisFault.makeFault(AxisFault.java:430)
at org.apache.axis2.transport.http.HTTPSender.sendViaPost(HTTPSender.java:203)
at org.apache.axis2.transport.http.HTTPSender.send(HTTPSender.java:76)domain
at org.apache.axis2.transport.http.CommonsHTTPTransportSender.writeMessageWithCommons(CommonsHTTPTransportSender.java:400)
at org.apache.axis2.transport.http.CommonsHTTPTransportSender.invoke(CommonsHTTPTransportSender.java:225)
at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:435)
at org.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java:402)
at org.apache.axis2.description.OutInAxisOperationClient.executeImpl(OutInAxisOperation.java:229)
at org.apache.axis2.client.OperationClient.execute(OperationClient.java:165)
at com.avamar.mc.sdk.client.McServiceStub.login(McServiceStub.java:5084)
at com.avamar.mc.sdk.client.McServiceClient.login(McServiceClient.java:119)
at com.avamar.mc.sdk.client.McServiceClient.connect(McServiceClient.java:105)
at com.emc.avamar.mcsdk.adapter.service.ServiceInstance.connect(ServiceInstance.java:313)
at com.emc.avamar.mcsdk.adapter.service.ServiceInstance.<init>(ServiceInstance.java:225)
at com.emc.avamar.mcsdk.adapter.service.ServiceInstance.<init>(ServiceInstance.java:187)
at com.emc.vdr.server.VDRExternalServerAccessConfig.getFromAdapterDefault(VDRExternalServerAccessConfig.java:67)
at com.emc.vdr.server.VDRExternalServerAccessConfig.getFromPropsDefault(VDRExternalServerAccessConfig.java:126)
at com.emc.vdr.server.ConnectionService.run(ConnectionService.java:40)
at java.lang.Thread.run(Unknown Source)
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(Unknown Source)
at java.net.PlainSocketImpl.connectToAddress(Unknown Source)
at java.net.PlainSocketImpl.connect(Unknown Source)
at java.net.SocksSocketImpl.connect(Unknown Source)
at java.net.Socket.connect(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.connect(Unknown Source)
at com.emc.avamar.mcsdk.adapter.service.TrustAllSSLProtocolSocketFactory.createSocket(TrustAllSSLProtocolSocketFactory.java:128)
at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707)
at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361)
at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387)
at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:3</init>97)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:346)
at org.apache.axis2.transport.http.AbstractHTTPSender.executeMethod(AbstractHTTPSender.java:542)
at org.apache.axis2.transport.http.HTTPSender.sendViaPost(HTTPSender.java:199)
... 17 more
2013-05-07 11:04:54,026 ERROR [com.emc.vdr.server.VDRServer$1]-service.AdapterUtils: ServiceInstance failed to obtain domain "/"
java.lang.NullPointerException
at com.emc.avamar.mcsdk.adapter.service.ServiceInstance.getDomain(ServiceInstance.java:496)
at com.emc.avamar.mcsdk.adapter.service.ServiceInstance.getRootDomain(ServiceInstance.java:462)
at com.emc.vdr.server.VDRServer.getVCenterClient(VDRServer.java:402)
at com.emc.vdr.server.VDRExternalServerAccessConfig.getFromAdapterDefault(VDRExternalServerAccessConfig.java:86)
at com.emc.vdr.server.VDRExternalServerAccessConfig.getFromPropsDefault(VDRExternalServerAccessConfig.java:126)
at com.emc.vdr.server.ConnectionService.run(ConnectionService.java:40)
at java.lang.Thread.run(Unknown Source)
2013-05-07 11:04:54,027 ERROR [com.emc.vdr.server.VDRServer$1]-server.VDRServer: VDRServer.getVCenterClient cannot obtain VCenterClient
java.lang.NullPointerException
at com.emc.vdr.server.VDRServer.getVCenterClient(VDRServer.java:403)
at com.emc.vdr.server.VDRExternalServerAccessConfig.getFromAdapterDefault(VDRExternalServerAccessConfig.java:86)
at com.emc.vdr.server.VDRExternalServerAccessConfig.getFromPropsDefault(VDRExternalServerAccessConfig.java:126)
at com.emc.vdr.server.ConnectionService.run(ConnectionService.java:40)
at java.lang.Thread.run(Unknown Source)
2013-05-07 11:04:54,027 ERROR [com.emc.vdr.server.VDRServer$1]-server.VDRExternalServerAccessConfig: Unable to fetch the vCenter credential from web service: null
2013-05-07 11:04:54,027 ERROR [com.emc.vdr.server.VDRServer$1]-server.ConnectionService: Unable to get the vi access: no protocol:
2013-05-07 11:04:54,027 INFO [com.emc.vdr.server.VDRServer$1]-server.ConnectionService: Retry after 5 seconds ...
2013-05-07 11:04:59,028 INFO [com.emc.vdr.server.VDRServer$1]-server.ConnectionService: Trying to establish connection with vCenter.
2013-05-07 11:04:59,028 INFO [com.emc.vdr.server.VDRServer$1]-server.VDRExternalServerAccessConfig: loading default values forcefully: true
2013-05-07 11:04:59,028 INFO [com.emc.vdr.server.VDRServer$1]-server.VDRExternalServerAccessConfig: Get mc access...
2013-05-07 11:04:59,039 INFO [com.emc.vdr.server.VDRServer$1]-service.AdapterUtils: MCS Web Services URL: https://castlerock.domain.edu:9443/services/mcService MCUserId="MCUser" MCUserPswd="*****************************"
2013-05-07 11:04:59,054 INFO [com.emc.vdr.server.VDRServer$1]-httpclient.HttpMethodDirector: I/O exception (java.net.ConnectException) caught when processing request: Connection refused
2013-05-07 11:04:59,055 INFO [com.emc.vdr.server.VDRServer$1]-httpclient.HttpMethodDirector: Retrying request
2013-05-07 11:04:59,055 INFO [com.emc.vdr.server.VDRServer$1]-httpclient.HttpMethodDirector: I/O exception (java.net.ConnectException) caught when processing request: Connection refused
2013-05-07 11:04:59,055 INFO [com.emc.vdr.server.VDRServer$1]-httpclient.HttpMethodDirector: Retrying request
2013-05-07 11:04:59,056 INFO [com.emc.vdr.server.VDRServer$1]-httpclient.HttpMethodDirector: I/O exception (java.net.ConnectException) caught when processing request: Connection refused
2013-05-07 11:04:59,056 INFO [com.emc.vdr.server.VDRServer$1]-httpclient.HttpMethodDirector: Retrying request
2013-05-07 11:04:59,057 INFO [com.emc.vdr.server.VDRServer$1]-http.HTTPSender: Unable to sendViaPost to url[https://castlerock.domain.edu:9443/services/mcService?wsdl]
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(Unknown Source)
at java.net.PlainSocketImpl.connectToAddress(Unknown Source)
at java.net.PlainSocketImpl.connect(Unknown Source)
at java.net.SocksSocketImpl.connect(Unknown Source)
at java.net.Socket.connect(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.connect(Unknown Source)
at com.emc.avamar.mcsdk.adapter.service.TrustAllSSLProtocolSocketFactory.createSocket(TrustAllSSLProtocolSocketFactory.java:128)
at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707)
at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361)
at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387)
at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:346)
at org.apache.axis2.transport.http.AbstractHTTPSender.executeMethod(AbstractHTTPSender.java:542)
at org.apache.axis2.transport.http.HTTPSender.sendViaPost(HTTPSender.java:199)
at org.apache.axis2.transport.http.HTTPSender.send(HTTPSender.java:76)
at org.apache.axis2.transport.http.CommonsHTTPTransportSender.writeMessageWithCommons(CommonsHTTPTransportSender.java:400)
at org.apache.axis2.transport.http.CommonsHTTPTransportSender.invoke(CommonsHTTPTransportSender.java:225)
at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:435)
at org.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java:402)
at org.apache.axis2.description.OutInAxisOperationClient.executeImpl(OutInAxisOperation.java:229)
at org.apache.axis2.client.OperationClient.execute(OperationClient.java:165)
at com.avamar.mc.sdk.client.McServiceStub.login(McServiceStub.java:5084)
at com.avamar.mc.sdk.client.McServiceClient.login(McServiceClient.java:119)
at com.avamar.mc.sdk.client.McServiceClient.connect(McServiceClient.java:105)
at com.emc.avamar.mcsdk.adapter.service.ServiceInstance.connect(ServiceInstance.java:313)
at com.emc.avamar.mcsdk.adapter.service.ServiceInstance.<init>(ServiceInstance.java:225)
at com.emc.avamar.mcsdk.adapter.service.ServiceInstance.<init>(ServiceInstance.java:187)
at com.emc.vdr.server.VDRExternalServerAccessConfig.getFromAdapterDefault(VDRExternalServerAccessConfig.java:67)
at com.emc.vdr.server.VDRExternalServerAccessConfig.getFromPropsDefault(VDRExternalServerAccessConfig.java:126)
at com.emc.vdr.server.ConnectionService.run(ConnectionService.java:40)
at java.lang.Thread.run(Unknown Source)
2013-05-07 11:04:59,057 ERROR [com.emc.vdr.server.VDRServer$1]-service.AdapterUtils: ServiceInstance failed to establish connection
org.apache.axis2.AxisFault: Connection refused
at org.apache.axis2.AxisFault.makeFault(AxisFault.java:430)
at org.apache.axis2.transport.http.HTTPSender.sendViaPost(HTTPSender.java:203)
at org.apache.axis2.transport.http.HTTPSender.send(HTTPSender.java:76)
at org.apache.axis2.transport.http.CommonsHTTPTransportSender.writeMessageWithCommons(CommonsHTTPTransportSender.java:400)
at org.apache.axis2.transport.http.CommonsHTTPTransportSender.invoke(CommonsHTTPTransportSender.java:225)
at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:435)
at org.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java:402)
at org.apache.axis2.description.OutInAxisOperationClient.executeImpl(OutInAxisOperation.java:229)
at org.apache.axis2.client.OperationClient.execute(OperationClient.java:165)
at com.avamar.mc.sdk.client.McServiceStub.login(McServiceStub.java:5084)
at com.avamar.mc.sdk.client.McServiceClient.login(McServiceClient.java:119)
at com.avamar.mc.sdk.client.McServiceClient.connect(McServiceClient.java:105)
at com.emc.avamar.mcsdk.adapter.service.ServiceInstance.connect(ServiceInstance.java:313)
at com.emc.avamar.mcsdk.adapter.service.ServiceInstance.<init>(ServiceInstance.java:225)
at com.emc.avamar.mcsdk.adapter.service.ServiceInstance.<init>(ServiceInstance.java:187)
at com.emc.vdr.server.VDRExternalServerAccessConfig.getFromAdapterDefault(VDRExternalServerAccessConfig.java:67)
at com.emc.vdr.server.VDRExternalServerAccessConfig.getFromPropsDefault(VDRExternalServerAccessConfig.java:126)
at com.emc.vdr.server.ConnectionService.run(ConnectionService.java:40)
at java.lang.Thread.run(Unknown Source)
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(Unknown Source)
at java.net.PlainSocketImpl.connectToAddress(Unknown Source)
at java.net.PlainSocketImpl.connect(Unknown Source)
at java.net.SocksSocketImpl.connect(Unknown Source)
at java.net.Socket.connect(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.connect(Unknown Source)
at com.emc.avamar.mcsdk.adapter.service.TrustAllSSLProtocolSocketFactory.createSocket(TrustAllSSLProtocolSocketFactory.java:128)
at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707)
at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361)
at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387)
at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:346)
at org.apache.axis2.transport.http.AbstractHTTPSender.executeMethod(AbstractHTTPSender.java:542)
at org.apache.axis2.transport.http.HTTPSender.sendViaPost(HTTPSender.java:199)
... 17 more
2013-05-07 11:04:59,057 ERROR [com.emc.vdr.server.VDRServer$1]-service.AdapterUtils: ServiceInstance failed to obtain domain "/"
java.lang.NullPointerException
at com.emc.avamar.mcsdk.adapter.service.ServiceInstance.getDomain(ServiceInstance.java:496)
at com.emc.avamar.mcsdk.adapter.service.ServiceInstance.getRootDomain(ServiceInstance.java:462)
at com.emc.vdr.server.VDRServer.getVCenterClient(VDRServer.java:402)
at com.emc.vdr.server.VDRExternalServerAccessConfig.getFromAdapterDefault(VDRExternalServerAccessConfig.java:86)
at com.emc.vdr.server.VDRExternalServerAccessConfig.getFromPropsDefault(VDRExternalServerAccessConfig.java:126)
at com.emc.vdr.server.ConnectionService.run(ConnectionService.java:40)
at java.lang.Thread.run(Unknown Source)
2013-05-07 11:04:59,058 ERROR [com.emc.vdr.server.VDRServer$1]-server.VDRServer: VDRServer.getVCenterClient cannot obtain VCenterClient
java.lang.NullPointerException
at com.emc.vdr.server.VDRServer.getVCenterClient(VDRServer.java:403)
at com.emc.vdr.server.VDRExternalServerAccessConfig.getFromAdapterDefault(VDRExternalServerAccessConfig.java:86)
at com.emc.vdr.server.VDRExternalServerAccessConfig.getFromPropsDefault(VDRExternalServerAccessConfig.java:126)
at com.emc.vdr.server.ConnectionService.run(ConnectionService.java:40)
at java.lang.Thread.run(Unknown Source)
2013-05-07 11:04:59,058 ERROR [com.emc.vdr.server.VDRServer$1]-server.VDRExternalServerAccessConfig: Unable to fetch the vCenter credential from web service: null
2013-05-07 11:04:59,058 ERROR [com.emc.vdr.server.VDRServer$1]-server.ConnectionService: Unable to get the vi access: no protocol:
2013-05-07 11:04:59,058 INFO [com.emc.vdr.server.VDRServer$1]-server.ConnectionService: Retry after 5 seconds ...
Something MUST have changed since this same error happens on all 8 VDP servers but I can't figure out what might have changed nor how to fix/workaround this issue now. Any help would be appreciated.