Version 1.5.1.1652. Health check started failing this morning on two prod servers with the below error. Tentacle log files and event logs do not show any errors. I can telnet to :10933 from the Octopus server, but cannot open it in a web browser from the Octopus server like I can other Tentacles.
System.ServiceModel.CommunicationException: An error occurred while receiving the HTTP response to http://:10933/. This could be due to the service endpoint binding not using the HTTP protocol. This could also be due to an HTTP request context being aborted by the server (possibly due to the service shutting down). See server logs for more details. —> System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a receive. —> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. —> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
— End of inner exception stack trace —
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
at System.Net.PooledStream.Read(Byte[] buffer, Int32 offset, Int32 size)
at System.Net.Connection.SyncRead(HttpWebRequest request, Boolean userRetrievedStream, Boolean probeRead)
— End of inner exception stack trace —
at System.Net.HttpWebRequest.GetResponse()
at System.ServiceModel.Channels.HttpChannelFactory.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout)
— End of inner exception stack trace —
Server stack trace:
at System.ServiceModel.Security.IssuanceTokenProviderBase1.DoNegotiation(TimeSpan timeout) at System.ServiceModel.Security.SspiNegotiationTokenProvider.OnOpen(TimeSpan timeout) at System.ServiceModel.Security.TlsnegoTokenProvider.OnOpen(TimeSpan timeout) at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout) at System.ServiceModel.Security.SymmetricSecurityProtocol.OnOpen(TimeSpan timeout) at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout) at System.ServiceModel.Channels.SecurityChannelFactory1.ClientSecurityChannel1.OnOpen(TimeSpan timeout) at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout) at System.ServiceModel.Security.SecuritySessionSecurityTokenProvider.DoOperation(SecuritySessionOperation operation, EndpointAddress target, Uri via, SecurityToken currentToken, TimeSpan timeout) at System.ServiceModel.Security.SecuritySessionSecurityTokenProvider.GetTokenCore(TimeSpan timeout) at System.IdentityModel.Selectors.SecurityTokenProvider.GetToken(TimeSpan timeout) at System.ServiceModel.Security.SecuritySessionClientSettings1.ClientSecuritySessionChannel.OnOpen(TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
at System.ServiceModel.Channels.ReliableChannelBinder1.ChannelSynchronizer.SyncWaiter.TryGetChannel() at System.ServiceModel.Channels.ReliableChannelBinder1.ChannelSynchronizer.SyncWaiter.TryWait(TChannel& channel)
at System.ServiceModel.Channels.ReliableChannelBinder1.ChannelSynchronizer.TryGetChannel(Boolean canGetChannel, Boolean canCauseFault, TimeSpan timeout, MaskingMode maskingMode, TChannel& channel) at System.ServiceModel.Channels.ClientReliableChannelBinder1.Request(Message message, TimeSpan timeout, MaskingMode maskingMode)
at System.ServiceModel.Channels.RequestReliableRequestor.OnRequest(Message request, TimeSpan timeout, Boolean last)
at System.ServiceModel.Channels.ReliableRequestor.Request(TimeSpan timeout)
at System.ServiceModel.Channels.ClientReliableSession.Open(TimeSpan timeout)
at System.ServiceModel.Channels.ReliableRequestSessionChannel.OnOpen(TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.OnOpen(TimeSpan timeout)
at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.CallOnceManager.CallOnce(TimeSpan timeout, CallOnceManager cascade)
at System.ServiceModel.Channels.ServiceChannel.EnsureOpened(TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)
Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at Octopus.Shared.Contracts.IHealthService.CheckHealth()
at Octopus.Server.Proxies.ClientBroker1.AssignableClosure1.Execute(TService service) in c:\w\e6923628be6eaf72\source\Octopus.Server\Proxies\ClientBroker.cs:line 135
at Octopus.Server.Proxies.ClientBroker1.CallOneWay(MachineEndpoint endpoint, Action1 callback) in c:\w\e6923628be6eaf72\source\Octopus.Server\Proxies\ClientBroker.cs:line 117
at Octopus.Server.Proxies.ClientBroker1.Call[TResult](MachineEndpoint endpoint, Func2 callback) in c:\w\e6923628be6eaf72\source\Octopus.Server\Proxies\ClientBroker.cs:line 30
at Octopus.Server.Tasks.Health.CheckTentacleHealthActivity.<>c__DisplayClass2.b__0() in c:\w\e6923628be6eaf72\source\Octopus.Server\Tasks\Health\CheckTentacleHealthActivity.cs:line 29