Recurring issue with my Tentacles/Machines

mball · 11 August 2015 19:09

I am running Octopus Deploy 2.6.4.951 and Octopus Tentacle 2.6.4.951-x64. I have one Tentacle configured as two Machines in Octopus Deploy so that I can have different sets of Roles for configured for my deployments. After a varying period of time of one to several days, the Machine will start failing the Health Check with either of the following errors:

Pipefish.Errors.PipefishCommunicationException: Unexpected character encountered while parsing value: S. Path ‘’, line 0, position 0.
at Microsoft.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at Microsoft.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccess(Task task)
at Octopus.Server.Communications.Handshaking.BackgroundPassiveTentacleSquidFinder.d__a.MoveNext() in y:\work\refs\heads\master\source\Octopus.Server\Communications\Handshaking\BackgroundPassiveTentacleSquidFinder.cs:line 96
Newtonsoft.Json.JsonReaderException: Unexpected character encountered while parsing value: S. Path ‘’, line 0, position 0.
at Newtonsoft.Json.JsonTextReader.ParseValue()
at Newtonsoft.Json.JsonTextReader.ReadInternal()
at Newtonsoft.Json.JsonTextReader.Read()
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.ReadForType(JsonReader reader, JsonContract contract, Boolean hasConverter)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
at Newtonsoft.Json.JsonSerializer.Deserialize[T](JsonReader reader)
at Octopus.Server.Communications.Handshaking.PassiveTentacleHandshaker.<>c__DisplayClass4.b__2(SecureTcpResponse response) in y:\work\refs\heads\master\source\Octopus.Server\Communications\Handshaking\PassiveTentacleHandshaker.cs:line 65
at Pipefish.Transport.SecureTcp.Client.SecureTcpClient.Send(SecureTcpRequest request, Action`1 response) in y:\work\3cbe05672d69a231\source\Pipefish.Transport.SecureTcp\Client\SecureTcpClient.cs:line 88
at Octopus.Server.Communications.Handshaking.PassiveTentacleHandshaker.Receive(PassiveTentacleHandshakeRequest message) in y:\work\refs\heads\master\source\Octopus.Server\Communications\Handshaking\PassiveTentacleHandshaker.cs:line 56
at Pipefish.Actor.OnReceivingTyped[TBody](Message message) in y:\work\3cbe05672d69a231\source\Pipefish\Actor.cs:line 113

Pipefish.PipefishException: The remote host aborted the connection. This can happen when the remote server does not trust the certificate that we provided. —> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. —> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
— End of inner exception stack trace —
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
at System.Net.FixedSizeReader.ReadPacket(Byte[] buffer, Int32 offset, Int32 count)
at System.Net.Security._SslStream.StartFrameHeader(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security._SslStream.StartReading(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security._SslStream.ProcessRead(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.Read(Byte[] buffer, Int32 offset, Int32 count)
at System.IO.Stream.InternalCopyTo(Stream destination, Int32 bufferSize)
at Pipefish.Transport.SecureTcp.Client.SecureTcpResponseExtensions.ReadContentAsText(SecureTcpResponse response) in y:\work\3cbe05672d69a231\source\Pipefish.Transport.SecureTcp\Client\SecureTcpResponseExtensions.cs:line 13
at Pipefish.Transport.SecureTcp.MessageExchange.Client.ClientWorker.<>c__DisplayClassf.b__a(SecureTcpResponse response) in y:\work\3cbe05672d69a231\source\Pipefish.Transport.SecureTcp\MessageExchange\Client\ClientWorker.cs:line 337
at Pipefish.Transport.SecureTcp.Client.SecureTcpClient.Send(SecureTcpRequest request, Action1 response) in y:\work\3cbe05672d69a231\source\Pipefish.Transport.SecureTcp\Client\SecureTcpClient.cs:line 88 --- End of inner exception stack trace --- at Pipefish.Transport.SecureTcp.Client.SecureTcpClient.Send(SecureTcpRequest request, Action1 response) in y:\work\3cbe05672d69a231\source\Pipefish.Transport.SecureTcp\Client\SecureTcpClient.cs:line 105
at Pipefish.Transport.SecureTcp.MessageExchange.Client.ClientWorker.PerformExchange() in y:\work\3cbe05672d69a231\source\Pipefish.Transport.SecureTcp\MessageExchange\Client\ClientWorker.cs:line 353
at Pipefish.Transport.SecureTcp.MessageExchange.Client.ClientWorker.Run() in y:\work\3cbe05672d69a231\source\Pipefish.Transport.SecureTcp\MessageExchange\Client\ClientWorker.cs:line 187

A “Delete this Tentacle instance” and then Configuring a new one and deleting and re-adding the Machines in the Octopus Deploy Web Portal usually works, but sometimes I have to fully uninstall the Tentacle software and re-run the install to get back to a working instance.

What is causing these issues to recur and what can I do to prevent it?

Shane_Gill · 13 August 2015 01:31

Hi,

Thanks for the question. Registering the exact same Tentacle multiple times in an Octopus Server does not work in version 2.x.

I think your safest bet would be to create another Tentacle instance on that machine. You can find instructions here:

You can then register both of your Tentacle instances with the Octopus Server and assigned them to whatever roles and environments are required without them interfering with each other.

Because of architecture changes in 3.0, you can register a Tentacle multiple times in the same Octopus Server.

Cheers,
Shane

mball · 19 August 2015 21:28

I have now installed multiple Tentacles on one server and deleted and then recreated the Machines pointing at the separate instances, yet the issue still persists.

mball · 19 August 2015 21:36

I am also getting this error on a machine that only has one Tentacle installed, so It’s failing the Connectivity check.

Pipefish.PipefishException: The request failed: BadRequest
The incoming request was on a communication link (subscription) that is no longer valid. Reset connectivity to perform a new handshake and reestablish communication.
at Pipefish.Transport.SecureTcp.MessageExchange.Client.ClientWorker.<>c__DisplayClassf.b__a(SecureTcpResponse response) in y:\work\3cbe05672d69a231\source\Pipefish.Transport.SecureTcp\MessageExchange\Client\ClientWorker.cs:line 345
at Pipefish.Transport.SecureTcp.Client.SecureTcpClient.Send(SecureTcpRequest request, Action`1 response) in y:\work\3cbe05672d69a231\source\Pipefish.Transport.SecureTcp\Client\SecureTcpClient.cs:line 88
at Pipefish.Transport.SecureTcp.MessageExchange.Client.ClientWorker.PerformExchange() in y:\work\3cbe05672d69a231\source\Pipefish.Transport.SecureTcp\MessageExchange\Client\ClientWorker.cs:line 353
at Pipefish.Transport.SecureTcp.MessageExchange.Client.ClientWorker.Run() in y:\work\3cbe05672d69a231\source\Pipefish.Transport.SecureTcp\MessageExchange\Client\ClientWorker.cs:line 187

Shane_Gill · 21 August 2015 00:37

Hi,

For the single Tentacle with the BadRequest error, could you try the “Reset” button on the Tentacle connectivity page in Octopus Server?

Screenshot attached.

Shane_Gill · 21 August 2015 00:57

Hi,

For the multiple instances of the Tentacles that are failing, would you mind sending me a screenshot of the connectivity page in Octopus Server for each of those Tentacles? Mark this discussion as private or obfuscate your IP addresses if necessary.

The error Unexpected character encountered while parsing value: S. indicates that something other than JSON was returned. If I had to hazard a guess it would be a proxy between your Octopus Server and Tentacle.

Cheers,
Shane

mball · 21 August 2015 13:25

For the single Tentacle with the BadRequest error, the Reset button usually fixes it, but only for about a day.

For the multiple instances of the Tentacles that are failing, they are all succeeding at this moment, so I’ll hop on later when I have a Tentacle failing the health check.