We started to get frequent (once a day or so) deployment failures. The only recent change in Octopus or infrastructure that I can think of was introduction of Stop App Pool and Start App Pool steps into the project’s Process.
On the client (octo.exe) side we get the following error (which is quite cryptic BTW) somewhere between Step 1 (Stop / Start Site) and Step 2 (Stop App Pool):
Deploying iRePORT 14.0.0-Test669 to: Test-pre-deploy (Guided Failure: Not Enabled)
Waiting for 1 deployment(s) to complete....
Success: Step 1: Stop the main Site and start the Down Page
Success: ireporttest.od.nih.gov
Info Executing script on 'ireporttest.od.nih.gov'
Info "Default Web Site" successfully stopped
Info "Down Page" successfully started.
Error from Octopus server (HTTP 200): Unable to process response from server: Error converting value "Canceled" to type 'Octopus.Client.Model.ActivityStatus'. Path 'ActivityLog.Children[2].Status', line 290, position 29.. Response content: {
"Task": {
"Id": "ServerTasks-3542",
"Name": "Deploy",
"Description": "Deploy iRe
Exit code: -7
Octopus Server logs contains no errors. Tentacle logs have the following errors, for example:
2016-01-28 10:21:43.7461 27 INFO listen://[::]:10933/ 27 Accepted TCP client: [::ffff:165.112.229.242]:63786
2016-01-28 10:21:43.7461 66 INFO listen://[::]:10933/ 66 Performing TLS server handshake
2016-01-28 10:21:43.7461 66 INFO listen://[::]:10933/ 66 Secure connection established, client is not yet authenticated, client connected with Tls12
2016-01-28 10:21:43.7617 66 INFO listen://[::]:10933/ 66 Client at [::ffff:165.112.229.242]:63786 authenticated as 45B9825C978A7D33607146A13F6FC409D55726BC
2016-01-28 10:32:31.3567 66 INFO listen://[::]:10933/ 66 Unhandled error when handling request from client: [::ffff:165.112.229.242]:63786
System.IO.IOException: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. ---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
--- End of inner exception stack trace ---
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
at System.Net.FixedSizeReader.ReadPacket(Byte[] buffer, Int32 offset, Int32 count)
at System.Net.Security._SslStream.StartFrameHeader(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security._SslStream.StartReading(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security._SslStream.ProcessRead(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.Read(Byte[] buffer, Int32 offset, Int32 count)
at System.IO.StreamReader.ReadBuffer()
at System.IO.StreamReader.ReadLine()
at Halibut.Transport.Protocol.MessageExchangeStream.ReadLine() in y:\work\7ab39c94136bc5c6\source\Halibut\Transport\Protocol\MessageExchangeStream.cs:line 91
at Halibut.Transport.Protocol.MessageExchangeStream.ExpectNextOrEnd() in y:\work\7ab39c94136bc5c6\source\Halibut\Transport\Protocol\MessageExchangeStream.cs:line 65
at Halibut.Transport.Protocol.MessageExchangeProtocol.ProcessClientRequests(Func`2 incomingRequestProcessor) in y:\work\7ab39c94136bc5c6\source\Halibut\Transport\Protocol\MessageExchangeProtocol.cs:line 110
at Halibut.Transport.SecureListener.ExecuteRequest(TcpClient client) in y:\work\7ab39c94136bc5c6\source\Halibut\Transport\SecureListener.cs:line 126
Tentacle service restart solves the issue.
Do you have any advice on how to resolve the issue?
So far, we have tried to upgrade to the latest version (3.2.13 to 3.2.20) and make sure that clocks on Server and Tentacle servers are synchronized, but this has not resolved the issue.