Lost Tentacle Connections after upgrading to 3.2.19

Legacy Uncategorised
1

Hi! After upgrading to 3.2.19 we’ve lost connectivity with all of our Tentacles. Any ideas? The server was at 3.2.14 prior to upgrading and the Tentacles are all at 3.2.14.

All Tentacles are reporting the error below.

Thanks!
Jason.

Here’s what I see when I upgrade:

Task ID: ServerTasks-15741
Task status: Executing
Task queued: Thursday, 21 January 2016 8:12 AM
Task started: Thursday, 21 January 2016 8:12 AM
Task duration: 4 minutes
Server version: 3.2.19+Branch.master.Sha.ab644ee12ada37fda9e736cd1d805d7c3cc8d1a3

                | == Running: Upgrade Tentacles == [34 of 47 machines upgraded (72%)]

08:12:46 Info | Starting Tentacle upgrade for all environments
08:12:46 Info | 47 machines will be upgraded to the latest Tentacle version.
|
| == Failed: AUTEST5VM03 ==
08:13:03 Fatal | An error occurred when sending a request to ‘https://10.11.10.133:10933/’, after the request began: Object reference not set to an instance of an object.
08:13:03 Verbose | An error occurred when sending a request to ‘https://10.11.10.133:10933/’, after the request began: Object reference not set to an instance of an object.
| Halibut.HalibutClientException: An error occurred when sending a request to ‘https://10.11.10.133:10933/’, after the request began: Object reference not set to an instance of an object. —> System.NullReferenceException: Object reference not set to an instance of an object.
| at Halibut.Transport.Protocol.MessageExchangeStream.ReadBsonMessageT in y:\work\7ab39c94136bc5c6\source\Halibut\Transport\Protocol\MessageExchangeStream.cs:line 198
| at Halibut.Transport.Protocol.MessageExchangeStream.ReceiveT in y:\work\7ab39c94136bc5c6\source\Halibut\Transport\Protocol\MessageExchangeStream.cs:line 154
| at Halibut.HalibutRuntime.<>c__DisplayClass6.b__5(MessageExchangeProtocol protocol) in y:\work\7ab39c94136bc5c6\source\Halibut\HalibutRuntime.cs:line 115
| at Halibut.Transport.SecureClient.ExecuteTransaction(Action`1 protocolHandler) in y:\work\7ab39c94136bc5c6\source\Halibut\Transport\SecureClient.cs:line 58
| — End of inner exception stack trace —
| Server stack trace:
| at Halibut.Transport.SecureClient.HandleError(Exception lastError, Boolean retryAllowed) in y:\work\7ab39c94136bc5c6\source\Halibut\Transport\SecureClient.cs:line 163
| at Halibut.HalibutRuntime.SendOutgoingHttpsRequest(RequestMessage request) in y:\work\7ab39c94136bc5c6\source\Halibut\HalibutRuntime.cs:line 117
| at Halibut.ServiceModel.HalibutProxy.Invoke(IMessage msg) in y:\work\7ab39c94136bc5c6\source\Halibut\ServiceModel\HalibutProxy.cs:line 33
| Exception rethrown at [0]:
| at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
| at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
| at Octopus.Shared.Contracts.IScriptService.StartScript(StartScriptCommand command)
| at Octopus.Worker.Tentacles.TentacleRemoteEndpointFacade.ExecuteCommand(StartScriptCommand command) in Y:\work\refs\tags\3.2.19\source\Octopus.Worker\Tentacles\TentacleRemoteEndpointFacade.cs:line 38
| at Octopus.Worker.Tentacles.TentacleUpgradeMediator.PerformUpgrade() in Y:\work\refs\tags\3.2.19\source\Octopus.Worker\Tentacles\TentacleUpgradeMediator.cs:line 31
| at Octopus.Server.Orchestration.Health.MachineTaskController.PerformTask(Machine machine) in Y:\work\refs\tags\3.2.19\source\Octopus.Server\Orchestration\Health\MachineTaskController.cs:line 86
| Octopus.Server version 3.2.19 (3.2.19+Branch.master.Sha.ab644ee12ada37fda9e736cd1d805d7c3cc8d1a3)

Here’s what I see when I check health:

Task ID: ServerTasks-15743
Task status: Failed
Task queued: Thursday, 21 January 2016 8:20 AM
Task started: Thursday, 21 January 2016 8:20 AM
Task duration: 2 seconds
Server version: 3.2.19+Branch.master.Sha.ab644ee12ada37fda9e736cd1d805d7c3cc8d1a3

                | == Failed: Check AUDEPLOYMENT health ==

08:20:48 Info | Starting health check for a limited set of deployment targets
08:20:48 Info | 1 machines will have their health checks taken.
08:20:49 Fatal | The health check failed. One or more deployment targets were not available.
|
| == Failed: Check deployment target: AUDEPLOYMENT ==
08:20:49 Fatal | An error occurred when sending a request to ‘https://audeployment:10934/’, after the request began: Object reference not set to an instance of an object.
08:20:49 Verbose | An error occurred when sending a request to ‘https://audeployment:10934/’, after the request began: Object reference not set to an instance of an object.
| Halibut.HalibutClientException: An error occurred when sending a request to ‘https://audeployment:10934/’, after the request began: Object reference not set to an instance of an object. —> System.NullReferenceException: Object reference not set to an instance of an object.
| at Halibut.Transport.Protocol.MessageExchangeStream.ReadBsonMessageT in y:\work\7ab39c94136bc5c6\source\Halibut\Transport\Protocol\MessageExchangeStream.cs:line 198
| at Halibut.Transport.Protocol.MessageExchangeStream.ReceiveT in y:\work\7ab39c94136bc5c6\source\Halibut\Transport\Protocol\MessageExchangeStream.cs:line 154
| at Halibut.HalibutRuntime.<>c__DisplayClass6.b__5(MessageExchangeProtocol protocol) in y:\work\7ab39c94136bc5c6\source\Halibut\HalibutRuntime.cs:line 115
| at Halibut.Transport.SecureClient.ExecuteTransaction(Action`1 protocolHandler) in y:\work\7ab39c94136bc5c6\source\Halibut\Transport\SecureClient.cs:line 58
| — End of inner exception stack trace —
| Server stack trace:
| at Halibut.Transport.SecureClient.HandleError(Exception lastError, Boolean retryAllowed) in y:\work\7ab39c94136bc5c6\source\Halibut\Transport\SecureClient.cs:line 163
| at Halibut.HalibutRuntime.SendOutgoingHttpsRequest(RequestMessage request) in y:\work\7ab39c94136bc5c6\source\Halibut\HalibutRuntime.cs:line 117
| at Halibut.ServiceModel.HalibutProxy.Invoke(IMessage msg) in y:\work\7ab39c94136bc5c6\source\Halibut\ServiceModel\HalibutProxy.cs:line 33
| Exception rethrown at [0]:
| at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
| at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
| at Octopus.Shared.Contracts.IScriptService.StartScript(StartScriptCommand command)
| at Octopus.Worker.Tentacles.TentacleRemoteEndpointFacade.ExecuteCommand(StartScriptCommand command) in Y:\work\refs\tags\3.2.19\source\Octopus.Worker\Tentacles\TentacleRemoteEndpointFacade.cs:line 38
| at Octopus.Worker.Tentacles.TentacleWorker.CheckHealth() in Y:\work\refs\tags\3.2.19\source\Octopus.Worker\Tentacles\TentacleWorker.cs:line 55
| at Octopus.Server.Orchestration.Health.MachineTaskController.PerformTask(Machine machine) in Y:\work\refs\tags\3.2.19\source\Octopus.Server\Orchestration\Health\MachineTaskController.cs:line 86
| Octopus.Server version 3.2.19 (3.2.19+Branch.master.Sha.ab644ee12ada37fda9e736cd1d805d7c3cc8d1a3)
|
| == Failed: Summary ==
08:20:49 Info | OFFLINE:
08:20:49 Info | - AUDEPLOYMENT at https://audeployment:10934/, error: An error occurred when sending a request to ‘https://audeployment:10934/’, af…
08:20:49 Fatal | One or more deployment targets were not online. Please see the output Log for details.

2

Hi Jason,

Thanks for getting in touch and sorry about the troubles. You will have to downgrade your server back to 3.2.14 and then we have a couple of strategies to help you upgrade from there. How many Tentacles do you have?

There was an unfortunate error in 3.2.14 that blocked Tentacle communication to other versions of Octopus. Very sorry that you have encountered this.

Vanessa

3

Thanks for getting back to me quickly! I’ve run the downgrade and now have connectivity to the Tentacles again. We’re running 47 Tentacles at the moment. What do you suggest for upgrading to latest?

Thanks!
Jason.

4

Hi Jason,

The most automated approach is to use Hydra with its latest package to upgrade the Tentacles from 3.2.14. Very similar to how it worked from 2.x -> 3.x it will upgrade the Tentacles for you, then you can upgrade the server version. How successful was Hydra for you when upgrading? Did you have any problems?

Vanessa

5

I do remember hitting some bumps with Hydra but don’t remember why. I’ll give Hydra a whirl.

Thanks!

6

This has worked. The biggest pain with Hydra is that it does not persist the windows account that runs the Tentacles.

Thanks again for your help! We’re now running 3.2.19 on the Server and all 47 Tentacles.

From: Jason Hess
Sent: Thursday, 21 January 2016 10:26 AM
To: ‘Vanessa Love’ tender2+d388549781@tenderapp.com
Subject: RE: Lost Tentacle Connections after upgrading to 3.2.19 [Problems #44027]

I do remember hitting some bumps with Hydra but don’t remember why. I’ll give Hydra a whirl.

Thanks!

7

Hi Jason,

Yeah, Hydra has a few issues like that, that for various reasons we can’t do a lot about.
We definitely did not intend for this to happen, but Hydra was the only way we could see to automate the upgrade to a newer version of Tentacle from an OD server and get out of this mess.

While we never intend for anything like this to happen again (or did at the time!) we are going to build in a little feature that if you have upgraded to a version that has a specific issue like this, we will add a notification in the bullhorn. We don’t know who you are but if you get notifications of upgrades we do know you are on a version and can then notify you.
We did try to pull the build early, but our wonderful customers who update early got caught and we had no way of knowing who you are.

I am very glad it wasn’t too painful and again we are really sorry that it happened.
Vanessa

8

Too easy. Thanks again for your help!