Some tentacles do not upgrade through Octopus Server

Legacy Uncategorised
1

Hi,

I’ve upgraded Octopus Server from 3.1.1 to 3.1.4. I ran the tentacle upgrade task and some tentacles upgraded fine but others did not upgrade. This issue occurred during a 3.0.24 to 3.1.1 upgrade as well. That time, I manually upgraded many of the tentacles using the tentacle installer exe directly on the machine, but I’d rather not go through that again.

Here is a log of one of the upgrade task runs:

Task ID:       ServerTasks-28436
Task status:   Success
Task queued:   October 14, 2015 10:30 AM
Task started:  October 14, 2015 10:30 AM
Task duration: 12 minutes

                    | == Warning: Upgrade Tentacles in (Environment) ==
10:30:12   Info     |   1 machines will be upgraded to the latest Tentacle version.
                    | 
                    |   == Warning: (Tentacle) ==
10:30:14   Info     |     Running Tentacle version 3.1.1
10:30:14   Info     |     Beginning upgrade
10:30:22   Verbose  |     Octopus Deploy: Tentacle version 3.1.1 (3.1.1+Branch.master.Sha.8dfcece92e5f815ad9e82fdd4dfbf2a3ac2890ce)
10:30:22   Verbose  |     Package: C:\Octopus\Work\20151014173020-23\Tentacle.nupkg
10:30:22   Verbose  |     Destination: C:\Octopus\Upgrade\20151014173012-R52BC
10:30:22   Verbose  |     5 files extracted
10:30:22   Info     |     Waiting for Tentacle to shut down and restart with the new version 3.1.4...
10:30:24   Info     |     Running Tentacle version 3.1.1
10:42:29   Warning  |     Object reference not set to an instance of an object.
                    |     System.NullReferenceException: Object reference not set to an instance of an object.
                    |     Server stack trace:
                    |     at Halibut.ServiceModel.HalibutProxy.Invoke(IMessage msg) in y:\work\7ab39c94136bc5c6\source\Halibut\ServiceModel\HalibutProxy.cs:line 33
                    |     Exception rethrown at [0]:
                    |     at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
                    |     at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
                    |     at Octopus.Shared.Contracts.IScriptService.StartScript(StartScriptCommand command)
                    |     at Octopus.Worker.Tentacles.TentacleRemoteEndpointFacade.ExecuteCommand(StartScriptCommand command) in Y:\work\refs\tags\3.1.4\source\Octopus.Worker\Tentacles\TentacleRemoteEndpointFacade.cs:line 38
                    |     at Octopus.Worker.Tentacles.TentacleUpgradeMediator.TryCheckVersion() in Y:\work\refs\tags\3.1.4\source\Octopus.Worker\Tentacles\TentacleUpgradeMediator.cs:line 96
                    |     Octopus.Server version 3.1.4 (3.1.4+Branch.master.Sha.3510f96241b0bb85f7189448bb7fef417340ebf8)
10:42:33   Info     |     Finishing upgrade
10:42:34   Verbose  |     Gathering logs from the upgrade
10:42:34   Verbose  |     C:\Octopus\Upgrade\20151014173012-R52BC\UpgradeLog-20151014173022-08d2f.log
10:42:34   Verbose  |     Octopus upgrader version 0.0.0.0
10:42:34   Verbose  |     Current directory: C:\Octopus\Upgrade\20151014173012-R52BC
10:42:34   Verbose  |     Arguments:
10:42:34   Verbose  |     [0] = "OctopusDeploy Tentacle"
10:42:34   Verbose  |     [1] = "C:\Octopus\Upgrade\20151014173012-R52BC\Octopus.Tentacle.msi"
10:42:34   Verbose  |     [2] = "C:\Octopus\Upgrade\20151014173012-R52BC\Octopus.Tentacle-x64.msi"
10:42:34   Verbose  |     Upgrade mutex acquired
10:42:34   Verbose  |     Stopping service: OctopusDeploy Tentacle
10:42:34   Verbose  |     Stopping service...
10:42:34   Verbose  |     Waiting for service to stop. Current status: StopPending
10:42:34   Verbose  |     Waiting for service to stop. Current status: Stopped
10:42:34   Verbose  |     Service stopped
10:42:34   Verbose  |     Running MSIEXEC on MSI: C:\Octopus\Upgrade\20151014173012-R52BC\Octopus.Tentacle-x64.msi, output will go to: c2fcbab3-f243-4e4f-8784-e8bdb846b873
10:42:34   Verbose  |     Starting: OctopusDeploy Tentacle
10:42:34   Verbose  |     Waiting for service to start. Current status: StartPending
10:42:34   Verbose  |     Waiting for service to start. Current status: Running
10:42:34   Verbose  |     Service started
10:42:34   Verbose  |     Error: System.ComponentModel.Win32Exception (0x80004005): Unknown error (0xfffffffe)
10:42:34   Verbose  |     at System.Diagnostics.Process.StartWithShellExecuteEx(ProcessStartInfo startInfo)
10:42:34   Verbose  |     at System.Diagnostics.Process.Start(ProcessStartInfo startInfo)
10:42:34   Verbose  |     at Octopus.Upgrader.SoftwareInstaller.Install(String msiPath) in Y:\work\refs\tags\3.1.4\source\Octopus.Upgrader\SoftwareInstaller.cs:line 16
10:42:34   Verbose  |     at Octopus.Upgrader.Program.Main(String[] args) in Y:\work\refs\tags\3.1.4\source\Octopus.Upgrader\Program.cs:line 39
10:42:40   Info     |     Running Tentacle version 3.1.1
                    |   
                    |   == Success: Summary ==
10:42:40   Info     |     ONLINE:
10:42:40   Info     |     - (Tentacle) at poll://thwquxnqq57p6hlpfuvv/, running version 3.1.1
2

Hi Roderick,

Is there anything different about the Tentacles that did not upgrade?
We have had report of Tentacles running under AD accounts that will not upgrade. Are your Tentacle running under an AD account?

Cheers,
Shane

3

Hi Shane,

All of the tentacles are running under an AD account. We need the tentacle to be running under a specific service account in order to perform tasks. This wasn’t an issue before in Octopus 2.x using the same accounts. I can also run the tentacle installer to upgrade the tentacle manually while logged in using the same account.

All of the machines are quite different so it’s difficult to single out any particular thing that is different between the tentacles that succeeded and failed upgrade.

4

Hi Roderick,

We have a ticket open to fix this: https://github.com/OctopusDeploy/Issues/issues/2023

I haven’t heard of anyone having success upgrading Tentacles under an AD account so if you had any insights into why some are successful I would appreciate your feedback.

We will get this fixed soon, thanks for your patience.

Cheers,
Shane

5

I tried a bunch of things to see if I could get any insight as to what’s happening, but I didn’t find out anything useful.