Issues with upgrades

Legacy Execution
1

We have noticed with the last couple of recent upgrades that our remote tentacle are not upgrading cleanly. We are seeing the following with large percentage of servers. We are manually restarting the agents to resolve the issues at this time.

Removing the ‘D:\Octopus\Upgrade\20170417122809-W7NDQ’ directory
April 17th 2017 08:28:22
Info
The process cannot access the file because it is being used by another process.
April 17th 2017 08:28:22
Fatal
The remote script failed with exit code 65535

2

Hi,

Thanks for getting in touch. Could you please send the raw task log for your Tentacle upgrade? (https://octopus.com/docs/how-to/get-the-raw-output-from-a-task)

Cheers,
Shane

3

We have also experienced this issue when upgrading to 3.12.5.0. Several (not all) of our machines went into this strange mode, marked as “Unhealthy”.

4

Hi,

We made some improvements to Tentacle upgrades in 3.12.6: https://github.com/OctopusDeploy/Issues/issues/3413

Please let me know if it continues to happen.

Cheers,
Shane

5

With the recent upgrade to 3.2.14 we are getting the following error with the tentacle upgrade.

This server does not have version 3.7.53 of the Calamari package. It will be pushed automatically.

April 21st 2017 13:55:49

Info
Calamari upgrade successful

April 21st 2017 13:55:56

Info
Running Tentacle version 3.12.4

April 21st 2017 13:55:56

Info
Beginning upgrade

April 21st 2017 13:56:04

Info
Waiting for the upgrade to complete

April 21st 2017 13:56:06

Fatal
The remote script failed with exit code 65532

6

Getting this on 3.12.8 for a handful of machines. Happened on a lot at first, but clicking “Try Again” fixed most of them.

7

Hi,

Thanks for reporting. Would it be possible to get the raw log from the Tentacle upgrades that are failing?

Thanks
Shane

8

Attached a snippet of the raw log output for two machines - one successful and one failed (some info redacted). In the UI, the Connectivity tab for these failed machines initially shows a Current Version of “0.0.0 (upgrade available)”.

It looks like the failure occurs after the upgrade when EnsureSuccessful() tries to find an exit code log. So far, it looks like all of our upgrades were actually successful. After running a health check, the Current Version is correct and the machine status is healthy.

OctopusUpgradeTentacles_RawLogSnippet_Issue53658.txt (26 KB)

9

Thank you.

It seems the process that performs the Tentacle upgrade is returning exit code 65532 but I haven’t figure out why that is happening. Unfortunately I am finding it very difficult to reproduce.

I have added a few tweaks and some extra logging to the upgrade process. I expect the error will be encountered again but the Tentacle upgrade should succeed.

Cheers,
Shane

10

I will let you know the outcome after we update in the near future.

Mike

image001.jpg

11

Hello,

I have upgraded today to 3.13.0 and encountered the same error while upgrading the tentacles.

12

I just performed an upgrade from 3.11.5 to 3.13.0 this morning and out of about 200 tentacle machines we had about 15 or so fail with this particular error.

Having it try to upgrade again worked successfully afterwards.

Here is a sample log entry of one that failed:
09:17:16 Verbose | Host Name:

09:17:16 Verbose | Running As: (Local Administrator: True)

09:17:17 Verbose | Package: C:\Octopus\Work\20170508141715-775\Tentacle.nupkg

09:17:17 Verbose | Destination: C:\Octopus\Upgrade\20170508141702-DTY9T

09:17:17 Verbose | Octopus Deploy: Tentacle version 3.11.5 (3.11.5+Branch.release-3.11.5.Sha.2ec95d93ccfb115671522932f84809b9bb4f3dca) instance Default

09:17:17 Verbose | Environment Information:

09:17:17 Verbose | OperatingSystem: Microsoft Windows NT 6.1.7601 Service Pack 1

09:17:17 Verbose | OsBitVersion: x64

09:17:17 Verbose | Is64BitProcess: True

09:17:17 Verbose | CurrentUser: NT AUTHORITY\SYSTEM

09:17:17 Verbose | MachineName:

09:17:17 Verbose | ProcessorCount: 8

09:17:17 Verbose | CurrentDirectory: C:\Octopus\Work\20170508141715-775

09:17:17 Verbose | TempDirectory: C:\Windows\TEMP\

09:17:17 Verbose | HostProcessName: Tentacle

09:17:17 Verbose | 5 files extracted

09:17:17 Info | Waiting for the upgrade to complete

09:17:17 Verbose | No exit code log exists at ‘C:\Octopus\Upgrade\20170508141702-DTY9T’

09:17:17 Fatal | The remote script failed with exit code 65532

09:17:17 Verbose | The remote script failed with exit code 65532

                |     Octopus.Shared.Tasks.ActivityFailedException: The remote script failed with exit code 65532

                |     at Octopus.Worker.Scripting.ScriptResult.EnsureSuccessful() in Z:\buildAgent\workDir\eec88466c176b607\source\Octopus.Worker\Scripting\ScriptResult.cs:line 91

                |     at Octopus.Worker.Tentacles.TentacleUpgradeMediator.WaitForExitCode(String installId) in Z:\buildAgent\workDir\eec88466c176b607\source\Octopus.Worker\Tentacles\TentacleUpgradeMediator.cs:line 117

                |     at Octopus.Worker.Tentacles.TentacleUpgradeMediator.PerformUpgrade(TargetManifest targetManifest) in Z:\buildAgent\workDir\eec88466c176b607\source\Octopus.Worker\Tentacles\TentacleUpgradeMediator.cs:line 56

                |     at Octopus.Server.Orchestration.Health.HealthCheckService.PerformHealthCheck(Machine machine, ExceptionHandling exceptionHandling, Action`2 customAction) in Z:\buildAgent\workDir\eec88466c176b607\source\Octopus.Server\Orchestration\Health\HealthCheckService.cs:line 77

                |     Octopus.Server version 3.13.0 (3.13.0+Branch.master.Sha.872899cdf487a79ef334d6b3c310d671b65a57df)

09:17:17 Verbose | Recording health check results

13

Thanks.

3.13.2 contains the extra logging and I am hoping will complete successfully. Is everyone who is running into this issue upgrading a large number of machines?

Cheers
Shane

14

Define “large number of machines” :slight_smile:

I upgraded > 100 machines. Does this qualify?

Cheers,

Mayank

15

Hi everyone,

It looks like this error is happening when running on PowerShell version 2. I have reproduced and created an issue to fix: https://github.com/OctopusDeploy/Issues/issues/3529

Does that seem consistent with what you are seeing?

Thanks for your patience.

Cheers,
Shane