Upgrading tentacles fails

Octopus Server
,
1

Hi,

We just upgraded our Octopus on-prem environment to version 2022.1.2495 successfully. Afterwards I tried to upgrade all Tentacles but several fail to update with the following error;
Octopus.Tentacle.Upgrader terminated unexpectedly with exit code -1073741502.

What causes the error code 1073741502?

Thanks in advance!

Best regards,
Jeffrey

3

Hi @jeffreywels,

Thanks for reaching out!

Would you be able to send over the raw task log of the update task within Octopus, please?
You can upload the log to our secure upload area here: Support Files.

Are you able to successfully run health checks against the targets that the upgrade fails on?
Is there anything the targets that fail to upgrade have in common?

Once we’ve been able to take a look at the task log for the upgrade we should have a better idea of what direction to take for troubleshooting.

Kind Regards,
Adam

4

Hi Adam,

I have included the full raw log of the first run after the update
ServerTasks-197244.log.txt (2.1 MB)
. There you will see some servers being updated but ending in the specific error.

As far as i can come up with, the servers don’t have particularly big things in common. Some are Windows 2012, some are Windows 2016. Some run the Tentacle Service with a service name, some don’t. But also the ones that succeed have the same things in common as those who fail.

Health checks run and all succeed with a healthy environment, but some aren’t updated but still run without problems, so the urgency ain’t that high.

Thanks in advance

5

Hi @jeffreywels,

Thanks for sending over the log file.

I found a few distinct errors within the log, some of which repeated.

Registry access errors:
13:29:27 Error | MSI (s) (7C:80) [13:28:47:497]: Product: Octopus Deploy Tentacle -- Error 1402. Could not open key: HKEY_CURRENT_USER\Software\Octopus\Tentacle. System error 1018. Verify that you have sufficient access to that key, or contact your support personnel.

Invalid Connection/Machine Unavailable:
13:36:27 Verbose | An error occurred when sending a request to 'https://scomp6172.wurnet.nl:10933/', before the request could begin: The client was unable to establish the initial connection within 00:01:00.

Unexpected terminations:
13:33:50 Error | Octopus.Tentacle.Upgrader terminated unexpectedly with exit code -1073741502.

Unfortunately, it seems that there is a variety of issues causing the tentacle upgrade to fail.

I would recommend individually inspecting the Tentacles that are failing and, if possible, trying the following troubleshooting methods:

  • Temporarily disable any anti-virus that may remove new files or hold a lock on them whilst scanning.
  • Checking permissions of the account the Tentacle is running under for registry access/folder access to the Tentacle Install and Tentacle Home folder.
  • Check EventViewer inside windows around the time of the failed upgrade as this may be able to provide you with a more relevant and verbose message.

As Tentacle changes and newer versions are released the way that it interacts with the system may also change slightly.
This isn’t something that happens often and not something you would notice if the difference between versions is small.
Do you perhaps know which version some of the failed Tentacles were on previously and which version they’re trying to be upgraded to?
If there is a large gap between versions it may explain the difficulties faced with upgrading them automatically.

Kind Regards,
Adam

6

Hi Adam,

Some of those errors you saw earlier, have been solved (by running the health checks a couple of times…strangely)

Some remain (see raw file)
ServerTasks-197320.log.txt (229.1 KB)

There remain two distinct issues;

  • Octopus.Tentacle.Upgrader terminated unexpectedly with exit code -1073741502. → on servers running tentacle version 6.0.489 and 6.1.732
  • two servers that appear to update, but after restarting the tentacle, it shows the old version of the tentacle (
    Running Tentacle version 6.1.732
    Beginning upgrade
    Waiting for the upgrade to complete
    Waiting for Tentacle to shut down and restart with the new version 6.1.1304…
    Running Tentacle version 6.1.732
    )

Also, when looking to the specific version it’s trying to update, I can’t find that version on Download Octopus Tentacle - Octopus Deploy which is strange since this version is running on all the other machines since the last Octopus update this morning.
I was thinking of running the installer on the machines after downloading it, but couldn’t find that specific version so.

I’ll check event manager to see what might goes wrong.
The anti-virus scanner isn’t blocking the folder, the tentacle is in, so that’s sorted
User permissions could be a problem, but strangely enough we use almost everywhere the “local system” account as the account for running the service, which works on most (and before this upgrade for all machines).

Jeffrey

7

Hi @jeffreywels,

Thanks for getting back to me!

The first error with the long exit code may be the ones you’d be most likely to get an answer for inside EventViewer, please let me know what you’re able to find.

The second error, it may be possible that a manual restart of the Tentacle service is all that’s required to complete the update, can you let me know if you’re able to try this?

As for the specific Tentacle version, we don’t widely advertise every build of Tentacle on the downloads page as not every build of Tentacle has changes that would be considered meaningful.

I can, however, provide a download link directly for that version for you:

Let me know what you’re able to find and if you have any further questions, I’ll do my best to assist as best as I can!

Kind Regards,
Adam

8

Hi Adam,

Thanks for the downloads.

I tried downloading and installing the executable (since the event viewer didn’t gave anything away). Sadly enough, the installer fails aswell.
By using the installer and writing verbose logging to a file, I was able to grab an error, which doesn’t gave much away for me (see also the full verbose log
OctopusVerboseInstall.txt (70.9 KB);

MSI (c) (08:1C) [17:07:06:844]: PROPERTY CHANGE: Deleting SECONDSEQUENCE property. Its current value is ‘1’.
Action ended 17:07:06: ExecuteAction. Return value 3.
MSI (c) (08:1C) [17:07:06:860]: Doing action: FatalError
Action start 17:07:06: FatalError.
Action ended 17:07:07: FatalError. Return value 2.
Action ended 17:07:07: INSTALL. Return value 3.

Maybe that helps?

Best regards,
Jeffrey

9

Hi @jeffreywels,

Thanks for getting back to me!

I wonder if it’s worth manually uninstalling the Tentacle on that machine and trying to re-install it with the new version MSI?
The phrasing of the error seems to point to some sort of conflict and performing the above may allow the installation to proceed as expected.

Let me know if this sounds like something you can try, please!

Kind Regards,
Adam

10

Hi Adam,

Will uninstalling remove the tentacle config as well? And by that meaning, that i need to re-add the machine to Octopus and therefore re-adding the deployment target in each and every single step/project it’s in?

Best regards,
Jeffrey

11

Hi @jeffreywels,

Manually uninstalling the Tentacle following the instructions in the docs I linked will remove the config file for the Tentacle.
You shouldn’t have to remove the Tentacle from within Octopus, however.

If the tentacle is a listening tentacle, as long as the new installation trusts the Octopus Server (via the Server Thumbprint), you should be able to modify the machine inside Octopus to match the tentacle thumbprint of the new Tentacle.

If the tentacle is a polling tentacle, when setting up the new Tentacle there is an option to overwrite a machine with the same name, so as long as you match the name of the new Tentacle with the machine name within Octopus, this should gracefully overwrite.

Once you’ve done either of the above, depending on the type of Tentacle, run a manual health check on the target to confirm everything is working correctly.

I hope this helps settle your concerns, please let me know how everything goes and if you’re still running into errors.

Kind Regards,
Adam

12

Hi Adam,

Followed the steps in deleting the tentacle from a server, but I need to wait for a restart tonight (since it’s a production environment). If it has been restarted tomorrow, I will try to run the installer again and see what happens.

Best regards,
Jeffrey

13

Hi @jeffreywels,

No worries and thanks for the update!
Let me know if you have success tomorrow or if you’re still experiencing issues.

Kind Regards,
Adam

14

Hi Adam,

Removing the tentacle completely and manually, and then restarting the server did work for one server.

This afternoon I’m going to try two others, but it looks like this should fix the issue.

Best regards,
Jeffrey

1 Like
15

Hi @jeffreywels,

Thanks for letting us know! It’s good to hear that seems to have resolved the issue.

Please let us know if the other two servers don’t respond in the same way.

Kind Regards,
Adam

16

Hi Adam,

Both servers worked out the same. Only three more remain, but not today.

You may close this issue, since the workaround works.

Thanks for your help!
Jeffrey

1 Like
17

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.