Environment "unavailable" and tentacle service will not start up

[tc.pugh]

We started having an issue a few days ago when one of our environments became unavailable. We first noticed the issue when we tried deploying something and received this error:

"Acquiring packages

Info

Making a list of packages to acquire

June 10th 2019 12:05:29

Info

Delta compression is enabled for package transfers from the Octopus Server to deployment targets

June 10th 2019 12:05:29

Info

Package OurProject.1.0.7100.21241-qa was found in cache. No need to download.

June 10th 2019 12:05:31

Fatal

The step failed: Activity failed with error ‘The remote script failed with exit code 255’."

I then checked the environment and it said that Calamari was outdated. I tried going to “environments page” to update Calamari, but I did not see the option to update(note:I am running octopus server 3.16.2). It also said that the Calamari should be updated automatically as part of a deployment, but I did not see this when I tried deploying.

Finally I remoted into the server and tried to reboot the Tentacle, but it would not reboot. Instead it gave me an error. I have posted the logs for the error received.

We also starting having errors come in our event logs around the same time(I’ll attached those also)

We have not had this issue before, and Octopus is working on our other environments fine.

The only recent change to this server is we had the following windows update applied:

"2019-03 Update for .NET Framework 3.5, 4.5.2, 4.6, 4.6.1, 4.6.2, 4.7, 4.7.1, 4.7.2 for Windows 8.1 and Server 2012 R2 for x64 (KB4489488)

Installation date: ‎6/‎6/‎2019 2:01 AM

Installation status: Succeeded

Update type: Important

Install this update to resolve issues in Windows. For a complete listing of the issues that are included in this update, see the associated Microsoft Knowledge Base article for more information. After you install this item, you may have to restart your computer.

More information:
http://support.microsoft.com/kb/4489488

Help and Support:

http://support.microsoft.com"

Since we have received the error we have tried to resolve this issue the following ways

1. Reinstalling the msi
   2)Manually uninstalling tentacle following this guide(https://octopus.com/docs/administration/managing-infrastructure/tentacle-configuration-and-file-storage/manually-uninstall-tentacle)
2. Installing latest tentacle version (from 3.15.7 to 4.0.1)
3. Rebooting the server w/ the Octopus tentacle.

Any suggestions would be greatly appreciated!

OctopusTentacle.txt (43.3 KB)

[Alex.Rolley]

Hi @tc.pugh

Thanks for getting in touch, that does seem to be a difficult one.

I see that you’ve already done some troubleshooting, so I can skip over some things. The first item that jumps out to me is that you mention that you upgraded to Tentacle 4.0.1, yet all I can see in the logs is attempts to start Tentacle 3.15.7.

What I would recommend as a starting point here is to re-run through the manual tentacle uninstall process (including the normal uninstall process), however this time also remove C:\Octopus\Tentacle\<Instance> and c:Octopus\Calamari directories. If you can then install Tentacle 4.0.1 and report back your results that would be greatly appreciated.

If this fails I think the next step would be to setup a support call to investigate in real time, although I’m hoping that removing the extra folders should sort this problem out.

Thanks @tc.pugh, I look forward to hearing from you with the results,

Regards,
Alex

[tc.pugh]

HI,
The logs are from when we had 3.15.7, but we get similar errors with 4.0.1 (I’ll attach logs). For these logs we also enabled trace mode in the log level.

We tried the manual tentacle uninstall and received similar results.

We would like to set up a support call and are available weekdays from 9am to 5pm eastern time. On the call would be myself and a coworker.

Also do you have a debug version of the tentacle we can install so we could get more debug information?

OctopusTentacle.LocalAppData.txt (20.5 KB)
OctopusTentacle.txt (23.4 KB)

Thanks,
TC

[Alex.Rolley]

Hi TC,

Unfortunately the logs haven’t provided much more info, and we don’t have a debug version of Tentacle on the shelf ready to go.

I’ve looked at the time overlap and it’s not super pretty (9am to 5pm Eastern is 11pm to 7am AEST). If you’re available at 4pm Eastern (which will be 6AM tomorrow for us) I have a victim willing volunteer from our engineering team who will be willing to join the call with us.

If you can confirm that you’re available I’ll send through the meeting information shortly before the meeting.

Regards
Alex

[tc.pugh]

Alex,

No need to have the meeting tomorrow. We found the fix. We had a profiler tool with a bad installation that seemed to be causing the issue. Uninstalling the profiler fixed this issues we were having and since then we have been able to deploy multiple times to the server.

Thanks,
TC

[Alex.Rolley]

Hi TC,

That’s great news, glad to hear that you got it sorted.

Let me know if there is anything else we can help with.

Regards,
Alex