Upgrading Polling Worker issue

[mikepower79]

Hi,
I am seeing an issue when I run the Upgrade on a polling worker to the latest version.

The upgrade just stays running before I cancel it and this is the messages I see while it is running.
There is no other tasks running but yet its says there are 5 tasks running.

We are running version 2022.2.6971.

Kind Regards,
Micheál Power

[clare.martin]

Hey @mikepower79,

Thank you for contacting us about this issue, this may be a silly question since you are a regular here over in the support forums and are quite the Octopus Wizard…

But I was wondering if you tried to restart the tentacle service on the worker (if you are able to) and see if it then completes the upgrade. You mentioned there are no other tasks running on that worker from inside Octopus itself so I am wondering if the worker has just gotten itself into a ‘stuck state’ and just needs a reboot in order for it to see there are no tasks running?

Let me know if you are able to restart the service on the worker, and if so, does that allow for the upgrade to go through?

I look forward to hearing from you,

Kind Regards,

Clare

[mikepower79]

Hi @clare.martin,
I have restarted the server a few times now but still same error. I have also restarted the worker service on the server.

Kind Regards,
Micheál Power

[dane.falvo]

Hi @mikepower79,

Clare’s just tied up with another ticket at the moment.

I’ve been looking into this one, which is intriguing. Can I get you to double check there are no “System Tasks” running in the “Tasks” tab? It requires a couple of extra clicks.
“Show Advanced Filters” → “Include system tasks”

Have you got any other polling tentacles having the same issue?

Regards,

[mikepower79]

Hi @dane.falvo,
No other system tasks running.
After I tried again this time the Worker upgrade did complete.
Would not work all day yesterday after rebooting worker server but now has passed so issue resolved.
Thanks for your time.

Kind Regards,
Micheál Power

[clare.martin]

Hey @mikepower79,

Excellent news thank you for letting us know, its always a bit annoying when these things happen though as techies we always want to know why things happen even if they appear to magically fix themselves.

I am not sure what went on here and it would be good to dig into it to really understand what the tasks were that was holding the upgrade up but apart from looking at Event Viewer for Applications and filtering by Octopus Tentacle to see if anything stands out there around the time of the upgrade its hard to say why the tentacle thought it had tasks to run.

Happy to close this one off but if it does happen again let us know and we can have a real good dig into it!

Kind Regards,

Clare

[clare.martin]

Hey @mikepower79,

Just a note here as I have been alerted to a discussion surrounding tentacle upgrades and wanted to share it with you and others if you are interested. Our engineers mentioned:

Between Tentacle 6.1.1184 and 6.1.1189 we added passing in the isolation mutex name to the Tentacle.

For newer tentacles we use a differently named mutex PackageRetention so that package retention runs based on a separate mutex to other tasks. The default RunningScript mutex is used by pretty much every other kind of task.

However for older tentacles which don’t take in a mutex name, we use the default name RunningScript for package retention. This means instead of requiring no other package retention tasks to be running, it instead requires no tasks to be running in order to continue. Hence the waiting.

I can see from your screenshot your worker was running 6.1.670 which was released in May 2021 (so is very old compared to your current Octopus Server version). Whilst this does not explain why there were no tasks running on that worker in the Octopus UI it does mean that because you upgraded, this issue will not happen again as we have accounted for other running tasks.

I hope this was of some value, there are discussions going on with regards to improving Tentacle upgrades and how Octopus handles them so in future we hope to pass on those improvements to you all.

It is definitely worth having runbooks setup or keeping on top of Tentacle upgrades so everyone gets the benefit of the latest features.

Kind Regards,

Clare

[mikepower79]

Hi @clare.martin,
Thanks for that update, its great to know.
Yes I agree and a lot of our Octopus users don’t keep on top of the Tentacle upgrades.

Kind Regards,
Micheál Power