Tentacle upgrade failure - This logging operation has already been completed or abandoned

Hi,

I’ve got a Tentacle stuck on a version as the upgrade fails every time. I’ve tried rebooting to see if that helped, but got the same problem. Any idea what it could be?

Server exception: 
System.InvalidOperationException: This logging operation has already been completed or abandoned.
   at Octopus.Core.Infrastructure.Logging.Operation.Complete() in ./source/Octopus.Core/Infrastructure/Logging/Operation.cs:line 59
   at Octopus.Core.Packages.PackageConversion.Streams.StreamExtensionMethods.CopyAndReportProgress(Stream source, Stream destination, Action`1 updateProgress) in ./source/Octopus.Core/Packages/PackageConversion/Streams/StreamExtensionMethods.cs:line 68
   at Halibut.Transport.Protocol.MessageExchangeStream.WriteEachStream(IEnumerable`1 streams)
   at Halibut.Transport.Protocol.MessageExchangeStream.Send[T](T message)
   at Halibut.Transport.Protocol.MessageExchangeProtocol.ProcessReceiverInternalAsync(IPendingRequestQueue pendingRequests, RequestMessage nextRequest) 

ServerTasks-608709.log.txt (10.9 KB)

Hey @matthewericford,

Thank you for contacting Octopus Support and sorry to hear you are having issues upgrading your tentacle. From the logs it looks like its trying to upgrade your tentacle from version 6.1.1157 to 6.1.1409.

I can see in your logs:

Waiting for Tentacle to shut down and restart with the new version 6.1.1409

It looks like the tentacle shuts down but on reboot it still has 6.1.1157 on it. I cant see why this fails from the task log you sent over. I am wondering if the tentacle log would give us some more information on why the tentacle wont upgrade.

I have created you a secure link here, are you able to grab the tentacle log for me and let us know when you have uploaded it to the secure link and I can take a look at it for you.

Kind Regards,

Clare

Hi Clare,

Attached in the secure store :slight_smile:

Hey @matthewericford,

Sorry for the long wait on our reply, I took a look at your logs but are they from the same machine you are trying to upgrade?

I noticed in your Octopus Server Upgrade logs for the Tentacle the machine name is MyTentantMachine, in the tentacle specific logs you just sent over the machine name is called something else. The log timings don’t correspond either (ie when your MyTenantMachine restarts for the upgrade it doesn’t show the restart in the other logs you sent over).

Did you change the machine name at all in the logs you sent first (because you didn’t want the machine name shown on the forum) or have the logs been taken from two different machines?

Do you have two tentacle instances on that box?

One thing you could try is to manually upgrade the tentacle by downloading Tentacle 6.1.1409 and running it on your tentacle box you are trying to upgrade. If it tells you to repair go ahead and try that, if it asks you to install it or upgrade try that and see if it errors out.

I look forward to seeing if that works for you.

Kind Regards,

Sorry that was me redacting when posting publicly and then uploading the wrong day log!

I’ve uploaded another from 2mins ago and not touch the logs :smiley:

Thanks

Matt

Hey @matthewericford,

Sorry it has taken so long to get back to you, I noticed you put in an internal ticket through support@octopus.com, I have responded to this on the email I sent you, we can continue this conversation there if you don’t mind and if we come to a conclusion I will post it in this thread so if anyone else has the issue.

Thank you for your patience so far with this and I will continue this investigation through emails.

Kind Regards,

Clare

Hi Clare,

Yes that’d be great thanks. I like to post these sort of things in case anyone else has the issue and searches :relaxed:

1 Like

To anyone having this same issue, this is related to our Octopus Cloud customers specifically and it was a feature toggle we implemented in order to allow for some extra log parameters to be generated for fault finding.

This feature seemed to have an effect on this instance and its tentacle upgrades, we have not heard of any other customers having this issue but the feature toggle was turned off which fixed @matthewericford’s issue posted here.

Our engineers will look at that feature toggle and workout what is causing the tentacle upgrade issue before turning it back on again.

This issue has been solved but please do reach out in future if anyone experiences something similar.

Kind Regards,

Clare

1 Like