I upgraded to Octopus Deploy 3.2.16 this morning and all of my tentacles are now failing their health checks. Here’s the stack trace of the error…
Halibut.HalibutClientException: An error occurred when sending a request to ‘https://mytentacleserver:10933/’, after the request began: Object reference not set to an instance of an object. —> System.NullReferenceException: Object reference not set to an instance of an object.
| at Halibut.Transport.Protocol.MessageExchangeStream.ReadBsonMessageT in y:\work\7ab39c94136bc5c6\source\Halibut\Transport\Protocol\MessageExchangeStream.cs:line 200
| at Halibut.Transport.Protocol.MessageExchangeStream.ReceiveT in y:\work\7ab39c94136bc5c6\source\Halibut\Transport\Protocol\MessageExchangeStream.cs:line 154
| at Halibut.HalibutRuntime.<>c__DisplayClass6.b__5(MessageExchangeProtocol protocol) in y:\work\7ab39c94136bc5c6\source\Halibut\HalibutRuntime.cs:line 115
| at Halibut.Transport.SecureClient.ExecuteTransaction(Action1 protocolHandler) in y:\work\7ab39c94136bc5c6\source\Halibut\Transport\SecureClient.cs:line 58 | --- End of inner exception stack trace --- | Server stack trace: | at Halibut.Transport.SecureClient.HandleError(Exception lastError, Boolean retryAllowed) in y:\work\7ab39c94136bc5c6\source\Halibut\Transport\SecureClient.cs:line 163 | at Halibut.Transport.SecureClient.ExecuteTransaction(Action1 protocolHandler) in y:\work\7ab39c94136bc5c6\source\Halibut\Transport\SecureClient.cs:line 103
| at Halibut.HalibutRuntime.SendOutgoingHttpsRequest(RequestMessage request) in y:\work\7ab39c94136bc5c6\source\Halibut\HalibutRuntime.cs:line 117
| at Halibut.ServiceModel.HalibutProxy.Invoke(IMessage msg) in y:\work\7ab39c94136bc5c6\source\Halibut\ServiceModel\HalibutProxy.cs:line 37
| Exception rethrown at [0]:
| at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
| at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
| at Octopus.Shared.Contracts.IScriptService.StartScript(StartScriptCommand command)
| at Octopus.Worker.Tentacles.TentacleRemoteEndpointFacade.ExecuteCommand(StartScriptCommand command) in Y:\work\refs\tags\3.2.16\source\Octopus.Worker\Tentacles\TentacleRemoteEndpointFacade.cs:line 38
| at Octopus.Worker.Tentacles.TentacleWorker.CheckHealth() in Y:\work\refs\tags\3.2.16\source\Octopus.Worker\Tentacles\TentacleWorker.cs:line 55
| at Octopus.Server.Orchestration.Health.MachineTaskController.PerformTask(Machine machine) in Y:\work\refs\tags\3.2.16\source\Octopus.Server\Orchestration\Health\MachineTaskController.cs:line 86
| Octopus.Server version 3.2.16 (3.2.16+Branch.master.Sha.9632f2eae1ea99934b139da39440c5ab5d018db7)
Did you upgrade (and update) your Tentacles to 3.2.14 ?
We had a communication error. You will have to downgrade your server back to 3.2.14.
Then all of your Tentacles need to be upgraded off 3.2.14 - how many do you have? We have a couple of strategies to do this. They all require a little bit of work.
I did downgrade the server to 3.2.14 to get back to a working state. I have nearly 100 tentacles, so I would prefer as much an automated solution as possible.
Great. I think the easiest automated approach would be to run hydra with the latest package available on the download page. The Hydra step itself can be run from 3.x it will behave as it did with the 2.6->3.x upgrade and when updated your Tentacles will appear offline until you finish and upgrade your Octopus Server.
Hi, I have also a problem (only with 1 out of 17 tentacles) after updating my octopus server to 3.2.16 (up from 3.2.13).
As you can see in my attached video-file the tentacle is running and the server is able to connect to the tentacle. However when i start the health check for this machine a folder is created on the tentacle (only for a second in the C:\Octopus\Work directory) but is instantly deleted again before the health check has finished. I also attached the raw error log for this case.
Thank you for the prompt replies and helping me resolve the issue.
I was able to downgrade my server to 3.2.14, use the Hyrda approach to push the 3.2.16 Tentacles to each of my servers, and then upgrade my server to 3.2.16.
While I understand that this breaking change was likely unintentional, I’m disappointed in how it was communicated (as far as I know it wasn’t). This caused me to burn cycles troubleshooting and then submit a support ticket.
With this issue and the several other support tickets I’ve needed to submit recently, some of which were for uncommunicated/unintentional breaking changes, my confidence of the product is starting to wane.
The error was an unintentional mistake. We pulled the build the morning after. The dev who made the mistake spent til 3am correcting the issue so we could get a new build out ASAP.
In the release notes for 3.2.15 we mentioned that we had to pull the build due to a Tentacle communication error that was also linked in the 3.2.15 Release notes.
We also updated our Known Issues page. http://docs.octopusdeploy.com/display/ODKB/Known+Issues
We knew that 160ish customers had installed the build, but this is all anon and we cannot link it directly to customers. We also did not know how many had directly updated their Tentacles. If a Server only upgrade had taken place, it would not have been an issue, being it was on the Tentacle side.
Our Known Issues page would not have helped, as we did update it at the same time as pulling the build, but it would have helped after. But viewing it after or hopefully google directing you to it would have helped. We try to be as open as we can with our release notes and convey everything we can to our customers.
We are terribly sorry that it happened, and also that I did not convey that immediately when you reported it, I just wanted to help you get working ASAP.
We knew we would have customers reach out, and any that did report it during the night were given instructions, but we just had no way of knowing who you were or what channels you watch.