For a number of release now, two of our tentacles don’t survive the upgrade process triggered by the Environments page. They get upgraded OK, but are left in a stopped state. Logging into the server remotely and starting the service in the Service panel work and they’re fine after that. This just happened again upgrading to 1.2.1.1447.
The exception in Event Viewer is:
@@@
2012-11-29 01:29:18,101 [1] ERROR Octopus [(null)] - System.InvalidOperationException: Cannot stop Octopus Tentacle service on computer ‘.’. —> System.ComponentModel.Win32Exception: The pipe has been ended
— End of inner exception stack trace —
at System.ServiceProcess.ServiceController.Stop()
at Octopus.Shared.Startup.ServiceInstaller.StopAndWaitForStop(String name) in c:\BuildAgent\work\e6923628be6eaf72\source\Octopus.Shared\Startup\ServiceInstaller.cs:line 597
at Octopus.Shared.Startup.ServiceInstaller.Restart(String serviceName) in c:\BuildAgent\work\e6923628be6eaf72\source\Octopus.Shared\Startup\ServiceInstaller.cs:line 43
at Octopus.Shared.Startup.CommandProcessor.Process(String[] args) in c:\BuildAgent\work\e6923628be6eaf72\source\Octopus.Shared\Startup\CommandProcessor.cs:line 40
@@@
Exact same situation and error on several of our servers.
The only similarities we see between these systems are very high memory and disk usage, so the systems are reacting slowly sometimes. Maybe it is a timeout issue during stopping of the service.
The upgrade to 1.3 works with all our tentacles without any problems now.
The upgrade to 1.3 seems to have worked for all our tentacles too.
Thanks.
Doesn’t work for me.
The task “upgrading tentacle” stays forever in ongoing, it is stuck on “The Tentacle has received a request to shut down and restart”
If I cancel the task and do a check health of the tentacle in question it turns out it had been upgraded just fine.
Our upgrade process is quite annoying right now. It looks something like this
- upgrade all tentacles
- wait 15 min, cancel task (it is in a endless loop waiting for tentacles to restart)
- run a check health on all tentacles to see which ones went through
- repeat 1-3 on all tentacles not upgraded
I have no guesses either on what could be wrong here.
Hi Ivar,
Are you using Octopus 2.x? The original thread here discusses 1.x which is significantly different in this respect.
We have seen this kind of behaviour when Windows Certificate Revocation List checking is enabled on the Tentacle machine, but the machine is unable to access the Internet. Could that be an issue here? (Details in the very last comment here: http://help.octopusdeploy.com/discussions/problems/15271-octopus-server-having-problems-connecting-with-tentacles-in-staging-environment-in-dmz)
Regards,
Nick
Ah, well yes we’re going to 2.5.3 from 2.4.x
I tried letting it run over the night… didn’t work. It’s totally random
which servers it hangs on too.
[image: Inline images 1]
Thanks for the follow-up Ivar. Did you have a chance to check CRL settings?
If you can get hold of them from a failing machine, Tentacle writes a series of log files to C:\Octopus\Logs. I believe we’ll find some clues in there as to what’s preventing the upgrade, if you can email them to our support@ email address (octopusdeploy.com) that may be preferable to attaching them here.
(A quick scan of the recent files might also point out some errors to you, if you’re inclined to dig in.)
Regards,
Nick
Same issue while upgrading 2.5.8 to 2.6.
1\3 of the tentacles don’t restart.
A health check fixes the status.
Log of a failing upgrade.
’’‘
2015-01-04 10:53:18.1490 INFO Beginning transfer of d:\Octopus\Files\Octopus.Tentacle.2.6.0.778.nupkg-79fce80d-a8c9-4c01-8271-5a1ba5821cef
2015-01-04 10:53:43.1613 INFO Reconfiguring the Octopus Tentacle Windows Service to use path: D:\Octopus\Applications.SQ-PRODUCTDB-SVR04-53D8DF09\Octopus.Tentacle\2.6.0.778\Tentacle.exe
2015-01-04 10:53:43.1703 INFO Tentacle instance: Tentacle
2015-01-04 10:53:43.6173 INFO Octopus Deploy: Tentacle version 2.6.0.778
2015-01-04 10:53:45.7435 INFO Service reconfigured
2015-01-04 10:53:45.9085 INFO Service reconfigured
2015-01-04 10:53:46.0676 INFO Reconfiguration complete!
2015-01-04 10:53:56.5125 INFO Stopping service…
2015-01-04 10:53:57.7966 INFO Waiting for service to stop. Current status: StopPending
2015-01-04 10:53:58.7987 INFO Waiting for service to stop. Current status: StopPending
2015-01-04 10:53:59.8008 INFO Waiting for service to stop. Current status: StopPending
2015-01-04 10:54:00.8019 INFO Waiting for service to stop. Current status: StopPending
2015-01-04 10:54:01.8040 INFO Waiting for service to stop. Current status: Stopped
2015-01-04 10:54:02.8069 INFO Service stopped
2015-01-04 10:54:02.9120 INFO Waiting for service to start. Current status: StartPending
2015-01-04 10:54:03.9132 INFO Waiting for service to start. Current status: Running
2015-01-04 10:54:04.9152 INFO Service started
2015-01-04 10:54:05.6903 INFO Distribution service listening on: 10933
2015-01-04 11:00:25.8693 INFO Distribution service listening on: 10933
2015-01-04 11:00:51.8776 INFO Distribution service listening on: 10933
2015-01-04 11:01:27.9809 ERROR Client failed authentication
System.Security.Authentication.AuthenticationException: A call to SSPI failed, see inner exception. —> System.ComponentModel.Win32Exception: The handle specified is invalid
— End of inner exception stack trace —
at System.Net.Security.SslState.StartSendAuthResetSignal(ProtocolToken message, AsyncProtocolRequest asyncRequest, Exception exception)
at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ProcessAuthentication(LazyAsyncResult lazyResult)
at Pipefish.Transport.SecureTcp.Server.SecureTcpServer.ExecuteRequest(TcpClient client)
2015-01-04 11:02:27.9151 INFO Distribution service listening on: 10933
’’’
Hi,
Sorry this is causing so much pain.
In 3.0 we are updating how we update the Tentacles, and this issue should go away. Currently there can be a lag time between the restart and notifying Octopus that the restart finished.
You have figured out the ‘workaround’. By running a health check on the machines you believe have been upgraded it will get their version and mark them as being upgraded.
Vanessa
