Hi there,
We’re trying to upgrade a tentacle hosted on a VM in Azure, via the Management Web UI which is failing. The tentacle is currently on version 3.0.6.2140.
From the logs I can see that there are consistent timeouts when trying to connect back to our Octopus Deploy server all of which seem to be going to different port numbers.
2015-10-13 12:12:41.5946 15 INFO listen://0.0.0.0:10933/ 15 Unhandled error when handling request from client: [ourIP]:55437
2015-10-13 13:12:46.1574 24 INFO listen://0.0.0.0:10933/ 24 Unhandled error when handling request from client: [ourIP]:55601
2015-10-13 11:13:04.7757 5 INFO listen://0.0.0.0:10933/ 5 Unhandled error when handling request from client: [ourIP]:55273
A full sample of a set of log messages is below:
2015-10-13 15:39:34.7807 7 INFO listen://0.0.0.0:10933/ 7 Accepted TCP client: [ourIP]:56012
2015-10-13 15:39:34.7807 21 INFO listen://0.0.0.0:10933/ 21 Performing SSL (TLS 1.0) server handshake
2015-10-13 15:39:34.8276 21 INFO listen://0.0.0.0:10933/ 21 Secure connection established, client is not yet authenticated
2015-10-13 15:39:34.9057 21 INFO listen://0.0.0.0:10933/ 21 Client at [ourIP]:56012 authenticated as BC48816DEB59AF9333153B48246092F31DE9D29B
2015-10-13 15:49:39.9954 21 INFO listen://0.0.0.0:10933/ 21 Unhandled error when handling request from client: [ourIP]:56012
System.IO.IOException: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. —> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
— End of inner exception stack trace —
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
at System.Net.FixedSizeReader.ReadPacket(Byte[] buffer, Int32 offset, Int32 count)
at System.Net.Security._SslStream.StartFrameHeader(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security._SslStream.StartReading(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security._SslStream.ProcessRead(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslStream.Read(Byte[] buffer, Int32 offset, Int32 count)
at System.IO.StreamReader.ReadBuffer()
at System.IO.StreamReader.ReadLine()
at Halibut.Transport.Protocol.MessageExchangeStream.ReadLine() in y:\work\7ab39c94136bc5c6\source\Halibut\Transport\Protocol\MessageExchangeStream.cs:line 91
at Halibut.Transport.Protocol.MessageExchangeStream.ExpectNextOrEnd() in y:\work\7ab39c94136bc5c6\source\Halibut\Transport\Protocol\MessageExchangeStream.cs:line 65
at Halibut.Transport.Protocol.MessageExchangeProtocol.ProcessClientRequests(Func2 incomingRequestProcessor) in y:\work\7ab39c94136bc5c6\source\Halibut\Transport\Protocol\MessageExchangeProtocol.cs:line 110 at Halibut.Transport.Protocol.MessageExchangeProtocol.ExchangeAsServer(Func2 incomingRequestProcessor, Func`2 pendingRequests) in y:\work\7ab39c94136bc5c6\source\Halibut\Transport\Protocol\MessageExchangeProtocol.cs:line 90
at Halibut.Transport.SecureListener.ExecuteRequest(TcpClient client) in y:\work\7ab39c94136bc5c6\source\Halibut\Transport\SecureListener.cs:line 122
Is there a port range that needs to be opened on the Octopus Deploy server for the tentacle upgrade? Or ideally, can we specify a single port to use?
Thanks,
Steve
Hi Stephen,
Thanks for getting in touch!
The upgrade doesn’t require a different port, however while the upgrade is in progress, you may not be able to communicate with the Tentacle. The Octopus Tentacle is listening on port 10933, but it looks like the server is unable to connect - at least for a period of time. The differing port numbers are the source port on the Octopus Server rather than the target port on the Tentacle.
We do have a current issue that may be causing the upgrade to fail, but just in case this is something else, are you able to give me some more info?
- Are you able to communicate with the Tentacle on port 10933 at all? The Listening Tentacle Troubleshooting guide may be of assistance here.
- Is the Tentacle service running as the default Local System account?
- Are there any errors in the Windows Event log on the Tentacle machine?
Thanks,
Damo
Hi Damo,
Sorry if my original email message wasn’t clear. The error messages are from the Tentacle’s logs and report an issue connecting back to the Octopus Server that is hosted on-site by us and is behind a firewall. [ourIP] is the redacted IP address of the server.
In answer to your questions:
- under normal circumstances (i.e. deploying) we can communicate with the tentacle perfectly.
- it’s running as “.\OctopusService”, which I guess was created on installation?
- yes, pasted below.
Thanks for your help with this.
Steve
Windows Events logs =>
The following appear fairly consistently together (i.e. at around the same time).
Information: “.NET Runtime version 4.0.30319.34209 - The profiler has requested that the CLR instance not load the profiler into this process. Profiler CLSID: ‘{71DA0A04-7777-4EC6-9643-7D28B46A8A41}’. Process ID (decimal): 3836. Message ID: [0x2516].”
Warning: “Windows detected your registry file is still in use by other applications or services. The file will be unloaded now. The applications or services that hold your registry file may not function properly afterwards.
DETAIL -
3 user registry handles leaked from \Registry\User\S-1-5-21-2131567522-1623411638-174046295-1005:
Process 6004 (\Device\HarddiskVolume1\Octopus\Upgrade\20151013151820-SDESQ\Octopus.Upgrader.exe) has opened key \REGISTRY\USER\S-1-5-21-2131567522-1623411638-174046295-1005
Process 6004 (\Device\HarddiskVolume1\Octopus\Upgrade\20151013151820-SDESQ\Octopus.Upgrader.exe) has opened key \REGISTRY\USER\S-1-5-21-2131567522-1623411638-174046295-1005\Control Panel\International
Process 5632 (\Device\HarddiskVolume1\Windows\System32\conhost.exe) has opened key \REGISTRY\USER\S-1-5-21-2131567522-1623411638-174046295-1005\Control Panel\International”
Warning: “Windows detected your registry file is still in use by other applications or services. The file will be unloaded now. The applications or services that hold your registry file may not function properly afterwards.
DETAIL -
1 user registry handles leaked from \Registry\User\S-1-5-21-2131567522-1623411638-174046295-1005_Classes:
Process 6004 (\Device\HarddiskVolume1\Octopus\Upgrade\20151013151820-SDESQ\Octopus.Upgrader.exe) has opened key \REGISTRY\USER\S-1-5-21-2131567522-1623411638-174046295-1005_CLASSES”
Error: “Application: Octopus.Upgrader.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: System.ComponentModel.Win32Exception
Stack:
at Octopus.Upgrader.Program.Main(System.String[])”
Error: "Faulting application name: Octopus.Upgrader.exe, version: 0.0.0.0, time stamp: 0x561c7b25
Faulting module name: KERNELBASE.dll, version: 6.2.9200.17366, time stamp: 0x554d4531
Exception code: 0xe0434352
Fault offset: 0x000000000004aea8
Faulting process id: 0x1774
Faulting application start time: 0x01d105ca6d65ceba
Faulting application path: C:\Octopus\Upgrade\20151013151820-SDESQ\Octopus.Upgrader.exe
Faulting module path: C:\Windows\system32\KERNELBASE.dll
Report Id: aeae93fc-71bd-11e5-9561-000d3a231b9e
Faulting package full name:
Faulting package-relative application ID: "
Information: "Fault bucket , type 0
Event Name: CLR20r3
Response: Not available
Cab Id: 0
Problem signature:
P1: Octopus.Upgrader.exe
P2: 0.0.0.0
P3: 561c7b25
P4: System
P5: 4.0.30319.34239
P6: 53e4531e
P7: 400e
P8: 2a9
P9: System.ComponentModel.Win32
P10:
Attached files:
These files may be available here:
C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppCrash_Octopus.Upgrader_f4c240d41c62fdbccc98381324e677d686d959b_0ac57c88
Analysis symbol:
Rechecking for solution: 0
Report Id: aeae93fc-71bd-11e5-9561-000d3a231b9e
Report Status: 4
Hashed bucket: "
Hi Stephen,
It looks like you’ve stumbled across a bug we’re aware of. We’re working on it and will hopefully release a fix soon.
Unfortunately the only workaround at the moment is to manually install the tentacle upgrade on each machine. I’m really sorry about that.
It’s worth noting as well that after an upgrade, the tentacle account will probably be reset to Local System. If you need your Tentacles to run under a different account, you will have to make the change after the upgrade completes.
You can do this manually, or using the following script:
Tentacle.exe service --instance "Tentacle" --reconfigure --username=DOMAIN\ACCOUNT --password=accountpassword --start --console
Again, sorry about the issue and hopefully we’ll have a fix soon!
Damo