Tentacle Won't Start After Stopped

Hello,
Today, we had a tentacle that thought that it was waiting on a task to complete when there was no task running on it in Octopus manager. We attempted to fix this issue by restarting the tentacle service, but found that the tentacle would not start. See the exceptions from the Event Viewer below.

Error 1:
Faulting application name: Tentacle.exe, version: 3.0.6.2130, time stamp: 0x55b53b97
Faulting module name: KERNELBASE.dll, version: 6.3.9600.17278, time stamp: 0x53eebf2e
Exception code: 0xe0434352
Fault offset: 0x000000000000606c
Faulting process id: 0x104c
Faulting application start time: 0x01d1010df240d822
Faulting application path: C:\Program Files\Octopus Deploy\Tentacle\Tentacle.exe
Faulting module path: C:\Windows\system32\KERNELBASE.dll
Report Id: 31ae220a-6d01-11e5-80d5-005056b33286
Faulting package full name:
Faulting package-relative application ID:

Error2:
Application: Tentacle.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: System.Exception
Stack:
at Octopus.Shared.Startup.WindowsServiceAdapter.RunService()
at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
at System.Threading.ThreadHelper.ThreadStart()

Error 3:

2015-10-07 09:39:20.8212 6 FATAL Unhandled AppDomain exception occurred: A required communications port is already in use. http://g.octopushq.com/HttpPortInUse See: http://g.octopushq.com/HttpPortInUse
System.Exception: A required communications port is already in use. http://g.octopushq.com/HttpPortInUse See: http://g.octopushq.com/HttpPortInUse —> System.Net.Sockets.SocketException: Only one usage of each socket address (protocol/network address/port) is normally permitted
at System.Net.Sockets.Socket.DoBind(EndPoint endPointSnapshot, SocketAddress socketAddress)
at System.Net.Sockets.Socket.Bind(EndPoint localEP)
at System.Net.Sockets.TcpListener.Start(Int32 backlog)
at Halibut.Transport.SecureListener.Start() in y:\work\7ab39c94136bc5c6\source\Halibut\Transport\SecureListener.cs:line 43
at Octopus.Tentacle.Communications.HalibutInitializer.Start() in y:\work\refs\heads\master\source\Octopus.Tentacle\Communications\HalibutInitializer.cs:line 39
at Octopus.Tentacle.Commands.RunAgentCommand.Start() in y:\work\refs\heads\master\source\Octopus.Tentacle\Commands\RunAgentCommand.cs:line 68
at Octopus.Shared.Startup.AbstractCommand.Octopus.Shared.Startup.ICommand.Start(String[] commandLineArguments, ICommandRuntime commandRuntime, OptionSet commonOptions) in y:\work\refs\heads\master\source\Octopus.Shared\Startup\AbstractCommand.cs:line 57
at Octopus.Shared.Startup.WindowsServiceHost.<>c__DisplayClass2.b__0() in y:\work\refs\heads\master\source\Octopus.Shared\Startup\WindowsServiceHost.cs:line 21
at Octopus.Shared.Startup.WindowsServiceAdapter.RunService() in y:\work\refs\heads\master\source\Octopus.Shared\Startup\WindowsServiceAdapter.cs:line 44
— End of inner exception stack trace —
at Octopus.Shared.Startup.WindowsServiceAdapter.RunService() in y:\work\refs\heads\master\source\Octopus.Shared\Startup\WindowsServiceAdapter.cs:line 53
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart()

Hi,

Thanks for reaching out. The error seems to be this: Unhandled AppDomain exception occurred: A required communications port is already in use. . Is it possible that you have another process (maybe a 2nd tentacle) listening on the same port as this Tentacle? You can check this by running this cmd netstat -ano.

Regards,

Dalmiro

Hi Dalmiro,
Thanks for the quick reply. I was able to resolve the issue by rebooting the box. When I run netstat –ano now, I just see one PID listening to port 10933. Is it recommended to use a different port for every deployment target?

Hi Bryan,

Yes. Each Tentacle instance should use a different port. Glad to hear a magic reboot fixed it for you :slight_smile:

Thanks,

Dalmiro

Just to be clear, I am talking about tentacles on different physical machines listening to the same port. From what I know about listening, as long as there’s nothing on the same box listening to the same port as the tentacle, it should work. Or is that incorrect?

as long as there’s nothing on the same box listening to the same port as the tentacle, it should work

Correct :). If they are on different machines, they can use the same ports.

Dalmiro

So back to the original issue… Any idea what could have caused this error?

Hi Bryan,

Not besides the “other process using the same port”. If this happens again, please let us know so we can investigate it further. Since now its working, its a bit difficult to figure out what went wrong back then :slight_smile:

Regards,

Dalmiro.

I’ve had the same issue today, same error as above. Documenting what I found in case it helps others.

TCPView (1.png) showed a non-existent process (previously the tentacle executable) still has a socket open on port 10933. This is what causes the starting up of the service to fail.

I then noticed in process explorer (2.png) that an instance of Calamari was still running, under a powershell process. This was for a hung task that I’d cancelled in the web gui but apparently was still running.

Killing this process caused the port to be freed up, so it looks like because the tentacle process launched this other process, the existence of the child process prevented the cleanup of the tentacle sockets.

Thanks for sharing your troubleshooting process :slight_smile:

Same issue.

Killed the powershell process that had the ports locked then which allowed the Octopus Tentacle to start. Running 3.3.12

We’re having tentacle lock-up issues too, in two different forms. One of those forms seems to match the errors as posted by the original poster. In our case I believe we were doing a Tentacle upgrade from 3.3.1 to 3.3.11. I have an event log call chain that might be helpful so here it is. I trimmed out excess lines to shorten my post. If you need more details let me know.

This happened on node “Apollo”.

/* --------------- */
06/07/2016 01:54:21 PM
LogName=System
TaskCategory=The operation completed successfully.
OpCode=The operation completed successfully.
Message=The OctopusDeploy Tentacle service entered the stopped state.

/* --------------- */
06/07/2016 01:54:22 PM
LogName=Application
SourceName=MsiInstaller
Message=Beginning a Windows Installer transaction: D:\Octopus\Upgrade\20160607205312-QEP87\Octopus.Tentacle-x64.msi. Client Process Id: 5468.

/* --------------- */
06/07/2016 01:54:24 PM
LogName=Application
SourceName=Microsoft-Windows-RestartManager
Message=Application ‘D:\Octopus Deploy\Tentacle\Tentacle.exe’ (pid 2112) cannot be restarted

  • Application SID does not match Conductor SID…

/* --------------- */
06/07/2016 01:54:24 PM
LogName=Application
Type=Error
Message=Application or service ‘Octopus Deploy: Tentacle’ could not be shut down.

/* --------------- */
06/07/2016 01:54:24 PM
LogName=Application
Type=Error
Message=Application or service ‘Octopus Deploy: Tentacle’ could not be shut down.

/* --------------- */
06/07/2016 01:54:25 PM
LogName=Application
Message=Application ‘D:\Octopus Deploy\Tentacle\Tentacle.exe’ (pid 2112) cannot be restarted- Application SID does not match Conductor SID…

/* --------------- */
06/07/2016 01:54:27 PM
LogName=Application
SourceName=MsiInstaller
Message=Product: Octopus Deploy Tentacle – Installation completed successfully.

/* --------------- */
06/07/2016 01:54:27 PM
LogName=Application
SourceName=MsiInstaller
Message=Windows Installer installed the product. Product Name: Octopus Deploy Tentacle. Product Version: 3.3.11. Product Language: 1033. Manufacturer: Octopus Deploy Pty. Ltd… Installation success or error status: 0.

/* --------------- */
06/07/2016 01:54:27 PM
LogName=Application
SourceName=MsiInstaller
Message=Ending a Windows Installer transaction: D:\Octopus\Upgrade\20160607205312-QEP87\Octopus.Tentacle-x64.msi. Client Process Id: 5468.

/* --------------- */
06/07/2016 01:54:28 PM
LogName=System
TaskCategory=The operation completed successfully.
OpCode=The operation completed successfully.
Message=The OctopusDeploy Tentacle service entered the running state.

/* --------------- */
06/07/2016 01:54:30 PM
LogName=Application
SourceName=Octopus Tentacle
EventCode=0
EventType=2
Type=Error
TaskCategory=%1
OpCode=Info
RecordNumber=9975881
Keywords=Classic
Message=2016-06-07 13:54:29.9993 7 FATAL Unhandled AppDomain exception occurred: A required communications port is already in use. http://g.octopushq.com/HttpPortInUse See: http://g.octopushq.com/HttpPortInUse
System.Exception: A required communications port is already in use. http://g.octopushq.com/HttpPortInUse See: http://g.octopushq.com/HttpPortInUse —> System.Net.Sockets.SocketException
: Only one usage of each socket address (protocol/network address/port) is normally permitted at System.Net.Sockets.Socket.DoBind(EndPoint endPointSnapshot, SocketAddress socketAddress)
at System.Net.Sockets.Socket.Bind(EndPoint localEP)
at System.Net.Sockets.TcpListener.Start(Int32 backlog)
at Halibut.Transport.SecureListener.Start() in y:\work\7ab39c94136bc5c6\source\Halibut\Transport\SecureListener.cs:line 47
at Octopus.Tentacle.Communications.HalibutInitializer.Start() in Y:\work\refs\tags\3.3.11
source\Octopus.Tentacle\Communications\HalibutInitializer.cs:line 39
at Octopus.Tentacle.Commands.RunAgentCommand.Start() in Y:\work\refs\tags\3.3.11\source\Octopus.Tentacle\Commands\RunAgentCommand.cs:line 80
at Octopus.Shared.Startup.AbstractCommand.Octopus.Shared.Startup.ICommand.Start(String[] commandLineArguments, ICommandRuntime commandRuntime, OptionSet commonOptions) in Y:\work\refs
\tags\3.3.11\source\Octopus.Shared\Startup\AbstractCommand.cs:line 57
at Octopus.Shared.Startup.WindowsServiceHost.<>c__DisplayClass1_0.b__0() in Y:\work\refs\tags\3.3.11\source\Octopus.Shared\Startup\WindowsServiceHost.cs:line 19
at Octopus.Shared.Startup.WindowsServiceAdapter.RunService() in Y:\work\refs\tags\3.3.11\source\Octopus.Shared\Startup\WindowsServiceAdapter.cs:line 44
— End of inner exception stack trace —
at Octopus.Shared.Startup.WindowsServiceAdapter.RunService() in Y:\work\refs\tags\3.3.11\source\Octopus.Shared\Startup\WindowsServiceAdapter.cs:line 53
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart()

/* --------------- */
06/07/2016 01:54:30 PM
LogName=Application
SourceName=.NET Runtime
EventCode=1026
EventType=2
Type=Error
TaskCategory=None
OpCode=Info
RecordNumber=9975882
Keywords=Classic
Message=Application: Tentacle.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: System.Exception
Stack:
at Octopus.Shared.Startup.WindowsServiceAdapter.RunService()
at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System
.Threading.ContextCallback, System.Object, Boolean)
at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
at System.Threading.ThreadHelper.ThreadStart()

/* --------------- */
06/07/2016 01:54:30 PM
LogName=Application
SourceName=Application Error
EventCode=1000
EventType=2
Type=Error
TaskCategory=Application Crashing Events
OpCode=Info
RecordNumber=9975883
Keywords=Classic
Message=Faulting application name: Tentacle.exe, version: 3.3.11.0, time stamp: 0x572834e5
Faulting module name: KERNELBASE.dll, version: 6.1.7601.23392, time stamp: 0x56eb3604
Exception code: 0xe0434352
Fault offset: 0x000000000001a06d
Faulting process id: 0x1d30
Faulting application start time: 0x01d1c0fec805a0e2
Faulting application path: D:\Octopus Deploy\Tentacle\Tentacle.exe
Faulting module path: C:\windows\system32\KERNELBASE.dll
Report Id: 072a46df-2cf2-11e6-8b8c-e839352a3268

/* --------------- */
06/07/2016 01:54:32 PM
LogName=System
SourceName=Microsoft-Windows-Service Control Manager
EventCode=7034
EventType=2
Type=Error
TaskCategory=The operation completed successfully.
OpCode=The operation completed successfully.
RecordNumber=211027
Keywords=Classic
Message=The OctopusDeploy Tentacle service terminated unexpectedly. It has done this 1 time(s).

Hi @rjhansen ,

  • How many Instances of the Tentacle are you running on that machine?
  • Have you already tried to check what is using that port and then terminating that process?

Thanks,
Dalmiro

Only one Tentacle instance. None of our boxes have multiple Tentacle instances. I’m pretty sure we were able to terminate and restart the process on this box without a reboot. However on some of our boxes we were only able to re-launch the process by rebooting the box. Some of our boxes are 24/7 and are on a reboot schedule so it is not always possible to reboot.

Tentacle is the only thing using that port. One instance of Tentacle.

Hi,

We brought this conversation once again to the team and decided to create a task to investigate this in more depth: https://github.com/OctopusDeploy/Issues/issues/2573 .

Please be aware that this might be one of those tasks that take some time to be able to consistently reproduce, and then (if possible) find a fix for it. Another developer might get back to this conversation and ask for more details to aid in his troubleshooting quest.

Best regards,
Dalmiro

Okay, great. For the sake of completeness here is the event log trail for the other node, Brizo. It is similar to the first node, Apollo, in that it is causing the Tentacle to stop unexpectedly. It is different though in that it is NOT giving us the “required communications port is already in use” error message. So maybe that will be helpful. Same situation with this node, one listening Tentacle on default port.

This happened on node “Brizo”.

/* --------------- */
06/27/2016 02:44:20 PM
LogName=System
TaskCategory=The operation completed successfully.
OpCode=The operation completed successfully.
Message=The OctopusDeploy Tentacle service entered the stopped state.

/* --------------- */
06/27/2016 02:45:01 PM
LogName=System
TaskCategory=The operation completed successfully.
OpCode=The operation completed successfully.
Message=The OctopusDeploy Tentacle service terminated unexpectedly. It has done this 1 time(s).

/* --------------- */
06/27/2016 02:45:01 PM
LogName=System
TaskCategory=The operation completed successfully.
OpCode=The operation completed successfully.
Message=The OctopusDeploy Tentacle service failed to start due to the following error:
The account name is invalid or does not exist, or the password is invalid for the account name specified.

/* --------------- */
06/27/2016 03:03:48 PM
LogName=Security
OpCode=Info
RecordNumber=6074525
Keywords=Audit Success
Message=A new process has been created.
New Process ID: 0x11f0
New Process Name: D:\Octopus Deploy\Tentacle\Octopus.Manager.Tentacle.exe
Token Elevation Type: TokenElevationTypeLimited (3)
Creator Process ID: 0x19d0
Process Command Line:
Token Elevation Type indicates the type of token that was assigned to the new process in accordance with User Account Control policy.
Type 1 is a full token with no privileges removed or groups disabled. A full token is only used if User Account Control is disabled or if the user is the built-in Administrator account or a service account.
Type 2 is an elevated token with no privileges removed or groups disabled. An elevated token is used when User Account Control is enabled and the user chooses to start the program using
Run as administrator. An elevated token is also used when an application is configured to always require administrative privilege or to always require maximum privilege, and the user is a member of the Administrators group.
Type 3 is a limited token with administrative privileges removed and administrative groups disabled. The limited token is used when User Account Control is enabled, the application does not require administrative privilege, and the user does not choose to start the program using Run as administrator.

/* --------------- */
06/27/2016 03:03:48 PM
LogName=Security
TaskCategory=Sensitive Privilege Use
OpCode=Info
Keywords=Audit Failure
Message=A privileged service was called.
Server: Security
Process ID: 0x11f0
Process Name: D:\Octopus Deploy\Tentacle\Octopus.Manager.Tentacle.exe
Service Request Information:
Privileges: SeCreateGlobalPrivilege

/* --------------- */
06/27/2016 03:03:50 PM
LogName=Security
TaskCategory=Process Creation
OpCode=Info
Keywords=Audit Success
Message=A new process has been created.
New Process ID: 0x1510
New Process Name: D:\Octopus Deploy\Tentacle\Octopus.Manager.Tentacle.exe
Token Elevation Type: TokenElevationTypeFull (2)
Creator Process ID: 0x11f0
Process Command Line:
Token Elevation Type indicates the type of token that was assigned to the new process in accordance with User Account Control policy.
Type 1 is a full token with no privileges removed or groups disabled. A full token is only used if User Account Control is disabled or if the user is the built-in Administrator account or a service account.
Type 2 is an elevated token with no privileges removed or groups disabled. An elevated token is used when User Account Control is enabled and the user chooses to start the program using Run as administrator. An elevated token is also used when an application is configured to always require administrative privilege or to always require maximum privilege, and the user is a member of the Administrators group.
Type 3 is a limited token with administrative privileges removed and administrative groups disabled. The limited token is used when User Account Control is enabled, the application does not require administrative privilege, and the user does not choose to start the program using Run as administrator.

/* --------------- */
06/27/2016 03:04:04 PM
LogName=Security
TaskCategory=Process Creation
OpCode=Info
RecordNumber=6074569
Keywords=Audit Success
Message=A new process has been created.
New Process ID: 0xaa8
New Process Name: D:\Octopus Deploy\Tentacle\Tentacle.exe
Token Elevation Type: TokenElevationTypeFull (2)
Creator Process ID: 0x1510
Process Command Line:
Token Elevation Type indicates the type of token that was assigned to the new process in accordance with User Account Control policy.
Type 1 is a full token with no privileges removed or groups disabled. A full token is only used if User Account Control is disabled or if the user is the built-in Administrator account or a service account.
Type 2 is an elevated token with no privileges removed or groups disabled. An elevated token is used when User Account Control is enabled and the user chooses to start the program using Run as administrator. An elevated token is also used when an application is configured to always require administrative privilege or to always require maximum privilege, and the user is a member of the Administrators group.
Type 3 is a limited token with administrative privileges removed and administrative groups disabled. The limited token is used when User Account Control is enabled, the application does not require administrative privilege, and the user does not choose to start the program using Run as administrator.

/* --------------- */
06/27/2016 03:04:04 PM
LogName=Security
TaskCategory=Process Creation
OpCode=Info
Keywords=Audit Success
Message=A new process has been created.
New Process ID: 0x1b80
New Process Name: D:\Octopus Deploy\Tentacle\Tentacle.exe
Token Elevation Type: TokenElevationTypeDefault (1)
Creator Process ID: 0x304
Process Command Line:
Token Elevation Type indicates the type of token that was assigned to the new process in accordance with User Account Control policy.
Type 1 is a full token with no privileges removed or groups disabled. A full token is only used if User Account Control is disabled or if the user is the built-in Administrator account or a service account.
Type 2 is an elevated token with no privileges removed or groups disabled. An elevated token is used when User Account Control is enabled and the user chooses to start the program using Run as administrator. An elevated token is also used when an application is configured to always require administrative privilege or to always require maximum privilege, and the user is a member of the Administrators group.
Type 3 is a limited token with administrative privileges removed and administrative groups disabled. The limited token is used when User Account Control is enabled, the application does not require administrative privilege, and the user does not choose to start the program using Run as administrator.

/* --------------- */
06/27/2016 03:04:05 PM
LogName=System
TaskCategory=The operation completed successfully.
OpCode=The operation completed successfully.
Message=The OctopusDeploy Tentacle service entered the running state.

/* --------------- */
06/27/2016 04:42:05 PM
LogName=Security
TaskCategory=Process Creation
OpCode=Info
Keywords=Audit Success
Message=A new process has been created.
New Process ID: 0xaf0
New Process Name: D:\Octopus Deploy\Tentacle\Octopus.Manager.Tentacle.exe
Token Elevation Type: TokenElevationTypeLimited (3)
Creator Process ID: 0x19b0
Process Command Line:
Token Elevation Type indicates the type of token that was assigned to the new process in accordance with User Account Control policy.
Type 1 is a full token with no privileges removed or groups disabled. A full token is only used if User Account Control is disabled or if the user is the built-in Administrator account or a service account.
Type 2 is an elevated token with no privileges removed or groups disabled. An elevated token is used when User Account Control is enabled and the user chooses to start the program using Run as administrator. An elevated token is also used when an application is configured to always require administrative privilege or to always require maximum privilege, and the user is a member of the Administrators group.
Type 3 is a limited token with administrative privileges removed and administrative groups disabled. The limited token is used when User Account Control is enabled, the application does not require administrative privilege, and the user does not choose to start the program using Run as administrator.

/* --------------- */
06/27/2016 04:42:05 PM
LogName=Security
TaskCategory=Sensitive Privilege Use
OpCode=Info
Message=A privileged service was called.
Process ID: 0xaf0
Process Name: D:\Octopus Deploy\Tentacle\Octopus.Manager.Tentacle.exe
Service Request Information:
Privileges: SeCreateGlobalPrivilege

/* --------------- */
06/27/2016 04:42:10 PM
LogName=Security
TaskCategory=Process Creation
OpCode=Info
Keywords=Audit Success
Message=A new process has been created.
New Process ID: 0x324
New Process Name: D:\Octopus Deploy\Tentacle\Octopus.Manager.Tentacle.exe
Token Elevation Type: TokenElevationTypeFull (2)
Creator Process ID: 0xaf0
Process Command Line:
Token Elevation Type indicates the type of token that was assigned to the new process in accordance with User Account Control policy.
Type 1 is a full token with no privileges removed or groups disabled. A full token is only used if User Account Control is disabled or if the user is the built-in Administrator account or a service account.
Type 2 is an elevated token with no privileges removed or groups disabled. An elevated token is used when User Account Control is enabled and the user chooses to start the program using Run as administrator. An elevated token is also used when an application is configured to always require administrative privilege or to always require maximum privilege, and the user is a member of the Administrators group.
Type 3 is a limited token with administrative privileges removed and administrative groups disabled. The limited token is used when User Account Control is enabled, the application does not require administrative privilege, and the user does not choose to start the program using Run as administrator.

Hi,

I have the same issue but I can’t find any process with this PID in Process Explorer (no Powershell or Calamari process).
In TCPView, I can see two non-existent process listenning on this port.

I don’t want to reboot the machine, even if it’s not a production machine, because if it’s happened on a production machine, I don’t want to reboot it.

I want to change the listenning port but it didn’t work (I try to add settings in the tentacle.exe.config).

I spent some more time digging through event logs today. I found that the majority of our “A required communications port is already in use” errors occurred immediately after a Tentacle install. Install the Tentacle software and 2-3 seconds later we get that error. 11 servers had 3.3.1 installed and were upgraded to 3.3.11. 1 server received a fresh install of 3.3.1. 2 servers received a fresh install of 3.3.11. I’m not sure whether or not the original poster was getting the error as a result of a Tentacle install.

We also see this problem more and more frequently. Our theory was that the Tentacle crashes and the dead process still holds on to the port casing the Tentacle not being able to restart. We’ve always restarted the server to solve it but that is not always practical. I will check for powershell or calamari processes the next time.