2.0 RC3 - Tentacle refuses connection when running as a service account

Hi,

I’m running in to an issue with running a tentacle in Octopus 2 as something other than Local System.

Firstly, cosmetically, in the Tentacle Manager, the tentacle is always listed as “Running as: Local System”, regardless of what account I am using.

Secondly, I cannot initiate communication with the tentacle when using the service account. The connection is refused. I’ve made sure the service account has full control on the install folder and the instance folders. From the server:

System.Net.Sockets.SocketException (0x80004005): No connection could be made because the target machine actively refused it 10.110.13.94:10933
at System.Net.Sockets.TcpClient.Connect(String hostname, Int32 port)
at Pipefish.Transport.SecureTcp.Client.SecureTcpClient.Send(SecureTcpRequest request) in c:\TeamCity\buildAgent\work\cf0b1f41263b24b9\source\Pipefish.Transport.SecureTcp\Client\SecureTcpClient.cs:line 39
at Pipefish.Transport.SecureTcp.MessageExchange.Client.ClientWorker.PerformExchange() in c:\TeamCity\buildAgent\work\cf0b1f41263b24b9\source\Pipefish.Transport.SecureTcp\MessageExchange\Client\ClientWorker.cs:line 298
at Pipefish.Transport.SecureTcp.MessageExchange.Client.ClientWorker.Run() in c:\TeamCity\buildAgent\work\cf0b1f41263b24b9\source\Pipefish.Transport.SecureTcp\MessageExchange\Client\ClientWorker.cs:line 173

From the tentacle logs, it would appear that I can load the cert ok, but the handshake fails when the service attempts to stop itself.

2013-12-23 19:33:02.3357 DEBUG Octopus version: 2.0.6 / c53995db/master
2013-12-23 19:33:02.5657 INFO Resolving for Pipefish.Hosting.ActivitySpace
2013-12-23 19:33:02.9927 TRACE Attaching Dispatcher as Dispatcher
2013-12-23 19:33:03.0297 TRACE Attaching TentacleRestarter as TentacleRestarter-AQ-eBQ_Cm0l
2013-12-23 19:33:03.0357 TRACE Attaching Undeliverable as Undeliverable
2013-12-23 19:33:03.0357 TRACE Attaching Clock as Clock
2013-12-23 19:33:03.0357 TRACE Attaching DeploymentMutex as Octopus.DeploymentMutex
2013-12-23 19:33:03.0357 DEBUG Starting activity space
2013-12-23 19:33:03.0357 TRACE Constructing subscription for SQ-SQL-IB-EB5BF1AE at C:\Octopus\Tentacle\Messages\SQ-SQL-IB-EB5BF1AE
2013-12-23 19:33:03.0837 TRACE Constructing subscription for SQ-OCTOPUS2-BCB5D28C at C:\Octopus\Tentacle\Messages\SQ-OCTOPUS2-BCB5D28C
2013-12-23 19:33:03.0837 TRACE Adding GET route: /
2013-12-23 19:33:03.0837 TRACE Adding POST route: /mx/v1
2013-12-23 19:33:03.0837 TRACE Adding POST route: /handshake
2013-12-23 19:33:03.0837 TRACE Route allows unauthorized access
2013-12-23 19:33:03.1077 DEBUG Loading certificate with thumbprint: DB1D3AF000354691AFC85E70ACA5CCEDC47D2DD5
2013-12-23 19:33:03.1117 DEBUG Certificate was found in store
2013-12-23 19:33:03.1117 INFO Distribution service listening on: 10933
2013-12-23 19:33:08.0682 TRACE Stored Pipefish.Messages.Timing.TimeoutElapsedEvent from Clock@SQ-SQL-IB-EB5BF1AE to TentacleRestarter-AQ-eBQ_Cm0l@SQ-SQL-IB-EB5BF1AE (id: 33a57953b1384660ba2e713f5cb90d37 env: 08D0CE5F3BDB234300000001)
2013-12-23 19:33:08.1892 INFO The Tentacle restarted running version 2.0.6.950.
2013-12-23 19:33:08.1892 TRACE Detected a Octopus.Platform.Deployment.Logging.LogMessageEvent message addressed to an anonymous actor; dropping.
2013-12-23 19:33:08.1892 TRACE Supervised actor TentacleRestarter-AQ-eBQ_Cm0l@SQ-SQL-IB-EB5BF1AE succeeding
2013-12-23 19:33:08.1892 TRACE Starting termination of supervised actor TentacleRestarter-AQ-eBQ_Cm0l@SQ-SQL-IB-EB5BF1AE
2013-12-23 19:33:08.1892 TRACE Completing supervised actor TentacleRestarter-AQ-eBQ_Cm0l@SQ-SQL-IB-EB5BF1AE immediately
2013-12-23 19:33:08.2052 TRACE Detaching TentacleRestarter-AQ-eBQ_Cm0l@SQ-SQL-IB-EB5BF1AE
2013-12-23 19:33:08.2052 TRACE Finished 0593f60641344672bfd5f838c5c6d268
2013-12-23 19:33:08.2052 TRACE Detected a Octopus.Platform.Deployment.Logging.ProgressMessageEvent message addressed to an anonymous actor; dropping.
2013-12-23 19:33:08.2052 TRACE Stored Pipefish.Messages.Timing.ClearTimeoutsCommand from TentacleRestarter-AQ-eBQ_Cm0l@SQ-SQL-IB-EB5BF1AE to Clock@SQ-SQL-IB-EB5BF1AE (id: 28c89864f79a49ddadff4d468383fe9f env: 08D0CE5F3BF1B92300000002)
2013-12-23 19:33:08.2052 TRACE TentacleRestarter-AQ-eBQ_Cm0l@SQ-SQL-IB-EB5BF1AE successfully detached
2013-12-23 19:33:08.5302 TRACE Accepted TCP client 10.110.13.200:55893
2013-12-23 19:33:08.5992 TRACE Routing Post request for /handshake…
2013-12-23 19:33:08.5992 TRACE Found handler.
2013-12-23 19:33:08.6402 TRACE Stored Octopus.Platform.Deployment.Messages.Restart.TentacleRestartCommand from Anonymous@SQ-SQL-IB-EB5BF1AE to Dispatcher@SQ-SQL-IB-EB5BF1AE (id: 2ecdeb0bd6fa455d9fb59dc9043a1e07 env: 08D0CE5F3C3293C800000003)
2013-12-23 19:33:08.6652 TRACE Attaching TentacleRestarter as TentacleRestarter-AQ-eBURs1hP
2013-12-23 19:33:08.6652 TRACE Stored Octopus.Platform.Deployment.Messages.Restart.TentacleRestartCommand from Anonymous@SQ-SQL-IB-EB5BF1AE to TentacleRestarter-AQ-eBURs1hP@SQ-SQL-IB-EB5BF1AE (id: 2ecdeb0bd6fa455d9fb59dc9043a1e07 env: 08D0CE5F3C37012200000004)
2013-12-23 19:33:08.6822 TRACE Starting supervised actor TentacleRestarter-AQ-eBURs1hP@SQ-SQL-IB-EB5BF1AE
2013-12-23 19:33:08.6822 TRACE Stored Pipefish.Messages.Timing.SetTimeoutCommand from TentacleRestarter-AQ-eBURs1hP@SQ-SQL-IB-EB5BF1AE to Clock@SQ-SQL-IB-EB5BF1AE (id: ed4fb015286645518d4145443ffc770e env: 08D0CE5F3C394AA400000005)
2013-12-23 19:33:08.6902 INFO The Tentacle has received a request to shut down and restart
2013-12-23 19:33:08.6902 TRACE Detected a Octopus.Platform.Deployment.Logging.LogMessageEvent message addressed to an anonymous actor; dropping.
2013-12-23 19:33:08.6902 TRACE Stored Pipefish.Messages.Timing.SetTimeoutCommand from TentacleRestarter-AQ-eBURs1hP@SQ-SQL-IB-EB5BF1AE to Clock@SQ-SQL-IB-EB5BF1AE (id: 4425823c77f64ba5be5c80d58e31319a env: 08D0CE5F3C3C310100000006)
2013-12-23 19:33:08.7142 DEBUG Stopping activity space
2013-12-23 19:33:12.5235 DEBUG Octopus version: 2.0.6 / c53995db/master
2013-12-23 19:33:12.5655 INFO Stopping service…

This user is purposefully not an administrative user. If I install the service using the --username and --password option (which would be nice to have in the config UI), does the tentacle service get the proper DACL set so it can start and stop itself?

Thanks,
Matt

Hi - it does look as if a restart issue is involved here - I don’t think the DACL you mention is applied.

If you manually restart the service after making the configuration change, does the service start back up successfully?

You may also need to set up URL ACLs with netsh, but that’s not the issue here as far as I can tell from the log.

Regards,
Nick

Hi Nick,

If I manually restart the service, it will get a handshake from the server and attempt to restart itself again. It appears that there is some additional configuration that happens that a manual restart does not replicate.

If you need me to run some tests, I can definitely do that. These servers are not production machines at this point.

I didn’t need to set up URL ACLs.

Interesting - cooking up a few theories here. One oddity is the multiple handshakes: this should really only happen once.

Tentacle uses a folder under its home directory called “Messages” to store/forward messages to and from the Octopus server. A couple of things could cause multiple handshakes - either:

a) for some reason Tentacle’s failing to write to this folder; or,
b) Tentacle wrote a message to the folder as admin, but can’t delete it as the un-privileged user

Neither is super-convincing, but to rule it out we can:

  1. manually stop the Tentacle service
  2. delete all files under Messages, and also under Actors
  3. start the service

Let me know how you go, I’ll have a think what else could be at the bottom of this.

Regards,
Nick

I haven’t gotten back to this specific situation due to time constraints. For now I have set the Tentacle’s service account to be a local admin, but when I circle back around to change that I’ll let you know.

-Matt

I had the same issue.

I tried manually stopping the tentacle service, deleting all files under Messages and Actors and re-starting service but this did not work.

Tried re-install but got the same error.

After a few minutes (while searching for a solution on-line) it seemed to finally connect! Looks like there is some delay in re-setting the connection with another user?