Tentacle locks up

Leslie · 19 June 2018 15:27

running tentacle version 3.22.0, We have now seen a number of times, where a tentacle locks up. The only thing that seems to correct it is restarting the tentacle. We have noticed, the few times that we have looked, that the server task listed does not exist when we query for it using the API.

here is a snippet of the log from the tentacle

2018-06-18 10:50:20.7739 3824 2018-06-18 10:50:21.6870 3824 2018-06-18 10:50:21.6870 3824 2018-06-18 10:55:22.7642 3824 2018-06-18 11:50:31.7229 3824 2018-06-18 11:50:31.7229 3824 2018-06-18 11:50:31.7229 3824 2018-06-18 11:50:31.7459 3824 2018-06-18 11:50:31.7959 3824 2018-06-18 11:50:32.7750 3824 2018-06-18 11:50:32.7750 3824 2018-06-18 11:55:33.7812 3824 2018-06-18 12:50:53.9271 3824 2018-06-18 12:50:53.9271 3824 2018-06-18 12:50:53.9271 3824 2018-06-18 12:50:53.9491 3824 2018-06-18 12:50:53.9981 3824 2018-06-18 12:50:54.9441 3824 2018-06-18 12:50:54.9441 3824 2018-06-18 12:55:55.9883 3824 2018-06-18 13:49:01.1402 3824 2018-06-18 13:49:01.1402 3824 2018-06-18 13:49:01.1622 3824 2018-06-18 13:49:01.1682 3824 2018-06-18 13:49:01.2242 3824 2018-06-18 13:49:02.1482 3824 2018-06-18 13:49:02.1482 3824 2018-06-18 13:50:58.9767 3824 2018-06-18 13:51:00.0798 3824 2018-06-18 13:51:00.0798 3824 2018-06-18 13:54:03.1634 3824 2018-06-18 13:54:34.8044 3824 2018-06-18 13:54:47.2292 3824 2018-06-18 13:54:47.2312 3824 2018-06-18 13:54:47.2312 3824 2018-06-18 13:54:47.2692 3824 2018-06-18 13:54:47.3292 3824 2018-06-18 13:54:48.2333 3824 2018-06-18 13:54:48.2333 3824 2018-06-18 13:54:53.0836 3824 2018-06-18 13:55:17.9162 3824 2018-06-18 13:55:19.0313 3824 2018-06-18 13:55:19.0313 3824 2018-06-18 14:06:22.6555 3824 2018-06-18 14:07:50.2851 3824 2018-06-18 14:07:51.3972 3824 2018-06-18 14:07:51.3972 3824 2018-06-18 14:08:44.6066 3824 2018-06-18 14:37:35.7203 3824 2018-06-18 14:37:35.7203 3824 2018-06-18 14:37:35.7343 3824 2018-06-18 14:37:35.7483 3824 2018-06-18 14:37:35.8224 3824 2018-06-18 14:37:36.7448 3824 2018-06-18 14:37:36.7448 3824 2018-06-18 14:37:38.3553 3824 2018-06-18 14:51:33.0473 3824 2018-06-18 14:51:33.0473 3824 2018-06-18 14:51:33.0593 3824 2018-06-18 14:51:33.0753 3824 2018-06-18 14:51:33.1483 3824 2018-06-18 14:51:34.0635 3824 2018-06-18 14:51:34.0635 3824 2018-06-18 14:55:58.3577 3824 2018-06-18 14:55:58.3577 3824 2018-06-18 14:55:58.3727 3824 2018-06-18 14:55:58.3877 3824 2018-06-18 14:55:58.4837 3824 2018-06-18 14:55:59.3578 3824 2018-06-18 14:55:59.3578 3824 2018-06-18 14:56:00.5480 3824 2018-06-18 14:56:34.4053 3824 2018-06-18 15:02:46.8521 3824 2018-06-18 15:02:47.4992 3824 2018-06-18 15:02:47.4992 3824 2018-06-18 15:02:56.2628 3824 2018-06-18 15:51:55.9246 3824 2018-06-18 15:51:55.9246 3824 2018-06-18 15:51:55.9246 3824 2018-06-18 15:51:55.9496 3824 2018-06-18 15:51:56.0126 3824 2018-06-18 15:51:56.9446 3824 2018-06-18 15:51:56.9446 3824 2018-06-18 15:56:57.9528 3824 2018-06-18 16:52:18.2883 3824 2018-06-18 16:52:18.2883 3824 2018-06-18 16:52:18.2943 3824 2018-06-18 16:52:18.3123 3824 2018-06-18 16:52:18.3693 3824 2018-06-18 16:52:19.3123 3824 2018-06-18 16:52:19.3123 3824 2018-06-18 16:57:20.3345 3824 12 INFO [ServerTasks-158836] [RunningScript] [Read Lock] [“ServerTasks-158326” (has a write lock)] Trying to acquire lock.
12 INFO [ServerTasks-158836] [RunningScript] [Read Lock] [“ServerTasks-158326” (has a write lock)] Failed to acquire lock within 00:00:00.1000000.
12 INFO [ServerTasks-158836] [RunningScript] [Read Lock] [“ServerTasks-158326” (has a write lock)] Trying to acquire lock with wait time of 00:05:00.
12 INFO [ServerTasks-158836] [RunningScript] [Read Lock] [“ServerTasks-158326” (has a write lock)] Failed to acquire lock within 00:05:00.
11 INFO listen://[::]:10933/ 11 Accepted TCP client: [::ffff:10.10.108.203]:52391
11 INFO listen://[::]:10933/ 11 Performing TLS server handshake
11 INFO listen://[::]:10933/ 11 Secure connection established, client is not yet authenticated, client connected with Tls12
11 INFO listen://[::]:10933/ 11 Client at [::ffff:10.10.108.203]:52391 authenticated as 48D5945E16C0D599D29B01BE5432D6F3FA65703F
19 INFO [ServerTasks-158842] [RunningScript] [Read Lock] [“ServerTasks-158326” (has a write lock)] Trying to acquire lock.
19 INFO [ServerTasks-158842] [RunningScript] [Read Lock] [“ServerTasks-158326” (has a write lock)] Failed to acquire lock within 00:00:00.1000000.
19 INFO [ServerTasks-158842] [RunningScript] [Read Lock] [“ServerTasks-158326” (has a write lock)] Trying to acquire lock with wait time of 00:05:00.
19 INFO [ServerTasks-158842] [RunningScript] [Read Lock] [“ServerTasks-158326” (has a write lock)] Failed to acquire lock within 00:05:00.
11 INFO listen://[::]:10933/ 11 Accepted TCP client: [::ffff:10.10.108.203]:53085
11 INFO listen://[::]:10933/ 11 Performing TLS server handshake
11 INFO listen://[::]:10933/ 11 Secure connection established, client is not yet authenticated, client connected with Tls12
11 INFO listen://[::]:10933/ 11 Client at [::ffff:10.10.108.203]:53085 authenticated as 48D5945E16C0D599D29B01BE5432D6F3FA65703F
18 INFO [ServerTasks-158854] [RunningScript] [Read Lock] [“ServerTasks-158326” (has a write lock)] Trying to acquire lock.
18 INFO [ServerTasks-158854] [RunningScript] [Read Lock] [“ServerTasks-158326” (has a write lock)] Failed to acquire lock within 00:00:00.1000000.
18 INFO [ServerTasks-158854] [RunningScript] [Read Lock] [“ServerTasks-158326” (has a write lock)] Trying to acquire lock with wait time of 00:05:00.
18 INFO [ServerTasks-158854] [RunningScript] [Read Lock] [“ServerTasks-158326” (has a write lock)] Failed to acquire lock within 00:05:00.
11 INFO listen://[::]:10933/ 11 Accepted TCP client: [::ffff:10.10.108.198]:64853
11 INFO listen://[::]:10933/ 11 Performing TLS server handshake
6 INFO listen://[::]:10933/ 6 Secure connection established, client is not yet authenticated, client connected with Tls12
6 INFO listen://[::]:10933/ 6 Client at [::ffff:10.10.108.198]:64853 authenticated as 48D5945E16C0D599D29B01BE5432D6F3FA65703F
12 INFO [ServerTasks-158861] [RunningScript] [Read Lock] [“ServerTasks-158326” (has a write lock)] Trying to acquire lock.
12 INFO [ServerTasks-158861] [RunningScript] [Read Lock] [“ServerTasks-158326” (has a write lock)] Failed to acquire lock within 00:00:00.1000000.
12 INFO [ServerTasks-158861] [RunningScript] [Read Lock] [“ServerTasks-158326” (has a write lock)] Trying to acquire lock with wait time of 00:05:00.
18 INFO [ServerTasks-158864] [RunningScript] [Read Lock] [“ServerTasks-158326” (has a write lock)] Trying to acquire lock.
18 INFO [ServerTasks-158864] [RunningScript] [Read Lock] [“ServerTasks-158326” (has a write lock)] Failed to acquire lock within 00:00:00.1000000.
18 INFO [ServerTasks-158864] [RunningScript] [Read Lock] [“ServerTasks-158326” (has a write lock)] Trying to acquire lock with wait time of 24.20:31:23.6470000.
12 INFO [ServerTasks-158861] [RunningScript] [Read Lock] [“ServerTasks-158326” (has a write lock)] Failed to acquire lock within 00:05:00.
18 INFO [ServerTasks-158864] [RunningScript] [Read Lock] [“ServerTasks-158326” (has a write lock)] Lock acquire canceled.
11 INFO listen://[::]:10933/ 11 Accepted TCP client: [::ffff:10.10.108.203]:53627
11 INFO listen://[::]:10933/ 11 Performing TLS server handshake
11 INFO listen://[::]:10933/ 11 Secure connection established, client is not yet authenticated, client connected with Tls12
11 INFO listen://[::]:10933/ 11 Client at [::ffff:10.10.108.203]:53627 authenticated as 48D5945E16C0D599D29B01BE5432D6F3FA65703F
20 INFO [ServerTasks-158538] [RunningScript] [Read Lock] [“ServerTasks-158326” (has a write lock)] Trying to acquire lock.
20 INFO [ServerTasks-158538] [RunningScript] [Read Lock] [“ServerTasks-158326” (has a write lock)] Failed to acquire lock within 00:00:00.1000000.
20 INFO [ServerTasks-158538] [RunningScript] [Read Lock] [“ServerTasks-158326” (has a write lock)] Trying to acquire lock with wait time of 24.20:31:23.6470000.
20 INFO [ServerTasks-158538] [RunningScript] [Read Lock] [“ServerTasks-158326” (has a write lock)] Lock acquire canceled.
9 INFO [ServerTasks-158539] [RunningScript] [Read Lock] [“ServerTasks-158326” (has a write lock)] Trying to acquire lock.
9 INFO [ServerTasks-158539] [RunningScript] [Read Lock] [“ServerTasks-158326” (has a write lock)] Failed to acquire lock within 00:00:00.1000000.
9 INFO [ServerTasks-158539] [RunningScript] [Read Lock] [“ServerTasks-158326” (has a write lock)] Trying to acquire lock with wait time of 24.20:31:23.6470000.
9 INFO [ServerTasks-158539] [RunningScript] [Read Lock] [“ServerTasks-158326” (has a write lock)] Lock acquire canceled.
29 INFO [ServerTasks-158867] [RunningScript] [Read Lock] [“ServerTasks-158326” (has a write lock)] Trying to acquire lock.
29 INFO [ServerTasks-158867] [RunningScript] [Read Lock] [“ServerTasks-158326” (has a write lock)] Failed to acquire lock within 00:00:00.1000000.
29 INFO [ServerTasks-158867] [RunningScript] [Read Lock] [“ServerTasks-158326” (has a write lock)] Trying to acquire lock with wait time of 24.20:31:23.6470000.
29 INFO [ServerTasks-158867] [RunningScript] [Read Lock] [“ServerTasks-158326” (has a write lock)] Lock acquire canceled.
11 INFO listen://[::]:10933/ 11 Accepted TCP client: [::ffff:10.10.108.203]:53662
11 INFO listen://[::]:10933/ 11 Performing TLS server handshake
11 INFO listen://[::]:10933/ 11 Secure connection established, client is not yet authenticated, client connected with Tls12
11 INFO listen://[::]:10933/ 11 Client at [::ffff:10.10.108.203]:53662 authenticated as 48D5945E16C0D599D29B01BE5432D6F3FA65703F
12 INFO [ServerTasks-158872] [RunningScript] [Read Lock] [“ServerTasks-158326” (has a write lock)] Trying to acquire lock.
12 INFO [ServerTasks-158872] [RunningScript] [Read Lock] [“ServerTasks-158326” (has a write lock)] Failed to acquire lock within 00:00:00.1000000.
12 INFO [ServerTasks-158872] [RunningScript] [Read Lock] [“ServerTasks-158326” (has a write lock)] Trying to acquire lock with wait time of 24.20:31:23.6470000.
12 INFO [ServerTasks-158872] [RunningScript] [Read Lock] [“ServerTasks-158326” (has a write lock)] Lock acquire canceled.
11 INFO listen://[::]:10933/ 11 Accepted TCP client: [::ffff:10.10.108.203]:53812
11 INFO listen://[::]:10933/ 11 Performing TLS server handshake
11 INFO listen://[::]:10933/ 11 Secure connection established, client is not yet authenticated, client connected with Tls12
11 INFO listen://[::]:10933/ 11 Client at [::ffff:10.10.108.203]:53812 authenticated as 48D5945E16C0D599D29B01BE5432D6F3FA65703F
15 INFO [ServerTasks-158874] [RunningScript] [Read Lock] [“ServerTasks-158326” (has a write lock)] Trying to acquire lock.
15 INFO [ServerTasks-158874] [RunningScript] [Read Lock] [“ServerTasks-158326” (has a write lock)] Failed to acquire lock within 00:00:00.1000000.
15 INFO [ServerTasks-158874] [RunningScript] [Read Lock] [“ServerTasks-158326” (has a write lock)] Trying to acquire lock with wait time of 00:05:00.
20 INFO listen://[::]:10933/ 20 Accepted TCP client: [::ffff:10.10.108.198]:49747
20 INFO listen://[::]:10933/ 20 Performing TLS server handshake
20 INFO listen://[::]:10933/ 20 Secure connection established, client is not yet authenticated, client connected with Tls12
20 INFO listen://[::]:10933/ 20 Client at [::ffff:10.10.108.198]:49747 authenticated as 48D5945E16C0D599D29B01BE5432D6F3FA65703F
13 INFO [ServerTasks-158881] [RunningScript] [Read Lock] [“ServerTasks-158326” (has a write lock)] Trying to acquire lock.
13 INFO [ServerTasks-158881] [RunningScript] [Read Lock] [“ServerTasks-158326” (has a write lock)] Failed to acquire lock within 00:00:00.1000000.
13 INFO [ServerTasks-158881] [RunningScript] [Read Lock] [“ServerTasks-158326” (has a write lock)] Trying to acquire lock with wait time of 24.20:31:23.6470000.
13 INFO [ServerTasks-158881] [RunningScript] [Read Lock] [“ServerTasks-158326” (has a write lock)] Lock acquire canceled.
15 INFO [ServerTasks-158874] [RunningScript] [Read Lock] [“ServerTasks-158326” (has a write lock)] Failed to acquire lock within 00:05:00.
31 INFO [ServerTasks-158878] [RunningScript] [Read Lock] [“ServerTasks-158326” (has a write lock)] Trying to acquire lock.
31 INFO [ServerTasks-158878] [RunningScript] [Read Lock] [“ServerTasks-158326” (has a write lock)] Failed to acquire lock within 00:00:00.1000000.
31 INFO [ServerTasks-158878] [RunningScript] [Read Lock] [“ServerTasks-158326” (has a write lock)] Trying to acquire lock with wait time of 24.20:31:23.6470000.
31 INFO [ServerTasks-158878] [RunningScript] [Read Lock] [“ServerTasks-158326” (has a write lock)] Lock acquire canceled.
20 INFO listen://[::]:10933/ 20 Accepted TCP client: [::ffff:10.10.108.203]:54846
20 INFO listen://[::]:10933/ 20 Performing TLS server handshake
20 INFO listen://[::]:10933/ 20 Secure connection established, client is not yet authenticated, client connected with Tls12
20 INFO listen://[::]:10933/ 20 Client at [::ffff:10.10.108.203]:54846 authenticated as 48D5945E16C0D599D29B01BE5432D6F3FA65703F
26 INFO [ServerTasks-158903] [RunningScript] [Read Lock] [“ServerTasks-158326” (has a write lock)] Trying to acquire lock.
26 INFO [ServerTasks-158903] [RunningScript] [Read Lock] [“ServerTasks-158326” (has a write lock)] Failed to acquire lock within 00:00:00.1000000.
26 INFO [ServerTasks-158903] [RunningScript] [Read Lock] [“ServerTasks-158326” (has a write lock)] Trying to acquire lock with wait time of 00:05:00.
26 INFO [ServerTasks-158903] [RunningScript] [Read Lock] [“ServerTasks-158326” (has a write lock)] Failed to acquire lock within 00:05:00.
20 INFO listen://[::]:10933/ 20 Accepted TCP client: [::ffff:10.10.108.203]:55870
20 INFO listen://[::]:10933/ 20 Performing TLS server handshake
20 INFO listen://[::]:10933/ 20 Secure connection established, client is not yet authenticated, client connected with Tls12
20 INFO listen://[::]:10933/ 20 Client at [::ffff:10.10.108.203]:55870 authenticated as 48D5945E16C0D599D29B01BE5432D6F3FA65703F
31 INFO [ServerTasks-158907] [RunningScript] [Read Lock] [“ServerTasks-158326” (has a write lock)] Trying to acquire lock.
31 INFO [ServerTasks-158907] [RunningScript] [Read Lock] [“ServerTasks-158326” (has a write lock)] Failed to acquire lock within 00:00:00.1000000.
31 INFO [ServerTasks-158907] [RunningScript] [Read Lock] [“ServerTasks-158326” (has a write lock)] Trying to acquire lock with wait time of 00:05:00.
31 INFO [ServerTasks-158907] [RunningScript] [Read Lock] [“ServerTasks-158326” (has a write lock)] Failed to acquire lock within 00:05:00.

henrik · 21 June 2018 04:06

Hi Leslie,

Thanks for getting in touch.

Next time you encounter this issue, could you check if there are any hung PowerShell.exe processes running. We’ve had reports of random time where a PowerShell.exe session hasn’t been closed properly or been abandoned/orphaned.

Thank you and kind regards,
Henrik

Leslie · 25 June 2018 18:31

Thank you for that suggestion, so far the one time we have seen that again, there was a hung powershell script. It popped-up a UI, which was not on a WinStation that is accessible by a user logging in. Please feel free to close the case

Thank You

system · 25 July 2018 18:31

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.