Hey @Lastbuilders,
Thanks for the logs, I have taken a look and it seems like the Tentacle service on the logs from today all start and stop fine.
Is the machine BRSPVXAGEMSUAT2 for Octopus EMEA working, the other two logs have one error message (shown below) but the EMEA logs look fine.
The Tentacle from the US (QASFVUA----) has this in the logs (notice it mentions Boolean shutdownCheck):
2022-09-23 08:35:14.7050 3160 28 INFO Stopping the Windows Service
2022-09-23 08:35:14.7206 3160 28 INFO listen://[::]:10933/ 28 Listener stopped
2022-09-23 08:35:14.7206 3160 28 INFO The Windows Service has stopped
2022-09-23 08:35:14.7519 3160 33 INFO listen://[::]:10933/ 33 Socket IO exception: [::ffff:172.27.4.140]:14227
System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'SslStream'.
at System.Net.Security.SslState.CheckThrow(Boolean authSuccessCheck, Boolean shutdownCheck)
at System.Net.Security._SslStream.ProcessFrameBody(Int32 readBytes, Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security._SslStream.StartFrameHeader(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security._SslStream.StartReading(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security._SslStream.ProcessRead(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
The same with the Octopus CA logs from TORLVUAX-----:
2022-09-23 09:16:51.7710 2580 17 INFO Stopping the Windows Service
2022-09-23 09:16:51.7710 2580 17 INFO listen://[::]:10933/ 17 Listener stopped
2022-09-23 09:16:51.7710 2580 17 INFO The Windows Service has stopped
2022-09-23 09:16:51.8574 2580 6 INFO listen://[::]:10933/ 6 Socket IO exception: [::ffff:172.27.4.140]:14790
System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'SslStream'.
Do those timings correspond to when you deployed and seen the failures? There is a forum post here where one of our users had the same issue and he mentioned he was not able to deploy apps until he restarted the server. The issue does not have a resolution though unfortunately.
There is also this user who runs into the same error message, its not Octopus related but this users code would not run (he got the error message you are seeing) until he rebooted his machine then the code would run.
This looks to potentially be coming from .net, there is an issue relating to it here and from the sounds of it it looks like something has been disposed of that the tentacle needs access to in order to deploy. This would potentially stop Octopus from actually running the restart script perhaps?
Once your ops team restart the server the tentacle then has access to the object it requires and continues to deploy. Unfortunately I don’t know why this would happen, what the disposed object is or why it only happens on tentacle version 6.1.1304. Have you tried upgrading the tentacles at all? We have not seen this error come up other than that one forum post so this does look to be environmental rather than an issue with Octopus but it would be worth upgrading a tentacle to see if the newer versions handle that disposed object better?
Also, are the tentacles all running the same .net versions?
Does the issue happen on BRSPVXAGEM---- for Octopus EMEA as that doesnt have the error in the log and is running on 6.1.1304 which is the same as the other two tentacles?
I look forward to hearing from you,
Kind Regards,
Clare