Deploy Tasks Failing after working minutes before

unknown
server
reliability
(Ops) #1

Connections to our Tentacles seem strangely flaky from our cloud instance.

We ran a staging deploy this morning with no issues.

20 minutes later we ran a production deploy (which deploys to exactly the same Tentacles) and recieved timeout errors on all 4 targets.

The step failed: Activity failed with error ‘An error occurred when sending a request to ‘https://:10933/’, before the request could begin: The client was unable to establish the initial connection within the timeout 00:01:00. The client was unable to establish the initial connection within the timeout 00:01:00.’.

I retried the deployment and again all 4 failed with the same error.

I went to the Infrastructure tab and all the targets were marked healthy.
I checked their connectivity health again and they all returned green.

I then ran the deployment for a third time and all succeeded.

Any idea what is going on?

(Tina) #3

Hi @ops,
Sorry to hear your running into issues with flaky Tentacle connections. This is strange indeed.
Are your listening tentacles all in the same network? Where there any network issues occurring around the same time that the Tentacle connection failures happened?

Thank for reaching out and hope we can get to the bottom of this strange behaviour shortly.

Tina

(Ops) #4

Hi @tina.bamford

Yes they are all on the same network.
From what we can tell there are no network issues and I was able to RDP to each server no problem when I wanted to check the service status.

It only seems to apply to Octopus Tentacles at the moment.
It certainly is strange behaviour but it also isn’t the first time it has happened.

The logs show nothing untoward so I’m not sure how to diagnose the issue further or help. I opened the topic in case this was a known issue or you had any other suggestions.
If it’s not already known then I will wait to see if it happens again and concentrate on diagnosing any latent network issues to see if there is an underlying cause behind the dropouts.

Many thanks for your reply and I will let you know how we get one.

Cheers,

Mike