Octopus.Server.exe export-certificate

Hi @finnian.dempsey. I allready did that, but it had no effect on the DDOS from the tentacles. Makes sense since the health checks themselves does not seem to be the culpit. Looks like the polling tentacles will constantly retry polling the server when they fail which is generating the DDOS behavior. Have you heard back from the devs?

Hi @kbl_sg,

Thank you for the update, and sorry to hear the new machine policy didnā€™t help things.

I didnā€™t see any new information on the internal thread Finnian created for this, so Iā€™ve relayed your latest message to our engineering team and have asked for an update, and we will let you know as soon as we hear back on this.

Regards,

Britton

Hi @kbl_sg,

Thanks for your patience while I contacted the team for an update on this!

In doing some initial internal testing, our engineering team has been unable to replicate the DDOS-like behavior you see from polling tentacles (even in tests with ~800 connecting polling tentacles).

From your most recent messages, it seems like some authentication failure might occur between the server and Tentacle(s), triggering this weird behavior. To help correlate this better, can you upload fresh Server and Tentacle logs with overlapping timestamps from when this failure occurs? It looks like the last logs we received may have expired, so it would be good grab the latest to review.

Here is a new secure link where you can upload these files as well.

Along with the data above, do you happen to notice any other processes kick-off or run on the server when this failure occurs (via Task Manager)? I know we had identified Microsoft Defender as a potential issue at one point, but Iā€™m not sure if this process is still ramping up during the failure like it was before.

Lastly, do you notice any other related messages or errors being written to Windows Event Viewer in your latest tests? I know we had checked this on a previous call, so I just wanted to confirm nothing new has cropped up here.

I appreciate you continuing to work with us on this, and Iā€™m sorry youā€™ve been running into some trouble here, but we will continue helping where we can to help get this resolved.

Regards,

Britton

After migrating 400 of our 700 tentacles by hand the ddos issue is resolved.

Hey @kbl_sg,

Sorry its taken awhile to get back to you, thank you for confirming the DDOS issue has been reaolved but I am sorry you had to manually update all of those tentacles! I appreciate that must have been very frustrating for you and I am sorry that was what it took to get this issue resolved, sometimes IT just does not play nice, hopefully now you are able to deploy and you never have to look at a tentacle installer again!

Reach out if you have any more issues as we are always on hand to help!

I will let @finnian.dempsey and @britton.riggs know too as they worked hard on this with you to get it resolved.

Kind Regards and Happy Deployments,
Clare

Hi all. I have been looking you a detailed log on the server of incoming requests from the tentacles, both successful and unsuccessful. Could you point me to where I can find those? If they are not available could you please note this as a feature request?

Hi @kbl_sg,

I was testing on my side to see if our Trace level logging would expose any of this information in the server logs, but unfortunately, it doesnā€™t look like this forces any more details to be shown regarding Tentacle connections.

With this being the case, Iā€™ve escalated your request to our engineering team to see if any other existing options would expose this information.

I will let you know as soon as I have an update from the team, and let me know if you come across anything new on your side in the meantime as well.

Regards,

Britton

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.