In incognito it fails even earlier and does not connect to the portal at all. See .har file as well. The problem is the same from my client computer and from the edge browser on the server itself.
Please let me know if you would like to shared screen session where we can reproduce the issue.
Hi @kbl_sg,
Cheers for that, confirming weâve received them!
It just looks like youâre being redirected and so the issue seems likely to be with Azure AD. I found the following in the logs which seems suspicious but Iâve not encountered this before:
WARN Provided value "octopus.signaturgruppen.dk" is not a valid IP Address so will not be included in this event.
Iâve reached out to the devs for some advice and will let you know if there is any extra info we need. Iâd be happy to arrange a call once I hear back from them as we should have enough evidence of the issue and just need to get a workaround sorted!
Iâll keep you posted with any updates, free to reach out with any questions!
Best Regards,
Hi @finnian.dempsey Any news on this issue? I tried disabling Azure AD in Octopus Settings but the issue is still present. I have also upgraded to latest Octopus Server build.
Kind regards
Kasper
Hi @kbl_sg,
Sorry for the delay with this, it looks like my original request to the devs was missed but theyâve confirmed itâs being investigated now.
The warning I shared previously about the Invalid IP isnât likely to be the cause and they are able to reproduce the private key issue without it causing the browser to hang. Are you using any network appliances like a Proxy/ReverseProxy?
Iâll keep you posted if they need any more info or are able to find any workarounds!
Best Regards,
Hi @kbl_sg,
Just an update from the engineers, in the HAR file it looks like the redirect to /app isnât occurring.
Could you please confirm the HTTPS binding certificate is the one you are expecting?
If you could also please test navigating to https://octopus.signaturgruppen.dk/app directly, does it still hang? It might also be worth removing the SSL redirect option in the Octopus Manager and trying http://octopus.signaturgruppen.dk/app instead:
Looking forward to hearing how you get on, feel free to reach out with any questions!
Best Regards,
Hi @finnian.dempsey It still hangs if I navigate to /app directly. I enabled http, removed ssl redirect and som http trafic seems to get though, but itâs very slow (116064ms for loading an image):
full log:
2023-03-08 22:34:22.4740 1808 15 INFO Scheduler started successfully.
2023-03-08 22:34:22.4740 1808 15 INFO The Octopus server is starting: Starting task queueâŚ
2023-03-08 22:34:22.4740 1808 15 INFO There are no tasks currently executing on nodes that are offline
2023-03-08 22:34:22.4740 1808 15 INFO Checking for tasks that are currently running on nodes that are offline or missingâŚ
2023-03-08 22:34:22.4740 1808 15 INFO There are no tasks currently executing on nodes that are offline
2023-03-08 22:34:22.4740 1808 15 INFO There are no tasks currently executing on nodes that are missing
2023-03-08 22:34:22.4740 1808 15 INFO There are no stale worker leases
2023-03-08 22:34:22.4740 1808 15 INFO The Octopus server is starting: Starting telemetryâŚ
2023-03-08 22:34:22.4740 1808 15 INFO Web server is ready to process requests.
2023-03-08 22:34:24.4935 1808 22 INFO âExecute non queryâ took 1859ms in transaction âUpdateDeploymentHistoryJob|819216f2-8aad-4813-9115-e3bcade3cd93|T15â: âUpdateDeploymentHistoryâ
2023-03-08 22:34:54.6349 1808 205 INFO âExecute non queryâ took 346ms in transaction âUpdateDeploymentHistoryJob|819216f2-8aad-4813-9115-e3bcade3cd93|T19â: âUpdateDeploymentHistoryâ
2023-03-08 22:35:10.7623 1808 205 INFO âHTTPâ âGETâ to âoctopus.signaturgruppen.dkââ/appâ âcompletedâ with 200 in 00:00:00.0931613 (93ms) by ââ
2023-03-08 22:35:24.8428 1808 22 INFO âExecute non queryâ took 412ms in transaction âUpdateDeploymentHistoryJob|819216f2-8aad-4813-9115-e3bcade3cd93|T22â: âUpdateDeploymentHistoryâ
2023-03-08 22:35:57.0526 1808 28 INFO âExecute non queryâ took 351ms in transaction âUpdateDeploymentHistoryJob|819216f2-8aad-4813-9115-e3bcade3cd93|T309â: âUpdateDeploymentHistoryâ
2023-03-08 22:36:23.1606 1808 28 INFO âHTTPâ âGETâ to âoctopus.signaturgruppen.dkââ/manifest.773135bab6309eef1766.hashedasset.jsâ âcompletedâ with 200 in 00:00:00.0050009 (5ms) by ââ
2023-03-08 22:36:27.1840 1808 309 INFO âExecute non queryâ took 355ms in transaction âUpdateDeploymentHistoryJob|819216f2-8aad-4813-9115-e3bcade3cd93|T307â: âUpdateDeploymentHistoryâ
2023-03-08 22:36:51.2717 1808 309 INFO âHTTPâ âGETâ to âoctopus.signaturgruppen.dkââ/â âcompletedâ with 302 in 00:00:00.0025388 (2ms) by ââ
2023-03-08 22:36:57.3037 1808 28 INFO âExecute non queryâ took 356ms in transaction âUpdateDeploymentHistoryJob|819216f2-8aad-4813-9115-e3bcade3cd93|T5â: âUpdateDeploymentHistoryâ
2023-03-08 22:37:15.4120 1808 307 INFO âHTTPâ âGETâ to âoctopus.signaturgruppen.dkââ/appâ âcompletedâ with 200 in 00:00:00.0026847 (2ms) by ââ
2023-03-08 22:37:27.4902 1808 309 INFO âExecute non queryâ took 341ms in transaction âUpdateDeploymentHistoryJob|819216f2-8aad-4813-9115-e3bcade3cd93|T391â: âUpdateDeploymentHistoryâ
2023-03-08 22:38:25.7860 1808 5 INFO âHTTPâ âGETâ to âoctopus.signaturgruppen.dkââ/loading-image.78d0e1acfba55a821561afe0852eddbd.hashedasset.svgâ âcompletedâ with 200 in 00:01:56.0640918 (116064ms) by ââ
2023-03-08 22:38:35.8173 1808 519 INFO âHTTPâ âGETâ to âoctopus.signaturgruppen.dkââ/appâ âcompletedâ with 200 in 00:00:00.0022044 (2ms) by ââ
2023-03-08 22:38:59.9747 1808 5 INFO âExecute non queryâ took 312ms in transaction âUpdateDeploymentHistoryJob|819216f2-8aad-4813-9115-e3bcade3cd93|T5â: âUpdateDeploymentHistoryâ
2023-03-08 22:39:06.0067 1808 5 INFO âHTTPâ âGETâ to âoctopus.signaturgruppen.dkââ/manifest.773135bab6309eef1766.hashedasset.jsâ âcompletedâ with 200 in 00:00:00.0010048 (1ms) by ââ
2023-03-08 22:40:00.2407 1808 5 INFO âExecute non queryâ took 302ms in transaction âUpdateDeploymentHistoryJob|819216f2-8aad-4813-9115-e3bcade3cd93|T519â: âUpdateDeploymentHistoryâ
2023-03-08 22:40:42.4589 1808 309 INFO âHTTPâ âGETâ to âoctopus.signaturgruppen.dkââ/outdatedBrowser.098c3fdd5fcf27d1f17e.hashedasset.jsâ âcompletedâ with 200 in 00:04:11.8744096 (251874ms) by ââ
2023-03-08 22:41:18.6285 1808 309 INFO âHTTPâ âGETâ to âoctopus.signaturgruppen.dkââ/manifest.773135bab6309eef1766.hashedasset.jsâ âcompletedâ with 200 in 00:00:00.0017198 (1ms) by ââ
Hi @kbl_sg,
Cheers for confirming that, I have to admit this is really strange and so it might be best to jump on a call to go over the process together.
Let me know a suitable time/timezone for you and Iâll get an invite sent out, depending on the time I might not be able to attend but our UK team should be able to help!
Best Regards,
Hi @finnian.dempsey OK. We need to do this outside european business hours. How does 10 PM CET (9 PM UK) work? Thursdays or mondays.
Hi @kbl_sg,
That works fine for me!
Iâve sent out an invite for Thursday 10pm CET (Friday 7am AEST) but feel free to adjust it to Monday if preferred.
Looking forward to meeting and hopefully getting to the bottom of this!
Best Regards,
Hi @kbl_sg,
No worries, Iâve generated another for you to use here - Secure Upload Portal
Let us know if there are any issues with it!
Best Regards,
Hi @kbl_sg,
Just an update, could you please check whether Check for publisher's certificate revocation is disabled in IE internet properties?
This article goes into how it could potentially cause slow page loads.
Iâll keep you posted with any other updates or suggestions!
Best Regards,
Hi @finnian.dempsey. Just did a test with the above setting and unfortunately it did not make any difference.
Hi @kbl_sg,
Cheers for confirming that.
I managed to find the following message in the process dump, which implies that it might be the TLS negotiation thatâs making it take so long:
Received an unexpected EOF or 0 bytes from the transport stream.
Which is odd since the browser certificate being negotiated shouldnât have changed, just the internal certificate.
Could you please confirm which TLS version/cipher is being negotiated? E.g.
openssl s_client -connect octopus.signaturgruppen.dk:443
It might pay to disable weaker tls versions and ciphers on the Octopus Instance to see if it will speed up the negotiation, I find IISCrypto is a great tool for this.
Otherwise if that isnât successful Iâd be happy to look at another process dump or logs to see whatâs going on, however it might require provisioning another VM as a clone to test if rotating the certificate also effects it too.
Let me know if you have any questions at all!
Best Regards,
Hi @finnian.dempsey @sean.stanway If I close the Tentacle communication port 10943 in the Windows firewall the portal is responsive again, but I of course canât talk to the tentacles. I have run your script on all tentacles to enable trust for the new certificate. Unfortunately the issue is now the same with the old certificate. This leads me to believe that the Octopus server does not handle tentacles what have (at least one) inactive certificate in the trustlist? I really need you to escalate this as I now am in a situation where i canât roll back and I have lost communication with all our tentacles.
Hi @kbl_sg ,
Sorry to hear youâre having trouble with it and are lost all communication with your tentacles!
The scripts should have created a backup of the original config and so as long as you still have the original certificate then you should be able to import it back and replace Tentacle.config with Tentacle.config.bak, however if your firewall is blocking port 10943 then you wonât be able to poll the Octopus Server to communicate with it.
It sounds like port 10943 might be bound to the old certificate. Octopus shouldnât have an issue with a multiple certificates (even when one isnât reachable) in a Tentacleâs config.
Could you please run the following command on the Octopus Server? If there is a certificate bound to 10943 then please either rotate it to the new certificate or remove it:
netsh http show sslcert
Iâll reach out to the devs for some more ideas about what could be causing this, looking forward to hearing how you get on!
Best Regards,
Hi @finnian.dempsey There is no binding on port 10943. I have begun configuring our tentacles by hand (30 out of 735). Right now I only allow access on port 10943 for tentacles that have the new certificate trust configured (and only the new trust). This seems to work. If I open up for trafic from all 735 tentacles the server becomes unresponsive again. For now we are handling I by hand but I would of course hope to be able to automate configuration of the rest of the tentacles.
Hi @kbl_sg,
Thanks for confirming that, it sounds like itâs getting DDOSâd from too many polling requests!
If I open up for trafic from all 735 tentacles the server becomes unresponsive again
Iâve reached out to the devs and will let you know if they have any suggestions around this but you might be able to change the health check behaviour to only perform a connection test to get around it.
Could you please create a new Machine Policy with the interval set to never and performs a connection only test and change the Tentacles to use that new policy?
That should prevent the initial health-check from DDOSâing the Server and allow you to move the Tentacles back to the original Machine policy in batches.
Our performance docs about Polling Tentacles: Performance | Documentation and Support
Let me know if that doesnât help or you have any questions at all!
Best Regards,