Octopus slowness issue

Hi,
We are testing Octopus deployments before we migrate our on-premise Octopus (version: 2021.3.12372) to on-premise Octopus hosted in Microsoft Azure (version: 2022.2.8011).
We are experiencing slowness issues in our Deployments in Azure Octopus and are taking longer in time compared to the on-premise (45mins vs 2Hrs 15 mins). The tasks are getting queued and tasks are waiting on other tasks to finish.
The slowness happens at the start of the deployment at the Acquire Packages step and with multiple deployments running in parallel, this causes a backlog of tasks and takes longer to run.

Octopus on-prem uses Listening Tentacles and the on-prem hosted in Azure uses Polling Tentacles.

In our test scenario which produces the result all tentacles (both listening-which connect to on-prem, and polling-which connect to off-prem) use tentacle version 6.1.736.
Our test case executes three separate releases in parallel. On-prem yields no issues, however, for off-prem each of the three projects block each other (waiting tasks) on the same infra.

One other thing, the version of Octopus is different in each:
On-prem = 2021.3.12372
Off-site = 2022.2.6971 (1st round of testing) and 2022.2.8011 (2nd round of testing – still no improvement).

While examining task logs I’ve noticed that the lines:

Executable name or full path: C:\windows\system32\WindowsPowershell\v1.0\PowerShell.exe
Starting C:\windows\system32\WindowsPowershell\v1.0\PowerShell.exe

Seem to have a much larger timespan between them in the cloud file (generally, though not always).

We have been in contact with Jason Gallup about this issue, he was going to open a ticket for it but not sure if he did.

We must get this issue fixed before we migrate from on-prem Octopus so maybe a call is needed to have a deep dive into the issue.

I will attach the raw output of runs that were executed. They’re not running exactly the same thing but close enough, I believe, so please provide a link to upload.

Kind regards,
Micheál Power

Hi Micheál,

Thanks for reaching out!

I’m sorry to hear you are seeing such a discrepancy in performance between your on-premise and Azure Octopus Deploy environments, but I’m happy to help investigate this further.

Here is a secure upload link for the task logs you mentioned, feel free to upload these at your convenience and I will review.

Looking forward to hearing back from you,

Britton

Hi @britton.riggs,
I have uploaded the log files, Have you had a chance to look into the issue?

Kind regards,
Micheál

Hi Micheál,

Thank you for the update! Apologies, I didn’t get a notification about the upload so I just had a chance to review these.

In taking a look, I do see that the Azure log shows about a +9 minute increase in the deploy time, and it looks like most of this increase had to do with package acquisitions. I noticed that the machine names in the Azure log are slightly different than the on-premise log - can you let me know if this process is deploying to the same deployment targets, or are the targets also new? I just want to eliminate this as being a potential source of the difference in performance before escalating this to our engineering team for a deeper review.

Best,

Britton

Hi @britton.riggs,
Yes - the deployment target machines are same only - The only difference is we are using the newly installed polling tentacle in each deployment target.

Kind regards,
Micheál Power

Hey @mikepower79,

Jumping in here for Britton as he is currently off shift as part of our US based team. Thank you for the extra information, I will collate everything and send it to our engineers, if they need more information I will let you know.

Kind Regards,

Clare

1 Like

Hey @mikepower79,

Sorry for the double post but I just had a thought, we had some issues not so long ago with some tentacle versions queuing tasks (not running them in parallel) even if the task cap on the Octopus server was not hit or the Bypass Mutex variable was setup on the project.

The GitHub issue for that is here, deployments seem to be fine in your On-Prem server, but that issue was seen in versions 2022.2 and above so it would affect your Azure newer server.

You have tentacle 6.1.736 installed, Ideally you would need tentacle 6.2 installed to make sure that issue isnt a factor here. Tentacle 6.2.5 is our latest build but you should be able to update them through the Octopus UI.

Whilst our engineers take a look at the logs are you able to update one of your tentacles that is slow please to the latest tentacle version and try the deploy again on your Azure server so we are able to rule this out.

I look forward to hearing from you,

Kind Regards,

Clare

Hi @clare.martin,
Yes we are aware of the Github issue and the Tentacles that were running the Acquire package step are upgraded to 6.2.50.

Kind reagrds,
Micheál Power

Hey @mikepower79,

Thanks for confirming that, your original post suggested all your tentacles were running 6.1.736 which is why I mentioned it. Having looked at your cloud logs you sent over though I can see you are using Calamari Version: 22.7.4 which would mean you are using tentacle v6.2 so I apologise for not seeing that before mentioning it.

The engineers have all the information you have provided though so hopefully they can take a look when they are free.

Kind Regards,

Clare

1 Like

Hey @mikepower79,

Our engineers have been looking into this and he has put together a sheet with the step times, its a google doc which you probably wont have access to so I have put it in picture format for you to see:

He thinks the “compressing the files” section of the step “Post Deployment script for archiving and cleanup” is a pretty good indicator that the cloud targets are slower than the on-prem targets.
Octopus has launched your powershell script and is waiting for it to run, so it’s basically just your script running. In the cloud, just that section of script takes a minute and a half longer than in their on-prem deployment.

I have asked the engineer if he thinks there is anything you can do here to alleviate this but do you know what that script is doing? You mentioned your Cloud Tentacles were using Azure VMs and polling tentacles. I am wondering if the same slowness is seen when using a Listening Tentacle in the Cloud. Are you able to set one up and see if it makes a difference?

I will keep you informed of our engineers recommendations if he has any, I am sorry we have not gotten to the bottom of this yet but thankfully it seems we are making progress.

Kind regards,

Clare

Hi @clare.martin,
We will be unable to use Listening Tentacles to compare as IT security only permits the use of Polling Tentacles.

Would the engineers be able concentrate on the Acquire Package step as we feel this step is the main contributor to the slowness we are experiencing.

Kind regards,
Micheál Power

Hey @mikepower79,

Thanks for confirming you are only able to use polling tentacles, I will ask the engineers to look into the Acquire package step and see if they can spot anything, they will have looked at this step as part of the wider investigation and will continue to do so but I will mention it.

Kind Regards,

Clare

2 Likes

Hi @clare.martin,
Any update on this issue?

Kind regards,
Micheál Power

Hey @mikepower79,

Sorry, I have been slowly going through my tickets updating customers with engineers responses so you have just come up on my list. The engineers did respond saying that in general package acquisition does take longer on your cloud instance compared with your on-prem one, but all the other tasks do too.

He mentioned the compression part because it is noticeably longer, and Octopus has next to no involvement in the time reported back. It demonstrates that operations on the cloud machines perform more slowly than the on-prem counterpart in general.

Package acquisition on the cloud instance is doing more than your on-prem instance due to package retention:

05:16:10 Verbose | Cleaning 28.875 GB space from the package cache.

That does account for some extra seconds, but the like-for-like acquisition in the cloud takes 48 seconds vs. 6 seconds on-prem.

Basically what the engineer is saying here is that your Cloud instances are generally slower to do things than your on-prem ones. As a thought, I am wondering if you could try downloading a package outside of Octopus, so using PowerShell to pull a package from the repository you are trying to pull it from in Octopus.

If you do this from an on-prem tentacle and then a cloud tentacle you can compare the two and see the difference in speed.

I am sorry I do not have anything more concrete here but since the compression part takes longer on your cloud instance and Octopus has nothing to do with the timing of that task it points more to the cloud instances being the factor here rather than Octopus.

But hopefully the PowerShell package test could dig something up? Let me know how you get on with that.

Kind Regards,

Clare

Hi @clare.martin,
Yes I have noticed that pulling a package from Cloud Octopus takes longer, 7 seconds compared to 2 seconds downloading from on-prem. I ran a powershell script from an Octopus on-prem worker server.

Kind reagrds,
Micheal Power

Hey @mikepower79,

Thank you for running that test, although it is only 5 seconds difference at least that tracks with what our engineers have found. The other tests will be harder to produce outside of Octopus so I am not sure where to go from here.

Unfortunately it does seem environmental as one of the steps that’s taking an unusual amount of time (the “compressing the files” section of the step “Post Deployment script for archiving and cleanup”) has nothing to do with Octopus and its taking a lot longer on cloud than on-prem.

Coupled with the extra few seconds to download a package plus the cleaning nearly 30GB from the package cache on cloud this will all add up.

I am not sure what the spec is for your cloud instances and how that compares to on-prem, and also your on-prem networking setup (link speed etc) compared to cloud, this will all factor in.

I am at a loss of where to go from here or what our next steps would be as we have done all we can over here to try and pinpoint this and it doesn’t seem like it is Octopus that is the cause of the slowness.

I know that is disappointing and not what you wanted to hear but I think your next steps would be to take a look at the servers themselves and see where the issue lies environmentally.

The only other thing we can suggest from an Octopus point of view would be to see if you can run a tentacle ping between your tentacles and server (two tests, one for your on-prem server and on-prem tentacles and one for your cloud server and cloud tentacles). That would show you the connection speed between the tentacles and servers of the respective hosting platforms but does not take the package stuff into account so you might see the same sort of timings for a ping which is a simple handshake using one protocol. It was just something I thought of in case you do run the ping and see big differences that would further clarify it is a machine speed issue.

If there is anything else I can help you with please let me know.

Kind Regards,

Clare

@mikepower79,

Sorry for the double post but 2 mins ago I got a response from our engineers which I wanted to share with you as it sort of validates what I mentioned above but mentions it wont be network speed thats the issue:

The engineer said the package is found on the machine in both cases:

06:41:21 Verbose | Package BMR_Server_PSH 4.14.21 hash 0b02ee48cf4a135fe848b26ba8ea2d955b4e9d3b has already been uploaded

So it doesn’t look like a difference in the network. In particular, the machine EUxxxxxD315-ITAAG89 seems to have the biggest discrepancies in speed.

On prem, the machine EUxxxxxD315 takes 5 seconds to perform the compression task. This is with everything already uploaded to the machine. PowerShell is already running, Calamari is running. There is nothing else for Octopus to do but observe the running script.

06:51:41   Info     |       compressing the files
06:51:41   Verbose  |       Preparing to compress...
06:51:42   Verbose  |       Adding '\\eu51xxxxxx\xxxx\OctopusConfigurations\POC44\GENAURA\EU50APVD315\BMR\BMR_Server_PSH\1.0\Areas\'.
06:51:42   Verbose  |       Adding '\\eu51xxxxxx\xxxx\OctopusConfigurations\POC44\GENAURA\EU50APVD315\BMR\BMR_Server_PSH\1.0\NLog.config'.
06:51:42   Verbose  |       Adding '\\eu51xxxxxx\xxxx\OctopusConfigurations\POC44\GENAURA\EU50APVD315\BMR\BMR_Server_PSH\1.0\web.config'.
06:51:42   Verbose  |       Adding '\\eu51xxxxxx\xxxx\OctopusConfigurations\POC44\GENAURA\EU50APVD315\BMR\BMR_Server_PSH\1.0\Areas\Management\'.
06:51:42   Verbose  |       Adding '\\eu51xxxxxx\xxxx\OctopusConfigurations\POC44\GENAURA\EU50APVD315\BMR\BMR_Server_PSH\1.0\Areas\Management\Views\Web.config'.
06:51:42   Verbose  |       Adding '\\eu51xxxxxx\xxxx\OctopusConfigurations\POC44\GENAURA\EU50APVD315\BMR\BMR_Server_PSH\1.0\Bin\Authorization.rules'.
06:51:42   Verbose  |       Adding '\\eu51xxxxxx\xxxx\OctopusConfigurations\POC44\GENAURA\EU50APVD315\BMR\BMR_Server_PSH\1.0\Bin\Cors.config'.
06:51:42   Verbose  |       Adding '\\eu51xxxxxx\xxxx\OctopusConfigurations\POC44\GENAURA\EU50APVD315\BMR\BMR_Server_PSH\1.0\Bin\ReportSettings.config'.
06:51:42   Verbose  |       Adding '\\eu51xxxxxx\xxxx\OctopusConfigurations\POC44\GENAURA\EU50APVD315\BMR\BMR_Server_PSH\1.0\Bin\StandardConfigurationSection.config'.
06:51:43   Verbose  |       Adding '\\eu51xxxxxx\xxxx\\OctopusConfigurations\POC44\GENAURA\EU50APVD315\BMR\BMR_WebClient\1.0\web.config'.
06:51:43   Verbose  |       Adding '\\eu51xxxxxx\xxxx\OctopusConfigurations\POC44\GENAURA\EU50APVD315\BMR\BMR_WebClient\1.0\de\assets\'.
06:51:43   Verbose  |       Adding '\\eu51xxxxxx\xxxx\OctopusConfigurations\POC44\GENAURA\EU50APVD315\BMR\BMR_WebClient\1.0\de\config.json'.
06:51:43   Verbose  |       Adding '\\eu51xxxxxx\xxxx\OctopusConfigurations\POC44\GENAURA\EU50APVD315\BMR\BMR_WebClient\1.0\de\index.html'.
06:51:43   Verbose  |       Adding '\\eu51xxxxxx\xxxx\OctopusConfigurations\POC44\GENAURA\EU50APVD315\BMR\BMR_WebClient\1.0\de\assets\user-manual\index.html'.
06:51:43   Verbose  |       Adding '\\eu51xxxxxx\xxxx\OctopusConfigurations\POC44\GENAURA\EU50APVD315\BMR\BMR_WebClient\1.0\en\assets\'.
06:51:43   Verbose  |       Adding '\\eu51xxxxxx\xxxx\OctopusConfigurations\POC44\GENAURA\EU50APVD315\BMR\BMR_WebClient\1.0\en\config.json'.
06:51:43   Verbose  |       Adding '\\eu51xxxxxx\xxxx\OctopusConfigurations\POC44\GENAURA\EU50APVD315\BMR\BMR_WebClient\1.0\en\index.html'.
06:51:43   Verbose  |       Adding '\\eu51xxxxxx\xxxx\OctopusConfigurations\POC44\GENAURA\EU50APVD315\BMR\BMR_WebClient\1.0\en\assets\user-manual\index.html'.
06:51:43   Verbose  |       Adding '\\eu51xxxxxx\xxxx\OctopusConfigurations\POC44\GENAURA\EU50APVD315\BMR\BMR_WebClient\1.0\en-150\assets\'.
06:51:43   Verbose  |       Adding '\\eu51xxxxxx\xxxx\OctopusConfigurations\POC44\GENAURA\EU50APVD315\BMR\BMR_WebClient\1.0\en-150\config.json'.
06:51:43   Verbose  |       Adding '\\eu51xxxxxx\xxxx\OctopusConfigurations\POC44\GENAURA\EU50APVD315\BMR\BMR_WebClient\1.0\en-150\index.html'.
06:51:43   Verbose  |       Adding '\\eu51xxxxxx\xxxx\OctopusConfigurations\POC44\GENAURA\EU50APVD315\BMR\BMR_WebClient\1.0\en-150\assets\user-manual\index.html'.
06:51:43   Verbose  |       Adding '\\eu51xxxxxx\xxxx\OctopusConfigurations\POC44\GENAURA\EU50APVD315\BMR\BMR_WebClient\1.0\fr\assets\'.
06:51:43   Verbose  |       Adding '\\eu51xxxxxx\xxxx\OctopusConfigurations\POC44\GENAURA\EU50APVD315\BMR\BMR_WebClient\1.0\fr\config.json'.
06:51:43   Verbose  |       Adding '\\eu51xxxxxx\xxxx\OctopusConfigurations\POC44\GENAURA\EU50APVD315\BMR\BMR_WebClient\1.0\fr\index.html'.
06:51:43   Verbose  |       Adding '\\eu51xxxxxx\xxxx\OctopusConfigurations\POC44\GENAURA\EU50APVD315\BMR\BMR_WebClient\1.0\fr\assets\user-manual\index.html'.
06:51:43   Verbose  |       Adding '\\eu51xxxxxx\xxxx\OctopusConfigurations\POC44\GENAURA\EU50APVD315\BMR\BMR_WebClient\1.0\nl\assets\'.
06:51:43   Verbose  |       Adding '\\eu51xxxxxx\xxxx\\OctopusConfigurations\POC44\GENAURA\EU50APVD315\BMR\BMR_WebClient\1.0\nl\config.json'.
06:51:43   Verbose  |       Adding '\\eu51xxxxxx\xxxx\OctopusConfigurations\POC44\GENAURA\EU50APVD315\BMR\BMR_WebClient\1.0\nl\index.html'.
06:51:43   Verbose  |       Adding '\\eu51xxxxxx\xxxx\OctopusConfigurations\POC44\GENAURA\EU50APVD315\BMR\BMR_WebClient\1.0\nl\assets\user-manual\index.html'.
06:51:46   Info     |       Deleting the folder

Compared to the cloud, where that machine’s counterpart EUxxxxxD315-ITAAG89 takes 1 minute 37 seconds to run the same section of the script.

05:33:18   Info     |       compressing the files
05:33:23   Verbose  |       Preparing to compress...
05:33:23   Verbose  |       Adding '\\eu51xxxxxx\xxxx\OctopusConfigurations\POC44\GENAURA\EU50APVD315\BMR\BMR_Server_PSH\1.0\Areas\'.
05:33:33   Verbose  |       Adding '\\eu51xxxxxx\xxxx\OctopusConfigurations\POC44\GENAURA\EU50APVD315\BMR\BMR_Server_PSH\1.0\NLog.config'.
05:33:59   Verbose  |       Adding '\\eu51xxxxxx\xxxx\OctopusConfigurations\POC44\GENAURA\EU50APVD315\BMR\BMR_Server_PSH\1.0\web.config'.
05:34:10   Verbose  |       Adding '\\eu51xxxxxx\xxxx\OctopusConfigurations\POC44\GENAURA\EU50APVD315\BMR\BMR_Server_PSH\1.0\Areas\Management\'.
05:34:15   Verbose  |       Adding '\\eu51xxxxxx\xxxx\OctopusConfigurations\POC44\GENAURA\EU50APVD315\BMR\BMR_Server_PSH\1.0\Areas\Management\Views\Web.config'.
05:34:30   Verbose  |       Adding '\\eu51xxxxxx\xxxx\OctopusConfigurations\POC44\GENAURA\EU50APVD315\BMR\BMR_Server_PSH\1.0\Bin\Authorization.rules'.
05:34:35   Verbose  |       Adding '\\eu51xxxxxx\xxxx\OctopusConfigurations\POC44\GENAURA\EU50APVD315\BMR\BMR_Server_PSH\1.0\Bin\Cors.config'.
05:34:35   Verbose  |       Adding '\\eu51xxxxxx\xxxx\OctopusConfigurations\POC44\GENAURA\EU50APVD315\BMR\BMR_Server_PSH\1.0\Bin\ReportSettings.config'.
05:34:40   Verbose  |       Adding '\\eu51xxxxxx\xxxx\OctopusConfigurations\POC44\GENAURA\EU50APVD315\BMR\BMR_Server_PSH\1.0\Bin\StandardConfigurationSection.config'.
05:34:40   Verbose  |       Adding '\\eu51xxxxxx\xxxx\OctopusConfigurations\POC44\GENAURA\EU50APVD315\BMR\BMR_WebClient\1.0\web.config'.
05:34:40   Verbose  |       Adding '\\eu51xxxxxx\xxxx\OctopusConfigurations\POC44\GENAURA\EU50APVD315\BMR\BMR_WebClient\1.0\de\assets\'.
05:34:40   Verbose  |       Adding '\\eu51xxxxxx\xxxx\OctopusConfigurations\POC44\GENAURA\EU50APVD315\BMR\BMR_WebClient\1.0\de\config.json'.
05:34:45   Verbose  |       Adding '\\eu51xxxxxx\xxxx\OctopusConfigurations\POC44\GENAURA\EU50APVD315\BMR\BMR_WebClient\1.0\de\index.html'.
05:34:45   Verbose  |       Adding '\\eu51xxxxxx\xxxx\OctopusConfigurations\POC44\GENAURA\EU50APVD315\BMR\BMR_WebClient\1.0\de\assets\user-manual\index.html'.
05:34:45   Verbose  |       Adding '\\eu51xxxxxx\xxxx\OctopusConfigurations\POC44\GENAURA\EU50APVD315\BMR\BMR_WebClient\1.0\en\assets\'.
05:34:45   Verbose  |       Adding '\\eu51xxxxxx\xxxx\OctopusConfigurations\POC44\GENAURA\EU50APVD315\BMR\BMR_WebClient\1.0\en\config.json'.
05:34:45   Verbose  |       Adding '\\eu51xxxxxx\xxxx\OctopusConfigurations\POC44\GENAURA\EU50APVD315\BMR\BMR_WebClient\1.0\en\index.html'.
05:34:45   Verbose  |       Adding '\\eu51xxxxxx\xxxx\OctopusConfigurations\POC44\GENAURA\EU50APVD315\BMR\BMR_WebClient\1.0\en\assets\user-manual\index.html'.
05:34:45   Verbose  |       Adding '\\eu51xxxxxx\xxxx\OctopusConfigurations\POC44\GENAURA\EU50APVD315\BMR\BMR_WebClient\1.0\en-150\assets\'.
05:34:45   Verbose  |       Adding '\\eu51xxxxxx\xxxx\OctopusConfigurations\POC44\GENAURA\EU50APVD315\BMR\BMR_WebClient\1.0\en-150\config.json'.
05:34:45   Verbose  |       Adding '\\eu51xxxxxx\xxxx\OctopusConfigurations\POC44\GENAURA\EU50APVD315\BMR\BMR_WebClient\1.0\en-150\index.html'.
05:34:45   Verbose  |       Adding '\\eu51xxxxxx\xxxx\OctopusConfigurations\POC44\GENAURA\EU50APVD315\BMR\BMR_WebClient\1.0\en-150\assets\user-manual\index.html'.
05:34:45   Verbose  |       Adding '\\eu51xxxxxx\xxxx\OctopusConfigurations\POC44\GENAURA\EU50APVD315\BMR\BMR_WebClient\1.0\fr\assets\'.
05:34:45   Verbose  |       Adding '\\eu51xxxxxx\xxxx\OctopusConfigurations\POC44\GENAURA\EU50APVD315\BMR\BMR_WebClient\1.0\fr\config.json'.
05:34:50   Verbose  |       Adding '\\eu51xxxxxx\xxxx\OctopusConfigurations\POC44\GENAURA\EU50APVD315\BMR\BMR_WebClient\1.0\fr\index.html'.
05:34:50   Verbose  |       Adding '\\eu51xxxxxx\xxxx\OctopusConfigurations\POC44\GENAURA\EU50APVD315\BMR\BMR_WebClient\1.0\fr\assets\user-manual\index.html'.
05:34:50   Verbose  |       Adding '\\eu51xxxxxx\xxxx\OctopusConfigurations\POC44\GENAURA\EU50APVD315\BMR\BMR_WebClient\1.0\nl\assets\'.
05:34:50   Verbose  |       Adding '\\eu51xxxxxx\xxxx\OctopusConfigurations\POC44\GENAURA\EU50APVD315\BMR\BMR_WebClient\1.0\nl\config.json'.
05:34:50   Verbose  |       Adding '\\eu51xxxxxx\xxxx\OctopusConfigurations\POC44\GENAURA\EU50APVD315\BMR\BMR_WebClient\1.0\nl\index.html'.
05:34:50   Verbose  |       Adding '\\eu51xxxxxx\xxxx\OctopusConfigurations\POC44\GENAURA\EU50APVD315\BMR\BMR_WebClient\1.0\nl\assets\user-manual\index.html'.
05:34:55   Info     |       Deleting the folder

I am not sure if that provides some more clarity for you?

Kind Regards,

Clare

Hi @clare.martin,
Thanks for the detailed feedback.
We will continue to test on our end and let you know if we make any progress.

Kind regards,
Micheál Power

1 Like