Can I acquire packages not in parallel when I have multiple "deploy a windows service" processes?

Advice
,
1

I have a project with multiple instances of the deploy a windows service process. The acquire packages process occurs before the deploy a windows service processes.

The deploy a windows service processes do not run in parallel, but the acquire packages process is running an upload package step for each of the deployment processes. The upload package steps are running in parallel. I can view this in the logs. Unfortunately, this is pushing the CPU usage of the deployment target towards 100%.

Is it possible to configure the acquire packages process to not run upload package steps in parallel?

3

Hi @richard.kneale

Firstly, welcome to the Octopus community!

Thank you for your question.

There are a couple of levers you can tweak to try and help with the CPU usage of your deployment targets.

The first is delta compression, it’s used to optimize the amount of data transferred between the Octopus Server and the target, but it can be CPU intensive. You can disable delta compression using a system variable called Octopus.Acquire.DeltaCompressionEnabled with a value of False. You can find all the details you need here in our documentation on that subject.

The second option directly relates to your question about the parallelism of package acquisition. Using another system variable called Octopus.Acquire.MaxParallelism you can limit the maximum number of packages concurrently deployed to targets. You set an integer value for this variable, the default is 10, so you may need to try out different values lower than that til you find the sweet spot for your deployments. You can find details on that variable and general performance hints for deployment targets here in our documentation.

I hope that helps, but please let me know if you have any follow-up questions!

Best,

4

Many thanks for your reply.

I have added the project variables and the packages are no longer acquired in parallel (Octopus.Acquire.MaxParallelism is 1), but I can still see the CPU being pushed towards 100% throughout the deployment.

Is there anything else I can try to mitigate this?

6

Hi @richard.kneale,

Thank you for getting back to us.

I hope you don’t mind, I have a few clarifying questions:

  • Can you confirm if the Octopus Server or the target machine is experiencing the high CPU usage?
  • What is the CPU usage normally when not running this particular deployment?
  • Can you share the system specifications of the affected machine? (CPU/RAM/HDD or SSD/Windows Edition)
  • For reference, can you share a raw task log from a recent deployment?
  • Additionally, can you generate a System Diagnostics Report and share that as well?

System Diagnostics Report
System Diagnostics Report1177×678 69.6 KB

Here is a secure link you may use to upload those files.

We look forward to hearing back from you.

Best Regards,
Donny Bell

7

Thank you for your reply.

It is the target machine that is experiencing high CPU usage.

The CPU is normally around 10%, but can momentarily be as high as 52% during normal operation when a deployment is not running.

The system specification is:
CPU - Processor Intel(R) Xeon(R) Gold 6136 CPU @ 3.00GHz, 2993 Mhz, 4 Core(s), 8 Logical Processor(s)
RAM - 175 GB
HDD
Windows Edition - Microsoft Windows Server 2012 Standard

I will send the raw task log. I do not have access to the System Diagnostics Report.

8

Hi @richard.kneale,

Thank you for getting back to me.

The specifications on that target machine certainly seem high enough to handle multiple package downloads at once. However, you may want to check that the Disk I/O isn’t maxing out since this system is running on a HDD. Also, are you able to confirm the exact process causing the CPU usage spike? One thing to consider is that sometimes we see antivirus/endpoint protections attempt to scan new files in realtime causing an increase in CPU usage.

Please send the raw task logs once you have a chance. Also, if you are able to capture a bit more information about what process is responsible for the CPU usage during the spike(s), that would be helpful as well.

Best Regards,
Donny Bell

9

I have now uploaded the raw task log at the secure link.

The CPU spikes throughout a deployment. It seems to be accompanied with a process called diawp.exe appearing many times and WMI Provider Host (as WmiPrvSE.exe). This is normally with terminated instances of powershell.exe, Tentacle.exe, or Calamari.exe.

10

Hi @richard.kneale,

Thank you for getting back to me.

diawp.exe and WmiPrvSE.exe are not processes we’ve heard any reports on regarding excess CPU usage during deployments.

Doing a bit of Googling, I was able to find a couple of relevant threads on these:

Can you confirm if this machine has antivirus/endpoint protection installed on it? If so, can you tell me what the specific product is?

Let me know at your earliest convenience.

Best Regards,
Donny Bell