Acquire Packages Takes Long Time

Hi we have one of our tentacles (targets) taking a long time to run deployments for example going from a few minutes to 50 minutes.
There have been no changes to this server - but there may have been network changes eg. domain controller. We have also noticed on some of our non octopus servers virus scanners interrupting processes which run in the C Users Temp Directory - perhaps scanning in this directory could be causing delays. Our other tentacles (targets) which are not being delayed like this tentacle is and we have rebooted the server and reinstalled the tentacle on this server and the problem persists.

Below I show a screen shot of how long the problem target is taking now vs before. The screenshot is from a task triggered on schedule which runs each day and installs the exact same package.

Any hints on what could be causing the delay for this - also note many steps are taking a long time not just acquire packages.

Good morning @Adam.Archibald,

Thank you for contacting Octopus Support and welcome to the forums!

I am sorry to hear one of your targets is struggling to acquire the package, 40 minutes to acquire a package is definitely not the behaviour you should be seeing!

Thank you for the comparison logs, they were really helpful in seeing the difference in wait times there and the fact you are using the same shared storage for both of those packages.

One thing that does come to mind is Antivirus, where I used to work some of the machines had ‘On Access Scanning’ turned on which meant a 2 minute software install on those machines took over 30 minutes as on access scanning will take any file put onto that machine and scan every bit of it, so your .nupkg you are trying to put on that machine will be scanned by your AV before it can be placed on that machine.

The only way to fully rule out AV would be to temporarily disable it on that specific machine and re-run the deployment, if it only takes a few minuities to acquire the package you know you need to take a look at your AV and either whitelist a specific folder where the packages are coming from, or turn off on access scanning (you would need to speak to your security department regarding this though).

On Access scanning for McAfee ePolicy Orchestrator (ePO) - which is what I used in my old company - was a bulk policy you could set for each agent on a machine but it could also be set individually on each agent so it may be that specific machine has it turned on by accident or that AV agent has not pulled down the AV policy from the main AV server.

Would you be able to temporarily turn off AV on that target and re-run the deployment to see if that fixes the issue?

I look forward to hearing from you,
Kind Regards,
Clare

Thanks Clare will do - yes we sometimes see differences between machines in virus scanning behaviour. Just a couple of other items to help:
i) Do you know how we can up the logging from verbose to see more detail
ii) Do you know if we can adjust through config the Temp Directory to somewhere else (we have in the past seen issues with the C Users Directory of the service running

The reason I ask is often the information security people will not turn off virus scanning so sometimes
we have to try other ways.

Thanks,

Adam

TempDirectory: C:\Users*service_account_user*\AppData\Local\Temp\

Hey @Adam.Archibald,

If you can get them to temporarily disable it just so you can test that one deployment that would be beneficial. Mainly because we have had customers in the past whitelist Octopus folders but once they have temporarily disabled AV the deployment started working, which then highlighted a folder they needed to whitelist so disabling it is the only way to fully rule out AV in situations such as this. Though I do fully appreciate there are security implications to be considered when doing so.

Do you know how we can up the logging from verbose to see more detail

If you mean the deployment logs we do have a page in our documentation here which does go through changing the log levels for Octopus Server and tentacle but that does not include deployment logs so there is no way to change a deployment log from its current logging level of Verbose.

You could try setting the tentacle log to a higher log level and see what that outputs if anything but usually the tentacle log doesnt show deployment processes it just shows service and connection level outputs.

You could also try checking out the AV logs on that machine or in event viewer to see if there are any indications that AV is scanning a file from Octopus.

Do you know if we can adjust through config the Temp Directory to somewhere else (we have in the past seen issues with the C Users Directory of the service running

Unfortunately, the temp directory is dictated by the OS so Octopus has no control over where that lives, it will use the service account that Octopus is running under and use that temp directory to store temp files in before they are transferred to the Octopus Work directory.

We do have a document on the folders you need to whitelist in Octopus but that does use the default drives so you would need to change those as appropriate for your instance. So that may help but if you have on access scanning on it will still scan the .nupkg before it gets put in a work folder so that would need to be turned off anyway in favour of on demand scanning.

This does seem environmental on that one machine from the way you have described it so it is doubtful it is a network issue or a change in network such as a DC change. The fact your other tentacles are working (assuming they are on the same subnet with the same external security policies in place) does point me to that one specific machine.

Hopefully we can get this sorted for you but if you send over that whitelist page to your AV team and ensure all of those folders have been whitelisted and see if they can temporarily disable AV on that one machine that will officially rule out AV and we can focus on some other things.

In the meantime, what are the RAM and CPU usage of the machine like at the time of the deployment, are they near max capacity or do they look ok?

Kind Regards,
Clare

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.