File copying too slow on tentacle server

Octopus Server
, , ,
1

Hello,

We are using 3.2.4 version currently. We are facing slowness for copying files for our web deployment (The purge and copy step takes about 8 to 10 mins ). The nuget package and the folder its coping are on the same tentacle server so there is no role coping to a shared network path.

attached is the deployment log.

deployment-log (29.6 KB)

Please suggest what could be the reason.

Thanks,
Rasmita

3

Hi Rasmita,

Thanks for getting in touch!

At that point all that Octopus is doing is interacting with the Windows file structure to perform the delete and copy so the issue is likely to be something environmental.
Can I ask what the size of the package being deployed is?
And is this something that has just started occurring or has it always been this slow?

A common cause of this would be an anti-virus that is scanning each file as it is copied causing the disk I/O to reach maximum levels and slowing down the copying.

Best regards,
Paul

4

Hi Paul,

Our package size is roughly 420 MB. The issue is happening from quite a long time and our package size constantly grows slightly with every release.

We checked our I/O activity and anti virus scanning and did not find any issue with them.

One more thing to add the copy process takes more time when we try to perform multiple deployment same time on the same server.

Thanks,
Rasmita

5

Hi Rasmita,

It would be useful if we could test whether the issue persists outside of Octopus. To do this could you try manually copying the contents of the package from the source location to the destination?
e.g.
Copy the files from D:\Octopus\Applications\LIFELINK-Training\TransplantConnect.Web.UI\2019.4.0.925_1
to D:\IIS\Training Sites\LIFELINK\WebUI

To avoid impacting a live application you could alter the destination folder slightly, perhaps .\WebUITest

If you can do this and let me know if it takes about the same time or if it is significantly quicker.

Regards,
Paul

6

Hi Paul,

I took some time to do some analysis on our end. so i tried different ways

  1. i compared the Octopus deployment Purge and copy file timing (which is appox 7 to 8 mins now) with manual copy to a path on the production server as you suggest previously. surprisingly the manual copy ( id do it through a powershell script to be accurate on time) it took ~40sec average to copy the files locally on the same server.

  2. i tried the copy process by zipping and unzipping the release folder on to a different path locally, this also was faster as compared the octopus deployment copy file timing.

I performed this test against our qa and 2 production servers for 4 different release.
deploymenttiming.xlsx (9.2 KB)

This number still surprises me why octopus is taking so much to just copy files locally, does it use any special zipping/unzipping. I also saw the deployment logs which it writes , does this have any impact on the copy process.

I do not see why we wont perform the purge and copy process through a power shell template and give it a try and see if it helps. Please suggest

Thanks,
Rasmita

7

Hi Rasmita,

Thanks for carrying out those tests, it does rule out external factors fairly well.
As you are running an older version of Octopus, an upgrade would likely see some improvement to these timings.
In the meantime though, adding some Powershell steps to carry out these functions may help. I would suggest splitting them at first, and trying with just a single script to perform the purge operation, then perform the deploy and see if that is enough to bring the time down.
Then add the copy on if needed afterwards.

Regards,
Paul