Azure Deploy Takes Forever

luke_barnett · 27 September 2015 21:42

My azure deploys are taking far longer than they should be, and it looks like, based on the logs, that it is updating files based on last write time? Which means it’s updating everything, every time.

The confusing part is that based on the Calamari source code it’s supposed to be using checksum?

github.com

OctopusDeploy/Calamari/blob/ce3b69e94b60c8c73619bc584eca52e11c68930a/source/Calamari.Azure/Deployment/Conventions/AzureWebAppConvention.cs#L91


    options.Trace += (sender, eventArgs) => LogDeploymentEvent(eventArgs);


    return options;
}


private static DeploymentSyncOptions DeploymentSyncOptions(VariableDictionary variables)
{
    var syncOptions = new DeploymentSyncOptions
    {
        WhatIf = false,
        UseChecksum = true,
        DoNotDelete = !variables.GetFlag(SpecialVariables.Action.Azure.RemoveAdditionalFiles, false)
    };


    if (variables.GetFlag(SpecialVariables.Action.Azure.PreserveAppData, false))
    {
       syncOptions.Rules.Add(new DeploymentSkipRule("SkipDeleteDataFiles", "Delete", "filePath", "\\\\App_Data\\\\.*", null)); 
       syncOptions.Rules.Add(new DeploymentSkipRule("SkipDeleteDataDir", "Delete", "dirPath", "\\\\App_Data(\\\\.*|$)", null)); 
    }


    return syncOptions;

Is there a bug in Calamri? and/or whatever library it is using underneath for msdeploy?

Michael_Richardson · 28 September 2015 03:14

Hi Luke,

What makes you think it is updating based on last-write-time?

In my testing, when I deploy a package to an Azure Web Site containing files where nothing has changed except the timestamp, I see lots of entries in the log similar to:
Attribute 'lastWriteTime' equality changed to True when comparing filePath (octowebapptest\PostDeploy.ps1) to C:\Octopus\Work\20150928023413-4\PostDeploy.ps1 because of rule IgnoreFileLastWriteTime

And at the end:
Successfully deployed to Azure. 0 objects added. 0 objects updated. 0 objects deleted.

This indicates to me that it is ignoring the the file timestamp.

And if I then modify one of the files, I see:
Source filePath (C:\Octopus\Work\20150928030636-10\Web.config) does not match destination (octowebapptest\Web.config) differing in attributes (size['314','315'],lastWriteTime['09/28/2015 03:06:43','09/17/2015 12:18:30']). Update pending.

And at the end:
Successfully deployed to Azure. 0 objects added. 1 objects updated. 0 objects deleted.

As you probably saw from the Calamari code, we rely on Microsoft Web Deploy for the synchronization implementation.

Regards,
Michael

luke_barnett · 30 September 2015 18:56

Ahhh, that explains a lot. I was missing that line and seemingly misreading the others

Sorry I am mistaken.

Michael_Richardson · 1 October 2015 22:51

No problem at all.

Happy deployments!

Michael

anjdreas · 4 December 2015 11:31

I would like to chime in that it’s still grudgingly slow. It’s possible that this is entirely a Web Deploy issue, or an Azure issue, but it’s spending minutes trying to compare these write times. Is there any optimization that can be done? It seems to just sequentially go through each file and check with the server if that file is touched, rather than uploading a large bulk of file information to compare on the server side and return back.

Mike_Carlisle · 5 December 2015 12:05

I’m also having an issue with this. It seems like it for large websites it’s quicker to deploy to a clean instance and add files, then crawl through looking for updates.

Vanessa_Love · 10 December 2015 11:10

Hi Folks,

We are playing around with an idea to add a variable to the project that allows you to instead of sync the files as it currently does, just purge and install.
We think in these cases it will be the best solution for a large amount of files as the webdeploy sync plus Azure is slow.

I will update when we have an issue to track.

Vanessa

James_Denning · 10 December 2015 12:14

Would definitely like the option to overwrite everything when deploying a web app to Azure. However do have some additional observations:

This happens to us on only one of our deployment types where we are deploying to a deployment slot that has previously been swapped with another slot. When we deploy to a slot that never gets swapped the deployments are quicker and rarely fail…
Sometimes the deployments never succeed, sometimes they take forever
Restarting the slot web apps (the one we’re trying to deploy to and the one we swap with), and cancelling the hanging release alleviates the problem for a day or two
Is it not possible to have an Octopus tentacle-like agent running as an azure plufg in or something similar so the deployment is not happening over the wire?

Vanessa_Love · 15 December 2015 05:52

Hi All,

So we investigated everything we could around what WebDeploy has available and there is no way for us to avoid the check-sum processing during sync. So we determined way to best use Octopus to speed up these deployments was to provide some documentation on using Deployment Slots to implement Blue-Green deployments. As most of you have noted deploying the first time is really fast, this mimics that each time.

The walk-through can be found here: http://docs.octopusdeploy.com/display/OD/Using+Deployment+Slots+with+Azure+Web+Apps

@James for your specific Q about Tentacle. Not at the moment, we are investigating how we can be a partner to have an agent on Azure similar to dropbox, Git etc.

Vanessa

James_Denning · 15 December 2015 09:01

Hi Vanessa
Many thanks for your reply, much appreciated, I’ll check out the blog article, make any changes based on recommendations and let you know if there are any further issues.
James

anjdreas · 15 December 2015 09:35

I just tried this recommendation. It does solve the part of not having to check for modified files, but it’s still slow due to having to upload a lot of unnecessary files that did not change. In my case the site is 200MB and that takes a lot of time to upload via web deploy. Based on internet speed tests it should take about 2 minutes to upload this with our uplink, so there is clearly some overhead or reduction towards Azure web deploy even though it is located in our region.

With checksum/last modified checking it took in the range of 3-6 minutes. When recreating deployment slot first it now takes 4 minutes on each of the first two attempts. I’ll keep running with this configuration to monitor over some time, but I’m unfortunately not seeing an improvement with this method.

anjdreas · 16 December 2015 12:49

I just realized you lose any changes you have made to that deployment slot, since you are effectively deleting it and recreating it. Obvious in hindsight, but I lost all my custom domain and application settings that you may have to re-add as part of the automation script if you go this route. I’ve reverted back for now as this method was not an improvement for our project.

Robin · 18 December 2015 09:33

We tried this solution aswell and it increased the deploy time with about 2 minutes, We are running a nodeJS project and have around 8000 files that needs to be added, Is there any other way to solve this?

anjdreas · 5 January 2016 13:42

I’m trying to find a place to discuss Web Deploy improvements if that is the core of the slow timestamp comparison in Octopus, but I can’t find it on github or Microsoft Connect or anywhere else. Any ideas?

anjdreas · 27 January 2016 11:19

I need to bump this. We are consistently having trouble deploying our web app, and typically have to Try Again 2-3 times or more. A successful deployment takes about 4-6 minutes, and it hangs forever on failed attempts.
I can provide logs and create a new thread if necessary, just tell me what to do. The logs in the admin panel just seems hang on an arbitrary place in the timestamp checking. This is a 200+ MB deployment so this checking takes around 2-3 minutes alone. I suspect there is a deadlock or timeout issue regarding this in Octopus Deploy, instead of in Web Deploy. I haven’t tried Web Deploy much lately so that is another path we could investigate if it helps you.

Thanks.

Michael_Richardson · 29 January 2016 05:58

Yes, could you create a new ticket for the hanging/failed deployments please?

And if you could upload the task log of a failed deployment here (and mention it in the new ticket), we’ll see if we can figure out what is going wrong.

Regards,
Michael

anjdreas · 29 January 2016 08:48

Created new ticket here: http://help.octopusdeploy.com/discussions/problems/44216-frequently-hang-on-deploying-large-azure-web-app

mohamed_abed · 1 February 2016 03:44

Any news regarding this annoying issue?

mevans · 23 February 2016 19:07

@michael Same issues for us. We are constantly having to retry. Same errors. Deployments are not reliable because of this.

Do you have any updates ? Is there something we could provide you to help get this issues resolved?

Mike_Carlisle · 23 February 2016 19:23

For us deleting and recreating a slot resolved the issue. It’s slow because webdeploy compares file for changes one at a time. Without any files to compare it’s quick and it’s just writing new files to disk.