Azure Deploy Takes Forever

Legacy Uncategorised
21

So you used a powershell script to delete a staging slot, and then re create it, then deploy?

22

@mevans yes that’s right in a first Octopus step we run custom PowerShell script to delete and recreate the Azure slot it’s deploying to. It takes a few seconds and massively speeds things up.

23

@mike, ok. Perhaps we will try that, the hanging and errors are painful.

Did you come across any other reasons that this process hangs or fails so often other than file checks?

24

@mevans no not that I can remember, it seemed to solve everything and deployments were quick afterwards, despite the large number of files.

25

@mike, i’ll give that a shot then.
Thanks so much for the help.

-Mark

26

@mevans Please note that you will lose any customizations you have done to that deployment slot. In our case we lost all our custom app settings and DB connection strings for the slot, and it also did not speed up the deployment in our case, but it did fix the hang. You may have to add back all the settings to the new slot via powershell if you use this workaround, which I personally don’t like, so I’m awaiting a proper fix to the problem instead.

27

@Mike, do you have a snippet of the powershell script you are running to delete/recreate the slots?

28

We are also seeing really long deployment times.

Looking at the raw log from the entire deployment process (single web app deployment and multiple server updates, windows services etc):

  • File has 4700 lines
  • Of them 2500 lines contain “Attribute ‘lastWriteTime’ equality changed to True” logged as verbose, the update web site step consists of 99% being these messages

I was wondering how much overhead does this verbose logging have? Is there a way to change logging level to be Info instead of Verbose? Or do I need to create a custom build of Calamari.

I’m not familiar how this logging is processed and can it be a bottleneck but it would be nice to try to have only info logged.

29

Same problem here :frowning:

Following the steps described on http://docs.octopusdeploy.com/display/OD/Using+Deployment+Slots+with+Azure+Web+Apps is not a solution, it’s ignoring the problem.

Have you planned any fix?

Thanks.

30

How many of you guys have process that are running in parallel? I’m testing it now but not doing any uploads in parallel.

31

We are running parallel processes, other ones than msdeploy are CPU intensive on target Tentacle machine, but they don’t eat server <-> tentacle bandwidth (very little logs).

32

The problem with the deployment slot solution is that it depends on a service plan of Standard or better. For some this might be problematic cost wise

33

Also, you have to recreate any settings you have stored on the deployment slot when deleting/creating slots. This added a lot of friction for us.

34

Hi Everyone,

Just thought I’d update you on this issue now that we’ve released that fix.

The core issue

The examples we have seen so far look look like threading dead-locks in WebDeploy. Microsoft.Web.Deployment has reportedly had issues with locking when using Checksum for file comparisons.

The solution

You can now use Timestamp instead of Checksum and see if this fixes the issue for your deployments. Other customers have reported success. Requires Octopus 3.3.3 or greater.

  • You will need to modify existing steps to use Timestamp (we do not automatically switch them over)
  • Any new Azure Web App steps you create will use Timestamp by default.

A note about timestamps

To get the most benefit of using timestamps for WebDeploy sync, start using zip packages instead of NuGet packages. We cannot maintain consistent timestamps when extracting NuGet packages - see this GitHub Issue for more information.

Plans for the future

We will continue to roll out further improvements to Azure Web App support, though we don’t have any fixed delivery timeframes for these:

  • ASP.NET Core 1.0 support
  • Improved Checksum support (bug fixes in WebDeploy)
  • Improved Proxy support (but fixes in WebDeploy)

All of these rely upon us upgrading to Microsoft.Web.Deploy.3.6.0 (track this GitHub Issue). We are working with Microsoft to update the NuGet package, otherwise we will have package and ship it ourselves. Release Notes for WebDeploy 3.6-beta

Hope that helps!
Mike

35

We use TeamCity w/ OctoPack. Are you recommending to still use that and somehow change it from using NuGet to Zip Packages?

Details?

36

Hi Mark,

We haven’t made OctoPack capable of producing zip packages yet, though we’d like to. How you proceed really depends on your situation. The benefit you get from OctoPack is that it is “smart” and can guess which files are required for your web app (like content files).

Right now, building zip packages means you need to ensure all of the files required for your site are published into a folder you can zip into a package. Take a look at an example on our blog. (This blog describes using the latest version of the Octopus TeamCity plugin for ASP.NET Core apps, but works the same for packaging and pushing any application from TeamCity).

If you’re relying on the “smarts” in OctoPack, the best thing would be to keep using OctoPack, but switch to Timestamp and see if your deployments become more reliable. Hopefully in the near future OctoPack will be able to produce zip packages.

Hope that helps!
Mike

37

Cool. Yes we are relying on the magic. One less thing we need to manage. Will likely give these suggestions a try because the deploy times are horrid.

Thank you guys so much for working on this.

38

Hi!

We also have been experiencing these issues with frozen deployments to Azure, but for now seems like upgrading to 3.3.3 and switching to timestamp comparisons probably helped.
Probably - because we have run only one deployment to Azure yet and that ran twice as fast as with old checksum setting. But hopefully that’s generic solution for the problem. Btw, we’re also using octopack in TeamCity and dont use zips instead nupkg yet.

39

Hi Andris,

Thanks for letting us know. To be clear, WebDeploy still compares the Timestamps for each and every file in your package with what’s currently deployed to the Web App - so you still pay the same performance penalty for checking. But if the Timestamp matches on 99% of your files, WebDeploy will only sync 1% of those files which could lead to a good performance improvement.

Hope that helps!
Mike

40

Is there a checkbox or something that we can use to set it to use Timestamps?
I just upgraded to 3.3.5 and I don’t see this option…